content/glyph/book/extending/internals.html
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 |
-----
permalink: t_31
title: Glyph – A quick look at Glyph's internals
type: page
-----
<nav class="navigation"><a href="/glyph/book/stats/links.html">← Link Statistics</a> | <a href="/glyph/book/index.html">Contents</a> | <a href="/glyph/book/extending/macro_def.html">Defining Custom Macros →</a></nav>
<p>If you plan on extending Glyph, knowing how it works inside helps. It is not mandatory by any means, but it definitely helps, especially when creating complex macros.</p>
<p>What happens behind the scenes when you call <code>glyph compile</code>? Glyph's code is parsed, analyzed and then translated into text, and here's how:</p>
<figure><img src="/glyph/book/images/glyph/document_generation.png" /><figcaption>A sequence diagram for document generation</figcaption></figure>
<p>From the diagram, it is possible to divide the document generation process into three phases:</p>
<ul>
<li>The <em>Parsing Phase</em> starts when a chunk of Glyph code is passed (by the <code>generate:document</code> Rake task, for example) to a <a href="http://rubydoc.info/gems/glyph/Glyph/Interpreter"><code>Glyph::Interpreter</code></a>. The interpreter initializes a <a href="http://rubydoc.info/gems/glyph/Glyph/Parser"><code>Glyph::Parser</code></a> that parses the code and returns an <em>Abstract Syntax Tree</em> (<span class="caps">AST</span>) of <a href="http://rubydoc.info/gems/glyph/Glyph/SyntaxNode"><code>Glyph::SyntaxNode</code></a> objects.</li>
<li>The <em>Analysis Phase</em> (Processing) starts when the interpreter method calls the <code>analyze</code> method, instantiating a new <a href="http://rubydoc.info/gems/glyph/Glyph/Document"><code>Glyph::Document</code></a>. The <code>Glyph::Document</code> object evaluates the <span class="caps">AST</span> expanding all macro nodesth (that’s when macros are executed) and generates string.</li>
<li>The <em>Finalization Phase</em> (Post-Processing) starts when the interpreter calls the <code>finalyze</code> method, causing the <code>Glyph::Document</code> object to perform a series of finalizations on the string obtained after analysis, i.e. it replaces escape sequences and placeholders.</li>
</ul>
<section class="section">
<header><h1 id="h_83" class="toc">Example: A short note</h1></header>
<p>As an example, consider the following Glyph code:</p>
<div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>fmi[something|#test]
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>...
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>section[
<span class="line-numbers"><a href="#n4" name="n4">4</a></span> @title[Test Section]
<span class="line-numbers"><a href="#n5" name="n5">5</a></span> @id[test]
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>...
<span class="line-numbers"><a href="#n7" name="n7">7</a></span>]</pre></div>
</div>
<p>This simple snippet uses the <a href="/glyph/book/macros/macros_inline.html#m_fmi"><code>fmi</code></a> macro to link to a section later on in the document. When parsed, the produced AST is the following:</p>
<div class="CodeRay">
<div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>{<span class="symbol">:name</span>=><span class="symbol"><span class="symbol">:</span><span class="delimiter">"</span><span class="content">--</span><span class="delimiter">"</span></span>}
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span> {<span class="symbol">:name</span>=><span class="symbol">:fmi</span>, <span class="symbol">:escape</span>=><span class="predefined-constant">false</span>}
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span> {<span class="symbol">:name</span>=><span class="symbol"><span class="symbol">:</span><span class="delimiter">"</span><span class="content">0</span><span class="delimiter">"</span></span>}
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="content">something</span><span class="delimiter">"</span></span>}
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span> {<span class="symbol">:name</span>=><span class="symbol"><span class="symbol">:</span><span class="delimiter">"</span><span class="content">1</span><span class="delimiter">"</span></span>}
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="content">#test</span><span class="delimiter">"</span></span>}
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="char">\n</span><span class="delimiter">"</span></span>}
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="char">\</span><span class="content">[</span><span class="delimiter">"</span></span>, <span class="symbol">:escaped</span>=><span class="predefined-constant">true</span>}
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="content">...</span><span class="delimiter">"</span></span>}
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="char">\</span><span class="content">]</span><span class="delimiter">"</span></span>, <span class="symbol">:escaped</span>=><span class="predefined-constant">true</span>}
<span class="line-numbers"><a href="#n11" name="n11">11</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="char">\n</span><span class="delimiter">"</span></span>}
<span class="line-numbers"><a href="#n12" name="n12">12</a></span> {<span class="symbol">:name</span>=><span class="symbol">:section</span>, <span class="symbol">:escape</span>=><span class="predefined-constant">false</span>}
<span class="line-numbers"><a href="#n13" name="n13">13</a></span> {<span class="symbol">:name</span>=><span class="symbol"><span class="symbol">:</span><span class="delimiter">"</span><span class="content">0</span><span class="delimiter">"</span></span>}
<span class="line-numbers"><a href="#n14" name="n14">14</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="char">\n</span><span class="char">\t</span><span class="delimiter">"</span></span>}
<span class="line-numbers"><a href="#n15" name="n15">15</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="char">\n</span><span class="char">\t</span><span class="delimiter">"</span></span>}
<span class="line-numbers"><a href="#n16" name="n16">16</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="char">\n</span><span class="delimiter">"</span></span>}
<span class="line-numbers"><a href="#n17" name="n17">17</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="char">\</span><span class="content">[</span><span class="delimiter">"</span></span>, <span class="symbol">:escaped</span>=><span class="predefined-constant">true</span>}
<span class="line-numbers"><a href="#n18" name="n18">18</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="content">...</span><span class="delimiter">"</span></span>}
<span class="line-numbers"><a href="#n19" name="n19">19</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="char">\</span><span class="content">]</span><span class="delimiter">"</span></span>, <span class="symbol">:escaped</span>=><span class="predefined-constant">true</span>}
<span class="line-numbers"><strong><a href="#n20" name="n20">20</a></strong></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="char">\n</span><span class="delimiter">"</span></span>}
<span class="line-numbers"><a href="#n21" name="n21">21</a></span> {<span class="symbol">:name</span>=><span class="symbol">:title</span>, <span class="symbol">:escape</span>=><span class="predefined-constant">false</span>}
<span class="line-numbers"><a href="#n22" name="n22">22</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="content">Test Section</span><span class="delimiter">"</span></span>}
<span class="line-numbers"><a href="#n23" name="n23">23</a></span> {<span class="symbol">:name</span>=><span class="symbol">:id</span>, <span class="symbol">:escape</span>=><span class="predefined-constant">false</span>}
<span class="line-numbers"><a href="#n24" name="n24">24</a></span> {<span class="symbol">:value</span>=><span class="string"><span class="delimiter">"</span><span class="content">test</span><span class="delimiter">"</span></span>}</pre></div>
</div>
<p>This output is produced by calling the <code>inspect</code> method on the AST. Each <a href="http://rubydoc.info/gems/glyph/Glyph/SyntaxNode"><code>Glyph::SyntaxNode</code></a> object in the tree is basically an ordinary Glyph Hash with a parent and 0 or more chidren, so the code snippets above shows how the syntax nodes are nested.</p>
<p>The AST contains information about macro, parameter and attribute names, and escaping, and raw text values (the nodes without a <code>:name</code> key), but nothing more.</p>
<p>When the AST is analyzed, the resulting textual output is the following:</p>
<div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="tag"><span</span> <span class="attribute-name">class</span>=<span class="string"><span class="delimiter">"</span><span class="content">fmi</span><span class="delimiter">"</span></span><span class="tag">></span>for more information on something, see ‡‡‡‡‡PLACEHOLDER ¤ 1‡‡‡‡‡
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="tag"></span></span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>\/[...\/]
<span class="line-numbers"><a href="#n4" name="n4">4</a></span><span class="tag"><div</span> <span class="attribute-name">class</span>=<span class="string"><span class="delimiter">"</span><span class="content">section</span><span class="delimiter">"</span></span><span class="tag">></span>
<span class="line-numbers"><a href="#n5" name="n5">5</a></span><span class="tag"><h2</span> <span class="attribute-name">id</span>=<span class="string"><span class="delimiter">"</span><span class="content">test</span><span class="delimiter">"</span></span><span class="tag">></span>Test Section<span class="tag"></h2></span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>\/[...\/]
<span class="line-numbers"><a href="#n7" name="n7">7</a></span>
<span class="line-numbers"><a href="#n8" name="n8">8</a></span><span class="tag"></div></span></pre></div>
</div>
<p>This looks almost perfect, except that:</p>
<ul>
<li>There's a nasty placeholder instead of a link: this is due to the fact that when the link is processed, there is no <code>#text</code> anchor in the document, but there may be one afterwards (and there will be).</li>
<li>There are some escaped brackets.</li>
</ul>
<p>Finally, when the document is finalized, placeholders and escape sequences are removed and the final result is the following:</p>
<div class="CodeRay">
<div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="tag"><span</span> <span class="attribute-name">class</span>=<span class="string"><span class="delimiter">"</span><span class="content">fmi</span><span class="delimiter">"</span></span><span class="tag">></span>for more information on something,
<span class="line-numbers"><a href="#n2" name="n2">2</a></span> see <span class="tag"><a</span> <span class="attribute-name">href</span>=<span class="string"><span class="delimiter">"</span><span class="content">#test</span><span class="delimiter">"</span></span><span class="tag">></span>Test Section<span class="tag"></a></span><span class="tag"></span></span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>[...]
<span class="line-numbers"><a href="#n4" name="n4">4</a></span><span class="tag"><div</span> <span class="attribute-name">class</span>=<span class="string"><span class="delimiter">"</span><span class="content">section</span><span class="delimiter">"</span></span><span class="tag">></span>
<span class="line-numbers"><a href="#n5" name="n5">5</a></span><span class="tag"><h2</span> <span class="attribute-name">id</span>=<span class="string"><span class="delimiter">"</span><span class="content">test</span><span class="delimiter">"</span></span><span class="tag">></span>Test Section<span class="tag"></h2></span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>[...]
<span class="line-numbers"><a href="#n7" name="n7">7</a></span>
<span class="line-numbers"><a href="#n8" name="n8">8</a></span><span class="tag"></div></span></pre></div>
</div>
</section>
<nav class="navigation"><a href="/glyph/book/stats/links.html">← Link Statistics</a> | <a href="/glyph/book/index.html">Contents</a> | <a href="/glyph/book/extending/macro_def.html">Defining Custom Macros →</a></nav>
|