all repos — h3rald @ 7c392647836be1fb022aea53f448c3c8508ae3d3

The sources of https://h3rald.com

content/glyph/book/extending/internals.html

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
-----
permalink: t_31
title: Glyph – A quick look at Glyph's internals
type: page
-----
<nav class="navigation"><a href="/glyph/book/stats/links.html">← Link Statistics</a> | <a href="/glyph/book/index.html">Contents</a> | <a href="/glyph/book/extending/macro_def.html">Defining Custom Macros →</a></nav>
  <p>If you plan on extending Glyph, knowing how it works inside helps. It is not mandatory by any means, but it definitely helps, especially when creating complex macros.</p>
  <p>What happens behind the scenes when you call <code>glyph compile</code>? Glyph's code is parsed, analyzed and then translated into text, and here's how:</p>
  <figure><img src="/glyph/book/images/glyph/document_generation.png" /><figcaption>A sequence diagram for document generation</figcaption></figure>
  <p>From the diagram, it is possible to divide the document generation process into three phases:</p>
<ul>
	<li>The <em>Parsing Phase</em> starts when a chunk of Glyph code is passed (by the <code>generate:document</code> Rake task, for example) to a <a href="http://rubydoc.info/gems/glyph/Glyph/Interpreter"><code>Glyph::Interpreter</code></a>. The interpreter initializes a <a href="http://rubydoc.info/gems/glyph/Glyph/Parser"><code>Glyph::Parser</code></a> that parses the code and returns an <em>Abstract Syntax Tree</em> (<span class="caps">AST</span>) of <a href="http://rubydoc.info/gems/glyph/Glyph/SyntaxNode"><code>Glyph::SyntaxNode</code></a> objects.</li>
	<li>The <em>Analysis Phase</em> (Processing) starts when the interpreter method calls the <code>analyze</code> method, instantiating a new <a href="http://rubydoc.info/gems/glyph/Glyph/Document"><code>Glyph::Document</code></a>. The <code>Glyph::Document</code> object evaluates the <span class="caps">AST</span> expanding all macro nodesth (that&#8217;s when macros are executed) and generates string.</li>
	<li>The <em>Finalization Phase</em> (Post-Processing) starts when the interpreter calls the <code>finalyze</code> method, causing the <code>Glyph::Document</code> object to perform a series of finalizations on the string obtained after analysis, i.e. it replaces escape sequences and placeholders.</li>
</ul>
  <section class="section">
<header><h1 id="h_83" class="toc">Example: A short note</h1></header>
<p>As an example, consider the following Glyph code:</p>
    <div class="CodeRay">
  <div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span>fmi[something|#test]
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>...
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>section[
<span class="line-numbers"><a href="#n4" name="n4">4</a></span>  @title[Test Section]
<span class="line-numbers"><a href="#n5" name="n5">5</a></span>  @id[test]
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>...
<span class="line-numbers"><a href="#n7" name="n7">7</a></span>]</pre></div>
</div>

    <p>This simple snippet uses the <a href="/glyph/book/macros/macros_inline.html#m_fmi"><code>fmi</code></a> macro to link to a section later on in the document. When parsed, the produced AST is the following:</p>
    <div class="CodeRay">
  <div class="code"><pre><span class="line-numbers"> <a href="#n1" name="n1">1</a></span>{<span class="symbol">:name</span>=&gt;<span class="symbol"><span class="symbol">:</span><span class="delimiter">&quot;</span><span class="content">--</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"> <a href="#n2" name="n2">2</a></span>  {<span class="symbol">:name</span>=&gt;<span class="symbol">:fmi</span>, <span class="symbol">:escape</span>=&gt;<span class="predefined-constant">false</span>}
<span class="line-numbers"> <a href="#n3" name="n3">3</a></span>    {<span class="symbol">:name</span>=&gt;<span class="symbol"><span class="symbol">:</span><span class="delimiter">&quot;</span><span class="content">0</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"> <a href="#n4" name="n4">4</a></span>      {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="content">something</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"> <a href="#n5" name="n5">5</a></span>    {<span class="symbol">:name</span>=&gt;<span class="symbol"><span class="symbol">:</span><span class="delimiter">&quot;</span><span class="content">1</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"> <a href="#n6" name="n6">6</a></span>      {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="content">#test</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"> <a href="#n7" name="n7">7</a></span>  {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="char">\n</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"> <a href="#n8" name="n8">8</a></span>  {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="char">\</span><span class="content">[</span><span class="delimiter">&quot;</span></span>, <span class="symbol">:escaped</span>=&gt;<span class="predefined-constant">true</span>}
<span class="line-numbers"> <a href="#n9" name="n9">9</a></span>  {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="content">...</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"><strong><a href="#n10" name="n10">10</a></strong></span>  {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="char">\</span><span class="content">]</span><span class="delimiter">&quot;</span></span>, <span class="symbol">:escaped</span>=&gt;<span class="predefined-constant">true</span>}
<span class="line-numbers"><a href="#n11" name="n11">11</a></span>  {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="char">\n</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"><a href="#n12" name="n12">12</a></span>  {<span class="symbol">:name</span>=&gt;<span class="symbol">:section</span>, <span class="symbol">:escape</span>=&gt;<span class="predefined-constant">false</span>}
<span class="line-numbers"><a href="#n13" name="n13">13</a></span>    {<span class="symbol">:name</span>=&gt;<span class="symbol"><span class="symbol">:</span><span class="delimiter">&quot;</span><span class="content">0</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"><a href="#n14" name="n14">14</a></span>      {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="char">\n</span><span class="char">\t</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"><a href="#n15" name="n15">15</a></span>      {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="char">\n</span><span class="char">\t</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"><a href="#n16" name="n16">16</a></span>      {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="char">\n</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"><a href="#n17" name="n17">17</a></span>      {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="char">\</span><span class="content">[</span><span class="delimiter">&quot;</span></span>, <span class="symbol">:escaped</span>=&gt;<span class="predefined-constant">true</span>}
<span class="line-numbers"><a href="#n18" name="n18">18</a></span>      {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="content">...</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"><a href="#n19" name="n19">19</a></span>      {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="char">\</span><span class="content">]</span><span class="delimiter">&quot;</span></span>, <span class="symbol">:escaped</span>=&gt;<span class="predefined-constant">true</span>}
<span class="line-numbers"><strong><a href="#n20" name="n20">20</a></strong></span>      {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="char">\n</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"><a href="#n21" name="n21">21</a></span>    {<span class="symbol">:name</span>=&gt;<span class="symbol">:title</span>, <span class="symbol">:escape</span>=&gt;<span class="predefined-constant">false</span>}
<span class="line-numbers"><a href="#n22" name="n22">22</a></span>      {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="content">Test Section</span><span class="delimiter">&quot;</span></span>}
<span class="line-numbers"><a href="#n23" name="n23">23</a></span>    {<span class="symbol">:name</span>=&gt;<span class="symbol">:id</span>, <span class="symbol">:escape</span>=&gt;<span class="predefined-constant">false</span>}
<span class="line-numbers"><a href="#n24" name="n24">24</a></span>      {<span class="symbol">:value</span>=&gt;<span class="string"><span class="delimiter">&quot;</span><span class="content">test</span><span class="delimiter">&quot;</span></span>}</pre></div>
</div>

    <p>This output is produced by calling the <code>inspect</code> method on the AST. Each <a href="http://rubydoc.info/gems/glyph/Glyph/SyntaxNode"><code>Glyph::SyntaxNode</code></a> object in the tree is basically an ordinary Glyph Hash with a parent and 0 or more chidren, so the code snippets above shows how the syntax nodes are nested.</p>
    <p>The AST contains information about macro, parameter and attribute names, and escaping, and raw text values (the nodes without a <code>:name</code> key), but nothing more.</p>
    <p>When the AST is analyzed, the resulting textual output is the following:</p>
    <div class="CodeRay">
  <div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="tag">&lt;span</span> <span class="attribute-name">class</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">fmi</span><span class="delimiter">&quot;</span></span><span class="tag">&gt;</span>for more information on something, see ‡‡‡‡‡PLACEHOLDER ¤ 1‡‡‡‡‡
<span class="line-numbers"><a href="#n2" name="n2">2</a></span><span class="tag">&lt;/span&gt;</span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>\/[...\/]
<span class="line-numbers"><a href="#n4" name="n4">4</a></span><span class="tag">&lt;div</span> <span class="attribute-name">class</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">section</span><span class="delimiter">&quot;</span></span><span class="tag">&gt;</span>
<span class="line-numbers"><a href="#n5" name="n5">5</a></span><span class="tag">&lt;h2</span> <span class="attribute-name">id</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">test</span><span class="delimiter">&quot;</span></span><span class="tag">&gt;</span>Test Section<span class="tag">&lt;/h2&gt;</span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>\/[...\/]
<span class="line-numbers"><a href="#n7" name="n7">7</a></span>
<span class="line-numbers"><a href="#n8" name="n8">8</a></span><span class="tag">&lt;/div&gt;</span></pre></div>
</div>

    <p>This looks almost perfect, except that:</p>
    <ul>
      <li>There's a nasty placeholder instead of a link: this is due to the fact that when the link is processed, there is no <code>#text</code> anchor in the document, but there may be one afterwards (and there will be).</li>
      <li>There are some escaped brackets.</li>
    </ul>
    <p>Finally, when the document is finalized, placeholders and escape sequences are removed and the final result is the following:</p>
    <div class="CodeRay">
  <div class="code"><pre><span class="line-numbers"><a href="#n1" name="n1">1</a></span><span class="tag">&lt;span</span> <span class="attribute-name">class</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">fmi</span><span class="delimiter">&quot;</span></span><span class="tag">&gt;</span>for more information on something, 
<span class="line-numbers"><a href="#n2" name="n2">2</a></span>  see <span class="tag">&lt;a</span> <span class="attribute-name">href</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">#test</span><span class="delimiter">&quot;</span></span><span class="tag">&gt;</span>Test Section<span class="tag">&lt;/a&gt;</span><span class="tag">&lt;/span&gt;</span>
<span class="line-numbers"><a href="#n3" name="n3">3</a></span>[...]
<span class="line-numbers"><a href="#n4" name="n4">4</a></span><span class="tag">&lt;div</span> <span class="attribute-name">class</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">section</span><span class="delimiter">&quot;</span></span><span class="tag">&gt;</span>
<span class="line-numbers"><a href="#n5" name="n5">5</a></span><span class="tag">&lt;h2</span> <span class="attribute-name">id</span>=<span class="string"><span class="delimiter">&quot;</span><span class="content">test</span><span class="delimiter">&quot;</span></span><span class="tag">&gt;</span>Test Section<span class="tag">&lt;/h2&gt;</span>
<span class="line-numbers"><a href="#n6" name="n6">6</a></span>[...]
<span class="line-numbers"><a href="#n7" name="n7">7</a></span>
<span class="line-numbers"><a href="#n8" name="n8">8</a></span><span class="tag">&lt;/div&gt;</span></pre></div>
</div>

</section>
<nav class="navigation"><a href="/glyph/book/stats/links.html">← Link Statistics</a> | <a href="/glyph/book/index.html">Contents</a> | <a href="/glyph/book/extending/macro_def.html">Defining Custom Macros →</a></nav>