1Twig Internals
2==============
3
4Twig is very extensible and you can hack it. Keep in mind that you
5should probably try to create an extension before hacking the core, as most
6features and enhancements can be handled with extensions. This chapter is also
7useful for people who want to understand how Twig works under the hood.
8
9How does Twig work?
10-------------------
11
12The rendering of a Twig template can be summarized into four key steps:
13
14* **Load** the template: If the template is already compiled, load it and go
15  to the *evaluation* step, otherwise:
16
17  * First, the **lexer** tokenizes the template source code into small pieces
18    for easier processing;
19
20  * Then, the **parser** converts the token stream into a meaningful tree
21    of nodes (the Abstract Syntax Tree);
22
23  * Finally, the *compiler* transforms the AST into PHP code.
24
25* **Evaluate** the template: It means calling the ``display()``
26  method of the compiled template and passing it the context.
27
28The Lexer
29---------
30
31The lexer tokenizes a template source code into a token stream (each token is
32an instance of ``\Twig\Token``, and the stream is an instance of
33``\Twig\TokenStream``). The default lexer recognizes 13 different token types:
34
35* ``\Twig\Token::BLOCK_START_TYPE``, ``\Twig\Token::BLOCK_END_TYPE``: Delimiters for blocks (``{% %}``)
36* ``\Twig\Token::VAR_START_TYPE``, ``\Twig\Token::VAR_END_TYPE``: Delimiters for variables (``{{ }}``)
37* ``\Twig\Token::TEXT_TYPE``: A text outside an expression;
38* ``\Twig\Token::NAME_TYPE``: A name in an expression;
39* ``\Twig\Token::NUMBER_TYPE``: A number in an expression;
40* ``\Twig\Token::STRING_TYPE``: A string in an expression;
41* ``\Twig\Token::OPERATOR_TYPE``: An operator;
42* ``\Twig\Token::PUNCTUATION_TYPE``: A punctuation sign;
43* ``\Twig\Token::INTERPOLATION_START_TYPE``, ``\Twig\Token::INTERPOLATION_END_TYPE``: Delimiters for string interpolation;
44* ``\Twig\Token::EOF_TYPE``: Ends of template.
45
46You can manually convert a source code into a token stream by calling the
47``tokenize()`` method of an environment::
48
49    $stream = $twig->tokenize(new \Twig\Source($source, $identifier));
50
51As the stream has a ``__toString()`` method, you can have a textual
52representation of it by echoing the object::
53
54    echo $stream."\n";
55
56Here is the output for the ``Hello {{ name }}`` template:
57
58.. code-block:: text
59
60    TEXT_TYPE(Hello )
61    VAR_START_TYPE()
62    NAME_TYPE(name)
63    VAR_END_TYPE()
64    EOF_TYPE()
65
66.. note::
67
68    The default lexer (``\Twig\Lexer``) can be changed by calling
69    the ``setLexer()`` method::
70
71        $twig->setLexer($lexer);
72
73The Parser
74----------
75
76The parser converts the token stream into an AST (Abstract Syntax Tree), or a
77node tree (an instance of ``\Twig\Node\ModuleNode``). The core extension defines
78the basic nodes like: ``for``, ``if``, ... and the expression nodes.
79
80You can manually convert a token stream into a node tree by calling the
81``parse()`` method of an environment::
82
83    $nodes = $twig->parse($stream);
84
85Echoing the node object gives you a nice representation of the tree::
86
87    echo $nodes."\n";
88
89Here is the output for the ``Hello {{ name }}`` template:
90
91.. code-block:: text
92
93    \Twig\Node\ModuleNode(
94      \Twig\Node\TextNode(Hello )
95      \Twig\Node\PrintNode(
96        \Twig\Node\Expression\NameExpression(name)
97      )
98    )
99
100.. note::
101
102    The default parser (``\Twig\TokenParser\AbstractTokenParser``) can be changed by calling the
103    ``setParser()`` method::
104
105        $twig->setParser($parser);
106
107The Compiler
108------------
109
110The last step is done by the compiler. It takes a node tree as an input and
111generates PHP code usable for runtime execution of the template.
112
113You can manually compile a node tree to PHP code with the ``compile()`` method
114of an environment::
115
116    $php = $twig->compile($nodes);
117
118The generated template for a ``Hello {{ name }}`` template reads as follows
119(the actual output can differ depending on the version of Twig you are
120using)::
121
122    /* Hello {{ name }} */
123    class __TwigTemplate_1121b6f109fe93ebe8c6e22e3712bceb extends \Twig\Template
124    {
125        protected function doDisplay(array $context, array $blocks = [])
126        {
127            // line 1
128            echo "Hello ";
129            echo twig_escape_filter($this->env, (isset($context["name"]) ? $context["name"] : null), "html", null, true);
130        }
131
132        // some more code
133    }
134
135.. note::
136
137    The default compiler (``\Twig\Compiler``) can be changed by calling the
138    ``setCompiler()`` method::
139
140        $twig->setCompiler($compiler);
141