1Twig Internals
2==============
3
4Twig is very extensible and you can easily hack it. Keep in mind that you
5should probably try to create an extension before hacking the core, as most
6features and enhancements can be handled with extensions. This chapter is also
7useful for people who want to understand how Twig works under the hood.
8
9How does Twig work?
10-------------------
11
12The rendering of a Twig template can be summarized into four key steps:
13
14* **Load** the template: If the template is already compiled, load it and go
15  to the *evaluation* step, otherwise:
16
17  * First, the **lexer** tokenizes the template source code into small pieces
18    for easier processing;
19  * Then, the **parser** converts the token stream into a meaningful tree
20    of nodes (the Abstract Syntax Tree);
21  * Eventually, the *compiler* transforms the AST into PHP code.
22
23* **Evaluate** the template: It basically means calling the ``display()``
24  method of the compiled template and passing it the context.
25
26The Lexer
27---------
28
29The lexer tokenizes a template source code into a token stream (each token is
30an instance of ``\Twig\Token``, and the stream is an instance of
31``\Twig\TokenStream``). The default lexer recognizes 13 different token types:
32
33* ``\Twig\Token::BLOCK_START_TYPE``, ``\Twig\Token::BLOCK_END_TYPE``: Delimiters for blocks (``{% %}``)
34* ``\Twig\Token::VAR_START_TYPE``, ``\Twig\Token::VAR_END_TYPE``: Delimiters for variables (``{{ }}``)
35* ``\Twig\Token::TEXT_TYPE``: A text outside an expression;
36* ``\Twig\Token::NAME_TYPE``: A name in an expression;
37* ``\Twig\Token::NUMBER_TYPE``: A number in an expression;
38* ``\Twig\Token::STRING_TYPE``: A string in an expression;
39* ``\Twig\Token::OPERATOR_TYPE``: An operator;
40* ``\Twig\Token::PUNCTUATION_TYPE``: A punctuation sign;
41* ``\Twig\Token::INTERPOLATION_START_TYPE``, ``\Twig\Token::INTERPOLATION_END_TYPE`` (as of Twig 1.5): Delimiters for string interpolation;
42* ``\Twig\Token::EOF_TYPE``: Ends of template.
43
44You can manually convert a source code into a token stream by calling the
45``tokenize()`` method of an environment::
46
47    $stream = $twig->tokenize(new \Twig\Source($source, $identifier));
48
49.. versionadded:: 1.27
50    ``\Twig\Source`` was introduced in version 1.27, pass the source and the
51    identifier directly on previous versions.
52
53As the stream has a ``__toString()`` method, you can have a textual
54representation of it by echoing the object::
55
56    echo $stream."\n";
57
58Here is the output for the ``Hello {{ name }}`` template:
59
60.. code-block:: text
61
62    TEXT_TYPE(Hello )
63    VAR_START_TYPE()
64    NAME_TYPE(name)
65    VAR_END_TYPE()
66    EOF_TYPE()
67
68.. note::
69
70    The default lexer (``\Twig\Lexer``) can be changed by calling
71    the ``setLexer()`` method::
72
73        $twig->setLexer($lexer);
74
75The Parser
76----------
77
78The parser converts the token stream into an AST (Abstract Syntax Tree), or a
79node tree (an instance of ``\Twig\Node\ModuleNode``). The core extension defines
80the basic nodes like: ``for``, ``if``, ... and the expression nodes.
81
82You can manually convert a token stream into a node tree by calling the
83``parse()`` method of an environment::
84
85    $nodes = $twig->parse($stream);
86
87Echoing the node object gives you a nice representation of the tree::
88
89    echo $nodes."\n";
90
91Here is the output for the ``Hello {{ name }}`` template:
92
93.. code-block:: text
94
95    \Twig\Node\ModuleNode(
96      \Twig\Node\TextNode(Hello )
97      \Twig\Node\PrintNode(
98        \Twig\Node\Expression\NameExpression(name)
99      )
100    )
101
102.. note::
103
104    The default parser (``\Twig\TokenParser\AbstractTokenParser``) can be changed by calling the
105    ``setParser()`` method::
106
107        $twig->setParser($parser);
108
109The Compiler
110------------
111
112The last step is done by the compiler. It takes a node tree as an input and
113generates PHP code usable for runtime execution of the template.
114
115You can manually compile a node tree to PHP code with the ``compile()`` method
116of an environment::
117
118    $php = $twig->compile($nodes);
119
120The generated template for a ``Hello {{ name }}`` template reads as follows
121(the actual output can differ depending on the version of Twig you are
122using)::
123
124    /* Hello {{ name }} */
125    class __TwigTemplate_1121b6f109fe93ebe8c6e22e3712bceb extends \Twig\Template
126    {
127        protected function doDisplay(array $context, array $blocks = [])
128        {
129            // line 1
130            echo "Hello ";
131            echo twig_escape_filter($this->env, (isset($context["name"]) ? $context["name"] : null), "html", null, true);
132        }
133
134        // some more code
135    }
136
137.. note::
138
139    The default compiler (``\Twig\Compiler``) can be changed by calling the
140    ``setCompiler()`` method::
141
142        $twig->setCompiler($compiler);
143