Lines Matching defs:token

39      * The goal of all lexer rules/methods is to create a token object.
41      * create a single token. `nextToken` will return this object after
44      * If you subclass to allow multiple token emissions, then set this
45      * to the last token to be matched or something nonnull so that
46      * the auto token emit mechanism will not emit another token.
50     public $token;
53      * What character index in the stream did the current token start at?
54      * Needed, for example, to get the text for current token. Set at
62      * The line on which the first character of the token resides.
76      * Once we see EOF on char stream, next token will be EOF.
84      * The channel number for the current token.
91      * The token type for the current token.
104      * You can set the text for the current token to override what is in the
133         $this->token = null;
151      * Return a token from this source; i.e., match a token on the char stream.
160         // guaranteed at least have text of current token
168                     return $this->token;
175                 $this->token = null;
216                 if ($this->token === null) {
220                 return $this->token;
230      * Instruct the lexer to skip creating a token for current lexer rule
231      * and look for another token. `nextToken` knows to keep looking when
232      * a lexer rule finishes with token set to SKIP_TOKEN. Recall that
233      * if `token === null` at end of any token rule, it creates one for you
310     public function emitToken(Token $token) : void
312         $this->token = $token;
316      * The standard method called to automatically emit a token at the
317      * outermost lexical rule. The token object should point into the
319      * use that to set the token's text. Override this method to emit
324         $token = $this->factory->createEx(
335         $this->emitToken($token);
337         return $token;
413      * Return the text matched so far for the current token or any text override.
429      * Set the complete text of this token; it wipes any previous changes to the text.
438         return $this->token;
444     public function setToken(Token $token) : void
446         $this->token = $token;
487      * Forces load of all tokens. Does not include EOF token.
494         $token = $this->nextToken();
496         while ($token && $token->getType() !== Token::EOF) {
497             $tokens[] = $token;
498             $token = $this->nextToken();
506      * a token, so do the easy thing and just kill a character and hope
541             \sprintf('token recognition error at: \'%s\'', $text),