Changes in Lexer.php

1c00c02121477c81b95fe750b94acdf109cba20a - fix(parser): validate inline formatting closers with a single memoized scan

Thu, 09 Jul 2026 12:09:52 +0000

fix(parser): validate inline formatting closers with a single memoized scanThe inline formatting modes only open a span when a valid closer existsahead. That check was a lookahead built on CONTENT_UNTIL_PARA, testedcharacter by character up to the next paragraph break and re-evaluatedfrom scratch for every opener candidate — openers times paragraphlength. With pcre.jit=0 a crafted 32KB page took 16s and an ordinary34KB page with long paragraphs 37s; with the JIT on (the PHP default)the per-character lookahead exhausted the JIT stack, the match silentlyfailed, and the formatting — or everything after it — rendered as plaintext.The check also decided the wrong thing. It scanned raw text, so a closerlookalike inside content the lexer consumes atomically — a nowiki or %%span, a backtick code span, a link, a URL — counted as a real closereven though the mode's exit pattern can never fire there. And it ignoredthe enclosing span: an inner delimiter whose only closer lay past thecloser of the mode it sits in was entered anyway, so a stray delimiterpaired with one in a following sibling span and dragged the boundaryalong — the `*` in ''glob/*.conf'' joined the `*` of the next ''...''span and corrupted the paragraph; the same held for //, ** and __ insidemonospace, and an emphasis opened inside ((...)) ran past the footnote's)) and the enclosing bold's **.Each formatting mode now declares its closer throughLexer::addCloserPattern(), mirroring addExitPattern(), and the lexeranswers "does a valid closer exist ahead" with one anchored possessivescan per range instead of a lookahead per opener:- The scan runs left to right from the opener, hopping over opaque spans derived from already-registered patterns — a plain or special match is consumed in one step, an entry into a verbatim mode (nowiki, the backtick code spans) extends to that mode's first exit — so a closer lookalike inside consumed content is never mistaken for a closer. Each hop finds the earliest of boundary, closer, or opaque span in a single leftmost search, keeping the check linear.- An opener is rejected when the nearest enclosing mode that has a closer of its own would close before the opener's own closer, so a delimiter that can never close within its span stays literal. That ancestor is found by walking the mode stack past modes that declare no closer (plugins, footnotes); the nearest guarded ancestor suffices, as it was itself validated against its own when it opened.- Both verdicts are memoized and reset per parse() run: a proven closer validates every earlier candidate, and a proven closer-free range rejects every later candidate before the next boundary. With the lexer consuming each opened span, the whole parse is linear in document size.Closer patterns match the closing delimiter itself with flanking contextin lookarounds — the convention exit patterns already follow — so closerpositions compare exactly across modes and a closer directly after aninner opener is seen. AbstractFormatting derives the closer from the exitpattern and registers it with the paragraph break as the boundary,preserving the rule that formatting never spans paragraphs; a mode withother needs can pass a different boundary or none.Footnote declares its )) as a closer rather than guarding its (( entrywith a (?=.*)) lookahead, so the footnote becomes a boundary the scansees and formatting inside it no longer pairs across the )); its closertakes no paragraph boundary, as footnotes are block-level. GfmEmphasisgains a closer pattern so single * emphasis is validated the same way,while its entry lookahead still enforces CommonMark nearest-delimiterpairing. GfmEmphasis and GfmStrong span bodies cannot contain theirdelimiter, so their in-pattern lookaheads stay linear on their own; theGFM backtick span bodies get deterministic alternatives with possessivequantifiers, removing their per-character backtracking.CONTENT_UNTIL_PARA is removed: any entry pattern built with it recreatesthe quadratic scan. ParallelRegex gains escapePattern() so embeddedcloser fragments follow the lexer's bare-parenthesis convention, reportsPREG_JIT_STACKLIMIT_ERROR so a future JIT exhaustion surfaces instead ofsilently truncating, and no longer rewrites its registered patterns inplace while compiling the compound regex.The adversarial 32KB page drops to 0.1s, the 37-second benign page to0.1s, and a 128KB variant stays under 0.6s. List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

e248eb6f8b6985dbe44dd2e8e16b6407ea1562a7 - fix(parser): don't drop document content after a blank indented line

Sun, 05 Jul 2026 15:58:46 +0000

fix(parser): don't drop document content after a blank indented linePreformatted's exit pattern (?=\n[^ \t\n]) is a lookahead-only exit: itconsumes no byte so the boundary \n stays in the stream for a following blockmode (an

or header after an indented code block) to anchor on.When that exit fired with nothing else consumed - a "blank" line that is emptyafter its indent, followed by a column-0 line - the match was zero-width at thecurrent offset and the lexer's no-advance guard aborted the whole parse,silently discarding the rest of the document. Reachable on ordinary trailingwhitespace ("abc\n \nmore") and inside footnotes.Exempt zero-width MODE_EXIT matches from the guard: popping the mode stack isreal progress and leaves the boundary byte for the parent mode to consume onthe next iteration. The stack strictly shrinks on each such exit, so thiscannot loop; the infinite-loop protection is unchanged for every otherzero-width match. Preformatted's pattern is left as-is, so

/header after anindented code block still works, including after a blank indented line.Add regressions for the blank-line, blank-line-after-code, andboundary-preservation (hr after a blank indented line) cases. List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

95f694202286c1add4c442936a5caa38db0dd603 - Lexer: dispatch zero-width EXIT events to mode handlers

Tue, 05 May 2026 08:20:33 +0000

Lexer: dispatch zero-width EXIT events to mode handlersinvokeHandler short-circuited on empty content for every lexer state,which silently dropped EXIT events from zero-width exit patterns(lookahead-only). The mode stack would still pop, but the mode'shandle() method never ran, so handler-side cleanup (restoring bufferedcall writers, emitting close calls) was skipped. Now empty content isonly suppressed for non-EXIT states.Also fix the parameter docblock: $is_match was annotated as booleanbut is actually one of the integer DOKU_LEXER_* constants. Renamed to$state to match. List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

864d6c6daac408fa4fc25801adcaf0f744288b30 - fix Lexer so exit-pattern lookbehinds see chars consumed by prior tokens

Tue, 21 Apr 2026 15:28:18 +0000

fix Lexer so exit-pattern lookbehinds see chars consumed by prior tokensLexer::reduce used to hand PCRE a shrinking tail of the subject — eachmatched token was chopped off the front of $raw and the next preg_matchran on what remained. Once a token was consumed, the bytes before thecursor were gone, and any lookbehind assertion in a subsequent patternsilently failed.The bug was latent for DokuWiki's entire history because literal exitpatterns like `\*\*`, ``, or `%%` don't care what's behind them.It surfaced with c3755410a ("require non-whitespace adjacency forinline formatting delimiters"), which added `(?<=[^\s])` to Strong,Emphasis, Underline, Monospace, Subscript, Superscript and Deleted atonce. After that commit, `**[[link]]**` stopped closing — the `]` thatwould satisfy the lookbehind had just been consumed by the link match,so Strong stayed open until end-of-section and swallowed everythingafter it (list items, headings, the lot).Fix:* Lexer::parse and Lexer::reduce track a byte offset into $raw instead of mutating $raw. $initialLength and the shrinking-length arithmetic for absolute match positions are replaced by straight offset increments; the no-progress guard and the trailing-unmatched dispatch both shift to the same cursor.* ParallelRegex::split takes an optional $offset and passes it to preg_match together with PREG_OFFSET_CAPTURE. PCRE scans from the offset forward but still sees the whole subject, so lookbehinds work across already-consumed tokens. The secondary preg_split call used to carve out pre/post is no longer needed — PREG_OFFSET_CAPTURE gives the match start for free, saving one regex operation per reduce() step.Regression tests at all three layers:* ParallelRegexTest — offset plumbing and pre/match accounting.* LexerTest::testIndexLookbehindAcrossConsumedToken — exit-pattern lookbehind targeting the `/>` of a self-closing `` that was consumed as a SPECIAL token on the previous step. Fails under the old Lexer.* FormattingTest — `**[[link]]**` and `**foo//bar//**` round-trip with correct open/close instructions through the full pipeline.Also updates ListsTest::testUnorderedListStrong, whose expectationsdocumented the pre-fix buggy behaviour ("formatting able to spreadacross list items"). With the fix, bold correctly stays within asingle list item; the expected call sequence and the comment areupdated to match. List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

71096e46fcbfaeaa808667aba794e77fe2780169 - move handler methods into ParserMode classes and rename Handler

Sat, 18 Apr 2026 15:35:30 +0000

move handler methods into ParserMode classes and rename HandlerEach ParserMode class now implements handle() from ModeInterface,containing the token handling logic that previously lived as individualmethods on Doku_Handler.The Handler class (formerly Doku_Handler) is the single dispatch point:Lexer passes tokens to Handler::handleToken() which routes to modeobjects, plugins, or returns false. The Lexer only tokenizes andresolves mapHandler aliases.Key changes:- Add handle() to ModeInterface, implemented by all mode classes- Move Doku_Handler to dokuwiki\Parsing\Handler namespace- File extends Code (shared parsing via $type property)- Quotes uses mapHandler() + Handler::getModeName() for sub-modes- Media::parseMedia() replaces Doku_Handler_Parse_Media()- Code::parseHighlightOptions() replaces parse_highlight_options()- Per-parse state (footnote, doublequote) stays on Handler- Deprecated wrappers kept for base/header/internallink/media- Class alias and rector rules added for backward compatibility List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

f8026da1c3e71254eda6c1c8588b1021e77eed8f - replace magic strings with class constants in Lexer

Thu, 16 Apr 2026 13:56:48 +0000

replace magic strings with class constants in LexerIntroduce MODE_EXIT and MODE_SPECIAL_PREFIX constants to replacethe undocumented "__exit" and "_" string conventions used formode transitions. List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

6c16a3a9aa602bb7e269fb6d5d18e1353e17f97f - Use str_starts_with/str_ends_with

Thu, 14 Sep 2023 20:32:18 +0000

Use str_starts_with/str_ends_with List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

d4f83172d9533c4d84f450fe22ef630816b21d75 - code style: line breaks

Thu, 31 Aug 2023 20:44:40 +0000

code style: line breaks List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

90fb952c4c30c09c8446076ba05991c89a3f0b01 - code style: operator spacing

Thu, 31 Aug 2023 20:38:07 +0000

code style: operator spacing List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

bcaec9f47d06126b3e653fea89a86d8b6a6cbef8 - Apply rector fixes to inc/Parsing

Tue, 29 Aug 2023 16:14:40 +0000

Apply rector fixes to inc/Parsing List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

ec34bb300b254ecd0dba0fac22d8115635141cc5 - Update core code to make use of sexplode()

Wed, 19 Oct 2022 13:18:31 +0000

Update core code to make use of sexplode()This makes use of our own explode mechanism everywhere were we expect afixed number of results. List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

46028c4c32caf8dfc675f5544a037bd5e5d8f9c2 - Move defines to their own file

Thu, 04 Jun 2020 06:22:16 +0000

Move defines to their own fileAs described inhttps://github.com/dwp-forge/columns/issues/5#issuecomment-638467603sometime the Lexer constants have not been (auto)loaded when a syntax pluginis invoked (I'm not sure why).In general PSR2 discourages a mix of main code and function/class setupwith the call to define() being considered main code.This patch moves these the define calls to a separate new file, solvingboth of the above problems.These are not all our defines. Instead I focused on the ones that areENUM-like.In the future we should think about what defines can be replaced byclass constants and what other define() calls should be moved. List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

368a782f1ad60530bdc034e6caf519a93b419846 - Let plugins access the lexer mode stack

Wed, 08 Apr 2020 16:22:45 +0000

Let plugins access the lexer mode stack List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

661c1ddc2c77658fc8c124036c2e706227865c5a - Make lexer/state stack more understandable

Wed, 23 May 2018 16:10:38 +0000

Make lexer/state stack more understandable- rename lexer $mode property to avoid two different uses of "mode" variables in the lexer- clarify/improve comments List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php

be906b566b9bdfd92c032ee07c4fd077d820a8d1 - moved all parsing related namespaces to their own

Fri, 04 May 2018 12:22:09 +0000

moved all parsing related namespaces to their own List of files: /dokuwiki/inc/Parsing/Lexer/Lexer.php