fix(parser): validate inline formatting closers with a single memoized scan

The inline formatting modes only open a span when a valid closer exists
ahead. That check was a lookahead built on CONTENT

fix(parser): validate inline formatting closers with a single memoized scan

The inline formatting modes only open a span when a valid closer exists
ahead. That check was a lookahead built on CONTENT_UNTIL_PARA, tested
character by character up to the next paragraph break and re-evaluated
from scratch for every opener candidate — openers times paragraph
length. With pcre.jit=0 a crafted 32KB page took 16s and an ordinary
34KB page with long paragraphs 37s; with the JIT on (the PHP default)
the per-character lookahead exhausted the JIT stack, the match silently
failed, and the formatting — or everything after it — rendered as plain
text.

The check also decided the wrong thing. It scanned raw text, so a closer
lookalike inside content the lexer consumes atomically — a nowiki or %%
span, a backtick code span, a link, a URL — counted as a real closer
even though the mode's exit pattern can never fire there. And it ignored
the enclosing span: an inner delimiter whose only closer lay past the
closer of the mode it sits in was entered anyway, so a stray delimiter
paired with one in a following sibling span and dragged the boundary
along — the `*` in ''glob/*.conf'' joined the `*` of the next ''...''
span and corrupted the paragraph; the same held for //, ** and __ inside
monospace, and an emphasis opened inside ((...)) ran past the footnote's
)) and the enclosing bold's **.

Each formatting mode now declares its closer through
Lexer::addCloserPattern(), mirroring addExitPattern(), and the lexer
answers "does a valid closer exist ahead" with one anchored possessive
scan per range instead of a lookahead per opener:

- The scan runs left to right from the opener, hopping over opaque spans
derived from already-registered patterns — a plain or special match is
consumed in one step, an entry into a verbatim mode (nowiki, the
backtick code spans) extends to that mode's first exit — so a closer
lookalike inside consumed content is never mistaken for a closer. Each
hop finds the earliest of boundary, closer, or opaque span in a single
leftmost search, keeping the check linear.
- An opener is rejected when the nearest enclosing mode that has a closer
of its own would close before the opener's own closer, so a delimiter
that can never close within its span stays literal. That ancestor is
found by walking the mode stack past modes that declare no closer
(plugins, footnotes); the nearest guarded ancestor suffices, as it was
itself validated against its own when it opened.
- Both verdicts are memoized and reset per parse() run: a proven closer
validates every earlier candidate, and a proven closer-free range
rejects every later candidate before the next boundary. With the lexer
consuming each opened span, the whole parse is linear in document size.

Closer patterns match the closing delimiter itself with flanking context
in lookarounds — the convention exit patterns already follow — so closer
positions compare exactly across modes and a closer directly after an
inner opener is seen. AbstractFormatting derives the closer from the exit
pattern and registers it with the paragraph break as the boundary,
preserving the rule that formatting never spans paragraphs; a mode with
other needs can pass a different boundary or none.

Footnote declares its )) as a closer rather than guarding its (( entry
with a (?=.*)) lookahead, so the footnote becomes a boundary the scan
sees and formatting inside it no longer pairs across the )); its closer
takes no paragraph boundary, as footnotes are block-level. GfmEmphasis
gains a closer pattern so single * emphasis is validated the same way,
while its entry lookahead still enforces CommonMark nearest-delimiter
pairing. GfmEmphasis and GfmStrong span bodies cannot contain their
delimiter, so their in-pattern lookaheads stay linear on their own; the
GFM backtick span bodies get deterministic alternatives with possessive
quantifiers, removing their per-character backtracking.

CONTENT_UNTIL_PARA is removed: any entry pattern built with it recreates
the quadratic scan. ParallelRegex gains escapePattern() so embedded
closer fragments follow the lexer's bare-parenthesis convention, reports
PREG_JIT_STACKLIMIT_ERROR so a future JIT exhaustion surfaces instead of
silently truncating, and no longer rewrites its registered patterns in
place while compiling the compound regex.

The adversarial 32KB page drops to 0.1s, the 37-second benign page to
0.1s, and a 128KB variant stays under 0.6s.

show more ...

fix(parser): don't drop document content after a blank indented line

Preformatted's exit pattern (?=\n[^ \t\n]) is a lookahead-only exit: it
consumes no byte so the boundary \n stays in the stream f

fix(parser): don't drop document content after a blank indented line

Preformatted's exit pattern (?=\n[^ \t\n]) is a lookahead-only exit: it
consumes no byte so the boundary \n stays in the stream for a following block
mode (an <hr> or header after an indented code block) to anchor on.

When that exit fired with nothing else consumed - a "blank" line that is empty
after its indent, followed by a column-0 line - the match was zero-width at the
current offset and the lexer's no-advance guard aborted the whole parse,
silently discarding the rest of the document. Reachable on ordinary trailing
whitespace ("abc\n \nmore") and inside footnotes.

Exempt zero-width MODE_EXIT matches from the guard: popping the mode stack is
real progress and leaves the boundary byte for the parent mode to consume on
the next iteration. The stack strictly shrinks on each such exit, so this
cannot loop; the infinite-loop protection is unchanged for every other
zero-width match. Preformatted's pattern is left as-is, so <hr>/header after an
indented code block still works, including after a blank indented line.

Add regressions for the blank-line, blank-line-after-code, and
boundary-preservation (hr after a blank indented line) cases.

show more ...

Lexer: dispatch zero-width EXIT events to mode handlers

invokeHandler short-circuited on empty content for every lexer state,
which silently dropped EXIT events from zero-width exit patterns
(lookah

Lexer: dispatch zero-width EXIT events to mode handlers

invokeHandler short-circuited on empty content for every lexer state,
which silently dropped EXIT events from zero-width exit patterns
(lookahead-only). The mode stack would still pop, but the mode's
handle() method never ran, so handler-side cleanup (restoring buffered
call writers, emitting close calls) was skipped. Now empty content is
only suppressed for non-EXIT states.

Also fix the parameter docblock: $is_match was annotated as boolean
but is actually one of the integer DOKU_LEXER_* constants. Renamed to
$state to match.

show more ...

fix Lexer so exit-pattern lookbehinds see chars consumed by prior tokens

Lexer::reduce used to hand PCRE a shrinking tail of the subject — each
matched token was chopped off the front of $raw and th

fix Lexer so exit-pattern lookbehinds see chars consumed by prior tokens

Lexer::reduce used to hand PCRE a shrinking tail of the subject — each
matched token was chopped off the front of $raw and the next preg_match
ran on what remained. Once a token was consumed, the bytes before the
cursor were gone, and any lookbehind assertion in a subsequent pattern
silently failed.

The bug was latent for DokuWiki's entire history because literal exit
patterns like `\*\*`, `</file>`, or `%%` don't care what's behind them.
It surfaced with c3755410a ("require non-whitespace adjacency for
inline formatting delimiters"), which added `(?<=[^\s])` to Strong,
Emphasis, Underline, Monospace, Subscript, Superscript and Deleted at
once. After that commit, `**[[link]]**` stopped closing — the `]` that
would satisfy the lookbehind had just been consumed by the link match,
so Strong stayed open until end-of-section and swallowed everything
after it (list items, headings, the lot).

Fix:

* Lexer::parse and Lexer::reduce track a byte offset into $raw instead
of mutating $raw. $initialLength and the shrinking-length arithmetic
for absolute match positions are replaced by straight offset
increments; the no-progress guard and the trailing-unmatched dispatch
both shift to the same cursor.

* ParallelRegex::split takes an optional $offset and passes it to
preg_match together with PREG_OFFSET_CAPTURE. PCRE scans from the
offset forward but still sees the whole subject, so lookbehinds work
across already-consumed tokens. The secondary preg_split call used
to carve out pre/post is no longer needed — PREG_OFFSET_CAPTURE
gives the match start for free, saving one regex operation per
reduce() step.

Regression tests at all three layers:

* ParallelRegexTest — offset plumbing and pre/match accounting.
* LexerTest::testIndexLookbehindAcrossConsumedToken — exit-pattern
lookbehind targeting the `/>` of a self-closing `<a/>` that was
consumed as a SPECIAL token on the previous step. Fails under the
old Lexer.
* FormattingTest — `**[[link]]**` and `**foo//bar//**` round-trip
with correct open/close instructions through the full pipeline.

Also updates ListsTest::testUnorderedListStrong, whose expectations
documented the pre-fix buggy behaviour ("formatting able to spread
across list items"). With the fix, bold correctly stays within a
single list item; the expected call sequence and the comment are
updated to match.

show more ...

move handler methods into ParserMode classes and rename Handler

Each ParserMode class now implements handle() from ModeInterface,
containing the token handling logic that previously lived as individ

move handler methods into ParserMode classes and rename Handler

Each ParserMode class now implements handle() from ModeInterface,
containing the token handling logic that previously lived as individual
methods on Doku_Handler.

The Handler class (formerly Doku_Handler) is the single dispatch point:
Lexer passes tokens to Handler::handleToken() which routes to mode
objects, plugins, or returns false. The Lexer only tokenizes and
resolves mapHandler aliases.

Key changes:
- Add handle() to ModeInterface, implemented by all mode classes
- Move Doku_Handler to dokuwiki\Parsing\Handler namespace
- File extends Code (shared parsing via $type property)
- Quotes uses mapHandler() + Handler::getModeName() for sub-modes
- Media::parseMedia() replaces Doku_Handler_Parse_Media()
- Code::parseHighlightOptions() replaces parse_highlight_options()
- Per-parse state (footnote, doublequote) stays on Handler
- Deprecated wrappers kept for base/header/internallink/media
- Class alias and rector rules added for backward compatibility

show more ...

replace magic strings with class constants in Lexer

Introduce MODE_EXIT and MODE_SPECIAL_PREFIX constants to replace
the undocumented "__exit" and "_" string conventions used for
mode transitions.

Use str_starts_with/str_ends_with

code style: line breaks

code style: operator spacing

Apply rector fixes to inc/Parsing

Update core code to make use of sexplode()

This makes use of our own explode mechanism everywhere were we expect a
fixed number of results.

Move defines to their own file

As described in
https://github.com/dwp-forge/columns/issues/5#issuecomment-638467603
sometime the Lexer constants have not been (auto)loaded when a syntax plugin
is in

Move defines to their own file

As described in
https://github.com/dwp-forge/columns/issues/5#issuecomment-638467603
sometime the Lexer constants have not been (auto)loaded when a syntax plugin
is invoked (I'm not sure why).

In general PSR2 discourages a mix of main code and function/class setup
with the call to define() being considered main code.

This patch moves these the define calls to a separate new file, solving
both of the above problems.

These are not all our defines. Instead I focused on the ones that are
ENUM-like.

In the future we should think about what defines can be replaced by
class constants and what other define() calls should be moved.

show more ...

Let plugins access the lexer mode stack

Make lexer/state stack more understandable
- rename lexer $mode property to avoid two different uses of "mode"
variables in the lexer
- clarify/improve comments

moved all parsing related namespaces to their own