fix(search): lock the metadata registry read-modify-write

updateMetadataRegistry() read metadata.idx, merged in new keys and wrote
the file back without holding a lock across the sequence. Two index

fix(search): lock the metadata registry read-modify-write

updateMetadataRegistry() read metadata.idx, merged in new keys and wrote
the file back without holding a lock across the sequence. Two indexer
processes each registering a different new key could both read the same
registry and the later writer would clobber the other's key, dropping it.
A dropped key is not cleared from its collection on deletePage(), leaving
orphaned index entries.

Guard the whole read-merge-write with a dedicated metadata lock so
concurrent registrations can no longer lose a key.

show more ...

fix(search): wait for a contended index lock and apply dperm

The Lock rewrite (c66b5ec65) made Lock::acquire() fail immediately when the
lock directory already existed, where the old indexer lock re

fix(search): wait for a contended index lock and apply dperm

The Lock rewrite (c66b5ec65) made Lock::acquire() fail immediately when the
lock directory already existed, where the old indexer lock retried until the
holder released it. Concurrent saves or indexer runs then aborted indexing
instead of serializing (recoverable only via the .indexed tag on the next
edit), and the lock directory was created without the configured directory
permissions, unlike every other directory DokuWiki creates.

Mirror the io_lock() convention: wait for a contended lock, bounded by a
tunable wait timeout, before throwing; keep clearing locks older than five
minutes as stale; and chmod the new lock directory to $conf['dperm']. The
give-up path still throws IndexLockException, so the self-healing retry via
the .indexed tag is unchanged.

show more ...

fix(search): restore histogram() as a Collection method with a BC shim

The indexer rework dropped Doku_Indexer::histogram(), which plugins reach
through idx_get_indexer()->histogram() to build tag c

fix(search): restore histogram() as a Collection method with a BC shim

The indexer rework dropped Doku_Indexer::histogram(), which plugins reach
through idx_get_indexer()->histogram() to build tag clouds (e.g. tagfilter
via the 'subject' metadata index). Calling it fatalled through
LegacyIndexer::__call as an undefined method.

Add histogram() to AbstractCollection: it sums each token's frequency across
all entities and returns them ordered by frequency, filtered by min/max/minlen,
handling both length-split (fulltext) and single-group (metadata) collections.
DirectCollection overrides it for the 1:1 title case (count shared values).
LegacyIndexer::histogram() is a deprecated shim dispatching by $key to the
matching collection.

show more ...

fix(search): don't let collections release a caller-owned index lock

Indexer::addPage() locks the page index once and hands the same instance to
several collections, each of which lock() and unlock(

fix(search): don't let collections release a caller-owned index lock

Indexer::addPage() locks the page index once and hands the same instance to
several collections, each of which lock() and unlock() it around their work.

AbstractCollection::lock() locked and tracked every index for later release.
When a shared AbstractIndex was already locked by the caller, locking it again
was a no-op but the collection still tracked it, so the collection's unlock()
released the caller's lock and flipped the shared index read-only. Each later
collection then re-acquired and re-released it, leaving windows in which another
process could grab the page lock mid-operation; contention could abort addPage()
with the title index updated but the fulltext index stale.

Treat a shared index that is already writable as owned by its creator: leave its
lock untouched and only manage the locks the collection acquires itself.

Add a regression test asserting a caller-held index lock survives repeated
collection lock()/unlock() cycles.

show more ...

fix(search): create index lock directories inside data/locks

Index locks were built by concatenating the configured lock directory and
the index name without a path separator.

That produced paths l

fix(search): create index lock directories inside data/locks

Index locks were built by concatenating the configured lock directory and
the index name without a path separator.

That produced paths like data/lockspage.index instead of
data/locks/page.index, so index lock directories were created in the
data root rather than inside the configured lock directory.

Fix this by joining the lock directory and index name with an explicit
slash, and update the lock test to assert the correct location and guard
against the old broken path.

show more ...

fix(search): read split index suffixes correctly

AbstractIndex::max() used /(\d)+\.idx$/ to determine the highest split
index suffix. For filenames like w10.idx or w12.idx that regex captured
only t

fix(search): read split index suffixes correctly

AbstractIndex::max() used /(\d)+\.idx$/ to determine the highest split
index suffix. For filenames like w10.idx or w12.idx that regex captured
only the last digit, so the maximum token-length group was truncated to 9.
Wildcard searches therefore skipped all fulltext index shards for words
with 10 or more characters.

Tighten the match to the basename and require the current index name
followed immediately by digits: ^<idx>(\d+)\.idx$. This captures the full
numeric suffix and also avoids counting unrelated index families that
share the same prefix, such as treating wiki2.idx as a numbered shard of
the w index.

Add regressions for both behaviors: multi-digit suffixes are parsed
correctly, and same-prefix indexes are ignored when determining the
maximum shard number.

show more ...

fix(search): fatals on invalid short wildcard terms

Searches such as "wiki a*" could crash in QueryEvaluator::opAnd() with
a TypeError. The query parser still emitted the wildcard term, but the
full

fix(search): fatals on invalid short wildcard terms

Searches such as "wiki a*" could crash in QueryEvaluator::opAnd() with
a TypeError. The query parser still emitted the wildcard term, but the
fulltext search later dropped it because its non-wildcard base was below
the minimum search term length.

That left the evaluator with an AND operator whose right-hand operand was
missing, causing a stack underflow during RPN evaluation.

Fix this in two places:
- filter invalid short wildcard terms already in Tokenizer::getWords()
when wildcard parsing is enabled, so they never enter the parsed query
- harden QueryEvaluator against missing operands for AND/OR/NOT to avoid
fatals if tokens disappear during later processing

Add regression tests covering the parser output for "wiki a*" and the
evaluator behavior when an operand is missing.

show more ...

fix(search): search index corruption in updateTuple

Require an end-of-tuple boundary when removing an existing tuple in
TupleOps::updateTuple(), so updating row ID 17 no longer corrupts
tuples for I

fix(search): search index corruption in updateTuple

Require an end-of-tuple boundary when removing an existing tuple in
TupleOps::updateTuple(), so updating row ID 17 no longer corrupts
tuples for IDs like 170 or 171.

Add regression tests covering both replacement and deletion when one
numeric row ID is a decimal prefix of another.

show more ...

BacklinksTest: give testLinksInDeletedPages its own page

testLinksInDeletedPages reused test:internallinks, the same page
testInternallink already saves and indexes. Since the data dir is
shared acr

BacklinksTest: give testLinksInDeletedPages its own page

testLinksInDeletedPages reused test:internallinks, the same page
testInternallink already saves and indexes. Since the data dir is
shared across the class, when the re-save and the earlier index land
in the same second, needsIndexing() now (correctly) reports the page
as up to date and addPage() skips reindexing, leaving stale link data.
backlinks('test:internallink') then returned an empty array.

Use a dedicated page (test:deletedlinks) with its own link targets so
the test no longer collides with testInternallink's index state.

show more ...

Indexer: signal nothing-to-do via boolean return instead of void

The TaskRunner runs indexing, sitemap, digest and changelog-trim tasks
in sequence and relies on each task returning false when it di

Indexer: signal nothing-to-do via boolean return instead of void

The TaskRunner runs indexing, sitemap, digest and changelog-trim tasks
in sequence and relies on each task returning false when it did no work
so the next one is tried. The indexer rewrite changed addPage(),
deletePage() and renamePage() to return void and only abort via
exceptions, breaking that contract: indexing always looked like work was
done and the following tasks never ran.

Restore the boolean return on these three methods (true when work was
done, false when there was nothing to do) while still using exceptions
to signal errors, and propagate it through TaskRunner::runIndexer().
runIndexer() also no longer forces reindexing on every call.

The legacy compatibility layer is adjusted to match: LegacyIndexer and
idx_addPage() forward the boolean, mapping SearchExceptions back to the
historic error-message/false returns. LegacyIndexer::renamePage()
restores the 'page is not in index' message that the move plugin expects.

Closes #4661

show more ...

Indexer: treat same-second save and index as up to date

needsIndexing() compared the .indexed tag mtime against the page mtime
with <=, so a page that was saved and indexed within the same second wa

Indexer: treat same-second save and index as up to date

needsIndexing() compared the .indexed tag mtime against the page mtime
with <=, so a page that was saved and indexed within the same second was
always reported as still needing indexing. Require the page to be
strictly newer than the index tag instead, so an equal mtime correctly
counts as up to date.

show more ...

fix(indexer): explicitly handle renames

In an attempt to simplify the index handling, the newly refactored
indexer implemented a rename as delete+add sequence.

This had unintended consequences for

fix(indexer): explicitly handle renames

In an attempt to simplify the index handling, the newly refactored
indexer implemented a rename as delete+add sequence.

This had unintended consequences for the move plugin which may move
several pages at once, requiring a working index even while some pages
have already been moved while others still remain at their old location.

Related to #4646

show more ...

SearchIndex: fix IntegrityTest not re-indexing between tests

The .indexed metadata tag persisted between test methods, causing
needsIndexing() to skip re-indexing when saveWikiText() didn't
update t

SearchIndex: fix IntegrityTest not re-indexing between tests

The .indexed metadata tag persisted between test methods, causing
needsIndexing() to skip re-indexing when saveWikiText() didn't
update the wiki file (identical content). Clean the tag in setUp.

show more ...

SearchIndex: remove write side effect from retrieveRow()

retrieveRow() padded the index file when the requested RID was beyond
the current length. This was an optimization for subsequent changeRow()

SearchIndex: remove write side effect from retrieveRow()

retrieveRow() padded the index file when the requested RID was beyond
the current length. This was an optimization for subsequent changeRow()
calls, but changeRow() already handles padding on its own. The side
effect was also inconsistent with retrieveRows() which is a pure read.

Added a cross-index integration test verifying RID consistency across
entity, token, frequency and reverse indexes when multiple entities
share tokens.

show more ...

SearchIndex: unify CollectionSearch API and optimize search pipeline

- Remove separate lookup() API from CollectionSearch. All searches now
use addTerm()/execute() with a single unified pipeline.

SearchIndex: unify CollectionSearch API and optimize search pipeline

- Remove separate lookup() API from CollectionSearch. All searches now
use addTerm()/execute() with a single unified pipeline.
- Add matches() predicate to Term using efficient string functions
(===, str_starts_with, str_ends_with, str_contains) instead of regex.
- Add caseInsensitive() support on CollectionSearch and Term for
metadata/title searches where indexed values preserve case.
- Remove callback support from MetadataSearch::lookupKey() — the only
real usage (case-insensitive substring) is replaced by
caseInsensitive() + wildcards.
- Remove min-length validation from Term. Add Tokenizer::isValidSearchTerm()
for callers that need it (FulltextSearch, Indexer::lookup).
- Optimize execute() from 4 group passes to 2: scan tokens + resolve
frequencies in one pass per group, batch entity name resolution, then
populate Terms.
- Store full match detail in Term: entity → token → frequency. New
accessors getMatches(), getEntityTokens(), getEntityFrequencies()
derive different views from this single data structure.
- Term no longer used as scratch pad by CollectionSearch. Index-internal
data (token IDs, entity IDs) stays local to execute(). Terms receive
only final resolved results.
- Use title from search results in MetadataSearch::pageLookupCallBack()
instead of re-fetching via p_get_first_heading().
- Update concept.txt documentation.

show more ...

SearchIndex: move search() function tests back to tests/inc/search/

The search.test.php file tests the search() function from inc/search.php,
not the Search namespace classes. It was incorrectly mov

SearchIndex: move search() function tests back to tests/inc/search/

The search.test.php file tests the search() function from inc/search.php,
not the Search namespace classes. It was incorrectly moved into
tests/Search/ during the test suite reorganization. Move it and its
data files (ns1/, ns2/) back to their original location, keeping only
searchtest.txt in tests/Search/data/ where it belongs.

show more ...

SearchIndex: add backward compatibility wrappers

Add deprecated wrappers for idx_* and ft_* functions that were removed
when inc/indexer.php and inc/fulltext.php were replaced by the new
Search clas

SearchIndex: add backward compatibility wrappers

Add deprecated wrappers for idx_* and ft_* functions that were removed
when inc/indexer.php and inc/fulltext.php were replaced by the new
Search classes. These wrappers delegate to the new architecture and
ensure existing plugins continue to work.

Deprecated standalone functions: idx_get_indexer, idx_getIndex,
idx_lookup, idx_listIndexLengths, idx_indexLengths, ft_pageSearch,
ft_backlinks, ft_mediause, ft_pageLookup, ft_snippet, ft_pagesorter,
ft_snippet_re_preprocess, ft_queryParser.

Deprecated methods on Indexer: lookupKey, getPages, addMetaKeys,
renameMetaValue, getPID, lookup.

Also migrates remaining core callers (Ajax, FeedCreator, ApiCore) to
use the new classes directly and fixes a UTF-8 case folding bug in
MetadataSearch title lookups.

show more ...

SearchIndex: add integrity checking to Collection architecture

Add checkIntegrity() to AbstractCollection and DirectCollection that
verifies paired indexes have matching line counts (token==frequenc

SearchIndex: add integrity checking to Collection architecture

Add checkIntegrity() to AbstractCollection and DirectCollection that
verifies paired indexes have matching line counts (token==frequency,
entity==reverse, entity==token for direct collections). Throws
IndexIntegrityException on the first inconsistency found.

Add Countable interface to AbstractIndex with count() implementations
in MemoryIndex and FileIndex. Add Indexer::checkIntegrity() and
Indexer::isIndexEmpty() to orchestrate checks across all collections.

Update infoutils.php to use the new Indexer API instead of the old
FulltextIndex/MetadataIndex classes.

Fix range(1, 0) bug in three places that produced [1, 0] instead of
an empty array when split-by-length indexes were empty.

show more ...

SearchIndex: rewrite MetadataSearch to use Collection classes

Replace MetadataIndex usage in MetadataSearch with the new Collection/Index
architecture. This completes the read-path migration so data

SearchIndex: rewrite MetadataSearch to use Collection classes

Replace MetadataIndex usage in MetadataSearch with the new Collection/Index
architecture. This completes the read-path migration so data written by the
Collection-based Indexer is read back correctly using TupleOps tuple format.

Generalize FrequencyCollectionSearch into CollectionSearch that works with any
AbstractCollection type (Frequency, Lookup, Direct) and handles both
split-by-length and non-split index layouts transparently. DirectCollection
participates via resolveTokenFrequencies() which maps token RID = entity RID.

Key changes:
- AbstractCollection gains isSplitByLength(), resolveTokenFrequencies(),
getEntitiesWithData(), and groupToSuffix() with validation
- Index groups are now int (0 = non-split, positive = token length)
- CollectionSearch provides both addTerm()/execute() for fulltext and
lookup() for metadata-style search (exact/wildcard/callback)
- MetadataSearch delegates entirely to collection APIs
- Shared filterPages() replaces duplicated page filtering logic
- All callers updated from MetadataIndex to MetadataSearch
- Tests moved to Search namespace with full coverage for new APIs

show more ...

SearchIndex: reorganize and expand test suite

Move all Search tests from _test/tests/inc/Search/ to _test/tests/Search/
to match the dokuwiki\test autoloader convention. Fix namespaces from
tests\*

SearchIndex: reorganize and expand test suite

Move all Search tests from _test/tests/inc/Search/ to _test/tests/Search/
to match the dokuwiki\test autoloader convention. Fix namespaces from
tests\* to dokuwiki\test\* so all tests work in isolation.

Extract inline test helpers into separate autoloadable mock files:
TestDirectCollection → MockDirectCollection, TestLookupCollection →
MockLookupCollection, TestFrequencyCollection → MockFrequencyCollection.

Rename AbstractIndexTest → AbstractIndexTestCase to fix PHPUnit warning
about abstract classes with Test suffix.

Replace dead xxxRealWord() with proper testWildcardSearch() verifying
exact token matches and frequencies for all three wildcard types.
Add testTokenizedPageSearch() using a dedicated test data file. Add
testNoMatchReturnsEmptyFrequencies() which exposed a bug in Term where
uninitialized $tokens/$frequencies caused crashes on zero-match terms.

Replace fulltext_query.test.php with modern QueryParserTest in the
Search\Query namespace.

Add new test files:
- LockTest: acquire/release, reference counting, stale lock override,
foreign lock rejection, releaseAll, independent locks
- NamespacePredicateTest: filter/exclude, sub-namespaces, partial prefix
safety, empty sets, score preservation
- PageSetTest: intersect, unite, subtract, isEmpty
- QueryEvaluatorTest: word lookups, AND/OR/NOT, namespace filtering,
combined queries, partial namespace prefix safety

Fix Term.php: initialize $tokens and $frequencies to [] instead of null.

show more ...