fix(search): stop FileIndex::retrieveRows() scanning past the last row

The early-exit check compared the array_shift() result against false, but
array_shift() returns null on an empty array, so the

fix(search): stop FileIndex::retrieveRows() scanning past the last row

The early-exit check compared the array_shift() result against false, but
array_shift() returns null on an empty array, so the break never fired and
every call read the index file to EOF after collecting the last requested
row. Compare against null and skip the file entirely when nothing is
requested.

show more ...

fix(search): wait for a contended index lock and apply dperm

The Lock rewrite (c66b5ec65) made Lock::acquire() fail immediately when the
lock directory already existed, where the old indexer lock re

fix(search): wait for a contended index lock and apply dperm

The Lock rewrite (c66b5ec65) made Lock::acquire() fail immediately when the
lock directory already existed, where the old indexer lock retried until the
holder released it. Concurrent saves or indexer runs then aborted indexing
instead of serializing (recoverable only via the .indexed tag on the next
edit), and the lock directory was created without the configured directory
permissions, unlike every other directory DokuWiki creates.

Mirror the io_lock() convention: wait for a contended lock, bounded by a
tunable wait timeout, before throwing; keep clearing locks older than five
minutes as stale; and chmod the new lock directory to $conf['dperm']. The
give-up path still throws IndexLockException, so the self-healing retry via
the .indexed tag is unchanged.

show more ...

fix(search): create index lock directories inside data/locks

Index locks were built by concatenating the configured lock directory and
the index name without a path separator.

That produced paths l

fix(search): create index lock directories inside data/locks

Index locks were built by concatenating the configured lock directory and
the index name without a path separator.

That produced paths like data/lockspage.index instead of
data/locks/page.index, so index lock directories were created in the
data root rather than inside the configured lock directory.

Fix this by joining the lock directory and index name with an explicit
slash, and update the lock test to assert the correct location and guard
against the old broken path.

show more ...

fix(search): read split index suffixes correctly

AbstractIndex::max() used /(\d)+\.idx$/ to determine the highest split
index suffix. For filenames like w10.idx or w12.idx that regex captured
only t

fix(search): read split index suffixes correctly

AbstractIndex::max() used /(\d)+\.idx$/ to determine the highest split
index suffix. For filenames like w10.idx or w12.idx that regex captured
only the last digit, so the maximum token-length group was truncated to 9.
Wildcard searches therefore skipped all fulltext index shards for words
with 10 or more characters.

Tighten the match to the basename and require the current index name
followed immediately by digits: ^<idx>(\d+)\.idx$. This captures the full
numeric suffix and also avoids counting unrelated index families that
share the same prefix, such as treating wiki2.idx as a numbered shard of
the w index.

Add regressions for both behaviors: multi-digit suffixes are parsed
correctly, and same-prefix indexes are ignored when determining the
maximum shard number.

show more ...

fix(search): search index corruption in updateTuple

Require an end-of-tuple boundary when removing an existing tuple in
TupleOps::updateTuple(), so updating row ID 17 no longer corrupts
tuples for I

fix(search): search index corruption in updateTuple

Require an end-of-tuple boundary when removing an existing tuple in
TupleOps::updateTuple(), so updating row ID 17 no longer corrupts
tuples for IDs like 170 or 171.

Add regression tests covering both replacement and deletion when one
numeric row ID is a decimal prefix of another.

show more ...

SearchIndex: remove write side effect from retrieveRow()

retrieveRow() padded the index file when the requested RID was beyond
the current length. This was an optimization for subsequent changeRow()

SearchIndex: remove write side effect from retrieveRow()

retrieveRow() padded the index file when the requested RID was beyond
the current length. This was an optimization for subsequent changeRow()
calls, but changeRow() already handles padding on its own. The side
effect was also inconsistent with retrieveRows() which is a pure read.

Added a cross-index integration test verifying RID consistency across
entity, token, frequency and reverse indexes when multiple entities
share tokens.

show more ...

SearchIndex: rector, phpcs, type hint fixes

SearchIndex: review fixes — auto-save MemoryIndex, cast TupleOps counts, style cleanups

- MemoryIndex: auto-save dirty data on unlock/destruction to prevent
silent index corruption when indexes ar

SearchIndex: review fixes — auto-save MemoryIndex, cast TupleOps counts, style cleanups

- MemoryIndex: auto-save dirty data on unlock/destruction to prevent
silent index corruption when indexes are used in tandem
- TupleOps::parseTuples(): cast exploded count strings to int
- FileIndex::retrieveRow(): document the write-on-read padding behavior
- Fix whitespace issues in ApiCore, common.php, Sitemap/Mapper
- Update concept.txt to reflect MemoryIndex auto-save behavior

show more ...

SearchIndex: unify CollectionSearch API and optimize search pipeline

- Remove separate lookup() API from CollectionSearch. All searches now
use addTerm()/execute() with a single unified pipeline.

SearchIndex: unify CollectionSearch API and optimize search pipeline

- Remove separate lookup() API from CollectionSearch. All searches now
use addTerm()/execute() with a single unified pipeline.
- Add matches() predicate to Term using efficient string functions
(===, str_starts_with, str_ends_with, str_contains) instead of regex.
- Add caseInsensitive() support on CollectionSearch and Term for
metadata/title searches where indexed values preserve case.
- Remove callback support from MetadataSearch::lookupKey() — the only
real usage (case-insensitive substring) is replaced by
caseInsensitive() + wildcards.
- Remove min-length validation from Term. Add Tokenizer::isValidSearchTerm()
for callers that need it (FulltextSearch, Indexer::lookup).
- Optimize execute() from 4 group passes to 2: scan tokens + resolve
frequencies in one pass per group, batch entity name resolution, then
populate Terms.
- Store full match detail in Term: entity → token → frequency. New
accessors getMatches(), getEntityTokens(), getEntityFrequencies()
derive different views from this single data structure.
- Term no longer used as scratch pad by CollectionSearch. Index-internal
data (token IDs, entity IDs) stays local to execute(). Terms receive
only final resolved results.
- Use title from search results in MetadataSearch::pageLookupCallBack()
instead of re-fetching via p_get_first_heading().
- Update concept.txt documentation.

show more ...

SearchIndex: add integrity checking to Collection architecture

Add checkIntegrity() to AbstractCollection and DirectCollection that
verifies paired indexes have matching line counts (token==frequenc

SearchIndex: add integrity checking to Collection architecture

Add checkIntegrity() to AbstractCollection and DirectCollection that
verifies paired indexes have matching line counts (token==frequency,
entity==reverse, entity==token for direct collections). Throws
IndexIntegrityException on the first inconsistency found.

Add Countable interface to AbstractIndex with count() implementations
in MemoryIndex and FileIndex. Add Indexer::checkIntegrity() and
Indexer::isIndexEmpty() to orchestrate checks across all collections.

Update infoutils.php to use the new Indexer API instead of the old
FulltextIndex/MetadataIndex classes.

Fix range(1, 0) bug in three places that produced [1, 0] instead of
an empty array when split-by-length indexes were empty.

show more ...

SearchIndex: rewrite Indexer to use Collection classes

Replace the intermediate #2943 classes (FulltextIndex, MetadataIndex)
with the new Collection-based architecture. The Indexer is now a thin
sta

SearchIndex: rewrite Indexer to use Collection classes

Replace the intermediate #2943 classes (FulltextIndex, MetadataIndex)
with the new Collection-based architecture. The Indexer is now a thin
stateless orchestrator that delegates all index work to collections.

Key changes:
- Indexer no longer extends AbstractIndex; page name passed to methods
- addPage/deletePage/clear use PageTitleCollection,
PageFulltextCollection, and PageMetaCollection
- New PageMetaCollection replaces separate ReferencesCollection and
MediaCollection with a single class that handles arbitrary metadata
keys dynamically
- Shared writable FileIndex('page') passed to all collections
- Logger callback replaces verbose parameter
- Methods return void instead of bool
- Index classes implement IteratorAggregate for clean data access
- Indexer tests consolidated into namespaced IndexerTest.php
- All callers updated to new stateless API

show more ...

SearchIndex: rewrite Lock as static registry with reference counting

Replace the instance-based Lock class with a static registry that
tracks held locks per-process with reference counting. This sol

SearchIndex: rewrite Lock as static registry with reference counting

Replace the instance-based Lock class with a static registry that
tracks held locks per-process with reference counting. This solves
three problems:

- Split indexes (w3, w4, ...) share a single lock name and now
coordinate naturally via the registry
- Multiple callers can acquire the same lock without conflict
- Indexes enforce their own writability through lock()/unlock()
methods on AbstractIndex

The Lock registry manages both the filesystem lock (mkdir) and the
in-process tracking. The first acquire creates the directory, subsequent
acquires increment the refcount. Release decrements, and only removes
the directory when the count reaches zero.

Note: I am not sure if implementing this as a static object is a great
idea or if we should pass an instance through the collection to the
indexes...

show more ...

SearchIndex: RID cache should not be static

A static var interferes when the same class is instantiated multiple
times

SearchIndex: TupleOps now work with frequencies of 1

We have indexes where we simply track that a relation between entity and
token exists, but there is no frequency. The frequency is always 1.
For

SearchIndex: TupleOps now work with frequencies of 1

We have indexes where we simply track that a relation between entity and
token exists, but there is no frequency. The frequency is always 1.
For those indexes we do not store *1 as frequency but omit it
completely.

The TupleOps class now can work with such indexes and will also store
frequencies of 1 like this.

show more ...

Working fulltext collection and search

This finalizes the FulltextCollection and FulltextCollectionSearch
classes. Proper locking is implemented, tests have been enhanced.

It should be possible to

Working fulltext collection and search

This finalizes the FulltextCollection and FulltextCollectionSearch
classes. Proper locking is implemented, tests have been enhanced.

It should be possible to reimplement the page full text search on top of
it.

show more ...

Indexes can now be opened in readonly mode

add method to retrieve multiple rows at once

added method to search an index by regular expression

keep modification state of MemoryIndexes

MemoryIndexes only need to be saved back when they have been modified

retrieveRow needs to pad the index, too

better method names

we need the same access methods in both index types

Changed IndexAccessor names based on access method not content

Add support to access multiple rows at once

When saving word indexs (w*.idx) often multiple words of the same length
will need to be accessed. This implements a new method that allows that
in an eff

Add support to access multiple rows at once

When saving word indexs (w*.idx) often multiple words of the same length
will need to be accessed. This implements a new method that allows that
in an efficient way.

Note: this removes the INDEX_MARK_DELETED mechanism to mark deleted
entries. Entries are now deleted using empty lines again. This makes the
batch handling much simpler. If a good reason exists that we should keep
it, it can be readded.

show more ...

rearranging the Index class structure

This is a first step at stuff at restructuring the indexing classes a
bit more.

Some background:

We have basically two different kind of index files:

a) RowI

rearranging the Index class structure

This is a first step at stuff at restructuring the indexing classes a
bit more.

Some background:

We have basically two different kind of index files:

a) RowIndex (like page.idx)

Each line in the index contains a single value. The line number is used
as primary ID. These files can be very large. Thus an index like that
should never be read into memory completely if it can be avoided.

b) TupleIndex (like i12.idx)

Each line contains a list of tuples. The files tend to be smaller so
loading them completely for search and replace is easier.

Since the the access is so completely different, I tried to model that
in the two different classes, basically moving the methods from
\dokuwiki\Search\AbstractIndex to the new classes.

While doing so, I tried to make the doc blocks, variable names and
interface easier to understand. I also added tests for each of the
methods.

The old code has not been touched yet. So these classes do not do
anything outside of tests currently.

show more ...