Use external program to split pages into words

An external tokenizer inserts extra spaces to mark words in the input text.
The text is sent through STDIN and STDOUT file handles.

A good choice for

Use external program to split pages into words

An external tokenizer inserts extra spaces to mark words in the input text.
The text is sent through STDIN and STDOUT file handles.

A good choice for Chinese and Japanese is MeCab.
http://sourceforge.net/projects/mecab/
With the command line 'mecab -O wakati'

show more ...

Repurpose io_runcmd for pipes

Indexer improvement: regex instead of arrays for lines

When updating a single line that line was split into an array and in a
loop over that array one entry was removed and afterwards a new one
adde

Indexer improvement: regex instead of arrays for lines

When updating a single line that line was split into an array and in a
loop over that array one entry was removed and afterwards a new one
added. Tests have shown that using a regex for doing that is much faster
which can be easily explained as that regex is very simple to match
while a loop over an array isn't that fast. As that update function is
called for every word in a page the impact of this change is
significant.

show more ...

Indexer improvement: Only write the words index when needed

This adds a simple boolean variable that tracks if new words have been
added. When editing a page in many cases all words have already bee

Indexer improvement: Only write the words index when needed

This adds a simple boolean variable that tracks if new words have been
added. When editing a page in many cases all words have already been
used somewhere else or just one or two words are new. Until this change
all words indexes read were always written, now only the changed ones
are written. The overhead of the new boolean variable should be low.

show more ...

Indexer improvement: replace _freadline by fgets

In PHP versions newer than 4.3.0 fgets reads a whole line regardless of
its length when no length is given. Thus the loop in _freadline isn't
needed.

Indexer improvement: replace _freadline by fgets

In PHP versions newer than 4.3.0 fgets reads a whole line regardless of
its length when no length is given. Thus the loop in _freadline isn't
needed. This increases the speed significantly as _freadline was called
very often.

show more ...

Indexer speed improvement: joined array vs. single lines

From my experience with a benchmark of the indexer it is faster to first
join the array of all index entries and then write them back togethe

Indexer speed improvement: joined array vs. single lines

From my experience with a benchmark of the indexer it is faster to first
join the array of all index entries and then write them back together
instead of writing every single entry. This might increase memory usage,
but I couldn't see a significant increase and this function is also only
used for the small index files, not for the large pagewords index.

show more ...

tokenizer was returning prematurely

Use native PHP JSON functions when available

Refactor tokenizer to avoid splitting multiple times

Measure length of multi-character Asian words

Remove unused idx_touchIndex function

improved some metadata comments

FS#2079: always show profile and subscribe links/buttons

Introduce metadata write wrapper p_save_metadata

p_purge_metadata now updates the metadata cache and the INFO array like the other metadata writing functions

Simpler ID trimming

Remove metadata conversion from 0a7e3bce (2006-11-26)

FS#1839: take favicon from mediadir (if it exists)

make custom buttons possible with html_btn() without the need of global $lang (more consistent with tpl_pagelink())

Chinese Language update

handle mailfrom replacements in a central place FS#2091

Only add successfully created sitemap items to the sitemap

Fix XSS vulnerability FS#2085

Basque language update

de-informal / typo fix
de / typo fix

de / typo fixed