Version upped

updated pdfparser

Version upped

Track the cosmocode prinsfrank/pdfparser dev fork

The dev fork adds native UTF-16BE Info-string decoding and Form-XObject
text extraction. Point composer at the fork's VCS repo, pin dev-dev, and
dro

Track the cosmocode prinsfrank/pdfparser dev fork

The dev fork adds native UTF-16BE Info-string decoding and Form-XObject
text extraction. Point composer at the fork's VCS repo, pin dev-dev, and
drop the now-redundant normalizePdfString() UTF-16BE shim (metadata values
just get trimmed). Refresh AGENTS.md accordingly.

show more ...

Return body text and metadata from a single extraction

Every extractor now returns an ExtractionResult (body text plus a
normalised metadata map) from one parse of the file, instead of just a
text s

Return body text and metadata from a single extraction

Every extractor now returns an ExtractionResult (body text plus a
normalised metadata map) from one parse of the file, instead of just a
text string. Metadata keys are a single canonical vocabulary shared
across all formats - Title, Author, Subject, Keywords, Description,
Created, Modified, Language, Producer, plus Copyright (image-only) - so
consumers never have to special-case the source format.

- ExtractionResult: final readonly value object carrying $text,
$metadata and a per-half ?Throwable $textError / $metadataError (with
isComplete()). Text and metadata are extracted independently: an
extractor throws only at its total-failure gate (container won't unzip,
PDF won't parse, file unreadable) and otherwise records a failed half's
error while still returning the salvaged half.

- OOXML/ODF metadata: new AbstractOoxmlExtractor and AbstractOdfExtractor
intermediaries declare each family's source map once (docProps/core.xml
+ app.xml; meta.xml); the concrete extractors only switch their extends
clause. AbstractZipXmlExtractor gains a generic mapMetadataFromXml()
walker.

- PDF: reads the Info dictionary (Title/Author/Subject/Keywords/Created/
Modified/Producer-else-Creator) alongside the body text from the same
parse. normalizePdfString() is a temporary shim for UTF-16BE literal
strings that prinsfrank <= v3.1.0 returns mojibake-expanded.

- Images: now metadata-only (text is ''); field-map keys renamed to the
canonical vocabulary (Caption->Description, Date->Created,
Camera->Producer).

- helper: extract() and extractMetadata() added; extractText() keeps its
string return and its throw-on-failure contract (rethrows textError),
so the docsearch plugin keeps working unchanged.

- cli: prints body text followed by metadata by default; --text/-t and
--meta/-m restrict to one. A salvaged-but-partial extraction warns on
STDERR and exits 0; nothing usable exits non-zero.

- ExtractionException::wrap() centralises wrapping a caught error as an
ExtractionException (passing one through if it already is).

Tests assert the real Tika samples' embedded metadata and the
partial-failure paths. The pre-existing PDF body-text Form-XObject
failure is unchanged.

show more ...

Replace smalot/pdfparser with prinsfrank/pdfparser

Switch the PDF extractor from the cosmocode/pdfparser memory fork
(an LGPL smalot/pdfparser fork) to prinsfrank/pdfparser (MIT, built
from scratch,

Replace smalot/pdfparser with prinsfrank/pdfparser

Switch the PDF extractor from the cosmocode/pdfparser memory fork
(an LGPL smalot/pdfparser fork) to prinsfrank/pdfparser (MIT, built
from scratch, zero PHP dependencies beyond prinsfrank/glyph-lists).

PdfExtractor now calls parseFile($path)->getText() in the default
in-memory mode, which benchmarked both faster and far lighter than the
old fork (on a 12 MB/92-page file: 63s/49.5MB vs 68s/100MB), so the
previous setRetainImageContent() tuning is no longer needed.

Known limitation: prinsfrank/pdfparser <= v3.1.0 does not extract text
inside Form XObjects (page content painted with the Do operator, common
in Quartz/Firefox/Chrome PDFs such as the Tika sample), so
PdfExtractorTest::testExtractsText fails against stock v3.1.0.

show more ...

Switch to the cosmocode/pdfparser memory fork

Replace the upstream smalot/pdfparser dependency with our own fork
(cosmocode/pdfparser, dev-memory branch) for its lower-peak-memory
streaming parser.

Switch to the cosmocode/pdfparser memory fork

Replace the upstream smalot/pdfparser dependency with our own fork
(cosmocode/pdfparser, dev-memory branch) for its lower-peak-memory
streaming parser. The fork keeps the smalot/pdfparser package name, so
it is wired in as a vcs repository in composer.json and required as
dev-memory; the vendored copy and lock file are regenerated accordingly.

PdfExtractor is unchanged - the public API it uses
(Config::setRetainImageContent(), Parser::parseFile()) is identical.

show more ...

Lower PDF parser peak memory by skipping image content

PDFs are parsed entirely into memory before any text is returned, so a
large file can exhaust the memory limit even when its text is tiny.
Sett

Lower PDF parser peak memory by skipping image content

PDFs are parsed entirely into memory before any text is returned, so a
large file can exhaust the memory limit even when its text is tiny.
Setting retainImageContent(false) makes smalot/pdfparser skip decoding
image streams - the largest streams in a typical PDF - which markedly
lowers peak memory without changing the extracted text.

show more ...

Use real Apache Tika sample files and trim the test suite

Replace the generated fixtures with real documents from the Apache Tika
test corpus (tika-* files plus download.sh and a short README) and r

Use real Apache Tika sample files and trim the test suite

Replace the generated fixtures with real documents from the Apache Tika
test corpus (tika-* files plus download.sh and a short README) and rework
the tests to run against them:

- Samples.php: drop the tempDir()/cleanup() single-call wrappers, let
withoutPart() and corrupt() manage their own temp files, and build the
"container missing a member" input with php-archive instead of ext-zip.
- ExtractorFactoryTest: stop re-typing the EXTRACTORS map; keep only the
case/path normalisation cases and own the missing-file/corrupt error
paths centrally.
- Per-format extractor tests: drop the duplicated shared error tests and
their setUp/tearDown, keeping the format-specific assertions; move the
multibyte-UTF8 check into DocxExtractorTest.
- Remove HelperTest and SampleFilesTest as redundant.
- Add a real-sample test for the plain-text extractor.

show more ...

Test against real files instead of generated fixtures

A hand-built fixture only encodes our belief about a file format; if that
belief is wrong the extractor is written to match it and both agree on

Test against real files instead of generated fixtures

A hand-built fixture only encodes our belief about a file format; if that
belief is wrong the extractor is written to match it and both agree on a
fiction that never matches reality. Replace the synthetic FixtureBuilder with
real files produced by real applications.

- _test/data/regenerate.sh rebuilds the edge-case samples from inline
plain-text sources: LibreOffice converts flat-ODF to the office formats,
ImageMagick + exiftool produce images with real IPTC/EXIF metadata.
- New samples cover headers/footers, tab/line-break handling, multi-sheet name
resolution, speaker notes, merged/covered cells and image metadata.
- Error-path tests derive broken inputs from real files (Samples::withoutPart
removes one ZIP member, Samples::corrupt writes garbage) rather than
fabricating containers.
- Scenarios real tools never emit (unnamed sheet, repeated filled-text cell,
the synthetic MAX_REPEAT extreme) are dropped rather than faked.

Delete FixtureBuilder; add the Samples helper. _test is export-ignored so the
sample binaries never ship in release tarballs.

show more ...

Drop the PDF parser availability check

smalot/pdfparser is bundled in the committed vendor/ directory, so the parser
is always present. The class_exists() guard could never fire and its
"run compose

Drop the PDF parser availability check

smalot/pdfparser is bundled in the committed vendor/ directory, so the parser
is always present. The class_exists() guard could never fire and its
"run composer install" message was misleading for a bundled dependency.

show more ...

Open extractors and factory for extension

Remove final from ExtractorFactory and the nine extractors, and widen every
private method, property and constant to protected, so the classes can be
subcla

Open extractors and factory for extension

Remove final from ExtractorFactory and the nine extractors, and widen every
private method, property and constant to protected, so the classes can be
subclassed and their helpers reused. The abstract base, the interface and the
exception hierarchy were already extensible and are unchanged.

show more ...

Make ExtractorFactory the single source of supported extensions

The file extensions were duplicated across forFile(), supportedExtensions()
and every extractor's supports()/extension(). Collapse the

Make ExtractorFactory the single source of supported extensions

The file extensions were duplicated across forFile(), supportedExtensions()
and every extractor's supports()/extension(). Collapse them into one
private const map (extension => extractor class) in ExtractorFactory, which
is now the sole routing authority: forFile() looks the extension up and
supportedExtensions() returns its keys.

Extractors are now dumb workers implementing only extract(); the unused
supports()/extensions() methods are dropped from ExtractorInterface and all
extractors. Per-extractor supports() tests are removed (routing is covered
by the factory tests).

show more ...

Use DokuWiki's temp dir and clean up in destructor

Allocate the archive-unpack temp dir in DokuWiki's own temp dir
($conf['tmpdir']) via core's io_mktmpdir(), and remove it with
io_rmdir(), instead

Use DokuWiki's temp dir and clean up in destructor

Allocate the archive-unpack temp dir in DokuWiki's own temp dir
($conf['tmpdir']) via core's io_mktmpdir(), and remove it with
io_rmdir(), instead of the system temp dir. Add a destructor that
repeats the cleanup as a safety net so the dir is removed even if the
process dies before extract()'s finally block runs. Align the test
FixtureBuilder on the same core helpers.

show more ...

Expand test coverage and fix two extractor bugs

Broaden the test suite from happy-path fixtures to the branching logic that
was previously unexercised, and fix two real bugs the new coverage exposed

Expand test coverage and fix two extractor bugs

Broaden the test suite from happy-path fixtures to the branching logic that
was previously unexercised, and fix two real bugs the new coverage exposed:

- XlsxExtractor: resolve sheet name->file mapping and tab order through
xl/workbook.xml + xl/_rels/workbook.xml.rels instead of pairing sorted
worksheet filenames with names positionally, which mismatched them whenever
tab order differed from file numbering. Falls back to positional/SheetN
naming when the workbook or its rels are absent.
- ImageExtractor: PHP's exif reader decodes Windows XP tags into a WINXP
section under short names (Title/Comment/Author/Keywords/Subject), not as
XPTitle etc., so those fields were silently dropped. List the short names
first, keep the XP* raw names as a cross-version fallback.

Tests:
- FixtureBuilder gains DOCX header/footer, PPTX notes, multi-sheet XLSX (+
no-workbook/no-worksheet variants), rich ODS (repeated columns, covered
cells, multi-paragraph cells, MAX_REPEAT cap, unnamed-sheet fallback), a
hand-packed TIFF with EXIF incl. a UTF-16LE XPTitle, and a public zip()
helper for composing broken containers.
- New critical-path coverage: header/footer, speaker notes, ODS edge cases,
XLSX multi-sheet/fallback/no-worksheet guards, EXIF/TIFF incl. XP decode,
and per-format missing-part/missing-file error paths.
- Routing, reject and supports() tests converted to data providers, plus a
drift guard that routes every supportedExtensions() entry through forFile().
- Real-world LibreOffice samples under _test/data covered by SampleFilesTest.

47 -> 155 tests, 104 -> 231 assertions.

show more ...

Initial implementation of the totext plugin

Extract plain text from various file formats using PHP only (no shell-outs,
no external binaries). Provides a CLI component (cli_plugin_totext) and a
help

Initial implementation of the totext plugin

Extract plain text from various file formats using PHP only (no shell-outs,
no external binaries). Provides a CLI component (cli_plugin_totext) and a
helper component (helper_plugin_totext).

Supported formats:
- OOXML: docx, xlsx, pptx
- OpenDocument: odt, ods, odp
- PDF (via bundled smalot/pdfparser)
- Plain-text family: txt, csv, md, markdown, log, text
- Image metadata (EXIF/IPTC): jpg, jpeg, tif, tiff

Includes a fixture-based test suite covering every extractor, the factory
routing, and the helper end to end.

show more ...