Changes in AGENTS.md

fd4f7c47d290ec79c2b911626d1d262c1e3d161e - Track the cosmocode prinsfrank/pdfparser dev fork

Mon, 06 Jul 2026 07:46:54 +0000

Track the cosmocode prinsfrank/pdfparser dev forkThe dev fork adds native UTF-16BE Info-string decoding and Form-XObjecttext extraction. Point composer at the fork's VCS repo, pin dev-dev, anddrop the now-redundant normalizePdfString() UTF-16BE shim (metadata valuesjust get trimmed). Refresh AGENTS.md accordingly. List of files: /plugin/totext/AGENTS.md

0fbee1892eb8b7339ce48e27e48e3fc821a8695f - Return body text and metadata from a single extraction

Mon, 22 Jun 2026 13:32:54 +0000

Return body text and metadata from a single extractionEvery extractor now returns an ExtractionResult (body text plus anormalised metadata map) from one parse of the file, instead of just atext string. Metadata keys are a single canonical vocabulary sharedacross all formats - Title, Author, Subject, Keywords, Description,Created, Modified, Language, Producer, plus Copyright (image-only) - soconsumers never have to special-case the source format.- ExtractionResult: final readonly value object carrying $text, $metadata and a per-half ?Throwable $textError / $metadataError (with isComplete()). Text and metadata are extracted independently: an extractor throws only at its total-failure gate (container won't unzip, PDF won't parse, file unreadable) and otherwise records a failed half's error while still returning the salvaged half.- OOXML/ODF metadata: new AbstractOoxmlExtractor and AbstractOdfExtractor intermediaries declare each family's source map once (docProps/core.xml + app.xml; meta.xml); the concrete extractors only switch their extends clause. AbstractZipXmlExtractor gains a generic mapMetadataFromXml() walker.- PDF: reads the Info dictionary (Title/Author/Subject/Keywords/Created/ Modified/Producer-else-Creator) alongside the body text from the same parse. normalizePdfString() is a temporary shim for UTF-16BE literal strings that prinsfrank <= v3.1.0 returns mojibake-expanded.- Images: now metadata-only (text is ''); field-map keys renamed to the canonical vocabulary (Caption->Description, Date->Created, Camera->Producer).- helper: extract() and extractMetadata() added; extractText() keeps its string return and its throw-on-failure contract (rethrows textError), so the docsearch plugin keeps working unchanged.- cli: prints body text followed by metadata by default; --text/-t and --meta/-m restrict to one. A salvaged-but-partial extraction warns on STDERR and exits 0; nothing usable exits non-zero.- ExtractionException::wrap() centralises wrapping a caught error as an ExtractionException (passing one through if it already is).Tests assert the real Tika samples' embedded metadata and thepartial-failure paths. The pre-existing PDF body-text Form-XObjectfailure is unchanged. List of files: /plugin/totext/AGENTS.md

b4ecfc82b02c9a76205381a28b47006474ccf569 - Replace smalot/pdfparser with prinsfrank/pdfparser

Mon, 22 Jun 2026 11:03:02 +0000

Replace smalot/pdfparser with prinsfrank/pdfparserSwitch the PDF extractor from the cosmocode/pdfparser memory fork(an LGPL smalot/pdfparser fork) to prinsfrank/pdfparser (MIT, builtfrom scratch, zero PHP dependencies beyond prinsfrank/glyph-lists).PdfExtractor now calls parseFile($path)->getText() in the defaultin-memory mode, which benchmarked both faster and far lighter than theold fork (on a 12 MB/92-page file: 63s/49.5MB vs 68s/100MB), so theprevious setRetainImageContent() tuning is no longer needed.Known limitation: prinsfrank/pdfparser <= v3.1.0 does not extract textinside Form XObjects (page content painted with the Do operator, commonin Quartz/Firefox/Chrome PDFs such as the Tika sample), soPdfExtractorTest::testExtractsText fails against stock v3.1.0. List of files: /plugin/totext/AGENTS.md

d8f4c70ae207510226ae1b1d6bf10a803069a4e4 - Switch to the cosmocode/pdfparser memory fork

Thu, 18 Jun 2026 21:03:07 +0000

Switch to the cosmocode/pdfparser memory forkReplace the upstream smalot/pdfparser dependency with our own fork(cosmocode/pdfparser, dev-memory branch) for its lower-peak-memorystreaming parser. The fork keeps the smalot/pdfparser package name, soit is wired in as a vcs repository in composer.json and required asdev-memory; the vendored copy and lock file are regenerated accordingly.PdfExtractor is unchanged - the public API it uses(Config::setRetainImageContent(), Parser::parseFile()) is identical. List of files: /plugin/totext/AGENTS.md

a5b145eb87e27f15d3d99ca28dabf3f8331db2c0 - Use real Apache Tika sample files and trim the test suite

Thu, 18 Jun 2026 09:41:19 +0000

Use real Apache Tika sample files and trim the test suiteReplace the generated fixtures with real documents from the Apache Tikatest corpus (tika-* files plus download.sh and a short README) and reworkthe tests to run against them:- Samples.php: drop the tempDir()/cleanup() single-call wrappers, let withoutPart() and corrupt() manage their own temp files, and build the "container missing a member" input with php-archive instead of ext-zip.- ExtractorFactoryTest: stop re-typing the EXTRACTORS map; keep only the case/path normalisation cases and own the missing-file/corrupt error paths centrally.- Per-format extractor tests: drop the duplicated shared error tests and their setUp/tearDown, keeping the format-specific assertions; move the multibyte-UTF8 check into DocxExtractorTest.- Remove HelperTest and SampleFilesTest as redundant.- Add a real-sample test for the plain-text extractor. List of files: /plugin/totext/AGENTS.md

6b6a7c2c4f3e5c6b7f0466abf36e5ee130b06f7a - Test against real files instead of generated fixtures

Wed, 17 Jun 2026 21:44:34 +0000

Test against real files instead of generated fixturesA hand-built fixture only encodes our belief about a file format; if thatbelief is wrong the extractor is written to match it and both agree on afiction that never matches reality. Replace the synthetic FixtureBuilder withreal files produced by real applications.- _test/data/regenerate.sh rebuilds the edge-case samples from inline plain-text sources: LibreOffice converts flat-ODF to the office formats, ImageMagick + exiftool produce images with real IPTC/EXIF metadata.- New samples cover headers/footers, tab/line-break handling, multi-sheet name resolution, speaker notes, merged/covered cells and image metadata.- Error-path tests derive broken inputs from real files (Samples::withoutPart removes one ZIP member, Samples::corrupt writes garbage) rather than fabricating containers.- Scenarios real tools never emit (unnamed sheet, repeated filled-text cell, the synthetic MAX_REPEAT extreme) are dropped rather than faked.Delete FixtureBuilder; add the Samples helper. _test is export-ignored so thesample binaries never ship in release tarballs. List of files: /plugin/totext/AGENTS.md

a67d0155efa368279db14d1247aa65c19b51d1ce - Make ExtractorFactory the single source of supported extensions

Wed, 17 Jun 2026 20:20:02 +0000

Make ExtractorFactory the single source of supported extensionsThe file extensions were duplicated across forFile(), supportedExtensions()and every extractor's supports()/extension(). Collapse them into oneprivate const map (extension => extractor class) in ExtractorFactory, whichis now the sole routing authority: forFile() looks the extension up andsupportedExtensions() returns its keys.Extractors are now dumb workers implementing only extract(); the unusedsupports()/extensions() methods are dropped from ExtractorInterface and allextractors. Per-extractor supports() tests are removed (routing is coveredby the factory tests). List of files: /plugin/totext/AGENTS.md

37c5feaa1dc3460cf196c7beda3a3c10741397bb - Use DokuWiki's temp dir and clean up in destructor

Wed, 17 Jun 2026 13:47:46 +0000

Use DokuWiki's temp dir and clean up in destructorAllocate the archive-unpack temp dir in DokuWiki's own temp dir($conf['tmpdir']) via core's io_mktmpdir(), and remove it withio_rmdir(), instead of the system temp dir. Add a destructor thatrepeats the cleanup as a safety net so the dir is removed even if theprocess dies before extract()'s finally block runs. Align the testFixtureBuilder on the same core helpers. List of files: /plugin/totext/AGENTS.md

e3bf24fb9dfc9f06d2338a5c5cb7500e628e4f71 - Expand test coverage and fix two extractor bugs

Wed, 17 Jun 2026 13:30:24 +0000

Expand test coverage and fix two extractor bugsBroaden the test suite from happy-path fixtures to the branching logic thatwas previously unexercised, and fix two real bugs the new coverage exposed:- XlsxExtractor: resolve sheet name->file mapping and tab order through xl/workbook.xml + xl/_rels/workbook.xml.rels instead of pairing sorted worksheet filenames with names positionally, which mismatched them whenever tab order differed from file numbering. Falls back to positional/SheetN naming when the workbook or its rels are absent.- ImageExtractor: PHP's exif reader decodes Windows XP tags into a WINXP section under short names (Title/Comment/Author/Keywords/Subject), not as XPTitle etc., so those fields were silently dropped. List the short names first, keep the XP* raw names as a cross-version fallback.Tests:- FixtureBuilder gains DOCX header/footer, PPTX notes, multi-sheet XLSX (+ no-workbook/no-worksheet variants), rich ODS (repeated columns, covered cells, multi-paragraph cells, MAX_REPEAT cap, unnamed-sheet fallback), a hand-packed TIFF with EXIF incl. a UTF-16LE XPTitle, and a public zip() helper for composing broken containers.- New critical-path coverage: header/footer, speaker notes, ODS edge cases, XLSX multi-sheet/fallback/no-worksheet guards, EXIF/TIFF incl. XP decode, and per-format missing-part/missing-file error paths.- Routing, reject and supports() tests converted to data providers, plus a drift guard that routes every supportedExtensions() entry through forFile().- Real-world LibreOffice samples under _test/data covered by SampleFilesTest.47 -> 155 tests, 104 -> 231 assertions. List of files: /plugin/totext/AGENTS.md

ca76a2e67052f7370ed3571784ecdc865c0c30b4 - Initial implementation of the totext plugin

Wed, 17 Jun 2026 11:42:17 +0000

Initial implementation of the totext pluginExtract plain text from various file formats using PHP only (no shell-outs,no external binaries). Provides a CLI component (cli_plugin_totext) and ahelper component (helper_plugin_totext).Supported formats:- OOXML: docx, xlsx, pptx- OpenDocument: odt, ods, odp- PDF (via bundled smalot/pdfparser)- Plain-text family: txt, csv, md, markdown, log, text- Image metadata (EXIF/IPTC): jpg, jpeg, tif, tiffIncludes a fixture-based test suite covering every extractor, the factoryrouting, and the helper end to end. List of files: /plugin/totext/AGENTS.md