1# AGENTS.md 2 3This file provides guidance to AI agents when working with code in this repository. 4 5**Always update this file automatically when you learn new things about the code base!** 6 7## Project Overview 8 9This is the totext DokuWiki plugin. It extracts plain text from various file 10formats using **PHP only** — no shell-outs and no external binaries. 11 12It exposes two entry points: 13 141. **CLI component** (`cli.php`, class `cli_plugin_totext`) — 15 `php bin/plugin.php totext <file>` prints the body text followed by the 16 metadata (`Key: value` lines, separated by a blank line) to STDOUT. `--text` 17 (`-t`) prints only the body text; `--meta` (`-m`) prints only the metadata; 18 the two are mutually exclusive "only this" switches. If a requested half fails 19 but the other is salvaged, it warns on STDERR and still prints the salvaged 20 half (exit 0); if *nothing* usable came through for what was requested it 21 re-throws, so the CLI exception handler reports it and exits non-zero. 222. **Helper component** (`helper.php`, class `helper_plugin_totext`) — gives other 23 plugins the same functionality via `plugin_load('helper', 'totext')`: 24 `extract($path): ExtractionResult` (both outputs; never throws for a *partial* 25 failure — inspect `->textError`/`->metadataError`), `extractMetadata($path): 26 array` (throws if metadata failed), and `extractText($path): string` (throws 27 if text failed — preserves the throw-on-failure contract the docsearch plugin 28 relies on to fall back to its own converters). 29 30### Architecture 31 32- `Extractor/ExtractorInterface.php` — the contract every extractor implements. 33 It is deliberately just `extract(): ExtractionResult`: extractors are dumb 34 workers and do not know their own extensions. All extension knowledge lives in 35 the factory. 36- `Extractor/ExtractionResult.php` — the `final` readonly value object every 37 extractor returns: `string $text` (body text), `array<string,string> 38 $metadata` (canonical key → value map), and `?\Throwable $textError` / 39 `?\Throwable $metadataError` recording a per-half failure (plus `isComplete()`). 40 Produced from **one** parse of the file. See *Metadata* below for the 41 vocabulary and *Failure model* below for the error semantics. 42- `Extractor/ExtractorFactory.php` — the sole routing authority (`forFile()`, 43 `extract()` → `ExtractionResult`, `supportedExtensions()`). Its `EXTRACTORS` 44 constant (extension → extractor class) is the single source of truth: 45 `forFile()` looks the file's extension up in it and `supportedExtensions()` 46 returns its keys. Add a new format with one new extractor class plus one entry 47 in that map — extensions are never written down anywhere else. 48- `Extractor/AbstractZipXmlExtractor.php` — shared base for all ZIP-of-XML formats 49 (OOXML *and* OpenDocument). `extract()` does one unzip — **opening the archive 50 is the total-failure gate**: if it can't be unpacked, nothing is recoverable 51 and it throws. Once open, the abstract `extractText()` and `extractMetadata()` 52 run as independent halves; whichever throws has its error recorded on the 53 result (normalised to an `ExtractionException` via `ExtractionException::wrap()`, 54 the shared wrap-a-caught-error helper) while the other half is still returned 55 (see *Failure model*). Provides 56 `readPart()`, `listParts()`, temp-dir handling, two streaming `XMLReader` text 57 walkers — `extractTextFromXml()` (text in a wrapper element, used by OOXML) and 58 `extractAllTextFromXml()` (text as character data, used by OpenDocument) — and 59 the generic `mapMetadataFromXml($xml, $map, $multiValueKeys)` primitive (walks 60 a metadata part, matching element **local names** to canonical keys, dropping 61 empty values, accumulating multi-value keys). The unpack temp dir is created in 62 DokuWiki's own temp dir (`$conf['tmpdir']`) via core's `io_mktmpdir()` and 63 removed with core's `io_rmdir($dir, true)` — never the system temp dir. 64 `extract()` removes it promptly in a `finally` block; the class `__destruct()` 65 repeats the cleanup as a safety net so the dir is gone even if the process 66 dies (e.g. a fatal error) before `finally` runs. 67- `Extractor/AbstractOoxmlExtractor.php` / `Extractor/AbstractOdfExtractor.php` — 68 intermediary classes between the ZIP base and the concrete extractors. They 69 exist solely to declare each family's metadata source **once**: OOXML reads 70 `docProps/core.xml` (`CORE_META_MAP`) + `docProps/app.xml` (`APP_META_MAP`); 71 ODF reads `meta.xml` (`META_MAP`, with `meta:keyword` accumulating into 72 `Keywords`). There is no family autodetection — each concrete ZIP extractor 73 simply `extends` the matching intermediary and carries no metadata code. 74- Concrete extractors: `DocxExtractor`/`XlsxExtractor`/`PptxExtractor` (extend 75 `AbstractOoxmlExtractor`), `OdtExtractor`/`OdsExtractor`/`OdpExtractor` (extend 76 `AbstractOdfExtractor`), `PdfExtractor`, `TextExtractor` (txt/csv/md/log/...), 77 `ImageExtractor` (EXIF/IPTC metadata of jpg/tiff — metadata only, not OCR). 78- `Exception/ExtractionException.php` and `Exception/UnsupportedFormatException.php` 79 (the latter extends the former). The helper and factory **throw**; the CLI does 80 not catch, relying on `splitbrain\phpcli\CLI` to print the message and exit non-zero. 81 82### Metadata 83 84`extract()` returns a metadata map keyed by a **single canonical vocabulary**, 85identical across all formats so consumers never special-case the source: 86 87`Title, Author, Subject, Keywords, Description, Created, Modified, Language, 88Producer` (all formats) plus `Copyright` (**image-only**). Values are non-empty 89UTF-8 strings; empty values are dropped (never stored as blank keys). 90 91- **`Producer`** is "what produced this file" — the authoring application for 92 office/PDF (PDF: `Producer`, else `Creator`) **and** the camera/software for an 93 image (`Simple.Camera` / EXIF `Model`). There is intentionally no separate 94 `Camera` key. 95- **`Copyright` is image-only by spec, not by accident.** OOXML 96 `CT_CoreProperties` (ECMA-376 / ISO 29500) has no `dc:rights`; ODF 97 `<office:meta>` has no `dc:rights` either; the PDF Info dictionary has no 98 copyright key (it lives only in an XMP stream prinsfrank does not surface). 99 Only IPTC/EXIF have dedicated copyright fields, which `ImageExtractor` reads. 100- Each family's source map is declared **once** in its intermediary class 101 (`AbstractOoxmlExtractor`, `AbstractOdfExtractor`); `PdfExtractor` and 102 `ImageExtractor` map their own native fields. The generic 103 `mapMetadataFromXml()` matches by element **local name** (namespace-agnostic) — 104 safe because OOXML core/app and ODF meta use distinct local names. 105- **Image behavior change:** `extractText()` on an image now returns `''` (its 106 descriptive fields moved into `metadata`), not the old `"Caption: …\nAuthor: 107 …"` text block. `ImageExtractor`'s field-map keys were renamed to the canonical 108 vocabulary (`Caption`→`Description`, `Date`→`Created`, `Camera`→`Producer`). 109 110### Failure model 111 112Text and metadata are extracted **independently**, so one half can fail while 113the other is salvaged. Each extractor has a single **total-failure gate** — the 114step that, if it fails, leaves *nothing* recoverable (the container won't unzip, 115the PDF won't parse, the file can't be read). The gate throws an 116`ExtractionException`. Everything after the gate is a per-half best-effort: 117 118- A half that throws has its error recorded in `ExtractionResult::$textError` / 119 `$metadataError`; its output is left empty (`''` / `[]`) and the *other* half 120 is still returned. `extract()` itself does **not** throw for a partial failure. 121- An output that is empty **by design** is not a failure and leaves its error 122 `null`: images have no body text (`text === ''`, `textError === null`), plain 123 text has no metadata (`metadata === []`, `metadataError === null`). Likewise a 124 document that simply carries no metadata yields `[]` with no error — only an 125 actual read/parse exception sets `metadataError`. 126- Single-output formats fold into this cleanly: their one product *is* the gate. 127 `TextExtractor` (text-only) and `ImageExtractor` (metadata-only) throw on 128 failure and never set the error fields. 129- The helper's `extractText()` / `extractMetadata()` **re-throw** the relevant 130 half's recorded error, so a single-output caller keeps its throw-on-failure 131 contract even when the other half was salvaged. The CLI warns on STDERR about 132 a failed half but still prints the salvaged half (exit 0); only when *nothing* 133 usable came through for what was requested does it re-throw (non-zero exit). 134 135### Dependencies 136 137- `prinsfrank/pdfparser` (v3.x, MIT, zero PHP dependencies — pulls only 138 `prinsfrank/glyph-lists`) is bundled in the committed `vendor/` directory (pulled 139 via the plugin's own `composer.json`). DokuWiki core auto-requires 140 `lib/plugins/totext/vendor/autoload.php` for enabled plugins. `PdfExtractor` 141 parses once via `(new PrinsFrank\PdfParser\PdfParser())->parseFile($path)`, takes 142 the body from `->getText()` and reads the Info dictionary 143 (`getInformationDictionary()`) best-effort for metadata — the default in-memory 144 mode, which benchmarked both faster and far lighter than the previous 145 smalot/cosmocode fork (no `setRetainImageContent` tuning needed). It requires 146 `ext-gd`/`ext-iconv`/`ext-zlib`, enforced transitively by the package. 147- **UTF-16BE Info-string shim** (`PdfExtractor::normalizePdfString()`): prinsfrank 148 ≤ v3.1.0 does not decode UTF-16BE text strings stored as PDF *literal* strings. 149 Their bytes arrive expanded one-codepoint-per-byte (`mb_chr` on the octal 150 escapes), so a UTF-16BE BOM surfaces as the mojibake `þÿ` 151 (`0xC3 0xBE 0xC3 0xBF`); the shim collapses that back to ISO-8859-1 to recover 152 the raw `0xFE 0xFF` BOM, then `iconv`s UTF-16BE→UTF-8. (`tika-sample.pdf`'s 153 Author `Bertrand Delacrétaz` exercises this.) A genuinely raw BOM is handled 154 too. **Remove once decoding is fixed upstream.** 155- **Known limitation:** prinsfrank/pdfparser ≤ v3.1.0 does **not** extract text that 156 lives inside a Form XObject (page content painted with the `Do` operator — common 157 in Quartz/macOS, Firefox and Chrome PDFs, including `_test/data/tika-sample.pdf`, 158 so `PdfExtractorTest::testExtractsText` and the factory roundtrip's `pdf` case 159 fail against stock v3.1.0). This is the **body text** only — the PDF **metadata** 160 test passes regardless, because the Info dictionary is read independently of 161 `getText()`. 162- `splitbrain\PHPArchive\Zip` is **not** bundled — core provides it globally. 163- Text encoding defers to core's `dokuwiki\Utf8\Clean` / `dokuwiki\Utf8\Conversion`; 164 JPEG metadata uses core's `JpegMeta`; TIFF metadata uses the `exif` extension. 165 166Legacy binary Office formats (`.doc`/`.xls`/`.ppt`) are intentionally unsupported. 167 168## Automated Testing 169 170Tests run via DokuWiki's PHPUnit-based testing framework. The calls MUST be made from within the plugin's repository root using a relative path to the `bin/plugin.php` script! 171 172```bash 173# Tests must be run from repository root 174../../../bin/plugin.php dev test 175 176# run individual test file 177../../../bin/plugin.php dev test _test/GeneralTest.php 178 179# create a new test file 180../../../bin/plugin.php dev addTest MyClass 181``` 182 183DokuWiki provides useful helper methods for testing: 184 185* `DokuWikiTest::getInaccessibleProperty()` to access private/protected properties 186* `DokuWikiTest::callInaccessibleMethod` to execute private/protected methods 187* read `../../../_test/core/DokuWikiTest.php` for more helper methods 188* use `../../../_test/TestRequest.php` to simulate HTTP requests for integration tests 189* use `../../../_test/phpQuery-onefile.php` if you need to parse HTML in tests 190 191Each test run will provide a fresh DokuWiki instance in a temporary directory via the default setupBeforeClass methods. 192 193### Test fixtures 194 195**All tests run against real files — never hand-built containers, never 196synthetic content.** A hand-authored fixture only encodes our *belief* about a 197format; if that belief is wrong the extractor is written to match and both agree 198on a fiction. So every sample is a genuine in-the-wild document, taken verbatim 199from the **Apache Tika test corpus** (Apache-2.0) and committed under 200`_test/data`. `_test` is `export-ignore`d, so these binaries never ship in 201release tarballs — only the git repo holds them. 202 203* **Provenance** — `_test/data/README.md` records, for every committed file, its 204 original Tika name + upstream path + license. Keep it accurate when files 205 change: committing third-party binaries here is fine *only* with that record. 206* **Refresh** — `_test/data/download.sh` re-downloads each file from its exact 207 upstream Tika path (needs only `curl`). The committed copies are authoritative; 208 the script just refreshes them and must reproduce them byte-for-byte. Every 209 committed file keeps a `tika-` filename prefix to mark its origin. 210* **What the samples cover** — `tika-sample.docx` (titles, headings, nested 211 tables, hyperlinks, custom style, header/footer); `tika-various.docx` (lists, 212 footnotes, Japanese + four-byte Gothic = multibyte UTF-8); `tika-sample.xlsx` 213 (three named sheets, tab-separated cells); `tika-sample.pptx` (three ordered 214 slides); `tika-various.pptx` (speaker notes); `tika-sample.odt` (one 215 paragraph); `tika-sample.ods` (numeric grid); `tika-sample.odp` (two slides); 216 `tika-sample.pdf` (the Tika homepage, full prose); `tika-meta.jpg` (Photoshop 217 IPTC caption/by-line/keywords + EXIF camera — IPTC is non-UTF-8, so only 218 ASCII-safe substrings are asserted); `tika-meta.tiff` (EXIF ImageDescription); 219 `tika-plain.jpg` (no metadata → empty string); `tika-sample.txt` (multilingual 220 UTF-8 pangram). 221* **Error-path tests** — derived from real files, not fabricated: `Samples.php` 222 exposes `withoutPart()` (unpack a real container minus one ZIP member and repack 223 it via php-archive, e.g. a DOCX missing `word/document.xml`) and `corrupt()` 224 (non-archive bytes); both manage their own temp files, and `path()` resolves a 225 committed sample. There is no fixture *builder*. Note the failure model (above): 226 `corrupt()` trips the total-failure gate so `extract()` **throws**, whereas 227 `withoutPart()` removing only the body part (`word/document.xml`, `content.xml`) 228 leaves the container openable, so `extract()` returns a **partial result** — 229 `textError` set, the independent metadata (`docProps/*`, `meta.xml`) salvaged. 230 `HelperTest` separately locks the `extractText()` re-throw contract. 231* **Edges no clean real file covers are not tested** — deliberately dropped when 232 the corpus had no document that targets them without inventing structure: XLSX 233 custom sheet-name↔file *re*ordering (only normal multi-sheet order is tested), 234 ODS merged/covered cells + multi-paragraph cells, and the Windows XP UTF-16LE 235 EXIF tag. Asserting these would mean asserting against invented inputs. 236 237### Format-specific gotchas learned 238 239* **XLSX sheet order/names** must be resolved through `xl/workbook.xml` + 240 `xl/_rels/workbook.xml.rels` (name → r:id → worksheet file). Worksheet file 241 numbering does NOT have to match tab order, so pairing sorted filenames with 242 names positionally mismatches them. `XlsxExtractor` follows the relationships 243 and only falls back to positional/`SheetN` naming when the workbook or its 244 rels are absent. 245* **EXIF "XP" tags** (TIFF): PHP's `exif_read_data` already decodes the Windows 246 XP tags and exposes them in a `WINXP` section under the short names 247 `Title`/`Comment`/`Author`/`Keywords`/`Subject` (UTF-8) — NOT as `XPTitle` 248 etc. `ImageExtractor::EXIF_FIELDS` lists the short names first and keeps the 249 `XP*` raw names (decoded from UTF-16LE by `normaliseExifValue()`) as a 250 cross-version fallback. This path still exists but is **no longer covered by a 251 fixture** — no real image in the corpus carries Windows XP tags. 252 253**Important:** Test classes that need the plugin must set `protected $pluginsEnabled = ['totext'];` to enable it in the test environment. 254 255**Important:** `setUp()` and `tearDown()` methods must be `public` (not `protected`) to match the `DokuWikiTest` base class. 256 257 258## Caching 259 260DokuWiki may cache JavaScript, CSS and rendered output. To reset the cache just touch the config file 261 262```bash 263touch ../../../conf/local.php 264``` 265 266## Linting, Formatting and Conventions 267 268Adhere to PSR-12 coding standards. Always add proper docblocks with descriptions, parameter types, and return types to all classes, methods and functions. 269 270```bash 271# Lint PHP files using PHP_CodeSniffer (must be run from repo root) 272../../../bin/plugin.php dev check 273 274# Auto-Fix formatting issues using PHP_CBF and Rector (must be run from repo root) 275../../../bin/plugin.php dev fix 276``` 277 278## Plugin Architecture 279 280Inspect the base plugin classes in `../../../inc/Extension/` to learn about the plugin system architecture. 281 282```bash 283# add new plugin components (must be run from repo root) 284../../../bin/plugin.php dev addComponent <type> 285# e.g. 286../../../bin/plugin.php dev addComponent action 287# if multiple of the same type are needed, give a name: 288../../../bin/plugin.php dev addComponent action foobar 289# -> creates action/foobar.php 290``` 291 292Additional classes are autoloaded when using the `dokuwiki\plugin\totext` namespace. 293
