1# AGENTS.md
2
3This file provides guidance to LLM Code Agents when working with code in this repository.
4
5## Overview
6
7This is a DokuWiki plugin that enables AI-powered chat functionality using LLMs (Large Language Models) and RAG (Retrieval-Augmented Generation). The plugin indexes wiki pages as embeddings in a vector database and allows users to ask questions about wiki content.
8
9## Development Commands
10
11### Testing
12
13```bash
14../../../bin/plugin.php dev test
15```
16
17The command does not accept any additional arguments or parameters and runs all tests in the `_test/` directory.
18
19PHPUnit can also be called directly, when special options are needed:
20
21```bash
22../../../_test/vendor/bin/phpunit -c ../../../_test/phpunit.xml --group plugin_aichat
23```
24
25### CLI Commands
26The plugin provides a CLI interface via `cli.php`:
27
28```bash
29# Get a list of available commands
30../../../bin/plugin.php aichat --help
31```
32
33## Architecture
34
35### Core Components
36
37**helper.php (helper_plugin_aichat)**
38- Main entry point for plugin functionality
39- Manages model factory and configuration
40- Handles question answering with context retrieval
41- Prepares messages with chat history and token limits
42- Implements question rephrasing for better context search
43
44**Embeddings.php**
45- Manages the vector embeddings index
46- Splits pages into chunks using TextSplitter
47- Creates and retrieves embeddings via embedding models
48- Performs similarity searches through storage backends
49- Handles incremental indexing (only updates changed pages)
50
51**TextSplitter.php**
52- Splits text into token-sized chunks (configurable, typically ~1000 tokens)
53- Prefers sentence boundaries using Vanderlee\Sentence
54- Handles long sentences by splitting at word boundaries
55- Maintains overlap between chunks (MAX_OVERLAP_LEN = 200 tokens) for context preservation
56
57**ModelFactory.php**
58- Creates and caches model instances (chat, rephrase, embedding)
59- Loads model configurations from Model/*/models.json files
60- Supports multiple providers: OpenAI, Gemini, Anthropic, Mistral, Ollama, Groq, Reka, VoyageAI
61
62### Model System
63
64**Model/AbstractModel.php**
65- Base class for all LLM implementations
66- Handles API communication with retry logic (MAX_RETRIES = 3)
67- Tracks usage statistics (tokens, costs, time, requests)
68- Implements debug mode for API inspection
69- Uses DokuHTTPClient for HTTP requests
70
71**Model Interfaces**
72- `ChatInterface`: For conversational models (getAnswer method)
73- `EmbeddingInterface`: For embedding models (getEmbedding method, getDimensions method)
74- `ModelInterface`: Base interface with token limits and pricing info
75
76**Model Providers**
77Each provider has its own namespace under Model/:
78- OpenAI/, Gemini/, Anthropic/, Mistral/, Ollama/, Groq/, Reka/, VoyageAI/
79- Each contains ChatModel.php and/or EmbeddingModel.php
80- Model info (token limits, pricing, dimensions) defined in models.json
81
82### Storage Backends
83
84**Storage/AbstractStorage.php**
85- Abstract base for vector storage implementations
86- Defines interface for chunk storage and similarity search
87
88**Available Implementations:**
89- SQLiteStorage: Local SQLite database
90- ChromaStorage: Chroma vector database
91- PineconeStorage: Pinecone cloud service
92- QdrantStorage: Qdrant vector database
93
94### Data Flow
95
961. **Indexing**: Pages → TextSplitter → Chunks → EmbeddingModel → Vector Storage
972. **Querying**: Question → EmbeddingModel → Vector → Storage.getSimilarChunks() → Filtered Chunks
983. **Chat**: Question + History + Context Chunks → ChatModel → Answer
99
100### Key Features
101
102**Question Rephrasing**
103- Converts follow-up questions into standalone questions using chat history
104- Controlled by `rephraseHistory` config (number of history entries to use)
105- Only applied when rephraseHistory > chatHistory to avoid redundancy
106
107**Context Management**
108- Chunks include breadcrumb trail (namespace hierarchy + page title)
109- Token counting uses tiktoken-php for accurate limits
110- Respects model's max input token length
111- Filters chunks by ACL permissions and similarity threshold
112
113**Language Support**
114- `preferUIlanguage` setting controls language behavior:
115  - LANG_AUTO_ALL: Auto-detect from question
116  - LANG_UI_ALL: Always use UI language
117  - LANG_UI_LIMITED: Use UI language and limit sources to that language
118
119### AJAX Integration
120
121**action.php**
122- Handles `AJAX_CALL_UNKNOWN` event for 'aichat' calls
123- Processes questions with chat history
124- Returns JSON with answer (as rendered Markdown), sources, and similarity scores
125- Implements access restrictions via helper->userMayAccess()
126- Optional logging of all interactions
127
128### Frontend
129- **script/**: JavaScript for UI integration
130- **syntax/**: DokuWiki syntax components
131- **renderer.php**: Custom renderer for AI chat output
132
133## Configuration
134
135Plugin configuration is in `conf/`:
136- **default.php**: Default config values
137- **metadata.php**: Config field definitions and validation
138
139Key settings:
140- Model selection: chatmodel, rephrasemodel, embedmodel
141- Storage: storage backend type
142- API keys: openai_apikey, gemini_apikey, etc.
143- Chunk settings: chunkSize, contextChunks, similarityThreshold
144- History: chatHistory, rephraseHistory
145- Access: restrict (user/group restrictions)
146- Indexing filters: skipRegex, matchRegex
147
148## Testing
149
150Tests are in `_test/` directory:
151- Extends DokuWikiTest base class
152- Uses @group plugin_aichat annotation
153
154## Important Implementation Notes
155
156- All token counting uses TikToken encoder for rough estimates
157- Chunk IDs are calculated as: pageID * 100 + chunk_sequence (pageIDs come from DokuWiki's internal search index)
158- Models are cached in ModelFactory to avoid re-initialization
159- API retries use exponential backoff (sleep for retry count seconds)
160- Breadcrumb trails provide context to AI without requiring full page content
161- Storage backends handle similarity search differently but provide unified interface
162- UTF-8 handling is critical for text splitting (uses dokuwiki\Utf8\PhpString)
163