store embedding meta data on start

This way errors during the a new embedding run will not trigger a clear
index on retry

auto codestyle cleanup

init ACLs in CLI

When using the text renderer, we might execute syntax plugins that check
ACLs. We don't have a user when running the CLI but we want to index
everything an anonymous user would be a

init ACLs in CLI

When using the text renderer, we might execute syntax plugins that check
ACLs. We don't have a user when running the CLI but we want to index
everything an anonymous user would be able to see. For that the ACLs
need to be loaded.

show more ...

rebuild the index when the embedding model changed

fix info output on used models

auto style fixes

emit the INDEXER_PAGE_ADD event

This allows plugins that add data to the fulltext index to add the same
data to the embeddings. This improves embedding searches with struct
data for example.

mechanisms to override things on command line

This should help with debugging/evaluating

various refactoring and introduction of a simulate command

The new command makes it easier to run the same chat questions against
multiple models and compare the results in a spreadsheet

separate the rephrasing model from the chat model

Rephrasing can be done with faster, simpler models as there is not much
reasoning needed.

fix info output

fix ouput prices

added Voyage AI for embeddings

added model command to CLI

This prints info about the available models

more refactoring on chat and embed model support

* differentiate between input and output tokens
* make use of much larger input contexts

First start on refactoring the class hierarchy

This splits embedding models from chat completion models.

record the times of embed and maintenance runs

This makes it easier to debug when something with the cronjob goes
wrong. Currently the data is only exposed in the cli info command.

We might want to

record the times of embed and maintenance runs

This makes it easier to debug when something with the cronjob goes
wrong. Currently the data is only exposed in the cli info command.

We might want to use it somewhere in the UI to warn about outdated data

show more ...

�� Automatic code style fixes

Regular expressions to limit the indexed pages. Implements #5

Both regular expressions (when set) need to apply at the same time. Eg a
page MUST match the matchRegex and MUST NOT match the skipRegex

Regular expressions to limit the indexed pages. Implements #5

Both regular expressions (when set) need to apply at the same time. Eg a
page MUST match the matchRegex and MUST NOT match the skipRegex to be
applied.

The regular expressions are applied when running the `embed` command
line command. Pages no longer adhering to a changed regex setup will be
removed from the vector store.

For the sqlite storage it is recommended to re-cluster the index when
the reges are changed by running the `maintenance` command.

show more ...

added chunk dumping to CLI page command

�� Automatic code style fixes

optionally search one language only

Added dumping of TSV files to SQLite store

This allows visualizing the embed vectors

Merge branch 'pineconestorage'

* pineconestorage:
implement Pinecone based storage
First go at syntax to display similar pages

use a k-means based cluster approach to speed up similarity searches