sqlite: avoid warnings on too short vectors

This should not happen in the real world. But when embeddings were
created with a shorter vector model than the model that is used to
embed the query, the

sqlite: avoid warnings on too short vectors

This should not happen in the real world. But when embeddings were
created with a shorter vector model than the model that is used to
embed the query, the cosineSimilarity method threw a whole bunch of
warnings. We now stop the comparison at the vector length.

In the real world the same model for embeddings and the query should be
used, results are unpredictable otherwise. So this is mostly a cosmetic
change for messed up states during development.

show more ...

add remote component to ask questions to the bot

The endpoint allows to override model and language settings on demand.

auto style fixes

emit the INDEXER_PAGE_ADD event

This allows plugins that add data to the fulltext index to add the same
data to the embeddings. This improves embedding searches with struct
data for example.

make threshold configurable

correctly use storage setting

more refactoring on chat and embed model support

* differentiate between input and output tokens
* make use of much larger input contexts

fixed overlong lines

�� Automatic code style fixes

codesniffer cleanups

�� Automatic code style fixes

generate clusters only if more than 3 clusters would be created

optionally search one language only

Added dumping of TSV files to SQLite store

This allows visualizing the embed vectors

Merge branch 'pineconestorage'

* pineconestorage:
implement Pinecone based storage
First go at syntax to display similar pages

use a k-means based cluster approach to speed up similarity searches

simplify cosine distance calculation

Since all OpenAI vectors are normalized, only the dotproduct needs to be
calculated for the distance. This saves a couple of floating point ops
per chunk, but do

simplify cosine distance calculation

Since all OpenAI vectors are normalized, only the dotproduct needs to be
calculated for the distance. This saves a couple of floating point ops
per chunk, but doesn't make a huge difference overall.

show more ...

First go at syntax to display similar pages

First go at syntax to display similar pages

use a cut-off point when considering similar documents

show similarity scores in CLI

refactoring to make models selectable

This makes it much easier to add new models. Models can now be selected
via the configuration