sqlite: avoid warnings on too short vectors

This should not happen in the real world. But when embeddings were
created with a shorter vector model than the model that is used to
embed the query, the

sqlite: avoid warnings on too short vectors

This should not happen in the real world. But when embeddings were
created with a shorter vector model than the model that is used to
embed the query, the cosineSimilarity method threw a whole bunch of
warnings. We now stop the comparison at the vector length.

In the real world the same model for embeddings and the query should be
used, results are unpredictable otherwise. So this is mostly a cosmetic
change for messed up states during development.

show more ...

add remote component to ask questions to the bot

The endpoint allows to override model and language settings on demand.

retry failed QDrant queries

hosted qdrant seems to be a bit unreliable. retrying might help.

auto codestyle cleanup

auto style fixes

emit the INDEXER_PAGE_ADD event

This allows plugins that add data to the fulltext index to add the same
data to the embeddings. This improves embedding searches with struct
data for example.

make threshold configurable

automatic stylefixes

better JSON exception handling in storages

do not hardcode dimensions in qdrant storage

fix syntax error in qdrant storage

correctly use storage setting

more refactoring on chat and embed model support

* differentiate between input and output tokens
* make use of much larger input contexts

Pinecone fixes

* should fix #12
* correctly handle JSON decoding errors
* fix empty filter requests
* match renamed API fields

Added Qdrant Storage

fixed overlong lines

�� Automatic code style fixes

Added ChromaDB vector storage support

codesniffer cleanups

�� Automatic code style fixes

generate clusters only if more than 3 clusters would be created

optionally search one language only

Added dumping of TSV files to SQLite store

This allows visualizing the embed vectors

use a k-means based cluster approach to speed up similarity searches

simplify cosine distance calculation

Since all OpenAI vectors are normalized, only the dotproduct needs to be
calculated for the distance. This saves a couple of floating point ops
per chunk, but do

simplify cosine distance calculation

Since all OpenAI vectors are normalized, only the dotproduct needs to be
calculated for the distance. This saves a couple of floating point ops
per chunk, but doesn't make a huge difference overall.

show more ...