added an AGENTS.md file for LLM based work

split sentences by token, not bytes. handle UTF-8

move text splitting into it's own class

Some enhancements on the subsentence splitting

When a sentence is longer than a chunk, it should be split forcefully in
smaller parts - these parts should NOT be the size of a full chunk since
we st

Some enhancements on the subsentence splitting

When a sentence is longer than a chunk, it should be split forcefully in
smaller parts - these parts should NOT be the size of a full chunk since
we still want to do some overlap with previous and following texts. I
chose to split into a quarter of a chunk.

This also ensures that whitespace is kept for the split sentences,
because they may be joined with follow up texts.

show more ...

Squashed commit of the following:

commit 4e0adf2a8d810e55db6d37ccc87c76d95ddcfd8d
Author: Henry <henry.krupp@gmail.com>
Date: Mon Feb 3 22:04:17 2025 +0100

Updated splitLongSentence()

commit

Squashed commit of the following:

commit 4e0adf2a8d810e55db6d37ccc87c76d95ddcfd8d
Author: Henry <henry.krupp@gmail.com>
Date: Mon Feb 3 22:04:17 2025 +0100

Updated splitLongSentence()

commit 9883844f1db6df9e11051c4c7b034e68baaca0be
Author: Henry <henry.krupp@gmail.com>
Date: Mon Feb 3 22:03:25 2025 +0100

Updated splitLongSentence()

commit 6f737f6fe4da25fa438211d5c00605c2df9c81ba
Author: Henry <henry.krupp@gmail.com>
Date: Mon Feb 3 21:43:16 2025 +0100

array_unshift($sentences, ...$this->splitLongSentence($sentence, $tiktok));

commit 21966eab02f87f632e82ad0055f9bc2aadb92053
Author: Henry <henry.krupp@gmail.com>
Date: Mon Feb 3 21:23:40 2025 +0100

Updated splitIntoChunks method

Push split sentences to the front of the queue with array_unshift($sentences, ...$this->splitLongSentence($sentence, $tiktok));

show more ...

added French Translations

upped version

check for ollama availability

move api endpoint init to AbstractModel

This should finally solve the chicken-egg issue. This also renames the
ollama url parameter from baseurl to apiurl and sets a sensible default

test: add test for Ollama model provider

test: add test for Generic model provider

Use Ionos for this test

test: add test for VoyageAI model provider

mark test incomplete on insufficient credits

test: add test for Reka model provider

test: add test for Mistral model provider

fix groq test and temperature issue

test: add test for Groq model provider

fixes based on tests, improve tests

test: add ModelGeminiTest for Gemini AI provider

first few tests to ensure model providers work

refactor HTTP Client initialization

translation update

add option to always send full page context

upped version

allow models to have a zero token limit

This allows for configuring completely unknown models. For these models
no token limit is known and we will simply do not apply any. Instead we
trust that the

allow models to have a zero token limit

This allows for configuring completely unknown models. For these models
no token limit is known and we will simply do not apply any. Instead we
trust that the model will be either large enough to handle our input or
at least throw useful error messages.

show more ...