This PR is a rework of `OpenAiTokenizer`.
Added `OpenAiTokenizerIT` with lots of tests to ensure that
`OpenAiTokenizer` calculates token usage very close to OpenAI.
In most cases calculation is 1:1, in some corner cases the difference is
within 5%.
Run `semitechnologies/weaviate` image using `GenericContainer` and
allow rapid feeback when using Weaviate. Relax `apiKey` constraint
to allow empty value for testing purposes.
Currently, IT in LocalAI module depends on a instance running.
This commit introduces the use of Testcontainers using
`GenericContainer`.
Also, due to the size of the image and the models. The image is
cached after the first run, improving next executions.
1. Added support for new QWen model features: repetitionPenalty,
temperature, stops, and incrementalOutput.
2. Implemented a new QwenTokenizer class.
3. Refactored code to utilize Lombok builder instead of manual builder
implementation.
Currently, integration tests in Ollama module are disabled because
it needs a Ollama instance running in order to execute them.
Testcontainers provides this infrastructure not only by running
the ollama container but also by automating the pull model step.
This commit, use the Singleton Container approach to reuse the single
instance across multiple IT. Also, pull step is only executed when the
image is `ollama/ollama`.
The behavior on this is:
1st execution:
1. Pull `ollama/ollama` image
2. Start the container based on `ollama/ollama` image
3. Download the `orca-mini` model
4. Create an image based on the current state (with the model in it)
5. Declare the container ready to use
6. Run test
Next executions:
1. Look for the local image created in the 1st execution
2. Start the container based on the local image
3. Declare the container ready to use
4. Run test
1st execution is expected to take longer because of the model (3GB).
Next execution are way more faster.
Currently, Milvus IT is disables because it requires a manual step
to run the compose file. Adding Testcontainers, the infrastructure
is in place to run the tests.
Current test execution depends on a running elasticserach instance,
which depends on a manual execution of docker command. This commit
introduces testcontainers as in other modules to run such instance
along with tests, improving DevEx.
`@Container` is used along with `@Testcontainers` for initialization.
The current test has `PostgreSQLContainer` annotated with `@Container`
and it is manually initialized.
The statistical logic on token usage is consistent with AiServiceStreamingResponseHandler.
And more:
When it comes to rolling messages in
MessageWindowChatMemory/TokenWindowChatMemory, LinkedList offers
superior performance. ArrayList moves all the elements when the first
element is deleted.
This PR introduces GraalVM integration to enable local execution of
JavaScript and Python scripts by Langchain4j agents.
Major updates include:
- New langchain4j-graal module for encapsulating integration logic.
- Necessary infrastructure for invoking GraalVM and executing scripts.
- Unit tests to validate the integration.
Julien Dubois