This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
| [actions/checkout](https://redirect.github.com/actions/checkout)
([changelog](de0fac2e45..df4cb1c069))
| action | digest | `de0fac2` → `df4cb1c` |
---
> [!WARNING]
> Some dependencies could not be looked up. Check the [Dependency
Dashboard](../issues/2069) for more information.
---
### Configuration
📅 **Schedule**: (UTC)
- Branch creation
- At any time (no schedule defined)
- Automerge
- At any time (no schedule defined)
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/langchain4j/langchain4j).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNDIuMiIsInVwZGF0ZWRJblZlciI6IjQzLjI0Mi4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiXX0=-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
This PR contains the following updates:
| Package | Type | Update | Change |
|---|---|---|---|
|
[mikepenz/action-junit-report](https://redirect.github.com/mikepenz/action-junit-report)
| action | major | `v4` -> `v5` |
---
> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.
---
### Release Notes
<details>
<summary>mikepenz/action-junit-report
(mikepenz/action-junit-report)</summary>
###
[`v5`](https://redirect.github.com/mikepenz/action-junit-report/releases/tag/v5)
[Compare
Source](https://redirect.github.com/mikepenz/action-junit-report/compare/v4...v5)
- no changes
</details>
---
### Configuration
📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/langchain4j/langchain4j).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0MS4xNy4yIiwidXBkYXRlZEluVmVyIjoiNDEuMTcuMiIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiZGVwZW5kZW5jaWVzIl19-->
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Add [JUnit Report](https://github.com/mikepenz/action-junit-report)
## Issue
Improve reporting in CI workflows
<img width="561" alt="image"
src="https://github.com/user-attachments/assets/190023e3-2715-4199-b86b-c94aaeab62e3">
## Change
This pull request introduces changes to multiple GitHub Actions
workflows to consistently publish test reports using the
`mikepenz/action-junit-report@v4` action. The updates ensure that test
reports are published regardless of the success or failure of the
previous steps.
Changes to GitHub Actions workflows:
*
[`.github/workflows/main.yaml`](diffhunk://#diff-71cabc4177e41ea8f15a89eb65398fa8187739f68a2762706d84885cd8b2ddc3R109-R115):
Added a step to publish test reports using
`mikepenz/action-junit-report@v4`, ensuring the step runs even if the
previous step fails.
*
[`.github/workflows/nightly.yaml`](diffhunk://#diff-e79198339eb3fcc6974b6912e52526068eab3b7952ed67d03276479a235eb58eL94-R105):
Updated the condition for uploading test reports and added a step to
publish test reports using `mikepenz/action-junit-report@v4`.
*
[`.github/workflows/release.yaml`](diffhunk://#diff-e426ed45842837026e10e66af23d9c7077e89eacbe6958ce7cb991130ad05adaR83-R89):
Added a step to publish test reports using
`mikepenz/action-junit-report@v4`, ensuring the step runs even if the
previous step fails.
*
[`.github/workflows/release_core_and_parent.yaml`](diffhunk://#diff-29983aaefd1016684ce940117feefedb37edefca433dde927389805f65cdfee6R36-R42):
Added a step to publish test reports using
`mikepenz/action-junit-report@v4`, ensuring the step runs even if the
previous step fails.
*
[`.github/workflows/snapshot_release.yaml`](diffhunk://#diff-cf84df4f6da634dce2ec7dc5dfc37ddc1f47181a6dd4f2e1d81df7537356dcd8R39-R45):
Added a step to publish test reports using
`mikepenz/action-junit-report@v4`, ensuring the step runs even if the
previous step fails.
*
[`.github/workflows/snapshot_release_core_and_parent.yaml`](diffhunk://#diff-178e48d6db6047dafaad65fe4eb44b9bdbe8a90f03397c2c653de9fa07e830dfR41-R47):
Added a step to publish test reports using
`mikepenz/action-junit-report@v4`, ensuring the step runs even if the
previous step fails.
## General checklist
<!-- Please double-check the following points and mark them like this:
[X] -->
- [x] There are no breaking changes
- [ ] I have added unit and integration tests for my change
- [ ] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- Before adding documentation and example(s) (below), please wait
until the PR is reviewed and approved. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Issue
Fixes#1909, Fixes#1899, Closes#1652
This pull request includes several updates to the GitHub Actions
workflows and Maven configuration files to modernize the Java version
and improve the build process. The most important changes include
upgrading the Java version, modifying the Maven build commands, and
updating dependencies.
## Change
- ⬆️ Min JDK version has been upgraded to 17
- 🗿 Hardening: run `mvn process-test-classes javadocs:aggregate` on
**ALL** modules to verify that code compiles before running integraiton
tests
- 💄 Reformat maven commands to multiline to fit on a screen
- Rename GoogleAiGeminiTokenizerTest to GoogleAiGeminiTokenizerIT
- Upgrade graalvm and Oracle image
- Run tests on every build, run integraiton tests only on
langchain4j/langchain4j repo
----
This pull request includes several updates to the GitHub workflows and
various project files to modernize the build environment and improve
testing processes. The most important changes include updating the Java
Development Kit (JDK) versions, modifying Maven commands for better
readability and functionality, and renaming test classes for
consistency.
### Updates to GitHub Workflows:
* **JDK Version Updates:**
*
[`.github/workflows/javadoc.yaml`](diffhunk://#diff-40d3687e6b489280a4b84d8a8ee6ceaee3d7ca3a8e161df603d1046daded3db3L30-R39):
Updated from JDK 8 to JDK 17 and changed the distribution to 'temurin'.
*
[`.github/workflows/main.yaml`](diffhunk://#diff-71cabc4177e41ea8f15a89eb65398fa8187739f68a2762706d84885cd8b2ddc3L31-R36):
Removed JDK 8 and 11, now only using JDK 17 and 21.
[[1]](diffhunk://#diff-71cabc4177e41ea8f15a89eb65398fa8187739f68a2762706d84885cd8b2ddc3L31-R36)
[[2]](diffhunk://#diff-71cabc4177e41ea8f15a89eb65398fa8187739f68a2762706d84885cd8b2ddc3L59-R68)
*
[`.github/workflows/nightly.yaml`](diffhunk://#diff-e79198339eb3fcc6974b6912e52526068eab3b7952ed67d03276479a235eb58eL13-L18):
Removed JDK 8 and 11, now only using JDK 17 and 21.
[[1]](diffhunk://#diff-e79198339eb3fcc6974b6912e52526068eab3b7952ed67d03276479a235eb58eL13-L18)
[[2]](diffhunk://#diff-e79198339eb3fcc6974b6912e52526068eab3b7952ed67d03276479a235eb58eL43-R46)
* **Maven Command Enhancements:**
*
[`.github/workflows/receive-pr.yml`](diffhunk://#diff-3f8fc2f08505435fdc4f66740e23234e3fee099725c4d72b02a67a0fc5db4c41L45-R47):
Reformatted Maven command for better readability.
*
[`.github/workflows/release.yaml`](diffhunk://#diff-e426ed45842837026e10e66af23d9c7077e89eacbe6958ce7cb991130ad05adaL38-R44):
Reformatted Maven command for better readability and maintainability.
*
[`.github/workflows/release_core_and_parent.yaml`](diffhunk://#diff-29983aaefd1016684ce940117feefedb37edefca433dde927389805f65cdfee6L27-R31):
Reformatted Maven command for better readability.
*
[`.github/workflows/snapshot_release.yaml`](diffhunk://#diff-cf84df4f6da634dce2ec7dc5dfc37ddc1f47181a6dd4f2e1d81df7537356dcd8L34-R38):
Reformatted Maven command for better readability.
*
[`.github/workflows/snapshot_release_core_and_parent.yaml`](diffhunk://#diff-178e48d6db6047dafaad65fe4eb44b9bdbe8a90f03397c2c653de9fa07e830dfL34-R40):
Reformatted Maven command for better readability.
### Project File Updates:
* **Dependency and Property Updates:**
*
[`code-execution-engines/langchain4j-code-execution-engine-graalvm-polyglot/pom.xml`](diffhunk://#diff-336b440bd3a43cdb2df4f9c6ee2a0cb257703526e8700f607cc67ec7ec73f0eaL19-R20):
Updated GraalVM version from 23.1.1 to 24.1.0.
*
[`langchain4j-parent/pom.xml`](diffhunk://#diff-db1dfc1ccffac6c8f8ae2414e6aad535940aaf08a488d30941157dc53ec573b0L17-R18):
Updated Maven compiler plugin version to 3.13.0 and removed Java 11+
profile.
[[1]](diffhunk://#diff-db1dfc1ccffac6c8f8ae2414e6aad535940aaf08a488d30941157dc53ec573b0L17-R18)
[[2]](diffhunk://#diff-db1dfc1ccffac6c8f8ae2414e6aad535940aaf08a488d30941157dc53ec573b0L458-R458)
[[3]](diffhunk://#diff-db1dfc1ccffac6c8f8ae2414e6aad535940aaf08a488d30941157dc53ec573b0L616-L624)
* **Test Class Renaming:**
*
[`langchain4j-google-ai-gemini/src/test/java/dev/langchain4j/model/googleai/GoogleAiGeminiTokenizerIT.java`](diffhunk://#diff-76accbd5060e23e5e16220546f59fa5074b91b36c2d42ece8109fa800c9a2f3dL14-R14):
Renamed from `GoogleAiGeminiTokenizerTest.java` to follow integration
test naming conventions.
* **Miscellaneous Fixes:**
*
[`langchain4j-oracle/src/test/java/dev/langchain4j/store/embedding/oracle/CommonTestOperations.java`](diffhunk://#diff-4a9654970655bfc6505ad1b1d334df5751c823922bdca974630c3641e29cf951L24):
Added a constant for the Oracle image name and updated container startup
and connect timeouts.
[[1]](diffhunk://#diff-4a9654970655bfc6505ad1b1d334df5751c823922bdca974630c3641e29cf951L24)
[[2]](diffhunk://#diff-4a9654970655bfc6505ad1b1d334df5751c823922bdca974630c3641e29cf951R63-R64)
[[3]](diffhunk://#diff-4a9654970655bfc6505ad1b1d334df5751c823922bdca974630c3641e29cf951L72-R75)
*
[`langchain4j-qdrant/src/test/java/dev/langchain4j/store/embedding/qdrant/QdrantFilterConverterTest.java`](diffhunk://#diff-2006ea9aece1dcbcf4d519a514526e8314ecebf5575d9197fc5385c99e2c3119L68-R68):
Corrected assertions for better readability.
[[1]](diffhunk://#diff-2006ea9aece1dcbcf4d519a514526e8314ecebf5575d9197fc5385c99e2c3119L68-R68)
[[2]](diffhunk://#diff-2006ea9aece1dcbcf4d519a514526e8314ecebf5575d9197fc5385c99e2c3119L78-R78)
[[3]](diffhunk://#diff-2006ea9aece1dcbcf4d519a514526e8314ecebf5575d9197fc5385c99e2c3119L88-R88)
[[4]](diffhunk://#diff-2006ea9aece1dcbcf4d519a514526e8314ecebf5575d9197fc5385c99e2c3119L98-R98)
----
## General checklist
<!-- Please double-check the following points and mark them like this:
[X] -->
- [x] There are no breaking changes
- [ ] I have added unit and integration tests for my change
- [ ] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [x] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## New EmbeddingStore (metadata) `Filter` API
Many embedding stores, such as
[Pinecone](https://docs.pinecone.io/docs/metadata-filtering) and
[Milvus](https://milvus.io/docs/boolean.md) support strict filtering
(think of an SQL "WHERE" clause) during similarity search.
So, if one has an embedding store with movies, for example, one could
search not only for the most semantically similar movies to the given
user query but also apply strict filtering by metadata fields like year,
genre, rating, etc. In this case, the similarity search will be
performed only on those movies that match the filter expression.
Since LangChain4j supports (and abstracts away) many embedding stores,
there needs to be an embedding-store-agnostic way for users to define
the filter expression.
This PR introduces a `Filter` interface, which can represent both simple
(e.g., `type = "documentation"`) and composite (e.g., `type in
("documentation", "tutorial") AND year > 2020`) filter expressions in an
embedding-store-agnostic manner.
`Filter` currently supports the following operations:
- Comparison:
- `IsEqualTo`
- `IsNotEqualTo`
- `IsGreaterThan`
- `IsGreaterThanOrEqualTo`
- `IsLessThan`
- `IsLessThanOrEqualTo`
- `IsIn`
- `IsNotIn`
- Logical:
- `And`
- `Not`
- `Or`
These operations are supported by most embedding stores and serve as a
good starting point. However, the list of operations will expand over
time to include other operations (e.g., `Contains`) supported by
embedding stores.
Currently, the DSL looks like this:
```java
Filter onlyDocs = metadataKey("type").isEqualTo("documentation");
Filter docsAndTutorialsAfter2020 = metadataKey("type").isIn("documentation", "tutorial").and(metadataKey("year").isGreaterThan(2020));
// or
Filter docsAndTutorialsAfter2020 = and(
metadataKey("type").isIn("documentation", "tutorial"),
metadataKey("year").isGreaterThan(2020)
);
```
## Filter expression as a `String`
Filter expression can also be specified as a `String`. This might be
necessary, for example, if the filter expression is generated
dynamically by the application or by the LLM (as in [self
querying](https://python.langchain.com/docs/modules/data_connection/retrievers/self_query/)).
This PR introduces a `FilterParser` interface with a simple `Filter
parse(String)` API, allowing for future support of multiple syntaxes (if
this will be required).
For the out-of-the-box filter syntax, ANSI SQL's `WHERE` clause is
proposed as a suitable candidate for several reasons:
- SQL is well-known among Java developers
- There is extensive tooling available for SQL (e.g., parsers)
- LLMs are pretty good at generating valid SQL, as there are tons of SQL
queries on the internet, which are included in the LLM training
datasets. There are also specialized LLMs that are trained for
text-to-SQL task, such as [SQLCoder](https://huggingface.co/defog).
The downside is that SQL's `WHERE` clause might not support all
operations and data types that could be supported in the future by
various embedding stores. In such case, we could extend it to a superset
of ANSI SQL `WHERE` syntax and/or provide an option to express filters
in the native syntax of the store.
An out-of-the-box implementation of the SQL `FilterParser` is provided
as a `SqlFilterParser` in a separate module
`langchain4j-embedding-store-filter-parser-sql`, using
[JSqlParser](https://github.com/JSQLParser/JSqlParser) under the hood.
`SqlFilterParser` can parse SQL "SELECT" (or just "WHERE" clause)
statement into a `Filter` object:
- `SELECT * FROM fake_table WHERE userId = '123-456'` ->
`metadataKey("userId").isEqualTo("123-456")`
- `userId = '123-456'` -> `metadataKey("userId").isEqualTo("123-456")`
It can also resolve `CURDATE()` and
`CURRENT_DATE`/`CURRENT_TIME`/`CURRENT_TIMESTAMP`:
`SELECT * FROM fake_table WHERE year = EXTRACT(YEAR FROM CURRENT_DATE`
-> `metadataKey("year").isEqualTo(LocalDate.now().getYear())`
## Changes in `Metadata` API
Until now, `Metadata` supported only `String` values. This PR expands
the list of supported value types to `Integer`, `Long`, `Float` and
`Double`. In the future, more types may be added (if needed).
The method `String get(String key)` will be deprecated later in favor
of:
- `String getString(String key)`
- `Integer getInteger(String key)`
- `Long getLong(String key)`
- etc
New overloaded `put(key, value)` methods are introduced to support more
value types:
- `put(String key, int value)`
- `put(String key, long value)`
- etc
## Changes in `EmbeddingStore` API
New method `search` is added that will become the main entry point for
search in the future. All `findRelevant` methods will be deprecated
later.
New `search` method accepts `EmbeddingSearchRequest` and returns
`EmbeddingSearchResult`.
`EmbeddingSearchRequest` contains all search criteria (e.g.
`maxResults`, `minScore`), including new `Filter`.
`EmbeddingSearchResult` contains a list of `EmbeddingMatch`.
```java
EmbeddingSearchResult search(EmbeddingSearchRequest request);
```
## Changes in `EmbeddingStoreContentRetriever` API
`EmbeddingStoreContentRetriever` can now be configured with a static
`filter` as well as dynamic `dynamicMaxResults`, `dynamicMinScore` and
`dynamicFilter` in the builder:
```java
ContentRetriever contentRetriever = EmbeddingStoreContentRetriever.builder()
.embeddingStore(embeddingStore)
.embeddingModel(embeddingModel)
...
.maxResults(3)
// or
.dynamicMaxResults(query -> 3) // You can define maxResults dynamically. The value could, for example, depend on the query or the user associated with the query.
...
.minScore(0.3)
// or
.dynamicMinScore(query -> 0.3)
...
.filter(metadataKey("userId").isEqualTo("123-456")) // Assuming your TextSegments contain Metadata with key "userId"
// or
.dynamicFilter(query -> metadataKey("userId").isEqualTo(query.metadata().chatMemoryId().toString()))
...
.build();
```
So now you can define `maxResults`, `minScore` and `filter` both
statically and dynamically (they can depend on the query, user, etc.).
These values will be propagated to the underlying `EmbeddingStore`.
##
["Self-querying"](https://python.langchain.com/docs/modules/data_connection/retrievers/self_query/)
This PR also introduces `LanguageModelSqlFilterBuilder` in
`langchain4j-embedding-store-filter-parser-sql` module which can be used
with `EmbeddingStoreContentRetriever`'s `dynamicFilter` to automatically
build a `Filter` object from the `Query` using language model and
`SqlFilterParser`.
For example:
```java
TextSegment groundhogDay = TextSegment.from("Groundhog Day", new Metadata().put("genre", "comedy").put("year", 1993));
TextSegment forrestGump = TextSegment.from("Forrest Gump", new Metadata().put("genre", "drama").put("year", 1994));
TextSegment dieHard = TextSegment.from("Die Hard", new Metadata().put("genre", "action").put("year", 1998));
// describe metadata keys as if they were columns in the SQL table
TableDefinition tableDefinition = TableDefinition.builder()
.name("movies")
.addColumn("genre", "VARCHAR", "one of [comedy, drama, action]")
.addColumn("year", "INT")
.build();
LanguageModelSqlFilterBuilder sqlFilterBuilder = new LanguageModelSqlFilterBuilder(model, tableDefinition);
ContentRetriever contentRetriever = EmbeddingStoreContentRetriever.builder()
.embeddingStore(embeddingStore)
.embeddingModel(embeddingModel)
.dynamicFilter(sqlFilterBuilder::build)
.build();
String answer = assistant.answer("Recommend me a good drama from 90s"); // Forrest Gump
```
## Which embedding store integrations will support `Filter`?
In the long run, all (provided the embedding store itself supports it).
In the first iteration, I aim to add support to just a few:
- `InMemoryEmbeddingStore`
- Elasticsearch
- Milvus
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Introduced filters for checking key's value existence in a collection
for improved data handling.
- **Enhancements**
- Updated `InMemoryEmbeddingStoreTest` to extend a different class for
improved testing coverage and added a new test method.
- **Refactor**
- Made minor formatting adjustments in the assertion block for better
readability.
- **Documentation**
- Updated class hierarchy information for clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
renovate[bot]