`MilvusMetadataFilterMapper.mapContains` builds a `ContainsString` query
as `field LIKE "%" + value + "%"` but never escapes the user value's own
LIKE wildcards. In Milvus, `%` and `_` are wildcards, so
`containsString("50%")` matches "50" + anything and
`containsString("a_b")` matches "axb" — but the contract is a literal
substring (`ContainsString#test` uses `String#contains`).
Parent PR #5577 hardened the same mapper's `formatValue` against
string-literal breakout but left the wildcard side undone. The identical
defect was already fixed in the Hibernate (#5522) and MongoDB (#5553)
mappers; Milvus is the store the sweep missed.
The fix adds a `formatLikePattern` that escapes `\`, `%`, `_`, `"` (the
existing `formatValue` is left untouched — its backslash-doubling would
corrupt `\%`). +4 regression tests. Verified by execution (Java 21): the
wildcard cases fail on the old code and pass after; non-wildcard values
are byte-identical.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Issue
Closes#5576
## Change
`MilvusMetadataFilterMapper.formatValue()` escaped double quotes but not
backslashes. Milvus treats backslash as the escape character in string
literals (`\\` represents a literal `\`), so a value containing a
backslash (e.g. a Windows path) produced a malformed expression — either
misinterpreted (`\b` → backspace) or rejected as an invalid escape
sequence. This affected every operator, `IsIn`/`IsNotIn`, and
`removeAll(ids)`, since all route through `formatValue`/`formatValues`.
Fix: escape backslashes first, then double quotes (`replace("\\",
"\\\\").replace("\"", "\\\"")`), mirroring sibling stores
(azure-cosmos-nosql, infinispan, pgvector). One production line. Added
`MilvusMetadataFilterMapperTest` (pure unit test) asserting the exact
generated expression for backslash, backslash+quote, collection values,
and a no-special-character regression. Values without backslashes are
unchanged — the only behaviour change is for backslash values, which
previously produced an invalid expression.
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
<!-- Unit tests green (6/6). The *IT are Testcontainers Docker-gated and
were not run locally. -->
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- Not run locally; change is isolated to langchain4j-milvus. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
<!-- Will add after review if requested. -->
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
<!-- N/A — small bug fix. -->
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
<!-- N/A -->
<!-- Checklist for adding new maven module: N/A — no new module. -->
<!-- Checklist for adding new embedding store integration: N/A — not a
new integration. -->
## Checklist for changing existing embedding store integration
- [ ] I have manually verified that the `MilvusEmbeddingStore` works
correctly with the data persisted using the latest released version of
LangChain4j
<!-- N/A — change only affects filter-expression string escaping, not
the persisted data format. -->
## Summary
- Convert Milvus L2 distances to relevance scores with lower distance
mapped higher.
- Pass the configured Milvus metric type into result mapping.
- Add mapper unit coverage for L2 and non-L2 score conversion.
## Verification
- MapperTest passed via Maven with embedding model download skipped.
---------
Co-authored-by: Q1Xuan <242799924+Q1Xuan@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Issue
Closes#3448
## Change
Added the ability to specify additional parameters when building a
MilvusEmbeddingStore that can be passed down during collection and index
creation to customize the behavior of the index created. These
parameters can impact the performance of indexes for large vector
collections.
## General checklist
- [X] There are no breaking changes
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
## Checklist for changing existing embedding store integration
- [X] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
## Issue
Fixes https://github.com/langchain4j/langchain4j/issues/3550
Changes here are limited to constructor arguments, ran spotless apply
because the changes were failing CI without them
## Change
Change "FiledName" variables to FieldName
## General checklist
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ x] There are no breaking changes
- [x ] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
<!--
Thank you so much for your contribution!
Please fill in all the sections below.
Please open the PR as a draft initially. Once it is reviewed and
approved, we will ask you to add documentation and examples.
Please note that PRs with breaking changes or without tests will be
rejected.
Please note that PRs will be reviewed based on the priority of the
issues they address.
We ask for your patience. We are doing our best to review your PR as
quickly as possible.
Please refrain from pinging and asking when it will be reviewed. Thank
you for understanding!
-->
## Issue
<!-- Please specify the ID of the issue this PR is addressing. For
example: "Closes #1234" or "Fixes #1234" -->
Closes #
## Change
<!-- Please describe the changes you made. -->
I noticed that in a lot of files there are duplicated checks for `null`
and not empty or `null` and not blank. Since there exist functions in
`Utils` class which do exactly that, I replaced all custom checks with
calls to such functions.
## General checklist
NB: I purposefully didn't commit formatting changes (i.e., `make lint`
and `make format`) because they'd change many lines in many files, much
more than what I changed in this PR, and that'd make PR difficult to
review. If you think I should do it, I'd gladly commit them.
<!-- Please double-check the following points and mark them like this:
[X] -->
- [x] There are no breaking changes
- [ ] I have added unit and/or integration tests for my change
- [ ] The tests cover both positive and negative cases
- [ ] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- Before adding documentation and example(s) (below), please wait
until the PR is reviewed and approved. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for adding new maven module
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added my new module in the root `pom.xml` and
`langchain4j-bom/pom.xml`
## Checklist for adding new embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that
extends from `EmbeddingStoreWithRemovalIT`
## Checklist for changing existing embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
---------
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
* Fix SQL injection vulnerability with PGVector, MariaDB and Milvus
- Add a test case for each database
- Fix the SQL injection vulnerability
- This is the same issue for all 3 databases, only Milvus has a different escape character
- I ran the full test suite for all 3 databases to test there is no regression
- This should fix the security warning at https://github.com/langchain4j/langchain4j/security/advisories/GHSA-hvxq-w43p-3ccr
* updated tests
* Fix Milvus test
---------
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
Fixes#3141
Main reason for this PR is to fix the security issue detailed in #3141
This also:
- Updates the Docker image used for tests to the latest version (2.5.10)
- Fixes a Spotless formatting issue
I have run the full test suite, both with TestContainers and using the
Milvus Cloud, and this is all green.
## Change
This PR adds a new MetadataFilters that allows filtering metadata values
by checking whether they contain or not contain a given string.
Whilst this is less powerful than #2342, it is easier to implement in
embedding stores, this PR implements the `ContainsString` filter for the
following embedding stores:
- Milvus
- PgVector
- Qdrant
I tried to implement it for Chroma as well, but didn't find a way in the
Chroma docs to do so.
## General checklist
- [X] There are no breaking changes
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [x] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [x] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
## Issue
Closes#2332
## Change
Milvus-sdk dependency upgraded to 2.5.3 and test containers to 2.4.20.
Added a simple README.md
## General checklist
- [X] There are no breaking changes
- [ ] I have added unit and/or integration tests for my change
- [ ] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for changing existing embedding store integration
- [X] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
## Issue
Fixes#1747
## Change
Fixed this bug for Milvus and Chroma.
Did not manage to fix it for Pinecone because Pinecone can store
`Integer`/`Long` only as `Double`. We could theoretically store them as
`String`s, but then metadata filtering for `Integer`/`Long` values will
not work. For now I've left the comment in the `PineconeEmbeddingStore`
Javadoc and Pinecone documentation about the bug and possible
workaround.
## General checklist
- [X] There are no breaking changes
- [X] I have added unit and integration tests for my change
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [x] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [X] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Issue
Closes#1924
## Change
> This PR is mainly for `ClickHosue` because a column can have only one
type.
1. Change `key2` from `"b"` to `UUID.randomUUID()`.
2. Rename `OracleEmbeddingStoreWithFilteringIT` to
`OracleEmbeddingStoreIT` to keep the consistence with other modules.
3. Add tinylog test dependencies in some modules which has no slf4j
implementation.
For all subclass of `EmbeddingStoreWithFilteringIT`, I have verified
them and they are all green. (except `AzureAiSearch` and `MilvusCloud`,
which I have no api key)
## General checklist
- [x] There are no breaking changes
- [ ] I have added unit and integration tests for my change
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [x] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
## Checklist for changing existing embedding store integration
- [x] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
## Issue
Closes#1842
## Change
Add `FieldDefinition` to hold customized filed name.
Replace default field name with customized filed name.
## General checklist
- [X] There are no breaking changes
- [X] I have added unit and integration tests for my change
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for changing existing embedding store integration
- [X] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
## Issue
Closes#1629
## Change
Upgrade to Milvus SDK 2.3.9 and replace com.alibaba.fastjson with
com.google.gson
## General checklist
- [X] There are no breaking changes
- [N/A] I have added unit and integration tests for my change
- [+-] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [+-] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
* I had to skip the MilvusEmbeddingStoreCloudIT test due to a lack of
credentials. However, other Milvus tests passed.
* While the [core] tests were successful, the [main] tests failed
because I lack credentials for several cloud providers.
<!-- Before adding documentation and example(s) (below), please wait
until the PR is reviewed and approved. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for adding new model integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added my new module in the
[BOM](https://github.com/langchain4j/langchain4j/blob/main/langchain4j-bom/pom.xml)
## Checklist for adding new embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added my new module in the
[BOM](https://github.com/langchain4j/langchain4j/blob/main/langchain4j-bom/pom.xml)
## Checklist for changing existing embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
## Issue
Closes#1066
## Change
These are changes for each split package (each change was done in a
separate commit, so they can be reviewed in isolation):
- `dev.langchain4j.retriever` -> Moved `EmbeddingStoreRetriever` into
`langchain4j-core` module
- `dev.langchain4j.agent.tool` -> Moved `DefaultToolExecutor` and
`ToolExecutor` into `dev.langchain4j.service.tool` package
- `dev.langchain4j.classification` -> Moved `TextClassifier` into
`langchian4j` module
- `dev.langchain4j.chain` -> Moved `Chain` into `langchain4j` module
- `dev.langchain4j.model.embedding` -> [All in-process embedding models
should have unique package
name](https://github.com/langchain4j/langchain4j-embeddings/pull/33)
- `dev.langchain4j.model.output` -> Moved `OutputParser` and all it's
implementations into `dev.langchain4j.service.output` package of the
`langchain4j` module
More details can be found
[here](https://docs.google.com/spreadsheets/d/1U7f2MIfDgWA1tydPpzWpOGTHiBjBVZjsu0uZnXBT9qE/edit?usp=sharing).
## Breaking Changes
- All in-process ONNX model classes moved into their own unique
packages:
- `AllMiniLmL6V2EmbeddingModel` moved into
`dev.langchain4j.model.embedding.onnx.allminilml6v2`
- `AllMiniLmL6V2QuantizedEmbeddingModel` moved into
`dev.langchain4j.model.embedding.onnx.allminilml6v2q`
- `OnnxEmbeddingModel` moved into `dev.langchain4j.model.embedding.onnx`
package
- etc
- `ToolExecutor` and `DefaultToolExecutor` moved into
`dev.langchain4j.service.tool` package
- Moved `OutputParser` and all it's implementations into
`dev.langchain4j.service.output` package of the `langchain4j` module
- Moved `Chain` into `langchain4j` module
- Moved `TextClassifier` into `langchian4j` module
## General checklist
- [ ] There are no breaking changes
- [ ] I have added unit and integration tests for my change
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- Before adding documentation and example(s) (below), please wait
until the PR is reviewed and approved. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Issue
<!-- Please paste the link to the issue this PR is addressing. For
example: https://github.com/langchain4j/langchain4j/issues/1012 -->
#1168
## Change
<!-- Please describe the changes you made. -->
Add the `MilvusEmbeddingStore` method to implement `remove(String id)`,
`removeAll(Collection<String> ids)`, `removeAll(Filter filter)`,
`removeAll()`.
Add unit test `MilvusEmbeddingStoreRemoveIT`
## General checklist
<!-- Please double-check the following points and mark them like this:
[X] -->
- [x] There are no breaking changes
- [x] I have added unit and integration tests for my change
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- Before adding documentation and example(s) (below), please wait
until the PR is reviewed and approved. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
## Checklist for changing existing embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
## New EmbeddingStore (metadata) `Filter` API
Many embedding stores, such as
[Pinecone](https://docs.pinecone.io/docs/metadata-filtering) and
[Milvus](https://milvus.io/docs/boolean.md) support strict filtering
(think of an SQL "WHERE" clause) during similarity search.
So, if one has an embedding store with movies, for example, one could
search not only for the most semantically similar movies to the given
user query but also apply strict filtering by metadata fields like year,
genre, rating, etc. In this case, the similarity search will be
performed only on those movies that match the filter expression.
Since LangChain4j supports (and abstracts away) many embedding stores,
there needs to be an embedding-store-agnostic way for users to define
the filter expression.
This PR introduces a `Filter` interface, which can represent both simple
(e.g., `type = "documentation"`) and composite (e.g., `type in
("documentation", "tutorial") AND year > 2020`) filter expressions in an
embedding-store-agnostic manner.
`Filter` currently supports the following operations:
- Comparison:
- `IsEqualTo`
- `IsNotEqualTo`
- `IsGreaterThan`
- `IsGreaterThanOrEqualTo`
- `IsLessThan`
- `IsLessThanOrEqualTo`
- `IsIn`
- `IsNotIn`
- Logical:
- `And`
- `Not`
- `Or`
These operations are supported by most embedding stores and serve as a
good starting point. However, the list of operations will expand over
time to include other operations (e.g., `Contains`) supported by
embedding stores.
Currently, the DSL looks like this:
```java
Filter onlyDocs = metadataKey("type").isEqualTo("documentation");
Filter docsAndTutorialsAfter2020 = metadataKey("type").isIn("documentation", "tutorial").and(metadataKey("year").isGreaterThan(2020));
// or
Filter docsAndTutorialsAfter2020 = and(
metadataKey("type").isIn("documentation", "tutorial"),
metadataKey("year").isGreaterThan(2020)
);
```
## Filter expression as a `String`
Filter expression can also be specified as a `String`. This might be
necessary, for example, if the filter expression is generated
dynamically by the application or by the LLM (as in [self
querying](https://python.langchain.com/docs/modules/data_connection/retrievers/self_query/)).
This PR introduces a `FilterParser` interface with a simple `Filter
parse(String)` API, allowing for future support of multiple syntaxes (if
this will be required).
For the out-of-the-box filter syntax, ANSI SQL's `WHERE` clause is
proposed as a suitable candidate for several reasons:
- SQL is well-known among Java developers
- There is extensive tooling available for SQL (e.g., parsers)
- LLMs are pretty good at generating valid SQL, as there are tons of SQL
queries on the internet, which are included in the LLM training
datasets. There are also specialized LLMs that are trained for
text-to-SQL task, such as [SQLCoder](https://huggingface.co/defog).
The downside is that SQL's `WHERE` clause might not support all
operations and data types that could be supported in the future by
various embedding stores. In such case, we could extend it to a superset
of ANSI SQL `WHERE` syntax and/or provide an option to express filters
in the native syntax of the store.
An out-of-the-box implementation of the SQL `FilterParser` is provided
as a `SqlFilterParser` in a separate module
`langchain4j-embedding-store-filter-parser-sql`, using
[JSqlParser](https://github.com/JSQLParser/JSqlParser) under the hood.
`SqlFilterParser` can parse SQL "SELECT" (or just "WHERE" clause)
statement into a `Filter` object:
- `SELECT * FROM fake_table WHERE userId = '123-456'` ->
`metadataKey("userId").isEqualTo("123-456")`
- `userId = '123-456'` -> `metadataKey("userId").isEqualTo("123-456")`
It can also resolve `CURDATE()` and
`CURRENT_DATE`/`CURRENT_TIME`/`CURRENT_TIMESTAMP`:
`SELECT * FROM fake_table WHERE year = EXTRACT(YEAR FROM CURRENT_DATE`
-> `metadataKey("year").isEqualTo(LocalDate.now().getYear())`
## Changes in `Metadata` API
Until now, `Metadata` supported only `String` values. This PR expands
the list of supported value types to `Integer`, `Long`, `Float` and
`Double`. In the future, more types may be added (if needed).
The method `String get(String key)` will be deprecated later in favor
of:
- `String getString(String key)`
- `Integer getInteger(String key)`
- `Long getLong(String key)`
- etc
New overloaded `put(key, value)` methods are introduced to support more
value types:
- `put(String key, int value)`
- `put(String key, long value)`
- etc
## Changes in `EmbeddingStore` API
New method `search` is added that will become the main entry point for
search in the future. All `findRelevant` methods will be deprecated
later.
New `search` method accepts `EmbeddingSearchRequest` and returns
`EmbeddingSearchResult`.
`EmbeddingSearchRequest` contains all search criteria (e.g.
`maxResults`, `minScore`), including new `Filter`.
`EmbeddingSearchResult` contains a list of `EmbeddingMatch`.
```java
EmbeddingSearchResult search(EmbeddingSearchRequest request);
```
## Changes in `EmbeddingStoreContentRetriever` API
`EmbeddingStoreContentRetriever` can now be configured with a static
`filter` as well as dynamic `dynamicMaxResults`, `dynamicMinScore` and
`dynamicFilter` in the builder:
```java
ContentRetriever contentRetriever = EmbeddingStoreContentRetriever.builder()
.embeddingStore(embeddingStore)
.embeddingModel(embeddingModel)
...
.maxResults(3)
// or
.dynamicMaxResults(query -> 3) // You can define maxResults dynamically. The value could, for example, depend on the query or the user associated with the query.
...
.minScore(0.3)
// or
.dynamicMinScore(query -> 0.3)
...
.filter(metadataKey("userId").isEqualTo("123-456")) // Assuming your TextSegments contain Metadata with key "userId"
// or
.dynamicFilter(query -> metadataKey("userId").isEqualTo(query.metadata().chatMemoryId().toString()))
...
.build();
```
So now you can define `maxResults`, `minScore` and `filter` both
statically and dynamically (they can depend on the query, user, etc.).
These values will be propagated to the underlying `EmbeddingStore`.
##
["Self-querying"](https://python.langchain.com/docs/modules/data_connection/retrievers/self_query/)
This PR also introduces `LanguageModelSqlFilterBuilder` in
`langchain4j-embedding-store-filter-parser-sql` module which can be used
with `EmbeddingStoreContentRetriever`'s `dynamicFilter` to automatically
build a `Filter` object from the `Query` using language model and
`SqlFilterParser`.
For example:
```java
TextSegment groundhogDay = TextSegment.from("Groundhog Day", new Metadata().put("genre", "comedy").put("year", 1993));
TextSegment forrestGump = TextSegment.from("Forrest Gump", new Metadata().put("genre", "drama").put("year", 1994));
TextSegment dieHard = TextSegment.from("Die Hard", new Metadata().put("genre", "action").put("year", 1998));
// describe metadata keys as if they were columns in the SQL table
TableDefinition tableDefinition = TableDefinition.builder()
.name("movies")
.addColumn("genre", "VARCHAR", "one of [comedy, drama, action]")
.addColumn("year", "INT")
.build();
LanguageModelSqlFilterBuilder sqlFilterBuilder = new LanguageModelSqlFilterBuilder(model, tableDefinition);
ContentRetriever contentRetriever = EmbeddingStoreContentRetriever.builder()
.embeddingStore(embeddingStore)
.embeddingModel(embeddingModel)
.dynamicFilter(sqlFilterBuilder::build)
.build();
String answer = assistant.answer("Recommend me a good drama from 90s"); // Forrest Gump
```
## Which embedding store integrations will support `Filter`?
In the long run, all (provided the embedding store itself supports it).
In the first iteration, I aim to add support to just a few:
- `InMemoryEmbeddingStore`
- Elasticsearch
- Milvus
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
- **New Features**
- Introduced filters for checking key's value existence in a collection
for improved data handling.
- **Enhancements**
- Updated `InMemoryEmbeddingStoreTest` to extend a different class for
improved testing coverage and added a new test method.
- **Refactor**
- Made minor formatting adjustments in the assertion block for better
readability.
- **Documentation**
- Updated class hierarchy information for clarity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Currently, Milvus IT is disables because it requires a manual step
to run the compose file. Adding Testcontainers, the infrastructure
is in place to run the tests.
greymoth