langchain4j/.github/workflows/release.yaml

67 lines
2.7 KiB
YAML
Raw Permalink Normal View History

2024-02-09 15:00:34 +08:00
name: release
2024-01-30 14:20:20 +08:00
on:
workflow_dispatch:
jobs:
2024-02-09 15:00:34 +08:00
release:
2024-01-30 14:20:20 +08:00
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
2024-02-09 15:00:34 +08:00
- name: Set up JDK 21
2024-01-30 14:20:20 +08:00
uses: actions/setup-java@v4
with:
2024-02-09 15:00:34 +08:00
java-version: '21'
2024-01-30 14:20:20 +08:00
distribution: 'temurin'
cache: maven
server-id: ossrh
server-username: OSSRH_USERNAME
server-password: OSSRH_PASSWORD
gpg-private-key: ${{ secrets.GPG_PRIVATE_KEY }}
gpg-passphrase: GPG_PASSPHRASE
- name: Authenticate to Google Cloud
uses: 'google-github-actions/auth@v2'
with:
project_id: ${{ secrets.GCP_PROJECT_ID }}
credentials_json: ${{ secrets.GCP_CREDENTIALS_JSON }}
2024-02-09 15:00:34 +08:00
- name: Setup Testcontainers Cloud Client
uses: atomicjar/testcontainers-cloud-setup-action@v1
with:
token: ${{ secrets.TC_CLOUD_TOKEN }}
- name: release
2024-03-12 03:08:55 +08:00
run: mvn -B -U --fail-at-end -DskipTests -DskipAnthropicITs -DskipLocalAiITs -DskipMilvusITs -DskipMongoDbAtlasITs -DskipOllamaITs -DskipVearchITs -DskipVertexAiGeminiITs -Psign clean deploy
2024-01-30 14:20:20 +08:00
env:
2024-03-12 03:08:55 +08:00
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
2024-01-30 14:20:20 +08:00
AZURE_OPENAI_ENDPOINT: ${{ secrets.AZURE_OPENAI_ENDPOINT }}
AZURE_OPENAI_KEY: ${{ secrets.AZURE_OPENAI_KEY }}
2024-03-19 23:23:33 +08:00
AZURE_SEARCH_ENDPOINT: ${{ secrets.AZURE_SEARCH_ENDPOINT }}
AZURE_SEARCH_KEY: ${{ secrets.AZURE_SEARCH_KEY }}
2024-01-30 14:20:20 +08:00
COHERE_API_KEY: ${{ secrets.COHERE_API_KEY }}
EmbeddingStore (Metadata) Filter API (#610) ## New EmbeddingStore (metadata) `Filter` API Many embedding stores, such as [Pinecone](https://docs.pinecone.io/docs/metadata-filtering) and [Milvus](https://milvus.io/docs/boolean.md) support strict filtering (think of an SQL "WHERE" clause) during similarity search. So, if one has an embedding store with movies, for example, one could search not only for the most semantically similar movies to the given user query but also apply strict filtering by metadata fields like year, genre, rating, etc. In this case, the similarity search will be performed only on those movies that match the filter expression. Since LangChain4j supports (and abstracts away) many embedding stores, there needs to be an embedding-store-agnostic way for users to define the filter expression. This PR introduces a `Filter` interface, which can represent both simple (e.g., `type = "documentation"`) and composite (e.g., `type in ("documentation", "tutorial") AND year > 2020`) filter expressions in an embedding-store-agnostic manner. `Filter` currently supports the following operations: - Comparison: - `IsEqualTo` - `IsNotEqualTo` - `IsGreaterThan` - `IsGreaterThanOrEqualTo` - `IsLessThan` - `IsLessThanOrEqualTo` - `IsIn` - `IsNotIn` - Logical: - `And` - `Not` - `Or` These operations are supported by most embedding stores and serve as a good starting point. However, the list of operations will expand over time to include other operations (e.g., `Contains`) supported by embedding stores. Currently, the DSL looks like this: ```java Filter onlyDocs = metadataKey("type").isEqualTo("documentation"); Filter docsAndTutorialsAfter2020 = metadataKey("type").isIn("documentation", "tutorial").and(metadataKey("year").isGreaterThan(2020)); // or Filter docsAndTutorialsAfter2020 = and( metadataKey("type").isIn("documentation", "tutorial"), metadataKey("year").isGreaterThan(2020) ); ``` ## Filter expression as a `String` Filter expression can also be specified as a `String`. This might be necessary, for example, if the filter expression is generated dynamically by the application or by the LLM (as in [self querying](https://python.langchain.com/docs/modules/data_connection/retrievers/self_query/)). This PR introduces a `FilterParser` interface with a simple `Filter parse(String)` API, allowing for future support of multiple syntaxes (if this will be required). For the out-of-the-box filter syntax, ANSI SQL's `WHERE` clause is proposed as a suitable candidate for several reasons: - SQL is well-known among Java developers - There is extensive tooling available for SQL (e.g., parsers) - LLMs are pretty good at generating valid SQL, as there are tons of SQL queries on the internet, which are included in the LLM training datasets. There are also specialized LLMs that are trained for text-to-SQL task, such as [SQLCoder](https://huggingface.co/defog). The downside is that SQL's `WHERE` clause might not support all operations and data types that could be supported in the future by various embedding stores. In such case, we could extend it to a superset of ANSI SQL `WHERE` syntax and/or provide an option to express filters in the native syntax of the store. An out-of-the-box implementation of the SQL `FilterParser` is provided as a `SqlFilterParser` in a separate module `langchain4j-embedding-store-filter-parser-sql`, using [JSqlParser](https://github.com/JSQLParser/JSqlParser) under the hood. `SqlFilterParser` can parse SQL "SELECT" (or just "WHERE" clause) statement into a `Filter` object: - `SELECT * FROM fake_table WHERE userId = '123-456'` -> `metadataKey("userId").isEqualTo("123-456")` - `userId = '123-456'` -> `metadataKey("userId").isEqualTo("123-456")` It can also resolve `CURDATE()` and `CURRENT_DATE`/`CURRENT_TIME`/`CURRENT_TIMESTAMP`: `SELECT * FROM fake_table WHERE year = EXTRACT(YEAR FROM CURRENT_DATE` -> `metadataKey("year").isEqualTo(LocalDate.now().getYear())` ## Changes in `Metadata` API Until now, `Metadata` supported only `String` values. This PR expands the list of supported value types to `Integer`, `Long`, `Float` and `Double`. In the future, more types may be added (if needed). The method `String get(String key)` will be deprecated later in favor of: - `String getString(String key)` - `Integer getInteger(String key)` - `Long getLong(String key)` - etc New overloaded `put(key, value)` methods are introduced to support more value types: - `put(String key, int value)` - `put(String key, long value)` - etc ## Changes in `EmbeddingStore` API New method `search` is added that will become the main entry point for search in the future. All `findRelevant` methods will be deprecated later. New `search` method accepts `EmbeddingSearchRequest` and returns `EmbeddingSearchResult`. `EmbeddingSearchRequest` contains all search criteria (e.g. `maxResults`, `minScore`), including new `Filter`. `EmbeddingSearchResult` contains a list of `EmbeddingMatch`. ```java EmbeddingSearchResult search(EmbeddingSearchRequest request); ``` ## Changes in `EmbeddingStoreContentRetriever` API `EmbeddingStoreContentRetriever` can now be configured with a static `filter` as well as dynamic `dynamicMaxResults`, `dynamicMinScore` and `dynamicFilter` in the builder: ```java ContentRetriever contentRetriever = EmbeddingStoreContentRetriever.builder() .embeddingStore(embeddingStore) .embeddingModel(embeddingModel) ... .maxResults(3) // or .dynamicMaxResults(query -> 3) // You can define maxResults dynamically. The value could, for example, depend on the query or the user associated with the query. ... .minScore(0.3) // or .dynamicMinScore(query -> 0.3) ... .filter(metadataKey("userId").isEqualTo("123-456")) // Assuming your TextSegments contain Metadata with key "userId" // or .dynamicFilter(query -> metadataKey("userId").isEqualTo(query.metadata().chatMemoryId().toString())) ... .build(); ``` So now you can define `maxResults`, `minScore` and `filter` both statically and dynamically (they can depend on the query, user, etc.). These values will be propagated to the underlying `EmbeddingStore`. ## ["Self-querying"](https://python.langchain.com/docs/modules/data_connection/retrievers/self_query/) This PR also introduces `LanguageModelSqlFilterBuilder` in `langchain4j-embedding-store-filter-parser-sql` module which can be used with `EmbeddingStoreContentRetriever`'s `dynamicFilter` to automatically build a `Filter` object from the `Query` using language model and `SqlFilterParser`. For example: ```java TextSegment groundhogDay = TextSegment.from("Groundhog Day", new Metadata().put("genre", "comedy").put("year", 1993)); TextSegment forrestGump = TextSegment.from("Forrest Gump", new Metadata().put("genre", "drama").put("year", 1994)); TextSegment dieHard = TextSegment.from("Die Hard", new Metadata().put("genre", "action").put("year", 1998)); // describe metadata keys as if they were columns in the SQL table TableDefinition tableDefinition = TableDefinition.builder() .name("movies") .addColumn("genre", "VARCHAR", "one of [comedy, drama, action]") .addColumn("year", "INT") .build(); LanguageModelSqlFilterBuilder sqlFilterBuilder = new LanguageModelSqlFilterBuilder(model, tableDefinition); ContentRetriever contentRetriever = EmbeddingStoreContentRetriever.builder() .embeddingStore(embeddingStore) .embeddingModel(embeddingModel) .dynamicFilter(sqlFilterBuilder::build) .build(); String answer = assistant.answer("Recommend me a good drama from 90s"); // Forrest Gump ``` ## Which embedding store integrations will support `Filter`? In the long run, all (provided the embedding store itself supports it). In the first iteration, I aim to add support to just a few: - `InMemoryEmbeddingStore` - Elasticsearch - Milvus <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Summary by CodeRabbit - **New Features** - Introduced filters for checking key's value existence in a collection for improved data handling. - **Enhancements** - Updated `InMemoryEmbeddingStoreTest` to extend a different class for improved testing coverage and added a new test method. - **Refactor** - Made minor formatting adjustments in the assertion block for better readability. - **Documentation** - Updated class hierarchy information for clarity. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2024-03-09 00:06:58 +08:00
ELASTICSEARCH_CLOUD_API_KEY: ${{ secrets.ELASTICSEARCH_CLOUD_API_KEY }}
ELASTICSEARCH_CLOUD_URL: ${{ secrets.ELASTICSEARCH_CLOUD_URL }}
2024-01-30 14:20:20 +08:00
GCP_CREDENTIALS_JSON: ${{ secrets.GCP_CREDENTIALS_JSON }}
GCP_LOCATION: ${{ secrets.GCP_LOCATION }}
GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
GCP_VERTEXAI_ENDPOINT: ${{ secrets.GCP_VERTEXAI_ENDPOINT }}
HF_API_KEY: ${{ secrets.HF_API_KEY }}
MILVUS_API_KEY: ${{ secrets.MILVUS_API_KEY }}
MISTRAL_AI_API_KEY: ${{ secrets.MISTRAL_AI_API_KEY }}
MONGODB_ATLAS_USERNAME: ${{ secrets.MONGODB_ATLAS_USERNAME }}
MONGODB_ATLAS_PASSWORD: ${{ secrets.MONGODB_ATLAS_PASSWORD }}
MONGODB_ATLAS_HOST: ${{ secrets.MONGODB_ATLAS_HOST }}
NOMIC_API_KEY: ${{ secrets.NOMIC_API_KEY }}
2024-02-09 15:00:34 +08:00
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
OPENAI_BASE_URL: ${{ secrets.OPENAI_BASE_URL }}
2024-01-30 14:20:20 +08:00
PINECONE_API_KEY: ${{ secrets.PINECONE_API_KEY }}
WEAVIATE_API_KEY: ${{ secrets.WEAVIATE_API_KEY }}
WEAVIATE_HOST: ${{ secrets.WEAVIATE_HOST }}
GPG_PASSPHRASE: ${{ secrets.GPG_PASSPHRASE }}
OSSRH_USERNAME: ${{ secrets.OSSRH_USERNAME }}
OSSRH_PASSWORD: ${{ secrets.OSSRH_PASSWORD }}