## Issue
Closes#4814
## Change
Added a null/empty guard for `getChoices()` in
`AzureOpenAiChatModel.doChat()` before accessing `.get(0)`.
**Before:**
```java
ChatChoice chatChoice = chatCompletions.getChoices().get(0);
```
**After:**
```java
if (isNullOrEmpty(chatCompletions.getChoices())) {
throw new IllegalArgumentException("Chat completion failed: no choices returned");
}
ChatChoice chatChoice = chatCompletions.getChoices().get(0);
```
This is consistent with the streaming counterpart
`AzureOpenAiStreamingChatModel` which already has this guard (line
287-288).
**Note:** No unit tests are added because the `OpenAIClient` is an Azure
SDK class that requires a real or mock Azure endpoint. The existing
tests in this module are integration tests (IT). The fix is a simple
null/empty guard with clear correctness from code inspection.
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [ ] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Summary
Fixes#5259
`InternalAzureOpenAiHelper.getOpenAIServiceVersion` silently returned
`OpenAIServiceVersion.getLatest()` whenever the user-provided version
string did not match any known enum value. This caused user-supplied
versions (e.g. `2024-10-21`) to be ignored without any warning — the API
call would then be made against a completely different version (the
latest preview), as shown in the issue report.
## Root cause
```java
// before
static OpenAIServiceVersion getOpenAIServiceVersion(String serviceVersion) {
for (OpenAIServiceVersion version : OpenAIServiceVersion.values()) {
if (version.getVersion().equals(serviceVersion)) {
return version;
}
}
return OpenAIServiceVersion.getLatest(); // ← silent fallback
}
```
Anyone who passed a version string not present in the
`com.azure.ai.openai.OpenAIServiceVersion` enum got the latest preview
version with no indication that their value was discarded.
## Fix
- `null` or empty `serviceVersion` continues to return the latest
version (this matches the documented contract: *"if not, the latest
version is used"*)
- A non-matching, non-empty `serviceVersion` now throws
`IllegalArgumentException` with a clear message listing all supported
versions
```java
// after
static OpenAIServiceVersion getOpenAIServiceVersion(String serviceVersion) {
if (serviceVersion == null || serviceVersion.isEmpty()) {
return OpenAIServiceVersion.getLatest();
}
for (OpenAIServiceVersion version : OpenAIServiceVersion.values()) {
if (version.getVersion().equals(serviceVersion)) {
return version;
}
}
List<String> supportedVersions = Arrays.stream(OpenAIServiceVersion.values())
.map(OpenAIServiceVersion::getVersion)
.collect(toList());
throw new IllegalArgumentException("Unsupported Azure OpenAI service version: '" + serviceVersion
+ "'. Supported versions are: " + supportedVersions
+ ". Leave serviceVersion null or empty to use the latest version.");
}
```
## Test plan
- [x] `InternalAzureOpenAiHelperTest` passes (16 tests)
- [x] Existing tests
`setupOpenAIClientShouldReturnClientWithCorrectConfiguration` and
`setupOpenAIAsyncClientShouldReturnClientWithCorrectConfiguration`
updated to use a real supported version (`2024-02-01`) instead of the
previously meaningless `test-service-version` string
- [x] The old
`getOpenAIServiceVersionShouldReturnLatestVersionIfIncorrect` test
(which documented the silent-fallback bug as intended behavior) was
replaced with `getOpenAIServiceVersionShouldThrowIfIncorrect`
- [x] Added `getOpenAIServiceVersionShouldReturnLatestVersionIfNull` and
`getOpenAIServiceVersionShouldReturnLatestVersionIfEmpty` to lock in the
documented fallback behavior
## Breaking change considerations
Code that previously relied on the silent-fallback (passing junk strings
and getting the latest version anyway) will now throw. This is
intentional — the silent fallback was the bug. Callers wanting the
latest version should pass `null` or omit `serviceVersion` entirely.
---------
Co-authored-by: Benamira05 <145583236+Benamira05@users.noreply.github.com>
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
### PR Description
Image token estimation in AzureOpenAiTokenCountEstimator was hardcoded
to 85 tokens regardless of detail level. This significantly
underestimates costs for HIGH detail images (which actually cost 765+
tokens for a typical 1024x1024 image).
Implement detail-level-aware token estimation:
- LOW: 85 tokens (unchanged, per OpenAI documentation)
- MEDIUM: 400 tokens (conservative estimate)
- HIGH/ULTRA_HIGH/AUTO: 765 tokens (typical for 1024x1024 image)
The exact token count for HIGH detail depends on image dimensions (85 +
170 per 512x512 tile), but since we don't have access to dimensions in
the estimator, we use a conservative default.
References:
- OpenAI Vision documentation:
https://platform.openai.com/docs/guides/vision
- ImageContent.DetailLevel enum in langchain4j-core
---
## Issue
Closes # (discovered via code audit)
## Change
Implement detail-level-aware image token estimation in
`AzureOpenAiTokenCountEstimator`.
**Before:** All images estimated at 85 tokens regardless of detail
level.
**After:**
- LOW: 85 tokens (unchanged, per OpenAI documentation)
- MEDIUM: 400 tokens (conservative estimate)
- HIGH/ULTRA_HIGH/AUTO: 765 tokens (typical for 1024x1024 image)
The exact token count for HIGH detail depends on image dimensions (85 +
170 per 512x512 tile), but since we don't have access to dimensions in
the estimator, we use a conservative default.
The `ImageContent.detailLevel()` method and `DetailLevel` enum already
exist in langchain4j-core but were being ignored.
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [ ] I have added unit and/or integration tests for my change
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have added/updated the documentation
<!--
Thank you so much for your contribution!
Please fill in all the sections below.
Please open the PR as a draft initially. Once it is reviewed and
approved, we will ask you to add documentation and examples.
Please note that PRs with breaking changes or without tests will be
rejected.
Please note that PRs will be reviewed based on the priority of the
issues they address.
We ask for your patience. We are doing our best to review your PR as
quickly as possible.
Please refrain from pinging and asking when it will be reviewed. Thank
you for understanding!
-->
## Issue
<!-- Please specify the ID of the issue this PR is addressing. For
example: "Closes #1234" or "Fixes #1234" -->
Closes #
## Change
<!-- Please describe the changes you made. -->
## General checklist
<!-- Please double-check the following points and mark them like this:
[X] -->
- [x] There are no breaking changes (API, behaviour)
- [ ] I have added unit and/or integration tests for my change
- [ ] The tests cover both positive and negative cases
- [ ] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- Before adding documentation and example(s) (below), please wait
until the PR is reviewed and approved. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for adding new maven module
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added my new module in the root `pom.xml` and
`langchain4j-bom/pom.xml`
## Checklist for adding new embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that
extends from `EmbeddingStoreWithRemovalIT`
## Checklist for changing existing embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
Signed-off-by: Anjali K <anjali.kakkar@ltts.com>
Co-authored-by: Anjali K <anjali.kakkar@ltts.com>
## Issue
Closes#5352
## Change
Makes `azure-core-http-netty` optional when a custom
`HttpClientProvider` is supplied via the builder.
### Details
- Removed hard constructor reference `NettyAsyncHttpClientProvider::new`
from `InternalAzureOpenAiHelper.setupOpenAIClientBuilder` — previously
baked into the constant pool as a `MethodHandle`, causing
`NoClassDefFoundError` at method invocation time even when a custom
provider
was set
- Replaced with a lazy `ServiceLoader` lookup that only runs when no
custom provider is supplied
- Added package-private overload
`loadDefaultHttpClientProvider(ClassLoader)` for clean testability
without reflection or thread manipulation
- Added tests covering: ServiceLoader discovery when Netty is on
classpath, custom provider is used and default is skipped, and
`IllegalStateException` when no provider is found
`azure-core-http-netty` already registers `NettyAsyncHttpClientProvider`
via `META-INF/services/com.azure.core.http.HttpClientProvider`, so
default behaviour is unchanged.
## General checklist
- [x] There are no breaking changes (API, behaviour)
- [x] I have added unit and/or integration tests for my change
- [x] The tests cover both positive and negative cases
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have manually run all the unit and integration tests in the core
and main modules, and they are all green
- [ ] I have added/updated the documentation
- [ ] I have added an example in the examples repo (only for "big"
features)
- [ ] I have added/updated Spring Boot starter(s) (if applicable)
## Checklist for adding new maven module
- [ ] I have added my new module in the root `pom.xml` and
`langchain4j-bom/pom.xml`
## Checklist for adding new embedding store integration
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that
extends from `EmbeddingStoreWithRemovalIT`
## Checklist for changing existing embedding store integration
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
---------
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
## Issue
Closes https://github.com/langchain4j/langchain4j/issues/4652
## Change
- Add support for `@Tool`-annotated methods to return multimodal content
(images) to the LLM, not just text. Tools can now return `Image`,
`ImageContent`, `Content`, `List<Content>`, `Content[]`, etc.
- Refactor `ToolExecutionResultMessage` to store `List<Content>`
internally instead of a plain `String`, with backward-compatible
`text()` accessor and new `contents()` / `hasSingleText()`
methods.
- Refactor `ToolExecutionResult` to support
`resultContents(List<Content>)` as an alternative to
`resultText(String)` and `resultTextSupplier(Supplier)`.
- Implement multimodal tool result mapping for providers that support
it: Anthropic, Amazon Bedrock, and Google AI Gemini.
- Add `UnsupportedFeatureException` guards in providers that do not
support non-text tool results: Azure OpenAI, GitHub Models, Jlama,
Mistral AI, Ollama, OpenAI (Chat Completions
& Responses), Vertex AI (Anthropic & Gemini), Watsonx, and Workers AI.
- Update `ToolExecutedEvent` / `DefaultToolExecutedEvent` to carry
`resultContents` alongside the existing `resultText` accessor.
- Add comprehensive unit and integration tests
(`should_execute_tool_returning_Image`,
`should_execute_tool_returning_ImageContent`,
`should_execute_tool_returning_ContentList`,
`should_fail_when_tool_returns_image_and_provider_does_not_support_it`).
- Update tools documentation with new "Returning Images and Multimodal
Content" and "Multimodal Tool Results" sections.
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [X] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
This PR contains the following updates:
| Package | Change |
[Age](https://docs.renovatebot.com/merge-confidence/) |
[Confidence](https://docs.renovatebot.com/merge-confidence/) |
|---|---|---|---|
| [io.netty:netty-codec-http2](https://netty.io/)
([source](https://redirect.github.com/netty/netty)) | `4.1.130.Final` →
`4.1.132.Final` |
|
|
---
> [!WARNING]
> Some dependencies could not be looked up. Check the [Dependency
Dashboard](../issues/2069) for more information.
### GitHub Vulnerability Alerts
####
[CVE-2026-33871](https://redirect.github.com/netty/netty/security/advisories/GHSA-w9fj-cfpg-grvv)
### Summary
A remote user can trigger a Denial of Service (DoS) against a Netty
HTTP/2 server by sending a flood of `CONTINUATION` frames. The server's
lack of a limit on the number of `CONTINUATION` frames, combined with a
bypass of existing size-based mitigations using zero-byte frames, allows
an user to cause excessive CPU consumption with minimal bandwidth,
rendering the server unresponsive.
### Details
The vulnerability exists in Netty's `DefaultHttp2FrameReader`. When an
HTTP/2 `HEADERS` frame is received without the `END_HEADERS` flag, the
server expects one or more subsequent `CONTINUATION` frames. However,
the implementation does not enforce a limit on the *count* of these
`CONTINUATION` frames.
The key issue is located in
`codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2FrameReader.java`.
The `verifyContinuationFrame()` method checks for stream association but
fails to implement a frame count limit.
Any user can exploit this by sending a stream of `CONTINUATION` frames
with a zero-byte payload. While Netty has a `maxHeaderListSize`
protection to limit the total size of headers, this check is never
triggered by zero-byte frames. The logic effectively evaluates to
`maxHeaderListSize - 0 < currentSize`, which will not trigger the limit
until a non-zero byte is added. As a result, the server is forced to
process an unlimited number of frames, consuming a CPU thread and
monopolizing the connection.
`codec-http2/src/main/java/io/netty/handler/codec/http2/DefaultHttp2FrameReader.java`
**`verifyContinuationFrame()` (lines 381-393)** — No frame count check:
```java
private void verifyContinuationFrame() throws Http2Exception {
verifyAssociatedWithAStream();
if (headersContinuation == null) {
throw connectionError(PROTOCOL_ERROR, "...");
}
if (streamId != headersContinuation.getStreamId()) {
throw connectionError(PROTOCOL_ERROR, "...");
}
// NO frame count limit!
}
```
**`HeadersBlockBuilder.addFragment()` (lines 695-723)** — Byte limit
bypassed by 0-byte frames:
```java
// Line 710-711: This check NEVER fires when len=0
if (headersDecoder.configuration().maxHeaderListSizeGoAway() - len <
headerBlock.readableBytes()) {
headerSizeExceeded(); // 10240 - 0 < 1 => FALSE always
}
```
When `len=0`: `maxGoAway - 0 < readableBytes` → `10240 < 1` → FALSE. The
byte limit is never triggered.
### Impact
This is a CPU-based Denial of Service (DoS). Any service using Netty's
default HTTP/2 server implementation is impacted. An unauthenticated
user can exhaust server CPU resources and block legitimate users,
leading to service unavailability. The low bandwidth requirement for the
attack makes it highly practical.
---
### Configuration
📅 **Schedule**: Branch creation - "" (UTC), Automerge - At any time (no
schedule defined).
🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.
♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box
---
This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/langchain4j/langchain4j).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My45MS41IiwidXBkYXRlZEluVmVyIjoiNDMuMTAwLjAiLCJ0YXJnZXRCcmFuY2giOiJtYWluIiwibGFiZWxzIjpbImRlcGVuZGVuY2llcyJdfQ==-->
---------
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
<!--
Thank you so much for your contribution!
Please fill in all the sections below.
Please open the PR as a draft initially. Once it is reviewed and
approved, we will ask you to add documentation and examples.
Please note that PRs with breaking changes or without tests will be
rejected.
Please note that PRs will be reviewed based on the priority of the
issues they address.
We ask for your patience. We are doing our best to review your PR as
quickly as possible.
Please refrain from pinging and asking when it will be reviewed. Thank
you for understanding!
-->
## Issue
<!-- Please specify the ID of the issue this PR is addressing. For
example: "Closes #1234" or "Fixes #1234" -->
Closes#2985
## Change
This PR adds OpenAI audio transcription support to langchain4j,
implementing the `/audio/transcriptions` API endpoint. The
implementation follows the existing patterns in the codebase and
integrates cleanly with the HTTP client infrastructure.
### Key Changes
- Added `AudioTranscriptionModel` interface in langchain4j-core with
convenience methods
- Implemented `OpenAiAudioModel` with support for Whisper and GPT-4o
transcription models
- Added multipart/form-data support to HTTP client infrastructure
(`addFile()` method)
- Created `MultipartBodyPublisher` for JDK HTTP client to handle file
uploads
- Implemented audio format detection and binary/base64 data handling
- Added comprehensive test coverage including integration tests
- Updated Azure OpenAI implementation to use new interface
### Sequence Diagram
```mermaid
sequenceDiagram
participant User
participant OpenAiAudioModel
participant OpenAiClient
participant DefaultOpenAiClient
participant JdkHttpClient
participant MultipartBodyPublisher
participant OpenAI API
User->>OpenAiAudioModel: transcribe(AudioTranscriptionRequest)
OpenAiAudioModel->>OpenAiAudioModel: validate(audioRequest)
OpenAiAudioModel->>OpenAiAudioModel: requestBuilder()
OpenAiAudioModel->>OpenAiClient: audioTranscription(OpenAiAudioTranscriptionRequest)
OpenAiClient->>DefaultOpenAiClient: audioTranscription()
DefaultOpenAiClient->>DefaultOpenAiClient: getBinaryDataFromAudio()
DefaultOpenAiClient->>DefaultOpenAiClient: getAudioExtension()
DefaultOpenAiClient->>DefaultOpenAiClient: build HttpRequest with multipart data
DefaultOpenAiClient->>JdkHttpClient: execute(HttpRequest)
JdkHttpClient->>JdkHttpClient: toJdkRequest()
JdkHttpClient->>MultipartBodyPublisher: ofMultipartData()
MultipartBodyPublisher->>MultipartBodyPublisher: addFormField()
MultipartBodyPublisher->>MultipartBodyPublisher: addFile()
MultipartBodyPublisher->>MultipartBodyPublisher: build()
JdkHttpClient->>OpenAI API: POST /audio/transcriptions
OpenAI API-->>JdkHttpClient: OpenAiAudioTranscriptionResponse
JdkHttpClient-->>DefaultOpenAiClient: SuccessfulHttpResponse
DefaultOpenAiClient-->>OpenAiClient: ParsedAndRawResponse
OpenAiClient-->>OpenAiAudioModel: OpenAiAudioTranscriptionResponse
OpenAiAudioModel->>OpenAiAudioModel: AudioTranscriptionResponse.from()
OpenAiAudioModel-->>User: AudioTranscriptionResponse
```
## General checklist
<!-- Please double-check the following points and mark them like this:
[X] -->
- [x] There are no breaking changes (API, behaviour)
- [x] I have added unit and/or integration tests for my change
- [ ] The tests cover both positive and negative cases
- [ ] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- Before adding documentation and example(s) (below), please wait
until the PR is reviewed and approved. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for adding new maven module
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added my new module in the root `pom.xml` and
`langchain4j-bom/pom.xml`
## Checklist for adding new embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that
extends from `EmbeddingStoreWithRemovalIT`
## Checklist for changing existing embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
<!--
Thank you so much for your contribution!
Please fill in all the sections below.
Please open the PR as a draft initially. Once it is reviewed and
approved, we will ask you to add documentation and examples.
Please note that PRs with breaking changes or without tests will be
rejected.
Please note that PRs will be reviewed based on the priority of the
issues they address.
We ask for your patience. We are doing our best to review your PR as
quickly as possible.
Please refrain from pinging and asking when it will be reviewed. Thank
you for understanding!
-->
## Issue
<!-- Please specify the ID of the issue this PR is addressing. For
example: "Closes #1234" or "Fixes #1234" -->
Closes#4051
## Change
Add support for the ReasoningEffortValue parameter in the
AzureOpenAiChatModel and AzureOpenAiStreamingChatModel.java class. This
enhancement would:
Extend the AzureOpenAiChatModel.Builder to accept a reasoningEffort
parameter
Store the reasoningEffort value in the model configuration
Pass the reasoningEffort value to the ChatCompletionsOptions when making
API calls
Allow users to specify reasoning effort levels (LOW, MEDIUM, HIGH) for
GPT-5 models
This would enable full compatibility with GPT-5 (o1 series) models and
provide users with control over the reasoning compute effort.
## General checklist
<!-- Please double-check the following points and mark them like this:
[X] -->
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [ ] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- Before adding documentation and example(s) (below), please wait
until the PR is reviewed and approved. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for adding new maven module
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added my new module in the root `pom.xml` and
`langchain4j-bom/pom.xml`
## Checklist for adding new embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that
extends from `EmbeddingStoreWithRemovalIT`
## Checklist for changing existing embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
## Issue
Closes https://github.com/langchain4j/langchain4j/issues/1146
## Change
Implemented streaming cancellation for the following APIs:
- `StreamingChatModel` + `StreamingChatResponseHandler`
- `TokenStream`
Implemented streaming cancellation for the following modules:
- Anthropic
- Azure OpenAI
- Bedrock
- Google AI Gemini
- Mistral
- Ollama
- OpenAI
- OpenAI Official
- Vertex AI Gemini
## Examples
### `StreamingChatModel` + `StreamingChatResponseHandler` APIs
If you wish to cancel the streaming, you can do so from one of the
following `StreamingChatResponseHandler` methods:
- `onPartialResponse(PartialResponse, PartialResponseContext)`
- `onPartialThinking(PartialThinking, PartialThinkingContext)`
- `onPartialToolCall(PartialToolCall, PartialToolCallContext)`
The context object contains the `StreamingHandle`, which can be used to
cancel the streaming:
```java
model.chat(userMessage, new StreamingChatResponseHandler() {
@Override
public void onPartialResponse(PartialResponse partialResponse, PartialResponseContext context) {
process(partialResponse);
if (shouldCancel()) {
context.streamingHandle().cancel();
}
}
@Override
public void onCompleteResponse(ChatResponse completeResponse) {
System.out.println("onCompleteResponse: " + completeResponse);
}
@Override
public void onError(Throwable error) {
error.printStackTrace();
}
});
```
### `TokenStream` API
If you wish to cancel the streaming, you can do so from one of the
following callbacks:
- `onPartialResponseWithContext(BiConsumer<PartialResponse,
PartialResponseContext>)`
- `onPartialThinkingWithContext(BiConsumer<PartialThinking,
PartialThinkingContext>)`
For example:
```java
tokenStream
.onPartialResponseWithContext((PartialResponse partialResponse, PartialResponseContext context) -> {
process(partialResponse);
if (shouldCancel()) {
context.streamingHandle().cancel();
}
})
.onCompleteResponse((ChatResponse response) -> futureResponse.complete(response))
.onError((Throwable error) -> futureResponse.completeExceptionally(error))
.start();
```
When `StreamingHandle.cancel()` is called, LangChain4j will close the
connection and stop the streaming.
Once `StreamingHandle.cancel()` has been called, `TokenStream` will not
receive any further callbacks.
## General checklist
- [x] There are no breaking changes (API, behaviour)
- [x] I have added unit and/or integration tests for my change
- [x] The tests cover both positive and negative cases
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [x] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [x] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring)
## Issue
Closes#3908
## Change
As of this implementation, the Azure OpenAI provider in LangChain4j now
supports sending images as base64-encoded data URIs. This allows you to
send images to GPT-4 Vision models without requiring a publicly
accessible URL.
## Background
Azure OpenAI Chat Completions API accepts images in the `image_url`
content block where the URL can be:
- An HTTP(S) URL pointing to an image
- A data URI in the format: `data:image/<mime>;base64,<payload>`
Previously, LangChain4j's Azure OpenAI provider only supported HTTP(S)
URLs and would throw an error when attempting to use base64 images. This
has been fixed.
## Main class that I used for testing
```java
package dev.langchain4j.model.azure;
import dev.langchain4j.data.message.ImageContent;
import dev.langchain4j.data.message.TextContent;
import dev.langchain4j.data.message.UserMessage;
import dev.langchain4j.model.chat.request.ChatRequest;
import dev.langchain4j.model.chat.response.ChatResponse;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Base64;
/**
* Simple example demonstrating base64 image support with Azure OpenAI.
*
* To run this example:
* 1. Set your Azure OpenAI endpoint and API key below
* 2. Set the deployment name (e.g., "gpt-4o", "gpt-4-vision-preview")
* 3. Set the path to an image file you want to analyze
* 4. Run the main method
*/
public class Base64ImageExample {
public static void main(String[] args) {
// ============================================
// CONFIGURATION - UPDATE THESE VALUES
// ============================================
// Your Azure OpenAI endpoint (e.g., "https://your-resource.openai.azure.com/")
String azureEndpoint = "...";
// Your Azure OpenAI API key
String azureApiKey = "...";
// Your deployment name (must be a vision-capable model like gpt-4o, gpt-4-vision-preview, etc.)
String deploymentName = "gpt-4.1";
// Path to the image you want to analyze
String imagePath = "humour.webp";
// Question to ask about the image
String question = "What do you see in this image? Please describe it in detail.";
// ============================================
// END CONFIGURATION
// ============================================
try {
System.out.println("=== Azure OpenAI Base64 Image Example ===\n");
// Validate configuration
if (azureEndpoint.contains("YOUR_") || azureApiKey.contains("YOUR_")) {
System.err.println("ERROR: Please update the Azure OpenAI endpoint and API key in the code.");
System.err.println("Look for the CONFIGURATION section at the top of this file.");
System.exit(1);
}
// Read and encode the image
System.out.println("1. Reading image from: " + imagePath);
Path imageFile = Paths.get(imagePath);
if (!Files.exists(imageFile)) {
System.err.println("ERROR: Image file not found: " + imagePath);
System.err.println("Please update the 'imagePath' variable with a valid image path.");
System.exit(1);
}
byte[] imageBytes = Files.readAllBytes(imageFile);
String base64Image = Base64.getEncoder().encodeToString(imageBytes);
System.out.println(" Image size: " + imageBytes.length + " bytes");
System.out.println(" Base64 length: " + base64Image.length() + " characters");
// Determine MIME type from file extension
String fileName = imageFile.getFileName().toString().toLowerCase();
String mimeType;
if (fileName.endsWith(".png")) {
mimeType = "image/png";
} else if (fileName.endsWith(".jpg") || fileName.endsWith(".jpeg")) {
mimeType = "image/jpeg";
} else if (fileName.endsWith(".gif")) {
mimeType = "image/gif";
} else if (fileName.endsWith(".webp")) {
mimeType = "image/webp";
} else {
mimeType = "image/png"; // default
System.out.println(" Warning: Unknown file extension, assuming PNG");
}
System.out.println(" MIME type: " + mimeType);
// Create Azure OpenAI chat model
System.out.println("\n2. Creating Azure OpenAI chat model...");
System.out.println(" Endpoint: " + azureEndpoint);
System.out.println(" Deployment: " + deploymentName);
AzureOpenAiChatModel model = AzureOpenAiChatModel.builder()
.endpoint(azureEndpoint)
.apiKey(azureApiKey)
.deploymentName(deploymentName)
.logRequestsAndResponses(false) // Set to true to see the full request/response
.build();
// Create message with base64 image
System.out.println("\n3. Creating message with base64 image...");
System.out.println(" Question: " + question);
UserMessage message = UserMessage.from(
TextContent.from(question),
ImageContent.from(base64Image, mimeType)
);
// Send request and get response
System.out.println("\n4. Sending request to Azure OpenAI...");
System.out.println(" (This may take a few seconds)\n");
ChatRequest chatRequest = ChatRequest.builder()
.messages(message)
.build();
ChatResponse response = model.chat(chatRequest);
// Display the response
System.out.println("=== RESPONSE ===");
System.out.println(response.aiMessage().text());
System.out.println("\n=== METADATA ===");
if (response.metadata() != null && response.metadata().tokenUsage() != null) {
System.out.println("Input tokens: " + response.metadata().tokenUsage().inputTokenCount());
System.out.println("Output tokens: " + response.metadata().tokenUsage().outputTokenCount());
System.out.println("Total tokens: " + response.metadata().tokenUsage().totalTokenCount());
}
if (response.metadata() != null && response.metadata().finishReason() != null) {
System.out.println("Finish reason: " + response.metadata().finishReason());
}
System.out.println("\n=== SUCCESS ===");
System.out.println("✓ Base64 image was successfully processed by Azure OpenAI!");
} catch (Exception e) {
System.err.println("\n=== ERROR ===");
System.err.println("Failed to process image: " + e.getMessage());
e.printStackTrace();
System.exit(1);
}
}
}
```
that produced the following output using file in copy.
```
=== Azure OpenAI Base64 Image Example ===
1. Reading image from: humour.webp
Image size: 61998 bytes
Base64 length: 82664 characters
MIME type: image/webp
2. Creating Azure OpenAI chat model...
Endpoint: XXX
Deployment: gpt-4.1
SLF4J(W): Class path contains multiple SLF4J providers.
SLF4J(W): Found provider [org.apache.logging.slf4j.SLF4JServiceProvider@97e1986]
SLF4J(W): Found provider [org.tinylog.slf4j.TinylogSlf4jServiceProvider@26f67b76]
SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J(I): Actual provider is of type [org.apache.logging.slf4j.SLF4JServiceProvider@97e1986]
3. Creating message with base64 image...
Question: What do you see in this image? Please describe it in detail.
4. Sending request to Azure OpenAI...
(This may take a few seconds)
=== RESPONSE ===
This image consists of two main parts: text at the top and a photo of a car at the bottom.
**Text (in French):**
"Je ne sais plus si je dois inculper le garagiste, l'opticien ou le bistro !!!"
Translation: "I no longer know whether to blame the mechanic, the optician, or the bar!!!"
There are two emojis next to the text:
- A laughing emoji with tears of joy
- A thinking face emoji (hand on chin)
**Image (bottom):**
There is a bright red vintage car parked on a street. It looks very unusual and distorted: the windows are misaligned, the wheels seem oddly placed, and the general shape of the car is very awkward, almost comically so. It looks like the car has been badly photoshopped or was constructed incorrectly, making it look absurd and nonfunctional.
**Context and mood:**
The text and the image together create a humorous effect. The joke is that the car looks so weird and dysfunctional that it is hard to know whether the fault lies with the mechanic (garagiste), the optician (who maybe didn't see the errors), or the bar (maybe someone was drunk when designing/repairing it). The emojis reinforce the funny and bewildered reaction.
**Summary:**
The image is a funny meme showing a badly assembled or photoshopped car, with text joking about which professional is to blame for the absurd result.
=== METADATA ===
Input tokens: 648
Output tokens: 297
Total tokens: 945
Finish reason: STOP
=== SUCCESS ===
✓ Base64 image was successfully processed by Azure OpenAI!
```
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [x] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- Before adding documentation and example(s) (below), please wait
until the PR is reviewed and approved. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
Farzad Sedaghatbin