## Issue
Closes#4796
## Change
Current vLLM returns the reasoning text of its OpenAI-compatible chat
responses in a `reasoning` field rather than `reasoning_content`, and
it's not just vLLM. OpenAI's own guidance for serving gpt-oss recommends
`reasoning` on chat completions responses, which is what prompted the
vLLM rename (vllm-project/vllm#27755). OpenRouter and Groq already use
`reasoning`, and SGLang is planning the same migration
(sgl-project/sglang#18219). `reasoning_content` came from DeepSeek and
is still what their API, llama.cpp and others return, so clients
realistically need to read both names for the foreseeable future.
Right now the module only parses `reasoning_content`, so
`returnThinking(true)` quietly returns null thinking against any of the
`reasoning` backends. Setting `thinkingFieldName` doesn't help because
it only affects how thinking is serialized back into request messages,
not response parsing.
This adds `@JsonAlias("reasoning")` on the `reasoningContent` builder
setters of `AssistantMessage` and `Delta`, so both field names
deserialize into `reasoningContent`. Serialization is untouched,
requests on the wire don't change, and `reasoning_content` keeps working
as before. Same approach that was suggested in the review of #5018
before that PR was closed.
Besides the unit tests I verified both models manually against a live
vLLM v0.25.1 server running a Qwen3.5 reasoning model: before the change
`thinking()` is null for `OpenAiChatModel` and `onPartialThinking` never
fires for the streaming model, after the change both come through.
## General checklist
- [x] There are no breaking changes (API, behaviour)
- [x] I have added unit and/or integration tests for my change
- [x] The tests cover both positive and negative cases
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [x] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
---------
Co-authored-by: Andrea Moccia <brokenlander@users.noreply.github.com>
## Issue
Closes#1153 — distinguish APIs for embedding queries vs. documents/keys
(adds `EmbeddingInputType.QUERY`/`DOCUMENT` as a per-call parameter,
plus opt-in `embeddingInputType(...)` on
`EmbeddingStoreContentRetriever` / `EmbeddingStoreIngestor`).
Partially addresses #4019 — adds the multimodal image-embedding API at
the core level (`EmbeddingInput` of `Content` parts) and wires Cohere,
Voyage, Jina, Google (Gemini Embedding 2), and Bedrock Titan; does
not implement it for `OnnxEmbeddingModel`.
Relates to #5142 — provider-specific / per-call parameters for OpenAI
embeddings (`OpenAiEmbeddingRequestParameters`: `user`,
`encodingFormat`, `customParameters`; e.g. NVIDIA NIM `input_type` via
custom
parameters).
Relates to #4273 — observability for `EmbeddingModel` via listeners
(`EmbeddingModelListener` + request/response/error contexts, wired
across providers).
## Change
Introduces an `EmbeddingModel.embed(EmbeddingRequest) →
EmbeddingResponse` API, structured like `ChatModel`'s request/response
API, so embeddings can carry **per-call parameters** and **multimodal
inputs** and
participate in **observability**. Everything is additive and
`@Experimental`; the existing `embed(String)` / `embed(TextSegment)` /
`embedAll(List)` methods keep working unchanged.
### Core (`langchain4j-core`)
- New request/response types: `EmbeddingRequest`, `EmbeddingResponse`,
`EmbeddingResponseMetadata`, `EmbeddingRequestParameters` (+
`DefaultEmbeddingRequestParameters` and typed `EmbeddingParameter<T>`
tokens), `EmbeddingInput`, `EmbeddingInputType`.
- New default methods on `EmbeddingModel`: `embed(EmbeddingRequest)`,
`doEmbed(...)`, `defaultRequestParameters()`, `supportedParameters()`,
`supportedContentTypes()`, `provider()`, `listeners()`.
- **Strict opt-in / fail-fast:** per-call parameters and content types
are token/type-checked; a request that uses something the model doesn't
declare is rejected with `UnsupportedFeatureException` instead of
being silently ignored. `overrideWith` preserves the provider-specific
parameters subtype (as on the chat side).
- **Multimodal:** an `EmbeddingInput` is an ordered list of `Content`
parts (text/image); models fuse them into one embedding (or
one-per-item, per provider). Modality is auto-detected — no manual flag.
- **Observability:** `EmbeddingModelListener` + request/response/error
contexts (same shape as `ChatModelListener`), fired inline from
`embed(EmbeddingRequest)`. `addListener(...)` still works.
- **RAG opt-in:** `EmbeddingStoreContentRetriever` and
`EmbeddingStoreIngestor` gain an optional `embeddingInputType(...)`
(QUERY / DOCUMENT). Default behavior is unchanged (no input type sent).
- `ModelProvider`: added `COHERE`, `VOYAGE_AI`, `JINA`, with matching
OpenTelemetry `gen_ai.provider.name` mappings (`cohere` is a well-known
OTel value; `voyage_ai` / `jina` are custom, as permitted by the
spec).
### Providers
- **OpenAI** (dimensions, `user`/`encodingFormat`/custom params),
**Cohere** (Embed v4 multimodal + input types), **Voyage** (multimodal +
input types), **Jina** (CLIP multimodal), **Google AI Gemini** (input
types; **Gemini Embedding 2** multimodal), **Amazon Bedrock Titan**
(multimodal).
- **Google Gen AI** (`langchain4j-google-genai`): input type → SDK
`task_type`, per-call dimensions → `outputDimensionality`, `provider()`,
listeners.
- **Ollama**: text-only — `provider()` + listeners (per-call params
correctly fail fast).
- **In-process models** (ONNX / `AbstractInProcessEmbeddingModel`):
already work via the default `doEmbed→embedAll` bridge (text-only,
image/param requests fail fast); observability via `addListener(...)`.
No
code change (no builders to wire listeners into, no dedicated
`ModelProvider`).
- **Gemini Embedding 2** dropped the `task_type` parameter, so input
types are applied as prompt instructions (`task: search result | query:
…` / `title: none | text: …`) automatically; `gemini-embedding-001`
still uses `task_type`.
- `modelName` in the response metadata reflects the API-reported model
where the provider returns one (OpenAI/Voyage/Jina), falling back to the
configured name.
### Tests
- `AbstractEmbeddingModelIT` — a shared IT base (like
`AbstractChatModelIT`) covering the new API, convenience methods,
listeners, and fail-fast; each provider adds a small
`common/…EmbeddingModelIT` that
parameterizes it and declares its capabilities via `supports*()`
overrides.
- Mock-based unit tests per provider for wire format / routing /
fail-fast (run in CI without keys), plus core value-type and listener
tests.
### Docs
- Embedding-model section in the RAG tutorial (request/response,
multimodal, query-vs-document opt-in), the EmbeddingModel listener
section in the Observability tutorial, the embedding contribution
guidance in
`CONTRIBUTING.md`, and the six provider integration pages.
### Notes
- `EmbeddingResponseMetadata` intentionally has no `finishReason`
(embeddings have no finish reason). No real provider is affected: the
only provider that emits `STOP` (Cloudflare WorkersAI) overrides the
convenience methods directly, and every other provider always returned
`null` here.
- `revapi.json` suppressions were added where the new (non-breaking)
types are exposed in provider APIs.
## General checklist
- [x] There are no breaking changes (API, behaviour)
- [x] I have added unit and/or integration tests for my change
- [x] The tests cover both positive and negative cases
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [x] I have added/updated the documentation
- [ ] I have manually run all the unit and integration tests in the core
and main modules, and they are all green
- [ ] I have added an example in the examples repo (only for "big"
features)
- [ ] I have added/updated Spring Boot starter(s) (if applicable)
---------
Co-authored-by: agent <agent@langchain4j.dev>
## Issue
Closes#5711
## Change
Some OpenAI-compatible gateways (LiteLLM, vLLM) emit a header chunk for
a streamed tool call that carries an `id` but no `function` object at
all, e.g. `{"index":0,"id":"call_x","type":"function"}`, with the
`function` details arriving in a later chunk.
Both `OpenAiStreamingChatModel.handle(...)` and
`OpenAiStreamingResponseBuilder.append(...)` dereferenced
`toolCall.function()` without a null check, causing an NPE that aborted
the entire stream. Note that `OpenAiStreamingResponseBuilder`'s
`isSentinel(...)` helper already null-checks `function()` correctly, but
the accumulation code right after it did not apply the same guard.
This PR adds the missing null guards in both places, so an id-only chunk
is accumulated (id recorded, name/arguments skipped) instead of crashing
the stream.
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Test plan
- Added `should_not_throw_npe_when_tool_call_has_no_function` in
`OpenAiStreamingResponseBuilderTest` (unit-level, reproduces the exact
NPE at `OpenAiStreamingResponseBuilder.java:206` without the fix).
- Added
`should_not_throw_npe_when_tool_call_delta_has_no_function_object` in
`OpenAiStreamingChatModelRawEventTest` (end-to-end via `MockHttpClient`,
reproduces the NPE at `OpenAiStreamingChatModel.java:251` without the
fix).
- Verified both new tests fail with the original NPE stack trace when
the fix is reverted, and pass with it applied.
- Ran the full `langchain4j-open-ai` (171 tests), `langchain4j-core`
(1131 tests), and `langchain4j` (1423 tests) suites — all green.
- Ran `./mvnw spotless:apply`/`spotless:check` — clean.
## Summary
Adds **Text-to-Speech (TTS)** support to LangChain4j: a new
provider-agnostic `TextToSpeechModel` abstraction in `langchain4j-core`,
an OpenAI implementation backed by the [OpenAI Speech
API](https://platform.openai.com/docs/api-reference/audio/createSpeech),
and the HTTP-client plumbing needed to return binary (audio) response
bodies intact.
## Changes
### `langchain4j-core` — new abstraction (`dev.langchain4j.model.audio`,
`@Experimental`)
- **`TextToSpeechModel`** — interface mirroring the existing
`AudioTranscriptionModel`:
- `synthesize(String text)` — convenience method using the model's
default voice
- `synthesize(TextToSpeechRequest request)`
- `provider()` defaulting to `OTHER`
- **`TextToSpeechRequest`** — carries `text` (validated non-blank) and
optional `voice`
- **`TextToSpeechResponse`** — wraps the core `Audio` type;
`from(Audio)` factory
### `langchain4j-open-ai` — OpenAI implementation
- **`OpenAiTextToSpeechModel`** — configurable `modelName`, `voice`
(default `alloy`), `timeout`, `maxRetries` (default `2`),
request/response logging, custom `baseUrl`, and `httpClientBuilder`.
Enforces
OpenAI's 4096-character input limit. `provider()` → `OPEN_AI`.
- **`OpenAiTextToSpeechModelName`** — `tts-1`, `tts-1-hd`,
`gpt-4o-mini-tts`, `gpt-4o-mini-tts-2025-12-15`
- **`OpenAiTextToSpeechRequest` / `OpenAiTextToSpeechResponse`** —
internal DTOs (`internal.audio.texttospeech`); the response is built
from the raw HTTP body (binary audio, not JSON)
- **`OpenAiTextToSpeechModelBuilderFactory`** — SPI factory for builder
customization
- **`OpenAiClient.textToSpeech(...)`** — `POST /audio/speech`, mapping
the raw bytes + `Content-Type` into the response
### HTTP client layer (shared) — binary response support
- **`SuccessfulHttpResponse`** — body is now stored as `byte[]`.
`body()` is preserved and returns a decoded `String` (using the charset
from the `Content-Type` header, UTF-8 fallback); adds `bodyBytes()` for
raw binary access and `contentType()`. The `body(String)` builder is
kept and a `body(byte[])` overload added — **additive, no public method
removed or changed**.
- **`JdkHttpClient`** — switched to `BodyHandlers.ofByteArray()`;
**`ApacheHttpClient` / `OkHttpClient`** read the body as raw bytes, so
binary payloads (audio) are no longer corrupted by text decoding.
- **`HttpResponseLogger`** — no longer decodes/dumps binary bodies as
text; logs `[binary body, N bytes, content-type: ...]` instead.
### Tests
- Core: `TextToSpeechModelTest`, `TextToSpeechRequestTest`
- HTTP: `SuccessfulHttpResponseTest` (charset decoding/edge cases),
`HttpClientIT` binary-response test asserting a valid MP3 is returned
across all client implementations
- OpenAI: `OpenAiTextToSpeechModelTest` (input-length and required-field
validation), `OpenAiTextToSpeechModelIT` (parameterized over every
supported model)
### Docs & API compatibility
- Added a **"Creating `OpenAiTextToSpeechModel`"** section to the OpenAI
docs page
- `revapi.json` updated to acknowledge the core
`TextToSpeechRequest`/`TextToSpeechResponse` types intentionally exposed
by `OpenAiTextToSpeechModel`
## Design notes
- Named `TextToSpeechModel` / `synthesize(...)` for discoverability (TTS
is the universally recognized term) and to read as a clear inverse of
the existing `AudioTranscriptionModel` / `transcribe(...)`.
- The shared HTTP-client change is **behavior-preserving**: for every
client (JDK / Apache / OkHttp), responses without an explicit charset
were already decoded as UTF-8, and explicit charsets are still
honored — verified empirically against `httpclient5 5.6.1` / `httpcore5
5.4`. The `String → byte[]` move is additive at the public API level.
## General checklist
- [x] There are no breaking changes (API, behaviour) — the shared
`SuccessfulHttpResponse` change is additive and verified
behavior-preserving; please review the HTTP-client layer regardless, as
it affects all
providers
- [x] I have added unit and/or integration tests for my change
- [x] The tests cover both positive and negative cases
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [x] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [x] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
---------
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
## Summary
- Add an opt-in `useInputImageFormat(boolean)` builder option to
`OpenAiChatModel`
and `OpenAiStreamingChatModel` for OpenAI-compatible servers that expect
the
`input_image` image payload.
- When enabled, image content is serialized as
`{"type":"input_image","image_url":"data:..."}` (string `image_url`, no
`detail`)
instead of the default Chat Completions
`{"type":"image_url","image_url":{"url":"data:...","detail":"..."}}`.
- Disabled by default — existing `image_url` object payload is
unchanged.
- `Content` now supports serialization **and** deserialization of both
the
object-shaped `image_url` and the string-shaped `input_image` form.
Fixes#4551
## Usage
```java
OpenAiChatModel model = OpenAiChatModel.builder()
.baseUrl(...)
.apiKey(...)
.modelName(...)
.useInputImageFormat(true) // opt in to the input_image payload
.build();
```
Testing
- Added OpenAiImageContentFormatTest covering:
- default image_url object format (serialization) for chat and streaming
chat
- input_image string format when useInputImageFormat(true) is set
- deserialization of both the object-shaped image_url and string-shaped
input_image forms
- <fill in: ran mvn -pl langchain4j-open-ai test locally / relying on
CI>
Checklist
- [x] No breaking changes (default behavior unchanged; new option
defaults to false)
- [ ] No new dependencies added
- [ ] New code follows existing OpenAI builder conventions (boolean
opt-in, mirrors sendThinking/returnThinking)
- [ ] Tests added and passing
## Issue
Closes#5116
## Change
Adds a new **`onUnmappedRawEvent`** streaming callback that gives
advanced users access to provider streaming events that LangChain4j does
**not** map to one of its typed callbacks — e.g. OpenAI server-tool
lifecycle events (`response.web_search_call.in_progress` / `searching` /
`completed`). This is an escape hatch so power users don't have to fall
back to a provider's native SDK for events we don't model yet.
**API** (both `@Experimental`, `@since 1.17.0`, `default` so nothing
breaks):
- `StreamingChatResponseHandler.onUnmappedRawEvent(Object rawEvent)` —
low-level API
- `TokenStream.onUnmappedRawEvent(Consumer<Object> rawEventHandler)` —
AI Services API
**Semantics — no duplication.** The callback fires **only** for events
that were *not* already delivered via a typed callback
(`onPartialResponse`, `onPartialThinking`, `onPartialToolCall`,
`onCompleteToolCall`, `onCompleteResponse`). So you can consume the
typed callbacks and the raw stream together without seeing the same
event twice. This is enforced by a small internal
`MappingTrackingStreamingChatResponseHandler` that records whether an
event was mapped to a typed callback (`wasMapped()`), and providers emit
the raw event only when it wasn't.
> Naming note: it's called `onUnmappedRawEvent` (not `onRawEvent`) to
leave room for a planned follow-up that also exposes the raw event
behind *mapped* callbacks (e.g. `PartialToolCall.rawEvent()`), giving a
clean mapped/unmapped split.
**The concrete `rawEvent` type is provider-specific:**
| Provider | Raw event type |
|---|---|
| OpenAI, Anthropic, Google AI Gemini, Mistral, Ollama |
`dev.langchain4j.http.client.sse.ServerSentEvent` |
| OpenAI (official) – Responses API |
`com.openai.models.responses.ResponseStreamEvent` |
| OpenAI (official) – Chat Completions API |
`com.openai.models.chat.completions.ChatCompletionChunk` |
| Amazon Bedrock |
`software.amazon.awssdk.services.bedrockruntime.model.ConverseStreamOutput`
|
| Google GenAI | `com.google.genai.types.GenerateContentResponse` |
**Providers wired:** OpenAI (Chat Completions + Responses),
OpenAI-official (Chat Completions + Responses), Anthropic, Google AI
Gemini, Google GenAI, Amazon Bedrock, Mistral, Ollama.
**Docs:** added a "Unmapped Raw Events" section to
`response-streaming.md` and the callback to the `TokenStream` example in
`ai-services.md`.
## General checklist
- [x] There are no breaking changes (API, behaviour) — all additions are
`default` and `@Experimental`
- [x] I have added unit and/or integration tests for my change
- [x] The tests cover both positive and negative cases — i.e. unmapped
events are forwarded *and* typed (text/tool) events are **not** repeated
as raw events
- [x] I have manually run all the unit and integration tests in the
module(s) I have added/changed, and they are all green
- [x] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [x] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
## Issue
Closes#5560
## Change
`OpenAiStreamingResponseBuilder` now accumulates logprobs from streamed
choices and exposes them in the response metadata.
- `append(ChatCompletionResponse)` reads `choice.logprobs().content()`
from each chunk and appends it, in order, to a thread-safe
`CopyOnWriteArrayList` (this builder is called from streaming threads).
- `buildMetadata()` converts the accumulated content with the existing
`OpenAiUtils.logProbsFrom(...)` helper and sets it via `.logProbs(...)`.
When nothing was accumulated (logprobs not requested), it stays `null`.
This closes the parity gap with the non-streaming
`OpenAiChatModel.doChat()`, which already populates logprobs. PR #4306
added logprobs only to the non-streaming path; this completes the
streaming sibling.
No public signature changes; behavior is unchanged when logprobs are not
requested (backward compatible).
Added two unit tests to `OpenAiStreamingResponseBuilderTest` (no API key
or mock needed): one asserts logprobs accumulate in order across chunks,
one asserts the field stays `null` when no logprobs are present.
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
<!-- N/A — core/main modules not touched; change is confined to
langchain4j-open-ai internal builder -->
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- N/A — documentation/examples are added after review per template
guidance -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
<!-- "new maven module" / "new embedding store" / "changing embedding
store" 섹션 삭제: 신규 모듈도 embedding store도 아닌 기존 통합 내부 버그픽스이므로 해당 없음 -->
## Summary
- Adds GPT image model names (`gpt-image-1`, `gpt-image-1-mini`,
`gpt-image-1.5`, `gpt-image-2`, `chatgpt-image-latest`) and deprecates
`DALL_E_2`/`DALL_E_3`
- Removes `style` and `response_format` parameters that are rejected by
the API (see #5356 for context on the 400 error)
- Adds new GPT image model parameters: `background`, `output_format`,
`output_compression`, `moderation`
- Captures full API response fields including token `usage` info
- Sets `mimeType` on `Image` from the response's `output_format`
- Updates both `langchain4j-open-ai` (legacy) and
`langchain4j-open-ai-official` modules
## Test plan
- [x] Both modules compile cleanly (`mvn test-compile`)
- [ ] Run `OpenAiImageModelIT` manually with `OPENAI_API_KEY` set
- [ ] Run `OpenAiOfficialImageModelIT` manually with `OPENAI_API_KEY`
set
Closes#5371
Signed-off-by: Eric Deandrea <eric.deandrea@ibm.com>
## Summary
Some OpenAI-compatible providers (observed with `deepseek-v4-flash`)
emit a trailing sentinel chunk after all real `tool_calls` have
streamed:
```json
{"index": 0, "id": "", "type": "function", "function": {"arguments": null}}
```
It carries an empty id, no `function.name`, and null
`function.arguments`. When its index does not match a previously-seen
tool call (e.g., arrives at a fresh `index`), the accumulator's
`computeIfAbsent` creates an empty builder that surfaces as a ghost
`ToolExecutionRequest { id = "", name = "", arguments = "" }` in the
final result. Downstream code in `OpenAiUtils#toOpenAiMessage` then
substitutes `"{}"` for the blank arguments when echoing the assistant
message back, producing the well-known `ToolExecutionRequest { id =
null, name = null, arguments = "{}" }` symptom in user-facing
tool-execution chains.
The fix adds an `isSentinel(toolCall)` guard at the top of the
`tool_calls` accumulation loop in `OpenAiStreamingResponseBuilder` that
treats end-of-stream marker frames as no-ops (skipped via `continue`)
before `computeIfAbsent` is invoked. Legitimate header chunks (id+name
set) and argument-fragment chunks (non-null arguments) are unaffected
because they have at least one of `id`, `function.name`, or non-null
`function.arguments` populated.
## Test plan
- [x] New `should_ignore_trailing_sentinel_chunk_from_deepseek_v4_flash`
exercises the exact stream pattern: header → argument fragments →
sentinel — asserts exactly 1 `ToolExecutionRequest` with the full
reconstructed arguments.
- [x] New `should_not_produce_ghost_for_orphan_sentinel_chunk` exercises
a sentinel arriving at a fresh `index` with no prior data — asserts no
ghost is produced. Without the fix this test produces
`ToolExecutionRequest { id = "", name = "", arguments = "" }`.
- [x] All 138 existing unit tests in `langchain4j-open-ai` continue to
pass.
---------
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
## Issue
Relates to #4838
## Change
This PR adds request-side support for OpenAI built-in tools in the
unofficial Responses models:
- `OpenAiResponsesChatModel`
- `OpenAiResponsesStreamingChatModel`
The unofficial API shape stays minimal and provider-specific by
introducing `serverTools` as `List<Map<String, Object>>` on
`OpenAiResponsesChatRequestParameters` and the two model builders.
`toolSpecifications` continues to represent LangChain4j function tools,
while `serverTools` carries raw OpenAI-shaped built-in tools. Both are
merged into the outgoing OpenAI `tools` array, with function tools
serialized first and built-in tools appended after them.
This keeps the unofficial implementation close to the OpenAI wire format
without introducing a new `OpenAiServerTool` abstraction in this patch.
Added coverage includes:
- storing and overriding `serverTools` in request parameters
- propagating `serverTools` through unofficial model builders
- payload construction for built-in tools only
- payload construction for function tools only
- payload construction for mixed function tools and built-in tools
- `toolChoice` serialization when mixed tools are present
Note: this change is intentionally limited to request support. It does
not add built-in tool result extraction yet.
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [X] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for adding new maven module
- [ ] I have added my new module in the root `pom.xml` and
`langchain4j-bom/pom.xml`
## Checklist for adding new embedding store integration
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that
extends from `EmbeddingStoreWithRemovalIT`
## Checklist for changing existing embedding store integration
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
Fixes#4931.
## Summary
- fail explicitly when an OpenAI-compatible chat completion response
contains more than one choice
- preserve the existing no-choice guard and avoid silently ignoring
later choices that may contain tool calls
- add a regression test for the reported text-first/tool-call-second
response shape
## Validation
-
`JAVA_HOME=/opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk/Contents/Home
PATH=/opt/homebrew/opt/openjdk@21/bin:$PATH ./mvnw -pl
langchain4j-open-ai -DskipITs -DskipIT -DskipIntegrationTests
-Dtest=OpenAiUtilsTest test`
- `git diff --check`
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
Closes#5128
## Summary
- Adds a `Boolean strict` field to `ToolSpecification` enabling per-tool
strict schema enforcement
- When `null` (default), defers to the model-level `strictTools` setting
— fully backward compatible
- When `true`/`false`, overrides the model-level default for that
specific tool
- Updates `AnthropicMapper` and `OpenAiUtils` to resolve per-tool strict
before falling back to the model default
- Includes JSON serialization/deserialization support and tests for all
three modules
## Motivation
Anthropic limits strict tools to **20 per request**. Agentic
architectures that send 20+ tools in a single request cannot use
`strictTools(true)` on the model without hitting this limit. Per-tool
strict allows selectively enforcing strict on high-value tools while
leaving others non-strict.
## Usage
```java
ToolSpecification.builder()
.name("runSql")
.description("Execute a SQL query")
.parameters(schema)
.strict(true) // enforce strict for this tool
.build();
```
## Test plan
- [x] `ToolSpecificationJsonTest` — round-trip for `strict` true, false,
null; rejection of non-boolean
- [x] `AnthropicMapperTest` — per-tool true overrides model null,
per-tool false overrides model true, per-tool null falls back to model
- [x] `OpenAiUtilsTest` — same three override/fallback scenarios
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Issue
Closes#5142
## Change
- Adds `customParameters(Map<String, Object>)` to `OpenAiEmbeddingModel`
so OpenAI-compatible embedding providers can receive provider-specific
JSON body parameters.
- Expands custom embedding parameters into the request body via
`@JsonAnyGetter`, following the existing OpenAI chat and Anthropic
`customParameters` style.
- Keeps the default request body unchanged when custom parameters are
not configured and snapshots the configured map when the model is built.
- Adds focused tests for opt-in behavior, default behavior, and
build-time map snapshotting.
- Adds a short documentation example for provider-specific embedding
parameters.
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [ ] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [X] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for adding new maven module
- [ ] I have added my new module in the root `pom.xml` and
`langchain4j-bom/pom.xml`
## Checklist for adding new embedding store integration
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that
extends from `EmbeddingStoreWithRemovalIT`
## Checklist for changing existing embedding store integration
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
## Tests run
- `./mvnw -pl langchain4j-open-ai -am -Dtest=OpenAiEmbeddingModelTest
-Dsurefire.failIfNoSpecifiedTests=false test`
- `git diff --check origin/main...HEAD`
Fixes#5075
## Summary
The `should_return_reasoning_summary` integration test is flaky. The
OpenAI Responses API can return an empty reasoning summary (`"summary":
[]`) even when `reasoning.summary=auto` is requested, causing
`assertThat(aiMessage.thinking()).isNotBlank()` to fail with `Expecting
not blank but was: null`.
This happens when the model doesn't produce internal reasoning (e.g.
`reasoning_tokens: 0`), particularly with `reasoningEffort("low")` on
simple factual questions.
Failed CI run: [integration_test
(21)](https://github.com/langchain4j/langchain4j/actions/runs/25183116986/job/73834305900)
## Change
Three adjustments to make the test more robust:
- **Use `reasoningEffort("medium")`** instead of `"low"` — consistent
with the encrypted reasoning tests in the same class that are stable
- **Replace simple factual question** ("What is the capital of
Germany?") with a CRT-style problem that forces step-by-step reasoning
- **Use `satisfiesAnyOf(isNotBlank(), isNull())`** as a safety net when
the model still doesn't produce a reasoning summary
Additional formatting changes are from `./mvnw spotless:apply` as
required by the contributing guidelines.
---------
Signed-off-by: Matheus Oliveira <matheus.6148@gmail.com>
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
## Issue
Closes#4810
## Change
Added null/empty guard for `choices` in two places:
1. **`OpenAiChatModel.doChat()`** — added
`isNullOrEmpty(openAiResponse.choices())` check before accessing
`.get(0)`, throwing `IllegalArgumentException` with a descriptive
message.
2. **`OpenAiUtils.aiMessageFrom()`** — added the same guard in this
shared utility method which also accessed `choices().get(0)` without
protection.
Both are consistent with the streaming counterpart
`OpenAiStreamingChatModel` which already has `isNullOrEmpty(choices)`
guard at line 188-189.
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
## Change
Adds PDF file content support across the OpenAI integrations, with
behavior aligned to the underlying OpenAI API used by each
implementation.
For `langchain4j-open-ai` (Responses API), `PdfFileContent` is now
supported in user messages. Inline/base64 PDFs are sent as `file_data`,
while URL-backed PDFs are sent as `file_url`. Unit tests cover both
mappings, and a disabled live integration test documents that public PDF
URLs work end-to-end against the Responses API.
For `langchain4j-open-ai-official`, PDF support is now split correctly
by API path:
- the Chat Completions-based models support inline/base64 PDFs, but
reject URL-backed PDFs explicitly because Chat Completions does not
support file URLs
- the official Responses streaming model supports both inline/base64
PDFs and URL-backed PDFs, and a disabled live integration test verifies
the URL-based case against the live API
This keeps the PDF behavior explicit and consistent with the actual
capabilities of the OpenAI endpoints behind each model.
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [ ] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for adding new maven module
- [ ] I have added my new module in the root `pom.xml` and
`langchain4j-bom/pom.xml`
## Checklist for adding new embedding store integration
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that
extends from `EmbeddingStoreWithRemovalIT`
## Checklist for changing existing embedding store integration
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
---------
Co-authored-by: Joshua Sperry <jrsperry@halosight.com>
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
## Change
- Add `reasoningSummary` parameter to `OpenAiResponsesChatModel` and
`OpenAiResponsesStreamingChatModel` to request reasoning summaries via
the Responses API (`reasoning.summary` field)
- Parse reasoning summary from response output items and expose via
`AiMessage.thinking()`
- For streaming, `onPartialThinking()` is called when reasoning summary
text is streamed
- Support round-tripping of encrypted reasoning content
(`reasoning.encrypted_content`) for stateless conversations: extracted
from responses into `AiMessage.attributes()` and automatically sent back
in follow-up requests
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [X] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Change
- Added `OpenAiResponsesChatModel` as a non-streaming `ChatModel`
implementation backed by OpenAI `/v1/responses`.
- Extended `OpenAiResponsesClient` with synchronous `chat(...)` support
and response parsing for:
- assistant text output
- tool execution requests
- response metadata and token usage
- Added a new integration test `OpenAiResponsesChatModelIT` that
exercises a basic tool-calling flow:
- first request asks model to call `create_person`
- second request sends tool result
- final assertion verifies person details in assistant response
- Updated OpenAI docs to include non-streaming Responses API usage and
note GPT-5.4 requirement to use Responses API for tools + reasoning
effort.
## General checklist
- [x] There are no breaking changes (API, behaviour)
- [x] I have added unit and/or integration tests for my change
- [x] The tests cover both positive and negative cases
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [x] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [x] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for adding new maven module
- [ ] I have added my new module in the root `pom.xml` and
`langchain4j-bom/pom.xml`
## Checklist for adding new embedding store integration
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that
extends from `EmbeddingStoreWithRemovalIT`
## Checklist for changing existing embedding store integration
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
## Validation performed
- `./mvnw -pl langchain4j-open-ai spotless:check`
- `./mvnw -pl langchain4j-open-ai -Dtest=OpenAiResponsesChatModelIT
-DfailIfNoTests=false test`
---------
Co-authored-by: Joshua Sperry <jrsperry@halosight.com>
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
Follow-up on #3816.
This PR tightens OpenAI Responses support in both official and
unofficial integrations:
- keep `strict` disabled by default and align Responses-mode JSON-schema
handling with actual supported behavior
- add explicit `previousResponseId` request parameters instead of shared
mutable model state
- restore unofficial client logging and remove extra local
`maxToolCalls` truncation
- improve streaming behavior and test coverage for
reasoning/tool-call/error/completion paths
- simplify Response-related AI-service test overrides via shared hooks
- add license metadata required for
`internal/langchain4j-internal-test-retry`
- fix brittle Vertex AI Anthropic integration-test initialization by
moving the quota-gating condition out of a class with provider-dependent
static initialization
## General checklist
- [x] There are no breaking changes (API, behaviour)
- [x] I have added unit and/or integration tests for my change
- [x] The tests cover both positive and negative cases
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [x] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for adding new maven module
- [ ] I have added my new module in the root `pom.xml` and
`langchain4j-bom/pom.xml`
## Checklist for adding new embedding store integration
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that
extends from `EmbeddingStoreWithRemovalIT`
## Checklist for changing existing embedding store integration
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
---------
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
<!--
Thank you so much for your contribution!
Please fill in all the sections below.
Please open the PR as a draft initially. Once it is reviewed and
approved, we will ask you to add documentation and examples.
Please note that PRs with breaking changes or without tests will be
rejected.
Please note that PRs will be reviewed based on the priority of the
issues they address.
We ask for your patience. We are doing our best to review your PR as
quickly as possible.
Please refrain from pinging and asking when it will be reviewed. Thank
you for understanding!
-->
## Issue
<!-- Please specify the ID of the issue this PR is addressing. For
example: "Closes #1234" or "Fixes #1234" -->
Closes#4298
## Change
### Summary
Implements `logprobs` and `top_logprobs` support for `OpenAiChatModel`.
Users can now retrieve log probability information from OpenAI
responses, which is useful for assessing model confidence and exploring
alternative token choices.
### Implementation Details
Added two new parameters to `OpenAiChatModel`:
- `logprobs(Boolean)` - enables log probability output
- `topLogprobs(Integer)` - specifies number of alternative tokens to
return (0-20)
**New DTOs:**
Created internal DTOs (`LogProbs`, `TokenLogProb`, `TopLogProb`) to
strictly match OpenAI's API structure.
**Metadata Exposure:**
As suggested by @dliubarskyi, the logprobs data is exposed via
`OpenAiChatResponseMetadata` rather than the generic metadata map,
ensuring type safety.
### Example Usage
```java
ChatRequest request = ChatRequest.builder()
.messages(UserMessage.from("Hello"))
.parameters(OpenAiChatRequestParameters.builder()
.logprobs(true)
.topLogprobs(2)
.build())
.build();
ChatResponse response = model.chat(request);
// Access strongly-typed metadata
OpenAiChatResponseMetadata metadata = (OpenAiChatResponseMetadata) response.metadata();
LogProbs logprobs = metadata.logprobs();
```
### Testing
- Added integration test should_return_logprobs in OpenAiChatModelIT.
- Verified that parameters are correctly sent to the API.
- Verified that the response structure (including nested top_logprobs)
is correctly parsed into the new DTOs.
- Tested successfully against the live OpenAI API.
## General checklist
<!-- Please double-check the following points and mark them like this:
[X] -->
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- Before adding documentation and example(s) (below), please wait
until the PR is reviewed and approved. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for adding new maven module
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added my new module in the root `pom.xml` and
`langchain4j-bom/pom.xml`
## Checklist for adding new embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that
extends from `EmbeddingStoreWithRemovalIT`
## Checklist for changing existing embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
---------
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
<!--
Thank you so much for your contribution!
Please fill in all the sections below.
Please open the PR as a draft initially. Once it is reviewed and
approved, we will ask you to add documentation and examples.
Please note that PRs with breaking changes or without tests will be
rejected.
Please note that PRs will be reviewed based on the priority of the
issues they address.
We ask for your patience. We are doing our best to review your PR as
quickly as possible.
Please refrain from pinging and asking when it will be reviewed. Thank
you for understanding!
-->
## Issue
<!-- Please specify the ID of the issue this PR is addressing. For
example: "Closes #1234" or "Fixes #1234" -->
Closes#4889
## Change
<!-- Please describe the changes you made. -->
Fixed a bug where `OpenAiStreamingResponseBuilder.append()` dropped
subsequent tool calls if a single streaming `Delta` chunk contained
multiple complete tool calls.
Changed the logic from extracting just the first element
(`delta.toolCalls().get(0)`) to iterating over all entries using a `for
(ToolCall toolCall : delta.toolCalls())` loop. Also added a unit test
`should_keep_all_tool_calls_from_same_delta` in
`OpenAiStreamingResponseBuilderTest` to verify that all tool calls in
the same delta are properly accumulated.
## General checklist
<!-- Please double-check the following points and mark them like this:
[X] -->
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- Before adding documentation and example(s) (below), please wait
until the PR is reviewed and approved. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
## Checklist for adding new maven module
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added my new module in the root `pom.xml` and
`langchain4j-bom/pom.xml`
## Checklist for adding new embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends
from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT`
- [ ] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that
extends from `EmbeddingStoreWithRemovalIT`
## Checklist for changing existing embedding store integration
<!-- Please double-check the following points and mark them like this:
[X] -->
- [ ] I have manually verified that the
`{NameOfIntegration}EmbeddingStore` works correctly with the data
persisted using the latest released version of LangChain4j
---------
Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
## Issue
Closes https://github.com/langchain4j/langchain4j/issues/4652
## Change
- Add support for `@Tool`-annotated methods to return multimodal content
(images) to the LLM, not just text. Tools can now return `Image`,
`ImageContent`, `Content`, `List<Content>`, `Content[]`, etc.
- Refactor `ToolExecutionResultMessage` to store `List<Content>`
internally instead of a plain `String`, with backward-compatible
`text()` accessor and new `contents()` / `hasSingleText()`
methods.
- Refactor `ToolExecutionResult` to support
`resultContents(List<Content>)` as an alternative to
`resultText(String)` and `resultTextSupplier(Supplier)`.
- Implement multimodal tool result mapping for providers that support
it: Anthropic, Amazon Bedrock, and Google AI Gemini.
- Add `UnsupportedFeatureException` guards in providers that do not
support non-text tool results: Azure OpenAI, GitHub Models, Jlama,
Mistral AI, Ollama, OpenAI (Chat Completions
& Responses), Vertex AI (Anthropic & Gemini), Watsonx, and Workers AI.
- Update `ToolExecutedEvent` / `DefaultToolExecutedEvent` to carry
`resultContents` alongside the existing `resultText` accessor.
- Add comprehensive unit and integration tests
(`should_execute_tool_returning_Image`,
`should_execute_tool_returning_ImageContent`,
`should_execute_tool_returning_ContentList`,
`should_fail_when_tool_returns_image_and_provider_does_not_support_it`).
- Update tools documentation with new "Returning Images and Multimodal
Content" and "Multimodal Tool Results" sections.
## General checklist
- [X] There are no breaking changes (API, behaviour)
- [X] I have added unit and/or integration tests for my change
- [X] The tests cover both positive and negative cases
- [X] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [X] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
- [X] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
- [ ] I have added/updated [Spring Boot
starter(s)](https://github.com/langchain4j/langchain4j-spring) (if
applicable)
brokenlander