Handle the three distinct endpoint host formats depending on location type (#5869)
Vertex AI uses different hostnames for global, multi-region (us/eu), and regional locations. Using the regional template for "global"/"us"/"eu" previously produced a non-existent host that resolved via wildcard DNS to an HTML 404 instead of a clear connection error. Fixes #5865 <!-- Thank you so much for your contribution! Please fill in all the sections below. Please open the PR as ready for review (not as a draft), with tests and documentation already included. Please note that PRs with breaking changes, or without tests and documentation, will be rejected. Please note that PRs will be reviewed based on the priority of the issues they address. We ask for your patience. We are doing our best to review your PR as quickly as possible. Please refrain from pinging and asking when it will be reviewed. Thank you for understanding! --> ## Issue <!-- Please specify the ID of the issue this PR is addressing. For example: "Closes #1234" or "Fixes #1234" --> Closes #5865 ## Change <!-- Please describe the changes you made. --> ## General checklist <!-- Please double-check the following points and mark them like this: [X] --> - [x] There are no breaking changes (API, behaviour) - [x] I have added unit and/or integration tests for my change - [x] The tests cover both positive and negative cases - [x] I have manually run all the unit and integration tests in the module I have added/changed, and they are all green - [x] I have manually run all the unit and integration tests in the [core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core) and [main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j) modules, and they are all green - [x] I have added/updated the [documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs) - [ ] I have added an example in the [examples repo](https://github.com/langchain4j/langchain4j-examples) (only for "big" features) - [ ] I have added/updated [Spring Boot starter(s)](https://github.com/langchain4j/langchain4j-spring) (if applicable) ## Checklist for adding new maven module <!-- Please double-check the following points and mark them like this: [X] --> - [x] I have added my new module in the root `pom.xml` and `langchain4j-bom/pom.xml` ## Checklist for adding new embedding store integration <!-- Please double-check the following points and mark them like this: [X] --> - [x] I have added a `{NameOfIntegration}EmbeddingStoreIT` that extends from either `EmbeddingStoreIT` or `EmbeddingStoreWithFilteringIT` - [x] I have added a `{NameOfIntegration}EmbeddingStoreRemovalIT` that extends from `EmbeddingStoreWithRemovalIT` ## Checklist for changing existing embedding store integration <!-- Please double-check the following points and mark them like this: [X] --> - [x] I have manually verified that the `{NameOfIntegration}EmbeddingStore` works correctly with the data persisted using the latest released version of LangChain4j --------- Co-authored-by: Dmytro Liubarskyi <ljubarskij@gmail.com>
This commit is contained in:
parent
45b058edf4
commit
2c504abaaf
|
|
@ -186,7 +186,7 @@ You can learn about the models in the [Claude model documentation](https://docs.
|
|||
```java
|
||||
ChatModel model = VertexAiAnthropicChatModel.builder()
|
||||
.project(PROJECT_ID) // your Google Cloud project ID
|
||||
.location(LOCATION) // the region where AI inference should take place
|
||||
.location(LOCATION) // where inference takes place, see "Locations" below
|
||||
.modelName(MODEL_NAME) // the Claude model used
|
||||
.maxTokens(4096) // the maximum number of tokens to generate
|
||||
.temperature(0.7) // temperature (between 0 and 1)
|
||||
|
|
@ -202,6 +202,30 @@ ChatModel model = VertexAiAnthropicChatModel.builder()
|
|||
|
||||
The same parameters are also available on the streaming chat model.
|
||||
|
||||
### Locations
|
||||
|
||||
The `location` decides where your requests are processed. Vertex AI offers three kinds of locations,
|
||||
and the value you pass to `.location(...)` determines which one is used:
|
||||
|
||||
| Location value | Kind | What it does |
|
||||
|---|---|---|
|
||||
| `"global"` | Global | Routes each request to any region with available capacity. Best availability, no pricing premium. Recommended unless you have data residency requirements. |
|
||||
| `"us"`, `"eu"` | Multi-region | Routes each request to a region within that geography, for data residency with high availability. |
|
||||
| `"us-east5"`, `"europe-west1"`, ... | Regional | Routes every request through one specific region. Required for single-region data residency and for provisioned throughput. |
|
||||
|
||||
```java
|
||||
ChatModel model = VertexAiAnthropicChatModel.builder()
|
||||
.project(PROJECT_ID)
|
||||
.location("global")
|
||||
.modelName(MODEL_NAME)
|
||||
.build();
|
||||
```
|
||||
|
||||
Note that model availability differs per location, and that multi-region and regional locations carry
|
||||
a pricing premium over the global one. See
|
||||
[Global, multi-region, and regional endpoints](https://platform.claude.com/docs/en/build-with-claude/claude-on-vertex-ai#global-multi-region-and-regional-endpoints)
|
||||
for details.
|
||||
|
||||
## More examples
|
||||
|
||||
Claude is a powerful multimodal model that accepts both text and images as input.
|
||||
|
|
|
|||
|
|
@ -106,7 +106,7 @@ public class VertexAiAnthropicChatModel implements ChatModel, Closeable {
|
|||
String requestModelName = getOrDefault(parameters.modelName(), modelName);
|
||||
|
||||
if (logRequests) {
|
||||
logger.debug("Base URL: {}-aiplatform.googleapis.com:443", location);
|
||||
logger.debug("Base URL: {}", VertexAiAnthropicClient.resolveEndpoint(location));
|
||||
logger.debug(
|
||||
"Using model name: {} (from parameters: {}, default: {})",
|
||||
requestModelName,
|
||||
|
|
|
|||
|
|
@ -17,13 +17,16 @@ import dev.langchain4j.model.vertexai.anthropic.internal.api.AnthropicResponse;
|
|||
import io.grpc.StatusRuntimeException;
|
||||
import java.io.IOException;
|
||||
import java.util.HashMap;
|
||||
import java.util.Locale;
|
||||
import java.util.Map;
|
||||
|
||||
public class VertexAiAnthropicClient {
|
||||
|
||||
private static final String PUBLISHER = "anthropic";
|
||||
|
||||
private static final String CREDENTIALS_ENDPOINT_TEMPLATE = "%s-aiplatform.googleapis.com:443";
|
||||
private static final String GLOBAL_ENDPOINT = "aiplatform.googleapis.com:443";
|
||||
private static final String MULTI_REGION_ENDPOINT_TEMPLATE = "aiplatform.%s.rep.googleapis.com:443";
|
||||
private static final String REGIONAL_ENDPOINT_TEMPLATE = "%s-aiplatform.googleapis.com:443";
|
||||
|
||||
private volatile PredictionServiceClient predictionServiceClient;
|
||||
private final String project;
|
||||
|
|
@ -49,7 +52,8 @@ public class VertexAiAnthropicClient {
|
|||
String ignoredModel = model;
|
||||
|
||||
this.project = project;
|
||||
this.location = location;
|
||||
// Vertex AI locations are lowercase, both in the endpoint host and in the resource name
|
||||
this.location = location.toLowerCase(Locale.ROOT);
|
||||
this.credentials = credentials;
|
||||
this.objectMapper = new ObjectMapper().disable(SerializationFeature.FAIL_ON_EMPTY_BEANS);
|
||||
this.predictionServiceClient = createClient();
|
||||
|
|
@ -58,7 +62,7 @@ public class VertexAiAnthropicClient {
|
|||
private PredictionServiceClient createClient() {
|
||||
try {
|
||||
PredictionServiceSettings.Builder settingsBuilder = PredictionServiceSettings.newBuilder();
|
||||
settingsBuilder.setEndpoint(String.format(CREDENTIALS_ENDPOINT_TEMPLATE, location.toLowerCase()));
|
||||
settingsBuilder.setEndpoint(resolveEndpoint(location));
|
||||
if (credentials != null) {
|
||||
GoogleCredentials scopedCredentials =
|
||||
credentials.createScoped("https://www.googleapis.com/auth/cloud-platform");
|
||||
|
|
@ -70,6 +74,30 @@ public class VertexAiAnthropicClient {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Maps a Vertex AI location to the API endpoint host serving it. Vertex AI uses a different host
|
||||
* format for each of its three location types:
|
||||
* <ul>
|
||||
* <li>{@code global} → {@code aiplatform.googleapis.com}</li>
|
||||
* <li>multi-region ({@code us}, {@code eu}) → {@code aiplatform.<location>.rep.googleapis.com}</li>
|
||||
* <li>region (e.g. {@code us-east5}) → {@code <location>-aiplatform.googleapis.com}</li>
|
||||
* </ul>
|
||||
* See <a href="https://platform.claude.com/docs/en/build-with-claude/claude-on-vertex-ai">Claude on Vertex AI</a>.
|
||||
*
|
||||
* @param location the Vertex AI location, case-insensitive
|
||||
* @return the {@code host:port} endpoint to use for {@link PredictionServiceSettings}
|
||||
*/
|
||||
public static String resolveEndpoint(String location) {
|
||||
String normalized = location.toLowerCase(Locale.ROOT);
|
||||
if ("global".equals(normalized)) {
|
||||
return GLOBAL_ENDPOINT;
|
||||
}
|
||||
if ("us".equals(normalized) || "eu".equals(normalized)) {
|
||||
return String.format(MULTI_REGION_ENDPOINT_TEMPLATE, normalized);
|
||||
}
|
||||
return String.format(REGIONAL_ENDPOINT_TEMPLATE, normalized);
|
||||
}
|
||||
|
||||
public AnthropicResponse generateContent(AnthropicRequest request, String modelName) throws IOException {
|
||||
return generateContentWithRetry(request, modelName, 1);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,46 @@
|
|||
package dev.langchain4j.model.vertexai.anthropic.internal.client;
|
||||
|
||||
import static org.assertj.core.api.Assertions.assertThat;
|
||||
|
||||
import org.junit.jupiter.api.Test;
|
||||
|
||||
/**
|
||||
* Basic validation of the {@link VertexAiAnthropicClient#resolveEndpoint(String)} behavior
|
||||
*/
|
||||
class VertexAiAnthropicClientTest {
|
||||
|
||||
@Test
|
||||
void should_resolve_global_endpoint_without_location_prefix() {
|
||||
assertThat(VertexAiAnthropicClient.resolveEndpoint("global")).isEqualTo("aiplatform.googleapis.com:443");
|
||||
}
|
||||
|
||||
@Test
|
||||
void should_resolve_multi_region_endpoints() {
|
||||
assertThat(VertexAiAnthropicClient.resolveEndpoint("us")).isEqualTo("aiplatform.us.rep.googleapis.com:443");
|
||||
assertThat(VertexAiAnthropicClient.resolveEndpoint("eu")).isEqualTo("aiplatform.eu.rep.googleapis.com:443");
|
||||
}
|
||||
|
||||
@Test
|
||||
void should_resolve_regional_endpoint() {
|
||||
assertThat(VertexAiAnthropicClient.resolveEndpoint("us-east5"))
|
||||
.isEqualTo("us-east5-aiplatform.googleapis.com:443");
|
||||
assertThat(VertexAiAnthropicClient.resolveEndpoint("europe-west1"))
|
||||
.isEqualTo("europe-west1-aiplatform.googleapis.com:443");
|
||||
}
|
||||
|
||||
@Test
|
||||
void should_not_treat_a_region_starting_with_a_multi_region_name_as_multi_region() {
|
||||
assertThat(VertexAiAnthropicClient.resolveEndpoint("us-central1"))
|
||||
.isEqualTo("us-central1-aiplatform.googleapis.com:443");
|
||||
assertThat(VertexAiAnthropicClient.resolveEndpoint("europe-west4"))
|
||||
.isEqualTo("europe-west4-aiplatform.googleapis.com:443");
|
||||
}
|
||||
|
||||
@Test
|
||||
void should_be_case_insensitive() {
|
||||
assertThat(VertexAiAnthropicClient.resolveEndpoint("GLOBAL")).isEqualTo("aiplatform.googleapis.com:443");
|
||||
assertThat(VertexAiAnthropicClient.resolveEndpoint("US")).isEqualTo("aiplatform.us.rep.googleapis.com:443");
|
||||
assertThat(VertexAiAnthropicClient.resolveEndpoint("US-East5"))
|
||||
.isEqualTo("us-east5-aiplatform.googleapis.com:443");
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue