Today the Azure OpenAI module does not have its own Tokenizer. It
actually uses the OpenAI one during its test phase
(`langchain4j-open-ai` dependency in `test` scope).
This PR is for Azure OpenAI to have its own Tokenizer. Not just for test
purposes (this PR removes the dependency to `langchain4j-open-ai`) but
also Azure OpenAI can be used for RAG without the need of the
`OpenAITokenizer`.
For that I had to change all the `ModelName` classes
(`AzureOpenAiChatModelName`, `AzureOpenAiEmbeddingModelName`...) to add
the model name (eg. `gpt-35-turbo-0301`), the model type (eg.
`gpt-3.5-turbo`, notice the dot `.` in `3.5`) and version.
The `AzureOpenAiTokenizer` is a copy/paste from `OpenAiTokenizer`.
All `AzureOpenAiTokenizerTest` tests pass.
## General checklist
<!-- Please double-check the following points and mark them like this:
[X] -->
- [x] There are no breaking changes
- [x] I have added unit and integration tests for my change
- [x] I have manually run all the unit and integration tests in the
module I have added/changed, and they are all green
- [ ] I have manually run all the unit and integration tests in the
[core](https://github.com/langchain4j/langchain4j/tree/main/langchain4j-core)
and
[main](https://github.com/langchain4j/langchain4j/tree/main/langchain4j)
modules, and they are all green
<!-- Before adding documentation and example(s) (below), please wait
until the PR is reviewed and approved. -->
- [ ] I have added/updated the
[documentation](https://github.com/langchain4j/langchain4j/tree/main/docs/docs)
- [ ] I have added an example in the [examples
repo](https://github.com/langchain4j/langchain4j-examples) (only for
"big" features)
There is currently a direct dependency on the `langchain4j-open-ai`
module in order to use the `OpenAiTokenizer` class.
This code moves that dependency to the `test` scope, so that end-users
don't get the full `langchain4j-open-ai` module (and its dependencies)
in their classpath at runtime.
LangChain4j