langchain4j/langchain4j-open-ai
LangChain4j 8e4254fc20
make OpenAI tokenizer more precise (#346)
This PR is a rework of `OpenAiTokenizer`.
Added `OpenAiTokenizerIT` with lots of tests to ensure that
`OpenAiTokenizer` calculates token usage very close to OpenAI.
In most cases calculation is 1:1, in some corner cases the difference is
within 5%.
2023-12-12 16:45:16 +01:00
..
src make OpenAI tokenizer more precise (#346) 2023-12-12 16:45:16 +01:00
pom.xml OpenAI: Support parallel tool calling (#338) 2023-12-08 17:29:56 +01:00