- did some refactorings
- added javadoc
- fixed NPE in PineconeEmbeddingStoreImpl when adding embedding by id
- PineconeEmbeddingStoreImpl now takes into account minScore and returns
score in EmbeddingMatch
- InMemoryEmbeddingStore now returns score instead of cosine similarity
- all-minilm-l6-v2
- all-minilm-l6-v2-q
- e5-small-v2
- e5-small-v2-q
The idea is to give users an option to embed documents/texts in the same
Java process without any external dependencies.
ONNX Runtime is used to run models inside JVM.
Each model resides in it's own maven module (inside the jar).
Now, the StreamingChatLanguageModel can be used in conjunction with
tools.
One can send tool specifications along with a message to the LLM, and
the LLM can either stream a response or initiate a request to execute a
tool (also as a stream of tokens).
Sorry for a huge PR...
- added retries to OpenAiChatModel
- added @UserName: an option to define a name of a user as a parameter
in AI Services API
- added an option to split multiple documents at once (see
DocumentSplitter)
- redesigned document loaders (see FileSystemDocumentLoader)
- renamed DocumentSegment into TextSegment
- redesigned ConversationalRetrievalChain
- added EmbeddingStoreIngestor
- misc refactorings/fixes
---------
Co-authored-by: deep-learning-dynamo <deep-learning-dynamo@gmail.com>
Have a parent pom that contains most/all common things for the
sub-projects.
Note that it is separate from the root aggregator pom: not mixing the
aggregator and the parents makes things slightly easier.
If this change makes it harder to do releases, there might be a
possibility to generate the effective poms for each subproject, but on
the other hand releasing everything should not be too problematic.
The title says it all. Relying on [this maven
plugin](https://github.com/chonton/license-maven-plugin) for it.
Note that this adds a separate build step because we need a more recent
JDK to run the needed plugin.
This PR:
adds a github action for running unit tests
tests that require an OpenAI/HuggingFace token and hit their API are now considered integration tests (and have been renamed to end in IT)
integration tests are now run through a separate goal (mvn integration-test) via the maven-failsafe-plugin
to fix the PromptTemplate tests a Clock has been added to that class. Its constructor is now private: whether this is the convention we want to follow can be discussed
deep-learning-dynamo