feature: add INSTRUCTOR models as embedding backend

[zhekazuev]

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

These instructor models are well tuned for embeddings, as demonstrated here.

But as I understand it, they cannot be used as an embedding backend in localai, because it is not bert or llama.cpp models.

And as I see here, there is no way to use different embedding models than llama.cpp compatible models and bert models.

There is a way to use these embedding models via chroma, but it may be more reasonable to use embeddings via localai REST API.

Why localai, but not chroma?

Because the simpler and more correct way would be to use localai as the backend for embeddings, because then end-user can scale this solution, e.g. in kubernetes and make changes to the architecture of that backend fairly quickly.

How this would work through chroma is hard for me to say.
I think it's not quite right to use chroma as embeddings backend and also complicates the code a bit if someone decides to switch these models to openai bert for example via environment variable or something like that.

Example of model configuration:

name: "instructor-large-embeddings"
license: "Apache 2.0"
urls:
- https://huggingface.co/hkunlp/instructor-large
description: |
    Large Instructor model that can be used for embeddings
config_file: |
    parameters:
      model: hkunlp-instructor-large
    backend: instructor-embeddings
    embeddings: true
files:
- filename: "hkunlp-instructor-large"
  sha256: "fb7f63752cf10103be938bc50dfad0b6fa1e63bc67b963471fc827838d9bbb41"
  uri: "https://huggingface.co/hkunlp/instructor-large/resolve/main/pytorch_model.bin"

Ref: https://github.com/go-skynet/model-gallery

Describe alternatives you've considered

Example of code with instructor models for embedding in chroma vector database:

# import
# from langchain.embeddings.sentence_transformer import SentenceTransformerEmbeddings
from chromadb.utils import embedding_functions
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import Chroma
from langchain.document_loaders import TextLoader

# use localai instead of openai api
base_path = os.environ.get('OPENAI_API_BASE', 'http://localhost:8080/v1')

# load the document and split it into chunks
loader = TextLoader('state_of_the_union.txt')
documents = loader.load()

# split it into chunks
text_splitter = CharacterTextSplitter(chunk_size=300, chunk_overlap=70)
texts = text_splitter.split_documents(documents)

# create the open-source embedding function
# embedding_function = SentenceTransformerEmbeddings(model_name="hkunlp/instructor-xl")
embedding_function = embedding_functions.InstructorEmbeddingFunction(model_name="hkunlp/instructor-xl", device="cpu")

# load it into Chroma
db = Chroma.from_documents(texts, embedding_function)

# query it
query = "What did the president say about Ketanji Brown Jackson"
texts = db.similarity_search(query)

# print results
print(texts[0].page_content)

Additional context

Links:

• https://instructor-embedding.github.io/
• https://github.com/HKUNLP/instructor-embedding
• https://huggingface.co/spaces/mteb/leaderboard
• https://docs.trychroma.com/embeddings#instructor-models
• https://python.langchain.com/docs/modules/data_connection/vectorstores/integrations/chroma
• https://mudler.pm/posts/localai-question-answering/
• Models to add go-skynet/model-gallery#1

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

Thank you for the detailed description and context. Based on your explanation, it seems like the issue is related to the lack of support for instructor models in llama.cpp backends in localai.
To address this issue, one possible solution would be to add support for instructor models in localai, similar to how it supports bert models. This could involve updating the backend configuration in localai to include instructor models and providing instructions on how to use them via the REST API.
Another alternative would be to use chroma as the backend for embeddings, which already supports instructor models. However, this may require additional configuration and could potentially complicate the code if someone decides to switch these models to openai bert in the future.
Overall, it seems like adding support for instructor models in localai would be a simpler and more straightforward solution that would allow users to easily scale this solution via kubernetes and make changes to the architecture of the backend fairly quickly.

Sources:

• https://localai.io/basics/news/index.html
• https://localai.io/features/embeddings/index.html
• https://github.com/go-skynet/LocalAI/tree/master/README.md
• https://localai.io/models/index.html

[mudler]

this is now possible as internal backends where moved to gRPC services. This allows to have a backend, let's say, in python, given it satisfies our gRPC server implementation.

[mudler]

With #778 it was introduced the huggingface embedding in python - will follow along with docs soon but an example can be seen in https://github.com/go-skynet/LocalAI/blob/master/tests/models_fixtures/grpc.yaml , didn't tried yet with instructor embeddings but worth a shot?

[zhekazuev]

With #778 it was introduced the huggingface embedding in python - will follow along with docs soon but an example can be seen in https://github.com/go-skynet/LocalAI/blob/master/tests/models_fixtures/grpc.yaml , didn't tried yet with instructor embeddings but worth a shot?

Hi,

Thanks for your feedback.

Not tested it yet. We can close an issue now.