Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(outlines): Add support outlines #1269

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

Aisuko
Copy link
Collaborator

@Aisuko Aisuko commented Nov 10, 2023

Description

This PR fixes #1261

Notes for Reviewers

Please kindly check the demo on Kaggle

Update

This PR has been tested

(outlines) @Aisuko ➜ /workspaces/LocalAI (feat/outlines) $  cd /workspaces/LocalAI ; /usr/bin/env /opt/conda/envs/outlines/bin/python /home/codespace/.vscode-remote/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 59803 -- /workspaces/LocalAI/backend/python/backend_outlines/backend_outlines.py 
Server started. Listening on: localhost:50051
Downloading (…)lve/main/config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 665/665 [00:00<00:00, 943kB/s]
Downloading model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 548M/548M [00:16<00:00, 32.5MB/s]
Downloading (…)neration_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 124/124 [00:00<00:00, 603kB/s]
Downloading (…)olve/main/vocab.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.04M/1.04M [00:00<00:00, 5.21MB/s]
Downloading (…)olve/main/merges.txt: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 456k/456k [00:00<00:00, 2.30MB/s]
Downloading (…)/main/tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.36M/1.36M [00:00<00:00, 65.7MB/s]
(outlines) @Aisuko ➜ /workspaces/LocalAI (feat/outlines) $  cd /workspaces/LocalAI ; /usr/bin/env /opt/conda/envs/outlines/bin/python /home/codespace/.vscode-remote/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 60939 -- /workspaces/LocalAI/backend/python/backend_outlines/backend_outlines.py 
Server started. Listening on: localhost:50051
(outlines) @Aisuko ➜ /workspaces/LocalAI (feat/outlines) $  cd /workspaces/LocalAI ; /usr/bin/env /opt/conda/envs/outlines/bin/python /home/codespace/.vscode-remote/extensions/ms-python.python-2023.20.0/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher 44581 -- /workspaces/LocalAI/backend/python/backend_outlines/test_backend_outlines.py 
...
----------------------------------------------------------------------
Ran 3 tests in 44.882s

OK

Signed commits

  • Yes, I signed my commits.

@Aisuko Aisuko changed the title Add support outlines feat(outlines): Add support outlines Nov 10, 2023
@Aisuko Aisuko self-assigned this Nov 10, 2023
@Aisuko Aisuko requested a review from mudler November 10, 2023 01:30
def Predict(self, request, context):
try:
#TODO: stop=["."] is a hard code, we need to make it configurable
output=generate.continuation(self.model, stop=["."])(self.prompt)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would make sense here to reuse the stopwords that are already sent as part of the request

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@mudler
Copy link
Owner

mudler commented Nov 10, 2023

great addition @Aisuko !

@mudler
Copy link
Owner

mudler commented Nov 10, 2023

@Aisuko how did you test it? can you give a small example to reproduce it?

@Aisuko
Copy link
Collaborator Author

Aisuko commented Nov 10, 2023

@Aisuko how did you test it? can you give a small example to reproduce it?

Hi, @mudler. I have not tested it. I am working on how to debug it. I will add an example after everything works out.

Signed-off-by: GitHub <[email protected]>
@Aisuko
Copy link
Collaborator Author

Aisuko commented Nov 15, 2023

@mudler The PR already rebased from master branch. And I also add the test cases for every functions. The result is in the above comment. Please take a look.

@@ -395,6 +396,7 @@ prepare-extra-conda-environments:
$(MAKE) -C backend/python/huggingface
$(MAKE) -C backend/python/vall-e-x
$(MAKE) -C backend/python/exllama
$(MAKE) -C backend/python/backend_outlines
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needs to be added also in the Dockerfile here:

## Duplicated from Makefile to avoid having a big layer that's hard to push

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

# get the directory where the bash script is located
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

python $DIR/backend_outlines.py $@
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure the file has executable bits, and the wrapper needs to be also listed here:

ENV EXTERNAL_GRPC_BACKENDS="huggingface-embeddings:/build/backend/python/huggingface/run.sh,autogptq:/build/backend/python/autogptq/run.sh,bark:/build/backend/python/bark/run.sh,diffusers:/build/backend/python/diffusers/run.sh,exllama:/build/backend/python/exllama/run.sh,vall-e-x:/build/backend/python/vall-e-x/run.sh,vllm:/build/backend/python/vllm/run.sh"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@mudler
Copy link
Owner

mudler commented Nov 15, 2023

is there maybe a way we can connect more complex features like https://github.com/outlines-dev/outlines#efficient-regex-guided-generation of outlines? I think the real gain would be to have guided output rather then generic output

@Aisuko
Copy link
Collaborator Author

Aisuko commented Nov 15, 2023

is there maybe a way we can connect more complex features like https://github.com/outlines-dev/outlines#efficient-regex-guided-generation of outlines? I think the real gain would be to have guided output rather then generic output

Agree with you, here the PR only including the basic one. And we can implement this.

However, if we want to connect regex-guided generation, we may need to add a new parameter for regex-guided to PredictOptions. Or there exists any other parameter we can use it.

What's your thought?

@rlouf
Copy link

rlouf commented Nov 19, 2023

Outlines author here, let me know if you need any help!

@mudler
Copy link
Owner

mudler commented Dec 17, 2023

is there maybe a way we can connect more complex features like https://github.com/outlines-dev/outlines#efficient-regex-guided-generation of outlines? I think the real gain would be to have guided output rather then generic output

Agree with you, here the PR only including the basic one. And we can implement this.

However, if we want to connect regex-guided generation, we may need to add a new parameter for regex-guided to PredictOptions. Or there exists any other parameter we can use it.

What's your thought?

maybe we can re-use the grammar field?

But I see that outlines support JSON schema already, and that we have already as we support OpenAI functions, maybe we can re-use that feature and pass down to the backends also the json schema?
https://github.com/outlines-dev/outlines#efficient-json-generation-following-a-pydantic-model

@rlouf is there an example that I can find with JSON schemas? that'd be great so could probably help @Aisuko and me here to speed it up :)

@rlouf
Copy link

rlouf commented Dec 17, 2023

generate.json takes JSON Schemas as well as Pydantic models. I'll add an example in the README soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feat(outlines): Add support pf outlines
3 participants