Consider turning on caching for Guide object
#1283
Labels
Comments
torymur
pushed a commit
to dottxt-ai/outlines-core
that referenced
this issue
Dec 2, 2024
I understand that `pickleable` is not your priority right now. But the `RegexGuide` needs to be pickled for `vllm` production use, which is multiprocessing-based. This PR reintroduces this pickling capability + some tests. I understand that this introduces more effort on your side. References: dottxt-ai/outlines#1274 vllm-project/vllm#10490 vllm-project/vllm#10576 vllm-project/vllm#10489 It would also tackle the current caching issues: huggingface/text-generation-inference#2766 dottxt-ai/outlines#1283 Closes: #95
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is a reminder to reconsider turning on caching for the guide, which was turned off due to pickling issues of python bindings provided by
outlines-core: #1278Most likely this will be done after this issue: dottxt-ai/outlines-core#95
The text was updated successfully, but these errors were encountered: