feat: extract output with regexes from LLMs #3491

mudler · 2024-09-07T10:09:18Z

This changset adds extract_regex to the LLM config. It is a list of regexes that can match output and will be used to re extract text from the LLM output. This is particularly useful for LLMs which outputs final results into tags.

This changset adds `extract_regex` to the LLM config. It is a list of regexes that can match output and will be used to re extract text from the LLM output. This is particularly useful for LLMs which outputs final results into tags. Signed-off-by: Ettore Di Giacinto <[email protected]>

netlify · 2024-09-07T10:09:33Z

✅ Deploy Preview for localai ready!

Name	Link
🔨 Latest commit	`866e8b2`
🔍 Latest deploy log	https://app.netlify.com/sites/localai/deploys/66e412892b355e0008b19ad8
😎 Deploy Preview	https://deploy-preview-3491--localai.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

dave-gray101 · 2024-09-07T18:57:50Z

core/backend/llm.go

+		mu.Lock()
+		reg, ok := cutstrings[r]
+		if !ok {
+			cutstrings[r] = regexp.MustCompile(r)


MustCompile strikes me as risky for user-submitted config files. Can we use Compile here and check for errors? I'd assume we rather either return an error or skip regex result selection versus a panic?

huh, good point. I didn't give much thought at this here as I mimicked what we already do with cutstrings here:

LocalAI/core/backend/llm.go

Line 184 in d51444d

cutstrings[c] = regexp.MustCompile(c)

From one end I think fatals error here should be required, because that would mean that the configuration is invalid - and probably should go back at the user into fixing it, from one hand probably we can smooth this up with some configuration to make errors more relaxed.

In any case the error here should be improved, panicking won't give a clear explanation to the user of what went wrong - I'm going with that first and then we can review what would be the best strategy here

Signed-off-by: Ettore Di Giacinto <[email protected]>

dave-gray101 reviewed Sep 7, 2024

View reviewed changes

Add tests, enhance output in case of configuration error

866e8b2

Signed-off-by: Ettore Di Giacinto <[email protected]>

mudler force-pushed the extract_regex branch from ccabb21 to 866e8b2 Compare September 13, 2024 10:23

mudler merged commit cf747bc into master Sep 13, 2024
29 of 30 checks passed

mudler deleted the extract_regex branch September 13, 2024 11:27

mudler added the enhancement New feature or request label Sep 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: extract output with regexes from LLMs #3491

feat: extract output with regexes from LLMs #3491

mudler commented Sep 7, 2024

netlify bot commented Sep 7, 2024 •

edited

Loading

dave-gray101 Sep 7, 2024

mudler Sep 13, 2024 •

edited

Loading

feat: extract output with regexes from LLMs #3491

feat: extract output with regexes from LLMs #3491

Conversation

mudler commented Sep 7, 2024

netlify bot commented Sep 7, 2024 • edited Loading

✅ Deploy Preview for localai ready!

dave-gray101 Sep 7, 2024

Choose a reason for hiding this comment

mudler Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

netlify bot commented Sep 7, 2024 •

edited

Loading

mudler Sep 13, 2024 •

edited

Loading