stop_regex now supported for OpenAI

[slundberg]

Now you can use regular expressions to stop generations from OpenAI models using the stop_regex parameter! (previously this only worked for open source Transformer LLMs)

Below is an example where we use stop_regex to facilitate a tool use syntax:

import guidance

guidance.llm = guidance.llms.OpenAI("text-davinci-003", caching=False)

program = guidance("""Please solve the following word problem and call a calcuator with CALC(EQUATION) = ANSWER whenever you need to compute equations. For example: CALC((4+3) * 2) = 14.
Problem: {{problem}}
Reason step by step: {{gen 'text' stop_regex="CALC\\(.*?\\) =" max_tokens=100 save_stop_text=True}}""")
program(problem="Joe has ten apples and needs run 5 tests on each apple, if each test takes 7 minutes how long will this take Joe?")

# here we can see the stop text that was saved
out["text_stop_text"]

How does it work?
It streams the results from OpenAI and breaks the stream when we hit a regex match.

[3rd]

This is nice and very helpful for getting structured output.

Assuming this is using streaming, does OpenAI charge for the full amount of requested tokens, or only for those received before the stream was closed? I think it charges for the full amount, but couldn't find a definitive answer, do you know?