Programmatic streaming now supported with for partial in program(stream=True)

[slundberg]

Guidance now supports the convenient streaming of partial results programmatically (before it was just streamed to the notebook for display).

Here is an example of synchronous streaming

import guidance
guidance.llm = guidance.llms.Transformers("databricks/dolly-v2-3b", caching=False)

program = guidance("""Generate a list of 5 company names:
{{#geneach 'companies' num_iterations=5~}}
{{@index}}. "{{gen 'this'}}"
{{/geneach}}""")

for p in program(stream=True, silent=True):
    print(p.get("companies", []))

Here is an example of asynchronous streaming (run in a notebook event loop)

import guidance
guidance.llm = guidance.llms.Transformers("databricks/dolly-v2-3b", caching=False)

program = guidance("""Generate a list of 5 company names:
{{#geneach 'companies' num_iterations=5~}}
{{@index}}. "{{gen 'this'}}"
{{/geneach}}""")

async for p in program(stream=True, async_mode=True, silent=True):
    print(p.get("companies", []))

Here is an example of tracking just the new content

import guidance
guidance.llm = guidance.llms.Transformers("databricks/dolly-v2-3b", caching=False)

program = guidance("""Generate a list of 5 company names:
{{gen 'companies' stop="\\n\\n"}}""")

pos = 0
for p in program(stream=True, silent=True):
    val = p.get("companies", "")
    print(val[pos:], end="", flush=True)
    pos = len(val)

How does it work?
It returns a reference to the partially completed program object at the same refresh rate used for notebook display streaming. If you want to get just the difference of what might be new, you can track the current position in whatever variable of interest yourself during the loop.

Caveats
Since this is a new feature I expect we will need to update some of the library functions to set variable values more frequently so they stream nicer. If you run into a scenario like this post an issue (or a PR!).

[fracorallo]

It works perfectly, thanks @slundberg !

Do you also plan to support the update of the internal program conversation "history" ( program.text ), as it happens without streaming?
e.g.:

program = program(stream=False, silent=True)  # this updates the conversation history internally

[AtmehEsraa]

AttributeError: module 'guidance' has no attribute 'llms' @slundberg