Support for streaming?

[aesnyder]

This is really cool. Does this support streaming responses?

From the docs & code it appears not to but it would be really nice. Something like how RubyLLM supports:

chat.ask('what is the capital of North Carolina') do |chunk| 
  broadcast_to_frontend chunk
end

[scmmishra]

Hey, yes you're correct, at the moment streaming is not supported. We do want to add it, and can do it with a very similar API surface.

runner.run("I need help with a technical issue") do | chunk |
  broadcast chunk
end

There's a chance that a simple passthrough of the streaming output from RubyLLM upwards to the user code may just work, but I have my doubts since we had to monkey patch a few parts of the Chat interface, and also we're doing a lot of context switching under the hood.

We are planning to add structured output soon, streaming can be the next priority, but I cannot guarantee anything. Any feedback or contribution is welcomed :)

[agarcher]

we had to monkey patch a few parts of the Chat interface, and also we're doing a lot of context switching under the hood

Any thoughts on working with the ruby_llm maintainers to contribute something like this directly to the gem? Monkey patching is likely going to get brittle in the long term and it would be great to have something like this supported more natively. I opened a feature request in their repo, but suspect it will only get traction if there are actual contributions.

[scmmishra]

Any thoughts on working with the ruby_llm maintainers to contribute something like this directly to the gem?

Yes absolutely. I have been exploring ways to do this, hoping to contribute soon, just getting the current version to be more stable and complete

Right now the roadmap in my mind is to get streaming to work, add tracing support, and then see if we can figure out a way to work with the RubyLLM interface directly instead of monkey patching.

This can be marked as the v1 release of the project.

Let me know if you have any ideas :)

[agarcher]

FYI, I was able to find a path to get ai-agents like behaviour on the vanilla gem with streaming support. Here's the details: crmne/ruby_llm#326 (comment)