Switch to Streaming (WebSockets, SSE, HTTP Streaming)

[jmccand]

Websockets will enable asynchronous communication between backend/frontend, eliminating the potential 5 second delay.

Wait for migration to FastAPI in #269

[RichardJSun]

@jmccand are web sockets what we need here? is the client ever going to be sending data over to the server? I think there's other options like streaming

[jmccand]

Good point! There are other options. I've updated the Issue title. As for which one is best, it seems like Server Sent Events (SSE) might be best?
https://blog.pixelfreestudio.com/best-practices-for-real-time-data-streaming-in-web-apps/

SSEs are uni-directional (server to client) and easier to maintain and scale than WebSockets. HTTP streaming (like how LLMs stream responses) is also an option, but I think that's usually used for requests that terminate eventually. SSE is designed for long lived connections.

[RichardJSun]

I agree with you, I think SSE is probably the best option here because WebSockets require the server and the client to send each other pings to keep the connection open.

[williamschen23]

gRPC is a more complicated option that could work across multiple platforms and applications

[jmccand]

I don't think browsers have standard support for gRPC. If we're concerned about mobile, we may want to use websockets, because I think they have universal support across platforms.

[RichardJSun]

gRPC is a more complicated option that could work across multiple platforms and applications

I considered GRPC but I don’t think it’s necessary SSE should work on all platforms