LLM gateway in Rapira: worker I/O offload, unified API, provider routing

[roxblnfk]

Motivation

LLM calls fit the PHP worker model poorly. A single call waits seconds to minutes on network I/O, streams tokens, keeps billing after the client disconnects, and is limited by tokens rather than requests. With one request per worker, 32 workers means 32 concurrent generations, each worker blocked on an upstream socket.

Rapira owns both the network layer and the workers, so it can take this I/O off PHP.

1. Worker I/O offload

PHP describes the call, Rapira executes it, and the worker is released:

• Stream-through: the worker returns a deferred-response descriptor. Rapira streams tokens to the client, then invokes an optional completion callback (text, tool calls, usage) on any free worker.
• Continuation: the request is suspended, and a worker is invoked again once the result is ready.
• Blocking RPC: fallback for simple cases and CLI.

Client disconnect aborts the upstream request.

2. Unified API (core)

One provider-agnostic API: messages, multimodal input, tool calling, structured output, a streaming event model, normalized errors and usage. It is exposed as a typed PHP package (transport goes through Rapira) and, optionally, as an OpenAI-compatible endpoint.

Provider-specific features (prompt caching, reasoning blocks, citations) are carried as typed extensions rather than dropped to a lowest common denominator.

3. Routing and fallbacks

• Model aliases (fast, smart) map to an ordered list of provider/model pairs.
• Fallback on 429, 5xx and timeouts, with a per-provider circuit breaker.
• Policies: priority, cost, latency, weighted.
• API keys stay in Rapira and never reach PHP.
• Token-based rate limits and budgets.
• Metrics: latency, TTFT, tokens, cost, fallback rate.

Fallback is transparent only until the first token reaches the client. After that, the policy has to be explicit.

Future: streaming to WebSocket connections

Once Rapira supports WebSockets, the stream target can be decoupled from the initiating request. A worker starts a generation and targets an existing WS connection or channel instead of the HTTP response. Rapira pushes the token events there without any worker involvement.

• Multiple concurrent generations per connection, multiplexed by stream ID.
• Cancellation on connection close or on a client cancel message.
• Backpressure for slow clients: buffer limits, and a policy to drop or abort when they are exceeded.
• Broadcasting a single generation to a channel (shared sessions, collaborative UIs).

Prior art

OpenRouter, LiteLLM and Portkey are gateways; symfony/ai and prism-php/prism are PHP abstractions. None of them combines the gateway with a server that owns the workers, which is what makes offload and cancellation possible.

Open questions

1. Canonical schema: adopt OpenAI or Anthropic, or define our own with adapters?
2. Core or plugin (via the plugin handler API)?
3. Handoff mechanism: a special response object, RPC, or both?
4. Tool loops: should Rapira drive the agent loop and call PHP only to execute tools?
5. Mid-stream failure: fail, replay, or let the app decide?
6. Transport adapters for symfony/ai and Prism?

[rustatian]

I really do like the idea of developing some generic contracts for PHP to use any model with streaming, tools, errors, and so on.