Skip to content

[Realtime] Use Local Studio as a first-class realtime voice provider #207

Description

@0xSero

Goal

Add Local Studio as a first-class realtime voice provider in Litter while preserving native WebRTC audio processing, shared-Rust lifecycle/transcript state, and iOS/Android parity.

Litter already has native libwebrtc sessions and Rust-owned voice transcript/handoff state. The new path should replace mobile-owned provider credentials/configuration with Local Studio capability discovery and session negotiation; it must not create a parallel voice UI or duplicate lifecycle reducers in Swift and Kotlin. The Local Studio broker contract is tracked in sybil-solutions/local-studio#346, and authenticated transport in 0xSero/alleycat#39.

Architectural boundary

  • Shared Rust owns provider selection, typed capabilities/errors, realtime session lifecycle, transcript normalization, reconnect state, and handoff orchestration.
  • AppClient owns direct server operations; AppStore receives authoritative events/snapshots and remains minimal.
  • Swift/Kotlin own libwebrtc peer connections, platform audio sessions/permissions, and render-only UI projection.
  • Local Studio owns credentials, provider/model selection policy, ephemeral session negotiation, and teardown.
  • Alleycat carries authenticated control/signaling; audio/media stays direct in the MVP.

Required behavior

  • Add a typed provider/capability model that distinguishes direct OpenAI realtime from Local Studio brokered realtime without platform string parsing.
  • Show only capabilities actually advertised by the selected server/model and surface explicit unavailability reasons.
  • Request an ephemeral session, exchange SDP or the negotiated signaling form, apply session updates, and close idempotently.
  • Persist enough shared state to recover cleanly after backgrounding/reconnect without duplicating a provider session.
  • Keep existing transcript, dynamic-tool, remote handoff, mute/speaker, interruption, and Live Activity/Watch behavior working.
  • Never persist or log provider secrets, SDP credentials, or controller credentials.

Acceptance criteria

  • Shared Rust unit/contract tests cover provider selection, capability normalization, lifecycle, reconnect, expiry, and redaction.
  • iOS and Android tests cover identical capability/error projections and native WebRTC offer/answer integration.
  • Direct OpenAI realtime remains available and regression-tested until an explicit removal decision is made.
  • A Local Studio + Alleycat + Litter e2e harness proves capability -> create -> media connected -> transcript/tool handoff -> background/reconnect -> close.
  • Physical iOS and Android device acceptance verifies microphone permission, two-way audio, interruption, speaker/mute, and reconnect; simulator/CI alone is insufficient.
  • End-to-end timings separate control negotiation, ICE/media connection, first transcript, and first audio response.
  • No shared protocol parsing or lifecycle state machine is added to Swift/Kotlin.

PR plan

  1. Shared Rust provider/capability and signaling contract with fixtures.
  2. iOS adapter and tests using existing voice UI.
  3. Android adapter and tests using existing voice UI.
  4. Cross-repository e2e harness and physical-device acceptance evidence.

This issue does not supersede #182; that issue tracks a bug in the existing direct Android realtime path and should close when its own fix is accepted.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpriority: criticalMust land in the current stabilization window

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions