Open
Conversation
This commit captures the working ElevenLabs voice integration with full
multi-session routing support. This is the stable baseline before
migrating to OpenAI Realtime.
WHAT'S IN THIS COMMIT:
realtimeClientTools.ts
- Complete rewrite to support multi-session routing
- Added resolveSessionId() fuzzy matcher (folder name → partial → summary → ID prefix)
- messageClaudeCode and processPermissionRequest now require 'session' param (not optional)
- Added new switchSession tool for explicit tab switching
- Removed auto-navigate on messageClaudeCode (was defocusing user mid-drive)
voiceHooks.ts / contextFormatters.ts
- onVoiceStarted() now builds initial context covering ALL active sessions
- getSessionLabel() returns folder name (short, stable, voice-friendly)
- Session roster uses folder names throughout, not raw IDs
RealtimeVoiceSession.tsx / .web.tsx
- On connect: immediately pushes full session roster via sendContextualUpdate
so agent knows all sessions from the moment voice connects
- Uses getSessionLabel() for human-readable session names
index.ts
- Added LogBox.ignoreAllLogs(true) to suppress 15 pre-existing RN warnings
that were freezing all UI interaction on the iPhone
docs/elevenlabs-agent-setup.md (new)
- Full setup guide for the ElevenLabs agent
- System prompt with {{initialConversationContext}} dynamic variable injection
- SILENCE BEHAVIOR section: agent must NOT fill silence (driving use case)
- All 3 tool JSON schemas (messageClaudeCode, processPermissionRequest, switchSession)
- Dynamic variable setup instructions, silence/inactivity settings
13-task plan to replace ElevenLabs SDK with custom WebRTC client connecting to OpenAI Realtime API (gpt-realtime GA). Key design: - VoiceSession interface is the abstraction boundary - OpenAIRealtimeClient class (~200 LOC) handles WebRTC + data channel - Tool translator bridges existing realtimeClientTools to OpenAI format - System prompt moves from ElevenLabs dashboard to code - Spike task gates everything to prove DataChannel works first - Includes ICE recovery and 30-min session auto-reconnect for driving
Custom WebRTC client connecting to OpenAI gpt-realtime via RTCPeerConnection and RTCDataChannel. Uses livekit react-native-webrtc which is already installed and proven via spike test. Changes: OpenAIRealtimeClient, tool translator, system prompt in code, RealtimeVoiceSession swapped, RealtimeProvider simplified, RealtimeSession updated for API key flow, semantic VAD with eagerness low for driving.
Use mediaDevices from @livekit/react-native-webrtc instead of navigator.mediaDevices (which doesn't exist in React Native). Also update default model to gpt-realtime-1.5.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Retro discovered 1 rule(s) from your sessions.
Approved via
retro dash.