Skip to content

voice.md

Latest commit

 

History

History
272 lines (214 loc) · 18.5 KB

voice.md

File metadata and controls

272 lines (214 loc) · 18.5 KB
Raw

Voice Mode

PortOS includes an optional voice assistant with support for fully local operation. Supports push-to-talk and hands-free continuous mode with barge-in (start speaking over a reply to interrupt it).

Privacy note: LLM, TTS, and pipeline orchestration always run locally. For speech-to-text, the default Web Speech API uses your browser's built-in recognition service — in Chrome/Chromium this forwards audio to a Google cloud endpoint. Switch stt.engine to whisper (whisper.cpp, local HTTP server) under Settings → Voice for fully-offline STT with no audio leaving the machine.

Stack

Stage Default engine Alternatives Local?
Speech-to-text Browser Web Speech API (default — note: Chromium browsers forward audio to a vendor cloud speech service) or whisper.cpp via whisper-server (HTTP :5562, fully local) — ✅ (whisper) / ⚠️ (web-speech)
LLM LM Studio (/v1/chat/completions) OpenAI-compatible local server ✅
Text-to-speech Kokoro-82M via kokoro-js (in-process) Piper (CLI) ✅
Voice activity AudioWorklet + RMS VAD (hands-free) or MediaRecorder (push-to-talk) — Web Speech mode bypasses server audio and posts final text via voice:text — ✅

The TTS engine is selectable in Settings → Voice → TTS engine.

FaceTime Audio control plane

FaceTime Audio controls are off by default and remain machine-local. On macOS, run npm run setup:facetime, grant the installed helper Accessibility permission in System Settings, choose the configured BlackHole devices in FaceTime, then enable FaceTime Audio in Settings → Features. Set a target name and E.164 phone number or email in Settings → Voice, save, and use Probe, Test call, or Hang up. The helper refuses ambiguous FaceTime surfaces and never uses coordinate clicks.

FaceTime Audio call bridge

The control plane can dial and hang up. Carrying the conversation takes two virtual audio devices and a browser tab.

Devices. npm run setup:facetime offers to run brew install blackhole-2ch blackhole-16ch. BlackHole is GPLv3 and is never bundled — PortOS asks, you install it, and declining is fine (dial and hang up keep working without it). In FaceTime, set the output to BlackHole 16ch and the microphone to BlackHole 2ch. Both must run at 48 kHz; Settings → Voice → Check setup verifies the label, rate, and channel count of each and names the exact problem when one is wrong. A device list that cannot be read at all reports that plainly rather than claiming the device is missing.

Call host. Open /voice/call-host in a browser tab on the Mac running PortOS and press Attach call host. Device permissions and setSinkId need a real browser profile, so this cannot live on the server. The page reads BlackHole 16ch (what FaceTime plays into), streams it to PortOS as 16 kHz mono PCM, and plays each reply back through BlackHole 2ch (what FaceTime hears as its microphone). An input-level meter and a one-second test tone confirm both directions before you rely on them.

The page fails closed and says why: a browser missing any required API names all of them at once, a missing or misconfigured device is named specifically, an unlabeled device list is reported as a missing microphone permission rather than a missing driver, and a second tab is refused — it holds a Web Lock, and the server refuses a second host independently, so two tabs can never double-answer one call.

Turns. A phone call has no push-to-talk, so the server decides where a turn ends: energy-based voice activity detection with 700 ms of trailing silence and a 20-second ceiling per utterance. Each utterance runs the existing voice pipeline — same persona, tools, confirm gate, and TTS settings as the widget. Speaking over a reply interrupts it, exactly as the widget's barge-in does.

Ending. The call ends when the caller hangs up, after 60 seconds of caller silence, at the configured maximum call length (Settings → Voice, default 15 minutes), or when the call-host tab goes away — a call nobody can hear is ended rather than left running. The helper's own view of the FaceTime window is the source of truth throughout: a probe that fails is treated as unknown, never as a hangup.

What is kept. A text transcript is appended to the daily journal, labelled Caller and PortOS. The call audio is never persisted, and the configured handle never appears in the transcript or its metadata.

Calls PortOS places on its own. Two opt-in paths can dial without you asking: the Persistent Mind's voice.call-user grant, and the critical-notification escalation in Settings → Voice. Both are off by default and share one gate and one budget — never while a browser tab can speak the message, never inside voice quiet hours, at most 3 calls per rolling 24 hours and at least 30 minutes apart, counted in durable state so a restart cannot reset them. Escalation additionally fires only for a critical notification that is still unread after escalateAfterMinutes. See Chief of Staff enhancement → Persistent Mind phone calls.

Calling PortOS back. Turn on Automatically answer incoming calls in Settings → Voice → FaceTime Audio (facetime.autoAnswer, off by default) to call the Mac from your phone or watch and talk to your Chief of Staff. Fail closed by construction: the helper's answer command presses only the Notification Center action naming your own configured identity — a call from any other handle is left ringing, and PortOS never logs anything that would name who it was. Answering also needs the call host tab open and attached on this Mac (same tab the outbound bridge uses); without it, an authorized call rings unanswered and a medium notification records the miss instead of silently dropping it. Quiet hours change only the greeting's tone, never whether the call is picked up — you placed it, so PortOS answers at any hour. Once answered, the call runs exactly like an outbound one: whisper STT → voice LLM/persona → Kokoro/Piper TTS, the same silence/max-duration/hangup rules, and a transcript in the daily journal. If the Persistent Mind is running, the call carries its persona and context and the transcript is handed back to it as a message on hangup, continuing the same conversation on its next wake; if the mind isn't running, it's the plain voice persona, same as the widget. The Mind tab shows an active-call chip with a Hang up button (voice:call:hangup) for either direction, from any tab — not just the one carrying the audio.

Meeting capture

The call-host page's second mode: /voice/call-host?mode=capture (or the Capture system audio tab on the same page) turns live audio from a Zoom/Meet/FaceTime meeting into a timestamped transcript in the daily journal and the Brain inbox — no dialing, no reply, no LLM call.

Setup. Reads the same BlackHole 16ch device as the call bridge, so it needs no separate driver — but only the input half: capture never plays anything back, so BlackHole 2ch is not required. Route the meeting app's output to BlackHole 16ch (a macOS Multi-Output Device lets you also hear it through your speakers at the same time).

Capturing. Press Start capture; PortOS transcribes continuously with the same whisper.cpp STT and energy-based endpointing (700 ms trailing silence) the call bridge uses, but stops there — it never runs the LLM/tools pipeline, so no AI provider is called while a meeting is being captured. Press Stop capture (or close the tab) to finalize: the transcript is appended to the daily journal under a "Meeting capture" heading with start/stop timestamps, and filed as a Brain inbox item with auto-classify off — exactly like a manually-typed thought with classification skipped. The usual summarize/tag flow applies only once you ask for it from the inbox.

Mutually exclusive with a call. Both modes read the same BlackHole device and the same host tab, so starting one while the other is active on this tab is refused with a specific reason rather than fighting over the device.

Why Kokoro is the default

Kokoro is a 82M-parameter frontier TTS model that runs in-process via ONNX Runtime + transformers.js — no Python, no extra binaries, cross-platform. Quality is significantly higher than Piper (more natural prosody, expressive pacing). First synthesis after server start has a 2–3 s cold start as the model loads; warm calls are 200–500 ms per sentence on CPU.

Use Piper instead if you need lower latency per call (~100 ms cold start), are on a memory-constrained machine, or want a particular pre-trained voice from rhasspy's catalogue.

First-time setup

  1. Open PortOS → Settings → Voice.
  2. Pick your TTS engine (default: Kokoro), Whisper model size, and CoreML toggle (macOS).
  3. Toggle Enable voice mode and click Save & Reconcile.

PortOS will:

  • Install whisper-cpp via Homebrew if missing (the whisper.cpp binary + its base models)
  • Download the selected Whisper ggml .bin model into ~/.portos/voice/models/
  • On macOS with CoreML enabled, download the matching <model>-encoder.mlmodelc (2–3× faster STT on Apple Silicon, requires a custom whisper.cpp build)
  • If Piper is selected, download the pre-built Piper binary + phonemize libs from rhasspy/piper and rhasspy/piper-phonemize GitHub Releases (Piper is not on Homebrew), then fetch the selected voice .onnx into ~/.portos/voice/voices/
  • Start portos-whisper under PM2

Kokoro models live under ~/.cache/huggingface/hub/ and download lazily on first synthesis.

You can also run the bootstrap script directly:

TTS_ENGINE=kokoro INSTALL_COREML=1 bash scripts/setup-voice.sh
TTS_ENGINE=piper VOICE_NAME=en_US-ryan-high bash scripts/setup-voice.sh
MODEL_NAME=ggml-small.en.bin bash scripts/setup-voice.sh

Configuration options

All options live in data/settings.json under voice (Settings UI patches this file).

Option Default Notes
enabled false Master toggle. Triggers reconcile on change.
hotkey Space Held to talk. Ignored while typing in inputs.
stt.model base.en tiny.en · base.en · small.en · medium.en · large-v3
stt.coreml false Optional on macOS. Enable to use the CoreML encoder companion (requires a custom whisper.cpp build with -DWHISPER_COREML=1).
stt.endpoint http://127.0.0.1:5562 whisper-server listen address (whisper engine only).
tts.engine kokoro kokoro or piper
tts.rate 1.0 Speech rate, 0.5–2.0
tts.kokoro.voice af_heart See Kokoro voices — A-grade are af_heart, af_bella.
tts.kokoro.dtype q8 q4 · q8 · fp16 · fp32 (size/quality trade-off)
tts.kokoro.modelId onnx-community/Kokoro-82M-v1.0-ONNX HuggingFace repo id
tts.piper.voice en_GB-jenny_dioco-medium Piper voice id (path-encoded)
tts.piper.voicePath ~/.portos/voice/voices/<voice>.onnx ONNX file location
llm.model auto auto picks first loaded LM Studio model
llm.systemPrompt (concise voice prompt) Edit to change personality
llm.fastPath.enabled false Fast-resolution cascade (see below). Off = every turn runs on the server LLM.
llm.fastPath.triggers true Tier 1: resolve navigation commands ("go to tasks") in the browser, no LLM.
llm.fastPath.browserLlm true Tier 2: answer simple/conversational turns with Chrome's on-device Gemini Nano.
llm.fastPath.browser.temperature 0.7 Nano sampling temperature (0–2).
llm.fastPath.browser.topK 3 Nano sampling top-K (1–128).

Using voice mode

Two modes, toggle with the headset icon in the voice widget:

  • Push-to-talk: click/hold the mic or hold the configured hotkey. Release to send. The hotkey ignores keypresses while an input/textarea is focused.
  • Hands-free: the mic stays live; an AudioWorklet computes an RMS envelope and auto-submits once you've been silent for vad.endOfSpeechMs (default 700 ms). Ambient noise is calibrated at session start.

Either mode: start speaking while the assistant is replying to interrupt it (barge-in). Click the square button to stop current TTS playback without sending a new turn.

With stt.engine = 'web-speech' the browser's SpeechRecognition handles STT entirely client-side; PortOS sends the final transcript as voice:text instead of shipping audio, which skips whisper.cpp and avoids a 250–800 ms server-side round-trip.

Fast-resolution cascade (lower latency)

The server LLM is the slowest part of a turn — a local model with tools attached can take several seconds. When llm.fastPath.enabled is on, the client triages each turn through faster tiers first and only falls through to the server LLM when it has to:

  1. Trigger (fastPath.triggers) — deterministic, offline. A navigation command ("go to tasks", "open the daily log") is matched against the ⌘K palette nav manifest and navigates immediately. No LLM.
  2. Browser LLM (fastPath.browserLlm) — Chrome's on-device Gemini Nano (the Prompt API: window.LanguageModel / legacy self.ai.languageModel) answers simple/conversational turns entirely in the browser (fast, private, offline). Nano is also asked to reply ESCALATE for anything that needs a real action.
  3. Server — the configured provider/model (recommend Ollama with a small model) via the existing pipeline. Handles every tool/action turn, personal-data retrieval, dictation, confirmations, and anything the fast tiers decline or can't run.

Notes:

  • The cascade only applies to client-produced transcripts — Web Speech STT or typed input. Whisper / hands-free audio turns have no client transcript to triage, so they stay fully server-driven. Set stt.engine = 'web-speech' to get the benefit.
  • Trigger/Nano replies are spoken through the server's configured TTS (POST /api/voice/public/synthesize), reusing the normal playback queue and echo-suppression, so barge-in keeps working.
  • Fast-tier turns are handled without a server round-trip, so they are not added to the server-side conversation history; a later server turn won't have that chit-chat in its context. Action turns (which always hit the server) are unaffected.
  • Nano availability is surfaced in Settings → Voice → Fast resolution. When it isn't downloaded/enabled (chrome://flags/#prompt-api-for-gemini-nano + #optimization-guide-on-device-model), tier 2 transparently falls through to the server.
  • Nothing here runs a model the user hasn't triggered — Nano only executes on a real spoken/typed turn, consistent with the no-cold-bootstrap AI policy.

Client modules: client/src/services/browserLlm.js (Nano client), client/src/services/voiceFastPath.js (the routing decision), wired into VoiceWidget.jsx.

Architecture

browser mic → MediaRecorder (PTT) OR AudioWorklet + RMS VAD (hands-free)
  → Socket.IO 'voice:turn' (audio) OR 'voice:text' (Web Speech final)
  → whisper.cpp /inference          (STT for audio path)
  → LM Studio /v1/chat (streaming)  (LLM)
  → sentence-boundary TTS dispatch  (Kokoro in-process | Piper CLI)
  → Socket.IO 'voice:tts:audio'     → Web Audio playback queue

Pipeline orchestration: server/services/voice/pipeline.js. The pipeline emits events as it runs:

  • voice:transcript — STT result
  • voice:llm:delta — each token delta from LM Studio
  • voice:llm:done — full assistant reply
  • voice:tts:audio — one WAV per sentence as soon as TTS finishes it
  • voice:idle — turn complete (or interrupted)

Barge-in works by aborting the shared AbortController tied to the current turn — the LLM stream is torn down and any queued TTS is discarded.

Endpoints

Method Path Purpose
GET /api/voice/status Health probes + active engines + binary/model presence
GET /api/voice/config Current merged voice config
PUT /api/voice/config Deep-merge patch; triggers PM2 + setup reconcile
GET /api/voice/voices Voices for the active TTS engine
POST /api/voice/test Body { text }, returns WAV bytes — verifies TTS

| GET | /api/voice/facetime/status | BlackHole device + helper + identity preflight | | POST | /api/voice/facetime/{probe,call,answer,hangup} | Machine-local call control |

Socket events are documented in server/sockets/voice.js — including the call-host bridge (voice:call:attach / voice:call:audio / voice:call:detach / voice:call:hangup inbound, voice:call:state / voice:call:tts outbound — voice:call:state broadcasts to every connected tab, not just the call host, so the Mind tab's active-call chip stays in sync) and meeting capture, which reuses the same voice:call:audio PCM frames (voice:capture:start / voice:capture:stop inbound, voice:capture:state outbound).

Troubleshooting

  • Whisper badge red — brew install whisper-cpp, then which whisper-server.
  • CoreML missing — re-run INSTALL_COREML=1 bash scripts/setup-voice.sh (or toggle voice off/on after enabling CoreML).
  • Kokoro shows lazy — model loads on first synthesis. Hit "Test voice" to warm it up.
  • Kokoro slow on first call — first call after server start downloads model (~80 MB for q8) and initializes the runtime. Subsequent calls are 200–500 ms.
  • Piper spawn fails — which piper and check voice file at ~/.portos/voice/voices/<name>.onnx.
  • LM Studio red — start LM Studio and load a chat model; the voice pipeline uses /v1/chat/completions.
  • No audio playback — browsers require a user gesture before AudioContext can play. Click the page once or press the mic button.

Performance notes

Engine Cold start Warm latency (per sentence) Quality
Kokoro q8 (CPU) 2–3 s 200–500 ms High
Kokoro fp32 (CPU) 3–5 s 400–900 ms Highest
Piper ~100 ms (CLI spawn) ~100 ms Mid
Whisper base.en (no CoreML) 0 (server resident) 400–800 ms / 2 s of audio Good
Whisper base.en + CoreML 0 150–300 ms / 2 s of audio Good
Whisper small.en + CoreML 0 300–600 ms / 2 s of audio Better