Feature: Windows/Linux voice support via browser audio playback

[MarkoSteenbergen]

Feature Request: Windows/Linux Support via Browser Audio Playback

Summary

Add cross-platform support for the voice system by streaming audio to a browser client instead of relying on macOS afplay.

Current Behavior

• Voice system uses afplay for audio playback (macOS only)
• Windows/Linux users cannot use voice notifications

Proposed Enhancement

Architecture

Hook (Stop) → events.jsonl → Observability Server → WebSocket → Browser → Voice Server → ElevenLabs → Audio

Instead of playing audio locally via afplay, the voice server returns audio data to a browser client that handles playback.

1. Voice Server Returns Audio (instead of playing locally)

// POST /tts returns audio/mpeg instead of playing via afplay
return new Response(audioBuffer, {
  headers: {
    "Content-Type": "audio/mpeg",
    "Access-Control-Allow-Origin": "*"
  }
});

2. Browser Client Fetches and Plays Audio

const response = await fetch(`${voiceServerUrl}/tts`, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ message: text, speaker: speaker })
});

const audioBlob = await response.blob();
const audio = new Audio(URL.createObjectURL(audioBlob));
await audio.play();

3. Audio Queue for Sequential Playback

Prevent overlapping speech when multiple events arrive:

const audioQueue: { text: string; speaker: string }[] = []
let isPlaying = false

async function processAudioQueue(): Promise<void> {
  if (audioQueue.length === 0) {
    isPlaying = false;
    return;
  }
  isPlaying = true;
  const item = audioQueue.shift()!;
  // fetch and play...
  currentAudio.onended = () => processAudioQueue();
}

4. WebSocket Event Streaming

Observability server watches the events JSONL file and streams new events via WebSocket:

// Server watches file, broadcasts new events
ws.send(JSON.stringify({ type: 'event', data: newEvent }));

// Client receives and triggers TTS
ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  if (shouldSpeak(data.data)) {
    queueSpeech(extractMessage(data.data), data.data.agent_name);
  }
};

Benefits

• Works on Windows, Linux, and macOS
• Browser handles audio context/permissions
• No platform-specific audio commands needed
• Can run on headless server with remote browser

Implementation Notes

• Voice server needs CORS headers for browser access
• Browser requires user interaction to unlock audio context (one-time)
• Fallback to browser's built-in SpeechSynthesis if ElevenLabs fails

Testing Done

• Tested on Windows with Dutch voices
• Audio queue prevents race conditions
• Works with multiple speakers/agents

---

Submitted by: Marko (happy PAI user)
Reference implementation: Custom observability + voice server setup

[3x-Projetos]

WSL2 Alternative: Windows PowerShell Playback

We implemented an alternative solution for WSL2 users in commit 85faf7b:

Approach

Instead of browser audio, we bypass WSLg's RDP audio pipeline by playing directly through Windows PowerShell:

WSL App → Copy WAV to Windows Temp → PowerShell MediaPlayer → Windows Audio → Speaker

Why?

WSLg's RDPSink causes audio artifacts (static, cuts). PowerShell playback is clean.

Implementation

• server-linux.ts detects WSL via WSL_DISTRO_NAME env var
• Copies audio to C:\Users\...\AppData\Local\Temp
• Uses System.Windows.Media.MediaPlayer for playback

Trade-offs

Approach	Pros	Cons
Browser Audio (#405)	Cross-platform, remote access	Requires browser open
PowerShell (our impl)	No browser needed, native	WSL2 only, ~100ms latency

Both solutions are valid - browser audio for flexibility, PowerShell for simplicity.

See: docs/WSLG-AUDIO-FIX.md

[Mutdogus]

We've implemented a working version of this in our local PAI setup that takes a slightly different architectural approach — thought it might be useful context.

What We Built

Instead of the WebSocket → browser architecture proposed here, we run a standalone Bun HTTP server (VoiceServer/server.ts) on port 8888 that:

1. Accepts POST /notify with {message, voice_enabled, voice_id} — hooks call this directly
2. Generates TTS audio via configurable backend (ElevenLabs, or any HTTP TTS API that returns audio)
3. Plays audio locally via afplay (macOS) — but the audio generation is decoupled from playback, so swapping afplay for browser playback or aplay (Linux) is a one-line change
4. Runs as a LaunchAgent (macOS) for persistence — could be a systemd service on Linux

The key insight: by making the VoiceServer a standalone HTTP service rather than embedded in hooks, any hook can trigger voice with a simple fetch() call. No WebSocket complexity needed.

Multi-Backend Support

We also extended it to support multiple TTS backends selectable via settings.json:

{
  "voiceServer": {
    "port": 8888,
    "defaultBackend": "elevenlabs"
  }
}

Hooks just POST to /notify and the server handles backend routing, audio format conversion, and playback.

Cross-Platform Path

The only macOS-specific piece is the afplay call in playAudio(). For cross-platform support:

• Linux: swap to aplay or paplay
• Windows: swap to powershell -c (New-Object Media.SoundPlayer $path).PlaySync()
• Universal: serve audio over HTTP and play in a browser tab (as this issue proposes)

Happy to submit a PR adapting this for upstream if there's interest. The VoiceServer is ~400 lines of TypeScript and has been running reliably in production.

[danielmiessler]

Hey @MarkoSteenbergen — Kai here, Daniel's AI assistant, doing the next pass of triage on the PAI repo.

Closing this out because Windows/WSL isn't something Daniel's planning to actively support in the near term — the toolchain assumptions (zsh/bash, launchd, kitty/cmux, macOS Shortcuts, etc.) run too deep through the codebase to maintain a parallel Windows path right now.

Honestly, our broader recommendation is macOS for AI work in general — not just for PAI. The current tooling ecosystem (Claude Code, Anthropic SDK, native voice/notification integrations, Bun on Apple Silicon) is meaningfully more pleasant on macOS today, and that's where most AI development gravity is right now. If you have a Mac available, that's the path we'd steer you toward.

Thanks for taking the time to file this — really appreciated. If a future PAI release adds Windows support, please reopen.

— Kai 🤖