Skip to content

fix(meet-audio): zero-PCM keepalive so Meet keeps the bot joined (#2945)#3323

Closed
CodeGhost21 wants to merge 5 commits into
tinyhumansai:mainfrom
CodeGhost21:fix/2945-meet-audio-keepalive
Closed

fix(meet-audio): zero-PCM keepalive so Meet keeps the bot joined (#2945)#3323
CodeGhost21 wants to merge 5 commits into
tinyhumansai:mainfrom
CodeGhost21:fix/2945-meet-audio-keepalive

Conversation

@CodeGhost21

@CodeGhost21 CodeGhost21 commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Part of slice A of #2945 (root cause of the ~5 s drop). This is PR-A of a two-PR split — PR-B (post-admission watchdog) will follow once PR #3321 (slice B — diagnostics) merges.

Adds a permanent looping zero-PCM AudioBufferSourceNode to audio_bridge.js inside ensureContext(). Without it, the bot's WebRTC audio track stays readyState: live but produces zero PCM during silence — some Meet builds drop the bot for "no audio activity" after a few seconds. Mirrors the camera-bridge 1-pixel-bob keepalive (meet_video/camera_bridge.js:179–181 "keeps the WebRTC encoder from dropping the stream as 'frozen'").

The brain's PCM mixes in via the existing __openhumanFeedPcm path — zero + real = real, so live speech is unaffected. The keepalive source is not tracked in activeSources, so barge-in (__openhumanFlushAudio) does not stop it.

Evidence the bug is real

  • speak_pump.rs:270–282 pushes nothing on idle ticks — confirmed.
  • meet_scanner exits at admission with no watchdog (meet_scanner/mod.rs:353–399).
  • Production Sentry issue TAURI-RUST-8TM4 065 events of [meet-agent] no session for request_id=… from a single 107-minute call (src/openhuman/meet_agent/session.rs:865) — consistent with shell-side pumps running long after core dropped the session.
  • The camera-bridge keepalive comment is an explicit, documented precedent for the same failure mode on the video side; the audio side never got the equivalent fix.

Acceptance criteria addressed (from #2945)

  • Agent stays joinedprobable; hypothesis-driven fix. Requires real-Meet manual verification (see Test Plan below) before this box should be considered closed. If hypothesis is wrong, PR-B (watchdog) will detect and we escalate to a comfort-noise variant.
  • Rejoin state resolves successfully — same caveat as above; if the bot doesn't drop, the rejoin state is never entered.

Acceptance criteria still open:

Spec and plan

Test plan

  • cargo fmt --check clean on root + app/src-tauri
  • cargo check --manifest-path app/src-tauri/Cargo.toml clean (27 pre-existing warnings, no new errors)
  • cargo test meet_audio:: green — 8/8 including the new presence test audio_bridge_contains_keepalive_block
  • pnpm compile (tsc --noEmit) clean
  • prettier --check clean on audio_bridge.js

Manual verification deferred to reviewer / maintainer — the implementing agent cannot launch a real Google Meet session from this environment. Before merge, please run:

  1. pnpm dev:app, open Skills → Meeting Bots, join a real Google Meet.
  2. Confirm the bot stays in the call ≥60 s without dropping (the bug being fixed).
  3. Open chrome://webrtc-internals/, find the outbound audio track for our bot, confirm bytesSent is monotonically increasing during silence (not stuck at zero).
  4. Trigger a brain reply (say wake word + question); confirm speech still plays correctly (keepalive doesn't mute live PCM).
  5. Trigger barge-in (new question mid-reply); confirm in-flight brain audio still cuts (keepalive isn't tracked in activeSources, so __openhumanFlushAudio should still stop the brain's sources only).

Real Meet E2E for the drop/rejoin path is intentionally NOT included — there is no mock-Meet harness in CI today. Same rationale as PR #3321.

Rollback

Revert this PR. Zero state, zero migration, behavior returns to prior immediately.

If the hypothesis is wrong

Bot still drops at ~5 s after merge. We learn DTX is suppressing pure silence on the wire. Next step: ship PR-B (watchdog) so the drop is surfaced in production telemetry, then file a follow-up swapping the zero buffer for a -60 dB pseudo-random noise buffer (~5 extra lines). The 20-line cost of this PR is the cheapest probe of the hypothesis.

Summary by CodeRabbit

  • New Features

    • Added an audio keepalive so the bot stays connected during extended silent periods, preventing unexpected disconnections.
  • Tests

    • Added automated verification to ensure the audio keepalive is present in shipped builds.
  • Documentation

    • Added a design spec and implementation plan describing the keepalive approach, testing, and rollout steps.

PR-A of a two-PR split for slice A of #2945. Adds a permanent
looping zero-PCM AudioBufferSourceNode to audio_bridge.js so
Meet's WebRTC engine doesn't drop the bot for "no audio
activity" during silence. Mirrors the existing camera-bridge
1-pixel-bob keepalive pattern. PR-B (post-admission watchdog)
follows once PR tinyhumansai#3321 (slice B — diagnostics) merges.
3-task TDD plan for PR-A of slice A of #2945. Task 1 adds a
Rust presence test asserting the keepalive code ships in
audio_bridge.js (red). Task 2 adds the keepalive code itself
(green). Task 3 verifies, pushes to aniketh, opens the PR.
@CodeGhost21 CodeGhost21 requested a review from a team June 3, 2026 21:46
@coderabbitai

coderabbitai Bot commented Jun 3, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: fd813d5a-86ea-46ac-84e0-7e7c860a8190

📥 Commits

Reviewing files that changed from the base of the PR and between 2105f1d and 8c4325b.

📒 Files selected for processing (2)
  • docs/superpowers/plans/2026-06-04-meet-audio-keepalive.md
  • docs/superpowers/specs/2026-06-04-meet-audio-keepalive-design.md
✅ Files skipped from review due to trivial changes (2)
  • docs/superpowers/plans/2026-06-04-meet-audio-keepalive.md
  • docs/superpowers/specs/2026-06-04-meet-audio-keepalive-design.md

📝 Walkthrough

Walkthrough

This PR implements an audio keepalive mechanism in the Meet bot's Web Audio bridge to prevent Google Meet from dropping the bot during periods of silence. The change adds a looping zero-sample audio source to the destination stream, backed by a Rust verification test and comprehensive design and execution documentation.

Changes

Meet Audio Keepalive Feature

Layer / File(s) Summary
Design & Specification
docs/superpowers/specs/2026-06-04-meet-audio-keepalive-design.md
Documents the observed failure (audio dropping after ~5s with Meet "Trying to reconnect"), investigation findings (missing audio keepalive), proposed architecture (looping zero-PCM source), signal chain behavior, edge cases (barge-in, suspension, sample-rate mismatches), risk/rollback strategy, testing approach (static regex presence test), and acceptance criteria.
JavaScript Keepalive Implementation
app/src-tauri/src/meet_audio/audio_bridge.js
Non-fatal keepalive setup in ensureContext() creates a ~100ms looping zero-sample buffer, connects it to the bridge destination, starts at time 0 with console logging, and catches setup failures as warnings without breaking speech synthesis.
Rust Verification Test
app/src-tauri/src/meet_audio/inject.rs
Unit test module verifying the bundled AUDIO_BRIDGE_JS contains the keepalive console-log marker and silenceSource.start(0) invocation, enforcing the keepalive code is present at bundle time.
Execution Plan & Checklist
docs/superpowers/plans/2026-06-04-meet-audio-keepalive.md
Step-by-step TDD-style execution plan (red/green tasks with formatting and commit commands), shipping verification gates, self-review checklist validating spec coverage and marker consistency, and execution handoff guidance.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Poem

🐰 A keepalive loop, silent and true,
Prevents the bot from a disconnection's rue,
With zero-samples that endlessly play,
The bridge stays alive, come what may!
No more Meet drops when the silence grows deep—
Just an echo of nothing to keep the bot's keep. 🎵

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding a zero-PCM keepalive mechanism to prevent Meet from dropping the bot during silence, which directly addresses the core issue (#2945).
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.


Comment @coderabbitai help to get the list of available commands and usage tips.

@CodeGhost21 CodeGhost21 marked this pull request as draft June 3, 2026 21:50

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
docs/superpowers/plans/2026-06-04-meet-audio-keepalive.md (1)

15-21: ⚡ Quick win

Add an explicit “run Codex PR checklist” step before PR creation.

The plan already lists gates, but it should directly require docs/agent-workflows/codex-pr-checklist.md to avoid missing merge gates in agentic runs.

Based on learnings: “Before opening AI-authored PRs from Codex web sessions or Linear-launched implementation agents, follow docs/agent-workflows/codex-pr-checklist.md.”

Also applies to: 326-401

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/superpowers/plans/2026-06-04-meet-audio-keepalive.md` around lines 15 -
21, The plan file (docs/superpowers/plans/2026-06-04-meet-audio-keepalive.md)
needs an explicit pre-PR step to run the Codex PR checklist; add a new bullet
under the pre-flight or PR creation section that instructs executors to run
docs/agent-workflows/codex-pr-checklist.md before opening the PR (e.g., "Run
Codex PR checklist: docs/agent-workflows/codex-pr-checklist.md") so agentic runs
cannot skip merge gates; apply the same insertion pattern to other affected
plans (326-401) mentioned in the comment.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/superpowers/plans/2026-06-04-meet-audio-keepalive.md`:
- Around line 265-269: The fenced commit-list block shown in the diff (the
triple-backtick block containing the three commit lines starting with
"fix(meet-audio):..." and ending with "docs(meet):...") lacks a language tag and
triggers markdownlint MD040; update that fenced code block to use a language tag
such as text (or bash) by changing the opening ``` to ```text so the block is
properly labeled.

In `@docs/superpowers/specs/2026-06-04-meet-audio-keepalive-design.md`:
- Around line 40-52: The fenced architecture block containing the ASCII diagram
starting with "Brain PCM (when speaking)" and ending with "Meet WebRTC" lacks a
language identifier and triggers markdownlint MD040; update the opening fence to
include a language tag (e.g., add "text" after the triple backticks) so the
block becomes ```text and the diagram (the lines referencing
MediaStreamDestinationNode, freshAudioStream().clone(),
getUserMedia({audio:true})) is treated as a text code block.

---

Nitpick comments:
In `@docs/superpowers/plans/2026-06-04-meet-audio-keepalive.md`:
- Around line 15-21: The plan file
(docs/superpowers/plans/2026-06-04-meet-audio-keepalive.md) needs an explicit
pre-PR step to run the Codex PR checklist; add a new bullet under the pre-flight
or PR creation section that instructs executors to run
docs/agent-workflows/codex-pr-checklist.md before opening the PR (e.g., "Run
Codex PR checklist: docs/agent-workflows/codex-pr-checklist.md") so agentic runs
cannot skip merge gates; apply the same insertion pattern to other affected
plans (326-401) mentioned in the comment.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: c771e410-50d0-4459-97ca-ed11402de018

📥 Commits

Reviewing files that changed from the base of the PR and between 6c4e300 and 2105f1d.

📒 Files selected for processing (4)
  • app/src-tauri/src/meet_audio/audio_bridge.js
  • app/src-tauri/src/meet_audio/inject.rs
  • docs/superpowers/plans/2026-06-04-meet-audio-keepalive.md
  • docs/superpowers/specs/2026-06-04-meet-audio-keepalive-design.md

Comment thread docs/superpowers/plans/2026-06-04-meet-audio-keepalive.md Outdated
Comment thread docs/superpowers/specs/2026-06-04-meet-audio-keepalive-design.md Outdated
@CodeGhost21 CodeGhost21 marked this pull request as ready for review June 4, 2026 08:46
@CodeGhost21

Copy link
Copy Markdown
Contributor Author

Manual verification — PASS ✅

Ran pnpm dev:app on this branch (commits bc6eab26c..2105f1dc7), joined a real Google Meet from a normal browser, sent OpenHuman to the call, admitted the bot from the waiting room, and observed.

Result: the bot stayed in the meeting for 3–4+ minutes without dropping.

This directly contradicts the bug's reported symptom from #2945 ("OpenHuman joins the meeting briefly, drops after around 5 seconds, shows a rejoining state, and then never actually rejoins the call"). The 3–4 min observation window is well past the 5-second drop point.

The hypothesis from the design spec held: pure zero-PCM into the MediaStreamDestinationNode is sufficient keepalive for Meet's WebRTC engine. DTX either isn't applied or doesn't trigger Meet's drop heuristic. The comfort-noise fallback noted in the spec is not needed.

What this verification did and did not cover

Covered: the core "agent stays joined" symptom from #2945.

🔍 Not exercised in this session (listed in the PR body for reviewer follow-up — none affect the PASS verdict on the core symptom):

  • chrome://webrtc-internals/ → bytesSent monotonic during silence — would confirm packets are on the wire (vs. DTX'd) but the empirical "doesn't drop" outcome implies whatever's happening is sufficient.
  • Brain reply (wake word + question) — would confirm live PCM still plays through the mixed graph. Zero + real = real, so this should hold trivially.
  • Barge-in mid-reply — would confirm __openhumanFlushAudio still cuts in-flight brain audio (keepalive is not in activeSources, so it shouldn't be touched).

I'll happily run the bonus checks if a reviewer wants them captured before merge — just say.

@CodeGhost21

Copy link
Copy Markdown
Contributor Author
Screenshot 2026-06-04 at 2 13 59 PM

@CodeGhost21

Copy link
Copy Markdown
Contributor Author

@senamakel — would appreciate a maintainer pass on this when you have a minute. Branch-protection requires it before the squash-merge can land, and this is the actual fix for the 5-second-drop symptom in #2945.

Quick summary:

  • What it does: 20-line zero-PCM AudioBufferSourceNode in audio_bridge.js so Meet's WebRTC engine keeps the bot in the call during silence. Mirrors the existing camera-bridge 1-pixel-bob keepalive pattern (meet_video/camera_bridge.js:179–181 "keeps the WebRTC encoder from dropping the stream as 'frozen'").
  • Manual verification (recorded above as a comment): bot stayed in a real Meet for 3–4+ minutes without dropping — directly contradicts the bug report's "drops after ~5 seconds" symptom.
  • CI: 12/12 checks green.
  • CodeRabbit: initial review CHANGES_REQUESTED → fixes pushed → re-review APPROVED.
  • Scope: zero Rust runtime changes; one JS file + one Rust presence test. Rollback = revert this one PR.

Companion PR #3321 (diagnostics + error toast) is independent and not blocking this one.

@senamakel-droid

Copy link
Copy Markdown
Contributor

we're moving this to the server side. closing this for now.

@CodeGhost21

Copy link
Copy Markdown
Contributor Author

okay

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants