Fix GA Realtime migration + port capability registry, gestures, ask_openclaw hard rules, shutdown - #1
Conversation
…ce identity, and MediaPipe numpy 2.x compat
- openai_realtime.py: migrate from the retired Realtime Beta API to the GA
API. OpenAI shut down the beta shape (beta_api_shape_disabled, close code
4000), which caused an endless reconnect loop. Uses client.realtime.connect,
the GA session.update shape (session.type, output_modalities, nested
session.audio.{input,output} with audio/pcm format objects, voice under
audio.output), and GA event names (response.output_audio.delta,
response.output_audio_transcript.delta/done).
- openclaw_bridge.py: update to current OpenClaw gateways.
* Protocol version 3 -> 4 (older gateways rejected with 'protocol mismatch')
* Identify as gateway-client/backend instead of impersonating
openclaw-control-ui (control-UI clients now require a secure context)
* Request valid scopes (operator.read, operator.write); the legacy 'chat'
scope no longer exists and broke pairing approval
* Implement Ed25519 device identity: keypair persisted at
~/.clawbody/device-identity.json, signs the connect.challenge nonce with
the v3 signature payload. Remote clients without a device identity get
their scopes stripped and chat.send fails with 'missing scope'.
Pairing is approved once on the gateway (openclaw devices approve).
- vision/mediapipe_tracker.py: make the BGR->RGB frame C-contiguous
(np.ascontiguousarray). With numpy 2.x mediapipe rejects negative-stride
views ('Reference mode is unavailable if data is not c_contiguous'),
which flooded logs and broke face tracking.
- .env.example: document that OPENCLAW_GATEWAY_URL must be the gateway
machine's LAN IP/mDNS name (not localhost) when running on the robot,
gateway.bind must be 'lan', note the one-time device pairing approval,
and replace the retired preview realtime model with a GA model.
Tested end-to-end on a Reachy Mini Wireless against OpenClaw 2026.7.1:
voice conversation, agent context fetch, ask_openclaw tool calls, and
MediaPipe face tracking all working.
Co-Authored-By: Oz <oz-agent@warp.dev>
…_openclaw hard rules, shutdown Selectively ports dAAAb's PR #2 (improve/ask-openclaw-prompt) onto our GA-Realtime codebase instead of merging it, since the PR was based on a stale main and would have reverted the GA Realtime migration, the reconnection loop, the OpenAI Vision camera fallback, and the bridge protocol-v4/device-identity work. Ported and adapted: - capabilities/ registry: runtime detection of dance/emotion libraries plus Reachy Mini daemon recorded-move datasets (localhost:8000), with macro fallbacks; new `capabilities` tool - emotion tool prefers daemon recorded expressions (full emotions library) with macro fallback; daemon HTTP calls run via asyncio.to_thread so the audio event loop never blocks - dance/emotion schemas accept any string (dynamic libraries) - body_sway tool (clamped body-yaw sway via goto_target) - ask_openclaw HARD RULES prompt section (never fake external actions, never claim "I can't do that"), adapted to English-default - turn-level + live keyword gestures, rewired to GA transcript event names with bilingual (English + Mandarin) cues; simplified to immediate rate-limited firing because the PR's audio-alignment scheduler (_schedule_gesture_at_char) was never committed and crashed the session on first use - shutdown tool + Gradio shutdown button; fixed the PR's missing `import os` and made exit graceful (SIGINT teardown, hard-exit fallback) - CLAWBODY_GESTURE_MODE env (natural/turn/off), documented in .env.example Deliberately not taken: the bridge Origin hardcoded to the PR author's personal server, the 120s timeout revert, the stripped ask_openclaw error handling, the 3-attempt reconnect revert, the README rewrite. Co-Authored-By: Ju Chun Ko <1607280+dAAAb@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request enhances the Reachy Mini OpenClaw integration by migrating to the GA OpenAI Realtime API, introducing speech-synchronized head gestures, implementing persistent device identity and authentication for the gateway, and adding new movement tools like body sway and runtime capability detection. Key feedback highlights several critical issues: the speech gesture handler does not track processed buffer length, leading to repeated gesture triggers; the body sway tool runs in an untracked background task that cannot be stopped and may cause hardware conflicts if triggered concurrently; dataset and move names are not URL-encoded when querying the daemon; and queuing multiple head look moves with the same initial start pose results in jerky, discontinuous motion.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
…encoding, continuous move chaining - Gestures: track _gesture_processed_len so a matched cue is consumed and can't re-trigger the same gesture after the cooldown expires (the stage-direction category had no once-per-response flag and would have re-fired while the cue stayed in the 48-char window) - body_sway: track the runner in a module-level task; a new sway or stop_moves cancels it (awaited) instead of letting concurrent runners fight over body_yaw, and a cancelled sway recenters to yaw 0 - Daemon URLs: percent-encode move names (LLM-supplied; spaces/non-ASCII made urllib reject the URL) and dataset names with slashes kept literal, since the daemon routes org/name as path segments and %2F would break route matching - Head-look sequences (gestures + emotion macros): chain each move's start pose to the previous move's target so back-to-back queued moves stay continuous instead of snapping back to the queue-time pose Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request upgrades the Reachy Mini OpenClaw integration to the GA OpenAI Realtime API, implements persistent Ed25519 device identity pairing, adds a runtime capability registry for dances and emotions, and introduces speech-synchronized head gestures. It also adds new tools (capabilities, body_sway, and shutdown) and a shutdown button in the Gradio UI. The review feedback identifies critical issues where synchronous, blocking SDK calls are executed directly on the asyncio event loop, which could freeze the application or cause audio stuttering. It is recommended to run these blocking calls in separate threads using asyncio.to_thread and to search the entire un-processed gesture buffer to avoid missing cues.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
… loop, widen gesture scan region - body_sway: robot.goto_target blocks its calling thread until the move completes (wait_for_task_completion -> threading.Event.wait in the SDK ws client), so every call now runs via asyncio.to_thread. The blocking also provides the pacing, so the extra sleeps are dropped (they were doubling the cycle time); recenter-on-cancel uses a short duration so stop_moves stays responsive - gestures: scan the entire unconsumed transcript region instead of a 48-char window, so cues that streamed in during a cooldown aren't lost; consumption via _gesture_processed_len keeps the region small Not changed: get_current_joint_positions/get_current_head_pose stay on the event loop — they are microsecond reads of state cached by the SDK's background websocket thread, not blocking calls. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request introduces several enhancements to the Reachy Mini OpenClaw integration, including upgrading to the GA OpenAI Realtime API, adding a persistent Ed25519 device identity for gateway pairing, and implementing speech-synchronized head gestures. It also introduces a runtime capability registry for dances and emotions, a body sway tool, and a graceful shutdown mechanism. The review feedback highlights two important improvements: refining the cue-matching regex to handle mixed CJK/Latin boundaries correctly, and adding type safety checks when parsing the gateway's connection challenge to prevent potential crashes.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
…andshake frame parsing - _find_cue: replace \b word boundaries with ASCII-alnum lookarounds; Python's \b counts CJK ideographs as word characters, so Latin cues never matched when adjacent to CJK text (e.g. "hi" in "你好hi"), defeating the bilingual cue design on mixed transcripts - openclaw_bridge: validate that the challenge frame is a dict before .get() (a non-dict JSON frame aborted the connect attempt with an AttributeError stack trace instead of a clear warning), and apply the same isinstance guards to the hello frame's payload/server/error fields Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request introduces several major updates, including upgrading to the GA OpenAI Realtime API, adding persistent Ed25519 device identity signing for gateway pairing, implementing speech-synchronized head gestures, and adding new tools like body_sway, capabilities, and shutdown. The feedback highlights a critical performance issue where synchronous, blocking robot SDK calls (get_current_joint_positions and get_current_head_pose) are executed directly on the main asyncio event loop in both openai_realtime.py and core_tools.py. Offloading these blocking calls to a worker thread using asyncio.to_thread is highly recommended to prevent audio stuttering, latency, or connection drops during real-time streaming.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
|
@gemini-code-assist review |
There was a problem hiding this comment.
Code Review
This pull request introduces several major updates, including upgrading to the GA OpenAI Realtime API, adding persistent device identity and pairing for the OpenClaw gateway, implementing speech-synchronized head gestures, and introducing new tools like body_sway, capabilities, and shutdown. Feedback focuses on critical performance and reliability improvements: blocking robot SDK calls are made directly on the main asyncio event loop and should be wrapped in asyncio.to_thread() to prevent audio stuttering, unreferenced background tasks (such as the shutdown task) need strong references to avoid premature garbage collection, and CJK cue matching should be made case-insensitive for mixed-language inputs.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
…, case-insensitive mixed cues - shutdown: hold the delayed-shutdown task in a module-level set with a done-callback discard; the event loop only keeps weak task refs, so the unreferenced task could be GC'd during its 4s grace sleep and the shutdown would silently never happen (body_sway was already safe via its tracked global) - _find_cue: match CJK-classified cues with re.search + IGNORECASE so a future mixed CJK/Latin cue isn't silently case-sensitive Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On the physical robot the mic picks up the robot's own speaker, so server VAD fired several times per second on echo/noise fragments (multilingual gibberish transcripts), each spawning a response that rehashed old context and chopped the real answer mid-sentence — perceived as the robot "caching" and repeating previous answers instead of prioritizing the new question. - Echo gate in receive(): while robot audio is playing (speaking flag or estimated playback window + 0.6s device tail), drop mic frames below CLAWBODY_BARGE_IN_RMS (default 0.06); louder speech still passes so the user can interrupt. 0 disables. - On genuine barge-in: send conversation.item.truncate with the audio position actually played so the server's history matches what the user heard, flush un-played audio, and clear stale queued gestures. - Stale tool guard: if the user starts a new turn while a tool call runs, keep the function_call_output in history but skip the forced response.create that would answer the old question over the new one. - Session config: far_field noise reduction, VAD threshold 0.5 -> 0.6, explicit create_response/interrupt_response. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When the user starts speaking, read DoA from the ReSpeaker (USB control read, run via asyncio.to_thread) and queue a proportional head turn toward the voice: yaw = 90° - angle, clamped to ±35°, with an 8° deadband so the robot doesn't twitch when already facing the speaker. HeadLookMove gains an optional target_yaw_deg for exact-yaw targets. Env knobs: CLAWBODY_DOA_MODE (on/off) and CLAWBODY_DOA_FLIP (invert if the hardware's left/right convention turns the robot away from you). Verified on hardware that the ReSpeaker answers DoA reads concurrently with the app's audio pipeline (device 38fb:1001, no USB contention). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On the robot, media/GStreamer teardown takes longer than 5s, so the voice shutdown was always hitting the hard-exit fallback instead of finishing app.stop() cleanly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ger-driven body_sway The MovementManager's 100Hz loop commanded set_target(body_yaw=0.0) on every tick (all moves evaluate to yaw 0), pinning the base and overriding any rotation — including body_sway's own goto_target, and any "turn around" request. Hardware probe confirmed the daemon accepts continuous body yaw through a full 360° with no clamp; the limitation was ours. - MovementManager: persistent base body yaw (current/target) slewed at 120°/s each tick and added to every command; thread-safe set_body_yaw(relative=...)/halt_body_yaw/get_body_yaw - New turn_body tool: relative rotation ±360°, clamped, prompt updated - body_sway rewired to oscillate the manager target instead of calling goto_target against the manager's own command stream - stop_moves halts any in-progress rotation at its current angle Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
OPENAI_TRANSCRIPTION_MODEL (default gpt-4o-transcribe) and OPENAI_VISION_MODEL (default gpt-4o-mini) join OPENAI_MODEL in Config so model swaps are a .env edit instead of a code change; documented them and the existing LOCAL_VISION_MODEL in .env.example. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
Summary
improve/ask-openclaw-prompt) onto this codebase instead of merging it, since that PR was based on a stale main and would have reverted the GA Realtime migration, the reconnection loop, the OpenAI Vision camera fallback, and the bridge protocol-v4/device-identity work (d7587e7):capabilities/registry: runtime detection of dance/emotion libraries plus Reachy Mini daemon recorded-move datasets, with macro fallbacks; newcapabilitiestoolemotion/dancetools accept any string and prefer daemon recorded moves, falling back to macros; daemon HTTP calls run off the audio event loop viaasyncio.to_threadbody_swaytool (clamped body-yaw sway)ask_openclawhard-rules prompt section (never fake external actions, never claim "I can't do that")shutdowntool + Gradio shutdown button, with the PR's missingimport osfixed and a graceful SIGINT-first exit (hard-exit fallback)CLAWBODY_GESTURE_MODEenv var (natural/turn/off)Deliberately not taken from PR #2: the bridge Origin hardcoded to the PR author's personal server, the 120s timeout revert, the stripped ask_openclaw error handling, the 3-attempt reconnect revert, and the README rewrite.
Test plan
python3 -m py_compile, plus real import in thereachy_mini_envvenv withopenai/fastrtcstubbed)capabilitiestool output against a running Reachy Mini daemon🤖 Generated with Claude Code