feat(mascot): load Human mascots from GitHub manifest#4312
Merged
senamakel merged 21 commits intoJun 30, 2026
Conversation
Fetch the published tinyhumansai/mascots dist/mascots.json directly over HTTPS with in-memory + localStorage caching and a sha256-keyed .riv binary cache (reusing rivCache). Adds typed manifest model, validation that drops malformed/runtime-less entries, and default/find selectors.
Constrain face->pose, viseme normalisation, idle flourish picks and channel defaults to a specific mascot's stateEngine vocabulary, degrading cleanly to idle/thinking/sil when an asset lacks a given value.
Loads a manifest mascot's .riv buffer (sha-cached), drives pose/viseme/colors constrained to its stateEngine, runs the per-mascot idle pose cycle, and auto-cycles channels (e.g. Toshi's eyes) when alive. Falls back to the bundled default while loading/on failure so the stage never blanks.
Refactor HumanPage to utilize the useMascotManifest hook, allowing for dynamic resolution of the active mascot based on the GitHub manifest. This change replaces the previous selectedMascotId logic with a fallback to ManifestRiveMascot, ensuring a seamless user experience while loading or in case of errors. Update tests to mock the mascot manifest appropriately.
useMascotManifest loads the manifest and resolves the active mascot from the selectedMascotId preference (falling back to the default ready mascot). HumanPage now renders ManifestRiveMascot instead of the backend-sourced BackendRiveMascot.
MascotPanel now lists mascots from the published manifest (via useMascotManifest) with per-mascot pose/viseme counts and a draft badge, and previews the active mascot with ManifestRiveMascot. Drops the backend /mascots library calls. Adds the settings.mascot.characterDraft i18n key across all locales. Voice + color sections are unchanged.
The cloud TTS backend ships per-frame timing as seconds (startSeconds/ endSeconds), but parse_cue/parse_alignment only read *_ms — so timing deserialized as missing and every frame collapsed to start=0/end=80, freezing the mascot mouth on the first viseme. Add a read_ms helper that falls back to the seconds keys (×1000) and wire it into both parsers. Adds a unit test for the seconds shape including a pause gap.
- Suppress the text-delta pseudo-lipsync while "Speak replies" is on so the mouth only moves with the synthesized audio (it was flapping ahead of, and faster than, the voice during streaming). - Replace the brittle rescale with normalizeVisemeTimeline: keep a real per-frame timeline (preserving gaps as pauses), and only even-distribute the viseme sequence across the measured audio when timestamps are degenerate. - Prefer the char alignment (which carries real timing incl. pauses) when the viseme starts are unusable (hasUsableStarts). - Resolve audio duration without an extra microtask when already known.
- Match resolved pose/viseme codes against the asset's actual enum option list
case-insensitively (Rive enums are case-sensitive and silently ignore an
unknown value, which froze the mouth/pose).
- Gate the enum-driving effects on a stable content key instead of the
per-render `values` array identity, which caused an infinite render loop
("Maximum update depth exceeded").
The mascot redux-persist whitelist omitted selectedMascotId, so the chosen GitHub-manifest mascot reset to the default on every reload. Add it to the whitelist (the slice's REHYDRATE guard re-validates it on restore).
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds manifest-based mascot selection and rendering for settings and human pages, persists the selected mascot id, updates draft translations and privacy metadata, and expands TTS viseme timing normalization and seconds-based parsing. It also adds meeting-bot mascot-id fallback routing. ChangesMascot Manifest System
TTS Lipsync and timing
Meeting Bot Mascot Routing
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
coderabbitai
Bot
added
agent
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 08c326fd34
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 7
🧹 Nitpick comments (1)
app/src/features/human/Mascot/manifest/manifestService.ts (1)
27-28: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftMove the manifest snapshot behind a store slice instead of raw
localStorage.This adds a second persistence path beside
mascotSlice, so manifest data andselectedMascotIdnow follow different invalidation/versioning rules. As per coding guidelines, "Prefer Redux Toolkit state slices over ad-hoclocalStorage; frontend state should live in the establishedstore/slices."Also applies to: 97-114
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/src/features/human/Mascot/manifest/manifestService.ts` around lines 27 - 28, The manifest snapshot is still being persisted directly through `SNAPSHOT_KEY` in `manifestService`, which bypasses the Redux store and creates a separate persistence/versioning path from `mascotSlice`. Move this snapshot state into an existing or new store slice under `store/` and update the relevant manifest load/save flow in `manifestService` to read and write through Redux Toolkit state instead of raw `localStorage`, keeping `selectedMascotId` and manifest data on the same persistence path.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/src/features/human/Mascot/manifest/manifestService.ts`:
- Around line 48-69: `isManifestEntry()` is too permissive for nested
`stateEngine` data, so remote entries with invalid `visemeCodes`,
`idlePoseCycle`, or missing/ill-typed `channels` can still be cast to
`MascotManifestEntry`. Tighten the validation in `manifestService.ts` by
checking each `stateEngine` collection element and the `channels` structure for
the expected string keys/values before returning true, so only fully valid
manifest entries pass `parseManifest()`.
- Around line 122-145: The stalled manifest request in fetchMascotManifest
prevents the snapshot fallback from ever running because the retry/offline logic
only happens inside the catch path. Add an abort/timeout around the fetch call
for MASCOT_MANIFEST_URL so a hung request rejects and can reach readSnapshot(),
while keeping the existing inflight, parseManifest, and writeSnapshot flow
unchanged.
In `@app/src/features/human/Mascot/manifest/useMascotManifest.ts`:
- Around line 28-31: The useMascotManifest effect contains a redundant
synchronous state update because loading already defaults to true, so remove the
setLoading(true) call from the useEffect while keeping the rest of the async
cancellation logic intact. Update the effect in useMascotManifest so it only
handles the async manifest fetch and cancellation, and rely on the existing
initial loading state instead of setting it again inside the effect.
In `@app/src/features/human/Mascot/ManifestRiveMascot.tsx`:
- Around line 256-281: The loader state in ManifestRiveMascot is being preserved
across entry changes because the component relies on callers remounting it with
a key, which can leave stale buffer/failed state attached to a new entry. Reset
the internal state inside ManifestRiveMascot when entry changes by clearing
buffer and failed before or as part of the load effect, so a new entry always
starts fresh regardless of whether the parent remounts it. Use the existing
ManifestRiveMascot component, its useEffect, and the buffer/failed state
variables as the place to make the reset boundary self-contained.
- Around line 166-169: The idle cleanup in ManifestRiveMascot’s effect is
writing restPose directly, which bypasses the case-insensitive enum
normalization used elsewhere. Update the cleanup path in the effect that returns
the timer teardown and setPoseRef.current to pass the idle/rest pose through
matchEnumValue so assets exposed as Idle or IDLE still resolve correctly when
clearing a flourish state.
In `@app/src/features/human/useHumanMascot.ts`:
- Around line 315-322: The lipsync fallback is not actually using a wall-clock
source yet; `useHumanMascot` still depends on `playback.currentMs()`, which
ultimately reads `audio.currentTime`, so the CEF blob-audio freeze can still
happen. Add a real playback-start timestamp/ref in `useHumanMascot` and derive
elapsed playback time from `performance.now()` (or equivalent monotonic clock)
when advancing viseme selection, rather than relying only on
`PlaybackHandle.currentMs()` or the existing log-throttle ref. Keep the change
localized to the playback timing path in `useHumanMascot` and, if needed, the
`PlaybackHandle` contract in `audioPlayer.ts` so the lipsync loop reads the new
wall-clock-based time source.
In `@src/openhuman/voice/reply_speech.rs`:
- Around line 307-315: The duration fallback in reply_speech::read_ms is missing
the short seconds aliases, so it can ignore valid inputs like startSec or
durationSec and fall back to the default. Update the fallback branch in the end
calculation so it uses the same alias set as the initial start parsing, or
remove the short alias support from the start parser for consistency. Keep the
alias handling aligned across the read_ms calls used for start, time, and
duration.
---
Nitpick comments:
In `@app/src/features/human/Mascot/manifest/manifestService.ts`:
- Around line 27-28: The manifest snapshot is still being persisted directly
through `SNAPSHOT_KEY` in `manifestService`, which bypasses the Redux store and
creates a separate persistence/versioning path from `mascotSlice`. Move this
snapshot state into an existing or new store slice under `store/` and update the
relevant manifest load/save flow in `manifestService` to read and write through
Redux Toolkit state instead of raw `localStorage`, keeping `selectedMascotId`
and manifest data on the same persistence path.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: c2102bc9-ddac-4fb7-b890-b0d155818fa2
📒 Files selected for processing (38)
app/src/components/settings/panels/MascotPanel.tsxapp/src/components/settings/panels/__tests__/MascotPanel.test.tsxapp/src/features/human/HumanPage.test.tsxapp/src/features/human/HumanPage.tsxapp/src/features/human/Mascot/ManifestRiveMascot.test.tsxapp/src/features/human/Mascot/ManifestRiveMascot.tsxapp/src/features/human/Mascot/index.tsapp/src/features/human/Mascot/manifest/manifestService.test.tsapp/src/features/human/Mascot/manifest/manifestService.tsapp/src/features/human/Mascot/manifest/stateEngine.test.tsapp/src/features/human/Mascot/manifest/stateEngine.tsapp/src/features/human/Mascot/manifest/types.tsapp/src/features/human/Mascot/manifest/useMascotManifest.test.tsxapp/src/features/human/Mascot/manifest/useMascotManifest.tsapp/src/features/human/useHumanMascot.lipsync.test.tsapp/src/features/human/useHumanMascot.test.tsapp/src/features/human/useHumanMascot.tsapp/src/features/human/voice/ttsClient.test.tsapp/src/features/human/voice/ttsClient.tsapp/src/lib/i18n/ar.tsapp/src/lib/i18n/bn.tsapp/src/lib/i18n/de.tsapp/src/lib/i18n/en.tsapp/src/lib/i18n/es.tsapp/src/lib/i18n/fr.tsapp/src/lib/i18n/hi.tsapp/src/lib/i18n/id.tsapp/src/lib/i18n/it.tsapp/src/lib/i18n/ko.tsapp/src/lib/i18n/pl.tsapp/src/lib/i18n/pt.tsapp/src/lib/i18n/ru.tsapp/src/lib/i18n/zh-CN.tsapp/src/store/index.tsapp/src/store/mascotSlice.tsapp/src/test/setup.tsapp/src/utils/config.tssrc/openhuman/voice/reply_speech.rs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f839a39139
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c684c24f02
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 47ab49b4a4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
coderabbitai
Bot
removed
the
agent
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 79477917ef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/src/features/human/Mascot/manifest/useMascotManifest.ts`:
- Around line 53-56: The fallback mascot is not being persisted because
useMascotManifest exits early when selectedMascotId is null, so the default
manifest mascot never gets written to Redux. Update the effect in
useMascotManifest to allow syncing fallbackEntry?.id into state when there is no
current selection, while still avoiding unnecessary writes when selectedEntry
already matches. Keep the logic centered around selectedMascotId,
fallbackEntry?.id, selectedEntry, and setSelectedMascotId so the
persisted-selection contract is maintained.
In `@app/src/features/human/useHumanMascot.lipsync.test.ts`:
- Around line 100-137: Update makePlaybackWithDeferredMetadata so the
FakePlayback handle’s stop() also settles ended, matching the playBase64Audio
contract used by useHumanMascot. When stopped, make ended reject or otherwise
resolve in the same way the real playback does so interruption and unmount paths
exercise the hook’s ended.catch(swallowAudioStop) cleanup instead of leaving the
promise pending. Keep the behavior aligned with finish() and the FakePlayback
shape in this test helper.
In `@src/openhuman/about_app/catalog_data.rs`:
- Around line 79-87: The disclosure for GITHUB_MASCOT_MANIFEST underreports
egress by naming only raw.githubusercontent.com, while the Persona Pack flow
also fetches runtime assets from the manifest’s files[].url. Update the
CapabilityPrivacy destinations used in catalog_data.rs for this manifest to
describe the broader GitHub/raw-plus-asset-host behavior, and keep the privacy
metadata aligned with the actual fetch path in the mascot picker / Human page
flow.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 285af5c6-6d4a-4705-8141-dfba634e165b
📒 Files selected for processing (16)
app/src/components/skills/MeetingBotsCard.tsxapp/src/components/skills/__tests__/MeetingBotsCard.test.tsxapp/src/features/human/Mascot/ManifestRiveMascot.test.tsxapp/src/features/human/Mascot/ManifestRiveMascot.tsxapp/src/features/human/Mascot/manifest/manifestService.test.tsapp/src/features/human/Mascot/manifest/manifestService.tsapp/src/features/human/Mascot/manifest/stateEngine.test.tsapp/src/features/human/Mascot/manifest/stateEngine.tsapp/src/features/human/Mascot/manifest/useMascotManifest.test.tsxapp/src/features/human/Mascot/manifest/useMascotManifest.tsapp/src/features/human/useHumanMascot.lipsync.test.tsapp/src/features/human/useHumanMascot.tsapp/test/playwright/specs/settings-feature-preferences.spec.tssrc/openhuman/about_app/catalog_data.rssrc/openhuman/about_app/catalog_tests.rssrc/openhuman/voice/reply_speech.rs
🚧 Files skipped from review as they are similar to previous changes (7)
- app/src/features/human/Mascot/ManifestRiveMascot.test.tsx
- app/src/features/human/Mascot/manifest/manifestService.ts
- app/src/features/human/Mascot/manifest/stateEngine.test.ts
- src/openhuman/voice/reply_speech.rs
- app/src/features/human/Mascot/manifest/stateEngine.ts
- app/src/features/human/Mascot/ManifestRiveMascot.tsx
- app/src/features/human/useHumanMascot.ts
coderabbitai
Bot
added
the
rust-core
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9765130594
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9efb1247f1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
coderabbitai
Bot
removed
the
rust-core
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ManifestRiveMascotso the Human stage can render manifest-provided Rive characters, poses, channels, idle cycles, and visemes.selectedMascotIdacross reloads.startSeconds/endSecondsalignment fields.Problem
Solution
selectedMascotIdin the mascot slice and update Settings/Human page tests around the manifest-backed selection flow.Submission Checklist
diff-cover) meet the gate enforced by.github/workflows/pr-ci.yml. Runpnpm test:coverageandpnpm test:rustlocally; PRs below 80% on changed lines will not merge.raw.githubusercontent.com) plus manifest-declared mascot asset hosts; tests mock the request per Testing Strategy.Impact
Related
AI Authored PR Metadata (required for Codex/Linear PRs)
Linear Issue
Commit & Branch
Validation Run
pnpm --filter openhuman-app format:check(via pre-pushpnpm format:check)pnpm typecheck(via pre-push)pnpm debug unit src/features/human/Mascot/manifest/manifestService.test.ts src/features/human/Mascot/manifest/stateEngine.test.ts src/features/human/Mascot/manifest/useMascotManifest.test.tsx src/features/human/Mascot/ManifestRiveMascot.test.tsx src/features/human/HumanPage.test.tsx src/features/human/useHumanMascot.test.ts src/features/human/useHumanMascot.lipsync.test.ts src/features/human/voice/ttsClient.test.ts src/components/settings/panels/__tests__/MascotPanel.test.tsx(9 files, 178 tests) andcargo test --manifest-path Cargo.toml voice::reply_speech --lib(8 tests)cargo fmt --manifest-path ../Cargo.toml --all --checkvia pre-pushcargo check --manifest-path app/src-tauri/Cargo.tomlvia pre-pushpnpm --filter openhuman-app test:e2e:web:buildbash app/scripts/e2e-web-session.sh test/playwright/specs/settings-feature-preferences.spec.ts --grep "persists manifest mascot selection"pnpm debug unit src/features/human/Mascot/manifest/stateEngine.test.ts src/features/human/useHumanMascot.lipsync.test.ts(2 files, 21 tests) andpnpm --filter openhuman-app exec eslint src/features/human/Mascot/ManifestRiveMascot.tsxpnpm debug unit src/features/human/Mascot/manifest/stateEngine.test.ts src/components/skills/__tests__/MeetingBotsCard.test.tsx(2 files, 48 tests)pnpm debug unit src/features/human/useHumanMascot.lipsync.test.ts(1 file, 9 tests),cargo test --manifest-path Cargo.toml persona_pack_reports_github_mascot_manifest_destination --lib, andcargo test --manifest-path Cargo.toml github_repo_memory_source_reports_github_destination --libpnpm debug unit src/features/human/Mascot/manifest/stateEngine.test.ts src/features/human/Mascot/manifest/useMascotManifest.test.tsx src/features/human/useHumanMascot.lipsync.test.ts(3 files, 27 tests) andcargo test --manifest-path Cargo.toml persona_pack_reports_github_mascot_manifest_destination --libpnpm debug unit src/features/human/Mascot/manifest/useMascotManifest.test.tsx src/features/human/Mascot/manifest/stateEngine.test.ts src/features/human/useHumanMascot.lipsync.test.ts(3 files, 28 tests)pnpm debug unit src/features/human/Mascot/ManifestRiveMascot.test.tsx src/features/human/Mascot/manifest/useMascotManifest.test.tsx src/features/human/Mascot/manifest/stateEngine.test.ts src/features/human/useHumanMascot.lipsync.test.ts(4 files, 33 tests)pnpm debug unit src/features/human/Mascot/manifest/useMascotManifest.test.tsx src/components/settings/panels/__tests__/MascotPanel.test.tsx src/features/human/Mascot/ManifestRiveMascot.test.tsx src/features/human/Mascot/manifest/stateEngine.test.ts src/features/human/useHumanMascot.lipsync.test.ts(5 files, 57 tests)pnpm --filter openhuman-app format:check,pnpm lint,pnpm typecheck,pnpm rust:check, andpnpm --filter openhuman-app lint:commands-tokensviagit push origin feat/github-mascot-manifestValidation Blocked
command:N/Aerror:N/Aimpact:N/ABehavior Changes
Parity Contract
Duplicate / Superseded PR Handling
Summary by CodeRabbit