feat(ios): iOS client with QR pairing, E2E-encrypted tunnel, and push-to-talk #1420
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,64 @@ | ||
| --- | ||
| name: Android Compile Sanity | ||
|
|
||
| on: | ||
| pull_request: | ||
| paths: | ||
| - 'app/src-tauri-mobile/**' | ||
| - 'packages/tauri-plugin-ptt/**' | ||
| - 'src/openhuman/devices/**' | ||
| - 'app/src/services/transport/**' | ||
| - 'app/src/lib/tunnel/**' | ||
| - 'app/src/pages/ios/**' | ||
| - '.github/workflows/android-compile.yml' | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
| pull-requests: read | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.head_ref || github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| android-compile: | ||
| name: Android Compile Check | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 1 | ||
| # Mobile crate uses stock Tauri (no CEF) — no submodules needed. | ||
| submodules: false | ||
|
|
||
| - name: Set up Rust | ||
| uses: dtolnay/rust-toolchain@stable | ||
| with: | ||
| toolchain: '1.93.0' | ||
| targets: aarch64-linux-android | ||
|
|
||
| - name: Cache Rust build artifacts | ||
| uses: Swatinem/rust-cache@v2 | ||
| with: | ||
| workspaces: | | ||
| app/src-tauri-mobile -> target | ||
| packages/tauri-plugin-ptt -> target | ||
| cache-on-failure: true | ||
|
|
||
| - name: Set up pnpm | ||
| uses: pnpm/action-setup@v4 | ||
|
|
||
| - name: Set up Node | ||
| uses: actions/setup-node@v4 | ||
|
Comment on lines
+29
to
+54
Contributor
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
for f in .github/workflows/android-compile.yml .github/workflows/ios-compile.yml; do
echo "== $f =="
echo "-- mutable action refs --"
rg -n '^\s*uses:\s*[^@]+@(v[0-9]+|stable|main|master)$' "$f" || true
echo "-- checkout credential persistence --"
if rg -nPzo 'uses:\s*actions/checkout@[^\n]+\n(?:.*\n){0,8}?\s*persist-credentials:\s*false' "$f" >/dev/null; then
echo "checkout hardening present"
else
echo "missing persist-credentials: false"
fi
doneRepository: tinyhumansai/openhuman Length of output: 750 Harden GitHub Actions workflows: pin Both 🧰 Tools🪛 zizmor (1.25.2)[warning] 29-33: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false (artipacked) [error] 30-30: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy) (unpinned-uses) [error] 36-36: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy) (unpinned-uses) [error] 42-42: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy) (unpinned-uses) [error] 50-50: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy) (unpinned-uses) [error] 53-53: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy) (unpinned-uses) 🤖 Prompt for AI Agents |
||
| with: | ||
| node-version: '24' | ||
| cache: 'pnpm' | ||
|
|
||
| - name: Install dependencies | ||
| run: pnpm install --frozen-lockfile | ||
|
|
||
| # Hard gate: mobile Tauri host compiles for Android. | ||
| - name: cargo check -- mobile host (aarch64-linux-android) | ||
| run: cargo check --manifest-path app/src-tauri-mobile/Cargo.toml --target aarch64-linux-android | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,93 @@ | ||
| --- | ||
| name: iOS Compile Sanity | ||
|
|
||
| on: | ||
| pull_request: | ||
| paths: | ||
| - 'app/src-tauri-mobile/**' | ||
| - 'packages/tauri-plugin-ptt/**' | ||
| - 'src/openhuman/devices/**' | ||
| - 'app/src/services/transport/**' | ||
| - 'app/src/lib/tunnel/**' | ||
| - 'app/src/pages/ios/**' | ||
| - '.github/workflows/ios-compile.yml' | ||
| workflow_dispatch: | ||
|
|
||
| permissions: | ||
| contents: read | ||
| pull-requests: read | ||
|
|
||
| concurrency: | ||
| group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.head_ref || github.ref }} | ||
| cancel-in-progress: true | ||
|
|
||
| jobs: | ||
| ios-compile: | ||
| name: iOS Compile Check | ||
| runs-on: macos-latest | ||
| env: | ||
| # Pin the deployment target so swift-rs invokes the Swift compiler with | ||
| # `-target arm64-apple-ios16.0`. Matches Package.swift in | ||
| # packages/tauri-plugin-ptt/ios/, which uses iOS 14+ APIs (OSLog). | ||
| IPHONEOS_DEPLOYMENT_TARGET: '16.0' | ||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 1 | ||
| # The mobile crate uses stock Tauri (no CEF), so we don't need | ||
| # `submodules: recursive` — which would try to clone the | ||
| # `app/src-tauri/vendor/tauri-cef` submodule, a step that | ||
| # intermittently fails on macOS runners for fork PRs. | ||
| submodules: false | ||
|
|
||
| - name: Set up Rust | ||
| uses: dtolnay/rust-toolchain@stable | ||
| with: | ||
| toolchain: '1.93.0' | ||
| targets: aarch64-apple-ios | ||
|
|
||
| - name: Cache Rust build artifacts | ||
| uses: Swatinem/rust-cache@v2 | ||
| with: | ||
| workspaces: | | ||
| . -> target | ||
| app/src-tauri-mobile -> target | ||
| packages/tauri-plugin-ptt -> target | ||
| cache-on-failure: true | ||
|
|
||
| - name: Set up pnpm | ||
| uses: pnpm/action-setup@v4 | ||
|
|
||
| - name: Set up Node | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '24' | ||
| cache: 'pnpm' | ||
|
|
||
| - name: Install dependencies | ||
| run: pnpm install --frozen-lockfile | ||
|
|
||
| # Hard gate: mobile Tauri host compiles for iOS. No more soft-gate | ||
| # `continue-on-error` — the mobile crate uses stock Tauri without CEF | ||
| # so cef-dll-sys is not in the dependency graph. | ||
| - name: cargo check -- mobile host (aarch64-apple-ios) | ||
| run: cargo check --manifest-path app/src-tauri-mobile/Cargo.toml --target aarch64-apple-ios | ||
|
|
||
| # Hard gate: PTT plugin (host-target check; Swift sources are built | ||
| # lazily by swift-rs during the iOS-target check above). | ||
| - name: cargo check -- tauri-plugin-ptt | ||
| run: cargo check --manifest-path packages/tauri-plugin-ptt/Cargo.toml | ||
|
|
||
| # Hard gate: TypeScript compile. | ||
| - name: pnpm compile | ||
| run: pnpm --dir app compile | ||
|
|
||
| # Hard gate: iOS-relevant Vitest suites. | ||
| - name: pnpm test (iOS suites) | ||
| run: > | ||
| pnpm --dir app test -- | ||
| src/services/transport | ||
| src/lib/tunnel | ||
| src/pages/ios | ||
| src/components/settings/panels/devices |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,148 @@ | ||
| ## Summary | ||
|
|
||
| - Adds an iOS client for OpenHuman: device pairing via QR code, mascot chat screen, and push-to-talk voice input. | ||
| - No Rust core ships on device; the iOS app connects to the desktop core via LAN HTTP, an E2E-encrypted socket.io tunnel, or cloud HTTP fallback. | ||
| - All changes are cfg-gated or platform-guarded; the desktop build is unaffected. | ||
| - Adds the `tauri-plugin-ptt` Swift plugin (`packages/tauri-plugin-ptt/`) for AVAudioEngine + SFSpeechRecognizer on iOS. | ||
| - Adds CI sanity-check workflow, build scripts, capability catalog entries, and full docs. | ||
|
|
||
| ## Problem | ||
|
|
||
| Users with iOS devices had no way to interact with their OpenHuman assistant on the go. The desktop app required a local machine. This PR adds the client-side scaffolding and transport layer needed to bridge iOS to an existing desktop core. | ||
|
|
||
| ## Solution | ||
|
|
||
| The iOS app is a subset of the existing React/TypeScript UI, compiled by Tauri v2 into an iOS bundle. A `TransportManager` selects the best transport at runtime. Pairing is secured by an X25519 key agreement; all tunnel traffic uses XChaCha20-Poly1305 encryption. The backend is a blind socket.io forwarder -- it never sees plaintext. | ||
|
|
||
| ## Layer-by-layer commits | ||
|
|
||
| | Commit | Layer | Summary | | ||
| |--------|-------|---------| | ||
| | `a99537f3` | Layer 1 | Rust devices domain -- pairing store, RPC handlers, event bus, crypto (`src/openhuman/devices/`) | | ||
| | `4ea14b78` | Layer 2 | TS transport refactor -- `TransportManager`, `LanHttpTransport`, `TunnelTransport`, `CloudHttpTransport`, tunnel crypto (`app/src/services/transport/`, `app/src/lib/tunnel/`) | | ||
| | `ba651705` | Layer 3 | Desktop `/settings/devices` UI -- `DevicesPanel`, `PairPhoneModal` with QR generation and 2-second poll | | ||
| | `3e0e2a67` | Layer 4 | Tauri shell cfg-gating -- `#[cfg(target_os = "ios")]` guards on CEF-specific code | | ||
| | `621fec98` | Layer 5 | iOS app shell -- `PairScreen` (QR scan via `AVCaptureSession`), `MascotScreen` (chat UI) | | ||
| | `5ca6cf21` | Layer 6 | `tauri-plugin-ptt` -- Swift PTT plugin (AVAudioEngine, SFSpeechRecognizer, AVSpeechSynthesizer) | | ||
| | `41a6a895` | Layer 6 fix | PTT Swift fix -- latest transcript tracking + `@unchecked Sendable` on PTTSpeaker | | ||
| | _(this PR)_ | Layers 7+8 | Build scripts, CI, Info.plist, capability catalog, docs, quality pass | | ||
|
|
||
| ## Test coverage | ||
|
|
||
| - **Vitest:** 1957 passed, 3 skipped, 1 todo across 218 test files (includes transport, tunnel, devices, iOS, PTT suites). | ||
| - **Rust (about_app):** 20 passed -- validates catalog uniqueness, Mobile category, and new capability entries. | ||
| - **cargo check (all three Cargo.toml files):** clean (warnings only, pre-existing). | ||
|
|
||
| ## What is gated behind the iOS target | ||
|
|
||
| The following only activates on `cfg(target_os = "ios")` or when explicitly called from iOS screens: | ||
|
|
||
| - CEF exclusions in `app/src-tauri/` (accounts webviews, etc.) | ||
| - `tauri-plugin-ptt` commands (`start_listening`, `stop_listening`, `speak`, `cancel_speech`, `list_voices`) -- return `NotSupported` on non-iOS targets. | ||
| - `packages/tauri-plugin-ptt/ios/` Swift sources -- not compiled for desktop. | ||
|
|
||
| Desktop users see no change. | ||
|
|
||
| ## Known TODOs for follow-up PRs | ||
|
|
||
| - **Keychain migration:** iOS symmetric session key is in-memory only; persist to Keychain so the app reconnects after restart without re-pairing. | ||
| - **Event-driven pairing detection:** `PairPhoneModal` polls `devices_list` every 2 s. Switch to a socket event subscription when the SSE/socket bridge for `DomainEvent::DevicePaired` lands. | ||
| - **Full Xcode CI:** `cargo check --target aarch64-apple-ios` runs with `continue-on-error: true` in the new CI workflow because third-party C deps (cef-dll-sys) may fail without full Xcode on the runner. A follow-up should pin an Xcode-enabled runner and harden this to a hard gate. | ||
| - **APNs push notifications:** real-time delivery requires the app to be foregrounded. | ||
| - **Multi-region tunnel:** single backend instance only; no failover. | ||
| - **Info.plist automation:** developer must manually copy `Info.ios.plist` keys into the generated Xcode project after `tauri ios init`. Should automate via `bundle.iOS.template` once Tauri v2 stabilises the iOS template pipeline. | ||
|
|
||
| ## Backend dependency | ||
|
|
||
| **`tinyhumansai/backend#709` must be merged and deployed before end-to-end pairing works.** The `devices_create_pairing` RPC will return a tunnel registration error until the `tunnel:register` / `tunnel:connect` / `tunnel:frame` socket.io contract is live. | ||
|
|
||
| ## Manual test plan for iOS reviewer | ||
|
|
||
| _(Requires a physical iPhone or iOS 17+ simulator paired with the desktop app.)_ | ||
|
|
||
| From `packages/tauri-plugin-ptt/README.md`: | ||
|
|
||
| - [ ] Permissions dialog appears on first `startListening` call. | ||
| - [ ] Partial transcripts update while speaking; final transcript matches. | ||
| - [ ] Hold button to record, release to stop, chat message is sent with transcript. | ||
| - [ ] TTS plays through speaker by default when iPhone is held away from ear. | ||
| - [ ] BT headset routes audio correctly; disconnecting mid-recording stops gracefully. | ||
| - [ ] App backgrounded mid-record produces a final transcript and stops cleanly. | ||
| - [ ] Phone call interruption emits `ptt://error` with `code: interrupted`. | ||
| - [ ] `cancelSpeech` during TTS emits `tts-ended` with `finished: false`. | ||
| - [ ] `listVoices` returns non-empty list of `AVSpeechSynthesisVoice` entries. | ||
|
|
||
| Additional pairing flow checks: | ||
|
|
||
| - [ ] Desktop: Settings > Devices > "Pair iPhone" shows QR code. | ||
| - [ ] iOS app: PairScreen scans QR and transitions to MascotScreen after handshake. | ||
| - [ ] Desktop: Devices panel lists the paired device with correct label. | ||
| - [ ] Desktop: Revoke device removes it from the list; iOS app shows reconnect prompt. | ||
| - [ ] QR code expiry: code expires after TTL, "Generate new code" creates a fresh session. | ||
|
|
||
| ## Screenshots | ||
|
|
||
| > **PLACEHOLDER:** Before opening the PR, attach screenshots of: | ||
| > - Desktop `/settings/devices` panel with a paired device. | ||
| > - iOS mascot screen showing a conversation. | ||
| > | ||
| > These require a device with Xcode signing configured and `tinyhumansai/backend#709` deployed. | ||
|
|
||
| ## Submission Checklist | ||
|
|
||
| - [x] Tests added or updated (transport, tunnel, devices, iOS, PTT suites -- see coverage statement above). | ||
| - [x] Diff coverage note: new Rust code in `src/openhuman/devices/` was covered in Layer 1 tests; new TS code in `app/src/services/transport/` and `app/src/lib/tunnel/` covered by Vitest suites. PTT Swift layer cannot be unit-tested without iOS toolchain (noted in README). | ||
| - [x] Coverage matrix: N/A for this layer (build scripts, CI, docs, catalog). | ||
| - [x] No new external network dependencies (all transport calls use existing mock backend or real backend behind feature flag). | ||
| - [ ] Manual smoke checklist: iOS path not in `docs/RELEASE-MANUAL-SMOKE.md` yet -- tracked as follow-up. | ||
| - [ ] Linked issue: N/A (tracked via Linear). | ||
|
|
||
| ## Impact | ||
|
|
||
| - Desktop runtime: no change. | ||
| - iOS target: new experimental app bundle (not in release pipeline yet). | ||
| - `packages/tauri-plugin-ptt/` is a new crate workspace member; adds to build time only when targeting iOS. | ||
| - Capability catalog adds three new `mobile.*` entries and a new `Mobile` category. | ||
|
|
||
| ## Related | ||
|
|
||
| - Closes: N/A (new feature) | ||
| - Follow-up PR(s): Keychain migration, event-driven pairing, full Xcode CI, APNs. | ||
| - Backend: tinyhumansai/backend#709 | ||
|
|
||
| --- | ||
|
|
||
| ## AI Authored PR Metadata (required for Codex/Linear PRs) | ||
|
|
||
| ### Linear Issue | ||
| - Key: N/A | ||
| - URL: N/A | ||
|
|
||
| ### Commit & Branch | ||
| - Branch: `feat/ios-client` | ||
| - Commit SHA: _(set after final commit)_ | ||
|
|
||
| ### Validation Run | ||
| - [x] `pnpm --filter openhuman-app format:check` -- clean | ||
| - [x] `pnpm typecheck` -- clean | ||
| - [x] Focused tests: Vitest 1957 passed; cargo about_app 20 passed | ||
| - [x] Rust fmt/check: `cargo fmt --all` + `cargo check` on all three Cargo.toml -- clean | ||
| - [x] Tauri fmt/check: included above | ||
|
|
||
| ### Validation Blocked | ||
| - command: `cargo check --target aarch64-apple-ios` | ||
| - error: May fail on cef-dll-sys C deps without full Xcode; guarded with `continue-on-error: true` in CI. | ||
| - impact: Soft gate only; does not block merge. | ||
|
|
||
| ### Behavior Changes | ||
| - Intended behavior change: Desktop users see new Settings > Devices panel. iOS users can pair and chat. | ||
| - User-visible effect: Desktop gains device management UI. iOS app becomes available for sideloading/TestFlight. | ||
|
|
||
| ### Parity Contract | ||
| - Legacy behavior preserved: All existing desktop flows unaffected. No CEF injection added. No new JS injection in webview accounts. | ||
| - Guard/fallback/dispatch parity: PTT commands return `NotSupported` on non-iOS. Transport falls back gracefully. | ||
|
|
||
| ### Duplicate / Superseded PR Handling | ||
| - Duplicate PR(s): None | ||
| - Canonical PR: This PR | ||
| - Resolution: N/A |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,6 +3,7 @@ dist | |
| coverage | ||
| app | ||
| src-tauri | ||
| src-tauri-mobile | ||
| rust-core | ||
| skills | ||
| *.config.js | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Expand trigger paths to include files this job actually depends on.
This workflow can be skipped even when its own install/compile inputs change (for example,
pnpm-workspace.yaml, lockfiles, root/app manifests), which creates false negatives for the Android sanity gate.Suggested patch
on: pull_request: paths: - 'app/src-tauri-mobile/**' - 'packages/tauri-plugin-ptt/**' - 'src/openhuman/devices/**' - 'app/src/services/transport/**' - 'app/src/lib/tunnel/**' - 'app/src/pages/ios/**' + - 'pnpm-workspace.yaml' + - 'pnpm-lock.yaml' + - 'app/package.json' + - 'Cargo.toml' + - 'Cargo.lock' - '.github/workflows/android-compile.yml'📝 Committable suggestion
🤖 Prompt for AI Agents