diff --git a/.gitignore b/.gitignore index ba30564..128382e 100644 --- a/.gitignore +++ b/.gitignore @@ -111,3 +111,10 @@ weight.bin # Local release credentials. This file can be bundled by local release builds, # but must never be committed. RunAnywhereAI/Resources/RunAnywhereLocalSecrets.plist + +# Claude / cross-agent tooling. `.claude/skills/` (if present) is the canonical, human-edited +# skill tree; `.agents/skills/` is a generated mirror for non-Claude tooling (e.g. Codex), kept +# local/ignored for now pending a decision on what is safe to publish from this public repo. +# See scripts/sync-skills.sh. +.claude/* +.agents/* diff --git a/AGENTS.md b/AGENTS.md index 0c8e580..b4c3a9e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,7 +1,8 @@ # AGENTS.md RunAnywhereAI for iOS and macOS One SwiftUI target, `RunAnywhereAI`, shipping to both the App Store and the Mac App Store, -plus a keyboard extension and a Live Activity widget. It consumes the RunAnywhere SDK from +plus a keyboard extension (`RunAnywhereKeyboard`) and a Live Activity widget +(`RunAnywhereActivityExtension`). It consumes the RunAnywhere SDK from `github.com/RunanywhereAI/runanywhere-swift`, the generated Swift-only SwiftPM distribution of the `runanywhere-sdks` monorepo. There is no monorepo checkout and no staged XCFramework: SwiftPM downloads checksum-verified binaries on resolve. @@ -25,7 +26,8 @@ path below is relative to this repository's root. `open RunAnywhereAI.xcodeproj` works too; SwiftPM resolves the SDK on open. `./scripts/verify.sh` resolves and runs a full simulator `xcodebuild`; `./scripts/smoke.sh` greps the sources for SDK call patterns and checks the Parakeet CTC catalog entry, without compiling. CI runs both plus -the unit tests. +the unit tests. There is one app target: these scripts build `RunAnywhereKeyboard` and +`RunAnywhereActivityExtension` along with the main app, not separately. Logs: `log stream --predicate 'subsystem CONTAINS "com.runanywhere"' --info --debug` on simulator and Mac, `idevicesyslog | grep "com.runanywhere"` on device. @@ -74,50 +76,18 @@ MVVM with Swift Observation. Views are SwiftUI with no business logic. View mode are `Codable` value types. Services are singletons for cross-feature concerns: `ConversationStore`, `KeychainService`, `DeviceInfoService`, `ModelCatalogBootstrap`. -### Navigation +Chat is the product; the SDK demos sit behind a secondary hub (`ConsumerAdvancedHubView`) +rather than top-level tabs. macOS uses `ConsumerMacShell` (`NavigationSplitView` over +`MacSidebar`, with Chat/Models/Advanced destinations); iOS uses `ConsumerCompactShell` +(`ChatInterfaceView` plus sheets). The full shell/sidebar/hub breakdown — including why +`.chat` and `.conversation(String)` are separate `MacSidebarSelection` cases and what each +Advanced-hub row is gated on — is in +[`docs/reference/ARCHITECTURE.md`](docs/reference/ARCHITECTURE.md). -Chat is the product; the SDK demos sit behind a secondary hub rather than top-level tabs. -`ContentView` branches on platform. - -| Platform | Shell | Structure | -|---|---|---| -| macOS | `ConsumerMacShell` | `NavigationSplitView` over `MacSidebar`. Three destinations (Chat, Models, Advanced) plus the conversation list, which is scoped to Chat. Detail is `ChatInterfaceView`, `SimplifiedModelsView`, or `ConsumerAdvancedHubView`. | -| iOS | `ConsumerCompactShell` | `ChatInterfaceView` alone, plus sheets. | - -`MacSidebarSelection` has four cases: `.chat` (the transcript, whatever is current), -`.conversation(String)`, `.models`, and `.advanced`. Splitting `.chat` from `.conversation` -is what lets ⌘1 land somewhere real before anything is saved. ⌘1/⌘2/⌘3 are published from -the shell through `focusedSceneValue(\.shellNavigationActions)` because the chat cannot -navigate away from itself. One `@SceneStorage` key, `mac.sidebar.visibility`, persists -whether the sidebar is showing; column width is fixed by `navigationSplitViewColumnWidth` -and the selection is re-derived from the current conversation on restore. - -On iOS, Settings and the Advanced hub are sheets, both opened from the conversation drawer -rather than the toolbar. Models is not the same kind of sheet: the chat presents -`ModelSelectionSheet` (a picker, cross-platform), while the full `SimplifiedModelsView` -management screen is reached through a `NavigationLink` inside `CombinedSettingsView`. On -macOS `SimplifiedModelsView` is the `.models` sidebar destination. - -`ConsumerAdvancedHubView` has five sections and eight rows: - -| Section | Rows | Availability | -|---|---|---| -| Connect | Host this Mac | macOS only (`#if os(macOS)`) | -| Voice Utilities | Transcribe, Read Aloud, Voice Activity | both | -| Voice Utilities | Diarization | iOS only (`#if canImport(UIKit)`) | -| Vision Utilities | Segmentation | iOS only, and so is the whole section | -| Agents | Talk, Computer Use | both | -| Management | Benchmarks | both | - -Storage and tool calling live in Settings and Manage Models instead. - -### Dependency injection - -Three layers, the first thinner than it looks. `RunAnywhereAIApp` injects exactly one -environment object, `FlowSessionManager`, and only on iOS; on macOS it injects nothing. -Everything else is reached as a singleton at the point of use (`ConversationStore.shared`, +Dependency injection is mostly singleton-at-point-of-use (`ConversationStore.shared`, `SettingsViewModel.shared`, `ToolSettingsViewModel.shared`, `ModelListViewModel.shared`, -`KeychainService.shared`), or through the static `RunAnywhere.*` SDK namespace. +`KeychainService.shared`) or the static `RunAnywhere.*` namespace. `RunAnywhereAIApp` injects +exactly one environment object, `FlowSessionManager`, and only on iOS; macOS injects nothing. ### Initialization gate @@ -140,11 +110,9 @@ gate a second call that arrives while the first is still awaiting. Both are `@St All four `register` calls run before `RunAnywhere.initialize` and with no `await` between them. That ordering is load-bearing: register later and a model load can race an empty -provider registry and fail with -422, "No provider could handle the request". - -The `priority:` argument is currently decorative. Every `register` declares it as -`priority _: Int = 100` and discards it; the real ordering comes from each plugin's base -priority in C++ commons. +provider registry and fail with -422, "No provider could handle the request". The +`priority:` argument is currently decorative — every call declares `priority _: Int = 100` +and discards it; the real ordering comes from each plugin's base priority in C++ commons. MLX executes only on a physical device or native macOS. On the arm64 simulator `MLX.register()` returns false and no MLX rows are seeded. @@ -163,267 +131,34 @@ bridging `UIColor` and `NSColor`. |---|---| | `RunAnywhereAI/App/` | `RunAnywhereAIApp` (entry, SDK init), `ContentView` (platform shells), `MacSidebar`, `ConsumerAdvancedHubView`, `AppCommands` (menu commands), `InitializationViews` | | `RunAnywhereAI/Core/DesignSystem/` | `AppColors`, `AppSpacing`, `AppType`, `Typography`, `Layout`, `Motion`, `Surface`, `Haptics`, `EmptyStateMark`, `AudioActivityBars`, `ViewCompatibility` | -| `RunAnywhereAI/Core/Models/` | `AppTypes` (`SystemDeviceInfo`, `Int64.formattedFileSize`), `MarkdownBlock` (block model plus `MarkdownBlockParser`) | | `RunAnywhereAI/Core/Services/` | `ConversationStore`, `DeviceInfoService`, `KeychainService`, `ModelCatalogBootstrap`, `HardwareTier`, `HuggingFaceHubClient` | -| `RunAnywhereAI/Features/Chat/` | 29 files across `Models/`, `ViewModels/`, `Views/` | -| `RunAnywhereAI/Features/Voice/` | STT, TTS, VAD, and the voice agent (15 files) | -| `RunAnywhereAI/Features/VoiceKeyboard/` | Dictation flow and Live Activity attributes (5 files) | -| `RunAnywhereAI/Features/Models/` | Model browser, download tracking, selection sheet, Hugging Face import (17 files) | -| `RunAnywhereAI/Features/Benchmarks/` | Scenario providers, runner, report formatting, share card (16 files) | -| `RunAnywhereAI/Features/Vision/` | `VLMViewModel`, `VLMCameraView`, `VLMCameraPreview` | -| `RunAnywhereAI/Features/Connect/` | Host management, client controller, status banner | -| `RunAnywhereAI/Features/ComputerUse/` | `ComputerUseAgentView` and its view model | -| `RunAnywhereAI/Features/Diarization/`, `Segmentation/` | One view plus one view model each | -| `RunAnywhereAI/Features/Settings/` | `CombinedSettingsView`, `SettingsViewModel`, `ToolSettingsView`, `CalendarTool`, `HealthKitTool` | -| `RunAnywhereAI/Features/RAG/Services/` | `DocumentService` only; text extraction for chat document attachments | -| `RunAnywhereAI/Features/Storage/` | `StorageViewModel` only; surfaced inside Settings and the models views | -| `RunAnywhereAI/Extensions/` | `ModelInfo+Logo`, `String+Markdown`, `RunAnywhere+ExampleShims` | -| `RunAnywhereAI/Helpers/` | `SmartMarkdownRenderer` (`AdaptiveMarkdownText`), `InlineMarkdownRenderer.swift` (declares `MarkdownText`, not a type of its own name), `AdaptiveLayout` | -| `RunAnywhereAI/Shared/` | `SharedConstants` (IPC keys, Darwin notification names, URL scheme), `SharedDataBridge` | +| `RunAnywhereAI/Features/*` | One folder per feature (Chat, Voice, VoiceKeyboard, Models, Benchmarks, Vision, Connect, ComputerUse, Diarization, Segmentation, Settings, RAG, Storage) — behavior, flow, and the SDK calls each one makes are in [`docs/reference/FEATURES.md`](docs/reference/FEATURES.md) | +| `RunAnywhereAI/Extensions/` | `ModelInfo+Logo`, `String+Markdown`, `RunAnywhere+ExampleShims` — app-local helpers only; a feature needing net-new C bridge code belongs in the SDK, not here | +| `RunAnywhereAI/Shared/` | `SharedConstants` (IPC keys, Darwin notification names, URL scheme) and `SharedDataBridge`, shared with both extension targets below | | `RunAnywhereKeyboard/` | `KeyboardViewController` (IPC via Darwin notifications), `KeyboardView`, `Info.plist` with `RequestsOpenAccess`, app-group entitlement | | `RunAnywhereActivityExtension/` | `WidgetBundle` entry and the Dynamic Island / Lock Screen Live Activity | -Diffusion and image generation are excluded from the v1 build; see the comment at the top -of `RunAnywhereAIApp.swift` and the matching one in `ModelCatalogBootstrap.swift`. Their -products, registration calls, feature folders, catalog rows, and `generateImage` APIs are -deliberately absent. - -## Features - -### Chat and LLM - -`LLMViewModel` is split across ten files by concern: core state and `sendMessage()` in -`LLMViewModel.swift`, then `+Generation` (streaming and non-streaming), `+ToolCalling`, -`+ModelManagement`, `+Analytics`, `+Events` (Combine subscription to `RunAnywhere.eventBus`), -`+Documents` (RAG-backed attachments), `+MessageActions`, `+Vision`, and the shared -`LLMViewModelTypes`. `ToolCallingModelPolicy` gates tools on context length alone -(`minimumContextTokens = 1024` against the model's `contextLength`), not on model identity; -its companion `ToolCallingExecutionPolicy` caps the run at two tool calls and 96 tokens of -final response, at temperature 0 with reasoning off. - -Flow: input, `sendMessage()`, `prepareMessagesForSending()` (creates the user message and an -empty assistant message), `executeGeneration()`, `performGeneration()`, then the streaming, -non-streaming, or tool-calling path, token-by-token message updates, `finalizeGeneration()`, -and persistence to `ConversationStore`. - -Tool calling runs through `RunAnywhere.llm.generate` with the registry active; the SDK owns -the call and execute loop and the format is auto-detected per model. - -LoRA lives almost entirely in the SDK. `ModelCatalogBootstrap.registerLoraAdapters()` seeds -the curated catalog as `RALoraAdapterCatalogEntry` values, mirroring Android's -`ModelBootstrap.seedLora`, and registers each one with `RunAnywhere.lora.registerArtifact`. -From there the app calls `RunAnywhere.lora.queryCatalog(_:)` (with an -`RALoraAdapterCatalogQuery`), `.download(_:artifact:)`, `.importAdapter(from:)`, -`.applyCatalogAdapter(_:localPath:scale:)`, `.apply(RALoraApplyRequest)` for a raw path, -`.remove(RALoraRemoveRequest)`, and `.state()`. Removal is id-keyed or `clearAll`; the old -path-keyed fallback was deleted. Scale is user-adjustable. - -Conversations persist as per-conversation JSON under `Documents/Conversations/`, attachments -under `Conversations/Attachments/{conversationID}/`. Search covers titles and message content. - -Titles are written by whichever model answered, through `RunAnywhere.llm`, not by a separate -`FoundationModels.LanguageModelSession`. Apple's Foundation Models will not serve two clients -against one on-device model: the old app-side title session hung and wedged every subsequent -turn behind it with no error and no timeout. Only one title task exists at a time and -`cancelPendingTitleGeneration()` hands the model back the moment the chat claims a new turn. - -Analytics: `MessageAnalytics` per message (time to first token, tokens per second, thinking -mode, completion status) and `ConversationAnalytics` rolled up onto the stored `Conversation`. -`ChatDetailsView` recomputes its figures from the per-message records rather than reading the -rolled-up type, and shows tokens per second, thinking usage, success rate, and average total -generation time. Per-message TTFT is rendered in `ChatMessageComponents`, not in -`ChatDetailsView`. - -Thinking mode: models with `supportsThinking` expose reasoning through the SDK's `reasoning` -options and `thinkingText` or stream thought events. Commons owns tag parsing and `/no_think` -directives; the app toggles the mode and renders the returned channel in a collapsible -section. - -Documents attached to a chat go through `ChatAttachmentLoader` and `DocumentService` for text -extraction, then `RunAnywhere.rag.open(embeddingModel:llmModel:)` for a `RagSession` cached on -the view model. The cache key is document plus embedding model plus answer model, so a session -is reused across turns rather than being strictly one per conversation. There is no separate -RAG screen. - -### Voice agent - -`RunAnywhere.voice.createSession(stt:llm:tts:)` then `session.start()`, which is the only -call that opens the microphone. `session.events` yields `agentStateChanged` (with -`.listening`, `.thinking`, `.speaking`), `speechStarted`, `speechEnded`, `userTranscribed`, -`agentResponse`, `inputSilent`, and `error`. `session.interrupt()` and `session.close()` are -the other two verbs. The SDK owns the whole audio pipeline including its own VAD. The user -loads STT, LLM, and TTS models independently through `ModelSelectionSheet`. - -`VoiceAssistantParticleView` is a Metal-rendered 2000-particle system: a Fibonacci-lattice -sphere that morphs to a ring while listening or speaking, with amplitude from the real -microphone level when listening and a simulated sine wave when speaking, and touch scatter -decaying at 0.92. - -### Speech, synthesis, and voice activity - -STT has three modes. Batch records audio then calls -`RunAnywhere.stt.transcribe(.pcm16(buffer, sampleRate: 16_000))`. Live yields microphone -chunks into `RunAnywhere.stt.transcribeStream(_:)`, which owns segmentation and emits -`.partial` and `.final`. Hybrid runs on-device first with cloud fallback through the SDK's -`HybridSTTRouter`. Capture is the SDK's `AudioCaptureManager` driven by the app-local -`AudioCapturePump`; no app-side silence detection exists. - -TTS is `RunAnywhere.tts.speak(text, options: TtsOptions(speed:))`, which synthesizes and plays -inside the SDK and hands back a handle. The app awaits `handle.waitForPlayout()` and interrupts -with `handle.interrupt()`; the whole-engine `RunAnywhere.tts.stop()` is deprecated and used only -as a fallback when no handle exists. Use `tts.synthesize(_:)` when the `Audio` buffer is wanted -instead of playback. - -VAD feeds microphone chunks to `RunAnywhere.vad.detectStream(_:)`, which emits `VadEvent` -values: `.speechStarted`, `.speechEnded`, per-chunk `.activity(isSpeech, _, _)`, `.failed`, and -`.completed`. Framing is the SDK's job. The activity log holds 50 entries. - -### Voice keyboard - -Cross-process dictation over two IPC channels: App Group `UserDefaults` -(`group.com.runanywhere.runanywhereai`) for shared state (session state, transcribed text, -audio level, heartbeat), and Darwin `CFNotificationCenter` for zero-latency signals (six names -in `SharedConstants.DarwinNotifications`). - -The keyboard's Run button opens `runanywhere://startFlow`. The main app activates a session, -loads the STT model, starts capture, and posts `sessionReady`. The user returns to the host -app, the keyboard sends `startListening`, the main app buffers audio, the keyboard sends -`stopListening`, the main app calls `RunAnywhere.stt.transcribe(_:)`, writes the result to -shared `UserDefaults`, and posts `transcriptionReady`; the keyboard inserts it through -`textDocumentProxy.insertText()`. - -`DictationActivityAttributes.ContentState` carries phase, elapsed seconds, transcript, and -word count for the Dynamic Island and Lock Screen. A one-second heartbeat lets the keyboard -detect a main-app crash after a three-second staleness window. - -### Vision - -Camera and photo-library image understanding, reached from the chat rather than its own tab. -`AVCaptureSession` with BGRA pixel format feeds -`RunAnywhere.vlm.generateStream(image: .pixelBuffer(frame), prompt:, options:)`. Live mode -captures every 2.5 seconds (`autoStreamInterval`) and clears on the first token, so an -unchanged scene does not repeat itself. Its token cap is `autoStreamMaxTokens = 64`; the -single-shot path has its own, larger `singleShotMaxTokens`. Pixel conversion belongs to the SDK: pass -`ImageInput` a `CVPixelBuffer` and do not bridge through `CIContext`. - -### Benchmarks - -Deterministic tests across LLM, STT, TTS, and VLM, each with a `BenchmarkScenarioProvider`. -`BenchmarkRunner` orchestrates with cooperative cancellation. Results persist as JSON, capped -at 50 runs. `BenchmarkExportFormat` offers Markdown and JSON; CSV exists only as a -`writeCSV(run:)` file writer with no picker entry. `SyntheticInputGenerator` produces silent -and sine-wave audio (440 Hz at 16 kHz) and solid and gradient 224x224 images. LLM scenarios -run at 50, 256, and 512 tokens measuring TTFT and decode speed. - -### Models - -`ModelListViewModel` is the canonical registry singleton, subscribed to -`RunAnywhere.eventBus.modelLifecycle` for live load and unload state. `ModelSelectionSheet` is -the universal picker, parameterized by `ModelSelectionContext`: `.llm`, `.stt`, `.tts`, `.vad`, -`.voice`, `.vlm`, `.ragEmbedding`, `.ragLLM`, `.diarization`, `.segmentation`. Custom models -arrive through `AddModelFromURLView` or `AddFromHuggingFaceView`. `ModelRecommendationEngine` -and `ModelCompatibilityLookup` drive the recommended set, and `HardwareTier` scopes it to the -device. - -### Storage - -`RunAnywhere.models.state()` gives used and free bytes; -`RunAnywhere.models.list(filter: ModelFilter(downloadedOnly: true))` gives the rows, filtered -to entries with a real on-disk size so Apple system pseudo-models drop out. Each row reads its -own `ModelInfo` for name, local path, framework, and `lastUsedAtUnixMs`. Deletion is -`RunAnywhere.models.delete(id:)`; cache and temp clearing are `RunAnywhere.clearCache()` and -`RunAnywhere.cleanTempFiles()`. `StorageViewModel` surfaces this inside Settings and the models -views. - -### Settings and tools - -`SettingsViewModel` (singleton) owns temperature, max tokens, and system prompt in -`UserDefaults`, and API key and base URL in the Keychain. Temperature, max tokens, and system -prompt each save on a Combine `debounce(0.5s)`; the thinking-mode toggle writes through -immediately. - -`ToolSettingsViewModel` is a separate singleton and registers tools through -`RunAnywhere.llm.tools`. Six are always available: `get_weather` (Open-Meteo), -`get_current_time`, `calculate` (a recursive-descent `SafeMathEvaluator`), `get_device_info`, -`get_battery_level`, and the SDK's own web search, added by `RunAnywhere.registerWebSearchTool()`. -Two more are opt-in behind a toggle and a permission prompt: `get_calendar_events` -(`CalendarTool`) and `get_health_data` (`HealthKitTool`, iOS only). `registerBuiltInTools()` -restores the enabled set at launch, because assigning a stored property inside `init` does not -fire `didSet`. - -## Markdown rendering - -One path, not a detect-and-route chain. `MarkdownBlockParser.parse(_:)` turns the reply into -`[MarkdownBlock]` (`paragraph`, `heading`, `list`, `quote`, `code`, `rule`) with no SwiftUI -involved, and `AdaptiveMarkdownText` renders one view per block: `MarkdownListView`, -`MarkdownQuoteView`, `MarkdownCodeBlock` (syntax-colored header, copy button, monospaced -scrollable body), and inline text through `MarkdownText` in `InlineMarkdownRenderer.swift`, -which uses `AttributedString(markdown:)` with bold as `.semibold`, italic as `.italic`, and -inline code monospaced and purple-tinted. - -## SDK surface used here - -Every call goes through the `RunAnywhere` enum. One namespace per modality; the SDK owns model -resolution, loading, downloading, and orchestration behind each verb. What follows is what this -app actually calls, not the SDK's full surface, which is larger. - -```swift -// Core -try RunAnywhere.initialize(apiKey:baseUrl:environment:) // one call, both phases -RunAnywhere.isReady -RunAnywhere.eventBus.events / .modelLifecycle // Combine, raw RASDKEvent protos -RunAnywhere.clearCache() / .cleanTempFiles() - -// Models -RunAnywhere.models.list(filter:) / .get(id:) / .register(_:) / .refresh() -RunAnywhere.models.download(id:) / .isResumable(...) / .checkCompatibility(...) -RunAnywhere.models.load(id:options:) / .unload(category:) / .delete(id:) / .state() - -// Generation -RunAnywhere.llm.generate(...) / .generateStream(...) -RunAnywhere.llm.tools.register(_:executor:) / .list() / .clear() -RunAnywhere.vlm.generate(image:prompt:options:) / .generateStream(...) - -// Audio and vision -RunAnywhere.stt.transcribe(_:options:) / .transcribeStream(_:options:) -RunAnywhere.tts.speak(_:options:) / .synthesize(_:options:) / .stop() -RunAnywhere.vad.detectStream(_:options:) -RunAnywhere.diarization.diarize(_:options:) -RunAnywhere.segmentation.segment(_:options:) - -// Sessions -let voice = try await RunAnywhere.voice.createSession(stt:llm:tts:) -try voice.start() // the only thing that opens the microphone -let rag = try await RunAnywhere.rag.open(embeddingModel:llmModel:) - -// LoRA -RunAnywhere.lora.registerArtifact(_:artifact:) / .allRegistered() / .queryCatalog(_:) -RunAnywhere.lora.download(_:artifact:) / .importAdapter(from:) / .applyCatalogAdapter(...) -RunAnywhere.lora.apply(RALoraApplyRequest) / .remove(RALoraRemoveRequest) / .state() -``` - -The SDK also exposes `embeddings`, `rerank`, `images`, `generateStructured`, `tts.voices()`, -`vad.detect(_:)`, `RunAnywhere.events`, `.version`, `.deviceId`, and `deleteStorage(_:)`. This -app calls none of them. Do not document them here as if it did. +Both extensions are thin (two source files each) and share their IPC protocol with the main +app through `RunAnywhereAI/Shared/`; see +[`docs/reference/FEATURES.md`](docs/reference/FEATURES.md) for the full dictation handshake. +Diffusion and image generation are excluded from the v1 build; see the comment at the top of +`RunAnywhereAIApp.swift` and the matching one in `ModelCatalogBootstrap.swift`. Their products, +registration calls, feature folders, catalog rows, and `generateImage` APIs are deliberately +absent. -Inputs are `AudioInput` and `ImageInput`. The app constructs `.pcm16`, `.uiImage`, and -`.pixelBuffer`; the other cases exist but go unused here. Options types carry all-optional -fields whose defaults come from the IDL, so the app passes only what it overrides: `LlmOptions` -and `TtsOptions` in practice. +## SDK surface -One-shot verbs throw `SDKException`. Stream factories are `async throws -> AsyncThrowingStream`, -so they throw on preflight failure and throw into the consumer mid-flight. No result carries a -`success` flag and no error text hides in a payload field. Cancel by cancelling the consuming -Task; there are no cancel verbs. +Every call goes through the `RunAnywhere` enum, one namespace per modality; the SDK owns +model resolution, loading, downloading, and orchestration behind each verb. The exact subset +this app calls — and what it deliberately does not (`embeddings`, `rerank`, `images`, +`generateStructured`, `tts.voices()`, `vad.detect(_:)`, `RunAnywhere.events`) — is in +[`docs/reference/FEATURES.md`](docs/reference/FEATURES.md). Do not assume an unused verb is +wired up here just because the SDK exposes it. The older flat verbs (`loadModel`, `transcribe`, `ragQuery`) are deprecated forwarders in the -SDK. Do not use them here. - -`RunAnywhere+ExampleShims.swift` holds one app-local helper, -`RunAnywhere.getRegisteredFrameworks() -> [RAInferenceFramework]`, which composes -`RunAnywhere.models.list()` into a framework filter sorted by descending model count. A new -feature needing net-new C bridge code belongs in the SDK; only UI plumbing over existing -canonical proto APIs belongs in that file. +SDK; do not use them here. One-shot verbs throw `SDKException`; stream factories are +`async throws -> AsyncThrowingStream` and throw into the consumer mid-flight — no result +carries a `success` flag. Cancel by cancelling the consuming `Task`; there are no cancel verbs. ## Design system @@ -431,23 +166,15 @@ No inline magic numbers or color literals in views. `AppColors` carries brand pr `#FF6900` and semantic tokens for text, backgrounds, bubbles, badges, and status; the canonical palette, motion tiers, and icon language live in the monorepo's [`docs/DESIGN_GUIDELINE.md`](https://github.com/RunanywhereAI/runanywhere-sdks/blob/main/docs/DESIGN_GUIDELINE.md). -`AppSpacing` runs xxSmall (2) to xxxLarge (40) plus icon sizes, button heights, corner radii, -and strokes. `AppType` and `AppTypography` cover text styles and weighted or monospaced -variants. `Layout` holds window sizes, content widths, and animation durations, `Motion` the -shared curves and springs, `Surface` the elevation treatments, and `AdaptiveSizing` the phone, -tablet, and desktop scaling. - -## Scripts and configuration +`AppSpacing`, `AppType`/`AppTypography`, `Layout`, `Motion`, `Surface`, and `AdaptiveSizing` +cover spacing/sizing tokens, text styles, window and content sizing, animation curves, +elevation treatments, and phone/tablet/desktop scaling respectively. -| Script | Purpose | -|---|---| -| `scripts/build_and_run_ios_sample.sh` | Resolve, build, and deploy to simulator, device, or Mac | -| `scripts/verify.sh` | Local gate: resolves the remote SDK release, runs a full `xcodebuild` | -| `scripts/smoke.sh` | Greps sources for SDK call patterns and asserts the Parakeet CTC catalog policy, without compiling. Gated in CI. | +## Configuration | File | Purpose | |---|---| -| `Package.swift` | One dependency: `github.com/RunanywhereAI/runanywhere-swift` at `from: "0.20.19"`, giving RunAnywhere, RunAnywhereONNX, RunAnywhereLlamaCPP, RunAnywhereMLX, RunAnywhereNeuRT. The Xcode project mirrors it with `upToNextMajorVersion` from the same minimum, and declares all five products on the app target. Its `RunAnywhereAITests` testTarget must point at `RunAnywhereAIUnitTests/`, not the XCUITest bundle. | +| `Package.swift` | One dependency: `github.com/RunanywhereAI/runanywhere-swift` at `from: "0.20.24"`, giving RunAnywhere, RunAnywhereONNX, RunAnywhereLlamaCPP, RunAnywhereMLX, RunAnywhereNeuRT. The Xcode project mirrors it with `upToNextMajorVersion` from the same minimum, and declares all five products on the app target. Its `RunAnywhereAITests` testTarget must point at `RunAnywhereAIUnitTests/`, not the XCUITest bundle. | | `Package.resolved` | Committed record of the resolved version and commit. CI fails if `swift package resolve` leaves it dirty, so commit it whenever the dependency moves. | | `Info.plist` | URL scheme `runanywhere`, `audio` background mode, Live Activities, and the local-network and Bonjour (`_runanywhere-connect._tcp`) declarations that Connect needs | | `RunAnywhereAI.entitlements` | App Sandbox, `device.camera`, `device.audio-input`, `network.client` and `network.server`, user-selected files (read-only and read-write), the `group.com.runanywhere.runanywhereai` app group, HealthKit, and `kernel.increased-memory-limit` | diff --git a/docs/reference/ARCHITECTURE.md b/docs/reference/ARCHITECTURE.md new file mode 100644 index 0000000..1046609 --- /dev/null +++ b/docs/reference/ARCHITECTURE.md @@ -0,0 +1,37 @@ +# Navigation architecture + +Full detail behind the one-paragraph summary in the root `AGENTS.md`. + +`ContentView` branches on platform. + +| Platform | Shell | Structure | +|---|---|---| +| macOS | `ConsumerMacShell` | `NavigationSplitView` over `MacSidebar`. Three destinations (Chat, Models, Advanced) plus the conversation list, which is scoped to Chat. Detail is `ChatInterfaceView`, `SimplifiedModelsView`, or `ConsumerAdvancedHubView`. | +| iOS | `ConsumerCompactShell` | `ChatInterfaceView` alone, plus sheets. | + +`MacSidebarSelection` has four cases: `.chat` (the transcript, whatever is current), +`.conversation(String)`, `.models`, and `.advanced`. Splitting `.chat` from `.conversation` +is what lets ⌘1 land somewhere real before anything is saved. ⌘1/⌘2/⌘3 are published from +the shell through `focusedSceneValue(\.shellNavigationActions)` because the chat cannot +navigate away from itself. One `@SceneStorage` key, `mac.sidebar.visibility`, persists +whether the sidebar is showing; column width is fixed by `navigationSplitViewColumnWidth` +and the selection is re-derived from the current conversation on restore. + +On iOS, Settings and the Advanced hub are sheets, both opened from the conversation drawer +rather than the toolbar. Models is not the same kind of sheet: the chat presents +`ModelSelectionSheet` (a picker, cross-platform), while the full `SimplifiedModelsView` +management screen is reached through a `NavigationLink` inside `CombinedSettingsView`. On +macOS `SimplifiedModelsView` is the `.models` sidebar destination. + +`ConsumerAdvancedHubView` has five sections and eight rows: + +| Section | Rows | Availability | +|---|---|---| +| Connect | Host this Mac | macOS only (`#if os(macOS)`) | +| Voice Utilities | Transcribe, Read Aloud, Voice Activity | both | +| Voice Utilities | Diarization | iOS only (`#if canImport(UIKit)`) | +| Vision Utilities | Segmentation | iOS only, and so is the whole section | +| Agents | Talk, Computer Use | both | +| Management | Benchmarks | both | + +Storage and tool calling live in Settings and Manage Models instead. diff --git a/docs/reference/FEATURES.md b/docs/reference/FEATURES.md new file mode 100644 index 0000000..df8d2ec --- /dev/null +++ b/docs/reference/FEATURES.md @@ -0,0 +1,247 @@ +# Feature behavior and SDK surface + +Per-feature detail behind the one-line index in the root `AGENTS.md`. Read the relevant +section before touching that feature's code — several of these encode a fix for a real bug, +not just a design choice. + +## Chat and LLM + +`LLMViewModel` is split across ten files by concern: core state and `sendMessage()` in +`LLMViewModel.swift`, then `+Generation` (streaming and non-streaming), `+ToolCalling`, +`+ModelManagement`, `+Analytics`, `+Events` (Combine subscription to `RunAnywhere.eventBus`), +`+Documents` (RAG-backed attachments), `+MessageActions`, `+Vision`, and the shared +`LLMViewModelTypes`. `ToolCallingModelPolicy` gates tools on context length alone +(`minimumContextTokens = 1024` against the model's `contextLength`), not on model identity; +its companion `ToolCallingExecutionPolicy` caps the run at two tool calls and 96 tokens of +final response, at temperature 0 with reasoning off. + +Flow: input, `sendMessage()`, `prepareMessagesForSending()` (creates the user message and an +empty assistant message), `executeGeneration()`, `performGeneration()`, then the streaming, +non-streaming, or tool-calling path, token-by-token message updates, `finalizeGeneration()`, +and persistence to `ConversationStore`. + +Tool calling runs through `RunAnywhere.llm.generate` with the registry active; the SDK owns +the call and execute loop and the format is auto-detected per model. + +LoRA lives almost entirely in the SDK. `ModelCatalogBootstrap.registerLoraAdapters()` seeds +the curated catalog as `RALoraAdapterCatalogEntry` values, mirroring Android's +`ModelBootstrap.seedLora`, and registers each one with `RunAnywhere.lora.registerArtifact`. +From there the app calls `RunAnywhere.lora.queryCatalog(_:)` (with an +`RALoraAdapterCatalogQuery`), `.download(_:artifact:)`, `.importAdapter(from:)`, +`.applyCatalogAdapter(_:localPath:scale:)`, `.apply(RALoraApplyRequest)` for a raw path, +`.remove(RALoraRemoveRequest)`, and `.state()`. Removal is id-keyed or `clearAll`; the old +path-keyed fallback was deleted. Scale is user-adjustable. + +Conversations persist as per-conversation JSON under `Documents/Conversations/`, attachments +under `Conversations/Attachments/{conversationID}/`. Search covers titles and message content. + +Titles are written by whichever model answered, through `RunAnywhere.llm`, not by a separate +`FoundationModels.LanguageModelSession`. Apple's Foundation Models will not serve two clients +against one on-device model: the old app-side title session hung and wedged every subsequent +turn behind it with no error and no timeout. Only one title task exists at a time and +`cancelPendingTitleGeneration()` hands the model back the moment the chat claims a new turn. + +Analytics: `MessageAnalytics` per message (time to first token, tokens per second, thinking +mode, completion status) and `ConversationAnalytics` rolled up onto the stored `Conversation`. +`ChatDetailsView` recomputes its figures from the per-message records rather than reading the +rolled-up type, and shows tokens per second, thinking usage, success rate, and average total +generation time. Per-message TTFT is rendered in `ChatMessageComponents`, not in +`ChatDetailsView`. + +Thinking mode: models with `supportsThinking` expose reasoning through the SDK's `reasoning` +options and `thinkingText` or stream thought events. Commons owns tag parsing and `/no_think` +directives; the app toggles the mode and renders the returned channel in a collapsible +section. + +Documents attached to a chat go through `ChatAttachmentLoader` and `DocumentService` for text +extraction, then `RunAnywhere.rag.open(embeddingModel:llmModel:)` for a `RagSession` cached on +the view model. The cache key is document plus embedding model plus answer model, so a session +is reused across turns rather than being strictly one per conversation. There is no separate +RAG screen. + +## Voice agent + +`RunAnywhere.voice.createSession(stt:llm:tts:)` then `session.start()`, which is the only +call that opens the microphone. `session.events` yields `agentStateChanged` (with +`.listening`, `.thinking`, `.speaking`), `speechStarted`, `speechEnded`, `userTranscribed`, +`agentResponse`, `inputSilent`, and `error`. `session.interrupt()` and `session.close()` are +the other two verbs. The SDK owns the whole audio pipeline including its own VAD. The user +loads STT, LLM, and TTS models independently through `ModelSelectionSheet`. + +`VoiceAssistantParticleView` is a Metal-rendered 2000-particle system: a Fibonacci-lattice +sphere that morphs to a ring while listening or speaking, with amplitude from the real +microphone level when listening and a simulated sine wave when speaking, and touch scatter +decaying at 0.92. + +## Speech, synthesis, and voice activity + +STT has three modes. Batch records audio then calls +`RunAnywhere.stt.transcribe(.pcm16(buffer, sampleRate: 16_000))`. Live yields microphone +chunks into `RunAnywhere.stt.transcribeStream(_:)`, which owns segmentation and emits +`.partial` and `.final`. Hybrid runs on-device first with cloud fallback through the SDK's +`HybridSTTRouter`. Capture is the SDK's `AudioCaptureManager` driven by the app-local +`AudioCapturePump`; no app-side silence detection exists. + +TTS is `RunAnywhere.tts.speak(text, options: TtsOptions(speed:))`, which synthesizes and plays +inside the SDK and hands back a handle. The app awaits `handle.waitForPlayout()` and interrupts +with `handle.interrupt()`; the whole-engine `RunAnywhere.tts.stop()` is deprecated and used only +as a fallback when no handle exists. Use `tts.synthesize(_:)` when the `Audio` buffer is wanted +instead of playback. + +VAD feeds microphone chunks to `RunAnywhere.vad.detectStream(_:)`, which emits `VadEvent` +values: `.speechStarted`, `.speechEnded`, per-chunk `.activity(isSpeech, _, _)`, `.failed`, and +`.completed`. Framing is the SDK's job. The activity log holds 50 entries. + +## Voice keyboard (RunAnywhereKeyboard + RunAnywhereActivityExtension) + +Cross-process dictation over two IPC channels: App Group `UserDefaults` +(`group.com.runanywhere.runanywhereai`) for shared state (session state, transcribed text, +audio level, heartbeat), and Darwin `CFNotificationCenter` for zero-latency signals (six names +in `SharedConstants.DarwinNotifications`, in `RunAnywhereAI/Shared/`). Both channel definitions +are shared code, not owned by either extension target. + +The keyboard's Run button opens `runanywhere://startFlow`. The main app activates a session, +loads the STT model, starts capture, and posts `sessionReady`. The user returns to the host +app, the keyboard sends `startListening`, the main app buffers audio, the keyboard sends +`stopListening`, the main app calls `RunAnywhere.stt.transcribe(_:)`, writes the result to +shared `UserDefaults`, and posts `transcriptionReady`; the keyboard inserts it through +`textDocumentProxy.insertText()`. + +`DictationActivityAttributes.ContentState` (Live Activity, in `RunAnywhereActivityExtension/`) +carries phase, elapsed seconds, transcript, and word count for the Dynamic Island and Lock +Screen. A one-second heartbeat lets the keyboard detect a main-app crash after a three-second +staleness window. + +Both extensions are thin: `RunAnywhereKeyboard/` is `KeyboardViewController` + `KeyboardView`; +`RunAnywhereActivityExtension/` is the `WidgetBundle` entry plus the Live Activity view. Neither +has its own build/test entry point — `scripts/build_and_run_ios_sample.sh` and `scripts/verify.sh` +build the whole app target including both extensions. + +## Vision + +Camera and photo-library image understanding, reached from the chat rather than its own tab. +`AVCaptureSession` with BGRA pixel format feeds +`RunAnywhere.vlm.generateStream(image: .pixelBuffer(frame), prompt:, options:)`. Live mode +captures every 2.5 seconds (`autoStreamInterval`) and clears on the first token, so an +unchanged scene does not repeat itself. Its token cap is `autoStreamMaxTokens = 64`; the +single-shot path has its own, larger `singleShotMaxTokens`. Pixel conversion belongs to the SDK: pass +`ImageInput` a `CVPixelBuffer` and do not bridge through `CIContext`. + +## Benchmarks + +Deterministic tests across LLM, STT, TTS, and VLM, each with a `BenchmarkScenarioProvider`. +`BenchmarkRunner` orchestrates with cooperative cancellation. Results persist as JSON, capped +at 50 runs. `BenchmarkExportFormat` offers Markdown and JSON; CSV exists only as a +`writeCSV(run:)` file writer with no picker entry. `SyntheticInputGenerator` produces silent +and sine-wave audio (440 Hz at 16 kHz) and solid and gradient 224x224 images. LLM scenarios +run at 50, 256, and 512 tokens measuring TTFT and decode speed. + +## Models + +`ModelListViewModel` is the canonical registry singleton, subscribed to +`RunAnywhere.eventBus.modelLifecycle` for live load and unload state. `ModelSelectionSheet` is +the universal picker, parameterized by `ModelSelectionContext`: `.llm`, `.stt`, `.tts`, `.vad`, +`.voice`, `.vlm`, `.ragEmbedding`, `.ragLLM`, `.diarization`, `.segmentation`. Custom models +arrive through `AddModelFromURLView` or `AddFromHuggingFaceView`. `ModelRecommendationEngine` +and `ModelCompatibilityLookup` drive the recommended set, and `HardwareTier` scopes it to the +device. + +## Storage + +`RunAnywhere.models.state()` gives used and free bytes; +`RunAnywhere.models.list(filter: ModelFilter(downloadedOnly: true))` gives the rows, filtered +to entries with a real on-disk size so Apple system pseudo-models drop out. Each row reads its +own `ModelInfo` for name, local path, framework, and `lastUsedAtUnixMs`. Deletion is +`RunAnywhere.models.delete(id:)`; cache and temp clearing are `RunAnywhere.clearCache()` and +`RunAnywhere.cleanTempFiles()`. `StorageViewModel` surfaces this inside Settings and the models +views. + +## Settings and tools + +`SettingsViewModel` (singleton) owns temperature, max tokens, and system prompt in +`UserDefaults`, and API key and base URL in the Keychain. Temperature, max tokens, and system +prompt each save on a Combine `debounce(0.5s)`; the thinking-mode toggle writes through +immediately. + +`ToolSettingsViewModel` is a separate singleton and registers tools through +`RunAnywhere.llm.tools`. Six are always available: `get_weather` (Open-Meteo), +`get_current_time`, `calculate` (a recursive-descent `SafeMathEvaluator`), `get_device_info`, +`get_battery_level`, and the SDK's own web search, added by `RunAnywhere.registerWebSearchTool()`. +Two more are opt-in behind a toggle and a permission prompt: `get_calendar_events` +(`CalendarTool`) and `get_health_data` (`HealthKitTool`, iOS only). `registerBuiltInTools()` +restores the enabled set at launch, because assigning a stored property inside `init` does not +fire `didSet`. + +## Markdown rendering + +One path, not a detect-and-route chain. `MarkdownBlockParser.parse(_:)` turns the reply into +`[MarkdownBlock]` (`paragraph`, `heading`, `list`, `quote`, `code`, `rule`) with no SwiftUI +involved, and `AdaptiveMarkdownText` renders one view per block: `MarkdownListView`, +`MarkdownQuoteView`, `MarkdownCodeBlock` (syntax-colored header, copy button, monospaced +scrollable body), and inline text through `MarkdownText` in `InlineMarkdownRenderer.swift`, +which uses `AttributedString(markdown:)` with bold as `.semibold`, italic as `.italic`, and +inline code monospaced and purple-tinted. + +## SDK surface used here + +Every call goes through the `RunAnywhere` enum. One namespace per modality; the SDK owns model +resolution, loading, downloading, and orchestration behind each verb. What follows is what this +app actually calls, not the SDK's full surface, which is larger. + +```swift +// Core +try RunAnywhere.initialize(apiKey:baseUrl:environment:) // one call, both phases +RunAnywhere.isReady +RunAnywhere.eventBus.events / .modelLifecycle // Combine, raw RASDKEvent protos +RunAnywhere.clearCache() / .cleanTempFiles() + +// Models +RunAnywhere.models.list(filter:) / .get(id:) / .register(_:) / .refresh() +RunAnywhere.models.download(id:) / .isResumable(...) / .checkCompatibility(...) +RunAnywhere.models.load(id:options:) / .unload(category:) / .delete(id:) / .state() + +// Generation +RunAnywhere.llm.generate(...) / .generateStream(...) +RunAnywhere.llm.tools.register(_:executor:) / .list() / .clear() +RunAnywhere.vlm.generate(image:prompt:options:) / .generateStream(...) + +// Audio and vision +RunAnywhere.stt.transcribe(_:options:) / .transcribeStream(_:options:) +RunAnywhere.tts.speak(_:options:) / .synthesize(_:options:) / .stop() +RunAnywhere.vad.detectStream(_:options:) +RunAnywhere.diarization.diarize(_:options:) +RunAnywhere.segmentation.segment(_:options:) + +// Sessions +let voice = try await RunAnywhere.voice.createSession(stt:llm:tts:) +try voice.start() // the only thing that opens the microphone +let rag = try await RunAnywhere.rag.open(embeddingModel:llmModel:) + +// LoRA +RunAnywhere.lora.registerArtifact(_:artifact:) / .allRegistered() / .queryCatalog(_:) +RunAnywhere.lora.download(_:artifact:) / .importAdapter(from:) / .applyCatalogAdapter(...) +RunAnywhere.lora.apply(RALoraApplyRequest) / .remove(RALoraRemoveRequest) / .state() +``` + +The SDK also exposes `embeddings`, `rerank`, `images`, `generateStructured`, `tts.voices()`, +`vad.detect(_:)`, `RunAnywhere.events`, `.version`, `.deviceId`, and `deleteStorage(_:)`. This +app calls none of them. Do not document them here as if it did. + +Inputs are `AudioInput` and `ImageInput`. The app constructs `.pcm16`, `.uiImage`, and +`.pixelBuffer`; the other cases exist but go unused here. Options types carry all-optional +fields whose defaults come from the IDL, so the app passes only what it overrides: `LlmOptions` +and `TtsOptions` in practice. + +One-shot verbs throw `SDKException`. Stream factories are `async throws -> AsyncThrowingStream`, +so they throw on preflight failure and throw into the consumer mid-flight. No result carries a +`success` flag and no error text hides in a payload field. Cancel by cancelling the consuming +Task; there are no cancel verbs. + +The older flat verbs (`loadModel`, `transcribe`, `ragQuery`) are deprecated forwarders in the +SDK. Do not use them here. + +`RunAnywhere+ExampleShims.swift` holds one app-local helper, +`RunAnywhere.getRegisteredFrameworks() -> [RAInferenceFramework]`, which composes +`RunAnywhere.models.list()` into a framework filter sorted by descending model count. A new +feature needing net-new C bridge code belongs in the SDK; only UI plumbing over existing +canonical proto APIs belongs in that file. diff --git a/scripts/sync-skills.sh b/scripts/sync-skills.sh new file mode 100755 index 0000000..feef132 --- /dev/null +++ b/scripts/sync-skills.sh @@ -0,0 +1,53 @@ +#!/bin/bash +# Skill trees: ONE canonical source, one generated mirror (DRY). +# +# `.claude/skills/` is the CANONICAL, human-edited skill tree (Claude Code reads it directly). +# `.agents/skills/` is a GENERATED MIRROR for non-Claude agent tooling (e.g. Codex, which has no native +# "skills" mechanism but can be pointed at plain markdown runbooks from AGENTS.md) — never hand-edit it. +# This script regenerates the mirror; `--check` verifies they are in sync. +# +# Why a copy and not a symlink: a checkout without core.symlinks=true (common on Windows) materializes +# git symlinks as text files, which would break Codex's reads. A generated copy + a drift check is portable. +# +# Both `.claude/*` and `.agents/*` are currently git-ignored in this repo — the skill content is being kept +# local until a public/private decision is made, but the two trees still need to agree with each other for +# whichever agent happens to read the mirror. +# +# Usage: +# scripts/sync-skills.sh # regenerate .agents/skills from .claude/skills +# scripts/sync-skills.sh --check # exit non-zero if the mirror is stale +set -euo pipefail + +# Repo root by WALKING UP to the nearest `.git`, not by counting parent hops. +_walk_up_to_repo_root() { + local d; d="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + while [ "$d" != "/" ]; do + if [ -e "$d/.git" ]; then printf '%s\n' "$d"; return 0; fi + d="$(dirname "$d")" + done + echo "FATAL: no repo root above ${BASH_SOURCE[0]} (looked for .git)" >&2 + return 1 +} +REPO="$(_walk_up_to_repo_root)" || exit 1 +SRC="$REPO/.claude/skills" +DST="$REPO/.agents/skills" + +if [[ ! -d "$SRC" ]]; then + echo "No .claude/skills/ yet — nothing to mirror." >&2 + exit 0 +fi + +if [[ "${1:-}" == "--check" ]]; then + if diff -rq "$SRC" "$DST" >/dev/null 2>&1; then + echo "skills in sync: .agents/skills mirrors .claude/skills" + exit 0 + fi + echo "ERROR: .agents/skills is STALE. Run scripts/sync-skills.sh (edit .claude/skills, never .agents/skills)." >&2 + diff -rq "$SRC" "$DST" || true + exit 1 +fi + +rm -rf "$DST" +mkdir -p "$(dirname "$DST")" +cp -R "$SRC" "$DST" +echo "regenerated .agents/skills from .claude/skills"