AGENTS.md

Scrawl: Agent Instructions

Scrawl is the human-to-agent half of a feedback loop: the agent builds, the human looks, points, talks and draws, and the agent understands what was meant and acts. A capture (⌥⇧S on macOS, a cabled iPhone (mirror plus native video), or ScrawlKit's three-finger long press inside an iOS app) records the screen, voice, cursor, clicks and ⇧ ink, and Scrawl compiles it into a compact .scrawl bundle (contact-sheet.png, frames/frame-XX.png, summary.md, scrawl.json, recording.mp4 or recording.mov, and audio.wav) in ~/Library/Application Support/scrawl/captures/. The Mac app is Rust + GPUI with a Swift helper (scrawl-helper); ScrawlKit recordings are imported with scrawl import (Sources/ScrawlKit). See README for the iPhone, ScrawlKit and testing.

Principles

1. The human's time is the scarce resource. Starting (⌥⇧S), marking (⇧) and delivering (auto-routing or ⌥⇧V) should feel instant and take as few steps as possible.
2. Don't make the agent guess what Scrawl can capture. When the OS, window server, Accessibility tree, browser page or host app knows which window, element, URL or session is on screen, record that fact instead of leaving the agent to infer it from pixels and words like "this part" or "here".
3. Images stay visual and text carries meaning. contact-sheet.png and frames/*.png show the screen, red ink, click ripples and each keyframe's number in a small white corner chip (no #, no time), with no rasterized captions or card chrome. summary.md and the clipboard carry the time-aligned transcript, interaction events and absolute file paths as plain text that pastes cleanly into a terminal or a GUI composer. Details are in docs/keyframes-and-text.md.
4. Spend the agent's context where the signal is. Cut keyframes only on real actions, marks or visual changes (never on speech pauses over a static screen), crop to the active region while keeping full frames on disk, and point the agent at scrawl zoom for a closer look around a moment.
5. Inspect real bundles after every change. Whenever working on Scrawl or after Kit tests a capture, inspect the latest bundles in ~/Library/Application Support/scrawl/captures/ (both summary.md / scrawl.json and visual contact-sheet.png / frames/*.png via read). Proactively surface concrete improvements whenever you spot issues in the output or UX, such as redundant or noisy text in summary.md, cluttered or low-density contact-sheet.png layouts, unnecessary keyframes (e.g. static frames cut on speech pauses), ghost strokes or misaligned annotations, slow startup or post-processing latency, or clipboard/paste friction.

Build, test and install

• Keep Cargo.toml and Cargo.lock stable and build/test with taskpolicy -c utility cargo build -j 3 --offline and taskpolicy -c utility cargo test -j 3 --offline -- --test-threads=2.
• After changing Rust code, rebuild with cargo build before running target/debug/scrawl: cargo test does not rebuild the binary.
• Keep everything local on main unless asked otherwise.
• Experiments: a change to what users get (the paste, summary.md, the images, routing) ships behind an experiment, off by default, until Kit turns it on for everyone. Kit enables one locally with "experiments": {"<name>": true} in ~/.config/scrawl/config.json; code asks handoff::experiment("<name>") (a value other than true/false, "yes"/"off" or a number is skipped and logged, and a bad value for any config key drops only that key) (tests see only handoff::enable_experiment, never the config file), and scrawl reprocess / scrawl inspect take --experiment <name> to try one on a bundle. Bug fixes ship normally. Current experiments are listed in docs/keyframes-and-text.md.
• Feature map: site/src/features.json lists every user-facing feature with its status (stable@<version>, main, experiment:<flag>, branch:<name>, review:#<pr>), platforms, shortcuts, config keys and docs. A PR that adds, removes, renames or changes the status of a user-facing feature updates it in the same PR. A release moves its main entries to stable@<version> (docs/release.md). cargo test feature_map fails when a config key or experiment the code reads has no entry, or an entry names one that's gone. The site renders it at the unlinked /map-tyyxsf5b/ (MAP_DIR in site/scripts/feature-audit.ts), with a status board and what looks inconsistent.
• Always install and launch via ./scripts/install-app.sh (~/Applications/Scrawl.app): Never run target/debug/scrawl directly as the background daemon because hivemind clean purges target/ directories, which unlinks the binary on disk and causes macOS tccd to revoke Screen Recording and Microphone access. ./scripts/install-app.sh copies the binary into ~/Applications/Scrawl.app, signs it with Kit's Developer ID (dev.kitlangton.scrawl) so TCC permissions persist across rebuilds, and restarts the app.
• Agents: install-app.sh starts the app with nohup from your shell, and the agent's shell tool can kill it when the command returns. If pgrep -fl Scrawl.app shows nothing afterwards, relaunch with open -a ~/Applications/Scrawl.app.
• Installing on another Mac (e.g. from the Studio onto the MacBook Pro): run ./scripts/install-app-remote.sh pro. Running install-app.sh over SSH can't sign (no keychain), and running it as a launchd job can't write into the installed app (App Management), so the remote script signs a staging copy in a launchd job and copies it into place over SSH.
  • Releasing: follow docs/release.md (version PR, release-app.sh with SCRAWL_SPARKLE_DIR, tag, the Pro's dev.kit.scrawl-appcast job over ~/.local/share/scrawl-release/updates, appcast PR, site deploy from the Pro, Homebrew PR). Sparkle's tools and the updates folder live under ~/.local/share/scrawl-release/, never target/ or /tmp.

Safety and recovery

• Never tccutil reset Kit's real grants to test permissions (preview the permission panel instead; see docs/onboarding-and-settings.md).
• Test more than one display with a CGVirtualDisplay (scripts/virtual-display.swift), never on Kit's desk Mac.
• Never wake or flash Kit's displays: no caffeinate -d/-u, no real-screen recordings or benchmarks on the Studio (perf/screen-guard.sh refuses there). Use the MacBook Pro under /tmp/pro-gui.lock, a virtual display or a macvm VM.
• Never play audio out loud on Kit's Macs (say, afplay, speaker playback) to test the mic or transcription: feed recorded WAVs (scrawl reprocess <bundle> --out … --bench 1 --live-speed 1 streams a bundle's audio.wav), use a silent virtual loopback device you remove afterwards, or use the Mac mini VMs (macvm).
• Tests never post to a real OpenCode session: swap the outside steps through IngestHooks (ScrawlKit ingest) and routing::Hooks (dev-site routing).
• Delivery recovery: Read the original timeline.json before re-sending an imported capture. Its top-level session is the configured destination; context.session only names the session visible in the recorded app. Never use the latter to choose scrawl send --session. scrawl send without --session uses the top-level destination and refuses a different --session unless --reroute is passed (handoff::recorded_destination). Preserve an explicit destination during recovery, check whether it already received the capture, and ask before overriding it. If no destination is recorded and app ownership is ambiguous, ask rather than guessing from the viewed session or recent activity. Do not change a private project's endpoint or recipient as a recovery shortcut.

Subsystem map

Read the matching docs/ file before changing a subsystem. Each one holds that area's invariants, CLI tools and test commands.

Subsystem	Code	Read first
Shortcuts (a press records the screen; the menu opens the picker), Mac recording, more than one display	src/shortcut.rs, src/capture.rs, src/screens.rs, src/ui.rs	docs/capture.md
Keyframe selection, transcription (SpeechAnalyzer, Hex, command), summary.md, frames and contact sheet	src/keyframes.rs, src/transcribe.rs, src/hex.rs, src/command_transcriber.rs, src/speech_model.rs, src/sketch.rs, src/render.rs	docs/keyframes-and-text.md
Onboarding, permissions, Settings, command-line tool and agent skill	src/onboarding.rs, src/settings.rs, src/command_line.rs, helper/Onboarding.swift, helper/PermissionPanel.swift, helper/Settings.swift	docs/onboarding-and-settings.md
iPhone capture, ScrawlKit ingest and delivery, dev-site routing and sdk.js	src/phone.rs, src/delivery.rs, src/routing.rs, src/web.rs, src/sdk.js, Sources/ScrawlKit/, helper/	docs/iphone-and-routing.md
Capsule motion, the pill lab, tuning	src/ui/lab.rs, src/ui/motion.rs, perf/pill/	docs/motion.md
Video export, retention and shrinking	src/export/, src/retention.rs, helper/Export.swift, helper/Media.swift	docs/export-and-retention.md
Windows: the OS shell behind platform::, overlay, hotkeys, Media Foundation, Parakeet, GPUI Settings and tour	src/capture/windows.rs, src/ui/windows.rs, src/win/, src/settings_window.rs, src/onboarding_window.rs, src/ax_other.rs	docs/windows.md