Skip to content

Commit 143baef

Browse files
committed
docs: align snapshot fallback and actions guidance
The snapshot guide claimed a zero-node XCTest result fails without ever switching to AX, but regular iOS capture has an explicit recursive-tree → query-sweep → private-AX recovery plan (ADR 0004, RunnerTests+SnapshotCapturePlan.swift). The public CLI also exposes `--actions`, `--force-full`, and `--timeout`, while both website reference pages published a three-flag snapshot usage line. - Add a schema-derived gate: `commands.md` must publish the exact usage `buildCommandUsage('snapshot', getCliCommandSchema('snapshot'))` produces, so the canonical invocation cannot drift from the command schema again. The flag list is never restated in the test. Proven red against the pre-fix `commands.md`. - Extract the fence walker both doc checks now share, and prove the new gate fails on a planted usage drift. - Publish the canonical snapshot usage in the command reference and describe `--actions` as iOS-simulator-only and planning-only. - Replace the "Backends (iOS)" list with an iOS capture behavior section written from ADR 0004 and the live capture plan: regular visible strategy with a bounded recovery ladder, raw diagnostic strategy preserving strict capture failures, and recovered/sparse/degraded output staying observable through quality warnings. Capture tiers are documented as internal, not as user-selectable backends. Custom-action discovery stays separate from invocation: the runner can read names but cannot trigger them (RunnerAXSnapshotBridge.h), so neither page implies otherwise. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014VzyMim5q4jbp1xDMm3jVo
1 parent 057f0e6 commit 143baef

3 files changed

Lines changed: 116 additions & 24 deletions

File tree

src/__tests__/command-doc-coverage.test.ts

Lines changed: 64 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,8 @@ import assert from 'node:assert/strict';
33
import { describe, test } from 'vitest';
44
import { PUBLIC_COMMANDS, isKnownCliCommandName } from '../command-catalog.ts';
55
import { cliCommandAlias } from '../commands/cli-command-aliases.ts';
6+
import { getCliCommandSchema } from '../cli-schema/command-schema.ts';
7+
import { buildCommandUsage } from '../cli-schema/usage.ts';
68

79
// Enumerates the public command surface (`PUBLIC_COMMANDS`, derived from the
810
// command descriptor registry) against the human-facing command reference at
@@ -29,31 +31,54 @@ const UNDOCUMENTED_PUBLIC_COMMAND_WAIVERS: readonly CommandDocWaiver[] = [];
2931
// companion binary). Empty today.
3032
const UNKNOWN_DOCUMENTED_COMMAND_WAIVERS: readonly CommandDocWaiver[] = [];
3133

32-
// Extracts the set of `agent-device <token>` command tokens documented inside
33-
// fenced code blocks, mapped to the 1-based line where each first appears. Prose
34-
// mentions outside code fences are ignored on purpose: only executable usage
35-
// lines count as "documenting" a command, so a passing reference like
36-
// "the effective agent-device state dir" is not read as an `agent-device state`
37-
// command.
38-
function extractDocumentedCommandTokens(markdown: string): Map<string, number> {
39-
const documented = new Map<string, number>();
34+
// Yields the lines inside fenced code blocks with their 1-based line numbers.
35+
// Prose outside code fences is skipped on purpose: only executable usage lines
36+
// count as "documenting" a command, so a passing reference like "the effective
37+
// agent-device state dir" is not read as an `agent-device state` command.
38+
function* fencedLines(markdown: string): Generator<{ text: string; lineNumber: number }> {
4039
let insideFence = false;
4140
const lines = markdown.split('\n');
4241
for (let index = 0; index < lines.length; index++) {
43-
const line = lines[index] ?? '';
44-
if (/^\s*(?:```|~~~)/.test(line)) {
42+
const text = lines[index] ?? '';
43+
if (/^\s*(?:```|~~~)/.test(text)) {
4544
insideFence = !insideFence;
4645
continue;
4746
}
48-
if (!insideFence) continue;
49-
const token = /^\s*agent-device\s+([a-z0-9-]+)/.exec(line)?.[1];
47+
if (insideFence) yield { text, lineNumber: index + 1 };
48+
}
49+
}
50+
51+
// Extracts the set of `agent-device <token>` command tokens documented inside
52+
// fenced code blocks, mapped to the 1-based line where each first appears.
53+
function extractDocumentedCommandTokens(markdown: string): Map<string, number> {
54+
const documented = new Map<string, number>();
55+
for (const { text, lineNumber } of fencedLines(markdown)) {
56+
const token = /^\s*agent-device\s+([a-z0-9-]+)/.exec(text)?.[1];
5057
if (token !== undefined && !documented.has(token)) {
51-
documented.set(token, index + 1);
58+
documented.set(token, lineNumber);
5259
}
5360
}
5461
return documented;
5562
}
5663

64+
// Locates a verbatim executable usage line inside a code block. Command names
65+
// are covered by the token checks above; this is the stricter per-command gate
66+
// that keeps a published invocation identical to the schema that produces it.
67+
function findUsageLine(markdown: string, usageLine: string): number | undefined {
68+
for (const { text, lineNumber } of fencedLines(markdown)) {
69+
if (text.trim() === usageLine) return lineNumber;
70+
}
71+
return undefined;
72+
}
73+
74+
// The canonical `snapshot` invocation, derived from the command schema rather
75+
// than restated here: `snapshot` grew `--actions`, `--force-full`, and
76+
// `--timeout` while the command reference still published a three-flag usage
77+
// line, so the flag list must stay owned by the schema.
78+
function canonicalSnapshotUsageLine(): string {
79+
return `agent-device ${buildCommandUsage('snapshot', getCliCommandSchema('snapshot'))}`;
80+
}
81+
5782
function isRegistryCommandToken(token: string): boolean {
5883
return isKnownCliCommandName(token) || cliCommandAlias(token) !== undefined;
5984
}
@@ -111,6 +136,14 @@ function undocumentedMessage(missing: readonly string[]): string {
111136
);
112137
}
113138

139+
function canonicalUsageMessage(usageLine: string): string {
140+
return (
141+
`${COMMANDS_DOC_PATH} does not publish the canonical usage line \`${usageLine}\`. ` +
142+
'The command schema owns CLI syntax: update the documented code block to match it rather ' +
143+
'than editing this expectation.'
144+
);
145+
}
146+
114147
function unknownMessage(unknown: readonly string[]): string {
115148
return (
116149
`${COMMANDS_DOC_PATH} documents command token(s) absent from the command registry: ` +
@@ -142,6 +175,15 @@ describe('command reference doc coverage', () => {
142175
assert.deepEqual(unknown, [], unknownMessage(unknown));
143176
});
144177

178+
test('commands.md publishes the canonical snapshot CLI usage', () => {
179+
const usageLine = canonicalSnapshotUsageLine();
180+
assert.notEqual(
181+
findUsageLine(markdown, usageLine),
182+
undefined,
183+
canonicalUsageMessage(usageLine),
184+
);
185+
});
186+
145187
test('no stale waivers', () => {
146188
assert.deepEqual(
147189
findStaleUndocumentedWaivers(
@@ -216,6 +258,15 @@ describe('command reference doc coverage gate behavior', () => {
216258
assert.match(message, /retired-command/);
217259
});
218260

261+
test('a drifted snapshot usage line fails, naming file and canonical usage', () => {
262+
const usageLine = canonicalSnapshotUsageLine();
263+
const driftedDocs = markdown.replaceAll(usageLine, 'agent-device snapshot [-i]');
264+
assert.equal(findUsageLine(driftedDocs, usageLine), undefined);
265+
const message = canonicalUsageMessage(usageLine);
266+
assert.match(message, /website\/docs\/docs\/commands\.md/);
267+
assert.ok(message.includes(usageLine), 'the failure names the usage line the schema produces');
268+
});
269+
219270
test('a waiver suppresses an intentional forward omission', () => {
220271
const withNewCommand = [...publicCommands, 'deliberately-hidden'];
221272
const missing = findUndocumentedPublicCommands(withNewCommand, documentedCommands, [

website/docs/docs/commands.md

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -316,7 +316,7 @@ agent-device close
316316
## Snapshot and inspect
317317
318318
```bash
319-
agent-device snapshot [--diff] [-i] [-d <depth>] [-s <scope>] [--raw]
319+
agent-device snapshot [--diff] [-i] [-d <depth>] [-s <scope>] [--raw] [--actions] [--force-full] [--timeout <ms>]
320320
agent-device diff snapshot [-i] [-d <depth>] [-s <scope>] [--raw]
321321
agent-device get text @e1
322322
agent-device get attrs @e1
@@ -333,6 +333,12 @@ agent-device get attrs @e1
333333
Android interactive window roots when available, so keyboard and system-overlay nodes can appear
334334
alongside the app root; `androidSnapshot.captureMode` and `androidSnapshot.windowCount` describe
335335
the capture.
336+
- `--actions` names the custom accessibility affordances an element merged away (iOS
337+
`UIAccessibilityCustomAction`, React Native `accessibilityActions`), so a card whose reply/options
338+
controls are not separate elements still lists them. It is iOS-simulator-only and exists for
339+
planning, not invocation: there is no API to trigger a named action, so reach the affordance
340+
through the element's detail screen, the same control exposed as a labeled element elsewhere, or
341+
coordinates from its rect. See [Snapshots](/docs/snapshots) for the full constraints.
336342
- `diff snapshot` compares the current snapshot with the previous session baseline and then updates baseline.
337343
- `snapshot --diff` is an alias for `diff snapshot`.
338344
- Default snapshot text is an agent-facing, token-efficient view for planning and targeting actions. It may collapse helper/accessibility noise; use `--raw` or `--json` when you need the full provider tree.

website/docs/docs/snapshots.md

Lines changed: 45 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -12,18 +12,37 @@ agent-device snapshot -i # Interactive elements only (recommende
1212
agent-device snapshot -d 3 # Limit depth to 3 levels
1313
agent-device snapshot -s "Contacts" # Scope to label/identifier
1414
agent-device snapshot -i -d 5 # Combine options
15+
agent-device snapshot --actions # Name custom actions merged into elements (iOS simulator)
1516
agent-device diff snapshot # Preferred structural diff vs previous session baseline
1617
agent-device snapshot --diff # Alias for the same diff operation
1718
```
1819

19-
| Option | Description |
20-
| ------------ | ---------------------------- |
21-
| `-i` | Interactive-only output |
22-
| `-d <depth>` | Limit tree depth |
23-
| `-s <scope>` | Scope to label or identifier |
20+
| Option | Description |
21+
| ---------------- | ------------------------------------------------------------------------------- |
22+
| `--diff` | Structural diff against the previous session baseline (alias for `diff snapshot`) |
23+
| `-i` | Interactive-only output |
24+
| `-d <depth>` | Limit tree depth |
25+
| `-s <scope>` | Scope to label or identifier |
26+
| `--raw` | Full provider tree instead of the visible-first agent view |
27+
| `--actions` | Name the custom accessibility actions merged inside an element (iOS simulator) |
28+
| `--force-full` | Re-emit the full tree even when it is unchanged since the previous snapshot |
29+
| `--timeout <ms>` | Maximum wall-clock time for the snapshot command |
2430

25-
Note: If XCTest returns 0 nodes (foreground app changed), agent-device fails explicitly.
26-
It does not automatically switch to AX.
31+
`--actions` constraints:
32+
33+
- iOS simulators only. Physical iOS devices, macOS, and Android targets reject the flag.
34+
- It names the affordances an element merged away (iOS `UIAccessibilityCustomAction`, React Native
35+
`accessibilityActions`), so a card whose reply/options controls are not separate elements still
36+
lists them.
37+
- The names are for planning and discovery, not invocation. There is no API to trigger one: reach
38+
the affordance through the element's detail screen, through the same control exposed as a labeled
39+
element elsewhere, or by coordinates from its rect.
40+
- Each merged element costs one accessibility round trip, so the pass is opt-in and bounded. When it
41+
cannot read every candidate, the response says how many it read — an absent list on an unread
42+
element is not evidence that it has none.
43+
- Do not combine it with `--raw`. The raw diagnostic strategy stays tree-first, and only the private
44+
accessibility backend can read custom actions, so `--raw --actions` returns a raw tree without
45+
them.
2746

2847
## Efficient snapshot usage
2948

@@ -66,7 +85,23 @@ agent-device snapshot -i
6685
# [off-screen below] 2 interactive items: "All Contacts", "New List"
6786
```
6887

69-
## Backends (iOS):
88+
## iOS capture behavior
89+
90+
Capture tiers are internal. There is no flag that selects a backend; `--raw` chooses a strategy, and
91+
the strategy owns which tiers it may use.
7092

71-
- `xctest` (default): full fidelity, fast, no Accessibility permission required.
72-
- `ax`: fast accessibility tree, may miss details, requires Accessibility permission; simulator-only.
93+
- Regular (non-`--raw`) capture uses the **regular visible strategy**: it starts with the recursive
94+
XCTest tree, and when that returns **sparse** output for a screen XCTest cannot serialize, it can
95+
recover through a query sweep and then, on simulators, a private accessibility backend.
96+
- The ladder is bounded by the capture budget rather than retried indefinitely; when the budget is
97+
spent, the best payload captured so far is returned.
98+
- Recovery and degradation stay observable instead of being presented as an empty UI. A
99+
**recovered** capture warns that it fell back to another backend and is safe to continue from; a
100+
**sparse** capture reports that no backend could read the screen and points you at `screenshot`
101+
as visual truth plus coordinate taps. Use `--json` and read `snapshotQuality` when you need the
102+
state, backend, and reason behind **degraded** output.
103+
- `--raw` uses the **raw diagnostic strategy**: it stays tree-first and preserves strict capture
104+
failures, so a real XCTest accessibility serialization error surfaces as an error rather than as
105+
an empty tree.
106+
- Private-accessibility recovery and `--actions` reads are simulator-specific. Physical iOS devices
107+
have no equivalent independent semantic backend; they bound the XCTest work with a probe instead.

0 commit comments

Comments
 (0)