feat: add selector-targeted drag gestures (#1567)

* feat: add selector-targeted drag gestures

* fix: address drag gesture review feedback

* fix: satisfy drag review quality gates

* fix(android): lower drag trajectories piecewise

* test(replay): validate drag fixture selectors

* fix(ios): ignore full-viewport chrome containers

* test(drag): prove destination on live devices

Author: thiagobrez

.github/workflows/replays-nightly.yml

@@ -138,6 +138,9 @@ jobs:
             BOOT_FINISHED_AT="$(date +%s)"
             pnpm build
             pnpm clean:daemon
+            adb -s "$ANDROID_SERIAL" install -r "${{ steps.fixture-app.outputs.apk-path }}"
+            node --experimental-strip-types src/bin.ts replay examples/test-app/replays/drag-android.ad --platform android --serial "$ANDROID_SERIAL" --session nightly-android-target-drag --json
+            pnpm clean:daemon
             FULL_STARTED_AT="$(date +%s)"
             AGENT_DEVICE_ANDROID_E2E=1 AGENT_DEVICE_ANDROID_E2E_TIER=full AGENT_DEVICE_ANDROID_SERIAL="$ANDROID_SERIAL" AGENT_DEVICE_FIXTURE_APP_PATH="${{ steps.fixture-app.outputs.apk-path }}" AGENT_DEVICE_FIXTURE_APP_ID="${{ steps.fixture-app.outputs.app-id }}" node --test test/integration/smoke-android-emulator.test.ts
             FULL_FINISHED_AT="$(date +%s)"
@@ -210,6 +213,13 @@ jobs:
         run: |
           echo "Fixture app source: ${{ steps.fixture-app.outputs.source }}" >> "$GITHUB_STEP_SUMMARY"
 
+      - name: Prove selector drag reaches its destination on iOS
+        run: |
+          xcrun simctl install "${{ steps.ios-simulator.outputs.simulator-udid }}" "${{ steps.fixture-app.outputs.app-path }}"
+          pnpm clean:daemon
+          node --experimental-strip-types src/bin.ts replay examples/test-app/replays/drag.ad --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --session nightly-ios-target-drag --json
+          pnpm clean:daemon
+
       - name: Run full fixture-backed iOS simulator E2E
         env:
           AGENT_DEVICE_FIXTURE_APP_ID: ${{ steps.fixture-app.outputs.app-id }}

docs/adr/0011-interaction-guarantee-contract.md

@@ -15,6 +15,7 @@ trade-offs independently:
 | --- | --- | --- |
 | `runtime-selector` | daemon tree capture → `resolveSelectorChain` → guards → coordinate tap | full semantics |
 | `runtime-ref` | session snapshot → ref lookup → guards → coordinate tap | full semantics |
+| `target-drag` | independently resolve source + destination → guards → one pointer plan | dual-endpoint drag semantics |
 | `direct-ios-selector` | selector sent to the XCTest runner, which queries and taps natively | saves a full snapshot round trip |
 | `native-ref` | `backend.tapTarget`/`fillTarget` for `click @ref` / `fill @ref` | saves resolution round trips |
 | `coordinate` | raw x/y tap | escape hatch; semantics intentionally minimal |
@@ -266,6 +267,10 @@ dimensions as their only frame source; cheaper frame sources such as
 intentionally excluded because they can diverge from full-screen screenshot
 coordinates on affected simulators.
 
+Target-authored `gesture drag` is classified as the `target-drag` path above because it resolves two
+elements and therefore inherits the element guarantees and ADR 0012 identity requirements for both.
+Coordinate-authored drag remains outside this matrix.
+
 Two-contact pan/pinch/rotate/transform planning is now owned by the typed
 gesture-plan contract in [ADR 0013](0013-unified-gesture-plans.md). It remains a
 sibling of this element-targeting matrix: coordinate gestures do not acquire

docs/adr/0012-interactive-replay.md

@@ -132,6 +132,15 @@ the corresponding provider contract cases in the same change.
 
 ### 3. Versioned `.ad` target-binding evidence
 
+> **Amendment (#1567): dual-endpoint target evidence.** An action that resolves two elements records
+> one `# agent-device:targets-v1 {"source":{...},"destination":{...}}` annotation instead of a
+> single `target-v1`. Each nested value is a complete `TargetAnnotationV1` under the same identity,
+> normalization, field-size, and verification rules. The wrapper is capped at 8,320 bytes. Replay
+> verifies both selectors and carries two post-resolution guards before dispatch; either endpoint's
+> selector miss, identity mismatch, unverifiable evidence, or dispatch-time guard mismatch refuses
+> the entire action before pointer-down. Unknown future `targets-vN` annotations remain ordinary
+> comments to an older reader, matching the single-target versioning rule.
+
 Recording writes evidence for every action that resolves an element target. The plain-text format is a
 versioned comment immediately before the action it annotates:
 

docs/adr/0013-unified-gesture-plans.md

@@ -33,7 +33,8 @@ while still routing through the canonical `pan` input.
 The runtime plans canonical intent in `packages/contracts/src/gesture-plan.ts`. Contact topology is separate
 from motion:
 
-- one contact: pan or fling with a complete pointer trajectory and an explicit execution profile;
+- one contact: pan, fling, or target-authored drag with a complete pointer trajectory and an explicit
+  execution profile;
 - two contacts: pan, pinch, rotate, or transform with two complete, synchronized trajectories.
 
 `swipe` without a duration remains public sugar for a fixed-duration fling. Timed public forms
@@ -133,8 +134,22 @@ Public two-finger pan is additive: `pointerCount?: 1 | 2` on pan and CLI
 `kind`, `durationMs`, `pointerCount`, `from`, and `to` fields, followed by backend evidence.
 Recording/replay keeps its existing public command identity and session semantics.
 
-ADR 0011's element dispatch-path matrix remains unchanged: coordinate gestures do not resolve
-selectors or refs and therefore cannot claim element-targeting guarantees.
+Target-authored drag is additive: `gesture drag <source> <destination>` accepts a selector or pinned
+snapshot ref at each endpoint, resolves both endpoints before device injection, then lowers their center
+points to one uninterrupted single-pointer plan. The plan holds at the source, moves over the authored
+duration, optionally holds at the destination, and releases. Both endpoint-resolution disclosures and
+portable selector chains are returned to the caller. Recordings replace session-local refs at both
+endpoints with those selector chains and attach one `targets-v1` annotation containing independent source
+and destination identity evidence. Replay verifies both identities before pointer-down and guards both
+independent endpoint resolutions against the verified elements. Ref admission happens for both endpoints
+before either is dispatched, and the usual mutation boundary expires the frame after the gesture. Because
+this contract includes the source hold, timed movement, and destination hold phases, target-authored drag
+is admitted only on Android touch devices and iOS/iPadOS, whose adapters preserve the full plan.
+
+ADR 0011's coordinate path remains unchanged: coordinate-authored gestures do not resolve selectors or
+refs and therefore cannot claim element-targeting guarantees. Target-authored drag explicitly runs the
+shared selector/ref resolution preflight before it enters the coordinate gesture executor; its endpoint
+resolution and recording contracts are tested at that composition seam.
 
 ## Consequences
 

examples/test-app/replays/drag-android.ad

@@ -0,0 +1,15 @@
+context platform=android kind=emulator timeout=60000
+
+env APP_TARGET="com.callstack.agentdevicelab"
+env APP_URL=""
+
+open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}"
+react-native dismiss-overlay
+wait "Gesture lab" 30000
+wait "gesture canary ready" 5000
+wait "drag completed no" 5000
+
+gesture drag id="drag-source" id="drag-destination" 600 500 0
+wait "drag completed yes" 5000
+
+close

examples/test-app/replays/drag.ad

@@ -0,0 +1,14 @@
+context platform=ios kind=simulator timeout=60000
+
+env APP_TARGET="Agent Device Tester"
+env APP_URL=""
+
+open "${APP_TARGET}" --relaunch --launch-url "${APP_URL}"
+wait "Gesture lab" 30000
+wait "gesture canary ready" 5000
+wait "drag completed no" 5000
+
+gesture drag id="drag-source" id="drag-destination" 600 500 0
+wait "drag completed yes" 5000
+
+close

examples/test-app/src/screens/GestureLab.tsx

@@ -1,5 +1,5 @@
 import { useEffect, useRef, useState } from 'react';
-import { Image, Platform, Text, View, type GestureResponderEvent } from 'react-native';
+import { Image, Platform, Pressable, Text, View, type GestureResponderEvent } from 'react-native';
 import {
   Directions,
   FlingGestureHandler,
@@ -64,6 +64,7 @@ export function GestureLab() {
   const [canaryReady, setCanaryReady] = useState(false);
   const [transform, setTransform] = useState<TransformState>(initialTransform);
   const [counts, setCounts] = useState<GestureCounts>(initialCounts);
+  const [dragCompleted, setDragCompleted] = useState(false);
   const transformRef = useRef<TransformState>(initialTransform);
   const gestureStartRef = useRef<TransformState>(initialTransform);
   const androidTouchStartRef = useRef<AndroidTouchStart | undefined>(undefined);
@@ -191,6 +192,15 @@ export function GestureLab() {
     .minDistance(4)
     .runOnJS(true)
     .onStart(handleTwoPointerPan);
+  const holdDragGesture = Gesture.Pan()
+    .activateAfterLongPress(500)
+    .minDistance(10)
+    .runOnJS(true)
+    .onEnd((event, completed) => {
+      if (completed && Math.hypot(event.translationX, event.translationY) > 60) {
+        setDragCompleted(true);
+      }
+    });
   const legacyFlingRefs = [
     legacyFlingLeftRef,
     legacyFlingRightRef,
@@ -282,6 +292,32 @@ export function GestureLab() {
       title="Gesture lab"
       testID="gesture-lab-card"
     >
+      <View style={styles.dragRow} testID="drag-gesture-fixture">
+        <GestureDetector gesture={holdDragGesture}>
+          <Pressable
+            accessibilityLabel="Drag source"
+            accessibilityRole="button"
+            onPress={() => undefined}
+            style={styles.dragEndpoint}
+            testID="drag-source"
+          >
+            <Text style={styles.dragEndpointLabel}>Source</Text>
+          </Pressable>
+        </GestureDetector>
+        <Pressable
+          accessibilityLabel="Drag destination"
+          accessibilityRole="button"
+          onPress={() => undefined}
+          style={styles.dragEndpoint}
+          testID="drag-destination"
+        >
+          <Text style={styles.dragEndpointLabel}>Destination</Text>
+        </Pressable>
+      </View>
+      <Text style={styles.metric} testID="drag-gesture-status">
+        drag completed {dragCompleted ? 'yes' : 'no'}
+      </Text>
+
       <View
         accessibilityLabel="Gesture test image"
         onTouchEnd={

examples/test-app/src/screens/gesture-lab-styles.ts

@@ -44,5 +44,25 @@ export function createGestureLabStyles(colors: AppColors) {
       fontWeight: '600',
       lineHeight: 18,
     },
+    dragRow: {
+      flexDirection: 'row',
+      gap: 20,
+      justifyContent: 'space-between',
+    },
+    dragEndpoint: {
+      alignItems: 'center',
+      backgroundColor: colors.cardStrong,
+      borderColor: colors.line,
+      borderRadius: 8,
+      borderWidth: 1,
+      flex: 1,
+      height: 56,
+      justifyContent: 'center',
+    },
+    dragEndpointLabel: {
+      color: colors.text,
+      fontSize: 13,
+      fontWeight: '700',
+    },
   });
 }

packages/ad-replay/src/internal/__tests__/plan-digest.test.ts

@@ -118,6 +118,32 @@ test('computeReplayPlanDigest changes when target evidence consumed before actio
   }
 });
 
+test('computeReplayPlanDigest binds both entries of multi-target evidence', () => {
+  const evidence = {
+    role: 'view',
+    ancestry: [],
+    sibling: 0,
+    viewportOrder: 0,
+    verification: 'verified' as const,
+  };
+  const targetEvidences = {
+    source: { ...evidence, id: 'source' },
+    destination: { ...evidence, id: 'destination' },
+  };
+  const original = digestFor([action({ targetEvidences })]);
+  assert.notEqual(
+    original,
+    digestFor([
+      action({
+        targetEvidences: {
+          ...targetEvidences,
+          destination: { ...targetEvidences.destination, id: 'different-destination' },
+        },
+      }),
+    ]),
+  );
+});
+
 test('computeReplayPlanDigest never changes based on unsubstituted ${VAR} text (variable VALUES never affect the digest)', () => {
   // The digest is computed over the still-unsubstituted action text; --env
   // values are resolved later, at invocation time, so two runs with

packages/ad-replay/src/internal/plan-digest.ts

@@ -59,6 +59,7 @@ function canonicalizeAction(
     flags: action.flags ?? {},
     runtime: action.runtime ?? null,
     targetEvidence: action.targetEvidence ?? null,
+    targetEvidences: action.targetEvidences ?? null,
     source: { path: sourcePath, line },
   };
 }