overlay with pseudo mouse support

[salir-admin]

Problem:

• No mouse move recordings
• During recording the video it is hard to catch where to look, usually mouse used as a right indicator. tracking the mouse moves usually gives video some important feedback.

Possible solution:

• create a pseudo mouse that shown on overlay and user can track it by eyes

[shreyaskarnik]

This already ships in Argo — cursorHighlight() (src/cursor.ts, exported from @argo-video/cli).

It injects a DOM ring that tracks the pointer, so it shows up in the recording even though the real OS cursor never does (Playwright's screencast doesn't capture it). It stays active until you call resetCursor(page).

import { test } from '@argo-video/cli';
import { cursorHighlight, resetCursor } from '@argo-video/cli';

test('my-demo', async ({ page, narration }) => {
  await page.goto('/');

  cursorHighlight(page, { color: '#60a5fa', radius: 18 });  // non-blocking, no await needed
  await narration.startRecording(page);

  narration.mark('intro');
  // ... your demo ...

  await resetCursor(page);
});

Options: color (default #3b82f6), radius (20px), pulse (true — breathing glow), clickRipple (true — ripple on every click), opacity (0.5).

The gotcha that probably made it look like "no mouse moves"

locator.click() / page.click() teleport the virtual mouse to the target in a single event, so the ring jumps rather than glides. To get a visible travel path, drive the mouse yourself with steps — each step is a real mousemove, which is exactly what the ring listens to:

async function moveTo(page, selector, steps = 25) {
  const box = await page.locator(selector).boundingBox();
  if (!box) return;
  await page.mouse.move(box.x + box.width / 2, box.y + box.height / 2, { steps });
}

await moveTo(page, '#submit');
await page.waitForTimeout(200);   // let it settle so the eye catches it
await page.click('#submit');      // clickRipple fires here

Bump steps for a slower glide. 20–30 reads well at 25–30fps.

Related

trackCursor(page, narration) is a separate thing — it records cursor positions into the narration timeline at 10Hz as telemetry (dwell/attention data for camera targeting), it does not draw anything.

Working example: demos/showcase.demo.ts uses both. Reference table is under API in the README.

If what you actually want is a pointer glyph (arrow / hand SVG) rather than a ring — that's not there today, and it'd be a reasonable addition. Let me know and I'll keep this open for it.

[salir-admin]

ref: https://github.com/CloverLabsAI/human-cursor

[salir-admin]

I implemented and validated a downstream prototype of the pointer-glyph idea.

The approach:

• Render a high-contrast arrow SVG in a fixed pointer-events: none overlay.
• Keep the existing ring concept only as a short, two-tone click ripple.
• Install the overlay with page.addInitScript() plus an immediate page.evaluate(), allowing it to work in the current document and survive navigation.
• Start visibly at the center of the frame.
• Maintain one cursor instance throughout the recording so each scene starts from the previous scene's last cursor position.
• Generate deterministic cubic Bezier paths with light distortion and ease-in/ease-out, adapting ideas from CloverLabsAI/human-cursor.
• Execute every path point through page.mouse.move(), following invisible_playwright's human-mouse guidance. This produces trusted browser events while seeded paths keep repeated recordings reproducible.
• Calculate movement duration from frame distance, bounded to approximately 300-1100 ms.
• Add a short pre-click dwell and place the pointer tip inside the control's padding so the glyph does not cover its label.
• Show a restrained click ripple, then preserve the resulting cursor position for the next action.

The usage looks roughly like this:

const cursor = await createHumanCursor(page, {
  seed: 'demo-name',
});

await cursor.click(page.getByRole('button', { name: 'New prompt' }));
await cursor.click(page.getByRole('button', { name: 'Cancel' }));

await cursor.hide(); // optional during dense reading
await cursor.dispose();

Complete prototype source

import type { Locator, Page } from '@playwright/test';

const CURSOR_ID = 'argo-human-cursor';
const CURSOR_STYLE_ID = 'argo-human-cursor-style';

export interface CursorPoint {
  x: number;
  y: number;
}

export interface HumanCursorOptions {
  /** Stable seed: identical inputs produce identical tutorial motion. */
  seed?: string;
  /** Click-feedback color. */
  accent?: string;
  /** Arrow size after video compression. */
  size?: number;
  /** Initial parking position as viewport fractions. */
  start?: CursorPoint;
}

export interface CursorMoveOptions {
  /** Override the calculated 300–1100 ms movement duration. */
  durationMs?: number;
  /** Point inside the target, where 0/0 is top-left and 1/1 is bottom-right. */
  target?: CursorPoint;
}

export interface CursorClickOptions extends CursorMoveOptions {
  dwellMs?: number;
  holdMs?: number;
  afterMs?: number;
}

function hashSeed(value: string): number {
  let hash = 2166136261;
  for (const character of value) {
    hash ^= character.charCodeAt(0);
    hash = Math.imul(hash, 16777619);
  }
  return hash >>> 0;
}

function random(seed: number): () => number {
  let state = seed;
  return () => {
    state += 0x6d2b79f5;
    let value = state;
    value = Math.imul(value ^ (value >>> 15), value | 1);
    value ^= value + Math.imul(value ^ (value >>> 7), value | 61);
    return ((value ^ (value >>> 14)) >>> 0) / 4_294_967_296;
  };
}

const easeInOutCubic = (value: number): number => (value < 0.5 ? 4 * value ** 3 : 1 - (-2 * value + 2) ** 3 / 2);

/**
 * Seeded tutorial variant of CloverLabsAI/human-cursor's curved, distorted, eased trajectory.
 * The seed is intentional: invisible_playwright's movement guidance recommends reproducible paths.
 */
export function buildHumanCursorPath(from: CursorPoint, to: CursorPoint, seed: string, steps: number): CursorPoint[] {
  const count = Math.max(2, steps);
  const next = random(hashSeed(seed));
  const dx = to.x - from.x;
  const dy = to.y - from.y;
  const distance = Math.hypot(dx, dy);
  const safeDistance = Math.max(distance, 1);
  const normal = { x: -dy / safeDistance, y: dx / safeDistance };
  const bend = Math.min(96, Math.max(18, distance * 0.13)) * (next() > 0.5 ? 1 : -1) * (0.72 + next() * 0.42);
  const firstControl = { x: from.x + dx * 0.32 + normal.x * bend, y: from.y + dy * 0.32 + normal.y * bend };
  const secondControl = {
    x: from.x + dx * 0.72 + normal.x * bend * 0.7,
    y: from.y + dy * 0.72 + normal.y * bend * 0.7,
  };

  return Array.from({ length: count }, (_, index) => {
    if (index === 0) return { ...from };
    if (index === count - 1) return { ...to };
    const t = easeInOutCubic(index / (count - 1));
    const inverse = 1 - t;
    const distortion = (next() - 0.5) * Math.min(1.4, distance / 180);
    return {
      x:
        inverse ** 3 * from.x +
        3 * inverse ** 2 * t * firstControl.x +
        3 * inverse * t ** 2 * secondControl.x +
        t ** 3 * to.x +
        normal.x * distortion,
      y:
        inverse ** 3 * from.y +
        3 * inverse ** 2 * t * firstControl.y +
        3 * inverse * t ** 2 * secondControl.y +
        t ** 3 * to.y +
        normal.y * distortion,
    };
  });
}

export function movementDurationMs(from: CursorPoint, to: CursorPoint, viewport: CursorPoint): number {
  const normalizedDistance = Math.hypot(to.x - from.x, to.y - from.y) / Math.hypot(viewport.x, viewport.y);
  return Math.round(Math.min(1100, Math.max(300, 300 + 700 * normalizedDistance)));
}

function installOverlay({
  accent,
  cursorId,
  size,
  styleId,
}: {
  accent: string;
  cursorId: string;
  size: number;
  styleId: string;
}): void {
  const previous = document.getElementById(cursorId) as (HTMLDivElement & { __cleanup?: () => void }) | null;
  previous?.__cleanup?.();
  previous?.remove();
  document.getElementById(styleId)?.remove();

  const style = document.createElement('style');
  style.id = styleId;
  style.textContent = `
    @keyframes argo-human-cursor-ripple {
      from { opacity: .75; transform: translate(-50%, -50%) scale(.35); }
      to { opacity: 0; transform: translate(-50%, -50%) scale(2.4); }
    }
    #${cursorId} {
      position: fixed; inset: 0 auto auto 0; z-index: 2147483647; pointer-events: none;
      width: ${size}px; height: ${Math.round(size * 1.3)}px; opacity: 0;
      transform: translate3d(-100px, -100px, 0); will-change: transform;
    }
    #${cursorId}[data-visible='true'] { opacity: 1; }
    #${cursorId} svg { display: block; width: 100%; height: 100%; filter: drop-shadow(0 2px 2px rgb(0 0 0 / .48)); }
    #${cursorId}[data-pressed='true'] svg { transform: scale(.9); transform-origin: 2px 2px; }
    .argo-human-cursor-ripple {
      position: fixed; z-index: 2147483646; pointer-events: none;
      width: 34px; height: 34px; border: 3px solid white; border-radius: 999px;
      box-shadow: 0 0 0 2px ${accent};
      animation: argo-human-cursor-ripple 420ms ease-out forwards;
    }
  `;
  document.head.appendChild(style);

  const cursor = document.createElement('div');
  cursor.id = cursorId;
  cursor.dataset.visible = 'false';
  cursor.dataset.pressed = 'false';
  cursor.setAttribute('aria-hidden', 'true');
  cursor.innerHTML = `
    <svg viewBox="0 0 30 39" aria-hidden="true">
      <path d="M2 2L2 30L9.4 23.5L15 36.5L21 33.8L15.4 21H26L2 2Z"
        fill="#fff" stroke="#111827" stroke-width="2.3" stroke-linejoin="round" />
    </svg>
  `;
  document.body.appendChild(cursor);

  const onMove = (event: MouseEvent): void => {
    cursor.dataset.visible = 'true';
    cursor.style.transform = `translate3d(${event.clientX}px, ${event.clientY}px, 0)`;
  };
  const onDown = (): void => {
    cursor.dataset.pressed = 'true';
  };
  const onUp = (event: MouseEvent): void => {
    cursor.dataset.pressed = 'false';
    const ripple = document.createElement('div');
    ripple.className = 'argo-human-cursor-ripple';
    ripple.style.left = `${event.clientX}px`;
    ripple.style.top = `${event.clientY}px`;
    document.body.appendChild(ripple);
    window.setTimeout(() => ripple.remove(), 450);
  };
  document.addEventListener('mousemove', onMove, true);
  document.addEventListener('mousedown', onDown, true);
  document.addEventListener('mouseup', onUp, true);
  (cursor as HTMLDivElement & { __cleanup?: () => void }).__cleanup = () => {
    document.removeEventListener('mousemove', onMove, true);
    document.removeEventListener('mousedown', onDown, true);
    document.removeEventListener('mouseup', onUp, true);
  };
}

async function setVisible(page: Page, visible: boolean): Promise<void> {
  await page.evaluate(
    ({ id, value }) => {
      const cursor = document.getElementById(id);
      if (cursor) cursor.dataset.visible = String(value);
    },
    { id: CURSOR_ID, value: visible },
  );
}

export async function createArgoHumanCursor(page: Page, options: HumanCursorOptions = {}) {
  const viewport = page.viewportSize() ?? { width: 1920, height: 1080 };
  const seed = options.seed ?? 'argo-human-cursor';
  const start = options.start ?? { x: 0.5, y: 0.5 };
  let position = { x: viewport.width * start.x, y: viewport.height * start.y };
  let movement = 0;
  const overlayOptions = {
    accent: options.accent ?? '#7c3aed',
    cursorId: CURSOR_ID,
    size: options.size ?? 30,
    styleId: CURSOR_STYLE_ID,
  };
  await page.addInitScript(installOverlay, overlayOptions);
  await page.evaluate(installOverlay, overlayOptions);
  await page.mouse.move(position.x, position.y);
  await setVisible(page, true);

  const moveTo = async (target: Locator, moveOptions: CursorMoveOptions = {}): Promise<void> => {
    await target.scrollIntoViewIfNeeded();
    const box = await target.boundingBox();
    if (!box) throw new Error('Cannot move tutorial cursor to a target without a visible bounding box.');
    const targetPoint = moveOptions.target ?? { x: 0.14, y: 0.7 };
    const destination = {
      x: box.x + box.width * targetPoint.x,
      y: box.y + box.height * targetPoint.y,
    };
    const duration =
      moveOptions.durationMs ?? movementDurationMs(position, destination, { x: viewport.width, y: viewport.height });
    const steps = Math.min(64, Math.max(18, Math.round(duration / 16)));
    const pathSeed = `${seed}:${movement}:${Math.round(destination.x)}:${Math.round(destination.y)}`;
    const path = buildHumanCursorPath(position, destination, pathSeed, steps);
    const cadence = random(hashSeed(`${pathSeed}:cadence`));
    await setVisible(page, true);
    for (const point of path.slice(1)) {
      await page.mouse.move(point.x, point.y);
      await page.waitForTimeout(Math.max(4, (duration / (path.length - 1)) * (0.72 + cadence() * 0.56)));
    }
    position = destination;
    movement += 1;
  };

  return {
    moveTo,
    async click(target, clickOptions = {}) {
      await moveTo(target, clickOptions);
      await page.waitForTimeout(clickOptions.dwellMs ?? 450);
      await page.mouse.down();
      await page.waitForTimeout(clickOptions.holdMs ?? 90);
      await page.mouse.up();
      await page.waitForTimeout(clickOptions.afterMs ?? 350);
    },
    hide: () => setVisible(page, false),
    async dispose() {
      await page.evaluate(
        ({ cursorId, styleId }) => {
          const cursor = document.getElementById(cursorId) as (HTMLDivElement & { __cleanup?: () => void }) | null;
          cursor?.__cleanup?.();
          cursor?.remove();
          document.getElementById(styleId)?.remove();
          document.querySelectorAll('.argo-human-cursor-ripple').forEach((element) => {
            element.remove();
          });
        },
        { cursorId: CURSOR_ID, styleId: CURSOR_STYLE_ID },
      );
    },
  };
}

Validation results:

• A real Argo recording demonstrated a visible center start.
• The cursor followed curved paths across two distant targets.
• The second movement began from the first target's last position.
• Both clicks produced trusted browser events and successful UI postconditions.
• A separate browser proof captured 84 trusted mouse events.
• The cursor remained readable in the final compressed video.

I see this as a complement to cursorHighlight(): the existing ring is useful for emphasis, while the pointer glyph and deliberate travel make the instructor's intent much easier for viewers to follow.

[salir-admin]

cursor-overlay-proof.mp4

[shreyaskarnik]

That video is great — the pointer actually travels, and the pre-click dwell is what sells it. Please do open a PR, this is well past prototype quality.

A few things in the design that are more right than they may look from the outside:

Driving every point through page.mouse.move() is the key decision. Because those are trusted browser events, the existing cursorHighlight() ring and trackCursor() telemetry both keep working on top of it for free — the ring follows because it listens for real mousemove, and trackCursor gets genuine dwell data instead of teleports. A DOM-only glyph animated with CSS would have looked identical in the video and quietly broken both.

Seeding matters more here than in a scraping context. Argo re-records demos constantly, and the shader transition cache is keyed on a content hash of the boundary frames — nondeterministic cursor paths would bust that cache on every export and force a browser relaunch each time. Reproducible paths keep that intact.

One cursor instance across scenes is the detail I'd have gotten wrong. Restarting each scene from center reads as a cut; continuing from the last position reads as one continuous session.

Licensing is clean — Argo is MIT, and both human-cursor and invisible_playwright are MIT. Just carry the attribution in a header comment on the file, the way src/transitions/shaders/ credits gl-transitions.

Worth deciding in the PR, not blockers

1. Relationship to cursorHighlight(). Complement or supersede? They overlap on the click ripple. My instinct is createHumanCursor() owns motion and click feedback, and cursorHighlight() stays as the lightweight "just show me where the pointer is" option for demos that don't need directed movement — but if the ripple ends up duplicated in both, one of them should delegate.

2. Composition with demoType(). Typing into a field the cursor never visibly traveled to is the seam most likely to show. Whether cursor.click() + demoType() is the documented pairing, or demoType grows a cursor-aware path, is worth a sentence in the README either way.

3. Timing against narration. Argo's convention is deriving waits from narration.durationFor(scene) rather than hardcoding ms, so voiceover and motion stay in sync when TTS text changes. A 300–1100ms move is short enough that it probably doesn't need to participate, but if a scene is mostly cursor travel it will drift. Worth thinking about whether movement duration can optionally be budgeted from the scene duration.

4. Camera moves. Post-export zoomTo applies zoompan in ffmpeg after the cursor is already burned into the frame, so the cursor scales with the zoom — which is what you'd want. No action needed, just confirming it composes rather than fights.

No need to resolve all of that before opening it. Happy to review, and thanks for building the thing rather than just filing for it.

[towfiqi]

@salir-admin @shreyaskarnik Very exciting. Can't wait for the PR!

[salir-admin]

I hope You saw that my comment contains the full source code of this feature. it is more about adopting it to codestyle of the project and properly name it. functionality is done, all is left is to merge it properly.

[salir-admin]

salir-admin and OleksandrKucherenko - is the same person :)