Skip to content
Merged
30 changes: 20 additions & 10 deletions docs/internal/video-engine-presentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,10 +160,11 @@ non-Chromium browsers: Safari 18.6 reports `hvc1` and `hev1` supported alongside

## Rendering Only On Change

Under push presentation Pixi's ticker paints nothing. Every render is an
explicit answer to something that changed: a presented frame, the prepared
window advancing, a layer turned on or off, restyling, hover or selection, a
viewport move, a resize, or the swapchain coming back after the tab was hidden.
Under push presentation every render goes through the scene scheduler; Pixi's
automatic application-render callback is detached. Every render answers something
that changed: a presented frame, the prepared window advancing, a layer turned on
or off, restyling, hover or selection, a viewport move, a resize, an opted-in focus
fade, or the swapchain coming back after the tab was hidden.

`scene-render-scheduler.ts` compares a description of everything a render would
put on screen against what was last drawn, so a notification that describes the
Expand All @@ -179,10 +180,17 @@ paused, untouched, for ten seconds" in
renders once the window has settled" in
`packages/web/src/renderers/pixi-media-scene.prepared-window.test.ts`.

Paused focus is static by policy. The focus layer's fade advances with the
presented media time, so it fades while the media moves and settles at once when
a hover or selection arrives with no frame behind it. No animation keeps the
scene rendering on its own.
A focus style may opt into a bounded `transitionDurationMs` fade. Its alpha
advances on wall time outside the atomic
present, including during playback, while annotation geometry stays tied to the
presented media frame. The initial resolved detection frame applies opted-in
focus immediately; later arrivals or replacement styles can transition. The
renderer stops its Pixi ticker when the transition settles, and its final tick
paints through the scheduler. Without that option, the focus layer's fade
advances with presented media time. Explicit hover, selection, and presentation
changes settle it immediately; content readiness, depth, and visibility redraws
preserve its current alpha. It does not keep the scene rendering on its own.
Still-image focus also settles before the first render without an explicit duration.

## The Prepared Annotation Window

Expand Down Expand Up @@ -310,6 +318,8 @@ the scene renders it on change through the same scheduler and signature as the
push path, with the presented sample as one more change. It answers
`getRenderCount()`, and a notification that describes what is already drawn
renders nothing, as on a paused video. Pixi's ticker starts only while
something on the frame moves by the clock, a looping GIF region or a loading
spinner, and stops with the render that removes the last of them. The behavior is pinned in
something on the frame moves by the clock: a looping GIF region, a loading
spinner, or an opted-in focus transition. Each animation tick paints through the
scheduler, which stops the ticker with the render that removes the last of them.
The behavior is pinned in
`packages/web/src/renderers/media-renderer.still.test.ts`.
22 changes: 22 additions & 0 deletions docs/public/guides/presentation-styles.md
Original file line number Diff line number Diff line change
Expand Up @@ -356,6 +356,28 @@ heatmap regions still cut out their actual shapes; rectangle-only detections
remain dimmed. Heatmap cutouts use the current frame's visible heat and disappear
when that heat is absent or still loading.

For still images, focus changes apply immediately by default. Set
`transitionDurationMs` to opt into a fade when focus appears or disappears,
including on paused [web video engine sources](media-sessions.md#web-video-engine-sources).
With this option, focus on the initial resolved detection frame applies immediately.
The value can be a number or a function of the focus context:

```ts
focusStyle: new BaseFocusStyle({
targetMode: FocusTargetMode.Ambient,
transitionDurationMs: ({ frame }) =>
frame.detections.length > 200 ? 0 : 150,
});
```

A positive finite duration bounds each fade while its resolved value stays
unchanged. Changing that value restarts the fade from its current alpha.
On still images and paused web video
engine sources, the extra focus renders stop when the fade finishes. The default
`src` video route keeps its regular render ticker even while paused. Zero, negative,
and non-finite values apply changes without a fade. The fade also runs during
video playback; annotation geometry continues to follow the presented media frames.

## Custom Styles

Custom styles implement the same `resolve(detection, context)` contract as the
Expand Down
4 changes: 4 additions & 0 deletions packages/core/src/styles/focus-style.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ const DEFAULT_FOCUS_FILL_ALPHA = 0.45;
const DEFAULT_FOCUS_CORNER_RADIUS = 8;

export interface BaseFocusStyleOptions {
/** Opt into bounded fades, including on still or paused media. Nonpositive or non-finite values apply changes immediately. */
readonly transitionDurationMs?: FocusStyleValue<number>;
/**
* Which interaction state should keep the media undimmed.
*/
Expand Down Expand Up @@ -46,9 +48,11 @@ export interface BaseFocusStyleOptions {
*/
export class BaseFocusStyle implements FocusStyle {
protected readonly options: BaseFocusStyleOptions;
readonly transitionDurationMs: FocusStyleValue<number> | undefined;

constructor(options: BaseFocusStyleOptions = {}) {
this.options = options;
this.transitionDurationMs = options.transitionDurationMs;
}

resolve(context: FocusStyleContext): FocusDrawInstruction | undefined {
Expand Down
2 changes: 2 additions & 0 deletions packages/core/src/types/focus-style.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,5 +39,7 @@ export interface FocusDrawInstruction {
}

export interface FocusStyle {
/** Opt into bounded fades, including on still or paused media. Nonpositive or non-finite values apply changes immediately. */
readonly transitionDurationMs?: FocusStyleValue<number> | undefined;
resolve(context: FocusStyleContext): FocusDrawInstruction | undefined;
}
281 changes: 281 additions & 0 deletions packages/web/src/renderers/media-renderer.focus-push.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,281 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { BaseFocusStyle, FocusTargetMode } from "supervision-js-core";
import type { MediaRendererSource } from "#types/media-renderer";

import {
createRenderer,
domMock,
flushPixiTicker,
pixiMock,
resetMocks,
} from "../../../../test/media-renderer-harness";
import type {
PresentedFrameChannel,
PresentedFrameChannelSignal,
PresentedFrameChannelStatus,
PresentedVideoFrame,
} from "./presented-frame-channel";

beforeEach(resetMocks);

describe("focus transitions on pushed video through the public renderer", () => {
it.each([0, Infinity])(
"applies a %sms duration immediately during a present",
async (duration) => {
const producer = createPushSource();
const renderer = await createRenderer(false, false, {
src: undefined,
source: producer.source,
detectionFrames: [detectionFrame],
focusStyle: new BaseFocusStyle({
targetMode: FocusTargetMode.Ambient,
fill: { alpha: 0.5, color: 0x000000 },
shouldRender: ({ mediaTime }) => mediaTime === 0,
transitionDurationMs: duration,
}),
});
try {
await vi.waitFor(() => expect(findFocus().alpha).toBe(1));
const renders = renderer.getRenderCount() ?? 0;
domMock.performanceNow.mockClear();
producer.present(0.04);
await vi.waitFor(() =>
expect(renderer.getRenderCount()).toBe(renders + 1),
);
expect(findFocus().alpha).toBe(0);
expect(domMock.performanceNow).not.toHaveBeenCalled();
expect(pixiMock.tickerStart).not.toHaveBeenCalled();
} finally {
renderer.destroy();
}
},
);

it.each([undefined, 300])(
"preserves default fades across redraws and updates to duration %s",
async (duration) => {
const producer = createPushSource();
const shouldRender = vi.fn(
({ mediaTime }: { mediaTime: number }) =>
mediaTime >= 0.04 && mediaTime < 0.12,
);
const renderer = await createRenderer(false, false, {
src: undefined,
source: producer.source,
detectionFrames: [detectionFrame],
focusStyle: new BaseFocusStyle({
targetMode: FocusTargetMode.Ambient,
fill: { alpha: 0.5, color: 0x000000 },
shouldRender,
}),
});
try {
await vi.waitFor(() => expect(shouldRender).toHaveBeenCalled());
const restoreVisibility = vi
.mocked(document.addEventListener)
.mock.calls.filter(([name]) => name === "visibilitychange")
.at(-1)?.[1];
if (typeof restoreVisibility !== "function") {
throw new Error("Expected the renderer's visibility listener");
}
for (const time of [0.04, 0.08, 0.12]) {
const renders = renderer.getRenderCount() ?? 0;
domMock.performanceNow.mockClear();
producer.present(time);
await vi.waitFor(() =>
expect(renderer.getRenderCount()).toBe(renders + 1),
);
expect(domMock.performanceNow).not.toHaveBeenCalled();
expect(findFocus().alpha).toBeGreaterThan(0);
expect(findFocus().alpha).toBeLessThan(1);
const alpha = findFocus().alpha;
restoreVisibility.call(document, new Event("visibilitychange"));
expect(findFocus().alpha).toBe(alpha);
expect(renderer.getRenderCount()).toBe(renders + 2);
}
const alpha = findFocus().alpha;
renderer.setPresentation({
focusStyle: new BaseFocusStyle({
targetMode: FocusTargetMode.Ambient,
shouldRender: () => false,
...(duration === undefined
? {}
: { transitionDurationMs: duration }),
}),
});
if (duration === undefined) {
expect(findFocus().alpha).toBe(0);
expect(pixiMock.tickerStart).not.toHaveBeenCalled();
} else {
await vi.waitFor(() =>
expect(pixiMock.tickerStart).toHaveBeenCalledOnce(),
);
expect(findFocus().alpha).toBe(alpha);
flushPixiTicker(150);
expect(findFocus().alpha).toBeGreaterThan(0);
expect(findFocus().alpha).toBeLessThan(alpha);
flushPixiTicker(300);
expect(findFocus().alpha).toBe(0);
expect(pixiMock.tickerStop).toHaveBeenCalledOnce();
}
expect(pixiMock.render).toHaveBeenCalledTimes(
renderer.getRenderCount() ?? -1,
);
} finally {
renderer.destroy();
}
},
);

it("paints each bounded fade tick once through the scheduler outside atomic presents", async () => {
const producer = createPushSource();
const renderer = await createRenderer(false, false, {
src: undefined,
source: producer.source,
detectionFrames: [detectionFrame],
focusStyle: new BaseFocusStyle({
targetMode: FocusTargetMode.Ambient,
fill: { alpha: 0.5, color: 0x000000 },
shouldRender: ({ mediaTime }) => mediaTime === 0,
transitionDurationMs: 150,
}),
});
try {
await vi.waitFor(() => expect(findFocus()).toBeDefined());
const focus = findFocus();
expect(focus.alpha).toBe(1);
expect(pixiMock.tickerStart).not.toHaveBeenCalled();
await renderer.play();
let clockReadsDuringPresent = 0;
domMock.performanceNow.mockImplementation(() => {
if (producer.hasOpenFrame()) clockReadsDuringPresent += 1;
return 0;
});
producer.present(0.04);
await vi.waitFor(() =>
expect(pixiMock.tickerStart).toHaveBeenCalledOnce(),
);
expect(clockReadsDuringPresent).toBe(0);
expect(focus.alpha).toBe(1);
for (const time of [0, 75, 150]) {
const renders = renderer.getRenderCount() ?? 0;
flushPixiTicker(time);
expect(renderer.getRenderCount()).toBe(renders + 1);
expect(pixiMock.render).toHaveBeenCalledTimes(renders + 1);
if (time === 75) {
expect(focus.alpha).toBeGreaterThan(0);
expect(focus.alpha).toBeLessThan(1);
}
}
expect(focus.alpha).toBe(0);
expect(focus.visible).toBe(false);
expect(pixiMock.tickerStop).toHaveBeenCalledOnce();
const settled = renderer.getRenderCount();
flushPixiTicker(500);
expect(renderer.getRenderCount()).toBe(settled);
} finally {
renderer.destroy();
}
});
});

const detectionFrame = {
detections: [
{ id: "subject", rect: { x: 50, y: 50, width: 20, height: 20 } },
],
mediaTime: 0,
};

function findFocus() {
const focus = pixiMock.graphicsInstances.find((graphics) =>
graphics.fill.mock.calls.some(
([fill]) => fill?.alpha === 0.5 && fill?.color === 0x000000,
),
);
if (!focus) throw new Error("Expected the drawn focus layer");
return focus;
}

function createPushSource() {
let handler: (frame: PresentedVideoFrame) => void = () => undefined;
let time = 0;
let paintSeq = 0;
let openFrame = false;
let status: PresentedFrameChannelStatus = "PAUSED";
const listeners = new Map<PresentedFrameChannelSignal, Set<() => void>>();
const setStatus = (next: PresentedFrameChannelStatus) => {
status = next;
for (const listener of listeners.get("state") ?? []) listener();
};
const present = (mediaTime: number) => {
time = mediaTime;
openFrame = true;
handler({
frame: {
close: () => {
openFrame = false;
},
} as VideoFrame,
frameId: { index: Math.round(time * 25), ticks: Math.round(time * 1000) },
mediaTimeS: time,
paintSeq: ++paintSeq,
});
};
const engine: PresentedFrameChannel = {
beginInteractiveSeek: vi.fn(),
commit: async (timeMs) => {
present(timeMs / 1000);
},
endInteractiveSeek: vi.fn(async () => undefined),
getDurationMs: () => 1000,
getPlaybackRate: () => 1,
getPlayhead: () => ({
frame: { index: Math.round(time * 25), ticks: time * 1000 },
mediaTimeS: time,
}),
getSeeking: () => false,
getStatus: () => status,
onPresentedFrame: (next) => {
handler = next;
},
pause: () => setStatus("PAUSED"),
play: async () => {
setStatus("PLAYING");
},
scrub: vi.fn(),
setPlaybackRate: vi.fn(),
step: vi.fn(async () => undefined),
subscribe: (signal, listener) => {
const signalListeners = listeners.get(signal) ?? new Set<() => void>();
signalListeners.add(listener);
listeners.set(signal, signalListeners);
return () => {
signalListeners.delete(listener);
};
},
};
const source: MediaRendererSource = {
open: async () => ({
engine,
input: { dispose: vi.fn() },
metadata: {
audioTrackCount: 0,
canRead: true,
duration: 1,
firstTimestamp: 0,
formatMimeType: null,
formatName: "test-push-source",
mimeType: null,
primaryVideoHeight: 240,
primaryVideoWidth: 320,
trackCount: 1,
videoTrackCount: 1,
},
sampleSink: {
getSample: vi.fn(async () => null),
samples: vi.fn(async function* () {}),
},
}),
};
return { source, present, hasOpenFrame: () => openFrame };
}
Loading
Loading