Skip to content

Add native hardware video decoding, HDR output, and audio continuity - #168

Draft
nocell wants to merge 3 commits into
omacom:mainfrom
nocell:feat/native-video-decode
Draft

Add native hardware video decoding, HDR output, and audio continuity#168
nocell wants to merge 3 commits into
omacom:mainfrom
nocell:feat/native-video-decode

Conversation

@nocell

@nocell nocell commented Sep 8, 2026

Copy link
Copy Markdown

ARM64 guest applications spend substantial CPU time decoding video, while the existing eight-bit display path loses HDR output. This adds host VideoToolbox decoding, a paired ten-bit HDR presentation path, consistent SDR interface brightness, and a fix for reproduced HDA audio dropouts.

Closes #167. CPU/RAM settings remain separate in #156.

Demonstration

The two owner-supplied recordings show the same Tron 4K60 HDR source. The accelerated setup uses a 120 Hz guest display; this is not a claim of 120 FPS source playback. The recordings are resized to 1920×1242 for upload, retain their original timing and audio, and preserve ten-bit BT.2020/PQ capture metadata.

Before — hardware video decoding and guest HDR disabled (42.64 seconds):

try-omarchy-without-acceleration-and-hdr.mp4

After — hardware video decoding and native HDR enabled (60.52 seconds):

try-omarchy-4k60-hdr-demo.mp4

These are visual demonstrations, not an isolated decoder benchmark: both acceleration and HDR differ, and the window layouts differ. Both files use the Mac's HDR screen-recording format, including the recording of the SDR guest. Playback appearance depends on the viewer's browser and display. Neither recording measures physical panel luminance or proves frame delivery at 120 FPS.

Implementation

  • Hardware-required VideoToolbox sessions support HEVC Main/Main10, AV1 Main and VP9 profiles 0/2. A supervised VA-API broker/helper exchanges bounded messages; private Mach capabilities pass decoder IOSurfaces to QEMU/ANGLE for GPU copies into VirGL textures. A bounded memory path supports CPU readback. Firefox retains its RDD seccomp sandbox.
  • The paired virtio-gpu module exposes ten-bit scanout and HDR metadata. Hyprland composites BT.2020/PQ; QEMU presents through an RGBA16Float Metal/EDR layer. Shared GPU events and a bounded three-surface pool avoid CPU readback and completion waits. Drawable acquisition runs off QEMU's main loop; an activity token prevents App Nap throttling while allowing normal system sleep.
  • The guest enables HDR only after capability negotiation. Scoped Mesa/mpv builds handle P010 and Wayland color metadata; Vivaldi enables Wayland color management. Older guest drivers keep the SDR path.
  • The active host display preset supplies an SDR-white hint, with a 250-nit fallback. A Hyprland patch advertises the same white level to color-managed clients, correcting dim Chromium interfaces beside brighter wallpaper. Chromium can also adapt HDR midtones to this white level; mpv's explicitly targeted PQ output remains unchanged in the tested comparison.
  • SDL's audio timer changes from 10 ms to 1 ms, preventing the reproduced full HDA-ring drops. PipeWire's existing quantum stays at 4096.
  • Sources, patches, binary hashes and corresponding-source notices are included in the native guest packages and build integration.

Validation

Development machine: M5 Pro, 18 vCPUs and 12 GiB guest RAM. Measurements are specific to the tested streams and machine.

  • Final make test passed with Python 3.12: 219 Swift tests in 48 suites, 87 guest unit tests, guest contracts, build/cache tests and macOS shell/storage suites. Native SDR-white and client-white policy tests passed separately.
  • Rebuilt and verified the QEMU runtime, native-video/HDR packages and patched Hyprland. The actual Metal presenter test passed SDR → HDR → SDR transitions, 100/1000-nit signal values, primaries, orientation, GPU synchronization and bounded submission under a stalled presentation queue.
  • FFmpeg: all 480 frames across four HEVC/AV1/VP9 streams matched software decoding exactly, including timestamps. GPU surface reuse and CPU readback were tested separately. Three alternating decode-only repetitions produced the medians below; guest CPU time excludes the host helper and is not total system power.
Decode-only stream Hardware FPS Software FPS Hardware guest CPU s Software guest CPU s
HEVC 4K, 10-bit 56.68 43.89 0.794 14.318
AV1 4K, 10-bit 115.16 17.34 0.403 103.364

Hardware decoding reduces guest CPU work but is not always faster for every codec. The AV1 software result is below real-time 60 FPS; this benchmark uses a different fixture from the Tron recordings.

  • Foreground playback: the complete 126.56-second 4K60 YouTube test had zero drops across 7,529 measured frames. mpv's complete 4K AV1 test had one presentation drop and zero decoder drops. Subsequent HDR checks covered YouTube VP9 Profile 2 4K60, local AV1 10-bit 4K60 and HEVC Main10 4K30, with media time checked against wall time.
  • Digital scanout checks confirmed Chromium white moving from approximately 205 to 597 nits for a 600-nit host hint, HDR highlights near 1000 nits, and unchanged explicitly targeted mpv PQ pixels across the SDR-white change. These are signal checks, not colorimeter readings.
  • Audio A/B during muted 4K60 YouTube playback: the 45-second stereo tone changed from 37 discontinuities/channel and 43.3787 seconds retained audio to zero discontinuities and 45.0000 seconds. QEMU and SDL callback captures agreed.
  • A clean factory image was assembled from an unprovisioned verified base, current overlays and verified packages, finalized/repacked with project scripts, and booted on a new user disk through graphical setup. No previous user state was imported. Source changes and added commit history were checked for private workspace references and common credential patterns before publication.

Remaining validation and tradeoffs

This remains a draft for architecture/implementation review. A full from-source Docker guest build is still blocked because pinned Rust 1:1.98.0-1 disappeared from the current Arch ARM repository; the clean-image validation reused the verified existing ttfx package without changing that pin. It does not establish a complete from-source rebuild.

HDR uses a private paired virtio extension and an exact 7.2.2-2-aarch64-ARCH kernel module. Optional private CoreDisplay APIs supply only a display capability hint and fall back safely when unavailable. The HDR surface pool adds about 190 MiB at 4K, excluding Metal drawables. Dolby Vision dynamic metadata is not transported. Other Mac generations, external displays, Bluetooth and physical acoustic output need separate validation. The 1 ms audio timer increases requested wakeups while audio is active.

See native video and native HDR for contracts, reproduction commands and detailed results.

@nocell nocell changed the title Add native hardware video decoding and fix audio dropouts Add native hardware video decoding, HDR output, and audio continuity Sep 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add host-backed hardware video decoding for the ARM64 guest

1 participant