#804: hardware video decoding (NVDEC / VA-API / VideoToolbox / Media Foundation) via filmcraft-platform

[vangel76]

Fixes #804.

Draft: this needs two maintainer decisions (see the last section): bumping the FilmCraft pin, and adding filmcraft-platform, a dependency that contains OS FFI.

Why

Gap #15 (docs/gaps.md, docs/hardware-parity.md): After Effects decodes H.264 / HEVC in hardware, while EffectCraft decodes every frame in software on every platform. FilmCraft already has the hardware path in filmcraft-platform: VideoToolbox, Media Foundation / DXVA, NVDEC and VA-API, with automatic software fallback. On this Linux machine (RTX 5090 for NVDEC, Radeon 610M iGPU for VA-API), decoding H.264 / HEVC in hardware cuts the CPU cost of a frame by 5–31× and the time of a random jump in a long-GOP file by 1.6–4×. The output is bit-exact with the software decoder. Full numbers are under Measurements.

Pin bump

All filmcraft-* git deps move from 2c5ba7a (2026-10-03) to FilmCraft main 4e2d0bc2 (2026-10-10, 1078 commits later), the first revision that has filmcraft-platform. That change is its own commit (Bump FilmCraft to 4e2d0bc2). What broke, and the fixes:

• API: MediaInfo.audio → audio_streams (audio() / has_audio()); VideoStreamInfo gained hdr. These were mechanical fixes in interchange, media and two tests.
• Behaviour, 6 tests (export round-trips for H.264 / HEVC / AV1 / ProRes, automation qa::kinetic): FilmCraft now rounds a frame request to the track's nearest timestamp unit, halves up (FilmCraft 2b49201e, for 1 ms WebM timescales). The media pool asked for the middle of frame i. When the timescale equals the frame rate (our own 10 fps MP4 / MOV exports), that request rounds onto frame i+1. The pool now asks a quarter of the way into the frame. That still tolerates timestamps rounded either way, and it is a one-line change in pool.rs.
• Behaviour, 8 VP9 encoder tests: filmcraft_vp9::Decoder::new() now uses frame threads, so a picture can come out of a later call. The tests expect one picture per call, so they now use Decoder::with_threads(1). Only test code uses the VP9 decoder.
• effectcraft-gpu's unit tests all passed (203/203), then the process segfaulted at exit during GPU teardown, the same class as the known ui_gpu segfault. It did not recur on the final run.
• Lockfile: +6 FilmCraft crates (filmcraft-export pulls in render, text, captions, audio-dsp) and the macOS-only objc2-core-media/-video/-video-toolbox. windows and libloading were already in the lockfile.

Fix

• apps/effectcraft (L6) is the only crate that depends on filmcraft-platform, and only on non-wasm targets. At startup it calls filmcraft_platform::register() on a background thread, because probing the drivers takes 0.3–0.6 s. All unsafe / FFI stays inside FilmCraft's crate. unsafe_code = "forbid" is unchanged for our workspace, and L0–L4, the web app and the CLI are untouched. cargo xtask layers needed no change, since its rules only cover effectcraft-* edges and UI crates.
• New setting Settings ▸ Import ▸ Enable Hardware Accelerated Decoding (import.hardwareDecoding, on by default like After Effects; automation id settings.import.hardwareDecoding; prefs.set; i18n rows in all four catalogs). Through a new FootageSource::set_hardware_decoding it flips filmcraft_codecs::hw::set_hardware_decoding. When the value changes, the media pool closes its open decoders, so footage that is already open follows the setting without a restart. Cached frames stay valid.
• GPU Information and Help ▸ System Compatibility Report (videoDecoding) show the backend and its counters, e.g. NVDEC (on; 1200 frames decoded in hardware).
• Software fallback is automatic, all inside FilmCraft: streams the hardware declines, decoders that fail mid-stream (HybridDecoder) and other codecs decode in software.
• Interplay with #329: scrubbing a long movie: superseded viewer frames call off their decodes #784 / #794: open movies in place instead of reading them whole #796: both merge cleanly with this branch textually; the one conflict, in render/src/lib.rs against #329: scrubbing a long movie: superseded viewer frames call off their decodes #784, is a union of trait methods, plus forwarding the two new methods in CancellableFootage, and is recorded. #794: open movies in place instead of reading them whole #796's open_reader / reader_openers exist at the new pin. On a local merge of all three, the media / render / engine / host tests pass (805), as do the new tests. The #329: scrubbing a long movie: superseded viewer frames call off their decodes #784 scrub harness with cancellation also works with hardware decoders: a drag (one far frame every 16 ms) has its last frame 161 ms after the request in software, 67 ms with NVDEC and 77 ms with VA-API, and nothing stale keeps decoding.

Measurements

These are headless runs of a release build through EffectCraft's MediaPool and FilmCraft's sources, on a Ryzen with 16 threads that other jobs were also loading. The fixtures are ffmpeg-made testsrc2 + noise clips with long GOPs (250). "CPU" is process CPU time per decoded frame. "Jump" is the mean of 12 random seeks. The bench is not committed.

File	Backend	Sequential ms/frame	CPU core-ms/frame	Random jump	Scrub +15 frames
H.264 1080p	software	4.2	42.7	245 ms	59 ms
	NVDEC	3.3	2.4	87 ms	24 ms
	VA-API (610M)	4.0	2.6	90 ms	25 ms
HEVC 1080p	software	3.9	27.1	445 ms	75 ms
	NVDEC	2.6	2.6	78 ms	15 ms
	VA-API	2.7	2.1	117 ms	23 ms
H.264 2160p	software	15.0	163	1375 ms	317 ms
	NVDEC	8.3	5.2	613 ms	158 ms
	VA-API	9.4	5.5	621 ms	161 ms
HEVC 2160p	software	14.1	123	1072 ms	228 ms
	NVDEC	6.8	4.7	271 ms	69 ms
	VA-API	17.9	5.0	618 ms	139 ms
H.264 1080p, 45 min (#329 fixture)	software	1.6	10.8	152 ms	23 ms
	NVDEC	2.6	2.5	89 ms	11 ms

• Software decoding gets its throughput by running on 7–12 cores. The hardware paths use about one core, which goes to the synchronous decode plus the GPU→CPU readback.
• On easy content (the 45-minute fixture) a single hardware decoder is slower sequentially than 7 software threads. Scrubbing and CPU load still improve, which leaves the cores free for compositing.

Determinism

Over the first 60 frames of every fixture above, the hardware output was 60/60 frames bit-exact with the software decoder (max diff 0, PSNR ∞) for NVDEC and VA-API, H.264 and HEVC, 1080p and 2160p. That is expected: H.264 / HEVC decoding is exactly specified, and FilmCraft's hardware path reuses the software decoder's parsers and DPB. The only documented difference is concealment of leading pictures after a seek into an open GOP, which neither decoder reconstructs correctly. So final renders don't need to force software decoding for determinism, and this PR doesn't do it. FilmCraft has no per-source switch in its media API anyway: make_video_decoder asks the process-wide registry. The CLI never registers hardware decoders, so headless renders stay software-only.

Tests

apps/effectcraft/tests/hardware_decoding.rs (its own binary, because the switch is process-wide; the fixtures are ffmpeg-made and the test skips without ffmpeg):

• off_decodes_everything_in_software: with the setting off, register() runs (twice, harmlessly) and no frame comes from the hardware.
• codecs_the_hardware_does_not_take_stay_in_software: VP9 with the setting on uses no hardware and gives the same pixels as with it off.
• hardware_decode_smoke: skips when no hardware decoder is present. Otherwise the stream is taken by the hardware, all 48 frames are > 60 dB against software (they are exact here), and switching off closes the decoders so new frames come from software.

Gates

• cargo fmt --check
• cargo clippy --workspace --all-targets -D warnings
• full cargo test --workspace --no-fail-fast: 2832 passed, 0 failed (25 ignored)
• cargo xtask layers ok
• cargo xtask wasm ok (32 crates)
• cargo xtask assets ok

Linux only: the macOS and Windows paths compile inside FilmCraft's crate but were not built or run here.

Needs maintainer decision

1. The pin bump (2c5ba7a → 4e2d0bc2). It is a large jump. Everything above passes, but you may prefer another revision or to bump it yourselves; the second commit rebases onto any revision that has filmcraft-platform.
2. A new dependency with OS FFI. filmcraft-platform contains unsafe for VideoToolbox / Media Foundation / CUDA-NVDEC / libva (with // SAFETY: comments, unsafe_code = "deny" outside its FFI modules; FilmCraft ADR 0001). On Suggestion: optional Nvidia CUDA / NVENC fast lane #498, CUDA/NVENC was declined because of the pure-Rust / no-unsafe rules. This PR keeps our own code unsafe-free and links no SDK: libva and the NVIDIA driver are dlopened at run time and nothing is needed at build time. Still, the shipped binary would call the OS and GPU drivers' video APIs. If that isn't acceptable, the pin bump (commit 1) can land on its own.

🤖 Generated with Claude Code

[vangel76]

Real-world evidence for this PR from the community Discord (#artcraft-bugs / #effectcraft): two users report H.264 10-bit 4:2:2 footage (brdn, Apple silicon; Cullen, Sony A6700 S-Log) showing as a black frame in EffectCraft 0.8, while the same file plays in FilmCraft. The pinned FilmCraft software decoder only handles 8-bit 4:2:0, and FilmCraft's filmcraft-platform registers VideoToolbox (4:2:2 / 10-bit pixel formats) first on macOS. With this PR EffectCraft would take the same hardware path. (I haven't been able to build or run it on macOS, only Linux with NVDEC and VA-API, so a macOS check would be welcome.) Separately I'm making the black frame show an explanatory notice when decoding fails.

[vangel76]

Updated against current main (merge of upstream/main, plus a follow-up commit).

Conflicts: crates/media/src/pool.rs, crates/render/src/lib.rs, crates/ui-egui/src/menus.rs, crates/ui-egui/src/i18n/ui.rs; both sides kept. CancellableFootage now forwards the hardware-decoding methods. GPU Information gets the "Video decoding" line from sysinfo::gpu_summary.

Because this branch carries the FilmCraft bump, two upstream tests failed against it, so I adapted the code. FilmCraft now rotates pure-rotation MP4/MOV frames itself, so video_turn only handles headers it leaves as stored. It also hands AAC 3.0 to 5.1 over in WAV order, so the AAC downmix gains follow that.

Checks run: cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, full tests of effectcraft (app), -engine, -export, -host, -interchange, -media, -render and -ui-egui (1627 passed, 1 failed), cargo xtask layers, cargo xtask wasm.

The one failure is theme::tests::interface_symbols_have_a_glyph_in_every_family in ui-egui. It also fails on a clean upstream/main on this machine (missing bullet glyph in a system font family), so it is unrelated to this PR.

🤖 Generated with Claude Code

[vangel76]

@echelon, @bflatastic: this is ready for review now, and it needs two yes/no decisions from you. Everything else is done and tested.

1. Bump the FilmCraft pin from 2c5ba7a to 4e2d0bc2 (the first revision with filmcraft-platform). It is its own commit (85a34554); I will also open it as a separate PR, so you can decide on it independently of the hardware path. The behaviour changes it brings are listed under "Pin bump" in the description.
2. Add filmcraft-platform as a dependency of apps/effectcraft (L6, non-wasm only). It contains the OS FFI (VideoToolbox, Media Foundation, NVDEC, VA-API), but all of it lives in FilmCraft's crate. Our workspace keeps unsafe_code = "forbid", and L0–L4, the web app and the CLI are untouched. I know Suggestion: optional Nvidia CUDA / NVENC fast lane #498 was declined over the no-unsafe rule; this route keeps the rule for our code.

Why it matters: two Discord users get a black frame for H.264 10-bit 4:2:2 footage (see my comment above), because the pinned software decoder only handles 8-bit 4:2:0. With this PR EffectCraft takes the same hardware path as FilmCraft. On my machine hardware decoding cuts the CPU cost of a frame by 5–31x, bit-exact with the software decoder.

If you would rather not take the dependency, a "no" on (2) is useful too: I can then land only the pin bump and the black-frame notice.

🤖 Generated with Claude Code

[vangel76]

The pin bump is now also its own PR: #1603.