🧪 RFC: an 'orchestration intelligence' benchmark for on-device AI

[rufolangus]

Current on-device AI benchmarks (MLPerf Mobile, AIBenchmark) measure one model in isolation — tokens/sec for a single LLM, ms for a single inference. They don't capture what JibarOS actually does: multiple capabilities resident and contending in real time.

The question we'd like to see a benchmark answer:

How much intelligence can a device orchestrate real-time?
How much can it see, think, hear, at the same time?

Possible shape — four metrics:

1. Resident capability count — how many capabilities can hold their model warm without OOM-killing each other?
2. See-think-speak latency — time from camera frame in → spoken response out, end-to-end through vision.describe → text.complete → audio.synthesize.
3. Concurrent agent capacity — how many independent agentic loops can run before tail latency degrades?
4. Intelligence bandwidth — useful tokens/sec produced across all capabilities under sustained load.

A standard rig (cuttlefish + a fixed model bundle) would let device makers compare meaningfully.

Open: does this resonate? Different metric shape? Anyone working on something adjacent we should align with?