Skip to content

docs: STT whisper->parakeet evaluation and resident swap#3

Merged
ai-hpc merged 1 commit into
mainfrom
docs/stt-parakeet-evaluation
Jun 7, 2026
Merged

docs: STT whisper->parakeet evaluation and resident swap#3
ai-hpc merged 1 commit into
mainfrom
docs/stt-parakeet-evaluation

Conversation

@ai-hpc

@ai-hpc ai-hpc commented Jun 7, 2026

Copy link
Copy Markdown
Member

Replaces the deployed STT engine (whisper.cpp ggml-small) with parakeet.cpp tdt_ctc-110m on the Jetson Orin Nano Super, and documents the head-to-head that motivated it. Addresses the STT-accuracy open item in #2.

Comparison (50 clean LibriSpeech clips, identical on-device conditions)

Metric whisper ggml-small (original) parakeet tdt_ctc-110m-q8 (current)
WER (Whisper-normalized) 1.02% 0.82%
RTFx (amortized) 19× 116×
Per-call latency (resident server) ~500 ms ~170 ms
Peak RAM Δ +726 MB +410 MB
Model on disk 466 MB 170 MB

parakeet is more accurate, ~6× faster compute, ~3× lower per-call latency, and ~300 MB leaner — and it's the small parakeet model. Three live ESP32-LyraT mic captures transcribed correctly.

What's in the PR

  • docs/stt-parakeet-evaluation.md — full report: method, comparison, real-capture check, caveats, build notes, optional streaming.
  • deploy/parakeet-stt/ — reproducible artifacts:
    • serve-mode.patch — adds a resident serve subcommand to parakeet.cpp (load model once; no per-call reload).
    • parakeet-stt-server.py — HTTP shim with the same POST /inference{"text": ...} contract as whisper-server, on :8178.
    • genie-parakeet.service — systemd unit (resident, Restart=always, on boot).
    • README.md — build / install / revert.

Integration & reversibility

genie-core is unchanged (it still posts to :8178; parakeet now answers there). Swap = systemctl disable --now genie-whisper && systemctl enable --now genie-parakeet; revert is the inverse.

Caveats

Clean LibriSpeech subset (both score low) — noisy-24 kHz real-capture WER is not yet quantified at scale (only the single live capture). tdt_ctc-110m is the lean model; tdt-0.6b-v2 is more accurate but heavier (swap via the service's PK_MODEL). q8_0 quant, device-specific.

Replace the deployed STT engine (whisper.cpp ggml-small) with parakeet.cpp
tdt_ctc-110m on :8178. On the Jetson Orin Nano Super, parakeet wins on every
axis: WER 0.82% vs 1.02%, RTFx 116x vs 19x, per-call latency ~170ms vs ~500ms,
peak RAM +410MB vs +726MB, model 170MB vs 466MB. Live LyraT captures transcribe
correctly.

Adds the comparison report (docs/stt-parakeet-evaluation.md) and reproducible
deploy artifacts (deploy/parakeet-stt/): a resident `serve` patch for
parakeet.cpp, an HTTP shim matching whisper-server's POST /inference, and a
systemd unit. genie-core is unchanged; the swap is reversible.

Addresses #2.
@ai-hpc ai-hpc merged commit b426c98 into main Jun 7, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant