Silent Interview Coach

A real-time, non-interruptive mock-interview coaching web app.

Silent Interview Coach runs a candidate through a mock interview and measures both how they deliver answers (camera alignment, speaking pace, pauses) and what they say (LLM content scoring). Feedback is delivered without interrupting — subtle visual nudges during the answer, and a scored report at the end.

For the full deep-dive (WebSocket contract, concurrency model, per-service internals, data flow), see ARCHITECTURE.md. This README is the high-level map.

---

Architecture at a glance

Two deployables connected by a single WebSocket (/ws/coach). All ML runs on the Python backend; the browser only captures media and renders feedback.

┌──────────────── Browser (Frontend) ─────────────────┐
│  Webcam + Mic → getUserMedia                         │
│    videoPump.ts   → OffscreenCanvas → JPEG @10fps     │
│    audioSource.ts → Int16 PCM 16 kHz (pcm-worklet)    │
│                                                      │
│  interview state machine · metricsStore (Zustand)    │
│  nudgeEngine.ts (live nudges) · ttsPlayer.ts (voice) │
│  report.ts (final score, computed client-side)       │
└───────────────────────┬──────────────────────────────┘
                        │  ONE WebSocket  /ws/coach
        binary = media  │  text = JSON control + metrics
                        ▼
┌──────────────── Python Backend (FastAPI) ───────────┐
│  CoachSession — one per connection                   │
│  asyncio I/O + 5 single-thread executors:            │
│    vision → FaceService     (MediaPipe Face Landmarker)
│    audio  → VadService      (Silero VAD v5)          │
│    stt    → SttService      (faster-whisper tiny.en) │
│    tts    → TtsService      (Piper, local ONNX)      │
│    llm    → content_evaluator (Google Gemini)        │
└──────────────────────────────────────────────────────┘

The two parts

Part	Stack	Responsibility
Frontend	Vite + React 19 + TypeScript, Tailwind v4, shadcn/ui, Framer Motion, Zustand	Capture camera + mic, run the interview state machine, render live nudges + final report, play the interviewer voice
Backend	Python 3.12, FastAPI, asyncio	All ML: face landmarks, voice-activity detection, speech-to-text, text-to-speech, LLM content scoring

Transport — one WebSocket

A single connection (/ws/coach) carries two frame kinds:

• Binary — [1-byte channel tag][payload]: 0x01 video JPEG (client→server), 0x02 audio PCM (client→server), 0x03 TTS WAV (server→client).
• Text (JSON) — control + metrics. Client sends init, calibrate, speak, answerStart, answerEnd, stop; server sends ready, faceStatus, calibration, metrics, speech, answerFeedback, etc.

There is no WebRTC and no API gateway — just the one WebSocket.

Backend services

Service	Model	Emits
FaceService	MediaPipe Face Landmarker (478-pt)	cameraAlignment, runs detecting → calibrating → live
VadService	Silero VAD v5	speaking flag; gates STT
SttService	faster-whisper tiny.en	paceWpm (3s tick); assembles per-answer transcript
TtsService	Piper en_US-lessac-medium (local ONNX/CPU)	interviewer speech WAV
content_evaluator	Google Gemini (gemini-flash-latest)	answerFeedback — 5 sub-scores, once per answer

Where scoring happens

• Per-answer content scoring runs on the backend (Gemini) on the assembled transcript only — never on video. Returns communication / clarity / confidence / technical + a justification.
• The composite final report is computed on the frontend (report.ts), blending delivery metrics (alignment, pace, pause) with content scores.
• Nudge timing is decided on the frontend (nudgeEngine.ts), not by the LLM — the app, not the model, decides when to interrupt.

---

Running locally

Backend (Python 3.11 or 3.12 — mediapipe/torch lack reliable 3.13 wheels):

cd Backend
py -3.12 -m venv .venv
.venv\Scripts\activate            # Windows  (POSIX: source .venv/bin/activate)
pip install -r requirements.txt
# Fetch the interviewer voice (~63 MB):
python -m piper.download_voices en_US-lessac-medium --data-dir models/piper
uvicorn app.main:app --port 8000

Frontend:

cd Frontend
npm install
npm run dev                       # Vite dev server; proxies /ws → ws://localhost:8000

Environment

Backend .env: GEMINI_API_KEY (content scoring). Optional: CONTENT_MODEL, PIPER_VOICE, FACE_LANDMARKER_PATH, ALLOWED_ORIGINS.

Models on disk: models/face_landmarker.task, models/piper/en_US-lessac-medium.onnx.

---

Current state & known gaps

This is a working local build, not yet production-hardened:

• No persistence — there is no database, ORM, or object storage. The report lives in the browser and is lost when the tab closes.
• /ws/coach is unauthenticated — anyone reaching the port gets a full ML session. Must be gated before hosting.
• Local-only deploy — uvicorn + Vite dev proxy; no CDN/edge, no metrics or error tracking wired up.
• Privacy note — because ML moved to the backend, raw camera + mic leave the device. The earlier "A/V never leaves the browser" guarantee no longer holds.

Verification

• Backend: cd Backend && .venv/Scripts/python -m pytest tests/ -q (13 tests).
• Frontend: cd Frontend && npx tsc -b && npm run build.

---

Repository layout

InterviewCoach/
├── ARCHITECTURE.md      # full system deep-dive
├── Frontend/            # Vite + React 19 + TypeScript SPA
│   └── src/
│       ├── coach/       # capture, WebSocket, TTS playback, orchestration
│       ├── interview/   # state machine, room UI, questions, report
│       ├── dashboard/   # results dashboard
│       └── state/       # Zustand metrics store
└── Backend/             # Python 3.12 + FastAPI ML backend
    └── app/
        ├── vision/      # FaceService, face geometry
        ├── audio/       # VadService, STT, turn detection
        ├── voice/       # Piper TTS
        └── llm/         # Gemini content evaluator