Skip to content

DRIFT.md

Latest commit

 

History

History
212 lines (146 loc) · 21.6 KB

DRIFT.md

File metadata and controls

212 lines (146 loc) · 21.6 KB
Raw

Live API Drift Detection

aimock produces responses shaped like real LLM APIs. Providers change their APIs over time. Drift means the mock no longer matches reality — your tests pass against aimock but break against the real API.

Three-Layer Approach

Drift detection compares three independent sources to triangulate the cause of any mismatch:

SDK types = Real API? Real API = aimock? Diagnosis
Yes No aimock drift — response builders need updating
No No Provider changed before SDK update — flag, wait for SDK catch-up
Yes Yes No drift — all clear
No Yes SDK drift — provider deprecated something SDK still references

Two-way comparison (mock vs real) can't distinguish between "we need to fix aimock" and "the SDK hasn't caught up yet." Three-way comparison can.

Running Drift Tests

# All providers (requires all three API keys)
OPENAI_API_KEY=sk-... ANTHROPIC_API_KEY=sk-... GOOGLE_API_KEY=... pnpm test:drift

# Single provider (others skip automatically)
OPENAI_API_KEY=sk-... pnpm test:drift

# Strict mode — warnings also fail
STRICT_DRIFT=1 OPENAI_API_KEY=sk-... pnpm test:drift

Required environment variables:

  • OPENAI_API_KEY — OpenAI API key
  • ANTHROPIC_API_KEY — Anthropic API key
  • GOOGLE_API_KEY — Google AI API key

Each provider's tests skip independently if its key is not set. You can run drift tests for just one provider.

Reading Results

Severity levels

  • critical — Test fails. aimock produces a different shape than the real API for a field that both the SDK and real API agree on. This means aimock needs an update.
  • warning — Test passes (unless STRICT_DRIFT=1). The real API has a field that neither the SDK nor aimock knows about, or the SDK and real API disagree. Usually means a provider added something new.
  • info — Always passes. Known intentional differences (usage fields are always zero, optional fields aimock omits, etc.).

Example report output

API DRIFT DETECTED: OpenAI Chat Completions (non-streaming text)

  1. [critical] LLMOCK DRIFT — field in SDK + real API but missing from mock
     Path:    usage.completion_tokens_details
     SDK:     object { reasoning_tokens: number }
     Real:    object { reasoning_tokens: number, accepted_prediction_tokens: number }
     Mock:    <absent>

  2. [warning] PROVIDER ADDED FIELD — in real API but not in SDK or mock
     Path:    system_fingerprint
     SDK:     <absent>
     Real:    string
     Mock:    <absent>

  3. [info] MOCK EXTRA FIELD — in mock but not in real API
     Path:    choices[0].logprobs
     SDK:     null | object
     Real:    <absent>
     Mock:    null

Fixing Detected Drift

When a critical drift is detected:

  1. Identify the response builder — the report path tells you which provider and field:

    • OpenAI Chat Completions → src/helpers.ts (buildTextCompletion, buildToolCallCompletion, buildTextChunks, buildToolCallChunks)
    • OpenAI Responses API → src/responses.ts (buildTextResponse, buildToolCallResponse, buildTextStreamEvents, buildToolCallStreamEvents)
    • Anthropic Claude → src/messages.ts (buildClaudeTextResponse, buildClaudeToolCallResponse, buildClaudeTextStreamEvents, buildClaudeToolCallStreamEvents)
    • Google Gemini → src/gemini.ts (buildGeminiTextResponse, buildGeminiToolCallResponse, buildGeminiTextStreamChunks, buildGeminiToolCallStreamChunks)
    • Gemini embedContent → src/gemini.ts (embedContent response builder)
    • Gemini Interactions → src/gemini-interactions.ts (buildInteractionsTextResponse, buildInteractionsToolCallResponse, buildInteractionsTextSSEEvents, buildInteractionsToolCallSSEEvents)
    • OpenAI Image Edit → src/images.ts (multipart /v1/images/edits handler)
    • OpenAI Audio Translation → src/transcription.ts (multipart /v1/audio/translations handler)
    • Ollama Embeddings → src/ollama.ts (/api/embed + legacy /api/embeddings response builder)
    • Cohere Embed → src/cohere.ts (/v2/embed response builder)
    • ElevenLabs TTS → src/elevenlabs-audio.ts (/v1/text-to-speech/{voice_id} response builder)

  2. Update the builder — add or modify the field to match the real API shape.

  3. Run conformance tests — pnpm test to verify existing API conformance tests still pass.

  4. Run drift tests — pnpm test:drift to verify the drift is resolved.

Model Deprecation

models.drift.ts normalizes each provider's live GET /models listing to family keys and subtracts the frozen classification in model-registry.ts. Two directions fall out of that subtraction: a live family we do not classify (new family — see the automated sync below), and a classified family the listing no longer contains (deprecation).

A deprecation needs nothing from you. The daily sync records it in deprecatedFamilies and aimock keeps mocking the family, so clients pinned to a retired model id keep working. The only thing worth doing by hand is the cheap live test model: if src/__tests__/drift/providers.ts names a model that no longer exists, the live drift legs cannot run at all, so point them at a current one and re-run pnpm test:drift.

Adding a New Provider

  1. Add the provider's SDK as a devDependency in package.json
  2. Add shape extraction functions to src/__tests__/drift/sdk-shapes.ts
  3. Add raw fetch client functions to src/__tests__/drift/providers.ts
  4. Create src/__tests__/drift/<provider>.drift.ts with 4 test scenarios
  5. Add model listing function to providers.ts and model check to models.drift.ts
  6. If the provider uses WebSocket, add protocol functions to ws-providers.ts and create ws-<provider>.drift.ts
  7. Update the allowlist in schema.ts if needed

WebSocket Drift Coverage

Alongside the 23 core drift tests (20 HTTP response-shape + 3 model deprecation), these endpoints are covered too:

Additional Endpoint Drift Coverage

Endpoint Provider Type Status
POST /v1beta/models/{model}:embedContent Gemini HTTP Covered
POST /v1/images/edits OpenAI HTTP (multipart) Covered
POST /v1/audio/translations OpenAI HTTP (multipart) Covered
POST /api/embed, /api/embeddings Ollama HTTP Covered
POST /v2/embed Cohere HTTP Covered
POST /v1/text-to-speech/{voice_id} ElevenLabs HTTP Covered
stream_options.include_usage OpenAI Streaming feature Covered
x-ratelimit-* / Retry-After 429 All providers Response headers Covered

WebSocket drift tests cover aimock's WS protocols (6 verified + 2 canary = 8 WS tests):

Gemini Interactions API (Beta)

The Gemini Interactions API (/v1beta/interactions) is covered by 4 drift tests in gemini-interactions.drift.ts:

  • Non-streaming text shape
  • Streaming text event sequence
  • Non-streaming tool call shape
  • Streaming tool call event sequence

Uses describe.skipIf(!GOOGLE_API_KEY) like other Gemini tests. The Interactions API is in Beta — shapes may shift as Google iterates on the endpoint.

Protocol Text Tool Call Real Endpoint Status
OpenAI Responses WS ✓ ✓ wss://api.openai.com/v1/responses Verified
OpenAI Realtime (GA) ✓ ✓ wss://api.openai.com/v1/realtime Verified
OpenAI Realtime (Beta) ✓ ✓ wss://api.openai.com/v1/realtime + OpenAI-Beta: realtime=v1 Verified
Gemini Live — — wss://generativelanguage.googleapis.com/ws/...BidiGenerateContent Unverified

Models: gpt-4o-mini for Responses WS, gpt-realtime-2 for Realtime GA (was gpt-4o-mini-realtime-preview).

GA Realtime Drift Tests:

  • Model canary — Verifies GA models exist (gpt-realtime, gpt-realtime-2, gpt-realtime-1.5, gpt-realtime-mini and dated snapshots) and flags unknown realtime models
  • Protocol probe — Connects with both GA and Beta protocol, normalizes event sequences, and verifies consistency
  • Event shape validation — GA event names (response.output_text.delta, conversation.item.added, conversation.item.done) and nested session config (session.audio.*, session.type, session.reasoning)

Auth: Uses the same OPENAI_API_KEY and GOOGLE_API_KEY environment variables as HTTP tests. No new secrets needed.

How it works: A TLS WebSocket client (ws-providers.ts) connects to real provider endpoints using node:tls with RFC 6455 framing. Each protocol function handles the setup sequence (e.g., Realtime session negotiation, Gemini Live setup/setupComplete) and collects messages until a terminal event. The mock side uses the existing ws-test-client.ts plaintext client against the local aimock server.

Gemini Live: graded on the AUDIO modality

aimock's Gemini Live handler implements the BidiGenerateContent protocol as documented in Google's Live API reference — setup/setupComplete handshake, clientContent with turns, serverContent with modelTurn.parts[], and toolCall responses.

A Live session carries exactly ONE response modality, and every model exposing bidiGenerateContent is a native-audio model that supports only AUDIO — Google's capabilities guide states the native audio models "only support AUDIO response modality". A session requesting TEXT is refused with an RFC 6455 CLOSE frame (code=1007, "The requested combination of response modalities (TEXT) is not supported by the model"), so ws-gemini-live.drift.ts drives responseModalities: ["AUDIO"] and grades the audio event sequence — inlineData parts plus turnComplete — along with the modality-independent toolCall. The mock side is driven by an audio fixture so both sides of the comparison see the same modality.

Model selection keys ONLY on the listing's declared bidiGenerateContent support. It must never re-derive a capability from the model name: a "native-audio" name-substring filter previously mis-classified gemini-3.1-flash-live-preview — a native-audio model that omits that substring from its name — as text-capable, which is how the leg came to request an unsupported modality.

aimock's TEXT serverContent path is exercised mock-only by ws-gemini-live.test.ts; it cannot be triangulated against a live endpoint while no Live model serves text. ws-gemini-live-modality.test.ts runs the whole three-way comparison locally against a fake provider that enforces Google's modality rule, so the mock side is verified without live credentials.

CI Schedule

Drift tests run on a schedule:

  • Daily: 6:00 AM UTC
  • Manual: Trigger via GitHub Actions UI (workflow_dispatch)
  • NOT on PR or push — these tests hit real APIs and cost money

See .github/workflows/test-drift.yml.

Automated Drift Remediation

There is no LLM/agent in the remediation loop. General (non-model-churn) drift is not auto-fixed by anything — it is caught by the daily drift test (which alerts on its own; see above) and fixed by a human like any other bug. The only automated remediation is the deterministic, zero-LLM model-family sync, which handles exactly one class of drift: a provider adding or retiring a model family. The fix-drift.yml workflow runs it on workflow_dispatch and a daily scheduled cron (independent of drift-test failure — a retired model family does not, by itself, fail the drift tests):

  1. Sync — scripts/drift-sync.ts fetches each provider's live /models listing directly and diffs it against the frozen classification in src/__tests__/drift/model-registry.ts:

    • a classified family a healthy live listing no longer contains → a provider-confirmed deprecation is a fact, not a decision, so it never routes to a human. drift-sync RECORDS it, mechanically, as a comment-marked entry in deprecatedFamilies[provider] (model-registry.ts), stamped with the date and with whether aimock's own source still references it. The mock keeps serving: includeFamilies is untouched, so every builder and fixture for that family still answers — users pin retired model ids in their own suites for years, and the upstream catalog shrinking is not a reason to break them. Recording it is also what makes it stop: the detector filters recorded families out of its candidate set, so the same retirement is not re-derived every morning for ever. Dropping a retired family from aimock altogether stays optional human cleanup (delete it from includeFamilies and deprecatedFamilies, then re-pin DATA_FROZEN["includeFamilies.<provider>"] in logic-pin.test.ts, all in one reviewed commit) — the re-pin is the reviewed decision the pin exists to force, which the sync's own changed-file allowlist forbids it from making. Nothing is broken while it is undone.
    • a genuinely new/unclassified family, or a registry structural mismatch (the AST locator could not find the array it had to edit) → not auto-applied: the decision itself is a human's. A family-keyed dedup note file is written under drift-proposals/ and the run is routed to a human (no PR spam on re-fire)

  2. Gate — scripts/drift-sync-check.ts re-verifies any mechanical edit before (inside drift-sync.ts) and after (workflow defense-in-depth) it is kept: a changed-file allowlist (only model-registry.ts data literals + drift-proposals/ notes), a checksum-pin re-assert over the frozen classification logic, and a clean re-collect. deprecatedFamilies is the one registry set deliberately not membership-pinned — a pin on the ledger the sync appends to would red on the sync's own append and revert it, every morning, forever. It gates no alert a human sees (isClassifiedFamily does not consult it), so there is nothing for a pin to defend; its invariants are asserted behaviourally in model-registry.test.ts instead.

  3. PR — the workflow opens a pull request for a human to review + merge (never auto-merged), unless an open PR already proposes the same changeset or a human has already rejected it. There are two distinct PR classes:

    • ok-applied — a successful mechanical registry edit: a recorded deprecation, or an addition a human already approved on a prior run. Pushed onto the fix/drift-* branch drift-sync.ts committed onto; a human reviews CI + the diff and merges. No alert, no red run — it is data-only bookkeeping.
    • needs-human — a routed decision, and now only a genuinely new/unclassified family or a registry structural mismatch. drift-sync.ts commits the drift-proposals/ note file(s), and the workflow pushes a distinct drift-needs-human/* branch and opens a PR so the note lands in the repo (the job also goes RED + Slack-alerts so the decision is seen). The PR is never auto-merged. To approve a new-family note, set its Decision: include line and merge the PR; the next drift-sync run reads the approved note from main and applies the mechanical registry edit (an ok-applied PR). That two-run hand-off is how the loop closes.

    Closing a drift-sync PR REJECTS that changeset, permanently. A CLOSED-but-never-merged PR carrying the <!-- drift-changeset: <key> --> marker tells the workflow a human decided against that exact changeset, so it stops re-proposing it (a genuinely different drift hashes to a different key and is unaffected; a merged PR is an accepted decision and is never read as a rejection). A still-open PR carrying the marker always wins over a closed one, so closing a duplicate does not reject the changeset the surviving PR is still proposing. The suppression is not silent, and not repetitive — the first run after the closure posts a Slack line naming the closing PR, then records an ack marker in that PR's body so the identical line is not re-posted every morning for as long as the rejection stands (which is for ever: the closure is permanent and the changeset key is date-independent). Delete that ack marker and the next run reports the suppression again. To un-suppress: REOPEN that PR — it becomes the pending proposal again, and the registry stays drifted until you do. Deleting the <!-- drift-changeset: … --> marker from the closed PR's body does not un-suppress: the marker self-heal now covers closed PRs and puts it back, because that marker going missing is far more often a human rewriting the body (to write down why they declined) than a deliberate un-suppression — and losing it that way used to resurrect the rejected changeset every morning, permanently. Reopening is the deliberate act; a body edit is not.

    Editing markers out of a drift-sync PR does nothing, on the other hand: the workflow restores the markers it owns on the PRs it can recognise as its own, warns in the run log, and dedups normally. That repair reaches a closed PR's changeset marker too — which is what makes closing one a durable rejection, since a later body edit can no longer erase the record of it. A merged PR is left alone: an accepted decision is neither a pending proposal nor a rejection, so nothing there is read and nothing is written.

    "No churn" is not the same as "could not look". Several things make drift-sync.ts SKIP a provider rather than fail: an unusable credential (a missing key, or a 401/402/403), and a live /models listing that comes back with fewer raw ids than MIN_LISTING_SIZE[provider] (a partial response, or an API that changed shape — the deprecation half then refuses to mass-remove off it). That floor is an explicit per-provider number, set below the smallest healthy listing there is evidence for. It is deliberately NOT the number of families aimock mocks: comparing raw ids against a family count is a unit mismatch, and it ratcheted anthropic's floor to 20 against a live listing of 11, abandoning that provider's deprecation half on every run for weeks while each one reported a quiet day. With nothing to diff the run reports ok-no-churn and exits 0, indistinguishable from a genuinely quiet day. So the sync prints a machine line, unchecked-providers=<csv>, and the workflow reclassifies such a run to provider-unchecked: the job goes RED and Slack points at the [skipped] <provider>: <reason> lines of the drift-sync-log artifact. Only transient classes (a 429 or a 5xx) are tolerated, and that is an allowlist of the tolerated class — not of the faults — so a skip class added later counts as unchecked by default instead of silently reading as "checked fine". An unreadable log, or a missing unchecked-providers= line, is treated as a fault too — an unprovable run must not pass as a quiet one.

    Re-fires never spam a second PR — idempotent in every run shape. Because a drift-sync PR is never auto-merged, an un-merged drift is re-detected on every daily cron run. Both PR classes therefore dedup on a stable changeset key: drift-sync.ts emits a date-independent changeset-key (a hash of the sorted set of applied + deferred family outcomes, independent of the date-stamped comment text and the run-id branch name), and each PR body carries a <!-- drift-changeset: <key> --> marker. Before opening a PR, the workflow skips if an open PR already carries that marker. This covers the mixed run — a mechanical removal of one family committed the same run a different family is deferred to a human (its note already on main) — whose committed diff is a registry edit with no new note file: a note-path-only key would be empty there and let a new PR open every day. A run that produces no new commit at all (note already on main, nothing applied) pushes nothing. The older per-note drift-proposal-note: <path> body marker is retained, but as a notice, not a guard: it used to skip the whole run on the first note some open PR already proposed, which silently discarded the rest of that run (a mixed run's registry edit, or a second note added since). Control only reaches it once the changeset key has found nothing, so the run's content is by then known to be un-proposed and standing down could only lose it. So the overlap is now logged as a warning — two open PRs carrying one note is worth explaining — and the run proceeds.

Artifacts

  • drift-report.json (test-drift.yml) / drift-sync-log, drift-sync-check-log (fix-drift.yml) — structured/plaintext run output (retained 30 days)

Cost

~31 API calls per run (20 HTTP response-shape + 3 model listing + 8 WS) using the cheapest available models (gpt-4o-mini, gpt-realtime-2, claude-haiku-4-5-20251001, gemini-2.5-flash) with 10-100 max tokens each. Under $0.25/week at daily cadence. The GA protocol probe adds a second Realtime WS connection (one GA, one Beta) per run. The 2 Gemini Live legs each open a real WS session and generate a short audio turn.