Skip to content

COWORK-TESTING.md

Latest commit

 

History

History
286 lines (234 loc) · 13.8 KB

COWORK-TESTING.md

File metadata and controls

286 lines (234 loc) · 13.8 KB
Raw

AI-Meter — Cowork test runbook (full system)

Paste this whole file to a Claude Cowork session (terminal + browser/computer-use) and say: "Follow this runbook and report results." It is self-contained — assume no prior context. This version tests everything current: backend, the dedup/supersede fix, all three real tokenizers, live capture on ChatGPT/Claude/Gemini, the polished monitor page, and the edge cases.

Mission & system shape

AI-Meter tracks a person's AI usage (ChatGPT/Claude/Gemini) and converts it to environmental impact (energy Wh, carbon gCO₂e, water mL). Two parts:

  1. Backend (Docker): FastAPI ingest → Redis stream → worker (computes impact, de-dupes by request_id) → TimescaleDB → read API + a live /monitor page.
  2. Browser extension (extension/, MV3): watches the chat web apps and POSTs token counts only (never text) to POST /v1/events. Each site loads a real per-provider tokenizer: OpenAI = exact, Gemini = near-exact (Gemma), Claude = approximate.

Your job: verify the whole chain works, the de-dup gives one event per turn, token counts are provider-accurate, and the monitor shows real data with working controls.

Ground rules

  • Repo root: D:\AI-Meter Backend (note the space — it matters for Docker mounts below).
  • Backend base URL: http://localhost:8000.
  • Everything backend runs in Docker. Install nothing on the machine (Node tests run in a container). Docker Desktop must be running.
  • Use the terminal for docker/curl; the browser for the extension + monitor UI.
  • Record PASS/FAIL (✅/⚠️/❌) per checkpoint with the actual output. Report template at the end.

Part 0 — Prereqs

docker --version && docker compose version
cd "D:/AI-Meter Backend"
  • ❌ If the Docker daemon is unreachable: open Docker Desktop, wait for "Engine running." If Docker Desktop is fully closed, launch it and wait ~30s — the containers auto-restart (they have restart: unless-stopped). Don't proceed until docker ps works.

Part 1 — Backend up & healthy

cd "D:/AI-Meter Backend"
docker compose up -d
docker compose ps                                            # C1
curl -s http://localhost:8000/healthz                        # C2
docker compose exec -T db psql -U aimeter -d aimeter -c "\dt" \
  -c "SELECT view_name FROM timescaledb_information.continuous_aggregates;"   # C3
curl -s http://localhost:8000/v1/methodology | python -m json.tool | head -20 # C4
  • C1 ✅ four services Up; db + redis (healthy).
  • C2 ✅ {"status":"ok"}.
  • C3 ✅ tables usage_events, impact_events; continuous aggregate impact_rollup_hourly.
  • C4 ✅ JSON with coefficients_version, energy_classes (small/mid/large), pue, water, disclaimer.

Part 2 — Data path + de-dup (no browser)

2a — one event flows end to end

curl -s "http://localhost:8000/v1/impact/summary?window=60m"; echo   # baseline
curl -s -X POST "http://localhost:8000/v1/events" -H "content-type: application/json" \
  -d '{"provider":"openai","model":"gpt-4o","input_tokens":120,"output_tokens":340,"region":"US","subject":"cowork-test"}'
echo; sleep 2
curl -s "http://localhost:8000/v1/impact/summary?window=60m"; echo
docker compose logs worker --tail 5
  • C5 ✅ POST returns 202 with {event_id, stream_id, status:"accepted"}.
  • C6 ✅ summary requests +1; gpt-4o in by_model with non-zero co2e/energy/water.
  • C7 ✅ worker log shows processed <uuid> model=gpt-4o ....

2b — supersede collapses a double (THE dedup fix)

Simulate a reasoning model's partial-then-final (same request_id):

RID="openai|/c/cowork|1|abc"
curl -s -X POST localhost:8000/v1/events -H 'content-type: application/json' \
  -d "{\"provider\":\"openai\",\"model\":\"gpt-5-5-thinking\",\"input_tokens\":8,\"output_tokens\":1,\"subject\":\"anon-x\",\"request_id\":\"$RID\"}"; echo
sleep 1
curl -s -X POST localhost:8000/v1/events -H 'content-type: application/json' \
  -d "{\"provider\":\"openai\",\"model\":\"gpt-5-5-thinking\",\"input_tokens\":8,\"output_tokens\":26,\"subject\":\"anon-x\",\"request_id\":\"$RID\"}"; echo
sleep 2
docker compose exec -T db psql -U aimeter -d aimeter -c "SELECT count(*) AS rows, max(output_tokens) AS out FROM usage_events WHERE request_id='$RID';"
docker compose logs worker --tail 6 | grep -E "processed|superseded|ignored"
  • C8 ✅ exactly 1 row, out = 26 (the partial was superseded, not duplicated).
  • C9 ✅ worker logs show processed ... then superseded req=... out 1->26.

Part 3 — Extension unit tests (in a container)

The repo path has a space, which breaks Docker bind-mounts in Git Bash — stage a copy to a no-space path first. (Adjust DL to the Windows username if different.)

mkdir -p /c/Users/DL/aimeter-ext-test
cp -r "/d/AI-Meter Backend/extension/." /c/Users/DL/aimeter-ext-test/
MSYS_NO_PATHCONV=1 docker run --rm -v "C:/Users/DL/aimeter-ext-test:/work:ro" \
  public.ecr.aws/docker/library/node:20-slim node /work/tests/run-tests.cjs
  • C10 ✅ ends with ALL PASSED: 24 assertions. These include the three real tokenizers: openai -> o200k_base, exact, google -> gemma-sentencepiece, near, anthropic -> claude-bpe, approx, and three tokenizers give provider-specific counts.
  • ❌ Image won't pull → confirm it's the public.ecr.aws/... ref (dodges Docker Hub rate limits).

Part 4 — Load the extension

In the browser:

  1. chrome://extensions → enable Developer mode.
  2. Load unpacked → select D:\AI-Meter Backend\extension.
  3. Confirm the card "AI-Meter — AI usage tap" has no errors (open "Errors" if present).
  4. Pin it; open the popup.
  • C11 ✅ loads clean (no manifest/service-worker errors).
  • C12 ✅ popup's "Last 24h impact" shows numbers (not "Backend unreachable"). If unreachable, open popup Settings, confirm Backend URL http://localhost:8000, Save.

Note: each AI site loads only its own tokenizer (per-site manifest). Gemini's is ~14 MB (the Gemma vocab), so gemini.google.com may take a beat longer to become active.

Part 5 — Live capture (the core) — ONE event per turn, accurate tokens

For each site: open it logged-in, send a short prompt, let the reply finish, wait ~2s, then check /v1/events/recent and the popup. Use a fresh prompt each time.

ChatGPT (https://chatgpt.com) — exact tokens, reasoning-safe

Send e.g. "In one sentence, what is entropy?"

curl -s "http://localhost:8000/v1/events/recent?limit=5"; echo
  • C13 ✅ exactly ONE new openai event for that turn — even on a thinking/reasoning model (no premature out:1 twin). Model is the real slug (e.g. gpt-5-...) from the interceptor.
  • C14 ✅ tokens are exact (popup shows no ~); input matches the real tokenizer (e.g. that prompt ≈ 8 tokens, not a chars/4 guess).

Claude (https://claude.ai) — captures + real Claude BPE

Send the same prompt.

  • C15 ✅ a new anthropic event appears (the .standard-markdown selector fix). Model claude-... from the interceptor. Exactly one event for the turn.
  • C16 ✅ counts come from the real Claude tokenizer and are flagged approx (popup ~), not chars/4.

Gemini (https://gemini.google.com) — one event, real Gemma tokens

Send the same prompt.

  • C17 ✅ exactly ONE google event (its selector matches 2 nodes but dedupeOverlaps
    • the request_id supersede collapse them).
  • C18 ✅ counts come from the real Gemma tokenizer (shown "accurate", no ~).

Cross-check in the worker logs while doing the above:

docker compose logs worker --tail 15 | grep -E "processed|superseded"
  • C19 ✅ one processed per turn (a superseded line is fine/expected on reasoning turns).

⚠️ Expected fragility: these sites change their HTML. If nothing appears for a site, its DOM selectors in extension/src/adapters/index.js have drifted — record which site and grab a snapshot (Troubleshooting). That's a maintenance item, not a backend failure.

Part 6 — The monitor page (polished dashboard)

Open http://localhost:8000/monitor in the browser.

IMPORTANT: it must be the localhost URL. The page shows REAL backend data only on localhost; opened as a file:// or any other host it shows a SYNTHETIC DEMO stream (random models). If you see models you never used, you're not on the localhost URL.

  • C20 ✅ shows your real events (the ones from Part 5, anon-… subjects = extension); the connection chip reads "live"; the intensity orb and the 60-min cards (CO₂/energy/water) reflect real numbers.
  • C21 — Pause: click Pause → chip shows "paused", the feed stops updating; send a new ChatGPT message → it does NOT appear until you click Resume (then it catches up). ✅/❌
  • C22 — Clear persists: click Clear (feed empties), then reload the page → the cleared rows do NOT come back (it's saved in localStorage), but the cumulative cards still populate and new events still arrive. ✅/❌
  • C23 — Supersede in place: send a ChatGPT reasoning prompt and watch the feed → you see one row whose output count settles to the final value (it updates in place, no duplicate row). ✅/❌
  • C24 — Theme + metric: the dark/light toggle works; the CO₂/Energy/Water metric switch changes the per-model bars / sparkline. ✅/❌
  • C25 — Methodology dialog: open "about the numbers" → it shows real coefficients (coefficients_version, energy classes small/mid/large, PUE, water, disclaimer) from /v1/methodology — not invented values. ✅/❌

Part 7 — Edge / negative tests

  1. Capture off (C26): popup toggle OFF, send a ChatGPT message → no new row in /v1/events/recent, no new processed worker log. Toggle back ON after.
  2. Offline queue (C27/C28):
    • docker compose stop api
    • send a ChatGPT message (POST fails → event queues in the extension; DB unchanged)
    • docker compose start api
    • in the popup, click Flush queue (or wait ~30s for the auto-flush alarm)
    • C27 ✅ while api was down, no new DB row; C28 ✅ after flush, the queued event appears in /v1/impact/summary.
  3. Privacy (C29): watch docker compose logs -f api (or the SW network panel) while sending a message → the POST body to /v1/events contains only provider, model, input_tokens, output_tokens, operation, region, subject — no prompt or response text.
  4. Resilience (C30, optional): in Docker Desktop, Restart the engine → after it's back, docker compose ps shows all four Up again without you running up (restart policy).

Part 8 — Report

# Check Result Notes
1 services up ✅/❌
2 /healthz ✅/❌
3 schema + cagg ✅/❌
4 /methodology ✅/❌
5 POST 202 ✅/❌
6 summary increments ✅/❌
7 worker processed log ✅/❌
8 supersede → 1 row, out=26 ✅/❌
9 worker superseded log ✅/❌
10 24 unit tests ✅/❌
11 extension loads clean ✅/❌
12 popup shows impact ✅/❌
13 ChatGPT 1 event (reasoning-safe) ✅/❌
14 ChatGPT tokens exact (no ~) ✅/❌ in:__
15 Claude captures, 1 event ✅/❌
16 Claude tokens real BPE (~) ✅/❌
17 Gemini 1 event ✅/❌
18 Gemini tokens real Gemma ✅/❌
19 worker: 1 processed/turn ✅/❌
20 monitor shows real data ✅/❌
21 monitor Pause halts ✅/❌
22 monitor Clear persists ✅/❌
23 monitor supersede in place ✅/❌
24 monitor theme + metric ✅/❌
25 methodology dialog (real) ✅/❌
26 toggle-off stops capture ✅/❌
27 offline queue survives ✅/❌
28 flush delivers ✅/❌
29 counts-only (no content) ✅/❌
30 engine-restart recovery ✅/⚠️/❌

Overall verdict: does a real prompt on each of the three sites produce exactly ONE accurate, content-free event that shows up live in the monitor? List any failed checkpoints with exact output.

Troubleshooting

  • Monitor shows random/unfamiliar models: you're not on the localhost URL (file:// or other host → demo data). Use http://localhost:8000/monitor. Force real with ?real, demo with ?demo.
  • Nothing captured on a site: selectors drifted. On the chat page DevTools console, grab a snapshot, e.g. ChatGPT: document.querySelector('[data-message-author-role="assistant"]')?.outerHTML?.slice(0,400) (Claude: .standard-markdown; Gemini: message-content) and paste it back — selectors live in extension/src/adapters/index.js.
  • Two events for one ChatGPT turn: the partial+final didn't share a request_id. Grab the two ids from docker compose logs worker | grep superseded (or the rows) and report them.
  • Popup "Backend unreachable": backend down or wrong URL — docker compose up -d; set Backend URL to http://localhost:8000 in popup Settings.
  • docker -v weird mount / "no such file": the repo path has a space — stage the extension to a no-space path first (Part 3).
  • Docker Hub rate limit on node pull: use the public.ecr.aws/docker/library/... ref (Part 3 does).
  • Reset the meter to empty: docker compose down -v && docker compose up -d (wipes the DB volume and re-inits schema).

What "success" looks like

One short prompt on each of ChatGPT / Claude / Gemini produces, within a few seconds: exactly one event with provider-accurate token counts (OpenAI exact, Gemini near-exact, Claude approx ~), visible live in the monitor with a working orb/pause/clear — and the transmitted payload carries counts only, never the conversation text.