M3 persist aggregate call origin metadata

Author: douglasmonsky

CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Unreleased
 
 - Remove low-value call/thread anchor diagnostics from the experimental call investigator to avoid an extra source-log scan per context load.
+- Persist call-origin metadata as categorical aggregate fields during indexing so normal dashboard payloads do not reopen source JSONL logs to infer user-vs-Codex initiation.
 
 ## 0.5.0 - 2026-06-10
 

docs/architecture.md

@@ -4,7 +4,8 @@ Codex Usage Tracker is a local sidecar app. It reads aggregate token counters fr
 
 ## Boundaries
 
-- `parser.py` converts local JSONL events into aggregate `UsageEvent` records. It must not persist prompts, assistant text, tool output, or transcript snippets.
+- `parser.py` converts local JSONL events into aggregate `UsageEvent` records. It also attaches metadata-only call-origin categories such as user message, tool result, post-compaction, and agent continuation. It must not persist prompts, assistant text, tool output, or transcript snippets.
+- `call_origin.py` owns the pure call-origin classifier and migrated-row fallback. It must not open source JSONL files; source-log reads belong in parser refresh or explicit context loading only.
 - `schema.py` owns persisted `usage_events` columns. Add columns there before changing SQLite migrations or export behavior.
 - `store.py` owns SQLite setup, refresh, rebuild, and query access. Keep filesystem scanning, database writes, SQL prefilters, counts, limits, and offsets here.
 - `reports.py` is the application-service layer for summaries, expensive-call reports, recommendations, pricing coverage, and filtered query payloads. CLI and MCP should call this layer instead of duplicating report assembly.
@@ -28,7 +29,7 @@ Codex Usage Tracker is a local sidecar app. It reads aggregate token counters fr
 4. Add dashboard-only interactions in `plugin_data/dashboard/dashboard.js` and keep URL state in `dashboard_state.js`.
 5. Keep all examples, screenshots, mocks, and tests synthetic. Never derive fixtures from real logs.
 6. When editing skill instructions, update both the source `skills/...` file and the bundled `src/codex_usage_tracker/plugin_data/skills/...` copy. `scripts/check_release.py` verifies that installable plugin assets stay complete and synced.
-7. When adding fields derived from `cwd`, Git metadata, or source paths, decide how they behave in `normal`, `redacted`, and `strict` privacy modes before exposing them in dashboard, JSON, CSV, MCP, or support-bundle output.
+7. When adding fields derived from `cwd`, Git metadata, source paths, or log-event metadata, decide how they behave in `normal`, `redacted`, and `strict` privacy modes before exposing them in dashboard, JSON, CSV, MCP, or support-bundle output.
 
 ## Validation
 

docs/call-drilldown-performance-checklist.md

@@ -38,9 +38,9 @@ Milestone 0 inspection ran on `perf/call-drilldown-performance-hardening` after
 
 Suspected hot paths confirmed by source inspection:
 
-- `src/codex_usage_tracker/dashboard.py` calls `annotate_rows_with_call_origin(...)` inside `dashboard_payload`.
-- `src/codex_usage_tracker/call_origin.py` groups rows by `source_file` and opens each JSONL file to infer call origin.
-- `src/codex_usage_tracker/server.py` serves `/api/usage` by calling `dashboard_payload`, so live refresh inherits the source-log scan.
+- M3 removed the `dashboard_payload` source-log call-origin scan. Call origin is now persisted as aggregate categorical metadata during parser refresh, with a cheap fallback for migrated rows.
+- M3 converted `src/codex_usage_tracker/call_origin.py` to pure classifiers that do not open source JSONL files.
+- `src/codex_usage_tracker/server.py` serves `/api/usage` by calling `dashboard_payload`; after M3, this no longer inherits call-origin source-log reads.
 - M2 removed `_read_call_anchors(...)` from `load_call_context`, so explicit context loading no longer performs the extra anchor scan.
 - M2 removed all dashboard reads of `payload.call_anchors` and `payload.thread_anchors`.
 - `src/codex_usage_tracker/plugin_data/dashboard/dashboard_data.js` builds helper indexes, but adjacent-call lookup and render paths still need a focused large-history review.
@@ -59,7 +59,7 @@ Already implemented before this branch:
 - [x] M0.1 contain calls-table horizontal overflow inside the table card.
 - [x] M1 validate and package the call investigator dashboard asset in CI, docs, and release checks.
 - [x] M2 remove low-value call/thread anchor diagnostics and their extra context source scan.
-- [ ] M3 persist aggregate call-origin metadata during indexing so dashboard payloads do not scan source logs.
+- [x] M3 persist aggregate call-origin metadata during indexing so dashboard payloads do not scan source logs.
 - [ ] M4 persist cheap performance-critical dashboard query helper fields where feasible.
 - [ ] M5 add optional timing diagnostics to `/api/usage` and `/api/context`.
 - [ ] M6 make explicit context loading single-pass where practical.
@@ -102,10 +102,21 @@ Full branch closeout should also run the release validation listed in `docs/deve
 - `docs/development.md`
 - `src/codex_usage_tracker/plugin_data/dashboard/dashboard.css`
 - `src/codex_usage_tracker/context.py`
+- `src/codex_usage_tracker/call_origin.py`
+- `src/codex_usage_tracker/dashboard.py`
+- `src/codex_usage_tracker/models.py`
+- `src/codex_usage_tracker/parser.py`
+- `src/codex_usage_tracker/schema.py`
+- `src/codex_usage_tracker/store.py`
 - `src/codex_usage_tracker/plugin_data/dashboard/dashboard.js`
 - `src/codex_usage_tracker/plugin_data/dashboard/dashboard_call_investigator.js`
+- `docs/privacy.md`
 - `tests/test_privacy.py`
+- `tests/test_call_origin.py`
+- `tests/test_parser.py`
+- `tests/test_schema.py`
 - `tests/test_store_dashboard_mcp.py`
+- `tests/test_store_migrations.py`
 
 ## Tests Run
 
@@ -126,27 +137,32 @@ Full branch closeout should also run the release validation listed in `docs/deve
   - `python -m pytest tests/test_privacy.py -q`
   - `python -m pytest tests/test_store_dashboard_mcp.py -q`
   - `python scripts/check_release.py`
+- M3 persisted call-origin metadata:
+  - `python -m pytest tests/test_call_origin.py tests/test_parser.py::test_parser_ignores_known_non_token_context_compaction_event tests/test_parser.py::test_parser_persists_call_origin_from_metadata_segments tests/test_store_dashboard_mcp.py::test_dashboard_payload_uses_persisted_call_origin_without_source_scan -q` failed before implementation because the pure classifier API was missing.
+  - `python -m pytest tests/test_call_origin.py tests/test_parser.py::test_parser_ignores_known_non_token_context_compaction_event tests/test_parser.py::test_parser_persists_call_origin_from_metadata_segments tests/test_schema.py tests/test_store_migrations.py::test_init_db_migrates_legacy_aggregate_table_without_data_loss tests/test_store_migrations.py::test_csv_export_keeps_current_columns_after_legacy_migration tests/test_store_dashboard_mcp.py::test_dashboard_payload_uses_persisted_call_origin_without_source_scan -q`
+  - `python -m pytest tests/test_parser.py tests/test_call_origin.py tests/test_store_migrations.py tests/test_privacy.py tests/test_store_dashboard_mcp.py -q`
+  - `python scripts/check_release.py`
 
 ## Benchmarks Run
 
-- None yet. Benchmarks start after implementation milestones change measurable behavior.
+- None yet. M3 removed a source-log scan path and added regression tests; benchmark coverage starts in M8.
 
 ## Known Remaining Slow Paths
 
-- Normal `dashboard_payload` currently runs source-file call-origin annotation.
-- Live `/api/usage` currently calls `dashboard_payload` and inherits that work.
+- Normal `dashboard_payload` no longer runs source-file call-origin annotation.
+- Live `/api/usage` still calls `dashboard_payload`, but after M3 it should not open source JSONL files for call-origin metadata.
 - Context loading still does selected-turn evidence and serialized-evidence work; Milestone 6 must verify whether that can be reduced to one source-file pass.
 - Large-history live dashboard still ships broad payloads before the SQLite-backed API slice work.
 
 ## Privacy Notes
 
 - Milestone 0 made no product behavior changes.
 - The branch must keep all test data synthetic and must not persist raw transcript content.
-- Persisted call-origin work must store only categorical labels, reasons, and confidence values.
+- Persisted call-origin stores only categorical labels, reasons, and confidence values. Parser tests and privacy tests cover this with synthetic secret-bearing message/tool/compaction payloads.
 
 ## Merge Blockers
 
-- `dashboard_payload` and `/api/usage` must stop opening source JSONL files.
+- `dashboard_payload` and `/api/usage` must stop opening source JSONL files. M3 covers the call-origin path; future milestones must preserve that invariant as APIs are split.
 - The call investigator asset must be syntax-checked in CI and release validation.
 - Raw call/thread anchors are removed; keep regression tests proving `call_anchors` and `thread_anchors` stay out of context payloads.
 - Focused privacy tests must prove no raw prompts, assistant messages, tool output, replacement history, or raw JSONL fragments are persisted by default.

docs/privacy.md

@@ -10,6 +10,7 @@ The local SQLite database is stored at `~/.codex-usage-tracker/usage.sqlite3` by
 - model, reasoning effort, context window
 - token counts and derived efficiency ratios
 - subagent source, role, nickname, parent session id, and parent thread name when present
+- call-origin category, reason, and confidence labels derived from event metadata during indexing
 - pricing, credit, allowance, recommendation, and project metadata derived from aggregate fields
 
 ## Not Stored
@@ -25,6 +26,8 @@ The parser intentionally does not store:
 
 Those fields are not written to SQLite, CSV exports, generated dashboard HTML, or synthetic screenshots.
 
+Call-origin metadata is heuristic and confidence-labeled. It stores categories such as `user`, `codex`, or `unknown` plus a reason such as `user_message`, `tool_result`, `post_compaction`, or `agent_continuation`. It does not store the message text, tool output, compaction replacement text, or raw JSONL fragment that produced the category.
+
 ## On-Demand Context
 
 `usage_call_context`, `codex-usage-tracker context`, and the `serve-dashboard` context endpoint read a single source JSONL file only when explicitly requested. Returned context is redacted for common secret patterns and capped in size by default for CLI/MCP requests. The call investigator uses the same endpoint at runtime and requests full redacted evidence for the selected call when the local context API is enabled; that still does not persist raw context into SQLite, CSV, support bundles, or generated dashboard HTML.

src/codex_usage_tracker/call_origin.py

@@ -2,93 +2,35 @@
 
 from __future__ import annotations
 
-import json
-from collections import defaultdict
+from collections.abc import Iterable, Mapping
 from dataclasses import dataclass
-from pathlib import Path
 from typing import Any
 
 
 @dataclass(frozen=True)
-class _EventFlags:
+class CallOriginFlags:
+    """Metadata-only signals observed before one token_count callback."""
+
     user_message: bool = False
     compaction: bool = False
     tool_result: bool = False
     codex_activity: bool = False
 
+    @property
+    def has_signal(self) -> bool:
+        return (
+            self.user_message
+            or self.compaction
+            or self.tool_result
+            or self.codex_activity
+        )
 
-def annotate_rows_with_call_origin(rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
-    """Annotate dashboard rows with derived call-level initiator metadata.
 
-    The persisted ``thread_source`` field is session-level. A normal user-created
-    thread can still contain many Codex-initiated model calls after tool results,
-    agent continuations, or compactions. This helper reads only source JSONL event
-    metadata around token-count lines. It does not copy prompt, assistant, or tool
-    text into the returned rows.
-    """
+def event_flags_from_envelope(envelope: object) -> CallOriginFlags:
+    """Return categorical call-origin flags without reading raw text fields."""
 
-    annotated = [dict(row) for row in rows]
-    rows_by_file: dict[str, dict[int, list[dict[str, Any]]]] = defaultdict(
-        lambda: defaultdict(list)
-    )
-    for row in annotated:
-        source_file = row.get("source_file")
-        line_number = _positive_int(row.get("line_number"))
-        if isinstance(source_file, str) and source_file and line_number is not None:
-            rows_by_file[source_file][line_number].append(row)
-        else:
-            row.update(_fallback_origin(row, reason="missing_source"))
-
-    for source_file, rows_by_line in rows_by_file.items():
-        annotations = _classify_source_file(Path(source_file), set(rows_by_line))
-        for line_number, line_rows in rows_by_line.items():
-            annotation = annotations.get(line_number)
-            for row in line_rows:
-                row.update(annotation or _fallback_origin(row, reason="source_unavailable"))
-    return annotated
-
-
-def _classify_source_file(path: Path, target_lines: set[int]) -> dict[int, dict[str, str]]:
-    if not target_lines or not path.exists():
-        return {}
-    max_line = max(target_lines)
-    annotations: dict[int, dict[str, str]] = {}
-    segment: list[_EventFlags] = []
-    try:
-        with path.open(encoding="utf-8") as handle:
-            for line_number, line in enumerate(handle, start=1):
-                if line_number > max_line:
-                    break
-                try:
-                    envelope = json.loads(line)
-                except json.JSONDecodeError:
-                    continue
-                if _is_token_count(envelope):
-                    if line_number in target_lines:
-                        annotations[line_number] = _classify_segment(segment)
-                    segment = []
-                    continue
-                segment.append(_event_flags(envelope))
-    except OSError:
-        return {}
-    return annotations
-
-
-def _classify_segment(segment: list[_EventFlags]) -> dict[str, str]:
-    if any(event.user_message for event in segment):
-        return _origin("user", "user_message", "high")
-    if any(event.compaction for event in segment):
-        return _origin("codex", "post_compaction", "high")
-    if any(event.tool_result for event in segment):
-        return _origin("codex", "tool_result", "high")
-    if any(event.codex_activity for event in segment):
-        return _origin("codex", "agent_continuation", "medium")
-    return _origin("unknown", "no_signal", "low")
-
-
-def _event_flags(envelope: object) -> _EventFlags:
     if not isinstance(envelope, dict):
-        return _EventFlags()
+        return CallOriginFlags()
     payload = envelope.get("payload")
     if not isinstance(payload, dict):
         payload = {}
@@ -116,34 +58,57 @@ def _event_flags(envelope: object) -> _EventFlags:
         and payload_type in {"message", "reasoning", "function_call", "tool_search_call"}
         and role != "user"
     )
-    return _EventFlags(
+    return CallOriginFlags(
         user_message=user_message,
         compaction=compaction,
         tool_result=tool_result,
         codex_activity=codex_activity,
     )
 
 
-def _is_token_count(envelope: object) -> bool:
-    if not isinstance(envelope, dict):
-        return False
-    payload = envelope.get("payload")
-    return (
-        envelope.get("type") == "event_msg"
-        and isinstance(payload, dict)
-        and payload.get("type") == "token_count"
-    )
+def classify_call_origin(segment: Iterable[CallOriginFlags]) -> dict[str, str]:
+    """Classify who most likely initiated a model call from metadata-only signals."""
+
+    flags = list(segment)
+    if any(event.user_message for event in flags):
+        return _origin("user", "user_message", "high")
+    if any(event.compaction for event in flags):
+        return _origin("codex", "post_compaction", "high")
+    if any(event.tool_result for event in flags):
+        return _origin("codex", "tool_result", "high")
+    if any(event.codex_activity for event in flags):
+        return _origin("codex", "agent_continuation", "medium")
+    return _origin("unknown", "no_signal", "low")
+
 
+def fallback_call_origin(row: Mapping[str, Any]) -> dict[str, str]:
+    """Return cheap categorical origin for migrated rows missing persisted metadata."""
 
-def _fallback_origin(row: dict[str, Any], *, reason: str) -> dict[str, str]:
     if (
         row.get("model") == "codex-auto-review"
         or row.get("thread_source") == "subagent"
         or row.get("subagent_type")
         or row.get("parent_session_id")
     ):
         return _origin("codex", "thread_source", "medium")
-    return _origin("unknown", reason, "low")
+    return _origin("unknown", "missing_origin", "low")
+
+
+def ensure_call_origin(row: Mapping[str, Any]) -> dict[str, Any]:
+    """Copy a row and fill missing persisted origin fields without source-log reads."""
+
+    copied = dict(row)
+    if (
+        isinstance(copied.get("call_initiator"), str)
+        and copied["call_initiator"]
+        and isinstance(copied.get("call_initiator_reason"), str)
+        and copied["call_initiator_reason"]
+        and isinstance(copied.get("call_initiator_confidence"), str)
+        and copied["call_initiator_confidence"]
+    ):
+        return copied
+    copied.update(fallback_call_origin(copied))
+    return copied
 
 
 def _origin(initiator: str, reason: str, confidence: str) -> dict[str, str]:
@@ -152,11 +117,3 @@ def _origin(initiator: str, reason: str, confidence: str) -> dict[str, str]:
         "call_initiator_reason": reason,
         "call_initiator_confidence": confidence,
     }
-
-
-def _positive_int(value: object) -> int | None:
-    try:
-        parsed = int(value)  # type: ignore[arg-type]
-    except (TypeError, ValueError):
-        return None
-    return parsed if parsed > 0 else None

src/codex_usage_tracker/dashboard.py

@@ -17,7 +17,7 @@
     load_allowance_config,
     summarize_allowance_usage,
 )
-from codex_usage_tracker.call_origin import annotate_rows_with_call_origin
+from codex_usage_tracker.call_origin import ensure_call_origin
 from codex_usage_tracker.i18n import dashboard_i18n_payload, language_direction, translations_for
 from codex_usage_tracker.paths import (
     DEFAULT_ALLOWANCE_PATH,
@@ -68,15 +68,16 @@ def dashboard_payload(
     privacy_mode = validate_privacy_mode(privacy_mode)
     normalized_offset = _normalize_offset(offset)
     rows = annotate_thread_attachments(
-        annotate_rows_with_call_origin(
-            query_dashboard_events(
+        [
+            ensure_call_origin(row)
+            for row in query_dashboard_events(
                 db_path=db_path,
                 limit=limit,
                 offset=normalized_offset,
                 since=since,
                 include_archived=include_archived,
             )
-        )
+        ]
     )
     pricing = load_pricing_config(pricing_path)
     allowance = load_allowance_config(allowance_path, rate_card_path=rate_card_path)

src/codex_usage_tracker/models.py

@@ -32,6 +32,9 @@ class UsageEvent:
     effort: str | None
     current_date: str | None
     timezone: str | None
+    call_initiator: str | None
+    call_initiator_reason: str | None
+    call_initiator_confidence: str | None
     thread_source: str | None
     subagent_type: str | None
     agent_role: str | None