Skip to content

work-tokens derives landings from swept traces, so per-LANDED-item divides two different populations — read landed-history.jsonl instead #332

Description

@thedavidmeister

landed-history.jsonl landed in #277 (merged 6be7010) and nothing reads it. work-tokens still derives landings from traces, which is the reason its headline figures cannot be read.

What work-tokens reports today

landed                      1 task   1 item    $2.28
delivered-awaiting-human    3 tasks  3 items   $9.22
churn                      49 tasks  4 items $104.84
main loop (11 runs)                           $38.41
output (whole-run)                            $71.70
TOTAL                      53 tasks          $226.46

per LANDED item        $226.46
per DELIVERED item      $56.61

per LANDED item is TOTAL ÷ 1. per DELIVERED item is TOTAL ÷ 4. The landed item's own cost was $2.28.

Why the ratio cannot be read

Both figures divide spend from one population by landings from a different one.

A run's spend enters the corpus when the run happens. Its item lands later — hours or days later, after human review. But a run only enters this corpus at all if its trace survives, and the tool's own skip lines say what that costs:

21 metrics rows skipped: the record names no trace
218 metrics rows skipped: the trace has been collected
42 metrics rows skipped: the run dispatched no subagent and recorded no work item

281 of 495 rows excluded. With KEEP_RUNS=20 (raised to 2000 on 2026-08-17) the surviving window was ~21 hours against a landing latency measured in days — so a landing's own run had usually aged out before the landing happened. The numerator and denominator were never drawn from the same set.

The consequence is not a wrong number, it is an unattributable one: the same landed item is charged against different spend depending when you ask, and neither figure is a property of any item. The tool already warns about the shape — "This is a snapshot of that corpus, not a trend — do not read it as one" — but the per-item lines invite exactly that reading, and got it.

What the feed changes

landed-history.jsonl records one line per landed unit, keyed on GitHub's own terminal timestamp:

{"ts":"2026-07-20T19:41:43Z","observedAt":"2026-07-22T13:31:08+00:00","kind":"pr","repo":"rainlanguage/rain.solmem","number":51}
  • ts is mergedAt/closedAt — GitHub's, verified byte-identical on spot-check.
  • It is append-only and retention-independent: a landing is recorded when it happens and stays, whatever became of the producing run's trace.
  • It is already seeded with 195 rows (185 PRs, 10 issues), ts 2026-07-13 → 2026-08-12, 0 duplicate keys.
  • An issue closed by a merged PR yields ONE row — the PR's — so the feed does not double-count a unit of work.

So the denominator now exists as a durable population that can be windowed on the merge date.

Proposed change

Take the landed count from landed-history.jsonl rather than from trace-derived work items, and window numerator and denominator on the same period — spend from runs in the window, landings whose ts falls in it.

Two things to settle while implementing, both of which decide whether the result means anything:

  • The window is the unit, not the corpus. "Total spend ever ÷ landings ever" is a coherent lifetime average; "spend in a truncated trace corpus ÷ landings in it" is what is broken now. Whichever is chosen, the report should name the window it computed over, so a reader cannot mistake one for the other.
  • Landing latency crosses the window boundary. Work spent in a period lands after it. A period's spend and a period's landings are not the same work, and a ratio of them is only meaningful once the series is long enough for that lag to wash out. Say so in the output rather than leaving a reader to discover it — the same honesty the skip lines already show.

Note work-tokens keeps needing traces for the numerator (per-task spend attribution by parent_tool_use_id). This change is only about where landings come from.

Done when

  • per LANDED item is computed from landed-history.jsonl, not from trace-derived items.
  • The report names the window both sides were computed over.
  • A landing whose producing run's trace no longer exists still counts.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions