Add rolling benchmark history tracking

.github/workflows/benchmark-history.yml

@@ -0,0 +1,50 @@
+name: Benchmark history
+
+on:
+  schedule:
+    - cron: "0 12 * * 1"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+concurrency:
+  group: benchmark-history-${{ github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  update-history:
+    name: Update benchmark history
+    runs-on: ubuntu-latest
+    timeout-minutes: 60
+    env:
+      HUSKY: 0
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Setup Bun
+        uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: 1.3.10
+
+      - name: Install dependencies
+        run: bun install --frozen-lockfile
+
+      - name: Refresh rolling benchmark history
+        run: bun run benchmark:history
+
+      - name: Commit history updates
+        run: |
+          if git diff --quiet -- benchmarks/history reports/known-ai-vs-solid-oss-history.md; then
+            echo "No history changes to commit"
+            exit 0
+          fi
+
+          git config user.name "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git add benchmarks/history reports/known-ai-vs-solid-oss-history.md
+          git commit -m "Update benchmark history"
+          git push

AGENTS.md

@@ -36,6 +36,40 @@
 - Output formats: `src/reporters/*`
 - Current language scope: `src/languages/javascript-like.ts`
 
+## Rolling benchmark history
+
+The repo now has **two benchmark modes**:
+
+1. **Pinned benchmark** for reproducible claims
+   - manifest: `benchmarks/sets/known-ai-vs-solid-oss.json`
+   - snapshot: `benchmarks/results/known-ai-vs-solid-oss.json`
+   - report: `reports/known-ai-vs-solid-oss-benchmark.md`
+2. **Rolling history** for default-branch-over-time trend tracking
+   - runner: `scripts/benchmark-history.ts`
+   - history logic: `src/benchmarks/history.ts`
+   - latest-ref parsing: `src/benchmarks/latest-ref.ts`
+   - report rendering: `src/benchmarks/history-report.ts`
+   - per-repo JSONL: `benchmarks/history/known-ai-vs-solid-oss/*.jsonl`
+   - aggregate summary: `benchmarks/history/known-ai-vs-solid-oss/latest.json`
+   - generated report: `reports/known-ai-vs-solid-oss-history.md`
+
+Rolling-history flow:
+
+- resolve each repo's current default branch with `git ls-remote --symref <url> HEAD`
+- resolve the commit that existed on that branch at the run's `recordedAt` timestamp
+- checkout that commit into `benchmarks/.cache/checkouts-history/<set-id>/`
+- analyze with the default registry and default config
+- write one history point per repo per **UTC week**
+- replace the same week's point on rerun instead of appending duplicates
+- use `bun run benchmark:history --recorded-at <iso>` to backfill prior weekly points honestly
+- backfills may skip newer repos for older weeks if the repo had no commit on its current default branch yet
+- compute two blended scores for each point:
+  - `vsCurrentCohort` for same-run relative ranking
+  - `vsPinnedBaseline` for cleaner long-term trend lines
+- regenerate `latest.json` and the markdown history report from the JSONL files
+
+When editing history logic, preserve the separation between pinned and rolling artifacts. Latest-ref scans must not overwrite the pinned snapshot/report used for reproducible benchmark claims.
+
 ## Working rules
 
 - Preserve determinism, stable ordering, and explainable evidence.
@@ -57,7 +91,5 @@
 - Stable self-scan runs the last published package, so newer config features may lag there; use the committed baseline in `tests/fixtures/self-scan-stable-baseline.json` as the source of truth for accepted stable-release behavior.
 - If rule behavior changes, update focused tests and `tests/fixtures-regression.test.ts`.
 - If stable self-scan regressions are intentional, refresh `tests/fixtures/self-scan-stable-baseline.json` with `bun run lint:self:update`.
-- If benchmark-facing behavior changes materially, rerun `bun run benchmark:update` intentionally.
-
-terially, rerun `bun run benchmark:update` intentionally.
-, rerun `bun run benchmark:update` intentionally.
+- If pinned benchmark-facing behavior changes materially, rerun `bun run benchmark:update` intentionally.
+- If rolling benchmark-history behavior changes materially, rerun `bun run benchmark:history` intentionally.

README.md

@@ -141,20 +141,7 @@ Current checks focus on patterns that often show up in unreviewed generated code
 - [placeholder comments](src/rules/placeholder-comments/README.md)
 - [duplicated test mock/setup patterns](src/rules/duplicate-mock-setup/README.md)
 
-## What you get back
-
-- raw repo score
-- normalized metrics:
-  - score / file
-  - score / KLOC
-  - score / function
-  - findings / file
-  - findings / KLOC
-  - findings / function
-- top file hotspots
-- top directory hotspots
-- grouped lint-style findings with `--lint`
-- full-fidelity findings with evidence in `--json`
+`scan` reports raw + normalized scores, hotspot tables, and grouped findings. Use `--json` when you want the full evidence payload.
 
 ## Supported files
 
@@ -187,36 +174,48 @@ _Why before Jan 1, 2025?_ Because this cutoff aims to catch mature OSS before AI
 | Findings / KLOC     |  **2.96** |          **1.38** | **2.14x** |
 | Findings / function |  **0.08** |          **0.04** | **2.21x** |
 
-### Pinned benchmark snapshot
-
-Ordered by blended score.
-
-| Repository                                                            | Cohort     | Ref       |  Blended | Score/file | Score/KLOC | Findings/file | Findings/KLOC |
-| --------------------------------------------------------------------- | ---------- | --------- | -------: | ---------: | ---------: | ------------: | ------------: |
-| [`garrytan/gstack`](https://github.com/garrytan/gstack)               | ai         | `6cc094c` | **5.33** |       2.34 |      21.71 |          0.52 |          4.85 |
-| [`redwoodjs/agent-ci`](https://github.com/redwoodjs/agent-ci)         | ai         | `4de00d6` | **3.57** |       0.99 |      10.95 |          0.31 |          3.42 |
-| [`jiayun/DevWorkbench`](https://github.com/jiayun/DevWorkbench)       | ai         | `ea50862` | **3.39** |       1.00 |      10.76 |          0.44 |          4.69 |
-| [`openclaw/openclaw`](https://github.com/openclaw/openclaw)           | ai         | `44cf747` | **3.06** |       1.04 |      10.60 |          0.32 |          3.22 |
-| [`robinebers/openusage`](https://github.com/robinebers/openusage)     | ai         | `857f537` | **3.02** |       1.27 |       7.92 |          0.33 |          2.07 |
-| [`emdash-cms/emdash`](https://github.com/emdash-cms/emdash)           | ai         | `dbaf8c6` | **2.17** |       0.73 |       6.54 |          0.22 |          1.98 |
-| [`FullAgent/fulling`](https://github.com/FullAgent/fulling)           | ai         | `d95060f` | **2.16** |       0.53 |       9.51 |          0.16 |          2.96 |
-| [`cloudflare/vinext`](https://github.com/cloudflare/vinext)           | ai         | `28980b0` | **1.99** |       0.48 |       9.20 |          0.15 |          2.76 |
-| [`withastro/astro`](https://github.com/withastro/astro)               | mature-oss | `f706899` | **1.58** |       0.28 |       6.75 |          0.10 |          2.31 |
-| [`payloadcms/payload`](https://github.com/payloadcms/payload)         | mature-oss | `f3f36d8` | **1.47** |       0.24 |       4.04 |          0.08 |          1.38 |
-| [`vitejs/vite`](https://github.com/vitejs/vite)                       | mature-oss | `a492253` | **1.47** |       0.25 |       8.19 |          0.08 |          2.52 |
-| [`pmndrs/zustand`](https://github.com/pmndrs/zustand)                 | mature-oss | `2e6d881` | **1.45** |       0.47 |       3.20 |          0.19 |          1.27 |
-| [`modem-dev/hunk`](https://github.com/modem-dev/hunk)                 | ai         | `b37663f` | **1.18** |       0.38 |       4.71 |          0.13 |          1.55 |
-| [`umami-software/umami`](https://github.com/umami-software/umami)     | mature-oss | `227b255` | **1.00** |       0.17 |       4.36 |          0.07 |          1.66 |
-| [`egoist/tsup`](https://github.com/egoist/tsup)                       | mature-oss | `cd03e1e` | **0.95** |       0.22 |       3.83 |          0.09 |          1.50 |
-| [`sindresorhus/execa`](https://github.com/sindresorhus/execa)         | mature-oss | `99d1741` | **0.89** |       0.17 |       4.86 |          0.05 |          1.37 |
-| [`mikaelbr/node-notifier`](https://github.com/mikaelbr/node-notifier) | mature-oss | `b36c237` | **0.41** |       0.08 |       0.90 |          0.04 |          0.47 |
-| [`vercel/hyper`](https://github.com/vercel/hyper)                     | mature-oss | `2a7bb18` | **0.41** |       0.65 |       1.12 |          0.16 |          0.28 |
+### Rolling benchmark snapshot
+
+Latest default-branch history, still normalized against the frozen pinned baseline. Ordered by latest pinned score.
+
+| Repository                                                            | Cohort     | Latest ref       | Current blended | Latest pinned | Highest pinned | Δ prev | Δ peak |
+| --------------------------------------------------------------------- | ---------- | ---------------- | --------------: | ------------: | -------------: | -----: | -----: |
+| [`garrytan/gstack`](https://github.com/garrytan/gstack)               | ai         | `main@c6e6a21`   |        **4.59** |      **4.77** |       **6.37** |  -0.64 |  -1.60 |
+| [`redwoodjs/agent-ci`](https://github.com/redwoodjs/agent-ci)         | ai         | `main@c61f27d`   |        **3.76** |      **3.91** |       **3.91** |  +0.51 |   0.00 |
+| [`jiayun/DevWorkbench`](https://github.com/jiayun/DevWorkbench)       | ai         | `main@ea50862`   |        **3.26** |      **3.39** |       **3.40** |   0.00 |  -0.02 |
+| [`robinebers/openusage`](https://github.com/robinebers/openusage)     | ai         | `main@06113d6`   |        **2.91** |      **3.03** |       **3.06** |  +0.01 |  -0.03 |
+| [`openclaw/openclaw`](https://github.com/openclaw/openclaw)           | ai         | `main@1de5610`   |        **2.81** |      **2.92** |       **3.15** |  -0.23 |  -0.23 |
+| [`FullAgent/fulling`](https://github.com/FullAgent/fulling)           | ai         | `main@d95060f`   |        **2.07** |      **2.16** |       **2.16** |   0.00 |   0.00 |
+| [`emdash-cms/emdash`](https://github.com/emdash-cms/emdash)           | ai         | `main@a1dac00`   |        **1.94** |      **2.01** |       **2.17** |  -0.16 |  -0.16 |
+| [`cloudflare/vinext`](https://github.com/cloudflare/vinext)           | ai         | `main@e81a621`   |        **1.85** |      **1.93** |       **1.99** |  -0.06 |  -0.07 |
+| [`vitejs/vite`](https://github.com/vitejs/vite)                       | mature-oss | `main@bc5c6a7`   |        **1.46** |      **1.52** |       **1.52** |  +0.02 |   0.00 |
+| [`modem-dev/hunk`](https://github.com/modem-dev/hunk)                 | ai         | `main@53242b4`   |        **1.46** |      **1.51** |       **1.51** |  +0.44 |   0.00 |
+| [`withastro/astro`](https://github.com/withastro/astro)               | mature-oss | `main@7fe40bc`   |        **1.40** |      **1.46** |       **1.55** |   0.00 |  -0.09 |
+| [`pmndrs/zustand`](https://github.com/pmndrs/zustand)                 | mature-oss | `main@00f96a3`   |        **1.33** |      **1.38** |       **1.38** |   0.00 |  -0.01 |
+| [`payloadcms/payload`](https://github.com/payloadcms/payload)         | mature-oss | `main@5afcef5`   |        **1.29** |      **1.34** |       **1.34** |  +0.02 |   0.00 |
+| [`umami-software/umami`](https://github.com/umami-software/umami)     | mature-oss | `master@3a31ad3` |        **1.00** |      **1.04** |       **1.04** |  +0.00 |   0.00 |
+| [`egoist/tsup`](https://github.com/egoist/tsup)                       | mature-oss | `main@b906f86`   |        **0.89** |      **0.92** |       **0.92** |   0.00 |   0.00 |
+| [`sindresorhus/execa`](https://github.com/sindresorhus/execa)         | mature-oss | `main@f3a2e84`   |        **0.85** |      **0.89** |       **0.89** |   0.00 |   0.00 |
+| [`mikaelbr/node-notifier`](https://github.com/mikaelbr/node-notifier) | mature-oss | `master@b36c237` |        **0.40** |      **0.41** |       **0.41** |   0.00 |   0.00 |
+| [`vercel/hyper`](https://github.com/vercel/hyper)                     | mature-oss | `canary@2a7bb18` |        **0.40** |      **0.41** |       **0.41** |   0.00 |   0.00 |
+
+Legend:
+
+- `Current blended` = latest repo score vs the current mature-OSS medians from the same rolling run
+- `Latest pinned` = latest repo score vs the frozen pinned mature-OSS baseline snapshot
+- `Highest pinned` = highest stored repo score on that same pinned baseline
+- `Δ prev` = latest pinned - previous week's pinned score
+- `Δ peak` = latest pinned - highest pinned score, so more negative means the repo is below its own historical high
+
+For exact pinned SHAs and the full per-metric breakdowns, see the saved snapshot and pinned benchmark report.
 
 Full benchmark assets:
 
 - manifest: [`benchmarks/sets/known-ai-vs-solid-oss.json`](benchmarks/sets/known-ai-vs-solid-oss.json)
-- snapshot: [`benchmarks/results/known-ai-vs-solid-oss.json`](benchmarks/results/known-ai-vs-solid-oss.json)
-- report: [`reports/known-ai-vs-solid-oss-benchmark.md`](reports/known-ai-vs-solid-oss-benchmark.md)
+- pinned snapshot: [`benchmarks/results/known-ai-vs-solid-oss.json`](benchmarks/results/known-ai-vs-solid-oss.json)
+- pinned report: [`reports/known-ai-vs-solid-oss-benchmark.md`](reports/known-ai-vs-solid-oss-benchmark.md)
+- rolling latest summary: [`benchmarks/history/known-ai-vs-solid-oss/latest.json`](benchmarks/history/known-ai-vs-solid-oss/latest.json)
+- rolling history report: [`reports/known-ai-vs-solid-oss-history.md`](reports/known-ai-vs-solid-oss-history.md)
 
 ## Configuration
 
@@ -273,23 +272,13 @@ See also:
 
 This repo also commits a root [`slop-scan.config.json`](slop-scan.config.json) for self-scans and local development. It keeps the scan focused on the tool itself by excluding heavyweight benchmark checkouts and intentionally disables directory-structure rules under `src/rules/**`.
 
-## How it works
-
-`slop-scan` is built as a pluggable engine:
-
-- language plugins
-- fact providers
-- rule plugins
-- reporters
-
-That keeps the analyzer deterministic and extensible without turning it into one giant loop of ad hoc checks.
-
 ## Docs
 
 - plugin guide: [`docs/plugins.md`](docs/plugins.md)
 - built-in rule docs: browse [`src/rules/`](src/rules)
 - benchmark guide: [`benchmarks/README.md`](benchmarks/README.md)
 - pinned benchmark report: [`reports/known-ai-vs-solid-oss-benchmark.md`](reports/known-ai-vs-solid-oss-benchmark.md)
+- rolling benchmark history: [`reports/known-ai-vs-solid-oss-history.md`](reports/known-ai-vs-solid-oss-history.md)
 - exploratory note on non-JS/TS candidates: [`reports/exploratory-vite-astro-openclaw-beads.md`](reports/exploratory-vite-astro-openclaw-beads.md)
 - contributing guide: [`CONTRIBUTING.md`](CONTRIBUTING.md)
 

benchmarks/README.md

@@ -53,14 +53,46 @@ Or do all three:
 bun run benchmark:update
 ```
 
+## Rolling history
+
+A separate rolling-history pipeline tracks the same repos at the **default-branch revision that existed at each recorded run time**.
+
+Refresh it locally with:
+
+```bash
+bun run benchmark:history
+```
+
+Backfill an earlier weekly point with:
+
+```bash
+bun run benchmark:history --recorded-at 2026-04-06T12:00:00Z
+```
+
+That writes:
+
+- per-repo JSONL histories under `benchmarks/history/known-ai-vs-solid-oss/*.jsonl`
+- a latest aggregate summary at `benchmarks/history/known-ai-vs-solid-oss/latest.json`
+- a generated markdown summary at `reports/known-ai-vs-solid-oss-history.md`
+
+If a repo did not exist yet for an older backfill date, that weekly point is skipped instead of fabricating a datapoint.
+
+The rolling history is intentionally separate from the pinned benchmark snapshot so reproducible benchmark claims still point at exact SHAs.
+
 ## Artifacts
 
-For the current set:
+For the current pinned set:
 
 - manifest: `benchmarks/sets/known-ai-vs-solid-oss.json`
 - saved snapshot: `benchmarks/results/known-ai-vs-solid-oss.json`
 - generated report: `reports/known-ai-vs-solid-oss-benchmark.md`
 
+For rolling history:
+
+- per-repo JSONL: `benchmarks/history/known-ai-vs-solid-oss/*.jsonl`
+- latest summary: `benchmarks/history/known-ai-vs-solid-oss/latest.json`
+- generated history report: `reports/known-ai-vs-solid-oss-history.md`
+
 ## Notes
 
 - Checkouts are stored under `benchmarks/.cache/` and are gitignored.