feat(health): expose fsck as kb.fsck on mcp/jsonl/capabilities #342
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # VouchBench season scoring — the GitHub-native retrieval competition. | |
| # | |
| # Two triggers: | |
| # * pull_request: practice scoring on the PUBLIC practice seeds — instant, | |
| # comparable feedback on every entry push. No secrets, no network beyond | |
| # checkout; scores are a pure function of (seed, code). | |
| # * workflow_dispatch: the SCORED run a maintainer triggers after the season | |
| # cutoff. Seeds are supplied at dispatch time (derive them from the drand | |
| # round at the cutoff — the commit-reveal step: entries are frozen before | |
| # the seeds exist, so nobody can pre-fit). All entries and main are scored | |
| # on the SAME seeds; paired comparison builds the margin band. | |
| # | |
| # See .superpowers/VOUCHBENCH-COMPETITION.md for the full season mechanics | |
| # (cutoff rules, payout shares, first-seen tie protection, review gates). | |
| name: vouchbench-season | |
| on: | |
| pull_request: | |
| paths: | |
| - "src/vouch/**" | |
| - "tests/**" | |
| workflow_dispatch: | |
| inputs: | |
| seeds: | |
| description: >- | |
| Comma-separated scored seeds (derive from the drand round at the | |
| season cutoff; document the round number in the season issue). | |
| required: true | |
| budget_chars: | |
| description: Context budget per query. | |
| default: "2000" | |
| env: | |
| PRACTICE_SEEDS: "1,2,3,4,5,6" | |
| jobs: | |
| score: | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 30 | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - uses: actions/setup-python@v5 | |
| with: | |
| python-version: "3.12" | |
| - name: Install vouch | |
| run: | | |
| python -m venv .venv | |
| .venv/bin/pip install -e '.[dev]' | |
| - name: Score | |
| env: | |
| # dispatch inputs pass through env, never interpolated into the | |
| # script body (workflow-injection hygiene); the python entrypoint | |
| # then int()-parses every seed, rejecting anything shell-shaped. | |
| INPUT_SEEDS: ${{ github.event.inputs.seeds }} | |
| INPUT_BUDGET: ${{ github.event.inputs.budget_chars }} | |
| run: | | |
| SEEDS="${INPUT_SEEDS:-$PRACTICE_SEEDS}" | |
| BUDGET="${INPUT_BUDGET:-2000}" | |
| .venv/bin/python -m vouch.cli bench run \ | |
| --seeds "$SEEDS" --budget-chars "$BUDGET" --json \ | |
| | tee bench-report.json | |
| - name: Summarize | |
| run: | | |
| .venv/bin/python - <<'PY' | |
| import json | |
| r = json.load(open("bench-report.json")) | |
| lines = [ | |
| "## VouchBench", | |
| "", | |
| f"composite **{r['composite_mean']:.3f} ± {r['composite_se']:.3f}** " | |
| f"(seeds {r['seeds']})", | |
| "", | |
| "| category | mean |", | |
| "|---|---|", | |
| ] | |
| lines += [f"| {k} | {v:.2f} |" for k, v in r["categories"].items()] | |
| open("summary.md", "w").write("\n".join(lines) + "\n") | |
| PY | |
| cat summary.md >> "$GITHUB_STEP_SUMMARY" | |
| - uses: actions/upload-artifact@v4 | |
| with: | |
| name: bench-report | |
| path: | | |
| bench-report.json | |
| summary.md |