Repository navigation
fix: replace binary auto-PR with a drift check (#114) - #122
Conversation
The "Keep binary updated" workflow has failed every night since 2026-08-14 (62 consecutive runs) with: Error creating blob for file 'proxy/caddy': Sorry, your input was too large to process. Root cause: `sign-commits: true` routes the commit through the GitHub Git API rather than `git push`, and that API caps blobs at 40 MiB. proxy/caddy is 46.0 MiB; it was ~38 MiB when this automation last worked, so it simply outgrew the ceiling. Bot-signed commits cannot be made to fit. Deploy keys are authentication-only and cannot sign (create-pull-request documents this explicitly), and there is no repository-scoped signing-key API -- signing keys attach only to user accounts. Every automated-signing path therefore requires a human's private key living in Actions secrets indefinitely, which is a poor trade for a nightly convenience. Renovate also pinned the Dockerfile to base-image digests in 8a559c9, which makes `make` deterministic. A nightly rebuild no longer discovers anything on its own; the binary can only drift when a pinned input changes. That is a pull-request condition, not a scheduled one. So the workflow becomes a check instead of an author: it rebuilds on PRs touching Dockerfile, compose.yaml, or Makefile and fails if the committed binary doesn't match, reporting the expected sha256 and caddy version so a maintainer can confirm their local build reproduces before committing it signed. Also drops the `-` prefix from the `docker compose cp` recipe in the Makefile. It caused a failed copy to be ignored, so a broken build would report "no drift" and pass green -- a false pass that would have undermined this check. Note: proxy/caddy on main is currently stale. The pinned builder bakes in caddy v2.11.7 while the committed binary is v2.11.4, so this check will flag it until someone rebuilds and commits.
The first live verification run surfaced a gap: on workflow_dispatch there is no pull request to comment on, so the expected sha256 and caddy version were written only to $GITHUB_OUTPUT and never shown anywhere a human could read them. Echo them to the job log too.
|
The committed Rebuild and commit it: make
sha256sum proxy/caddyExpected after rebuilding:
If your local sha256 differs from the value above, the build is not The binary must be committed by a human because |
| schedule: | ||
| # Runs every night around 4:20am EST/1:20am PST | ||
| - cron: "20 8 * * *" | ||
| pull_request: | ||
| types: | ||
| - opened | ||
| - synchronize | ||
| - reopened | ||
| - ready_for_review | ||
| paths: | ||
| - Dockerfile | ||
| - compose.yaml | ||
| - Makefile |
There was a problem hiding this comment.
This looks like it only runs in response to pull-requests, but we're not regularly making PRs in this repository... So what is the mechanism that triggers the check on a regular basis?
I think this should still be cron-driven, and I like that it's a staleness drift-check. However, in response to drift, it should open a PR that's unmergeable without an updated binary (implying a separate PR check). The created PR should include instructions for the user to update the binary:
- Check out the branch
- Run the build command
- Make a signed commit
- Push to the branch
- Wait for PR check to pass
- Merge the PR
mogul
left a comment
There was a problem hiding this comment.
I'm a dummy, the trigger is the Renovate PR, which already detects the drift and prompts action, and this check is the gating PR check that requires you to commit a binary, and provides instructions.
Fixes #114.
Context
The "Keep binary updated" workflow has failed every night since 2026-08-14 — 62 consecutive runs — with:
Root cause:
sign-commits: trueroutes the commit through the GitHub Git API rather thangit push, and that API caps blobs at 40 MiB.proxy/caddyis 46.0 MiB. It was ~38 MiB when this automation last worked, so it simply outgrew the ceiling. The failure is increateBlob, which matches thegit show-flavored error text noted in the issue discussion.Why not keep the auto-PR with bot signing? It can't be made to fit:
Why a check rather than a schedule? Renovate pinned the Dockerfile to base-image digests in 8a559c9 (Oct 14), which makes
makedeterministic. A nightly rebuild no longer discovers anything on its own — the binary can only drift when a pinned input changes, which is a pull-request condition, not a scheduled one.Plan
keep-binary-updated.yml→ drift check on PRs touchingDockerfile,compose.yaml,Makefilecreate-pull-request+sign-commitspermissions: contents: write→readMakefile: drop-fromdocker compose cpThe Makefile change is a real bug, not cosmetics
Makefile:7read- docker compose cp caddy:/usr/bin/caddy proxy/caddy(introduced in 521788f while fixing M1 builds). The leading-makesmakeignore a failed copy — so a broken build produced no drift and passed green. That false pass would have silently undermined this entire check. Removing it surfaces the failure (AGENTS.md §14.5).Tradeoff I'm not hiding: a failed
cpnow skips thedocker compose downon line 8, leaving a container running. Irrelevant on an ephemeral runner, a minor annoyance locally. Guaranteeing teardown is more machinery than the problem warrants, but say so if you'd prefer it.Verification
Live runs, no mocks (§8.3).
Build reproducibility — the assumption everything rests on. Ran
maketwice locally; byte-identical both times (3a84bf57…). Also confirmed historically: runs on 2026-08-08 → 08-13 passed withCreate PR with changesskipped, meaninggit statuswas clean aftermake. A diff-based check won't false-positive.Fail case — run 38761199302:
Pass case — run 38762029557, on a scratch branch carrying the rebuilt binary (since deleted):
The committed sha256 matched CI's expected value exactly, confirming the reported digest is actionable: a maintainer can trust it to verify their local build.
Path filter + comment step: this PR touches
Makefile, so the check should fire on it and post the failure comment. Verified by this PR's own checks.Static checks: YAML parses; the
runblock passesbash -n; thegithub-scriptbody passesnode --check; the comment markdown was rendered and eyeballed.One defect found and fixed mid-verification: the first run (38759707698) wrote the sha256 only to
$GITHUB_OUTPUT, so onworkflow_dispatch— where there's no PR to comment on — the value was invisible. Fixed in bd14634 by echoing to the job log too.Expected: this check fails on arrival
proxy/caddyonmainis stale. The pinned builder bakes inCADDY_VERSION=v2.11.7; the committed binary is v2.11.4, built 2026-08-06 (d025c6f) — three patch releases behind, dating from right before the automation broke.forwardproxyis unchanged.That's the check doing its job on its first real input. Resolving it means rebuilding and committing the binary:
make sha256sum proxy/caddy # expect 3a84bf57fa36e219d84fbe20cf91a166969214527a6851f5bed21a20c6484559Worth noting independently: there are open
govulncheckalerts against the committed binary, including called-function vulnerabilities innet/http,crypto/tls, andgolang.org/x/net/http2. A v2.11.7 rebuild on Go 1.27.2 (vs. the committed 1.26.5) likely clears a number of them. I'd treat the rebuild as worth doing promptly rather than whenever convenient.Rollback
git revertthese commits. Three files, no state outside the repo.Security impact
Net reduction:
contents: writefrom the workflow.mainrequires signed commits).vulnscan.ymlstill runsgovulncheckagainst the committed binary daily, so a stale binary remains independently detectable.Reviewer focus
Dockerfile,compose.yaml,Makefile) — I excludedproxy/Caddyfile: it's amakeprerequisite but doesn't affect the binary's bytes. Please sanity-check that.-removal tradeoff noted above.Not addressed here, tracked separately: #121 (
ubuntu-latest→ Ubuntu 26 migration, starts Oct 19).🤖 Written by an AI agent. Requires human review before merge; I'm not in a position to vouch for my own output.