Skip to content

fix: replace binary auto-PR with a drift check (#114) - #122

Merged
mogul merged 3 commits into
mainfrom
fix-114-binary-drift-check
Oct 15, 2026
Merged

mogul merged 3 commits into
mainfrom
fix-114-binary-drift-check

Conversation

@rahearn

@rahearn rahearn commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Fixes #114.

Context

The "Keep binary updated" workflow has failed every night since 2026-08-14 — 62 consecutive runs — with:

Error creating blob for file 'proxy/caddy': Sorry, your input was too
large to process.

Root cause: sign-commits: true routes the commit through the GitHub Git API rather than git push, and that API caps blobs at 40 MiB. proxy/caddy is 46.0 MiB. It was ~38 MiB when this automation last worked, so it simply outgrew the ceiling. The failure is in createBlob, which matches the git show-flavored error text noted in the issue discussion.

Why not keep the auto-PR with bot signing? It can't be made to fit:

  • Deploy keys cannot sign commits — they're repo-scoped and authentication-only. Upstream states it plainly: "You cannot use deploy keys with commit signature verification for bots."
  • There is no repository-scoped signing-key API. Signing keys attach only to user accounts (I probed for a repo-level endpoint; 404).
  • So every automated-signing path requires a human's private key sitting in Actions secrets indefinitely, with no expiry management. This repo currently has zero Actions secrets; that seemed a poor trade for a nightly convenience.

Why a check rather than a schedule? Renovate pinned the Dockerfile to base-image digests in 8a559c9 (Oct 14), which makes make deterministic. A nightly rebuild no longer discovers anything on its own — the binary can only drift when a pinned input changes, which is a pull-request condition, not a scheduled one.

Plan

Change Why
keep-binary-updated.yml → drift check on PRs touching Dockerfile, compose.yaml, Makefile The inputs that determine the binary's bytes. No cron.
Drop create-pull-request + sign-commits The 40 MiB ceiling is unavoidable for bot-signed commits.
permissions: contents: write → read The workflow no longer writes.
Makefile: drop - from docker compose cp See below — it was masking failures.
README: document the rebuild-and-commit procedure The check now expects a human in this loop.

The Makefile change is a real bug, not cosmetics

Makefile:7 read - docker compose cp caddy:/usr/bin/caddy proxy/caddy (introduced in 521788f while fixing M1 builds). The leading - makes make ignore a failed copy — so a broken build produced no drift and passed green. That false pass would have silently undermined this entire check. Removing it surfaces the failure (AGENTS.md §14.5).

Tradeoff I'm not hiding: a failed cp now skips the docker compose down on line 8, leaving a container running. Irrelevant on an ephemeral runner, a minor annoyance locally. Guaranteeing teardown is more machinery than the problem warrants, but say so if you'd prefer it.

Verification

Live runs, no mocks (§8.3).

Build reproducibility — the assumption everything rests on. Ran make twice locally; byte-identical both times (3a84bf57…). Also confirmed historically: runs on 2026-08-08 → 08-13 passed with Create PR with changes skipped, meaning git status was clean after make. A diff-based check won't false-positive.

Fail case — run 38761199302:

Expected sha256 after rebuilding: 3a84bf57fa36e219d84fbe20cf91a166969214527a6851f5bed21a20c6484559
Caddy version in fresh build:      v2.11.7
::error::proxy/caddy does not match a fresh build. Rebuild it with 'make' and commit the result.
Process completed with exit code 1

Pass case — run 38762029557, on a scratch branch carrying the rebuilt binary (since deleted):

proxy/caddy is current: a fresh build is byte-identical.

The committed sha256 matched CI's expected value exactly, confirming the reported digest is actionable: a maintainer can trust it to verify their local build.

Path filter + comment step: this PR touches Makefile, so the check should fire on it and post the failure comment. Verified by this PR's own checks.

Static checks: YAML parses; the run block passes bash -n; the github-script body passes node --check; the comment markdown was rendered and eyeballed.

One defect found and fixed mid-verification: the first run (38759707698) wrote the sha256 only to $GITHUB_OUTPUT, so on workflow_dispatch — where there's no PR to comment on — the value was invisible. Fixed in bd14634 by echoing to the job log too.

Expected: this check fails on arrival

proxy/caddy on main is stale. The pinned builder bakes in CADDY_VERSION=v2.11.7; the committed binary is v2.11.4, built 2026-08-06 (d025c6f) — three patch releases behind, dating from right before the automation broke. forwardproxy is unchanged.

That's the check doing its job on its first real input. Resolving it means rebuilding and committing the binary:

make
sha256sum proxy/caddy   # expect 3a84bf57fa36e219d84fbe20cf91a166969214527a6851f5bed21a20c6484559

Worth noting independently: there are open govulncheck alerts against the committed binary, including called-function vulnerabilities in net/http, crypto/tls, and golang.org/x/net/http2. A v2.11.7 rebuild on Go 1.27.2 (vs. the committed 1.26.5) likely clears a number of them. I'd treat the rebuild as worth doing promptly rather than whenever convenient.

Rollback

git revert these commits. Three files, no state outside the repo.

Security impact

Net reduction:

  • Drops contents: write from the workflow.
  • Adds no secrets, and avoids putting a long-lived signing key in Actions.
  • Every binary-bearing commit stays signed by an accountable human (main requires signed commits).
  • Removes a third-party action from a write-privileged position.
  • Fixes a check that could pass while the build was broken.

vulnscan.yml still runs govulncheck against the committed binary daily, so a stale binary remains independently detectable.

Reviewer focus

  1. The path list (Dockerfile, compose.yaml, Makefile) — I excluded proxy/Caddyfile: it's a make prerequisite but doesn't affect the binary's bytes. Please sanity-check that.
  2. Renovate interaction — digest-bump PRs will now fail this check until a human rebuilds. Intended, but it's a workflow change for you. I deliberately did not add this to the required status checks, so you can watch it on the next Renovate bump before it can block a merge.
  3. The - removal tradeoff noted above.

Not addressed here, tracked separately: #121 (ubuntu-latest → Ubuntu 26 migration, starts Oct 19).


🤖 Written by an AI agent. Requires human review before merge; I'm not in a position to vouch for my own output.

The "Keep binary updated" workflow has failed every night since
2026-08-14 (62 consecutive runs) with:

  Error creating blob for file 'proxy/caddy': Sorry, your input was
  too large to process.

Root cause: `sign-commits: true` routes the commit through the GitHub
Git API rather than `git push`, and that API caps blobs at 40 MiB.
proxy/caddy is 46.0 MiB; it was ~38 MiB when this automation last
worked, so it simply outgrew the ceiling.

Bot-signed commits cannot be made to fit. Deploy keys are
authentication-only and cannot sign (create-pull-request documents
this explicitly), and there is no repository-scoped signing-key API --
signing keys attach only to user accounts. Every automated-signing
path therefore requires a human's private key living in Actions
secrets indefinitely, which is a poor trade for a nightly convenience.

Renovate also pinned the Dockerfile to base-image digests in 8a559c9,
which makes `make` deterministic. A nightly rebuild no longer
discovers anything on its own; the binary can only drift when a
pinned input changes. That is a pull-request condition, not a
scheduled one.

So the workflow becomes a check instead of an author: it rebuilds on
PRs touching Dockerfile, compose.yaml, or Makefile and fails if the
committed binary doesn't match, reporting the expected sha256 and
caddy version so a maintainer can confirm their local build
reproduces before committing it signed.

Also drops the `-` prefix from the `docker compose cp` recipe in the
Makefile. It caused a failed copy to be ignored, so a broken build
would report "no drift" and pass green -- a false pass that would
have undermined this check.

Note: proxy/caddy on main is currently stale. The pinned builder
bakes in caddy v2.11.7 while the committed binary is v2.11.4, so
this check will flag it until someone rebuilds and commits.
The first live verification run surfaced a gap: on workflow_dispatch
there is no pull request to comment on, so the expected sha256 and
caddy version were written only to $GITHUB_OUTPUT and never shown
anywhere a human could read them. Echo them to the job log too.
@github-actions

Copy link
Copy Markdown
Contributor

The committed proxy/caddy does not match what this branch builds.

Rebuild and commit it:

make
sha256sum proxy/caddy

Expected after rebuilding:

sha256 3a84bf57fa36e219d84fbe20cf91a166969214527a6851f5bed21a20c6484559
caddy version v2.11.7

If your local sha256 differs from the value above, the build is not
reproducing and something other than the pinned inputs has changed --
investigate before committing.

The binary must be committed by a human because main requires signed
commits, and bot-signed commits cannot carry a file this large (see #114).

@rahearn
rahearn requested a review from a team October 15, 2026 16:17
Comment on lines -11 to +27
schedule:
# Runs every night around 4:20am EST/1:20am PST
- cron: "20 8 * * *"
pull_request:
types:
- opened
- synchronize
- reopened
- ready_for_review
paths:
- Dockerfile
- compose.yaml
- Makefile

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like it only runs in response to pull-requests, but we're not regularly making PRs in this repository... So what is the mechanism that triggers the check on a regular basis?

I think this should still be cron-driven, and I like that it's a staleness drift-check. However, in response to drift, it should open a PR that's unmergeable without an updated binary (implying a separate PR check). The created PR should include instructions for the user to update the binary:

  1. Check out the branch
  2. Run the build command
  3. Make a signed commit
  4. Push to the branch
  5. Wait for PR check to pass
  6. Merge the PR

@mogul mogul left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a dummy, the trigger is the Renovate PR, which already detects the drift and prompts action, and this check is the gating PR check that requires you to commit a binary, and provides instructions.

@mogul
mogul merged commit c64bb26 into main Oct 15, 2026
11 checks passed
@mogul
mogul deleted the fix-114-binary-drift-check branch October 15, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"Keep binary updated" action failing

2 participants