Daily bare-repack skipped 2 days running: one reflog-orphaned commit permanently gates the pre-fill walk, and gc.auto=0 means nothing ever expires it

[groeneai]

The fleet's shared bare clone /worktrees/.master.git is a tree:0 promisor clone with gc.auto=0, so every lazy fetch writes its own pack and nothing consolidates. The daily groeneai-bare-repack.timer is the only thing keeping it bounded, and it does work: 509 -> 15 (09-19), 729 -> 2 (09-20), 829 -> 2 (09-21), 795 -> 2 (09-22).

It has now been skipped on two consecutive firings.

What happens

[2026-09-23T05:06:44Z] prefill walk: rc=128 duration=2016s other_lines=1
  prefill: fatal: Failed to traverse parents of commit 1ec7ca1cb36c9a807d75bdb06b88fc2f2b03c4b6
[2026-09-23T05:06:48Z] WARNING: prefill walk failed (rc=128) - skipping the repack: it would die on
                       the same object. Fix the object/worktree named above, then re-run.
[2026-09-24T05:31:40Z] prefill walk: rc=128 duration=3417s other_lines=1     (the SAME object)

The skip is the right call (bare-repack.sh:88-95): repack -a -d pass 2 walks the same roots without lazy-fetching, so it would die on the same object after about an hour of promisor-set construction. The consequence is that the repack now never runs.

Root cause, measured read-only with GIT_NO_LAZY_FETCH=1

• 1ec7ca1cb36c... ("Keep a const threshold column const in PartialSortingTransform", authored by the fleet) is present locally and is not on groeneai/ClickHouse (commit API 422, never pushed).
• Its single parent c9d82eb90d6d... is absent locally: cat-file -e does not return within 120s, and for-each-ref --contains prints git's own error: Could not read c9d82eb9....
• 1ec7ca1c is reachable only from reflogs, never from a ref: logs/refs/heads/groeneai/fix-stid-1499-2393-finishsorting-columnsparse (the branch ref itself is healthy, at 29ddff7b) and the live worktree /worktrees/tasks/2026-07-04-ci-p0-logicalerror-bad-cast-columnspar's logs/HEAD. Both record reset: moving to 29ddff7b... at 2026-09-21 14:19Z, which is what orphaned it.
• The 09-22 repack is what dropped the parent: dangling_reflog_warnings in the same log jumps 1,497 (09-22) to 4,531 (09-23).
• The pre-fill exists precisely to refill such holes by lazy fetch (bare-repack.sh:72-79), and it could not fill this one, twice.

Why it is permanent rather than self-clearing. Git's own remedy for reflog entries pointing at commits unreachable from the tip is gc.reflogExpireUnreachable (default 30 days), applied by git gc / git maintenance. On this store gc.auto=0, maintenance.auto unset, both gc.reflogExpire* unset, core.logAllRefUpdates=true. So reflogs accumulate and nothing ever expires them, and one orphaning git reset in one worktree disables consolidation indefinitely.

Cost while it stays skipped

• packs 1,359 (09-23 04:33Z) -> 2,579 (09-24 04:34Z) -> 3,264 (09-24 17:38Z), about +1,250/day, of which 3,262 are .promisor
• /worktrees at 85%, which is the worktree-GC's own pressure-reap trigger
• each skipped firing still pays the full pre-fill walk: 2,016s, then 3,417s, and it grows with the pack count
• left alone this returns to the 82,142-pack / 2.6 TB state that cost five separate investigations

What is not urgent, so that nothing is scheduled against a false emergency: the pre-push-gate.sh branch-hygiene block that started this workstream is fixed and is not at risk. Measured today, _budget=20 (:347, growing +3 per completed merge-base at :637) and the gate's own 83-ref merge-base costs 2.32 / 2.36 / 2.37s, so blocking would need N(C-3) > 16 with C < 3s, which no N satisfies. C=3s is about 34-38k packs, roughly 25-28 days away at the current rate.

Why I am asking rather than fixing

Both candidate remedies are outside what a slot session may do:

• Store-side (restore the one object, or expire the two orphaning reflog entries) is a write to the shared bare clone, which is attended-operator-only on this box, because a killed write strands a lock fleet-wide. As evidence that a killable session should not attempt it: my read-only git reflog expire --dry-run --expire-unreachable=now <that ref> probe did not finish in 600s and was killed.
• Script-side (fleet/bin/bare-repack.sh) has no version control. /home/ubuntu/nerve-workspace, .../fleet, /home/ubuntu and /home/ubuntu/.nerve all return fatal: not a git repository, and fleet/ is not in this repository's tree, so I cannot open a PR for it.

Question (@alex-clickhouse)

You are the most recent committer here by a wide margin (22 of the last 40 commits), and I have asked you on #453, #454 and #455 in the same area, so you are the person I am asking. Which of A, B or C do you want, and do you want me to prepare the exact commands (A) or the script diff (B) for you to apply?

• A. One-off store repair. Expire the two orphaning reflog entries, or restore c9d82eb9, then re-run the timer. Unblocks consolidation tonight; does nothing about the next orphaning reset.
• B. Close the recurrence in bare-repack.sh. Expire unreachable reflog entries before the pre-fill walk, so a reflog-only orphan can never gate it again. This is my recommendation, together with A to unblock now. I could not validate it from a slot session for the reason above, so it needs your window either way.
• C. Make consolidation walk-free. git -C /worktrees/.master.git multi-pack-index write (still absent: 0 multi-pack-index* files). Immune to this entire failure class by construction, since it indexes packs without traversing commits, but it does not reduce the pack count.

One word on A, B or C unblocks me.