Skip to content

Commit ce0daeb

Browse files
committed
feat(publish): push each gateway as it lands, not all at the end
A fourteen-gateway run went dark for its entire duration: every result was committed locally and nothing was pushed until the final sweep, so a run that died at box twelve published nothing at all despite eleven gateways having been measured cleanly. The reason given for holding it was that a half-finished run leaves some rows fresh and others stale. That is true, and it is also what the board is built to show: every row carries its own measured_at, gen-data ages each row against it, and app.js badges a stale one. A partial board that says which rows are new is honest; a board that shows nothing for an hour is just less useful. What actually made this unsafe was C8 - a board must not mix harness engines - and that guard now lives in the publish path, where it fails the BUILD rather than the push. So a mixed board cannot reach the site no matter when rows are pushed, and the push no longer has to be the thing enforcing it. Worth being precise about what this does and does not buy, because it is not the same during a run that changes the engine: - A rerun of one gateway on the SAME engine as the rest of the board now appears as soon as that box finishes. That is the steady state and it is the common case. - A full-field rerun that also moves the engine still shows nothing until the last box lands, because until then the newest snapshot per gateway spans two engines and C8 correctly refuses the board. The data is in git the moment each box finishes either way, which is the part that was actually being lost.
1 parent 69446a8 commit ce0daeb

1 file changed

Lines changed: 21 additions & 5 deletions

File tree

‎run-on-ec2.sh‎

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -468,11 +468,27 @@ its result so the board updates just this row (matrix-sole-source)." \
468468
# stale HEAD is rejected non-fast-forward; fetch/rebase-then-push in a bounded retry loop (still
469469
# inside the flock so the whole fetch-rebase-push is serialized across boxes).
470470
#
471-
# Committed here, but NOT pushed: the commit happens per gateway so a later crash cannot lose a
472-
# measurement that was already pulled, but nothing reaches the board until the whole run has shut
473-
# down cleanly, so a run killed halfway never leaves a partial set of gateways live while the rest
474-
# are missing. The final sweep at the end of the run does the push once.
475-
echo "[$gw] committed locally; nothing is published until the run finishes cleanly"
471+
# PUSHED AS IT LANDS, not held to the end of the run.
472+
#
473+
# This used to commit here and push only in the final sweep, so nothing appeared until every box
474+
# had finished - a fourteen-gateway run went dark for its whole duration, and a run that died at
475+
# box twelve published nothing at all despite eleven gateways having been measured cleanly.
476+
#
477+
# The reason given for holding it was that a half-finished run would leave some rows fresh and
478+
# others stale. That is true and it is also what the board is built to show: every row carries its
479+
# own measured_at, gen-data ages each row against it, and app.js badges a stale one. A partial
480+
# board that says which rows are new is honest; a board that shows nothing for an hour is just
481+
# less useful.
482+
#
483+
# What actually made this unsafe was C8 - a board must not mix harness engines, and a run
484+
# spanning an engine change would have published a mixed board one row at a time. That guard is
485+
# in the publish path now (site/check-consistency.mjs) and it fails the BUILD rather than the
486+
# push, so a mixed board never reaches the site regardless of when rows are pushed.
487+
if push_with_rebase "[$gw]" echo; then
488+
echo "[$gw] published"
489+
else
490+
echo "[$gw] publish: push failed; the commit is local and the final sweep will retry it"
491+
fi
476492
)
477493
}
478494

0 commit comments

Comments
 (0)