Skip to content

fix(release): self-heal the release-please "untagged, merged" abort#7149

Merged
JSONbored merged 1 commit into
mainfrom
fix/release-please-reconcile-stale-tagging
Jul 18, 2026
Merged

fix(release): self-heal the release-please "untagged, merged" abort#7149
JSONbored merged 1 commit into
mainfrom
fix/release-please-reconcile-stale-tagging

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

Root cause traced as far as possible into release-please's own source: createReleases() (the step that actually tags a merged Release PR) sometimes finds nothing to tag, and createPullRequests() then aborts every subsequent run without ever completing the tag. Confirmed live across two separate linked-versions release cycles this session (#7127, #7133), both requiring a manual dispatch of each publish-*.yml workflow plus manually flipping the merged PR's autorelease: pending label. This matches several long-open, unresolved upstream issues (googleapis/release-please#1946, #1444, #1406 -- same exact message), so it isn't something fixable from this repo's config alone.

Adds a reconcile-stale-releases job that runs after every release-please attempt, independent of its outputs (which are unreliable exactly when this bug is in play):

  • Compares each package's committed package.json version against what's actually live on npm, and dispatches that package's publish workflow whenever they disagree -- the same "bare manual dispatch" human-override path each publish-*.yml already documents, just triggered automatically instead of by a human noticing red CI on an unrelated PR days later.
  • Engine publishes first and is awaited before mcp/miner (they carry a real runtime dependency on it, and their own isolated pack/smoke-test resolves @loopover/engine from the real npm registry, so it fails with ETARGET if engine hasn't actually published yet -- confirmed live when I first tried dispatching all three in parallel).
  • Once every package is confirmed live, flips any merged PR still carrying autorelease: pending to autorelease: tagged, unsticking release-please's own tracking for its next run.

Real npm publishes still go through the same environment: release required-reviewer gate as always -- this doesn't remove that security control, it just eliminates the diagnostic/dispatch/label-cleanup toil around it.

Side effect I verified is safe and beneficial: running the version-comparison logic locally against this repo's real current state found one ancient, silently-stuck instance of the exact same bug (engine v1.0.0's Release PR, still autorelease: pending since long before this session) -- the new job's cleanup step will resolve that too.

Test plan

  • npm run actionlint -- clean
  • Extracted both run: scripts and ran bash -n on each -- syntax valid
  • Ran the actual version-comparison script for real against this repo's live state: correctly identified all 4 packages as already in sync and dispatched nothing (safe to execute as a real validation since nothing was out of sync)
  • Workflow-only change; not measured by Codecov. Real verification: the next release-please abort should self-resolve within this same job's run instead of needing manual intervention.

Root cause traced as far as possible: release-please's own createReleases()
(the step that actually tags a merged Release PR) sometimes finds nothing
to tag, and createPullRequests() then aborts every subsequent run without
ever completing the tag -- confirmed live across two separate
linked-versions release cycles this session (#7127, #7133), both requiring
a manual dispatch of each publish-*.yml workflow plus manually flipping
the merged PR's autorelease:pending label. Matches multiple long-open,
unresolved upstream issues (googleapis/release-please#1946/#1444/#1406),
so it isn't something fixable from this repo's config alone.

Adds a reconcile-stale-releases job that runs after every release-please
attempt, independent of its outputs: compares each package's committed
package.json version against what's actually live on npm, dispatches
that package's publish workflow whenever they disagree (the same
documented human-override path, automated), and flips any merged PR's
autorelease label once every package is confirmed live -- unsticking
release-please's own tracking for the next run. Also cleaned up one
ancient, silently-stuck instance of the exact same bug (engine v1.0.0,
autorelease:pending since long before this session).
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.75%. Comparing base (c060b2a) to head (240f29c).
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #7149   +/-   ##
=======================================
  Coverage   93.75%   93.75%           
=======================================
  Files         692      692           
  Lines       68832    68832           
  Branches    18781    18781           
=======================================
  Hits        64535    64535           
  Misses       3302     3302           
  Partials      995      995           
Flag Coverage Δ
shard-1 43.75% <ø> (ø)
shard-2 36.91% <ø> (ø)
shard-3 33.14% <ø> (ø)
shard-4 34.44% <ø> (ø)
shard-5 32.27% <ø> (+<0.01%) ⬆️
shard-6 45.90% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

@JSONbored
JSONbored merged commit c4eb125 into main Jul 18, 2026
17 checks passed
@JSONbored
JSONbored deleted the fix/release-please-reconcile-stale-tagging branch July 18, 2026 01:11
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 18, 2026
galuis116 pushed a commit to galuis116/gittensory that referenced this pull request Jul 18, 2026
…liation job

The reconcile-stale-releases job (JSONbored#7149) shipped with two real bugs,
found by actually running it live rather than trusting its "success"
conclusion:

1. dispatch_and_wait() used `gh run list --limit 1` right after `gh
   workflow run` to find the just-dispatched run's ID -- unreliable, since
   the new run isn't always indexed by the API yet. It grabbed a STALE,
   already-completed prior run instead, reported that run's old "success",
   and silently never published the new version at all (confirmed live:
   packages/loopover-engine committed 3.2.3 while npm still had 3.2.2,
   undetected until manually checking the actual log content -- exactly
   the kind of unverified-success this whole session has been about not
   repeating). Fixed by parsing the run ID directly from `gh workflow
   run`'s own stdout (which prints the created run's URL), with a
   timestamp-filtered `gh run list` fallback for the rare case that URL
   isn't returned. Verified this against several real dispatches before
   shipping.

2. mcp/miner's own isolated pack/smoke-test resolves @loopover/engine
   from the real npm registry (no local workspace symlink in that bare
   temp dir), and can transiently ETARGET for a short window after
   engine's own publish reports success -- npm's registry is CDN-backed,
   not instantly consistent across every edge node (confirmed live:
   mcp and miner dispatched at the same moment against the same
   already-published engine version, mcp's smoke-test hit a
   already-propagated edge node and succeeded, miner's hit one that
   hadn't and failed with the exact ETARGET this job exists to fix).
   Added a settle delay after engine publishes plus automatic retry
   (with backoff) for its dependents -- this is registry propagation
   lag, not a genuine test regression, which would fail identically on
   every retry instead of resolving itself.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant