Skip to content

fix(windows-miner): back off transient header failures#8018

Merged
Scottcjn merged 3 commits into
Scottcjn:mainfrom
guoqiangliu-ocean:agent/windows-header-retry-policy
Jul 21, 2026
Merged

fix(windows-miner): back off transient header failures#8018
Scottcjn merged 3 commits into
Scottcjn:mainfrom
guoqiangliu-ocean:agent/windows-header-retry-policy

Conversation

@guoqiangliu-ocean

Copy link
Copy Markdown
Contributor

Closes #7368

Bounty: Scottcjn/rustchain-bounties#16252
BCOS: BCOS-L1

What this completes

#7565 stopped the original hot loop and surfaced the rejection diagnostic, but it marked every failed slot handled. A temporary connection failure or restarting node therefore never retried during the slot. This PR completes the missing retry-policy part of #7368:

  • terminal client/auth responses are marked handled and never resubmitted;
  • connection failures plus HTTP 408/425/429/5xx responses use 10, 20, 40, 80, 160, then capped 300 second backoff;
  • each retry returns through the mining loop, which rebuilds and signs a fresh header;
  • headless output says class=terminal no_retry or class=retryable retry_in=Ns and retains the node's diagnostic;
  • bootstrap SHA-256 pins are updated with the miner artifact.

Before / after reproduction

Before this PR on current main, a network blip was treated as permanently handled:

12:00:00 [share] slot=27456 submitted=1 accepted=0 FAIL error=header request failed: connection refused
12:00:10 slot=27456 suppressed by _last_submitted_slot (no retry after recovery)
12:00:20 slot=27456 suppressed by _last_submitted_slot (no retry after recovery)

After this PR, temporary failures rebuild the header and back off:

12:00:00 [share] slot=27456 submitted=1 accepted=0 FAIL error=HTTP 503 error=node restarting class=retryable retry_in=10s
12:00:10 [share] slot=27456 submitted=2 accepted=0 FAIL error=HTTP 503 error=node restarting class=retryable retry_in=20s
12:00:20 retry gate closed (next attempt remains at 12:00:30)

Terminal rejection remains one-shot. This is the actual output from a one-time, identity-free invalid-header probe against the repository's public node endpoint on 2026-07-20:

[share] slot=-7368 submitted=1 accepted=0 FAIL error=HTTP 400 error=missing fields class=terminal no_retry

The same run recorded last_submitted_slot=-7368, retry_slot=null; _header_submission_due(-7368) stays false at later timestamps, so there is no second POST of that rejected header.

Verification

34 passed in 13.00s

Covered suites:

  • Windows header retry regression tests (terminal JSON/plain-text responses, connection failure, 503, exponential cap, log formatting)
  • Windows bootstrap checksum, Python 3.9 annotation, chain identity, and headless lifecycle tests
  • attestation signing and setup-miner download tests
  • python miners/windows/check_bootstrap_hashes.py -> Windows bootstrap hashes are in sync.
  • python -m py_compile miners/windows/rustchain_windows_miner.py

The live probe bypassed miner initialization and sent no wallet, key, signature, or user identity; it only exercised the public endpoint's terminal validation response.

@github-actions

Copy link
Copy Markdown
Contributor

Welcome to RustChain! Thanks for your first pull request.

Before we review, please make sure:

  • Non-doc PRs have a BCOS-L1 or BCOS-L2 label
  • Doc-only PRs are exempt from BCOS tier labels when they only touch docs/**, *.md, or common image/PDF files
  • New code files include an SPDX license header
  • You've tested your changes against the live node

Bounty tiers: Micro (1-10 RTC) | Standard (20-50) | Major (75-100) | Critical (100-150)

A maintainer will review your PR soon. Thanks for contributing!

@github-actions github-actions Bot added BCOS-L1 Beacon Certified Open Source tier BCOS-L1 (required for non-doc PRs) tests Test suite changes size/M PR: 51-200 lines labels Jul 19, 2026

@elevasyncsolutions-jpg elevasyncsolutions-jpg left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of PR #8018 — fix(windows-miner): back off transient header failures

Overall this is a well-structured fix for #7368. Two substantive observations:

  1. Missing jitter in exponential backoff — The _schedule_header_retry uses pure exponential backoff (10, 20, 40, 80…). If multiple miners lose connectivity at the same time (e.g. a node restart), they will retry in lockstep. Adding a small random jitter (±25% of the delay) would spread retries and reduce node load on recovery. See AWS retry best practices.

  2. Redundant isinstance check — In _is_retryable_header_status, the check isinstance(status_code, int) and 500 <= status_code <= 599 has an unnecessary isinstance guard — requests always returns int for status_code. The real guard is against the case where response is None (network error), but that path already sets last_header_retryable = True in the exception handler. The 5xx range check alone would be cleaner.

Positive: The separation of terminal vs retryable failures with the _reset_header_retry() mechanism is a significant improvement over the previous binary handled/not-handled logic. The bounded cap at 300s prevents runaway waits. Good defensive design.

@github-actions github-actions Bot added size/L PR: 201-500 lines and removed size/M PR: 51-200 lines labels Jul 21, 2026
@guoqiangliu-ocean

Copy link
Copy Markdown
Contributor Author

Addressed the review feedback in 38b2890: retry jitter is now ±25% (while preserving the 300 s cap), and the redundant status-code type guard was removed. Bootstrap hashes were refreshed in abb066f. All 12 upstream checks, including the full 3,640-test CI job and Windows build, are green. Ready for maintainer review.

@Scottcjn
Scottcjn merged commit d684e61 into Scottcjn:main Jul 21, 2026
12 checks passed
@Scottcjn

Copy link
Copy Markdown
Owner

Merged — bounded exponential backoff (10s base, 300s cap, retryable on 408/425/429) with the diagnostic surfaced, exactly what #16252 asked for. Awarded 12 RTC, sent to your wallet with the other 65. Thanks for the consistently clean work. — Sophia

@github-actions

Copy link
Copy Markdown
Contributor

RTC Reward

This merged PR earned 5 RTC — sent to guoqiangliu-ocean.

RustChain Bounty Program

@JHON12091986 JHON12091986 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: Looks good! Minor suggestion: ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

BCOS-L1 Beacon Certified Open Source tier BCOS-L1 (required for non-doc PRs) size/L PR: 201-500 lines tests Test suite changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows headless miner retries a rejected header every poll and hides the server diagnostic

4 participants