Input blocks: derive difficulty only from a parent bound to the best chain

[cafebedouin]

Prepared with Claude (Anthropic): Claude Fable 5.1 and Claude Opus 5 found the issue; Claude Opus 5 wrote the patch and tests.

In processInputBlock, an unknown parent gives expectedNBits = None, so valid checks proof-of-work against the announced nBits. A known parent is used without checking its position. The todo at InputBlocksProcessor.scala:862 anticipates this.

Line numbers refer to weak-blocks at c216c5b.

Check. The parent is bound when its header is known, is in the best header chain, and header.height == parent.height + 1. Only then is expectedNBits derived (requiredDifficultyAfter; at genesis height, the initial difficulty, as in header validation). A bound announcement failing valid is penalized as before. Height gates are unchanged. The parent need not be the best full block, so the todo at ErgoNodeViewSynchronizer.scala:1472 (last 1-2 ordering blocks) stays open. An input block extending the best full block is processed as before.

Policy. One commented branch, if (expectedNBits.isEmpty). A known parent outside the best header chain also reaches it, so whether fork announcements are processed is decided there. Default: not processed, no penalty; an unknown parent header is requested from the sender with requestBlockSection, guarded by status == Unknown, as in the height + 2 branch.

Request trade-off. A tracked request is re-sent to other peers after the timeout, so an invented parent id can get those peers penalized for not delivering it (when no other modifier arrived in the meantime). An untracked request gets the honest sender's reply penalized as spam. The alternative, one tracked request to the sender only, expiring without a delivery check, was implemented and tested and can be provided.

Open question. Keep unbound input blocks, as the paper's disconnected waitlist (papers/inputblocks/main.tex:366-374) and todo: save input block? (ErgoNodeViewSynchronizer.scala:1537) suggest?

Not covered. The fullBlockHeight + 2 branch (:1531-1545) requests a header with no proof-of-work check.

Tests. testOnly org.ergoplatform.network.InputBlockParentBindingSpec (real Autolykos proof-of-work; the default test configuration uses a fake scheme). 6 of 10 fail without the change.

Merge notes. Applies cleanly on @a-shannon's #2367, #2368, #2500, #2502, #2503 and #2506. From a code reading of #2506: a replayed input tip one height above the receiver's best full block, whose ordering parent the receiver does not have or has outside its best header chain, reaches the policy branch and is dropped by default. Without this change it is processed, and when the parent is unknown, against the difficulty the tip itself declares. Replays between nodes on the same best full block are processed as before.

[a-shannon]

Update, 17 September: sender-only correction checked. @cafebedouin, I ran both parent-validation suites on #2552 at 622974210 plus #2553 at 54b3322ee, composed with current weak-blocks at 31a8de804 (including #2551). 28/28 passed, 14 per suite, with no failures or ignored tests. Same Java 8u504, Scala 2.12.20, sbt 1.11.1 and cached Sigma snapshot as below.

The previously red timeout regression now passes. Both paths cover expiration without another-peer retry or non-delivery penalty, protection against an earlier attempt's expiration, and acceptance of a timely requested header. The two input-announcement replay properties pass too. Focused source review found no new issue in this increment. These are receiver/history tests, not full network or UTXO-application evidence. The unchanged input height+2 path and generic delivery-attempt hardening in #2528 remain separate.

Review order: #2552 first, then #2553's remaining increment. #2553 contains #2552 in its ancestry. Both still target weak-blocks, so #2553's full GitHub diff includes #2552 until the prerequisite merges. The earlier ordering commit is patch-equivalent after rebasing; the linked increment also adds sender-only handling and its tests for ordering announcements.

Hosted CI is still blocked before tests by the unavailable Sigma snapshot: #2552 run and #2553 run. This does not establish green CI or settle the maintainer decision on dropping versus retaining announcements with unknown/off-best-chain parents.

Earlier evidence below refers to the previous heads. Its red timeout result and pending-variant status are superseded by the update above.

Tested composition: weak-blocks c216c5b6e3310911d5f3822b40c5e57eee6fc45a, with #2551 0aed58d79edf4401c2933b3204a0575209059178, #2552 79ce9375f1ca7272aeac1d61aabdd302b2be42ef, and #2553 ad56cad55d2433d2174f3382d7d063bd1b75fac4. Java 8u504, Scala 2.12.20, sbt 1.11.1, and the declared Sigma 6.0.5-22-368a860b-SNAPSHOT from the local cache.

• Both published suites passed: 22/22, before the additional patch.
• The two added replay cases passed. One first drops an announcement with an unknown same-height parent, then explicitly replays it after that parent joins the best header chain while the best full block stays unchanged. The other starts in the fullBlockHeight + 2 branch, selects the missing parent as the best full block in history, then replays at the new fullBlockHeight + 1.
• The added sender-only expiration regression fails on the current code, as expected: requests go to another peer. It also requires the request to become Unknown, with no remaining RequestedInfo, so silently retaining a request forever cannot satisfy it. This reproduces the retry concern already documented in the PR; the production correction is still pending.

This clarifies my earlier recovery wording: best-header selection and best-full-block selection are distinct. These tests cover history selection and explicit receiver-side replay, relevant to #2506. Coverage stops at the synchronizer handoff; the #2506 sender and UTXO-state application are outside this patch. Waitlist policy remains a separate decision.

Please also preserve request-attempt identity on expiration, the invariant in #2528, and the existing legitimate-response test when adding your variant. The patch changes only InputBlockParentBindingSpec.scala, with longer deadlines for replay fixtures and the existing short deadline for expiration.

Commands:

sbt "testOnly org.ergoplatform.network.InputBlockParentBindingSpec org.ergoplatform.network.OrderingBlockAnnouncementParentCheckSpec"
sbt 'testOnly org.ergoplatform.network.InputBlockParentBindingSpec -- -z replay -z "discovery expires"'

The second command ran three additional tests: two passed and the expiration regression failed. It must become green with the correction.

CI update, 17 September: maintainer approval was granted and all three runs executed. Every failed job in #2551, #2552, and #2553 stopped during dependency resolution: org.scorexfoundation:sigma-state_<Scala binary version>:6.0.5-22-368a860b-SNAPSHOT was unavailable. Remaining jobs were cancelled. These runs provide no test result. Local tests used a cached Sigma build; they do not establish hosted CI success.

#2551 was approved and merged into weak-blocks at 31a8de80. Its effective diff matches the #2551 increment tested above, so the merge alone does not invalidate that local evidence. The sender-only request correction on #2552 remains pending.

The unavailable snapshot already has a source-pinned CI bootstrap in #2501, isolated in 13352c1e (two CI files). That existing prerequisite can be reviewed independently of #2501's broader composition. It has not been added to these parent-validation PRs. Rerunning unchanged workflows will not restore the missing dependency. Hosted validation and the sender-only correction are separate outstanding gates; these local results are not a merge-ready claim.

Three focused regression cases (one expected red until the request correction lands)

diff --git a/src/test/scala/org/ergoplatform/network/InputBlockParentBindingSpec.scala b/src/test/scala/org/ergoplatform/network/InputBlockParentBindingSpec.scala
index d43fb133d..0bd3ca481 100644
--- a/src/test/scala/org/ergoplatform/network/InputBlockParentBindingSpec.scala
+++ b/src/test/scala/org/ergoplatform/network/InputBlockParentBindingSpec.scala
@@ -58,7 +58,9 @@ class InputBlockParentBindingSpec extends AnyPropSpec with Matchers with FileUti
     * @param initialDifficultyHex overrides the test chain's difficulty (the local chain keeps it)
     * @param applyLocalChain      whether the generated chain is applied to the local history
     */
-  class Fixture(initialDifficultyHex: Option[String] = None, applyLocalChain: Boolean = true) extends AkkaFixture {
+  class Fixture(initialDifficultyHex: Option[String] = None,
+                applyLocalChain: Boolean = true,
+                requestTimeout: FiniteDuration = 2.seconds) extends AkkaFixture {
     implicit val ec: ExecutionContextExecutor = system.dispatcher
 
     val historySettings: ErgoSettings = {
@@ -68,11 +70,11 @@ class InputBlockParentBindingSpec extends AnyPropSpec with Matchers with FileUti
 
     private val cs = historySettings.chainSettings
     val realPowScheme = new AutolykosPowScheme(cs.powScheme.k, cs.powScheme.n)
-    // a short delivery timeout, so that what happens after it can be observed
+    // Expiration tests use a short timeout; replay tests keep unrelated deadlines outside their window.
     val synchronizerSettings: ErgoSettings =
       historySettings.copy(chainSettings = cs.copy(powScheme = realPowScheme),
         scorexSettings = historySettings.scorexSettings.copy(
-          network = historySettings.scorexSettings.network.copy(deliveryTimeout = 2.seconds)))
+          network = historySettings.scorexSettings.network.copy(deliveryTimeout = requestTimeout)))
 
     val ncProbe = TestProbe("NetworkControllerProbe")
     val viewHolderProbe = TestProbe("ViewHolderProbe")
@@ -226,6 +228,94 @@ class InputBlockParentBindingSpec extends AnyPropSpec with Matchers with FileUti
     }
   }
 
+  property("an input announcement is accepted on replay after its parent joins the best header chain") {
+    withFixture(new Fixture(requestTimeout = 30.seconds)) { f =>
+      val originalTip = f.hist.bestFullBlockOpt.get
+      val parent = nextBlock(Some(f.chain(1)), originalTip.blockTransactions.txs, defaultExtension)
+      parent.height shouldBe originalTip.height
+      f.hist.contains(parent.header) shouldBe false
+      val ib = announcement(f, parent.id, originalTip.height + 1, requiredNBitsAfter(f, parent.header))
+      ib.valid(f.realPowScheme, f.state.stateContext.currentParameters,
+        Some(requiredNBitsAfter(f, parent.header))) shouldBe true
+
+      f.process(ib)
+
+      viewHolderGotInputBlock(f) shouldBe false
+      val initialMessages = networkMessages(f)
+      headerRequests(initialMessages) shouldBe Seq(Seq(parent.id) -> SendToPeer(f.peer))
+      penalized(initialMessages) shouldBe false
+
+      f.hist.append(parent.header).get
+      f.hist.isInBestChain(parent.header) shouldBe false
+      val nextHeader = nextBlock(Some(parent), originalTip.blockTransactions.txs, defaultExtension).header
+      f.hist.append(nextHeader).get
+      f.hist.isInBestChain(parent.header) shouldBe true
+      f.hist.bestFullBlockIdOpt shouldBe Some(originalTip.id)
+      ib.header.height shouldBe f.hist.fullBlockHeight + 1
+
+      f.process(ib)
+
+      f.viewHolderProbe.expectMsg(ProcessInputBlock(ib, f.peer))
+      val replayMessages = networkMessages(f)
+      penalized(replayMessages) shouldBe false
+      headerRequests(replayMessages) shouldBe empty
+    }
+  }
+
+  property("an input announcement is accepted on replay after its parent becomes the best full block in history") {
+    withFixture(new Fixture(requestTimeout = 30.seconds)) { f =>
+      val originalTip = f.hist.bestFullBlockOpt.get
+      val parent = nextBlock(Some(originalTip), originalTip.blockTransactions.txs, defaultExtension)
+      f.hist.contains(parent.header) shouldBe false
+      val ib = announcement(f, parent.id, parent.height + 1, requiredNBitsAfter(f, parent.header))
+      ib.header.height shouldBe f.hist.fullBlockHeight + 2
+      ib.valid(f.realPowScheme, f.state.stateContext.currentParameters,
+        Some(requiredNBitsAfter(f, parent.header))) shouldBe true
+
+      f.process(ib)
+
+      viewHolderGotInputBlock(f) shouldBe false
+      val initialMessages = networkMessages(f)
+      headerRequests(initialMessages) shouldBe Seq(Seq(parent.id) -> SendToPeer(f.peer))
+      penalized(initialMessages) shouldBe false
+
+      applyChain(f.hist, Seq(parent))
+      f.hist.bestFullBlockIdOpt shouldBe Some(parent.id)
+      f.hist.isInBestChain(parent.header) shouldBe true
+      ib.header.height shouldBe f.hist.fullBlockHeight + 1
+
+      f.process(ib)
+
+      f.viewHolderProbe.expectMsg(ProcessInputBlock(ib, f.peer))
+      val replayMessages = networkMessages(f)
+      penalized(replayMessages) shouldBe false
+      headerRequests(replayMessages) shouldBe empty
+    }
+  }
+
+  property("unknown-parent discovery expires without requesting the header from another peer") {
+    withFixture { f =>
+      val otherPeer = f.initialize()
+      val unknownParent = bytesToId(Array.fill(32)(0x5a.toByte))
+      val tip = f.hist.bestFullBlockOpt.get.header
+      val ib = announcement(f, unknownParent, tip.height + 1, DifficultySerializer.encodeCompactBits(1))
+
+      f.process(ib)
+
+      // Observe two delivery deadlines, including a potential retry against the other peer.
+      val messages = f.ncProbe.receiveWhile(max = 5.seconds, idle = 5.seconds) { case m => m }
+      val requests = headerRequests(messages).filter(_._1.contains(unknownParent))
+      requests.map(_._2 == SendToPeer(f.peer)) shouldBe Seq(true)
+      messages.exists {
+        case p: PenalizePeer => p.address == otherPeer.connectionId.remoteAddress
+        case _ => false
+      } shouldBe false
+      f.deliveryTracker.getRequestedInfo(Header.modifierTypeId, unknownParent) shouldBe None
+      f.deliveryTracker.status(unknownParent, Header.modifierTypeId, Seq.empty) shouldBe ModifiersStatus.Unknown
+      viewHolderGotInputBlock(f) shouldBe false
+    }
+  }
+
   property("input block whose parent is a known header outside the best chain is dropped") {
     withFixture { f =>
       val tip = f.hist.bestFullBlockOpt.get

[jozanek]

What do you return here?

[cafebedouin]

Nothing: processInputBlock returns Unit, so the bare return only ends the method, like the three early returns above it (height gap, digest mode, already known input block). The announcement is not processed further.

[cafebedouin]

@a-shannon, pushed 622974210: the unknown parent header is now requested from the sender only, once, via requestHeaderFromSenderOnly. The request is tracked, so the sender's delivery is accepted as before. After the delivery timeout a SenderOnlyRequestExpired message forgets the request instead of running a delivery check: no other peer is asked, no peer is penalized, and the header is back to Unknown with no RequestedInfo (until then it is Requested, so other paths do not fetch it).

The expiration carries its own timer, the Cancellable the delivery tracker stores with the request, and the handler clears the entry only if the tracker's current RequestedInfo.cancellable is that timer. That binds an expiration to the request attempt that scheduled it without a request id on this branch; with #2528 the comparison can become requestId.

Tests: your three cases are included as posted, plus one where the header is requested twice and the first attempt's expiration, arriving after the second request, leaves the second request intact and the sender's delivery is still accepted. Removing the timer check makes that test fail. testOnly org.ergoplatform.network.InputBlockParentBindingSpec: 14/14 on the branch; the expiration case fails on the previous head, as in your run. The six existing input/ordering block suites give the same per-test results as before (162 tests, the two pre-existing OrderingBlockMessageFlowSpec timeouts included).

#2553 carries the same change for ordering block announcements and is now stacked on this branch, since both use the helper.

[cafebedouin]

Superseded by #2554, consolidated as suggested.