Restore V2 fork synchronization and block-download progress

[a-shannon]

A V2 sync cache keyed only by header height can return a reduced one-header summary to a later full-summary request at the same height. This happens when the reduced request populates the cache first; header delivery and history application are asynchronous. Keying by selected header ID and requested mode preserves the full sampling ladder and also refreshes summaries after a same-height tip replacement.

Review follow-up, 21 September: prioritize the cache-only unit on current master, separately from genesis sampling and download scheduling. Source review identifies the synchronizer cache hunks and their direct spec as a standalone unit. A merge-tree check of portable G against master 23aabead is conflict-free; this is source composition evidence, not a compiled or rebased candidate. No full #2535 merge or #2501 merge is required to begin that focused work. The published branch remains unchanged pending this restructuring.

The external short-chain experiments below compare the cumulative PR with v6.0.5. They do not isolate the cache change or exercise the far-behind download branch under the stated chain conditions. A cache-only execution remains an open attribution check. The requested clamped sampling alternative, explicit rollback-depth naming, scheduling scan measurement, V1 stale-tip cache handling and terminal-height guard description are recorded for their appropriate scope. None is claimed implemented by this status update. The related #1815 remains open; this PR does not claim to resolve it.

Full V2 synchronization summaries retain the locally available genesis header alongside recent samples. The cache is keyed by selected header ID and requested mode, so full requests cannot reuse reduced summaries and a replacement tip refreshes it even at the same height. Tip order, deduplication, reduced and empty summaries, wire encoding and validation rules are preserved.

During active synchronization, periodic body downloads can extend backward to a confirmed common ancestor within the configured search bound. The forward ceiling remains anchored to the best full block, so finding an older retained ancestor does not shorten forward progress. Pruning floors, filtering and request counts are preserved; arithmetic is bounded at the terminal height. A truncated search does not invent an ancestor, and rollback remains limited by retained state versions.

Current head: 8c00df3a9850660d2cfd13f5c23a99c5b5b070aa, targeting master. The latest one-file increment reuses the shared startup-reader fixture without changing production. The preceding history increment added two real-history regressions. All three production files remain unchanged from reviewed revision c6e566900e5b12aeece8f6f3aa78d778da91e4e7.

The synthetic-history fixture now owns its startup history/mempool responses, preventing a late empty real-holder response from replacing its populated readers. State/vault handling, all exact full-summary expectations and their deadlines remain intact. This is the exact shared reader increment R, also reused by #2433. Normal compilation and 47/47 tests passed on this actual master composition: synchronizer 26, cache 4, history 6 and scheduling 11. Independent composition review found no blockers.

The new cases construct and append actual header histories sharing prefixes of 3 and 18 headers, with fork tips 9/13 and 249/70. In both directions the old sparse summary has no recognizable common header and yields an empty continuation; the full summary with genesis returns the exact expected chain continuation, including the divergent section. Serializer round trips are checked. All six history tests pass; restoring only the old summary implementation makes both added cases fail, and restoring this correction passes both. These prove a header-continuation defect and its correction, not full-block network convergence or exclusive attribution of an earlier CI timeout.

Additional external evidence, 21 September: cafebedouin reports four-node devnet experiments on this exact head: the lighter-fork follower switched in 8/8 runs within 6 seconds, whereas the v6.0.5 release stayed on the lighter fork for the 300-second observation in 9/13 runs. Reverse controls passed 4/4. The report describes matching best full blocks and a UTXO rollback, extending beyond the local header-continuation fixtures above. These are contributor-reported measurements, not independently rerun here. The comparison uses the release jar versus this cumulative PR, and the runs span multiple script revisions; it does not isolate a single change or measure production incidence.

Shared test support remains owned by #2535. Review R once, then the portable seven-file synchronization increment G, which this PR owns. Both exact commits are ancestors of the current head and of #2535's release composition. This order requires no full #2535 merge before #2511. The master branch also retains historical support 5eedfdc91d7a515f66822643e730b4ca9b3197df; the older cumulative increment includes diagnostics. Upstream's full diff remains cumulative until prerequisites are integrated. Maintainers select release integration or a focused port of G.

For release qualification, fork integration PR #3 at 168bd7e16860d3e3a6fbae80a000a4478f19c007 passed all eight CI jobs. Its tree exactly equals #2535's efbd1e917fe74f0f680a2e3c9e4d93595a59b06a, including #2549's separate cache wake-up prerequisite. The seven G source/direct-test blobs match this PR exactly; 49 local checks include the two wake-up regressions. Integration fixtures and deadlines are unchanged. Historical run 34724671004 covered the earlier d72/7d release composition, including 51/51 integration tests and one ignored case. This PR remains the synchronization correction's owner; native master CI remains separate.

Earlier review increments remain available:

• Candidate isolation composition.
• ForkResolution diagnostic increment: topology, fixed target, per-node accepted-value latching and 15-minute post-startup budget retained.
• Settled-seed composition: bounded fully applied non-mining common seed below height 50, with peer isolation and 50/200 targets, 20-minute outer and ten-minute final convergence deadlines retained. Shared helper/test blobs match that prerequisite exactly.

Native CI for 8c00df3a9 passed all eight jobs, including node and integration. The preceding run at 40c88de93 passed integration, but wallet Scala 2.11 failed during Java setup before sbt and matrix siblings were cancelled. The new source update supersedes that old-head rerun request. The earlier 8/8 CI applies to 1eaec0059; its 32 pure observation tests and integration compilation remain evidence for unchanged inputs. Earlier 8/8 CI and ForkResolution timeout likewise retain their original scope.

The scheduler correction requested on #2508 remains here. Missing-parent handling remains with #2433 and #2312; pruning admissibility remains with #2359. Candidate expiry #2478 targets v6.0.7 separately. Wallet/support consumers do not require this production increment merely to acquire shared test fixtures. Review order, reviewed pins and current CI are tracked in #2533.

[cafebedouin]

Prepared with Claude (Anthropic), model Claude Opus 5, building on earlier test-setup work with Claude Opus 4.8.

@a-shannon, this is independent multi-node evidence for the case the description explicitly scopes out ("… not full-block network convergence …"). A follower on the lighter of two static forks, connected only to a peer that holds the heavier fork, stayed on the lighter fork for the whole 300 s window in 9 of 13 runs on a v6.0.5 release jar. On this head (8c00df3a9) it switched in 8 of 8 runs, each within 6 s.

Setup. Four node processes run with --devnet on an isolated network. One miner builds a prefix of 12–18 blocks (21–22 in the reverse runs), and a second miner syncs it from the same genesis. The two miners are then partitioned, and each extends its own fork. Follower L takes the lighter chain X and follower S takes the heavier chain Y. Both followers are then cut off from their miners, so neither held chain grows, and no node is restarted. L and S are then connected to each other, and L's header id at the fork height is polled for up to 300 s.

build	runs	L switched to Y	S's Extension is empty while comparison is younger warnings after L and S connect
v6.0.5 release jar	13	4 (within 6 s)	12–15 in each of the 9 runs where L did not switch; 0 in the 4 where it did
#2511 8c00df3a9	8	8 (within 6 s)	0 in every run

Across these runs, fork heights were 13–19, L held 2–23 blocks above the fork point, and Y led X by 9–36 in chain score. Each run is a single 300 s observation, and the counts are descriptive. Not counted: one early release run in which L stopped below height 16 (too low for any of its sync samples to reach the shared prefix), two release runs in which S stalled before it was ahead by 8 (inconclusive), and one further #2511 switch whose log was not kept.

Reverse control on #2511, with X heavier than Y by 8–10: in 4 of 4 runs L stayed on X for the whole window, and S switched to X within 5 s.

The warning is emitted when the continuation for a younger peer is empty. On the release jar it matched the outcome in all 13 runs: it was present in every run where L stayed on X and absent in every run where L switched. In the runs where L stayed on X, L and S were connected for the whole window, but L never received a header from S. In one of these runs L received two block sections from S (an extension and a transactions section); in the others it received nothing. The runs where L stayed on X are consistent with the empty continuation that the new history cases reproduce. The release runs where L switched are consistent with v6.0.5 sometimes sending a summary that does reach the shared prefix. Which summary form was sent was not captured. On #2511, L received Y's headers and block sections, rolled back its UTXO state once, and ended with the same best full block as S.

Build. sbt assembly on JDK 8 at 8c00df3a9, unmodified. Its base is 5528ef569, which was master at the time of these runs. The script runs logged OpenJDK 21 for the nodes. /info.appVersion reported 6.0.5-12-8c00df3a-SNAPSHOT on all four nodes in every #2511 run. The release runs used a v6.0.5 release jar (sha256 2a7e2978…), and every release run but the first reported 6.0.5 on all four nodes. The first release run predates that check; its configuration used the same jar.

Reproduce. The script below is self-contained. It needs the following:

• Linux with unprivileged user namespaces
• util-linux (unshare)
• iproute2 (ip, tc) with the sch_netem module
• jq, curl, unzip and bash 4
• a Java runtime and about 1.5 GB of free RAM (four nodes at -Xmx512m; peak measured at 1.36 GB)

No Docker is required, and no root either, except sudo modprobe sch_netem once if netem isn't loaded. The script checks for user namespaces and netem and says what is missing. A run takes about 3 minutes when L switches and about 7 minutes otherwise. Use a fresh WORKDIR (the default) for each run. The v6.0.5 outcome varies from run to run, so run main several times on that jar.

bash ergo-fork-convergence.sh <ergo-node.jar> main          # L switches on #2511; on v6.0.5, often stays on X (9 of 13 here)
bash ergo-fork-convergence.sh <ergo-node.jar> reverse       # on #2511: L stays on X, S switches
LMIN=32 bash ergo-fork-convergence.sh <ergo-node.jar> main  # deeper fork on L (run on #2511 here)

The exact script below (sha256 752a6b20…) produced 2 of the release runs (L switched in 1 of them), 2 of the #2511 runs and 1 of the reverse runs. The other runs used an earlier test setup and earlier revisions of this script, with the same topology and the same held-fork procedure.

ergo-fork-convergence.sh

#!/usr/bin/env bash
# ergo-fork-convergence.sh: does a follower on a lighter, static fork adopt a heavier fork that its
# only peer holds?
#
#   ./ergo-fork-convergence.sh <ergo-node.jar> [main|reverse]
#
# Four real node processes (--devnet, private magic), each in its own network namespace, with one
# veth pair per link. No root is needed: the script runs itself under `unshare -Urmn` (an
# unprivileged user, mount and net namespace). The one exception: if the sch_netem kernel module is not
# loaded and cannot be auto-loaded, run `sudo modprobe sch_netem` once.
#
#   A (miner, chain X) --- L (follower)          A mines a prefix alone; C starts later with its
#   |                      |                     genesis pinned to A's, syncs the prefix, and the
#   C (miner, chain Y) --- S (follower)          A-C link is then cut. Each miner extends its own fork.
#                                                The L-S link starts fully lossy.
#
# main:    Y is the heavier fork. L is cut from A once it has synced X; S is cut from C once it is
#          ahead by DELTA. Both held chains are then static, and no node is restarted. L-S is opened
#          and L's header id at the fork height is polled for up to TMAX seconds.
#          Expected on a converging node: L switches to Y.
# reverse: X is the heavier fork. L must NOT switch; S should switch to X.
#
# The witness is the header id at the fork height, not a height.
#
# Requires: Linux with unprivileged user namespaces, util-linux (unshare, mount), iproute2 (ip, tc) with
#           sch_netem, jq, curl, unzip, bash 4+, and a Java runtime the jar supports on PATH (the
#           published runs used OpenJDK 21).
# Env: PREFIX_MIN (default 14), LMIN (17: minimum height L is held at; 32 gives a ~15-block fork on L),
#      TMAX (300), DELTA (8), WORKDIR (default: mktemp -d).
# Runtime: about 3 minutes when L switches, about 7 otherwise. RAM: 4 JVMs at -Xmx512m; peak measured ~1.4 GB.
# Use a fresh WORKDIR for each run. If the script is killed hard, `pkill -f <workdir>/conf_` stops the nodes.
set -uo pipefail

if [[ "${FC_INNER:-0}" != 1 ]]; then
  JAR="$(readlink -f "${1:?usage: $0 <ergo-node.jar> [main|reverse]}")"; MODE="${2:-main}"
  [[ -f "$JAR" ]] || { echo "no jar: $JAR" >&2; exit 2; }
  [[ "$MODE" == main || "$MODE" == reverse ]] || { echo "mode must be main or reverse" >&2; exit 2; }
  for c in unshare ip tc jq curl unzip java; do command -v "$c" >/dev/null || { echo "missing: $c" >&2; exit 2; }; done
  WORKDIR="${WORKDIR:-$(mktemp -d)}"; mkdir -p "$WORKDIR"; WORKDIR="$(readlink -f "$WORKDIR")"
  [[ -e "$WORKDIR/node_A.log" ]] && { echo "WORKDIR $WORKDIR was already used; pass a fresh one" >&2; exit 2; }
  unshare -Urmn true 2>/dev/null || { echo "unprivileged user namespaces are unavailable here (on Ubuntu 23.10+:" \
    "sudo sysctl kernel.apparmor_restrict_unprivileged_userns=0, or use a VM)" >&2; exit 2; }
  export FC_INNER=1 JAR MODE WORKDIR
  exec unshare -Urmn bash "$(readlink -f "${BASH_SOURCE[0]}")"
fi

# ---------------- inside the namespaces ----------------
PREFIX_MIN=${PREFIX_MIN:-14}; LMIN=${LMIN:-17}; TMAX=${TMAX:-300}; DELTA=${DELTA:-8}; HOLD=60
MAGIC='[99,97,102,101]'
# Ergo's own public test mnemonic (src/main/resources/nodeTestnet/application.conf); gives miners a reward address.
MNEMONIC="ozone drill grab fiber curtain grace pudding thank cruise elder eight picnic"
API_KEY=hello; API_HASH=324dcf027dd4a30a932c441f365a25e86b173defa4b8e58948253471b81b72cf
echo "[fc] jar=$JAR sha256=$(sha256sum "$JAR" | cut -c1-16) mode=$MODE workdir=$WORKDIR"
echo "[fc] java: $(java -version 2>&1 | head -1)"

mount --make-rprivate / 2>/dev/null || true
mount -t tmpfs tmpfs /run || { echo "FAIL: cannot mount tmpfs on /run in the user namespace"; exit 10; }
mkdir -p /run/netns
# The node resolves some fallback configs relative to its CWD; give it the jar's own copies.
RT="$WORKDIR/rt"; mkdir -p "$RT/src/main/resources"
unzip -o -q "$JAR" application.conf devnet.conf mainnet.conf testnet.conf -d "$RT/src/main/resources"

NODES=(A C L S)
declare -A LIP PRIMARY PID
for n in "${NODES[@]}"; do ip netns add "ns_$n"; ip -n "ns_$n" link set lo up; done
LINKS=("A L" "A C" "C S" "L S")              # link i -> subnet 10.9.(20+i).0/30; first node .1, second .2
for i in "${!LINKS[@]}"; do
  read -r a b <<< "${LINKS[$i]}"; s="10.9.$((20+i))"
  ip link add "ve_${a}_${b}" type veth peer name "ve_${b}_${a}"
  ip link set "ve_${a}_${b}" netns "ns_$a"; ip link set "ve_${b}_${a}" netns "ns_$b"
  ip -n "ns_$a" addr add "$s.1/30" dev "ve_${a}_${b}"; ip -n "ns_$b" addr add "$s.2/30" dev "ve_${b}_${a}"
  ip -n "ns_$a" link set "ve_${a}_${b}" up; ip -n "ns_$b" link set "ve_${b}_${a}" up
  LIP["$a,$b"]="$s.1"; LIP["$b,$a"]="$s.2"
  [[ -z "${PRIMARY[$a]:-}" ]] && PRIMARY[$a]="$s.1"; [[ -z "${PRIMARY[$b]:-}" ]] && PRIMARY[$b]="$s.2"
done
netem(){ ip netns exec "ns_$1" tc qdisc replace dev "ve_${1}_${2}" root netem $3; }   # $1->$2 direction
{ netem L S "loss 100%" && netem S L "loss 100%"; } || { echo "FAIL: tc netem unavailable (try: sudo modprobe sch_netem)"; exit 10; }

if [[ "$MODE" == main ]]; then POLL_A=3s; POLL_C=1s; else POLL_A=1s; POLL_C=3s; fi
declare -A MINING=([A]=true [C]=true [L]=false [S]=false) POLL=([A]=$POLL_A [C]=$POLL_C)
declare -A PEERS=([A]="" [C]="A" [L]="A S" [S]="C L")
declare -A EXTRA
conf(){ local n=$1; local f="$WORKDIR/conf_$n.conf" kp="" p
  for p in ${PEERS[$n]}; do kp+="\"${LIP[$p,$n]}:9021\","; done
  { echo "ergo.directory=\"$WORKDIR/data_$n\""
    echo "ergo.node.mining=${MINING[$n]}"; echo "ergo.node.offlineGeneration=${MINING[$n]}"
    echo "ergo.node.useExternalMiner=false"
    [[ ${MINING[$n]} == true ]] && echo "ergo.node.internalMinerPollingInterval=${POLL[$n]}"
    echo "ergo.wallet.testMnemonic=\"$MNEMONIC\""; echo "ergo.wallet.testKeysQty=5"
    echo "scorex.network.bindAddress=\"0.0.0.0:9021\""
    echo "scorex.network.declaredAddress=\"${PRIMARY[$n]}:9021\""
    echo "scorex.network.knownPeers=[${kp%,}]"
    echo "scorex.network.allowLocal=true"; echo "scorex.network.nodeName=\"$n\""
    echo "scorex.network.magicBytes=$MAGIC"
    echo "scorex.restApi.bindAddress=\"127.0.0.1:9052\""; echo "scorex.restApi.apiKeyHash=\"$API_HASH\""
    # settings from Ergo's src/it/resources/devnetTemplate.conf, so a sole-peer follower fully syncs
    echo "ergo.node.stateType=utxo"; echo "ergo.node.verifyTransactions=true"; echo "ergo.node.blocksToKeep=-1"
    echo "ergo.node.PoPoWBootstrap=false"; echo "ergo.node.minimalSuffix=10"; echo "ergo.node.mempoolCapacity=10000"
    echo "ergo.chain.powScheme.powType=autolykos"; echo "ergo.chain.powScheme.k=32"; echo "ergo.chain.powScheme.n=26"
    [[ -n "${EXTRA[$n]:-}" ]] && echo "${EXTRA[$n]}"
  } > "$f"; echo "$f"; }
# Each node gets its own java.io.tmpdir so that four nodes starting at once on one host do not interfere at startup.
launch(){ local n=$1 c; c=$(conf "$n"); mkdir -p "$WORKDIR/data_$n" "$WORKDIR/tmp_$n"
  ip netns exec "ns_$n" bash -c "cd '$RT' && exec java -Xmx512m -Djava.io.tmpdir='$WORKDIR/tmp_$n' -jar '$JAR' --devnet -c '$c'" >> "$WORKDIR/node_$n.log" 2>&1 &
  PID[$n]=$!; }
rest(){ ip netns exec "ns_$1" curl -s --max-time 4 "http://127.0.0.1:9052$2"; }
f(){ local v; v=$(rest "$1" /info | jq -r ".$2 // \"null\"" 2>/dev/null); echo "${v:-null}"; }   # "null" if unreachable
header_at(){ rest "$1" "/blocks/at/$2" | jq -r '.[0] // empty'; }
wait_up(){ local end=$((SECONDS+90)); while [[ $SECONDS -lt $end ]]; do rest "$1" /info | jq -e .appVersion >/dev/null 2>&1 && return 0; sleep 2; done; return 1; }
cleanup(){ for n in "${NODES[@]}"; do [[ -n "${PID[$n]:-}" ]] && kill "${PID[$n]}" 2>/dev/null; done
           pkill -f "$WORKDIR/conf_" 2>/dev/null; wait 2>/dev/null; }
trap cleanup EXIT
cut_(){  netem "$1" "$2" "loss 100%" && netem "$2" "$1" "loss 100%" || die "netem failed"; echo "[fc] cut $1<->$2"; }
open_(){ netem "$1" "$2" "delay 0ms" && netem "$2" "$1" "delay 0ms" || die "netem failed"; echo "[fc] open $1<->$2"; }
snap(){ for n in "${NODES[@]}"; do printf '%s[full=%s hdr=%s score=%s] ' "$n" "$(f $n fullHeight)" "$(f $n headersHeight)" "$(f $n headersScore)"; done; echo; }
synced_to(){ local fh hh; fh=$(f "$1" fullHeight); hh=$(f "$1" headersHeight)   # holder full==hdr and on its miner's chain
  [[ "$fh" != null && "$fh" == "$hh" ]] || return 1; local id; id=$(f "$1" bestFullHeaderId)
  [[ "$id" != null && "$id" == "$(header_at "$2" "$fh")" ]]; }
dial(){ ip netns exec "ns_$1" curl -s --max-time 4 -X POST -H "api_key: $API_KEY" -H "Content-Type: application/json" \
          --data "\"$2:9021\"" http://127.0.0.1:9052/peers/connect >/dev/null; }
conn_to(){ rest "$1" /peers/connected | jq -r --arg p "$2" '[.[]?|select(.name==$p)]|length' 2>/dev/null; }
die(){ echo "[fc] INCONCLUSIVE: $*"; snap; exit 3; }

for n in A L S; do launch $n; done
for n in A L S; do wait_up $n || die "$n did not start"; done
echo "[fc] versions: A=$(f A appVersion) L=$(f L appVersion) S=$(f S appVersion)"

# 1. prefix: A mines alone; C starts with genesisId pinned to A's genesis and syncs A's prefix
end=$((SECONDS+180)); until [[ "$(f A fullHeight)" != null && "$(f A fullHeight)" -ge $PREFIX_MIN ]] || [[ $SECONDS -ge $end ]]; do sleep 2; done
[[ -n "$(header_at A $PREFIX_MIN)" ]] || die "A did not reach height $PREFIX_MIN"
G=$(header_at A 1); [[ -n "$G" ]] || die "A mined no genesis"
EXTRA[C]="ergo.chain.genesisId=\"$G\""; launch C; wait_up C || die "C did not start"
echo "[fc] C version=$(f C appVersion); A genesis=${G:0:16}"
end=$((SECONDS+150)); ok=no
while [[ $SECONDS -lt $end ]]; do
  [[ "$(header_at C 1)" == "$G" && "$(f C fullHeight)" != null && "$(header_at C $PREFIX_MIN)" == "$(header_at A $PREFIX_MIN)" ]] && { ok=yes; break; }; sleep 1
done
[[ $ok == yes ]] || die "C did not sync A's prefix"
# 2. partition the miners
cut_ A C; P0=$(f A fullHeight); echo "[fc] partitioned at A.full=$P0"
# 3-4. freeze the lighter holder, then the heavier holder once it leads by DELTA
if [[ "$MODE" == main ]]; then LH=L; LM=A; HH=S; HM=C; else LH=S; LM=C; HH=L; HM=A; fi
LMIN_EFF=$LMIN; [[ "$MODE" == reverse ]] && LMIN_EFF=0
end=$((SECONDS+180)); ok=no
while [[ $SECONDS -lt $end ]]; do lf=$(f $LH fullHeight)
  [[ "$lf" != null && "$lf" -ge $((P0+2)) && "$lf" -ge $LMIN_EFF ]] && synced_to $LH $LM && { ok=yes; break; }; sleep 1; done
[[ $ok == yes ]] || die "lighter holder $LH did not sync"
cut_ $LH $LM; sleep 3; LIGHT=$(f $LH fullHeight); echo "[fc] $LH frozen at $LIGHT"
end=$((SECONDS+240)); ok=no
while [[ $SECONDS -lt $end ]]; do hf=$(f $HH fullHeight)
  [[ "$hf" != null && "$hf" -ge $((LIGHT+DELTA)) ]] && synced_to $HH $HM && { ok=yes; break; }; sleep 1; done
[[ $ok == yes ]] || die "heavier holder $HH did not reach +$DELTA"
cut_ $HH $HM; sleep 5
# 5. preconditions: static, L fully synced, margin, common genesis, fork height, divergent tips
s1="$(f L bestHeaderId) $(f S bestHeaderId)"; sleep 20; s2="$(f L bestHeaderId) $(f S bestHeaderId)"
LF=$(f L fullHeight); LHH=$(f L headersHeight); SF=$(f S fullHeight); LS=$(f L headersScore); SS=$(f S headersScore)
snap
for v in LF SF LS SS; do [[ "${!v}" =~ ^[0-9]+$ ]] || die "a follower's REST is not answering ($v=${!v})"; done
[[ "$s1" == "$s2" ]] || die "held chains not static"
[[ "$LF" == "$LHH" ]] || die "L not fully synced (full=$LF hdr=$LHH)"
if [[ "$MODE" == main ]]; then M=$((SS-LS)); else M=$((LS-SS)); fi
[[ $M -ge $DELTA ]] || die "score margin $M < $DELTA"
[[ "$(header_at L 1)" == "$(header_at S 1)" ]] || die "L and S have different genesis"
HF=""; for ((h=1; h<=LF; h++)); do [[ "$(header_at L $h)" != "$(header_at S $h)" ]] && { HF=$h; break; }; done
[[ -n "$HF" && "$HF" -gt 1 ]] || die "no fork height found"
X=$(header_at L $HF); Y=$(header_at S $HF); low=$(( LF<SF ? LF : SF ))
[[ "$(header_at L $low)" != "$(header_at S $low)" ]] || die "observer control: tips not divergent"
echo "[fc] fork height=$HF L=$LF (score $LS) S=$SF (score $SS) margin=$M"
echo "[fc] X@fork=$X"; echo "[fc] Y@fork=$Y"
W0=$(grep -c "Extension is empty while comparison is younger" "$WORKDIR/node_S.log")
W0L=$(grep -c "Extension is empty while comparison is younger" "$WORKDIR/node_L.log")
# 6. open L-S and observe
open_ L S; dial L "${LIP[S,L]}"; dial S "${LIP[L,S]}"; last=$SECONDS; t0=$SECONDS; tL=""; tS=""
while [[ $((SECONDS-t0)) -lt $TMAX ]]; do
  lh=$(header_at L $HF); sh=$(header_at S $HF); c=$(conn_to L S)
  [[ "${c:-0}" == 0 && $((SECONDS-last)) -ge 20 ]] && { dial L "${LIP[S,L]}"; dial S "${LIP[L,S]}"; last=$SECONDS; }
  stL="?"; [[ "$lh" == "$X" ]] && stL=X; [[ "$lh" == "$Y" ]] && stL=Y
  stS="?"; [[ "$sh" == "$X" ]] && stS=X; [[ "$sh" == "$Y" ]] && stS=Y
  echo "  t+$((SECONDS-t0))s L@fork=$stL S@fork=$stS L.full=$(f L fullHeight) S.full=$(f S fullHeight) connected=${c:-0}"
  same=no; [[ "$(f L bestFullHeaderId)" == "$(f S bestFullHeaderId)" ]] && same=yes
  [[ $stL == Y && -z "$tL" && $same == yes ]] && tL=$((SECONDS-t0))
  [[ $stS == X && -z "$tS" && $same == yes ]] && tS=$((SECONDS-t0))
  [[ "$MODE" == main && -n "$tL" ]] && break
  sleep 5
done
[[ -n "$tL$tS" ]] && { sleep $HOLD; echo "[fc] after ${HOLD}s: L@fork=$(header_at L $HF | cut -c1-16) S@fork=$(header_at S $HF | cut -c1-16)"; }
W=$(( $(grep -c "Extension is empty while comparison is younger" "$WORKDIR/node_S.log") - W0 ))
WL=$(( $(grep -c "Extension is empty while comparison is younger" "$WORKDIR/node_L.log") - W0L ))
echo "[fc] 'Extension is empty while comparison is younger' after open: S=$W L=$WL"
echo "[fc] L rollbacks: $(grep -c 'Rollback UtxoState' "$WORKDIR/node_L.log")  S rollbacks: $(grep -c 'Rollback UtxoState' "$WORKDIR/node_S.log")"
if [[ "$MODE" == main ]]; then
  [[ -n "$tL" ]] && echo "RESULT main: L SWITCHED to the heavier fork at t+${tL}s (fork height $HF, margin $M)" \
                 || echo "RESULT main: L did NOT switch in ${TMAX}s (fork height $HF, margin $M)"
else
  [[ -z "$tL" ]] && echo "RESULT reverse: L stayed on its heavier fork for ${TMAX}s (expected); S switched: $([[ -n "$tS" ]] && echo "yes at t+${tS}s" || echo no)" \
                 || echo "RESULT reverse: L SWITCHED to the lighter fork at t+${tL}s (unexpected)"
fi
echo "[fc] node logs: $WORKDIR/node_{A,C,L,S}.log"

Not covered.

• which of the three production changes is responsible (the sync payloads were not captured)
• a port onto v6.0.6
• longer chains (the followers' held chains were at most 70 blocks high)
• more than one peer per follower
• the PR's integration specs

[jozanek]

Thanks for this, and for stating the attribution limits up front rather than burying them. Three things I'd value your view on.

The cache hunks may be the whole mechanism. getV2SyncInfo(hr, full = false) at :741 fires after every header batch and, under a height-only key, poisoned the full = true broadcast with a one-header summary — disabling the 16/128/512 ladder during active sync. That explains the empty continuations, and it isn't in the description.

The evidence may not reach the genesis anchor. farAwayFromBeingSynced needs fb.height < headersHeight − 128, so at ≤70 blocks the ToDownloadProcessor change never fires in those runs. Could you run the script against a v6.0.5 jar carrying only the cache-key hunks? If L switches ~8/8, that settles it.

Nothing here looks blocked by #2535. All three production files auto-merge onto master; the four conflicts are in DeepRollBackSpec, CandidateGeneratorSpec and add/add on ConvergenceObservations{,Spec} — the latter two already upstream via #2512. git merge-tree --write-tree origin/master d62c945a4 is clean. Would you consider retargeting at that chain?

Also worth citing #1815 (open since 2022), with a note that this doesn't fix it. I'd be glad to see the cache change land on its own straight away.

[jozanek]

Genesis is picked only when no other rung matched — a fork rooted >512 back. continuationIdsV2 then answers heights 2..400, which can't reach a divergence at 1.5M, so it gets re-sent every round. And a fork that deep is fatally rejected by hdrTooOld (rule 209, keepVersions=200) on the peer we're serving.

Would you be open to clamping instead? offsets.map(o => Math.max(GenesisHeight, h - o)).distinct gives {50,34,1} at h=50 — the same fix your new tests cover — and today's exact four samples on mainnet, at no added cost.

[jozanek]

This looks like the load-bearing fix. getV2SyncInfo(hr, full = false) at :741 runs after every header batch and, under the old height-only key, stored a one-header summary that was then broadcast to the full = true callers at :358 and :492 — disabling the 16/128/512 ladder during active sync.

That mechanism isn't in the description, and it's the strongest thing here. Would you lead with it, and consider landing these three hunks alone? They merge onto master cleanly.

[jozanek]

keepVersions is the right concept here — FullBlockProcessor:105,133 and rule 209 already use it as the fork-root depth bound, so no objection to the value.

What I'd ask for is a name. Reached for directly, an operator raising it to 10000 for deep rollback turns a storage setting into a 10192-height scheduling walk — and the near-tip branch below hardcodes 100 for the same question.

Would a def maxRollbackDepth on the trait, used at all four sites, work for you?

[jozanek]

Couldn't settle this by reading: downloadRequired filters in-flight ids, so once this window is fully requested the accumulator stays empty and the loop re-walks 21..252 on every call — on the synchronizer actor thread, at up to 20 calls/s (50 ms floor, ErgoNodeViewSynchronizer:1403).

isInBestChain(fb.id) gates it and continuation early-exits, so this may be nothing. Do you have a measurement? If not, would resuming from the last scanned height beat re-walking?

[jozanek]

syncInfoV1CacheByHeadersHeight two lines up says the same thing as a tuple, and it keeps the identical stale-tip weakness — a stale 1000-id list to V1 peers after a same-height reorg.

The class does buy the SyncInfoV2CacheSpec seam, which I think justifies it. If it stays, would you route getV1SyncInfo through it too? Two cache shapes with two correctness properties in one file is the part I'd like to avoid.

[jozanek]

This took me a moment to place — an Int-overflow guard for the branches passing maxHeight = Int.MaxValue, where height + 1 wraps negative and the height > maxHeight check never fires. Correct, and otherwise behaviour-equivalent.

It's unrelated to the three changes the description covers, and unmentioned there. Could it get a sentence in the body, or land as its own one-liner?