fix: block-staleness watchdog for substrate subscription

[dylanuys]

Problem

SubstrateConnectionManager.start_subscription() only reconnects when the connect/subscribe coroutine raises. On Aug 12 a validator's data service hit a reconnect that hung instead: the websocket dropped (Unable to reconnect because there are currently open subscriptions), the retry loop logged Connecting to async substrate..., and the AsyncSubstrateInterface connect stalled forever — no exception, no return.

Every existing safety layer misses this mode:

• the manager's retry loop is stuck awaiting the hung connect;
• callers' substrate_task.done() checks (data_service.start(), BaseNeuron.check_substrate_connection()) never fire because a hung task is never done;
• the validator heartbeat watches self.step, which increments every second in the main loop regardless of chain connectivity.

Result: the data service ran as a zombie for a week — process "online", no block callbacks, no dataset downloads, no HF uploads (~20k media backlog).

Fix

Run connect+subscribe as an inner task and watch block arrival — the one signal that's true in every failure mode. Every block stamps last_block_time; if no block lands for 120s (finney block time ~12s), cancel the inner task and rebuild the connection. This also bounds hung connects. The raise-and-retry path is preserved.

Public interface (start_subscription_task() / stop()) is unchanged, so the validator and data service both inherit the fix with zero call-site changes; their existing .done() checks remain as an outer layer.

Verification

Simulated four scenarios against the patched class (subclass with fake _connect_and_subscribe):

• hung connect → detected via staleness, cancelled, reconnected, blocks resume
• connect raises → reconnects (existing behavior preserved)
• subscription goes silent after delivering blocks → detected, reconnected
• healthy stream → no false reconnects

Also running in production on the affected validator's data service since 2026-08-20; block subscription and hourly HF upload cycles healthy.