Skip to content

Cold resolution fails on ChainHead disjointed, 41% of resolution failures #171

Description

@BigTava

ChainHead disjointed is the second largest cause of cold resolution failure, at 170 of the 412 failures in the last 7 days. Sentry groups it as DOTLI-2E, 1247 occurrences and 901 users since 2026-04-07, still firing as of today.

Reproduction Steps

No deterministic repro yet. It reproduces on its own against paseo.li at roughly 10% of cold resolutions, so the reliable path is to watch it rather than trigger it.

  1. Open a cold <label>.paseo.li in a fresh profile, so no CID cache and no warm smoldot.
  2. Repeat on both backends. It happens on smoldot-direct and on smoldot-shared-worker.
  3. Watch surface:host_main_resolve in Sentry. Roughly one in ten attempts ends with outcome:error and error.value:"ChainHead disjointed".

The synthetic prober already reproduces it several times a day across explore.paseo.li, getsome.paseo.li and hackme3.paseo.li.

Expected Behavior

A chainHead follow being dropped is a normal, recoverable smoldot event. The resolver should re-follow and retry the read, the same way it already handles StopError. The user should see the name resolve, at most a little slower.

Current Behavior

The read rejects, resolution fails, and the shell shows a resolution error.

Two separate defects make it worse than it needs to be.

1. DisjointError is not handled where its sibling StopError is.

withStopGuard in packages/resolver/src/api.ts:142-154 catches only StopError:

} catch (err) {
  if (err instanceof StopError) {
    markStopped(err);
    throw new ApiStoppedError(err);
  }
  throw err;
}

StopError and DisjointError are siblings from the same module, @polkadot-api/substrate-client/dist/chainhead/errors.js. Both mean the follow this read was issued against is gone. Only one of them is recognised, so DisjointError falls through the throw err and aborts the whole resolution.

2. The worker boundary discards the error type.

packages/protocol/src/client.ts:213 rebuilds every failed protocol response as a generic error:

const err = new Error(msg.error || "Unknown protocol error");
err.name = "ProtocolResponseError";

Only message survives the postMessage hop, so DisjointError arrives as ProtocolResponseError with a single stack frame pointing at that constructor. That is why every event in DOTLI-2E has the same useless culprit and no origin frame, and it is why callers cannot branch on the error type even if they wanted to retry. A separate Sentry group, DOTLI-9Q, holds the 14 events that surfaced as a real DisjointError outside that path.

There is already a comment acknowledging this failure mode at apps/protocol/src/protocol-shared-worker.ts:185, where sharing one Asset Hub follow through the broker was the mitigation. That helped the shared-worker path but did not remove the error, and it does not cover smoldot-direct at all.

Impact

Over the last 7 days, surface:host_main_resolve:

count share of failures
attempts (outcome:pending) 1689
resolveDotName timed out after 90000ms 234 57%
ChainHead disjointed 170 41%
Asset Hub sync timeouts 6 1%
other 2 <1%

That is a 24% overall failure rate for cold resolution, and this issue is 41% of it, so about 10% of all attempts.

Worth checking whether the two dominant rows are the same bug. A dropped follow that never rejects would present as the 90 second resolveDotName timeout rather than as this error, which would make the real share larger than 41%.

Scope

dimension breakdown
chain backend smoldot-direct 533, smoldot-shared-worker 329, unset 105
browser HeadlessChrome 797, Chrome 127, Chrome Mobile 12, Firefox 11, other 20
environment paseo.li
releases present in every release sampled from June to 9b66f3f5, so not a regression
handled yes, it is caught and reported, not a crash

Most of the volume is the synthetic prober, which is why the user count reads high. Real browsers account for roughly 170 of the 967 events in the 90 day window, so the user-facing rate is lower than the raw count suggests but the failure is real and not an artefact of monitoring.

Suggested fix

  1. Treat DisjointError like StopError in withStopGuard, and re-follow plus retry the read once rather than failing the resolution.
  2. Preserve the error name across the protocol worker boundary so callers can branch on it and so Sentry groups by real cause. Send the name alongside the message and reconstruct it in client.ts.
  3. Once 1 and 2 are in, recheck whether the 234 resolveDotName timeouts shrink. If they do, they were the silent half of this bug.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions