perf(search): multi-index fan-out executes concurrently (#875)

[xerj-org]

The wildcard/multi-index search loop spawned one task per index and awaited it before spawning the next — serial by construction, so ax-* over N datasets cost the SUM of per-index latencies instead of the maximum.

Two-phase now: phase 1 spawns every per-index search up front (per-index request prep is sync map lookups and stays serial); phase 2 awaits and merges in the SAME deterministic index order, so hit ordering and agg-merge order are unchanged. Concurrency is bounded by the existing global search permit (Index::search → governor.acquire_search, sized by limits.max_concurrent_searches); excess searches queue deadline-aware on that semaphore.

What this is worth, measured honestly

The issue opened with a 701 ms ax-* p50 against 11.7 ms for the same query on one index. That baseline does not reproduce on a settled corpus, and the PR should not claim it. Re-measured on the identical corpus state, released rc.71 (serial) as the control, p50 of 21:

query	rc.71 serial	this PR
identifier ax-*	3.3 ms	2.7 ms
common word ax-*	1.8 ms	1.5 ms
phrase ax-*	7.7 ms	6.6 ms

Result parity: identical totals and top-hit scores.

The original 701 ms was compound — the serial loop multiplying a per-index cost that was itself pathological (~17 ms/index while 100k-doc datasets sat memtable-resident during post-ingest merge churn, the #873/#876 state). So this change bounds ax-* by max(per-index) instead of Σ(per-index): ~1.2× when indices are cheap, and the ~N× the issue measured when they are not. It is the insurance half; #878 and #876 remove the expensive state itself.

The merge gate is the deterministic concurrency test, not the latency table.

Proof

wildcard_fanout_runs_per_index_searches_concurrently asserts BOTH per-index searches START while NEITHER can complete — impossible under the serial loop (the second never spawned until the first resolved), immediate under the fan-out. Abort propagation to all spawned searches is asserted in the same test. The pending-search test injection was extended to a Vec of signal pairs with prefix-matched index names to make that expressible.

Behaviour changes, disclosed

• queries_by_index now increments for every resolvable index even when a later index errors the request — those searches really do run now.
• Index resolution happens for all indices before any await, so a request naming both a missing index and an erroring index now returns the 404 rather than the search error. Closer to ES, which resolves at the coordinator first.
• Per-index search deadlines all start at request time rather than sequentially, so a search queued behind the global permits can return timed_out: true with partial results where the serial loop would have returned complete results slowly. Only reachable when ceil(N/permits) × per-index latency exceeds the timeout.

Review follow-ups

Raised and deliberately not taken in this PR, recorded for #875: the fan-out is unbounded per request (up to N tasks, capped process-wide by the 64-permit pool rather than per-request). Bounding it to min(N, cpus) would cost nothing and remove a fairness/peak-memory question on single-user deployments; it is a small follow-up rather than a change to make under a release gate.

Closes #875.

🤖 Generated with Claude Code

[xerj-org]

Same-state A/B, and an honest correction to the headline expectation.

Measured today on the identical corpus state (41 autoindex datasets, settled and merged), same 21-rep p50 method, released rc.71 (serial loop) as the control:

query	rc.71 serial	this PR	single index
identifier ax-*	3.3 ms	2.7 ms	0.3 ms
common word ax-*	1.8 ms	1.5 ms	—
phrase ax-*	7.7 ms	6.6 ms	—

Result parity: identical totals and top-hit scores across both arms.

The issue's 701 ms baseline does NOT reproduce on a settled corpus — that number was compound: the serial loop multiplying a per-index cost that was itself pathological (~17 ms/index while 100k-doc datasets sat memtable-resident during post-ingest merge churn — the #873/#876 state).

What this PR is worth, stated precisely: it bounds ax-* latency by max(per-index) instead of sum(per-index). That is ~1.2x when indices are cheap (above), and the ~Nx the issue measured when they are not (N=41 expensive indices: sum ~700 ms vs max ~17 ms). It is the insurance half; #878 (idle-age flush) removes the expensive state itself. The deterministic concurrency test, not the latency table, is the merge gate.