ci/bench: MS MARCO concurrent validation regression — queries 9351 and 9340 fail per-rank score check on main (post-#366)

[tjgreen42]

Summary

The nightly Benchmarks workflow on main has been failing for 8+ consecutive
runs (last success: run 25420768617
on 2026-05-06). The most recent fully-completed run
(25846493777,
scheduled 2026-05-14 06:54Z) failed at:

psql:benchmarks/datasets/msmarco/validate_queries.sql:240:
  NOTICE: VALIDATION FAILED: 2 of 80 queries failed
    (per-rank scores differ beyond 0.001 tolerance, excluding allowlist)
ERROR: MS MARCO concurrent validation FAILED!

The two failing queries are 9351 and 9340, both with the same
shape as #363 (now closed): per-rank score swaps among docs with
near-tied BM25 scores on the concurrent-build path.

Evidence

From the artefact msmarco_concurrent_validation.log for run 25846493777:

query_id	scores_match	docs_match	max_abs_diff	worst_rank	details (excerpt)
158	f	f	0.621404	4	(allowlisted — #361, stale GT)
9351	f	t	0.049454	2	rank 2: gt=11.6665 (doc 8794243), tapir=11.6170 (doc 7162299); rank 3: gt=11.6170 (doc 7162299), tapir=11.6665 (doc 8794243)
9340	f	t	0.017698	8	rank 6: gt=12.8823 (doc 8799571), tapir=12.8725 (doc 1370509); rank 7: gt=12.8725 (doc 1370509), tapir=12.8646 (doc 8612630); rank 8: gt=12.8646 (doc 8612630), tapir=12.8823 (doc 8799571)

Both failures are clean adjacent-rank swaps (docs_match=t, same top-10
doc set as ground truth — only the order differs). The score deltas are
small (0.018 and 0.049, vs the 0.001 tolerance).

Why this is not #363 / not fixed by #366

#363 was closed by #366. PR #366's body explicitly lists 9351 and 9340
in its scope:

the "concurrent-INSERT-only" validation failures repeatedly hit after
#360 (queries 1267, 130, 1517, 222, 9351, 9340 across runs)

#366's theory was that the validator was asking BMW for K=1000 instead
of K=10, hitting the BMW K-scaling bug from #365. The fix was to inline
LIMIT 10 into the inner subquery so it propagates to the BM25 index
scan. That fix is present on main today
(benchmarks/datasets/msmarco/validate_queries.sql:96-103):

SELECT row_number() OVER ()::int as rank, t.passage_id, t.score FROM (
    SELECT
        passage_id,
        -(passage_text <@> to_bm25query(p_query_text, 'msmarco_bm25_idx'))::float8 as score
    FROM msmarco_passages
    ORDER BY passage_text <@> to_bm25query(p_query_text, 'msmarco_bm25_idx')
    LIMIT 10
) t;

Yet queries 9351 and 9340 still fail on the concurrent-build path on
main post-#366. That means the K=1000 hypothesis was not the
complete root cause for these two queries. The other four queries
mentioned in #366 (1267, 130, 1517, 222) do appear stable post-#366.

Likely root cause (hypothesis)

Adjacent-rank swaps among docs whose BM25 scores differ by ~0.02–0.05
are consistent with non-deterministic segment layout producing
slightly different per-segment top-K candidates depending on which
docs landed in which segment. The concurrent-pgbench build path
generates segments with different doc-to-segment assignments than a
serial INSERT, and the standalone validator's <@> recomputation
runs over the full corpus (no segments) → ground-truth ordering.

If two docs have BM25 scores 11.6665 and 11.6170 (Δ=0.0495), a tiny
per-segment normalization difference (avg doc length, IDF rounding,
etc.) can flip the BMW comparison. The scores reported in the log
after the swap (tapir=11.6170 at rank 2 and tapir=11.6665 at
rank 3) are the standalone-recomputed scores, not the BMW scores — so
this looks like BMW assigning rank 2 to doc 7162299 even though doc
8794243 (the GT rank 2) has a marginally higher score. Worth
verifying with pg_textsearch.log_scores enabled.

Cascade effect on the rest of the job

The failure at this step causes all subsequent steps in insert-benchmark
to be skipped (the wikipedia concurrent insert / queries / validation
steps). Their metric files get emitted by the if: always() extraction
step with all fields null, the formatter (format_for_action.sh)
correctly produces [], and the publish step fails with:

No benchmark result was found in .../wikipedia_concurrent_action.json.
Benchmark output was '[]'

That cascade has been the visible "Wikipedia concurrent failure" in
recent runs, but it is not an independent bug. Fixing the MS MARCO
concurrent regression unblocks the wikipedia_concurrent publish step.

Repro

1. Run the workflow against the current main (e.g. via workflow_dispatch
   with dataset=all and wikipedia_size=100K).
2. The job insert-benchmark will fail at the
   "Validate MS MARCO concurrent results" step with the message above.
3. Inspect msmarco_concurrent_validation.log in the artefact for
   per-query failure detail.

Proposed work

1. Add pg_textsearch.log_scores = on for one bench run and capture
   BMW vs standalone scores for queries 9351, 9340 on the
   concurrent-build path. Confirm whether the discrepancy is in BMW
   scoring or in upstream segment statistics.
2. If BMW scoring: investigate whether block_max_* upper bounds in
   the concurrent-build segment layout differ from a serial-build
   layout in a way that causes near-tied docs to be skipped/included
   differently.
3. If segment statistics: trace total_docs / total_tokens /
   per-term doc_freq for the affected segments and compare against a
   serial-build run. (Recall: VACUUM does not refresh these, but
   that's not in play here since we're on a fresh load.)
4. Until root-caused, either add (9351, 9340) to the known_mismatches
   allowlist with a link to this issue (unblocks the nightly bench
   workflow on main) or relax the tolerance for adjacent-rank
   swaps where max_abs_diff < 0.05 and docs_match = true (cleaner
   but may mask a real regression class).

Side note: full-benchmark on 2026-05-12

Run 25718524040
also failed full-benchmark (separately from insert-benchmark).
Different log shape; potentially worth its own ticket if it
reproduces.

[tjgreen42]

Not reproduced on post-merge main

Post-merge of #375, manually-dispatched Benchmarks run 25894810538 ran insert-benchmark cleanly:

VALIDATION PASSED: All 79 queries match within tolerance
  (1 allowlisted known mismatches)

Only query 158 (the stemmer-drift case from #361) shows up in the failure detail; queries 9351 and 9340 did not reproduce. So this PR appears to dodge the failure mode — most likely because the on-disk memtable v2 changes the concurrent-build segment layout enough that the near-tied-score adjacency that triggered the per-rank swap no longer arises in the same way.

However, leaving this issue open: the underlying behaviour was empirically non-deterministic on the old code (varied run-to-run, surfaced different query IDs over time). Memtable v2 happening to dodge the race is not a root-cause fix. If a future change shifts segment timings, the swap pattern could resurface — possibly on different queries. The investigation steps in the original issue are still worth doing once we have a free hand.

Monitoring tomorrow's scheduled run; if it stays green for ~7 consecutive nights I'll downgrade to "monitor and document" status.

[tjgreen42]

Closing as no longer reproducible after the on-disk memtable rewrite in #375. The originally reported queries stopped failing immediately after that merge, and the latest scheduled benchmark on current main (f8616b84, run 35062172854) again passed the MS MARCO concurrent validation, including queries 9351 and 9340.

We never established a definitive root cause for the old architecture, so please reopen if this failure pattern recurs.