validate_queries.sql: MS MARCO query 1267 fails only on concurrent-build path (rank-5 doc missing from top-10)

[tjgreen42]

Surfaced by the hardened benchmark validation watchdog in #360 on the concurrent-INSERT MS MARCO path. Distinct from #361 (different query, different failure mode).

Symptom

Validate MS MARCO concurrent results reports:

 query_id | docs_match | scores_match | missing_docs | extra_docs | max_abs_diff | details
----------+------------+--------------+--------------+------------+--------------+-----------------------------------
     1267 | f          | t            |            1 |          1 |     0.000003 | Missing: 1956260; Extra: 3906882

Key facts

• Query 1267: 3 ways a log can move when bucking on side hill (token bucket 5)
• Failure is on concurrent-INSERT path only. The single-txn COPY path validates this query cleanly.
• Doc 1956260 is rank 5 in ground truth (gt score 20.297, well above the rank-10 cutoff of 18.786). Not a rank-10 tie-break: it's solidly in the top-10 in GT but absent from the concurrent-build index's top-10.
• Overlapping docs (those in both index and GT top-10) match scores within 0.000003 — so the BM25 scoring formula is correct for what the index returns.
• The widened tie-break tolerance (0.01) in PR fix: sort posting list when spilling memtable (root-cause MS MARCO bucket-8 hang) #360 (dd466229) caught and forgave one of the two tied-cluster pairs (2205571 / 3906879); this remaining pair is the genuine non-tie discrepancy.

Hypotheses

1. Real concurrent-insert correctness issue. Despite the segment-sort fix (PR fix: sort posting list when spilling memtable (root-cause MS MARCO bucket-8 hang) #360) making bm25_check_segment_consistency return 0 inconsistencies, there may be another concurrent-path-specific bug — e.g., a tokenization race during tp_aminsert, a stale total_docs / avg_doc_len field, or BMW pruning incorrectly excluding doc 1956260.
2. Row never inserted. pgbench draws from insert_seq; if a transaction failed silently (constraint violation, etc.) doc 1956260 might be missing from msmarco_passages entirely. The index would correctly exclude it. Easy to diagnose: SELECT count(*) FROM msmarco_passages WHERE passage_id = 1956260 in the failing CI environment.
3. Stale ground truth. ground_truth_pg17.tsv was generated at some earlier index state; if doc 1956260's heap row has changed (or its tokenization differs in a corner case the GT precompute didn't model), GT might disagree with the current index even on a correctly-built version.

Repro & diagnosis

Diagnostic queries to run in a failing CI environment:

-- Was the doc actually inserted?
SELECT count(*) FROM msmarco_passages WHERE passage_id = 1956260;

-- What does the index return for the query?
SELECT passage_id,
       passage_text <@> to_bm25query('3 ways a log can move when bucking on side hill', 'msmarco_bm25_idx') AS neg_score
FROM msmarco_passages
ORDER BY passage_text <@> to_bm25query('3 ways a log can move when bucking on side hill', 'msmarco_bm25_idx')
LIMIT 20;

-- Standalone score for doc 1956260
SELECT passage_id, passage_text <@> to_bm25query('3 ways a log can move when bucking on side hill', 'msmarco_bm25_idx') AS neg_score
FROM msmarco_passages WHERE passage_id = 1956260;

-- Verify consistency check still passes
SELECT bm25_check_segment_consistency('msmarco_bm25_idx');  -- (function removed in #360 but can be restored from git)

Temporary mitigation

PR #360 allowlists query 1267 in validate_queries.sql so the watchdog can still catch new regressions without permanently red CI on this one query. Remove the allowlist entry once root cause is fixed.

[tjgreen42]

Updated evidence from benchmark run 25704684298 (post-merge of #360, on commit c68b991b):

Cross-run non-determinism

The failing query is not stable across runs:

Run	Failing query	Missing doc	gt rank of missing	gt score of missing	rank-10 cutoff	gap above cutoff
25700662521	1267	1956260	5	20.297	18.786	1.511
25704684298	130	5780442	8	29.157	28.882	0.275

In the second run, query 1267 passed cleanly despite the same allowlist entry being inert (allowlisted but not triggered). Different pgbench scheduling produces different rank-10 boundary outcomes — and in some runs different queries fail entirely.

What's the same in both failures

• scores_match: t — overlapping docs in both top-10s agree on scores within 0.001 (in fact within ~8e-6 in the latest run).
• docs_match: f — exactly 1 missing, 1 extra at the rank-10 boundary.
• Missing doc is at a genuinely higher gt rank than rank-10, not in the immediate tie-cluster.
• Only the concurrent-INSERT path. Single-txn COPY validates cleanly.

Inference

The pattern "overlapping docs agree perfectly, but one specific doc is mis-ranked by the index" plus "different doc each run" plus "only on pgbench-concurrent-insert" strongly suggests a per-doc indexing race during tp_aminsert under concurrent backends — not a stale ground truth, not a tokenization issue.

Specifically, for the missing-from-tapir doc to have a lower tapir score than its gt score (so it drops out), either:

1. The missing doc has fewer postings in the index than the document text actually contains (some term postings were dropped during a concurrent insert).
2. The missing doc's stored doc_length / fieldnorm is higher than reality (making BM25's length normalization penalize it more than it should).

The extra-in-tapir doc must symmetrically have:

3. More postings stored than it actually contains (got someone else's posting append by mistake), OR
4. Stored doc_length / fieldnorm lower than reality.

(1) and (3) together would be a lost-update / cross-doc-write race in the memtable posting list append path. (2) and (4) would be a similar race in the doc_lengths dshash table updates.

Tracking this here rather than per-query allowlisting; opening a deep investigation post-#360 merge.