bench: Wikipedia validation is broken — ~0% top-10 doc overlap with ground truth (was silently masked)

[tjgreen42]

Summary

The Wikipedia benchmark validator (benchmarks/datasets/wikipedia/validate_queries.sql) reports that 0/80 queries have matching top-10 doc sets with the bundled ground_truth.tsv, and 1/80 has a real score-diff of ~0.086. This has been latent for an unknown period and was masked by two compounding bugs that are now fixed.

Discovery

Surfaced by the benchmark run on PR #367:

• https://github.com/timescale/pg_textsearch/actions/runs/25713784697

Pre-#367 (and pre-#360), the same failure mode existed but two layered bugs hid it:

1. MS MARCO validator aborted before cleanup. benchmarks/datasets/msmarco/validate_queries.sql:176 had an ambiguous query_id SQL error. With \set ON_ERROR_STOP on this halted the script before the closing DROP FUNCTION IF EXISTS validate_single_query; ran. That left the MS MARCO validate_single_query(int, text, float8) overload in the bench_test database.
2. Wikipedia validator defines a colliding overload. validate_queries.sql:34 defines validate_single_query(int, text, int DEFAULT 4). With the MS MARCO overload still around, LATERAL validate_single_query(q.query_id, q.query_text) at line 141 became:

   ERROR:  function validate_single_query(integer, text) is not unique
   

3. Watchdog masked both failures. Before fix: sort posting list when spilling memtable (root-cause MS MARCO bucket-8 hang) #360, the workflow's Wikipedia validation step only grepped for VALIDATION FAILED. The SQL error produced neither FAILED nor PASSED markers, and set -o pipefail was not in effect, so Wikipedia validation PASSED was echoed unconditionally.

PR #360 fixed (1) and (3). Now MS MARCO reaches its DROP, and Wikipedia validation runs to completion — exposing the real GT/index mismatch.

What the real validation output looks like

From PR #367 benchmark run:

 total_queries | docs_match_count | scores_match_count | docs_match_pct | scores_match_pct | worst_abs_diff
---------------+------------------+--------------------+----------------+------------------+----------------
            80 |                0 |                 79 |            0.0 |             98.8 |       0.086028

• 0/80 queries have full top-10 doc match with GT
• The 20 worst (sorted by missing_docs DESC) all show 10 missing + 10 extra (completely disjoint top-10s), max_abs_diff=0.000000 — score comparison is vacuous when no docs overlap
• 1 query has a real score diff of ~0.086 (> 4 decimal places) → fails the score-match gate

Likely causes (not investigated yet)

• Wikipedia ground_truth.tsv was generated against a different corpus snapshot, different doc_id assignment, or different tokenization config than what download.sh / load.sql produces today.
• Or it was never regenerated after a stemmer / text_config / data-source change. (We've already hit this on MS MARCO at validate_queries.sql: MS MARCO query 158 fails by 1.5 score points (pre-existing tokenization discrepancy) #361 with Snowball stemmer drift.)

Why this is a separate issue from #365

PR #367 fixes #365 (BMW K-dependent score under-reporting). That fix:

• Makes MS MARCO validation pass (it was failing before)
• Passes all 17 standard CI checks (sanitizers pg17/18, performance, scoring1..6, bmw, wand, bmw_skip_advance, etc.)
• Does not change algorithmic invariants in a way that would zero out top-10 overlap on a 100K-doc corpus

The Wikipedia 0%-overlap pattern is structural, not a scoring drift.

Proposed work

1. Decouple validator function names. Rename validate_single_query → validate_msmarco_query / validate_wikipedia_query (or always require a 3-arg call). Eliminates the cross-dataset overload-collision foot-gun even if a future SQL error skips cleanup again.
2. Regenerate benchmarks/datasets/wikipedia/ground_truth.tsv against the current download.sh+load.sql corpus snapshot using precompute_ground_truth.sql. Compare against current Tapir output; investigate any remaining real discrepancies the same way we did on MS MARCO.
3. Track the single score-diff query (~0.086). Document it once GT is regenerated; either it disappears, or it's a real bug like BMW reports under-scored results for some docs when K (default_limit) is much larger than the actual LIMIT #365.

Unblocking PR #367 / v1.2

PR #367 contains the v1.2 blocker fix for #365. The Wikipedia regression here is pre-existing and orthogonal. Suggest one of:

• (a) Temporarily allow the Wikipedia validation step to soft-fail / be marked continue-on-error: true until GT is regenerated
• (b) Merge a stub benchmarks/datasets/wikipedia/known_mismatches allowlist akin to MS MARCO's, accepting the current Wikipedia state for now
• (c) Block fix(bmw): include non-pivot term max scores in block-skip safety check (#365) #367 on regenerating Wikipedia GT first

Recommend (a) for speed — the validator's existence is more important than its current accuracy on Wikipedia, and option (b) requires understanding which docs are 'expected' to mismatch which we currently can't justify.

[tjgreen42]

Resolved by PR #372 (62308b82).

The validator was switched from doc-set comparison to per-rank score comparison (mirroring msmarco/validate_queries.sql), which correctly absorbs the tie-cluster reordering between BMW and ground-truth that was producing the 0/80 doc-overlap symptom.

Current state on main (2026-05-28 nightly benchmark, run 26560367054):

VALIDATION PASSED: All 80 queries match within tolerance (0 allowlisted known mismatches)

The single ~0.086 score divergence reported in this issue is no longer flagged: with per-rank comparison at 0.001 tolerance, only true rank-position score drift fails, and there's none.

Two of the three proposed work items here were not done — they are footgun-prevention / hygiene rather than current bugs, and can be filed separately if desired:

• Decouple the validate_single_query function names between msmarco and wikipedia (still a footgun if one validator's SQL aborts before DROP FUNCTION again).
• Regenerate benchmarks/datasets/wikipedia/ground_truth.tsv against the current corpus snapshot (the per-rank approach makes this optional rather than blocking).

Closing as the reported broken behaviour is fixed.