[shan-nvidia]

What

Stacked on #89. Make grouped query-disjoint splitting over a shared full corpus the portable retrieval export protocol.

Why

In-domain adaptation retrieves new queries against the customer's existing collection. Document overlap is intentional; known query rewrites must stay together. This new feature does not retain document-disjoint export compatibility.

Usage

from data_designer_retrieval_sdg.retrieval import export_retrieval_data, SplitRatios

export_retrieval_data("generated.jsonl", "bundle", dataset_id="service-policies",
                      ratios=SplitRatios(train=0.8, validation=0, evaluation=0.2))

Optionally attach query_provenance keyed by final candidate index, e.g. {"0": {"query_group_id": "warranty", "seed_query_id": "seed-42"}}. Parent query IDs are supported too. Missing provenance produces singleton groups joined by normalized duplicates. Lexical near-duplicate grouping is opt-in with group_near_duplicates=True; unknown semantic equivalents cannot be guaranteed detectable.

How

Resolve explicit groups, derivations and normalized duplicates before splitting. Assign groups once across export views. Do not group independent queries merely because they share a document, page, summary or context. Retain existing acceptance gates unchanged. Each view uses a shared full eligible corpus, including distractors; training/evaluation query assignments remain disjoint. No dataset-specific adapters or new VLM defaults. Paired consumer: NVIDIA-NeMo/Nemotron#353.

Validation

• make sync, make format, make all: passed (lint, all isolated plugin tests, validation, metadata checks, strict docs).
• Retrieval plugin: 371 passed, one optional cross-repository test skipped in the standalone environment.
• Cross-repository test with revised Nemotron353 on PYTHONPATH: passed using actual DD export and full recipe handoff validation.
• Additional tests cover explicit/derived groups, normalized/optional near duplicates, shared-document questions crossing splits, full corpus retention and deterministic assignments.

No changes were deployed to ongoing experiment workers or their frozen input/config/source snapshots.

[shan-nvidia]

Superseded by the consolidated generic multimodal SDG EA candidate in #92, with recipe integration in NVIDIA-NeMo/Nemotron#359. Generic grouping/export work is retained there. Closing the duplicate proposal only: this branch and commit history are preserved for existing pinned consumers. No running experiment is migrated.