[shan-nvidia]

Add a multimodal retrieval data generation pipeline alongside the existing text QA workflow.

The new pipeline takes prepared text passages, images, or mixed collections and generates retrieval queries paired with supporting source units and graded relevance labels. It exports a portable dataset containing the original corpus, training records, synthetic evaluation queries, and the evidence behind each accepted query.

Why

Retrieval training needs queries grounded in the material a model will retrieve, including information found in figures, tables, and combinations of sources. This workflow supports those inputs and produces a shared dataset handoff for downstream hard-negative mining, fine-tuning, and evaluation with Nemotron and AutoModel.

How

• Plan generation around the source collection. Generate from individual units, documents, or sections, with optional embedding-based combinations of related sections. Planning summaries supplement the original source content.
• Generate and assess grounded queries. Support varied query types and explicit abstention when a requested task is unsupported. Assess relevance, self-sufficiency, answer leakage, and localized source support, retaining accepted and rejected candidates for inspection.
• Preserve evidence through export. Export the full corpus with grouped query splits and text, image, combined, and mixed views. Each view retains a query only when all its supporting sources are representable.
• Support bounded, resumable runs. Bound model requests, validate structured responses, cache completed work, and publish complete bundles with integrity checks.
• Separate the two workflows. Move the existing text QA implementation under text_qa and expose the new pipeline under multimodal. The text QA CLI and package-root API remain available; direct submodule imports move to text_qa.

The documentation includes a prepared NVIDIA hardware example, a generation guide, configuration reference, and guidance on evidence and export views.

Usage

From the repository root:

uv pip install ./plugins/data-designer-retrieval-sdg
python -m data_designer_retrieval_sdg.multimodal --config sdg.yaml

See the first-run guide for source preparation and model configuratio

[shan-nvidia]

Companion recipe integration is now available at NVIDIA-NeMo/Nemotron#359 (stacked above the #351 consumer work). It pins this PR at c786eb3, selects the retrieval-first multimodal API, and keeps AutoModel #3915/native HNM unchanged. All 86 focused recipe/profile/handoff tests passed; both stage lockfiles validate. Hosted quality and the full GPU release-qualification run remain outstanding; neither PR publishes release assets.

[shan-nvidia]

Recipe integration is now directly in NVIDIA-NeMo/Nemotron#351 at commit 6c27b5ec94e31be5f1641bbc45f47dfbefc8fefe; no stacked recipe PR is required. It pins this producer at c786eb3. The former companion #359 and its supporting #353/#356/#357/#358 stack are consolidated and closed, with branches/commits retained. Validation on the direct consolidation: 198 focused tests passed, 10 skipped; Ruff and both dependency locks passed. Existing experiment sources remain unchanged; release qualification is still pending.