[shan-nvidia]

What

Add opt-in failed_record_policy: skip to multimodal retrieval SDG and the retrieval-structured column. A permanently malformed model response becomes a recorded failure after the existing native validation retries, allowing usable records to reach export. The default remains error.

Why

Currently one persistently invalid response can prevent an entire run from exporting, even though other responses are already cached. Explicit resume preserves the good work but can repeatedly fail on the same record. Operators need a way to continue while accounting for those failures.

Usage

Add this to a new run's existing multimodal YAML configuration:

failed_record_policy: skip

For example, if one context's query generation exhausts validation retries, its requested slots are recorded as failed and the other contexts continue through the existing quality gates. A failed required judgment or localization excludes the affected query. Original corpus units remain available.

Inspect generation_failures.json in the run directory and portable bundle, and report.json.generation_failures for counts by stage, failed requested slots, and excluded query IDs. Failed slots are separate from abstentions and quality rejections.

Generated queries remain in funnel.generated and the retention denominator if a later judgment or localization fails. Three generated queries with two accepted and one localization failure report three generated and 2/3 retention. Multiple failed judgments for the same query count once. Direct structured-column users receive None in configured trace and reasoning columns for failed records, allowing downstream expressions to handle even an entirely failed batch.

Omit the option, or use error, for strict behavior. Changing policy or implementation requires a new output directory because ordinary resume checks the frozen run identity.

How

• Catch only ModelGenerationValidationFailureError after Data Designer's existing native retry budget. Other errors and unexplained missing/duplicate identities remain errors.
• Persist successes and tagged failures as uniform JSON-string cells in skip mode, then validate and cache them under the existing request identities.
• Preserve record alignment and apply the disposition at each existing stage: omit failed enrichment/summaries, explicitly exclude failed summary judgments, and exclude affected query slots or queries.
• Reuse both successes and terminal failures on resume. Include the portable failure ledger in manifest checksums and verify the run ledger on completed resume.
• Preserve parser schemas, quality thresholds, corpus export, and the error when no queries pass. Add no retry loop or dependency changes; missing_response_attempts retains its existing separate meaning.

Validation

• make all passed: Ruff lint/format, isolated tests for every plugin, plugin validation, generated metadata and license checks, and the strict documentation build.
• 799 tests passed across devtools and plugins, including 423 retrieval plugin tests. Two optional Nemotron consumer tests skipped because that consumer is not installed.
• Regression coverage uses the real Data Designer parser, retry loop, batching, cache, workflow, and export with mocked model transport. It covers every affected stage, synchronous/asynchronous column behavior, provider-error propagation, missing/duplicate identities, all-failed stages, failure accounting, interrupted/completed resume, and tampered-cache/ledger rejection.
• Reproduced the generated-count and missing-trace-column bugs before fixing them. Regression tests check later-stage failure retention, multiple failures for one query, configured/disabled trace and reasoning columns, and real downstream expressions on an all-failed batch.
• Focused development checks used workspace Data Designer 0.9.4; isolated plugin CI installed Data Designer 0.9.5.
• make plugin-docs regenerated the documentation; git diff --check passed.

Linked Issue

Repository collaborator performing a focused maintenance fix.