@@ -16,7 +16,8 @@ image paths inside source records resolve relative to their JSONL file.
1616| ` source_directory ` | ` null ` | Directory preparation settings below. Mutually exclusive with ` sources_file ` . |
1717| ` output_dir ` | Required | New run directory, or an existing compatible directory when resuming. |
1818| ` dataset_id ` | Required | Nonempty corpus identifier recorded in exports. |
19- | ` generator ` , ` judge ` | Required | Independent model settings, described below. |
19+ | ` generator ` , ` judge ` | Required | Generator settings and default judge settings, described below. |
20+ | ` judges ` | ` {} ` | Optional complete model settings for individual judgment stages. |
2021| ` contexts_file ` | ` null ` | Explicit context JSONL; replaces automatic context grouping. |
2122| ` concurrency ` | ` 8 ` | Model-call concurrency, from 1 to 128. |
2223| ` batch_size ` | ` 30 ` | Request batch size, from 1 to 128. |
@@ -27,7 +28,7 @@ image paths inside source records resolve relative to their JSONL file.
2728| ` ratios ` | ` {train: 0.8, validation: 0.0, evaluation: 0.2} ` | Query-group split fractions, each from 0 to 1 and summing to 1. |
2829| ` group_near_duplicates ` | ` false ` | Add conservative lexical query grouping before splitting. |
2930
30- The plugin groups requests by model role , response schema, and presence of images.
31+ The plugin groups requests by resolved model settings , response schema, and presence of images.
3132Each batch runs a Data Designer structured-response column. Matching requests
3233reuse cached responses keyed by their prompt, schema, model settings, and image
3334content.
@@ -276,14 +277,91 @@ reasoning models, `max_tokens` can include both reasoning and the final response
276277
277278To enable local token preflight for a role, set `context_window_tokens`,
278279` tokenizer_file` , and `request_overhead_tokens`. Also set
279- ` image_tokens_per_image` when the corpus contains images. Use values appropriate
280+ ` image_tokens_per_image` for the generator, relevance judge, and localization
281+ judge when the corpus contains images. Text-only judgment stages do not require
282+ an image allowance. Use values appropriate
280283to the deployment. The check includes the structured-response instructions,
281284source input, image allowances, and output reserve.
282285
283286Without local token preflight, Data Designer's rendered-prompt limit still
284287applies, but it is not a model context-window guarantee. Provider context and
285288image limits remain applicable.
286289
290+ # ## Judge Model Overrides
291+
292+ Use `judge` for the default judgment model. Override individual stages under
293+ `judges` when a different deployment or sampling configuration suits the task :
294+
295+ | Stage | Input | Configuration key |
296+ | --- | --- | --- |
297+ | Summary quality | Generated summary and persona | `judges.summary` |
298+ | Query classification and answer leakage | Query only | `judges.query_metadata` |
299+ | Self-sufficiency | Query and source text | `judges.self_sufficiency` |
300+ | Relevance | Query, source text, and available images | `judges.relevance` |
301+ | Support localization | Query, source text, and available images | `judges.localization` |
302+
303+ An omitted or `null` stage uses `judge`. Each override is a complete
304+ [Model Settings](#model-settings) object. Required fields must be supplied;
305+ optional fields use their own defaults rather than inheriting values from
306+ ` judge` . This includes sampling parameters, output limits, credentials, and
307+ local token budgets. Configure vision-capable deployments for image-bearing
308+ requests; the plugin does not infer model capabilities or switch models on errors.
309+
310+ This configuration uses one text judge for summary and query checks, and the
311+ default vision judge for relevance and support localization. Replace the model
312+ identifiers and endpoints with your serving deployments, and set both credential
313+ environment variables before running :
314+
315+ ` ` ` yaml
316+ sources_file: sources.jsonl
317+ output_dir: runs/stage-judges
318+ dataset_id: service-manuals
319+ generator:
320+ model: your-generation-model
321+ endpoint: https://generation.example.com/v1
322+ credential_env: GENERATION_API_KEY
323+ judge:
324+ model: your-vision-judge-model
325+ endpoint: https://vision.example.com/v1
326+ credential_env: JUDGE_API_KEY
327+ temperature: 0
328+ max_tokens: 4096
329+ judges:
330+ summary: &text_judge
331+ model: your-text-judge-model
332+ endpoint: https://text.example.com/v1
333+ credential_env: JUDGE_API_KEY
334+ temperature: 0
335+ max_tokens: 1024
336+ query_metadata: *text_judge
337+ self_sufficiency: *text_judge
338+ ` ` `
339+
340+ You can also configure `judges.relevance` and `judges.localization` independently.
341+ For Python configuration, import `JudgeModels` from
342+ ` data_designer_retrieval_sdg.multimodal` . `config.model_for("judge", "relevance")`
343+ returns the effective settings for that stage.
344+
345+ Automatic section planning (`context_strategy : sections` without `contexts_file`)
346+ uses the fallback `judge` for text-only table-of-contents confirmation. This judge
347+ remains active even when all five named stages are overridden or
348+ `judge_summaries : false`. Its local token budget is validated before creating the
349+ run, and its tokenizer content hash is included in the frozen run identity.
350+ Explicit contexts bypass this confirmation step.
351+
352+ `judge_summaries : false` disables the summary judgment stage, including its
353+ model-budget validation. The run manifest records the fallback under
354+ ` models.judge` and the effective models for enabled stages under `models.judges`.
355+ ` shared_generator_judge` is true only when every active judge, including the
356+ fallback used for TOC confirmation, uses the generator's model identifier and endpoint.
357+
358+ Cached responses include the resolved model settings and tokenizer content hash.
359+ Changing one override cannot reuse that stage's incompatible response. Changing
360+ settings in an existing run still violates its frozen identity; use a new output
361+ directory. Omitting overrides preserves the original judge request cache keys.
362+ Stage configuration does not establish better judgment quality; compare reviewed
363+ examples when choosing models.
364+
287365# # Context Planning
288366
289367A context specifies the units available for one generation request. These fields
0 commit comments