Skip to content

Commit ef6b6da

Browse files
Allow separate models for multimodal retrieval judges (#103)
* Allow separate models for multimodal retrieval judges Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> * Include the TOC judge in validation and provenance Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com> --------- Signed-off-by: Oliver Holworthy <1216955+oliverholworthy@users.noreply.github.com>
1 parent daf67f2 commit ef6b6da

10 files changed

Lines changed: 628 additions & 31 deletions

File tree

‎docs/plugins/data-designer-retrieval-sdg/multimodal-reference.md‎

Lines changed: 81 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@ image paths inside source records resolve relative to their JSONL file.
1616
| `source_directory` | `null` | Directory preparation settings below. Mutually exclusive with `sources_file`. |
1717
| `output_dir` | Required | New run directory, or an existing compatible directory when resuming. |
1818
| `dataset_id` | Required | Nonempty corpus identifier recorded in exports. |
19-
| `generator`, `judge` | Required | Independent model settings, described below. |
19+
| `generator`, `judge` | Required | Generator settings and default judge settings, described below. |
20+
| `judges` | `{}` | Optional complete model settings for individual judgment stages. |
2021
| `contexts_file` | `null` | Explicit context JSONL; replaces automatic context grouping. |
2122
| `concurrency` | `8` | Model-call concurrency, from 1 to 128. |
2223
| `batch_size` | `30` | Request batch size, from 1 to 128. |
@@ -27,7 +28,7 @@ image paths inside source records resolve relative to their JSONL file.
2728
| `ratios` | `{train: 0.8, validation: 0.0, evaluation: 0.2}` | Query-group split fractions, each from 0 to 1 and summing to 1. |
2829
| `group_near_duplicates` | `false` | Add conservative lexical query grouping before splitting. |
2930

30-
The plugin groups requests by model role, response schema, and presence of images.
31+
The plugin groups requests by resolved model settings, response schema, and presence of images.
3132
Each batch runs a Data Designer structured-response column. Matching requests
3233
reuse cached responses keyed by their prompt, schema, model settings, and image
3334
content.
@@ -276,14 +277,91 @@ reasoning models, `max_tokens` can include both reasoning and the final response
276277

277278
To enable local token preflight for a role, set `context_window_tokens`,
278279
`tokenizer_file`, and `request_overhead_tokens`. Also set
279-
`image_tokens_per_image` when the corpus contains images. Use values appropriate
280+
`image_tokens_per_image` for the generator, relevance judge, and localization
281+
judge when the corpus contains images. Text-only judgment stages do not require
282+
an image allowance. Use values appropriate
280283
to the deployment. The check includes the structured-response instructions,
281284
source input, image allowances, and output reserve.
282285

283286
Without local token preflight, Data Designer's rendered-prompt limit still
284287
applies, but it is not a model context-window guarantee. Provider context and
285288
image limits remain applicable.
286289

290+
### Judge Model Overrides
291+
292+
Use `judge` for the default judgment model. Override individual stages under
293+
`judges` when a different deployment or sampling configuration suits the task:
294+
295+
| Stage | Input | Configuration key |
296+
| --- | --- | --- |
297+
| Summary quality | Generated summary and persona | `judges.summary` |
298+
| Query classification and answer leakage | Query only | `judges.query_metadata` |
299+
| Self-sufficiency | Query and source text | `judges.self_sufficiency` |
300+
| Relevance | Query, source text, and available images | `judges.relevance` |
301+
| Support localization | Query, source text, and available images | `judges.localization` |
302+
303+
An omitted or `null` stage uses `judge`. Each override is a complete
304+
[Model Settings](#model-settings) object. Required fields must be supplied;
305+
optional fields use their own defaults rather than inheriting values from
306+
`judge`. This includes sampling parameters, output limits, credentials, and
307+
local token budgets. Configure vision-capable deployments for image-bearing
308+
requests; the plugin does not infer model capabilities or switch models on errors.
309+
310+
This configuration uses one text judge for summary and query checks, and the
311+
default vision judge for relevance and support localization. Replace the model
312+
identifiers and endpoints with your serving deployments, and set both credential
313+
environment variables before running:
314+
315+
```yaml
316+
sources_file: sources.jsonl
317+
output_dir: runs/stage-judges
318+
dataset_id: service-manuals
319+
generator:
320+
model: your-generation-model
321+
endpoint: https://generation.example.com/v1
322+
credential_env: GENERATION_API_KEY
323+
judge:
324+
model: your-vision-judge-model
325+
endpoint: https://vision.example.com/v1
326+
credential_env: JUDGE_API_KEY
327+
temperature: 0
328+
max_tokens: 4096
329+
judges:
330+
summary: &text_judge
331+
model: your-text-judge-model
332+
endpoint: https://text.example.com/v1
333+
credential_env: JUDGE_API_KEY
334+
temperature: 0
335+
max_tokens: 1024
336+
query_metadata: *text_judge
337+
self_sufficiency: *text_judge
338+
```
339+
340+
You can also configure `judges.relevance` and `judges.localization` independently.
341+
For Python configuration, import `JudgeModels` from
342+
`data_designer_retrieval_sdg.multimodal`. `config.model_for("judge", "relevance")`
343+
returns the effective settings for that stage.
344+
345+
Automatic section planning (`context_strategy: sections` without `contexts_file`)
346+
uses the fallback `judge` for text-only table-of-contents confirmation. This judge
347+
remains active even when all five named stages are overridden or
348+
`judge_summaries: false`. Its local token budget is validated before creating the
349+
run, and its tokenizer content hash is included in the frozen run identity.
350+
Explicit contexts bypass this confirmation step.
351+
352+
`judge_summaries: false` disables the summary judgment stage, including its
353+
model-budget validation. The run manifest records the fallback under
354+
`models.judge` and the effective models for enabled stages under `models.judges`.
355+
`shared_generator_judge` is true only when every active judge, including the
356+
fallback used for TOC confirmation, uses the generator's model identifier and endpoint.
357+
358+
Cached responses include the resolved model settings and tokenizer content hash.
359+
Changing one override cannot reuse that stage's incompatible response. Changing
360+
settings in an existing run still violates its frozen identity; use a new output
361+
directory. Omitting overrides preserves the original judge request cache keys.
362+
Stage configuration does not establish better judgment quality; compare reviewed
363+
examples when choosing models.
364+
287365
## Context Planning
288366

289367
A context specifies the units available for one generation request. These fields

‎plugins/data-designer-retrieval-sdg/docs/multimodal-reference.md‎

Lines changed: 81 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,8 @@ image paths inside source records resolve relative to their JSONL file.
1616
| `source_directory` | `null` | Directory preparation settings below. Mutually exclusive with `sources_file`. |
1717
| `output_dir` | Required | New run directory, or an existing compatible directory when resuming. |
1818
| `dataset_id` | Required | Nonempty corpus identifier recorded in exports. |
19-
| `generator`, `judge` | Required | Independent model settings, described below. |
19+
| `generator`, `judge` | Required | Generator settings and default judge settings, described below. |
20+
| `judges` | `{}` | Optional complete model settings for individual judgment stages. |
2021
| `contexts_file` | `null` | Explicit context JSONL; replaces automatic context grouping. |
2122
| `concurrency` | `8` | Model-call concurrency, from 1 to 128. |
2223
| `batch_size` | `30` | Request batch size, from 1 to 128. |
@@ -27,7 +28,7 @@ image paths inside source records resolve relative to their JSONL file.
2728
| `ratios` | `{train: 0.8, validation: 0.0, evaluation: 0.2}` | Query-group split fractions, each from 0 to 1 and summing to 1. |
2829
| `group_near_duplicates` | `false` | Add conservative lexical query grouping before splitting. |
2930

30-
The plugin groups requests by model role, response schema, and presence of images.
31+
The plugin groups requests by resolved model settings, response schema, and presence of images.
3132
Each batch runs a Data Designer structured-response column. Matching requests
3233
reuse cached responses keyed by their prompt, schema, model settings, and image
3334
content.
@@ -276,14 +277,91 @@ reasoning models, `max_tokens` can include both reasoning and the final response
276277

277278
To enable local token preflight for a role, set `context_window_tokens`,
278279
`tokenizer_file`, and `request_overhead_tokens`. Also set
279-
`image_tokens_per_image` when the corpus contains images. Use values appropriate
280+
`image_tokens_per_image` for the generator, relevance judge, and localization
281+
judge when the corpus contains images. Text-only judgment stages do not require
282+
an image allowance. Use values appropriate
280283
to the deployment. The check includes the structured-response instructions,
281284
source input, image allowances, and output reserve.
282285

283286
Without local token preflight, Data Designer's rendered-prompt limit still
284287
applies, but it is not a model context-window guarantee. Provider context and
285288
image limits remain applicable.
286289

290+
### Judge Model Overrides
291+
292+
Use `judge` for the default judgment model. Override individual stages under
293+
`judges` when a different deployment or sampling configuration suits the task:
294+
295+
| Stage | Input | Configuration key |
296+
| --- | --- | --- |
297+
| Summary quality | Generated summary and persona | `judges.summary` |
298+
| Query classification and answer leakage | Query only | `judges.query_metadata` |
299+
| Self-sufficiency | Query and source text | `judges.self_sufficiency` |
300+
| Relevance | Query, source text, and available images | `judges.relevance` |
301+
| Support localization | Query, source text, and available images | `judges.localization` |
302+
303+
An omitted or `null` stage uses `judge`. Each override is a complete
304+
[Model Settings](#model-settings) object. Required fields must be supplied;
305+
optional fields use their own defaults rather than inheriting values from
306+
`judge`. This includes sampling parameters, output limits, credentials, and
307+
local token budgets. Configure vision-capable deployments for image-bearing
308+
requests; the plugin does not infer model capabilities or switch models on errors.
309+
310+
This configuration uses one text judge for summary and query checks, and the
311+
default vision judge for relevance and support localization. Replace the model
312+
identifiers and endpoints with your serving deployments, and set both credential
313+
environment variables before running:
314+
315+
```yaml
316+
sources_file: sources.jsonl
317+
output_dir: runs/stage-judges
318+
dataset_id: service-manuals
319+
generator:
320+
model: your-generation-model
321+
endpoint: https://generation.example.com/v1
322+
credential_env: GENERATION_API_KEY
323+
judge:
324+
model: your-vision-judge-model
325+
endpoint: https://vision.example.com/v1
326+
credential_env: JUDGE_API_KEY
327+
temperature: 0
328+
max_tokens: 4096
329+
judges:
330+
summary: &text_judge
331+
model: your-text-judge-model
332+
endpoint: https://text.example.com/v1
333+
credential_env: JUDGE_API_KEY
334+
temperature: 0
335+
max_tokens: 1024
336+
query_metadata: *text_judge
337+
self_sufficiency: *text_judge
338+
```
339+
340+
You can also configure `judges.relevance` and `judges.localization` independently.
341+
For Python configuration, import `JudgeModels` from
342+
`data_designer_retrieval_sdg.multimodal`. `config.model_for("judge", "relevance")`
343+
returns the effective settings for that stage.
344+
345+
Automatic section planning (`context_strategy: sections` without `contexts_file`)
346+
uses the fallback `judge` for text-only table-of-contents confirmation. This judge
347+
remains active even when all five named stages are overridden or
348+
`judge_summaries: false`. Its local token budget is validated before creating the
349+
run, and its tokenizer content hash is included in the frozen run identity.
350+
Explicit contexts bypass this confirmation step.
351+
352+
`judge_summaries: false` disables the summary judgment stage, including its
353+
model-budget validation. The run manifest records the fallback under
354+
`models.judge` and the effective models for enabled stages under `models.judges`.
355+
`shared_generator_judge` is true only when every active judge, including the
356+
fallback used for TOC confirmation, uses the generator's model identifier and endpoint.
357+
358+
Cached responses include the resolved model settings and tokenizer content hash.
359+
Changing one override cannot reuse that stage's incompatible response. Changing
360+
settings in an existing run still violates its frozen identity; use a new output
361+
directory. Omitting overrides preserves the original judge request cache keys.
362+
Stage configuration does not establish better judgment quality; compare reviewed
363+
examples when choosing models.
364+
287365
## Context Planning
288366

289367
A context specifies the units available for one generation request. These fields

‎plugins/data-designer-retrieval-sdg/src/data_designer_retrieval_sdg/multimodal/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77

88
from data_designer_retrieval_sdg.multimodal.models import (
99
GenerationContext,
10+
JudgeModels,
1011
ModelSettings,
1112
MultimodalSDGConfig,
1213
QueryInstruction,
@@ -18,6 +19,7 @@
1819

1920
__all__ = [
2021
"GenerationContext",
22+
"JudgeModels",
2123
"ModelSettings",
2224
"MultimodalSDGConfig",
2325
"QueryInstruction",

‎plugins/data-designer-retrieval-sdg/src/data_designer_retrieval_sdg/multimodal/bounds.py‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -40,12 +40,11 @@ def validate_model_budget(model) -> None:
4040

4141
def validate_deployment(config, has_images: bool) -> None:
4242
"""Validate model budgets/assets before creating an immutable run directory."""
43-
for role in ("generator", "judge"):
44-
model = getattr(config, role)
43+
for role, model in {"generator": config.generator, **config.active_judges()}.items():
4544
if model.context_window_tokens is None:
4645
continue
4746
validate_model_budget(model)
48-
if has_images and model.image_tokens_per_image is None:
47+
if has_images and role in ("generator", "relevance", "localization") and model.image_tokens_per_image is None:
4948
raise ValueError(f"Configure image_tokens_per_image for {role} before running image SDG")
5049
tokenizer(str(model.tokenizer_file), digest(model.tokenizer_file))
5150

@@ -63,7 +62,7 @@ def check_request(request, config) -> None:
6362
"""
6463
if len(request.text) > environment.MAX_RENDERED_LEN:
6564
raise RequestTooLarge(f"Rendered prompt exceeds DD limit ({environment.MAX_RENDERED_LEN} characters)")
66-
model = getattr(config, request.role)
65+
model = config.model_for(request.role, request.judge_stage)
6766
if model.context_window_tokens is not None:
6867
validate_model_budget(model)
6968
if request.images and model.image_tokens_per_image is None:
@@ -80,7 +79,7 @@ def check_request(request, config) -> None:
8079
)
8180
if required > model.context_window_tokens:
8281
raise RequestTooLarge(
83-
f"{request.role} request needs {required} tokens including images, overhead and output; "
82+
f"{request.judge_stage or request.role} request needs {required} tokens including images, overhead and output; "
8483
f"configured context window is {model.context_window_tokens}"
8584
)
8685

‎plugins/data-designer-retrieval-sdg/src/data_designer_retrieval_sdg/multimodal/export.py‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -346,6 +346,7 @@ def write_multimodal_bundle(
346346
exclude={
347347
"generator",
348348
"judge",
349+
"judges",
349350
"sources_file",
350351
"source_directory",
351352
"contexts_file",
@@ -357,9 +358,12 @@ def write_multimodal_bundle(
357358
"models": {
358359
"generator": config.generator.model_dump(mode="json"),
359360
"judge": config.judge.model_dump(mode="json"),
361+
"judges": {stage: model.model_dump(mode="json") for stage, model in config.resolved_judges().items()},
360362
},
361-
"shared_generator_judge": config.generator.model == config.judge.model
362-
and config.generator.endpoint == config.judge.endpoint,
363+
"shared_generator_judge": all(
364+
config.generator.model == model.model and config.generator.endpoint == model.endpoint
365+
for model in config.active_judges().values()
366+
),
363367
"dataset_id": config.dataset_id,
364368
"split_protocol": "grouped_query_disjoint",
365369
"corpus_scope": "full_collection",

0 commit comments

Comments
 (0)