Skip to content

bug: validation_ctx parameter in SamplingStrategy.sample() exists but is not being used in the validation call #668

@akihikokuroda

Description

@akihikokuroda

The validation_ctx parameter in SamplingStrategy.sample() exists but is currently not being used in the validation call!

In mellea/stdlib/sampling/base.py line 128:
validation_ctx = validation_ctx if validation_ctx is not None else context

But at line 200-208, validation uses result_ctx instead:
val_scores_co = mfuncs.avalidate(
reqs=reqs,
context=result_ctx, # ← Uses result_ctx, NOT validation_ctx!
backend=backend,
output=result,
...
)

Solution Path
To enable HallucinationRequirement to work seamlessly with instruct():

Option 1: Fix the Sampling Strategy (Recommended)
Change line 202 in mellea/stdlib/sampling/base.py from:
context=result_ctx,
to:
context=validation_ctx,

This would allow passing a custom validation context with documents attached.

Option 2: Enhance instruct() to Support RAG Validation
Modify instruct() to:

Accept a validation_documents parameter
Automatically attach documents to the generated message
Pass the modified context as validation_ctx to the sampling strat

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions