The validation_ctx parameter in SamplingStrategy.sample() exists but is currently not being used in the validation call!
In mellea/stdlib/sampling/base.py line 128:
validation_ctx = validation_ctx if validation_ctx is not None else context
But at line 200-208, validation uses result_ctx instead:
val_scores_co = mfuncs.avalidate(
reqs=reqs,
context=result_ctx, # ← Uses result_ctx, NOT validation_ctx!
backend=backend,
output=result,
...
)
Solution Path
To enable HallucinationRequirement to work seamlessly with instruct():
Option 1: Fix the Sampling Strategy (Recommended)
Change line 202 in mellea/stdlib/sampling/base.py from:
context=result_ctx,
to:
context=validation_ctx,
This would allow passing a custom validation context with documents attached.
Option 2: Enhance instruct() to Support RAG Validation
Modify instruct() to:
Accept a validation_documents parameter
Automatically attach documents to the generated message
Pass the modified context as validation_ctx to the sampling strat