Skip to content

bugfix: handle negative deductions returned by evaluator#280

Open
JellyPork wants to merge 1 commit into
interviewstreet:mainfrom
JellyPork:fix/negative-deductions-validation
Open

bugfix: handle negative deductions returned by evaluator#280
JellyPork wants to merge 1 commit into
interviewstreet:mainfrom
JellyPork:fix/negative-deductions-validation

Conversation

@JellyPork

Copy link
Copy Markdown

Closes #24

Summary

Fix a crash in resume evaluation when the model returns a negative value for deductions.total.

The evaluator expects deductions to be stored as a positive magnitude and applied later as a subtraction. In some runs, the model returns values like -10, which causes EvaluationData validation to fail and stops the scoring flow.

Changes

  • normalize deductions.total to a positive number during validation
  • clarify the prompt contract so deductions are returned as magnitude-only values
  • add a regression test covering the negative-deductions case

Repro

An evaluator response like this can cause the run to fail:

"deductions": {
  "total": -10,
  "reasons": "Broken or inactive links"
}

Prompt change examples

Before:

"deductions": {
  "total": -10,
  "reasons": "Broken or inactive links"
}

After:

"deductions": {
  "total": 10,
  "reasons": "Broken or inactive links"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Negative deduction total can crash resume evaluation with Pydantic validation error

1 participant