feat(agent-eval): add LLMAgent* evaluators for agent trace monitoring#432
Merged
Conversation
Add 8 LLM evaluators + 3 rule evaluators for AI agent trace quality assessment, aligned with DeepEval 3-layer taxonomy (Execution/Action/Reasoning) + Recovery. LLM evaluators (dingo/model/llm/agent_eval/): - LLMAgentTaskCompletion, LLMAgentStepEfficiency (Execution) - LLMAgentToolCorrectness, LLMAgentArgumentCorrectness (Action) - LLMAgentPlanQuality, LLMAgentPlanAdherence (Reasoning) - LLMAgentErrorRecovery (Recovery), LLMAgentTraceConclusion (synthesis) Rule evaluators: RuleAgentTraceLoopDetection, RuleAgentTraceTokenBudget, RuleAgentTraceLatencyAnomaly Shared BaseLLMAgentEval with 0-10→0.0-1.0 normalization, CJK language detection, and configurable threshold.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive suite of LLM-as-Judge and rule-based evaluators for assessing AI agent execution traces, including a base class for standardized scoring and a synthesizer for overall trace diagnosis. The review feedback highlights several critical opportunities to improve code quality and robustness. Specifically, it recommends avoiding boilerplate duplication in LLMAgentErrorRecovery and LLMAgentPlanQuality by leveraging inheritance and calling super(). Additionally, the feedback points out potential TypeError and AttributeError vulnerabilities in JSON parsing and attribute access across multiple files, and advises removing an overriding dynamic_config = None statement that could break configuration inheritance.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Audit fixes to the agent evaluators:
- base: _parse_json_response now tolerates prose around the JSON and
uppercase/missed code fences by extracting the first balanced object,
so a recoverable response is no longer penalized as a hard parse failure
(which previously surfaced as a misleading score of 0).
- TraceConclusion: derive severity from the overall score (good >= 0.6,
warning >= 0.3, else critical) instead of a separate LLM field, so the
pass/fail status, severity, and numeric score can never diverge.
- ErrorRecovery: drop the redundant `recovery_quality` field (the base reads
`score`); make the no-error short-circuit parse the {error_events, steps}
payload instead of exact-string matching, so a clean run skips the LLM call.
- TaskCompletion / PlanQuality: anchor the overall score to the rated
sub-dimensions so they can no longer contradict each other; PlanQuality
also drops an unused keyword list.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LLMAgentErrorRecovery and LLMAgentPlanQuality each copied the full retry/parse/error-fallback loop from BaseOpenAI.eval. Replace the duplication with the template-method hooks: - ErrorRecovery: keep the no-error short-circuit, then `return super().eval()` instead of re-implementing the loop. - PlanQuality: override `process_response` to handle the score=-1 "no planning" sentinel and delegate everything else to `super().process_response()`; drop the custom `eval`. Behavior is identical (verified: short-circuit, sentinel, and normal 8/10 -> 0.8 scoring paths). Also removes now-unused imports. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fix three crashes in the deterministic agent-trace rules and wire them for trace-level orchestration: - _extract_tool_names / _extract_steps: a present-but-null "tool_calls" /"steps" key (JSON null) made dict.get return None and broke iteration (TypeError); use `a or b or []` plus a list guard. - _extract_tool_names: parenthesize the filter so a non-dict item no longer reaches `item.get(...)` and raises AttributeError (operator precedence: `and` binds tighter than `or`). - RuleAgentTraceTokenBudget.dynamic_config: None -> EvaluatorRuleArgs( threshold=500_000) so set_config_rule's model_copy() in the local/spark executors does not raise, matching every other rule. - Add input_data_type="agent_trace_json" / eval_layer to all three rules so agent orchestrators feed them the trace-level tool-call sequence as JSON instead of running them per-span on plain text. Adds test_rule_agent.py (previously zero coverage) covering the malformed-input edge cases and the registration/attribute contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_extract_tool_names and _extract_steps had an identical parse-and-normalize block (JSON string/list/dict → list of items, with the null-key and non-list guards) differing only in key priority. Hoist it into a module-level _load_trace_items() so each extractor keeps only its distinct mapping. No behavior change — covered by the existing test_rule_agent.py edge-case suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.