Agent Evaluation Notebooks

Companion notebooks for Evaluating AI Agents: A Comprehensive Guide — covering AI agent evaluation using Vertex AI, Google ADK, and open-source frameworks.

Start Here

00_eval_frameworks_overview.ipynb   ← start here

Notebooks

#	Notebook	What it covers
00	00_eval_frameworks_overview.ipynb	The core measurement problem — why cosine similarity fails for semantic evaluation, the two-layer Loss Meter (vibe check + fact check), framework decision tree
01	01_vertex_ai_trajectory_eval.ipynb	Vertex AI native agent eval — trajectory metrics (exact match, precision, recall, ordering), synthetic eval case generation with Gemini, ADK agent runnable wrapper
02	02_deepeval_agent_metrics.ipynb	DeepEval agent-specific metrics — ToolCorrectnessMetric, TaskCompletionMetric, GEval, AAEF metrics (TUE, MCR, CSS, SPI), Gemini as judge via LiteLLM
03	03_ragas_rag_evaluation.ipynb	RAGAS for retrieval-grounded agents — faithfulness (atomic fact check), context precision/recall, answer relevancy, Gemini as LLM evaluator and embedder
04	04_phoenix_observability.ipynb	Phoenix/Arize observability tracing — ADK auto-instrumentation with zero code changes, OpenTelemetry setup, TruLens RAG Triad
05	05_production_eval_pipeline.ipynb	Full 4-phase production pipeline — development unit tests, CI/CD release gates, production sampling + drift detection, improvement cycle with failure export

Setup

pip install "google-cloud-aiplatform[evaluation]>=1.111.0" google-adk deepeval

For RAGAS notebooks:

pip install ragas google-genai datasets

For Phoenix/observability notebooks:

pip install arize-phoenix opentelemetry-sdk openinference-instrumentation-google-adk

Set your project:

PROJECT_ID = "your-project-id"
LOCATION = "us-central1"

All notebooks use gemini-2.5-flash as the default model for inference and as the LLM judge.

Key Concepts

Two-layer Loss Meter — the core measurement framework used throughout:

• Method A (Vibe Check): Cosine similarity of text embeddings — catches large semantic drift cheaply but is fooled by negation and antonyms
• Method B (Fact Check): Propositional recall — decompose source into atomic facts, verify each one independently — catches precise factual errors that embeddings miss

Two eval dimensions for agents:

• Final Response Quality — correctness, completeness, groundedness, safety
• Trajectory Quality — tool selection, ordering, efficiency, parameter correctness

Three measurement methods:

• Automated metrics — fast, deterministic, CI/CD friendly
• LLM-as-Judge — scales to nuanced evaluation, catches negation/semantic opposition
• Human evaluation — ground truth calibration, high-stakes validation