Assignment 1

[Joeghaly86]

TITLE:
UofT-DSI | AI Evaluation - Assignment 1

What changes are you trying to make? (e.g. Adding or removing code, refactoring existing code, adding reports)

Implemented automated evaluation for LLM-generated summaries.
Integrated OpenAI model output with structured JSON schema.
Extracted the summary field from the model response object.
Added DeepEval summarization metric with bespoke assessment questions.
Added GEval metrics for coherence/clarity, tonality, and safety.
Structured evaluation output to return scores and explanations for each metric.

What did you learn from the changes you have made?

How DeepEval uses LLM-as-a-judge to assess summarization quality.
How evaluation dimensions like coherence, tone, and safety complement summarization accuracy.

Was there another approach you were thinking about making? If so, what approach(es) were you thinking of?

Returning unstructured evaluation text instead of structured key-value outputs.
Performing manual human evaluation instead of automated scoring.

Were there any challenges? If so, what issue(s) did you face? How did you overcome it?

Difficulty extracting the summary from the structured OpenAI response object.
Resolved by inspecting response.output[0].content[0].model_dump() to identify correct fields.

JSON vs dictionary confusion when using model_dump_json() vs model_dump().
Fixed by switching to dictionary parsing for indexing.

Aligning DeepEval input format with the OpenAI response structure.
Addressed by separating original text and generated summary before evaluation.

How were these changes tested?

Generated a summary using the OpenAI model.
Extracted the summary from the structured response object.
Ran DeepEval metrics locally using .measure() for each evaluation metric.
Verified outputs returned valid scores and explanations.
Confirmed the structured result format matched assignment requirements.

A reference to a related issue in your repository (if applicable)

N/A

I can confirm that my changes are working as intended

[Dmytro-Bonislavskyi]

Well done!