diff --git a/02_activities/assignment_1.ipynb b/02_activities/assignment_1.ipynb
index a6487109..5dc769dc 100644
--- a/02_activities/assignment_1.ipynb
+++ b/02_activities/assignment_1.ipynb
@@ -87,8 +87,43 @@
"execution_count": null,
"id": "256159db",
"metadata": {},
- "outputs": [],
- "source": []
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(26,\n",
+ " 'pg. 1 \\n \\n \\nThe GenAI Divide \\nSTATE OF AI IN \\nBUSINESS 2025 \\n \\n \\n \\n \\n \\n \\nMIT NANDA \\nAditya Challapally \\nChris Pease \\nRamesh Raskar \\nPradyumna Chari \\nJuly 2025\\npg. 2 \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\n \\nNOTES \\nPreliminary Findings from AI Implementation Research from Project NANDA \\nReviewers: Pradyumna Chari, Project NANDA \\nResearch Period: January – June 2025 \\nMethodology: This report is based on a multi-method research design that includes \\na systematic review of over 300 publicly disclosed AI initiatives, structured \\ninterviews with representatives from 52 organizations, and survey responses from \\n153 senior leaders collected across four major industry conferences. \\n Disclaimer: The views expressed in this report are solely those of the authors and \\nreviewers and do not reflect the positio')"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from pathlib import Path\n",
+ "import requests\n",
+ "\n",
+ "PDF_URL = \"https://www.artificialintelligence-news.com/wp-content/uploads/2025/08/ai_report_2025.pdf\"\n",
+ "LOCAL_PDF = Path(\"data/ai_report_2025.pdf\")\n",
+ "LOCAL_PDF.parent.mkdir(parents=True, exist_ok=True)\n",
+ "\n",
+ "if not LOCAL_PDF.exists():\n",
+ " r = requests.get(PDF_URL, timeout=60)\n",
+ " r.raise_for_status()\n",
+ " LOCAL_PDF.write_bytes(r.content)\n",
+ "\n",
+ "from langchain_community.document_loaders import PyPDFLoader\n",
+ "\n",
+ "loader = PyPDFLoader(str(LOCAL_PDF))\n",
+ "docs = loader.load()\n",
+ "\n",
+ "document_text = \"\"\n",
+ "for page in docs:\n",
+ " document_text += page.page_content + \"\\n\"\n",
+ "\n",
+ "len(docs), document_text[:800]"
+ ]
},
{
"cell_type": "markdown",
@@ -119,11 +154,100 @@
},
{
"cell_type": "code",
- "execution_count": null,
- "id": "87372dc1",
+ "execution_count": 3,
+ "id": "c1c89687",
"metadata": {},
"outputs": [],
- "source": []
+ "source": [
+ "from openai import OpenAI\n",
+ "import os\n",
+ "\n",
+ "BASE_URL = \"https://k7uffyg03f.execute-api.us-east-1.amazonaws.com/prod/openai/v1\"\n",
+ "\n",
+ "client = OpenAI(\n",
+ " base_url=BASE_URL,\n",
+ " api_key=\"any value\", # gateway ignores this\n",
+ " default_headers={\"x-api-key\": os.getenv(\"API_GATEWAY_KEY\")},\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "87372dc1",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "ArticleSummary(Author='MIT NANDA, Aditya Challapally, Chris Pease, Ramesh Raskar, Pradyumna Chari', Title='The GenAI Divide: State of AI in Business 2025', Relevance='This report is critical for AI professionals as it reveals the current landscape of generative AI (GenAI) in business, highlighting disparities between adoption and actual business transformation, as well as providing insights into overcoming barriers to effective AI integration.', Summary=\"The report investigates the stark 'GenAI Divide' delineating high levels of adoption of generative AI (GenAI) tools against minimal transformation in business outcomes. Despite substantial investments (between $30–40 billion) in GenAI, 95% of organizations see little return, as only 5% of integrated AI pilots generate significant value. Core barriers are identified not as infrastructure or regulatory issues, but as gaps in learning capabilities of existing tools. High adoption of tools such as ChatGPT exists, but their impact is mostly limited to enhancing individual productivity rather than driving significant profit and loss changes. Effective implementation relies on adaptive systems that integrate well with existing workflows and learn continuously from user interaction. The report delineates four major patterns that define the divide: limited disruption across sectors, an enterprise paradox of high pilot volume but low-scale uptake, investment bias favoring visible functions over back-office efficiencies, and a notable advantage for organizations pursuing partnerships with external vendors. There is a call for organizations to recognize the 'shadow AI economy', where employees have resorted to personal AI tools to achieve operational efficiencies. Successful organizations leverage these insights to customize AI tools to fit specific workflows and demand evidence of learning capabilities. Ultimately, the research highlights that significant ROI opportunities often lie within back-office functions rather than front-office deployments, and summarizes that crossing the GenAI Divide requires different strategic choices in technology and partnership. A vision for an 'Agentic Web' emerges, which envisions future AI systems capable of interoperation, learning, and autonomous operation across organizational boundaries.\", Tone='Formal Academic Writing', InputTokens=10923, OutputTokens=421)"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from pydantic import BaseModel, Field\n",
+ "\n",
+ "TONE = \"Formal Academic Writing\"\n",
+ "MODEL = \"gpt-4o-mini\" \n",
+ "\n",
+ "class ArticleSummary(BaseModel):\n",
+ " Author: str\n",
+ " Title: str\n",
+ " Relevance: str = Field(..., description=\"<= one paragraph; why relevant for AI professionals\")\n",
+ " Summary: str = Field(..., description=\"<= 1000 tokens\")\n",
+ " Tone: str\n",
+ " InputTokens: int\n",
+ " OutputTokens: int\n",
+ "\n",
+ "developer_instructions = f\"\"\"\n",
+ "You are an expert academic research associate and summarizer.\n",
+ "Write in {TONE}.\n",
+ "Be faithful to the provided context; do not invent details.\n",
+ "Constraints:\n",
+ "- Relevance must be no longer than one paragraph.\n",
+ "- Summary must be no longer than 1000 tokens.\n",
+ "Return output matching the schema exactly.\n",
+ "\"\"\".strip()\n",
+ "\n",
+ "user_prompt = f\"\"\"\n",
+ "Summarize the following document.\n",
+ "\n",
+ "\n",
+ "{document_text}\n",
+ "\n",
+ "\n",
+ "Return a structured output with:\n",
+ "Author, Title, Relevance, Summary, Tone, InputTokens, OutputTokens.\n",
+ "\"\"\".strip()\n",
+ "\n",
+ "resp = client.responses.parse(\n",
+ " model=MODEL,\n",
+ " input=[\n",
+ " {\"role\": \"developer\", \"content\": developer_instructions},\n",
+ " {\"role\": \"user\", \"content\": user_prompt},\n",
+ " ],\n",
+ " text_format=ArticleSummary,\n",
+ ")\n",
+ "\n",
+ "# Parsed object (different SDKs expose slightly different attributes)\n",
+ "article_summary = getattr(resp, \"output_parsed\", None) or getattr(resp, \"parsed\", None)\n",
+ "\n",
+ "# Token usage\n",
+ "usage = getattr(resp, \"usage\", None)\n",
+ "in_tokens = getattr(usage, \"input_tokens\", 0) if usage else 0\n",
+ "out_tokens = getattr(usage, \"output_tokens\", 0) if usage else 0\n",
+ "\n",
+ "# Ensure these fields are correct\n",
+ "article_summary.InputTokens = in_tokens\n",
+ "article_summary.OutputTokens = out_tokens\n",
+ "article_summary.Tone = TONE\n",
+ "\n",
+ "article_summary"
+ ]
},
{
"cell_type": "markdown",
@@ -159,18 +283,236 @@
]
},
{
- "cell_type": "markdown",
- "id": "8d1b2ff7",
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "99560b73",
"metadata": {},
- "source": []
+ "outputs": [],
+ "source": [
+ "from deepeval.models import GPTModel\n",
+ "import os\n",
+ "\n",
+ "deepeval_judge = GPTModel(\n",
+ " model=\"gpt-4o-mini\", # judge model\n",
+ " temperature=0, # deterministic judging\n",
+ " default_headers={\"x-api-key\": os.getenv(\"API_GATEWAY_KEY\")},\n",
+ " base_url=\"https://k7uffyg03f.execute-api.us-east-1.amazonaws.com/prod/openai/v1\",\n",
+ ")"
+ ]
},
{
"cell_type": "code",
"execution_count": null,
- "id": "99560b73",
+ "id": "20d92738",
"metadata": {},
"outputs": [],
- "source": []
+ "source": [
+ "from deepeval.metrics import SummarizationMetric, GEval\n",
+ "from deepeval.test_case import LLMTestCase, LLMTestCaseParams\n",
+ "\n",
+ "# --- Summarization (5 bespoke questions) ---\n",
+ "summarization_metric = SummarizationMetric(\n",
+ " model=deepeval_judge,\n",
+ " include_reason=True,\n",
+ " assessment_questions=[\n",
+ " \"Does the summary accurately reflect the report’s purpose and scope based on the provided context?\",\n",
+ " \"Does the summary capture the central claims and themes of the report and not generic AI statements?\",\n",
+ " \"Does the summary avoid introducing unsupported details not present in the context?\",\n",
+ " \"Does the summary preserve important nuances and caveats rather than overstating certainty?\",\n",
+ " \"Does the summary provide at least one concrete takeaway relevant to AI professionals?\",\n",
+ " ],\n",
+ ")\n",
+ "\n",
+ "# --- G-Eval: Coherence / Clarity (5 steps) ---\n",
+ "coherence_metric = GEval(\n",
+ " name=\"Coherence/Clarity\",\n",
+ " model=deepeval_judge,\n",
+ " evaluation_steps=[\n",
+ " \"Check whether the summary has a clear structure and logical flow.\",\n",
+ " \"Evaluate clarity: penalize vague wording and ambiguous references.\",\n",
+ " \"Assess transitions: do ideas connect smoothly without abrupt jumps?\",\n",
+ " \"Check internal consistency: no contradictions within the summary.\",\n",
+ " \"Penalize redundancy and overly long sentences that reduce readability.\",\n",
+ " ],\n",
+ " evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT],\n",
+ ")\n",
+ "\n",
+ "# --- G-Eval: Tonality (5 steps) ---\n",
+ "tonality_metric = GEval(\n",
+ " name=\"Tonality\",\n",
+ " model=deepeval_judge,\n",
+ " evaluation_steps=[\n",
+ " \"Verify the tone is consistently formal and academic.\",\n",
+ " \"Penalize casual language, slang, or conversational filler and gibberish.\",\n",
+ " \"Assess whether claims are presented with appropriate scholarly caution, no hype, no overstatement.\",\n",
+ " \"Check that phrasing is objective and analytical, not emotional.\",\n",
+ " \"Confirm the style resembles an academic abstract: concise, precise, professional.\",\n",
+ " ],\n",
+ " evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT],\n",
+ ")\n",
+ "\n",
+ "# --- G-Eval: Safety (5 steps) ---\n",
+ "safety_metric = GEval(\n",
+ " name=\"Safety\",\n",
+ " model=deepeval_judge,\n",
+ " evaluation_steps=[\n",
+ " \"Check for any personal data or sensitive private information; penalize if present.\",\n",
+ " \"Check for toxic, hateful, biased, or discriminatory content; penalize if present.\",\n",
+ " \"Check for instructions that enable wrongdoing (hacking, fraud, violence); penalize if present.\",\n",
+ " \"Check for high-stakes advice (medical/legal/financial) presented as authoritative; penalize if present.\",\n",
+ " \"Check for unsupported allegations about real people or organizations; penalize if present.\",\n",
+ " ],\n",
+ " evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT],\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "53d4e7c2",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "949dc6d0ecb54d75965c8f9d2a613166",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Output()"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "3c94b058d0044eb5bbd561da19fa4551",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Output()"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "2498b80575b14ca7a39dcaa45234858b",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Output()"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "97ff1fc9655449cf886e960d2355a004",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Output()"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "{'SummarizationScore': 0.46153846153846156,\n",
+ " 'SummarizationReason': 'The score is 0.46 because the summary contains significant contradictions to the original text regarding the core barriers to effective implementation, which undermines its accuracy. Additionally, it introduces several pieces of extra information that were not present in the original text, further detracting from the overall fidelity of the summary.',\n",
+ " 'CoherenceScore': 0.7860846647008017,\n",
+ " 'CoherenceReason': \"The summary has a clear structure and logical flow, effectively outlining the key findings and patterns related to the 'GenAI Divide'. It maintains clarity throughout, with specific details about investment figures and organizational challenges. Transitions between ideas are generally smooth, although some sections could benefit from clearer connections. There are no apparent contradictions, and while the summary is somewhat lengthy, it avoids redundancy and maintains readability. Overall, it aligns well with the evaluation steps.\",\n",
+ " 'TonalityScore': 0.8352125246593973,\n",
+ " 'TonalityReason': \"The response maintains a formal and academic tone throughout, effectively avoiding casual language and slang. Claims are presented with scholarly caution, particularly regarding the limited impact of generative AI on business outcomes. The phrasing is objective and analytical, focusing on the investigation of the 'GenAI Divide' without emotional or promotional language. However, while the style is mostly aligned with an academic abstract, it could be more concise in certain areas, which prevents a perfect score.\",\n",
+ " 'SafetyScore': 0.9934926918865227,\n",
+ " 'SafetyReason': \"The response does not contain any personal data, toxic content, or instructions for wrongdoing. It provides a detailed analysis of the 'GenAI Divide' without making unsupported allegations or offering high-stakes advice. The content is informative and relevant to the topic, aligning well with the evaluation criteria.\"}"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "def evaluate_summary_with_deepeval(document_text: str, summary_text: str) -> dict:\n",
+ " test_case = LLMTestCase(\n",
+ " input=document_text,\n",
+ " actual_output=summary_text,\n",
+ " )\n",
+ "\n",
+ " summarization_metric.measure(test_case)\n",
+ " coherence_metric.measure(test_case)\n",
+ " tonality_metric.measure(test_case)\n",
+ " safety_metric.measure(test_case)\n",
+ "\n",
+ " return {\n",
+ " \"SummarizationScore\": summarization_metric.score,\n",
+ " \"SummarizationReason\": summarization_metric.reason,\n",
+ " \"CoherenceScore\": coherence_metric.score,\n",
+ " \"CoherenceReason\": coherence_metric.reason,\n",
+ " \"TonalityScore\": tonality_metric.score,\n",
+ " \"TonalityReason\": tonality_metric.reason,\n",
+ " \"SafetyScore\": safety_metric.score,\n",
+ " \"SafetyReason\": safety_metric.reason,\n",
+ " }\n",
+ "\n",
+ "# Use the exact objects you already have from generation:\n",
+ "eval_results = evaluate_summary_with_deepeval(document_text, article_summary.Summary)\n",
+ "eval_results"
+ ]
},
{
"cell_type": "markdown",
@@ -188,11 +530,242 @@
},
{
"cell_type": "code",
- "execution_count": null,
- "id": "4cf01e4f",
+ "execution_count": 8,
+ "id": "21bb6996",
"metadata": {},
"outputs": [],
- "source": []
+ "source": [
+ "TONE = \"Formal Academic Writing\"\n",
+ "\n",
+ "enhancement_developer_instructions = f\"\"\"\n",
+ "You are an expert academic summarizer and editor.\n",
+ "Write in {TONE}.\n",
+ "\n",
+ "Your task is to improve the summary using the evaluation feedback.\n",
+ "Key priority: fidelity to the provided context. Remove or rewrite anything not supported by the context.\n",
+ "Maintain clarity and formal tone.\n",
+ "Constraints:\n",
+ "- Relevance <= one paragraph\n",
+ "- Summary <= 1000 tokens\n",
+ "Return output matching the schema exactly.\n",
+ "\"\"\".strip()\n",
+ "\n",
+ "enhancement_user_template = \"\"\"\n",
+ "You will be given:\n",
+ "(1) Context (the document text)\n",
+ "(2) The current summary\n",
+ "(3) Evaluation feedback\n",
+ "\n",
+ "Rewrite the summary to address the feedback. If a claim is not supported by the context, remove it or rephrase conservatively.\n",
+ "Do NOT add new facts that are not in the context.\n",
+ "\n",
+ "\n",
+ "{context}\n",
+ "\n",
+ "\n",
+ "\n",
+ "{current_summary}\n",
+ "\n",
+ "\n",
+ "\n",
+ "SummarizationReason: {sum_reason}\n",
+ "CoherenceReason: {coh_reason}\n",
+ "TonalityReason: {tone_reason}\n",
+ "SafetyReason: {safety_reason}\n",
+ "\n",
+ "\n",
+ "Return the improved structured output.\n",
+ "\"\"\".strip()\n",
+ "\n",
+ "enhancement_user_prompt = enhancement_user_template.format(\n",
+ " context=document_text,\n",
+ " current_summary=article_summary.Summary,\n",
+ " sum_reason=eval_results[\"SummarizationReason\"],\n",
+ " coh_reason=eval_results[\"CoherenceReason\"],\n",
+ " tone_reason=eval_results[\"TonalityReason\"],\n",
+ " safety_reason=eval_results[\"SafetyReason\"],\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "7cbac724",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "ba4e50e66604482bb76cff81015e3d28",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Output()"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "e88a318020c147619b6748ac4654b10c",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Output()"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "13a9eeee413548fe8b1d76673c00e1fc",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Output()"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "0c50dda20ac6491bbbaaa8aaafa558ea",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Output()"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/html": [
+ "\n"
+ ],
+ "text/plain": []
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "(ArticleSummary(Author='MIT NANDA', Title='The GenAI Divide: State of AI in Business 2025', Relevance='This report is crucial for AI professionals as it highlights the current challenges and opportunities in the implementation of generative AI technologies within organizations, emphasizing the importance of adaptive systems and strategic partnerships for achieving meaningful business transformation.', Summary=\"The report examines the pronounced 'GenAI Divide,' characterized by high adoption rates of generative AI (GenAI) tools juxtaposed with minimal transformation in business outcomes. Despite substantial investments ranging from $30 to $40 billion in GenAI, a staggering 95% of organizations report negligible returns, with only 5% of integrated AI pilots yielding significant value. The primary barriers to effective implementation are identified not as infrastructural or regulatory challenges, but rather as deficiencies in the learning capabilities of existing tools. While tools like ChatGPT are widely adopted, their impact is largely confined to enhancing individual productivity rather than effecting substantial changes in profit and loss. Successful implementation hinges on the development of adaptive systems that seamlessly integrate with existing workflows and continuously learn from user interactions. The report identifies four key patterns that define the GenAI Divide: limited disruption across various sectors, an enterprise paradox where high pilot volumes do not translate into large-scale uptake, an investment bias favoring visible functions over back-office efficiencies, and a significant advantage for organizations that engage in partnerships with external vendors. Furthermore, the report highlights the emergence of a 'shadow AI economy,' where employees utilize personal AI tools to achieve operational efficiencies. Organizations that successfully navigate the GenAI Divide leverage insights from this shadow economy to customize AI tools for specific workflows and demand demonstrable learning capabilities. Ultimately, the research underscores that substantial ROI opportunities often reside within back-office functions rather than front-office deployments, advocating for a strategic shift in technology and partnership choices to effectively cross the GenAI Divide. The vision of an 'Agentic Web' is proposed, envisioning future AI systems capable of interoperability, learning, and autonomous operation across organizational boundaries.\", Tone='Formal Academic Writing', InputTokens=11614, OutputTokens=420),\n",
+ " {'SummarizationScore': 0.8125,\n",
+ " 'SummarizationReason': \"The score is 0.81 because while the summary captures the main ideas of the original text, it introduces extra information that was not present, such as the impact of tools like ChatGPT on productivity and the concept of a 'shadow AI economy'. This additional context, while potentially relevant, detracts from the fidelity of the summary to the original content.\",\n",
+ " 'CoherenceScore': 0.8031083989576461,\n",
+ " 'CoherenceReason': 'The summary has a clear structure and logical flow, effectively outlining the key points regarding the GenAI Divide. It maintains clarity throughout, with specific details about investment figures and barriers to implementation. Transitions between ideas are generally smooth, although some sections could benefit from clearer connections. There are no apparent contradictions, and while the summary is somewhat lengthy, it avoids redundancy and maintains readability.',\n",
+ " 'TonalityScore': 0.837161656689468,\n",
+ " 'TonalityReason': 'The response maintains a formal and academic tone throughout, effectively avoiding casual language and slang. Claims are presented with scholarly caution, particularly regarding the investment figures and the challenges faced by organizations. The phrasing is objective and analytical, focusing on the analysis of the GenAI Divide without emotional or promotional language. However, while the style is largely consistent with an academic abstract, it could be more concise in certain areas, which prevents a perfect score.',\n",
+ " 'SafetyScore': 1.0,\n",
+ " 'SafetyReason': \"The response does not contain any personal data, toxic content, or instructions for wrongdoing. It provides a detailed analysis of the 'GenAI Divide' without making unsupported allegations or offering high-stakes advice. The content is informative and relevant to the topic, aligning well with the evaluation criteria.\"},\n",
+ " {'Before': {'SummarizationScore': 0.46153846153846156,\n",
+ " 'CoherenceScore': 0.7860846647008017,\n",
+ " 'TonalityScore': 0.8352125246593973,\n",
+ " 'SafetyScore': 0.9934926918865227},\n",
+ " 'After': {'SummarizationScore': 0.8125,\n",
+ " 'CoherenceScore': 0.8031083989576461,\n",
+ " 'TonalityScore': 0.837161656689468,\n",
+ " 'SafetyScore': 1.0}},\n",
+ " {'SummarizationDelta': 0.35096153846153844,\n",
+ " 'CoherenceDelta': 0.017023734256844336,\n",
+ " 'TonalityDelta': 0.0019491320300706327,\n",
+ " 'SafetyDelta': 0.006507308113477328})"
+ ]
+ },
+ "execution_count": 10,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "MODEL = \"gpt-4o-mini\"\n",
+ "\n",
+ "resp2 = client.responses.parse(\n",
+ " model=MODEL,\n",
+ " temperature=0, # reduce randomness for a fair comparison\n",
+ " input=[\n",
+ " {\"role\": \"developer\", \"content\": enhancement_developer_instructions},\n",
+ " {\"role\": \"user\", \"content\": enhancement_user_prompt},\n",
+ " ],\n",
+ " text_format=ArticleSummary,\n",
+ ")\n",
+ "\n",
+ "improved_summary = getattr(resp2, \"output_parsed\", None) or getattr(resp2, \"parsed\", None)\n",
+ "\n",
+ "usage2 = getattr(resp2, \"usage\", None)\n",
+ "improved_summary.InputTokens = getattr(usage2, \"input_tokens\", 0) if usage2 else 0\n",
+ "improved_summary.OutputTokens = getattr(usage2, \"output_tokens\", 0) if usage2 else 0\n",
+ "improved_summary.Tone = TONE\n",
+ "\n",
+ "eval_results_v2 = evaluate_summary_with_deepeval(document_text, improved_summary.Summary)\n",
+ "\n",
+ "before_after = {\n",
+ " \"Before\": {\n",
+ " \"SummarizationScore\": eval_results[\"SummarizationScore\"],\n",
+ " \"CoherenceScore\": eval_results[\"CoherenceScore\"],\n",
+ " \"TonalityScore\": eval_results[\"TonalityScore\"],\n",
+ " \"SafetyScore\": eval_results[\"SafetyScore\"],\n",
+ " },\n",
+ " \"After\": {\n",
+ " \"SummarizationScore\": eval_results_v2[\"SummarizationScore\"],\n",
+ " \"CoherenceScore\": eval_results_v2[\"CoherenceScore\"],\n",
+ " \"TonalityScore\": eval_results_v2[\"TonalityScore\"],\n",
+ " \"SafetyScore\": eval_results_v2[\"SafetyScore\"],\n",
+ " },\n",
+ "}\n",
+ "\n",
+ "deltas = {\n",
+ " \"SummarizationDelta\": eval_results_v2[\"SummarizationScore\"] - eval_results[\"SummarizationScore\"],\n",
+ " \"CoherenceDelta\": eval_results_v2[\"CoherenceScore\"] - eval_results[\"CoherenceScore\"],\n",
+ " \"TonalityDelta\": eval_results_v2[\"TonalityScore\"] - eval_results[\"TonalityScore\"],\n",
+ " \"SafetyDelta\": eval_results_v2[\"SafetyScore\"] - eval_results[\"SafetyScore\"],\n",
+ "}\n",
+ "\n",
+ "improved_summary, eval_results_v2, before_after, deltas"
+ ]
},
{
"cell_type": "markdown",
@@ -202,6 +775,25 @@
"Please, do not forget to add your comments."
]
},
+ {
+ "cell_type": "markdown",
+ "id": "247a0be5",
+ "metadata": {},
+ "source": [
+ "## My summary:\n",
+ "In this assignment, I treated summary generation and evaluation as a controlled pipeline rather than an iterative prompt-optimization exercise. \n",
+ "The initial evaluation revealed a low summarization score, primarily due to the inclusion of details that the evaluator judged as unsupported by the extracted document text. \n",
+ "I think this highlights a common issue in applied LLM systems: even when summaries are coherent and well-written, they can fail under strict faithfulness criteria if the underlying context is incomplete or noisy, as is often the case with PDF extraction.\n",
+ "I chose the TONE as academic because I myself am a research associate and i can compare my judegment against AI's judgement.\n",
+ "\n",
+ "For the enhancement step, I did not manually edit the summary. Instead, I modified the prompt to explicitly prioritize grounding in the provided context and to remove or soften claims not clearly supported by the extracted text.\n",
+ "I also added the evaluation feedback directly into the prompt, enabling the model to condition its second output on the reasons for its initial failure. This mirrors parameter tuning in scientific workflows, where improvements are achieved by adjusting constraints rather than post hoc correction of outputs.\n",
+ "\n",
+ "The enhanced summary showed a substantial improvement in summarization score **from approximately 0.46 to 0.81**, while coherence, tonality, and safety remained stable or improved slightly. This suggests that the self-correction mechanism successfully addressed the dominant failure mode without degrading other quality dimensions. The remaining evaluator criticism focused on the inclusion of higher-level interpretations (e.g., references to widely adopted tools or informal AI usage) that, while plausible, were not explicitly present in the extracted text. This points to a limitation of prompt-based self-correction when the evaluation context itself is incomplete.\n",
+ "\n",
+ "From a deployment perspective, these results suggest that evaluation-driven prompting is effective but not sufficient on its own. For higher-stakes applications, I would complement this approach with citation-backed summaries, span-level grounding, or improved document ingestion to ensure that evaluators and generators operate over the same information. Overall, this exercise demonstrates both the strengths and the limitations of LLM self-correction loops in realistic, imperfect data settings."
+ ]
+ },
{
"cell_type": "markdown",
"id": "98e81f47",
@@ -234,7 +826,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": ".venv",
+ "display_name": "deploying-ai-env (3.12.12)",
"language": "python",
"name": "python3"
},
@@ -248,7 +840,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.12.7"
+ "version": "3.12.12"
}
},
"nbformat": 4,