Sample.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Azure AI Document Intelligence and Azure OpenAI GPT-3.5 Turbo to extract structured data from documents\n",
    "\n",
    "This notebook demonstrates [how to use the new Markdown content extraction feature of Azure AI Document Intelligence's pre-built Layout model](https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/document-intelligence-preview-adds-more-prebuilts-support-for/ba-p/4084608) to convert documents, such as invoices, into Markdown, then use GPT-3.5 Turbo to extract structured JSON data using the [Azure OpenAI Service](https://learn.microsoft.com/en-us/azure/ai-services/openai/overview).\n",
    "\n",
    "## Pre-requisites\n",
    "\n",
    "The notebook uses [PowerShell](https://learn.microsoft.com/powershell/scripting/install/installing-powershell) and [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) to deploy all necessary Azure resources. Both tools are available on Windows, macOS and Linux environments. It also uses [.NET 8](https://dotnet.microsoft.com/download/dotnet/8.0) to run the C# code that interacts with Azure AI Document Intelligence and Azure OpenAI Service.\n",
    "\n",
    "Running this notebook will deploy the following resources in your Azure subscription:\n",
    "- Azure Resource Group\n",
    "- Azure AI Document Intelligence (East US)\n",
    "- Azure OpenAI Service (East US)\n",
    "  - **Note**: You must be registered to deploy Azure OpenAI resources in your Azure Subscription. If you are not, please follow the [instructions in the registration process](https://learn.microsoft.com/en-us/legal/cognitive-services/openai/limited-access#registration-process) to enable this feature in Azure.\n",
    "- GPT-3.5 Turbo 16K model deployment (120K capacity)\n",
    "\n",
    "**Note**: Any GPT-3.5 Turbo model can be used with this sample. To provide a single region deployment for both the Azure AI Document Intelligence and Azure OpenAI Service, the GPT-3.5 Turbo 16K model is used in this sample in the East US region. For more information on the available GPT models, see the [Azure OpenAI Service documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-35). For more information on the available regions for Azure AI Document Intelligence preview features, see the [Azure AI Document Intelligence documentation](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/whats-new?view=doc-intel-4.0.0&tabs=csharp#february-2024)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy infrastructure with Az CLI & Bicep\n",
    "\n",
    "The following will prompt you to login to Azure. Once logged in, the current default subscription in your available subscriptions will be set for deployment.\n",
    "\n",
    "> **Note:** If you have multiple subscriptions, you can change the default subscription by running `az account set --subscription <subscription_id>`.\n",
    "\n",
    "Then, all the necessary Azure resources will be deployed, previously listed, using [Azure Bicep](https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/).\n",
    "\n",
    "The deployment occurs at the subscription level, creating a new resource group. The location of the deployment is set to **East US** and this can be changed to another location that supports the combination of [Azure AI Document Intelligence preview features](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/whats-new?view=doc-intel-4.0.0&tabs=csharp#february-2024) and a GPT-3.5 Turbo model deployment. This can be changed in the parameters provided to the PowerShell script in the next cell. You can tweak the model details also in the [`main.bicep`](./infra/main.bicep) file.\n",
    "\n",
    "Once deployed, the endpoints and API keys will be stored in the [`./config.env`](./config.env) file for use in the .NET code.\n",
    "\n",
    "### Understanding the deployment\n",
    "\n",
    "#### AI Document Intelligence\n",
    "\n",
    "An [Azure AI Document Intelligence](https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/overview) instance is deployed in the East US region. This is to enable support for the new Markdown content extraction feature of the pre-built Layout model.\n",
    "\n",
    "#### OpenAI Services\n",
    "\n",
    "An [Azure OpenAI Service](https://learn.microsoft.com/en-us/azure/ai-services/openai/overview) instance is deployed in the East US region. This is deployed with the `gpt-35-turbo-16k` model to be used for inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "pwsh"
    },
    "polyglot_notebook": {
     "kernelName": "pwsh"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "# Login to Azure\n",
    "Write-Host \"Checking if logged in to Azure...\"\n",
    "\n",
    "$loggedIn = az account show --query \"name\" -o tsv\n",
    "\n",
    "if ($loggedIn -ne $null) {\n",
    "    Write-Host \"Already logged in as $loggedIn\"\n",
    "} else {\n",
    "    Write-Host \"Logging in...\"\n",
    "    az login\n",
    "}\n",
    "\n",
    "# Retrieve the default subscription ID\n",
    "$subscriptionId = (\n",
    "    (\n",
    "        az account list -o json `\n",
    "            --query \"[?isDefault]\"\n",
    "    ) | ConvertFrom-Json\n",
    ").id\n",
    "\n",
    "# Set the subscription\n",
    "az account set --subscription $subscriptionId\n",
    "Write-Host \"Subscription set to $subscriptionId\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "pwsh"
    },
    "polyglot_notebook": {
     "kernelName": "pwsh"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "# Run Deploy-Infrastructure.ps1\n",
    ".\\Deploy-Infrastructure.ps1 -DeploymentName 'docintel-gpt-document-extraction' -Location 'eastus'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install .NET dependencies\n",
    "\n",
    "This notebook uses .NET to interact with the Azure AI Document Intelligence and Azure OpenAI Service. It takes advantage of the following NuGet packages:\n",
    "\n",
    "### Azure.AI.DocumentIntelligence\n",
    "\n",
    "The [Azure.AI.DocumentIntelligence](https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/documentintelligence/Azure.AI.DocumentIntelligence) library is used to interact with the Azure AI Document Intelligence service. This library provides a client to interact with the Azure AI Document Intelligence service to perform operations such as analyzing documents and extracting information from them.\n",
    "\n",
    "### Azure.AI.OpenAI\n",
    "\n",
    "The [Azure.AI.OpenAI](https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/openai/Azure.AI.OpenAI) library is used to interact with the Azure OpenAI Service. This library provides a client to interact with the Azure OpenAI Service to perform inference on the GPT-3.5 Turbo model.\n",
    "\n",
    "### DotNetEnv\n",
    "\n",
    "The [DotNetEnv](https://github.com/tonerdo/dotnet-env) library is used to load environment variables from a `.env` file which can be accessed via the `Environment.GetEnvironmentVariable(string)` method. This library is used to load the Azure OpenAI Service endpoint, key and model deployment name from the [`./config.env`](./config.env) file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "#r \"nuget:System.Text.Json, 8.0.1\"\n",
    "#r \"nuget:DotNetEnv, 3.0.0\"\n",
    "#r \"nuget:Azure.AI.OpenAI, 1.0.0-beta.14\"\n",
    "#r \"nuget:Azure.AI.DocumentIntelligence, 1.0.0-beta.2\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "using System.Net;\n",
    "using System.Net.Http;\n",
    "using System.Text.Json.Nodes;\n",
    "using System.Text.Json;\n",
    "using System.IO; \n",
    "\n",
    "using Azure;\n",
    "using Azure.AI.DocumentIntelligence;\n",
    "using Azure.AI.OpenAI;\n",
    "using DotNetEnv;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "Env.Load(\"config.env\");\n",
    "\n",
    "var openAIEndpoint = Environment.GetEnvironmentVariable(\"AZURE_OPENAI_ENDPOINT\");\n",
    "var openAIApiKey = Environment.GetEnvironmentVariable(\"AZURE_OPENAI_API_KEY\");\n",
    "var openAIModelDeployment = Environment.GetEnvironmentVariable(\"AZURE_OPENAI_MODEL_DEPLOYMENT_NAME\");\n",
    "var openAIApiVersion = \"2023-12-01-preview\";\n",
    "var documentIntelligenceEndpoint = Environment.GetEnvironmentVariable(\"AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT\");\n",
    "var documentIntelligenceApiKey = Environment.GetEnvironmentVariable(\"AZURE_DOCUMENT_INTELLIGENCE_KEY\");\n",
    "\n",
    "var pdfName = \"Invoice_1.pdf\";\n",
    "\n",
    "var documentIntelligenceClient = new DocumentIntelligenceClient(new Uri(documentIntelligenceEndpoint), new AzureKeyCredential(documentIntelligenceApiKey));\n",
    "var openAIClient = new OpenAIClient(new Uri(openAIEndpoint), new AzureKeyCredential(openAIApiKey));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform layout analysis on a document to extract Markdown content\n",
    "\n",
    "To be able to extract structured JSON data from a document, the document must first be converted into Markdown content. The following code demonstrates how to use the Azure AI Document Intelligence SDK to perform layout analysis on a document and return the result as Markdown.\n",
    "\n",
    "### Important notes for document analysis with Azure AI Document Intelligence\n",
    "\n",
    "- The document must be in one of the supported formats: PDF, JPEG/JPG, PNG, BMP, TIFF, HEIF, DOCX, XLSX, PPTX, or HTML.\n",
    "- The document file size limit is 500MB.\n",
    "- You can process only specific pages of a document by specifying the page numbers in the `pages` parameter of the `AnalyzeDocumentAsync` method in the example format `1,3-5,7-10`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [],
   "source": [
    "var markdownAnalysisContent = new AnalyzeDocumentContent()\n",
    "{\n",
    "    Base64Source = BinaryData.FromBytes(File.ReadAllBytes(pdfName))\n",
    "};\n",
    "\n",
    "Operation<AnalyzeResult> markdownAnalysisOperation = await documentIntelligenceClient.AnalyzeDocumentAsync(WaitUntil.Completed, \"prebuilt-layout\", markdownAnalysisContent, outputContentFormat: ContentFormat.Markdown);\n",
    "var markdown = markdownAnalysisOperation.Value.Content;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use GPT-3.5 Turbo to extract structured JSON data from the Markdown content\n",
    "\n",
    "Now that the document has been converted to Markdown, the GPT-3.5 Turbo model can be used to extract structured JSON data from the content. The following code demonstrates how to use the deployed Azure OpenAI Service using the .NET SDK to extract the structured JSON data.\n",
    "\n",
    "In this example, the payload object contains the following details:\n",
    "\n",
    "### System Prompt\n",
    "\n",
    "The system prompt is the instruction to the model that prescribes the model's behavior. They allow you to constrain the model's behavior to a specific task, making it more adaptable for specific use cases, such as extracting structured JSON data from documents.\n",
    "\n",
    "In this case, it is to extract structured JSON data from the content. Here is what we have provided:\n",
    "\n",
    "**You are an AI assistant that extracts data from documents and returns them as structured JSON objects. Do not return as a code block.**\n",
    "\n",
    "Learn more about [system prompts](https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/system-message).\n",
    "\n",
    "### User Prompt\n",
    "\n",
    "The user prompt is the input to the model that provides context for the model's response. It is the input that the model uses to generate a response. \n",
    "\n",
    "In this case, it is the content of the document plus some additional text context to help the model understand the task. Here is what we have provided:\n",
    "\n",
    "**Extract the data from this invoice. If a value is not present, provide null. Use the following structure: {\"company_name\":\"\",\"invoice_date\":\"\",\"products\":[{\"id\":\"\",\"unit_price\":\"\",\"quantity\":\"\",\"total\":\"\"}],\"total_amount\":\"\",\"signatures\":[{\"type\":\"\",\"has_signature\":\"\",\"signed_on\":\"\"}]}**\n",
    "\n",
    "Followed by the Markdown content extracted from the document.\n",
    "\n",
    "> **Note:** For the user prompt, it is ideal to provide a structure for the JSON response. Without one, the model will determine this for you and you may not get consistency across responses. \n",
    "\n",
    "This prompt ensures that the model understands the task, and the additional text context provides the model with the necessary information to extract the structured JSON data from the Markdown. This approach would result in a response similar to the following:\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"company_name\": \"CONTOSO\",\n",
    "  \"invoice_date\": \"2/27/2024\",\n",
    "  \"products\": [\n",
    "    {\n",
    "      \"id\": \"5-01-XX\",\n",
    "      \"unit_price\": \"1.00\",\n",
    "      \"quantity\": \"1.0\",\n",
    "      \"total\": \"1.00\"\n",
    "    },\n",
    "    {\n",
    "      \"id\": \"5-02-XX\",\n",
    "      \"unit_price\": \"1.50\",\n",
    "      \"quantity\": \"5.0\",\n",
    "      \"total\": \"7.50\"\n",
    "    }\n",
    "  ],\n",
    "  \"total_amount\": \"8.50\",\n",
    "  \"signatures\": [\n",
    "    {\n",
    "      \"type\": \"Distributor\",\n",
    "      \"has_signature\": true,\n",
    "      \"signed_on\": \"2/27/2024\"\n",
    "    },\n",
    "    {\n",
    "      \"type\": \"Customer\",\n",
    "      \"has_signature\": false,\n",
    "      \"signed_on\": null\n",
    "    }\n",
    "  ]\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "dotnet_interactive": {
     "language": "csharp"
    },
    "polyglot_notebook": {
     "kernelName": "csharp"
    },
    "vscode": {
     "languageId": "polyglot-notebook"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"company_name\": \"CONTOSO\",\n",
      "  \"invoice_date\": \"2/27/2024\",\n",
      "  \"products\": [\n",
      "    {\n",
      "      \"id\": \"5-01-XX\",\n",
      "      \"unit_price\": \"1.00\",\n",
      "      \"quantity\": \"1.0\",\n",
      "      \"total\": \"1.00\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-02-XX\",\n",
      "      \"unit_price\": \"1.50\",\n",
      "      \"quantity\": \"5.0\",\n",
      "      \"total\": \"7.50\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-03-XX\",\n",
      "      \"unit_price\": \"5.75\",\n",
      "      \"quantity\": \"2.0\",\n",
      "      \"total\": \"11.50\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-04-XX\",\n",
      "      \"unit_price\": \"2.80\",\n",
      "      \"quantity\": \"6.0\",\n",
      "      \"total\": \"16.80\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-05-XX\",\n",
      "      \"unit_price\": \"4.45\",\n",
      "      \"quantity\": \"13.0\",\n",
      "      \"total\": \"57.85\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-06-XX\",\n",
      "      \"unit_price\": \"2.20\",\n",
      "      \"quantity\": \"11.0\",\n",
      "      \"total\": \"24.20\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-07-XX\",\n",
      "      \"unit_price\": \"20.05\",\n",
      "      \"quantity\": \"5.0\",\n",
      "      \"total\": \"100.25\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-08-XX\",\n",
      "      \"unit_price\": \"9.50\",\n",
      "      \"quantity\": \"1.0\",\n",
      "      \"total\": \"9.50\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-09-XX\",\n",
      "      \"unit_price\": \"5.00\",\n",
      "      \"quantity\": \"5.0\",\n",
      "      \"total\": \"25.00\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-10-XX\",\n",
      "      \"unit_price\": \"0.75\",\n",
      "      \"quantity\": \"2.0\",\n",
      "      \"total\": \"1.50\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-11-XX\",\n",
      "      \"unit_price\": \"1.00\",\n",
      "      \"quantity\": \"8.0\",\n",
      "      \"total\": \"8.0\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-12-XX\",\n",
      "      \"unit_price\": \"15.00\",\n",
      "      \"quantity\": \"2.5\",\n",
      "      \"total\": \"37.50\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-13-XX\",\n",
      "      \"unit_price\": \"11.50\",\n",
      "      \"quantity\": \"1.0\",\n",
      "      \"total\": \"11.50\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-14-XX\",\n",
      "      \"unit_price\": \"5.25\",\n",
      "      \"quantity\": \"4.0\",\n",
      "      \"total\": \"21.00\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-15-XX\",\n",
      "      \"unit_price\": \"99.99\",\n",
      "      \"quantity\": \"1.0\",\n",
      "      \"total\": \"99.99\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-16-XX\",\n",
      "      \"unit_price\": \"1,350.00\",\n",
      "      \"quantity\": \"2.0\",\n",
      "      \"total\": \"2,700.00\"\n",
      "    },\n",
      "    {\n",
      "      \"id\": \"5-17-XX\",\n",
      "      \"unit_price\": \"56.95\",\n",
      "      \"quantity\": \"3.0\",\n",
      "      \"total\": \"170.85\"\n",
      "    }\n",
      "  ],\n",
      "  \"total_amount\": \"3,303.94\",\n",
      "  \"signatures\": [\n",
      "    {\n",
      "      \"type\": \"Distributor signature\",\n",
      "      \"has_signature\": true,\n",
      "      \"signed_on\": \"2/27/2024\"\n",
      "    },\n",
      "    {\n",
      "      \"type\": \"Customer signature\",\n",
      "      \"has_signature\": false,\n",
      "      \"signed_on\": null\n",
      "    }\n",
      "  ]\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "var jsonStructure = new {\n",
    "    company_name = \"\",\n",
    "    invoice_date = \"\",\n",
    "    products = new [] {\n",
    "        new {\n",
    "            id = \"\",\n",
    "            unit_price = \"\",\n",
    "            quantity = \"\",\n",
    "            total = \"\"\n",
    "        }\n",
    "    },\n",
    "    total_amount = \"\",\n",
    "    signatures = new [] {\n",
    "        new {\n",
    "            type = \"\",\n",
    "            has_signature = \"\",\n",
    "            signed_on = \"\"\n",
    "        }\n",
    "    }\n",
    "};\n",
    "\n",
    "ChatCompletionsOptions options = new ChatCompletionsOptions()\n",
    "{\n",
    "    DeploymentName = openAIModelDeployment,\n",
    "    MaxTokens = 4096,\n",
    "    Temperature = 0.1f,\n",
    "    NucleusSamplingFactor = 0.1f\n",
    "};\n",
    "\n",
    "options.Messages.Add(new ChatRequestSystemMessage(\"You are an AI assistant that extracts data from documents and returns them as structured JSON objects. Do not return as a code block.\"));\n",
    "options.Messages.Add(new ChatRequestUserMessage($\"Extract the data from this invoice. If a value is not present, provide null. Use the following structure: {JsonSerializer.Serialize(jsonStructure)}\"));\n",
    "options.Messages.Add(new ChatRequestUserMessage(markdown));\n",
    "\n",
    "Response<ChatCompletions> response = await openAIClient.GetChatCompletionsAsync(options);\n",
    "foreach(var completion in response.Value.Choices)\n",
    "{\n",
    "    Console.WriteLine(completion.Message.Content);\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}