examples/langchain-bedrock/rag_with_bedrock/RAG_with_reranking_claudev2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1421cda1-9407-4269-8e1f-aef69c69a28c",
   "metadata": {},
   "source": [
    "# Question Answering on HuggingFace blog posts using RAG\n",
    "\n",
    "In this notebook we demonstrate building a simple Question Answering application with GenAI.\n",
    "\n",
    "## Building a RAG application with Anthropic Claude v2 and Cohere Embed v3 on Amazon Bedrock\n",
    "\n",
    "Models used:\n",
    "- Large Language Model: **anthropic.claude-v2** on Amazon Bedrock\n",
    "- Embedding Model: **cohere.embed-english-v3** on Amazon Bedrock\n",
    "\n",
    "\n",
    "Vector Store (to store embeddings): **[Qdrant](https://qdrant.tech/documentation/quick-start/)**\n",
    "\n",
    "LangChain's [LCEL](https://python.langchain.com/docs/expression_language/why#lcel) to implement a sequential chain to answer questions on a blog posts from the internet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "923276b2-0293-4d62-abcf-f43adddb1ee4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install boto3 anthropic_bedrock qdrant_client transformers langchain rich  \"unstructured[all-docs]\" --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9052957e-d255-4169-b22a-a34a01e35ac3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "sys.path.append(os.path.abspath(os.path.join(os.getcwd(), os.pardir)) + \"/utils\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5b1720aa-61ff-40d4-a966-7980c1fed42b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "from typing import List\n",
    "\n",
    "import boto3\n",
    "from anthropic_bedrock import AI_PROMPT, HUMAN_PROMPT\n",
    "from rich import print\n",
    "from utils import get_inference_parameters, get_model_ids\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "%load_ext rich\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf59ad36-577a-423c-8488-df963776ecc2",
   "metadata": {},
   "source": [
    "### Get Bedrock Model IDs for Cohere and Anthropic\n",
    "\n",
    "We need model ids to instantiate both LLM and Embeddings with LangChain\n",
    "\n",
    "`utils.get_model_ids` function can help get the model ids. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "774b8613-ed75-43a1-b950-43f2bffa9772",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span><span style=\"color: #008000; text-decoration-color: #008000\">'cohere.embed-english-v3'</span>, <span style=\"color: #008000; text-decoration-color: #008000\">'cohere.embed-multilingual-v3'</span><span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m[\u001b[0m\u001b[32m'cohere.embed-english-v3'\u001b[0m, \u001b[32m'cohere.embed-multilingual-v3'\u001b[0m\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Model ids for Embeddings\n",
    "provider = \"Cohere\"  # Providers can be Amazon, Anthropic, Cohere\n",
    "output_modality = \"EMBEDDING\"  # Can be TEXT, EMBEDDING\n",
    "embed_model_ids = get_model_ids(provider, output_modality)\n",
    "print(embed_model_ids)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0b5c9e1f-5f51-46a4-a172-04fe4071991c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">[</span>\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'anthropic.claude-instant-v1:2:100k'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'anthropic.claude-instant-v1'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'anthropic.claude-v1:3:18k'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'anthropic.claude-v1:3:100k'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'anthropic.claude-v1'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'anthropic.claude-v2:0:18k'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'anthropic.claude-v2:0:100k'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'anthropic.claude-v2:1:18k'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'anthropic.claude-v2:1:200k'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'anthropic.claude-v2:1'</span>,\n",
       "    <span style=\"color: #008000; text-decoration-color: #008000\">'anthropic.claude-v2'</span>\n",
       "<span style=\"font-weight: bold\">]</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m[\u001b[0m\n",
       "    \u001b[32m'anthropic.claude-instant-v1:2:100k'\u001b[0m,\n",
       "    \u001b[32m'anthropic.claude-instant-v1'\u001b[0m,\n",
       "    \u001b[32m'anthropic.claude-v1:3:18k'\u001b[0m,\n",
       "    \u001b[32m'anthropic.claude-v1:3:100k'\u001b[0m,\n",
       "    \u001b[32m'anthropic.claude-v1'\u001b[0m,\n",
       "    \u001b[32m'anthropic.claude-v2:0:18k'\u001b[0m,\n",
       "    \u001b[32m'anthropic.claude-v2:0:100k'\u001b[0m,\n",
       "    \u001b[32m'anthropic.claude-v2:1:18k'\u001b[0m,\n",
       "    \u001b[32m'anthropic.claude-v2:1:200k'\u001b[0m,\n",
       "    \u001b[32m'anthropic.claude-v2:1'\u001b[0m,\n",
       "    \u001b[32m'anthropic.claude-v2'\u001b[0m\n",
       "\u001b[1m]\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Model ids for LLMs\n",
    "provider = \"Anthropic\"\n",
    "output_modality = \"TEXT\"\n",
    "llm_model_ids = get_model_ids(provider, output_modality)\n",
    "print(llm_model_ids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52604004-1615-44f6-9d72-79cc11b7e769",
   "metadata": {},
   "source": [
    "### Instantiate LLM and Embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a5882eb7-917a-4d02-bd51-0cb5ae83558f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.embeddings.bedrock import BedrockEmbeddings\n",
    "from langchain.llms.bedrock import Bedrock\n",
    "\n",
    "region = \"us-west-2\"\n",
    "b_client = boto3.client(\"bedrock-runtime\", region_name=region)\n",
    "model_kwargs = get_inference_parameters(\n",
    "    \"anthropic\"\n",
    ")  # We need pass in model_kwargs for a model\n",
    "# llm_model_id = \"anthropic.claude-v2\"\n",
    "llm_model_id = \"anthropic.claude-instant-v1\"\n",
    "embed_model_id = \"cohere.embed-english-v3\"\n",
    "\n",
    "llm = Bedrock(\n",
    "    client=b_client,\n",
    "    model_kwargs=model_kwargs,\n",
    "    model_id=llm_model_id,\n",
    "    region_name=region,\n",
    ")\n",
    "embeddings = BedrockEmbeddings(\n",
    "    client=b_client, model_id=embed_model_id, region_name=region\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "839bc199-077d-40e7-a927-2e4fe1eeb2e1",
   "metadata": {},
   "source": [
    "### Scrape a few blogs posts for encoding\n",
    "\n",
    "We use LangChain `AsyncHtmlLoader` document loader to download blog posts as html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0ce1aa4b-044e-42e7-96b4-5cb357490955",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fetching pages: 100%|########################################################################################################################| 5/5 [00:00<00:00,  5.14it/s]\n"
     ]
    }
   ],
   "source": [
    "from langchain.document_loaders import AsyncHtmlLoader\n",
    "\n",
    "urls = [\n",
    "    \"https://huggingface.co/blog/moe\",\n",
    "    \"https://huggingface.co/blog/setfit-absa\",\n",
    "    \"https://huggingface.co/blog/prodigy-hf\",\n",
    "    \"https://huggingface.co/blog/personal-copilot\",\n",
    "    \"https://aws.amazon.com/blogs/machine-learning/mitigate-hallucinations-through-retrieval-augmented-generation-using-pinecone-vector-database-llama-2-from-amazon-sagemaker-jumpstart/\",\n",
    "]\n",
    "html_loader = AsyncHtmlLoader(urls)\n",
    "html_docs = html_loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f34c24d5-8df5-4eae-aae2-4a2a68172d43",
   "metadata": {},
   "source": [
    "### Convert HTML docs into Text\n",
    "\n",
    "We use Unstructured [partition_html](https://unstructured-io.github.io/unstructured/core/partition.html#partition-html) to extract text from html. `partition_html` helps to clean and group html text.\n",
    "\n",
    "- group articles by title using `chunking_strategy='by_title'`\n",
    "- `assemble_articles = True`\n",
    "- `skip_headers_and_footers = True`\n",
    "- Clean any non ascii chars in text with `clean_non_ascii_chars`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8044c08e-b7e5-4c22-860f-222e8eab5fc8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.docstore.document import Document\n",
    "from unstructured.cleaners.core import clean_non_ascii_chars\n",
    "from unstructured.partition.html import partition_html\n",
    "\n",
    "\n",
    "# Add documentation to the below function\n",
    "def extract_text_chunks_from_html(urls, html_docs) -> List[Document]:\n",
    "    \"\"\" \"\n",
    "    Function to reformat html_docs from html to plain text\n",
    "    Input: urls, html_docs\n",
    "    Output: List[Document]\n",
    "    \"\"\"\n",
    "    extracted_docs = []\n",
    "    for url, doc in zip(urls, html_docs):\n",
    "        elements = partition_html(\n",
    "            text=doc.page_content,\n",
    "            html_assemble_articles=True,\n",
    "            skip_headers_and_footers=True,\n",
    "            chunking_strategy=\"by_title\",\n",
    "        )\n",
    "        extracted_text = \"\".join([e.text for e in elements])\n",
    "        # extract metadata\n",
    "        metadata = doc.metadata\n",
    "        metadata[\"language\"] = \"en\"\n",
    "        # extract links if available and append to metadata\n",
    "        extracted_links = []\n",
    "        for element in elements:\n",
    "            if element.metadata.links is not None:\n",
    "                print(element.metadata.links)\n",
    "                link = element.metadata.links[0][\"url\"][1:]\n",
    "                extracted_links.append(link)\n",
    "        # Add extracted links to metadata as references\n",
    "        if len(extracted_links) > 0:\n",
    "            metadata[\"references\"] = extracted_links\n",
    "        doc.page_content = clean_non_ascii_chars(extracted_text)\n",
    "        doc.metadata = metadata\n",
    "        extracted_docs.append(doc)\n",
    "    return extracted_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "db017348-be84-4ddd-9bf9-35e3d02d60c8",
   "metadata": {},
   "outputs": [],
   "source": [
    "extracted_docs = extract_text_chunks_from_html(urls, html_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6930ab2a-18ac-4e16-9df9-2f295f1c97a5",
   "metadata": {},
   "source": [
    "### Split docs into chunks the size of Embedding models max length (512)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8bcf5fdc-cdc9-400a-86f7-c6d863cd5614",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">46</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;36m46\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "# Remember TextSplitter chunk_size is != model max length\n",
    "splitter = RecursiveCharacterTextSplitter(\n",
    "    add_start_index=True, chunk_size=2048, chunk_overlap=0\n",
    ")\n",
    "doc_chunks = splitter.split_documents(documents=extracted_docs)\n",
    "print(len(doc_chunks))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19bef56c-3f06-4697-8630-0f57790e47e7",
   "metadata": {},
   "source": [
    "`cohere.embed-english-v3` model max input length is *512* tokens and output is **1024** dimensional vector\n",
    "\n",
    "To calculate the number of tokens we need the model's tokenizer. \n",
    "\n",
    "We use `Cohere's` tokenizer from HuggingFace available [here](https://huggingface.co/Cohere/Cohere-embed-english-v3.0/tree/main)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5a5f96b2-6bba-4918-9972-2eb23c39cd65",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n",
      "Token indices sequence length is longer than the specified maximum sequence length for this model (578 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Chunk <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span> has <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">578</span> tokens\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Chunk \u001b[1;36m4\u001b[0m has \u001b[1;36m578\u001b[0m tokens\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Chunk <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">22</span> has <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">526</span> tokens\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Chunk \u001b[1;36m22\u001b[0m has \u001b[1;36m526\u001b[0m tokens\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Chunk <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">23</span> has <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">619</span> tokens\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Chunk \u001b[1;36m23\u001b[0m has \u001b[1;36m619\u001b[0m tokens\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Chunk <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">31</span> has <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">545</span> tokens\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Chunk \u001b[1;36m31\u001b[0m has \u001b[1;36m545\u001b[0m tokens\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Chunk <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">32</span> has <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">530</span> tokens\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Chunk \u001b[1;36m32\u001b[0m has \u001b[1;36m530\u001b[0m tokens\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Chunk <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">37</span> has <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">537</span> tokens\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Chunk \u001b[1;36m37\u001b[0m has \u001b[1;36m537\u001b[0m tokens\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from transformers import AutoTokenizer\n",
    "from unstructured.staging.huggingface import chunk_by_attention_window\n",
    "\n",
    "hf_model_id = \"Cohere/Cohere-embed-english-v3.0\"\n",
    "tokenizer = AutoTokenizer.from_pretrained(hf_model_id)\n",
    "\n",
    "\n",
    "def get_token_len(text) -> int:\n",
    "    num_tokens = tokenizer.tokenize(text)\n",
    "    return len(num_tokens)\n",
    "\n",
    "\n",
    "# Sanity check if chunks have more tokens then the model can accept\n",
    "for idx, _chunk in enumerate(doc_chunks):\n",
    "    num_tokens = get_token_len(_chunk.page_content)\n",
    "    if num_tokens > 512:\n",
    "        print(f\"Chunk {idx} has {num_tokens} tokens\")\n",
    "        # print(_chunk)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8743d1e1-4796-47b0-817d-f9d36b14c36b",
   "metadata": {},
   "source": [
    "### Add docs to vectorstore (Qdrant)\n",
    "\n",
    "Install and run `qdrant` vector store locally using docker\n",
    "\n",
    "Refer here for Installation: <https://qdrant.tech/documentation/quick-start/>\n",
    "\n",
    "Qdrant should be running at port `6333` on localhost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f37788b9-eb24-44c9-b731-fd739a5df56d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Connected to collection: <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\">mlblogs_coherev3</span> ✅\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Connected to collection: \u001b[1;35mmlblogs_coherev3\u001b[0m ✅\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from langchain.vectorstores.qdrant import Qdrant\n",
    "from qdrant_client import QdrantClient\n",
    "\n",
    "collection_name = \"mlblogs_coherev3\"  # define collection name\n",
    "qclient = QdrantClient(location=\"localhost\", port=6333)\n",
    "collection_status = qclient.get_collection(collection_name=collection_name).status\n",
    "\n",
    "if collection_status == \"green\":\n",
    "    print(f\"Connected to collection: [b magenta]{collection_name}[/b magenta] ✅\")\n",
    "    # Instantiating Qdrant client is weird with LangChain\n",
    "    db = Qdrant(\n",
    "        client=qclient,\n",
    "        collection_name=collection_name,\n",
    "        distance_strategy=\"cosine\",\n",
    "        embeddings=embeddings,\n",
    "    )\n",
    "else:\n",
    "    # Add documents to vector db with force_recreate = True for testing\n",
    "    db = db.from_documents(\n",
    "        documents=doc_chunks,\n",
    "        embedding=embeddings,\n",
    "        collection_name=collection_name,\n",
    "        force_recreate=True,  # Set this to false in PROD\n",
    "    )\n",
    "    print(\n",
    "        f\"Added [b]{len(doc_chunks)}[/b] to collection: [b green]{collection_name}[/b green] ✅\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5236003c-f642-4831-a8e6-44651827c11a",
   "metadata": {},
   "source": [
    "### Testing Qdrant retriever\n",
    "\n",
    "For a given query, retrieve **top 5** documents and test if the document chunks returned are relevant to the query.\n",
    "\n",
    "We set our `search_type` to `similarity`, we can also try with `mmr`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "33bab13b-51df-4820-b3a7-158658b221f5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "\n",
       "\u001b[1m[\u001b[0m\n",
       "    \u001b[1;35mDocument\u001b[0m\u001b[1m(\u001b[0m\n",
       "        \u001b[33mpage_content\u001b[0m=\u001b[32m'Above output shows that were returning relevant contexts to help us answer our question. Since we top_k = 1, index.query returned the top result along side the metadata which reads Managed Spot Training can be used with all instances supported in Amazon.Augmenting the Prompt\\n\\nUse the retrieved contexts to augment the prompt and decide on a maximum amount of context to feed into the LLM. Use the 1000 characters limit to iteratively add each returned context to the prompt until you exceed the content length.Augmenting the Prompt\\n\\nFeed the context_str into the LLM prompt as shown in the following screen capture:\\n\\n\u001b[0m\u001b[32m[\u001b[0m\u001b[32mInput\u001b[0m\u001b[32m]\u001b[0m\u001b[32m: Which instances can I use with Managed Spot Training in SageMaker?\\n\\n\u001b[0m\u001b[32m[\u001b[0m\u001b[32mOutput\u001b[0m\u001b[32m]\u001b[0m\u001b[32m: Based on the context provided, you can use Managed Spot Training with all instances supported in Amazon SageMaker. Therefore, the answer is:\\n\\n\\nAll instances supported in Amazon SageMaker.\\n\\nThe logic works, so wrap it up into a single function to keep things clean.You can now ask questions like those shown in the following:\\n\\nClean up\\n\\nTo stop incurring any unwanted charges, delete the model and endpoint.Conclusion\\n\\nIn this post, we introduced you to RAG with open-access LLMs on SageMaker. We also showed how to deploy Amazon SageMaker Jumpstart models with Llama 2, Hugging Face LLMs with Flan T5, and embedding models with MiniLM.\\n\\nWe implemented a complete end-to-end RAG pipeline using our open-access models and a Pinecone vector index. Using this, we showed how to minimize hallucinations, and keep LLM knowledge up to date, and ultimately enhance the user experience and trust in our systems.To run this example on your own, clone this GitHub repository and walkthrough the previous steps using the Question Answering notebook on GitHub.About the authors'\u001b[0m,\n",
       "        \u001b[33mmetadata\u001b[0m=\u001b[1m{\u001b[0m\n",
       "            \u001b[32m'language'\u001b[0m: \u001b[32m'en'\u001b[0m,\n",
       "            \u001b[32m'source'\u001b[0m: \u001b[32m'https://aws.amazon.com/blogs/machine-learning/mitigate-hallucinations-through-retrieval-augmented-generation-using-pinecone-vector-database-llama-2-from-amazon-sagemaker-jumpstart/'\u001b[0m,\n",
       "            \u001b[32m'start_index'\u001b[0m: \u001b[1;36m8724\u001b[0m,\n",
       "            \u001b[32m'title'\u001b[0m: \u001b[32m'Mitigate hallucinations through Retrieval Augmented Generation using Pinecone vector database & Llama-2 from Amazon SageMaker JumpStart | AWS Machine Learning Blog'\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "    \u001b[1m)\u001b[0m,\n",
       "    \u001b[1;35mDocument\u001b[0m\u001b[1m(\u001b[0m\n",
       "        \u001b[33mpage_content\u001b[0m=\u001b[32m'Pinecone will handle the retrieval component of RAG, but you need two more critical components: somewhere to run the LLM inference and somewhere to run the embedding model.Amazon SageMaker Studio an integrated development environment \u001b[0m\u001b[32m(\u001b[0m\u001b[32mIDE\u001b[0m\u001b[32m)\u001b[0m\u001b[32m that provides a single web-based visual interface where you can access purpose-built tools to perform all machine learning \u001b[0m\u001b[32m(\u001b[0m\u001b[32mML\u001b[0m\u001b[32m)\u001b[0m\u001b[32m development. It provides SageMaker JumpStart which is a model hub where users can locate, preview, and launch a particular model in their own SageMaker account. It provides pretrained, publicly available and proprietary models for a wide range of problem types, including Foundation Models.Amazon SageMaker Studio provides the ideal environment for developing RAG-enabled LLM pipelines. First, using the AWS console, go to Amazon SageMaker & create a SageMaker Studio domain and open a Jupyter Studio notebook.\\n\\nPrerequisites\\n\\nComplete the following prerequisite steps:\\n\\nSet up Amazon SageMaker Studio.\\n\\nOnboard to an Amazon SageMaker Domain.\\n\\nSign up for a free-tier Pinecone Vector Database.\\n\\nPrerequisite libraries: SageMaker Python SDK, Pinecone ClientSolution Walkthrough\\n\\nUsing SageMaker Studio notebook, we first need install prerequisite libraries:Deploying an LLM\\n\\nIn this post, we discuss two approaches to deploying an LLM. The first is through the HuggingFaceModel object. You can use this when deploying LLMs \u001b[0m\u001b[32m(\u001b[0m\u001b[32mand embedding models\u001b[0m\u001b[32m)\u001b[0m\u001b[32m directly from the Hugging Face model hub.\\n\\nFor example, you can create a deployable config for the google/flan-t5-xl model as shown in the following screen capture:\\n\\nWhen deploying models directly from Hugging Face, initialize the my_model_configuration with the following:An env config tells us which model we want to use and for what task.\\n\\nOur SageMaker execution role gives us permissions to deploy our model.\\n\\nAn image_uri is an image config specifically for deploying LLMs from Hugging Face.'\u001b[0m,\n",
       "        \u001b[33mmetadata\u001b[0m=\u001b[1m{\u001b[0m\n",
       "            \u001b[32m'language'\u001b[0m: \u001b[32m'en'\u001b[0m,\n",
       "            \u001b[32m'source'\u001b[0m: \u001b[32m'https://aws.amazon.com/blogs/machine-learning/mitigate-hallucinations-through-retrieval-augmented-generation-using-pinecone-vector-database-llama-2-from-amazon-sagemaker-jumpstart/'\u001b[0m,\n",
       "            \u001b[32m'start_index'\u001b[0m: \u001b[1;36m1326\u001b[0m,\n",
       "            \u001b[32m'title'\u001b[0m: \u001b[32m'Mitigate hallucinations through Retrieval Augmented Generation using Pinecone vector database & Llama-2 from Amazon SageMaker JumpStart | AWS Machine Learning Blog'\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "    \u001b[1m)\u001b[0m,\n",
       "    \u001b[1;35mDocument\u001b[0m\u001b[1m(\u001b[0m\n",
       "        \u001b[33mpage_content\u001b[0m=\u001b[32m'Mitigate hallucinations through Retrieval Augmented Generation using Pinecone vector database & Llama-2 from Amazon SageMaker JumpStartDespite the seemingly unstoppable adoption of LLMs across industries, they are one component of a broader technology ecosystem that is powering the new AI wave. Many conversational AI use cases require LLMs like Llama 2, Flan T5, and Bloom to respond to user queries. These models rely on parametric knowledge to answer questions. The model learns this knowledge during training and encodes it into the model parameters. In order to update this knowledge, we must retrain the LLM, which takes a lot of time and money.Fortunately, we can also use source knowledge to inform our LLMs. Source knowledge is information fed into the LLM through an input prompt. One popular approach to providing source knowledge is Retrieval Augmented Generation \u001b[0m\u001b[32m(\u001b[0m\u001b[32mRAG\u001b[0m\u001b[32m)\u001b[0m\u001b[32m. Using RAG, we retrieve relevant information from an external data source and feed that information into the LLM.In this blog post, well explore how to deploy LLMs such as Llama-2 using Amazon Sagemaker JumpStart and keep our LLMs up to date with relevant information through Retrieval Augmented Generation \u001b[0m\u001b[32m(\u001b[0m\u001b[32mRAG\u001b[0m\u001b[32m)\u001b[0m\u001b[32m using the Pinecone vector database in order to prevent AI Hallucination.Retrieval Augmented Generation \u001b[0m\u001b[32m(\u001b[0m\u001b[32mRAG\u001b[0m\u001b[32m)\u001b[0m\u001b[32m in Amazon SageMaker'\u001b[0m,\n",
       "        \u001b[33mmetadata\u001b[0m=\u001b[1m{\u001b[0m\n",
       "            \u001b[32m'language'\u001b[0m: \u001b[32m'en'\u001b[0m,\n",
       "            \u001b[32m'source'\u001b[0m: \u001b[32m'https://aws.amazon.com/blogs/machine-learning/mitigate-hallucinations-through-retrieval-augmented-generation-using-pinecone-vector-database-llama-2-from-amazon-sagemaker-jumpstart/'\u001b[0m,\n",
       "            \u001b[32m'start_index'\u001b[0m: \u001b[1;36m0\u001b[0m,\n",
       "            \u001b[32m'title'\u001b[0m: \u001b[32m'Mitigate hallucinations through Retrieval Augmented Generation using Pinecone vector database & Llama-2 from Amazon SageMaker JumpStart | AWS Machine Learning Blog'\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "    \u001b[1m)\u001b[0m,\n",
       "    \u001b[1;35mDocument\u001b[0m\u001b[1m(\u001b[0m\n",
       "        \u001b[33mpage_content\u001b[0m=\u001b[32m'Alternatively, SageMaker has a set of models directly compatible with a simpler JumpStartModel object. Many popular LLMs like Llama 2 are supported by this model, which can be initialized as shown in the following screen capture:For both versions of my_model, deploy them as shown in the following screen capture:Querying the pre-trained LLM\\n\\nWith our initialized LLM endpoint, you can begin querying. The format of our queries may vary \u001b[0m\u001b[32m(\u001b[0m\u001b[32mparticularly between conversational and non-conversational LLMs\u001b[0m\u001b[32m)\u001b[0m\u001b[32m, but the process is generally the same. For the Hugging Face model, do the following:\\n\\nYou can find the solution in the GitHub repository.\\n\\nThe generated answer were receiving here doesnt make much sense  it is a hallucination.Providing Additional Context to LLM\\n\\nLlama 2 attempts to answer our question based solely on internal parametric knowledge. Clearly, the model parameters do not store knowledge of which instances we can with managed spot training in SageMaker.\\n\\nTo answer this question correctly, we must use source knowledge. That is, we give additional information to the LLM via the prompt. Lets add that information directly as additional context for the model.We now see the correct answer to the question; that was easy! However, a user is unlikely to insert contexts into their prompts, they would already know the answer to their question.\\n\\nRather than manually inserting a single context, automatically identify relevant information from a more extensive database of information. For that, you will need Retrieval Augmented Generation.Retrieval Augmented Generation'\u001b[0m,\n",
       "        \u001b[33mmetadata\u001b[0m=\u001b[1m{\u001b[0m\n",
       "            \u001b[32m'language'\u001b[0m: \u001b[32m'en'\u001b[0m,\n",
       "            \u001b[32m'source'\u001b[0m: \u001b[32m'https://aws.amazon.com/blogs/machine-learning/mitigate-hallucinations-through-retrieval-augmented-generation-using-pinecone-vector-database-llama-2-from-amazon-sagemaker-jumpstart/'\u001b[0m,\n",
       "            \u001b[32m'start_index'\u001b[0m: \u001b[1;36m3238\u001b[0m,\n",
       "            \u001b[32m'title'\u001b[0m: \u001b[32m'Mitigate hallucinations through Retrieval Augmented Generation using Pinecone vector database & Llama-2 from Amazon SageMaker JumpStart | AWS Machine Learning Blog'\u001b[0m\n",
       "        \u001b[1m}\u001b[0m\n",
       "    \u001b[1m)\u001b[0m\n",
       "\u001b[1m]\u001b[0m"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# define retriever args\n",
    "retriever_kwargs = {\"search_type\": \"similarity\", \"top_k\": 5}\n",
    "retriever = db.as_retriever(**retriever_kwargs)\n",
    "query = \"How to do RAG in Amazon SageMaker\"  # Change this to another question to test\n",
    "retriever.get_relevant_documents(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3870a9f7-dca3-40c6-8ab0-7fc3aaadd9e8",
   "metadata": {},
   "source": [
    "### Create RAG prompt\n",
    "\n",
    "Here we create a RAG prompt with re-ranking built in. \n",
    "\n",
    "We ask Claude to first rerank the documents in context from 1 to 5 and evaluate the relevance accordingly. Then we ask to use *only the top ranked documents* to answer the question.\n",
    "\n",
    "Adding the `<question>` and the end of the prompt increases LLM output quality (Best practice)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7dd8c241-99b0-44ca-96d1-d3bf37b9fa3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.prompts.prompt import PromptTemplate\n",
    "\n",
    "rag_template = \"\"\"Given the following retrieved context documents, your task is to rerank the contexts based on their relevance to truthfully and completely answering the user's question provided in the <question> tags.\n",
    "Then use only the top ranked context to provide an answer to the question.\n",
    "If you don't have the information just say so. Sometimes the retrieved documents may not contain the information you need. In such cases, say 'Sorry, I don't have enough information'.\n",
    "\n",
    "Retrieved documents: \n",
    "\n",
    "{context}\n",
    "\n",
    "please rerank the documents above from most (1) to least (5) relevant in directly and fully answering the user's specific question \"<question>\".\n",
    "Evaluate relevance based on how precisely each document answers this question if taken alone.\n",
    "\n",
    "Document ranking:\n",
    "1.\n",
    "2.\n",
    "3.\n",
    "4.\n",
    "5.\n",
    "\n",
    "Now using only the top ranked documents, please provide a clear and concise answer to the question in <answer> tags.\n",
    "\n",
    "Do NOT output <answer> with any preamble. Just answer the question in a direct manner.\n",
    "\n",
    "User's question: <question> {question} </question>\"\"\"\n",
    "\n",
    "# We need to add HUMAN_PROMPT and AI_PROMPT to the template (Anthropic specific)\n",
    "rag_prompt = PromptTemplate.from_template(\n",
    "    template=f\"{HUMAN_PROMPT}{rag_template}{AI_PROMPT}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a100cce6-80d8-4905-8e1e-8ddfb9b8355e",
   "metadata": {},
   "source": [
    "### Create sequential chain\n",
    "\n",
    "- First, we retrieve the docs from vector store\n",
    "- Then the retrieved context docs are formatted with index like below, this helps the model to re-rank the docs efficiently.\n",
    "  ```text\n",
    "  <context1>..</context1>\n",
    "  <context2>..</context2>\n",
    "  ...\n",
    "  ```\n",
    "  - `format_context_docs` function format docs and returns the formatted string back\n",
    "- before the prompt, we call `context` with `retriever` and pipe it to function `format_context_docs` that'd inject the formatted context docs into the `{context}` variable in the prompt.\n",
    "- Also, we define `question` to be a `RunnablePassthrough`, this allows the question to be passed in directly into the `invoke_chain` call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "6b0589f1-bd13-4d3f-934a-9ecf377f3bb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.schema.output_parser import StrOutputParser\n",
    "from langchain.schema.runnable import RunnablePassthrough\n",
    "\n",
    "# Passing this to context will get back 5 docs\n",
    "retriever_kwargs = {\"search_type\": \"similarity\", \"top_k\": 5}\n",
    "retriever = db.as_retriever(**retriever_kwargs)\n",
    "\n",
    "\n",
    "# format retrieved docs within <context{idx}></context{idx}> tags\n",
    "def format_context_docs(docs):\n",
    "    context_string = \"\"\n",
    "    for idx, _d in enumerate(docs):\n",
    "        otag = f\"<context{idx+1}>\"\n",
    "        ctag = f\"</context{idx+1}>\"\n",
    "        c_text = f\"{otag} {_d.page_content} {ctag}\\n\"\n",
    "        context_string += c_text\n",
    "    return context_string\n",
    "\n",
    "\n",
    "# Input variables to RAG prompt are `context` and `question`\n",
    "rag_chain = (\n",
    "    {\n",
    "        \"context\": retriever | format_context_docs,\n",
    "        \"question\": RunnablePassthrough(),\n",
    "        # \"context\": RunnableLambda(lambda output: format_context_docs(query=output, retriever=retriever)),\n",
    "    }\n",
    "    | rag_prompt\n",
    "    | llm\n",
    "    | StrOutputParser()\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b77682ba-a6f6-4d4b-9b51-496d666a2af4",
   "metadata": {},
   "source": [
    "### Let's ask a few questions based on following posts\n",
    "\n",
    "- https://huggingface.co/blog/moe\n",
    "- https://huggingface.co/blog/setfit-absa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c2cc9962-c956-4e72-803d-0dbf6461adb0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Question: </span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">What is MoE?</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mQuestion: \u001b[0m\u001b[1;32mWhat is MoE?\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       " <answer>MoE stands for Mixture of Experts. In the context of transformer models, a MoE consists of two main elements: sparse MoE layers used instead of dense feed-forward network (FFN) layers, and a gate network or router that determines which tokens are sent to which expert neural network.</answer>"
      ],
      "text/plain": [
       "\u001b[1m<\u001b[0m\u001b[1;95mIPython.core.display.Markdown\u001b[0m\u001b[39m object\u001b[0m\u001b[1m>\u001b[0m"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">=============================================\n",
       "</pre>\n"
      ],
      "text/plain": [
       "=============================================\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Question: </span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">What is SetFit? How does it work</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mQuestion: \u001b[0m\u001b[1;32mWhat is SetFit? How does it work\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       " <answer>\n",
       "SetFit is a sentence classification framework. It works by taking a training sentence and aspect candidate, concatenating them together to create a training instance with a label, and using these generated instances to train a few-shot domain-specific classifier to extract aspects from text reviews. It has three steps - extracting aspect candidates from text, classifying candidates as aspects or non-aspects, and associating a sentiment polarity to each extracted aspect.\n",
       "</answer>"
      ],
      "text/plain": [
       "\u001b[1m<\u001b[0m\u001b[1;95mIPython.core.display.Markdown\u001b[0m\u001b[39m object\u001b[0m\u001b[1m>\u001b[0m"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">=============================================\n",
       "</pre>\n"
      ],
      "text/plain": [
       "=============================================\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Question: </span><span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold\">What are model sizes of SetFit compared to the others</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mQuestion: \u001b[0m\u001b[1;32mWhat are model sizes of SetFit compared to the others\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"></pre>\n"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       " <answer>\n",
       "We see a clear advantage of SetFitABSA when the number of training instances is low, despite being 2x smaller than T5 and x3 smaller than GPT2-medium.  Even when compared to Llama 2, which is x64 larger, the performance is on par or better.\n",
       "</answer>"
      ],
      "text/plain": [
       "\u001b[1m<\u001b[0m\u001b[1;95mIPython.core.display.Markdown\u001b[0m\u001b[39m object\u001b[0m\u001b[1m>\u001b[0m"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">=============================================\n",
       "</pre>\n"
      ],
      "text/plain": [
       "=============================================\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import Markdown, display\n",
    "\n",
    "queries = [\n",
    "    \"What is MoE?\",\n",
    "    \"What is SetFit? How does it work\",\n",
    "    \"What are model sizes of SetFit compared to the others\",\n",
    "]\n",
    "\n",
    "for q in queries:\n",
    "    print(f\"[b]Question: [b green]{q}[/b green]\")\n",
    "    output = rag_chain.invoke(q)\n",
    "    display(Markdown(output))\n",
    "    print(\"===\" * 15)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "AnthropicGH",
   "language": "python",
   "name": "anthropicgh"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}