L3-Vectorstores_and_Embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "02a0e38d",
   "metadata": {},
   "source": [
    "# Vectorstores and Embeddings\n",
    "Recall the overall workflow for retrieval augmented generation (RAG):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f07dae27",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import openai\n",
    "import sys\n",
    "sys.path.append('../..')\n",
    "\n",
    "from dotenv import load_dotenv, find_dotenv\n",
    "_ = load_dotenv(find_dotenv()) # read local .env file\n",
    "\n",
    "openai.api_key  = os.environ['OPENAI_API_KEY']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "028d8db7",
   "metadata": {},
   "source": [
    "We just discussed Document Loading and Splitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f8595586",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import PyPDFLoader\n",
    "\n",
    "# Load PDF\n",
    "loaders = [\n",
    "    # Duplicate documents on purpose - messy data\n",
    "    PyPDFLoader(\"docs/cs229_lectures/MachineLearning-Lecture01.pdf\"),\n",
    "    PyPDFLoader(\"docs/cs229_lectures/MachineLearning-Lecture01.pdf\"),\n",
    "    PyPDFLoader(\"docs/cs229_lectures/MachineLearning-Lecture02.pdf\"),\n",
    "    PyPDFLoader(\"docs/cs229_lectures/MachineLearning-Lecture03.pdf\")\n",
    "]\n",
    "docs = []\n",
    "for loader in loaders:\n",
    "    docs.extend(loader.load())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "38c47949",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size = 1500,\n",
    "    chunk_overlap = 150\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bffb67f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "splits = text_splitter.split_documents(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c471804f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "209"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7aa38802",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "Let's take our splits and embed them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9baa8eec",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
    "embedding = OpenAIEmbeddings()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "01324ec9",
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence1 = \"i like dogs\"\n",
    "sentence2 = \"i like canines\"\n",
    "sentence3 = \"the weather is ugly outside\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "95d87cd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding1 = embedding.embed_query(sentence1)\n",
    "embedding2 = embedding.embed_query(sentence2)\n",
    "embedding3 = embedding.embed_query(sentence3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f7cbe5a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "824e41dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9631853877103518"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(embedding1, embedding2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "df2bbd39",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7709997651294671"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(embedding1, embedding3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "379dc355",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7596334120325524"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.dot(embedding2, embedding3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03e60563",
   "metadata": {},
   "source": [
    "## Vectorstores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "7d8f8b25",
   "metadata": {},
   "outputs": [],
   "source": [
    "# ! pip install chromadb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "fc5943fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.vectorstores import Chroma"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "79d84ab7",
   "metadata": {},
   "outputs": [],
   "source": [
    "persist_directory = 'docs/chroma/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "95c9d3fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "!rm -rf ./docs/chroma  # remove old database files if any"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "86b1746a",
   "metadata": {},
   "outputs": [],
   "source": [
    "vectordb = Chroma.from_documents(\n",
    "    documents=splits,\n",
    "    embedding=embedding,\n",
    "    persist_directory=persist_directory\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b0145579",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "209\n"
     ]
    }
   ],
   "source": [
    "print(vectordb._collection.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fba605ca",
   "metadata": {},
   "source": [
    "## Similarity Search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "2fc85706",
   "metadata": {},
   "outputs": [],
   "source": [
    "question = \"is there an email i can ask for help\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "15a4de34",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = vectordb.similarity_search(question,k=3) # k is number of documents "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "8e8be2a9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d3329787",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"cs229-qa@cs.stanford.edu. This goes to an acc ount that's read by all the TAs and me. So \\nrather than sending us email individually, if you send email to this account, it will \\nactually let us get back to you maximally quickly with answers to your questions.  \\nIf you're asking questions about homework probl ems, please say in the subject line which \\nassignment and which question the email refers to, since that will also help us to route \\nyour question to the appropriate TA or to me  appropriately and get the response back to \\nyou quickly.  \\nLet's see. Skipping ahead — let's see — for homework, one midterm, one open and term \\nproject. Notice on the honor code. So one thi ng that I think will help you to succeed and \\ndo well in this class and even help you to enjoy this cla ss more is if you form a study \\ngroup.  \\nSo start looking around where you' re sitting now or at the end of class today, mingle a \\nlittle bit and get to know your classmates. I strongly encourage you to form study groups \\nand sort of have a group of people to study with and have a group of your fellow students \\nto talk over these concepts with. You can also  post on the class news group if you want to \\nuse that to try to form a study group.  \\nBut some of the problems sets in this cla ss are reasonably difficult.  People that have \\ntaken the class before may tell you they were very difficult. And just I bet it would be \\nmore fun for you, and you'd probably have a be tter learning experience if you form a\""
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[0].page_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "ed8c256a",
   "metadata": {},
   "outputs": [],
   "source": [
    "vectordb.persist()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9da949f",
   "metadata": {},
   "source": [
    "## Failure modes\n",
    "This seems great, and basic similarity search will get you 80% of the way there very easily.\n",
    "\n",
    "But there are some failure modes that can creep up.\n",
    "\n",
    "Here are some edge cases that can arise - we'll fix them in the next class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "dd63fa7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "question = \"what did they say about matlab?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "f303aa30",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = vectordb.similarity_search(question,k=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3c86465",
   "metadata": {},
   "source": [
    "Notice that we're getting duplicate chunks (because of the duplicate MachineLearning-Lecture01.pdf in the index).\n",
    "\n",
    "Semantic search fetches all similar documents, but does not enforce diversity.\n",
    "\n",
    "docs[0] and docs[1] are indentical."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "b6badb8c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content='those homeworks will be done in either MATLA B or in Octave, which is sort of — I \\nknow some people call it a free ve rsion of MATLAB, which it sort  of is, sort of isn\\'t.  \\nSo I guess for those of you that haven\\'t s een MATLAB before, and I know most of you \\nhave, MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to \\nplot data. And it\\'s sort of an extremely easy to  learn tool to use for implementing a lot of \\nlearning algorithms.  \\nAnd in case some of you want to work on your  own home computer or something if you \\ndon\\'t have a MATLAB license, for the purposes of  this class, there\\'s also — [inaudible] \\nwrite that down [inaudible] MATLAB — there\\' s also a software package called Octave \\nthat you can download for free off the Internet. And it has somewhat fewer features than MATLAB, but it\\'s free, and for the purposes of  this class, it will work for just about \\neverything.  \\nSo actually I, well, so yeah, just a side comment for those of you that haven\\'t seen \\nMATLAB before I guess, once a colleague of mine at a different university, not at \\nStanford, actually teaches another machine l earning course. He\\'s taught it for many years. \\nSo one day, he was in his office, and an old student of his from, lik e, ten years ago came \\ninto his office and he said, \"Oh, professo r, professor, thank you so much for your', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture01.pdf', 'page': 8})"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "b3fbdc99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content='those homeworks will be done in either MATLA B or in Octave, which is sort of — I \\nknow some people call it a free ve rsion of MATLAB, which it sort  of is, sort of isn\\'t.  \\nSo I guess for those of you that haven\\'t s een MATLAB before, and I know most of you \\nhave, MATLAB is I guess part of the programming language that makes it very easy to write codes using matrices, to write code for numerical routines, to move data around, to \\nplot data. And it\\'s sort of an extremely easy to  learn tool to use for implementing a lot of \\nlearning algorithms.  \\nAnd in case some of you want to work on your  own home computer or something if you \\ndon\\'t have a MATLAB license, for the purposes of  this class, there\\'s also — [inaudible] \\nwrite that down [inaudible] MATLAB — there\\' s also a software package called Octave \\nthat you can download for free off the Internet. And it has somewhat fewer features than MATLAB, but it\\'s free, and for the purposes of  this class, it will work for just about \\neverything.  \\nSo actually I, well, so yeah, just a side comment for those of you that haven\\'t seen \\nMATLAB before I guess, once a colleague of mine at a different university, not at \\nStanford, actually teaches another machine l earning course. He\\'s taught it for many years. \\nSo one day, he was in his office, and an old student of his from, lik e, ten years ago came \\ninto his office and he said, \"Oh, professo r, professor, thank you so much for your', metadata={'source': 'docs/cs229_lectures/MachineLearning-Lecture01.pdf', 'page': 8})"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abfb77aa",
   "metadata": {},
   "source": [
    "We can see a new failure mode.\n",
    "\n",
    "The question below asks a question about the third lecture, but includes results from other lectures as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "692d7269",
   "metadata": {},
   "outputs": [],
   "source": [
    "question = \"what did they say about regression in the third lecture?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "83b85399",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = vectordb.similarity_search(question,k=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "19344ee2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 0}\n",
      "{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 14}\n",
      "{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 4}\n",
      "{'source': 'docs/cs229_lectures/MachineLearning-Lecture02.pdf', 'page': 0}\n",
      "{'source': 'docs/cs229_lectures/MachineLearning-Lecture03.pdf', 'page': 6}\n"
     ]
    }
   ],
   "source": [
    "for doc in docs:\n",
    "    print(doc.metadata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "8259e486",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data sets as well. So don’t want to talk about  that. If you’re interested, look up the work \n",
      "of Andrew Moore on KD-trees. He, sort of, fi gured out ways to fit these models much \n",
      "more efficiently. That’s not something I want  to go into today. Okay? Let me move one. \n",
      "Let’s take more questions later.  \n",
      "So, okay. So that’s locally weighted regres sion. Remember the outline I had, I guess, at \n",
      "the beginning of this lecture. What I want to do now is talk about a probabilistic interpretation of linear regres sion, all right? And in partic ular of the – it’ll be this \n",
      "probabilistic interpretati on that let’s us move on to talk  about logistic regression, which \n",
      "will be our first classification algorithm. So le t’s put aside locally weighted regression for \n",
      "now. We’ll just talk about ordinary unwei ghted linear regression. Let’s ask the question \n",
      "of why least squares, right? Of all the thi ngs we could optimize how do we come up with \n",
      "this criteria for minimizing the square of  the area between the predictions of the \n",
      "hypotheses and the values Y predicted. So w hy not minimize the absolute value of the \n",
      "areas or the areas to the power of four or something? What I’m going to do now is \n",
      "present one set of assumptions that will serve to “justify” why we’re minimizing the sum \n",
      "of square zero. Okay?  \n",
      "It turns out that there are many assumptions th at are sufficient to justify why we do least \n",
      "squares and this is just one of them. So ju st because I present one set of assumptions\n"
     ]
    }
   ],
   "source": [
    "print(docs[4].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b433da8e",
   "metadata": {},
   "source": [
    "Approaches discussed in the next lecture can be used to address both!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88940169",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}