all_rag_techniques/HyDe_Hypothetical_Document_Embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypothetical Document Embedding (HyDE) in Document Retrieval\n",
    "\n",
    "## Overview\n",
    "\n",
    "This code implements a Hypothetical Document Embedding (HyDE) system for document retrieval. HyDE is an innovative approach that transforms query questions into hypothetical documents containing the answer, aiming to bridge the gap between query and document distributions in vector space.\n",
    "\n",
    "## Motivation\n",
    "\n",
    "Traditional retrieval methods often struggle with the semantic gap between short queries and longer, more detailed documents. HyDE addresses this by expanding the query into a full hypothetical document, potentially improving retrieval relevance by making the query representation more similar to the document representations in the vector space.\n",
    "\n",
    "## Key Components\n",
    "\n",
    "1. PDF processing and text chunking\n",
    "2. Vector store creation using FAISS and OpenAI embeddings\n",
    "3. Language model for generating hypothetical documents\n",
    "4. Custom HyDERetriever class implementing the HyDE technique\n",
    "\n",
    "## Method Details\n",
    "\n",
    "### Document Preprocessing and Vector Store Creation\n",
    "\n",
    "1. The PDF is processed and split into chunks.\n",
    "2. A FAISS vector store is created using OpenAI embeddings for efficient similarity search.\n",
    "\n",
    "### Hypothetical Document Generation\n",
    "\n",
    "1. A language model (GPT-4) is used to generate a hypothetical document that answers the given query.\n",
    "2. The generation is guided by a prompt template that ensures the hypothetical document is detailed and matches the chunk size used in the vector store.\n",
    "\n",
    "### Retrieval Process\n",
    "\n",
    "The `HyDERetriever` class implements the following steps:\n",
    "\n",
    "1. Generate a hypothetical document from the query using the language model.\n",
    "2. Use the hypothetical document as the search query in the vector store.\n",
    "3. Retrieve the most similar documents to this hypothetical document.\n",
    "\n",
    "## Key Features\n",
    "\n",
    "1. Query Expansion: Transforms short queries into detailed hypothetical documents.\n",
    "2. Flexible Configuration: Allows adjustment of chunk size, overlap, and number of retrieved documents.\n",
    "3. Integration with OpenAI Models: Uses GPT-4 for hypothetical document generation and OpenAI embeddings for vector representation.\n",
    "\n",
    "## Benefits of this Approach\n",
    "\n",
    "1. Improved Relevance: By expanding queries into full documents, HyDE can potentially capture more nuanced and relevant matches.\n",
    "2. Handling Complex Queries: Particularly useful for complex or multi-faceted queries that might be difficult to match directly.\n",
    "3. Adaptability: The hypothetical document generation can adapt to different types of queries and document domains.\n",
    "4. Potential for Better Context Understanding: The expanded query might better capture the context and intent behind the original question.\n",
    "\n",
    "## Implementation Details\n",
    "\n",
    "1. Uses OpenAI's ChatGPT model for hypothetical document generation.\n",
    "2. Employs FAISS for efficient similarity search in the vector space.\n",
    "3. Allows for easy visualization of both the hypothetical document and retrieved results.\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "Hypothetical Document Embedding (HyDE) represents an innovative approach to document retrieval, addressing the semantic gap between queries and documents. By leveraging advanced language models to expand queries into hypothetical documents, HyDE has the potential to significantly improve retrieval relevance, especially for complex or nuanced queries. This technique could be particularly valuable in domains where understanding query intent and context is crucial, such as legal research, academic literature review, or advanced information retrieval systems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: center;\">\n",
    "\n",
    "<img src=\"../images/HyDe.svg\" alt=\"HyDe\" style=\"width:40%; height:auto;\">\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: center;\">\n",
    "\n",
    "<img src=\"../images/hyde-advantages.svg\" alt=\"HyDe\" style=\"width:100%; height:auto;\">\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "\n",
    "sys.path.append(os.path.abspath(os.path.join(os.getcwd(), '..'))) # Add the parent directory to the path sicnce we work with notebooks\n",
    "from helper_functions import *\n",
    "from evaluation.evalute_rag import *\n",
    "\n",
    "# Load environment variables from a .env file\n",
    "load_dotenv()\n",
    "\n",
    "# Set the OpenAI API key environment variable\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_API_KEY')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define document(s) path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = \"../data/Understanding_Climate_Change.pdf\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define the HyDe retriever class - creating vector store, generating hypothetical document, and retrieving"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class HyDERetriever:\n",
    "    def __init__(self, files_path, chunk_size=500, chunk_overlap=100):\n",
    "        self.llm = ChatOpenAI(temperature=0, model_name=\"gpt-4o-mini\", max_tokens=4000)\n",
    "\n",
    "        self.embeddings = OpenAIEmbeddings()\n",
    "        self.chunk_size = chunk_size\n",
    "        self.chunk_overlap = chunk_overlap\n",
    "        self.vectorstore = encode_pdf(files_path, chunk_size=self.chunk_size, chunk_overlap=self.chunk_overlap)\n",
    "    \n",
    "        \n",
    "        self.hyde_prompt = PromptTemplate(\n",
    "            input_variables=[\"query\", \"chunk_size\"],\n",
    "            template=\"\"\"Given the question '{query}', generate a hypothetical document that directly answers this question. The document should be detailed and in-depth.\n",
    "            the document size has be exactly {chunk_size} characters.\"\"\",\n",
    "        )\n",
    "        self.hyde_chain = self.hyde_prompt | self.llm\n",
    "\n",
    "    def generate_hypothetical_document(self, query):\n",
    "        input_variables = {\"query\": query, \"chunk_size\": self.chunk_size}\n",
    "        return self.hyde_chain.invoke(input_variables).content\n",
    "\n",
    "    def retrieve(self, query, k=3):\n",
    "        hypothetical_doc = self.generate_hypothetical_document(query)\n",
    "        similar_docs = self.vectorstore.similarity_search(hypothetical_doc, k=k)\n",
    "        return similar_docs, hypothetical_doc\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a HyDe retriever instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever = HyDERetriever(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Demonstrate on a use case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_query = \"What is the main cause of climate change?\"\n",
    "results, hypothetical_doc = retriever.retrieve(test_query)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the hypothetical document and the retrieved documnets "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_content = [doc.page_content for doc in results]\n",
    "\n",
    "print(\"hypothetical_doc:\\n\")\n",
    "print(text_wrap(hypothetical_doc)+\"\\n\")\n",
    "show_context(docs_content)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}