gemini_basic_rag.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4756b174-9e08-453e-b52c-6f768cdd7188",
   "metadata": {},
   "source": [
    "# PDF yüklemesinin gerçekleştirilmesi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9ce62089-413b-4cee-8fe6-8d6da4e6e5ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "\n",
    "loader = PyPDFLoader(\"attentionisallyouneed.pdf\")\n",
    "data = loader.load()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "97dd6d4e-b6d0-4b39-ae48-b9474a00e8f8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2ad6e34-ae3f-488f-afa7-9ea2b13d962c",
   "metadata": {},
   "source": [
    "# Veriyi parçalara ayırma (Chunking İşlemi)\n",
    "\n",
    "Tek bir büyük metin parçası yerine, dokümantasyonu daha küçük parçalara böldük?\n",
    "Doğru bilgiyi daha kolay bulmasını sağlayabilmek için bu işlemi gerçekleştirdik.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c098753e-01cb-4a30-b583-c6522d39949a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total 52\n"
     ]
    }
   ],
   "source": [
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000)\n",
    "docs = text_splitter.split_documents(data)\n",
    "\n",
    "print(\"Total\", len(docs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c728a5bc-7a8a-4e30-bd93-81cd51f0f16b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(metadata={'producer': 'pdfTeX-1.40.25', 'creator': 'LaTeX with hyperref', 'creationdate': '2024-04-10T21:11:43+00:00', 'author': '', 'keywords': '', 'moddate': '2024-04-10T21:11:43+00:00', 'ptex.fullbanner': 'This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5', 'subject': '', 'title': '', 'trapped': '/False', 'source': 'attentionisallyouneed.pdf', 'total_pages': 15, 'page': 1, 'page_label': '2'}, page_content='in the distance between positions, linearly for ConvS2S and logarithmically for ByteNet. This makes\\nit more difficult to learn dependencies between distant positions [ 12]. In the Transformer this is\\nreduced to a constant number of operations, albeit at the cost of reduced effective resolution due\\nto averaging attention-weighted positions, an effect we counteract with Multi-Head Attention as\\ndescribed in section 3.2.\\nSelf-attention, sometimes called intra-attention is an attention mechanism relating different positions\\nof a single sequence in order to compute a representation of the sequence. Self-attention has been\\nused successfully in a variety of tasks including reading comprehension, abstractive summarization,\\ntextual entailment and learning task-independent sentence representations [4, 27, 28, 22].\\nEnd-to-end memory networks are based on a recurrent attention mechanism instead of sequence-')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[7]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ba6a261-6d90-4942-8f7e-f7baa22ee525",
   "metadata": {},
   "source": [
    "# Google Generative AI Embeddings'i kullanarak Embedding oluşturma kısmı\n",
    "\n",
    "- Text veriyi sayısal bir veriye çevirmeye çalıştık."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ce7b9397-d50d-4012-a3b6-bae23e817f19",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.05168594419956207,\n",
       " -0.030764883384108543,\n",
       " -0.03062233328819275,\n",
       " -0.02802734263241291,\n",
       " 0.01813093200325966]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain_chroma import Chroma\n",
    "from langchain_google_genai import GoogleGenerativeAIEmbeddings\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "embeddings = GoogleGenerativeAIEmbeddings(model=\"models/embedding-001\")\n",
    "vector = embeddings.embed_query(\"hello, world!\")\n",
    "vector[:5]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7a6a28f-a969-4cd1-9f72-610603e24025",
   "metadata": {},
   "source": [
    "# ChromaDB üzerine kayıtı işlemi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4de61eb8-008b-4c35-938b-b7f31e6ed678",
   "metadata": {},
   "outputs": [],
   "source": [
    "vectorstore = Chroma.from_documents(documents=docs,embedding=embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a0bffe54-f372-4cb8-aa0d-c6360461c583",
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever = vectorstore.as_retriever(search_type=\"similarity\", search_kwargs= {\"k\" : 10})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c7c6cc0c-08ea-42ad-8fab-904a7d5e4a2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "retrieved_docs = retriever.invoke(\"What is encoder?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "34bc943d-68b2-49d5-b29b-dffdb3839a4c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(retrieved_docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "faf549c5-cda0-4831-9b33-5a74ad41cb3f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention Visualizations\n",
      "Input-Input Layer5\n",
      "It\n",
      "is\n",
      "in\n",
      "this\n",
      "spirit\n",
      "that\n",
      "a\n",
      "majority\n",
      "of\n",
      "American\n",
      "governments\n",
      "have\n",
      "passed\n",
      "new\n",
      "laws\n",
      "since\n",
      "2009\n",
      "making\n",
      "the\n",
      "registration\n",
      "or\n",
      "voting\n",
      "process\n",
      "more\n",
      "difficult\n",
      ".\n",
      "<EOS>\n",
      "<pad>\n",
      "<pad>\n",
      "<pad>\n",
      "<pad>\n",
      "<pad>\n",
      "<pad>\n",
      "It\n",
      "is\n",
      "in\n",
      "this\n",
      "spirit\n",
      "that\n",
      "a\n",
      "majority\n",
      "of\n",
      "American\n",
      "governments\n",
      "have\n",
      "passed\n",
      "new\n",
      "laws\n",
      "since\n",
      "2009\n",
      "making\n",
      "the\n",
      "registration\n",
      "or\n",
      "voting\n",
      "process\n",
      "more\n",
      "difficult\n",
      ".\n",
      "<EOS>\n",
      "<pad>\n",
      "<pad>\n",
      "<pad>\n",
      "<pad>\n",
      "<pad>\n",
      "<pad>\n",
      "Figure 3: An example of the attention mechanism following long-distance dependencies in the\n",
      "encoder self-attention in layer 5 of 6. Many of the attention heads attend to a distant dependency of\n",
      "the verb ‘making’, completing the phrase ‘making...more difficult’. Attentions here shown only for\n",
      "the word ‘making’. Different colors represent different heads. Best viewed in color.\n",
      "13\n"
     ]
    }
   ],
   "source": [
    "print(retrieved_docs[5].page_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93925e18-e97c-43ff-974a-2099897f2e89",
   "metadata": {},
   "source": [
    "# Google Gemini API yapısını kullanarak LLM Invoke işlemleri\n",
    "\n",
    "Temperature Değer Aralığı\n",
    "\n",
    "- Düşük Değerler(0.1-0.4):Daha kesin ve daha tutarlı cevaplar vermemizi sağlayabilir. Model daha tahmin edilebilir hale gelir.\n",
    "- Orta Değerler(0.4-0.7): Hem mantıklı hem de yaratıcı cevaplar verebiliyor.\n",
    "- Yüksek Değerler(0.8-1.0) : Daha rastgele ve yaratıcı cevaplar ve bazen de belki tutarsız cevaplar verebilir. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "fb83800c-7f56-403d-b792-0d49f708017d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_google_genai import ChatGoogleGenerativeAI\n",
    "\n",
    "llm = ChatGoogleGenerativeAI(\n",
    "    model=\"gemini-1.5-pro\",\n",
    "    temperature=0.3,\n",
    "    max_tokens=500\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "18f702c9-0cc3-4d30-845c-876e2de74f51",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import create_retrieval_chain\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2b6d30db-1867-4946-a160-0b3e7dfeee6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = (\"You are an assistant for question-answering tasks\"\n",
    "                 \"Use the following pieces of retrieved context to answer\"\n",
    "                 \"If you don't know the answer, say that you don't know\"\n",
    "                 \"Use three sentences maximum and keep the answer corrects.\"\n",
    "                 \"\\n\\n\"\n",
    "                 \"{context}\"\n",
    "                )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c7de0141-012a-4690-acdf-77f1dd76000b",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\",system_prompt),\n",
    "        (\"human\",\"{input}\")\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e461dd48-aaf9-4943-8f26-5d95af059cb1",
   "metadata": {},
   "source": [
    "# Soru-Cevap Üretme Zinciri Oluşturu (LLM+PROMPT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "4bf6a2ec-4224-4fae-9df1-b1e22f8bee75",
   "metadata": {},
   "outputs": [],
   "source": [
    "question_answer_chain = create_stuff_documents_chain(llm,prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a09f9959-4847-4a6b-81d7-e15d488ba0ef",
   "metadata": {},
   "source": [
    "# RAG Zinciri Oluşturma ( RAG + LLM Entegrasyonun Gerçekleşmesi ) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "cf6b2bcc-0dba-461d-b5c3-eebfc6984ebc",
   "metadata": {},
   "outputs": [],
   "source": [
    "rag_chain = create_retrieval_chain(retriever,question_answer_chain)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e94c50f-5c59-49ae-9ff2-c06737c5a59f",
   "metadata": {},
   "source": [
    "# Kullanıcı Sorgusunu Çalıştırarak Cevap Üretme İşlemi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "75c1a1de-6914-4bad-81f5-b8aa2db3354e",
   "metadata": {},
   "outputs": [],
   "source": [
    "response = rag_chain.invoke({\"input\":\"What is encoder?\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "909ffa20-06d3-48c4-99b1-127066337f02",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'input': 'What is encoder?', 'context': [Document(id='9e6909eb-36bb-4829-9fd7-5e33b18907a6', metadata={'author': '', 'creationdate': '2024-04-10T21:11:43+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2024-04-10T21:11:43+00:00', 'page': 1, 'page_label': '2', 'producer': 'pdfTeX-1.40.25', 'ptex.fullbanner': 'This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5', 'source': 'attentionisallyouneed.pdf', 'subject': '', 'title': '', 'total_pages': 15, 'trapped': '/False'}, page_content='Here, the encoder maps an input sequence of symbol representations (x1, ..., xn) to a sequence\\nof continuous representations z = (z1, ..., zn). Given z, the decoder then generates an output\\nsequence (y1, ..., ym) of symbols one element at a time. At each step the model is auto-regressive\\n[10], consuming the previously generated symbols as additional input when generating the next.\\n2'), Document(id='0a931f03-1e4f-47dc-9d39-aef55b39fd33', metadata={'author': '', 'creationdate': '2024-04-10T21:11:43+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2024-04-10T21:11:43+00:00', 'page': 2, 'page_label': '3', 'producer': 'pdfTeX-1.40.25', 'ptex.fullbanner': 'This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5', 'source': 'attentionisallyouneed.pdf', 'subject': '', 'title': '', 'total_pages': 15, 'trapped': '/False'}, page_content='Figure 1: The Transformer - model architecture.\\nThe Transformer follows this overall architecture using stacked self-attention and point-wise, fully\\nconnected layers for both the encoder and decoder, shown in the left and right halves of Figure 1,\\nrespectively.\\n3.1 Encoder and Decoder Stacks\\nEncoder: The encoder is composed of a stack of N = 6 identical layers. Each layer has two\\nsub-layers. The first is a multi-head self-attention mechanism, and the second is a simple, position-\\nwise fully connected feed-forward network. We employ a residual connection [11] around each of\\nthe two sub-layers, followed by layer normalization [ 1]. That is, the output of each sub-layer is\\nLayerNorm(x + Sublayer(x)), where Sublayer(x) is the function implemented by the sub-layer\\nitself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding\\nlayers, produce outputs of dimension dmodel = 512.'), Document(id='0f6b6543-894e-4166-8641-39d70f376706', metadata={'author': '', 'creationdate': '2024-04-10T21:11:43+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2024-04-10T21:11:43+00:00', 'page': 2, 'page_label': '3', 'producer': 'pdfTeX-1.40.25', 'ptex.fullbanner': 'This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5', 'source': 'attentionisallyouneed.pdf', 'subject': '', 'title': '', 'total_pages': 15, 'trapped': '/False'}, page_content='itself. To facilitate these residual connections, all sub-layers in the model, as well as the embedding\\nlayers, produce outputs of dimension dmodel = 512.\\nDecoder: The decoder is also composed of a stack of N = 6identical layers. In addition to the two\\nsub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head\\nattention over the output of the encoder stack. Similar to the encoder, we employ residual connections\\naround each of the sub-layers, followed by layer normalization. We also modify the self-attention\\nsub-layer in the decoder stack to prevent positions from attending to subsequent positions. This\\nmasking, combined with fact that the output embeddings are offset by one position, ensures that the\\npredictions for position i can depend only on the known outputs at positions less than i.\\n3.2 Attention\\nAn attention function can be described as mapping a query and a set of key-value pairs to an output,'), Document(id='2408583f-c75e-4b7d-ab6d-031b23e2b4fb', metadata={'author': '', 'creationdate': '2024-04-10T21:11:43+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2024-04-10T21:11:43+00:00', 'page': 7, 'page_label': '8', 'producer': 'pdfTeX-1.40.25', 'ptex.fullbanner': 'This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5', 'source': 'attentionisallyouneed.pdf', 'subject': '', 'title': '', 'total_pages': 15, 'trapped': '/False'}, page_content='positional encodings in both the encoder and decoder stacks. For the base model, we use a rate of\\nPdrop = 0.1.\\nLabel Smoothing During training, we employed label smoothing of value ϵls = 0.1 [36]. This\\nhurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.\\n6 Results\\n6.1 Machine Translation\\nOn the WMT 2014 English-to-German translation task, the big transformer model (Transformer (big)\\nin Table 2) outperforms the best previously reported models (including ensembles) by more than 2.0\\nBLEU, establishing a new state-of-the-art BLEU score of 28.4. The configuration of this model is\\nlisted in the bottom line of Table 3. Training took 3.5 days on 8 P100 GPUs. Even our base model\\nsurpasses all previously published models and ensembles, at a fraction of the training cost of any of\\nthe competitive models.\\nOn the WMT 2014 English-to-French translation task, our big model achieves a BLEU score of 41.0,'), Document(id='6fb5d9f0-d596-4675-9eb4-555109c09726', metadata={'author': '', 'creationdate': '2024-04-10T21:11:43+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2024-04-10T21:11:43+00:00', 'page': 4, 'page_label': '5', 'producer': 'pdfTeX-1.40.25', 'ptex.fullbanner': 'This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5', 'source': 'attentionisallyouneed.pdf', 'subject': '', 'title': '', 'total_pages': 15, 'trapped': '/False'}, page_content='encoder.\\n• Similarly, self-attention layers in the decoder allow each position in the decoder to attend to\\nall positions in the decoder up to and including that position. We need to prevent leftward\\ninformation flow in the decoder to preserve the auto-regressive property. We implement this\\ninside of scaled dot-product attention by masking out (setting to −∞) all values in the input\\nof the softmax which correspond to illegal connections. See Figure 2.\\n3.3 Position-wise Feed-Forward Networks\\nIn addition to attention sub-layers, each of the layers in our encoder and decoder contains a fully\\nconnected feed-forward network, which is applied to each position separately and identically. This\\nconsists of two linear transformations with a ReLU activation in between.\\nFFN(x) = max(0, xW1 + b1)W2 + b2 (2)\\nWhile the linear transformations are the same across different positions, they use different parameters'), Document(id='0dd37aa6-c64b-4d89-b650-3a9789f60653', metadata={'author': '', 'creationdate': '2024-04-10T21:11:43+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2024-04-10T21:11:43+00:00', 'page': 12, 'page_label': '13', 'producer': 'pdfTeX-1.40.25', 'ptex.fullbanner': 'This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5', 'source': 'attentionisallyouneed.pdf', 'subject': '', 'title': '', 'total_pages': 15, 'trapped': '/False'}, page_content='Attention Visualizations\\nInput-Input Layer5\\nIt\\nis\\nin\\nthis\\nspirit\\nthat\\na\\nmajority\\nof\\nAmerican\\ngovernments\\nhave\\npassed\\nnew\\nlaws\\nsince\\n2009\\nmaking\\nthe\\nregistration\\nor\\nvoting\\nprocess\\nmore\\ndifficult\\n.\\n<EOS>\\n<pad>\\n<pad>\\n<pad>\\n<pad>\\n<pad>\\n<pad>\\nIt\\nis\\nin\\nthis\\nspirit\\nthat\\na\\nmajority\\nof\\nAmerican\\ngovernments\\nhave\\npassed\\nnew\\nlaws\\nsince\\n2009\\nmaking\\nthe\\nregistration\\nor\\nvoting\\nprocess\\nmore\\ndifficult\\n.\\n<EOS>\\n<pad>\\n<pad>\\n<pad>\\n<pad>\\n<pad>\\n<pad>\\nFigure 3: An example of the attention mechanism following long-distance dependencies in the\\nencoder self-attention in layer 5 of 6. Many of the attention heads attend to a distant dependency of\\nthe verb ‘making’, completing the phrase ‘making...more difficult’. Attentions here shown only for\\nthe word ‘making’. Different colors represent different heads. Best viewed in color.\\n13'), Document(id='62981fe3-00a3-4136-9bf9-3fba66b09fc8', metadata={'author': '', 'creationdate': '2024-04-10T21:11:43+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2024-04-10T21:11:43+00:00', 'page': 1, 'page_label': '2', 'producer': 'pdfTeX-1.40.25', 'ptex.fullbanner': 'This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5', 'source': 'attentionisallyouneed.pdf', 'subject': '', 'title': '', 'total_pages': 15, 'trapped': '/False'}, page_content='textual entailment and learning task-independent sentence representations [4, 27, 28, 22].\\nEnd-to-end memory networks are based on a recurrent attention mechanism instead of sequence-\\naligned recurrence and have been shown to perform well on simple-language question answering and\\nlanguage modeling tasks [34].\\nTo the best of our knowledge, however, the Transformer is the first transduction model relying\\nentirely on self-attention to compute representations of its input and output without using sequence-\\naligned RNNs or convolution. In the following sections, we will describe the Transformer, motivate\\nself-attention and discuss its advantages over models such as [17, 18] and [9].\\n3 Model Architecture\\nMost competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35].\\nHere, the encoder maps an input sequence of symbol representations (x1, ..., xn) to a sequence\\nof continuous representations z = (z1, ..., zn). Given z, the decoder then generates an output'), Document(id='d4fc1af7-e699-4b9a-9c9f-c4b0c240ff6c', metadata={'author': '', 'creationdate': '2024-04-10T21:11:43+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2024-04-10T21:11:43+00:00', 'page': 4, 'page_label': '5', 'producer': 'pdfTeX-1.40.25', 'ptex.fullbanner': 'This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5', 'source': 'attentionisallyouneed.pdf', 'subject': '', 'title': '', 'total_pages': 15, 'trapped': '/False'}, page_content='FFN(x) = max(0, xW1 + b1)W2 + b2 (2)\\nWhile the linear transformations are the same across different positions, they use different parameters\\nfrom layer to layer. Another way of describing this is as two convolutions with kernel size 1.\\nThe dimensionality of input and output is dmodel = 512, and the inner-layer has dimensionality\\ndff = 2048.\\n3.4 Embeddings and Softmax\\nSimilarly to other sequence transduction models, we use learned embeddings to convert the input\\ntokens and output tokens to vectors of dimension dmodel. We also use the usual learned linear transfor-\\nmation and softmax function to convert the decoder output to predicted next-token probabilities. In\\nour model, we share the same weight matrix between the two embedding layers and the pre-softmax\\nlinear transformation, similar to [30]. In the embedding layers, we multiply those weights by √dmodel.\\n5'), Document(id='451c83ac-d3a9-43be-9c1e-57bf3edb5036', metadata={'author': '', 'creationdate': '2024-04-10T21:11:43+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2024-04-10T21:11:43+00:00', 'page': 5, 'page_label': '6', 'producer': 'pdfTeX-1.40.25', 'ptex.fullbanner': 'This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5', 'source': 'attentionisallyouneed.pdf', 'subject': '', 'title': '', 'total_pages': 15, 'trapped': '/False'}, page_content='tokens in the sequence. To this end, we add \"positional encodings\" to the input embeddings at the\\nbottoms of the encoder and decoder stacks. The positional encodings have the same dimension dmodel\\nas the embeddings, so that the two can be summed. There are many choices of positional encodings,\\nlearned and fixed [9].\\nIn this work, we use sine and cosine functions of different frequencies:\\nP E(pos,2i) = sin(pos/100002i/dmodel )\\nP E(pos,2i+1) = cos(pos/100002i/dmodel )\\nwhere pos is the position and i is the dimension. That is, each dimension of the positional encoding\\ncorresponds to a sinusoid. The wavelengths form a geometric progression from 2π to 10000 · 2π. We\\nchose this function because we hypothesized it would allow the model to easily learn to attend by\\nrelative positions, since for any fixed offset k, P Epos+k can be represented as a linear function of\\nP Epos.\\nWe also experimented with using learned positional embeddings [9] instead, and found that the two'), Document(id='81f44aa2-aa8b-43a3-a412-8ec77f718893', metadata={'author': '', 'creationdate': '2024-04-10T21:11:43+00:00', 'creator': 'LaTeX with hyperref', 'keywords': '', 'moddate': '2024-04-10T21:11:43+00:00', 'page': 8, 'page_label': '9', 'producer': 'pdfTeX-1.40.25', 'ptex.fullbanner': 'This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5', 'source': 'attentionisallyouneed.pdf', 'subject': '', 'title': '', 'total_pages': 15, 'trapped': '/False'}, page_content='bigger models are better, and dropout is very helpful in avoiding over-fitting. In row (E) we replace our\\nsinusoidal positional encoding with learned positional embeddings [9], and observe nearly identical\\nresults to the base model.\\n6.3 English Constituency Parsing\\nTo evaluate if the Transformer can generalize to other tasks we performed experiments on English\\nconstituency parsing. This task presents specific challenges: the output is subject to strong structural\\nconstraints and is significantly longer than the input. Furthermore, RNN sequence-to-sequence\\nmodels have not been able to attain state-of-the-art results in small-data regimes [37].\\nWe trained a 4-layer transformer with dmodel = 1024on the Wall Street Journal (WSJ) portion of the\\nPenn Treebank [25], about 40K training sentences. We also trained it in a semi-supervised setting,\\nusing the larger high-confidence and BerkleyParser corpora from with approximately 17M sentences')], 'answer': \"The encoder maps an input sequence of symbol representations (x1, ..., xn) to a sequence of continuous representations z = (z1, ..., zn).  Given z, the decoder then generates an output sequence (y1, ..., ym) of symbols.  The Transformer's encoder is composed of a stack of N = 6 identical layers, each with two sub-layers: a multi-head self-attention mechanism and a position-wise fully connected feed-forward network.\"}\n"
     ]
    }
   ],
   "source": [
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "5eb7402e-4b53-40f8-8a41-46577189a654",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The encoder maps an input sequence of symbol representations (x1, ..., xn) to a sequence of continuous representations z = (z1, ..., zn).  Given z, the decoder then generates an output sequence (y1, ..., ym) of symbols.  The Transformer's encoder is composed of a stack of N = 6 identical layers, each with two sub-layers: a multi-head self-attention mechanism and a position-wise fully connected feed-forward network.\n"
     ]
    }
   ],
   "source": [
    "print(response[\"answer\"]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cd1366b-86c0-4c72-86c7-9367825e68c6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef5d72bc-d5f2-486d-8d4a-b45e1c96d116",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72b412b4-89e4-45a4-84a7-445f8b56eb6b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (env_langchain)",
   "language": "python",
   "name": "env_langchain"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}