pytorch/natural_language_processing/represent_words_with_embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Embeddings\n",
    "\n",
    "In the [Representing text as tensors](https://www.kaggle.com/code/aisuko/representing-text-as-tensors) notebook, we operated on high-dimensional bag-of-words vectors with the length of `vocab_size`, and we were explicity converting from low-dimentional positional representation vectors into sparse one-hot representations. Here is an example for embedding classifier:\n",
    "\n",
    "\n",
    "The goal of using word embeddings and reducing the dimensionality are:\n",
    "* Finding the meaning of words based on their word approximation to other words. This is done by taken two word vectors and analyzing how often the words in the vectors are used together. The higher the frequency, the more you can find a correlation and relationship between the words.\n",
    "* This process of training the word embedding to find word approcimations between words in a given dimension is how we reduce the word representation of low-dimensions.\n",
    "* Embedding vectors serve as numeric representations of words and are used as input to other machine learning network layers.\n",
    "* The embedding vector becomes the stored lookup table for words in the vocabulary.\n",
    "\n",
    "Here we are continue exploring the **News AG** dataset. To begin. let's load the data and get some definitions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://github.com/pytorch/data/issues/1093\n",
    "pip install portalocker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu118"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torchtext\n",
    "from torchtext.data import get_tokenizer\n",
    "import numpy as np\n",
    "from torchinfo import summary\n",
    "import collections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check the paltform, Apple Silicon or Linux\n",
    "import os, platform\n",
    "\n",
    "torch_device=\"cpu\"\n",
    "\n",
    "if 'kaggle' in os.environ.get('KAGGLE_URL_BASE','localhost'):\n",
    "    torch_device = 'cuda'\n",
    "else:\n",
    "    torch_device = 'mps' if platform.system() == 'Darwin' else 'cpu'\n",
    "\n",
    "torch_device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with variable sequence size\n",
    "\n",
    "When working with words, you are going to have text sequences or sentences that are of different lengths. This can be problematic in training the word embeddings neural network. For consistency in the word embedding and improve training performance, we would have to apply some padding. This can be done using the `torch.nn.functional.pad` on a tokenized dataset. It adds zero values to the empty indices at the end of the vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading dataset\n",
    "train_dataset, test_dataset = torchtext.datasets.AG_NEWS(root='./data')\n",
    "train_dataset, test_dataset = list(train_dataset), list(test_dataset)\n",
    "classes = ['World','Sports','Business','Sci/Tech']\n",
    "\n",
    "tokenizer = torchtext.data.utils.get_tokenizer('basic_english')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a vocabulary of all tokens\n",
    "\n",
    "We first build the dictionary using the `Counter` object, and then create `Vocab` object that would help us deal with vectorization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Building vocab\n",
    "def build_vocab(train_dataset,ngrams=1,min_freq=1):\n",
    "    counter = collections.Counter()\n",
    "    for (label, line) in train_dataset:\n",
    "        counter.update(torchtext.data.utils.ngrams_iterator(tokenizer(line), ngrams=ngrams))\n",
    "    vocab = torchtext.vocab.vocab(counter, min_freq=min_freq)\n",
    "    return vocab\n",
    "\n",
    "vocab = build_vocab(train_dataset,ngrams=1,min_freq=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def encode(x, voc=None,tokenizer=tokenizer):\n",
    "    v =vocab if not voc else voc\n",
    "    return [v.get_stoi()[s] for s in tokenizer(x)]\n",
    "\n",
    "def padify(b):\n",
    "    # b is the list of tuples of length batch_size\n",
    "    # - first element of a tuple = label\n",
    "    # - second = feature (text, sequence)\n",
    "    # build vectorized sequence\n",
    "    v = [encode(x[1]) for x in b]\n",
    "    # first, compute max length of a sequnce in this minibatch\n",
    "    l = max(map(len,v))\n",
    "    return (\n",
    "        # tuple of two tensors - labels and features\n",
    "        torch.LongTensor([t[0]-1 for t in b]),\n",
    "        torch.stack([torch.nn.functional.pad(torch.tensor(t),(0,l-len(t)), mode='constant', value=0) for t in v])\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the first 2 sentences as example to view the text length difference and effects of padding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_sentence = train_dataset[0][1]\n",
    "second_sentence = train_dataset[1][1]\n",
    "\n",
    "f_tokens = encode(first_sentence)\n",
    "s_tokens = encode(second_sentence)\n",
    "\n",
    "print(f'First sentence in dataset:\\n{first_sentence}')\n",
    "print('Length:', len(train_dataset[0][1]), '\\n')\n",
    "print(f'\\nSecond sentence in dataset:\\n{second_sentence}')\n",
    "print('Length:', len(train_dataset[1][1]), '\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use the text sequence from the news article headlines in our dataset to change into a tokenize vector. As you will see, the text sequence have different lengths. We will apply padding so all the text sequence wil have a fixed length. This approach is used when you have a large set of text sequences in your dataset.\n",
    "* The length of the 1st and 2nd sentences displayed have difference lengths.\n",
    "* The max length of the dataset tensors is the length of the longest sentence length in the entire dataset.\n",
    "* The zeros are added to the empty indexes in the tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size =len(vocab)\n",
    "labels, features = padify(train_dataset)\n",
    "print(f'features:{features}')\n",
    "\n",
    "print(f'\\nlength of first sentence: {len(f_tokens)}')\n",
    "print(f'length of second sentence: {len(s_tokens)}')\n",
    "print(f'size of features: {features.size()}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is embedding?\n",
    "\n",
    "The idea of `embedding` is the process of mapping words into vectors, which reflects the `semantic meaning of a word`. The length of its vectors are the embedding dimensions size. We will later discuss how to build meningful word embeddings, but for now let's just think of embeddings as a way to lower dimensionality of a word vector.\n",
    "\n",
    "So, embedding layer would take a word as an input, and produce an output vector of specified `embedding_size`. In a sense, it is very similar to `Linear` layer, but instead of taking one-hot encoded vector, it will be able to take a word number as an input.\n",
    "\n",
    "By using embedding layer as a first layer in our network, we can switch from bag-or-words to `embedding bag` model, where we first convert each word in our text into corresponding embedding, and then compute some aggregate function over all those embeddings, such as `sum`, `avergae` or `max`.\n",
    "\n",
    "<figure><img src=\"https://hostux.social/system/media_attachments/files/110/750/619/792/006/183/original/94496909c3a93b6e.png\" alt=\"\" width=\"1000\"><figcaption><p>Source from Microsoft Learning </a> </p></figcaption></figure>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time our classifier neural network will start with an embedding layer, then aggregation layer, and a linear classifier on top of it:\n",
    "* `vocab_size` are the size of the total number of words we have in our vocabulary\n",
    "* `embed_dim` are the length of the word dimensions that show relationship between words passed as in the network\n",
    "* `num_class` are the number of new categories we are trying to classify (e.g. Word, Sports, Business, Sci/Tech)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EmbedClassifier(torch.nn.Module):\n",
    "    def __init__(self, vocab_size, embed_dim, num_class):\n",
    "        super().__init__()\n",
    "        self.embedding = torch.nn.EmbeddingBag(vocab_size, embed_dim, sparse=True)\n",
    "        self.fc = torch.nn.Linear(embed_dim, num_class)\n",
    "    \n",
    "    def forward(self, x):\n",
    "        x = self.embedding(x)\n",
    "        x = torch.mean(x, dim=1) # torch.mean() computes the mean of all elements in a tensor, it is a reduction operation\n",
    "        return self.fc(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training embedding classifer\n",
    "\n",
    "Now we'll define our training dataloader and use the `collate_fn` to apply the padify function to the datasets as they loaded in each batch. As a result, the training dataset will be padded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, shuffle=True, collate_fn=padify)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can train the model using rh training function to run the embedding network. The training output serves as a vector lookup store based on the unique index tokens from the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epoch(net, dataloader, lr=0.01, optimizer=None, loss_fn=torch.nn.CrossEntropyLoss(), epoch_size=None, report_freq=200):\n",
    "    optimizer = optimizer or torch.optim.Adam(net.parameters(), lr=lr)\n",
    "    net.train()\n",
    "    total_loss, acc, count,i = 0,0,0,0\n",
    "    for labels, features in dataloader:\n",
    "        labels, features = labels.to(torch_device), features.to(torch_device)\n",
    "        optimizer.zero_grad()\n",
    "        output = net(features)\n",
    "        loss = loss_fn(output, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        total_loss += loss\n",
    "        _, predicted = torch.max(output, 1)\n",
    "        acc += (predicted == labels).sum()\n",
    "        count += len(labels)\n",
    "        i+=1\n",
    "        if i%report_freq == 0:\n",
    "            print(f'iteration {count}, loss {total_loss.item()/count}, accuracy {acc.item()/count}') # item() is used to get the value of a tensor\n",
    "        if epoch_size  and count >= epoch_size:\n",
    "            break\n",
    "    \n",
    "    return total_loss.item()/count, acc.item()/count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = EmbedClassifier(vocab_size, embed_dim=32, num_class=len(classes)).to(torch_device)\n",
    "\n",
    "train_epoch(net, train_loader,lr=1, epoch_size=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">Note: We are only training for 25k records here (less than one full epoch) for the sake of time, but you can continue training, write a function to train for several epochs, and experiement with learning rate parameter to achieve higher accuracy. You should be able to go to the accuracy of about 90%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EmbeddingBag Layer and Variable_length Sequnce Representation\n",
    "\n",
    "In the previous architecture, we need to pad all sequences to the same length in order to fit them into a minibatch. This is not the most efficient way to represent variable length sequynces - another approach would be to use `offset` vector, which would hold offsets to all sequences stored in one large vector.\n",
    "\n",
    "<figure><img src=\"https://hostux.social/system/media_attachments/files/110/751/844/099/559/089/original/c67bdb8c6e8afa7f.png\" alt=\"\" width=\"1000\"><figcaption><p>Source from Microsoft Learning </a> </p></figcaption></figure>\n",
    "\n",
    ">Note: On the picture above, we show a sequence of characters, but in our example we are working with sequences of words. However, the general principle of representing sequences with offset vector remains the same.\n",
    "\n",
    "To work with offset representation, we use PyTorch's `EmbeddingBag` layer. It is similar to `Embedding`, but it takes content vector and offset vector as input, and it also includes averaging layer, which can be `mean`, `sum` or `max`. Here is modified network that uses `EmbeddingBag` layer:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class EmbedClassifier(torch.nn.Module):\n",
    "    def __init__(self, vocab_size, embed_dim, num_class):\n",
    "        super().__init__()\n",
    "        self.embedding = torch.nn.EmbeddingBag(vocab_size, embed_dim, sparse=True) # sparse=True is used to save memory\n",
    "        self.fc = torch.nn.Linear(embed_dim, num_class)\n",
    "    \n",
    "    def forward(self, text, off):\n",
    "        x = self.embedding(text, off)\n",
    "        return self.fc(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To prepare the dataset for training, we need to provide a conversition funciton that will prepare the offset vector:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def offsetify(b):\n",
    "    # first, compute data tensot from all sequences\n",
    "    x = [torch.tensor(encode(t[1])) for t in b]\n",
    "    # now, compute the offsets by accumulating the tensor of sequence lengths\n",
    "    o = [0] + [len(t) for t in x]\n",
    "    o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
    "    return (\n",
    "        torch.LongTensor([t[0]-1 for t in b]), # labels\n",
    "        torch.cat(x), # text\n",
    "        o\n",
    "    )\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, shuffle=True, collate_fn=offsetify)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The offset vector is calculated by first combining the sentences indices into one tensor sequence, then extracting the staring index location of each sentence in the sequence. For example:\n",
    "* The length of the first sentence in our training dataset is 29. Meaning the first index of the offset will 0\n",
    "* The length of the second sentence in the dataset is 42. Meaning the second index of the offset of will be 29, where the first sentence ended.\n",
    "* The third index of the offset will be 29 +42=71, where the 2nd sentence ended."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels, features, offset = offsetify(train_dataset)\n",
    "print(f'offset:{offset}')\n",
    "print(f'\\nlength of first sentence: {len(f_tokens)}')\n",
    "print(f'length of second sentence: {len(s_tokens)}')\n",
    "print(f'size of data vector: {features.size()}')\n",
    "print(f'size of offset vector: {offset.size()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Note: that unlike in all previous example, our network now accepts two parameters: data vector and offset vector, which are of different sizes. Similarly, our data loader also provides us with 3 values instead of 2: both text and offset vectors are provided as features. Therefore, we need to slightly adjust our training function to take care of that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = EmbedClassifier(vocab_size,32,len(classes)).to(torch_device)\n",
    "\n",
    "def train_epoch_emb(net, dataloader, lr=0.01, optimizer=None, loss_fn=torch.nn.CrossEntropyLoss(),epoch_size=None, report_freq=200):\n",
    "    optimizer  = optimizer or torch.optim.Adam(net.parameters(), lr=lr)\n",
    "    loss_fn = loss_fn.to(torch_device)\n",
    "    net.train()\n",
    "    total_loss, acc, count, i = 0,0,0,0\n",
    "    for labels, text,off in dataloader:\n",
    "        optimizer.zero_grad()\n",
    "        labels, text, off = labels.to(torch_device), text.to(torch_device), off.to(torch_device)\n",
    "        output = net(text, off)\n",
    "        loss = loss_fn(output, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        total_loss += loss\n",
    "        _, predicted = torch.max(output,1) # torch.max() is used to get the max value of a tensor\n",
    "        acc += (predicted == labels).sum()\n",
    "        count += len(labels)\n",
    "        i += 1\n",
    "        if i%report_freq == 0:\n",
    "            print(f'iteration {count}, loss {total_loss.item()/count}, accuracy {acc.item()/count}')\n",
    "        if epoch_size and count >= epoch_size:\n",
    "            break\n",
    "    return total_loss.item()/count, acc.item()/count\n",
    "\n",
    "train_epoch_emb(net, train_loader, lr=4, epoch_size=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Semantic Embeddings: Word2Vec\n",
    "\n",
    "In our previous example, the model embedding layer learnt to map words to vector representaion, however, this representaion did not have much semantical meaning. It would be nice to learn such vector representaion, that similar words ot symonims would correspond to vectors that are close to each other in terms of some vector distance (eg. euclidian distance).\n",
    "\n",
    "To do that, we need to pre-train our embedding model on a large collection of text of in a specific way. One of the first wats to train semantic embeddings is called **Word2Vec**. It helps `map the probability of a word`, based on the contexts from texts in the sequence. It is based on two main architectures that are used to produce a distributed representation of words:\n",
    "* **Continuous Bag-of-Words** (CBOW) - in this architecture, we train the model to predict a word from surrounding context. Given the ngram ($W_{-2},W_{-1},W_{0},W_{1},W_{2}$), the goal of the model is to predict $W_{0}$ from ($W_{-2},W_{-1},W_{1},W_{2}$). For example, \"I like my hot dog on a _\". Here the predicted word would be \"bun\".\n",
    "* **Continuous skip-gram** is opposite to CBoW. The model uses surrounding window of context words to predict the current word. For example: you can predict `dog` to be more associated with the word `veterinary`.\n",
    "\n",
    "CBoW is faster, while skip-gram is slower, bu does a better job of representing infrequent words.\n",
    "\n",
    "<figure><img src=\"https://hostux.social/system/media_attachments/files/110/755/104/068/247/744/original/157b705602d3e7c4.png\" alt=\"\" width=\"1000\"><figcaption><p>Source from Microsoft Learning </a> </p></figcaption></figure>\n",
    "\n",
    "Both CBOW and Skip-Grams are \"predictive\" embeddings, in that they only take local contexts into account. Word2Vec does not take advantage of global context.\n",
    "* **GloVe**(Global Vectors) - derives the relationship between words. It determines how often a particular word-pair occurs together in a text by leveraging the idea of co-occurence matrix and uses neural methods to decompose co-occurence matrix into more expressive and non-linear word vectors.\n",
    "* **FastText** - builds on Word2Vec by learning vector representations are then averaged into one vector at each training step. While this adds a lot of additional computation ro pre-training, it enables word embeddings to encode sub-word information.\n",
    "* **Gensim**(Generate Similar) - is an open source NLP Python library that provides a unified interface to build word vectors,  corpus, perform topic identification,and other NLP tasks.\n",
    "\n",
    "FastText and Glove are other word embedding techniques that predict the probably of words appearing together.\n",
    "\n",
    "In our Word2Vec example, we'll using pre-trained semantic embeddings, but it is interesting to see how those embeddings can be trained using either FastText, CBoW or Skip-gram architectures. This exercise goes beyond this module, but those interested can reference Word Embeddings tutorials on PyTorch's website.\n",
    "\n",
    "## Genim\n",
    "\n",
    "The `gensim` framework can be used with PyTorch to train most commonly used to embeddings in a few lines of code. To experiment with word2vec embedding pre-trained on Google News dataset, we can use the `gensim` library. Below we find the words that are most similar to `neural`\n",
    "\n",
    ">Note: When you first create word vectors, downloading them can take some time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim.downloader as api\n",
    "w2v = api.load('word2vec-google-news-300')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at words that arte similar to `dog`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for w,p in w2v.most_similar('dog'):\n",
    "    print(f'{w} {p}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also extract vector embeddings from the word, to be used in training classification model(we'll only show first 20 components of the vector for clarity):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w2v.word_vec('play')[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The great thing about semantical embeddings is that you can manipulate vector encoding to change the semantics. For example, we can ask to find a word, whose vectot representation would be as close as possible to words `king` and `woman`, asn as far away from the word `man`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w2v.most_similar(positive=['king','woman'],negative=['man'])[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Pre-trained Embeddings in PyTorch\n",
    "\n",
    "We can modify the example above to pre-populate the matrix in our embedding layer with semantical embeddings, such as Word2Vec. We need to take into account that vocabularies of pre-trained embedding are an addition to the existing text corpus that we already have so they will likely not match. As a result, we will initialize weights for the missing words with random values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "embed_size =len(w2v.get_vector('hello'))\n",
    "print(f'Embedding size: {embed_size}')\n",
    "\n",
    "net = EmbedClassifier(vocab_size, embed_size, len(classes)).to(torch_device)\n",
    "\n",
    "print('Populating matrix, this will take some time...', end='')\n",
    "found,not_found = 0,0\n",
    "for i,w in enumerate(vocab.get_itos()):\n",
    "    try:\n",
    "        net.embedding.weight[i].data = torch.tensor(w2v.get_vector(w))\n",
    "        found += 1\n",
    "    except:\n",
    "        net.embedding.weight[i].data = torch.normal(0.0,1.0,(embed_size,))\n",
    "        not_found += 1\n",
    "\n",
    "print(f'Done, found {found} words, {not_found} words missing')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train our model. Note that the time it takes to train the model is significantly larger than in the provious example, due to larger embedding layer size, and thus much higher number of parameters. Also, because of this, we may need to train our model on more examples if we want to avoid overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_epoch_emb(net, train_loader, lr=4, epoch_size=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our csse, we do not see a huge increase in accuracy, which is likely due to thr quite different vocabularies. To overcome the problem of different vocabularies, we can use one of the followinf solutions:\n",
    "\n",
    "* Re-train word2vec model on our vocabulary\n",
    "* Load our dataset with the vocabulary form the pre-trained word2vec model. Vocabulary used to load the dataset can be specified during loading.\n",
    "\n",
    "The latter approach seems easier, especially because PyTorch `torchtext` framework contains built-in support for embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Glove Embeddings\n",
    "\n",
    "To load our dataset wiht the covabulary from a pre-trained word2vec model,we use Glove embeddings. We'll start by instantiating Glove-based vocabulary in the following manner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = torchtext.vocab.GloVe(name='6B', dim=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loaded vocabulary has the following basic operations:\n",
    "* `vocab.get_stoi()` dictionary allows us to convert word into its dictionary index\n",
    "* `vocab.get_itos()` does the opposite - converts number into word\n",
    "* `vocab.vectors` is the array of embedding vectors, so to get the embedding of a word we nee to use `vocab.vectors[vocab.get_stoi(s)]`\n",
    "\n",
    "Here is the example of manipulating embeddings to demonstrate the equation `kind-man+woman=queen`(the coefficient was tweak a bit to make it work):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the vector corresponding to kind-man+woman\n",
    "qvec = vocab.vectors[vocab.stoi['king']]-vocab.vectors[vocab.stoi['man']]+1.3*vocab.vectors[vocab.stoi['woman']]\n",
    "# find the index of the closest embedding vector\n",
    "d = torch.sum((vocab.vectors-qvec)**2,dim=1)\n",
    "min_idx = torch.argmin(d)\n",
    "# find the corresponding word\n",
    "vocab.itos[min_idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train the classsifier using those embeedings, we first need to encode our dataset Glove vocabulary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def offsetify(b):\n",
    "    # first, compute data tensot from all sequences\n",
    "    x = [torch.tensor(encode(t[1], voc=vocab)) for t in b] # pass the instance of vocab to encode function\n",
    "    # now, compute the offsets by accumulating the tensor of sequence lengths\n",
    "    o = [0] + [len(t) for t in x]\n",
    "    o = torch.tensor(o[:-1]).cumsum(dim=0)\n",
    "    return (\n",
    "        torch.LongTensor([t[0]-1 for t in b]), # labels\n",
    "        torch.cat(x), # text\n",
    "        o\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen above, all vector embeddings are stored in `vocab.vectors` matrix. It makes it super-easy to load those weights into weights of embedding layer using simple copying:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = EmbedClassifier(len(vocab), len(vocab.vectors[0]), len(classes))\n",
    "net.embedding.weight.data = vocab.vectors\n",
    "net = net.to(torch_device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train our model and see if we get better results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, shuffle=True, collate_fn=offsetify)\n",
    "train_epoch_emb(net, train_loader, lr=4, epoch_size=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the reasons we are not seeing a significant increase in accuracy is due to the fact that some words from our dataset are missing in the pre-trained Glove vocabulary, and thus they are essentially ignored. To overcome this fact, we can train our own embeddings on our dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contextual Embeddings\n",
    "\n",
    "One key limitation of trafitional pretrained embedding representations such as Word2Vec is the problem of word sense and removing ambiguity by making them clear. While pretrained embeddings can capture some of the meaning of words in context, every possible meaning of a word is encoded into the same embedding. This can cause problems in downstream models, since many words such as the word `play` have different meanings depending on the context they are used in.\n",
    "\n",
    "For example, the word `play` in these two different sentences have quite different meaning:\n",
    "* I went to `play` at the theatre\n",
    "* John wants to `play` with his friends\n",
    "\n",
    "The pretrained embeddings above represent both meanings of the word 'play' in the same embedding. To overcome this limitation, we need to build embeddings based on the `language model`, which is trained on a large corpus of text, and knows how words can be put together in different contexts."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}