POS-task.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__This seminar:__ after you're done coding your own recurrent cells, it's time you learn how to train recurrent networks easily with Keras. We'll also learn some tricks on how to use keras layers and model. We also want you to note that this is a non-graded assignment, meaning you are not required to pass it for a certificate.\n",
    "\n",
    "Enough beatin' around the bush, let's get to the task!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Part Of Speech Tagging\n",
    "\n",
    "<img src=https://i.stack.imgur.com/6pdIT.png width=320>\n",
    "\n",
    "Unlike our previous experience with language modelling, this time around we learn the mapping between two different kinds of elements.\n",
    "\n",
    "This setting is common for a range of useful problems:\n",
    "* Speech Recognition - processing human voice into text\n",
    "* Part Of Speech Tagging - for morphology-aware search and as an auxuliary task for most NLP problems\n",
    "* Named Entity Recognition - for chat bots and web crawlers\n",
    "* Protein structure prediction - for bioinformatics\n",
    "\n",
    "Our current guest is part-of-speech tagging. As the name suggests, it's all about converting a sequence of words into a sequence of part-of-speech tags. We'll use a reduced tag set for simplicity:\n",
    "\n",
    "### POS-tags\n",
    "- ADJ - adjective (new, good, high, ...)\n",
    "- ADP - adposition\t(on, of, at, ...)\n",
    "- ADV - adverb\t(really, already, still, ...)\n",
    "- CONJ\t- conjunction\t(and, or, but, ...)\n",
    "- DET - determiner, article\t(the, a, some, ...)\n",
    "- NOUN\t- noun\t(year, home, costs, ...)\n",
    "- NUM - numeral\t(twenty-four, fourth, 1991, ...)\n",
    "- PRT -\tparticle (at, on, out, ...)\n",
    "- PRON - pronoun (he, their, her, ...)\n",
    "- VERB - verb (is, say, told, ...)\n",
    "- .\t- punctuation marks\t(. , ;)\n",
    "- X\t- other\t(ersatz, esprit, dunno, ...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package brown to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/brown.zip.\n",
      "[nltk_data] Downloading package universal_tagset to\n",
      "[nltk_data]     /home/jovyan/nltk_data...\n",
      "[nltk_data]   Unzipping taggers/universal_tagset.zip.\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "import sys\n",
    "import numpy as np\n",
    "nltk.download('brown')\n",
    "nltk.download('universal_tagset')\n",
    "data = nltk.corpus.brown.tagged_sents(tagset='universal')\n",
    "all_tags = ['#EOS#','#UNK#','ADV', 'NOUN', 'ADP', 'PRON', 'DET', '.', 'PRT', 'VERB', 'X', 'NUM', 'CONJ', 'ADJ']\n",
    "\n",
    "data = np.array([ [(word.lower(),tag) for word,tag in sentence] for sentence in data ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import train_test_split\n",
    "train_data,test_data = train_test_split(data,test_size=0.25,random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table><tr><td>NOUN</td><td>ADP</td><td>NOUN</td><td>NOUN</td><td>NOUN</td><td>NOUN</td><td>VERB</td><td>ADV</td><td>VERB</td><td>ADP</td><td>DET</td><td>ADJ</td><td>NOUN</td><td>.</td></tr><td>implementation</td><td>of</td><td>georgia's</td><td>automobile</td><td>title</td><td>law</td><td>was</td><td>also</td><td>recommended</td><td>by</td><td>the</td><td>outgoing</td><td>jury</td><td>.</td><tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tr><td>PRON</td><td>VERB</td><td>ADP</td><td>DET</td><td>NOUN</td><td>.</td><td>VERB</td><td>NOUN</td><td>PRT</td><td>VERB</td><td>.</td><td>DET</td><td>NOUN</td><td>.</td></tr><td>it</td><td>urged</td><td>that</td><td>the</td><td>city</td><td>``</td><td>take</td><td>steps</td><td>to</td><td>remedy</td><td>''</td><td>this</td><td>problem</td><td>.</td><tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table><tr><td>NOUN</td><td>VERB</td></tr><td>merger</td><td>proposed</td><tr></table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import HTML, display\n",
    "def draw(sentence):\n",
    "    words,tags = zip(*sentence)\n",
    "    display(HTML('<table><tr>{tags}</tr>{words}<tr></table>'.format(\n",
    "                words = '<td>{}</td>'.format('</td><td>'.join(words)),\n",
    "                tags = '<td>{}</td>'.format('</td><td>'.join(tags)))))\n",
    "    \n",
    "    \n",
    "draw(data[11])\n",
    "draw(data[10])\n",
    "draw(data[7])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building vocabularies\n",
    "\n",
    "Just like before, we have to build a mapping from tokens to integer ids. This time around, our model operates on a word level, processing one word per RNN step. This means we'll have to deal with far larger vocabulary.\n",
    "\n",
    "Luckily for us, we only receive those words as input i.e. we don't have to predict them. This means we can have a large vocabulary for free by using word embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coverage = 0.92876\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "word_counts = Counter()\n",
    "for sentence in data:\n",
    "    words,tags = zip(*sentence)\n",
    "    word_counts.update(words)\n",
    "\n",
    "all_words = ['#EOS#','#UNK#']+list(list(zip(*word_counts.most_common(10000)))[0])\n",
    "\n",
    "#let's measure what fraction of data words are in the dictionary\n",
    "print(\"Coverage = %.5f\"%(float(sum(word_counts[w] for w in all_words)) / sum(word_counts.values())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "word_to_id = defaultdict(lambda:1,{word:i for i,word in enumerate(all_words)})\n",
    "tag_to_id = {tag:i for i,tag in enumerate(all_tags)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "convert words and tags into fixed-size matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def to_matrix(lines,token_to_id,max_len=None,pad=0,dtype='int32',time_major=False):\n",
    "    \"\"\"Converts a list of names into rnn-digestable matrix with paddings added after the end\"\"\"\n",
    "    \n",
    "    max_len = max_len or max(map(len,lines))\n",
    "    matrix = np.empty([len(lines),max_len],dtype)\n",
    "    matrix.fill(pad)\n",
    "\n",
    "    for i in range(len(lines)):\n",
    "        line_ix = list(map(token_to_id.__getitem__,lines[i]))[:max_len]\n",
    "        matrix[i,:len(line_ix)] = line_ix\n",
    "\n",
    "    return matrix.T if time_major else matrix\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Word ids:\n",
      "[[   2 3057    5    2 2238 1334 4238 2454    3    6   19   26 1070   69\n",
      "     8 2088    6    3    1    3  266   65  342    2    1    3    2  315\n",
      "     1    9   87  216 3322   69 1558    4    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0]\n",
      " [  45   12    8  511 8419    6   60 3246   39    2    1    1    3    2\n",
      "   845    1    3    1    3   10 9910    2    1 3470    9   43    1    1\n",
      "     3    6    2 1046  385   73 4562    3    9    2    1    1 3250    3\n",
      "    12   10    2  861 5240   12    8 8936  121    1    4]\n",
      " [  33   64   26   12  445    7 7346    9    8 3337    3    1 2811    3\n",
      "     2  463  572    2    1    1 1649   12    1    4    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0]]\n",
      "Tag ids:\n",
      "[[ 6  3  4  6  3  3  9  9  7 12  4  5  9  4  6  3 12  7  9  7  9  8  4  6\n",
      "   3  7  6 13  3  4  6  3  9  4  3  7  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0]\n",
      " [ 5  9  6  9  3 12  6  3  7  6 13  3  7  6 13  3  7 13  7  5  9  6  3  3\n",
      "   4  6 13  3  7 12  6  3  6 13  3  7  4  6  3  9  3  7  9  4  6 13  3  9\n",
      "   6  3  2 13  7]\n",
      " [ 4  6  5  9 13  4  3  4  6 13  7 13  3  7  6  3  4  6 13  3  3  9  9  7\n",
      "   0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0\n",
      "   0  0  0  0  0]]\n"
     ]
    }
   ],
   "source": [
    "batch_words,batch_tags = zip(*[zip(*sentence) for sentence in data[-3:]])\n",
    "\n",
    "print(\"Word ids:\")\n",
    "print(to_matrix(batch_words,word_to_id))\n",
    "print(\"Tag ids:\")\n",
    "print(to_matrix(batch_tags,tag_to_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Build model\n",
    "\n",
    "Unlike our previous lab, this time we'll focus on a high-level keras interface to recurrent neural networks. It is as simple as you can get with RNN, allbeit somewhat constraining for complex tasks like seq2seq.\n",
    "\n",
    "By default, all keras RNNs apply to a whole sequence of inputs and produce a sequence of hidden states `(return_sequences=True` or just the last hidden state `(return_sequences=False)`. All the recurrence is happening under the hood.\n",
    "\n",
    "At the top of our model we need to apply a Dense layer to each time-step independently. As of now, by default keras.layers.Dense would apply once to all time-steps concatenated. We use __keras.layers.TimeDistributed__ to modify Dense layer so that it would apply across both batch and time axes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "import keras.layers as L\n",
    "\n",
    "model = keras.models.Sequential()\n",
    "model.add(L.InputLayer([None],dtype='int32'))\n",
    "model.add(L.Embedding(len(all_words),50))\n",
    "model.add(L.SimpleRNN(64,return_sequences=True))\n",
    "\n",
    "#add top layer that predicts tag probabilities\n",
    "stepwise_dense = L.Dense(len(all_tags),activation='softmax')\n",
    "stepwise_dense = L.TimeDistributed(stepwise_dense)\n",
    "model.add(stepwise_dense)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Training:__ in this case we don't want to prepare the whole training dataset in advance. The main cause is that the length of every batch depends on the maximum sentence length within the batch. This leaves us two options: use custom training code as in previous seminar or use generators.\n",
    "\n",
    "Keras models have a __`model.fit_generator`__ method that accepts a python generator yielding one batch at a time. But first we need to implement such generator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from keras.utils.np_utils import to_categorical\n",
    "BATCH_SIZE=32\n",
    "def generate_batches(sentences,batch_size=BATCH_SIZE,max_len=None,pad=0):\n",
    "    assert isinstance(sentences,np.ndarray),\"Make sure sentences is q numpy array\"\n",
    "    \n",
    "    while True:\n",
    "        indices = np.random.permutation(np.arange(len(sentences)))\n",
    "        for start in range(0,len(indices)-1,batch_size):\n",
    "            batch_indices = indices[start:start+batch_size]\n",
    "            batch_words,batch_tags = [],[]\n",
    "            for sent in sentences[batch_indices]:\n",
    "                words,tags = zip(*sent)\n",
    "                batch_words.append(words)\n",
    "                batch_tags.append(tags)\n",
    "\n",
    "            batch_words = to_matrix(batch_words,word_to_id,max_len,pad)\n",
    "            batch_tags = to_matrix(batch_tags,tag_to_id,max_len,pad)\n",
    "\n",
    "            batch_tags_1hot = to_categorical(batch_tags,len(all_tags)).reshape(batch_tags.shape+(-1,))\n",
    "            yield batch_words,batch_tags_1hot\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "__Callbacks:__ Another thing we need is to measure model performance. The tricky part is not to count accuracy after sentence ends (on padding) and making sure we count all the validation data exactly once.\n",
    "\n",
    "While it isn't impossible to persuade Keras to do all of that, we may as well write our own callback that does that.\n",
    "Keras callbacks allow you to write a custom code to be ran once every epoch or every minibatch. We'll define one via LambdaCallback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_test_accuracy(model):\n",
    "    test_words,test_tags = zip(*[zip(*sentence) for sentence in test_data])\n",
    "    test_words,test_tags = to_matrix(test_words,word_to_id),to_matrix(test_tags,tag_to_id)\n",
    "\n",
    "    #predict tag probabilities of shape [batch,time,n_tags]\n",
    "    predicted_tag_probabilities = model.predict(test_words,verbose=1)\n",
    "    predicted_tags = predicted_tag_probabilities.argmax(axis=-1)\n",
    "\n",
    "    #compute accurary excluding padding\n",
    "    numerator = np.sum(np.logical_and((predicted_tags == test_tags),(test_words != 0)))\n",
    "    denominator = np.sum(test_words != 0)\n",
    "    return float(numerator)/denominator\n",
    "\n",
    "\n",
    "class EvaluateAccuracy(keras.callbacks.Callback):\n",
    "    def on_epoch_end(self,epoch,logs=None):\n",
    "        sys.stdout.flush()\n",
    "        print(\"\\nMeasuring validation accuracy...\")\n",
    "        acc = compute_test_accuracy(self.model)\n",
    "        print(\"\\nValidation accuracy: %.5f\\n\"%acc)\n",
    "        sys.stdout.flush()\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1343/1343 [============================>.] - ETA: 0s - loss: 0.2624\n",
      "Measuring validation accuracy...\n",
      "14304/14335 [============================>.] - ETA: 0s\n",
      "Validation accuracy: 0.93946\n",
      "\n",
      "1344/1343 [==============================] - 82s - loss: 0.2622    \n",
      "Epoch 2/5\n",
      "1342/1343 [============================>.] - ETA: 0s - loss: 0.0581\n",
      "Measuring validation accuracy...\n",
      "14304/14335 [============================>.] - ETA: 0s\n",
      "Validation accuracy: 0.94430\n",
      "\n",
      "1344/1343 [==============================] - 80s - loss: 0.0581    \n",
      "Epoch 3/5\n",
      "1342/1343 [============================>.] - ETA: 0s - loss: 0.0511\n",
      "Measuring validation accuracy...\n",
      "14335/14335 [==============================] - 16s    \n",
      "\n",
      "Validation accuracy: 0.94533\n",
      "\n",
      "1344/1343 [==============================] - 80s - loss: 0.0511    \n",
      "Epoch 4/5\n",
      "1342/1343 [============================>.] - ETA: 0s - loss: 0.0462\n",
      "Measuring validation accuracy...\n",
      "14304/14335 [============================>.] - ETA: 0s\n",
      "Validation accuracy: 0.94447\n",
      "\n",
      "1344/1343 [==============================] - 81s - loss: 0.0462    \n",
      "Epoch 5/5\n",
      "1343/1343 [============================>.] - ETA: 0s - loss: 0.0420\n",
      "Measuring validation accuracy...\n",
      "14335/14335 [==============================] - 17s    \n",
      "\n",
      "Validation accuracy: 0.94510\n",
      "\n",
      "1344/1343 [==============================] - 86s - loss: 0.0420    \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f38b9418b38>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile('adam','categorical_crossentropy')\n",
    "\n",
    "model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,\n",
    "                    callbacks=[EvaluateAccuracy()], epochs=5,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Measure final accuracy on the whole test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14335/14335 [==============================] - 19s    \n",
      "Final accuracy: 0.94510\n"
     ]
    }
   ],
   "source": [
    "acc = compute_test_accuracy(model)\n",
    "print(\"Final accuracy: %.5f\"%acc)\n",
    "\n",
    "assert acc>0.94, \"Keras has gone on a rampage again, please contact course staff.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task I: getting all bidirectional\n",
    "\n",
    "Since we're analyzing a full sequence, it's legal for us to look into future data.\n",
    "\n",
    "A simple way to achieve that is to go both directions at once, making a __bidirectional RNN__.\n",
    "\n",
    "In Keras you can achieve that both manually (using two LSTMs and Concatenate) and by using __`keras.layers.Bidirectional`__. \n",
    "\n",
    "This one works just as `TimeDistributed` we saw before: you wrap it around a recurrent layer (SimpleRNN now and LSTM/GRU later) and it actually creates two layers under the hood.\n",
    "\n",
    "Your first task is to use such a layer our POS-tagger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Define a model that utilizes bidirectional SimpleRNN\n",
    "model = keras.models.Sequential()\n",
    "\n",
    "model.add(L.InputLayer([None],dtype='int32'))\n",
    "model.add(L.Embedding(len(all_words),50))\n",
    "model.add(L.Bidirectional(L.SimpleRNN(64,return_sequences=True)))\n",
    "\n",
    "#add top layer that predicts tag probabilities\n",
    "stepwise_dense = L.Dense(len(all_tags),activation='softmax')\n",
    "stepwise_dense = L.TimeDistributed(stepwise_dense)\n",
    "model.add(stepwise_dense)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "1343/1343 [============================>.] - ETA: 0s - loss: 0.1830\n",
      "Measuring validation accuracy...\n",
      "14335/14335 [==============================] - 38s    \n",
      "\n",
      "Validation accuracy: 0.95669\n",
      "\n",
      "1344/1343 [==============================] - 165s - loss: 0.1829   \n",
      "Epoch 2/5\n",
      "1343/1343 [============================>.] - ETA: 0s - loss: 0.0421\n",
      "Measuring validation accuracy...\n",
      "14335/14335 [==============================] - 34s    \n",
      "\n",
      "Validation accuracy: 0.96090\n",
      "\n",
      "1344/1343 [==============================] - 153s - loss: 0.0421   \n",
      "Epoch 3/5\n",
      "1343/1343 [============================>.] - ETA: 0s - loss: 0.0299\n",
      "Measuring validation accuracy...\n",
      "14335/14335 [==============================] - 37s    \n",
      "\n",
      "Validation accuracy: 0.96292\n",
      "\n",
      "1344/1343 [==============================] - 164s - loss: 0.0299   \n",
      "Epoch 5/5\n",
      "1343/1343 [============================>.] - ETA: 0s - loss: 0.0253\n",
      "Measuring validation accuracy...\n",
      "14335/14335 [==============================] - 65s    \n",
      "\n",
      "Validation accuracy: 0.96212\n",
      "\n",
      "1344/1343 [==============================] - 195s - loss: 0.0253   \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f38907e9e48>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile('adam','categorical_crossentropy')\n",
    "\n",
    "model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,\n",
    "                    callbacks=[EvaluateAccuracy()], epochs=5,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14335/14335 [==============================] - 52s    \n",
      "\n",
      "Final accuracy: 0.96212\n",
      "Well done!\n"
     ]
    }
   ],
   "source": [
    "acc = compute_test_accuracy(model)\n",
    "print(\"\\nFinal accuracy: %.5f\"%acc)\n",
    "\n",
    "assert acc>0.96, \"Bidirectional RNNs are better than this!\"\n",
    "print(\"Well done!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task II: now go and improve it\n",
    "\n",
    "You guesses it. We're now gonna ask you to come up with a better network.\n",
    "\n",
    "Here's a few tips:\n",
    "\n",
    "* __Go beyond SimpleRNN__: there's `keras.layers.LSTM` and `keras.layers.GRU`\n",
    "  * If you want to use a custom recurrent Cell, read [this](https://keras.io/layers/recurrent/#rnn)\n",
    "  * You can also use 1D Convolutions (`keras.layers.Conv1D`). They are often as good as recurrent layers but with less overfitting.\n",
    "* __Stack more layers__: if there is a common motif to this course it's about stacking layers\n",
    "  * You can just add recurrent and 1dconv layers on top of one another and keras will understand it\n",
    "  * Just remember that bigger networks may need more epochs to train\n",
    "* __Gradient clipping__: If your training isn't as stable as you'd like, set `clipnorm` in your optimizer.\n",
    "  * Which is to say, it's a good idea to watch over your loss curve at each minibatch. Try tensorboard callback or something similar.\n",
    "* __Regularization__: you can apply dropouts as usuall but also in an RNN-specific way\n",
    "  * `keras.layers.Dropout` works inbetween RNN layers\n",
    "  * Recurrent layers also have `recurrent_dropout` parameter\n",
    "* __More words!__: You can obtain greater performance by expanding your model's input dictionary from 5000 to up to every single word!\n",
    "  * Just make sure your model doesn't overfit due to so many parameters.\n",
    "  * Combined with regularizers or pre-trained word-vectors this could be really good cuz right now our model is blind to >5% of words.\n",
    "* __The most important advice__: don't cram in everything at once!\n",
    "  * If you stuff in a lot of modiffications, some of them almost inevitably gonna be detrimental and you'll never know which of them are.\n",
    "  * Try to instead go in small iterations and record experiment results to guide further search.\n",
    "  \n",
    "There's some advanced stuff waiting at the end of the notebook.\n",
    "  \n",
    "Good hunting!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Define a model that utilizes bidirectional SimpleRNN\n",
    "model = keras.models.Sequential()\n",
    "\n",
    "model.add(L.InputLayer([None],dtype='int32'))\n",
    "model.add(L.Embedding(len(all_words),50))\n",
    "model.add(L.Bidirectional(L.GRU(128,return_sequences=True,activation='relu')))\n",
    "model.add(L.Dropout(0.5))\n",
    "model.add(L.Bidirectional(L.GRU(64,return_sequences=True,activation='relu')))\n",
    "model.add(L.Dropout(0.5))\n",
    "\n",
    "#add top layer that predicts tag probabilities\n",
    "stepwise_dense = L.Dense(len(all_tags),activation='softmax')\n",
    "stepwise_dense = L.TimeDistributed(stepwise_dense)\n",
    "model.add(stepwise_dense)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      " 969/1343 [====================>.........] - ETA: 287s - loss: 0.2875"
     ]
    }
   ],
   "source": [
    "#feel free to change anything here\n",
    "\n",
    "model.compile('adam','categorical_crossentropy')\n",
    "\n",
    "model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,\n",
    "                    callbacks=[EvaluateAccuracy()], epochs=5,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "acc = compute_test_accuracy(model)\n",
    "print(\"\\nFinal accuracy: %.5f\"%acc)\n",
    "\n",
    "if acc >= 0.99:\n",
    "    print(\"Awesome! Sky was the limit and yet you scored even higher!\")\n",
    "elif acc >= 0.98:\n",
    "    print(\"Excellent! Whatever dark magic you used, it certainly did it's trick.\")\n",
    "elif acc >= 0.97:\n",
    "    print(\"Well done! If this was a graded assignment, you would have gotten a 100% score.\")\n",
    "elif acc > 0.96:\n",
    "    print(\"Just a few more iterations!\")\n",
    "else:\n",
    "    print(\"There seems to be something broken in the model. Unless you know what you're doing, try taking bidirectional RNN and adding one enhancement at a time to see where's the problem.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "#### Some advanced stuff\n",
    "Here there are a few more tips on how to improve training that are a bit trickier to impliment. We strongly suggest that you try them _after_ you've got a good initial model.\n",
    "* __Use pre-trained embeddings__: you can use pre-trained weights from [there](http://ahogrammer.com/2017/01/20/the-list-of-pretrained-word-embeddings/) to kickstart your Embedding layer.\n",
    "  * Embedding layer has a matrix W (layer.W) which contains word embeddings for each word in the dictionary. You can just overwrite them with tf.assign.\n",
    "  * When using pre-trained embeddings, pay attention to the fact that model's dictionary is different from your own.\n",
    "  * You may want to switch trainable=False for embedding layer in first few epochs as in regular fine-tuning.  \n",
    "* __More efficient baching__: right now TF spends a lot of time iterating over \"0\"s\n",
    "  * This happens because batch is always padded to the length of a longest sentence\n",
    "  * You can speed things up by pre-generating batches of similar lengths and feeding it with randomly chosen pre-generated batch.\n",
    "  * This technically breaks the i.i.d. assumption, but it works unless you come up with some insane rnn architectures.\n",
    "* __Structured loss functions__: since we're tagging the whole sequence at once, we might as well train our network to do so.\n",
    "  * There's more than one way to do so, but we'd recommend starting with [Conditional Random Fields](http://blog.echen.me/2012/01/03/introduction-to-conditional-random-fields/)\n",
    "  * You could plug CRF as a loss function and still train by backprop. There's even some neat tensorflow [implementation](https://www.tensorflow.org/api_guides/python/contrib.crf) for you.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}