Lab3_FunWithMNIST.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Lab3_FunWithMNIST",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "REaV1GblZovs",
        "colab_type": "text"
      },
      "source": [
        "#Fun with MNIST\n",
        "Convolutional Neural Networks (CNNs) are able to solve a wide range of computer vision tasks. In this Lab you will learn about\n",
        "\n",
        "- Image classification\n",
        "- Convolutional AutoEncoders\n",
        "- Denoising AutoEncoders\n",
        "- Image super resolution\n",
        "- Image regression\n",
        "- Image segmentation\n",
        "- Object detection\n",
        "- Few-shot learning with Siamese networks\n",
        "- Generative Adversarial Networks (GANs)\n",
        "\n",
        "The purpose of the Lab is to give your some intuition about how to tweak CNNs to solve different tasks.\n",
        "\n",
        "**Before we start - remember to set runtime to GPU**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qc8RJkIJaxap",
        "colab_type": "text"
      },
      "source": [
        "##Task 1: Downloading and pre-processing the MNIST dataset\n",
        "The MNIST dataset of handwritten digits is so commonly used that it comes with most deep learning frameworks, including Keras. Let's download the dataset and explore a little bit."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "r-_9ZwQkiGQg",
        "colab_type": "code",
        "outputId": "c5139861-bce9-4bae-9c68-5277bc0482ac",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 72
        }
      },
      "source": [
        "from __future__ import print_function\n",
        "import keras\n",
        "from keras.datasets import mnist\n",
        "from keras import backend as K\n",
        "\n",
        "num_classes = 10\n",
        "\n",
        "# input image dimensions\n",
        "img_rows, img_cols = 28, 28\n",
        "\n",
        "# the data, split between train and test sets\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "\n",
        "if K.image_data_format() == 'channels_first':\n",
        "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
        "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
        "    input_shape = (1, img_rows, img_cols)\n",
        "else:\n",
        "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
        "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
        "    input_shape = (img_rows, img_cols, 1)\n",
        "\n",
        "# Pre-process inputs\n",
        "x_train = x_train.astype('float32')\n",
        "x_test = x_test.astype('float32')\n",
        "x_train /= 255\n",
        "x_test /= 255\n",
        "\n",
        "# Convert class indices to one-hot vectors\n",
        "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
        "y_test = keras.utils.to_categorical(y_test, num_classes)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n",
            "11493376/11490434 [==============================] - 2s 0us/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xHvyXrdliPFE",
        "colab_type": "text"
      },
      "source": [
        "###Questions 1.1\n",
        "1. What is the input shape?\n",
        "2. How many training examples are there?\n",
        "3. How many test examples are there?\n",
        "4. What does `to_categorical` do?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5OX-mswCi2vY",
        "colab_type": "text"
      },
      "source": [
        "##Task 2: Logistic regression\n",
        "Now, let's define a tiny Keras model for logistic regression. Mathematically this model outputs a 10-dimensional vector `y` of class probabilities, where\n",
        "\n",
        "\n",
        "```\n",
        "y = softmax(W*x + b)\n",
        "```\n",
        "\n",
        "and\n",
        "- `x` is a 28x28 = 784-dimensional vector corresponding to the input image, \n",
        "- `W` is a 10 x 784 matrix of weights\n",
        "- `b` is a 10-dimensional vector of biases\n",
        "\n",
        "Defining models in Keras is not very intuitive from a mathematical perspective. Here is one way to implement the equation above using Keras' [Sequential API](https://keras.io/getting-started/sequential-model-guide/). A Sequential model is a just a linear stack of layers."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HSqZ3O4ocxQ-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.models import Sequential\n",
        "from keras.layers import Dense, Flatten, Activation\n",
        "\n",
        "# Model\n",
        "model = Sequential()\n",
        "model.add(Flatten(input_shape=input_shape))\n",
        "model.add(Dense(num_classes))\n",
        "model.add(Activation('softmax'))\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CJBX3bW9kaq8",
        "colab_type": "text"
      },
      "source": [
        "###Questions 2.1\n",
        "1. What does Flatten() do and why is it necessary to have it here?\n",
        "2. What does Dense() do, and why does it have 7850 parameters?\n",
        "3. What does Activation() do?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7PqpHs6EgXMP",
        "colab_type": "text"
      },
      "source": [
        "###Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eABHA5aSmBdy",
        "colab_type": "text"
      },
      "source": [
        "Now, let's train the model for 10 iterations (epochs). We will be using the cross entropy loss for multiple classes and stochastic gradient descent (SGD). The difference between normal gradient descent and SGD is that normal gradient descent calculates the gradients based on all training examples, whereas SGD approximates the gradient by calculating it on small batches (of size 128 in this example)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qimN5at5djhI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "batch_size = 128\n",
        "epochs = 10\n",
        "\n",
        "# Compile the model before training\n",
        "model.compile(optimizer='sgd',loss='categorical_crossentropy',metrics=['accuracy'])\n",
        "\n",
        "# Fit model\n",
        "history = model.fit(x_train, y_train,\n",
        "            batch_size=batch_size,\n",
        "            epochs=epochs,\n",
        "            verbose=1,\n",
        "            validation_data=(x_test, y_test))\n",
        "\n",
        "score = model.evaluate(x_test, y_test, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q_r7oO_Pm8TK",
        "colab_type": "text"
      },
      "source": [
        "###Model evaluation\n",
        "You can visualize the loss and accuracy curves like this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gj2-tNJShfjj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def show_history(history):\n",
        "  plt.figure(figsize=(20,6))\n",
        "\n",
        "  # summarize history for accuracy\n",
        "  plt.subplot(121)\n",
        "  plt.plot(history.history['acc'])\n",
        "  plt.plot(history.history['val_acc'])\n",
        "  plt.title('model accuracy')\n",
        "  plt.ylabel('accuracy')\n",
        "  plt.xlabel('epoch')\n",
        "  plt.legend(['train', 'test'], loc='upper left')\n",
        "\n",
        "  # summarize history for loss\n",
        "  plt.subplot(122)\n",
        "  plt.plot(history.history['loss'])\n",
        "  plt.plot(history.history['val_loss'])\n",
        "  plt.title('model loss')\n",
        "  plt.ylabel('loss')\n",
        "  plt.xlabel('epoch')\n",
        "  plt.legend(['train', 'test'], loc='upper left')\n",
        "  plt.show()\n",
        "\n",
        "show_history(history)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U8v3SUB2o-Tp",
        "colab_type": "text"
      },
      "source": [
        "###Task 2.1: Training on a smaller data set\n",
        "Training on the entire MNIST training data set is guaranteed to work (almost) always. Simply because of the large number of images in the training set. This is boring. So let's make our problem a little more challenging by reducing the number of training examples to just 10 from each class. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wxOTzj5tp46I",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "# create smaller training set\n",
        "digit_indices = np.asarray([np.where(np.argmax(y_train,axis=1) == i)[0][np.random.randint(0,5000,10)] for i in range(num_classes)]).flatten()\n",
        "x_train_small = x_train[digit_indices,:]\n",
        "y_train_small = y_train[digit_indices,:]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G8WwvRml4ebZ",
        "colab_type": "text"
      },
      "source": [
        "... and re-train the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0J_53yTrryTY",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "batch_size = 100 # There are only 10 x 10 training images, so batch size should be 100\n",
        "epochs = 200 # We need more epoch becaue we have fewer training samples\n",
        "\n",
        "# Model (redefine the model in order to reinitialize the weights to random values)\n",
        "model = Sequential()\n",
        "model.add(Flatten(input_shape=input_shape))\n",
        "model.add(Dense(num_classes))\n",
        "model.add(Activation('softmax'))\n",
        "\n",
        "# Compile the model before training\n",
        "model.compile(optimizer=keras.optimizers.SGD(),loss='categorical_crossentropy',metrics=['accuracy'])\n",
        "\n",
        "# Fit model (this will take a little while. Set verbose to 1 if you want to see how training progresses)\n",
        "history = model.fit(x_train_small, y_train_small,\n",
        "            batch_size=batch_size,\n",
        "            epochs=epochs,\n",
        "            validation_data=(x_test, y_test),\n",
        "            verbose=0,\n",
        "            shuffle=True)\n",
        "\n",
        "score = model.evaluate(x_test, y_test, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])\n",
        "\n",
        "# Plot old vs new loss\n",
        "show_history(history)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nkmF52862hxn",
        "colab_type": "text"
      },
      "source": [
        "###Question 2.2\n",
        "1. What is the accuracy of our model on the small data set?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jypf_eK3Iopc",
        "colab_type": "text"
      },
      "source": [
        "###Task 2.2: Finding a better learning rate\n",
        "Your task is to find a better learning rate than the default used above, which is 0.01. You can adjust the learning rate by setting the `lr` argument of keras.optimizer.SGD:\n",
        "\n",
        "```\n",
        "keras.optimizers.SGD(lr=0.01)\n",
        "```\n",
        "\n",
        "To make the process of finding a better learning rate faster, you may want to initially lower the number of epochs. When you are done searhing for a better learning rate, retrain your model with 200 epochs.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bTIMPPIXnmps",
        "colab_type": "text"
      },
      "source": [
        "###Task 2.3: Displaying the learned weights\n",
        "Figure out a way to extract the learned weight matrix `W` from the model.\n",
        "\n",
        "Then display them as images (there are 10 classes and one 28x28 weight image per class):\n",
        "\n",
        "What do you observe?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eTDJCvLqn_3n",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "W = # Your code goes here\n",
        "W = W.reshape((28,28,10)) # there are 10 classes and one 28x28 weight image per class\n",
        "plt.figure(figsize=(20,4))\n",
        "for i in range(10):\n",
        "  ax = plt.subplot(2,5,i+1)\n",
        "  plt.imshow(W[:,:,i])\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)\n",
        "  plt.gray()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d_HLWIZ7MUms",
        "colab_type": "text"
      },
      "source": [
        "###Task 2.4: Weight decay\n",
        "With only 10 observations per class in our training data set, it is very likely that our model overfits the training data. This leads to poor generalization (i.e., the model doesn't work that well on unseen data).\n",
        "\n",
        "One way to avoid overfitting is by means of regularization. The best kind of regularization is \"adding more data\" (of course). Another possibility is to use weight decay.\n",
        "\n",
        "So let's try to modify the loss function of the model by adding an L2 regularization term. The regularization term is added using an extra parameter to the Dense layer function.\n",
        "\n",
        "Compare the performance (accuracy) with the first model with the default learning rate. Don't expect a huge gain in performance (I got 70%)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uiY_I0FKLmBv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "epochs = 200\n",
        "\n",
        "# lamda is the weight of the L2 penalty term\n",
        "lamda = 1\n",
        "L2_regularizer = keras.regularizers.l2(lamda)\n",
        "\n",
        "# Model\n",
        "model = Sequential()\n",
        "model.add(Flatten(input_shape=input_shape))\n",
        "model.add(Dense(num_classes,\n",
        "                activation='softmax',\n",
        "                kernel_regularizer=L2_regularizer))\n",
        "\n",
        "# Training\n",
        "model.compile(optimizer=keras.optimizers.SGD(),loss='categorical_crossentropy',metrics=['accuracy'])\n",
        "history_reg = model.fit(x_train_small, y_train_small,\n",
        "            batch_size=batch_size,\n",
        "            epochs=epochs,\n",
        "            verbose=0,\n",
        "            validation_data=(x_test, y_test))\n",
        "\n",
        "# Evaluation\n",
        "score = model.evaluate(x_test, y_test, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])\n",
        "\n",
        "# Plot old vs new loss\n",
        "print(\"Loss curves with old learning rate (0.01)\")\n",
        "show_history(history)\n",
        "print(\"Loss curves with new learning rate (0.1)\")\n",
        "show_history(history_reg)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-OigocSPpfuo",
        "colab_type": "text"
      },
      "source": [
        "**Now comes the interesting part:** Try once more to display the weights - this time of the regularized model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4Ry1Am1ddFV9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#W = # Your code goes here\n",
        "W = W.reshape((28,28,10))\n",
        "plt.figure(figsize=(20,4))\n",
        "for i in range(10):\n",
        "  ax = plt.subplot(2,5,i+1)\n",
        "  plt.imshow(W[:,:,i])\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)\n",
        "  plt.gray()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1rivkoCuppm3",
        "colab_type": "text"
      },
      "source": [
        "###Questions 2.2\n",
        "1. How do the weights of the regularized model differ from the weights of the non-regularized model?\n",
        "2. Can you explain why?."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Y2gg6tWesR1",
        "colab_type": "text"
      },
      "source": [
        "###Functional API instead of Sequential API\n",
        "The models above have been specified using Keras' [Sequential API](https://keras.io/getting-started/sequential-model-guide/). Keras also allows you to specify models a [Functional API](https://keras.io/getting-started/functional-api-guide/). The Keras functional API is the way to go for defining complex models, such as multi-output models, directed acyclic graphs, or models with shared layers.\n",
        "\n",
        "Here is how to set up the logistic regression model using the functional API:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rZ3KOeSjpoTJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.layers import Input\n",
        "from keras.models import Model\n",
        "\n",
        "# This returns a tensor\n",
        "inputs = Input(shape=input_shape)\n",
        "\n",
        "# a layer instance is callable on a tensor, and returns a tensor\n",
        "x = Flatten()(inputs)\n",
        "x = Dense(num_classes)(x)\n",
        "predictions = Activation('softmax')(x)\n",
        "\n",
        "# This creates a model that includes the Input layer and the prediction layer\n",
        "model = Model(inputs=inputs, outputs=predictions)\n",
        "\n",
        "# Training\n",
        "model.compile(optimizer='sgd',loss='categorical_crossentropy',metrics=['accuracy'])\n",
        "model.fit(x_train, y_train,\n",
        "          batch_size=batch_size,\n",
        "          epochs=epochs,\n",
        "          verbose=1,\n",
        "          validation_data=(x_test, y_test))\n",
        "\n",
        "# Evaluation\n",
        "score = model.evaluate(x_test, y_test, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zHf86VZYlrAB",
        "colab_type": "text"
      },
      "source": [
        "##Task 3: Our first CNN\n",
        "Here we will build and train our own CNN:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wvHAmMda0uNE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.layers import Dropout\n",
        "from keras.layers import Conv2D, MaxPooling2D\n",
        "\n",
        "inputs = Input(shape=(28, 28, 1))\n",
        "\n",
        "# Encoder (convolutional base)\n",
        "x = Conv2D(8, kernel_size=(3, 3), activation='relu')(inputs)\n",
        "x = MaxPooling2D((2, 2))(x)\n",
        "x = Conv2D(16, kernel_size=(3, 3), activation='relu')(x)\n",
        "x = MaxPooling2D((2, 2))(x)\n",
        "x = Conv2D(32, kernel_size=(3, 3), activation='relu')(x)\n",
        "encoded = Flatten()(x)\n",
        "\n",
        "# Decoder (2 fully connected layers)\n",
        "x = Dense(64, activation='relu')(encoded)\n",
        "x = Dropout(0.5)(x)\n",
        "predictions = Dense(num_classes,activation='softmax')(x)\n",
        "\n",
        "# This creates a callable model that includes the Input layer and the prediction layer\n",
        "model = Model(inputs=inputs, outputs=predictions)\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SNATzsjAnWCi",
        "colab_type": "text"
      },
      "source": [
        "###Questions 3.1\n",
        "1. How may layers does this CNN have?\n",
        "2. What does the MaxPooling2D layer do?\n",
        "3. What does the Dropout layer do?\n",
        "4. What is the shape of the input of the last convolutiona layer (i.e., just before the flatten layer)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5zgpNVDP2nKi",
        "colab_type": "text"
      },
      "source": [
        "For the record, the same model can also be defined using the sequential API:\n",
        "\n",
        "\n",
        "```\n",
        "from keras.layers import Dropout\n",
        "from keras.layers import Conv2D ![alt text](https://), MaxPooling2D\n",
        "\n",
        "input_img = Input(shape=(28, 28, 1))\n",
        "model = Sequential()\n",
        "model.add(Conv2D(8,\n",
        "                 kernel_size=(3, 3),\n",
        "                 activation='relu',\n",
        "                 input_shape=input_shape))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "model.add(Conv2D(16,\n",
        "                 kernel_size=(3, 3),\n",
        "                 activation='relu'))\n",
        "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
        "model.add(Conv2D(32,\n",
        "                 kernel_size=(3, 3),\n",
        "                 activation='relu'))\n",
        "model.add(Flatten())\n",
        "model.add(Dense(64,\n",
        "                activation='relu'))\n",
        "model.add(Dropout(0.5))\n",
        "model.add(Dense(num_classes, activation='softmax'))\n",
        "\n",
        "model.summary()\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "44dU5LYWrMXY",
        "colab_type": "text"
      },
      "source": [
        "###Training\n",
        "Let's train our model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tB02sLW7mbX6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(loss='categorical_crossentropy',\n",
        "              optimizer='SGD',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "history = model.fit(x_train, y_train,\n",
        "                    batch_size=batch_size,\n",
        "                    epochs=epochs,\n",
        "                    verbose=1,\n",
        "                    validation_data=(x_test, y_test))\n",
        "\n",
        "score = model.evaluate(x_test, y_test, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "G_Sy5ZMP5ZPQ",
        "colab": {}
      },
      "source": [
        "show_history(history)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x_R_hovT5bfA",
        "colab_type": "text"
      },
      "source": [
        "The CNN should perform better than the logistic regression model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jOxQ0LletyuB",
        "colab_type": "text"
      },
      "source": [
        "###Task 3.1\n",
        "In the above model the input shape to the Flatten layer  is 3x3x32, which is then flattened to a 288-dimensional vector (this is the variable named `encoded`)\n",
        "\n",
        "Your task is to modify the network such that variable `encoded` has dimensionality 2 instead of 288.\n",
        "\n",
        "Hint: You could insert an extra layer before the Flatten layer that reduces the 3x3x32 input tensor to a 1x1x2 tensor. There are several solutions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nF0XJUxJ4Qpj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.layers import Dropout\n",
        "from keras.layers import Conv2D, MaxPooling2D\n",
        "\n",
        "inputs = Input(shape=(28, 28, 1))\n",
        "\n",
        "# Encoder (convolutional base)\n",
        "x = Conv2D(8, kernel_size=(3, 3), activation='relu')(inputs)\n",
        "x = MaxPooling2D((2, 2))(x)\n",
        "x = Conv2D(16, kernel_size=(3, 3), activation='relu')(x)\n",
        "x = MaxPooling2D((2, 2))(x)\n",
        "x = Conv2D(32, kernel_size=(3, 3), activation='relu')(x)\n",
        "x = # Your code goes here\n",
        "encoded = Flatten()(x)\n",
        "\n",
        "# Decoder (2 fully connected layers)\n",
        "x = Dense(64, activation='relu')(encoded)\n",
        "x = Dropout(0.5)(x)\n",
        "predictions = Dense(num_classes,activation='softmax')(x)\n",
        "\n",
        "# This creates a callable model that includes the Input layer and the prediction layer\n",
        "model = Model(inputs=inputs, outputs=predictions)\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "69KNBxBEv53g",
        "colab_type": "text"
      },
      "source": [
        "###Training\n",
        "Now, train the modified model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eyDbES-izBol",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(loss=keras.losses.categorical_crossentropy,\n",
        "              optimizer=keras.optimizers.Adadelta(),\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.fit(x_train, y_train,\n",
        "          batch_size=batch_size,\n",
        "          epochs=epochs,\n",
        "          verbose=1,\n",
        "          validation_data=(x_test, y_test))\n",
        "\n",
        "score = model.evaluate(x_test, y_test, verbose=0)\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9HrN8wbHwA0q",
        "colab_type": "text"
      },
      "source": [
        "###Task 3.2\n",
        "Run all test examples (x_test) through your model and for each example extract the 2-dimensional vector output of the Flatten layer (variable named `encoded`)\n",
        "\n",
        "Then plot those vectors in a 2D plot, where each class gets its own color.\n",
        "\n",
        "You might find this code useful:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AGUoaLHVwn5T",
        "colab_type": "code",
        "outputId": "62739315-a821-4ee0-f8b2-84230e57949a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 269
        }
      },
      "source": [
        "model_encoded = Model(inputs=model.input, outputs=encoded)\n",
        "\n",
        "# Plot 10 dots with 10 different colors\n",
        "for i in range(10):\n",
        "  plt.plot(i,i,'.C'+str(i))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADi1JREFUeJzt3V9snfV9x/HP5xzbaR1aYxFDRXBj\noi4dUNQ4HEIKajWVqupUBDetxLaijIlys7UUdaq6TlUvpt51XTupQgspCJWo1ZYyraoQY9KolAkR\nOI4jpSFthFwHEwgYZExJJvznfHdhU3DmxE+CHz/nm/N+3YSEE/ujR+Sth5998jgiBADIo1b1AADA\nuSHcAJAM4QaAZAg3ACRDuAEgGcINAMkQbgBIhnADQDKEGwCS6Srjg27YsCGGhobK+NAAcEEaGRl5\nNSIGiry2lHAPDQ2p2WyW8aEB4IJk+1jR13JUAgDJEG4ASIZwA0AyhBsAkiHcAJAM4QaAZAg3AKyC\n5vRJ/fOxl9WcPln65yrl+7gBoJM0p0/qCwef02wr1F2z9m79iBp960v7fNxxA8B79OTrb2q2FZqX\nNNsKPfn6m6V+PsINAO/RjRdfpO6aVZfUXbNuvPiiUj8fRyUA8B41+tZr79aP6MnX39SNF19U6jGJ\nRLgBYFU0+taXHuy3cVQCAMkQbgBIhnADQDKEGwCSIdwAkAzhBoBkCDcAJEO4ASAZwg0AyRBuAEiG\ncANAMoQbAJIh3ACQDOEGgGQINwAkUyjctu+1fdj2r23/1Pb7yh4GAFjeiuG2vVHSVyU1IuJjkuqS\nbi97GAAUMT19QOPj92l6+kDVU9ZM0SfgdEl6v+1ZSb2SXixvEgAUMz19QAdG71CrNaNarUfbhn+i\nvr5tVc8q3Yp33BFxXNL3JD0v6SVJ0xHx+Omvs3237abt5uTk5OovBYDTTE3tV6s1I6mlVmtWU1P7\nq560JooclfRLuk3SlZIul7Te9pdOf11E7IqIRkQ0BgYGVn8pAJymv/8G1Wo9kuqq1brV339D1ZPW\nRJGjks9I+l1ETEqS7Uck3Sjp4TKHAcBK+vq2advwTzQ1tV/9/Td0xDGJVCzcz0vaYbtX0v9KullS\ns9RVAFBQX9+2jgn224qcce+XtFfSAUmHFn/PrpJ3AQDOoNB3lUTEdyR9p+QtAIACeOckACRDuAEg\nGcINAMkQbgBIhnADQDKEGwCSIdwAkAzhBoBkCDcAJEO4ASAZwg0AyRBuAEiGcANAMoQbAJIh3ADO\n28TEhPbt26eJiYmqp3SUok95B4AlJiYm9NBDD2l+fl71el07d+7U4OBg1bM6AnfcAM7L+Pi45ufn\nFRGan5/X+Ph41ZM6BuEGcF6GhoZUr9dlW/V6XUNDQ1VP6hgclQA4L4ODg9q5c6fGx8c1NDTEMcka\nItwAztvg4CDBrgBHJQCQDOEGgGQINwAkQ7gBIBnCDQDJEG4ASIZwA0AyhBsAkiHcAJAM4QaAZAg3\nACRDuAEgGcINAMkQbgBIplC4bV9se6/t39g+YvsTZQ8DACyv6N/H/UNJj0XEF2z3SOotcRMA4CxW\nDLftPkmfkvSXkhQRM5Jmyp0F4GzeOvaG3hqb1rrNfVq36YNVz8EaK3LHfaWkSUkP2v64pBFJ90TE\nyVKXAVjWW8fe0Ku7DynmWnJXTRvuupZ4d5giZ9xdkrZJui8ihiWdlPTN019k+27bTdvNycnJVZ4J\n4G1vjU0r5lpSSDHX0ltj01VPwhorEu4XJL0QEfsXf75XCyFfIiJ2RUQjIhoDAwOruRHAu6zb3Cd3\n1SRL7qpp3ea+qidhja14VBIRJ2xP2P5oRPxW0s2Sni1/GoDlrNv0QW2461rOuDtY0e8q+YqkPYvf\nUTIm6c7yJgFYybpNHyTYHaxQuCPioKRGyVsAAAXwzkkASIZwA0AyhBsAkiHcAJAM4QaAZAg3ACRD\nuAEgGcINAMkQbgBIhnADQDKEGwCSIdwAkAzhBoBkCDcAJEO4ASAZwg2cgxePHtH+f/9XvXj0SNVT\n0MGKPgEH6HgvHj2if/uHv9f83JzqXV364re/q8u3XFX1LHQg7riBgiYOH9L83Jyi1dL83JwmDh+q\nehI6FOEGChq85lrVu7rkWk31ri4NXnNt1ZPQoTgqAQq6fMtV+uK3v6uJw4c0eM21HJOgMoQbOAeX\nb7mKYKNyHJUAQDKEGwCSIdwAkAzhBoBkCDcAJEO4ASAZwg0AyRBuAEiGcANAMoQbAJIh3ACQDOEG\ngGQINwAkQ7gBIJnC4bZdtz1q+5dlDgIAnN253HHfI4knpAJAxQqF2/YVkj4vaXe5c4DlnRib1shj\n4zoxNl31FKByRZ+A8wNJ35D0gRK3AMs6MTat//inUc3PtVTvqum2e4f1oc19Vc8CKrPiHbftWyS9\nEhEjK7zubttN283JyclVGwgcPzql+bmWIqT5+ZaOH52qehJQqSJHJTdJutX2uKSfSfq07YdPf1FE\n7IqIRkQ0BgYGVnkmOtnGLf2qd9XkmlSv17RxS3/Vk4BKOSKKv9j+E0l/GxG3nO11jUYjms3me5wG\nvOPE2LSOH53Sxi39HJPggmR7JCIaRV7LU96Rwoc29xFsYNE5hTsifiXpV6UsAQAUwjsnASAZwg0A\nyRBuAEiGcANAMoQbAJIh3ACQDOEGgGQINwAkQ7gBIBnCDQDJEG4ASIZwA0AyhBsAkiHcAJAM4QaA\nZAg3zurU6Khe/ZddOjU6WvUUAIt4Ag7O6NToqJ6/868UMzNyT48+/OAD6h0ernoW0PG448YZnXr6\nGcXMjNRqKWZnderpZ6qeBECEG2fRu/16uadHqtfl7m71br++6kkAxFEJzqJ3eFgffvABnXr6GfVu\nv55jEqBNEG6cVe/wMMEG2gxHJQCQDOEGgGQINwAkQ7gBIBnCDQDJEG4ASIZwA0AyhBsAkiHcAJAM\n4QaAZAg3ACRDuAEgGcINAMkQbgBIZsVw2x60/YTtZ20ftn3PWgwDACyvyN/HPSfp6xFxwPYHJI3Y\n/q+IeLbkbQCAZax4xx0RL0XEgcV//r2kI5I2lj2s0x185aB2H9qtg68crHoKgDZzTk/AsT0kaVjS\n/jLGYMHBVw7qy49/WTPzM+qp9+j+z96vrZdurXoWgDZR+IuTti+S9HNJX4uIN5b593fbbtpuTk5O\nrubGjtN8uamZ+Rm11NJsa1bNl5tVTwLQRgqF23a3FqK9JyIeWe41EbErIhoR0RgYGFjNjR2ncVlD\nPfUe1V1Xd61bjcsaVU8C0EZWPCqxbUk/lnQkIr5f/iRsvXSr7v/s/Wq+3FTjsgbHJACWKHLGfZOk\nOyQdsv32V8q+FRGPljcLWy/dSrABLGvFcEfE/0jyGmwBABTAOycBIBnCDQDJEG4ASIZwA0AyhBsA\nkiHcAJAM4QaAZAg3ACRDuAEgGcINAMkQbgBIhnADQDKEGwCSIdwAkAzhBoBkCPdyJp6W9v3jwo8A\n0GbO6SnvHWHiaemhW6X5GaneI+38hTS4vepVAPAH3HGfbnzfQrRjfuHH8X1VLwKAJQj36YY+uXCn\n7frCj0OfrHoRACzBUcnpBrcvHI+M71uINsckANoM4V7O4HaCDaBtcVQCAMkQbgBIhnADQDKEGwCS\nIdwAkAzhBoBkCDcAJEO4ASAZwg0AyRBuAEiGcANAMoQbAJIh3ACQDOEGgGQKhdv252z/1vZztr9Z\n9igAwJmtGG7bdUk/kvSnkq6W9Ge2ry57GABgeUXuuLdLei4ixiJiRtLPJN1WxpiRY1P60RPPaeTY\nVBkfHgAuCEWegLNR0sS7fv6CpBtWe8jIsSn9xe6nNDPXUk9XTXvu2qHrNvWv9qcBgPRW7YuTtu+2\n3bTdnJycPOff/9TYa5qZa6kV0uxcS0+NvbZa0wDgglIk3MclDb7r51cs/toSEbErIhoR0RgYGDjn\nITs2X6Kerprqlrq7atqx+ZJz/hgA0AmKHJU8I+mPbF+phWDfLunPV3vIdZv6teeuHXpq7DXt2HwJ\nxyQAcAYrhjsi5mz/jaT/lFSX9EBEHC5jzHWb+gk2AKygyB23IuJRSY+WvAUAUADvnASAZAg3ACRD\nuAEgGcINAMkQbgBIxhGx+h/UnpR07Dx/+wZJr67inMy4FktxPZbierzjQrgWmyKi0LsXSwn3e2G7\nGRGNqne0A67FUlyPpbge7+i0a8FRCQAkQ7gBIJl2DPeuqge0Ea7FUlyPpbge7+ioa9F2Z9wAgLNr\nxztuAMBZtE24eSDxO2wP2n7C9rO2D9u+p+pNVbNdtz1q+5dVb6ma7Ytt77X9G9tHbH+i6k1Vsn3v\n4p+TX9v+qe33Vb2pbG0Rbh5I/P/MSfp6RFwtaYekv+7w6yFJ90g6UvWINvFDSY9FxB9L+rg6+LrY\n3ijpq5IaEfExLfzV07dXu6p8bRFureEDiTOIiJci4sDiP/9eC38wN1a7qjq2r5D0eUm7q95SNdt9\nkj4l6ceSFBEzEfF6tasq1yXp/ba7JPVKerHiPaVrl3Av90Dijg3Vu9kekjQsaX+1Syr1A0nfkNSq\nekgbuFLSpKQHF4+OdtteX/WoqkTEcUnfk/S8pJckTUfE49WuKl+7hBvLsH2RpJ9L+lpEvFH1nirY\nvkXSKxExUvWWNtElaZuk+yJiWNJJSR37NSHb/Vr4v/MrJV0uab3tL1W7qnztEu5CDyTuJLa7tRDt\nPRHxSNV7KnSTpFttj2vhCO3Tth+udlKlXpD0QkS8/X9ge7UQ8k71GUm/i4jJiJiV9IikGyveVLp2\nCfcfHkhsu0cLX1z4RcWbKmPbWjjDPBIR3696T5Ui4u8i4oqIGNLCfxf/HREX/B3VmUTECUkTtj+6\n+Es3S3q2wklVe17SDtu9i39ublYHfLG20DMny7aWDyRO4iZJd0g6ZPvg4q99a/HZn8BXJO1ZvMkZ\nk3RnxXsqExH7be+VdEAL3401qg54FyXvnASAZNrlqAQAUBDhBoBkCDcAJEO4ASAZwg0AyRBuAEiG\ncANAMoQbAJL5P6WBbB8P8fJ1AAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9iR1BvO15aCQ",
        "colab_type": "text"
      },
      "source": [
        "###Comment\n",
        "The point of this little exercise is to show you that you can use CNN encoders to compress images down to just 2 dimensions. This is a powerful tool for visualization. In this example, the 2D representation actually separates the 10 classes quite nicely."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6YJnv8WI7b4B",
        "colab_type": "text"
      },
      "source": [
        "##Task 4: Convolutional Autoencoder\n",
        "Autoencoders are special types of neural networks that map the input X to the same output (namely X). So the autoencoder (AE) is an identity function:\n",
        "\n",
        "```\n",
        "X = AE(X)\n",
        "```\n",
        "\n",
        "So what's the point? The point is that the autoencoder compresses the image down to a low-dimensional representation, which can be decoded again to reconstruct the original input image. This has many useful applications, such as data compression and representation learning. Only the important information is stored in the low-dimensional representation.\n",
        "\n",
        "The autoencoder consists of a trained encoder (E) and a trained decoder (D):\n",
        "\n",
        "```\n",
        "X = AE(X) = D(E(X))\n",
        "```\n",
        "\n",
        "It is typically (but not always) the encoding E(X) that we are interested in.\n",
        "\n",
        "Note that the autoecoder does not need the class labels to train. So it is an *unsupervised* machine learning technique.\n",
        "\n",
        "Here is an example of a convolutional autoencoder:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KNUtSwB-1thF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.layers import UpSampling2D\n",
        "\n",
        "inputs = Input(shape=(28, 28, 1))\n",
        "\n",
        "# Encoder (convolutional base)\n",
        "x = Conv2D(16, kernel_size=(3, 3), activation='relu', padding='same')(inputs)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "x = Conv2D(8, kernel_size=(3, 3), activation='relu', padding='same')(x)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "x = Conv2D(8, kernel_size=(3, 3), activation='relu', padding='same')(x)\n",
        "encoded = MaxPooling2D((2, 2), padding='same')(x)\n",
        "print((\"shape of encoded\", K.int_shape(encoded)))\n",
        "\n",
        "# Decoder (upsamling)\n",
        "x = Conv2D(8, kernel_size=(3, 3), activation='relu', padding='same')(encoded)\n",
        "x = UpSampling2D((2, 2))(x)\n",
        "x = Conv2D(8, kernel_size=(3, 3), activation='relu', padding='same')(x)\n",
        "x = UpSampling2D((2, 2))(x)\n",
        "x = Conv2D(16, kernel_size=(3, 3), activation='relu', padding='same')(x) \n",
        "x = UpSampling2D((2, 2))(x)\n",
        "decoded = Conv2D(1, kernel_size=(5, 5), padding='valid')(x)\n",
        "print((\"shape of decoded\", K.int_shape(decoded)))\n",
        "\n",
        "autoencoder = Model(inputs, decoded)\n",
        "autoencoder.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nk0IV2fFAfuw",
        "colab_type": "text"
      },
      "source": [
        "###Questions 4.1\n",
        "1. What is the shape of the encoded image?\n",
        "2. Why are we not flattening (i.e., vectorizing) the encoded image like we did before?\n",
        "3. What does UpSampling2D do?\n",
        "4. Why do you think upsampling is followed by a convolution?\n",
        "5. What is the difference between using `padding='same'` and `padding='valid'`?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bKXQlf8tBnmc",
        "colab_type": "text"
      },
      "source": [
        "###Training\n",
        "Let's train the autoencoder:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vw_OqAHD5k1-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "autoencoder.compile(optimizer='adadelta', loss='mse')\n",
        "autoencoder.fit(x_train, x_train, epochs=20, batch_size=128,\n",
        "               shuffle=True, validation_data=(x_test, x_test), verbose=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FSYL9fe757yF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# utility function for showing images\n",
        "def show_imgs(x_test, n=10):\n",
        "    sz = x_test.shape[1]\n",
        "    plt.figure(figsize=(20, 4))\n",
        "    for i in range(n):\n",
        "        ax = plt.subplot(2, n, i+1)\n",
        "        plt.imshow(x_test[i].reshape(sz,sz))\n",
        "        plt.gray()\n",
        "        ax.get_xaxis().set_visible(False)\n",
        "        ax.get_yaxis().set_visible(False)\n",
        "    plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "edNyzCMW6PaS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "decoded_imgs = autoencoder.predict(x_test)\n",
        "print(\"input (upper row)\")\n",
        "show_imgs(x_test)\n",
        "print(\"decoded (bottom row)\")\n",
        "show_imgs(decoded_imgs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_LAsmAerCSLh",
        "colab_type": "text"
      },
      "source": [
        "###Improving the autoencoder\n",
        "This is good, but not great. Let's use cross entropy loss instead. This requires that the decoding is a probability, hence the sigmoid below:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nJJhiPNeCiAg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "decoded_sigmoid = Activation('sigmoid')(decoded) # decoded is the output of the first autoencoder\n",
        "autoencoder2 = Model(inputs, decoded_sigmoid)\n",
        "autoencoder2.compile(optimizer='adadelta', loss='binary_crossentropy')\n",
        "autoencoder2.fit(x_train, x_train, epochs=20, batch_size=128,\n",
        "               shuffle=True, validation_data=(x_test, x_test), verbose=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OyRb-DNXDBBM",
        "colab_type": "code",
        "outputId": "32bb8063-da95-4f72-f69b-4b3fa2814b90",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 267
        }
      },
      "source": [
        "decoded_imgs = autoencoder2.predict(x_test)\n",
        "print(\"input (upper row)\")\n",
        "show_imgs(x_test)\n",
        "print(\"decoded (bottom row)\")\n",
        "show_imgs(decoded_imgs)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "input (upper row)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABGoAAABsCAYAAAAyoVQIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAGxtJREFUeJzt3Xu4zVUex/F1QkXqFDlk3I9Rcs29\nhobypNyiKMM0DZEmTbq4NGVKqJ6HUpLIPKOQpFyKyFQGJZkehmPcHxp0co9OKXI5e/6Yx7fvWs5v\n2+ec/fvt39n7/frrs6x19l7TPr+99/nN+q6VFolEDAAAAAAAABLvvERPAAAAAAAAAP/HjRoAAAAA\nAICQ4EYNAAAAAABASHCjBgAAAAAAICS4UQMAAAAAABASxaN1pqWlcSRU4hyKRCLl4vFAvI6JE4lE\n0uLxOLyGCcW1mAS4FpMC12IS4FpMClyLSYBrMSlwLSYBr2uRFTXhtSvREwBgjOFaBMKCaxEIB65F\nIBy4FpMYN2oAAAAAAABCghs1AAAAAAAAIcGNGgAAAAAAgJDgRg0AAAAAAEBIcKMGAAAAAAAgJLhR\nAwAAAAAAEBLcqAEAAAAAAAgJbtQAAAAAAACERPFETwCpY9CgQZJLlixp9dWvX19yt27dPB9j4sSJ\nkr/44gurb/r06YWdIgAAAAAACcWKGgAAAAAAgJDgRg0AAAAAAEBIcKMGAAAAAAAgJNijBr6aNWuW\n5Gh7z2i5ubmeff3795fctm1bq2/58uWSd+/eHesUkUC1atWy2lu2bJE8cOBAyePHjw9sTqnuoosu\nkjxmzBjJ+tozxpg1a9ZI7t69u9W3a9cun2YHAAAQvMsuu0xylSpVYvoZ9/vQww8/LHnDhg2St23b\nZo3LysoqyBSRZFhRAwAAAAAAEBLcqAEAAAAAAAgJSp8QV7rUyZjYy510ycs//vEPyTVq1LDGderU\nSXJmZqbV16tXL8nPPfdcTM+LxLrmmmusti57y87ODno6MMZcccUVkvv16yfZLUls3Lix5I4dO1p9\nEyZM8Gl2OKNRo0aS586da/VVq1bNt+e96aabrPbmzZslf/311749L2KjPyONMWb+/PmSH3jgAcmT\nJk2yxp0+fdrfiSWZjIwMye+8847klStXWuMmT54seefOnb7P64z09HSrff3110tevHix5JMnTwY2\nJ6Ao6NChg+TOnTtbfa1bt5Zcs2bNmB7PLWmqWrWq5AsuuMDz54oVKxbT4yO5saIGAAAAAAAgJLhR\nAwAAAAAAEBKUPqHQmjRpIrlr166e4zZu3CjZXU546NAhyUePHpV8/vnnW+NWrVoluUGDBlZf2bJl\nY5wxwqJhw4ZW+8cff5Q8b968oKeTksqVK2e1p06dmqCZID/atWsnOdry6XhzS2v69OkjuUePHoHN\nA7/Qn32vvvqq57hXXnlF8pQpU6y+Y8eOxX9iSUSf9mKM/X1Glxnt37/fGpeocid9Kp8x9vu8Llvd\nvn27/xMrgi655BKrrcvp69atK9k9fZRSsvDS2yUMGDBAsi7xNsaYkiVLSk5LSyv087qnmwL5wYoa\nAAAAAACAkOBGDQAAAAAAQEhwowYAAAAAACAkAt2jxj2qWdcF7tmzx+o7fvy45BkzZkjet2+fNY76\n2sTTx/m69Zy6jlvvqbB3796YHvvRRx+12ldffbXn2IULF8b0mEgsXd+tj4s1xpjp06cHPZ2U9OCD\nD0ru0qWL1desWbN8P54++tUYY84775f/DyArK0vyp59+mu/Hxi+KF//lI7t9+/YJmYO798Ujjzwi\n+aKLLrL69J5T8I++/ipVquQ5bubMmZL1dyzk7fLLL5c8a9Ysq69MmTKS9b5Af/7zn/2fmIdhw4ZJ\nrl69utXXv39/yXxvzluvXr0kP/PMM1Zf5cqV8/wZdy+bb7/9Nv4TQ1zo98aBAwf6+lxbtmyRrP8O\nQnzpI9L1+7Ux9p6p+lh1Y4zJzc2VPGnSJMmff/65NS4M75WsqAEAAAAAAAgJbtQAAAAAAACERKCl\nT6NHj7ba1apVi+nn9JLNH374weoLcklZdna2ZPd/y+rVqwObR9gsWLBAsl6GZoz9eh0+fDjfj+0e\n91qiRIl8PwbC5aqrrpLslkq4y8vhjxdffFGyXgJaULfddptne9euXZLvvPNOa5xbRoPo2rRpI/na\na6+V7H4e+ck9pliXo5YqVcrqo/TJH+5x7E888URMP6dLSyORSFznlIwaNWok2V06r40YMSKA2Zyt\nTp06VluXis+bN8/q47M1b7oc5qWXXpKsj7w3xvt6GT9+vNXW5dwF+c6Lc3NLXHQZky5dWbx4sTXu\n559/lpyTkyPZ/ZzS30s/+ugjq2/Dhg2S//Wvf0leu3atNe7YsWOej4/80dslGGNfY/q7pvt7Eavm\nzZtLPnXqlNW3detWyStWrLD69O/diRMnCvTcsWBFDQAAAAAAQEhwowYAAAAAACAkuFEDAAAAAAAQ\nEoHuUaOP4zbGmPr160vevHmz1Ve7dm3J0eqEW7RoIfnrr7+W7HWUXl50TdrBgwcl62OnXbt377ba\nqbxHjab3oyiowYMHS65Vq5bnOF0fmlcb4TRkyBDJ7u8L15F/Fi1aJFkfn11Q+hjSo0ePWn1Vq1aV\nrI+J/fLLL61xxYoVK/Q8kplbm62PV96xY4fkZ599NrA53XrrrYE9F/JWr149q924cWPPsfr7zYcf\nfujbnJJBRkaG1b799ts9x95zzz2S9fdGv+l9aT755BPPce4eNe7+jvi/QYMGSdZHrsfK3Xft5ptv\nluwe8a33s/FzT4tkFG3fmAYNGkjWRzK7Vq1aJVn/Xblz505rXJUqVSTrvUmNic+efsibvicwYMAA\nye41dskll+T58998843V/uyzzyT/97//tfr03yF6r8RmzZpZ4/R7Qvv27a2+rKwsyfqI73hjRQ0A\nAAAAAEBIcKMGAAAAAAAgJAItfVqyZEnUtuYeq3aGezRow4YNJevlS02bNo15XsePH5e8bds2yW45\nll4CpZedo/A6duwoWR91ef7551vjDhw4IPkvf/mL1ffTTz/5NDsURrVq1ax2kyZNJOvrzRiOMYyn\n3/72t1b7yiuvlKyX78a6lNdd2qmXH+ujLo0x5oYbbpAc7ejgP/3pT5InTpwY0zxSybBhw6y2Xv6t\nl9i7pWfxpj/73N8rloIHL1pJjsstE4C3F154wWr//ve/l6y/XxpjzLvvvhvInFytWrWSXL58eavv\njTfekPzmm28GNaUiRZflGmNM79698xy3fv16q71//37Jbdu29Xz89PR0ybqsyhhjZsyYIXnfvn3n\nnmwKc7/7v/XWW5J1qZMxdulvtHJAzS130tytLeCP1157zWrrsrVoR23rewf/+c9/JD/++OPWOP23\nveu6666TrL+HTpkyxRqn7zHo9wBjjJkwYYLkOXPmSI53KSwragAAAAAAAEKCGzUAAAAAAAAhEWjp\nUzwcOXLEai9dujTPcdHKqqLRS4rdMiu9xGrWrFkFenzkTZfDuEseNf3fffny5b7OCfHhlkpoQZ6W\nkQp0mdnbb79t9UVbSqrpk7j0cs6nn37aGhet1FA/xr333iu5XLly1rjRo0dLvvDCC62+V155RfLJ\nkyfPNe2k0a1bN8nuKQPbt2+XHOQJabp8zS11WrZsmeTvvvsuqCmltOuvv96zzz1NJlrpIWyRSMRq\n69/1PXv2WH1+ntpTsmRJq62X9N9///2S3fn26dPHtzklC13KYIwxF198sWR9Soz7vUV/Pv3ud7+T\n7JZbZGZmSq5QoYLV9/7770u+5ZZbJB8+fDimuSe70qVLS3a3NtDbIxw6dMjqe/755yWzBUK4uN/r\n9GlLffv2tfrS0tIk678N3LL4MWPGSC7odglly5aVrE8fHT58uDVOb8Pilk0GhRU1AAAAAAAAIcGN\nGgAAAAAAgJDgRg0AAAAAAEBIFLk9avyQkZEh+dVXX5V83nn2fSx9bDQ1pYXz3nvvWe2bbropz3HT\npk2z2u5xtQi/evXqefbpPUpQeMWL//KWHuueNO5eTz169JDs1oLHSu9R89xzz0keO3asNa5UqVKS\n3d+F+fPnS96xY0eB5lEUde/eXbL+72OM/fnkN73fUa9evSSfPn3aGjdq1CjJqbSXUND0caI6u9ya\n/XXr1vk2p1TSoUMHq62PPdd7M7n7KcRK74nSunVrq69FixZ5/szs2bML9Fyp7IILLrDaep+fF198\n0fPn9FG/r7/+umT9fm2MMTVq1PB8DL1/ip97HBVVXbp0kfzYY49ZffrIbH1EvTHG5OTk+DsxFJj7\nXjZ48GDJek8aY4z55ptvJOv9Yr/88ssCPbfee6Zy5cpWn/7bctGiRZLdvWk1d77Tp0+X7Of+fKyo\nAQAAAAAACAlu1AAAAAAAAIQEpU/GmAEDBkjWx8e6R4Fv3bo1sDkloyuuuEKyu3RbL0fV5RZ6Wb0x\nxhw9etSn2SGe9FLt3r17W31r166V/PHHHwc2J/xCH+3sHula0HInL7qESZfQGGNM06ZN4/pcRVF6\nerrV9ipzMKbgZRUFoY9V12V0mzdvtsYtXbo0sDmlslivlSB/R5LNuHHjrHabNm0kV6xY0erTR6Tr\nJfGdO3cu0HPrx3CP3da++uorye7R0Dg3fbS2S5e3ueX5Xpo0aRLzc69atUoy32XPFq2kU39vzM7O\nDmI6iANdfmTM2aXT2qlTpyQ3b95ccrdu3axxV111VZ4/f+zYMatdu3btPLMx9vfc8uXLe85J279/\nv9UOquybFTUAAAAAAAAhwY0aAAAAAACAkEjJ0qff/OY3VtvdXfwMvQO5McZs2LDBtzmlgjlz5kgu\nW7as57g333xTciqd9pJM2rZtK7lMmTJW3+LFiyXrkxQQX+6pdZpeVuo3vaTfnVO0OQ4fPlzyXXfd\nFfd5hYV7CsmvfvUryTNnzgx6OiIzMzPPf+dzMDGilVjE49QhGLNmzRqrXb9+fckNGza0+m6++WbJ\n+iSTgwcPWuOmTp0a03PrE0SysrI8x61cuVIy34/yz31P1aVqurzQLa/Qp1d27dpVsntKjL4W3b5+\n/fpJ1q/3pk2bYpp7snNLXDR9vT311FNW3/vvvy+ZU+7C5Z///KfV1qXS+u8EY4ypUqWK5Jdfflly\ntFJQXUrllllF41XulJuba7XnzZsn+cEHH7T69u7dG/PzFQYragAAAAAAAEKCGzUAAAAAAAAhwY0a\nAAAAAACAkEjJPWrat29vtUuUKCF5yZIlkr/44ovA5pSsdP1vo0aNPMctW7ZMslt/iqKnQYMGkt36\n0tmzZwc9nZRx3333SXZrbROlU6dOkq+55hqrT8/Rna/eoyaZ/fDDD1Zb19jrPTKMsfd7Onz4cFzn\nkZGRYbW99gtYsWJFXJ8X3lq2bCm5Z8+enuNycnIkc3Rt/Bw5ckSyewy9bg8dOrTQz1WjRg3Jel8v\nY+z3hEGDBhX6uVLZJ598YrX1taP3oXH3jfHaJ8N9vAEDBkj+4IMPrL5f//rXkvV+F/pzO5WVK1dO\nsvt9QO/l9uSTT1p9w4YNkzxp0iTJ+jh0Y+w9ULZv3y5548aNnnOqU6eO1dZ/F/Jee27ukdl6f6dL\nL73U6tP7xeq9ZL/99ltr3O7duyXr3wv9d4cxxjRr1izf8508ebLVfvzxxyXr/aeCxIoaAAAAAACA\nkOBGDQAAAAAAQEikTOlTyZIlJetj3owx5sSJE5J12c3Jkyf9n1iScY/d1svGdImZSy/tPXr0aPwn\nBt9VqFBBcqtWrSRv3brVGqePu0N86TKjIOkly8YYc/XVV0vW7wHRuMfapsr7r7s0WB+5e/vtt1t9\nCxculDx27Nh8P1fdunWtti63qFatmtXntdQ/LCV1qUB/nkY7yv7jjz8OYjrwkS7ncK89XVrlvk8i\nf9yS0TvuuEOyLstOT0/3fIzx48dLdsvejh8/Lnnu3LlWny7taNeuneTMzExrXKoeu/78889LfuSR\nR2L+Of3eeP/99+eZ40Vff3rLhh49esT9uZKdW0qkr4+CmDZtmtWOVvqkS87179obb7xhjdPHfycK\nK2oAAAAAAABCghs1AAAAAAAAIcGNGgAAAAAAgJBImT1qBg8eLNk9Inbx4sWSV65cGdicktGjjz5q\ntZs2bZrnuPfee89qcyR30ffHP/5Rsj7q98MPP0zAbBCkJ554wmrrI0qj2blzp+S7777b6tNHMKYS\n/V7oHtPboUMHyTNnzsz3Yx86dMhq670wLr/88pgew63hhn+8jkh3a/tfe+21IKaDOOrevbvV/sMf\n/iBZ759gzNnH0yJ+9PHa+nrr2bOnNU5fc3o/Ib0njWvkyJFWu3bt2pI7d+6c5+MZc/ZnYarQe5TM\nmjXL6nvrrbckFy9u/+lauXJlydH28ooHvR+f/n3RR4QbY8yoUaN8nQf+b8iQIZLzs0/QfffdJ7kg\n36WCxIoaAAAAAACAkOBGDQAAAAAAQEgkbemTXiJujDF//etfJX///fdW34gRIwKZUyqI9Ui9Bx54\nwGpzJHfRV7Vq1Tz//ciRIwHPBEFYtGiR5CuvvLJAj7Fp0ybJK1asKPScksGWLVsk66NjjTGmYcOG\nkmvWrJnvx9bHz7qmTp1qtXv16pXnOPc4ccRPpUqVrLZbfnFGdna21V69erVvc4I/brnlFs++Dz74\nwGr/+9//9ns6MHYZlM4F5b5X6nIeXfrUpk0ba1yZMmUku8eJJzN9FLL7nlarVi3Pn7vxxhsllyhR\nQvLw4cOtcV5bMRSULk1u3LhxXB8b3vr27StZl5y5JXHaxo0brfbcuXPjPzGfsKIGAAAAAAAgJLhR\nAwAAAAAAEBJJVfpUtmxZyS+//LLVV6xYMcl6yb4xxqxatcrfieEsemmnMcacPHky34+Rk5Pj+Rh6\n+WN6errnY1x66aVWO9bSLb1Ec+jQoVbfTz/9FNNjJJuOHTvm+e8LFiwIeCapSy/FjXb6QbRl95Mn\nT5ZcsWJFz3H68XNzc2OdoqVTp04F+rlUtW7dujxzPHz11Vcxjatbt67V3rBhQ1znkcquu+46q+11\nDbunJqLocd+Df/zxR8kvvPBC0NNBAN555x3JuvTpzjvvtMbprQHYmuHclixZkue/61JhY+zSp1On\nTkl+/fXXrXF/+9vfJD/00ENWn1c5KvzTrFkzq63fH0uXLu35c3pLDX3KkzHG/Pzzz3Ganf9YUQMA\nAAAAABAS3KgBAAAAAAAICW7UAAAAAAAAhESR36NG7z2zePFiydWrV7fG7dixQ7I+qhuJsX79+kI/\nxrvvvmu19+7dK7l8+fKS3frfeNu3b5/VfuaZZ3x9vrBo2bKl1a5QoUKCZoIzJk6cKHn06NGe4/Tx\nr9H2l4l175lYx02aNCmmcQie3t8or/YZ7EnjH73PnuvQoUOSx40bF8R0EGd6nwT9HcUYYw4cOCCZ\n47iTk/6c1J/Pt956qzXuqaeekvz2229bfdu2bfNpdsnno48+str6u7k+yrlfv37WuJo1a0pu3bp1\nTM+VnZ1dgBkiFu5ehhdffHGe4/Q+X8bY+0B9/vnn8Z9YQFhRAwAAAAAAEBLcqAEAAAAAAAiJIl/6\nlJmZKblx48ae4/Sxy7oMCvHlHn3uLumMp+7duxfo5/SxfNFKNubPny959erVnuM+++yzAs2jqOva\ntavV1mWIa9eulfzpp58GNqdUN3fuXMmDBw+2+sqVK+fb8x48eNBqb968WfK9994rWZcnIlwikUjU\nNvzXrl07z77du3dLzsnJCWI6iDNd+uReXwsXLvT8Ob3U/7LLLpOsfydQtKxbt07yk08+afWNGTNG\n8rPPPmv13XXXXZKPHTvm0+ySg/4eYox9PPodd9zh+XNt2rTx7Dt9+rRkfc0+9thjBZkiPOj3vCFD\nhsT0MzNmzLDay5Yti+eUEoYVNQAAAAAAACHBjRoAAAAAAICQ4EYNAAAAAABASBS5PWqqVq1qtd3j\n185w92fQx9HCP7fddpvV1rWFJUqUiOkx6tSpIzk/R2tPmTJF8s6dOz3HzZkzR/KWLVtifnwYU6pU\nKcnt27f3HDd79mzJuqYX/tq1a5fkHj16WH1dunSRPHDgwLg+r3sk/YQJE+L6+PDfhRde6NnHXgj+\n0Z+Les891/HjxyWfPHnS1zkhePpzslevXlbfww8/LHnjxo2S7777bv8nBt9NmzbNavfv31+y+516\nxIgRktevX+/vxIo493ProYcekly6dGnJTZo0scZlZGRIdv+WmD59uuThw4fHYZY4Q78mmzZtkhzt\nb0d9DejXN5mwogYAAAAAACAkuFEDAAAAAAAQEkWu9Ekf9WqMMVWqVMlz3PLly602R40mxujRowv1\n8z179ozTTBAPesn9kSNHrD59nPm4ceMCmxPy5h6Lrtu6ZNR9T+3UqZNk/ZpOnjzZGpeWliZZL1NF\n0dS7d2+r/d1330keOXJk0NNJGbm5uZJXr15t9dWtW1fy9u3bA5sTgte3b1/J99xzj9X397//XTLX\nYvI5ePCg1W7btq1kt/Rm6NChkt0SOUS3f/9+yfp7jj7y3BhjWrRoIfnpp5+2+g4cOODT7HDDDTdI\nrlSpkuRof7/rslBdHpxMWFEDAAAAAAAQEtyoAQAAAAAACIm0aEuK0tLSQlEv1LJlS8mLFi2y+vQu\n0VqzZs2strukuAhYE4lEmpx72LmF5XVMRZFIJO3co86N1zChuBaTANdidAsWLLDaY8eOlbx06dKg\np+Mlqa/FihUrWu1Ro0ZJXrNmjeSifqpaql6L+rusPr3HGLs0deLEiVafLjM+ceKET7PLt6S+FsPC\nPdn22muvldy8eXPJBS0/TtVrMckkxbWYlZUluV69ep7jxowZI1mXAhZ1XtciK2oAAAAAAABCghs1\nAAAAAAAAIcGNGgAAAAAAgJAoEsdzt2rVSrLXnjTGGLNjxw7JR48e9XVOAAAkC31cKRJjz549VrtP\nnz4Jmgn8sGLFCsn6KFrAS7du3ay23sejZs2akgu6Rw0QFmXKlJGclvbLdi3ukegvvfRSYHMKA1bU\nAAAAAAAAhAQ3agAAAAAAAEKiSJQ+RaOXAd54442SDx8+nIjpAAAAAEChfP/991a7evXqCZoJ4K+x\nY8fmmUeOHGmN27t3b2BzCgNW1AAAAAAAAIQEN2oAAAAAAABCghs1AAAAAAAAIZEWiUS8O9PSvDvh\ntzWRSKRJPB6I1zFxIpFI2rlHnRuvYUJxLSYBrsWkwLWYBLgWkwLXYhLgWkwKXItJwOtaZEUNAAAA\nAABASHCjBgAAAAAAICTOdTz3IWPMriAmgrNUjeNj8TomBq9hcuB1LPp4DZMDr2PRx2uYHHgdiz5e\nw+TA61j0eb6GUfeoAQAAAAAAQHAofQIAAAAAAAgJbtQAAAAAAACEBDdqAAAAAAAAQoIbNQAAAAAA\nACHBjRoAAAAAAICQ+B91hS6/T+74OwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 1440x288 with 10 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "decoded (bottom row)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABGoAAABsCAYAAAAyoVQIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3WmQVdXVxvFFNEoEZ5FJVBBFENEo\ngzOoMU5Y0WjAUhONE8FowIoRnKtCSmJicIqWY9RUSCliKShqNA4MEhGRQUURHAAFEQWRQRwS3g/W\nu3zWsu+lhe7m9O3/79PT2cfmps/d55x7a6+9Gq1Zs8YAAAAAAACw4X1vQ78AAAAAAAAAfI0vagAA\nAAAAAAqCL2oAAAAAAAAKgi9qAAAAAAAACoIvagAAAAAAAApi43KDjRo1oiXUhvPRmjVrmtXEL+I8\nbjhr1qxpVBO/h3O4QTEXKwBzsSIwFysAc7EiMBcrAHOxIjAXK0CpuciKmuKau6FfAAAzYy4CRcFc\nBIqBuQgUA3OxgvFFDQAAAAAAQEHwRQ0AAAAAAEBB8EUNAAAAAABAQfBFDQAAAAAAQEHwRQ0AAAAA\nAEBBlG3PDdSk733vm+8FmzZtGsa+//3ve169erXnxo0bh+N0bNWqVWFszRq6ytV3G2/8zSXpf//7\nX5UZAAAAACoZK2oAAAAAAAAKgi9qAAAAAAAACoLSJ9SoRo0ahZ+1dKlHjx6eBw4cGI7bdtttPS9a\ntMjzF198EY6bPHmy53vvvTeMLV261DNlUPWDlryZmXXv3t3za6+95vmTTz6ps9fU0Okc1lK0PLe/\n/PJLz8w3AACA8vKzVHXwjNVwsaIGAAAAAACgIPiiBgAAAAAAoCD4ogYAAAAAAKAg2KMG603bbrdr\n1y6MDRgwwPNRRx3lebvttgvHff7555533nlnz7k9d8+ePT23bds2jA0ePNhzbt2N4tD63HwOhw4d\n6vnSSy/1PGHChNp/YQ2InoMmTZqEsb59+3o+66yzPOt+NWZmQ4YM8TxmzJgwRjv1DUvPL7XtDYve\nj83ivXb58uWeP/vsszp7TZVok0028azPKStXrgzH/fe//62z16TK7YPBNQGonjyP9Pqq836LLbYI\nx7Vq1cpz69atw9hee+3leeHChZ7nzJkTjnv55Zc9r1ixIozxjNVwsKIGAAAAAACgIPiiBgAAAAAA\noCAofcJ603KnESNGhLEOHTp4Xr16tWdd7mdm9uabb3redNNNPXfp0iUcp8sLTzjhhDA2cuRIz+PG\njavWa0fd05bcZ555Zhjr3Lmz56222spzXn7K0u31o2VM++yzTxjTkrMdd9zRcz4H119/vecpU6aE\nsQULFtTI60RUqnV6bnOvZRla4vLVV1+V/N15Tm200Uae9bp7wAEHhOP0Wj5t2rQwVu7fQ83R5fjn\nnXdeGBs0aJDnq6++2vNtt90WjmMp/beVK9P9+c9/7nnu3LmeR40aFY5btmyZ55r+G+cyt2bNmnnO\nz0dvv/2252eeecYzc/Qber5z2X2pa+oXX3xR+y8M6yQ/s2yzzTae9T622WabhePat2/vWZ9Dzcx6\n9OjhuU2bNp71fpn/bX3v5N+ppZGLFy8Ox5177rmen3322TD25ZdfeuZ5uLKxogYAAAAAAKAg+KIG\nAAAAAACgIOq09Ckv0yy3XIulXMWVz2OvXr08a6lEPlZ3NL/uuuvCca+++qpnLZHSTlFmZpdffrnn\nbbfdNoydfPLJnsePH++Z91KxaJchfe+YxeWjH374oWfOYc3S8sJ8DrbeemvPunw3L9vXZcS/+MUv\nwtiwYcM8szS85ugSal12nTty6dzR0obvUnqhc06vteeff344btasWZ5fe+21MEZZRd3YYYcdPGvn\nPDOzpk2bej766KM959InfJuWRNxwww1hrGvXrp7vvfdezw888EA4rjbvXVtuuWX4+eGHH/asZcRm\nZk899ZRnfT5ijn5D/556DzMz22+//Txreds111wTjvvkk09q6dWhOvQzR/48omXdhx12mOdcmqTy\n84uWF+pzVD5O3wf6mcYszjktW9ZrtVm8/uTPXVi7UqXiZvGc63ORdiDOYxsKZx4AAAAAAKAg+KIG\nAAAAAACgIPiiBgAAAAAAoCBqfY8arbk77bTTwljHjh095/ZoWtc7YcIEz7n956JFizyvXLnSc64r\n0/q+XOun/7bWC2rbtPz7ly9fHsaKUMdWV/L/10ceecSz1sqbxRaHd955p+fcvrfUPhYPPfRQ+Llv\n376ec+vufL5QDLlForY5bdmyZRjTvTXefffdWn1dDY3u/9OpUyfPeX8ZvR5qLbW2gzSL5/Xss88O\nYzvttJPnIUOGeKZt93eT71XNmzf3rHsm6LwxM5s/f77ndb0u6n+3dOlSz6tWrQrHbb755iVfL2pH\n/jufeuqpnnUPsOyll17y3JCeWdaV7kNz8MEHhzFt0fzoo49W+b+b1fweNXrdPfLII8PYPvvs4zk/\nU919992e854ZDZnOpf3339+z7nloZvaDH/zAc6tWrTx/+umn4bhbbrml5Bh77dUO3X9EnymPPfbY\ncJzuq/f66697zudp7ty5Jf8tbeutn2/zZ9OxY8d6znNx11139ax7AupnXTOzxx9/3HPeO6Uhv5f0\nc/p2223nWVunm5kNGDDAsz7zmsW/58SJEz3rXl5mZk888YTnhQsXhrG6uofyVAUAAAAAAFAQfFED\nAAAAAABQEDVS+pRLG7TtlS4X1WVIZmY777yzZ21zln+nLs3X8iMzs2XLlnnWcqS8/FSX7edlaLpU\nTpfG5f9fM2fO9NyvX78wpq2nG9qS4o8++sjztddeW/I4XW5b7m+kJRq5NEbPTz7Hs2fPXvuLxQan\n14S8hP++++7zvGTJkjp7TQ2BLrEdPHiwZ13GbRZLXlasWOE5L73VearLT83iNfvAAw/0fPzxx4fj\n3nnnHc8NeSlvKbltqC7l1lbLuSVwTZSB6vnQpeV63zaL5Wz5nonaka+bJ554oud8DvS9oG2F8W16\nTTMzGzhwoOdcnj9lyhTPr7zyiufafv7Tuaglb2bx2VZL0s3MnnzySc9ca7+h5fnnnXeeZy11yvS6\nnLd00PfJ8OHDw9hbb73lOZcSo/r0nJmZHXTQQZ579uzpOX+uHD16tOdZs2Z51i00zOJzT573Y8aM\n8ayfQd5+++2SvyPPN71G61i+djTkearlTVpKb2Z2+umne+7du7fn/GyiZdn5b6nt01u3bu05l/Fr\neekdd9wRxiZNmuQ5l8/VJFbUAAAAAAAAFARf1AAAAAAAABQEX9QAAAAAAAAURI3sUZNrv7Sl6wsv\nvOD5V7/6VThO6+333nvvMLb99ttX+W/lulGtFdU9GHJtv9bu6r42+VitadMaObPYTnyXXXYJY1qf\n2ND2qNEa+LyHkNbSl6u31HOgLdaGDRsWjtP23/PmzQtj48aNq9a/hbqV52KfPn085zahWtOt1xF8\nd/nvftlll3nW/U3ycVpbre0I83VTr6k6L/PP7du39/zYY4+F437zm994/ve//x3GGtp19P/pNbND\nhw5h7IwzzvC85ZZbei5XA1+qHn5tdC+Mk046ybPeB7OGes7q2rbbbht+zu8TpbX45drOwqxFixbh\nZ91fK+/79PDDD3vO++XVJm0h3b179zCm1+srr7wyjOU9xhqqvIeTtlvWvU7ytVLvix988IFn/dxh\nFj/n6PXazOyGG27wrM+2XDe/LZ8n/UzYv3//MKb7iLz77rueb7311nCcttBetWqV53yu9ee894ju\nyanPQDyvfnd5r7XmzZt71n1ofvnLX4bjdM7p5/R8jdb7ne6HaBZbcuv57tWrVzhOr7f5Pnv77bd7\nvuuuuzznzzXrixU1AAAAAAAABcEXNQAAAAAAAAVRI6VPmS4/Wrp0qeexY8eG48aPH+85L3PTNona\nYk1Lk/LP2iI2/z5dtpiXsmnrrzPPPNPzYYcdFo7Tpf/z588PYyxd/FpeQqh/Fz2nTZs2Dcdp294h\nQ4Z41nbcZrEVbF7+qO3TURydO3cOP2vphJarmX27nA3fjV738rL4s846y7O2t8zXLl0S/Prrr3ue\nPn16OE7nW9u2bcPYcccd57lbt26etQzKzOzOO+8s+Xp1eXlDUqqNuplZu3btPM+ZM8fzjBkzwnHr\nsgw73zObNGni+ZhjjvGsJVFmZkuWLPFMy9m6oc8pZrGFbD73V111lefabCFaX+n7XktTzOLzR37m\n0/bXtV32oG1ndYl9fo66/vrrPWsJCL6R2y3r/NAx/exiZjZy5EjP2po9l2XoNg65LKpv376e//rX\nv3qu6VKJ+krnYi4zGTp0qOdOnTqFsTfeeMPzX/7yF89a6mRW/XlaqnTYLN7j+Ny3dvnvp5/Zjzji\niDB2/vnne95tt9085xIpbXOvzz76nYKZ2Ysvvui53BzbZ599PB911FFhTL9XyJ9HDz74YM8jRoyo\n1r+1LlhRAwAAAAAAUBB8UQMAAAAAAFAQtVL6VF3llo1p+ZTurK0lTGZxebwuh8rLs1Uuz9Gl/rNn\nz/asu8GbxdIn7aRQ1e/E13TZm3Yp0Y5fZmYXX3yxZ+1okZefXnLJJZ61o5gZu64XiS5V1HNmFju3\nPf3002GMzhTrR3fAHzBgQBjTUhal1z+zuITz/vvv9/zhhx+G4/Ra/Oqrr4YxXXI8cOBAz9ptyiwu\nK+3Xr18Y0xLISl5inJf16rXx8MMPD2NauqJlDlqiZhavhet6b9KyimbNmlX5u83iuc5dF1BztFzx\nnHPOCWP6Hsrd2XQOV/I8Wlfa9a5r165hTJ9ftNuLWfyba1l3dedbLgnQa7eWOpnFkiztfqPPvGax\nlJTnoW/o3zqXYu+5555V/je5bOaWW27xrHNMt04wiyU7uXOsfi4p9xmlodJukQ8++GAY0/d9fva/\n8MILPWv5/LrOAZ3D5bblYI6tXe7U/Otf/9pzLhvUz37Lly/3/MQTT4Tj/va3v3nWbk75fOh5zNum\naBm5bnOSr706h/W7CLPY7bk23wusqAEAAAAAACgIvqgBAAAAAAAoCL6oAQAAAAAAKIh6VySZ63/1\nZ83larHzngBag9alSxfPWrtsFusi2aOmerSeU9vv5hr7Fi1aeNa9L2699dZw3OOPP+6Z+tDi0jaX\nBx54YBjTdr7a8tKMPS7Wl9Zx77fffmFMa611L6AxY8aE44YNG+ZZ96XJ56bcNfa9997zrPP0kEMO\nCcdttdVWnvMeNdddd53nSm4rnPcq+NnPflZybNSoUZ61PXBNtMXO9zC9L+peQnkfqUcffbTk70DN\n0Xaler80i3/3CRMmhDG93uLb9Lr4yiuvhDFtx7rjjjuGsUGDBnm+7777POueCWbx3Og+Q7m1tu6V\n8pOf/CSM6b4nH3/8sWfdmyOP4Rt6jvfff/8wps+oixYt8nzTTTeF43Q/Cj2n2gLYLO4xlvevmTVr\nlmf2i/qa3uOuvvpqz+3btw/HaQtubeNsZjZ//nzPNfF3LdeeO/+M8nbdddfw8xlnnOG5ZcuWYUw/\nm+sz5KRJk8JxOqZ7LOr11cxs66239pz3nNX9ErU9d95T57PPPvOc960aPny459p8RmVFDQAAAAAA\nQEHwRQ0AAAAAAEBB1LvSp3JKlUGZxSVVm266aRjr1auX527dunnOS4bvuecez7ocCqW1bt3a8+DB\ngz137NgxHKdL98eOHev5oYceCsdpGUW5JYgswa97ej46derkWduym8Vzmls+Y/3o3z23I9S5M2PG\nDM+///3vw3Hvv/++53UtRdPlx1ommpcl63smlwLo+6aSS5/ycl1tD5nvM88884zn1atX1+jryNdT\nLcXQtpkrV64Mx+l7ietuzdKyDC2HyW1/9f6pJYNmlFisjZby9e/fP4xdeumlnk888cQw1qdPH8/H\nHXec51yGqHNYx/K9T8uFmzdvHsb02v3ss896/s9//hOO41xXTa9Luc26ltq/9tprnrWcxizORc1a\nvmsWW/jma7T+rL+jIdNy7WOOOcZz3qJCnxsXLFgQxmr6fa//dj5P+kzE/W7t8r1KnzPy31Z/1rJT\nbZ9tFlt3azv23FpbP8/nLRj0WH1Wzu+lmTNnetbSPDOzd99913NtbtvAihoAAAAAAICC4IsaAAAA\nAACAgqio0qdydElV7oZyySWXeNayKF32amb26quveqY7TdXycsVDDz3U8x577OE5dzOZOnWq5xtv\nvNGz7sJvVv3d2Dk/dU/n2Kmnnuo5n6cRI0Z45jzVLC19ynNx6dKlnnWO5S4lpc7Jd1nmW2p5q+7Q\nbxaXjedOerm7UKVq0qRJ+FlLofLfZOHChZ7XpftEuWvmFltsEca0s4Z2QtDuh2Zmy5Yt+86vA9Wj\n74Ujjjii5HE6t6dPn16rr6nS6HVNu4mYmf32t7/1/Nxzz4Ux7VK39957e9ZOI2alu+1leh3IZaBz\n5871/M9//tNzTZc/Vio9x/l6pfckLTHLZRQ6pvct7RhjFu93uTOplpBq1lKOhkbvO/q+z58RdBuF\nvH3F+j6z5PIcncN5jun7Rec2ZVBV09Ihs/jsOWDAgDCm3SX1nGrHQzOz0047zbOWMuYyRL0u53JS\nfQ/pvzV79uxw3EUXXeR5ypQpYayurr+sqAEAAAAAACgIvqgBAAAAAAAoCL6oAQAAAAAAKIiK3aMm\n1+K3aNHC88UXXxzGmjVr5nn06NGen3rqqXAc+2msXW4127t3b89ac6p7LZiZ3XHHHZ6nTZvmObe6\n1NZp+RzrnhxaL0rLytpRbo4dfPDBnnP99csvv1y7L6wByfvQdOnSpeSxr7/+uufx48d71naiWXXr\nrvN7Qee6vhfyPij63+X9WBrK/gv5b6x/k7x/zQEHHOBZa7Pz3j9ac6/3Ld1rxizW/e+7775hrGPH\njp61xfBtt90Wjsv7MGDdlbum6p4Z+T2j19RKbmVf1/R9r3urmZk9+OCDntu0aeM5t4HVPRFbtmzp\nOT8DHXTQQZ7z++DJJ5/0rPsk8Gzz3eV9LPSZtdQzjFnpPVJ0Xw2z8p8Tunbt6nnQoEGeL7zwwnBc\nQ7n3mcU26NruuF27duG4H/3oR57z3iZvvfWWZ92DKD976nPJJpts4ln3MjGL+ww9++yzYWzWrFme\nV65c6TnfB5mbX8vPJrfeeqvnvO+X7mm6YsUKz3nPLp1/ek71mcUsPu/o+TaL50ffdxdccEE4Tvfk\n21DPOqyoAQAAAAAAKAi+qAEAAAAAACiIii19yu3btNVojx49wpgulbv99ts957IbrF0uvdAlhbok\n++GHHw7HTZgwwfO6LvvU5cJaEsISxNqRy260dEKXF2tbezOzxYsX1+4La0DyEvmddtqp5JjOKy13\nKlfepL+jXDvoPNa+fXvPp59+uud8XS5XWqDLiiuZtlY2i6UNhxxySBjTv+XRRx/tOZc06fJ7LbEo\n1y42t6/cbLPNPOucHTduXBX/L1ATdMm9WemywdzmWcvRKNGuG/p31qXz8+bNC8c99NBDnrfffnvP\nHTp0CMfptfvDDz8MY4888ohnvWbiu3vnnXfCz3rutDyibdu24bjNN9/csz7L5na+em/N98Xdd9/d\n84knnuh51KhR4bgnnnii5OuvNHqf19IXLQ0zi6WBAwcODGNaIqyf2/KzjY6VymaxrFjL1czMJk6c\n6Fm3aZg8eXI4TsuuaN39DX0G0XJ8M7O5c+d61pKm/HyjzyatWrXynD9/6pzNnwO15O6MM87wnM9j\nET4/sqIGAAAAAACgIPiiBgAAAAAAoCD4ogYAAAAAAKAgKmqPGq3v7t69exg77bTTPGvrUrNYD/rG\nG294pq6werQO98gjjwxjW265pWfdC+j9998Px2ndov6+XB+o5yTvkVJqj5pc96+/s1xr3FL/Lr6m\nNaRmsZZXx1588cVwHHs/1Zw8B7R2N1/ntD5+jz328JzbYuv50bmTz7f+/tyi9JZbbvHcuXNnz3l+\nzZkzx/O9994bxopQG1wXcnv0IUOGeL7yyivDmO5xoe2atRbbLNb9654Wb775ZjhO9zrR945Z/Pvr\n78jtNlFzcgtR3aNI52Lew2Ts2LGeuVdtWPm6pXNHn4G0VbdZvIbmvRt0PwX2IFo/kyZNCj+fcsop\nnvWc5OdGpffMvMeYthLWa7SZ2RVXXOG5U6dOni+77LJwnD4zLVmypOTrqDQLFizw/Lvf/S6MdevW\nzfPNN98cxrQtsz6n5HOoc0dz3rtNP7fofntmZj179vSs83nYsGHhuLvuustz3lOMa/TX8rVMn1v0\n2Tbvy6XPILqXVN5PSO+n+t4yi89Wui9gEZ87WVEDAAAAAABQEHxRAwAAAAAAUBAVVfrUuHFjz/36\n9Qtjuqw0t+cbOnSoZ22phurRJWrHHntsGNMlhCtWrPBcrvVuuTbA1X0dmsu1rs3L3LbZZhvPO+yw\ng+dcHqJLz7VVo9m3l1FWEj032i7WzOzwww/3rH/zvIybZZ81p9xcyWVR2opZ52k+P7rUWpeO5vbN\n2o69f//+YWz//fev8nXkZdxXXXWVZ20jbdZw3if5/+fMmTM9a8muWZxXWlLWunXrcJyWlmrOZYe6\nbDifQ72f5msoao7OYZ2jZmY9evTwrPNIl2qb8dxSZHp+9RqqJTdmsT331KlTw1guncC6y9dAbQn8\n3nvvec73T1Xd8nktWTOLJU5//OMfPee2wpdffrnniy++OIxV8vOlyiXBL7zwgudjjjkmjGm77r59\n+3rW53mzOI/0HObyNf28mJ9ztbRKr9dnn312OG7kyJGec6kqqlaq7Cj/71peqPMjlxrqZ70//elP\nYWzUqFGeiz6nWFEDAAAAAABQEHxRAwAAAAAAUBD1vvRJlxnqju2HHnpoOE6XJ1533XVhbN68eZ6L\nuONz0elSwBYtWpQca9asmecTTjghHKdlD9oJZvXq1eE4XbqYlzW2adOmyrz33nuH4/S/y0v6dSf+\nrbfe2nNeWn733Xd7vv3228PY/y+jrMT3ks63Dh06hLFdd93Vs/5/nz59ejiuEv8uG0reNf+pp57y\nrO9ls/heP/744z3nZbmPPfaYZ10C3KdPn3DcYYcd5lmX7ZvFea9LmO+5555wnHbcK/ry07qi8yMv\n/9afn3/+ec/VLRfNx2nZpnY8NDM77rjjPJcrF8X60RKL3GFE75l67vX+Y0YnoPpCy6l32223MKal\nhrnDXrkyHNScUl2B1lXuVjNx4kTP2tUo3xf1WUrLPMy+XYbfUOj5yB18tKRs9OjRnnN54VZbbeVZ\nn49yJ0P9m+fSJ6WfK7WErqrfiXWXO5hqd+H99tvPc75OTp482XO+Z9anclKu/gAAAAAAAAXBFzUA\nAAAAAAAFwRc1AAAAAAAABVHv96jR9rEnn3yyZ91bwSy2KL3vvvvCGDX360frNHP7Xd0rZtNNN/Xc\nq1evcNyee+7pWdt455pDHcu1o6XqSvXfNTPbaKONqnztZnEPBx3T+nGzWN+a97nJrf4qidbOaztu\ns7j3z8cff+w5ty9Hzcl19DfffLPn3MJyr7328tyqVSvPF110UThOr6NaG6xz2Sy+73VOmcX9Zm66\n6SbPV1xxRTgu78GCdbOurcz13qf7Z5iZbbbZZp7z9Q81R+dOvi/qHNNr6rRp02r9daFm6NzUvUdy\nK/ZyLZ9RGfS+OGPGDM+6T6ZZvD+3bt06jDXUPWrK0Zbr2sb7gw8+CMf17t3b8w9/+EPPeQ8Uve7m\n/aJ0burv/8Mf/hCOW7lyZZX/DapH/+7du3cPY9dcc41nfU5ZvHhxOO6kk07yXJ/2pMlYUQMAAAAA\nAFAQfFEDAAAAAABQEPWu9CkvUdtjjz08ayu2XDLzwAMPeM6tlrF+tHzh3HPPDWMjR470rKVJ+fzo\nMuDmzZt7zu1kdYljpsfqcvL83+jPuX2ilsi99NJLnrWtollsjbto0aIwVsmldFpqqHPPLJ5TXfaZ\nW6yj9mir7auuuiqMaXvCJk2aeN58883DcbvvvnuVvzuXN+n7PM+jP//5z56HDh3qudz8Re3LS7DL\ntQLX66met+q2Akf16DNN586dw5gu/9YSUr2+otj0vqjnN5dkaxlrixYtwlguv0D9p3P4/vvvD2NH\nHHGEZ71Xm5Uuz8fXtLzsvffeC2OvvPKKZ32eyS3Qy31+0M8I/fr18zx16tRwXCV/Dqgt+t7W0rS/\n//3v4Tj9vKjnW1vem5ktXLiwpl/iBsGKGgAAAAAAgILgixoAAAAAAICC4IsaAAAAAACAgqgXha9a\nt9a2bdswpu1etY2dtrI0Mxs2bJhnagdrz4svvhh+7tixo+du3bp5zvubaGtYPce5BfdHH33kObfB\nXrVqlecFCxZ4zrWjWreYf4fuX6S1j+U0pDphraPXVulm8W85bty4kseh9uh78cknnwxjl156aZU5\nt2UutR9Cbm84adIkz/379w9jb775pmeut8Wl9fd57zbdW0P308j7i2H96H4ILVu2DGM6n3X/qere\nm7Dh6fndZZddPOf9FvW6265duzCme8OhMuh98emnnw5jBxxwgOdDDjkkjE2bNs1z3lcMUd5fZs6c\nOZ71eXW77bYLx+nc1P/GzOynP/2p51mzZnluSJ8Daos+Z+gzar4v6tx5/PHHPee9nioFT1wAAAAA\nAAAFwRc1AAAAAAAABVEvSp8aN27sWduhmZn17Nmzyv9m+PDh4WddNoy6oy0In3vuuSoz6g9davvg\ngw+GMZ1jd955p2dKnzaMvOz3rrvu8jxmzBjPffr0CcftueeenrUl8G233RaOmz17tmfKm+onPW/z\n5s0rOfbBBx94Zrl9zdK/5/z588NY+/btPY8fP77K/wbFpqWCek3O10z9ecmSJWFMS45RefJWDdts\ns43n008/PYxNmTLFs14TKIf8tjxv9B7Xu3dvz/vuu284bvvtt/c8evToMPbJJ5/U5EuEOPzwwz1r\ni3otHzWLnykuuOACz5V6X2RFDQAAAAAAQEHwRQ0AAAAAAEBBNCq3U3WjRo0KsY317rvv7jl3MmnT\npo1n7fSjS6jMzN54441aenW1ZsqaNWu61sQvKsp5bIjWrFnTaO1HrV0Rz6Hu0G4Wd73XJd4VsBs+\nc7ECVPJcXFfaUVHvpWZm116wo6ASAAACo0lEQVR7recbb7zR8/PPPx+Oq+P5XXFzUUtjunTpEsa0\nDPFf//qX58WLF4fj6ts1tiHNRZ1j2vXpnHPOCcdpmemoUaPC2FtvveV59erVnjfwea+4ubih5A5g\nZ599tmct7TCL3RYHDRrkeV23d2hIc7GC1cu5mEuaJk6c6Ll79+6e83VuxIgRnk855RTP9b0Ev9Rc\nZEUNAAAAAABAQfBFDQAAAAAAQEHwRQ0AAAAAAEBBFLY9t9Zt//jHP/asbdOyZ555xvPcuXNr54UB\nMDOzzz//fEO/BADrQWu/c3turf3WNqf1bT+UotO6+unTp4exGTNmVHkc6g+dL3PmzPE8ePDgav03\nqHy6p5+Z2T/+8Y+Sx7Zt29azfh7KLb71esH7CUXUpEmT8HPTpk09a6vtqVOnhuPOPfdczw3hvsiK\nGgAAAAAAgILgixoAAAAAAICCKGzpk9JlgcuWLQtjulz7iiuu8KzLpgAAQPV99dVXG/olNDi5RIGS\nhcrFuUUpK1as8JzLoNq0aeN5+fLlnjfeOH6c089NvNdQRKtWrQo/n3rqqZ4bN27sOZc+NbRtF1hR\nAwAAAAAAUBB8UQMAAAAAAFAQfFEDAAAAAABQEIXdo0Zbbg0fPtzz5MmTw3EzZ870nOvdAAAAAKA+\n0D1ldL8aM7PZs2dX+d/kPcXYlwZFl9+z06ZN20CvpNhYUQMAAAAAAFAQfFEDAAAAAABQEGsrffrI\nzObWxQsp59NPP/X80ksvbcBXUqd2qsHfVYjz2ABxDisD57H+4xxWBs5j/cc5rAycxzqQS5i07XYN\n4BxWBs5j/VfyHDaijhEAAAAAAKAYKH0CAAAAAAAoCL6oAQAAAAAAKAi+qAEAAAAAACgIvqgBAAAA\nAAAoCL6oAQAAAAAAKIj/AzCeX77bPf+IAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 1440x288 with 10 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SYNuOKadDk_P",
        "colab_type": "text"
      },
      "source": [
        "###Questions 4.2\n",
        "1. What is the difference between 'mse' loss and 'binary_crossentropy' loss?\n",
        "2. Can you explain why 'binary_crossentropy' works better?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LEU1ZDqg6bk9",
        "colab_type": "text"
      },
      "source": [
        "##Task 5: Denoising Autoencoder\n",
        "Autoencoders can get really advanced, like [Variational Autoencoders](https://towardsdatascience.com/intuitively-understanding-variational-autoencoders-1bfe67eb5daf). A slightly less complicated, yet powerful autoecoder variant is the [Denoising Autoencoder](https://www.cs.toronto.edu/~larocheh/publications/icml-2008-denoising-autoencoders.pdf).\n",
        "\n",
        "As stated above autoencoders have many useful applications. One of these is *noise reduction*. The underlying idea is very simple: Add random noise to the input X, and teach the autoencoder to remove the noise. That is, the autoencoder should learn the mapping:\n",
        "\n",
        "```\n",
        "X = AE(X + noise)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MnzL99s2IYtR",
        "colab_type": "text"
      },
      "source": [
        "###Task 5.1\n",
        "Create two new data sets based on x_train and x_test, where you have added noise such that\n",
        "\n",
        "```\n",
        "x_train_noisy = x_train + noise\n",
        "x_test_noisy = x_test + noise\n",
        "```\n",
        "\n",
        "You may want to look at numpy functions like np.random.normal and np.clip\n",
        "\n",
        "Retrain autoencoder2 on the noisy images (input = x_train_noisy and output = x_train).\n",
        "\n",
        "How much noise can you add before the thing breaks down?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wD06QF9EwDbp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Your code goes here\n",
        "#x_train_noisy = ???\n",
        "#x_test_noisy = ???"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "outputId": "17e10560-d9e4-4cf6-d22c-228356e411ac",
        "id": "ZNC70CzX6YaE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        }
      },
      "source": [
        "# it takes more epochs to converge\n",
        "autoencoder2.fit(x_train_noisy, x_train, epochs=10, batch_size=128,\n",
        "                shuffle=True, validation_data=(x_test_noisy, x_test))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Train on 60000 samples, validate on 10000 samples\n",
            "Epoch 1/10\n",
            "60000/60000 [==============================] - 3s 58us/step - loss: 0.1424 - val_loss: 0.1325\n",
            "Epoch 2/10\n",
            "60000/60000 [==============================] - 3s 57us/step - loss: 0.1310 - val_loss: 0.1266\n",
            "Epoch 3/10\n",
            "60000/60000 [==============================] - 3s 57us/step - loss: 0.1286 - val_loss: 0.1253\n",
            "Epoch 4/10\n",
            "60000/60000 [==============================] - 3s 57us/step - loss: 0.1273 - val_loss: 0.1264\n",
            "Epoch 5/10\n",
            "60000/60000 [==============================] - 3s 56us/step - loss: 0.1268 - val_loss: 0.1266\n",
            "Epoch 6/10\n",
            "60000/60000 [==============================] - 3s 57us/step - loss: 0.1260 - val_loss: 0.1239\n",
            "Epoch 7/10\n",
            "60000/60000 [==============================] - 3s 57us/step - loss: 0.1257 - val_loss: 0.1245\n",
            "Epoch 8/10\n",
            "60000/60000 [==============================] - 3s 57us/step - loss: 0.1252 - val_loss: 0.1220\n",
            "Epoch 9/10\n",
            "60000/60000 [==============================] - 3s 57us/step - loss: 0.1246 - val_loss: 0.1254\n",
            "Epoch 10/10\n",
            "60000/60000 [==============================] - 3s 56us/step - loss: 0.1244 - val_loss: 0.1210\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<keras.callbacks.History at 0x7f23d9bab908>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ynvpaB666wlm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# denoising\n",
        "print(\"denoising\")\n",
        "decoded_imgs = autoencoder2.predict(x_test_noisy)\n",
        "print(\"input (upper row)\")\n",
        "show_imgs(x_test_noisy)\n",
        "print(\"decoded (bottom row)\")\n",
        "show_imgs(decoded_imgs)\n",
        "\n",
        "# what if we feed the original noise-free test images?\n",
        "decoded_imgs = autoencoder2.predict(x_test)\n",
        "print(\"\\nof course, it works with original noise-less images\")\n",
        "print(\"input (upper row)\")\n",
        "show_imgs(x_test)\n",
        "print(\"decoded (bottom row)\")\n",
        "show_imgs(decoded_imgs)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F1cRW2XrKHif",
        "colab_type": "text"
      },
      "source": [
        "##Task 6: Super resolution\n",
        "The convolutional autoencoder is a network that maps an image to another image. There are other types of these *image-to-image networks*.\n",
        "\n",
        "One example is a super resolution network. This is pretty much an autoencoder, except that the input image has lower spatial resolution than the output image. Super resolution networks learn to increase the spatial of the input image.\n",
        "\n",
        "Your task is to modify the autoencoder such that it takes an 14x14x1 image as input and transforms it to a 28x28x1 image. Specifically, the training and test inputs should be\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Dvcl2mGLu0v3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Low resolution images (pick every other pixel)\n",
        "x_train_small = x_train[:,::2,::2,:] \n",
        "x_test_small = x_test[:,::2,::2,:]\n",
        "\n",
        "# Show example\n",
        "print(\"Input low resolution images\")\n",
        "show_imgs(x_train_small)\n",
        "print(\"Output high resolution images (target)\")\n",
        "show_imgs(x_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SjUB_cm4yRj6",
        "colab_type": "text"
      },
      "source": [
        "##Task 7: Image regression\n",
        "Recall that regression problems are when the output of the model is one or more scalar values, rather than class labels. Both the autoencoder and super resolution network are examples of regression models. Another example of image regression is [facial landmark prediction](https://medium.com/@rishiswethan.c.r/emotion-detection-using-facial-landmarks-and-deep-learning-b7f54fe551bf), which can be used for emotion recognition.\n",
        "\n",
        "In this task we will estimate the rotation angle of rotated MNIST images (but it could just as well have been estimating pixel coordinates of facial landmarks).\n",
        "\n",
        "As a first step, we need an image generator that generates batches of randomly rotated images, along with the target rotation angles that the model should learn to predict. This code was modified from https://d4nst.github.io/2017/01/12/image-orientation/\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EAhmL_jj_5m-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.preprocessing.image import Iterator\n",
        "from keras.utils.np_utils import to_categorical\n",
        "import cv2\n",
        "\n",
        "class RotNetDataGenerator(Iterator):\n",
        "\n",
        "    def __init__(self, input, batch_size=64,\n",
        "                 preprocess_func=None, shuffle=False):\n",
        "\n",
        "        self.images = input\n",
        "        self.batch_size = batch_size\n",
        "        self.input_shape = self.images.shape[1:]\n",
        "        self.preprocess_func = preprocess_func\n",
        "        self.shuffle = shuffle\n",
        "\n",
        "        # add dimension if the images are greyscale\n",
        "        if len(self.input_shape) == 2:\n",
        "            self.input_shape = self.input_shape + (1,)\n",
        "        N = self.images.shape[0]\n",
        "\n",
        "        super(RotNetDataGenerator, self).__init__(N, batch_size, shuffle, None)\n",
        "        \n",
        "    def _get_batches_of_transformed_samples(self, index_array):\n",
        "        # create array to hold the images\n",
        "        batch_x = np.zeros((len(index_array),) + self.input_shape, dtype='float32')\n",
        "        # create array to hold the labels\n",
        "        batch_y = np.zeros(len(index_array), dtype='float32')\n",
        "\n",
        "        # iterate through the current batch\n",
        "        for i, j in enumerate(index_array):\n",
        "          \n",
        "            image = self.images[j].squeeze()\n",
        "\n",
        "            # get a random angle\n",
        "            rotation_angle = np.random.randint(-45,45)\n",
        "\n",
        "            # rotate the image\n",
        "            rows,cols = image.shape\n",
        "            M = cv2.getRotationMatrix2D(((cols-1)/2.0,(rows-1)/2.0),rotation_angle,1)\n",
        "            rotated_image = cv2.warpAffine(image,M,(cols,rows))\n",
        "\n",
        "            # add dimension to account for the channels if the image is greyscale\n",
        "            if rotated_image.ndim == 2:\n",
        "                rotated_image = np.expand_dims(rotated_image, axis=2)\n",
        "\n",
        "            # store the image and label in their corresponding batches\n",
        "            batch_x[i] = rotated_image\n",
        "            batch_y[i] = rotation_angle\n",
        "\n",
        "        # preprocess input images\n",
        "        if self.preprocess_func:\n",
        "            batch_x = self.preprocess_func(batch_x)\n",
        "\n",
        "        return batch_x, batch_y\n",
        "\n",
        "    def next(self):\n",
        "        with self.lock:\n",
        "            # get input data index and size of the current batch\n",
        "            index_array = next(self.index_generator)\n",
        "        # create array to hold the images\n",
        "        return self._get_batches_of_transformed_samples(index_array)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PBRKLikE37WE",
        "colab_type": "text"
      },
      "source": [
        "###Test the generator"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LJSiRdxcBcEQ",
        "colab_type": "code",
        "outputId": "e6826f81-b911-455f-d256-9b1e34aec268",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 267
        }
      },
      "source": [
        "# Instantiate\n",
        "datagen = RotNetDataGenerator(\n",
        "        x_train,\n",
        "        batch_size=32,\n",
        "        preprocess_func=None,\n",
        "        shuffle=False\n",
        "    )\n",
        "\n",
        "# Generate batch\n",
        "rotated_images, angles = datagen.next()\n",
        "\n",
        "# Display\n",
        "print(\"Images (before rotation)\")\n",
        "show_imgs(x_train)\n",
        "print(\"Images after random rotation\")\n",
        "show_imgs(rotated_images)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Images (before rotation)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABGoAAABsCAYAAAAyoVQIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAGqNJREFUeJzt3XmQlMUZx/FeBFcIriAQDwwL4RKC\nsAioIIWGQxQRUAKC3GowEFCTQDBCFMMpKqkFRC6BcFSACpcYDBK5PJACCVatgAGiIIRTwYVFFjCb\nP6x6fLrd2X1n952Zd979fv76vemed1pm3znedPeTkpeXZwAAAAAAAJB4pRI9AAAAAAAAAHyHGzUA\nAAAAAAABwY0aAAAAAACAgOBGDQAAAAAAQEBwowYAAAAAACAgShfUmJKSQkmoxDmVl5dXxY8T8Tom\nTl5eXoof5+E1TCiuxRDgWgwFrsUQ4FoMBa7FEOBaDAWuxRCIdC0yoya4DiZ6AACMMVyLQFBwLQLB\nwLUIBAPXYohxowYAAAAAACAguFEDAAAAAAAQENyoAQAAAAAACAhu1AAAAAAAAAQEN2oAAAAAAAAC\nghs1AAAAAAAAAcGNGgAAAAAAgIDgRg0AAAAAAEBAcKMGAAAAAAAgILhRAwAAAAAAEBDcqAEAAAAA\nAAgIbtQAAAAAAAAEROlEDwCIRpMmTSQPGTLEauvbt6/kBQsWSJ46darVb+fOnTEaHQAAwHcyMzMl\nP/nkk5KzsrKsfh07dpR88ODB2A8MAFBk77zzjuSUlBTJrVu39vV5mFEDAAAAAAAQENyoAQAAAAAA\nCIikX/p0xRVXSL7mmms8PcZdMlOuXDnJdevWlfzrX//a6vfyyy9L7tmzp9V24cIFyRMnTpT8wgsv\neBoT8peRkWEdr1+/XnJaWprVlpeXJ7lPnz6SO3XqZPWrVKmSn0NEArRp00by4sWLrba77rpL8qef\nfhq3MSF/o0aNkuy+H5Yq9f3/V3D33XdbbZs3b47puICwuPrqqyWXL1/earv//vslV6lSRfLkyZOt\nfrm5uTEaXclSvXp167h3796S//e//0muV6+e1e/mm2+WzNKnxKtTp451XKZMGcmtWrWSPH36dKuf\nfo2LavXq1ZJ79OhhtV28eLHY5y+p9GvYokULyePHj7f63XnnnXEbE5LHn//8Z+tY/w3p7Tb8xowa\nAAAAAACAgOBGDQAAAAAAQEAEZulTtWrVrOMrr7xSsp5e1LJlS6tfhQoVJHft2rXY4zh8+LDkKVOm\nWG0PPvig5LNnz1ptH3/8sWSm7BfPbbfdJnn58uVWm17eppc6GWO/Jnp6qLvU6Y477pDsVoAK27RS\nPUVX/zusXLkyEcPxTbNmzSRv3749gSNBfvr37y95xIgRkguaFu5ezwC+p5fU6GvKGGOaN28uuUGD\nBp7Od8MNN1jHuiIRiu7kyZPW8ZYtWyS7y7CReD/72c8k68+tbt26Wf30Mt0bb7xRsvuZ5sfnmP47\nmTFjhtX29NNPS87Ozi72c5Uk+vfDxo0bJR87dszqd/3110dsQ8mitzL51a9+ZbVdunRJsq4A5Tdm\n1AAAAAAAAAQEN2oAAAAAAAACghs1AAAAAAAAAZHQPWp06eUNGzZYbV5LbftBrzHVpWTPnTtn9dNl\ngI8ePWq1nT59WjIlgQunS6IbY8ytt94qedGiRZLddfQF2bdvn+RJkyZJXrJkidXv/fffl6xfb2OM\nmTBhgufnSwa65HHt2rUlJ9seNXp9uDHG1KhRQ3J6errVlpKSEpcxITL9mlx11VUJHEnJdPvtt0vW\n5YF16Xpj7P0ZXMOGDZP83//+V7K7T5x+v962bVv0g4XQJZqNsfej6NWrl+SyZcta/fR73hdffGG1\n6b3bdEno7t27W/10meG9e/dGM2woOTk51jGltoNNf+fr0KFDAkeSv759+1rHr7/+umT9XRZFp/ek\ncY/Zo6Zk03ua6vLuxhjz3nvvSV62bFnMxsCMGgAAAAAAgIDgRg0AAAAAAEBAJHTp06FDhyR/+eWX\nVltxlz65U7DPnDkj+ec//7nVpksyL1y4sFjPC29mzpxpHffs2bPY59TLp8qXLy/ZLZeulwM1bNiw\n2M8bZHra7NatWxM4kuJxl8D98pe/lKyXXhjDtP1EaNu2rXU8dOjQfPu5r03Hjh0lHz9+3P+BlRAP\nP/ywdZyZmSm5cuXKkt1lgZs2bZJcpUoVq+2ll17K97ncc+jH9ejRw9uASzj9/ebFF1+U7L6OV199\ntafz6WW/7du3t9r0dG19/em/i/yOUTQVKlSwjhs1apSgkcCL9evXSy5o6dOJEyck6+VH7rJst1y3\n1qJFC8nuMlQkDsvlk0urVq0kjxw5UrL7O/Krr76K+tzuORo0aCD5wIEDVpteHh5LzKgBAAAAAAAI\nCG7UAAAAAAAABAQ3agAAAAAAAAIioXvU6PVjw4cPt9r03gX/+te/JE+ZMiXi+Xbt2iW5Xbt2Vpsu\nmeiWJH3qqac8jhjF0aRJE8n333+/1RZpjai7v8yaNWskv/zyy1abLiGr/2Z06XRjjGndunWhzxsW\n7vrpZDVnzpyIbXp/BsSPLtM8b948qy3SHmPuvieUro1O6dLff2Q3bdpU8uzZs61+5cqVk7xlyxbJ\nY8aMsfrp8pKpqalWmy43ec8990Qc044dOwobNhwPPvig5Mcffzzqx7tr5fX3Hbc8d61ataI+P4pO\nX3vGGFOtWjVPj2vWrJlkdy8v3idj57XXXpO8atWqiP0uXbokuaglm9PS0iRnZWVJvvHGGyM+xh0T\n77f+y8vLs46vuuqqBI0EXsyaNUty7dq1JdevX9/qp7/fePXss89ax5UqVZKs98Y0xpiPP/446vMX\nRTh+xQEAAAAAAIQAN2oAAAAAAAACIqFLnzR3et+GDRsknz17VrJb6vCxxx6TrJfC6KVOrk8++cQ6\nHjhwYHSDhWcZGRmSdRlEPQXUGHvq4VtvvSXZLZWmSxqOGjXKatPLY06ePCnZnZ6myye6S7B0ie+d\nO3eaZOOWG7/uuusSNBJ/RVpKY4z9d4X46devn+SCpm7rEtALFiyI5ZBCr3fv3pILWg6orwld8jk7\nOzviY9zS0JGWOx0+fNg6/stf/hLxnMhft27dPPX7/PPPJW/fvl3yiBEjrH7ucietXr160Q0OxaKX\nYBtjzPz58yWPHj064uN025kzZ6y2adOm+TE05OPy5cuSC7qO/NC+fXvJFStW9PQY9/02NzfX1zHh\nh/Sy4g8//DCBI0F+zp8/L1n/dizqkjX9OzU9Pd1q078XE7Ukjhk1AAAAAAAAAcGNGgAAAAAAgIAI\nzNInV6Qp2l9//XXEx+gdmZcuXWq16elLiJ06depYx7qal16+curUKavf0aNHJeup9OfOnbP6/f3v\nf883F1XZsmWt49/97neSe/XqVezzx1uHDh2sY/e/L5noZVs1atSI2O/IkSPxGE6JV7lyZev40Ucf\nley+v+qp+2PHjo3twELMrdKkKxLoKb/Tp0+3+ulloQUtd9JGjhzpqd+TTz5pHetlpvBGf1fRS6/f\nfvttq9/+/fslnzhxokjPFZblr8lKX8MFLX1C+PTo0cM61te91+9mzz33nK9jKsn0Mjf9W9JdWl+z\nZs24jQmFc78H3XLLLZL37NkjOZoqTD/60Y8k66XEbtU+vfTtb3/7m+fz+4kZNQAAAAAAAAHBjRoA\nAAAAAICA4EYNAAAAAABAQAR2j5pI3DW+TZo0kaxLN7dt29bq5679hn9SU1Ml6xLpxth7pugy6337\n9rX67dixQ3Ii91WpVq1awp7bD3Xr1o3Y5palDzr9t+Tus/Dvf/9bsv67gr+qV68uefny5Z4fN3Xq\nVMkbN270c0ihp/ck0HvSGGPMxYsXJa9bt06yW675m2++yffcbnlJXYLbfe9LSUmRrPcZWr16dcSx\nwxtdwjnW+5Y0b948pueHd6VKff//jbJvYji4exk+88wzkmvVqmW1lSlTxtM5d+3aJfnSpUvFGB00\nvXfeu+++K7ljx46JGA4K8JOf/ESy3tvJGHuvoSFDhkiOZr+8yZMnS+7WrZtk/dlsjDF33nmn53PG\nCjNqAAAAAAAAAoIbNQAAAAAAAAGRdEufcnJyrGM9JWrnzp2SZ8+ebfXT0+/1MhtjjHn11Vcl65Kn\n8KZx48aS3fLQWufOnSVv3rw5pmPCD23fvj3RQzDGGJOWlib53nvvtdp69+4tWS/LcOlyfXo6K/yl\nX5+GDRtG7PfOO+9Yx5mZmTEbU9hUqFDBOh48eLBk9/NIL3fq0qWLp/Pr6feLFy+22vTSYZcuRTlp\n0iRPz4XY0WXRdWnRwuhSptoHH3xgHW/durVoA4NnerkT3zWDQS/v7dOnj2R3+4RIWrZsaR17fV2z\ns7Ml6+VSxhizdu1ayZGWsQJh06BBA8krV66UXLlyZaufXlrv9bfksGHDrOP+/fvn22/cuHGezhdP\nzKgBAAAAAAAICG7UAAAAAAAABETSLX1yHThwQLKeyjRv3jyrn57SqLMx9jTiBQsWSD569Khfwww1\nvXu2rhRijD0tLSjLnUpq5YVrr722SI9r1KiRZP36ulODb7rpJslXXnmlZLcqgv73d6f1btu2TXJu\nbq7k0qXtt6qPPvrI09gRPb2kZuLEiRH7vffee5L79etntX399df+Dyyk9LVizA+n+Wp6+cuPf/xj\nyQMGDLD6derUSbKeTly+fHmrn56m707ZX7RokWR3yTH8U65cOcn169e32p5//nnJBS0r9vqZpita\nuH8z3377beGDBZKcfj80xpg33nhDcjyrfuqqQ7NmzYrb86JwlSpVSvQQQkt/l9dbHRhjzOuvvy65\noM80XcnwD3/4g2T9W9QY+zePruxkjP1bRv/unzlzZsH/AQnAjBoAAAAAAICA4EYNAAAAAABAQHCj\nBgAAAAAAICCSfo8aTZfz2rdvn9Wm1661adPGahs/frzk9PR0yW6ZriNHjvgyzmTXsWNH6zgjI0Oy\nu8+BXv8bFAWVyNy1a1e8h+Mrd88X/d83Y8YMyc8++6znc+qyzHpd5+XLl61+58+fl7x7927Jc+fO\ntfrt2LFDsrtv0fHjxyUfPnxYctmyZa1+e/fu9TR2FE6XJzXGmOXLl3t63H/+8x/J+nVDdC5evGgd\nnzx5UnKVKlWsts8++0yy1zKwel8SXRLWGGNuuOEGyadOnbLa1qxZ4+n8KFyZMmWs48aNG0vW15t+\nPYyx38/16+iW0r733nsl6z1vXHp/gIceeshqy8zMlOz+TQJhpb/TuHsseqH30jDG+76H+nv0fffd\nZ7W99dZbUY8D/tF7vMFfPXr0kDxnzhyrTX+n0dfR/v37rX5NmzbNN3fu3NnqV7VqVcnuZ6v+nvXo\no496GnuiMKMGAAAAAAAgILhRAwAAAAAAEBChWvqkZWVlWcfdu3eX/MADD1htupT3E088Ibl27dpW\nv3bt2vk5xKTlLkPR5WVPnDhhtS1dujQuY3KlpqZKHj16dMR+GzZssI51qbdkNHjwYOv44MGDklu0\naFGkcx46dEjyqlWrJO/Zs8fq9+GHHxbp/NrAgQMl62UfepkN/DVixAjr2OvU7YJKd8O7M2fOWMe6\nPPqbb75ptelykwcOHJC8evVqq9/8+fMlf/XVV5KXLFli9dPTgd02FI/+XNRLk4wxZsWKFfk+5oUX\nXrCO9efT+++/L1n/Hbj93PLDmn5PnTBhgtUW6X3eGGNyc3MjnhPeeS2j3qpVK+t42rRpMRtTSeP+\nNrj77rsl63LB69ats/pduHAh6ud67LHHrOOhQ4dGfQ7ExsaNGyW72znAPw8//LB1rH9vX7p0yWrT\n34UeeeQRyadPn7b6vfLKK5LvuusuyXoZlDH2UkZ3qXjlypUlf/HFF5L1+4Ex9vesRGFGDQAAAAAA\nQEBwowYAAAAAACAguFEDAAAAAAAQEKHdo8al174tXLjQatMlwnT5SnedsF67tmnTJn8HGBLuWvaj\nR4/G7bn1vjSjRo2SPHz4cKufLvus1zoaY8y5c+diNLrEePHFFxM9hKi0adMm3//da8loeJORkSH5\nnnvu8fQYdx+UTz/91Ncx4Tvbtm2T7JbnLgr9OabXcxtj75PBPlDF45bg1vvNuJ9Bmi7FO3XqVKtN\nf2/Rfwtr1661+t1yyy2S3dLakyZNkqz3r3FLmS5evFjyP//5T6tNf464+wVou3btitgG+3pz90zQ\n3NLp9evXl7x7927/B1aC6X38xo0b5+u53f0R2aMmOPSeXC79Xp6enm616b8XFE7v+2qM/e8+duxY\nq03vX1MQfR3NnDlTcvPmzT2PS+9fo/crCsKeNC5m1AAAAAAAAAQEN2oAAAAAAAACIrRLnxo2bGgd\n/+IXv5DcrFkzq00vd9LcKaZbtmzxaXTh9cYbb8TtufTyDWPs6eW6JJy7ZKNr166xHRh8t3LlykQP\nIVTefvttyRUrVozYT5dc79+/fyyHhBgpW7asZLcksF5+QXnu6F1xxRWSx4wZY7UNGzZMck5OjtX2\nzDPPSNb/7m6pdl1uVJdobty4sdVv3759kgcNGmS16WndaWlpklu0aGH169Wrl+ROnTpZbevXrzf5\n0WVNjTGmRo0a+fbDd2bMmCHZXRJQkIEDB0p++umnfR0TYqd9+/aJHgIiuHz5csQ2vSxGb6mA6Lm/\nv1asWCHZ/fzwSpfW1st5XT179pSclZUVsZ/eDiOImFEDAAAAAAAQENyoAQAAAAAACIikX/pUt25d\nyUOGDJHs7pp//fXXezrft99+K9mtWOROGy+p9LRA97hLly5W21NPPeXrc//mN7+R/Mc//tFqu+aa\nayTrChZ9+/b1dQxAsqtUqZLkgt7Xpk+fLjlsFdFKinXr1iV6CKGll6TopU7GGHP+/HnJ7jIXvfTw\njjvukDxgwACr33333SdZL2H705/+ZPXT1TIKmk6enZ0t+R//+IfVpo/1lHFjjHnkkUfyPZ/+PEbh\n9u7dm+ghlAhuBTZd2XDDhg1W2zfffOPrc+trODMz09dzwz96SY57Xd58882S3aWGgwcPju3AQsaP\na0D/tjPGmG7duknWy3ndik3Lli0r9nMHATNqAAAAAAAAAoIbNQAAAAAAAAHBjRoAAAAAAICASIo9\navT+Mu7aab0vTfXq1Yt0/h07dkgeN26c5HiWmk4muqSre+zuBTRlyhTJc+fOlfzll19a/fQ6/T59\n+khu1KiR1e+mm26SfOjQIatN78Wg99ZActJ7H9WpU8dq02Wj4Y3ex6JUKW/36D/44INYDQdxQonY\n2HnuuecitunS3cOHD7faRo8eLblWrVqenks/ZsKECVab3lvPD3/9618LPEbRTJ06VfLQoUOttpo1\na0Z8nN7rT5/D3ZOhJGvZsqXkkSNHWm3t2rWT7JaQL0qJ4GuvvVZyhw4drLbJkydLLleuXMRz6L1x\nLly4EPUY4B+9Z5gxxlStWlXyb3/723gPBw53X6BBgwZJPnHihOTWrVvHbUzxxIwaAAAAAACAgOBG\nDQAAAAAAQEAEZunTddddZx3Xr19f8rRp0yTrsmnR2LZtm+SXXnrJatNl2ijBXTx6urcx9pS1rl27\nStZlQo0xpnbt2p7Or5dibNy40WoraBo6ko9eUud1qQ6+l5GRYR23bdtWsn6fu3jxotXv1VdflXz8\n+PEYjQ7x8tOf/jTRQwitY8eOSa5SpYrVlpqaKtldwqutXbtW8pYtW6y2VatWSf78888l+73UCfH3\nySefWMcFXad8Ly2c/p3QoEGDiP1+//vfW8dnz56N+rn0Uqpbb73VanO3BtA2bdok+bXXXpPsfpdF\nYunX0P1+hPhIT0+X/Pjjj1tt+vWZNWuW5MOHD8d+YAnArx8AAAAAAICA4EYNAAAAAABAQHCjBgAA\nAAAAICDiukeNLmlnjDEzZ86U7O6nUJR19Xr/kldeecVq06WbdVk8RG/r1q3W8fbt2yU3a9Ys4uN0\n6W53TyJNl+5esmSJ1abLVKLkaN68uXU8f/78xAwkiVSoUME61tefduTIEet42LBhMRsT4u/dd9+V\n7O71xN4XxdOqVSvJXbp0sdr03hW6hKgxxsydO1fy6dOnJbMfQsmh91YwxpgHHnggQSMpWXRp31jQ\n1/qaNWusNv39lZLcwZWWlia5c+fOVtvKlSvjPZwSaf369ZL1fjXGGLNo0SLJzz//fNzGlCjMqAEA\nAAAAAAgIbtQAAAAAAAAEREyWPt1+++2Shw8fLvm2226z+lWtWjXqc58/f946njJliuTx48dLzsnJ\nifrc8MYtgfbQQw9JfuKJJ6y2UaNGeTpnZmamZF22cP/+/UUZIkIgJSUl0UMAkl5WVpbkffv2WW16\niXHNmjWttpMnT8Z2YCGgS/suXLjQanOPAW337t3W8Z49eyTXq1cv3sNJev3795c8dOhQq61fv37F\nPv+BAwck698hemmpMfaSNv3ei+Dq3r27dZybmytZX5eIn3nz5kkeM2aM1bZ69ep4DyehmFEDAAAA\nAAAQENyoAQAAAAAACIiUvLy8yI0pKZEbCzBx4kTJeulTQdxpoG+++abky5cvS3arOZ05c6YoQ0wG\nH+Xl5TX140RFfR1RfHl5eb6s3ykpr6Gevqwro8yePdvq5y6xi7GkvBbdKk9Lly6V3LJlS8mfffaZ\n1a9WrVqxHViCcC3a15cxxsyZM0fy5s2brTa9fMD9fE6gpLwWYeNaDIXAXoupqanWsX7fGzt2rNVW\nsWJFyatWrZKsq84YYy+3OHbsmB/DDASuxR9Wl9VLDzt16mS1HTx4MC5jilJgr0V4F+laZEYNAAAA\nAABAQHCjBgAAAAAAICC4UQMAAAAAABAQMdmjBr5gzWEIsP43FLgWQ4Br0Zi0tDTreNmyZZLbtm1r\nta1YsULygAEDJOfk5MRodJ5wLYYA12IocC2GANdiKHAthgB71AAAAAAAAAQcN2oAAAAAAAAConSi\nBwAAAGIvOzvbOu7evbvkcePGWW2DBg2SPHr0aMkBKtUNAAAQWsyoAQAAAAAACAhu1AAAAAAAAAQE\nN2oAAAAAAAACgvLcwUW5tRCg9GEocC2GANdiKHAthgDXYihwLYYA12IocC2GAOW5AQAAAAAAAo4b\nNQAAAAAAAAFRWHnuU8aYg/EYCH4g3cdz8TomBq9hOPA6Jj9ew3DgdUx+vIbhwOuY/HgNw4HXMflF\nfA0L3KMGAAAAAAAA8cPSJwAAAAAAgIDgRg0AAAAAAEBAcKMGAAAAAAAgILhRAwAAAAAAEBDcqAEA\nAAAAAAiI/wOlDS3HrbUb+QAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 1440x288 with 10 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "Images after random rotation\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAABGoAAABsCAYAAAAyoVQIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xm81nP6x/GrMX6RtFBJq9NqGFki\nWxExY6cm2ZvKFo0Hhsk+GDOa8MBYC8PQIElRGVuihUQ1JUsJKUqT0iJly/n9MQ/XvK9P5z6d6izf\nc5/X86/3/ft8Or5z7vNd7vv3uT5XtcLCQgMAAAAAAEDF+1lFHwAAAAAAAAD+iy9qAAAAAAAAMoIv\nagAAAAAAADKCL2oAAAAAAAAygi9qAAAAAAAAMuLnxQ1Wq1aNllAVZ2lhYWH90vhBvI8Vp7CwsFpp\n/BzewwrFuZgHOBfzAudiHuBczAuci3mAczEvcC5uQL169cLr1atXe/7mm2/K+3CKlOtcLPaLGlSo\n+RV9AADMjHMRyArORSAbOBeBbOBc3ICuXbuG1xMnTvQ8e/bs8j6cjULpEwAAAAAAQEawogbAJvnZ\nz+L3vA0bNvS8aNGi8j4cAAAAAFXcKaec4rl3795hrE6dOp5ZUQMAAAAAAIAS4YsaAAAAAACAjOCL\nGgAAAAAAgIxgjxoAm+Skk04Kr7fYYgvPb731luc5c+aU2zEBAAAAqDr0M4iZWa1atTy3bds2jDVr\n1sxz48aNPS9cuLCMjm7TsaIGAAAAAAAgI/iiBgAAAAAAICMofQIQbL311uH1t99+63nQoEGemzRp\nEubNnTvX85gxY8ro6AAAufzsZ/H///bjjz9W0JEAAFA+1q1bF17vuOOOntPPNa1atfKcxXInxYoa\nAAAAAACAjOCLGgAAAAAAgIzgixoAAAAAAICMqPR71Oywww6ef/jhB8/ff/99mLdq1apyOyaUj512\n2im8PvDAAz23aNHCs7aKNjObOnWq52XLloWxwsLCUjzCykP3m2nfvn0Y69mzp2f9nU+ZMiXMu/XW\nWz2vWLGilI8QQHn6v//7P8/fffddBR4JzMx+/vP/Pa41b948jHXo0MFzek97//33PX/66adldHTY\nWHp+mZnttddeno866qgwpvu/DRkypGwPDEXaaqutPOtnDc0AKk6NGjXC6+22286z7rVpZrZy5UrP\nTZs29ZzFeyQragAAAAAAADKCL2oAAAAAAAAyotKVPqVLm8444wzPJ598sufbb789zJs3b57nmTNn\nhrHVq1eX5iGiDO28886etdTGzKygoMCztl6bPn16mPfYY4951nbTZusvj8tXzZo1C68vvvhiz126\ndAlj2267refnnnvO88CBA8O8+fPnl+YhIk8cccQRns8555ww1qhRI8933XVXGBs1apRnSlfLhl4n\nDzjggDD2ySefeJ4wYUJ5HRKEljvtvffenk899dQwT8+xbbbZJoz985//9HzTTTd5TkukUL7S9+mK\nK67w3KlTp5z/rl69ep4HDx4cxtasWVNKR1d1aDt7/d1qObhZLI/Qzwx6XpqZnXfeeZ4XLVoUxhYv\nXuxZn5f0/57+u7R0f/bs2UX8r8BP9ttvP8+6zYEZZWr5Lr3+TZo0yfPxxx8fxvS8//rrr8v2wDYT\nK2oAAAAAAAAygi9qAAAAAAAAMqLSlz4dfvjhnnfbbTfP6c74L7/8suc5c+aEsWHDhnmeNm2a57Vr\n14Z569at24QjRklsscUWnvU93n///cM8Xbq9yy675PwZWsKkfxdmZldffbXndOlwPtNyp0svvTSM\nHXPMMZ7TZYCjR4/2fMcdd3im1Am5HHzwwZ779+/ved999w3ztt56a88NGzYMY3o+Y+NUr17dc58+\nfTyn97SWLVt67t69exhbuHChZ71ezJgxo9SOE2bVqlXznJakXnDBBZ67du3quUGDBmHe8uXLPe+4\n445hrG/fvp7Hjh3reeLEiWHeN998szGHjc1Uq1at8Fq7QNWpUyeMaTdKLZPRvx1sGj1fdCuFww47\nLMzT500tSdx+++3DPH1PtFxqY2gJhz5/mZldcsklnj///PNN+vmVXfp337p1a8/6+9HSFzOzhx56\nyDPl1PlP72lpJ2j9jNimTRvPb7zxRtkf2EZiRQ0AAAAAAEBG8EUNAAAAAABARvBFDQAAAAAAQEZU\nuj1q0tpE3XtG63o7dOgQ5mnL4Xbt2oWxE044wfOLL77o+corrwzzvvrqK8+09N482vLZzGzXXXf1\n3LFjR89ab2oW2yeme1hoPeKHH37oeenSpWGetvju3LlzGNO/p7SmsTLS3/P555/v+cgjjwzzvvji\nC88333xzGNP2kOxLg5IoKCgoMuveKan0mkorzeLVrVvX82mnnRbGrrrqKs/6O1+5cmWYp/szpO+N\n1nCne8Oh9LRt29aztvY1i2249f1+5plnwrwxY8Z47tmzZxjTe9whhxziedasWWFe2iIYpU/3oUn3\n66pdu3bOf/f66697njt3ruest5WtDOrXr+/50EMP9aznipnZjz/+6Lms96vU661+djGL14iBAwd6\nrkp/C9pa2Sw+Y+jnufRz4Pjx4z2/8847nnnWyE9ffvmlZ90P0Sw+Mz311FOe0+8YdH+wisKKGgAA\nAAAAgIzgixoAAAAAAICMqHSlT8uWLQuvhw8f7vmDDz7wfPfdd4d5jRo18qxLHVO9evXyfOCBB4ax\nESNGeE6XDT/33HOedbkV/merrbbyrO17zcyuv/56z3vssYfndNlZce0odUzbFk6ePDnM23vvvT3f\neeedYeyaa67xPHTo0Jz/rcpCy/W0DGqHHXYI8/7yl7941uWhZrEsCijKXnvtFV737t3bs/6tLVq0\nKMwbMGCA5+effz6Mpa2kEdsy69Lds88+O+e/efjhhz3rcm+z4u93+r7pPS2LS4Mrm8aNG3vWFtw9\nevQI87TUV8+dYcOGhXmvvvqq57QlsJY+aSlA2gKa0qeyoeeLljfpuWdmtttuu+X8GVqSnV5DsXn0\n737UqFGetQW3WXx++uSTTzx/9NFHYd4222zjuWbNmmGsWbNmnrXcv7jPJHoNMDPbc889PWvpqpb7\n57u09OzNN9/0PHPmTM/t27cP8/Seefvtt3teuHBhaR8iStGWW24ZXpd0W4qPP/7Y87Rp08KYlp7W\nqlXLcxafZ1hRAwAAAAAAkBF8UQMAAAAAAJARfFEDAAAAAACQEZVujxptkWdmNm/ePM9a86mtms1i\nq9E1a9aEMa2/11rR1q1bh3mXXXaZZ22XaGb2q1/9yrO29a7qtY9a+6f70px55plhnr53+h6ndfNT\npkzxrPW5ZnGfDN2b5T//+U+Y995773nefvvtw5i2QM0Hui+Q1jCnbba7devmefTo0WV/YKj0dt99\nd89pS3c9F7Ulbbrv08iRIz0vWbIkjKXX+qoo3SdBf6/9+/f3nP6udO+fBx54wHN6T0vbQSu91mpN\neBZruLNIW8imezhde+21nvXZQc8Vs/gcM2HCBM/pfk7aXlb3Z0tpXX5670PZ02tmmzZtwpjubZLu\nCzdp0iTP7FFTuvT58MEHH/Sc3qt0ryFthf3dd9+FebovTboPVJMmTTyfe+65nrt37x7m6XUg/byi\n53f6366qdD+7v/3tb54feeSRME/3qHnhhRc8p88eJd0DBaWrYcOGnq+66irPTzzxRJj37rvvel6+\nfHnOn7fFFlt4TvfW0/uz7m+bRayoAQAAAAAAyAi+qAEAAAAAAMiISlf6lNJSlaOPPtpzWhYzZMgQ\nz9rK0sxs1apVnvv16+f58MMPD/N06VTadk+P46CDDvKctt0bPHiwZy3HyhfpUs+OHTt61jak+n83\ni0v3dVnbLbfcEubpsl9tO2pmNnHixCKzlsel/20tzTLLv1bUunT+0Ucf9VxQUBDmacvHQYMGhbG+\nfft61iWm6ZJcVC26XDst7dBl/DNmzPCsS8vN4pJjSp3Wl5YZ6TVP70Hjxo0L88aOHetZz1N9X8xi\nu+C0XPjTTz/1nLagxYZVr17ds5b2mpkddthhnrXMYeXKlWHe9OnTPetScH1mMYt/JytWrAhjWtqh\nLdfTc1ZLgotbTo6No++NltlreXYqLfnW1tA1atTwzD148+n7o7/PTf3d6vmXnot6rmvL4eLufXpe\nmpmNGDHC82effbZJx5hv9N41ZswYz2lLZm3XraVnKb1/ovw0btzY8x577OFZS0bNYtn3G2+8kfPn\n6XUz/Wyq5Yvp9wVZw4oaAAAAAACAjOCLGgAAAAAAgIyo9KVPq1ev9qxLGNMSlpkzZ3rWEpCU/rvt\nttsujM2aNcvz008/HcbatWvn+b777vOcljdpGcnZZ58dxl577bWcx5Vl2pmkR48eYezEE0/0rOU1\n77zzTpi3dOlSzy+99JJn7XRhZrZs2TLPaUmTLn/UeevWrQvzdLfvFi1ahDHtiqJL5dIlrJWRlpn8\n8Y9/DGNXX3215w4dOoQx/Vu/8cYbPU+dOjXMy3rZmHbAMlu/1APFS6+HvXr18qxLTM1iacY999zj\n+ZNPPgnzKHdan5bYNmrUKIxpCY3eW9Llv7m6GBTXhTC9V2nZjXZPSK+nKJqWibZt2zaM6d+9liY9\n9dRTYd7jjz/uWc+d4jpv6f3NLJbb6Hmadn3SZymUDT3/0uX8+p7+8pe/DGP6Wkv3KX3KtvQ9PuGE\nEzzn6oyYSkuf9DrA/XN9et299957w9jdd9/tuUuXLp7T5xLtssZzYvnR90E/V+6yyy5hXqdOnTy/\n/fbbntProd4LtazKLJYeZr17GitqAAAAAAAAMoIvagAAAAAAADKCL2oAAAAAAAAyotLvUaO1nbqP\nSLqfQnEtvJTWyD3xxBNh7M477/T8ww8/hDFtZao14d26dQvz9LiOO+64MKb7CmR9vw/VqlUrz336\n9Alj2mLthRde8Ky1omaxha/WFepeM2Zm33//vecFCxZs0vE2b97cs9YpmsX2t08++aTnfNijRqX/\ne6699lrPul+NWWxTr+3l0/dQ91jSGt+KpO91Wvev146RI0eW2zFVJjvttJPntLV2w4YNc/67IUOG\neNZ2otR7b5jupXTkkUeGsaOPPtqz7uWV3qt0Hxmt9dY9Esxiu+7PP/88jOk+Q+xLs3mef/758Frv\n7/rMkT6n6Dzdj0LbbJuZFRQUeNb7sVm8zul+OOneF3pvRdno3bu3Z92LKjV37tycr2mdXrp036am\nTZt63nrrrcM8PRf1GV/3yzAzO/XUUz3rnodm8Z6p53C6r5Ree/WZyyz3/mP4L33GGDZsWBjT++nB\nBx/sWfcLSsf0cwvKln7eGzp0qOfrr78+zGvQoIHn9LO40rbb6Wdq3aNt/vz5nvWZyCzeFytqLxtW\n1AAAAAAAAGQEX9QAAAAAAABkRKUvfdJ2XHXr1vWctiGtX79+iX7e4sWLPd92221hrLhWeFq6o2VQ\nr7zySpinSx/TpeaVqdxJXXfddZ7bt28fxnRZ92WXXeZ5zpw5YV5x7UZLm5ZzpPRv6KuvviqHo8mG\nr7/+2nPauvv111/3fMwxx3i+9NJLwzwtLapdu7bnZ599ttSOsyi1atUKr7WVX/fu3T3rsmazeK6P\nGzcujK1cubI0D7FS0XJAXSq83377hXnasnnixIlhTK+dLNXfOFqud9JJJ4UxPU+nTp3qWe85ZrHc\nqWPHjp4vvPDCME/vR7qc2CyWyWiZRnleq/OFXkPNYuvz4tpi16xZ07P+LaRlnNr+Oy23aNmypecl\nS5Z41vPXLN770utfSdsAawkHrYPX17NnT8/FnUdvvvlmeP3++++X2TFVNY0aNQqvTz/9dM/77LOP\n57SEUEsstARi1113DfP0/pm23dbz49tvv/WclopredPHH38cxjivSi59htd23U2aNPGclj7pc276\nbEiJaPnQ+1H6O9dzUZ9N0nuaXjdr1KgRxnQbkhtvvNHzokWLwrzf//73nil9AgAAAAAAqOL4ogYA\nAAAAACAjKn3pk3bI0CWCabelM844w7Pu2G5mNmXKlCJ/9qYuMdSdq9PuOrNnz96kn1nRdAf89Hf7\n61//2nPa1WX48OGeK+p/e/Xq1cNr3Qk8fY91Oaou/a9K0mWGo0eP9jxt2jTPaYeS3/3ud5779+/v\nec899wzzbr31Vs9pNxn9/RdXbnH22Wd7Puigg8LYiSee6Pmtt97ynC4vvv/++z2vXbvWqqq089lp\np53meeDAgTnnTZgwwfNZZ50VxubNm+eZUpmNs9tuu3neeeedw5jeWyZPnuw5PWcPOeQQz5dffrnn\nxo0bh3m6hDi9dut/Oytd3CqrtDNFrnKntIyza9eunq+55hrPadcnld7vlC7/1mu0WSyfGjNmTM7j\n1eXfaWccvWa/8847OY+jKtHnIz2n0uuilmmkz6SUPm0eLQVt0aJFGOvXr59nLYcpqbR7V3H3O33e\nnDVrlmfdFsAslj6ln1ew6V5++WXPWhaf/k0cf/zxntPSs3RLDJQeLRXUroTFlelq6eLSpUvDvB49\nenhu1qxZGNOyYu2mmXb50u5QutVKeWJFDQAAAAAAQEbwRQ0AAAAAAEBG8EUNAAAAAABARhS7R021\natW83jmtX88KPa5Ro0Z51j1pzMyOOuooz+PHjw9jufaoKQ3pHhyVldbzabtPs1inrm1HzeIeNRWl\nuHbcuieKWdXdl6aktHXdQw89FMa0Ffuxxx7r+dRTTw3z9P1Izz1tZ691w2nbd21zqnsrmMV24Jr1\n+mAW9/uoqLZ7FUXbhKa/2wcffLDIf5P+jnTPqbSOm31pSi6tv9Y26NpC0izuC6W/8y5duoR5ufaI\nWrBgQZinezdsu+22Yax+/fobPHZsPt1T5rjjjgtj55xzjudtttnG89ixY3P+vE6dOoXXtWvX9qzP\nS7pXm1ls3a77jZnFdtHaIn6PPfYI82bMmOFZ9wqrynr16lWieXovTPekqWr3p9Kme8MsXrw4jI0c\nOdJzx44dPaf3ML0W615fxe0JldKfqe9pel1mX5qy98wzz3hu165dGNN78HnnnRfG7rvvPs96Pc2X\nz3oVSfeoOfTQQz3rXrRm8fOF7seXftZbvny55/T5Rs2cOdOz7l1pZvbFF19s4KjLHitqAAAAAAAA\nMoIvagAAAAAAADKi2NKnmjVr2r777mtmZk2bNg1jr7/+uue0Pd3cuXM961KmtAWuLrtO21cW15pX\n6ZLGVatWedbWXmZmbdq0KTKbxVaXlL4UTZddp0vitTWsLiEzK98Wnfo3o+3bdBmjWfxbS5fBvvba\na2V0dPlPy5bmz5/vWdt7m5mdf/75nn/zm9+EMW0lrEtJ07bC2mb2z3/+cxjTFoyvvvpqSQ69ytFl\n3FryYLZ+y/qfrFy5Mry+4447NvhvsGF6HzSLy6nTsigt19O2v7/97W/DPL2n3XPPPZ4HDx4c5j32\n2GOe0zb3ep1v0KCB5yVLlhTxvwLFSZ+R9Pzr27ev5wsuuCDM03KYu+66y/OIESPCPL2mpu+j3uNu\nueWWEh1Tq1atwlhBQYHn5s2bF/mzzcwGDhzoOf27Tp/x8pX+fszMTj75ZM/6LJv+/rXcSctKsfn0\n/jRv3rwwduutt3rWZxV9hknp55patWqFsV/84heee/fuHcYOPPDAIv9dWrKhJeYoG88//7zntAy0\nXr16ntP35pJLLvGs99a0NTQ2XqNGjTxrybZur2FmtmLFCs/6OSF9D/Rza+rJJ5/0/Kc//clzeu5l\nYdsXVtQAAAAAAABkBF/UAAAAAAAAZARf1AAAAAAAAGREsXvU1KtXz2sstbbSLNbXprWcWsOuLXG1\nraOZ2eeff+5Z28WaxXZ1uh9FuoeM1p5qS7tHH300zNNWX2lLSa1PZI+aoul7UKdOnTCmbZknTZoU\nxtL216UprfHW2vATTjjBs9aIm8W2tmkrtvQ1No3+jtMaT91PI23Pre+hns9pu9JHHnnE8xNPPBHG\nOIeLpnW+ur+JtiRNrVmzxvPTTz8dxthHoXSkbT2feuopzxdddFEY69atm2fd30n3rjGLe888/vjj\nntNWsrqXW+fOncPY7rvv7ln3IcPGS9uL6jl36aWXetZ99szMBg0a5Hny5MmedQ8Fs9jGO23lrHt4\nPfDAAzmPUduXrl69Ooztsssunps1a+b5ww8/DPPKc0+6rEqvp9redfvtt/es11azeH1euHBhGR0d\n0uutftZI22SXRPrZ5YMPPvCse4WZxc9R+nyT/i2gfE2YMCG8btu2rec//OEPYezMM8/0rO/hww8/\nHOZ99dVXpXmIeSn9DKf7O+l+NelnCG2RrvtQ6vXVzOyss87ynD7f6F40H330Uc7/VhawogYAAAAA\nACAj+KIGAAAAAAAgI4otfVqwYIH169fPzMzuvffeMNajRw/PukTJLLZh1OXZaStH9dlnn+Uc02Xd\nuozULLYCnzFjhue99947zNMlUbp01yyW8mg71HSJZFVWs2ZNz+lyTm0zmC6ZLm265HufffYJY0cd\ndZTnn9rKm61ffvWPf/zD8/Dhw8NYWR9/VaHlhV27dg1jRxxxhOcWLVqEMS2j01bdaelT1pcqZpG2\nOL/77rs967ltZrZ27VrP//rXvzynS4BpyV060rbFc+bM8Xz11VeHMV2+q9fhtO32Qw89VOR/Ky3B\n0bJBXdJtFsuF9Xo6bty4MC8ttcH6WrZsGV736dPHs94/09+tlhLpvBtuuCHM03bBQ4cODWPahra4\nUuTixqZPn15kxvrS5fe5fq9aTm4WtwaoXbt2GFu5cmUpHV3+0hKkJk2ahDF9ltctF8w2/z6W/nv9\nPJE+3yhtMcwzTMXS50mzWOatbaLN4r1w11139VyW2zzkq7R9dps2bTw3bNjQs54rZvG59M033/Sc\ntlnXUqqDDz44jO28886etSQ8i+ciK2oAAAAAAAAygi9qAAAAAAAAMqLY0qd169b5kqNTTjkljI0d\nO9Zz3759w5ju5Ny+fXvPacmMzituTKXdJ3S5me7i3Lp165w/L11WqkusKHcqmi6tTjtONGjQwHOH\nDh3CmJas6JJTLTEzi0vPli9f7vmggw4K804//XTP7dq1C2O69HX8+PGe//73v4d5b7/9tqH06TLQ\n/v37ez7kkEPCvMLCQs/adcYsvr9aDpl2jMOGbbnlluH1Tjvt5FnLKNKl29q1T8sE0440KBv6t65d\nf8xiqZLe70oqXdarnYTSkhntzqZZ/40ZpU+56D1OS8XN4vL5MWPGeB4wYECYp89F+h5o9y+z+DyW\n3u/Srpwoffpep91e9PlIpc+a2vWJstKSadWqlWft9Kllm2axFHT06NFhrKSlDrk+k6SlMeedd55n\nLcdPaXdKrqHZ8sYbb3hOy0wvvPBCz9rhLS1h1r8XfebF/6TXQC1V0ufQ9D3Qcie1dOnS8Fqvxem2\nKfrZI+tla6yoAQAAAAAAyAi+qAEAAAAAAMgIvqgBAAAAAADIiGL3qCmO1kFrqyyz2H53r7328typ\nU6cwT1sOpnW8WjfavHlzz2ntrrZCLCgoyHm8Wp82ZcqUMKb7qFBXWDRt4Vu/fv0wpnthpLX4devW\n9azt09N9gvbbbz/PuteJ/nuzWEuoNYxmZqNGjfKsf5+ffvqpofRpfbhZ3JdGW+G9++67Yd4LL7zg\necSIEWEsbZOIjaP7NLVt2zaMXXHFFZ5z1dubxWulnrMof9oqvajXm0vbY86cOTPnvJNPPtnzzTff\nHMZmzZrlOd1DripJzyl93jnyyCPDmNbODx8+3HP6++vXr5/nY445xvOaNWvCvGHDhnlO30eeY8qe\n7rXQuHHjMKbv9Xbbbec5bRM9derUIv8NctP9Qo4//njP6ecJfa5I96RZtmyZ59WrV3vWvRLN4j5E\nupfGBRdcEObps6zuBWcW9zHR+2x6PiM7Jk2aFF7vs88+nvWzysCBA8O8wYMHe/7www/L6Ogqt3S/\nU71Xffnll551z6Di6D5fZmZ16tTxnLbu1s+g2267rWfacwMAAAAAACAnvqgBAAAAAADIiE0ufVLp\nEk5thTdy5Mic/05LmtL2u7okSpcxNmrUKMzTZaZNmjTxrEtMzWJ76XSp4uLFiz2zTLhoWr6ivy+z\nuKSsRYsWYeyMM87wrO0T0/IppeUbuqzezOzf//63Z21Va2Y2e/Zsz+nfJEqHtni+/PLLw5i24dal\nnldeeWWYN2fOHM9a/ojNpyWKukTXLLYR1aX66bJcLRvU5afIPytWrPD81ltvhbFx48Z57ty5s2dt\nP2tm9te//tVzem9I7+v5LC1z0Nat7du3D2NaVtG9e3fP1113XZinS7K1ZPu+++4L81588UXPPMOU\nP32+1PfMzKxGjRpF/ptmzZqF11rWX7169TCW9faxFaVp06ae9TODluObxTbZ6bmo5U76PJJeu1q3\nbu25Xbt2nvW9N4slUh9//HEY03NYr6+rVq0yVA6vvPKKZy1p7dq1a5inpU9pWSzX6P9Kt6WYPn26\n57Fjx3qeO3duiX5eWjKq52L6LKuv09bdWcOKGgAAAAAAgIzgixoAAAAAAICM4IsaAAAAAACAjCiV\nPWqKozXwGzOm+yboniVac2YW291pq+7vvvsuzNM2XQsWLCjmiFGURYsWeb744ovD2EUXXeQ5rdPU\n9q9ad/3ZZ5+Fedoicfz48Z7TVrD6M+bPn1+iY8fm0X2htAV3ly5dwjyt+dQ9FNL6UvalKTtaL5/u\n56XtCLUd6Kuvvhrm3X///WVzcMg0rQ83MxswYIBn3QvutNNOC/MaNmzo+cknnwxjei1fuHChZ71v\n54v0uqb7Yuj+I2bxXDzssMM8p+1FH3jgAc+PP/6453Q/IfYwqVi6L83atWvDWLrfzE+0ZbRZ3PuC\n97Nk9Bql97Ru3bqFebq3nubipO+B7jOi76k+u5rFNuuDBg0KY7qPou5f8/3335fomFDxZsyY4Vnf\nt/Qa36lTJ8/p550stoDOAt0fUfcxLeled+l+UfqepK3Ada7us5rF/U1ZUQMAAAAAAJARfFEDAAAA\nAACQEWVe+lQadPlSurxMLV26NOcY7e82j7bzTZddX3vttZ5feumlMKbLTOvWret58uTJYZ62YtMl\n8ih/2uLZzKxnz56edTm/liRxcHaXAAADFUlEQVSamQ0ZMsTzyy+/7Flb0aJsadtHPWfN1i8H/QnL\n7GEWSwfM4jVZy1v79OkT5mkb+ObNm4cxbYWrZazpUvB8pKUseu8zi88jWlY8dOjQMO/ZZ5/1rMu/\nae+aLXrupNddbf+sJVIjRowI89LWstgwLTO68sorPaflSL169fKclhfOmzfPs74HaWm9vnfvvfee\n5+HDh4d5urVC2n44/dtA5XbTTTd5btOmTRjTazelTiWjv6f03CmJ9FlWy5HTEn8919PW3VnDihoA\nAAAAAICM4IsaAAAAAACAjKhW3BLaatWqsb624kwrLCzcuzR+EO9jxSksLKy24VkbVp7vYY0aNcLr\n2267zfOxxx7rWZd2mpndfvvtnt99990yOroKUWnORe1G0blz5zCm5StNmjTxPGzYsDDvhhtuKJuD\nq2CV8VzMCu2QkHZNad26tecDDjggjC1ZssTzHXfc4XkzSncqzbmo5U7akdLMrGXLlp61TCO9bqYd\nhPJFVToXd999d89agvPoo4+GeW+//bbnXGWqGZOpc1E7wjZt2jSM7b///p61S51Z7D6rZVDa3Sed\np7bccsvwurJ1cKpK52Iey9S5mBV6TSgoKAhj2lk6K3Kdi6yoAQAAAAAAyAi+qAEAAAAAAMgIvqgB\nAAAAAADICPaoyS5qDvNAPtT/1q9f3/O5557recyYMWFeWtOdRzgX80A+nItZpHXg5dB+lnMxD3Au\n5gXOxTzAuZgXOBfzAHvUAAAAAAAAZBxf1AAAAAAAAGTEzyv6AABk2xdffOF5wIABnsuhzAFAxnEd\nAAAAKH2sqAEAAAAAAMgIvqgBAAAAAADICL6oAQAAAAAAyAj2qAFQYuxHAQAAAABlixU1AAAAAAAA\nGcEXNQAAAAAAABmxodKnpWY2vzwOBOtpXoo/i/exYvAe5gfex8qP9zA/8D5WfryH+YH3sfLjPcwP\nvI+VX873sFphYWF5HggAAAAAAAByoPQJAAAAAAAgI/iiBgAAAAAAICP4ogYAAAAAACAj+KIGAAAA\nAAAgI/iiBgAAAAAAICP+H+q/Frdnt3SYAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 1440x288 with 10 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-rWkMWvh9hNt",
        "colab_type": "text"
      },
      "source": [
        "###Task 7.1\n",
        "Make a small CNN that takes as input an 28x28x1 image and outputs a single scalar value (the rotation angle).\n",
        "\n",
        "The last layer of your network should be\n",
        "\n",
        "```\n",
        "angle = Dense(1)(x)\n",
        "```\n",
        "\n",
        "This is a dense layer without any activation function, hence the output of this layer is simply `angle = W*x + b`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f3EMvU5N94L3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "inputs = Input(shape=(28, 28, 1))\n",
        "\n",
        "# Encoder (convolutional base)\n",
        "# Your codes goes here\n",
        "\n",
        "# Decoder (predict angle)\n",
        "x = # Your code goes here\n",
        "angle = Dense(1)(x)\n",
        "\n",
        "angle_estimator = Model(input=inputs, output=angle)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H9eeVPFI-tVq",
        "colab_type": "text"
      },
      "source": [
        "###Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bW2RzBPJG0P-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "angle_estimator.compile(optimizer='rmsprop',loss='mse')\n",
        "\n",
        "# training loop\n",
        "angle_estimator.fit_generator(\n",
        "    RotNetDataGenerator(\n",
        "        x_train,\n",
        "        batch_size=128,\n",
        "        preprocess_func=None,\n",
        "        shuffle=True\n",
        "    ),\n",
        "    steps_per_epoch=80,\n",
        "    epochs=100,\n",
        "    validation_data=RotNetDataGenerator(\n",
        "        x_test,\n",
        "        batch_size=128,\n",
        "        preprocess_func=None),\n",
        "    validation_steps=80)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "98or9_7_6YNu",
        "colab_type": "text"
      },
      "source": [
        "###Model evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jYmyHlZ5MEKH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Set up generator\n",
        "datagen = RotNetDataGenerator(\n",
        "        x_test,\n",
        "        batch_size=32,\n",
        "        preprocess_func=None,\n",
        "        shuffle=False\n",
        "    )\n",
        "\n",
        "# Generate test images\n",
        "rotated_images, angles = datagen.next()\n",
        "print(\"Test images before rotation\")\n",
        "show_imgs(x_test)\n",
        "print(\"Test images after rotation\")\n",
        "show_imgs(rotated_images)\n",
        "\n",
        "# Predict angles\n",
        "angles_pred = angle_estimator.predict(rotated_images)\n",
        "\n",
        "# Plot angles\n",
        "plt.plot(angles)\n",
        "plt.plot(angles_pred)\n",
        "plt.legend(['True','Predicted']);"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dqtngpfbAaM8",
        "colab_type": "text"
      },
      "source": [
        "**HELP**: The predicted angles should match the true angles reasonably well. If your model fails to predict the angles, it could be because the model is underfitting. This indicates that the capacity of the model is too low. To increase capacity, add more connections in dense layers and/or more output maps in convolutional layers (I will explain this in the class)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xLrZfojj8Cd7",
        "colab_type": "text"
      },
      "source": [
        "###De-rotate images\n",
        "Now that we have estimated the rotation angles, let's de-rotate the images back to their original alignment."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H78jSdQoNC5-",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "de_rotated_images = np.zeros(rotated_images.shape)\n",
        "\n",
        "for i in range(rotated_images.shape[0]):\n",
        "  image = rotated_images[i,:,:,:].squeeze()\n",
        "\n",
        "  # get predicted angle\n",
        "  rotation_angle = -angles_pred[i]\n",
        "\n",
        "  # rotate the image\n",
        "  rows,cols = image.shape\n",
        "  M = cv2.getRotationMatrix2D(((cols-1)/2.0,(rows-1)/2.0),rotation_angle,1)\n",
        "  de_rotated_image = cv2.warpAffine(image,M,(cols,rows))\n",
        "\n",
        "  de_rotated_images[i,:,:,0] = de_rotated_image\n",
        "\n",
        "print('Images before rotation (ground truth)')\n",
        "show_imgs(x_test)\n",
        "print('Images after rotation (to be de-rotated)')\n",
        "show_imgs(rotated_images)\n",
        "print('De-rotated images (should match ground truth)')\n",
        "show_imgs(de_rotated_images)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NV6vSV3WCQZd",
        "colab_type": "text"
      },
      "source": [
        "##Task 8: Object detection\n",
        "**Motivation:** Classification CNNs assign one label to each input image. This is problematic if the image contains multiple objects. \n",
        "\n",
        "Object detection is about detecting and classifying multiple objects in images. Object detection networks output the corner coordinates of the bounding box of each detect object, along with a class label.\n",
        "\n",
        "There are many ways to implement object detection with CNNs. You may want to take a look at this 3-part tutorial:\n",
        "- https://towardsdatascience.com/beginners-guide-to-object-detection-algorithms-6620fb31c375\n",
        "- https://www.analyticsvidhya.com/blog/2018/11/implementation-faster-r-cnn-python-object-detection/?utm_source=blog&utm_medium=a-step-by-step-introduction-to-the-basic-object-detection-algorithms-part-1\n",
        "- https://www.analyticsvidhya.com/blog/2018/12/practical-guide-object-detection-yolo-framewor-python/?utm_source=blog&utm_medium=implementation-faster-r-cnn-python-object-detection\n",
        "\n",
        "The basic idea of our simple object detector below is as follows:\n",
        "\n",
        "- The output image is divided into a 2-by-2 grid\n",
        "- Each grid cell can contain one object, or no object. We want our model to output whether it thinks there is an object in the cell or not.\n",
        "- If a cell contains an object, we want our model to output the corner coordinates of the bounding box (relative to the center of the grid cell).\n",
        "- Also, if there is an object in a cell, we want to predict its class label.\n",
        "\n",
        "If the input image has shape 64x64, the output of the model will have shape 2x2x15:\n",
        "- 1 output per cell for the confidence (is there an object or not?)\n",
        "- 4 outputs per cell corresponding to the upper left and lower right coordinates of the bounding box\n",
        "- 10 outputs per cell corresponding to the class probabalities (assuming we have 10 classes).\n",
        "\n",
        "This totals 15 outputs per cell.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rHkjYyA09Slk",
        "colab_type": "text"
      },
      "source": [
        "###Task 8.1\n",
        "Fill in the empty spots below (marked with ???), then run the code block to set up the object detection network."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DBaW6otN9dHO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.layers import concatenate\n",
        "from keras.activations import softmax\n",
        "\n",
        "def softMaxAxis3(x):\n",
        "    return softmax(x,axis=3)\n",
        "\n",
        "input_img = Input(shape=(64, 64, 1))\n",
        "\n",
        "x = Conv2D(8, 33, activation='relu', padding='same')(input_img) #nb_filter, nb_row, nb_col\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "x = Conv2D(16, 3, activation='relu', padding='same')(x)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "x = Conv2D(32, 3, activation='relu', padding='same')(x)\n",
        "encoded = MaxPooling2D((2, 2), padding='same')(x)\n",
        "\n",
        "# Notice that all three outputs use the same encoder\n",
        "\n",
        "# 1. This predicts whether there is an object in a cell or not\n",
        "x = Conv2D(4, 3, activation='relu', padding='same')(encoded)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "x = Conv2D(4, 3, activation='relu', padding='same')(x)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "confidence = Conv2D(???, 1, activation='sigmoid', padding='same')(x)\n",
        "\n",
        "# 2. This predicts the bounding box coordinates for each cell\n",
        "x = Conv2D(4, 3, activation='relu', padding='same')(encoded)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "x = Conv2D(4, 3, activation='relu', padding='same')(x)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "box = Conv2D(???, 1, padding='same')(x)\n",
        "\n",
        "# 3. This predicts the class probabilities for each cell\n",
        "x = Conv2D(16, 3, activation='relu', padding='same')(encoded)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "x = Conv2D(16, 3, activation='relu', padding='same')(x)\n",
        "x = MaxPooling2D((2, 2), padding='same')(x)\n",
        "classes = Conv2D(???, 1, activation=softMaxAxis3, padding='same')(x)\n",
        "\n",
        "# Merge output\n",
        "merged = concatenate([confidence, box, classes])\n",
        "\n",
        "objdet = Model(input_img, merged)\n",
        "objdet.compile(optimizer='adagrad', loss='mse')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1FGcUFY18kOO",
        "colab_type": "text"
      },
      "source": [
        "###Quesions 8.1\n",
        "1. What does softMaxAxis3 do, and why is it needed?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VtLlialYGZVm",
        "colab_type": "text"
      },
      "source": [
        "###Training data\n",
        "Let's generate some training data for our object detector.\n",
        "\n",
        "The training images (`x_train_obj`) will be 64x64, where two of the four quadrants will contain one handwritten digit. This just serves to illustrate that we can teach a network to detect and classify more than one digit per input image.\n",
        "\n",
        "The output is (`y_train_obj`) is 2x2x15 as explained above:\n",
        "- 1 output per cell for the confidence (is there an object or not?)\n",
        "- 4 outputs per cell corresponding to the upper left and lower right coordinates of the bounding box\n",
        "- 10 outputs per cell corresponding to the class probabalities (assuming we have 10 classes).\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s8EGlKP8Gd28",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_train_obj = np.zeros((5000,64,64,1))\n",
        "y_train_obj = np.zeros((5000,2,2,15))\n",
        "\n",
        "for i in range(5000):\n",
        "  \n",
        "  ## 1 \n",
        "  q = np.random.randint(0,2) # 1st or 2nd image quadrant?\n",
        "  \n",
        "  # Random image\n",
        "  rand_ix = np.random.randint(0,x_train.shape[0])\n",
        "  x_off_start = np.random.randint(0,5)\n",
        "  y_off_start = np.random.randint(0,5)\n",
        "  x_train_obj[i,0+x_off_start:28+x_off_start,32*q+y_off_start:32*q+28+y_off_start,:] = x_train[rand_ix,:,:,:]\n",
        "  \n",
        "  # Set confidence to 1\n",
        "  y_train_obj[i,0,q,0] = 1\n",
        "\n",
        "  # Class label\n",
        "  label = np.argmax(y_train[rand_ix,:])\n",
        "  y_train_obj[i,0,q,5+label] = 1\n",
        "\n",
        "  # Bounding box corners coordinates\n",
        "  tmp = x_train[rand_ix,:,:,:]\n",
        "  ix = np.where(tmp>0.1)\n",
        "  rows = ix[0]\n",
        "  cols = ix[1]  \n",
        "  y_train_obj[i,0,q,1] = np.min(rows) + x_off_start\n",
        "  y_train_obj[i,0,q,2] = np.min(cols) + y_off_start\n",
        "  y_train_obj[i,0,q,3] = np.max(rows) + x_off_start\n",
        "  y_train_obj[i,0,q,4] = np.max(cols) + y_off_start\n",
        "\n",
        "  ## 2\n",
        "  q = np.random.randint(0,2) # 3rd or 4rd image quadrant?\n",
        "  \n",
        "  # Random image\n",
        "  rand_ix = np.random.randint(0,x_train.shape[0])\n",
        "  x_off_start = np.random.randint(0,5)\n",
        "  y_off_start = np.random.randint(0,5)\n",
        "  x_train_obj[i,32+x_off_start:32+28+x_off_start,32*q+y_off_start:32*q+28+y_off_start,:] = x_train[rand_ix,:,:,:]\n",
        "  \n",
        "  # Set confidence to 1\n",
        "  y_train_obj[i,1,q,0] = 1\n",
        "\n",
        "  # Class label\n",
        "  label = np.argmax(y_train[rand_ix,:])\n",
        "  y_train_obj[i,1,q,5+label] = 1\n",
        "\n",
        "  # Bounding box corners coordinates\n",
        "  tmp = x_train[rand_ix,:,:,:]\n",
        "  ix = np.where(tmp>0.1)\n",
        "  rows = ix[0]\n",
        "  cols = ix[1]  \n",
        "  y_train_obj[i,1,q,1] = np.min(rows) + x_off_start\n",
        "  y_train_obj[i,1,q,2] = np.min(cols) + y_off_start\n",
        "  y_train_obj[i,1,q,3] = np.max(rows) + x_off_start\n",
        "  y_train_obj[i,1,q,4] = np.max(cols) + y_off_start"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DLAKnAV0KkSC",
        "colab_type": "text"
      },
      "source": [
        "Display example outputs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3Usjz_hbI7k4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import cv2\n",
        "plt.figure(figsize=(20,4))\n",
        "for k in range(10):\n",
        "  result = np.tile(x_train_obj[k,:,:,:],(1,1,3))\n",
        "\n",
        "  for i in range(2):\n",
        "    for j in range(2):\n",
        "      object_present = np.round(y_train_obj[k,i,j,0])\n",
        "      if object_present:\n",
        "        class_index = np.argmax(y_train_obj[k,i,j,5:])\n",
        "        xmin = int(y_train_obj[k,i,j,1] + i*32) # row\n",
        "        ymin = int(y_train_obj[k,i,j,2] + j*32) # col\n",
        "        xmax = int(y_train_obj[k,i,j,3] + i*32) # row\n",
        "        ymax = int(y_train_obj[k,i,j,4] + j*32) # col\n",
        "        cv2.rectangle(result,(ymin,xmin),(ymax,xmax),(0,255,0),1)\n",
        "        cv2.putText(result, str(class_index), (ymin-10, xmin+10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0),lineType=cv2.LINE_AA)\n",
        "  ax = plt.subplot(2,5,k+1)\n",
        "  plt.imshow(result)\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q3x93OVGOLvO",
        "colab_type": "text"
      },
      "source": [
        "###Training\n",
        "We are ready to start training the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "osyiFWyhWUri",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "objdet.fit(x_train_obj, y_train_obj, epochs=50, batch_size=128,shuffle=True,verbose=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TveS0ULsOR75",
        "colab_type": "text"
      },
      "source": [
        "###Model evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c1kETljRYh7f",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "out = objdet.predict(x_train_obj[0:10,:,:,:])\n",
        "\n",
        "plt.figure(figsize=(20,4))\n",
        "for k in range(10):\n",
        "  result = np.tile(x_train_obj[k,:,:,:],(1,1,3))\n",
        "\n",
        "  for i in range(2):\n",
        "    for j in range(2):\n",
        "      object_present = np.round(out[k,i,j,0])\n",
        "      if object_present:\n",
        "        class_index = np.argmax(out[k,i,j,5:])\n",
        "        xmin = int(out[k,i,j,1] + i*32) # row\n",
        "        ymin = int(out[k,i,j,2] + j*32) # col\n",
        "        xmax = int(out[k,i,j,3] + i*32) # row\n",
        "        ymax = int(out[k,i,j,4] + j*32) # col\n",
        "        cv2.rectangle(result,(ymin,xmin),(ymax,xmax),(0,255,0),1)\n",
        "        cv2.putText(result, str(class_index), (ymin-10, xmin+10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0),lineType=cv2.LINE_AA)\n",
        "  ax = plt.subplot(2,5,k+1)\n",
        "  plt.imshow(result)\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sK3z-27PBVLn",
        "colab_type": "text"
      },
      "source": [
        "**Note:** Do not expect perfect results. Our object detection network is over-simplified compared to state-of-the-art. Or more precisely, the loss function is far from ideal, so we are optmizing *the wrong objective*, so to say."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QJzrjPFn455A",
        "colab_type": "text"
      },
      "source": [
        "##Task 9: Image segmentation\n",
        "Image segmentation is a form of image-to-image transformation. It outputs a softmax classification per pixel. So if the input image has size 64x64, and there are 10 classes, the output will have shape 64x64x10. That is, for each pixel the network outputs a vector of class probabilities:\n",
        "\n",
        "![alt text](https://miro.medium.com/max/498/1*P1ooLjeSwhxeJGyFawCvaQ.png)\n",
        "\n",
        "Source of inspiration: https://medium.com/100-shades-of-machine-learning/https-medium-com-100-shades-of-machine-learning-rediscovering-semantic-segmentation-part1-83e1462e0805\n",
        "\n",
        "Let's first make a new training set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qn4GeiUOLYe-",
        "colab_type": "text"
      },
      "source": [
        "###Training images\n",
        "The training images (x_train_seg) will be 64x64, where two of the four quadrants will contain one handwritten digit. This is to illustrate that we can teach a network to identify and segment more than one digit per input image.\n",
        "\n",
        "The target output (`y_train_seg`) will be 64x64x10, with a one-hot vector for each pixel indicating the correct class."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5jS8Dwk6H2Qr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_train_seg = np.zeros((5000,64,64,1))\n",
        "y_train_seg = np.zeros((5000,64,64,10))\n",
        "\n",
        "for i in range(5000):\n",
        "  \n",
        "  ## 1\n",
        "  q = np.random.randint(0,2) # 1st or 2nd image quadrant?\n",
        "  rand_ix = np.random.randint(0,x_train.shape[0])\n",
        "  x_off_start = np.random.randint(0,5)\n",
        "  y_off_start = np.random.randint(0,5)\n",
        "  x_train_seg[i,0+x_off_start:28+x_off_start,32*q+y_off_start:32*q+28+y_off_start,:] = x_train[rand_ix,:,:,:]\n",
        "\n",
        "  # Mask\n",
        "  tmp = x_train[rand_ix,:,:,:]\n",
        "  ix = np.where(tmp>0.1)\n",
        "  tmp = np.zeros(tmp.shape)\n",
        "  tmp[ix] = 1\n",
        "  label = np.argmax(y_train[rand_ix,:])\n",
        "  y_train_seg[i,0+x_off_start:28+x_off_start,32*q+y_off_start:32*q+28+y_off_start,label] = tmp.squeeze()\n",
        "  \n",
        "  ## 2\n",
        "  q = np.random.randint(0,2) #  3rd or 4th image quadrant?\n",
        "  rand_ix = np.random.randint(0,x_train.shape[0])\n",
        "  x_off_start = np.random.randint(0,5)\n",
        "  y_off_start = np.random.randint(0,5)\n",
        "  x_train_seg[i,32+x_off_start:32+28+x_off_start,32*q+y_off_start:32*q+28+y_off_start,:] = x_train[rand_ix,:,:,:]\n",
        "\n",
        "  # Mask\n",
        "  tmp = x_train[rand_ix,:,:,:]\n",
        "  ix = np.where(tmp>0.1)\n",
        "  tmp = np.zeros(tmp.shape)\n",
        "  tmp[ix] = 1\n",
        "  label = np.argmax(y_train[rand_ix,:])\n",
        "  y_train_seg[i,32+x_off_start:32+28+x_off_start,32*q+y_off_start:32*q+28+y_off_start,label] = tmp.squeeze()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sNh7RfpjM9rl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Show examples of input to output mappings\n",
        "for ex in range(5):\n",
        "  plt.figure(figsize=(20,6))\n",
        "  rand_ix = np.random.randint(0,5000)\n",
        "  ax = plt.subplot(1,11,1)\n",
        "  plt.imshow(x_train_seg[rand_ix,:,:,:].squeeze())\n",
        "  plt.title('Input image')\n",
        "  for i in range(10):\n",
        "    ax = plt.subplot(1,11,i+2)\n",
        "    plt.imshow(y_train_seg[rand_ix,:,:,i].squeeze())\n",
        "    plt.title(\"Out class \"+str(i))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sVfoiPrt_210",
        "colab_type": "text"
      },
      "source": [
        "###U-Net for image segmentation\n",
        "We will use a light version of the so-called U-Net:\n",
        "- https://arxiv.org/pdf/1505.04597.pdf\n",
        "- https://towardsdatascience.com/understanding-semantic-segmentation-with-unet-6be4f42d4b47\n",
        "\n",
        "![alt text](https://miro.medium.com/max/720/1*OkUrpDD6I0FpugA_bbYBJQ.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a_Yw6_PkkWkU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.layers import concatenate\n",
        "\n",
        "# See last layer of network\n",
        "def softMaxAxis3(x):\n",
        "    return softmax(x,axis=3)\n",
        "\n",
        "inputs = Input(shape=(64, 64, 1))\n",
        "\n",
        "# Encoder\n",
        "conv1 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)\n",
        "conv1 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv1)\n",
        "pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)\n",
        "conv2 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool1)\n",
        "conv2 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv2)\n",
        "pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)\n",
        "conv3 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool2)\n",
        "conv3 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv3)\n",
        "pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)\n",
        "conv4 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool3)\n",
        "conv4 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv4)\n",
        "drop4 = Dropout(0.5)(conv4)\n",
        "\n",
        "# Decoder\n",
        "up7 = Conv2D(32, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(drop4))\n",
        "merge7 = concatenate([conv3,up7], axis = 3)\n",
        "conv7 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge7)\n",
        "conv7 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv7)\n",
        "up8 = Conv2D(16, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv7))\n",
        "merge8 = concatenate([conv2,up8], axis = 3)\n",
        "conv8 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge8)\n",
        "conv8 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv8)\n",
        "up9 = Conv2D(8, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv8))\n",
        "merge9 = concatenate([conv1,up9], axis = 3)\n",
        "conv9 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge9)\n",
        "conv9 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)\n",
        "\n",
        "# Perform softmax on each pixel, so axis should be 3 because output has shape: batch_size x 64 x 64 x num_classes\n",
        "conv10 = Conv2D(num_classes, 1, activation = softMaxAxis3)(conv9)\n",
        "\n",
        "model = Model(input = inputs, output = conv10)\n",
        "model.summary()\n",
        "model.compile(optimizer = keras.optimizers.RMSprop(lr = 1e-3), loss = 'binary_crossentropy')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pFtbTD9CC9v_",
        "colab_type": "text"
      },
      "source": [
        "###Questions 9.1\n",
        "1. What does \"concatenate\" do?\n",
        "2. Which layers of the encoder are being concated with which layers of the decoder?\n",
        "3. Why do you think U-Net concatenates outputs from layers of the encoder and layers of the decoder?\n",
        "\n",
        "Hint: Use K.int_shape(...) to get the shapes of the layers that are being concatenated"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "By1_TT4pOrpu",
        "colab_type": "text"
      },
      "source": [
        "###Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iu1bZx4J65lM",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.fit(x_train_seg, y_train_seg, nb_epoch=30, batch_size=64,shuffle=True,verbose=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ECy9tSPLOuG2",
        "colab_type": "text"
      },
      "source": [
        "###Model evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fOz5f-5-7rn4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Pick 4 random examples\n",
        "rand_ix = np.random.randint(0,5000,4)\n",
        "out = model.predict(x_train_seg[rand_ix,:,:,:])\n",
        "ref = y_train_seg[rand_ix,:,:,:].squeeze()\n",
        "for k in range(4):\n",
        "  plt.figure(figsize=(20,4))\n",
        "  plt.subplot(2,11,1)\n",
        "  plt.imshow(x_train_seg[rand_ix[k],:,:,:].squeeze())\n",
        "  plt.title('Input image')\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)\n",
        "  for i in range(10):\n",
        "    ax = plt.subplot(2,11,i+2)\n",
        "    plt.imshow(out[k,:,:,i].squeeze(),vmin=0,vmax=1)\n",
        "    plt.title('Predicted ' + str(i))\n",
        "    ax.get_xaxis().set_visible(False)\n",
        "    ax.get_yaxis().set_visible(False)\n",
        "    ax = plt.subplot(2,11,11+i+2)\n",
        "    plt.imshow(ref[k,:,:,i].squeeze(),vmin=0,vmax=1)\n",
        "    plt.title('True ' + str(i))\n",
        "    ax.get_xaxis().set_visible(False)\n",
        "    ax.get_yaxis().set_visible(False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "psBtdJUcFt_n",
        "colab_type": "text"
      },
      "source": [
        "###Task 9.1\n",
        "Remove the concatenation layers (i.e., fill in the empty slots marked with ???). Then run the code block and train the model."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rwFBtvPcGZgz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "inputs = Input(shape=(64, 64, 1))\n",
        "\n",
        "# Encoder\n",
        "conv1 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs)\n",
        "conv1 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv1)\n",
        "pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)\n",
        "conv2 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool1)\n",
        "conv2 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv2)\n",
        "pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)\n",
        "conv3 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool2)\n",
        "conv3 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv3)\n",
        "pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)\n",
        "conv4 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool3)\n",
        "conv4 = Conv2D(64, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv4)\n",
        "drop4 = Dropout(0.5)(conv4)\n",
        "\n",
        "# Decoder\n",
        "up7 = Conv2D(32, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(drop4))\n",
        "#merge7 = concatenate([conv3,up7], axis = 3)\n",
        "conv7 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(???)\n",
        "conv7 = Conv2D(32, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv7)\n",
        "up8 = Conv2D(16, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv7))\n",
        "#merge8 = concatenate([conv2,up8], axis = 3)\n",
        "conv8 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(???)\n",
        "conv8 = Conv2D(16, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv8)\n",
        "up9 = Conv2D(8, 2, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(UpSampling2D(size = (2,2))(conv8))\n",
        "#merge9 = concatenate([conv1,up9], axis = 3)\n",
        "conv9 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(???)\n",
        "conv9 = Conv2D(8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv9)\n",
        "\n",
        "# Perform softmax on each pixel, so axis should be 3 because output has shape: batch_size x 64 x 64 x num_classes\n",
        "conv10 = Conv2D(num_classes, 1, activation = softMaxAxis3)(conv9)\n",
        "\n",
        "model = Model(input = inputs, output = conv10)\n",
        "model.summary()\n",
        "model.compile(optimizer = keras.optimizers.RMSprop(lr = 1e-3), loss = 'binary_crossentropy')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YCHTqAt7HLiH",
        "colab_type": "text"
      },
      "source": [
        "###Training\n",
        "Train the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lVlwZA9SGPao",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.fit(x_train_seg, y_train_seg, nb_epoch=30, batch_size=64,shuffle=True,verbose=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "scV-vmHuHPLL",
        "colab_type": "text"
      },
      "source": [
        "###Model evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sRe5Y2gRGwGc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Pick 4 random examples\n",
        "rand_ix = np.random.randint(0,5000,4)\n",
        "out = model.predict(x_train_seg[rand_ix,:,:,:])\n",
        "ref = y_train_seg[rand_ix,:,:,:].squeeze()\n",
        "for k in range(4):\n",
        "  plt.figure(figsize=(20,4))\n",
        "  plt.subplot(2,11,1)\n",
        "  plt.imshow(x_train_seg[rand_ix[k],:,:,:].squeeze())\n",
        "  plt.title('Input image')\n",
        "  ax.get_xaxis().set_visible(False)\n",
        "  ax.get_yaxis().set_visible(False)\n",
        "  for i in range(10):\n",
        "    ax = plt.subplot(2,11,i+2)\n",
        "    plt.imshow(out[k,:,:,i].squeeze(),vmin=0,vmax=1)\n",
        "    plt.title('Predicted ' + str(i))\n",
        "    ax.get_xaxis().set_visible(False)\n",
        "    ax.get_yaxis().set_visible(False)\n",
        "    ax = plt.subplot(2,11,11+i+2)\n",
        "    plt.imshow(ref[k,:,:,i].squeeze(),vmin=0,vmax=1)\n",
        "    plt.title('True ' + str(i))\n",
        "    ax.get_xaxis().set_visible(False)\n",
        "    ax.get_yaxis().set_visible(False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MsDvcgV9HS0j",
        "colab_type": "text"
      },
      "source": [
        "###Questions 9.2\n",
        "1. Compare the results with and without the concatenation layers. What do you observe? Can you explain it?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jJrlKE5CPWvz",
        "colab_type": "text"
      },
      "source": [
        "##10 Ideas for further work:\n",
        "###Few-shot learning (or one-shot learning) with Siamese networks\n",
        "What? Why? How?\n",
        "\n",
        "Can you modify the code below to train on just 10 examples of each class?\n",
        "\n",
        "Read here: https://towardsdatascience.com/one-shot-learning-with-siamese-networks-using-keras-17f34e75bb3d\n",
        "\n",
        "Code here: https://keras.io/examples/mnist_siamese/"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9CYmzCDXI3Q3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from __future__ import absolute_import\n",
        "from __future__ import print_function\n",
        "import numpy as np\n",
        "\n",
        "import random\n",
        "from keras.datasets import mnist\n",
        "from keras.models import Model\n",
        "from keras.layers import Input, Flatten, Dense, Dropout, Lambda\n",
        "from keras.optimizers import RMSprop\n",
        "from keras import backend as K\n",
        "\n",
        "num_classes = 10\n",
        "epochs = 20\n",
        "\n",
        "def euclidean_distance(vects):\n",
        "    x, y = vects\n",
        "    sum_square = K.sum(K.square(x - y), axis=1, keepdims=True)\n",
        "    return K.sqrt(K.maximum(sum_square, K.epsilon()))\n",
        "\n",
        "def eucl_dist_output_shape(shapes):\n",
        "    shape1, shape2 = shapes\n",
        "    return (shape1[0], 1)\n",
        "\n",
        "def contrastive_loss(y_true, y_pred):\n",
        "    '''Contrastive loss from Hadsell-et-al.'06\n",
        "    http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf\n",
        "    '''\n",
        "    margin = 1\n",
        "    sqaure_pred = K.square(y_pred)\n",
        "    margin_square = K.square(K.maximum(margin - y_pred, 0))\n",
        "    return K.mean(y_true * sqaure_pred + (1 - y_true) * margin_square)\n",
        "\n",
        "def create_pairs(x, digit_indices):\n",
        "    '''Positive and negative pair creation.\n",
        "    Alternates between positive and negative pairs.\n",
        "    '''\n",
        "    pairs = []\n",
        "    labels = []\n",
        "    n = min([len(digit_indices[d]) for d in range(num_classes)]) - 1\n",
        "    for d in range(num_classes):\n",
        "        for i in range(n):\n",
        "            z1, z2 = digit_indices[d][i], digit_indices[d][i + 1]\n",
        "            pairs += [[x[z1], x[z2]]]\n",
        "            inc = random.randrange(1, num_classes)\n",
        "            dn = (d + inc) % num_classes\n",
        "            z1, z2 = digit_indices[d][i], digit_indices[dn][i]\n",
        "            pairs += [[x[z1], x[z2]]]\n",
        "            labels += [1, 0]\n",
        "    return np.array(pairs), np.array(labels)\n",
        "\n",
        "def create_base_network(input_shape):\n",
        "    '''Base network to be shared (eq. to feature extraction).\n",
        "    '''\n",
        "    input = Input(shape=input_shape)\n",
        "    x = Flatten()(input)\n",
        "    x = Dense(128, activation='relu')(x)\n",
        "    x = Dropout(0.1)(x)\n",
        "    x = Dense(128, activation='relu')(x)\n",
        "    x = Dropout(0.1)(x)\n",
        "    x = Dense(128, activation='relu')(x)\n",
        "    return Model(input, x)\n",
        "\n",
        "def compute_accuracy(y_true, y_pred):\n",
        "    '''Compute classification accuracy with a fixed threshold on distances.\n",
        "    '''\n",
        "    pred = y_pred.ravel() < 0.5\n",
        "    return np.mean(pred == y_true)\n",
        "\n",
        "def accuracy(y_true, y_pred):\n",
        "    '''Compute classification accuracy with a fixed threshold on distances.\n",
        "    '''\n",
        "    return K.mean(K.equal(y_true, K.cast(y_pred < 0.5, y_true.dtype)))\n",
        "\n",
        "# the data, split between train and test sets\n",
        "(x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
        "x_train = x_train.astype('float32')\n",
        "x_test = x_test.astype('float32')\n",
        "x_train /= 255\n",
        "x_test /= 255\n",
        "input_shape = x_train.shape[1:]\n",
        "\n",
        "# create training+test positive and negative pairs\n",
        "digit_indices = [np.where(y_train == i)[0] for i in range(num_classes)]\n",
        "tr_pairs, tr_y = create_pairs(x_train, digit_indices)\n",
        "\n",
        "digit_indices = [np.where(y_test == i)[0] for i in range(num_classes)]\n",
        "te_pairs, te_y = create_pairs(x_test, digit_indices)\n",
        "\n",
        "# network definition\n",
        "base_network = create_base_network(input_shape)\n",
        "\n",
        "input_a = Input(shape=input_shape)\n",
        "input_b = Input(shape=input_shape)\n",
        "\n",
        "# because we re-use the same instance `base_network`,\n",
        "# the weights of the network\n",
        "# will be shared across the two branches\n",
        "processed_a = base_network(input_a)\n",
        "processed_b = base_network(input_b)\n",
        "\n",
        "distance = Lambda(euclidean_distance,\n",
        "                  output_shape=eucl_dist_output_shape)([processed_a, processed_b])\n",
        "\n",
        "model = Model([input_a, input_b], distance)\n",
        "\n",
        "# train\n",
        "rms = RMSprop()\n",
        "model.compile(loss=contrastive_loss, optimizer=rms, metrics=[accuracy])\n",
        "model.fit([tr_pairs[:, 0], tr_pairs[:, 1]], tr_y,\n",
        "          batch_size=128,\n",
        "          epochs=epochs,\n",
        "          validation_data=([te_pairs[:, 0], te_pairs[:, 1]], te_y))\n",
        "\n",
        "# compute final accuracy on training and test sets\n",
        "y_pred = model.predict([tr_pairs[:, 0], tr_pairs[:, 1]])\n",
        "tr_acc = compute_accuracy(tr_y, y_pred)\n",
        "y_pred = model.predict([te_pairs[:, 0], te_pairs[:, 1]])\n",
        "te_acc = compute_accuracy(te_y, y_pred)\n",
        "\n",
        "print('* Accuracy on training set: %0.2f%%' % (100 * tr_acc))\n",
        "print('* Accuracy on test set: %0.2f%%' % (100 * te_acc))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a_5WzyIk9LFB",
        "colab_type": "text"
      },
      "source": [
        "###Generative Adversarial Networks\n",
        "What on earth is this thing doing? See if you can guess :-)\n",
        "\n",
        "Read here: https://towardsdatascience.com/dcgans-deep-convolutional-generative-adversarial-networks-c7f392c2c8f8\n",
        "\n",
        "Code here: https://github.com/eriklindernoren/Keras-GAN/blob/master/dcgan/dcgan.py"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8eYaGm4oV2oo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from __future__ import print_function, division\n",
        "\n",
        "from keras.datasets import mnist\n",
        "from keras.layers import Input, Dense, Reshape, Flatten, Dropout\n",
        "from keras.layers import BatchNormalization, Activation, ZeroPadding2D\n",
        "from keras.layers.advanced_activations import LeakyReLU\n",
        "from keras.layers.convolutional import UpSampling2D, Conv2D\n",
        "from keras.models import Sequential, Model\n",
        "from keras.optimizers import Adam\n",
        "\n",
        "import sys\n",
        "\n",
        "class DCGAN():\n",
        "    def __init__(self):\n",
        "        # Input shape\n",
        "        self.img_rows = 28\n",
        "        self.img_cols = 28\n",
        "        self.channels = 1\n",
        "        self.img_shape = (self.img_rows, self.img_cols, self.channels)\n",
        "        self.latent_dim = 10\n",
        "\n",
        "        optimizer = Adam(0.0002, 0.5)\n",
        "\n",
        "        # Build and compile the discriminator\n",
        "        self.discriminator = self.build_discriminator()\n",
        "        self.discriminator.compile(loss='binary_crossentropy',\n",
        "            optimizer=optimizer,\n",
        "            metrics=['accuracy'])\n",
        "\n",
        "        # Build the generator\n",
        "        self.generator = self.build_generator()\n",
        "\n",
        "        # The generator takes noise as input and generates imgs\n",
        "        z = Input(shape=(self.latent_dim,))\n",
        "        img = self.generator(z)\n",
        "\n",
        "        # For the combined model we will only train the generator\n",
        "        self.discriminator.trainable = False\n",
        "\n",
        "        # The discriminator takes generated images as input and determines validity\n",
        "        valid = self.discriminator(img)\n",
        "\n",
        "        # The combined model  (stacked generator and discriminator)\n",
        "        # Trains the generator to fool the discriminator\n",
        "        self.combined = Model(z, valid)\n",
        "        self.combined.compile(loss='binary_crossentropy', optimizer=optimizer)\n",
        "\n",
        "    def build_generator(self):\n",
        "\n",
        "        model = Sequential()\n",
        "\n",
        "        model.add(Dense(128 * 7 * 7, activation=\"relu\", input_dim=self.latent_dim))\n",
        "        model.add(Reshape((7, 7, 128)))\n",
        "        model.add(UpSampling2D())\n",
        "        model.add(Conv2D(128, kernel_size=3, padding=\"same\"))\n",
        "        model.add(BatchNormalization(momentum=0.8))\n",
        "        model.add(Activation(\"relu\"))\n",
        "        model.add(UpSampling2D())\n",
        "        model.add(Conv2D(64, kernel_size=3, padding=\"same\"))\n",
        "        model.add(BatchNormalization(momentum=0.8))\n",
        "        model.add(Activation(\"relu\"))\n",
        "        model.add(Conv2D(self.channels, kernel_size=3, padding=\"same\"))\n",
        "        model.add(Activation(\"tanh\"))\n",
        "\n",
        "        model.summary()\n",
        "\n",
        "        noise = Input(shape=(self.latent_dim,))\n",
        "        img = model(noise)\n",
        "\n",
        "        return Model(noise, img)\n",
        "\n",
        "    def build_discriminator(self):\n",
        "\n",
        "        model = Sequential()\n",
        "\n",
        "        model.add(Conv2D(32, kernel_size=3, strides=2, input_shape=self.img_shape, padding=\"same\"))\n",
        "        model.add(LeakyReLU(alpha=0.2))\n",
        "        model.add(Dropout(0.25))\n",
        "        model.add(Conv2D(64, kernel_size=3, strides=2, padding=\"same\"))\n",
        "        model.add(ZeroPadding2D(padding=((0,1),(0,1))))\n",
        "        model.add(BatchNormalization(momentum=0.8))\n",
        "        model.add(LeakyReLU(alpha=0.2))\n",
        "        model.add(Dropout(0.25))\n",
        "        model.add(Conv2D(128, kernel_size=3, strides=2, padding=\"same\"))\n",
        "        model.add(BatchNormalization(momentum=0.8))\n",
        "        model.add(LeakyReLU(alpha=0.2))\n",
        "        model.add(Dropout(0.25))\n",
        "        model.add(Conv2D(256, kernel_size=3, strides=1, padding=\"same\"))\n",
        "        model.add(BatchNormalization(momentum=0.8))\n",
        "        model.add(LeakyReLU(alpha=0.2))\n",
        "        model.add(Dropout(0.25))\n",
        "        model.add(Flatten())\n",
        "        model.add(Dense(1, activation='sigmoid'))\n",
        "\n",
        "        model.summary()\n",
        "\n",
        "        img = Input(shape=self.img_shape)\n",
        "        validity = model(img)\n",
        "\n",
        "        return Model(img, validity)\n",
        "\n",
        "    def train(self, epochs, batch_size=128, save_interval=500):\n",
        "\n",
        "        # Load the dataset\n",
        "        (X_train, _), (_, _) = mnist.load_data()\n",
        "\n",
        "        # Rescale -1 to 1\n",
        "        X_train = X_train / 127.5 - 1.\n",
        "        X_train = np.expand_dims(X_train, axis=3)\n",
        "\n",
        "        # Adversarial ground truths\n",
        "        valid = np.ones((batch_size, 1))\n",
        "        fake = np.zeros((batch_size, 1))\n",
        "\n",
        "        for epoch in range(epochs):\n",
        "\n",
        "            # ---------------------\n",
        "            #  Train Discriminator\n",
        "            # ---------------------\n",
        "\n",
        "            # Select a random half of images\n",
        "            idx = np.random.randint(0, X_train.shape[0], batch_size)\n",
        "            imgs = X_train[idx]\n",
        "\n",
        "            # Sample noise and generate a batch of new images\n",
        "            noise = np.random.normal(0, 1, (batch_size, self.latent_dim))\n",
        "            gen_imgs = self.generator.predict(noise)\n",
        "\n",
        "            # Train the discriminator (real classified as ones and generated as zeros)\n",
        "            d_loss_real = self.discriminator.train_on_batch(imgs, valid)\n",
        "            d_loss_fake = self.discriminator.train_on_batch(gen_imgs, fake)\n",
        "            d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)\n",
        "\n",
        "            # ---------------------\n",
        "            #  Train Generator\n",
        "            # ---------------------\n",
        "\n",
        "            # Train the generator (wants discriminator to mistake images as real)\n",
        "            g_loss = self.combined.train_on_batch(noise, valid)\n",
        "\n",
        "            # Plot the progress\n",
        "            #print (\"%d [D loss: %f, acc.: %.2f%%] [G loss: %f]\" % (epoch, d_loss[0], 100*d_loss[1], g_loss))\n",
        "\n",
        "            # If at save interval => save generated image samples\n",
        "            if epoch % save_interval == 0:\n",
        "                print (\"%d [D loss: %f, acc.: %.2f%%] [G loss: %f]\" % (epoch, d_loss[0], 100*d_loss[1], g_loss))\n",
        "                self.save_imgs(epoch)\n",
        "\n",
        "    def save_imgs(self, epoch):\n",
        "        r, c = 5, 5\n",
        "        noise = np.random.normal(0, 1, (r * c, self.latent_dim))\n",
        "        gen_imgs = self.generator.predict(noise)\n",
        "\n",
        "        # Rescale images 0 - 1\n",
        "        gen_imgs = 0.5 * gen_imgs + 0.5\n",
        "\n",
        "        plt.figure()\n",
        "        fig, axs = plt.subplots(r, c)\n",
        "        cnt = 0\n",
        "        for i in range(r):\n",
        "            for j in range(c):\n",
        "                axs[i,j].imshow(gen_imgs[cnt, :,:,0], cmap='gray')\n",
        "                axs[i,j].axis('off')\n",
        "                cnt += 1\n",
        "        #fig.savefig(\"mnist_%d.png\" % epoch)\n",
        "        plt.show()\n",
        "\n",
        "if __name__ == '__main__':\n",
        "    dcgan = DCGAN()\n",
        "    dcgan.train(epochs=4000, batch_size=32, save_interval=500)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mOnzOzU0LtNK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}