dlnd_image_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Image Classification\n",
    "In this project, I'll classify images from the [CIFAR-10 dataset](https://www.cs.toronto.edu/~kriz/cifar.html).  The dataset consists of airplanes, dogs, cats, and other objects. I've preprocessed the images, then trained a convolutional neural network on all the samples. The images needed to be normalized and the labels needed to be one-hot encoded. \n",
    "## Get the Data\n",
    "Run the following cell to download the [CIFAR-10 dataset for python](https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "CIFAR-10 Dataset: 171MB [00:19, 8.66MB/s]                              \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All files found!\n"
     ]
    }
   ],
   "source": [
    "from urllib.request import urlretrieve\n",
    "from os.path import isfile, isdir\n",
    "from tqdm import tqdm\n",
    "import problem_unittests as tests\n",
    "import tarfile\n",
    "\n",
    "cifar10_dataset_folder_path = 'cifar-10-batches-py'\n",
    "\n",
    "class DLProgress(tqdm):\n",
    "    last_block = 0\n",
    "\n",
    "    def hook(self, block_num=1, block_size=1, total_size=None):\n",
    "        self.total = total_size\n",
    "        self.update((block_num - self.last_block) * block_size)\n",
    "        self.last_block = block_num\n",
    "\n",
    "if not isfile('cifar-10-python.tar.gz'):\n",
    "    with DLProgress(unit='B', unit_scale=True, miniters=1, desc='CIFAR-10 Dataset') as pbar:\n",
    "        urlretrieve(\n",
    "            'https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz',\n",
    "            'cifar-10-python.tar.gz',\n",
    "            pbar.hook)\n",
    "\n",
    "if not isdir(cifar10_dataset_folder_path):\n",
    "    with tarfile.open('cifar-10-python.tar.gz') as tar:\n",
    "        tar.extractall()\n",
    "        tar.close()\n",
    "\n",
    "\n",
    "tests.test_folder_path(cifar10_dataset_folder_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Exploring the Data\n",
    "The dataset is broken into batches to prevent your machine from running out of memory.  The CIFAR-10 dataset consists of 5 batches, named `data_batch_1`, `data_batch_2`, etc.. Each batch contains the labels and images that are one of the following:\n",
    "* airplane\n",
    "* automobile\n",
    "* bird\n",
    "* cat\n",
    "* deer\n",
    "* dog\n",
    "* frog\n",
    "* horse\n",
    "* ship\n",
    "* truck\n",
    "\n",
    "Understanding a dataset is part of making predictions on the data. The `batch_id` is the id for a batch (1-5). The `sample_id` is the id for a image and label pair in the batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Stats of batch 1:\n",
      "Samples: 10000\n",
      "Label Counts: {0: 1005, 1: 974, 2: 1032, 3: 1016, 4: 999, 5: 937, 6: 1030, 7: 1001, 8: 1025, 9: 981}\n",
      "First 20 Labels: [6, 9, 9, 4, 1, 1, 2, 7, 8, 3, 4, 7, 7, 2, 9, 9, 9, 3, 2, 6]\n",
      "\n",
      "Example of Image 5:\n",
      "Image - Min Value: 0 Max Value: 252\n",
      "Image - Shape: (32, 32, 3)\n",
      "Label - Label Id: 1 Name: automobile\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfoAAAH0CAYAAADVH+85AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAWJQAAFiUBSVIk8AAAHF9JREFUeJzt3UmPZOl1HuAvxsyMrKzKqsqau6rYA5vNbropkjJJmYIs\nUIBXWtn+BV7YO/8Yr73wymtDNAwIggwSMEmBNMeW2Wz2VOzumquyco6M2QttzI2Bc5gChYPn2Z88\nEd+9cd+8q7ezWq0aAFBT9w/9AQCAfzyCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhQl6AChM0ANA\nYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bh/T/0B/jH8l/+w79fZebGx9PwTK+f\n+3+pc/tGeGZvtJHa9faFYWruk1/+LDzznR/+PLVrbzILz/R6ybPvdFJzg7X18MylKzupXec34t/t\n83eupHb9+be+Hp6Zz+LXq7XWnu0fpeYGWxfDM+9+8NvUrr/97g/jQ8nnwNogN3dhMAjPDPuL1K5p\n4lrPZ7nfWFstU2NrvbXwzMkq/rxvrbUXp/F46eZ+Lu073/+75EH+P7t/3z8AAPzTJegBoDBBDwCF\nCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGFl2+te3P84NddfxJuT\nBv1UUV67v5qEZ94f5yqQ3v7iK6m55TT+Ga/t5NraNlLfLXf22fa6k0n8PPZ3X6R2HXXiTWOT03Fq\n15e/+o3wzOzkNLXr2fPceVxbjzc3LqcHqV0ba/H7atlyrWtXt86l5r70ymvhmadP7qd2jceH4Zmj\no1xLYevGW/laa22tPw/P3Lx+IbVrNrwanvngV/dSu86CN3oAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9\nABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUFjZUpuPT9dScyfj/fDMsJMr92iLeKFCtzNMrXr2\n28epuZ88+Cw88+snudKS1SReSpEtp1lfX0/NzebxopnWzf0/vb4Rv4f3xrlilR+983545sblXCHI\nZJ67ZpkCo7XkE24wSHzG3NG3L7z6amruc3fuhme2t0apXY8e3gvPLGe55+K5izdSc4tBvPRotJYr\n3rm5Ey8i+rSXO/uz4I0eAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCY\noAeAwgQ9ABQm6AGgsLLtdeNeriFrtxtvJ+ssJqldl/vx4z93/mJq1+lxvJWvtdb2DuPf7eB0ltq1\nSpz9YpFok2ut9ZKfsZ/533gWb11rrbXjafzsz61yu370i1+GZ15/7bXUrjdevZOa6w/j7V+f+1yu\nGe54OQjPPH74NLXr4HCcmmvrm+GRP/6zt1Orfv7j74VnxvN4G2VrrR3Oci1vz4/jz8ZL41zD3q3e\nYXjm9Cjb2vj780YPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhQl6AChM0ANA\nYYIeAAorW2qz1tlNzd0YxYsYtlu8AKO11i5d3AjPfLyKlym01trmxjI1t9aJl6SMOrnbara5Fp+Z\n58ppTie5IqJF4n/jjVGupGO4Fr+vrt++kdp186Xb4ZlnR7lCkEcHuRKXb3zj6+GZ3cePUrv+9b/5\nVnjmf/z3v07t+uEP/i41d+dLXw3PfPvtr6V2fXj/o/DMx9//cWrX/nQrNXc0jz/jvvjP42fYWmvj\n2YvwzM7OemrXWfBGDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNAD\nQGGCHgAKE/QAUFjZ9rrhZu6rvbJ1NTzz8iq368Iw0Wa0/1lq12g73gzXWmvHw5PwzHKwSO364z+K\nN0lduxq/Xq219tEHH6TmPv3kfnim28u1G67m8Xa49W7u7P/kG/Gzfxq/NVprrf3oe99Nzb333p3w\nzGKc/JCbF8Mje8e5RsSjWe5964OHz8Mzx8teatfxPP4Zn+zlzmOyfi419/m7r4Rntq/dTO16+jx+\n9t/+9lupXWfBGz0AFCboAaAwQQ8AhQl6AChM0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCF\nCXoAKEzQA0BhZdvrjqa5xrALvc3wzOzZi9SuT/fiTWh/+uU3UrvG0+PU3K1lfGZ9tErt+uZ2/Ozf\nvLKT2nWyzH3GZ2vxFsCT/dz9sZjGZ/rTw9Suu598HJ7Z2Jundl26sp2am/39z8Iz2ebAH/7q3fDM\new8epHadznMtb/c/iTdZPnn+NLXr61/5Znjm7vbt1K7/9F//W2puOn4UnvnJj5+ldj1+/GF45qt/\nkXt2nwVv9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGg\nsLKlNld666m5W60Xnjl/fiu16+cv4qUULyb7qV13r99Izf3bJy+HZwYHuQKdy+/Hz2Ptw4epXYvl\nLDX3uU58ZrBIDLXWuv34Pbzo5EpcJj/6aXjmQrKMZbkTLy9qrbXFPNGwdLBI7TrfOxeemRzn7vtL\n8UdOa6210Wocnjl49NvUrltffD08s7WZewZ//dVbqbkn+/EWqEdHJ6ldJye74ZmP3n8/tesseKMH\ngMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAorGx7\n3Rtbo9Tc5vNn4ZleN9Gq1Vp7/aWXwjOHj5+mdrVVrkHtVmcVnhkNc7t6iUaozjL++VprLd5z9Q8m\n3cT/xsO11K7BKv7d+pmGt9baoBtv85tt5WrXVie51rv5JH4ei5a7F69143fItzdyrXzTzjA1t7h5\nLTyzfu9eatdJ5iMmWz3feuO11NyNk/g1uzGbp3a9/urN8MxrO/FGxLPijR4AChP0AFCYoAeAwgQ9\nABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFFa21Gb3wUepuck8XoIx7uWKRE4u\nxEsONk7i5SOttXb67oepuUVvEZ6Zb+Zuq24vXkqxlixx6bT11Nw8UQ60WOY+42owiM+kNuXm+ldf\nSe3a2su9X5wmLtn07sXUrovzo/DM5mmuKmm+lytWOXqyH545efD91K6H//sX4Znzb72e2vX8Ua64\nazq6FJ6Zj1Or2snzF+GZg0G2Suv3540eAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bh\ngh4AChP0AFCYoAeAwgQ9ABQm6AGgsLLtdc+P9lJznx6fhmfmy1z71LBzPTwzuriT2vV8fJiau95b\nC89snOb+f1wcxJv5JtNcm1/byZ3j5uuvhWdOE01orbV29OwgPLO2jLfrtdZabzIJz0ye5u6ptpZr\nlOtsx9se+51cn9/yIP4c2Hgr1+bXhvHv1Vproyfx6rXj+/dTu/Z+/UF4ZvnJ49SurUtbqbnd7XhL\n5PNHud/mwyefhWdeHt5I7ToL3ugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGg\nMEEPAIUJegAoTNADQGGCHgAKK9te9+I03j7VWmuPTuJtRrOD49SunWtXwjOr21dTu9Yu5hqh1g7i\nzXz9B09Tu6ZHJ+GZoxZvrGqttcW5jdTc4O6d8Ey/s0jt2tyOn8fsN5+kds0SLYCn3Vxz4NafvZma\nO9l7Fh9679epXW2eeAd6mPh8rbXJMte0Obh+Mzxz/V9+M7VrbaMXntn9zYepXdsn8V2ttXbhbrxp\n85NHuYa9jV68FXEwGKZ2nQVv9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCY\noAeAwgQ9ABQm6AGgsLKlNrdvv5Sa6358PzyzMU6taotpvBhhrTNI7XpxfJCa+8Gnn4Vnbp4epna9\n0eIHOUmUsbTW2vh+/Dq31tr0p7+K72rx69xaa51bt8Izp69fT+06mY/CM2+/miunOe6eS82NH9wL\nzwz3c+VW8/PxApLpJ8lCoce5UqzB1SfhmZNruVKswaUL4ZmLf/HV1K69Tx+m5rZ34mU4Xz13N7Xr\nb/7Xi/DM2na8xOyseKMHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm\n6AGgMEEPAIUJegAorGx73fWb11Jzh/efhWdGFzupXa2zFh4ZdHO7Hj57npr7z7/4P+GZL1zOtZP9\nx/XN8Mwo+a/q6vgoNbf7Try9bvdKvPmrtdY+msRbzabJprybr98Mz9y5mPte04ePU3PnEq1mneU0\ntasdxn9na92N1KqD8UlqbvHRR+GZ1YNHqV0vtuLPqs0v5BpEb778amru9FH8vroyij9zWmvtK196\nLTxz++XceZwFb/QAUJigB4DCBD0AFCboAaAwQQ8AhQl6AChM0ANAYYIeAAoT9ABQmKAHgMIEPQAU\nJugBoLCypTb7ixepuf5qPzwz6OeOcdqLF5DszcepXbvjXNnJfBX/bgeDXLnH/cEoPLO9mqd2Tbu5\nudVqEp7ZX+ZKSz57Ei+1Od9dT+16kbhkf3X/r1K7vnDrVmru1Uvx73Z57Xpq1/G9++GZxTh+vVpr\nbbXI3YsvXjxN7Mo9B6br8VKb2X68IKy11qa/fD81N0oUOk3WB6ldd998Kzwze/Db1K6z4I0eAAoT\n9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgsLLtdcPV\nMjXXX87CMzvdXAPStBdvrerPpqldJ6e587h15Up45qWXb6d23T9KNPOtcm1cw2RrVWce/8lMl/HG\nu9Zau3F5JzzTzxWhtYOnj8Izq91cK9+D57mWt/3RMDxzZxL/PbfWWvdZvL2ujXOH353n3rfG8/g5\nnixyz49VohVxNO6kdj28/1lqbtSJ7zue567Z9iQ+t/P266ldZ8EbPQAUJugBoDBBDwCFCXoAKEzQ\nA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAorGypzcZ4lJp7ML8QnrnaPU3tujjeC8/0\nnzxM7ZofvkjNffHNl8Mzd77w+dSu3V+8F5650emldrVBrgxnsIr/b7xxlCtx6bf4ZxyNNlK7fvPh\nvfDMznHuPeGVz11KzX02jBfUPP4g93vZONwNz3TmuXuqs8jdw6eJUqxpN3fNpsfxXbuLw9Su0eh8\nau5wGi+POp7krtnu/cfhmf6d66ldZ8EbPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeA\nwgQ9ABQm6AGgMEEPAIUJegAoTNADQGFl2+v2j+NNV6219t39eEvT/HJqVfvWchqe2XjyKLVrfXaS\nmvvK174dnrl5+7XUru/86J3wzP4k1xy46Ofuj1miLW9j1UntOv0sfq17l3LNcK9c3AnPnC72U7v6\nm8PU3Nt/+vXwzG680Owf5n7yJDwzWeaa0Jb9tdTcOHFfbW4mH1Ybm+GR8TDXyre8fDE1d9ri+x49\njbcUttba/t6z8MyLX7+f2vWXqanf5Y0eAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bh\ngh4AChP0AFCYoAeAwgQ9ABQm6AGgsLLtddODB6m5D54/Ds+MZ7k2ru2X4o1hXx7kWte2+vFWvtZa\ne/n27fDM+XO5BrXJIt7mNzmJz7TW2nCwSM2druL7ht3c/TGcxq/ZeDfXxtXtxx8Fy16ure3x81wD\n44t3fxWeGa3nGtQO18/FZzZGqV2Tc1upuePj4/DMaCf329ydxlsiD+e531h3Nk7NPXx0FN+1Hm/l\na621g1n8ObB5kGt7PAve6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8A\nhQl6AChM0ANAYWVLbf7V3VxZwdPdeJnFjz8+Se36m3vxkoONV3Lfa3RuLTW31YsXdcwO4wUYrbW2\n6MRLMI4nuV3rvdytv+gl/jfu5P6fXnbjc7vH8WKP1lpbncYLdIbHubOf7eWKiFYffhKeGSXfZaaj\n8+GZd+aT1K57z56k5taX8ZnhMlcYM1iP/146s05q1+lerpjpeBUvB+qfG6R2LQbx73b34nZq11nw\nRg8AhQl6AChM0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFBY\n2fa612/mvtq/G90Jz9xeu5/a9T/fizeN/e29WWrXH929mZo7+vDj8Mxe8v/H3jJex7U3zTUHXhnF\nm65aa22x6oVnZsvcNXu6ip/Hs1G8fbG11k778fa6rU7uN7Z5IXf2y2n8M7bnB6lda2vxlsjPTnPN\ncM8Xq9Tc9UG8eW20mbs/tjbj57Ea59oNn01z59jvxZ8Fvd3c8+NLq2F45txh7jlwFrzRA0Bhgh4A\nChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCypbaTJJlJ5fWO+GZ\nP3l9J7Xr2XG8tOQn9/dTu959/CI19/lEUcd0mLutVsv4/52Hp5Pcrkm8lKK11gbr8e+2WuZKS1pi\nbmNtPbXqcBUvIDm4cy216/Jbb6TmevGfS3vnr7+X2nU7cV+9dPFKalebTFNj6/34gezPcoUxx8/j\nz9PryYKlmzuXU3PDbvy3OdjNPU/vHsYLyW5vb6d2nQVv9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoA\nKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIWVba/r9HJfrTOPt1bd2M41hv2Lly+EZw6m\n8Zax1lq7t5dr8zvpxdv8rt6+ndrVG47CM6fzXDPc6eFhaq4/W4RnhoON1K743dHa/PHT1K7zi3l4\nZnKQu6d2Z4kautba9sWL8ZlO7l1mcBr/brc2N1O7hsn3rc7mWnxmkPuM3aN4w961fvz33FpriQLR\n1lpr3Un8t3mSfA5c6MXvj1fv5HLiLHijB4DCBD0AFCboAaAwQQ8AhQl6AChM0ANAYYIeAAoT9ABQ\nmKAHgMIEPQAUJugBoDBBDwCFCXoAKKxse91qlatAWi0T7WTLeONda629eSl+/E9vnEvtOp7kPuN8\nHG/L27l8JbVr/Vy8r21vmWuvm01nqbl5Ym7SyzUOdju98Mz55L/umV6t6cF+btlp7jxWj56EZ15q\nuefAoBdv89sa587jai/Xbvgi0Ui5thVvAGytteUsfmPNT/ZSuw4muVbERHldW06OU7tuvHk1PPPy\nndxz8Sx4oweAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8A\nhZUttVl2cv/DLFq8SKTNcwUpF/rxwo2v3N5J7Xp+uJuamz5+GJ6ZHeeKIoab8XKP0+R1nq1yc91l\n/FovZom2jdZaZxG/P+bJ85gOMuUv8eKX1lrrzHPnsegN40PdXKnNYh7/bqtkWc/6YpCaW82m4ZlH\n67mimdla/OyXa6lVbbCZO4+Tk/h5DFfL1K4rd66HZ9b7ifv3jHijB4DCBD0AFCboAaAwQQ8AhQl6\nAChM0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKKxse91wYzM111sfhWeme0epXZlW\ns5vb8c/XWmv/bD/XrPXu3uPwzKMHn6R2HYwPwjNHy1z71Gk39z/uYLkKz8xXuba27ir+8zzu5Nra\nTlbxuX7yPWE5yV2z5SR+D3eS7XUtcZ1P+7nrvEw05bXW2nHmM65NUrtaN/7d1ge5+rrlIt5C11pr\nm8v4d3vt2lZq18Vh/OxPnueaA3Of8Hd5oweAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QA\nUJigB4DCBD0AFCboAaAwQQ8AhZUttWndXmqs0xmEZ/obqVXttDsLzwwSZQqttXbnRq4M5+PP4gUT\n08lxatdiGd+1N88VYDzr5G79rV78vuqscteskyio2c/1xbRH03hpSbeTe0/oJQp0srJvMoMWv86P\nl/Hfc2ut7bdcGc5R4lrfSpb8bCcKuHq7h6ld1/rrqbmv3b4ennn1du7hPRrHi8wmybIepTYAwP+X\noAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhdVtr1vm\n/oeZjE/CM9k2rk6iSWo1zTVkndvcTM3tnI83Lu0+fZLadfgoPrffy13nHySbxi4miujOJxoRW2tt\nM9FeN+vmmvIO5vG502TrWra7rteNX+thom2wtdZGqU+Z29Xv5CoHR4lrvZzNU7umi/h5bCTvjwvn\ncp+xzQ7CI0cvcmd/cD7+m+7Mc8+cndTU7/JGDwCFCXoAKEzQA0Bhgh4AChP0AFCYoAeAwgQ9ABQm\n6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUFjZ9rrFMtfitUrMdZINasP+MDyzGucakFruONrVzfhn\n/Ok7f5/a9fzB0/DMvJO7hZ8mO9QO5vE2v9Ei2U6W+IhryXtxNYxf526iTa611jqJVr7WWuv3441h\ni1WynWwR/53N57m2tlXyMw4zx59sr1sm7qtuP/fQWbbcM27vaC8801vlzmOtuxWe6Sz/cHHrjR4A\nChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFFa21KY7iBdg\ntNbaINHD0EkWxnR6ieNf5IozFsdHqbkbW6PwzOVB7jMOTsfhmfPLXEHKaSf3P243MTfv50pLjpfx\nuXHyXmyJEpfePLeskywU6iYKhVarZLlVJ372uW/V2qDTy80lnh8byfv+XGJss5N8DuTGWmvxwcn4\nOLUp8zgddePP0rPijR4AChP0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJig\nB4DCBD0AFCboAaCwuu11/dxX660S//uscu1kLdVel2vl63dz3VrnOvHGsD9762Zq1/5JfNfPPnmW\n2vVsMk/NnS7jbWiTZK/ZMnF/LJP/uy8S36ubrG3sJGveut1sNV9cL9Hy1k9+vI1u7lk16safBVv9\n3OFvdePPuMvJdBklb5BBi/+mh8l7arWI7zpNtHOeFW/0AFCYoAeAwgQ9ABQm6AGgMEEPAIUJegAo\nTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaCwsqU2bbieHIyXFXRWyTaLRPHOfD5LrVomL3WmvOHG\nKLWq/eWXb4Vnrg1yhUIfPD5IzT0+jp//i3mupON02QvPTJK34rwTv86rRPFLa611e/Hv1VprvcRc\nsj+nDRIlP/1kt9VmptyqtbaWOP+1Tu5Dnu8twjMXkwU6m73cfbU+iJ9jP3crttks/hw46cTP8Kx4\noweAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhQl6ACis\ns8o2rwEA/+R5oweAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAw\nQQ8AhQl6AChM0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bhgh4AChP0AFCY\noAeAwgQ9ABQm6AGgMEEPAIUJegAoTNADQGGCHgAKE/QAUJigB4DCBD0AFCboAaAwQQ8AhQl6AChM\n0ANAYYIeAAoT9ABQmKAHgMIEPQAUJugBoDBBDwCFCXoAKEzQA0Bh/xfkBwlHN40TWAAAAABJRU5E\nrkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f8146ea8a20>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 250,
       "width": 253
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "import helper\n",
    "import numpy as np\n",
    "\n",
    "# Explore the dataset\n",
    "batch_id = 1\n",
    "sample_id = 5\n",
    "helper.display_stats(cifar10_dataset_folder_path, batch_id, sample_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Implementing Preprocessing Functions\n",
    "### Normalize\n",
    "In the cell below,I implemented the `normalize` function to take in image data, `x`, and return it as a normalized Numpy array. The values should be in the range of 0 to 1, inclusive.  The return object should be the same shape as `x`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def normalize(x):\n",
    "   \n",
    "    normalized_x = ( x - x.min())  /( x.max() - x.min() )\n",
    "    return normalized_x\n",
    "\n",
    "\n",
    "tests.test_normalize(normalize)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### One-hot encode\n",
    "Here I've implemented the `one_hot_encode` function. The inputs, `x`, are a list of labels.The function returns the list of labels as One-Hot encoded Numpy array.  The possible values for labels are 0 to 9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "# from sklearn.preprocessing import LabelBinarizer\n",
    "# one_hot_encoder = LabelBinarizer()\n",
    "# one_hot_encoder.fit(range(0,10))\n",
    "def one_hot_encode(x):\n",
    "    \n",
    "#     encoded_x = one_hot_encoder.transform(x)\n",
    "    return np.eye(10)[x]\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_one_hot_encode(one_hot_encode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Randomize Data\n",
    "As you saw from exploring the data above, the order of the samples are randomized.  It doesn't hurt to randomize it again, but you don't need to for this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Preprocess all the data and save it\n",
    "Running the code cell below will preprocess all the CIFAR-10 data and save it to file. The code below also uses 10% of the training data for validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Preprocess Training, Validation, and Testing Data\n",
    "helper.preprocess_and_save_data(cifar10_dataset_folder_path, normalize, one_hot_encode)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Check Point\n",
    "This is the first checkpoint.  If anyone ever decides to come back to this notebook or has to restart the notebook, they can start from here.  The preprocessed data has been saved to disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\n",
    "import pickle\n",
    "import problem_unittests as tests\n",
    "import helper\n",
    "\n",
    "# Load the Preprocessed Validation data\n",
    "valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Build the network\n",
    "For the neural network, I've built each layer into a function. \n",
    "\n",
    "### Input\n",
    "The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implemented the following functions\n",
    "* Implemented `neural_net_image_input`\n",
    " * Returns a [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder)\n",
    " * Set the shape using `image_shape` with batch size set to `None`.\n",
    " * Named the TensorFlow placeholder \"x\" using the TensorFlow `name` parameter in the [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder).\n",
    "* Implemented `neural_net_label_input`\n",
    " * Returns a [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder)\n",
    " * Set the shape using `n_classes` with batch size set to `None`.\n",
    " * Named the TensorFlow placeholder \"y\" using the TensorFlow `name` parameter in the [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder).\n",
    "* Implemented `neural_net_keep_prob_input`\n",
    " * Returns a [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder) for dropout keep probability.\n",
    " * Named the TensorFlow placeholder \"keep_prob\" using the TensorFlow `name` parameter in the [TF Placeholder](https://www.tensorflow.org/api_docs/python/tf/placeholder).\n",
    "\n",
    "These names will be used at the end of the project to load your saved model.\n",
    "\n",
    "Note: `None` for shapes in TensorFlow allow for a dynamic size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Image Input Tests Passed.\n",
      "Label Input Tests Passed.\n",
      "Keep Prob Tests Passed.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "def neural_net_image_input(image_shape):\n",
    "    \"\"\"\n",
    "    Return a Tensor for a bach of image input\n",
    "    : image_shape: Shape of the images\n",
    "    : return: Tensor for image input.\n",
    "    \"\"\"\n",
    "    \n",
    "    input_image = tf.placeholder(tf.float32,\n",
    "                                 shape = [None, image_shape[0], image_shape[1], image_shape[2]],\n",
    "                                 name = 'x')\n",
    "    return input_image\n",
    "\n",
    "\n",
    "def neural_net_label_input(n_classes):\n",
    "    \"\"\"\n",
    "    Return a Tensor for a batch of label input\n",
    "    : n_classes: Number of classes\n",
    "    : return: Tensor for label input.\n",
    "    \"\"\"\n",
    "    \n",
    "    input_labels = tf.placeholder(tf.float32, shape= [None, n_classes], name = 'y')\n",
    "    return input_labels\n",
    "\n",
    "\n",
    "def neural_net_keep_prob_input():\n",
    "    \"\"\"\n",
    "    Return a Tensor for keep probability\n",
    "    : return: Tensor for keep probability.\n",
    "    \"\"\"\n",
    "    \n",
    "    dropout_prob = tf.placeholder(tf.float32, name = 'keep_prob')\n",
    "    return dropout_prob\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tf.reset_default_graph()\n",
    "tests.test_nn_image_inputs(neural_net_image_input)\n",
    "tests.test_nn_label_inputs(neural_net_label_input)\n",
    "tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Convolution and Max Pooling Layer\n",
    "Convolution layers have a lot of success with images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def conv2d_maxpool(x_tensor, conv_num_outputs, conv_ksize, conv_strides, pool_ksize, pool_strides):\n",
    "    \"\"\"\n",
    "    Apply convolution then max pooling to x_tensor\n",
    "    :param x_tensor: TensorFlow Tensor\n",
    "    :param conv_num_outputs: Number of outputs for the convolutional layer\n",
    "    :param conv_ksize: kernal size 2-D Tuple for the convolutional layer\n",
    "    :param conv_strides: Stride 2-D Tuple for convolution\n",
    "    :param pool_ksize: kernal size 2-D Tuple for pool\n",
    "    :param pool_strides: Stride 2-D Tuple for pool\n",
    "    : return: A tensor that represents convolution and max pooling of x_tensor\n",
    "    \"\"\"\n",
    "    \n",
    "    n_filters = conv_num_outputs\n",
    "    \n",
    "    color_channels = int(x_tensor.shape[3])\n",
    "    \n",
    "    #initialising weights and biases\n",
    "    weight = tf.Variable(tf.truncated_normal(shape=[*conv_ksize, color_channels, n_filters], stddev=0.1))\n",
    "    bias = tf.Variable(tf.zeros(n_filters))\n",
    "\n",
    "    conv_layer = tf.nn.conv2d(x_tensor, weight, strides=[1, *conv_strides, 1], padding='SAME')\n",
    "    conv_layer = tf.nn.bias_add(conv_layer, bias)\n",
    "    conv_layer = tf.nn.relu(conv_layer)\n",
    "    \n",
    "    conv_layer = tf.nn.max_pool(conv_layer, \n",
    "                               ksize=[1, *pool_ksize, 1],\n",
    "                               strides=[1, *pool_strides, 1],\n",
    "                                padding = 'SAME')\n",
    "    \n",
    "    return conv_layer \n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_con_pool(conv2d_maxpool)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Flatten Layer\n",
    "Implemented the `flatten` function to change the dimension of `x_tensor` from a 4-D tensor to a 2-D tensor.  The output should be the shape (*Batch Size*, *Flattened Image Size*). Shortcut option: we can use classes from the [TensorFlow Layers](https://www.tensorflow.org/api_docs/python/tf/layers) or [TensorFlow Layers (contrib)](https://www.tensorflow.org/api_guides/python/contrib.layers) packages for this layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def flatten(x_tensor):\n",
    "    \n",
    "    input_shape = x_tensor.shape.as_list()\n",
    "#     print(input_shape)\n",
    "    flattened_image_shape = np.prod(input_shape[1:])\n",
    "    flat_x = tf.reshape(x_tensor, shape=[-1, flattened_image_shape])\n",
    "    return flat_x\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_flatten(flatten)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Fully-Connected Layer\n",
    "Implemented the `fully_conn` function to apply a fully connected layer to `x_tensor` with the shape (*Batch Size*, *num_outputs*). Shortcut option: you can use classes from the [TensorFlow Layers](https://www.tensorflow.org/api_docs/python/tf/layers) or [TensorFlow Layers (contrib)](https://www.tensorflow.org/api_guides/python/contrib.layers) packages for this layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def fully_conn(x_tensor, num_outputs):\n",
    "    \n",
    "    n_inputs = x_tensor.shape.as_list()[1]\n",
    "    \n",
    "    weights = tf.Variable(tf.truncated_normal([n_inputs, num_outputs], stddev=0.1))\n",
    "    biases = tf.Variable(tf.zeros(num_outputs))\n",
    "    \n",
    "    layer = tf.nn.bias_add(tf.matmul(x_tensor, weights),biases)\n",
    "    layer = tf.nn.relu(layer)\n",
    "    return layer\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_fully_conn(fully_conn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Output Layer\n",
    "Implemented the `output` function to apply a fully connected layer to `x_tensor` with the shape (*Batch Size*, *num_outputs*). Shortcut option: we can use classes from the [TensorFlow Layers](https://www.tensorflow.org/api_docs/python/tf/layers) or [TensorFlow Layers (contrib)](https://www.tensorflow.org/api_guides/python/contrib.layers) packages for this layer. \n",
    "\n",
    "**Note:** Activation, softmax, or cross entropy should **not** be applied to this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def output(x_tensor, num_outputs):\n",
    "    \n",
    "    n_inputs = x_tensor.shape.as_list()[1]\n",
    "    \n",
    "    weights = tf.Variable(tf.truncated_normal([n_inputs, num_outputs], stddev = 0.1))\n",
    "    biases = tf.Variable(tf.zeros(num_outputs))\n",
    "    \n",
    "    layer = tf.nn.bias_add(tf.matmul(x_tensor, weights),biases)\n",
    "\n",
    "    return layer\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_output(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Create Convolutional Model\n",
    "Implement the function `conv_net` to create a convolutional neural network model. The function takes in a batch of images, `x`, and outputs logits.  Use the layers you created above to create this model:\n",
    "\n",
    "* Apply 1, 2, or 3 Convolution and Max Pool layers\n",
    "* Apply a Flatten Layer\n",
    "* Apply 1, 2, or 3 Fully Connected Layers\n",
    "* Apply an Output Layer\n",
    "* Return the output\n",
    "* Apply [TensorFlow's Dropout](https://www.tensorflow.org/api_docs/python/tf/nn/dropout) to one or more layers in the model using `keep_prob`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Neural Network Built!\n"
     ]
    }
   ],
   "source": [
    "def conv_net(x, keep_prob):\n",
    "   \n",
    "    # x is 32x32x3\n",
    "    x = conv2d_maxpool(x, 64, (3,3), (1,1), (2,2), (2,2))\n",
    "    x = tf.nn.dropout(x, keep_prob)\n",
    "    \n",
    "    #x is 16x16x64\n",
    "    x = conv2d_maxpool(x, 128, (3,3), (1,1), (2,2), (2,2))\n",
    "    x = tf.nn.dropout(x, keep_prob)\n",
    "    \n",
    "    #x is 8x8x128\n",
    "    x = conv2d_maxpool(x, 256, (3,3), (1,1), (2,2), (2,2))\n",
    "    x = tf.nn.dropout(x, keep_prob)\n",
    "    \n",
    "    #x is 4x4x256\n",
    "    \n",
    "    # TODO: Apply a Flatten Layer\n",
    "    # Function Definition from Above:\n",
    "    #   flatten(x_tensor)\n",
    "    x = flatten(x)\n",
    "    \n",
    "    # x is [None, 4096]\n",
    "    \n",
    "#     x = fully_conn(x, 2043)\n",
    "#     x = tf.nn.dropout(x, keep_prob)\n",
    "    \n",
    "#     x = fully_conn(x, 1056)\n",
    "#     x = tf.nn.dropout(x, keep_prob)\n",
    "    \n",
    "    x = fully_conn(x, 512)\n",
    "    x = tf.nn.dropout(x, keep_prob)\n",
    "    \n",
    "    out = output(x, 10)\n",
    "    \n",
    "   \n",
    "    return out\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "##############################\n",
    "## the Neural Network ##\n",
    "##############################\n",
    "\n",
    "# Remove previous weights, bias, inputs, etc..\n",
    "tf.reset_default_graph()\n",
    "\n",
    "# Inputs\n",
    "x = neural_net_image_input((32, 32, 3))\n",
    "y = neural_net_label_input(10)\n",
    "keep_prob = neural_net_keep_prob_input()\n",
    "\n",
    "# Model\n",
    "logits = conv_net(x, keep_prob)\n",
    "\n",
    "# Name logits Tensor, so that it can be loaded from disk after training\n",
    "logits = tf.identity(logits, name='logits')\n",
    "\n",
    "# Loss and Optimizer\n",
    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y))\n",
    "optimizer = tf.train.AdamOptimizer().minimize(cost)\n",
    "\n",
    "# Accuracy\n",
    "correct_pred = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32), name='accuracy')\n",
    "\n",
    "tests.test_conv_net(conv_net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Train the Neural Network\n",
    "### Single Optimization\n",
    "Implemented the function `train_neural_network` to do a single optimization.  The optimization should use `optimizer` to optimize in `session` with a `feed_dict` of the following:\n",
    "* `x` for image input\n",
    "* `y` for labels\n",
    "* `keep_prob` for keep probability for dropout\n",
    "\n",
    "This function will be called for each batch, so `tf.global_variables_initializer()` has already been called.\n",
    "\n",
    "Note: Nothing needs to be returned. This function is only optimizing the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tests Passed\n"
     ]
    }
   ],
   "source": [
    "def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):\n",
    "   \n",
    "   \n",
    "    session.run(optimizer, feed_dict ={\n",
    "        x: feature_batch,\n",
    "        y: label_batch,\n",
    "        keep_prob: keep_probability\n",
    "    })\n",
    "\n",
    "\n",
    "\"\"\"\n",
    "DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE\n",
    "\"\"\"\n",
    "tests.test_train_nn(train_neural_network)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Show Stats\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def print_stats(session, feature_batch, label_batch, cost, accuracy):\n",
    "   \n",
    "   \n",
    "    global valid_features, valid_labels\n",
    "    \n",
    "    loss = session.run(cost, feed_dict={\n",
    "        x: feature_batch,\n",
    "        y: label_batch,\n",
    "        keep_prob: 1.0\n",
    "    })\n",
    "    \n",
    "    validation_accuracy = session.run(accuracy, feed_dict={\n",
    "        x: valid_features,\n",
    "        y: valid_labels,\n",
    "        keep_prob: 1.0\n",
    "    })\n",
    "    \n",
    "    \n",
    "    print('Loss: {:.4f} Validation accuracy: {:.4f}'.format(loss, validation_accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Hyperparameters\n",
    "Tuning the following parameters:\n",
    "* Set `epochs` to the number of iterations until the network stops learning or start overfitting\n",
    "* Set `batch_size` to the highest number that your machine has memory for.  Most people set them to common sizes of memory:\n",
    " * 64\n",
    " * 128\n",
    " * 256\n",
    " * ...\n",
    "* Set `keep_probability` to the probability of keeping a node using dropout"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "epochs = 100\n",
    "batch_size = 128\n",
    "keep_probability = 0.5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Train the model on a Single CIFAR-10 Batch\n",
    "Instead of training the neural network on all the CIFAR-10 batches of data, I've used a single batch. This saves time while iterating on the model to get a better accuracy.  Once the final validation accuracy is 50% or greater, run the model on all the data in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking the Training on a Single Batch...\n",
      "Epoch  1, CIFAR-10 Batch 1:  Loss: 2.2998 Validation accuracy: 0.1380\n",
      "Epoch  2, CIFAR-10 Batch 1:  Loss: 2.2953 Validation accuracy: 0.1680\n",
      "Epoch  3, CIFAR-10 Batch 1:  Loss: 2.2741 Validation accuracy: 0.1860\n",
      "Epoch  4, CIFAR-10 Batch 1:  Loss: 2.2353 Validation accuracy: 0.1712\n",
      "Epoch  5, CIFAR-10 Batch 1:  Loss: 2.1880 Validation accuracy: 0.2246\n",
      "Epoch  6, CIFAR-10 Batch 1:  Loss: 2.1639 Validation accuracy: 0.2330\n",
      "Epoch  7, CIFAR-10 Batch 1:  Loss: 2.1561 Validation accuracy: 0.2550\n",
      "Epoch  8, CIFAR-10 Batch 1:  Loss: 2.0121 Validation accuracy: 0.3124\n",
      "Epoch  9, CIFAR-10 Batch 1:  Loss: 1.9450 Validation accuracy: 0.3354\n",
      "Epoch 10, CIFAR-10 Batch 1:  Loss: 1.9584 Validation accuracy: 0.3436\n",
      "Epoch 11, CIFAR-10 Batch 1:  Loss: 1.8627 Validation accuracy: 0.3800\n",
      "Epoch 12, CIFAR-10 Batch 1:  Loss: 1.8929 Validation accuracy: 0.3520\n",
      "Epoch 13, CIFAR-10 Batch 1:  Loss: 1.8079 Validation accuracy: 0.3846\n",
      "Epoch 14, CIFAR-10 Batch 1:  Loss: 1.7634 Validation accuracy: 0.3990\n",
      "Epoch 15, CIFAR-10 Batch 1:  Loss: 1.7564 Validation accuracy: 0.3914\n",
      "Epoch 16, CIFAR-10 Batch 1:  Loss: 1.6780 Validation accuracy: 0.4388\n",
      "Epoch 17, CIFAR-10 Batch 1:  Loss: 1.7323 Validation accuracy: 0.4136\n",
      "Epoch 18, CIFAR-10 Batch 1:  Loss: 1.6971 Validation accuracy: 0.4210\n",
      "Epoch 19, CIFAR-10 Batch 1:  Loss: 1.6715 Validation accuracy: 0.4320\n",
      "Epoch 20, CIFAR-10 Batch 1:  Loss: 1.6245 Validation accuracy: 0.4476\n",
      "Epoch 21, CIFAR-10 Batch 1:  Loss: 1.6493 Validation accuracy: 0.4438\n",
      "Epoch 22, CIFAR-10 Batch 1:  Loss: 1.5361 Validation accuracy: 0.4768\n",
      "Epoch 23, CIFAR-10 Batch 1:  Loss: 1.5875 Validation accuracy: 0.4554\n",
      "Epoch 24, CIFAR-10 Batch 1:  Loss: 1.5018 Validation accuracy: 0.4810\n",
      "Epoch 25, CIFAR-10 Batch 1:  Loss: 1.4704 Validation accuracy: 0.4800\n",
      "Epoch 26, CIFAR-10 Batch 1:  Loss: 1.5320 Validation accuracy: 0.4562\n",
      "Epoch 27, CIFAR-10 Batch 1:  Loss: 1.4457 Validation accuracy: 0.4906\n",
      "Epoch 28, CIFAR-10 Batch 1:  Loss: 1.4289 Validation accuracy: 0.4906\n",
      "Epoch 29, CIFAR-10 Batch 1:  Loss: 1.3272 Validation accuracy: 0.5050\n",
      "Epoch 30, CIFAR-10 Batch 1:  Loss: 1.4318 Validation accuracy: 0.4782\n",
      "Epoch 31, CIFAR-10 Batch 1:  Loss: 1.3554 Validation accuracy: 0.4900\n",
      "Epoch 32, CIFAR-10 Batch 1:  Loss: 1.3298 Validation accuracy: 0.4970\n",
      "Epoch 33, CIFAR-10 Batch 1:  Loss: 1.3613 Validation accuracy: 0.4746\n",
      "Epoch 34, CIFAR-10 Batch 1:  Loss: 1.2911 Validation accuracy: 0.5202\n",
      "Epoch 35, CIFAR-10 Batch 1:  Loss: 1.2052 Validation accuracy: 0.5234\n",
      "Epoch 36, CIFAR-10 Batch 1:  Loss: 1.1994 Validation accuracy: 0.5296\n",
      "Epoch 37, CIFAR-10 Batch 1:  Loss: 1.1643 Validation accuracy: 0.5344\n",
      "Epoch 38, CIFAR-10 Batch 1:  Loss: 1.1847 Validation accuracy: 0.5256\n",
      "Epoch 39, CIFAR-10 Batch 1:  Loss: 1.1347 Validation accuracy: 0.5302\n",
      "Epoch 40, CIFAR-10 Batch 1:  Loss: 1.1206 Validation accuracy: 0.5398\n",
      "Epoch 41, CIFAR-10 Batch 1:  Loss: 1.0951 Validation accuracy: 0.5558\n",
      "Epoch 42, CIFAR-10 Batch 1:  Loss: 1.0540 Validation accuracy: 0.5532\n",
      "Epoch 43, CIFAR-10 Batch 1:  Loss: 1.1149 Validation accuracy: 0.5260\n",
      "Epoch 44, CIFAR-10 Batch 1:  Loss: 1.0417 Validation accuracy: 0.5580\n",
      "Epoch 45, CIFAR-10 Batch 1:  Loss: 1.0354 Validation accuracy: 0.5544\n",
      "Epoch 46, CIFAR-10 Batch 1:  Loss: 0.9653 Validation accuracy: 0.5640\n",
      "Epoch 47, CIFAR-10 Batch 1:  Loss: 1.0160 Validation accuracy: 0.5540\n",
      "Epoch 48, CIFAR-10 Batch 1:  Loss: 1.0947 Validation accuracy: 0.5430\n",
      "Epoch 49, CIFAR-10 Batch 1:  Loss: 0.9946 Validation accuracy: 0.5648\n",
      "Epoch 50, CIFAR-10 Batch 1:  Loss: 0.8627 Validation accuracy: 0.5800\n",
      "Epoch 51, CIFAR-10 Batch 1:  Loss: 0.8532 Validation accuracy: 0.5874\n",
      "Epoch 52, CIFAR-10 Batch 1:  Loss: 0.7821 Validation accuracy: 0.5946\n",
      "Epoch 53, CIFAR-10 Batch 1:  Loss: 0.8329 Validation accuracy: 0.5884\n",
      "Epoch 54, CIFAR-10 Batch 1:  Loss: 0.8477 Validation accuracy: 0.5918\n",
      "Epoch 55, CIFAR-10 Batch 1:  Loss: 0.7652 Validation accuracy: 0.5982\n",
      "Epoch 56, CIFAR-10 Batch 1:  Loss: 0.7395 Validation accuracy: 0.6060\n",
      "Epoch 57, CIFAR-10 Batch 1:  Loss: 0.7108 Validation accuracy: 0.6102\n",
      "Epoch 58, CIFAR-10 Batch 1:  Loss: 0.7373 Validation accuracy: 0.6066\n",
      "Epoch 59, CIFAR-10 Batch 1:  Loss: 0.6756 Validation accuracy: 0.6262\n",
      "Epoch 60, CIFAR-10 Batch 1:  Loss: 0.7055 Validation accuracy: 0.6058\n",
      "Epoch 61, CIFAR-10 Batch 1:  Loss: 0.6281 Validation accuracy: 0.6216\n",
      "Epoch 62, CIFAR-10 Batch 1:  Loss: 0.6346 Validation accuracy: 0.6104\n",
      "Epoch 63, CIFAR-10 Batch 1:  Loss: 0.6445 Validation accuracy: 0.6122\n",
      "Epoch 64, CIFAR-10 Batch 1:  Loss: 0.6108 Validation accuracy: 0.6240\n",
      "Epoch 65, CIFAR-10 Batch 1:  Loss: 0.5828 Validation accuracy: 0.6280\n",
      "Epoch 66, CIFAR-10 Batch 1:  Loss: 0.5750 Validation accuracy: 0.6278\n",
      "Epoch 67, CIFAR-10 Batch 1:  Loss: 0.5300 Validation accuracy: 0.6348\n",
      "Epoch 68, CIFAR-10 Batch 1:  Loss: 0.5007 Validation accuracy: 0.6448\n",
      "Epoch 69, CIFAR-10 Batch 1:  Loss: 0.4846 Validation accuracy: 0.6554\n",
      "Epoch 70, CIFAR-10 Batch 1:  Loss: 0.5677 Validation accuracy: 0.6158\n",
      "Epoch 71, CIFAR-10 Batch 1:  Loss: 0.4312 Validation accuracy: 0.6408\n",
      "Epoch 72, CIFAR-10 Batch 1:  Loss: 0.4528 Validation accuracy: 0.6372\n",
      "Epoch 73, CIFAR-10 Batch 1:  Loss: 0.4201 Validation accuracy: 0.6460\n",
      "Epoch 74, CIFAR-10 Batch 1:  Loss: 0.3987 Validation accuracy: 0.6536\n",
      "Epoch 75, CIFAR-10 Batch 1:  Loss: 0.3912 Validation accuracy: 0.6550\n",
      "Epoch 76, CIFAR-10 Batch 1:  Loss: 0.3958 Validation accuracy: 0.6486\n",
      "Epoch 77, CIFAR-10 Batch 1:  Loss: 0.3606 Validation accuracy: 0.6622\n",
      "Epoch 78, CIFAR-10 Batch 1:  Loss: 0.3319 Validation accuracy: 0.6580\n",
      "Epoch 79, CIFAR-10 Batch 1:  Loss: 0.3025 Validation accuracy: 0.6640\n",
      "Epoch 80, CIFAR-10 Batch 1:  Loss: 0.3155 Validation accuracy: 0.6550\n",
      "Epoch 81, CIFAR-10 Batch 1:  Loss: 0.2928 Validation accuracy: 0.6552\n",
      "Epoch 82, CIFAR-10 Batch 1:  Loss: 0.3330 Validation accuracy: 0.6616\n",
      "Epoch 83, CIFAR-10 Batch 1:  Loss: 0.3889 Validation accuracy: 0.6556\n",
      "Epoch 84, CIFAR-10 Batch 1:  Loss: 0.2710 Validation accuracy: 0.6654\n",
      "Epoch 85, CIFAR-10 Batch 1:  Loss: 0.2610 Validation accuracy: 0.6606\n",
      "Epoch 86, CIFAR-10 Batch 1:  Loss: 0.2719 Validation accuracy: 0.6708\n",
      "Epoch 87, CIFAR-10 Batch 1:  Loss: 0.2282 Validation accuracy: 0.6782\n",
      "Epoch 88, CIFAR-10 Batch 1:  Loss: 0.2303 Validation accuracy: 0.6726\n",
      "Epoch 89, CIFAR-10 Batch 1:  Loss: 0.2206 Validation accuracy: 0.6700\n",
      "Epoch 90, CIFAR-10 Batch 1:  Loss: 0.2224 Validation accuracy: 0.6690\n",
      "Epoch 91, CIFAR-10 Batch 1:  Loss: 0.1952 Validation accuracy: 0.6688\n",
      "Epoch 92, CIFAR-10 Batch 1:  Loss: 0.1771 Validation accuracy: 0.6838\n",
      "Epoch 93, CIFAR-10 Batch 1:  Loss: 0.1586 Validation accuracy: 0.6776\n",
      "Epoch 94, CIFAR-10 Batch 1:  Loss: 0.1403 Validation accuracy: 0.6862\n",
      "Epoch 95, CIFAR-10 Batch 1:  Loss: 0.2622 Validation accuracy: 0.6520\n",
      "Epoch 96, CIFAR-10 Batch 1:  Loss: 0.1263 Validation accuracy: 0.6846\n",
      "Epoch 97, CIFAR-10 Batch 1:  Loss: 0.1774 Validation accuracy: 0.6708\n",
      "Epoch 98, CIFAR-10 Batch 1:  Loss: 0.1569 Validation accuracy: 0.6634\n",
      "Epoch 99, CIFAR-10 Batch 1:  Loss: 0.1699 Validation accuracy: 0.6538\n",
      "Epoch 100, CIFAR-10 Batch 1:  Loss: 0.1317 Validation accuracy: 0.6778\n"
     ]
    }
   ],
   "source": [
    "print('Checking the Training on a Single Batch...')\n",
    "with tf.Session() as sess:\n",
    "    # Initializing the variables\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # Training cycle\n",
    "    for epoch in range(epochs):\n",
    "        batch_i = 1\n",
    "        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):\n",
    "            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)\n",
    "        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')\n",
    "        print_stats(sess, batch_features, batch_labels, cost, accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Fully Train the Model\n",
    "Now that we have a good accuracy with a single CIFAR-10 batch, let's try it with all five batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n",
      "Epoch  1, CIFAR-10 Batch 1:  Loss: 2.3009 Validation accuracy: 0.1312\n",
      "Epoch  1, CIFAR-10 Batch 2:  Loss: 2.2966 Validation accuracy: 0.1166\n",
      "Epoch  1, CIFAR-10 Batch 3:  Loss: 2.2551 Validation accuracy: 0.1618\n",
      "Epoch  1, CIFAR-10 Batch 4:  Loss: 2.1632 Validation accuracy: 0.2190\n",
      "Epoch  1, CIFAR-10 Batch 5:  Loss: 2.1724 Validation accuracy: 0.2328\n",
      "Epoch  2, CIFAR-10 Batch 1:  Loss: 2.1156 Validation accuracy: 0.2742\n",
      "Epoch  2, CIFAR-10 Batch 2:  Loss: 1.9941 Validation accuracy: 0.3098\n",
      "Epoch  2, CIFAR-10 Batch 3:  Loss: 1.9873 Validation accuracy: 0.2364\n",
      "Epoch  2, CIFAR-10 Batch 4:  Loss: 1.8721 Validation accuracy: 0.2832\n",
      "Epoch  2, CIFAR-10 Batch 5:  Loss: 1.9756 Validation accuracy: 0.3012\n",
      "Epoch  3, CIFAR-10 Batch 1:  Loss: 1.9577 Validation accuracy: 0.3544\n",
      "Epoch  3, CIFAR-10 Batch 2:  Loss: 1.7492 Validation accuracy: 0.3526\n",
      "Epoch  3, CIFAR-10 Batch 3:  Loss: 1.7122 Validation accuracy: 0.3612\n",
      "Epoch  3, CIFAR-10 Batch 4:  Loss: 1.7943 Validation accuracy: 0.3604\n",
      "Epoch  3, CIFAR-10 Batch 5:  Loss: 1.6870 Validation accuracy: 0.3872\n",
      "Epoch  4, CIFAR-10 Batch 1:  Loss: 1.8558 Validation accuracy: 0.4112\n",
      "Epoch  4, CIFAR-10 Batch 2:  Loss: 1.6368 Validation accuracy: 0.3680\n",
      "Epoch  4, CIFAR-10 Batch 3:  Loss: 1.5669 Validation accuracy: 0.3974\n",
      "Epoch  4, CIFAR-10 Batch 4:  Loss: 1.6641 Validation accuracy: 0.4280\n",
      "Epoch  4, CIFAR-10 Batch 5:  Loss: 1.6441 Validation accuracy: 0.4074\n",
      "Epoch  5, CIFAR-10 Batch 1:  Loss: 1.7690 Validation accuracy: 0.4368\n",
      "Epoch  5, CIFAR-10 Batch 2:  Loss: 1.5276 Validation accuracy: 0.4052\n",
      "Epoch  5, CIFAR-10 Batch 3:  Loss: 1.5285 Validation accuracy: 0.4342\n",
      "Epoch  5, CIFAR-10 Batch 4:  Loss: 1.6003 Validation accuracy: 0.4426\n",
      "Epoch  5, CIFAR-10 Batch 5:  Loss: 1.6132 Validation accuracy: 0.4214\n",
      "Epoch  6, CIFAR-10 Batch 1:  Loss: 1.6267 Validation accuracy: 0.4620\n",
      "Epoch  6, CIFAR-10 Batch 2:  Loss: 1.6436 Validation accuracy: 0.3992\n",
      "Epoch  6, CIFAR-10 Batch 3:  Loss: 1.4933 Validation accuracy: 0.4574\n",
      "Epoch  6, CIFAR-10 Batch 4:  Loss: 1.5665 Validation accuracy: 0.4826\n",
      "Epoch  6, CIFAR-10 Batch 5:  Loss: 1.5181 Validation accuracy: 0.4470\n",
      "Epoch  7, CIFAR-10 Batch 1:  Loss: 1.5132 Validation accuracy: 0.4866\n",
      "Epoch  7, CIFAR-10 Batch 2:  Loss: 1.4494 Validation accuracy: 0.4850\n",
      "Epoch  7, CIFAR-10 Batch 3:  Loss: 1.4715 Validation accuracy: 0.4466\n",
      "Epoch  7, CIFAR-10 Batch 4:  Loss: 1.3983 Validation accuracy: 0.5276\n",
      "Epoch  7, CIFAR-10 Batch 5:  Loss: 1.5944 Validation accuracy: 0.4626\n",
      "Epoch  8, CIFAR-10 Batch 1:  Loss: 1.4462 Validation accuracy: 0.5076\n",
      "Epoch  8, CIFAR-10 Batch 2:  Loss: 1.4249 Validation accuracy: 0.4720\n",
      "Epoch  8, CIFAR-10 Batch 3:  Loss: 1.3801 Validation accuracy: 0.4748\n",
      "Epoch  8, CIFAR-10 Batch 4:  Loss: 1.3706 Validation accuracy: 0.5332\n",
      "Epoch  8, CIFAR-10 Batch 5:  Loss: 1.3845 Validation accuracy: 0.5240\n",
      "Epoch  9, CIFAR-10 Batch 1:  Loss: 1.3082 Validation accuracy: 0.5510\n",
      "Epoch  9, CIFAR-10 Batch 2:  Loss: 1.3806 Validation accuracy: 0.5020\n",
      "Epoch  9, CIFAR-10 Batch 3:  Loss: 1.4330 Validation accuracy: 0.4828\n",
      "Epoch  9, CIFAR-10 Batch 4:  Loss: 1.3405 Validation accuracy: 0.5344\n",
      "Epoch  9, CIFAR-10 Batch 5:  Loss: 1.3973 Validation accuracy: 0.5166\n",
      "Epoch 10, CIFAR-10 Batch 1:  Loss: 1.2954 Validation accuracy: 0.5474\n",
      "Epoch 10, CIFAR-10 Batch 2:  Loss: 1.3396 Validation accuracy: 0.5400\n",
      "Epoch 10, CIFAR-10 Batch 3:  Loss: 1.4473 Validation accuracy: 0.4650\n",
      "Epoch 10, CIFAR-10 Batch 4:  Loss: 1.2279 Validation accuracy: 0.5804\n",
      "Epoch 10, CIFAR-10 Batch 5:  Loss: 1.3139 Validation accuracy: 0.5422\n",
      "Epoch 11, CIFAR-10 Batch 1:  Loss: 1.0410 Validation accuracy: 0.6090\n",
      "Epoch 11, CIFAR-10 Batch 2:  Loss: 1.2615 Validation accuracy: 0.5634\n",
      "Epoch 11, CIFAR-10 Batch 3:  Loss: 1.1250 Validation accuracy: 0.5564\n",
      "Epoch 11, CIFAR-10 Batch 4:  Loss: 1.1863 Validation accuracy: 0.5890\n",
      "Epoch 11, CIFAR-10 Batch 5:  Loss: 1.2297 Validation accuracy: 0.5770\n",
      "Epoch 12, CIFAR-10 Batch 1:  Loss: 0.9984 Validation accuracy: 0.6274\n",
      "Epoch 12, CIFAR-10 Batch 2:  Loss: 1.2932 Validation accuracy: 0.5596\n",
      "Epoch 12, CIFAR-10 Batch 3:  Loss: 1.1562 Validation accuracy: 0.5540\n",
      "Epoch 12, CIFAR-10 Batch 4:  Loss: 1.2773 Validation accuracy: 0.5650\n",
      "Epoch 12, CIFAR-10 Batch 5:  Loss: 1.4252 Validation accuracy: 0.5294\n",
      "Epoch 13, CIFAR-10 Batch 1:  Loss: 0.9651 Validation accuracy: 0.6288\n",
      "Epoch 13, CIFAR-10 Batch 2:  Loss: 1.1549 Validation accuracy: 0.6118\n",
      "Epoch 13, CIFAR-10 Batch 3:  Loss: 1.0859 Validation accuracy: 0.5894\n",
      "Epoch 13, CIFAR-10 Batch 4:  Loss: 0.9761 Validation accuracy: 0.6484\n",
      "Epoch 13, CIFAR-10 Batch 5:  Loss: 1.3089 Validation accuracy: 0.5492\n",
      "Epoch 14, CIFAR-10 Batch 1:  Loss: 1.1357 Validation accuracy: 0.6072\n",
      "Epoch 14, CIFAR-10 Batch 2:  Loss: 1.1400 Validation accuracy: 0.6118\n",
      "Epoch 14, CIFAR-10 Batch 3:  Loss: 1.0738 Validation accuracy: 0.5866\n",
      "Epoch 14, CIFAR-10 Batch 4:  Loss: 1.0489 Validation accuracy: 0.6170\n",
      "Epoch 14, CIFAR-10 Batch 5:  Loss: 1.2634 Validation accuracy: 0.5832\n",
      "Epoch 15, CIFAR-10 Batch 1:  Loss: 0.8820 Validation accuracy: 0.6430\n",
      "Epoch 15, CIFAR-10 Batch 2:  Loss: 1.1491 Validation accuracy: 0.5940\n",
      "Epoch 15, CIFAR-10 Batch 3:  Loss: 0.9031 Validation accuracy: 0.6048\n",
      "Epoch 15, CIFAR-10 Batch 4:  Loss: 0.9095 Validation accuracy: 0.6564\n",
      "Epoch 15, CIFAR-10 Batch 5:  Loss: 1.0876 Validation accuracy: 0.6156\n",
      "Epoch 16, CIFAR-10 Batch 1:  Loss: 0.8890 Validation accuracy: 0.6584\n",
      "Epoch 16, CIFAR-10 Batch 2:  Loss: 0.9459 Validation accuracy: 0.6302\n",
      "Epoch 16, CIFAR-10 Batch 3:  Loss: 1.0481 Validation accuracy: 0.5924\n",
      "Epoch 16, CIFAR-10 Batch 4:  Loss: 0.8751 Validation accuracy: 0.6594\n",
      "Epoch 16, CIFAR-10 Batch 5:  Loss: 1.0667 Validation accuracy: 0.6194\n",
      "Epoch 17, CIFAR-10 Batch 1:  Loss: 0.8666 Validation accuracy: 0.6582\n",
      "Epoch 17, CIFAR-10 Batch 2:  Loss: 1.0003 Validation accuracy: 0.6436\n",
      "Epoch 17, CIFAR-10 Batch 3:  Loss: 0.8939 Validation accuracy: 0.6282\n",
      "Epoch 17, CIFAR-10 Batch 4:  Loss: 0.8431 Validation accuracy: 0.6700\n",
      "Epoch 17, CIFAR-10 Batch 5:  Loss: 0.9496 Validation accuracy: 0.6396\n",
      "Epoch 18, CIFAR-10 Batch 1:  Loss: 0.8313 Validation accuracy: 0.6740\n",
      "Epoch 18, CIFAR-10 Batch 2:  Loss: 0.9236 Validation accuracy: 0.6158\n",
      "Epoch 18, CIFAR-10 Batch 3:  Loss: 0.8290 Validation accuracy: 0.6378\n",
      "Epoch 18, CIFAR-10 Batch 4:  Loss: 0.9443 Validation accuracy: 0.6390\n",
      "Epoch 18, CIFAR-10 Batch 5:  Loss: 1.0639 Validation accuracy: 0.6206\n",
      "Epoch 19, CIFAR-10 Batch 1:  Loss: 0.7940 Validation accuracy: 0.6746\n",
      "Epoch 19, CIFAR-10 Batch 2:  Loss: 0.8460 Validation accuracy: 0.6474\n",
      "Epoch 19, CIFAR-10 Batch 3:  Loss: 0.7217 Validation accuracy: 0.6648\n",
      "Epoch 19, CIFAR-10 Batch 4:  Loss: 0.8221 Validation accuracy: 0.6766\n",
      "Epoch 19, CIFAR-10 Batch 5:  Loss: 1.0973 Validation accuracy: 0.6338\n",
      "Epoch 20, CIFAR-10 Batch 1:  Loss: 0.7386 Validation accuracy: 0.7014\n",
      "Epoch 20, CIFAR-10 Batch 2:  Loss: 0.7792 Validation accuracy: 0.6770\n",
      "Epoch 20, CIFAR-10 Batch 3:  Loss: 0.7171 Validation accuracy: 0.6416\n",
      "Epoch 20, CIFAR-10 Batch 4:  Loss: 0.8555 Validation accuracy: 0.6640\n",
      "Epoch 20, CIFAR-10 Batch 5:  Loss: 0.9096 Validation accuracy: 0.6666\n",
      "Epoch 21, CIFAR-10 Batch 1:  Loss: 0.7290 Validation accuracy: 0.7046\n",
      "Epoch 21, CIFAR-10 Batch 2:  Loss: 0.6811 Validation accuracy: 0.6846\n",
      "Epoch 21, CIFAR-10 Batch 3:  Loss: 0.6495 Validation accuracy: 0.6776\n",
      "Epoch 21, CIFAR-10 Batch 4:  Loss: 0.7156 Validation accuracy: 0.6958\n",
      "Epoch 21, CIFAR-10 Batch 5:  Loss: 0.8650 Validation accuracy: 0.6772\n",
      "Epoch 22, CIFAR-10 Batch 1:  Loss: 0.7139 Validation accuracy: 0.7054\n",
      "Epoch 22, CIFAR-10 Batch 2:  Loss: 0.6628 Validation accuracy: 0.7008\n",
      "Epoch 22, CIFAR-10 Batch 3:  Loss: 0.5412 Validation accuracy: 0.6906\n",
      "Epoch 22, CIFAR-10 Batch 4:  Loss: 0.6848 Validation accuracy: 0.6844\n",
      "Epoch 22, CIFAR-10 Batch 5:  Loss: 0.7964 Validation accuracy: 0.6800\n",
      "Epoch 23, CIFAR-10 Batch 1:  Loss: 0.6938 Validation accuracy: 0.7056\n",
      "Epoch 23, CIFAR-10 Batch 2:  Loss: 0.8036 Validation accuracy: 0.6644\n",
      "Epoch 23, CIFAR-10 Batch 3:  Loss: 0.5539 Validation accuracy: 0.6788\n",
      "Epoch 23, CIFAR-10 Batch 4:  Loss: 0.7457 Validation accuracy: 0.6798\n",
      "Epoch 23, CIFAR-10 Batch 5:  Loss: 0.7273 Validation accuracy: 0.7090\n",
      "Epoch 24, CIFAR-10 Batch 1:  Loss: 0.6819 Validation accuracy: 0.7058\n",
      "Epoch 24, CIFAR-10 Batch 2:  Loss: 0.6345 Validation accuracy: 0.7070\n",
      "Epoch 24, CIFAR-10 Batch 3:  Loss: 0.6260 Validation accuracy: 0.6778\n",
      "Epoch 24, CIFAR-10 Batch 4:  Loss: 0.5730 Validation accuracy: 0.7082\n",
      "Epoch 24, CIFAR-10 Batch 5:  Loss: 0.9087 Validation accuracy: 0.6540\n",
      "Epoch 25, CIFAR-10 Batch 1:  Loss: 0.7109 Validation accuracy: 0.7134\n",
      "Epoch 25, CIFAR-10 Batch 2:  Loss: 0.6650 Validation accuracy: 0.6830\n",
      "Epoch 25, CIFAR-10 Batch 3:  Loss: 0.4939 Validation accuracy: 0.7080\n",
      "Epoch 25, CIFAR-10 Batch 4:  Loss: 0.5442 Validation accuracy: 0.7208\n",
      "Epoch 25, CIFAR-10 Batch 5:  Loss: 0.8176 Validation accuracy: 0.6778\n",
      "Epoch 26, CIFAR-10 Batch 1:  Loss: 0.5714 Validation accuracy: 0.7366\n",
      "Epoch 26, CIFAR-10 Batch 2:  Loss: 0.6957 Validation accuracy: 0.6988\n",
      "Epoch 26, CIFAR-10 Batch 3:  Loss: 0.5831 Validation accuracy: 0.7008\n",
      "Epoch 26, CIFAR-10 Batch 4:  Loss: 0.5245 Validation accuracy: 0.7228\n",
      "Epoch 26, CIFAR-10 Batch 5:  Loss: 0.6835 Validation accuracy: 0.7062\n",
      "Epoch 27, CIFAR-10 Batch 1:  Loss: 0.5712 Validation accuracy: 0.7412\n",
      "Epoch 27, CIFAR-10 Batch 2:  Loss: 0.5726 Validation accuracy: 0.7198\n",
      "Epoch 27, CIFAR-10 Batch 3:  Loss: 0.3986 Validation accuracy: 0.7134\n",
      "Epoch 27, CIFAR-10 Batch 4:  Loss: 0.6398 Validation accuracy: 0.6998\n",
      "Epoch 27, CIFAR-10 Batch 5:  Loss: 0.7021 Validation accuracy: 0.7086\n",
      "Epoch 28, CIFAR-10 Batch 1:  Loss: 0.5598 Validation accuracy: 0.7478\n",
      "Epoch 28, CIFAR-10 Batch 2:  Loss: 0.5557 Validation accuracy: 0.7246\n",
      "Epoch 28, CIFAR-10 Batch 3:  Loss: 0.3783 Validation accuracy: 0.7206\n",
      "Epoch 28, CIFAR-10 Batch 4:  Loss: 0.5630 Validation accuracy: 0.7090\n",
      "Epoch 28, CIFAR-10 Batch 5:  Loss: 0.6738 Validation accuracy: 0.7098\n",
      "Epoch 29, CIFAR-10 Batch 1:  Loss: 0.5959 Validation accuracy: 0.7374\n",
      "Epoch 29, CIFAR-10 Batch 2:  Loss: 0.6098 Validation accuracy: 0.7222\n",
      "Epoch 29, CIFAR-10 Batch 3:  Loss: 0.3549 Validation accuracy: 0.7336\n",
      "Epoch 29, CIFAR-10 Batch 4:  Loss: 0.6836 Validation accuracy: 0.6978\n",
      "Epoch 29, CIFAR-10 Batch 5:  Loss: 0.5891 Validation accuracy: 0.7388\n",
      "Epoch 30, CIFAR-10 Batch 1:  Loss: 0.5397 Validation accuracy: 0.7470\n",
      "Epoch 30, CIFAR-10 Batch 2:  Loss: 0.5314 Validation accuracy: 0.7212\n",
      "Epoch 30, CIFAR-10 Batch 3:  Loss: 0.3303 Validation accuracy: 0.7290\n",
      "Epoch 30, CIFAR-10 Batch 4:  Loss: 0.6081 Validation accuracy: 0.6868\n",
      "Epoch 30, CIFAR-10 Batch 5:  Loss: 0.6326 Validation accuracy: 0.7192\n",
      "Epoch 31, CIFAR-10 Batch 1:  Loss: 0.5334 Validation accuracy: 0.7402\n",
      "Epoch 31, CIFAR-10 Batch 2:  Loss: 0.5599 Validation accuracy: 0.7134\n",
      "Epoch 31, CIFAR-10 Batch 3:  Loss: 0.3502 Validation accuracy: 0.7216\n",
      "Epoch 31, CIFAR-10 Batch 4:  Loss: 0.6156 Validation accuracy: 0.6854\n",
      "Epoch 31, CIFAR-10 Batch 5:  Loss: 0.5165 Validation accuracy: 0.7406\n",
      "Epoch 32, CIFAR-10 Batch 1:  Loss: 0.4773 Validation accuracy: 0.7534\n",
      "Epoch 32, CIFAR-10 Batch 2:  Loss: 0.5264 Validation accuracy: 0.7334\n",
      "Epoch 32, CIFAR-10 Batch 3:  Loss: 0.3194 Validation accuracy: 0.7476\n",
      "Epoch 32, CIFAR-10 Batch 4:  Loss: 0.5463 Validation accuracy: 0.7078\n",
      "Epoch 32, CIFAR-10 Batch 5:  Loss: 0.5274 Validation accuracy: 0.7402\n",
      "Epoch 33, CIFAR-10 Batch 1:  Loss: 0.4680 Validation accuracy: 0.7462\n",
      "Epoch 33, CIFAR-10 Batch 2:  Loss: 0.5335 Validation accuracy: 0.7376\n",
      "Epoch 33, CIFAR-10 Batch 3:  Loss: 0.3650 Validation accuracy: 0.7416\n",
      "Epoch 33, CIFAR-10 Batch 4:  Loss: 0.5602 Validation accuracy: 0.7218\n",
      "Epoch 33, CIFAR-10 Batch 5:  Loss: 0.6958 Validation accuracy: 0.6994\n",
      "Epoch 34, CIFAR-10 Batch 1:  Loss: 0.4722 Validation accuracy: 0.7548\n",
      "Epoch 34, CIFAR-10 Batch 2:  Loss: 0.5052 Validation accuracy: 0.7276\n",
      "Epoch 34, CIFAR-10 Batch 3:  Loss: 0.3127 Validation accuracy: 0.7412\n",
      "Epoch 34, CIFAR-10 Batch 4:  Loss: 0.3930 Validation accuracy: 0.7536\n",
      "Epoch 34, CIFAR-10 Batch 5:  Loss: 0.5411 Validation accuracy: 0.7334\n",
      "Epoch 35, CIFAR-10 Batch 1:  Loss: 0.5564 Validation accuracy: 0.7432\n",
      "Epoch 35, CIFAR-10 Batch 2:  Loss: 0.4345 Validation accuracy: 0.7490\n",
      "Epoch 35, CIFAR-10 Batch 3:  Loss: 0.3238 Validation accuracy: 0.7366\n",
      "Epoch 35, CIFAR-10 Batch 4:  Loss: 0.3843 Validation accuracy: 0.7594\n",
      "Epoch 35, CIFAR-10 Batch 5:  Loss: 0.4476 Validation accuracy: 0.7540\n",
      "Epoch 36, CIFAR-10 Batch 1:  Loss: 0.4368 Validation accuracy: 0.7576\n",
      "Epoch 36, CIFAR-10 Batch 2:  Loss: 0.5213 Validation accuracy: 0.7410\n",
      "Epoch 36, CIFAR-10 Batch 3:  Loss: 0.2934 Validation accuracy: 0.7498\n",
      "Epoch 36, CIFAR-10 Batch 4:  Loss: 0.3315 Validation accuracy: 0.7594\n",
      "Epoch 36, CIFAR-10 Batch 5:  Loss: 0.5248 Validation accuracy: 0.7290\n",
      "Epoch 37, CIFAR-10 Batch 1:  Loss: 0.4758 Validation accuracy: 0.7542\n",
      "Epoch 37, CIFAR-10 Batch 2:  Loss: 0.4549 Validation accuracy: 0.7330\n",
      "Epoch 37, CIFAR-10 Batch 3:  Loss: 0.2760 Validation accuracy: 0.7516\n",
      "Epoch 37, CIFAR-10 Batch 4:  Loss: 0.3448 Validation accuracy: 0.7626\n",
      "Epoch 37, CIFAR-10 Batch 5:  Loss: 0.4612 Validation accuracy: 0.7534\n",
      "Epoch 38, CIFAR-10 Batch 1:  Loss: 0.4503 Validation accuracy: 0.7660\n",
      "Epoch 38, CIFAR-10 Batch 2:  Loss: 0.4484 Validation accuracy: 0.7472\n",
      "Epoch 38, CIFAR-10 Batch 3:  Loss: 0.2351 Validation accuracy: 0.7462\n",
      "Epoch 38, CIFAR-10 Batch 4:  Loss: 0.2826 Validation accuracy: 0.7678\n",
      "Epoch 38, CIFAR-10 Batch 5:  Loss: 0.4590 Validation accuracy: 0.7422\n",
      "Epoch 39, CIFAR-10 Batch 1:  Loss: 0.4717 Validation accuracy: 0.7708\n",
      "Epoch 39, CIFAR-10 Batch 2:  Loss: 0.4467 Validation accuracy: 0.7362\n",
      "Epoch 39, CIFAR-10 Batch 3:  Loss: 0.2619 Validation accuracy: 0.7482\n",
      "Epoch 39, CIFAR-10 Batch 4:  Loss: 0.4140 Validation accuracy: 0.7434\n",
      "Epoch 39, CIFAR-10 Batch 5:  Loss: 0.4203 Validation accuracy: 0.7468\n",
      "Epoch 40, CIFAR-10 Batch 1:  Loss: 0.4056 Validation accuracy: 0.7726\n",
      "Epoch 40, CIFAR-10 Batch 2:  Loss: 0.3444 Validation accuracy: 0.7546\n",
      "Epoch 40, CIFAR-10 Batch 3:  Loss: 0.2233 Validation accuracy: 0.7584\n",
      "Epoch 40, CIFAR-10 Batch 4:  Loss: 0.3060 Validation accuracy: 0.7662\n",
      "Epoch 40, CIFAR-10 Batch 5:  Loss: 0.3682 Validation accuracy: 0.7660\n",
      "Epoch 41, CIFAR-10 Batch 1:  Loss: 0.3625 Validation accuracy: 0.7674\n",
      "Epoch 41, CIFAR-10 Batch 2:  Loss: 0.4729 Validation accuracy: 0.7300\n",
      "Epoch 41, CIFAR-10 Batch 3:  Loss: 0.2666 Validation accuracy: 0.7516\n",
      "Epoch 41, CIFAR-10 Batch 4:  Loss: 0.3350 Validation accuracy: 0.7604\n",
      "Epoch 41, CIFAR-10 Batch 5:  Loss: 0.3772 Validation accuracy: 0.7582\n",
      "Epoch 42, CIFAR-10 Batch 1:  Loss: 0.3942 Validation accuracy: 0.7756\n",
      "Epoch 42, CIFAR-10 Batch 2:  Loss: 0.4813 Validation accuracy: 0.7186\n",
      "Epoch 42, CIFAR-10 Batch 3:  Loss: 0.2079 Validation accuracy: 0.7580\n",
      "Epoch 42, CIFAR-10 Batch 4:  Loss: 0.2458 Validation accuracy: 0.7702\n",
      "Epoch 42, CIFAR-10 Batch 5:  Loss: 0.3338 Validation accuracy: 0.7546\n",
      "Epoch 43, CIFAR-10 Batch 1:  Loss: 0.3755 Validation accuracy: 0.7750\n",
      "Epoch 43, CIFAR-10 Batch 2:  Loss: 0.4399 Validation accuracy: 0.7462\n",
      "Epoch 43, CIFAR-10 Batch 3:  Loss: 0.2001 Validation accuracy: 0.7612\n",
      "Epoch 43, CIFAR-10 Batch 4:  Loss: 0.2912 Validation accuracy: 0.7786\n",
      "Epoch 43, CIFAR-10 Batch 5:  Loss: 0.3062 Validation accuracy: 0.7678\n",
      "Epoch 44, CIFAR-10 Batch 1:  Loss: 0.4086 Validation accuracy: 0.7740\n",
      "Epoch 44, CIFAR-10 Batch 2:  Loss: 0.4027 Validation accuracy: 0.7656\n",
      "Epoch 44, CIFAR-10 Batch 3:  Loss: 0.1631 Validation accuracy: 0.7832\n",
      "Epoch 44, CIFAR-10 Batch 4:  Loss: 0.3224 Validation accuracy: 0.7648\n",
      "Epoch 44, CIFAR-10 Batch 5:  Loss: 0.2670 Validation accuracy: 0.7748\n",
      "Epoch 45, CIFAR-10 Batch 1:  Loss: 0.3738 Validation accuracy: 0.7678\n",
      "Epoch 45, CIFAR-10 Batch 2:  Loss: 0.3651 Validation accuracy: 0.7568\n",
      "Epoch 45, CIFAR-10 Batch 3:  Loss: 0.1508 Validation accuracy: 0.7742\n",
      "Epoch 45, CIFAR-10 Batch 4:  Loss: 0.2891 Validation accuracy: 0.7648\n",
      "Epoch 45, CIFAR-10 Batch 5:  Loss: 0.4204 Validation accuracy: 0.7386\n",
      "Epoch 46, CIFAR-10 Batch 1:  Loss: 0.3550 Validation accuracy: 0.7776\n",
      "Epoch 46, CIFAR-10 Batch 2:  Loss: 0.3821 Validation accuracy: 0.7416\n",
      "Epoch 46, CIFAR-10 Batch 3:  Loss: 0.2111 Validation accuracy: 0.7602\n",
      "Epoch 46, CIFAR-10 Batch 4:  Loss: 0.3406 Validation accuracy: 0.7542\n",
      "Epoch 46, CIFAR-10 Batch 5:  Loss: 0.3079 Validation accuracy: 0.7642\n",
      "Epoch 47, CIFAR-10 Batch 1:  Loss: 0.3368 Validation accuracy: 0.7812\n",
      "Epoch 47, CIFAR-10 Batch 2:  Loss: 0.4243 Validation accuracy: 0.7494\n",
      "Epoch 47, CIFAR-10 Batch 3:  Loss: 0.1870 Validation accuracy: 0.7732\n",
      "Epoch 47, CIFAR-10 Batch 4:  Loss: 0.2943 Validation accuracy: 0.7746\n",
      "Epoch 47, CIFAR-10 Batch 5:  Loss: 0.3376 Validation accuracy: 0.7630\n",
      "Epoch 48, CIFAR-10 Batch 1:  Loss: 0.2983 Validation accuracy: 0.7802\n",
      "Epoch 48, CIFAR-10 Batch 2:  Loss: 0.3401 Validation accuracy: 0.7596\n",
      "Epoch 48, CIFAR-10 Batch 3:  Loss: 0.2431 Validation accuracy: 0.7472\n",
      "Epoch 48, CIFAR-10 Batch 4:  Loss: 0.2416 Validation accuracy: 0.7828\n",
      "Epoch 48, CIFAR-10 Batch 5:  Loss: 0.3260 Validation accuracy: 0.7650\n",
      "Epoch 49, CIFAR-10 Batch 1:  Loss: 0.4203 Validation accuracy: 0.7760\n",
      "Epoch 49, CIFAR-10 Batch 2:  Loss: 0.3602 Validation accuracy: 0.7672\n",
      "Epoch 49, CIFAR-10 Batch 3:  Loss: 0.2028 Validation accuracy: 0.7716\n",
      "Epoch 49, CIFAR-10 Batch 4:  Loss: 0.3143 Validation accuracy: 0.7670\n",
      "Epoch 49, CIFAR-10 Batch 5:  Loss: 0.2929 Validation accuracy: 0.7778\n",
      "Epoch 50, CIFAR-10 Batch 1:  Loss: 0.3068 Validation accuracy: 0.7854\n",
      "Epoch 50, CIFAR-10 Batch 2:  Loss: 0.3333 Validation accuracy: 0.7758\n",
      "Epoch 50, CIFAR-10 Batch 3:  Loss: 0.1780 Validation accuracy: 0.7840\n",
      "Epoch 50, CIFAR-10 Batch 4:  Loss: 0.1759 Validation accuracy: 0.7898\n",
      "Epoch 50, CIFAR-10 Batch 5:  Loss: 0.2833 Validation accuracy: 0.7582\n",
      "Epoch 51, CIFAR-10 Batch 1:  Loss: 0.2796 Validation accuracy: 0.7930\n",
      "Epoch 51, CIFAR-10 Batch 2:  Loss: 0.3627 Validation accuracy: 0.7500\n",
      "Epoch 51, CIFAR-10 Batch 3:  Loss: 0.1800 Validation accuracy: 0.7796\n",
      "Epoch 51, CIFAR-10 Batch 4:  Loss: 0.2393 Validation accuracy: 0.7864\n",
      "Epoch 51, CIFAR-10 Batch 5:  Loss: 0.2397 Validation accuracy: 0.7740\n",
      "Epoch 52, CIFAR-10 Batch 1:  Loss: 0.3247 Validation accuracy: 0.7796\n",
      "Epoch 52, CIFAR-10 Batch 2:  Loss: 0.3026 Validation accuracy: 0.7674\n",
      "Epoch 52, CIFAR-10 Batch 3:  Loss: 0.1730 Validation accuracy: 0.7910\n",
      "Epoch 52, CIFAR-10 Batch 4:  Loss: 0.2583 Validation accuracy: 0.7726\n",
      "Epoch 52, CIFAR-10 Batch 5:  Loss: 0.2196 Validation accuracy: 0.7788\n",
      "Epoch 53, CIFAR-10 Batch 1:  Loss: 0.4214 Validation accuracy: 0.7518\n",
      "Epoch 53, CIFAR-10 Batch 2:  Loss: 0.3944 Validation accuracy: 0.7486\n",
      "Epoch 53, CIFAR-10 Batch 3:  Loss: 0.1476 Validation accuracy: 0.7740\n",
      "Epoch 53, CIFAR-10 Batch 4:  Loss: 0.2140 Validation accuracy: 0.7808\n",
      "Epoch 53, CIFAR-10 Batch 5:  Loss: 0.2617 Validation accuracy: 0.7674\n",
      "Epoch 54, CIFAR-10 Batch 1:  Loss: 0.2609 Validation accuracy: 0.7938\n",
      "Epoch 54, CIFAR-10 Batch 2:  Loss: 0.2499 Validation accuracy: 0.7738\n",
      "Epoch 54, CIFAR-10 Batch 3:  Loss: 0.1502 Validation accuracy: 0.7756\n",
      "Epoch 54, CIFAR-10 Batch 4:  Loss: 0.1793 Validation accuracy: 0.7850\n",
      "Epoch 54, CIFAR-10 Batch 5:  Loss: 0.1937 Validation accuracy: 0.7770\n",
      "Epoch 55, CIFAR-10 Batch 1:  Loss: 0.3051 Validation accuracy: 0.7806\n",
      "Epoch 55, CIFAR-10 Batch 2:  Loss: 0.2890 Validation accuracy: 0.7500\n",
      "Epoch 55, CIFAR-10 Batch 3:  Loss: 0.1653 Validation accuracy: 0.7772\n",
      "Epoch 55, CIFAR-10 Batch 4:  Loss: 0.1595 Validation accuracy: 0.7824\n",
      "Epoch 55, CIFAR-10 Batch 5:  Loss: 0.1911 Validation accuracy: 0.7754\n",
      "Epoch 56, CIFAR-10 Batch 1:  Loss: 0.2250 Validation accuracy: 0.7918\n",
      "Epoch 56, CIFAR-10 Batch 2:  Loss: 0.2951 Validation accuracy: 0.7878\n",
      "Epoch 56, CIFAR-10 Batch 3:  Loss: 0.1285 Validation accuracy: 0.7808\n",
      "Epoch 56, CIFAR-10 Batch 4:  Loss: 0.2379 Validation accuracy: 0.7756\n",
      "Epoch 56, CIFAR-10 Batch 5:  Loss: 0.1885 Validation accuracy: 0.7774\n",
      "Epoch 57, CIFAR-10 Batch 1:  Loss: 0.2594 Validation accuracy: 0.7924\n",
      "Epoch 57, CIFAR-10 Batch 2:  Loss: 0.3128 Validation accuracy: 0.7570\n",
      "Epoch 57, CIFAR-10 Batch 3:  Loss: 0.1476 Validation accuracy: 0.7640\n",
      "Epoch 57, CIFAR-10 Batch 4:  Loss: 0.2178 Validation accuracy: 0.7702\n",
      "Epoch 57, CIFAR-10 Batch 5:  Loss: 0.2304 Validation accuracy: 0.7680\n",
      "Epoch 58, CIFAR-10 Batch 1:  Loss: 0.2689 Validation accuracy: 0.7930\n",
      "Epoch 58, CIFAR-10 Batch 2:  Loss: 0.3053 Validation accuracy: 0.7706\n",
      "Epoch 58, CIFAR-10 Batch 3:  Loss: 0.1252 Validation accuracy: 0.7884\n",
      "Epoch 58, CIFAR-10 Batch 4:  Loss: 0.2797 Validation accuracy: 0.7656\n",
      "Epoch 58, CIFAR-10 Batch 5:  Loss: 0.2125 Validation accuracy: 0.7656\n",
      "Epoch 59, CIFAR-10 Batch 1:  Loss: 0.2430 Validation accuracy: 0.7940\n",
      "Epoch 59, CIFAR-10 Batch 2:  Loss: 0.3980 Validation accuracy: 0.7430\n",
      "Epoch 59, CIFAR-10 Batch 3:  Loss: 0.1469 Validation accuracy: 0.7680\n",
      "Epoch 59, CIFAR-10 Batch 4:  Loss: 0.1431 Validation accuracy: 0.7886\n",
      "Epoch 59, CIFAR-10 Batch 5:  Loss: 0.2281 Validation accuracy: 0.7842\n",
      "Epoch 60, CIFAR-10 Batch 1:  Loss: 0.2104 Validation accuracy: 0.7974\n",
      "Epoch 60, CIFAR-10 Batch 2:  Loss: 0.2313 Validation accuracy: 0.7764\n",
      "Epoch 60, CIFAR-10 Batch 3:  Loss: 0.1534 Validation accuracy: 0.7708\n",
      "Epoch 60, CIFAR-10 Batch 4:  Loss: 0.2055 Validation accuracy: 0.7812\n",
      "Epoch 60, CIFAR-10 Batch 5:  Loss: 0.1605 Validation accuracy: 0.7858\n",
      "Epoch 61, CIFAR-10 Batch 1:  Loss: 0.2185 Validation accuracy: 0.7918\n",
      "Epoch 61, CIFAR-10 Batch 2:  Loss: 0.3022 Validation accuracy: 0.7768\n",
      "Epoch 61, CIFAR-10 Batch 3:  Loss: 0.1311 Validation accuracy: 0.7738\n",
      "Epoch 61, CIFAR-10 Batch 4:  Loss: 0.1823 Validation accuracy: 0.7960\n",
      "Epoch 61, CIFAR-10 Batch 5:  Loss: 0.1947 Validation accuracy: 0.7726\n",
      "Epoch 62, CIFAR-10 Batch 1:  Loss: 0.2108 Validation accuracy: 0.7918\n",
      "Epoch 62, CIFAR-10 Batch 2:  Loss: 0.2189 Validation accuracy: 0.7958\n",
      "Epoch 62, CIFAR-10 Batch 3:  Loss: 0.1121 Validation accuracy: 0.7890\n",
      "Epoch 62, CIFAR-10 Batch 4:  Loss: 0.1637 Validation accuracy: 0.7792\n",
      "Epoch 62, CIFAR-10 Batch 5:  Loss: 0.1579 Validation accuracy: 0.7782\n",
      "Epoch 63, CIFAR-10 Batch 1:  Loss: 0.1977 Validation accuracy: 0.8020\n",
      "Epoch 63, CIFAR-10 Batch 2:  Loss: 0.3488 Validation accuracy: 0.7688\n",
      "Epoch 63, CIFAR-10 Batch 3:  Loss: 0.1014 Validation accuracy: 0.7934\n",
      "Epoch 63, CIFAR-10 Batch 4:  Loss: 0.1538 Validation accuracy: 0.7826\n",
      "Epoch 63, CIFAR-10 Batch 5:  Loss: 0.1574 Validation accuracy: 0.7784\n",
      "Epoch 64, CIFAR-10 Batch 1:  Loss: 0.2288 Validation accuracy: 0.7984\n",
      "Epoch 64, CIFAR-10 Batch 2:  Loss: 0.2557 Validation accuracy: 0.7912\n",
      "Epoch 64, CIFAR-10 Batch 3:  Loss: 0.1366 Validation accuracy: 0.7850\n",
      "Epoch 64, CIFAR-10 Batch 4:  Loss: 0.1378 Validation accuracy: 0.7906\n",
      "Epoch 64, CIFAR-10 Batch 5:  Loss: 0.1824 Validation accuracy: 0.7624\n",
      "Epoch 65, CIFAR-10 Batch 1:  Loss: 0.1800 Validation accuracy: 0.7996\n",
      "Epoch 65, CIFAR-10 Batch 2:  Loss: 0.2769 Validation accuracy: 0.7864\n",
      "Epoch 65, CIFAR-10 Batch 3:  Loss: 0.1548 Validation accuracy: 0.7804\n",
      "Epoch 65, CIFAR-10 Batch 4:  Loss: 0.2001 Validation accuracy: 0.7840\n",
      "Epoch 65, CIFAR-10 Batch 5:  Loss: 0.1688 Validation accuracy: 0.7872\n",
      "Epoch 66, CIFAR-10 Batch 1:  Loss: 0.1745 Validation accuracy: 0.8028\n",
      "Epoch 66, CIFAR-10 Batch 2:  Loss: 0.2126 Validation accuracy: 0.8016\n",
      "Epoch 66, CIFAR-10 Batch 3:  Loss: 0.1027 Validation accuracy: 0.7996\n",
      "Epoch 66, CIFAR-10 Batch 4:  Loss: 0.1373 Validation accuracy: 0.7946\n",
      "Epoch 66, CIFAR-10 Batch 5:  Loss: 0.1478 Validation accuracy: 0.7954\n",
      "Epoch 67, CIFAR-10 Batch 1:  Loss: 0.1911 Validation accuracy: 0.7978\n",
      "Epoch 67, CIFAR-10 Batch 2:  Loss: 0.2119 Validation accuracy: 0.7738\n",
      "Epoch 67, CIFAR-10 Batch 3:  Loss: 0.1078 Validation accuracy: 0.7992\n",
      "Epoch 67, CIFAR-10 Batch 4:  Loss: 0.1382 Validation accuracy: 0.8014\n",
      "Epoch 67, CIFAR-10 Batch 5:  Loss: 0.2373 Validation accuracy: 0.7394\n",
      "Epoch 68, CIFAR-10 Batch 1:  Loss: 0.2340 Validation accuracy: 0.7902\n",
      "Epoch 68, CIFAR-10 Batch 2:  Loss: 0.2786 Validation accuracy: 0.7854\n",
      "Epoch 68, CIFAR-10 Batch 3:  Loss: 0.1526 Validation accuracy: 0.7874\n",
      "Epoch 68, CIFAR-10 Batch 4:  Loss: 0.1247 Validation accuracy: 0.7998\n",
      "Epoch 68, CIFAR-10 Batch 5:  Loss: 0.1383 Validation accuracy: 0.7774\n",
      "Epoch 69, CIFAR-10 Batch 1:  Loss: 0.1686 Validation accuracy: 0.8028\n",
      "Epoch 69, CIFAR-10 Batch 2:  Loss: 0.1522 Validation accuracy: 0.7910\n",
      "Epoch 69, CIFAR-10 Batch 3:  Loss: 0.1122 Validation accuracy: 0.7866\n",
      "Epoch 69, CIFAR-10 Batch 4:  Loss: 0.1293 Validation accuracy: 0.7992\n",
      "Epoch 69, CIFAR-10 Batch 5:  Loss: 0.1595 Validation accuracy: 0.7754\n",
      "Epoch 70, CIFAR-10 Batch 1:  Loss: 0.1860 Validation accuracy: 0.7960\n",
      "Epoch 70, CIFAR-10 Batch 2:  Loss: 0.2320 Validation accuracy: 0.7810\n",
      "Epoch 70, CIFAR-10 Batch 3:  Loss: 0.1015 Validation accuracy: 0.7930\n",
      "Epoch 70, CIFAR-10 Batch 4:  Loss: 0.1391 Validation accuracy: 0.7888\n",
      "Epoch 70, CIFAR-10 Batch 5:  Loss: 0.1219 Validation accuracy: 0.7876\n",
      "Epoch 71, CIFAR-10 Batch 1:  Loss: 0.1839 Validation accuracy: 0.7998\n",
      "Epoch 71, CIFAR-10 Batch 2:  Loss: 0.2616 Validation accuracy: 0.7748\n",
      "Epoch 71, CIFAR-10 Batch 3:  Loss: 0.1152 Validation accuracy: 0.7928\n",
      "Epoch 71, CIFAR-10 Batch 4:  Loss: 0.1537 Validation accuracy: 0.7924\n",
      "Epoch 71, CIFAR-10 Batch 5:  Loss: 0.1493 Validation accuracy: 0.7744\n",
      "Epoch 72, CIFAR-10 Batch 1:  Loss: 0.2029 Validation accuracy: 0.7908\n",
      "Epoch 72, CIFAR-10 Batch 2:  Loss: 0.1624 Validation accuracy: 0.7638\n",
      "Epoch 72, CIFAR-10 Batch 3:  Loss: 0.0934 Validation accuracy: 0.7886\n",
      "Epoch 72, CIFAR-10 Batch 4:  Loss: 0.1471 Validation accuracy: 0.7932\n",
      "Epoch 72, CIFAR-10 Batch 5:  Loss: 0.1407 Validation accuracy: 0.7748\n",
      "Epoch 73, CIFAR-10 Batch 1:  Loss: 0.1643 Validation accuracy: 0.8012\n",
      "Epoch 73, CIFAR-10 Batch 2:  Loss: 0.1730 Validation accuracy: 0.7902\n",
      "Epoch 73, CIFAR-10 Batch 3:  Loss: 0.0917 Validation accuracy: 0.7874\n",
      "Epoch 73, CIFAR-10 Batch 4:  Loss: 0.1485 Validation accuracy: 0.7936\n",
      "Epoch 73, CIFAR-10 Batch 5:  Loss: 0.1152 Validation accuracy: 0.7808\n",
      "Epoch 74, CIFAR-10 Batch 1:  Loss: 0.1633 Validation accuracy: 0.8062\n",
      "Epoch 74, CIFAR-10 Batch 2:  Loss: 0.1678 Validation accuracy: 0.7840\n",
      "Epoch 74, CIFAR-10 Batch 3:  Loss: 0.1269 Validation accuracy: 0.7852\n",
      "Epoch 74, CIFAR-10 Batch 4:  Loss: 0.1312 Validation accuracy: 0.7986\n",
      "Epoch 74, CIFAR-10 Batch 5:  Loss: 0.1139 Validation accuracy: 0.7892\n",
      "Epoch 75, CIFAR-10 Batch 1:  Loss: 0.1657 Validation accuracy: 0.8112\n",
      "Epoch 75, CIFAR-10 Batch 2:  Loss: 0.1654 Validation accuracy: 0.7862\n",
      "Epoch 75, CIFAR-10 Batch 3:  Loss: 0.1159 Validation accuracy: 0.7912\n",
      "Epoch 75, CIFAR-10 Batch 4:  Loss: 0.1433 Validation accuracy: 0.7920\n",
      "Epoch 75, CIFAR-10 Batch 5:  Loss: 0.1360 Validation accuracy: 0.7724\n",
      "Epoch 76, CIFAR-10 Batch 1:  Loss: 0.1961 Validation accuracy: 0.7906\n",
      "Epoch 76, CIFAR-10 Batch 2:  Loss: 0.2075 Validation accuracy: 0.7870\n",
      "Epoch 76, CIFAR-10 Batch 3:  Loss: 0.0951 Validation accuracy: 0.7842\n",
      "Epoch 76, CIFAR-10 Batch 4:  Loss: 0.1121 Validation accuracy: 0.7996\n",
      "Epoch 76, CIFAR-10 Batch 5:  Loss: 0.1288 Validation accuracy: 0.7854\n",
      "Epoch 77, CIFAR-10 Batch 1:  Loss: 0.1840 Validation accuracy: 0.8010\n",
      "Epoch 77, CIFAR-10 Batch 2:  Loss: 0.1892 Validation accuracy: 0.7832\n",
      "Epoch 77, CIFAR-10 Batch 3:  Loss: 0.1098 Validation accuracy: 0.7830\n",
      "Epoch 77, CIFAR-10 Batch 4:  Loss: 0.1419 Validation accuracy: 0.7840\n",
      "Epoch 77, CIFAR-10 Batch 5:  Loss: 0.1269 Validation accuracy: 0.7782\n",
      "Epoch 78, CIFAR-10 Batch 1:  Loss: 0.1320 Validation accuracy: 0.8048\n",
      "Epoch 78, CIFAR-10 Batch 2:  Loss: 0.2269 Validation accuracy: 0.7710\n",
      "Epoch 78, CIFAR-10 Batch 3:  Loss: 0.0833 Validation accuracy: 0.7986\n",
      "Epoch 78, CIFAR-10 Batch 4:  Loss: 0.1124 Validation accuracy: 0.8016\n",
      "Epoch 78, CIFAR-10 Batch 5:  Loss: 0.1172 Validation accuracy: 0.7918\n",
      "Epoch 79, CIFAR-10 Batch 1:  Loss: 0.1883 Validation accuracy: 0.7976\n",
      "Epoch 79, CIFAR-10 Batch 2:  Loss: 0.2122 Validation accuracy: 0.7744\n",
      "Epoch 79, CIFAR-10 Batch 3:  Loss: 0.0791 Validation accuracy: 0.7946\n",
      "Epoch 79, CIFAR-10 Batch 4:  Loss: 0.1171 Validation accuracy: 0.7970\n",
      "Epoch 79, CIFAR-10 Batch 5:  Loss: 0.0984 Validation accuracy: 0.7922\n",
      "Epoch 80, CIFAR-10 Batch 1:  Loss: 0.1423 Validation accuracy: 0.8040\n",
      "Epoch 80, CIFAR-10 Batch 2:  Loss: 0.1860 Validation accuracy: 0.7740\n",
      "Epoch 80, CIFAR-10 Batch 3:  Loss: 0.1034 Validation accuracy: 0.7886\n",
      "Epoch 80, CIFAR-10 Batch 4:  Loss: 0.1106 Validation accuracy: 0.8004\n",
      "Epoch 80, CIFAR-10 Batch 5:  Loss: 0.1366 Validation accuracy: 0.7788\n",
      "Epoch 81, CIFAR-10 Batch 1:  Loss: 0.1414 Validation accuracy: 0.7990\n",
      "Epoch 81, CIFAR-10 Batch 2:  Loss: 0.1533 Validation accuracy: 0.7986\n",
      "Epoch 81, CIFAR-10 Batch 3:  Loss: 0.1284 Validation accuracy: 0.7786\n",
      "Epoch 81, CIFAR-10 Batch 4:  Loss: 0.1335 Validation accuracy: 0.7990\n",
      "Epoch 81, CIFAR-10 Batch 5:  Loss: 0.1116 Validation accuracy: 0.7894\n",
      "Epoch 82, CIFAR-10 Batch 1:  Loss: 0.1537 Validation accuracy: 0.8024\n",
      "Epoch 82, CIFAR-10 Batch 2:  Loss: 0.1856 Validation accuracy: 0.8002\n",
      "Epoch 82, CIFAR-10 Batch 3:  Loss: 0.0824 Validation accuracy: 0.7992\n",
      "Epoch 82, CIFAR-10 Batch 4:  Loss: 0.1401 Validation accuracy: 0.7850\n",
      "Epoch 82, CIFAR-10 Batch 5:  Loss: 0.1414 Validation accuracy: 0.7782\n",
      "Epoch 83, CIFAR-10 Batch 1:  Loss: 0.1195 Validation accuracy: 0.8096\n",
      "Epoch 83, CIFAR-10 Batch 2:  Loss: 0.2610 Validation accuracy: 0.7682\n",
      "Epoch 83, CIFAR-10 Batch 3:  Loss: 0.0832 Validation accuracy: 0.8072\n",
      "Epoch 83, CIFAR-10 Batch 4:  Loss: 0.0907 Validation accuracy: 0.7986\n",
      "Epoch 83, CIFAR-10 Batch 5:  Loss: 0.0920 Validation accuracy: 0.7930\n",
      "Epoch 84, CIFAR-10 Batch 1:  Loss: 0.1348 Validation accuracy: 0.8110\n",
      "Epoch 84, CIFAR-10 Batch 2:  Loss: 0.2541 Validation accuracy: 0.7706\n",
      "Epoch 84, CIFAR-10 Batch 3:  Loss: 0.0797 Validation accuracy: 0.7844\n",
      "Epoch 84, CIFAR-10 Batch 4:  Loss: 0.1078 Validation accuracy: 0.7934\n",
      "Epoch 84, CIFAR-10 Batch 5:  Loss: 0.1159 Validation accuracy: 0.7942\n",
      "Epoch 85, CIFAR-10 Batch 1:  Loss: 0.1271 Validation accuracy: 0.8014\n",
      "Epoch 85, CIFAR-10 Batch 2:  Loss: 0.1177 Validation accuracy: 0.8046\n",
      "Epoch 85, CIFAR-10 Batch 3:  Loss: 0.0624 Validation accuracy: 0.7976\n",
      "Epoch 85, CIFAR-10 Batch 4:  Loss: 0.1103 Validation accuracy: 0.7942\n",
      "Epoch 85, CIFAR-10 Batch 5:  Loss: 0.1341 Validation accuracy: 0.7906\n",
      "Epoch 86, CIFAR-10 Batch 1:  Loss: 0.1201 Validation accuracy: 0.8062\n",
      "Epoch 86, CIFAR-10 Batch 2:  Loss: 0.1765 Validation accuracy: 0.7882\n",
      "Epoch 86, CIFAR-10 Batch 3:  Loss: 0.0708 Validation accuracy: 0.8062\n",
      "Epoch 86, CIFAR-10 Batch 4:  Loss: 0.1349 Validation accuracy: 0.7956\n",
      "Epoch 86, CIFAR-10 Batch 5:  Loss: 0.1211 Validation accuracy: 0.7834\n",
      "Epoch 87, CIFAR-10 Batch 1:  Loss: 0.1157 Validation accuracy: 0.8106\n",
      "Epoch 87, CIFAR-10 Batch 2:  Loss: 0.2043 Validation accuracy: 0.7710\n",
      "Epoch 87, CIFAR-10 Batch 3:  Loss: 0.0819 Validation accuracy: 0.7948\n",
      "Epoch 87, CIFAR-10 Batch 4:  Loss: 0.1311 Validation accuracy: 0.7950\n",
      "Epoch 87, CIFAR-10 Batch 5:  Loss: 0.1913 Validation accuracy: 0.7636\n",
      "Epoch 88, CIFAR-10 Batch 1:  Loss: 0.1117 Validation accuracy: 0.8058\n",
      "Epoch 88, CIFAR-10 Batch 2:  Loss: 0.1439 Validation accuracy: 0.7954\n",
      "Epoch 88, CIFAR-10 Batch 3:  Loss: 0.0522 Validation accuracy: 0.8056\n",
      "Epoch 88, CIFAR-10 Batch 4:  Loss: 0.1357 Validation accuracy: 0.7950\n",
      "Epoch 88, CIFAR-10 Batch 5:  Loss: 0.1165 Validation accuracy: 0.7768\n",
      "Epoch 89, CIFAR-10 Batch 1:  Loss: 0.1221 Validation accuracy: 0.8062\n",
      "Epoch 89, CIFAR-10 Batch 2:  Loss: 0.1127 Validation accuracy: 0.7964\n",
      "Epoch 89, CIFAR-10 Batch 3:  Loss: 0.0645 Validation accuracy: 0.7946\n",
      "Epoch 89, CIFAR-10 Batch 4:  Loss: 0.1321 Validation accuracy: 0.7924\n",
      "Epoch 89, CIFAR-10 Batch 5:  Loss: 0.0726 Validation accuracy: 0.8034\n",
      "Epoch 90, CIFAR-10 Batch 1:  Loss: 0.1096 Validation accuracy: 0.8016\n",
      "Epoch 90, CIFAR-10 Batch 2:  Loss: 0.1396 Validation accuracy: 0.7910\n",
      "Epoch 90, CIFAR-10 Batch 3:  Loss: 0.0635 Validation accuracy: 0.7964\n",
      "Epoch 90, CIFAR-10 Batch 4:  Loss: 0.0912 Validation accuracy: 0.7990\n",
      "Epoch 90, CIFAR-10 Batch 5:  Loss: 0.1533 Validation accuracy: 0.7870\n",
      "Epoch 91, CIFAR-10 Batch 1:  Loss: 0.1010 Validation accuracy: 0.8040\n",
      "Epoch 91, CIFAR-10 Batch 2:  Loss: 0.0984 Validation accuracy: 0.7956\n",
      "Epoch 91, CIFAR-10 Batch 3:  Loss: 0.0590 Validation accuracy: 0.7966\n",
      "Epoch 91, CIFAR-10 Batch 4:  Loss: 0.1195 Validation accuracy: 0.8028\n",
      "Epoch 91, CIFAR-10 Batch 5:  Loss: 0.0825 Validation accuracy: 0.7942\n",
      "Epoch 92, CIFAR-10 Batch 1:  Loss: 0.0901 Validation accuracy: 0.8040\n",
      "Epoch 92, CIFAR-10 Batch 2:  Loss: 0.1974 Validation accuracy: 0.7726\n",
      "Epoch 92, CIFAR-10 Batch 3:  Loss: 0.0514 Validation accuracy: 0.8028\n",
      "Epoch 92, CIFAR-10 Batch 4:  Loss: 0.0845 Validation accuracy: 0.8076\n",
      "Epoch 92, CIFAR-10 Batch 5:  Loss: 0.0710 Validation accuracy: 0.7964\n",
      "Epoch 93, CIFAR-10 Batch 1:  Loss: 0.0888 Validation accuracy: 0.8094\n",
      "Epoch 93, CIFAR-10 Batch 2:  Loss: 0.1388 Validation accuracy: 0.8002\n",
      "Epoch 93, CIFAR-10 Batch 3:  Loss: 0.0655 Validation accuracy: 0.7954\n",
      "Epoch 93, CIFAR-10 Batch 4:  Loss: 0.0965 Validation accuracy: 0.8032\n",
      "Epoch 93, CIFAR-10 Batch 5:  Loss: 0.1014 Validation accuracy: 0.7820\n",
      "Epoch 94, CIFAR-10 Batch 1:  Loss: 0.0963 Validation accuracy: 0.8052\n",
      "Epoch 94, CIFAR-10 Batch 2:  Loss: 0.0870 Validation accuracy: 0.7954\n",
      "Epoch 94, CIFAR-10 Batch 3:  Loss: 0.0633 Validation accuracy: 0.7970\n",
      "Epoch 94, CIFAR-10 Batch 4:  Loss: 0.1384 Validation accuracy: 0.7906\n",
      "Epoch 94, CIFAR-10 Batch 5:  Loss: 0.1351 Validation accuracy: 0.7794\n",
      "Epoch 95, CIFAR-10 Batch 1:  Loss: 0.1170 Validation accuracy: 0.8086\n",
      "Epoch 95, CIFAR-10 Batch 2:  Loss: 0.1256 Validation accuracy: 0.7874\n",
      "Epoch 95, CIFAR-10 Batch 3:  Loss: 0.0698 Validation accuracy: 0.7910\n",
      "Epoch 95, CIFAR-10 Batch 4:  Loss: 0.1098 Validation accuracy: 0.8066\n",
      "Epoch 95, CIFAR-10 Batch 5:  Loss: 0.0763 Validation accuracy: 0.7908\n",
      "Epoch 96, CIFAR-10 Batch 1:  Loss: 0.1211 Validation accuracy: 0.8110\n",
      "Epoch 96, CIFAR-10 Batch 2:  Loss: 0.1398 Validation accuracy: 0.7974\n",
      "Epoch 96, CIFAR-10 Batch 3:  Loss: 0.0468 Validation accuracy: 0.8050\n",
      "Epoch 96, CIFAR-10 Batch 4:  Loss: 0.1001 Validation accuracy: 0.8054\n",
      "Epoch 96, CIFAR-10 Batch 5:  Loss: 0.0914 Validation accuracy: 0.7926\n",
      "Epoch 97, CIFAR-10 Batch 1:  Loss: 0.0984 Validation accuracy: 0.8104\n",
      "Epoch 97, CIFAR-10 Batch 2:  Loss: 0.1671 Validation accuracy: 0.7886\n",
      "Epoch 97, CIFAR-10 Batch 3:  Loss: 0.0755 Validation accuracy: 0.8018\n",
      "Epoch 97, CIFAR-10 Batch 4:  Loss: 0.1085 Validation accuracy: 0.7872\n",
      "Epoch 97, CIFAR-10 Batch 5:  Loss: 0.0829 Validation accuracy: 0.7832\n",
      "Epoch 98, CIFAR-10 Batch 1:  Loss: 0.1153 Validation accuracy: 0.7980\n",
      "Epoch 98, CIFAR-10 Batch 2:  Loss: 0.0839 Validation accuracy: 0.8014\n",
      "Epoch 98, CIFAR-10 Batch 3:  Loss: 0.0712 Validation accuracy: 0.7966\n",
      "Epoch 98, CIFAR-10 Batch 4:  Loss: 0.0978 Validation accuracy: 0.7998\n",
      "Epoch 98, CIFAR-10 Batch 5:  Loss: 0.1060 Validation accuracy: 0.7782\n",
      "Epoch 99, CIFAR-10 Batch 1:  Loss: 0.0852 Validation accuracy: 0.8100\n",
      "Epoch 99, CIFAR-10 Batch 2:  Loss: 0.1131 Validation accuracy: 0.8000\n",
      "Epoch 99, CIFAR-10 Batch 3:  Loss: 0.0601 Validation accuracy: 0.8044\n",
      "Epoch 99, CIFAR-10 Batch 4:  Loss: 0.1330 Validation accuracy: 0.7878\n",
      "Epoch 99, CIFAR-10 Batch 5:  Loss: 0.0995 Validation accuracy: 0.7818\n",
      "Epoch 100, CIFAR-10 Batch 1:  Loss: 0.1144 Validation accuracy: 0.8092\n",
      "Epoch 100, CIFAR-10 Batch 2:  Loss: 0.1201 Validation accuracy: 0.7886\n",
      "Epoch 100, CIFAR-10 Batch 3:  Loss: 0.0844 Validation accuracy: 0.7842\n",
      "Epoch 100, CIFAR-10 Batch 4:  Loss: 0.0666 Validation accuracy: 0.8002\n",
      "Epoch 100, CIFAR-10 Batch 5:  Loss: 0.0634 Validation accuracy: 0.8002\n"
     ]
    }
   ],
   "source": [
    "save_model_path = './image_classification'\n",
    "\n",
    "print('Training...')\n",
    "with tf.Session() as sess:\n",
    "    # Initializing the variables\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # Training cycle\n",
    "    for epoch in range(epochs):\n",
    "        # Loop over all batches\n",
    "        n_batches = 5\n",
    "        for batch_i in range(1, n_batches + 1):\n",
    "            for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):\n",
    "                train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)\n",
    "            print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')\n",
    "            print_stats(sess, batch_features, batch_labels, cost, accuracy)\n",
    "            \n",
    "    # Save Model\n",
    "    saver = tf.train.Saver()\n",
    "    save_path = saver.save(sess, save_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Checkpoint\n",
    "The model has been saved to disk.\n",
    "## Test Model\n",
    "Test your model against the test dataset.  This will be your final accuracy. You should have an accuracy greater than 50%. If you don't, keep tweaking the model architecture and parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing Accuracy: 0.7826344936708861\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAscAAAJ/CAYAAACUb342AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAWJQAAFiUBSVIk8AAAIABJREFUeJzs3XecZFWd///Xp3u6e3pyDkygGdIMAgJDEJGki4KYFVDU\nFdz1K6ioGFbXsMK6hlVXUEzrGlAXBbO/VVFM5CRBEJghzNAMk2NPd890rs/vj3Oq7u071dXV07n7\n/Xw86nGr7jn33FP51KnPOcfcHRERERERgYrhroCIiIiIyEihxrGIiIiISKTGsYiIiIhIpMaxiIiI\niEikxrGIiIiISKTGsYiIiIhIpMaxiIiIiEikxrGIiIiISKTGsYiIiIhIpMaxiIiIiEikxrGIiIiI\nSKTGsYiIiIhIpMaxiIiIiEikxrGIiIiISKTG8TAzswPN7DVmdqmZ/auZfdjMLjOz88zseDObMtx1\n7ImZVZjZK83sejN7yswazcxTl18Odx1FRhozq8u8T64YiLwjlZmdkbkPFw13nURESpkw3BUYj8xs\nFnAp8DbgwF6y58zsMeA24DfAn9y9dZCr2Kt4H34KnDncdZGhZ2bXAm/pJVsn0ABsBx4gvIZ/5O67\nB7d2IiIi+089x0PMzF4GPAb8B703jCE8R0cSGtO/Bl43eLXrk+/Th4axeo/GpQnAHGA5cCHwdWCD\nmV1hZvphPopk3rvXDnd9REQGk76ghpCZnQ/8EKjMJDUCfwc2A23ATGApsIIR+APGzJ4HnJva9Qxw\nJXAf0JTav3co6yWjwmTgE8BpZnaOu7cNd4VERETS1DgeImZ2MKG3Nd0wfgT4KPBbd+8scswU4HTg\nPODVwLQhqGo5XpO5/Up3f2hYaiIjxQcJYTZpE4D5wAuAdxB+8OWdSehJfuuQ1E5ERKRMahwPnU8B\nNanbfwRe4e4tPR3g7s2EOOPfmNllwD8TepeH28rU9Xo1jAXY7u71RfY/BdxhZl8GriP8yMu7yMy+\n7O5/G4oKjkbxMbXhrkd/uPvNjPL7ICLjy4j7y34sMrNa4BWpXR3AW0o1jLPcvcndr3L3Pw54Bftu\nXur6xmGrhYwa8bX+RuCJ1G4DLhmeGomIiBSnxvHQOA6oTd2+091Hc6MyPb1cx7DVQkaV2EC+KrP7\nRcNRFxERkZ4orGJoLMjc3jCUJzezacCpwCJgNmHQ3BbgHndftz9FDmD1BoSZLSOEeywGqoF64C/u\nvrWX4xYTYmKXEO7Xpnjc+n7UZRHwHGAZMCPu3gmsA+4a51OZ/Slz+2Azq3T3rr4UYmZHAkcACwmD\n/Ord/YdlHFcDPJ8wU8w8oIvwXnjY3R/uSx16KP9Q4ETgAKAVWA/c6+5D+p4vUq/DgGOAuYTX5F7C\na/0R4DF3zw1j9XplZkuA5xFi2KcS3k8bgdvcvWGAz7WM0KGxhDBGZAtwh7uv7UeZhxMe/wWEzoVO\noBl4FngSWO3u3s+qi8hAcXddBvkCvB7w1OXGITrv8cCNQHvm/OnLw4RptqxEOWeUOL6ny83x2Pr9\nPTZTh2vTeVL7Twf+AuSKlNMOfA2YUqS8I4Df9nBcDvgZsKjMx7ki1uPrwJpe7lsXId78zDLL/l7m\n+G/24fn/TObYX5d6nvv42ro2U/ZFZR5XW+QxmVckX/p1c3Nq/8WEBl22jIZeznsk8BNgT4nn5lng\nvUDVfjwepwD39FBuJ2HswMqYty6TfkWJcsvOW+TYGcC/E36UlXpNbgO+A5zQy3Nc1qWMz4+yXivx\n2POBv5U4XwfwB+B5fSjz5tTx9an9JxF+vBX7THDgbuDkPpynCng/Ie6+t8etgfCZc9ZAvD910UWX\n/l2GvQLj4QK8MPNB2ATMGMTzGfC5Eh/yxS43AzN7KC/75VZWefHY+v09NlOHbl/Ucd+7y7yPfyXV\nQCbMtrG3jOPqgaVlPN5v3Y/76MB/AZW9lD0ZWJU57vVl1OmszGOzHpg9gK+xazN1uqjM4yYWeRzm\nFsmXft3cTBjM+uMSj2XRxjHhh8vnCT9Kyn1eHqLMH0bxHB8p83XYToi7rsvsv6JE2WXnzRz3amBX\nH1+Pf+vlOS7rUsbnR6+vFcLMPH/s47mvBirKKPvm1DH1cd9llO5ESD+H55dxjrmEhW/6+vj9cqDe\no7roosv+XxRWMTTuJ3w556dxmwJ838wu9DAjxUD7H+CfMvvaCT0fGwk9SscTFmjIOx241cxOc/dd\ng1CnARXnjP5SvOmE3qU1hB8GxwAHp7IfD1wDXGxmZwI3kIQUrY6XdsK80keljjuQ0HPb22In2dj9\nFuBRwt/WjYTe0qXA0YSQj7z3EXq+PtxTwe6+x8wuIPRKToy7v2lm97n7U8WOMbMFwA9Iwl+6gAvd\nfUcv92MoLM7cdkIjrjdXE6Y0zB/zIEkDehlwUPYAM6skPNevzSTtJbwnNxHekwcDzyV5vI4G7jSz\nE919S6lKmdl7CTPRpHURnq9nCSEAxxLCP6oIDc7se3NAxTp9kX3DnzYT/inaDkwiPBdH0X0WnWFn\nZlOBWwjv47RdwL1xu5AQZpGu+3sIn2lv6uP53gh8ObXrEUJvbxvhtbGS5LGsAq41swfd/ckeyjPg\n54TnPW0LYT777YQfU9Nj+YegEEeRkWW4W+fj5UL4SzvbS7CRsCDCUQzc391vyZwjR2hYzMjkm0D4\nkt6dyf+jImVOJPRg5S/rU/nvzqTlLwvisYvj7WxoyQd6OK5wbKYO12aOz/eK/QY4uEj+8wmN1PTj\ncHJ8zB24EzimyHFnADsy53ppL495foq9z8RzFO29Ivwo+RDd/9rPASeV8bxekqnTfUB1kXwVhL+Z\n03k/Pgiv5+zzcVGZx/2/zHFP9ZCvPpWnKXX9B8DiIvnriuz7VOZcWwhhGcUet4PZ9z36217uy1Hs\n29v4w+zrNz4n5wNbY56dmWOuKHGOunLzxvwvYd9e8lsIcdb7fMYQGpcvJ/ylf38mbQ7JezJd3k/p\n+b1b7Hk4oy+vFeC7mfyNwNvJhLsQGpf/xb699m/vpfybU3mbST4nfgEcUiT/CsK/Celz3FCi/HMz\neZ8kDDwt+hlP+HfolcD1wE8G+r2qiy669P0y7BUYLxdCz1Rr5kMzfdlBaOh9nPCX+OT9OMcU9v0r\n9fJejjmJfeMwS8a90UM8aC/H9OkLssjx1xZ5zK6jxN+ohCW3izWo/wjUlDjuZeV+Ecb8C0qVVyT/\nyZnXQsnyU8fdkKnXl4rk+Wgmz59LPUb9eD1nn49en0/Cj6xsiEjRGGqKh+N8tg/1O4nujcTHKfKj\nK3NMBfvGeJ9TIv9fMnm/2kv5z2HfhvGANY4JvcFbMvm/Uu7zD8wvkZYu89o+vlbKfu8TBsem8+4F\nTuml/HdljmmmhxCxmP/mIs/BVyg97mI+3T9b23o6B2HsQT5fB3BQHx6riX15bHXRRZfBuWgqtyHi\nYaGMNxMaRcXMAl5KGEBzE7DLzG4zs7fH2SbK8RaS2REAfufu2amzsvW6B/i3zO73lHm+4bSR0ENU\napT9twk943n5Ufpv9hLLFrv7rwmNqbwzSlXE3TeXKq9I/ruAr6Z2vSrOotCbtxFCR/LebWavzN8w\nsxcQlvHO2wa8sZfHaEiY2URCr+/yTNJ/l1nE3wgN/3J9mCTcpRN4lbuXXEAnPk5vp/tsMu8tltfM\njqD76+IJ4PJeyn8U+JeSte6ft9F9DvK/AJeV+/x7LyEkQyT72XOlu99R6gB3/wqh1z9vMn0LXXmE\n0IngJc6xhdDozasmhHUUk14J8m/u/nS5FXH3nr4fRGQIqXE8hNz9J4S/N28vI3sVoRflG8BaM3tH\njGUr5Y2Z258os2pfJjSk8l5qZrPKPHa4fNN7idd293Yg+8V6vbtvKqP8P6euz4txvAPpV6nr1ewb\nX7kPd28khKe0p3Z/18yWxufrRyRx7Q78Y5n3dSDMMbO6zOUQM3u+mf0L8Bjwuswx17n7/WWWf5WX\nOd1bnEovvejOD919VTnHxsbJN1O7zjSzSUWyZuNaPxdfb735DiEsaTC8LXO7ZINvpDGzycCrUrt2\nEULCyvGxzO2+xB1f5e7lzNf+28zt55ZxzNw+1ENERgg1joeYuz/o7qcCpxF6NkvOwxvNJvQ0Xm9m\n1cUyxJ7H41K71rr7vWXWqYMwzVWhOHruFRkpbioz35rM7T+UeVx2sFufv+QsmGpmB2Qbjuw7WCrb\no1qUu99HiFvOm0loFH+P7oPdPu/uv+trnfvh88DTmcuThB8n/8m+A+buYN/GXCm/7j1LwRl0/2z7\nWR+OBbg1db0KOKFInpNT1/NT//Uq9uL+tI/16ZWZzSWEbeT91Uffsu4n0H1g2i/K/Ucm3tfHUruO\nigP7ylHu+2R15nZPnwnpf50ONLN3llm+iIwQGiE7TNz9NuA2KPxF+3zCrAonEHoRi/1wOZ8w0rnY\nh+2RdB+5fU8fq3Q38I7U7ZXs21MykmS/qHrSmLn9eNFcvR/Xa2hLnB3hHwizKpxAaPAW/TFTxMwy\n8+HuV5vZGYRBPBBeO2l307cQhKHUQphl5N/K7K0DWOfuO/twjlMyt3fFHyTlqszcXkYY1JaW/iH6\npPdtIYq/9iFvuU7K3L5tEM4x2FZmbu/PZ9gR8XoF4XO0t8eh0ctfrTS7eE9PnwnX0z3E5itm9irC\nQMMbfRTMBiQy3qlxPAK4+2OEXo9vAZjZDMLfi5cTppVKe4eZfafI39HZXoyi0wyVkG00jvS/A8td\nZa5zgI6rKpXZzE4mxM8eVSpfCeXGleddTIjDXZrZ3wC8wd2z9R8OXYTHewdh6rXbCCEOfWnoQveQ\nn3Jkp4u7tWiu8nULMYr/0qSfr+y/E70pOgVfP2XDfsoKIxlhhuMzrOzVKt29IxPZVvQzwd3vNbOv\n0b2z4R/iJWdmfyeE1t1KGNBczr+HIjKEFFYxArl7g7tfS+j5+PciWS4rsm9G5na257M32S+Jsnsy\nh0M/BpkN+OA0MzubMPhpfxvG0Mf3Yux9+nSRpPe7e30/6rG/LnZ3y1wmuPtsdz/M3S9w96/sR8MY\nwuwDfTHQ8fJTMrez743+vtcGwuzM7QFdUnmIDMdn2GANVn0X4d+bvZn9FYRY5XcSZp/ZZGZ/MbPX\nlTGmRESGiBrHI5gHnyB8iKb9QzmH9/F0+mDeD3Eg3P/SPaSlHvgkcA5wOOFLf2K64UiRRSv6eN7Z\nhGn/st5kZuP9fV2yl38/9PbeGInvtVEzEK+Ekfi4liV+dn+aEJLzIeAu9v03CsJ38BmEMR+3mNnC\nIaukiPRIYRWjwzXABanbi8ys1t1bUvuyPUXT+3iO7N/6iosrzzvo3mt3PfCWMmYuKHew0D5iD9P3\ngEVFks8kjNwv9o/DeJHune4Eagc4zCT73ujve20gZHvks72wo8GY+wyLU8B9DvicmU0BTgROJbxP\nT6H7d/CpwO/iyoxlTw0pIgNvvPcwjRbFRp1n/zLMxmUe0sdzHNZLeVLcuanru4F/LnNKr/5MDXd5\n5rz30n3Wk38zs1P7Uf5ol56vdwL97KXPig2X9F/+B/eUtwd9fW+WIzuH84pBOMdgG9OfYe7e7O5/\ndvcr3f0MwhLYHyMMUs07GnjrcNRPRBJqHI8OxeLisvF4j9B9/tvs6PXeZKduK3f+2XKNhb95i0l/\ngd/u7nvKPG6/psozs+OBz6Z27SLMjvGPJI9xJfDDGHoxHt2duf2iQTjHA6nrh8ZBtOUqNjVcf91N\n9/fYaPxxlP3M6c9nWI4wYHXEcvft7v4p9p3S8OXDUR8RSahxPDocnrndnF0AI/Zmpb9cDjaz7NRI\nRZnZBEIDq1AcfZ9GqTfZvwnLneJspEv/9VvWAKIYFvGGvp4orpR4A91jat/q7uvc/feEuYbzFhOm\njhqP/pi5fdEgnOOu1PUK4LXlHBTjwc/rNWMfufs24NHUrhPNrD8DRLPS79/Beu/+le5xua/uaV73\nrHhf0/M8P+LuTQNZuUF0A91XTq0bpnqISKTG8RAws/lmNr8fRWT/Zru5h3w/zNzOLgvdk3fRfdnZ\nG919R5nHlis7knygV5wbLuk4yezfuj15M/v3t/c3CQN88q5x91+mbn+U7r2mLzez0bAU+IBy96eA\nP6V2nWRm2dUj++u6zO1/MbNyBgK+leKx4gPhm5nbXxzAGRDS799Bee/Gf13SK0fOovic7sV8MnP7\nfwekUkMgxsOnZ7UoJyxLRAaRGsdDYwVhCejPmtm8XnOnmNlrgUszu7OzV+R9j+5fYq8ws3f0kDdf\n/gns+8Xy5b7UsUxrgfSiDy8chHMMh7+nrq80s9NLZTazEwkDLPvEzP4f3QdlPgh8MJ0nfsm+ge4N\n9s+ZWXrBivHiiszt/zGzs/pSgJktNLOXFktz90fpvjDIYcBVvZR3BGFw1mD5Nt3jrf8BuLrcBnIv\nP+DTcwifEAeXDYbsZ88n42dUj8zsUpIFcQD2EB6LYWFml8YVC8vNfw7dpx8sd6EiERkkahwPnUmE\nKX3Wm9kvzOy1pT5AzWyFmX0T+DHdV+x6gH17iAGIfyO+L7P7GjP7vJl1G/ltZhPM7GLCcsrpL7of\nx7/oB1QM+0gvZ326mX3LzF5kZodmllceTb3K2aWAf2Zmr8hmMrNaM7uc0KM5jbDSYVnM7Ejg6tSu\nZuCCYiPa4xzH6RjGauCGPiylOya4++10nwe6ljATwNfM7NCejjOzGWZ2vpndQJiS7x9LnOYyuv/g\ne6eZXZd9/ZpZhZmdR/jHZyaDNAexu+8l1Dc9RuHdwJ/iIjX7MLMaM3uZmf2U0itiphdSmQL8xsxe\nHT+nskuj9+c+3Ar8ILVrMvAHM/unbM+8mU0zs88BX8kU88H9nE97oHwIWBdfC6/q6b0XP4P/kbD8\ne9qo6fUWGas0ldvQqyKsfvcqADN7ClhHaCzlCF+eRwBLihy7Hjiv1AIY7v4dMzsNeEvcVQF8ALjM\nzO4CNhGmeToBmJM5fBX79lIPpGvovrTvP8VL1i2EuT9Hg+8QZo/IN7hmA78ys2cIP2RaCX9Dn0T4\ngQRhdPqlhLlNSzKzSYR/CmpTuy9x9x5XD3P3n5rZN4BL4q5DgK8DbyrzPo0VHyesIJi/3xWEx/3S\n+Pw8RhjQWEV4TxxKH+I93f3vZvYh4Iup3RcCF5jZ3cCzhIbkSsLMBBBiai9nkOLB3f0mM/sA8F8k\n8/6eCdxpZpuAhwkrFtYS4tKPJpmju9isOHnfAt4PTIy3T4uXYvobyvEuwkIZ+dVBp8fz/6eZ3Uv4\ncbEAODlVn7zr3f3r/Tz/QJhIeC1cCLiZPQE8TTK93ELgWPadru6X7v5/Q1ZLESlKjeOhsZPQ+M02\nRiE0XMqZsuiPwNvKXP3s4njO95J8UdVQusF5O/DKwexxcfcbzOwkQuNgTHD3tthT/GeSBhDAgfGS\n1UwYkLW6zFNcQ/ixlPddd8/GuxZzOeGHSH5Q1hvN7E/uPm4G6cUfkW82s4eA/6D7Qi09PT9ZJefK\ndfer4g+YT5K81yrp/iMwr5PwY7C/y1mXFOu0gdCgTPdaLqT7a7QvZdab2UWERn1tL9n7xd0bY3jS\nzwkN+7zZhIV1evJVQk/5SGOEQdXZgdVZN5B0aojIMFJYxRBw94cJPR0vJPQy3Qd0lXFoK+EL4uXu\nfla5ywLH1ZneR5ja6CaKr8yU9yjhA/m0ofgrMtbrJMIX2V8JvVijegCKu68GjiP8HdrTY90MfB84\n2t1/V065ZvYGug/GXE3xpcOL1amVEKOcHuhzjZktL+f4scTdv0AYyHg1+84HXMzjhB8lJ7t7r/+k\nxOm4TqN72FBajvA+PMXdv19WpfvJ3X9MmN/5C3SPQy5mC2EwX8mGmbvfQBg/cSUhRGQT3efoHTDu\n3kCYgu9CQm93T7oIoUqnuPu7+rGs/EB6JeExupveP9tyhPqf6+6v1+IfIiODuY/V6WdHttjbdFi8\nzCPp4Wkk9Po+Cjw2ECt7xXjj0wij5GcRGmpbgHvKbXBLeeLcwqcR/p6fSHicNwC3xZhQGWZxYNzR\nhH9yZhB+hDYAa4BH3X1ricN7K/tQwo/ShbHcDcC97v5sf+vdjzoZIUzhOcBcQqhHc6zbo8AqH+Ff\nBGa2lPC4zid8Vu4ENhLeV8O+El5PzGwicCTh38EFhMe+gzBw+inggWGOjxaRItQ4FhERERGJFFYh\nIiIiIhKpcSwiIiIiEqlxLCIiIiISqXEsIiIiIhKpcSwiIiIiEqlxLCIiIiISqXEsIiIiIhKpcSwi\nIiIiEqlxLCIiIiISqXEsIiIiIhKpcSwiIiIiEqlxLCIiIiISqXEsIiIiIhKpcSwiIiIiEqlxLCIi\nIiISqXEsIiIiIhKpcSwiIiIiEqlxLCIiIiISqXEsIiIiIhKpcSwiIiIiEqlxLCIiIiISqXEsIiIi\nIhKpcSwiIiIiEqlxLCIiIiISTRjuCkhxZnYRUAf80t3/Nry1ERERERkf1DgeuS4CTgfqATWORURE\nRIaAwipERERERCI1jkVEREREIjWO94OZrTCzb5jZE2a2x8wazOzvZvZlM1uZyldtZuea2f+Y2UNm\ntt3MWs3sGTO7Lp03dcxFZuaEkAqA75qZpy71Q3Q3RURERMYdc/fhrsOoYmaXAVcBlXHXHsKPjNp4\n+xZ3PyPmfRnwf6nD98a8E+PtTuCt7v6DVPkXAF8CZgFVQCPQkirjWXc/YQDvkoiIiIhE6jnuAzM7\nD/gyoWH8U+AId58CTAYOAN4E3J86pBn4LvAiYI67T3b3WuBA4GrCgMhvmtnS/AHufoO7LwDujLve\n4+4LUhc1jEVEREQGiXqOy2RmVcBaYDHwI3e/cADK/DbwVuAKd78yk3YzIbTiYne/tr/nEhEREZHe\nqee4fC8iNIy7gA8OUJn5kItTBqg8EREREekHzXNcvufF7UPuvqHcg8xsFvBO4BzgcGA6Sbxy3gED\nUkMRERER6Rc1jss3P27XlXuAmR0B/Dl1LEATYYCdA9XATELMsoiIiIgMM4VVlM/245jvEhrGDwBn\nA1PdfZq7z4+D7s7rR9kiIiIiMsDUc1y+zXF7YDmZ4wwUJxJilF/RQyjG/CL7RERERGSYqOe4fHfH\n7dFmtqiM/IvjdluJGOV/KHF8Lm7VqywiIiIyRNQ4Lt+fgA2EwXSfLyP/7ridb2bzsolmdhRQajq4\nxrid0ZdKioiIiMj+U+O4TO7eAbw/3nyDmf3YzJbn081soZm9zcy+HHetAtYTen5vMLNDYr4qM3sN\n8AfCIiE9eTRuX2Nm0wfyvoiIiIhIcVoEpI/M7H2EnuP8D4tmQm9yseWjX01YSS+ftwmoIcxSsQ74\nKPAD4Bl3r8ucZznwUMzbCWwFOoD17v6CQbhrIiIiIuOeeo77yN2/CBxLmImiHqgCWoGHgS8Bl6fy\n/gJ4IaGXuCnmfQb4QixjfYnzrAbOAn5HCNFYQBgMuLinY0RERESkf9RzLCIiIiISqedYRERERCRS\n41hEREREJFLjWEREREQkUuNYRERERCRS41hEREREJFLjWEREREQkUuNYRERERCRS41hEREREJFLj\nWEREREQkmjDcFRARGYvM7GlgGmGZeRER6bs6oNHdDxrKk47ZxnF71xYHyOWS5bHNDIBKCx3mbkn+\nDs+FPHjcJonmmW3qPOnr+woHdMZMbvuWWZFavbui+2G9Fb5f0suFV1TEx2PC/EE4k8i4N622tnbW\nihUrZg13RURERqNVq1bR0tIy5Ocds43jrq4uADpSjcHK2FDeuWlbSMt1FdKmL5of8lRWAmCp4/Kt\n1EKDNn2imC/f8PZux8WjY2Oc1PkK9Uxdz+Ubz8Waqt6/FnO+6HT9crl843i/ihSR0upXrFgx6/77\n7x/ueoiIjEorV67kgQceqB/q8yrmWETGPTO72cz2/WUrIiLjjvoMRUQGySMbdlP34d8MdzVERIZF\n/WfPHe4q7Jcx3DiOYQ6VyZ6d23cCcMeNNwEwpWpiIe2AZQcC8JzjjgGgYmJ1Ia0jBgbnYoyu5ZIy\nK6x7mIOl44rj9SceWw3APXfcVUg7ZuWxYXvcscl5YtwzhfOkOrJs4MOCbRDKFBERERnNFFYhIqOK\nmZ1oZjeY2QYzazOzTWZ2k5mdn8pzkZn9zMzWmlmLmTWa2R1m9qZMWXUxnOL0eNtTl5uH9p6JiMhI\nMHZ7jj20+ytzSe9oRbw+edpkAA6Lg/AANm58EoCH7g+jIo8/80WFtFy+F7or9Ox2VSZl5vt2q2Ke\nXFtnIe3u2+8AYNX99wHw5OOrC2nzZ88F4LiVJxX2Wa49br1b2SISmNnbgK8TxrL+f8CTwDzgeOAd\nwI9j1q8DjwG3ApuA2cBLgR+Y2eHu/vGYrwG4ErgIODBez6sfxLsiIiIj1NhtHIvImGJmRwBfAxqB\nU9390Uz64tTNI919TSa9GrgR+LCZfcPdN7h7A3CFmZ0BHOjuV+xHvXqajmJ5X8sSEZHhN4YbxzFu\ntyuJHJk6ZRoAiw4M36FtHbsLaXPmzwDgzoceAODx9ZsLaceuPBGAefNjT3NNVSEtlwu9ybs2bgHg\nwTvvKaQ1bA1Txs2dOhWA5z7niELaQcsOBsAtCYr2GOVi7Dvlm4hwKeEz65PZhjGAu69PXV9TJL3d\nzL4KvBB4EfD9QayriIiMUmO4cSwiY8zz4vbG3jKa2VLgQ4RG8FKgNpNl0UBVyt1X9lCH+4HjBuo8\nIiIyNNQ4FpHRYkbcbiiVycyWAfcCM4HbgJuA3YQ45TrgLUDNoNVSRERGtTHfOHb2XS55Yk34Xmzd\nm6S1NLcCUNkRwhxW3fNQIW3jU88CsKhuCQDVkycV0hobGgHY/Gz4R7e1qbmQduTyEEYxZVIIw5g2\na1ohbc78eQB0puqXy8ysponWRLppiNtFwOoS+d5HGIB3sbtfm04wszcQGsciIiJFjfnGsYiMGXcT\nZqU4h9KN40Pi9mdF0k7v4ZguADOrdPcBC/o/ctF07h+lk+CLiIxX46pxnF/0YlfDLgC69rYV0hp3\nhSnc9mwLPb9zq6cmx3WE3t0Na9cCMGlKktbcuAeA/BC9Tk/6e9v2hjIrpoQFRSqrk4F8+eu5Ej3H\nlZrLTSR/WhFmAAAgAElEQVTt68AlwMfN7Pfu/lg60cwWx0F59XHXGcD/pdJfAvxzD2XviNulwNMD\nWGcRERllxlXjWERGL3d/zMzeAXwDeNDMfkWY53g2oUe5CTiTMN3bxcBPzOxnhBjlI4GzCfMgX1Ck\n+D8B5wE/N7PfAi3AM+7+g8G9VyIiMtKocSwio4a7/4+ZPQJ8gNAz/CpgO/Aw8K2Y52EzOxP4D8LC\nHxOAh4DXEOKWizWOv0VYBOT1wL/EY24B1DgWERlnxm7jOIYkhJVhg3XrngHgztvCynVTqpMB67Nn\nhhXr2uO8xdOmTU+Kqgj7OmMYRnUq/qG1OYRV1E4MM0U1NDQl56vcGI7LdQBQUZPMuXx0XAUPzxX2\ndcXr+cgMS4VcWFnD8yyz3Vd6gKJryJ+MQu5+F/DaXvLcSZjPuJh9Xvgxzvgj8SIiIuNYRe9ZRERE\nRETGhzHbc1wZO4e2btlU2HfHbbcA0LBjJwC5ycnUau0efic83RBWuptp7YW0mbWTAZgWB+K5JR1P\nz24MU652xg7glraOQtrO1lDGtqawEl9XZzIA8LknngzAsQsXFPZ57OWuqIy/WTqTsrwrDKAvNkYv\nP9Awv8JeqR5h73NvtIiIiMj4oZ5jEREREZFozPYcb9m0GYCH77+/sK9h8zYApsQe4OpJyYqyHfFn\nwtJDlwHQ2ZVMdTptaog/3rl5KwCPrH68kLZxayhzT+wlrp6YLBCydEY4rqUjpO3YnPRi33l7iHvO\nLwYCsLs5TCNncbGSmdOTuOd58+cDUFkZFinJperXleseq1y6Q1i9xSIiIiI9Uc+xiIiIiEikxrGI\niIiISDRmwyo2PbsegK7mlsK+gxctBWD6jCkAbI0D8wDqN4SBdR2EEIX5MYwBoL1pLwCH1B0EQPOe\n1kLatu2hjLPPOguABQcsLqRt3RYW3Xrm6acA2N2UTPN22623ArB3797CvrVxBb5n1q0DYPbcuYW0\nl740LEF73PHHAzAvlTZz5kwA2jwM4Mt5atCdKYxCREREpFzqORYRERERicZsz/HE6moApqUGyC0+\nqA6AnY2ht3dZXV0hrWVv6GGuXx96nGdOmlpI68iFh2lSdRjAt2zJgYW0TRvCILvJ8aF89K8PFNK2\n7twFQE1tVTg+NQBwV0zbUL+usG/JvDCt27o19QCs+vuqQtraJ58OeZYsAeDQQw8tpL34JS8G4MRT\nTgKguqYqqXtXZ7yW70FOLwIiIiIiImnqORYRERERicZsz/GChWGKtKbNGwv7Hlv1CAAduTC12rLl\nKwppLzznbAA2x+napk+fVUirnRRilG/44fUAtLYni3lc8MY3A/Dogw8BMKEiifGdPTtMxdbQ2BDK\nnJosOpKrCT26jbEHGWDJ3BDnvGT+QgB2NO4ppLW1h/w7toep4yzXWUh7/JFw7hW/ew4Ab3jzmwpp\nhx8R7mNHXGCkIxWPnCxcLSIiIiKgnmMRERERkQI1jkVEREREojEbVjFpagiFWFi3pLBvzrzZALS0\nhJXo2lMj0qxmMgBHHnciAF1d6eFqYVW6ZctD2MLCxck0akcceggAUyaHwXYL1i8qpP199WMANOzZ\nDUDTrsZCWlscALjswLrCvtraUEZlRThfe2sSvrF1WwinaG0KAwwrc0lQxOTaiQDcHVfdW/PUmkLa\nWTFc5CUvfykA0+fOTu6zVssTERER6UY9xyIyopjZu83sMTNrMTM3s/cOd51ERGT8GLM9x9WxN3XG\nAcliHk3btwPg1gVA4/ZkMNytf/4DAAcfHKZIqztwWSEtlws9rIcuqwOgK5csAvJsXLhj2+Ywpdvq\nvz9cSGvrCD2/c2eH3trZ06YX0qZPCVPFTalNpprbsycMwMv3INdW1xTSairDU5XrCAPxWpuTwXqT\nqsLUbTOnhQF/u3cmi5t8/7vXAvDkmicBeOPFFxXSDlueTAcnMhKY2euBLwEPAlcDbcDdw1opEREZ\nV8Zs41hERqWX5bfuvrFkzlHgkQ27qfvwbwCo/+y5w1wbEREph8IqRGQkOQBgLDSMRURkdBqzPcde\nEdr9VZOTsIVcQxjo1tUZwiqmVid3f+mcEJKQ2xNCLepXJ+ERk6fE+YnjGLjN658tpD18f1gRb0tc\nKa+zK5l/eMb8MHBv+vQQTlFdPbGQNmvGTADaW5MQjaefDAPpdsa5jyem8tfWhOsTLFSidc/eQtqO\n9jBv84TqEF5RUZWskNcVK/3Xe/4KwJLUqoBLli4Oj0MS7SEyLMzsCuATqduFEbHubvH2LcDrgf8A\nzgEWAP/k7tfGYxYCHwPOJTSydwO3AZ9y9/uLnHM6cCXwOmAOUA98E/glsAb4nrtfNKB3VERERrwx\n2zgWkVHl5ri9CDiQ0GjNmkWIP24Gfk74uboFwMwOAm4nNIr/DPwIWAKcB5xrZq9191/nCzKziTHf\ncYT45uuA6cBHgVP7UnEz26fhHS3vSzkiIjIyjNnGcX6is0pLel9rKsJAt917Q0/rlnXrC2lr658B\nYG97BwDVk6cU0ibFnt+aiaEXekqqZ3be/Dnhisce3bakJ7hmYjUAk+MAu4kTJxfS9jaG6eQ2bd5S\n2OeVodwdu0OvcHtHMp3clCnh2EmhSGakesR37wwr8LW3hEF6lVQX0g46NAy6a24L9+uuO+8spJ14\n8vMAOO6EZLo7keHg7jcDN5vZGcCB7n5FkWxHAT8A3urunZm0bxAaxh9z90/ld5rZ14Bbge+Z2YHu\n3hyTPkhoGF8PXOgelo40s08BDwzU/RIRkdFHMcciMlq0Ax/INozNbDHwYmAd8Ll0mrvfSehFngW8\nJpX0FsJv6H/NN4xj/mcJs2SUzd1XFrsAq/tSjoiIjAxjtueYuMCFxQU1ACxOh9bcGnpmt+3YXkh7\ndt06AOYdEBbxmDtnTiGtvfDdGbbtqd7hefPDVHG1k0PPbnNzUyFtT5xura0lLPixY3tDIa0rltmZ\nWsyjIk7dVhFjh7s8WQRkxqzQ+7xs6TwAZk+dWkhb+0SYTm577EFu7ewopE2MZU2NMc4TUlPHVVcl\nPcwio0C9u28tsv/YuL3N3TuKpP8ZeFPM930zmwYcDDzr7vVF8t8+EJUVEZHRST3HIjJabO5hf35I\n6aYe0vP7Z8RtHGHLliJ5S+0XEZFxQI1jERktvIf9u+N2QQ/pCzP58uu4zy+St9R+EREZB8ZwWEWQ\nS32fTpkZOphmHRC+Q7s6Wwpph68IA9dmzwlhCy2tSVhjYwyPaGwKY3nWP5sM5Nu4PlyfOTOELVgM\n5wDYu3dvPC4cv33n7kLa9Bjm0NKW/Av81OrH47lDOMWUqckKefMXTIvHhbCI6VNrC2mz5kzpdr4a\nT9K2bgydbQ17wn2dPjsJF8mvticyyj0Yty8wswlFBuudGbcPALh7o5mtBerMrK5IaMULBqpiRy6a\nzv1a/ENEZFRRz7GIjGruvh74A1AHvDedZmYnARcCu4BfpJK+T/j8+4yZWSr/kmwZIiIyvozZnuN8\nD26657iyNvTEzo49x5vWry2ktXWG3trOuK2qSH43dO4NPb97doXFOTrakoFyVXFatz17Yu9yY2Mh\nLf+d25kLi47MmT+vkNYQ8z38yKrCvnzP8uQ4jdyunXsKaTVx9ri5M8NAvIaG5kIaHgYdVsTfOhMm\nJE9r/nHYsC4uXGLJAMUJSZtAZLS7BLgD+LyZvRi4j2Se4xxwsbs3pfJ/DngVYVGRw83sJkLs8vmE\nqd9eRTIjpIiIjCPqORaRUc/d1wLHE+Y7Phz4AGEVvd8Bp7j7rzL5WwjhFtcQYpUvj7c/DXwmZmtE\nRETGnTHccxx6ayvT++L0aTPnhGWdn3viKYW0x/72NwDWbggzRTXtSr4Xq2Jv68SaEMt7wMK5hTSP\nPdO7doVe361bk2WdPRfS8stH72pMeoLv/et9QBKPDFBZFRYsaYlTv1lX0uvd2Rbuj7eF3t7qqall\nsYlTy1WGfTYhWaTE41R21bVhqrm9calpgHUbNgBwxPGIjAjufkYP+3v9m8PdNwCX9uFcDcC746XA\nzN4Wr67a5yARERnz1HMsIuOSmR1QZN8S4ONAJ/DrfQ4SEZExb8z2HIuI9OJnZlYF3A80EAb0vQyY\nRFg5b8Mw1k1ERIbJmG0cWwx3qPTUmJo4AC3/D+30Ocl0piuffzoA9WvWhO3jTxbS2ppDqEQ+RGFP\nczIYrrEphF90xsmjZs+aXUirnRRCGfa0hAF8j65KyuyMq9jlQy4Ach468hsaQojGhNRgwnxISEdb\nCIuoSU3JNnVGWNtg+45Qrz17kwGDbfHc+bpv3b6jkLb6iVCfsxEZl34AvBl4LWEwXjNwD/AVd//5\ncFZMRESGz5htHIuIlOLuXwO+Ntz1EBGRkWXMN47TS2plR/TkUr3KVROrATh8xXIAFs5NepU3Pxv+\nXd21YzsAEyYlw/xmzg3TwlXEQXueCuPe8OxGALZsC8c3NSczSU2dEqZkq56YLNixZVuYKq6rKwy+\nq62pLqQ17GoAoHF6mOZt/rxkWrhcV67bcdu3b0vu5IRQr/yUc4fWLS0kHXX0UYiIiIhIQgPyRERE\nREQiNY5FRERERKIxH1ZRSjrkIhdjLipjGMKsBUlYRVVcca5yQvgtMa1zWiGtuTnMU7x3b5hreGdc\n5Q5gcxz8Vv/MOgDaU3MMV1WHMIe9e5J5kZuaQthFRRw8N2PmzELa0cuXAbBobtiXDhHp6AiD+zwO\n2suHWQBMmhIGBZqHkIsFCxYW0o48SmEVIiIiImnqORYRERERicZ3z7El/a9m4XfCzp1h4NuTjyaL\nY7XsDj26Xe2hh3bqzKmFtHxv8IZnQ+/wk08+XUjbuj0OotsTepVbO7qS88VtW0uyQt7c6aGXd+GC\nMMjvoKWLC2kH1y0Jx+XCnHE7du4qpO1sCOdpiFPMdabvVxyI5x2hV3nR4iWFtMmTpiAiIiIiCfUc\ni4iIiIhE47vnOL0+SEX4nXDnHXcD8MsbflJIWzh7bsgfY3kPO/KwQtoRhx8CwIypodd3xrSkN3bj\nlhBz3EmIIa6dmsQQV+RCL3Suo7Ww77CDwjRrRx0RppObGOOSATpzoee3qTX0VO+NccYATS0tAKzb\nvBmAmolJHdpjYHVV7SQATjjp5ELaxNrJiIiIiEhCPcciIiIiIpEaxyIyqphZvZnVD3c9RERkbBrX\nYRUVqfnQPBdCJhYcEKY6O/KY5xbSdu/YCUAu5qmZnIQt1MRBbfMWHABARy55SBvbwm+Pphjb0Nra\nkpyvsw2ARUuSAXJz5sdV7+J0cg3NyWC99o4wEK9pTyijaW8SjrFtRxicN7E2rLY3Z+7cQtqEOCBv\n5tzZAMyaPTupQzquRERERETUcywiIiIikjeue44ttQqIxwFvRx55JABLly4tpHV2hl7b6upqACZP\nTAayVXSFgXFrVoep33auWlNIe3rdegCeWLMWgO07txbSDqkL5T/vxOcX9k2qDj3GrXvClGzbUtO1\ntcaBeHtawnbz1u2FtPbYATx3buh5np3uHY7TuuXvw4YNGwppXbkwtVwy7E9ERERkfFPPsYiMOBa8\ny8weNbNWM9tgZl8xs+k95K8xsw+b2cNmttfMGs3sNjM7v0T57zGzx7LlK6ZZRGR8G9c9x+k1mCsq\nw++EirhEdD5Gt3v+cEBbc7IM9IMPPATA+jWhx3jTxqR3uH5dWBhkTf0zoeyKJMbX47l3NjYW9lXP\nnQFALtZl4pSkHbB5eyirsSn0KndZ8rtmblwSuqZqYv7OFNJ27gzx0k17wkImLS3JctWVlfptJCPW\n1cC7gU3AN4EO4JXASUA1UHgTmlk18HvgdGA18FVgEvA64AYzO8bdP5Ip/6vApcDGWH478ArgRMKf\nKR2IiMi4NL4bxyIy4pjZ8wkN4zXAie6+M+7/KPAXYCHwTOqQ9xMaxjcCr3D3zpj/SuBe4F/N7Nfu\nfmfcfyqhYfwEcJK7N8T9HwH+CByQKb+3+t7fQ9LycssQEZGRQ12HIjLSXBy3n8o3jAHcvRX41yL5\n3wo48L58wzjm3wp8Mt7851T+t6TKb0jlb++hfBERGUfGdc+xk4zIy1lnNjG56uFGVVUNAA889FAh\n7dtf+zYAs2vCCnQtqZXrtmzbEsqOU8BVV9QW0p566lkAnl2/vrBv5qwwLdy0qWHbkcz8xt44hdsB\nB4Qp42bNnFRIa20N08I9tS6UuTc1ZdzkyeGcyw4+CIAVRyadWRXj+tmXEey4uL2lSNptQOHNamZT\ngUOADe6+ukj+P8ftsal9+eu3F8l/d7r8crj7ymL7Y4/yccXSRERk5FLPsYiMNPlg+y3ZBHfvAnYU\nybuph7Ly+2fsZ/kiIjLOjOu+Q+89CwBWmA4t9ApPm5YsAnL0sUcBsOWZ0Gs7tWJiIe0Fp4Vp2h56\n+AkA1j29sZDWGqdka2pKBsh5HKXXsidMsbZ5UzJdW2dn6H1uaAq9wpWVyaC71rawIMiCBQsAeMlL\nX1JIO/XUUwBYvvwwABYvOaCQ1hXvT6ookZFgd9zOB9amE8ysEpgNbMjkXdBDWQsz+QDyo2DLKV9E\nRMYZ9RyLyEjzQNyeXiTtVFI/6t29iTBwb5GZHVok/5mZMgEejNsXFMn/PMZ5p4GIyHinxrGIjDTX\nxu1HzWxWfqeZTQQ+UyT/dwgTM34+9vzm888BPp7Kk/f9VPnTU/mrgU/3u/YiIjKqqYdkPxx2yLLC\n9TlvvgCArZs2A1AzMQmrmDV3DgBr1oQ5ih968NFC2saNYSDeho3JgLzGOOdxc3NzPH5eIa2zK4RV\nLF26BIDjjz++kDZ//nwAjj36aAAOjoPvAKpr8k9xOD6XSwYMOl293leRoebud5jZNcBlwCNm9lOS\neY53sW988ReAc2L6Q2b2W8I8x+cB84DPufvtqfJvMbNvAv8PeNTMfhbLfzkh/GIj+TeMiIiMO2oc\ni8hI9B7CPMTvBN5OGCT3C+AjwEPpjO7ebmZnAe8DLiQ0qjtjvve6+4+KlH8pYcGQtwOXZMpfTwjV\n6K+6VatWsXJl0cksRESkF6tWrQKoG+rzWn6aMhGR8S7GLT8BXO/ub+hnWW1AJZnGvMgQy8/fWWyq\nQ5GhtD+vxTqg0d0P6i3jQFLPsYiMO2a2ANjq7rnUvkmEZash9CL31yPQ8zzIIkMhv4KjXocy3EbT\na1GNYxEZj94LvMHMbibEMC8AXgQsJixD/ZPhq5qIiAwnNY5FZDz6A/Bc4MXALEKM8hPAl4GrXfFm\nIiLjlhrHIjLuuPufgD8Ndz1ERGTk0TzHIiIiIiKRGsciIiIiIpGmchMRERERidRzLCIiIiISqXEs\nIiIiIhKpcSwiIiIiEqlxLCIiIiISqXEsIiIiIhKpcSwiIiIiEqlxLCIiIiISqXEsIiIiIhKpcSwi\nUgYzW2xm3zGzjWbWZmb1Zna1mc3sYzmz4nH1sZyNsdzFg1V3GVsG4rVoZjebmZe4TBzM+yCjn5m9\nzsyuMbPbzKwxvm7+dz/LGpDP14EyYThOKiIympjZwcCdwDzgV8Bq4ETgPcDZZnaKu+8oo5zZsZzD\ngD8D1wPLgYuBc83sZHdfOzj3QsaCgXotplzZw/7OflVUxoOPAc8FmoH1hM+yPhuE13S/qXEsItK7\nrxE+uN/t7tfkd5rZF4HLgU8Bl5RRzqcJDeOr3P19qXLeDXwpnufsAay3jD0D9VoEwN2vGOgKyrhx\nOaFR/BRwOvCX/SxnQF/TA8HcfSjPJyIyqpjZMmANUA8c7O65VNpUYBNgwDx331OinMnANiAHLHT3\nplRaRTxHXTyHeo9lHwP1Woz5bwZOd3cbtArLuGFmZxAax9e5+5v6cNyAvaYHkmKORURKe2Hc3pT+\n4AaIDdw7gEnA83op52SgFrgj3TCO5eSAm+LNM/tdYxmrBuq1WGBmF5jZh83sfWZ2jpnVDFx1RXo1\n4K/pgaDGsYhIaYfH7RM9pD8Zt4cNUTkyfg3Ga+h64DPAfwG/BdaZ2ev2r3oifTYiPxfVOBYRKW16\n3O7uIT2/f8YQlSPj10C+hn4FvBxYTPhHYzmhkTwDuMHMzulHPUXKNSI/FzUgT0Skf/Ixm/0dwDFQ\n5cj4VfZryN2vyux6HPiImW0EriEMHr1xYKsn0mfD8rmonmMRkdLyPRfTe0iflsk32OXI+DUUr6Fv\nEaZxOyYOiBIZTCPyc1GNYxGR0h6P255i3g6N255i5ga6HBm/Bv015O6tQH7A6OT9LUekTCPyc1GN\nYxGR0vJzd744TrlWEHvWTgFagLt7KefumO+UbI9cLPfFmfOJZA3Ua7FHZnY4MJPQQN6+v+WIlGnQ\nX9P7Q41jEZES3H0NYZq1OuCdmeQrCb1r30/PwWlmy82s22pR7t4M/CDmvyJTzrti+b/XHMfSk4F6\nLZrZMjNblC3fzOYA3403r3d3rZInA8LMquJr8eD0/v15TQ8FLQIiItKLIsubrgJOIsxJ/ATw/PTy\npmbmANkFFoosH30vsAJ4JbA1lrNmsO+PjF4D8Vo0s4sIscW3EBZg2AksBV5KiP28DzjL3RsG/x7J\naGVmrwJeFW8uAF4CrAVui/u2u/sHYt464GngGXevy5TTp9f0UFDjWESkDGa2BPh3wvLOswkrN/0S\nuNLdd2byFm0cx7RZwCcIXyoLgR2EWQH+zd3XD+Z9kLGhv69FMzsKeD+wEjiAMOipCXgU+DHw3+7e\nPvj3REYzM7uC8FnWk0JDuFTjOKaX/ZoeCmoci4iIiIhEijkWEREREYnUOBYRERERidQ47gMz83ip\nG+66iIiIiMjAU+NYRERERCRS41hEREREJFLjWEREREQkUuNYRERERCRS4zjFzCrM7DIze8jMWsxs\nm5n9n5mdXMaxc83sM2b2dzNrNrM9ZvaImX0qTvpf6tgjzew7Zva0mbWaWYOZ3WFml5hZVZH8dfnB\ngfH288zsp2a2ycy6zOzq/X8URERERMavCcNdgZHCzCYAPyUs4wrQSXh8XgacbWYXlDj2BYQlD/ON\n4HagC3hOvLzZzM5y98eLHPsu4EskP1T2AFOA58fLBWZ2rrvv7eHc5wPXxbrujucVERERkf2gnuPE\nhwgN4xzwQWC6u88ElgF/BL5T7CAzOxD4P0LD+FvAcqAWmAwcCfwOWAL83MwqM8e+ErgGaAE+Asx3\n9ynx+BcDjwNnAFeVqPe3CQ3zg9x9BjAJUM+xiIiIyH7Q8tGAmU0GNhLWl7/S3a/IpNcADwBHxF0H\nuXt9TPtf4I3Al939PUXKrgbuBZ4LnOfuP437K4E1wIHAa9z9F0WOPQj4O1ADLHX3TXF/HWGNcoA7\ngNPcPbd/915ERERE8tRzHLyY0DBuo0gvrbu3AV/I7jezWuC8ePOLxQp293ZCuAbAWamkMwgN4/pi\nDeN47NPA3YSQiTN6qPt/qWEsIiIiMjAUcxwcF7d/c/fdPeS5pci+44HqeP0eM+up/Nq4XZLa9/y4\nPcDMNpeo2/Qix6bdVeJYEREREekDNY6DuXG7sUSeDUX2LUxdn1/GeSYVObZ6P45N21bGsSIiIiJS\nBjWO+ycflrLL3UtO11bi2F+4+2v2twLurtkpRERERAaIYo6DfO/rASXyFEvbErczzWxBH8+ZP/aI\nkrlEREREZMiocRw8ELfHmNm0HvKcXmTffYT5kAH62vubjxU+3Mye08djRURERGQQqHEc/B5oJEyZ\n1tN0bO/P7nf3JuBn8ebHzKzH2GEzm2BmU1K7/gSsi9evys6BnDl2Zq/3QERERET6TY1jIK4+97l4\n8xNm9r44TVt+TuFf0PNsER8GdhIG2N1pZq+O8yITjz/EzN4LrCLMbpE/ZwdwGeCEKd5uMrOTLE55\nERvTK83ss8DaAbuzIiIiItIjLQIS9bB8dDMwI16/gKSXuLAISDz2BOCXJHHJnYSlnKcQeqPzznD3\nblPCmdnFwDdIpoRrJSwhPQMo9Ca7u6WOqSMuApLeLyIiIiL9o57jyN07gdcC7wYeJjRwu4DfAKe7\n+89LHPtXwrLRHwLuBJoIjdsWQlzyfwInZBvG8djvAocTlnx+NJ53OrAD+AvwAaBuIO6jiIiIiJSm\nnmMRERERkUg9xyIiIiIikRrHIiIiIiKRGsciIiIiIpEaxyIiIiIikRrHIiIiIiKRGsciIiIiIpEa\nxyIiIiIikRrHIiIiIiKRGsciIiIiItGE4a6AiMhYZGZPA9OA+mGuiojIaFUHNLr7QUN50jHbOD74\nOQc5QEVFsjz25Cm1AMydOxuA6TOmFNKqq8PWYv4JVanCrBMA9y4AqphYSGpqDNu1axsAWL9hdyGt\noyuUVVkZOugrSOqSv5ZevtvMwrknTIh1qSykVVWEMmZOCXX29tZC2vTJkwCYOjWkVVclfwh0te4F\nINeVC2VWTSqk7d7TAsDNd/7JEJGBNq22tnbWihUrZg13RURERqNVq1bR0tIy5Ocds43jvIqKpKGY\nb3x2dITGbkd7rpDW1RnydXaFRmdN0v4tXK+dVBPy7E0ett27msO2YU9Ii2VDqgFs+e2+dbFuzdJw\nI99errCuQkouF9Ja9nQAMG1SbSFt5sxpAMyZGSq6bcuWQtqOXU0AzJ45P2xnJN/TUyd3ICKDpn7F\nihWz7r///uGuh4jIqLRy5UoeeOCB+qE+r2KORWTEMLM6M3Mzu7bM/BfF/BcNYB3OiGVeMVBliojI\n6KHGsYiIiIhINGbDKro6Q0iCpeIW2tvbAQrxK+6psIIYwlBREcMXWpPfDflY49mzQqzy5Ook5qJh\n55Z4vhALUVVVXUjrjMdVWIwd9iSG2GKIRf58AJb/rRKflVxNkj/XFcIo2ohbS9J27G0DoKkxxDvv\n2tFYSNvVEu5jU1dImzh1aiFtaio0Q2SU+gVwN7BpuCtSzCMbdlP34d8MdzVERqT6z5473FUQKWrM\nNo5FZOxz993A7l4zioiIlGnMNo67urr3BAPkusL1tjiDQ2VFTSGtqir0xOZ7dHMdyXFtraGsLS27\nALQAt5wAACAASURBVFg8L5nlYkLsKe6qiLNHTEh6nCfkYu9wfoBdKoilMtYrPSOFxQF5ufxQvlx6\nJos4Y0ZnqHtbe/LUbdsReo47W8PgwI6OZCBfRyyjYW9Iq9+6sZA2uSa5/yIjjZktBz4LnAbUAA8C\n/+7uN6XyXAR8F7jY3a9N7a+PV48GrgBeAywCPuXuV8Q884FPAy8jTLn2OHAV8Myg3SkRERnxxmzj\nWERGtYOAu4BHgP8GFgIXADea2YXufkMZZVQDfwZmATcBjcDTAGY2G7gTWAbcHi8LgW/EvGUzs56m\no1jel3JERGRkGLON49b20JtqlUkMcHW+Vzh2yHqqK7cyTnScj1FO9zjPnjEn5O8MU795VzLH8Ixp\n4bhN28P5cp1Jr+0ED7HJVfl5jqtS8cWxOzmXS+KeO+OxuXysclfSc1yYki7GLXd0JPcrPwVcLvYu\nT5meTNI8YULoHe5oD2U2NSXna9ub1FVkhDkN+IK7fzC/w8y+Qmgwf8PMbnT3xh6PDhYCjwGnu/ue\nTNpnCA3jq9398iLnEBGRcUqzVYjISLQb+Pf0Dne/D7gOmAG8usxy3p9tGJtZFfBGoIkQclHsHGVz\n95XFLsDqvpQjIiIjgxrHIjISPeDuTUX23xy3x5ZRRivwcJH9y4FJwN/igL6eziEiIuPQmA2rOOSg\nRUD3qdxaW0M4RFVVCDuYmBqQNmXaZAByHkIn0ssVdnSFUIRcZwhbaPUkHGH6pFDWsgU1MU/ykFZa\nLNNCCERHavnorlwoq7k5+W7u7Ar7JtXWdqsnQPOeMBDPY0zIzNTS19OmzQBgxtSQf+7smUkdKkNZ\nu3eF+7N1S/JPdE21fhvJiLWlh/2b43Z6GWVs9fT67In8sb2dQ0RExiG1jkRkJJrfw/4FcVvO9G3F\nGsbpY3s7h4iIjENjtuf4dWc/H4D29qSXd+uW7WFfW+ih3bY76UV1Dz2sU6aEHtlmmgtpe/aEkMWK\nXOiZ7cglvynq5s8C4OxTnwvA1KnJwhpdHvJv3REWH9mzt7OQlvOwb29LUocZMycBMG9uGABoJD3H\nO3bsjHUJvd+TJ08rpNVODMe1d4QyKyuTp7WiIlxvaw3n3nPg7EJadU1SvsgIc5yZTS0SWnFG3D7Y\nj7JXA3uBY8xsepHQijP2PWT/HLloOvdroQMRkVFFPcciMhJNB/4tvcPMjicMpNtNWBlvv3hYGvM6\nYCqZAXmpc4iIyDg1ZnuORWRUuxX4ZzM7CbiDZJ7jCuDtZUzj1puPAC8C3hsbxPl5ji8Afgu8op/l\ni4jIKDVmG8fHHB4G5E2qTcbtdHWGwXk7tjcAcNtdyT+zG7aFMIpcV5iveMG0OYW01ooQatHYGAbF\ntbQl8xw37QohFzNjOMbChVMLaRZX3Zs2OYQ7WJz3GGDO3BAWUTkhV9g3Ic6DPKEqPC3rNyTjgqbU\nhHIPPui4eF+SEI3WllCH9ZtC2Egu9bTOmRPCKGprw7m7ckmYSXt7OyIj1NPAJYQV8i4hrJD3AGGF\nvN/3t3B3325mpxBWyHs5cDxhhbxLgXrUOBYRGbfGbONYREYfd68nv6pN8Mpe8l8LXFtkf10Z59oM\nvLWHZOthv4iIjHFjtnH8/7N35/GRHeW9/z9Pt6TWMlpn38c2tmfAxuBhsdls4MdqkhACISELJgkJ\nSy577vVNQrAhBH5JbkIuCZAbEsx2AyQkkAAOzguwzWYINpvt8W55PPuqXeq17h9P9TnHGkmj0Wik\nUev7fr2Goz5Vp04dqZGrHz1V1dPrkdmWpvbk3PFjHh0uxojppU94XFK2+YhPeNu3z6O1TU1phLW3\n3dtY3eXLtTU3p9+2gcGjAHz1a98DYPuOc5OyJ+68GEh3t7PMf2/LFV9araOzIzk3MT4R2/R+/vjO\nezJlHmHetOkCf5ajh9O+b/bocO/Krd73fLpEXX05uPqKVtmVrcbGJm8aJiIiIrK8aUKeiIiIiEjU\nsJHj+gqnhw6nebvFCY8Gd3X7cmsPP7w3Kevt80jzUy/3jbe6utKIc6Xqkebmek5wU3ZjEd8g5KEH\n9wPwnW/dnpT98LYvArB16xYAtmxdnZStWut96OxN85CHxj0X+ugxj+geOpDmNp97zvkA7NlzyOse\nP5KUrY55xZYrxf6Vk7LBIT8X4uYmhUJ6v1o1zXcWEREREUWORUREREQSGhyLiIiIiEQNm1bRFCei\nrVzVl5yrVjytolz2ZdDaV5yXlLUUfGJcc4u/Hp9IN+YqFn0Jt4FBn7QXSNMWNqzfDMC55/pkuJam\nFUnZf33/JwA81P8IAHfec0dS9uSnXgTAtvPOT86tXefpEbt3e729e9KNu/K5Pd73ki/v+rjMxL96\nCsnoiE/yM0sn3dXTKHLmP+paNU0J6exMvzciIiIiosixiIiIiEiiYSPHXV2+aUa5nEZ5h4c96trS\n6p8JOlakS57Vgp87dvTRS7oBdHf3AtDXtz7WTdsstPhEvuYmDzl396ZtXvEc37DjnEc8uvxfP/hx\nUtb/kLf/5S99KzmXN58EeMdP7wfggfseTsouvNAj05c/bafftzmdTFec8Il7E+MeMW5qSvuwcuU6\nAHK5+mTCfFLWlNdSriIiIiJZihyLiIiIiEQNGzk+OuhLqxHSSGmx5LnGtZpHTHu60g04Wszrdbb5\nEmvnbt2UlHXGraFb2zwiO1FLt26uxChyPl6/ck2ax1uO92uLbW7ZsCYpO3DQN/G45ebvJuceuH83\nAGOj3ubQ8aGk7Hvfug2AvQ977vHKuPQcQHt73Bq66n0oh/QzT2HFfV4WPLp8wXnrk7KLtp8DwAZE\nREREBBQ5FhERERFJaHAsIiIiIhI1bFrF4KAvv1YupefyVnhU2cCxND1iRaenTjQ1+eeFfL4lKTs+\n6jvW1YbTpdXq2tt9Et3E2AgAzc3pt3Q0nhsb8evb29Jl3i640HfNq5SqybmD+30y4OjIUQA6OtI+\nPLLbUy76H/JJeq2FtqSsEL8eHvXUiZFiMSmrxYl7lvdzz3nWJUlZV9dzAXjcCU8lIiIisjwpciwi\nZxUz6zez/sXuh4iILE8NGznesH4bAOViuiHGwYMemR0f9Ylu1Wq6JBtNHlltbfVo7eCR42mZefS1\n0OLfrq44AQ6gJS6HVq15BLgl+x1t83u3NPvEv1xTOgFwcMSj1/fHDUIAdu/1SXqlordVKKRLrdVq\n9Q1MvM185mNNqeR9n5iIz5NLrysVPTq+YdMqADZu3JyU3XtfPwAvQkRERESggQfHIiKL7Y69g2y7\n5suL3Q2ZZ/3vv2qxuyAiZ5DSKkREREREooaNHB88uBeAc7ZdkDnrnwVqVU+TqFbSCXk9cbe8elpF\nrtKclPX2dgNQKPi3q1hMUy4qZZ8819zs19cyE+wO7TsEwPdu9x3v+nen6xbfH1MaHnxgT3KuOBFi\nL/0+zbn0s0tzs/enteD9q2XWWq6nXOSayic8Vz7u3FeK6SWP9Kd977so+70RWThmZsAbgdcD5wFH\ngX8F/mCGa34Z+G3gCUAb8BDwaeDPQgjFKepvB64BngusAQaArwHXhRDumVT3euDVsS9XAa8Fzge+\nF0K4cu5PKiIiS03DDo5F5Kz2AeBNwH7g/wBl4OeApwItQClb2cz+HvgNYA/wL/hA9zLgPcBzzex5\nIYRKpv4LY71m4N+B+4FNwMuAq8zs2SGE26fo118BzwS+DHwFqE5RR0REGljDDo6PHD0CQHdPb3LO\nzCfSta3wSHBrU7pU2vBxnwx39x13AFBoTie1TRz35dpGRjzyuztGhAFGx/y/4cUJ/+/y0NBoUnbw\ngLd574MH/fqJdHJgLe5il7N0SbYYfCYXI9z5alo/xK+LMSpcsjQ6jHlZpTIOwKbNG5Oi5zzvBQBs\n3brV+zCYLkeXyymrRhaemT0NHxg/ADwlhHAsnv8D4BvAeuDhTP2r8YHxvwK/EkIYz5RdC7wLj0L/\nVTzXC/wjMAY8K4RwV6b+44DvAR8FLp2ie5cCTwwhPHQKz3PbNEXbZ9uGiIicPTQ6EpGF9pp4fG99\nYAwQQpgA/ucU9d8MVIDfyA6Mo/fgKRm/kjn360AP8K7swDje407g74Anmtljp7jXn57KwFhERBpP\nw0aOm5s8Irtv74Hk3Mq+DQCsXunHh+9/ICn77i1fB6CnpweAyy+/LClbt24dAGNjvslGc2sS1OL2\nH3qkeffDfu7o0TSnd3jI6xeaOwGwfC0pm4hLrBXL6V9tk08qIUatLY1e15eKC7UYTa6mZd0xJ/oZ\nT38OAFde8bSkbNMmX7ptYsLHFAfSbwfjY4/6y7XIQqlHbG+eouyb+EAYADNrBy4BjgBvscz/JzKK\nwI7M68vj8ZIYWZ6snmy/A7hrUtn3Z+r4VEIIO6c6HyPKU0WnRUTkLNawg2MROWt1x+PByQUhhKqZ\nHc2c6gUMWI2nT8zGynh87UnqrZji3IEpzomIyDKitAoRWWj1xPe1kwvMLE86uM3W/WEIwWb6N8U1\nl5zkmo9P0bcwxTkREVlGGjZyfPFFTwKgNJ7ugldo9gl5B/d5wKo8MpCUXf7kJ3ud9i4ABofTdIf9\nR31ZuFLZ0yJaW7uSsmdd+WwALn2S73h3153pClG33/4jAI4e91SL6kg6Wa/+l+Psn4lD8PZrtfrr\npkxZ/G92rJOrpf8Nv+xJTwDgd177agA62ltOuK5c9mfvaMsnZcUJTcSXRXE7nm5wBfDgpLJnkvm9\nFEIYMbM7gceZWV82R3kGtwK/ENv6yfx0eW4u2tjNbdowQkRkSVHkWEQW2vXx+Adm1lc/ab6czPum\nqP8X+PJu/2BmPZMLzazXzLK5vR/Dl3p7l5k9ZYr6OTO7cu7dFxGRRtawkeO1a7cAUBlPI6y33uzz\nf/b2+6Yc1px+NigFj6gODe/36yppmeFrrOVy/u1auSaNHK/fuB6A1es2AfCM1ekyak2FDr/vd78b\n2x5J24wR45CJAFerHsmtxb8Q1yyN7Da3+rnLL/MxwIUXbEnKnnTpRQCs6fMl5yYmxpKy+h+bmwv+\nPF0dq5KyWk1/QZaFF0L4tpl9EPhvwB1m9s+k6xwfx9c+ztb/BzPbCbwBeMDMvgrsBvqAc4Bn4QPi\n18X6R83s5fjSb7ea2deAO4EasAWfsLcSaD3TzyoiIktPww6OReSs9mbgXnx94t8h3SHv94EfT64c\nQnijmd2AD4D/P3yptmP4IPnPgE9Nqv81M3s88A7gBXiKRQnYB3wd+PwZeSoREVnyGnZwPDLk0dMf\n/ddPk3Pf/Y6v0hQqXnZkeDgpG6v69sw9Xb5pSFuM+gJ0xhzetjaPILcU0oDTSMwjPnjINx0ptLQn\nZWvXeRT5vPPPB2C8lC7ReiDWHx5Lz5XjBh8hRpULhTR6fcWzfGm5V73iJQBsWJ9ubtIa+zc66jnU\nlXKaZ10ql2Jb/ny1Wvojr2S2mRZZSMGT4f86/pts2zTXfAn40incox/43VnWvRq4erZti4hI41LO\nsYiIiIhIpMGxiIiIiEjUsGkV//VdT6H44hf+IzlXrfgyaF3dnjLRFHeuA1i9wtMhOjp9f4KmQiEp\n6+7zZVc7VsSd7kgnyh0+5HsGjI56ikapmPaho80n4j/ucRcDsKI7TdW46ZZbADhw5HDaP/P+BTyt\nYsP6dHLflU97KgBtOa9zaO8jSVlLR0fsn+9p0NKS/bF6XytVT7UoVdK+j46MICIiIiIpRY5FRERE\nRKKGjRzf+p3bARgfTSOlvTECnDP/TFCzdOJaS4tPauvq8mXa6tFigEJbGwDlGHkeGEgn8g0c9824\nSuXx2HZzUjYy6mUdXT6Bb9WqtM1ztm0D4JG9mVWrJnzyXFubR7EvufjipKi1w/s1Uqw9qs6jxOep\nVEvJqaYmf9ZCwaPKtVr6/RioaBMQERERkSxFjkVEREREIg2ORURERESihk2rGBzyNX97e9PdZru7\nfELdwMBRAIqZdYcLrT4Bb3TM1y1u70wn6+WbPeWiVvUd5UItn15XqE/u888ZlUomVaPZ642NxYlv\nOUvKNmz0yXYXZ1Inaub1L7zwQgC2bliXPlCTp2aMxU3tJjLrI2/b6Okaq1d7+kctpH1obn707n4D\nAwPpc1nD/vhFRERE5kSRYxERERGRqGFDh1u3edS1WkkfceXK1QAcOuQR3IMHDyVlbTFy3NHu0deu\nzhVJWa7J2yiWfAJbtVpLylqa69FaYll2kpvvQFdo8+stl/alNe6yt337juScxfv0dvvku+yScccH\nh7wvLR5dXtXTkpQ1t/gkwEqcFDheHE3K8nkvK5e9g2Nj6WS9oM9GIiIiIo+i0ZGIiIiISNSwkePx\nkkdIa9V0V45y8Gjwhi3r/UTmo0FLk0eOO1f4MW/pdZWS5wzng0dfq9U0MpvLef32Nt88JN+Ufkv7\nVno+8vrNHrEeGk7bPHbcl4PraE03GynHdsslz4kukS4LVyrm4v383BhtSdnAUIh1/HW1XEn7XvM2\n6/nMTU3pdZZv2B+/iIiIyJwociwiIiIiEmlwLCIiIiISNezf1ffv9SXL2jvStIX1G/yzwJq1vQAU\nJyaSsmOHhuJXPllvZHgkKWtu8euGR7xOuZJeV4kZDF3dnqqx9dzNSVlHp1/Xt7or3j9NaSiOl2Mf\n0sl95YrvetfW5mkSTZmPLqWin6uUvX5nV5pyQc7PTRS9M+ViLVPmP+JCm6d41JelAygW0zQPERER\nEVHkWEQEM7vJzMJi90NERBZfw0aOV67y6HChNR3/t7X7444XPaq8Zk13UrZh7QYAOrt80l5bWxpx\nzsXI7OGjhwEYHkk34Dh02CfPNbf4smvtHen9mls8kjsxHjck6Um/3U94wlZva2gsOTcWNyCp1Xwy\nYceKtA9m3q7FzzOdXR1JWTFOGBwdHo7Xp/+Nr1S9jYmiR5pDNZ2sVy4rcixyJt2xd5Bt13x5xjr9\n779qgXojIiKzocixiIiIiEjUsJHjJ+zcAsBwjKYC9PR5zu/WrZ4XvHrl2qSsELdnDniUuJTZWnqi\n6NHd9k5fkm1gIN2eee1a3wa6VvOI7NDQYFLWNOH1Ojp8GbXO9jQS3Nbq92ttSX8E5XJXbD+2kUsj\nu/VIcbnsbQ6NHEzKarW4LXWMLhfaMrnNw37v8THvX1NbGlU20u2sRZYKM3sK8HbgGcAq4BjwU+Cj\nIYTPxTpXAz8DPBFYD5RjnQ+HED6VaWsb8FDmdTa14uYQwpVn7klERORs1LCDYxFpPGb2WuDDQBX4\nN+A+YA3wJOANwOdi1Q8DdwG3APuBlcCLgU+a2YUhhHfGegPAdcDVwNb4dV3/GXwUERE5S2lwLCJL\ngpk9FvgQMAQ8M4Rw56TyTZmXF4UQHphU3gLcAFxjZh8JIewNIQwA15rZlcDWEMK1c+jXbdMUbT/V\ntkREZPE17OC40OLpBOsesy05t3mzp1OsWOEpCvUJcAADYz5prj7xrZZZDa1W9RdNeU+LKDSnqdqt\nLT7RrRLXdKtkJrkdOnAcgLDKJ/mt7ku/3aOjXi/kMjcyT3OYiDvljQwMJUVHjx0BoFj0dI98Uz4p\n6+joiH32c/l8WlZojekYcR5eqKT3q4XMvUXOfq/Hf2e9Z/LAGCCEsCfz9QNTlJfM7G+A5wDPBT5x\nBvsqIiJLVMMOjkWk4VwWjzecrKKZbQH+Bz4I3gKZ/dbdxvnqVAhh5zR9uA24dL7uIyIiC6NhB8et\nzR6tXdGeLtc2OuIT6w7s3wfAoUPppLb2dt+A45xzHgNAb09fUmZ4JDaX92/XmtXpRLZyyUOyg3Ei\nXrVaSsoKm3y5ttWre2I7qVLJ+1KqpZP7Rid8SbbBQW+rVqomZZU4Ea8eOa5vTAJQnPAodCVOzCu0\npsu1dbTFSHjwc5YJFoda2r7IEtATj3tnqmRm5wLfB3qBbwI3AoN4nvI24NVAYbrrRURkeWvYwbGI\nNJyBeNwI3D1DvbfhE/BeE0K4PltgZr+MD45FRESmpMGxiCwVt+KrUryImQfHj4nHz09RdsU011QB\nzCwfQpi3P6lctLGb27TJh4jIktKwg+OenpUATEykaQtHj/hudqXKBADFYvrfwDivjtYWT01sqZ8A\nRkc93aEc1wo20rJq1ZdFbWrylIYtW9YnZW1xveGmZi8bGx9JysrHPf3i+LF0XeShuPPe2JjnPjRb\nZmJdcyHe239k5fJEUjYy4W2V43/TR0bStIqxgvcvl4vrOGcm4U1MDCCyhHwYeB3wTjP7agjhrmyh\nmW2Kk/L646krgX/PlL8A+K1p2j4aj1vIrHssIiLLT8MOjkWksYQQ7jKzNwAfAX5oZl/E1zleiUeU\nh4Fn48u9vQb4JzP7PJ6jfBHwQnwd5FdO0fzXgFcA/2JmXwHGgYdDCJ88jS5v27VrFzt3TjlfT0RE\nTmLXrl3gc0UWlIUQTl5LROQsYWaXA+8AnolP0jsC/ATfIe+fY52nAX+M75DXBPwY+HM8b/kbwHXZ\nNY3NLA+8B/glYHO85rR2yDOzIpCP9xY5G9XX4p4pTUlkMV0CVEMICzqJWoNjEZEzoL45yHRLvYks\nNr1H5Wy3WO/R3MmriIiIiIgsDxoci4iIiIhEGhyLiIiIiEQaHIuIiIiIRBoci4iIiIhEWq1CRERE\nRCRS5FhEREREJNLgWEREREQk0uBYRERERCTS4FhEREREJNLgWEREREQk0uBYRERERCTS4FhERERE\nJNLgWEREREQk0uBYRGQWzGyTmf2Dme0zs6KZ9ZvZB8ys9xTb6YvX9cd29sV2N52pvsvyMB/vUTO7\nyczCDP9az+QzSOMys5eb2QfN7JtmNhTfT5+aY1vz8vt4Ok3z0YiISCMzs/OA7wBrgC8CdwNPAd4M\nvNDMnh5CODqLdlbGdi4Avg58BtgOvAa4yswuDyE8eGaeQhrZfL1HM66b5nzltDoqy9kfApcAI8Ae\n/HffKTsD7/UTaHAsInJyH8J/Eb8phPDB+kkz+wvgrcB7gdfNop0/wQfGfxlCeFumnTcBfxXv88J5\n7LcsH/P1HgUghHDtfHdQlr234oPi+4ErgG/MsZ15fa9PxUIIp3O9iEhDM7NzgQeAfuC8EEItU9YJ\n7AcMWBNCGJ2hnQ7gMFAD1ocQhjNluXiPbfEeih7LrM3XezTWvwm4IoRgZ6zDsuyZ2ZX44PjTIYRf\nPYXr5u29PhPlHIuIzOw58Xhj9hcxQBzgfhtoBy47STuXA23At7MD49hODbgxvnz2afdYlpv5eo8m\nzOyVZnaNmb3NzF5kZoX5667InM37e30qGhyLiMzswni8d5ry++LxggVqR2SyM/He+gzwPuB/AV8B\ndpvZy+fWPZF5syC/RzU4FhGZWXc8Dk5TXj/fs0DtiEw2n++tLwI/A2zC/9KxHR8k9wCfNbMXnUY/\nRU7Xgvwe1YQ8EZHTU8/NPN0JHPPVjshks35vhRD+ctKpe4DfN7N9wAfxSaU3zG/3RObNvPweVeRY\nRGRm9UhE9zTlXZPqnel2RCZbiPfWR/Fl3J4QJz6JLIYF+T2qwbGIyMzuicfpctjOj8fpcuDmux2R\nyc74eyuEMAHUJ5J2zLUdkdO0IL9HNTgWEZlZfS3O58cl1xIxgvZ0YBy49STt3BrrPX1y5C22+/xJ\n9xOZrfl6j07LzC4EevEB8pG5tiNyms74ex00OBYRmVEI4QF8mbVtwBsnFV+HR9E+kV1T08y2m9mj\ndn8KIYwAn4z1r53Uzu/G9r+qNY7lVM3Xe9TMzjWzjZPbN7NVwMfiy8+EELRLnpxRZtYc36PnZc/P\n5b0+p/trExARkZlNsV3pLuCp+JrE9wJPy25XamYBYPJGClNsH/19YAfwc8Ch2M4DZ/p5pPHMx3vU\nzK7Gc4tvxjdaOAZsAV6M53j+AHheCGHgzD+RNBozeynw0vhyHfAC4EHgm/HckRDCO2LdbcBDwMMh\nhG2T2jml9/qc+qrBsYjIyZnZZuDd+PbOK/GdmL4AXBdCODap7pSD41jWB7wL/4/EeuAoPvv/j0II\ne87kM0hjO933qJldDLwd2AlswCc3DQN3Ap8D/jaEUDrzTyKNyMyuxX/3TScZCM80OI7ls36vz6mv\nGhyLiIiIiDjlHIuIiIiIRBoci4iIiIhEGhyLiIiIiEQaHJ8mM7vazIKZ3TSHa7fFa5X4LSIiInIW\n0OBYRERERCRqWuwOLHNl0q0QRURERGSRaXC8iEIIe4HtJ60oIiIiIgtCaRUiIiIiIpEGx1MwsxYz\ne7OZfcfMBsysbGYHzezHZvY3Znb5DNf+jJl9I143Yma3mtkvT1N32gl5ZnZ9LLvWzFrN7Dozu9vM\nxs3skJn9o5ldMJ/PLSIiIrLcKa1iEjNrAm4EroinAjCIb0+4Bnh8/Pq7U1z7Tnw7wxq+5WYHvt/3\n/zWztSGED8yhSwXgG8BlQAmYAFYDvwT8rJm9KIRwyxzaFREREZFJFDk+0avwgfEY8GtAewihFx+k\nbgV+F/jxFNddgu8Z/k5gZQihB1gH/HMsf5+Z9c2hP6/HB+SvBlaEELqBJwK3A+3A58ysdw7tioiI\niMgkGhyf6LJ4/EQI4VMhhAmAEEI1hLA7hPA3IYT3TXFdD/CuEMIfhxAG4jUH8QH2YaAVeMkc+tMN\n/HYI4RMhhHJs90fAC4CjwFrgjXNoV0REREQm0eD4REPxuP4Ur5sATkibiIPrr8aXF82hPw8D/3eK\ndo8AfxtfvnwO7YqIiIjIJBocn+iGePw5M/s3M3uZma2cxXV3hRBGpynbG49zSX+4OYQw3Q56N8fj\nRWbWMoe2RURERCRDg+NJQgg3A38EVICfAT4PHDGzXWb252Z2/jSXDs/Q7EQ8Ns+hS3tnUZZnbgNv\nEREREcnQ4HgKIYT3ABcA/xNPiRjCN+t4O3CXmf36InYvyxa7AyIiIiKNRIPjaYQQHgohvD+E8EKg\nD3g2cAu+/N2HzGzNAnVlwwxl9bzoKnB8AfoiIiIi0tA0OJ6FuFLFTfhqE2V8/eInLdDtr5hFbYjA\nRQAAIABJREFU2R0hhNJCdEZERESkkWlwPMlJJraV8Cgt+LrHC2HbVDvsxTWTfzu+/KcF6ouIiIhI\nQ9Pg+ESfMLOPmdkLzKyzftLMtgEfx9crHge+uUD9GQT+zsx+Ne7eh5k9Hs+FXg0cAj60QH0RERER\naWjaPvpErcArgauBYGaDQAu+Gx145Ph34jrDC+HDwJXAJ4GPmlkR6IplY8ArQgjKNxYRERGZB4oc\nn+ga4L8D/wE8iA+M88ADwMeAS0MIn1zA/hTxyYDvxjcEacF33PtM7MstC9gXERERkYZm0+8vIYvJ\nzK4HXg1cF0K4dnF7IyIiIrI8KHIsIiIiIhJpcCwiIiIiEmlwLCIiIiISaXAsIiIiIhJpQp6IiIiI\nSKTIsYiIiIhIpMGxiIiIiEikwbGIiIiISKTBsYiIiIhI1LTYHRARaURm9hDQBfQvcldERJaqbcBQ\nCOGchbxpww6On3rZMwJAc3Nzcq65o9WP7e0AhFzm8c2D6MViEYCmXD4pypkBMDwyAkBvX3dSVgs1\nAEaGvaxaqSRlra3xfrEPpWo1KRudmABgYmgsOVfItwCwZp23XyhYUjYxXgKgXPXVRUJIg/4tsf0q\n3pdiaTwpe87llwHwlje8wZ+rJX3m+MisaO1JbyQi86Wrra2tb8eOHX2L3RERkaVo165djI+Pn7zi\nPGvYwbGILD1mtg14CPh4COHqWdS/GvgY8JoQwvXz1IcrgW8A14UQrj2Npvp37NjRd9ttt81Ht0RE\nlp2dO3dy++239y/0fRt2cJwveBQ2n4kc16O6leEhANo7ViRlq1Z5cKdY9m9JW3taVojfpYMHPTKb\ny8RZx8c90tzbUQCgq2tlUta+og2AUqUMwJFjw0nZ6Ji3VctEmscmPDo8NOAh3dbW9MdTjRHjlpbm\nR7UN0NnZCcDho0cBCJVaUlYqeZvVmket85mIM0Ep5yIiIiJZDTs4FpFl4V+BW4H9i92Rqdyxd5Bt\n13x5sbuxpPW//6rF7oKILDMaHIvIkhVCGAQGF7sfIiLSOBp2cNzc6mkH2e2xK0VPLcjhx83npfNk\nNm9cA8CRQZ9YNxon5gFs2rAKgBUdntKwe/eh9D5N9bSNmKJgmYl8cVJfsextFYvltIMl70OtXEpP\nFT3F4uhhr7dqTdq/Ws2fY2TI+zcxmk7ka857nkcltlWMk/2ycrnco46TvxY525jZduD9wLOAAvBD\n4N0hhBszda5mipxjM+uPXz4euBZ4GbAReG89j9jM1gJ/ArwEX1XiHuAvgYfP2EOJiMhZr2EHxyKy\npJ0DfBe4A/hbYD3wSuAGM3tVCOGzs2ijBfg60AfcCAzhk/0ws5XAd4BzgW/Ff+uBj8S6IiKyTDXs\n4LgWl18LpLPnajG629npS7n19fUkZb29XQCMl+JSboU0AhzimmfVGGmtkUajh4c9gjscI7mHj6TR\n2L7V3v5ExaPEo4PphLymqk+aq0d9AZo749JvTfl4TMtKpThxL06sGx9LlzYZGvJ2K7GOpd3LRM4t\n/m921Tat4CZnrWcBfx5C+L36CTP7a3zA/BEzuyGEMHSSNtYDdwFXhBBGJ5W9Dx8YfyCE8NYp7jFr\nZjbdchTbT6UdERE5O+jv6iJyNhoE3p09EUL4AfBpoAf4+Vm28/bJA2MzawZ+BRjGUy6muoeIiCxT\nDRs5bi54znEls6xZa5dHZmnxiOnewwNJ2fGYy5tv8khrNZOr/JO77gdgJC7bVh5L85HHBz3PN9/s\n0d7WzBJw4zHHuFT2o4W0Ly0tMQe4qSM5V47LunW0eT9bM8vQNcX149oKXlYtp0vA5WJkuiluMtKe\nTz/zVMoxt7n+OJmoci3ztchZ5vYQwvAU528CXg08Efj4SdqYAH4yxfntQDvwzTihb7p7zEoIYedU\n52NE+dLZtiMiImcHRY5F5Gx0cJrzB+Kxe5ryrEMhOyM3Vb/2ZPcQEZFlSINjETkbrZ3m/Lp4nM3y\nbdP9baR+7cnuISIiy1DDplUURz31IYR00lklTmYbrHoqxKFDx5Ky1rgN3sYNvqRblXTZtbE4+a05\n76kT+ZY03aHW5u0X2n2HvKamdCJfU31SYJzIV9+1D6C1Ne7gZ2lbA0ePx/v4uWo1TcOop2jkc17W\nlEt/dBZTR0pxUmB2RFBPv6hPIqxmSnPTjh1EFt2lZtY5RWrFlfH4w9No+25gDHiCmXVPkVpx5YmX\nzM1FG7u5TZtYiIgsKYoci8jZqBv4o+wJM3sSPpFuEN8Zb05CCGV80l0nkybkZe4hIiLLVMNGjptz\nHmnNWfqIoxMeTW5r8XOd3Zml3Lo9DfH4MY8mDwynk/UKrV6/JUaFy5lodC14m9XgUeli3NwDoKPD\nJ8+V4+Yc5VK6OceKdo8c12rpxLrCCo8+t7TFyHE5jV63N3sf6hMMR0fSTUBG659xgtcJZCb+1Sf1\nxSBxWgJMmY4pcla4BfgtM3sq8G3SdY5zwO/MYhm3k/l94LnAW+KAuL7O8SuBrwA/e5rti4jIEqXI\nsYicjR4CngYcB14H/CJwO/DiWW4AMqMQwhHg6fjuetuBtwBPAF6P75InIiLLVMNGjp98qa+/X98g\nA6Cewlvf1GOsmG7dbDHHuBa3em7J5ALn4ppnTTF3uBIy20DHkGw9WpvPbMncmuQYe9nKrnSZt1Dy\ne7fFXGWAWsxDbo/HXC6NUI+OeN5zOS7NtirT1kjMNc7F5wqZXOJqqd5XRYnl7BdC6OfRu9P83Enq\nXw9cP8X5bbO41wHgN6Yp1g45IiLLlCLHIiIiIiKRBsciIiIiIlHDplX0rloJwJp1a5Jz5apPfrv7\n7ocAOH5sKFPmaQfFCa9TyUysa2r23fZqwSfkhUqaVtHT4ekNazasAqCtJf28UZ/81tHqKRrrVq9K\nyh55qN/vOz6enKvWd8iLk+i6O9PUieaYFmHmfahVM0uyVTxFozleV8tMtBsaHI7nvDeWmUxops9G\nIiIiIlkaHYmIiIiIRA0bOd53uL7BRxopHRnxpdTueWA/AN1dXUlZ3HeDlnaPzLa0pdHX9q5OAErF\niVgn/bYdP3YUgFyzT6I797ytSVk5bjZSjEvIZT+JbN7m9SqlYnKuEpduKxRa45k0ep1v9Yl7yVJu\n4+lSbtVkEmCMfhfTJePWtPl1cT8SzTISERERmYEixyIiIiIiUcNGjntivu7xoZHk3KGDvrFHsejx\n07GJNGrb2eHfiuERr1+PFgOUKp4XXItx14lyunFHPb584JBv/Wzp7tFs3bTWr4th21pmNbWmGGnu\n7U7vU4m5wzZF/dYe37CkWvGTHZk+NNWXfIvbY0+MjSZlF1z4mNjmo6PLIiIiInIiRY5FRERERCIN\njkVEREREooZNq3jMhj4ASqvTSXfFuFve8eO+hFtLLs2BKMRUhuEjPpGvPJwusbbl/A0AHI0pF0NH\nBpKy3hZvoxKXZNu7N0136I6pGR0dvhRctZpOsBsf8wl1Y6V08lxTk7eVz+fj6/THU4o73YW4FFtT\nU7qDHzlPlcjHso6O9qRoRWcrIiIiIjI7ihyLiIiIiEQNGzle09cBQEdTOv5f2fl4APIxSnzPvXuT\nsvFyjOoWPQJ8/OjxpOzC83wjkXPX9XrZRLoJSFucBDcWJ9hV2tKo7YEDRwBYFzcIqWWWZqvEjUTy\n1bR/+Yr/OAyPPueb0ih0reZLuNUjx7lcWpYzL8vHGXwhTuwDGBmrR8D9uux0PC3rJiIiIvJoihyL\niIiIiEQNGzned9g359i+Md0+evMqzz/+2WfvBOCfhtMlz6plz/298mkvAjKRZGDjeo8Yjw15nQf3\npznHrTEa3LRlNQBrLkg3Abn7gd3edj2vOEaXXcxVrtaSM7WaR4ObY67x+Fiaj5xvqudHx+hwSPOl\nLdYvxSg2IZOrHCZ//knjxVrUTUREROTRFDkWkbOKmb3JzO4ys3EzC2b2lsXuk4iILB8NGzkWkaXH\nzH4J+Cvgh8AHgCJw66J2SkRElpWGHRw/0O+T4VY0pakMm9f5LnPr13QD8Ksvf05SNjLiy7xV65Pb\nCm1JWVe7t/G1b/wIgB/2P5KU/ebLnwvAyu0bAQiFNN3h/G2e0nHwuC8B17/nUFJ26IjfbyJk0hzi\nTncheMJDLpcG9mtVP2exTqWcndznX1tMmbBMwkSyM55p+p0sCS+pH0MI+xa1J/Pgjr2DbLvmy4vd\njUXR//6rFrsLIiJzorQKETmbbABohIGxiIgsTQ0bOd4bJ811FtLNMjra/HFXmkdW1/SmG2T0dXq9\nB/d6dLdSTJdDy7X6dSNjHgFekdlYZP35mwCotXh0eaKULvPW0e6fPTpaPBr9lIvPScoOHvbNRvYc\nGUnOHY4R5mI5Rq/zad+LsT/NOe9LIJ3IV4pl9SB0LWTKyvE5QsgeAC3lJmcPM7sWeFfmdfJODSFY\nfH0z8EvAHwMvAtYBvxlCuD5esx74Q+AqfJA9CHwTeG8I4bYp7tkNXAe8HFgF9AP/B/gC8ADw8RDC\n1fP6oCIictZr2MGxiCwpN8Xj1cBWfNA6WR+efzwC/AtQAw4CmNk5wLfwQfHXgX8ENgOvAK4ys18I\nIXyp3pCZtcZ6l+L5zZ8GuoE/AJ55Kh03sxMG3tH2U2lHRETODg07OB6PEdz+vYeTc4/Z4jnAQ0O+\ndbNloq/tbR5F3rTBl2Qby0SOQ4zErlvjS7pd+JJnJWWF3hV+v5j3m8+EZuv5y/fevweA8zdvTsrO\n37wWgA1rVyfn7tvtUeu7HvS/KI/GLaYB6sHgarxPLZy4EFs15hpXa1NtHqKF2+TsFUK4CbjJzK4E\ntoYQrp2i2sXAJ4HfCCFUJpV9BB8Y/2EI4b31k2b2IeAW4ONmtjWEUP9Tze/hA+PPAK8K8f8gZvZe\n4Pb5ei4REVl6lHMsIktFCXjH5IGxmW0Cng/sBv40WxZC+A4eRe4DXpYpejUeef6fIfPJMYTwCL5K\nxqyFEHZO9Q+4+1TaERGRs4MGxyKyVPSHEA5Ncf6J8fjNEEJ5ivKvZ+uZWRdwHrA3hNA/Rf1vnW5H\nRURk6WrYtIoLz/Ol1Y4cPZacGy8WAehs9c8EIyPFpCzE6Wlt7b6EW3Nz+rlhKO6kt23LegDGhtOd\n6779g7sA2HGh74y3qiddAi6f92/vYy84H4ADew4kZffd9wAAF+x4THJu/Spfaq6nsx2AR/amE/YP\nHPa/Blcs/sia0sl6teCpFvX4V7WSBtbaWupL2cVCS1NJ9NlIlpgD05zvjsf905TXz/fEY31G7cFp\n6k93XkRElgGNjkRkqZgucX4wHtdNU75+Ur2heFw7Tf3pzouIyDLQsJHjCzavBGBtTyE5l8951NTi\n5hrVavrf2tHR4qPOda7oSMo6O3zS3ejwOAADx9O/3B4f8ut27zsU75FOsBsdjfXixh29mcl340Uv\n+8lP70/O7brnQQCevPNiAJ5x2ROTsof3ejDrzvseAmCklEava3GDj3yu+VHPB1Cr1qPIU40rNElP\nGsIP4/EZZtY0xWS9Z8fj7QAhhCEzexDYZmbbpkiteMZ8deyijd3cps0wRESWFEWORWRJCyHsAf4T\n2Aa8JVtmZk8FXgUcB/41U/QJ/Pff+8zS7SPNbPPkNkREZHlp2MixiCwrrwO+DfyZmT0f+AHpOsc1\n4DUhhOFM/T8FXopvKnKhmd2I5y7/Ir7020vjdSIissw07OB4Iq7vu2JFOkGuKU6ys/pktmr619f6\n+sGlmO4wFNKd6zo7Pa1iw9pVAFRq6d5y+456GmOuOQ9A/750XeX9+4/F+t5mZ1u6I9+6lX0AtHem\n6Rv1ZI1ixddYfmT3w0lZa8En1p23yVMzDg+OJmVHRjzdY3C43uf0DwL1SXqZ4FiG9siTxhBCeNDM\nnoTvkPdi4Eo8t/g/8B3y/mtS/XEzezbwbnyHvLcCDwF/gu+q91LS3GQREVlGGnZwLCJLTwjhymnO\nn/STXAhhL/D6U7jXAPCm+C9hZq+NX+6abVsiItI4GnZwXJ+kVgrpX0bb8h4xLlX8XEchXQ5tYsyj\nr8UYOc7uKDdQ9QBST1cnAGv6OpOyJ1/sy7QNjXidsXIajd5xwblAGrHOWWYCXNXv096eRra3nuM7\n+FnN6+Wqad/zzf6j6u72Vas2b8mn/RvxnfT69/ikvUf2Zlei0qQ7kamY2YYQwr5J5zYD7wQqwJem\nvFBERBpaww6ORURO4vNm1gzcBgzgE/peArTjO+ftXcS+iYjIImnYwfGaXs8TrmUix5WS5xUfOe5R\n3lpXGrXtWeH1jx09CkAul5bVWzg66Ne1ZjYIWdfn1/V1ev1Dg2maYi7n0d2WGOTNN6eR6qYWzz+2\nzJyfSowm54JfkGtO/5JcjBt7hBgRb81Evdc0+54GHe3el+JEurlJpZx+TbyjiADwSeDXgF/AJ+ON\nAN8D/jqE8C+L2TEREVk8DTs4FhGZSQjhQ8CHFrsfIiJydtE6xyIiIiIiUcNGjssVT1GoL9EGMDrq\nu8rt2f0IAEM9XUnZlg2+Y2zfKl9i7eDBQ0lZV5fXs7ynO4wU0zZXt/u3MBfTHSqZiXzlODmvo8Mn\n0Q3HJdcA9vTv9rZjOgfARHEituGvWzNpGENx6baRmDLR2pKWDQ748q3Hhvw4Xi4lZTvXb/AvkmwK\nfR4SERERmY5GSiIiIiIiUcNGjmtVD79mArm0tLYD0Ld6baxUTsqODHlk1mK0dvXa1WnZgQMAtK9Y\nCcBgMY3MHh32srj6GkOj6QS4objE2t79PklvaDiNHD+yz69ryqWfT+qR5olSXE6umkaoK3FZt/oy\ndPnMph65uKlJDGzT3lFIysziSa3oJiIiInJSihyLiIiIiEQNGzl+8IBHa8vlNDoczD8LlEoe+Q3V\ntKxtwqO01Zic29LUk5R19XjOcP/DewDYezhdrq3a7Ns6790/AMDYeBo5rn9di2HbXD6zrXP8XNLU\nlIZ0a3E76+JEOb5OI8eFDl/6LVfwSHA+86PLxbXicjmPKo+XJpKy8bF6f+qR5mwIWcu6iYiIiGQp\nciwiIiIiEmlwLCIiIiISNWxaxdCwT4bLN+WTcy1xZ7vuzg4AmnJpWkEhpiYU2nxy20Bm2bWeTk9p\n2LzZl0U7eGg4KSvFdddK5t/KSi7d8S7vl9Gc9CFNacjHiXjVTOpEiDvjWai/TvvX1OST7CbGPGVi\n//6DSdnqtT5RsKngN5wopRMGx8bKJ7SV3k+z9ERERESyFDkWEREREYkaNnK841yP8uZyaeS4UqlH\nVOvLvKVR3npctRqXTBurpRHd2ohPautZ0QnAxTsuTMq+d+f9AAwXPdJcKaUT8srjHrVtbfeIbqGt\nLW0Tb//h+3en9YuV2K/JvYKxuPlHIU4ALE6k95mIy8fVxv3CsbgkHMDouN/HQv1zUBotNtNnIxEA\nM7sJuCJM9ScWERFZVjQ6EhERERGJGjZybHgEuFpJo6i1GCmu1fyYTbmt5wCHWj2qnH5uGI+bcgzs\n2wvAuWvXJWUr1/h205VHDgNQmkiXhxuJ2zpXin5u8NhoUtYel2ZbtSrdbKRU88h2fQOTgYPHk7L+\nu+4DYM3qNQA0ZT7WlCe83RoevS60tSZlhdZ6xRglDw37Ixc569yxd5Bt13x5sbuxIPrff9Vid0FE\nZF4ociwiS4qZPcXMPmtme82saGb7zexGM/vFTJ2rzezzZvagmY2b2ZCZfdvMfnVSW9vMLABXxNch\n8++mhX0yERE5GyiMKCJLhpm9FvgwUAX+DbgPWAM8CXgD8LlY9cPAXcAtwH5gJfBi4JNmdmEI4Z2x\n3gBwHXA1sDV+Xdd/Bh9FRETOUg07OG5u8aXPSqU0zcHiJLtc/sSl1UJMZajvXBeyZTWfozMw7GkP\nPzz2UFI2WvayYn3pt0yuRn3JuFLcpe7AvsNJWWd3FwCbz9+cnOtb0QvAirZ2AO4p35eUHTpwAIBj\nx48BsLI33cGvIz5rW1yGLt/enPZhRfwR15eYe9R0I809kqXDzB4LfAgYAp4ZQrhzUvmmzMuLQggP\nTCpvAW4ArjGzj4QQ9oYQBoBrzexKYGsI4do59Ou2aYq2n2pbIiKy+JRWISJLxevxD/TvmTwwBggh\n7Ml8/cAU5SXgb2Ibzz2D/RQRkSWsYSPHxwYGAahkJuRV40S8QsEjrWZp5LRez0I9gpyqlD36PFby\niPPDDyT/DWY0TrYbG/RJcS3N6be0vdsnxnXGCXIHDhxJyibGPdLcuSJd3m3DOo8cV+OSbhs2pJP1\niEu/VYoevW7JLFFnsbfVokeog6XL0JXH0yXfvK5lvhZZUi6LxxtOVtHMtgD/Ax8EbwHaJlXZOF+d\nCiHsnKYPtwGXztd9RERkYTTs4FhEGk49l2jvTJXM7Fzg+0Av8E3gRmAQ/4S5DXg1UDhjvRQRkSWt\nYQfH+bxnjJilj9gcI8X1qHB26+ZS3HK5vrnG6OhYUjYx4VHeXGEFABu2pamNh494DvBD9z8CwNBA\nGqld2bcFgPUxAjxeTMuGR7z94shQci5X9Fzje396FwA1S6PDqzpiDnXOe1+upH0fG/eodbW+bXRm\ny+zhwfpzxAyaTE50QNtHy5IyEI8bgbtnqPc2fALea0II12cLzOyX8cGxiIjIlJRzLCJLxa3x+KKT\n1HtMPH5+irIrprmmCmCW+UQqIiLLUsNGjkWk4XwYeB3wTjP7agjhrmyhmW2Kk/L646krgX/PlL8A\n+K1p2j4aj1uAh6apc8ou2tjNbdocQ0RkSWnYwXE9JJ7Lp8HxnMVl2mI2QbC0bHR4BIC9ew4CMDY2\nkZTl49Jvazd0A1COO+0BdPf4uSc+6fEAjBxLJ91R9jb27vb/1rYXMkusdfh1hXya2nBo3z4AOuqT\n9DJZD7Wyp1E0xeXaAi1JWWt7S3wu75fl0uc6f7sH0dpa4yTEfPojz+f0hwNZOkIId5nZG4CPAD80\nsy/i6xyvxNc5HgaejS/39hrgn8zs83iO8kXAC/F1kF85RfNfA14B/IuZfQUYBx4OIXzyzD6ViIic\nbRp2cCwijSeE8HdmdgfwDjwy/FLgCPAT4KOxzk/M7NnAH+MbfzQBPwZehuctTzU4/ii+CcgvAf89\nXnMzcDqD4227du1i584pF7MQEZGT2LVrF/hE6gVlIWhSlojIfDOzIpDHB+YiZ6P6RjUzTXAVWUyX\nANUQwoKuMKTIsYjImXEHTL8Osshiq+/uqPeonK1m2IH0jFLSqYiIiIhIpMGxiIiIiEikwbGIiIiI\nSKTBsYiIiIhIpMGxiIiIiEikpdxERERERCJFjkVEREREIg2ORUREREQiDY5FRERERCINjkVERERE\nIg2ORUREREQiDY5FRERERCINjkVEREREIg2ORUREREQiDY5FRGbBzDaZ2T+Y2T4zK5pZv5l9wMx6\nT7Gdvnhdf2xnX2x305nquywP8/EeNbObzCzM8K/1TD6DNC4ze7mZfdDMvmlmQ/H99Kk5tjUvv4+n\n0zQfjYiINDIzOw/4DrAG+CJwN/AU4M3AC83s6SGEo7NoZ2Vs5wLg68BngO3Aa4CrzOzyEMKDZ+Yp\npJHN13s047ppzldOq6OynP0hcAkwAuzBf/edsjPwXj+BBsciIif3IfwX8ZtCCB+snzSzvwDeCrwX\neN0s2vkTfGD8lyGEt2XaeRPwV/E+L5zHfsvyMV/vUQBCCNfOdwdl2XsrPii+H7gC+MYc25nX9/pU\nLIRwOteLiDQ0MzsXeADoB84LIdQyZZ3AfsCANSGE0Rna6QAOAzVgfQhhOFOWi/fYFu+h6LHM2ny9\nR2P9m4ArQgh2xjosy56ZXYkPjj8dQvjVU7hu3t7rM1HOsYjIzJ4TjzdmfxEDxAHut4F24LKTtHM5\n0AZ8Ozswju3UgBvjy2efdo9luZmv92jCzF5pZteY2dvM7EVmVpi/7orM2by/16eiwbGIyMwujMd7\npym/Lx4vWKB2RCY7E++tzwDvA/4X8BVgt5m9fG7dE5k3C/J7VINjEZGZdcfj4DTl9fM9C9SOyGTz\n+d76IvAzwCb8Lx3b8UFyD/BZM3vRafRT5HQtyO9RTcgTETk99dzM053AMV/tiEw26/dWCOEvJ526\nB/h9M9sHfBCfVHrD/HZPZN7My+9RRY5FRGZWj0R0T1PeNanemW5HZLKFeG99FF/G7Qlx4pPIYliQ\n36MaHIuIzOyeeJwuh+38eJwuB26+2xGZ7Iy/t0IIE0B9ImnHXNsROU0L8ntUg2MRkZnV1+J8flxy\nLREjaE8HxoFbT9LOrbHe0ydH3mK7z590P5HZmq/36LTM7EKgFx8gH5lrOyKn6Yy/10GDYxGRGYUQ\nHsCXWdsGvHFS8XV4FO0T2TU1zWy7mT1q96cQwgjwyVj/2knt/G5s/6ta41hO1Xy9R83sXDPbOLl9\nM1sFfCy+/EwIQbvkyRllZs3xPXpe9vxc3utzur82ARERmdkU25XuAp6Kr0l8L/C07HalZhYAJm+k\nMMX20d8HdgA/BxyK7Txwpp9HGs98vEfN7Go8t/hmfKOFY8AW4MV4jucPgOeFEAbO/BNJozGzlwIv\njS/XAS8AHgS+Gc8dCSG8I9bdBjwEPBxC2DapnVN6r8+prxoci4icnJltBt6Nb++8Et+J6QvAdSGE\nY5PqTjk4jmV9wLvw/0isB47is///KISw50w+gzS2032PmtnFwNuBncAGfHLTMHAn8Dngb0MIpTP/\nJNKIzOxa/HffdJKB8EyD41g+6/f6nPqqwbGIiIiIiFPOsYiIiIhIpMGxiIiIiEikwfESZGbbzCzU\nc8ZEREREZH4s6+2j48zcbcAXQgg/WtzeiIiIiMhiW9aDY+Bq4AqgH9DgWERERGSZU1qFiIiIiEik\nwbGIiIiISLQsB8dmdnWczHZFPPWx+gS3+K8/W8/Mboqvf8XMbjazo/H8S+P56+Pra2e4502xztXT\nlDeb2W+b2dfM7LCZFc3sYTO7MZ7vOIXnu8TMDsb7fcrMlnv6jIiIiMisLNdB0zhwEOhETgWGAAAg\nAElEQVQDmoGheK7u8OQLzOx/A/8NqAGD8Tgv4l72XwKeEE/VYp8241t3Pg/fEvGmWbT1NODLQA/w\nYeCNQTu9iIiIiMzKsowchxA+G0JYh+/NDfDmEMK6zL8nT7pkJ/C7+LaHK0MIfUBv5vo5M7MC8G/4\nwPgI8GqgK4TQC3QATwY+wKMH79O19XzgP/GB8f8fQniDBsYiIiIis7dcI8enagXwvhDCu+snQghD\neHT3dP0mcClQBJ4bQvhJ5h7jwA/ivxmZ2cuAfwRagN8PIbxvHvomIiIisqxocDw7VeAvzlDbvx6P\nH8sOjE+Fmb0G+Dv8LwFvDCF8aL46JyIiIrKcLMu0ijm4P4RwZL4bNbNmPGUD4CtzbOPNwN8DAfh1\nDYxFRERE5k6R49k5YYLePOkj/RnsnmMbH4jHd4cQPnX6XRIRERFZvhQ5np3qGWrX5qGNz8TjO8zs\nKfPQnoiIiMiypcHx/KjEY+sMdbqnOHc0c+3WOd7714DPA13AV83s0jm2IyIiIrLsLffBcX2t4tON\n4A7E46apCuMGHjsmnw8hlIHb4ssXz+XGIYQK8MvAv+NLuN1oZo+fS1siIiIiy91yHxzXl2LrOc12\nfhqPzzezqaLHbwUK01z7iXi8eq6D2jjIfjlwA7AS+E8zO2EwLiIiIiIzW+6D4zvj8WVmNlXaw2z9\nO75Jx2rgE2a2BsDMus3sD4Br8V31pvL3wI/wwfPXzOzXzKw9Xt9mZk8xs78zs6fO1IEQQgl4GfA1\nYE1s6/zTeCYRERGRZWe5D44/CZSAZwBHzGyvmfWb2bdOpZEQwjHgmvjyFcBBMzsOHAP+GHg3PgCe\n6toi8LPAHcAqPJI8ZGbHgFHge8BvAW2z6MdEbOtmYD3wdTM791SeRURERGQ5W9aD4xDC3cDzgP/A\nI7vr8IlxU+YOn6St/w28ErgVGMO/t98Gfj67s9401z4CPAl4E/AtYBhox5d3+yrwWuD7s+zHGPCS\neO9N+AB5y6k+j4iIiMhyZCGExe6DiIiIiMhZYVlHjkVEREREsjQ4FhERERGJNDgWEREREYk0OBYR\nERERiTQ4FhERERGJNDgWEREREYk0OBYRERERiTQ4FhERERGJNDgWEREREYk0OBYRERERiZoWuwMi\nIo3IzB4CuoD+Re6KiMhStQ0YCiGcs5A3bdjB8Z//w+sDwMDxweRccbwEQGtTKwChGpKyegi9rbUD\ngIlSNSkbHfPrMP92ff/WnyZlQ4PjAFzxjGcC8KzLn5mU/eDOh7zORA2AfC6flLXk/Y7BLDmXx++z\nYW0fAD1dhfSBJg75YeQoAD+6d09S1LtyMwCXPf2p3s20Sb777VsA2H/Y608UR5KySnz+f/741zJX\niMg86Wpra+vbsWNH32J3RERkKdq1axfj4+MLft+GHRyXxscAqFYryTnL+xiwVPVBaAjpADgXy6zm\nA1ny6Xixuc3byMfv1tOf8di0TWsBoFguArD74L1JWW9PJwCDe3yAHprSLJYq3mah0Jp2OvjX+w8O\nA3DgwNGkqLvDB7LlCX9da1qRlB0b8Pp33bkLgJ6+7qRsYMQvmCj585Sr6QCdWvrhQORsYGbbgIeA\nj4cQrp5F/auBjwGvCSFcP099uBL4BnBdCOHa02iqf8eOHX233XbbfHRLRGTZ2blzJ7fffnv/Qt9X\nOcciIiIiIlHDRo5FZFn4V+BWYP9id2Qqd+wdZNs1X17sbojIMtX//qsWuwtLUsMOjg8f9hzdgYGh\n5JzhKQW1mGtbaG1Jyrp6PQWiFhN287k0qB7Tg8k3expGd1fbCfcr1+J1rQPJue5WT284dNTLJkqZ\nHOeYf1ytpqkdTTFvI9fcFPuS5hwfHPLUiba21QCsO3drUjZ0qB+AvXv9eOBImqrR3NYFwJq2dgAO\nH+5PymqVsROeQ2QpCSEMAoMnrSgiIjJLSqsQkbOSmW03sy+Y2TEzGzWzb5nZ8yfVudrMQsw9zp7v\nj/+6zOwv4tdlM7s2U2etmf29mR00s3Ez+5GZvXphnk5ERM5WDRs5PnrMJ7M1kU5Ay8XoroWmE8pC\nxSO41XyM6IZMlLfJzxUnfNJdOR6zZYVCnJhXTINYpfEHAWgpeAS5VE6/3VZfUiK7TkTOX9TwvtQq\n6WTC5ia/thwXzmgNaVs9vR4VHhvyKPnhzAodj338JQD09awC4K4702ceG92LyFnqHOC7wB3A3wLr\ngVcCN5jZq0IIn51FGy3A14E+4EZgCJ/sh5mtBL4DnAt8K/5bD3wk1p01M5tuxt32U2lHRETODg07\nOBaRJe1ZwJ+HEH6vfsLM/hofMH/EzG4IIQxNe7VbD9wFXBFCGJ1U9j58YPyBEMJbp7iHiIgsUw07\nOF7V2wtAaSyN8lLzLJLWgq9l3NzWnBTlCl5WiTnAuUzGST2qbLEsX0ivq4VYr+KR42DpdeWyr81X\nCZ6jXK6kYeJcXDKulk9/BKHi11ZjbnNzZhm6loLXG4vPM3T8WFLW2eH129u8zsp8T6bMo8pbNm7y\ndmpPTsruvmfh1w4UmaVB4N3ZEyGEH5jZp4FXAz8PfHwW7bx98sDYzJqBXwGGgWtnuMeshBB2TnU+\nRpQvnW07IiJydlDOsYicjW4PIQxPcf6meHziLNqYAH4yxfntQDvw/9q79yBJr/K+49+nL9M998ve\ntau9S1rZkhZJNldjpDhgjJxAGWI7tpMAFcpysMUlpAoDMcK2jAsnRhQOlQsBBexAUjHEFQOBlEFG\nFpaMVjJC0gqkXe39vjuXnVv3TPfJH8/p9zSjmb3O7M70/D5VW73znrff97y7XbNnnn2e5/x9LOib\n6x4iIrIMaXEsIovR8TmOH4uvvXOMNzsRQphtp5vGe893DxERWYZaNq2iHAvkLHVKw+LPAoU2fy2W\n0+PXcp7mkItFcdTqL7pmX5+3RauRxkZHY9pGPu5gV53Kxuo1v9bElKdHVKyzaS5+7xDStdqnYqpF\n0edXaWonVwixkM4axYEp5aIj7y3fCvF+awfSc7VzxOcw6PPsb0/t4Vb0dL/oGUUWiTVzHF8bXy+k\nfdtcW0A23nu+e4iIyDLUsotjEVnSbjOz7llSK+6Ir09cxrWfBcaBl5hZ7yypFXe8+C2X5qb1vexS\nE34RkSWldRfHdQ8adXV2ZYdysVgu5OOGHW1Njx+L3wqxlVu1Xk3vi7uAWCEWzFVSOLqt4BHqUlvc\nUKSWor2Do/77tpzPodZUYNcTr1klHZuO08lX4yYgzWONSHMsKjRLLdmGx73e6JoeL75ry6cixBPH\nfuDnHPOxztJAmruSamTx6gV+B2juVvETeCHdML4z3iUJIUzFort34AV5zd0qGvcQEZFlqnUXxyKy\nlH0b+Jdm9jLgYVKf4xzw6xfQxu18PgD8DPDuuCBu9Dn+JeCrwD++zOuLiMgSpdihiCxGLwCvBAaB\nu4FfBB4H3nCBG4CcUwjhFPAq4LN494p3Ay8BfgP4+OVeX0RElq6WjRwXC96LuLO9IzvWKJarmadc\nhKlUDDc96WPFtvi+Ynu6WGPnuphNYU0/U8QMDaYmYg9kS2kca1Ztj6/eY9iGB7Oxteb3qRw5kh17\ndNqDYcfjWKGS5lcPcde8NInMyISnZXbmYo/mUvprrUz5XDs7vZjQcmksN2shv8jVE0LYx4/uG/nG\n85z/APDALMc3X8C9jgFvn2PY5jguIiItTpFjEREREZGoZSPHbcUYfc2nwrWQj+3a6h4UspB+Nmjv\n8DZrZj5WnZ7MxmoxwjoVd8oLliKuHSVvh7ZmYAsA27ekDbG2b78FgJ4VqwAYOpV2tdvzhBfbrz5x\nMDs2NuHzO9uWj/NN88vFVnO1ms/Lmv7qchP++4mzvuNdRy4VDJZiW7dau78vX06t3KYGm/rciYiI\niIgixyIiIiIiDS0bOW60X6s3bdiRK3hUOEw3vk5R5VwMBtcq/ps8bdlYI1e5rezR5a3bd2Rj123d\n6a9bbgagu2tlNjY56dHaJ575PgB/+8h3srGDTz4FwOunUtu1Rg5wacDvPZ1PcxidHPdjMTd6ZTHl\nNtdjxLlc8mj56v6eNPcOP//M6Al/vlyKiE/VlVYpIiIi0kyRYxERERGRSItjEREREZGoddMqYsuy\n4ZG0V0CjDVpboQzAyGAqkCOmVZTpB1L7NYAdN1znrzd6gd2mjduzsXLZd5wbjikUj/592tX2+7v8\n93ufexaAk/v3ZGPVOK/HTqf2biGmRYzv9LSI0NWbjU1O+o59ne1+rFRIKRG1Nn+OfEynGA8T6Zmn\nPYfEpv0Bh86cysa6ymm3PBERERFR5FhEREREJNOykeOpynR8Te3KOto9Yjw+6kVwU7G4DSAXNw3Z\nsMWjxHe8+q5sbOPG6/39nR61nRhPkdldT34PgO/u+i4Azz/1bDY2csqL4GxsFIB8SMV3tHsx4LPD\nQ9mhYvxZZXTffgC6tm7OxkKI7d1iq7nxfIoAD2zy13KsKpyuVbMxi8WE9Qn/86hMpbl3FNJ5IiIi\nIqLIsYiIiIhIpmUjx0OnPae3UEiPODnhUdRQ8+hrZyFtLd3f57nGP3nrKwDYcUPazGN8wiOsZ+Mm\nG1/5+teyscf+9lsAnD54AICx4RQdLnd7pLqnx9uuDdfTzyIvnDwGgIWUO9wWE58njxwFYE3fimxs\noN/nNzxyHIDR/P5s7Jab/bxS8A0+VvWmdnJTsTXdweohAPL11B7u8PNHEREREZFEkWMRERERkUiL\nYxERERGRqGXTKvI5L2Br7G4HQCxm6yh3A1CqpZSGndfdBMDKmJJgubR7XnunpytMVrxd23Pfeywb\nO/T4o0Da3a7SnnanG5uK7dNGfQe7fKkzGzt71sdOjKUd66brPtf1HT6HfCHtglerxeeoeeu3gf5S\nNlad8rFqOOsHxtPYji0v9/tN+hyqZ8eysY5cStsQWSzM7B7gbmALUAbeE0K4/+rOSkRElouWXRyL\nyNJjZr8MfAJ4ArgfqACPXNVJiYjIstKyi+Nim0dPK5PT2bGOdi/AGxr2aG3B0iYb39vj7dbWXBcz\nTVJQmVzjixgdnhhOG3dY2YvuKkV/34mhNFavePS5O++R4CMHns/GTp04DcBYNRXwrVi1GoCN27yd\nXG9XezZ2/PgzAAx0eVHg9ZtvzMb2HfHivIE1GwG48eZXprlPNTY88YhxZ2xnB7CiaZMRkUXi5xuv\nIYQjV3UmIiKyLLXs4lhElqRrAFplYfzU4WE2v/8rC3qPfX941/lPEhGRC6aCPBG56szsXjMLwJ3x\n69D41fT1g2a21sw+bWaHzaxmZm9tusY6M/sPZrbPzKpmdtLMvmRmt89xz14zu9/MDpnZpJk9a2bv\nNbOt8X4PXIFHFxGRRaZlI8d79/oOch2dfdmxrt5rAGjPNQrs0uO/cMzTL4bH0o56M4W6p1VQTqkJ\nhyqxGG7Kr9XemYrcQuxbfPSw9ybeu2dfNtbR5UWBN+z4sezYxvXXAtDb40V9p48/l42VzdMitmxe\nA0BcMwDQV/b0iB0bPB2jenI0G3v421/335g/1+ru7mysM6efjWTReDC+vhXYBHxklnMG8PzjUeBL\nQB04DmBmW4C/wSPP3wS+AFwL/BPgLjN7cwjhLxsXMrNyPO82PL/5z4Be4IPAq+f1yUREZElp2cWx\niCwdIYQHgQfN7A5gUwjh3llOuxn4PPD2EML0jLH/iC+MPxRCuK9x0Mw+BXwb+G9mtimE0PjJ8d/g\nC+MvAr8SQmhEqO8DHr+YuZvZrjmGdlzMdUREZHFo2cVx3by4rb17Y3ZspOoFeRXf8I6pSmrzVszH\ndmu5pkq8jB9rRIIPnDmdjQxW/X0DK7zobnR4Ihs7uP8FPzbmu/X19Q5kY1u2bQNgw/rN2bFy3Lzu\n+NG9AOQqw9nYTTv9vP4ubwc3Nl5N7yv7c5085rvuTQwfyMZW9fhYW6cX93WVUpFfuT21fBNZAqrA\n+2YujM1sA/A64ADwseaxEMJ3zOwLwK8BvwB8Lg79Czzy/NuNhXE8/6CZ3Q/8/oI9hYiILGotuzgW\nkZazL4RwYpbjt8bXh0IIU7OMfxNfHN8KfM7MeoBtwMEQwr5Zzv+bi5lUCGGunOZdeHRaRESWkJZd\nHPcPbAFgdLyeHZuue1TXCh4JbsunvN1Gt7andnvLtMbGHwCrVnpUuFbza1lbyjnOFfy8w0c8ajs8\nmPJ9p6c8z3fDhk0AbN++Oc2vz3Oha9UUBDt2cB8AxZy3mnv5rel/ZTdv8KjzyKRHoQ8cOpqNjY37\n+es3+P16O/uzsYEef1+h8YBtaXOTUGpDZAk5NsfxRk/Co3OMN443ChAaO/Ucn+P8uY6LiMgyoIos\nEVkqwhzHG/lHa+cYXzfjvJH4umaO8+c6LiIiy4AWxyKy1D0RX3/KzGb737A74+vjACGEEWAvsN7M\nNs9y/k/N9wRFRGTpaNm0itExTzWwpvq6Qt5TCizEArtaSrmwvKdH7HnhEACnh0eysY4OL2KbnPRr\nPvPU7mxs//Pebq1Q8lSLtlgcB3BtTKfYuGE9AJ3lVAw3OjwEwNDJtNfB6gEfv/lGb++2biDtYNcb\nW78NDcdiwEoKooUJ//3zz3gh36aNm7Kx4gpPtehs92uP1VMax6qzMwv+RZaeEMIhM/t/wGuBdwP/\nrjFmZi8DfgUYBL7c9LbPAfcCHzWz5m4V18ZrzIub1veyS5t0iIgsKS27OBaRZeVu4GHgj8zsdcBj\npD7HdeBtIYSzTed/DHgT8MvADWb2DTx3+Rfx1m9viu8TEZFlpmUXxxY3uGjuzFYoeOS4UPTHrtfT\nv30hZpiMjvlmGydOpdqf4ZFBAE6e8Lqe48dTtLexGcdAn9f4bNq8NRtbscLbyZXLfs7IqfS+M4f9\nWlvWpzTJW27yTTzKRZ9XIZc2JKlWKgC0t3UBcO26FKE2PGo9NOz/9nd3dGZjnaXOH5nnxERqNRcG\n9W+/tIYQwl4z+wngQ8AbgDvw3OL/C9wXQvjujPMnzOxO4HeBtwDvAV4A/gB4CF8cjyAiIstOyy6O\nRWTpCSHcMcfx2RqQzzznMPAbF3GvIeCe+CtjZu+Iv939ojeJiEjLa9nFcVubtylr6u9PiAnI9Xis\n3jQ2Pe35xKHq0drK5Fg2NjbiRe5h2nN01667JhsrxrZua1d7gXtne4ranh3293V3+n127EhbS0+s\n8Ajwyu60MUjj+uQ9ij3UlPd8dswjvlPTfq18IbWaGxqK84s51NNTqdVrPW553YigN1dgDtskIsuV\nmV0TQjgy49i1wL8FpoG/nPWNIiLS0lp2cSwich5/bmZFYBcwBGwGfh7owHfOO3wV5yYiIleJFsci\nslx9HvhnwJvxYrxR4FHgT0IIX7qaExMRkaun5RfHzWkVubhLXK1We9F5xWIRgELB0zHK5fRHUy55\nCsPkhBfF1Zq2IijFsdqUpygcO3QwG6tNeLu2W16yGYC2Umqdlqt4OkZzwWAInhZRqfh5taYkiIPH\nfNOuybijXldXTzaWq/tFOmPLuVzTRcdjekghppnUptMcTrUprUKWrxDCp4BPXe15iIjI4qJNQERE\nREREopaNHE/HCOl0U6S0sSFIo5VbqdSWjeUard/ijwuNtm8A+ZyfX4xR5ebI8+DpEwCcOHIAgBU9\nqcXaDbdtB+Ca1V6k9+T3v5+NdeBFfX193WkOsd1aI7pbbNo0JJdri8/g51Sq1Wysq93vuWrVSv+6\no5yNjY56e7fJiRglbopUn60pciwiIiLSTJFjEREREZFIi2MRERERkahl0yoCsb9vLq3/LRaqNQrW\nmlMu8nlPozDzP5JQT++r171v8Pj4KACnjh3KxgZPepvUdSs8PeInb92ejfWv9PSG6aoX8nUV+7Ox\nvnJfnFSac6MoMNbX0d2ZUi7Wr/X51EJM6WgqupsY9x7IExPjfu2eph3yOjri3D0No72zKxurnKkg\nIiIiIokixyIiIiIiUctGji0fo8RNkdlc3SPFhZwXuoWmnw1KxVgMF7xIbXIy9Ws7fcz3Aji8/zm/\ndn08G9u2fj0AO2/eAcDAmlQMV8l7G7WhQd/BrrOzLxs7edoL5fbt/0F27LYd1wHQ3+PzO3osbd5V\njkV37e1+/VCvZ2M9q1bF5/LnGRtN82vv8ihyoerPM3b2bDZWDEVEREREJFHkWEREREQkatnI8VjM\nw+0opbZra1d55Pb0kB8LNpWN9XR7znFfv7dMO3D4aDZWsEEAtqz392/dcks21tXmkdmumNs7NZ3y\neI+d8DZv+/fvB2D1yg3Z2OCY3ztXSNHbYsH/Oupxl5HGxh0AHZ1+/c7Y3q3UNFad9muFaX+uSjU9\n1/CQR627Ozx/OTS1oSs0tYoTEREREUWORUREREQyWhyLyKJiZvvMbN/VnoeIiCxPLZtWUch5Yd26\ntSuzY0Xz1InK5AgAfQOpWq/c5sV6XR2e5tCWO5mNbYzpFF2FHgByzbvnxRZwIXgqxMjgWDZ2bO8p\nv5Z5+sJUJbWOm57ytI8d123NjvX1+/WLOZ9DdzH99VSqfv74hL+Wyqnwr1T0FIvxireaKxbT/Kqx\nXV2t5q/tHWkHv3pTUZ+IiIiItPDiWETkanvq8DCb3/+VWcf2/eFdV3g2IiJyIVp2cbxlm0dfw1Rq\na3b0kP9+Km7mkRtIjx/qXqh25pQX1BWadueoVj37xAolAPJN/eEaEdlG67diMUVmN63f5ver+TWH\nRlIbtdFTHr1mVTU7VoyR4hiEJt+0gUm55JHiehycmJxIDxvnPm3+2t6eCu3qE/H8icb5k9lYR1MU\nWURERESUcywiV4G53zSzp81s0swOm9mfmFnvOd7zT83sW2Y2GN+z28w+ZGalOc7fYWYPmNlBM6uY\n2XEz++9mdsMs5z5gZsHMtprZb5nZk2Y2YWYPzuNji4jIEtCykeNczvN9j54ayo7VprydWW/Zc3Q7\nSinntpHTWyr5H0m1KT/40OHjABQ2ekS2uy1FXKcaEdyS//s8NJSiw6UYCV45sNrvV+zJxjau9qhy\nb0/aznm6FqO61tjeOj1PqHsEuNThcxibTLnNlYrPoW6xBVxnWiuMjHmUvBYj2/39aQvr5u2zRa6w\n+4F7gKPAfwamgDcCLwPagGrzyWb2X4G3A4eALwFDwMuB3wN+xsxeG0KYbjr/9fG8IvB/gOeBDcAv\nAHeZ2Z0hhMdnmdcngFcDXwG+CtRmOUdERFpYyy6ORWRxMrNX4gvjPcBLQwhn4vEPAt8C1gH7m85/\nK74w/jLwqyGEiaaxe4EPA+/EF7aYWT/wBWAc+OkQwjNN5/848CjwaeC2WaZ3G3BrCOGFi3ieXXMM\n7bjQa4iIyOKhtAoRudLeFl/vayyMAUIIk8Bvz3L+u4Bp4O3NC+Po94DTwK82HfvnQB/w4eaFcbzH\n08B/AW41sx+b5V4fu5iFsYiItJ6WjRwPnjwGwOREKkDr7/QUhv42T2+oT6YxuvznhHzJd7x7bk/6\n93FF10YAnn3Bj61fvyobWzvgaQrTNU/RaG/adS6f9/SIypT/z2wun3bD6+3z9nBtpdR2bXzcd7Yr\n5ONcmv52xkbjNeKOf8XuNNbR43MeHPXd8Eaq2XqD8ar/7/TKXp9nuf3FRYgiV1gjYvvXs4w9hC+E\nATCzDmAncAp4t5nN8hYqwI1NX78ivu6MkeWZro+vNwLPzBj7u3NNfDYhhNtnOx4jyrNFp0VEZBFr\n2cWxiCxajaK74zMHQgg1MzvddKgfMGAVnj5xIVbE13ec57yuWY4du8B7iIhIi2rZxXFnaQCAekf6\nX9ixES9OK0x7QV6tNpWNbVjtodihET//9KnRbGz4pKc/nhzyyGy+LWRjHW0e+e0oequ1rq5UdNco\nlBs560V6habI8dQZXxd0dXdmx0qxVZzh169Mp7mPV2PkuB7byuVT1Hti3KPDFuc1NZ2eq6PD//0P\nMYHmzFCKKnd3pii3yBU0HF/XAHubB8wsjy9uD88494kQwoVGYRvv2RlCePIi5xbOf4qIiLSyll0c\ni8ii9TiebvAaZiyO8U4R2felEMKomT0N/LiZDTTnKJ/DI8Cb47UudnE8r25a38subfYhIrKkqCBP\nRK60B+LrB81soHHQzMrAR2c5/4/x9m6fMbO+mYNm1m9mzVHlz+Kt3j5sZi+d5fycmd1x6dMXEZFW\n1rKR486iF82N109kx3J5T1Mo93hRz/hIaqVaGfOxU0cH/VxLhXKnT3nP5B3Xe2emm27Ylo2VYn1Q\nMf5RToynHflOn/H3lWORXr49pVVMxmLAqak0h54eT8moxXSKSj31Mh6v+P/2lgdikV9Tn+MTJ08C\n0N5ejs/elo1Z3VMsqpX4vkq6X6ktzUfkSgkhPGxmnwR+C3jKzP4Xqc/xIN77uPn8z5jZ7cC/AvaY\n2deBA8AAsAX4aXxBfHc8/7SZvQVv/faImf0V8DRQBzbiBXsrgPJCP6uIiCw9Lbs4FpFF7V3AD/H+\nxL+Ot2P7MvAB4HszTw4hvNPMvoYvgP8h3qrtDL5I/iPgT2ec/1dmdgvwPuBn8RSLKnAE+Cbw5wvy\nVD9q8+7du7n99lmbWYiIyHns3r0bYPOVvq+FoPoTEZH5ZmYVIM8si32RRaKxUc2zV3UWInPbCdRC\nCKXznjmPFDkWEVkYT8HcfZBFrrbG7o76jMpidY4dSBeUCvJERERERCItjkVEREREIi2ORUREREQi\nLY5FRERERCItjkVEREREIrVyExERERGJFDkWEREREYm0OBYRERERibQ4FhERERGJtDgWEREREYm0\nOBYRERERibQ4FhERERGJtDgWEREREYm0OBYRuQBmtsHMPmNmR8ysYmb7zOx+M+u/yOsMxPfti9c5\nEq+7YaHmLsvDfHxGzexBMwvn+FVeyGeQ1mVmbzGzT5rZQ2Y2Ej9Pf3qJ15qX78dzKczHRUREWpmZ\nbQO+A6wG/gJ4Fngp8C7g9Wb2qhDC6Qu4zop4neuBbwJfBHYAbwPuMrNXhBD2Lian/noAAAQNSURB\nVMxTSCubr89ok4/McXz6siYqy9mHgJ3AKHAI/9530Rbgs/4iWhyLiJzfp/BvxPeEED7ZOGhmfwy8\nB7gPuPsCrvMH+ML44yGE9zZd5x7gE/E+r5/HecvyMV+fUQBCCPfO9wRl2XsPvih+HngN8K1LvM68\nftZno+2jRUTOwcy2AnuAfcC2EEK9aawbOAoYsDqEMHaO63QCJ4E6sC6EcLZpLBfvsTneQ9FjuWDz\n9RmN5z8IvCaEYAs2YVn2zOwOfHH8ZyGEX7uI983bZ/1clHMsInJu/yC+fqP5GzFAXOA+DHQALz/P\ndV4BtAMPNy+M43XqwDfil3de9oxluZmvz2jGzH7JzN5vZu81s58zs9L8TVfkks37Z302WhyLiJzb\nDfH1h3OMPxdfr79C1xGZaSE+W18EPgr8e+CrwAEze8ulTU9k3lyR76NaHIuInFtvfB2eY7xxvO8K\nXUdkpvn8bP0F8I+ADfj/dOzAF8l9wP8ws5+7jHmKXK4r8n1UBXkiIpenkZt5uQUc83UdkZku+LMV\nQvj4jEM/AD5gZkeAT+JFpV+b3+mJzJt5+T6qyLGIyLk1IhG9c4z3zDhvoa8jMtOV+Gx9Gm/j9pJY\n+CRyNVyR76NaHIuInNsP4utcOWzXxde5cuDm+zoiMy34ZyuEMAk0Ckk7L/U6Ipfpinwf1eJYROTc\nGr04XxdbrmViBO1VwATwyHmu80g871UzI2/xuq+bcT+RCzVfn9E5mdkNQD++QD51qdcRuUwL/lkH\nLY5FRM4phLAHb7O2GXjnjOGP4FG0zzX31DSzHWb2I7s/hRBGgc/H8++dcZ3fjNf/unocy8War8+o\nmW01s/Uzr29mK4HPxi+/GELQLnmyoMysGD+j25qPX8pn/ZLur01ARETObZbtSncDL8N7Ev8QeGXz\ndqVmFgBmbqQwy/bRfwfcCLwROBGvs2ehn0daz3x8Rs3srXhu8V/jGy2cATYCb8BzPB8DXhtCGFr4\nJ5JWY2ZvAt4Uv1wL/CywF3goHjsVQnhfPHcz8AKwP4SwecZ1Luqzfklz1eJYROT8zOxa4Hfx7Z1X\n4Dsx/W/gIyGEMzPOnXVxHMcGgA/j/0isA07j1f+/E0I4tJDPIK3tcj+jZnYz8K+B24Fr8OKms8DT\nwP8E/lMIobrwTyKtyMzuxb/3zSVbCJ9rcRzHL/izfklz1eJYRERERMQp51hEREREJNLiWEREREQk\n0uJYRERERCTS4lhEREREJNLiWEREREQk0uJYRERERCTS4lhEREREJNLiWEREREQk0uJYRERERCTS\n4lhEREREJNLiWEREREQk0uJYRERERCTS4lhEREREJNLiWEREREQk0uJYRERERCTS4lhEREREJNLi\nWEREREQk+v8wrCiHCOpAHQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f81aa8c6ef0>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 319,
       "width": 355
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "import tensorflow as tf\n",
    "import pickle\n",
    "import helper\n",
    "import random\n",
    "\n",
    "# Set batch size if not already set\n",
    "try:\n",
    "    if batch_size:\n",
    "        pass\n",
    "except NameError:\n",
    "    batch_size = 64\n",
    "\n",
    "save_model_path = './image_classification'\n",
    "n_samples = 4\n",
    "top_n_predictions = 3\n",
    "\n",
    "def test_model():\n",
    "    \"\"\"\n",
    "    Test the saved model against the test dataset\n",
    "    \"\"\"\n",
    "\n",
    "    test_features, test_labels = pickle.load(open('preprocess_training.p', mode='rb'))\n",
    "    loaded_graph = tf.Graph()\n",
    "\n",
    "    with tf.Session(graph=loaded_graph) as sess:\n",
    "        # Load model\n",
    "        loader = tf.train.import_meta_graph(save_model_path + '.meta')\n",
    "        loader.restore(sess, save_model_path)\n",
    "\n",
    "        # Get Tensors from loaded model\n",
    "        loaded_x = loaded_graph.get_tensor_by_name('x:0')\n",
    "        loaded_y = loaded_graph.get_tensor_by_name('y:0')\n",
    "        loaded_keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')\n",
    "        loaded_logits = loaded_graph.get_tensor_by_name('logits:0')\n",
    "        loaded_acc = loaded_graph.get_tensor_by_name('accuracy:0')\n",
    "        \n",
    "        # Get accuracy in batches for memory limitations\n",
    "        test_batch_acc_total = 0\n",
    "        test_batch_count = 0\n",
    "        \n",
    "        for train_feature_batch, train_label_batch in helper.batch_features_labels(test_features, test_labels, batch_size):\n",
    "            test_batch_acc_total += sess.run(\n",
    "                loaded_acc,\n",
    "                feed_dict={loaded_x: train_feature_batch, loaded_y: train_label_batch, loaded_keep_prob: 1.0})\n",
    "            test_batch_count += 1\n",
    "\n",
    "        print('Testing Accuracy: {}\\n'.format(test_batch_acc_total/test_batch_count))\n",
    "\n",
    "        # Print Random Samples\n",
    "        random_test_features, random_test_labels = tuple(zip(*random.sample(list(zip(test_features, test_labels)), n_samples)))\n",
    "        random_test_predictions = sess.run(\n",
    "            tf.nn.top_k(tf.nn.softmax(loaded_logits), top_n_predictions),\n",
    "            feed_dict={loaded_x: random_test_features, loaded_y: random_test_labels, loaded_keep_prob: 1.0})\n",
    "        helper.display_image_predictions(random_test_features, random_test_labels, random_test_predictions)\n",
    "\n",
    "\n",
    "test_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}