PyTorch-Deep-Learning/feedforward-nn.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"x0sQkmyWpeT4"},"source":["# Training Deep Neural Networks on a GPU with PyTorch\n","\n","Unlike the linear or logistic model we used in previous notebooks, we will use a feed-forward neural network here to capture non-linear relationships between inputs and targets. See the [Universal Approximation Theorem](http://neuralnetworksanddeeplearning.com/chap4.html) for more info. For the calculation of gradients in neural networks with multiple layers and the subsequent weight update, see [Stanford CS229 Lecture notes on Backpropagation](https://github.com/BirajCoder/File-host-repo/blob/main/backprop.pdf)."]},{"cell_type":"markdown","metadata":{"id":"h1N-aPtrpeT5"},"source":["## Example: MNIST digit classification with feed-forward neural network"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"WapU423CpeT5"},"outputs":[],"source":["import torch\n","import torchvision\n","import numpy as np\n","import matplotlib\n","import matplotlib.pyplot as plt\n","import torch.nn as nn\n","import torch.nn.functional as F\n","from torchvision.datasets import MNIST\n","from torchvision.transforms import ToTensor\n","from torchvision.utils import make_grid\n","from torch.utils.data.dataloader import DataLoader\n","from torch.utils.data import random_split\n","%matplotlib inline\n","\n","# Use a white background for matplotlib figures\n","matplotlib.rcParams['figure.facecolor'] = '#ffffff'"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Cgek18qlpeT5"},"outputs":[],"source":["# Prepare data\n","\n","# Download dataset\n","dataset = MNIST(root='data/', download=True, transform=ToTensor())\n","\n","# Split dataset into training and validation datasets\n","\n","val_size = 10000\n","train_size = len(dataset) - val_size\n","\n","train_ds, val_ds = random_split(dataset, [train_size, val_size])\n","len(train_ds), len(val_ds)\n","\n","# Create data loaders\n","batch_size=128\n","train_loader = DataLoader(train_ds, batch_size, shuffle=True, num_workers=4, pin_memory=True)\n","val_loader = DataLoader(val_ds, batch_size*2, num_workers=4, pin_memory=True)\n","# Note: see documentation for the use of num_workers and pin_memory arguments\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"YshQj9srpeT6"},"outputs":[],"source":["# To inspect the images:\n","# for images, _ in train_loader:\n","#     print('images.shape:', images.shape)\n","#     plt.figure(figsize=(16,8))\n","#     plt.axis('off')\n","#     plt.imshow(make_grid(images, nrow=16).permute((1, 2, 0))) # make_grid: PyTorch function for arranging images in a grid; # plt.imshow expects channels to be last dimension, so use permute to rearrange dimensions\n","#     break"]},{"cell_type":"markdown","metadata":{"id":"OKEjn2zvpeT6"},"source":["## Hidden Layers\n","\n","The following model has two layers of neural network:\n","- one hidden layer (a linear model)\n","- one output layer (a linear model)\n","\n","Between the two layers, we will add an activation function to capture non-linear relationships between inputs and outputs.\n"]},{"cell_type":"markdown","source":["\n","\n","## Activation function and non-linearity\n","\n","ReLU will be used as the activation function for the outputs:\n","\n","`relu(x) = max(0,x)`\n","\n","ReLU is a non-linear function that converts negative values to 0.\n","\n","<img src=\"https://i.imgur.com/yijV4xF.png\" width=\"420\">\n","\n","The resulting neural network looks like this:\n","\n","<img src=\"https://i.imgur.com/eN7FrpF.png\" width=\"480\">"],"metadata":{"id":"vki1Uyrj0WnI"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"6zXN1c4PpeT6"},"outputs":[],"source":["# Example of a neural network with two layers\n","\n","# Get a batch of input tensors for demonstration\n","for images, labels in train_loader:\n","    inputs = images.reshape(-1, 784) # flatten 28*28 image tensors to 1*784 tensors\n","    break\n","\n","# Create first NN layer (layer1)\n","input_size = inputs.shape[-1] # same as batch size (i.e. 128)\n","hidden_size = 32 # learning capacity\n","layer1 = nn.Linear(input_size, hidden_size)\n","layer1_outputs = layer1(inputs) # the size of layer1_outputs should be 128*32\n","\n","# Activation function\n","relu_outputs = F.relu(layer1_outputs)\n","\n","# Create 2nd NN layer (layer2). This is the final output layer\n","output_size = 10 # The final output has 10 scores for 10 labels\n","layer2 = nn.Linear(hidden_size, output_size)\n","layer2_outputs = layer2(relu_outputs)\n","\n","# # The above neural network can be written in one line\n","# outputs_expanded = (F.relu(inputs @ layer1.weight.t() + layer1.bias)) @ layer2.weight.t() + layer2.bias\n","# torch.allclose(outputs_expanded, layer2_outputs, 1e-3) # use allclose to verify if two tensors are equal (tolerated range of error: 1e-3)"]},{"cell_type":"markdown","metadata":{"id":"l8dOeJyfpeT8"},"source":["## Model\n","\n","Like building a logistic model in the previous notebook, we will extend the nn.Module to build the model with a feedforward neural network."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"_fKPW6tSpeT8"},"outputs":[],"source":["class MnistModel(nn.Module):\n","    \"\"\"Feedfoward neural network with 1 hidden layer\"\"\"\n","    def __init__(self, in_size, hidden_size, out_size):\n","        super().__init__()\n","        # hidden layer\n","        self.linear1 = nn.Linear(in_size, hidden_size)\n","        # output layer\n","        self.linear2 = nn.Linear(hidden_size, out_size)\n","\n","    def forward(self, xb):\n","        # Flatten the image tensors\n","        xb = xb.view(xb.size(0), -1)\n","        # Get intermediate outputs using hidden layer\n","        out = self.linear1(xb)\n","        # Apply activation function\n","        out = F.relu(out)\n","        # Get predictions using output layer\n","        out = self.linear2(out)\n","        return out\n","\n","    def training_step(self, batch):\n","        images, labels = batch\n","        out = self(images)                  # Generate predictions\n","        loss = F.cross_entropy(out, labels) # Calculate loss\n","        return loss\n","\n","    def validation_step(self, batch):\n","        images, labels = batch\n","        out = self(images)                    # Generate predictions\n","        loss = F.cross_entropy(out, labels)   # Calculate loss\n","        acc = accuracy(out, labels)           # Calculate accuracy\n","        return {'val_loss': loss, 'val_acc': acc}\n","\n","    def validation_epoch_end(self, outputs):\n","        batch_losses = [x['val_loss'] for x in outputs]\n","        epoch_loss = torch.stack(batch_losses).mean()   # Combine losses\n","        batch_accs = [x['val_acc'] for x in outputs]\n","        epoch_acc = torch.stack(batch_accs).mean()      # Combine accuracies\n","        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}\n","\n","    def epoch_end(self, epoch, result):\n","        print(\"Epoch [{}], val_loss: {:.4f}, val_acc: {:.4f}\".format(epoch, result['val_loss'], result['val_acc']))\n","\n","# evaluation metric\n","def accuracy(outputs, labels):\n","    _, preds = torch.max(outputs, dim=1)\n","    return torch.tensor(torch.sum(preds == labels).item() / len(preds))"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"beSRnIQTpeT8"},"outputs":[],"source":["# Create the model\n","input_size = 784\n","hidden_size = 32\n","num_classes = 10\n","model = MnistModel(input_size, hidden_size=32, out_size=num_classes)"]},{"cell_type":"markdown","metadata":{"id":"EtWfv0N9peT8"},"source":["The model has one set of weights and one set of bias for each layers. (For two layers, we have 4 tensors.)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"psecOJfZpeT8","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1703547257118,"user_tz":300,"elapsed":11,"user":{"displayName":"Man Ho Wong","userId":"01009376394376826107"}},"outputId":"83585a84-0151-4f5e-83d6-570e611aea1e"},"outputs":[{"output_type":"stream","name":"stdout","text":["torch.Size([32, 784])\n","torch.Size([32])\n","torch.Size([10, 32])\n","torch.Size([10])\n"]}],"source":["for t in model.parameters():\n","    print(t.shape)"]},{"cell_type":"markdown","metadata":{"id":"vmw6iiTXpeT8"},"source":["## Using a GPU\n","\n","For Google Colab or Kaggle users:\n","- Google Colab: Runtime > Change Runtime Type > select \"GPU\"\n","- Kaggle: In \"Settings\" select \"GPU\""]},{"cell_type":"code","execution_count":null,"metadata":{"id":"gt2Ve_cFpeT8","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1703547257118,"user_tz":300,"elapsed":9,"user":{"displayName":"Man Ho Wong","userId":"01009376394376826107"}},"outputId":"a4e6f75a-1438-4a51-df45-a77d66b4c2d9"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["True"]},"metadata":{},"execution_count":31}],"source":["# Check if GPU is available\n","torch.cuda.is_available()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"oY8kxA4tpeT8","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1703547257414,"user_tz":300,"elapsed":301,"user":{"displayName":"Man Ho Wong","userId":"01009376394376826107"}},"outputId":"57276be9-0bc5-449f-e32d-87195c58bf24"},"outputs":[{"output_type":"stream","name":"stdout","text":["torch.Size([128, 1, 28, 28])\n","cuda:0\n"]}],"source":["# Function to select GPU if it's available\n","def get_default_device():\n","    \"\"\"Pick GPU if available, else CPU\"\"\"\n","    if torch.cuda.is_available():\n","        return torch.device('cuda')\n","    else:\n","        return torch.device('cpu')\n","\n","# Function to move data and model to a chosen device\n","def to_device(data, device):\n","    \"\"\"Move tensor(s) to chosen device\"\"\"\n","    if isinstance(data, (list,tuple)):\n","        return [to_device(x, device) for x in data] # for a list of tensors/models\n","    return data.to(device, non_blocking=True)  # for one tensor/model\n","\n","# Example use\n","device = get_default_device()\n","for images, labels in train_loader:\n","    print(images.shape)\n","    images = to_device(images, device)\n","    print(images.device)\n","    break"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Mncx5AEcpeT9"},"outputs":[],"source":["# To move data to the selected device, we can simply wrap the existing data loaders with the following function.\n","# (This function return a data loader with the same batches of data after moving the data to the selected device.)\n","\n","class DeviceDataLoader():\n","    \"\"\"Wrap a dataloader to move data to a device. Return a dataloader (generator).\"\"\"\n","    def __init__(self, dl, device):\n","        self.dl = dl\n","        self.device = device\n","\n","    def __iter__(self):\n","        \"\"\"Yield a batch of data after moving it to device\"\"\"\n","        for b in self.dl:\n","            yield to_device(b, self.device) # yield: see note below\n","\n","    def __len__(self):\n","        \"\"\"Number of batches\"\"\"\n","        return len(self.dl)\n","\n","# Examples\n","train_loader = DeviceDataLoader(train_loader, device)\n","val_loader = DeviceDataLoader(val_loader, device)\n","\n","# Note: You can create a generator function using yield.\n","# def my_generator():\n","#     # Create a generator function with 3 items\n","#     yield 10\n","#     yield 20\n","#     yield 30\n","# # the above generator function can be used in a for loop:\n","# for item in my_generator():\n","#     print(item)"]},{"cell_type":"markdown","metadata":{"id":"sfPhGrIRpeT9"},"source":["## Training the Model\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"JRw6Zf0wpeT9"},"outputs":[],"source":[" # Define functions for training\n","\n","def evaluate(model, val_loader):\n","    \"\"\"Evaluate the model's performance on the validation set\"\"\"\n","    outputs = [model.validation_step(batch) for batch in val_loader]\n","    return model.validation_epoch_end(outputs)\n","\n","def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):\n","    \"\"\"Train the model using gradient descent\"\"\"\n","    history = []\n","    optimizer = opt_func(model.parameters(), lr) # parameters include weights and biases from all layers\n","    for epoch in range(epochs):\n","        # Training Phase\n","        for batch in train_loader:\n","            loss = model.training_step(batch)\n","            loss.backward()\n","            optimizer.step()\n","            optimizer.zero_grad()\n","        # Validation phase\n","        result = evaluate(model, val_loader)\n","        model.epoch_end(epoch, result)\n","        history.append(result)\n","    return history"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"bF_wEBAIpeT9","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1703547335239,"user_tz":300,"elapsed":77832,"user":{"displayName":"Man Ho Wong","userId":"01009376394376826107"}},"outputId":"1189c757-c594-488e-f17f-d55f55e26b00"},"outputs":[{"output_type":"stream","name":"stdout","text":["Epoch [0], val_loss: 0.2331, val_acc: 0.9337\n","Epoch [1], val_loss: 0.1782, val_acc: 0.9500\n","Epoch [2], val_loss: 0.1563, val_acc: 0.9563\n","Epoch [3], val_loss: 0.1640, val_acc: 0.9530\n","Epoch [4], val_loss: 0.1449, val_acc: 0.9590\n","Epoch [5], val_loss: 0.1375, val_acc: 0.9602\n","Epoch [6], val_loss: 0.1323, val_acc: 0.9635\n","Epoch [7], val_loss: 0.1323, val_acc: 0.9634\n","Epoch [8], val_loss: 0.1504, val_acc: 0.9592\n","Epoch [9], val_loss: 0.1295, val_acc: 0.9645\n"]}],"source":["# Move model to the same device as the data\n","model = MnistModel(input_size, hidden_size=hidden_size, out_size=num_classes)\n","to_device(model, device)\n","\n","# Training\n","history = fit(10, 0.5, model, train_loader, val_loader)"]},{"cell_type":"markdown","metadata":{"id":"Lf_6nWkCBGB5"},"source":["## Testing with individual images in the test dataset"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ZJ6E4f-iBBTm"},"outputs":[],"source":["# Define test dataset\n","test_dataset = MNIST(root='data/',\n","                     train=False,\n","                     transform=ToTensor())\n","\n","# Helper function\n","def predict_image(img, model):\n","    xb = to_device(img.unsqueeze(0), device)\n","    yb = model(xb)\n","    _, preds  = torch.max(yb, dim=1)\n","    return preds[0].item()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"JsNnl42DBXog","colab":{"base_uri":"https://localhost:8080/","height":447},"executionInfo":{"status":"ok","timestamp":1703547335550,"user_tz":300,"elapsed":332,"user":{"displayName":"Man Ho Wong","userId":"01009376394376826107"}},"outputId":"1cb717ee-5990-4f3d-b3a8-8a9e4be9a9f3"},"outputs":[{"output_type":"stream","name":"stdout","text":["Label: 7 , Predicted: 7\n"]},{"output_type":"display_data","data":{"text/plain":["<Figure size 640x480 with 1 Axes>"],"image/png":"iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAaqElEQVR4nO3df2xV9f3H8VeL9ILaXiylvb2jQEEFwy8ng9rwYygNtC4GtEtA/QMWAoFdzLDzx7qIKFvSjSWOuCD+s8BMxF+JQCRLMym2hNliqDDCph3tugGBFsVxbylSGP18/yDer1cKeMq9ffdeno/kJPTe8+l9ezzhyWlvT9Occ04AAPSxdOsBAAA3JwIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBM3GI9wLd1d3frxIkTyszMVFpamvU4AACPnHPq6OhQMBhUevrVr3P6XYBOnDihgoIC6zEAADfo2LFjGj58+FWf73dfgsvMzLQeAQAQB9f7+zxhAdq4caNGjRqlQYMGqaioSB9//PF3WseX3QAgNVzv7/OEBOjtt99WRUWF1q5dq08++USTJ0/WvHnzdOrUqUS8HAAgGbkEmDZtmguFQtGPL1265ILBoKuqqrru2nA47CSxsbGxsSX5Fg6Hr/n3fdyvgC5cuKDGxkaVlJREH0tPT1dJSYnq6+uv2L+rq0uRSCRmAwCkvrgH6IsvvtClS5eUl5cX83heXp7a2tqu2L+qqkp+vz+68Q44ALg5mL8LrrKyUuFwOLodO3bMeiQAQB+I+88B5eTkaMCAAWpvb495vL29XYFA4Ir9fT6ffD5fvMcAAPRzcb8CysjI0JQpU1RTUxN9rLu7WzU1NSouLo73ywEAklRC7oRQUVGhxYsX6wc/+IGmTZumDRs2qLOzUz/5yU8S8XIAgCSUkAAtXLhQn3/+uV544QW1tbXp3nvvVXV19RVvTAAA3LzSnHPOeohvikQi8vv91mMAAG5QOBxWVlbWVZ83fxccAODmRIAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATMQ9QC+++KLS0tJitnHjxsX7ZQAASe6WRHzS8ePHa9euXf//Irck5GUAAEksIWW45ZZbFAgEEvGpAQApIiHfAzpy5IiCwaBGjx6tJ554QkePHr3qvl1dXYpEIjEbACD1xT1ARUVF2rJli6qrq7Vp0ya1trZq5syZ6ujo6HH/qqoq+f3+6FZQUBDvkQAA/VCac84l8gXOnDmjkSNH6uWXX9bSpUuveL6rq0tdXV3RjyORCBECgBQQDoeVlZV11ecT/u6AIUOG6O6771Zzc3OPz/t8Pvl8vkSPAQDoZxL+c0Bnz55VS0uL8vPzE/1SAIAkEvcAPf3006qrq9O///1vffTRR3rkkUc0YMAAPfbYY/F+KQBAEov7l+COHz+uxx57TKdPn9awYcM0Y8YMNTQ0aNiwYfF+KQBAEkv4mxC8ikQi8vv91mMAAG7Q9d6EwL3gAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATCf+FdOhbP/7xjz2vWbZsWa9e68SJE57XnD9/3vOaN954w/OatrY2z2skXfUXJwKIP66AAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYCLNOeesh/imSCQiv99vPUbS+te//uV5zahRo+I/iLGOjo5erfv73/8e50kQb8ePH/e8Zv369b16rf379/dqHS4Lh8PKysq66vNcAQEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJm6xHgDxtWzZMs9rJk2a1KvX+vTTTz2vueeeezyvue+++zyvmT17tuc1knT//fd7XnPs2DHPawoKCjyv6Uv/+9//PK/5/PPPPa/Jz8/3vKY3jh492qt13Iw0sbgCAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMcDPSFFNTU9Mna3qrurq6T17njjvu6NW6e++91/OaxsZGz2umTp3qeU1fOn/+vOc1//znPz2v6c0NbbOzsz2vaWlp8bwGiccVEADABAECAJjwHKA9e/bo4YcfVjAYVFpamrZv3x7zvHNOL7zwgvLz8zV48GCVlJToyJEj8ZoXAJAiPAeos7NTkydP1saNG3t8fv369XrllVf02muvad++fbrttts0b968Xn1NGQCQujy/CaGsrExlZWU9Puec04YNG/T8889r/vz5kqTXX39deXl52r59uxYtWnRj0wIAUkZcvwfU2tqqtrY2lZSURB/z+/0qKipSfX19j2u6uroUiURiNgBA6otrgNra2iRJeXl5MY/n5eVFn/u2qqoq+f3+6FZQUBDPkQAA/ZT5u+AqKysVDoej27Fjx6xHAgD0gbgGKBAISJLa29tjHm9vb48+920+n09ZWVkxGwAg9cU1QIWFhQoEAjE/WR+JRLRv3z4VFxfH86UAAEnO87vgzp49q+bm5ujHra2tOnjwoLKzszVixAitXr1av/71r3XXXXepsLBQa9asUTAY1IIFC+I5NwAgyXkO0P79+/XAAw9EP66oqJAkLV68WFu2bNGzzz6rzs5OLV++XGfOnNGMGTNUXV2tQYMGxW9qAEDSS3POOeshvikSicjv91uPAcCj8vJyz2veeecdz2sOHz7sec03/9HsxZdfftmrdbgsHA5f8/v65u+CAwDcnAgQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGDC869jAJD6cnNzPa959dVXPa9JT/f+b+B169Z5XsNdrfsnroAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABPcjBTAFUKhkOc1w4YN87zmv//9r+c1TU1Nntegf+IKCABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwwc1IgRQ2ffr0Xq37xS9+EedJerZgwQLPaw4fPhz/QWCCKyAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQ3IwVS2EMPPdSrdQMHDvS8pqamxvOa+vp6z2uQOrgCAgCYIEAAABOeA7Rnzx49/PDDCgaDSktL0/bt22OeX7JkidLS0mK20tLSeM0LAEgRngPU2dmpyZMna+PGjVfdp7S0VCdPnoxub7755g0NCQBIPZ7fhFBWVqaysrJr7uPz+RQIBHo9FAAg9SXke0C1tbXKzc3V2LFjtXLlSp0+ffqq+3Z1dSkSicRsAIDUF/cAlZaW6vXXX1dNTY1++9vfqq6uTmVlZbp06VKP+1dVVcnv90e3goKCeI8EAOiH4v5zQIsWLYr+eeLEiZo0aZLGjBmj2tpazZkz54r9KysrVVFREf04EokQIQC4CST8bdijR49WTk6Ompube3ze5/MpKysrZgMApL6EB+j48eM6ffq08vPzE/1SAIAk4vlLcGfPno25mmltbdXBgweVnZ2t7OxsvfTSSyovL1cgEFBLS4ueffZZ3XnnnZo3b15cBwcAJDfPAdq/f78eeOCB6Mdff/9m8eLF2rRpkw4dOqQ//elPOnPmjILBoObOnatf/epX8vl88ZsaAJD00pxzznqIb4pEIvL7/dZjAP3O4MGDPa/Zu3dvr15r/Pjxntc8+OCDntd89NFHntcgeYTD4Wt+X597wQEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMBE3H8lN4DEeOaZZzyv+f73v9+r16qurva8hjtbwyuugAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAE9yMFDDwox/9yPOaNWvWeF4TiUQ8r5GkdevW9Wod4AVXQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACW5GCtygoUOHel7zyiuveF4zYMAAz2v+/Oc/e14jSQ0NDb1aB3jBFRAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIKbkQLf0JsbflZXV3teU1hY6HlNS0uL5zVr1qzxvAboK1wBAQBMECAAgAlPAaqqqtLUqVOVmZmp3NxcLViwQE1NTTH7nD9/XqFQSEOHDtXtt9+u8vJytbe3x3VoAEDy8xSguro6hUIhNTQ06IMPPtDFixc1d+5cdXZ2Rvd56qmn9P777+vdd99VXV2dTpw4oUcffTTugwMAkpunNyF8+5utW7ZsUW5urhobGzVr1iyFw2H98Y9/1NatW/Xggw9KkjZv3qx77rlHDQ0Nuv/+++M3OQAgqd3Q94DC4bAkKTs7W5LU2NioixcvqqSkJLrPuHHjNGLECNXX1/f4Obq6uhSJRGI2AEDq63WAuru7tXr1ak2fPl0TJkyQJLW1tSkjI0NDhgyJ2TcvL09tbW09fp6qqir5/f7oVlBQ0NuRAABJpNcBCoVCOnz4sN56660bGqCyslLhcDi6HTt27IY+HwAgOfTqB1FXrVqlnTt3as+ePRo+fHj08UAgoAsXLujMmTMxV0Ht7e0KBAI9fi6fzyefz9ebMQAASczTFZBzTqtWrdK2bdu0e/fuK36ae8qUKRo4cKBqamqijzU1Neno0aMqLi6Oz8QAgJTg6QooFApp69at2rFjhzIzM6Pf1/H7/Ro8eLD8fr+WLl2qiooKZWdnKysrS08++aSKi4t5BxwAIIanAG3atEmSNHv27JjHN2/erCVLlkiSfv/73ys9PV3l5eXq6urSvHnz9Oqrr8ZlWABA6khzzjnrIb4pEonI7/dbj4Gb1N133+15zWeffZaASa40f/58z2vef//9BEwCfDfhcFhZWVlXfZ57wQEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMBEr34jKtDfjRw5slfr/vKXv8R5kp4988wzntfs3LkzAZMAdrgCAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMcDNSpKTly5f3at2IESPiPEnP6urqPK9xziVgEsAOV0AAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAluRop+b8aMGZ7XPPnkkwmYBEA8cQUEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJjgZqTo92bOnOl5ze23356ASXrW0tLiec3Zs2cTMAmQXLgCAgCYIEAAABOeAlRVVaWpU6cqMzNTubm5WrBggZqammL2mT17ttLS0mK2FStWxHVoAEDy8xSguro6hUIhNTQ06IMPPtDFixc1d+5cdXZ2xuy3bNkynTx5MrqtX78+rkMDAJKfpzchVFdXx3y8ZcsW5ebmqrGxUbNmzYo+fuuttyoQCMRnQgBASrqh7wGFw2FJUnZ2dszjb7zxhnJycjRhwgRVVlbq3LlzV/0cXV1dikQiMRsAIPX1+m3Y3d3dWr16taZPn64JEyZEH3/88cc1cuRIBYNBHTp0SM8995yampr03nvv9fh5qqqq9NJLL/V2DABAkup1gEKhkA4fPqy9e/fGPL58+fLonydOnKj8/HzNmTNHLS0tGjNmzBWfp7KyUhUVFdGPI5GICgoKejsWACBJ9CpAq1at0s6dO7Vnzx4NHz78mvsWFRVJkpqbm3sMkM/nk8/n680YAIAk5ilAzjk9+eST2rZtm2pra1VYWHjdNQcPHpQk5efn92pAAEBq8hSgUCikrVu3aseOHcrMzFRbW5skye/3a/DgwWppadHWrVv10EMPaejQoTp06JCeeuopzZo1S5MmTUrIfwAAIDl5CtCmTZskXf5h02/avHmzlixZooyMDO3atUsbNmxQZ2enCgoKVF5erueffz5uAwMAUoPnL8FdS0FBgerq6m5oIADAzYG7YQPf8Le//c3zmjlz5nhe8+WXX3peA6QabkYKADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJhIc9e7xXUfi0Qi8vv91mMAAG5QOBxWVlbWVZ/nCggAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAICJfhegfnZrOgBAL13v7/N+F6COjg7rEQAAcXC9v8/73d2wu7u7deLECWVmZiotLS3muUgkooKCAh07duyad1hNdRyHyzgOl3EcLuM4XNYfjoNzTh0dHQoGg0pPv/p1zi19ONN3kp6eruHDh19zn6ysrJv6BPsax+EyjsNlHIfLOA6XWR+H7/Jrdfrdl+AAADcHAgQAMJFUAfL5fFq7dq18Pp/1KKY4DpdxHC7jOFzGcbgsmY5Dv3sTAgDg5pBUV0AAgNRBgAAAJggQAMAEAQIAmEiaAG3cuFGjRo3SoEGDVFRUpI8//th6pD734osvKi0tLWYbN26c9VgJt2fPHj388MMKBoNKS0vT9u3bY553zumFF15Qfn6+Bg8erJKSEh05csRm2AS63nFYsmTJFedHaWmpzbAJUlVVpalTpyozM1O5ublasGCBmpqaYvY5f/68QqGQhg4dqttvv13l5eVqb283mjgxvstxmD179hXnw4oVK4wm7llSBOjtt99WRUWF1q5dq08++USTJ0/WvHnzdOrUKevR+tz48eN18uTJ6LZ3717rkRKus7NTkydP1saNG3t8fv369XrllVf02muvad++fbrttts0b948nT9/vo8nTazrHQdJKi0tjTk/3nzzzT6cMPHq6uoUCoXU0NCgDz74QBcvXtTcuXPV2dkZ3eepp57S+++/r3fffVd1dXU6ceKEHn30UcOp4++7HAdJWrZsWcz5sH79eqOJr8IlgWnTprlQKBT9+NKlSy4YDLqqqirDqfre2rVr3eTJk63HMCXJbdu2Lfpxd3e3CwQC7ne/+130sTNnzjifz+fefPNNgwn7xrePg3POLV682M2fP99kHiunTp1yklxdXZ1z7vL/+4EDB7p33303us+nn37qJLn6+nqrMRPu28fBOed++MMfup/97Gd2Q30H/f4K6MKFC2psbFRJSUn0sfT0dJWUlKi+vt5wMhtHjhxRMBjU6NGj9cQTT+jo0aPWI5lqbW1VW1tbzPnh9/tVVFR0U54ftbW1ys3N1dixY7Vy5UqdPn3aeqSECofDkqTs7GxJUmNjoy5evBhzPowbN04jRoxI6fPh28fha2+88YZycnI0YcIEVVZW6ty5cxbjXVW/uxnpt33xxRe6dOmS8vLyYh7Py8vTZ599ZjSVjaKiIm3ZskVjx47VyZMn9dJLL2nmzJk6fPiwMjMzrccz0dbWJkk9nh9fP3ezKC0t1aOPPqrCwkK1tLTol7/8pcrKylRfX68BAwZYjxd33d3dWr16taZPn64JEyZIunw+ZGRkaMiQITH7pvL50NNxkKTHH39cI0eOVDAY1KFDh/Tcc8+pqalJ7733nuG0sfp9gPD/ysrKon+eNGmSioqKNHLkSL3zzjtaunSp4WToDxYtWhT988SJEzVp0iSNGTNGtbW1mjNnjuFkiREKhXT48OGb4vug13K147B8+fLonydOnKj8/HzNmTNHLS0tGjNmTF+P2aN+/yW4nJwcDRgw4Ip3sbS3tysQCBhN1T8MGTJEd999t5qbm61HMfP1OcD5caXRo0crJycnJc+PVatWaefOnfrwww9jfn1LIBDQhQsXdObMmZj9U/V8uNpx6ElRUZEk9avzod8HKCMjQ1OmTFFNTU30se7ubtXU1Ki4uNhwMntnz55VS0uL8vPzrUcxU1hYqEAgEHN+RCIR7du376Y/P44fP67Tp0+n1PnhnNOqVau0bds27d69W4WFhTHPT5kyRQMHDow5H5qamnT06NGUOh+udxx6cvDgQUnqX+eD9bsgvou33nrL+Xw+t2XLFvePf/zDLV++3A0ZMsS1tbVZj9anfv7zn7va2lrX2trq/vrXv7qSkhKXk5PjTp06ZT1aQnV0dLgDBw64AwcOOEnu5ZdfdgcOHHD/+c9/nHPO/eY3v3FDhgxxO3bscIcOHXLz5893hYWF7quvvjKePL6udRw6Ojrc008/7err611ra6vbtWuXu++++9xdd93lzp8/bz163KxcudL5/X5XW1vrTp48Gd3OnTsX3WfFihVuxIgRbvfu3W7//v2uuLjYFRcXG04df9c7Ds3NzW7dunVu//79rrW11e3YscONHj3azZo1y3jyWEkRIOec+8Mf/uBGjBjhMjIy3LRp01xDQ4P1SH1u4cKFLj8/32VkZLjvfe97buHCha65udl6rIT78MMPnaQrtsWLFzvnLr8Ve82aNS4vL8/5fD43Z84c19TUZDt0AlzrOJw7d87NnTvXDRs2zA0cONCNHDnSLVu2LOX+kdbTf78kt3nz5ug+X331lfvpT3/q7rjjDnfrrbe6Rx55xJ08edJu6AS43nE4evSomzVrlsvOznY+n8/deeed7plnnnHhcNh28G/h1zEAAEz0++8BAQBSEwECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABg4v8AjVqFRqQZEfIAAAAASUVORK5CYII=\n"},"metadata":{}}],"source":["img, label = test_dataset[0]\n","plt.imshow(img[0], cmap='gray')\n","print('Label:', label, ', Predicted:', predict_image(img, model))"]},{"cell_type":"markdown","metadata":{"id":"CZz2M4WPBzlT"},"source":["# Check the accuracy/loss on the test set\n","The numbers for the test set should be similar to those for the validation set. If not, we might need another validation set which has similar data distribution as the test set."]},{"cell_type":"code","execution_count":null,"metadata":{"id":"rvNoyn0RBtBm","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1703547336637,"user_tz":300,"elapsed":1098,"user":{"displayName":"Man Ho Wong","userId":"01009376394376826107"}},"outputId":"a02b0b09-5ede-4257-c424-a82337ca7577"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["{'val_loss': 0.10948429256677628, 'val_acc': 0.96728515625}"]},"metadata":{},"execution_count":38}],"source":["test_loader = DeviceDataLoader(DataLoader(test_dataset, batch_size=256), device)\n","result = evaluate(model, test_loader)\n","result"]},{"cell_type":"markdown","source":["## Save the model parameters"],"metadata":{"id":"ORUyEqmmRuH7"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"BT6v860WByIJ"},"outputs":[],"source":["torch.save(model.state_dict(), 'mnist-feedforward.pth')"]}],"metadata":{"accelerator":"GPU","colab":{"provenance":[{"file_id":"1_qNrqSsgz6I3gCgVNHkzKms3KhFOgpqD","timestamp":1703297675310}]},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.8.5"}},"nbformat":4,"nbformat_minor":0}