pytorch/computer_vision/pre_trained_models_and_transfer_learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Trasnfer Learning\n",
    "\n",
    "Training CNNs can take a lot of time, and a lot of data is required for that task. However, much of the time is spent to learn the best low-level filters that a network is using to extract patterns from images. A neural question arises - can we use a neural network trained on one dataset and adapt it to classifying different images without full training process?\n",
    "\n",
    "This approach is called `transfer learning`, because transfer some knowledge from one neural network model to another. In transfer learning, we typically start with a pre-trained model, which has been trained on some large image dataset, such as `ImageNet`. Those models can already do a good job extracting different features from generic images, and in many cases just building a classifier on top of those extracted features can yield a good result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.utils import data\n",
    "import torchvision\n",
    "from torchvision.transforms import transforms\n",
    "from torchinfo import summary\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from PIL import Image\n",
    "import glob, os, zipfile # glob for reading files, os for path, zipfile for unzipping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check the paltform, Apple Silicon or Linux\n",
    "import os, platform\n",
    "\n",
    "torch_device=\"cpu\"\n",
    "\n",
    "if 'kaggle' in os.environ.get('KAGGLE_URL_BASE','localhost'):\n",
    "    torch_device = 'cuda'\n",
    "else:\n",
    "    torch_device = 'mps' if platform.system() == 'Darwin' else 'cpu'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ['PYTORCH_ENABLE_MPS_FALLBACK'] = '1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch_device"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cats vs. Dogs Dataset\n",
    "\n",
    "We will solve a real-life problem of classifying images of cats and dogs. For this reason, we will use [Kaggle Cats vs. Dogs Dataset](https://www.kaggle.com/c/dogs-vs-cats) which can also be downloaded [from Microsoft](https://www.microsoft.com/en-us/download/details.aspx?id=54765).\n",
    "\n",
    "Let's download this dataset and extract it into `data` directory (this process may take some time!):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists('data/kagglecatsanddogs_5340.zip'):\n",
    "    !wget -P data https://download.microsoft.com/download/3/E/1/3E1C3F21-ECDB-4869-8368-6DEBA77B919F/kagglecatsanddogs_5340.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "if not os.apth.exists('data/PetImages'):\n",
    "    with zipfile.ZipFile('data/kagglecatsanddogs_5340.zip', 'r') as zip_ref: # zipfile.ZipFile() is a class here we use it as a context manager to open the zip file\n",
    "        zip_ref.extractall('data') # extractall() is a method of the class zipfile.ZipFile() to extract all the files in the zip file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunatelly, there are some corrupt image files in the dataset. We need to do quick cleaning to check for corrupted files. In order not to clobber this tutorial, we moved the code to verify dataset into a module."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_image(fn):\n",
    "    try:\n",
    "        im= Image.open(fn)\n",
    "        im.verify() # verify() is a method of the class Image() to check if the image is broken\n",
    "        return True\n",
    "    except:\n",
    "        return False\n",
    "\n",
    "def check_image_dir(path):\n",
    "    for fn in glob.glob(path):\n",
    "        if not check_image(fn):\n",
    "            print('Corrupt image: {}'.format(fn))\n",
    "            os.remove(fn)\n",
    "\n",
    "check_image_dir('data/PetImages/Cat/*.jpg')\n",
    "check_image_dir('data/PetImages/Dog/*.jpg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's load the images into PyTorch dataset, converting them to tensors and doing some normalization. We will apply `std_normalize` transform to bring images to the range expected by pre-trained VGG network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "std_normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],\n",
    "                                     std=[0.229, 0.224, 0.225])\n",
    "trans = transforms.Compose([\n",
    "    transforms.Resize(256),\n",
    "    transforms.CenterCrop(224),\n",
    "    transforms.ToTensor(),\n",
    "    std_normalize\n",
    "])\n",
    "\n",
    "dataset = torchvision.datasets.ImageFolder('data/PetImages', transform=trans)\n",
    "trainset, testset = torch.utils.data.random_split(dataset, [20000, len(dataset)-20000])\n",
    "\n",
    "\n",
    "def display_dataset(dataset, n=10, classes=None):\n",
    "    fig,ax=plt.subplots(1,n,figsize=(15,3))\n",
    "    mn = min(dataset[i][0].min() for i in range(n))\n",
    "    mx = max(dataset[i][0].max() for i in range(n))\n",
    "    for i in range(n):\n",
    "        ax[i].imshow(np.transpose((dataset[i][0]-mn)/(mx-mn),(1,2,0)))\n",
    "        ax[i].axis('off')\n",
    "        if classes:\n",
    "            ax[i].set_title(classes[dataset[i][1]])\n",
    "\n",
    "display_dataset(dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-trained models\n",
    "\n",
    "There are many different pre-trained models avaliable inside `torchvision` module, and even more models can be found on the Internet. Let's see how simplest VGG-16 model can be loaded and used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vgg = torchvision.models.vgg16(pretrained=True) # vgg16 is a class here we use it as a function to get the model which is names vgg\n",
    "sample_image = dataset[0][0].unsqueeze(0) # unsqueeze() is a method of the class Tensor() to add a dimension to the tensor\n",
    "res = vgg(sample_image) # here we use the model as a function to get the output of the model\n",
    "print(res[0].argmax()) # argmax() is a method of the class Tensor() to get the index of the maximum value in the tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result that we have reveived is a number of an `ImageNet` class, which can be looked up [here](https://gist.github.com/yrevar/942d3a0ac09ec9e5eb3a). We can use the following code to automatically load this class table and return the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json, requests\n",
    "\n",
    "class_map = json.loads(requests.get('https://raw.githubusercontent.com/hololandscape/notebooks/main/pytorch/computer_vision/imagenet_class_index.json').text)\n",
    "class_map = {int(k):v for k,v in class_map.items()}\n",
    "\n",
    "class_map[res[0].argmax().item()] # res[0].argmax().item() is the index of the class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking the architecture of the VGG-16 networks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary(vgg, input_size=(1,3,224,224))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the layer we already know, there is also another layer type called `Dropout`. These layers act as `regularization` technique.\n",
    "\n",
    "Regularization makes slight modifications to the learning algorithm so the model generalizes better. During training, dropout layers discard some proportion (around 30%) of the neurons in the previous layer, and training happens without them. This helps to get the optimization process out of local minima, and to distribute decisive power between neural paths, which improves overall stability of the network."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GPU computaions\n",
    "\n",
    "Deep neural networks, such as VGG-16 and other more modern architectures require quite a lot of computational power to run. It makes sense to use GPU acceleration, if it is available."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vgg.to(torch_device)\n",
    "\n",
    "sample_image = sample_image.to(torch_device)\n",
    "\n",
    "vgg(sample_image).argmax()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting VGG features\n",
    "\n",
    "If we want to use VGG-16 to extract features from our images, we need the model without final classification layers. In fact, this \"feature extractor\" can be obtained using `vgg.features` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = vgg.features(sample_image).cpu() # features() is a method of the class VGG() to get the output of the convolutional layers\n",
    "plt.figure(figsize=(15, 3))\n",
    "plt.imshow(res.detach().view(-1,512))\n",
    "print(res.size())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dimension of feature tensor is 512x7x7, but in order to visualize it we had to reshape it to 2D form.\n",
    "\n",
    "Now let's try to see if those features can be used to classify images. Let's manually take some portion of images (800 in our case), and pre-compute their feature vectors. We will store the result in one big tensor called `feature_tensor`, and also labels into `label_tensor`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 8 # batch size\n",
    "dl = torch.utils.data.DataLoader(dataset, batch_size=bs, shuffle=True)\n",
    "num = bs*100 # number of images\n",
    "feature_tensor = torch.zeros(num, 512*7*7).to(torch_device) # we use torch.zrtos() to create a tensor of zeros, num is the number of images, 512*7*7 is the size of the output of the convolutional layers\n",
    "label_tensor = torch.zeros(num).to(torch_device) # torch.zeros() is a function to create a tensor of zeros\n",
    "i=0\n",
    "for x, l in dl:\n",
    "    with torch.no_grad():\n",
    "        f = vgg.features(x.to(torch_device))\n",
    "        feature_tensor[i:i+bs]=f.view(bs,-1) # view() is a method of the class Tensor() to reshape the tensor\n",
    "        label_tensor[i:i+bs]=l\n",
    "        i+=bs\n",
    "        print('.',end='')\n",
    "        if i>=num:\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can define `vgg_dataset` that takes data from thei tensor, split it into training and test sets using `random_split` function, and train a small one-layer dense classifier network on top of extracted features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_epoch(net, dataloader, lr=0.01, optimizer=None, loss_fn=nn.NLLLoss()):\n",
    "    optimizer = optimizer or torch.optim.Adam(net.parameters(), lr=lr) # use Adam optimizer if not provided\n",
    "    net.train() # put the network into training mode make sure the parameters are trainable\n",
    "    total_loss,acc,count =0,0,0\n",
    "    for features, labels in dataloader:\n",
    "        features,labels = features.to(torch_device), labels.to(torch_device)\n",
    "        optimizer.zero_grad() # reset the gradients to zero before each batch to avoid accumulation\n",
    "        out=net(features) # forward pass of the mini-batch through the network to obtain the outputs\n",
    "        loss=loss_fn(out,labels) # compute the loss\n",
    "        loss.backward() # compute the gradients of the loss with respect to all the parameters of the network\n",
    "        optimizer.step() # update the parameters of the network using the gradients to minimize the loss\n",
    "        total_loss+=loss # accumulate the loss for inspection\n",
    "        _,preds=torch.max(out,dim=1) # compute the predictions to obtain the accuracy\n",
    "        acc+=(preds==labels).sum() # accumulate the correct predictions\n",
    "        count+=len(labels) # accumulate the total number of examples\n",
    "    return total_loss.item()/count, acc.item()/count # return the loss and accuracy\n",
    "\n",
    "def validate(net, dataloader, loss_fn=nn.NLLLoss()):\n",
    "    net.eval() # put the network into evaluation mode to deactivate the dropout layers\n",
    "    count,acc,loss =0,0,0\n",
    "    with torch.no_grad(): # deactivate autograd to save memory and speed up computations\n",
    "        for features, labels in dataloader:\n",
    "            features,labels = features.to(torch_device), labels.to(torch_device)\n",
    "            out=net(features) # forward pass of the mini-batch through the network to obtain the outputs\n",
    "            loss += loss_fn(out,labels) # compute the loss\n",
    "            preds=torch.max(out,dim=1)[1] # compute the predictions to obtain the accuracy\n",
    "            acc+=(preds==labels).sum() # accumulate the correct predictions\n",
    "            count+=len(labels) # accumulate the total number of examples\n",
    "    return loss.item()/count, acc.item()/count # return the loss and accuracy\n",
    "\n",
    "def train(net, train_loader, test_loader, optimizer=None, lr=0.01, epochs=10, loss_fn=nn.NLLLoss()):\n",
    "    optimizer = optimizer or torch.optim.Adam(net.parameters(), lr=lr) # use Adam optimizer if not provided\n",
    "    res = {'train_loss': [], 'train_acc': [], 'val_loss': [], 'val_acc':[]}\n",
    "    for ep in range(epochs):\n",
    "        tl, ta = train_epoch(net, train_loader, optimizer=optimizer,lr=lr, loss_fn=loss_fn)\n",
    "        vl,va = validate(net, test_loader, loss_fn=loss_fn)\n",
    "        print(f\"Epoch {ep:2}, Train acc={ta:.3f}, Val acc={va:.3f}, Train loss={tl:.3f}, Val loss={vl:.3f}\")\n",
    "        res['train_loss'].append(tl)\n",
    "        res['train_acc'].append(ta)\n",
    "        res['val_loss'].append(vl)\n",
    "        res['val_acc'].append(va)\n",
    "    return res\n",
    "\n",
    "def plot_results(hist):\n",
    "    plt.figure(figsize=(15,5)) # figure is used to create a new figure\n",
    "    plt.subplot(121) # subplot is used to create a new subplot on a grid\n",
    "    plt.plot(hist['train_acc'], label='Training acc')\n",
    "    plt.plot(hist['val_acc'], label='Validation acc')\n",
    "    plt.legend() # legend is used to add a legend to the plot\n",
    "    plt.subplot(122) # plot the loss\n",
    "    plt.plot(hist['train_loss'], label='Training loss')\n",
    "    plt.plot(hist['val_loss'], label='Validation loss')\n",
    "    plt.legend()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# TensorDataset() is a class to create a dataset from tensors of features and labels. Here we use `torch.long` to convert the labels to long type in order to use them in the loss function.\n",
    "vgg_dataset = torch.utils.data.TensorDataset(feature_tensor, label_tensor.to(torch.long))\n",
    "train_ds, test_ds = torch.utils.data.random_split(vgg_dataset, [700, 100])\n",
    "\n",
    "train_loader =torch.utils.data.DataLoader(train_ds, batch_size=32, shuffle=True)\n",
    "test_loader =torch.utils.data.DataLoader(test_ds, batch_size=32, shuffle=True)\n",
    "\n",
    "net = torch.nn.Sequential(\n",
    "    torch.nn.Linear(512*7*7, 2),\n",
    "    torch.nn.LogSoftmax(dim=0)\n",
    ").to(torch_device)\n",
    "\n",
    "history = train(net,train_loader,test_loader)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is great, we can distinguish  between a cat and a dog with almost 98% probability! However, we have only tested this aproach on a small subset of all images, because manual feature extraction seems to take a lot of time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Transder learning using one VGG network\n",
    "\n",
    "We can also avoid manually pre-computing the features by using the original VGG-16 network as a whole during training> Let's look at the VGG-16 object structure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(vgg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that the network contains:\n",
    "* feature extractor `features`, comprised of a number of convolutional and pooling layers\n",
    "* average pooling layer `avgpool`\n",
    "* final `classifier`, consisting of several dense layers, which turns 25088 input features into 1000 classes (which is the number of classes in ImageNet)\n",
    "\n",
    "To train the end-to-end model that will classify our dataset,  we need to:\n",
    "* `replace the final classifier` with the one that will produce required number of classes. In our case, we can use one `Linear` layer with 25088 inputs and 2 output neurons.\n",
    "* `freeze weights of convolutional feature extractor`, so that they are no trained. It is recommended to initially do this freezing, because otherwise untrained classifier layer can destroy the original pre-trained weights of convolutional extractor. Freezing weights can be accomplished by setting `requires_grad` property of all parameters of `False`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vgg.classifier = torch.nn.Linear(25088,2).to(torch_device)\n",
    "\n",
    "for x in vgg.features.parameters():\n",
    "    x.requires_grad=False\n",
    "\n",
    "summary(vgg, input_size=(1,3,224,224))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you cane see from the summary, this model contain around 15 million total parameters, but only 50k of them are trainable - those are the weights of classification layer. That is good, because we are able to fine-tune smaller number of parameters with smaller number of examples.\n",
    "\n",
    "Now, let's train the model using our original dataset. This process will take a long time, so we will use `train_long` function that will print some intermediate results without waiting for thr end of epoch. It is highly recommended to run this training on GPU-enabled computer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_long(net, train_loader, test_loader, epochs=5, lr=0.01, optimizer=None, loss_fn=nn.NLLLoss(), print_freq=10):\n",
    "    optimizer = optimizer or torch.optim.Adam(net.parameters(), lr=lr) # use Adam optimizer if not provided\n",
    "    for epoch in range(epochs):\n",
    "        net.train() # put the network into training mode make sure the parameters are trainable\n",
    "        total_loss,acc,count =0,0,0\n",
    "        for i, (features, labels) in enumerate(train_loader):\n",
    "            lbls = labels.to(torch_device)\n",
    "            optimizer.zero_grad() # reset the gradients to zero before each batch to avoid accumulation\n",
    "            out=net(features.to(torch_device)) # forward pass of the mini-batch through the network to obtain the outputs\n",
    "            loss = loss_fn(out, lbls) # compute the loss\n",
    "            loss.backward() # compute the gradients of the loss with respect to all the parameters of the network\n",
    "            optimizer.step() # update the parameters of the network using the gradients to minimize the loss\n",
    "            total_loss+=loss # accumulate the loss for inspection\n",
    "            _,preds=torch.max(out,dim=1) # compute the predictions to obtain the accuracy\n",
    "            acc+=(preds==lbls).sum() # accumulate the correct predictions\n",
    "            count+=len(lbls) # accumulate the total number of examples\n",
    "            if i%print_freq==0:\n",
    "                print(f'Epoch {epoch}, iter {i}, loss={total_loss.item()/count:.3f}, acc={acc.item()/count:.3f}')\n",
    "        vl, va = validate(net, test_loader, loss_fn=loss_fn)\n",
    "        print(f'Epoch {epoch}, val_loss={vl:.3f}, val_acc={va:.3f}')\n",
    "\n",
    "trainset, testset = torch.utils.data.random_split(dataset, [20000, len(dataset)-20000])\n",
    "train_loader = torch.utils.data.DataLoader(trainset, batch_size=16, shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(testset, batch_size=16, shuffle=True)\n",
    "\n",
    "train_long(vgg,   train_loader, test_loader, loss_fn=torch.nn.CrossEntropyLoss(), epochs=1, print_freq=90) # print_freq=90 means print the results every 90 batches"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Saving the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(vgg, 'data/cats_dogs.pth')\n",
    "\n",
    "# load model from the file anytime\n",
    "# vgg= torch.load('data/cats_dogs.pth')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning transfer learning\n",
    "\n",
    "In the previous section, we have trained the final classifier layer to classify images in our own dataset. However, we did not re-train the feature extractor, and our model relied on the features that the model has learned on ImageNet data. If your objects visually differ from ordinary ImageNet images, thiw combination of features might not work best. Thus it makes sense to start training convolutional layers as well.\n",
    "\n",
    "To do that, we can unfreeze the convolutional filter parameters that we have previously frozen.\n",
    "\n",
    "> Note: It is important that you freeze parameters first and perform several epochs of training in order to stabilize weights in the classification layer. If you immediately start training end-to-end network with unfrozen parameters, large errors are likely to destroy the pre-trained weights in the convolutional layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for x in vgg.features.parameters():\n",
    "    x.requires_grad=True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After unfreezing, we can do a few more epochs of training. You can also select lower learning rate, in order to minimize the impact on the pre-trained weights. However, even with low learning rate, you can expect the accuracy to drop in the beginning of the training, until finally reaching slightly higher level than  in the case of fixed weights.\n",
    "\n",
    "> Note: This training happens mush slower, because we need to propagate gradients back through many layers of the network! You mat want to watch the first few minibatches to see the tendency, ansd then stop the computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_long(vgg, train_loader, test_loader, loss_fn=torch.nn.CrossEntropyLoss(), epochs=1, print_freq=90, lr=0.0001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other computer vision models\n",
    "\n",
    "VGG-16 is one of the simplest computer vision architectures. `torchvision` package provides many ,pre pre-trained networks. The most frequently used one among thise are `ResNet` architectures, developed by Microsoft, and `Inception` by Google. For example, let's explore the architecture of the simplest ResNet-18 model (ResNet is a family of model with different depth, you can try experimenting with ResNet-151 if you want to see what a really deep model looks like):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "resnet = torchvision.models.resnet18()\n",
    "print(resnet)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Batch Normalization\n",
    "\n",
    "This network contains yet another type of layer: `Batch normalization`. The idea of batch mnormalization is to bring values that flow thorugh the neural network to right interval. Usually neural networks work best when all values are in the range of [-1,1] or [0,1], and that is the reason that we scale/normalize our input data accordingly. However, during training of a deep network, it can happen that values get significantly out of this range, which makes training problematic. Batch normalization layer computes average and standards deviation for all values of the current minibatch, and uses them to normalize the signal before passing it through a neural network layer. This significantly improves the stability of deep networks.\n",
    "\n",
    "\n",
    "Using transfer learing, we were able to quickly put together a classifier for our custom object classification task, and achieve high accuracy. However, this example was not completely fair, because original VGG-16 network was pre-trained to recognize cats and dogs, and thus we were just reusing mosr of the patterns that were already presernt in the network. You can expect lower accuracy on more  exotic domain-specific obejcts, such as details on production line in a plant, or different tree leaves.\n",
    "\n",
    "You can see that more complex tasks taht we are solving now require higher computational power, and cannot be easily solved on the CPU."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "imp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}