object_detection_yolov4_pretrained_image.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "object_detection_yolov4_pretrained_image.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lF5rGAnOF1Gm"
      },
      "source": [
        "# Object Detection using the YOLO V4 pre-trained model\n",
        "\n",
        "*by Georgios K. Ouzounis, June 10th, 2021*\n",
        "\n",
        "In this exercise we will experiment with object detection in still images using the YOLO V4 pretrained model "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jB9dN1Io78DA"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tXzDYD5Jqxdo"
      },
      "source": [
        "# import the relevant libraries\n",
        "import numpy as np\n",
        "import cv2 # openCV"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ewrJSoXrGadW"
      },
      "source": [
        "# check the opencv version\n",
        "print(cv2.__version__)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pmXkC427Gkqx"
      },
      "source": [
        "# if the openCV version is < 4.4.0 update to the latest otherwise skip this step\n",
        "!pip install opencv-python==4.5.2.52"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Ms4LYX18B-1"
      },
      "source": [
        "## Get the model\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x1TwKBXhHcXu"
      },
      "source": [
        "# first create a directory to store the model\n",
        "%mkdir model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cY4WZZF1H1nZ"
      },
      "source": [
        "# enter the directory and download the necessary files \n",
        "%cd model\n",
        "!wget https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v3_optimal/yolov4.weights\n",
        "!wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/cfg/yolov4.cfg\n",
        "!wget https://raw.githubusercontent.com/AlexeyAB/darknet/master/data/coco.names\n",
        "%cd .."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U0u02442T7Bk"
      },
      "source": [
        "##  Get the test data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H1ocmIkmUDRP"
      },
      "source": [
        "# option 1: download a test image from the web\n",
        "!wget https://github.com/georgiosouzounis/object-detection-yolov4/raw/main/data/pretrained/people_bicycles.jpg"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_g-43RaiUWL7"
      },
      "source": [
        "#option 2: mount your Google Drive and select the image of your own\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6C5Qqhvg8Mpg"
      },
      "source": [
        "# copy a test file locally or use it directly from the mounted drive \n",
        "# customize this example for your drive structure\n",
        "%cp drive/MyDrive/object_detection/people_bicycles.jpg ."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zlGZYGMXWo_0"
      },
      "source": [
        "## Read test image"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pjmJxTYQ-06i"
      },
      "source": [
        "# read file\n",
        "test_img = cv2.imread('people_bicycles.jpg')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mge8rb5CCq1z"
      },
      "source": [
        "# display test image\n",
        "# import the cv2_imshow as a replacement of cv2.imshow to prevent errors\n",
        "from google.colab.patches import cv2_imshow\n",
        "cv2_imshow(test_img)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rALpx_t2YUN5"
      },
      "source": [
        "## Image2Blob\n",
        "\n",
        "convert the image to blob for model compatibility using the opencv builtin  dnn.blobFromImage() method\n",
        "\n",
        "Argument explanations:\n",
        "- scalefactor: multiplication factor for each pixel to rescale its intensity in  the range of [0,1]. No contrast stretching is performed. It is set to 1/255.0 = 0.003922.\n",
        "- new_size: rescaling size for network compatibility. We use (416, 416). Other supported sizes are (320, 320) and (609, 609). The greater the size is the better the prediction accuracy will be but at the cost of higher computational load.\n",
        "- swapRB: binary flag that if set instructs opencv to swap the red and blue channels. That is because opencv stores images in a BGR order but YOLO requires them in RGB.\n",
        "- crop: binary flag that if set instructs opencv to crop the image after resizing.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DGYR5dKX_fsJ"
      },
      "source": [
        "scalefactor = 1.0/255.0\n",
        "new_size = (416, 416)\n",
        "blob = cv2.dnn.blobFromImage(test_img, scalefactor, new_size, swapRB=True, crop=False)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NRoo5k-QdB5K"
      },
      "source": [
        "## Customize the YOLO detector\n",
        "\n",
        "class labels:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dvPiC9uxKrHk"
      },
      "source": [
        "class_labels_path = \"/content/model/coco.names\"\n",
        "class_labels = open(class_labels_path).read().strip().split(\"\\n\")\n",
        "class_labels"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XJu-efpG4f8X"
      },
      "source": [
        "bounding box color definitions; two options:\n",
        "\n",
        "1. create a template set of colors that is repeated till each class gets one assigned"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "80wm8OWX_oxr"
      },
      "source": [
        "# declare repeating bounding box colors for each class \n",
        "# 1st: create a list colors as an RGB string array\n",
        "# Example: Red, Green, Blue, Yellow, Magenda\n",
        "class_colors = [\"255,0,0\",\"0,255,0\",\"0,0,255\",\"255,255,0\",\"255,0, 255\"]\n",
        "\n",
        "#2nd: split the array on comma-seperated strings and for change each string type to integer\n",
        "class_colors = [np.array(every_color.split(\",\")).astype(\"int\") for every_color in class_colors]\n",
        "\n",
        "#3d: convert the array or arrays to a numpy array\n",
        "class_colors = np.array(class_colors)\n",
        "\n",
        "#4th: tile this to get 80 class colors, i.e. as many as the classes  (16rows of 5cols each). \n",
        "# If you want unique colors for each class you may randomize the color generation \n",
        "# or set them manually\n",
        "class_colors = np.tile(class_colors,(16,1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TivFzg45Ln_1"
      },
      "source": [
        "to visualize the effect of this code execute the next block"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T3cox1p5I5ol"
      },
      "source": [
        "def colored(r, g, b, text):\n",
        "    return \"\\033[38;2;{};{};{}m{} \\033[38;2;255;255;255m\".format(r, g, b, text)\n",
        "\n",
        "for i in range(16):\n",
        "  line = \"\"\n",
        "  for j in range(5):\n",
        "    class_id = i*5 + j\n",
        "    class_id_str = str(class_id)\n",
        "    text = \"class\" + class_id_str\n",
        "    colored_text = colored(class_colors[class_id][0], class_colors[class_id][1], class_colors[class_id][2], text)\n",
        "    line += colored_text\n",
        "  print(line)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PxNlQ1FO4mOI"
      },
      "source": [
        "2. or simply create random colors:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ItyWr4oX4pb8"
      },
      "source": [
        "class_colors = np.random.randint(0, 255, size=(len(class_labels), 3), dtype=\"uint8\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VHo_MgL40uma"
      },
      "source": [
        "## Load and run the model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WBTX_4xYAhZc"
      },
      "source": [
        "# Load the pre-trained model \n",
        "yolo_model = cv2.dnn.readNetFromDarknet('model/yolov4.cfg','model/yolov4.weights')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4cew-LCPDJOR"
      },
      "source": [
        "# Read the network layers/components. The YOLO V4 neural network has 379 components.\n",
        "# They consist of convolutional layers (conv), rectifier linear units (relu) etc.:\n",
        "model_layers = yolo_model.getLayerNames()\n",
        "print(\"number of network components: \" + str(len(model_layers))) \n",
        "print(model_layers)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CkBIXXd9E_lK"
      },
      "source": [
        "# extract the output layers\n",
        "\n",
        "# in the code that follows:\n",
        "# - model_layer[0]: returns the index of each output layer in the range of 1 to 379\n",
        "# - model_layer[0] - 1: corrects  this to the range of 0 to 378\n",
        "# - model_layers[model_layer[0] - 1]: returns the indexed layer name \n",
        "output_layers = [model_layers[model_layer[0] - 1] for model_layer in yolo_model.getUnconnectedOutLayers()]\n",
        "\n",
        "# YOLOv4 deploys the same YOLO head as YOLOv3 for detection with   \n",
        "# the anchor based detection steps, and three levels of detection granularity. \n",
        "print(output_layers)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fZaZEi9mDKld"
      },
      "source": [
        "# input pre-processed blob into the model\n",
        "yolo_model.setInput(blob)\n",
        "\n",
        "# compute the forward pass for the input, storing the results per output layer in a list\n",
        "obj_detections_in_layers = yolo_model.forward(output_layers)\n",
        "\n",
        "# verify the number of sets of detections\n",
        "print(\"number of sets of detections: \" + str(len(obj_detections_in_layers)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mgw3oGS2RXwc"
      },
      "source": [
        "Note: the yolo architecture offers 3 detection layers that generate feature maps of sizes 15x15x255, 30x30x255 and 60x60x255 respectively. Each detection layer is tasked with finding objects of a given size range.\n",
        "\n",
        "<img src=\"https://blog.roboflow.com/content/images/2020/06/image-19.png\" width=\"300\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I6hPes5MTnHc"
      },
      "source": [
        "## Analyze the results\n",
        "\n",
        "The objective now is to get each object detection from each output layer and evaluate  the algorithm's cofidence score against a threshold. For high confidence detections we extract the class ID and the bounding box info."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rsqLHbckDOH_"
      },
      "source": [
        "def object_detection_analysis(test_image, obj_detections_in_layers, confidence_threshold): \n",
        "\n",
        "  # get the image dimensions  \n",
        "  img_height = test_img.shape[0]\n",
        "  img_width = test_img.shape[1]\n",
        "\n",
        "  result = test_image.copy()\n",
        "  \n",
        "  # loop over each output layer \n",
        "  for object_detections_in_single_layer in obj_detections_in_layers:\n",
        "\t  # loop over the detections in each layer\n",
        "      for object_detection in object_detections_in_single_layer:  \n",
        "        # obj_detection[1]: bbox center pt_x\n",
        "        # obj_detection[2]: bbox center pt_y\n",
        "        # obj_detection[3]: bbox width\n",
        "        # obj_detection[4]: bbox height\n",
        "        # obj_detection[5]: confidence scores for all detections within the bbox \n",
        "\n",
        "        # get the confidence scores of all objects detected with the bounding box\n",
        "        prediction_scores = object_detection[5:]\n",
        "        # consider the highest score being associated with the winning class\n",
        "        # get the class ID from the index of the highest score \n",
        "        predicted_class_id = np.argmax(prediction_scores)\n",
        "        # get the prediction confidence\n",
        "        prediction_confidence = prediction_scores[predicted_class_id]\n",
        "    \n",
        "        # consider object detections with confidence score higher than threshold\n",
        "        if prediction_confidence > confidence_threshold:\n",
        "            # get the predicted label\n",
        "            predicted_class_label = class_labels[predicted_class_id]\n",
        "            # compute the bounding box cooridnates scaled for the input image \n",
        "            # scaling is a multiplication of the float coordinate with the appropriate  image dimension\n",
        "            bounding_box = object_detection[0:4] * np.array([img_width, img_height, img_width, img_height])\n",
        "            # get the bounding box centroid (x,y), width and height as integers\n",
        "            (box_center_x_pt, box_center_y_pt, box_width, box_height) = bounding_box.astype(\"int\")\n",
        "            # to get the start x and y coordinates we to subtract from the centroid half the width and half the height respectively \n",
        "            # for even values of width and height of bboxes adjacent to the  image border\n",
        "            #  this may genetrate a -1 which is prevented by the max() operator below  \n",
        "            start_x_pt = max(0, int(box_center_x_pt - (box_width / 2)))\n",
        "            start_y_pt = max(0, int(box_center_y_pt - (box_height / 2)))\n",
        "            end_x_pt = start_x_pt + box_width\n",
        "            end_y_pt = start_y_pt + box_height\n",
        "            \n",
        "            # get a random mask color from the numpy array of colors\n",
        "            box_color = class_colors[predicted_class_id]\n",
        "            \n",
        "            # convert the color numpy array as a list and apply to text and box\n",
        "            box_color = [int(c) for c in box_color]\n",
        "            \n",
        "            # print the prediction in console\n",
        "            predicted_class_label = \"{}: {:.2f}%\".format(predicted_class_label, prediction_confidence * 100)\n",
        "            print(\"predicted object {}\".format(predicted_class_label))\n",
        "            \n",
        "            # draw the rectangle and text in the image\n",
        "            cv2.rectangle(result, (start_x_pt, start_y_pt), (end_x_pt, end_y_pt), box_color, 1)\n",
        "            cv2.putText(result, predicted_class_label, (start_x_pt, start_y_pt-5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, box_color, 1)\n",
        "  return result\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I4unbfTsVMBb"
      },
      "source": [
        "confidence_threshold = 0.2\n",
        "result_raw = object_detection_analysis(test_img, obj_detections_in_layers, confidence_threshold)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nKvlX0mkLQaP"
      },
      "source": [
        "## Visualize the detections"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YT8vlllrD-vL"
      },
      "source": [
        "cv2_imshow(result_raw)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kXYCReuy5e7Q"
      },
      "source": [
        "## Non Maxima Suppression (NMS)\n",
        "\n",
        "supress fully- or partially- overlapping bounding boxes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2l-CLsIQ5c89"
      },
      "source": [
        "# step 1: declare lists for the arguments of interest: classID, bbox info, detection confidences\n",
        "class_ids_list = []\n",
        "boxes_list = []\n",
        "confidences_list = []"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C2-NTv1_yApP"
      },
      "source": [
        "The next 4 blocks of code area available as a single function in ```object_detections_functions.py```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HGwd6fOK6EkV"
      },
      "source": [
        "# step 2: populate those lists from the detections\n",
        "\n",
        "def object_detection_attributes(test_image, obj_detections_in_layers, confidence_threshold):\n",
        "  # get the image dimensions  \n",
        "  img_height = test_img.shape[0]\n",
        "  img_width = test_img.shape[1]\n",
        "  \n",
        "  # loop over each output layer \n",
        "  for object_detections_in_single_layer in obj_detections_in_layers:\n",
        "    # loop over the detections in each layer\n",
        "    for object_detection in object_detections_in_single_layer:  \n",
        "      # get the confidence scores of all objects detected with the bounding box\n",
        "      prediction_scores = object_detection[5:]\n",
        "      # consider the highest score being associated with the winning class\n",
        "      # get the class ID from the index of the highest score \n",
        "      predicted_class_id = np.argmax(prediction_scores)\n",
        "      # get the prediction confidence\n",
        "      prediction_confidence = prediction_scores[predicted_class_id]\n",
        "      \n",
        "      # consider object detections with confidence score higher than threshold\n",
        "      if prediction_confidence > confidence_threshold:\n",
        "        # get the predicted label\n",
        "        predicted_class_label = class_labels[predicted_class_id]\n",
        "        # compute the bounding box cooridnates scaled for the input image\n",
        "        bounding_box = object_detection[0:4] * np.array([img_width, img_height, img_width, img_height])\n",
        "        (box_center_x_pt, box_center_y_pt, box_width, box_height) = bounding_box.astype(\"int\")\n",
        "        start_x_pt = max(0, int(box_center_x_pt - (box_width / 2)))\n",
        "        start_y_pt = max(0, int(box_center_y_pt - (box_height / 2)))\n",
        "        \n",
        "        # update the 3 lists for nms processing\n",
        "        # - confidence is needed as a float \n",
        "        # - the bbox info has the openCV Rect format\n",
        "        class_ids_list.append(predicted_class_id)\n",
        "        confidences_list.append(float(prediction_confidence))\n",
        "        boxes_list.append([int(start_x_pt), int(start_y_pt), int(box_width), int(box_height)])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mM5t6MqX_V4p"
      },
      "source": [
        "populate the lists with our detections"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "voE1aEpo_WMm"
      },
      "source": [
        "score_threshold = 0.5\n",
        "object_detection_attributes(test_img, obj_detections_in_layers, score_threshold)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lvRtC6458vDl"
      },
      "source": [
        "compute the [NMS](https://docs.opencv.org/master/d6/d0f/group__dnn.html#ga9d118d70a1659af729d01b10233213ee)\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IeaKSqOC7HMA"
      },
      "source": [
        "# NMS for a set of overlapping bboxes returns the ID of the one with highest \n",
        "# confidence score while suppressing all others (non maxima)\n",
        "# - score_threshold: a threshold used to filter boxes by score \n",
        "# - nms_threshold: a threshold used in non maximum suppression. \n",
        "\n",
        "score_threshold = 0.5\n",
        "nms_threshold = 0.4\n",
        "winner_ids = cv2.dnn.NMSBoxes(boxes_list, confidences_list, score_threshold, nms_threshold)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "C7BuWUIEDD8H"
      },
      "source": [
        "# loop through the final set of detections remaining after NMS and draw bounding box and write text\n",
        "for winner_id in winner_ids:\n",
        "    max_class_id = winner_id[0]\n",
        "    box = boxes_list[max_class_id]\n",
        "    start_x_pt = box[0]\n",
        "    start_y_pt = box[1]\n",
        "    box_width = box[2]\n",
        "    box_height = box[3]\n",
        "    \n",
        "    #get the predicted class id and label\n",
        "    predicted_class_id = class_ids_list[max_class_id]\n",
        "    predicted_class_label = class_labels[predicted_class_id]\n",
        "    prediction_confidence = confidences_list[max_class_id]\n",
        "\n",
        "    #obtain the bounding box end co-oridnates\n",
        "    end_x_pt = start_x_pt + box_width\n",
        "    end_y_pt = start_y_pt + box_height\n",
        "    \n",
        "    #get a random mask color from the numpy array of colors\n",
        "    box_color = class_colors[predicted_class_id]\n",
        "    \n",
        "    #convert the color numpy array as a list and apply to text and box\n",
        "    box_color = [int(c) for c in box_color]\n",
        "    \n",
        "    # print the prediction in console\n",
        "    predicted_class_label = \"{}: {:.2f}%\".format(predicted_class_label, prediction_confidence * 100)\n",
        "    print(\"predicted object {}\".format(predicted_class_label))\n",
        "    \n",
        "    # draw rectangle and text in the image\n",
        "    cv2.rectangle(test_img, (start_x_pt, start_y_pt), (end_x_pt, end_y_pt), box_color, 1)\n",
        "    cv2.putText(test_img, predicted_class_label, (start_x_pt, start_y_pt-5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, box_color, 1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v5zjbPZ7DSur"
      },
      "source": [
        "cv2_imshow(test_img)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}