GradientDescent.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RvidLkZCc4A2"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/porterjenkins/CS180/blob/main/data_science_labs/data_science_lab_8.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a><p><b>After clicking the \"Open in Colab\" link, copy the notebook to your own Google Drive before getting started, or it will not save your work</b></p>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k0NimIjkdTzz"
      },
      "source": [
        "# BYU CS 180 Lab 8"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gHbfWVe-dMYz"
      },
      "source": [
        "## Introduction:\n",
        "For this assignment, you will build a simple univariate linear regression model to predict profits for a food truck."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7NewZD37dbrM"
      },
      "source": [
        "Download the data that you will use by running this command, and read it in using:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "xbrM8d_3b91Y",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "95afb71f-4759-4f2b-ded3-a63b32651f19"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "--2023-03-08 18:55:42--  https://raw.githubusercontent.com/porterjenkins/cs180-intro-data-science/master/data/food-truck.csv\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 1359 (1.3K) [text/plain]\n",
            "Saving to: ‘food-truck.csv’\n",
            "\n",
            "\rfood-truck.csv        0%[                    ]       0  --.-KB/s               \rfood-truck.csv      100%[===================>]   1.33K  --.-KB/s    in 0s      \n",
            "\n",
            "2023-03-08 18:55:42 (67.2 MB/s) - ‘food-truck.csv’ saved [1359/1359]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "\n",
        "!wget https://raw.githubusercontent.com/porterjenkins/cs180-intro-data-science/master/data/food-truck.csv\n",
        "data = pd.read_csv(\"food-truck.csv\", header=None, names=['X', 'Y'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h5eqHpQ9yx7Q"
      },
      "source": [
        "We will restrict ourselves to a linear hypothesis space, constructing a model that adheres to the following form:\n",
        "$$ f _\\Theta(x) = \\theta _0 + \\theta _1x $$\n",
        "\n",
        "You might notice that this equation is similar to the linear equation: $$ f(x) = b + mx $$\n",
        "\n",
        "(Yes you did use y=mx+b after 8th grade 😜)\n",
        "\n",
        "\n",
        "In this lab, you will be writing a machine learning model that learns/approximates 2 parameters. The first one is $\\theta_0$ which represents learning *b* (the bias or the intercept). The second one is $\\theta_1$ which represents learning *m* (the weight, or the slope). This type of machine learning model is traditionally called Least Squares model."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jDiIwcCh1Owd"
      },
      "source": [
        "Given data, your goal will be to estimate the parameters of this model using the method of steepest gradient descent. The parameters are defined within the construct:\n",
        "$$ \\theta_p = \\{\\theta_0,\\theta_1\\} $$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jc-wWrUc12NG"
      },
      "source": [
        "which is the vector of learnable coefficients that weight the observed variables, and where $\\theta_1$ is a single bias coefficient. We can learn these parameters by minimizing average squared error. Thus, the loss function you will want to implement is:\n",
        "\n",
        "Equation 1: $$ L(\\Theta) = \\frac{1}{2m}\\sum_{i=1}^m (f(x^i)-y^i)^2 $$\n",
        "\n",
        "This equation may look very scary at first, but it's really not that scary, so let's break it down, and define our variables.\n",
        "*   m is the number of datapoints in the data set.\n",
        "*   i is the index of the data point tuple that we're looking at in the sum.\n",
        "*   L is the loss function (think of it like f(x)).\n",
        "*   $\\Theta$ is the list of parameters to estimate.\n",
        "*   $(f(x^i)-y^i)^2$ is the squared difference between the predicted output and the actual output.\n",
        "\n",
        "So wrapping it all together, the loss function is taking the sum of the squared differences, and dividing it by double the number of datapoints in the dataset.\n",
        "\n",
        "The goal of this lab is to minimize the loss function, because that means that our predicted values and the actual values are getting closer together. (Quick thought exercise, ask yourself why we want our predicted values to get closer to the actual values)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LiYPG03jKvWQ"
      },
      "source": [
        "## Exercise 1: The Dark Descent (Compute Loss)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gp6q-MueNuPh"
      },
      "source": [
        "In order to correctly recreate the gradient descent algorithm, you need to compute the loss function. Use the equation for loss given in the introduction to fill out the functions below to correctly produce the right loss."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "vVDwIWcJOegx"
      },
      "outputs": [],
      "source": [
        "def predict(X, theta):\n",
        "  # TODO: Write the code to output the predicted y values based on the X values\n",
        "  # Hint: Remember that theta is a tuple consisting of theta_0 and theta_1\n",
        "  y_predictions = theta[0] + theta[1]*X\n",
        "  return y_predictions\n",
        "\n",
        "def sum_mean_squared_error(y, y_hat):\n",
        "  # TODO: Write the sum of the mean squared error\n",
        "  # Hint: Follow the loss equation\n",
        "  mean_squared_error = np.sum(np.power((y - y_hat), 2))\n",
        "  return mean_squared_error\n",
        "\n",
        "def calculate_loss(X, y, theta):\n",
        "\t# TODO: Write your compute loss function below\n",
        "  # Hint: You'll use the predict and the sum_mean_squared_error functions defined above\n",
        "  y_hat = predict(X, theta)\n",
        "  m = y.shape[0]\n",
        "  loss = (1/(2*m))*sum_mean_squared_error(y, y_hat)\n",
        "  return loss"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xAUhDKl1QkxU"
      },
      "source": [
        "Explain to me like I'm five what the code above is doing:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kc2Dc_tzQsLz"
      },
      "source": [
        "> the predict(X, theta) function takes in an X value and theta which is a list of (x,y) pairs except in this case they are represented as (theta_0, theta_1). Predict calculates all the y prediction values by adding the first theta values to the second theta values multipled by the x value. This is how we get y_hat. sum_mean_squared_error uses a y value and the y prediction to find the difference of those, squared, and summed and returns the error value. calculate_loss is the function that combines what these did to return to final loss function"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LzsatAR4GGUq"
      },
      "source": [
        "## Exercise 2: Partial Derivatives"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5zGn1zlbGK56"
      },
      "source": [
        "Analytically derive the gradient of the loss function with respect to the model parameters, $\\theta_0$ and $\\theta_1$. (take the partial derivative with respect to the given parameters)\n",
        "\n",
        "Hint: You will need to find these two derivatives in order to calculate the gradient (L is just a less fancy version of the L(θ) above):\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ye_vjnFKGiR-"
      },
      "source": [
        "Give the partial derivative for the parameter $\\theta_0$ below:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Csf84DuKGw_o"
      },
      "source": [
        "> (1/m)*(y_hat - y)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4_Dg07RwGrgn"
      },
      "source": [
        "Give the partial derivative for the parameter $\\theta_1$ below:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jvD2WWiWKid0"
      },
      "source": [
        "> (1/m)*(y_hat - y)*X"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C82MvdtJK-si"
      },
      "source": [
        "Using the two equations above, fill out the gradient calculation function below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "INIE_kqpK7sm"
      },
      "outputs": [],
      "source": [
        "def calculate_gradient(X,y, theta):\n",
        "  # TODO: The gradient with respect to the bias (the y intercept)\n",
        "  y_hat = predict(X, theta)\n",
        "  m = y.shape[0]\n",
        "  dL_d0 = (1/m)*sum((y_hat - y))\n",
        "  # TODO: The gradient with respect to the weight (the slope)\n",
        "  dL_d1 = (1/m)*sum((y_hat - y)*X)\n",
        "  \n",
        "  # nabla represents the full gradient, or a vector of the partial derivatives\n",
        "  nabla = (dL_d0, dL_d1)\n",
        "  return nabla"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x3P97BfsQ3JB"
      },
      "source": [
        "## Exercise 3: Hold The Line (Training Your Least Squared Algorithm)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IY0BRQMpRlC-"
      },
      "source": [
        "### There are a couple of hyperparameters that you should know about before training your algorithm:\n",
        "\n",
        "#### Epochs / Iterations:\n",
        "Epochs is the number of times that the algorithm runs through the dataset. For example, if the epoch count is 500, then the algorithm with iterate through the dataset 500 times.\n",
        "\n",
        "#### Learning Rate / Step Size:\n",
        "Learning rate (traditionally denoted as α or alpha) is the rate at which your algorithm learns. You can also think of it as the step size for updating the weight and bias ($\\theta_0$ and $\\theta_1$). If the step size is too large, the loss value will blow up and never converge. If the learning rate is too small, the algorithm will take forever to converge, and won't be an efficient use of your time. If you have a smaller learning rate, I recommend using a higher epoch count.\n",
        "\n",
        "#### Batch Size:\n",
        "You won't use batch size for this lab, but I still feel like it's an important concept to learn. Batch size is splitting the dataset up into smaller chunks and then feeding those smaller chuncks of data into the ML model. When it comes to larger models such as Neural Networks, it's important to have the right batch size otherwise your computer may run out of memory.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3X5_vs4rrJYu"
      },
      "source": [
        "### Now that you understand the hyperparameters, let's train our model!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JUtKOGZOGxqF"
      },
      "source": [
        "You will update the current value for each theta with the function:\n",
        "\n",
        "Equation 2:\n",
        "$$\\theta_n = \\theta_p - α \\frac{dL(\\Theta)}{d\\theta_p}$$\n",
        "\n",
        "Again, there are some scary letters, but that's not to worry, I'm going to break them down for you below.\n",
        "\n",
        "* $\\theta_n$ is the updated parameter tuple.\n",
        "* $\\theta_p$ is the previous parameter tuple.\n",
        "* α is the learning rate. (I recommend a small value for this, like 0.1, 0.01, or 0.001. Once the lab is finished you should try different values for your learning rate and see how it changes the convergence of the parameters).\n",
        "* $\\frac{dL(\\Theta)}{d\\theta_p}$ is the gradient (vector of partial derviatives) that you'll be updating.\n",
        "\n",
        "Putting it all together, we have the previous parameters being subtracted from the gradient multiplied by a learning rate.\n",
        "\n",
        "Attempt to explain why/how this function uses the loss function minimize the parameters. (Credit will be given at an attempt of an explanation, not based on correctness):"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sdCxSZAcKsS9"
      },
      "source": [
        "> this function uses the derivative of the loss function to minimize the difference between the updated and previous values to reach a value of 0. When the rate is 0 that means the loss is unchanging. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oxx63wSgZVLJ"
      },
      "source": [
        "Fill out the TODO parts of the training function below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "7EFH9H1cRhSX"
      },
      "outputs": [],
      "source": [
        "def train_model(dataframe, epochs, learning_rate):\n",
        "  # Get the X and y values of the column\n",
        "  X = dataframe.iloc[:, 0]\n",
        "  y = dataframe.iloc[:, 1]\n",
        "\n",
        "  # Turn the X and y values into numpy arrays\n",
        "  X = np.array(X.values)\n",
        "  y = np.array(y.values)\n",
        "\n",
        "  # convert to numpy arrays and initalize the parameter array theta\n",
        "  w = np.zeros(1)\n",
        "  b = np.zeros(1)\n",
        "  theta = (b, w)\n",
        "\n",
        "  # TODO: calculate the initial loss\n",
        "  initial_L = calculate_loss(X, y, theta)\n",
        "\n",
        "  # Initialize the list of loss values\n",
        "  loss_values = [initial_L]\n",
        "\n",
        "  # Initialize I to 0\n",
        "  i = 0\n",
        "\n",
        "  while i < epochs:\n",
        "    # TODO: Calculate Gradient\n",
        "    dL_db, dL_dw = calculate_gradient(X, y, theta)\n",
        "    t_0 = theta[0]\n",
        "    t_1 = theta[1]\n",
        "\n",
        "    # TODO: update theta with respect to the calculated gradient\n",
        "    # Hint: use the equation above\n",
        "    updated_t_0 = t_0 - learning_rate*calculate_gradient(X, y, theta)[0]\n",
        "    updated_t_1 = t_1 - learning_rate*calculate_gradient(X, y, theta)[1]\n",
        "\n",
        "    theta = (updated_t_0, updated_t_1)\n",
        "\n",
        "    # TODO: Calculated new loss using the updated theta values, and add it to the loss_values list\n",
        "    L = calculate_loss(X, y , theta)\n",
        "    loss_values.append(L)\n",
        "\n",
        "    # Update I\n",
        "    i += 1\n",
        "    \n",
        "  return (loss_values, theta)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "_sv8tD_aYaPK"
      },
      "outputs": [],
      "source": [
        "def plot_line(dataframe, theta, epoch, learning_rate):\n",
        "  # This function will plot your model based on some sampled values\n",
        "  # There is no need to change this function, you'll just need to call it\n",
        "  \n",
        "  X = dataframe.iloc[:, 0]\n",
        "  y = dataframe.iloc[:, 1]\n",
        "\n",
        "  kludge = 0.25\n",
        "  X_test = np.linspace(data.X.min(), data.X.max(), 100)\n",
        "  X_test = np.expand_dims(X_test, axis=1)\n",
        "  \n",
        "  plt.plot(X_test, predict(X_test, theta), label=\"Model\")\n",
        "  plt.title(f\"Value for {epoch} epochs and {learning_rate} step\")\n",
        "  plt.scatter(X, y, edgecolor='g', s=20, label=\"Samples\")\n",
        "  plt.xlabel(\"x\")\n",
        "  plt.ylabel(\"y\")\n",
        "  plt.xlim((np.amin(X_test) - kludge, np.amax(X_test) + kludge))\n",
        "  plt.ylim((np.amin(y) - kludge, np.amax(y) + kludge))\n",
        "  plt.legend(loc=\"best\")\n",
        "  plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oYajIvGMZr8j"
      },
      "source": [
        "Use the two functions above to do plot the models after 5, 100, 1000, and 10,000 epochs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IsijuvK4ZSCZ"
      },
      "outputs": [],
      "source": [
        "# Write the code to plot the line for the following cases:\n",
        "# 5 Epochs\n",
        "loss, theta = train_model(data,5, 0.001)\n",
        "plot_line(data,theta, 5, 0.001)\n",
        "\n",
        "# 100 Epochs\n",
        "loss, theta = train_model(data,100, 0.001)\n",
        "plot_line(data,theta, 100, 0.001)\n",
        "# 1000 Epochs\n",
        "loss, theta = train_model(data,1000, 0.001)\n",
        "plot_line(data,theta, 1000, 0.001)\n",
        "# 10,000 Epochs\n",
        "loss, theta = train_model(data,10000, 0.001)\n",
        "plot_line(data,theta, 10000, 0.001)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9UuayB93aVOJ"
      },
      "source": [
        "Using the results after 10,000 epochs, plot the loss values with respect to epochs using a line graph (epochs on the x axis and loss values on the y axis)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "id": "H7d77l9HaTEo",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 283
        },
        "outputId": "24bc0877-df8b-4db3-bf61-49a65ef35674"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[<matplotlib.lines.Line2D at 0x7f8cd4ea9e80>]"
            ]
          },
          "metadata": {},
          "execution_count": 31
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXEAAAD4CAYAAAAaT9YAAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAAsTAAALEwEAmpwYAAAVVUlEQVR4nO3deYwkZ33G8efp7jn2PuzxsvGRNWAR+Y+wJiPLFiginI4VBYhQhBXBBhwZJUGCBCWy4Y9AFCkQcSQoCWCwwYrAnCa2LBJiHCMHKTJZg4+11xvbYIKX9c74ml3v7lzdv/xRb892zdR4Zmd6tued/X6kVlW99XbVW1MzT9e8dbQjQgCAPNV63QAAwNIR4gCQMUIcADJGiANAxghxAMhY43Su7Oyzz45du3adzlUCQPbuvffepyNiqGreaQ3xXbt2ae/evadzlQCQPds/n28e3SkAkDFCHAAyRogDQMYIcQDIGCEOABkjxAEgY4Q4AGQsixC/c/9h/fMPHut1MwBg1ckixH9wYFRf/K+f9boZALDqZBHiksSXVwDAXFmEuN3rFgDA6pRFiAMAqmUT4nSmAMBcWYQ4vSkAUC2LEJckzmsCwFxZhLg5swkAlbIIcYlLDAGgSjYhDgCYK5sQ5zgcAObKIsTpEgeAaguGuO1B2z+yfb/th2x/NJVfaPse24/Z/rrt/pVvLgCg02KOxCckvS4iXilpt6QrbF8m6eOSPh0RL5f0nKSrV6yVEv0pAFBhwRCPwgtpsi+9QtLrJH0rld8k6a0r0UBJMrf7AEClRfWJ267bvk/SiKQ7JD0u6fmImE5VnpR07jzvvcb2Xtt7R0dHl9xQDsQBYK5FhXhENCNit6TzJF0q6dcWu4KIuD4ihiNieGhoaEmN5MQmAFQ7patTIuJ5SXdJulzSVtuNNOs8SQe727Q5617JxQNAlhZzdcqQ7a1pfJ2kN0raryLM356q7ZF06wq1kR5xAJhHY+Eq2inpJtt1FaH/jYi43fbDkr5m+28k/UTSDSvYTvrEAaDCgiEeEQ9IuqSi/Kcq+sdXHH3iAFAtizs2AQDVsglxzmsCwFxZhDjPEweAalmEuCQFpzYBYI4sQpzjcAColkWIS/SJA0CVPEKcQ3EAqJRHiAMAKmUT4vSmAMBcWYQ4zxMHgGpZhLgkDsUBoEIWIc69PgBQLYsQl7jZBwCqZBHiHIgDQLUsQlziZh8AqJJFiNMnDgDVsghxAEC1bEKc3hQAmCuLEOdmHwColkWIS1JwZhMA5sgixDmxCQDVsghxiT5xAKiSRYhzIA4A1bIIcYmbfQCgSh4hTqc4AFTKI8QBAJUIcQDI2IIhbvt823fZftj2Q7bfn8o/Yvug7fvS68qVaiSdKQBQrbGIOtOSPhgRP7a9SdK9tu9I8z4dEZ9YueaVRYRM/zgAzFgwxCPikKRDafyo7f2Szl3phnUitwGg2in1idveJekSSfekovfZfsD2jba3zfOea2zvtb13dHR0WY3lMkMAKFt0iNveKOnbkj4QEUckfVbSyyTtVnGk/smq90XE9RExHBHDQ0NDS2okD8ACgGqLCnHbfSoC/CsRcYskRcThiGhGREvSFyRdunLNBABUWczVKZZ0g6T9EfGpjvKdHdXeJmlf95tXRm8KAJQt5uqUV0t6p6QHbd+Xyj4k6Srbu1Vk6xOS3rsC7ZPEiU0AmM9irk75oaov1f5u95uzYFvmaQoAnJmyuGOT2AaAalmEeBt94gBQlkWI0ycOANWyCPE2bvYBgLIsQpznpQBAtSxCHABQLasQD05tAkBJViEOACjLKsQ5sQkAZVmEOOc1AaBaFiEOAKiWRYjzPHEAqJZFiLfRJw4AZVmEOH3iAFAtixAHAFTLKsS52QcAyrIIcXpTAKBaFiHexolNACjLIsQ5sQkA1bII8TYOxAGgLIsQ52YfAKiWRYgDAKplFeLBmU0AKMkixDmxCQDVsgjxNo7DAaAsqxAHAJRlFeJ0iQNA2YIhbvt823fZftj2Q7bfn8q3277D9qNpuG2lGmk6xQGg0mKOxKclfTAiLpZ0maQ/tX2xpGsl3RkRF0m6M02vLI7EAaBkwRCPiEMR8eM0flTSfknnSnqLpJtStZskvXWF2sitPgAwj1PqE7e9S9Ilku6RtCMiDqVZT0naMc97rrG91/be0dHR5bQVADDLokPc9kZJ35b0gYg40jkvirtwKjs7IuL6iBiOiOGhoaFlNZbniQNA2aJC3HafigD/SkTckooP296Z5u+UNLIyTeRmHwCYz2KuTrGkGyTtj4hPdcy6TdKeNL5H0q3db14ZlxgCQFljEXVeLemdkh60fV8q+5Ckj0n6hu2rJf1c0u+vSAvFiU0AmM+CIR4RP9T8Ofr67jZngbaczpUBQAayuGOTm30AoFoWId7Go2gBoCyLEOdAHACqZRHiAIBqWYU4nSkAUJZFiNObAgDVsgjxNs5rAkBZHiHOmU0AqJRHiCc8AAsAyrIIcY7DAaBaFiEOAKiWV4jTmwIAJVmEOOc1AaBaFiHexoE4AJRlEeLm1CYAVMoixNu42QcAyrIIcfrEAaBaFiHexs0+AFCWRYhzIA4A1bIIcQBAtaxCnBObAFCWRYhzYhMAqmUR4m0ciANAWRYhzs0+AFAtixBvCzrFAaAkjxDnQBwAKuUR4gkH4gBQtmCI277R9ojtfR1lH7F90PZ96XXlSjaSA3EAqLaYI/EvS7qiovzTEbE7vb7b3WYBABZjwRCPiLslPXsa2gIAOEXL6RN/n+0HUnfLtvkq2b7G9l7be0dHR5e0InO3DwBUWmqIf1bSyyTtlnRI0ifnqxgR10fEcEQMDw0NLXF17WUt6+0AsOYsKcQj4nBENCOiJekLki7tbrPKOA4HgGpLCnHbOzsm3yZp33x1u4nniQNAWWOhCrZvlvRaSWfbflLSX0l6re3dKh5n8oSk965cE3kAFgDMZ8EQj4irKopvWIG2LIg+cQAoy+qOTQBAWRYhTncKAFTLIsTb6E0BgLIsQpzniQNAtSxCvI3niQNAWRYhTp84AFTLIsTbOA4HgLKsQhwAUEaIA0DGsgpxzmsCQFkWIc7zxAGgWh4hnoZcYggAZXmEeEpxIhwAyrII8VpKcQ7EAaAskxAvhi1SHABKsgjx9olNQhwAyvII8TQkwwGgLIsQr3EkDgCV8gjx1EoyHADKsghx+sQBoFoWIX6yO6XHDQGAVSaLEOeOTQColkWIz9zs0+N2AMBqk0mIF8MW/SkAUJJFiJs+cQColEmIF0P6xAGgbMEQt32j7RHb+zrKttu+w/ajabhtRRtJnzgAVFrMkfiXJV0xq+xaSXdGxEWS7kzTK4YHYAFAtQVDPCLulvTsrOK3SLopjd8k6a3dbVYZfeIAUG2pfeI7IuJQGn9K0o4utacSR+IAUG3ZJzajONs4b7ravsb2Xtt7R0dHl7QOz3wpBCEOAJ2WGuKHbe+UpDQcma9iRFwfEcMRMTw0NLSkldVmrk5Z0tsBYM1aaojfJmlPGt8j6dbuNKcaz04BgGqLucTwZkn/LekVtp+0fbWkj0l6o+1HJb0hTa8Y0ycOAJUaC1WIiKvmmfX6LrdlXhZ94gBQJYs7NvlSCAColkeI0ycOAJUyCfFiSJ84AJRlEeLtr4UgxAGgLIsQ5zpxAKiWSYi3n2JIigNAp6xCvNXqcUMAYJXJIsS52QcAqmUV4mQ4AJRlEeL0iQNAtaxCnJt9AKAskxAvhk1SHABKsgjxekpxQhwAyrII8f5G0czJaa4xBIBOeYV4kxAHgE55hHi9aOYER+IAUJJFiNtWf71GdwoAzJJFiEtSX92EOADMkk2I9zdqmmw2e90MAFhV8gpxjsQBoIQQB4CMZRPiW9f167njU71uBgCsKtmE+I7NAzp8ZLzXzQCAVaXR6wYs1svP2aTv7x/RX3zzfp2/fb1esmVQO7cMaueWddq5ZVAbBrLZFADommyS7z2v2aUDTx3RXQdG9PQLk3Pmbx5sFIG+9WS4v2TLoF6yeVDnbB7Q0MYBbVvfr1r7aVoAsAZkE+LnbBrUl959qSRpfKqpkSMT+uXYCT01Nq5DY+M6NHZiZrjv4Fhl0Ddq1tkbB2ZCvT0c2jxYDDcN6JxNxXCwr366NxEATlk2Id5psK+uC85arwvOWj9vnYnppg6PTeipI+MaPTqhkaPt4YRGj07ol2Pjuv/JMT1zbKLyG4M2DTS0fWO/tq3v11kb+rW947Vtw9yyjQMN2RzlAzi9lhXitp+QdFRSU9J0RAx3o1HdMNBYOOglabrZ0rPHJmfCfeTouEaOTOiZY5N6Nr0OjY3roV8e0bPHJud9CFd/vaZtG/q0bX0R6lvW9c28Nqfh1vV9pfIt6/q0abBv5lG7AHCqunEk/lsR8XQXltMTjXpN52we1DmbBxesGxE6NtnUc8cmU8hP6NljU3OGzx2f1KMjL2jsxJTGTky96PXttrRxoDEn5DcN9GnjYEMbBxraNNjQhoFifONgQ5vScONAQ5sG+rRhoK5GPZsLjQB0UZbdKb1iuwjSgYbO3/7iR/idxqeaM4E+dmJKY8en9HzH9JHOeSemdPjIC3phfFovTBSvxVjXVy8F/Ib+9rCudf0Nre+vp1cxvm5muq51fSfnr+uoM9Co0UUErHLLDfGQ9B+2Q9LnI+L62RVsXyPpGkm64IILlrm6PA321TXYV9eORRztz9ZqhY5NpkAfn9bRiWkd6xjvDPuj7fHxKb0wMa1fPHtcxyebOj7Z1InJaR2falb2/8+n5uLDYV1/QxsG6mm8rsFGXYN9NQ3MHvbVNdgohgON8vRgo6bBVD7YV9dAX02DjblDrh4CTs1yQ/w1EXHQ9jmS7rD9SETc3VkhBfv1kjQ8PMz3q52iWs3aNFj0nWvL8pYVEZqYbqVgn9aJFPDHOsZPpHnHp5ozZTMfApNNnZhqamKqpWeOTWpiqqXx6ebMcHyqqYnp1il9UMxWrxWPHe6rW/2NuvrrVl+jlspq6mvUNFCvqa9h9dVTeZrf31leKqt1lBXz+9I66rWaGnWrUbMa843Xa2lo1WtWX62mer0YtuvwHwt6ZVkhHhEH03DE9nckXSrp7hd/F3rF9sx/Bds39K/IOiJCk82WJqZbRahPtTQx3dR4xbAd+p3DqWZLU83Q5HRLk82WptrDZiuVhaamWxqfauno+PTJes2Wpqaj9J7J5vI+UE5FvdYO+DSs10rD2R8O9ZpVt1VLw3qtPV4sq1Yqc0fZrPlpWJrfsdxabfa6VFru3PVbNRe/K53Dmi1XTNdsWcXBRru+VZSX6rhdlup0TNfSB2B7GTPr0snpWlFQmi7ac3K6vewzzZJD3PYGSbWIOJrG3yTpr7vWMmTJtgYadQ006to82NfTtkSEmq0ofSi0Q36q2dJ0KzTdDE235hlv12m10rz0arZmlttstdIwNNVqqTlTr+M9pXUV85qtmHlNt1qamA41o+g+a7ZCrdT2ZkRRFqFWS3PKmq1Z81PZmSzlfekDRTNlxYfD7DrqnJ41z6lCu7w2q87MOhdY9t/+3q/r0gu3d317l3MkvkPSd9JGNCR9NSL+vSutArrATkfAdWld/5l181Yp5GfCXnPKSvMj1GxJoaJuK0IRaTra00VZe3qmzuzpjmW0ovhADc2ajpPTVcueeU/rZB1p9nuKZbU6lqX2OtRuS3pvGo84WR4d9aST7YxZ7y/mn2zz7PeXlj1TfnJaIW0YWJnfwSWHeET8VNIru9gWAF1Sq1k1Wdx4vPZxcTEAZIwQB4CMEeIAkDFCHAAyRogDQMYIcQDIGCEOABkjxAEgY47T9XAJSbZHJf18iW8/W1K2zy1fIrb5zMA2nxmWs82/GhFDVTNOa4gvh+29q+mbg04HtvnMwDafGVZqm+lOAYCMEeIAkLGcQnzOtwadAdjmMwPbfGZYkW3Opk8cADBXTkfiAIBZCHEAyFgWIW77CtsHbD9m+9pet2epbJ9v+y7bD9t+yPb7U/l223fYfjQNt6Vy2/5M2u4HbL+qY1l7Uv1Hbe/p1TYtlu267Z/Yvj1NX2j7nrRtX7fdn8oH0vRjaf6ujmVcl8oP2H5zjzZlUWxvtf0t24/Y3m/78rW+n23/Wfq93mf7ZtuDa20/277R9ojtfR1lXduvtn/D9oPpPZ+xF/GloTHztUir8yWpLulxSS+V1C/pfkkX97pdS9yWnZJelcY3SfpfSRdL+jtJ16byayV9PI1fKenfVHxF32WS7knl2yX9NA23pfFtvd6+Bbb9zyV9VdLtafobkt6Rxj8n6Y/T+J9I+lwaf4ekr6fxi9O+H5B0YfqdqPd6u15ke2+S9EdpvF/S1rW8nyWdK+lnktZ17N8/XGv7WdJvSnqVpH0dZV3br5J+lOo6vfe3F2xTr38oi/ihXS7pex3T10m6rtft6tK23SrpjZIOSNqZynZKOpDGPy/pqo76B9L8qyR9vqO8VG+1vSSdJ+lOSa+TdHv6BX1aUmP2Ppb0PUmXp/FGqufZ+72z3mp7SdqSAs2zytfsfk4h/osUTI20n9+8FvezpF2zQrwr+zXNe6SjvFRvvlcO3SntX462J1NZ1tK/j5dIukfSjog4lGY9peJLqKX5tz23n8nfS/pLSa00fZak5yNiOk13tn9m29L8sVQ/p22+UNKopC+lLqQv2t6gNbyfI+KgpE9I+j9Jh1Tst3u1tvdzW7f267lpfHb5i8ohxNcc2xslfVvSByLiSOe8KD6C18x1n7Z/R9JIRNzb67acRg0V/3J/NiIukXRMxb/ZM9bgft4m6S0qPsB+RdIGSVf0tFE90Iv9mkOIH5R0fsf0eaksS7b7VAT4VyLillR82PbONH+npJFUPt+25/QzebWk37X9hKSvqehS+QdJW203Up3O9s9sW5q/RdIzymubn5T0ZETck6a/pSLU1/J+foOkn0XEaERMSbpFxb5fy/u5rVv79WAan13+onII8f+RdFE6y92v4iTIbT1u05KkM803SNofEZ/qmHWbpPYZ6j0q+srb5e9KZ7kvkzSW/m37nqQ32d6WjoDelMpWnYi4LiLOi4hdKvbdf0bEH0i6S9LbU7XZ29z+Wbw91Y9U/o50VcOFki5ScRJo1YmIpyT9wvYrUtHrJT2sNbyfVXSjXGZ7ffo9b2/zmt3PHbqyX9O8I7YvSz/Dd3Usa369PkmwyBMJV6q4kuNxSR/udXuWsR2vUfGv1gOS7kuvK1X0Bd4p6VFJ35e0PdW3pH9K2/2gpOGOZb1H0mPp9e5eb9sit/+1Onl1yktV/HE+JumbkgZS+WCafizNf2nH+z+cfhYHtIiz9j3e1t2S9qZ9/a8qrkJY0/tZ0kclPSJpn6R/UXGFyZraz5JuVtHnP6XiP66ru7lfJQ2nn9/jkv5Rs06OV7247R4AMpZDdwoAYB6EOABkjBAHgIwR4gCQMUIcADJGiANAxghxAMjY/wPBR4RKaDElMgAAAABJRU5ErkJggg==\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# Write the code to plot the loss values for each of the 10,000 iterations:\n",
        "epochs = [item for item in range(0,10001)]\n",
        "plt.plot(epochs,loss)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ke029wcPasAb"
      },
      "source": [
        "## Exercise 4: Discussion"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3yT8gXHhau04"
      },
      "source": [
        "Since learning a linear model is a convex optimization problem, you should see the model converge to a low mean squared error. However, you will need to tune the learning rate/step size, α (bear in mind that values that are too big will result in divergence).\n",
        "Write a few sentences describing what you learned from the training/model fitting process. \n",
        "\n",
        "Things to discuss: What happens when you change the step size α? How many epochs did you need to converge on a reasonable solution (for any given step size)?\n",
        "\n",
        "You should also discuss the hyperparameters you tried, and which ones worked best. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tKUndHnBa1C0"
      },
      "source": [
        ">when you change the size of the learning rate, the prediction line becomes more accurate as alpha increases. I needed my epochs size to be about 0.001 for my function to converge to a reasonable solution. "
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.0"
    },
    "vscode": {
      "interpreter": {
        "hash": "921445e9336842bf3da7ce0cce684b71b836ed2d0c3aed21b6777cfb0e4f3429"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}