notebooks/02_numerical_pipeline_introduction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# First model with scikit-learn\n",
    "\n",
    "In this notebook, we present how to build predictive models on tabular\n",
    "datasets, with only numerical features.\n",
    "\n",
    "In particular we will highlight:\n",
    "\n",
    "* the scikit-learn API: `.fit(X, y)`/`.predict(X)`/`.score(X, y)`;\n",
    "* how to evaluate the generalization performance of a model with a train-test\n",
    "  split.\n",
    "\n",
    "## Loading the dataset with Pandas\n",
    "\n",
    "We will use the same dataset \"adult_census\" described in the previous\n",
    "notebook. For more details about the dataset see\n",
    "<http://www.openml.org/d/1590>.\n",
    "\n",
    "Numerical data is the most natural type of data used in machine learning and\n",
    "can (almost) directly be fed into predictive models. We will load a\n",
    "subset of the original data with only the numerical columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census-numeric.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the first records of this dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult_census.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that this CSV file contains all information: the target that we would\n",
    "like to predict (i.e. `\"class\"`) and the data that we want to use to train\n",
    "our predictive model (i.e. the remaining columns). The first step is to\n",
    "separate columns to get on one side the target and on the other side the\n",
    "data.\n",
    "\n",
    "## Separate the data and the target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_name = \"class\"\n",
    "target = adult_census[target_name]\n",
    "target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = adult_census.drop(columns=[target_name, ])\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now linger on the variables, also denominated features, that we will\n",
    "use to build our predictive model. In addition, we can also check how many\n",
    "samples are available in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The dataset contains {data.shape[0]} samples and \"\n",
    "      f\"{data.shape[1]} features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fit a model and make predictions\n",
    "\n",
    "We will build a classification model using the \"K-nearest neighbors\"\n",
    "strategy. To predict the target of a new sample, a k-nearest neighbors takes\n",
    "into account its `k` closest samples in the training set and predicts the\n",
    "majority target of these samples.\n",
    "\n",
    "<div class=\"admonition caution alert alert-warning\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Caution!</p>\n",
    "<p class=\"last\">We use a K-nearest neighbors here. However, be aware that it is seldom useful\n",
    "in practice. We use it because it is an intuitive algorithm. In the next\n",
    "notebook, we will introduce better models.</p>\n",
    "</div>\n",
    "\n",
    "The `fit` method is called to train the model from the input (features) and\n",
    "target data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "model = KNeighborsClassifier()\n",
    "_ = model.fit(data, target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Learning can be represented as follows:\n",
    "\n",
    "![Predictor fit diagram](../figures/api_diagram-predictor.fit.svg)\n",
    "\n",
    "The method `fit` is composed of two elements: (i) a **learning algorithm**\n",
    "and (ii) some **model states**. The learning algorithm takes the training\n",
    "data and training target as input and sets the model states. These model\n",
    "states will be used later to either predict (for classifiers and regressors)\n",
    "or transform data (for transformers).\n",
    "\n",
    "Both the learning algorithm and the type of model states are specific to each\n",
    "type of model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">Here and later, we use the name <tt class=\"docutils literal\">data</tt> and <tt class=\"docutils literal\">target</tt> to be explicit. In\n",
    "scikit-learn documentation, <tt class=\"docutils literal\">data</tt> is commonly named <tt class=\"docutils literal\">X</tt> and <tt class=\"docutils literal\">target</tt> is\n",
    "commonly called <tt class=\"docutils literal\">y</tt>.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use our model to make some predictions using the same dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_predicted = model.predict(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can illustrate the prediction mechanism as follows:\n",
    "\n",
    "![Predictor predict diagram](../figures/api_diagram-predictor.predict.svg)\n",
    "\n",
    "To predict, a model uses a **prediction function** that will use the input\n",
    "data together with the model states. As for the learning algorithm and the\n",
    "model states, the prediction function is specific for each type of model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now have a look at the computed predictions. For the sake of\n",
    "simplicity, we will look at the five first predicted targets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_predicted[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed, we can compare these predictions to the actual data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and we could even check if the predictions agree with the real targets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target[:5] == target_predicted[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Number of correct prediction: \"\n",
    "      f\"{(target[:5] == target_predicted[:5]).sum()} / 5\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we see that our model makes a mistake when predicting for the first\n",
    "sample.\n",
    "\n",
    "To get a better assessment, we can compute the average success rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(target == target_predicted).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This result means that the model makes a correct prediction for\n",
    "approximately 82 samples out of 100. Note that we used the same data\n",
    "to train and evaluate our model. Can this evaluation be trusted or is\n",
    "it too good to be true?\n",
    "\n",
    "## Train-test data split\n",
    "\n",
    "When building a machine learning model, it is important to evaluate the\n",
    "trained model on data that was not used to fit it, as **generalization** is\n",
    "more than memorization (meaning we want a rule that generalizes to new data,\n",
    "without comparing to data we memorized).\n",
    "It is harder to conclude on never-seen instances than on already seen ones.\n",
    "\n",
    "Correct evaluation is easily done by leaving out a subset of the data when\n",
    "training the model and using it afterwards for model evaluation.\n",
    "The data used to fit a model is called training data while the data used to\n",
    "assess a model is called testing data.\n",
    "\n",
    "We can load more data, which was actually left-out from the original data\n",
    "set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "adult_census_test = pd.read_csv('../datasets/adult-census-numeric-test.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this new data, we separate our input features and the target to predict,\n",
    "as in the beginning of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_test = adult_census_test[target_name]\n",
    "data_test = adult_census_test.drop(columns=[target_name, ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check the number of features and samples available in this new set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The testing dataset contains {data_test.shape[0]} samples and \"\n",
    "      f\"{data_test.shape[1]} features\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Instead of computing the prediction and manually computing the average\n",
    "success rate, we can use the method `score`. When dealing with classifiers\n",
    "this method returns their performance metric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = model.score(data_test, target_test)\n",
    "model_name = model.__class__.__name__\n",
    "\n",
    "print(f\"The test accuracy using a {model_name} is \"\n",
    "      f\"{accuracy:.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the underlying mechanism when the `score` method is called:\n",
    "\n",
    "![Predictor score diagram](../figures/api_diagram-predictor.score.svg)\n",
    "\n",
    "To compute the score, the predictor first computes the predictions (using\n",
    "the `predict` method) and then uses a scoring function to compare the\n",
    "true target `y` and the predictions. Finally, the score is returned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we compare with the accuracy obtained by wrongly evaluating the model\n",
    "on the training set, we find that this evaluation was indeed optimistic\n",
    "compared to the score obtained on a held-out test set.\n",
    "\n",
    "It shows the importance to always testing the generalization performance of\n",
    "predictive models on a different set than the one used to train these models.\n",
    "We will discuss later in more detail how predictive models should be\n",
    "evaluated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">In this MOOC, we will refer to <strong>generalization performance</strong> of a model when\n",
    "referring to the test score or test error obtained by comparing the\n",
    "prediction of a model and the true targets. Equivalent terms for\n",
    "<strong>generalization performance</strong> are predictive performance and statistical\n",
    "performance. We will refer to <strong>computational performance</strong> of a predictive\n",
    "model when assessing the computational costs of training a predictive model\n",
    "or using it to make predictions.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook Recap\n",
    "\n",
    "In this notebook we:\n",
    "\n",
    "* fitted a **k-nearest neighbors** model on a training dataset;\n",
    "* evaluated its generalization performance on the testing data;\n",
    "* introduced the scikit-learn API `.fit(X, y)` (to train a model),\n",
    "  `.predict(X)` (to make predictions) and `.score(X, y)`\n",
    "  (to evaluate a model)."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}