notebooks/cross_validation_ex_01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \ud83d\udcdd Exercise M2.01\n",
    "\n",
    "The aim of this exercise is to make the following experiments:\n",
    "\n",
    "* train and test a support vector machine classifier through cross-validation;\n",
    "* study the effect of the parameter gamma of this classifier using a\n",
    "  validation curve;\n",
    "* use a learning curve to determine the usefulness of adding new samples in\n",
    "  the dataset when building a classifier.\n",
    "\n",
    "To make these experiments we will first load the blood transfusion dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">If you want a deeper overview regarding this dataset, you can refer to the\n",
    "Appendix - Datasets description section at the end of this MOOC.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "blood_transfusion = pd.read_csv(\"../datasets/blood_transfusion.csv\")\n",
    "data = blood_transfusion.drop(columns=\"Class\")\n",
    "target = blood_transfusion[\"Class\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use a support vector machine classifier (SVM). In its most simple\n",
    "form, a SVM classifier is a linear classifier behaving similarly to a logistic\n",
    "regression. Indeed, the optimization used to find the optimal weights of the\n",
    "linear model are different but we don't need to know these details for the\n",
    "exercise.\n",
    "\n",
    "Also, this classifier can become more flexible/expressive by using a so-called\n",
    "kernel that makes the model become non-linear. Again, no requirement regarding\n",
    "the mathematics is required to accomplish this exercise.\n",
    "\n",
    "We will use an RBF kernel where a parameter `gamma` allows to tune the\n",
    "flexibility of the model.\n",
    "\n",
    "First let's create a predictive pipeline made of:\n",
    "\n",
    "* a [`sklearn.preprocessing.StandardScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html)\n",
    "  with default parameter;\n",
    "* a [`sklearn.svm.SVC`](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html)\n",
    "  where the parameter `kernel` could be set to `\"rbf\"`. Note that this is the\n",
    "  default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the generalization performance of your model by cross-validation with\n",
    "a `ShuffleSplit` scheme. Thus, you can use\n",
    "[`sklearn.model_selection.cross_validate`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_validate.html)\n",
    "and pass a\n",
    "[`sklearn.model_selection.ShuffleSplit`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ShuffleSplit.html)\n",
    "to the `cv` parameter. Only fix the `random_state=0` in the `ShuffleSplit` and\n",
    "let the other parameters to the default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As previously mentioned, the parameter `gamma` is one of the parameters\n",
    "controlling under/over-fitting in support vector machine with an RBF kernel.\n",
    "\n",
    "Evaluate the effect of the parameter `gamma` by using the\n",
    "[`sklearn.model_selection.validation_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.validation_curve.html)\n",
    "function. You can leave the default `scoring=None` which is equivalent to\n",
    "`scoring=\"accuracy\"` for classification problems. You can vary `gamma` between\n",
    "`10e-3` and `10e2` by generating samples on a logarithmic scale with the help\n",
    "of `np.logspace(-3, 2, num=30)`.\n",
    "\n",
    "Since we are manipulating a `Pipeline` the parameter name will be set to\n",
    "`svc__gamma` instead of only `gamma`. You can retrieve the parameter name\n",
    "using `model.get_params().keys()`. We will go more into detail regarding\n",
    "accessing and setting hyperparameter in the next section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the validation curve for the train and test scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you can perform an analysis to check whether adding new samples to the\n",
    "dataset could help our model to better generalize. Compute the learning curve\n",
    "(using [`sklearn.model_selection.learning_curve`](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.learning_curve.html))\n",
    "by computing the train and test scores for different training dataset size.\n",
    "Plot the train and test scores with respect to the number of samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "encoding": "# -*- coding: utf-8 -*-",
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}