notebooks/parameter_tuning_grid_search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hyperparameter tuning by grid-search\n",
    "\n",
    "In the previous notebook, we saw that hyperparameters can affect the\n",
    "generalization performance of a model. In this notebook, we will show how to\n",
    "optimize hyperparameters using a grid-search approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our predictive model\n",
    "\n",
    "Let us reload the dataset as we did previously:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import set_config\n",
    "\n",
    "set_config(display=\"diagram\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We extract the column containing the target."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_name = \"class\"\n",
    "target = adult_census[target_name]\n",
    "target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We drop from our data the target and the `\"education-num\"` column which\n",
    "duplicates the information from the `\"education\"` column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = adult_census.drop(columns=[target_name, \"education-num\"])\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the dataset is loaded, we split it into a training and testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will define a pipeline as seen in the first module. It will handle both\n",
    "numerical and categorical features.\n",
    "\n",
    "The first step is to select all the categorical columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import make_column_selector as selector\n",
    "\n",
    "categorical_columns_selector = selector(dtype_include=object)\n",
    "categorical_columns = categorical_columns_selector(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we will use a tree-based model as a classifier\n",
    "(i.e. `HistGradientBoostingClassifier`). That means:\n",
    "\n",
    "* Numerical variables don't need scaling;\n",
    "* Categorical variables can be dealt with an `OrdinalEncoder` even if the \n",
    "  coding order is not meaningful;\n",
    "* For tree-based models, the `OrdinalEncoder` avoids having high-dimensional \n",
    "  representations.\n",
    "\n",
    "We now build our `OrdinalEncoder` by passing it the known categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "\n",
    "categorical_preprocessor = OrdinalEncoder(handle_unknown=\"use_encoded_value\",\n",
    "                                          unknown_value=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then use a `ColumnTransformer` to select the categorical columns and\n",
    "apply the `OrdinalEncoder` to them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "preprocessor = ColumnTransformer([\n",
    "    ('cat_preprocessor', categorical_preprocessor, categorical_columns)],\n",
    "    remainder='passthrough', sparse_threshold=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we use a tree-based classifier (i.e. histogram gradient-boosting) to\n",
    "predict whether or not a person earns more than 50 k$ a year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import HistGradientBoostingClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "model = Pipeline([\n",
    "    (\"preprocessor\", preprocessor),\n",
    "    (\"classifier\",\n",
    "     HistGradientBoostingClassifier(random_state=42, max_leaf_nodes=4))])\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tuning using a grid-search\n",
    "\n",
    "In the previous exercise we used one `for` loop for each hyperparameter to find the \n",
    "best combination over a fixed grid of values. `GridSearchCV` is a scikit-learn class \n",
    "that implements a very similar logic with less repetitive code.\n",
    "\n",
    "Let's see how to use the `GridSearchCV` estimator for doing such search.\n",
    "Since the grid-search will be costly, we will only explore the combination\n",
    "learning-rate and the maximum number of nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "param_grid = {\n",
    "    'classifier__learning_rate': (0.01, 0.1, 1, 10),\n",
    "    'classifier__max_leaf_nodes': (3, 10, 30)}\n",
    "model_grid_search = GridSearchCV(model, param_grid=param_grid,\n",
    "                                 n_jobs=2, cv=2)\n",
    "model_grid_search.fit(data_train, target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we will check the accuracy of our model using the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = model_grid_search.score(data_test, target_test)\n",
    "print(\n",
    "    f\"The test accuracy score of the grid-searched pipeline is: \"\n",
    "    f\"{accuracy:.2f}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition warning alert alert-danger\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Warning</p>\n",
    "<p>Be aware that the evaluation should normally be performed through\n",
    "cross-validation by providing <tt class=\"docutils literal\">model_grid_search</tt> as a model to the\n",
    "<tt class=\"docutils literal\">cross_validate</tt> function.</p>\n",
    "<p class=\"last\">Here, we used a single train-test split to to evaluate <tt class=\"docutils literal\">model_grid_search</tt>.\n",
    "In a future notebook will go into more detail about nested cross-validation,\n",
    "when you use cross-validation both for hyperparameter tuning and model\n",
    "evaluation.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `GridSearchCV` estimator takes a `param_grid` parameter which defines\n",
    "all hyperparameters and their associated values. The grid-search will be in\n",
    "charge of creating all possible combinations and test them.\n",
    "\n",
    "The number of combinations will be equal to the product of the\n",
    "number of values to explore for each parameter (e.g. in our example 4 x 3\n",
    "combinations). Thus, adding new parameters with their associated values to be\n",
    "explored become rapidly computationally expensive.\n",
    "\n",
    "Once the grid-search is fitted, it can be used as any other predictor by\n",
    "calling `predict` and `predict_proba`. Internally, it will use the model with\n",
    "the best parameters found during `fit`.\n",
    "\n",
    "Get predictions for the 5 first samples using the estimator with the best\n",
    "parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_grid_search.predict(data_test.iloc[0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can know about these parameters by looking at the `best_params_`\n",
    "attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"The best set of parameters is: \"\n",
    "      f\"{model_grid_search.best_params_}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy and the best parameters of the grid-searched pipeline are\n",
    "similar to the ones we found in the previous exercise, where we searched the\n",
    "best parameters \"by hand\" through a double for loop.\n",
    "\n",
    "In addition, we can inspect all results which are stored in the attribute\n",
    "`cv_results_` of the grid-search. We will filter some specific columns\n",
    "from these results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_results = pd.DataFrame(model_grid_search.cv_results_).sort_values(\n",
    "    \"mean_test_score\", ascending=False)\n",
    "cv_results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us focus on the most interesting columns and shorten the parameter\n",
    "names to remove the `\"param_classifier__\"` prefix for readability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the parameter names\n",
    "column_results = [f\"param_{name}\" for name in param_grid.keys()]\n",
    "column_results += [\n",
    "    \"mean_test_score\", \"std_test_score\", \"rank_test_score\"]\n",
    "cv_results = cv_results[column_results]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def shorten_param(param_name):\n",
    "    if \"__\" in param_name:\n",
    "        return param_name.rsplit(\"__\", 1)[1]\n",
    "    return param_name\n",
    "\n",
    "\n",
    "cv_results = cv_results.rename(shorten_param, axis=1)\n",
    "cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With only 2 parameters, we might want to visualize the grid-search as a\n",
    "heatmap. We need to transform our `cv_results` into a dataframe where:\n",
    "\n",
    "- the rows will correspond to the learning-rate values;\n",
    "- the columns will correspond to the maximum number of leaf;\n",
    "- the content of the dataframe will be the mean test scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pivoted_cv_results = cv_results.pivot_table(\n",
    "    values=\"mean_test_score\", index=[\"learning_rate\"],\n",
    "    columns=[\"max_leaf_nodes\"])\n",
    "\n",
    "pivoted_cv_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use a heatmap representation to show the above dataframe visually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "ax = sns.heatmap(pivoted_cv_results, annot=True, cmap=\"YlGnBu\", vmin=0.7,\n",
    "                 vmax=0.9)\n",
    "ax.invert_yaxis()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above tables highlights the following things:\n",
    "\n",
    "* for too high values of `learning_rate`, the generalization performance of the\n",
    "  model is degraded and adjusting the value of `max_leaf_nodes` cannot fix\n",
    "  that problem;\n",
    "* outside of this pathological region, we observe that the optimal choice\n",
    "  of `max_leaf_nodes` depends on the value of `learning_rate`;\n",
    "* in particular, we observe a \"diagonal\" of good models with an accuracy\n",
    "  close to the maximal of 0.87: when the value of `max_leaf_nodes` is\n",
    "  increased, one should decrease the value of `learning_rate` accordingly\n",
    "  to preserve a good accuracy.\n",
    "\n",
    "The precise meaning of those two parameters will be explained later.\n",
    "\n",
    "For now we will note that, in general, **there is no unique optimal parameter\n",
    "setting**: 4 models out of the 12 parameter configurations reach the maximal\n",
    "accuracy (up to small random fluctuations caused by the sampling of the\n",
    "training set)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we have seen:\n",
    "\n",
    "* how to optimize the hyperparameters of a predictive model via a\n",
    "  grid-search;\n",
    "* that searching for more than two hyperparamters is too costly;\n",
    "* that a grid-search does not necessarily find an optimal solution."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}