notebooks/ensemble_ex_03.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \ud83d\udcdd Exercise M6.03\n",
    "\n",
    "The aim of this exercise is to:\n",
    "\n",
    "* verifying if a random forest or a gradient-boosting decision tree overfit\n",
    "  if the number of estimators is not properly chosen;\n",
    "* use the early-stopping strategy to avoid adding unnecessary trees, to\n",
    "  get the best generalization performances.\n",
    "\n",
    "We will use the California housing dataset to conduct our experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data, target = fetch_california_housing(return_X_y=True, as_frame=True)\n",
    "target *= 100  # rescale the target in k$\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, random_state=0, test_size=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">If you want a deeper overview regarding this dataset, you can refer to the\n",
    "Appendix - Datasets description section at the end of this MOOC.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a gradient boosting decision tree with `max_depth=5` and\n",
    "`learning_rate=0.5`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Also create a random forest with fully grown trees by setting `max_depth=None`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "For both the gradient-boosting and random forest models, create a validation\n",
    "curve using the training set to assess the impact of the number of trees on\n",
    "the performance of each model. Evaluate the list of parameters `param_range =\n",
    "[1, 2, 5, 10, 20, 50, 100]` and use the mean absolute error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both gradient boosting and random forest models will always improve when\n",
    "increasing the number of trees in the ensemble. However, it will reach a\n",
    "plateau where adding new trees will just make fitting and scoring slower.\n",
    "\n",
    "To avoid adding new unnecessary tree, unlike random-forest gradient-boosting\n",
    "offers an early-stopping option. Internally, the algorithm will use an\n",
    "out-of-sample set to compute the generalization performance of the model at\n",
    "each addition of a tree. Thus, if the generalization performance is not\n",
    "improving for several iterations, it will stop adding trees.\n",
    "\n",
    "Now, create a gradient-boosting model with `n_estimators=1_000`. This number\n",
    "of trees will be too large. Change the parameter `n_iter_no_change` such\n",
    "that the gradient boosting fitting will stop after adding 5 trees that do not\n",
    "improve the overall generalization performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Estimate the generalization performance of this model again using\n",
    "the `sklearn.metrics.mean_absolute_error` metric but this time using\n",
    "the test set that we held out at the beginning of the notebook.\n",
    "Compare the resulting value with the values observed in the validation\n",
    "curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your code here."
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "encoding": "# -*- coding: utf-8 -*-",
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}