Ground-02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ground & Machine Learning Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this second exercise, we expand our use of Ground to consider how versioning and lineage interact with an application model.\n",
    "\n",
    "To illustrate this, we explore one of the motivating applications for Ground: managing lifecycles for machine learning models. In particular, we are interested in tracking both the code and the data that combine to output a particular ML model. As we've already learned, Ground treats versioning (change over time) independently. As a result, we'll explore a standard scenario in which Ground helps ML developers track the versions of data and code used to train the model. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/3-layer.png\", width=400, alt=\"The Full 3-Layer Data Model\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that bugs in ML applications can show up in subtle ways beyond just breakage. We'll consider a simple example application that takes an ongoing feed from Twitter, and uses the content of each tweet to predict the location that tweet was posted from. During our scenario, Twitter changes the format of its feed. This does not cause our code to *break*; rather, it degrades the quality of our predictions. The evidence of the problem (poor predictions) can be hard to trace back to a root cause (data format change). Ground's versioning and lineage tracking provide the tools to debug the situation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stepping into the specifics of our exercise, we provide some Aboveground tools that call into the Ground API, and allow us to easily interact with and debug ML models. Throughout this exercise, we will be using these higher-level tools instead of interacting directly with Ground. \n",
    "\n",
    "We're first going to look at the working version of our Twitter modeling pipeline and understand the context that is captured in Ground. Second, we'll see an unexpected change in the feed format that causes a significant degradation in prediction quality. We will then use Ground to understand what changed & why and fix the bug."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Please first run the cell below to set up the tutorial before continuing further. The cell will take a minute to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from ml import tutorial\n",
    "tutorial.setup()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this particular example, we will be toying with a model that predicts the location of a Twitter user based on the content of the tweet. Below is a rough description of the pipeline that we have put together:\n",
    "\n",
    "1. Tweets are crawled from Twitter to generate a training set and a test set.\n",
    "2. Those tweets are cleaned and transformed into a tabular csv format.\n",
    "3. The model is trained on the cleaned training set. \n",
    "4. The model trained in step 3 is validated on the cleaned testing data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Model training pipeline](images/target_test_simple.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the aboveground functions we are providing — these are the tools we will need in the steps below:\n",
    "\n",
    "* `setup`: prepares and configures the system and data for this tutorial\n",
    "* `display_data`: displays a dataframe containing the data we will use throughout this tutorial; this data is the pre-labeled training & test data that we are going to use to train the model. There are some funny tweets for you to browse through. :-)\n",
    "* `test_model`: executes the machine learning pipeline to both train and test the model, reports prediction accuracy\n",
    "* `show_model_version`: return the information about the most recent model built.\n",
    "* `show_all_model_versions`: return the information about all the models Ground knows about.\n",
    "* `show_model_dependencies`: return the code & data dependencies for a particular model; if no **Ground version** is passed in, the default is assumed to be the most recent version of the model.\n",
    "* `show_data_schema`: prints out the schema of the dataset the model is trained on; if no **Ground version** is specified, prints out the most recent version of the schema.\n",
    "* `diff_data_schemas`: takes in the **Ground versions** of two data schemas and prints out the differences between them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by examining two points in our pipeline above: one right after cleaning, and the other after the model is trained and tested.  \n",
    "\n",
    "First, let's look at the data that comes out of the cleaning step in our pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tutorial.display_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, now let's train and test the model. `test_model` will print out some information followed by the test accuracy of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "output = tutorial.test_model()\n",
    "print(output.decode('UTF-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point we have a baseline model, and in fact it does pretty well: 60% accuracy. To see why that's pretty good, let's compare to what a *random guess* model would achieve. There are about 170 different countries that we have tweets from in this dataset. If we were to guess _totally_ at random, we would expect $\\frac{1}{170} = 0.6\\%$ accuracy. Even without machine learning, we can do better. As you might have guessed, the plurality of tweets in the world come from the United States. Thus, we can say that the default case would be to always guess the United States, which would give us about 35% accuracy. (That is the proportion of tweets that comes from the United States.) \n",
    "\n",
    "Next, we're going to inspect the data context -- in particular, the lineage information -- to understand what exactly Ground has learned here. Follow the steps below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# first, let's retrieve the most recent model version since that's the source of our score\n",
    "# we should see that there's a model number associated with each new version of the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# next, let's look at the dependencies that the model has"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# now that we know the Ground version of the dataset, let's also inspect the schema of that dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dealing with Change"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's say that we fast-forward time by a week."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from ml import ground_time\n",
    "ground_time.fast_forward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, as soon we as get to the future, a colleague tells us that the model that we trained last week isn't working well anymore. Let's first test the model and verify this for ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "output = tutorial.test_model()\n",
    "print(output.decode('UTF-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, it seems to be true. Something has changed pretty significantly. We've gone from 60% accuracy down to 35%. Remember from above that this is no better than our intelligent random guess. We haven't changed anything in our pipeline, so something external must have changed, and caused the sudden drop from fairly accurate to no-better-than-random.\n",
    "\n",
    "The question we have to answer next is what changed that caused our prediction quality to degrade. We can imagine a long list of things that might have changed, but the context stored in Ground will make it fairly easy to narrow our focus down to the real culprit. \n",
    "\n",
    "Your task is to use the aboveground helper functions listed above to identify the root causes of the degraded prediction quality, and remedy it (them). If you're stuck, we've written a description below that will help walk you through the investigative steps. \n",
    "\n",
    "**HINTS**: \n",
    "\n",
    "1. You're probably going to find the tutorial helper functions described above above very helpful.\n",
    "2. The ultimate solution will be to modify the `clean` function somehow. We provided a skeleton below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your experiments and exploration go here. \n",
    "# The cells below are meant to be for extra commands. You don't necessarily need to use them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your experiments and exploration go here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your experiments and exploration go here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Your experiments and exploration go here. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**MORE HINTS**: \n",
    "\n",
    "* We suggest that you start by comparing the schemas that different version of the model depend on.\n",
    "* You can switch two columns in a dataframe by running code that looks like this:\n",
    "```python\n",
    "temp = df[\"col1\"].copy()\n",
    "df[\"col1\"] = df[\"col2\"]\n",
    "df[\"col2\"] = temp\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%%writefile ml/my_cleaner.py\n",
    "#!/usr/bin/env python\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "def clean(df):\n",
    "    pass\n",
    "    # your solution here\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now run the model one last time to make sure things are back to normal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# The model should now be back to normal.\n",
    "output = tutorial.test_model()\n",
    "print(output.decode('UTF-8'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The full solution is provided [here](https://github.com/ground-context/tutorial/blob/master/solutions/Ground-02.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# if you run into any problems, run this cell to reset Ground\n",
    "!bash ../reset_ground.sh >> /dev/null"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}