|
20 | 20 | "cell_type": "markdown", |
21 | 21 | "metadata": {}, |
22 | 22 | "source": [ |
23 | | - "" |
| 23 | + "<img src=\"images/3-layer.png\", width=400, alt=\"The Full 3-Layer Data Model\"/>" |
24 | 24 | ] |
25 | 25 | }, |
26 | 26 | { |
|
36 | 36 | "source": [ |
37 | 37 | "Stepping into the specifics of our exercise, we provide some Aboveground tools that call into the Ground API, and allow us to easily interact with and debug ML models. Throughout this exercise, we will be using these higher-level tools instead of interacting directly with Ground. \n", |
38 | 38 | "\n", |
39 | | - "We're first going to look at the working version of our Twitter modeling pipeline and understand the context that is captured in Ground. Next, we'll see that the unexpected change in the feed format that causes a significant degradation in prediction quality. We will then use Ground to understand what changed & why and fix the bug." |
| 39 | + "We're first going to look at the working version of our Twitter modeling pipeline and understand the context that is captured in Ground. Second, we'll see an unexpected change in the feed format that causes a significant degradation in prediction quality. We will then use Ground to understand what changed & why and fix the bug." |
40 | 40 | ] |
41 | 41 | }, |
42 | 42 | { |
|
97 | 97 | "* `show_all_model_versions`: return the information about all the models Ground knows about.\n", |
98 | 98 | "* `show_model_dependencies`: return the code & data dependencies for a particular model; if no **Ground version** is passed in, the default is assumed to be the most recent version of the model.\n", |
99 | 99 | "* `show_data_schema`: prints out the schema of the dataset the model is trained on; if no **Ground version** is specified, prints out the most recent version of the schema.\n", |
100 | | - "* `diff_deta_schemas`: takes in the **Ground versions** of two data schemas and prints out the differences between them." |
| 100 | + "* `diff_data_schemas`: takes in the **Ground versions** of two data schemas and prints out the differences between them." |
101 | 101 | ] |
102 | 102 | }, |
103 | 103 | { |
|
144 | 144 | "cell_type": "markdown", |
145 | 145 | "metadata": {}, |
146 | 146 | "source": [ |
147 | | - "Okay, so we have a baseline model, and in fact it does pretty well: 60% accuracy. To see why that's pretty good, let's compare to what a *random guess* model would achieve. There are about 170 different countries that we have tweets from in this dataset. If we were to guess _totally_ at random, we would expect $\\frac{1}{170} = 0.6\\%$ accuracy. Even without machine learning, we can do better. As you might have guessed, the plurality of tweets in the world comes from the United States. Thus, we can say that the default case would be to always guess the United States, which would give us about 35% accuracy. (That is the proportion of tweets that comes from the United States.) \n", |
| 147 | + "At this point we have a baseline model, and in fact it does pretty well: 60% accuracy. To see why that's pretty good, let's compare to what a *random guess* model would achieve. There are about 170 different countries that we have tweets from in this dataset. If we were to guess _totally_ at random, we would expect $\\frac{1}{170} = 0.6\\%$ accuracy. Even without machine learning, we can do better. As you might have guessed, the plurality of tweets in the world come from the United States. Thus, we can say that the default case would be to always guess the United States, which would give us about 35% accuracy. (That is the proportion of tweets that comes from the United States.) \n", |
148 | 148 | "\n", |
149 | 149 | "Next, we're going to inspect the data context -- in particular, the lineage information -- to understand what exactly Ground has learned here. Follow the steps below." |
150 | 150 | ] |
|
157 | 157 | }, |
158 | 158 | "outputs": [], |
159 | 159 | "source": [ |
160 | | - "# first, let's retrieve the most recent model version to see what Ground knows\n", |
| 160 | + "# first, let's retrieve the most recent model version since that's the source of our score\n", |
161 | 161 | "# we should see that there's a model number associated with each new version of the model" |
162 | 162 | ] |
163 | 163 | }, |
|
232 | 232 | "cell_type": "markdown", |
233 | 233 | "metadata": {}, |
234 | 234 | "source": [ |
235 | | - "Unfortunately, it seems to be true. Something has changed pretty significantly. We've gone from 60% accuracy down to 35%. Remember from above that this is no better than our intelligent random guess. It's clear that something external to our own pipeline has changed, causing the sudden drop from fairly accurate to no-better-than-random.\n", |
| 235 | + "Unfortunately, it seems to be true. Something has changed pretty significantly. We've gone from 60% accuracy down to 35%. Remember from above that this is no better than our intelligent random guess. We haven't changed anything in our pipeline, so something external must have changed, and caused the sudden drop from fairly accurate to no-better-than-random.\n", |
236 | 236 | "\n", |
237 | | - "The question we have to answer next is what changed that caused our prediction quality to degrade. We can imagine a long list of things that might have changed. If you're stuck, we've written a description below that will help walk you through the investigative steps. \n", |
| 237 | + "The question we have to answer next is what changed that caused our prediction quality to degrade. We can imagine a long list of things that might have changed, but the context stored in Ground will make it fairly easy to narrow our focus down to the real culprit. \n", |
| 238 | + "\n", |
| 239 | + "Your task is to use the aboveground helper functions listed above to identify the root causes of the degraded prediction quality, and remedy it (them). If you're stuck, we've written a description below that will help walk you through the investigative steps. \n", |
238 | 240 | "\n", |
239 | 241 | "**HINTS**: \n", |
240 | 242 | "\n", |
|
346 | 348 | "source": [ |
347 | 349 | "The full solution is provided [here](https://github.com/ground-context/tutorial/blob/master/solutions/Ground-02.ipynb)." |
348 | 350 | ] |
| 351 | + }, |
| 352 | + { |
| 353 | + "cell_type": "code", |
| 354 | + "execution_count": null, |
| 355 | + "metadata": { |
| 356 | + "collapsed": true |
| 357 | + }, |
| 358 | + "outputs": [], |
| 359 | + "source": [ |
| 360 | + "# if you run into any problems, run this cell to reset Ground\n", |
| 361 | + "!bash ../reset_ground.sh >> /dev/null" |
| 362 | + ] |
349 | 363 | } |
350 | 364 | ], |
351 | 365 | "metadata": { |
|
0 commit comments