examples/Magics in IPython Kernel.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demonstrating sparkmagic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## This notebook will demonstrate how we can use the spark magic to interspere our Python code with code that is running against a Spark cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s say we’re working in an IPython notebook and we want to use Spark to analyze some data. So, we'll load `sparkmagic` in order to be able to talk to Spark from our Python notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext sparkmagic.magics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With it, the `%manage_spark` line magic and the `%%spark` magic are available.\n",
    "\n",
    "The %%manage_spark line magic lets you manage Livy endpoints and Spark sessions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by adding an Endpoint.\n",
    "\n",
    "An Endpoint is a [Livy](https://github.com/cloudera/livy) installation running on a Spark cluster. \n",
    "\n",
    "`sparkmagic` allows us to specify the Livy endpoint along with a username and password to authenticate to it. If the Livy endpoint is on your local machine or has no password, simply leave the text fields for username and password blank."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added endpoint https://localhost:8998\n",
      "Creating SparkContext as 'sc'\n",
      "Creating HiveContext as 'sqlContext'\n"
     ]
    }
   ],
   "source": [
    "%manage_spark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![add_endpoint](images/addendpoint.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, add a session to the endpoint you added. The name you give to the session will be used with the `%%spark` magic to run Spark code. You can also specify the configuration you want to start the session with. You can create either Python (PySpark) or Scala (Spark) sessions.\n",
    "\n",
    "Creating a session will create a `SparkContext` with the name `sc` and a `HiveContext` with the name `sqlContext`.\n",
    "\n",
    "We'll start by adding a PySpark session."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![add_session](images/addsession.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now run Spark code against your Livy session.  For information on the available commands, run %spark?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%spark?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pyspark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can run code against your Spark session by adding `%%spark` at the beginning of the cell.  Since we’ve only created a single session, we don’t need to specify the session name.\n",
    "\n",
    "In the following cell, I'll create a Resilient Distributed Dataset (RDD) called fruits, and print its first element."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First element of numbers is 1 and its description is:\n",
      "(16) ParallelCollectionRDD[0] at parallelize at PythonRDD.scala:423 []"
     ]
    }
   ],
   "source": [
    "%%spark\n",
    "numbers = sc.parallelize([1, 2, 3, 4])\n",
    "print('First element of numbers is {} and its description is:\\n{}'.format(numbers.first(), numbers.toDebugString()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, you've created your session and executed some statements. If you want to look at the Livy logs for this session, simply run a cell like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "16/06/09 23:43:28 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.\n",
      "16/06/09 23:43:28 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.\n",
      "16/06/09 23:43:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n",
      "16/06/09 23:43:30 INFO TimelineClientImpl: Timeline service address: http://localhost:8188/ws/v1/timeline/\n",
      "16/06/09 23:43:30 INFO Client: Requesting a new application from cluster with 10 NodeManagers\n",
      "16/06/09 23:43:30 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (25600 MB per container)\n",
      "16/06/09 23:43:30 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead\n",
      "16/06/09 23:43:30 INFO Client: Setting up container launch context for our AM\n",
      "16/06/09 23:43:30 INFO Client: Setting up the launch environment for our AM container\n",
      "16/06/09 23:43:30 INFO Client: Preparing resources for our AM container\n",
      "16/06/09 23:43:31 INFO MetricsConfig: loaded properties from hadoop-metrics2-azure-file-system.properties\n",
      "16/06/09 23:43:31 INFO WasbAzureIaasSink: Init starting.\n",
      "16/06/09 23:43:31 INFO AzureIaasSink: Init starting. Initializing MdsLogger.\n",
      "16/06/09 23:43:31 INFO AzureIaasSink: Init completed.\n",
      "16/06/09 23:43:31 INFO WasbAzureIaasSink: Init completed.\n",
      "16/06/09 23:43:31 INFO MetricsSinkAdapter: Sink azurefs2 started\n",
      "16/06/09 23:43:31 INFO MetricsSystemImpl: Scheduled snapshot period at 60 second(s).\n",
      "16/06/09 23:43:31 INFO MetricsSystemImpl: azure-file-system metrics system started\n",
      "16/06/09 23:43:31 INFO Client: Uploading resource file:/usr/hdp/current/spark-client/python/lib/pyspark.zip -> wasb://MYSTORAGE@store.blob.core.windows.net/user/spark/.sparkStaging/application_1464100251524_0001/pyspark.zip\n",
      "16/06/09 23:43:32 INFO Client: Uploading resource file:/usr/hdp/current/spark-client/python/lib/py4j-0.8.2.1-src.zip -> wasb://MYSTORAGE@store.blob.core.windows.net/user/spark/.sparkStaging/application_1464100251524_0001/py4j-0.8.2.1-src.zip\n",
      "16/06/09 23:43:32 INFO Client: Uploading resource file:/usr/hdp/current/spark-client/conf/hive-site.xml -> wasb://MYSTORAGE@store.blob.core.windows.net/user/spark/.sparkStaging/application_1464100251524_0001/hive-site.xml\n",
      "16/06/09 23:43:33 INFO Client: Uploading resource file:/usr/hdp/2.3.3.1-7/spark/python/lib/pyspark.zip -> wasb://MYSTORAGE@store.blob.core.windows.net/user/spark/.sparkStaging/application_1464100251524_0001/pyspark.zip\n",
      "16/06/09 23:43:33 INFO Client: Uploading resource file:/usr/hdp/2.3.3.1-7/spark/python/lib/py4j-0.8.2.1-src.zip -> wasb://MYSTORAGE@store.blob.core.windows.net/user/spark/.sparkStaging/application_1464100251524_0001/py4j-0.8.2.1-src.zip\n",
      "16/06/09 23:43:34 INFO Client: Uploading resource file:/tmp/spark-1c0a5560-2256-4f63-ba1f-c0dfd25dc24b/__spark_conf__324896021270563710.zip -> wasb://MYSTORAGE@store.blob.core.windows.net/user/spark/.sparkStaging/application_1464100251524_0001/__spark_conf__324896021270563710.zip\n",
      "16/06/09 23:43:34 WARN Client: spark.yarn.am.extraJavaOptions will not take effect in cluster mode\n",
      "16/06/09 23:43:34 INFO SecurityManager: Changing view acls to: spark\n",
      "16/06/09 23:43:34 INFO SecurityManager: Changing modify acls to: spark\n",
      "16/06/09 23:43:34 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); users with modify permissions: Set(spark)\n",
      "16/06/09 23:43:34 INFO Client: Submitting application 1 to ResourceManager\n",
      "16/06/09 23:43:35 INFO YarnClientImpl: Submitted application application_1464100251524_0001\n",
      "16/06/09 23:43:36 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:36 INFO Client: \n",
      "\t client token: N/A\n",
      "\t diagnostics: N/A\n",
      "\t ApplicationMaster host: N/A\n",
      "\t ApplicationMaster RPC port: -1\n",
      "\t queue: default\n",
      "\t start time: 1465515814875\n",
      "\t final status: UNDEFINED\n",
      "\t tracking URL: http://localhost:8088/proxy/application_1464100251524_0001/\n",
      "\t user: spark\n",
      "16/06/09 23:43:37 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:38 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:39 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:40 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:41 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:42 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:43 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:44 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:45 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:46 INFO Client: Application report for application_1464100251524_0001 (state: ACCEPTED)\n",
      "16/06/09 23:43:47 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:47 INFO Client: \n",
      "\t client token: N/A\n",
      "\t diagnostics: N/A\n",
      "\t ApplicationMaster host: 10.0.0.4\n",
      "\t ApplicationMaster RPC port: 0\n",
      "\t queue: default\n",
      "\t start time: 1465515814875\n",
      "\t final status: UNDEFINED\n",
      "\t tracking URL: http://localhost:8088/proxy/application_1464100251524_0001/\n",
      "\t user: spark\n",
      "16/06/09 23:43:48 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:49 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:50 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:51 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:52 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:53 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:54 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:55 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:56 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:57 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:58 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:43:59 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:00 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:01 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:02 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:03 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:04 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:05 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:06 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:07 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:08 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:09 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:10 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:11 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:12 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:13 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:14 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:15 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:16 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:17 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:18 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:19 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:20 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:21 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:22 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:23 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:24 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)\n",
      "16/06/09 23:44:25 INFO Client: Application report for application_1464100251524_0001 (state: RUNNING)"
     ]
    }
   ],
   "source": [
    "%spark logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SparkSQL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can run SQL queries by passing the arguments `-c sql` to the %%spark magic\n",
    "\n",
    "First, let's see what tables we have defined:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>isTemporary</th>\n",
       "      <th>tableName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>hivesampletable</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>hvac</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>table4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>table5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>test</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>False</td>\n",
       "      <td>test1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>False</td>\n",
       "      <td>test2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>False</td>\n",
       "      <td>test3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>False</td>\n",
       "      <td>testnew</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  isTemporary        tableName\n",
       "0       False  hivesampletable\n",
       "1       False             hvac\n",
       "2       False           table4\n",
       "3       False           table5\n",
       "4       False             test\n",
       "5       False            test1\n",
       "6       False            test2\n",
       "7       False            test3\n",
       "8       False          testnew"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%spark -c sql\n",
    "SHOW TABLES"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's query one of the available tables.\n",
    "\n",
    "Notice that we are passing the `--output` or `-o` parameter with a value of `df_hvac` so that the output of our SQL query is saved in the `df_hvac` variable in the IPython kernel context as a [Pandas](http://pandas.pydata.org/) DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clientid</th>\n",
       "      <th>country</th>\n",
       "      <th>devicemake</th>\n",
       "      <th>devicemodel</th>\n",
       "      <th>deviceplatform</th>\n",
       "      <th>market</th>\n",
       "      <th>querydwelltime</th>\n",
       "      <th>querytime</th>\n",
       "      <th>sessionid</th>\n",
       "      <th>sessionpagevieworder</th>\n",
       "      <th>state</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>United States</td>\n",
       "      <td>Samsung</td>\n",
       "      <td>SCH-i500</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>13.920401</td>\n",
       "      <td>2016-06-09 18:54:20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>California</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>23</td>\n",
       "      <td>United States</td>\n",
       "      <td>HTC</td>\n",
       "      <td>Incredible</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2016-06-09 19:19:44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Pennsylvania</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>23</td>\n",
       "      <td>United States</td>\n",
       "      <td>HTC</td>\n",
       "      <td>Incredible</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>1.475742</td>\n",
       "      <td>2016-06-09 19:19:46</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Pennsylvania</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23</td>\n",
       "      <td>United States</td>\n",
       "      <td>HTC</td>\n",
       "      <td>Incredible</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>0.245968</td>\n",
       "      <td>2016-06-09 19:19:47</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Pennsylvania</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>20.309534</td>\n",
       "      <td>2016-06-09 01:37:50</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Colorado</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>16.298167</td>\n",
       "      <td>2016-06-09 00:53:31</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Colorado</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>1.771523</td>\n",
       "      <td>2016-06-09 00:53:50</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Colorado</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>11.675599</td>\n",
       "      <td>2016-06-09 16:44:21</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Utah</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>36.944689</td>\n",
       "      <td>2016-06-09 16:43:41</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>Utah</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>28.981142</td>\n",
       "      <td>2016-06-09 01:37:19</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Colorado</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   clientid        country devicemake devicemodel deviceplatform market  \\\n",
       "0         8  United States    Samsung    SCH-i500        Android  en-US   \n",
       "1        23  United States        HTC  Incredible        Android  en-US   \n",
       "2        23  United States        HTC  Incredible        Android  en-US   \n",
       "3        23  United States        HTC  Incredible        Android  en-US   \n",
       "4        28  United States   Motorola     Droid X        Android  en-US   \n",
       "5        28  United States   Motorola     Droid X        Android  en-US   \n",
       "6        28  United States   Motorola     Droid X        Android  en-US   \n",
       "7        28  United States   Motorola     Droid X        Android  en-US   \n",
       "8        28  United States   Motorola     Droid X        Android  en-US   \n",
       "9        28  United States   Motorola     Droid X        Android  en-US   \n",
       "\n",
       "   querydwelltime           querytime  sessionid  sessionpagevieworder  \\\n",
       "0       13.920401 2016-06-09 18:54:20          0                     0   \n",
       "1             NaN 2016-06-09 19:19:44          0                     0   \n",
       "2        1.475742 2016-06-09 19:19:46          0                     1   \n",
       "3        0.245968 2016-06-09 19:19:47          0                     2   \n",
       "4       20.309534 2016-06-09 01:37:50          1                     1   \n",
       "5       16.298167 2016-06-09 00:53:31          0                     0   \n",
       "6        1.771523 2016-06-09 00:53:50          0                     1   \n",
       "7       11.675599 2016-06-09 16:44:21          2                     1   \n",
       "8       36.944689 2016-06-09 16:43:41          2                     0   \n",
       "9       28.981142 2016-06-09 01:37:19          1                     0   \n",
       "\n",
       "          state  \n",
       "0    California  \n",
       "1  Pennsylvania  \n",
       "2  Pennsylvania  \n",
       "3  Pennsylvania  \n",
       "4      Colorado  \n",
       "5      Colorado  \n",
       "6      Colorado  \n",
       "7          Utah  \n",
       "8          Utah  \n",
       "9      Colorado  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%spark -c sql -o df_hvac --maxrows 10\n",
    "SELECT * FROM hivesampletable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">SQL queries also have other parameters you can pass in, like `--samplemethod`, `--maxrows`, `--samplefraction`, and `--quiet`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now simply use the Pandas dataframe from the IPython notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clientid</th>\n",
       "      <th>country</th>\n",
       "      <th>devicemake</th>\n",
       "      <th>devicemodel</th>\n",
       "      <th>deviceplatform</th>\n",
       "      <th>market</th>\n",
       "      <th>querydwelltime</th>\n",
       "      <th>querytime</th>\n",
       "      <th>sessionid</th>\n",
       "      <th>sessionpagevieworder</th>\n",
       "      <th>state</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>United States</td>\n",
       "      <td>Samsung</td>\n",
       "      <td>SCH-i500</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>13.920401</td>\n",
       "      <td>2016-06-09 18:54:20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>California</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>23</td>\n",
       "      <td>United States</td>\n",
       "      <td>HTC</td>\n",
       "      <td>Incredible</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2016-06-09 19:19:44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Pennsylvania</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>23</td>\n",
       "      <td>United States</td>\n",
       "      <td>HTC</td>\n",
       "      <td>Incredible</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>1.475742</td>\n",
       "      <td>2016-06-09 19:19:46</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Pennsylvania</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23</td>\n",
       "      <td>United States</td>\n",
       "      <td>HTC</td>\n",
       "      <td>Incredible</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>0.245968</td>\n",
       "      <td>2016-06-09 19:19:47</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Pennsylvania</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>20.309534</td>\n",
       "      <td>2016-06-09 01:37:50</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Colorado</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   clientid        country devicemake devicemodel deviceplatform market  \\\n",
       "0         8  United States    Samsung    SCH-i500        Android  en-US   \n",
       "1        23  United States        HTC  Incredible        Android  en-US   \n",
       "2        23  United States        HTC  Incredible        Android  en-US   \n",
       "3        23  United States        HTC  Incredible        Android  en-US   \n",
       "4        28  United States   Motorola     Droid X        Android  en-US   \n",
       "\n",
       "   querydwelltime           querytime  sessionid  sessionpagevieworder  \\\n",
       "0       13.920401 2016-06-09 18:54:20          0                     0   \n",
       "1             NaN 2016-06-09 19:19:44          0                     0   \n",
       "2        1.475742 2016-06-09 19:19:46          0                     1   \n",
       "3        0.245968 2016-06-09 19:19:47          0                     2   \n",
       "4       20.309534 2016-06-09 01:37:50          1                     1   \n",
       "\n",
       "          state  \n",
       "0    California  \n",
       "1  Pennsylvania  \n",
       "2  Pennsylvania  \n",
       "3  Pennsylvania  \n",
       "4      Colorado  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_hvac.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to visualize the data in the Pandas dataframe, you can write your own code to do so, or you can use our autovisualization widget:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from autovizwidget.widget.utils import display_dataframe\n",
    "display_dataframe(df_hvac)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![widget](images/widget.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">You could also choose to have this widget display by default for *all* Pandas dataframes from here on by running this piece of code:\n",
    "\n",
    "```\n",
    "ip = get_ipython()\n",
    "ip.display_formatter.ipython_display_formatter.for_type_by_name('pandas.core.frame', 'DataFrame', display_dataframe)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scala support"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to write your Spark code in Scala, you can easily do that."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add a Scala session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating SparkContext as 'sc'\n",
      "Creating HiveContext as 'sqlContext'\n"
     ]
    }
   ],
   "source": [
    "%manage_spark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![add_session](images/addsession_s.PNG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And just run some Spark code. Notice that we now specify the session we want to use, `-s my_spark`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "res0: Int = 1"
     ]
    }
   ],
   "source": [
    "%%spark -s my_spark\n",
    "val hvacText = sc.parallelize(Array(1, 2, 3, 4))\n",
    "hvacText.first()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can query the table with **SparkSQL** too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clientid</th>\n",
       "      <th>country</th>\n",
       "      <th>devicemake</th>\n",
       "      <th>devicemodel</th>\n",
       "      <th>deviceplatform</th>\n",
       "      <th>market</th>\n",
       "      <th>querydwelltime</th>\n",
       "      <th>querytime</th>\n",
       "      <th>sessionid</th>\n",
       "      <th>sessionpagevieworder</th>\n",
       "      <th>state</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>United States</td>\n",
       "      <td>Samsung</td>\n",
       "      <td>SCH-i500</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>13.920401</td>\n",
       "      <td>2016-06-09 18:54:20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>California</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>23</td>\n",
       "      <td>United States</td>\n",
       "      <td>HTC</td>\n",
       "      <td>Incredible</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2016-06-09 19:19:44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Pennsylvania</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>23</td>\n",
       "      <td>United States</td>\n",
       "      <td>HTC</td>\n",
       "      <td>Incredible</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>1.475742</td>\n",
       "      <td>2016-06-09 19:19:46</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Pennsylvania</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23</td>\n",
       "      <td>United States</td>\n",
       "      <td>HTC</td>\n",
       "      <td>Incredible</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>0.245968</td>\n",
       "      <td>2016-06-09 19:19:47</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Pennsylvania</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>20.309534</td>\n",
       "      <td>2016-06-09 01:37:50</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Colorado</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>16.298167</td>\n",
       "      <td>2016-06-09 00:53:31</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Colorado</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>1.771523</td>\n",
       "      <td>2016-06-09 00:53:50</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Colorado</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>11.675599</td>\n",
       "      <td>2016-06-09 16:44:21</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>Utah</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>36.944689</td>\n",
       "      <td>2016-06-09 16:43:41</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>Utah</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>28.981142</td>\n",
       "      <td>2016-06-09 01:37:19</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>Colorado</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   clientid        country devicemake devicemodel deviceplatform market  \\\n",
       "0         8  United States    Samsung    SCH-i500        Android  en-US   \n",
       "1        23  United States        HTC  Incredible        Android  en-US   \n",
       "2        23  United States        HTC  Incredible        Android  en-US   \n",
       "3        23  United States        HTC  Incredible        Android  en-US   \n",
       "4        28  United States   Motorola     Droid X        Android  en-US   \n",
       "5        28  United States   Motorola     Droid X        Android  en-US   \n",
       "6        28  United States   Motorola     Droid X        Android  en-US   \n",
       "7        28  United States   Motorola     Droid X        Android  en-US   \n",
       "8        28  United States   Motorola     Droid X        Android  en-US   \n",
       "9        28  United States   Motorola     Droid X        Android  en-US   \n",
       "\n",
       "   querydwelltime           querytime  sessionid  sessionpagevieworder  \\\n",
       "0       13.920401 2016-06-09 18:54:20          0                     0   \n",
       "1             NaN 2016-06-09 19:19:44          0                     0   \n",
       "2        1.475742 2016-06-09 19:19:46          0                     1   \n",
       "3        0.245968 2016-06-09 19:19:47          0                     2   \n",
       "4       20.309534 2016-06-09 01:37:50          1                     1   \n",
       "5       16.298167 2016-06-09 00:53:31          0                     0   \n",
       "6        1.771523 2016-06-09 00:53:50          0                     1   \n",
       "7       11.675599 2016-06-09 16:44:21          2                     1   \n",
       "8       36.944689 2016-06-09 16:43:41          2                     0   \n",
       "9       28.981142 2016-06-09 01:37:19          1                     0   \n",
       "\n",
       "          state  \n",
       "0    California  \n",
       "1  Pennsylvania  \n",
       "2  Pennsylvania  \n",
       "3  Pennsylvania  \n",
       "4      Colorado  \n",
       "5      Colorado  \n",
       "6      Colorado  \n",
       "7          Utah  \n",
       "8          Utah  \n",
       "9      Colorado  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%spark -s my_spark -c sql -o my_df_from_scala --maxrows 10\n",
    "SELECT * FROM hivesampletable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can still access the result of the Spark query from Scala as a Pandas dataframe!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>clientid</th>\n",
       "      <th>country</th>\n",
       "      <th>devicemake</th>\n",
       "      <th>devicemodel</th>\n",
       "      <th>deviceplatform</th>\n",
       "      <th>market</th>\n",
       "      <th>querydwelltime</th>\n",
       "      <th>querytime</th>\n",
       "      <th>sessionid</th>\n",
       "      <th>sessionpagevieworder</th>\n",
       "      <th>state</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8</td>\n",
       "      <td>United States</td>\n",
       "      <td>Samsung</td>\n",
       "      <td>SCH-i500</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>13.920401</td>\n",
       "      <td>2016-06-09 18:54:20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>California</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>23</td>\n",
       "      <td>United States</td>\n",
       "      <td>HTC</td>\n",
       "      <td>Incredible</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2016-06-09 19:19:44</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>Pennsylvania</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>23</td>\n",
       "      <td>United States</td>\n",
       "      <td>HTC</td>\n",
       "      <td>Incredible</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>1.475742</td>\n",
       "      <td>2016-06-09 19:19:46</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Pennsylvania</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>23</td>\n",
       "      <td>United States</td>\n",
       "      <td>HTC</td>\n",
       "      <td>Incredible</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>0.245968</td>\n",
       "      <td>2016-06-09 19:19:47</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>Pennsylvania</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>28</td>\n",
       "      <td>United States</td>\n",
       "      <td>Motorola</td>\n",
       "      <td>Droid X</td>\n",
       "      <td>Android</td>\n",
       "      <td>en-US</td>\n",
       "      <td>20.309534</td>\n",
       "      <td>2016-06-09 01:37:50</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Colorado</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   clientid        country devicemake devicemodel deviceplatform market  \\\n",
       "0         8  United States    Samsung    SCH-i500        Android  en-US   \n",
       "1        23  United States        HTC  Incredible        Android  en-US   \n",
       "2        23  United States        HTC  Incredible        Android  en-US   \n",
       "3        23  United States        HTC  Incredible        Android  en-US   \n",
       "4        28  United States   Motorola     Droid X        Android  en-US   \n",
       "\n",
       "   querydwelltime           querytime  sessionid  sessionpagevieworder  \\\n",
       "0       13.920401 2016-06-09 18:54:20          0                     0   \n",
       "1             NaN 2016-06-09 19:19:44          0                     0   \n",
       "2        1.475742 2016-06-09 19:19:46          0                     1   \n",
       "3        0.245968 2016-06-09 19:19:47          0                     2   \n",
       "4       20.309534 2016-06-09 01:37:50          1                     1   \n",
       "\n",
       "          state  \n",
       "0    California  \n",
       "1  Pennsylvania  \n",
       "2  Pennsylvania  \n",
       "3  Pennsylvania  \n",
       "4      Colorado  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_df_from_scala.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cleaning up\n",
    "\n",
    "Now that you’re done with your Livy sessions, you should clean them up.\n",
    "\n",
    "Simply click on the `Delete` buttons!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%manage_spark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![clean_up](images/cleanup.PNG)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  },
  "widgets": {
   "state": {
    "002f8ae4f9064d21a99b1a3b5e477aa3": {
     "views": []
    },
    "004c2f98654d4c21b0e619ed9c9f49e4": {
     "views": []
    },
    "009c4ca9cff846e68a60bbc657f65381": {
     "views": []
    },
    "00c7108e57664937b3c7c2a387e9a0a9": {
     "views": []
    },
    "00d992d8cac14b6fb53f56208890216e": {
     "views": []
    },
    "00f5950d301142ad8fb892969d94b592": {
     "views": []
    },
    "018db6fa89df437abd40973ea9eea562": {
     "views": []
    },
    "019d0b3be4bc4202b4c7acac38ed3420": {
     "views": []
    },
    "02db41ae16524c4988773ad4c86f82fb": {
     "views": []
    },
    "035c94ab02104d6e82ec7dabe8c1857e": {
     "views": []
    },
    "03a2e48efd634e3f980e2cfb3ada9cf6": {
     "views": []
    },
    "03a6eadcdc694192b26a2ad3b7fc0c1b": {
     "views": []
    },
    "03ab2b795edd4490aac7ce8f877fc6a0": {
     "views": []
    },
    "04696ca667544c69bdfec3103c1d4ff7": {
     "views": []
    },
    "046b13c070814437bb1f0f881e81b341": {
     "views": []
    },
    "04a4e4f7dbff4a678dba6936902c8f47": {
     "views": []
    },
    "04d484521e454983ad7baf39d81f0407": {
     "views": []
    },
    "04fd12a0280f4366929d04f732cdd228": {
     "views": []
    },
    "054a28c54021495789ec308b8f3d4480": {
     "views": []
    },
    "05a081b82e7a44248a670702ead6e6f8": {
     "views": []
    },
    "05f65dc0da2b4a18b5e7ed05e399335b": {
     "views": []
    },
    "06b6840f09874791aa57d97147c93430": {
     "views": []
    },
    "07b9939ac2cc4eb2a282d65606763a7a": {
     "views": []
    },
    "08c8b61556294808885a61098aa5ac86": {
     "views": []
    },
    "08f8a41c399a40c1bf8e17930fc69098": {
     "views": []
    },
    "09c9728142b74370b6d177d6bd9859b2": {
     "views": []
    },
    "0a6e74d776ca48128f74aeb1bcf5a7b0": {
     "views": []
    },
    "0bd679072aef45008ec4fd8ffacb8328": {
     "views": []
    },
    "0c677213b62c475caad8dbe92b2ab816": {
     "views": []
    },
    "0d7718a9e4144a1ca721e6039da51d5b": {
     "views": []
    },
    "0edfb4fff97a4ec8a9c5da54a9240039": {
     "views": []
    },
    "0f00b5152a284452993746a058e1118a": {
     "views": []
    },
    "0f5d611c9b294132a001bb79adfd3ce0": {
     "views": []
    },
    "0fb9c6d25b40472b9781e7aa0434e0f6": {
     "views": []
    },
    "1030af1a1acb47aa8b914ebd9d21e42c": {
     "views": []
    },
    "10956737c8b64c0aaa09a2ee747c07b5": {
     "views": []
    },
    "10b51e316f5a485ab2da0f3ce11d760c": {
     "views": []
    },
    "10be30034079447fa908cdf1f5d2e151": {
     "views": []
    },
    "10d12c2da458489394588cec6b2e3f0b": {
     "views": []
    },
    "11294cd045fc4dbabf8c8a53ec7cbc60": {
     "views": []
    },
    "117ffda0a5d145e4b5e5cf811c726872": {
     "views": []
    },
    "12a7a24cefdc494eb99cd0fb46e0f5dc": {
     "views": []
    },
    "12d698351adc4448bc3efc0c57c2ab81": {
     "views": []
    },
    "13282c201b744352ad1c2a6c22ea1c99": {
     "views": []
    },
    "137b242fbebf4b8c8b99e6ab415b967d": {
     "views": []
    },
    "13de53ca3fe24a389640a7516ce105fd": {
     "views": []
    },
    "143cb702034a440e88e68f6ae62bc032": {
     "views": []
    },
    "15754d7af7d1470a9cc21f97684e38d8": {
     "views": []
    },
    "1661a94a6ce845fdbe323cc46084a981": {
     "views": []
    },
    "16bec478883347788cc19f474713798c": {
     "views": []
    },
    "16fc6f971f56469392b4f379ddfe74fc": {
     "views": []
    },
    "17499fa9aa1a4c1bb64b37bc38cc5ebd": {
     "views": []
    },
    "1853618953434a67b21a1d9ed5ed7ed0": {
     "views": []
    },
    "18bf210df8884b37a407a058c988f1f1": {
     "views": []
    },
    "18e51416a1b74e65b0ae23b259dc8766": {
     "views": []
    },
    "198ca3addbe141bead2b47d0f638ae06": {
     "views": []
    },
    "1ab6bc4ed23c4ee5bc6872f9892a00e9": {
     "views": []
    },
    "1b8e619288e842d0a0db51f6359c0019": {
     "views": []
    },
    "1baee760b0ec4d27a37ccc6adc169e3b": {
     "views": []
    },
    "1bc5bc1c4a0143d2a792c8ea248660d5": {
     "views": []
    },
    "1bd7fa1fca284f73abccfe96acec1f59": {
     "views": []
    },
    "1d0b08f5ddeb499b9246ea2720fc724e": {
     "views": []
    },
    "1e627f7c330c40dea2f4cd85c7a1291f": {
     "views": []
    },
    "1f0c57b21b5c43d086d8e5ba58dedb13": {
     "views": []
    },
    "1f9b6b7628434a6ca51cf780373d9c8f": {
     "views": []
    },
    "1fa98a768cb44ace8a894ab5948f6042": {
     "views": []
    },
    "1fe697b0defa402fa2b61946528febfc": {
     "views": []
    },
    "20589d964e4843bb821dbad786404875": {
     "views": []
    },
    "2333daceae5b43bea7e62edb4c4832d3": {
     "views": []
    },
    "238a0a911a1349ed8380228ef3cdfc82": {
     "views": []
    },
    "23cc4384a44f4532ac5e2540533d0e47": {
     "views": []
    },
    "23cf3274a84f414c9643769286b4b155": {
     "views": []
    },
    "246eb5ef07104cdd8c6138b5f96fbd75": {
     "views": []
    },
    "2495f052b81e4a8a9feb49fca781a58c": {
     "views": []
    },
    "249b545bcee040bf90873762c057ecc9": {
     "views": []
    },
    "24df30ddcfee47f1a34f72dd624bb3cf": {
     "views": []
    },
    "24e6add1e6b94329b6147ce5217f58a6": {
     "views": []
    },
    "25f258856c464574aa41788464e8f11b": {
     "views": []
    },
    "264fdb8033e14d9ab95440b7b6be1c7d": {
     "views": []
    },
    "27699877565543a8a43d4f06244f07ca": {
     "views": []
    },
    "276e908827664d6ca836a41946575ee4": {
     "views": []
    },
    "27d87d70d7894c9d930c147e460d91db": {
     "views": []
    },
    "282884489d8740a6af022c86b9e99c65": {
     "views": []
    },
    "282b035d4ab04ca79eb343f8ddc3e774": {
     "views": []
    },
    "28438bc6e51d4c119ca45fb1f3ecb35b": {
     "views": []
    },
    "29eefb5c7c28475b818d3ad36a509238": {
     "views": []
    },
    "29efbc027eba484aab2fa1360454445c": {
     "views": []
    },
    "2a66c91e175b4e92b4e4ae76bb490490": {
     "views": []
    },
    "2a776d1c846f49cb8d728143e2a35ce1": {
     "views": []
    },
    "2bc2b0ace5354d0b883c5eafba8eef62": {
     "views": []
    },
    "2c37f830b78f410db283fde0a010e64e": {
     "views": []
    },
    "2c8b324519c14279ba138ee643f2cbf5": {
     "views": []
    },
    "2d1742064a0241c9a860081d93e8abce": {
     "views": []
    },
    "2d460f58343546418ceb2bf755aa8987": {
     "views": []
    },
    "2d98fc68931b46649de3302db7f65d86": {
     "views": []
    },
    "2de034e8ae544572887ecc83624ac3d3": {
     "views": []
    },
    "2f425327636b40d39abdc24394a31715": {
     "views": []
    },
    "3077d7a1a6e14af58ef121788892bdba": {
     "views": []
    },
    "3097db359f32428380bd84844c4179f0": {
     "views": []
    },
    "3169e07189584897975927dce5adf764": {
     "views": []
    },
    "31849bc7b28d4589836a0b6e605dee29": {
     "views": []
    },
    "322aee57512f4f5ab41eca43707c46dc": {
     "views": []
    },
    "329851df8dea468abe21ae8a0cda45c6": {
     "views": []
    },
    "3344debb3acd430faea0994bd692fe4c": {
     "views": []
    },
    "33d217abd48c4c3e83f25b735bc891d3": {
     "views": []
    },
    "33edc73d8e344c258937eeb75375994b": {
     "views": []
    },
    "34c0790ee2254a6881040daa11b9df5d": {
     "views": []
    },
    "35bdcccf9df84ef986dd590c3bc0e9f5": {
     "views": []
    },
    "35fa85a408c144e1a3c3fb03e97ec67f": {
     "views": []
    },
    "362c101871774640aaef37c497d719f9": {
     "views": []
    },
    "363577c09cd1475dadf371d1b097608e": {
     "views": []
    },
    "3835d3b5959544bc8b00dfcdc981c438": {
     "views": []
    },
    "38b9afbba069482d8db02aa976e29827": {
     "views": []
    },
    "392548f96b7249049480dfdc2666bada": {
     "views": []
    },
    "3aa855c83cbe448cb94b45db33d4182d": {
     "views": []
    },
    "3aade98a21954759b4d434231e4a2875": {
     "views": []
    },
    "3bad19589d8e47c4a022986c6ca1b428": {
     "views": []
    },
    "3c5392c494fb44df82aa6ee566c91fa7": {
     "views": []
    },
    "3ce3aaae5783451a98e605cbd7ea8b61": {
     "views": []
    },
    "3d38096eb76247378bacdc3e7b70d75b": {
     "views": []
    },
    "3d774014e1d24bf9a3f2d74389bee570": {
     "views": []
    },
    "3de944a68eb84bc592b1d3f14725dc8f": {
     "views": []
    },
    "3eb0ac1b71944f7982acef824529c3ea": {
     "views": []
    },
    "3f9c8628ceee4ecaa74a7ef36225532d": {
     "views": []
    },
    "402d7bd614584f5aabc2b02a2c45bcb6": {
     "views": []
    },
    "4049067402d84dfaa1a12684449dc53b": {
     "views": []
    },
    "40888c9d2ccc47a0a88143ab40d175fc": {
     "views": []
    },
    "415abd94649241d4a9d3037131679cf8": {
     "views": []
    },
    "422f739af31a4e19b1ec558c05692e6b": {
     "views": []
    },
    "42b424b4eafb43a2b7604dbecab792e6": {
     "views": []
    },
    "433b70114bf848089eed2a346559b0f5": {
     "views": []
    },
    "43e77dfd670e4cfda1678637cf93cb7c": {
     "views": []
    },
    "4448c88750ea4e3b9f6839342d4627f9": {
     "views": []
    },
    "444fd0b6cbdd4855b09919b11183ba33": {
     "views": []
    },
    "445cb7eeaaaa420fbc58e1f884f99f2f": {
     "views": []
    },
    "44c235f9720e48648a90bd8ad90f1119": {
     "views": []
    },
    "44f8efaec90d4c05b82e028b06f7c1d1": {
     "views": []
    },
    "45582badaca0467690a10655931e6acc": {
     "views": []
    },
    "4575b6e315d64062beea7b0c2f10ab95": {
     "views": []
    },
    "459cea80c3124785a13d8b49ef866c8d": {
     "views": []
    },
    "4672e5bda4d8471baaef1a70c90bcf5e": {
     "views": []
    },
    "46917e92c24749dfa76c051a66f39476": {
     "views": []
    },
    "4696dfe7fd724a2d95e382a0fd335a6f": {
     "views": []
    },
    "46b8549d804a4fdc975086e0fec5143a": {
     "views": []
    },
    "46c5ed588ce2404881003e8ae0f5e90b": {
     "views": []
    },
    "4701b38eeae44891b444bb5d95097eb7": {
     "views": []
    },
    "473be50b845e4aa9ad488fd564657fdb": {
     "views": []
    },
    "47b308b300a1422ab1076b5cb2414b67": {
     "views": []
    },
    "481a458e68604b428c70dffe4bb9528f": {
     "views": []
    },
    "491f59eeec8b4451b4100b534130acad": {
     "views": []
    },
    "4975b31264f043ecb410d2adeab39e6c": {
     "views": []
    },
    "4a71e8b610bf410eb4f0d2ac9167dbcc": {
     "views": []
    },
    "4b044052e7e44688a7532662638cfd49": {
     "views": []
    },
    "4b1b84c19e544868b970c9ce10c1f2f8": {
     "views": []
    },
    "4b50155fcd4d41969f218edc1eab9998": {
     "views": []
    },
    "4bf025112f8d471285173d340cc4ca0a": {
     "views": []
    },
    "4c4727b4f60e4919894c916efbef41df": {
     "views": []
    },
    "4dc406d1f5c8408c8b5d47a094404a37": {
     "views": []
    },
    "4ddebd9065534901beed6503f6291fae": {
     "views": []
    },
    "4f759d4c91124ad786bf2c9386bb3375": {
     "views": []
    },
    "4fdf12b30d234288ba0ef0bb760ab988": {
     "views": []
    },
    "5027215f4ce84a0ba00321fec1e612c7": {
     "views": []
    },
    "50f19f55dccc4c3b9b44a1c320c5b008": {
     "views": []
    },
    "517a453018db47d0b507295272fdcebe": {
     "views": []
    },
    "5256d05ab29d49bdb503f02481e5c4c1": {
     "views": []
    },
    "53051047cbb341dc9fa87a087ca93cbd": {
     "views": []
    },
    "5317a548c0434bf0ba205331f6d8598b": {
     "views": []
    },
    "542423514d3645abbd350fb413621223": {
     "views": []
    },
    "546e50c7c2f946418e907ca7824250c4": {
     "views": []
    },
    "548779cf3af64522b2d0ad0d2c8133bd": {
     "views": []
    },
    "5596337ff6864cbd9eb46154b461f189": {
     "views": []
    },
    "55a6cd5ce243492d8063c8d0c1e0d929": {
     "views": []
    },
    "55f8811663074906865afe5e9181b92e": {
     "views": []
    },
    "56d8825e4b3c4faf81fb8195985c47a6": {
     "views": []
    },
    "570d2c105a544fa783294a4450929ee6": {
     "views": []
    },
    "570ed0bbcc364bc0942805ee4408167b": {
     "views": []
    },
    "572ae3f59f8140a999f62af27433c975": {
     "views": []
    },
    "589166c6504747259e8cb1fd9b49b4a6": {
     "views": []
    },
    "59391b7deded46bab2e797a2f69bfd34": {
     "views": []
    },
    "5a6dc19dcfe24ff4b8e3c584e711fb61": {
     "views": []
    },
    "5a6fe61218734b4a853f39beed16d8e4": {
     "views": []
    },
    "5a76e0a8b250439aba0aa5542304cefb": {
     "views": []
    },
    "5c625be86290481c8b090ada16146417": {
     "views": []
    },
    "5d3afeef0d074f6e9e64d72dd74f29c0": {
     "views": []
    },
    "5d48ace46b5d45ce9b24cf5acf67ad0c": {
     "views": []
    },
    "5dfcee09b67b4a3ba04bd63415d7225b": {
     "views": []
    },
    "5e08e21398734a7e8db6ae506a20cd0a": {
     "views": []
    },
    "5e42397e8d6f458cbcd370e69d47f8c4": {
     "views": []
    },
    "5e5c63507657454b98fc57dfe35bf936": {
     "views": []
    },
    "5f654c7a180147258e05b6c9da016de4": {
     "views": []
    },
    "5f810a7855f346b38d82280605b7c74e": {
     "views": []
    },
    "5fe5e92541534bd19c15248f7d5270ef": {
     "views": []
    },
    "5ffb942be7e744ac9a5d9b861e046069": {
     "views": []
    },
    "6085f053623b4c7cbd5292e31984bbae": {
     "views": []
    },
    "60a6717e2fdf4de38b40d0c645397b44": {
     "views": []
    },
    "60af173b13b241fc93d18514a743ccad": {
     "views": []
    },
    "60fc7fe6ff614afb8b9910e54e32039a": {
     "views": []
    },
    "61ca149f3e5847f49020b6fd6b8997d8": {
     "views": []
    },
    "623c022f68534a58aa870a6ba3cea641": {
     "views": []
    },
    "62d045f626ae4b399fdcbd4104e24d85": {
     "views": []
    },
    "63fd19904cf346cbbc4f4c3ac642bf42": {
     "views": []
    },
    "6403964963be4611a9f9cc534d6314c3": {
     "views": []
    },
    "6455039fbc244720b3b8e30277a73104": {
     "views": []
    },
    "64e752cbf7d241c491483595c6db885f": {
     "views": []
    },
    "64e89e2e7531477b8d2c441d7b2e316b": {
     "views": []
    },
    "65b90d8275eb4dc98aca60b1200d2bbb": {
     "views": []
    },
    "65da2fd144d042aa8029753a14a1c62f": {
     "views": []
    },
    "65fc8ae2fa9249d3a87eed74b90c3f7f": {
     "views": []
    },
    "663154f2a2834b968b8ba1e8fe715212": {
     "views": []
    },
    "6642f76bdaa44e69b9031e107ddcad40": {
     "views": []
    },
    "6683c6256f35443d8a5ac6129a570703": {
     "views": []
    },
    "670ca1732799483aadbf1a0efee67f0f": {
     "views": []
    },
    "671e73335eaf4637b36183453662b5f6": {
     "views": []
    },
    "672a0d46f9e444dba91a60b0dc077ca8": {
     "views": []
    },
    "674ee5fbb23a40778c7dbd6b213671ad": {
     "views": []
    },
    "67968ff65c8447aa92f88077ee31999b": {
     "views": []
    },
    "67d2296927b04999abc4e31de12649d9": {
     "views": []
    },
    "67fb241301f548ef988403c333f59b0f": {
     "views": []
    },
    "685b9f522b88435ea64e43462bdccd73": {
     "views": []
    },
    "68bd27b63aa64df3a541de2bbf317194": {
     "views": []
    },
    "68c332448b4549639a4242d827eb13bb": {
     "views": []
    },
    "6b1aabe535924dd79317641ab314bd1d": {
     "views": []
    },
    "6b2a7e2dd4d7425fa71309c300460225": {
     "views": []
    },
    "6bce7fd4c92f405082e36f6e652ac52e": {
     "views": []
    },
    "6c287ca8c86046ab8c67641cc7d497f7": {
     "views": []
    },
    "6cae767bd0444a0793028a847a35716a": {
     "views": []
    },
    "6cb2d3d2f1d9422bb30ec830e3ca4656": {
     "views": []
    },
    "6d79c61a8f6f4229a1489a47a0a56c51": {
     "views": []
    },
    "6d83179436ab478fad16b3f21f08e228": {
     "views": []
    },
    "6dba3f2ca39d497f8afcd9fd8dc1375e": {
     "views": []
    },
    "6dbf6318bb994343b141a1b17809faf8": {
     "views": []
    },
    "6dfcb5f7979a4eeaa0eecadf46c50ecb": {
     "views": []
    },
    "6e17c3c177974b0db1b5bedffbd5abb7": {
     "views": []
    },
    "6fa1ee774231417a84f0838de35109f5": {
     "views": []
    },
    "70ab4433d1214b09b848c7107c1eb181": {
     "views": []
    },
    "70ddb0cc4abc4006bca26ac54b64ba23": {
     "views": []
    },
    "71f7e418ed7f4903aa41dce7b01529cb": {
     "views": []
    },
    "72dc6d4e381040749e0658249313aae7": {
     "views": []
    },
    "730077e765794fcbad4dfb0abbf93797": {
     "views": []
    },
    "7342f7075fc949cdb9ea7e0e4c4b1252": {
     "views": []
    },
    "73aa7e6af0ef4aafacb383e609955996": {
     "views": []
    },
    "747ec4b3b1574357a65da93186074953": {
     "views": []
    },
    "74ad9c932cf4460db73aa6e6b7b081fe": {
     "views": []
    },
    "76df612393c54c71826fccd8039b3803": {
     "views": []
    },
    "777dcf72b5564ddeb4fd2949670308d3": {
     "views": []
    },
    "78a6e9c0747b448ca9fddf5a7d750b2e": {
     "views": []
    },
    "795c25007857492b9f26e5d831bfb496": {
     "views": []
    },
    "7a7615fb4a3a43d2820f19ab0f5417e0": {
     "views": []
    },
    "7b13fc0849ab44b0ae4560916f880cdd": {
     "views": []
    },
    "7b1d7be6a10c4b378bd899c08879257f": {
     "views": []
    },
    "7bd220e11804440686db7cca4c2425a4": {
     "views": []
    },
    "7c1c03f8a35645ef875b14cb3f3ebb4b": {
     "views": []
    },
    "7c400f5db4a64a5aa44ccb62dea332a9": {
     "views": []
    },
    "7d75d4f1ca4d4995974dedd35995d7c0": {
     "views": []
    },
    "7dc224ea134d4645a1010b347ced01e2": {
     "views": []
    },
    "7dc22cc9c07d40489216512eac46182c": {
     "views": []
    },
    "7ddfccf431174040a1e1c82e81864466": {
     "views": []
    },
    "7e784a69a23c41429ca9bc6515d36856": {
     "views": []
    },
    "7eacf8bae8e04943a985e0c0e522424d": {
     "views": []
    },
    "7ed6ab5e64234726af18f0cb3d72ff78": {
     "views": []
    },
    "7f34bd72ce984ef2a31bc4762d40840a": {
     "views": []
    },
    "808b862d23e9452caf0ae47d1b9f4b01": {
     "views": []
    },
    "80ead7e851af4a62a29e171393140ff6": {
     "views": []
    },
    "82c4f335b3c64979ab59f42239a4e422": {
     "views": []
    },
    "82e1594827514f5f93baf32fef3c4355": {
     "views": []
    },
    "83117696251c4149ad98cff1d7ea99dc": {
     "views": []
    },
    "8354f4a445db4e71bf934e20b0a68e60": {
     "views": []
    },
    "83f5876c25744d36a3ae39e07c9ad705": {
     "views": []
    },
    "845ee392539d4efdaeb34a9ea945f6c3": {
     "views": []
    },
    "84d0ea03663244759f5aa32a0bdfeefe": {
     "views": []
    },
    "855d9d01c57841cb9e060f8365e8bda4": {
     "views": []
    },
    "8586a4d3cce4412bbf6bd070206962b9": {
     "views": []
    },
    "8587971c26ca46539229a092eff9af5f": {
     "views": []
    },
    "86076e49796f412ea0b5040037a6ed65": {
     "views": []
    },
    "8663169e1df3448ba9ae8867118daa0d": {
     "views": []
    },
    "87446f9e85954d1eb5a6459fc25c87d0": {
     "views": []
    },
    "87451e56ebc2445d8d0604ef9abd8921": {
     "views": []
    },
    "88dad515792c4dc0a0beb8868fe3d5a2": {
     "views": []
    },
    "88f8b30e65d94ffaa3ad4a42edd6d9af": {
     "views": []
    },
    "8956b2ba91ff49eb98900ef5ad76a9e6": {
     "views": []
    },
    "89ce1ebf75b74a34a54acee9a7957d9d": {
     "views": []
    },
    "8a4880dfeaaf4b7ea1624b04719afbf8": {
     "views": []
    },
    "8a965e13dfc6462d9b51124197b40e44": {
     "views": []
    },
    "8b031e693b5144c389079df0c5fe9bcc": {
     "views": []
    },
    "8b153feb1a9f4071a5a6276bdbb61fe0": {
     "views": []
    },
    "8bab50e956d14f4c9cec96362e20eea9": {
     "views": []
    },
    "8beb4a359bbf40b0a092ea80349dda73": {
     "views": []
    },
    "8ce77cdf3bcc49b7b4dd0f705b3b283e": {
     "views": []
    },
    "8d9edff75e754616b1cb04c3c8820dd4": {
     "views": []
    },
    "8dbcae4a829b4eccb1925e8c3af261c4": {
     "views": []
    },
    "8dca34d52cc74dd8b727de4c3066c3e1": {
     "views": []
    },
    "8e72b91a7a6b4fb889edcb2f57ffee0d": {
     "views": []
    },
    "8e89167d51d64dd4b4def1287a5f9e70": {
     "views": []
    },
    "8ed2bfc5f0f14b889528a0f1a277df73": {
     "views": []
    },
    "8f513db325fc421497ee709831a1c2f5": {
     "views": []
    },
    "8fec883cb038446a9c4d6db8e1770f3d": {
     "views": []
    },
    "904097ea22684113b5080684929937b5": {
     "views": []
    },
    "908cc4dcbdd847e29f6a8343b580a1ca": {
     "views": []
    },
    "90e8be9783004ff4a8d72432f3f9e205": {
     "views": []
    },
    "9261f331e1a745879d54c764522a8f86": {
     "views": []
    },
    "93469278a0474154bb02b9c50a893e91": {
     "views": []
    },
    "938c369269d5489f8c77193bfc71d20d": {
     "views": []
    },
    "93ef853b3cd34ad39102b0e31c0c990f": {
     "views": []
    },
    "9487b8da5d604db8881dcd7e32a8c76c": {
     "views": []
    },
    "94e076fece7f44798c862b27475fd6c0": {
     "views": []
    },
    "9549aa1af9da4326b8beda04cc9965df": {
     "views": []
    },
    "9553fb427cd44a749bd738f7c3e97d9d": {
     "views": []
    },
    "971407e795ba4a0683242d7bd560e759": {
     "views": []
    },
    "97aedcaa2584402196bed40dffd6db2f": {
     "views": []
    },
    "97ec777d6ea3466eaa925e751d213738": {
     "views": []
    },
    "9a41c6ea99b64d80bf3d415ad0af4e30": {
     "views": []
    },
    "9acfbb3334804de6bae91534c989ca5d": {
     "views": []
    },
    "9b87a37acfe94f64a68178320f4cf442": {
     "views": []
    },
    "9ba03755d3ed4d20ae713d31c6c72bb2": {
     "views": []
    },
    "9bb3d7ce1cef4282947a22e3b520ebfd": {
     "views": []
    },
    "9bc34217f77541a0b501da965f04d446": {
     "views": []
    },
    "9c8ced28a4b8408894300df6a0b74a63": {
     "views": []
    },
    "9cb8dbedaa1f4cf191e41ef1bbff56b7": {
     "views": []
    },
    "9d1f7bf45ac44fda957a52b010886774": {
     "views": []
    },
    "9d471f6a96314aab93140e3a90986acc": {
     "views": []
    },
    "9db39ad0952142a1bac85999874b11b2": {
     "views": []
    },
    "9ee4abf8a8194b8d8296ccdc3c1d3a58": {
     "views": []
    },
    "9f47e144552f48569e862eb418004f29": {
     "views": []
    },
    "9fe674c4ddb747d7916bc61bd1230871": {
     "views": []
    },
    "a10eba8636b444d6a31012b191e8fa07": {
     "views": []
    },
    "a1259877889b495d85e6ce12e750b89b": {
     "views": []
    },
    "a187cbf0b95d42ec84ae60b4be650f3d": {
     "views": []
    },
    "a1885ce9d42c4421909f530920525de1": {
     "views": []
    },
    "a1b7d45189a441819ebe16d49e1e8aa9": {
     "views": []
    },
    "a1c82b3f292244a5b60b9de436121f88": {
     "views": []
    },
    "a25ab1dc75164dd3bdde1a3dd4f4edd6": {
     "views": []
    },
    "a2665683a88c4f24aa054a937672955d": {
     "views": []
    },
    "a3c14f14214248ccb8919d438b539ef8": {
     "views": []
    },
    "a538dadcd7754de2afc5fd16672d0de3": {
     "views": []
    },
    "a58ab9363f7541338b282a6e81d03c47": {
     "views": []
    },
    "a5ce36495fb8438d87f2f3523aa64f04": {
     "views": []
    },
    "a6ec33b1b5174f35b6c938373c122a98": {
     "views": []
    },
    "a720092102ae4c9691f136a67ce71f12": {
     "views": []
    },
    "a771e1949812402aa5d39635f57432ce": {
     "views": []
    },
    "a8413baecf71466aaf62b79e190d6a06": {
     "views": []
    },
    "a94aa3138a4941f6a909f0fe35510a9e": {
     "views": []
    },
    "a9935ae41a7143d1b6405adca33713c9": {
     "views": []
    },
    "a9c34bb9b2cb489eb00cd888dfe3823b": {
     "views": []
    },
    "ab3a517dbc0a4b03affb4dbc512e8d73": {
     "views": []
    },
    "abb433f63cf5451080a0d54e1ab8cbaa": {
     "views": []
    },
    "abcba84c7b4c44198397065c6e0f2f92": {
     "views": []
    },
    "acfd856a7c9f400a8914f0ecc3790508": {
     "views": []
    },
    "ad4c647f8f3343fc9d4b43534c585588": {
     "views": []
    },
    "adb449d9448b447592bb6a95bfc7337c": {
     "views": []
    },
    "adfeb551fe1c47bf8eabdfedd76d7fd1": {
     "views": []
    },
    "af68ea7411de492488055fcade00861e": {
     "views": []
    },
    "b100927498244b3c94965c4e024fb7d8": {
     "views": []
    },
    "b1b8d8fb420d425681f81d0827be57fb": {
     "views": []
    },
    "b1bb0ce4e0a14099b64ad266a85f5441": {
     "views": []
    },
    "b1e81131ec9440618c236c3cd1e17c3c": {
     "views": []
    },
    "b2a66a13c82240c5912da1fcde674bbe": {
     "views": []
    },
    "b332ddc932d441c5876ec52d7ee0def6": {
     "views": []
    },
    "b35d8e6c4cbc43e5a14616d38eba36c9": {
     "views": []
    },
    "b4c726ddd4a64523a545f4c500e9c014": {
     "views": []
    },
    "b4cf431de33449d98c64e4b13d18c935": {
     "views": []
    },
    "b5c649812c23466aae8eb78fc0236e31": {
     "views": []
    },
    "b5d23520dc014e40a59696c38b942337": {
     "views": []
    },
    "b5e48857b2734c6e859242629fa251a8": {
     "views": []
    },
    "b6458f0a953646db8f6e880b5faad2af": {
     "views": []
    },
    "b65dbc60d4c44ac3b00d6fb82193a102": {
     "views": []
    },
    "b6e97a2306894a278c693c7198b352ad": {
     "views": []
    },
    "b766ac82ba0a4829a0e28cd4253d65ea": {
     "views": []
    },
    "b771e402dd7b4249b9fc0b680075122f": {
     "views": []
    },
    "b7f05f9c1de547ba983dbad0c7b4ae13": {
     "views": []
    },
    "b850fa1c709b4279900c9f11a87ea146": {
     "views": []
    },
    "b8eb234453e84a5a9a5fa765f4a5d1cb": {
     "views": []
    },
    "b9d8ab3e20c44357abb11c99b9f7c36e": {
     "views": []
    },
    "b9f12425601b416d956a0bb768f88e54": {
     "views": []
    },
    "ba719d94af224d6eaf7bacdea87d344b": {
     "views": []
    },
    "bb2908860204416fb59ac36d727232de": {
     "views": []
    },
    "bb4b0b69ff9442458c3c5c6dccb4880c": {
     "views": []
    },
    "bb6e18a974ed47289a5b51f2aa739dc5": {
     "views": []
    },
    "bccc58aafdb549f8bcb852041f6be337": {
     "views": []
    },
    "bcec7b62b9e4469a9e62d92ef585464d": {
     "views": []
    },
    "bd24c08617314e5aaae77b0d25545f9e": {
     "views": []
    },
    "bdb7d64997994ba69c62470cfb60858e": {
     "views": []
    },
    "be1f132405c240ad8ab5e4f5ebb9e745": {
     "views": []
    },
    "be6a0890a32c4698ac978b197266510b": {
     "views": []
    },
    "be91a016f0e6463ba1eefb4eb3efb997": {
     "views": []
    },
    "bed02ab3a20d4fa0ae848fc7b4479652": {
     "views": []
    },
    "bf21ae3564f046ab86d90a034945ffd6": {
     "views": []
    },
    "bf2e3202cbd943a6af473a804bd2b6f2": {
     "views": []
    },
    "bf904c08b2bf47b190cda19f213565da": {
     "views": []
    },
    "c1589808d9e2467680d036a37105b5b1": {
     "views": []
    },
    "c21ad1c067a24277b2d3fed42970427d": {
     "views": []
    },
    "c3a0851ef9554a4fad467fbb0fd2a369": {
     "views": []
    },
    "c3b565c39dad40c28b67a41a0a26e79f": {
     "views": []
    },
    "c47ecb8c98684578af14f63e29828562": {
     "views": []
    },
    "c484a8dafa3d4563b6148076858e33bc": {
     "views": []
    },
    "c50e55e118fa420398ab5466bd6c6cee": {
     "views": []
    },
    "c51df391bb7e466c9b3be6162286a432": {
     "views": []
    },
    "c52566dd0b194d5b866ca8c20bf0748e": {
     "views": []
    },
    "c5598250b79241f7a55ca0573cc9da2d": {
     "views": []
    },
    "c58ac451c30446e9804247cd39ee7ba0": {
     "views": []
    },
    "c599fcb19de44d509f041d92820adf0a": {
     "views": []
    },
    "c71febd4d3a84a62a6bd48867ba34e1a": {
     "views": []
    },
    "c7a30d6a6f23417e8540ae786054484b": {
     "views": []
    },
    "c988e0f363544fdfb70916e1bb199f4d": {
     "views": []
    },
    "c98c2c53733942d6a87af3481cd17e42": {
     "views": []
    },
    "c9a6e51954354815ada1faba970a77f1": {
     "views": []
    },
    "caace157e7b947c683dfa4d1199d7e4c": {
     "views": []
    },
    "cae79ced27084b7ea6491e285b8269e2": {
     "views": []
    },
    "cb0d745166284fd893667411119516ec": {
     "views": []
    },
    "cbb826cb40c446de9c9f60cd35e8007e": {
     "views": []
    },
    "cd3d3d45091648ba924445fd0ef58806": {
     "views": []
    },
    "cd5b82ef8f8c4453954f65663acce521": {
     "views": []
    },
    "cdfb1960ddd94d8384856b603c146140": {
     "views": []
    },
    "cefdb90b823949edb8cc828dee33c421": {
     "views": []
    },
    "cf0acaebe42642069ca503d9974277f6": {
     "views": []
    },
    "cf50521ae02146b3b139a55c0f1b9be2": {
     "views": []
    },
    "cf95457cc4d24bb6a478fcfc6350c4bc": {
     "views": []
    },
    "cf95a7ebf1d04a10802d76fc1fba0a16": {
     "views": []
    },
    "cfd91142673745088428f7d7ddb813f5": {
     "views": []
    },
    "d047fb6ab9ba47a4985417d5fc073b99": {
     "views": []
    },
    "d228512ec637477d959cae10453e4147": {
     "views": []
    },
    "d23e27d78a4b4239ac1f2ea9cb0380a3": {
     "views": []
    },
    "d290e78eb99e446588972989163e2dee": {
     "views": []
    },
    "d37ffb626d0c405cb43a1a70b1cd48d9": {
     "views": []
    },
    "d3944b93c9bc4f07b4f14e6c3b7e2455": {
     "views": []
    },
    "d3c1f01d8de9474c8adefe5f9628ed2c": {
     "views": []
    },
    "d411ce291f19460fb415cde6bffca849": {
     "views": []
    },
    "d577384c378e4f679de59952b2883608": {
     "views": []
    },
    "d5a49c390bb84f7a9aa1d615eca3aafe": {
     "views": []
    },
    "d604bee51f274eaf9ecf52aa4ea2f979": {
     "views": []
    },
    "d614e53a114f4d378526a8ae32a5905a": {
     "views": []
    },
    "d66af11efe274c37adfc09bfac2bf5c0": {
     "views": []
    },
    "d6dfdcabe4084df6968595b09cf7667a": {
     "views": []
    },
    "d6eb5bfab110447d87b8f6e0c1c3cf85": {
     "views": []
    },
    "d7603a0318694b72aa57799d224f7d0f": {
     "views": []
    },
    "d818eea5d08b480b956a67399068deac": {
     "views": []
    },
    "d84eb6a1eb414b729f7d22f886c81ceb": {
     "views": []
    },
    "d978c26cbf894e41b6b380c58e4c91ff": {
     "views": []
    },
    "d99df7e1d52d4d05a4ae85c83bec631f": {
     "views": []
    },
    "da405edb34234f3394ba6e13ab446e9c": {
     "views": []
    },
    "da5bb9accec448a6a8a61d2209f7627d": {
     "views": []
    },
    "dc1e0be9f82d4270bdb56e8d64083ffd": {
     "views": []
    },
    "dc28c0ec49644be4951648af53788a25": {
     "views": []
    },
    "dd10c5a44524444294dc3cda692ec20d": {
     "views": []
    },
    "dd5980cf0367466796862a73d6a94795": {
     "views": []
    },
    "dda953e815fa4c2680e0d2662dece6ff": {
     "views": []
    },
    "dedd17c82c174700b79b19926bd06894": {
     "views": []
    },
    "df02c4a87f04406292f968bde14516cf": {
     "views": []
    },
    "df3bb1376a854008bd572ff126ad56ef": {
     "views": []
    },
    "df3ea6e6efb348e7a997033d27d334c9": {
     "views": []
    },
    "df6ec0e521b546cf87339de430e08dcf": {
     "views": []
    },
    "df871ce7b5fa43999cd43ed47e911d32": {
     "views": [
      {
       "cell_index": 26
      }
     ]
    },
    "df88f9a4978b4eeb86885d7e4d3364ed": {
     "views": []
    },
    "df93b5a138e549f1a619fb8cf9df8b2b": {
     "views": []
    },
    "df93f56ed887465ebca5255f3f42c41e": {
     "views": []
    },
    "dfab6549731c47e49818c9b39fee5f61": {
     "views": []
    },
    "e0f6cfe9fc5e436dbcff43c60c66b499": {
     "views": []
    },
    "e1562c0f6cc44224873c729ba0b11ef5": {
     "views": []
    },
    "e19b9430857b4a66a9ef81480641944f": {
     "views": []
    },
    "e220878bc02f4281981a30f243d2d6f1": {
     "views": []
    },
    "e22f0223f90a4f04960687427b29858a": {
     "views": []
    },
    "e2bf345d46744a6a896e6d20cc3f4856": {
     "views": []
    },
    "e3306f5a9a9a435dab1f31e6058a7f53": {
     "views": []
    },
    "e3ae4bb0673d4aeda958bcbb56b969e2": {
     "views": []
    },
    "e58532e0ab5b4b4286f63e2fc1351ace": {
     "views": []
    },
    "e58c119ec1ef475189475e16946ad442": {
     "views": []
    },
    "e5ba48c814ae4ca6b062ffb9823783ab": {
     "views": []
    },
    "e6d6f73994ce4874999ec3451381fafa": {
     "views": []
    },
    "e6e3fa69789e472682739ffd832acedf": {
     "views": []
    },
    "e76a005db4994331a4e7f09e1c9490fd": {
     "views": []
    },
    "e8ded0a47d434e488f8b92f330502082": {
     "views": []
    },
    "e96f3ec1301f403a9e95bbfb27441fa8": {
     "views": []
    },
    "e98811e5aead4198a3a299a86588d7c7": {
     "views": []
    },
    "ea78e6557e504e89a472a3367d777b0a": {
     "views": []
    },
    "ea8e0a32e83a41dd92dbe3c612dd79ca": {
     "views": []
    },
    "eaf84aaa0d92444f85493552d027b4ea": {
     "views": []
    },
    "eb1b3305ccc64b78990a8c975927375c": {
     "views": []
    },
    "ec7272b56f1241e2a6773609743350fb": {
     "views": []
    },
    "ece58ce41a1e475885422305f685a8f8": {
     "views": []
    },
    "ecee1ac34a0745509ccfd98d48588843": {
     "views": []
    },
    "ed1947f3ef3a480a9d51371761e1a12a": {
     "views": []
    },
    "ed2eb4d74dc94954ac03496fd46cc126": {
     "views": []
    },
    "ed318f2d7721427985a6c3cc64f97a09": {
     "views": []
    },
    "ed501b37aa824c6e8940fca9082f5e81": {
     "views": []
    },
    "ee1ec3813476421287e392bc2d4e74f7": {
     "views": []
    },
    "ee212adcb5c845f29af37494747d5530": {
     "views": []
    },
    "ef31b74a2ea24bf484dbd62149c1bebd": {
     "views": []
    },
    "ef82a0f368f743d59b6cc75de1decc57": {
     "views": []
    },
    "efbd0d44cad441df8492b584c28c7672": {
     "views": []
    },
    "f16f28922f0a4c78bf54e685f7ef3ac1": {
     "views": []
    },
    "f267c28fbc0440b190154ab917b980c5": {
     "views": []
    },
    "f52fd702226245d9bbeaf6c7300b46e4": {
     "views": []
    },
    "f547f55ab16a4cd389643f2b0ba46c3c": {
     "views": []
    },
    "f570ef7cde684202b423ebe3a44892f6": {
     "views": []
    },
    "f7ef10df938445a39a67b50a3f967c75": {
     "views": []
    },
    "f7f529ddfa6b4127824238becb344699": {
     "views": []
    },
    "f84450459bb14dc8bfc2b482d37545b7": {
     "views": []
    },
    "f864605325634aaf858a83b9db62404d": {
     "views": []
    },
    "f94155f1363a41e897da6378f2f09326": {
     "views": []
    },
    "f95438ae6d634279a27fd1388c8e3222": {
     "views": []
    },
    "fa09e12bbe444d4ebdc674ff04a08ff8": {
     "views": []
    },
    "fa7cedbf8bd64bee9d6cbd3c4d117e32": {
     "views": []
    },
    "fac54524e99e46b5b4d983c6f67b9d6b": {
     "views": []
    },
    "fc595dd1ca2a489b8032c7483aa64e66": {
     "views": []
    },
    "fc97f733dd684a55b455c07fb694df4c": {
     "views": []
    },
    "fca6520357824983a8e333a075ee380a": {
     "views": []
    },
    "fd1b4f0486614e7cbd134f9e6aba8245": {
     "views": []
    },
    "fd514058da1a4ec3a4a7ad6ce33fec2d": {
     "views": []
    },
    "fe29a1533c864679bba888bc43f36321": {
     "views": []
    },
    "ff2b812935e545b685aee08298c8e394": {
     "views": []
    },
    "ff8d85998641490495da0b85d54f9204": {
     "views": []
    }
   },
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}