monary.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:cdd92477a18ef68f6193046382f526e5a9fe0b0311322c0b27c2ac63fe36ea63"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Where did we just go..?\n",
      "\n",
      "### This is an *IPython Notebook*!\n",
      "\n",
      "### http://ipython.org/notebook.html\n",
      "\n",
      "<br><br><br><br>\n",
      "\n",
      "Our data set: NYC Taxi Trips of 2013\n",
      "====================================\n",
      "\n",
      "You can find the [NYC MTA Taxi Data](http://chriswhong.com/open-data/foil_nyc_taxi/) at http://chriswhong.com/open-data/foil_nyc_taxi/.\n",
      "\n",
      "<br><br><br><br>\n",
      "\n",
      "Connecting\n",
      "----------\n",
      "\n",
      "First, let's create a connection with Monary.\n",
      "\n",
      "<br>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from monary import Monary\n",
      "\n",
      "client = Monary()\n",
      "client = Monary(\"mongodb://localhost:27017\")\n",
      "db = client.taxi"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br><br>\n",
      "Retrieving Data\n",
      "---------------\n",
      "\n",
      "As an example, we will find the average distance traveled by cabs:\n",
      "\n",
      "<br>"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Simliar sytax to the mongo shell: db.colleciton.find(..)\n",
      "\n",
      "distances, = db.trips.find({}, ['distance'], ['float64'])\n",
      "distances.mean()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br>\n",
      "\n",
      "How many were there?"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "db.trips.count()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br>\n",
      "\n",
      "So that takes up a lot of memory. Can we do it in chunks?\n",
      "\n",
      "Yes."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "avg_distance = 0.0\n",
      "num_trips = db.trips.count()\n",
      "\n",
      "for distances, in db.trips.block_find(\n",
      "    {}, ['distance'], ['float64'], block_size=10000):\n",
      "    avg_distance += distances.sum() / num_trips\n",
      "\n",
      "avg_distance"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br>\n",
      "\n",
      "Great! Even better, all of this data is in NumPy, so we can make graphs with matplotlib:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%matplotlib inline\n",
      "import matplotlib.pyplot as plt\n",
      "\n",
      "# Remove masks to prevent skewing the histogram.\n",
      "data = distances.data[~distances.mask]\n",
      "\n",
      "# The histogram of the data.\n",
      "plt.hist(data, 100, normed=0,\n",
      "         facecolor='blue', alpha=0.75)\n",
      "\n",
      "plt.xlabel('Distance')\n",
      "plt.ylabel('Number of Rides')\n",
      "plt.title('Distance of Cab Rides')\n",
      "plt.axis([0, 14, 0, 2300])\n",
      "plt.grid(True)\n",
      "\n",
      "plt.show()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br><br>\n",
      "\n",
      "More Computations Using Inserts\n",
      "-------------------------------\n",
      "\n",
      "With Monary Inserts, we can store intermediate values in a new collection:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "num_inserted = 0\n",
      "\n",
      "for tips, total_fares in db.fares.block_find(\n",
      "    {}, ['tip_amount', 'total_amount'],\n",
      "    ['float64', 'float64'], block_size = 100000):\n",
      "\n",
      "    # Calculate the tip percentage.\n",
      "    percent_tip = tips / (total_fares - tips)\n",
      "    \n",
      "    # Insert the values.\n",
      "    ids = db.tips.insert([percent_tip],\n",
      "                         [\"tip\"],\n",
      "                         [\"float64\"])\n",
      "    num_inserted += len(ids)\n",
      "\n",
      "# Prove that it worked.\n",
      "(num_inserted, db.tips.count())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br>\n",
      "\n",
      "Now we can use this data."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pct, = db.tips.find({}, ['tip'], ['float64'])\n",
      "(pct.mean(), pct.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br>\n",
      "\n",
      "We can draw another histogram:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Remove masks to prevent skewing the histogram.\n",
      "data = pct.data[~pct.mask]\n",
      "\n",
      "# Remove outliers\n",
      "data.sort()\n",
      "data = data[:-600]\n",
      "\n",
      "# The histogram of the data.\n",
      "plt.hist(data, 50, normed=0,\n",
      "         facecolor='red', alpha=0.75)\n",
      "\n",
      "plt.xlabel('Tip')\n",
      "plt.ylabel('Count')\n",
      "plt.title('Distribution of Tips')\n",
      "plt.axis([0, .5, 0, 1200000])\n",
      "plt.grid(True)\n",
      "\n",
      "plt.show()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br><br>\n",
      "\n",
      "Utilizing MongoDB's Aggregation Framework\n",
      "-----------------------------------------\n",
      "\n",
      "With Monary's **aggregation** function, we have access to the full power of MongoDB's aggregation pipeline."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from datetime import datetime\n",
      "pipeline = [{\"$match\": {\"pickup_time\":\n",
      "                        {\"$lt\": datetime(2013, 1, 2)}}},\n",
      "            {\"$group\": {\"_id\": \"$pickup_time\",\n",
      "                        \"count\": {\"$sum\": 1}}},\n",
      "            {\"$sort\": {\"_id\": 1}}]\n",
      "\n",
      "pickups, counts = db.fares.aggregate(\n",
      "    pipeline, [\"_id\", \"count\"], [\"date\", \"float64\"])\n",
      "\n",
      "# Make timing relative to January 1st 2013\n",
      "pickups -= 1356998400000\n",
      "len(pickups)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br>\n",
      "But it looks like we have some data problems..."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "plt.plot(pickups, counts, 'yo')\n",
      "plt.xlabel('Millis')\n",
      "plt.ylabel('Pickups')\n",
      "plt.title('Taxi Pickups on New Years 2013')\n",
      "plt.axis([-1000000, 87400000, 0, 310])\n",
      "plt.show()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br>\n",
      "\n",
      "But that's okay, because we can use Numpy to compact the data into usable bins:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "bin_width = 300\n",
      "\n",
      "# Truncate to multiple of bin_width\n",
      "if (len(counts) % bin_width) > 0:\n",
      "    counts = counts[:-1 * (len(counts) % bin_width)]\n",
      "    pickups = pickups[:-1 * (len(pickups) % bin_width)]\n",
      "\n",
      "# Sum into bins of width bin_width (reshape and sum)\n",
      "counts = counts.reshape(len(counts) / bin_width, bin_width)\n",
      "counts = counts.sum(axis=1)\n",
      "\n",
      "# Pick every bin_width values\n",
      "pickups = pickups[::bin_width]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br>\n",
      "\n",
      "And finally, we can plot our data:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Convert times from millis to hours\n",
      "pickups = pickups.astype(\"float64\") / (60 * 60 * 1000)\n",
      "\n",
      "\n",
      "plt.plot(pickups, counts, 'go')\n",
      "plt.xlabel('Time')\n",
      "plt.ylabel('Pickups')\n",
      "plt.title('Taxi Pickups on New Years 2013')\n",
      "plt.axis([-.5, 24.5, 1200, 2400])\n",
      "\n",
      "# Pretty labels for the x-axis\n",
      "hours = [a + b for b in [\"a\", \"p\"]\n",
      "               for a in [\"12\", \"2\", \"4\", \"6\", \"8\", \"10\"]]\n",
      "plt.xticks(range(0, 25, 2), hours + [\"12a\"])\n",
      "\n",
      "plt.show()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br><br>\n",
      "\n",
      "Removing Documents with Monary\n",
      "------------------------------\n",
      "\n",
      "We can use Monary to remove documents we don't want. Let's remove all documents that have a fare more than two standard deviations away from the mean, or an unlisted payment type:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ids, fares, payment_type = db.fares.find(\n",
      "    {}, ['_id', 'fare_amount', 'payment_type'],\n",
      "    ['id', 'float64', 'type'])\n",
      "\n",
      "mean, std = fares.mean(), fares.std()\n",
      "\n",
      "# Find all of the fares within 1 std of the mean.\n",
      "keep = fares > (mean - 2*std)\n",
      "keep &= fares < (mean + 2*std)\n",
      "\n",
      "# Only keep those with an unmasked payment type.\n",
      "keep &= ~(payment_type.mask & (payment_type == 'UNK'))\n",
      "\n",
      "# Select the ids we want to remove.\n",
      "ids = ids[~keep]\n",
      "\n",
      "db.fares.remove([ids], [\"_id\"], [\"id\"])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br><br>\n",
      "\n",
      "And we can see that these have actually been removed:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "db.fares.count()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "<br>\n",
      "## Hooray for Monary!\n",
      "<br>\n"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}