sentiment-pipeline-sklearn-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to build a social media sentiment analysis pipeline with scikit-learn\n",
    "\n",
    "*This is Part 2 of 5 in a series on building a sentiment analysis pipeline using scikit-learn. You can find Part 3 [here](./sentiment-pipeline-sklearn-3.ipynb), and the introduction [here](./sentiment-pipeline-sklearn-1.ipynb).*\n",
    "\n",
    "*Jump to:* \n",
    "\n",
    "* *[**Part 1 - Introduction and requirements**](./sentiment-pipeline-sklearn-1.ipynb)*\n",
    "* *[**Part 3 - Adding a custom function to a pipeline**](./sentiment-pipeline-sklearn-3.ipynb)*\n",
    "* *[**Part 4 - Adding a custom feature to a pipeline with FeatureUnion**](./sentiment-pipeline-sklearn-4.ipynb)*\n",
    "* *[**Part 5 - Hyperparameter tuning in pipelines with GridSearchCV**](./sentiment-pipeline-sklearn-5.ipynb)*\n",
    "\n",
    "# Part 2 - Building a basic scikit-learn pipeline\n",
    "\n",
    "In this post, we're going to build a very simple pipeline, consisting of a count vectorizer for feature extraction and a logistic regression for classification.\n",
    "\n",
    "# Get The Data\n",
    "\n",
    "First, we're going to read in a CSV that has the ids of 1000 tweets and the posts' labeled sentiments, then we're going to fetch the contents of those tweets via Twython and return a dataframe. This is defined in a function in `fetch_twitter_data.py`. You'll need to populate the variables for Twitter API app key and secret, plus a user token and secret at the top of that file in order to fetch from Twitter. You can create an app [here](https://apps.twitter.com/app/new)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "got 92 posts from page 1...\n",
      "got 90 posts from page 2...\n",
      "got 89 posts from page 3...\n",
      "got 91 posts from page 4...\n",
      "got 87 posts from page 5...\n",
      "got 88 posts from page 6...\n",
      "got 95 posts from page 7...\n",
      "got 93 posts from page 8...\n",
      "got 86 posts from page 9...\n",
      "got 90 posts from page 10...\n",
      "got all pages - 901 posts in total\n",
      "CPU times: user 412 ms, sys: 193 ms, total: 605 ms\n",
      "Wall time: 6.17 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from fetch_twitter_data import fetch_the_data\n",
    "df = fetch_the_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're following along at home, a nonzero amount of those 1000 tweets will be unable to be fetched. That happens when the posts are deleted by the users. All we can do is fondly remember them and carry on. ¯\\\\_(ツ)_/¯\n",
    "\n",
    "We're going to take a quick peek at the head of the data and move on to building the model, since the emphasis of this post is on building a robust classifier pipeline. You're a great data scientist, though, so you already know the importance of getting really friendly with the contents of the data, right?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>tweet_id</th>\n",
       "      <th>sentiment</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>750068048607010816</td>\n",
       "      <td>negative</td>\n",
       "      <td>When you're drinking a beer and you get to tha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>773620706441474048</td>\n",
       "      <td>neutral</td>\n",
       "      <td>Yupppp https://t.co/mpzJ6yGI0r</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>778301474816294912</td>\n",
       "      <td>negative</td>\n",
       "      <td>@JetBlue three places just took off! Let us go...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>750079494560550912</td>\n",
       "      <td>positive</td>\n",
       "      <td>Enjoying a cold #beer @gbbrewingco @ATLairport...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>776834202205552640</td>\n",
       "      <td>neutral</td>\n",
       "      <td>@SuperNovs1 @zjlaing not sure how airlines han...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             tweet_id sentiment  \\\n",
       "0  750068048607010816  negative   \n",
       "1  773620706441474048   neutral   \n",
       "2  778301474816294912  negative   \n",
       "3  750079494560550912  positive   \n",
       "4  776834202205552640   neutral   \n",
       "\n",
       "                                                text  \n",
       "0  When you're drinking a beer and you get to tha...  \n",
       "1                     Yupppp https://t.co/mpzJ6yGI0r  \n",
       "2  @JetBlue three places just took off! Let us go...  \n",
       "3  Enjoying a cold #beer @gbbrewingco @ATLairport...  \n",
       "4  @SuperNovs1 @zjlaing not sure how airlines han...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building a basic pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A quick note before we begin: I'm taking liberties with assessing the performance of the model, and we're in danger of overfitting the model to the testing data. We're looking at the results of a particular set to show how the model changes with new features, preprocessing, and hyperparameters. This is for illustratory purposes only. Always remember to practice safe model evaluation, using proper cross-validation.\n",
    "\n",
    "Okay, time for some fun - we're going to make our first pipeline that we'll continue to build upon.\n",
    "\n",
    "## Train-test split\n",
    "\n",
    "First, we'll split up the data between training and testing sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "format": "row"
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X, y = df.text, df.sentiment\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenizer\n",
    "\n",
    "We're going to override sklearn's default tokenizer with NLTK's TweetTokenizer. This has the benefit of tokenizing hashtags and emoticons correctly, and shortening repeated characters (e.g., \"stop ittttttttttt\" -> \"stop ittt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "tokenizer = nltk.casual.TweetTokenizer(preserve_case=False, reduce_len=True) # Your milage may vary on these arguments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pipeline steps\n",
    "Now let's initialize our [Count Vectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html) and our [Logistic Regression](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) for classification. Many models tend to use Naive Bayes approaches for text classification problems. However, I'm not going to for this example for a few reasons:\n",
    "\n",
    "1. Later on, we're going to be adding continuous features to the pipeline, which is difficult to do with scikit-learn's implementation of NB. For example, Gaussian NB (the flavor which produces best results most of the time from continuous variables) requires dense matrices, but the output of a `CountVectorizer` is sparse. It's more work, and from my tests LR will still outperform even on this small of a dataset.\n",
    "1. We're thinking big, creating a pipeline that will still be useful on a much bigger dataset than we have in front of us right now. While Naive Bayes tends to converge quicker (albeit with a higher error rate than Logistic Regression) on smaller datasets, LR should outperform NB in the long run with more data to learn from.\n",
    "1. Finally, and probably most importantly, I've simply observed LR working better than NB on this kind of text data in actual production use cases. Data sparsity is likely our enemy here. Twitter data is short-form, so each example will have few \"rows\" to look up from our vectorized vocabulary, and our model may not see the same unigram/phrase enough times to really get its head around the true class probabilities for every word. sklearn's Logistic Regression's built-in regularization will handle these kinds of cases better than NB (again, based on my experience in this specific domain of social media). The experiences I've seen with LR vs. NB on social media posts probably warrant their own post.\n",
    "\n",
    "For more information, here's a nice book excerpt on Naive Bayes vs. Logistic Regression [here](http://www.cs.cmu.edu/~tom/mlbook/NBayesLogReg.pdf), a quick article on how to pick the right classifier [here](http://blog.echen.me/2011/04/27/choosing-a-machine-learning-classifier/), and a great walkthrough of how logistic regression works [here](http://nbviewer.jupyter.org/github/justmarkham/DAT8/blob/master/notebooks/12_logistic_regression.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "count_vect = CountVectorizer(tokenizer=tokenizer.tokenize) \n",
    "classifier = LogisticRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now to put them together in a pipeline..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sentiment_pipeline = Pipeline([\n",
    "        ('vectorizer', count_vect),\n",
    "        ('classifier', classifier)\n",
    "    ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alright, we've got a pipeline. Let's unleash this beast all over the training data!\n",
    "\n",
    "We'll import a helper function to make training and testing a more pleasant experience. This function trains, predicts on test data, checks accuracy score against null accuracy, and displays a confusion matrix and classification report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "null accuracy: 47.35%\n",
      "accuracy score: 61.95%\n",
      "model is 14.60% more accurate than null accuracy\n",
      "---------------------------------------------------------------------------\n",
      "Confusion Matrix\n",
      "\n",
      "Predicted  negative  neutral  positive  __all__\n",
      "Actual                                         \n",
      "negative         23       19        15       57\n",
      "neutral           6       45        11       62\n",
      "positive         10       25        72      107\n",
      "__all__          39       89        98      226\n",
      "---------------------------------------------------------------------------\n",
      "Classification Report\n",
      "\n",
      "                precision    recall  F1_score support\n",
      "Classes                                              \n",
      "negative         0.589744  0.403509  0.479167      57\n",
      "neutral          0.505618  0.725806  0.596026      62\n",
      "positive         0.734694  0.672897  0.702439     107\n",
      "__avg / total__  0.635292  0.619469  0.616934     226\n"
     ]
    }
   ],
   "source": [
    "from sklearn_helpers import train_test_and_evaluate\n",
    "\n",
    "sentiment_pipeline, confusion_matrix = train_test_and_evaluate(sentiment_pipeline, X_train, y_train, X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# All done\n",
    "\n",
    "That wasn't so bad, was it? Pipelines are great, as you can easily keep adding steps and tweaking the ones you have. In the [next lesson](./sentiment-pipeline-sklearn-3.ipynb) we're going to define a custom preprocessing function and add it as a step in the model. [Hit it!](./sentiment-pipeline-sklearn-3.ipynb)\n",
    "\n",
    "*This is Part 2 of 5 in a series on building a sentiment analysis pipeline using scikit-learn. You can find Part 3 [here](./sentiment-pipeline-sklearn-3.ipynb), and the introduction [here](./sentiment-pipeline-sklearn-1.ipynb).*\n",
    "\n",
    "*Jump to:* \n",
    "\n",
    "* *[**Part 1 - Introduction and requirements**](./sentiment-pipeline-sklearn-1.ipynb)*\n",
    "* *[**Part 3 - Adding a custom function to a pipeline**](./sentiment-pipeline-sklearn-3.ipynb)*\n",
    "* *[**Part 4 - Adding a custom feature to a pipeline with FeatureUnion**](./sentiment-pipeline-sklearn-4.ipynb)*\n",
    "* *[**Part 5 - Hyperparameter tuning in pipelines with GridSearchCV**](./sentiment-pipeline-sklearn-5.ipynb)*"
   ]
  }
 ],
 "metadata": {
  "_draft": {
   "nbviewer_url": "https://gist.github.com/bfeb07535373d908f1bba6842e7797d9"
  },
  "gist": {
   "data": {
    "description": "Sentiment Blog Rough Draft",
    "public": false
   },
   "id": "bfeb07535373d908f1bba6842e7797d9"
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}