sentiment-pipeline-sklearn-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to build a social media sentiment analysis pipeline with scikit-learn\n",
    "\n",
    "*This is Part 1 of 5 in a series on building a sentiment analysis pipeline using scikit-learn. You can find Part 2 [here](./sentiment-pipeline-sklearn-2.ipynb).*\n",
    "\n",
    "*Jump to:* \n",
    "\n",
    "* *[**Part 2 - Building a basic pipeline**](./sentiment-pipeline-sklearn-2.ipynb)*\n",
    "* *[**Part 3 - Adding a custom function to a pipeline**](./sentiment-pipeline-sklearn-3.ipynb)*\n",
    "* *[**Part 4 - Adding a custom feature to a pipeline with FeatureUnion**](./sentiment-pipeline-sklearn-4.ipynb)*\n",
    "* *[**Part 5 - Hyperparameter tuning in pipelines with GridSearchCV**](./sentiment-pipeline-sklearn-5.ipynb)*\n",
    "\n",
    "# Part 1 - Introduction and requirements\n",
    "\n",
    "As a data scientist at [Earshot](http://www.earshotinc.com), I'm always looking for ways to cut through the noise of social media and get to posts that are interesting for our clients. One such signal that we use to determine relevance is the sentiment of a post - is this user expressing positive or negative feelings? If an influential person is talking unfavorably on social media about you, you want to get out ahead of it quickly.\n",
    "\n",
    "Previously, we were using various off-the-shelf packages and APIs to get sentiment on our posts. We found their results to be somewhat dissatisfactory. These third party sentiment solutions did not perform well on social media posts. As we dug in to figure out why, some were trained on longer texts - Yelp reviews, IMDB reviews, Amazon reviews, etc., some did not handle emojis at all, and some were based on predefined dictionaries of specific adjectives that couldn't take into account the way slang and the English language changes. Because of this, we thought \"maybe we can do better\".\n",
    "\n",
    "We could, and we did.\n",
    "\n",
    "By creating a sentiment analysis engine that is attuned to the \"unique\" way users express themselves on social media, we've created a solution that works better for our customers.\n",
    "\n",
    "How did we do it? With blood, sweat, and tears – and a little TLC. More specifically, we used the magic of [scikit-learn pipelines](http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) to help rapidly build, iterate, and productionize our model. They're unimpeachably awesome, and if you're using Python for machine learning and not using these things, your life is about to get much easier. Come with me as I show you how to build a pipeline and add all sorts of fun* steps to it! \n",
    "\n",
    "<sub>*Fun not guaranteed</sub>\n",
    "\n",
    "# What are we going to cover?\n",
    "\n",
    "* Basic scikit-learn pipeline building\n",
    "* Adding custom functions as pipeline steps for text preprocessing\n",
    "* Adding custom functions as additional features \n",
    "* Using `GridSearchCV` to search for optimal parameters for each step\n",
    "\n",
    "\n",
    "# What are we not going to cover?\n",
    "\n",
    "* Machine learning basics\n",
    "* Machine learning with text basics\n",
    "* What is sentiment analysis?\n",
    "* How to scrape Twitter data (though the code is available in the repo)\n",
    "\n",
    "# Requirements\n",
    "\n",
    "First things first - we have to make sure you have everything you need to do this thing. You'll need the following packages for maximum enjoyment:\n",
    "\n",
    "* [pandas](https://github.com/pydata/pandas)\n",
    "* [NumPy](http://www.numpy.org/)\n",
    "* [scikit-learn](https://github.com/scikit-learn/scikit-learn) (>= 0.18)\n",
    "* [Twython](https://github.com/ryanmcgrath/twython) (for getting Twitter data)\n",
    "* [NTLK](http://www.nltk.org/)\n",
    "* [pandas_confusion](https://github.com/pandas-ml/pandas_confusion) (for neato confusion matrices)\n",
    "* [Jupyter](http://jupyter.org/) (if you want to run the notebooks yourself)\n",
    "\n",
    "# Code\n",
    "\n",
    "There are several helper functions that we will refer to in `sklearn_helpers.py` and `fetch_twitter_data.py`. If you want to follow along at home, clone the repo from [GitHub](https://github.com/ryan-cranfill/sentiment-pipeline-sklearn) and run the Jupyter notebooks found therein. Note - you will need to modify `fetch_twitter_data.py` with your Twitter credentials in order to download the data.\n",
    "\n",
    "# Ready?\n",
    "[Let's go to part 2!](./sentiment-pipeline-sklearn-2.ipynb)\n",
    "\n",
    "*This is Part 1 of 5 in a series on building a sentiment analysis pipeline using scikit-learn. You can find Part 2 [here](./sentiment-pipeline-sklearn-2.ipynb).*\n",
    "\n",
    "*Jump to:* \n",
    "\n",
    "* *[**Part 2 - Building a basic pipeline**](./sentiment-pipeline-sklearn-2.ipynb)*\n",
    "* *[**Part 3 - Adding a custom function to a pipeline**](./sentiment-pipeline-sklearn-3.ipynb)*\n",
    "* *[**Part 4 - Adding a custom feature to a pipeline with FeatureUnion**](./sentiment-pipeline-sklearn-4.ipynb)*\n",
    "* *[**Part 5 - Hyperparameter tuning in pipelines with GridSearchCV**](./sentiment-pipeline-sklearn-5.ipynb)*\n",
    "\n",
    "\n",
    "\n",
    "*[** -> ***[\n",
    "**] -> ]\n",
    ".ipynb)* -> )***"
   ]
  }
 ],
 "metadata": {
  "_draft": {
   "nbviewer_url": "https://gist.github.com/bfeb07535373d908f1bba6842e7797d9"
  },
  "gist": {
   "data": {
    "description": "Sentiment Blog Rough Draft",
    "public": false
   },
   "id": "bfeb07535373d908f1bba6842e7797d9"
  },
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}