Chapters7-8.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chapters 7 and 8: Sentiment Analysis\n",
    "\n",
    "## Data loading and preliminary inspection\n",
    "\n",
    "We will be using popular `polarity dataset 2.0` collected by [Bo Pang and colleagues from Cornell Univeristy](http://www.cs.cornell.edu/people/pabo/movie-review-data/). Let's first read it in and inspect the data. This code is very similar to the routine applied to spam filtering in Chapter 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, codecs\n",
    "\n",
    "def read_in(folder):\n",
    "    files = os.listdir(folder)\n",
    "    a_dict = {}\n",
    "    for a_file in sorted(files):\n",
    "        if not a_file.startswith(\".\"):\n",
    "            with codecs.open(folder + a_file, encoding='ISO-8859-1', errors ='ignore') as f:\n",
    "                file_id = a_file.split(\".\")[0].strip()\n",
    "                a_dict[file_id] = f.read()\n",
    "            f.close()\n",
    "    return a_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, when you download the dataset, it comes as two subfolders named `pos/` for all positive reviews and `neg/` for all negative ones, put within a folder called `review_polarity/txt_sentoken/`. If you don't change the folder names, you can simply read in the contents of all positive and negative reviews and put them in separate Python dictionaries of review titles mapped to the reviews content, using the method `read_in` from above.\n",
    "\n",
    "Let's also print out the number of reviews in positive and negative dictionaries, as well as the very first positive and very first negative reviews in the dictionaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000\n",
      "films adapted from comic books have had plenty of success , whether they're about superheroes ( batman , superman , spawn ) , or geared toward kids ( casper ) or the arthouse crowd ( ghost world ) , but there's never really been a comic book like from hell before . \n",
      "for starters , it was created by alan moore ( and eddie campbell ) , who brought the medium to a whole new level in the mid '80s with a 12-part series called the watchmen . \n",
      "to say moore and campbell thoroughly researched the subject of jack the ripper would be like saying michael jackson is starting to look a little odd . \n",
      "the book ( or \" graphic novel , \" if you will ) is over 500 pages long and includes nearly 30 more that consist of nothing but footnotes . \n",
      "in other words , don't dismiss this film because of its source . \n",
      "if you can get past the whole comic book thing , you might find another stumbling block in from hell's directors , albert and allen hughes . \n",
      "getting the hughes brothers to direct this seems almost as ludicrous as casting carrot top in , well , anything , but riddle me this : who better to direct a film that's set in the ghetto and features really violent street crime than the mad geniuses behind menace ii society ? \n",
      "the ghetto in question is , of course , whitechapel in 1888 london's east end . \n",
      "it's a filthy , sooty place where the whores ( called \" unfortunates \" ) are starting to get a little nervous about this mysterious psychopath who has been carving through their profession with surgical precision . \n",
      "when the first stiff turns up , copper peter godley ( robbie coltrane , the world is not enough ) calls in inspector frederick abberline ( johnny depp , blow ) to crack the case . \n",
      "abberline , a widower , has prophetic dreams he unsuccessfully tries to quell with copious amounts of absinthe and opium . \n",
      "upon arriving in whitechapel , he befriends an unfortunate named mary kelly ( heather graham , say it isn't so ) and proceeds to investigate the horribly gruesome crimes that even the police surgeon can't stomach . \n",
      "i don't think anyone needs to be briefed on jack the ripper , so i won't go into the particulars here , other than to say moore and campbell have a unique and interesting theory about both the identity of the killer and the reasons he chooses to slay . \n",
      "in the comic , they don't bother cloaking the identity of the ripper , but screenwriters terry hayes ( vertical limit ) and rafael yglesias ( les mis ? rables ) do a good job of keeping him hidden from viewers until the very end . \n",
      "it's funny to watch the locals blindly point the finger of blame at jews and indians because , after all , an englishman could never be capable of committing such ghastly acts . \n",
      "and from hell's ending had me whistling the stonecutters song from the simpsons for days ( \" who holds back the electric car/who made steve guttenberg a star ? \" ) . \n",
      "don't worry - it'll all make sense when you see it . \n",
      "now onto from hell's appearance : it's certainly dark and bleak enough , and it's surprising to see how much more it looks like a tim burton film than planet of the apes did ( at times , it seems like sleepy hollow 2 ) . \n",
      "the print i saw wasn't completely finished ( both color and music had not been finalized , so no comments about marilyn manson ) , but cinematographer peter deming ( don't say a word ) ably captures the dreariness of victorian-era london and helped make the flashy killing scenes remind me of the crazy flashbacks in twin peaks , even though the violence in the film pales in comparison to that in the black-and-white comic . \n",
      "oscar winner martin childs' ( shakespeare in love ) production design turns the original prague surroundings into one creepy place . \n",
      "even the acting in from hell is solid , with the dreamy depp turning in a typically strong performance and deftly handling a british accent . \n",
      "ians holm ( joe gould's secret ) and richardson ( 102 dalmatians ) log in great supporting roles , but the big surprise here is graham . \n",
      "i cringed the first time she opened her mouth , imagining her attempt at an irish accent , but it actually wasn't half bad . \n",
      "the film , however , is all good . \n",
      "2 : 00 - r for strong violence/gore , sexuality , language and drug content \n",
      "\n",
      "1000\n",
      "plot : two teen couples go to a church party , drink and then drive . \n",
      "they get into an accident . \n",
      "one of the guys dies , but his girlfriend continues to see him in her life , and has nightmares . \n",
      "what's the deal ? \n",
      "watch the movie and \" sorta \" find out . . . \n",
      "critique : a mind-fuck movie for the teen generation that touches on a very cool idea , but presents it in a very bad package . \n",
      "which is what makes this review an even harder one to write , since i generally applaud films which attempt to break the mold , mess with your head and such ( lost highway & memento ) , but there are good and bad ways of making all types of films , and these folks just didn't snag this one correctly . \n",
      "they seem to have taken this pretty neat concept , but executed it terribly . \n",
      "so what are the problems with the movie ? \n",
      "well , its main problem is that it's simply too jumbled . \n",
      "it starts off \" normal \" but then downshifts into this \" fantasy \" world in which you , as an audience member , have no idea what's going on . \n",
      "there are dreams , there are characters coming back from the dead , there are others who look like the dead , there are strange apparitions , there are disappearances , there are a looooot of chase scenes , there are tons of weird things that happen , and most of it is simply not explained . \n",
      "now i personally don't mind trying to unravel a film every now and then , but when all it does is give me the same clue over and over again , i get kind of fed up after a while , which is this film's biggest problem . \n",
      "it's obviously got this big secret to hide , but it seems to want to hide it completely until its final five minutes . \n",
      "and do they make things entertaining , thrilling or even engaging , in the meantime ? \n",
      "not really . \n",
      "the sad part is that the arrow and i both dig on flicks like this , so we actually figured most of it out by the half-way point , so all of the strangeness after that did start to make a little bit of sense , but it still didn't the make the film all that more entertaining . \n",
      "i guess the bottom line with movies like this is that you should always make sure that the audience is \" into it \" even before they are given the secret password to enter your world of understanding . \n",
      "i mean , showing melissa sagemiller running away from visions for about 20 minutes throughout the movie is just plain lazy ! ! \n",
      "okay , we get it . . . there \n",
      "are people chasing her and we don't know who they are . \n",
      "do we really need to see it over and over again ? \n",
      "how about giving us different scenes offering further insight into all of the strangeness going down in the movie ? \n",
      "apparently , the studio took this film away from its director and chopped it up themselves , and it shows . \n",
      "there might've been a pretty decent teen mind-fuck movie in here somewhere , but i guess \" the suits \" decided that turning it into a music video with little edge , would make more sense . \n",
      "the actors are pretty good for the most part , although wes bentley just seemed to be playing the exact same character that he did in american beauty , only in a new neighborhood . \n",
      "but my biggest kudos go out to sagemiller , who holds her own throughout the entire film , and actually has you feeling her character's unraveling . \n",
      "overall , the film doesn't stick because it doesn't entertain , it's confusing , it rarely excites and it feels pretty redundant for most of its runtime , despite a pretty cool ending and explanation to all of the craziness that came before it . \n",
      "oh , and by the way , this is not a horror or teen slasher flick . . . it's \n",
      "just packaged to look that way because someone is apparently assuming that the genre is still hot with the kids . \n",
      "it also wrapped production two years ago and has been sitting on the shelves ever since . \n",
      "whatever . . . skip \n",
      "it ! \n",
      "where's joblo coming from ? \n",
      "a nightmare of elm street 3 ( 7/10 ) - blair witch 2 ( 7/10 ) - the crow ( 9/10 ) - the crow : salvation ( 4/10 ) - lost highway ( 10/10 ) - memento ( 10/10 ) - the others ( 9/10 ) - stir of echoes ( 8/10 ) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "folder = \"review_polarity/txt_sentoken/\"\n",
    "pos_dict = read_in(folder + \"pos/\")\n",
    "print(len(pos_dict))\n",
    "print(pos_dict.get(next(iter(pos_dict))))\n",
    "neg_dict = read_in(folder + \"neg/\")\n",
    "print(len(neg_dict))\n",
    "print(neg_dict.get(next(iter(neg_dict))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So far so good! There are a couple of observation that you can make at this point:\n",
    "\n",
    "- The dataset is balanced between positive and negative reviews – both subsets have `1000` texts.\n",
    "- Texts are already preprocessed: they are tokenized (i.e., each word token is separated from its neighbours by a white space, all words are converted to lower case).\n",
    "\n",
    "What can you tell about the word content? Does the first positive review contain many positive words: i.e., if you didn't know it came from the `pos/` folder, would you be able to tell it was a positive one?\n",
    "\n",
    "Let's next calculate some simple statistics, including: \n",
    "- Average length of a review in positive and negative subsets\n",
    "- Average number of words per sentence\n",
    "- Size of the vocabulary (unique words in each subset)\n",
    "- *Diversity* of sentiment vocabulary – average number of times each word occurs in a text with a specific sentiment\n",
    "\n",
    "Many of these calculations will rely on having words extracted from text. Since texts are already tokenized and words are separated by white spaces, it's a matter of simply splitting texts into words by white spaces – see method `tokenize`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Category  Avg_Len(Review)  Avg_Len(Sent)  Vocabulary Size  Diversity \n",
      " Positive  787.051000       23.191531      36805.000000     21.384350 \n",
      " Negative  705.630000       21.524266      34542.000000     20.428174 \n"
     ]
    }
   ],
   "source": [
    "def tokenize(text):\n",
    "    text.replace(\"\\n\", \" \")\n",
    "    return text.split()\n",
    "\n",
    "def statistics(a_dict):\n",
    "    length = 0\n",
    "    sent_length = 0\n",
    "    num_sents = 0\n",
    "    vocab = []\n",
    "    for review in a_dict.values():\n",
    "        length += len(tokenize(review))\n",
    "        sents = review.split(\"\\n\")\n",
    "        num_sents += len(sents)\n",
    "        for sent in sents:\n",
    "            sent_length += len(tokenize(sent))\n",
    "        vocab += tokenize(review)\n",
    "    avg_length = float(length)/len(a_dict)\n",
    "    avg_sent_length = float(sent_length)/num_sents\n",
    "    vocab_size = len(set(vocab))\n",
    "    diversity = float(length)/float(vocab_size)\n",
    "    return avg_length, avg_sent_length, vocab_size, diversity\n",
    "        \n",
    "categories = [\"Positive\", \"Negative\"]\n",
    "rows = []\n",
    "rows.append([\"Category\", \"Avg_Len(Review)\", \"Avg_Len(Sent)\", \"Vocabulary Size\", \"Diversity\"])\n",
    "stats = {}\n",
    "stats[\"Positive\"] = statistics(pos_dict)\n",
    "stats[\"Negative\"] = statistics(neg_dict)\n",
    "for cat in categories:\n",
    "    rows.append([cat, f\"{stats.get(cat)[0]:.6f}\", \n",
    "                f\"{stats.get(cat)[1]:.6f}\",\n",
    "                f\"{stats.get(cat)[2]:.6f}\",\n",
    "                f\"{stats.get(cat)[3]:.6f}\"])\n",
    "\n",
    "columns = zip(*rows)\n",
    "column_widths = [max(len(item) for item in col) for col in columns]\n",
    "for row in rows:\n",
    "    print(''.join(' {:{width}} '.format(row[i], width=column_widths[i]) \n",
    "                  for i in range(0, len(row))))   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As it shows, positive reviews are overall longer than negative ones; they also have longer sentences with larger and more diverse vocabularies. Let's see how many words there are in overlap between positive and negative vocabularies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['asssss', 'assuaged', 'asthmatic', 'astin', 'astonishment', 'astor', \"astronauts'\", 'astronomer/writer', 'astronomical', 'astronomically', 'asuma', 'at-odds', 'atf', 'atheism', 'atheists', 'atkine', \"atlanta's\", \"atlantic's\", 'atlantis', \"atlantis'\", 'atmospheres', 'atmospherically', 'atmostpheric', 'atomizers', 'atone', 'atreus', 'atrophied', 'atrophy', 'attachs', 'attainable', \"attanasio's\", \"attendant's\", 'attendees', 'attention-grabbing', 'attentive', 'attentiveness', 'attest', 'attilla-looking', 'attractions', 'attractive-seeming', 'attractiveness', 'au/~koukoula/', 'auberjonois', 'audacious', 'audacious--and', 'audience--in', 'audience-friendly', 'audience-pleaser', 'audience-pleasing', 'audiotapes', 'auditioned', 'auditor', 'audre', 'aug', 'aughra', 'augment', \"august's\", 'augusts', 'aulon', 'aumont', 'aunjanue', 'auntie', 'aural', 'aurally', 'aurelien', 'aurelius', \"aurelius'\", 'auriga', 'auriol', 'aurora', 'auschwitz', 'austen-like', \"australian's\", 'australian/belgian', 'auteil', 'auteuil', \"auteuil's\", \"auteur's\", 'authentically', 'authorial', 'authoritarian', 'authoritarianism', 'authoritatively', \"authorities'\", 'authoritive', 'authorizes', 'authorship', 'authorties', 'auto-arousal', 'autocrat', 'automation', 'autonomous', 'autonomy', \"autumn's\", 'autumnal', 'aux', 'auxiliary', 'availability', 'avant', 'avary']\n",
      "['baffling', 'bafflingly', 'bafta', 'baggy', 'baghdad', 'bah', 'bahia', 'bahns', 'baigelman', \"baigelman's\", 'baileygaites', 'bails', 'baio', 'baird', 'bakalian', 'baker-enhanced', 'bakersfield', 'bakker', 'bakkers', 'bakula', 'balaban', 'balasko', 'balbricker', 'baldly', 'balinski', 'balki', 'ball-peen', 'ball-picking', 'ball-sniffing', 'ballard', 'ballet-type', 'ballhaus', 'ballisitic', 'ballistics', 'ballooning', 'ballyhooed', 'balm', 'bamboozled', \"band's\", 'bandaras', \"banderas'\", \"banderas's\", 'banderes', 'bandies', 'bane', 'bangkok', 'bangy', 'banish', 'bankability', 'bankroll', 'banshee', 'banzai', 'baotian', 'bar-owner', 'bar-room', 'bar/whorehouse', 'barbarino', 'barbarity', 'barbecue', 'barbell', 'barbeque', 'barby', 'barcalow', 'bare-chested', 'barely-funny', 'barely-in-control', 'barest', 'barfing', 'bark', 'barked', 'barkeep', 'barkeeps', 'barker', 'barnacles', 'barnett', 'barney', 'barnfield', 'barr', 'barracking', 'barrie', 'barring', 'barta', 'bartenders', 'barter', 'bartholomew', 'bartkiwiak', 'bartusiak', 'bascially', 'baseball-related', 'baseless', 'baser', 'bashfully', 'basic--and', \"basinger's\", 'basquiat', 'bastardizing', 'bastards', 'bastion', 'bat-loathing', 'bat-people']\n",
      "\n",
      "16378 unique words in positive reviews only\n",
      "14115 unique words in negative reviews only\n"
     ]
    }
   ],
   "source": [
    "def vocab_difference(list1, list2):\n",
    "    vocab1 = []\n",
    "    vocab2 = []\n",
    "    for rev in list1:\n",
    "        vocab1 += tokenize(rev)\n",
    "    for rev in list2:\n",
    "        vocab2 += tokenize(rev)\n",
    "    return sorted(list(set(vocab1) - set(vocab2)))\n",
    "\n",
    "pos_wordlist = pos_dict.values()\n",
    "neg_wordlist = neg_dict.values()\n",
    "\n",
    "print(vocab_difference(pos_wordlist, neg_wordlist)[1500:1600])\n",
    "print(vocab_difference(neg_wordlist, pos_wordlist)[1500:1600])\n",
    "print()\n",
    "print(str(len(vocab_difference(pos_wordlist, neg_wordlist))) + \" unique words in positive reviews only\")\n",
    "print(str(len(vocab_difference(neg_wordlist, pos_wordlist))) + \" unique words in negative reviews only\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are $16378$ unique words in the `positive` dictionary and $14115$ unique words in the `negative` one.\n",
    "\n",
    "So far, you've been looking at word forms – i.e., words like *movie* and *movies*, *like* and *liked* were considered as separate tokens in the statistics above. How much diversity is there in reviews when you take lemmas? Let's apply further NLP analysis and see how many word lemmas, not word forms (i.e., original tokens), there are in each subset.\n",
    "\n",
    "First, import `spacy` and implement the method `lemmatize`, that, when the \"switch\" is turned on, will return lemmas rather than word tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load(\"en_core_web_md\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['film', 'adapt', 'from', 'comic', 'book', 'have', 'have', 'plenty', 'of', 'success', ',', 'whether', '-PRON-', 'be', 'about', 'superhero', '(', 'batman', ',', 'superman', ',', 'spawn', ')', ',', 'or', 'gear', 'toward', 'kid', '(', 'casper', ')', 'or', 'the', 'arthouse', 'crowd', '(', 'ghost', 'world', ')', ',', 'but', 'there', 'be', 'never', 'really', 'be', 'a', 'comic', 'book', 'like', 'from', 'hell', 'before', '.', ' ', 'for', 'starter', ',', '-PRON-', 'be', 'create', 'by', 'alan', 'moore', '(', 'and', 'eddie', 'campbell', ')', ',', 'who', 'bring', 'the', 'medium', 'to', 'a', 'whole', 'new', 'level', 'in', 'the', 'mid', \"'\", '80', 'with', 'a', '12-part', 'series', 'call', 'the', 'watchman', '.', ' ', 'to', 'say', 'moore', 'and', 'campbell', 'thoroughly', 'research', 'the', 'subject', 'of', 'jack', 'the', 'ripper', 'would', 'be', 'like', 'say', 'michael', 'jackson', 'be', 'start', 'to', 'look', 'a', 'little', 'odd', '.', ' ', 'the', 'book', '(', 'or', '\"', 'graphic', 'novel', ',', '\"', 'if', '-PRON-', 'will', ')', 'be', 'over', '500', 'page', 'long', 'and', 'include', 'nearly', '30', 'more', 'that', 'consist', 'of', 'nothing', 'but', 'footnote', '.', ' ', 'in', 'other', 'word', ',', 'do', 'not', 'dismiss', 'this', 'film', 'because', 'of', '-PRON-', 'source', '.', ' ', 'if', '-PRON-', 'can', 'get', 'past', 'the', 'whole', 'comic', 'book', 'thing', ',', '-PRON-', 'may', 'find', 'another', 'stumble', 'block', 'in', 'from', 'hell', \"'s\", 'director', ',', 'albert', 'and', 'allen', 'hughes', '.', ' ', 'get', 'the', 'hughe', 'brother']\n",
      "\n",
      "[films, adapted, from, comic, books, have, had, plenty, of, success, ,, whether, they, 're, about, superheroes, (, batman, ,, superman, ,, spawn, ), ,, or, geared, toward, kids, (, casper, ), or, the, arthouse, crowd, (, ghost, world, ), ,, but, there, 's, never, really, been, a, comic, book, like, from, hell, before, .,  , for, starters, ,, it, was, created, by, alan, moore, (, and, eddie, campbell, ), ,, who, brought, the, medium, to, a, whole, new, level, in, the, mid, ', 80s, with, a, 12-part, series, called, the, watchmen, .,  , to, say, moore, and, campbell, thoroughly, researched, the, subject, of, jack, the, ripper, would, be, like, saying, michael, jackson, is, starting, to, look, a, little, odd, .,  , the, book, (, or, \", graphic, novel, ,, \", if, you, will, ), is, over, 500, pages, long, and, includes, nearly, 30, more, that, consist, of, nothing, but, footnotes, .,  , in, other, words, ,, do, n't, dismiss, this, film, because, of, its, source, .,  , if, you, can, get, past, the, whole, comic, book, thing, ,, you, might, find, another, stumbling, block, in, from, hell, 's, directors, ,, albert, and, allen, hughes, .,  , getting, the, hughes, brothers]\n"
     ]
    }
   ],
   "source": [
    "def lemmatize(sentence, switch):\n",
    "    text = nlp(sentence.replace(\"\\n\", \" \"))\n",
    "    if switch==\"on\":\n",
    "        lemmas = [text[i].lemma_ for i in range(len(text))]\n",
    "        return lemmas\n",
    "    else:\n",
    "        tokens = [text[i] for i in range(len(text))]\n",
    "        return tokens\n",
    "\n",
    "#print out the results and compare\n",
    "print(lemmatize(pos_dict.get(next(iter(pos_dict))), \"on\")[:200])\n",
    "print()\n",
    "print(lemmatize(pos_dict.get(next(iter(pos_dict))), \"off\")[:200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since processing with `spacy` might take time, let's run it once and store the results in specific data structures as we did in the previous chapters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200 reviews processed\n",
      "400 reviews processed\n",
      "600 reviews processed\n",
      "800 reviews processed\n",
      "Dataset processed\n",
      "200 reviews processed\n",
      "400 reviews processed\n",
      "600 reviews processed\n",
      "800 reviews processed\n",
      "Dataset processed\n"
     ]
    }
   ],
   "source": [
    "def spacy_preprocess_reviews(source):\n",
    "    source_docs = {}\n",
    "    index = 0\n",
    "    for review_id in source.keys():\n",
    "        #to speed processing up, you can disable \"ner\" – Named Entity Recognition module of spaCy\n",
    "        source_docs[review_id] = nlp(source.get(review_id).replace(\"\\n\", \"\"), disable=[\"ner\"])\n",
    "        if index>0 and (index%200)==0:\n",
    "            print(str(index) + \" reviews processed\")\n",
    "        index += 1\n",
    "    print(\"Dataset processed\")\n",
    "    return source_docs\n",
    "\n",
    "pos_docs = spacy_preprocess_reviews(pos_dict)\n",
    "neg_docs = spacy_preprocess_reviews(neg_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's calculate statistics as before, but this time taking lemmas rather than word tokens into account – intuitively, this should make the space of unique 'words' smaller:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Category  Avg_Len(Review)  Vocabulary Size  Diversity \n",
      " Positive  818.722000       24432.000000     33.510232 \n",
      " Negative  737.379000       22839.000000     32.285958 \n"
     ]
    }
   ],
   "source": [
    "def statistics_lem(source_docs):\n",
    "    length = 0\n",
    "    vocab = []\n",
    "    for review_id in source_docs.keys():\n",
    "        review_doc = source_docs.get(review_id)\n",
    "        lemmas = []\n",
    "        for token in review_doc:\n",
    "            lemmas.append(token.lemma_)\n",
    "        length += len(lemmas)\n",
    "        vocab += lemmas\n",
    "    avg_length = float(length)/len(source_docs)\n",
    "    vocab_size = len(set(vocab))\n",
    "    diversity = float(length)/float(vocab_size)\n",
    "    return avg_length, vocab_size, diversity\n",
    "        \n",
    "categories = [\"Positive\", \"Negative\"]\n",
    "rows = []\n",
    "rows.append([\"Category\", \"Avg_Len(Review)\", \"Vocabulary Size\", \"Diversity\"])\n",
    "stats = {}\n",
    "stats[\"Positive\"] = statistics_lem(pos_docs)\n",
    "stats[\"Negative\"] = statistics_lem(neg_docs)\n",
    "for cat in categories:\n",
    "    rows.append([cat, f\"{stats.get(cat)[0]:.6f}\", \n",
    "                f\"{stats.get(cat)[1]:.6f}\",\n",
    "                f\"{stats.get(cat)[2]:.6f}\"])\n",
    "\n",
    "columns = zip(*rows)\n",
    "column_widths = [max(len(item) for item in col) for col in columns]\n",
    "for row in rows:\n",
    "    print(''.join(' {:{width}} '.format(row[i], width=column_widths[i]) \n",
    "                  for i in range(0, len(row))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference gets considerably smaller, but you can still see that positive reviews are longer and more diverse than negative ones. What do you think it says about the nature of positive and negative reviews?\n",
    "\n",
    "Let's now extract the non-overlapping lemmas from the positive and negative subsets. This requires you to modify the code for `vocab_difference` above slightly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9230 unique lemmas in positive reviews only\n",
      "7637 unique lemmas in negative reviews only\n"
     ]
    }
   ],
   "source": [
    "def vocab_lem_difference(source_docs1, source_docs2):\n",
    "    vocab1 = []\n",
    "    vocab2 = []\n",
    "    for rev_id in source_docs1.keys():\n",
    "        rev = source_docs1.get(rev_id)\n",
    "        for token in rev:\n",
    "            vocab1.append(token.lemma_)\n",
    "    for rev_id in source_docs2.keys():\n",
    "        rev = source_docs2.get(rev_id)\n",
    "        for token in rev:\n",
    "            vocab2.append(token.lemma_)\n",
    "    return sorted(list(set(vocab1) - set(vocab2)))\n",
    "\n",
    "print(str(len(vocab_lem_difference(pos_docs, neg_docs))) + \" unique lemmas in positive reviews only\")\n",
    "print(str(len(vocab_lem_difference(neg_docs, pos_docs))) + \" unique lemmas in negative reviews only\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare these numbers to the ones for word tokens above.\n",
    "\n",
    "Finally, let's see if the number of unique adjectives and adverbs differs across the reviews with different polarity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Category  Unique adj's  Unique adv's \n",
      " Positive  2374.000000   574.000000   \n",
      " Negative  1906.000000   495.000000   \n"
     ]
    }
   ],
   "source": [
    "def vocab_pos_difference(source_docs1, source_docs2, pos):\n",
    "    vocab1 = []\n",
    "    vocab2 = []\n",
    "    for rev_id in source_docs1.keys():\n",
    "        rev = source_docs1.get(rev_id)\n",
    "        for token in rev:\n",
    "            if token.pos_==pos:\n",
    "                vocab1.append(token.text)\n",
    "    for rev_id in source_docs2.keys():\n",
    "        rev = source_docs2.get(rev_id)\n",
    "        for token in rev:\n",
    "            if token.pos_==pos:\n",
    "                vocab2.append(token.text)\n",
    "    return sorted(list(set(vocab1) - set(vocab2)))\n",
    "\n",
    "\n",
    "categories = [\"Positive\", \"Negative\"]\n",
    "rows = []\n",
    "rows.append([\"Category\", \"Unique adj's\", \"Unique adv's\"])\n",
    "stats = {}\n",
    "stats[\"Positive\"] = (len(vocab_pos_difference(pos_docs, neg_docs, \"ADJ\")), \n",
    "                     len(vocab_pos_difference(pos_docs, neg_docs, \"ADV\"))) \n",
    "stats[\"Negative\"] = (len(vocab_pos_difference(neg_docs, pos_docs, \"ADJ\")), \n",
    "                     len(vocab_pos_difference(neg_docs, pos_docs, \"ADV\"))) \n",
    "for cat in categories:\n",
    "    rows.append([cat, f\"{stats.get(cat)[0]:.6f}\", \n",
    "                f\"{stats.get(cat)[1]:.6f}\"])\n",
    "\n",
    "columns = zip(*rows)\n",
    "column_widths = [max(len(item) for item in col) for col in columns]\n",
    "for row in rows:\n",
    "    print(''.join(' {:{width}} '.format(row[i], width=column_widths[i]) \n",
    "                  for i in range(0, len(row))))      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, positive reviews turn out to have more unique adjectives and adverbs.\n",
    "\n",
    "## Simple approach with sentiment lexicons\n",
    "\n",
    "[William Hamilton and his colleagues from Stanford](https://nlp.stanford.edu/projects/socialsent/) collected lists of words with sentiment scores assigned:\n",
    "\n",
    "- per part of speech, for instance for all words vs for adjectives only\n",
    "- per decade, registering how sentiment changed over time, e.g. between 1990's and 2000's (take a look at how the sentiment of different words changed, e.g., '*wicked*' becoming much more positive these days, changing from $-3.09$ in 1890's up to $-0.76$ in 2000's)\n",
    "- per domain, e.g. movie-related words specifically.\n",
    "\n",
    "To this end, let's build a simple \"classifier\" that will decide upon the sentiment taking the overall weight of different sentimental words in a review into account. \n",
    "\n",
    "Let's first implement a method that will collect sentiment score per word, depending on the source list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.28\n",
      "1968\n",
      "1.19\n",
      "2041\n",
      "4924\n",
      "4924\n",
      "4981\n"
     ]
    }
   ],
   "source": [
    "def collect_wordlist(input_file):\n",
    "    word_dict = {}\n",
    "    with codecs.open(input_file, encoding='ISO-8859-1', errors ='ignore') as f:\n",
    "        for a_line in f.readlines():\n",
    "            cols = a_line.split(\"\\t\")\n",
    "            if len(cols)>2:\n",
    "                word = cols[0].strip()\n",
    "                score = float(cols[1].strip())\n",
    "                word_dict[word] = score\n",
    "    f.close()\n",
    "    return word_dict\n",
    "\n",
    "\n",
    "adj_90 = collect_wordlist(\"sentiment_words/adjectives/1990.tsv\")\n",
    "print(adj_90.get(\"cool\"))\n",
    "print(len(adj_90))\n",
    "adj_00 = collect_wordlist(\"sentiment_words/adjectives/2000.tsv\")\n",
    "print(adj_00.get(\"cool\"))\n",
    "print(len(adj_00))\n",
    "all_90 = collect_wordlist(\"sentiment_words/frequent_words/1990.tsv\")\n",
    "print(len(all_90))\n",
    "all_00 = collect_wordlist(\"sentiment_words/frequent_words/2000.tsv\")\n",
    "print(len(all_00))\n",
    "movie_words = collect_wordlist(\"sentiment_words/subreddits/movies.tsv\")\n",
    "print(len(movie_words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's implement a simple threshold-based approach, based on the idea that if the cumulative score is positive, it's a positive review, and if it is negative, it is a negative one. You have a choice between two strategies:\n",
    "- each positive word is worth +1 point and each negative one -1\n",
    "- each positive and negative word has different weight defined by the score from the lexicon.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " List    Acc(positive)  Acc(negative)  Acc(all) \n",
      " Adj_90  0.889000       0.267000       0.578000 \n",
      " Adj_00  0.825000       0.354000       0.589500 \n",
      " All_90  1.000000       0.000000       0.500000 \n",
      " All_00  0.935000       0.086000       0.510500 \n",
      " Movies  0.014000       0.996000       0.505000 \n"
     ]
    }
   ],
   "source": [
    "def bin_decisions(a_dict, label, sent_dict):\n",
    "    decisions = []\n",
    "    for rev_id in a_dict.keys():\n",
    "        score = 0\n",
    "        for token in a_dict.get(rev_id):\n",
    "            if token.text in sent_dict.keys():\n",
    "                if sent_dict.get(token.text)<0:\n",
    "                    score -= 1\n",
    "                else:\n",
    "                    score += 1\n",
    "        if score < 0:\n",
    "            decisions.append((-1, label))\n",
    "        else:\n",
    "            decisions.append((1, label))\n",
    "    return decisions\n",
    "\n",
    "def weighted_decisions(a_dict, label, sent_dict):\n",
    "    decisions = []\n",
    "    for rev_id in a_dict.keys():\n",
    "        score = 0\n",
    "        for token in a_dict.get(rev_id):\n",
    "            if token.text in sent_dict.keys():\n",
    "                score += sent_dict.get(token.text)\n",
    "        if score < 0:\n",
    "            decisions.append((-1, label))\n",
    "        else:\n",
    "            decisions.append((1, label))\n",
    "    return decisions\n",
    "\n",
    "def get_accuracy(pos_docs, neg_docs, sent_dict):\n",
    "    decisions_pos = bin_decisions(pos_docs, 1, sent_dict)\n",
    "    decisions_neg = bin_decisions(neg_docs, -1, sent_dict)\n",
    "    decisions_all = decisions_pos + decisions_neg\n",
    "    lists = [decisions_pos, decisions_neg, decisions_all]\n",
    "    accuracies = []\n",
    "    for i in range(0, len(lists)):\n",
    "        match = 0\n",
    "        for item in lists[i]:\n",
    "            if item[0]==item[1]:\n",
    "                match += 1\n",
    "        accuracies.append(float(match)/float(len(lists[i])))\n",
    "    return accuracies\n",
    "\n",
    "        \n",
    "categories = [\"Adj_90\", \"Adj_00\", \"All_90\", \"All_00\", \"Movies\"]\n",
    "rows = []\n",
    "rows.append([\"List\", \"Acc(positive)\", \"Acc(negative)\", \"Acc(all)\"])\n",
    "accs = {}\n",
    "accs[\"Adj_90\"] = get_accuracy(pos_docs, neg_docs, adj_90) \n",
    "accs[\"Adj_00\"] = get_accuracy(pos_docs, neg_docs, adj_00) \n",
    "accs[\"All_90\"] = get_accuracy(pos_docs, neg_docs, all_90) \n",
    "accs[\"All_00\"] = get_accuracy(pos_docs, neg_docs, all_00) \n",
    "accs[\"Movies\"] = get_accuracy(pos_docs, neg_docs, movie_words) \n",
    "for cat in categories:\n",
    "    rows.append([cat, f\"{accs.get(cat)[0]:.6f}\", \n",
    "                f\"{accs.get(cat)[1]:.6f}\",\n",
    "                f\"{accs.get(cat)[2]:.6f}\"])\n",
    "\n",
    "columns = zip(*rows)\n",
    "column_widths = [max(len(item) for item in col) for col in columns]\n",
    "for row in rows:\n",
    "    print(''.join(' {:{width}} '.format(row[i], width=column_widths[i]) \n",
    "                  for i in range(0, len(row))))    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the summary of the **results:**\n",
    "\n",
    "- With binary approach:\n",
    "    - adj_90: Positive: 0.889; Negative: 0.267; All: 0.578\n",
    "    - adj_00: Positive: 0.825; Negative: 0.354; All: **0.5895**\n",
    "    - all_90: Positive: **1.0**; Negative: 0.0; All: 0.5\n",
    "    - all_00: Positive: 0.935; Negative: 0.086; All: 0.5105\n",
    "    - movie_words: Positive: 0.014; Negative: **0.996**; All: 0.505\n",
    "\n",
    "- With weighted scores approach:\n",
    "    - adj_90: Positive: 0.788; Negative: 0.507; All: **0.6475**\n",
    "    - adj_00: Positive: 0.818; Negative: 0.424; All: 0.621\n",
    "    - all_90: Positive: **0.984**; Negative: 0.017; All: 0.5005\n",
    "    - all_00: Positive: 0.805; Negative: 0.373; All: 0.589\n",
    "    - movie_words: Positive: 0.008; Negative: **0.997**; All: 0.5025\n",
    "    \n",
    "Let's check how many words from the lexicons occur in each type of reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " List    Occurs(pos)  Occurs(neg) \n",
      " Adj_90  1726.000000  1667.000000 \n",
      " Adj_00  1782.000000  1731.000000 \n",
      " All_90  4772.000000  4712.000000 \n",
      " All_00  4789.000000  4732.000000 \n",
      " Movies  4675.000000  4660.000000 \n"
     ]
    }
   ],
   "source": [
    "def occurrences(a_dict, sent_dict):\n",
    "    occur = []\n",
    "    for rev_id in a_dict.keys():\n",
    "        for token in a_dict.get(rev_id):\n",
    "            if token.text in sent_dict.keys():\n",
    "                occur.append(token.text)\n",
    "    return len(set(occur))\n",
    "\n",
    "categories = [\"Adj_90\", \"Adj_00\", \"All_90\", \"All_00\", \"Movies\"]\n",
    "rows = []\n",
    "rows.append([\"List\", \"Occurs(pos)\", \"Occurs(neg)\"])\n",
    "occs = {}\n",
    "occs[\"Adj_90\"] = occurrences(pos_docs, adj_90), occurrences(neg_docs, adj_90)\n",
    "occs[\"Adj_00\"] = occurrences(pos_docs, adj_00), occurrences(neg_docs, adj_00)\n",
    "occs[\"All_90\"] = occurrences(pos_docs, all_90), occurrences(neg_docs, all_90)\n",
    "occs[\"All_00\"] = occurrences(pos_docs, all_00), occurrences(neg_docs, all_00)\n",
    "occs[\"Movies\"] = occurrences(pos_docs, movie_words), occurrences(neg_docs, movie_words)\n",
    "for cat in categories:\n",
    "    rows.append([cat, f\"{occs.get(cat)[0]:.6f}\", \n",
    "                f\"{occs.get(cat)[1]:.6f}\"])\n",
    "\n",
    "columns = zip(*rows)\n",
    "column_widths = [max(len(item) for item in col) for col in columns]\n",
    "for row in rows:\n",
    "    print(''.join(' {:{width}} '.format(row[i], width=column_widths[i]) \n",
    "                  for i in range(0, len(row))))    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment Analysis using SentiWordNet\n",
    "\n",
    "One of the problems with the previous approach is that words may express different meanings, and each may have different polarity or strength of sentiment.\n",
    "\n",
    "NLTK provides an interface to the `WordNet` (a lexical database that helps distinguish between vaiorus senses of the word) and [SentiWordNet](http://www.nltk.org/howto/sentiwordnet.html) that was created by researchers from the [Text Learning Group of the University of Pisa](http://sentiwordnet.isti.cnr.it).\n",
    "\n",
    "First, let's explore what the SentiWordNet contains. For instance, here is how you can check how many different senses a given word has (you can check the examples of use and the definition of each sense through [the online version of the WordNet](http://wordnetweb.princeton.edu/perl/webwn)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[SentiSynset('joy.n.01'), SentiSynset('joy.n.02'), SentiSynset('rejoice.v.01'), SentiSynset('gladden.v.01')]\n",
      "[SentiSynset('trouble.n.01'), SentiSynset('fuss.n.02'), SentiSynset('trouble.n.03'), SentiSynset('trouble.n.04'), SentiSynset('worry.n.02'), SentiSynset('trouble.n.06'), SentiSynset('disturb.v.01'), SentiSynset('trouble.v.02'), SentiSynset('perturb.v.01'), SentiSynset('trouble_oneself.v.01'), SentiSynset('trouble.v.05')]\n"
     ]
    }
   ],
   "source": [
    "from nltk.corpus import sentiwordnet as swn\n",
    "\n",
    "print(list(swn.senti_synsets('joy')))\n",
    "print(list(swn.senti_synsets('trouble')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code above shows that *joy* may be either a noun meaning the \"emotion of great happiness\" (the synset `'joy.n.01'`) or \"something / someone providing a source of happiness as in \"*a joy to behold*\" (the synset `'joy.n.02'`). As a verb, it can mean \"rejoice\" i.e. \"feel happiness or joy\" (`'rejoice.v.01'`) or \"gladden\" i.e. \"make glad or happy\" (`'rejoice.v.02'`). \n",
    "\n",
    "*Trouble* is a more complex case, with as many as 6 different meanings as a noun (for instance, it can mean a particular event causing pain as in \"*heart trouble*\", or a difficulty as in \"he went to a lot of trouble\"), and 5 senses for the verb! The differences may be quite subtle, yet leading to different interpretations, potentially with sentiments of different strength. Let's check this out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " List      Positive score  Negative Score \n",
      " Joy1      0.500           0.250          \n",
      " Joy2      0.375           0.000          \n",
      " Trouble1  0.000           0.625          \n",
      " Trouble2  0.000           0.500          \n"
     ]
    }
   ],
   "source": [
    "joy1 = swn.senti_synset('joy.n.01')\n",
    "joy2 = swn.senti_synset('joy.n.02')\n",
    "\n",
    "trouble1 = swn.senti_synset('trouble.n.03')\n",
    "trouble2 = swn.senti_synset('trouble.n.04')\n",
    "\n",
    "\n",
    "categories = [\"Joy1\", \"Joy2\", \"Trouble1\", \"Trouble2\"]\n",
    "rows = []\n",
    "rows.append([\"List\", \"Positive score\", \"Negative Score\"])\n",
    "accs = {}\n",
    "accs[\"Joy1\"] = [joy1.pos_score(), joy1.neg_score()]\n",
    "accs[\"Joy2\"] = [joy2.pos_score(), joy2.neg_score()]\n",
    "accs[\"Trouble1\"] = [trouble1.pos_score(), trouble1.neg_score()]\n",
    "accs[\"Trouble2\"] = [trouble2.pos_score(), trouble2.neg_score()] \n",
    "for cat in categories:\n",
    "    rows.append([cat, f\"{accs.get(cat)[0]:.3f}\", \n",
    "                f\"{accs.get(cat)[1]:.3f}\"])\n",
    "\n",
    "columns = zip(*rows)\n",
    "column_widths = [max(len(item) for item in col) for col in columns]\n",
    "for row in rows:\n",
    "    print(''.join(' {:{width}} '.format(row[i], width=column_widths[i]) \n",
    "                  for i in range(0, len(row)))) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, here is how we can check the sentiment for all synsets assigned to a word, additionally taking its part of speech into account:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pos: +0.25 neg: -0.25\n",
      "pos: +0.75 neg: -0.0\n",
      "pos: +0.0 neg: -0.625\n"
     ]
    }
   ],
   "source": [
    "synsets = swn.senti_synsets('terrific', 'a')\n",
    "for synset in synsets:\n",
    "    print(\"pos: +\" + str(synset.pos_score()) + \" neg: -\" + str(synset.neg_score()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's update our sentiment analyzer with SentiWordNet scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import wordnet as wn\n",
    "\n",
    "#use token.tag_ and see https://spacy.io/api/annotation\n",
    "def convert_tags(pos_tag):\n",
    "    if pos_tag.startswith(\"JJ\"):\n",
    "         return wn.ADJ\n",
    "    elif pos_tag.startswith(\"NN\"):\n",
    "         return wn.NOUN\n",
    "    elif pos_tag.startswith(\"RB\"):\n",
    "         return wn.ADV\n",
    "    elif pos_tag.startswith(\"VB\") or pos_tag.startswith(\"MD\"):\n",
    "         return wn.VERB\n",
    "    return None\n",
    " \n",
    "def swn_decisions(a_dict, label):   \n",
    "    decisions = []\n",
    "    for rev_id in a_dict.keys():\n",
    "        score = 0\n",
    "        neg_count = 0\n",
    "        pos_count = 0\n",
    "        for token in a_dict.get(rev_id):\n",
    "            wn_tag = convert_tags(token.tag_)\n",
    "            if wn_tag in (wn.ADJ, wn.ADV, wn.NOUN, wn.VERB):\n",
    "                synsets = list(swn.senti_synsets(token.lemma_, pos=wn_tag))\n",
    "                if len(synsets)>0:\n",
    "                    temp_score = 0.0\n",
    "                    for synset in synsets:\n",
    "                        temp_score += synset.pos_score() - synset.neg_score()\n",
    "                    score += temp_score/len(synsets)\n",
    "        if score < 0:\n",
    "            decisions.append((-1, label))  \n",
    "        else:\n",
    "            decisions.append((1, label))\n",
    "    return decisions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " List          Acc(positive)  Acc(negative)  Acc(all) \n",
      " SentiWordNet  0.686000       0.690000       0.688000 \n"
     ]
    }
   ],
   "source": [
    "def get_swn_accuracy(pos_docs, neg_docs):\n",
    "    decisions_pos = swn_decisions(pos_docs, 1)\n",
    "    decisions_neg = swn_decisions(neg_docs, -1)\n",
    "    decisions_all = decisions_pos + decisions_neg\n",
    "    lists = [decisions_pos, decisions_neg, decisions_all]\n",
    "    accuracies = []\n",
    "    for i in range(0, len(lists)):\n",
    "        match = 0\n",
    "        for item in lists[i]:\n",
    "            if item[0]==item[1]:\n",
    "                match += 1\n",
    "        accuracies.append(float(match)/float(len(lists[i])))\n",
    "    return accuracies\n",
    "\n",
    "\n",
    "accuracies = get_swn_accuracy(pos_docs, neg_docs)  \n",
    "\n",
    "rows = []\n",
    "rows.append([\"List\", \"Acc(positive)\", \"Acc(negative)\", \"Acc(all)\"])\n",
    "rows.append([\"SentiWordNet\", f\"{accuracies[0]:.6f}\", \n",
    "                f\"{accuracies[1]:.6f}\",\n",
    "                f\"{accuracies[2]:.6f}\"])\n",
    "\n",
    "columns = zip(*rows)\n",
    "column_widths = [max(len(item) for item in col) for col in columns]\n",
    "for row in rows:\n",
    "    print(''.join(' {:{width}} '.format(row[i], width=column_widths[i]) \n",
    "                  for i in range(0, len(row))))   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before moving on, let's visualize the results as you did before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAVkUlEQVR4nO3df/BddZ3f8eerRGyNKFC/MiD+QKWwrJWI38FFRlZEI6YqaP0BpW4AbdYpUt3ZdsSxs1rs7mitte7oMo0RpQwgK7NZscMGUuqPXQXkmyWL/FIwxiUGyZdG/MFuNwO++8c9X7kk9+Z7br7fBP34fMzcued8zudzzuck5/u6537uOfemqpAktesfPd4dkCTtXQa9JDXOoJekxhn0ktQ4g16SGmfQS1Lj5g36JEcl2Tj0+EmS9yQ5OMn6JHd3zweNab+yq3N3kpWLvwuSpN3JJNfRJ9kP+AHwEuA8YHtVfTjJBcBBVfXeneofDMwA00ABG4AXV9WPFqn/kqR5TDp0cwrw3ar6PnAacElXfglw+oj6rwbWV9X2LtzXA6fuaWclSZNbMmH9M4AruulDquo+gKq6L8nTR9R/BnDv0PyWrmwXSVYBqwCWLl364qOPPnrCrknSr68NGzY8UFVTo5b1Dvok+wOvB943wbYzomzkWFFVrQZWA0xPT9fMzMwEm5GkX29Jvj9u2SRDN68B/rqq7u/m709yaLeBQ4FtI9psAZ45NH84sHWCbUqSFmiSoD+TR4dtAK4G5q6iWQl8cUSba4HlSQ7qrspZ3pVJkvaRXkGf5EnAq4A/Gyr+MPCqJHd3yz7c1Z1OsgagqrYDHwJu7h4XdmWSpH1kossr9xXH6CVpMkk2VNX0qGXeGStJjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMm/VIzSQt07mV//3h3Qb+kLj7rn+yV9XpGL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxvb4CIcmBwBrgBUAB5wLvAY7qqhwIPFhVy0a03Qz8FHgEeHjcT11JkvaOvt918wlgXVW9Kcn+wJOq6q1zC5N8DPjxbtqfXFUPLKCfkqQ9NG/QJ3kKcBJwNkBV7QB2DC0P8BbgFXuni5KkhegzRv9cYBb4bJJbkqxJsnRo+cuA+6vq7jHtC7guyYYkqxbYX0nShPoE/RLgOOCiqnoR8BBwwdDyM4ErdtP+xKo6DngNcF6Sk0ZVSrIqyUySmdnZ2X69lyTNq0/QbwG2VNVN3fxVDIKfJEuANwJXjmtcVVu7523AWuD4MfVWV9V0VU1PTU313wNJ0m7NG/RV9UPg3iRzV9icAtzRTb8SuKuqtoxqm2RpkgPmpoHlwG0L7rUkqbe+V92cD1zWXXGzCTinKz+DnYZtkhwGrKmqFcAhwNrB57UsAS6vqnWL0XFJUj+9gr6qNgK7XP9eVWePKNsKrOimNwHHLqyLkqSF8M5YSWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXG9gj7JgUmuSnJXkjuTnJDkg0l+kGRj91gxpu2pSb6d5J4kFyxu9yVJ8+l7Rv8JYF1VHc3gx77v7Mo/XlXLusc1OzdKsh/wKeA1wDHAmUmOWYR+S5J6mjfokzwFOAn4DEBV7aiqB3uu/3jgnqraVFU7gM8Dp+1pZyVJk+tzRv9cYBb4bJJbkqxJsrRb9q4ktya5OMlBI9o+A7h3aH5LV7aLJKuSzCSZmZ2dnWQfJEm70SfolwDHARdV1YuAh4ALgIuA5wHLgPuAj41omxFlNWojVbW6qqaranpqaqpP3yVJPSzpUWcLsKWqburmrwIuqKr75yok+TTwv8a0febQ/OHA1j3say9LT75sb65ev8Ie+vJZj3cXpMfFvGf0VfVD4N4kR3VFpwB3JDl0qNobgNtGNL8ZODLJEUn2B84Arl5gnyVJE+hzRg9wPnBZF9abgHOAP06yjMFQzGbgdwGSHAasqaoVVfVwkncB1wL7ARdX1e2LvA+SpN3oFfRVtRGY3qn4bWPqbgVWDM1fA+xy6aUkad/wzlhJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY3r9VOCSQ4E1gAvYPAbsecCbwReB+wAvgucU1UPjmi7Gfgp8AjwcFXt/JOEkqS9qO8Z/SeAdVV1NHAscCewHnhBVb0Q+A7wvt20P7mqlhnykrTvzRv0SZ4CnAR8BqCqdlTVg1V1XVU93FW7ETh873VTkrSn+pzRPxeYBT6b5JYka5Is3anOucBfjGlfwHVJNiRZNW4jSVYlmUkyMzs726vzkqT59Qn6JcBxwEVV9SLgIeCCuYVJ3g88DFw2pv2JVXUc8BrgvCQnjapUVaurarqqpqempibZB0nSbvQJ+i3Alqq6qZu/ikHwk2Ql8FrgrKqqUY2ramv3vA1YCxy/0E5LkvqbN+ir6ofAvUmO6opOAe5IcirwXuD1VfV3o9omWZrkgLlpYDlw26L0XJLUS6/LK4HzgcuS7A9sAs4BbgaeCKxPAnBjVb0zyWHAmqpaARwCrO2WLwEur6p1i7wPkqTd6BX0VbUR2PnSyOePqbsVWNFNb2JwOaYk6XHinbGS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhrXK+iTHJjkqiR3JbkzyQlJDk6yPsnd3fNBY9qu7OrcnWTl4nZfkjSfvmf0nwDWVdXRDH4D9k7gAuD6qjoSuL6bf4wkBwMfAF4CHA98YNwLgiRp75g36JM8BTgJ+AxAVe2oqgeB04BLumqXAKePaP5qYH1Vba+qHwHrgVMXo+OSpH76nNE/F5gFPpvkliRrkiwFDqmq+wC656ePaPsM4N6h+S1d2S6SrEoyk2RmdnZ2op2QJI3XJ+iXAMcBF1XVi4CHGDFMM0ZGlNWoilW1uqqmq2p6amqq5+olSfPpE/RbgC1VdVM3fxWD4L8/yaEA3fO2MW2fOTR/OLB1z7srSZrUvEFfVT8E7k1yVFd0CnAHcDUwdxXNSuCLI5pfCyxPclD3IezyrkyStI8s6VnvfOCyJPsDm4BzGLxI/GmStwN/C7wZIMk08M6qekdVbU/yIeDmbj0XVtX2Rd0DSdJu9Qr6qtoITI9YdMqIujPAO4bmLwYu3tMOSpIWxjtjJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1rtdPCSbZDPwUeAR4uKqmk1wJzP1g+IHAg1W1rE/bRei3JKmnvj8ODnByVT0wN1NVb52bTvIx4Md920qS9p1Jgn6kJAHeArxi4d2RJC22vmP0BVyXZEOSVTstexlwf1XdvQdtfyHJqiQzSWZmZ2d7dkuSNJ++Z/QnVtXWJE8H1ie5q6q+1i07E7hiD9v+QlWtBlYDTE9P1wT7IEnajV5n9FW1tXveBqwFjgdIsgR4I3DlpG0lSfvGvEGfZGmSA+amgeXAbd3iVwJ3VdWWPWgrSdoH+gzdHAKsHXzmyhLg8qpa1y07g52GbZIcBqypqhXztJUk7QPzBn1VbQKOHbPs7BFlW4EV87WVJO0b3hkrSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxvYI+yeYk30qyMclMV/bBJD/oyjYmWTGm7alJvp3kniQXLGbnJUnz6/Pj4HNOrqoHdir7eFX913ENkuwHfAp4FbAFuDnJ1VV1x+RdlSTtib09dHM8cE9VbaqqHcDngdP28jYlSUP6Bn0B1yXZkGTVUPm7ktya5OIkB41o9wzg3qH5LV3ZLpKsSjKTZGZ2drZntyRJ8+kb9CdW1XHAa4DzkpwEXAQ8D1gG3Ad8bES7jCirURuoqtVVNV1V01NTUz27JUmaT6+gr6qt3fM2YC1wfFXdX1WPVNXPgU8zGKbZ2RbgmUPzhwNbF9ZlSdIk5g36JEuTHDA3DSwHbkty6FC1NwC3jWh+M3BkkiOS7A+cAVy98G5Lkvrqc9XNIcDaJHP1L6+qdUkuTbKMwVDMZuB3AZIcBqypqhVV9XCSdwHXAvsBF1fV7XthPyRJY8wb9FW1CTh2RPnbxtTfCqwYmr8GuGYBfZQkLYB3xkpS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJalyf34wlyWbgp8AjwMNVNZ3ko8DrgB3Ad4FzqurBPm0Xp+uSpD4mOaM/uaqWDQX1euAFVfVC4DvA+yZoK0naR/Z46Kaqrquqh7vZG4HDF6dLkqTF1DfoC7guyYYkq0YsPxf4iz1sK0nai3qN0QMnVtXWJE8H1ie5q6q+BpDk/cDDwGWTth3WvQisAnjWs5418Y5IkkbrdUZfVVu7523AWuB4gCQrgdcCZ1VVTdJ2RL3VVTVdVdNTU1OT7ockaYx5gz7J0iQHzE0Dy4HbkpwKvBd4fVX93SRtF6vzkqT59Rm6OQRYm2Su/uVVtS7JPcATGQzHANxYVe9MchiwpqpWjGu7F/ZDkjTGvEFfVZuAY0eUP39M/a3Ait21lSTtO94ZK0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDWuV9An2ZzkW0k2Jpnpyg5Osj7J3d3zQWParuzq3J1k5WJ2XpI0v0nO6E+uqmVVNd3NXwBcX1VHAtd384+R5GDgA8BLgOOBD4x7QZAk7R0LGbo5Dbikm74EOH1EnVcD66tqe1X9CFgPnLqAbUqSJrSkZ70CrktSwP+oqtXAIVV1H0BV3Zfk6SPaPQO4d2h+S1e2iySrgFXd7M+SfLtn3zTe04AHHu9O/LJI/vXj3QWN5nHa+ezCDtFnj1vQN+hPrKqtXZivT3JXz3YZUVajKnYvHqt7rlc9JJkZGmqTfil5nO59vYZuqmpr97wNWMtgvP3+JIcCdM/bRjTdAjxzaP5wYOtCOixJmsy8QZ9kaZID5qaB5cBtwNXA3FU0K4Evjmh+LbA8yUHdh7DLuzJJ0j7SZ+jmEGBtkrn6l1fVuiQ3A3+a5O3A3wJvBkgyDbyzqt5RVduTfAi4uVvXhVW1fdH3QuM4FKZfBR6ne1mqRg6ZS5Ia4Z2xktQ4g16SGmfQ70aS9ye5Pcmt3dc/vGQP17MsyYqh+dcn2eVO4sWU5OVJXjpm2dlJPrkI27gmyYELXY8eH4txfI87tpMcmOT/pvtwL8kJSSrJ4d38U5NsT7LHGZTkZ93zc7p1nz+07JNJzp6n/elJjtnT7f8qMejHSHIC8FrguKp6IfBKHnvz1ySWAb/4Y6iqq6vqwwvv5W69HBgZ9IulqlZU1YN7cxvaOxbx+B55bHfHxQ+B3+gWvRS4hUePyd8Cbqqqn/fs73wXjmwD3p1k/wn6fjpg0P+aOxR4oKr+AaCqHpi7nyDJi5N8NcmGJNcO3U/wlSQfSfLNJN9J8rLuwLsQeGt31vTW4TPqJJ9LclGSLyfZlOS3k1yc5M4kn5vrTJLlSW5I8tdJvpDkyV355iT/qSv/VpKjkzwHeCfwe902X9Znh0dtozvz+naSo7o6VyT5N0Pbflo3/TvdmeHfJLm0K3t2kuu78uuTPGton/84yTe6fX5TV35okq91fb6tb7+1R0Ye34t5bANf59Fgfynw8Z3mv9Gte1mSG7vjZG2678PqtvlHSb7KIMSP6I7PmzO4mm/YLIPv3NrlixOTPC/Jum6f/rL7G3kp8Hrgo13fn7cY/6i/tKrKx4gH8GRgI/Ad4E+A3+7Kn8DgAJ3q5t8KXNxNfwX4WDe9Avjf3fTZwCeH1v2LeeBzwOcZ3EV8GvAT4J8zeBHewOCM6WnA14ClXZv3An/QTW8Gzu+m/y2wppv+IPDvx+zbY/rTle1uG68CbgDOANYNtdnctftN4NvA07ryg7vnLwEru+lzgT8f2ucvdPt4DHBPV/77wPu76f2AAx7v46DVx6jjey8c22cPtb8F+MfAX3Xz64FXdNO38ujf14XAfx/a5p8Mrftq4He66fOAn3XTz2Fwb88RwF3dsfNJ4Oxu+fXAkd30S4D/M3Qcvunx/r/YF4++X4Hwa6eqfpbkxcDLgJOBKzMYV58BXsDgqyBgcFDdN9T0z7rnDQwOwD6+VFWV5FvA/VX1LYAkt3frOJxBIH692+b+DIJ31Dbf2H8vH+O3xm2jqtYneTPwKeDYEW1fAVxVVQ909efulThhqD+XAv9lqM2f1+Bt+x1JDunKbgYuTvKEbvnGPdwXzWPU8Q38Zxb32P46cEGSI4DNVfX/MvBk4MXAN5M8FTiwqr7atbmEwUnAnCuHpk8E/mU3fSnwkZ326XtJvgn8q7myblsvBb7Q7RPAE3v0vSkG/W5U1SMMziq+0oXwSgYH+e1VdcKYZv/QPT9C/3/fuTY/H5qem1/SrWt9VZ25iNvcWcZtI4MPzH4D+HvgYAZfbbFz2z43ZAzXGd7PAFTV15KcBPwL4NIkH62q/9l/FzSJEcf3eSzisV1Vd3fDMK/j0ROTDcA5wPe6F5unzrOah3Ze7Tz1/wi4isG7Uxi8a3ywqpbN19+WOUY/RpKjkhw5VLQM+D6DIYqpDD7MIskTkvzmPKv7KXDAArpzI3Bikud323xSkn+2yNvc3TZ+D7gTOJNHz7iHXQ+8Jck/7doe3JV/g8FwD8BZwF/trgNJng1sq6pPA58Bjpug/5rAmOP7Thb/2L4BeDePBv0NwHvoxuer6sfAj4Y+j3kb8NWdV9L5Oo89nnZRVXcBdzD4oJmq+gnwve4dKd07irl3pQv9u/yVYdCP92TgkiR3JLmVwbDGB6tqB/Am4CNJ/obBOOd8V7d8GThm7gOrSTtSVbMMxjuv6PpyI3D0PM2+BLwh4z+MPTvJlrkHg7ezu2yjC/t3AL9fVX/J4EzpP+7Uv9uBPwS+2v2b/Ldu0b8DzunW9zYGf/C783JgY5JbGLxF/8Q89bXnRh3ff8DiH9tfZ/DFhjPd/A3Ac+mCvrOSwYeitzJ4wblwzLbeDZyXwdev7O6dwB8yGO6ccxbw9m6fbmfwWRgMPhv7D0luaf3DWL8CQZIa5xm9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mN+//eIpmYZklW9gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function matplotlib.pyplot.legend(*args, **kwargs)>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "a = [\"Sentiment Lexicons\", \"SentiWordNet\"]\n",
    "index = range(len(a))\n",
    "b = [64.75, 68.80] # Accuracy scores for the compared methods\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "axes = plt.gca()\n",
    "# Let's set 50 as the lower bound as the majority class baseline is at 50 for this dataset\n",
    "axes.set_ylim([50,70])\n",
    "\n",
    "ax.bar(index, b, color=['#0A40A4', '#61A4F6'])\n",
    "plt.xticks(index, a) \n",
    "\n",
    "plt.show()\n",
    "\n",
    "import matplotlib\n",
    "matplotlib.axes.Axes.plot\n",
    "matplotlib.pyplot.plot\n",
    "matplotlib.axes.Axes.legend\n",
    "matplotlib.pyplot.legend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data-driven approach with a machine learning classifier\n",
    "\n",
    "First, let's filter out punctuation marks and stopwords (you can add any other filters if you'd like) and prepare the data for the machine learning pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000 2000\n",
      "the central focus of michael winterbottom 's welcome to sarajevo is sarajevo itself the city under siege and its different effect on the characters unfortunate enough to be stuck there it proves the backdrop for a stunningly realized story which refreshingly strays from mythic portents platoon racial tumultuosness the risible the walking dead or a tinge of schmaltziness schindler 's list the two leads stephen dillane as a reporter and emira nusevic as an orphan with a plight few can identify with are extremely believable not one moment with them involved rings false the question is not what went right the question is what went wrong for one the film fails to provide a political overview of the war as it progresses the dillane characters reports an american plane departing from sarajevo as it departs and that 's about it the assortment of high profile supporting actors ranging from woody harrelson as a yankee reporter into liquor and cigarrettes to marisa tomei as a huggable children 's aid or somesuch are incapable of rising above the sketchiness of their characters albeit they strive the interrupted use of authentic war footage somewhat hampers the rest of the film it makes the fictional characters seem powerless by comparison still winterbottom eschews mawkishness through flashy frantic editing and imaginative use of music and it 's a plus because he does n't toy with our emotions with sentimental blandness he wants us to know that in war no one is victorious the sweet hereafter starring ian holm sarah polley bruce greenwood tom mccamus gabrielle rose arsinee khanjian alberta watson maury chaykin caerthan banks produced by atom egoyan and camelia frieberg script by atom egoyan based on the novel by russell banks directed by atom egoyan running time 110 mins rated r atom egoyan 's powerfully meditative the sweet hereafter is as anything as haunting and transcendental as i 've seen this year it not only explores the aftermaths of a terrible tragedy with magnificent subtlety with cold and stunning shots of the backdrop of this mistfortune but also by telegraphing the sense of devastation that has permeated the small town with an enormous amount of dignity and respect for each and every soul affected ian holm as the somber lonely lawyer seeking compensation for the townspeople has a decency and a restraint uncommon with lawyers he 's a lion for the wrong reasons the pain for having lost his daughter she 's a druggie who frequently calls him begging for money is reflected on the town and in some mesmerizing flashbacks we witness how she lost her purity it 's one of the most strikingly breathtaking takes on the loss of innocence i 've ever seen but the most astounding part goes to sarah polly the peaceful benevolent girl who in contrast to the other characters remains the most mentally stable during past present or future her ability to convey concealed pain and unconditional love is the opposite yet near revolutionary role that won emily watson an oscar nomination for breaking the waves both women are torn apart by an extraordinary incident but it is only then that we truly see truly feel their wordless omnipotent love   the sweet hereafter though is atom egoyan 's movie and as of now he is emerging as the definite face of independent cinema after surging to worldwide recognition with 1994 's exotica the sweet hereafter makes him something few people in this industry can call themselves an artist he interweaves time with a delicacy and sees a soul with a purity that is just not common\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import string\n",
    "#from spacy.lang.en.stop_words import STOP_WORDS as stopwords_list # stopwords list\n",
    "punctuation_list = [punct for punct in string.punctuation]\n",
    "\n",
    "def text_filter(a_dict, label, exclude_lists):\n",
    "    data = []\n",
    "    for rev_id in a_dict.keys():\n",
    "        tokens = []\n",
    "        for token in a_dict.get(rev_id):\n",
    "            if not token.text in exclude_lists:\n",
    "                tokens.append(token.text)\n",
    "                #tokens.append(token.lemma_) # for the use of lemmas instead of word tokens\n",
    "        data.append((' '.join(tokens), label))\n",
    "    return data\n",
    "\n",
    "def prepare_data(pos_docs, neg_docs, exclude_lists):\n",
    "    data = text_filter(pos_docs, 1, exclude_lists)\n",
    "    data += text_filter(neg_docs, -1, exclude_lists)\n",
    "    random.seed(42)\n",
    "    random.shuffle(data)\n",
    "    texts = []\n",
    "    labels = []\n",
    "    for item in data:\n",
    "        texts.append(item[0])\n",
    "        labels.append(item[1])\n",
    "    return texts, labels\n",
    "\n",
    "# for the use of both lists in filtering:\n",
    "#texts, labels = prepare_data(pos_docs, neg_docs, list(stopwords_list) + punctuation_list)\n",
    "\n",
    "texts, labels = prepare_data(pos_docs, neg_docs, punctuation_list) \n",
    "\n",
    "print(len(texts), len(labels))\n",
    "print(texts[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note, how the content is filtered down to meaningful words only. Let's set up a small experiment: first set $80\\%$ of the data for training and rest for testing in this randomly shuffled set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1600\n",
      "1600\n",
      "400\n",
      "400\n",
      "[1, -1, 1, 1, -1, -1, -1, -1, 1, -1]\n",
      "[-1, 1, 1, -1, -1, 1, -1, 1, 1, 1]\n"
     ]
    }
   ],
   "source": [
    "def split(texts, labels, proportion):\n",
    "    train_data = []\n",
    "    train_targets = []\n",
    "    test_data = []\n",
    "    test_targets = []\n",
    "    for i in range(0, len(texts)):\n",
    "        if i < proportion*len(texts):\n",
    "            train_data.append(texts[i])\n",
    "            train_targets.append(labels[i])\n",
    "        else:\n",
    "            test_data.append(texts[i])\n",
    "            test_targets.append(labels[i])\n",
    "    return train_data, train_targets, test_data, test_targets\n",
    "\n",
    "train_data, train_targets, test_data, test_targets = split(texts, labels, 0.8)\n",
    "        \n",
    "print(len(train_data)) # 1600?\n",
    "print(len(train_targets)) # 1600?      \n",
    "print(len(test_data)) # 400?       \n",
    "print(len(test_targets)) # 400? \n",
    "print(train_targets[:10]) # print out the targets for the first 10 training reviews \n",
    "print(test_targets[:10]) # print out the targets for the first 10 test reviews "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's estimate the distribution of words across texts using `sklearn`'s `CountVectorizer`. Here is an example of how it will work on a random selection of 1000 reviews:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1600, 36094)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "count_vect = CountVectorizer()\n",
    "train_counts = count_vect.fit_transform(train_data)\n",
    "# Check the dimensionality \n",
    "print(train_counts.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows that our training set contains over $35,000$ distinct words (the exact number may change depending on your split). This is our training set vocabulary, and it will be applied to all test reviews only. Note that this vocabulary is learned on the training data only. Let's look 'under the hood' and print out the counts for some words in the first $10$ reviews from the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 32056)\t41\n",
      "  (0, 5161)\t1\n",
      "  (0, 12240)\t1\n",
      "  (0, 22070)\t18\n",
      "  (0, 20177)\t1\n",
      "  (0, 35454)\t2\n",
      "  (0, 35114)\t1\n",
      "  (0, 32450)\t9\n",
      "  (0, 27596)\t3\n",
      "  (0, 16795)\t11\n",
      "  (0, 16856)\t1\n",
      "  (0, 5707)\t1\n",
      "  (0, 33478)\t1\n",
      "  (0, 28858)\t1\n",
      "  (0, 1445)\t16\n",
      "  (0, 16855)\t1\n",
      "  (0, 8681)\t1\n",
      "  (0, 10024)\t1\n",
      "  (0, 22176)\t4\n",
      "  (0, 5285)\t5\n",
      "  (0, 33682)\t1\n",
      "  (0, 10525)\t1\n",
      "  (0, 2862)\t1\n",
      "  (0, 30800)\t1\n",
      "  (0, 32104)\t1\n",
      "  :\t:\n",
      "  (10, 8382)\t1\n",
      "  (10, 24977)\t1\n",
      "  (10, 18290)\t1\n",
      "  (10, 27020)\t1\n",
      "  (10, 15133)\t1\n",
      "  (10, 13610)\t1\n",
      "  (10, 23180)\t1\n",
      "  (10, 12099)\t1\n",
      "  (10, 24500)\t1\n",
      "  (10, 22155)\t1\n",
      "  (10, 30112)\t1\n",
      "  (10, 30275)\t1\n",
      "  (10, 30764)\t1\n",
      "  (10, 10569)\t1\n",
      "  (10, 14226)\t1\n",
      "  (10, 10259)\t1\n",
      "  (10, 11863)\t1\n",
      "  (10, 30584)\t1\n",
      "  (10, 18070)\t1\n",
      "  (10, 21762)\t1\n",
      "  (10, 33771)\t1\n",
      "  (10, 29624)\t1\n",
      "  (10, 26149)\t1\n",
      "  (10, 2265)\t1\n",
      "  (10, 396)\t1\n"
     ]
    }
   ],
   "source": [
    "print(train_counts[:11])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, I have the following results: (0, 5240)\t5 and (0, 33493)\t1. What does it mean? The first review (index 0) contains $5$ occurrences of some word with an index $5240$ and $2$ occurrences of a word with an index $33493$ from the vocabulary. Let's see what those indexes correspond to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'unfortunate'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect.get_feature_names()[33682]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'stuck'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "count_vect.get_feature_names()[30800]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this particular iteration, index $5240$ corresponds to the word *characters* and index $33493$ to the word *unfortunate*. By the way, according to the `train_targets`, the first review is positive (label `1`). Here is how you can check the whole list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array(['00', '000', '0009f', ..., 'anticipating', 'anticipation',\n",
      "       'anticlimactic'], dtype='<U32')]\n"
     ]
    }
   ],
   "source": [
    "print(count_vect.inverse_transform(train_data))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1600, 36094)\n",
      "  (0, 32056)\t1\n",
      "  (0, 5161)\t1\n",
      "  (0, 12240)\t1\n",
      "  (0, 22070)\t1\n",
      "  (0, 20177)\t1\n",
      "  (0, 35454)\t1\n",
      "  (0, 35114)\t1\n",
      "  (0, 32450)\t1\n",
      "  (0, 27596)\t1\n",
      "  (0, 16795)\t1\n",
      "  (0, 16856)\t1\n",
      "  (0, 5707)\t1\n",
      "  (0, 33478)\t1\n",
      "  (0, 28858)\t1\n",
      "  (0, 1445)\t1\n",
      "  (0, 16855)\t1\n",
      "  (0, 8681)\t1\n",
      "  (0, 10024)\t1\n",
      "  (0, 22176)\t1\n",
      "  (0, 5285)\t1\n",
      "  (0, 33682)\t1\n",
      "  (0, 10525)\t1\n",
      "  (0, 2862)\t1\n",
      "  (0, 30800)\t1\n",
      "  (0, 32104)\t1\n",
      "  :\t:\n",
      "  (0, 32193)\t1\n",
      "  (0, 20907)\t1\n",
      "  (0, 21810)\t1\n",
      "  (0, 10294)\t1\n",
      "  (0, 8110)\t1\n",
      "  (0, 11343)\t1\n",
      "  (0, 16040)\t1\n",
      "  (0, 5659)\t1\n",
      "  (0, 940)\t1\n",
      "  (0, 31239)\t1\n",
      "  (0, 35640)\t1\n",
      "  (0, 25810)\t1\n",
      "  (0, 194)\t1\n",
      "  (0, 11115)\t1\n",
      "  (0, 29672)\t1\n",
      "  (0, 23331)\t1\n",
      "  (0, 16105)\t1\n",
      "  (0, 4658)\t1\n",
      "  (0, 32085)\t1\n",
      "  (0, 1975)\t1\n",
      "  (0, 16595)\t1\n",
      "  (0, 8175)\t1\n",
      "  (0, 28157)\t1\n",
      "  (0, 17325)\t1\n",
      "  (0, 6270)\t1\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import Binarizer\n",
    "\n",
    "transformer = Binarizer()\n",
    "train_bin = transformer.fit_transform(train_counts)\n",
    "print(train_bin.shape)\n",
    "print(train_bin[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, let's run the classifier on the designated test set: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'lengthy and lousy are two words to describe the boring drama the english patient great acting music ' => pos\n",
      "'capsule a short punchy action sequel to the two dinosaur films made by steven spielberg joe johnston' => neg\n",
      "'now lets first look into the history of shark films there was the unforgettable jaws the exciting ja' => neg\n",
      "'five girls spend a day in a closed building doing inventory when a strange box gets delivered there ' => neg\n",
      "'one sided doom and gloom documentary about the possible annihilation of the human race as foretold b' => neg\n",
      "\"robert redford 's a river runs through it is not a film i watch often it is a masterpiece -- one of \" => pos\n",
      "\"susan granger 's review of america 's sweethearts columbia sony what a waste of a talented cast bill\" => neg\n",
      "'  the fugitive is probably one of the greatest thrillers ever made it takes realistic believable cha' => pos\n",
      "'just look back two years ago at the coen brothers comedic gem the big lebowski change the actors tak' => pos\n",
      "'plot during a 10-week span in london in the fall of 1888 jack the ripper a man whose identity was ne' => pos\n"
     ]
    }
   ],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "clf = MultinomialNB().fit(train_counts, train_targets)\n",
    "test_counts = count_vect.transform(test_data)\n",
    "predicted = clf.predict(test_counts)\n",
    "\n",
    "for text, label in list(zip(test_data, predicted))[:10]:\n",
    "    if label==1:\n",
    "        print('%r => %s' % (text[:100], \"pos\"))\n",
    "    else:\n",
    "        print('%r => %s' % (text[:100], \"neg\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, do the same using `sklearn`'s pipeline: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline(memory=None,\n",
      "         steps=[('vect',\n",
      "                 CountVectorizer(analyzer='word', binary=False,\n",
      "                                 decode_error='strict',\n",
      "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
      "                                 input='content', lowercase=True, max_df=0.5,\n",
      "                                 max_features=None, min_df=10,\n",
      "                                 ngram_range=(1, 1), preprocessor=None,\n",
      "                                 stop_words=None, strip_accents=None,\n",
      "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
      "                                 tokenizer=None, vocabulary=None)),\n",
      "                ('binarizer', Binarizer(copy=True, threshold=0.0)),\n",
      "                ('clf',\n",
      "                 MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))],\n",
      "         verbose=False)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import Binarizer\n",
    "\n",
    "text_clf = Pipeline([('vect', CountVectorizer(min_df=10, max_df=0.5)), \n",
    "                     ('binarizer', Binarizer()), # include this for detecting presence-absence of features\n",
    "                     ('clf', MultinomialNB())\n",
    "                    ])\n",
    "\n",
    "text_clf.fit(train_data, train_targets) \n",
    "print(text_clf)\n",
    "predicted = text_clf.predict(test_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Confusion matrix:\n",
      "[[173  29]\n",
      " [ 41 157]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          -1       0.81      0.86      0.83       202\n",
      "           1       0.84      0.79      0.82       198\n",
      "\n",
      "    accuracy                           0.82       400\n",
      "   macro avg       0.83      0.82      0.82       400\n",
      "weighted avg       0.83      0.82      0.82       400\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "print(\"\\nConfusion matrix:\")\n",
    "print(metrics.confusion_matrix(test_targets, predicted))\n",
    "print(metrics.classification_report(test_targets, predicted))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now with the cross-validation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.87  0.805 0.87  0.785 0.86  0.82  0.845 0.85  0.81  0.845]\n",
      "Accuracy: 0.8360000000000001\n",
      "\n",
      "Confusion matrix:\n",
      "[[862 138]\n",
      " [190 810]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          -1       0.82      0.86      0.84      1000\n",
      "           1       0.85      0.81      0.83      1000\n",
      "\n",
      "    accuracy                           0.84      2000\n",
      "   macro avg       0.84      0.84      0.84      2000\n",
      "weighted avg       0.84      0.84      0.84      2000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score, cross_val_predict\n",
    "\n",
    "scores = cross_val_score(text_clf, texts, labels, cv=10)\n",
    "print(scores)\n",
    "print(\"Accuracy: \" + str(sum(scores)/10))\n",
    "predicted = cross_val_predict(text_clf, texts, labels, cv=10)\n",
    "print(\"\\nConfusion matrix:\")\n",
    "print(metrics.confusion_matrix(labels, predicted))\n",
    "print(metrics.classification_report(labels, predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD8CAYAAABuHP8oAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAU0ElEQVR4nO3df7RlZX3f8fcnjAQZFQa8skZNFAkFjdUJ3hLBSlTELkkUtBq1xgzGOKb1d5MsadMqNTVVg6EmJqyOik6sUpWKoFUqmSr+Ri8wwgAaDCCCI1wUNYBRwW//OM91DpdzuecO5zDzOO/XWnedvZ+zf3z32Wd/7t7PPufeVBWSpP78ws4uQJK0YwxwSeqUAS5JnTLAJalTBrgkdcoAl6ROjRXgSV6ZZGuSS5O8qrXtl+TcJFe0xzXTLVWSNGzZAE/ySODFwOHAo4HfSnIwcCKwuaoOBja3cUnSPWScM/CHA1+sqlur6jbgPOAZwHHApjbNJuD46ZQoSRpl1RjTbAXekGR/4IfAscAccEBVbQOoqm1JHjBq5iQbgA0Aq1evfsyhhx46kcIlaXdxwQUX3FhVM4vblw3wqro8yZuAc4Gbga8At4274qraCGwEmJ2drbm5ubGLliRBkm+Mah/rJmZVvbOqDquqo4DvAlcA1ydZ2xa+FrhhUsVKkpY37qdQHtAefxl4JnA6cDawvk2yHjhrGgVKkkYbpw8c4H+3PvCfAC+tqpuSvBH4QJIXAdcAz55WkZKkOxsrwKvq8SPavgMcPfGKJElj8ZuYktQpA1ySOmWAS1KnDHBJ6pQBLkmdMsAlqVMGuCR1ygCXpE4Z4JLUKQNckjplgEtSpwxwSeqUAS5JnTLAJalTBrgkdcoAl6ROGeCS1CkDXJI6Ne4/NX51kkuTbE1yepK9krw7yVVJtrSfddMuVpK03bL/EzPJg4BXAI+oqh8m+QDw3Pb0H1fVGdMsUJI02rhdKKuAeydZBewNfGt6JUmSxrFsgFfVdcDJwDXANuD7VfWJ9vQbklyc5JQkvzjFOiVJiywb4EnWAMcBBwIPBFYn+R3gPwCHAv8C2A94zRLzb0gyl2Rufn5+YoVL0u5unC6UJwNXVdV8Vf0E+BBwZFVtq4EfAe8CDh81c1VtrKrZqpqdmZmZXOWStJsbJ8CvAR6bZO8kAY4GLk+yFqC1HQ9snV6ZkqTFlv0USlWdn+QM4ELgNuAiYCPw8SQzQIAtwB9Ms1BJ0h0tG+AAVfU64HWLmp80+XIkSePym5iS1CkDXJI6NVYXiqTdw9f3+J2dXcLPrV+5/X9OfJmegUtSpwxwSeqUAS5JnTLAJalTBrgkdcoAl6ROGeCS1CkDXJI6ZYBLUqcMcEnqlAEuSZ0ywCWpUwa4JHXKAJekThngktQpA1ySOjVWgCd5dZJLk2xNcnqSvZIcmOT8JFckeX+SPaddrCRpu2UDPMmDgFcAs1X1SGAP4LnAm4BTqupg4CbgRdMsVJJ0R+N2oawC7p1kFbA3sI3Bf6U/oz2/CTh+8uVJkpaybIBX1XXAycA1DIL7+8AFwPeq6rY22bXAg0bNn2RDkrkkc/Pz85OpWpI0VhfKGuA44EDggcBq4KkjJq1R81fVxqqararZmZmZu1OrJGnIOF0oTwauqqr5qvoJ8CHgSGDf1qUC8GDgW1OqUZI0wjgBfg3w2CR7JwlwNHAZ8EngWW2a9cBZ0ylRkjTKOH3g5zO4WXkhcEmbZyPwGuDfJ/k6sD/wzinWKUlaZNXyk0BVvQ543aLmK4HDJ16RJGksfhNTkjplgEtSpwxwSeqUAS5JnTLAJalTBrgkdcoAl6ROGeCS1CkDXJI6ZYBLUqcMcEnqlAEuSZ0ywCWpUwa4JHXKAJekThngktQpA1ySOmWAS1Knlv2XakkOAd4/1PQw4LXAvsCLgfnW/h+r6mMTr1CSNNKyAV5VXwPWASTZA7gOOBN4IXBKVZ081QolSSOttAvlaOAfquob0yhGkjS+lQb4c4HTh8ZfluTiJKclWTNqhiQbkswlmZufnx81iSRpB4wd4En2BJ4OfLA1nQocxKB7ZRvwllHzVdXGqpqtqtmZmZm7Wa4kacFKzsCfClxYVdcDVNX1VXV7Vf0UeDtw+DQKlCSNtpIAfx5D3SdJ1g499wxg66SKkiQtb9lPoQAk2Rs4BnjJUPObk6wDCrh60XOSpCkbK8Cr6lZg/0VtL5hKRZKksfhNTEnqlAEuSZ0ywCWpUwa4JHXKAJekThngktQpA1ySOmWAS1KnDHBJ6pQBLkmdMsAlqVMGuCR1ygCXpE4Z4JLUKQNckjplgEtSpwxwSeqUAS5JnVo2wJMckmTL0M8PkrwqyX5Jzk1yRXtcc08ULEkaWDbAq+prVbWuqtYBjwFuBc4ETgQ2V9XBwOY2Lkm6h6y0C+Vo4B+q6hvAccCm1r4JOH6ShUmS7tpKA/y5wOlt+ICq2gbQHh8waoYkG5LMJZmbn5/f8UolSXcwdoAn2RN4OvDBlaygqjZW1WxVzc7MzKy0PknSElZyBv5U4MKqur6NX59kLUB7vGHSxUmSlraSAH8e27tPAM4G1rfh9cBZkypKkrS8sQI8yd7AMcCHhprfCByT5Ir23BsnX54kaSmrxpmoqm4F9l/U9h0Gn0qRJO0EfhNTkjo11hm4tCN+770/3Nkl/Nw67fn33tklaBfgGbgkdcoAl6ROGeCS1CkDXJI6ZYBLUqcMcEnqlAEuSZ0ywCWpUwa4JHXKAJekThngktQpA1ySOmWAS1KnDHBJ6pQBLkmdMsAlqVPj/k/MfZOckeSrSS5PckSSk5Jcl2RL+zl22sVKkrYb9z/yvBU4p6qelWRPYG/gXwGnVNXJU6tOkrSkZQM8yf2Ao4ATAKrqx8CPk0y3MknSXRqnC+VhwDzwriQXJXlHktXtuZcluTjJaUnWjJo5yYYkc0nm5ufnJ1W3JO32xgnwVcBhwKlV9WvALcCJwKnAQcA6YBvwllEzV9XGqpqtqtmZmZnJVC1JGqsP/Frg2qo6v42fAZxYVdcvTJDk7cBHp1Dfz6x+4nunufjd2i2ffP7OLkHSDlj2DLyqvg18M8khrelo4LIka4cmewawdQr1SZKWMO6nUF4OvLd9AuVK4IXAXyZZBxRwNfCSqVQoSRpprACvqi3A7KLmF0y+HEnSuPwmpiR1ygCXpE4Z4JLUKQNckjplgEtSpwxwSeqUAS5JnTLAJalTBrgkdcoAl6ROGeCS1CkDXJI6ZYBLUqcMcEnqlAEuSZ0ywCWpUwa4JHXKAJekTo0V4En2TXJGkq8muTzJEUn2S3Jukiva45ppFytJ2m7cM/C3AudU1aHAo4HLgROBzVV1MLC5jUuS7iHLBniS+wFHAe8EqKofV9X3gOOATW2yTcDx0ypSknRn45yBPwyYB96V5KIk70iyGjigqrYBtMcHjJo5yYYkc0nm5ufnJ1a4JO3uxgnwVcBhwKlV9WvALaygu6SqNlbVbFXNzszM7GCZkqTFxgnwa4Frq+r8Nn4Gg0C/PslagPZ4w3RKlCSNsmyAV9W3gW8mOaQ1HQ1cBpwNrG9t64GzplKhJGmkVWNO93LgvUn2BK4EXsgg/D+Q5EXANcCzp1OiJGmUsQK8qrYAsyOeOnqy5UiSxuU3MSWpUwa4JHXKAJekThngktQpA1ySOmWAS1KnDHBJ6pQBLkmdMsAlqVMGuCR1ygCXpE4Z4JLUKQNckjplgEtSpwxwSeqUAS5JnTLAJalTBrgkdWqsAE9ydZJLkmxJMtfaTkpyXWvbkuTY6ZYqSRo27j81BnhiVd24qO2Uqjp5kgVJksZjF4okdWrcAC/gE0kuSLJhqP1lSS5OclqSNVOoT5K0hHED/HFVdRjwVOClSY4CTgUOAtYB24C3jJoxyYYkc0nm5ufnJ1GzJIkxA7yqvtUebwDOBA6vquur6vaq+inwduDwJebdWFWzVTU7MzMzqbolabe3bIAnWZ3kvgvDwFOArUnWDk32DGDrdEqUJI0yzqdQDgDOTLIw/fuq6pwk70myjkH/+NXAS6ZWpSTpTpYN8Kq6Enj0iPYXTKUiSdJY/BihJHXKAJekThngktQpA1ySOmWAS1KnDHBJ6pQBLkmdMsAlqVMGuCR1ygCXpE4Z4JLUKQNckjplgEtSpwxwSeqUAS5JnTLAJalTBrgkdcoAl6ROGeCS1Klx/qkxSa4G/hG4HbitqmaT7Ae8H3gog39q/NtVddN0ypQkLbaSM/AnVtW6qppt4ycCm6vqYGBzG5ck3UPuThfKccCmNrwJOP7ulyNJGleqavmJkquAm4AC/kdVbUzyvarad2iam6pqzYh5NwAb2ughwNcmUvmu7/7AjTu7CI3N/dWf3WmfPaSqZhY3jhvgD6yqbyV5AHAu8HLg7HECfHeVZG6ou0m7OPdXf9xnY3ahVNW32uMNwJnA4cD1SdYCtMcbplWkJOnOlg3wJKuT3HdhGHgKsBU4G1jfJlsPnDWtIiVJdzbOxwgPAM5MsjD9+6rqnCRfBj6Q5EXANcCzp1dmlzbu7AK0Iu6v/uz2+2ysPnBJ0q7Hb2JKUqcMcEnq1C4b4En+JMmlSS5OsiXJr+/gctYlOXZo/OlJpvqt0SRPSHLkEs+dkORtE1jHx5Lsu/yUu75J7Oul9nOSfZN8J+0mTpIjklSSB7fxfZJ8N8kOHwtJbm6PD23LfvnQc29LcsIy8x+f5BE7uv6doW3ne4bGVyWZT/LRNj6R9/kyNfzcHAM7apcM8CRHAL8FHFZVjwKeDHxzBxe3DvjZgV1VZ1fVG+9+lXfpCcDIAJ+Uqjq2qr43zXXcEya4r0fu5/YafRt4eHvqSOAitu+fxwLnV9VPx6x3uRv/NwCvTLLnCmo/HugqwIFbgEcmuXcbPwa47p4sYKXHwBj7rju7ZIADa4Ebq+pHAFV148Jn0ZM8Jsl5SS5I8n+HPov+qSRvSvKlJH+f5PHtIHo98Jx2Zvec4TODJO9OcmqSTya5MslvJDktyeVJ3r1QTJKnJPlCkguTfDDJfVr71Un+S2u/JMmhSR4K/AHw6rbOx4+zwaPW0c4Ov5bkkDbN6UlePLTu+7fh321nr19ZOCtK8pAkm1v75iS/PLTNf5nk822bn9Xa1yb5dKt567h1T8DIfT3J/Qx8ju2BfSRwyqLxz7dlr0vyxfaanZlkzdA6/yzJeQzC+cC2r76c5E8Xbc88g78NtH5RO0kOSnJO26bPtPfLkcDTgT9vtR80iRf1HvJx4Dfb8POA03d0Qe01nm3D98/gD+gtnMl/qL1uVyR589A8w8fAf07y1STntuPkj4aWO7zvnpbk/CQXJfm7JAe06U5KsinJJ9pyn5nkze24PifJvdp0b0xyWXuPnLyj2zsxVbXL/QD3AbYAfw/8DfAbrf1eDA62mTb+HOC0Nvwp4C1t+Fjg79rwCcDbhpb9s3Hg3cD/AsLgb7v8APjnDH6xXcDgrO7+wKeB1W2e1wCvbcNXAy9vw/8OeEcbPgn4oyW27Q71tLa7WscxwBeA5wLnDM1zdZvvVxn8eYL7t/b92uNHgPVt+PeADw9t8wfbNj4C+Hpr/0PgT9rwHsB9d9a+nsJ+PmFo/ouAvYDPtvFzgSe14YvZ/l57PfDfh9b5N0PLPhv43Tb8UuDmNvxQBt+ROBD4ansd3wac0J7fDBzchn8d+H9D++RZO/u4W+F+uxl4FHBGez23MLjy/OhS7/NllvcpYHboeLh6aDlXAvu09XwD+KVFx8BsW/+9gfsCV9COvxH7bg3bP333+0PvpZOAz7b33qOBW4GntufOZHCVtB+DY21h/n139n7YJS8pqurmJI8BHg88EXh/Bv3Wc8AjgXMz6NLcA9g2NOuH2uMFDA6mcXykqirJJcD1VXUJQJJL2zIezCDoPtfWuSeDQB21zmeOv5V38Nil1lFV5yZ5NvDXDN5Yiz0JOKOqbmzTf7e1HzFUz3uANw/N8+EadBlctnAGAnwZOK2daXy4qrbs4LasyKh9DfxXJrufPwecmORABsHwTxm4D/AY4EtJ9mFwQJ7X5tnE4BfdgvcPDT8O+Ndt+D3AmxZt01VJvgT8m4W2tq4jgQ+2bQL4xTFq32VV1cUZXHE+D/jYFFe1uaq+D5DkMuAh3LGb7V8CZ1XVD9s0H1k0//C+ezCDPFnL4Di7aui5j1fVT1oW7AGc09ovYfA++yjwT8A7kvyfNr5T7ZIBDlBVtzP47fmp9oKuZ3DAXlpVRywx24/a4+2Mv20L8/x0aHhhfFVb1rlV9bwJrnOxLLWODG6uPRz4IYMzgGtHzDvOh/mHpxnezgBU1aeTHMXgkvg9Sf68qv52/E3YcSP29UuZ4H6uqitad8jT2P7L9wLghcBV7ZfIPsss5pbFi11m+j9jcHb66Tb+C8D3qmrdcvV25mzgZAZn3/vfjeXcxvYu3b0WPTf8fh21z8NdG953fwX8RVWdneQJDM6877Ceqvppkp9UO82mZUFV3ZbkcOBoBlfEL2NwArXT7JJ94EkOSXLwUNM6BpdOXwNmMrjxRZJ7JfnVZRb3jwwuq3bUF4HHJfmVts69k/yzCa/zrtbxauByBmc5C2fIwzYDv51k/zbvfq398wzeZADPZ3B5uKQkDwFuqKq3A+8EDltB/TtsiX19OZPfz18AXsn2AP8C8Cpa/3c7w7sp2/v+XwCct3ghzee442t7J1X1VeAyBjdoqaofAFe1qynaFcDCFdXdfY/uTKcBr1+4cl1Okr9tIbjY1QyuhgCetcIaPgs8Lcle7UrnN+9i2n3YfrP1Tvcp7kpb9j5V9TEG752d/st4lwxwBv2imxZuFjDoXjipqn7MYOe+KclXGPR7Lfdpj08Cj2g3iJ6z0kKqap5BP9zprZYvAocuM9tHgGdk6ZuYJyS5duGHwaX0ndbRQvz3gT+sqs8wOJv7T4vquxR4A3Bee03+oj31CuCFbXkvYBBed+UJwJYkFzHoHnjrMtNPyqh9/Vomv58/B/wSg244GAT4w2gB3qxncDPxYgYH5+uXWNcrgZdm8Ock7urM/Q0MLtkXPB94UdumSxncd4HBfZg/bjfWerqJSVVdW1VLvVfu8D7P4KObj+KO3WELTgb+bZLPM+jXXkkNX2ZwJfAVBt1rc8D3l5j8JAbdWJ9h5X+K9r7AR9v74zwGJ1c7lV+ll3SPSHI/4J1VNfG/m5TkPq0rbG8GJzobqurCSa9nV2OAS+pekvcxuHrbC9hUVf9tJ5d0jzDAJalTu2ofuCRpGQa4JHXKAJekThngktQpA1ySOvX/Afq4Cf6XY0xiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function matplotlib.pyplot.legend(*args, **kwargs)>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "a = [\"Sentiment Lexicons\", \"SentiWordNet\", \"ML, unigrams\"]\n",
    "index = range(len(a))\n",
    "b = [64.75, 68.80, 83.60] # Accuracy scores for the compared methods\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "axes = plt.gca()\n",
    "# Let's set 50 as the lower bound as the majority class baseline is at 50 for this dataset\n",
    "axes.set_ylim([50,90])\n",
    "\n",
    "ax.bar(index, b, color=['#0A40A4', '#61A4F6', '#DB025B'])\n",
    "plt.xticks(index, a) \n",
    "\n",
    "plt.show()\n",
    "\n",
    "import matplotlib\n",
    "matplotlib.axes.Axes.plot\n",
    "matplotlib.pyplot.plot\n",
    "matplotlib.axes.Axes.legend\n",
    "matplotlib.pyplot.legend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, run the `CountVectorizer` on a set of word uni- and bi-grams (with `ngram_range=(1, 2)`) or even uni- + bi- + tri-grams (with `ngram_range=(1, 3)`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline(memory=None,\n",
      "         steps=[('vect',\n",
      "                 CountVectorizer(analyzer='word', binary=False,\n",
      "                                 decode_error='strict',\n",
      "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
      "                                 input='content', lowercase=True, max_df=1.0,\n",
      "                                 max_features=None, min_df=1,\n",
      "                                 ngram_range=(1, 3), preprocessor=None,\n",
      "                                 stop_words=None, strip_accents=None,\n",
      "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
      "                                 tokenizer=None, vocabulary=None)),\n",
      "                ('binarizer', Binarizer(copy=True, threshold=0.0)),\n",
      "                ('clf',\n",
      "                 MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))],\n",
      "         verbose=False)\n"
     ]
    }
   ],
   "source": [
    "text_clf = Pipeline([('vect', CountVectorizer(ngram_range=(1, 3))), \n",
    "                     ('binarizer', Binarizer()), #presence-absence of features\n",
    "                     ('clf', MultinomialNB())\n",
    "                    ])\n",
    "\n",
    "text_clf.fit(train_data, train_targets) \n",
    "print(text_clf)\n",
    "predicted = text_clf.predict(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Confusion matrix:\n",
      "[[158  44]\n",
      " [ 24 174]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          -1       0.87      0.78      0.82       202\n",
      "           1       0.80      0.88      0.84       198\n",
      "\n",
      "    accuracy                           0.83       400\n",
      "   macro avg       0.83      0.83      0.83       400\n",
      "weighted avg       0.83      0.83      0.83       400\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(\"\\nConfusion matrix:\")\n",
    "print(metrics.confusion_matrix(test_targets, predicted))\n",
    "print(metrics.classification_report(test_targets, predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.89  0.86  0.87  0.835 0.895 0.82  0.86  0.855 0.825 0.88 ]\n",
      "Accuracy: 0.859\n",
      "\n",
      "Confusion matrix:\n",
      "[[810 190]\n",
      " [ 92 908]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          -1       0.90      0.81      0.85      1000\n",
      "           1       0.83      0.91      0.87      1000\n",
      "\n",
      "    accuracy                           0.86      2000\n",
      "   macro avg       0.86      0.86      0.86      2000\n",
      "weighted avg       0.86      0.86      0.86      2000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "scores = cross_val_score(text_clf, texts, labels, cv=10)\n",
    "print(scores)\n",
    "print(\"Accuracy: \" + str(sum(scores)/10))\n",
    "predicted = cross_val_predict(text_clf, texts, labels, cv=10)\n",
    "print(\"\\nConfusion matrix:\")\n",
    "print(metrics.confusion_matrix(labels, predicted))\n",
    "print(metrics.classification_report(labels, predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAYQElEQVR4nO3de5RcVZnG4d9LYoQEMAEaJqLIRQZEHAK0GQFBIIjCqICi4FKMyhAZRQWvOI4QHHVAQQZkiURAMg5mkECGiIpEhouIIJ3QQCAgCjEEQmgQkIsCCd/8sXebolLddbpT1c1Ov89avarOrerbp7rfPrXr7FOKCMzMrDzrDHcBZmY2OA5wM7NCOcDNzArlADczK5QD3MysUA5wM7NCVQpwSZ+WtFDSHZKOzfM2kjRP0j35dkJ7SzUzs1pNA1zSjsBRwGRgJ+AdkrYFjgeuiohtgavytJmZDZEqR+CvA26MiGciYgVwLXAIcBAwM68zEzi4PSWamVkjVQJ8IbCXpI0ljQUOBF4NbBYRywDy7abtK9PMzOqNbrZCRCySdAowD3gKuBVYUfUJJE0DpgGMGzdu1+23336QpZqZjUzz589/JCI66udroNdCkfQNYCnwaWDviFgmaSJwTURs19+2nZ2d0dXVNaDnMzMb6STNj4jO+vlVz0LZNN9uAbwbmAXMBabmVaYCl7WmVDMzq6JpF0p2iaSNgeeBT0TEY5JOBn4s6UhgCfDedhVpZmarqxTgEbFng3mPAlNaXpGZmVXikZhmZoVygJuZFcoBbmZWKAe4mVmhHOBmZoVygJuZFcoBbmZWKAe4mVmhHOBmZoVygJuZFcoBbmZWKAe4mVmhHOBmZoVygJuZFcoBbmZWKAe4mVmhHOBmZoWq+pVqZmZD7qsTvzbcJbTMCcv+reWPWfVLjY+TdIekhZJmSVpX0gWS7pPUnX8mtbw6MzPrU9MjcEmbA58CdoiIv0j6MXB4Xvz5iJjdzgLNzKyxqn3go4H1JI0GxgIPtq8kMzOrommAR8QDwKnAEmAZ8EREXJkXf13SbZJOl/TyNtZpZmZ1qnShTAAOArYCHgculvRB4EvAQ8AYYAbwReCrDbafBkwD2GKLLVpWuNlIcHXncFfQOvt0DXcFa58qXSj7AfdFRE9EPA9cCuweEcsieRb4ATC50cYRMSMiOiOis6Ojo3WVm5mNcFUCfAnwJkljJQmYAiySNBEgzzsYWNi+Ms3MrF7TLpSIuEnSbGABsAK4hdRl8nNJHYCAbuDodhZqZmYvVmkgT0ScCJxYN3vf1pdjZmZVeSi9mVmhHOBmZoVygJuZFcoXs7KXvN+P+uBwl9Ayr13538Ndgq1FfARuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhaoU4JKOk3SHpIWSZklaV9JWkm6SdI+kiySNaXexZma2StMAl7Q58CmgMyJ2BEYBhwOnAKdHxLbAY8CR7SzUzMxerGoXymhgPUmjgbHAMtK30s/Oy2cCB7e+PDMz60vTAI+IB4BTgSWk4H4CmA88HhEr8mpLgc0bbS9pmqQuSV09PT2tqdrMzCp1oUwADgK2Al4JjAMOaLBqNNo+ImZERGdEdHZ0dKxJrWZmVqNKF8p+wH0R0RMRzwOXArsD43OXCsCrgAfbVKOZmTVQJcCXAG+SNFaSgCnAncDVwKF5nanAZe0p0czMGqnSB34T6cPKBcDteZsZwBeBz0j6PbAxcF4b6zQzszqjm68CEXEicGLd7HuByS2vyMzMKvFITDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK1fQr1SRtB1xUM2tr4ARgPHAU0JPn/2tE/KzlFZqZWUNNAzwi7gYmAUgaBTwAzAE+ApweEae2tUIzM2tooF0oU4A/RMQf21GMmZlVN9AAPxyYVTN9jKTbJJ0vaUKjDSRNk9Qlqaunp6fRKmZmNgiVA1zSGOBdwMV51tnANqTulWXAaY22i4gZEdEZEZ0dHR1rWK6ZmfUayBH4AcCCiFgOEBHLI2JlRLwAfB+Y3I4CzcyssYEE+Pup6T6RNLFm2SHAwlYVZWZmzTU9CwVA0ljgrcDHamZ/U9IkIIDFdcvMzKzNKgV4RDwDbFw374i2VGRmZpV4JKaZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhWoa4JK2k9Rd8/NnScdK2kjSPEn35NsJQ1GwmZklTQM8Iu6OiEkRMQnYFXgGmAMcD1wVEdsCV+VpMzMbIgPtQpkC/CEi/ggcBMzM82cCB7eyMDMz699AA/xwYFa+v1lELAPIt5s22kDSNEldkrp6enoGX6mZmb1I5QCXNAZ4F3DxQJ4gImZERGdEdHZ0dAy0PjMz68NAjsAPABZExPI8vVzSRIB8+3CrizMzs74NJMDfz6ruE4C5wNR8fypwWauKMjOz5ioFuKSxwFuBS2tmnwy8VdI9ednJrS/PzMz6MrrKShHxDLBx3bxHSWelmJnZMPBITDOzQlU6Arfh99EL/zLcJbTE+R9Yb7hLMFtr+AjczKxQDnAzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MCuUANzMrVNXvxBwvabakuyQtkrSbpOmSHpDUnX8ObHexZma2StVv5DkDuCIiDpU0BhgLvA04PSJObVt1ZmbWp6YBLmlDYC/gwwAR8RzwnKT2VmZmZv2q0oWyNdAD/EDSLZLOlTQuLztG0m2Szpc0oX1lmplZvSoBPhrYBTg7InYGngaOB84GtgEmAcuA0xptLGmapC5JXT09Pa2p2szMKvWBLwWWRsRNeXo2cHxELO9dQdL3gcsbbRwRM4AZAJ2dnTHYQsftc+FgN33JefrqDwx3CWa2Fmh6BB4RDwH3S9ouz5oC3ClpYs1qhwAL21CfmZn1oepZKJ8ELsxnoNwLfAQ4U9IkIIDFwMfaUqGZmTVUKcAjohvorJt9ROvLMTOzqjwS08ysUA5wM7NCOcDNzArlADczK5QD3MysUA5wM7NCOcDNzArlADczK5QD3MysUA5wM7NCOcDNzArlADczK5QD3MysUA5wM7NCOcDNzArlADczK5QD3MysUA5wM7NCVQpwSeMlzZZ0l6RFknaTtJGkeZLuybcT2l2smZmtUvUI/AzgiojYHtgJWAQcD1wVEdsCV+VpMzMbIk0DXNKGwF7AeQAR8VxEPA4cBMzMq80EDm5XkWZmtroqR+BbAz3ADyTdIulcSeOAzSJiGUC+3bSNdZqZWZ0qAT4a2AU4OyJ2Bp5mAN0lkqZJ6pLU1dPTM8gyzcysXpUAXwosjYib8vRsUqAvlzQRIN8+3GjjiJgREZ0R0dnR0dGKms3MjAoBHhEPAfdL2i7PmgLcCcwFpuZ5U4HL2lKhmZk1NLriep8ELpQ0BrgX+Agp/H8s6UhgCfDe9pRoZmaNVArwiOgGOhssmtLacszMrCqPxDQzK5QD3MysUA5wM7NCOcDNzArlADczK5QD3MysUA5wM7NCOcDNzArlADczK5QD3MysUA5wM7NCOcDNzArlADczK5QD3MysUA5wM7NCOcDNzArlADczK5QD3MysUJUCXNJiSbdL6pbUledNl/RAntct6cD2lmpmZrWqfqkxwD4R8UjdvNMj4tRWFmRmZtW4C8XMrFBVAzyAKyXNlzStZv4xkm6TdL6kCW2oz8zM+lA1wPeIiF2AA4BPSNoLOBvYBpgELANOa7ShpGmSuiR19fT0tKJmMzOjYoBHxIP59mFgDjA5IpZHxMqIeAH4PjC5j21nRERnRHR2dHS0qm4zsxGvaYBLGidpg977wP7AQkkTa1Y7BFjYnhLNzKyRKmehbAbMkdS7/o8i4gpJP5Q0idQ/vhj4WNuqNDOz1TQN8Ii4F9ipwfwj2lKRmZlV4tMIzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFBVvtQYSYuBJ4GVwIqI6JS0EXARsCXpS43fFxGPtadMMzOrN5Aj8H0iYlJEdObp44GrImJb4Ko8bWZmQ2RNulAOAmbm+zOBg9e8HDMzq0oR0Xwl6T7gMSCAcyJihqTHI2J8zTqPRcSEBttOA6blye2Au1tSeftsAjwy3EUME7d95BrJ7S+h7a+JiI76mVUD/JUR8aCkTYF5wCeBuVUCvDSSumq6iUYUt31kth1GdvtLbnulLpSIeDDfPgzMASYDyyVNBMi3D7erSDMzW13TAJc0TtIGvfeB/YGFwFxgal5tKnBZu4o0M7PVVTmNcDNgjqTe9X8UEVdIuhn4saQjgSXAe9tX5pCaMdwFDCO3feQaye0vtu2V+sDNzOylxyMxzcwK5QA3MytUUQEu6cuS7pB0m6RuSf84iMeYJOnAmul3SWrrKFJJe0vavZ3P0eA5i9xXTerZUtLCunnTJX2un206JZ3Z/uraR9L5kh6ub3vN8ldLulrSovyaf3qoa2ynCu1fV9JvJd2a23/SUNc4XCpdC+WlQNJuwDuAXSLiWUmbAGMG8VCTgE7gZwARMZd0Rk077Q08BdzQ5ucBit9XLRURXUBX1fWVPq1XRLzQvqoG7ALgLOC/+li+AvhsRCzIZ4zNlzQvIu4czJNJGh0RKwZXaltcQP/tfxbYNyKekvQy4HpJP4+IGwfzZJJGRcTKwZU6tEo6Ap8IPBIRzwJExCN5cNGukq6VNF/SL2rOTb9G0in5P/PvJO0paQzwVeCwfFR6mKQPSzorb3OBpLPz0cy9kt6S//svknRBbyGS9pf0G0kLJF0saf08f7Gkk/L82yVtL2lL4GjguPyce64N++qlplEb8vy9JV2e73dImpdfn3Mk/VHSJvnIfpGk7wILgFfn34Ou+iO6/Bp/I7/+XZJ2yfvyD5KOzutMlHRd3m8L1/Q1j4jrgD/1s3xZRCzI958EFgGb97GfviLprrwfZvW+e8n77xuSrgU+Lemdkm6SdIukX0raLK83XdJMSVfmffFuSd/Mv+9X5ABF0smS7lR6B3hqm9sfEfFUnnxZ/lnt7AxJ60j6bn5NL5f0M0mH5mWLJZ0g6XrgvZKOknSz0lH9JZLG5vWaZoSkUXm9hXm/HLcm7e9XRBTxA6wPdAO/A74LvCW/UDcAHXmdw4Dz8/1rgNPy/QOBX+b7HwbOqnncv02T/tP/DyDStV7+DLyB9I9uPumIdBPgOmBc3uaLwAn5/mLgk/n+x4Fz8/3pwOfWpn01TL8DWwIL6+ZNBz7XTxv2Bi7P988CvpTvv530R75JftwXgDfVPO5G+XZUfux/qHmN/yXfPx24DdgA6AAezvM/C3y5ZvsN2tH2ftZbAmzYYFln/r1YL9d8T+/vZW7jd2vWncCqs9T+uWbfTgeuz79POwHPAAfkZXNI10TaiHTJjN7tx7e7/Xk/d5Pe6Z7SxzqHkt5NrgP8HenyIIfWvK5fqFl345r7X2PV3/UFNM+IXYF5Nduvcfv7+immCyXS26NdgT2BfUiXsv0asCMwT+k89VHAsprNLs2380m/AFX8JCJC0u3A8oi4HUDSHfkxXgXsAPw6P+cY4Dd9POe7q7ewdYZwXw21vs557Z3frA1vBg4BiDSWofbyx3+MF7/lfp/SdXxGk97R7EAKa1jVjXQ7sH6ko94nJf1V0njgZuD8fDT6vxHRXbWBayK/E7wEODYi/txglTcDl0XEX/L6P6lbflHN/VcBF+V3aWOA+2qW/Twins9/I6OAK/L820n7/XLgr8C5kn6ap9sqUpfHpLz/50jaMSLq+8zfDFwcqXvsIUlX1y2vbf+Okr4GjCcdEP2iZlmzjLgW2FrSd4CfAle2ppWrKybA4W8v0jXANXnnfQK4IyJ262OTZ/PtSqq3tXebF2ru906Pzo81LyLe38LnbLl27itJo0ghCemaOCesecWVPEo6Mqy1EavCpVkb1M9jP/23laStSEf1b4yIx/Jb43Vr1u33dyQirpO0F/BPwA8lfSsi+uq/HTBJrwZ6w/d7EfG9/M/iEuDCiLi00Xr0336o2QfAd4BvR8RcSXuTjrx79XbNvSDp+ciHmaxq/wpJk4EpwOHAMcC+A29pY43a37ssIh6XdA3wdqWR4+fkRScwsPZfABwcEbdK+jDpnVyvZq//Y5J2At5G+rt7H/DRSo0boGL6wCVtJ2nbmlmTSH19HUof2iHpZZJe3+ShniS9fRysG4E9JL02P+dYSX/f5ucckHbvq4hYGena8JOGMLyJ1M+5TNIUAKUvFXk76S19FdeT/piQtD+r/zPotSHpj/mJ3Pd7wEDqlPQaUnfK94HzgF0Gsn0zEXF/zf7/ntJbqvOARRHx7b7WI7X/nUpnbaxP+gfTl1cAD+T7U/tZbzX5sV8RET8DjiX9/rVMg/Z35CNvJK0H7AfcFRE31aw3l9T+9+S+8M14cSjX24D0u/Yy4AMDqU/ppIF1IuIS4Cu0+PWvVUyAk97GzOz9YIT0lvYEUr/WKZJuJfWBNTtd72pgB+UP5gZaRET0kPqCZ+U6bgS2b7LZT4BDNHQfYr4k9lWbfAj4N0ndwP8BJ0XEHypuexKwv6QFpFBeRvon9SIRcStwC3AHcD7w6wHWuDfQLekW4D3AGQPc/kUkzSJ1020naanS5Stq7QEcAeybX6tu1Zz+2SsibiZ1/9xK6m7qAp7o42mnAxdL+hUDv9TqBsDl+XfvWmCNPsSr0P6JwNX5+W4mvUNu1G1zCbCUdC2nc4Cb6Lv9X8nL5wF3DbDkzUnvfLtJR/JfGuD2lXkovY0Ykl4OrMxv8XcDzo6Ilh4dvtRJWj9/RjKW9GH8tMhnsIwENe3fGPgtsEdEPDTcdQ1WUX3gZmtoC9IF2NYBngOOGuZ6hsMMSTuQ+vRnjqTwzi7P3S1jgH8vObzBR+BmZsUqqQ98NZKear5W08c4WtKHWlHPUJF0uqRja6Z/IencmunTJH1mkI/9t6HpeTDCA7nrAaVBL4ubbD9e0scH89xmNjBFB3grRMT3WnmK1xC5gfwBZO4O2ASoPaNkdyp88JZPB2xmJQM7BWo8aRCTmbXZWhfg+ZSiS5SGwd4saY88/0xJJ+T7b1Ma6rxO3RHna5WGDd+qNNx6GyXf0qphsYfldfdWGn48W2lo8oX5dK6WDiPuw69ZdQbJ60mfqj8paUI+Wn4d6SyIvuq+WtKPSAMvei98dbekX5K+eLrWf5IuA7Da5yWSPp/38W1aNdz8ZGCbfCbEt1rdcDNbZW38EPMM4PSIuF7SFqQRVK8DjgduzqdFnQkcmAci1G57IXByRMyRtC7pH9y7Seex7kQ60r1Z0nV5/Z1JAfogKVT3kHQnabTf9nm01nhaLNJ1TVbk9u1OOsVqc2A30mlRt5EuZtVX3ZOBHSPiPqURm4fntowmXQtkfs3TLSGdP3sEqwZP9J5HvW1+LAFzlQavHJ8fe0Sd3WE2HNbGAN+PdO5y7/SGkjaIiCclHUU6deq4+nOHla7itnlEzAGIiL/m+W8GZuWRjcuVLvbzRtI1EH4bEUvzet2kYbQ3MjTDiHuPwncHvk0K8N1JAX4Dadhwf3X3jl7cE5gTEc/kdjS62uA3SOcP/7Rm3v7555Y8vT4p0Je0qoFm1r+1McDXAXbrvd5DnTeQhmO/ssGyvobZ9jf8tnYY7UqGYBhxjd5+8DeQulDuJ11E6c+kwSdT+tn26brpfk9Fiojf539Q76uZLeA/IuKc2nWVrr5oZkNgresDJ1045pjeCUmT8u1rSAG3M3CA6r7gIF/8Z6mkg/P6L68Z7HCY0iUiO4C9SAMAGlKbhxHX+DWpm+RPeWj7n0gfIO5G6lKpWvd1pFGi6+V3Ie/s4/m+Tro+SK9fAB/Vqkvpbi5pU4b4sgFmI1npAT5WaWht789ngE8BnfmDtTuBo/OHi+eRLp35IHAkqYtj3brHOwL4lNKQ3BtIl5ycQ+pTvpU0dPsLTU7+b+kw4n7cTurbvrFu3hMR8QgV684DOS4iDa2/BPhVoyeLiDtI/eO901cCPwJ+o3SxrNmky6Y+SrpS40J/iGnWXh7IY2ZWqNKPwM3MRiwHuJlZoRzgZmaFcoCbmRXKAW5mVigHuJlZoRzgZmaFcoCbmRXq/wFBmtoM8jZ31AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function matplotlib.pyplot.legend(*args, **kwargs)>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "a = [\"Sentiment\\nLexicons\", \"Senti-\\nWordNet\", \"Unigrams\", \"1-2-grams\", \"1-3-grams\"]\n",
    "index = range(len(a))\n",
    "b = [64.75, 68.80, 83.60, 85.05, 85.90] # Accuracy scores for the compared methods\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "axes = plt.gca()\n",
    "# Let's set 50 as the lower bound as the majority class baseline is at 50 for this dataset\n",
    "axes.set_ylim([50,90])\n",
    "\n",
    "ax.bar(index, b, color=['#0A40A4', '#61A4F6', '#DB025B', '#BE33FF', '#811883'])\n",
    "plt.xticks(index, a) \n",
    "\n",
    "plt.show()\n",
    "\n",
    "import matplotlib\n",
    "matplotlib.axes.Axes.plot\n",
    "matplotlib.pyplot.plot\n",
    "matplotlib.axes.Axes.legend\n",
    "matplotlib.pyplot.legend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional exercise: Consider only pre-defined words as features\n",
    "\n",
    "You can use selected words (e.g., adjectives identified by their part-of-speech tag or words from specific  sentiment lexicons) only.\n",
    "\n",
    "First with the lexicons:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000 2000\n",
      "at end an action scene wall near warrior her face floor she into then abruptly says oh my car keys mention scene because only moment whole damned production made me smile based an incredibly popular video game lousy movie structure goes poorly action sequence boring exposition poorly action sequence boring exposition etc etc end basically film exists jolie puffy lips tits close ups framed oh carefully include her million dollar running scenes highlight her breasts fashion there even include shower scene offers brief side view them filmmakers ca even flash audience correctly one extended display nudity get ready guy no particular reason muscular supporting character around naked about 30 seconds each shot barely cover his package la austin powers now enjoy good looking male body much next gay guy what hell doing flick aimed at males course what else should one expect film does virtually nothing right sets up elaborate action set pieces then them incomprehensible needless jump cut editing promising dual battle ruined by excessive cuts promises series exotic then cavernous sets grimy paintings throw some basement computer graphics end up movie come down many intended indiana jones style adventure any sense tension low point comes when threatened by monkey warriors giant armed figure come life cgi easily ever seen creatures move fall apart single shot from gun if ever by monsters pray get monkey warriors tells story sort once every 5 years planets group very bad men out find two an object if planets will give them control over time itself goal stop them rescue her long missing jon jolie real life dad none matters though because stunning internal logic plot lame ever other aspect film except jolie lips breasts\n"
     ]
    }
   ],
   "source": [
    "def text_filter_incl(a_dict, label, include_lists):\n",
    "    data = []\n",
    "    for rev_id in a_dict.keys():\n",
    "        tokens = []\n",
    "        for token in a_dict.get(rev_id):\n",
    "            if token.text in include_lists:\n",
    "                tokens.append(token.text)\n",
    "        data.append((' '.join(tokens), label))\n",
    "    return data\n",
    "\n",
    "def prepare_data_incl(pos_docs, neg_docs, include_lists):\n",
    "    data = text_filter_incl(pos_docs, 1, include_lists)\n",
    "    data += text_filter_incl(neg_docs, -1, include_lists)\n",
    "    random.shuffle(data)\n",
    "    texts = []\n",
    "    labels = []\n",
    "    for item in data:\n",
    "        texts.append(item[0])\n",
    "        labels.append(item[1])\n",
    "    return texts, labels\n",
    "\n",
    "#texts_incl, labels_incl = prepare_data(pos_docs, neg_docs, set(list(adj_90.keys()) + list(adj_00.keys())))\n",
    "texts_incl, labels_incl = prepare_data_incl(pos_docs, neg_docs, \n",
    "                                       set(list(adj_90.keys()) + list(adj_00.keys()) + \n",
    "                                       list(all_90.keys()) + list(all_00.keys()) + list(movie_words.keys())))\n",
    "\n",
    "print(len(texts_incl), len(labels_incl))\n",
    "print(texts_incl[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pipeline(memory=None,\n",
      "         steps=[('vect',\n",
      "                 CountVectorizer(analyzer='word', binary=False,\n",
      "                                 decode_error='strict',\n",
      "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
      "                                 input='content', lowercase=True, max_df=1.0,\n",
      "                                 max_features=None, min_df=1,\n",
      "                                 ngram_range=(1, 2), preprocessor=None,\n",
      "                                 stop_words=None, strip_accents=None,\n",
      "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
      "                                 tokenizer=None, vocabulary=None)),\n",
      "                ('binarizer', Binarizer(copy=True, threshold=0.0)),\n",
      "                ('clf',\n",
      "                 MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))],\n",
      "         verbose=False)\n"
     ]
    }
   ],
   "source": [
    "text_clf = Pipeline([('vect', CountVectorizer(ngram_range=(1, 2))), \n",
    "                     #options: ngram_range=(1, 2) or min_df=10, max_df=0.5\n",
    "                     ('binarizer', Binarizer()), #presence-absence of features\n",
    "                     ('clf', MultinomialNB()),\n",
    "                    ])\n",
    "\n",
    "print(text_clf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.88  0.82  0.87  0.835 0.85  0.885 0.87  0.82  0.855 0.865]\n",
      "Accuracy: 0.8550000000000001\n",
      "\n",
      "Confusion matrix:\n",
      "[[830 170]\n",
      " [120 880]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          -1       0.87      0.83      0.85      1000\n",
      "           1       0.84      0.88      0.86      1000\n",
      "\n",
      "    accuracy                           0.85      2000\n",
      "   macro avg       0.86      0.85      0.85      2000\n",
      "weighted avg       0.86      0.85      0.85      2000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "scores = cross_val_score(text_clf, texts_incl, labels_incl, cv=10)\n",
    "print(scores)\n",
    "print(\"Accuracy: \" + str(sum(scores)/10))\n",
    "predicted = cross_val_predict(text_clf, texts_incl, labels_incl, cv=10)\n",
    "print(\"\\nConfusion matrix:\")\n",
    "print(metrics.confusion_matrix(labels_incl, predicted))\n",
    "print(metrics.classification_report(labels_incl, predicted))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now with the part-of-speech tags:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000 2000\n",
      "last sore when not aloud virtually n't as much second first great goofy dandy energetic inspired flat most fun zippy retro straight slapstick then extremely rich venetian unusual high half very special together lucky where simple n there ahead first wrong next few flashy trashy big erudite conservative more little wary together clear fierce competitive currently most hated all enormous high vengeful new cheerful odd little italian also narcoleptic asleep most inopportune wayne hyper tense human fearful angry together so mini green young crooked stupid poorly only close finally smart first blandly wholesome when periodically where right bizarre few flat high intricately bizarre outdoor utterly tasteless screamingly funny filmmaking where instead same also different really yet smash enough terrific diverse again when hilarious surely\n"
     ]
    }
   ],
   "source": [
    "def text_filter_postag(a_dict, label, pos_lists):\n",
    "    data = []\n",
    "    for rev_id in a_dict.keys():\n",
    "        tokens = []\n",
    "        for token in a_dict.get(rev_id):\n",
    "            if token.pos_ in pos_lists:\n",
    "                tokens.append(token.text)\n",
    "        data.append((' '.join(tokens), label))\n",
    "    return data\n",
    "\n",
    "def prepare_data_postag(pos_docs, neg_docs, pos_lists):\n",
    "    data = text_filter_postag(pos_docs, 1, pos_lists)\n",
    "    data += text_filter_postag(neg_docs, -1, pos_lists)\n",
    "    random.shuffle(data)\n",
    "    texts = []\n",
    "    labels = []\n",
    "    for item in data:\n",
    "        texts.append(item[0])\n",
    "        labels.append(item[1])\n",
    "    return texts, labels\n",
    "\n",
    "texts_postag, labels_postag = prepare_data_postag(pos_docs, neg_docs, [\"ADJ\", \"ADV\"])\n",
    "\n",
    "print(len(texts_postag), len(labels_postag))\n",
    "print(texts_postag[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.845 0.875 0.845 0.875 0.855 0.82  0.85  0.81  0.82  0.865]\n",
      "Accuracy: 0.8460000000000001\n",
      "\n",
      "Confusion matrix:\n",
      "[[828 172]\n",
      " [136 864]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          -1       0.86      0.83      0.84      1000\n",
      "           1       0.83      0.86      0.85      1000\n",
      "\n",
      "    accuracy                           0.85      2000\n",
      "   macro avg       0.85      0.85      0.85      2000\n",
      "weighted avg       0.85      0.85      0.85      2000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "scores = cross_val_score(text_clf, texts_postag, labels_postag, cv=10)\n",
    "print(scores)\n",
    "print(\"Accuracy: \" + str(sum(scores)/10))\n",
    "predicted = cross_val_predict(text_clf, texts_postag, labels_postag, cv=10)\n",
    "print(\"\\nConfusion matrix:\")\n",
    "print(metrics.confusion_matrix(labels_postag, predicted))\n",
    "print(metrics.classification_report(labels_postag, predicted))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Addressing negation\n",
    "\n",
    "Use NLTK's negation \"trick\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2000 2000\n",
      "the central focus of michael winterbottom's welcome to sarajevo is sarajevo itself the city under siege and its different effect on the characters unfortunate enough to be stuck there it proves the backdrop for a stunningly realized story which refreshingly strays from mythic portents platoon racial tumultuosness the risible the walking dead or a tinge of schmaltziness schindler's list the two leads stephen dillane as a reporter and emira nusevic as an orphan with a plight few can identify with are extremely believable not one_NEG moment_NEG with_NEG them_NEG involved_NEG rings_NEG false_NEG the question is not what_NEG went_NEG right_NEG the question is what went wrong for one the film fails to provide a political overview of the war as it progresses the dillane characters reports an american plane departing from sarajevo as it departs and that's about it the assortment of high-profile supporting actors ranging from woody harrelson as a yankee reporter into liquor and cigarrettes to marisa tomei as a huggable children's aid or somesuch are incapable of rising above the sketchiness of their characters albeit they strive the interrupted use of authentic war footage somewhat hampers the rest of the film it makes the fictional characters seem powerless by comparison still winterbottom eschews mawkishness through flashy frantic editing and imaginative use of music and it's a plus because he doesn't toy_NEG with_NEG our_NEG emotions_NEG with_NEG sentimental_NEG blandness_NEG he wants us to know that in war no one_NEG is_NEG victorious_NEG the sweet hereafter starring ian holm sarah polley bruce greenwood tom mccamus gabrielle rose arsinee khanjian alberta watson maury chaykin caerthan banks produced by atom egoyan and camelia frieberg script by atom egoyan based on the novel by russell banks directed by atom egoyan running time 110 mins rated r ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ ___ __ atom egoyan's powerfully meditative the sweet hereafter is as anything as haunting and transcendental as i've seen this year it not only_NEG explores_NEG the_NEG aftermaths_NEG of_NEG a_NEG terrible_NEG tragedy_NEG with_NEG magnificent_NEG subtlety_NEG ,_NEG with_NEG cold_NEG ,_NEG and_NEG stunning_NEG shots_NEG of_NEG the_NEG backdrop_NEG of_NEG this_NEG mistfortune_NEG but_NEG also_NEG by_NEG telegraphing_NEG the_NEG sense_NEG of_NEG devastation_NEG that_NEG has_NEG permeated_NEG the_NEG small_NEG town_NEG with_NEG an_NEG enormous_NEG amount_NEG of_NEG dignity_NEG and_NEG respect_NEG for_NEG each_NEG and_NEG every_NEG soul_NEG affected_NEG ian holm as the somber lonely lawyer seeking compensation for the townspeople has a decency and a restraint uncommon with lawyers he's a lion for the wrong reasons the pain for having lost his daughter she's a druggie who frequently calls him begging for money is reflected on the town and in some mesmerizing flashbacks we witness how she lost her purity it's one of the most strikingly breathtaking takes on the loss of innocence i've ever seen but the most astounding part goes to sarah polly the peaceful benevolent girl who in contrast to the other characters remains the most mentally stable during past present or future her ability to convey concealed pain and unconditional love is the opposite yet near-revolutionary role that won emily watson an oscar nomination for breaking the waves both women are torn apart by an extraordinary incident but it is only then that we truly see truly feel their wordless omnipotent love the sweet hereafter though is atom egoyan's movie and as of now he is emerging as the definite face of independent cinema after surging to worldwide recognition with 1994's exotica the sweet hereafter makes him something few people in this industry can call themselves an artist he interweaves time with a delicacy and sees a soul with a purity that is just not common_NEG\n"
     ]
    }
   ],
   "source": [
    "from nltk.sentiment.util import mark_negation\n",
    "\n",
    "def text_filter_neg(a_dict, label, exclude_lists):\n",
    "    data = []\n",
    "    for rev_id in a_dict.keys():\n",
    "        tokens = []\n",
    "        for sent in a_dict.get(rev_id).sents:\n",
    "            neg_tokens = mark_negation(sent.text.split())\n",
    "            #print (neg_tokens)\n",
    "            for token in neg_tokens:\n",
    "                if not token in exclude_lists:\n",
    "                    tokens.append(token)\n",
    "        data.append((' '.join(tokens), label))\n",
    "    return data\n",
    "\n",
    "def prepare_data_neg(pos_docs, neg_docs, exclude_lists):\n",
    "    data = text_filter_neg(pos_docs, 1, exclude_lists)\n",
    "    data += text_filter_neg(neg_docs, -1, exclude_lists)\n",
    "    random.seed(42)\n",
    "    random.shuffle(data)\n",
    "    texts = []\n",
    "    labels = []\n",
    "    for item in data:\n",
    "        texts.append(item[0])\n",
    "        labels.append(item[1])\n",
    "    return texts, labels\n",
    "\n",
    "\n",
    "texts_neg, labels_neg = prepare_data_neg(pos_docs, neg_docs, punctuation_list) \n",
    "\n",
    "print(len(texts_neg), len(labels_neg))\n",
    "print(texts_neg[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.89  0.865 0.875 0.82  0.895 0.8   0.855 0.885 0.82  0.88 ]\n",
      "Accuracy: 0.8585\n",
      "\n",
      "Confusion matrix:\n",
      "[[828 172]\n",
      " [111 889]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "          -1       0.88      0.83      0.85      1000\n",
      "           1       0.84      0.89      0.86      1000\n",
      "\n",
      "    accuracy                           0.86      2000\n",
      "   macro avg       0.86      0.86      0.86      2000\n",
      "weighted avg       0.86      0.86      0.86      2000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "text_clf = Pipeline([('vect', CountVectorizer(ngram_range=(1, 2))), \n",
    "                     #options: ngram_range=(1, 2) or min_df=10, max_df=0.5\n",
    "                     ('binarizer', Binarizer()), #presence-absence of features\n",
    "                     ('clf', MultinomialNB()),\n",
    "                    ])\n",
    "\n",
    "scores = cross_val_score(text_clf, texts_neg, labels_neg, cv=10)\n",
    "print(scores)\n",
    "print(\"Accuracy: \" + str(sum(scores)/10))\n",
    "predicted = cross_val_predict(text_clf, texts_neg, labels_neg, cv=10)\n",
    "print(\"\\nConfusion matrix:\")\n",
    "print(metrics.confusion_matrix(labels_neg, predicted))\n",
    "print(metrics.classification_report(labels_neg, predicted))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAZoUlEQVR4nO3de5hcVZ3u8e9LYoQEMAk0nIgiiBmQQYnQMlwEg1EURgUcFHgcJihDZEZEcBzFM4rRUQ8oyNHxEQgXk3FiDnKJRHSAyHAZVJBOCBAIDgoYAiE0CsjFC4Tf+WOtJjuV6q7dnap0r/B+nqefqr1r7arfqqp+e9eqvXYrIjAzs/JsMtwFmJnZ0DjAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwKVSvAJX1c0lJJd0k6Oa+bKGmhpHvz5YTOlmpmZlUtA1zSbsDxwF7A7sC7JU0GTgWujYjJwLV52czMNpA6e+CvB26OiGcj4nngBuBw4FBgTm4zBzisMyWamVkzdQJ8KXCApK0kjQUOAV4NbBsRKwHy5TadK9PMzBqNbtUgIpZJOgNYCDwN3A48X/cBJM0AZgCMGzduz1122WWIpZqZvTQtWrTosYjoalyvwZ4LRdJXgBXAx4GpEbFS0iTg+ojYeaBtu7u7o6enZ1CPZ2b2UidpUUR0N66vexTKNvlye+B9wDxgATA9N5kOXNGeUs3MrI6WQyjZZZK2Ap4DPhoRj0s6Hfi+pOOA5cD7O1WkmZmtq1aAR8T+Tdb9FpjW9orMzKwWz8Q0MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MytU3X+pZmYbgS9O+tJwl1DLaSs/W7vtu7e9sIOVtM+Vq45r+33W/afGp0i6S9JSSfMkbSpptqT7JS3JP1PaXp2ZmfWr5R64pO2Ak4BdI+IPkr4PHJVv/ueIuLSTBZqZWXN1x8BHA5tJGg2MBR7uXElmZlZHywCPiIeAM4HlwErgyYi4Jt/8ZUl3SDpb0ss7WKeZmTWoM4QyATgU2BF4ArhE0t8CnwEeAcYAs4BPA19ssv0MYAbA9ttv37bCzTaE67qHu4J6DuwZ7gpsONQZQnk7cH9E9EbEc8DlwL4RsTKSPwHfAfZqtnFEzIqI7ojo7urqal/lZmYvcXUCfDmwt6SxkgRMA5ZJmgSQ1x0GLO1cmWZm1qjlEEpE3CLpUmAx8DxwG2nI5D8ldQEClgAndLJQMzNbW62JPBHxeeDzDavf1v5yzMysLk+lNzMrlAPczKxQDnAzs0L5ZFbWNr8a9bfDXUItr1v9H8NdgllbeA/czKxQDnAzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MClUrwCWdIukuSUslzZO0qaQdJd0i6V5JF0sa0+lizcxsjZYBLmk74CSgOyJ2A0YBRwFnAGdHxGTgceC4ThZqZmZrqzuEMhrYTNJoYCywkvRf6S/Nt88BDmt/eWZm1p+WAR4RDwFnAstJwf0ksAh4IiKez81WANs1217SDEk9knp6e3vbU7WZmdUaQpkAHArsCLwSGAcc3KRpNNs+ImZFRHdEdHd1da1PrWZmVlFnCOXtwP0R0RsRzwGXA/sC4/OQCsCrgIc7VKOZmTVRJ8CXA3tLGitJwDTgbuA64IjcZjpwRWdKNDOzZuqMgd9C+rJyMXBn3mYW8GngE5J+BWwFXNjBOs3MrMHo1k0gIj4PfL5h9X3AXm2vyMzMavFMTDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK5QA3MyuUA9zMrFAOcDOzQjnAzcwK1fJfqknaGbi4suq1wGnAeOB4oDev/98R8eO2V2hmZk21DPCI+CUwBUDSKOAhYD7wIeDsiDizoxWamVlTgx1CmQb8OiJ+04lizMysvsEG+FHAvMryiZLukHSRpAnNNpA0Q1KPpJ7e3t5mTczMbAhqB7ikMcB7gUvyqnOAnUjDKyuBs5ptFxGzIqI7Irq7urrWs1wzM+szmD3wg4HFEbEKICJWRcTqiHgBOB/YqxMFmplZc4MJ8KOpDJ9ImlS57XBgabuKMjOz1loehQIgaSzwDuAjldVflTQFCOCBhtvMzKzDagV4RDwLbNWw7piOVGRmZrV4JqaZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhWoZ4JJ2lrSk8vN7SSdLmihpoaR78+WEDVGwmZklLQM8In4ZEVMiYgqwJ/AsMB84Fbg2IiYD1+ZlMzPbQAY7hDIN+HVE/AY4FJiT188BDmtnYWZmNrDBBvhRwLx8fduIWAmQL7dptoGkGZJ6JPX09vYOvVIzM1tL7QCXNAZ4L3DJYB4gImZFRHdEdHd1dQ22PjMz68dg9sAPBhZHxKq8vErSJIB8+Wi7izMzs/4NJsCPZs3wCcACYHq+Ph24ol1FmZlZa7UCXNJY4B3A5ZXVpwPvkHRvvu309pdnZmb9GV2nUUQ8C2zVsO63pKNSzMxsGHgmpplZoWrtgVtnfHjuH4a7hFou+uBmw12CmTXhPXAzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MCuUANzMrlAPczKxQdf8n5nhJl0q6R9IySftIminpIUlL8s8hnS7WzMzWqPsfeb4BXBURR0gaA4wF3gmcHRFndqw6MzPrV8sAl7QlcABwLEBE/Bn4s6TOVmZmZgOqM4TyWqAX+I6k2yRdIGlcvu1ESXdIukjShM6VaWZmjeoE+GhgD+CciHgT8AxwKnAOsBMwBVgJnNVsY0kzJPVI6unt7W1P1WZmVmsMfAWwIiJuycuXAqdGxKq+BpLOB65stnFEzAJmAXR3d8dQCx134NyhbrpBPXPdB4e7BDN7iWi5Bx4RjwAPSto5r5oG3C1pUqXZ4cDSDtRnZmb9qHsUyseAufkIlPuADwHflDQFCOAB4CMdqdDMzJqqFeARsQToblh9TPvLMTOzujwT08ysUA5wM7NCOcDNzArlADczK5QD3MysUA5wM7NCOcDNzArlADczK5QD3MysUA5wM7NCOcDNzArlADczK5QD3MysUA5wM7NCOcDNzArlADczK5QD3MysUA5wM7NC1QpwSeMlXSrpHknLJO0jaaKkhZLuzZcTOl2smZmtUXcP/BvAVRGxC7A7sAw4Fbg2IiYD1+ZlMzPbQFoGuKQtgQOACwEi4s8R8QRwKDAnN5sDHNapIs3MbF119sBfC/QC35F0m6QLJI0Dto2IlQD5cpsO1mlmZg3qBPhoYA/gnIh4E/AMgxgukTRDUo+knt7e3iGWaWZmjeoE+ApgRUTckpcvJQX6KkmTAPLlo802johZEdEdEd1dXV3tqNnMzKgR4BHxCPCgpJ3zqmnA3cACYHpeNx24oiMVmplZU6NrtvsYMFfSGOA+4EOk8P++pOOA5cD7O1OimZk1UyvAI2IJ0N3kpmntLcfMzOryTEwzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MCuUANzMrlAPczKxQDnAzs0I5wM3MClUrwCU9IOlOSUsk9eR1MyU9lNctkXRIZ0s1M7Oquv/UGODAiHisYd3ZEXFmOwsyM7N6PIRiZlaougEewDWSFkmaUVl/oqQ7JF0kaUIH6jMzs37UDfD9ImIP4GDgo5IOAM4BdgKmACuBs5ptKGmGpB5JPb29ve2o2czMqBngEfFwvnwUmA/sFRGrImJ1RLwAnA/s1c+2syKiOyK6u7q62lW3mdlLXssAlzRO0hZ914GDgKWSJlWaHQ4s7UyJZmbWTJ2jULYF5kvqa/+9iLhK0nclTSGNjz8AfKRjVZqZ2TpaBnhE3Afs3mT9MR2pyMzMavFhhGZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlYoB7iZWaEc4GZmhXKAm5kVygFuZlaoOv/UGEkPAE8Bq4HnI6Jb0kTgYmAH0j81/kBEPN6ZMs3MrNFg9sAPjIgpEdGdl08Fro2IycC1ednMzDaQ9RlCORSYk6/PAQ5b/3LMzKwuRUTrRtL9wONAAOdFxCxJT0TE+EqbxyNiQpNtZwAz8uLOwC/bUnl7bA08NtxFtNnG1qeNrT+w8fVpY+sPjLw+vSYiuhpX1g3wV0bEw5K2ARYCHwMW1AnwkUxST2VIaKOwsfVpY+sPbHx92tj6A+X0qdYQSkQ8nC8fBeYDewGrJE0CyJePdqpIMzNbV8sAlzRO0hZ914GDgKXAAmB6bjYduKJTRZqZ2brqHEa4LTBfUl/770XEVZJuBb4v6ThgOfD+zpXZMbOGu4AO2Nj6tLH1Bza+Pm1s/YFC+lRrDNzMzEYez8Q0MyuUA9zMrFAjOsAl/YukuyTdIWmJpL8awn1MkXRIZfm9kjo6a1TSVEn7tuF+iui/pB0kLW1YN1PSJwfYplvSN9tZRztIukjSo439qdz+aknXSVqWX5uPb+gaB6tGnzaV9AtJt+c+fWFD11gKScdKemVl+QJJuw5XPSM2wCXtA7wb2CMi3gi8HXhwCHc1BXgxwCJiQUSc3p4q+zUVWK8AL7z/LUVET0ScVLe9kg3xfp0NvGuA258H/ikiXg/sDXx0fX6BJdU6H9F6ms3AffoT8LaI2J30fnmXpL2H+mCSRg1120E8xlRJszv9OE0cC7wY4BHx9xFx9zDUAYzgAAcmAY9FxJ8AIuKxPJloT0k3SFok6erKsejXSzoj70n8j6T9JY0Bvggcmfdgj8x/Qb+Vt5kt6Zy8R3WfpLfmvZVl1TeHpIMk/VzSYkmXSNo8r39A0hfy+jsl7SJpB+AE4JT8mPuP1P5vCM3qyuunSroyX++StDA/j+dJ+o2krfOe/TJJ3wYWA6/Or1dP455ifi2+kl+nHkl75Ofn15JOyG0mSboxPxdLm702EXEj8Lv++hMRKyNicb7+FLAM2K6fvn9O0j25b/P6PpHk5+Qrkm4APi7pPZJukXSbpJ9I2ja3mylpjqRrcv/eJ+mr+b12laSX5XanS7pb6ZPamUPoU0TE03nxZflnnaMbJG0i6dv5ub9S0o8lHVF5/k+TdBPwfknHS7pVaa/+Mkljc7uWv3OSRuV2S3NfT+mv9ob6+nuvjZL0tVzPHZI+UqM/p+X2SyXNUnIE0A3Mze+hzfJjdudtjs71LpV0RqWupyV9OT8XN/e9vm0RESPyB9gcWAL8D/Bt4K2kN9bPgK7c5kjgonz9euCsfP0Q4Cf5+rHAtyr3++Iyac/k/wEindvl98AbSH/YFpH2RrYGbgTG5W0+DZyWrz8AfCxf/0fggnx9JvDJkd7/Nr5WOwBLG9bNBD45QF1TgSvz9W8Bn8nX30UKj63z/b4A7F2534n5clS+7zdWXot/yNfPBu4AtgC6gEfz+n8C/qWy/RZ1+zNAu+XAlk1u686v32a5jnv73hO57m9X2k5gzRFhf195vmYCN+XXfXfgWeDgfNt80vmHJpJOT9G3/fih9Ck/H0uAp4Ez+mlzBPBj0u/H/yKdXuOIyvP/qUrbrSrXv8Sa35PZtP6d2xNYWNl+nT7l98/shnXX0/y9NgP4bL7+cqAH2LFFfyZW7ve7wHsqj9Hd8JjdpL3y5aT322jgv4DDcpuobP/Vvlra8bMhPr4NSUQ8LWlPYH/gQNKpa78E7AYsVDoufRSwsrLZ5flyEekNW8cPIyIk3Qmsiog7ASTdle/jVcCuwE/zY44Bft7PY76vfg8HtgH73w79HYvat75VXW8BDgeINMegelri30TEzZXlDyidX2c06VPKrqSwhjS5DOBOYPNIe8hPSfqjpPHArcBFec/1BxGxpG4HGyl9CrsMODkift9Pn66IiD/k9j9suP3iyvVXARcrfZoaA9xfue0/I+K5/P4cBVxV6eMOwJXAH4ELJP0oLw9aRKwGpuTnab6k3SKiccz8LcAlEfEC8Iik6wbo026SvgSMJ+2MXF25rdXv3A3AayX9G/Aj4Jq+DSXdQgrhzYGJkvpew0/ny2bvtYOAN/btXQOvACa36M+Bkj4FjCX9kbwLaHwNq94MXB8RvbnOucABwA+AP7PmdVkEvGOA+xmUERvg8OKb6nrg+vxifxS4KyL26WeTP+XL1dTvW982L1Su9y2Pzve1MCKObuNj1tLJ/iuNUy7Kiwsi4rT1KPW3pL3IqomsCaJWdWmA+37mxUbSjqS9+jdHxOP5I/emlbYDvpYRcaOkA4C/Br4r6WsR8e8DPDaSXs2aX9xzI+Lc/AfgMmBuRFzerF2LPq3VL+DfgK9HxAJJU0l73mv1KSJekPRc5N24Sp+el7QXMA04CjgReNtg+9R3W0Q8Iel60jj4OOC8fNNpg+zTbNIe6O2SjiXtMa/VJ/p/nR6XtDvwTtJ7/gPAh3N9f5X7MBU4NiKOrfTrMzR/r4n0CaD6RwRJf92sE5I2JX3q7Y6IByXNZO33WdPNBrit+rq1NSdG7Bi4pJ0lTa6smkIab+xS+oIPSS+T9Jct7uop0kfYoboZ2E/S6/JjjpX0Fx1+zI73PyJWRzq/+5T1DG8ijZ+ulDQt1zWRNBRyU827uIn0S4qkg1j3j0GfLUkh8WQeRzx4MHVKeg1pOOV84EJgj1bbRMSDlefpXKWPPhcCyyLi6/21y316j9IRHpuT/mj05xXAQ/n69AHaNevT5sArIuLHwMmk98lg+9SV97yRtBnpC/N7IuKWSrsFuU9/k8eOt2XtUG60Bek98TLgg4Ps09bAJhFxGfA5arxOLVwN/IPWfGfwF/mPU3/96Qvrx/Lze0Tlvvr73b4FeKvSdzejgKNJnyQ6asQGOOkj0hzlL2dIH5VPIz2ZZ0i6nTRm1+poj+uAXfOXDkcOtoj8kehYYF6u42Zglxab/RA4XOv3JeaI6P8g/B3w2fyR9r+AL0TEr2tu+wXgIEmLSaG8kvSLspaIuB24jfRx9iLgp4OscSqwRNJtwN8A32hsIGkeaYhsZ0krlE4VUbUfcAzwtvycLlHlMM1KrbeShnRuJ32s7wGe7KeumcAlkv6bwZ/CdAvgyvweuQFY5wu/Gn2aBFyX7+NW0ifOZkMxlwErSOdCOo8UWv316XP59oXAPYPs03akT51LSHvynxnk9o0uAO4GFisdSnkeaS+4aX8i4gngfNIw1Q9Iz0mf2cC5+XXfrG9lRKzMdV5Hes0XR0THzw/lqfQ27CS9HFidhwP2Ac6JiJZ7kiOdpM3zdxljSV+Ez4h8BEupKn3aCvgFsF9EPDLcdQ1V6f0Z0WPg9pKxPenEaJuQvvA5fpjraZdZSseIbwrMKT28syvzcMsY4F9LCrt+FN0f74GbmRVqJI+Br0XS061btbyPEyT9XTvqWV+SzpZ0cmX5akkXVJbPkvSJId73i9PYlSZEPJSHKchfsjzQYvvxkv5xKI9tZhtOMQHeDhFxbqvDxjagn5G/gMxDB1sD1SNK9qXGl3SqN215NfkwrJrGkyYmmdkIVnSA58OfLlOa8nqrpP3y+m9KOi1ff6fS9OlNGvZMX6c0bfl2pSncOyn5mtZM4T0yt52qNGX2UqWp0XPz4WQtpzEP4KesOYLkL0nfhD8laULeW3496YiJ/uq5TtL3SN+U95346peSfkL659FV/5c0tX+d7zwk/bPWTDHum5p+OrBT/qb9a4Pok5ltQKV/ifkN4OyIuEnS9qTjPV8PnArcmg/L+iZwSJ4IUd12LnB6RMxXOnB/E9JMyimkactb5/u4Mbd/EyloHyaF736S7ibNINwlzywbT02RzmvyfK57X9JhXtsB+5AOzbqDdDKr/urZC9gtIu5XmrF5VK5xNOm8IYsqD7ecdMzrMVRmkykdcz0535eABUoTXU7N9138kSBmG7PSA/ztpGOc+5a3lLRFRDwl6XjSoVunNB6PrPQ/PreLiPkAEfHHvP4twLw8A3KV0smG3kw6X8MvImJFbreENE33ZtZvGnPfXvi+wNdJAb4vKcB/RprqO1A9fTMd9wfmR8Szub4FrOsrpOOSf1RZd1D+uS0vb04K9OWD7IeZDYPSA3wTYJ++8000eANpivcrm9zW37TXgabDVqf8rmaI05gb9I2Dv4E0hPIg6YRLvydNVJk2wLbPNCwPeDhRRPwq/+H5QGW1gP8TEedV2yqdUdHMRriix8BJJ7k5sW9B0pR8+RpSEL4JOFgN/wghn3xohaTDcvuXVyZbHKl0+sku0sloftHfg2sI05gb/JQ0TPK7PLX9d6QvEPchDanUredG0szPzfKni/f083hfJp1LpM/VwIe15vS420nahjacCsDMOq+kAB+rNA247+cTwElAd/4C7m7ghPzl4oWkU3c+DBxHGuJoPBnNMcBJStOHf0Y6neR80tjz7aTp4J9qcWB/y2nMLdxJGtu+uWHdkxHxWN168gSRi0lT6y8D/rvZg0XEXaTx8b7la4DvAT9XOlnWpaRTrP6WdPbFpf4S02zk8kQeM7NClbQHbmZmFQ5wM7NCOcDNzArlADczK5QD3MysUA5wM7NCOcDNzArlADczK9T/B/Qb+bUrQRYvAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<function matplotlib.pyplot.legend(*args, **kwargs)>"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "a = [\"Sentiment\\nLexicons\", \"Senti-\\nWordNet\", \"Unigrams\", \"1-2-grams\", \"1-3-grams\", \"+negation\"]\n",
    "index = range(len(a))\n",
    "b = [64.75, 68.80, 83.60, 85.05, 85.90, 85.85] # Accuracy scores for the compared methods\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "axes = plt.gca()\n",
    "# Let's set 50 as the lower bound as the majority class baseline is at 50 for this dataset\n",
    "axes.set_ylim([50,90])\n",
    "\n",
    "ax.bar(index, b, color=['#0A40A4', '#61A4F6', '#DB025B', '#BE33FF', '#811883', '#4B169C'])\n",
    "plt.xticks(index, a) \n",
    "\n",
    "plt.show()\n",
    "\n",
    "import matplotlib\n",
    "matplotlib.axes.Axes.plot\n",
    "matplotlib.pyplot.plot\n",
    "matplotlib.axes.Axes.legend\n",
    "matplotlib.pyplot.legend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}