IWI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Informal Word Identification (IWI)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The IWI component identifies each word as informal, or not. The system only attempts to paraphrase only the informal words in the rest of the pipeline.\n",
    "\n",
    "We derive our dataset from a lexical substitution dataset called Concepts in Context (CoInCo) (Kremer et al., 2014). The CoInCo dataset is a All-Words lexical substitution dataset, where all words that could be substituted are manually annotated. The corpus is sampled from newswire and fiction genres of the Manually Annotated Sub-Corpus (MASC) corpus. While the targets (words that are going to be substituted) are used to build the informal word identification dataset, the candidates are further processed to perform the academic paraphrase ranking task. A total of 1,608 train and 866 test sentences are compiled out of 2,474 sentences from the CoInCo dataset.\n",
    "\n",
    "We automatically generated an IWI dataset from CoInCo dataset as follows. For each non-academic target word, we determine if its substitution candidates include at least one academic word. If so, it is labelled as formal. All academic target words and all words without substitution are labelled as formal. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random, pickle, re\n",
    "from collections import Counter\n",
    "import xml.etree.ElementTree as ET\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from gensim.test.utils import datapath, get_tmpfile\n",
    "from gensim.models import KeyedVectors\n",
    "from gensim.scripts.glove2word2vec import glove2word2vec\n",
    "\n",
    "import spacy\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn import metrics\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.ensemble import GradientBoostingClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42) # to replicate the results\n",
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "specify the paths to the following resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CoInCo = '<path-to-coinco.xml>'\n",
    "COCA_ALL = 'path-to-(COCA)allWords.xlsx>'\n",
    "COMPILED_LIST = '<path-to-academic_keyphrases.xlsx>'\n",
    "COCA_LIST = '<path-to-(COCA)acadCore.xlsx>'\n",
    "NAWL = '<path-to-NAWL_Headwords.txt>'\n",
    "ACL_FREQ = '<path-to-academic_unigrams.pkl>' # obained while compiling the resources\n",
    "BEAUTIFUL_DATA = '<path-to-(beautiful_data)count_1w.txt>'\n",
    "GLOVE_PATH = 'path-to-glove.840B.300d.txt'\n",
    "\n",
    "PRECONTEXT = 0\n",
    "TARGETSENTENCE = 1\n",
    "POSTCONTEXT = 2\n",
    "TOKENS = 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "parse the XML CoInCo dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tree = ET.parse(CoInCo)\n",
    "root = tree.getroot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentences = list()\n",
    "for child in root:\n",
    "    sentences.append(child[TARGETSENTENCE].text.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "perform train test split over the list of available sentences in the CoInCo dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random.Random(9).shuffle(sentences)\n",
    "train_sentences = sentences[ : int(0.65 * len(sentences))]\n",
    "test_sentences = sentences[int(0.65 * len(sentences)) : ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_sentences = train_sentences[ : int(0.8 * len(train_sentences))]\n",
    "v_sentences = train_sentences[int(0.8 * len(train_sentences)) : ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "split the train and test sentences from the CoInCo dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t_d = dict()\n",
    "v_d = dict()\n",
    "test_d = dict()\n",
    "\n",
    "for child in root:\n",
    "    for token in child[TOKENS]:\n",
    "        token_id = token.get('id')\n",
    "        t = dict()\n",
    "        t['precontenxt'] = child[PRECONTEXT].text.strip()\n",
    "        t['postcontext'] = child[POSTCONTEXT].text.strip()\n",
    "        t['wordform'] = token.get('wordform')\n",
    "        t['lemma'] = token.get('lemma')\n",
    "        t['posMASC'] = token.get('posMASC')\n",
    "        t['posTT'] = token.get('posTT')\n",
    "        t['problematic'] = token.get('problematic')\n",
    "        l = list()\n",
    "        for substitutions in token:\n",
    "            for subst in substitutions:\n",
    "                s = (subst.get('lemma'), subst.get('pos'), subst.get('freq'))\n",
    "                l.append(s)\n",
    "        t['substitutions'] = l\n",
    "        \n",
    "        if(token_id != 'XXX' and (child[TARGETSENTENCE].text.strip() in t_sentences)):\n",
    "            t['targetsentence'] = child[TARGETSENTENCE].text.strip()\n",
    "            t_d[token_id] = t\n",
    "        elif(token_id != 'XXX' and (child[TARGETSENTENCE].text.strip() in v_sentences)):\n",
    "            t['targetsentence'] = child[TARGETSENTENCE].text.strip()\n",
    "            v_d[token_id] = t\n",
    "        elif(token_id != 'XXX' and (child[TARGETSENTENCE].text.strip() in test_sentences)):\n",
    "            t['targetsentence'] = child[TARGETSENTENCE].text.strip()\n",
    "            test_d[token_id] = t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_d = dict()\n",
    "for t in t_d:\n",
    "    train_d[t] = t_d[t]\n",
    "for t in v_d:\n",
    "    train_d[t] = v_d[t]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(t_d), len(v_d), len(train_d), len(test_d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "load the compiled list of academic phrases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "academic_df = pd.read_excel(COMPILED_LIST, sheet_name='<sheet-name>')\n",
    "academic_list = academic_df.phrase.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "load COCA academic list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "coca_df = pd.read_excel(COCA_LIST, sheet_name='list')\n",
    "coca_list = coca_df.word.tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "load NAWL list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(NAWL, 'r') as f:\n",
    "    s = f.read()\n",
    "    nawl_list = s.split()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "load GloVe vectors file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glove_file = datapath(GLOVE_PATH)\n",
    "tmp_file = get_tmpfile(\"test_word2vec.txt\")\n",
    "_ = glove2word2vec(glove_file, tmp_file)\n",
    "model = KeyedVectors.load_word2vec_format(tmp_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(ACL_FREQ, 'rb') as f:\n",
    "    acl_freq = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "load Beatiful Data corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "beatiful_data_freq = Counter()\n",
    "with open(BEAUTIFUL_DATA, 'r') as f:\n",
    "    tmp = f.read().strip().split('\\n')\n",
    "    for c in tmp:\n",
    "        word, freq = c.strip().split('\\t')\n",
    "        beatiful_data_freq[word] = freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "load COCA all words list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "allwords_df = pd.read_excel(COCA_ALL, sheet_name='list')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "identify the formal and informal words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "informal_t_d = dict()\n",
    "non_informal_t_d = dict()\n",
    "for token_id in t_d:\n",
    "    lemma = t_d[token_id]['lemma']\n",
    "    if(lemma not in coca_list or lemma not in nawl_list or lemma not in academic_list):\n",
    "        c = 0\n",
    "        for subst in t_d[token_id]['substitutions']:\n",
    "            s = subst[0]\n",
    "            if(s in coca_list or s in nawl_list):\n",
    "                informal_t_d[token_id] = t_d[token_id]\n",
    "                c = 1\n",
    "                break\n",
    "        if(c == 0):\n",
    "            non_informal_t_d[token_id] = t_d[token_id]\n",
    "    else:\n",
    "        non_informal_t_d[token_id] = t_d[token_id]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "identify the formal and informal words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "informal_v_d = dict()\n",
    "non_informal_v_d = dict()\n",
    "for token_id in v_d:\n",
    "    lemma = v_d[token_id]['lemma']\n",
    "    if(lemma not in coca_list or lemma not in nawl_list or lemma not in academic_list):\n",
    "        c = 0\n",
    "        for subst in v_d[token_id]['substitutions']:\n",
    "            s = subst[0]\n",
    "            if(s in coca_list or s in nawl_list):\n",
    "                informal_v_d[token_id] = v_d[token_id]\n",
    "                c = 1\n",
    "                break\n",
    "        if(c == 0):\n",
    "            non_informal_v_d[token_id] = v_d[token_id]\n",
    "    else:\n",
    "        non_informal_v_d[token_id] = v_d[token_id]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "identify the formal and informal words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "informal_train_d = dict()\n",
    "non_informal_train_d = dict()\n",
    "for token_id in informal_t_d:\n",
    "    informal_train_d[token_id] = informal_t_d[token_id]\n",
    "for token_id in non_informal_t_d:\n",
    "    non_informal_train_d[token_id] = non_informal_t_d[token_id]\n",
    "for token_id in informal_v_d:\n",
    "    informal_train_d[token_id] = informal_v_d[token_id]\n",
    "for token_id in non_informal_v_d:\n",
    "    non_informal_train_d[token_id] = non_informal_v_d[token_id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(informal_t_d), len(non_informal_t_d))\n",
    "print(len(informal_v_d), len(non_informal_v_d))\n",
    "print(len(informal_train_d), len(non_informal_train_d))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "UNK_embed = np.random.rand(300,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "obtain the word embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_word_embedding(word):\n",
    "    try:\n",
    "        return model.wv[word]\n",
    "    except:\n",
    "        return UNK_embed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "compute the sentence embedding - average of all the word embeddings of the words present in the sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_sentence_embedding(sentence):\n",
    "    # remove special characters\n",
    "    sentence = ' '.join(re.findall(r\"[a-zA-Z0-9]+\", sentence))\n",
    "\n",
    "    words_embed = list()\n",
    "    word_list = sentence.split()\n",
    "    for word in word_list:\n",
    "        words_embed.append(get_word_embedding(word))\n",
    "    \n",
    "    return np.mean(words_embed, axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_ner_tag(lemma, sentence):\n",
    "    doc = nlp(sentence)\n",
    "    for ent in doc.ents:\n",
    "        if(ent.text == lemma):\n",
    "            return ent.label_\n",
    "    return 'UNK'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "compute the features from the resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = list()\n",
    "for token_id in informal_t_d:\n",
    "    t = list()\n",
    "    lemma = informal_t_d[token_id]['lemma']\n",
    "    # freq_beautiful_data\n",
    "    try:\n",
    "        t.append(beatiful_data_freq[lemma])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_coca_general\n",
    "    try:\n",
    "        t.append(allwords_df.loc[allwords_df['word'] == lemma].iloc[0]['COCA-All'])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_acl\n",
    "    try:\n",
    "        t.append(acl_freq[tuple(lemma.split())])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # cos_target\n",
    "    sentence_embed = get_sentence_embedding(informal_t_d[token_id]['targetsentence'])\n",
    "    word_embed = get_word_embedding(lemma)\n",
    "    t.append(np.dot(sentence_embed, word_embed)/(np.sqrt(np.dot(sentence_embed, sentence_embed))*np.sqrt(np.dot(word_embed, word_embed))))\n",
    "    \n",
    "    # euclidean_distance\n",
    "    t.append(np.linalg.norm(sentence_embed-word_embed))\n",
    "    \n",
    "    # posMASC_tag\n",
    "    t.append(informal_t_d[token_id]['posMASC'])\n",
    "    \n",
    "    # is_problematic\n",
    "    t.append(1 if informal_t_d[token_id]['problematic']=='yes' else 0)\n",
    "    \n",
    "    # word_length\n",
    "    t.append(len(lemma))\n",
    "    \n",
    "    # count_vowel\n",
    "    t.append(sum(list(map(lemma.lower().count, 'aeiou'))))\n",
    "    \n",
    "    # y\n",
    "    t.append(1)\n",
    "    \n",
    "    features.append(t)\n",
    "    \n",
    "for token_id in non_informal_t_d:\n",
    "    t = list()\n",
    "    lemma = non_informal_t_d[token_id]['lemma']\n",
    "    # freq_beautiful_data\n",
    "    try:\n",
    "        t.append(beatiful_data_freq[lemma])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_coca_general\n",
    "    try:\n",
    "        t.append(allwords_df.loc[allwords_df['word'] == lemma].iloc[0]['COCA-All'])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_acl\n",
    "    try:\n",
    "        t.append(acl_freq[tuple(lemma.split())])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # cos_target\n",
    "    sentence_embed = get_sentence_embedding(non_informal_t_d[token_id]['targetsentence'])\n",
    "    word_embed = get_word_embedding(lemma)\n",
    "    t.append(np.dot(sentence_embed, word_embed)/(np.sqrt(np.dot(sentence_embed, sentence_embed))*np.sqrt(np.dot(word_embed, word_embed))))\n",
    "    \n",
    "    # euclidean_distance\n",
    "    t.append(np.linalg.norm(sentence_embed-word_embed))\n",
    "    \n",
    "    # posMASC\n",
    "    t.append(non_informal_t_d[token_id]['posMASC'])\n",
    "    \n",
    "    # is_problematic\n",
    "    t.append(1 if non_informal_t_d[token_id]['problematic']=='yes' else 0)\n",
    "    \n",
    "    # word_length\n",
    "    t.append(len(lemma))\n",
    "    \n",
    "    # count_vowel\n",
    "    t.append(sum(list(map(lemma.lower().count, 'aeiou'))))\n",
    "    \n",
    "    # y\n",
    "    t.append(0)\n",
    "    \n",
    "    features.append(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# specify the feature columns and the target variable\n",
    "train_features_cols = ['freq_beautiful', 'freq_coca_general', 'freq_acl', 'cos_target', 'euclidean_distance', 'posMASC', 'is_problematic', 'word_length', 'count_vowel']\n",
    "y_feature = ['y']\n",
    "df_t = pd.DataFrame(features, columns=train_features_cols+y_feature)\n",
    "\n",
    "# shuffle the dataframe\n",
    "df_t = df_t.sample(frac=1).reset_index(drop=True)\n",
    "\n",
    "label_encoder = LabelEncoder()\n",
    "# label encode the POS tag\n",
    "df_t['posMASC_le'] = label_encoder.fit_transform(df_t.posMASC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_t.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "compute the features from the resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = list()\n",
    "for token_id in informal_v_d:\n",
    "    t = list()\n",
    "    lemma = informal_v_d[token_id]['lemma']\n",
    "    # freq_beautiful_data\n",
    "    try:\n",
    "        t.append(beatiful_data_freq[lemma])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_coca_general\n",
    "    try:\n",
    "        t.append(allwords_df.loc[allwords_df['word'] == lemma].iloc[0]['COCA-All'])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_acl\n",
    "    try:\n",
    "        t.append(acl_freq[tuple(lemma.split())])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # cos_target\n",
    "    sentence_embed = get_sentence_embedding(informal_v_d[token_id]['targetsentence'])\n",
    "    word_embed = get_word_embedding(lemma)\n",
    "    t.append(np.dot(sentence_embed, word_embed)/(np.sqrt(np.dot(sentence_embed, sentence_embed))*np.sqrt(np.dot(word_embed, word_embed))))\n",
    "    \n",
    "    # euclidean_distance\n",
    "    t.append(np.linalg.norm(sentence_embed-word_embed))\n",
    "    \n",
    "    # posMASC\n",
    "    t.append(informal_v_d[token_id]['posMASC'])\n",
    "\n",
    "    # is_problematic\n",
    "    t.append(1 if informal_v_d[token_id]['problematic']=='yes' else 0)    \n",
    "    \n",
    "    # word_length\n",
    "    t.append(len(lemma)) \n",
    "\n",
    "    # count_vowel\n",
    "    t.append(sum(list(map(lemma.lower().count, 'aeiou'))))\n",
    "\n",
    "    # y\n",
    "    t.append(1)\n",
    "    \n",
    "    features.append(t)\n",
    "    \n",
    "for token_id in non_informal_v_d:\n",
    "    t = list()\n",
    "    lemma = non_informal_v_d[token_id]['lemma']\n",
    "    # freq_beautiful_data\n",
    "    try:\n",
    "        t.append(beatiful_data_freq[lemma])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_coca_general\n",
    "    try:\n",
    "        t.append(allwords_df.loc[allwords_df['word'] == lemma].iloc[0]['COCA-All'])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_acl\n",
    "    try:\n",
    "        t.append(acl_freq[tuple(lemma.split())])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # cos_target\n",
    "    sentence_embed = get_sentence_embedding(non_informal_v_d[token_id]['targetsentence'])\n",
    "    word_embed = get_word_embedding(lemma)\n",
    "    t.append(np.dot(sentence_embed, word_embed)/(np.sqrt(np.dot(sentence_embed, sentence_embed))*np.sqrt(np.dot(word_embed, word_embed))))\n",
    "    \n",
    "    # euclidean_distance\n",
    "    t.append(np.linalg.norm(sentence_embed-word_embed))\n",
    "    \n",
    "    # posMASC\n",
    "    t.append(non_informal_v_d[token_id]['posMASC'])\n",
    "\n",
    "    # is_problematic\n",
    "    t.append(1 if non_informal_v_d[token_id]['problematic']=='yes' else 0)\n",
    "    \n",
    "    # word_length\n",
    "    t.append(len(lemma))\n",
    "    \n",
    "    # count_vowel\n",
    "    t.append(sum(list(map(lemma.lower().count, 'aeiou'))))\n",
    "    \n",
    "    # y\n",
    "    t.append(0)\n",
    "    \n",
    "    features.append(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_val = pd.DataFrame(features, columns=train_features_cols+y_feature)\n",
    "\n",
    "# shuffle the dataframe\n",
    "df_val = df_val.sample(frac=1).reset_index(drop=True)\n",
    "\n",
    "# label encode the POS tag\n",
    "df_val['posMASC_le'] = label_encoder.fit_transform(df_val.posMASC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_val.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "compute the features from the resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = list()\n",
    "for token_id in informal_train_d:\n",
    "    t = list()\n",
    "    lemma = informal_train_d[token_id]['lemma']\n",
    "    # freq_beautiful_data\n",
    "    try:\n",
    "        t.append(beatiful_data_freq[lemma])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_coca_general\n",
    "    try:\n",
    "        t.append(allwords_df.loc[allwords_df['word'] == lemma].iloc[0]['COCA-All'])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_acl\n",
    "    try:\n",
    "        t.append(acl_freq[tuple(lemma.split())])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # cos_target\n",
    "    sentence_embed = get_sentence_embedding(informal_train_d[token_id]['targetsentence'])\n",
    "    word_embed = get_word_embedding(lemma)\n",
    "    t.append(np.dot(sentence_embed, word_embed)/(np.sqrt(np.dot(sentence_embed, sentence_embed))*np.sqrt(np.dot(word_embed, word_embed))))\n",
    "    \n",
    "    # euclidean_distance\n",
    "    t.append(np.linalg.norm(sentence_embed-word_embed))\n",
    "    \n",
    "    # posMASC_tag\n",
    "    t.append(informal_train_d[token_id]['posMASC'])\n",
    "    \n",
    "    # is_problematic\n",
    "    t.append(1 if informal_train_d[token_id]['problematic']=='yes' else 0)\n",
    "    \n",
    "    # word_length\n",
    "    t.append(len(lemma))\n",
    "    \n",
    "    # count_vowel\n",
    "    t.append(sum(list(map(lemma.lower().count, 'aeiou'))))\n",
    "    \n",
    "    # y\n",
    "    t.append(1)\n",
    "    \n",
    "    features.append(t)\n",
    "    \n",
    "for token_id in non_informal_train_d:\n",
    "    t = list()\n",
    "    lemma = non_informal_train_d[token_id]['lemma']\n",
    "    # freq_beautiful_data\n",
    "    try:\n",
    "        t.append(beatiful_data_freq[lemma])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_coca_general\n",
    "    try:\n",
    "        t.append(allwords_df.loc[allwords_df['word'] == lemma].iloc[0]['COCA-All'])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_acl\n",
    "    try:\n",
    "        t.append(acl_freq[tuple(lemma.split())])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # cos_target\n",
    "    sentence_embed = get_sentence_embedding(non_informal_train_d[token_id]['targetsentence'])\n",
    "    word_embed = get_word_embedding(lemma)\n",
    "    t.append(np.dot(sentence_embed, word_embed)/(np.sqrt(np.dot(sentence_embed, sentence_embed))*np.sqrt(np.dot(word_embed, word_embed))))\n",
    "    \n",
    "    # euclidean_distance\n",
    "    t.append(np.linalg.norm(sentence_embed-word_embed))\n",
    "    \n",
    "    # posMASC\n",
    "    t.append(non_informal_train_d[token_id]['posMASC'])\n",
    "    \n",
    "    # is_problematic\n",
    "    t.append(1 if non_informal_train_d[token_id]['problematic']=='yes' else 0)\n",
    "    \n",
    "    # word_length\n",
    "    t.append(len(lemma))\n",
    "    \n",
    "    # count_vowel\n",
    "    t.append(sum(list(map(lemma.lower().count, 'aeiou'))))\n",
    "    \n",
    "    # y\n",
    "    t.append(0)\n",
    "    \n",
    "    features.append(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = pd.DataFrame(features, columns=train_features_cols+y_feature)\n",
    "\n",
    "# shuffle the dataframe\n",
    "df_train = df_train.sample(frac=1).reset_index(drop=True)\n",
    "\n",
    "# label encode the POS tag\n",
    "df_train['posMASC_le'] = label_encoder.fit_transform(df_train.posMASC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# update the train features to perform the various experiments\n",
    "train_features = ['freq_beautiful', 'freq_coca_general', 'freq_acl', 'cos_target', 'euclidean_distance', 'posMASC_le', 'is_problematic', 'word_length', 'count_vowel']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Classifers (provided by scikit learn) to perform Informal Word Identification (IWI)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = LogisticRegression()\n",
    "clf.fit(df_train[train_features], df_train[y_feature])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = GaussianNB()\n",
    "clf.fit(df_train[train_features], df_train[y_feature])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = SVC(kernel='rbf')\n",
    "clf.fit(df_train[train_features], df_train[y_feature])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = DecisionTreeClassifier()\n",
    "clf.fit(df_train[train_features], df_train[y_feature])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = RandomForestClassifier(n_estimators=10)\n",
    "clf.fit(df_train[train_features], df_train[y_feature])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = GradientBoostingClassifier(n_estimators=600)\n",
    "clf.fit(df_train[train_features], df_train[y_feature])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = clf.predict(df_val[train_features])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Accuracy : ', metrics.accuracy_score(df_val[y_feature], y_pred))\n",
    "print('Precision : ', metrics.precision_score(df_val[y_feature], y_pred))\n",
    "print('Recall : ', metrics.recall_score(df_val[y_feature], y_pred))\n",
    "print('F-Measure : ', metrics.f1_score(df_val[y_feature], y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "precision, recall, thresholds = metrics.precision_recall_curve(df_val[y_feature], y_pred)\n",
    "f1 = metrics.f1_score(df_val[y_feature], y_pred)\n",
    "auc = metrics.auc(recall, precision)\n",
    "ap = metrics.average_precision_score(df_val[y_feature], y_pred)\n",
    "\n",
    "plt.plot([0, 1], [0.5, 0.5], linestyle='--')\n",
    "plt.plot(recall, precision, marker='.')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "identify the formal and informal words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "informal_test_d = dict()\n",
    "non_informal_test_d = dict()\n",
    "for token_id in test_d:\n",
    "    lemma = test_d[token_id]['lemma']\n",
    "    if(lemma not in coca_list or lemma not in nawl_list or lemma not in academic_list):\n",
    "        c = 0\n",
    "        for subst in test_d[token_id]['substitutions']:\n",
    "            s = subst[0]\n",
    "            if(s in coca_list or s in nawl_list):\n",
    "                informal_test_d[token_id] = test_d[token_id]\n",
    "                c = 1\n",
    "                break\n",
    "        if(c == 0):\n",
    "            non_informal_test_d[token_id] = test_d[token_id]\n",
    "    else:\n",
    "        non_informal_test_d[token_id] = test_d[token_id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(informal_test_d), len(non_informal_test_d))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "compute the features from the resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = list()\n",
    "for token_id in informal_test_d:\n",
    "    t = list()\n",
    "    lemma = informal_test_d[token_id]['lemma']\n",
    "    # freq_beautiful_data\n",
    "    try:\n",
    "        t.append(beatiful_data_freq[lemma])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_coca_general\n",
    "    try:\n",
    "        t.append(allwords_df.loc[allwords_df['word'] == lemma].iloc[0]['COCA-All'])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_acl\n",
    "    try:\n",
    "        t.append(acl_freq[tuple(lemma.split())])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # cos_target\n",
    "    sentence_embed = get_sentence_embedding(informal_test_d[token_id]['targetsentence'])\n",
    "    word_embed = get_word_embedding(lemma)\n",
    "    t.append(np.dot(sentence_embed, word_embed)/(np.sqrt(np.dot(sentence_embed, sentence_embed))*np.sqrt(np.dot(word_embed, word_embed))))\n",
    "    \n",
    "    # euclidean_distance\n",
    "    t.append(np.linalg.norm(sentence_embed-word_embed))\n",
    "    \n",
    "    # posMASC\n",
    "    t.append(informal_test_d[token_id]['posMASC'])\n",
    "    \n",
    "    # is_problematic\n",
    "    t.append(1 if informal_test_d[token_id]['problematic']=='yes' else 0)\n",
    "    \n",
    "    # word_length\n",
    "    t.append(len(lemma))\n",
    "    \n",
    "    # count_vowel\n",
    "    t.append(sum(list(map(lemma.lower().count, 'aeiou'))))\n",
    "    \n",
    "    # y\n",
    "    t.append(1)\n",
    "    \n",
    "    features.append(t)\n",
    "    \n",
    "for token_id in non_informal_test_d:\n",
    "    t = list()\n",
    "    lemma = non_informal_test_d[token_id]['lemma']\n",
    "    # freq_beautiful_data\n",
    "    try:\n",
    "        t.append(beatiful_data_freq[lemma])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_coca_general\n",
    "    try:\n",
    "        t.append(allwords_df.loc[allwords_df['word'] == lemma].iloc[0]['COCA-All'])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_acl\n",
    "    try:\n",
    "        t.append(acl_freq[tuple(lemma.split())])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # cos_target\n",
    "    sentence_embed = get_sentence_embedding(non_informal_test_d[token_id]['targetsentence'])\n",
    "    word_embed = get_word_embedding(lemma)\n",
    "    t.append(np.dot(sentence_embed, word_embed)/(np.sqrt(np.dot(sentence_embed, sentence_embed))*np.sqrt(np.dot(word_embed, word_embed))))\n",
    "    \n",
    "    # euclidean_distance\n",
    "    t.append(np.linalg.norm(sentence_embed-word_embed))\n",
    "    \n",
    "    # posMASC\n",
    "    t.append(non_informal_test_d[token_id]['posMASC'])\n",
    "    \n",
    "    # is_problematic\n",
    "    t.append(1 if non_informal_test_d[token_id]['problematic']=='yes' else 0)\n",
    "    \n",
    "    # word_length\n",
    "    t.append(len(lemma))\n",
    "    \n",
    "    # count_vowel\n",
    "    t.append(sum(list(map(lemma.lower().count, 'aeiou'))))\n",
    "    \n",
    "    # y\n",
    "    t.append(0)\n",
    "    \n",
    "    features.append(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test = pd.DataFrame(features, columns=train_features_cols+y_feature)\n",
    "\n",
    "# shuffle the dataframe\n",
    "df_test = df_test.sample(frac=1).reset_index(drop=True)\n",
    "\n",
    "# label encode the POS tag\n",
    "df_test['posMASC_le'] = label_encoder.fit_transform(df_test.posMASC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = clf.predict(df_test[train_features])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Accuracy : ', metrics.accuracy_score(df_test[y_feature], y_pred))\n",
    "print('Precision : ', metrics.precision_score(df_test[y_feature], y_pred))\n",
    "print('Recall : ', metrics.recall_score(df_test[y_feature], y_pred))\n",
    "print('F-Measure : ', metrics.f1_score(df_test[y_feature], y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "compute the features from the resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features = list()\n",
    "for token_id in informal_test_d:\n",
    "    t = list()\n",
    "    lemma = informal_test_d[token_id]['lemma']\n",
    "    # freq_beautiful_data\n",
    "    try:\n",
    "        t.append(beatiful_data_freq[lemma])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_coca_general\n",
    "    try:\n",
    "        t.append(allwords_df.loc[allwords_df['word'] == lemma].iloc[0]['COCA-All'])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # freq_acl\n",
    "    try:\n",
    "        t.append(acl_freq[tuple(lemma.split())])\n",
    "    except:\n",
    "        t.append(0)\n",
    "    \n",
    "    # cos_target\n",
    "    sentence_embed = get_sentence_embedding(informal_test_d[token_id]['targetsentence'])\n",
    "    word_embed = get_word_embedding(lemma)\n",
    "    t.append(np.dot(sentence_embed, word_embed)/(np.sqrt(np.dot(sentence_embed, sentence_embed))*np.sqrt(np.dot(word_embed, word_embed))))\n",
    "    \n",
    "    # euclidean_distance\n",
    "    t.append(np.linalg.norm(sentence_embed-word_embed))\n",
    "    \n",
    "    # posMASC\n",
    "    t.append(informal_test_d[token_id]['posMASC'])\n",
    "    \n",
    "    # is_problematic\n",
    "    t.append(1 if informal_test_d[token_id]['problematic']=='yes' else 0)\n",
    "    \n",
    "    # word_length\n",
    "    t.append(len(lemma))\n",
    "    \n",
    "    # count_vowel\n",
    "    t.append(sum(list(map(lemma.lower().count, 'aeiou'))))\n",
    "    \n",
    "    # y\n",
    "    t.append(1)\n",
    "    \n",
    "    features.append(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_gold_test = pd.DataFrame(features, columns=train_features_cols+y_feature)\n",
    "\n",
    "# shuffle the dataframe\n",
    "df_gold_test = df_gold_test.sample(frac=1).reset_index(drop=True)\n",
    "\n",
    "# label encode the POS tag\n",
    "df_gold_test['posMASC_le'] = label_encoder.fit_transform(df_gold_test.posMASC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_gold_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_gold_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = clf.predict(df_gold_test[train_features])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Accuracy : ', metrics.accuracy_score(df_gold_test[y_feature], y_pred))\n",
    "print('Precision : ', metrics.precision_score(df_gold_test[y_feature], y_pred))\n",
    "print('Recall : ', metrics.recall_score(df_gold_test[y_feature], y_pred))\n",
    "print('F-Measure : ', metrics.f1_score(df_gold_test[y_feature], y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}