DistilBert Model for Fake News Detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sFU9LTOyMiMj"
   },
   "source": [
    "# Deep Learning for NLP\n",
    "\n",
    "**Fake news classifier**: Train a text classification model to detect fake news articles!\n",
    "\n",
    "* Used Dataset: https://www.kaggle.com/clmentbisaillon/fake-and-real-news-dataset\n",
    "* Develop an NLP model for classification that uses a pretrained language model\n",
    "* Finetune model on the dataset, and generate an AUC curve of your model on the test set. \n",
    "* Model on HuggingFace: https://huggingface.co/ahmednasser/DistilBert-FakeNews.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:46:20.598811Z",
     "iopub.status.busy": "2022-04-08T10:46:20.597848Z",
     "iopub.status.idle": "2022-04-08T10:46:40.498386Z",
     "shell.execute_reply": "2022-04-08T10:46:40.497630Z",
     "shell.execute_reply.started": "2022-04-08T10:46:20.598692Z"
    },
    "id": "E90i018KyJH3"
   },
   "outputs": [],
   "source": [
    "!pip install transformers\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import spacy\n",
    "import torch\n",
    "import os\n",
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:46:40.500439Z",
     "iopub.status.busy": "2022-04-08T10:46:40.500168Z",
     "iopub.status.idle": "2022-04-08T10:46:40.509621Z",
     "shell.execute_reply": "2022-04-08T10:46:40.508721Z",
     "shell.execute_reply.started": "2022-04-08T10:46:40.500406Z"
    }
   },
   "outputs": [],
   "source": [
    "device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:46:40.511752Z",
     "iopub.status.busy": "2022-04-08T10:46:40.511246Z",
     "iopub.status.idle": "2022-04-08T10:46:40.517053Z",
     "shell.execute_reply": "2022-04-08T10:46:40.516313Z",
     "shell.execute_reply.started": "2022-04-08T10:46:40.511715Z"
    }
   },
   "outputs": [],
   "source": [
    "#sequence max length\n",
    "MAX_LEN = 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:46:40.519696Z",
     "iopub.status.busy": "2022-04-08T10:46:40.519341Z",
     "iopub.status.idle": "2022-04-08T10:46:41.011882Z",
     "shell.execute_reply": "2022-04-08T10:46:41.011056Z",
     "shell.execute_reply.started": "2022-04-08T10:46:40.519643Z"
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "# Uncomment to download \"stopwords\"\n",
    "nltk.download(\"stopwords\")\n",
    "from nltk.corpus import stopwords\n",
    "import re\n",
    "\n",
    "def text_preprocessing(s):\n",
    "    \"\"\"\n",
    "    - Lowercase the sentence\n",
    "    - Change \"'t\" to \"not\"\n",
    "    - Remove \"@name\"\n",
    "    - Remove other special characters\n",
    "    - Remove stop words except \"not\" and \"can\"\n",
    "    - Remove trailing whitespace\n",
    "    \"\"\"\n",
    "    s = s.lower()\n",
    "    \n",
    "    s = s.split(\"(reuters) -\")[-1]\n",
    "    s= s.replace(\"reuters\", \"\")\n",
    "    \n",
    "    # Change 't to 'not'\n",
    "    s = re.sub(r\"\\'t\", \" not\", s)\n",
    "    # Remove @name\n",
    "    s = re.sub(r'(@.*?)[\\s]', ' ', s)\n",
    "\n",
    "    # Remove some special characters\n",
    "    s = re.sub(r'([\\;\\:\\|•«\\n])', ' ', s)\n",
    "    # Remove stopwords except 'not' and 'can'\n",
    "    s = \" \".join([word for word in s.split()\n",
    "                  if word not in stopwords.words('english')\n",
    "                  or word in ['not', 'can']])\n",
    "    # Remove trailing whitespace\n",
    "    s = re.sub(r'\\s+', ' ', s).strip()\n",
    "    \n",
    "    \n",
    "    return s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:46:41.014096Z",
     "iopub.status.busy": "2022-04-08T10:46:41.013151Z",
     "iopub.status.idle": "2022-04-08T10:46:45.067807Z",
     "shell.execute_reply": "2022-04-08T10:46:45.066872Z",
     "shell.execute_reply.started": "2022-04-08T10:46:41.014053Z"
    }
   },
   "outputs": [],
   "source": [
    "fake_news = pd.read_csv('../input/fake-and-real-news-dataset/Fake.csv', engine='python')\n",
    "true_news = pd.read_csv('../input/fake-and-real-news-dataset/True.csv', engine='python')\n",
    "#adding target variable 'fake' for both sets\n",
    "fake_news['fake'] = np.ones(len(fake_news))\n",
    "true_news['fake'] = np.zeros(len(true_news))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:46:45.076000Z",
     "iopub.status.busy": "2022-04-08T10:46:45.073469Z",
     "iopub.status.idle": "2022-04-08T10:46:45.087581Z",
     "shell.execute_reply": "2022-04-08T10:46:45.086707Z",
     "shell.execute_reply.started": "2022-04-08T10:46:45.075954Z"
    }
   },
   "outputs": [],
   "source": [
    "len(true_news) + len(fake_news)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**The total number of points is *44898*, for feasability I am selecting 2000 from each data set for training and evaluation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:46:45.094647Z",
     "iopub.status.busy": "2022-04-08T10:46:45.093062Z",
     "iopub.status.idle": "2022-04-08T10:46:45.116916Z",
     "shell.execute_reply": "2022-04-08T10:46:45.116046Z",
     "shell.execute_reply.started": "2022-04-08T10:46:45.094603Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "news = pd.concat([true_news[:2000], fake_news[:2000]])\n",
    "news = news[['text', 'fake']]\n",
    "\n",
    "\n",
    "X = news.text.values\n",
    "y = news.fake.values\n",
    "\n",
    "X_train, X_val, y_train, y_val =\\\n",
    "    train_test_split(X, y, test_size=0.2, random_state=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:46:45.124254Z",
     "iopub.status.busy": "2022-04-08T10:46:45.121757Z",
     "iopub.status.idle": "2022-04-08T10:46:45.136021Z",
     "shell.execute_reply": "2022-04-08T10:46:45.134927Z",
     "shell.execute_reply.started": "2022-04-08T10:46:45.124139Z"
    }
   },
   "outputs": [],
   "source": [
    "print('training points: ', len(X_train))\n",
    "print('evaluation points: ', len(X_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tokenization and Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:46:45.142950Z",
     "iopub.status.busy": "2022-04-08T10:46:45.141139Z",
     "iopub.status.idle": "2022-04-08T10:46:50.183674Z",
     "shell.execute_reply": "2022-04-08T10:46:50.182885Z",
     "shell.execute_reply.started": "2022-04-08T10:46:45.142908Z"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import DistilBertModel,DistilBertTokenizer\n",
    "tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:46:50.187252Z",
     "iopub.status.busy": "2022-04-08T10:46:50.187023Z",
     "iopub.status.idle": "2022-04-08T10:46:50.194875Z",
     "shell.execute_reply": "2022-04-08T10:46:50.194082Z",
     "shell.execute_reply.started": "2022-04-08T10:46:50.187223Z"
    }
   },
   "outputs": [],
   "source": [
    "def preprocessing_for_bert(data):\n",
    "    \"\"\"Perform required preprocessing steps for pretrained BERT.\n",
    "    @param    data (np.array): Array of texts to be processed.\n",
    "    @return   input_ids (torch.Tensor): Tensor of token ids to be fed to a model.\n",
    "    @return   attention_masks (torch.Tensor): Tensor of indices specifying which\n",
    "                  tokens should be attended to by the model.\n",
    "    \"\"\"\n",
    "    # Create empty lists to store outputs\n",
    "    input_ids = []\n",
    "    attention_masks = []\n",
    "\n",
    "    # For every sentence...\n",
    "    for sent in data:\n",
    "        # `encode_plus` will:\n",
    "        #    (1) Tokenize the sentence\n",
    "        #    (2) Add the `[CLS]` and `[SEP]` token to the start and end\n",
    "        #    (3) Truncate/Pad sentence to max length\n",
    "        #    (4) Map tokens to their IDs\n",
    "        #    (5) Create attention mask\n",
    "        #    (6) Return a dictionary of outputs\n",
    "        encoded_sent = tokenizer.encode_plus(\n",
    "            text=text_preprocessing(sent),  # Preprocess sentence\n",
    "            add_special_tokens=True,        # Add `[CLS]` and `[SEP]`\n",
    "            max_length=MAX_LEN,                  # Max length to truncate/pad\n",
    "            pad_to_max_length=True,         # Pad sentence to max length\n",
    "            #return_tensors='pt',           # Return PyTorch tensor\n",
    "            return_attention_mask=True      # Return attention mask\n",
    "            )\n",
    "        \n",
    "        # Add the outputs to the lists\n",
    "        input_ids.append(encoded_sent.get('input_ids'))\n",
    "        attention_masks.append(encoded_sent.get('attention_mask'))\n",
    "\n",
    "    # Convert lists to tensors\n",
    "    input_ids = torch.tensor(input_ids)\n",
    "    attention_masks = torch.tensor(attention_masks)\n",
    "\n",
    "    return input_ids, attention_masks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**preprocess & tokenize training and evaluation sets**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:46:50.196624Z",
     "iopub.status.busy": "2022-04-08T10:46:50.196178Z",
     "iopub.status.idle": "2022-04-08T10:50:42.057605Z",
     "shell.execute_reply": "2022-04-08T10:50:42.056863Z",
     "shell.execute_reply.started": "2022-04-08T10:46:50.196583Z"
    }
   },
   "outputs": [],
   "source": [
    "train_inputs, train_masks = preprocessing_for_bert(X_train)\n",
    "val_inputs, val_masks = preprocessing_for_bert(X_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:50:42.059074Z",
     "iopub.status.busy": "2022-04-08T10:50:42.058821Z",
     "iopub.status.idle": "2022-04-08T10:50:42.075332Z",
     "shell.execute_reply": "2022-04-08T10:50:42.074654Z",
     "shell.execute_reply.started": "2022-04-08T10:50:42.059039Z"
    }
   },
   "outputs": [],
   "source": [
    "from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
    "\n",
    "# Convert other data types to torch.Tensor\n",
    "train_labels = torch.LongTensor(y_train) \n",
    "val_labels = torch.LongTensor(y_val)\n",
    "\n",
    "batch_size = 32\n",
    "\n",
    "# Create the DataLoader for our training set\n",
    "train_data = TensorDataset(train_inputs, train_masks, train_labels)\n",
    "train_sampler = RandomSampler(train_data)\n",
    "train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n",
    "\n",
    "# Create the DataLoader for our validation set\n",
    "val_data = TensorDataset(val_inputs, val_masks, val_labels)\n",
    "val_sampler = SequentialSampler(val_data)\n",
    "val_dataloader = DataLoader(val_data, sampler=val_sampler, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining the classifier class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:50:42.077074Z",
     "iopub.status.busy": "2022-04-08T10:50:42.076763Z",
     "iopub.status.idle": "2022-04-08T10:50:42.088091Z",
     "shell.execute_reply": "2022-04-08T10:50:42.087338Z",
     "shell.execute_reply.started": "2022-04-08T10:50:42.077037Z"
    }
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from transformers import AutoConfig\n",
    "from transformers.modeling_utils import PreTrainedModel\n",
    "\n",
    "# Create the BertClassfier class\n",
    "class BertClassifier(PreTrainedModel):\n",
    "    \"\"\"Bert Model for Classification Tasks.\n",
    "    \"\"\"\n",
    "    def __init__(self, config, freeze_bert=False):\n",
    "        \"\"\"\n",
    "        @param    bert: a DistilBertModel object\n",
    "        @param    classifier: a torch.nn.Module classifier\n",
    "        @param    freeze_bert (bool): Set `False` to fine-tune the BERT model\n",
    "        \"\"\"\n",
    "        super(BertClassifier, self).__init__(config)\n",
    "        # Specify hidden size of BERT, hidden size of our classifier, and number of labels\n",
    "        D_in, H, D_out = 768, 50, 2\n",
    "\n",
    "        # Instantiate BERT model\n",
    "        self.bert = DistilBertModel.from_pretrained('distilbert-base-uncased')\n",
    "        #self.bert.confi\n",
    "        \n",
    "        # Instantiate an one-layer feed-forward classifier\n",
    "        self.classifier = nn.Sequential(\n",
    "            nn.Linear(D_in, H),\n",
    "            nn.ReLU(),\n",
    "            nn.Linear(H, D_out)\n",
    "        )\n",
    "\n",
    "        # Freeze the BERT model\n",
    "        if freeze_bert:\n",
    "            for param in self.bert.parameters():\n",
    "                param.requires_grad = False\n",
    "        \n",
    "    def forward(self, input_ids, attention_mask):\n",
    "        \"\"\"\n",
    "        Feed input to BERT and the classifier to compute logits.\n",
    "        @param    input_ids (torch.Tensor): an input tensor with shape (batch_size,\n",
    "                      max_length)\n",
    "        @param    attention_mask (torch.Tensor): a tensor that hold attention mask\n",
    "                      information with shape (batch_size, max_length)\n",
    "        @return   logits (torch.Tensor): an output tensor with shape (batch_size,\n",
    "                      num_labels)\n",
    "        \"\"\"\n",
    "        # Feed input to BERT\n",
    "        outputs = self.bert(input_ids=input_ids,\n",
    "                            attention_mask=attention_mask)\n",
    "        \n",
    "        # Extract the last hidden state of the token `[CLS]` for classification task\n",
    "        last_hidden_state_cls = outputs[0][:, 0, :]\n",
    "\n",
    "        # Feed input to classifier to compute logits\n",
    "        logits = self.classifier(last_hidden_state_cls)\n",
    "\n",
    "        return logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:50:42.089813Z",
     "iopub.status.busy": "2022-04-08T10:50:42.089420Z",
     "iopub.status.idle": "2022-04-08T10:50:42.103514Z",
     "shell.execute_reply": "2022-04-08T10:50:42.102792Z",
     "shell.execute_reply.started": "2022-04-08T10:50:42.089759Z"
    }
   },
   "outputs": [],
   "source": [
    "from transformers import AdamW, get_linear_schedule_with_warmup\n",
    "\n",
    "def initialize_model(epochs=4):\n",
    "    \"\"\"Initialize the Bert Classifier, the optimizer and the learning rate scheduler.\n",
    "    \"\"\"\n",
    "    # Instantiate Bert Classifier\n",
    "    bert_classifier = BertClassifier(config=AutoConfig.from_pretrained('distilbert-base-uncased'), freeze_bert=False)\n",
    "\n",
    "    # Tell PyTorch to run the model on GPU\n",
    "    bert_classifier.to(device)\n",
    "\n",
    "    # Create the optimizer\n",
    "    optimizer = AdamW(bert_classifier.parameters(),\n",
    "                      lr=5e-5,    # Default learning rate\n",
    "                      eps=1e-8    # Default epsilon value\n",
    "                      )\n",
    "\n",
    "    # Total number of training steps\n",
    "    total_steps = len(train_dataloader) * epochs\n",
    "\n",
    "    # Set up the learning rate scheduler\n",
    "    scheduler = get_linear_schedule_with_warmup(optimizer,\n",
    "                                                num_warmup_steps=0, # Default value\n",
    "                                                num_training_steps=total_steps)\n",
    "    return bert_classifier, optimizer, scheduler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:50:42.105260Z",
     "iopub.status.busy": "2022-04-08T10:50:42.104998Z",
     "iopub.status.idle": "2022-04-08T10:50:42.124059Z",
     "shell.execute_reply": "2022-04-08T10:50:42.123121Z",
     "shell.execute_reply.started": "2022-04-08T10:50:42.105224Z"
    }
   },
   "outputs": [],
   "source": [
    "import random\n",
    "import time\n",
    "\n",
    "# Specify loss function\n",
    "loss_fn = nn.CrossEntropyLoss()\n",
    "\n",
    "def set_seed(seed_value=100):\n",
    "    \"\"\"Set seed for reproducibility.\n",
    "    \"\"\"\n",
    "    random.seed(seed_value)\n",
    "    np.random.seed(seed_value)\n",
    "    torch.manual_seed(seed_value)\n",
    "    torch.cuda.manual_seed_all(seed_value)\n",
    "\n",
    "def train(model, train_dataloader, val_dataloader=None, epochs=4, evaluation=False):\n",
    "    \"\"\"Train the BertClassifier model.\n",
    "    \"\"\"\n",
    "    # Start training loop\n",
    "    print(\"Start training...\\n\")\n",
    "    for epoch_i in range(epochs):\n",
    "        # =======================================\n",
    "        #               Training\n",
    "        # =======================================\n",
    "        # Print the header of the result table\n",
    "        print(f\"{'Epoch':^7} | {'Batch':^7} | {'Train Loss':^12} | {'Val Loss':^10} | {'Val Acc':^9} | {'Elapsed':^9}\")\n",
    "        print(\"-\"*70)\n",
    "\n",
    "        # Measure the elapsed time of each epoch\n",
    "        t0_epoch, t0_batch = time.time(), time.time()\n",
    "\n",
    "        # Reset tracking variables at the beginning of each epoch\n",
    "        total_loss, batch_loss, batch_counts = 0, 0, 0\n",
    "\n",
    "        # Put the model into the training mode\n",
    "        model.train()\n",
    "\n",
    "        # For each batch of training data...\n",
    "        for step, batch in enumerate(train_dataloader):\n",
    "            batch_counts +=1\n",
    "            # Load batch to GPU\n",
    "            b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)\n",
    "\n",
    "            # Zero out any previously calculated gradients\n",
    "            model.zero_grad()\n",
    "\n",
    "            # Perform a forward pass. This will return logits.\n",
    "            logits = model(b_input_ids, b_attn_mask)\n",
    "\n",
    "            # Compute loss and accumulate the loss values\n",
    "            loss = loss_fn(logits, b_labels)\n",
    "            batch_loss += loss.item()\n",
    "            total_loss += loss.item()\n",
    "\n",
    "            # Perform a backward pass to calculate gradients\n",
    "            loss.backward()\n",
    "\n",
    "            # Clip the norm of the gradients to 1.0 to prevent \"exploding gradients\"\n",
    "            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)\n",
    "\n",
    "            # Update parameters and the learning rate\n",
    "            optimizer.step()\n",
    "            scheduler.step()\n",
    "\n",
    "            # Print the loss values and time elapsed for every 20 batches\n",
    "            if (step % 20 == 0 and step != 0) or (step == len(train_dataloader) - 1):\n",
    "                # Calculate time elapsed for 20 batches\n",
    "                time_elapsed = time.time() - t0_batch\n",
    "\n",
    "                # Print training results\n",
    "                print(f\"{epoch_i + 1:^7} | {step:^7} | {batch_loss / batch_counts:^12.6f} | {'-':^10} | {'-':^9} | {time_elapsed:^9.2f}\")\n",
    "\n",
    "                # Reset batch tracking variables\n",
    "                batch_loss, batch_counts = 0, 0\n",
    "                t0_batch = time.time()\n",
    "\n",
    "        # Calculate the average loss over the entire training data\n",
    "        avg_train_loss = total_loss / len(train_dataloader)\n",
    "\n",
    "        print(\"-\"*70)\n",
    "        # =======================================\n",
    "        #               Evaluation\n",
    "        # =======================================\n",
    "        if evaluation == True:\n",
    "            # After the completion of each training epoch, measure the model's performance\n",
    "            # on our validation set.\n",
    "            val_loss, val_accuracy = evaluate(model, val_dataloader)\n",
    "\n",
    "            # Print performance over the entire training data\n",
    "            time_elapsed = time.time() - t0_epoch\n",
    "            \n",
    "            print(f\"{epoch_i + 1:^7} | {'-':^7} | {avg_train_loss:^12.6f} | {val_loss:^10.6f} | {val_accuracy:^9.2f} | {time_elapsed:^9.2f}\")\n",
    "            print(\"-\"*70)\n",
    "        print(\"\\n\")\n",
    "    \n",
    "    print(\"Training complete!\")\n",
    "\n",
    "\n",
    "def evaluate(model, val_dataloader):\n",
    "    \"\"\"After the completion of each training epoch, measure the model's performance\n",
    "    on our validation set.\n",
    "    \"\"\"\n",
    "    # Put the model into the evaluation mode. The dropout layers are disabled during\n",
    "    # the test time.\n",
    "    model.eval()\n",
    "\n",
    "    # Tracking variables\n",
    "    val_accuracy = []\n",
    "    val_loss = []\n",
    "\n",
    "    # For each batch in our validation set...\n",
    "    for batch in val_dataloader:\n",
    "        # Load batch to GPU\n",
    "        b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)\n",
    "\n",
    "        # Compute logits\n",
    "        with torch.no_grad():\n",
    "            logits = model(b_input_ids, b_attn_mask)\n",
    "\n",
    "        # Compute loss\n",
    "        loss = loss_fn(logits, b_labels)\n",
    "        val_loss.append(loss.item())\n",
    "\n",
    "        # Get the predictions\n",
    "        preds = torch.argmax(logits, dim=1).flatten()\n",
    "\n",
    "        # Calculate the accuracy rate\n",
    "        accuracy = (preds == b_labels).cpu().numpy().mean() * 100\n",
    "        val_accuracy.append(accuracy)\n",
    "\n",
    "    # Compute the average accuracy and loss over the validation set.\n",
    "    val_loss = np.mean(val_loss)\n",
    "    val_accuracy = np.mean(val_accuracy)\n",
    "\n",
    "    return val_loss, val_accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:50:42.125354Z",
     "iopub.status.busy": "2022-04-08T10:50:42.125060Z",
     "iopub.status.idle": "2022-04-08T10:52:06.403165Z",
     "shell.execute_reply": "2022-04-08T10:52:06.402395Z",
     "shell.execute_reply.started": "2022-04-08T10:50:42.125319Z"
    }
   },
   "outputs": [],
   "source": [
    "set_seed(100)    # Set seed for reproducibility\n",
    "bert_classifier, optimizer, scheduler = initialize_model(epochs=2)\n",
    "train(bert_classifier, train_dataloader, val_dataloader, epochs=2, evaluation=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:52:06.404778Z",
     "iopub.status.busy": "2022-04-08T10:52:06.404498Z",
     "iopub.status.idle": "2022-04-08T10:52:06.411771Z",
     "shell.execute_reply": "2022-04-08T10:52:06.410880Z",
     "shell.execute_reply.started": "2022-04-08T10:52:06.404742Z"
    }
   },
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "\n",
    "def bert_predict(model, test_dataloader):\n",
    "    \"\"\"Perform a forward pass on the trained BERT model to predict probabilities\n",
    "    on the test set.\n",
    "    \"\"\"\n",
    "    # Put the model into the evaluation mode. The dropout layers are disabled during\n",
    "    # the test time.\n",
    "    model.eval()\n",
    "\n",
    "    all_logits = []\n",
    "\n",
    "    # For each batch in our test set...\n",
    "    for batch in test_dataloader:\n",
    "        # Load batch to GPU\n",
    "        b_input_ids, b_attn_mask = tuple(t.to(device) for t in batch)[:2]\n",
    "\n",
    "        # Compute logits\n",
    "        with torch.no_grad():\n",
    "            logits = model(b_input_ids, b_attn_mask)\n",
    "        all_logits.append(logits)\n",
    "    \n",
    "    # Concatenate logits from each batch\n",
    "    all_logits = torch.cat(all_logits, dim=0)\n",
    "\n",
    "    # Apply softmax to calculate probabilities\n",
    "    probs = F.softmax(all_logits, dim=1).cpu().numpy()\n",
    "\n",
    "    return probs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:52:06.413682Z",
     "iopub.status.busy": "2022-04-08T10:52:06.413149Z",
     "iopub.status.idle": "2022-04-08T10:52:06.423058Z",
     "shell.execute_reply": "2022-04-08T10:52:06.422354Z",
     "shell.execute_reply.started": "2022-04-08T10:52:06.413626Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score, roc_curve, auc\n",
    "\n",
    "def evaluate_roc(probs, y_true):\n",
    "    \"\"\"\n",
    "    - Print AUC and accuracy on the test set\n",
    "    - Plot ROC\n",
    "    @params    probs (np.array): an array of predicted probabilities with shape (len(y_true), 2)\n",
    "    @params    y_true (np.array): an array of the true values with shape (len(y_true),)\n",
    "    \"\"\"\n",
    "    preds = probs[:, 1]\n",
    "    fpr, tpr, threshold = roc_curve(y_true, preds)\n",
    "    roc_auc = auc(fpr, tpr)\n",
    "    print(f'AUC: {roc_auc:.4f}')\n",
    "       \n",
    "    # Get accuracy over the test set\n",
    "    y_pred = np.where(preds >= 0.5, 1, 0)\n",
    "    accuracy = accuracy_score(y_true, y_pred)\n",
    "    print(f'Accuracy: {accuracy*100:.2f}%')\n",
    "    \n",
    "    # Plot ROC AUC\n",
    "    plt.title('Receiver Operating Characteristic')\n",
    "    plt.plot(fpr, tpr, 'b', label = 'AUC = %0.2f' % roc_auc)\n",
    "    plt.legend(loc = 'lower right')\n",
    "    plt.plot([0, 1], [0, 1],'r--')\n",
    "    plt.xlim([0, 1])\n",
    "    plt.ylim([0, 1])\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section I will be testing the model against test (unseen data), for feasiblity I will only test on 4000 instances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:52:06.424622Z",
     "iopub.status.busy": "2022-04-08T10:52:06.424358Z",
     "iopub.status.idle": "2022-04-08T10:56:20.370887Z",
     "shell.execute_reply": "2022-04-08T10:56:20.370131Z",
     "shell.execute_reply.started": "2022-04-08T10:52:06.424587Z"
    }
   },
   "outputs": [],
   "source": [
    "test_data = pd.concat([true_news[6000:8000], fake_news[6000:8000]])\n",
    "\n",
    "test_labels = test_data.fake\n",
    "test_inputs, test_masks = preprocessing_for_bert(test_data.text)\n",
    "test_dataset = TensorDataset(test_inputs, test_masks)\n",
    "test_sampler = SequentialSampler(test_dataset)\n",
    "test_dataloader = DataLoader(test_dataset, sampler=test_sampler, batch_size=32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:56:20.372473Z",
     "iopub.status.busy": "2022-04-08T10:56:20.372240Z",
     "iopub.status.idle": "2022-04-08T10:56:32.938314Z",
     "shell.execute_reply": "2022-04-08T10:56:32.937430Z",
     "shell.execute_reply.started": "2022-04-08T10:56:20.372441Z"
    }
   },
   "outputs": [],
   "source": [
    "probs = bert_predict(bert_classifier, test_dataloader)\n",
    "evaluate_roc(probs, test_labels.to_numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kpInVUMLyJ24"
   },
   "source": [
    "**Write up**: \n",
    "* Link to the model on Hugging Face Hub: https://huggingface.co/ahmednasser/DistilBert-FakeNews \n",
    "* **Observations**: I have used 4000 instances for training and evalutaiton with split of .2, and used another 4000 samples for testing. The training routine consisted of only 2 epochs even though, the accuracy is ver high 99.78%. There have been a discussion about the validity of this dataset and why it's easy for the models to differentiate between fake and true news. For example, most of the True news contain the word *reuters* and even when I removed this word while preprocessing the model is still performing well which suggests that data may have biases/systematic difference between fake and true news."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-08T10:56:32.948803Z",
     "iopub.status.busy": "2022-04-08T10:56:32.948390Z",
     "iopub.status.idle": "2022-04-08T10:56:33.495366Z",
     "shell.execute_reply": "2022-04-08T10:56:33.494626Z",
     "shell.execute_reply.started": "2022-04-08T10:56:32.948758Z"
    }
   },
   "outputs": [],
   "source": [
    "tokenizer.save_pretrained('./')\n",
    "bert_classifier.save_pretrained('./')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}