notebooks/text_classification/tfhub_text_classification/BERT_Binary_Text_Classification.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Binary text classification using BERT models from TF Hub\n",
    "\n",
    "This notebook demonstrates fine tuning BERT models from [TF Hub](https://tfhub.dev) with binary text classification datasets.\n",
    "\n",
    "The notebook performs the following steps:\n",
    "1. [Import dependencies and setup parameters](#1.-Import-dependencies-and-setup-parameters)\n",
    "2. [Prepare the dataset](#2.-Prepare-the-dataset)\n",
    "3. [Build the model](#3.-Build-the-model)\n",
    "4. [Fine tuning and evaluation](#4.-Fine-tuning-and-evaluation)\n",
    "5. [Export the model](#5.-Export-the-model)\n",
    "6. [Reload the model and make predictions](#6.-Reload-the-model-and-make-predictions)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Import dependencies and setup parameters\n",
    "\n",
    "This notebook assumes that you have already followed the instructions in the [README.md](/notebooks/README.md) to setup a TensorFlow environment with all the dependencies required to run the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "import tensorflow_hub as hub\n",
    "import tensorflow_datasets as tfds\n",
    "\n",
    "from bert_utils import get_model_map\n",
    "from tlt.utils.file_utils import download_and_extract_zip_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note that tensorflow_text isn't used directly but the import is required to register ops used by the\n",
    "# BERT text preprocessor\n",
    "! pip3 install tensorflow-text==2.12 --no-deps\n",
    "import tensorflow_text"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook will run one of the supported [BERT models from TF Hub](https://tfhub.dev/google/collections/bert/1). The table below has a list of the available models and links to their URLs in TF Hub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the TF Hub model map from json and print a list of the supported models\n",
    "tfhub_model_map, models_df = get_model_map(\"tfhub_bert_model_map_classifier.json\", return_data_frame=True)\n",
    "models_df.style.hide(axis=\"index\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Specify the name of the BERT model to use. This string must match one of the models listed in the table above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = \"small_bert/bert_en_uncased_L-2_H-128_A-2\"\n",
    "if model_name not in tfhub_model_map.keys():\n",
    "    raise ValueError(\"The specified model name ({}) is not supported\".format(model_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define a directory to download the dataset\n",
    "dataset_directory = os.environ[\"DATASET_DIR\"] if \"DATASET_DIR\" in os.environ else \\\n",
    "    os.path.join(os.environ[\"HOME\"], \"dataset\")\n",
    "\n",
    "# Define an output directory for the saved model to be exported\n",
    "output_directory = os.environ[\"OUTPUT_DIR\"] if \"OUTPUT_DIR\" in os.environ else \\\n",
    "    os.path.join(os.environ[\"HOME\"], \"output\")\n",
    "\n",
    "# Output directory for logs and checkpoints generated during training\n",
    "if not os.path.isdir(output_directory):\n",
    "    os.makedirs(output_directory)\n",
    "    \n",
    "tfhub_preprocess = tfhub_model_map[model_name][\"preprocess\"]\n",
    "tfhub_bert_encoder = tfhub_model_map[model_name][\"bert_encoder\"]\n",
    "\n",
    "print(\"Using TF Hub model:\", model_name)\n",
    "print(\"BERT encoder URL:\", tfhub_bert_encoder)\n",
    "print(\"Preprocessor URL:\", tfhub_preprocess)\n",
    "print(\"Dataset directory:\", dataset_directory)\n",
    "print(\"Output directory:\", output_directory)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Prepare the dataset\n",
    "\n",
    "The notebook has two options for getting a dataset:\n",
    "* Option A: Use a dataset from the [TensorFlow Datasets catalog](https://www.tensorflow.org/datasets/catalog/overview)\n",
    "* Option B: Use a custom dataset (downloaded from another source or from your local system)\n",
    "\n",
    "In both cases, the code ends up defining [`tf.data.Dataset`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset) objects for each split (train, validation, and test) and a map for the translating the numerical to string label.\n",
    "\n",
    "Execute the following cell to set the batch size and declare the base class used for the dataset setup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the dataset batch size\n",
    "batch_size = 32\n",
    "\n",
    "# Base class used for defining the binary text classification dataset being used\n",
    "class BinaryTextClassificationData():\n",
    "    def __init__(self, batch_size, label_map):\n",
    "        self.batch_size = batch_size\n",
    "        self.label_map = label_map\n",
    "        self.reverse_label_map = {}\n",
    "        self.train_ds = None\n",
    "        self.val_ds = None\n",
    "        self.test_ds = None\n",
    "        self.dataset_name = \"\"\n",
    "        \n",
    "        for k, v in self.label_map.items():\n",
    "            self.reverse_label_map[v] = k\n",
    "        \n",
    "    def get_str_label(self, numerical_value):\n",
    "        if not isinstance(numerical_value, int):\n",
    "            numerical_value = int(tf.math.round(numerical_value))\n",
    "        \n",
    "        if numerical_value in self.label_map.keys():\n",
    "            return label_map[numerical_value]\n",
    "        else:\n",
    "            raise ValueError(\"The key {} was not found in the label map\".format(numerical_value))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the base class is defined, either run [Option A to use the TensorFlow Dataset catalog](#Option-A:-Use-a-TensorFlow-dataset) or [Option B for a custom dataset](#Option-B:-Use-a-custom-dataset) downloaded from online or from your local system."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option A: Use a TensorFlow dataset\n",
    "\n",
    "[TensorFlow Datasets](https://www.tensorflow.org/datasets) has a [catalog of datasets](https://www.tensorflow.org/datasets/catalog/overview) that can be specified by name. Information about the dataset is available in the catalog (including information on the size of the dataset and the splits).\n",
    "\n",
    "The next cell demonstrates using the [`imdb_reviews`](https://www.tensorflow.org/datasets/catalog/imdb_reviews) dataset from the TensorFlow datasets catalog to get splits for training, validation, and test. Skip the next cell if you would like to instead use \"Option B\" for a custom dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TFDSBinaryTextClassificationData(BinaryTextClassificationData):\n",
    "    def __init__(self, dataset_dir, tfds_name, train_split, val_split, test_split, label_map, batch_size):\n",
    "        \"\"\"\n",
    "        Intialize the TFDSBinaryTextClassificationData class for a dataset binary text classification dataset\n",
    "        from the TensorFlow dataset catalog.\n",
    "        \n",
    "        :param dataset_dir: Path to a dataset directory to read/write data\n",
    "        :param tfds_name: String name of the TensorFlow dataset to load\n",
    "        :param train_split: String specifying which split to load for training (e.g. \"train[:80%]\"). See the\n",
    "                            https://www.tensorflow.org/datasets/splits documentation for more information on\n",
    "                            defining splits.\n",
    "        :param val_split: String specifying the split to load for validation.\n",
    "        :param test_split: String specifying the split to load for test.\n",
    "        :param label_map: Dictionary where the key is a numerical value and the value is the string label\n",
    "        :param batch_size: Batch size\n",
    "        \"\"\"\n",
    "        # Init base class\n",
    "        BinaryTextClassificationData.__init__(self, batch_size, label_map) \n",
    "        \n",
    "        [self.train_ds, self.val_ds, self.test_ds], info = tfds.load(tfds_name,\n",
    "                     data_dir=dataset_dir,\n",
    "                     split=[train_split, val_split, test_split],\n",
    "                     batch_size=batch_size,\n",
    "                     as_supervised=True,\n",
    "                     shuffle_files=True,\n",
    "                     with_info=True)\n",
    "        self.dataset_name = tfds_name\n",
    "        print(info)\n",
    "\n",
    "\n",
    "# Name of the TFDS to use\n",
    "tfds_name=\"imdb_reviews\"\n",
    "\n",
    "# Location where the dataset will be downloaded\n",
    "dataset_directory = os.path.join(dataset_directory, tfds_name)\n",
    "if not os.path.isdir(dataset_directory):\n",
    "    os.makedirs(dataset_directory)\n",
    "\n",
    "# Label map for sentiment analysis\n",
    "label_map = {\n",
    "    1: \"Positive\",\n",
    "    0: \"Negative\"\n",
    "}\n",
    "    \n",
    "# Initialize the dataset splits using a dataset from the TensorFlow datasets catalog\n",
    "dataset = TFDSBinaryTextClassificationData(dataset_dir=dataset_directory,\n",
    "                                           tfds_name=tfds_name,\n",
    "                                           train_split=\"train[:50%]\",\n",
    "                                           val_split=\"train[:20%]\",\n",
    "                                           test_split=\"test[:20%]\",\n",
    "                                           label_map=label_map,\n",
    "                                           batch_size=batch_size)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Skip to the next step [3. Build the model](#3.-Build-the-model) to continue using the TF dataset."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option B: Use a custom dataset\n",
    "\n",
    "Instead of using a dataset from TensorFlow datasets, another dataset from your local system or a download can be used. \n",
    "\n",
    "In this example, we download the [SMS Spam Collection dataset](https://archive.ics.uci.edu/ml/datasets/sms+spam+collection). (Note: Please see this dataset's applicable license for terms and conditions. Intel Corporation does not own the rights to this data set and does not confer any rights to it.) The zip file has a single tab-separated value file with two columns. The first column is the label (`ham` or `spam`) and the second column is the text of the SMS message:\n",
    "```\n",
    "<ham or spam>\t<text>\n",
    "<ham or spam>\t<text>\n",
    "<ham or spam>\t<text>\n",
    "...\n",
    "```\n",
    "If you are using a custom dataset that has a similarly formatted csv or tsv file, you can still use the class defined below. Just create your object passing in custom values for delimiter, header (whether the file has a header row), the label map, mapping function, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomCsvBinaryTextClassificationData(BinaryTextClassificationData):\n",
    "    def __init__(self, csv_file, delimiter, header, train_percent, val_percent,\n",
    "                 test_percent, label_map, batch_size, dataset_name, map_function=None):\n",
    "        \"\"\"\n",
    "        Intialize the CustomCsvBinaryTextClassificationData class for a dataset binary text\n",
    "        classification dataset that uses a single csv file.\n",
    "        \n",
    "        :param csv_file: Path to the csv file\n",
    "        :param delimiter: String character that separates the fields in each row\n",
    "        :param header: Boolean indicating whether or not the csv file has a header line that should be skipped\n",
    "        :param train_percent: Decimal value for the percentage of the dataset that should be used for training\n",
    "                              (e.g. 0.8 for 80%)\n",
    "        :param val_percent: Decimal value for the percentage of the dataset that should be used for validation\n",
    "                            (e.g. 0.1 for 10%)\n",
    "        :param test_percent: Decimal value for the percentage of the dataset that should be used for test\n",
    "                             (e.g. 0.1 for 10%)\n",
    "        :param label_map: Dictionary where the key is a numerical value and the value is the string label\n",
    "        :param batch_size: Batch size\n",
    "        :param dataset_name: Name of the dataset. This is used later in this notebook for naming the saved model\n",
    "                             export folder and determining which input strings to use when testing the reloaded model\n",
    "        :param map_function: (Optional) If the csv file has string labels instead of the numerical values, provide a\n",
    "                             map function to apply on the dataset\n",
    "        \"\"\"\n",
    "        # Init base class\n",
    "        BinaryTextClassificationData.__init__(self, batch_size, label_map)\n",
    "        \n",
    "        self.dataset_name = dataset_name\n",
    "        \n",
    "        if (train_percent + val_percent + test_percent) > 1:\n",
    "            raise ValueError(\"The combined value of the train percentage, validation percentage, and \" \\\n",
    "                             \"test percentage cannot be greater than 1\")\n",
    "        \n",
    "        if not os.path.exists(csv_file):\n",
    "            raise FileNotFoundError(\"Unable to find the csv file at\", csv_file)\n",
    "        \n",
    "        custom_dataset = tf.data.experimental.CsvDataset(filenames=csv_file,\n",
    "                                                         record_defaults=[tf.string, tf.string],\n",
    "                                                         field_delim=delimiter,\n",
    "                                                         use_quote_delim=False,\n",
    "                                                         header=header)\n",
    "        \n",
    "        # Count the number of lines in the csv file to get the dataset length\n",
    "        custom_dataset_len = sum(1 for line in open(csv_file))\n",
    "        \n",
    "        if header:\n",
    "            custom_dataset_len -= 1\n",
    "        \n",
    "        # Optionally map the dataset labels using the map_function\n",
    "        if map_function:\n",
    "            custom_dataset = custom_dataset.map(lambda x, y: (y, map_function(x)))\n",
    "        \n",
    "        # Create batches based on the specified batch size\n",
    "        custom_dataset = custom_dataset.batch(batch_size)\n",
    "        \n",
    "        # Calculate sizes for the splits\n",
    "        total_num_batches = int(custom_dataset_len / batch_size)\n",
    "        train_size = int(train_percent * total_num_batches)\n",
    "        val_size = int(val_percent * total_num_batches)\n",
    "        test_size = int(test_percent * total_num_batches)\n",
    "\n",
    "        # Create the train, validation, and test splits\n",
    "        self.train_ds = custom_dataset.take(train_size)    \n",
    "        self.val_ds = custom_dataset.skip(train_size).take(val_size)\n",
    "        self.test_ds = custom_dataset.skip(train_size).skip(val_size)\n",
    "\n",
    "        # Set the cardinality so that progress bars will work properly\n",
    "        self.train_ds = self.train_ds.apply(tf.data.experimental.assert_cardinality(train_size))\n",
    "        self.val_ds = self.val_ds.apply(tf.data.experimental.assert_cardinality(val_size))\n",
    "        self.test_ds = self.test_ds.apply(tf.data.experimental.assert_cardinality(test_size))\n",
    "\n",
    "\n",
    "# Modify the variables below to use a different dataset or a csv file on your local system.\n",
    "# The csv_path variable should be pointing to a csv file with 2 columns (the label and the text)\n",
    "dataset_url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
    "dataset_directory = os.path.join(dataset_directory, \"smsspamcollection\")\n",
    "csv_name = \"SMSSpamCollection\"\n",
    "delimiter = \"\\t\"\n",
    "header = False  # Set to true if the csv file has a header row\n",
    "csv_path = os.path.join(dataset_directory, csv_name)\n",
    "\n",
    "# If we don't already have the csv file, download and extract the zip file to get it.\n",
    "if not os.path.exists(csv_path):\n",
    "    download_and_extract_zip_file(dataset_url, dataset_directory)\n",
    "\n",
    "# Define the label map for your dataset. The label map below is for the SMS Spam Collection dataset.\n",
    "# The labels defined in this dictionary should match the labels in the csv file.\n",
    "label_map = {\n",
    "    1: \"spam\",\n",
    "    0: \"ham\"\n",
    "}\n",
    "\n",
    "# Map function to translate labels in the csv file to numerical values when loading the dataset\n",
    "def map_spam(x):\n",
    "    if x == \"spam\":\n",
    "        return 1\n",
    "    else:\n",
    "        return 0\n",
    "\n",
    "# Initialize the dataset splits using the custom dataset\n",
    "dataset = CustomCsvBinaryTextClassificationData(csv_file=csv_path,\n",
    "                                                delimiter=delimiter,\n",
    "                                                header=header,\n",
    "                                                train_percent=0.8,\n",
    "                                                val_percent=0.1,\n",
    "                                                test_percent=0.1,\n",
    "                                                label_map=label_map,\n",
    "                                                batch_size=batch_size,\n",
    "                                                dataset_name=csv_name,\n",
    "                                                map_function=map_spam)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Build the model\n",
    "\n",
    "Create the BERT model to fine tune using a input layer, the preprocessing layer (from TF Hub), the BERT encoder layer (from TF Hub), one dense layer, and a dropout layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_layer = tf.keras.layers.Input(shape=(), dtype=tf.string, name='input_layer')\n",
    "preprocessing_layer = hub.KerasLayer(tfhub_preprocess, name='preprocessing')\n",
    "encoder_inputs = preprocessing_layer(input_layer)\n",
    "encoder_layer = hub.KerasLayer(tfhub_bert_encoder, trainable=True, name='encoder')\n",
    "outputs = encoder_layer(encoder_inputs)\n",
    "net = outputs['pooled_output']\n",
    "net = tf.keras.layers.Dropout(0.1)(net)\n",
    "net = tf.keras.layers.Dense(1, activation=None, name='classifier')(net)\n",
    "classifier_model = tf.keras.Model(input_layer, net)\n",
    "\n",
    "classifier_model.summary()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Fine tuning and evaluation\n",
    "\n",
    "Train the model for the specified number of epochs, then evaluate the model using the test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# The number of training epochs to run\n",
    "num_train_epochs = 2\n",
    "\n",
    "# Learning rate\n",
    "learning_rate = 3e-5\n",
    "\n",
    "# Maximum total input sequence length after WordPiece tokenization (longer sequences will be truncated)\n",
    "max_seq_length = 128\n",
    "\n",
    "classifier_model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=learning_rate,epsilon=1e-08),\n",
    "                         loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n",
    "                         metrics=tf.metrics.BinaryAccuracy())\n",
    "\n",
    "history = classifier_model.fit(dataset.train_ds,\n",
    "                               validation_data=dataset.val_ds,\n",
    "                               epochs=num_train_epochs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the accuracy using the test dataset. If the accuracy does not meet your expectations, try to increasing the size of the training dataset split or the number of training epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss, accuracy = classifier_model.evaluate(dataset.test_ds)\n",
    "\n",
    "print(f'Loss: {loss}')\n",
    "print(f'Accuracy: {accuracy}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict using a single batch from the test dataset, and then display the results along with the input text and the actual label."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_steps = 1\n",
    "predictions = classifier_model.predict(dataset.test_ds, batch_size=batch_size, steps=num_steps)\n",
    "\n",
    "prediction_list = []\n",
    "step_count = 0\n",
    "\n",
    "for batch in dataset.test_ds:\n",
    "    label_list = list(batch[1].numpy())\n",
    "    text_list = list(batch[0].numpy())\n",
    "    \n",
    "    for i, (text, actual_label) in enumerate(zip(text_list, label_list)):\n",
    "        score = tf.math.sigmoid(predictions[i])\n",
    "        prediction = int(tf.math.round(score))\n",
    "        prediction = dataset.get_str_label(prediction)\n",
    "        prediction_list.append([text.decode('utf-8'),\n",
    "                                tf.get_static_value(score)[0],\n",
    "                                prediction,\n",
    "                                dataset.get_str_label(actual_label)])\n",
    "    \n",
    "    step_count += 1\n",
    "    if num_steps <= step_count:\n",
    "        break\n",
    "    \n",
    "result_df = pd.DataFrame(prediction_list, columns=[\"Input Text\", \"Score\", \"Predicted Label\", \"Actual Label\"])\n",
    "result_df.style.hide(axis=\"index\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Export the model\n",
    "\n",
    "Since training has completed, export the `saved_model.pb` to the output directory in a folder with the model and dataset name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_dir = \"{}_{}\".format(model_name, dataset.dataset_name)\n",
    "model_dir = os.path.join(output_directory, model_dir)\n",
    "classifier_model.save(model_dir, include_optimizer=False)\n",
    "\n",
    "saved_model_path = os.path.join(model_dir, \"saved_model.pb\")\n",
    "if os.path.exists(saved_model_path):\n",
    "    print(\"Saved model location:\", saved_model_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Reload the model and make predictions\n",
    "\n",
    "Reload from the `saved_model.pb` in the output directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reloaded_model = tf.saved_model.load(model_dir)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next section defines a list of strings to send as input to the reloaded model. If you are using a dataset other than the [IMDB movie reviews](https://www.tensorflow.org/datasets/catalog/imdb_reviews) or the [SMS Spam Collection](https://archive-beta.ics.uci.edu/ml/datasets/sms+spam+collection), you can update the snippet below with your own list of input text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if dataset.dataset_name == \"imdb_reviews\":\n",
    "    input_text = [\"Awesome movie\",\n",
    "                  \"It was entertaining, but completely predictable.\",\n",
    "                  \"Wasn't what I expected, but I still enjoyed it\",\n",
    "                  \"I wouldn't recommend this movie to my worst enemy\",\n",
    "                  \"I'm not sure how good the movie was, because I fell asleep\"]\n",
    "elif dataset.dataset_name == \"SMSSpamCollection\":\n",
    "    input_text = [\"Happy Birthday!\",\n",
    "                  \"Thank you for your order, please click the following link for tracking info 12345678\",\n",
    "                  \"Congratulations! You have won a free trip to Australia!!! Reply back with your full name and address.\",\n",
    "                  \"Can you get some milk while you're at the store?\",\n",
    "                  \"On my way\",\n",
    "                  \"OMG LOL :D\",\n",
    "                  \"Urgent! The IRS has been trying to contact you regarding your tax return. Please call 555-555-5555 immediately\"]\n",
    "else:\n",
    "    # Define your own list of input text for another dataset\n",
    "    input_text = []\n",
    "    \n",
    "if not input_text:\n",
    "    raise ValueError(\"Please define the list of input_text strings.\")\n",
    "\n",
    "# Send the input text to the reloaded model\n",
    "predict_results = tf.sigmoid(reloaded_model(tf.constant(input_text)))\n",
    "\n",
    "# Get the results into a data frame to display\n",
    "result_list = [[input_text[i],\n",
    "                tf.get_static_value(predict_results[i])[0],\n",
    "                dataset.get_str_label(tf.get_static_value(predict_results[i])[0])] for i in range(len(input_text))]\n",
    "result_df = pd.DataFrame(result_list, columns=[\"Input Text\", \"Score\", \"Predicted Label\"])\n",
    "result_df.style.hide(axis=\"index\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Citations\n",
    "\n",
    "```\n",
    "@InProceedings{maas-EtAl:2011:ACL-HLT2011,\n",
    "  author    = {Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher},\n",
    "  title     = {Learning Word Vectors for Sentiment Analysis},\n",
    "  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},\n",
    "  month     = {June},\n",
    "  year      = {2011},\n",
    "  address   = {Portland, Oregon, USA},\n",
    "  publisher = {Association for Computational Linguistics},\n",
    "  pages     = {142--150},\n",
    "  url       = {http://www.aclweb.org/anthology/P11-1015}\n",
    "}\n",
    "\n",
    "@misc{misc_sms_spam_collection_228,\n",
    "  author       = {Almeida, Tiago},\n",
    "  title        = {{SMS Spam Collection}},\n",
    "  year         = {2012},\n",
    "  howpublished = {UCI Machine Learning Repository}\n",
    "}\n",
    "```\n",
    "\n",
    "Please see this dataset's applicable license for terms and conditions. Intel Corporation does not own the rights to this data set and does not confer any rights to it."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}