code/topic-modeling-and-latent-dirichlet-allocation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classification of Security-Relevant Configuration Settings Using Topic Modeling and Latent Dirichlet Allocation\n",
    "\n",
    "This notebook is part of the paper *Automated Identification of Security-Relevant Configuration Settings Using NLP* submitted to the [**37<sup>th</sup> IEEE/ACM International Conference on Automated Software Engineering (ASE)**](https://conf.researchr.org/track/ase-2022/ase-2022-industry-showcase).\n",
    "\n",
    "The other notebooks can be found here\n",
    "\n",
    "- [Sentiment Analysis](https://www.kaggle.com/tumin4/sentiment-analysis/)\n",
    "- [Transformer-based Machine Learning](https://www.kaggle.com/tumin4/transformer-based-machine-learning)\n",
    "\n",
    "and on [GitHub](https://github.com/tum-i4/Automated-Identification-of-Security-Relevant-Configuration-Settings-Using-NLP/)\n",
    "\n",
    "## Contact\n",
    "\n",
    "If you have any questions, please contact [Patrick Stöckle](mailto:patrick.stoeckle@tum.de?subject=Kaggle%20Notebook%20%22Topic%20Modeling%20and%20Latent%20Dirichlet%20Allocation%22).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Acknowledgements\n",
    "2. Import libraries\n",
    "3. Load Data\n",
    "4. Prepare Data\n",
    "5. Generating security stop words\n",
    "6. Coherence evaluation\n",
    "6. Latent Dirichlet Allocation - Topic model generation\n",
    "7. Classification\n",
    "8. Evaluation\n",
    "9. Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Acknowledgements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create topic clusters from security settings with latent dirichlet allocation\n",
    "\n",
    "Reference: \n",
    "* https://towardsdatascience.com/topic-modelling-in-python-with-nltk-and-gensim-4ef03213cd21\n",
    "* https://towardsdatascience.com/topic-modeling-and-latent-dirichlet-allocation-in-python-9bf156893c24#:~:text=Latent%20Dirichlet%20Allocation%20(LDA)%20is,and%20split%20them%20into%20topics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:15:55.925819Z",
     "iopub.status.busy": "2022-04-25T11:15:55.925408Z",
     "iopub.status.idle": "2022-04-25T11:15:57.761638Z",
     "shell.execute_reply": "2022-04-25T11:15:57.760744Z",
     "shell.execute_reply.started": "2022-04-25T11:15:55.925747Z"
    }
   },
   "outputs": [],
   "source": [
    "from json import load\n",
    "import matplotlib.pyplot as plt\n",
    "from gensim.corpora import Dictionary\n",
    "from pandas import DataFrame\n",
    "import numpy as np\n",
    "from nltk.stem import SnowballStemmer, WordNetLemmatizer\n",
    "from nltk import Text, word_tokenize, FreqDist\n",
    "from random import shuffle\n",
    "from gensim import corpora, models\n",
    "import gensim\n",
    "import pyLDAvis.gensim\n",
    "import pickle\n",
    "from gensim import models\n",
    "from sklearn.metrics import precision_score, recall_score, f1_score, balanced_accuracy_score\n",
    "\n",
    "stop_w = gensim.parsing.preprocessing.STOPWORDS\n",
    "np.random.seed(2018)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:15:57.763827Z",
     "iopub.status.busy": "2022-04-25T11:15:57.763557Z",
     "iopub.status.idle": "2022-04-25T11:15:57.773503Z",
     "shell.execute_reply": "2022-04-25T11:15:57.772421Z",
     "shell.execute_reply.started": "2022-04-25T11:15:57.7638Z"
    }
   },
   "outputs": [],
   "source": [
    "def load_prepare_data(path, name):\n",
    "    \"\"\"\n",
    "    1. Load labeled configuration settings from ase2022 dataset\n",
    "    2. Remove Internet Explorer settings\n",
    "    3. Prepare dataframe for binary classification\n",
    "    \n",
    "    Args:\n",
    "    path: path to data\n",
    "    name: name of system variant\n",
    "    \n",
    "    Returns: \n",
    "    A dataframe of the system variant\n",
    "    \n",
    "    \"\"\"\n",
    "    with open(path) as f_read:\n",
    "        docs_all = load(f_read)\n",
    "    shuffle(docs_all)\n",
    "    df = DataFrame([doc for doc in docs_all if doc[\"text\"].lower().find(\"windows components. internet explorer.\")==-1])\n",
    "    df['sec']=df['is_security_relevant'].apply(lambda x: 1 if x==True else 0)\n",
    "    df = df.drop('is_security_relevant', 1)\n",
    "    print(f\"{name} settings: {len(df)}\")\n",
    "    return df.text.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:15:57.775596Z",
     "iopub.status.busy": "2022-04-25T11:15:57.775042Z",
     "iopub.status.idle": "2022-04-25T11:15:58.227647Z",
     "shell.execute_reply": "2022-04-25T11:15:58.226977Z",
     "shell.execute_reply.started": "2022-04-25T11:15:57.775558Z"
    }
   },
   "outputs": [],
   "source": [
    "docs: dict = {\"CIS_18_sec\": load_prepare_data(\"../input/ase2022/docs/cis/1803/sec_docs.json\",\"SEC CIS 1803\"),\n",
    "              \"CIS_18_non_sec\": load_prepare_data(\"../input/ase2022/docs/cis/1803/non_sec_docs.json\",\"NON-SEC CIS 1803\"),\n",
    "              \"CIS_19_sec\": load_prepare_data(\"../input/ase2022/docs/cis/1909/sec_docs.json\", \"SEC CIS 1909\"), \n",
    "              \"CIS_19_non_sec\": load_prepare_data(\"../input/ase2022/docs/cis/1909/non_sec_docs.json\", \"NON-SEC CIS 1909\"), \n",
    "              \"CIS_Server_sec\": load_prepare_data(\"../input/ase2022/docs/cis/server2016/sec_docs.json\", \"SEC CIS Server2016\" ), \n",
    "              \"CIS_Server_non_sec\": load_prepare_data(\"../input/ase2022/docs/cis/server2016/non_sec_docs.json\", \"NON-SEC CIS Server2016\" ), \n",
    "              \"SIE_19_sec\": load_prepare_data(\"../input/ase2022/docs/siemens/1909/sec_docs.json\", \"SEC Siemens 1909\"), \n",
    "              \"SIE_19_non_sec\": load_prepare_data(\"../input/ase2022/docs/siemens/1909/non_sec_docs.json\", \"NON-SEC Siemens 1909\"), \n",
    "              \"SIE_Server_sec\": load_prepare_data(\"../input/ase2022/docs/siemens/server2016/sec_docs.json\",\"SEC Siemens Server2016\"),\n",
    "              \"SIE_Server_non_sec\": load_prepare_data(\"../input/ase2022/docs/siemens/server2016/non_sec_docs.json\",\"NON-SEC Siemens Server2016\")}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:15:58.22892Z",
     "iopub.status.busy": "2022-04-25T11:15:58.228564Z",
     "iopub.status.idle": "2022-04-25T11:15:58.233701Z",
     "shell.execute_reply": "2022-04-25T11:15:58.232622Z",
     "shell.execute_reply.started": "2022-04-25T11:15:58.228894Z"
    }
   },
   "outputs": [],
   "source": [
    "def lemmatize_stemming(text):\n",
    "    \"\"\"\n",
    "    1. apply WordNetLemmatizer on text input\n",
    "    2. apply SnowballStemmer\n",
    "    \"\"\"\n",
    "    return SnowballStemmer(\"english\").stem(WordNetLemmatizer().lemmatize(text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:15:58.237732Z",
     "iopub.status.busy": "2022-04-25T11:15:58.237357Z",
     "iopub.status.idle": "2022-04-25T11:15:58.244849Z",
     "shell.execute_reply": "2022-04-25T11:15:58.243976Z",
     "shell.execute_reply.started": "2022-04-25T11:15:58.237693Z"
    }
   },
   "outputs": [],
   "source": [
    "def preprocess(text):\n",
    "    \"\"\"\n",
    "    1. apply simple_preprocess\n",
    "    2. remove stop words\n",
    "    3. keep words w of length 2 < w < 16\n",
    "    4. apply stemmer/lemmatizer\n",
    "    \"\"\"\n",
    "    result = [lemmatize_stemming(token) for token in gensim.utils.simple_preprocess(text) if (token not in gensim.parsing.preprocessing.STOPWORDS) and (len(token) > 2) and (len(token)<16)]\n",
    "    return [token for token in result if (token not in security_stop_words)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating security stop words:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Most common words in security vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:15:58.247247Z",
     "iopub.status.busy": "2022-04-25T11:15:58.246919Z",
     "iopub.status.idle": "2022-04-25T11:16:00.664674Z",
     "shell.execute_reply": "2022-04-25T11:16:00.663553Z",
     "shell.execute_reply.started": "2022-04-25T11:15:58.247222Z"
    }
   },
   "outputs": [],
   "source": [
    "sec_lemma_tokens= Text(lemmatize_stemming(token) for document in docs['CIS_19_sec'] for token in word_tokenize(document.lower()) if token.isalpha() and (token not in gensim.parsing.preprocessing.STOPWORDS))\n",
    "f_dist_sec_lemma_tokens = FreqDist(sec_lemma_tokens)\n",
    "most_common = f_dist_sec_lemma_tokens.most_common(300)\n",
    "most_common_words = [w for (w,c) in most_common]\n",
    "print(most_common_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Manual extraction of security stop words: frequent, but not security identifying words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:16:00.666805Z",
     "iopub.status.busy": "2022-04-25T11:16:00.666408Z",
     "iopub.status.idle": "2022-04-25T11:16:00.674731Z",
     "shell.execute_reply": "2022-04-25T11:16:00.674046Z",
     "shell.execute_reply.started": "2022-04-25T11:16:00.66676Z"
    }
   },
   "outputs": [],
   "source": [
    "security_stop_words={'abl', 'accept', 'activ', 'add', 'addit', 'affect', 'algorithm', 'allow', 'alreadi', 'also', 'alway', 'appear', 'avail', 'back', 'bar', 'behavior', 'box', 'charact', 'check', 'choos', 'client', 'client', 'com', 'command', 'comput', 'comput', 'configur', 'configur', 'control', 'creat', 'decid', 'default', 'desktop', 'determin', 'disabl', 'domain', 'edg', 'effect', 'employe', 'enabl', 'exampl', 'field', 'fix', 'follow', 'host', 'howev', 'includ', 'input', 'instead', 'kilobyt', 'let', 'local', 'longer', 'make', 'manag', 'may', 'megabyt', 'method', 'method', 'microsoft', 'ms', 'must', 'new', 'offer', 'one', 'oper', 'option', 'option', 'order', 'polici', 'polici', 'powershel', 'prevent', 'process', 'prompt', 'properti', 'public', 'rdp', 'reach', 'remov', 'requir', 'resum', 'select', 'set', 'set', 'specifi', 'state', 'still', 'system', 'take', 'task', 'temporari', 'terabyt', 'tool', 'tpm', 'turn', 'type', 'usb', 'use', 'user', 'user', 'valu', 'via', 'vista', 'want', 'whether', 'window', 'winrm', 'without', 'folder', 'devic', 'server', 'remot', 'password', 'data', 'featur'}\n",
    "print(security_stop_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Coherence evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:16:00.675989Z",
     "iopub.status.busy": "2022-04-25T11:16:00.675632Z",
     "iopub.status.idle": "2022-04-25T11:16:00.685625Z",
     "shell.execute_reply": "2022-04-25T11:16:00.68449Z",
     "shell.execute_reply.started": "2022-04-25T11:16:00.675964Z"
    }
   },
   "outputs": [],
   "source": [
    "# https://www.tutorialspoint.com/gensim/gensim_documents_and_lda_model.htm\n",
    "# https://towardsdatascience.com/evaluate-topic-model-in-python-latent-dirichlet-allocation-lda-7d57484bb5d0#:~:text=Topic%20Coherence%20measures%20score%20a,are%20artifacts%20of%20statistical%20inference.\n",
    "def coherence_values_computation(dictionary, corpus, texts, limit, start=2, step=3):\n",
    "    \"\"\"\n",
    "    build coherence models for different numbers of topics\n",
    "    \"\"\"\n",
    "    coherence_values = []\n",
    "    model_list = []\n",
    "    for num_topics in range(start, limit, step):\n",
    "        model = gensim.models.LdaModel(corpus, num_topics=num_topics, id2word=dictionary, passes=4, alpha='auto', per_word_topics=True)\n",
    "        model_list.append(model)\n",
    "        coherencemodel = gensim.models.coherencemodel.CoherenceModel(model=model, texts=texts, dictionary=dictionary)\n",
    "        coherence_values.append(coherencemodel.get_coherence())\n",
    "    return model_list, coherence_values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:16:00.687322Z",
     "iopub.status.busy": "2022-04-25T11:16:00.686806Z",
     "iopub.status.idle": "2022-04-25T11:16:13.616905Z",
     "shell.execute_reply": "2022-04-25T11:16:13.61604Z",
     "shell.execute_reply.started": "2022-04-25T11:16:00.687272Z"
    }
   },
   "outputs": [],
   "source": [
    "def coherence_evaluation(docs_input): \n",
    "    \"\"\"\n",
    "    evaluate and plot coherence values\n",
    "    \"\"\"\n",
    "    text_data = [preprocess(doc.lower()) for doc in docs_input]\n",
    "    dictionary = gensim.corpora.Dictionary.load('../input/ase2022/dictionary.gensim')\n",
    "    corpus = pickle.load(open('../input/ase2022/corpus.pkl', 'rb'))\n",
    "    tfidf = models.TfidfModel(corpus)\n",
    "    corpus_tfidf = tfidf[corpus]\n",
    "    model_list, coherence_values = coherence_values_computation (dictionary=dictionary, corpus=corpus_tfidf, texts=text_data, start=1, limit=30)\n",
    "    x = range(1, 30, 3)\n",
    "    plt.plot(x, coherence_values)\n",
    "    plt.xlabel(\"Num Topics\")\n",
    "    plt.ylabel(\"Coherence score\")\n",
    "    plt.legend(\"coherence_values\", loc='best')\n",
    "    plt.savefig('CoherenceEval.png', bbox_inches='tight')\n",
    "    plt.show()\n",
    "    for m, cv in zip(x, coherence_values):\n",
    "        print(\"Num Topics =\", m, \" is having Coherence Value of\", round(cv, 4))\n",
    "    return model_list\n",
    "coherence_evaluation(docs['CIS_19_sec'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Latent Dirichlet Allocation - Topic model generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:16:13.618603Z",
     "iopub.status.busy": "2022-04-25T11:16:13.618354Z",
     "iopub.status.idle": "2022-04-25T11:16:13.625396Z",
     "shell.execute_reply": "2022-04-25T11:16:13.624271Z",
     "shell.execute_reply.started": "2022-04-25T11:16:13.618578Z"
    }
   },
   "outputs": [],
   "source": [
    "def lda_topic_model_generation(docs_input):\n",
    "    \"\"\"\n",
    "    build lda topic model with 9 topics using TF-IDF model\n",
    "    \"\"\"\n",
    "    text_data = [preprocess(doc.lower()) for doc in docs_input]\n",
    "    dictionary = corpora.Dictionary(text_data)\n",
    "    corpus = [dictionary.doc2bow(text) for text in text_data]\n",
    "    pickle.dump(corpus, open('corpusNew.pkl', 'wb'))\n",
    "    dictionary.save('dictionaryNew.gensim')\n",
    "    tfidf = models.TfidfModel(corpus)\n",
    "    corpus_tfidf = tfidf[corpus]\n",
    "    n_topics = 9\n",
    "    ldamodel = gensim.models.LdaModel(corpus_tfidf, num_topics=n_topics, id2word=dictionary,passes=4, alpha='auto')\n",
    "    topics = ldamodel.print_topics(num_words=10)\n",
    "    for topic in topics:\n",
    "        print(topic)\n",
    "    ldamodel.save(f\"modelNew.gensim\")  \n",
    "\n",
    "    return ldamodel\n",
    "\n",
    "# Uncomment to train a new topic model\n",
    "# model = lda_topic_model_generation(docs['CIS_19_sec'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:16:13.6272Z",
     "iopub.status.busy": "2022-04-25T11:16:13.626876Z",
     "iopub.status.idle": "2022-04-25T11:16:13.638543Z",
     "shell.execute_reply": "2022-04-25T11:16:13.637591Z",
     "shell.execute_reply.started": "2022-04-25T11:16:13.627164Z"
    }
   },
   "outputs": [],
   "source": [
    "def classification(docs_input: [], model, threshold, trained_dict, tfidf): \n",
    "    \"\"\"\n",
    "    classify input documents: if topic probability for one of the topics is > threshold -> doc is sec-relevant\n",
    "    \"\"\"\n",
    "    sec_counter = []\n",
    "    for doc in docs_input:\n",
    "        sec_relevant = False\n",
    "        new_doc = preprocess(doc.lower())\n",
    "        new_doc_bow = trained_dict.doc2bow(new_doc)\n",
    "        doc_tfidf = tfidf[new_doc_bow]\n",
    "        res = model.get_document_topics(doc_tfidf)\n",
    "        for (t,r) in res:\n",
    "            if r>threshold:\n",
    "                sec_relevant = True\n",
    "        sec_counter.append(1) if sec_relevant else sec_counter.append(0)\n",
    "    return sec_counter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:16:13.641835Z",
     "iopub.status.busy": "2022-04-25T11:16:13.641574Z",
     "iopub.status.idle": "2022-04-25T11:16:13.650542Z",
     "shell.execute_reply": "2022-04-25T11:16:13.649685Z",
     "shell.execute_reply.started": "2022-04-25T11:16:13.641811Z"
    }
   },
   "outputs": [],
   "source": [
    "def evaluation(sec_docs: [], non_sec_docs: [],model, threshold, trained_dict):\n",
    "    \"\"\"\n",
    "    evaluate the model on the input data\n",
    "    \"\"\"\n",
    "    all_docs = sec_docs+non_sec_docs\n",
    "    text_data = [preprocess(doc.lower()) for doc in all_docs]\n",
    "    corpus = [trained_dict.doc2bow(text) for text in text_data]\n",
    "    tfidf = models.TfidfModel(corpus)\n",
    "    \n",
    "    test_y = [1 for i in sec_docs]+[0 for i in non_sec_docs]\n",
    "    y_pred_sec = classification(sec_docs, model, threshold, trained_dict, tfidf)\n",
    "    y_pred_non_sec = classification(non_sec_docs, model, threshold, trained_dict, tfidf)\n",
    "    y_pred=y_pred_sec+y_pred_non_sec\n",
    "    \n",
    "    prec = precision_score(test_y, y_pred, zero_division=0)\n",
    "    rec = recall_score(test_y, y_pred, zero_division=0)\n",
    "    f1 = f1_score(test_y,y_pred, zero_division=0)\n",
    "    bal_acc = balanced_accuracy_score(test_y,y_pred)\n",
    "    \n",
    "    print('Precision: {:4.2f}'.format(prec))\n",
    "    print('Recall: {:4.2f}'.format(rec))\n",
    "    print('F-Score: {:4.2f}'.format(f1))\n",
    "    print('Balanced accuracy: {:4.2f}'.format(bal_acc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:16:13.652235Z",
     "iopub.status.busy": "2022-04-25T11:16:13.651816Z",
     "iopub.status.idle": "2022-04-25T11:16:13.66188Z",
     "shell.execute_reply": "2022-04-25T11:16:13.661148Z",
     "shell.execute_reply.started": "2022-04-25T11:16:13.652205Z"
    }
   },
   "outputs": [],
   "source": [
    "def classify_and_evaluate(eval_sec, eval_non_sec, name):\n",
    "    \"\"\"\n",
    "    classify input docs with threshold of 0.7 and evaluate\n",
    "    \"\"\"\n",
    "    dictionary = gensim.corpora.Dictionary.load('../input/ase2022/dictionary.gensim')\n",
    "    ldamodel = gensim.models.ldamodel.LdaModel.load('../input/ase2022/model.gensim')\n",
    "    t=0.7\n",
    "    print(f\"\\nThreshold {t}:\")\n",
    "    print(name)\n",
    "    evaluation(eval_sec, eval_non_sec, ldamodel, t, dictionary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:16:13.663593Z",
     "iopub.status.busy": "2022-04-25T11:16:13.663066Z",
     "iopub.status.idle": "2022-04-25T11:17:05.622442Z",
     "shell.execute_reply": "2022-04-25T11:17:05.621505Z",
     "shell.execute_reply.started": "2022-04-25T11:16:13.663559Z"
    }
   },
   "outputs": [],
   "source": [
    "classify_and_evaluate(docs['CIS_18_sec'], docs['CIS_18_non_sec'], \"CIS 1803: \")\n",
    "classify_and_evaluate(docs['CIS_19_sec'], docs['CIS_19_non_sec'], \"CIS 1909: \")\n",
    "classify_and_evaluate(docs['SIE_19_sec'], docs['SIE_19_non_sec'], \"Siemens 1909: \")\n",
    "classify_and_evaluate(docs['CIS_Server_sec'], docs['CIS_Server_non_sec'],\"CIS Server 2016: \")\n",
    "classify_and_evaluate(docs['SIE_Server_sec'], docs['SIE_Server_non_sec'],\"Siemens Server 2016: \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Topic Visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-25T11:17:05.624153Z",
     "iopub.status.busy": "2022-04-25T11:17:05.623881Z",
     "iopub.status.idle": "2022-04-25T11:17:08.145547Z",
     "shell.execute_reply": "2022-04-25T11:17:08.144099Z",
     "shell.execute_reply.started": "2022-04-25T11:17:05.624127Z"
    }
   },
   "outputs": [],
   "source": [
    "dct = gensim.corpora.Dictionary.load('../input/ase2022/dictionary.gensim')\n",
    "corp = pickle.load(open('../input/ase2022/corpus.pkl', 'rb'))\n",
    "lda = gensim.models.ldamodel.LdaModel.load('../input/ase2022/model.gensim')\n",
    "lda_display = pyLDAvis.gensim.prepare(lda, corp, dct, sort_topics=False)\n",
    "pyLDAvis.display(lda_display)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}