topic_model_clustering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering documents using topic modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal in clustering is to group together similar documents creating a number of different groups. To judge the similarity of the documents it is common for each document to be represented by a vector of weights which are assigned to each word in the document. These weights in most cases are the <a src=https://en.wikipedia.org/wiki/Tf%E2%80%93idf>tf-idf</a> frequencies of the words. Thus, an NxM dimensional matrix is created where N is the number of documents and M the dimensions of the vector space (number of words in the dictionary of the document collection). The end result of clustering is a number of clusters with each document assigned to a cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another method of organizing a document collection is topic modeling. In topic modeling each word of the dictionary is associated with a probability of occurence in a topic and each topic is associated with a probability of occurence in a document. Each document is represented by a vector of the probabilities of each topic in it. This means that topic modeling results to a NxM matrix representation of the collection where N is the number of documents and M is the number of topics. Since we have a vector space representation of the documents we can use it for clustering. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topic modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use <a src=http://radimrehurek.com/gensim/>Gensim</a> for topic modeling as it offers time and space efficient implementations of the topic modeling algorithms and it's free. It also supports parallelization although I don't make us of this feature here. With Gensim you can use either Latent Semantic Analysis (LSA) or Latent Dirichlet Allocation (LDA) to produce the topic vector space described above. We won't get into how these work now but you can find more information <a src=https://en.wikipedia.org/wiki/Latent_semantic_analysis>here</a> and <a src=https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation>here</a>. In this implementation i will use LSA as it is faster that LDA. It is also less accurate but the goal of this notebook is a proof of concept. \n",
    "\n",
    "The data we are going to use are greek WikiPedia articles xml formatted and bz2 compressed. You can download them from <a src=https://dumps.wikimedia.org/elwiki/20170601/elwiki-20170601-pages-meta-history.xml.bz2>here</a>. Gensim can work directly on the compressed collection.\n",
    "\n",
    "First we need to create a Dictionary object, which is the dictionary of the document collection and associates each word to an id, and a MmCorpus object, which stores the corpus in a <a src=http://math.nist.gov/MatrixMarket/formats.html>Matrix Market</a> format. In a MmCorpus object each document can be represented by a vector of either integer count frequencies of it's word or tf-idf frequencies, but either way the resulting matrix is stored in the Matrix Market format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I have saved the compressed WikiPedia XML file in a directory called data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-08-01 14:55:56,120 : INFO : running /home/theovasi/anaconda3/lib/python3.6/site-packages/gensim/scripts/make_wikicorpus.py data/elwiki-20170720-pages-articles.xml.bz2 data/grwiki\n",
      "2017-08-01 14:55:56,323 : INFO : adding document #0 to Dictionary(0 unique tokens: [])\n",
      "2017-08-01 14:56:25,444 : INFO : adding document #10000 to Dictionary(361468 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...)\n",
      "2017-08-01 14:56:46,820 : INFO : adding document #20000 to Dictionary(500714 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...)\n",
      "2017-08-01 14:57:05,578 : INFO : adding document #30000 to Dictionary(598622 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...)\n",
      "2017-08-01 14:57:24,443 : INFO : adding document #40000 to Dictionary(686151 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...)\n",
      "2017-08-01 14:57:42,056 : INFO : adding document #50000 to Dictionary(757946 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...)\n",
      "2017-08-01 14:58:01,081 : INFO : adding document #60000 to Dictionary(834243 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...)\n",
      "2017-08-01 14:58:18,331 : INFO : adding document #70000 to Dictionary(903189 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...)\n",
      "2017-08-01 14:58:38,320 : INFO : adding document #80000 to Dictionary(968007 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...)\n",
      "2017-08-01 14:58:56,657 : INFO : adding document #90000 to Dictionary(1032638 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...)\n",
      "2017-08-01 14:59:16,328 : INFO : adding document #100000 to Dictionary(1099346 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...)\n",
      "2017-08-01 14:59:36,327 : INFO : adding document #110000 to Dictionary(1158563 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...)\n",
      "2017-08-01 14:59:55,921 : INFO : adding document #120000 to Dictionary(1216908 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...)\n",
      "2017-08-01 15:00:01,163 : INFO : finished iterating over Wikipedia corpus of 122813 documents with 59080781 positions (total 282747 articles, 59624135 positions before pruning articles shorter than 50 words)\n",
      "2017-08-01 15:00:01,200 : INFO : built Dictionary(1252353 unique tokens: ['αθλητισμός', 'είναι', 'συστηματική', 'σωματική', 'καλλιέργεια']...) from 122813 documents (total 59080781 corpus positions)\n",
      "2017-08-01 15:00:02,988 : INFO : discarding 1153232 tokens: [('είναι', 89144), ('και', 118661), ('με', 104775), ('την', 104476), ('ως', 65718), ('σε', 88015), ('στο', 90990), ('ένας', 28946), ('οποίος', 24450), ('τη', 73154)]...\n",
      "2017-08-01 15:00:02,991 : INFO : keeping 99121 tokens which were in no less than 20 and no more than 12281 (=10.0%) documents\n",
      "2017-08-01 15:00:03,409 : INFO : resulting dictionary: Dictionary(99121 unique tokens: ['αθλητισμός', 'συστηματική', 'σωματική', 'καλλιέργεια', 'δράση']...)\n",
      "2017-08-01 15:00:03,511 : INFO : storing corpus in Matrix Market format to data/grwiki_bow.mm\n",
      "2017-08-01 15:00:03,544 : INFO : saving sparse matrix to data/grwiki_bow.mm\n",
      "2017-08-01 15:00:03,720 : INFO : PROGRESS: saving document #0\n",
      "2017-08-01 15:00:35,520 : INFO : PROGRESS: saving document #10000\n",
      "2017-08-01 15:00:59,077 : INFO : PROGRESS: saving document #20000\n",
      "2017-08-01 15:01:19,233 : INFO : PROGRESS: saving document #30000\n",
      "2017-08-01 15:01:39,448 : INFO : PROGRESS: saving document #40000\n",
      "2017-08-01 15:01:58,664 : INFO : PROGRESS: saving document #50000\n",
      "2017-08-01 15:02:19,289 : INFO : PROGRESS: saving document #60000\n",
      "2017-08-01 15:02:39,004 : INFO : PROGRESS: saving document #70000\n",
      "2017-08-01 15:03:04,299 : INFO : PROGRESS: saving document #80000\n",
      "2017-08-01 15:03:24,200 : INFO : PROGRESS: saving document #90000\n",
      "2017-08-01 15:03:46,027 : INFO : PROGRESS: saving document #100000\n",
      "2017-08-01 15:04:07,752 : INFO : PROGRESS: saving document #110000\n",
      "2017-08-01 15:04:29,058 : INFO : PROGRESS: saving document #120000\n",
      "2017-08-01 15:04:34,727 : INFO : finished iterating over Wikipedia corpus of 122813 documents with 59080781 positions (total 282747 articles, 59624135 positions before pruning articles shorter than 50 words)\n",
      "2017-08-01 15:04:34,765 : INFO : saved 122813x99121 matrix, density=0.172% (20926208/12173347373)\n",
      "2017-08-01 15:04:34,769 : INFO : saving MmCorpus index to data/grwiki_bow.mm.index\n",
      "2017-08-01 15:04:34,777 : INFO : saving dictionary mapping to data/grwiki_wordids.txt.bz2\n",
      "2017-08-01 15:04:36,774 : INFO : loaded corpus index from data/grwiki_bow.mm.index\n",
      "2017-08-01 15:04:36,775 : INFO : initializing corpus reader from data/grwiki_bow.mm\n",
      "2017-08-01 15:04:36,779 : INFO : accepted corpus with 122813 documents, 99121 features, 20926208 non-zero entries\n",
      "2017-08-01 15:04:36,783 : INFO : collecting document frequencies\n",
      "2017-08-01 15:04:36,789 : INFO : PROGRESS: processing document #0\n",
      "2017-08-01 15:04:48,723 : INFO : PROGRESS: processing document #10000\n",
      "2017-08-01 15:04:56,872 : INFO : PROGRESS: processing document #20000\n",
      "2017-08-01 15:05:04,013 : INFO : PROGRESS: processing document #30000\n",
      "2017-08-01 15:05:10,662 : INFO : PROGRESS: processing document #40000\n",
      "2017-08-01 15:05:16,828 : INFO : PROGRESS: processing document #50000\n",
      "2017-08-01 15:05:23,696 : INFO : PROGRESS: processing document #60000\n",
      "2017-08-01 15:05:29,950 : INFO : PROGRESS: processing document #70000\n",
      "2017-08-01 15:05:36,434 : INFO : PROGRESS: processing document #80000\n",
      "2017-08-01 15:05:42,250 : INFO : PROGRESS: processing document #90000\n",
      "2017-08-01 15:05:48,638 : INFO : PROGRESS: processing document #100000\n",
      "2017-08-01 15:05:54,826 : INFO : PROGRESS: processing document #110000\n",
      "2017-08-01 15:06:00,951 : INFO : PROGRESS: processing document #120000\n",
      "2017-08-01 15:06:02,641 : INFO : calculating IDF weights for 122813 documents and 99120 features (20926208 matrix non-zeros)\n",
      "2017-08-01 15:06:02,711 : INFO : saving TfidfModel object under data/grwiki.tfidf_model, separately None\n",
      "2017-08-01 15:06:02,774 : INFO : saved data/grwiki.tfidf_model\n",
      "2017-08-01 15:06:02,775 : INFO : storing corpus in Matrix Market format to data/grwiki_tfidf.mm\n",
      "2017-08-01 15:06:02,845 : INFO : saving sparse matrix to data/grwiki_tfidf.mm\n",
      "2017-08-01 15:06:02,849 : INFO : PROGRESS: saving document #0\n",
      "2017-08-01 15:06:27,633 : INFO : PROGRESS: saving document #10000\n",
      "2017-08-01 15:06:44,348 : INFO : PROGRESS: saving document #20000\n",
      "2017-08-01 15:07:00,110 : INFO : PROGRESS: saving document #30000\n",
      "2017-08-01 15:07:14,022 : INFO : PROGRESS: saving document #40000\n",
      "2017-08-01 15:07:26,996 : INFO : PROGRESS: saving document #50000\n",
      "2017-08-01 15:07:41,036 : INFO : PROGRESS: saving document #60000\n",
      "2017-08-01 15:07:54,004 : INFO : PROGRESS: saving document #70000\n",
      "2017-08-01 15:08:08,622 : INFO : PROGRESS: saving document #80000\n",
      "2017-08-01 15:08:20,712 : INFO : PROGRESS: saving document #90000\n",
      "2017-08-01 15:08:34,029 : INFO : PROGRESS: saving document #100000\n",
      "2017-08-01 15:08:46,987 : INFO : PROGRESS: saving document #110000\n",
      "2017-08-01 15:08:59,666 : INFO : PROGRESS: saving document #120000\n",
      "2017-08-01 15:09:03,165 : INFO : saved 122813x99121 matrix, density=0.172% (20926208/12173347373)\n",
      "2017-08-01 15:09:03,166 : INFO : saving MmCorpus index to data/grwiki_tfidf.mm.index\n",
      "2017-08-01 15:09:03,174 : INFO : finished running make_wikicorpus.py\n"
     ]
    }
   ],
   "source": [
    "%run -m gensim.scripts.make_wikicorpus data/elwiki-20170720-pages-articles.xml.bz2 data/grwiki"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above script takes the compressed XML file as its first argument. The second argument is the prefix of the output files. When run this script creates the dictionary and saves it in /data/grwiki_wordids.txt.bz2 and the tf-idf representation of the corpus saved in Matrix Market format in /data/grwiki_tfidf.mm. Now lets load these files and create the needed objects."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First unzip the dictionary file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!bzip2 -dk data/grwiki_wordids.txt.bz2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create the Dictionary and MmCorpus objects by loading the files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-08-03 11:41:02,035 : INFO : loaded corpus index from ../data/grwiki_tfidf.mm.index\n",
      "2017-08-03 11:41:02,036 : INFO : initializing corpus reader from ../data/grwiki_tfidf.mm\n",
      "2017-08-03 11:41:02,037 : INFO : accepted corpus with 122813 documents, 99121 features, 20926208 non-zero entries\n"
     ]
    }
   ],
   "source": [
    "import logging, gensim, bz2\n",
    "logging.basicConfig(\n",
    "    format='%(asctime)s : %(levelname)s : %(message)s',\n",
    "    level=logging.INFO) # Allow gensim to print additional info.\n",
    "\n",
    "dictionary = gensim.corpora.Dictionary.load_from_text('../data/grwiki_wordids.txt')\n",
    "corpus = gensim.corpora.MmCorpus('../data/grwiki_tfidf.mm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the LSA of the Greek WikiPedia:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-08-03 11:41:06,631 : INFO : using serial LSI version on this node\n",
      "2017-08-03 11:41:06,633 : INFO : updating model with new documents\n",
      "2017-08-03 11:41:30,159 : INFO : preparing a new chunk of documents\n",
      "2017-08-03 11:41:31,077 : INFO : using 100 extra samples and 2 power iterations\n",
      "2017-08-03 11:41:31,078 : INFO : 1st phase: constructing (99121, 200) action matrix\n",
      "2017-08-03 11:41:32,597 : INFO : orthonormalizing (99121, 200) action matrix\n",
      "2017-08-03 11:41:48,634 : INFO : 2nd phase: running dense svd on (200, 20000) matrix\n",
      "2017-08-03 11:41:49,800 : INFO : computing the final decomposition\n",
      "2017-08-03 11:41:49,801 : INFO : keeping 100 factors (discarding 22.315% of energy spectrum)\n",
      "2017-08-03 11:41:50,003 : INFO : processed documents up to #20000\n",
      "2017-08-03 11:41:50,031 : INFO : topic #0(14.900): 0.274*\"έλληνας\" + 0.217*\"αμερικανός\" + 0.199*\"πολιτικός\" + 0.188*\"ηθοποιός\" + 0.145*\"συγγραφέας\" + 0.134*\"βασιλιάς\" + 0.131*\"γεννήσεις\" + 0.129*\"θάνατοι\" + 0.128*\"γρηγοριανό\" + 0.127*\"hμερολόγιο\"\n",
      "2017-08-03 11:41:50,035 : INFO : topic #1(11.223): -0.169*\"έλληνας\" + -0.161*\"αμερικανός\" + -0.128*\"ηθοποιός\" + -0.120*\"πολιτικός\" + -0.101*\"hμερολόγιο\" + -0.099*\"γρηγοριανό\" + -0.099*\"γεννήσεις\" + -0.097*\"θάνατοι\" + -0.085*\"συγγραφέας\" + -0.085*\"γάλλος\"\n",
      "2017-08-03 11:41:50,040 : INFO : topic #2(9.450): -0.246*\"έλληνας\" + -0.245*\"αμερικανός\" + 0.244*\"γεννήσεις\" + 0.236*\"θάνατοι\" + 0.230*\"hμερολόγιο\" + 0.228*\"τρέχουσα\" + 0.227*\"γρηγοριανό\" + -0.224*\"ηθοποιός\" + 0.192*\"σελίδα\" + 0.181*\"βασιλιάς\"\n",
      "2017-08-03 11:41:50,044 : INFO : topic #3(8.749): -0.551*\"ποδοσφαιριστές\" + -0.185*\"εθνική\" + -0.149*\"εθνικής\" + -0.147*\"πρωτάθλημα\" + -0.139*\"αεκ\" + -0.137*\"ελλάδος\" + -0.129*\"ελλάδας\" + -0.124*\"ολυμπιακού\" + -0.121*\"αγωνίστηκε\" + -0.115*\"ολυμπιακός\"\n",
      "2017-08-03 11:41:50,046 : INFO : topic #4(7.665): 0.399*\"δήμος\" + 0.206*\"δήμο\" + 0.202*\"κατοίκους\" + 0.197*\"χωριό\" + 0.182*\"νομού\" + 0.175*\"δήμου\" + 0.156*\"άγιος\" + 0.154*\"πληθυσμό\" + 0.121*\"χωριού\" + 0.110*\"απογραφή\"\n",
      "2017-08-03 11:42:06,236 : INFO : preparing a new chunk of documents\n",
      "2017-08-03 11:42:06,909 : INFO : using 100 extra samples and 2 power iterations\n",
      "2017-08-03 11:42:06,912 : INFO : 1st phase: constructing (99121, 200) action matrix\n",
      "2017-08-03 11:42:08,052 : INFO : orthonormalizing (99121, 200) action matrix\n",
      "2017-08-03 11:42:20,704 : INFO : 2nd phase: running dense svd on (200, 20000) matrix\n",
      "2017-08-03 11:42:21,816 : INFO : computing the final decomposition\n",
      "2017-08-03 11:42:21,817 : INFO : keeping 100 factors (discarding 20.707% of energy spectrum)\n",
      "2017-08-03 11:42:22,023 : INFO : merging projections: (99121, 100) + (99121, 100)\n",
      "2017-08-03 11:42:24,058 : INFO : keeping 100 factors (discarding 13.140% of energy spectrum)\n",
      "2017-08-03 11:42:24,320 : INFO : processed documents up to #40000\n",
      "2017-08-03 11:42:24,327 : INFO : topic #0(17.919): 0.161*\"ποδοσφαιριστές\" + 0.136*\"έλληνας\" + 0.109*\"δήμος\" + 0.101*\"πολιτικός\" + 0.095*\"αμερικανός\" + 0.088*\"ηθοποιός\" + 0.086*\"βασιλιάς\" + 0.085*\"χωριό\" + 0.079*\"px\" + 0.075*\"εθνική\"\n",
      "2017-08-03 11:42:24,334 : INFO : topic #1(14.464): -0.467*\"ποδοσφαιριστές\" + 0.285*\"δήμος\" + 0.173*\"δημοτικά\" + 0.171*\"διαμερίσματα\" + 0.164*\"νομού\" + 0.160*\"δήμο\" + 0.137*\"δήμου\" + 0.123*\"χωριό\" + 0.120*\"κατοίκους\" + -0.115*\"εθνική\"\n",
      "2017-08-03 11:42:24,340 : INFO : topic #2(14.086): 0.390*\"ποδοσφαιριστές\" + 0.260*\"δήμος\" + 0.160*\"δημοτικά\" + 0.158*\"διαμερίσματα\" + 0.143*\"νομού\" + 0.142*\"δήμο\" + -0.128*\"μυθολογίας\" + 0.119*\"δήμου\" + -0.114*\"patsi\" + -0.114*\"garin\"\n",
      "2017-08-03 11:42:24,345 : INFO : topic #3(13.793): 0.263*\"μυθολογίας\" + 0.236*\"patsi\" + 0.236*\"garin\" + 0.235*\"επίτομο\" + 0.230*\"ποδοσφαιριστές\" + 0.229*\"emmy\" + 0.226*\"πάτση\" + 0.213*\"μυθολογία\" + 0.188*\"λεξικό\" + 0.175*\"οίκος\"\n",
      "2017-08-03 11:42:24,349 : INFO : topic #4(12.269): -0.219*\"δήμος\" + -0.192*\"ποδοσφαιριστές\" + -0.186*\"έλληνας\" + -0.162*\"δημοτικά\" + -0.160*\"διαμερίσματα\" + -0.156*\"αμερικανός\" + -0.126*\"πολιτικός\" + -0.125*\"ηθοποιός\" + -0.110*\"μυθολογίας\" + -0.110*\"patsi\"\n",
      "2017-08-03 11:42:39,374 : INFO : preparing a new chunk of documents\n",
      "2017-08-03 11:42:40,011 : INFO : using 100 extra samples and 2 power iterations\n",
      "2017-08-03 11:42:40,013 : INFO : 1st phase: constructing (99121, 200) action matrix\n",
      "2017-08-03 11:42:41,097 : INFO : orthonormalizing (99121, 200) action matrix\n",
      "2017-08-03 11:42:53,695 : INFO : 2nd phase: running dense svd on (200, 20000) matrix\n",
      "2017-08-03 11:42:54,611 : INFO : computing the final decomposition\n",
      "2017-08-03 11:42:54,613 : INFO : keeping 100 factors (discarding 19.959% of energy spectrum)\n",
      "2017-08-03 11:42:54,785 : INFO : merging projections: (99121, 100) + (99121, 100)\n",
      "2017-08-03 11:42:56,725 : INFO : keeping 100 factors (discarding 13.409% of energy spectrum)\n",
      "2017-08-03 11:42:56,980 : INFO : processed documents up to #60000\n",
      "2017-08-03 11:42:56,986 : INFO : topic #0(22.623): 0.209*\"εκτιμάται\" + 0.201*\"μέση\" + 0.194*\"αστεροειδών\" + 0.186*\"φασματικός\" + 0.184*\"παρατηρούσε\" + 0.183*\"άλβεδό\" + 0.181*\"jpl\" + 0.181*\"αστεροειδής\" + 0.180*\"τροχιά\" + 0.180*\"διάμετρος\"\n",
      "2017-08-03 11:42:57,000 : INFO : topic #1(20.653): 0.220*\"ποδοσφαιριστές\" + 0.106*\"έλληνας\" + 0.098*\"δήμος\" + 0.098*\"εθνική\" + 0.096*\"px\" + 0.091*\"χωριό\" + 0.082*\"πρωτάθλημα\" + 0.076*\"νομού\" + 0.074*\"εθνικής\" + 0.074*\"πολιτικός\"\n",
      "2017-08-03 11:42:57,008 : INFO : topic #2(17.180): -0.621*\"ποδοσφαιριστές\" + 0.148*\"δήμος\" + -0.146*\"εθνική\" + -0.126*\"πρωτάθλημα\" + -0.121*\"αγωνίστηκε\" + -0.120*\"κύπελλο\" + -0.119*\"γκολ\" + -0.114*\"εθνικής\" + 0.107*\"χωριό\" + 0.106*\"νομού\"\n",
      "2017-08-03 11:42:57,013 : INFO : topic #3(15.666): 0.335*\"δήμος\" + 0.219*\"νομού\" + 0.209*\"ποδοσφαιριστές\" + 0.189*\"δήμο\" + 0.182*\"δήμου\" + 0.173*\"χωριό\" + 0.168*\"διαμερίσματα\" + 0.166*\"δημοτικά\" + 0.142*\"κατοίκους\" + -0.131*\"έλληνας\"\n",
      "2017-08-03 11:42:57,016 : INFO : topic #4(14.062): -0.268*\"μυθολογίας\" + -0.237*\"patsi\" + -0.237*\"garin\" + -0.235*\"επίτομο\" + -0.232*\"μυθολογία\" + -0.229*\"emmy\" + -0.227*\"πάτση\" + -0.191*\"λεξικό\" + -0.178*\"οίκος\" + -0.171*\"ελληνικής\"\n",
      "2017-08-03 11:43:11,777 : INFO : preparing a new chunk of documents\n",
      "2017-08-03 11:43:12,435 : INFO : using 100 extra samples and 2 power iterations\n",
      "2017-08-03 11:43:12,436 : INFO : 1st phase: constructing (99121, 200) action matrix\n",
      "2017-08-03 11:43:13,656 : INFO : orthonormalizing (99121, 200) action matrix\n",
      "2017-08-03 11:43:26,443 : INFO : 2nd phase: running dense svd on (200, 20000) matrix\n",
      "2017-08-03 11:43:27,361 : INFO : computing the final decomposition\n",
      "2017-08-03 11:43:27,362 : INFO : keeping 100 factors (discarding 20.790% of energy spectrum)\n",
      "2017-08-03 11:43:27,535 : INFO : merging projections: (99121, 100) + (99121, 100)\n",
      "2017-08-03 11:43:29,718 : INFO : keeping 100 factors (discarding 13.019% of energy spectrum)\n",
      "2017-08-03 11:43:30,002 : INFO : processed documents up to #80000\n",
      "2017-08-03 11:43:30,005 : INFO : topic #0(30.987): 0.231*\"φυσικά\" + 0.217*\"αστεροειδών\" + 0.212*\"jpl\" + 0.209*\"κύριας\" + 0.209*\"java\" + 0.207*\"αστεροειδής\" + 0.207*\"τροχιά\" + 0.204*\"ηλιακό\" + 0.203*\"απόλυτο\" + 0.201*\"παρατηρούσε\"\n",
      "2017-08-03 11:43:30,009 : INFO : topic #1(23.591): 0.348*\"ποδοσφαιριστές\" + 0.138*\"px\" + 0.125*\"εθνική\" + 0.116*\"πρωτάθλημα\" + 0.098*\"εθνικής\" + 0.088*\"έλληνας\" + 0.083*\"κύπελλο\" + 0.079*\"αγώνες\" + 0.075*\"ελλάδας\" + 0.074*\"χωριό\"\n",
      "2017-08-03 11:43:30,013 : INFO : topic #2(20.043): -0.644*\"ποδοσφαιριστές\" + -0.113*\"εθνική\" + -0.103*\"γκολ\" + -0.100*\"εθνικής\" + -0.097*\"αγωνίστηκε\" + 0.097*\"δήμος\" + 0.095*\"χωριό\" + -0.095*\"πρωτάθλημα\" + -0.085*\"κύπελλο\" + -0.085*\"καριέρα\"\n",
      "2017-08-03 11:43:30,017 : INFO : topic #3(16.724): -0.304*\"δήμος\" + -0.260*\"ποδοσφαιριστές\" + 0.242*\"px\" + -0.219*\"νομού\" + -0.210*\"χωριό\" + -0.185*\"δήμο\" + -0.177*\"δήμου\" + -0.158*\"κατοίκους\" + -0.145*\"κοινότητα\" + -0.140*\"διαμερίσματα\"\n",
      "2017-08-03 11:43:30,022 : INFO : topic #4(16.404): 0.582*\"px\" + -0.295*\"ποδοσφαιριστές\" + 0.146*\"πρωτάθλημα\" + 0.139*\"παναθηναϊκός\" + 0.136*\"ολυμπιακός\" + 0.113*\"πανελλήνιος\" + 0.101*\"δήμος\" + 0.093*\"αγώνες\" + 0.087*\"πανιώνιος\" + 0.087*\"αεκ\"\n",
      "2017-08-03 11:43:44,271 : INFO : preparing a new chunk of documents\n",
      "2017-08-03 11:43:44,896 : INFO : using 100 extra samples and 2 power iterations\n",
      "2017-08-03 11:43:44,898 : INFO : 1st phase: constructing (99121, 200) action matrix\n",
      "2017-08-03 11:43:45,947 : INFO : orthonormalizing (99121, 200) action matrix\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-08-03 11:43:59,315 : INFO : 2nd phase: running dense svd on (200, 20000) matrix\n",
      "2017-08-03 11:44:00,288 : INFO : computing the final decomposition\n",
      "2017-08-03 11:44:00,289 : INFO : keeping 100 factors (discarding 22.928% of energy spectrum)\n",
      "2017-08-03 11:44:00,476 : INFO : merging projections: (99121, 100) + (99121, 100)\n",
      "2017-08-03 11:44:02,576 : INFO : keeping 100 factors (discarding 12.315% of energy spectrum)\n",
      "2017-08-03 11:44:02,999 : INFO : processed documents up to #100000\n",
      "2017-08-03 11:44:03,005 : INFO : topic #0(31.376): 0.232*\"φυσικά\" + 0.217*\"αστεροειδών\" + 0.212*\"jpl\" + 0.209*\"κύριας\" + 0.209*\"java\" + 0.207*\"αστεροειδής\" + 0.207*\"τροχιά\" + 0.203*\"ηλιακό\" + 0.202*\"απόλυτο\" + 0.201*\"ζώνης\"\n",
      "2017-08-03 11:44:03,009 : INFO : topic #1(26.170): 0.360*\"ποδοσφαιριστές\" + 0.138*\"εθνική\" + 0.135*\"πρωτάθλημα\" + 0.119*\"px\" + 0.106*\"κύπελλο\" + 0.102*\"εθνικής\" + 0.096*\"ποδοσφαίρου\" + 0.081*\"αγώνες\" + 0.080*\"έλληνας\" + 0.079*\"ομάδες\"\n",
      "2017-08-03 11:44:03,012 : INFO : topic #2(22.064): -0.590*\"ποδοσφαιριστές\" + -0.112*\"εθνική\" + -0.106*\"πρωτάθλημα\" + -0.101*\"κύπελλο\" + -0.095*\"γκολ\" + -0.094*\"ποδοσφαίρου\" + 0.094*\"χωριό\" + -0.092*\"αγωνίστηκε\" + -0.090*\"εθνικής\" + 0.082*\"δήμος\"\n",
      "2017-08-03 11:44:03,016 : INFO : topic #3(17.997): 0.266*\"δήμος\" + 0.233*\"χωριό\" + 0.217*\"νομού\" + 0.183*\"ποδοσφαιριστές\" + 0.183*\"δήμο\" + 0.169*\"δήμου\" + 0.161*\"κατοίκους\" + 0.148*\"κοινότητα\" + -0.144*\"κόμμα\" + 0.139*\"απογραφή\"\n",
      "2017-08-03 11:44:03,019 : INFO : topic #4(17.692): 0.500*\"px\" + -0.434*\"ποδοσφαιριστές\" + 0.191*\"πρωτάθλημα\" + 0.130*\"κύπελλο\" + 0.127*\"ολυμπιακός\" + 0.121*\"παναθηναϊκός\" + 0.119*\"ομάδες\" + 0.103*\"αγώνες\" + 0.100*\"ανδρών\" + 0.098*\"πανελλήνιος\"\n",
      "2017-08-03 11:44:17,564 : INFO : preparing a new chunk of documents\n",
      "2017-08-03 11:44:18,167 : INFO : using 100 extra samples and 2 power iterations\n",
      "2017-08-03 11:44:18,168 : INFO : 1st phase: constructing (99121, 200) action matrix\n",
      "2017-08-03 11:44:19,242 : INFO : orthonormalizing (99121, 200) action matrix\n",
      "2017-08-03 11:44:32,532 : INFO : 2nd phase: running dense svd on (200, 20000) matrix\n",
      "2017-08-03 11:44:33,892 : INFO : computing the final decomposition\n",
      "2017-08-03 11:44:33,893 : INFO : keeping 100 factors (discarding 21.281% of energy spectrum)\n",
      "2017-08-03 11:44:34,142 : INFO : merging projections: (99121, 100) + (99121, 100)\n",
      "2017-08-03 11:44:36,605 : INFO : keeping 100 factors (discarding 10.845% of energy spectrum)\n",
      "2017-08-03 11:44:36,956 : INFO : processed documents up to #120000\n",
      "2017-08-03 11:44:36,961 : INFO : topic #0(31.663): 0.230*\"φυσικά\" + 0.215*\"αστεροειδών\" + 0.209*\"jpl\" + 0.208*\"κύριας\" + 0.206*\"java\" + 0.206*\"αστεροειδής\" + 0.206*\"τροχιά\" + 0.202*\"ηλιακό\" + 0.200*\"απόλυτο\" + 0.200*\"ζώνης\"\n",
      "2017-08-03 11:44:36,965 : INFO : topic #1(28.221): 0.315*\"ποδοσφαιριστές\" + 0.135*\"πρωτάθλημα\" + 0.131*\"εθνική\" + 0.127*\"px\" + 0.103*\"κύπελλο\" + 0.095*\"εθνικής\" + 0.087*\"ποδοσφαίρου\" + 0.084*\"αγώνες\" + 0.080*\"έλληνας\" + 0.080*\"ομάδες\"\n",
      "2017-08-03 11:44:36,969 : INFO : topic #2(23.604): -0.593*\"ποδοσφαιριστές\" + -0.123*\"πρωτάθλημα\" + -0.120*\"εθνική\" + -0.112*\"κύπελλο\" + -0.099*\"αγωνίστηκε\" + -0.098*\"ποδοσφαίρου\" + -0.094*\"γκολ\" + -0.093*\"εθνικής\" + 0.087*\"χωριό\" + -0.085*\"λιγκ\"\n",
      "2017-08-03 11:44:36,974 : INFO : topic #3(19.477): 0.347*\"px\" + -0.209*\"κόμμα\" + -0.185*\"ποδοσφαιριστές\" + -0.175*\"εκλογές\" + 0.172*\"δήμος\" + 0.170*\"χωριό\" + 0.142*\"νομού\" + -0.123*\"βουλευτές\" + 0.120*\"δήμο\" + -0.114*\"βουλευτής\"\n",
      "2017-08-03 11:44:36,978 : INFO : topic #4(19.383): 0.500*\"px\" + -0.407*\"ποδοσφαιριστές\" + -0.168*\"χωριό\" + -0.162*\"δήμος\" + -0.137*\"νομού\" + 0.137*\"πρωτάθλημα\" + -0.116*\"δήμο\" + -0.112*\"δήμου\" + -0.111*\"κατοίκους\" + -0.101*\"απογραφή\"\n",
      "2017-08-03 11:44:39,081 : INFO : preparing a new chunk of documents\n",
      "2017-08-03 11:44:39,169 : INFO : using 100 extra samples and 2 power iterations\n",
      "2017-08-03 11:44:39,172 : INFO : 1st phase: constructing (99121, 200) action matrix\n",
      "2017-08-03 11:44:39,345 : INFO : orthonormalizing (99121, 200) action matrix\n",
      "2017-08-03 11:44:52,609 : INFO : 2nd phase: running dense svd on (200, 2813) matrix\n",
      "2017-08-03 11:44:52,886 : INFO : computing the final decomposition\n",
      "2017-08-03 11:44:52,889 : INFO : keeping 100 factors (discarding 23.297% of energy spectrum)\n",
      "2017-08-03 11:44:53,130 : INFO : merging projections: (99121, 100) + (99121, 100)\n",
      "2017-08-03 11:44:55,731 : INFO : keeping 100 factors (discarding 3.617% of energy spectrum)\n",
      "2017-08-03 11:44:56,119 : INFO : processed documents up to #122813\n",
      "2017-08-03 11:44:56,122 : INFO : topic #0(31.676): 0.230*\"φυσικά\" + 0.214*\"αστεροειδών\" + 0.209*\"jpl\" + 0.207*\"κύριας\" + 0.206*\"java\" + 0.205*\"αστεροειδής\" + 0.205*\"τροχιά\" + 0.201*\"ηλιακό\" + 0.200*\"απόλυτο\" + 0.199*\"ζώνης\"\n",
      "2017-08-03 11:44:56,127 : INFO : topic #1(28.461): 0.305*\"ποδοσφαιριστές\" + 0.133*\"πρωτάθλημα\" + 0.129*\"εθνική\" + 0.124*\"px\" + 0.101*\"κύπελλο\" + 0.093*\"εθνικής\" + 0.086*\"ποδοσφαίρου\" + 0.083*\"αγώνες\" + 0.080*\"έλληνας\" + 0.079*\"κόμμα\"\n",
      "2017-08-03 11:44:56,131 : INFO : topic #2(23.765): -0.594*\"ποδοσφαιριστές\" + -0.127*\"πρωτάθλημα\" + -0.123*\"εθνική\" + -0.114*\"κύπελλο\" + -0.100*\"αγωνίστηκε\" + -0.100*\"ποδοσφαίρου\" + -0.095*\"γκολ\" + -0.095*\"εθνικής\" + 0.087*\"χωριό\" + -0.086*\"λιγκ\"\n",
      "2017-08-03 11:44:56,135 : INFO : topic #3(19.716): -0.228*\"κόμμα\" + 0.225*\"χωριό\" + 0.223*\"δήμος\" + -0.189*\"εκλογές\" + 0.186*\"νομού\" + 0.157*\"δήμο\" + 0.149*\"κατοίκους\" + 0.148*\"δήμου\" + 0.141*\"απογραφή\" + -0.133*\"βουλευτές\"\n",
      "2017-08-03 11:44:56,139 : INFO : topic #4(19.473): 0.585*\"px\" + -0.458*\"ποδοσφαιριστές\" + 0.169*\"πρωτάθλημα\" + 0.111*\"κύπελλο\" + 0.107*\"ομάδες\" + 0.103*\"αγώνες\" + 0.103*\"ολυμπιακός\" + 0.093*\"παναθηναϊκός\" + 0.090*\"ανδρών\" + -0.083*\"χωριό\"\n"
     ]
    }
   ],
   "source": [
    "lsi = gensim.models.lsimodel.LsiModel(corpus=corpus, id2word=dictionary, num_topics=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save LSI model for future use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-08-02 14:42:32,157 : INFO : saving Projection object under ../data/model.lsi.projection, separately None\n",
      "2017-08-02 14:42:32,940 : INFO : saved ../data/model.lsi.projection\n",
      "2017-08-02 14:42:32,942 : INFO : saving LsiModel object under ../data/model.lsi, separately None\n",
      "2017-08-02 14:42:32,943 : INFO : not storing attribute projection\n",
      "2017-08-02 14:42:32,944 : INFO : not storing attribute dispatcher\n",
      "2017-08-02 14:42:33,008 : INFO : saved ../data/model.lsi\n"
     ]
    }
   ],
   "source": [
    "lsi.save('../data/model.lsi')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have created a model that can transform a vector from the tf-idf vector space to the topic vector space. This model extracted 400 topics from the documents. The first 10 topics are printed below. Each topic is represented by its 10 most contributing words (negative or positive)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2017-08-03 11:46:31,007 : INFO : topic #0(31.676): 0.230*\"φυσικά\" + 0.214*\"αστεροειδών\" + 0.209*\"jpl\" + 0.207*\"κύριας\" + 0.206*\"java\" + 0.205*\"αστεροειδής\" + 0.205*\"τροχιά\" + 0.201*\"ηλιακό\" + 0.200*\"απόλυτο\" + 0.199*\"ζώνης\"\n",
      "2017-08-03 11:46:31,011 : INFO : topic #1(28.461): 0.305*\"ποδοσφαιριστές\" + 0.133*\"πρωτάθλημα\" + 0.129*\"εθνική\" + 0.124*\"px\" + 0.101*\"κύπελλο\" + 0.093*\"εθνικής\" + 0.086*\"ποδοσφαίρου\" + 0.083*\"αγώνες\" + 0.080*\"έλληνας\" + 0.079*\"κόμμα\"\n",
      "2017-08-03 11:46:31,015 : INFO : topic #2(23.765): -0.594*\"ποδοσφαιριστές\" + -0.127*\"πρωτάθλημα\" + -0.123*\"εθνική\" + -0.114*\"κύπελλο\" + -0.100*\"αγωνίστηκε\" + -0.100*\"ποδοσφαίρου\" + -0.095*\"γκολ\" + -0.095*\"εθνικής\" + 0.087*\"χωριό\" + -0.086*\"λιγκ\"\n",
      "2017-08-03 11:46:31,018 : INFO : topic #3(19.716): -0.228*\"κόμμα\" + 0.225*\"χωριό\" + 0.223*\"δήμος\" + -0.189*\"εκλογές\" + 0.186*\"νομού\" + 0.157*\"δήμο\" + 0.149*\"κατοίκους\" + 0.148*\"δήμου\" + 0.141*\"απογραφή\" + -0.133*\"βουλευτές\"\n",
      "2017-08-03 11:46:31,022 : INFO : topic #4(19.473): 0.585*\"px\" + -0.458*\"ποδοσφαιριστές\" + 0.169*\"πρωτάθλημα\" + 0.111*\"κύπελλο\" + 0.107*\"ομάδες\" + 0.103*\"αγώνες\" + 0.103*\"ολυμπιακός\" + 0.093*\"παναθηναϊκός\" + 0.090*\"ανδρών\" + -0.083*\"χωριό\"\n",
      "2017-08-03 11:46:31,029 : INFO : topic #5(18.708): -0.370*\"ταινίες\" + 0.294*\"κόμμα\" + -0.254*\"ταινία\" + 0.236*\"εκλογές\" + 0.171*\"βουλευτές\" + 0.154*\"βουλευτής\" + 0.117*\"υπουργός\" + 0.106*\"εξελέγη\" + -0.099*\"ταινίας\" + 0.095*\"δήμος\"\n",
      "2017-08-03 11:46:31,032 : INFO : topic #6(17.311): -0.712*\"px\" + -0.296*\"ποδοσφαιριστές\" + 0.219*\"ολυμπιονίκες\" + 0.174*\"αρχαίοι\" + 0.163*\"πρωτάθλημα\" + 0.134*\"ος\" + 0.113*\"κύπελλο\" + 0.109*\"ομάδες\" + 0.101*\"αγώνες\" + 0.097*\"αιώνας\"\n",
      "2017-08-03 11:46:31,036 : INFO : topic #7(16.899): 0.369*\"ταινίες\" + -0.311*\"ολυμπιονίκες\" + -0.272*\"αρχαίοι\" + 0.226*\"ταινία\" + 0.143*\"κόμμα\" + -0.140*\"αιώνας\" + -0.135*\"ποδοσφαιριστές\" + -0.117*\"ος\" + 0.115*\"δήμος\" + -0.114*\"px\"\n",
      "2017-08-03 11:46:31,040 : INFO : topic #8(16.429): 0.420*\"ολυμπιονίκες\" + 0.325*\"αρχαίοι\" + 0.302*\"ταινίες\" + 0.188*\"px\" + 0.174*\"ταινία\" + 0.174*\"αιώνας\" + 0.149*\"ος\" + 0.141*\"ολυμπιονίκης\" + 0.139*\"ους\" + 0.112*\"ποδοσφαιριστές\"\n",
      "2017-08-03 11:46:31,044 : INFO : topic #9(15.274): 0.340*\"κόμμα\" + 0.207*\"άλμπουμ\" + 0.164*\"εκλογές\" + 0.163*\"ολυμπιονίκες\" + -0.145*\"έλληνας\" + 0.118*\"έδρες\" + 0.118*\"αρχαίοι\" + -0.107*\"πολιτικός\" + -0.092*\"βασιλιάς\" + -0.092*\"αμερικανός\"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  '0.230*\"φυσικά\" + 0.214*\"αστεροειδών\" + 0.209*\"jpl\" + 0.207*\"κύριας\" + 0.206*\"java\" + 0.205*\"αστεροειδής\" + 0.205*\"τροχιά\" + 0.201*\"ηλιακό\" + 0.200*\"απόλυτο\" + 0.199*\"ζώνης\"'),\n",
       " (1,\n",
       "  '0.305*\"ποδοσφαιριστές\" + 0.133*\"πρωτάθλημα\" + 0.129*\"εθνική\" + 0.124*\"px\" + 0.101*\"κύπελλο\" + 0.093*\"εθνικής\" + 0.086*\"ποδοσφαίρου\" + 0.083*\"αγώνες\" + 0.080*\"έλληνας\" + 0.079*\"κόμμα\"'),\n",
       " (2,\n",
       "  '-0.594*\"ποδοσφαιριστές\" + -0.127*\"πρωτάθλημα\" + -0.123*\"εθνική\" + -0.114*\"κύπελλο\" + -0.100*\"αγωνίστηκε\" + -0.100*\"ποδοσφαίρου\" + -0.095*\"γκολ\" + -0.095*\"εθνικής\" + 0.087*\"χωριό\" + -0.086*\"λιγκ\"'),\n",
       " (3,\n",
       "  '-0.228*\"κόμμα\" + 0.225*\"χωριό\" + 0.223*\"δήμος\" + -0.189*\"εκλογές\" + 0.186*\"νομού\" + 0.157*\"δήμο\" + 0.149*\"κατοίκους\" + 0.148*\"δήμου\" + 0.141*\"απογραφή\" + -0.133*\"βουλευτές\"'),\n",
       " (4,\n",
       "  '0.585*\"px\" + -0.458*\"ποδοσφαιριστές\" + 0.169*\"πρωτάθλημα\" + 0.111*\"κύπελλο\" + 0.107*\"ομάδες\" + 0.103*\"αγώνες\" + 0.103*\"ολυμπιακός\" + 0.093*\"παναθηναϊκός\" + 0.090*\"ανδρών\" + -0.083*\"χωριό\"'),\n",
       " (5,\n",
       "  '-0.370*\"ταινίες\" + 0.294*\"κόμμα\" + -0.254*\"ταινία\" + 0.236*\"εκλογές\" + 0.171*\"βουλευτές\" + 0.154*\"βουλευτής\" + 0.117*\"υπουργός\" + 0.106*\"εξελέγη\" + -0.099*\"ταινίας\" + 0.095*\"δήμος\"'),\n",
       " (6,\n",
       "  '-0.712*\"px\" + -0.296*\"ποδοσφαιριστές\" + 0.219*\"ολυμπιονίκες\" + 0.174*\"αρχαίοι\" + 0.163*\"πρωτάθλημα\" + 0.134*\"ος\" + 0.113*\"κύπελλο\" + 0.109*\"ομάδες\" + 0.101*\"αγώνες\" + 0.097*\"αιώνας\"'),\n",
       " (7,\n",
       "  '0.369*\"ταινίες\" + -0.311*\"ολυμπιονίκες\" + -0.272*\"αρχαίοι\" + 0.226*\"ταινία\" + 0.143*\"κόμμα\" + -0.140*\"αιώνας\" + -0.135*\"ποδοσφαιριστές\" + -0.117*\"ος\" + 0.115*\"δήμος\" + -0.114*\"px\"'),\n",
       " (8,\n",
       "  '0.420*\"ολυμπιονίκες\" + 0.325*\"αρχαίοι\" + 0.302*\"ταινίες\" + 0.188*\"px\" + 0.174*\"ταινία\" + 0.174*\"αιώνας\" + 0.149*\"ος\" + 0.141*\"ολυμπιονίκης\" + 0.139*\"ους\" + 0.112*\"ποδοσφαιριστές\"'),\n",
       " (9,\n",
       "  '0.340*\"κόμμα\" + 0.207*\"άλμπουμ\" + 0.164*\"εκλογές\" + 0.163*\"ολυμπιονίκες\" + -0.145*\"έλληνας\" + 0.118*\"έδρες\" + 0.118*\"αρχαίοι\" + -0.107*\"πολιτικός\" + -0.092*\"βασιλιάς\" + -0.092*\"αμερικανός\"')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lsi.print_topics(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Apply the LSI tranformation to the whole collection that is represented in\n",
    "the corpus object in tf-df form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus_lsi = lsi[corpus]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have managed to create a topic space representation of the Greek \n",
    "WikiPedia and reduced teh dimensions of the vector space from NxM (N is\n",
    "the number of documents and M is sthe number of words in the dictionary) \n",
    "to Nx100."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A document is represented in the new vector sapce like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 0.043097616110488325),\n",
       " (1, 0.10538208571697072),\n",
       " (2, 0.049693755551678806),\n",
       " (3, 0.013177885592721289),\n",
       " (4, 0.022210603056187848),\n",
       " (5, -0.047895680123100901),\n",
       " (6, 0.02779818360480461),\n",
       " (7, -0.055389267100266051),\n",
       " (8, -0.028944547760499775),\n",
       " (9, 0.059686660962608683),\n",
       " (10, -0.054395228223970249),\n",
       " (11, -0.043106698028264161),\n",
       " (12, 0.046429293537655916),\n",
       " (13, 0.028013281470275854),\n",
       " (14, 0.035825578529825368),\n",
       " (15, 0.026825692676210662),\n",
       " (16, 0.011081979206418789),\n",
       " (17, 0.011744732711968509),\n",
       " (18, -0.051268652323500509),\n",
       " (19, 0.043715181426642165),\n",
       " (20, -0.0096210998635424903),\n",
       " (21, -0.014752740761021722),\n",
       " (22, -0.010672577099673727),\n",
       " (23, 0.043502740153385573),\n",
       " (24, -0.023750172517982331),\n",
       " (25, 0.014728183535870146),\n",
       " (26, 0.02601227250022865),\n",
       " (27, 0.027615780775637599),\n",
       " (28, 0.030004526702247201),\n",
       " (29, -0.0086155325075867786),\n",
       " (30, 0.016341604498913483),\n",
       " (31, -0.045182979662826972),\n",
       " (32, 0.025455021008494342),\n",
       " (33, -0.025658191464714552),\n",
       " (34, 0.0035999498007744579),\n",
       " (35, 0.0111600414312878),\n",
       " (36, 0.038776974834303798),\n",
       " (37, -0.00082264793032000227),\n",
       " (38, 0.010776408257858711),\n",
       " (39, 0.0011992401342626054),\n",
       " (40, -0.0081005694354021138),\n",
       " (41, 0.029568087377897873),\n",
       " (42, 0.022085124215805461),\n",
       " (43, 0.00084320842904005383),\n",
       " (44, -0.0072955650134990026),\n",
       " (45, 0.01603244148973574),\n",
       " (46, -0.012970902863821437),\n",
       " (47, 4.7892394120141269e-05),\n",
       " (48, -0.0072208949462653675),\n",
       " (49, -0.0044220102392043339),\n",
       " (50, 0.0047885207866066055),\n",
       " (51, 0.0014296640115295325),\n",
       " (52, 0.017584408050847444),\n",
       " (53, 0.0093329433629210194),\n",
       " (54, 0.012373086777296887),\n",
       " (55, -0.0104299606385441),\n",
       " (56, 0.0037396315388798802),\n",
       " (57, 0.021442020110219312),\n",
       " (58, 0.016712428580239105),\n",
       " (59, 0.014039665326129495),\n",
       " (60, 0.024684401342571965),\n",
       " (61, -0.0092356232406805901),\n",
       " (62, -0.013112419116026739),\n",
       " (63, 0.0003923398982748427),\n",
       " (64, -0.0051762617185081711),\n",
       " (65, 0.010197656801414016),\n",
       " (66, 0.017085607906586718),\n",
       " (67, -0.015547334988163278),\n",
       " (68, -0.0013402668017882631),\n",
       " (69, -0.00046597760540792145),\n",
       " (70, -0.0092200117425666941),\n",
       " (71, 0.0053548666729911426),\n",
       " (72, 0.010972512804772508),\n",
       " (73, -0.027127850870126841),\n",
       " (74, 0.0024990742870729999),\n",
       " (75, 0.0037855257098892404),\n",
       " (76, 0.016238818695353936),\n",
       " (77, 0.0017851012101436732),\n",
       " (78, -0.0041622640867442365),\n",
       " (79, 0.023489901633314141),\n",
       " (80, 0.013095534291488476),\n",
       " (81, 0.014431577423331508),\n",
       " (82, -0.0034405348159308358),\n",
       " (83, 0.0062321175149999582),\n",
       " (84, -0.012272396148852784),\n",
       " (85, 0.014976197319684421),\n",
       " (86, 0.0084628692408278486),\n",
       " (87, 0.012724782314272232),\n",
       " (88, 0.013399550090073631),\n",
       " (89, 0.0055691948666048334),\n",
       " (90, 0.002179379680153908),\n",
       " (91, 0.014010280392885318),\n",
       " (92, 0.00052033625536153003),\n",
       " (93, 0.012785642494219238),\n",
       " (94, 0.0060977389692078443),\n",
       " (95, -0.006489275344030156),\n",
       " (96, -0.010721454601017262),\n",
       " (97, -0.00077834373861534968),\n",
       " (98, -0.00099422963395850083),\n",
       " (99, 0.0047361680551845043)]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus_lsi[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step is the clustering."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to use scikit-learn for the clustering. Scikit's algorithms require for the vector space matrix to be in a sparse matrix format. Gensim \n",
    "provides a function to do just that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100, 122813)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lsi_sparse = gensim.matutils.corpus2csc(corpus_lsi)\n",
    "lsi_sparse.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This sparse matrix has the documents as the columns. We need to reverse it\n",
    "before fitting the model, because sklearn expects the documents to be in rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(122813, 100)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import scipy\n",
    "lsi_sparse = scipy.sparse.csc_matrix.transpose(lsi_sparse)\n",
    "lsi_sparse.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I am using <a src=https://en.wikipedia.org/wiki/K-means_clustering>k-means</a> to create 8 clusters without any refinements apart from a maximum iteration number. MiniBatchKMeans is a more computationally effiecient implementation of k-means, that uses a fixed size subsample of the collection in each iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cluster import MiniBatchKMeans\n",
    "kmodel = MiniBatchKMeans(n_clusters=8, max_iter=10, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Finally fit the k-means model on the documents x topic weights matrix we\n",
    "created above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Init 1/3 with method: k-means++\n",
      "Inertia for init 1/3: 25.764056\n",
      "Init 2/3 with method: k-means++\n",
      "Inertia for init 2/3: 25.381525\n",
      "Init 3/3 with method: k-means++\n",
      "Inertia for init 3/3: 23.299360\n",
      "Minibatch iteration 1/12290: mean batch inertia: 0.094869, ewa inertia: 0.094869 \n",
      "Minibatch iteration 2/12290: mean batch inertia: 0.079461, ewa inertia: 0.094844 \n",
      "Minibatch iteration 3/12290: mean batch inertia: 0.089083, ewa inertia: 0.094835 \n",
      "Minibatch iteration 4/12290: mean batch inertia: 0.081780, ewa inertia: 0.094814 \n",
      "Minibatch iteration 5/12290: mean batch inertia: 0.089899, ewa inertia: 0.094806 \n",
      "Minibatch iteration 6/12290: mean batch inertia: 0.091719, ewa inertia: 0.094801 \n",
      "Minibatch iteration 7/12290: mean batch inertia: 0.103704, ewa inertia: 0.094815 \n",
      "Minibatch iteration 8/12290: mean batch inertia: 0.066246, ewa inertia: 0.094769 \n",
      "Minibatch iteration 9/12290: mean batch inertia: 0.078329, ewa inertia: 0.094742 \n",
      "Minibatch iteration 10/12290: mean batch inertia: 0.101849, ewa inertia: 0.094753 \n",
      "[MiniBatchKMeans] Reassigning 3 cluster centers.\n",
      "Minibatch iteration 11/12290: mean batch inertia: 0.087866, ewa inertia: 0.094742 \n",
      "Minibatch iteration 12/12290: mean batch inertia: 0.072545, ewa inertia: 0.094706 \n",
      "Minibatch iteration 13/12290: mean batch inertia: 0.082251, ewa inertia: 0.094686 \n",
      "Minibatch iteration 14/12290: mean batch inertia: 0.102994, ewa inertia: 0.094699 \n",
      "Minibatch iteration 15/12290: mean batch inertia: 0.089606, ewa inertia: 0.094691 \n",
      "Minibatch iteration 16/12290: mean batch inertia: 0.096388, ewa inertia: 0.094694 \n",
      "Minibatch iteration 17/12290: mean batch inertia: 0.110547, ewa inertia: 0.094719 \n",
      "Minibatch iteration 18/12290: mean batch inertia: 0.085138, ewa inertia: 0.094704 \n",
      "Minibatch iteration 19/12290: mean batch inertia: 0.079863, ewa inertia: 0.094680 \n",
      "Minibatch iteration 20/12290: mean batch inertia: 0.095834, ewa inertia: 0.094682 \n",
      "Minibatch iteration 21/12290: mean batch inertia: 0.095678, ewa inertia: 0.094683 \n",
      "Minibatch iteration 22/12290: mean batch inertia: 0.059811, ewa inertia: 0.094626 \n",
      "Minibatch iteration 23/12290: mean batch inertia: 0.080389, ewa inertia: 0.094603 \n",
      "Minibatch iteration 24/12290: mean batch inertia: 0.078862, ewa inertia: 0.094578 \n",
      "Minibatch iteration 25/12290: mean batch inertia: 0.074307, ewa inertia: 0.094545 \n",
      "Minibatch iteration 26/12290: mean batch inertia: 0.081233, ewa inertia: 0.094523 \n",
      "Minibatch iteration 27/12290: mean batch inertia: 0.085605, ewa inertia: 0.094508 \n",
      "Minibatch iteration 28/12290: mean batch inertia: 0.063627, ewa inertia: 0.094458 \n",
      "Minibatch iteration 29/12290: mean batch inertia: 0.079169, ewa inertia: 0.094433 \n",
      "Minibatch iteration 30/12290: mean batch inertia: 0.092554, ewa inertia: 0.094430 \n",
      "Minibatch iteration 31/12290: mean batch inertia: 0.086186, ewa inertia: 0.094417 \n",
      "Minibatch iteration 32/12290: mean batch inertia: 0.087627, ewa inertia: 0.094406 \n",
      "Minibatch iteration 33/12290: mean batch inertia: 0.078958, ewa inertia: 0.094381 \n",
      "Minibatch iteration 34/12290: mean batch inertia: 0.095785, ewa inertia: 0.094383 \n",
      "Minibatch iteration 35/12290: mean batch inertia: 0.079245, ewa inertia: 0.094358 \n",
      "Minibatch iteration 36/12290: mean batch inertia: 0.085310, ewa inertia: 0.094343 \n",
      "Minibatch iteration 37/12290: mean batch inertia: 0.083180, ewa inertia: 0.094325 \n",
      "Minibatch iteration 38/12290: mean batch inertia: 0.086928, ewa inertia: 0.094313 \n",
      "Minibatch iteration 39/12290: mean batch inertia: 0.062296, ewa inertia: 0.094261 \n",
      "Minibatch iteration 40/12290: mean batch inertia: 0.075275, ewa inertia: 0.094230 \n",
      "Minibatch iteration 41/12290: mean batch inertia: 0.067073, ewa inertia: 0.094186 \n",
      "Minibatch iteration 42/12290: mean batch inertia: 0.091804, ewa inertia: 0.094182 \n",
      "Minibatch iteration 43/12290: mean batch inertia: 0.091516, ewa inertia: 0.094178 \n",
      "Minibatch iteration 44/12290: mean batch inertia: 0.072535, ewa inertia: 0.094142 \n",
      "Minibatch iteration 45/12290: mean batch inertia: 0.075757, ewa inertia: 0.094113 \n",
      "Minibatch iteration 46/12290: mean batch inertia: 0.076520, ewa inertia: 0.094084 \n",
      "Minibatch iteration 47/12290: mean batch inertia: 0.089236, ewa inertia: 0.094076 \n",
      "Minibatch iteration 48/12290: mean batch inertia: 0.094250, ewa inertia: 0.094076 \n",
      "Minibatch iteration 49/12290: mean batch inertia: 0.082820, ewa inertia: 0.094058 \n",
      "Minibatch iteration 50/12290: mean batch inertia: 0.073816, ewa inertia: 0.094025 \n",
      "Minibatch iteration 51/12290: mean batch inertia: 0.082587, ewa inertia: 0.094006 \n",
      "Minibatch iteration 52/12290: mean batch inertia: 0.088909, ewa inertia: 0.093998 \n",
      "Minibatch iteration 53/12290: mean batch inertia: 0.086401, ewa inertia: 0.093986 \n",
      "Minibatch iteration 54/12290: mean batch inertia: 0.078771, ewa inertia: 0.093961 \n",
      "Minibatch iteration 55/12290: mean batch inertia: 0.081233, ewa inertia: 0.093940 \n",
      "Minibatch iteration 56/12290: mean batch inertia: 0.076018, ewa inertia: 0.093911 \n",
      "Minibatch iteration 57/12290: mean batch inertia: 0.070193, ewa inertia: 0.093872 \n",
      "Minibatch iteration 58/12290: mean batch inertia: 0.081188, ewa inertia: 0.093852 \n",
      "Minibatch iteration 59/12290: mean batch inertia: 0.083228, ewa inertia: 0.093834 \n",
      "Minibatch iteration 60/12290: mean batch inertia: 0.084381, ewa inertia: 0.093819 \n",
      "Minibatch iteration 61/12290: mean batch inertia: 0.081999, ewa inertia: 0.093800 \n",
      "Minibatch iteration 62/12290: mean batch inertia: 0.070366, ewa inertia: 0.093762 \n",
      "Minibatch iteration 63/12290: mean batch inertia: 0.097412, ewa inertia: 0.093768 \n",
      "Minibatch iteration 64/12290: mean batch inertia: 0.077910, ewa inertia: 0.093742 \n",
      "Minibatch iteration 65/12290: mean batch inertia: 0.070026, ewa inertia: 0.093703 \n",
      "Minibatch iteration 66/12290: mean batch inertia: 0.094276, ewa inertia: 0.093704 \n",
      "Minibatch iteration 67/12290: mean batch inertia: 0.078961, ewa inertia: 0.093680 \n",
      "Minibatch iteration 68/12290: mean batch inertia: 0.074542, ewa inertia: 0.093649 \n",
      "Minibatch iteration 69/12290: mean batch inertia: 0.071163, ewa inertia: 0.093612 \n",
      "Minibatch iteration 70/12290: mean batch inertia: 0.079508, ewa inertia: 0.093589 \n",
      "Minibatch iteration 71/12290: mean batch inertia: 0.086524, ewa inertia: 0.093578 \n",
      "Minibatch iteration 72/12290: mean batch inertia: 0.086532, ewa inertia: 0.093566 \n",
      "Minibatch iteration 73/12290: mean batch inertia: 0.079555, ewa inertia: 0.093543 \n",
      "Minibatch iteration 74/12290: mean batch inertia: 0.069261, ewa inertia: 0.093504 \n",
      "Minibatch iteration 75/12290: mean batch inertia: 0.069906, ewa inertia: 0.093465 \n",
      "Minibatch iteration 76/12290: mean batch inertia: 0.071401, ewa inertia: 0.093430 \n",
      "Minibatch iteration 77/12290: mean batch inertia: 0.070126, ewa inertia: 0.093392 \n",
      "Minibatch iteration 78/12290: mean batch inertia: 0.080489, ewa inertia: 0.093371 \n",
      "Minibatch iteration 79/12290: mean batch inertia: 0.072725, ewa inertia: 0.093337 \n",
      "Minibatch iteration 80/12290: mean batch inertia: 0.074635, ewa inertia: 0.093307 \n",
      "Minibatch iteration 81/12290: mean batch inertia: 0.067313, ewa inertia: 0.093264 \n",
      "Minibatch iteration 82/12290: mean batch inertia: 0.064750, ewa inertia: 0.093218 \n",
      "Minibatch iteration 83/12290: mean batch inertia: 0.072918, ewa inertia: 0.093185 \n",
      "Minibatch iteration 84/12290: mean batch inertia: 0.079463, ewa inertia: 0.093162 \n",
      "Minibatch iteration 85/12290: mean batch inertia: 0.090343, ewa inertia: 0.093158 \n",
      "Minibatch iteration 86/12290: mean batch inertia: 0.071368, ewa inertia: 0.093122 \n",
      "Minibatch iteration 87/12290: mean batch inertia: 0.081472, ewa inertia: 0.093103 \n",
      "Minibatch iteration 88/12290: mean batch inertia: 0.099821, ewa inertia: 0.093114 \n",
      "Minibatch iteration 89/12290: mean batch inertia: 0.082301, ewa inertia: 0.093097 \n",
      "Minibatch iteration 90/12290: mean batch inertia: 0.084780, ewa inertia: 0.093083 \n",
      "Minibatch iteration 91/12290: mean batch inertia: 0.095480, ewa inertia: 0.093087 \n",
      "Minibatch iteration 92/12290: mean batch inertia: 0.088486, ewa inertia: 0.093079 \n",
      "Minibatch iteration 93/12290: mean batch inertia: 0.103840, ewa inertia: 0.093097 \n",
      "Minibatch iteration 94/12290: mean batch inertia: 0.078194, ewa inertia: 0.093073 \n",
      "Minibatch iteration 95/12290: mean batch inertia: 0.065715, ewa inertia: 0.093028 \n",
      "Minibatch iteration 96/12290: mean batch inertia: 0.083873, ewa inertia: 0.093013 \n",
      "Minibatch iteration 97/12290: mean batch inertia: 0.079149, ewa inertia: 0.092991 \n",
      "Minibatch iteration 98/12290: mean batch inertia: 0.113036, ewa inertia: 0.093023 \n",
      "Minibatch iteration 99/12290: mean batch inertia: 0.082618, ewa inertia: 0.093006 \n",
      "Minibatch iteration 100/12290: mean batch inertia: 0.072266, ewa inertia: 0.092973 \n",
      "Minibatch iteration 101/12290: mean batch inertia: 0.101399, ewa inertia: 0.092986 \n",
      "Minibatch iteration 102/12290: mean batch inertia: 0.075200, ewa inertia: 0.092957 \n",
      "Minibatch iteration 103/12290: mean batch inertia: 0.082037, ewa inertia: 0.092940 \n",
      "Minibatch iteration 104/12290: mean batch inertia: 0.069922, ewa inertia: 0.092902 \n",
      "Minibatch iteration 105/12290: mean batch inertia: 0.083117, ewa inertia: 0.092886 \n",
      "Minibatch iteration 106/12290: mean batch inertia: 0.070766, ewa inertia: 0.092850 \n",
      "Minibatch iteration 107/12290: mean batch inertia: 0.130387, ewa inertia: 0.092911 \n",
      "Minibatch iteration 108/12290: mean batch inertia: 0.060326, ewa inertia: 0.092858 \n",
      "Minibatch iteration 109/12290: mean batch inertia: 0.093023, ewa inertia: 0.092858 \n",
      "Minibatch iteration 110/12290: mean batch inertia: 0.070531, ewa inertia: 0.092822 \n",
      "Minibatch iteration 111/12290: mean batch inertia: 0.077305, ewa inertia: 0.092797 \n",
      "Minibatch iteration 112/12290: mean batch inertia: 0.068985, ewa inertia: 0.092758 \n",
      "Minibatch iteration 113/12290: mean batch inertia: 0.086230, ewa inertia: 0.092747 \n",
      "Minibatch iteration 114/12290: mean batch inertia: 0.066694, ewa inertia: 0.092705 \n",
      "Minibatch iteration 115/12290: mean batch inertia: 0.059564, ewa inertia: 0.092651 \n",
      "Minibatch iteration 116/12290: mean batch inertia: 0.073180, ewa inertia: 0.092619 \n",
      "Minibatch iteration 117/12290: mean batch inertia: 0.079417, ewa inertia: 0.092598 \n",
      "Minibatch iteration 118/12290: mean batch inertia: 0.089162, ewa inertia: 0.092592 \n",
      "Minibatch iteration 119/12290: mean batch inertia: 0.077957, ewa inertia: 0.092568 \n",
      "Minibatch iteration 120/12290: mean batch inertia: 0.091334, ewa inertia: 0.092566 \n",
      "Minibatch iteration 121/12290: mean batch inertia: 0.074305, ewa inertia: 0.092537 \n",
      "Minibatch iteration 122/12290: mean batch inertia: 0.086773, ewa inertia: 0.092527 \n",
      "Minibatch iteration 123/12290: mean batch inertia: 0.069179, ewa inertia: 0.092489 \n",
      "Minibatch iteration 124/12290: mean batch inertia: 0.081051, ewa inertia: 0.092471 \n",
      "Minibatch iteration 125/12290: mean batch inertia: 0.092969, ewa inertia: 0.092471 \n",
      "Minibatch iteration 126/12290: mean batch inertia: 0.064391, ewa inertia: 0.092426 \n",
      "Minibatch iteration 127/12290: mean batch inertia: 0.104984, ewa inertia: 0.092446 \n",
      "Minibatch iteration 128/12290: mean batch inertia: 0.078529, ewa inertia: 0.092424 \n",
      "Minibatch iteration 129/12290: mean batch inertia: 0.081577, ewa inertia: 0.092406 \n",
      "Minibatch iteration 130/12290: mean batch inertia: 0.081887, ewa inertia: 0.092389 \n",
      "Minibatch iteration 131/12290: mean batch inertia: 0.087398, ewa inertia: 0.092381 \n",
      "Minibatch iteration 132/12290: mean batch inertia: 0.070978, ewa inertia: 0.092346 \n",
      "Minibatch iteration 133/12290: mean batch inertia: 0.087074, ewa inertia: 0.092337 \n",
      "Minibatch iteration 134/12290: mean batch inertia: 0.085292, ewa inertia: 0.092326 \n",
      "Minibatch iteration 135/12290: mean batch inertia: 0.091777, ewa inertia: 0.092325 \n",
      "Minibatch iteration 136/12290: mean batch inertia: 0.097874, ewa inertia: 0.092334 \n",
      "Minibatch iteration 137/12290: mean batch inertia: 0.093485, ewa inertia: 0.092336 \n",
      "Minibatch iteration 138/12290: mean batch inertia: 0.095131, ewa inertia: 0.092340 \n",
      "Minibatch iteration 139/12290: mean batch inertia: 0.085535, ewa inertia: 0.092329 \n",
      "Minibatch iteration 140/12290: mean batch inertia: 0.079796, ewa inertia: 0.092309 \n",
      "Minibatch iteration 141/12290: mean batch inertia: 0.091420, ewa inertia: 0.092307 \n",
      "Minibatch iteration 142/12290: mean batch inertia: 0.071160, ewa inertia: 0.092273 \n",
      "Minibatch iteration 143/12290: mean batch inertia: 0.106429, ewa inertia: 0.092296 \n",
      "Minibatch iteration 144/12290: mean batch inertia: 0.064958, ewa inertia: 0.092251 \n",
      "Minibatch iteration 145/12290: mean batch inertia: 0.069079, ewa inertia: 0.092214 \n",
      "Minibatch iteration 146/12290: mean batch inertia: 0.093842, ewa inertia: 0.092216 \n",
      "Minibatch iteration 147/12290: mean batch inertia: 0.078780, ewa inertia: 0.092194 \n",
      "Minibatch iteration 148/12290: mean batch inertia: 0.090281, ewa inertia: 0.092191 \n",
      "Minibatch iteration 149/12290: mean batch inertia: 0.062830, ewa inertia: 0.092144 \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Minibatch iteration 150/12290: mean batch inertia: 0.074904, ewa inertia: 0.092115 \n",
      "Minibatch iteration 151/12290: mean batch inertia: 0.067760, ewa inertia: 0.092076 \n",
      "Minibatch iteration 152/12290: mean batch inertia: 0.066583, ewa inertia: 0.092034 \n",
      "Minibatch iteration 153/12290: mean batch inertia: 0.072471, ewa inertia: 0.092002 \n",
      "Minibatch iteration 154/12290: mean batch inertia: 0.065788, ewa inertia: 0.091960 \n",
      "Minibatch iteration 155/12290: mean batch inertia: 0.094243, ewa inertia: 0.091963 \n",
      "Minibatch iteration 156/12290: mean batch inertia: 0.064424, ewa inertia: 0.091919 \n",
      "Minibatch iteration 157/12290: mean batch inertia: 0.067222, ewa inertia: 0.091878 \n",
      "Minibatch iteration 158/12290: mean batch inertia: 0.085169, ewa inertia: 0.091867 \n",
      "Minibatch iteration 159/12290: mean batch inertia: 0.100383, ewa inertia: 0.091881 \n",
      "Minibatch iteration 160/12290: mean batch inertia: 0.085629, ewa inertia: 0.091871 \n",
      "Minibatch iteration 161/12290: mean batch inertia: 0.079591, ewa inertia: 0.091851 \n",
      "Minibatch iteration 162/12290: mean batch inertia: 0.089972, ewa inertia: 0.091848 \n",
      "Minibatch iteration 163/12290: mean batch inertia: 0.074260, ewa inertia: 0.091819 \n",
      "Minibatch iteration 164/12290: mean batch inertia: 0.111888, ewa inertia: 0.091852 \n",
      "Minibatch iteration 165/12290: mean batch inertia: 0.085088, ewa inertia: 0.091841 \n",
      "Minibatch iteration 166/12290: mean batch inertia: 0.095600, ewa inertia: 0.091847 \n",
      "Minibatch iteration 167/12290: mean batch inertia: 0.077958, ewa inertia: 0.091825 \n",
      "Minibatch iteration 168/12290: mean batch inertia: 0.073576, ewa inertia: 0.091795 \n",
      "Minibatch iteration 169/12290: mean batch inertia: 0.080037, ewa inertia: 0.091776 \n",
      "Minibatch iteration 170/12290: mean batch inertia: 0.080947, ewa inertia: 0.091758 \n",
      "Minibatch iteration 171/12290: mean batch inertia: 0.078198, ewa inertia: 0.091736 \n",
      "Minibatch iteration 172/12290: mean batch inertia: 0.078274, ewa inertia: 0.091714 \n",
      "Minibatch iteration 173/12290: mean batch inertia: 0.100465, ewa inertia: 0.091728 \n",
      "Minibatch iteration 174/12290: mean batch inertia: 0.095551, ewa inertia: 0.091735 \n",
      "Minibatch iteration 175/12290: mean batch inertia: 0.074186, ewa inertia: 0.091706 \n",
      "Minibatch iteration 176/12290: mean batch inertia: 0.079114, ewa inertia: 0.091685 \n",
      "Minibatch iteration 177/12290: mean batch inertia: 0.081686, ewa inertia: 0.091669 \n",
      "Minibatch iteration 178/12290: mean batch inertia: 0.076380, ewa inertia: 0.091644 \n",
      "Minibatch iteration 179/12290: mean batch inertia: 0.080912, ewa inertia: 0.091627 \n",
      "Minibatch iteration 180/12290: mean batch inertia: 0.093708, ewa inertia: 0.091630 \n",
      "Minibatch iteration 181/12290: mean batch inertia: 0.099198, ewa inertia: 0.091643 \n",
      "Minibatch iteration 182/12290: mean batch inertia: 0.089980, ewa inertia: 0.091640 \n",
      "Minibatch iteration 183/12290: mean batch inertia: 0.073257, ewa inertia: 0.091610 \n",
      "Minibatch iteration 184/12290: mean batch inertia: 0.086066, ewa inertia: 0.091601 \n",
      "Minibatch iteration 185/12290: mean batch inertia: 0.084551, ewa inertia: 0.091589 \n",
      "Minibatch iteration 186/12290: mean batch inertia: 0.099456, ewa inertia: 0.091602 \n",
      "Minibatch iteration 187/12290: mean batch inertia: 0.068795, ewa inertia: 0.091565 \n",
      "Minibatch iteration 188/12290: mean batch inertia: 0.079003, ewa inertia: 0.091545 \n",
      "Minibatch iteration 189/12290: mean batch inertia: 0.101008, ewa inertia: 0.091560 \n",
      "Minibatch iteration 190/12290: mean batch inertia: 0.090423, ewa inertia: 0.091558 \n",
      "Minibatch iteration 191/12290: mean batch inertia: 0.064164, ewa inertia: 0.091514 \n",
      "Minibatch iteration 192/12290: mean batch inertia: 0.085794, ewa inertia: 0.091504 \n",
      "Minibatch iteration 193/12290: mean batch inertia: 0.070809, ewa inertia: 0.091471 \n",
      "Minibatch iteration 194/12290: mean batch inertia: 0.075215, ewa inertia: 0.091444 \n",
      "Minibatch iteration 195/12290: mean batch inertia: 0.079460, ewa inertia: 0.091425 \n",
      "Minibatch iteration 196/12290: mean batch inertia: 0.057958, ewa inertia: 0.091370 \n",
      "Minibatch iteration 197/12290: mean batch inertia: 0.083298, ewa inertia: 0.091357 \n",
      "Minibatch iteration 198/12290: mean batch inertia: 0.094499, ewa inertia: 0.091362 \n",
      "Minibatch iteration 199/12290: mean batch inertia: 0.074466, ewa inertia: 0.091334 \n",
      "Minibatch iteration 200/12290: mean batch inertia: 0.083807, ewa inertia: 0.091322 \n",
      "Minibatch iteration 201/12290: mean batch inertia: 0.070408, ewa inertia: 0.091288 \n",
      "Minibatch iteration 202/12290: mean batch inertia: 0.070713, ewa inertia: 0.091255 \n",
      "Minibatch iteration 203/12290: mean batch inertia: 0.084127, ewa inertia: 0.091243 \n",
      "Minibatch iteration 204/12290: mean batch inertia: 0.073767, ewa inertia: 0.091215 \n",
      "Minibatch iteration 205/12290: mean batch inertia: 0.090114, ewa inertia: 0.091213 \n",
      "Minibatch iteration 206/12290: mean batch inertia: 0.082882, ewa inertia: 0.091199 \n",
      "Minibatch iteration 207/12290: mean batch inertia: 0.078033, ewa inertia: 0.091178 \n",
      "Minibatch iteration 208/12290: mean batch inertia: 0.081784, ewa inertia: 0.091163 \n",
      "Minibatch iteration 209/12290: mean batch inertia: 0.082186, ewa inertia: 0.091148 \n",
      "Minibatch iteration 210/12290: mean batch inertia: 0.075437, ewa inertia: 0.091122 \n",
      "Minibatch iteration 211/12290: mean batch inertia: 0.103040, ewa inertia: 0.091142 \n",
      "Minibatch iteration 212/12290: mean batch inertia: 0.092009, ewa inertia: 0.091143 \n",
      "Minibatch iteration 213/12290: mean batch inertia: 0.071743, ewa inertia: 0.091112 \n",
      "Minibatch iteration 214/12290: mean batch inertia: 0.089967, ewa inertia: 0.091110 \n",
      "Minibatch iteration 215/12290: mean batch inertia: 0.075602, ewa inertia: 0.091084 \n",
      "Minibatch iteration 216/12290: mean batch inertia: 0.077269, ewa inertia: 0.091062 \n",
      "Minibatch iteration 217/12290: mean batch inertia: 0.082275, ewa inertia: 0.091048 \n",
      "Minibatch iteration 218/12290: mean batch inertia: 0.074119, ewa inertia: 0.091020 \n",
      "Minibatch iteration 219/12290: mean batch inertia: 0.091852, ewa inertia: 0.091021 \n",
      "Minibatch iteration 220/12290: mean batch inertia: 0.074120, ewa inertia: 0.090994 \n",
      "Minibatch iteration 221/12290: mean batch inertia: 0.092977, ewa inertia: 0.090997 \n",
      "Minibatch iteration 222/12290: mean batch inertia: 0.078126, ewa inertia: 0.090976 \n",
      "Minibatch iteration 223/12290: mean batch inertia: 0.068463, ewa inertia: 0.090939 \n",
      "Minibatch iteration 224/12290: mean batch inertia: 0.093295, ewa inertia: 0.090943 \n",
      "Minibatch iteration 225/12290: mean batch inertia: 0.091888, ewa inertia: 0.090945 \n",
      "Minibatch iteration 226/12290: mean batch inertia: 0.079948, ewa inertia: 0.090927 \n",
      "Minibatch iteration 227/12290: mean batch inertia: 0.074753, ewa inertia: 0.090901 \n",
      "Minibatch iteration 228/12290: mean batch inertia: 0.084537, ewa inertia: 0.090890 \n",
      "Minibatch iteration 229/12290: mean batch inertia: 0.084220, ewa inertia: 0.090879 \n",
      "Minibatch iteration 230/12290: mean batch inertia: 0.093143, ewa inertia: 0.090883 \n",
      "Minibatch iteration 231/12290: mean batch inertia: 0.091597, ewa inertia: 0.090884 \n",
      "Minibatch iteration 232/12290: mean batch inertia: 0.078092, ewa inertia: 0.090863 \n",
      "Minibatch iteration 233/12290: mean batch inertia: 0.062286, ewa inertia: 0.090817 \n",
      "Minibatch iteration 234/12290: mean batch inertia: 0.075401, ewa inertia: 0.090792 \n",
      "Minibatch iteration 235/12290: mean batch inertia: 0.077206, ewa inertia: 0.090770 \n",
      "Minibatch iteration 236/12290: mean batch inertia: 0.078131, ewa inertia: 0.090749 \n",
      "Minibatch iteration 237/12290: mean batch inertia: 0.085416, ewa inertia: 0.090740 \n",
      "Minibatch iteration 238/12290: mean batch inertia: 0.072174, ewa inertia: 0.090710 \n",
      "Minibatch iteration 239/12290: mean batch inertia: 0.080216, ewa inertia: 0.090693 \n",
      "Minibatch iteration 240/12290: mean batch inertia: 0.057876, ewa inertia: 0.090640 \n",
      "Minibatch iteration 241/12290: mean batch inertia: 0.076913, ewa inertia: 0.090617 \n",
      "Minibatch iteration 242/12290: mean batch inertia: 0.080921, ewa inertia: 0.090601 \n",
      "Minibatch iteration 243/12290: mean batch inertia: 0.062333, ewa inertia: 0.090555 \n",
      "Minibatch iteration 244/12290: mean batch inertia: 0.075628, ewa inertia: 0.090531 \n",
      "Minibatch iteration 245/12290: mean batch inertia: 0.071810, ewa inertia: 0.090501 \n",
      "Minibatch iteration 246/12290: mean batch inertia: 0.091316, ewa inertia: 0.090502 \n",
      "Minibatch iteration 247/12290: mean batch inertia: 0.082192, ewa inertia: 0.090488 \n",
      "Minibatch iteration 248/12290: mean batch inertia: 0.087285, ewa inertia: 0.090483 \n",
      "Minibatch iteration 249/12290: mean batch inertia: 0.100779, ewa inertia: 0.090500 \n",
      "Minibatch iteration 250/12290: mean batch inertia: 0.069666, ewa inertia: 0.090466 \n",
      "Minibatch iteration 251/12290: mean batch inertia: 0.065949, ewa inertia: 0.090426 \n",
      "Minibatch iteration 252/12290: mean batch inertia: 0.091467, ewa inertia: 0.090428 \n",
      "Minibatch iteration 253/12290: mean batch inertia: 0.102898, ewa inertia: 0.090448 \n",
      "Minibatch iteration 254/12290: mean batch inertia: 0.096301, ewa inertia: 0.090458 \n",
      "Minibatch iteration 255/12290: mean batch inertia: 0.095501, ewa inertia: 0.090466 \n",
      "Minibatch iteration 256/12290: mean batch inertia: 0.084741, ewa inertia: 0.090457 \n",
      "Minibatch iteration 257/12290: mean batch inertia: 0.083296, ewa inertia: 0.090445 \n",
      "Minibatch iteration 258/12290: mean batch inertia: 0.076779, ewa inertia: 0.090423 \n",
      "Minibatch iteration 259/12290: mean batch inertia: 0.082057, ewa inertia: 0.090409 \n",
      "Minibatch iteration 260/12290: mean batch inertia: 0.065451, ewa inertia: 0.090368 \n",
      "Minibatch iteration 261/12290: mean batch inertia: 0.093651, ewa inertia: 0.090374 \n",
      "Minibatch iteration 262/12290: mean batch inertia: 0.075932, ewa inertia: 0.090350 \n",
      "Minibatch iteration 263/12290: mean batch inertia: 0.080911, ewa inertia: 0.090335 \n",
      "Minibatch iteration 264/12290: mean batch inertia: 0.095565, ewa inertia: 0.090343 \n",
      "Minibatch iteration 265/12290: mean batch inertia: 0.082429, ewa inertia: 0.090330 \n",
      "Minibatch iteration 266/12290: mean batch inertia: 0.083962, ewa inertia: 0.090320 \n",
      "Minibatch iteration 267/12290: mean batch inertia: 0.078744, ewa inertia: 0.090301 \n",
      "Minibatch iteration 268/12290: mean batch inertia: 0.084247, ewa inertia: 0.090291 \n",
      "Minibatch iteration 269/12290: mean batch inertia: 0.082772, ewa inertia: 0.090279 \n",
      "Minibatch iteration 270/12290: mean batch inertia: 0.092096, ewa inertia: 0.090282 \n",
      "Minibatch iteration 271/12290: mean batch inertia: 0.074575, ewa inertia: 0.090256 \n",
      "Minibatch iteration 272/12290: mean batch inertia: 0.095346, ewa inertia: 0.090265 \n",
      "Minibatch iteration 273/12290: mean batch inertia: 0.077003, ewa inertia: 0.090243 \n",
      "Minibatch iteration 274/12290: mean batch inertia: 0.084376, ewa inertia: 0.090234 \n",
      "Minibatch iteration 275/12290: mean batch inertia: 0.087949, ewa inertia: 0.090230 \n",
      "Minibatch iteration 276/12290: mean batch inertia: 0.084330, ewa inertia: 0.090220 \n",
      "Minibatch iteration 277/12290: mean batch inertia: 0.086781, ewa inertia: 0.090215 \n",
      "Minibatch iteration 278/12290: mean batch inertia: 0.061714, ewa inertia: 0.090168 \n",
      "Minibatch iteration 279/12290: mean batch inertia: 0.074308, ewa inertia: 0.090142 \n",
      "Minibatch iteration 280/12290: mean batch inertia: 0.081602, ewa inertia: 0.090129 \n",
      "Minibatch iteration 281/12290: mean batch inertia: 0.066482, ewa inertia: 0.090090 \n",
      "Minibatch iteration 282/12290: mean batch inertia: 0.076401, ewa inertia: 0.090068 \n",
      "Minibatch iteration 283/12290: mean batch inertia: 0.086043, ewa inertia: 0.090061 \n",
      "Minibatch iteration 284/12290: mean batch inertia: 0.106908, ewa inertia: 0.090089 \n",
      "Minibatch iteration 285/12290: mean batch inertia: 0.079803, ewa inertia: 0.090072 \n",
      "Minibatch iteration 286/12290: mean batch inertia: 0.060168, ewa inertia: 0.090023 \n",
      "Minibatch iteration 287/12290: mean batch inertia: 0.082804, ewa inertia: 0.090011 \n",
      "Minibatch iteration 288/12290: mean batch inertia: 0.088040, ewa inertia: 0.090008 \n",
      "Minibatch iteration 289/12290: mean batch inertia: 0.108857, ewa inertia: 0.090039 \n",
      "Minibatch iteration 290/12290: mean batch inertia: 0.080899, ewa inertia: 0.090024 \n",
      "Minibatch iteration 291/12290: mean batch inertia: 0.057152, ewa inertia: 0.089971 \n",
      "Minibatch iteration 292/12290: mean batch inertia: 0.078877, ewa inertia: 0.089952 \n",
      "Minibatch iteration 293/12290: mean batch inertia: 0.072445, ewa inertia: 0.089924 \n",
      "Minibatch iteration 294/12290: mean batch inertia: 0.092363, ewa inertia: 0.089928 \n",
      "Minibatch iteration 295/12290: mean batch inertia: 0.094876, ewa inertia: 0.089936 \n",
      "Minibatch iteration 296/12290: mean batch inertia: 0.094114, ewa inertia: 0.089943 \n",
      "Minibatch iteration 297/12290: mean batch inertia: 0.100498, ewa inertia: 0.089960 \n",
      "Minibatch iteration 298/12290: mean batch inertia: 0.072185, ewa inertia: 0.089931 \n",
      "Minibatch iteration 299/12290: mean batch inertia: 0.078754, ewa inertia: 0.089913 \n",
      "Minibatch iteration 300/12290: mean batch inertia: 0.103519, ewa inertia: 0.089935 \n",
      "Minibatch iteration 301/12290: mean batch inertia: 0.069414, ewa inertia: 0.089902 \n",
      "Minibatch iteration 302/12290: mean batch inertia: 0.108970, ewa inertia: 0.089933 \n",
      "Minibatch iteration 303/12290: mean batch inertia: 0.082246, ewa inertia: 0.089920 \n",
      "Minibatch iteration 304/12290: mean batch inertia: 0.091405, ewa inertia: 0.089922 \n",
      "Minibatch iteration 305/12290: mean batch inertia: 0.091445, ewa inertia: 0.089925 \n",
      "Minibatch iteration 306/12290: mean batch inertia: 0.095742, ewa inertia: 0.089934 \n",
      "Minibatch iteration 307/12290: mean batch inertia: 0.085448, ewa inertia: 0.089927 \n",
      "Minibatch iteration 308/12290: mean batch inertia: 0.097939, ewa inertia: 0.089940 \n",
      "Minibatch iteration 309/12290: mean batch inertia: 0.056471, ewa inertia: 0.089886 \n",
      "Minibatch iteration 310/12290: mean batch inertia: 0.083437, ewa inertia: 0.089875 \n",
      "Minibatch iteration 311/12290: mean batch inertia: 0.078784, ewa inertia: 0.089857 \n",
      "Minibatch iteration 312/12290: mean batch inertia: 0.093528, ewa inertia: 0.089863 \n",
      "Minibatch iteration 313/12290: mean batch inertia: 0.093836, ewa inertia: 0.089870 \n",
      "Minibatch iteration 314/12290: mean batch inertia: 0.089322, ewa inertia: 0.089869 \n",
      "Minibatch iteration 315/12290: mean batch inertia: 0.074414, ewa inertia: 0.089844 \n",
      "Minibatch iteration 316/12290: mean batch inertia: 0.076192, ewa inertia: 0.089821 \n",
      "Minibatch iteration 317/12290: mean batch inertia: 0.076505, ewa inertia: 0.089800 \n",
      "Minibatch iteration 318/12290: mean batch inertia: 0.083536, ewa inertia: 0.089789 \n",
      "Minibatch iteration 319/12290: mean batch inertia: 0.073123, ewa inertia: 0.089762 \n",
      "Minibatch iteration 320/12290: mean batch inertia: 0.089334, ewa inertia: 0.089762 \n",
      "Minibatch iteration 321/12290: mean batch inertia: 0.100434, ewa inertia: 0.089779 \n",
      "Minibatch iteration 322/12290: mean batch inertia: 0.095194, ewa inertia: 0.089788 \n",
      "Minibatch iteration 323/12290: mean batch inertia: 0.087999, ewa inertia: 0.089785 \n",
      "Minibatch iteration 324/12290: mean batch inertia: 0.090344, ewa inertia: 0.089786 \n",
      "Minibatch iteration 325/12290: mean batch inertia: 0.093321, ewa inertia: 0.089792 \n",
      "Minibatch iteration 326/12290: mean batch inertia: 0.072977, ewa inertia: 0.089764 \n",
      "Minibatch iteration 327/12290: mean batch inertia: 0.069913, ewa inertia: 0.089732 \n",
      "Minibatch iteration 328/12290: mean batch inertia: 0.082188, ewa inertia: 0.089720 \n",
      "Minibatch iteration 329/12290: mean batch inertia: 0.077036, ewa inertia: 0.089699 \n",
      "Minibatch iteration 330/12290: mean batch inertia: 0.074653, ewa inertia: 0.089674 \n",
      "Minibatch iteration 331/12290: mean batch inertia: 0.103773, ewa inertia: 0.089697 \n",
      "Minibatch iteration 332/12290: mean batch inertia: 0.086948, ewa inertia: 0.089693 \n",
      "Minibatch iteration 333/12290: mean batch inertia: 0.077904, ewa inertia: 0.089674 \n",
      "Minibatch iteration 334/12290: mean batch inertia: 0.074274, ewa inertia: 0.089649 \n",
      "Minibatch iteration 335/12290: mean batch inertia: 0.080938, ewa inertia: 0.089634 \n",
      "Minibatch iteration 336/12290: mean batch inertia: 0.072197, ewa inertia: 0.089606 \n",
      "Minibatch iteration 337/12290: mean batch inertia: 0.072762, ewa inertia: 0.089579 \n",
      "Minibatch iteration 338/12290: mean batch inertia: 0.101831, ewa inertia: 0.089599 \n",
      "Minibatch iteration 339/12290: mean batch inertia: 0.080240, ewa inertia: 0.089583 \n",
      "Minibatch iteration 340/12290: mean batch inertia: 0.093554, ewa inertia: 0.089590 \n",
      "Minibatch iteration 341/12290: mean batch inertia: 0.080589, ewa inertia: 0.089575 \n",
      "Minibatch iteration 342/12290: mean batch inertia: 0.085418, ewa inertia: 0.089568 \n",
      "Minibatch iteration 343/12290: mean batch inertia: 0.067267, ewa inertia: 0.089532 \n",
      "Minibatch iteration 344/12290: mean batch inertia: 0.066262, ewa inertia: 0.089494 \n",
      "Minibatch iteration 345/12290: mean batch inertia: 0.092961, ewa inertia: 0.089500 \n",
      "Minibatch iteration 346/12290: mean batch inertia: 0.065306, ewa inertia: 0.089460 \n",
      "Minibatch iteration 347/12290: mean batch inertia: 0.083394, ewa inertia: 0.089450 \n",
      "Minibatch iteration 348/12290: mean batch inertia: 0.103772, ewa inertia: 0.089474 \n",
      "Minibatch iteration 349/12290: mean batch inertia: 0.083864, ewa inertia: 0.089465 \n",
      "Minibatch iteration 350/12290: mean batch inertia: 0.054678, ewa inertia: 0.089408 \n",
      "Minibatch iteration 351/12290: mean batch inertia: 0.105636, ewa inertia: 0.089434 \n",
      "Minibatch iteration 352/12290: mean batch inertia: 0.083304, ewa inertia: 0.089424 \n",
      "Minibatch iteration 353/12290: mean batch inertia: 0.085484, ewa inertia: 0.089418 \n",
      "Minibatch iteration 354/12290: mean batch inertia: 0.078281, ewa inertia: 0.089400 \n",
      "Minibatch iteration 355/12290: mean batch inertia: 0.088503, ewa inertia: 0.089398 \n",
      "Minibatch iteration 356/12290: mean batch inertia: 0.089945, ewa inertia: 0.089399 \n",
      "Minibatch iteration 357/12290: mean batch inertia: 0.114633, ewa inertia: 0.089440 \n",
      "Minibatch iteration 358/12290: mean batch inertia: 0.079733, ewa inertia: 0.089425 \n",
      "Minibatch iteration 359/12290: mean batch inertia: 0.084885, ewa inertia: 0.089417 \n",
      "Minibatch iteration 360/12290: mean batch inertia: 0.079674, ewa inertia: 0.089401 \n",
      "Minibatch iteration 361/12290: mean batch inertia: 0.059128, ewa inertia: 0.089352 \n",
      "Minibatch iteration 362/12290: mean batch inertia: 0.059637, ewa inertia: 0.089304 \n",
      "Minibatch iteration 363/12290: mean batch inertia: 0.073787, ewa inertia: 0.089278 \n",
      "Minibatch iteration 364/12290: mean batch inertia: 0.072278, ewa inertia: 0.089251 \n",
      "Minibatch iteration 365/12290: mean batch inertia: 0.076447, ewa inertia: 0.089230 \n",
      "Minibatch iteration 366/12290: mean batch inertia: 0.119171, ewa inertia: 0.089279 \n",
      "Minibatch iteration 367/12290: mean batch inertia: 0.094720, ewa inertia: 0.089287 \n",
      "Minibatch iteration 368/12290: mean batch inertia: 0.076551, ewa inertia: 0.089267 \n",
      "Minibatch iteration 369/12290: mean batch inertia: 0.100561, ewa inertia: 0.089285 \n",
      "Minibatch iteration 370/12290: mean batch inertia: 0.110997, ewa inertia: 0.089320 \n",
      "Minibatch iteration 371/12290: mean batch inertia: 0.089370, ewa inertia: 0.089321 \n",
      "Minibatch iteration 372/12290: mean batch inertia: 0.078507, ewa inertia: 0.089303 \n",
      "Minibatch iteration 373/12290: mean batch inertia: 0.069710, ewa inertia: 0.089271 \n",
      "Minibatch iteration 374/12290: mean batch inertia: 0.098115, ewa inertia: 0.089285 \n",
      "Minibatch iteration 375/12290: mean batch inertia: 0.084068, ewa inertia: 0.089277 \n",
      "Converged (lack of improvement in inertia) at iteration 375/12290\n",
      "Computing label assignment and total inertia\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "MiniBatchKMeans(batch_size=100, compute_labels=True, init='k-means++',\n",
       "        init_size=None, max_iter=10, max_no_improvement=10, n_clusters=8,\n",
       "        n_init=3, random_state=None, reassignment_ratio=0.01, tol=0.0,\n",
       "        verbose=True)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kmodel.fit(lsi_sparse) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have successfully trained the k-means model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the k-means model we just trained we can get the topic space representation of the centers of the clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "centers = kmodel.cluster_centers_.argsort()[:, ::-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have each cluster center vector in ascending topic weight."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1  4  6  7  3  0 18 19  5 27 15 17 11 59 41 57 16 80 48 66 25 62 94 77 34\n",
      " 46 49 29 70 90 82 92 36 42 60 98 30 93 81 51 23 85 56 44 88 35 71 83 14 47\n",
      " 50 67 97 74 79 40 38 24 65 55 73 54 28 95 84 78 99 53 10 63 96 91 45 26 75\n",
      " 64 89 33 39 43 72 76 12 87 61 52 58 37 32 86 68 31 69 20  9 21 22 13  8  2]\n"
     ]
    }
   ],
   "source": [
    "print(centers[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The vector printed above is the topic space representation of the first cluster. We can see that topic number 7 is the topic with the highest weight for this cluster. One way to visualize the clustering is to do the above for all the clusters, pick the 3 most contributing topics and display\n",
    "some information about them. Gensim provides a way to get information about a topic by printing the most contributing words for the direction of the topic. This contribution can be positive or negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster: 0\n",
      "    Topic 0: 0.305*\"ποδοσφαιριστές\" + 0.133*\"πρωτάθλημα\" + 0.129*\"εθνική\" + 0.124*\"px\" + 0.101*\"κύπελλο\"\n",
      "    Topic 1: 0.585*\"px\" + -0.458*\"ποδοσφαιριστές\" + 0.169*\"πρωτάθλημα\" + 0.111*\"κύπελλο\" + 0.107*\"ομάδες\"\n",
      "    Topic 2: -0.712*\"px\" + -0.296*\"ποδοσφαιριστές\" + 0.219*\"ολυμπιονίκες\" + 0.174*\"αρχαίοι\" + 0.163*\"πρωτάθλημα\"\n",
      "Cluster: 1\n",
      "    Topic 0: -0.228*\"κόμμα\" + 0.225*\"χωριό\" + 0.223*\"δήμος\" + -0.189*\"εκλογές\" + 0.186*\"νομού\"\n",
      "    Topic 1: 0.305*\"ποδοσφαιριστές\" + 0.133*\"πρωτάθλημα\" + 0.129*\"εθνική\" + 0.124*\"px\" + 0.101*\"κύπελλο\"\n",
      "    Topic 2: -0.594*\"ποδοσφαιριστές\" + -0.127*\"πρωτάθλημα\" + -0.123*\"εθνική\" + -0.114*\"κύπελλο\" + -0.100*\"αγωνίστηκε\"\n",
      "Cluster: 2\n",
      "    Topic 0: 0.305*\"ποδοσφαιριστές\" + 0.133*\"πρωτάθλημα\" + 0.129*\"εθνική\" + 0.124*\"px\" + 0.101*\"κύπελλο\"\n",
      "    Topic 1: 0.230*\"φυσικά\" + 0.214*\"αστεροειδών\" + 0.209*\"jpl\" + 0.207*\"κύριας\" + 0.206*\"java\"\n",
      "    Topic 2: -0.594*\"ποδοσφαιριστές\" + -0.127*\"πρωτάθλημα\" + -0.123*\"εθνική\" + -0.114*\"κύπελλο\" + -0.100*\"αγωνίστηκε\"\n",
      "Cluster: 3\n",
      "    Topic 0: -0.370*\"ταινίες\" + 0.294*\"κόμμα\" + -0.254*\"ταινία\" + 0.236*\"εκλογές\" + 0.171*\"βουλευτές\"\n",
      "    Topic 1: 0.305*\"ποδοσφαιριστές\" + 0.133*\"πρωτάθλημα\" + 0.129*\"εθνική\" + 0.124*\"px\" + 0.101*\"κύπελλο\"\n",
      "    Topic 2: -0.594*\"ποδοσφαιριστές\" + -0.127*\"πρωτάθλημα\" + -0.123*\"εθνική\" + -0.114*\"κύπελλο\" + -0.100*\"αγωνίστηκε\"\n",
      "Cluster: 4\n",
      "    Topic 0: 0.305*\"ποδοσφαιριστές\" + 0.133*\"πρωτάθλημα\" + 0.129*\"εθνική\" + 0.124*\"px\" + 0.101*\"κύπελλο\"\n",
      "    Topic 1: 0.230*\"φυσικά\" + 0.214*\"αστεροειδών\" + 0.209*\"jpl\" + 0.207*\"κύριας\" + 0.206*\"java\"\n",
      "    Topic 2: 0.420*\"ολυμπιονίκες\" + 0.325*\"αρχαίοι\" + 0.302*\"ταινίες\" + 0.188*\"px\" + 0.174*\"ταινία\"\n",
      "Cluster: 5\n",
      "    Topic 0: 0.369*\"ταινίες\" + -0.311*\"ολυμπιονίκες\" + -0.272*\"αρχαίοι\" + 0.226*\"ταινία\" + 0.143*\"κόμμα\"\n",
      "    Topic 1: 0.305*\"ποδοσφαιριστές\" + 0.133*\"πρωτάθλημα\" + 0.129*\"εθνική\" + 0.124*\"px\" + 0.101*\"κύπελλο\"\n",
      "    Topic 2: 0.329*\"άλμπουμ\" + -0.233*\"ταινίες\" + -0.206*\"κόμμα\" + 0.135*\"τραγούδια\" + 0.133*\"συγκροτήματος\"\n",
      "Cluster: 6\n",
      "    Topic 0: 0.305*\"ποδοσφαιριστές\" + 0.133*\"πρωτάθλημα\" + 0.129*\"εθνική\" + 0.124*\"px\" + 0.101*\"κύπελλο\"\n",
      "    Topic 1: -0.166*\"αθηνών\" + -0.155*\"βουλευτές\" + -0.142*\"βουλευτής\" + 0.142*\"βασιλιάς\" + 0.132*\"κόμμα\"\n",
      "    Topic 2: -0.594*\"ποδοσφαιριστές\" + -0.127*\"πρωτάθλημα\" + -0.123*\"εθνική\" + -0.114*\"κύπελλο\" + -0.100*\"αγωνίστηκε\"\n",
      "Cluster: 7\n",
      "    Topic 0: 0.279*\"επαρχία\" + -0.209*\"δήμος\" + -0.152*\"κόμμα\" + -0.138*\"δημοτικά\" + 0.129*\"περιφέρεια\"\n",
      "    Topic 1: -0.228*\"κόμμα\" + 0.225*\"χωριό\" + 0.223*\"δήμος\" + -0.189*\"εκλογές\" + 0.186*\"νομού\"\n",
      "    Topic 2: -0.272*\"jpg\" + -0.254*\"file\" + 0.192*\"επαρχία\" + -0.187*\"βουλευτές\" + -0.169*\"βουλευτής\"\n"
     ]
    }
   ],
   "source": [
    "for i, center in enumerate(centers):\n",
    "    # Pick the 3 most contributing topics.\n",
    "    print(\"Cluster: \" + str(i))\n",
    "    for j in range(3):\n",
    "        # Print top 10 most contributing words.\n",
    "        print(\"    Topic \" + str(j) + \": \" +\n",
    "              lsi.print_topic(center[j], topn=5))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}