LinkPred.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span>\n",
    "<b>Author:</b> <a href=\"http://about.giuliorossetti.net\">Giulio Rossetti</a><br/>\n",
    "<b>Python version:</b>  3.6<br/>\n",
    "<b>LinkPred version:</b>  0.4.1<br/>\n",
    "<b>Last update:</b> 12/05/2019\n",
    "</span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='top'></a>\n",
    "# *Intro to LinkPred: Link Prediction Library*\n",
    "\n",
    "``linkpred`` is a python library designed to provide support to *unsupervised* link prediction analysis.\n",
    "\n",
    "In this notebook are introduced some of the main features of the library and an overview of its functionalities.\n",
    "\n",
    "**Note:** \n",
    "- this notebook is purposely not 100% comprehensive, it only discusses the basic things you need to get started.\n",
    "- LinkPred is developed and maintained by Raf Guns (University of Antwerp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "1. [Installing LinkPred](#install)\n",
    "2. [Prediction Workflow](#workflow)\n",
    "    1. [Loading the network from file](#graph)\n",
    "    2. [Computing the desired unsupervied predictors](#pred)\n",
    "    3. [Evaluation and comparison of different predictors](#eval)\n",
    "3. [Conclusions](#conclusion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='install'></a>\n",
    "## 1. Installing LinkPred ([to top](#top))\n",
    "\n",
    "As a first step, we need to make sure that ``linkpred`` is installed and working.\n",
    "\n",
    "The library can be installed using ``pip``:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    pip install linkpred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to check if ``linkpred`` has been correctly installed just try to import it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import linkpred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='workflow'></a>\n",
    "## 2. Prediction Workflow\n",
    "\n",
    "``linkpred`` offers complete support to all stages of the Link Prediction workflow:\n",
    "\n",
    "1. Network Loading\n",
    "2. Predictor selection and application\n",
    "3. Results evaluation \n",
    "\n",
    "<a id='graph'></a>\n",
    "### 2.A Loading the network from file\n",
    "\n",
    "To get started, ``linkpred`` requires to read the graph from file: a check on the format is made on the file extension. </br>\n",
    "In our example we'll use standard edgelist graph representation (required extension *.edgelist*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import networkx as nx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "g = linkpred.read_network('data/gs6.edgelist')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='pred'></a>\n",
    "### 2.B Computing the desired unsupervied predictors\n",
    "\n",
    "``linkpred`` offers a wide number of unsupervied predictors organized into four families:\n",
    "- **Neighborhood:**\n",
    "    - *AdamicAdar*, AssociationStrength, *CommonNeighbours*, Cosine, DegreeProduct, *Jaccard*, MaxOverlap, MinOverlap, NMeasure, Pearson,ResourceAllocation\n",
    "- **Paths:**\n",
    "    - GraphDistance, *Katz*\n",
    "- **Ranking:**\n",
    "    - *SimRank*, RootedPageRank\n",
    "- **Miscellanea:**\n",
    "    - Community, Copy, *Random*\n",
    "    \n",
    "In the following we'll test only few of them.\n",
    "\n",
    "For sake of simplicity here we'll show only the top-5 forecasts for each selected predictor."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.B.1 Neighborhood\n",
    "\n",
    "The first family of predictors is the one that tries to relate neighborhoods of node pairs with the likelihood of observing a tie among them. </br>\n",
    "The question such predictors try to answer is: *How many friend we have to share in order to become friends?*\n",
    "\n",
    "##### **Common Neighbors**\n",
    "\n",
    "The more friends we share, the more likely we will become friends."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TOMMEN - MACE 8.0\n",
      "NED - LADY_CRANE 8.0\n",
      "NED - CLARENZO 8.0\n",
      "MARGAERY - IZEMBARO 8.0\n",
      "MARGAERY - ARYA 8.0\n"
     ]
    }
   ],
   "source": [
    "cn = linkpred.predictors.CommonNeighbours(g, excluded=g.edges()) # We aim to predict only new links, thus we exclude existing ones\n",
    "cn_results = cn.predict()\n",
    "\n",
    "top = cn_results.top(5)\n",
    "for edge, score in top.items():\n",
    "    print(edge, score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### **Jaccard**\n",
    "The more similar our friends circles are, the more likely we will become friends."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RED_PRIEST - KINVARA 1.0\n",
      "LITTLE_SAM - DICKON 1.0\n",
      "TOMMEN - MACE 0.7272727272727273\n",
      "JOFFREY - CAMELLO 0.7272727272727273\n",
      "OLD_NAN - BENJEN 0.7142857142857143\n"
     ]
    }
   ],
   "source": [
    "jc = linkpred.predictors.Jaccard(g, excluded=g.edges())\n",
    "jc_results = jc.predict()\n",
    "\n",
    "top = jc_results.top(5)\n",
    "for edge, score in top.items():\n",
    "    print(edge, score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### **Adamic Adar**\n",
    "The more selective our mutual friends are, the more likely we will become friends."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TOMMEN - MACE 3.187557281030789\n",
      "NED - LADY_CRANE 3.126371165725784\n",
      "NED - CLARENZO 3.126371165725784\n",
      "SANSA - ROBB 3.096644315133247\n",
      "JOFFREY - CAMELLO 2.8944937119329133\n"
     ]
    }
   ],
   "source": [
    "aa = linkpred.predictors.AdamicAdar(g, excluded=g.edges())\n",
    "aa_results = aa.predict()\n",
    "\n",
    "top = aa_results.top(5)\n",
    "for edge, score in top.items():\n",
    "    print(edge, score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.B.2 Paths\n",
    "\n",
    "The second family of predictors is the one that tries to relate node pairs distance with the likelihood of observing a tie among them in the future. </br>\n",
    "The question such predictors try to answer is: *How distant are we?*\n",
    "\n",
    "\n",
    "##### **Katz**\n",
    "Katz computes the weighted sum over all the paths between two nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": []
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "MARGAERY - IZEMBARO 8.094303942000002e-06\n",
      "MARGAERY - ARYA 8.093345628999996e-06\n",
      "CERSEI - CAMELLO 8.090298797999998e-06\n",
      "NED - LADY_CRANE 8.083239761000003e-06\n",
      "NED - CLARENZO 8.078180928e-06\n"
     ]
    }
   ],
   "source": [
    "kz = linkpred.predictors.Katz(g, excluded=g.edges())\n",
    "kz_results = kz.predict()\n",
    "\n",
    "top = kz_results.top(5)\n",
    "for edge, score in top.items():\n",
    "    print(edge, score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.B.3 Rankig\n",
    "\n",
    "The third family of predictors is the one that tries to relate node pairs position in the graph with the likelihood of observing a tie among them in the future. </br>\n",
    "The question such predictors try to answer is: *How similar are we?\n",
    "\n",
    "##### **SimRank**\n",
    "Two nodes are similar to the extent that their neighborhoods are similar."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LITTLE_SAM - DICKON 0.08833386746204597\n",
      "MORGAN - HOUND 0.0868844615447422\n",
      "HOUND - FLYNN 0.0868844615447422\n",
      "SANSA - ROBB 0.07299147669159453\n",
      "NED - MEERA 0.07081009555968669\n"
     ]
    }
   ],
   "source": [
    "simrank = linkpred.predictors.SimRank(g, excluded=g.edges())\n",
    "simrank_results = simrank.predict(c=0.5)\n",
    "\n",
    "top = simrank_results.top(5)\n",
    "for edge, score in top.items():\n",
    "    print(edge, score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.B.4 Miscellanea\n",
    "\n",
    "In this family fall alternative definitios of link predictors. </br>\n",
    "``linkpred`` groups here approaches that are commonly used as baselines.\n",
    "\n",
    "##### **Random**\n",
    "\n",
    "Random guessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LITTLE_SAM - DICKON 0.08833386746204597\n",
      "MORGAN - HOUND 0.0868844615447422\n",
      "HOUND - FLYNN 0.0868844615447422\n",
      "SANSA - ROBB 0.07299147669159453\n",
      "NED - MEERA 0.07081009555968669\n"
     ]
    }
   ],
   "source": [
    "rnd = linkpred.predictors.Random(g, excluded=g.edges())\n",
    "rnd_results = rnd.predict()\n",
    "\n",
    "top = simrank_results.top(5)\n",
    "for edge, score in top.items():\n",
    "    print(edge, score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='eval'></a>\n",
    "### 2.C Evaluation and comparison of different predictors\n",
    "\n",
    "To evaluate a link predictor we have to separate the network used for training from the one used for testing purposes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import itertools\n",
    "from linkpred.evaluation import Pair\n",
    "\n",
    "# Building the test network\n",
    "test = linkpred.read_network('data/gs7.edgelist')\n",
    "\n",
    "# Exclude test network from learning phase\n",
    "training = g.copy()\n",
    "\n",
    "# Node set\n",
    "nodes = list(g.nodes())\n",
    "nodes.extend(list(test.nodes()))\n",
    "\n",
    "# Compute the test set and the universe set\n",
    "test = [Pair(i) for i in test.edges()]\n",
    "universe = set([Pair(i) for i in itertools.product(nodes, nodes) if i[0]!=i[1]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, we can apply the predictors to the training network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "cn = linkpred.predictors.CommonNeighbours(training, excluded=training.edges())\n",
    "cn_results = cn.predict()\n",
    "\n",
    "aa = linkpred.predictors.AdamicAdar(training, excluded=training.edges())\n",
    "aa_results = aa.predict()\n",
    "\n",
    "jc = linkpred.predictors.Jaccard(training, excluded=training.edges())\n",
    "jc_results = jc.predict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, we can evaluate the obtained prediction against the test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "cn_evaluation = linkpred.evaluation.EvaluationSheet(cn_results, test, universe)\n",
    "aa_evaluation = linkpred.evaluation.EvaluationSheet(aa_results, test, universe)\n",
    "jc_evaluation = linkpred.evaluation.EvaluationSheet(jc_results, test, universe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results can be easlily compared using a ROC plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEKCAYAAAAb7IIBAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XdYVFf6wPHvmQFBRJEiWFABxY5YEHuNGlPUGONGU9ZUYzb+ds2WtM1ms+m9Z5MYY/qaZkzcbKLR2Ltg7GJHASugCCgw5fz+uARpwgAzzADv53l8mDnl3ncU7zv33nPPUVprhBBCNDwmdwcghBDCPSQBCCFEAyUJQAghGihJAEII0UBJAhBCiAZKEoAQQjRQkgCEEKKBkgQghBANlEMJQCk1Tim1Tyl1UCn1UDn1M5VSO5VS25RSa5VS3YrVPVzYb59S6kpnBi+EEKL6VGVPAiulzMB+YAyQCmwBpmmt9xRr00xrfb7w9QTgD1rrcYWJYD4QD7QGlgGdtNa2y+0vJCRER0RE1OhDCSFEQ5OYmJiutW5RlT5eDrSJBw5qrQ8DKKW+ACYCRQngt4N/oSbAb1llIvCF1jofOKKUOli4vQ2X21lERAQJCQlV+QxCCNHgKaWOVrWPIwmgDZBS7H0q0L+cnd8H/BloBIwq1ndjqb5tqhqkEEII53PaTWCt9dta6w7Ag8CjVemrlJqhlEpQSiWcOXPGWSEJIYSogCMJIA1oW+x9eGHZ5XwBXFeVvlrrOVrrOK11XIsWVbqEJYQQopocuQS0BYhWSkViHLynAjcVb6CUitZaHyh8ew3w2+tFwH+UUq9g3ASOBjZXNUiLxUJqaip5eXlV7SrqMV9fX8LDw/H29nZ3KELUSZUmAK21VSk1C1gCmIF5WuvdSqkngASt9SJgllJqNGABzgLTC/vuVkp9hXHD2ArcV9EIoMtJTU2ladOmREREoJSqandRD2mtycjIIDU1lcjISHeHI0Sd5MgZAFrrH4EfS5U9Vuz1nyro+zTwdHUDBMjLy5ODvyhBKUVwcDByz0iI6qszTwLLwV+UJr8TQtSMQ2cAQgghXOPzTUc5lZXHhZOv08jbygPT5tTavuvMGYC7nTx5kqlTp9KhQwf69u3L1Vdfzf79+90dlkM++ugjTCYTO3bsKCrr0aMHycnJFfa766672LNnT4VtbrvtNr755psy5StXruTaa6+tVrxCNBSZuQX8feEu3lh+kB25G9iYvbHyTk4kZwAO0FozadIkpk+fzhdffAHA9u3bOXXqFJ06dXJzdI4JDw/n6aef5ssvv3S4z9y5c10YUcVsNhtms9lt+xfClTYfyeTR73aSb7Xzgtd7XBlwlAdNeWR7+9VqHHIG4IAVK1bg7e3NzJkzi8piY2MZOnQoWmv+9re/0aNHD2JiYooOsCtXrmT48OFMnDiRqKgoHnroIT7//HPi4+OJiYnh0KFDgPEN+t5772XAgAFERUWxcuVK7rjjDrp27cptt91WtL/58+cTExNDjx49ePDBB4vK/f39+fvf/05sbCwDBgzg1KlT5X6Ga6+9lt27d7Nv374ydT///DMDBw6kT58+TJkyhZycHABGjBhRNC3HBx98QKdOnYiPj+fuu+9m1qxZRf1Xr17NoEGDiIqKKnE2cP78ea655ho6d+7MzJkzsdvtlX6Wv/zlL8TGxrJhwwYeeughunXrRs+ePfnrX//q2D+WEJ7Gmg8nd5b4s2/7esyndzM68AzjGyXg56XALwT8w2o1tDp3BvCv/+5mz/HzlTesgm6tm/HP8d0vW79r1y769u1bbt23337Ltm3b2L59O+np6fTr149hw4YBxlnC3r17CQoKIioqirvuuovNmzfz+uuv8+abb/Laa68BcPbsWTZs2MCiRYuYMGEC69atY+7cufTr149t27YRGhrKgw8+SGJiIoGBgYwdO5bvvvuO6667jtzcXAYMGMDTTz/NAw88wPvvv8+jj5Z9ENtkMvHAAw/wzDPP8PHHHxeVp6en89RTT7Fs2TKaNGnC888/zyuvvMJjjxUN8uL48eM8+eSTbN26laZNmzJq1ChiY2OL6k+cOMHatWtJSkpiwoQJ3HDDDQBs3ryZPXv20L59e8aNG8e3337LoEGDKvws/fv35+WXXyYjI4M777yTpKQklFKcO3euCv+iQniQnx6ExA9LFN0K3OqDMTkOQI8JwGkoyK7V0OQMoIbWrl3LtGnTMJvNhIWFMXz4cLZs2QJAv379aNWqFT4+PnTo0IGxY8cCEBMTU+L6+/jx41FKERMTQ1hYGDExMZhMJrp3705ycjJbtmxhxIgRtGjRAi8vL26++WZWr14NQKNGjYqutfft27fC6/o33XQTGzdu5MiRI0VlGzduZM+ePQwePJhevXrx8ccfc/RoyTmlNm/ezPDhwwkKCsLb25spU6aUqL/uuuswmUx069atxBlIfHw8UVFRmM1mpk2bxtq1ayv8LGazmcmTJwMQEBCAr68vd955J99++y1+frV7aixEjeWmw4pn4eg6aNoabvwMbvyMQ6Pe5Z6C+3nA/EBh2ecw5M8kZSahqXh2Zmerc2cAFX1Td5Xu3buXe6OzMj4+PkWvTSZT0XuTyYTVai3Trnib4u0qetLV29u7aDik2Wwusd3SvLy8+Mtf/sLzzz9fVKa1ZsyYMcyfP7+Kn+6S4jEXn1689DDNyoZt+vr6Fl339/LyYvPmzfzyyy988803vPXWWyxfvrzaMQpR6/b9CKueA5MXdJ0AXccD8NTGzaywN+PO3pHQtWjpFEzKxKkL5V/CdRU5A3DAqFGjyM/PZ86cS8OzduzYwZo1axg6dChffvklNpuNM2fOsHr1auLj4526//j4eFatWkV6ejo2m4358+czfPjwam3rtttuY9myZUUPUA0YMIB169Zx8OBBAHJzc8uMburXrx+rVq3i7NmzWK1WFixY4NC+Nm/ezJEjR7Db7Xz55ZcMGTLE4c+Sk5NDVlYWV199Na+++irbt2+v1ucVwm3Wvmr8nL0TpnzIsYwLjHttNesPZRDbtjn/uLYbH+36iKsWXMVVC64iMy+TAa0G1GqIde4MwB2UUixcuJDZs2fz/PPP4+vrS0REBK+99hpDhgxhw4YNxMbGopTihRdeoGXLliQlJTlt/61ateK5555j5MiRaK255pprmDhxYrW21ahRI/74xz/ypz8ZD2+3aNGCjz76iGnTppGfnw/AU089VWJ0U5s2bXjkkUeIj48nKCiILl26EBAQUOm++vXrx6xZszh48CAjR45k0qRJmEwmhz5LdnY2EydOJC8vD601r7zySrU+rxBuYzPOxndmNYbzWWxOziTpZDbDO7Xghr7hWO1WPt37Kfm2fIa2GUovejGp46RaDbHSFcFqW1xcnC69IMzevXvp2rWrmyISYHwj9/f3x2q1MmnSJO644w4mTardX9byyO+G8Fiv9WRJTiT35MwoUfzz/cPoFNaUT3Z/wosJLxLfMp4PrvygxrtTSiVqreOq0kfOAIRDHn/8cZYtW0ZeXh5jx47luuuuq7yTEA3V2WQ4dxQL7RnZuQU3928PQFNfL6JD/QHIthgjfp4eUqOp0mpEEoBwyEsvveTuEISoO7YbzwPtVxG0DfJjdLfLj+9v2aRlLQVVliQAIYRwUJ7Fxo3vbSA9p4DW9pO8YHmaxjq/TDt/cvEH3swbx621H6bDJAEIIRquvCw44/icXlnn8zClJTKsZVMGNjpE5KlU9vgPJNereZm2p3zac0NIOJP7hJeps9lt7E7fXaPQnUESgBCi4Vo40xiv76AwYKEPxrJXhbrd+gqEdSu3/eWmQ1x4cCFr0tbgbXLvanaSAIQQDdPe/8LxbRDaDcY86VCXHanneHnpfh65uiudw5qCbzMIrdootHxbPq9tNaaB+Xjcx5W0di15EKwKvvvuO5RSFY7xv9z0yNUxaNCgKvfp1asXU6dOvWx9cnIyPXr0qElYQtQPyx6HnFPQYRREj3boT2brYayyx5LbboRR1jYeqrgw0a70XWTlZ2FSJroFl3/mUFvkDKAK5s+fz5AhQ5g/fz7/+te/XL6/9evXV6n93r17sdlsrFmzhtzcXJo0aVLjGKxWK15e8msi6qjTSTB/KljzytblnIYe18OV5Q/DvPOjLew6nlWiLN9qzGhbk7XoDmcdBuDd0e9iNrl3ynM5A3BQTk4Oa9eu5YMPPihaEwCMuW9mzZpF586dGT16NKdPny6qe+KJJ+jXrx89evRgxowZRfPkjBgxgvvvv5+4uDi6du3Kli1buP7664mOji4xk6e/v3/R6+eff56YmBhiY2N56KGHyo1x/vz53HrrrYwdO5bvv/++qDwxMZHY2FhiY2N5++23i8qTk5MZOnQoffr0oU+fPkUJZ+XKlQwdOpQJEybQrZt7v6EIUSMntsHZI9CmL0SPKfmn980Qd+dlu647lE6gXyNGdg4t+jOue0vuGhJJ11bNqh1SduGMn62atKr2Npyl7n21++khY05tZ2oZA1c9V2GT77//nnHjxtGpUyeCg4NJTEykb9++LFy4kH379rFnzx5OnTpFt27duOOOOwCYNWtW0bTKt956Kz/88APjxxsTQjVq1IiEhARef/11Jk6cSGJiIkFBQXTo0IH777+f4ODgSx/5p5/4/vvv2bRpE35+fmRmZpYb45dffsnSpUtJSkrizTff5KabbgLg9ttv56233mLYsGH87W9/K2ofGhrK0qVL8fX15cCBA0ybNq1o/v+tW7eya9cuIiMjq/mXKoQH2P2d8XPsUxBU9d/l4Z1a8PDVzn3SfGnyUgACfQOdut3qkDMAB82fP7/o2vrUqVOLZs9cvXp10XTQrVu3ZtSoUUV9VqxYQf/+/YmJiWH58uXs3n1p2NeECRMAY2ro7t27F00bHRUVRUpKSol9L1u2jNtvv71oSuSgoKAy8SUkJBASEkK7du244oor+PXXX8nMzOTcuXOcO3euaI2CW2+9NCrZYrFw9913ExMTw5QpU0os/xgfHy8Hf1F3HV0PP/4NMg4Y76t48Ndak2exOz2sbae3sStjF4E+gQT4VD6flqvVvTOASr6pu0JmZibLly9n586dKKWw2WwopXjxxRcv2ycvL48//OEPJCQk0LZtWx5//HHy8i5dh6xsCuiqmj9/PklJSURERADGalwLFiwoM3d/ca+++iphYWFs374du92Or69vUZ0z7h8I4TYb3jaGd/oGQM8bq9z9eJbxfzW3oOr/Fyvy+d7PAbg66mqnbre65AzAAd988w233norR48eJTk5mZSUFCIjI1mzZg3Dhg0rmg76xIkTrFixAqDoYB8SEkJOTk6NRgaNGTOGDz/8kAsXLgCUuQRkt9v56quv2LlzJ8nJySQnJ/P9998zf/58mjdvTvPmzVm7di0An3/+eVG/rKwsWrVqhclk4tNPP8Vms1U7RiE8xukkSPoBWnSBB5Ph+jmVdintxLmLAMS0qfm3dLu2c+uPtzL8y+H8cuwXogKieCi+/Pt4tU0SgAPmz59fZubLyZMnF5VHR0fTrVs3fv/73zNw4EAAmjdvzt13302PHj248sor6devX7X3P27cOCZMmEBcXBy9evUqMy/PmjVraNOmDa1bty4qGzZsGHv27OHEiRN8+OGH3HffffTq1avEgi1/+MMf+Pjjj4mNjSUpKUm+9Yv64XThpdbCBViqI+uiBYDmfo1qHI7VbmXbmW208W/DpI6TuLfXvTXeprM4NB20Umoc8DpgBuZqrZ8rVf9n4C7ACpwB7tBaHy2sswG/3bU9prWeUNG+ZDpoURXyuyHKWPoYrHsd/m8rBHeo1ibe/OUALy/dz6JZg+kZXnaah6pIzkpm/HfjuaPHHdzf9/4abasiLpkOWillBt4GxmAsYbxFKbVIa72nWLNfgTit9QWl1L3AC8BvF94uaq17VSUoIYQo4+AvxqWdyhzfZvwMaFvtXa3YZwznbhdUs7Wo7drOy4kvA9A1yPO+qDhyEzgeOKi1PgyglPoCmAgUJQCt9Ypi7TcCtzgzSCGEYP2bkLwGGjswfLLtAGMt3mo6dCYXgCY+NRsnczznOCtTVgIQHRhdo225giOfrg1QfFxiKtC/gvZ3Aj8Ve++rlErAuDz0nNb6uypHKYRo2Na8AodXGAf2O5fUaFNfbjnGsz8lUdHV76yLFm7u3w5vc81ukyaeSgTgqcFP0aF59S5HuZJTh4EqpW4B4oDiq3y311qnKaWigOVKqZ1a60Ol+s0AZgC0a9fOmSEJIeqq9IOQVfjdc3/hQX/YX2u82W+3ppGbby1apetypsSVnca5qrac3AJAn7A+Nd6WKziSANKA4hfTwgvLSlBKjQb+DgzX+tIKCVrrtMKfh5VSK4HeQIkEoLWeA8wB4yZw1T6CEKJemjMCCqdNAIxv/9FjarTJPIuNTUcyUQoen9C9ZvFVIteSy/eHjClZQv1CXbqv6nIkAWwBopVSkRgH/qnATcUbKKV6A+8B47TWp4uVBwIXtNb5SqkQYDDGDWIhhLjkwDLY+33JsoJsiL0J+vzeeB9S82voBTbj6d67hrj2KffMvEze2PoGALd0vQUfs08lPdyj0gSgtbYqpWYBSzCGgc7TWu9WSj0BJGitFwEvAv7A18qYGvW34Z5dgfeUUnaMZw6eKzV6qE7x9/cnJyfHbftfuXIlL730Ej/84MBICCHqko1vw5E10CTkUlmzcGMsf/uBTt9dWDPfyhvVwOaTm1lwYAGhfqGMbj/apfuqCYfuAWitfwR+LFX2WLHX5X5CrfV6IKYmATZkNpsNs9m908UK4TKWPHhvKJw/AZZcCI+v8Q3e0vaeOM+tH2wqmtfnt+eeTFWcw7+qlhwxPsfcsXOJDPDcObXq3lxAbpaTk8PEiRM5e/YsFouFp556iokTJwLwySef8NJLL6GUomfPnnz66aecOnWKmTNncviwMQf4O++8w6BBg7juuutISUkhLy+PP/3pT8yYMQMwzjLuueceli1bxttvv01OTg6zZ8/Gz8+PIUOGuO1zC+E06QeNKZovnoP0/RA5DMJioOMVgHHQPnm+nPn7qyEx+SzpOQVM6t2GoCbGU71eZsVVMS2dsv3SMvMy2Z2+m8w8Y7qW9s0qvtHsbnUuATy/+XmSMi+/Ild1dAnqwoPxDzrU1tfXl4ULF9KsWTPS09MZMGAAEyZMYM+ePTz11FOsX7+ekJCQovl6/vjHPzJ8+HAWLlyIzWYruoQ0b948goKCuHjxIv369WPy5MkEBweTm5tL//79efnll8nLyyM6Oprly5fTsWNHbryx6pNaCeFxPh4P2ccvve/9e+hpTFqYb7Ux4a21WGzOHQvy5zGdaFvDh7oc8fTGp/n56M8AxITEYFKePdtOnUsA7qa15pFHHmH16tWYTCbS0tI4deoUy5cvZ8qUKYSEGNcwf5uyefny5XzyyScAmM1mAgKMyaXeeOMNFi5cCEBKSgoHDhwgODgYs9nM5MmTAUhKSiIyMpLoaOPm1y233MKcOVWf2EoIt1n/JpzeW7Is9wx0nwQDZ4HZ2/j2X8hm11hsmukD23Nd7zZOCSGgsXetHPwPnTvEz0d/JjIgkqcGP0XbptV/Erm21LkE4Og3dVf5/PPPOXPmDImJiXh7exMREVFimmdHrFy5kmXLlrFhwwb8/PwYMWJE0TZ8fX3lur+oP5Y9Dl6NoXGx+XSatjJu7oZfftqa1s0b07ud+xdMqYqfk41v/iPajqBni55ujsYxdS4BuFtWVhahoaF4e3uzYsUKjh49CsCoUaOYNGkSf/7znwkODiYzM5OgoCCuuOIK3nnnHWbPnl10CSgrK4vAwED8/PxISkpi48aN5e6rS5cuJCcnc+jQITp06FC0CI0QHm3hvbDbOLvFboUBM2HUoxX3AR79bidfJ6QCrr9JW1MWm4XrF13PydyTRWVWuxWF4v4+rpvwzdkkATjIarXi4+PDzTffzPjx44mJiSEuLo4uXboA0L17d/7+978zfPhwzGYzvXv35qOPPuL1119nxowZfPDBB5jNZt555x3GjRvHu+++S9euXencuTMDBgwod5++vr7MmTOHa665Bj8/P4YOHUp2dna5bYXwGCe2Q7NW0OVaUCbodVOZJlprtiSf5XzhtMsA6w9mEOLvw8Rerbk21v3r5VYkqyCL5PPJ9G/Zn27Bl9bNjgiIQHl48ipOEoCDdu/eTYcOHQgJCWHDhg3ltpk+fTrTp08vURYWFlZigfbf/PTTT2XKgDLPGYwbN46kJOfe9BbC5UK7wdgnL1t96Ewuv3uv7P+ja3q24oFxXVwZmVOsTTMWWBrVbhQ3dS2b4OoKSQAOePfdd3njjTd47bXX3B2KEJ4rbStseheyUitdgzfPYqw+9+g1XekfGVxU3iHUMxclSr+Yzlu/vkWezbhXl3LemKNoRNsRboyq5iQBOGDmzJnMnDnT3WEI4dl2fgM7voTASIgcXnl7oH1wE2LC3b84emUSTyWy4MACwvzCaGQ2nifoHdqb4MbBlfT0bHUmAWit69S1NeF6jqxmJ2pRwgfGHPx/2lZp0398vwsAUx35L734yGIA5oyZQ1TzKDdH4zx1IgH4+vqSkZFBcHCwJAEBGAf/jIwMfH1dO6eLqEReljGHDxq8/cC7cYXN7XbN+kMZnMwyLqUMiPLsb9DZBdlsObml6MneiIAI9wbkZHUiAYSHh5OamsqZM2fcHYrwIL6+voSH13zOdlED616HNS9fet+z4gXPd6RlccsHmwCY2q9tjVfccrWPd3/MezveA+rGk71V5dl/+4W8vb2JjPTcCZWEaLAsecY3/zuNh6AI6VyiOs9i44kf9hQN98zIKQDguetjmNTHOU/6utKS5CWYlImvrv2KVv6ePTS1OupEAhBCeDBlhpblT/p78HQO/9l0jLBmPkXf9nuGBzCySyg+Xp7/xLvFbsGu7XQO6lx54zpIEoAQwjEfXg1H15Ut9700iufhb3cwf3NKmSbPTIrhiq5hrozOaRYeWMjjGx7Hro0ppMdHjXdzRK4jCUAI4Zij67C36k1y4CCs9ksjsLIDOnFihzG756bDmYQHNub6PpfuzTT2Nnv8zd7ijmQdQaGYGWsM/R7TvmbLUHoySQBCiMrlnQfgtDmUUVsHl9Pg16JX1/RsxZ/HdKqlwFzD2+TNfb3uc3cYLicJQAhR1vkTsPQxsBbOdGszbt4uyjQWOJn7+zjaB5c/xXJtTL3sKu9tf4/lKcvdHUatkQQghCjr2HrY+ZXxVK+X8ayFbhnD/44al3YGdQzGr1H9OnxorXlvx3s08W5Sry/7FFe//gWFEDWX9CMsuNt4ffPXEBLNuoPpTJ+3GavWzBrZsd4d/AFuX3I7FruFydGTmd13trvDqRX1719RCFFGdp6FlfvOYK9s+gxtZ9C612ihbezq8QCHUhpDahobDmVgtWvuGR7F1HjPX+nKUTa7jVWpq7hovciejD0ATOk8xc1R1R5JAEI0APM3H+OZHyufVryHOsxEH+NJ3ckJ3chP2F5U18hs4r6RHWnm6+2yOGvbr6d/5U8r/lT0/u6Yu2nj7/kPqDmLJAAh6onvt6Xx3+0nyq07nG6sM7H0/mGYi83AFrjpJRql7y56b84/Cyfg1Nh/81N0yevgAY2969XBX2vNM5ufAeCFYS/QLbgb4f4Na2oRSQBCeCi7XWOx2x1u//mmY+xKyyIiuOyc+v5mO9f1CKZjkHfJCRV//Tf4NDVW8PpNu4GE9RgFzfxrEr7Hy7Zkc+DsAQAGtBpAoG/dWoPYGSQBCOGhpr2/kU1HMqvUZ1CHYP5zd6klRte/CT8/CueAp8vp1HcWXPFYteOsq15LNBZ4ejj+4QZ58AcHE4BSahzwOmAG5mqtnytV/2fgLsAKnAHu0FofLaybDvy2IvRTWuuPnRS7EPVW0snzbDqSSc/wAK7s3tLhfkM6hhgvck7DoeWgNexfAt5NYNhfynZQJohpODc9j+ccJ+FUAgA703cCcFXkVe4Mya0qTQBKKTPwNjAGSAW2KKUWaa33FGv2KxCntb6glLoXeAG4USkVBPwTiAM0kFjY96yzP4gQ9cn9Xxo3X4dGh3DfyI5V38CaV2DTO5feh3aDoeUkgAbm5YSX+fnoz0XvB7Ue1GC//YNjZwDxwEGt9WEApdQXwESgKAForVcUa78RuKXw9ZXAUq11ZmHfpcA4YH7NQxeifsnNt/LwtzvJzrNwJD2H2PAA/jKmGrNQHttkHPx9AuCeVUaZf6hzg/Vgq1NX80XSF+XW7c7YTVRAFG9d8RYAoX4N5++lPI4kgDZA8en9UoH+FbS/E/ipgr4NZ4yVaDDyrTastpotUbkzLYtF248TEexHp7CmTB8UgcnRNRNt1kvTNuz5zvjZ++ZKF2evTy5YLgDww6Ef2HhiI50Cy85H1KpJK0a1G0XbpvXnWYaacOpNYKXULRiXexxbEfpSvxnADIB27do5MyQhXO5kVh4jXlpBnsXxETsV+dfEHgzv1MLxDlrDG70h69ilskb+MO5Zp8RTF/x72795Z/ulS17tm7Xni2vLPwsQlziSANKA4ukyvLCsBKXUaODvwHCtdX6xviNK9V1Zuq/Weg4wByAuLk5W+hZ1yhdbjpFnsXND33A6hdVs6KSfFwy8uAq2XnS8k7YbB/8OoyBqpFHWon4uYFKeC5YLfLjrQ5p6N2VGzxkAdA/p7uao6gZHEsAWIFopFYlxQJ8K3FS8gVKqN/AeME5rfbpY1RLgGaXUb3dZxgIP1zhqITxE2rmLvLbMGEt+U/929GlXwxuKRzfAh3dWr2/XCRB3e832Xwf9eORH8mx5hDcN57Yet7k7nDql0gSgtbYqpWZhHMzNwDyt9W6l1BNAgtZ6EfAi4A98XfiQyTGt9QStdaZS6kmMJALwxG83hIWo0/YsQid8SHrqOT7xthDVwp/wVXNrvt0Lhf89pnwMbfo63s9khqb1b81aR3y+93MA3hn9TiUtRWkO3QPQWv8I/Fiq7LFir0dX0HceMK+6AQrhUWxWKMiB7V/A0fXYLOH4Kwjx9oZ8a823b/Y2LuNEDIUmdWcVrdpywXIBqy7595yZZyR3zDRzAAAgAElEQVTNFo2rcN9EAPIksBBV88mEonVx9+l2XF/wBM9eH0OfeBm84GorU1byf8v/r9y6GzvfiNnk+YvMexpJAEKUIzffyv92nsBis+Obn074qZWAptfxXWQH9OBw2Die2+lP26DGXB3TMC+9ONv+s/vZdnrbZesTThpP8N7X6z6aeF+a70ihGNlupMvjq48kAQhRjh93nuCBb3YA8JDXfCZ7/beobm7GcF481QeAV0Z3IqBx/Zkh052e2fQMiacSK2zTxLsJ07tPp7FX41qKqn6TBCBEOXamZQGw5Lb2dP7iv2izDxl3GmMZpjQJZYpSmE2KYH8fd4ZZZ6Vkp/DspmcpKFxrGCApM4m4sDheGPbCZfs18W4iB38nkgQgRCkXC2zY7BofCuiYvRkA1WEUIa3buzmy+uF8wXk2HN/AmrQ1dAvuhq/ZWHO4c2BnxncYTws/uZlbWyQBCFFM1gULA579BbvlIgm+f8D8P2N6AcY9497A6onSN3JfGfFKg1qBy9NIAhDiXArsXwxAQXY+N9gP0L9tI5qeuQDdroOu4yGw4cyp40o/Jxszcf6x9x8JbxpO6yat3RxRwyYJQIi1r0LCBwC0AJ70xljVAoy58rte667I6p11x40htDd3vRk/bz83RyMkAYiG4eJZ+PYeyM8uW5dxEN0klCfazmVXWhaHzuTy8R3xxLQNgsYNd6746rpgucCDax7kfP75MnXn889zZcSVcvD3EJIARP1WkAv5OZCWCAeWQFgMNG5esk2Lzlxo1Z8PV+TQNsifuG6tiWjXDurRAui1ISs/C4vdwv6z+1mZspKOzTsS5BtUok2fsD5cGyVnVJ5CEoCovyx58HJXyM+6VHbVcxAxpESznalZTHh7LQC/HxDB3cOiajPKemHLyS3cseSOEmV/7vtnhoYPdVNEwhGSAET9lHHIWBglP8u4kRs5zJgjv23ZtYzSc/LRGmaN7MiUuHA3BFt3rTi2grScNPZm7gXg3th7CWkcgq+XL/1bVbRulPAEkgBE/bT6RdheuPJozA3GSJ5KjO4WRnO/Ri4OrP6w2CzMXjkbuzYWwmns1ZipXaaWuewjPJckAFE/HVkDzdvDvevAp2m5TbYkZ/LC4iQycwvKrRcV02js2s49Pe/h1m634mP2wdfL191hiSqQBCDqh7zzUHzUiY8/5JzC4tWEM+fKX11r8a6TbEk+y5COIXRu2ZSOoTVbzauh8vXyJcAnwN1hiGqQBCDqPksevNq9ZAIA6DaR2V9u4387Tly2q7dZ8emd8RQuZCSq4MxF42GJi9YqLF8pPIokAFH3FeQYB//u10OHS9MCW9sN4X8v7SUqpAn3DC9/ZE/bID85+FfTujTjoa5wf7lxXldJAhB1X+Zh42fLGOjz+6LiXSnnAAhs0ogb+8mCLc722d7PAIgOjHZzJKK6JAGIustaAF/deikBtOpZovqbxBQA/niFHKCc7dM9n3Ik6wiDWw+mR0gPd4cjqkkSgKi7sk8Yk7iFdjPG+rfqXVSltSYh+SwAMW3kBmVN2Ow2TuSWvI/yv8P/A2Byp8nuCEk4iSQAUXdteNv4OeR+6Pm7ElWfbzpG0sls+rRrTlATGdtfEy9seYH/JP2nTPmQNkMY036MGyISziIJQNRddovxs0vZuWUWbE0F4Nnre5apExXblb6LjSc2Fr3fcmoLIY1DmN1ndol2vUN7l+4q6hhJAKJua9ICGpWdWfLXY8YN4E5hMra/ql5NfJXNJzeXKBsWPoyJHSe6KSLhKpIARL3UyGxi+qD2MsSzAidzT/Lg6gfLjONPPp9MXFgc7455t6jM2yQzo9ZHkgBE3WUtALu13CqlwGSSg//lXLBcYP3x9Ww9vZXeob0JaHTpRnmYXxjjIsfhY5YF7+s7SQCi7jq2Aaz57o6iTrp76d3sOLMDgEf6P0KXoC5ujki4g0MJQCk1DngdMANztdbPlaofBrwG9ASmaq2/KVZnA3YWvj2mtZ7gjMBF/bMi6TS70rIqb6jtxJxYQO/ciyhTMz7+5UCZJla7dkGEnu3Y+WMsTl6M1pV/9qPnjxLbIpa7Yu6ic2DnWohOeKJKE4BSygy8DYwBUoEtSqlFWus9xZodA24D/lrOJi5qrXs5IVZRzz2ycCcnsvIqbddBpfF/Ps8D8B/rKF5eur/cdpHBTZwan6f7bO9nzE+a73D7wW0GM6LtCNcFJDyeI2cA8cBBrfVhAKXUF8BEoCgBaK2TC+vsLohRNABncws4kZXH7+LCeWZSTJl60/d/QJ3YZryx5UMm2CbN5Xfdr+d3l7nR62U2uTJkj/HfQ/9l3q55nL5wmkCfQJb/brlD/bxMcgW4oXPkN6ANkFLsfSpQlaV+fJVSCYAVeE5r/V3pBkqpGcAMgHbtZM6Whibt3EUSjxpP7YY18y174M5Kgz0LoVlrCOtulLWJwxw1FLzMtRytZ0i/mF606PrPyT+TlpPG4NaD6dmipxzYhcNq4zelvdY6TSkVBSxXSu3UWh8q3kBrPQeYAxAXF9fwLt42YLvSsrj2zbVF7wd1CCnZIP0gvNXXeN3jBhj191qMzjOdLzjPmG/GYC02Aqpj8468OvJVN0Yl6iJHEkAa0LbY+/DCModordMKfx5WSq0EegOHKuwkGozvEg5xt/kHrohqQmgzHyKObodjxS7pnC/8VRv6Fxh4n3uCdIO9GXtZkbKi3LocSw5Wu5XJ0ZMZ0GoAIDNyiupxJAFsAaKVUpEYB/6pwE2ObFwpFQhc0FrnK6VCgMHAC9UNVtQ/ydtWMtf7PyUvMpbm7Qcxv4PGzWstLnd7f+f7LD269LL1XiYvro68mvhW8bUYlahvKk0AWmurUmoWsARjGOg8rfVupdQTQILWepFSqh+wEAgExiul/qW17g50Bd4rvDlswrgHsOcyuxINjM2uuZhfAI2A2xdD+4HuDqlWLTq0iLk755ZbdzL3JJ0CO7FgwoJajko0JA7dA9Ba/wj8WKrssWKvt2BcGirdbz1QdkiHEEDSyfO0wIFx//VQnjWPpclLOZFzguFth5ep7xTYicGtB7shMtGQyHAB4RYbDmUw/f017Pf9t1Hg7evegGrZo+seZWXqSiIDInlp+EvuDkc0UJIAhFvsP5WNH8ZDX7YOozG3qt/PClrtVj7b8xnnC4yhm7vSdxHRLILXRr7m5shEQyYJQLjFuQsWItQpAMwRg4zZ2+qxA2cP8HLiyygUJmU853B99PVEBZS/WL0QtUESgKh1D3+7g592nWSqqXA8QFj9uk1ksVmYsXQGpy+cLirLtxmT1r0x6g2ZfkF4DEkAolZdLLAxf3MKHQK9uTYkF04D4XHuDsspcgpyOJZ9jHN550g4lUCP4B60a3bpyfbGXo3p1aJ+X+oSdYskAFGr/vH9LgCe8f+SHqf/C74B4Bfk5qic42+r/8batEtPNU/rOo0JHWTyW+G5JAGIWmO3a75JTAU08ekLwS8Ypv/X3WE5TcLJBDoHdua+XvfhbfYmvqU8pCU8myQAUWuSM3IBiA+xoHJsoMyXJnerBzQam7Yxst1Id4cihEMkAQjX2voJrH4RgIDcAtY0shFiK5zBs55M7GbXdmYsnUGBrYD+raoyUa4Q7iUJQLjOuRQsu77HlJNOVsRVJOZmkq2tXBPZCnx9oeNod0dYY1pr9mbuZdOJTXQL7sb4qPHuDkkIh0kCEK5hzYe34vC25rHPHs6VuyYDcHVMSyb/rq+bg3Oerae3ctvi2wCY2GEi3UPqzyUtUf9JAhDOVXAB1r0GF8+CNY8vrSP4OeQWPhzXD4CY8AA3B+i4fZn7WHRoUYVtUrKNaUwf6f8I10dfXxthCeE0kgCEc6UlwqrnwcuXi+am/FQQT4+uPRnZJdTdkVXZ/KT5LDiwAD8vvwrbhfqFMqb9GHzMPrUUmRDOIQlAlLXzG1j6GOhqLM5mNeb32TTkA25crGjq68VHV3Z2coCutf/sfmavmM2ZC2cI8wtj2ZRl7g5JCJeQBCAu0RpO7oS9/4Wc0xA7tcqbyMm3kYsvC08GA5k8OK6L8+N0sX2Z+0jJTmFM+zEMbTPU3eEI4TKSAMQlR9fDR1cbr5u2holvVal7vtVG38d/Jt9qBzJp5uvFLQPaOz9OF9udsRuAB/o9QMsmLd0cjRCuIwlAXLLlfePnVS9Cxyuq1HVF0mmW7T1FvtXO1H5tuTqmFW0CG7sgSNeya3vRjV85+Iv6ThKAuGT/EuNn9+vAv2o3bd9YfoAdqVkENWnEVTGtGNaphQsCdL3jOcfJLsh2dxhC1ApJAMKw6gWwXID+Mx0++N/9SQK/HjsHwLkLBQzpGMLHd9Tt+W/+vc1YoeyZIc+4ORIhXE8SgICzybDvJ+N1r5vLbXI6O4/DZ3JLlK07mE7bQD/6RgQCcFWPun/JJDUnFYCBrRvWAvWiYZIEIOC94ZB3DqJGQKue5Ta559PEom/7xY3tHsZfxtatYZ4VUSj6t+xPSOMQd4cihMtJAmjozh41Dv4xU+DKZ8tU7z1xnvmbj3H4TC7xEUHMHhNdVKdQ9GrbvDajdZqFBxYWjfYp7lj2MToEdHBDRELUPkkADd3ewqkOOo4G/7I3br9OSOWTDUcJatKI0d1CGdShfnwzfm3ra+QU5NDEu0mZutjQWDdEJETtkwTQEB1aDt/eA3YrWC4aZV3H8/T/9hQu2HJJboGNZr5ebP3HGDcEWnOzV8wm8VRimfJz+eeY2nkqfx9QP6akFqI6JAE0JBmHICsF9iyC3NPQZzqYG2Fp1paElDxW7juDr7eZMd3CSnTr3rqZmwKuuV+O/ULH5h2JCyu57rBSisnRk90UlRCewaEEoJQaB7wOmIG5WuvnStUPA14DegJTtdbfFKubDjxa+PYprfXHzghcVMP7o4zr/QBmH7jqBfD2Zd6qQzz7/kYAruwexhMTe7gxSOe5YLkAQIfmHeSbvhDlqDQBKKXMwNvAGCAV2KKUWqS13lOs2THgNuCvpfoGAf8E4gANJBb2Peuc8EWVFORCzO8g7nbwDwNvX5btOcWi7ccB+OqegXQOa+rmIJ3HYrcA0DWoq5sjEcIzOXIGEA8c1FofBlBKfQFMBIoSgNY6ubDOXqrvlcBSrXVmYf1SYBwwv8aRi+oJCIf2g4revrvqEAdO5TCkYwjxkUFuDMx1fL183R2CEB7JkQTQBkgp9j4VcHTh0/L6tindSCk1A5gB0K5dOwc3LQBeXbqfeeuOONR2K3Y+WnuYN9YuKSrLzbcyuGMIn95Z/9ayzbiYAYDNbnNzJEJ4Jo+4Cay1ngPMAYiLi6vGJPQNz94T5zmdnc/KfadpZDYxoVfrijtoO95bbXRp2Ywb2oSXqCp907euS7+Yzr7MfRw6dwiAZj519ya2EK7kSAJIA9oWex9eWOaINGBEqb4rHewrLiM338q1b67FZjdy5aAOwfxzfCVr0Savha0wNMKfoePq97q1/1j3D9amrS16HxUQ5cZohPBcjiSALUC0UioS44A+FbjJwe0vAZ5RSgUWvh8LPFzlKAXfJKay6bBxSaPAZmeqWsoNbU7RNsiPpr5e8N2nFW/gfOH4/q4TXByp+2TmZfLvbf9mb8ZeugZ15ZH+j+Dn7Ud08+jKOwvRAFWaALTWVqXULIyDuRmYp7XerZR6AkjQWi9SSvUDFgKBwHil1L+01t211plKqScxkgjAE7/dEBZV8+byA5w6n0eQXyMA/ue9gGZZFsyWKty4De0GIfX3YLjl5Ba+3Pclwb7BjGw7kl6hvdwdkhAezaF7AFrrH4EfS5U9Vuz1FozLO+X1nQfMq0GMDd7iXSc4mnGBSb3b8GrMMfjuD0A29L4brnnJ3eG5zfq09fx11V+xaisAVrvxc96V84hqLpd9hKiMR9wEFpekZF7gwOmSC5Is3nUSgNsGRcCaZ6EgGwb932Wnbm4ILlgusOzYMrIt2UzrMg0fsw8AAT4BtG9W95ahFMIdJAF4mBmfJrL3xPky5f4+XsQ0L4D9i42CMU+CUrUcnef4ePfHfL3/a7xN3tzf934ae9W95SeFcDdJAB7k4Ols9p44z9DoEP5aao79VioT05KHjDcjH22wB/9fjv3CL0d/YU/GHrxN3vxv0v/k4C9ENUkC8CA/7TQu9YztFkZs6Xn2t/4Xdi2AoCiIrpszczrD53s/Z/vp7bTwa8GItiNo5d/K3SEJUWdJAvBAN6kl8OQ/MKZPKvTb06zTf4CAMg9T13uPrXuMHw7/gMVuoV/Lfsy7UsYVCFFTkgA8kGnHF4CGgfeVrPAPg2aVPPFbD53MPcnCgwsJ9w/nyogrGdxmsLtDEqJekATgYRqThzq+1Xgz+nF3huIxntjwBGCs1DW772w3RyNE/SEJwIMc2pPAi94fGm+GPeDeYDzI5pObCfAJ4PGBj7s7FCHqFUkAHqRP7lquNW9Ct+iC6jja3eF4DD8vPwJ9A2VaZyGcTBKAh1i96wi/z/sMADVzHZjlnwbg/hX3czb/LGMjxro7FCHqHTnK1KJDZ3LYdzK73Lpd23cxDDgbNpDABnzwz7iYUWIR919P/wrATV0dnX9QCOGohnukcYNZ//m13Kd8Ae4wLwFvaDLg9lqOyrO8+eubLDiwoETZjZ1vlCmdhXABSQAu9vmmo6w7mA7A0YxchndqwSNXl12jtv3XT0EGNOpcfx/yWnRoEStTVlbYZmf6TkL9Qnl39LtFZRHNIlwbmBANlCQAF7DbLz3A9cHaI5w+n0+rAF/aNG/MNTGt6Nyy1MLrWkOusTA7vgG1GKlr2HXppaEN/9n7Hw5nHaZ1k8s/y9DEqwmD2gwiOrD+TlsthKeQBOBku9KymPzOevKtlw6CE3u15vWpvS/faf40yDsH/e4Ck7kWonSddWnrmPXLrKIpmksbHj6ct654q5ajEkKURxKAky3edZJ8q51bBrSjhb8xbPHKHpWsuXsmyfjZ/14XR+c6FywXWJu2ljVpa7BqK7d3v53G3mUnaRsWPswN0QkhyiMJwMneWnEQgHtHdKRNcwdnqVQm6HEDhHR0YWSu9e2Bb3l+y/MA+Jh9mBk7Ez9vPzdHJYSoiCSAGli65xRfJ6SUKZ8Q29rxg389kW/LB+Dr8V8T6hcqB38h6gBJADXwTWIKK/efISqkSVFZt1bNmNjLgQnb7Haw/3adXFfY1NNY7JYyIdu0MVtpRLMIeWJXiDpCEkANRYU0YfHsKl7X1hre7ANnj1wqC+/n3MBc5Ofkn/nrqr+iL5O0TMpUyxEJIapLEkA1XCiwsmzvaY6fy6veBs4fNw7+kcMhsjB5dLnWeQE60UXrRValrDK+9QOrUleh0dwbey/eJu8Sbdv4t6GRuZE7whRCVIMkgGpYtO04D327E4BBHYKrvoGVzxo/u00whn56sMVHFvPY+sdKlPl7+3N3z7vLJAAhRN0iCaAaCmzGGP/v7htMl9IPdTnCVgBevhB3p5Mjc47EU4l8tOsjNJq0nDQAPrv6M4J8ggAI8A2Qg78Q9YAkgBpoG9gYX+8qPLhlsxh/7FZo2tIjF3a32Cz8dOQnVqetpnNgZ7xN3owIH0H34O54meTXRYj6RP5H15bcdHg9FgpyjPfBnjfmPykziWn/m4bVbqW5T3O+Gv+Vu0MSQriQQwlAKTUOeB0wA3O11s+VqvcBPgH6AhnAjVrrZKVUBLAX2FfYdKPWeqZzQq9jzqcZB/+YKRDWA9r0dXdEZfyc/DNWu5VpXaYxtM1Qd4cjhHCxShOAUsoMvA2MAVKBLUqpRVrrPcWa3Qmc1Vp3VEpNBZ4HbiysO6S17uXkuOueDOMJYTqNg5gb3BvLZXy0+yMAft/t94Q3DXdvMEIIl3PkDCAeOKi1PgyglPoCmAgUTwATgccLX38DvKWUB17gdpLLzelfod9umrbo4txgquh8wXkeX/84FywXytRZ7BaujrxaDv5CNBCOJIA2QPH5DlKB/pdro7W2KqWygN/GR0YqpX4FzgOPaq3X1CxkT2DktuZ+nj/m3a7tJQ72O87sYOnRpUQGRNLUu+QIpp4tenJN1DW1HaIQwk1cfRP4BNBOa52hlOoLfKeU6q61LvEVWik1A5gB0K5dOxeHVDMWm535m4/h623CbPL8k5w/rfhTuYuwPDbgMeJaxtV+QEIIj+FIAkgD2hZ7H15YVl6bVKWUFxAAZGitNZAPoLVOVEodAjoBCcU7a63nAHMA4uLiPHpinILCef4HRFXjAbBaci7vHCtSVmDTNvZk7KFj845c1/G6ono/bz9iQ2PdGKEQwhM4kgC2ANFKqUiMA/1UoPQK3YuA6cAG4AZgudZaK6VaAJlaa5tSKgqIBg47LXo3qtYTwLXkmwPf8PrW14vej2w7kundp7sxIiGEJ6o0ARRe058FLMEYBjpPa71bKfUEkKC1XgR8AHyqlDoIZGIkCYBhwBNKKQtgB2ZqrTNd8UE8gs0K3/8Bsk+WrctNr5UQ0i+mFx38l96wFIWihV+LWtm3EKJucegegNb6R+DHUmWPFXudB0wpp98CYEENY/RsdruxnCMYB/4dX0JgJPiXWgXMp6kxBDQo0mm7ttgs5FpyS5QlnDSurnUP7k6YXxj1eDCWEKKG5EngKjp53pgBNN9SuObvolmw7fOSjYb9DXrf7PJYJi2axNHzR8ute2zgY3LwF0JUSBKAA3aknmN7ahYAqWeNIZUdfc7ClrmQshmat4eB9xmNzd7QdbzLYll2dBkZFzMASMtOo3+r/oxsO7JEm6aNmtIlyL3PGwghPJ8kAAc8uGBniYe/TAr6p3wA+74wCrpPgv73uDyO0xdOc//K+0uUjWw7kpu7uv5sQwhR/0gCcIDFZmd011BeHGSnyZonMWkbXqkHoFk4zFgBfq4bEWSxW3h4zcNkXMygwFYAwAP9HuCqyKswKRNBvkEu27cQon6TBFARuw0uZNDcfpYQLAQeWw/H1kC7QRDS2VjNyz/U6bu9aL1YdHP3VO4pliQvIaJZBCGNQxjUehBD2wwlpHGI0/crhGhYJAFUZNH/wbbP+QYgBzgCKBP8/nvwcs00EPm2fK74+gqyC7JLlN8Tew/XRnnmspFCiLpJEkB5dn9nDOlM2YS9eXseOzOKHq0DmBrf1rjh64KDv8Vm4YfDP5CRl0F2QTZj2o+hf0tjyqVG5kaMajvK6fsUQjRskgBKy82Ary89NXum3TV8dnIM1wS2Ymq/Pi7b7a+nfy2x9u5VkVcxpv0Yl+1PCCEkARSXvA6W/ROAdVGzeT97EKmZ3sBF7h3ewam7enf7u6w/vr7o/fl8Y5TRO6PfoVeLXvg38nfq/oQQojRJAL/RGvYugtQt5EdcwZyTHdmVb6JLKz+iWwYQ1aKJU3f3RdIXaDTRgdEAhPiFEBEQQWyLWDn4CyFqhSSA3yx/Eja9i8XkQ+ekOwGY2CuE16f2dvqutp7aSkZeBqPbjebVka86fftCCOEISQD5OZxc8yGmHYtp5tWM55o+gq/FxD/Hd2dIR9cMtczIM57kLT5FsxBC1DZJAAeW0HLtowBssnfhoxPt6NbKn2nxrl+YppV/K5fvQwghLqdhJYBl/4IjqwBIz8knM7eApvbztAJmN3+DJ2fcyE6TmcbeZvfGKYQQtaDhJICcM7B9PnatKQjuSsrFbHKtXuT5BnDY1J2BA4bQ1M/XpSForTl14RRn8866dD9CCOGIhpEAUhNg7hUAfGUfzUPpdwDQPzKIL+8ZWGthfLrnU15MeLHofSOT5y8qL4Sov+pvArBc5Pz6eew9doqA3MN0ATaE38UrB3szqXcbBnYIpk+75i4P42TuSRYfWYwdO2tS12BWZv458J8E+ATQvll7l+9fCCEup/4mgOR1NFvxCP0L317Ujbj/YCzpKpAJsa0Z2cX5k7iV56t9X/H+zveL3ncI6MCk6Em1sm8hhKhI/U0AhU/03pD/GJ8+OgNM3qwwe6MU+FbhJu/iI4uZt2tetcM4deEU3iZv1k1bB8hlHyGE56i/CeDcMQB26wgaN2larU1YbBYWJy/mcNZhBrQaUK1thPqF0imwE429GlervxBCuEr9TQAmMx9ZxxIXHV7tTTy67lF+OfYLEc0ieOuKt5wYnBBCuF+9TAApZ87R9qIx1DK0acVDO/ef3c+qlFXl1u1M30m7pu14afhLTo9RCCHcrV4mgOW//MR0wIyd/lEVL5k4Z8ccliQvuWz9hA4T6BzU2ckRCiGE+9W7BHBs/zYG7HkKTDDttv/Dq2PbMm0eWvMQB84eAOB4znE6BHTg6/Ffl7s9L1O9+ysSQgigHiaA9F9/oI8plZ3+Q4hpE1uirsBWQGpOKj8d+Yl2TdsRFRBFuH84g9sMxtvs7aaIhRDCPRxKAEqpccDrgBmYq7V+rlS9D/AJ0BfIAG7UWicX1j0M3AnYgD9qrS9/vaWmcjPos9d40jZkyivQOLBE9T/W/YMfj/wIwLVR13JP7D0uC0UIITxdpQlAKWUG3gbGAKnAFqXUIq31nmLN7gTOaq07KqWmAs8DNyqlugFTge5Aa2CZUqqT1trm7A8CwIGfWdzEjyXeHel0djmcXV6iemf6Tto2bcsf+/yRwa0HuyQEIYSoKxw5A4gHDmqtDwMopb4AJgLFE8BE4PHC198AbymlVGH5F1rrfOCIUupg4fY2OCf8UrbM5dGQIPJN51i2/d/lNpnYYSLjIsa5ZPdCCFGXOJIA2gApxd6nQtEMC2XaaK2tSqksILiwfGOpvm2qHW0FUk4c4P9IJd/UiO5+1zH/hidcsRshhKg3POImsFJqBjADoF276i3EYvZqRJBujskWxrTu12KcgAghhLgcRxJAGlB8LGV4YVl5bVKVUl5AAMbNYEf6orWeA8wBiIuL044GX1zrFu2ZN2NTdboKIUSDZHKgzRYgWikVqZRqhHFTd1GpNouA6YWvbwCWa611YflUpZSPUioSiAY2Oyd0IYQQNfB7QXEAAAanSURBVFHpGUDhNf1ZwBKMYaDztNa7lVJPAAla60XAB8CnhTd5MzGSBIXtvsK4YWwF7nPZCCAhhBBVoowv6p4jLi5OJyQkuDsMIYSoU5RSiVrruKr0ceQSkBBCiHpIEoAQQjRQkgCEEKKBkgQghBANlCQAIYRooDxuFJBS6gxwtAabCAHSnRSOs0ls1efJ8Uls1efJ8dW12NprrVtUZSMelwBqSimVUNWhULVFYqs+T45PYqs+T46vIcQml4CEEKKBkgQghBANVH1MAHPcHUAFJLbq8+T4JLbq8+T46n1s9e4egBBCCMfUxzMAIYQQDqgzCUApNU4ptU8pdVAp9VA59T5KqS8L6zcppSKK1T1cWL5PKXWlp8SmlBqjlEpUSu0s/DnK2bHVJL5i9e2UUjlKqb96UmxKqZ5KqQ1K/X975xZiVRXG8d/XTN4RR+kmPZiUDxpqhUZQvYiVBRUlVFJqPVqE0EXKHqYgKLOi8KGXMIsCUyqCJB2FUKKI8jKlkpdRZLKwMonRyEtfD2spe7Y648xe55w97P8PNmfttdfa+8/aZ/Gt216fbY9lOKQs+szsYjNbEXXtNLPnGqDtVjPbbGYnzWx27to8M9sdj3n5vI3SZmZTM++03cweKIu2zPWRZtZpZstSayuqL9bVdfE/tyNfl8/C3Ut/ELah3guMBwYB24CJuTQLgHdi+EFgZQxPjOkHA1fF+zSVRNt1wNgYvhb4pUxll7m+GlgFPF0WbYStzNuBKfF8TMr3mkDfHII/bIBhwH5gXJ21jQMmA+8DszPxo4GO+NsSwy0l0TYBuCaGxwK/AqPKoC1z/S3gI2BZyv9bCn3AV8DMGB4BDOvpeQOlB3DGMb27HwdOO6bPcg+wIoZXAzPMujumd/d9wGnH9A3X5u5b3P1gjN8ODDWzwQm1FdIHYGb3AvuivtQU0XYb0O7u2wDc/U9P72uiiD4HhlvwkDcUOA78XU9t7r7f3duB/3J5bwfa3P2wu/8FtAF3lEGbu+9y990xfBA4BPTp46ZaaQMwsxuAy4B1CTUl0WdmE4Fmd2+L6brc/VhPDxsoBuBcjunzzuW7OaYHso7pe8vbKG1Z7gc2u/u/CbUV0mdmI4BFwIuJNRXWRmgpupmtjd3hZ0umbzVwlNCCPQAsdffDddZWi7x1u7+ZTSe0gvcm0gUFtJnZRcDrQPKh0AxFym4CcMTMPjGzLWb2mpk19ZShFE7hq46ZTQJeJbRqy0Qr8Ka7d8UOQZloBm4GpgHHgA0WHGJsaKysM0wHThGGMVqATWa23t07GitrYGBmVwAfAPPc/ayWeINYAKxx984S1gcIdeIWwtDyAWAlMJ/gsfGcDJQeQF8c02P9cEzfIG2Y2ZXAp8Bcd0/Z0kmh70ZgiZntBxYCz1twD1oGbZ3ARnf/I3Zz1wDXJ9RWVN8c4Et3P+Huh4CvgZTbChT5X5ehTpwXMxsJfAEsdvdvE+oqqu0m4IlYH5YCc83slbTyCunrBLbG4aOTwGf0VidST2LU4iBYtg7CJO7piZFJuTSP030y7uMYnkT3SeAO0k4CF9E2Kqa/r4xll0vTSvpJ4CJl1wJsJkywNgPrgbtKpG8RsDyGhxP8Yk+up7ZM2vc4exJ4XyzDlhgeXRJtg4ANwMLUdaGotty1+dRmErhI2TXF9JfE8+UEP+znf14tCrlGL+5OYBdhPHBxjHsJuDuGhxBWquwBvgPGZ/Iujvl+BmaVRRvwAmGceGvmuLQs+nL3aCWxAUjwXh8mTE7/BCwp0/+OsAJjVdS3A3imAdqmEVqFRwm9ku2ZvI9FzXuAR8uiLb7TE7k6MbUM2nL3mE8NDECC9zqTsDruR4KBGNTTs/QlsBBCVJSBMgcghBAiMTIAQghRUWQAhBCiosgACCFERZEBEEKIiqIvgYUAzGwMYf05wOWEr3h/j+dTCOurm4GdhK9Tj5nZKcJyu2bCWvpH3P1IXYULUQAtAxUih5m1Al3uvjSed7n7iBj+EPjB3d/Ixa8Adrn7y43SLURf0RCQEH1jE3D1OeK/Ie2GakLUHBkAIS6QuNfPLMKwTza+CZgBfN4IXUL0FxkAIXpnqJltBb4n7LL4bi7+N8Ie8W0N0idEv9AksBC984+7Tz1fvJkNA9YSNoZ7u77ShOg/6gEIURAP21E/CTwVh4mEGBDIAAiRAHffQtiF8aFGaxHiQtEyUCGEqCjqAQghREWRARBCiIoiAyCEEBVFBkAIISqKDIAQQlQUGQAhhKgoMgBCCFFRZACEEKKi/A+gxxO9vnoH3QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(cn_evaluation.fallout(), cn_evaluation.recall(), label=\"Common Neighbors\")\n",
    "plt.plot(aa_evaluation.fallout(), aa_evaluation.recall(), label=\"Adamic Adar\")\n",
    "plt.plot(jc_evaluation.fallout(), jc_evaluation.recall(), label=\"Jaccard\")\n",
    "\n",
    "plt.xlabel(\"FPR\")\n",
    "plt.xlabel(\"TPR\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple way to summarize the results offered by the ROC curve is through its AUC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Area Under Roc Curve (AUROC)\n",
      "Common Neigh.: \t 0.025717681925924538\n",
      "Adamic Adar: \t 0.02589950662044348\n",
      "Jaccard: \t 0.018159887369919418\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import auc\n",
    "\n",
    "print(\"Area Under Roc Curve (AUROC)\")\n",
    "print(f\"Common Neigh.: \\t {auc(cn_evaluation.fallout(), cn_evaluation.recall())}\")\n",
    "print(f\"Adamic Adar: \\t {auc(aa_evaluation.fallout(), aa_evaluation.recall())}\")\n",
    "print(f\"Jaccard: \\t {auc(jc_evaluation.fallout(), jc_evaluation.recall())}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"conclusion\"></a>\n",
    "## 3. Conclusions ([to top](#top))\n",
    "\n",
    "In this notebook we introduced the basic facilities offered by ``linkpred``.\n",
    "\n",
    "For any issue, suggestion, bug report refer the official [GitHub repository](https://github.com/rafguns/linkpred) of the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}