diff --git a/2021_scipy_tutorial/fairness-in-AI-systems-instructors.ipynb b/2021_scipy_tutorial/fairness-in-AI-systems-instructors.ipynb index 249bddf..fabd3f3 100644 --- a/2021_scipy_tutorial/fairness-in-AI-systems-instructors.ipynb +++ b/2021_scipy_tutorial/fairness-in-AI-systems-instructors.ipynb @@ -2,9 +2,6 @@ "cells": [ { "cell_type": "markdown", - "metadata": { - "id": "NDxtnoIr7mIN" - }, "source": [ "# SciPy 2021 Tutorial
_Fairness in AI systems:
From social context to practice using Fairlearn_\n", "\n", @@ -14,47 +11,57 @@ "[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)._\n", "\n", "---" - ] + ], + "metadata": { + "id": "NDxtnoIr7mIN" + } }, { "cell_type": "markdown", - "metadata": { - "id": "Sch9KDWg7SL8" - }, "source": [ "Fairness in AI systems is an interdisciplinary field of research and practice that aims to understand and address some of the negative impacts of AI systems on society. In this tutorial, we will walk through the process of assessing and mitigating fairness-related harms in the context of the U.S. health care system. This tutorial will consist of a mix of instructional content and hands-on demonstrations using Jupyter notebooks. Participants will use the Fairlearn library to assess ML models for performance disparities across different racial groups and mitigate those disparities using a variety of algorithmic techniques." - ] + ], + "metadata": { + "id": "Sch9KDWg7SL8" + } }, { "cell_type": "markdown", - "metadata": { - "id": "fboVvqvVvpKz" - }, "source": [ "# **Prepare environment**" - ] + ], + "metadata": { + "id": "fboVvqvVvpKz" + } }, { "cell_type": "markdown", - "metadata": { - "id": "o0p461hmgrmz" - }, "source": [ "## Install packages" - ] + ], + "metadata": { + "id": "o0p461hmgrmz" + } }, { "cell_type": "markdown", - "metadata": { - "id": "9b5Nsb2Rgux7" - }, "source": [ "Note that the runtime environment needs to be restarted after installing `model-card-toolkit`." - ] + ], + "metadata": { + "id": "9b5Nsb2Rgux7" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "!pip install --upgrade fairlearn==0.7.0\r\n", + "!pip install --upgrade scikit-learn\r\n", + "!pip install --upgrade seaborn\r\n", + "!pip install model-card-toolkit" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -62,183 +69,173 @@ }, "id": "w_uVNMHUdb2w", "outputId": "8ef912ea-10cf-47ef-c85d-574a044283e3" - }, - "outputs": [], - "source": [ - "!pip install --upgrade fairlearn==0.7.0\n", - "!pip install --upgrade scikit-learn\n", - "!pip install --upgrade seaborn\n", - "!pip install model-card-toolkit" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "fVtpwFGJLcgU" - }, "source": [ "## Import and set up packages" - ] + ], + "metadata": { + "id": "fVtpwFGJLcgU" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "cEwLsWyTLgJn" - }, - "outputs": [], "source": [ - "import numpy as np\n", - "import pandas as pd\n", - "\n", + "import numpy as np\r\n", + "import pandas as pd\r\n", + "\r\n", "pd.set_option(\"display.float_format\", \"{:.3f}\".format)" - ] + ], + "outputs": [], + "metadata": { + "id": "cEwLsWyTLgJn" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "wBOSsyc48MK0" - }, - "outputs": [], "source": [ - "import matplotlib.pyplot as plt\n", - "import seaborn as sns\n", - "\n", + "import matplotlib.pyplot as plt\r\n", + "import seaborn as sns\r\n", + "\r\n", "sns.set()" - ] + ], + "outputs": [], + "metadata": { + "id": "wBOSsyc48MK0" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "from sklearn.linear_model import LogisticRegression\r\n", + "from sklearn.model_selection import train_test_split\r\n", + "from sklearn.preprocessing import StandardScaler\r\n", + "from sklearn.pipeline import Pipeline\r\n", + "from sklearn.utils import Bunch\r\n", + "from sklearn.metrics import (\r\n", + " balanced_accuracy_score,\r\n", + " roc_auc_score,\r\n", + " accuracy_score,\r\n", + " recall_score,\r\n", + " confusion_matrix,\r\n", + " roc_auc_score,\r\n", + " roc_curve,\r\n", + " plot_roc_curve)\r\n", + "from sklearn import set_config\r\n", + "\r\n", + "set_config(display=\"diagram\")" + ], + "outputs": [], "metadata": { "id": "VhbFzU6GLgW0" - }, - "outputs": [], - "source": [ - "from sklearn.linear_model import LogisticRegression\n", - "from sklearn.model_selection import train_test_split\n", - "from sklearn.preprocessing import StandardScaler\n", - "from sklearn.pipeline import Pipeline\n", - "from sklearn.utils import Bunch\n", - "from sklearn.metrics import (\n", - " balanced_accuracy_score,\n", - " roc_auc_score,\n", - " accuracy_score,\n", - " recall_score,\n", - " confusion_matrix,\n", - " roc_auc_score,\n", - " roc_curve,\n", - " plot_roc_curve)\n", - "from sklearn import set_config\n", - "\n", - "set_config(display=\"diagram\")" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "GcilMnWhLgaT" - }, - "outputs": [], "source": [ - "from fairlearn.metrics import (\n", - " MetricFrame,\n", - " true_positive_rate,\n", - " false_positive_rate,\n", - " false_negative_rate,\n", - " selection_rate,\n", - " count,\n", - " false_negative_rate_difference\n", - ")\n", - "\n", - "from fairlearn.postprocessing import ThresholdOptimizer, plot_threshold_optimizer\n", - "from fairlearn.postprocessing._interpolated_thresholder import InterpolatedThresholder\n", - "from fairlearn.postprocessing._threshold_operation import ThresholdOperation\n", + "from fairlearn.metrics import (\r\n", + " MetricFrame,\r\n", + " true_positive_rate,\r\n", + " false_positive_rate,\r\n", + " false_negative_rate,\r\n", + " selection_rate,\r\n", + " count,\r\n", + " false_negative_rate_difference\r\n", + ")\r\n", + "\r\n", + "from fairlearn.postprocessing import ThresholdOptimizer, plot_threshold_optimizer\r\n", + "from fairlearn.postprocessing._interpolated_thresholder import InterpolatedThresholder\r\n", + "from fairlearn.postprocessing._threshold_operation import ThresholdOperation\r\n", "from fairlearn.reductions import ExponentiatedGradient, EqualizedOdds, TruePositiveRateParity" - ] + ], + "outputs": [], + "metadata": { + "id": "GcilMnWhLgaT" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "oMbCFBSZr2A4" - }, - "outputs": [], "source": [ - "# Model Card Toolkit works in Google Colab, but it does not work on all local environments\n", - "# that we tested. If the import fails, define a dummy function in place of the function\n", - "# for saving figures into images in a model card..\n", - "\n", - "try:\n", - " from model_card_toolkit import ModelCardToolkit\n", - " from model_card_toolkit.utils.graphics import figure_to_base64str\n", - " model_card_imported = True\n", - "except Exception:\n", - " model_card_imported = False\n", - " def figure_to_base64str(*args):\n", + "# Model Card Toolkit works in Google Colab, but it does not work on all local environments\r\n", + "# that we tested. If the import fails, define a dummy function in place of the function\r\n", + "# for saving figures into images in a model card..\r\n", + "\r\n", + "try:\r\n", + " from model_card_toolkit import ModelCardToolkit\r\n", + " from model_card_toolkit.utils.graphics import figure_to_base64str\r\n", + " model_card_imported = True\r\n", + "except Exception:\r\n", + " model_card_imported = False\r\n", + " def figure_to_base64str(*args):\r\n", " return None" - ] + ], + "outputs": [], + "metadata": { + "id": "oMbCFBSZr2A4" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "S4ANMj1M8k0h" - }, - "outputs": [], "source": [ - "from IPython import display\n", + "from IPython import display\r\n", "from datetime import date" - ] + ], + "outputs": [], + "metadata": { + "id": "S4ANMj1M8k0h" + } }, { "cell_type": "markdown", - "metadata": { - "id": "4OoPTetsDv-v" - }, "source": [ "# **Overview of fairness in AI systems**" - ] + ], + "metadata": { + "id": "4OoPTetsDv-v" + } }, { "cell_type": "markdown", - "metadata": { - "id": "vt9n43o6DPbg" - }, "source": [ "Please refer to the slides here: https://raw.githubusercontent.com/fairlearn/talks/main/2021_scipy_tutorial/overview.pdf\n", "\n" - ] + ], + "metadata": { + "id": "vt9n43o6DPbg" + } }, { "cell_type": "markdown", - "metadata": { - "id": "0gZTFumIxCfK" - }, "source": [ "# **Introduction of Fairlearn and other tutorial resources**\n" - ] + ], + "metadata": { + "id": "0gZTFumIxCfK" + } }, { "cell_type": "markdown", - "metadata": { - "id": "7KhsuCB-imFk" - }, "source": [ "This tutorial builds on the following open source projects:\n", "\n", "* **machine learning and data processing**: _scikit-learn_, _pandas_, _numpy_\n", "* **plotting**: _seaborn_, _matplotlib_\n", "* **AI fairness**: _Fairlearn_, _Model Card Toolkit_" - ] + ], + "metadata": { + "id": "7KhsuCB-imFk" + } }, { "cell_type": "markdown", - "metadata": { - "id": "ORfTAukTuEPN" - }, "source": [ "### [Fairlearn](https://fairlearn.org)\n", "\n", @@ -249,33 +246,33 @@ "* Educational resources covering organizational and technical processes for unfairness mitigation (user guide, case studies, Jupyter notebooks, etc.)\n", "\n", "The project was started in 2018 at Microsoft Research. In 2021 it adopted neutral governance structure and since then it is completely community-driven." - ] + ], + "metadata": { + "id": "ORfTAukTuEPN" + } }, { "cell_type": "markdown", - "metadata": { - "id": "AE3bFABst9ze" - }, "source": [ "### [Model Card Toolkit](https://github.com/tensorflow/model-card-toolkit)\n", "\n", "The Model Card Toolkit (MCT) streamlines and automates generation of _model cards_, machine learning documents that provide context and transparency into a model's development and performance. It was released by Google in 2020." - ] + ], + "metadata": { + "id": "AE3bFABst9ze" + } }, { "cell_type": "markdown", - "metadata": { - "id": "_j1vtg6TD7Fi" - }, "source": [ "# **Introduction to the health care scenario**" - ] + ], + "metadata": { + "id": "_j1vtg6TD7Fi" + } }, { "cell_type": "markdown", - "metadata": { - "id": "HUkG_zdEylGU" - }, "source": [ "Our scenario builds on previous research that highlighted racial disparities in how health care resources are allocated in the U.S. ([Obermeyer et al., 2019](https://science.sciencemag.org/content/366/6464/447.full)).\n", "Motivated by that work, in this tutorial we consider an automated system for recommending patients for _high-risk care management_ programs, which are described by Obermeyer et al. 2019 as follows:\n", @@ -285,22 +282,22 @@ "**Convenience restriction**\n", "\n", "* In practice, the modeling of health needs would use large data sets covering a wide range of diagnoses. In this tutorial, we will work with a [publicly available clinical dataset](https://archive.ics.uci.edu/ml/datasets/Diabetes+130-US+hospitals+for+years+1999-2008) that focuses on _diabetic patients only_ ([Strack et al., 2014](https://www.hindawi.com/journals/bmri/2014/781670/))." - ] + ], + "metadata": { + "id": "HUkG_zdEylGU" + } }, { "cell_type": "markdown", - "metadata": { - "id": "q-j4KN95wLLS" - }, "source": [ "## Dataset and task" - ] + ], + "metadata": { + "id": "q-j4KN95wLLS" + } }, { "cell_type": "markdown", - "metadata": { - "id": "zOwrRsB7wEeM" - }, "source": [ "We will be working with a clincial dataset of hospital re-admissions over a ten-year period (1998-2008) for diabetic patients across 130 different hospitals in the US. Each record represents the hospital admission records for a patient diagnosed with diabetes whose stay lasted one to fourteen days.\n", "\n", @@ -315,69 +312,69 @@ "* Because of the class imbalance, we will be measuring our performance via **balanced accuracy**. Another key performance consideration is how many patients are recommended for care, metric we refer to as **selection rate**.\n", "\n", "Ideally, health care professionals would be involved in both designing and using the model, including formalizing the task definition. \n" - ] + ], + "metadata": { + "id": "zOwrRsB7wEeM" + } }, { "cell_type": "markdown", - "metadata": { - "id": "2BE26iXWwUqr" - }, "source": [ "## Fairness considerations" - ] + ], + "metadata": { + "id": "2BE26iXWwUqr" + } }, { "cell_type": "markdown", - "metadata": { - "id": "eZUcQVZYyRvz" - }, "source": [ "* _Which groups are most likely to be disproportionately negatively affected?_ Previous work suggests that groups with different race and ethnicity can be differently affected.\n", "\n", "* _What are the harms?_ The key harms here are allocation harms. In particular, false negatives, i.e., don't recommend somebody who will be readmitted.\n", "\n", "* _How should we measure those harms?_\n" - ] + ], + "metadata": { + "id": "eZUcQVZYyRvz" + } }, { "cell_type": "markdown", - "metadata": { - "id": "2kD6G-yF1Tcf" - }, "source": [ "In the remainder of the tutorial we will:\n", "* First examine the dataset and our choice of label with an eye towards a variety of fairness issues.\n", "* Then train a logistic regression model and assess its performance as well as fairness.\n", "* Finally, look at two unfairness mitigation strategies." - ] + ], + "metadata": { + "id": "2kD6G-yF1Tcf" + } }, { "cell_type": "markdown", - "metadata": { - "id": "FkTmOAh8Bp1D" - }, "source": [ "\n", "## Discussion: Fairness-related harms\n", "\n", "* How can we determine which type of harm is relevant in a particular scenario?\n", "* What are ways to find out which (groups of) individuals are most likely to be disproportionately negatively affected?\n" - ] + ], + "metadata": { + "id": "FkTmOAh8Bp1D" + } }, { "cell_type": "markdown", - "metadata": { - "id": "R7tpRumX_4Nh" - }, "source": [ "# Task definition and dataset characteristics" - ] + ], + "metadata": { + "id": "R7tpRumX_4Nh" + } }, { "cell_type": "markdown", - "metadata": { - "id": "3iJGhfCgPiy-" - }, "source": [ "Two critical decisions when desiging an AI system are\n", "1. how we define the machine learning task\n", @@ -394,40 +391,47 @@ "\n", "The dataset characteristics can be systematically documented through the **datasheets** practice. We will touch on this later on. By documenting our understanding of the dataset, we communicate any concerns we have about the data and highlight downstream issues that may arise during the model training, evaluation and deployment.\n", "\n" - ] + ], + "metadata": { + "id": "3iJGhfCgPiy-" + } }, { "cell_type": "markdown", - "metadata": { - "id": "3-ABnntZT8Fn" - }, "source": [ "## Load the dataset\n" - ] + ], + "metadata": { + "id": "3-ABnntZT8Fn" + } }, { "cell_type": "markdown", - "metadata": { - "id": "bvyHqcLIT8Fo" - }, "source": [ "We next load the dataset and review the meaning of its columns.\n" - ] + ], + "metadata": { + "id": "bvyHqcLIT8Fo" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "gkfwniFQT8Fp" - }, - "outputs": [], "source": [ "df = pd.read_csv(\"https://raw.githubusercontent.com/fairlearn/talks/main/2021_scipy_tutorial/data/diabetic_preprocessed.csv\")" - ] + ], + "outputs": [], + "metadata": { + "id": "gkfwniFQT8Fp" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "df.head()" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -435,17 +439,10 @@ }, "id": "mIFN96kiT8Fq", "outputId": "9dbf587e-5f51-4dc4-877a-3ffb9a7374a2" - }, - "outputs": [], - "source": [ - "df.head()" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "KXQNItgRT8Fu" - }, "source": [ "The columns contain mostly boolean and categorical data (including age and various test results), with just the following exceptions: `time_in_hospital`, `num_lab_procedures`, `num_procedures`, `num_medications`, `number_diagnoses`.\n", "\n", @@ -462,11 +459,25 @@ "| readmitted, readmit_binary,
readmit_30_days | readmission information |\n", "\n", "\n" - ] + ], + "metadata": { + "id": "KXQNItgRT8Fu" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Show the values of all binary and categorical features\r\n", + "categorical_values = {}\r\n", + "for col in df:\r\n", + " if col not in {'time_in_hospital', 'num_lab_procedures',\r\n", + " 'num_procedures', 'num_medications', 'number_diagnoses'}:\r\n", + " categorical_values[col] = pd.Series(df[col].value_counts().index.values)\r\n", + "categorical_values_df = pd.DataFrame(categorical_values).fillna('')\r\n", + "categorical_values_df.T" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -474,79 +485,65 @@ }, "id": "gPtaDcfHT8Fv", "outputId": "42347f1f-d3c8-4221-f98d-fdeaa25ef4fd" - }, - "outputs": [], - "source": [ - "# Show the values of all binary and categorical features\n", - "categorical_values = {}\n", - "for col in df:\n", - " if col not in {'time_in_hospital', 'num_lab_procedures',\n", - " 'num_procedures', 'num_medications', 'number_diagnoses'}:\n", - " categorical_values[col] = pd.Series(df[col].value_counts().index.values)\n", - "categorical_values_df = pd.DataFrame(categorical_values).fillna('')\n", - "categorical_values_df.T" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "8loEiuFSWb8A" - }, "source": [ "We mark all categorical features: " - ] + ], + "metadata": { + "id": "8loEiuFSWb8A" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "categorical_features = [\r\n", + " \"race\",\r\n", + " \"gender\",\r\n", + " \"age\",\r\n", + " \"discharge_disposition_id\",\r\n", + " \"admission_source_id\",\r\n", + " \"medical_specialty\",\r\n", + " \"primary_diagnosis\",\r\n", + " \"max_glu_serum\",\r\n", + " \"A1Cresult\",\r\n", + " \"insulin\",\r\n", + " \"change\",\r\n", + " \"diabetesMed\",\r\n", + " \"readmitted\"\r\n", + "]" + ], + "outputs": [], "metadata": { "id": "P4y1FRMdWduE" - }, - "outputs": [], - "source": [ - "categorical_features = [\n", - " \"race\",\n", - " \"gender\",\n", - " \"age\",\n", - " \"discharge_disposition_id\",\n", - " \"admission_source_id\",\n", - " \"medical_specialty\",\n", - " \"primary_diagnosis\",\n", - " \"max_glu_serum\",\n", - " \"A1Cresult\",\n", - " \"insulin\",\n", - " \"change\",\n", - " \"diabetesMed\",\n", - " \"readmitted\"\n", - "]" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "Qyo3GQ4RWduG" - }, - "outputs": [], "source": [ - "for col_name in categorical_features:\n", + "for col_name in categorical_features:\r\n", " df[col_name] = df[col_name].astype(\"category\")" - ] + ], + "outputs": [], + "metadata": { + "id": "Qyo3GQ4RWduG" + } }, { "cell_type": "markdown", - "metadata": { - "id": "6LTax67Em4q8" - }, "source": [ "## Group sample sizes " - ] + ], + "metadata": { + "id": "6LTax67Em4q8" + } }, { "cell_type": "markdown", - "metadata": { - "id": "ft28kXKHm4q9" - }, "source": [ "From the perspective of fairness assessment, a key data characteristic is the sample size of groups with respect to which we conduct fairness assessment.\n", "\n", @@ -558,26 +555,33 @@ "\n", "Let's examine the sample sizes of the groups according to `race`:\n", "\n" - ] + ], + "metadata": { + "id": "ft28kXKHm4q9" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "df[\"race\"].value_counts()" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "wEGJjCrOm4q9", "outputId": "a7430662-00d7-405b-b9af-1dcaf8fc5c28" - }, - "outputs": [], - "source": [ - "df[\"race\"].value_counts()" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "df[\"race\"].value_counts().plot(kind='bar', rot=45);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -585,107 +589,100 @@ }, "id": "HznSDLWEm4q-", "outputId": "009b8aea-9ae4-4810-eb13-67ac8efae163" - }, - "outputs": [], - "source": [ - "df[\"race\"].value_counts().plot(kind='bar', rot=45);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "ZkI6KA3rm4q-" - }, "source": [ "Normalized as frequencies:" - ] + ], + "metadata": { + "id": "ZkI6KA3rm4q-" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "df[\"race\"].value_counts(normalize=True)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "qE2UoDU3m4q-", "outputId": "c23f07b1-493d-4df2-db5a-06bc921eefde" - }, - "outputs": [], - "source": [ - "df[\"race\"].value_counts(normalize=True)" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "DEUrg6J3m4q_" - }, "source": [ "In our dataset, our patients are predominantly *Caucasian* (75%). The next largest racial group is *AfricanAmerican*, making up 19% of the patients. The remaining race categories (including *Unknown*) compose only 6% of the data." - ] - }, - { - "cell_type": "markdown", + ], "metadata": { - "id": "xIdDhgtAm4rA" - }, + "id": "DEUrg6J3m4q_" + } + }, + { + "cell_type": "markdown", "source": [ "We also examine the dataset composition by `gender`:" - ] + ], + "metadata": { + "id": "xIdDhgtAm4rA" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "df[\"gender\"].value_counts() # counts" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0h26CH50m4rA", "outputId": "20d290a6-c8a7-4d21-c676-2a6f552bc3a6" - }, - "outputs": [], - "source": [ - "df[\"gender\"].value_counts() # counts" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "df[\"gender\"].value_counts(normalize=True) # frequencies" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "UTbI5v48m4rA", "outputId": "e69992ee-8b33-4aef-a903-b51eb63094e8" - }, - "outputs": [], - "source": [ - "df[\"gender\"].value_counts(normalize=True) # frequencies" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "i3xBANdpm4rA" - }, "source": [ "Gender is in our case effectively binary (and we have no further information how it was operationalized), with both *Female* represented at 54% and *Male* represented at 46%. There are only 3 samples annotated as *Unknown/Invalid*." - ] + ], + "metadata": { + "id": "i3xBANdpm4rA" + } }, { "cell_type": "markdown", - "metadata": { - "id": "NyLBXpKWm4rA" - }, "source": [ "### Decision point: How do we address smaller group sizes?" - ] + ], + "metadata": { + "id": "NyLBXpKWm4rA" + } }, { "cell_type": "markdown", - "metadata": { - "id": "wT-BPQ2Gm4rB" - }, "source": [ "When the data set lacks coverage of certain groups, it means that we will not be able to reliably assess any fairness-related issues. There are three interventions (which could be carried out in a combination):\n", "\n", @@ -702,69 +699,76 @@ "* merge the three smallest race groups *Asian*, *Hispanic*, *Other* (similar to [Strack et al., 2014](https://www.hindawi.com/journals/bmri/2014/781670/)), but also retain the original groups for auxiliary assessments\n", "\n", "* drop the gender group *Unknown/Invalid*, because the sample size is so small that no meaningful fairness assessment is possible" - ] + ], + "metadata": { + "id": "wT-BPQ2Gm4rB" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "iBN_fIhpm4rB" - }, - "outputs": [], "source": [ - "# drop gender group Unknown/Invalid\n", - "df = df.query(\"gender != 'Unknown/Invalid'\")\n", - "\n", - "# retain the original race as race_all, and merge Asian+Hispanic+Other \n", - "df[\"race_all\"] = df[\"race\"]\n", + "# drop gender group Unknown/Invalid\r\n", + "df = df.query(\"gender != 'Unknown/Invalid'\")\r\n", + "\r\n", + "# retain the original race as race_all, and merge Asian+Hispanic+Other \r\n", + "df[\"race_all\"] = df[\"race\"]\r\n", "df[\"race\"] = df[\"race\"].replace({\"Asian\": \"Other\", \"Hispanic\": \"Other\"})" - ] + ], + "outputs": [], + "metadata": { + "id": "iBN_fIhpm4rB" + } }, { "cell_type": "markdown", - "metadata": { - "id": "N_Sb8ISAnRQF" - }, "source": [ "### Exercise" - ] + ], + "metadata": { + "id": "N_Sb8ISAnRQF" + } }, { "cell_type": "markdown", - "metadata": { - "id": "ns1Tr9wLm4rB" - }, "source": [ "Please examine the distribution of the `age` feature in the dataset." - ] + ], + "metadata": { + "id": "ns1Tr9wLm4rB" + } }, { "cell_type": "markdown", - "metadata": { - "id": "MDiLp-cDoWGv" - }, "source": [ "### Answer" - ] + ], + "metadata": { + "id": "MDiLp-cDoWGv" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "df[\"age\"].value_counts()" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "UOn4j1o8m4rB", "outputId": "577b1636-57f5-4ab1-903a-c6a42f8193ba" - }, - "outputs": [], - "source": [ - "df[\"age\"].value_counts()" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "df[\"age\"].value_counts().plot(kind='bar', rot=0);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -772,35 +776,28 @@ }, "id": "ns7RxP5um4rB", "outputId": "81364da1-1f2e-4053-ced1-c93394ba125e" - }, - "outputs": [], - "source": [ - "df[\"age\"].value_counts().plot(kind='bar', rot=0);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "HAXa7MoQnjq4" - }, "source": [ "As we might expect, most patients admitted into the hospital in our data set belong to the *Over 60 years* category. Although we will not be assessing for age-based fairness-related harms in this tutorial, we will want to document the age imbalance in our dataset." - ] + ], + "metadata": { + "id": "HAXa7MoQnjq4" + } }, { "cell_type": "markdown", - "metadata": { - "id": "pK0LbF_sylTU" - }, "source": [ "## Examining the choice of label" - ] + ], + "metadata": { + "id": "pK0LbF_sylTU" + } }, { "cell_type": "markdown", - "metadata": { - "id": "AtgLr5Cs_YhF" - }, "source": [ "Next we dive into the question of whether our choice of label (readmission within 30 days) aligns with our goal (identify patients that would benefit from the care management program).\n", "\n", @@ -811,22 +808,22 @@ "In our case, the **measurement** coincides with the **classification label**.\n", "\n", "The act of _operationalizing_ the construct via a specific measurement corresponds to making certain assumptions. In our case, we are making the following assumption: **the greatest benefit from the care management program would go to patients that are** (in the absence of such a program) **most likely to be readmitted into the hospital within 30 days.**" - ] + ], + "metadata": { + "id": "AtgLr5Cs_YhF" + } }, { "cell_type": "markdown", - "metadata": { - "id": "fA19isovvCew" - }, "source": [ "### How can we check whether our assumptions apply?" - ] + ], + "metadata": { + "id": "fA19isovvCew" + } }, { "cell_type": "markdown", - "metadata": { - "id": "oY95_kOFO6Wb" - }, "source": [ "In the terminology of measurement modeling, how do we establish _construct validity_? Following, [Jacobs and Wallach, 2021](https://arxiv.org/abs/1912.05511),\n", "\n", @@ -850,42 +847,50 @@ "related to the construct purported to be measured, but not incorporated into the operationalization.\n", "\n", "The predictions do not need to be chronological, meaning that we do not necessarily need to be predicting future from the past. Also, the predictions do not need to be causal (going from causes to effects). We just need to ensure that the predicted property is not part of the measurement whose validity we're checking. \n" - ] + ], + "metadata": { + "id": "oY95_kOFO6Wb" + } }, { "cell_type": "markdown", - "metadata": { - "id": "8IBUG3Sa96LX" - }, "source": [ "### Predictive validity" - ] + ], + "metadata": { + "id": "8IBUG3Sa96LX" + } }, { "cell_type": "markdown", - "metadata": { - "id": "rVrLpuwy98uG" - }, "source": [ "We would like to show that our measurement `readmit_30_days` is correlated with patient characteristics that are related to our construct \"benefiting from care management\". One such characteristic is the general patient health, where we expect that patients that are less healthy are more likely to benefit from care management.\n", "\n", "While our data does not contain full health records that would enable us to holistically measure general patient health, the data does contain two relevant features: `had_emergency` and `had_inpatient_days`, which indicate whether the patient spent any days in the emergency room or in the hospital (but non-emergency) in the preceding year.\n", "\n", "To establish predictive validity, we would like to show that our measurement `readmit_30_days` is predictive of these two observable characteristics." - ] + ], + "metadata": { + "id": "rVrLpuwy98uG" + } }, { "cell_type": "markdown", - "metadata": { - "id": "BQWxJEN-M6VD" - }, "source": [ "First, let's check the rate at which the patients with different `readmit_30_days` labels were readmitted in the previous year:" - ] + ], + "metadata": { + "id": "BQWxJEN-M6VD" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.pointplot(y=\"had_emergency\", x=\"readmit_30_days\",\r\n", + " data=df, ci=95, join=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -893,34 +898,34 @@ }, "id": "3t11OgTgZfV8", "outputId": "d2467bd3-0148-4373-cd41-0b99d9f54088" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"had_emergency\", x=\"readmit_30_days\",\n", - " data=df, ci=95, join=False);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "Ptl-tHkDf_GJ" - }, "source": [ "The plot shows that indeed patients with `readmit_30_days=0` have a lower rate of emergency visits in the prior year, whereas patients with `readmit_30_days=1` have a larger rate. (The vertical lines indicate 95% confidence intervals obtained via boostrapping.)" - ] + ], + "metadata": { + "id": "Ptl-tHkDf_GJ" + } }, { "cell_type": "markdown", - "metadata": { - "id": "07GU8IGIY9KC" - }, "source": [ "We see a similar pattern when `readmit_30_days` is used to predict the rate of (non-emergency) hospital visits in the previous year:" - ] + ], + "metadata": { + "id": "07GU8IGIY9KC" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.pointplot(y=\"had_inpatient_days\", x=\"readmit_30_days\",\r\n", + " data=df, join=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -928,25 +933,26 @@ }, "id": "GuPPVpXrO5uE", "outputId": "dcc740aa-0805-4ef9-9c77-d64fe1b0111a" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"had_inpatient_days\", x=\"readmit_30_days\",\n", - " data=df, join=False);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "wN8NU8QkRMqM" - }, "source": [ "Now let's take a look whether the predictiveness is similar across different race groups. First, let's check how well `readmit_30_days` predicts `had_emergency`:" - ] + ], + "metadata": { + "id": "wN8NU8QkRMqM" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Visualize predictiveness using a categorical pointplot\r\n", + "sns.catplot(y=\"had_emergency\", x=\"readmit_30_days\", hue=\"race\", data=df,\r\n", + " kind=\"point\", ci=95, dodge=True, join=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -954,37 +960,36 @@ }, "id": "tgLhUlZzOvHC", "outputId": "59590db9-f5c7-44a0-85d2-bfcf79eab25e" - }, - "outputs": [], - "source": [ - "# Visualize predictiveness using a categorical pointplot\n", - "sns.catplot(y=\"had_emergency\", x=\"readmit_30_days\", hue=\"race\", data=df,\n", - " kind=\"point\", ci=95, dodge=True, join=False);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "IQ7zH3Wn416g" - }, "source": [ "The patients in the group *Unknown* have a substantially lower rate of emergency visits in the prior year, regardless of whether they are readmitted in 30 days. The readmission is still positively correlated with `had_emergency`, but note the large error bars (due to small sample sizes).\n", "\n", "We also see that the group with feature value *AfricanAmerican* has a higher rate of emergency visits compared with other groups. However, generally the groups *Caucasian*, *AfricanAmerican* and *Other* follow similar dependence patterns." - ] + ], + "metadata": { + "id": "IQ7zH3Wn416g" + } }, { "cell_type": "markdown", - "metadata": { - "id": "G13abbdS7oM-" - }, "source": [ "We see a similar pattern when `readmit_30_days` is used to predict the rate of (non-emergency) hospital visits in the previous year:" - ] + ], + "metadata": { + "id": "G13abbdS7oM-" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.catplot(y=\"had_inpatient_days\", x=\"readmit_30_days\", hue=\"race\", data=df,\r\n", + " kind=\"point\", ci=95, dodge=True, join=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -992,18 +997,10 @@ }, "id": "Kx52yWYNQuSH", "outputId": "d0acd516-57d5-4910-feaa-f3b1c1176900" - }, - "outputs": [], - "source": [ - "sns.catplot(y=\"had_inpatient_days\", x=\"readmit_30_days\", hue=\"race\", data=df,\n", - " kind=\"point\", ci=95, dodge=True, join=False);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "HrCelj5_hR6a" - }, "source": [ "Again, for *Unknown* the rate of (non-emergency) hospital visits in the previous year is lower than for other groups.In all groups there is a strong positive correlation between `readmit_30_days` and `had_inpatient_days`.\n", "\n", @@ -1012,30 +1009,39 @@ "The analysis is also surfacing the fact that patients with the value of race *Unknown* have fewer hospital visits in the preceding year (both emergency and non-emergency) than other groups. In practice, this would be a good reason to reach out to health professionals to investigate this patient cohort, to make sure that we understand why there is the systematic difference.\n", "\n", "Note that we have only investigated _predictive validity_, but there are other important aspects of construct validity which we may want to establish (see [Jacobs and Wallach, 2021](https://arxiv.org/abs/1912.05511))." - ] + ], + "metadata": { + "id": "HrCelj5_hR6a" + } }, { "cell_type": "markdown", - "metadata": { - "id": "V_4CZt-UBaOQ" - }, "source": [ "\n", "### Exercise" - ] + ], + "metadata": { + "id": "V_4CZt-UBaOQ" + } }, { "cell_type": "markdown", - "metadata": { - "id": "rW9ktRlqBhZA" - }, "source": [ "Check the predictive validity with respect to `gender` and `age`. Do you see any differences? Can you form a hypothesis why?" - ] + ], + "metadata": { + "id": "rW9ktRlqBhZA" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Check for predictive validity by gender\r\n", + "sns.catplot(y=\"had_inpatient_days\",x=\"readmit_30_days\",hue=\"gender\", data=df,\r\n", + " kind=\"point\", ci=95, dodge=True, join=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1043,17 +1049,17 @@ }, "id": "VLom8GFbUfqR", "outputId": "91418775-b94d-4b3e-dba4-1305ef908a29" - }, - "outputs": [], - "source": [ - "# Check for predictive validity by gender\n", - "sns.catplot(y=\"had_inpatient_days\",x=\"readmit_30_days\",hue=\"gender\", data=df,\n", - " kind=\"point\", ci=95, dodge=True, join=False);" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Check for predictive validity by age\r\n", + "sns.catplot(y=\"had_inpatient_days\", x=\"readmit_30_days\", hue=\"age\", data=df,\r\n", + " kind=\"point\", ci=95, dodge=True, join=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1061,29 +1067,20 @@ }, "id": "7PdFAu3lT-pD", "outputId": "7c3b6008-fdff-46dd-f749-ca0198a5193b" - }, - "outputs": [], - "source": [ - "# Check for predictive validity by age\n", - "sns.catplot(y=\"had_inpatient_days\", x=\"readmit_30_days\", hue=\"age\", data=df,\n", - " kind=\"point\", ci=95, dodge=True, join=False);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "3i9KdmTWKUJZ" - }, "source": [ "## Label imbalance\n", "\n" - ] + ], + "metadata": { + "id": "3i9KdmTWKUJZ" + } }, { "cell_type": "markdown", - "metadata": { - "id": "ViGqA5VTGrEo" - }, "source": [ "Now that we have established the validity of our label, we will check frequency of its values in our data. The frequency of different labels is an important descriptive characteristic in classification settings for several reasons:\n", "\n", @@ -1091,72 +1088,79 @@ "* in binary classification settings, our ability to evaluate error is often driven by the size of the smaller of the two classes (again, the smaller the sample the larger the uncertainty in estimates)\n", "* label imbalance may exacerbate the problems due to smaller group sizes in fairness assessment\n", "\n" - ] + ], + "metadata": { + "id": "ViGqA5VTGrEo" + } }, { "cell_type": "markdown", - "metadata": { - "id": "cos3--59EiZt" - }, "source": [ "Let's check how many samples in our data are labeled as positive and how many as negative." - ] + ], + "metadata": { + "id": "cos3--59EiZt" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "df[\"readmit_30_days\"].value_counts() # counts" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "eR5ULLYGE4UK", "outputId": "02f2e533-2a7f-44af-e57f-c24c7b8b4d4d" - }, - "outputs": [], - "source": [ - "df[\"readmit_30_days\"].value_counts() # counts" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "df[\"readmit_30_days\"].value_counts(normalize=True) # frequencies" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0UWBbg4Cz90t", "outputId": "a0e1e14b-47f9-4f49-da41-ba53a7a63082" - }, - "outputs": [], - "source": [ - "df[\"readmit_30_days\"].value_counts(normalize=True) # frequencies" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "b-_K_KHXz9MR" - }, "source": [ "As we can see, the target label is heavily skewed towards the patients not being readmitted within 30 days. In our dataset, only 11% of patients were readmitted within 30 days.\n", "\n", "Since there are fewer positive examples, we expect that we will have a much larger uncertainty (error bars) in our estimates of *false negative rates* (FNR), compared with *false positive rates* (FPR). This means that there will be larger differences between training FNR and test FNR, even if there is no overfitting, simply because of the smaller sample sizes. \n", "\n", "Our target metric is *balanced error rate*, which is the average of FPR and FNR. The value of this metric is robust to different frequencies of positives and negatives. However, since half of the metric is contributed by FNR, we expect the uncertainty in balanced error values to behave similarly to the uncertainty of FNR." - ] + ], + "metadata": { + "id": "b-_K_KHXz9MR" + } }, { "cell_type": "markdown", - "metadata": { - "id": "OkopoyE3GQ8g" - }, "source": [ "Now, let's examine how much the label frequencies vary within each group defined by `race`:" - ] + ], + "metadata": { + "id": "OkopoyE3GQ8g" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.barplot(x=\"readmit_30_days\", y=\"race\", data=df, ci=95);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1164,36 +1168,29 @@ }, "id": "tcH3Mxwmm4rC", "outputId": "75641d75-670d-4a81-a354-855f80c38613" - }, - "outputs": [], - "source": [ - "sns.barplot(x=\"readmit_30_days\", y=\"race\", data=df, ci=95);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "ox06aTMmm4rB" - }, "source": [ "We see the rate of *30-day readmission* is similar for the *AfricanAmerican* and *Caucasian* groups, but appears smaller for *Other* and smallest for *Unknown* (this is consistent with an overall lower rate of hospital visits in the prior year). The smaller sample size of the *Other* and *Unknown* groups mean that there is more uncertainty around the estimate for these two groups." - ] + ], + "metadata": { + "id": "ox06aTMmm4rB" + } }, { "cell_type": "markdown", - "metadata": { - "id": "0AA5uoqAKUSx" - }, "source": [ "## Proxies for sensitive features\n", "\n" - ] + ], + "metadata": { + "id": "0AA5uoqAKUSx" + } }, { "cell_type": "markdown", - "metadata": { - "id": "7PSup7dMJjJg" - }, "source": [ "We next investigate which of the features are highly predictive of the sensitive feature *race*; such features are called *proxies*.\n", "\n", @@ -1202,29 +1199,36 @@ "Another reason to understand the proxies is because they might explain why we see differences in impact on different groups even when our model does not have access to the sensitive features directly.\n", "\n", "In this section we briefly examine the identification of such proxies (but we don't go into legal or causality considerations).\n" - ] + ], + "metadata": { + "id": "7PSup7dMJjJg" + } }, { "cell_type": "markdown", - "metadata": { - "id": "1PHULa3eQEcn" - }, "source": [ "In the United States, *Medicare* and *Medicaid* are joint federal and state programs to help qualified individuals pay for healthcare expenses. *Medicare* is available to people over the age of 65 and younger individuals with severe illnesses. *Medicaid* is available to all individuals under the age of 65 whose adjusted gross income falls below the Federal Poverty Line. " - ] + ], + "metadata": { + "id": "1PHULa3eQEcn" + } }, { "cell_type": "markdown", - "metadata": { - "id": "VNYx0OaVElUC" - }, "source": [ "First, let's explore the relationship between patients who paid with *Medicaid* and our demographic features. Because *Medicaid* is available to low-income individuals, and race is correlated with socioeconomic status in the United States, we expect there to be a relationship between `race` and paying with *Medicaid*. " - ] + ], + "metadata": { + "id": "VNYx0OaVElUC" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.pointplot(y=\"medicaid\", x=\"race\", data=df, join=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1232,44 +1236,44 @@ }, "id": "dR3eNMeY0HTi", "outputId": "aaa8498f-1f82-4003-c73c-ab6eaa3c2476" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"medicaid\", x=\"race\", data=df, join=False);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "MTcAD0oxbarW" - }, "source": [ "From our analysis, we see that paying with *Medicaid* does appear to have some relationship with the patient's race. *Caucasian* patients are the least likely to pay with *Medicaid* compared with other groups. If paying with *Medicaid* is a proxy for socioeconomic status, then the patterns we find align with our understanding of wealth and race in the United States." - ] + ], + "metadata": { + "id": "MTcAD0oxbarW" + } }, { "cell_type": "markdown", - "metadata": { - "id": "9BLSciTKaBLd" - }, "source": [ "## Additional validity checks" - ] + ], + "metadata": { + "id": "9BLSciTKaBLd" + } }, { "cell_type": "markdown", - "metadata": { - "id": "-PzKOA59ezFs" - }, "source": [ "Similarly as we used predictive validity to check that our label aligns with the construct of \"likely to benefit from the care management program\", we can use predictive validity to verify that our various features are coherent with each other.\n", "\n", "For example, based on the eligibility criteria for *Medicaid* vs *Medicare*, we expect `medicaid` to be negatively correlated with age and `medicare` to be positively correlated with age:" - ] + ], + "metadata": { + "id": "-PzKOA59ezFs" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.pointplot(y=\"medicaid\", x=\"age\", data=df, ci=95, join=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1277,15 +1281,15 @@ }, "id": "869rzUjSUe3C", "outputId": "bd4de952-204d-4159-addc-19b9d1b75167" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"medicaid\", x=\"age\", data=df, ci=95, join=False);" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.pointplot(y=\"medicare\", x=\"age\", data=df, ci=95, join=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1293,43 +1297,43 @@ }, "id": "nnI8lFHFZyBL", "outputId": "e0596d0e-efb7-4fc0-d775-99fdae53c8bf" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"medicare\", x=\"age\", data=df, ci=95, join=False);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "dBgmuoz8aXw4" - }, "source": [ "As we see, that's indeed the case." - ] + ], + "metadata": { + "id": "dBgmuoz8aXw4" + } }, { "cell_type": "markdown", - "metadata": { - "id": "DdrS4By1dVk-" - }, "source": [ "\n", "## Exercise" - ] + ], + "metadata": { + "id": "DdrS4By1dVk-" + } }, { "cell_type": "markdown", - "metadata": { - "id": "V-sOl0rqblsb" - }, "source": [ "Now, let's explore the relationship between paying with `medicare` and other demographic features. In the below sections, feel free to perform any analysis you would like to better understand the relationship between `medicare` and `race` and `gender` in this dataset." - ] + ], + "metadata": { + "id": "V-sOl0rqblsb" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.pointplot(y=\"medicare\", x=\"race\", data=df, ci=95, join=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1337,15 +1341,15 @@ }, "id": "lp02oUbWZx54", "outputId": "40c1cc55-625f-47d5-a21a-8950db45ab23" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"medicare\", x=\"race\", data=df, ci=95, join=False);" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.pointplot(y=\"medicare\", x=\"age\", data=df, ci=95, join=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1353,15 +1357,15 @@ }, "id": "ONJcedCWULXN", "outputId": "1f0e82df-b12a-459b-9b77-13285ae44008" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"medicare\", x=\"age\", data=df, ci=95, join=False);" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.pointplot(y=\"medicare\", x=\"gender\", data=df, ci=95, join=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1369,27 +1373,20 @@ }, "id": "jXtY7_xdULLe", "outputId": "67948de6-24eb-4d61-9116-1d465534ee53" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"medicare\", x=\"gender\", data=df, ci=95, join=False);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "ZVSerRqDG3we" - }, "source": [ "\n", "## Datasheets for datasets" - ] + ], + "metadata": { + "id": "ZVSerRqDG3we" + } }, { "cell_type": "markdown", - "metadata": { - "id": "ACLMWSAwGaLc" - }, "source": [ "The _datasheets_ practice was proposed by [Gebru et al. (2018)](https://arxiv.org/abs/1803.09010). A datasheet of a given dataset documents the motivation behind the dataset creation, the dataset composition, collection process, recommended uses and many other characteristics. In the words of Gebru et al., the goal is to\n", "> facilitate better communication between dataset creators\n", @@ -1397,117 +1394,117 @@ "> community to prioritize transparency and accountability.\n", "\n", "In this section, we show how to fill in some of the sections of the datasheet for the dataset that we are using. The information is obtained directly from [Strack et al. (2014)](https://www.hindawi.com/journals/bmri/2014/781670/)." - ] + ], + "metadata": { + "id": "ACLMWSAwGaLc" + } }, { "cell_type": "markdown", - "metadata": { - "id": "fcc4cUMhZnCb" - }, "source": [ "### Example sections of a datasheet [OPTIONAL SECTION]" - ] + ], + "metadata": { + "id": "fcc4cUMhZnCb" + } }, { "cell_type": "markdown", - "metadata": { - "id": "0TIPJIhX1IKJ" - }, "source": [ "**For what purpose was the dataset created?** *Was there a specific task in mind? Was there a specific gap that needed to be filled?*" - ] + ], + "metadata": { + "id": "0TIPJIhX1IKJ" + } }, { "cell_type": "markdown", - "metadata": { - "id": "sBNKbKJQJbmf" - }, "source": [ "In the words of the dataset authors:\n", "> [...] the management of hyperglycemia in the hospitalized patient has a significant bearing on outcome, in terms of both morbidity and mortality. This recognition has led to the development of formalized protocols in the intensive care unit (ICU) setting [...] However, the same cannot be said for most non-ICU inpatient admissions. [...] there are few national assessments of diabetes care in the hospitalized patient which could serve as a baseline for change [in the non-ICU protocols]. The present analysis of a large clinical database was undertaken to examine historical patterns of diabetes care in patients with diabetes admitted to a US hospital and to inform future directions which might lead to improvements in patient safety." - ] + ], + "metadata": { + "id": "sBNKbKJQJbmf" + } }, { "cell_type": "markdown", - "metadata": { - "id": "iA-XRu0o1ErL" - }, "source": [ "**Who created the dataset (e.g., which team) and on behalf of which entity?**" - ] + ], + "metadata": { + "id": "iA-XRu0o1ErL" + } }, { "cell_type": "markdown", - "metadata": { - "id": "zQSHxl26LQkt" - }, "source": [ "The dataset was created by [Strack et al. (2014)](https://www.hindawi.com/journals/bmri/2014/781670/): a team of researchers from a variety of disciplines, ranging from computer science to public health, from three institutions (Virginia Commonwealth University, University of Cordoba, and Polish Academy of Sciences)." - ] + ], + "metadata": { + "id": "zQSHxl26LQkt" + } }, { "cell_type": "markdown", - "metadata": { - "id": "0Qx32mSJG3zP" - }, "source": [ "#### **Composition**" - ] + ], + "metadata": { + "id": "0Qx32mSJG3zP" + } }, { "cell_type": "markdown", - "metadata": { - "id": "8RS2V8001F3E" - }, "source": [ "**What do the instances that comprise the dataset represent?**\n", "\n" - ] + ], + "metadata": { + "id": "8RS2V8001F3E" + } }, { "cell_type": "markdown", - "metadata": { - "id": "JPy_TXp_1Gub" - }, "source": [ "Each instance in this dataset represents a hospital admission for diabetic patient (diabetes was entered as a possible diagnosis for the patient) whose hospital stay lasted between one to fourteen days." - ] + ], + "metadata": { + "id": "JPy_TXp_1Gub" + } }, { "cell_type": "markdown", - "metadata": { - "id": "eOb0FPeOJqxm" - }, "source": [ "**Is any information missing from individual instances?**" - ] + ], + "metadata": { + "id": "eOb0FPeOJqxm" + } }, { "cell_type": "markdown", - "metadata": { - "id": "4vlZWeQjJq8w" - }, "source": [ "The features `Payer Code` and `Medical Specialty` have 40,255 and 49,947 missing values, respectively. For `Payer Code`, these missing values are reflected in the category *Unknown*. For `Medical Specialty`, these missing values are reflecting in the category *Missing*. \n", "\n", "For our demographic features, we are missing the `Gender` information for three patients in the dataset. These three records were dropped from our final dataset. Regarding `Race`, the 2,271 missing values were recoded into the `Unknown` race category. \n", "\n" - ] + ], + "metadata": { + "id": "4vlZWeQjJq8w" + } }, { "cell_type": "markdown", - "metadata": { - "id": "Uh0lLV6mJrSp" - }, "source": [ "**Does the dataset identify any subpopulations (e.g., by age, gender)?**" - ] + ], + "metadata": { + "id": "Uh0lLV6mJrSp" + } }, { "cell_type": "markdown", - "metadata": { - "id": "U-ZnQfibJrcQ" - }, "source": [ "Patients are identified by gender, age group, and race. \n", "\n", @@ -1537,83 +1534,83 @@ "AfricanAmerican | 19210 | 18.9% \n", "Other | 4183 | 4.1%\n", "Unknown | 2271 | 2.2%" - ] + ], + "metadata": { + "id": "U-ZnQfibJrcQ" + } }, { "cell_type": "markdown", - "metadata": { - "id": "yzXw0egqG4J4" - }, "source": [ "#### **Preprocessing**" - ] + ], + "metadata": { + "id": "yzXw0egqG4J4" + } }, { "cell_type": "markdown", - "metadata": { - "id": "rGfxGcI21Fyj" - }, "source": [ "**Was any preprocessing/cleaning/labeling of the data done?**" - ] + ], + "metadata": { + "id": "rGfxGcI21Fyj" + } }, { "cell_type": "markdown", - "metadata": { - "id": "5jO4Pf911GrL" - }, "source": [ "For the `race` feature, the categories of *Asian* and *Hispanic* and *Other* were merged into the *Other* category. The `age` feature was bucketed into 30-year intervals (*30 years and below*, *30 to 60 years*, and *Over 60 years*). The `discharge_disposition_id` was binarized into a boolean outcome on whether an patient was discharged to home.\n", "\n", "The full preprocessing code is provided in the file `preprocess.py` of the tutorial [GitHub repository](https://github.com/fairlearn/talks/blob/main/2021_scipy_tutorial/).\n", "\n", "\n" - ] + ], + "metadata": { + "id": "5jO4Pf911GrL" + } }, { "cell_type": "markdown", - "metadata": { - "id": "C8b5nXfPIA7a" - }, "source": [ "#### **Uses**\n", "\n" - ] + ], + "metadata": { + "id": "C8b5nXfPIA7a" + } }, { "cell_type": "markdown", - "metadata": { - "id": "_YH8evvN1HX2" - }, "source": [ "**Has the dataset been used for any tasks already?** " - ] + ], + "metadata": { + "id": "_YH8evvN1HX2" + } }, { "cell_type": "markdown", - "metadata": { - "id": "H8RW1LKW1Hbg" - }, "source": [ "This dataset has been used by [Strack et al. (2014)](https://www.hindawi.com/journals/bmri/2014/781670/) to model the relationship between patient readmission and HbA1c measurement during admission, based on primary medical diagnosis.\n", "\n", "The dataset is publicly available through the UCI Machine Learning Repository and, as of May 2021, has received over 350,000 views." - ] + ], + "metadata": { + "id": "H8RW1LKW1Hbg" + } }, { "cell_type": "markdown", - "metadata": { - "id": "HB7zfhA1UKiW" - }, "source": [ "# Training the initial model" - ] + ], + "metadata": { + "id": "HB7zfhA1UKiW" + } }, { "cell_type": "markdown", - "metadata": { - "id": "0f7jZOzGX24Z" - }, "source": [ "We next train a classification model to predict our target variable (readmission within 30 days) while optimizing balanced accuracy.\n", "\n", @@ -1629,103 +1626,110 @@ "* **Model expressiveness.** How well can the model separate positive examples from negative examples? How well can it do so given the available dataset size? Can it do so across all groups or does it need to trade off performance on one group against performance on another group?\n", "\n", "Some additional considerations are training time (this impacts the ability to iterate), familiarity (this impacts the ability to fine tune and debug), and carbon footprint (this impacts the Earth climate both directly and indirectly by normalizing unnecessarily heavy workloads)." - ] + ], + "metadata": { + "id": "0f7jZOzGX24Z" + } }, { "cell_type": "markdown", - "metadata": { - "id": "7hbsqXap9Mzp" - }, "source": [ "### Decision point: Model type" - ] + ], + "metadata": { + "id": "7hbsqXap9Mzp" + } }, { "cell_type": "markdown", - "metadata": { - "id": "iftAwdfoVDM0" - }, "source": [ "We will use a logistic regression model. Our reasoning:\n", "\n", "* **Interpretability**. Logistic models over a small number of variables (as used here) are highly interpretable in the sense that stakeholders can simulate them easily.\n", "\n", "* **Model expressiveness**. Logistic regression predictions are described by a linear weighting of the feature values. The concern might be that this is too simple. The previous work by [Strack et al. (2014)](https://www.hindawi.com/journals/bmri/2014/781670/), which also used a logistic model to predict readmission rates concluded that a slightly more expressive model might be useful (their analysis uncovered 8 pairwise interactions that were significant, see their Table 5)." - ] + ], + "metadata": { + "id": "iftAwdfoVDM0" + } }, { "cell_type": "markdown", - "metadata": { - "id": "52wWLOFoXkho" - }, "source": [ "## Prepare training and test datasets" - ] + ], + "metadata": { + "id": "52wWLOFoXkho" + } }, { "cell_type": "markdown", - "metadata": { - "id": "vCwtDyx8yqSG" - }, "source": [ "As we mentioned in the task definition, our target variable is **readmission within 30 days**, and our sensitive feature for the purposes of fairness assessment is **race**.\n" - ] + ], + "metadata": { + "id": "vCwtDyx8yqSG" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "WpiSRj2JyqSH" - }, - "outputs": [], "source": [ - "target_variable = \"readmit_30_days\"\n", - "demographic = [\"race\", \"gender\"]\n", - "sensitive = [\"race\"]\n", + "target_variable = \"readmit_30_days\"\r\n", + "demographic = [\"race\", \"gender\"]\r\n", + "sensitive = [\"race\"]\r\n", "# If multiple sensitive features are chosen, the rest of the script considers intersectional groups." - ] + ], + "outputs": [], + "metadata": { + "id": "WpiSRj2JyqSH" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "uaNqDyqvi1QE" - }, - "outputs": [], "source": [ "Y, A = df.loc[:, target_variable], df.loc[:, sensitive]" - ] + ], + "outputs": [], + "metadata": { + "id": "uaNqDyqvi1QE" + } }, { "cell_type": "markdown", - "metadata": { - "id": "niu49A9YXQmf" - }, "source": [ "We next drop the features that we don't want to use in our model and expand the categorical features into 0/1 indicators (\"dummies\")." - ] + ], + "metadata": { + "id": "niu49A9YXQmf" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "JXyRuCsri1cY" - }, - "outputs": [], "source": [ - "X = pd.get_dummies(df.drop(columns=[\n", - " \"race\",\n", - " \"race_all\",\n", - " \"discharge_disposition_id\",\n", - " \"readmitted\",\n", - " \"readmit_binary\",\n", - " \"readmit_30_days\"\n", + "X = pd.get_dummies(df.drop(columns=[\r\n", + " \"race\",\r\n", + " \"race_all\",\r\n", + " \"discharge_disposition_id\",\r\n", + " \"readmitted\",\r\n", + " \"readmit_binary\",\r\n", + " \"readmit_30_days\"\r\n", "]))" - ] + ], + "outputs": [], + "metadata": { + "id": "JXyRuCsri1cY" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "X.head() # sanity check" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1733,122 +1737,124 @@ }, "id": "kAA0sIhQUWFa", "outputId": "73d62224-bf2e-40d2-a29a-8909e26d3cf9" - }, - "outputs": [], - "source": [ - "X.head() # sanity check" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "ATzi8cKCD7V3" - }, "source": [ "We split our data into a training and test portion. The test portion will be used to evaluate our performance metric (i.e., balanced accuracy), but also for fairness assessment. The split is half/half for training and test to ensure that we have sufficient sample sizes for fairness assessment." - ] + ], + "metadata": { + "id": "ATzi8cKCD7V3" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "-wpnURazmJ4-" - }, - "outputs": [], "source": [ - "random_seed = 445\n", + "random_seed = 445\r\n", "np.random.seed(random_seed)" - ] + ], + "outputs": [], + "metadata": { + "id": "-wpnURazmJ4-" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "xgl_b-CUl7TW" - }, - "outputs": [], "source": [ - "X_train, X_test, Y_train, Y_test, A_train, A_test, df_train, df_test = train_test_split(\n", - " X,\n", - " Y,\n", - " A,\n", - " df,\n", - " test_size=0.50,\n", - " stratify=Y,\n", - " random_state=random_seed\n", + "X_train, X_test, Y_train, Y_test, A_train, A_test, df_train, df_test = train_test_split(\r\n", + " X,\r\n", + " Y,\r\n", + " A,\r\n", + " df,\r\n", + " test_size=0.50,\r\n", + " stratify=Y,\r\n", + " random_state=random_seed\r\n", ")" - ] + ], + "outputs": [], + "metadata": { + "id": "xgl_b-CUl7TW" + } }, { "cell_type": "markdown", - "metadata": { - "id": "eSMbXR9iVqr8" - }, "source": [ "Our performance metric is **balanced accuracy**, so for the purposes of training (but not evaluation!) we will resample the data set, so that it has the same number of positive and negative examples. This means that we can use estimators that optimize standard accuracy (although some estimators allow the use us importance weights).\n" - ] + ], + "metadata": { + "id": "eSMbXR9iVqr8" + } }, { "cell_type": "markdown", - "metadata": { - "id": "nPNQpb2ZN1ku" - }, "source": [ "Because we are downsampling the number of negative examples, we create a training dataset with a significantly lower number of data points. For more complex machine learning models, this lower number of training data points may affect the model's accuracy." - ] + ], + "metadata": { + "id": "nPNQpb2ZN1ku" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "L1aVgzyFNa4B" - }, - "outputs": [], "source": [ - "def resample_dataset(X_train, Y_train, A_train):\n", - "\n", - " negative_ids = Y_train[Y_train == 0].index\n", - " positive_ids = Y_train[Y_train == 1].index\n", - " balanced_ids = positive_ids.union(np.random.choice(a=negative_ids, size=len(positive_ids)))\n", - "\n", - " X_train = X_train.loc[balanced_ids, :]\n", - " Y_train = Y_train.loc[balanced_ids]\n", - " A_train = A_train.loc[balanced_ids, :]\n", + "def resample_dataset(X_train, Y_train, A_train):\r\n", + "\r\n", + " negative_ids = Y_train[Y_train == 0].index\r\n", + " positive_ids = Y_train[Y_train == 1].index\r\n", + " balanced_ids = positive_ids.union(np.random.choice(a=negative_ids, size=len(positive_ids)))\r\n", + "\r\n", + " X_train = X_train.loc[balanced_ids, :]\r\n", + " Y_train = Y_train.loc[balanced_ids]\r\n", + " A_train = A_train.loc[balanced_ids, :]\r\n", " return X_train, Y_train, A_train" - ] + ], + "outputs": [], + "metadata": { + "id": "L1aVgzyFNa4B" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "6Ogw-r3DQsds" - }, - "outputs": [], "source": [ "X_train_bal, Y_train_bal, A_train_bal = resample_dataset(X_train, Y_train, A_train)" - ] + ], + "outputs": [], + "metadata": { + "id": "6Ogw-r3DQsds" + } }, { "cell_type": "markdown", - "metadata": { - "id": "fRddJS7XXv5n" - }, "source": [ "## Save descriptive statistics of training and test data" - ] + ], + "metadata": { + "id": "fRddJS7XXv5n" + } }, { "cell_type": "markdown", - "metadata": { - "id": "hZ-T4lGxX0IQ" - }, "source": [ "We next evaluate and save descriptive statistics of the training dataset. These will be provided as part of _model cards_ to document our training." - ] + ], + "metadata": { + "id": "hZ-T4lGxX0IQ" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.countplot(x=\"race\", data=A_train_bal)\r\n", + "plt.title(\"Sensitive Attributes for Training Dataset\")\r\n", + "sensitive_train = figure_to_base64str(plt)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1856,17 +1862,17 @@ }, "id": "n3GhcUCm2LjD", "outputId": "9a3be0c7-99ae-4de6-b61b-cf0fd0bd6b0e" - }, - "outputs": [], - "source": [ - "sns.countplot(x=\"race\", data=A_train_bal)\n", - "plt.title(\"Sensitive Attributes for Training Dataset\")\n", - "sensitive_train = figure_to_base64str(plt)" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.countplot(x=Y_train_bal)\r\n", + "plt.title(\"Target Label Histogram for Training Dataset\")\r\n", + "outcome_train = figure_to_base64str(plt)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1874,17 +1880,17 @@ }, "id": "lIp3j8fD2LjE", "outputId": "48b4cbdf-c0ad-4a5b-d5f2-987d6b75229b" - }, - "outputs": [], - "source": [ - "sns.countplot(x=Y_train_bal)\n", - "plt.title(\"Target Label Histogram for Training Dataset\")\n", - "outcome_train = figure_to_base64str(plt)" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.countplot(x=\"race\", data=A_test)\r\n", + "plt.title(\"Sensitive Attributes for Testing Dataset\")\r\n", + "sensitive_test = figure_to_base64str(plt)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1892,17 +1898,17 @@ }, "id": "czIGZYhk2LjF", "outputId": "07490aa7-c22e-4614-8580-be8b8b6db229" - }, - "outputs": [], - "source": [ - "sns.countplot(x=\"race\", data=A_test)\n", - "plt.title(\"Sensitive Attributes for Testing Dataset\")\n", - "sensitive_test = figure_to_base64str(plt)" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "sns.countplot(x=Y_test)\r\n", + "plt.title(\"Target Label Histogram for Test Dataset\")\r\n", + "outcome_test = figure_to_base64str(plt)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1910,49 +1916,47 @@ }, "id": "YjhW0t9-2LjF", "outputId": "0785b4e1-88b8-4ab3-cb0b-9bf1d3df5ffe" - }, - "outputs": [], - "source": [ - "sns.countplot(x=Y_test)\n", - "plt.title(\"Target Label Histogram for Test Dataset\")\n", - "outcome_test = figure_to_base64str(plt)" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "4V523GQbYobT" - }, "source": [ "## Train the model" - ] + ], + "metadata": { + "id": "4V523GQbYobT" + } }, { "cell_type": "markdown", - "metadata": { - "id": "g7jwN2cVbO0g" - }, "source": [ "We train a logistic regression model and save its predictions on test data for analysis." - ] + ], + "metadata": { + "id": "g7jwN2cVbO0g" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "f6nKDzt164vw" - }, - "outputs": [], "source": [ - "unmitigated_pipeline = Pipeline(steps=[\n", - " (\"preprocessing\", StandardScaler()),\n", - " (\"logistic_regression\", LogisticRegression(max_iter=1000))\n", + "unmitigated_pipeline = Pipeline(steps=[\r\n", + " (\"preprocessing\", StandardScaler()),\r\n", + " (\"logistic_regression\", LogisticRegression(max_iter=1000))\r\n", "])" - ] + ], + "outputs": [], + "metadata": { + "id": "f6nKDzt164vw" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "unmitigated_pipeline.fit(X_train_bal, Y_train_bal)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1960,36 +1964,37 @@ }, "id": "ld9clGbHl7tv", "outputId": "89fa7f0d-680f-424f-a7cd-96686987a943" - }, - "outputs": [], - "source": [ - "unmitigated_pipeline.fit(X_train_bal, Y_train_bal)" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "Ok-eREU0xbAD" - }, - "outputs": [], "source": [ - "Y_pred_proba = unmitigated_pipeline.predict_proba(X_test)[:,1]\n", + "Y_pred_proba = unmitigated_pipeline.predict_proba(X_test)[:,1]\r\n", "Y_pred = unmitigated_pipeline.predict(X_test)" - ] + ], + "outputs": [], + "metadata": { + "id": "Ok-eREU0xbAD" + } }, { "cell_type": "markdown", - "metadata": { - "id": "nkA0K8KV0HeD" - }, "source": [ "Check model performance on test data." - ] + ], + "metadata": { + "id": "nkA0K8KV0HeD" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Plot ROC curve of probabilistic predictions\r\n", + "plot_roc_curve(unmitigated_pipeline, X_test, Y_test);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -1997,54 +2002,54 @@ }, "id": "nz7QJOLx0RVH", "outputId": "ef446998-8269-4e9b-c8ba-43a777b947d8" - }, - "outputs": [], - "source": [ - "# Plot ROC curve of probabilistic predictions\n", - "plot_roc_curve(unmitigated_pipeline, X_test, Y_test);" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Show balanced accuracy rate of the 0/1 predictions\r\n", + "balanced_accuracy_score(Y_test, Y_pred)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "pxYppCAy1owq", "outputId": "b9febbe3-8016-43a7-c14a-98a94f05716a" - }, - "outputs": [], - "source": [ - "# Show balanced accuracy rate of the 0/1 predictions\n", - "balanced_accuracy_score(Y_test, Y_pred)" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "jmWrDs5N2HVD" - }, "source": [ "As we see, the performance of the model is well above the performance of a coin flip (whose performance would be 0.5 in both cases), albeit it is quite far from a perfect classifier (whose performance would be 1.0 in both cases).\n" - ] + ], + "metadata": { + "id": "jmWrDs5N2HVD" + } }, { "cell_type": "markdown", - "metadata": { - "id": "AmhwS1Z9VnK9" - }, "source": [ "## Inspect the coefficients of trained model\n", "\n", "We check the coefficients of the fitted model to make sure that they \"makes sense\". While subjective, this step is important and helps catch mistakes and might point out to some fairness issues. However, we will systematically assess the fairness of the model in the next section.\n", "\n", "*Note that coefficients are also a proxy for \"feature importance\", but this interpretation can be misleading when features are highly correlated.*" - ] + ], + "metadata": { + "id": "AmhwS1Z9VnK9" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "coef_series = pd.Series(data=unmitigated_pipeline.named_steps[\"logistic_regression\"].coef_[0], index=X.columns)\r\n", + "coef_series.sort_values().plot.barh(figsize=(4, 12), legend=False);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2052,39 +2057,31 @@ }, "id": "Owzkar8R9Cyy", "outputId": "7c073a93-e3e5-4f22-e089-217f97967df7" - }, - "outputs": [], - "source": [ - "coef_series = pd.Series(data=unmitigated_pipeline.named_steps[\"logistic_regression\"].coef_[0], index=X.columns)\n", - "coef_series.sort_values().plot.barh(figsize=(4, 12), legend=False);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "hX8CrWjhD7MB" - }, "source": [ "# **Fairness assessment**" - ] + ], + "metadata": { + "id": "hX8CrWjhD7MB" + } }, { "cell_type": "markdown", - "metadata": { - "id": "0CS9-jaxtxh2" - }, "source": [ "## Measuring fairness-related harms\n", "\n", "\n", "\n" - ] + ], + "metadata": { + "id": "0CS9-jaxtxh2" + } }, { "cell_type": "markdown", - "metadata": { - "id": "s8TMm9w8duVY" - }, "source": [ "The goal of fairness assessment is to answer the question: *Which groups of people may be disproportionately negatively impacted by an AI system and in what ways?*\n", "\n", @@ -2095,26 +2092,26 @@ "4. Compare quantified harms across the groups\n", "\n", "We next examine these four steps in more detail." - ] + ], + "metadata": { + "id": "s8TMm9w8duVY" + } }, { "cell_type": "markdown", - "metadata": { - "id": "X6hMFmzmPbL6" - }, "source": [ "### 1. Identify harms\n", "\n", "For example, in a system for screening job applications, qualified candidates that are automatically rejected experience an allocation harm. In a speech-to-text transcription system, high error rates constitute harm in the quality of service.\n", "\n", "**In the health care scenario**, the patients that would benefit from a care management program, but are not recommended for it experience an allocation harm. In the context of the classification scenario these are **FALSE NEGATIVES**." - ] + ], + "metadata": { + "id": "X6hMFmzmPbL6" + } }, { "cell_type": "markdown", - "metadata": { - "id": "aqqUk1mjPnpM" - }, "source": [ "### 2. Identify the groups that might be harmed\n", "\n", @@ -2123,13 +2120,13 @@ "It is also important to consider group intersections, for example, in addition to considering groups according to gender and groups according to race, it is also important to consider their intersections (e.g., Black women, Latinx nonbinary people, etc.).\n", "\n", "**In the health care scenario**, based on the previous work, we focus on groups defined by **RACE**." - ] + ], + "metadata": { + "id": "aqqUk1mjPnpM" + } }, { "cell_type": "markdown", - "metadata": { - "id": "nmvSqI3dPrVk" - }, "source": [ "### 3. Quantify harms\n", "\n", @@ -2144,13 +2141,13 @@ " * **selection rate**: overall fraction of patients that are recommended for the care management program (regardless of whether they are readmittted with 30 days or no); this quantifies benefit; here the assumption is that all patients benefit similarly from the extra care.\n", "\n", "There are several reasons for including selection rate in addition to false negative rate. We would like to monitor how the benefits are allocated, focusing on groups that might be disadvantaged. Another reason is to get extra robustness in our assessement, because our measure (i.e., readmission within 30 days) is only an imperfect measure of our construct (who is most likely to benefit from the care management program). The auxiliary metrics, like selection rate, may alert us to large disparities in how the benefit is allocated, and allow us to catch issues that we might have missed.\n" - ] + ], + "metadata": { + "id": "nmvSqI3dPrVk" + } }, { "cell_type": "markdown", - "metadata": { - "id": "fpJXt6miPvRX" - }, "source": [ "### 4. Compare quantified harms across the groups\n", "\n", @@ -2159,13 +2156,13 @@ "\n", "To summarize the disparities in errors (or other metrics), we may want to report quantities such as the **difference** or **ratio** of the metric values between the best and the worst slice. In settings where the goal is to guarantee certain minimum quality of service (such as speech recognition), it is also meaningful to report the **worst performance** across all considered groups.\n", "\n" - ] + ], + "metadata": { + "id": "fpJXt6miPvRX" + } }, { "cell_type": "markdown", - "metadata": { - "id": "7Is_zdXvnW0s" - }, "source": [ "For example, when comparing false negative rate across groups defined by race, we may summarize our findings with a table like the following:\n", "\n", @@ -2179,78 +2176,100 @@ "|_largest difference_| 0.24   (best is 0.0)|\n", "|_smallest ratio_| 0.64   (best is 1.0)|\n", "|_maximum_
_(worst-case) FNR_|0.67|" - ] + ], + "metadata": { + "id": "7Is_zdXvnW0s" + } }, { "cell_type": "markdown", - "metadata": { - "id": "9CjHlopBDgSG" - }, "source": [ "## Fairness assessment with `MetricFrame`" - ] + ], + "metadata": { + "id": "9CjHlopBDgSG" + } }, { "cell_type": "markdown", - "metadata": { - "id": "epJO2baHV2Dy" - }, "source": [ "Fairlearn provides the data structure called `MetricFrame` to enable evaluation of disaggregated metrics. We will show how to use a `MetricFrame` object to assess the trained `LogisticRegression` classifier for potential fairness-related harms.\n", "\n" - ] + ], + "metadata": { + "id": "epJO2baHV2Dy" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# In its simplest form MetricFrame takes four arguments:\r\n", + "# metric_function with signature metric_function(y_true, y_pred)\r\n", + "# y_true: array of labels\r\n", + "# y_pred: array of predictions\r\n", + "# sensitive_features: array of sensitive feature values\r\n", + "\r\n", + "mf1 = MetricFrame(metrics=false_negative_rate,\r\n", + " y_true=Y_test,\r\n", + " y_pred=Y_pred,\r\n", + " sensitive_features=df_test['race'])\r\n", + "\r\n", + "# The disaggregated metrics are stored in a pandas Series mf1.by_group:\r\n", + "\r\n", + "mf1.by_group" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0iiAYRvoduPh", "outputId": "3287b55e-3610-424d-a99d-a6e19c2b1693" - }, - "outputs": [], - "source": [ - "# In its simplest form MetricFrame takes four arguments:\n", - "# metric_function with signature metric_function(y_true, y_pred)\n", - "# y_true: array of labels\n", - "# y_pred: array of predictions\n", - "# sensitive_features: array of sensitive feature values\n", - "\n", - "mf1 = MetricFrame(metrics=false_negative_rate,\n", - " y_true=Y_test,\n", - " y_pred=Y_pred,\n", - " sensitive_features=df_test['race'])\n", - "\n", - "# The disaggregated metrics are stored in a pandas Series mf1.by_group:\n", - "\n", - "mf1.by_group" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# The largest difference, smallest ratio and worst-case performance are accessed as\r\n", + "# mf1.difference(), mf1.ratio(), mf1.group_max()\r\n", + "\r\n", + "print(f\"difference: {mf1.difference():.3}\\n\"\r\n", + " f\"ratio: {mf1.ratio():.3}\\n\"\r\n", + " f\"max across groups: {mf1.group_max():.3}\")" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "5tl1Qxt1fT2v", "outputId": "e2af1585-7602-4664-f396-2f0d395d4ad2" - }, - "outputs": [], - "source": [ - "# The largest difference, smallest ratio and worst-case performance are accessed as\n", - "# mf1.difference(), mf1.ratio(), mf1.group_max()\n", - "\n", - "print(f\"difference: {mf1.difference():.3}\\n\"\n", - " f\"ratio: {mf1.ratio():.3}\\n\"\n", - " f\"max across groups: {mf1.group_max():.3}\")" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# You can also evaluate multiple metrics by providing a dictionary\r\n", + "\r\n", + "metrics_dict = {\r\n", + " \"selection_rate\": selection_rate,\r\n", + " \"false_negative_rate\": false_negative_rate,\r\n", + " \"balanced_accuracy\": balanced_accuracy_score,\r\n", + "}\r\n", + "\r\n", + "metricframe_unmitigated = MetricFrame(metrics=metrics_dict,\r\n", + " y_true=Y_test,\r\n", + " y_pred=Y_pred,\r\n", + " sensitive_features=df_test['race'])\r\n", + "\r\n", + "# The disaggregated metrics are then stored in a pandas DataFrame:\r\n", + "\r\n", + "metricframe_unmitigated.by_group" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2258,48 +2277,38 @@ }, "id": "R2zmBHo5gk-F", "outputId": "e970f25b-7218-4430-8969-d8f891ddce27" - }, - "outputs": [], - "source": [ - "# You can also evaluate multiple metrics by providing a dictionary\n", - "\n", - "metrics_dict = {\n", - " \"selection_rate\": selection_rate,\n", - " \"false_negative_rate\": false_negative_rate,\n", - " \"balanced_accuracy\": balanced_accuracy_score,\n", - "}\n", - "\n", - "metricframe_unmitigated = MetricFrame(metrics=metrics_dict,\n", - " y_true=Y_test,\n", - " y_pred=Y_pred,\n", - " sensitive_features=df_test['race'])\n", - "\n", - "# The disaggregated metrics are then stored in a pandas DataFrame:\n", - "\n", - "metricframe_unmitigated.by_group" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# The largest difference, smallest ratio, and the maximum and minimum values\r\n", + "# across the groups are then all pandas Series, for example:\r\n", + "\r\n", + "metricframe_unmitigated.difference()" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Hc29jRJrhlSC", "outputId": "e2ffee50-2764-434f-d322-2a8fdafe6a09" - }, - "outputs": [], - "source": [ - "# The largest difference, smallest ratio, and the maximum and minimum values\n", - "# across the groups are then all pandas Series, for example:\n", - "\n", - "metricframe_unmitigated.difference()" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# You'll probably want to view them transposed:\r\n", + "\r\n", + "pd.DataFrame({'difference': metricframe_unmitigated.difference(),\r\n", + " 'ratio': metricframe_unmitigated.ratio(),\r\n", + " 'group_min': metricframe_unmitigated.group_min(),\r\n", + " 'group_max': metricframe_unmitigated.group_max()}).T" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2307,20 +2316,18 @@ }, "id": "bVbjFa4Aig9Y", "outputId": "228f2de1-4de8-4059-b166-c75550e52de2" - }, - "outputs": [], - "source": [ - "# You'll probably want to view them transposed:\n", - "\n", - "pd.DataFrame({'difference': metricframe_unmitigated.difference(),\n", - " 'ratio': metricframe_unmitigated.ratio(),\n", - " 'group_min': metricframe_unmitigated.group_min(),\n", - " 'group_max': metricframe_unmitigated.group_max()}).T" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# You can also easily plot all of the metrics using DataFrame plotting capabilities\r\n", + "\r\n", + "metricframe_unmitigated.by_group.plot.bar(subplots=True, layout= [1,3], figsize=(12, 4),\r\n", + " legend=False, rot=-45, position=1.5);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2328,212 +2335,209 @@ }, "id": "DvjRBIcjjSkl", "outputId": "0aebd5a1-f287-4d75-cdd3-8af1daf1d190" - }, - "outputs": [], - "source": [ - "# You can also easily plot all of the metrics using DataFrame plotting capabilities\n", - "\n", - "metricframe_unmitigated.by_group.plot.bar(subplots=True, layout= [1,3], figsize=(12, 4),\n", - " legend=False, rot=-45, position=1.5);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "b5C3SITjuPUm" - }, "source": [ "According to the above bar chart, it seems that the group *Unknown* is selected for the care management program less often than other groups as reflected by the selection rate. Also this group experiences the largest false negative rate, so a larger fraction of group members that are likely to benefit from the care management program are not selected. Finally, the balanced accuracy on this group is also the lowest.\n", "\n" - ] + ], + "metadata": { + "id": "b5C3SITjuPUm" + } }, { "cell_type": "markdown", - "metadata": { - "id": "c2Qs68rv2_Vg" - }, "source": [ "We observe disparity, even though we did not include race in our model. There's a variety of reasons why such disparities may occur. It could be due to representational issues (i.e., not enough instances per group), or because the feature distribution itself differs across groups (i.e., different relationship between features and target variable, obvious example would be people with darker skin in facial recognition systems, but can be much more subtle). Real-world applications often exhibit both kinds of issues at the same time." - ] + ], + "metadata": { + "id": "c2Qs68rv2_Vg" + } }, { "cell_type": "markdown", - "metadata": { - "id": "n_1Rm8PbPmmk" - }, "source": [ "\n", "## Exercise: Train other fairness-unaware models" - ] + ], + "metadata": { + "id": "n_1Rm8PbPmmk" + } }, { "cell_type": "markdown", - "metadata": { - "id": "oeQF5qT6Qs-C" - }, "source": [ "In this section, you'll be training your own fairness-unaware model and evaluate the model using the `MetricFrame` for fairness-related harms." - ] + ], + "metadata": { + "id": "oeQF5qT6Qs-C" + } }, { "cell_type": "markdown", - "metadata": { - "id": "SsHy-Os0oVQU" - }, "source": [ "We encourage you to explore the model's performance across different sensitive features (such as `age` or `gender`) as well as different model performance metrics." - ] + ], + "metadata": { + "id": "SsHy-Os0oVQU" + } }, { "cell_type": "markdown", - "metadata": { - "id": "61GU_zrFSC-6" - }, "source": [ "1.) First, let's train our machine learning model. We'll create a `HistGradientBoostingClassifier` and fit it to the balanced training data set." - ] + ], + "metadata": { + "id": "61GU_zrFSC-6" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "cUPOfey0QtJ_" - }, - "outputs": [], "source": [ - "from sklearn.experimental import enable_hist_gradient_boosting\n", - "from sklearn.ensemble import HistGradientBoostingClassifier\n", - "\n", - "# Create your model here\n", - "clf = HistGradientBoostingClassifier()\n", - "\n", - "# Fit the model to the training data\n", - "clf.fit(X_train_bal, Y_train_bal)\n", + "from sklearn.experimental import enable_hist_gradient_boosting\r\n", + "from sklearn.ensemble import HistGradientBoostingClassifier\r\n", + "\r\n", + "# Create your model here\r\n", + "clf = HistGradientBoostingClassifier()\r\n", + "\r\n", + "# Fit the model to the training data\r\n", + "clf.fit(X_train_bal, Y_train_bal)\r\n", "exercise_pred = clf.predict(X_test)" - ] + ], + "outputs": [], + "metadata": { + "id": "cUPOfey0QtJ_" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "bSIDkuV4_Rou" - }, - "outputs": [], "source": [ - "#from sklearn.experimental import enable_hist_gradient_boosting\n", - "#from sklearn.ensemble import HistGradientBoostingClassifier\n", - "\n", - "# Create your model here\n", - "#clf = HistGradientBoostingClassifier()\n", - "\n", - "# Fit the model to the training data\n", - "#clf.fit(__________, ________)\n", + "#from sklearn.experimental import enable_hist_gradient_boosting\r\n", + "#from sklearn.ensemble import HistGradientBoostingClassifier\r\n", + "\r\n", + "# Create your model here\r\n", + "#clf = HistGradientBoostingClassifier()\r\n", + "\r\n", + "# Fit the model to the training data\r\n", + "#clf.fit(__________, ________)\r\n", "#exercise_pred = clf.predict(______)" - ] + ], + "outputs": [], + "metadata": { + "id": "bSIDkuV4_Rou" + } }, { "cell_type": "markdown", + "source": [ + "2.) Next, let's evaluate the fairness of the model using the `MetricFrame`. In the below cells, create a `MetricFrame` that looks at the following metrics:\r\n", + "\r\n", + "\r\n", + "* _Count_: The number of data points belonging to each sensitive feature category.\r\n", + "* _False Positive Rate_: $\\dfrac{FP}{FP+TN}$\r\n", + "* _Recall Score_: $\\dfrac{TP}{TP+FN}$\r\n", + "\r\n", + "As an extra challenge, you can use the prediction probabilities to compute the _ROC AUC Score_ for each sensitive group pair.\r\n", + "\r\n" + ], "metadata": { "id": "Fnnles-p6OXr" - }, - "source": [ - "2.) Next, let's evaluate the fairness of the model using the `MetricFrame`. In the below cells, create a `MetricFrame` that looks at the following metrics:\n", - "\n", - "\n", - "* _Count_: The number of data points belonging to each sensitive feature category.\n", - "* _False Positive Rate_: $\\dfrac{FN}{FN+TP}$\n", - "* _Recall Score_: $\\dfrac{TP}{TP+FN}$\n", - "\n", - "As an extra challenge, you can use the prediction probabilities to compute the _ROC AUC Score_ for each sensitive group pair.\n", - "\n" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "EXUYQPuwQtNm" - }, - "outputs": [], "source": [ - "# Define additional fairness metrics of interest here\n", - "exercise_metrics = {\n", - " \"count\": count,\n", - " \"false_positive_rate\": false_positive_rate,\n", - " \"recall_score\": recall_score\n", + "# Define additional fairness metrics of interest here\r\n", + "exercise_metrics = {\r\n", + " \"count\": count,\r\n", + " \"false_positive_rate\": false_positive_rate,\r\n", + " \"recall_score\": recall_score\r\n", "}" - ] + ], + "outputs": [], + "metadata": { + "id": "EXUYQPuwQtNm" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "bcf-x1oA_jP5" - }, - "outputs": [], "source": [ - "#exercise_metrics = {\n", - "# \"count\": count,\n", - "# \"false_positive_rate\": _______,\n", - "# \"recall_score\": _______\n", + "#exercise_metrics = {\r\n", + "# \"count\": count,\r\n", + "# \"false_positive_rate\": _______,\r\n", + "# \"recall_score\": _______\r\n", "#}" - ] + ], + "outputs": [], + "metadata": { + "id": "bcf-x1oA_jP5" + } }, { "cell_type": "markdown", - "metadata": { - "id": "Bll-8GAWJF6p" - }, "source": [ "Now, let's create our `MetricFrame` using the metrics listed above with the sensitive groups of `race` and `gender`." - ] + ], + "metadata": { + "id": "Bll-8GAWJF6p" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "zWLsCa_xQtbE" - }, - "outputs": [], "source": [ - "# Create a MetricFrame on your model's results\n", - "metricframe_exercise = MetricFrame(\n", - " metrics=exercise_metrics,\n", - " y_true=Y_test,\n", - " y_pred=exercise_pred,\n", - " sensitive_features=A_test\n", + "# Create a MetricFrame on your model's results\r\n", + "metricframe_exercise = MetricFrame(\r\n", + " metrics=exercise_metrics,\r\n", + " y_true=Y_test,\r\n", + " y_pred=exercise_pred,\r\n", + " sensitive_features=A_test\r\n", ")" - ] + ], + "outputs": [], + "metadata": { + "id": "zWLsCa_xQtbE" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "jAjzjCqh_fNx" - }, - "outputs": [], "source": [ - "#metricframe_exercise = MetricFrame(\n", - "# metrics=__________,\n", - "# y_true=Y_test,\n", - "# y_pred=__________,\n", - "# sensitive_features=_____\n", + "#metricframe_exercise = MetricFrame(\r\n", + "# metrics=__________,\r\n", + "# y_true=Y_test,\r\n", + "# y_pred=__________,\r\n", + "# sensitive_features=_____\r\n", "#)" - ] + ], + "outputs": [], + "metadata": { + "id": "jAjzjCqh_fNx" + } }, { "cell_type": "markdown", - "metadata": { - "id": "QeghVCbLZOf5" - }, "source": [ "3.) Finally, play around with the plotting capabilities of the `MetricFrame` in the below section.\n", "\n" - ] + ], + "metadata": { + "id": "QeghVCbLZOf5" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "metricframe_exercise.by_group" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2541,26 +2545,28 @@ }, "id": "6dQNSB02D_ba", "outputId": "a19a040b-adba-4a98-a7ea-673f892375ed" - }, - "outputs": [], - "source": [ - "metricframe_exercise.by_group" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "Nd4D17ME_hB2" - }, - "outputs": [], "source": [ "#metricframe_exercise._______" - ] + ], + "outputs": [], + "metadata": { + "id": "Nd4D17ME_hB2" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Plot some of the performance disparities here\r\n", + "metricframe_exercise.by_group.plot.bar(subplots=True, layout=[1,4], figsize=(12, 4),\r\n", + " legend=False, rot=-45, position=1.5)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2568,157 +2574,156 @@ }, "id": "-ylW8y2CQtjb", "outputId": "444e074c-b955-462b-a0fd-61e3f1778714" - }, - "outputs": [], - "source": [ - "# Plot some of the performance disparities here\n", - "metricframe_exercise.by_group.plot.bar(subplots=True, layout=[1,4], figsize=(12, 4),\n", - " legend=False, rot=-45, position=1.5)" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "_xaLx6Br_hyc" - }, - "outputs": [], "source": [ "#metricframe_exercise.by_group.____.bar(subplots=_____, layout=[1,4], figsize=(12, 4), legend=False, rot=-45, position=1.5)" - ] + ], + "outputs": [], + "metadata": { + "id": "_xaLx6Br_hyc" + } }, { "cell_type": "markdown", - "metadata": { - "id": "Me1ocEi2kEgw" - }, "source": [ "The charts above are based on test data, so without any uncertainty quantification (such as error bars or confidence intervals), we cannot reliably compare these data statistics. Next optional section shows how to augment MetricFrame with the report of error bars.\n", "\n", "## Adding error bars [OPTIONAL SECTION]" - ] + ], + "metadata": { + "id": "Me1ocEi2kEgw" + } }, { "cell_type": "markdown", - "metadata": { - "id": "9l8YJ8qQdehm" - }, "source": [ "In this section, we define new custom metrics that quantify errors in our estimates of selection rate, false negative rate and balanced accuracy, and then review our metrics again." - ] - }, - { - "cell_type": "code", - "execution_count": null, + ], "metadata": { - "id": "OiP-uXr_FLtz" - }, - "outputs": [], - "source": [ - "# All of our error bar calculations are based on normal approximation to\n", - "# the binomial variables.\n", - "\n", - "def error_bar_normal(n_successes, n_trials, z=1.96):\n", - " \"\"\"\n", - " Computes the error bars for the parameter p of a binomial variable\n", - " using normal approximation. The default value z corresponds to the 95%\n", - " confidence interval.\n", - " \"\"\"\n", - " point_est = n_successes / n_trials\n", - " error_bar = z*np.sqrt(point_est*(1-point_est))/np.sqrt(n_trials)\n", - " return error_bar\n", - "\n", - "def fpr_error(Y_true, Y_pred):\n", - " \"\"\"\n", - " Compute the 95%-error bar for the false positive rate\n", - " \"\"\"\n", - " tn, fp, fn, tp = confusion_matrix(Y_true, Y_pred).ravel()\n", - " return error_bar_normal(fp, tn+fp)\n", - "\n", - "def fnr_error(Y_true, Y_pred):\n", - " \"\"\"\n", - " Compute the 95%-error bar for the false negative rate\n", - " \"\"\"\n", - " tn, fp, fn, tp = confusion_matrix(Y_true, Y_pred).ravel()\n", - " return error_bar_normal(fn, fn+tp)\n", - "\n", - "def selection_rate_error(Y_true, Y_pred):\n", - " \"\"\"\n", - " Compute the 95%-error bar for the selection rate\n", - " \"\"\"\n", - " tn, fp, fn, tp = confusion_matrix(Y_true, Y_pred).ravel()\n", - " return error_bar_normal(tp+fp, tn+fp+fn+tp)\n", - "\n", - "def balanced_accuracy_error(Y_true, Y_pred):\n", - " \"\"\"\n", - " Compute the 95%-error bar for the balanced accuracy\n", - " \"\"\"\n", - " fnr_err, fpr_err = fnr_error(Y_true, Y_pred), fpr_error(Y_true, Y_pred)\n", + "id": "9l8YJ8qQdehm" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "# All of our error bar calculations are based on normal approximation to\r\n", + "# the binomial variables.\r\n", + "\r\n", + "def error_bar_normal(n_successes, n_trials, z=1.96):\r\n", + " \"\"\"\r\n", + " Computes the error bars for the parameter p of a binomial variable\r\n", + " using normal approximation. The default value z corresponds to the 95%\r\n", + " confidence interval.\r\n", + " \"\"\"\r\n", + " point_est = n_successes / n_trials\r\n", + " error_bar = z*np.sqrt(point_est*(1-point_est))/np.sqrt(n_trials)\r\n", + " return error_bar\r\n", + "\r\n", + "def fpr_error(Y_true, Y_pred):\r\n", + " \"\"\"\r\n", + " Compute the 95%-error bar for the false positive rate\r\n", + " \"\"\"\r\n", + " tn, fp, fn, tp = confusion_matrix(Y_true, Y_pred).ravel()\r\n", + " return error_bar_normal(fp, tn+fp)\r\n", + "\r\n", + "def fnr_error(Y_true, Y_pred):\r\n", + " \"\"\"\r\n", + " Compute the 95%-error bar for the false negative rate\r\n", + " \"\"\"\r\n", + " tn, fp, fn, tp = confusion_matrix(Y_true, Y_pred).ravel()\r\n", + " return error_bar_normal(fn, fn+tp)\r\n", + "\r\n", + "def selection_rate_error(Y_true, Y_pred):\r\n", + " \"\"\"\r\n", + " Compute the 95%-error bar for the selection rate\r\n", + " \"\"\"\r\n", + " tn, fp, fn, tp = confusion_matrix(Y_true, Y_pred).ravel()\r\n", + " return error_bar_normal(tp+fp, tn+fp+fn+tp)\r\n", + "\r\n", + "def balanced_accuracy_error(Y_true, Y_pred):\r\n", + " \"\"\"\r\n", + " Compute the 95%-error bar for the balanced accuracy\r\n", + " \"\"\"\r\n", + " fnr_err, fpr_err = fnr_error(Y_true, Y_pred), fpr_error(Y_true, Y_pred)\r\n", " return np.sqrt(fnr_err**2 + fpr_err**2)/2" - ] + ], + "outputs": [], + "metadata": { + "id": "OiP-uXr_FLtz" + } }, { "cell_type": "markdown", - "metadata": { - "id": "qHaXfBYWp6ob" - }, "source": [ "We next create a metric frame that includes the sample sizes and error bar sizes in addition to the metrics that we have used previously." - ] - }, - { - "cell_type": "code", - "execution_count": null, + ], "metadata": { - "id": "OlEq6ogfyHb6" - }, - "outputs": [], - "source": [ - "metrics_with_err_bars = {\n", - " \"count\": count,\n", - " \"selection_rate\": selection_rate,\n", - " \"selection_err_bar\": selection_rate_error,\n", - " \"false_negative_rate\": false_negative_rate,\n", - " \"fnr_err_bar\": fnr_error,\n", - " \"balanced_accuracy\": balanced_accuracy_score,\n", - " \"bal_acc_err_bar\": balanced_accuracy_error\n", - "}\n", - "\n", - "# sometimes we will only want to display metrics without error bars\n", - "metrics_to_display = [\n", - " \"count\",\n", - " \"selection_rate\",\n", - " \"false_negative_rate\",\n", - " \"balanced_accuracy\"\n", - "]\n", - "\n", - "# sometimes we will only want to show the difference values of the metrics other than count\n", - "differences_to_display = [\n", - " \"selection_rate\",\n", - " \"false_negative_rate\",\n", - " \"balanced_accuracy\"\n", + "id": "qHaXfBYWp6ob" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "metrics_with_err_bars = {\r\n", + " \"count\": count,\r\n", + " \"selection_rate\": selection_rate,\r\n", + " \"selection_err_bar\": selection_rate_error,\r\n", + " \"false_negative_rate\": false_negative_rate,\r\n", + " \"fnr_err_bar\": fnr_error,\r\n", + " \"balanced_accuracy\": balanced_accuracy_score,\r\n", + " \"bal_acc_err_bar\": balanced_accuracy_error\r\n", + "}\r\n", + "\r\n", + "# sometimes we will only want to display metrics without error bars\r\n", + "metrics_to_display = [\r\n", + " \"count\",\r\n", + " \"selection_rate\",\r\n", + " \"false_negative_rate\",\r\n", + " \"balanced_accuracy\"\r\n", + "]\r\n", + "\r\n", + "# sometimes we will only want to show the difference values of the metrics other than count\r\n", + "differences_to_display = [\r\n", + " \"selection_rate\",\r\n", + " \"false_negative_rate\",\r\n", + " \"balanced_accuracy\"\r\n", "]" - ] + ], + "outputs": [], + "metadata": { + "id": "OlEq6ogfyHb6" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "55GOqFTYxbCi" - }, - "outputs": [], "source": [ - "metricframe_unmitigated_w_err = MetricFrame(\n", - " metrics=metrics_with_err_bars,\n", - " y_true=Y_test,\n", - " y_pred=Y_pred,\n", - " sensitive_features=A_test\n", + "metricframe_unmitigated_w_err = MetricFrame(\r\n", + " metrics=metrics_with_err_bars,\r\n", + " y_true=Y_test,\r\n", + " y_pred=Y_pred,\r\n", + " sensitive_features=A_test\r\n", ")" - ] + ], + "outputs": [], + "metadata": { + "id": "55GOqFTYxbCi" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "unmitigated_groups = metricframe_unmitigated_w_err.by_group\r\n", + "unmitigated_groups # show both the metrics as well as the error bars" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2726,56 +2731,55 @@ }, "id": "MlZKPNkHxbFb", "outputId": "9d6cb95c-538a-42e7-f3b4-349c74258902" - }, - "outputs": [], - "source": [ - "unmitigated_groups = metricframe_unmitigated_w_err.by_group\n", - "unmitigated_groups # show both the metrics as well as the error bars" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "hh5F-A5C5rSV" - }, "source": [ "We see that for smaller sample sizes we have larger error bars. The problem is further exacerbated for false negative rate, which is estimated only over *positive examples* and so its sample sizes is further reduced due to label imbalance.\n", "\n", "We next visualize the metrics with the corresponding error bars using a custom plotting function." - ] - }, - { - "cell_type": "code", - "execution_count": null, + ], "metadata": { - "id": "rqFPLsATwROr" - }, - "outputs": [], - "source": [ - "def plot_group_metrics_with_error_bars(metricframe, metric, error_name):\n", - " \"\"\"\n", - " Plots the disaggregated `metric` for each group with an associated\n", - " error bar. Both metric and the erro bar are provided as columns in the \n", - " provided metricframe.\n", - " \"\"\"\n", - " grouped_metrics = metricframe.by_group\n", - " point_estimates = grouped_metrics[metric]\n", - " error_bars = grouped_metrics[error_name]\n", - " lower_bounds = point_estimates - error_bars\n", - " upper_bounds = point_estimates + error_bars\n", - "\n", - " x_axis_names = [str(name) for name in error_bars.index.to_flat_index().tolist()]\n", - " plt.vlines(x_axis_names, lower_bounds, upper_bounds, linestyles=\"dashed\", alpha=0.45)\n", - " plt.scatter(x_axis_names, point_estimates, s=25)\n", - " plt.xticks(rotation=0)\n", - " y_start, y_end = np.round(min(lower_bounds), decimals=2), np.round(max(upper_bounds), decimals=2)\n", - " plt.yticks(np.arange(y_start, y_end, 0.05))\n", + "id": "hh5F-A5C5rSV" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "def plot_group_metrics_with_error_bars(metricframe, metric, error_name):\r\n", + " \"\"\"\r\n", + " Plots the disaggregated `metric` for each group with an associated\r\n", + " error bar. Both metric and the erro bar are provided as columns in the \r\n", + " provided metricframe.\r\n", + " \"\"\"\r\n", + " grouped_metrics = metricframe.by_group\r\n", + " point_estimates = grouped_metrics[metric]\r\n", + " error_bars = grouped_metrics[error_name]\r\n", + " lower_bounds = point_estimates - error_bars\r\n", + " upper_bounds = point_estimates + error_bars\r\n", + "\r\n", + " x_axis_names = [str(name) for name in error_bars.index.to_flat_index().tolist()]\r\n", + " plt.vlines(x_axis_names, lower_bounds, upper_bounds, linestyles=\"dashed\", alpha=0.45)\r\n", + " plt.scatter(x_axis_names, point_estimates, s=25)\r\n", + " plt.xticks(rotation=0)\r\n", + " y_start, y_end = np.round(min(lower_bounds), decimals=2), np.round(max(upper_bounds), decimals=2)\r\n", + " plt.yticks(np.arange(y_start, y_end, 0.05))\r\n", " plt.ylabel(metric)" - ] + ], + "outputs": [], + "metadata": { + "id": "rqFPLsATwROr" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "plot_group_metrics_with_error_bars(metricframe_unmitigated_w_err, \"selection_rate\", \"selection_err_bar\")" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2783,15 +2787,15 @@ }, "id": "dsRFpuXfzrUA", "outputId": "0f4adb40-6c34-45a2-b8ca-b50d151c4b55" - }, - "outputs": [], - "source": [ - "plot_group_metrics_with_error_bars(metricframe_unmitigated_w_err, \"selection_rate\", \"selection_err_bar\")" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "plot_group_metrics_with_error_bars(metricframe_unmitigated_w_err, \"false_negative_rate\", \"fnr_err_bar\")" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2799,15 +2803,15 @@ }, "id": "B68Q2ZIgzrcE", "outputId": "368fed7e-80c6-4fca-ebef-d61cdd09e800" - }, - "outputs": [], - "source": [ - "plot_group_metrics_with_error_bars(metricframe_unmitigated_w_err, \"false_negative_rate\", \"fnr_err_bar\")" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "plot_group_metrics_with_error_bars(metricframe_unmitigated_w_err, \"balanced_accuracy\", \"bal_acc_err_bar\")" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2815,46 +2819,39 @@ }, "id": "htoUhyirpqZt", "outputId": "c83ac73e-ccbf-4a53-8002-c2dc012b8e4c" - }, - "outputs": [], - "source": [ - "plot_group_metrics_with_error_bars(metricframe_unmitigated_w_err, \"balanced_accuracy\", \"bal_acc_err_bar\")" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "SAMhqvycR7eE" - }, "source": [ "As we see above, even accounting for the larger uncertainty in estimating the false negative rate for *Unknown*, this group is experiencing substantially larger false negative rate than other groups and thus experiences the harm of allocation." - ] + ], + "metadata": { + "id": "SAMhqvycR7eE" + } }, { "cell_type": "markdown", - "metadata": { - "id": "8ZqVGZkam1eH" - }, "source": [ "\n", "\n", "---\n" - ] + ], + "metadata": { + "id": "8ZqVGZkam1eH" + } }, { "cell_type": "markdown", - "metadata": { - "id": "-dgITdRiD7Yu" - }, "source": [ "# **Mitigating fairness-related harms in ML models**" - ] + ], + "metadata": { + "id": "-dgITdRiD7Yu" + } }, { "cell_type": "markdown", - "metadata": { - "id": "sbUSG1jVA06G" - }, "source": [ "We have found that the logistic regression predictor leads to a large difference in false negative rates between the groups. We next look at **algorithmic mitigation strategies** of this fairness issue (and similar ones).\n", "\n", @@ -2869,31 +2866,31 @@ "* **Postprocessing**: The output of a trained model is transformed to mitigate fairness issues; for example, the predicted probability of readmission is thresholded according to a group-specific threshold.\n", "\n", "We will now dive into two algorithms: a postprocessing approach and a reductions approach (which is a training-time algorithm). Both of them are in fact **meta-algorithms** in the sense that they act as wrappers around *any* standard (fairness-unaware) machine learning algorithms. This makes them quite versatile in practice.\n" - ] + ], + "metadata": { + "id": "sbUSG1jVA06G" + } }, { "cell_type": "markdown", - "metadata": { - "id": "rX8QycCL0mJj" - }, "source": [ "## Postprocessing with `ThresholdOptimizer`" - ] + ], + "metadata": { + "id": "rX8QycCL0mJj" + } }, { "cell_type": "markdown", - "metadata": { - "id": "fRZfSFzcFaXP" - }, "source": [ "**Postprocessing** techniques are a class of unfairness-mitigation algorithms that take an already trained model and a dataset as an input and seek to fit a transformation function to model's outputs to satisfy some (group) fairness constraint(s). They might be the only feasible unfairness mitigation approach when developers cannot influence training of the model, due to practical reasons or due to security or privacy.\n" - ] + ], + "metadata": { + "id": "fRZfSFzcFaXP" + } }, { "cell_type": "markdown", - "metadata": { - "id": "6PgzZkK9Wbni" - }, "source": [ "Here we use the `ThresholdOptimizer` algorithm from Fairlearn, which follows the approach of [Hardt, Price, and Srebro (2016)](https://arxiv.org/abs/1610.02413).\n", "\n", @@ -2901,13 +2898,13 @@ "\n", "The constraint **false negative rate parity** requires that all the groups have equal values of false negative rate.\n", "\n" - ] + ], + "metadata": { + "id": "6PgzZkK9Wbni" + } }, { "cell_type": "markdown", - "metadata": { - "id": "OFOovaN7AwDr" - }, "source": [ "To instatiate our `ThresholdOptimizer`, we pass in:\n", "\n", @@ -2915,38 +2912,45 @@ "* The fairness `constraints` we want to satisfy.\n", "* The `objective` metric we want to maximize.\n", "\n" - ] + ], + "metadata": { + "id": "OFOovaN7AwDr" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "8je0grKPWHhy" - }, - "outputs": [], "source": [ - "# Now we instantite ThresholdOptimizer with the logistic regression estimator\n", - "postprocess_est = ThresholdOptimizer(\n", - " estimator=unmitigated_pipeline,\n", - " constraints=\"false_negative_rate_parity\",\n", - " objective=\"balanced_accuracy_score\",\n", - " prefit=True,\n", - " predict_method='predict_proba'\n", + "# Now we instantite ThresholdOptimizer with the logistic regression estimator\r\n", + "postprocess_est = ThresholdOptimizer(\r\n", + " estimator=unmitigated_pipeline,\r\n", + " constraints=\"false_negative_rate_parity\",\r\n", + " objective=\"balanced_accuracy_score\",\r\n", + " prefit=True,\r\n", + " predict_method='predict_proba'\r\n", ")" - ] + ], + "outputs": [], + "metadata": { + "id": "8je0grKPWHhy" + } }, { "cell_type": "markdown", - "metadata": { - "id": "VDD86L7eCSe0" - }, "source": [ "In order to use the `ThresholdOptimizer`, we need access to the sensitive features **both during training time and once it's deployed**." - ] + ], + "metadata": { + "id": "VDD86L7eCSe0" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "postprocess_est.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2954,43 +2958,46 @@ }, "id": "VCHJBB7x1rAK", "outputId": "dfc6338e-2be0-4007-bde2-fd8544cc4a89" - }, - "outputs": [], - "source": [ - "postprocess_est.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "YscNZsYU1rCY" - }, - "outputs": [], "source": [ - "# Record and evaluate the output of the trained ThresholdOptimizer on test data\n", - "\n", - "Y_pred_postprocess = postprocess_est.predict(X_test, sensitive_features=A_test)\n", - "metricframe_postprocess = MetricFrame(\n", - " metrics=metrics_dict,\n", - " y_true=Y_test,\n", - " y_pred=Y_pred_postprocess,\n", - " sensitive_features=A_test\n", + "# Record and evaluate the output of the trained ThresholdOptimizer on test data\r\n", + "\r\n", + "Y_pred_postprocess = postprocess_est.predict(X_test, sensitive_features=A_test)\r\n", + "metricframe_postprocess = MetricFrame(\r\n", + " metrics=metrics_dict,\r\n", + " y_true=Y_test,\r\n", + " y_pred=Y_pred_postprocess,\r\n", + " sensitive_features=A_test\r\n", ")" - ] + ], + "outputs": [], + "metadata": { + "id": "YscNZsYU1rCY" + } }, { "cell_type": "markdown", + "source": [ + "We can now inspect how the metric values differ between the postprocessed model and the unmitigated model:" + ], "metadata": { "id": "_izbGv6tQ1KD" - }, - "source": [ - "We can now inspect how the metric values differ between the postprocessed model and the unmitigated model:" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "pd.concat([metricframe_unmitigated.by_group,\r\n", + " metricframe_postprocess.by_group],\r\n", + " keys=['Unmitigated', 'ThresholdOptimizer'],\r\n", + " axis=1)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -2998,27 +3005,27 @@ }, "id": "-9mtWyWc1rH5", "outputId": "982dcb16-b82f-42bc-addb-1caaf2d0aa09" - }, - "outputs": [], - "source": [ - "pd.concat([metricframe_unmitigated.by_group,\n", - " metricframe_postprocess.by_group],\n", - " keys=['Unmitigated', 'ThresholdOptimizer'],\n", - " axis=1)" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "mzPCUFsXPU_S" - }, "source": [ "We next zoom in on differences between the largest and the smallest metric values:" - ] + ], + "metadata": { + "id": "mzPCUFsXPU_S" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "pd.concat([metricframe_unmitigated.difference(),\r\n", + " metricframe_postprocess.difference()],\r\n", + " keys=['Unmitigated: difference', 'ThresholdOptimizer: difference'],\r\n", + " axis=1).T" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -3026,36 +3033,34 @@ }, "id": "dsC4v8Ap1rKt", "outputId": "ebf489b6-07ac-45d5-ae2b-848d6488dbb4" - }, - "outputs": [], - "source": [ - "pd.concat([metricframe_unmitigated.difference(),\n", - " metricframe_postprocess.difference()],\n", - " keys=['Unmitigated: difference', 'ThresholdOptimizer: difference'],\n", - " axis=1).T" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "Hhi_RSxSRoyg" - }, "source": [ "As we see, `ThresholdOptimizer` was able to substantiallydecrease the difference between the values of false negative rate." - ] + ], + "metadata": { + "id": "Hhi_RSxSRoyg" + } }, { "cell_type": "markdown", - "metadata": { - "id": "GarQvopkVN2S" - }, "source": [ "Finally, we save the disagregated statistics:" - ] + ], + "metadata": { + "id": "GarQvopkVN2S" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "metricframe_postprocess.by_group.plot.bar(subplots=True, layout=[1,3], figsize=(12, 4), legend=False, rot=-45, position=1.5)\r\n", + "postprocess_performance = figure_to_base64str(plt)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -3063,36 +3068,44 @@ }, "id": "EsTehBH2SW7f", "outputId": "3cf05d43-1389-41a2-e2dc-7e7511833c2b" - }, - "outputs": [], - "source": [ - "metricframe_postprocess.by_group.plot.bar(subplots=True, layout=[1,3], figsize=(12, 4), legend=False, rot=-45, position=1.5)\n", - "postprocess_performance = figure_to_base64str(plt)" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "umd3slsDmk0d" - }, "source": [ "Next optional section shows that `ThresholdOptimizer` more closely satisfies constraints on the training data than on the test data.\n", "\n", "### Postprocessing: Correctness check [OPTIONAL SECTION]" - ] + ], + "metadata": { + "id": "umd3slsDmk0d" + } }, { "cell_type": "markdown", - "metadata": { - "id": "Cs1kWzda8f0z" - }, "source": [ "We can verify that `ThresholdOptimizer` achieves false negative rate parity on the training dataset, meaning that the values of the false negative rate parity with respect to all groups are close on the training data." - ] + ], + "metadata": { + "id": "Cs1kWzda8f0z" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Record and evaluate the output of the ThresholdOptimizer on the training data\r\n", + "\r\n", + "Y_pred_postprocess_training = postprocess_est.predict(X_train_bal, sensitive_features=A_train_bal)\r\n", + "metricframe_postprocess_training = MetricFrame(\r\n", + " metrics=metrics_dict,\r\n", + " y_true=Y_train_bal,\r\n", + " y_pred=Y_pred_postprocess_training,\r\n", + " sensitive_features=A_train_bal\r\n", + ")\r\n", + "metricframe_postprocess_training.by_group" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -3100,194 +3113,186 @@ }, "id": "ttOZAVbgmplf", "outputId": "c6bb375e-5e32-4508-cfd8-45f775618519" - }, - "outputs": [], - "source": [ - "# Record and evaluate the output of the ThresholdOptimizer on the training data\n", - "\n", - "Y_pred_postprocess_training = postprocess_est.predict(X_train_bal, sensitive_features=A_train_bal)\n", - "metricframe_postprocess_training = MetricFrame(\n", - " metrics=metrics_dict,\n", - " y_true=Y_train_bal,\n", - " y_pred=Y_pred_postprocess_training,\n", - " sensitive_features=A_train_bal\n", - ")\n", - "metricframe_postprocess_training.by_group" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Evaluate the difference between the largest and smallest value of each metric\r\n", + "metricframe_postprocess_training.difference()" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Tty0cmJomp1h", "outputId": "1d821a8d-3e0a-4082-8efe-740d09ceb1e0" - }, - "outputs": [], - "source": [ - "# Evaluate the difference between the largest and smallest value of each metric\n", - "metricframe_postprocess_training.difference()" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "Kvdn7wBhCqXw" - }, "source": [ "The value of `false_negative_rate_difference` on the training data is smaller than on the test data." - ] + ], + "metadata": { + "id": "Kvdn7wBhCqXw" + } }, { "cell_type": "markdown", - "metadata": { - "id": "z2vLNUnK_P66" - }, "source": [ "\n", "### Exercise: ThresholdOptimizer" - ] + ], + "metadata": { + "id": "z2vLNUnK_P66" + } }, { "cell_type": "markdown", - "metadata": { - "id": "0YUembo02yQ8" - }, "source": [ "In this exercise, we will create a `ThresholdOptimizer` by constraining the *true positive rate* (also known as the *recall score*). For any model, the *true positive rate* + *false negative rate* = 1. \n", "\n", "By trying to achieve the *true positive rate parity*, we should produce a `ThresholdOptimizer` with the same performance as our original `ThresholdOptimizer`.\n", "\n" - ] + ], + "metadata": { + "id": "0YUembo02yQ8" + } }, { "cell_type": "markdown", - "metadata": { - "id": "ZJxE2eMuNF3_" - }, "source": [ "#### 1.) Create a new ThresholdOptimizer with the constraint `true_positive_rate_parity` and objective `balanced_accuracy_score`." - ] + ], + "metadata": { + "id": "ZJxE2eMuNF3_" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "wmF8QGQ62tpF" - }, - "outputs": [], "source": [ - "# Instatitate ThresholdOptimizer\n", - "thresopt_exercise = ThresholdOptimizer(\n", - " estimator=unmitigated_pipeline,\n", - " constraints=\"true_positive_rate_parity\",\n", - " objective=\"balanced_accuracy_score\",\n", - " prefit=True,\n", - " predict_method='predict_proba'\n", + "# Instatitate ThresholdOptimizer\r\n", + "thresopt_exercise = ThresholdOptimizer(\r\n", + " estimator=unmitigated_pipeline,\r\n", + " constraints=\"true_positive_rate_parity\",\r\n", + " objective=\"balanced_accuracy_score\",\r\n", + " prefit=True,\r\n", + " predict_method='predict_proba'\r\n", ")" - ] + ], + "outputs": [], + "metadata": { + "id": "wmF8QGQ62tpF" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Fit to data and predict on test data\r\n", + "thresopt_exercise.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)\r\n", + "threshopt_pred = thresopt_exercise.predict(X_test, sensitive_features=A_test)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "JIuirDkd2ttM", "outputId": "44873f39-a5a5-4c83-d195-db0de81113ad" - }, - "outputs": [], - "source": [ - "# Fit to data and predict on test data\n", - "thresopt_exercise.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)\n", - "threshopt_pred = thresopt_exercise.predict(X_test, sensitive_features=A_test)" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "zD5kQu6gqyl6" - }, - "outputs": [], "source": [ - "#thresopt_exercise = ThresholdOptimizer(\n", - "# estimator=______________,\n", - "# constraints=____________,\n", - "# objective=\"balanced_accuracy_score\",\n", - "# prefit=True,\n", - "# predict_method='predict_proba'\n", + "#thresopt_exercise = ThresholdOptimizer(\r\n", + "# estimator=______________,\r\n", + "# constraints=____________,\r\n", + "# objective=\"balanced_accuracy_score\",\r\n", + "# prefit=True,\r\n", + "# predict_method='predict_proba'\r\n", "#)" - ] + ], + "outputs": [], + "metadata": { + "id": "zD5kQu6gqyl6" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "CxdAikCKqyuW" - }, - "outputs": [], "source": [ - "#thresopt_exercise.____(X_train_bal, Y_train_bal, sensitive_features=_______)\n", + "#thresopt_exercise.____(X_train_bal, Y_train_bal, sensitive_features=_______)\r\n", "#threshopt_pred = thresopt_exercise._________(X_test, sensitive_features=_______)" - ] + ], + "outputs": [], + "metadata": { + "id": "CxdAikCKqyuW" + } }, { "cell_type": "markdown", - "metadata": { - "id": "xBLD77OENFN-" - }, "source": [ "#### 2.) Create a new `MetricFrame` object to process the results of this classifier." - ] + ], + "metadata": { + "id": "xBLD77OENFN-" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "RmSwK3P42tzz" - }, - "outputs": [], "source": [ - "thresop_metricframe = MetricFrame(\n", - " metrics=metrics_dict,\n", - " y_true=Y_test,\n", - " y_pred=threshopt_pred,\n", - " sensitive_features=A_test\n", + "thresop_metricframe = MetricFrame(\r\n", + " metrics=metrics_dict,\r\n", + " y_true=Y_test,\r\n", + " y_pred=threshopt_pred,\r\n", + " sensitive_features=A_test\r\n", ")" - ] + ], + "outputs": [], + "metadata": { + "id": "RmSwK3P42tzz" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "B12UNsZErImo" - }, - "outputs": [], "source": [ - "#thresop_metricframe = MetricFrame(\n", - "# metrics=metrics_dict,\n", - "# y_true=Y_test,\n", - "# y_pred=____________,\n", - "# sensitive_features=_______\n", + "#thresop_metricframe = MetricFrame(\r\n", + "# metrics=metrics_dict,\r\n", + "# y_true=Y_test,\r\n", + "# y_pred=____________,\r\n", + "# sensitive_features=_______\r\n", "#)" - ] + ], + "outputs": [], + "metadata": { + "id": "B12UNsZErImo" + } }, { "cell_type": "markdown", - "metadata": { - "id": "DEyEie-9NEew" - }, "source": [ "#### 3.) Compare the performance of the two `ThresholdOptimizers`." - ] + ], + "metadata": { + "id": "DEyEie-9NEew" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Visualize the performance of the new ThresholdOptimizer\r\n", + "thresop_metricframe.by_group" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -3295,16 +3300,16 @@ }, "id": "2P_w3ltB2uAO", "outputId": "015bf2e9-cf56-49da-dffc-9fa95abcb587" - }, - "outputs": [], - "source": [ - "# Visualize the performance of the new ThresholdOptimizer\n", - "thresop_metricframe.by_group" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Compare the performance to the original ThresholdOptimizer\r\n", + "metricframe_postprocess.by_group" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -3312,51 +3317,43 @@ }, "id": "R2phiinPLXwQ", "outputId": "6c1a1037-b535-4cdd-a7d0-b1f6b410e072" - }, - "outputs": [], - "source": [ - "# Compare the performance to the original ThresholdOptimizer\n", - "metricframe_postprocess.by_group" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "uz9mS66YrWka" - }, - "outputs": [], "source": [ "#thresop_metricframe._______" - ] + ], + "outputs": [], + "metadata": { + "id": "uz9mS66YrWka" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "oByEABVGrXWo" - }, - "outputs": [], "source": [ "#metricframe_postprocess.______" - ] + ], + "outputs": [], + "metadata": { + "id": "oByEABVGrXWo" + } }, { "cell_type": "markdown", - "metadata": { - "id": "wAUI3LdQbBCs" - }, "source": [ "Similar to many unfairness mitigation approaches, `ThresholdOptimizer` produces randomized classifiers. Next optional section presents a heuristic strategy for converting a randomized `ThresholdOptimizer` into a deterministic one. In our scenario, this heursitic is quite effective and the resulting deterministic classifier has similar performance as the original `ThresholdOptimizer`.\n", "\n", "### Deployment considerations: Randomized predictions [OPTIONAL SECTION]" - ] + ], + "metadata": { + "id": "wAUI3LdQbBCs" + } }, { "cell_type": "markdown", - "metadata": { - "id": "wlfeHwVC6dna" - }, "source": [ "When we were describing `ThresholdOptimizer` we said that it picks a separate threshold for each group. However, that is not quite correct. In fact,`ThresholdOptimizer`, for each group, picks two thresholds that are close to each other (say `threshold0` and `threshold1`) and then, at deployment time, randomizes between the two: choosing `threshold0` with some probability `p0` and `threshold1` with the remaining probability `p1=1-p0` (the specific probabilities are determined during training; for certain kinds of constraints, three thresholds are considered.)\n", "\n", @@ -3364,22 +3361,30 @@ "\n", "One derandomization heuristic is to replace the two thresholds by their weighted average, i.e., `threshold = p0*threshold0 + p1*threshold1`. That corresponds to the assumption that the values of the scores between the two thresholds are approximately uniformly distributed. Using this heuristic, we derandomize `ThresholdOptimizer`.\n", "\n" - ] + ], + "metadata": { + "id": "wlfeHwVC6dna" + } }, { "cell_type": "markdown", - "metadata": { - "id": "nia26AD2BBKf" - }, "source": [ "The randomized model of the `ThresholdOptimizer` is stored as the field\n", "`interpolated_thresholder_` in the fitted ThresholdOptimizer, which is itself a\n", "valid estimator of type `InterpolatedThresholder`:" - ] + ], + "metadata": { + "id": "nia26AD2BBKf" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "interpolated = postprocess_est.interpolated_thresholder_\n", + "interpolated" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -3387,29 +3392,20 @@ }, "id": "5w-c22tsZsvj", "outputId": "48c9f6ef-9457-41d2-afe4-78965966e470" - }, - "outputs": [], - "source": [ - "interpolated = postprocess_est.interpolated_thresholder_\n", - "interpolated" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "8PkqiznTuL0l" - }, "source": [ "The `interpolation_dict` is a dictionary which assign to each sensitive feature value two thresholds and two respective probabilities. Using our derandomization strategy, we can create a dictionary that represents a deterministic rule:\n" - ] + ], + "metadata": { + "id": "8PkqiznTuL0l" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "F3a1Mw8koLUa" - }, - "outputs": [], "source": [ "def create_deterministic(interpolate_dict):\n", " \"\"\"\n", @@ -3429,50 +3425,58 @@ " operation1=ThresholdOperation(operator=\">\",threshold=(p0*op0 + p1*op1))\n", " )\n", " return deterministic_dict" - ] + ], + "outputs": [], + "metadata": { + "id": "F3a1Mw8koLUa" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "deterministic_dict = create_deterministic(interpolated.interpolation_dict)\n", + "deterministic_dict" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "lbynyyy2CKxv", "outputId": "09ac99a7-5338-4e31-9e02-eb4acbb9d5ec" - }, - "outputs": [], - "source": [ - "deterministic_dict = create_deterministic(interpolated.interpolation_dict)\n", - "deterministic_dict" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "euqycBW_3EB2" - }, "source": [ "Now, we can create an `InterpolatedThresholder` that uses the same pre-fit estimator, but with derandomized thresholds." - ] + ], + "metadata": { + "id": "euqycBW_3EB2" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "R3qKiaMiPOaG" - }, - "outputs": [], "source": [ "deterministic_thresholder = InterpolatedThresholder(estimator=interpolated.estimator,\n", " interpolation_dict=deterministic_dict,\n", " prefit=True,\n", " predict_method='predict_proba')" - ] + ], + "outputs": [], + "metadata": { + "id": "R3qKiaMiPOaG" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "deterministic_thresholder.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -3480,30 +3484,22 @@ }, "id": "XLNNadraaFMT", "outputId": "b3a7edc0-dfe1-4746-f208-0526ad4019a9" - }, - "outputs": [], - "source": [ - "deterministic_thresholder.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "TPRwWuuWY2Yy" - }, - "outputs": [], "source": [ "y_pred_postprocess_deterministic = deterministic_thresholder.predict(X_test, sensitive_features=A_test)" - ] + ], + "outputs": [], + "metadata": { + "id": "TPRwWuuWY2Yy" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "q3834uLWeXWO" - }, - "outputs": [], "source": [ "mf_deterministic = MetricFrame(\n", " metrics=metrics_dict,\n", @@ -3511,20 +3507,28 @@ " y_pred=y_pred_postprocess_deterministic,\n", " sensitive_features=A_test\n", ")" - ] + ], + "outputs": [], + "metadata": { + "id": "q3834uLWeXWO" + } }, { "cell_type": "markdown", - "metadata": { - "id": "2tnGhMYzDloo" - }, "source": [ "Now compare the two models in terms of their disaggregated metrics:" - ] + ], + "metadata": { + "id": "2tnGhMYzDloo" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "mf_deterministic.by_group" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -3532,15 +3536,15 @@ }, "id": "NIkoljGTeXZ0", "outputId": "353905d4-3954-421a-9bf1-adbd6a8febd1" - }, - "outputs": [], - "source": [ - "mf_deterministic.by_group" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "metricframe_postprocess.by_group" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -3548,35 +3552,28 @@ }, "id": "mVimPa2nlFpv", "outputId": "7c197ff9-4eed-4e1d-d00d-6b16ddd34bb6" - }, - "outputs": [], - "source": [ - "metricframe_postprocess.by_group" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "JOi4dwLKECHS" - }, "source": [ "The differences are generally small except for the *Unknown* group, whose false negative rate goes down and balanced accuracy goes up." - ] + ], + "metadata": { + "id": "JOi4dwLKECHS" + } }, { "cell_type": "markdown", - "metadata": { - "id": "NU_rncBQ0lab" - }, "source": [ "## Reductions approach with `ExponentiatedGradient`" - ] + ], + "metadata": { + "id": "NU_rncBQ0lab" + } }, { "cell_type": "markdown", - "metadata": { - "id": "bpyBnqNLZ-w_" - }, "source": [ "With the `ThresholdOptimizer`, we took a fairness-unaware model and transformed the model's decision boundary to satisfy our fairness constraints. One limitation of `ThresholdOptimizer` is that it needs access to the sensitive features at deployment time.\n", "\n", @@ -3589,13 +3586,13 @@ "retraining process is guaranteed to find a model that satisfies the fairness constraints while optimizing the performance metric.\n", "\n", "The model returned by `ExponentiatedGradient` consists of several inner models, returned by the wrapped estimator. At deployment time, `ExponentiatedGradient` randomizes among these models according to a specific probability weights." - ] + ], + "metadata": { + "id": "bpyBnqNLZ-w_" + } }, { "cell_type": "markdown", - "metadata": { - "id": "ZvT_qeduHCn8" - }, "source": [ "To instantiate an `ExponentiatedGradient` model, we pass in two parameters:\n", "\n", @@ -3603,34 +3600,42 @@ "* Fairness `constraints` (an object of type `Moment`).\n", "\n", "The constraints supported by `ExponentiatedGradient` are more general than those supported by `ThresholdOptimizer`. For example, rather than requiring that false negative rates be equal, it is possible to specify the maxium allowed difference or ratio between the largest and the smallest value.\n" - ] + ], + "metadata": { + "id": "ZvT_qeduHCn8" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "ToZdrYen0tJZ" - }, - "outputs": [], "source": [ "expgrad_est = ExponentiatedGradient(\n", " estimator=LogisticRegression(max_iter=1000, random_state=random_seed),\n", " constraints=TruePositiveRateParity(difference_bound=0.02)\n", ")" - ] + ], + "outputs": [], + "metadata": { + "id": "ToZdrYen0tJZ" + } }, { "cell_type": "markdown", - "metadata": { - "id": "bLFk3SCxrskU" - }, "source": [ "The constraints above are expressed for the true positive parity, they require that the difference between the largest and the smallest true positive rate (TPR) across all groups be at most 0.02. Since false negative rate (FNR) is equal to 1-TPR, this is equivalent to requiring that the difference between the largest and smallest FNR be at most 0.02." - ] + ], + "metadata": { + "id": "bLFk3SCxrskU" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Fit the exponentiated gradient model\n", + "expgrad_est.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -3638,34 +3643,20 @@ }, "id": "UFSF5Wn-3M-H", "outputId": "43f639cf-58ef-4425-b772-3daec9d9ff3d" - }, - "outputs": [], - "source": [ - "# Fit the exponentiated gradient model\n", - "expgrad_est.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "VsCeOlKFDQZZ" - }, "source": [ "Similarly to `ThresholdOptimizer` the predictions of `ExponentiatedGradient` models are randomized. If we want to assure reproducible results, we can pass `random_state` to the `predict` function. " - ] + ], + "metadata": { + "id": "VsCeOlKFDQZZ" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 203 - }, - "id": "YYz7GAqf4cbp", - "outputId": "72957bdb-41c5-41c6-b9c5-7f0ba8fdf610" - }, - "outputs": [], "source": [ "# Record and evaluate predictions on test data\n", "\n", @@ -3677,78 +3668,83 @@ " sensitive_features=A_test\n", ")\n", "metricframe_reductions.by_group" - ] + ], + "outputs": [], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 203 + }, + "id": "YYz7GAqf4cbp", + "outputId": "72957bdb-41c5-41c6-b9c5-7f0ba8fdf610" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Evaluate the difference between the largest and smallest value of each metric\n", + "metricframe_reductions.difference()" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "idYvm9lq4mh3", "outputId": "0848b976-8651-45ae-ddb3-b906b9417ac5" - }, - "outputs": [], - "source": [ - "# Evaluate the difference between the largest and smallest value of each metric\n", - "metricframe_reductions.difference()" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "qpxYOozouVx9" - }, "source": [ "While there is a decrease in the false negative rate difference from the unmitigated model, this decrease is not as substantial as with `ThresholdOptimizer`. Note, however, that `ThresholdOptimizer` was able to use the sensitive feature (i.e., race) at deployment time." - ] + ], + "metadata": { + "id": "qpxYOozouVx9" + } }, { "cell_type": "markdown", - "metadata": { - "id": "IzTeHhWG4nJJ" - }, "source": [ "### Explore individual predictors" - ] + ], + "metadata": { + "id": "IzTeHhWG4nJJ" + } }, { "cell_type": "markdown", - "metadata": { - "id": "o7qCmHeYKWGp" - }, "source": [ "During the training process, the `ExponentiatedGradient` algorithm iteratively trains multiple inner models on a reweighted training dataset. The algorithm stores each of these predictors and then randomizes among them at deployment time.\n", "\n", "In many applications, the randomization is undesirable, and also using multiple inner models can pose issues for interpretability. However, the inner models that `ExponentiatedGradient` relies on span a variety of fairness-accuracy trade-offs, and they could be considered for stand-alone deployment: addressing the randomization and interpretability issues, while possibly offering additional flexibility thanks to a variety of trade-offs. \n", "\n", "In this section explore the performance of the individual predictors learned by the `ExponentiatedGradient` algorithm. First, note that since the base estimator was `LogisticRegression` all these predictors are different logistic regression models:" - ] + ], + "metadata": { + "id": "o7qCmHeYKWGp" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "predictors = expgrad_est.predictors_\n", + "predictors" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "II_YtUZg3Ue4", "outputId": "76948908-4240-4d6b-d1db-38cbeda7d5be" - }, - "outputs": [], - "source": [ - "predictors = expgrad_est.predictors_\n", - "predictors" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "K3Bh2IVm4Ynj" - }, - "outputs": [], "source": [ "# Collect predictions by all predictors and calculate balanced error\n", "# as well as the false negative difference for all of them\n", @@ -3756,20 +3752,15 @@ "sweep_preds = [clf.predict(X_test) for clf in predictors]\n", "balanced_error_sweep = [1-balanced_accuracy_score(Y_test, Y_sweep) for Y_sweep in sweep_preds]\n", "fnr_diff_sweep = [false_negative_rate_difference(Y_test, Y_sweep, sensitive_features=A_test) for Y_sweep in sweep_preds]" - ] + ], + "outputs": [], + "metadata": { + "id": "K3Bh2IVm4Ynj" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 260 - }, - "id": "r9TKGsNY8Myu", - "outputId": "06146e22-ea39-4d70-9488-0dd9b03eddca" - }, - "outputs": [], "source": [ "# Show the balanced error / fnr difference values of all predictors on a raster plot \n", "\n", @@ -3797,56 +3788,70 @@ "plt.ylabel(\"False Negative Rate Difference\")\n", "plt.legend(bbox_to_anchor=(1.9,1))\n", "plt.show()" - ] + ], + "outputs": [], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 260 + }, + "id": "r9TKGsNY8Myu", + "outputId": "06146e22-ea39-4d70-9488-0dd9b03eddca" + } }, { "cell_type": "markdown", - "metadata": { - "id": "1MozjJKkZqz_" - }, "source": [ "\n", "### Exercise: Train an `ExponentiatedGradient` model" - ] + ], + "metadata": { + "id": "1MozjJKkZqz_" + } }, { "cell_type": "markdown", - "metadata": { - "id": "lLeGnB4juJsM" - }, "source": [ "In this section, we will explore how changing the base model for the `ExponentiatedGradient` affects the overall performance of the classifier. \n", "\n", "We will instatiate a new `ExponentiatedGradient` classifier with a base `HistGradientBoostingClassifer` estimator. We will use the same `difference_bound` as above." - ] + ], + "metadata": { + "id": "lLeGnB4juJsM" + } }, { "cell_type": "markdown", - "metadata": { - "id": "ghjfKhtB3Kl9" - }, "source": [ "1.) First, let's create our new `ExponentiatedGradient` instance in the cells below and fit it to the training data." - ] + ], + "metadata": { + "id": "ghjfKhtB3Kl9" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "F5lvfDtGZp6O" - }, - "outputs": [], "source": [ "# Create ExponentiatedGradient instance here\n", "expgrad_exercise = ExponentiatedGradient(\n", " estimator=HistGradientBoostingClassifier(),\n", " constraints=TruePositiveRateParity(difference_bound=0.02)\n", ")" - ] + ], + "outputs": [], + "metadata": { + "id": "F5lvfDtGZp6O" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Fit the new instance to the balanced training dataset\n", + "expgrad_exercise.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -3854,59 +3859,45 @@ }, "id": "TYyadS7DZqFk", "outputId": "991d214f-5922-4c4a-ba98-32460a5d3bdf" - }, - "outputs": [], - "source": [ - "# Fit the new instance to the balanced training dataset\n", - "expgrad_exercise.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "CNjDh4JUAc6i" - }, - "outputs": [], "source": [ "#expgrad_exercise = ExponentiatedGradient(\n", "# estimator=_______,\n", "# constraints=TruePositiveRateParity(difference_bound=____)\n", "#)" - ] + ], + "outputs": [], + "metadata": { + "id": "CNjDh4JUAc6i" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "MFIAEkEBAc9P" - }, - "outputs": [], "source": [ "#expgrad_exercise.fit(________, _________, sensitive_features=________)" - ] + ], + "outputs": [], + "metadata": { + "id": "MFIAEkEBAc9P" + } }, { "cell_type": "markdown", - "metadata": { - "id": "8H_UDlAs3M-D" - }, "source": [ "2.) Now, let's compute the performance of the `ExponentiatedGradient` model and compare it with the performance of `ExponentiatedGradient` model with logistic regression as base estimator" - ] + ], + "metadata": { + "id": "8H_UDlAs3M-D" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 203 - }, - "id": "CEUVENjeZqJF", - "outputId": "eddfbe7f-71b7-4da8-8492-4bc2798e11f0" - }, - "outputs": [], "source": [ "# Save the predictions and report the disagregated metrics\n", "# of the exponantiated gradient model\n", @@ -3918,11 +3909,26 @@ " sensitive_features=A_test\n", ")\n", "mf_expgrad_exercise.by_group" - ] + ], + "outputs": [], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 203 + }, + "id": "CEUVENjeZqJF", + "outputId": "eddfbe7f-71b7-4da8-8492-4bc2798e11f0" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "# Compare with the disaggregated metric values of the\n", + "# exponentiated gradient model based on logistic regression\n", + "metricframe_reductions.by_group" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -3930,21 +3936,11 @@ }, "id": "Q1nPfHpWRwWZ", "outputId": "ea0e57d9-ed9e-4d03-fd5b-d57329fe1d1a" - }, - "outputs": [], - "source": [ - "# Compare with the disaggregated metric values of the\n", - "# exponentiated gradient model based on logistic regression\n", - "metricframe_reductions.by_group" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "_Qkfth4ZZBQV" - }, - "outputs": [], "source": [ "#Y_expgrad_exercise = expgrad_exercise.predict(X_test)\n", "#mf_expgrad_exercise = MetricFrame(\n", @@ -3954,75 +3950,70 @@ "# sensitive_features=________\n", "#)\n", "#mf_expgrad_exercise.______" - ] + ], + "outputs": [], + "metadata": { + "id": "_Qkfth4ZZBQV" + } }, { "cell_type": "markdown", - "metadata": { - "id": "Boo771yFUyJ7" - }, "source": [ "3.) Next, calculate the balanced error rate and false negative rate difference of each of the inner models learned by this new `ExponentiatedGradient` classifier." - ] + ], + "metadata": { + "id": "Boo771yFUyJ7" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "jws7w2z6RWUM" - }, - "outputs": [], "source": [ "# Save the inner predictors of the new model\n", "predictors_exercise = expgrad_exercise.predictors_" - ] + ], + "outputs": [], + "metadata": { + "id": "jws7w2z6RWUM" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "EAtkvVOEZqMb" - }, - "outputs": [], "source": [ "# Compute the balanced error rate and false negative rate difference for each of the predictors on the test data.\n", "balanced_error_exercise = [(1 - balanced_accuracy_score(Y_test, pred.predict(X_test))) for pred in predictors_exercise]\n", "false_neg_exercise = [(false_negative_rate_difference(Y_test, pred.predict(X_test), sensitive_features=A_test)) for pred in predictors_exercise]" - ] + ], + "outputs": [], + "metadata": { + "id": "EAtkvVOEZqMb" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "aVTqrrSRAtGc" - }, - "outputs": [], "source": [ "#balanced_error_exercise = [(1 - ______(Y_test, pred.predict(X_test))) for pred in predictors_exercise]\n", "#false_neg_exercise = [(______(Y_test, pred.predict(X_test), sensitive_features=_____)) for pred in predictors_exercise]" - ] + ], + "outputs": [], + "metadata": { + "id": "aVTqrrSRAtGc" + } }, { "cell_type": "markdown", - "metadata": { - "id": "Aos3gdHjDQEi" - }, "source": [ "4.) Finally, let's plot the performances of these individual inner models. In the below cells, plot the individual inner predictors against the performance of their corresponding exponentiated gradient model as well as the unmitigated logistic regression model, and the `ThresholdOptimizer`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 260 - }, - "id": "IxTjb8CeZqPI", - "outputId": "8e06051c-4629-42f0-93f6-51931da2615b" - }, - "outputs": [], + ], + "metadata": { + "id": "Aos3gdHjDQEi" + } + }, + { + "cell_type": "code", + "execution_count": null, "source": [ "# Plot the individual predictors against the Unmitigated Model and the ThresholdOptimizer\n", "plt.scatter(balanced_error_exercise, false_neg_exercise,\n", @@ -4044,42 +4035,47 @@ "plt.ylabel(\"False Negative Rate Difference\")\n", "plt.legend(bbox_to_anchor=(1.9,1))\n", "plt.show()" - ] + ], + "outputs": [], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 260 + }, + "id": "IxTjb8CeZqPI", + "outputId": "8e06051c-4629-42f0-93f6-51931da2615b" + } }, { "cell_type": "markdown", - "metadata": { - "id": "iyknfgrsW1gi" - }, "source": [ "## Comparing performance of different techniques" - ] + ], + "metadata": { + "id": "iyknfgrsW1gi" + } }, { "cell_type": "markdown", - "metadata": { - "id": "Vee-7c2Tw33O" - }, "source": [ "Now we have covered two different class of techniques for mitigating the fairness-related harms we found in our fairness-unaware model. In this section, we will compare the performance of the models we trained above across our key metrics." - ] + ], + "metadata": { + "id": "Vee-7c2Tw33O" + } }, { "cell_type": "markdown", - "metadata": { - "id": "XEXnEeLl7mgc" - }, "source": [ "#### Model performance - by group" - ] + ], + "metadata": { + "id": "XEXnEeLl7mgc" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "SNyCZxHJuXZV" - }, - "outputs": [], "source": [ "def plot_technique_comparison(mf_dict, metric):\n", " \"\"\"\n", @@ -4091,15 +4087,15 @@ " plt.title(metric)\n", " plt.xticks(rotation=0, ha='center');\n", " plt.legend(bbox_to_anchor=(1.01,1), loc='upper left')" - ] + ], + "outputs": [], + "metadata": { + "id": "SNyCZxHJuXZV" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "9dNb-kI3uHzM" - }, - "outputs": [], "source": [ "test_dict = {\n", " \"Reductions\": metricframe_reductions,\n", @@ -4107,11 +4103,19 @@ " \"Postprocessing\": metricframe_postprocess,\n", " \"Postprocessing (DET)\": mf_deterministic\n", "}" - ] + ], + "outputs": [], + "metadata": { + "id": "9dNb-kI3uHzM" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "plot_technique_comparison(test_dict, \"false_negative_rate\")" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -4119,15 +4123,15 @@ }, "id": "SweXBa-vEWFM", "outputId": "a230ed5d-2665-477d-adba-287c76020032" - }, - "outputs": [], - "source": [ - "plot_technique_comparison(test_dict, \"false_negative_rate\")" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "plot_technique_comparison(test_dict, \"balanced_accuracy\")" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -4135,15 +4139,15 @@ }, "id": "FJOwW9db3wKe", "outputId": "a34b7c03-6bfc-4843-c67e-bc5997ebf86c" - }, - "outputs": [], - "source": [ - "plot_technique_comparison(test_dict, \"balanced_accuracy\")" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "plot_technique_comparison(test_dict, \"selection_rate\")" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -4151,30 +4155,22 @@ }, "id": "dACpOpm67K-m", "outputId": "65a6de47-9ca4-49d8-e97a-16ef1db40c02" - }, - "outputs": [], - "source": [ - "plot_technique_comparison(test_dict, \"selection_rate\")" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "OYkgAggpW11N" - }, "source": [ "\n", "\n", "#### Model performance - overall" - ] + ], + "metadata": { + "id": "OYkgAggpW11N" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "w3Rwe98m6Pv2" - }, - "outputs": [], "source": [ "overall_df = pd.DataFrame.from_dict({\n", " \"Unmitigated\": metricframe_unmitigated.overall,\n", @@ -4182,11 +4178,19 @@ " \"Postprocessing (DET)\": mf_deterministic.overall,\n", " \"Reductions\": metricframe_reductions.overall\n", "})" - ] + ], + "outputs": [], + "metadata": { + "id": "w3Rwe98m6Pv2" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "overall_df.T" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -4194,15 +4198,15 @@ }, "id": "44Q-lYTa6PzX", "outputId": "376ce1db-ead9-4984-83c6-7e59e815a1b2" - }, - "outputs": [], - "source": [ - "overall_df.T" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "overall_df.transpose().plot.bar(subplots=True, layout= [1,3], figsize=(12, 5), legend=False, rot=-45, position=1.5);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -4210,19 +4214,11 @@ }, "id": "yUaAejwa6P3F", "outputId": "c6d84d35-fdde-47be-af88-69881afaa163" - }, - "outputs": [], - "source": [ - "overall_df.transpose().plot.bar(subplots=True, layout= [1,3], figsize=(12, 5), legend=False, rot=-45, position=1.5);" - ] + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "pIGS8f2M2Afu" - }, - "outputs": [], "source": [ "difference_df = pd.DataFrame.from_dict({\n", " \"Unmitigated\": metricframe_unmitigated.difference(),\n", @@ -4231,11 +4227,19 @@ " \"Reductions\": metricframe_reductions.difference()\n", "}\n", ")" - ] + ], + "outputs": [], + "metadata": { + "id": "pIGS8f2M2Afu" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "difference_df.T" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -4243,15 +4247,15 @@ }, "id": "3pBkP7QCDVs6", "outputId": "8bec1174-4a94-4cf6-8cab-634e179c0ea7" - }, - "outputs": [], - "source": [ - "difference_df.T" - ] + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "difference_df.T.plot.bar(subplots=True, layout= [1,3], figsize=(12, 5), legend=False, rot=-45, position=1.5);" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -4259,49 +4263,42 @@ }, "id": "_qXEkByRDkK0", "outputId": "36ac6599-3113-4862-b5a0-7fe90bc71a5b" - }, - "outputs": [], - "source": [ - "difference_df.T.plot.bar(subplots=True, layout= [1,3], figsize=(12, 5), legend=False, rot=-45, position=1.5);" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "7S6CIoJ-W1jH" - }, "source": [ "### Randomized predictions" - ] + ], + "metadata": { + "id": "7S6CIoJ-W1jH" + } }, { "cell_type": "markdown", - "metadata": { - "id": "MzdFLJ9BA15F" - }, "source": [ "Both the `ExponentiatedGradient` and the `ThresholdOptimizer` yield randomized predictions (may return different result given the same instance). Due to legal regulations or other concerns, a practitioner may not be able to deploy a randomized model. To address these restrictions:\n", "\n", "* We created a deterministic predictor based on the randomized thresholds learned by the `ThresholdOptimizer`. This deteministic predictor achieved similar performance as the `ThresholdOptimizer`.\n", "* For the `ExponentiatedGradient` model, we can deploy one of the deterministic inner models rather than the overall `ExponentiatedGradient` model.\n", "\n" - ] + ], + "metadata": { + "id": "MzdFLJ9BA15F" + } }, { "cell_type": "markdown", - "metadata": { - "id": "P9dvkQvKW1lv" - }, "source": [ "### Access to sensitive features\n", "\n" - ] + ], + "metadata": { + "id": "P9dvkQvKW1lv" + } }, { "cell_type": "markdown", - "metadata": { - "id": "L56tawsAW14B" - }, "source": [ "\n", "\n", @@ -4309,71 +4306,71 @@ "* The `ExponentiatedGradient` model requires access to the sensitive features ONLY during training time. \n", "\n", "\n" - ] + ], + "metadata": { + "id": "L56tawsAW14B" + } }, { "cell_type": "markdown", - "metadata": { - "id": "fQOR2R3XVa-U" - }, "source": [ "# Model cards for model reporting" - ] + ], + "metadata": { + "id": "fQOR2R3XVa-U" + } }, { "cell_type": "markdown", - "metadata": { - "id": "MzFzSHrQBAgi" - }, "source": [ "_Note: The Python code in this section works in Google Colab, but it does not work on all local environments that we tested._" - ] + ], + "metadata": { + "id": "MzFzSHrQBAgi" + } }, { "cell_type": "markdown", - "metadata": { - "id": "2RAXJDoyVbBT" - }, "source": [ "[Mitchell et al. (2019)](https://arxiv.org/abs/1810.03993) proposed the *model cards* framework for documenting and reporting model training details and deployment considerations. A _model card_ documents, for example, training and evaluation dataset summaries, ethical considerations, and quantitative performance results.\n", "\n" - ] + ], + "metadata": { + "id": "2RAXJDoyVbBT" + } }, { "cell_type": "markdown", - "metadata": { - "id": "oP_tklbmMoFH" - }, "source": [ "### Fill out the model card [OPTIONAL SECTION]" - ] + ], + "metadata": { + "id": "oP_tklbmMoFH" + } }, { "cell_type": "markdown", - "metadata": { - "id": "QfI9oEwIkS6v" - }, "source": [ "In this section, we will create a model card for one of the models we trained." - ] + ], + "metadata": { + "id": "QfI9oEwIkS6v" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "OViYhOAHPftT" - }, - "outputs": [], "source": [ "mct = ModelCardToolkit()\n", "model_card = mct.scaffold_assets()\n" - ] + ], + "outputs": [], + "metadata": { + "id": "OViYhOAHPftT" + } }, { "cell_type": "markdown", - "metadata": { - "id": "Lnr6aRjIF_t6" - }, "source": [ "The first section of the Model Card is the _model details_ section. In _model details_, we fill in some basic information for our model.\n", "\n", @@ -4383,15 +4380,14 @@ "* _Owners_: Name of individual(s) or group who created the model.\n", "* _References_: Any external links or references\n", "\n" - ] + ], + "metadata": { + "id": "Lnr6aRjIF_t6" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "YKvUW25cPewp" - }, - "outputs": [], "source": [ "model_card.model_details.name = \"Diabetes Re-Admission Risk model\"\n", "model_card.model_details.overview = \"This model predicts whether a patient will be re-admitted into a hospital within 30 days.\"\n", @@ -4405,15 +4401,15 @@ "model_card.model_details.version.name = \"v1.0\"\n", "model_card.model_details.version.date = str(date.today())\n", "model_card.model_details.license = \"MIT License\"\n" - ] + ], + "outputs": [], + "metadata": { + "id": "YKvUW25cPewp" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "avEdkdzAPe2S" - }, - "outputs": [], "source": [ "model_card.considerations.use_cases = [\"High-Risk Patient Care Management\"]\n", "model_card.considerations.users = [\"Medical Professionals\", \"ML Researchers\"]\n", @@ -4428,24 +4424,24 @@ " \"name\": (\"Low sample sizes of certain racial groups could lead to poorer performance on these groups\"),\n", " \"mitigation_strategy\": \"Collect additional data points from more hospitals.\"\n", "}]" - ] + ], + "outputs": [], + "metadata": { + "id": "avEdkdzAPe2S" + } }, { "cell_type": "markdown", - "metadata": { - "id": "orPNYAZFwGVM" - }, "source": [ "The next two sections of the model card are meant to provide the reader with information about the data used to train and evaluate the model. For each of these sections, we provide a brief `description` of the data and then submit a `visualization` of the distribution of labels in the dataset." - ] + ], + "metadata": { + "id": "orPNYAZFwGVM" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "khWJN_vqm35e" - }, - "outputs": [], "source": [ "model_card.model_parameters.data.train.graphics.description = (\n", " f\"{X_train_bal.shape[0]} rows with {X_train_bal.shape[1]} features. \"\n", @@ -4456,15 +4452,15 @@ " {\"name\": \"Sensitive Features\", \"image\": sensitive_train},\n", " {\"name\": \"Target Label\", \"image\": outcome_train} \n", "]" - ] + ], + "outputs": [], + "metadata": { + "id": "khWJN_vqm35e" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "h3fiatJ6nGQf" - }, - "outputs": [], "source": [ "model_card.model_parameters.data.eval.graphics.description = (\n", " f\"{X_test.shape[0]} rows with {X_test.shape[1]} columns\"\n", @@ -4474,24 +4470,24 @@ " {\"name\": \"Sensitive Features\", \"image\": sensitive_test},\n", " {\"name\": \"Target Label\", \"image\": outcome_test}\n", "]" - ] + ], + "outputs": [], + "metadata": { + "id": "h3fiatJ6nGQf" + } }, { "cell_type": "markdown", - "metadata": { - "id": "frGlrXHAyn2C" - }, "source": [ "In the last section, we fill out the `quantitative_analysis` section where we describe the model's performance metrics on the evaluation dataset. In particular, we want to report the model's disagregated performance with respect to our three metrics including false negative rate, which quantifies fairness-related harms." - ] + ], + "metadata": { + "id": "frGlrXHAyn2C" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "1cyHIMlAJqod" - }, - "outputs": [], "source": [ "def metricframe_to_dictionary(mframe, feature_name):\n", " \"\"\"\n", @@ -4501,15 +4497,15 @@ " group_metrics = mframe.by_group[feature_name].reset_index()\n", " group_metrics = group_metrics.melt(id_vars=\"race\", var_name=\"type\", value_vars=feature_name).rename(columns={\"race\":\"slice\"})\n", " return group_metrics.to_dict(orient=\"records\")" - ] + ], + "outputs": [], + "metadata": { + "id": "1cyHIMlAJqod" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "kETWDtN68Gmq" - }, - "outputs": [], "source": [ "model_card.quantitative_analysis.graphics.description = (\n", " f\"These graphs show the models performance on the test dataset for disagregated racial categories.\"\n", @@ -4518,41 +4514,49 @@ "model_card.quantitative_analysis.graphics.collection = [\n", " {\"name\": \"ThresholdOptimizer\", \"image\": postprocess_performance}\n", "]" - ] + ], + "outputs": [], + "metadata": { + "id": "kETWDtN68Gmq" + } }, { "cell_type": "markdown", - "metadata": { - "id": "WvL2mHlwp-l0" - }, "source": [ "Finally, we pass our filled-out `model_card` to the `mct` object to generate an HTML version of the `model_card` that can be rendered within a Jupyter notebook." - ] + ], + "metadata": { + "id": "WvL2mHlwp-l0" + } }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "A-j_4mBaPe5z" - }, - "outputs": [], "source": [ "mct.update_model_card_json(model_card)\n", "html_modelcard = mct.export_format()" - ] + ], + "outputs": [], + "metadata": { + "id": "A-j_4mBaPe5z" + } }, { "cell_type": "markdown", - "metadata": { - "id": "CkuNE_MvMy7P" - }, "source": [ "### Display the model card" - ] + ], + "metadata": { + "id": "CkuNE_MvMy7P" + } }, { "cell_type": "code", "execution_count": null, + "source": [ + "display.HTML(html_modelcard)" + ], + "outputs": [], "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -4560,26 +4564,19 @@ }, "id": "89kqC0Jj9D6O", "outputId": "a64294f5-205a-4dc9-8e8a-fcd245a7182f" - }, - "outputs": [], - "source": [ - "display.HTML(html_modelcard)" - ] + } }, { "cell_type": "markdown", - "metadata": { - "id": "1j9WzWcCVbZV" - }, "source": [ "# Discussion and conclusion" - ] + ], + "metadata": { + "id": "1j9WzWcCVbZV" + } }, { "cell_type": "markdown", - "metadata": { - "id": "xtvW54LtB8hp" - }, "source": [ "In this tutorial we have explored in depth a health care scenario through all stages of the AI lifecycle except the model deployment stage. We have seen how fairness-related harms can arise at the stage of task definition, data collection, model training, and model evaluation. We have also seen how to use a variety of tools and practices, such as datasheets for datasets, Fairlearn, and model cards.\n", "\n", @@ -4590,7 +4587,10 @@ "If you would like to learn more about fairness of AI systems, or to contribute to Fairlearn, we welcome you to join our community. Fairlearn is built and maintained by contributors with a variety of backgrounds and expertise.\n", "\n", "Further resources can also be found [on our website](https://fairlearn.org/main/user_guide/further_resources.html)." - ] + ], + "metadata": { + "id": "xtvW54LtB8hp" + } } ], "metadata": { @@ -4616,5 +4616,5 @@ } }, "nbformat": 4, - "nbformat_minor": 0 -} + "nbformat_minor": 2 +} \ No newline at end of file diff --git a/2021_scipy_tutorial/fairness-in-AI-systems-student.ipynb b/2021_scipy_tutorial/fairness-in-AI-systems-student.ipynb index d6c4813..9ac54af 100644 --- a/2021_scipy_tutorial/fairness-in-AI-systems-student.ipynb +++ b/2021_scipy_tutorial/fairness-in-AI-systems-student.ipynb @@ -1,4382 +1,7054 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "NDxtnoIr7mIN" - }, - "source": [ - "# SciPy 2021 Tutorial
_Fairness in AI systems:
From social context to practice using Fairlearn_\n", - "\n", - "---\n", - "\n", - "_SciPy 2021 Tutorial: Fairness in AI systems: From social context to practice using Fairlearn by Manojit Nandi, Miroslav Dudík, Triveni Gandhi, Lisa Ibañez, Adrin Jalali, Michael Madaio, Hanna Wallach, Hilde Weerts is licensed under\n", - "[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)._\n", - "\n", - "---" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Sch9KDWg7SL8" - }, - "source": [ - "Fairness in AI systems is an interdisciplinary field of research and practice that aims to understand and address some of the negative impacts of AI systems on society. In this tutorial, we will walk through the process of assessing and mitigating fairness-related harms in the context of the U.S. health care system. This tutorial will consist of a mix of instructional content and hands-on demonstrations using Jupyter notebooks. Participants will use the Fairlearn library to assess ML models for performance disparities across different racial groups and mitigate those disparities using a variety of algorithmic techniques." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "fboVvqvVvpKz" - }, - "source": [ - "# **Prepare environment**" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "o0p461hmgrmz" - }, - "source": [ - "## Install packages" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "9b5Nsb2Rgux7" - }, - "source": [ - "Note that the runtime environment needs to be restarted after installing `model-card-toolkit`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - }, - "id": "w_uVNMHUdb2w", - "outputId": "8ef912ea-10cf-47ef-c85d-574a044283e3" - }, - "outputs": [], - "source": [ - "!pip install --upgrade fairlearn==0.7.0\n", - "!pip install --upgrade scikit-learn\n", - "!pip install --upgrade seaborn\n", - "!pip install model-card-toolkit" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "fVtpwFGJLcgU" - }, - "source": [ - "## Import and set up packages" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "cEwLsWyTLgJn" - }, - "outputs": [], - "source": [ - "import numpy as np\n", - "import pandas as pd\n", - "\n", - "pd.set_option(\"display.float_format\", \"{:.3f}\".format)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "wBOSsyc48MK0" - }, - "outputs": [], - "source": [ - "import matplotlib.pyplot as plt\n", - "import seaborn as sns\n", - "\n", - "sns.set()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "VhbFzU6GLgW0" - }, - "outputs": [], - "source": [ - "from sklearn.linear_model import LogisticRegression\n", - "from sklearn.model_selection import train_test_split\n", - "from sklearn.preprocessing import StandardScaler\n", - "from sklearn.pipeline import Pipeline\n", - "from sklearn.utils import Bunch\n", - "from sklearn.metrics import (\n", - " balanced_accuracy_score,\n", - " roc_auc_score,\n", - " accuracy_score,\n", - " recall_score,\n", - " confusion_matrix,\n", - " roc_auc_score,\n", - " roc_curve,\n", - " plot_roc_curve)\n", - "from sklearn import set_config\n", - "\n", - "set_config(display=\"diagram\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "GcilMnWhLgaT" - }, - "outputs": [], - "source": [ - "from fairlearn.metrics import (\n", - " MetricFrame,\n", - " true_positive_rate,\n", - " false_positive_rate,\n", - " false_negative_rate,\n", - " selection_rate,\n", - " count,\n", - " false_negative_rate_difference\n", - ")\n", - "\n", - "from fairlearn.postprocessing import ThresholdOptimizer, plot_threshold_optimizer\n", - "from fairlearn.postprocessing._interpolated_thresholder import InterpolatedThresholder\n", - "from fairlearn.postprocessing._threshold_operation import ThresholdOperation\n", - "from fairlearn.reductions import ExponentiatedGradient, EqualizedOdds, TruePositiveRateParity" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "oMbCFBSZr2A4" - }, - "outputs": [], - "source": [ - "# Model Card Toolkit works in Google Colab, but it does not work on all local environments\n", - "# that we tested. If the import fails, define a dummy function in place of the function\n", - "# for saving figures into images in a model card..\n", - "\n", - "try:\n", - " from model_card_toolkit import ModelCardToolkit\n", - " from model_card_toolkit.utils.graphics import figure_to_base64str\n", - " model_card_imported = True\n", - "except Exception:\n", - " model_card_imported = False\n", - " def figure_to_base64str(*args):\n", - " return None" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "S4ANMj1M8k0h" - }, - "outputs": [], - "source": [ - "from IPython import display\n", - "from datetime import date" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "4OoPTetsDv-v" - }, - "source": [ - "# **Overview of fairness in AI systems**" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "vt9n43o6DPbg" - }, - "source": [ - "Please refer to the slides here: https://raw.githubusercontent.com/fairlearn/talks/main/2021_scipy_tutorial/overview.pdf\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0gZTFumIxCfK" - }, - "source": [ - "# **Introduction of Fairlearn and other tutorial resources**\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "7KhsuCB-imFk" - }, - "source": [ - "This tutorial builds on the following open source projects:\n", - "\n", - "* **machine learning and data processing**: _scikit-learn_, _pandas_, _numpy_\n", - "* **plotting**: _seaborn_, _matplotlib_\n", - "* **AI fairness**: _Fairlearn_, _Model Card Toolkit_" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ORfTAukTuEPN" - }, - "source": [ - "### [Fairlearn](https://fairlearn.org)\n", - "\n", - "Fairlearn is an open-source, community-driven project to help data scientists improve fairness of AI systems. It includes:\n", - "\n", - "* A Python library for fairness assessment and improvement (fairness metrics, mitigation algorithms, plotting, etc.)\n", - "\n", - "* Educational resources covering organizational and technical processes for unfairness mitigation (user guide, case studies, Jupyter notebooks, etc.)\n", - "\n", - "The project was started in 2018 at Microsoft Research. In 2021 it adopted neutral governance structure and since then it is completely community-driven." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "AE3bFABst9ze" - }, - "source": [ - "### [Model Card Toolkit](https://github.com/tensorflow/model-card-toolkit)\n", - "\n", - "The Model Card Toolkit (MCT) streamlines and automates generation of _model cards_, machine learning documents that provide context and transparency into a model's development and performance. It was released by Google in 2020." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "_j1vtg6TD7Fi" - }, - "source": [ - "# **Introduction to the health care scenario**" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "HUkG_zdEylGU" - }, - "source": [ - "Our scenario builds on previous research that highlighted racial disparities in how health care resources are allocated in the U.S. ([Obermeyer et al., 2019](https://science.sciencemag.org/content/366/6464/447.full)).\n", - "Motivated by that work, in this tutorial we consider an automated system for recommending patients for _high-risk care management_ programs, which are described by Obermeyer et al. 2019 as follows:\n", - "\n", - "> These programs seek to improve the care of patients with complex health needs by providing additional resources, including greater attention from trained providers, to help ensure that care is well coordinated. Most health systems use these programs as the cornerstone of population health management efforts, and they are widely considered effective at improving outcomes and satisfaction while reducing costs. [...] Because the programs are themselves expensive—with costs going toward teams of dedicated nurses, extra primary care appointment slots, and other scarce resources—**health systems rely extensively on algorithms to identify patients who will benefit the most.**\n", - "\n", - "**Convenience restriction**\n", - "\n", - "* In practice, the modeling of health needs would use large data sets covering a wide range of diagnoses. In this tutorial, we will work with a [publicly available clinical dataset](https://archive.ics.uci.edu/ml/datasets/Diabetes+130-US+hospitals+for+years+1999-2008) that focuses on _diabetic patients only_ ([Strack et al., 2014](https://www.hindawi.com/journals/bmri/2014/781670/))." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "q-j4KN95wLLS" - }, - "source": [ - "## Dataset and task" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "zOwrRsB7wEeM" - }, - "source": [ - "We will be working with a clincial dataset of hospital re-admissions over a ten-year period (1998-2008) for diabetic patients across 130 different hospitals in the US. Each record represents the hospital admission records for a patient diagnosed with diabetes whose stay lasted one to fourteen days.\n", - "\n", - "The features describing each encounter include demographics, diagnoses, diabetic medications, number of visits in the year preceding the encounter, and payer information, as well as whether the patient was readmitted after release, and whether the readmission occurred within 30 days of the release.\n", - "\n", - "We would like to develop a classification model, which decides whether the patients should be suggested to their primary care physicians for an enrollment into the high-risk care management program. The positive prediction will mean recommendation into the care program.\n", - "\n", - "**Decision point: Task definition**\n", - "\n", - "* A hospital **readmission within 30 days** can be viewed as a proxy that the patients needed more assistance at the release time, so it will be the label we wish to predict.\n", - "\n", - "* Because of the class imbalance, we will be measuring our performance via **balanced accuracy**. Another key performance consideration is how many patients are recommended for care, metric we refer to as **selection rate**.\n", - "\n", - "Ideally, health care professionals would be involved in both designing and using the model, including formalizing the task definition. \n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "2BE26iXWwUqr" - }, - "source": [ - "## Fairness considerations" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "eZUcQVZYyRvz" - }, - "source": [ - "* _Which groups are most likely to be disproportionately negatively affected?_ Previous work suggests that groups with different race and ethnicity can be differently affected.\n", - "\n", - "* _What are the harms?_ The key harms here are allocation harms. In particular, false negatives, i.e., don't recommend somebody who will be readmitted.\n", - "\n", - "* _How should we measure those harms?_\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "2kD6G-yF1Tcf" - }, - "source": [ - "In the remainder of the tutorial we will:\n", - "* First examine the dataset and our choice of label with an eye towards a variety of fairness issues.\n", - "* Then train a logistic regression model and assess its performance as well as fairness.\n", - "* Finally, look at two unfairness mitigation strategies." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "FkTmOAh8Bp1D" - }, - "source": [ - "\n", - "## Discussion: Fairness-related harms\n", - "\n", - "* How can we determine which type of harm is relevant in a particular scenario?\n", - "* What are ways to find out which (groups of) individuals are most likely to be disproportionately negatively affected?\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "R7tpRumX_4Nh" - }, - "source": [ - "# Task definition and dataset characteristics" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "3iJGhfCgPiy-" - }, - "source": [ - "Two critical decisions when desiging an AI system are\n", - "1. how we define the machine learning task\n", - "2. what dataset we use to train our models\n", - "\n", - "These choices are often intertwined, because the dataset is often a convenience dataset, based on availability, which leads to a specific choice of label and performance metric (that's also the case in our scenario).\n", - "\n", - "In this part of the tutorial, we first load the dataset, and then we examine it for a variety of fairness issues:\n", - "1. sample sizes of different demographic groups, and in particular different racial groups\n", - "2. appropriateness of our choice of label (readmission within 30 days)\n", - "3. representativeness/informativeness of different features for different groups\n", - "\n", - "Besides dataset characteristics, one additional aspect of dataset fairness is whether the data was collected in a manner that respects the autonomy of individuals in the dataset.\n", - "\n", - "The dataset characteristics can be systematically documented through the **datasheets** practice. We will touch on this later on. By documenting our understanding of the dataset, we communicate any concerns we have about the data and highlight downstream issues that may arise during the model training, evaluation and deployment.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "3-ABnntZT8Fn" - }, - "source": [ - "## Load the dataset\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "bvyHqcLIT8Fo" - }, - "source": [ - "We next load the dataset and review the meaning of its columns.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "gkfwniFQT8Fp" - }, - "outputs": [], - "source": [ - "df = pd.read_csv(\"https://raw.githubusercontent.com/fairlearn/talks/main/2021_scipy_tutorial/data/diabetic_preprocessed.csv\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 360 - }, - "id": "mIFN96kiT8Fq", - "outputId": "9dbf587e-5f51-4dc4-877a-3ffb9a7374a2" - }, - "outputs": [], - "source": [ - "df.head()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "KXQNItgRT8Fu" - }, - "source": [ - "The columns contain mostly boolean and categorical data (including age and various test results), with just the following exceptions: `time_in_hospital`, `num_lab_procedures`, `num_procedures`, `num_medications`, `number_diagnoses`.\n", - "\n", - "\n", - "|features| description|\n", - "|---|---|\n", - "| race, gender, age | demographic features |\n", - "| medicare, medicaid | insurance information |\n", - "| admission_source_id | emergency, referral, or other |\n", - "| had_emergency, had_inpatient_days,
had_outpatient_days | hospital visits in prior year |\n", - "| medical_specialty | admitting physician's specialty |\n", - "| time_in_hospital, num_lab_procedures,
num_procedures, num_medications,
primary_diagnosis, number_diagnoses,
max_glu_serum, A1Cresult, insulin
change, diabetesMed | description of the hospital visit
|\n", - "| discharge_disposition_id | discharched to home or not |\n", - "| readmitted, readmit_binary,
readmit_30_days | readmission information |\n", - "\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 738 - }, - "id": "gPtaDcfHT8Fv", - "outputId": "42347f1f-d3c8-4221-f98d-fdeaa25ef4fd" - }, - "outputs": [], - "source": [ - "# Show the values of all binary and categorical features\n", - "categorical_values = {}\n", - "for col in df:\n", - " if col not in {'time_in_hospital', 'num_lab_procedures',\n", - " 'num_procedures', 'num_medications', 'number_diagnoses'}:\n", - " categorical_values[col] = pd.Series(df[col].value_counts().index.values)\n", - "categorical_values_df = pd.DataFrame(categorical_values).fillna('')\n", - "categorical_values_df.T" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "8loEiuFSWb8A" - }, - "source": [ - "We mark all categorical features: " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "P4y1FRMdWduE" - }, - "outputs": [], - "source": [ - "categorical_features = [\n", - " \"race\",\n", - " \"gender\",\n", - " \"age\",\n", - " \"discharge_disposition_id\",\n", - " \"admission_source_id\",\n", - " \"medical_specialty\",\n", - " \"primary_diagnosis\",\n", - " \"max_glu_serum\",\n", - " \"A1Cresult\",\n", - " \"insulin\",\n", - " \"change\",\n", - " \"diabetesMed\",\n", - " \"readmitted\"\n", - "]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "Qyo3GQ4RWduG" - }, - "outputs": [], - "source": [ - "for col_name in categorical_features:\n", - " df[col_name] = df[col_name].astype(\"category\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "6LTax67Em4q8" - }, - "source": [ - "## Group sample sizes " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ft28kXKHm4q9" - }, - "source": [ - "From the perspective of fairness assessment, a key data characteristic is the sample size of groups with respect to which we conduct fairness assessment.\n", - "\n", - "Small sample sizes have two implications:\n", - "\n", - "* **assessment**: the impacts of the AI system on smaller groups are harder to assess, because due to fewer data points we have a much larger uncertainty (error bars) in our estimates\n", - "\n", - "* **model training**: fewer training data points mean that our model fails to appropriately capture any data patterns specific to smaller groups, which means that its predictive performance on these groups could be worse\n", - "\n", - "Let's examine the sample sizes of the groups according to `race`:\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "wEGJjCrOm4q9", - "outputId": "a7430662-00d7-405b-b9af-1dcaf8fc5c28" - }, - "outputs": [], - "source": [ - "df[\"race\"].value_counts()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 330 - }, - "id": "HznSDLWEm4q-", - "outputId": "009b8aea-9ae4-4810-eb13-67ac8efae163" - }, - "outputs": [], - "source": [ - "df[\"race\"].value_counts().plot(kind='bar', rot=45);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ZkI6KA3rm4q-" - }, - "source": [ - "Normalized as frequencies:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "qE2UoDU3m4q-", - "outputId": "c23f07b1-493d-4df2-db5a-06bc921eefde" - }, - "outputs": [], - "source": [ - "df[\"race\"].value_counts(normalize=True)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "DEUrg6J3m4q_" - }, - "source": [ - "In our dataset, our patients are predominantly *Caucasian* (75%). The next largest racial group is *AfricanAmerican*, making up 19% of the patients. The remaining race categories (including *Unknown*) compose only 6% of the data." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "xIdDhgtAm4rA" - }, - "source": [ - "We also examine the dataset composition by `gender`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "0h26CH50m4rA", - "outputId": "20d290a6-c8a7-4d21-c676-2a6f552bc3a6" - }, - "outputs": [], - "source": [ - "df[\"gender\"].value_counts() # counts" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "UTbI5v48m4rA", - "outputId": "e69992ee-8b33-4aef-a903-b51eb63094e8" - }, - "outputs": [], - "source": [ - "df[\"gender\"].value_counts(normalize=True) # frequencies" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "i3xBANdpm4rA" - }, - "source": [ - "Gender is in our case effectively binary (and we have no further information how it was operationalized), with both *Female* represented at 54% and *Male* represented at 46%. There are only 3 samples annotated as *Unknown/Invalid*." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "NyLBXpKWm4rA" - }, - "source": [ - "### Decision point: How do we address smaller group sizes?" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "wT-BPQ2Gm4rB" - }, - "source": [ - "When the data set lacks coverage of certain groups, it means that we will not be able to reliably assess any fairness-related issues. There are three interventions (which could be carried out in a combination):\n", - "\n", - "* **collect more data**: collect more data for groups with fewer samples\n", - "* **buckets**: merge some of the groups\n", - "* **drop small groups**\n", - "\n", - "The choice of strategy depends on our existing understanding of which groups are at the greatest risk of a harm. In particular, pooling the groups with widely different risks could mask the extent of harms. We generally caution against dropping small groups as this leads to the representational harm of erasure.\n", - "\n", - "If any groups are merged or dropped, these decisions should be annotated / explained (in the datasheet, which we discuss below).\n", - "\n", - "In our case, we will:\n", - "\n", - "* merge the three smallest race groups *Asian*, *Hispanic*, *Other* (similar to [Strack et al., 2014](https://www.hindawi.com/journals/bmri/2014/781670/)), but also retain the original groups for auxiliary assessments\n", - "\n", - "* drop the gender group *Unknown/Invalid*, because the sample size is so small that no meaningful fairness assessment is possible" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "iBN_fIhpm4rB" - }, - "outputs": [], - "source": [ - "# drop gender group Unknown/Invalid\n", - "df = df.query(\"gender != 'Unknown/Invalid'\")\n", - "\n", - "# retain the original race as race_all, and merge Asian+Hispanic+Other \n", - "df[\"race_all\"] = df[\"race\"]\n", - "df[\"race\"] = df[\"race\"].replace({\"Asian\": \"Other\", \"Hispanic\": \"Other\"})" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "N_Sb8ISAnRQF" - }, - "source": [ - "### Exercise" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ns1Tr9wLm4rB" - }, - "source": [ - "Please examine the distribution of the `age` feature in the dataset." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "MDiLp-cDoWGv" - }, - "source": [ - "### Answer" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "UOn4j1o8m4rB", - "outputId": "577b1636-57f5-4ab1-903a-c6a42f8193ba" - }, - "outputs": [], - "source": [ - "df[\"age\"].value_counts()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 268 - }, - "id": "ns7RxP5um4rB", - "outputId": "81364da1-1f2e-4053-ced1-c93394ba125e" - }, - "outputs": [], - "source": [ - "df[\"age\"].value_counts().plot(kind='bar', rot=0);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "HAXa7MoQnjq4" - }, - "source": [ - "As we might expect, most patients admitted into the hospital in our data set belong to the *Over 60 years* category. Although we will not be assessing for age-based fairness-related harms in this tutorial, we will want to document the age imbalance in our dataset." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "pK0LbF_sylTU" - }, - "source": [ - "## Examining the choice of label" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "AtgLr5Cs_YhF" - }, - "source": [ - "Next we dive into the question of whether our choice of label (readmission within 30 days) aligns with our goal (identify patients that would benefit from the care management program).\n", - "\n", - "A framework particularly suited for this analysis is called _measurement modeling_ (see, e.g., [Jacobs and Wallach, 2021](https://arxiv.org/abs/1912.05511)). The goal of measurement modeling is to describe the relationship between what we care about and what we can measure. The thing that we care about is referred to as the _construct_ and what we can observe is referred to as the _measurement_. In our case:\n", - "* **construct** = greatest benefit from the care management program\n", - "* **measurement** = readmission within 30 days (in the absence of such program)\n", - "\n", - "In our case, the **measurement** coincides with the **classification label**.\n", - "\n", - "The act of _operationalizing_ the construct via a specific measurement corresponds to making certain assumptions. In our case, we are making the following assumption: **the greatest benefit from the care management program would go to patients that are** (in the absence of such a program) **most likely to be readmitted into the hospital within 30 days.**" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "fA19isovvCew" - }, - "source": [ - "### How can we check whether our assumptions apply?" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "oY95_kOFO6Wb" - }, - "source": [ - "In the terminology of measurement modeling, how do we establish _construct validity_? Following, [Jacobs and Wallach, 2021](https://arxiv.org/abs/1912.05511),\n", - "\n", - "> Establishing construct\n", - "validity means demonstrating, in a variety of ways, that the measurements obtained from measurement model are both meaningful\n", - "and useful:\n", - "> * Does the operationalization capture all relevant aspects\n", - "of the construct purported to be measured?\n", - "> * Do the measurements\n", - "look plausible?\n", - "> * Do they correlate with other measurements of the\n", - "same construct? Or do they vary in ways that suggest that the\n", - "operationalization may be inadvertently capturing aspects of other\n", - "constructs?\n", - "> * Are the measurements predictive of measurements of\n", - "any relevant observable properties (and other unobservable theoretical constructs) thought to be related to the construct, but not incorporated into the operationalization?\n", - "\n", - "We focus on one aspect of construct validity, called _predictive validity_, which refers to the extent\n", - "to which the measurements obtained from a measurement model\n", - "are predictive of measurements of any relevant observable properties \n", - "related to the construct purported to be measured, but not incorporated into the operationalization.\n", - "\n", - "The predictions do not need to be chronological, meaning that we do not necessarily need to be predicting future from the past. Also, the predictions do not need to be causal (going from causes to effects). We just need to ensure that the predicted property is not part of the measurement whose validity we're checking. \n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "8IBUG3Sa96LX" - }, - "source": [ - "### Predictive validity" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "rVrLpuwy98uG" - }, - "source": [ - "We would like to show that our measurement `readmit_30_days` is correlated with patient characteristics that are related to our construct \"benefiting from care management\". One such characteristic is the general patient health, where we expect that patients that are less healthy are more likely to benefit from care management.\n", - "\n", - "While our data does not contain full health records that would enable us to holistically measure general patient health, the data does contain two relevant features: `had_emergency` and `had_inpatient_days`, which indicate whether the patient spent any days in the emergency room or in the hospital (but non-emergency) in the preceding year.\n", - "\n", - "To establish predictive validity, we would like to show that our measurement `readmit_30_days` is predictive of these two observable characteristics." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "BQWxJEN-M6VD" - }, - "source": [ - "First, let's check the rate at which the patients with different `readmit_30_days` labels were readmitted in the previous year:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 285 - }, - "id": "3t11OgTgZfV8", - "outputId": "d2467bd3-0148-4373-cd41-0b99d9f54088" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"had_emergency\", x=\"readmit_30_days\",\n", - " data=df, ci=95, join=False);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Ptl-tHkDf_GJ" - }, - "source": [ - "The plot shows that indeed patients with `readmit_30_days=0` have a lower rate of emergency visits in the prior year, whereas patients with `readmit_30_days=1` have a larger rate. (The vertical lines indicate 95% confidence intervals obtained via boostrapping.)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "07GU8IGIY9KC" - }, - "source": [ - "We see a similar pattern when `readmit_30_days` is used to predict the rate of (non-emergency) hospital visits in the previous year:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 285 - }, - "id": "GuPPVpXrO5uE", - "outputId": "dcc740aa-0805-4ef9-9c77-d64fe1b0111a" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"had_inpatient_days\", x=\"readmit_30_days\",\n", - " data=df, join=False);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "wN8NU8QkRMqM" - }, - "source": [ - "Now let's take a look whether the predictiveness is similar across different race groups. First, let's check how well `readmit_30_days` predicts `had_emergency`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 382 - }, - "id": "tgLhUlZzOvHC", - "outputId": "59590db9-f5c7-44a0-85d2-bfcf79eab25e" - }, - "outputs": [], - "source": [ - "# Visualize predictiveness using a categorical pointplot\n", - "sns.catplot(y=\"had_emergency\", x=\"readmit_30_days\", hue=\"race\", data=df,\n", - " kind=\"point\", ci=95, dodge=True, join=False);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "IQ7zH3Wn416g" - }, - "source": [ - "The patients in the group *Unknown* have a substantially lower rate of emergency visits in the prior year, regardless of whether they are readmitted in 30 days. The readmission is still positively correlated with `had_emergency`, but note the large error bars (due to small sample sizes).\n", - "\n", - "We also see that the group with feature value *AfricanAmerican* has a higher rate of emergency visits compared with other groups. However, generally the groups *Caucasian*, *AfricanAmerican* and *Other* follow similar dependence patterns." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "G13abbdS7oM-" - }, - "source": [ - "We see a similar pattern when `readmit_30_days` is used to predict the rate of (non-emergency) hospital visits in the previous year:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 382 - }, - "id": "Kx52yWYNQuSH", - "outputId": "d0acd516-57d5-4910-feaa-f3b1c1176900" - }, - "outputs": [], - "source": [ - "sns.catplot(y=\"had_inpatient_days\", x=\"readmit_30_days\", hue=\"race\", data=df,\n", - " kind=\"point\", ci=95, dodge=True, join=False);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "HrCelj5_hR6a" - }, - "source": [ - "Again, for *Unknown* the rate of (non-emergency) hospital visits in the previous year is lower than for other groups.In all groups there is a strong positive correlation between `readmit_30_days` and `had_inpatient_days`.\n", - "\n", - "In all cases, we see that readmission in 30 days is predictive of our two measurements of general patient health.\n", - "\n", - "The analysis is also surfacing the fact that patients with the value of race *Unknown* have fewer hospital visits in the preceding year (both emergency and non-emergency) than other groups. In practice, this would be a good reason to reach out to health professionals to investigate this patient cohort, to make sure that we understand why there is the systematic difference.\n", - "\n", - "Note that we have only investigated _predictive validity_, but there are other important aspects of construct validity which we may want to establish (see [Jacobs and Wallach, 2021](https://arxiv.org/abs/1912.05511))." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "V_4CZt-UBaOQ" - }, - "source": [ - "\n", - "### Exercise" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "rW9ktRlqBhZA" - }, - "source": [ - "Check the predictive validity with respect to `gender` and `age`. Do you see any differences? Can you form a hypothesis why?" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 382 - }, - "id": "VLom8GFbUfqR", - "outputId": "91418775-b94d-4b3e-dba4-1305ef908a29" - }, - "outputs": [], - "source": [ - "# Check for predictive validity by gender\r\n", - "sns.catplot(y=\"had_inpatient_days\",x=\"readmit_30_days\",hue=_____, data=df,\r\n", - " kind=\"point\", ci=95, dodge=True, join=False);" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 382 - }, - "id": "7PdFAu3lT-pD", - "outputId": "7c3b6008-fdff-46dd-f749-ca0198a5193b" - }, - "outputs": [], - "source": [ - "# Check for predictive validity by age\r\n", - "sns.catplot(y=\"had_inpatient_days\", x=\"readmit_30_days\", hue=_____, data=df,\r\n", - " kind=\"point\", ci=95, dodge=True, join=False);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "3i9KdmTWKUJZ" - }, - "source": [ - "## Label imbalance\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ViGqA5VTGrEo" - }, - "source": [ - "Now that we have established the validity of our label, we will check frequency of its values in our data. The frequency of different labels is an important descriptive characteristic in classification settings for several reasons:\n", - "\n", - "* some classification algorithms and performance measures might not work well with data sets with extreme class imbalance\n", - "* in binary classification settings, our ability to evaluate error is often driven by the size of the smaller of the two classes (again, the smaller the sample the larger the uncertainty in estimates)\n", - "* label imbalance may exacerbate the problems due to smaller group sizes in fairness assessment\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "cos3--59EiZt" - }, - "source": [ - "Let's check how many samples in our data are labeled as positive and how many as negative." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "eR5ULLYGE4UK", - "outputId": "02f2e533-2a7f-44af-e57f-c24c7b8b4d4d" - }, - "outputs": [], - "source": [ - "df[\"readmit_30_days\"].value_counts() # counts" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "0UWBbg4Cz90t", - "outputId": "a0e1e14b-47f9-4f49-da41-ba53a7a63082" - }, - "outputs": [], - "source": [ - "df[\"readmit_30_days\"].value_counts(normalize=True) # frequencies" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "b-_K_KHXz9MR" - }, - "source": [ - "As we can see, the target label is heavily skewed towards the patients not being readmitted within 30 days. In our dataset, only 11% of patients were readmitted within 30 days.\n", - "\n", - "Since there are fewer positive examples, we expect that we will have a much larger uncertainty (error bars) in our estimates of *false negative rates* (FNR), compared with *false positive rates* (FPR). This means that there will be larger differences between training FNR and test FNR, even if there is no overfitting, simply because of the smaller sample sizes. \n", - "\n", - "Our target metric is *balanced error rate*, which is the average of FPR and FNR. The value of this metric is robust to different frequencies of positives and negatives. However, since half of the metric is contributed by FNR, we expect the uncertainty in balanced error values to behave similarly to the uncertainty of FNR." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "OkopoyE3GQ8g" - }, - "source": [ - "Now, let's examine how much the label frequencies vary within each group defined by `race`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 285 - }, - "id": "tcH3Mxwmm4rC", - "outputId": "75641d75-670d-4a81-a354-855f80c38613" - }, - "outputs": [], - "source": [ - "sns.barplot(x=\"readmit_30_days\", y=\"race\", data=df, ci=95);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ox06aTMmm4rB" - }, - "source": [ - "We see the rate of *30-day readmission* is similar for the *AfricanAmerican* and *Caucasian* groups, but appears smaller for *Other* and smallest for *Unknown* (this is consistent with an overall lower rate of hospital visits in the prior year). The smaller sample size of the *Other* and *Unknown* groups mean that there is more uncertainty around the estimate for these two groups." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0AA5uoqAKUSx" - }, - "source": [ - "## Proxies for sensitive features\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "7PSup7dMJjJg" - }, - "source": [ - "We next investigate which of the features are highly predictive of the sensitive feature *race*; such features are called *proxies*.\n", - "\n", - "While in this tutorial we examine fairness issues through the **impact** of the machine-learning model on different populations, there are other concepts of fairness that seek to analyze how the **model might be using information** contained in the sensitive features, and which of the information uses are justified (often using causal reasoning). More pragmatically, certain uses of sensitive features (or proxies of it) might be illegal in some contexts.\n", - "\n", - "Another reason to understand the proxies is because they might explain why we see differences in impact on different groups even when our model does not have access to the sensitive features directly.\n", - "\n", - "In this section we briefly examine the identification of such proxies (but we don't go into legal or causality considerations).\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "1PHULa3eQEcn" - }, - "source": [ - "In the United States, *Medicare* and *Medicaid* are joint federal and state programs to help qualified individuals pay for healthcare expenses. *Medicare* is available to people over the age of 65 and younger individuals with severe illnesses. *Medicaid* is available to all individuals under the age of 65 whose adjusted gross income falls below the Federal Poverty Line. " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "VNYx0OaVElUC" - }, - "source": [ - "First, let's explore the relationship between patients who paid with *Medicaid* and our demographic features. Because *Medicaid* is available to low-income individuals, and race is correlated with socioeconomic status in the United States, we expect there to be a relationship between `race` and paying with *Medicaid*. " - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 285 - }, - "id": "dR3eNMeY0HTi", - "outputId": "aaa8498f-1f82-4003-c73c-ab6eaa3c2476" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"medicaid\", x=\"race\", data=df, join=False);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "MTcAD0oxbarW" - }, - "source": [ - "From our analysis, we see that paying with *Medicaid* does appear to have some relationship with the patient's race. *Caucasian* patients are the least likely to pay with *Medicaid* compared with other groups. If paying with *Medicaid* is a proxy for socioeconomic status, then the patterns we find align with our understanding of wealth and race in the United States." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "9BLSciTKaBLd" - }, - "source": [ - "## Additional validity checks" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "-PzKOA59ezFs" - }, - "source": [ - "Similarly as we used predictive validity to check that our label aligns with the construct of \"likely to benefit from the care management program\", we can use predictive validity to verify that our various features are coherent with each other.\n", - "\n", - "For example, based on the eligibility criteria for *Medicaid* vs *Medicare*, we expect `medicaid` to be negatively correlated with age and `medicare` to be positively correlated with age:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 285 - }, - "id": "869rzUjSUe3C", - "outputId": "bd4de952-204d-4159-addc-19b9d1b75167" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"medicaid\", x=\"age\", data=df, ci=95, join=False);" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 285 - }, - "id": "nnI8lFHFZyBL", - "outputId": "e0596d0e-efb7-4fc0-d775-99fdae53c8bf" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"medicare\", x=\"age\", data=df, ci=95, join=False);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "dBgmuoz8aXw4" - }, - "source": [ - "As we see, that's indeed the case." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "DdrS4By1dVk-" - }, - "source": [ - "\n", - "## Exercise" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "V-sOl0rqblsb" - }, - "source": [ - "Now, let's explore the relationship between paying with `medicare` and other demographic features. In the below sections, feel free to perform any analysis you would like to better understand the relationship between `medicare` and `race` and `gender` in this dataset." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 285 - }, - "id": "lp02oUbWZx54", - "outputId": "40c1cc55-625f-47d5-a21a-8950db45ab23" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"medicare\", x=____, data=df, ci=95, join=False);" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 285 - }, - "id": "ONJcedCWULXN", - "outputId": "1f0e82df-b12a-459b-9b77-13285ae44008" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"medicare\", x=____, data=df, ci=95, join=False);" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 285 - }, - "id": "jXtY7_xdULLe", - "outputId": "67948de6-24eb-4d61-9116-1d465534ee53" - }, - "outputs": [], - "source": [ - "sns.pointplot(y=\"medicare\", x=____, data=df, ci=95, join=False);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ZVSerRqDG3we" - }, - "source": [ - "\n", - "## Datasheets for datasets" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ACLMWSAwGaLc" - }, - "source": [ - "The _datasheets_ practice was proposed by [Gebru et al. (2018)](https://arxiv.org/abs/1803.09010). A datasheet of a given dataset documents the motivation behind the dataset creation, the dataset composition, collection process, recommended uses and many other characteristics. In the words of Gebru et al., the goal is to\n", - "> facilitate better communication between dataset creators\n", - "> and dataset consumers, and encourage the machine learning\n", - "> community to prioritize transparency and accountability.\n", - "\n", - "In this section, we show how to fill in some of the sections of the datasheet for the dataset that we are using. The information is obtained directly from [Strack et al. (2014)](https://www.hindawi.com/journals/bmri/2014/781670/)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "fcc4cUMhZnCb" - }, - "source": [ - "### Example sections of a datasheet [OPTIONAL SECTION]" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0TIPJIhX1IKJ" - }, - "source": [ - "**For what purpose was the dataset created?** *Was there a specific task in mind? Was there a specific gap that needed to be filled?*" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "sBNKbKJQJbmf" - }, - "source": [ - "In the words of the dataset authors:\n", - "> [...] the management of hyperglycemia in the hospitalized patient has a significant bearing on outcome, in terms of both morbidity and mortality. This recognition has led to the development of formalized protocols in the intensive care unit (ICU) setting [...] However, the same cannot be said for most non-ICU inpatient admissions. [...] there are few national assessments of diabetes care in the hospitalized patient which could serve as a baseline for change [in the non-ICU protocols]. The present analysis of a large clinical database was undertaken to examine historical patterns of diabetes care in patients with diabetes admitted to a US hospital and to inform future directions which might lead to improvements in patient safety." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "iA-XRu0o1ErL" - }, - "source": [ - "**Who created the dataset (e.g., which team) and on behalf of which entity?**" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "zQSHxl26LQkt" - }, - "source": [ - "The dataset was created by [Strack et al. (2014)](https://www.hindawi.com/journals/bmri/2014/781670/): a team of researchers from a variety of disciplines, ranging from computer science to public health, from three institutions (Virginia Commonwealth University, University of Cordoba, and Polish Academy of Sciences)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0Qx32mSJG3zP" - }, - "source": [ - "#### **Composition**" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "8RS2V8001F3E" - }, - "source": [ - "**What do the instances that comprise the dataset represent?**\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "JPy_TXp_1Gub" - }, - "source": [ - "Each instance in this dataset represents a hospital admission for diabetic patient (diabetes was entered as a possible diagnosis for the patient) whose hospital stay lasted between one to fourteen days." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "eOb0FPeOJqxm" - }, - "source": [ - "**Is any information missing from individual instances?**" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "4vlZWeQjJq8w" - }, - "source": [ - "The features `Payer Code` and `Medical Specialty` have 40,255 and 49,947 missing values, respectively. For `Payer Code`, these missing values are reflected in the category *Unknown*. For `Medical Specialty`, these missing values are reflecting in the category *Missing*. \n", - "\n", - "For our demographic features, we are missing the `Gender` information for three patients in the dataset. These three records were dropped from our final dataset. Regarding `Race`, the 2,271 missing values were recoded into the `Unknown` race category. \n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Uh0lLV6mJrSp" - }, - "source": [ - "**Does the dataset identify any subpopulations (e.g., by age, gender)?**" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "U-ZnQfibJrcQ" - }, - "source": [ - "Patients are identified by gender, age group, and race. \n", - "\n", - "For gender, patients are identified as Male, Female, or Unknown. There were only three instances where the patient gender is *Unknown*, so these records were removed from our dataset.\n", - "\n", - "Gender | Count| Percentage\n", - "------ | ------|----------\n", - "Male | 47055 | 46.2%\n", - "Female | 54708 | 53.7% \n", - "\n", - "\n", - "\n", - "For age group, patients are binned into three age buckets: *30 years or younger*, *30-60 years*, *Older than 60 years*.\n", - "\n", - "Age Group |Count| Percentage\n", - "------ | ------|----------\n", - "30 years or younger | 2509 | 2.4%\n", - "30-60 years | 30716 | 30.2%\n", - "Older than 60 years | 68538 | 67.4% \n", - "\n", - "\n", - "For race, patients are identified as *AfricanAmerican*, *Caucasian*, and *Other*. For individuals whose race information was not collected during hospital admission, their race is listed as *Unknown*.\n", - "\n", - "Race | Count| Percentage\n", - "------ | ------|----------\n", - "Caucasian | 76099 | 74.8%\n", - "AfricanAmerican | 19210 | 18.9% \n", - "Other | 4183 | 4.1%\n", - "Unknown | 2271 | 2.2%" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "yzXw0egqG4J4" - }, - "source": [ - "#### **Preprocessing**" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "rGfxGcI21Fyj" - }, - "source": [ - "**Was any preprocessing/cleaning/labeling of the data done?**" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "5jO4Pf911GrL" - }, - "source": [ - "For the `race` feature, the categories of *Asian* and *Hispanic* and *Other* were merged into the *Other* category. The `age` feature was bucketed into 30-year intervals (*30 years and below*, *30 to 60 years*, and *Over 60 years*). The `discharge_disposition_id` was binarized into a boolean outcome on whether an patient was discharged to home.\n", - "\n", - "The full preprocessing code is provided in the file `preprocess.py` of the tutorial [GitHub repository](https://github.com/fairlearn/talks/blob/main/2021_scipy_tutorial/).\n", - "\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "C8b5nXfPIA7a" - }, - "source": [ - "#### **Uses**\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "_YH8evvN1HX2" - }, - "source": [ - "**Has the dataset been used for any tasks already?** " - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "H8RW1LKW1Hbg" - }, - "source": [ - "This dataset has been used by [Strack et al. (2014)](https://www.hindawi.com/journals/bmri/2014/781670/) to model the relationship between patient readmission and HbA1c measurement during admission, based on primary medical diagnosis.\n", - "\n", - "The dataset is publicly available through the UCI Machine Learning Repository and, as of May 2021, has received over 350,000 views." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "HB7zfhA1UKiW" - }, - "source": [ - "# Training the initial model" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0f7jZOzGX24Z" - }, - "source": [ - "We next train a classification model to predict our target variable (readmission within 30 days) while optimizing balanced accuracy.\n", - "\n", - "What kind of model should we train? Deep neural nets? Random forests? Logistic regression?\n", - "\n", - "There are a variety of considerations. We highlight two:\n", - "\n", - "* **Interpretability.** Interpretability is tightly linked with questions of fairness. There are several reasons why it is important to have interpretable models that are open to the stakeholder scrutiny:\n", - " * It allows discovery of fairness issues that were not discovered by the data science team.\n", - " * It provides a path toward recourse for those that are affected by the model.\n", - " * It allows for a *face validity* check, a \"sniff test\", by experts to verify that the model \"makes sense\" (at the face value). While this step is subjective, it is really important when the model is applied to different populations than those on which the assessment was conducted.\n", - "\n", - "* **Model expressiveness.** How well can the model separate positive examples from negative examples? How well can it do so given the available dataset size? Can it do so across all groups or does it need to trade off performance on one group against performance on another group?\n", - "\n", - "Some additional considerations are training time (this impacts the ability to iterate), familiarity (this impacts the ability to fine tune and debug), and carbon footprint (this impacts the Earth climate both directly and indirectly by normalizing unnecessarily heavy workloads)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "7hbsqXap9Mzp" - }, - "source": [ - "### Decision point: Model type" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "iftAwdfoVDM0" - }, - "source": [ - "We will use a logistic regression model. Our reasoning:\n", - "\n", - "* **Interpretability**. Logistic models over a small number of variables (as used here) are highly interpretable in the sense that stakeholders can simulate them easily.\n", - "\n", - "* **Model expressiveness**. Logistic regression predictions are described by a linear weighting of the feature values. The concern might be that this is too simple. The previous work by [Strack et al. (2014)](https://www.hindawi.com/journals/bmri/2014/781670/), which also used a logistic model to predict readmission rates concluded that a slightly more expressive model might be useful (their analysis uncovered 8 pairwise interactions that were significant, see their Table 5)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "52wWLOFoXkho" - }, - "source": [ - "## Prepare training and test datasets" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "vCwtDyx8yqSG" - }, - "source": [ - "As we mentioned in the task definition, our target variable is **readmission within 30 days**, and our sensitive feature for the purposes of fairness assessment is **race**.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "WpiSRj2JyqSH" - }, - "outputs": [], - "source": [ - "target_variable = \"readmit_30_days\"\n", - "demographic = [\"race\", \"gender\"]\n", - "sensitive = [\"race\"]\n", - "# If multiple sensitive features are chosen, the rest of the script considers intersectional groups." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "uaNqDyqvi1QE" - }, - "outputs": [], - "source": [ - "Y, A = df.loc[:, target_variable], df.loc[:, sensitive]" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "niu49A9YXQmf" - }, - "source": [ - "We next drop the features that we don't want to use in our model and expand the categorical features into 0/1 indicators (\"dummies\")." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "JXyRuCsri1cY" - }, - "outputs": [], - "source": [ - "X = pd.get_dummies(df.drop(columns=[\n", - " \"race\",\n", - " \"race_all\",\n", - " \"discharge_disposition_id\",\n", - " \"readmitted\",\n", - " \"readmit_binary\",\n", - " \"readmit_30_days\"\n", - "]))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 274 - }, - "id": "kAA0sIhQUWFa", - "outputId": "73d62224-bf2e-40d2-a29a-8909e26d3cf9" - }, - "outputs": [], - "source": [ - "X.head() # sanity check" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ATzi8cKCD7V3" - }, - "source": [ - "We split our data into a training and test portion. The test portion will be used to evaluate our performance metric (i.e., balanced accuracy), but also for fairness assessment. The split is half/half for training and test to ensure that we have sufficient sample sizes for fairness assessment." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "-wpnURazmJ4-" - }, - "outputs": [], - "source": [ - "random_seed = 445\n", - "np.random.seed(random_seed)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "xgl_b-CUl7TW" - }, - "outputs": [], - "source": [ - "X_train, X_test, Y_train, Y_test, A_train, A_test, df_train, df_test = train_test_split(\n", - " X,\n", - " Y,\n", - " A,\n", - " df,\n", - " test_size=0.50,\n", - " stratify=Y,\n", - " random_state=random_seed\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "eSMbXR9iVqr8" - }, - "source": [ - "Our performance metric is **balanced accuracy**, so for the purposes of training (but not evaluation!) we will resample the data set, so that it has the same number of positive and negative examples. This means that we can use estimators that optimize standard accuracy (although some estimators allow the use us importance weights).\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "nPNQpb2ZN1ku" - }, - "source": [ - "Because we are downsampling the number of negative examples, we create a training dataset with a significantly lower number of data points. For more complex machine learning models, this lower number of training data points may affect the model's accuracy." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "L1aVgzyFNa4B" - }, - "outputs": [], - "source": [ - "def resample_dataset(X_train, Y_train, A_train):\n", - "\n", - " negative_ids = Y_train[Y_train == 0].index\n", - " positive_ids = Y_train[Y_train == 1].index\n", - " balanced_ids = positive_ids.union(np.random.choice(a=negative_ids, size=len(positive_ids)))\n", - "\n", - " X_train = X_train.loc[balanced_ids, :]\n", - " Y_train = Y_train.loc[balanced_ids]\n", - " A_train = A_train.loc[balanced_ids, :]\n", - " return X_train, Y_train, A_train" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "6Ogw-r3DQsds" - }, - "outputs": [], - "source": [ - "X_train_bal, Y_train_bal, A_train_bal = resample_dataset(X_train, Y_train, A_train)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "fRddJS7XXv5n" - }, - "source": [ - "## Save descriptive statistics of training and test data" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "hZ-T4lGxX0IQ" - }, - "source": [ - "We next evaluate and save descriptive statistics of the training dataset. These will be provided as part of _model cards_ to document our training." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 301 - }, - "id": "n3GhcUCm2LjD", - "outputId": "9a3be0c7-99ae-4de6-b61b-cf0fd0bd6b0e" - }, - "outputs": [], - "source": [ - "sns.countplot(x=\"race\", data=A_train_bal)\n", - "plt.title(\"Sensitive Attributes for Training Dataset\")\n", - "sensitive_train = figure_to_base64str(plt)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 301 - }, - "id": "lIp3j8fD2LjE", - "outputId": "48b4cbdf-c0ad-4a5b-d5f2-987d6b75229b" - }, - "outputs": [], - "source": [ - "sns.countplot(x=Y_train_bal)\n", - "plt.title(\"Target Label Histogram for Training Dataset\")\n", - "outcome_train = figure_to_base64str(plt)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 301 - }, - "id": "czIGZYhk2LjF", - "outputId": "07490aa7-c22e-4614-8580-be8b8b6db229" - }, - "outputs": [], - "source": [ - "sns.countplot(x=\"race\", data=A_test)\n", - "plt.title(\"Sensitive Attributes for Testing Dataset\")\n", - "sensitive_test = figure_to_base64str(plt)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 301 - }, - "id": "YjhW0t9-2LjF", - "outputId": "0785b4e1-88b8-4ab3-cb0b-9bf1d3df5ffe" - }, - "outputs": [], - "source": [ - "sns.countplot(x=Y_test)\n", - "plt.title(\"Target Label Histogram for Test Dataset\")\n", - "outcome_test = figure_to_base64str(plt)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "4V523GQbYobT" - }, - "source": [ - "## Train the model" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "g7jwN2cVbO0g" - }, - "source": [ - "We train a logistic regression model and save its predictions on test data for analysis." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "f6nKDzt164vw" - }, - "outputs": [], - "source": [ - "unmitigated_pipeline = Pipeline(steps=[\n", - " (\"preprocessing\", StandardScaler()),\n", - " (\"logistic_regression\", LogisticRegression(max_iter=1000))\n", - "])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 111 - }, - "id": "ld9clGbHl7tv", - "outputId": "89fa7f0d-680f-424f-a7cd-96686987a943" - }, - "outputs": [], - "source": [ - "unmitigated_pipeline.fit(X_train_bal, Y_train_bal)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "Ok-eREU0xbAD" - }, - "outputs": [], - "source": [ - "Y_pred_proba = unmitigated_pipeline.predict_proba(X_test)[:,1]\n", - "Y_pred = unmitigated_pipeline.predict(X_test)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "nkA0K8KV0HeD" - }, - "source": [ - "Check model performance on test data." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 285 - }, - "id": "nz7QJOLx0RVH", - "outputId": "ef446998-8269-4e9b-c8ba-43a777b947d8" - }, - "outputs": [], - "source": [ - "# Plot ROC curve of probabilistic predictions\n", - "plot_roc_curve(unmitigated_pipeline, X_test, Y_test);" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "pxYppCAy1owq", - "outputId": "b9febbe3-8016-43a7-c14a-98a94f05716a" - }, - "outputs": [], - "source": [ - "# Show balanced accuracy rate of the 0/1 predictions\n", - "balanced_accuracy_score(Y_test, Y_pred)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "jmWrDs5N2HVD" - }, - "source": [ - "As we see, the performance of the model is well above the performance of a coin flip (whose performance would be 0.5 in both cases), albeit it is quite far from a perfect classifier (whose performance would be 1.0 in both cases).\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "AmhwS1Z9VnK9" - }, - "source": [ - "## Inspect the coefficients of trained model\n", - "\n", - "We check the coefficients of the fitted model to make sure that they \"makes sense\". While subjective, this step is important and helps catch mistakes and might point out to some fairness issues. However, we will systematically assess the fairness of the model in the next section.\n", - "\n", - "*Note that coefficients are also a proxy for \"feature importance\", but this interpretation can be misleading when features are highly correlated.*" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 703 - }, - "id": "Owzkar8R9Cyy", - "outputId": "7c073a93-e3e5-4f22-e089-217f97967df7" - }, - "outputs": [], - "source": [ - "coef_series = pd.Series(data=unmitigated_pipeline.named_steps[\"logistic_regression\"].coef_[0], index=X.columns)\n", - "coef_series.sort_values().plot.barh(figsize=(4, 12), legend=False);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "hX8CrWjhD7MB" - }, - "source": [ - "# **Fairness assessment**" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0CS9-jaxtxh2" - }, - "source": [ - "## Measuring fairness-related harms\n", - "\n", - "\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "s8TMm9w8duVY" - }, - "source": [ - "The goal of fairness assessment is to answer the question: *Which groups of people may be disproportionately negatively impacted by an AI system and in what ways?*\n", - "\n", - "The steps of the assesment are as follows:\n", - "1. Identify harms\n", - "2. Identify the groups that might be harmed\n", - "3. Quantify harms\n", - "4. Compare quantified harms across the groups\n", - "\n", - "We next examine these four steps in more detail." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "X6hMFmzmPbL6" - }, - "source": [ - "### 1. Identify harms\n", - "\n", - "For example, in a system for screening job applications, qualified candidates that are automatically rejected experience an allocation harm. In a speech-to-text transcription system, high error rates constitute harm in the quality of service.\n", - "\n", - "**In the health care scenario**, the patients that would benefit from a care management program, but are not recommended for it experience an allocation harm. In the context of the classification scenario these are **FALSE NEGATIVES**." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "aqqUk1mjPnpM" - }, - "source": [ - "### 2. Identify the groups that might be harmed\n", - "\n", - "In most applications, we consider demographic groups including historically marginalized groups (e.g., based on gender, race, ethnicity). We should also consider groups that are relevant to a particular application. For example, for speech-to-text transcription, groups based on the regional dialect or being a native or a non-native speaker.\n", - "\n", - "It is also important to consider group intersections, for example, in addition to considering groups according to gender and groups according to race, it is also important to consider their intersections (e.g., Black women, Latinx nonbinary people, etc.).\n", - "\n", - "**In the health care scenario**, based on the previous work, we focus on groups defined by **RACE**." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "nmvSqI3dPrVk" - }, - "source": [ - "### 3. Quantify harms\n", - "\n", - "Define metrics that quantify harms or benefits:\n", - "\n", - "* In job screening scenario, we need to quantify the number of candidates that are classified as \"negative\" (not recommended for the job), but whose true label is \"positive\" (they are qualified). One possible metric is the **false negative rate**: fraction of qualified candidates that are screened out.\n", - "\n", - "* In speech-to-text scenario, the harm could be measured by **word error rate**, number of mistakes in a transcript divided by the overall number of words.\n", - "\n", - "* **In the health care scenario**, we could consider two metrics for quantifying harms / benefits:\n", - " * **false negative rate**: fraction of patients that are readmitted within 30 days, but that are not recommended for the care management program; this quantifies harm\n", - " * **selection rate**: overall fraction of patients that are recommended for the care management program (regardless of whether they are readmittted with 30 days or no); this quantifies benefit; here the assumption is that all patients benefit similarly from the extra care.\n", - "\n", - "There are several reasons for including selection rate in addition to false negative rate. We would like to monitor how the benefits are allocated, focusing on groups that might be disadvantaged. Another reason is to get extra robustness in our assessement, because our measure (i.e., readmission within 30 days) is only an imperfect measure of our construct (who is most likely to benefit from the care management program). The auxiliary metrics, like selection rate, may alert us to large disparities in how the benefit is allocated, and allow us to catch issues that we might have missed.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "fpJXt6miPvRX" - }, - "source": [ - "### 4. Compare quantified harms across the groups\n", - "\n", - "The workhorse of fairness assessment are _disaggregated metrics_, which are **metrics evaluated on slices of data**. For example, to measure harms due to errors, we would begin by evaluating the errors on each slice of the data that corresponds to a group we identified in Step 2.\n", - "If some of the groups are seeing much larger errors than other groups, we would flag this as a fairness harm.\n", - "\n", - "To summarize the disparities in errors (or other metrics), we may want to report quantities such as the **difference** or **ratio** of the metric values between the best and the worst slice. In settings where the goal is to guarantee certain minimum quality of service (such as speech recognition), it is also meaningful to report the **worst performance** across all considered groups.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "7Is_zdXvnW0s" - }, - "source": [ - "For example, when comparing false negative rate across groups defined by race, we may summarize our findings with a table like the following:\n", - "\n", - "| | false negative rate
(FNR) |\n", - "|---|---|\n", - "| AfricanAmerican | 0.43 |\n", - "| Caucasian | 0.44 |\n", - "| Other | 0.52 |\n", - "| Unknown | 0.67 |\n", - "| | |\n", - "|_largest difference_| 0.24   (best is 0.0)|\n", - "|_smallest ratio_| 0.64   (best is 1.0)|\n", - "|_maximum_
_(worst-case) FNR_|0.67|" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "9CjHlopBDgSG" - }, - "source": [ - "## Fairness assessment with `MetricFrame`" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "epJO2baHV2Dy" - }, - "source": [ - "Fairlearn provides the data structure called `MetricFrame` to enable evaluation of disaggregated metrics. We will show how to use a `MetricFrame` object to assess the trained `LogisticRegression` classifier for potential fairness-related harms.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "0iiAYRvoduPh", - "outputId": "3287b55e-3610-424d-a99d-a6e19c2b1693" - }, - "outputs": [], - "source": [ - "# In its simplest form MetricFrame takes four arguments:\n", - "# metric_function with signature metric_function(y_true, y_pred)\n", - "# y_true: array of labels\n", - "# y_pred: array of predictions\n", - "# sensitive_features: array of sensitive feature values\n", - "\n", - "mf1 = MetricFrame(metrics=false_negative_rate,\n", - " y_true=Y_test,\n", - " y_pred=Y_pred,\n", - " sensitive_features=df_test['race'])\n", - "\n", - "# The disaggregated metrics are stored in a pandas Series mf1.by_group:\n", - "\n", - "mf1.by_group" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "5tl1Qxt1fT2v", - "outputId": "e2af1585-7602-4664-f396-2f0d395d4ad2" - }, - "outputs": [], - "source": [ - "# The largest difference, smallest ratio and worst-case performance are accessed as\n", - "# mf1.difference(), mf1.ratio(), mf1.group_max()\n", - "\n", - "print(f\"difference: {mf1.difference():.3}\\n\"\n", - " f\"ratio: {mf1.ratio():.3}\\n\"\n", - " f\"max across groups: {mf1.group_max():.3}\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 203 - }, - "id": "R2zmBHo5gk-F", - "outputId": "e970f25b-7218-4430-8969-d8f891ddce27" - }, - "outputs": [], - "source": [ - "# You can also evaluate multiple metrics by providing a dictionary\n", - "\n", - "metrics_dict = {\n", - " \"selection_rate\": selection_rate,\n", - " \"false_negative_rate\": false_negative_rate,\n", - " \"balanced_accuracy\": balanced_accuracy_score,\n", - "}\n", - "\n", - "metricframe_unmitigated = MetricFrame(metrics=metrics_dict,\n", - " y_true=Y_test,\n", - " y_pred=Y_pred,\n", - " sensitive_features=df_test['race'])\n", - "\n", - "# The disaggregated metrics are then stored in a pandas DataFrame:\n", - "\n", - "metricframe_unmitigated.by_group" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "Hc29jRJrhlSC", - "outputId": "e2ffee50-2764-434f-d322-2a8fdafe6a09" - }, - "outputs": [], - "source": [ - "# The largest difference, smallest ratio, and the maximum and minimum values\n", - "# across the groups are then all pandas Series, for example:\n", - "\n", - "metricframe_unmitigated.difference()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 172 - }, - "id": "bVbjFa4Aig9Y", - "outputId": "228f2de1-4de8-4059-b166-c75550e52de2" - }, - "outputs": [], - "source": [ - "# You'll probably want to view them transposed:\n", - "\n", - "pd.DataFrame({'difference': metricframe_unmitigated.difference(),\n", - " 'ratio': metricframe_unmitigated.ratio(),\n", - " 'group_min': metricframe_unmitigated.group_min(),\n", - " 'group_max': metricframe_unmitigated.group_max()}).T" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 311 - }, - "id": "DvjRBIcjjSkl", - "outputId": "0aebd5a1-f287-4d75-cdd3-8af1daf1d190" - }, - "outputs": [], - "source": [ - "# You can also easily plot all of the metrics using DataFrame plotting capabilities\n", - "\n", - "metricframe_unmitigated.by_group.plot.bar(subplots=True, layout= [1,3], figsize=(12, 4),\n", - " legend=False, rot=-45, position=1.5);" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "b5C3SITjuPUm" - }, - "source": [ - "According to the above bar chart, it seems that the group *Unknown* is selected for the care management program less often than other groups as reflected by the selection rate. Also this group experiences the largest false negative rate, so a larger fraction of group members that are likely to benefit from the care management program are not selected. Finally, the balanced accuracy on this group is also the lowest.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "c2Qs68rv2_Vg" - }, - "source": [ - "We observe disparity, even though we did not include race in our model. There's a variety of reasons why such disparities may occur. It could be due to representational issues (i.e., not enough instances per group), or because the feature distribution itself differs across groups (i.e., different relationship between features and target variable, obvious example would be people with darker skin in facial recognition systems, but can be much more subtle). Real-world applications often exhibit both kinds of issues at the same time." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "n_1Rm8PbPmmk" - }, - "source": [ - "\n", - "## Exercise: Train other fairness-unaware models" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "oeQF5qT6Qs-C" - }, - "source": [ - "In this section, you'll be training your own fairness-unaware model and evaluate the model using the `MetricFrame` for fairness-related harms." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "SsHy-Os0oVQU" - }, - "source": [ - "We encourage you to explore the model's performance across different sensitive features (such as `age` or `gender`) as well as different model performance metrics." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "61GU_zrFSC-6" - }, - "source": [ - "1.) First, let's train our machine learning model. We'll create a `HistGradientBoostingClassifier` and fit it to the balanced training data set." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "bSIDkuV4_Rou" - }, - "outputs": [], - "source": [ - "from sklearn.experimental import enable_hist_gradient_boosting\r\n", - "from sklearn.ensemble import HistGradientBoostingClassifier\r\n", - "\r\n", - "# Create your model here\r\n", - "clf = HistGradientBoostingClassifier()\r\n", - "\r\n", - "# Fit the model to the training data\r\n", - "clf.fit(__________, ________)\r\n", - "exercise_pred = clf.predict(______)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Fnnles-p6OXr" - }, - "source": [ - "2.) Next, let's evaluate the fairness of the model using the `MetricFrame`. In the below cells, create a `MetricFrame` that looks at the following metrics:\n", - "\n", - "\n", - "* _Count_: The number of data points belonging to each sensitive feature category.\n", - "* _False Positive Rate_: $\\dfrac{FN}{FN+TP}$\n", - "* _Recall Score_: $\\dfrac{TP}{TP+FN}$\n", - "\n", - "As an extra challenge, you can use the prediction probabilities to compute the _ROC AUC Score_ for each sensitive group pair.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "bcf-x1oA_jP5" - }, - "outputs": [], - "source": [ - "# Define exercise fairness metrics of interest here\r\n", - "exercise_metrics = {\r\n", - " \"count\": count,\r\n", - " \"false_positive_rate\": _______,\r\n", - " \"recall_score\": _______\r\n", - "}" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Bll-8GAWJF6p" - }, - "source": [ - "Now, let's create our `MetricFrame` using the metrics listed above with the sensitive groups of `race` and `gender`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "jAjzjCqh_fNx" - }, - "outputs": [], - "source": [ - "metricframe_exercise = MetricFrame(\r\n", - " metrics=__________,\r\n", - " y_true=Y_test,\r\n", - " y_pred=__________,\r\n", - " sensitive_features=_____\r\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "QeghVCbLZOf5" - }, - "source": [ - "3.) Finally, play around with the plotting capabilities of the `MetricFrame` in the below section.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "Nd4D17ME_hB2" - }, - "outputs": [], - "source": [ - "metricframe_exercise._______" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "_xaLx6Br_hyc" - }, - "outputs": [], - "source": [ - "# Plot some of the performance disparities here\r\n", - "metricframe_exercise.by_group.____.bar(subplots=_____, layout=[1,3], figsize=(12, 4),\r\n", - " legend=False, rot=-45, position=1.5)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Me1ocEi2kEgw" - }, - "source": [ - "The charts above are based on test data, so without any uncertainty quantification (such as error bars or confidence intervals), we cannot reliably compare these data statistics. Next optional section shows how to augment MetricFrame with the report of error bars.\n", - "\n", - "## Adding error bars [OPTIONAL SECTION]" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "9l8YJ8qQdehm" - }, - "source": [ - "In this section, we define new custom metrics that quantify errors in our estimates of selection rate, false negative rate and balanced accuracy, and then review our metrics again." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "OiP-uXr_FLtz" - }, - "outputs": [], - "source": [ - "# All of our error bar calculations are based on normal approximation to\n", - "# the binomial variables.\n", - "\n", - "def error_bar_normal(n_successes, n_trials, z=1.96):\n", - " \"\"\"\n", - " Computes the error bars for the parameter p of a binomial variable\n", - " using normal approximation. The default value z corresponds to the 95%\n", - " confidence interval.\n", - " \"\"\"\n", - " point_est = n_successes / n_trials\n", - " error_bar = z*np.sqrt(point_est*(1-point_est))/np.sqrt(n_trials)\n", - " return error_bar\n", - "\n", - "def fpr_error(Y_true, Y_pred):\n", - " \"\"\"\n", - " Compute the 95%-error bar for the false positive rate\n", - " \"\"\"\n", - " tn, fp, fn, tp = confusion_matrix(Y_true, Y_pred).ravel()\n", - " return error_bar_normal(fp, tn+fp)\n", - "\n", - "def fnr_error(Y_true, Y_pred):\n", - " \"\"\"\n", - " Compute the 95%-error bar for the false negative rate\n", - " \"\"\"\n", - " tn, fp, fn, tp = confusion_matrix(Y_true, Y_pred).ravel()\n", - " return error_bar_normal(fn, fn+tp)\n", - "\n", - "def selection_rate_error(Y_true, Y_pred):\n", - " \"\"\"\n", - " Compute the 95%-error bar for the selection rate\n", - " \"\"\"\n", - " tn, fp, fn, tp = confusion_matrix(Y_true, Y_pred).ravel()\n", - " return error_bar_normal(tp+fp, tn+fp+fn+tp)\n", - "\n", - "def balanced_accuracy_error(Y_true, Y_pred):\n", - " \"\"\"\n", - " Compute the 95%-error bar for the balanced accuracy\n", - " \"\"\"\n", - " fnr_err, fpr_err = fnr_error(Y_true, Y_pred), fpr_error(Y_true, Y_pred)\n", - " return np.sqrt(fnr_err**2 + fpr_err**2)/2" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "qHaXfBYWp6ob" - }, - "source": [ - "We next create a metric frame that includes the sample sizes and error bar sizes in addition to the metrics that we have used previously." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "OlEq6ogfyHb6" - }, - "outputs": [], - "source": [ - "metrics_with_err_bars = {\n", - " \"count\": count,\n", - " \"selection_rate\": selection_rate,\n", - " \"selection_err_bar\": selection_rate_error,\n", - " \"false_negative_rate\": false_negative_rate,\n", - " \"fnr_err_bar\": fnr_error,\n", - " \"balanced_accuracy\": balanced_accuracy_score,\n", - " \"bal_acc_err_bar\": balanced_accuracy_error\n", - "}\n", - "\n", - "# sometimes we will only want to display metrics without error bars\n", - "metrics_to_display = [\n", - " \"count\",\n", - " \"selection_rate\",\n", - " \"false_negative_rate\",\n", - " \"balanced_accuracy\"\n", - "]\n", - "\n", - "# sometimes we will only want to show the difference values of the metrics other than count\n", - "differences_to_display = [\n", - " \"selection_rate\",\n", - " \"false_negative_rate\",\n", - " \"balanced_accuracy\"\n", - "]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "55GOqFTYxbCi" - }, - "outputs": [], - "source": [ - "metricframe_unmitigated_w_err = MetricFrame(\n", - " metrics=metrics_with_err_bars,\n", - " y_true=Y_test,\n", - " y_pred=Y_pred,\n", - " sensitive_features=A_test\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 223 - }, - "id": "MlZKPNkHxbFb", - "outputId": "9d6cb95c-538a-42e7-f3b4-349c74258902" - }, - "outputs": [], - "source": [ - "unmitigated_groups = metricframe_unmitigated_w_err.by_group\n", - "unmitigated_groups # show both the metrics as well as the error bars" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "hh5F-A5C5rSV" - }, - "source": [ - "We see that for smaller sample sizes we have larger error bars. The problem is further exacerbated for false negative rate, which is estimated only over *positive examples* and so its sample sizes is further reduced due to label imbalance.\n", - "\n", - "We next visualize the metrics with the corresponding error bars using a custom plotting function." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "rqFPLsATwROr" - }, - "outputs": [], - "source": [ - "def plot_group_metrics_with_error_bars(metricframe, metric, error_name):\n", - " \"\"\"\n", - " Plots the disaggregated `metric` for each group with an associated\n", - " error bar. Both metric and the erro bar are provided as columns in the \n", - " provided metricframe.\n", - " \"\"\"\n", - " grouped_metrics = metricframe.by_group\n", - " point_estimates = grouped_metrics[metric]\n", - " error_bars = grouped_metrics[error_name]\n", - " lower_bounds = point_estimates - error_bars\n", - " upper_bounds = point_estimates + error_bars\n", - "\n", - " x_axis_names = [str(name) for name in error_bars.index.to_flat_index().tolist()]\n", - " plt.vlines(x_axis_names, lower_bounds, upper_bounds, linestyles=\"dashed\", alpha=0.45)\n", - " plt.scatter(x_axis_names, point_estimates, s=25)\n", - " plt.xticks(rotation=0)\n", - " y_start, y_end = np.round(min(lower_bounds), decimals=2), np.round(max(upper_bounds), decimals=2)\n", - " plt.yticks(np.arange(y_start, y_end, 0.05))\n", - " plt.ylabel(metric)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 268 - }, - "id": "dsRFpuXfzrUA", - "outputId": "0f4adb40-6c34-45a2-b8ca-b50d151c4b55" - }, - "outputs": [], - "source": [ - "plot_group_metrics_with_error_bars(metricframe_unmitigated_w_err, \"selection_rate\", \"selection_err_bar\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 268 - }, - "id": "B68Q2ZIgzrcE", - "outputId": "368fed7e-80c6-4fca-ebef-d61cdd09e800" - }, - "outputs": [], - "source": [ - "plot_group_metrics_with_error_bars(metricframe_unmitigated_w_err, \"false_negative_rate\", \"fnr_err_bar\")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 268 - }, - "id": "htoUhyirpqZt", - "outputId": "c83ac73e-ccbf-4a53-8002-c2dc012b8e4c" - }, - "outputs": [], - "source": [ - "plot_group_metrics_with_error_bars(metricframe_unmitigated_w_err, \"balanced_accuracy\", \"bal_acc_err_bar\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "SAMhqvycR7eE" - }, - "source": [ - "As we see above, even accounting for the larger uncertainty in estimating the false negative rate for *Unknown*, this group is experiencing substantially larger false negative rate than other groups and thus experiences the harm of allocation." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "8ZqVGZkam1eH" - }, - "source": [ - "\n", - "\n", - "---\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "-dgITdRiD7Yu" - }, - "source": [ - "# **Mitigating fairness-related harms in ML models**" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "sbUSG1jVA06G" - }, - "source": [ - "We have found that the logistic regression predictor leads to a large difference in false negative rates between the groups. We next look at **algorithmic mitigation strategies** of this fairness issue (and similar ones).\n", - "\n", - "*Note that while we currently focus on the training stage of the AI lifecycle mitigation should not be limited to this stage. In fact, we have already discussed mitigation strategies that are applicable at the task definition stage (e.g., checking for construct validity) and data collection stage (e.g., collecting more data).*\n", - "\n", - "Within the model training stage, mitigation may occur at different steps relative to model training:\n", - "\n", - "* **Preprocessing**: A mitigation algorithm is applied to transform the input data to the training algorithm; for example, some strategies seek to remove and dependence between the input features and sensitive features.\n", - "\n", - "* **At training time**: The model is trained by an (optimization) algorithm that seeks to satisfy fairness constraints.\n", - "\n", - "* **Postprocessing**: The output of a trained model is transformed to mitigate fairness issues; for example, the predicted probability of readmission is thresholded according to a group-specific threshold.\n", - "\n", - "We will now dive into two algorithms: a postprocessing approach and a reductions approach (which is a training-time algorithm). Both of them are in fact **meta-algorithms** in the sense that they act as wrappers around *any* standard (fairness-unaware) machine learning algorithms. This makes them quite versatile in practice.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "rX8QycCL0mJj" - }, - "source": [ - "## Postprocessing with `ThresholdOptimizer`" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "fRZfSFzcFaXP" - }, - "source": [ - "**Postprocessing** techniques are a class of unfairness-mitigation algorithms that take an already trained model and a dataset as an input and seek to fit a transformation function to model's outputs to satisfy some (group) fairness constraint(s). They might be the only feasible unfairness mitigation approach when developers cannot influence training of the model, due to practical reasons or due to security or privacy.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "6PgzZkK9Wbni" - }, - "source": [ - "Here we use the `ThresholdOptimizer` algorithm from Fairlearn, which follows the approach of [Hardt, Price, and Srebro (2016)](https://arxiv.org/abs/1610.02413).\n", - "\n", - "`ThresholdOptimizer` takes in an exisiting (possibly pre-fit) machine learning model whose predictions act as a scoring function and identifies a separate thrceshold for each group in order to optimize some specified objective metric (such as **balanced accuracy**) subject to specified fairness constraints (such as **false negative rate parity**). Thus, the resulting classifier is just a suitably thresholded version of the underlying machinelearning model.\n", - "\n", - "The constraint **false negative rate parity** requires that all the groups have equal values of false negative rate.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "OFOovaN7AwDr" - }, - "source": [ - "To instatiate our `ThresholdOptimizer`, we pass in:\n", - "\n", - "* An existing `estimator` that we wish to threshold. \n", - "* The fairness `constraints` we want to satisfy.\n", - "* The `objective` metric we want to maximize.\n", - "\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "8je0grKPWHhy" - }, - "outputs": [], - "source": [ - "# Now we instantite ThresholdOptimizer with the logistic regression estimator\n", - "postprocess_est = ThresholdOptimizer(\n", - " estimator=unmitigated_pipeline,\n", - " constraints=\"false_negative_rate_parity\",\n", - " objective=\"balanced_accuracy_score\",\n", - " prefit=True,\n", - " predict_method='predict_proba'\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "VDD86L7eCSe0" - }, - "source": [ - "In order to use the `ThresholdOptimizer`, we need access to the sensitive features **both during training time and once it's deployed**." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 344 - }, - "id": "VCHJBB7x1rAK", - "outputId": "dfc6338e-2be0-4007-bde2-fd8544cc4a89" - }, - "outputs": [], - "source": [ - "postprocess_est.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "YscNZsYU1rCY" - }, - "outputs": [], - "source": [ - "# Record and evaluate the output of the trained ThresholdOptimizer on test data\n", - "\n", - "Y_pred_postprocess = postprocess_est.predict(X_test, sensitive_features=A_test)\n", - "metricframe_postprocess = MetricFrame(\n", - " metrics=metrics_dict,\n", - " y_true=Y_test,\n", - " y_pred=Y_pred_postprocess,\n", - " sensitive_features=A_test\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "_izbGv6tQ1KD" - }, - "source": [ - "We can now inspect how the metric values differ between the postprocessed model and the unmitigated model:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 254 - }, - "id": "-9mtWyWc1rH5", - "outputId": "982dcb16-b82f-42bc-addb-1caaf2d0aa09" - }, - "outputs": [], - "source": [ - "pd.concat([metricframe_unmitigated.by_group,\n", - " metricframe_postprocess.by_group],\n", - " keys=['Unmitigated', 'ThresholdOptimizer'],\n", - " axis=1)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "mzPCUFsXPU_S" - }, - "source": [ - "We next zoom in on differences between the largest and the smallest metric values:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 127 - }, - "id": "dsC4v8Ap1rKt", - "outputId": "ebf489b6-07ac-45d5-ae2b-848d6488dbb4" - }, - "outputs": [], - "source": [ - "pd.concat([metricframe_unmitigated.difference(),\n", - " metricframe_postprocess.difference()],\n", - " keys=['Unmitigated: difference', 'ThresholdOptimizer: difference'],\n", - " axis=1).T" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Hhi_RSxSRoyg" - }, - "source": [ - "As we see, `ThresholdOptimizer` was able to substantiallydecrease the difference between the values of false negative rate." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "GarQvopkVN2S" - }, - "source": [ - "Finally, we save the disagregated statistics:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 314 - }, - "id": "EsTehBH2SW7f", - "outputId": "3cf05d43-1389-41a2-e2dc-7e7511833c2b" - }, - "outputs": [], - "source": [ - "metricframe_postprocess.by_group.plot.bar(subplots=True, layout=[1,3], figsize=(12, 4), legend=False, rot=-45, position=1.5)\n", - "postprocess_performance = figure_to_base64str(plt)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "umd3slsDmk0d" - }, - "source": [ - "Next optional section shows that `ThresholdOptimizer` more closely satisfies constraints on the training data than on the test data.\n", - "\n", - "### Postprocessing: Correctness check [OPTIONAL SECTION]" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Cs1kWzda8f0z" - }, - "source": [ - "We can verify that `ThresholdOptimizer` achieves false negative rate parity on the training dataset, meaning that the values of the false negative rate parity with respect to all groups are close on the training data." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 203 - }, - "id": "ttOZAVbgmplf", - "outputId": "c6bb375e-5e32-4508-cfd8-45f775618519" - }, - "outputs": [], - "source": [ - "# Record and evaluate the output of the ThresholdOptimizer on the training data\n", - "\n", - "Y_pred_postprocess_training = postprocess_est.predict(X_train_bal, sensitive_features=A_train_bal)\n", - "metricframe_postprocess_training = MetricFrame(\n", - " metrics=metrics_dict,\n", - " y_true=Y_train_bal,\n", - " y_pred=Y_pred_postprocess_training,\n", - " sensitive_features=A_train_bal\n", - ")\n", - "metricframe_postprocess_training.by_group" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "Tty0cmJomp1h", - "outputId": "1d821a8d-3e0a-4082-8efe-740d09ceb1e0" - }, - "outputs": [], - "source": [ - "# Evaluate the difference between the largest and smallest value of each metric\n", - "metricframe_postprocess_training.difference()" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Kvdn7wBhCqXw" - }, - "source": [ - "The value of `false_negative_rate_difference` on the training data is smaller than on the test data." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "z2vLNUnK_P66" - }, - "source": [ - "\n", - "### Exercise: ThresholdOptimizer" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "0YUembo02yQ8" - }, - "source": [ - "In this exercise, we will create a `ThresholdOptimizer` by constraining the *true positive rate* (also known as the *recall score*). For any model, the *true positive rate* + *false negative rate* = 1. \n", - "\n", - "By trying to achieve the *true positive rate parity*, we should produce a `ThresholdOptimizer` with the same performance as our original `ThresholdOptimizer`.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "ZJxE2eMuNF3_" - }, - "source": [ - "#### 1.) Create a new ThresholdOptimizer with the constraint `true_positive_rate_parity` and objective `balanced_accuracy_score`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "zD5kQu6gqyl6" - }, - "outputs": [], - "source": [ - "thresopt_exercise = ThresholdOptimizer(\r\n", - " estimator=______________,\r\n", - " constraints=____________,\r\n", - " objective=\"balanced_accuracy_score\",\r\n", - " prefit=True,\r\n", - " predict_method='predict_proba'\r\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "CxdAikCKqyuW" - }, - "outputs": [], - "source": [ - "thresopt_exercise.____(X_train_bal, Y_train_bal, sensitive_features=_______)\r\n", - "threshopt_pred = thresopt_exercise._________(X_test, sensitive_features=_______)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "xBLD77OENFN-" - }, - "source": [ - "#### 2.) Create a new `MetricFrame` object to process the results of this classifier." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "B12UNsZErImo" - }, - "outputs": [], - "source": [ - "thresop_metricframe = MetricFrame(\r\n", - " metrics=metrics_dict,\r\n", - " y_true=Y_test,\r\n", - " y_pred=____________,\r\n", - " sensitive_features=_______\r\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "DEyEie-9NEew" - }, - "source": [ - "#### 3.) Compare the performance of the two `ThresholdOptimizers`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "uz9mS66YrWka" - }, - "outputs": [], - "source": [ - "# Visualize the performance of the new ThresholdOptimizer\r\n", - "thresop_metricframe._______" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "oByEABVGrXWo" - }, - "outputs": [], - "source": [ - "# Compare the performance to the original ThresholdOptimizer\r\n", - "metricframe_postprocess.______" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "wAUI3LdQbBCs" - }, - "source": [ - "Similar to many unfairness mitigation approaches, `ThresholdOptimizer` produces randomized classifiers. Next optional section presents a heuristic strategy for converting a randomized `ThresholdOptimizer` into a deterministic one. In our scenario, this heursitic is quite effective and the resulting deterministic classifier has similar performance as the original `ThresholdOptimizer`.\n", - "\n", - "### Deployment considerations: Randomized predictions [OPTIONAL SECTION]" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "wlfeHwVC6dna" - }, - "source": [ - "When we were describing `ThresholdOptimizer` we said that it picks a separate threshold for each group. However, that is not quite correct. In fact,`ThresholdOptimizer`, for each group, picks two thresholds that are close to each other (say `threshold0` and `threshold1`) and then, at deployment time, randomizes between the two: choosing `threshold0` with some probability `p0` and `threshold1` with the remaining probability `p1=1-p0` (the specific probabilities are determined during training; for certain kinds of constraints, three thresholds are considered.)\n", - "\n", - "This means that the predictions are randomized. To achieve reproducible randomization, it is possible to provide an argument `random_state` to the `predict` method. However, in some settings, even such reproducible randomization is not acceptable and can be in fact viewed as a fairness issue, because of its arbitrariness.\n", - "\n", - "One derandomization heuristic is to replace the two thresholds by their weighted average, i.e., `threshold = p0*threshold0 + p1*threshold1`. That corresponds to the assumption that the values of the scores between the two thresholds are approximately uniformly distributed. Using this heuristic, we derandomize `ThresholdOptimizer`.\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "nia26AD2BBKf" - }, - "source": [ - "The randomized model of the `ThresholdOptimizer` is stored as the field\n", - "`interpolated_thresholder_` in the fitted ThresholdOptimizer, which is itself a\n", - "valid estimator of type `InterpolatedThresholder`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 112 - }, - "id": "5w-c22tsZsvj", - "outputId": "48c9f6ef-9457-41d2-afe4-78965966e470" - }, - "outputs": [], - "source": [ - "interpolated = postprocess_est.interpolated_thresholder_\n", - "interpolated" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "8PkqiznTuL0l" - }, - "source": [ - "The `interpolation_dict` is a dictionary which assign to each sensitive feature value two thresholds and two respective probabilities. Using our derandomization strategy, we can create a dictionary that represents a deterministic rule:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "F3a1Mw8koLUa" - }, - "outputs": [], - "source": [ - "def create_deterministic(interpolate_dict):\n", - " \"\"\"\n", - " Creates a deterministic interpolation_dictionary from a randomized\n", - " interpolation_dictionary. The determinstic thresholds are created by taking\n", - " the weighted combinations of the two randomized thresholds for each sensitive\n", - " group.\n", - " \"\"\"\n", - " deterministic_dict = {}\n", - " for (race, operations) in interpolate_dict.items():\n", - " op0, op1 = operations[\"operation0\"]._threshold, operations[\"operation1\"]._threshold\n", - " p0, p1 = operations[\"p0\"], operations[\"p1\"]\n", - " deterministic_dict[race] = Bunch(\n", - " p0=0.0,\n", - " p1=1.0,\n", - " operation0=ThresholdOperation(operator=\">\",threshold=(p0*op0 + p1*op1)),\n", - " operation1=ThresholdOperation(operator=\">\",threshold=(p0*op0 + p1*op1))\n", - " )\n", - " return deterministic_dict" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "lbynyyy2CKxv", - "outputId": "09ac99a7-5338-4e31-9e02-eb4acbb9d5ec" - }, - "outputs": [], - "source": [ - "deterministic_dict = create_deterministic(interpolated.interpolation_dict)\n", - "deterministic_dict" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "euqycBW_3EB2" - }, - "source": [ - "Now, we can create an `InterpolatedThresholder` that uses the same pre-fit estimator, but with derandomized thresholds." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "R3qKiaMiPOaG" - }, - "outputs": [], - "source": [ - "deterministic_thresholder = InterpolatedThresholder(estimator=interpolated.estimator,\n", - " interpolation_dict=deterministic_dict,\n", - " prefit=True,\n", - " predict_method='predict_proba')" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 273 - }, - "id": "XLNNadraaFMT", - "outputId": "b3a7edc0-dfe1-4746-f208-0526ad4019a9" - }, - "outputs": [], - "source": [ - "deterministic_thresholder.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "TPRwWuuWY2Yy" - }, - "outputs": [], - "source": [ - "y_pred_postprocess_deterministic = deterministic_thresholder.predict(X_test, sensitive_features=A_test)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "q3834uLWeXWO" - }, - "outputs": [], - "source": [ - "mf_deterministic = MetricFrame(\n", - " metrics=metrics_dict,\n", - " y_true=Y_test,\n", - " y_pred=y_pred_postprocess_deterministic,\n", - " sensitive_features=A_test\n", - ")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "2tnGhMYzDloo" - }, - "source": [ - "Now compare the two models in terms of their disaggregated metrics:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 203 - }, - "id": "NIkoljGTeXZ0", - "outputId": "353905d4-3954-421a-9bf1-adbd6a8febd1" - }, - "outputs": [], - "source": [ - "mf_deterministic.by_group" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 203 - }, - "id": "mVimPa2nlFpv", - "outputId": "7c197ff9-4eed-4e1d-d00d-6b16ddd34bb6" - }, - "outputs": [], - "source": [ - "metricframe_postprocess.by_group" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "JOi4dwLKECHS" - }, - "source": [ - "The differences are generally small except for the *Unknown* group, whose false negative rate goes down and balanced accuracy goes up." - ] + "cells": [ + { + "cell_type": "markdown", + "source": [ + "# SciPy 2021 Tutorial
_Fairness in AI systems:
From social context to practice using Fairlearn_\n", + "\n", + "---\n", + "\n", + "_SciPy 2021 Tutorial: Fairness in AI systems: From social context to practice using Fairlearn by Manojit Nandi, Miroslav Dudík, Triveni Gandhi, Lisa Ibañez, Adrin Jalali, Michael Madaio, Hanna Wallach, Hilde Weerts is licensed under\n", + "[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)._\n", + "\n", + "---" + ], + "metadata": { + "id": "NDxtnoIr7mIN" + } + }, + { + "cell_type": "markdown", + "source": [ + "Fairness in AI systems is an interdisciplinary field of research and practice that aims to understand and address some of the negative impacts of AI systems on society. In this tutorial, we will walk through the process of assessing and mitigating fairness-related harms in the context of the U.S. health care system. This tutorial will consist of a mix of instructional content and hands-on demonstrations using Jupyter notebooks. Participants will use the Fairlearn library to assess ML models for performance disparities across different racial groups and mitigate those disparities using a variety of algorithmic techniques." + ], + "metadata": { + "id": "Sch9KDWg7SL8" + } + }, + { + "cell_type": "markdown", + "source": [ + "# **Prepare environment**" + ], + "metadata": { + "id": "fboVvqvVvpKz" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Install packages" + ], + "metadata": { + "id": "o0p461hmgrmz" + } + }, + { + "cell_type": "markdown", + "source": [ + "Note that the runtime environment needs to be restarted after installing `model-card-toolkit`." + ], + "metadata": { + "id": "9b5Nsb2Rgux7" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "!pip install --upgrade fairlearn==0.7.0\r\n", + "!pip install --upgrade scikit-learn\r\n", + "!pip install --upgrade seaborn\r\n", + "!pip install model-card-toolkit" + ], + "outputs": [], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000 }, - { - "cell_type": "markdown", - "metadata": { - "id": "NU_rncBQ0lab" - }, - "source": [ - "## Reductions approach with `ExponentiatedGradient`" - ] + "id": "w_uVNMHUdb2w", + "outputId": "8ef912ea-10cf-47ef-c85d-574a044283e3" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Import and set up packages" + ], + "metadata": { + "id": "fVtpwFGJLcgU" + } + }, + { + "cell_type": "code", + "execution_count": 1, + "source": [ + "import numpy as np\r\n", + "import pandas as pd\r\n", + "\r\n", + "pd.set_option(\"display.float_format\", \"{:.3f}\".format)" + ], + "outputs": [], + "metadata": { + "id": "cEwLsWyTLgJn" + } + }, + { + "cell_type": "code", + "execution_count": 2, + "source": [ + "import matplotlib.pyplot as plt\r\n", + "import seaborn as sns\r\n", + "\r\n", + "sns.set()" + ], + "outputs": [], + "metadata": { + "id": "wBOSsyc48MK0" + } + }, + { + "cell_type": "code", + "execution_count": 3, + "source": [ + "from sklearn.linear_model import LogisticRegression\r\n", + "from sklearn.model_selection import train_test_split\r\n", + "from sklearn.preprocessing import StandardScaler\r\n", + "from sklearn.pipeline import Pipeline\r\n", + "from sklearn.utils import Bunch\r\n", + "from sklearn.metrics import (\r\n", + " balanced_accuracy_score,\r\n", + " roc_auc_score,\r\n", + " accuracy_score,\r\n", + " recall_score,\r\n", + " confusion_matrix,\r\n", + " roc_auc_score,\r\n", + " roc_curve,\r\n", + " plot_roc_curve)\r\n", + "from sklearn import set_config\r\n", + "\r\n", + "set_config(display=\"diagram\")" + ], + "outputs": [], + "metadata": { + "id": "VhbFzU6GLgW0" + } + }, + { + "cell_type": "code", + "execution_count": 4, + "source": [ + "from fairlearn.metrics import (\r\n", + " MetricFrame,\r\n", + " true_positive_rate,\r\n", + " false_positive_rate,\r\n", + " false_negative_rate,\r\n", + " selection_rate,\r\n", + " count,\r\n", + " false_negative_rate_difference\r\n", + ")\r\n", + "\r\n", + "from fairlearn.postprocessing import ThresholdOptimizer, plot_threshold_optimizer\r\n", + "from fairlearn.postprocessing._interpolated_thresholder import InterpolatedThresholder\r\n", + "from fairlearn.postprocessing._threshold_operation import ThresholdOperation\r\n", + "from fairlearn.reductions import ExponentiatedGradient, EqualizedOdds, TruePositiveRateParity" + ], + "outputs": [], + "metadata": { + "id": "GcilMnWhLgaT" + } + }, + { + "cell_type": "code", + "execution_count": 5, + "source": [ + "# Model Card Toolkit works in Google Colab, but it does not work on all local environments\r\n", + "# that we tested. If the import fails, define a dummy function in place of the function\r\n", + "# for saving figures into images in a model card..\r\n", + "\r\n", + "try:\r\n", + " from model_card_toolkit import ModelCardToolkit\r\n", + " from model_card_toolkit.utils.graphics import figure_to_base64str\r\n", + " model_card_imported = True\r\n", + "except Exception:\r\n", + " model_card_imported = False\r\n", + " def figure_to_base64str(*args):\r\n", + " return None" + ], + "outputs": [], + "metadata": { + "id": "oMbCFBSZr2A4" + } + }, + { + "cell_type": "code", + "execution_count": 6, + "source": [ + "from IPython import display\r\n", + "from datetime import date" + ], + "outputs": [], + "metadata": { + "id": "S4ANMj1M8k0h" + } + }, + { + "cell_type": "markdown", + "source": [ + "# **Overview of fairness in AI systems**" + ], + "metadata": { + "id": "4OoPTetsDv-v" + } + }, + { + "cell_type": "markdown", + "source": [ + "Please refer to the slides here: https://raw.githubusercontent.com/fairlearn/talks/main/2021_scipy_tutorial/overview.pdf\n", + "\n" + ], + "metadata": { + "id": "vt9n43o6DPbg" + } + }, + { + "cell_type": "markdown", + "source": [ + "# **Introduction of Fairlearn and other tutorial resources**\n" + ], + "metadata": { + "id": "0gZTFumIxCfK" + } + }, + { + "cell_type": "markdown", + "source": [ + "This tutorial builds on the following open source projects:\n", + "\n", + "* **machine learning and data processing**: _scikit-learn_, _pandas_, _numpy_\n", + "* **plotting**: _seaborn_, _matplotlib_\n", + "* **AI fairness**: _Fairlearn_, _Model Card Toolkit_" + ], + "metadata": { + "id": "7KhsuCB-imFk" + } + }, + { + "cell_type": "markdown", + "source": [ + "### [Fairlearn](https://fairlearn.org)\n", + "\n", + "Fairlearn is an open-source, community-driven project to help data scientists improve fairness of AI systems. It includes:\n", + "\n", + "* A Python library for fairness assessment and improvement (fairness metrics, mitigation algorithms, plotting, etc.)\n", + "\n", + "* Educational resources covering organizational and technical processes for unfairness mitigation (user guide, case studies, Jupyter notebooks, etc.)\n", + "\n", + "The project was started in 2018 at Microsoft Research. In 2021 it adopted neutral governance structure and since then it is completely community-driven." + ], + "metadata": { + "id": "ORfTAukTuEPN" + } + }, + { + "cell_type": "markdown", + "source": [ + "### [Model Card Toolkit](https://github.com/tensorflow/model-card-toolkit)\n", + "\n", + "The Model Card Toolkit (MCT) streamlines and automates generation of _model cards_, machine learning documents that provide context and transparency into a model's development and performance. It was released by Google in 2020." + ], + "metadata": { + "id": "AE3bFABst9ze" + } + }, + { + "cell_type": "markdown", + "source": [ + "# **Introduction to the health care scenario**" + ], + "metadata": { + "id": "_j1vtg6TD7Fi" + } + }, + { + "cell_type": "markdown", + "source": [ + "Our scenario builds on previous research that highlighted racial disparities in how health care resources are allocated in the U.S. ([Obermeyer et al., 2019](https://science.sciencemag.org/content/366/6464/447.full)).\n", + "Motivated by that work, in this tutorial we consider an automated system for recommending patients for _high-risk care management_ programs, which are described by Obermeyer et al. 2019 as follows:\n", + "\n", + "> These programs seek to improve the care of patients with complex health needs by providing additional resources, including greater attention from trained providers, to help ensure that care is well coordinated. Most health systems use these programs as the cornerstone of population health management efforts, and they are widely considered effective at improving outcomes and satisfaction while reducing costs. [...] Because the programs are themselves expensive—with costs going toward teams of dedicated nurses, extra primary care appointment slots, and other scarce resources—**health systems rely extensively on algorithms to identify patients who will benefit the most.**\n", + "\n", + "**Convenience restriction**\n", + "\n", + "* In practice, the modeling of health needs would use large data sets covering a wide range of diagnoses. In this tutorial, we will work with a [publicly available clinical dataset](https://archive.ics.uci.edu/ml/datasets/Diabetes+130-US+hospitals+for+years+1999-2008) that focuses on _diabetic patients only_ ([Strack et al., 2014](https://www.hindawi.com/journals/bmri/2014/781670/))." + ], + "metadata": { + "id": "HUkG_zdEylGU" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Dataset and task" + ], + "metadata": { + "id": "q-j4KN95wLLS" + } + }, + { + "cell_type": "markdown", + "source": [ + "We will be working with a clincial dataset of hospital re-admissions over a ten-year period (1998-2008) for diabetic patients across 130 different hospitals in the US. Each record represents the hospital admission records for a patient diagnosed with diabetes whose stay lasted one to fourteen days.\n", + "\n", + "The features describing each encounter include demographics, diagnoses, diabetic medications, number of visits in the year preceding the encounter, and payer information, as well as whether the patient was readmitted after release, and whether the readmission occurred within 30 days of the release.\n", + "\n", + "We would like to develop a classification model, which decides whether the patients should be suggested to their primary care physicians for an enrollment into the high-risk care management program. The positive prediction will mean recommendation into the care program.\n", + "\n", + "**Decision point: Task definition**\n", + "\n", + "* A hospital **readmission within 30 days** can be viewed as a proxy that the patients needed more assistance at the release time, so it will be the label we wish to predict.\n", + "\n", + "* Because of the class imbalance, we will be measuring our performance via **balanced accuracy**. Another key performance consideration is how many patients are recommended for care, metric we refer to as **selection rate**.\n", + "\n", + "Ideally, health care professionals would be involved in both designing and using the model, including formalizing the task definition. \n" + ], + "metadata": { + "id": "zOwrRsB7wEeM" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Fairness considerations" + ], + "metadata": { + "id": "2BE26iXWwUqr" + } + }, + { + "cell_type": "markdown", + "source": [ + "* _Which groups are most likely to be disproportionately negatively affected?_ Previous work suggests that groups with different race and ethnicity can be differently affected.\n", + "\n", + "* _What are the harms?_ The key harms here are allocation harms. In particular, false negatives, i.e., don't recommend somebody who will be readmitted.\n", + "\n", + "* _How should we measure those harms?_\n" + ], + "metadata": { + "id": "eZUcQVZYyRvz" + } + }, + { + "cell_type": "markdown", + "source": [ + "In the remainder of the tutorial we will:\n", + "* First examine the dataset and our choice of label with an eye towards a variety of fairness issues.\n", + "* Then train a logistic regression model and assess its performance as well as fairness.\n", + "* Finally, look at two unfairness mitigation strategies." + ], + "metadata": { + "id": "2kD6G-yF1Tcf" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "## Discussion: Fairness-related harms\n", + "\n", + "* How can we determine which type of harm is relevant in a particular scenario?\n", + "* What are ways to find out which (groups of) individuals are most likely to be disproportionately negatively affected?\n" + ], + "metadata": { + "id": "FkTmOAh8Bp1D" + } + }, + { + "cell_type": "markdown", + "source": [ + "# Task definition and dataset characteristics" + ], + "metadata": { + "id": "R7tpRumX_4Nh" + } + }, + { + "cell_type": "markdown", + "source": [ + "Two critical decisions when desiging an AI system are\n", + "1. how we define the machine learning task\n", + "2. what dataset we use to train our models\n", + "\n", + "These choices are often intertwined, because the dataset is often a convenience dataset, based on availability, which leads to a specific choice of label and performance metric (that's also the case in our scenario).\n", + "\n", + "In this part of the tutorial, we first load the dataset, and then we examine it for a variety of fairness issues:\n", + "1. sample sizes of different demographic groups, and in particular different racial groups\n", + "2. appropriateness of our choice of label (readmission within 30 days)\n", + "3. representativeness/informativeness of different features for different groups\n", + "\n", + "Besides dataset characteristics, one additional aspect of dataset fairness is whether the data was collected in a manner that respects the autonomy of individuals in the dataset.\n", + "\n", + "The dataset characteristics can be systematically documented through the **datasheets** practice. We will touch on this later on. By documenting our understanding of the dataset, we communicate any concerns we have about the data and highlight downstream issues that may arise during the model training, evaluation and deployment.\n", + "\n" + ], + "metadata": { + "id": "3iJGhfCgPiy-" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Load the dataset\n" + ], + "metadata": { + "id": "3-ABnntZT8Fn" + } + }, + { + "cell_type": "markdown", + "source": [ + "We next load the dataset and review the meaning of its columns.\n" + ], + "metadata": { + "id": "bvyHqcLIT8Fo" + } + }, + { + "cell_type": "code", + "execution_count": 7, + "source": [ + "df = pd.read_csv(\"https://raw.githubusercontent.com/fairlearn/talks/main/2021_scipy_tutorial/data/diabetic_preprocessed.csv\")" + ], + "outputs": [], + "metadata": { + "id": "gkfwniFQT8Fp" + } + }, + { + "cell_type": "code", + "execution_count": 8, + "source": [ + "df.head()" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
racegenderagedischarge_disposition_idadmission_source_idtime_in_hospitalmedical_specialtynum_lab_proceduresnum_proceduresnum_medications...changediabetesMedmedicaremedicaidhad_emergencyhad_inpatient_dayshad_outpatient_daysreadmittedreadmit_binaryreadmit_30_days
0CaucasianFemale30 years or youngerOtherReferral1Other4101...NoNoFalseFalseFalseFalseFalseNO00
1CaucasianFemale30 years or youngerDischarged to HomeEmergency3Missing59018...ChYesFalseFalseFalseFalseFalse>3010
2AfricanAmericanFemale30 years or youngerDischarged to HomeEmergency2Missing11513...NoYesFalseFalseFalseTrueTrueNO00
3CaucasianMale30-60 yearsDischarged to HomeEmergency2Missing44116...ChYesFalseFalseFalseFalseFalseNO00
4CaucasianMale30-60 yearsDischarged to HomeEmergency1Missing5108...ChYesFalseFalseFalseFalseFalseNO00
\n", + "

5 rows × 25 columns

\n", + "
" + ], + "text/plain": [ + " race gender age discharge_disposition_id \\\n", + "0 Caucasian Female 30 years or younger Other \n", + "1 Caucasian Female 30 years or younger Discharged to Home \n", + "2 AfricanAmerican Female 30 years or younger Discharged to Home \n", + "3 Caucasian Male 30-60 years Discharged to Home \n", + "4 Caucasian Male 30-60 years Discharged to Home \n", + "\n", + " admission_source_id time_in_hospital medical_specialty num_lab_procedures \\\n", + "0 Referral 1 Other 41 \n", + "1 Emergency 3 Missing 59 \n", + "2 Emergency 2 Missing 11 \n", + "3 Emergency 2 Missing 44 \n", + "4 Emergency 1 Missing 51 \n", + "\n", + " num_procedures num_medications ... change diabetesMed medicare medicaid \\\n", + "0 0 1 ... No No False False \n", + "1 0 18 ... Ch Yes False False \n", + "2 5 13 ... No Yes False False \n", + "3 1 16 ... Ch Yes False False \n", + "4 0 8 ... Ch Yes False False \n", + "\n", + " had_emergency had_inpatient_days had_outpatient_days readmitted \\\n", + "0 False False False NO \n", + "1 False False False >30 \n", + "2 False True True NO \n", + "3 False False False NO \n", + "4 False False False NO \n", + "\n", + " readmit_binary readmit_30_days \n", + "0 0 0 \n", + "1 1 0 \n", + "2 0 0 \n", + "3 0 0 \n", + "4 0 0 \n", + "\n", + "[5 rows x 25 columns]" + ] + }, + "metadata": {}, + "execution_count": 8 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 360 }, - { - "cell_type": "markdown", - "metadata": { - "id": "bpyBnqNLZ-w_" - }, - "source": [ - "With the `ThresholdOptimizer`, we took a fairness-unaware model and transformed the model's decision boundary to satisfy our fairness constraints. One limitation of `ThresholdOptimizer` is that it needs access to the sensitive features at deployment time.\n", - "\n", - "In this section, we will show how to use the _reductions_ approach of [Agarwal et. al (2018)](https://arxiv.org/abs/1803.02453) to obtain a model that satisfies the fairness constraints, but does not need access to sensitive features at deployment time.\n", - "\n", - "Terminology \"reductions\" refers to another kind of a wrapper approach, which instead of wrapping an already trained model, wraps any standard classification or regression algorithm, such as \n", - "`LogisticRegression`. In other words, an input to a reduction algorithm is an object that supports training on any provided (weighted) dataset. In addition, a reduction algorithm receives a data set that includes sensitive features. The goal, like with post-processing, is to optimize a performance metric (such as classification accuracy) subject to fairness constraints (such as an upper bound on differences between false negative rates).\n", - "\n", - "The main reduction algorithm algorithm in Fairlearn is `ExponentiatedGradient`. It creates a sequence of reweighted datasets and retrains the wrapped model on each of them. The \n", - "retraining process is guaranteed to find a model that satisfies the fairness constraints while optimizing the performance metric.\n", - "\n", - "The model returned by `ExponentiatedGradient` consists of several inner models, returned by the wrapped estimator. At deployment time, `ExponentiatedGradient` randomizes among these models according to a specific probability weights." - ] + "id": "mIFN96kiT8Fq", + "outputId": "9dbf587e-5f51-4dc4-877a-3ffb9a7374a2" + } + }, + { + "cell_type": "markdown", + "source": [ + "The columns contain mostly boolean and categorical data (including age and various test results), with just the following exceptions: `time_in_hospital`, `num_lab_procedures`, `num_procedures`, `num_medications`, `number_diagnoses`.\n", + "\n", + "\n", + "|features| description|\n", + "|---|---|\n", + "| race, gender, age | demographic features |\n", + "| medicare, medicaid | insurance information |\n", + "| admission_source_id | emergency, referral, or other |\n", + "| had_emergency, had_inpatient_days,
had_outpatient_days | hospital visits in prior year |\n", + "| medical_specialty | admitting physician's specialty |\n", + "| time_in_hospital, num_lab_procedures,
num_procedures, num_medications,
primary_diagnosis, number_diagnoses,
max_glu_serum, A1Cresult, insulin
change, diabetesMed | description of the hospital visit
|\n", + "| discharge_disposition_id | discharched to home or not |\n", + "| readmitted, readmit_binary,
readmit_30_days | readmission information |\n", + "\n", + "\n" + ], + "metadata": { + "id": "KXQNItgRT8Fu" + } + }, + { + "cell_type": "code", + "execution_count": 9, + "source": [ + "# Show the values of all binary and categorical features\n", + "categorical_values = {}\n", + "for col in df:\n", + " if col not in {'time_in_hospital', 'num_lab_procedures',\n", + " 'num_procedures', 'num_medications', 'number_diagnoses'}:\n", + " categorical_values[col] = pd.Series(df[col].value_counts().index.values)\n", + "categorical_values_df = pd.DataFrame(categorical_values).fillna('')\n", + "categorical_values_df.T" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
012345
raceCaucasianAfricanAmericanUnknownHispanicOtherAsian
genderFemaleMaleUnknown/Invalid
ageOver 60 years30-60 years30 years or younger
discharge_disposition_idDischarged to HomeOther
admission_source_idEmergencyReferralOther
medical_specialtyMissingOtherInternalMedicineEmergency/TraumaFamily/GeneralPracticeCardiology
primary_diagnosisOtherRespiratory IssuesDiabetesGenitourinary IssuesMusculoskeletal Issues
max_glu_serumNoneNorm>200>300
A1CresultNone>8Norm>7
insulinNoSteadyDownUp
changeNoCh
diabetesMedYesNo
medicareFalseTrue
medicaidFalseTrue
had_emergencyFalseTrue
had_inpatient_daysFalseTrue
had_outpatient_daysFalseTrue
readmittedNO>30<30
readmit_binary0.0001.000
readmit_30_days0.0001.000
\n", + "
" + ], + "text/plain": [ + " 0 1 \\\n", + "race Caucasian AfricanAmerican \n", + "gender Female Male \n", + "age Over 60 years 30-60 years \n", + "discharge_disposition_id Discharged to Home Other \n", + "admission_source_id Emergency Referral \n", + "medical_specialty Missing Other \n", + "primary_diagnosis Other Respiratory Issues \n", + "max_glu_serum None Norm \n", + "A1Cresult None >8 \n", + "insulin No Steady \n", + "change No Ch \n", + "diabetesMed Yes No \n", + "medicare False True \n", + "medicaid False True \n", + "had_emergency False True \n", + "had_inpatient_days False True \n", + "had_outpatient_days False True \n", + "readmitted NO >30 \n", + "readmit_binary 0.000 1.000 \n", + "readmit_30_days 0.000 1.000 \n", + "\n", + " 2 3 \\\n", + "race Unknown Hispanic \n", + "gender Unknown/Invalid \n", + "age 30 years or younger \n", + "discharge_disposition_id \n", + "admission_source_id Other \n", + "medical_specialty InternalMedicine Emergency/Trauma \n", + "primary_diagnosis Diabetes Genitourinary Issues \n", + "max_glu_serum >200 >300 \n", + "A1Cresult Norm >7 \n", + "insulin Down Up \n", + "change \n", + "diabetesMed \n", + "medicare \n", + "medicaid \n", + "had_emergency \n", + "had_inpatient_days \n", + "had_outpatient_days \n", + "readmitted <30 \n", + "readmit_binary \n", + "readmit_30_days \n", + "\n", + " 4 5 \n", + "race Other Asian \n", + "gender \n", + "age \n", + "discharge_disposition_id \n", + "admission_source_id \n", + "medical_specialty Family/GeneralPractice Cardiology \n", + "primary_diagnosis Musculoskeletal Issues \n", + "max_glu_serum \n", + "A1Cresult \n", + "insulin \n", + "change \n", + "diabetesMed \n", + "medicare \n", + "medicaid \n", + "had_emergency \n", + "had_inpatient_days \n", + "had_outpatient_days \n", + "readmitted \n", + "readmit_binary \n", + "readmit_30_days " + ] + }, + "metadata": {}, + "execution_count": 9 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 738 }, - { - "cell_type": "markdown", - "metadata": { - "id": "ZvT_qeduHCn8" - }, - "source": [ - "To instantiate an `ExponentiatedGradient` model, we pass in two parameters:\n", - "\n", - "* A base `estimator` (an object that supports training)\n", - "* Fairness `constraints` (an object of type `Moment`).\n", - "\n", - "The constraints supported by `ExponentiatedGradient` are more general than those supported by `ThresholdOptimizer`. For example, rather than requiring that false negative rates be equal, it is possible to specify the maxium allowed difference or ratio between the largest and the smallest value.\n" - ] + "id": "gPtaDcfHT8Fv", + "outputId": "42347f1f-d3c8-4221-f98d-fdeaa25ef4fd" + } + }, + { + "cell_type": "markdown", + "source": [ + "We mark all categorical features: " + ], + "metadata": { + "id": "8loEiuFSWb8A" + } + }, + { + "cell_type": "code", + "execution_count": 10, + "source": [ + "categorical_features = [\n", + " \"race\",\n", + " \"gender\",\n", + " \"age\",\n", + " \"discharge_disposition_id\",\n", + " \"admission_source_id\",\n", + " \"medical_specialty\",\n", + " \"primary_diagnosis\",\n", + " \"max_glu_serum\",\n", + " \"A1Cresult\",\n", + " \"insulin\",\n", + " \"change\",\n", + " \"diabetesMed\",\n", + " \"readmitted\"\n", + "]" + ], + "outputs": [], + "metadata": { + "id": "P4y1FRMdWduE" + } + }, + { + "cell_type": "code", + "execution_count": 11, + "source": [ + "for col_name in categorical_features:\r\n", + " df[col_name] = df[col_name].astype(\"category\")" + ], + "outputs": [], + "metadata": { + "id": "Qyo3GQ4RWduG" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Group sample sizes " + ], + "metadata": { + "id": "6LTax67Em4q8" + } + }, + { + "cell_type": "markdown", + "source": [ + "From the perspective of fairness assessment, a key data characteristic is the sample size of groups with respect to which we conduct fairness assessment.\n", + "\n", + "Small sample sizes have two implications:\n", + "\n", + "* **assessment**: the impacts of the AI system on smaller groups are harder to assess, because due to fewer data points we have a much larger uncertainty (error bars) in our estimates\n", + "\n", + "* **model training**: fewer training data points mean that our model fails to appropriately capture any data patterns specific to smaller groups, which means that its predictive performance on these groups could be worse\n", + "\n", + "Let's examine the sample sizes of the groups according to `race`:\n", + "\n" + ], + "metadata": { + "id": "ft28kXKHm4q9" + } + }, + { + "cell_type": "code", + "execution_count": 12, + "source": [ + "df[\"race\"].value_counts()" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "Caucasian 76099\n", + "AfricanAmerican 19210\n", + "Unknown 2273\n", + "Hispanic 2037\n", + "Other 1506\n", + "Asian 641\n", + "Name: race, dtype: int64" + ] + }, + "metadata": {}, + "execution_count": 12 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "ToZdrYen0tJZ" - }, - "outputs": [], - "source": [ - "expgrad_est = ExponentiatedGradient(\n", - " estimator=LogisticRegression(max_iter=1000, random_state=random_seed),\n", - " constraints=TruePositiveRateParity(difference_bound=0.02)\n", - ")" - ] + "id": "wEGJjCrOm4q9", + "outputId": "a7430662-00d7-405b-b9af-1dcaf8fc5c28" + } + }, + { + "cell_type": "code", + "execution_count": 13, + "source": [ + "df[\"race\"].value_counts().plot(kind='bar', rot=45);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAExCAYAAACTeL4rAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA8KElEQVR4nO3de1hU5fr/8TcwiCgkogMS+UUrs8g81JSZBqklEKB5KhVDd6VZO1O3sSPFlNStuVGwdtjOs6AlpELqgIdMzGNoKprWzgOKqDAcEkEHhmH9/vBi/SBLkZRh8H5dV1fysIa57xmYz1rPOtkoiqIghBBCALaWLkAIIUT9IaEghBBCJaEghBBCJaEghBBCJaEghBBCJaEghBBCpbF0AX9VYWEJFRV1c1RtixZO5OcX18lzWYL0Z92kP+tVl73Z2trQvHnTP/2+1YdCRYVSZ6FQ+XwNmfRn3aQ/61VfepPpIyGEECoJBSGEECoJBSGEECoJBSGEECoJBSGEECoJBSGEECoJBSGEECoJBSGEECqrP3mtNpzvcaSxQ+1a12qdb/kxxtJyLhddrdXzCSFEXborQ6Gxg4bgicl19nzr5/bjcp09mxBC1J5MHwkhhFBJKAghhFBJKAghhFBJKAghhFBJKAghhFBJKAghhFBJKAghhFBJKAghhFBJKAghhFBJKAghhFBJKAghhFDd9NpHiYmJxMfHq1+fO3eOfv368fzzzzNr1ixKS0sJCAhgwoQJABw/fpyIiAiKi4vR6XRERkai0Wg4f/48YWFh5Ofn07ZtW6KiomjatClFRUW89957ZGVl4erqSkxMDFqt9s51LIQQ4k/ddEth8ODBJCcnk5ycTFRUFC1atGDUqFFMmjSJ2NhY9Ho9R48eJS0tDYCwsDCmTJnCpk2bUBSFhIQEACIjIxk2bBipqal06NCB2NhYAGJiYtDpdKSkpDB48GBmzpx5B9sVQghxI7c0fTRt2jQmTJhAVlYWXl5etG7dGo1GQ3BwMKmpqWRnZ2M0GuncuTMAAwYMIDU1FZPJRHp6On5+ftXGAbZv305wcDAAQUFB7NixA5PJdBtbFEIIUVM1vnT27t27MRqNBAQEsGHDhmpTPG5ubuTk5JCbm1ttXKvVkpOTQ2FhIU5OTmg0mmrjQLXHaDQanJycKCgowN3dvUZ1tWjhVNMWLKo292GwBGups7akP+vWkPurL73VOBS++uor/va3vwGgKMp137exsbnl8T9ja1vzDZj8/GIqKq7/+TdiiRffYKj/d1TQap2tos7akv6sW0Pury57s7W1ueHKdI0+fcvKykhPT6dXr14AuLu7k5eXp34/NzcXNze368YNBgNubm64urpSXFyM2WyuNg7XtjIqH1NeXk5xcTEuLi631qUQQojbokah8Msvv9CmTRuaNGkCQKdOnTh9+jRnzpzBbDazYcMGfHx88PT0xMHBgQMHDgCQlJSEj48P9vb26HQ69Hp9tXEAX19fkpKSANDr9eh0Ouzt7W93n0IIIWqgRtNHWVlZtGrVSv3awcGB2bNnM3bsWEpLS/H19cXf3x+AqKgoIiIiKCkpwdvbm9DQUACmTp1KeHg4CxYswMPDg3nz5gEwbtw4wsPDCQwMxNnZmaioqNvdoxBCiBqyUf5owt+K1HafQl3fo9ka5kIb8pwtSH/WriH3Z3X7FIQQQtwdJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoJBSEEEKoahQK27ZtY8CAAfj7+zNjxgwAdu/eTXBwMH369CE6Olpd9vjx4wwcOBA/Pz8mT55MeXk5AOfPnyckJAR/f3/eeustSkpKACgqKmL06NEEBAQQEhKCwWC43T0KIYSooZuGQlZWFlOnTiU2Npb169dz7Ngx0tLSmDRpErGxsej1eo4ePUpaWhoAYWFhTJkyhU2bNqEoCgkJCQBERkYybNgwUlNT6dChA7GxsQDExMSg0+lISUlh8ODBzJw58w62K4QQ4kZuGgpbtmzhxRdfpFWrVtjb2xMdHY2joyNeXl60bt0ajUZDcHAwqampZGdnYzQa6dy5MwADBgwgNTUVk8lEeno6fn5+1cYBtm/fTnBwMABBQUHs2LEDk8l0h9oVQghxI5qbLXDmzBns7e15/fXXMRgM9OzZk3bt2qHVatVl3NzcyMnJITc3t9q4VqslJyeHwsJCnJyc0Gg01caBao/RaDQ4OTlRUFCAu7v7bW1UCCHEzd00FMxmM/v37ycuLo4mTZrw9ttv4+joeN1yNjY2KIpyS+N/xta25vu/W7RwqvGylqTVOlu6hBqxljprS/qzbg25v/rS201DoWXLlnTr1g1XV1cAevfuTWpqKnZ2duoyubm5uLm54e7uTl5enjpuMBhwc3PD1dWV4uJizGYzdnZ26jhc28rIy8ujVatWlJeXU1xcjIuLS40byM8vpqLi+tC5EUu8+AbD5Tp/zlul1TpbRZ21Jf1Zt4bcX132Zmtrc8OV6Zuukvfs2ZOdO3dSVFSE2Wzm+++/x9/fn9OnT3PmzBnMZjMbNmzAx8cHT09PHBwcOHDgAABJSUn4+Phgb2+PTqdDr9dXGwfw9fUlKSkJAL1ej06nw97e/q/2LYQQohZuuqXQqVMn3njjDYYNG4bJZKJ79+4MHTqU+++/n7Fjx1JaWoqvry/+/v4AREVFERERQUlJCd7e3oSGhgIwdepUwsPDWbBgAR4eHsybNw+AcePGER4eTmBgIM7OzkRFRd3BdoUQQtyIjfJHE/5WpLbTR8ETk+9QRddbP7efVWz2NuTNc5D+rF1D7s+qpo+EEELcPSQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqCQUhBBCqDQ1WSg0NJT8/Hw0mmuLf/TRR5w9e5YFCxZgMpkYOXIkISEhAOzevZtZs2ZRWlpKQEAAEyZMAOD48eNERERQXFyMTqcjMjISjUbD+fPnCQsLIz8/n7Zt2xIVFUXTpk3vULtCCCFu5KZbCoqicOrUKZKTk9X/WrVqRXR0NKtWrSI5OZnVq1dz4sQJjEYjkyZNIjY2Fr1ez9GjR0lLSwMgLCyMKVOmsGnTJhRFISEhAYDIyEiGDRtGamoqHTp0IDY29s52LIQQ4k/dNBROnTqFjY0No0aNom/fvsTHx7N7926efvppXFxcaNKkCX5+fqSmppKRkYGXlxetW7dGo9EQHBxMamoq2dnZGI1GOnfuDMCAAQNITU3FZDKRnp6On59ftXEhhBCWcdPpo6KiIrp168a0adMwGo2EhoYSEBCAVqtVl3FzcyMjI4Pc3NzrxnNycq4b12q15OTkUFhYiJOTkzotVTl+K1q0cLql5S1Fq3W2dAk1Yi111pb0Z90acn/1pbebhkKXLl3o0qULAE2aNGHQoEHMmjWLMWPGVFvOxsYGRVGue3xtxm9Ffn4xFRXX/5wbscSLbzBcrvPnvFVarbNV1Flb0p91a8j91WVvtrY2N1yZvun00f79+9mzZ4/6taIoeHp6kpeXp47l5ubi5uaGu7t7jcYNBgNubm64urpSXFyM2WyuNi6EEMIybhoKly9fZs6cOZSWllJcXMy6dev497//zZ49eygoKODq1ats3rwZHx8fOnXqxOnTpzlz5gxms5kNGzbg4+ODp6cnDg4OHDhwAICkpCR8fHywt7dHp9Oh1+urjQshhLCMm04f9ezZk8OHD/PSSy9RUVHBsGHDeOKJJ5gwYQKhoaGYTCYGDRpEx44dAZg9ezZjx46ltLQUX19f/P39AYiKiiIiIoKSkhK8vb0JDQ0FYOrUqYSHh7NgwQI8PDyYN2/eHWxXCCHEjdgofzSxb0Vqu08heGLyHaroeuvn9rOKudCGPGcL0p+1a8j9WdU+BSGEEHcPCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBAqCQUhhBCqGofCxx9/THh4OADHjx9n4MCB+Pn5MXnyZMrLywE4f/48ISEh+Pv789Zbb1FSUgJAUVERo0ePJiAggJCQEAwGAwBlZWWEhYUREBBA//79OXny5O3uTwghxC2oUSjs2bOHdevWqV+HhYUxZcoUNm3ahKIoJCQkABAZGcmwYcNITU2lQ4cOxMbGAhATE4NOpyMlJYXBgwczc+ZMAOLi4nB0dCQlJYVJkyapoSOEEMIybhoKv/32G9HR0YwZMwaA7OxsjEYjnTt3BmDAgAGkpqZiMplIT0/Hz8+v2jjA9u3bCQ4OBiAoKIgdO3ZgMpnYvn07ffv2BeDJJ5+ksLCQ8+fP3/YmhRBC1IzmZgt8+OGHTJgwgQsXLgCQm5uLVqtVv6/VasnJyaGwsBAnJyc0Gk218d8/RqPR4OTkREFBwR/+rIsXL3LvvffWuIEWLZxqvKwlabXOli6hRqylztqS/qxbQ+6vvvR2w1BITEzEw8ODbt26sXbtWgAURbluORsbmz8d/zO2tn+8kfJn438mP7+Yiorrn/tGLPHiGwyX6/w5b5VW62wVddaW9GfdGnJ/ddmbra3NDVembxgKer0eg8FAv379uHTpEleuXMHGxoa8vDx1GYPBgJubG66urhQXF2M2m7Gzs1PHAdzc3MjLy6NVq1aUl5dTXFyMi4sLbm5uGAwGvLy8qv0sIYQQlnHD1fKlS5eyYcMGkpOTeffdd+nVqxezZs3CwcGBAwcOAJCUlISPjw/29vbodDr0en21cQBfX1+SkpKAa0Gj0+mwt7fH19eX5ORkAPbv34+Dg8MtTR0JIYS4vWp1nkJUVBSzZs0iICCAq1evEhoaCsDUqVNJSEjgxRdfZP/+/YwfPx6AcePGcejQIQIDA1m1ahUffvghAK+++iplZWUEBgYyc+ZM5syZc3u6EkIIUSs2yh/tDLAitd2nEDwx+Q5VdL31c/tZxVxoQ56zBenP2jXk/urTPgU5o1kIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYRKQkEIIYSqRqEwf/58XnzxRQIDA1m6dCkAu3fvJjg4mD59+hAdHa0ue/z4cQYOHIifnx+TJ0+mvLwcgPPnzxMSEoK/vz9vvfUWJSUlABQVFTF69GgCAgIICQnBYDDc7h6FEELU0E1D4YcffmDv3r188803rFmzhri4OH7++WcmTZpEbGwser2eo0ePkpaWBkBYWBhTpkxh06ZNKIpCQkICAJGRkQwbNozU1FQ6dOhAbGwsADExMeh0OlJSUhg8eDAzZ868g+0KIYS4kZuGwlNPPcWKFSvQaDTk5+djNpspKirCy8uL1q1bo9FoCA4OJjU1lezsbIxGI507dwZgwIABpKamYjKZSE9Px8/Pr9o4wPbt2wkODgYgKCiIHTt2YDKZ7lC7QgghbkRTk4Xs7e355JNPWLJkCf7+/uTm5qLVatXvu7m5kZOTc924VqslJyeHwsJCnJyc0Gg01caBao/RaDQ4OTlRUFCAu7t7jRpo0cKpZp1amFbrbOkSasRa6qwt6c+6NeT+6ktvNQoFgHfffZdRo0YxZswYMjMzr/u+jY0NiqLc0vifsbWt+f7v/PxiKiqu//k3YokX32C4XOfPeau0WmerqLO2pD/r1pD7q8vebG1tbrgyfdNP35MnT3L8+HEAHB0d6dOnD/v27SMvL09dJjc3Fzc3N9zd3auNGwwG3NzccHV1pbi4GLPZXG0crm1lVD6mvLyc4uJiXFxcbr1TIYQQf9lNQ+HcuXNERERQVlZGWVkZ3377LUOGDOH06dOcOXMGs9nMhg0b8PHxwdPTEwcHBw4cOABAUlISPj4+2Nvbo9Pp0Ov11cYBfH19SUpKAkCv16PT6bC3t79D7QohhLiRm04f+fr6cvjwYV566SXs7Ozo06cPgYGBuLq6MnbsWEpLS/H19cXf3x+AqKgoIiIiKCkpwdvbm9DQUACmTp1KeHg4CxYswMPDg3nz5gEwbtw4wsPDCQwMxNnZmaioqDvYrhBCiBuxUf5owt+K1HafQvDE5DtU0fXWz+1nFXOhDXnOFqQ/a9eQ+7OqfQpCCCHuHhIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVBIKQgghVDUKhf/85z8EBgYSGBjInDlzANi9ezfBwcH06dOH6Ohoddnjx48zcOBA/Pz8mDx5MuXl5QCcP3+ekJAQ/P39eeuttygpKQGgqKiI0aNHExAQQEhICAaD4Xb3KIQQooZuGgq7d+9m586drFu3jqSkJH766Sc2bNjApEmTiI2NRa/Xc/ToUdLS0gAICwtjypQpbNq0CUVRSEhIACAyMpJhw4aRmppKhw4diI2NBSAmJgadTkdKSgqDBw9m5syZd7BdIYQQN3LTUNBqtYSHh9OoUSPs7e154IEHyMzMxMvLi9atW6PRaAgODiY1NZXs7GyMRiOdO3cGYMCAAaSmpmIymUhPT8fPz6/aOMD27dsJDg4GICgoiB07dmAyme5Qu0IIIW5Ec7MF2rVrp/47MzMTvV7Pq6++ilarVcfd3NzIyckhNze32rhWqyUnJ4fCwkKcnJzQaDTVxoFqj9FoNDg5OVFQUIC7u3uNGmjRwqlGy1maVuts6RJqxFrqrC3pz7o15P7qS283DYVKv/76K2+++Sbvv/8+Go2G06dPV/u+jY0NiqJc97gbjf8ZW9ua7//Ozy+mouL6n38jlnjxDYbLdf6ct0qrdbaKOmtL+rNuDbm/uuzN1tbmhivTNfr0PXDgACNHjmTixIn0798fd3d38vLy1O/n5ubi5uZ23bjBYMDNzQ1XV1eKi4sxm83VxuHaVkblY8rLyykuLsbFxeWWGxVCCPHX3XRL4cKFC/z9738nOjqabt26AdCpUydOnz7NmTNnuO+++9iwYQMDBw7E09MTBwcHDhw4wBNPPEFSUhI+Pj7Y29uj0+nQ6/UEBwer4wC+vr4kJSUxZswY9Ho9Op0Oe3v7O9t1A+d8jyONHWq8EVhNbbaijKXlXC66WqvnE0LULzf95Fi8eDGlpaXMnj1bHRsyZAizZ89m7NixlJaW4uvri7+/PwBRUVFERERQUlKCt7c3oaGhAEydOpXw8HAWLFiAh4cH8+bNA2DcuHGEh4cTGBiIs7MzUVFRd6LPu0pjBw3BE5Pr7PnWz+1Hw9yoF+LuY6P80YS/FantPoW6/tCsy7nQht5fbTXkOWmQ/qyZ1e1TEEIIcXeQUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGSUBBCCKGqcSgUFxcTFBTEuXPnANi9ezfBwcH06dOH6Ohodbnjx48zcOBA/Pz8mDx5MuXl5QCcP3+ekJAQ/P39eeuttygpKQGgqKiI0aNHExAQQEhICAaD4Xb2J4QQ4hbUKBQOHz7M0KFDyczMBMBoNDJp0iRiY2PR6/UcPXqUtLQ0AMLCwpgyZQqbNm1CURQSEhIAiIyMZNiwYaSmptKhQwdiY2MBiImJQafTkZKSwuDBg5k5c+YdaFMIIURN1CgUEhISmDp1Km5ubgBkZGTg5eVF69at0Wg0BAcHk5qaSnZ2Nkajkc6dOwMwYMAAUlNTMZlMpKen4+fnV20cYPv27QQHBwMQFBTEjh07MJlMt7tPIYQQNaCpyUK/X3vPzc1Fq9WqX7u5uZGTk3PduFarJScnh8LCQpycnNBoNNXGf/+zNBoNTk5OFBQU4O7u/tc6E0IIcctqFAq/pyjKdWM2Nja3PP5nbG1rvv+7RQunGi9rSVqts6VLuKOspT9rqbO2pD/rVV96q1UouLu7k5eXp36dm5uLm5vbdeMGgwE3NzdcXV0pLi7GbDZjZ2enjsO1rYy8vDxatWpFeXk5xcXFuLi41LiW/PxiKiquD50bscSLbzBcrrPnauj91ZZW62wVddaW9Ge96rI3W1ubG65M1+qQ1E6dOnH69GnOnDmD2Wxmw4YN+Pj44OnpiYODAwcOHAAgKSkJHx8f7O3t0el06PX6auMAvr6+JCUlAaDX69HpdNjb29emLCGEEH9RrbYUHBwcmD17NmPHjqW0tBRfX1/8/f0BiIqKIiIigpKSEry9vQkNDQVg6tSphIeHs2DBAjw8PJg3bx4A48aNIzw8nMDAQJydnYmKirpNrQkhhLhVtxQK27ZtU//drVs3vvnmm+uWefjhh/n666+vG/f09CQuLu66cRcXFz7//PNbKUMIIcQdImc0CyGEUEkoCCGEUEkoCCGEUEkoCCGEUEkoCCGEUEkoCCGEUEkoCCGEUEkoCCGEUEkoCCGEUEkoCCGEUEkoCCGEUEkoCCGEUEkoCCGEUEkoCCGEUEkoCCGEUEkoCCGEUNXqzmtCWJLzPY40dqjdr25t7l9tLC3nctHVWj1fbTT0/kT9JqEgrE5jBw3BE5Pr7PnWz+1HXd4uvqH3J+o3CQUhRJ2SLaH6rV6Ewvr161mwYAEmk4mRI0cSEhJi6ZKEEHeIbAnVbxYPhZycHKKjo1m7di2NGjViyJAhdO3alQcffNDSpQkhxC1pCFtBFg+F3bt38/TTT+Pi4gKAn58fqampvPPOOzV6vK2tTa2e1625Y60eV1u1rbO2pL/bS/q7vRpqf40dNLw+Y3OdPBfA4og+lNxibzd7LWwURVH+SlF/1X//+1+uXLnChAkTAEhMTCQjI4Pp06dbsiwhhLgrWfw8hT/KJBubul1rEUIIcY3FQ8Hd3Z28vDz169zcXNzc3CxYkRBC3L0sHgrPPPMMe/bsoaCggKtXr7J582Z8fHwsXZYQQtyVLL6j2d3dnQkTJhAaGorJZGLQoEF07NjR0mUJIcRdyeI7moUQQtQfFp8+EkIIUX9IKAghhFBJKAghhFBJKAghhFBJKAghhFBJKIg6IQe5idupof8+WbI/CQVxxymKol665H//+x+XL1+mtLTUwlX9uYqKiuvGGvqHkDWp+vu0ceNG0tLSLFzR7VW1vzNnzpCTk1Onz2/xk9casqpvbllZGY0aNbJwRXWv6muwcuVKli5dysMPP4xOp2PAgAHcc889Fq6wOkVRsLW9tq60e/du8vPzCQgIQKOx/j+Vyvfixx9/pKioCLPZTO/evS1d1i2r/H3S6/UsWbKE+Ph4C1d0+1RUVKi/f8uXL2flypU89NBD3H///fzjH/+okxpkS+EOqfphmJiYyPvvv09CQgLFxcUWrqxuVb4GW7du5ejRo8TFxdGzZ0/Onj3LmjVrKCoqsnCF1VXWu2rVKqZPn87KlSsZNmxYtetzWSOz2YyNjQ07duzggw8+4ODBg1y8eNHSZdVadnY2mzZtwtHREUfHa5fh/qMtPGtTGQgHDhwgMzOTBQsWEBoayqlTp5g7d27d1FAnz3IXqrp5u2bNGp566ini4uL4+uuvKSgosHB1d17ldIuiKBgMBmJiYiguLsbDw4P+/fvTqVMnsrKyWLlyJZcv16/7Yh0+fJidO3eydu1avvrqK1q3bk1YWBj5+fmWLu2WZWVlkZ+fj52dHSUlJSxZsoSwsDAmTJhASEgIxcXFbNq0ydJl3lTV6buKigo8PDx4+eWXcXFxISYmBrPZjK2trdUGQ2V/FRUV/Prrr4waNQqz2cwDDzxA586def3118nMzGTGjBl3vBYJhTsoIyODxMRE3nvvPYYOHcrUqVPZtm0b33zzTYMOhqpbSVevXkWr1RIREcH+/ftZsWIFtra29OvXj0ceeYRLly5hNpstXPH/l5GRQWpqKllZWezduxeAuXPn4urqyujRo63ufdu2bRvnzp3DZDLRtGlTWrRoQfPmzYFrH0AGg4GEhATKysosXOmNVZ2C/Pe//01UVBTOzs689NJL5OXl8fnnn6vBYG2q/r2Ul5fTrl07pk+fzq5du9izZw+NGjXiscceIzQ0lMLCwjv+O2h9r2A99vu1lCtXrmBjY8PSpUv57bff0Ol0jB8/nuTkZFJSUqx2reZGqv6Cx8XF8fbbb7NkyRLatm3Lp59+yqpVq4iLiwNg8ODBvPPOO+pd9yztxIkTbNu2jRdeeIEePXqwZ88e0tPTgWvB0L59e4xGo4WrvDUjRoxQt84KCgpo3749kydPxmg0YmtrS15eHhUVFVaxIz0xMZGUlBRGjhxJYmIi6enpdO/eHX9/f3799VcWL15s6RJvWdW/lxUrVjB+/Hiio6Px9fVl/PjxTJkyhT179qDRaOjSpQv/+te/cHV1vaM1Wf/es3qi6g7K7777jkuXLuHv788//vEP1q5dy4IFC3j77bd5/PHHmTZtGi1btrTKtZqbqbofRa/XExoaSkxMDJcuXWLQoEHMnDmTd955B41Gw9ChQ3FycrJwxdcoisI999zD0aNHefzxxxkzZgxffPEF27Zto7y8nG7duvGvf/3L0mXWWNUPGzc3N7p27crbb7/NsmXLKCwsZMiQIXTr1o3du3czbtw4HBwcLFzxn1MUBbPZzLFjx5g0aRI7duygc+fO9O/fn2XLljFs2DBsbW2t8r7ule9RfHw8qampDBkyhNWrV5OXl0d4eDi2tra8++67fPbZZzz11FN1csCDXCX1L6r6xwfX3tzVq1fj4uKC2WwmJiaGs2fPkpKSQllZGWFhYfXuiJvbSVEUcnNzmTRpEpGRkVRUVPDxxx/j4ODAvffeS9++fSktLaVZs2b83//9n6XLBSApKYmsrCyGDh1KXl4ekydPZv78+ZjNZlasWIGTkxNvvfUWDg4OVnVXwPT0dA4ePEi3bt1o27YtX3zxBXv27GH58uWcPHmS/Px8mjVrRpcuXa77Pa5PzGYzdnZ2xMTEcOjQIRwdHfnss8+wtbUlMDCQpUuXWt2Nuf73v/+Rn59Pt27d+O233/jwww+ZM2cOp0+f5vPPP6dVq1ZcvXqViRMnkp6eTrt27fDy8qqT2hreqmodMxgM6r937tzJxo0bSUpK4vXXX+fMmTNERETQtm1bevbsiYuLS72fu62NqusVNjY2uLu707VrV7Kzs9m4cSPvvvsuffv2JSkpibVr1/Lggw9aNBB+vx6UnZ1NbGwsb775Jj/99BMDBgwgPT0dLy8vBg8eTGhoKI0bN663H5p/JD09ncjISI4cOUJMTAzfffcdr7/+Ot27d6d///60bNmS5557ji5dugD16xa4hYWF6nuk1+uJi4vj9OnT9O7dm8zMTIYOHUp5eTmbNm1Co9HU662cP2IymThy5Ajt27cnLy+PZs2aYWdnx6FDh/j2228ZNmwYHTt2ZN++fbz33ns899xzdRYIIFsKtaYoCoWFhbzxxhvEx8fTpEkTDh48yM6dO3nhhRfYsmULAwcOZPz48QD885//5JFHHqFp06aWLfw2q7qGmZCQwC+//ELz5s0ZNGgQV69eJTw8nOXLl7Nv3z7i4uKYNWsWWq3WwlVf8+2336rH6S9btgyALVu2cOXKFQoLC9myZQv29vYWrLB2Tpw4wcyZMwkLC8Pb25u4uDiOHDmCr68v3bt3Z+nSpfTo0YMnn3zS0qVeJzMzkyVLltC3b19OnTrFwoULefbZZ1m9ejXJyclkZmby5ZdfAmA0GpkyZQoPP/ywhauuucq/F7PZTFZWFp9++il9+/ale/fu5OXl8Y9//INVq1axZcsWvv/+e/7+97/j7u5epzXKPoVasrGxwdXVlVWrVrFz506Ki4t58cUXcXBw4MyZM1y5coV7772XPn36sH37dry8vBpcIED14/rXrVtHSEgI//d//0erVq3IyMigXbt2/Pvf/+bgwYPMmTPHooFQNcAqKir48ssviYuLY8SIEVy6dIk2bdqwaNEitm/fzqpVq8jJyeG+++6zWL21de7cOX799VdSUlLw9vbm1VdfZeXKlWzatIny8nLGjh2LRqOpd1NGV65cwcPDA0dHR5KTkzEYDCxatAgvLy8efPBBXnrpJdatW8fixYs5f/48Dg4OtGjRwtJl11jV17u0tJSWLVvy4IMPsn37dkwmE4888giKojB58mQyMjKYO3dunQcCyJZCrfz+j2nr1q288847fPLJJ/Tp04ePPvoIT09PWrVqxerVq5k5cyatW7e2YMW3X0ZGBr/++isDBw4EYMqUKbz44ot069YNgJKSEqKjo7nvvvu4cuUKffr0seiOwKrv2YEDB1AUhbZt26LX6zGbzezdu5eDBw8yYcIEhgwZgslkspqthMrefv31V2xsbGjdujW7d+9m2bJl9OrVixEjRgDXzpDt2rVrvVyzNhgMbNy4keDgYC5dusTXX3/N3r17GTZsGH379qVRo0Z8+eWXREZGEh8fj06ns3TJtZaQkMAPP/xAr1696NGjB1999RXZ2dl06dKFVq1asXfvXoKDg3nggQcsUp9sKdyiqh8uW7duxcHBgZ49e7Jw4UJGjx5NbGwsffv25dNPP6WkpIRp06Y1uECoPK+gR48enDx5kgceeICSkhK+++67aqFgNBoJCQmpFx+uVY9zT0hI4MEHHyQ7O5shQ4bQsWNHHnroIc6ePcv3339PUFBQvTkqqiZsbGzYunUrs2fPpnHjxvTp04fBgwfz2muvER8fT1lZGaNGjVLDoT6ys7MjJyeH8ePH07ZtW8LCwnBwcODYsWN4eHjw9NNPM3ToUBwcHNTzLKzR2rVrWbduHR9++KF6UmFAQAApKSmkp6cTFBSkTjlbioTCLap6CFlSUhL+/v54enry7LPPsmLFCl577TXmz5/P4sWLuXLlCk2aNLFwxbeXoijY2dnRsWNHCgoKmD59Oi+88AL//Oc/CQkJwcXFhbfffpt9+/bxyy+/UFJSUm/OQzh69Chr1qxh4cKFuLm5odfr0ev1eHp68swzz/Dpp5/SrFkzqwoEuLbVtnjxYhITE9m8eTMxMTE0bdqU4OBghgwZQnx8PAEBAXh6etar6SK4dvZ4dnY2RqORy5cvc/LkSTp16sSVK1fUlawtW7ZQVlaGj48PAwYMsHTJt6S0tBR7e3tsbW0xGo0cPHiQ119/nbKyMnbu3Mn7779Pz549GTx4MA4ODhbbOqhKjj6qhaysLDZu3MgXX3xB165d+eGHHxg/fjwmk4kvvviCDz74gOLiYvWaLA1J1X0IMTExjBo1im3btvHtt9+yfPly9Ho9EydOZOnSpcyaNcuigfD7mVEXFxcefPBBWrRoQXl5OS+++CJeXl58+eWXKIrCAw88QMuWLS1Ube3l5OTw6KOPYjQaOX/+PO+99x6rV6/miy++wMnJiblz53LffffVu0BIS0tjypQpZGVlodfr+eWXXygqKuLy5cskJSWRk5PDxIkTATh48GC9vrLuH9m7dy8vv/wy8+fPJysri8aNG9OrVy+mTJnCxx9/TLt27Vi5ciUFBQVoNBpGjBhRLw6tlS2FGqh65UKA5s2b07ZtW6ZOnUpRURFeXl48+uijrF69mvnz57Nt27YGuVO5UkJCAikpKcyePRtPT09sbW1ZuHAhTZo0YcOGDZSUlFBaWnrHz7y8karTfOfPn8fOzg5XV1d++ukn1q9fz0svvQRAmzZtrO7M8sreSkpKaNq0Ka1bt6Z58+YcO3YMBwcHBg4cyKlTp8jIyGD48OEWfR/+zKFDh5g5cyZz5syhc+fOjBo1CoPBwOzZs9m0aRM9e/Zk8+bNNG/enB49etC5c2er2+pu2rQpjRo14sKFC4SEhBAUFMQzzzzD5s2badq0KZcvX+Z///sfeXl59eo9klC4iapnKuv1es6ePUv//v0JDAwkJyeHp556ivvuu4+tW7eyf/9+ysrKrO6X92YqP4Qq/3/x4kXS09O5cuUKAJ06dWL06NHMnTuXS5cuMXLkSIuGYtVAWLp0KWvXrqVp06YMHz6cTz75hOHDh3P8+HFsbW3ZvXs3c+bMqXdr0TdiY2PDtm3bSEpKAqBv37507tyZr776iiZNmpCRkUFGRgbvvvsubdq0sWitf+b06dO88MILdO7cWb2svLu7O+PHj8fW1pYOHTpw8eJFvvvuO+bMmWOVW3Du7u44Ojry8ssvM3ToUI4ePcobb7xBYGAghYWFnDp1ijZt2vDRRx/h4eFh6XJVEgo3UXW6ZPXq1Xh7ezNkyBAWLFhA9+7dWbJkCefOnePHH39kzpw5De6eCVU/YE+fPo2npyfvvvsudnZ2DB8+nI0bN9KyZUsee+wx3n//fTw9PS1c8f9/zw4fPkxGRgb/+c9/uHDhAlOnTmXcuHEkJyezZcsWioqKiImJoW3bthau+NYcOXKETz75hEmTJvHDDz+wc+dOMjMzMRqNnDp1ivj4eCIiIurleQiVv0+ZmZkUFhYCYG9vr463atWKs2fP0rt3b1555RVMJpPVTsO6ubnRr18/IiMjWb9+PRUVFbi6uqLRaPjtt99QFIU5c+bUiymjahRxUwcOHFBefvll5fLly8r+/fsVf39/JTAwUDlx4oSSlJSkbN68WTl16pSly7yjli9frrz22mtKZGSkEh4eriiKosyYMUN55plnlIsXLyqKoigVFRWWLFExm83qv3/++WclKChImTp1qjq2f/9+JSAgQFm8eLEFqrs9MjMzlYkTJyqTJ09Wx1JSUpTRo0crubm5yrlz55TTp08rimL59+NG9uzZo/Tp00dJT09XFOXae2cymRRFUZTZs2cr33//vSXLu23y8/OVSZMmKR9//LHy/PPPK9u3b1e/l5uba8HK/pzsaP4Dyu92ULZo0YIePXpw9uxZdu3aRVxcHA8//DCvv/46zZo144UXXrC6tc1bkZaWRkpKCgsWLCAvL4/y8nKuXr3K5MmTeeGFFxg+fHi9uPx15TTfpk2buO+++wgKCuLkyZPs378fk8nEE088wbRp09Dr9dUupVDfVdZZUVFB48aNad68OVlZWRw8eBAAf39/4NqtGz09PdUpo/o8Jda5c2d69erFl19+yb59+7C1tUWj0bBx40Z27dpVb6e9bpWrqyuurq4sWbKEKVOm4Ovri8lkAqg3Z/b/nkwf/Y5SZbokLS0No9FIkyZN8Pf3Jzc3l7y8PFq2bMmTTz6JwWColycC/VWVr0HlDvbi4mL8/f1JSkqipKSE2bNns3TpUtq2bcu0adMwGAzY2dlZrN7jx4+TlJTEBx98gNFoJD4+Hh8fH958802MRiMLFy5k1KhRdOzYkaeeeoqVK1dazfVyKt+Dffv2sWPHDnr27Mno0aNZuXIlu3btwmAw0KZNGzIzM3F2drZ0uTXWuHFjRowYQXx8POHh4Tz22GM0a9aMgwcPMm/ePKs8k/z3Kv+Oxo4dy7Fjx9TLrteH83ZuRLYUfqcyEBYtWsQnn3xCamoqjo6OtGvXjv379+Ps7KyegDJ9+nRatWpl4Ypvr6qhWHnr0LZt2xIfH09iYiKLFy+mSZMmHD9+XN3RbOmdgGVlZZw8eZLo6GgcHBxo2rQpp06dAmDcuHE89dRTzJ07l59++gnAKgKh8sKJtra27Nq1i8jISNq1a8eFCxfQarUEBwdjMpn47LPPmDZtGh9++CHt27e3qiOpWrVqxbhx45g/fz7PPPMMvXv35r///S8PPfSQpUu7LSoPzrC1taV58+ZkZGRYxdapbCn8gby8PNLS0oiPj1d3cpWVleHs7MyPP/7IiRMnmDZtWr259PPtVPXM37Vr1+Ln58fjjz9Ov379yM3N5euvv0aj0ZCZmclTTz1V7TF1rTLAvL29GTduHAsWLCAyMpLc3FxSU1PZsGEDAQEBBAUFYTabrSbAi4qKWLFiBaGhoTRt2hS9Xs8HH3xAp06dSExMpG/fvvTu3Ztnn30WRVEoKipStxKs7R4d9vb2dOzYkY4dO1q6lDvCxsYGjUbD22+/jZ2dXb2e0qsk1z76nUuXLtGoUSOCgoKYM2cOTzzxBOXl5axevZqSkhJGjx6tHkLXkFQ9F+PgwYMsXLiQAQMGsGHDBry9vdXN+W+++YbmzZszYsQIi06d/X6LxsnJiYyMDJYtW4ZerycmJobt27djMBgoLCxUr2Rb3505c4ZFixYxffp0cnJyMBqNHDlyhBkzZuDu7k6fPn1o27YtycnJ/POf/8TR0ZFly5bh4ODAW2+9ZRU9ivpNthSqOHDgAHq9nhEjRvC3v/2Nr7/+GkdHR7y9vbGxseHkyZNWdaG0mlKqnIuRlpZGRkYGXbt25fnnn8fd3Z1Vq1ZRXl5Onz59+Pzzzy36Gvx+HSY+Pp7NmzfzyCOPMHz4cEaMGEF+fj4///wzs2fPBqCgoMBqPixtbW1JTEzE1dUVDw8P9uzZw7vvvsuKFSvQarU0bdoUo9HI0qVLMZlMPPDAA4SEhODs7Gw1PYr6zbq2NW+z33/AVB4VsGbNGu655x4eeeQR/v73vzNjxgyWLl3KqFGjsLe3t4pNwFtR9Raa06dP5+zZs8yaNYucnBz1huE///wzmzdvtngoZmVlYWNjg42NDXq9ni1btjB27FiOHz9OfHw899xzDxMnTuTIkSPMnTsXoN5ce+lmTCYTrVu3ZufOnaxatYpDhw7Ro0cPFi1ahKIoZGdnM3DgQEJCQnjjjTd4+OGHURQFLy+venVGrLBuMn0EfP/99zz77LMA7N+/n7S0NGxtbenbty9Xr16lsLCQtm3bNogjIv7MoUOHiIqKYtasWbRu3ZpZs2axdu1akpOTuffee9Wb51jqRBtFUTAajXTt2pU33niD0NBQJk6cyPPPP8/QoUMpKChg6tSpeHp6MnDgQEpLS9FqtRa5Hv2tqpz+Ki8vV+/B++qrr7J//3569+6Nr68vhw8fpl+/fnh6emI0Grn//vvr3f0QRMNwV24pVOagoiiUlZXxn//8hzFjxgCg0+no0aMH+/btY9myZTRp0oRnn322QQdCWVkZmZmZXLp0Sb2r1QcffMCgQYPo1asXFy9epH379hY/89LR0ZG1a9eycuVKPvjgA7p27cr69evJyMjA1dWV6dOn8/PPP7N+/Xrat29vFYFw5coVQkND+eqrr9BoNJjNZsaMGUPXrl3ZuXMnP/zwA7t27eLRRx8lMTGRli1bcv/99wP1+zwEYb3uui2FqmtX58+f59577yUvL48PP/wQjUbDJ598AsD06dNp0qQJf/vb3xrcpnnlB01oaCgODg44OTlRXFzM7t272bp1K+3bt+f1118HIDo6mn79+qkfRJZUuYP/zJkzDBo0CB8fH55++ml+/PFHQkJC6NChA5cuXeLq1atWc6QRXDvZLjo6mjfffJOdO3ei1WoJDw8H4NSpUwQFBfHcc88xZcqUenWNHNEw3VWhUDUQli1bxpo1a+jSpQv9+/fnvvvuY8qUKVy5cgU/Pz/WrFnDZ5991uD+CEtLS/nnP//Jli1bGDFiBBcuXGDkyJE89thj2NnZsWnTJrZv3869997L2LFjLV0u6enpZGdnq1c1rQyGzMxMXnnlFXx9fenSpQt79uzh7bffttqTCbdt28YHH3zAQw89RFxcHHDtHsSNGzcmMzOTc+fO0aNHDwtXKe4Gd9X0UWUgJCUl8e233xITE8PRo0eJj48nMzOTefPm8dBDD3HkyBFmzZrV4AIBrp24NXLkSDp06ECbNm3w8vLi/fffZ8KECaxfv57u3buj0+kwGo3qBcssqaioiI8//pj169cD0KhRI8rKymjTpg0LFy7k9OnT6kEB1nxHrl69evHxxx9z8eJF9Ho9cO2s38pee/ToYRUnPgnrd1ccknrkyBGKiorIyspi4MCBfP/994wZMwZHR0c6dOgAwJIlS3jllVeIiIiotsOvoVEUhYceeoiOHTvSuHFjJkyYgIODA4sWLWLPnj0kJCTwyCOPMHbs2Hpx2YTevXtjZ2fH9OnTMZvNvPTSS2g0GsrLy3nggQdo27Yt3bp1w9/f36KX2rgdnnvuOSoqKvj444+5cuUKgwYNqnY+jOxDEHWhYX7yVZGWlsbcuXPp2bMnR48e5bnnnqNr167Y2dmRmppK//798fLyYsiQIXz33Xd06dKFZs2aWbrs26rqtJmNjQ1NmzbF1dWVhIQEWrVqRXJyMrGxsTz55JN8/fXX+Pr61otAqFT5YTl79mxsbGzo168ftra2fPvtt5w6dUq9RWhD0KtXL8xmMzNnzuTZZ5/Fzc1NwkDUrTt8FVaLOnTokBIUFKQcPnxYURRFvTSvoijKb7/9pvTv31/55ZdflF27dinDhw9XLwHdkFS9fHJ2drZiMBjUrwcPHqw89thjyubNm/9w+fpm+/btSteuXZV58+Ypn376qfr+NUR5eXmWLkHcpRr0juY1a9Zw4cIF3nnnnWqXpsjPz2fXrl0cO3aMoqIiDh8+TExMDO3atbNwxbdX1UtXLFu2jMTERO6//34ee+wxRo8ezfLly/n+++9ZtGgRgFUc937s2DE2b96Mvb09/v7+9eJG50I0JA1y+qjyw+3kyZPqlT6rnoWr0WhYsGABzz33HL169WLUqFEN8n4IlYFw6NAhTpw4wdy5c7l48aJ6HaChQ4fy+eefs2HDBoKCgup9IAB4e3vj7e1t6TKEaLAaZChUfrj16NGDyMhI9u/fj06no6KigoqKCpo1a0b37t0JDAxUdzQ3JJWhqCgKP/30E2+88QaDBw/m4Ycfpk2bNjRq1IjFixdz+fJlJk2aRJcuXSxdshCinmjQh6R26dKFXr168dVXX5Genq7e3SklJYUDBw40uJPS4PopoA4dOjB+/Hi+/fZbjh07RuPGjdHpdISGhnLixAmefvrpenFfZSFE/dCg9ykA5OTksHLlSlJTU3n88cdxcnJi3759zJs3r8HtQ6gaCPHx8Rw8eJD27dvz2muvER8fT0JCAlFRUXh7e1NWVobZbLbam6ILIe6MBh8KcO0s2KNHj7J//360Wi1PPPFEg7xBTqW4uDg2btzIgAEDWLduHR07duS9997jyy+/5L///S+LFy+22jN/hRB3VoPcp/B7jRo14vHHH+fxxx+3dCl3nMFgYO/evaxcuZKffvqJFi1aUFpaSnR0NOPHj8fR0VGuuy+E+FN3RSg0ZL/fh3DPPfdQUVHBwYMH2bt3L6+++iqZmZksXLiQc+fOMX/+fKs4ykgIYRkSClasaiAkJiZy4cIFnJyc+OSTTzh79iyHDh3inXfeIT8/Hx8fH958800JBCHEDTXoo48ausrdQQkJCaxYsYJOnTrh4eGBvb09ZWVlXL58mRkzZhAbG8vQoUOt4v4CQgjLuit2NDc0aWlptGzZkkcffZSysjLee+89XnrpJXr16gXAhQsXWLx4Mb169WLLli0MHz5czvwVQtSITB9ZodLSUpydnSkoKMDV1ZXmzZtjMBjU77do0YITJ04QERHBM888Y8FKhRDWRqaPrIjZbAagT58+mM1mnnnmGTIyMvDz8yM+Pp69e/dSXFzMd999h9Fo5MqVK3INfiHELZHpIytReRcugOXLl3P//fdz6tQpoqOjWbt2LadOnWLVqlU4OzuTk5NDZGQk7du3t3DVQghrI9NHVmDbtm3s2bOHyZMns3XrVrZu3cpHH33Es88+i0ajoW/fvqxbt44lS5aQlZWFg4MDbm5uli5bCGGFJBTqueLiYpYuXcrAgQPJycnhyy+/xGQycenSJUwmEyEhIdjZ2REcHEx8fDw6nc7SJQshrJiEQj2n0WjQ6XSsX7+effv2MWvWLObNm8e6detwcXFR7xrXqFEjq75HsRCifpAdzfVUaWkpZrOZxo0bc++99/LLL79gZ2eHo6MjERERFBUVERcXp96OcsCAAXLYqRDiL5NQqIf27t3Lyy+/zPz58zl//jxPPPEEixYtwt7entjYWC5dusSMGTM4e/YsX3/9NeXl5ZYuWQjRQEgo1ENNmzalUaNGXLhwgZCQEBITEykuLua1117jt99+45tvviEvL4/58+czcuTIaneVE0KIv0JCoR5yd3fH0dGRV155hXnz5uHh4cGrr77K8uXL0Wg0HDlyhO+++45GjRrJpSuEELeVhEI95ObmRr9+/YiMjKRLly48+uijuLi4UFhYSHZ2NgcOHCAgIACNRo4TEELcXnLyWj1VUFDA3LlzadasGVu2bCEiIgJfX18AcnNz5TwEIcQdIaua9ZSrqyuurq4sXLiQL774Ah8fH0wmE/b29hIIQog7RkKhHqq8T8LYsWM5duwYRqMRQHYoCyHuONmnUA/Z2NigKAq2trY0b96cjIwMubCdEKJOyD6Feu7UqVPY2dnh5eVl6VKEEHcBCQUhhBAqmT4SQgihklAQQgihklAQQgihklAQQgihklAQQgihklAQQgihklAQQgih+n+sj6K93/jN/gAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 330 }, - { - "cell_type": "markdown", - "metadata": { - "id": "bLFk3SCxrskU" - }, - "source": [ - "The constraints above are expressed for the true positive parity, they require that the difference between the largest and the smallest true positive rate (TPR) across all groups be at most 0.02. Since false negative rate (FNR) is equal to 1-TPR, this is equivalent to requiring that the difference between the largest and smallest FNR be at most 0.02." - ] + "id": "HznSDLWEm4q-", + "outputId": "009b8aea-9ae4-4810-eb13-67ac8efae163" + } + }, + { + "cell_type": "markdown", + "source": [ + "Normalized as frequencies:" + ], + "metadata": { + "id": "ZkI6KA3rm4q-" + } + }, + { + "cell_type": "code", + "execution_count": 14, + "source": [ + "df[\"race\"].value_counts(normalize=True)" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "Caucasian 0.748\n", + "AfricanAmerican 0.189\n", + "Unknown 0.022\n", + "Hispanic 0.020\n", + "Other 0.015\n", + "Asian 0.006\n", + "Name: race, dtype: float64" + ] + }, + "metadata": {}, + "execution_count": 14 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 81 - }, - "id": "UFSF5Wn-3M-H", - "outputId": "43f639cf-58ef-4425-b772-3daec9d9ff3d" - }, - "outputs": [], - "source": [ - "# Fit the exponentiated gradient model\n", - "expgrad_est.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" - ] + "id": "qE2UoDU3m4q-", + "outputId": "c23f07b1-493d-4df2-db5a-06bc921eefde" + } + }, + { + "cell_type": "markdown", + "source": [ + "In our dataset, our patients are predominantly *Caucasian* (75%). The next largest racial group is *AfricanAmerican*, making up 19% of the patients. The remaining race categories (including *Unknown*) compose only 6% of the data." + ], + "metadata": { + "id": "DEUrg6J3m4q_" + } + }, + { + "cell_type": "markdown", + "source": [ + "We also examine the dataset composition by `gender`:" + ], + "metadata": { + "id": "xIdDhgtAm4rA" + } + }, + { + "cell_type": "code", + "execution_count": 15, + "source": [ + "df[\"gender\"].value_counts() # counts" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "Female 54708\n", + "Male 47055\n", + "Unknown/Invalid 3\n", + "Name: gender, dtype: int64" + ] + }, + "metadata": {}, + "execution_count": 15 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "markdown", - "metadata": { - "id": "VsCeOlKFDQZZ" - }, - "source": [ - "Similarly to `ThresholdOptimizer` the predictions of `ExponentiatedGradient` models are randomized. If we want to assure reproducible results, we can pass `random_state` to the `predict` function. " - ] + "id": "0h26CH50m4rA", + "outputId": "20d290a6-c8a7-4d21-c676-2a6f552bc3a6" + } + }, + { + "cell_type": "code", + "execution_count": 16, + "source": [ + "df[\"gender\"].value_counts(normalize=True) # frequencies" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "Female 0.538\n", + "Male 0.462\n", + "Unknown/Invalid 0.000\n", + "Name: gender, dtype: float64" + ] + }, + "metadata": {}, + "execution_count": 16 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 203 - }, - "id": "YYz7GAqf4cbp", - "outputId": "72957bdb-41c5-41c6-b9c5-7f0ba8fdf610" - }, - "outputs": [], - "source": [ - "# Record and evaluate predictions on test data\n", - "\n", - "Y_pred_reductions = expgrad_est.predict(X_test, random_state=random_seed)\n", - "metricframe_reductions = MetricFrame(\n", - " metrics=metrics_dict,\n", - " y_true=Y_test,\n", - " y_pred=Y_pred_reductions,\n", - " sensitive_features=A_test\n", - ")\n", - "metricframe_reductions.by_group" - ] + "id": "UTbI5v48m4rA", + "outputId": "e69992ee-8b33-4aef-a903-b51eb63094e8" + } + }, + { + "cell_type": "markdown", + "source": [ + "Gender is in our case effectively binary (and we have no further information how it was operationalized), with both *Female* represented at 54% and *Male* represented at 46%. There are only 3 samples annotated as *Unknown/Invalid*." + ], + "metadata": { + "id": "i3xBANdpm4rA" + } + }, + { + "cell_type": "markdown", + "source": [ + "### Decision point: How do we address smaller group sizes?" + ], + "metadata": { + "id": "NyLBXpKWm4rA" + } + }, + { + "cell_type": "markdown", + "source": [ + "When the data set lacks coverage of certain groups, it means that we will not be able to reliably assess any fairness-related issues. There are three interventions (which could be carried out in a combination):\n", + "\n", + "* **collect more data**: collect more data for groups with fewer samples\n", + "* **buckets**: merge some of the groups\n", + "* **drop small groups**\n", + "\n", + "The choice of strategy depends on our existing understanding of which groups are at the greatest risk of a harm. In particular, pooling the groups with widely different risks could mask the extent of harms. We generally caution against dropping small groups as this leads to the representational harm of erasure.\n", + "\n", + "If any groups are merged or dropped, these decisions should be annotated / explained (in the datasheet, which we discuss below).\n", + "\n", + "In our case, we will:\n", + "\n", + "* merge the three smallest race groups *Asian*, *Hispanic*, *Other* (similar to [Strack et al., 2014](https://www.hindawi.com/journals/bmri/2014/781670/)), but also retain the original groups for auxiliary assessments\n", + "\n", + "* drop the gender group *Unknown/Invalid*, because the sample size is so small that no meaningful fairness assessment is possible" + ], + "metadata": { + "id": "wT-BPQ2Gm4rB" + } + }, + { + "cell_type": "code", + "execution_count": 17, + "source": [ + "# drop gender group Unknown/Invalid\n", + "df = df.query(\"gender != 'Unknown/Invalid'\")\n", + "\n", + "# retain the original race as race_all, and merge Asian+Hispanic+Other \n", + "df[\"race_all\"] = df[\"race\"]\n", + "df[\"race\"] = df[\"race\"].replace({\"Asian\": \"Other\", \"Hispanic\": \"Other\"})" + ], + "outputs": [], + "metadata": { + "id": "iBN_fIhpm4rB" + } + }, + { + "cell_type": "markdown", + "source": [ + "### Exercise" + ], + "metadata": { + "id": "N_Sb8ISAnRQF" + } + }, + { + "cell_type": "markdown", + "source": [ + "Please examine the distribution of the `age` feature in the dataset." + ], + "metadata": { + "id": "ns1Tr9wLm4rB" + } + }, + { + "cell_type": "markdown", + "source": [ + "### Answer" + ], + "metadata": { + "id": "MDiLp-cDoWGv" + } + }, + { + "cell_type": "code", + "execution_count": 18, + "source": [ + "df[\"age\"].value_counts()" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "Over 60 years 68538\n", + "30-60 years 30716\n", + "30 years or younger 2509\n", + "Name: age, dtype: int64" + ] + }, + "metadata": {}, + "execution_count": 18 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "idYvm9lq4mh3", - "outputId": "0848b976-8651-45ae-ddb3-b906b9417ac5" - }, - "outputs": [], - "source": [ - "# Evaluate the difference between the largest and smallest value of each metric\n", - "metricframe_reductions.difference()" - ] + "id": "UOn4j1o8m4rB", + "outputId": "577b1636-57f5-4ab1-903a-c6a42f8193ba" + } + }, + { + "cell_type": "code", + "execution_count": 19, + "source": [ + "df[\"age\"].value_counts().plot(kind='bar', rot=0);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAD7CAYAAACFfIhNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmv0lEQVR4nO3de1zUdaL/8dfAIF6gTJtBMtbT2exs6gl8NMe0C3PwwkUYKcwuUFTb1TY1K04s4oVObm6S2GVxT9tuW+52dulUcCQcayvtgq7oOZu5WcfKvJEDKC4OCQzD5/eHj9/3sYTKeCXx/Xw8fDycz3y+8/1chnl/v58vfMdmjDGIiIgAYT3dABER+f5QKIiIiEWhICIiFoWCiIhYFAoiImJRKIiIiEWhICIiFntPN+BENTY209HRO//UYvDgKPbu9fd0M+Q4af7OXL157sLCbJx33oAjPt9tKLz66qv87ne/sx7v2rWLzMxMJk6cyBNPPEFraytpaWnMnj0bgC1btlBYWIjf78flclFUVITdbqe2tpa8vDz27t3LRRddRHFxMQMGDKCpqYlHHnmEnTt3MmjQIJYuXYrD4Qi5gx0dpteGAtCr+3Y20Pyduc7Wuet2+WjatGlUVFRQUVFBcXExgwcP5u6776agoIDS0lKqqqrYvHkza9asASAvL4+5c+eyatUqjDGUlZUBUFRURHZ2Nl6vl1GjRlFaWgrA0qVLcblcrFy5kmnTprFw4cJT2F0RETmaY7qmsGDBAmbPns3OnTsZNmwYcXFx2O12PB4PXq+X3bt309LSQkJCAgBZWVl4vV4CgQA1NTWkpKR0KgdYvXo1Ho8HgIyMDN5//30CgcBJ7KKIiIQq5FCorq6mpaWFtLQ06urqOi3xOJ1OfD5fl3KHw4HP56OxsZGoqCjsdnuncqDTNna7naioKPbt23dSOiciIscm5AvNf/jDH7jjjjsAONw99Gw22zGXH0lYWOgnMIMHR4Vc90zkcET3dBPkBGj+zlxn69yFFAptbW3U1NSwaNEiAGJiYmhoaLCer6urw+l0dimvr6/H6XQyaNAg/H4/wWCQ8PBwqxwOnWU0NDQwZMgQ2tvb8fv9DBw4MOQO7N3r77UXhByOaOrrD/R0M+Q4af7OXL157sLCbEc9mA7pkPzzzz/nH/7hH+jfvz8A8fHxbNu2je3btxMMBqmsrCQxMZGhQ4cSGRnJxo0bASgvLycxMZGIiAhcLhdVVVWdygHcbjfl5eUAVFVV4XK5iIiIOO4Oi4jI8QvpTGHnzp0MGTLEehwZGcmiRYuYMWMGra2tuN1uUlNTASguLqawsJDm5mZGjBhBbm4uAPPnzyc/P59ly5YRGxvLkiVLAJg1axb5+fmkp6cTHR1NcXHxye6jiIiEyHamf8nO6Vw+ij6nH30jz/i/9zuiltZ2DjQd7Olm9Bq9eQmit+vNc9fd8lHv/YQ7BfpG2vE8XNHTzThlVjyVSe/8MRCRUOneRyIiYlEoiIiIRaEgIiIWhYKIiFgUCiIiYlEoiIiIRaEgIiIWhYKIiFgUCiIiYlEoiIiIRaEgIiIWhYKIiFgUCiIiYlEoiIiIRaEgIiIWhYKIiFgUCiIiYlEoiIiIRaEgIiIWhYKIiFhCCoV3332XrKwsUlNTefzxxwGorq7G4/GQnJxMSUmJVXfLli1MnTqVlJQU5syZQ3t7OwC1tbXk5OSQmprK9OnTaW5uBqCpqYl77rmHtLQ0cnJyqK+vP9l9FBGREHUbCjt37mT+/PmUlpayYsUKPv30U9asWUNBQQGlpaVUVVWxefNm1qxZA0BeXh5z585l1apVGGMoKysDoKioiOzsbLxeL6NGjaK0tBSApUuX4nK5WLlyJdOmTWPhwoWnsLsiInI03YbC22+/zeTJkxkyZAgRERGUlJTQr18/hg0bRlxcHHa7HY/Hg9frZffu3bS0tJCQkABAVlYWXq+XQCBATU0NKSkpncoBVq9ejcfjASAjI4P333+fQCBwirorIiJHY++uwvbt24mIiODOO++kvr6epKQkhg8fjsPhsOo4nU58Ph91dXWdyh0OBz6fj8bGRqKiorDb7Z3KgU7b2O12oqKi2LdvHzExMSe1oyIi0r1uQyEYDLJhwwaWL19O//79uf/+++nXr1+XejabDWPMMZUfSVhY6Ne/Bw+OCrmudM/hiO7pJvQqGs8z19k6d92Gwvnnn8+4ceMYNGgQABMmTMDr9RIeHm7Vqaurw+l0EhMTQ0NDg1VeX1+P0+lk0KBB+P1+gsEg4eHhVjkcOstoaGhgyJAhtLe34/f7GThwYMgd2LvXT0dH19A5Fc6GN0l9/YGebkKv4XBEazzPUL157sLCbEc9mO72kDwpKYkPP/yQpqYmgsEgH3zwAampqWzbto3t27cTDAaprKwkMTGRoUOHEhkZycaNGwEoLy8nMTGRiIgIXC4XVVVVncoB3G435eXlAFRVVeFyuYiIiDjRfouIyHHo9kwhPj6eu+66i+zsbAKBAFdddRU333wz//iP/8iMGTNobW3F7XaTmpoKQHFxMYWFhTQ3NzNixAhyc3MBmD9/Pvn5+SxbtozY2FiWLFkCwKxZs8jPzyc9PZ3o6GiKi4tPYXdFRORobOZwC/5nkNO9fOR5uOK07KsnrHgqs9eeMveE3rwE0dv15rk74eUjERE5eygURETEolAQERGLQkFERCwKBRERsSgURETEolAQERGLQkFERCwKBRERsSgURETEolAQERGLQkFERCwKBRERsSgURETEolAQERGLQkFERCwKBRERsSgURETEolAQERGLQkFERCwKBRERsSgURETEYg+lUm5uLnv37sVuP1T9scceY8eOHSxbtoxAIMDtt99OTk4OANXV1TzxxBO0traSlpbG7NmzAdiyZQuFhYX4/X5cLhdFRUXY7XZqa2vJy8tj7969XHTRRRQXFzNgwIBT1F0RETmabs8UjDF89dVXVFRUWP+GDBlCSUkJr7zyChUVFfzxj3/kiy++oKWlhYKCAkpLS6mqqmLz5s2sWbMGgLy8PObOncuqVaswxlBWVgZAUVER2dnZeL1eRo0aRWlp6antsYiIHFG3ofDVV19hs9m4++67mTJlCr/73e+orq5m7NixDBw4kP79+5OSkoLX62XTpk0MGzaMuLg47HY7Ho8Hr9fL7t27aWlpISEhAYCsrCy8Xi+BQICamhpSUlI6lYuISM/odvmoqamJcePGsWDBAlpaWsjNzSUtLQ2Hw2HVcTqdbNq0ibq6ui7lPp+vS7nD4cDn89HY2EhUVJS1LPX/y4/F4MFRx1Rfjs7hiO7pJvQqGs8z19k6d92GwujRoxk9ejQA/fv35/rrr+eJJ57gvvvu61TPZrNhjOmy/fGUH4u9e/10dHR9nVPhbHiT1Ncf6Okm9BoOR7TG8wzVm+cuLMx21IPpbpePNmzYwNq1a63HxhiGDh1KQ0ODVVZXV4fT6SQmJiak8vr6epxOJ4MGDcLv9xMMBjuVi4hIz+g2FA4cOMCTTz5Ja2srfr+fN954g8WLF7N27Vr27dvHwYMHeeutt0hMTCQ+Pp5t27axfft2gsEglZWVJCYmMnToUCIjI9m4cSMA5eXlJCYmEhERgcvloqqqqlO5iIj0jG6Xj5KSkvj444+59tpr6ejoIDs7m8svv5zZs2eTm5tLIBDg+uuv57LLLgNg0aJFzJgxg9bWVtxuN6mpqQAUFxdTWFhIc3MzI0aMIDc3F4D58+eTn5/PsmXLiI2NZcmSJaewuyIicjQ2c7iF/TPI6b6m4Hm44rTsqyeseCqz166j9oTevC7d2/XmuTvhawoiInL2UCiIiIhFoSAiIhaFgoiIWBQKIiJiUSiIiIhFoSAiIhaFgoiIWBQKIiJiUSiIiIhFoSAiIhaFgoiIWBQKIiJiUSiIiIhFoSAiIhaFgoiIWBQKIiJiUSiIiIhFoSAiIhaFgoiIWBQKIiJiCTkUfv7zn5Ofnw/Ali1bmDp1KikpKcyZM4f29nYAamtrycnJITU1lenTp9Pc3AxAU1MT99xzD2lpaeTk5FBfXw9AW1sbeXl5pKWlcd111/Hll1+e7P6JiMgxCCkU1q5dyxtvvGE9zsvLY+7cuaxatQpjDGVlZQAUFRWRnZ2N1+tl1KhRlJaWArB06VJcLhcrV65k2rRpLFy4EIDly5fTr18/Vq5cSUFBgRU6IiLSM7oNhf3791NSUsJ9990HwO7du2lpaSEhIQGArKwsvF4vgUCAmpoaUlJSOpUDrF69Go/HA0BGRgbvv/8+gUCA1atXM2XKFAD+5V/+hcbGRmpra096J0VEJDT27irMmzeP2bNn88033wBQV1eHw+Gwnnc4HPh8PhobG4mKisJut3cq/+42drudqKgo9u3bd9jX2rNnDxdccEHIHRg8OCrkutI9hyO6p5vQq2g8z1xn69wdNRReffVVYmNjGTduHK+//joAxpgu9Ww22xHLjyQs7PAnKUcqP5K9e/10dHTd96lwNrxJ6usP9HQTeg2HI1rjeYbqzXMXFmY76sH0UUOhqqqK+vp6MjMz+dvf/sa3336LzWajoaHBqlNfX4/T6WTQoEH4/X6CwSDh4eFWOYDT6aShoYEhQ4bQ3t6O3+9n4MCBOJ1O6uvrGTZsWKfXEhGRnnHUw/IXX3yRyspKKioqmDlzJuPHj+eJJ54gMjKSjRs3AlBeXk5iYiIRERG4XC6qqqo6lQO43W7Ky8uBQ0HjcrmIiIjA7XZTUVEBwIYNG4iMjDympSMRETm5juvvFIqLi3niiSdIS0vj4MGD5ObmAjB//nzKysqYPHkyGzZs4MEHHwRg1qxZ/OUvfyE9PZ1XXnmFefPmAXDrrbfS1tZGeno6Cxcu5Mknnzw5vRIRkeNiM4e7GHAGOd3XFDwPV5yWffWEFU9l9tp11J7Qm9ele7vePHfdXVPQXzSLiIhFoSAiIhaFgoiIWBQKIiJiUSiIiIhFoSAiIhaFgoiIWBQKIiJiUSiIiIhFoSAiIhaFgoiIWBQKIiJiUSiIiIhFoSAiIhaFgoiIWBQKIiJiUSiIiIhFoSAiIhaFgoiIWBQKIiJiUSiIiIglpFB4+umnmTx5Munp6bz44osAVFdX4/F4SE5OpqSkxKq7ZcsWpk6dSkpKCnPmzKG9vR2A2tpacnJySE1NZfr06TQ3NwPQ1NTEPffcQ1paGjk5OdTX15/sPoqISIjs3VVYv34969at47//+79pb29n8uTJjBs3joKCApYvX05sbCz33nsva9aswe12k5eXx+OPP05CQgIFBQWUlZWRnZ1NUVER2dnZpKen84tf/ILS0lLy8vJYunQpLpeL559/nvLychYuXMjSpUtPQ9flbBN9Tj/6Rnb7lj+pHI7o07avltZ2DjQdPG37k96p25+QMWPG8PLLL2O32/H5fASDQZqamhg2bBhxcXEAeDwevF4vF198MS0tLSQkJACQlZXFM888w7Rp06ipqeEXv/iFVX7LLbeQl5fH6tWr+f3vfw9ARkYGjz32GIFAgIiIiFPUZTlb9Y2043m4oqebccqseCqTAz3dCDnjhbR8FBERwTPPPEN6ejrjxo2jrq4Oh8NhPe90OvH5fF3KHQ4HPp+PxsZGoqKisNvtncqBTtvY7XaioqLYt2/fSeugiIiELuRz6ZkzZ3L33Xdz33338fXXX3d53mazYYw5pvIjCQsL/fr34MFRIdeV7p3O5Q45+TR/J8/ZOpbdhsKXX35JW1sbl156Kf369SM5ORmv10t4eLhVp66uDqfTSUxMDA0NDVZ5fX09TqeTQYMG4ff7CQaDhIeHW+Vw6CyjoaGBIUOG0N7ejt/vZ+DAgSF3YO9ePx0dXUPnVDgb3iT19b13AULzJ6FyOKJ77ViGhdmOejDd7SH5rl27KCwspK2tjba2Nt555x1uuukmtm3bxvbt2wkGg1RWVpKYmMjQoUOJjIxk48aNAJSXl5OYmEhERAQul4uqqqpO5QBut5vy8nIAqqqqcLlcup4gItJDuj1TcLvdfPzxx1x77bWEh4eTnJxMeno6gwYNYsaMGbS2tuJ2u0lNTQWguLiYwsJCmpubGTFiBLm5uQDMnz+f/Px8li1bRmxsLEuWLAFg1qxZ5Ofnk56eTnR0NMXFxaewuyIicjQ2c7gF/zPI6V4+6u2/vdJbT5lB8yeh0/KRiIgICgUREfk7CgUREbEoFERExKJQEBERi0JBREQsCgUREbEoFERExKJQEBERi0JBREQsCgUREbEoFERExKJQEBERi0JBREQsCgUREbEoFERExKJQEBERi0JBREQsCgUREbEoFERExKJQEBERS0ih8Nxzz5Genk56ejpPPvkkANXV1Xg8HpKTkykpKbHqbtmyhalTp5KSksKcOXNob28HoLa2lpycHFJTU5k+fTrNzc0ANDU1cc8995CWlkZOTg719fUnu48iIhKibkOhurqaDz/8kDfeeIPy8nL++te/UllZSUFBAaWlpVRVVbF582bWrFkDQF5eHnPnzmXVqlUYYygrKwOgqKiI7OxsvF4vo0aNorS0FIClS5ficrlYuXIl06ZNY+HChaewuyIicjTdhoLD4SA/P58+ffoQERHBD3/4Q77++muGDRtGXFwcdrsdj8eD1+tl9+7dtLS0kJCQAEBWVhZer5dAIEBNTQ0pKSmdygFWr16Nx+MBICMjg/fff59AIHCKuisiIkfTbSgMHz7c+pD/+uuvqaqqwmaz4XA4rDpOpxOfz0ddXV2ncofDgc/no7GxkaioKOx2e6dyoNM2drudqKgo9u3bd9I6KCIiobOHWnHr1q3ce++9PProo9jtdrZt29bpeZvNhjGmy3ZHKz+SsLDQr38PHhwVcl3pnsMR3dNNkBOg+Tt5ztaxDCkUNm7cyMyZMykoKCA9PZ3169fT0NBgPV9XV4fT6SQmJqZTeX19PU6nk0GDBuH3+wkGg4SHh1vlcOgso6GhgSFDhtDe3o7f72fgwIEhd2DvXj8dHV1D51Q4G94k9fUHeroJp4zmT0LlcET32rEMC7Md9WC620Pyb775hp/85CcUFxeTnp4OQHx8PNu2bWP79u0Eg0EqKytJTExk6NChREZGsnHjRgDKy8tJTEwkIiICl8tFVVVVp3IAt9tNeXk5AFVVVbhcLiIiIk6o0yIicny6PVP49a9/TWtrK4sWLbLKbrrpJhYtWsSMGTNobW3F7XaTmpoKQHFxMYWFhTQ3NzNixAhyc3MBmD9/Pvn5+SxbtozY2FiWLFkCwKxZs8jPzyc9PZ3o6GiKi4tPRT9FRCQENnO4Bf8zyOlePvI8XHFa9tUTVjyV2WtPmUHzJ6HT8pGIiAgKBRER+TsKBRERsSgURETEolAQERGLQkFERCwKBRERsSgURETEolAQERGLQkFERCwKBRERsSgURETEolAQERGLQkFERCwKBRERsSgURETEolAQERGLQkFERCwKBRERsSgURETEolAQERFLyKHg9/vJyMhg165dAFRXV+PxeEhOTqakpMSqt2XLFqZOnUpKSgpz5syhvb0dgNraWnJyckhNTWX69Ok0NzcD0NTUxD333ENaWho5OTnU19efzP6JiMgxCCkUPv74Y26++Wa+/vprAFpaWigoKKC0tJSqqio2b97MmjVrAMjLy2Pu3LmsWrUKYwxlZWUAFBUVkZ2djdfrZdSoUZSWlgKwdOlSXC4XK1euZNq0aSxcuPAUdFNEREIRUiiUlZUxf/58nE4nAJs2bWLYsGHExcVht9vxeDx4vV52795NS0sLCQkJAGRlZeH1egkEAtTU1JCSktKpHGD16tV4PB4AMjIyeP/99wkEAie7nyIiEgJ7KJW+e/ReV1eHw+GwHjudTnw+X5dyh8OBz+ejsbGRqKgo7HZ7p/LvvpbdbicqKop9+/YRExNzYj0TEZFjFlIofJcxpkuZzWY75vIjCQsL/fr34MFRIdeV7jkc0T3dBDkBmr+T52wdy+MKhZiYGBoaGqzHdXV1OJ3OLuX19fU4nU4GDRqE3+8nGAwSHh5ulcOhs4yGhgaGDBlCe3s7fr+fgQMHhtyWvXv9dHR0DZ1T4Wx4k9TXH+jpJpwymj8JlcMR3WvHMizMdtSD6eP6ldT4+Hi2bdvG9u3bCQaDVFZWkpiYyNChQ4mMjGTjxo0AlJeXk5iYSEREBC6Xi6qqqk7lAG63m/LycgCqqqpwuVxEREQcT7NEROQEHdeZQmRkJIsWLWLGjBm0trbidrtJTU0FoLi4mMLCQpqbmxkxYgS5ubkAzJ8/n/z8fJYtW0ZsbCxLliwBYNasWeTn55Oenk50dDTFxcUnqWsiInKsbOZwC/5nkNO9fOR5uOK07KsnrHgqs9eeMoPmT0Kn5SMREREUCiIi8ncUCiIiYlEoiIiIRaEgIiIWhYKIiFgUCiIiYlEoiIiIRaEgIiIWhYKIiFgUCiIiYlEoiIiIRaEgIiIWhYKIiFgUCiIiYlEoiIiIRaEgIiIWhYKIiFiO6zuaRUROt+hz+tE38vR9ZDkc0adtXy2t7RxoOnja9nc0CgUROSP0jbT32u/YXvFUJt+Xb4TW8pGIiFgUCiIiYvlehMKKFSuYPHkykyZN4ve//31PN0dE5KzV49cUfD4fJSUlvP766/Tp04ebbrqJK664gosvvrinmyYictbp8VCorq5m7NixDBw4EICUlBS8Xi8PPPBASNuHhdlOYeu6cp7X77Tu73Q73eN5umn+zmy9ef5O19x1t58eD4W6ujocDof12Ol0smnTppC3P++8AaeiWUf068Lk07q/023w4KiebsIppfk7s/Xm+fu+zF2PX1MwxnQps9l699GOiMj3VY+HQkxMDA0NDdbjuro6nE5nD7ZIROTs1eOhcOWVV7J27Vr27dvHwYMHeeutt0hMTOzpZomInJV6/JpCTEwMs2fPJjc3l0AgwPXXX89ll13W080SETkr2czhFvVFROSs1OPLRyIi8v2hUBAREYtCQURELAoFERGxKBSOoLm5maKiIiZNmsSUKVPIzs5m7dq1p2RfZWVlXHfddaSkpPCrX/3KKteNAk+up59+msmTJ5Oens6LL75olVdXV+PxeEhOTqakpOSI27/77rtkZWWRmprK448/fszbn21OdLylhxjpoqOjw9xyyy1m4cKFprW11RhjzF//+ldz1VVXmXXr1p3UfdXU1JiUlBTT1NRk/H6/mTRpktm6davZs2ePSUpKMo2Njaa5udl4PB6zdevWk7rvs8mf//xnc9NNN5lAIGAOHjxokpKSzJdffmkOHjxo3G632bFjhwkEAubHP/6xWb16dZftd+zYYa6++mrzzTffmLa2NnPzzTeb1atXh7z92eZEx1t6To//ncL30fr166mtreXll1+2brkxYsQIpk+fTmlpKeeeey6PPPIIlZWVALz33nv88Y9/5Je//CXPP/88K1euJBgMcvXVV5OXl8fu3bu56667OO+884iMjOS3v/2tta+VK1eSnZ1NdPShr/77zW9+w8CBA3n77be7vVHg0qVL6ejo4KGHHgLgpz/9Kddccw1jxoxh3rx57NmzB5vNxsMPP8yVV16Jz+ejoKCAAwcOUF9fT3p6Oo888givv/46b7zxBvv37ycpKYnhw4fzwgsvEB4ezoUXXsjixYuJjIw8DSN/6owZM4aXX34Zu92Oz+cjGAzSv39/Nm3axLBhw4iLiwPA4/Hg9Xpxu92dtn/77beZPHkyQ4YMAaCkpITIyMiQtn/11VdZt24dTz31FADPPfccffr0IScnh8cee4ytW7cSDAa5++67ycjIwO/3U1BQgM/no66uDpfLxZNPPsn69etZvHgxHR0dDB8+nGuvvZbFixcDcO655/LUU08xaNCgUz6WoTiR8e7J8Xrttdd48cUXsdlsjBw5krlz5zJgwADGjh3LyJEjaWho4L/+67+IiIgAIC8vD5fLxY033gjArbfeyiOPPMI555zDvHnz2L9/P/3792fOnDlcdtll5OfnM2bMGLKysgD4p3/6Jz7//HOeffZZfD4f27dvZ/fu3UybNo3p06cTCASYP38+GzduJCYmBpvNxv33388VV1xxzJ81IevpVPo++tWvfmVmzpzZpfzzzz83o0ePNsYY4/F4zOeff26MMeahhx4yb775plmzZo2ZMWOGaW9vN8Fg0Dz00EOmvLzc7Ny501xyySVm586dXV7zzjvvNCUlJSYnJ8dMmTLFvPzyy8YYY375y1+aJUuWWPXKyspMYWFhp2137NhhkpKSTEdHh2lubjZut9u0traaBx980PzpT38yxhjj8/nMhAkTzIEDB8wLL7xgXn/9dWOMMU1NTWb06NFm79695rXXXjOTJk0ygUDAGGPM+PHjTUNDgzHGmCVLlphPP/30hMbz++Tpp5828fHx5tFHHzUdHR1mxYoV5uGHH7ae/+ijj8wdd9zRZbt58+aZf//3fzc//vGPjcfjMUuWLAl5e7/fb8aNG2f8fr/p6OgwycnJZs+ePWbx4sXmpZdeMsYYc+DAAZOenm527NhhVqxYYUpLS40xxrS2tpqJEyeaTz75xKxbt85cfvnlpqmpyRhjzC233GI+/vhjY4wxL730kvnggw9O7mCdBMcz3j01Xp999pmZOHGi2bdvnzHGmAULFphFixYZY4y55JJLDrtKsHbtWpOdnW2MMWbXrl1m8uTJxhhjpk6dalatWmWMMeZ///d/zb/+67+a1tZW8+ijj5rXXnvN2v6SSy4xxhjzzDPPmOuvv960traahoYGk5CQYP72t7+Zl19+2Tz44IOmo6PD7Nq1y4wePdqsW7fuuD5rQqUzhcOw2WwEg8Eu5YFAwPp/ZmYmb775JnFxcaxfv56f/exnLF26lE2bNllHAS0tLVxwwQVcfvnlDB48mAsvvLDLawaDQf7nf/6H//iP/6C9vZ1bbrmF4cOHh3SjwLi4OIYOHUpNTQ21tbW43W769OlDdXU1X331Fc888wwA7e3t7Ny5kzvvvJN169bx61//mq1btxIIBDh48NCXhY8YMQK7/dDbISkpiZtvvpkJEyaQkpLCpZdeepwj+f0zc+ZM7r77bu677z7Kysro379/lzqHuyFjMBhkw4YNLF++nP79+3P//ffzxhtvWEeMR9t+wIABuN1u3nrrLeLi4oiLiyMmJobq6mpaWlp47bXXAPj222/ZunUrGRkZbNq0id/+9rd89dVX7N+/n2+//RaAiy66yDqrnDBhAg888AATJ05kwoQJXHXVVSc8Pifb8Yx3T41XTU0NSUlJnHfeeQDceOON/PSnP7Wej4+P79L2K664grlz57Jr1y4qKirIzMykubmZHTt2kJx86I6uCQkJnHvuuXz11VdHHasrrriCPn36MHjwYAYOHMiBAwf46KOPuOGGG7DZbAwdOpRx48YBsHbt2mP+rAmVQuEw4uPjWb58OYFAoNMP/V/+8hf++Z//GYCMjAxuu+02fvSjH3H11VcTGRlJMBjktttu44477gCgqamJ8PBwGhsb6du372H3df755zNy5EgGDDh0C/BrrrmGTz75hJiYGDZs2GDVO9KNAqdOnUplZSW1tbXMmDEDgI6ODl566SVr6cnn83H++eezaNEidu7cSUZGBhMnTqS6utoKn79vX2FhIZ999hlr1qwhLy+PBx54gMzMzOMdzu+FL7/8kra2Ni699FL69etHcnIyn3/+OampqYe9IeM777xjher48eM5//zzGTdunLXcMGHCBDZt2sTkyZNDuqHj1KlTWbZsGRdeeKH1g9zR0cHixYsZOXIkAA0NDZx77rksX76cVatWccMNN3DllVfyf//3f4edp9tvv52kpCTee+89Fi9ezKZNm5g+ffpJHrnjc6zj/V09MV4dHR2d2mCMob293Xp8uJ9hm83Gtddey5tvvonX6+WFF17AGNPloM4YQzAYxGazWc/9/UEm0GmJ9v/XCw8P79Iu4Lg+a0Kl3z46DJfLxcUXX8zPfvYza+I2b97MsmXLuP/++4FD92yKjY3l+eefZ8qUKQCMHTuWiooKmpubaW9v5yc/+QmrVq066r6SkpJ4++23aWtro6WlhXXr1jFq1KiQbxSYmprK2rVraWhosI5kxo4dyyuvvALAF198wZQpUzh48CAfffQRd955J2lpaXzzzTf4fL4ub7j29naSk5M577zzuPfee8nMzGTLli0nNqDfA7t27aKwsJC2tjba2tp45513uPzyy4mPj2fbtm1s376dYDBIZWUliYmJTJgwgYqKCioqKpg1axZJSUl8+OGHNDU1EQwG+eCDDxg5cuQRt/8ul8vFnj17+POf/8zEiROBQ/P0n//5n8ChD8cpU6bwzTff8NFHH3HjjTcyZcoUbDYbn3322WE/GKZNm0ZzczO33347t99+O59++umpHcRjcKzj/V09MV5jxozh3XffZf/+/cCh3wq84ooruu1rVlYWf/jDHxgyZAgxMTFERUURFxfHW2+9BRw6mGxoaGD48OEMHDiQL774AoA//elP3b72lVdeSVVVFcYYfD4f69evx2azHddnTah0pnAEzz33HCUlJWRkZBAeHs65557L4sWLO71JMjMzKSkpscrGjx/PZ599xg033EAwGOSaa67huuuuY/fu3Ufcz+TJk9mxYwfXXXcd7e3tZGZmWqeIodwosG/fviQkJHDJJZdYZYWFhcybNw+PxwPAk08+SVRUFPfeey//9m//xjnnnMPgwYMZNWoUu3bt6vR6drudmTNncscdd9C3b1/OOeccfv7znx//QH5PuN1uPv74Y6699lrCw8NJTk4mPT0dgEWLFjFjxgxaW1txu92kpqZ22T4+Pp677rqL7OxsAoEAV111FVOnTiUsLCyk7QEmTZrE/v376dOnDwAPPPAACxYsICMjg2AwSF5eHj/4wQ+47bbbWLBgAb/5zW8YMGAAo0ePZteuXfzgBz/o9HoPPfQQ+fn52O12IiMjKSoqOsmjdvxOdLzh9I/Xj370I+69915uvfVWAoEAI0eODGlMY2NjiY2N5brrrrPKFi9ezIIFC3j22WeJiIjg2WefpU+fPmRnZ/Pggw/i8XgYO3Zspy8YO5wbbriBzz77DI/Hg8Ph4IILLqBv376MGTPmmD9rQnbcVyOkx3V0dJgDBw6YyZMnm7q6up5ujhxBR0eHaW1tNdnZ2Wbz5s093ZzvvTNpvDo6OsyePXvMpEmTrF9fP5nee+898+677xpjDv1yyPjx401jY+NJ38/f0/LRGeyTTz5h/Pjx3HDDDd0ecUjPqa+v56qrriI+Pt5aD5cjO5PGa9WqVWRmZvLQQw9ZZzQn0w9/+EOef/55MjMzueWWW5g5c6Z1rfBU0a2zRUTEojMFERGxKBRERMSiUBAREYtCQURELAoFERGxKBRERMTy/wDkZQeBiVeSGwAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 268 }, - { - "cell_type": "markdown", - "metadata": { - "id": "qpxYOozouVx9" - }, - "source": [ - "While there is a decrease in the false negative rate difference from the unmitigated model, this decrease is not as substantial as with `ThresholdOptimizer`. Note, however, that `ThresholdOptimizer` was able to use the sensitive feature (i.e., race) at deployment time." - ] + "id": "ns7RxP5um4rB", + "outputId": "81364da1-1f2e-4053-ced1-c93394ba125e" + } + }, + { + "cell_type": "markdown", + "source": [ + "As we might expect, most patients admitted into the hospital in our data set belong to the *Over 60 years* category. Although we will not be assessing for age-based fairness-related harms in this tutorial, we will want to document the age imbalance in our dataset." + ], + "metadata": { + "id": "HAXa7MoQnjq4" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Examining the choice of label" + ], + "metadata": { + "id": "pK0LbF_sylTU" + } + }, + { + "cell_type": "markdown", + "source": [ + "Next we dive into the question of whether our choice of label (readmission within 30 days) aligns with our goal (identify patients that would benefit from the care management program).\n", + "\n", + "A framework particularly suited for this analysis is called _measurement modeling_ (see, e.g., [Jacobs and Wallach, 2021](https://arxiv.org/abs/1912.05511)). The goal of measurement modeling is to describe the relationship between what we care about and what we can measure. The thing that we care about is referred to as the _construct_ and what we can observe is referred to as the _measurement_. In our case:\n", + "* **construct** = greatest benefit from the care management program\n", + "* **measurement** = readmission within 30 days (in the absence of such program)\n", + "\n", + "In our case, the **measurement** coincides with the **classification label**.\n", + "\n", + "The act of _operationalizing_ the construct via a specific measurement corresponds to making certain assumptions. In our case, we are making the following assumption: **the greatest benefit from the care management program would go to patients that are** (in the absence of such a program) **most likely to be readmitted into the hospital within 30 days.**" + ], + "metadata": { + "id": "AtgLr5Cs_YhF" + } + }, + { + "cell_type": "markdown", + "source": [ + "### How can we check whether our assumptions apply?" + ], + "metadata": { + "id": "fA19isovvCew" + } + }, + { + "cell_type": "markdown", + "source": [ + "In the terminology of measurement modeling, how do we establish _construct validity_? Following, [Jacobs and Wallach, 2021](https://arxiv.org/abs/1912.05511),\n", + "\n", + "> Establishing construct\n", + "validity means demonstrating, in a variety of ways, that the measurements obtained from measurement model are both meaningful\n", + "and useful:\n", + "> * Does the operationalization capture all relevant aspects\n", + "of the construct purported to be measured?\n", + "> * Do the measurements\n", + "look plausible?\n", + "> * Do they correlate with other measurements of the\n", + "same construct? Or do they vary in ways that suggest that the\n", + "operationalization may be inadvertently capturing aspects of other\n", + "constructs?\n", + "> * Are the measurements predictive of measurements of\n", + "any relevant observable properties (and other unobservable theoretical constructs) thought to be related to the construct, but not incorporated into the operationalization?\n", + "\n", + "We focus on one aspect of construct validity, called _predictive validity_, which refers to the extent\n", + "to which the measurements obtained from a measurement model\n", + "are predictive of measurements of any relevant observable properties \n", + "related to the construct purported to be measured, but not incorporated into the operationalization.\n", + "\n", + "The predictions do not need to be chronological, meaning that we do not necessarily need to be predicting future from the past. Also, the predictions do not need to be causal (going from causes to effects). We just need to ensure that the predicted property is not part of the measurement whose validity we're checking. \n" + ], + "metadata": { + "id": "oY95_kOFO6Wb" + } + }, + { + "cell_type": "markdown", + "source": [ + "### Predictive validity" + ], + "metadata": { + "id": "8IBUG3Sa96LX" + } + }, + { + "cell_type": "markdown", + "source": [ + "We would like to show that our measurement `readmit_30_days` is correlated with patient characteristics that are related to our construct \"benefiting from care management\". One such characteristic is the general patient health, where we expect that patients that are less healthy are more likely to benefit from care management.\n", + "\n", + "While our data does not contain full health records that would enable us to holistically measure general patient health, the data does contain two relevant features: `had_emergency` and `had_inpatient_days`, which indicate whether the patient spent any days in the emergency room or in the hospital (but non-emergency) in the preceding year.\n", + "\n", + "To establish predictive validity, we would like to show that our measurement `readmit_30_days` is predictive of these two observable characteristics." + ], + "metadata": { + "id": "rVrLpuwy98uG" + } + }, + { + "cell_type": "markdown", + "source": [ + "First, let's check the rate at which the patients with different `readmit_30_days` labels were readmitted in the previous year:" + ], + "metadata": { + "id": "BQWxJEN-M6VD" + } + }, + { + "cell_type": "code", + "execution_count": 20, + "source": [ + "sns.pointplot(y=\"had_emergency\", x=\"readmit_30_days\",\n", + " data=df, ci=95, join=False);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEJCAYAAACKWmBmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgqElEQVR4nO3dfVxUZf438M/wKIiK8mNGs0z9KeATrdUKktIaEoIg3IT5wC3SBsVaklOroCSmZZm6Ir1SUnNrM7jFlJc03cqyepfrBrhirqAQhuZTDzM8iPEwCAzn/sNfs84Ch4PMcXj4vF+vXnTOXNc537FdP5zrOuc6CkEQBBAREXXAytIFEBFRz8agICIiUQwKIiISxaAgIiJRDAoiIhLFoCAiIlEMCiIiEmVj6QLkcPNmPVpb+XgIEZEUVlYKDB06sMPP+2RQtLYKDAoiIjPh0BMREYliUBARkSgGBRERiWJQEBGRKAYFERGJYlAQEZEoBgUR9Trnyivxbvo3OFdeaelS+oU++RwFEfVth09+j6vaWjQ2GfDIuP+ydDl9Hq8oiKjXaWxqMflJ8mJQEBGRKAYFERGJYlAQEZEoWSezNRoN0tLS0NzcjOjoaERGRrbbLiEhAV5eXggPD0dVVRV+//vfGz+rra3FzZs3cfbsWTlLJSKiDsgWFFqtFikpKcjKyoKdnR0WLlwILy8vjBs3zqTNunXrkJ+fDy8vLwCAi4sLsrOzAQCtra1YunQp1Gq1XGUSEVEnZBt6ysvLg7e3N5ydneHo6IiAgADk5OSYtNFoNPDz80NgYGC7xzh06BAcHBwQEhIiV5lERNQJ2a4odDodXF1djdtKpRJFRUUmbWJiYgAAZ86cadPfYDAgLS0NaWlpcpVIREQSyBYUgtD2xUEKhUJy/5MnT2LMmDFwd3fv8rldXJy63IeIeg9rayvjT1fXQRaupu+TLShUKhUKCwuN2zqdDkqlUnL/Y8eOISgo6J7OXVVVxzfcEfVhBkOr8WdFRa2Fq+n9rKwUor9gyzZH4ePjg/z8fFRXV0Ov1yM3Nxe+vr6S+//rX//C448/Lld5REQkkWxBoVKpoFarERUVhbCwMAQHB8PT0xOxsbEoLi7utP/169cxfPhwucojIiKJFEJ7kwm9HIeeiPq21bvyob2ph2qoA955cbqly+n1LDb0REQkB93NBtTpmwEA9Y0tuFXfZOGK+j5eURBRr/F/868g68Rl3P3/bhtrBZ6fOxFeE1UWq6u34xUFEfUJZy9W4NB/hAQAtBgE7PmiBDd0dRapqz9gUBBRr5B7+nqHn7W2Cjj+zY37WE3/wqAgol7hys/iz0t8/9Mv96mS/odBQUS9gr2t+F9XA2yt71Ml/Q+Dgoh6hcc8xFd26OxzuncMCiLqFeZ6P4zBA+3a/Wyk60DM9BxxnyvqPxgURNQrDBs8AKsjH4Xnf7uY7J8+aThWLZqKAXayvoetX2NQEFGvoRrmiBXzH4Gr8wAAgKuzA2JDJmKQY/tXGmQeDAoi6nWs/ueVBVbS31xA3cCgICIiUQwKIiISxaAgIiJRDAoiIhLFoCAiIlEMCiIiEsWgICIiUQwKIiISJWtQaDQaBAUFwd/fH+np6R22S0hIQFZWlnFbp9PhhRdeQFhYGBYuXIgbN7jOPBGRpcgWFFqtFikpKcjIyEB2djYyMzNRXl7epk1cXBxycnJM9q9atQqzZs3C4cOHERoaiq1bt8pVJhERdUK2VbTy8vLg7e0NZ2dnAEBAQABycnLw8ssvG9toNBr4+fkZ2wBAdXU1vv32W3z00UcAgGeeeQbTp0+Xq0wiIuqEbFcUOp0Orq6uxm2lUgmtVmvSJiYmBvPnzzfZd/36dTzwwAN4++23MW/ePMTHx8PW1lauMomIqBOyXVEIwn++Ah1QKDpfwaulpQUlJSVYvnw5kpKS8NlnnyExMRH79u2TfG4XF6cu1UpEvYu1tZXxp6vrIAtX0/fJFhQqlQqFhYXGbZ1OB6Wy8zdQubq6YuDAgZg1axYAIDg4GG+99VaXzl1VVYfW1rZBRUR9g8HQavxZUSH+Lm3qnJWVQvQXbNmGnnx8fJCfn4/q6mro9Xrk5ubC19e3036jRo2CSqXCiRMnAABffvklJk2aJFeZRNQL/fqSIr6s6P6QLShUKhXUajWioqIQFhaG4OBgeHp6IjY2FsXFxaJ933//fXz44YcIDg7GJ598grfffluuMomoFwqbOQYeo5wRNnOMpUvpFxRCe5MJvRyHnoiIpLPY0BMREfUNDAoiIhLFoCAiIlEMCiIiEsWgICIiUQwKIiISxaAgIiJRDAoiIhLFoCAiIlEMCiIiEsWgICIiUQwKIiISxaAgIiJRDAoiIhLFoCAiIlEMCiIiEiUpKJYvX468vDy5ayEioh5IUlA8/fTT2LlzJwICArB3717U1NTIXBYREfUUXXoV6qVLl3Do0CH87W9/w29+8xssWbIEnp6ectZ3T/gqVCIi6cz2KtTW1lZcvXoVV65cQUtLC1xcXPDGG29gy5YtHfbRaDQICgqCv78/0tPTO2yXkJCArKws4/bhw4cxY8YMhIaGIjQ0FCkpKVLLJCIiM7OR0iglJQVZWVl46KGHsHjxYqSmpsLW1hYNDQ2YNWsWVq5c2aaPVqs19rOzs8PChQvh5eWFcePGmbRZt24d8vPz4eXlZdxfXFyMxMREBAcHm+ErEhFRd0gKiurqauzZswceHh4m+x0dHfGnP/2p3T55eXnw9vaGs7MzACAgIAA5OTl4+eWXjW00Gg38/PyMbX5VXFyMq1evYvfu3XBzc8PatWsxZMiQLnwtIiIyF0lDTy+99BL2798PALh8+TKWLVuGiooKAMCMGTPa7aPT6eDq6mrcViqV0Gq1Jm1iYmIwf/78Nn1dXV2xfPlyZGdnY8SIEdiwYYO0b0NERGYn6YoiMTERTz31FABg5MiRmDZtGtasWYM9e/Z02Ke9OXKFQiGpqB07dhj/PSYmBrNnz5bU71dikzJERNQ1koLi5s2biIqKAgDY29sjOjoahw8fFu2jUqlQWFho3NbpdFAqlZ2eq7a2FocOHUJ0dDSAO4FjYyOpTCPe9UREJJ1Z7noyGAwmw0aVlZXtXjHczcfHB/n5+aiuroZer0dubi58fX07PZejoyM+/PBDnDt3DgDw6aefwt/fX0qZREQkA0m/qkdHRyMsLAwzZ86EQqFAXl4eVq1aJdpHpVJBrVYjKioKzc3NiIiIgKenJ2JjYxEfH48pU6a028/a2hrbt2/HG2+8gcbGRowePRqbN2/u+jcjIiKzkPzA3bfffouCggJYW1vDy8sLbm5uctd2zzj0REQkXWdDT5KD4ocffsCtW7dMhpwmTZrU/QplwKAgIpKus6CQNPS0ZcsWfPrpp3BxcTHuUygUOH78ePcrJCKiHk1SUBw9ehS5ublQqVRy10NERD2MpLueRowYwZAgIuqnJM1RvP/++2hoaICfnx8GDBhg3M85CiKi3s8sk9m/PpVt0rEHz1EwKIiIpDPbXU+9CYOCiEg6szyZXV9fjw0bNmDp0qWoqalBcnIy6uvrzVYkERH1XJKC4q233sKgQYNQVVUFe3t71NXVITk5We7aiIioB5AUFKWlpVCr1bCxsYGDgwO2bt2K0tJSuWsjIqIeQFJQWFmZNjMYDG32ERFR3yTpgbvf/va32LJlCxobG3Hy5Emkp6ebvLqUiIj6Lkl3PTU3N2P37t346quvYDAYMHPmTCxbtgz29vb3o8Yu411PRETS8fZYIiISZZZFAZ966imT15gqFAo4ODhg/PjxSExMlPTmOiIi6p0kBcXs2bNRX1+PyMhIWFlZ4eDBg6ivr4e7uzuSk5PxwQcfyF0nERFZiKShp/DwcGRlZZnsi4iIwMGDBxESEgKNRiNbgfeCQ09ERNKZ7cnsuro643ZdXR0aGxu7Xx0REfV4koaennnmGTz77LOYM2cOBEFAbm4u5s+fj3379mHs2LFy10hERBYk+fbYgoIC/P3vf4eNjQ2efPJJeHt74/z58xg9ejScnDq+ZLEEDj0REUlnlttjw8LCcPjw4S6fXKPRIC0tDc3NzYiOjkZkZGS77RISEuDl5YXw8HCT/SUlJXj22Wdx/vz5Lp2XQUFEJJ1Z5igGDBiAn3/+uUsn1mq1SElJQUZGBrKzs5GZmYny8vI2beLi4pCTk9Omv16vx4YNG9Dc3Nyl8xIRkXlJmqPQ6/Xw8/PD8OHD4ejoaNwvdrdTXl4evL294ezsDAAICAhATk4OXn75ZZP+fn5+xjZ327RpE6Kjo3H27FmJX4WIiOQgKSiSkpK6fGCdTgdXV1fjtlKpRFFRkUmbmJgYAMCZM2dM9h8/fhyNjY2YM2dOl88LQPQSioiIukZSUEybNg1FRUUoKSlBeHg4Lly4gKlTp4r2aW/q4+6nuztSUVGBtLQ0fPzxx1JKaxfnKIiIpDPLHEVWVhZWr16NDz/8ELW1tVi2bBkOHDgg2kelUqGystK4rdPpJC318dVXX6GmpgaRkZEIDQ0FAISGhpo8x0FERPePpKDYt28fMjMz4eTkBBcXF2RlZeEvf/mLaB8fHx/k5+ejuroaer0eubm58PX17fRc8+fPx7Fjx5CdnY3s7GwAQHZ2do+7BZeIqL+Q/OKiu/+iHjFiBKytrUX7qFQqqNVqREVFISwsDMHBwfD09ERsbCyKi4u7VzUREd03kuYonJ2dUVpaapxj+PzzzzFkyJBO+4WEhCAkJMRk3549e9q027RpU4fHKCsrk1IiERHJRNIDd5cuXcIrr7yCa9euYfDgwbC3t8fOnTvh7u5+P2rsMk5mExFJZ7YXFxkMBly5cgUGgwFjxoyBra2t2Yo0NwYFEZF0ZgmKxsZGHD9+HDU1NSb7O1qSw9IYFERE0pnlDXdxcXH45Zdf8OCDDxr3KRSKHhsURERkPpKCQqvV4siRI5IemCMior5F0u2xbm5uJg/PERFR/yHpimLOnDkIDAyEm5sbbGz+3eWTTz6RrTAiIuoZJAXFli1b8OKLL2LUqFFy10NERD2MpKAYOHAgYmNj5a6FiIh6IElzFLNmzUJ6ejp0Oh1qamqM/xARUd8n6TkKT09PNDU1mXZUKFBaWipbYd3B5yiIiKQz25PZvQmDgohIOrO8j6K1tRV79+5FYmIi6urqsGvXLhgMBrMVSUREPZekoNi8eTPKyspw7tw5CIKAkydP4p133pG7NiIi6gEkBUV+fj42bdoEe3t7DBo0CH/+85/x9ddfy10bERH1AJKCwsbGBlZW/25qZ2dn8uAdERH1XZL+tndzc0N6ejoMBgMuX76Mjz/+GB4eHnLXRkREPYCkK4qkpCRcuHABVVVVWLx4MRoaGrBmzRq5ayMioh6g27fHbtq0CYmJieaqxyx4eywRkXRmuT1WzKlTpzr8TKPRICgoCP7+/khPT++wXUJCArKysozbhYWFCA8PR0hICOLi4nDr1q3ulklERPeo20HR0QWJVqtFSkoKMjIykJ2djczMTJSXl7dpExcXh5ycHJP9q1evxubNm6HRaDBu3Djs3bu3u2USEdE96nZQdPQyo7y8PHh7e8PZ2RmOjo4ICAhoEwgajQZ+fn4IDAw02X/kyBGMGzcOzc3N0Gq1GDx4cHfLJCKieyTbPa46nQ6urq7GbaVSiaKiIpM2MTExAIAzZ86Y7Le1tUVZWRmee+452NjY4NVXX+3SucXG2oiIqGtkC4r2hqS68ipVd3d35OXlYf/+/VCr1di/f7/kvpzMJiKSTvbJ7I7mKFQqlcnrU3U6HZRKZafHu337No4dO2bcnjdvHsrKyrpbJhER3aNuB8Xq1avb3e/j44P8/HxUV1dDr9cjNzcXvr6+nR7PxsYG69evx/nz5wEAR48exaOPPtrdMomI6B6JDj15eHiIDheVlpbCy8ur3c9UKhXUajWioqLQ3NyMiIgIeHp6IjY2FvHx8ZgyZUq7/aytrZGSkoLk5GQYDAaoVCps3LixC1+JiIjMSfSBu5s3b0IQBKSmpmLkyJFYsGABrK2tkZWVhR9//LHHPWj3K85REBFJ1605iqFDh2LYsGE4f/48XnjhBQwZMgROTk6IiorCP//5T7MXS0REPY+kOQq9Xo/Lly8bt8vKytDc3CxbUURE1HNIuj12xYoVWLBgAdzd3dHa2opLly5h69atctdGREQ9gORFAauqqnDmzBkoFAo89thjGDZsmNy13TPOURARSWeW5yiamprwzTffoKGhAfX19fjyyy+RkpJitiKJiKjnkjT0pFarcf36dVRUVGDixIk4d+4cpk2bJndtRETUA0i6oigtLUVWVhb8/PywZs0a7N+/H7W1tXLXRkREPYCkoFAqlbCxscHo0aNx8eJFjBs3Dnq9Xu7aiIioB5AUFI6OjtBoNPDw8MDRo0dRVlaGmpoamUsjIqKeQFJQJCcn49tvv8WMGTNgbW2NJUuW4Pnnn5e7NiIi6gEkBUVVVRUuXLiAefPmoby8HCqVChkZGXLXRkREPYCku57Wrl2LZ599FhMmTOjSOyWIiKj3kxQUdnZ2iI6OlrkUIiLqiSQNPY0dOxbFxcVy10JERD2Q6BIeISEhAID6+nrodDo89NBDsLH590WIRqORv8J7wCU8iIik62wJD9Ghp7Vr15q9ICIi6l0kLwrYm/CKgohIOrMsCkhERP0Xg4KIiETJGhQajQZBQUHw9/dHenp6h+0SEhKQlZVl3D5z5gyeeeYZhIaGYunSpfjhhx/kLJOIiETIFhRarRYpKSnIyMhAdnY2MjMzUV5e3qZNXFwccnJyTPavXLkSGzduRHZ2NkJCQvDWW2/JVSYREXVCtqDIy8uDt7c3nJ2d4ejoiICAgDaBoNFo4Ofnh8DAQOO+pqYmvPLKK/Dw8AAAuLu746effpKrTCIi6oSkJ7PvhU6ng6urq3FbqVSiqKjIpE1MTAyAO0NNv7Kzs0NoaCgAoLW1Fe+//z5mz57dpXOLzd4TEVHXyBYU7d1125V1opqampCYmIiWlha8+OKLXTo3b48lIpLOYrfHqlQqVFZWGrd1Oh2USqWkvvX19YiJiUFLSwvS0tJga2srV5lERNQJ2YLCx8cH+fn5qK6uhl6vR25uLnx9fSX1XblyJR5++GGkpqbCzs5OrhKJiEgC2YaeVCoV1Go1oqKi0NzcjIiICHh6eiI2Nhbx8fGYMmVKu/1KSkpw/PhxjBs3DmFhYQDuzG/s2bNHrlKJiEgEl/AgIurnuIQHERF1C4OCiIhEMSiIiEgUg4KIiEQxKIiISBSDgoiIRDEoiIhIFIOCiIhEMSiIiEgUg4KIiEQxKIiISBSDgoiIRDEoiIhIFIOCiIhEMSiIiEgUg4KIiEQxKIiISBSDgoiIRDEoiIhIlOxBodFoEBQUBH9/f6Snp3fYLiEhAVlZWSb7Dh48iMTERLlLJCIiEbIGhVarRUpKCjIyMpCdnY3MzEyUl5e3aRMXF4ecnBzjvtu3b2Pr1q3YuHGjnOUREZEEsgZFXl4evL294ezsDEdHRwQEBJgEAnDnisPPzw+BgYHGfadPn0ZraytWrlwpZ3lERCSBrEGh0+ng6upq3FYqldBqtSZtYmJiMH/+fJN9M2bMwKpVqzBgwAA5yyMiIgls5Dy4IAht9ikUCjlPCQBwcXGS/RxERP2FrEGhUqlQWFho3NbpdFAqlXKeEgBQVVWH1ta2IUVERG1ZWSlEf8GWdejJx8cH+fn5qK6uhl6vR25uLnx9feU8JRERmZmsQaFSqaBWqxEVFYWwsDAEBwfD09MTsbGxKC4ulvPURERkJgqhvYmEXo5DT0RE0ll06ImIiHo/BgUREYliUFCHzpVX4t30b3CuvNLSpRCRBcl6eyz1bodPfo+r2lo0NhnwyLj/snQ5RGQhvKKgNvS3W/D/vrmBn6vrAQDVvzSiobHFwlURkaXwricyUVGjx5b/cxaVtxpN9g91sscfF/0GI1wGWqgyIpIL73qiLtmtudAmJADgZt1tpB0+3+6yLETUtzEoyOjqz7W49MMvHX5+o6IeF6/X3L+CiKhHYFCQ0Y+V9Z23qWq4D5UQUU/CoCCjgQ62nbZxktCGiPoWBgUZTXh4KAY7dhwEDvY28Bzrch8rIqKegEFBRrY2VvjfT7ujo1eGLPIbD3s76/tbFBFZHG+PpTZKrlRD8/UVlP3PxLWttRWW/a/JfOiOqI/i7bHUZRNHD0NC5KNQDnUAAAwbbM+QIOrHGBTUIQe7Oyu8DLDjSi9E/RmDgjoUNnMMPEY5I2zmGEuXQkQWxDkKIqJ+jnMURETULQwKIiISJWtQaDQaBAUFwd/fH+np6R22S0hIQFZWlnH7xx9/RGRkJObMmYM//OEPqK/vfGkJIiKSh2xBodVqkZKSgoyMDGRnZyMzMxPl5eVt2sTFxSEnJ8dk//r167F48WLk5ORg8uTJ2Llzp1xlEhFRJ2QLiry8PHh7e8PZ2RmOjo4ICAhoEwgajQZ+fn4IDAw07mtubsbp06cREBAAAAgPD2/Tj4iI7h/ZbpDX6XRwdXU1biuVShQVFZm0iYmJAQCcOXPGuO/mzZtwcnKCjc2d0lxdXaHVauUqk4iIOiFbULR3162io0WEzNDvbmK3eRERUdfINvSkUqlQWVlp3NbpdFAqlZ32GzZsGOrq6mAwGAAAFRUVkvoREZE8ZAsKHx8f5Ofno7q6Gnq9Hrm5ufD19e20n62tLR5//HEcOXIEAHD48GFJ/YiISB6yXlGo1WpERUUhLCwMwcHB8PT0RGxsLIqLi0X7rlu3DgcOHEBQUBAKCwuxYsUKucokIqJO9MklPIiIyHz4ZDYREYliUBARkSgGBRERiWJQEBGRKAYFERGJYlAQEZEoBgW1S+oS8USWUldXh+DgYNy4ccPSpfR5DApqQ8oS8USWdO7cOSxatAhXrlyxdCn9AoOC2pCyRDyRJR04cADr1q3jOnD3iWyrx1LvJWWJeCJL2rhxo6VL6Fd4RUFtmGOpdyLqOxgU1Ma9LhFPRH0Tg4LauNcl4omob+IcBbVx9xLxzc3NiIiIgKenp6XLIiIL4TLjREQkikNPREQkikFBRESiGBRERCSKQUFERKIYFEREJIpBQXSXqVOndmk10uLiYsTHxwMAioqKkJyc3Gmf2tpaxMfHIzg4GEFBQdi9e7fxsytXrmDx4sUICgpCREQELl261OXvEBwcjFOnTnW5H1FHGBRE3TBlyhS89957AIDy8nJotdpO+6SmpkKlUuGLL77AwYMHsX//fpw9exYA8Mc//hGLFi3CkSNHsHz5csTHx7e7pArR/cQH7qjXOnXqFDZu3AhHR0c0NDQgPj4eu3btQnNzMwYMGICEhARMnToVlZWVSE5ORlVVFSoqKjBy5Ehs374dLi4uKCwsxJtvvgmFQoEpU6agtbXVeOxt27ZBqVTiu+++g4ODA5YvX459+/bh+++/x9NPP401a9bg1KlTePPNN7Fnzx689957qK2txerVq/HOO+90WHdSUhIMBgMAoKKiAk1NTRg0aBC0Wi0uX76MuXPnAgCefPJJrF+/HiUlJZg0aVKHxysvL8eaNWug1+sxduxYNDQ0GD/74IMPcOzYMdy+fRt6vR4JCQmYPXs25syZg7Vr12LGjBkAgNdffx3jx4/HjBkzkJSUhKamJgiCgIiICERGRnb7vxX1cgJRL1VQUCB4eHgIN27cEL7//nshODhYqK6uFgRBEC5evCg88cQTQn19vfDxxx8Lu3btEgRBEFpbW4WYmBhh7969wu3btwUfHx8hLy9PEARB0Gg0gpubm3D9+nWhoKBAmDBhgnDhwgVBEATh+eefFxYsWCDcvn1bqKqqEiZNmiT8/PPPQkFBgTB37lxBEATh0KFDwgsvvCC5/tdee02YPHmyoFarhZaWFuHs2bNCQECASZuFCxcKx44dEz1OaGiocODAAUEQBKGwsFBwd3cXCgoKhBs3bghLliwR9Hq9IAiC8MUXXwjBwcGCIAjCRx99JMTHxwuCIAi1tbWCt7e3cOvWLWH16tXGPyudTiesWLFCMBgMkr8T9U28oqBebcSIERg5ciTS09Oh0+kQHR1t/EyhUODatWtYunQpCgsL8dFHH+HKlSv47rvv8Mgjj+DixYuwsbHB9OnTAdwZ2797juHBBx/ExIkTAQCjRo3CoEGDYGdnh2HDhmHgwIG4detWt2rfunUr1q9fj/j4eOzYscP42/1/sra27vAYN2/eRFlZGcLCwgAAjz32GMaPHw8AGDlyJN59911oNBpcvXoV586dQ319PQAgPDwcO3bsQHV1NXJycvC73/0OgwcPhr+/PxISElBUVITp06fj9ddfh5UVR6j7O/4vgHo1R0dHAEBrayumT5+O7Oxs4z8HDhzA+PHjsWXLFqSmpmLo0KFYsGABnnjiCQiCAIVC0Wb838bm37872dnZdfhZd5w8edI4lzFw4EDMnTsXJSUleOCBB1BZWWlSk1arxfDhwzs81q/Lv9/d59c6L1y4gIULF6Kurg5PPPEEYmJijG0GDx6MOXPm4PPPP8ehQ4ewaNEiAMCsWbPw17/+FYGBgSgtLUVISAiuXbtmlu9NvReDgvoEb29vfP3118a7hE6cOIF58+bh9u3b+Mc//oGlS5ciLCwMLi4uyMvLg8FggJubGwRBwIkTJwAAx48f79ZVgrW1NVpaWjptd/ToUezYsQOCIKCpqQlHjx6Ft7c3hg8fjlGjRuHIkSMA7gSKlZUV3NzcOjyWs7MzJk2ahM8++wzAnXC4ePEiAOD06dOYPHkynnvuOUybNg3Hjx83zo0AQGRkJD755BMIgmBc9PG1117DkSNHMHfuXKxbtw5OTk746aef7vnPhPoGDj1RnzB+/Hhs2LABr776KgRBgI2NDdLS0uDo6IiXXnoJmzdvxs6dO2FtbY1HH30U165dg62tLXbs2IE33ngD27Ztw4QJE+Di4nLPNUydOhXbt2/HSy+9hB07dnTYLjExEevWrUNISAgUCgX8/PwQFRUFANi2bRvWrl2LtLQ02NnZITU1tdOhn23btmH16tXYv38/Ro0ahbFjxwK4M5SWm5uLoKAg2NraYvr06bh16xbq6urg5OQEDw8PDBkyBAsXLjQea9myZUhKSkJmZiasra0xe/ZsTJs27Z7/TKhv4OqxRP3UtWvXsGTJEuTk5MDBwcHS5VAPxisKIjO7fPky1Gp1u5+NGTMG27dv79LxCgoKOrzd1svLC2vWrOlqiUhNTcWBAweQlJTEkKBO8YqCiIhEcTKbiIhEMSiIiEgUg4KIiEQxKIiISBSDgoiIRDEoiIhI1P8HfVl/V2+A3tkAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 285 }, - { - "cell_type": "markdown", - "metadata": { - "id": "IzTeHhWG4nJJ" - }, - "source": [ - "### Explore individual predictors" - ] + "id": "3t11OgTgZfV8", + "outputId": "d2467bd3-0148-4373-cd41-0b99d9f54088" + } + }, + { + "cell_type": "markdown", + "source": [ + "The plot shows that indeed patients with `readmit_30_days=0` have a lower rate of emergency visits in the prior year, whereas patients with `readmit_30_days=1` have a larger rate. (The vertical lines indicate 95% confidence intervals obtained via boostrapping.)" + ], + "metadata": { + "id": "Ptl-tHkDf_GJ" + } + }, + { + "cell_type": "markdown", + "source": [ + "We see a similar pattern when `readmit_30_days` is used to predict the rate of (non-emergency) hospital visits in the previous year:" + ], + "metadata": { + "id": "07GU8IGIY9KC" + } + }, + { + "cell_type": "code", + "execution_count": 21, + "source": [ + "sns.pointplot(y=\"had_inpatient_days\", x=\"readmit_30_days\",\n", + " data=df, join=False);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEJCAYAAAC61nFHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAphElEQVR4nO3de1xVdb7/8RdXFcXQYKPZOJODaRSajifRHMmSKBAxxQQ9UadCm1NRu0ZFxUo8Phot8TIZR5saR9PMKYOYo4iXtFHQtBq1UvEyeZcNosZVtrB+f/hoz/ATcblli+L7+XjMI75rfdfys5wevVnf71rf5WYYhoGIiMgVcm/sAkRE5MakABEREacoQERExCkKEBERcYoCREREnKIAERERpyhARETEKZ6NXcC1dPp0GTU1eu1FRMQMd3c32rRpecn9N1WA1NQYChARkQaiISwREXGKAkRERJyiABEREacoQERExCkKEBERcYoCREREnKIAEZEmZcf+IqYv+YYd+4sau5Qm76Z6D0REmr6Mv/+TQwUlVFZV0z3Iv7HLadJ0ByIiTUpl1fla/xTXUYCIiIhTXBogWVlZREZGEh4ezpIlSy7a/8477zBgwABiYmKIiYlx9Nm9ezfDhg0jIiKCSZMmcf78hd8kjh8/zqhRo3jkkUf43e9+R1lZmSvLFxGRergsQAoKCpg1axZLly4lMzOTjz/+mP3799fq891335GWlkZmZiaZmZmMGjUKgLFjxzJ58mRWr16NYRgsX74cgClTpjBy5Eiys7O55557ePfdd11VvojcYOzna9i+x0ZZ5YVfOLXsneu5LEByc3MJDQ3Fz88PHx8fIiIiyM7OrtXnu+++47333iM6OprU1FTOnTvHsWPHqKys5N577wVg6NChZGdnY7fb2bZtGxEREbW2i4jkHznDuP/N5d2M7yitsANQdKaCTTtPNHJlTZvLnsKy2WwEBAQ42haLhZ07dzraZWVl3HXXXYwfP54OHTqQnJzMu+++ywMPPFDruICAAAoKCjh9+jStWrXC09Oz1vYrceutra7yqkTkemM7Xc6cT3ZSca72pLkB/Hnlbjr9og3d7wyo+2C5Ki4LEMO4+P7Rzc3N8XPLli157733HO2nn36aiRMnEhYWVudxlzufGadOlWo5d5Em5pMN+y8Kj58ZwEc5e7itTfNrW1QT4e7uVu8v3i4bwgoMDKSo6F8v8thsNiwWi6N9/PhxPvnkE0fbMAw8PT0vOq6wsBCLxULbtm0pLS2lurq61nYRubnlHzlzVfvFeS4LkL59+5KXl0dxcTEVFRXk5OTQv39/x/7mzZvz1ltvceTIEQzDYMmSJYSHh9OhQweaNWvG119/DUBGRgb9+/fHy8uLXr16sXLlylrbReTm5uFe/3/GPNyvbKRCzHMz6hobaiBZWVnMnz8fu91ObGwsiYmJJCYmkpSUREhICKtXr+aPf/wjdrudnj17MmXKFLy9vdmzZw8pKSmUlZURHBzMm2++ibe3N8eOHSM5OZlTp07Rvn170tLSuOWWW0zXoyEskaZn1ZZD/HXDgUvu7x0cyJjBd1/DipqOyw1huTRArjcKEJGmp6zSzhsffMWpn85dtM/b052UhF7cbtEDNM5otDkQEZFroWVzL8aN7EnXjn61tnu4u/HKiHsVHi6kABGRG16AXwvGjezJtMTe+LXyBsD/lubc+Qu/xi2siVOAiEiT0f7WljTz8mjsMm4aChAREXGKAkRERJyiABEREacoQESkSWnu7Vnrn+I6ChARaVKG/PYOunb0Y8hv72jsUpo8vUgoIiJ10ouEIiLiEgoQERFxigJEREScogARERGnKEBERMQpChAREXGKAkRERJzi0lc1s7KySE9Px26389RTTzFq1Kg6+23YsIHU1FTWr18PwNChQx3fPq+srOTIkSN8+eWXVFVVERUVRceOHQHw9/fn/fffd+UliIjIJbgsQAoKCpg1axYrVqzA29ubuLg4evfuTVBQUK1+RUVFTJ8+vda2FStWOH4eN24cjz32GP7+/qxevZro6GhSU1NdVbaIiJjksiGs3NxcQkND8fPzw8fHh4iICLKzsy/ql5KSwgsvvFDnOfLy8tizZw+JiYkA7Nq1i/z8fIYOHUpCQgJ79+51VfkiInIZLgsQm81GQECAo22xWCgoKKjVZ9GiRQQHB9O9e/c6zzF37lysViseHhc+ENOsWTOGDBnCihUreOaZZ3j++eepqqpy1SWIiEg9XDaEVdcSW25ubo6f8/PzycnJYeHChZw8efKivvv27eP06dMMGDDAse3FF190/BwWFsbMmTM5ePAgXbt2NVVTfWu6iIjIlXFZgAQGBrJ9+3ZH22azYbFYHO3s7GwKCwsZNmwYdrsdm83GyJEjWbp0KQBr164lMjKy1jkXL17MoEGDaNOmDXAhpDw9zV+CFlMUETGv0RZT7Nu3L3l5eRQXF1NRUUFOTg79+/d37E9KSmL16tVkZmayYMECLBaLIzwA/vGPf9CrV69a59y2bRuffPIJAF999RU1NTV06tTJVZcgIiL1cOkdiNVqJSEhAbvdTmxsLN26dSMxMZGkpCRCQkLqPf7IkSMEBgbW2jZp0iSSk5PJzMykWbNmzJw5E3d3vcoiItIY9D0QERGpk74HIiIiLqEAERERpyhARETEKQoQERFxigJEREScogARERGnKEBERMQpChAREXGKAkRERJyiABEREacoQERExCkKEBERccoVB0hpaSnHjx93RS0iInIDMRUga9asYerUqZSWljJ48GBiYmL4y1/+4uraRETkOmYqQObPn8/jjz9OTk4O9957L1988QWff/65q2sTEZHrmKkAMQyDLl26kJubS//+/WnVqlWd3zwXEZGbh6kAcXd3Z+XKlWzatIn777+fjRs3urouERG5zpkKkOTkZJYvX47VaiUgIID09HRSUlIue1xWVhaRkZGEh4ezZMmSS/bbsGEDDz74oKO9bds2evfuTUxMDDExMUyYMAGAn376idGjR/Poo48yatQoCgsLzZQvIiIuYOqTtuvXr2fAgAG4ubmZPnFBQQHx8fGsWLECb29v4uLiSEtLIygoqFa/oqIinnjiCc6dO8f69esB+OCDD7Db7YwZM6ZW39TUVNq1a8fo0aPJyMhgw4YNzJ4923RN+qStiIh5DfJJ2w8//JCHHnqId9991/Rv/bm5uYSGhuLn54ePjw8RERFkZ2df1C8lJYUXXnih1rZdu3axefNmhgwZwnPPPceJEyeAC3cq0dHRAAwaNIgvv/wSu91uqh4REWlYnmY6ffDBBxw+fJjly5czfPhwunXrRnx8PH369LnkMTabjYCAAEfbYrGwc+fOWn0WLVpEcHAw3bt3r7Xd19eXqKgoBg4cyEcffYTVamXZsmW1zunp6UmrVq0oLi4mMDDQ1MXWl6QiInJlTAUIQMeOHbFarYSEhDBjxgxeeeUV/P39ef311+nVq9dF/esaGfv3IbD8/HxycnJYuHAhJ0+erNUvNTXV8XN8fDwzZ86kpKSkzrrc3c2/C6khLBER8xpkCOvQoUO89dZbhIWFsXz5ciZOnMjmzZuZMmUK48aNq/OYwMBAioqKHG2bzYbFYnG0s7OzKSwsZNiwYYwePRqbzcbIkSOpqakhPT2d6urqWufz9PTEYrE4znn+/HlKS0vx8/MzcwkiItLATAXI8OHDOX/+PB9++CHvv/8+Dz30EO7u7vTs2ZP77ruvzmP69u1LXl4excXFVFRUkJOTQ//+/R37k5KSWL16NZmZmSxYsACLxcLSpUtxd3dnzZo1rF69GoCMjAy6d+9OixYtCAsLIyMjA4CVK1fSq1cvvLy8rvKvQEREnGHqKazKykqaN29+xSfPyspi/vz52O12YmNjSUxMJDExkaSkJEJCQhz9jh49SkJCguMprH379jF58mRKSkpo27YtM2bMoH379pw5c4bk5GSOHDmCr68vb7/9NrfffrvpejSEJSJi3uWGsEwFyLfffsuCBQsoLy/HMAxqamo4evQoGzZsaMhaXU4BIiJiXoPMgaSkpNCjRw9KS0uJjo6mVatWPPzwww1WpIiI3HhMPYXl5ubG6NGjOX36NJ06dWLw4MHEx8e7ujYREbmOmboDadmyJXDhUd59+/bRrFmzi56SEhGRm4upO5CQkBBefvllXnrpJcaMGcOPP/6Ih4eHq2sTEZHrmKlJdMMw2LFjB/feey8bNmwgNzeXuLg4OnXqdC1qbDCaRBcRMe+qnsK63Kdrb7vtNucrawQKEBER8y4XIPUOYUVFReHm5oZhGFRWVuLj44Onpyc//fQTt956K5s2bWrwgkVE5MZQb4B8++23ALz22mv07t2bqKgoANatW8fatWtdX52IiFy3TD2F9d133znCA+Chhx5iz549LitKRESuf6YCpKamhq1btzraX3755RV9XEpERJoeU4/xpqSk8PLLL+Pl5YVhGBiGwbx581xdm4iIXMdMPcYLYLfbyc/Px83NjTvvvBNPzwvZ88EHH/D000+7tMiGoqewRETMa5C1sAC8vLy4++67CQ4OdoQHXFhxV0REbj7mP+d3CSZvYEREpIm56gDRZLqIyM3pqgNERERuTi4NkKysLCIjIwkPD2fJkiWX7LdhwwYefPBBR/vAgQOMHDmSmJgYRowYwe7du4ELS6v06NGDmJgYYmJieOaZZ1xZvoiI1MPUY7z1udQcSEFBAbNmzWLFihV4e3sTFxdH7969CQoKqtWvqKiI6dOn19qWkpLC6NGjGTBgAHl5eYwfP57PP/+cXbt2ER0dTWpq6tWWLSIiV8nUHcjSpUsv2rZgwQKAS94F5ObmEhoaip+fHz4+PkRERJCdnX1Rv5SUFF544YVa24YPH07//v0B6NKlCydOnABg165d5OfnM3ToUBISEti7d6+Z8kVExAXqvQP56KOPqKysZOHChZw7d86x3W63s3jxYkaPHk10dHSdx9psNgICAhxti8XCzp07a/VZtGgRwcHBdO/evdb2oUOHOn6eO3cuAwcOBKBZs2YMGTKEuLg4Nm7cyPPPP8/KlSvx9vY2ebkiItJQ6g0QT09P8vPzqaysJD8/37Hdw8ODyZMn13viuoa2/v2Jrfz8fHJycli4cCEnT56s8/gZM2awY8cOFi1aBMCLL77o2B8WFsbMmTM5ePAgXbt2rbeWn9X3QoyIiFyZegNk+PDhDB8+nLVr1zruAswKDAxk+/btjrbNZsNisTja2dnZFBYWMmzYMOx2OzabjZEjR7J06VLOnz/P+PHjKSgoYNGiRfj6+gKwePFiBg0aRJs2bYALIfPvLzVejt5EFxEx76q+B/Kz7t27884773DmzJla21NSUi55TN++ffnjH/9IcXExLVq0ICcnh6lTpzr2JyUlkZSUBMDRo0dJSEhwzLVMnz6d0tJSPvjgg1rDU9u2baOyspLExES++uorampqbrivIoqINBWmAsRqteLr60twcLDpFwcDAwOxWq0kJCRgt9uJjY2lW7duJCYmkpSUREhISJ3HFRcXs2TJEm6//XaGDx/u2J6ZmcmkSZNITk4mMzOTZs2aMXPmTNzd9SqLiEhjMLWY4qOPPsqqVauuRT0upSEsERHzGmQxxdtuu43y8vIGK0pERG58poawLBYLQ4YM4b777qN58+aO7fXNgYiISNNmKkA6dOhAhw4dXF2LiIjcQEx/UKqyspJDhw7RuXNnqqqqat2J3Cg0ByIiYl6DzIHs2LGDgQMHMmbMGGw2G2FhYXzzzTcNVqSIiNx4TAXI9OnTWbhwIX5+frRr144ZM2Ywbdo0V9cmIiLXMVMBUllZWWsV3bCwMKqrq11WlIiIXP9MBYinpydnz551vER48OBBlxYlIiLXP1NPYT333HP853/+J0VFRbzyyits3rxZ3+QQEbnJmX4K69ChQ2zevJmamhpCQ0Mv+jDUjUBPYYmImHe5p7DqDZADBw7w61//mu+//77O/XfffffVV3gNKUBERMy7qgAZM2YM8+fPr/W9cseBbm6sW7euYaq8RhQgIiLmXVWA/OzkyZO0a9eu1rZ9+/bRuXPnq6/wGlKAiIiYd1UvEp45c4YzZ84wevRozp49y5kzZzh79ixFRUU8//zzDV6siIjcOOp9CuvVV19l8+bNAPTu3ftfB3l6XvEXCkVEpGkxNYQ1YcIE3nzzzWtRj0tpCEtExLwGWQvrzTff5MyZM5w4cYLjx49z5MgRx51JfbKysoiMjCQ8PJwlS5Zcst+GDRtqTdT/9NNPjB49mkcffZRRo0ZRWFgIQFVVFWPHjuXRRx/lscce48CBA2bKFxERFzD1IuHcuXOZP38+AB4eHtjtdoKCgsjKyrrkMQUFBcyaNYsVK1bg7e1NXFwcvXv3vuj9kaKiIqZPn15r2+zZs+nVqxcLFiwgIyODadOmMXv2bBYvXkyLFi1YtWoV27ZtIzk5mb/+9a9Xes0iItIATN2BZGRk8MUXXxAREUFOTg5/+MMfLvsiYW5uLqGhofj5+eHj40NERATZ2dkX9UtJSeGFF16otW3Dhg1ER0cDMGjQIL788kvsdjsbNmxg8ODBAPzHf/wHp0+f5vjx46YuVEREGpapAGnbti0Wi4VOnTqxZ88eYmJiOHToUL3H2Gw2AgICHG2LxUJBQUGtPosWLSI4OJju3btf8lhPT09atWpFcXHxRecMCAjg5MmTZi5BREQamKkhLE9PTw4fPkynTp3Yvn07/fr146effqr3mLrm5n9ejBEgPz+fnJwcFi5caCoE3N3rzrpLba9LfZNBIiJyZUwFyJgxY5g8eTLp6enMnj2bjIwMHnjggXqPCQwMZPv27Y62zWbDYrE42tnZ2RQWFjJs2DDsdjs2m42RI0eydOlSLBYLRUVFtGvXjvPnz1NaWoqfnx8Wi4XCwkJ++ctfAlBYWFjrnJejp7BERMxrkKewBgwYwF/+8hd8fHzIzMzkT3/6E1OnTq33mL59+5KXl0dxcTEVFRXk5OTQv39/x/6kpCRWr15NZmYmCxYswGKxsHTpUuDC90YyMjIAWLlyJb169cLLy4uwsDAyMzMB2L59O82aNeO2224zcwkiItLATN2BlJWVMW/ePDZt2oSHhwcPPvggnTp1wtvb+5LHBAYGYrVaSUhIwG63ExsbS7du3UhMTCQpKYmQkJBLHvvSSy+RnJxMVFQUvr6+vP322wA88cQTvPbaa0RFReHt7c2MGTOu8HJFRKShmHqR0Gq14u7uTmxsLIZhsHz5clq3bn3DfRNEQ1giIuZdbgjL1B3IDz/8wOrVqx3t0NBQoqKirr46ERG5YZmaA/H396e4uNjRLi8vp02bNi4rSkRErn+m7kD8/f0ZNmwYjzzyCB4eHqxbtw5/f3/+53/+B7jwMqCIiNxcTAVI586da337Q8NXIiJi+pvoTYEm0UVEzGuQSfRVq1bx1ltvcfbs2VpvmH/zzTdXX6GIiNyQTAXI7NmzSU5OJjg4uNZyJCIicvMyFSCtW7fm4YcfdnUtIiJyAzH1GG/37t3ZuHGjq2sREZEbiKlJ9PDwcI4cOYKXlxdeXl4YhoGbm9sNNweiSXQREfMuN4luKkCOHTtW5/YOHTo4X1kjUICIiJh3VU9h5eXl0adPH77//vs6999oASIiIg2n3gD5v//7P/r06cPixYsv2ufm5qaJdRGRm9hVv0j4hz/8geTk5Iaqx6U0hCUiYl6DfFCqPlu3br3aU4iIyA3oqgPkJloJRURE/s1VB4jeTBcRuTmZehPdWVlZWaSnp2O323nqqacYNWpUrf1r1qxh7ty51NTUEBISQmpqKiUlJTz99NOOPiUlJZw+fZpvv/2Wbdu28cILL9CuXTsAgoODefPNN115CSIicgkuC5CCggJmzZrFihUr8Pb2Ji4ujt69exMUFARc+ChVamoqn332Gf7+/litVj777DNGjBhBZmYmADU1NTz55JNYrVYAdu3axdNPP82YMWNcVbaIiJjksjmQ3NxcQkND8fPzw8fHh4iICLKzsx37fXx8WL9+Pf7+/pSXl3Pq1Clat25d6xyffvopLVq0IDo6GrgQIJs3b2bIkCE899xznDhx4mrLFxERJ131HciECRPq3G6z2QgICHC0LRYLO3furNXHy8uLjRs3Mm7cOCwWC/369XPsq66uJj09nfT0dMc2X19foqKiGDhwIB999BFWq5Vly5aZrrW+x9FEROTK1BsgXbt2rXeSfPfu3fTu3bvOfXXdmdR1rrCwMLZu3UpaWhpvvPEGM2fOBODvf/87d9xxB126dHH0TU1NdfwcHx/PzJkzKSkpwdfXt77LcNB7ICIi5l3VeyB5eXls3ryZxx9/HKvVypYtW9i2bRsTJkzgySefrPcPDgwMpKioyNG22WxYLBZH+8yZM2zatMnRjo6OZu/evY722rVriYyMdLRrampIT0+nurq61p/j6enS5wBEROQS6g2QNm3a0LZtW7777jtGjx7NLbfcQqtWrUhISOCrr76q98R9+/YlLy+P4uJiKioqyMnJoX///o79hmEwduxYjh8/Dlz46mHPnj0d+//xj3/Qq1evfxXq7s6aNWtYvXo1ABkZGXTv3p0WLVpc+VWLiMhVM/Xre0VFBQcPHqRTp04A7N27F7vdXu8xgYGBWK1WEhISsNvtxMbG0q1bNxITE0lKSiIkJISpU6cyZswY3NzcCAoKYsqUKY7jjxw54nhc92fTp09n8uTJzJs3j7Zt2zJjxowrvV4REWkgptbCysnJYdKkSXTp0oWamhoOHDjA22+/zW9/+9trUWOD0RyIiIh5DfI9EIBTp07x9ddf4+bmxm9+8xvatm3bYEVeKwoQERHzGmQxxaqqKr755hvKy8spKyvjiy++YNasWQ1WpIiI3HhMzYFYrVaOHDlCYWEhwcHB7Nixg/vuu8/VtYmIyHXM1B3I7t27WbFiBQ899BATJ05k2bJllJSUuLo2ERG5jpkKEIvFgqenJ7/61a/Iz88nKCiIiooKV9cmIiLXMVMB4uPjQ1ZWFl27dmXVqlXs3buXM2fOuLg0ERG5npkKkNdee409e/bQr18/PDw8eOKJJ3jmmWdcXZuIiFzHTAXIqVOn+P777xk8eDD79+8nMDCQpUuXuro2ERG5jpl6Cmvy5Mk8/vjj3HXXXfoCoYiIACYDxNvbm6eeesrFpYiIyI3E1BBWp06d2LVrl6trERGRG0i9S5n8/CXAsrIybDYbv/jFL2otn56VleX6ChuQljIRETHvckuZ1DuENXny5AYvSEREmgbTiyk2BboDERExr0EWUxQREfn/KUBERMQpLg2QrKwsIiMjCQ8PZ8mSJRftX7NmDdHR0URFRZGcnExVVRVw4XO1/fr1IyYmhpiYGMfS8cePH2fUqFE88sgj/O53v6OsrMyV5YuISD1cNgdSUFBAfHw8K1aswNvbm7i4ONLS0ggKCgKgvLyciIgIPvvsM/z9/bFarYSGhjJixAimTp1Kjx49GDRoUK1zjhkzhsGDBxMVFcW8efMoLy9n7NixpmvSHIiIiHmNNgeSm5tLaGgofn5++Pj4EBERQXZ2tmO/j48P69evx9/fn/Lyck6dOkXr1q0B2LVrFxkZGQwePJjf//73nD17FrvdzrZt24iIiABg6NChtc4nIiLXlssCxGazERAQ4GhbLBYKCgpq9fHy8mLjxo0MGDCA06dP069fPwACAgJ48cUXyczMpH379qSmpnL69GlatWrleA8lICDgovOJiMi1Y2opE2fUNTJW1zpaYWFhbN26lbS0NN544w1mzpzJvHnzHPufffZZBg4cyLhx40ydrz713YqJiMiVcVmABAYGsn37dkfbZrNhsVgc7TNnzvDdd9857jqio6OxWq2UlJTw6aefOtbeMgwDT09P2rZtS2lpKdXV1Xh4eFBYWFjrfGZoDkRExLxGmwPp27cveXl5FBcXU1FRQU5ODv3793fsNwyDsWPHcvz4cQBWrVpFz5498fHx4U9/+hM7duwA4MMPPyQ8PBwvLy969erFypUrgQtPav37+URE5Npy6ZvoWVlZzJ8/H7vdTmxsLImJiSQmJpKUlERISAhr165lzpw5uLm5ERQUxJQpU/D19WX79u1MmzaNyspKfvWrXzFjxgx8fX05duwYycnJnDp1ivbt25OWlsYtt9xiuh7dgYiImHe5OxAtZSIiInXSUiYiIuISChAREXGKAkRERJyiABEREacoQERExCkKEBERcYoCREREnKIAERERpyhARETEKQoQERFxigJEREScogARERGnKEBERMQpChAREXGKAkRERJyiABEREacoQERExCkuDZCsrCwiIyMJDw9nyZIlF+1fs2YN0dHRREVFkZycTFVVFQBff/01w4YNIyYmhieffJJjx44BsG3bNnr37k1MTAwxMTFMmDDBleWLiEg9XPZJ24KCAuLj41mxYgXe3t7ExcWRlpZGUFAQAOXl5URERPDZZ5/h7++P1WolNDSUESNG8OCDD/Luu+/StWtXPvnkE9atW0d6ejoffPABdrudMWPGOFWTPmkrImJeo33SNjc3l9DQUPz8/PDx8SEiIoLs7GzHfh8fH9avX4+/vz/l5eWcOnWK1q1bU1VVxUsvvUTXrl0B6NKlCydOnABg165dbN68mSFDhvDcc885touIyLXn6aoT22w2AgICHG2LxcLOnTtr9fHy8mLjxo2MGzcOi8VCv3798Pb2JiYmBoCamhreeecdBg4cCICvry9RUVEMHDiQjz76CKvVyrJly0zXVF+SiojIlXFZgNQ1Mubm5nbRtrCwMLZu3UpaWhpvvPEGM2fOBKCqqork5GTOnz/vGLJKTU11HBcfH8/MmTMpKSnB19fXVE0awhIRMa/RhrACAwMpKipytG02GxaLxdE+c+YMmzZtcrSjo6PZu3cvAGVlZTz77LOcP3+e9PR0vLy8qKmpIT09nerq6lp/jqenyzJQRETq4bIA6du3L3l5eRQXF1NRUUFOTg79+/d37DcMg7Fjx3L8+HEAVq1aRc+ePQEYO3Ysv/zlL5kzZw7e3t4XCnV3Z82aNaxevRqAjIwMunfvTosWLVx1CSIiUg+XPYUFFx7jnT9/Pna7ndjYWBITE0lMTCQpKYmQkBDWrl3LnDlzcHNzIygoiClTpnDkyBEee+wxgoKCHHcXFouF9957j3379jF58mRKSkpo27YtM2bMoH379qbr0RCWiIh5lxvCcmmAXG8UIA3jXFU19uoaWjb3rHNeS0SahssFiCYQxLRDJ0tY8eVBdh08BUBrHy+i77+DB3t2UJCI3IR0ByKmHDh+lhlLv8V+vuaifQN/czsjw+9shKpExJUa7SksaVqWrdtXZ3gArP36KMcKS69xRSLS2BQgcllFZys4cOynevts3W27RtWIyPVCASKXVV553kQf+zWoRESuJwoQuawAvxZ4edb/r8rtAVomRuRmowCRy2rRzJN+IZd+36Zlc096Bwdew4pE5HqgABFTHh8QRNeOfhdtb+7twQtDQ2jRTE+Ei9xs9BivmFZTY/CP/UWs3X6Eo4VldPv1rTw+IIjWLb0buzQRcQG9if5vFCAiIubpPRAREXEJBYiIiDhFASIiIk5RgIiIiFMUICIi4hQFiIiIOOWmevvL3V3frBARMety/828qd4DERGRhqMhLBERcYoCREREnKIAERERpyhARETEKQoQERFxigJEREScogARERGnKEBERMQpChAREXGKAkSuWFZWFpGRkYSHh7NkyZLGLkekltLSUgYNGsTRo0cbu5QmTwEiV6SgoIBZs2axdOlSMjMz+fjjj9m/f39jlyUCwI4dO4iPj+fHH39s7FJuCgoQuSK5ubmEhobi5+eHj48PERERZGdnN3ZZIgAsX76c119/HYvF0til3BRuqtV45erZbDYCAgIcbYvFws6dOxuxIpF/mTZtWmOXcFPRHYhckboWb3Zz0zL5IjcjBYhckcDAQIqKihxtm82m4QKRm5QCRK5I3759ycvLo7i4mIqKCnJycujfv39jlyUijUBzIHJFAgMDsVqtJCQkYLfbiY2NpVu3bo1dlog0An2RUEREnKIhLBERcYoCREREnKIAERERpyhARETEKQoQERFxigJExIQePXpc0equu3btIikpCYCdO3fy2muvXfaYkpISkpKSGDRoEJGRkSxYsMCx78cff2TkyJFERkYSGxvLgQMHrvgaBg0axNatW6/4OJFLUYCIuEBISAhz584FYP/+/RQUFFz2mDlz5hAYGMjf/vY3PvnkE5YtW8a3334LwO9//3vi4+NZuXIlL774IklJSXUuKyNyLelFQmlytm7dyrRp0/Dx8aG8vJykpCTmz5+P3W6nefPmjB8/nh49elBUVMRrr73GqVOnKCwspEOHDsyePZtbb72V7du3M3XqVNzc3AgJCaGmpsZx7rS0NCwWC/v27aNFixa8+OKLLF68mH/+8588/PDDTJw4ka1btzJ16lTee+895s6dS0lJCRMmTODNN9+8ZN2TJk2iuroagMLCQqqqqvD19aWgoICDBw8SFRUFQFhYGFOmTOGHH37g7rvvvuT59u/fz8SJE6moqKBTp06Ul5c79v3v//4va9eu5dy5c1RUVDB+/HgGDhzII488wuTJk+nXrx8AKSkpdO7cmX79+jFp0iSqqqowDIPY2FhGjRp11f9fyQ3OEGlitmzZYnTt2tU4evSo8c9//tMYNGiQUVxcbBiGYeTn5xv333+/UVZWZixcuNCYP3++YRiGUVNTYzz77LPG+++/b5w7d87o27evkZubaxiGYWRlZRl33nmnceTIEWPLli3GXXfdZXz//feGYRjGM888Y4wYMcI4d+6ccerUKePuu+82Tp48aWzZssWIiooyDMMwPv30U2P06NGm63/11VeNe+65x7Barcb58+eNb7/91oiIiKjVJy4uzli7dm2954mJiTGWL19uGIZhbN++3ejSpYuxZcsW4+jRo8YTTzxhVFRUGIZhGH/729+MQYMGGYZhGH/+85+NpKQkwzAMo6SkxAgNDTXOnj1rTJgwwfF3ZbPZjJdfftmorq42fU3SNOkORJqk9u3b06FDB5YsWYLNZuOpp55y7HNzc+Pw4cM8+eSTbN++nT//+c/8+OOP7Nu3j+7du5Ofn4+npyd9+vQBLswd/Pscxu23305wcDAAHTt2xNfXF29vb9q2bUvLli05e/bsVdX+9ttvM2XKFJKSkpg3b57jbuD/5+HhcclznD59mr179zJkyBAAfvOb39C5c2cAOnTowPTp08nKyuLQoUPs2LGDsrIyAIYOHcq8efMoLi4mOzubBx54gNatWxMeHs748ePZuXMnffr0ISUlBXd3jYDf7PRvgDRJPj4+ANTU1NCnTx8yMzMd/1u+fDmdO3fmrbfeYs6cObRp04YRI0Zw//33YxgGbm5uF80veHr+63ctb2/vS+67Gn//+98dcyUtW7YkKiqKH374gdtuu42ioqJaNRUUFNCuXbtLnuvnJfb//Zif6/z++++Ji4ujtLSU+++/n2effdbRp3Xr1jzyyCN8/vnnfPrpp8THxwMwYMAAVq9ezaOPPsru3buJjo7m8OHDDXLdcuNSgEiTFhoayubNmx1PLW3cuJHBgwdz7tw5Nm3axJNPPsmQIUO49dZbyc3Npbq6mjvvvBPDMNi4cSMA69atu6q7Cg8PD86fP3/ZfqtWrWLevHkYhkFVVRWrVq0iNDSUdu3a0bFjR1auXAlcCBp3d3fuvPPOS57Lz8+Pu+++m7/+9a/AhdDIz88HYNu2bdxzzz3813/9F/fddx/r1q1zzL0AjBo1ikWLFmEYhmOhzFdffZWVK1cSFRXF66+/TqtWrThx4oTTfyfSNGgIS5q0zp07k5qayiuvvIJhGHh6epKeno6Pjw/PP/88M2bM4N1338XDw4OePXty+PBhvLy8mDdvHm+88QZpaWncdddd3HrrrU7X0KNHD2bPns3zzz/PvHnzLtkvOTmZ119/nejoaNzc3HjooYdISEgAIC0tjcmTJ5Oeno63tzdz5sy57BBSWloaEyZMYNmyZXTs2JFOnToBF4bkcnJyiIyMxMvLiz59+nD27FlKS0tp1aoVXbt25ZZbbiEuLs5xrv/+7/9m0qRJfPzxx3h4eDBw4EDuu+8+p/9OpGnQarwiUsvhw4d54oknyM7OpkWLFo1djlzHdAcico0cPHgQq9Va57477riD2bNnX9H5tmzZcsnHgnv37s3EiROvtETmzJnD8uXLmTRpksJDLkt3ICIi4hRNoouIiFMUICIi4hQFiIiIOEUBIiIiTlGAiIiIUxQgIiLilP8HdmA7N111pnsAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 285 }, - { - "cell_type": "markdown", - "metadata": { - "id": "o7qCmHeYKWGp" - }, - "source": [ - "During the training process, the `ExponentiatedGradient` algorithm iteratively trains multiple inner models on a reweighted training dataset. The algorithm stores each of these predictors and then randomizes among them at deployment time.\n", - "\n", - "In many applications, the randomization is undesirable, and also using multiple inner models can pose issues for interpretability. However, the inner models that `ExponentiatedGradient` relies on span a variety of fairness-accuracy trade-offs, and they could be considered for stand-alone deployment: addressing the randomization and interpretability issues, while possibly offering additional flexibility thanks to a variety of trade-offs. \n", - "\n", - "In this section explore the performance of the individual predictors learned by the `ExponentiatedGradient` algorithm. First, note that since the base estimator was `LogisticRegression` all these predictors are different logistic regression models:" - ] + "id": "GuPPVpXrO5uE", + "outputId": "dcc740aa-0805-4ef9-9c77-d64fe1b0111a" + } + }, + { + "cell_type": "markdown", + "source": [ + "Now let's take a look whether the predictiveness is similar across different race groups. First, let's check how well `readmit_30_days` predicts `had_emergency`:" + ], + "metadata": { + "id": "wN8NU8QkRMqM" + } + }, + { + "cell_type": "code", + "execution_count": 22, + "source": [ + "# Visualize predictiveness using a categorical pointplot\n", + "sns.catplot(y=\"had_emergency\", x=\"readmit_30_days\", hue=\"race\", data=df,\n", + " kind=\"point\", ci=95, dodge=True, join=False);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdkAAAFqCAYAAABWGSDZAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABB7ElEQVR4nO3deVwV9f7H8ddhUwgURUDzZtnNfS1NcL8mRiIqefWXy03N1MzK4nc19zXNm5rbLUmre9s0tSSQMjRt0QQNzYXMXDLT1FhdgEDgnPn94a9TJOJJHA7K+/l4+Die+c7M+YwF7zPfmfl+LYZhGIiIiMh15+LsAkRERG5WClkRERGTKGRFRERMopAVERExiUJWRETEJApZERERk7g5u4CylJGRjc2mJ5ZEpOz5+/s4uwRxAp3JioiImEQhKyIiYhKFrIiIiEkUsiIiIiZRyIqIiJhEISsiImIShayIiIhJFLIiIiImUciKiIiYRCErIiJiEoWsiIiISRSyIiIiJlHIishV7Tuazgsrv2bf0XRnlyJyQ6lQs/CIyLWJ2fYDP6ZkkZdvpcVdNZxdjsgNQ2eyInJVefmFRV5FxDEKWREREZMoZEVEREyikBURETGJQlZERMQkClkRERGTKGRFRERMopAVERExiUJWRETEJApZERERkyhkRURETKKQFRERMYlCVkRExCQKWREREZMoZEVEREyikBURETGJQlZERMQkpoZsXFwcYWFhdOvWjZUrV17WvnnzZnr37k2vXr0YPXo058+fB+D06dMMGjSIBx54gMcff5ycnBwALly4wMiRI+nevTuDBg0iLS3NzPJFRERKxbSQTUlJYdGiRaxatYrY2FjWrFnD0aNH7e3Z2dnMmDGDFStWsH79eho0aMC///1vAGbOnMnAgQOJj4+nadOmLFu2DIDFixfTunVrPv74Y/r168ecOXPMKl9ERKTUTAvZhIQEgoOD8fX1xcvLi9DQUOLj4+3tBQUFzJgxg8DAQAAaNGjAmTNnKCgoICkpidDQUAD69Olj3+7zzz+nZ8+eAISHh7N161YKCgrMOgQREZFScTNrx6mpqfj7+9vfBwQEsH//fvv7atWqERISAkBeXh4rVqzg4Ycf5uzZs3h7e+Pmdqk0f39/UlJSLtunm5sb3t7eZGZm2oP6avz8vK/LsYlUFFarjU93neRs1kUAzmZdZO+xTO67tw6uLhYnVydS/pkWsoZhXLbMYrn8hzIrK4vRo0fTsGFDHnzwQXugXm27X7m4OH4ynpGRjc12eV0icjmrzcYrMQfYffi3ex/yC20sXbuX7XtPMSqiCa5/4uevovP393F2CeIEpv2EBAYGkp6ebn+fmppKQEBAkXVSU1MZOHAgDRs2tF9frV69OtnZ2VitVgDS0tLs2wUEBNj3WVhYSHZ2Nr6+vmYdgkiFlpD8c5GA/b3dh9NI+ObnMq5I5MZjWsi2a9eOxMREMjMzyc3NZdOmTXTq1MnebrVaGTVqFN27d2fy5Mn2s1V3d3dat27Nhg0bAIiJibFv17lzZ2JiYgDYsGEDrVu3xt3d3axDEKnQtu0/U6p2EQGLUVy/7nUSFxfH8uXLKSgooG/fvowYMYIRI0YwZswYfv75Z5566ikaNGhgX79p06bMmTOHU6dOMWHCBDIyMqhVqxYLFy6katWqnDt3jgkTJnDy5El8fHxYsGABf/nLXxyuR93FIo4bu2w7mRcuXrG9epVKLBjdvgwrurGpu7hiMjVkyxuFrIjjnn97N0dPnb9i+11/qcqkf7Qqw4pubArZikl3LYhIsTo2r1WqdhFRyIrIFbRvVotW9f2LbWtV35/2TRWyIlej7mIRuSKrzUbCNz/zzqbDFBTacHdz4R/316d901q4FPOc7DfpB9l84gtC6nSmaY1GTqi4/FJ3ccWkM1kRuSJXFxc6Nr+V6j6VAKjuU4mOzW8tNmABPvxhE0fOHePDHzaVZZki5ZZCVkSum4uFF4u8ilR0ClkRERGTKGRFRERMopAVERExiUJWRETEJApZERERkyhkRURETKKQFRERMYlCVkRExCQKWREREZMoZEVEREyikBURETGJQlZERMQkClkRERGTKGRFRERMopAVERExiUJWRETEJApZERERkyhkRURETKKQFRERMYlCVkRExCQKWRG5qsoebkVeRcQxClkRuaqIjnVpWMeXiI51nV2KyA1FX0tF5Kpa3FWDFnfVcHYZIjccncmKiIiYxNQz2bi4OKKioigoKGDo0KEMGjSo2PXGjx9PUFAQffr0ISMjg2HDhtnbsrKyOHv2LHv27CEpKYknn3ySmjVrAtC4cWPmzp1r5iGIiIhcM9NCNiUlhUWLFhEdHY2Hhwf9+/cnKCiIu+66q8g606dPJzExkaCgIAD8/PyIjY0FwGazMWTIECIjIwFITk5m2LBhPPbYY2aVLSIict2Y1l2ckJBAcHAwvr6+eHl5ERoaSnx8fJF14uLi6Nq1K927dy92H+vWrcPT05OePXsCl0J2+/btREREMGrUKM6cOWNW+SIiIqVm2plsamoq/v7+9vcBAQHs37+/yDrDhw8HYPfu3Zdtb7VaiYqKIioqyr7Mx8eHHj16EBISwrvvvktkZCSrV692uCY/P+8/exgi8ie4urrYX/39fZxcjYjzmRayhmFctsxisTi8/bZt26hbty4NGjSwL5s1a5b97wMGDODFF18kKysLHx/HfpgzMrKx2S6vS0SuD6vVZn9NS8tycjXli750VEymdRcHBgaSnp5uf5+amkpAQIDD22/evJmwsDD7e5vNRlRUFFartch6bm56CklERMon00K2Xbt2JCYmkpmZSW5uLps2baJTp04Ob793715at25tf+/i4sInn3zCxo0bAYiJiaFFixZ4enpe99pFRESuB1PPZCMjIxk8eDARERGEh4fTvHlzRowYQXJy8lW3P3nypP1RnV+98MILvPXWW/To0YN169Yxe/Zss8oXEREpNYtR3MXTm5SuyYqYa2biPFJz0wnwrMH0ts86u5xyRddkKyaN+CQiImIShayIiIhJFLIiIiImUciKyDUpPLGXX+LmUnhir7NLESm39JCpiFyTi7s+wJb+IxcL8nCr09LZ5YiUSzqTFZFrYhTkFXkVkcspZEVEREyikBURETGJQlZERMQkClkRERGTKGRFRERMopAVkT/FsFkp+G4rRnbmpffZmZfe22xOrkyk/NFzsiLiMMNmJW/zMgqP7/5tobWAvK3/we3EPgwfTcAh8ns6kxURhxUe3l40YH/fdnw35OuZWZHfU8iKiMPyD20tsd0oyC2jSkRuDApZEXHYr9dh/8gKJPlU5pzFCsDZi+dIOJ2EzdB1WqnYFLIi4jCLd/XLllmBVTWrsC6wCoUWCwAFtkJWfvcer3/zDlabtYyrFCk/FLIi4jCPBp0uW/a1T2UOeFcG4/KbnvamfcNXP39dFqWJlEsKWRFxmFv9Drjd0arIsl1VKl/6y/+fxf5Rwpkks8sSKbcUsiLiMIuLC5VDRlO586Pg6g7AOfeSnwQ8m3euDCoTKZ8UsiLyp1hcXHFv0NF+fdbXKPnXSLXKvmVQlUj5pJAVkVK59yqPxrardW/ZFCJSDilkRaRUWuVBS/+mxba19G9KUK1WxbaJVAQKWREpFRcsDGsyiH807Ie7y6Xrs+4ubvyjYT8ebfoPXCz6NSMVl/7vF5FSc3Vxpe2t91Ktki8A1Sr50vbWexWwUuHpJ0BERMQkClkRuSYW98pFXkXkcgpZEbkmlVo/iGuthlRq/aCzSxEptzSfrIhcE7c6LXGr09LZZYiUa6aeycbFxREWFka3bt1YuXLlFdcbP3480dHR9vcxMTF06NCB3r1707t3bxYtWgTA6dOnGTRoEA888ACPP/44OTk5ZpYvIiJSKqaFbEpKCosWLWLVqlXExsayZs0ajh49etk6o0aNIj4+vsjy5ORkJkyYQGxsLLGxsURGRgIwc+ZMBg4cSHx8PE2bNmXZsmVmlS8iIlJqpoVsQkICwcHB+Pr64uXlRWho6GVhGhcXR9euXenevXuR5cnJycTExNCrVy/Gjh3L+fPnKSgoICkpidDQUAD69Olz2f5ERETKE9OuyaampuLv729/HxAQwP79+4usM3z4cAB2795dZLm/vz8jR46kefPmLFy4kFmzZjF+/Hi8vb1xc3Ozr5OSkvKnavLz876WQxERB7m6uthf/f19nFyNiPOZFrJGMXNLWq4wFdYfvfzyy/a/Dx8+nJCQEJ599tlr3t+vMjKysdkur0tErg+r1WZ/TUvLcnI15Yu+dFRMpnUXBwYGkp6ebn+fmppKQEDAVbfLysrijTfesL83DAM3NzeqV69OdnY2VqsVgLS0NIf2JyIi4iymhWy7du1ITEwkMzOT3NxcNm3aRKdOna66nZeXF6+99hr79u0D4J133qFbt264u7vTunVrNmzYAFy6A9mR/YmIiDiLqWeykZGRDB48mIiICMLDw2nevDkjRowgOTn5itu5urqyePFiZsyYQffu3Tlw4ADjxo0DYPr06axdu5awsDB27drFM888Y1b5IiIipWYxirt4+gdPPfUUAwYMoF27dmVRk2l0TVbEXDMT55Gam06AZw2mt738PoqKTNdkKyaHzmTvv/9+li1bRmhoKK+//jrnzp0zuSwREZEbn0Mh27NnT9555x2WLVtGRkYG/fr1Y9y4cZc9kiMiIiK/cfiarM1m48cff+T48eMUFhbi5+fHjBkzmD9/vpn1iYiI3LAcek520aJFREdHc9tttzFw4ECWLFmCu7s7v/zyC126dLHfmCQiIiK/cShkMzMzefXVV2nYsGGR5V5eXrz44oumFCYiInKjc6i7+IknnmD16tUAHDt2jNGjR5OWlgZAhw4dzKtORETkBuZQyE6YMIE777wTgNq1a9OmTRsmTZpkamEiIiI3OodC9uzZswwePBiASpUqMXToUPuZrIiIiBTPoZC1Wq1FZrxJT08vdgIAERER+Y1DNz4NHTqUiIgIOnbsiMViISEhodhZcaT82nc0nfidJ3ggqA4t7qrh7HJERCoEh0K2b9++NG3alB07duDq6sqjjz5K/fr1za5NrqOYbT/wY0oWeflWhayISBlxeD5ZHx8f2rRpg2EYFBQUcODAAZo0aWJmbXId5eUXFnkVERHzORSy8+fP55133sHPz8++zGKxsGXLFtMKExERudE5FLIff/wxmzZtIjAw0Ox6REREbhoO3V1cq1YtBewNymqzsW3faTIvXAQg88JFtu07rSn/RETKgENnsm3btmXevHl07dqVypUr25frmmz5ZrXZeCXmALsP//ZMc4HVxn8//o7932cwKqIJri4OzxEhIiJ/kkMhGx0dDUB8fLx9ma7Jln8JyT8XCdjf2304jYRvfqZj81vLuCoRkYrDoZD99NNPza5DTLBt/5mrtitkRUTM41BfYU5ODrNmzWLIkCGcO3eOadOmkZOTY3ZtUkqZWXklt18ouV1ERErHoZCdPXs2Pj4+ZGRkUKlSJbKzs5k2bZrZtUkpVfepXHJ7lZLbRUSkdBwK2YMHDxIZGYmbmxuenp4sWLCAgwcPml2blFLH5rVK1S4iIqXjUMi6/OEOVKvVetkyKX/aN6tFq/r+xba1qu9P+6YKWRERMzl049O9997L/PnzycvLY9u2baxcuZKgoCCza5NScnGxMCqiCQnf/Mw7mw5TUGjD3c2Ff9xfn/ZNa+HiYnF2iSIiNzWHQnbs2LGsWLECHx8fFi1aRMeOHRk9erTZtcl14OriQsfmt7Ih8UdSzuZS3aeS7igWkWu2c+dO5syZg5eXFzk5Odxzzz18++235OTkYBgGs2fPplWrVuTk5DB79my+/vprXF1dCQkJITIykoKCAhYsWEBSUhJWq5XGjRszZcoUvL29nX1opnAoZN3d3XniiSd44oknzK5HRETKuSNHjrB582ZSU1P573//y5o1a3BxcWHFihW8+uqrtGrViqVLl3Lx4kU2bNiA1Wpl2LBhfPXVVyQlJeHq6kp0dDQWi4WFCxeyYMECZsyY4ezDMoVDIXvfffdhsfzWtWixWPD09KRevXpMmDCBgIAA0wqU66Oyh1uRVxGRa1WrVi1q165N7dq1qVq1KqtXr+bkyZPs3LmTW265BYCEhAQmTpyIq6srrq6uvPPOO8ClCWeysrJISEgAoKCgoMjkMzcbh37jhoSEkJOTw6BBg3BxceH9998nJyeHBg0aMG3aNF555RWz65RSiuhYl41fnSC0TR1nlyIiNzgvLy8APv/8c+bMmcMjjzxC165dufPOO1m/fj0Abm5uRU7Ozpw5Q+XKlbHZbEyaNInOnTsDl8ZhuHjxYtkfRBlx6BbhXbt2MWfOHBo3bkzDhg2ZMmUKR44cYejQoZw6dcrsGuU6aHFXDZ4deI8mbBeR62b79u106dKFgQMH0qxZMzZv3ozVagUujXn/wQcfYLPZyM/PZ8yYMSQlJdGhQwdWrlxJfn4+NpuNqVOnsnDhQicfiXkcHvEpOzvb/j47O5u8PI0WJCJSkfXv35+kpCR69uzJQw89xG233cZPP/2EzWbjySefxN3dnd69exMREUHnzp25//77GT16NLVr1+bBBx8kLCwMwzCYMGGCsw/FNBbDMK4659mKFSuIiYnhgQcewDAMNm3aRL9+/XB1dWXXrl0sWbKk2O3i4uKIioqioKCAoUOHMmjQoGLXGz9+PEFBQfTp0weA3bt38/zzz1NYWIivry/PP/88tWvXJikpiSeffJKaNWsC0LhxY+bOnevwwWZkZGuKNxETzUycR2puOgGeNZje9llnl1Ou+Pv7OLsEcQKHrsk+8sgjNGrUiK1bt+Lm5sbUqVMJDg7mm2++4cEHHyx2m5SUFBYtWkR0dDQeHh7079+foKAg7rrrriLrTJ8+ncTExCLP3Y4bN45ly5bRsGFD3n//fWbPnk1UVBTJyckMGzaMxx57rJSHLSIiYj6HQrZfv37ExMTQsWPHIsubNm16xW0SEhIIDg7G19cXgNDQUOLj43nyySft68TFxdG1a1f7OgD5+fk8/fTTNGzYEIAGDRrY70pLTk4mIyODjz/+mJo1azJ9+nRq1dKoRSLlRSW3SkVeRSo6h0K2cuXK/Pzzz/ZuWkekpqbi7//bkH4BAQHs37+/yDrDhw8HLnUP/8rDw4PevXsDYLPZeOmllwgJCQHAx8eHHj16EBISwrvvvktkZCSrV692uCY/v5vzYWeR8mJQy97EHdpMzwYh6h4VwcGQzc3NpWvXrtSsWdN+6zZcOhO9kuIu9f7+du6ryc/PZ8KECRQWFtq7h2fNmmVvHzBgAC+++CJZWVn4+Dj2w6xrsiLmus39DkY3vfTlOS0ty8nVlC/60lExORSykydP/tM7DgwMZNeuXfb3qampDg9akZOTw+OPP46vry9RUVG4u7tjs9lYvnw5I0eOxNXV1b6um5sGVxARkfLJoUd42rRpQ+XKlTl27BgtW7bE3d2dNm3alLhNu3btSExMJDMzk9zcXDZt2kSnTp0cKmrcuHHcfvvtLFmyBA8Pj0uFurjwySefsHHjRgBiYmJo0aIFnp6eDu1TRESkrDkUstHR0UycOJHXXnuNrKwsRo8ezdq1a0vcJjAwkMjISAYPHkxERATh4eE0b96cESNGkJycfMXtvv32W7Zs2cLXX39NREQEvXv3ZsSIEQC88MILvPXWW/To0YN169Yxe/bsP3GoIiIiZcuh52QffPBB3n77bf7xj38QExPDmTNnGD58OB999FFZ1Hjd6JqsiDjLtVyTtVptfLrrJJ98dYK0c7n4+3rSrU0d7ru3Dq7XcarKw4cP07NnT5YuXUpoaCgAX3zxBTNmzOCee+7hxRdfLLJ+7969iY2NvW6f/6sxY8bwww8/lHi/T2kkJyezevVq5syZY8r+i+PQBU0XF5ci0xDVqlWryHVRERG5vqxWGy+8vYvE5DP2Zenncjl4PJOkgymMf7g1rq4OdUZeVXR0NKGhoaxevdoesvHx8YwaNYqHHnrosvXNCNizZ8/y7bffUqNGDXbv3k2rVq2u+2c0a9aMZs2aXff9lsShkPX19eXgwYP2u4PXr19P1apVTS1MRKQi+3TXySIB+3uJyWf4bPdJQtrcXurPKSwsZP369axcuZL+/ftz4sQJdu7cyZYtW0hMTMTFxcX+O//IkSMsXryYiIgIDh06xLlz55g8eTLHjh3Dw8ODCRMm0LZtW9555x1iY2PJzc3FYrGwePFi/vrXv3LffffRq1cvvvzyS3Jzc3nhhRfs4y3ExcXRunVr6tevz5o1a+whGx0dzeeff05qaio///wzQ4YM4fTp0+zYsQNfX19ee+01KlWqRExMDG+++SY2m40mTZowffp0KlWqRHBwME2aNCE9PZ1nn32WV155hbfffpuDBw8ybdo08vLyqFq1KgsWLKBGjRrMmDGDI0eOkJ6eTt26dXnppZdIT0/nySefpF69ehw8eBA/Pz+WLFlSZIyHK3Hoa9CkSZMYN24c33//PR06dGDJkiVMmTLl2v+riohIiT756kSJ7Zt2ltzuqM8//5xbb72VunXrEhISwurVq+nXrx/33XcfY8aMoV+/fsClgYE2btxIo0aN7NsuWbKEOnXq8PHHHzNv3jwWL15MdnY2mzdv5u233+bDDz8kJCSEVatW2bfx9fXl/fffp3///ixfvty+PDo6mu7du9O9e3c2btzIuXPn7G3Jycm89tprrFy5kn/961906tTJ3qW8bds2jhw5wtq1a1m9ejWxsbH4+fnx+uuvA5fOkEeOHElsbGyRp1HGjh3L6NGjiYuLIywsjDfffJM9e/bg7u7OmjVr+OSTT7h48SJffPEFAN999x2PPPIIH374IVWqVHG4S9uhM9m//vWvxMbGcvz4caxWK3Xr1sXd3d2hDxARkT8v7VxuqdodFR0dTXh4OABhYWGMHTuWZ5555rL1mjdvftmypKQkFixYAFwK4TVr1gDw4osv8tFHH3H8+HG2bdtWJJh/HTmwXr16bNq0CYCDBw9y5swZ2rVrh7u7O40aNSImJoahQ4cCcM899+Dt7W2/bNm2bVsAateuzYULF9i5cyc//vgj//M//wNcmqO2cePG9s9s0aJFkbozMzNJS0ujS5cuAAwcONDe5uvry8qVKzl27BjHjx/nl19+AcDPz8++z3r16nH+/PmS/2H/n0Mhm5eXx5YtW+zfLJKSkgCuOOC/iIiUjr+vJ+klBKm/b+kfX8zIyGDr1q188803vPXWWxiGwYULF+zh93uVK1e+bNkfxyn4/vvvqVy5MkOGDOEf//gHnTp1okaNGhw8eNC+TqVKl4bc/P3gROvWrSM/P99+PTgnJ4fVq1fbQ/aPJ3V//Fyr1Ur37t3tPaw5OTn2KfeKq/2P+7t48SKpqakcPnyYpUuXMnjwYPr06cPZs2ftAyv9WvevtTtwzzDgYHfxqFGjeP3119m5c6f9z1dffeXQB4iIyJ/XrU2dEtvvDyq53RHr168nODiYrVu38umnn/LZZ58xatQo+xnp1bRu3ZoNGzYAlwJ2xIgRfPPNN9x+++0MHTqUFi1asHXr1iKB90f5+fnExcXxxhtv8Omnn/Lpp5+yZcsW0tLS2Llzp0N1BAUF8cknn5CRkYFhGMyYMYM333zziuv7+PhQs2ZNtm/fDly6kWvJkiUkJibSvXt3/v73v1OjRg2SkpJKrN0RDp3JpqSksGHDhj81LKKIiFy7++6tQ9LBlGJvfmrbrBZdWpc+ZKOjo4mMjCyybODAgbz22mvUqFHjqtuPGTOGKVOm0KtXL9zc3Jg3bx6NGjVi9erVhIWF4eHhQfPmzTly5MgV9/HZZ59Ru3btIl263t7e9OvXj9WrV182MU1xGjZsyJNPPsmQIUOw2Ww0atSIkSNHlrjN/PnzmTFjBvPmzaNatWrMmzePs2fPMnbsWOLj4/Hw8KBly5b89NNPV/38kjj0nOzTTz/NlClTigz4fyPSc7JX9036QTaf+IKQOp1pWqPR1TcQEYdc63Oyn+0+yaadvz0ne39QHbq0vr7PyYp5HDqTfeCBB+jevTv169cv0hf+1ltvmVaYOMeHP2ziZNYp8qwXFbIiTubq6kJIm9uvy6M64hwOhez8+fN57LHHqFOn9N0TUr5dLLxY5FVERK6dQyF7yy232McPFhEREcc4dHdxly5dWLlyJampqZw7d87+R0RERK7MoTPZN954g/z8fJ577jn7MovFUuTZJxERESnKoZDdv3+/2XWIiIjcdBzqLrbZbLz++utMmDCB7Oxsli9fXuoHdEVExLmys7OZOXMm4eHh9O7dm4cffpgDBw44rZ7evXs77bPN4tCZ7Lx588jMzCQ5ORnDMNi2bRtpaWmaJEBExESGzUrW/s/J2reFwgsZuFXxw6dFV3ya/w2LS+mmG7XZbIwYMYKgoCBiYmJwc3Njx44djBgxgo8++ohq1apdp6NwnBlT6DmbQyGbmJjIBx98QJ8+ffDx8eE///nPTfmNQ0SkvDBsVlKiX+SXQ78NLWi9kM7Fnw7xy9HdBPb5Z6mCdufOnaSmpjJmzBhcXC51agYHBzN37lxsNhtTpkwpdsq3wYMH8+mnnwLw73//G4CnnnqKuLg4oqKisFgsNGvWjOeee47MzEwmTZpEVlYWaWlp9OjRg7Fjx/Ldd98xbdo0CgsLqVSpEnPnzuWOO+6gQYMGHDp0iJSUlGK3i46OZtu2bZw/f56TJ0/Svn17ZsyYce3/yGXAoe5iNzc3+38EAA8Pj8sGaBYRkesna//nRQL29345tJPs5C9Ktf9vv/2WZs2aFfndDtC5c2eOHTt2xSnfipOSksLcuXP5z3/+w0cffYTVauWLL77gww8/JDw8nLVr17J+/XpWrVpFZmYmb775Jo888gjR0dE8/PDD7N27t8j+rrQdwJ49e1i6dCnr16/ns88+49ChQ6X6dzCbQ0lZv359Vq5cidVq5dixY7zxxhs0bNjQ7NpERCqsrH1bSmy/sHcLPi3uu+b9u7i4XHEmmXvvvfeKU74VZ8+ePdxzzz3UrFkTuDSA0a927NjB66+/zpEjRygoKCA3N5fOnTsza9Ystm3bRpcuXeyz7/zq0UcfLXY7gLvvvts+5d1tt93m8JRzzuLQmezkyZM5cOAAGRkZDBw4kF9++YVJkyaZXZuISIVVeCHjKu3ppdp/06ZN+fbbby8L2oULF7J582bGjh1L5cqV6dOnD/feey+GYVw2xVthYSFw+dRzmZmZZGZm8q9//Yu3336bW2+9lccff5xq1aphGAYPPPAAH3zwAc2bN+fNN99k+vTpRba/0nZw7VPOOYtDIevt7c3zzz9PQkICO3bsYMGCBfaL4v/6179MLVDKhtVmJeF0EmcvngPg7MVzJJxOwmbYnFuYSAXlVsXvKu1XnyWnJK1bt8bPz4+XXnrJ/rTItm3b7Nc9i5vyrUqVKpw/f57MzEzy8/PZtm0bAM2aNWPfvn2kpaUB8Pzzz7Nlyxa2b9/Oo48+Svfu3Tlz5gwpKSnYbDaeeeYZ9u/fT//+/Xn66af59ttvi9R2pe1uRKW+sOrofH9SflltVv5zYCV7076xLyuwFbLyu/c4kHGQYU0G4VrKOxlF5M/xadGViz9d+XpjlZZdS7V/i8XCsmXLmDt3LuHh4bi5uVGtWjVWrFiBq6trsVO++fj48Oijj9K3b19q1qxJs2bNAAgMDGTy5Mk8+uij2Gw2WrZsSZ8+ffD09OTZZ5+lSpUq+Pn50bRpU3766SdGjRrF5MmTWbZsGa6urkyYMKFIbY899lix292IHJrqriQRERHExMRcp3LMpanuipdwOomV3713xfZ/NOxH21vvLcOKRG4+f3aqu+LuLv6VV4OgUt9dLGWj1Geymsj9xpd45qsS2xPOJClkRcqYxcWVwD7/JDv5Cy7s3ULhhXTcqtSgSsuueDfrrIC9Qeg5HOFsXsl3553NO1c2hYhIERYXV3xa3Fequ4jFuRy68UlubtUqV71Ku2/ZFCIicpMpdciW99un5era1mpTYnu7WuoqFhG5FqUO2YkTJ16POsSJgmu1oqV/02LbWvo3JahWqzKuSETk5lDi3cUNGzYs8camG20+Wd1dXFThib3k7/sYjxbdsfylGV/9/DVrDn9Aga0Qdxc3Hqr/IEG1WuFi0VUFkdL6s3cXy82hxBufEhMTMQyDJUuWULt2bR566CFcXV2Jjo7m9OnTV935rwNGFxQUMHToUAYNGlTseuPHjycoKIg+ffoAcPr0acaNG0dGRgZ169ZlwYIF3HLLLVy4cIGxY8dy8uRJqlevzuLFi/H397+Gw67YDJuVwsPbyUtYCYUXyU07TuV2gwiu34FNP35Gam461Sr56o5iEZFSKvEUpVq1alSvXp1vvvmGkSNHUrVqVby9vRk8eDBffVXyYx8pKSksWrSIVatWERsby5o1azh69Ohl64waNYr4+Pgiy2fOnMnAgQOJj4+nadOmLFu2DIDFixfTunVrPv74Y/r168ecOXOu5ZgrNMNmJW/zMvK2/gcKL15aWHiRvK3/IW/zyxjoTF+kvLDarHx6LIGpm+fzeNwkpm6ez6fHEq7b6Ec5OTnMnDmTbt260atXLwYOHEhiYiIAa9as4cMPPwRgwoQJREdHX5fPrGgc6gfMzc3l2LFj9veHDh2ioKCgxG0SEhIIDg7G19cXLy8vQkNDLwvTuLg4unbtSvfu3e3LCgoKSEpKsg8Y3adPH/t2n3/+OT179gQgPDycrVu3XrUOKarw8HYKj+8uvu34bsjPK+OKRKQ4VpuVRYmv8UrS2xzKOEbGL2c5lHGMV5LeZmHiq1ht1lLt3zAMRo0ahbu7Ox999BHr169nypQpjBs3jp07d7Jnzx7y8/Ov09FUXA49J/vMM8/w0EMP0aBBA2w2G99//z0LFiwocZvU1NQiXbkBAQHs37+/yDrDhw8HYPfu337pnz17Fm9vb/uA0/7+/qSkpFy2Tzc3N7y9vcnMzCQwMNCRw8DPz9uh9W5mpzZsL3mFglxwAVdXF11DEnGiL47v5Kuf9hbb9tVPe9l6fCdd7mx3zfv/6quvOH36NG+99Zb93pvGjRvz+OOPM3ToUHx8fNixY4f9d+7nn3/OqlWryMjIYNSoUTz00EPk5OQwa9Ysjhw5gtVqZcSIEYSHhxMdHc0HH3zAuXPn6NKlC//7v/97zXXe6BwK2fvvv59WrVqxe/duLBYLrVq1onr16iVuU9z9VI6MDvVnt/vjXIgl0Y1PkH82rcR2w2YFFwtWq420tKwyqkrk5vdnv7R+dqzkL8Sf/pBQqpBNTk6madOml/1+vffee/H09OS+++6jTZs2dOzYkY8++oj8/Hzee+89jhw5wuDBg3nooYeIioqiSZMmvPDCC2RnZ9O/f39atGgBXLocuGHDhgo/97hDR5+fn8/XX39tn0/ws88+48SJE0RGRl5xm8DAQHbt2mV/n5qaSkBAwFU/q3r16mRnZ2O1WnF1dSUtLc2+XUBAAOnp6dSsWZPCwkKys7Px9fV15BDk/1m8q2PkZF55BRdX4Mac7ULkZpKee7bk9l9K+Dl2gMVisc++83tXugTXtWtXLBYL9erV4+zZS7UlJCSQl5fHunXrAPjll184cuQIcOmsuKIHLDgYspGRkZw8eZK0tDQaN27Mvn37aNPmKgMYtGvHv//9bzIzM/H09GTTpk0899xzV/0sd3d3WrduzYYNG+jZsycxMTF06tQJgM6dOxMTE8OoUaPYsGEDrVu3xt3d3ZFDkP/n0aATeSlHr9hucfcEa04ZViQixanhWY2MX64ctDW8Su5NvJoWLVrw9ttvU1BQUOT36N69e+2z6/yeq+ulsZJ/f+Zrs9mYP38+TZo0ASA9PZ2qVasSFxdH5cqVS1XfzcKhvtaDBw8SHR1N165dmTRpEqtXryYrq+SuxMDAQCIjIxk8eDARERGEh4fTvHlzRowYQXJyconbTp8+nbVr1xIWFsauXbt45plnAHj66afZu3cvPXr0YNWqVUybNs2xoxQ7t/odcLuj+MEl3O5oBR6eZVyRiBSny53tS2y/r+61dxXDpflk77rrLp5//nn72es333xDVFQUo0ePxtXVtdgz3d8LDg7m3XffBS71Vvbq1YszZ86Uqq6bjUNnsgEBAbi5uXHHHXdw+PBhunfvTm5u7lW369mzp/1u4F+9+uqrl633x4nfa9euzdtvv33Zer6+vrzyyiuOlCxXYHFxoXLIaAqPJJC/72NsF1JxqRKAR4vuuNVrT+Xd/wagklslJ1cqUrH97Y5gvj6TXOzNT23+0pLOdwSX+jNeeuklFi1aRHh4OK6urlStWpX58+cTFBREeno6CxcuxMfnyteSn3zySWbMmEF4eDhWq5Vx48ZRp06dIpcKKzqH5pMdNmwYDz74INWqVWPt2rU88cQTPPHEE2zevLksarxudOPT1X2TfpDNJ74gpE5nmtZo5OxyRG4a13K3vtVmZevxnXz6QwLpv2RSw6s699VtR+c7gv/UTZ/iPA6F7PHjx3nvvfcYN24ckZGRbN++ncjISAYMGFAWNV43ClkRcRY9ElcxOfRVKCMjgwMHDtCrVy+OHj1KYGAgq1atMrs2ERGRG5pD12SnTp3K//zP/9CoUSOHnnUVERERB0PWw8ODoUOHmlyKiIjIzcWh7uI777zzqo/diIiISFEl3vj06+M3OTk5pKamcttttxUZwSMuLs78Cq8j3fgkIs6iG58qphK7i6dOnVpWdYiISBn76aefGDx4MJ9++mmR5Q0aNODQoUPFbrNz505eeumlYscykMuVGLJXGzpRRETMY1itpH72OSmfbOFiegaVavgR2K0rAV3+huX/hzmU8k2jN4uIlEOG1cqh+S+SkbjTviw/PZ2s7w5xdtduGoz7p6lBGx0dzbZt2zh//jwnT56kffv2zJgxo8g6b775Jps3b2bFihWMHDmSZs2asXv3bjIzM5kyZQqdO3cmPT2dyZMnc/r0adzc3IiMjKRp06b06tWLL7/8EoCOHTsyceJEwsLCWLFiBRaLhby8PFJSUvjxxx85deoU/fr14/HHHzfteM2iIUNERMqh1M8+LxKwv5eRuJPUz74wvYY9e/awdOlS1q9fz2effVakC3ndunVs2rSJ5cuX4+l5aczzgoIC1qxZw8SJE1myZAkAzz33HMHBwcTFxbF06VImTZqEzWbj1ltv5fDhw3z//fdYrVa++uorALZu3UqXLl0AOHToEK+//jrvvfceK1as4MKFC6Yf8/WmkBURKYdSPtlScvvmktsdUdzQjIZh2MdDuPvuu/H29sbT05PbbruN8+fPA3D48GGmTZvG4MGD8fLysm/bsWNHAOrVq8e5c+cA2LFjB3379gXgtttuo0WLFuzbt4/OnTuTmJjIjh07GDx4MLt37yYrK4u0tDTuuusuAIKCgvDw8MDPzw9fX9+rTkxTHilkRUTKoYvpGSW3p6WX+jOqVKlyWXBlZGRQtWpVACpV+m2iEIvFwq8Po9xyyy0sXbqUefPm2ecZ//36vx+06I8PsBiGgdVqtYfszp076datGy4uLsTFxdmDuqTPv5EoZEVEyqFKNfxKbvevUerP8Pb25vbbb2fjxo32ZWvWrKFt27Ylble7dm26du1KmzZtWLp0aYnrBgcH8/777wNw8uRJvv76a1q2bEmTJk344YcfOH78OH/9618JCgoiKiqKv/3tb6U+rvJEISsiUg4FdutacntIye2Omj9/PqtWraJXr150796dI0eOODxX97PPPktcXBwHDhy44jqTJ09mx44d9OzZk9GjRzN79mwCAgKwWCy0atWKv/71r8ClMM7Ozr7pnmpxaBaem4UGoxARZ/mzg1EUd3fxr/zaBpl+d7FcHwpZETFV9v69nI3/mGoPdMe7eUtnl+M01zLi06XnZL8gZfMWLqalU8m/BoEhXQno0lkBe4PQc7IiYqqMmA+4eOJHbHl5FTpkr4XF1ZXAkPsIDLnP2aXINdI1WRExle1iXpFXkYpEISsiImIShayIiIhJFLIiIiImUciKiIiYRCErIiJiEoWsiIiISRSyIiIiJlHIioiImEQhKyIiYhKFrIiIiElMHbs4Li6OqKgoCgoKGDp0KIMGDSrSfvDgQaZMmUJ2djatW7dm5syZnD9/nmHDhtnXycrK4uzZs+zZs4ekpCSefPJJatasCUDjxo2ZO3eumYcgIiJyzUwL2ZSUFBYtWkR0dDQeHh7079+foKAg7rrrLvs648aNY/bs2bRs2ZJJkyaxdu1aBg4cSGxsLAA2m40hQ4YQGRkJQHJyMsOGDeOxxx4zq2wREZHrxrTu4oSEBIKDg/H19cXLy4vQ0FDi4+Pt7adOnSIvL4+WLVsC0KdPnyLtAOvWrcPT05OePXsCl0J2+/btREREMGrUKM6cOWNW+SIiIqVm2plsamoq/v7+9vcBAQHs37//iu3+/v6kpKTY31utVqKiooiKirIv8/HxoUePHoSEhPDuu+8SGRnJ6tWrHa7Jz8/7Wg9HRK7RCVcXCgBXV5drmlNV5EZmWsgWNxe8xWJxuH3btm3UrVuXBg0a2JfNmjXL/vcBAwbw4osvkpWVhY+PYz+4mrRdpOxZrTb7a1palpOrcR59waiYTOsuDgwMJD093f4+NTWVgICAK7anpaUVad+8eTNhYWH29zabjaioKKxWa5HPcXPTvPMiIlI+mRay7dq1IzExkczMTHJzc9m0aROdOnWyt9euXZtKlSqxe/duAGJiYoq07927l9atW/9WqIsLn3zyCRs3brSv36JFCzw9Pc06BBERkVIx9Uw2MjKSwYMHExERQXh4OM2bN2fEiBEkJycDsGDBAubOnUv37t3Jzc1l8ODB9u1Pnjxpf1TnVy+88AJvvfUWPXr0YN26dcyePdus8kVERErNYhR3cfQmpWuyImXvh8njKUhJwT0wkLpzXnB2OU6ja7IVk0Z8EhERMYlCVkRExCQKWREREZMoZEVEREyikBURETGJQlZERMQkClkRERGTKGRFRERMopAVERExiUJWRETEJApZERERkyhkRURETKKQFRERMYlCVkRExCQKWREREZMoZEVEREyikBURETGJQlZERMQkClkRERGTKGRFRERMopAVERExiUJWRETEJApZERERkyhkRcQUhtXK+S+3UpiZCUBhZibnv9yKYbM5uTKRsmMxDMNwdhFlJSMjG5utwhyuiNMYVitnli8j++vdl7V539OKWo+NxuLq6oTKnMff38fZJYgT6ExWRK67C4nbiw1YgOyvd3MhMaGMKxJxDoWsiFx357dtLbn9y5LbRW4WClkRue4Kz2aW3J6ZUUaViDiXQlZErju3atVLbq/uV0aViDiXqSEbFxdHWFgY3bp1Y+XKlZe1Hzx4kL///e+EhoYyefJkCgsLAYiJiaFDhw707t2b3r17s2jRIgBOnz7NoEGDeOCBB3j88cfJyckxs3wRuUZVO3Yqub1Dye0iNwvTQjYlJYVFixaxatUqYmNjWbNmDUePHi2yzrhx45g6dSobN27EMAzWrl0LQHJyMhMmTCA2NpbY2FgiIyMBmDlzJgMHDiQ+Pp6mTZuybNkys8oXkVKo0q4D3ve0KrbN+55WVGnXvowrEnEO00I2ISGB4OBgfH198fLyIjQ0lPj4eHv7qVOnyMvLo2XLlgD06dPH3p6cnExMTAy9evVi7NixnD9/noKCApKSkggNDb1sfREpXywuLtR6bDSBQx/F4u5+aZm7O4FDH6XWqCewuOhKlVQMbmbtODU1FX9/f/v7gIAA9u/ff8V2f39/UlJS7H8fOXIkzZs3Z+HChcyaNYvx48fj7e2Nm5vbZes7ys/PuzSHJCJ/UsCDYZzftIG802eo5F+Dux4Mc3ZJImXKtJAtbowLi8XiUPvLL79sXzZ8+HBCQkJ49tlnS9yfIzQYhUjZs1pt9te0tCwnV+M8GoyiYjKtzyYwMJD09HT7+9TUVAICAq7YnpaWRkBAAFlZWbzxxhv25YZh4ObmRvXq1cnOzsZqtRZZX0REpLwyLWTbtWtHYmIimZmZ5ObmsmnTJjp1+u2Owtq1a1OpUiV27740KkxMTAydOnXCy8uL1157jX379gHwzjvv0K1bN9zd3WndujUbNmwosr6IiEh5ZerYxXFxcSxfvpyCggL69u3LiBEjGDFiBGPGjKFZs2Z89913TJkyhZycHBo3bszcuXPx8PBg165dzJkzh7y8PO644w7mzZuHj48Pp06dYsKECWRkZFCrVi0WLlxI1apVHa5H3cUiZe+HyeMpSEnBPTCQunNecHY5TqPu4opJEwSIiKkUspcoZCsm3UcvIiJiEoWsiIiISRSyIiIiJlHIioiImEQhKyIiYhKFrIiIiEkUsiIiIiZRyMpVZe/fy8l5c8nev9fZpYiI3FBMmyBAbh4ZMR9w8cSP2PLy8G7e0tnliIjcMHQmK1dlu5hX5FVERByjkBURETGJQlZERMQkClkRERGTKGRFRERMopAVERExiUJWrsiwWjn/5VYKMzMBKMzM5PyXWzFsNidXJiJyY9BzslIsw2rlzPJlZH+9+7dlBQWkvPEfcvbvo9Zjo7G4ujqxQhGR8k9nslKsC4nbiwTs72V/vZsLiQllXJGIyI1HISvFOr9ta8ntX5bcLiIiClm5gsKzmSW3Z2aUUSUiIjcuhawUy61a9ZLbq/uVUSUiIjcuhawUq2rHTiW3dyi5XUREFLJyBVXadcD7nlbFtnnf04oq7dqXcUUiIjcehawUy+LiQq3HRhM49FEs7u6Xlrm7Ezj0UWqNegKLi/7XERG5Gv2mlCuyuLpStUNH3Kpfuj7rVr06VTt0VMCKiDhIvy1FRERMopAVERExiUJWRETEJApZERERk5gasnFxcYSFhdGtWzdWrlx5WfvBgwf5+9//TmhoKJMnT6awsBCA3bt38/e//53evXszZMgQTp06BUBSUhJBQUH07t2b3r17M3HiRDPLFxERKRXTQjYlJYVFixaxatUqYmNjWbNmDUePHi2yzrhx45g6dSobN27EMAzWrl1rXz5nzhxiY2Pp2bMns2fPBiA5OZlhw4YRGxtLbGwsc+fONat8EblOXCpVLvIqUpGYFrIJCQkEBwfj6+uLl5cXoaGhxMfH29tPnTpFXl4eLVu2BKBPnz7Ex8eTn5/P008/TcOGDQFo0KABZ86cAS6F7Pbt24mIiGDUqFH25WIu/ZKU0vCLeBDPBg3xi3jQ2aWIlDnT5pNNTU3F39/f/j4gIID9+/dfsd3f35+UlBQ8PDzo3bs3ADabjZdeeomQkBAAfHx86NGjByEhIbz77rtERkayevVqh2vy8/Mu7WFVSK5DBnHqg1hqP9ib6v4+zi5HbjD+XTtSt2tHZ5ch4hSmhaxhGJcts1gsDrfn5+czYcIECgsLeeyxxwCYNWuWvX3AgAG8+OKLZGVl4ePj2C/+jIxsbLbLP1eu4vb61HxmHFYgLS3L2dWI3JD89QW1QjKtuzgwMJD09HT7+9TUVAICAq7YnpaWZm/Pyclh+PDhFBYWEhUVhbu7OzabjaioKKxWa5HPcXMz7XuCiIhIqZgWsu3atSMxMZHMzExyc3PZtGkTnTr9NnNL7dq1qVSpErt37wYgJibG3j5u3Dhuv/12lixZgoeHx6VCXVz45JNP2Lhxo339Fi1a4OnpadYhiIiIlIrFKK7f9jqJi4tj+fLlFBQU0LdvX0aMGMGIESMYM2YMzZo147vvvmPKlCnk5OTQuHFj5s6dy9GjR3nwwQe566677GepAQEBvPrqqxw5coSpU6eSlZVF9erVmTdvHrVq1XK4HnUXi4izqLu4YjI1ZMsbhayIOItCtmLSiE8iIiImUciKiIiYRCErIiJiEoWsiIiISRSyIiIiJlHIioiImKRCDZfk4mK5+koiIiLXSYV6TlZERKQsqbtYRETEJApZERERkyhkRURETKKQFRERMYlCVkRExCQKWREREZMoZEVEREyikBURETGJQlZERMQkClkhLi6OsLAwunXrxsqVK51djtyAsrOzCQ8P56effnJ2KSLlikK2gktJSWHRokWsWrWK2NhY1qxZw9GjR51dltxA9u3bx4ABAzh+/LizSxEpdxSyFVxCQgLBwcH4+vri5eVFaGgo8fHxzi5LbiBr165l+vTpBAQEOLsUkXKnQs3CI5dLTU3F39/f/j4gIID9+/c7sSK50cyZM8fZJYiUWzqTreCKm4TJYtGUgCIi14NCtoILDAwkPT3d/j41NVXdfiIi14lCtoJr164diYmJZGZmkpuby6ZNm+jUqZOzyxIRuSnommwFFxgYSGRkJIMHD6agoIC+ffvSvHlzZ5clInJTsBjFXZQTERGRUlN3sYiIiEkUsiIiIiZRyIqIiJhEISsiImIShayIiIhJFLJyU7j77rv/1AwwycnJjBkzBoD9+/czbdq0q26TlZXFmDFjCA8PJywsjBUrVtjbjh8/zsCBAwkLC6Nv3758//33f/oYwsPD2blz55/eTkTKL4WsVEjNmjVj6dKlABw9epSUlJSrbrNkyRICAwP58MMPef/991m9ejV79uwBYOzYsQwYMIANGzbw1FNPMWbMmGKHrBSRikWDUUiZ27lzJ3PmzMHLy4tffvmFMWPGsHz5cgoKCqhcuTLjx4/n7rvvJj09nWnTppGRkUFaWhq1a9dm8eLF+Pn5sWvXLp577jksFgvNmjXDZrPZ971w4UICAgI4cuQInp6ePPXUU7z99tv88MMP3H///UyaNImdO3fy3HPP8eqrr7J06VKysrKYOHEic+fOvWLdkydPxmq1ApCWlkZ+fj4+Pj6kpKRw7NgxevToAUDnzp2ZOXMm3377LU2aNLni/o4ePcqkSZPIzc3lzjvv5JdffrG3vfLKK2zevJmLFy+Sm5vL+PHjCQkJ4YEHHmDq1Kl06NABgClTplCvXj06dOjA5MmTyc/PxzAM+vbty6BBg0r930pESskQKWM7duwwGjZsaPz000/GDz/8YISHhxuZmZmGYRjG4cOHjfbt2xs5OTnGG2+8YSxfvtwwDMOw2WzG8OHDjddff924ePGi0a5dOyMhIcEwDMOIi4sz6tevb5w8edLYsWOH0ahRI+PAgQOGYRjGo48+ajz00EPGxYsXjYyMDKNJkybGzz//bOzYscPo0aOHYRiGsW7dOmPkyJEO1//Pf/7TaNq0qREZGWkUFhYae/bsMUJDQ4us079/f2Pz5s0l7qd3797G2rVrDcMwjF27dhkNGjQwduzYYfz000/Gww8/bOTm5hqGYRgffvihER4ebhiGYfz3v/81xowZYxiGYWRlZRnBwcHG+fPnjYkTJ9r/rVJTU41nnnnGsFqtDh+TiJhDZ7LiFLVq1aJ27dqsXLmS1NRUhg4dam+zWCycOHGCIUOGsGvXLv773/9y/Phxjhw5QosWLTh8+DBubm60bdsWuHQt8/fXVP/yl7/QuHFjAOrUqYOPjw8eHh5Ur16dW265hfPnz5eq9gULFjBz5kzGjBnDyy+/bD+r/CNXV9cr7uPs2bMcOnSIiIgIAFq1akW9evUAqF27Ni+88AJxcXH8+OOP7Nu3j5ycHAD69OnDyy+/TGZmJvHx8fztb3+jSpUqdOvWjfHjx7N//37atm3LlClTcHHR1SARZ9NPoTiFl5cXADabjbZt2xIbG2v/s3btWurVq8f8+fNZsmQJ1apV46GHHqJ9+/YYhoHFYrnseqeb22/fFz08PK7YVhrbtm2zX7u95ZZb6NGjB99++y233nor6enpRWpKSUmhZs2aV9zXr9MJ/n6bX+s8cOAA/fv3Jzs7m/bt2zN8+HD7OlWqVOGBBx5g/fr1rFu3jgEDBgDQpUsXNm7cSPfu3Tl48CA9e/bkxIkT1+W4ReTaKWTFqYKDg9m+fbv9btwvvviCXr16cfHiRb788kuGDBlCREQEfn5+JCQkYLVaqV+/PoZh8MUXXwCwZcuWUp2durq6UlhYeNX1Pv74Y15++WUMwyA/P5+PP/6Y4OBgatasSZ06ddiwYQNwKYxdXFyoX7/+Fffl6+tLkyZNeO+994BLwXr48GEAkpKSaNq0KY888ght2rRhy5Yt9mvBAIMGDeKtt97CMAz7ZA7//Oc/2bBhAz169GD69Ol4e3tz5syZa/43EZHrQ93F4lT16tVj1qxZ/O///i+GYeDm5kZUVBReXl488cQTzJs3j2XLluHq6so999zDiRMncHd35+WXX2bGjBksXLiQRo0a4efnd8013H333SxevJgnnniCl19++YrrTZgwgenTp9OzZ08sFgtdu3Zl8ODBACxcuJCpU6cSFRWFh4cHS5YsuWp37cKFC5k4cSKrV6+mTp063HnnncCl7u9NmzYRFhaGu7s7bdu25fz582RnZ+Pt7U3Dhg2pWrUq/fv3t+9r9OjRTJ48mTVr1uDq6kpISAht2rS55n8TEbk+NAuPyA3mxIkTPPzww8THx+Pp6ensckSkBDqTFfl/x44dIzIysti2unXrsnjx4j+1vx07dlzxkaCgoCAmTZr0Z0tkyZIlrF27lsmTJytgRW4AOpMVERExiW58EhERMYlCVkRExCQKWREREZMoZEVEREyikBURETGJQlZERMQk/wf9GtL6w97wegAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 382 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/" - }, - "id": "II_YtUZg3Ue4", - "outputId": "76948908-4240-4d6b-d1db-38cbeda7d5be" - }, - "outputs": [], - "source": [ - "predictors = expgrad_est.predictors_\n", - "predictors" - ] + "id": "tgLhUlZzOvHC", + "outputId": "59590db9-f5c7-44a0-85d2-bfcf79eab25e" + } + }, + { + "cell_type": "markdown", + "source": [ + "The patients in the group *Unknown* have a substantially lower rate of emergency visits in the prior year, regardless of whether they are readmitted in 30 days. The readmission is still positively correlated with `had_emergency`, but note the large error bars (due to small sample sizes).\n", + "\n", + "We also see that the group with feature value *AfricanAmerican* has a higher rate of emergency visits compared with other groups. However, generally the groups *Caucasian*, *AfricanAmerican* and *Other* follow similar dependence patterns." + ], + "metadata": { + "id": "IQ7zH3Wn416g" + } + }, + { + "cell_type": "markdown", + "source": [ + "We see a similar pattern when `readmit_30_days` is used to predict the rate of (non-emergency) hospital visits in the previous year:" + ], + "metadata": { + "id": "G13abbdS7oM-" + } + }, + { + "cell_type": "code", + "execution_count": 23, + "source": [ + "sns.catplot(y=\"had_inpatient_days\", x=\"readmit_30_days\", hue=\"race\", data=df,\n", + " kind=\"point\", ci=95, dodge=True, join=False);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdwAAAFqCAYAAACwMOudAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABALklEQVR4nO3dd1gU1/4/8PeyCyoiosCi8Xu9alQsgDVSVLCgKCKo0dgi0RjrNVxJ1KhYEDXGhiWWRHMTG1w1EUEsiNgVJFiiWCIYYwvKUhTpLrvz+4OfG7nAuoo7tPfrefJsZs7M7mdJ9M2ZmXOORBAEAURERKRXBuVdABERUXXAwCUiIhIBA5eIiEgEDFwiIiIRMHCJiIhEwMAlIiISgay8C9CHtLQsqNUc7UREFY+lZZ3yLoHKCXu4REREImDgEhERiYCBS0REJAIGLhERkQgYuERERCJg4BIREYmAgUtERCQCBi4REZEIGLhEREQiYOASERGJgIFLREQkAr3OpRweHo7NmzdDqVRi7NixGD16dJH2DRs2YN++fTA1NQUAfPTRRxg9enSp+4mIiCorvQVucnIy1qxZg5CQEBgZGWHEiBGwt7dH8+bNNcdcv34dgYGB6NChQ5FzS9tPROXj6p1URMQ+QD/7xmjX3KK8yyGqlPR2STk6OhoODg4wMzODsbEx3NzcEBERUeSY69evY+vWrRg4cCACAgKQn5+vdT8RlY/Qs3/i9sNnCD37Z3mXQlRp6a2Hq1AoYGlpqdmWy+W4du2aZjs7OxutW7fGV199hUaNGmH27NnYtGkTJk6cWOJ+X19fnT/b3NzknX4XoupOqVJrXrm8HNHb0VvgCkLx9WglEonm32vXro2tW7dqtj/99FPMnTsXvr6+pe7XFdfDJXq3VP8/cFUqNVJSMsu5msqNv7BUX3q7pGxlZYXU1FTNtkKhgFwu12wnJSXhl19+0WwLggCZTFbqfiIiospMb4Hr5OSEmJgYpKenIzc3F5GRkXB2dta016xZEytXrsTDhw8hCAKCgoLQp0+fUvcTERFVZnrrOlpZWcHX1xfe3t5QKpUYOnQo7OzsMGHCBPj4+MDW1hYBAQGYMmUKlEolOnbsiHHjxsHIyKjE/URERJWZRCjpZmslx3u4RO/WnO9jkPw0F1b1amHZJMfyLqdS4z3c6oszTREREYmAgUtERCQCBi4REZEIGLhEREQiYOASERGJgIFLRKVSqdU4ezUJ6c8L5zNPf56Ps1eTOAqA6C1wWBARlUilVuO70Bu4lJBSrK1TS0tMHtQWUgP+zv6mOCyo+uKfFiIqUXT8kxLDFgAuJaQg+voTkSsiqtwYuERUorPXHr9x+/XUW1h7+TtcT72lr7KIKi2uCkBEJUrPzNPe/rx4+8E/I/Ew8y/kqfJhY9FaX6URVUrs4RJRierXqam93bR4e35BfpFXIvobA5eIStTdrmGZ2omoKAYuEZWoq21DdGppWWJbp5aW6GrDwCV6EwxcIiqRgYEEkwe1xTj3VjCUFf5VYSgzwDj3VpgyyAYGBpJyrpCocmHgElGppAYG6G73HurXqQEAqF+nBrrbvcewJXoLDFwiIiIRMHCJiIhEwMAlIiISAQOXiIhIBAxcIiIiETBwiYiIRMDAJSIiEgEDl4iISAQMXCIiIhEwcInotWoayYq8EtGbY+AS0WsN6t4UrRqbYVD3puVdClGlxV9Xiei12jW3QLvmFuVdBlGlxh4uERGRCPQauOHh4XB3d0efPn0QFBRUrH3Dhg3o2bMnvLy84OXlpTnm1q1b+PDDD+Hm5gY/Pz8UFBTos0wiegsFD35DTvgyFDz4rbxLIaoU9HZJOTk5GWvWrEFISAiMjIwwYsQI2Nvbo3nz5ppjrl+/jsDAQHTo0KHIuTNnzsSSJUvQvn17zJ07F3v37sWoUaP0VSoRvYX8i/uhTr2PfGUeZI3bl3c5RBWe3nq40dHRcHBwgJmZGYyNjeHm5oaIiIgix1y/fh1bt27FwIEDERAQgPz8fPz111/Iy8tD+/btAQBDhgwpdh4RlT9BmVfklYi001vgKhQKWFpaarblcjmSk5M129nZ2WjdujW++uor7N+/H8+fP8emTZuKnWdpaVnkPCIiospIb5eUBUEotk8ikWj+vXbt2ti6datm+9NPP8XcuXPh4uKi9TxdmJubvNHxRPTm8qQGUAKQSQ1gaVkHACCVGmheX+4jokJ6C1wrKytcvHhRs61QKCCXyzXbSUlJiI6OxtChQwEUBrRMJoOVlRVSU1M1x6WkpBQ5TxdpaVlQq4sHPhGVnaBWoSDhPJQZhX9OlRmpSDp7CLKW3aBSqQEAKpUaKSmZ5VlmhcVfRKovvV1SdnJyQkxMDNLT05Gbm4vIyEg4Oztr2mvWrImVK1fi4cOHEAQBQUFB6NOnDxo1aoQaNWrg0qVLAIDQ0NAi5xFR+RHUKuRFbULemR8BlbJwp0qJvDM/Ii9qIwTwF12i0ugtcK2srODr6wtvb28MGjQIHh4esLOzw4QJExAfH4/69esjICAAU6ZMQb9+/SAIAsaNGwcAWLVqFZYtW4b+/fsjNzcX3t7e+iqTiN5AQcJ5FNy7VHLbvUvACz5ARVQaiVDSzdZKjpeUifQjO2wJ1Ml3Sm1f1awBUg3UkNeywELHWSJWVnnwknL1xZmmiEhnQla69gPUKnEKIaqEGLhEpDOJSX3tBxhIxSmEqBJi4BKRzoystT/AKDGsJVIlRJUPA5eIdCZr2Q2yJp2K7VcBuNzUGs+EfADA0/xniE6Kg1pQi1whUcXFh6aI6I0IahUKEqORd24HoFJCJTXEntatcS3vSbFj21va4NO2oyHlpWYNPjRVfbGHS0RvRGIghaF1d8393CsW9UoMWwD4LeU6fn1yWczyiCosBi4RlUlcTe3t0Y/jxCmEqIJj4BLRW5EYFiZthlT7XOdP856JUA1RxcfAJaK3UqPzYEgbtkI9Y3Otx9WraSZOQUQVHAOXiN6KrHF7GA+cDacmPbUe59TwA5EqIqrYGLhEVCYODTuhvaVNiW3tLW1g37D4MCKi6ojDgoiozFRqFX59chl7EvZDqS6AoYEMw1sOhn3DTjCQ8Pf6V3FYUPXFPwlEVGZSAykc3/sA9WqYAQDq1TCD43sfMGyJXsE/DURERCJg4BIREYmAgUtERCQCBi4REZEIGLhEREQiYOASERGJgIFLREQkAgYuERGRCBi4REREImDgEhERiYCBS0REJAIGLhERkQgYuERERCJg4BIREYmAgUtERCQCBi4REZEI9Bq44eHhcHd3R58+fRAUFFTqcadOnUKvXr0023FxcbC3t4eXlxe8vLwwZ84cfZZJRESkdzJ9vXFycjLWrFmDkJAQGBkZYcSIEbC3t0fz5s2LHJeamorly5cX2RcfH49PP/0UkyZN0ld5REREotJbDzc6OhoODg4wMzODsbEx3NzcEBERUey4efPmYdq0aUX2xcfH4/z58xg0aBAmT56Mx48f66tMIiIiUeith6tQKGBpaanZlsvluHbtWpFjduzYgTZt2qBdu3ZF9tepUwcDBgyAq6sr/vvf/8LX1xe7d+/W+bPNzU3KVjwRvRWp1EDzamlZp5yrIapY9Ba4giAU2yeRSDT/npCQgMjISGzbtg1PnjwpclxAQIDm30eOHInVq1cjMzMTdero9gc4LS0LanXxzyci/VKp1JrXlJTMcq6mYuIvItWX3i4pW1lZITU1VbOtUCggl8s12xEREUhJScGHH36IiRMnQqFQYNSoUVCr1di8eTNUKlWR95PJ9Pa7ARERkd7pLXCdnJwQExOD9PR05ObmIjIyEs7Ozpp2Hx8fHD16FGFhYdiyZQvkcjmCg4NhYGCAY8eO4ejRowCA0NBQtGvXDrVq1dJXqURERHqn1x6ur68vvL29MWjQIHh4eMDOzg4TJkxAfHy81nOXL1+OHTt2YMCAAdi3bx+WLFmirzKJiIhEIRFKutlayfEeLlH5WBSzAorcVMhrWWCh46zyLqdC4j3c6oszTREREYmAgUtERCQCBi4REZEIGLhEREQiYOASERGJgIFLRO9MDVmNIq9E9DcGLhG9Mx5N+6KFWTN4NO1b3qUQVTgch0tEJCKOw62+2MMlIiISAQOXiIhIBAxcIiIiEbxx4GZlZSEpKUkftRAREVVZOgXusWPHsHjxYmRlZcHT0xNeXl7Yvn27vmsjIiKqMnQK3O+//x4fffQRIiMj0b59e5w8eRIHDhzQd21ERERVhk6BKwgCrK2tER0dDWdnZ5iYmKAKjiYiIiLSG50C18DAAIcPH8a5c+fQtWtXnD59Wt91ERERVSk6Be7s2bOxd+9e+Pr6wtLSEps3b8a8efP0XRsREVGVodNMUydOnEDPnj0hkUjEqKnMONMUEVVUnGmq+tKph7tr1y707t0bmzZtQkpKir5rIiIiqnJ0nkv5wYMH2Lt3Lw4ePAg7OzuMHDkSjo6O+q7vrbCHS0QVFXu41dcbLV6gUqkQFRWFFStWICcnBxYWFli4cCE6d+6szxrfGAOXiCoqBm71JdPloPv372Pv3r0ICwuDtbU15s6di549e+K3337DjBkzcOLECX3XSUREVKnpFLjDhg3D4MGDsWvXLjRp0kSzv2PHjujSpYu+aiMiIqoydLqknJeXh5o1a4pRzzvBS8pEVFHxknL1pVMP99atW9iyZQtycnIgCALUajUePXqEU6dO6bk8IiKiqkGnYUHz5s1Dhw4dkJWVhYEDB8LExAR9+/bVd21ERERVhk49XIlEgokTJ+Lp06do1qwZPD09MXLkSH3XRkREVGXo1MOtXbs2AKBx48ZITExEjRo1oFKp9FoYERFRVaJT4Nra2mL69OlwcHDAjz/+iG+++QZSqfS154WHh8Pd3R19+vRBUFBQqcedOnUKvXr10mw/f/4cEydORP/+/TF69GjObkVERJWeToHr5+eHsWPHomnTppg7dy7UajVWrVql9Zzk5GSsWbMGwcHBCAsLw549e3Dnzp1ix6WmpmL58uVF9q1duxadO3fGkSNHMGzYMCxduvQNvhIREVHFozVwk5KSkJSUhMePH0MulyMpKQktW7bE2LFjXztMKDo6Gg4ODjAzM4OxsTHc3NwQERFR7Lh58+Zh2rRpRfadOnUKAwcOBAB4eHjgzJkzUCqVb/rdiIiIKgytD00NGDAAEokEgiAgLy8PxsbGkMlkeP78OczNzXHu3LlSz1UoFLC0tNRsy+VyXLt2rcgxO3bsQJs2bdCuXbtSz5XJZDAxMUF6ejqsrKze+AsSERFVBFoD98qVKwCABQsWwN7eHgMGDAAAHD9+HFFRUVrfuKT5NF5d3i8hIQGRkZHYtm0bnjx58tpCDQx0uvoNADA3N9H5WCLSr/SLl/DX/jA0GuyF+p07lXc5ROVGp2FB169fR0BAgGa7d+/e2LBhg9ZzrKyscPHiRc22QqGAXC7XbEdERCAlJQUffvghlEolFAoFRo0aheDgYMjlcqSmpqJBgwYoKChAVlYWzMzMdP5SnGmKqOK4vz0I+Q/uI/95FlT/bFne5ZQ7zjRVfenUbVSr1YiNjdVsnzlz5rWL0Ts5OSEmJgbp6enIzc1FZGQknJ2dNe0+Pj44evQowsLCsGXLFsjlcgQHBwMAXFxcEBoaCgA4fPgwOnfuDENDwzf9bkRUAajz84q8ElVXOvVw582bh+nTp8PQ0BCCIEAQBGzcuFHrOVZWVvD19YW3tzeUSiWGDh0KOzs7TJgwAT4+PrC1tS313H//+9+YPXs2BgwYgDp16rz2iWgiIqKKTuf1cJVKJRISEiCRSNCyZUvIZIVZ/eOPP+LTTz/Va5FvipeUiSqOP/2+gjI5GYZWVmi6dPnrT6jieEm5+tL5SSRDQ0O0bdsWbdq00YQtUDi5BREREWmn+6O/pdCxg0xERFStlTlwX/fwFBEREb2DwCUiIqLXY+ASERGJgPdwiYiIRKBT4L6ckOJVW7ZsAQCMHz/+3VZERERUBWmd+OK///0v8vLysG3bNuTn52v2K5VK7Ny5ExMnTtSs6kNERNVHbGwsli5dCmNjY2RnZ6Njx464efMmsrOzIQgClixZgk6dOiE7OxtLlizB5cuXIZVK4erqCl9fXyiVSqxatQpxcXFQqVRo06YN5s2bBxOTqjsXvtbAlclkSEhIQF5eHhISEjT7pVIp5s+fr/fiiIio4kpMTERUVBQUCgV++ukn7NmzBwYGBtiyZQu2bt2KTp06Yf369cjPz8fhw4ehUqnw6aef4tdff0VcXBykUilCQkIgkUgQGBiIVatWwd/fv7y/lt5oDdxhw4Zh2LBhiIqKgqurq1g1ERFRJdCwYUM0atQIjRo1Qt26dbF79248fPgQsbGxqF27NoDCtdHnzJkDqVQKqVSKXbt2AQBWrlyJzMxMREdHAyi8cmpubl5u30UMOs2l3K5dO2zYsAHPnj0rsn/evHn6qImIiCoBY2NjAMCpU6ewdOlSjBs3Dr1790azZs1w4MABAIVXSl+dr+Hx48eoWbMm1Go15s6dCxcXFwBAdnZ2kVuXVZFOD035+vrixo0bqFu3LszMzDT/EBERnT9/Hj179sSoUaNga2uLqKgoqFQqAICjoyP2798PtVqNFy9ewMfHB3FxcejWrRuCgoLw4sULqNVqzJ8/H4GBgeX8TfRLpx5uWlqa5jIAERHRq0aMGIEZM2Zg4MCBkEql6Ny5MyIjI6FWqzFt2jQsXboUXl5eUKlUcHd3R9++feHs7Izly5dj8ODBUKlUaN26NWbPnl3eX0WvdFotaPz48fj22281lw8qOq4WRFRxcLWgorhaUPWlUw9XLpdj0KBB6NKlC2rWrKnZz3u4REREutEpcF8+hUZERERvR6fAnTZtGvLy8nD//n20aNECL168KNLTJSIiIu10ekr56tWrcHV1xaRJk6BQKODi4oLLly/ruzYiIqIqQ6fAXb58ObZt2wYzMzM0aNAAK1aswNKlS/VdGxERUZWhU+Dm5eWhefPmmm0XFxfNGCsiIiJ6PZ0CVyaTISMjQzNbyN27d/VaFBERUVWjU+BOnjwZH3/8MZ48eYIvvvgCI0eOxJQpU/RdGxERAVCp1DgWex+zvj2LcYsjMevbszgWex+qdzzfQEJCAqytrXH06FHNvtOnT6Nnz5748ssvix3v5eX1Tj//JR8fH72uRBcfHw8/Pz+9vX9pdJr4AgDu37+P8+fPQ61Ww8HBocgl5oqGE18QVRyc+KKoN534QqVSY/nOi4iJf1yszdG2Ib4a0xlSqU59p9f65ptvkJSUhMzMTPz0008AgDlz5qB9+/YYPnz4O/mM13n69CmGDRsGCwsLzJw5E506dRLlc8WgdVjQH3/8gffffx83btwAULiIAQDk5+fjxo0baNu2rf4rJCKqxk5cfFhi2AJATPxjnLz0EK5d/lnmzykoKMCBAwcQFBSEESNG4MGDB4iNjcXx48cRExMDAwMDHDhwAHXr1kViYiLWrl2LQYMG4fbt23j27Bn8/Pxw9+5dGBkZYfbs2XB0dMSuXbsQFhaG3NxcSCQSrF27Fu+//z569eoFT09PnDt3Drm5uVi+fDlsbGwAAOHh4ejcuTNatmyJPXv2aAI3JCQEp06dgkKhwJMnT/DJJ58gKSkJFy5cgJmZGX744QfUqFEDoaGh2L59O9RqNdq2bYuFCxeiRo0acHBwQNu2bZGamopZs2bhu+++w86dO3Hr1i0sWLAAeXl5qFu3LlatWgULCwv4+/sjMTERqampaNq0KTZs2IDU1FRMmzYNLVq0wK1bt2Bubo5169bpvLaA1l+LVqxYAQD4/PPPi/3j4+NThv+0RESki2O/PtDaHhmrvV1Xp06dwnvvvYemTZvC1dUVu3fvxrBhw9CrVy/4+Phg2LBhAKC55Ny6dWvNuevWrUPjxo1x5MgRrFixAmvXrkVWVhaioqKwc+dOHDx4EK6urggODtacY2Zmhl9++QUjRozA999/r9kfEhKC/v37o3///jh69GiRVeri4+Pxww8/ICgoCN988w2cnZ0RHh4OADh79iwSExOxd+9e7N69G2FhYTA3N8d//vMfAIU954kTJyIsLAwy2d99zRkzZmDq1KkIDw+Hu7s7tm/fjitXrsDQ0BB79uzBsWPHkJ+fj9OnTwMAfv/9d4wbNw4HDx6Eqamp5vN1obWH+/KHEBwcjAYNGhRpS0xM1PlDiIjo7aQ8yy1Tu65CQkLg4eEBAHB3d8eMGTMwffr0YsfZ2dkV2xcXF4dVq1YBKAzkPXv2AABWr16NQ4cO4d69ezh79myRkO7evTsAoEWLFoiMjAQA3Lp1C48fP4aTkxMMDQ3RunVrhIaGYuzYsQCAjh07wsTEBCYmJgAKVyICCmdDfP78OWJjY3H//n189NFHAArX2G3Tpo3mM19epX0pPT0dKSkp6NmzJwBg1KhRmjYzMzMEBQXh7t27uHfvHnJycgAA5ubmmvds0aIFMjIytP9gX6E1cF/+ZjFx4kTs3LkTgiBAIpFAqVTiX//6l+aHRERE+mFpVgupWkLV0qxWmT8jLS0NZ86cwfXr17Fjxw4IgoDnz5+X+Hd8SbMMvtpjBApvR9asWROffPIJPv74Yzg7O8PCwgK3bt3SHFOjRg0AKLJW7r59+/DixQu4ubkBKFwjd/fu3ZrANTQ01Pq5KpUK/fv318zzn52dXWQI6//W/r/vl5+fD4VCgYSEBKxfvx7e3t4YMmQInj59ipePO72s+2XtOj4GBeA1l5S//PJLODg4ICEhAfb29nBwcIC9vT169OhR5LcGIiLSjz5dGmtt72uvvV0XBw4cgIODA86cOYMTJ07g5MmTmDx5sqan+jqdO3fG4cOHARSG7YQJE3D9+nX885//xNixY9GuXTucOXNG6/wNL168QHh4OLZt24YTJ07gxIkTOH78OFJSUhAbG6tTHfb29jh27BjS0tIgCAL8/f2xffv2Uo+vU6cOGjRogPPnzwMAwsLCsG7dOsTExKB///748MMPYWFhgbi4uHcy94TWHu7La99z5szBsmXLyvxhRET0Znp90Bhxt5JLfUq5Z+eyB25ISAh8fX2L7Bs1ahR++OEHWFhYvPZ8Hx8fzJs3D56enpDJZFixYgVat26N3bt3w93dHUZGRrCzs9N6K/LkyZNo1KhRkcu+JiYmGDZsGHbv3q25BK1Nq1atMG3aNHzyySdQq9Vo3bo1Jk6cqPWclStXwt/fHytWrEC9evWwYsUKPH36FDNmzEBERASMjIzQvn17PHr06LWf/zo6Dwt69uwZcnNzIQgCVCoVHjx4gK5du2o9Jzw8HJs3b4ZSqcTYsWMxevToIu3Hjh3D+vXroVarYWtri4CAABgZGSE0NBSrVq2Cubk5AKBHjx7F/mfQhsOCiCoODgsq6m3Ww1Wp1Dh56SEiYx8g5VkuLM1qoa99Y/Ts3BhSA8nr34AqBJ1WC1q/fr3mASqpVAqlUonmzZtrfTorOTkZa9asQUhICIyMjDBixAjY29trxu/m5OQgICAA+/fvh4WFBXx9fbF//34MHz4c8fHxmD17tuYGPhFRdSaVGsC1yz/fyfAfKj86jZYODQ3FyZMn4ebmhsjISHzzzTevnfgiOjoaDg4OMDMzg7GxMdzc3BAREaFpNzY2xokTJ2BhYYGcnBykpaXB1NQUQOGj36GhofD09MSMGTPe6CkwIiKiikinHm79+vUhl8vRrFkz/P777/Dy8tJ6IxoAFAoFLC0tNdtyuRzXrl0rcoyhoSFOnz6NWbNmQS6Xo1u3bgAAS0tLTJw4EXZ2dggMDERAQABWr16t85cyNzfR+Vgi0q8HUgMoUdhLe5vLqURVhU6BK5PJ8ODBAzRr1gwXL15Et27d8Pz5c63nlHRr+NXHv19ycXFBbGwsAgMD4e/vj9WrV2Pjxo2a9s8++wyurq66lKnBe7hEFYdKpda8pqRklnM15Y+/dFRfOl1SnjRpEubPn48ePXogMjISPXr0gIODg9ZzrKyskJqaqtlWKBSQy+Wa7WfPnuHcuXOa7YEDB+L27dvIzMzEtm3bNPsFQSg21oqIiKiy0Slwe/bsie3bt8PY2BhhYWH44YcfsHjxYq3nODk5ISYmBunp6cjNzUVkZCScnZ017YIgYObMmUhKSgIAHDlyBB07doSxsTF++OEHXL16FQCwa9cu9OnT522/HxERUYWgU9cxOzsbGzduxLlz5yCVStGrVy80a9YMRkZGpZ5jZWUFX19feHt7Q6lUYujQobCzs8OECRPg4+MDW1tbLF68GJMmTYJEIkHz5s2xaNEiSKVSrF27Fv7+/sjLy0OTJk00czoTEdG7l5WVhdWrVyMuLg5SqRSmpqaYPXt2uS1Q4+XlhbCwsHL5bH3SaRyur68vDAwMMHToUAiCgL1798LU1BQBAQFi1PjGeA+XqOLgONyi3uYerqBWIfPaKWRePY6C52mQmZqjTrveqGPXAxIDaZnqUavVGD16NOzt7TFt2jTIZDJcuHABX3zxBQ4dOoR69eqV6f3pbzr1cG/evFlkQWIHBwcMGDBAb0VR2V29k4qI2AfoZ98Y7Zq/fqYYIqqYBLUKySGrkXP77+kNVc9Tkf/oNnLuXILVkC/LFLqxsbFQKBTw8fGBgUHhXUYHBwcsW7YMarUa8+bNK3GZOm9vb5w4cQIA8O233wIoXFnu5YRHEolEcyUzPT0dc+fORWZmJlJSUjBgwADMmDEDv//+OxYsWICCggLUqFEDy5YtQ5MmTWBtbY3bt28jOTm5xPNCQkJw9uxZZGRk4OHDh+jatSv8/f3f/ocsEp3u4VpYWCA9PV2znZOTw996KrjQs3/i9sNnCD37Z3mXQkRlkHntVJGwfVXO7VhkxZ8u0/vfvHkTtra2mrB9ycXFBXfv3i11mbqSJCcnY9myZfjxxx9x6NAhqFQqnD59GgcPHoSHhwf27t2LAwcOIDg4GOnp6di+fTvGjRuHkJAQjBkzBr/99luR9yvtPAC4cuUK1q9fjwMHDuDkyZO4fft2mX4OYtCph2thYYEPP/wQ/fr1g1QqxfHjx2FhYYElS5YAgGZlBqo48l4UFHklosop8+pxre3PfzuOOu16vfX7GxgYlLrizQcffFDqMnUluXLlCjp27KhZznXlypWatgsXLuA///kPEhMToVQqkZubCxcXFwQEBODs2bPo2bOnZpWgl8aPH1/ieQDQoUMHzTJ9//jHPyrFBEk6BW6LFi3QokULzTYvJ1dcKrUa0fFPkP48HwCQ/jwfZ68moattQxhwzlWiSqfgedpr2lO1tr+OjY0NgoODNcuvvhQYGAg7Ozt8++23xZap+99l6QoKCiCTyYoN4XzZG92yZQsePnwIDw8PuLq6Ijo6GoIgoF+/fujQoQNOnjyJ7du34/Tp05qOHAB88803JZ4HlG2ZvPKiU+BOmzZN33XQO6BSq/Fd6A1cSkjR7FOq1PjpyO+49kcaJg9qC6mBTncRiKiCkJmaQ6UlVGWmZXtGo3PnzjA3N8eGDRswdepUSKVSnD17FiEhIcjIyNAsU5ecnIy4uDg4OjrC1NQUGRkZSE9Ph4mJiaaHamtri0WLFiElJQWWlpb4+uuvYW9vj/Pnz2PRokXo2LEjLly4gOTkZKjVakyfPh0DBgzAiBEj8P777xdbla608yornQL3yJEjWLlyJTIyMor8FnH58mW9FUZvLjr+SZGwfdWlhBREX3+C7nbviVwVEZVFnXa9kf+o9PuTpu17l+n9JRIJNm3ahGXLlsHDwwMymQz16tXDli1bIJVKS1ymrk6dOhg/fjyGDh2KBg0awNbWFkDhcFA/Pz+MHz8earUa7du3x5AhQ1CrVi3MmjULpqamMDc3h42NDR49eoTJkyfDz88PmzZtglQqxezZs4vUNmnSpBLPq6x0Ghbk5uaGL7/8Em3atClyyaFRo0Z6Le5tVddhQV/vvIQ7f5V+H6P5/9XF3I87iVgRVWeCSoXnMeehCNoJQamExNAQ8tFjYOrUDZJqfKXlTYcFlfSU8kvG1vZlfkqZxKNTD9fU1BR9+/bVdy1URumZedrbn2tvJ3pXBJUKj7/fhKzLl/7ep1QieduPyL52FQ0nTYVEypDQhcRACqshXyIr/jSe/3YcBc9TITO1gGn73jCxdWHYViI6BW67du1w+vRpuLi46LseKoP6dWpqHpYqsd20pojVUHX2POZ8kbB9VdblS3geE4263bqLXFXlJTGQok67XmV6GpnKn06Be/r0aezatQuGhoYwNDTUPKXGe7gVS3e7hlovKXe3ayhiNVSdZZw9o7393BkGLlU7OgXuq6v3UMXV1bYhrv2RVuKDU51aWqKrDQOXxFHwNF17e7r2oS5EVZHWwI2JiYGjoyNu3LhRYntFfWiqujIwkGDyoLaIvv4EuyIToCxQw1BmgI/7tkRXG47DJfHI6tVHQXrpoSurby5iNUQVg9bAPXToEBwdHbFz585ibRKJhA9SVUBSAwN0t3sPJy79hfvJmXjPvDaHApHo6nZ3Rt4fd0pv7+ZcahtRVaXTsCBtvvnmm2Jjp8pbdR0W9Kqrd1Jx9NcHcOvCxQtIfIJajcffbSzxwSmTjp3QcPK/qu3QoLdZLYiqhjIH7uDBg7F///53Vc87wcAlKn+F43CjoQja8co4XG+YOnWttmELvF3gqtQqnL4Xi5N3zyM19yksatVDz2Zd0aOJQ7FFB95GdnY2Vq1ahXPnzqFWrVowMTHB559/DkdHR+zZswe1a9eGh4cHZs+ejS5dumDIkCFl/szqSKeHprSpDPNXVneCWoWChPN4cfsMhKx0SEzqw8jaGbKW1XsCAtIviVSKut26I/3IQSiTkyGrX59PJr8FlVqFNTE/4NdHv2n2peU8xe20u7j8OB6+jp9BWoaxuIIgYPLkyWjdujUOHToEIyMj3Lx5ExMnTsTq1atx5coVdOnS5R18Eypz4L468xRVPIJahbyoTSi498oEBNnpyEu+A9mDq6jpOpUD54kqsNP3YouE7at+ffQbztyLRc9mTm/9/r/++iuSkpKwY8cOzd/nbdq0wZQpUzB27FjUqVMHFy5cgKWlJQDg1KlTCA4ORlpaGiZPnozhw4cjOzsbAQEBSExMhEqlwoQJE+Dh4YGQkBDs378fz549Q8+ePfHFF1+8dZ1VQZkDlyq2goTzRcK2SNu9SyhIjIahNXsdRBXVybvntbaf+DO6TIEbHx8PGxubYp2nDz74ALVq1UKvXr3QpUsXdO/eHYcOHcKLFy/w888/IzExEd7e3hg+fDg2b96Mtm3bYvny5cjKysKIESPQrl07AIVr5B4+fLjYSkLVEX8CVdyL29onIFD+foaBS1SBpeY+1d6eo33M8+tIJBKoVKpi+5VKZYnH9+7dGxKJBC1atMDTp4W1RUdHIy8vD/v27QMA5OTkIDExEUBhb5lhW4j3cKs4IUv7H0Z1FicgIKrILGrVQ1pO6aFrYVy/TO/frl077Ny5E0qlEoaGhpr9v/32m2YVoFdJ//8c2K/2iNVqNVauXIm2bdsCAFJTU1G3bl2Eh4ejZk1OKftSmZ+YmTNnzruog/REYqL9D6OBCScgIKrIejbrqrW9V9O3v5wMFK6H27x5c3z99deaXu3169exefNmzfq4JfWAX+Xg4ID//ve/AACFQgFPT088fvy4THVVRVp7uK1atdL6UNStW7dgb2//zouid8fI2hl5yaVPQGDYihMQEFVkPZo44PLj+BIfnOryf+3h0sShzJ+xYcMGrFmzBh4eHpBKpahbty5WrlwJe3t7pKamIjAwEHXqlD6cadq0afD394eHhwdUKhVmzpyJxo0b4+LFi2WurSrROg736dOnEAQB69atQ6NGjTB8+HBIpVKEhIQgKSmpwk148RLH4f5NUKuRF7WxxAenZE06oaZr9Z2AgMTxp99XUCYnw9DKCk2XLi/vcsrd247DPXMvFif+jEZqTjosjOujV1MnuLyjcbgkDq093Hr16gEovLywaNEizX5vb28OfK4kJAYGqOk6FQWJ0Xhx9QjUzxUwMJXDqF1/yFpU7wkIiCoLqYEUPZs5lelpZCp/Oj00lZubi7t376JZs2YAgNu3b5f6BBtVPBIDKQytu/NpZCKicqRT4E6fPh3Dhw+HtbU11Go1/vjjD6xatUrftREREVUZOgVu37590alTJ1y6dAkSiQSdOnVC/fplexSdiIioOtHpBt6LFy9w+fJl5OTkIDs7GydPnsSaNWv0XRsREVGVoVMP19fXFw8fPkRKSgratGmDq1evcjJrIiKiN6BTD/fWrVsICQlB7969MXfuXOzevRuZmZmvPS88PBzu7u7o06cPgoKCirUfO3YMAwcOxIABAzB79my8ePECAJCUlITRo0ejX79+mDJlCrKzs9/waxERka4ePXqEXr16FdtvbW1d6jmxsbEYM2aMPsuqcnQKXLlcDplMhiZNmiAhIQHNmzdHbm6u1nOSk5OxZs0aBAcHIywsDHv27MGdO39PwJCTk4OAgAD89NNPOHToEPLz8zXr6i5atAijRo1CREQEbGxssGnTpjJ8RSKiyk1QqZAcdRzXvpqLuPGTcO2ruUiOOg7hNTNAUcWi0yVlY2NjhIeHo1WrVti7dy+aNWuGZ8+eaT0nOjoaDg4OMDMzAwC4ubkhIiIC06ZN07zniRMnYGhoiJycHKSlpcHU1BRKpRJxcXHYuHEjAGDIkCH4+OOPMXPmzLf/lkRElZSgUuH2ytVIi4nV7HuRmorM32/j6cVLsJ75JSRS/S2xGRISgrNnzyIjIwMPHz5E165d4e/vX+SY7du3IyoqClu2bMHEiRNha2uLS5cuIT09HfPmzYOLiwtSU1Ph5+eHpKQkyGQy+Pr6wsbGBp6enjh37hwAoHv37pgzZw7c3d2xZcsWSCQS5OXlITk5Gffv38dff/2FYcOGYcqUKXr7vvqkU+AuWLAAP//8M2bOnIl9+/ZhzJgx8PX11XqOQqHQrJ8IFPaSr127VuQYQ0NDnD59GrNmzYJcLke3bt3w9OlTmJiYaFaXsLS0RHJy8ht9KXNzkzc6noj054HUAEoAUqnBW82yVN0pTp4qEravSouJheLkaVi5Fr8c/C5duXIFBw8ehFQqRb9+/TBy5EhN2759+xAZGYmtW7eiVq1aAApXGtqzZw9OnDiBdevWwcXFBYsXL4aDgwPGjRuHhw8fYuTIkQgNDcV7772HhIQEzZzNv/76K9zd3XHmzBn4+/vjyJEjuH37NoKCgpCZmQlXV1eMHj0apqamev3O+qBT4KalpeHGjRvw9PSEIAiwsrJCcHBwkR/6/yppxsiS5mV2cXFBbGwsAgMD4e/vj1mzZul0nvZ6ObUjUUWhUqk1rykpr3/2o6p70186ko8d194edbzMgVvS9JCCIGj+7u3QoQNMTAo7Mv/4xz+QkZEBAEhISMCCBQsQGBgIY2NjzbnduxdOstOiRQvN1dALFy5gyZIlmvdo164drl69ChcXF8TExEAmk8Hb2xuHDh1CZmYmUlJS0Lx5cwCAvb09jIyMYG5uDjMzM2RmZlbdwJ0/fz4++ugjtG7dWufws7KyKjJxtUKhgFwu12w/e/YM169fR7du3QAAAwcOhK+vL+rXr4+srCyoVCpIpVKkpKQUOY+IqDrJT9W+hGZ+SmqZP8PU1LTYg7BpaWmoW7cuAKBGjRqa/RKJRNOhql27Nr7++mt8/fXX6N69uyZ0Xx7/al78bydMEASoVCq4uLhgw4YNMDIywr///W8cOXIE4eHhmtDW9vmVjU4PTRkZGWHs2LGwt7dHly5dNP9o4+TkhJiYGKSnpyM3NxeRkZFwdv57ZRpBEDBz5kwkJSUBAI4cOYKOHTvC0NAQnTt3xuHDhwEAoaGhRc4jIqpOalhoX0KzhqVFmT/DxMQE//znP3H06FHNvj179sDR0VHreY0aNULv3r3RpUsXrF+/XuuxDg4O+OWXXwAADx8+xOXLl9G+fXu0bdsWf/75J+7du4f3338f9vb22Lx5M3r06FHm71XR6BS4zZo1Q3x8/Bu9sZWVFXx9feHt7Y1BgwbBw8MDdnZ2mDBhAuLj41GvXj0sXrwYkyZNgqenJ+7du6d5MGrhwoXYu3cv3N3dcfHiRUyfPv2NvxgRUVVg1ae39nZX7e26WrlyJYKDg+Hp6Yn+/fsjMTERCxYs0OncWbNmITw8HDdu3Cj1GD8/P1y4cAEDBw7E1KlTsWTJEsjlcs3she+//z6AwmDOysqqknM9aF2eb+DAgQCA7OxsKBQK/OMf/9A8zAQUjrOtiHgPl6ji4PJ8Rb3pPdySnlJ+ydzRXu9PKdO7o/Ue7vz588Wqg4iISiCRSmE980soTp5GctRx5KekooalBaxce0Pe04VhW4loDdyq2KUnIqpsJFIprFx76X34D+kXVx8nIiISAQOXiIhIBAxcIiIiETBwiYiIRMDAJSIiEgEDt5q7nnoLay9/h+upt8q7FCKiKk2nuZSp6jr4ZyQeZv6FPFU+bCxal3c5RERVFnu41Vx+QX6RVyIi0g8GLhERkQgYuERERCJg4BIREYmAgUtERCQCBi4REZEIGLjVlEqtQnRSHJ7mPwMAPM1/huikOKgFdfkWRkRURXEcbjWkUqvw440g/JZyXbNPqS5A0O8/40baLXzadjSkBlxjk4joXWIPtxqKfXK5SNi+6reU6/j1yWWRKyIiqvoYuNVQzONftbZHP44TqRIiouqDgVsNPc3LeE37M3EKISKqRhi41VC9mnVf024mTiFERNUIA7cacmzYRWu7U8MPRKqEiKj6YOBWQw4NO6G9pU2Jbe0tbWDfsJPIFRERVX0M3GrIQGKAT9uOxsethsHQoHBkmKGBDB+3GobxNh/DQML/LYiI3jX+zVpNSQ2kcHzvA9SrYQYAqFfDDI7vfcCwpXfOoEbNIq9E1RX/diUivTIfNBi1rFvBfNDg8i6FqFxxpiki0isTu/YwsWtf3mUQlTv2cImIiESg18ANDw+Hu7s7+vTpg6CgoGLtUVFR8PLygqenJ6ZOnYqMjMIJGUJDQ9GtWzd4eXnBy8sLa9as0WeZREREeqe3S8rJyclYs2YNQkJCYGRkhBEjRsDe3h7NmzcHAGRlZcHf3x/79u2DlZUV1q1bh2+//Rbz5s1DfHw8Zs+eDQ8PD32VR/9fDVmNIq9ERKQfeuvhRkdHw8HBAWZmZjA2NoabmxsiIiI07UqlEv7+/rCysgIAWFtb4/HjxwCA+Ph4hIaGwtPTEzNmzND0fOnd82jaFy3MmsGjad/yLoWIqErTW+AqFApYWlpqtuVyOZKTkzXb9erVg6urKwAgLy8PW7Zs0WxbWlri888/R1hYGBo2bIiAgAB9lVnt2Vi0xvSOk2Fj0bq8SyEiqtL0dklZEIRi+yQSSbF9mZmZmDp1Klq1aoXBgwuHDWzcuFHT/tlnn2mCWFfm5iZvWC0REZF+6S1wrayscPHiRc22QqGAXC4vcoxCocD48ePh4OCAuXPnAigM4H379mHs2LEACoNbJnuzMtPSsqBWFw98IqLyZmlZp7xLoHKit0vKTk5OiImJQXp6OnJzcxEZGQlnZ2dNu0qlwuTJk9G/f3/4+flper/Gxsb44YcfcPXqVQDArl270KdPH32VSUREJAq99nB9fX3h7e0NpVKJoUOHws7ODhMmTICPjw+ePHmCmzdvQqVS4ejRowAAGxsbLF26FGvXroW/vz/y8vLQpEkTrFixQl9lEhERiUIilHSztZLjJWUiqqh4Sbn64kxTREREImDgEhERiYCBS0REJAIGLhERkQgYuERERCJg4BIREYmAgUtERCQCBi4REZEIGLhEREQiYOASERGJgIFLREQkAgYuERGRCBi4REREImDgEhERiYCBS0REJAIGLhERkQgYuERERCJg4BIREYmAgUtERCQCBi4REZEIGLhEREQiYOASERGJgIFLREQkAgYuERGRCBi4REREImDgEhERiYCBS8VkXfsND1csQ9a138q7FCKiKkNW3gVQxZMWuh/5D+5DnZcHE7v25V0OEVGVwB4uFaPOzyvySkREZafXwA0PD4e7uzv69OmDoKCgYu1RUVHw8vKCp6cnpk6dioyMDABAUlISRo8ejX79+mHKlCnIzs7WZ5lERER6p7fATU5Oxpo1axAcHIywsDDs2bMHd+7c0bRnZWXB398fW7ZswYEDB2BtbY1vv/0WALBo0SKMGjUKERERsLGxwaZNm/RVJhERkSj0FrjR0dFwcHCAmZkZjI2N4ebmhoiICE27UqmEv78/rKysAADW1tZ4/PgxlEol4uLi4ObmBgAYMmRIkfOIiIgqI709NKVQKGBpaanZlsvluHbtmma7Xr16cHV1BQDk5eVhy5YtGDNmDJ4+fQoTExPIZIWlWVpaIjk5+Y0+29zc5B18g+rrgdQASgBSqQEsLeuUdzlERFWC3gJXEIRi+yQSSbF9mZmZmDp1Klq1aoXBgweXGK4lnadNWloW1Orin0+6UanUmteUlMxyroaoauEvsdWX3i4pW1lZITU1VbOtUCggl8uLHKNQKDBq1Ci0atUKS5cuBQDUr18fWVlZUKlUAICUlJRi5xEREVU2egtcJycnxMTEID09Hbm5uYiMjISzs7OmXaVSYfLkyejfvz/8/Pw0vVhDQ0N07twZhw8fBgCEhoYWOY+IiKgy0tslZSsrK/j6+sLb2xtKpRJDhw6FnZ0dJkyYAB8fHzx58gQ3b96ESqXC0aNHAQA2NjZYunQpFi5ciNmzZ2Pz5s1o2LAhAgMD9VUmERGRKCRCSTdbKznew307gkqF5zHnoQjaCUGphMTQEPLRY2Dq1A0SA86RQvQu8B5u9cXAJQCFYfv4+03IunypWJtJx05oOGkqJFJpOVRGVLUwcKsvdlsIAPA85nyJYQsAWZcv4XlMtMgVERFVLQxcAgBknD2jvf2c9nYiItKOgUsAgIKn6drb09NEqoSIqGpi4BIAQFavvvb2+uYiVUJEVDUxcAkAULe79rHOdbtxLDQRUVkwcAkAYOrUDSYdO5XYZtKxE0yduopcERFR1cJhQaRROA43GoqgHa+Mw/WGqVNXjsMlekc4LKj64t+ipCGRSlG3W3fI6hfez5XVr4+63bozbImI3gH+TUpERCQCBi4REZEIGLhEREQiYOASERGJgIFLREQkAgYuERGRCBi4REREImDgUjEGNWoWeSUiorJj4FIx5oMGo5Z1K5gPGlzepRARVRmc2pGISESc2rH6Yg+XiIhIBAxcIiIiETBwiYiIRMDAJSIiEgEDl4iISAQMXCIiIhEwcImIiETAwCUiIhIBA5eIiEgEDFwiIiIRyMq7AH0wMJCUdwlERERFVMm5lImIiCoaXlImIiISAQOXiIhIBAxcIiIiETBwiYiIRMDAJSIiEgEDl4iISAQMXCIiIhEwcImIiETAwCUiIhIBA7caCg8Ph7u7O/r06YOgoKDyLoeqgKysLHh4eODRo0flXQpRhcXArWaSk5OxZs0aBAcHIywsDHv27MGdO3fKuyyqxK5evYqRI0fi3r175V0KUYXGwK1moqOj4eDgADMzMxgbG8PNzQ0RERHlXRZVYnv37sXChQshl8vLuxSiCq1KrhZEpVMoFLC0tNRsy+VyXLt2rRwrospu6dKl5V0CUaXAHm41U9LiUBIJlzMkItI3Bm41Y2VlhdTUVM22QqHgpUAiIhEwcKsZJycnxMTEID09Hbm5uYiMjISzs3N5l0VEVOXxHm41Y2VlBV9fX3h7e0OpVGLo0KGws7Mr77KIiKo8iVDSTT0iIiJ6p3hJmYiISAQMXCIiIhEwcImIiETAwCUiIhIBA5eIiEgEDFyq1Dp06PBGK9TEx8fDx8cHAHDt2jUsWLDgtedkZmbCx8cHHh4ecHd3x5YtWzRt9+7dw6hRo+Du7o6hQ4fijz/+eOPv4OHhgdjY2Dc+j4gqFwYuVSu2trZYv349AODOnTtITk5+7Tnr1q2DlZUVDh48iF9++QW7d+/GlStXAAAzZszAyJEjcfjwYXz++efw8fEpcfpMIiJOfEGiiY2NxdKlS2FsbIycnBz4+Pjg+++/h1KpRM2aNfHVV1+hQ4cOSE1NxYIFC5CWloaUlBQ0atQIa9euhbm5OS5evIjFixdDIpHA1tYWarVa896BgYGQy+VITExErVq18Pnnn2Pnzp34888/0bdvX8ydOxexsbFYvHgxtm7divXr1yMzMxNz5szBsmXLSq3bz88PKpUKAJCSkoIXL16gTp06SE5Oxt27dzFgwAAAgIuLCxYtWoSbN2+ibdu2pb7fnTt3MHfuXOTm5qJZs2bIycnRtH333XeIiopCfn4+cnNz8dVXX8HV1RX9+vXD/Pnz0a1bNwDAvHnz0KJFC3Tr1g1+fn548eIFBEHA0KFDMXr06DL/tyIiPRCIRHLhwgWhVatWwqNHj4Q///xT8PDwENLT0wVBEISEhASha9euQnZ2trBt2zbh+++/FwRBENRqtfDZZ58J//nPf4T8/HzByclJiI6OFgRBEMLDw4WWLVsKDx8+FC5cuCC0bt1auHHjhiAIgjB+/Hhh+PDhQn5+vpCWlia0bdtWePLkiXDhwgVhwIABgiAIwr59+4SJEyfqXP+XX34p2NjYCL6+vkJBQYFw5coVwc3NrcgxI0aMEKKiorS+j5eXl7B3715BEATh4sWLgrW1tXDhwgXh0aNHwpgxY4Tc3FxBEATh4MGDgoeHhyAIgvDTTz8JPj4+giAIQmZmpuDg4CBkZGQIc+bM0fysFAqFMH36dEGlUun8nYhIPOzhkqgaNmyIRo0aISgoCAqFAmPHjtW0SSQSPHjwAJ988gkuXryIn376Cffu3UNiYiLatWuHhIQEyGQyODo6Aii89/nqPdj/+7//Q5s2bQAAjRs3Rp06dWBkZIT69eujdu3ayMjIKFPtq1atwqJFi+Dj44ONGzdqepv/SyqVlvoeT58+xe3btzFo0CAAQKdOndCiRQsAQKNGjbB8+XKEh4fj/v37uHr1KrKzswEAQ4YMwcaNG5Geno6IiAj06NEDpqam6NOnD7766itcu3YNjo6OmDdvHgwMeKeIqCLin0wSlbGxMQBArVbD0dERYWFhmn/27t2LFi1aYOXKlVi3bh3q1auH4cOHo2vXrhAEARKJpNj9UZns798ZjYyMSm0ri7Nnz2ru9dauXRsDBgzAzZs38d577yE1NbVITcnJyWjQoEGp7/VyKcRXz3lZ540bNzBixAhkZWWha9eu+OyzzzTHmJqaol+/fjhw4AD27duHkSNHAgB69uyJo0ePon///rh16xYGDhyIBw8evJPvTUTvFgOXyoWDgwPOnz+vear39OnT8PT0RH5+Ps6dO4dPPvkEgwYNgrm5OaKjo6FSqdCyZUsIgoDTp08DAI4fP16mXqtUKkVBQcFrjzty5Ag2btwIQRDw4sULHDlyBA4ODmjQoAEaN26Mw4cPAygMZgMDA7Rs2bLU9zIzM0Pbtm3x888/AygM2YSEBABAXFwcbGxsMG7cOHTp0gXHjx/X3DsGgNGjR2PHjh0QBEGz4MSXX36Jw4cPY8CAAVi4cCFMTEzw+PHjt/6ZEJH+8JIylYsWLVogICAAX3zxBQRBgEwmw+bNm2FsbIx//etfWLFiBTZt2gSpVIqOHTviwYMHMDQ0xMaNG+Hv74/AwEC0bt0a5ubmb11Dhw4dsHbtWvzrX//Cxo0bSz1u9uzZWLhwIQYOHAiJRILevXvD29sbABAYGIj58+dj8+bNMDIywrp16157STcwMBBz5szB7t270bhxYzRr1gxA4SXyyMhIuLu7w9DQEI6OjsjIyEBWVhZMTEzQqlUr1K1bFyNGjNC819SpU+Hn54c9e/ZAKpXC1dUVXbp0eeufCRHpD1cLIqokHjx4gDFjxiAiIgK1atUq73KI6A2xh0vV3t27d+Hr61tiW9OmTbF27do3er8LFy6UOszI3t4ec+fOfdMSsW7dOuzduxd+fn4MW6JKij1cIiIiEfChKSIiIhEwcImIiETAwCUiIhIBA5eIiEgEDFwiIiIRMHCJiIhE8P8ArLVKLnmkJq0AAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 382 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "K3Bh2IVm4Ynj" - }, - "outputs": [], - "source": [ - "# Collect predictions by all predictors and calculate balanced error\n", - "# as well as the false negative difference for all of them\n", - "\n", - "sweep_preds = [clf.predict(X_test) for clf in predictors]\n", - "balanced_error_sweep = [1-balanced_accuracy_score(Y_test, Y_sweep) for Y_sweep in sweep_preds]\n", - "fnr_diff_sweep = [false_negative_rate_difference(Y_test, Y_sweep, sensitive_features=A_test) for Y_sweep in sweep_preds]" - ] + "id": "Kx52yWYNQuSH", + "outputId": "d0acd516-57d5-4910-feaa-f3b1c1176900" + } + }, + { + "cell_type": "markdown", + "source": [ + "Again, for *Unknown* the rate of (non-emergency) hospital visits in the previous year is lower than for other groups.In all groups there is a strong positive correlation between `readmit_30_days` and `had_inpatient_days`.\n", + "\n", + "In all cases, we see that readmission in 30 days is predictive of our two measurements of general patient health.\n", + "\n", + "The analysis is also surfacing the fact that patients with the value of race *Unknown* have fewer hospital visits in the preceding year (both emergency and non-emergency) than other groups. In practice, this would be a good reason to reach out to health professionals to investigate this patient cohort, to make sure that we understand why there is the systematic difference.\n", + "\n", + "Note that we have only investigated _predictive validity_, but there are other important aspects of construct validity which we may want to establish (see [Jacobs and Wallach, 2021](https://arxiv.org/abs/1912.05511))." + ], + "metadata": { + "id": "HrCelj5_hR6a" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "### Exercise" + ], + "metadata": { + "id": "V_4CZt-UBaOQ" + } + }, + { + "cell_type": "markdown", + "source": [ + "Check the predictive validity with respect to `gender` and `age`. Do you see any differences? Can you form a hypothesis why?" + ], + "metadata": { + "id": "rW9ktRlqBhZA" + } + }, + { + "cell_type": "code", + "execution_count": 24, + "source": [ + "# Check for predictive validity by gender\n", + "sns.catplot(y=\"had_inpatient_days\",x=\"readmit_30_days\",hue=\"gender\", data=df,\n", + " kind=\"point\", ci=95, dodge=True, join=False);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdsAAAFqCAYAAABS7PDkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxfUlEQVR4nO3deViVdf7/8deBAyqCkbKUNlamae7bV3AlF1JBxNRK7Ss1lbYOxVSGipaaV6PlOmOmNd+s1EnGXKKUSE1TQVMrNMdcMrcsEEgEAYFz7t8f/jzFgHRUblnO83FdXnjfn3t538dLXudzL5/bYhiGIQAAYBq3yi4AAICajrAFAMBkhC0AACYjbAEAMBlhCwCAyQhbAABMZq3sAsyQmZkru50nmgBUPf7+PpVdAioBPVsAAExG2AIAYDLCFgAAkxG2AACYjLAFAMBkhC0AACYjbAEAMBlhCwCAyQhbAABMRtgCAGAywhYAAJMRtgAAmIywBeC01CMZmrHsa6UeySgxv/jEt8pLeE3FJ76tnMKAKq5GvvUHgDnWbP1Rx9NyVFBoU7umfo75F3avlj3juC4UFcjauH3lFQhUUfRsATitoLC4xM9LjKKCEj8BlETYAgBgMsIWAACTEbYAAJiMsAUAwGSELQAAJiNsAQAwGWELAIDJCFsAAExG2AIAYDJTwzYhIUFhYWEKDQ3VsmXLSrX/4x//UO/evRUZGanIyEjHMgcOHNCwYcPUv39/TZw4UcXFxaXWBQCgujBtbOS0tDTNmTNHq1atkqenp0aMGKGgoCA1bdrUscx3332n2bNnq0OHDiXWffHFF/Xqq6+qffv2mjBhguLj4zVq1CizSgUAwFSm9WyTk5MVHBwsX19feXl5qX///kpMTCyxzHfffae3335bERERmjp1qi5cuKCffvpJBQUFat++vSRp6NChpdYDcH3Z7HZtTT2trHMXJElZ5y5oa+pp2e1GJVcGVA+m9WzT09Pl7+/vmA4ICNDevXsd0+fPn9ddd92ll156SY0aNVJsbKzefPNN3X333SXW8/f3V1pa2hXtu0ED72s/AACSJJvNrhkf7FbKvp8d84psdr27/nsdPPmrnmqXK53PuthwPku1fvpKPm3vlsXNvZIqBqoe08LWMEp/47VYLI6/161bV2+//bZj+pFHHtGECRMUEhJS7nrOyMzM5Rs3UEG2pp4uEbSXuMmuFsdWKPP0Ccc8o7hIGZ++qbP7d6p2v6cI3DL4+/tUdgmoBKadRg4MDFRGxm8vmE5PT1dAQIBj+vTp01q5cqVj2jAMWa3WUuudOXOmxHoArq+te0sHrST9j+cPaud5QmV9rS0+tkfFh5PNLQyoRkwL227duiklJUVZWVnKz89XUlKSevXq5WivXbu2Xn/9dZ08eVKGYWjZsmUKDQ1Vo0aNVKtWLe3Zs0eStGbNmhLrAbi+snLKfkdtcK0jkqTLnXcq+v5LkyoCqh/TTiMHBgYqJiZGUVFRKioq0vDhw9W2bVuNGTNG0dHRatOmjaZOnaonn3xSRUVF6tixo/785z9Lkt544w3FxcXp/PnzatmypaKioswqE8AfqO9T23Fj1O/d6Ha+3PXsuZlmlQRUOxajrIur1RzXbIGKszX1tN5d/32p+c/6rFcTjzOXXc89sJm8IieaWVq1xDVb18QIUgDK1b3Nzep0p3+p+TsuNC1j6d94tODyD3AJYQugXG5uFj0xpJX+HNZCHtaLvzI8rG5q2Tdc7rd1KnMd622dZG3W/XqWCVRphC2AP+Tu5qaebRuqvk8tSVJ9n1rq2e4W1en3lGqHPCq5e/z/BT1UO+RR1e73tCxu/HoBLuF/A4CrZnFzl0fznrJ417847V3/4jRBC5TA/wgAAExG2AIAYDLCFgAAkxG2AACYjLAFAMBkhC0AACYjbAEAMBlhCwCAyQhbAABMRtgCAGAywhYAAJMRtgCcVtvTWuInAOcQtgCcNqTn7WrR2FdDet5eYr7Fo3aJnwBKshiGYVR2ERUtMzNXdnuNOyygyio+8a0KUxPl2W6ArI3bV3Y5VZq/v09ll4BKQNgCwHVE2LomTiMDAGAywhYAAJMRtgAAmIywBQDAZIQtAAAmI2wBADAZYQsAgMkIWwAATEbYAgBgMsIWAACTEbYAAJiMsAUAwGSELQAAJiNsAQAwGWELAIDJCFsAAExG2AIAYDLCFgAAkxG2AACYjLAFAMBkhC0AACYjbAEAMBlhCwCAyQhbAABMRtgCAGAywhYAAJMRtgAAmIywBQDAZIQtAAAmI2wBADAZYQsAgMkIWwAATEbYAgBgMsIWAACTEbYAAJiMsAUAwGSELQAAJiNsAQAwGWELAIDJCFsAAExG2AIAYDJTwzYhIUFhYWEKDQ3VsmXLLrvc5s2b1adPH8f0rl27FBQUpMjISEVGRmr8+PFmlgkAgKmsZm04LS1Nc+bM0apVq+Tp6akRI0YoKChITZs2LbFcRkaGZsyYUWLevn379Mgjj+jxxx83qzwAAK4b03q2ycnJCg4Olq+vr7y8vNS/f38lJiaWWi4uLk7PPPNMiXn79u3T9u3bNWTIED3xxBP6+eefzSoTAADTmdazTU9Pl7+/v2M6ICBAe/fuLbHM+++/r5YtW6pdu3Yl5vv4+Cg8PFz9+vXTv/71L8XExOjDDz90et8NGnhfW/EAAFQg08LWMIxS8ywWi+Pvhw4dUlJSkpYsWaJffvmlxHJTp051/H3kyJGaNWuWcnJy5OPj49S+MzNzZbeX3j8AVDZ/f+d+j6FmMe00cmBgoDIyMhzT6enpCggIcEwnJibqzJkzGjZsmMaOHav09HSNGjVKdrtdCxculM1mK7E9q9W07wUAAJjKtLDt1q2bUlJSlJWVpfz8fCUlJalXr16O9ujoaH322Wdau3atFi9erICAAC1fvlxubm76/PPP9dlnn0mS1qxZo3bt2qlOnTpmlQoAgKlM7dnGxMQoKipKQ4YM0aBBg9S2bVuNGTNG+/btK3fdGTNm6P3331d4eLg++ugjvfrqq2aVCQCA6SxGWRdXqzmu2QKoqrhm65oYQQoAAJMRtgAAmIywBQDAZIQtAAAmu+Kwzc3N1enTp82oBQCAGsmpsP388881bdo05ebmavDgwYqMjNR7771ndm0AANQIToXtokWLdP/99yspKUnt27fXF198oY8//tjs2gAAqBGcClvDMNS8eXMlJyerV69e8vb2LnPsYwAAUJpTYevm5qZ169Zp27Zt6t69u7Zs2WJ2XQAA1BhOhW1sbKzi4+MVExMjf39/LVy4UHFxcWbXBgBAjeDUcI2bNm1S7969S7wirypjuEYAVRXDNbomp3q2S5cuVd++ffXmm2/qzJkzZtcEAECN4vSLCE6cOKH4+Hh98sknatu2rUaOHKmuXbuaXd9VoWcLoKqiZ+uaruitPzabTRs2bNDMmTOVl5cnPz8/vfzyy+rcubOZNV4xwhZAVUXYuiarMwsdP35c8fHxWrt2rZo3b64JEyaod+/e+vbbb/XCCy9o06ZNZtcJAEC15VTY3nfffbr33nu1dOlS3XbbbY75HTt2VJcuXcyqDQCAGsGp08gFBQWqXbv29ainQnAaGUBVxWlk1+RUz/bAgQNavHix8vLyZBiG7Ha7Tp06pc2bN5tcHgAA1Z9Tj/7ExcWpQ4cOys3NVUREhLy9vXXPPfeYXRsAADWCUz1bi8WisWPH6tdff1WTJk00ePBgjRw50uzaAACoEZzq2datW1eS1LhxYx0+fFi1atWSzWYztTAAAGoKp3q2bdq00XPPPadnn31Wjz/+uI4dOyZ3d3ezawMAoEZw6m5kwzCUmpqq9u3ba/PmzUpOTtaIESPUpEmT61HjFeNuZABVFXcju6Zyw/b06dPlrtywYcMKL6giELYAqirC1jWVexo5PDxcFotFhmGooKBAXl5eslqtOnfunBo0aKBt27ZdrzoBAKi2yg3bb775RpI0efJkBQUFKTw8XJK0ceNGbdiwwfzqAACoAZy6G/m7775zBK0k9e3bV99//71pRQEAUJM4FbZ2u107d+50TH/55ZfV5kXyAABUNqce/YmLi9Nzzz0nDw8PGYYhwzC0YMECs2sDAKBGcPp9tkVFRTp06JAsFovuvPNOWa0Xc/r//u//9Mgjj5ha5JXibmQAVRV3I7smp04jS5KHh4datWqlli1bOoJWkhISEkwpDACAmsLpsL0cJzvGAAC4rGsOW26UAgCgfNcctgAAoHyELQAAJuOaLQAAJnMqbJcvX15q3uLFiyVJjz76aMVWBABADVPuoBb/+te/VFBQoCVLlujChQuO+UVFRfrggw80duxYRUREmF4kAADVWblha7VadejQIRUUFOjQoUOO+e7u7po0aZLpxQEAUBM4NYLUhg0b1K9fv+tRT4VgBCkAVRUjSLkmp8L2zJkzWrFihc6ePVtiflxcnFl1XRPCFkBVRdi6JqdeRBATEyMfHx+1bNmSQSwAALhCToVtZmamli5danYtAADUSE49+tOwYUPl5eWZXQsAwEUkJiZq9OjRlV3GdeNUzzYgIEBDhgxRly5dVLt2bcf8qnrNFgCAqsSpsG3UqJEaNWpkdi0AgCpm8eLFWrlyperWravOnTtr48aNSkxM1BtvvKFdu3bJZrOpZcuWiouLk7e3t/r06aN7771XKSkp+vnnnzVw4ECNGzdOkjRv3jwlJCTI19dXt956q2MfhYWF5W6vbdu2OnjwoP76178qNDS0sj6Ka+JU2D7zzDMqKCjQ8ePH1axZMxUWFpbo4QIAap6tW7dq1apVWrlypXx8fDRx4kRJFwPY3d1dq1atksVi0ezZs/XGG2/olVdekSTl5eVp+fLlSktLU2hoqEaOHKmDBw8qKSlJa9asUe3atfX000879vNH22vWrJnmzp17nY++YjkVtqmpqXr66adltVr14YcfKjIyUgsXLlTHjh3Nrg8AUEm2bNmiAQMGqF69epKkBx98UDt27NDmzZuVk5Oj5ORkSRdHFWzQoIFjvb59+0qSAgMD1aBBA2VnZyslJUWhoaHy9vaWJA0bNkwffPCBJP3h9jp37mz+wZrMqbCdMWOGlixZohdeeEE33XSTZs6cqenTp+ujjz4yuz4AQCWxWq0lXjbj7u4uSbLb7ZowYYJCQkIkSefPny8xpG+tWrUcf7dYLDIMw/Hzv7flzPa8vLwq+MiuP6fuRi4oKFDTpk0d0yEhIbLZbKYVBQCofCEhIUpKSlJOTo4kaeXKlZKkHj16aNmyZSosLJTdbtekSZM0e/bscrfVs2dPJSYm6ty5c7Lb7Vq7dq2j7Wq2V904FbZWq1XZ2dmOAS2OHj1qalEAgMrXtWtX3X///XrggQc0dOhQ5eTkqE6dOnrqqafUqFEj3XvvvQoLC5NhGIqNjS13WyEhIRo2bJiGDRum++67Tz4+v42kdTXbq26cGq5x06ZNmjNnjjIyMtS1a1dt375dU6dOVf/+/a9HjVeM4RoBVFXVabjGffv26ZtvvlFUVJQk6d1331Vqamq1v1mpMjgVtpJ0/Phxbd++XXa7XcHBwSVOK1c1hC2Aqqo6hW1ubq4mTJigo0ePymKx6Oabb9a0adMUGBhY2aVVO+WG7Q8//KA77rhD+/fvL7O9VatWphV2LQhbAFVVdQpbVJxyw/bxxx/XokWL1KdPn9IrWizauHGjqcVdLcIWQFVF2Lomp04j//LLL7rppptKzDt8+LCaNWtmWmHXgrAFUFURtq6p3LuRz549q7Nnz2rs2LHKzs7W2bNnlZ2drYyMjBKjfwAAgMsrd1CL559/Xtu3b5ckBQUF/baS1ap+/fqZWxkAADWEU6eRx48fr9dee+161FMhOI0MoKriNLJrcvrRn7Nnzyo/P1+GYchms+nEiRPq3r272fVdFcIWQFV1NWFrs9m1afdJff7VCZ05my9/3zoK7dJYff6nsdzdLNdc06lTpzRgwADdcccdJea/9dZbuvnmm695+/+9r6ioKG3atKlCt1vVOTU28vz587Vo0SJJF8ezLCoqUtOmTZWQkFDuegkJCVq4cKGKior08MMP68EHHyxzuc2bN2vq1KmOD//cuXN64YUXdPLkSdWvX19z586Vv7//lRwXANQINptdMz7YrZR9PzvmZZzN14FjWdp1IE0vje4sd3enBgMsV0BAQIkhFFGxnArbNWvW6IsvvtDf/vY3jRs3Tjt37tTmzZvLXSctLU1z5szRqlWr5OnpqREjRigoKKjUYBgZGRmaMWNGiXlz585V586dtXjxYq1Zs0bTp09nxBIALmnT7pMlgvb3Uvb9rC/2nFS/LreW2X6tMjIyNHnyZP3yyy+yWCx6/vnn1a1bN/3973/X6dOndfDgQWVmZuq5557Tjh07lJqaqhYtWmjOnDmy2Wx65ZVXdPjwYWVkZOj222/XP/7xD6e2XxM5Fbb169dXQECAmjRpou+//16RkZF67733yl0nOTlZwcHB8vX1lST1799fiYmJeuaZZ0osFxcXp2eeeUazZs1yzNu8ebOWLVsmSRo0aJCmTp2qoqIieXh4XMmxAUC19/lXJ8ptT9p5okLCNj09XZGRkY7piIgI7d+/X8OGDVPfvn2Vnp6uUaNGac2aNZKkQ4cOKT4+Xl9//bUeeughJSQk6LbbblNYWJgOHjyonJwceXh4aMWKFbLb7XrooYe0ZcuWEoMhTZ8+vcztX3oNX03iVNharVadOHFCTZo00e7du9WjRw+dO3eu3HXS09NLnPoNCAjQ3r17Syzz/vvvq2XLlmrXrt1l17VarfL29lZWVpbTQ4Q1aFDz/qEAuKYzZ/Ovqd1ZZZ1GDgoK0tGjRzV//nxJUnFxsU6ePClJ6t69u6xWqxo2bCh/f3/HWcvAwEBlZ2crKChIvr6+WrZsmY4ePapjx44pLy+vxPaTk5PL3P5dd91VIcdUlTgVto8//rgmTZqkhQsXau7cuVqzZo3uvvvuctcp676rS28Nki5+K0pKStKSJUv0yy+//GENbm7OX5PgBikAVdWV3iDl71tHGeUEqr9vnWst6bLsdrvee+89xxnKtLQ0+fn5acOGDSXONFqtpaNk48aNmj9/vqKiojR06FD9+uuvpXLhctuviZxKsN69e+u9996Tl5eX1q5dq3feeUfTpk0rd53AwEBlZGQ4ptPT0xUQEOCYTkxM1JkzZzRs2DCNHTvWcQpBuvgN69K6xcXFys3NdfxjAIArCe3SuNz2e4LKb78WwcHBWr58uSTpyJEjGjx4sPLznetJp6SkaODAgRo2bJj8/Py0a9euUu9Bv5btVzdO9WzPnz+vBQsWaNu2bXJ3d1efPn3UpEkTeXp6XnadSxfRs7KyVKdOHSUlJZUI6OjoaEVHR0v67VbwSx96SEiI1qxZoyeeeELr1q1T586duV4LwCX1+Z/G2nUgrcybpLq2uVm9O5sXtnFxcZo8ebIiIiIkSTNnznT6eup9992nF154QYmJifL09FT79u116tSpCtt+dePUc7YxMTFyc3PT8OHDZRiG4uPjVa9ePU2dOrXc9RISErRo0SIVFRVp+PDhGjNmjMaMGaPo6Gi1adPGsdx/P3d19uxZxcbG6uTJk/Lx8dEbb7yhW265xemD4jQygKrqap+z/WLPSSXt/O0523uCGqt354p5zhbmcyps+/fvr88++8wxbbfbFR4ervXr15ta3NUibAFUVYwg5Zqcumbr5+enrKwsx3ReXp5uvPFG04oCAKAmceqarZ+fn4YNG6YBAwbI3d1dGzdulJ+fn1599VVJF8+7AwCAsjkVts2aNSvx7trw8HDTCgIAoKZx+kUE1QnXbAFUVVyzdU1O9WzXr1+v119/XdnZ2SUeSv76669NKwwAgJrCqRuk5s6dq9jYWK1du1YJCQmOPwCA6u/UqVNq3ry5Jk+eXGL+gQMH1Lx5c61ateqy6/bp06fU87Mozamebb169XTPPfeYXQsAoAyG3aacvZuVk7pRxecyZa3XQD7t+sqn7d2yuLlXyD58fX21detW2Ww2ubtf3Oa6detUv379Ctm+q3MqbNu1a6ctW7YoJCTE7HoAAL9j2G1KWzVLeQd3OubZzmXowqmDyjuyR4FDn6+QwK1bt65atGihXbt2KTg4WJK0fft2xyvvli5dqrVr1yo/P18Wi0Vz584t8bJ5m82mmTNn6quvvpLNZtPQoUP18MMPX3NdNYVTp5G3bNmixx9/XG3atFHHjh3VoUMHdezY0ezaAMDl5ezdXCJofy/v4E7l7ttSYfsaOHCgYwCjvXv3qnnz5vLw8FBubq42bNigDz74QJ988on69evnGF73kvj4eEnS6tWrtXLlSm3cuFG7d++usNqqO6d6tkuWLDG5DABAWXJSN5bbfu7bjfJp16dC9tW7d2/NnTtXdrtd69ev18CBA7Vu3Tp5e3tr1qxZ+vTTT3Xs2DFt3bq11GvwUlJSdODAAe3YsUPSxcGPDh48qM6dO1dIbdVduWGbkpKirl27av/+/WW2N2rUyJSiAAAXFZ/L/IP2jHLbr4S3t7datGihPXv2aMeOHXr++ee1bt06/fzzz3rggQf0v//7v+rVq5f8/Px04MCBEuvabDa9+OKLjvt7srKy5OXlVWG1VXflhu2nn36qrl276oMPPijVZrFYuGkKAExmrddAtnIC1VqvYt//OnDgQM2aNUutW7d2vKfWy8tLt956qx5++GEVFhbqrbfeKnXjVHBwsOLj49W7d28VFhZq1KhRmjJlioKCgiq0vuqq3LC9NBxjWWF7yd/+9jfFxsZWbFUAAEmST7u+unDq4GXb67XvW6H76927tyZOnKhnn33WMc/Dw0N2u11hYWHy9PRU27Ztdfjw4RLrjRgxQsePH9e9996r4uJiDR06lKD9nWseQeree+/V6tWrK6qeCsEIUgCqqisdQaqsu5Ev8WoeVGF3I8NcTt0gVZ4aONojAFQZFjd3BQ59Xrn7tujctxtVfC5D1np+qte+r7zbhBC01cQ1h63FwouLAcBMFjd3+bTrU2F3HeP6c+o5WwAAcPUIWwAATHbNYcs1WwAAynfNYTt+/PiKqAMAgBqr3Ed/WrRoUe4NUP89gkhVwaM/AKoqXh7vmsrt2aakpGj79u26//77FRMTox07dmjXrl0aP368HnrooetVIwC4NJvdpk1HkzVpw+t6MmGCJm14XZuOJstut1fI9k+dOqU+fUrf6dy8efPLrrNz506NHj26QvZ/tQoLCzV69OjL1n8tYmNjtWrVKqWlpWnMmDFlLlPe5/Pfyn3058Ybb5Qkfffdd5oyZYpjflRUlIYOHer0TgAAV8dmt2lOyjv66tS3jnmZeb/qYOZRff3zPsV0fUzuLvqs7a5du0x/0UFgYKDefvvta96OU8/Z5ufn6+jRo2rSpIkk6eDBgyoqKrrmnQMAyrfl2M4SQft7X536Vl8e26neTbqZtv9Vq1Zp69atys7O1smTJ9W9e3e98sorJZZ57733tGHDBi1evFhjx45VmzZttGfPHmVlZSkuLk4hISHKyMjQxIkTdfr0aVmtVsXExKh169YaPHiwtm3bJknq2bOnxo8fr7CwMC1evFgWi0UFBQVKS0vT8ePH9dNPP+m+++7Tk08+KUn68ssvNWDAAKfqfeaZZzRo0CDH8kOHDtW0adN0/vx5zZkzRwUFBcrOztaLL76ogQMHOrZ36tQpRUVFadOmTTp16pRefPFF5eXlqV27dlf0OTp1g9Rzzz3neOPDqFGjFBUVpXHjxl3RjgAAV+6Lo9vLbd/0Y7LpNXzzzTeaP3++Pv74Y33xxRc6ePC3sZo/+ugjJSUladGiRapTp44kqaioSCtWrND48eM1b948SdK0adMUHByshIQEzZ8/XxMmTJDdblfDhg116NAh/fDDD7LZbPrqq68kXQzS3r17S7rYwfvnP/+pf//731q8eLHOnTsnSUpNTVXbtm2dqjcyMlLr1q2TJB07dkwXLlxQq1attHTpUr366qtavXq1pk+frjfffPOyn8O0adM0dOhQrV279orf6e5Uz/aee+5Rp06dtGfPHlksFnXq1KnUGx8AABUvI//X8tvzsq55H25upftdhmE4bpDt0KGDvL29JUl/+tOflJ2dLUk6dOiQJk+erNmzZ5d4nV7Pnj0lSc2aNdPZs2clSTt27HC83OZPf/qT2rVrp9TUVIWEhCglJUVWq1VRUVH69NNPlZOTozNnzqhp06aSpKCgIHl6eqpBgwby9fVVTk6Ozp07p4YNG8rdvfQp9LLqDQkJ0bRp05Sbm6tPPvlEERERkqTXX39dX3zxhRITE5Wamqrz589f9nP66quvNGvWLEnS4MGDFRcX5+Qn7GTPtrCwUF9//bXy8vJ0/vx5ffHFF5ozZ47TOwEAXB2/OjeW3+517R2fevXqKScnp8S8zMxM3XDDDZKkWrVqOeZbLBbH+Ap169bV/PnzNXPmTOXl5TmWubT8759m+e8HXwzDkM1mc4Ttzp07FRoaKjc3NyUkJDgC+3L7//LLL9WrV68yj6es5T09PXX33Xdr06ZNSkxMdITtqFGjtHfvXrVu3VpPPPHEH35Wl47DYrFc0XDFToVtTEyM/v73v2vGjBlKSEjQa6+9Vur1SgCAite7Sfdy2/vcfu3Xa729vXXrrbfqs88+c8xbsWKFunbtWu56jRo1Ut++fdWlSxfNnz+/3GWDg4O1cuVKSdLJkyf19ddfq3379mrVqpV+/PFHHTt2THfccYeCgoK0cOFC3X333eVub9u2berRo4dzB/j/RUZG6t1339UNN9ygRo0a6ezZszp27JieffZZhYSEaPv27bLZbJddv1u3bvr4448lSUlJSSosLHR6306F7YEDB7Rq1Sr17dtXEyZM0IcffljqWxAAoOLdfVuwutzSvsy2Lre0V8htwRWyn9dff13Lly/X4MGDNXDgQB0+fFiTJ092at1x48YpISFB+/fvv+wyEydO1I4dOxQREaGnnnpKr776qgICAhyXJu+44w5JF0M5NzdXXbp0uey2CgsLlZOTIz8/vys6xk6dOiknJ0eDBw+WJPn6+uq+++5TeHi4hgwZoszMTBUUFJTopf/e5MmT9dlnnykiIkJbtmxR3bp1nd63U++zHTFihD788EO98847atSokQYOHKjhw4c7vqVUNQxqAaCquppBLWx2m748tlObfkxWRl6W/Lzqq8/t3RRyW3CZ11tR9Th1g5SXl5cSEhLUokULxcfHq0mTJo6L3gAAc7m7uat3k26mPuIDczn1lWjy5Mn6/vvv1aNHD7m7u2v06NF69NFHza4NAIAawamwzczM1P79+zV48GAdOXJEgYGBWr58udm1AQBQIzh1GnnSpEm6//77ddddd13Rrc4AAMDJsPX09NTDDz9scikAANRMTp1GbtKkifbt22d2LQAA1EjlPvpzaYSN8+fPKz09XX/6059ktf7WGU5ISDC/wqvAoz8AqireZ+uayj2NPGnSpOtVBwAANZZTg1pUN/RsAVRV9GxdE0OPAABgMsIWAACTEbYAAJiMsAUAwGSELQAAJiNsAQAwGWELAIDJCFsAAExG2AIAYDLCFgAAkxG2AACYjLAFAMBkhC0AACYjbAEAMBlhCwCAyQhbAABMRtgCAGAywhYAAJMRtgAAmIywBQDAZIQtAAAmMzVsExISFBYWptDQUC1btqxU++eff66IiAiFh4crNjZWhYWFkqQ1a9aoR48eioyMVGRkpObMmWNmmQAAmMpq1obT0tI0Z84crVq1Sp6enhoxYoSCgoLUtGlTSVJeXp6mTp2q1atXy8/PTzExMVq9erUeeOAB7du3T7GxsRo0aJBZ5QEAcN2Y1rNNTk5WcHCwfH195eXlpf79+ysxMdHR7uXlpU2bNsnPz095eXnKzMxUvXr1JEn79u3TmjVrNHjwYL3wwgvKzs42q0wAAExnWs82PT1d/v7+jumAgADt3bu3xDIeHh7asmWLxo0bp4CAAPXo0UOS5O/vr7Fjx6pt27aaPXu2pk6dqlmzZjm97wYNvCvmIAAAqACmha1hGKXmWSyWUvNCQkK0c+dOzZ49W6+88opmzZqlBQsWONofe+wx9evX74r2nZmZK7u99P4BoLL5+/tUdgmoBKadRg4MDFRGRoZjOj09XQEBAY7ps2fPatu2bY7piIgIHTx4UDk5OVqyZIljvmEYslpN+04AAIDpTAvbbt26KSUlRVlZWcrPz1dSUpJ69erlaDcMQy+++KJOnz4tSVq/fr06duwoLy8vvfPOO0pNTZUkLV26VKGhoWaVCQCA6SxGWed7K0hCQoIWLVqkoqIiDR8+XGPGjNGYMWMUHR2tNm3aaMOGDZo3b54sFouaNm2qKVOmyMfHR7t379b06dNVUFCg2267TTNnzpSPj/OnXjiNDKCq4jSyazI1bCsLYQugqiJsXRMjSAEAYDLCFgAAkxG2AACYjLAFAMBkhC0AACYjbAEAMBlhCwCAyQhbAABMRtgCAGAywhYAAJMRtgAAmIywBQDAZIQtAAAmI2wBADAZYQsAgMkIWwAATEbYAgBgMsIWAACTEbYAAJiMsAUAwGSELQAAJiNsAQAwGWELAIDJCFsAAExG2AIAYDLCFgAAkxG2AACYjLCt4VKPZGjGsq+VeiSjsksBAJdlrewCYK41W3/U8bQcFRTa1K6pX2WXAwAuiZ5tDVdQWFziJwDg+iNsayib3a6tqaeVde6CJCnr3AVtTT0tu92o5MoAwPVYDMOocb99MzNzXTpUbHa73lqzX3sOnSnV1ulOfz0xpJXc3fieBVQGf3+fyi4BlYDfuDVQ8r5fygxaSdpz6IySv/vlOlcEAK6NsK2Btu79+ZraAQAVi7CtgbJyCspvP1d+OwCgYhG2NVB9n9rlt9crvx0AULEI2xqoZ9ubr6kdAFCxCNsaqHubm9XpTv8y2zrd6a/urQlbALieePSnhrLZ7Ur+7hctTTqkomK7PKxu+t977lT31jfLzc1S2eUBLotHf1wTPdsayt3NTT3bNlR9n1qSpPo+tdSzbUOCFgAqAWFbw9X2tJb4CQC4/gjbGm5Iz9vVorGvhvS8vbJLAQCXxTVbALiOuGbrmujZAgBgMsIWAACTEbYuqPjEt8pLeE3FJ76t7FIAwCVwi6oLurB7tewZx3WhqEDWxu0ruxwAqPHo2bogo6igxE8AgLkIWwAATEbYuhDDblPR91/KyM26OJ2bdXHabq/kygCgZuM5Wxdh2G0q2PCmio/tKdVmva2Tavd7ShY390qoDHAtPGfrmujZuojiQ9vLDFpJKj62R8WHk69zRQDgOghbF1F48Mty24u+L78dAHD1CFsXcek67eXYczOvUyUA4HoIWxdh8a5fbrubd4PrVAkAuB7C1kV4Nu9VbrtHi/LbAQBXj7B1EdY7e8h6W6ey227rJGuz7te5IgBwHTz640IMu03Fh5NVsO19yVYkuXuodo8oWZt1l8WN713A9cCjP66J37AuxOLmLo/mPR3Xby3e9S9OE7QAYCp+y7ogi0ftEj8BAOYyNWwTEhIUFham0NBQLVu2rFT7559/roiICIWHhys2NlaFhYWSpNOnT+vBBx/UgAED9OSTT+r8+fNmlulyanW+V+43t1CtzvdWdikA4BJMC9u0tDTNmTNHy5cv19q1a7VixQodOXLE0Z6Xl6epU6fq3Xff1aeffqoLFy5o9erVkqQpU6Zo1KhRSkxMVOvWrfXmm2+aVaZLsjZuL6+IWF6vBwDXiWlhm5ycrODgYPn6+srLy0v9+/dXYmKio93Ly0ubNm2Sn5+f8vLylJmZqXr16qmoqEi7du1S//79JUlDhw4tsR4AANWNaS+PT09Pl7+/v2M6ICBAe/fuLbGMh4eHtmzZonHjxikgIEA9evTQr7/+Km9vb1mtF0vz9/dXWlraFe27QQPvaz8AAAAqiGlhW9YTRRaLpdS8kJAQ7dy5U7Nnz9Yrr7yicePGObVeeXj0B0BVxaM/rsm008iBgYHKyMhwTKenpysgIMAxffbsWW3bts0xHRERoYMHD6p+/frKzc2VzWaTJJ05c6bEegAAVDemhW23bt2UkpKirKws5efnKykpSb16/TYkoGEYevHFF3X69GlJ0vr169WxY0d5eHioc+fOWrdunSRpzZo1JdYDAKC6MXUEqYSEBC1atEhFRUUaPny4xowZozFjxig6Olpt2rTRhg0bNG/ePFksFjVt2lRTpkyRj4+PfvrpJ8XGxiozM1M333yzZs+erRtuuMHp/XIaGUBVxWlk18RwjQBwHRG2rokRpAAAMBlhCwCAyQhbAABMZtpztpXJze3KnssFAMBMNfIGKQAAqhJOIwMAYDLCFgAAkxG2AACYjLAFAMBkhC0AACYjbAEAMBlhCwCAyQhbAABMRtgCAGAywtYFJSQkKCwsTKGhoVq2bFlll4MaIDc3V4MGDdKpU6cquxSgSiJsXUxaWprmzJmj5cuXa+3atVqxYoWOHDlS2WWhGktNTdXIkSN17Nixyi4FqLIIWxeTnJys4OBg+fr6ysvLS/3791diYmJll4VqLD4+Xi+//LICAgIquxSgyqqRb/3B5aWnp8vf398xHRAQoL1791ZiRajupk+fXtklAFUePVsXU9ZLniwWXkkIAGYibF1MYGCgMjIyHNPp6emc/gMAkxG2LqZbt25KSUlRVlaW8vPzlZSUpF69elV2WQBQo3HN1sUEBgYqJiZGUVFRKioq0vDhw9W2bdvKLgsAajSLUdZFPAAAUGE4jQwAgMkIWwAATEbYAgBgMsIWAACTEbYAAJiMsEW11qFDhyt608y+ffsUHR0tSdq7d68mT578h+vk5OQoOjpagwYNUlhYmBYvXuxoO3bsmEaNGqWwsDANHz5cP/zwwxUfw6BBg7Rz584rXg9A9UHYwqW0adNG8+fPlyQdOXJEaWlpf7jOvHnzFBgYqE8++UQrV67Uhx9+qG+++UaS9MILL2jkyJFat26d/vKXvyg6OrrMITEBuDYGtcB1s3PnTk2fPl1eXl7Ky8tTdHS0Fi1apKKiItWuXVsvvfSSOnTooIyMDE2ePFmZmZk6c+aMGjVqpLlz56pBgwbavXu3pk2bJovFojZt2shutzu2PXv2bAUEBOjw4cOqU6eO/vKXv+iDDz7Qjz/+qHvuuUcTJkzQzp07NW3aNL399tuaP3++cnJyNH78eL322muXrXvixImy2WySpDNnzqiwsFA+Pj5KS0vT0aNHFR4eLkkKCQnRlClT9J///EetWrW67PaOHDmiCRMmKD8/X02aNFFeXp6j7a233tKGDRt04cIF5efn66WXXlK/fv00YMAATZo0ST169JAkxcXFqVmzZurRo4cmTpyowsJCGYah4cOH68EHH7zmfysAFcwArpMdO3YYLVq0ME6dOmX8+OOPxqBBg4ysrCzDMAzj0KFDRvfu3Y3z588bS5YsMRYtWmQYhmHY7XbjscceM/75z38aFy5cMLp162YkJycbhmEYCQkJxp133mmcPHnS2LFjh3HXXXcZ+/fvNwzDMB599FHjgQceMC5cuGBkZmYarVq1Mn755Rdjx44dRnh4uGEYhvHRRx8ZY8eOdbr+559/3mjdurURExNjFBcXG998843Rv3//EsuMGDHC2LBhQ7nbiYyMNOLj4w3DMIzdu3cbzZs3N3bs2GGcOnXKGD16tJGfn28YhmF88sknxqBBgwzDMIx3333XiI6ONgzDMHJycozg4GAjOzvbGD9+vOOzSk9PN5577jnDZrM5fUwArg96triubr75ZjVq1EjLli1Tenq6Hn74YUebxWLRiRMn9NBDD2n37t169913dezYMR0+fFjt2rXToUOHZLVa1bVrV0kXr3X+/prrLbfcopYtW0qSGjduLB8fH3l6eqp+/fqqW7eusrOzr6n2N954Q1OmTFF0dLQWLFjg6GX+N3d398tu49dff9XBgwc1ZMgQSVKnTp3UrFkzSVKjRo00Y8YMJSQk6Pjx40pNTdX58+clSUOHDtWCBQuUlZWlxMRE3X333apXr55CQ0P10ksvae/everatavi4uLk5sbVIaCq4X8lrisvLy9Jkt1uV9euXbV27VrHn/j4eDVr1kyvv/665s2bpxtvvFEPPPCAunfvLsMwZLFYSl0PtVp/+77o6el52bZrsXXrVse13bp16yo8PFz/+c9/1LBhQ2VkZJSoKS0tTTfddNNlt3XpdYa/X+dSnfv379eIESOUm5ur7t2767HHHnMsU69ePQ0YMEAff/yxPvroI40cOVKS1Lt3b3322WcaOHCgDhw4oIiICJ04caJCjhtAxSFsUSmCg4O1fft2x927W7Zs0eDBg3XhwgVt27ZNDz30kIYMGaIGDRooOTlZNptNd955pwzD0JYtWyRJGzduvKbeqru7u4qLi/9wufXr12vBggUyDEOFhYVav369goODddNNN6lx48Zat26dpIuh7ObmpjvvvPOy2/L19VWrVq3073//W9LFgD106JAkadeuXWrdurX+/Oc/q0uXLtq4caPjWrEkPfjgg3r//fdlGIbj5RHPP/+81q1bp/DwcL388svy9vbWzz//fNWfCQBzcBoZlaJZs2aaOnWq/vrXv8owDFmtVi1cuFBeXl56+umnNXPmTL355ptyd3dXx44ddeLECXl4eGjBggV65ZVXNHv2bN11111q0KDBVdfQoUMHzZ07V08//bQWLFhw2eViY2P18ssvKyIiQhaLRX379lVUVJQkafbs2Zo0aZIWLlwoT09PzZs37w9P486ePVvjx4/Xhx9+qMaNG6tJkyaSLp4WT0pKUlhYmDw8PNS1a1dlZ2crNzdX3t7eatGihW644QaNGDHCsa2nnnpKEydO1IoVK+Tu7q5+/fqpS5cuV/2ZADAHb/0BqokTJ05o9OjRSkxMVJ06dSq7HABXgJ4tXN7Ro0cVExNTZtvtt9+uuXPnXtH2duzYcdlHiYKCgjRhwoQrLVHz5s1TfHy8Jk6cSNAC1RA9WwAATMYNUgAAmIywBQDAZIQtAAAmI2wBADAZYQsAgMkIWwAATPb/AJHIr0gM2IOBAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 382 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 260 - }, - "id": "r9TKGsNY8Myu", - "outputId": "06146e22-ea39-4d70-9488-0dd9b03eddca" - }, - "outputs": [], - "source": [ - "# Show the balanced error / fnr difference values of all predictors on a raster plot \n", - "\n", - "plt.scatter(balanced_error_sweep, fnr_diff_sweep, label=\"ExponentiatedGradient - Iterations\")\n", - "for i in range(len(predictors)):\n", - " plt.annotate(str(i), xy=(balanced_error_sweep[i]+0.001, fnr_diff_sweep[i]+0.001))\n", - "\n", - "# Also include in the plot the combined ExponentiatedGradient model\n", - "# as well as the three previously fitted models\n", - "\n", - "plt.scatter(1-balanced_accuracy_score(Y_test, Y_pred_reductions),\n", - " false_negative_rate_difference(Y_test, Y_pred_reductions, sensitive_features=A_test),\n", - " label=\"ExponentiatedGradient - Combined model\")\n", - "plt.scatter(1-balanced_accuracy_score(Y_test, Y_pred),\n", - " false_negative_rate_difference(Y_test, Y_pred, sensitive_features=A_test),\n", - " label=\"Unmitigated\")\n", - "plt.scatter(1-balanced_accuracy_score(Y_test, Y_pred_postprocess),\n", - " false_negative_rate_difference(Y_test, Y_pred_postprocess, sensitive_features=A_test),\n", - " label=\"ThresholdOptimizer\")\n", - "plt.scatter(1-balanced_accuracy_score(Y_test, y_pred_postprocess_deterministic),\n", - " false_negative_rate_difference(Y_test, y_pred_postprocess_deterministic, sensitive_features=A_test),\n", - " label=\"ThresholdOptimizer (DET)\")\n", - "\n", - "plt.xlabel(\"Balanced Error Rate\")\n", - "plt.ylabel(\"False Negative Rate Difference\")\n", - "plt.legend(bbox_to_anchor=(1.9,1))\n", - "plt.show()" - ] + "id": "VLom8GFbUfqR", + "outputId": "91418775-b94d-4b3e-dba4-1305ef908a29" + } + }, + { + "cell_type": "code", + "execution_count": 25, + "source": [ + "# Check for predictive validity by age\n", + "sns.catplot(y=\"had_inpatient_days\", x=\"readmit_30_days\", hue=\"age\", data=df,\n", + " kind=\"point\", ci=95, dodge=True, join=False);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAewAAAFqCAYAAADYw2mlAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2dUlEQVR4nO3deVhUZf8G8HuYARFFUTaXtPQVFRFQMQE3QlQQWVxIUAM1UyuV5FfmEpbZW24pai5pZZZaiqYhpkjuKYiSCW6I5oKkgoDIDsPM+f3B2yQJ4wAOw4H7c11evGeec858B1+75znnOc8jEQRBABEREdVperougIiIiJ6NgU1ERCQCDGwiIiIRYGATERGJAAObiIhIBBjYREREIiDTdQE1kZmZB6WST6URkXiYmxvrugQSKfawiYiIRICBTUREJAIMbCIiIhFgYBMREYkAA5uIiEgEGNhEREQiwMAmIiISAQY2ERGRCDCwiYiIRICBTUREJAIMbCIiIhFgYBMREYkAA5uINJJwIwNLt59Hwo0MXZdC1CCJerUuIqo9P/92C3fSclFUooB9JzNdl0PU4LCHTUQaKSopLfeTiGoXA5uIiEgEGNhEREQiwMAmIiISAQY2ERGRCDCwiYiIRICBTUREJAIMbCIiIhFgYBMREYkAA5uIiEgEGNhEREQiwMAmIiISAQY2ERGRCDCwiYiIRICBTUREJAIMbCIiIhFgYBMREYkAA5uIiEgEGNhEREQiwMAmIiISAQY2ERGRCDCwiYiIRICBTUREJAIMbCIiIhFgYBORWgqlEr8l3ENWTjEAICunGL8l3INSKei4MqKGRSIIgmj/1WVm5vE/GkRapFAq8eXPl/F78sOn2hw6m+PNETaQ6vF7f1WYmxvrugQSKf5LI6JKxVx8UGFYA8DvyQ8Rc+lBLVdE1HAxsImoUr8l3q9ROxE9PwxsIqpUVm6R+vYc9e1E9PwwsImoUi2NDdW3N1PfTkTPDwObiCo1wK51jdqJ6PlhYBNRpfrZtoZDZ/MK2xw6m6NfdwY2UW3hY11EpJZCqUTMpQfYFp0MeakS+jI9vDa0M/p1bw09PYmuyxMdPtZF1cUeNhGpJdXTwwC7Nmhp3AgA0NK4EQbYtWFYE9UymTZPHhkZiQ0bNkAul2PixIkYP368qu3q1auYO3euajsrKwvNmzfH/v37tVkSERGRKGktsNPS0hAWFoY9e/bAwMAAAQEBcHR0RKdOnQAA1tbWiIiIAAAUFhbi1VdfxcKFC7VVDhERkahp7ZJ4TEwMnJycYGJiAiMjI7i7uyMqKqrCfTdu3IiXX34ZvXv31lY5REREoqa1HnZ6ejrMzf8ZXWphYYHExMSn9svJyUF4eDgiIyO1VQoREZHoaS2wKxp8LpE8PUglMjISgwcPhqmpaZXfw9S0abVqI6Kqk0r1VD850pmo9mktsC0tLREfH6/aTk9Ph4WFxVP7HT58GNOmTavWe/CxLqLao1AoVT8fPszVcTXixS87VF1au4fdt29fxMbGIisrC4WFhYiOjsbAgQPL7SMIAi5fvoyePXtqqwwiIqJ6QWuBbWlpiZCQEAQFBWHEiBHw8vKCnZ0dpkyZgosXLwIoe5RLX18fjRo10lYZRERE9QJnOiMijczbGIu0R4WwbNEYi6c567oc0eIlcaouznRGREQkAgxsIiIiEWBgExERiQADm4iISAQY2ERERCLAwCYiIhIBBjYREZEIMLCJiIhEgIFNREQkAgxsIiIiEWBgExERiQADm4iISAQY2ERERCLAwCYiIhIBBjYREZEIMLCJSCOGBrJyP4modjGwiUgjIwZ0QNf2JhgxoIOuSyFqkCSCIAi6LqK6MjPzoFSKtnwiaoDMzY11XQKJFHvYREREIsDAJiIiEgEGNhERkQgwsImIiESAgU1ERCQCDGwiIiIRYGATERGJAAObiIhIBBjYREREIsDAJiIiEgEGNhERkQgwsImIiESAgU1ERCQCDGwiIiIRYGATERGJAAObiIhIBBjYREREIsDAJiIiEgEGNhERkQgwsImIiESAgU1ERCQCDGwiIiIRYGATERGJAAObiIhIBBjYREREIsDAJiIiEgEGNhERkQgwsImIiERAq4EdGRkJT09PDBkyBNu3b3+q/ebNmwgMDISPjw8mT56Mx48fa7McIiIi0dJaYKelpSEsLAw//PADIiIisHPnTty4cUPVLggC3nrrLUyZMgX79u2DtbU1Nm3apK1yiIiIRE1rgR0TEwMnJyeYmJjAyMgI7u7uiIqKUrVfvnwZRkZGGDhwIADgzTffxPjx47VVDhERkajJqnpAXl4ecnJy0KZNG7X7paenw9zcXLVtYWGBxMRE1XZKSgrMzMwwZ84cXLlyBZ07d8aCBQuqVIupadOqFU9ERCRSGgX2r7/+ijNnziAkJAQ+Pj7Izc3FjBkzMGHChEqPEQThqdckEonqf5eWluLs2bPYtm0bbG1tsWrVKixZsgRLlizRuPjMzDwolU+/DxFRXWVubqzrEkikNLokvnHjRowZMwbR0dHo0aMHjh07hn379qk9xtLSEhkZGart9PR0WFhYqLbNzc3x4osvwtbWFgDg5eVVrgdORERE/9AosAVBQJcuXRATE4OBAweiadOmFfagn9S3b1/ExsYiKysLhYWFiI6OVt2vBoCePXsiKysLSUlJAICjR4/CxsamBh+FiIio/tLokrienh4OHDiAU6dOYc6cOThx4sQzj7G0tERISAiCgoIgl8vh5+cHOzs7TJkyBcHBwbC1tcW6desQGhqKwsJCtGrVCsuWLavxByIiIqqPJMKzusoAfv/9d3zxxRcYNmwY/P39ERAQgPfffx+9evWqjRorxXvYRCQ2vIdN1aVRYB89ehSurq7lBo3VBQxsIhIbBjZVl0b3sLdt2wY3NzesX78eDx8+1HZNRERE9C8a9bCBsuemw8PDsX//ftjZ2WHs2LFwdnbWdn1qsYdNRGLDHjZVl8aBDQAKhQKHDx/GsmXLUFBQADMzM3z00Ufo3bu3NmusFAObiMSGgU3VpdEo8Tt37iA8PBwRERHo0qUL5s+fD1dXV1y4cAHvvfcejh49qu06iYiIGjSNAvvVV1/FyJEjsW3bNrz00kuq13v16oU+ffpoqzYiIiL6H40uiRcVFcHQ0LA26qkSXhInIrHhJXGqLo162FevXsWmTZtQUFAAQRCgVCqRmpqK48ePa7k8IiIiAjR8rCs0NBQ9e/ZEXl4evL290bRpUwwdOlTbtREREdH/aNTDlkgkmDp1Kh49eoSOHTvCx8cHY8eO1XZtRERE9D8a9bCbNGkCAGjfvj2uX7+ORo0aQaFQaLUwIiIi+odGPWxbW1vMmjUL77zzDqZNm4bbt29DKpVquzYiIiL6H41GiQuCgISEBPTo0QPHjx9HTEwMAgIC0LFjx9qosVIcJU5EYsNR4lRdagP73r17ag9u06bNcy+oKhjYRCQ2DGyqLrWXxIcPHw6JRAJBEFBUVAQjIyPIZDLk5OTA1NQUp06dqq06iagOKk25gJKEgzCwHwZZ+x66LoeoXlMb2H/88QcA4MMPP4SjoyOGDx8OADhy5AgOHz6s/eqIqE4rjt8LZcYdFMuLGNhEWqbRKPFLly6pwhoA3NzckJSUpLWiiEgcBHlRuZ9EpD0aBbZSqURcXJxq++TJk5BIJForioiIiMrT6LGu0NBQzJo1C/r6+hAEAYIgYN26ddqujYiIiP5Ho8Du3bs3jh07huTkZEgkEnTu3BkyWdmhmzdvxuuvv67VIomIiBo6jS6JA4C+vj5sbGzQrVs3VVgDQGRkpFYKIyIion9oHNiV0WDeFSKqZwSlAvKkkxDyssq287LKtpVKHVdGVH9pdElcHQ4+I2pYBKUCRYfXo/T27/+8qJCj6ORmyFISYDj4bUj0OHUx0fNW4x42ETUspcmny4f1k223f0fp9ZharoioYWBgE1GVlFw7qbZdnqS+nYiqh/ewiahK/r5vXRllXmYtVULUsGgU2D/88MNTr23atAkAMHny5OdbERHVaZKmLdW26zU1raVKiBoWtYPOfvzxRxQVFWHLli0oLi5WvS6Xy7F161ZMnToV3t7eWi+SiOoOgy4DUZR2o9J2/a4Da7EaooZDbWDLZDIkJyejqKgIycnJqtelUikWLFig9eKIqO6Rde4PWUpChQPPZC85QGbVTwdVEdV/atfD/tvhw4cxePDg2qinSrgeNpFuCEoFSq/HoOjU94BCDkj1Ydg/CDKrfpDocSyrOlwPm6pLo+ew7e3tsXbtWmRnZ5d7PTQ0VBs1EVEdJ9GTQr/LABRf2A/hcRokTVtCv8sAXZdFVK9pFNghISEwNjZGt27dOFEKERGRDmgU2JmZmdi2bZu2ayEiIqJKaHSzqU2bNigoKNB2LURERFQJjXrYFhYWGDFiBPr06QNDQ0PV67yHTUREVDs0Cuy2bduibdu22q6FiERGom8I4X8/iSqiVCrx2WefISEhAfn5+RAEAf/973/RoUMHzJs3DykpKTAxMYG5uTmsrKwwc+ZM/Pnnn/j000+RnZ0NhUKBwMBA+Pn56fqj6JxGgT1jxgwUFRXhzp07sLKyQklJSbmeNhE1TI16j0RJQhQM7D10XQrVUQkJCUhPT8fOnTuhp6eHTZs24auvvoKRkRE6deqEjRs3Ij09HaNGjYKVlRVKS0sRHByMZcuWwcbGBrm5ufD390enTp3Qo0cPXX8cndIosBMSEjB9+nTIZDLs2LEDvr6+2LBhA3r16qXt+oioDpO17wFZ+x66LoPqsJ49e6J58+bYsWMH7t69i7i4ODRp0gTnzp3D3r17AZTddvXwKPvSd/v2baSkpGD+/PmqcxQVFeHKlSsNPrA1GnS2dOlSbNmyBSYmJmjVqhWWLVuGTz/9VNu1ERGRyB0/fhzTpk0DALi5uWHs2LEAymbSfHLeLr3/TbijUCjQrFkzREREqP6Eh4dj9OjRtV98HaNRYBcVFaFTp06qbRcXFygUCq0VRURE9cPp06fh6uqKcePGwdbWFocPH4ZCoYCLiwt2794NAHj06BEOHz4MiUSCDh06oFGjRoiIiAAA3L9/H15eXrh06ZIuP0adoNElcZlMhsePH6smTbl586ZWiyIiovohICAA7733Hry9vSGVStG7d29ER0dj3bp1CA0Nhbe3N0xMTNCmTRsYGhrCwMAA69evx6effoqvv/4apaWleOedd+Dg4KDrj6JzGs0lfvToUYSFhSEjIwPOzs44ffo0Fi1aBHd399qosVKcS5yIxIZziZfZvn07unXrhp49e6KkpATjxo3DzJkz4eLiouvS6iyNAhsA7ty5g9OnT0OpVMLJyancJXJdYWATkdgwsMvExcVh6dKlUCqVkMvl8PDwwMyZM3VdVp2mNrD//PNP/Oc//8Hly5crbLexsdFaYZpgYBOR2DCwqbrUBva0adOwceNGDBo06OkDJRIcOXJEq8U9CwObiMSGgU3VpdEl8QcPHqBVq1blXrt+/TqsrKy0VpgmGNhEJDYMbKoutY91ZWdnIzs7G1OnTsXjx4+RnZ2Nx48fIyMjA9OnT3/mySMjI+Hp6YkhQ4Zg+/btT7WvXbsWrq6u8PX1ha+vb4X7EBER0TMe63r33Xdx+vRpAICjo+M/B8lkGDx4sNoTp6WlISwsDHv27IGBgQECAgLg6OhYbrDapUuXsHLlSvTs2bMmn4GIiKje0+iS+Lx587B48eIqnXjv3r04d+4cPvvsMwDAunXrIAgCZsyYodqnf//+sLOzw927d/Hyyy9jzpw5aNSokcbvwUviRCQ2vCRO1aXRxCmLFy9GdnY2CgsLIQgCFAoFUlJS0K9fv0qPSU9Ph7m5uWrbwsICiYmJqu38/HxYW1tjzpw5aNu2LebOnYv169cjJCRE4+JNTZtqvC8RkVgpFEocjb+LX8+m4GF2IcxNGmNIn/YY9HJ7SPUkNT7/6tWrcejQIUgkEvj5+WHSpEkAgJiYGCxevBjFxcUYNmxYlf77TM+fRoG9Zs0abNy4EQAglUohl8vRqVMnREZGVnpMRR33v2dKA4AmTZrgq6++Um2//vrrmD9/fpX+D8EeNhGJTVV72AqFEku3xiP24n3VaxnZhbh6OwvnrqZhTmBvSKUazTJdobNnz+LMmTPYt28fSktL4enpCRcXF7Rp0wbz58/H1q1b0bp1a0ybNg0nTpzgxCY6pNHf8s8//4xjx47B3d0d0dHRWLJkyTMnTrG0tERGRoZqOz09HRYWFqrte/fuqeaRBcoCXibT6PsDEVGDcTT+brmwflLsxfs49vvdGp2/T58++P777yGTyZCZmQmFQgEjIyMkJibixRdfRLt27SCTyeDt7Y2oqKhyx+7atQvvvvuuanvt2rXYtGkT8vPzMWfOHIwaNQq+vr7Yv38/ACAvLw/BwcHw9/eHq6srZs+eDUEQEBcXBz8/P4waNQpz5sxBbGwsRo0ahVGjRmHSpEnIysp6qu6ffvoJXl5e8Pb2xty5c5Gfnw8AcHJywuTJk+Hr6wu5XK7af/bs2di5c6dqOzAwEAkJCbh16xYCAwPh7e0Nf39/1ZXguXPnYs+ePar9u3TpAgD44osvEBoaisDAQAwaNAgbNmwAAMjlcsyfPx/u7u4ICgrChAkTEBcXBwDYtGkTRo4cCR8fHyxbtgyCICA1NRUeHh4YO3YsJk6cqNHflUaB3bJlS1hYWKBjx45ISkqCr68v7ty5o/aYvn37IjY2FllZWSgsLER0dDQGDhyoajc0NMTy5ctx9+5dCIKA7du3Y8iQIRoVTUTUUPx6NkVte3Sc+nZN6OvrY82aNRg+fDicnZ1haWlZ4W3NtLS0csd5enoiNjYW+fn5EAQBkZGRquWXbWxssGfPHmzfvh1ffvkl7t69i+PHj8Pa2ho7d+7EoUOHcOHCBdXEXLdv38Z3332HpUuXYv369Vi4cCH27NkDV1dXXLlypdz7Xrt2DV9++SW2bt2KyMhING7cGGvXrgVQtpDI1KlTERERAX19fdUxo0ePxr59+wAAf/31F7KysmBvb4/Zs2cjMDAQkZGRmDdvHt555x2UlJSo/X1du3YN33zzDXbt2oVNmzYhJycHO3bsQGFhIaKiorB48WJcvHgRAHDy5ElcunQJu3fvxs8//4y0tDRVHbdu3cLy5cuxZcsWjf6eNF78IyUlBR07dkR8fDz69++PnJwctcdYWloiJCQEQUFBkMvl8PPzg52dHaZMmYLg4GDY2tpi0aJFeOuttyCXy9GrVy/VfRMiIirzMLuwRu2aCg4OxpQpU/Dmm28iPDwcRkZGT+3z5G1NoOzWpouLC6Kjo9GuXTu0a9cOlpaWiImJQVFREX766ScAQEFBAa5fvw4vLy8kJiZiy5YtuHnzJrKzs1FQUAAA6NChA4yNy24XuLm5YcaMGRg8eDDc3NyeGi917tw5uLq6okWLFgAAf39/zJs3T9Vub2//VO2Ojo5YsGABUlNTERERAV9fX+Tn5yMlJQVDhw4FAPTo0QPNmzd/5gJXjo6OMDAwgKmpKUxMTJCbm4vTp09jzJgxkEgkaNu2LZydnQEAsbGxSExMxKhRowCUrX7Zpk0bODg4wNTUFC+88ILa93qSRoE9bdo0LFiwABs2bMCqVavw888/45VXXnnmcd7e3vD29i732pP3rd3d3XW+gAgRUV1mbtIYGWpC2dykcY3O/+eff6KkpATW1tZo3Lgxhg4dimvXrsHDw0Ptbc2/jR49Ghs2bMALL7ygCiWlUonly5erpq/OyMhA8+bNsXXrVhw6dAhjxoxB3759kZycrBrvZGhoqDrnxIkT4erqimPHjmH58uVITEzEW2+9pWpXKpXlahAEAaWlpartJ8/1N4lEghEjRuCXX35BVFQUvv76awiC8NR4q78HVkskElXbk5fWAZR7munv/aRS6VN1AWXre0+YMEHVIc3JyYFUKsWjR48qrFMdjS6Ju7q64rvvvoORkREiIiLw9ddf45NPPqnSGxERUdUN6dNebftQR/Xtz5KamorQ0FCUlJSgpKQER44cgYODA+zt7XHr1i3cuXMHCoUC+/fvL3db82+9e/fGgwcPEBcXp5qfw8nJCT/++COAsqD38fHB/fv3cfr0afj7+8PHxwcSiQRJSUkVhtyrr76K/Px8TJw4ERMnTnzqknifPn1w9OhRZGdnAwDCw8PLzRVSmVGjRmHHjh1o1aoVLC0t0bRpU7Rr1w7R0dEAgAsXLiAjIwNWVlYwMTHBjRs3AACHDx9+5rn79u2LAwcOQBAEpKWl4ezZs5BIJHByckJERATy8/NRWlqK6dOn49ChQ888X0U06mHn5+dj3bp1OHXqFKRSKQYNGoSOHTvCwMCgWm9KRESaGfRye5y7mlbhwDNn29Zw7V2zwHZxcUFCQgJGjBgBqVSKoUOHYvjw4QCAJUuWYObMmSguLoaLiws8PDwqPMeQIUOQnZ2tyoQZM2Zg4cKF8PLygkKhwOzZs9G+fXtMmDABCxcuxObNm9GkSRP07NkTqampaN++/Gf4v//7P8ydOxcymQyNGjXCxx9/XK69a9eumDZtGgIDAyGXy2FjY/PUPhVp3bo1WrdujZEjR6peW758ORYuXIgvvvgC+vr6+OKLL2BgYIBx48Zh1qxZ8Pb2hpOTU7n7+RUZM2YMkpKS4O3tDXNzc9X63n369EFSUhLGjBkDhUKBAQMGYOTIkfjrr7+eWe+/aTRxSkhICPT09ODn5wdBEBAeHo5mzZph0aJFVX7D54mPdRGR2FRn4hSFQoljv99FdNw/z2EPdWwP197P5zns6hIEAXK5HJMmTcL8+fN1voKjOoIgID09HYGBgdi/f/9z73AeP34cgiDA1dUVubm5GDFiBH766SeYmJg8t/fQqId95cqVcl14Jycn1TcwIqJ/u5RxFYdTTmBwexd0N7PWdTmiJ5XqYXCfFzG4z4u6LqWchw8fYvjw4Xj11VfrdFgDwKFDh7Bw4UIsXLhQK1eH//Of/+D999/HqlWrAJQN4nueYQ1oGNhmZmbIyspCy5YtAZSN+Pt7dB4R0b/tvxWNu7l/oUhRzMCuxywsLHDu3Dldl6ERDw+PSi/pPw/t2rVT3bfXFo0De/To0fDw8IBUKsWRI0dgZmaG//73vwCA0NBQrRZJROJSXFpc7icR1ZxGgW1lZVVu7WteDieiiiiUCsQ9OI9HxdkAgEfF2Yi5dw5OrR2gJ6n+9JlEpOGgs7qKg86I6g6FUoHNl7fjwsNLT7X1MO+O123GQ6on1UFldQtX66Lq0qiHffDgQSxfvhyPHz8u95D5+fPntVYYEYlL3IPzFYY1AFx4eAlnH5yHc5uXa7kqovpDo8BetWoV5s6di27duj01NR0REQDE3j+rtj3m/jkGdh1V0+U1jx49irVr16KgoAD9+/dXjWvi8pzPl0aB3axZM9Vcq0REFXlU9PgZ7dm1U0g9JCgVyE08jtyEIyjNyYSsmSmM7d1gbPcKJDW8zVDT5TXv3r2Ljz76CLt27YKpqSkmTJiAEydOwNHRkctzPmcaBba9vT1/0USkVgvD5qrBZhW3m9RaLfWJoFQgbc8KFFyLU72myMlAceo1FNz4HZaj3q1RaD+5vGZaWlqFy2sCUC2v+e8c+PXXX+Hp6YlWrVoBAMLCwtCoUSONjt+1axfOnDmDFStWAChbntPAwADjx4/HokWLcP36dSgUCkyZMgVeXl7Iy8vD/PnzkZaWhvT0dPTu3RvLli3D2bNnsXz5ciiVSlhZWWHEiBFYvnw5AKB58+ZYsWKF6rFkMdMosE+cOIFt27ZBX18f+vr6EAQBEomE97CJSMW5dR/cfFz5srt9W/NyeHXkJh4vF9ZPKrgWh7yLJ2BsP6hG7/H38pqbN2+Gh4cHLC0tER8f/8zlNQHgzp070NfXx+TJk/Hw4UO4urpi1qxZGi/PGRYWhvz8fBgZGSEyMhLff/+9annOpUuXIi8vDwEBAbC3t0dCQgKsra2xZs0alJSUYPjw4eWW5zx27BiMjY0RGBiIhQsXws7ODt9//z2uXLmC/v371+h3VBdoFNiartVJRA2XU2sHXM68WukoccfWDjqoSvxyE46obc+5cKTGgQ1Ub3lNoGw1qvj4eGzduhVGRkZ4++23sXfv3nJrUVd2vK6W5xQrtYEdGxsLZ2dn1TeYf2vbtq1WiiIi8dGT6OF1m/E4++A8dibvhVxZCn09Gfw7j4Qjn8OuttKczGe0Z6htf5aqLq955MgRrFmzBgAwaNAgmJmZwdnZWXXJ2c3NDYmJifD09Kyzy3OKldp/Qb/88gsAYOvWrU/92bZtW60USETiIdWTwrnNy2jRyAQA0KKRCZzbvMywrgFZM9NntJvV6PxVXV7Tzc0NERERiIiIwDvvvANXV1ecOnUKOTk5UCgU+O2332BjY1Onl+cUK7U97L+nHt26dWul+yxZsgRz5859vlURkag1kjUq95Oqz9jeDcWp1yptb9bDrUbnr+nymvb29njjjTcwbtw4yOVy9OvXD6NHj4aenl6dXZ5TrGo809nIkSOxd+/e51VPlXCmM6K6iat1Va6qM51VNEr8b0ZdHGs8SlyXxLQ8Z12g0aAzdUQ8sykRaUl3M2sG9XMi0ZPCctS7yLt4AjkXjqA0JwOyZmZo1sMNTW1dRBvWgLiW56wLahzYnPmMiEi7JHpSGNsPei6jwesSMS3PWRdwJAgREZEIMLCJiIhEoMaBzXvYRERE2lfjwJ43b97zqIOIiIjUUPtYV9euXdUOKrt69apWitIUH+siIrGp6mNdRH975tSkgiBg9erVaNu2Lfz9/SGVSrFnzx7cu3evtmokImrQFEoFTtyOw7Gbp5FR+AhmjVvAtWM/vPKSE/T0aj4UKT8/H59//jlOnTqFxo0bo2nTppg5cyacnZ2fQ/XlhYeH48cff0RBQQH8/PwwZcoUAEBkZCQ2bNgAuVyOiRMnYvz48c/9vcVObWC3aNECAHDp0qVyM8UEBQWp5nwlIiLtUSgVCIv9GmdTL6heyyx4hGuZN3H+/kWEOL8BaQ2exRYEAW+++Sasra3xyy+/wMDAAFeuXMHUqVOxYsUKODo6PodPUSY+Ph6bN2/Grl27oKenh5EjR8LV1RXGxsYICwvDnj17YGBggICAADg6OqJTp07P7b3rA42+mhUWFuLmzZuq7WvXrkEul2utKCIiKnPidly5sH7S2dQLOHm74qU3NXX27Fncu3cP8+bNU00N2q1bN7z11ltYv349kpKS4OXlpdr/2LFjePPNNwEAmzZtwsiRI+Hj44Nly5ZBEASkpqbCw8MDY8eOxcSJE8u918GDBzFu3DgYGxujSZMm2Lx5M1q1aoWYmBg4OTnBxMQERkZGcHd3R1RUVLljV61ahZUrV6q2582bhwMHDiAjIwNvv/02Ro0ahdGjRyMmJgYAkJaWhsmTJ2PMmDFwdXXF559/DgDYs2cPAgMD4e3tjZUrVyIyMhK+vr4YNWoUgoODUVxcXKPfpzZpNHHKrFmz4O/vjy5dukCpVOLPP/9UfXgiItKeYzdPq20/eisGrh37Vvv8Fy9eRPfu3Z8ar/Tyyy9jxYoV6Nq1K/T09JCcnIzOnTtj//798PHxwcmTJ3Hp0iXs3r0bEokEs2fPxr59++Dg4IBbt27h66+/xgsvvFDunHfu3IGxsTFee+015Obmws/PD4GBgRWunZ2YmFju2NGjR2PChAkICQlBYWEhYmNj8fHHH2POnDkYPXo03NzckJ6ejnHjxuHnn3/G/v374eXlhZEjRyI3NxcuLi54/fXXAZSF+YEDByCTyeDm5obw8HCYmpoiLCwMN2/ehLV13ZylT6PAHjp0KBwcHPD7779DIpHAwcFBtZQaERFpT0bhI/XtBVk1Or9EIoFCoXjq9Sevovr6+uKXX35Bu3btcPbsWXz22WdYtWoVEhMTVbdHi4qK0KZNGzg4OMDU1PSpsAbK1s4+f/48Nm7ciNLSUrz22muwsrKq8PHgf3+BaNeuHdq2bYtz587h3r17cHFxgYGBAWJiYnDz5k3Vkp+lpaW4e/cuJk+ejDNnzuCbb77B9evXIZfLUVhYCKDsCoJMVhZ/rq6uGDt2LNzc3ODu7l5nwxrQMLBLSkpw/vx51ULhx44dQ0pKCkJCQrRaHBFRQ2fWuAUyCyoPbTOjmnWe7O3tsXXrVsjlcujr66tev3DhAmxtbQEAXl5emDBhArp27Yr+/fujUaNGUCgUmDBhAiZNmgQAyMnJgVQqxaNHj8qtTV2uVjMz2NjYoEmTJgCAAQMG4OLFi7C0tER8fLxqP3VrZ+/fvx/37t3DzJkzAZStnf3dd9/BxMQEQFnv2czMDEuWLMHdu3fh5eWFwYMHIyYmpsK1s0NDQ5GUlIQTJ05g9uzZmDFjBnx9fav769Qqje5hh4SE4IsvvsDSpUsRGRmJxYsX4/r169qujYiowXPt2E9t+6AO1b8cDpStR92pUyd89tlnql71pUuXsGHDBrz99tsAAEtLS7Ru3RqbNm2Cj48PgLI1qyMiIpCfn4/S0lJMnz4dhw4dUv9ZXF3x66+/oqSkBEVFRThz5gy6d++Ovn37IjY2FllZWSgsLER0dHSFa2d7eHggNjYWGRkZsLe3V9Xxww8/AABu3LgBHx8fFBYW4vTp05g8eTKGDRuG+/fvIy0t7am1s0tLSzF06FC0aNEC06ZNg6+vr84fV1ZHox721atXER0djYULF2LSpEkQBKHerC9KRFSXvfKSE87fv1jhwLM+L/SAy0tONX6PtWvXIiwsDF5eXpBKpWjevDmWL19eboS4r68vwsLCVK8NGjQISUlJGDNmDBQKBQYMGICRI0fir7/+qvR9PD09kZKSgpEjR6K0tBS+vr6qR8dCQkIQFBQEuVwOPz8/2NnZPXW8oaEhevTogc6dO6teCw0NxYcffghvb28AwLJly9C0aVNMmzYN77//Ppo1awZTU1N0794dqamp5c4nk8kQHByMSZMmwdDQEM2aNcPSpUur/4vUMo3Www4ICMCOHTvw9ddfo23bthg2bBj8/Pywe/fu2qixUpw4hYjEpjoTpyiUCpy8HYejt2KQUZAFM6OWGNShL1ye03PYYiAIAvLz8+Hv748tW7aUG6TWUGjUwzYyMkJkZCS6du2K8PBwdOzYEdnZ2VoujYiIAECqJ4Vrx741Gg0udhcvXsQbb7yB6dOnN8iwBjS8h/3hhx8iKSkJ/fv3h1QqRWBgICZPnqzt2oiIiAAAdnZ2OHv2LCZMmKDrUnRGo8DOzMzE5cuX4ePjgxs3bsDS0lJ1k5+IiIi0T6NL4gsWLMCYMWNgbW2tdjEQIiIi0g6NAtvAwOCpKeaIiIio9mh0Sbxjx464ePGitmshIiKiSqh9rOvv59ry8/ORnp6Odu3aqaZzA8qWQ9MlPtZFRGLD9bCputReEl+wYEFt1UFERERqaDRxSl3FHjYRiQ172FRdDWOKHCIiIpFjYBMREYmAVgM7MjISnp6eGDJkCLZv317pfsePH8egQYO0WQoREZGoafQcdnWkpaUhLCwMe/bsgYGBAQICAuDo6IhOnTqV2y8jI6NOr45CRERUF2ithx0TEwMnJyeYmJjAyMgI7u7uiIqKemq/0NBQzJgxQ1tlEBER1QtaC+z09PRyK6pYWFggLS2t3D7ff/89unXrplqInIiIiCqmtUviFT0t9uQ85MnJyYiOjsaWLVvw4MGDar2HqWnTatdHREQkJloLbEtLS8THx6u209PTYWFhodqOiorCw4cPMXr0aMjlcqSnp2PcuHFVWgWMz2ETkdjwOWyqLq1NnJKWloaxY8di9+7daNy4MQICAvDJJ5/Azs7uqX1TU1MRFBSEo0ePVuk9GNhEJDYMbKourd3DtrS0REhICIKCgjBixAh4eXnBzs4OU6ZM4UIiREREVcSpSYmIahF72FRdnOmMiIhIBBjYREREIsDAJiIiEgEGNhERkQgwsImIiESAgU1ERCQCDGwiIiIRYGATERGJAAObiIhIBBjYREREIsDAJiIiEgEGNhERkQgwsImIiESAgU1ERCQCDGwiIiIRYGATERGJAAObiIhIBBjYREREIsDAJiIiEgEGNhERkQgwsImIiESAgU1ERCQCDGwiIiIRYGATERGJAAObiIhIBBjYREREIsDAJiIiEgEGNhERkQgwsImIiESAgU1ERCQCDGwiIiIRkOm6AKp9CqUCcQ/OI/b+WTwqeowWhs3h3LoPnFo7QE/C73BERHURA7uBUSgV2Hx5Oy48vKR67VFxNm4+voPLmVfxus14SPWkOqyQiIgqwu5UAxP34Hy5sH7ShYeXcPbB+VquiIiINMHAbmBi759V2x5z/1wtVUJERFXBwG5gHhU9fkZ7du0UQkREVcLAbmBaGDZ/RrtJ7RRCRERVwsBuYJxb91Hb3rf1y7VUCRERVQUDu4Fxau2AHubdK2zrYd4djq0darkiIiLShEQQBEHXRVRXZmYelErRlq8zCqUCZx+cx+GUE8gozIJZ45YY3N4FjnwOm0jrzM2NdV0CiRQDm4ioFjGwqbrYnSIiIhIBBjYREZEIMLCJiIhEgIFNREQkAgxsIiIiEdBqYEdGRsLT0xNDhgzB9u3bn2r/9ddf4e3tjeHDh2Pu3LkoKSnRZjlERESipbXATktLQ1hYGH744QdERERg586duHHjhqq9oKAAixYtwrfffotffvkFxcXF2Lt3r7bKISIiEjWtBXZMTAycnJxgYmICIyMjuLu7IyoqStVuZGSEo0ePwszMDAUFBcjMzESzZs20VQ4REZGoybR14vT0dJibm6u2LSwskJiYWG4ffX19nDhxAu+//z4sLCzQv39/bZXTIAlKBUqTT6M48SCEnIeQNDNHI7thkHXuD4kehy8QEYmJ1gK7ognUJBLJU6+5uLggLi4OK1euxMKFC7FixQqN38PUtGmNaqzPBKUCaXtWoOha3D+vZd9H0cnNMEq7DMtR70KiJ9VhhUREVBVaC2xLS0vEx8erttPT02FhYaHazs7OxqVLl1S9am9vb4SEhFTpPTg1aeXkSSfLhfWTCq7F4X7MIeh3GVDLVRERpyal6tLaddG+ffsiNjYWWVlZKCwsRHR0NAYOHKhqFwQBs2fPxr179wAABw8eRK9evbRVToNTcu2k2nZ5kvp2IiKqW7Taww4JCUFQUBDkcjn8/PxgZ2eHKVOmIDg4GLa2tvjkk08wbdo0SCQSdOrUCR9//LG2ymlwhLwste3KvMxaqoSIiJ4HrtZVT+VH/BfKtBuVtkstrWDk+0EtVkREAC+JU/VxqHA9ZdBloNp2/a7q24mIqG5hYNdTss79IXvJoeK2lxwgs+pXyxUREVFN8JJ4PZJwIwNRcSnwcGwP+05mZc9hX49BScJBKHPSodfMAgb2wyCz6sfnsIl0hJfEqbq0NuiMat/Pv93CnbRcFJUoYN/JDBI9KfS7DODjW0RE9QC7WfVIUUlpuZ9ERFR/MLCJiIhEgIFNREQkAgxsIiIiEWBgExERiQADux5QKJX4LeEesnKKAQBZOcX4LeEeH3kjIqpH+By2yCmUSnz582X8nvzwqTaHzuZ4c4QNpHzmmqjO4HPYVF38L7nIxVx8UGFYA8DvyQ8Rc+lBLVdERETawMAWud8S79eonYiIxIGBLXJZuUXq23PUtxMRkTgwsEWupbGh+vZm6tuJiEgcGNgiN8CudY3aiYhIHBjYItfPtjUcOptX2ObQ2Rz9ujOwiYjqAz7WVQ8olErEXHqAbdHJkJcqoS/Tw2tDO6Nf99bQ05PoujwiegIf66LqYg+7HpDq6WGAXRu0NG4EAGhp3AgD7NowrImI6hEGNhERkQgwsImIiESAgU1ERCQCDGwiIiIRYGDXI4YGsnI/iYio/mBg1yMjBnRA1/YmGDGgg65LISKi54zPYRMR1SI+h03VxR42ERGRCDCwiYiIRICBTUREJAIMbCIiIhFgYBMREYkAA5uIiEgEGNhEREQiwMAmIiISAQY2ERGRCDCwiYiIREDUq0To6Ul0XQIREVGtEPVc4kRERA0FL4kTERGJAAObiIhIBBjYREREIsDAJiIiEgEGNhERkQgwsImIiESAgU1ERCQCDGwiIiIRYGATERGJAAO7nouMjISnpyeGDBmC7du367ocqmfy8vLg5eWF1NRUXZdCVO8xsOuxtLQ0hIWF4YcffkBERAR27tyJGzdu6LosqicSEhIwduxY3L59W9elEDUIDOx6LCYmBk5OTjAxMYGRkRHc3d0RFRWl67KonggPD8dHH30ECwsLXZdC1CCIerUuUi89PR3m5uaqbQsLCyQmJuqwIqpPPv30U12XQNSgsIddj1W0EJtEwiVJiYjEiIFdj1laWiIjI0O1nZ6ezsuXREQixcCux/r27YvY2FhkZWWhsLAQ0dHRGDhwoK7LIiKiauA97HrM0tISISEhCAoKglwuh5+fH+zs7HRdFhERVYNEqOhGJxEREdUpvCROREQkAgxsIiIiEWBgExERiQADm4iISAQY2ERERCLAwCbR6dmzZ5VWh7p48SKCg4MBAImJifjwww+feUxubi6Cg4Ph5eUFT09PbNq0SdV2+/ZtjBs3Dp6envDz88Off/5Z5c/g5eWFuLi4Kh9HRA0XA5vqPVtbW6xZswYAcOPGDaSlpT3zmNWrV8PS0hL79+/H7t27sWPHDvzxxx8AgPfeew9jx47FgQMHMHPmTAQHB1c4DSwR0fPEiVNIq+Li4vDpp5/CyMgIBQUFCA4OxsaNGyGXy2FoaIg5c+agZ8+eyMjIwIcffojMzEw8fPgQbdu2xapVq2Bqaor4+Hh88sknkEgksLW1hVKpVJ175cqVsLCwwPXr19G4cWPMnDkTW7duxa1btzB06FDMnz8fcXFx+OSTT/DVV19hzZo1yM3Nxbx587B48eJK6/7ggw+gUCgAAA8fPkRJSQmMjY2RlpaGmzdvYvjw4QAAFxcXfPzxx7hy5QpsbGwqPd+NGzcwf/58FBYWomPHjigoKFC1ffnllzh8+DCKi4tRWFiIOXPmYPDgwfDw8MCCBQvQv39/AEBoaCisrKzQv39/fPDBBygpKYEgCPDz88P48eNr/HdFRHWcQKRFZ86cEbp27SqkpqYKt27dEry8vISsrCxBEAQhOTlZ6Nevn5Cfny9s2bJF2LhxoyAIgqBUKoU33nhD+Oabb4Ti4mKhb9++QkxMjCAIghAZGSl07txZuHv3rnDmzBnB2tpauHz5siAIgjB58mTB399fKC4uFjIzMwUbGxvhwYMHwpkzZ4Thw4cLgiAIP/30kzB16lSN63/33XeF7t27CyEhIUJpaanwxx9/CO7u7uX2CQgIEA4fPqz2PL6+vkJ4eLggCIIQHx8vdOnSRThz5oyQmpoqBAYGCoWFhYIgCML+/fsFLy8vQRAE4dtvvxWCg4MFQRCE3NxcwcnJSXj8+LEwb9481e8qPT1dmDVrlqBQKDT+TEQkTuxhk9a1bt0abdu2xfbt25Geno6JEyeq2iQSCVJSUjBhwgTEx8fj22+/xe3bt3H9+nXY29sjOTkZMpkMzs7OAMru/T55D/qFF15At27dAADt27eHsbExDAwM0LJlSzRp0gSPHz+uUe2ff/45Pv74YwQHB2PdunWq3u6/SaXSSs/x6NEjXLt2DSNGjAAAODg4wMrKCgDQtm1bLF26FJGRkbhz5w4SEhKQn58PABg1ahTWrVuHrKwsREVF4ZVXXkGzZs0wZMgQzJkzB4mJiXB2dkZoaCj09Hh3i6i+479y0jojIyMAgFKphLOzMyIiIlR/wsPDYWVlheXLl2P16tVo0aIF/P390a9fPwiCAIlE8tT9YZnsn++ZBgYGlbbVxG+//aa6192kSRMMHz4cV65cQZs2bZCRkVGuprS0NLRq1arSc/29pOmTx/xd5+XLlxEQEIC8vDz069cPb7zxhmqfZs2awcPDA/v27cNPP/2EsWPHAgBcXV1x6NAhDBs2DFevXoW3tzdSUlKey+cmorqLgU21xsnJCadPn1aNqj5x4gR8fHxQXFyMU6dOYcKECRgxYgRMTU0RExMDhUKBzp07QxAEnDhxAgBw5MiRGvWapVIpSktLn7nfwYMHsW7dOgiCgJKSEhw8eBBOTk5o1aoV2rdvjwMHDgAoC3Y9PT107ty50nOZmJjAxsYGu3btAlAW0snJyQCAc+fOoXv37pg0aRL69OmDI0eOqO6dA8D48ePx/fffQxAE1cIt7777Lg4cOIDhw4fjo48+QtOmTXH//v1q/06ISBx4SZxqjZWVFRYtWoT/+7//gyAIkMlk2LBhA4yMjDB9+nQsW7YM69evh1QqRa9evZCSkgJ9fX2sW7cOCxcuxMqVK2FtbQ1TU9Nq19CzZ0+sWrUK06dPx7p16yrdb+7cufjoo4/g7e0NiUQCNzc3BAUFAQBWrlyJBQsWYMOGDTAwMMDq1aufeUl65cqVmDdvHnbs2IH27dujY8eOAMou8UdHR8PT0xP6+vpwdnbG48ePkZeXh6ZNm6Jr165o3rw5AgICVOd6++238cEHH2Dnzp2QSqUYPHgw+vTpU+3fCRGJA1frIqrDUlJSEBgYiKioKDRu3FjX5RCRDrGHTQ3SzZs3ERISUmFbhw4dsGrVqiqd78yZM5U+Jubo6Ij58+dXtUSsXr0a4eHh+OCDDxjWRMQeNhERkRhw0BkREZEIMLCJiIhEgIFNREQkAgxsIiIiEWBgExERiQADm4iISAT+HzHMb3syAshoAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 382 }, - { - "cell_type": "markdown", - "metadata": { - "id": "1MozjJKkZqz_" - }, - "source": [ - "\n", - "### Exercise: Train an `ExponentiatedGradient` model" - ] + "id": "7PdFAu3lT-pD", + "outputId": "7c3b6008-fdff-46dd-f749-ca0198a5193b" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Label imbalance\n", + "\n" + ], + "metadata": { + "id": "3i9KdmTWKUJZ" + } + }, + { + "cell_type": "markdown", + "source": [ + "Now that we have established the validity of our label, we will check frequency of its values in our data. The frequency of different labels is an important descriptive characteristic in classification settings for several reasons:\n", + "\n", + "* some classification algorithms and performance measures might not work well with data sets with extreme class imbalance\n", + "* in binary classification settings, our ability to evaluate error is often driven by the size of the smaller of the two classes (again, the smaller the sample the larger the uncertainty in estimates)\n", + "* label imbalance may exacerbate the problems due to smaller group sizes in fairness assessment\n", + "\n" + ], + "metadata": { + "id": "ViGqA5VTGrEo" + } + }, + { + "cell_type": "markdown", + "source": [ + "Let's check how many samples in our data are labeled as positive and how many as negative." + ], + "metadata": { + "id": "cos3--59EiZt" + } + }, + { + "cell_type": "code", + "execution_count": 26, + "source": [ + "df[\"readmit_30_days\"].value_counts() # counts" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "0 90406\n", + "1 11357\n", + "Name: readmit_30_days, dtype: int64" + ] + }, + "metadata": {}, + "execution_count": 26 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "markdown", - "metadata": { - "id": "lLeGnB4juJsM" - }, - "source": [ - "In this section, we will explore how changing the base model for the `ExponentiatedGradient` affects the overall performance of the classifier. \n", - "\n", - "We will instatiate a new `ExponentiatedGradient` classifier with a base `HistGradientBoostingClassifer` estimator. We will use the same `difference_bound` as above." - ] + "id": "eR5ULLYGE4UK", + "outputId": "02f2e533-2a7f-44af-e57f-c24c7b8b4d4d" + } + }, + { + "cell_type": "code", + "execution_count": 27, + "source": [ + "df[\"readmit_30_days\"].value_counts(normalize=True) # frequencies" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "0 0.888\n", + "1 0.112\n", + "Name: readmit_30_days, dtype: float64" + ] + }, + "metadata": {}, + "execution_count": 27 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "markdown", - "metadata": { - "id": "ghjfKhtB3Kl9" - }, - "source": [ - "1.) First, let's create our new `ExponentiatedGradient` instance in the cells below and fit it to the training data." - ] + "id": "0UWBbg4Cz90t", + "outputId": "a0e1e14b-47f9-4f49-da41-ba53a7a63082" + } + }, + { + "cell_type": "markdown", + "source": [ + "As we can see, the target label is heavily skewed towards the patients not being readmitted within 30 days. In our dataset, only 11% of patients were readmitted within 30 days.\n", + "\n", + "Since there are fewer positive examples, we expect that we will have a much larger uncertainty (error bars) in our estimates of *false negative rates* (FNR), compared with *false positive rates* (FPR). This means that there will be larger differences between training FNR and test FNR, even if there is no overfitting, simply because of the smaller sample sizes. \n", + "\n", + "Our target metric is *balanced error rate*, which is the average of FPR and FNR. The value of this metric is robust to different frequencies of positives and negatives. However, since half of the metric is contributed by FNR, we expect the uncertainty in balanced error values to behave similarly to the uncertainty of FNR." + ], + "metadata": { + "id": "b-_K_KHXz9MR" + } + }, + { + "cell_type": "markdown", + "source": [ + "Now, let's examine how much the label frequencies vary within each group defined by `race`:" + ], + "metadata": { + "id": "OkopoyE3GQ8g" + } + }, + { + "cell_type": "code", + "execution_count": 28, + "source": [ + "sns.barplot(x=\"readmit_30_days\", y=\"race\", data=df, ci=95);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAckAAAEJCAYAAAD/19zFAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAknUlEQVR4nO3deXiNd97H8ffJKiS2CJ6a6mjHVkKpEjWqlg4iwmN5bCXtWKqLoFJbUFrD1FZRqjXdUB7a8iCh2ospk1ajGFNKSqaqRNvIJpLIfn7PHy7nqpGbiCQnic/rL7mX3/39npPr/uR3n+O+bcYYg4iIiNzAxdkFiIiIlFcKSREREQsKSREREQsKSREREQsKSREREQsKSREREQsKSREREQtuzi5A7lxqaiZ2e+X7766+vt4kJ2c4u4wSV1n7gsrbW2XtC+7O3lxcbNSqVa1IYygkKwG73VTKkATUVwVUWXurrH2BersZXW4VERGxoJAUERGxYNO9W0VExJmyc/JIv5xd4uP6+fmQmJh+w3IXFxu+vt5FGkOfSVYCoQu3kZSa6ewyRESKZeOiEaRT8iFZEnS5VURExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExEKph+Tp06dp2rQpn332mWPZ/v376dq1K1OmTLlh+379+pVKHaGhofTt27dUxgY4fvw44eHhpTa+iIiUvVIPya1bt9KzZ082bdrkWLZ7927Gjx/P0qVLb9h++/btJV5DamoqJ0+epFq1ahw5cqTExwfw9/fnL3/5S6mMLSIizuFWmoPn5+ezY8cONmzYwNChQzl37hwHDx5k7969fP3117i4uLBjxw5q1KhBXFwcy5cvp3///pw6dYpLly4RHh7OmTNn8PDwYPr06XTs2JEPP/yQ7du3k5WVhc1mY/ny5TzwwAN069aN4OBgvvzyS7Kysnjttddo2bIlAJGRkbRr144mTZqwefNmHn74YeBqgO/bt4+LFy/y66+/EhISws8//0xMTAw1a9bknXfewdPTk23btrF27VrsdjstWrTg5ZdfxtPTk4CAAFq0aEFSUhJTp07lrbfeYv369cTGxjJnzhyys7OpUaMGS5YsoU6dOsydO5e4uDiSkpJo1KgRK1euJCkpiRdeeIHGjRsTGxuLr68vERER1KxZszTfGhERKYJSDcl9+/Zxzz330KhRI3r06MGmTZuYOnUqR44coX379gwYMIAdO3bQtGlTVq5ced2+ERERNGzYkFWrVnHq1CnmzJmDv78/e/bsYf369VSpUoWIiAg2btzI7NmzAahZsyaffPIJ69ev5+233+aNN94Arobh5MmTadKkCREREcycOdMRQsePHycyMpK0tDS6devGO++8Q3h4OCNHjiQ6Opr77ruPjz76iE2bNuHp6cnSpUt59913ee6550hNTWXcuHF06NCBgwcPOmoPCwsjLCyMrl27snHjRtauXUu3bt1wd3dn8+bN2O12QkJC2L9/Py1atOD7779nwYIFPPjgg0yYMIHIyEhGjhxZmm+NiEipyEk7T8bP/8QU5BV5n1GjPqOgwNzWcby8qjB48HDatm13uyXellINya1btxIUFARAYGAgYWFhTJo06YbtWrVqdcOyQ4cOsWTJEgCaNm3K5s2bAVi6dCk7d+7k7NmzREdH07x5c8c+nTt3BqBx48Z8/vnnAMTGxvLLL7/w6KOP4u7uTvPmzdm2bRtPPfUUAG3btsXb2xtvb28AOnbsCECDBg24fPkyBw8e5KeffuJ//ud/AMjLy+PBBx90HLN169bX1Z2SkkJiYiJdu3YFYPjw4Y51NWvWZMOGDZw5c4azZ89y5coVAHx9fR1jNm7cmLS0tJu/sCIi5VTmr8fJv5J8W/tcuHC5WMeKitpWcUMyOTmZf/zjH3z33XesW7cOYwyXL192hNdvValS5cbC3K4v7YcffqBKlSqEhITw5JNP8thjj1GnTh1iY2Md23h6egJgs9kcy7Zs2UJubi49e/YEIDMzk02bNjlC0t3d/abHLSgooHfv3syaNcuxf0FBgWXt/zleTk4OFy9e5PTp06xYsYJRo0YxYMAAUlNTMcZcV/e12q8tFxGpaKrV9yfj57zbmknWr+NTrJlkUFD/26zu9pVaSO7YsYOAgADeeecdx7I33niDzZs306BBg1vu365dO3bt2kWzZs344YcfGDt2LNOmTeO+++7jqaeeIjc3l7feeovatWtbjpGbm0tkZCQffPCBY8aXkZFBly5drrs8ejMdOnTgvffe49lnn6V27drMnTuXhg0bMmHChEK39/HxoX79+nz11Vd06tSJ7du3880331CzZk169+7NwIEDSUhI4NChQ45Zq4hIZeFZ4148a9x7W/usWzSCxMT0UqrozpRaSF77HPC3hg8fzjvvvEOdOnVuuX9oaCizZs0iODgYNzc3Fi1aRPPmzdm0aROBgYF4eHjQqlUr4uLiLMf44osvaNCgwXWXRL29vRk8eDCbNm1yXJ69mWbNmvHCCy8QEhKC3W6nefPmjBs37qb7LF68mLlz57Jo0SJq1arFokWLSE1NJSwsjN27d+Ph4cFDDz1EfHz8LY8vIiLOYzO6tlfhhS7cRlJqprPLEBEplo2lNJP08/MpdFwXFxu+vt5FGkN33BEREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbGgkBQREbFgM8YYZxchIiJ3r+ycPNIvZ5f4uH5+PiQmpt+w3MXFhq+vd5HGcCvpoqTsJSdnYLdXvr91rH7BK7rK2hdU3t4qa19QuXsrCbrcKiIiYkEhKSIiYkEhKSIiYkEhKSIiYkEhKSIiYkEhKSIiYkEhKSIiYkEhKSIiYkEhKSIiYkG3pRMRkXIhPzeH1LTcEhtPt6UTAI6/NY3cy8nOLkNE5I48PPUdoORCsiTocquIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIiFSheSGRkZzJs3j6CgIPr168fIkSM5ceKE0+rp16+f044tIiJ3xs3ZBZQku93O2LFj6dChA9u2bcPNzY2YmBjGjh3Lzp07qVWrVpnXtH379jI/poiIlIxKFZIHDx7k4sWLhIaG4uJydZIcEBDAwoULsdvtzJo1i7i4OJKSkmjUqBErV64kKSmJUaNG8fe//x2AN954A4AJEyYQGRnJ6tWrsdls+Pv78+qrr5KSksLMmTNJT08nMTGRPn36EBYWxvfff8+cOXPIz8/H09OThQsX8vvf/56mTZty6tQpEhISCt1v69atREdHk5aWxvnz5+nUqRNz58511ksoIlKqYhOv8Pm/U8nJt9+wzvO7URQUmEL38/KqwuDBw2nbtl1pl3idShWSJ0+exN/f3xGQ13Tp0oVDhw7h7u7O5s2bsdvthISEsH//flq0aFHoWAkJCSxcuJCtW7dSv359XnrpJfbv389PP/1EUFAQ//3f/016ejpdunThz3/+M2vXruXpp5+md+/e7Nq1i3/961/8/ve/d4wXFRVV6H4AR48eJSoqCldXV3r16sWwYcNo2rRpqb1OIiLOsv/HNC5czi185ZULN903KmqbQvJOuLi4YEzhf4U88sgj1KxZkw0bNnDmzBnOnj3LlStXLMc6evQobdu2pX79+gAsXrzYsS4mJoZ3332XuLg48vLyyMrKokuXLrzyyitER0fTtWtXevbsed14o0ePLnQ/gDZt2uDt7Q3AvffeS1pa2h29DiIi5VWXRjXIKbAXPpOsVe+mM8mgoP6lXN2NKlVItmzZko0bN2KMwWazOZYvW7aMVq1a8cYbbzBq1CgGDBhAamqqY7vfBmt+fj5ubm64uV3/0qSkpACwZs0azp8/T1BQED169ODAgQMYY+jVqxdt2rThiy++YO3atezfv5/58+c79v/rX/9a6H4Anp6eju3+sx4RkcqkuV9VmvtVLXTdw1PfITExvYwrurlK9e3Wdu3a4evry8qVKykoKAAgOjra8blf7969GThwIHXq1OHQoUMUFBRQvXp10tLSSElJITc3l+joaAD8/f359ttvSUxMBGDBggXs3buXr776itGjR9O7d29++eUXEhISsNvtTJo0iWPHjjF06FAmTpzIyZMnr6vNaj8RESm/KtVM0maz8eabb7Jw4UKCgoJwc3OjVq1arFmzBldXV8LCwti9ezceHh489NBDxMfH4+Pjw+jRoxk0aBD169fH398fgHr16hEeHs7o0aOx2+089NBDDBgwAC8vL6ZOnUr16tXx9fWlZcuWxMfHM378eMLDw3nzzTdxdXVl+vTp19X2zDPPFLqfiIiUXzaja3sV3vG3ppF7OdnZZYiI3JGSvtzq5+dT6HguLjZ8fb2LNEalutwqIiJSkhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFooUkpmZmcybN4+QkBAuXbrEnDlzyMzMLO3aREREnKpIITl//nyqV69OcnIynp6eZGRkMGfOnNKuTURExKmKFJKxsbFMnjwZNzc3vLy8WLJkCbGxsaVdm4iIiFO5FWUjF5frs7SgoOCGZeI8/uNfc3YJIiJ3LD83x9kl3KBIIfnII4+wePFisrOziY6O5sMPP6RDhw6lXZsUUXJyBna7cXYZJc7qqeIVXWXtCypvb5W1L6jcvZWEIk0Hw8LCqFq1Kj4+Prz++us0a9aMqVOnlnZtIiIiTlWkmaS7uzvt27fn+eef59KlSxw+fBhPT8/Srk1ERMSpijSTfP3111mxYgUA2dnZrFmzhjfffLNUCxMREXG2IoXk3r17ee+99wCoX78+H374Ibt27SrVwkRERJytSCGZl5eHu7u742d3d3dsNlupFSUiIlIeFOkzybZt2zJlyhQGDRqEzWZj27ZttG7durRrExERcaoizSRnz55NnTp1WLhwIYsWLcLX15fw8PDSrk1ERMSpijSTrFq1KjNmzCjtWkRERMqVIoXk0aNHWbNmDVeuXMEYg91uJz4+nn379pVyeSIiIs5jM8bc8lYtffr0oV+/fnz22WcMHTqUvXv30rBhQ2bOnFkWNYqISCnLyc3lclr5uy3cnbC6m5CLiw1fX+8ijVGkmaTNZmPcuHGkpqZy//33ExwczLBhw26vWik1YR/PIykjxdlliEgF9sHTEUDlCsmSUKQv7lStWhWAhg0bEhcXh6enJwUFBaVamIiIiLMVaSb54IMPMmnSJCZOnMgzzzzD2bNncXV1Le3aREREnKpIM8kjR47w9NNP06hRI2bOnIndbmfJkiWlXZuIiIhTFWkmWa1aNerVqwfA448/zuOPP16aNYmIiJQLRQrJrKwsunfvTv369R2fTwJERkaWWmEiIiLOVqSQ1N11RETkblSkkGzfvn1p1yEiIlLuFOmLOyIiIncjhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFhaSIiIgFheRtyszMZN68eTzxxBMEBwczfPhwvv76awA2b95MVFQUANOnT2fr1q3OLFVERO5QkW5LJ1cZYxg/fjzNmzdn586deHh4cPLkScaNG8fSpUs5evSobuEnIlKJKCRvwzfffMPPP//MunXrsNlswNUHUj/77LM89dRT+Pj4EBMTg5+fHwD79u1j48aNJCcnM378eIYMGUJmZiavvPIKcXFxFBQUMHbsWIKCgti6dSv/93//x6VLl+jatSsvvviiM1sVEREUkrfl+PHjtGzZ0hGQ1zzyyCN4eXnRrVs32rdvT+fOndm5cye5ubl8/PHHxMXFMWrUKIYMGcLq1atp0aIFr732GhkZGQwdOpTWrVsDkJCQwK5du3Bz09siIpAVn07asQRMnr3UjzXqi1EUFJhSPw6Al1cVBg8eTtu27crkeHdCZ+PbYLPZKCgouGF5Xl5eodt3794dm81G48aNSU1NBeDAgQNkZ2ezZcsWAK5cuUJcXBxwdVaqgBSRa9JPJpKXkl0mx7qQfqFMjnNNVNQ2hWRl07p1a9avX09eXh7u7u6O5f/617/w9/e/YXtXV1eA62aedrudxYsX06JFCwCSkpKoUaMGkZGRVKlSpZQ7EJGKxOdBP+z5ZTOTrFfdr0xnkkFB/cvkWHdKIXkb2rVrxx/+8AcWLFjAzJkzcXd357vvvmP16tUsXbqUHTt2FDrT/K2AgAD+93//l/nz53Px4kX69+/Ppk2byqgDEalIvH7ng9fvfMrkWB88HUFiYnqZHKsiUUjeppUrV/L6668TFBSEq6srNWrUYPHixXTo0IGkpCSWLVuGj4/1L/ULL7zA3LlzCQoKoqCggJdeeomGDRty+PDhMuxCRESKwmaMKZv5tZSasI/nkZSR4uwyRKQCq4wzST8/n0J7cnGx4evrXaQxdDMBERERCwpJERERCwpJERERCwpJERERCwpJERERCwpJERERCwpJERERCwpJERERCwpJERERCwpJERERCwpJERERCwpJERERCwpJERERCwpJERERCwpJERERCwpJERERC3rosoiIkJOby+W0HGeXUaJK4qHLbiVdlJS95OQM7PbK97eO1S94RVdZ+4LK21tl7Qsqd28lQZdbRURELCgkRURELCgkRURELCgkRURELCgkRURELCgkRURELCgkRURELCgkRURELCgkRURELOiOO5VAUW+vVBH5+fk4u4RSUVn7gsrb2836ysvO4VJ6bhlWI2VFIVkJfPFiGFlJyc4uQ+SuFbjufVBIVkq63CoiImJBISkiImJBISkiImJBISkiImJBISkiImJBISkiImJBISkiImJBISkiImJBISkiImJBISkiImJBISkiImJBISkiImJBISkiImJBISkiImJBISkiImJBISkiImLhrgjJ+Ph4unXrdsPypk2bWu5z8OBBRo4cWZpliYhIOXdXhKSIiEhxuDm7AGfbunUr0dHRpKWlcf78eTp16sTcuXOv22bt2rXs2bOHNWvWMG7cOPz9/Tly5AgpKSnMmjWLLl26kJSURHh4OD///DNubm5MnjyZli1bEhwczJdffglA586dmTFjBoGBgaxZswabzUZ2djYJCQn89NNPXLhwgcGDB/Pss8864ZUQEZH/dNeHJMDRo0eJiorC1dWVXr16MWzYMMe6LVu28Pnnn/O3v/0NLy8vAPLy8ti8eTN///vfiYiIoEuXLrz66qsEBATw9NNPc/78eYYNG8a2bdu45557OH36NK6urhQUFPDNN98QGBjIP/7xD+bOncunn37KqVOn2LBhA+np6fTo0YMRI0ZQvXp1Z70cIhXamewsvrp8iTy7KbNjbho1ioKCsjveNV5eVRg8eDht27Yr82PfLe6KkHRxufGqsjEGm80GQJs2bfD29gbg3nvvJS0tDYDTp08zZ84cli1bRtWqVR37du7cGYDGjRtz6dIlAGJiYpg/f75jjNatW/Ptt9/SpUsXvv76a9zc3Bg1ahQ7d+4kPT2dxMRE/vCHPwDQoUMHPDw88PX1pWbNmqSnpyskRYrpUPplLubllekxUy9cKNPj/VZU1DaFZCm6K0KyevXqpKenX7csOTmZGjVqAODp6elYbrPZMObqX4TVqlVjwYIFLFiwgM6dOzuC8tr210IWcOzz258LCgro0qULK1euxMPDg4kTJ/Lpp58SGRnpCNqbHV9Ebt8jPtXJLeOZZNX69Zw2kwwK6l/mx72b3BUh6e3tzX333cdnn31Gz549Adi8eTMdO3a86X4NGjSge/fu7NmzhxUrVjB9+nTLbQMCAvjkk08cl1v/+c9/MnfuXPz8/Pjxxx/x9PTkgQceoEOHDqxevZqFCxeWaI8ictX9Vby4v4pXmR4zcN37JCam33pDqXDumm+3Ll68mI0bNxIcHEzv3r2Ji4tjzpw5Rdp36tSpREZGcuLECcttwsPDiYmJoW/fvjz33HPMnz+funXrYrPZePjhh3nggQeAq2GakZFB+/btS6QvEREpPTaja3sV3hcvhpGVlOzsMkTuWhV5Junn51Nha78Vq95cXGz4+noXaYy7ZiYpIiJyuxSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFhSSIiIiFtycXYDcua7Llji7BJG7Wl52jrNLkFKikKwEkpMzsNuNs8socX5+PiQmpju7jBJXWfuCyttbZe1Lbk2XW0VERCwoJEVERCzocmsl4OJic3YJpaay9lZZ+4LK21tl7Qvuvt5up1+bMabyfZglIiJSAnS5VURExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCUkRExIJCshyLjIwkMDCQJ554gg0bNtywPjY2loEDB9KzZ0/Cw8PJz88H4Oeff2bEiBH06tWLZ599lszMzLIu/aaK29eRI0cYOHAg/fr1IyQkhAsXLpR16bdU3N6uOXnyJC1btiyrcm9LcXu7ePEi48aNo3///gwdOpT4+PiyLv2mittXfHw8I0aMoF+/fowcObJC/j5eM23aNLZu3er4uaKfQ675z76KdQ4xUi79+uuvpmvXriY1NdVkZmaavn37mri4uOu26dOnjzl69KgxxpgZM2aYDRs2GGOMGTdunImKijLGGLNy5UqzaNGiMq39Zu6kr65du5rY2FhjjDEff/yxGT9+fJnWfit30psxxly5csUMGTLENGnSpCzLLpI76S0kJMRs3LjRGGPMxo0bzcSJE8uy9Ju6k77CwsIc/163bp2ZMmVKmdZ+K0Xp7ddffzXPPPOMadWqldmyZYtjeUU/h1j1VZxziGaS5dSBAwcICAigZs2aVK1alZ49e7J7927H+gsXLpCdnc1DDz0EwIABA9i9ezd5eXkcOnSInj17Xre8vChuX7m5uUycOJFmzZoB0LRpU3755RdntGCpuL1d89e//pWnnnqqjKsumuL2lpKSwvfff8/QoUMBGDhwIJMmTXJCB4W7k/fMbreTkZEBQFZWFlWqVCnz+m/mVr3B1RlZ9+7d6d27t2NZRT+HQOF9FfccopAspy5evIifn5/j57p165KQkGC53s/Pj4SEBFJTU/H29sbNze265eVFcfvy8PCgX79+wNWT08qVK+nRo0fZFV4Exe0NYO/evWRnZ9OrV6+yK/g2FLe38+fPc88997BgwQKCg4MJDQ3F3d29TGu/mTt5zyZOnMgHH3xA586dee+99xg7dmzZFV4Et+oNYMyYMQwePPi6ZRX9HAKF91Xcc4hCspwyhTycxWaz3XL9rfZztuL2dU1ubi5hYWHk5+fzzDPPlE6RxVTc3hITE1m9ejWzZ88u1fruRHF7y8/P5+TJkzz66KPs2LGD7t27M3369FKt9Xbcye/jtGnTeOWVV4iOjmbevHm88MILhW7vLMU9F1T0c8it3O45RCFZTtWrV4+kpCTHzxcvXqRu3bqW6xMTE6lbty61a9cmIyODgoKC65aXF8XtCyAzM5MxY8aQn5/P6tWry9WMBIrf2759+7h06ZLjSyAA/fr1c1zKKw+K25ufnx/VqlWja9euAAQFBXHs2LGyK/wWittXSkoKZ86cccxEevbsSWJiIqmpqWVX/C3cqjcrFf0ccjPFOYcoJMupRx99lK+//pqUlBSysrL4/PPPeeyxxxzrGzRogKenJ0eOHAFg27ZtPPbYY7i7u9OuXTt27dp13fLyorh9Abz00kvcd999RERE4OHh4ZT6b6a4vQ0ePJg9e/awfft2tm/fDsD27dvx9vZ2Sh+FKW5vDRs2pF69euzfvx+AL774ghYtWjilh8IUt69atWrh6enJ4cOHgavfmqxWrRq1a9d2Sh+FuVVvVir6OeRminUOucMvGkkp2rFjh+nTp4/505/+ZNasWWOMMWbMmDHm2LFjxhhjYmNjzcCBA02vXr3Miy++aHJycowxxsTHx5snn3zS9O7d2/z5z382ly5dcloPhSlOXydOnDBNmjQxgYGBJjg42AQHB5sxY8Y4s41CFfc9+63y+O1WY4rf2w8//GCefPJJ06dPHzNkyBDz448/OquFQhW3r2+//dYMGjTIBAUFmSFDhpgTJ044rQcrt+rtmmnTpl33LdCKfg655rd9FfccYjOmHF1EFxERKUd0uVVERMSCQlJERMSCQlJERMSCQlJERMSCQlJERMSCQlLkLtWmTZvbeiLH8ePHCQ0NBeDYsWPMmTPnlvukp6cTGhpKUFAQgYGBrFmzxrHu7NmzDB8+nMDAQAYNGsQPP/xw2z0EBQVx8ODB295PpKgUkiJSJP7+/qxYsQKAf//730W6n2dERAT16tUjKiqKTz75hE2bNnH06FEAwsLCGDZsGLt27WLChAmEhoaWq9u6iQC4ObsAEbm1gwcP8pe//IWqVaty5coVQkNDefvtt8nLy6NKlSpMmzaNNm3akJSUxJw5c0hOTiYxMZEGDRqwfPlyfH19OXz4MK+++io2mw1/f3/sdrtj7GXLllG3bl3i4uLw8vJiwoQJrF+/nh9//JE//elPzJw5k4MHD/Lqq6/yt7/9jRUrVpCens6MGTNYuHChZd3h4eHX3d4sNzcXHx8fEhISOHPmDH369AGgS5cuzJs3j5MnT970jjz//ve/mTlzJllZWdx///1cuXLFse6tt95iz5495OTkkJWVxbRp0+jRowe9evVi9uzZ/PGPfwRg1qxZNG7cmD/+8Y+Eh4eTm5uLMYZBgwYxYsSIO36vpJIp2fsgiEhpiImJMc2aNTPx8fHmxx9/NEFBQSYlJcUYY8zp06dNp06dTGZmpvnggw/M22+/bYwxxm63mzFjxph3333X5OTkmEcffdQcOHDAGGNMZGSkadKkiTl//ryJiYkxzZs3d9wxZvTo0WbIkCEmJyfHJCcnmxYtWphff/3VxMTEmD59+hhjjNmyZYsZN25ckeufMmWKadmypZk8ebLJz883R48eNT179rxum6FDh5o9e/bcdJx+/fqZjz76yBhjzOHDh03Tpk1NTEyMiY+PNyNHjjRZWVnGGGOioqJMUFCQMcaY999/34SGhhpjjElPTzcBAQEmLS3NzJgxw/FaXbx40UyaNMkUFBQUuSe5O2gmKVJB/Nd//RcNGjRgw4YNXLx48bpnT9psNs6dO0dISAiHDx/m/fff5+zZs8TFxdG6dWtOnz6Nm5sbHTt2BK5+lvfbzxR/97vf8eCDDwLQsGFDfHx88PDwoHbt2lSrVo20tLQ7qn3JkiXMmzeP0NBQVq1a5ZjV/SdXV1fLMVJTUzl16hT9+/cH4OGHH6Zx48bA1Xusvvbaa0RGRvLTTz/x7bffkpmZCVx9HuKqVatISUlh9+7dPP7441SvXp0nnniCadOmcezYMTp27MisWbNwcdEnUHI9/UaIVBBVq1YFrj4Lr2PHjo4bom/fvp2PPvqIxo0bs3jxYiIiIqhVqxZDhgyhU6dOGGMKfYzatecFAjfc7Pm36+5EdHS047PLatWq0adPH06ePMk999xDUlLSdTUlJCRQv359y7GuPQ7pt/tcq/PEiRMMHTqUjIwMOnXqxJgxYxzbVK9enV69erFjxw62bNnCsGHDAOjatSufffYZvXv3JjY2lr59+3Lu3LkS6VsqD4WkSAUTEBDAV1995fg26P79+wkODiYnJ4cvv/ySkJAQ+vfvj6+vLwcOHKCgoIAmTZpgjHE8jWPv3r13NDt0dXUlPz//ltt9+umnrFq1CmMMubm5fPrppwQEBFC/fn0aNmzoeNJEdHQ0Li4uNGnSxHKsmjVr0qJFCz7++GPgajCePn0agEOHDtGyZUuefvpp2rdvz969ex2fhQKMGDGCdevWYYyhVatWAEyZMoVdu3bRp08fXn75Zby9vYv0pHq5u+hyq0gF07hxY1555RVefPFFjDG4ubmxevVqqlatyvPPP8+iRYt48803cXV1pW3btpw7dw53d3dWrVrF3LlzWbZsGc2bN8fX17fYNbRp04bly5fz/PPPs2rVKsvtpk+fzssvv0zfvn2x2Wx0796dUaNGAbBs2TJmz57N6tWr8fDwICIi4paXO5ctW8aMGTPYtGkTDRs25P777weuXj7+/PPPCQwMxN3dnY4dO5KWlkZGRgbe3t40a9aMGjVqMHToUMdYzz33HOHh4WzevBlXV1d69OhB+/bti/2aSOWkp4CISKV37tw5Ro4cye7du/Hy8nJ2OVKBaCYpIsV25swZJk+eXOi6Ro0asXz58tsaLyYmxvK/lHTo0IGZM2febolERETw0UcfER4eroCU26aZpIiIiAV9cUdERMSCQlJERMSCQlJERMSCQlJERMSCQlJERMSCQlJERMTC/wPSUeYnexBr/AAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 285 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "CNjDh4JUAc6i" - }, - "outputs": [], - "source": [ - "# Create ExponentiatedGradient instance here\r\n", - "expgrad_exercise = ExponentiatedGradient(\r\n", - " estimator=_______,\r\n", - " constraints=TruePositiveRateParity(difference_bound=____)\r\n", - ")" - ] + "id": "tcH3Mxwmm4rC", + "outputId": "75641d75-670d-4a81-a354-855f80c38613" + } + }, + { + "cell_type": "markdown", + "source": [ + "We see the rate of *30-day readmission* is similar for the *AfricanAmerican* and *Caucasian* groups, but appears smaller for *Other* and smallest for *Unknown* (this is consistent with an overall lower rate of hospital visits in the prior year). The smaller sample size of the *Other* and *Unknown* groups mean that there is more uncertainty around the estimate for these two groups." + ], + "metadata": { + "id": "ox06aTMmm4rB" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Proxies for sensitive features\n", + "\n" + ], + "metadata": { + "id": "0AA5uoqAKUSx" + } + }, + { + "cell_type": "markdown", + "source": [ + "We next investigate which of the features are highly predictive of the sensitive feature *race*; such features are called *proxies*.\n", + "\n", + "While in this tutorial we examine fairness issues through the **impact** of the machine-learning model on different populations, there are other concepts of fairness that seek to analyze how the **model might be using information** contained in the sensitive features, and which of the information uses are justified (often using causal reasoning). More pragmatically, certain uses of sensitive features (or proxies of it) might be illegal in some contexts.\n", + "\n", + "Another reason to understand the proxies is because they might explain why we see differences in impact on different groups even when our model does not have access to the sensitive features directly.\n", + "\n", + "In this section we briefly examine the identification of such proxies (but we don't go into legal or causality considerations).\n" + ], + "metadata": { + "id": "7PSup7dMJjJg" + } + }, + { + "cell_type": "markdown", + "source": [ + "In the United States, *Medicare* and *Medicaid* are joint federal and state programs to help qualified individuals pay for healthcare expenses. *Medicare* is available to people over the age of 65 and younger individuals with severe illnesses. *Medicaid* is available to all individuals under the age of 65 whose adjusted gross income falls below the Federal Poverty Line. " + ], + "metadata": { + "id": "1PHULa3eQEcn" + } + }, + { + "cell_type": "markdown", + "source": [ + "First, let's explore the relationship between patients who paid with *Medicaid* and our demographic features. Because *Medicaid* is available to low-income individuals, and race is correlated with socioeconomic status in the United States, we expect there to be a relationship between `race` and paying with *Medicaid*. " + ], + "metadata": { + "id": "VNYx0OaVElUC" + } + }, + { + "cell_type": "code", + "execution_count": 29, + "source": [ + "sns.pointplot(y=\"medicaid\", x=\"race\", data=df, join=False);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEJCAYAAACKWmBmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAoO0lEQVR4nO3dfVxUdd7/8dcAgiIqhgxuPLLcLPMGJTUhrxWzNJLw5kLd3G7ItguzfkbRapuCpWZiRprm5mo3m2t4iSWBZKF5GwWaknmTeJ+tkjsDgqYEMsD8/vBqNlIOUzKA+H4+Hj2mM59z5nwGZ+Y958w532Oy2+12REREauDW0A2IiEjjpqAQERFDCgoRETGkoBAREUMKChERMaSgEBERQwoKEREx5NHQDbhCcXEJVVU6PURExBlubibatm1ZY71JBkVVlV1BISJSR7TrSUREDCkoRETEkIJCREQMKShERMSQgkJERAwpKERExJCCQkSq2XW4kJeTv2LX4cKGbkUaCZcGRUZGBhEREQwePJjk5OSL6nl5eYwcOZLw8HDi4+OpqKgAYPfu3YwcOZKhQ4fy2GOPUVBQ4Mo2ReRn0rK+5cDx06RlfdvQrUgj4bKgsFgszJs3j+XLl5Oenk5KSgqHDx+uNs+kSZOYOnUqa9euxW63s3LlSux2O7GxsUyaNImMjAyGDx/O1KlTXdWmiPxCWXlFtVsRlwVFdnY2oaGh+Pr64u3tTXh4OJmZmY56fn4+ZWVlBAcHAxAVFUVmZibFxcWUlZURGhoKwMCBA/n8888pLy93VasiImLAZUFhtVrx9/d3TJvNZiwWS411f39/LBYLbdu2xdvbm88//xyANWvWYLPZKC4udlWrIiJiwGVjPdntF4+1ZDKZaq2bTCYWLFjAyy+/TFJSEsOHD8fX15dmzZo5vW4/P5/f1rSI4O7u5rj192/VwN1IY+CyoAgICGDHjh2OaavVitlsrlYvLPzPURUFBQWOuoeHB8uWLQPg9OnTvPHGG/j6+jq97lOnzmlQQJHfqLKyynFbUHC2gbuR+uDmZjL8gu2yXU/9+vUjJyeHoqIiSktLWbduHWFhYY56YGAgXl5e5ObmApCWluaoT5kyhd27dwPwzjvvcM899+DmpiN5RUQagku3KOLi4oiOjsZmszFq1Ch69OhBTEwMsbGxBAUFkZSUREJCAiUlJXTt2pXo6GgApk2bxgsvvEBpaSmdO3fmpZdeclWbIiJSC5P9Uj8WXOG060nkt5u8OAdLcSkBbVuQ+NjtDd2O1IMG2/UkIiJNg4JCREQMKShERMSQgkJERAwpKERExJCCQkREDCkoRETEkIJCREQMKShERMSQgkJERAwpKERExJCCQkREDLls9FgRubIcPH6azV/nU/TDeQDO2yqx2+3VLjgmVyeNHisifPjZUTKyj110f0jXAGIiu+LmprBoyjR6rIgYyvuu+JIhAbBtn4XPdn1fvw1Jo6OgELnKbd6Zb1jfVEtdmj4FhchVzlL0o3G92LguTZ+CQuQq17qlp3Hd27guTZ+CQuQqd3v39ob1frXUpelTUIhc5fp2MdPzRr9L1jqYfQjv26GeO5LGRkEhcpVzd3Pj/0UF8ceBnQho28Jxf8vmHvz1gV608NLpVlc7lwZFRkYGERERDB48mOTk5IvqeXl5jBw5kvDwcOLj46moqADgxIkTPPDAAwwfPpyHHnqI/HwddSHiSh7ubtwT0oHEx253hIVPi2YKCQFcGBQWi4V58+axfPly0tPTSUlJ4fDhw9XmmTRpElOnTmXt2rXY7XZWrlwJwPz587n33ntJT0/n7rvvZt68ea5qU0REauGyoMjOziY0NBRfX1+8vb0JDw8nMzPTUc/Pz6esrIzg4GAAoqKiHPWqqirOnTsHQGlpKc2bN3dVmyIiUguXbVdarVb8/f0d02azmd27d9dY9/f3x2KxAPDUU08xZswYli1bhs1mIyUlxVVtiohILVwWFJcaQurng4sZ1f/6178yY8YMBg0axNq1a5kwYQKrV692enAyozFLRMSYu7ub49bfv1UDdyONgcuCIiAggB07djimrVYrZrO5Wr2wsNAxXVBQgNlspqioiKNHjzJo0CAAwsPDeeGFFyguLuaaa65xat0aFFDkt6usrHLcFhScbeBupD402KCA/fr1Iycnh6KiIkpLS1m3bh1hYWGOemBgIF5eXuTm5gKQlpZGWFgYbdu2xcvLyxEyubm5tGzZ0umQEBGRuuXSLYq4uDiio6Ox2WyMGjWKHj16EBMTQ2xsLEFBQSQlJZGQkEBJSQldu3YlOjoak8nEwoULefHFFykrK6Nly5a8/vrrrmpTRERqoetRiEg1kxfnYCkuJaBtCxIfu72h25F6oOtRiIjIZVFQiIiIIQWFiIgYUlCIiIghBYWIiBhSUIiIiCEFhYiIGFJQiIiIIQWFiIgYUlCIiIghBYWIiBhSUIiIiCEFhYhU09zTo9qtiIJCRKoZ0b8jt3TwZUT/jg3dijQSGmZcROQqp2HGRUTksigoRETEkIJCREQMKShERMSQgkJERAwpKERExJBLz6jJyMhg0aJF2Gw2xo4dywMPPFCtnpeXR0JCAufOnaNPnz5Mnz6dM2fO8Oc//9kxz9mzZykuLmbnzp2ubFVERGrgsi0Ki8XCvHnzWL58Oenp6aSkpHD48OFq80yaNImpU6eydu1a7HY7K1euxM/Pj/T0dNLT0/nwww8JDAxkxowZrmpTRERq4bKgyM7OJjQ0FF9fX7y9vQkPDyczM9NRz8/Pp6ysjODgYACioqKq1QFWrVpFixYtGDp0qKvaFBGRWrgsKKxWK/7+/o5ps9mMxWKpse7v71+tXllZyaJFi/jLX/7iqhZFRMQJLvuN4lIjg5hMJqfrWVlZdOzYkc6dO//qdRudii4iIr+Oy4IiICCAHTt2OKatVitms7lavbCw0DFdUFBQrb5+/XoiIiJ+07o11pOIiPMabKynfv36kZOTQ1FREaWlpaxbt46wsDBHPTAwEC8vL3JzcwFIS0urVv/666/p06ePq9oTEREnuSwoAgICiIuLIzo6mhEjRhAZGUmPHj2IiYlhz549ACQlJZGYmMiQIUMoLS0lOjrasfzx48dp3769q9oTEREnaZhxEZGrnIYZFxGRy6KgEBERQwoKERExpKAQERFDCgoRETGkoBAREUMKChERMaSgEBERQwoKERExZDgo4C233FJtRNdfysvLq/OGRESkcTEMipycHOx2O/PnzycwMJD77rsPd3d3UlNT+f777+urRxERaUCGu57atm3LNddcw969exk3bhxt2rTBx8eH6Ohovvzyy/rqUUREGpBTv1GUlpZy9OhRx/SBAwew2Wwua0pERBoPpy5c9PTTT3PffffRuXNnqqqqOHLkCElJSa7uTUREGgGnhxk/deoUubm5mEwmevfuzTXXXOPq3n4zDTMuIuK82oYZN9yiSE9PZ/jw4fzjH/+odv+JEycAeOSRR+qgRRERacwMg+K7774D4ODBg/XSjIiIND66wp2IyFXusnY9/WTnzp0sWbKEH3/8EbvdTlVVFSdOnGDz5s111aeIiDRSTh0em5CQwK233sq5c+cYOnQoPj4+3H333a7uTUREGgGntihMJhPjxo2juLiY3//+9wwbNow//elPru5NREQaAae2KFq2bAlAhw4dOHToEF5eXlRWVta6XEZGBhEREQwePJjk5OSL6nl5eYwcOZLw8HDi4+OpqKgAwGq1Mm7cOEaMGMGYMWMcR1mJiEj9cyoogoKCePrppwkNDeWdd95h9uzZuLu7Gy5jsViYN28ey5cvJz09nZSUFA4fPlxtnkmTJjF16lTWrl2L3W5n5cqVADz77LMMHDiQtLQ0hg8frpP7REQakFNBER8fz9ixY+nYsSPx8fHY7fZaP7yzs7MJDQ3F19cXb29vwsPDyczMdNTz8/MpKysjODgYgKioKDIzMykqKmL//v2MGTMGgJEjR/L000//tmcnIiKXzamgsFqtpKWlAXDddddx/PhxWrVqVesy/v7+jmmz2YzFYqmx7u/vj8Vi4fjx41x77bXMmjWLYcOGERsbS7NmzX7NcxIRkTrk1I/Zf/3rX7nzzjsBCAwMpG/fvkyZMoU333yzxmUudXrGz69tUVO9oqKCffv28eSTTxIfH8/777/Pc889x7Jly5xpFcDweGAREfl1nAqK4uJioqOjAfDy8mLs2LGOLYyaBAQEsGPHDse01WrFbDZXqxcWFjqmCwoKMJvN+Pv707JlSwYOHAhAZGQkM2fOdPoJgU64ExH5NWo74c6pXU+VlZXVdhsVFhZecovg5/r160dOTg5FRUWUlpaybt06wsLCHPXAwEC8vLzIzc0FIC0tjbCwMDp06EBAQABbtmwBYNOmTXTr1s2ZNkVExAWcGsLjgw8+4NVXX6V///6YTCays7N59tlnGTp0qOFyGRkZLF68GJvNxqhRo4iJiSEmJobY2FiCgoLYv38/CQkJlJSU0LVrVxITE/H09OTo0aO88MILFBcX4+Pjw+zZs7nhhhucflLaohARcV5tWxROj/W0f/9+tm7diru7OyEhIdx888111mRda+xBsetwIZnb/sU9IR3o2aldQ7cjIle5y9r1dOTIEQC++eYbKisrue222+jVqxc2m41vvvmmbju9iqRlfcuB46dJy/q2oVsREamV4Y/ZL7/8MkuWLOHJJ5+8qGYymdiwYYPLGmvKysorqt2KiDRmhkGxZMkSADZu3FgvzYiISONjGBQLFy40XHjChAl12oyIiDQ+hkFRXFwMwNGjR/n2228ZNGgQHh4ebNiwgc6dO9dLgyIi0rAMg2Lq1KkAREdHk5qayjXXXAPA448/zhNPPOH67pqg7/59lnOlNgDKyiupqKzCw92p01lERBqEU59QBQUFjpAAaN26NadOnXJZU01RRWUVSzK+Yfq72ykpu/Aj9pmScuLf3MrJUyUN3J2ISM2cCorOnTszefJktm7dSk5ODhMnTqRnz56u7q1J+fCzo2z9xnLR/QWny3jt/V1UVFY1QFciIrVz6oS7c+fOsWDBAnJycjCZTPTv358nn3yS5s2b10ePv1pjO+HufHklcQs/p6y85os9PT6iO7fdYq6xLiLiKrWdcOfUoIA+Pj4888wzHDt2jJtvvpny8vJGGxKN0fenSgxDAuBI/hkFhYg0Sk7tevr6668ZNGgQ48ePx2q1MmDAAL766itX99ZkeDYzvhqgs/OIiDQEp4Jizpw5vPvuu/j6+tK+fXvmzJnDSy+95Oremoxr/bxpf4234Ty9b/Y3rIuINBSngqKsrIxOnTo5pgcMGEBlpfGuFPkPk8nE6DtuxFRDPaRrANe3N75ioIhIQ3EqKDw8PDhz5ozjCnVHjx51aVNN0a03+zMhKqjaloUJuCekA4/e26XhGhMRqYVTQTF+/HgefPBBTp48yTPPPMOf/vQnHn/8cVf31uTcerM/L8WE4NfaCwB/3xb8cWAnnXAnIo2aU0c93XnnnbRq1Yr9+/fj5ubGY489hru7fnz9LUwmkyMYTDXtixIRaUScCorExESSk5Px8fnPcbYmk4mcnByXNSYiIo2DU0Hx6aefkpWVRdu2bV3dj4iINDJO7Ry/4YYbaN26tat7ERFpUnYdLuTl5K/YdbiwoVu5LE5tUTz00EM8+OCDhISE4OHxn0V0PQoRkZqlZX3Ld5azlJVX0rNTu4Zu5zdzKihef/11/Pz8OHv2rKv7uSo09/SodisiTVNTueyxU59UpaWlvPnmm7/6wTMyMli0aBE2m42xY8fywAMPVKvn5eWRkJDAuXPn6NOnD9OnT8fDw4O0tDSSkpLw8/MD4I477iAuLu5Xr7+xGtG/I2u//BfhfTs0dCsiIrVyKihuuukm9u/fzy233OL0A1ssFubNm0dqaiqenp6MGTOGkJCQamd4T5o0iZkzZxIcHMyUKVNYuXIl999/P3v27OG5554jMjLy1z+jK0DPTu2u6M1QEbm6OPVjttVqZdSoUYSHhzN06FDHf0ays7MJDQ3F19cXb29vwsPDyczMdNTz8/MpKysjODgYgKioKEd9z549pKWlMWzYMCZOnMiZM2d+49MTEZHL5dQWxTPPPPOrH9hqteLv/5+B7sxmM7t3766x7u/vj8Vicfz/uHHj6NGjB3PnzmXGjBm8+uqrv7oHERG5fE4FRd++fX/1A1/qekimn52KbFT/29/+5rjvf/7nfxg0aNCvWrfRBThEROqL+/+NwuDu7oa//5U78KfLDrsJCAhgx44djmmr1YrZbK5WLyz8z7HFBQUFmM1mzp49y6pVqxg7dixwIVB+fkiuMxrbFe5E5OpU+X+XOK6srKKgoPEeNVrbFe5cNhpdv379yMnJoaioiNLSUtatW0dYWJijHhgYiJeXF7m5uQCkpaURFhaGt7c3b731Frt27QLgvffeY/Dgwa5qU0REauHSLYq4uDiio6Ox2WyMGjWKHj16EBMTQ2xsLEFBQSQlJZGQkEBJSQldu3YlOjoad3d3XnvtNaZNm0ZZWRk33HADc+bMcVWbIiJSC5P9Uj8WXOG060lEGoPJi3OwFJcS0LYFiY/d3tDt1KjBdj2JiEjToKAQERFDCgoRETGkoBAREUMKChERMaSgEBERQwoKERExpKAQERFDCgoRETGkoBAREUMKChERMaSgEBGpYz+WVfBR9jFO/VAGwOlz5/nmWFEDd/XbaVBAEZE6dPbHcmYnf8XJUz9eVBt9x40MCb2+AboypkEBRUTq0aotRy4ZEgDvbz7C94Ul9dzR5VNQiIjUkXJbJVu/sRjO8/nuk/XUTd1RUIiI1JFzpTbKK6oM5yk6W1ZP3dQdBYWISB3xadEMTw/jj9VrWjWvp27qjoJCRKSOeDZzJ6RrgOE8f+jxu3rqpu4oKERE6tCoO27kd37eNdaubdeynju6fDo8VkSkjv1YVsGGr06w+otvqay049nMjQlRQXTv6NfQrV2SDo8VEaln3s09GNrvBtq1vvB7RFsfr0YbEs5waVBkZGQQERHB4MGDSU5Ovqiel5fHyJEjCQ8PJz4+noqKimr1ffv20b17d1e2KCIitXBZUFgsFubNm8fy5ctJT08nJSWFw4cPV5tn0qRJTJ06lbVr12K321m5cqWjVlpayowZM7DZbK5qUUREnOCyoMjOziY0NBRfX1+8vb0JDw8nMzPTUc/Pz6esrIzg4GAAoqKiqtVnz57N2LFjXdWeiIg4yWVBYbVa8ff3d0ybzWYsFkuNdX9/f0d9w4YNlJWVcc8997iqPRERcZKHqx74UgdTmUymWusFBQUsWrSId9999zev2+jXexGR+uLu7ua49fdv1cDd/HYuC4qAgAB27NjhmLZarZjN5mr1wsJCx3RBQQFms5nNmzdz+vRpHnjgAUdt+PDhJCcn4+PjXADo8FgRaQwqK6sctwUFZxu4m5o12OGx/fr1Iycnh6KiIkpLS1m3bh1hYWGOemBgIF5eXuTm5gKQlpZGWFgYo0ePZv369aSnp5Oeng5Aenq60yEhIiJ1y2VBERAQQFxcHNHR0YwYMYLIyEh69OhBTEwMe/bsASApKYnExESGDBlCaWkp0dHRrmpHRER+I52ZLSLiIpMX52ApLiWgbQsSH7u9odupkc7MFhGRy6KgEBERQwoKERExpKAQERFDCgoRETGkoBAREUMKChERMaSgEBERQwoKERExpKAQEXGR5p4e1W6vVAoKEREXGdG/I7d08GVE/44N3cpl0VhPIiJXOY31JCIil0VBISIihhQUIiJiSEEhIiKGFBQiImJIQSEiIoYUFCIiYkhBISIihhQUIiJiyKVBkZGRQUREBIMHDyY5Ofmiel5eHiNHjiQ8PJz4+HgqKioA2LFjB1FRUQwdOpTx48dz5swZV7YpIiIGXBYUFouFefPmsXz5ctLT00lJSeHw4cPV5pk0aRJTp05l7dq12O12Vq5cCcDkyZOZM2cOGRkZdOrUibfffttVbYqISC1cFhTZ2dmEhobi6+uLt7c34eHhZGZmOur5+fmUlZURHBwMQFRUlKP+8ccf06lTJ2w2GxaLhdatW7uqTRERqYXLgsJqteLv7++YNpvNWCyWGuv+/v6OerNmzThw4AADBgxg27Zt3Hvvva5qU0REauGyQdIvNSityWRyut65c2eys7NZsWIFcXFxrFixwul1G42CKCIiv47LgiIgIIAdO3Y4pq1WK2azuVq9sLDQMV1QUIDZbOb8+fNkZWUxaNAgAIYNG8bLL7/8q9atYcZFRJzXYMOM9+vXj5ycHIqKiigtLWXdunWEhYU56oGBgXh5eZGbmwtAWloaYWFheHh4MH36dPbu3QvAJ598Qq9evVzVpoiI1MKlFy7KyMhg8eLF2Gw2Ro0aRUxMDDExMcTGxhIUFMT+/ftJSEigpKSErl27kpiYiKenJzt27GDWrFlUVlYSEBDAjBkzaN++vdPr1RaFiIjzatui0BXuRESucrUFxZV9xW+56uUXlnAk/wwe7ia6d/SjdUvPhm5JpMlRUMgV6ccyG0sy9rH7yCnHfW4mCA/pwMgBN+L2syPoROTyaKwnueLY7Xb+9uHeaiEBUGWHT7b+i4+yjzVMYyJNlIJCrjhH8n8g77viGutrvzzOeVtlPXYk0rQpKOSKs++7IsN66fkKvvv32XrqRqTpU1DIFUe/PojULwWFXHG6dfQzrLds7sEN7VvVUzciTZ+CQq44HX/XiqDf1xwWEaHX49nMvR47EmnaFBRyxTGZTIwf3o3bbjFftBtqxB86ck9IhwbpS6Sp0pnZckUrOF3Kxq9OsOdoEcP+6wb6dglo6JZErjgawkNERAw12OixIiLSNCgoRETEkIJCREQMKShERMSQgkJERAwpKERExFCTvB6Fm5tGAxIRcVZtn5lN8jwKERGpO9r1JCIihhQUIiJiSEEhIiKGFBQiImJIQSEiIoYUFCIiYkhBISIihhQUIiJiSEEhIiKGroqgOHjwIJ07d2bt2rWO+7Zs2cLAgQP5y1/+ctH8w4cPd0kfsbGxDB061CWPDbBnzx7i4+Nd9vi/1blz55g+fTqRkZEMHz6chx56iG+++abB+nHVv29jVVJSwvTp0xk8eDDDhg3j/vvvJycnB4CUlBQ++ugjAJ577jlSU1MbstUGdeLECe68886L7u/cuXONy2zbto2HHnrIlW01Ck1yrKdfSk1NJTw8nBUrVhAeHg5AZmYm48eP57777rto/vT09Drvobi4mH379tGuXTtyc3Pp3bt3na8jKCiIoKCgOn/cy1FVVUVMTAwhISGkpaXh4eHB1q1biYmJYc2aNbRt27bee3LFv29jZbfbGT9+PF26dGHNmjV4enqyb98+xo0bx6uvvsrOnTvp27dvQ7cpjVyTD4qKigpWr15NcnIyY8aM4V//+hfbtm1jw4YN5OTk4ObmxurVq2nTpg2HDh3itddeY8SIERw4cIDTp08THx/P0aNH8fT05LnnnuP222/nvffeIz09ndLSUkwmE6+99ho33ngjd955J8OGDePzzz+ntLSUl19+me7duwOQkZFBnz59uPnmm0lJSXEERWpqKps3b8ZqtfLvf/+bhx9+mO+//56tW7fi6+vLW2+9hZeXF2lpaSxdupSqqiq6devGCy+8gJeXF6GhoXTr1o3CwkKeffZZ/v73v7Ns2TLy8vJ4/vnnKSsro02bNiQlJdGuXTumTZvGoUOHKCwspGPHjixcuJDCwkImTJjATTfdRF5eHn5+fsyfPx9fX9/L/vtv27YNq9VKbGwsbm4XNmBDQ0NJTEykqqqKhISES/YTHR3Nxo0bAXj99dcBePLJJ8nIyGDRokWYTCaCgoJ48cUXKSoqYsqUKZw9e5aCggLuvfdeJk6cyP79+3n++eepqKjAy8uLxMREbrjhBjp37syBAwewWCyXXC41NZWsrCzOnDnD8ePH+a//+i+mTZt22X+LhvDll1/y/fff889//hOT6cLAb127duXxxx9n7NixtGrViq1bt+Lv7w/A5s2bWb58OadOnXJ8kSopKWHGjBkcOnSIyspKYmJiiIyMJDU1lQ8//JDTp08zcOBAnnnmmYZ8qi7lzGti6dKlrF+/niVLljBu3DiCgoLIzc2lqKiIhIQEBgwYQGFhIfHx8Xz//fd4eHgQFxdH9+7dHZ8bAP3792fy5MlERESwZMkSTCYTZWVlWCwWvvvuO/Lz8xk9ejSPP/54vT3/Jr/rafPmzVx77bV07NiRQYMGsWLFCkaPHs2dd95JbGwso0ePBnDsmurSpYtj2fnz59OhQwc++eQT5syZw2uvvca5c+dYv349y5Yt46OPPmLQoEEsX77csYyvry8ffPABY8aMYfHixY77U1NTGTJkCEOGDGHt2rWcPn3aUduzZw9vvfUWycnJzJ49m7CwMDIyMgDIysri0KFDrFy5khUrVpCeno6fnx9vv/02cGFLZdy4caSnp+Ph8Z/cnzhxIk888QQZGRlERESwdOlSdu7cSbNmzUhJSeHTTz/l/PnzbNmyBYD9+/fzyCOP8NFHH9G6dWvH+i/Xvn37CAoKcoTETwYMGMDRo0dr7OdSLBYLiYmJvPPOO6xZs4bKykq2bNnCRx99RGRkJCtXrmT16tUsX76coqIili5dyiOPPEJqaioPPfQQX3/9dbXHq2k5gJ07d7JgwQJWr17Npk2bOHDgQJ38Perbnj176N69uyMkfnLbbbfRokULx/ugf//+AJSXl/P++++zePFi5s2bB8CiRYvo1q0bqampJCcn8/e//53jx48DF/5NPvzwwyYdEj8xek2sWrWKdevWsXjxYlq0aAGAzWYjJSWFyZMnM3/+fABefPFFQkNDycjIYMGCBUyZMoWqqiquvfZaDh48yJEjR6isrOTLL78E4LPPPmPgwIEAHDhwgLfffpv333+fJUuW8MMPP9Tbc2/yWxSpqalERkYCEBERwcSJE3n66acvmq9Hjx4X3bd9+3aSkpKAC0GSkpICwKuvvsqaNWs4duwYWVlZ1cLlpzfcTTfdxLp16wDIy8vj5MmT9OvXj2bNmtGlSxfS0tIYO3YsAL169cLHxwcfHx8Abr/9dgACAwP54Ycf2LZtG9999x1//OMfgQsvwK5duzrW2bNnz2p9FxUVUVBQ4HiB3X///Y6ar68vycnJHD16lGPHjvHjjz8C4Ofn53jMm266iTNnzhj/YZ3k5uZGTQMU33bbbTX2cyk7d+6kV69etG/fHoBXXnnFUdu6dStvv/02hw4dwmazUVpayoABA5gxYwZZWVkMHDjQsdvxJ48++ugllwO49dZbHf8e1113XZ39PeqbyWSisrLyovttNtsl57/rrrswmUzcdNNNFBcXA5CdnU1ZWRmrVq0C4Mcff+TQoUPAha2Tn39BuZL98ssMXNh191PI1vSaOHjwIM8//zxz587F29vbsezPPwt++mK4detWZs6c6XiMnj17smvXLgYMGEBOTg4eHh5ER0ezZs0ax5Zup06dAAgJCcHT0xM/Pz98fX05e/YsrVu3ds0f4xeaxr9wDU6dOsVnn33G3r17+ec//4ndbueHH35wfID/XPPmzS+675dvgCNHjtC8eXMefvhhHnzwQcLCwmjXrh15eXmOeby8vACqfYNbtWoV5eXljg+qkpISVqxY4QiKZs2aGa63srKSIUOGkJCQ4Fj+52/+X/b+y8c7f/48VquVgwcPsmDBAqKjo4mKiqK4uNjxIf5T3z/1Xlejz3fv3p3ly5dXe8MBzJ07lx49evD6669f1M8v119RUYGHh8dFf5efvv0vWbKE48ePExkZyaBBg8jOzsZut3PPPfdw6623smnTJpYuXcqWLVscb1KA2bNnX3I5V/496lvPnj1ZtmwZNput2uvi66+/vuTvWe7u7kD1129VVRWvvPIK3bp1A6CwsJA2bdqQkZFxyffNlap169acPXu22n2nTp2iTZs2QM2viZYtWzJr1ixmzZpF//79HWFxqc+CX76O7HY7lZWVDBgwgIULF+Lp6clTTz3FJ598QkZGhiNsjNZfH5r0rqfVq1cTGhrKZ599xsaNG9m0aRPjx493bBnUpk+fPnz88cfAhZCIiYlh7969XH/99YwdO5aePXvy2WefXfIb20/Ky8vJyMjg3XffZePGjWzcuJENGzZQUFDAtm3bnOojJCSETz/9lFOnTmG325k2bRpLly6tcf5WrVrRvn17vvjiC+DCj7fz588nJyeHIUOGMHLkSNq1a8f27dsNe68Lffr0wc/Pj4ULFzrWlZWV5djne6l+WrduzZkzZygqKqK8vJysrCzgwo/1u3btoqCgAIBZs2axYcMGvvjiCx599FGGDBnCyZMnsVgsVFVV8fTTT7N7927GjBnDU089xb59+6r1VtNyTUmfPn3o1KkTs2bNcmxF7N27l0WLFvHEE0/g7u5e62sgNDSU//3f/wXAarUybNgwTp486fLe65uPjw/XX399taMjU1JSHFv4NQkMDOSuu+6ib9++LFiwwHDe0NBQPvjgAwCOHz/OV199RXBwMN26dePbb7/l2LFj3HjjjYSEhLBo0SLuuOOOy35edaFJb1GkpqYSFxdX7b7777+ft956i3bt2tW6fGxsLAkJCQwbNgwPDw/mzJlDly5dWLFiBREREXh6etKjRw/HZvilbNq0icDAwGq7h3x8fBg9ejQrVqyo9o2hJrfccgsTJkzg4Ycfpqqqii5dujBu3DjDZV555RWmTZvGnDlzaNu2LXPmzKG4uJiJEyeSmZmJp6cnwcHBnDhxotb1Xw6TycQbb7xBYmIikZGReHh40LZtW5YsWYK7u/sl+2nVqhWPPvooo0aNon379o5vvgEBAcTHx/Poo49SVVVFcHAwUVFRtGjRgmeffZbWrVvj5+dH9+7dOXHiBOPHjyc+Pp433ngDd3d3nnvuuWq9PfbYY5dcrqlZuHAh8+bNIzIyEnd3d9q0acMrr7xCSEgIhYWFzJ07l1atWtW4/IQJE5g2bRqRkZFUVlYyadIkOnTowI4dO+rxWdSPn943f/vb37DZbHTu3Jnnn3+ezZs317rss88+S2RkpOEh8PHx8Tz//POOw5BnzpyJ2WwGoHfv3o5dn6Ghobz//vuN5og0XeFOREQMNeldTyIicvkUFCIiYkhBISIihhQUIiJiSEEhIiKGFBQiImJIQSEiIoaa9Al3IvVh27ZtvPTSS3h7e1NSUkKvXr3Yt28fJSUl2O12Zs6cSe/evSkpKWHmzJl89dVXuLu7M2jQIOLi4rDZbCQlJTnOTO/atSsJCQmOcYVEGpqCQqQOHDp0iPXr12O1WvnHP/5BSkoKbm5uLFmyhDfffJPevXuzYMECzp8/z8cff0xlZSV//vOf+fLLL9m+fTvu7u6kpqZiMpmYO3cuSUlJV+zQ5tL0KChE6sDvfvc7AgMDCQwMpE2bNqxYsYLjx4+zbds2WrZsCVwYhXXy5Mm4u7vj7u7Oe++9B1wYNuLs2bNkZ2cDF0Z29fPza7DnIvJLCgqROvDTiKGbN2/mpZde4pFHHuGuu+7i97//PatXrwYujAr885FET548SfPmzamqqmLKlCkMGDAAuDA68Pnz5+v/SYjUQD9mi9ShL774goEDB3L//fcTFBTE+vXrHaOz3n777Xz44YdUVVVRXl5ObGws27dv5w9/+APJycmUl5dTVVXF1KlTmTt3bgM/E5H/UFCI1KExY8awfft2hg4dyn333cd1113HiRMnqKqqYsKECTRr1ozhw4czYsQIBgwYwN13380TTzxBYGAg//3f/01ERAR2u/2ikW5FGpJGjxUREUPaohAREUMKChERMaSgEBERQwoKERExpKAQERFDCgoRETGkoBAREUMKChERMfT/AVE8tWcMcS4kAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 285 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "MFIAEkEBAc9P" - }, - "outputs": [], - "source": [ - "# Fit the new instance to the balanced training dataset\r\n", - "expgrad_exercise.fit(________, _________, sensitive_features=________)" - ] + "id": "dR3eNMeY0HTi", + "outputId": "aaa8498f-1f82-4003-c73c-ab6eaa3c2476" + } + }, + { + "cell_type": "markdown", + "source": [ + "From our analysis, we see that paying with *Medicaid* does appear to have some relationship with the patient's race. *Caucasian* patients are the least likely to pay with *Medicaid* compared with other groups. If paying with *Medicaid* is a proxy for socioeconomic status, then the patterns we find align with our understanding of wealth and race in the United States." + ], + "metadata": { + "id": "MTcAD0oxbarW" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Additional validity checks" + ], + "metadata": { + "id": "9BLSciTKaBLd" + } + }, + { + "cell_type": "markdown", + "source": [ + "Similarly as we used predictive validity to check that our label aligns with the construct of \"likely to benefit from the care management program\", we can use predictive validity to verify that our various features are coherent with each other.\n", + "\n", + "For example, based on the eligibility criteria for *Medicaid* vs *Medicare*, we expect `medicaid` to be negatively correlated with age and `medicare` to be positively correlated with age:" + ], + "metadata": { + "id": "-PzKOA59ezFs" + } + }, + { + "cell_type": "code", + "execution_count": 30, + "source": [ + "sns.pointplot(y=\"medicaid\", x=\"age\", data=df, ci=95, join=False);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEJCAYAAACKWmBmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnC0lEQVR4nO3de1xUdf4/8NcAggEJXmbGIi1bb3nF1a8iJoSAKIKDgJqoqKuktoqXldREK/O22kaWj0xty/SLG5bGRG2AkvprBVOpRENMtFRQZ0BQAQeZy+f3h79m44ceRuTIxdfzHz3ncz7nvIc58JrzOZdRCCEEiIiI7sGuoQsgIqLGjUFBRESSGBRERCSJQUFERJIYFEREJIlBQUREkhgUREQkyaGhC5BDaWkFLBbeHkJEZAs7OwVat3a5Z3uzDAqLRTAoiIjqCYeeiIhIEoOCiIgkMSiIiEgSg4KIiCTJGhQpKSkIDg5GYGAgEhMT77nc4sWLsXfv3hrzc3Nz0atXLzlLJCKiWsgWFDqdDgkJCdi1axe0Wi2SkpKQn59fY5lZs2YhNTW1Rn+DwYCVK1fCaDTKVSIREdlAtqDIzMyEl5cX3N3d4ezsjKCgoBqBkJKSAn9/f4wcObJG/3Xr1mHq1Klyldckncgvxt8Tf8CJ/OKGLoWIHiGy3Ueh1+uhVCqt0yqVCjk5OdWWmTFjBgAgOzu72vyMjAxUVlZixIgRddp227auderX2H31v9k4V3ADJiEQMLhTQ5dDRI8I2YLibl+cp1Aoau1XVFSEzZs3Y/v27XXe9rVr5c3yhrvyiirrv0VFZQ1cDRE1F3Z2CskP2LINPanVahQX/3eIRK/XQ6VS1drv4MGDuH79OiZOnAiNRgMA0Gg0KC8vl6tUIiKSIFtQeHt7IysrCyUlJTAYDEhPT4ePj0+t/caOHYv9+/dDq9VCq9UCALRaLVxdm+dwEhFRYyfrEcWCBQsQHR2NsLAwhISEoE+fPoiJicHJkyfl2iwREdUzhbjbyYQmrrmeo1i6JQu6UgPUrR/D2pmDG7ocImomGuwcBRERNQ8MCiIiksSgICIiSQwKIiKSxKAgIiJJDAoiIpLEoCAiIkkMCiIiksSgICIiSQyKJqQZ3mxORE0Ag6IJyLtQijX/m42i6wYAQPGNShz8sfCuj3InIqpvsn0fBdWPnHPX8O7nJ6odTZgtAjvSzqCkrBLhPn9quOKI6JHAI4pGzCIE/pVx9p5DTl9nXUDJzcqHWxQRPXIYFI3YJV05dCW37tkuBHAsT/8QKyKiRxGDohG7VWmsdRnDbdNDqISIHmUMikbsyXYusKvla8Y7qPjNf0QkLwZFI+bm6oSBz6nv2d62VUv07dzuIVZERI8iBkUjN2l4V/zpyVY15ru5OGJuRG842PMtJCJ58atQmwCzxYKfzl7DP7/ORWWVGY87t8C6mYPxmBOvbiaiB8evQm0G7O3s0L+bEm4ujgAAZycHhgQRPTQMCiIiksSgICIiSQwKIiKSJGtQpKSkIDg4GIGBgUhMTLzncosXL8bevXut09nZ2YiIiIBGo8GUKVNQWFgoZ5lERCRBtqDQ6XRISEjArl27oNVqkZSUhPz8/BrLzJo1C6mpqdXmx8XFYfXq1dBqtQgNDcWqVavkKpOIiGohW1BkZmbCy8sL7u7ucHZ2RlBQUI1ASElJgb+/P0aOHGmdV1VVhXnz5qF79+4AgG7duuHKlStylUlERLWQ7RpLvV4PpVJpnVapVMjJyam2zIwZMwDcGWr6naOjIzQaDQDAYrFg06ZNCAgIuK9tS10P3JTZ/7+b6+zt7aBUPt7A1RDRo0K2oLjbfXwKRS0PLvqDqqoqLFmyBCaTCTNnzryvbTe3G+5+ZzZbrP8WFZU1cDVE1Fw02A13arUaxcXF1mm9Xg+VSmVT34qKCsyYMQMmkwmbN29GixYt5CqTiIhqIVtQeHt7IysrCyUlJTAYDEhPT4ePj49NfePi4vD0009j48aNcHR0lKtEIiKygWxDT2q1GgsWLEB0dDSMRiMiIyPRp08fxMTEIDY2Fr17975rv9zcXGRkZKBz584ICwsDcOf8xrZt2+QqlYiIJPChgE3I0i1Z0JUaoG79GNbOHNzQ5RBRM8GHAhIR0QNhUBARkSQGRRPS0tGh2r9ERA8Dg6IJCRvaCd07uiNsaKeGLoWIHiE8mU1E9IjjyWwiInogDAoiIpLEoCAiIkkMCiIiksSgICIiSQwKIiKSxKAgIiJJDAoiIpLEoCAiIkkMCiIiksSgICIiSQwKIiKSxKAgIiJJDAoiIpLEoCAiIkkMCiIiksSgICIiSQwKIiKSJGtQpKSkIDg4GIGBgUhMTLzncosXL8bevXut05cvX8bEiRMxYsQIzJ49GxUVFXKWSUREEmQLCp1Oh4SEBOzatQtarRZJSUnIz8+vscysWbOQmppabf4bb7yBqKgopKamolevXnj//fflKpOIiGohW1BkZmbCy8sL7u7ucHZ2RlBQUI1ASElJgb+/P0aOHGmdZzQacezYMQQFBQEAwsPDa/QjIqKHx0GuFev1eiiVSuu0SqVCTk5OtWVmzJgBAMjOzrbOKy0thaurKxwc7pSmVCqh0+nkKpOIiGohW1AIIWrMUygUsvX7o7ZtXe9reSIiujfZgkKtVuP48ePWab1eD5VKVWu/Nm3aoLy8HGazGfb29igqKrKp3x9du1YOi6Vm4BARUU12dgrJD9iynaPw9vZGVlYWSkpKYDAYkJ6eDh8fn1r7tWjRAgMGDMC///1vAEBycrJN/YiISB6yBYVarcaCBQsQHR2NsLAwhISEoE+fPoiJicHJkycl+7722mvYvXs3goODcfz4ccyfP1+uMomIqBYKcbeTAk0ch56IiGzXYENPRETUPDAoiIhIEoOCiIgkMSiIiEgSg4KIiCQxKIiISBKDgoiIJDEoiIhIkuSznrp37y75QL7Tp0/Xe0FERNS4SAZFVlYWhBDYuHEjPDw8MH78eNjb22Pv3r24fPnyw6qRiIgakOTQU+vWrdGmTRucOnUKL730Etzc3ODq6oro6GgcPXr0YdVIREQNyKZzFAaDAefPn7dOnzlzBkajUbaiiIio8bDp+yjmz5+P8ePHo1u3brBYLDh37hzeeustuWsjIqJGwOanx167dg3Z2dlQKBTo378/2rRpI3dtdcanxxIR2a62p8dKHlFotVpoNBp8/PHH1eYXFBQAAKZNm1YPJRIRUWMmGRQXLlwAAPzyyy8PpRgiImp8+MVFRESPuAcaevrdjz/+iK1bt+LWrVsQQsBisaCgoAAHDx6srzqJiKiRsuny2Pj4ePTr1w/l5eUIDQ2Fq6srhg8fLndtRETUCNh0RKFQKPDSSy+htLQUzz77LEaPHo0JEybIXRsRETUCNh1RuLi4AAA6duyIs2fPwsnJCWazWdbCiIiocbDpiKJ3796YP38+5s2bh5kzZ+K3336Dvb293LUREVEjYNNVT0IInDhxAp6enjh06BAyMzMxfvx4PPvssw+jxvvGq56IiGxX21VPNg096fV6JCcnAwA6dOiAS5cu4fHHH6+1X0pKCoKDgxEYGIjExMQa7adPn0ZERASCgoKwbNkymEwmAHdu6Js4cSI0Gg0mT56MwsJCW8okIiIZ2BQUixcvth49eHh4YODAgXj11Vcl++h0OiQkJGDXrl3QarVISkpCfn5+tWXi4uKwfPlypKWlQQiB3bt3AwA2btyIUaNGQavVYvjw4UhISKjLayMionpgU1CUlpYiOjoaAODk5ISpU6eiqKhIsk9mZia8vLzg7u4OZ2dnBAUFITU11dpeWFiIyspKeHp6AgDCw8Ot7RaLBeXl5QDuPLm2ZcuW9/3CiIiofth0MttsNkOn00GtVgMAiouLUdupDb1eD6VSaZ1WqVTIycm5Z7tSqYROpwMAzJs3Dy+++CJ27twJo9GIpKQk218RERHVK5uCYurUqQgLC8PQoUOhUCiQmZmJV155RbLP3YLkj1+rKtW+ePFirFy5EgEBAUhLS8OcOXPw5ZdfSn4t6x9JnZQhIqL7Y1NQREZGolevXjhy5Ajs7e0xffp0dO3aVbKPWq3G8ePHrdN6vR4qlapae3FxsXW6qKgIKpUKJSUlOH/+PAICAgAAQUFBeO2111BaWmrzo8151RMRke0e6Kqnc+fOAQB+/vlnmM1m/M///A/+/Oc/w2g04ueff5bcsLe3N7KyslBSUgKDwYD09HT4+PhY2z08PODk5ITs7GwAQHJyMnx8fNC6dWs4OTlZQyY7OxsuLi6N+vsviIiaM8n7KF566SVs3boVw4YNq9lRoUBGRobkylNSUrBlyxYYjUZERkYiJiYGMTExiI2NRe/evZGXl4f4+HhUVFSgR48eWLt2LRwdHZGTk4M333wTlZWVcHFxwYoVK9CjRw+bXxSPKIiIbFfbEQUfM05E9Ih7oMeMb9q0SXLlc+bMqVtVRETUZEgGRWlpKQDg/Pnz+PXXXxEQEAAHBwdkZGSgW7duD6VAIiJqWDYNPUVHR+Odd96xnlC+ceMGXn755bs+lqMx4NATEZHt6uVZT0VFRdWuOmrVqhWuXbv24NUREVGjZ9N9FN26dcPSpUuh0WgghMDnn3+Ovn37yl0bERE1AjYNPZWXl+Pdd99FVlYWFAoFhg4dirlz5zbaZzBx6ImIyHb1dnlsZWUlfvvtN3Tt2hVVVVWNNiQABgUR0f2ol3MUP/30EwICAjBr1izo9Xr4+vrihx9+qLciiYio8bIpKNavX4/t27fD3d0d7du3x/r167F69Wq5ayMiokbApqCorKxE586drdO+vr4wm82yFUVERI2HTUHh4OCAGzduWB/zff78eVmLIiKixsOmy2NnzZqFSZMmoaioCAsXLsThw4excuVKuWsjIqJGwOarno4dO4a8vDzY2dlhwIABsLe3rzYc1ZjwqiciIts90EMBf7d27VokJibC1fW/K1IoFMjKynrwComIqFGzKSj27duH7777Dq1bt5a7HiIiamRsOpn9zDPPoFWrVnLXQkREjZBNRxSTJ0/GpEmTMGjQIDg4/LcLv4+CyDZGkxn/OXkFmSevIsT7GfTt3K6hSyKymU1B8d5776Ft27YoKyuTux6iZsVoMuOL//MrDv1UCEPVnXuPPtD+jFei+qHTEzxKp6bBpqueRo4ciW+++eZh1FMveNUTNQYWIfDu5znIOVfzkfwtHOywdNKf8Ux7hgU1vHp51lOXLl2Ql5dXb0URPQpyfyu5a0gAgNFkwZ5DvHGVmgabhp70ej0iIyPh4eEBR0dH6/yUlBTZCiNq6o7n6SXbf/61BLcqTXBuadOvIVGDsWkPXbhwodx1EDU7lVW1Pw/tttHMoKBGz6Y9dODAgXLXQdTsPN3+cRw9fe+jCndXR7i5ON6znaixsOkcRV2lpKQgODgYgYGBSExMrNF++vRpREREICgoCMuWLYPJZAJwZ6jrpZdeQlhYGF588UUUFBTIWSaRLJ7v/QQec7K/Z3vAgA6ws1M8xIqI6ka2oNDpdEhISMCuXbug1WqRlJSE/Pz8asvExcVh+fLlSEtLgxACu3fvBgC88sor8PPzQ3JyMjQaDd566y25yiSSzePOjoiN6ANnp5oH7kN6t8eIgR0boCqi+ydbUGRmZsLLywvu7u5wdnZGUFAQUlNTre2FhYWorKyEp6cnACA8PBypqakoKSlBXl4eXnzxRQBAREQE5s+fL1eZRLLq1rE11s8ejImBXfGY452jizatnDB9VA8eTVCTIVtQ6PV6KJVK67RKpYJOp7tnu1KphE6nw6VLl/Dkk09izZo1GD16NGJjY9GiRQu5yiSSnXPLFvDv/xRUrZ0BAI8/xvMS1LTIdrnF3e7j+/2Lj6TaTSYTcnNzMXfuXCxbtgyfffYZlixZgp07d9q8bakbR4gaypSQHvji4DmMeeFPUCofb+hyiGwmW1Co1WocP37cOq3X66FSqaq1FxcXW6eLioqgUqmgVCrh4uICPz8/AEBISAhWrVp1X9vmndnUGD2jdMGCsX0AAEVFfBwONR71cmd2XXh7eyMrKwslJSUwGAxIT0+Hj4+Ptd3DwwNOTk7Izs4GACQnJ8PHxwcdO3aEWq3GoUOHAAAHDhxAz5495SqTiIhqYfM33NVFSkoKtmzZAqPRiMjISMTExCAmJgaxsbHo3bs38vLyEB8fj4qKCvTo0QNr166Fo6Mjzp8/j9deew2lpaVwdXXFunXr8Mwzz9i8XR5REBHZrrYjClmDoqEwKIiIbNdgQ09ERNQ8MCiIiEgSg4KIiCQxKIiISBKDgoiIJDEoiIhIEoOCiIgkMSiIiEgSg4KIiCQxKIiISBKDgoiIJDEoiIhIEoOCiIgkMSiIiEgSg4KIiCQxKIiISBKDgoiIJDEoiIhIEoOCiIgkMSiIiEgSg4KIiCQxKIiISBKDgoiIJMkaFCkpKQgODkZgYCASExNrtJ8+fRoREREICgrCsmXLYDKZqrXn5uaiV69ecpZIRES1kC0odDodEhISsGvXLmi1WiQlJSE/P7/aMnFxcVi+fDnS0tIghMDu3butbQaDAStXroTRaJSrRCIisoFsQZGZmQkvLy+4u7vD2dkZQUFBSE1NtbYXFhaisrISnp6eAIDw8PBq7evWrcPUqVPlKo+IiGwkW1Do9XoolUrrtEqlgk6nu2e7Uqm0tmdkZKCyshIjRoyQqzwiIrKRg1wrFkLUmKdQKGptLyoqwubNm7F9+/Y6b7ttW9c69yUioupkCwq1Wo3jx49bp/V6PVQqVbX24uJi63RRURFUKhUOHjyI69evY+LEidY2jUaDxMREuLraFgDXrpXDYqkZREREVJOdnULyA7ZsQ0/e3t7IyspCSUkJDAYD0tPT4ePjY2338PCAk5MTsrOzAQDJycnw8fHB2LFjsX//fmi1Wmi1WgCAVqu1OSSIiKh+yRYUarUaCxYsQHR0NMLCwhASEoI+ffogJiYGJ0+eBAC89dZbWLt2LUaOHAmDwYDo6Gi5yiEiojpSiLudLGjiOPRERGS7Bht6IiKi5oFBQUREkhgUREQkiUFBRESSGBRERCSJQUFERJIYFEREJIlBQUREkhgUREQkiUFBRESSGBRERCSJQUFERJIYFEREJIlBQUREkhgUREQkiUFBRESSGBRERCSJQUFERJIYFEREJIlBQUREkhgUREQkiUFBRESSGBRERCSJQUFERJJkDYqUlBQEBwcjMDAQiYmJNdpPnz6NiIgIBAUFYdmyZTCZTACA7OxsREREQKPRYMqUKSgsLJSzTCIikiBbUOh0OiQkJGDXrl3QarVISkpCfn5+tWXi4uKwfPlypKWlQQiB3bt3W+evXr0aWq0WoaGhWLVqlVxlEhFJulF+GxnZBfjyP7/ieJ4eJrOloUt66GQLiszMTHh5ecHd3R3Ozs4ICgpCamqqtb2wsBCVlZXw9PQEAISHhyM1NRVVVVWYN28eunfvDgDo1q0brly5IleZRET39M2RC1j0fiYS9/2C5P/8iveTT2HxB1n49crNhi7toXKQa8V6vR5KpdI6rVKpkJOTc892pVIJnU4HR0dHaDQaAIDFYsGmTZsQEBBwX9tu29b1AasnokfdoR8K8NnBczXml5bdxjuf5WDz4mFwc3VqgMoePtmCQghRY55CobC5vaqqCkuWLIHJZMLMmTPva9vXrpXDYqm5fiIiWyXtO3PPtrJbVdAeOIuRXk8/xIrkY2enkPyALdvQk1qtRnFxsXVar9dDpVLds72oqMjaXlFRgRkzZsBkMmHz5s1o0aKFXGUSEdVQWWXCJX255DJnC248pGoanmxB4e3tjaysLJSUlMBgMCA9PR0+Pj7Wdg8PDzg5OSE7OxsAkJycbG2Pi4vD008/jY0bN8LR0VGuEomI7sreTgFFbcvY17ZE8yHb0JNarcaCBQsQHR0No9GIyMhI9OnTBzExMYiNjUXv3r3x1ltvIT4+HhUVFejRoweio6ORm5uLjIwMdO7cGWFhYQDunN/Ytm2bXKUSEVXTwsEePTq1wc+/ltxzGc/O7R5iRQ1LIe52sqCJ4zkKInpQ+YU38PfEH2C+y9+Sp5QuWD5lAFo42DdAZfWvwc5REBE1ZZ093LBgXF+0b+NsnacA0K9LOyya0K/ZhIQteERBRCRBCIELujKUG4x4sq0L2rRq2dAl1bvajihkO0dBRNQcKBQKPNO+VUOX0aA49ERERJIYFEREJIlBQUREkhgUREQkqVmezLaze3TumCQielC1/c1slpfHEhFR/eHQExERSWJQEBGRJAYFERFJYlAQEZEkBgUREUliUBARkSQGBRERSWJQEBGRJAYFERFJeuSCYuPGjQgODsaoUaPw8ccfW+dnZmYiNDQUw4cPR0JCQgNWSA/Tg+4P3377LcLDwzFixAisWrXqvvuTbSoqKvDGG28gMDAQo0ePRlRUFLKysmTZ1u7duzFmzBgEBQVh27Zt1vkpKSkIDg5GYGAgEhMTZdl2oyUeId9//7148cUXhdFoFAaDQfj5+Ylz584Jg8EgfH19xcWLF4XRaBR/+ctfxMGDBxu6XJLZg+4PFy9eFM8//7y4cuWKqKqqEhMmTBAHDx7k/lTPLBaLmDRpkli9erW4ffu2EEKIn3/+WQwZMkQcOXKkXrd17NgxERQUJG7evCnKy8tFYGCgOHv2rLh69arw8/MTpaWloqKiQoSGhoqzZ8/W67Ybs2b5UMB7GThwIHbs2AEHBwfodDqYzWY4OzsjJycHTz/9NDp06AAACA0NRWpqKnx9fa19P/vsMxw5cgT/+Mc/AACbNm2Co6MjJk6ciJUrV+Ls2bMwm82IiYlBSEgIysvL8eqrr0Kn00Gv12PAgAFYv349jh49ig0bNsBisaBLly4ICwvDhg0bAABubm74xz/+gTZt2lSre8+ePfj444+hUCjQs2dPLF++HC4uLvDy8kLPnj1RXFyMzz//HC1atAAAxMXFYcCAARg/fjwAYPLkyVi0aBFatWqFFStW4Pr163B2dsayZcvQp08fLFmyBAMHDkR4eDgAoFu3bjhz5gzee+896HQ6XLhwAYWFhRg7dixmz54No9GI1157DdnZ2VCr1VAoFHj55ZcxaNAgbN26Fd988w3MZjOef/55xMXFobCwEDNmzEDr1q3h5OSE7du3y/o+2+pB9gcA2LdvH4KDg9G+fXsAQEJCApycnBr9/tTUHD16FJcvX8aOHTugUNx5eF2PHj0we/ZsvP/++3Bzc8OiRYvw1VdfAQAOHDiApKQkfPDBB/e9P37zzTeIiorC448/DgD46KOP4O7ujn379sHLywvu7u4AgKCgIKSmpmLOnDnWvu+88w4sFgsWLlwIAFi6dCmGDh2KgQMHYsWKFbh69SoUCgX+9re/wdvbGzqdDq+++irKyspQVFSEUaNGYdGiRdi7dy+++OILXL9+HX5+fujSpQs+/PBD2Nvb46mnnsKGDRvg5OT0EH7yf9DQSdUQNm7cKPr27SsWL14sLBaLSElJEX/729+s7YcPHxbTpk2r1qe8vFwMHjxYlJeXC4vFIoYPHy6uXr0qNmzYID755BMhhBBlZWVi1KhR4uLFiyIlJUW8//77Qgghbt++LQICAsTJkyfFkSNHRP/+/cXNmzeFEEJMmjRJnDhxQgghxCeffCK+++67atvNy8sTAQEBoqSkRAghxOuvvy7WrVsnhBCia9eud/1ElZWVJaKiooQQQhQUFIjg4GAhhBAREREiLS1NCCHEjz/+KF544QVx+/ZtsXjxYrFnzx5r/65duwohhHj33XdFZGSkuH37tiguLhaenp7ixo0bYseOHWL+/PnCYrGIgoIC0a9fP3HkyBFx6NAhMXfuXGEymYTZbBYLFy4UycnJ4tKlS6Jr167i0qVL9/1ePQx12R+EEGLFihXizTffFH/5y19EaGioePvttxv9/tQUbdu2TcTGxtaYf+bMGdGvXz8hhBChoaHizJkzQgghFi5cKL7++us67Y/Tp08XCQkJYuLEiWL06NFix44dQgghPvjgA/H2229bl9u9e7eIj4+v1vfixYvCz89PWCwWUVFRIXx9fcXt27fF/Pnzxf79+4UQQuh0OuHv7y/KysrEhx9+KPbu3SuEEOLmzZuiX79+4tq1a2LPnj0iMDBQGI1GIYQQw4YNE8XFxUIIId5++22Rm5v7QD/Punikjih+Fxsbi5iYGMyaNQu7d++Gs7NzjWV+/+TyOxcXF/j6+iI9PR0dOnRAhw4doFarkZmZicrKSuzZswcAcOvWLZw9exYhISHIycnB9u3bcf78eVy/fh23bt0CAHTq1Mn6icXf3x9z5sxBQEAA/P39MWTIkGrbPXbsGPz8/NC6dWsAwPjx47F06VJre9++fWvUPmjQICxfvhwFBQXQarXQaDSoqKjAxYsXMXz4cACAp6cn3NzccP78ecmf1aBBg+Do6Ii2bdvC3d0dZWVlOHz4MMaNGweFQgEPDw8MHjwYAJCVlYWcnBzrkUllZSWefPJJ9O/fH23btsVTTz0lua2GUpf9AQDMZjOOHz+OnTt3wtnZGS+//DK++OIL65GdVP+G2p+aIoVCAbPZXGO+0Wi0/l+j0eDrr79Ghw4dcPToUaxZswbvvPPOfe+PZrMZP/zwA7Zs2QKTyYRJkyahS5cuEHd5yPb//5526NABHh4eOHbsGC5fvgxfX184OjoiMzMT58+fx7vvvgsAMJlMuHTpEqZPn44jR47gn//8J86ePQuj0QiDwQDgzhGTg8OdP89+fn6YMGEC/P39ERQUhOeee66OP8m6e6SC4ty5c6iqqsJzzz2Hxx57DMOHD8eZM2cwYsQIFBcXW5fT6/VQqVQ1+kdERGDz5s146qmnrDufxWLBhg0b0LNnTwBAcXEx3NzcsHPnTqSlpWHcuHHw9vbGL7/8Yt3ZWrZsaV3n1KlT4efnhwMHDmDDhg3IycnB7Nmzre0Wi6VaDUIImEwm6/Qf1/U7hUKBsLAwfP3110hNTcWHH34IIUSNnV0IAbPZDIVCYW374y8fgGqHuL8vZ29vX6Mu4M4v2ZQpUzBt2jQAwM2bN2Fvb4/S0tK71tnQ7nd/yMjIsP6yDxs2DO3atcPgwYOtQzv+/v7IyclBcHBwo92fmqK+ffti586dMBqN1UL4p59+Qu/evQEAISEhmDJlCrp3747nn38eTk5Oddof27Vrh549e8LFxQUAMHToUJw8eRJqtRrHjx+3Lif1nn711Ve4fPky5s6dC+DOe/rJJ59Yh610Oh3atWuHdevW4dKlSwgJCUFAQAAyMzPv+p7Gx8cjLy8Phw4dQlxcHObMmQONRlPXH2edPFJXPRUUFCA+Ph5VVVWoqqpCRkYG+vfvj759++LXX3/FhQsXYDab8dVXX8HHx6dG/wEDBuDq1av4/vvvERAQAADw8vLCv/71LwB3dp7Ro0fjypUrOHz4MMaPH4/Ro0dDoVAgLy/vrn9cx44di4qKCkydOhVTp05Fbm5utfaBAwfi22+/xfXr1wHcuSJj0KBBtb7W8PBwfPrpp2jfvj3UajVcXV3RoUMHpKenA7jzS1ZcXIwuXbrA3d0d+fn5AID9+/fXum5vb2/8+9//hhACOp0OR48ehUKhgJeXF7RaLSoqKmAymfDXv/4VaWlpta6vodzv/uDv7w+tVgutVot58+bBz88P//nPf3Dz5k2YzWZ899136NmzZ6Pen5qiAQMGoHPnzlizZo31g8ypU6ewefNmvPzyywAAtVqNJ554Alu3bsXo0aMBoE77o5+fH/bt24eqqipUVlbiyJEj6NWrF7y9vZGVlYWSkhIYDAakp6ff9T0dMWIEsrKyUFxcbD3a9/Lywq5duwAA+fn5GD16NAwGAw4fPozp06dj5MiRuHLlCnQ6XY331GQyYfjw4WjdujVmzpwJjUaD06dPP9gPtA4eqSMKX19fnDhxAmFhYbC3t8fw4cMxatQoAMC6deswd+5c3L59G76+vhgxYsRd1xEYGIjr16/D0dERADBnzhy8/vrrCAkJgdlsRlxcHDp27IgpU6bg9ddfx0cffQQXFxf069cPBQUF6NixY7X1LVy4EEuWLIGDgwOcnJzwxhtvVGvv3r07Zs6cicmTJ8NoNKJnz541lrmbJ554Ak888QTGjBljnbdhwwa8/vrreO+999CiRQu89957cHR0RFRUFObPn4/Q0FB4eXlBqVRKrnvcuHHIy8tDaGgolEolnnzySbRs2RIDBw5EXl4exo0bB7PZjKFDh2LMmDEoLCystd6G8KD7Q9++fTFjxgxERUXBaDRiyJAhiIiIgJ2dXaPdn5qqTZs2ISEhASEhIbC3t4ebmxs2bNhQ7UOTRqNBQkKCdd6wYcPue38MDg7GxYsXMWbMGJhMJmg0GuvQ6oIFCxAdHQ2j0YjIyEj06dOnRv+WLVvC09MTXbt2tc6Lj4/HihUrEBoaCgBYv349XF1dMXPmTLzyyito1aoV2rZti169eqGgoKDa+hwcHBAbG4tp06ahZcuWaNWqFf7+97/X/QdZVw/9rEgTZbFYxO3bt0VUVJQ4depUQ5cjyWKxiKtXr4rAwEDr5YT16cCBA+Lbb78VQtw5CTds2DBRWlpa79tpzprS/kS2sVgsoqysTAQHBwu9Xt/Q5dSrR2ro6UEUFRVhyJAh6Nu3r3X8uLFKS0uDRqPBwoULrZ9U69Of/vQnbN26FRqNBpMmTUJsbKx1/JVs05T2J7LNyZMnMWzYMIwbN67Wo/Kmht+ZTUREknhEQUREkhgUREQkiUFBRESSGBRERCSJQUFERJIeqRvuiORisViwZs0anDhxAhUVFRBCYNWqVejUqROWLl2Kixcvwt3dHUqlEl26dMHcuXNx7tw5rF69GtevX4fZbMbkyZMRGRnZ0C+FqAYGBVE9OHHiBPR6PZKSkmBnZ4etW7di27ZtcHZ2RufOnbFlyxbo9XqEh4ejS5cuMJlMiI2Nxfr169GzZ0+UlZVh/Pjx6Ny5Mzw9PRv65RBVw6Agqgf9+vWDm5sbPv30U1y6dAnff/89XFxccOzYMXzxxRcAAJVKZX2Ux2+//YaLFy/i1Vdfta6jsrISubm5DApqdBgURPXg4MGDWL16NaZNmwZ/f388++yz+PLLL+Hg4FDtqb12dndOC5rNZrRq1QpardbaVlxcbH1cOFFjwpPZRPXg8OHD8PPzQ1RUFHr37o39+/fDbDbD19cXn3/+OQCgtLQU+/fvh0KhQKdOneDk5GQNiitXriAkJASnTp1qyJdBdFd8hAdRPTh37hwWLVoEk8kEe3t7DBgwAOnp6dBqtYiPj7eezBZC4IUXXsCMGTOQl5dnPZltMpkQHR2NCRMmNPRLIaqBQUEko8TERPTo0QP9+vVDVVUVoqKiMHfu3Brfv03UmPEcBZGMOnfujDfffBMWiwVGoxEjRoxgSFCTwyMKIiKSxJPZREQkiUFBRESSGBRERCSJQUFERJIYFEREJIlBQUREkv4v77zd1b8cYI0AAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 285 }, - { - "cell_type": "markdown", - "metadata": { - "id": "8H_UDlAs3M-D" - }, - "source": [ - "2.) Now, let's compute the performance of the `ExponentiatedGradient` model and compare it with the performance of `ExponentiatedGradient` model with logistic regression as base estimator" - ] + "id": "869rzUjSUe3C", + "outputId": "bd4de952-204d-4159-addc-19b9d1b75167" + } + }, + { + "cell_type": "code", + "execution_count": 31, + "source": [ + "sns.pointplot(y=\"medicare\", x=\"age\", data=df, ci=95, join=False);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEJCAYAAACKWmBmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAowklEQVR4nO3de1xUdf4/8NcAgo6AeJkZTW2t8JIXEGOFEGERRhCB8UJeUElTKvkqiWGauYYXuugKupat5HczC7/ZhsJSinhJ1xU0aUsow/VSeUFmQEABuQwzn98fPprfsuBhVAZQX8+/POdzPue8B4685nzOTSaEECAiIroDq7YugIiI2jcGBRERSWJQEBGRJAYFERFJYlAQEZEkBgUREUliUBARkSSbti7AEsrKqmA08vYQIiJzWFnJ0LVr5zu2P5RBYTQKBgURUQvh0BMREUliUBARkSQGBRERSWJQEBGRpIfyZDYRUUuo0xvwj9OFOHlGi8qaejzWXQ7/Z/pgcL9ubV1aq5I9jI8Zv369klc9EdF9qa6tx4Zd3+Ni4c1GbRN9nkSoV7/WL8pCrKxk6N7d/s7trVgLEdEDI+P4L02GBADs+cdF/FpU0coVtR0GBRHRfzEKgWN5hZLL/OO0dPvDhEFBRPRfamoNqKqpl1zm+s2aVqqm7TEoiIj+S0dba3SytZZcxsnerpWqaXsMCiKi/2JlJYPXsF6Sy3i7SLc/TBgURERN0Hg/gd6Kph+UFzTycTj37tLKFbUdXh5LRHQHt2rqcTD3Mk6c0aKyWo8+is4YM6IPnhmogEwma+vyWkxzl8cyKIiIHnG8j4KIiO4Lg4KIiCRZNCgyMjIQHBwMtVqNlJSUOy535MgRjBkzxjR98+ZNvPjiixg3bhxmzJiB4uJiS5ZJREQSLBYUWq0WSUlJ2LlzJ9LT07Fr1y6cP3++0XIlJSV49913G8zbuHEj3N3dsW/fPjz33HNISEiwVJlERNQMiwVFdnY2PD094eTkBLlcjsDAQGRmZjZabsWKFViwYEGDeUeOHEFoaCgAICQkBP/4xz+g1+stVSoREUmwWFDodDooFArTtFKphFarbbDMjh07MHjwYLi6ut6xr42NDezt7VFaWmqpUomISILF3kfR1FW3/3nd8b///W9kZWVh+/btKCoqanZ9VlbmZ5rUZV5ERHR3LBYUKpUKubm5pmmdTgelUmmazszMRHFxMSZPngy9Xg+dToeIiAjs3LkTSqUSJSUl6NmzJ+rr61FZWQknJyezt837KIiIzNdm91F4eXkhJycHpaWlqK6uRlZWFnx8fEztMTEx2L9/P9LT05GcnAylUomdO3cCAHx9fZGWlgYA2Lt3L9zd3dGhQwdLlUpERBIsFhQqlQqxsbGIjIzEhAkTEBISAhcXF0RFRSE/P1+y7yuvvILvv/8e48ePx86dO7Fy5UpLlUlERM3gIzyIiB5xfIQHERHdFwYFERFJYlAQEZEkBgUREUliUBARkSQGBRERSWJQEBGRJAYFERFJYlAQEZEkBgUREUliUBARkSQGBRERSWJQEBGRJAYFERFJYlAQEZEkiwZFRkYGgoODoVarkZKS0qj9wIEDCA0Nxfjx47Fs2TLU1dUBANLS0uDt7Q2NRgONRoOkpCRLlklERBIs9s5srVaLpKQk7N69G7a2tpg2bRo8PDzg7OwMALh16xZWr16NPXv2oEePHoiNjcWePXswdepU5OfnY9myZQgJCbFUeUREZCaLHVFkZ2fD09MTTk5OkMvlCAwMRGZmpqldLpfj8OHD6NGjB27duoXr16/D0dERAJCfn4+0tDSEhYUhLi4ON27csFSZRETUDIsFhU6ng0KhME0rlUpotdoGy3To0AFHjx6Fn58fysrK4O3tDQBQKBRYuHAh0tPT0atXL6xevdpSZRIRUTMsNvTU1Ku4ZTJZo3m+vr44efIkEhMTER8fjw0bNuD99983tc+bNw8BAQF3tW2pd78SEdHdsVhQqFQq5ObmmqZ1Oh2USqVpury8HD/88IPpKCI0NBSxsbGoqKhAamoqZs+eDeB24NjY3F2Z169XwmhsHFRERNSYlZVM8gu2xYaevLy8kJOTg9LSUlRXVyMrKws+Pj6mdiEElixZgsLCQgDAvn37MGLECMjlcmzbtg2nT58GAHz66adQq9WWKpOIiJohE02NEbWQjIwMbN26FXq9HuHh4YiKikJUVBRiYmIwbNgwHDx4EJs2bYJMJoOzszNWrVoFBwcH5ObmIiEhATU1NejXrx/WrVsHBwcHs7fLIwoiIvM1d0Rh0aBoKwwKIiLztdnQExERPRwYFEREJIlBQUREkhgUREQkiUFBRESSGBRERCSJQUFERJIYFEREJIlBQUREkhgUREQkiUFBRESSGBRERCSJQUFERJIYFEREJIlBQUREkhgUREQkyaJBkZGRgeDgYKjVaqSkpDRqP3DgAEJDQzF+/HgsW7YMdXV1AIDCwkLMmDEDQUFBmD9/PqqqqixZJhERSbBYUGi1WiQlJWHnzp1IT0/Hrl27cP78eVP7rVu3sHr1anz00Uf46quvUFtbiz179gAAVq1ahYiICGRmZmLo0KHYsmWLpcokIqJmWCwosrOz4enpCScnJ8jlcgQGBiIzM9PULpfLcfjwYfTo0QO3bt3C9evX4ejoCL1ej1OnTiEwMBAAMGnSpAb9iIioddlYasU6nQ4KhcI0rVQqkZeX12CZDh064OjRo3jttdegVCrh7e2NsrIy2Nvbw8bmdmkKhQJarfauti317lciIro7FgsKIUSjeTKZrNE8X19fnDx5EomJiYiPj8drr71mVj8p169XwmhsvH0iImrMykom+QXbYkNPKpUKJSUlpmmdTgelUmmaLi8vxz//+U/TdGhoKM6ePYtu3bqhsrISBoMBAFBcXNygHxERtS6LBYWXlxdycnJQWlqK6upqZGVlwcfHx9QuhMCSJUtQWFgIANi3bx9GjBiBDh06wN3dHXv37gUApKWlNehHREStSyaaGiNqIRkZGdi6dSv0ej3Cw8MRFRWFqKgoxMTEYNiwYTh48CA2bdoEmUwGZ2dnrFq1Cg4ODrh69SqWLVuG69evo1evXkhMTESXLl3M3i6HnoiIzNfc0JNFg6KtMCiIiMzXZucoiIjo4cCgICIiSQwKIiKSxKAgIiJJDAoiIpLEoCAiIkkMCiIiksSgICIiSQwKIiKSxKAgIiJJZgVFVVUVVq1aheeffx7l5eVYuXIlX09KRPSIMCso1q5dC0dHR1y/fh12dnaorKzEypUrLV0bERG1A2YFxU8//YTY2FjY2NigU6dO+NOf/oSffvrJ0rUREVE7YFZQWFk1XMxgMDSaR0REDyezXoX6+9//HuvXr0dNTQ2OHTuGlJQUeHh4WLo2IiJqB8w6LIiLi4NcLoeDgwOSkpIwcODAJt9t/d8yMjIQHBwMtVqNlJSURu0HDx6ERqNBWFgYoqOjcePGDQC332rn7e0NjUYDjUaDpKSku/xYRETUUsx6cdGGDRvw6quv3tWKtVotpk+fjt27d8PW1hbTpk1DYmIinJ2dAQCVlZUICgpCamoqVCoVNm3ahIqKCqxYsQJr1qyBm5sbQkJC7ulD8cVFRETma5EXFx05cuSuN5ydnQ1PT084OTlBLpcjMDAQmZmZpna9Xo/4+HioVCoAwMCBA3Ht2jUAQH5+PtLS0hAWFoa4uDjTkQYREbU+s85R9OnTBy+88AJGjBiBzp07m+bPmTPnjn10Oh0UCoVpWqlUIi8vzzTdtWtXBAQEAABqamqQnJyMWbNmAQAUCgVefPFFuLi4IDExEatXr8aGDRvu7pMREVGLMCsonJycAABXr141e8VNjWjJZLJG8yoqKhAdHY1BgwZh4sSJAID333/f1D5v3jxToJhL6hCKiIjujllB8fbbb9/1ilUqFXJzc03TOp0OSqWywTI6nQ5z586Fp6cnli9fDuB2cKSmpmL27NkAbgeOjY1ZZZrwHAURkfmaO0dh1l/g7777DsnJybh16xaEEDAajbhy5YrkuQsvLy9s3rwZpaWl6NSpE7KysrBmzRpTu8FgwMsvv4xx48YhOjraNF8ul2Pbtm1wc3ODq6srPv30U6jVanPKJCIiCzArKFasWAGNRoP9+/dj2rRpOHToEMaOHSvZR6VSITY2FpGRkdDr9QgPD4eLiwuioqIQExODoqIinDlzBgaDAfv37wcADB06FAkJCdi4cSPi4+NRU1ODfv36Yd26dff/SYmI6J6YdXlsSEgIvvzyS7z77rsICAjA0KFDTZe+tkcceiIiMl+LXB7725VOjz/+OM6dOwc7OzsYDIaWqZCIiNo1s4aehg0bhkWLFuGVV17BSy+9hF9++QXW1taWro2IiNoBs4aehBA4ffo0hg8fjqNHjyI7OxtTp07Fk08+2Ro13jUOPRERma9Fhp50Oh3S0tIAAH379sXly5fh4ODQIgUSEVH7ZlZQLF261HT00Lt3b4wcOdJ03wMRET3czAqKsrIyREZGAgDs7Owwe/ZsFBcXW7QwIiJqH8wKCoPBAK1Wa5ouKSlp8hEdRET08DHrqqfZs2djwoQJGD16NGQyGbKzs816HwURET34zLrqCQAKCgpw4sQJWFtbw8PDAwMGDLB0bfeMVz0REZmvuaueJIPiwoULeOqpp/Djjz822T5kyJD7r9ACGBREROa7r6B48cUXkZycjDFjxjTuKJPh0KFDLVNlC2NQEBGZ776C4kHFoCAiMt99PWb8vffek1z5ggUL7q0qIiJ6YEgGRVlZGQDg4sWL+PnnnxEQEAAbGxscOnQIAwcObJUCiYiobZk19BQZGYmNGzeiW7duAIAbN24gOjoaKSkpFi/wXnDoiYjIfC3yrKfi4mJTSACAo6Mjrl+/3my/jIwMBAcHQ61WNxkqBw8ehEajQVhYGKKjo3Hjxg0AQGFhIWbMmIGgoCDMnz8fVVVV5pRJREQWYFZQDBw4EK+//jpOnDiBnJwcxMXFwdXVVbKPVqtFUlISdu7cifT0dOzatQvnz583tVdWViI+Ph7Jycn4+9//joEDB2Lz5s0AgFWrViEiIgKZmZkYOnQotmzZch8fkYiI7odZQbF27Vo4ODggISEBb7/9Nnr27IlVq1ZJ9snOzoanpyecnJwgl8sRGBiIzMxMU7ter0d8fDxUKhWA22F07do16PV6nDp1CoGBgQCASZMmNehHRESty6xHeNjb22Px4sX45ZdfMGDAANTV1aFjx46SfXQ6HRQKhWlaqVQiLy/PNN21a1cEBAQAAGpqapCcnIxZs2ahrKwM9vb2sLG5XZpCoWjwnCkiImpdZgXF999/jwULFsDGxgafffYZNBoNPvjgA4wYMeKOfZo6Ry6TyRrNq6ioQHR0NAYNGoSJEyc2GQpN9ZMidVKGiIjujllBsW7dOmzfvh1xcXHo2bMn1q1bh4SEBKSmpt6xj0qlQm5urmlap9NBqVQ2WEan02Hu3Lnw9PQ0vd+iW7duqKyshMFggLW1NYqLixv1aw6veiIiMl+LXPVUU1MDZ2dn07Svry8MBoNkHy8vL+Tk5KC0tBTV1dXIysqCj4+Pqd1gMODll1/GuHHj8MYbb5iOGjp06AB3d3fs3bsXAJCWltagHxERtS6zjihsbGxw48YN0x/zixcvNttHpVIhNjYWkZGR0Ov1CA8Ph4uLC6KiohATE4OioiKcOXMGBoMB+/fvBwAMHToUCQkJePPNN7Fs2TJ88MEH6NWrFxITE+/jIxIR0f0w64a7w4cPIykpCcXFxfDy8sLx48exevVq05VJ7Q2HnoiIzNdiDwU8deoUCgoKYGVlBXd3d1hbWzcYjmpPGBREROa7r4cC/ubtt99GSkoK7O3//4pkMhlycnLuv0IiImrXzAqKAwcO4NixY+jataul6yEionbGrKue+vXrB0dHR0vXQkRE7ZBZRxSzZs3CzJkz4eHhYbpjGuD7KIiIHgVmBcXmzZvRvXt3VFRUWLoeIiJqZ8wKiurqanz44YeWroWIiNohs85R9O/fHwUFBZauhYiI2iGzjih0Oh3Cw8PRu3dv2NramuZnZGRYrDAiImofzAqKxYsXW7oOIiJqp8y+M/tBwjuziYjM1yJPjyUiokcXg4KIiCQxKIiISBKDgoiIJDEoiIhIkkWDIiMjA8HBwVCr1UhJSbnjckuXLsXu3btN02lpafD29oZGo4FGo0FSUpIlyyQiIglm3UdxL7RaLZKSkrB7927Y2tpi2rRp8PDwaPCyI61WizfffBM5OTnw8PAwzc/Pz8eyZcsQEhJiqfKIiMhMFjuiyM7OhqenJ5ycnCCXyxEYGIjMzMwGy2RkZMDf3x/jxo1rMD8/Px9paWkICwtDXFwcbty4YakyiYioGRY7otDpdFAoFKZppVKJvLy8BsvMmzcPAPDtt982mK9QKPDiiy/CxcUFiYmJWL16NTZs2GD2tqVuHCEiortjsaBo6oZvmUxmVt/333/f9O958+YhICDgrrbNO7OJiMzXZndmq1QqlJSUmKZ1Oh2USmWz/SoqKrB9+3bTtBCiwcuSiIiodVksKLy8vJCTk4PS0lJUV1cjKysLPj4+zfaTy+XYtm0bTp8+DQD49NNPoVarLVUmERE1w2Jf1VUqFWJjYxEZGQm9Xo/w8HC4uLggKioKMTExGDZsWJP9rK2tsXHjRsTHx6Ompgb9+vXDunXrLFUmERE1g0+PJSJ6xPHpsUREdF8YFEREJIlBQUREkhgUREQkiUFBRESSGBRERCSJQUFERJIYFEREJIlBQUREkvi0PSILEkLgxI9aHPz2Mi7rKtHJzgYjn1Zh/LO/g5O9XVuXR2QWPsKDyII+O3QOWacuN5rf1cEOb8x6Bt0cO7ZBVUQN8REeRG3kl6KbTYYEAJRV1OKLIxdauSKie8OgILKQ7PwiyfZTBTrU1hlaqRqie8egILKQG1V1ku0Go0BVjb6VqiG6dwwKIgtRdesk2d7J1hqOnW1bqRqie2fRoMjIyEBwcDDUajVSUlLuuNzSpUuxe/du03RhYSFmzJiBoKAgzJ8/H1VVVZYsk8giRrs8BiuJ18SPcukFG2t+V6P2z2J7qVarRVJSEnbu3In09HTs2rUL58+fb7TMyy+/jMzMzAbzV61ahYiICGRmZmLo0KHYsmWLpcokshiFUydEBg1CU1nxVG9HTBz9ZKvXRHQvLBYU2dnZ8PT0hJOTE+RyOQIDAxsFQkZGBvz9/TFu3DjTPL1ej1OnTiEwMBAAMGnSpEb9iB4UPq6P4c05v4ePay880csBg/t1xexxg/Da9BHoZMfbmOjBYLE9VafTQaFQmKaVSiXy8vIaLDNv3jwAwLfffmuaV1ZWBnt7e9jY3C5NoVBAq9Vaqkwii3tc5YDZ455u6zKI7pnFgqKp+/hkMokB2/vs95+kbhwhIqK7Y7GgUKlUyM3NNU3rdDoolcpm+3Xr1g2VlZUwGAywtrZGcXGxWf3+E+/MJiIyX5vdme3l5YWcnByUlpaiuroaWVlZ8PHxabZfhw4d4O7ujr179wIA0tLSzOpHRESWYbGgUKlUiI2NRWRkJCZMmICQkBC4uLggKioK+fn5kn3ffPNNfP755wgODkZubi4WLVpkqTKJiKgZfCggEdEjjg8FJCKi+8KgICIiSQwKIiKSxKAgIiJJDAoiIpLEoCAiIkkMCiIiksSgICIiSQwKIiKSxKAgIiJJDAoiIpLEoCAiIkkMCiIiksSgICIiSQwKIiKSZNGgyMjIQHBwMNRqNVJSUhq1//TTT5g8eTICAwPxxhtvoL6+HsDtt9p5e3tDo9FAo9EgKSnJkmUSEZEEi724SKvVYvr06di9ezdsbW0xbdo0JCYmwtnZ2bRMSEgI1q5di+HDh2P58uUYOnQoIiIisGbNGri5uSEkJOSets0XFxERma/NXlyUnZ0NT09PODk5QS6XIzAwEJmZmab2q1evoqamBsOHDwcATJo0ydSen5+PtLQ0hIWFIS4uDjdu3LBUmURE1AyLBYVOp4NCoTBNK5VKaLXaO7YrFApTu0KhwMKFC5Geno5evXph9erVliqTiIiaYWOpFTc1oiWTycxqf//9903z5s2bh4CAgLvattQhFBER3R2LBYVKpUJubq5pWqfTQalUNmgvKSkxTRcXF0OpVKKiogKpqamYPXs2gNuBYmNzd2XyHAURkfna7ByFl5cXcnJyUFpaiurqamRlZcHHx8fU3rt3b9jZ2eHbb78FcPtKJx8fH8jlcmzbtg2nT58GAHz66adQq9WWKpOIiJphsauegNuXx27duhV6vR7h4eGIiopCVFQUYmJiMGzYMBQUFGDFihWoqqrC4MGD8fbbb8PW1ha5ublISEhATU0N+vXrh3Xr1sHBwcHs7fKIgojIfM0dUVg0KNoKg4KIyHxtNvREREQPBwYFERFJsthVT9Ry6g1G5J7V4ch3hSgsqYJb/x54zs8Z9p06tHVpRPQI4DmKdq6yWo8Nu77Hr0UVDebbd+qAxVNd0a+nYxtVRkQPC56jeMB9sv9so5AAbgfI5tR81BuMbVAVET1KGBTtWHllLXLP6u7YXlZRi+/OldyxnYioJTAo2rFrJVVobmDwiq6ydYohokcWg6Idk3ds/mR1Z57QJiILY1C0Y4+r7NGru/yO7VYy4PeDlHdsJyJqCQyKdkwmkyFCPQDWVrIm28NGPYGuDnatXBURPWp4eewD4NyVcqQd+xk//VoGAOjqYIeJo5/EqGE9Gzy6nYjoXvBZTw+Rmrp61BsEOne0YUAQUYtpLih4Z/YDpKMtf11E1Pp4joKIiCQxKIiISBKDgoiIJDEoiIhIEoOCiIgkPZSX0Vjd4QY1IiJqrLm/mQ/lfRRERNRyOPRERESSGBRERCSJQUFERJIYFEREJIlBQUREkhgUREQkiUFBRESSGBRERCSJQUFERJIeuaDYtGkTgoODMX78eHz00Uem+dnZ2QgNDcXYsWORlJTUhhVSa7rf/eHw4cOYNGkSgoKCsHbt2rvuT+apqqrCqlWroFarERYWhoiICOTk5FhkW59//jkmTpyIwMBAfPjhh6b5GRkZCA4OhlqtRkpKikW23W6JR8jJkyfFtGnThF6vF9XV1cLPz09cuHBBVFdXC19fX3Hp0iWh1+vFCy+8II4cOdLW5ZKF3e/+cOnSJeHt7S2uXbsm6urqxPTp08WRI0e4P7Uwo9EoZs6cKRISEkRtba0QQogff/xRjBo1Spw4caJFt3Xq1CkRGBgobt68KSorK4VarRbnzp0TRUVFws/PT5SVlYmqqioRGhoqzp0716Lbbs8eyocC3snIkSOxY8cO2NjYQKvVwmAwQC6XIy8vD7/73e/Qt29fAEBoaCgyMzPh6+tr6vu3v/0NJ06cwIYNGwAA7733HmxtbTFjxgysXr0a586dg8FgQFRUFEJCQlBZWYnly5dDq9VCp9PB3d0d69atwzfffIP169fDaDSif//+mDBhAtavXw8A6NKlCzZs2IBu3bo1qDs1NRUfffQRZDIZhgwZgj/+8Y/o3LkzPD09MWTIEJSUlOCLL75Ahw4dAABLliyBu7s7pk6dCgCYNWsW4uLi4OjoiJUrV6K8vBxyuRxvvPEGXFxcsGzZMowcORKTJk0CAAwcOBBnz57F5s2bodVq8euvv+Lq1at47rnnMH/+fOj1erz55pv49ttvoVKpIJPJEB0dDQ8PDyQnJ2Pfvn0wGAzw9vbGkiVLcPXqVcybNw9du3aFnZ0dtm/fbtHfs7nuZ38AgAMHDiA4OBg9e/YEACQlJcHOzq7d708Pmm+++QaFhYXYsWOH6V3xgwcPxvz587FlyxZ06dIFcXFx+PLLLwEAX3/9NXbt2oW//OUvd70/7tu3DxEREXBwcAAA/PWvf4WTkxMOHDgAT09PODk5AQACAwORmZmJBQsWmPpu3LgRRqMRixcvBgC8/vrrGD16NEaOHImVK1eiqKgIMpkMr776Kry8vKDVarF8+XJUVFSguLgY48ePR1xcHHbv3o09e/agvLwcfn5+6N+/P7Zt2wZra2v06dMH69evh52dXSv85P9DWydVW9i0aZNwdXUVS5cuFUajUWRkZIhXX33V1H78+HExZ86cBn0qKyvFs88+KyorK4XRaBRjx44VRUVFYv369eLjjz8WQghRUVEhxo8fLy5duiQyMjLEli1bhBBC1NbWioCAAJGfny9OnDghnnnmGXHz5k0hhBAzZ84Up0+fFkII8fHHH4tjx4412G5BQYEICAgQpaWlQggh4uPjxTvvvCOEEGLAgAFNfqPKyckRERERQgghrly5IoKDg4UQQkyePFns379fCCHEd999J/7whz+I2tpasXTpUpGammrqP2DAACGEEH/+859FeHi4qK2tFSUlJWL48OHixo0bYseOHWLRokXCaDSKK1euCDc3N3HixAlx9OhRsXDhQlFfXy8MBoNYvHixSEtLE5cvXxYDBgwQly9fvuvfVWu4l/1BCCFWrlwp1qxZI1544QURGhoqEhMT2/3+9CD68MMPRUxMTKP5Z8+eFW5ubkIIIUJDQ8XZs2eFEEIsXrxYfPXVV/e0P86dO1ckJSWJGTNmiLCwMLFjxw4hhBB/+ctfRGJiomm5zz//XKxYsaJB30uXLgk/Pz9hNBpFVVWV8PX1FbW1tWLRokXi4MGDQgghtFqt8Pf3FxUVFWLbtm1i9+7dQgghbt68Kdzc3MT169dFamqqUKvVQq/XCyGEGDNmjCgpKRFCCJGYmCjOnDlzXz/Pe/FIHVH8JiYmBlFRUXj55Zfx+eefQy6XN1rmt28uv+ncuTN8fX2RlZWFvn37om/fvlCpVMjOzkZNTQ1SU1MBALdu3cK5c+cQEhKCvLw8bN++HRcvXkR5eTlu3boFAHjiiSdM31j8/f2xYMECBAQEwN/fH6NGjWqw3VOnTsHPzw9du3YFAEydOhWvv/66qd3V1bVR7R4eHvjjH/+IK1euID09HRqNBlVVVbh06RLGjh0LABg+fDi6dOmCixcvSv6sPDw8YGtri+7du8PJyQkVFRU4fvw4pkyZAplMht69e+PZZ58FAOTk5CAvL890ZFJTU4PHHnsMzzzzDLp3744+ffpIbqut3Mv+AAAGgwG5ubn45JNPIJfLER0djT179piO7KT6t9X+9CCSyWQwGAyN5uv1etO/NRoNvvrqK/Tt2xfffPMN3nrrLWzcuPGu90eDwYB//etf2Lp1K+rr6zFz5kz0798foomHbP/377Rv377o3bs3Tp06hcLCQvj6+sLW1hbZ2dm4ePEi/vznPwMA6uvrcfnyZcydOxcnTpzA//7v/+LcuXPQ6/Worq4GcPuIycbm9p9nPz8/TJ8+Hf7+/ggMDMTTTz99jz/Je/dIBcWFCxdQV1eHp59+Gp06dcLYsWNx9uxZBAUFoaSkxLScTqeDUqls1H/y5Mn44IMP0KdPH9POZzQasX79egwZMgQAUFJSgi5duuCTTz7B/v37MWXKFHh5eeHf//63aWfr2LGjaZ2zZ8+Gn58fvv76a6xfvx55eXmYP3++qd1oNDaoQQiB+vp60/R/rus3MpkMEyZMwFdffYXMzExs27YNQohGO7sQAgaDATKZzNT2n//5ADQ4xP1tOWtr60Z1Abf/kz3//POYM2cOAODmzZuwtrZGWVlZk3W2tbvdHw4dOmT6zz5mzBj06NEDzz77rGlox9/fH3l5eQgODm63+9ODyNXVFZ988gn0en2DEP7+++8xbNgwAEBISAief/55DBo0CN7e3rCzs7un/bFHjx4YMmQIOnfuDAAYPXo08vPzoVKpkJuba1pO6nf65ZdforCwEAsXLgRw+3f68ccfm4attFotevTogXfeeQeXL19GSEgIAgICkJ2d3eTvdMWKFSgoKMDRo0exZMkSLFiwABqN5l5/nPfkkbrq6cqVK1ixYgXq6upQV1eHQ4cO4ZlnnoGrqyt+/vln/PrrrzAYDPjyyy/h4+PTqL+7uzuKiopw8uRJBAQEAAA8PT3xf//3fwBu7zxhYWG4du0ajh8/jqlTpyIsLAwymQwFBQVN/nF97rnnUFVVhdmzZ2P27Nk4c+ZMg/aRI0fi8OHDKC8vB3D7igwPD49mP+ukSZPw2WefoWfPnlCpVLC3t0ffvn2RlZUF4PZ/spKSEvTv3x9OTk44f/48AODgwYPNrtvLywt79+6FEAJarRbffPMNZDIZPD09kZ6ejqqqKtTX1+N//ud/sH///mbX11budn/w9/dHeno60tPT8corr8DPzw///Oc/cfPmTRgMBhw7dgxDhgxp1/vTg8jd3R3Ozs546623TF9kfvjhB3zwwQeIjo4GAKhUKvTq1QvJyckICwsDgHvaH/38/HDgwAHU1dWhpqYGJ06cwNChQ+Hl5YWcnByUlpaiuroaWVlZTf5Og4KCkJOTg5KSEtPRvqenJ3bu3AkAOH/+PMLCwlBdXY3jx49j7ty5GDduHK5duwatVtvod1pfX4+xY8eia9eueOmll6DRaPDTTz/d3w/0HjxSRxS+vr44ffo0JkyYAGtra4wdOxbjx48HALzzzjtYuHAhamtr4evri6CgoCbXoVarUV5eDltbWwDAggULEB8fj5CQEBgMBixZsgSPP/44nn/+ecTHx+Ovf/0rOnfuDDc3N1y5cgWPP/54g/UtXrwYy5Ytg42NDezs7LBq1aoG7YMGDcJLL72EWbNmQa/XY8iQIY2WaUqvXr3Qq1cvTJw40TRv/fr1iI+Px+bNm9GhQwds3rwZtra2iIiIwKJFixAaGgpPT08oFArJdU+ZMgUFBQUIDQ2FQqHAY489ho4dO2LkyJEoKCjAlClTYDAYMHr0aEycOBFXr15ttt62cL/7g6urK+bNm4eIiAjo9XqMGjUKkydPhpWVVbvdnx5U7733HpKSkhASEgJra2t06dIF69evb/ClSaPRICkpyTRvzJgxd70/BgcH49KlS5g4cSLq6+uh0WhMQ6uxsbGIjIyEXq9HeHg4XFxcGvXv2LEjhg8fjgEDBpjmrVixAitXrkRoaCgAYN26dbC3t8dLL72E1157DY6OjujevTuGDh2KK1euNFifjY0NYmJiMGfOHHTs2BGOjo5499137/0Hea9a/azIA8poNIra2loREREhfvjhh7YuR5LRaBRFRUVCrVabLidsSV9//bU4fPiwEOL2SbgxY8aIsrKyFt/Ow+xB2p/IPEajUVRUVIjg4GCh0+naupwW9UgNPd2P4uJijBo1Cq6urqbx4/Zq//790Gg0WLx4sembakt66qmnkJycDI1Gg5kzZyImJsY0/krmeZD2JzJPfn4+xowZgylTpjR7VP6g4TuziYhIEo8oiIhIEoOCiIgkMSiIiEgSg4KIiCQxKIiISNIjdcMdkaUYjUa89dZbOH36NKqqqiCEwNq1a/HEE0/g9ddfx6VLl+Dk5ASFQoH+/ftj4cKFuHDhAhISElBeXg6DwYBZs2YhPDy8rT8KUSMMCqIWcPr0aeh0OuzatQtWVlZITk7Ghx9+CLlcDmdnZ2zduhU6nQ6TJk1C//79UV9fj5iYGKxbtw5DhgxBRUUFpk6dCmdnZwwfPrytPw5RAwwKohbg5uaGLl264LPPPsPly5dx8uRJdO7cGadOncKePXsAAEql0vQoj19++QWXLl3C8uXLTeuoqanBmTNnGBTU7jAoiFrAkSNHkJCQgDlz5sDf3x9PPvkk/v73v8PGxqbBU3utrG6fFjQYDHB0dER6erqpraSkxPS4cKL2hCeziVrA8ePH4efnh4iICAwbNgwHDx6EwWCAr68vvvjiCwBAWVkZDh48CJlMhieeeAJ2dnamoLh27RpCQkLwww8/tOXHIGoSH+FB1AIuXLiAuLg41NfXw9raGu7u7sjKykJ6ejpWrFhhOpkthMAf/vAHzJs3DwUFBaaT2fX19YiMjMT06dPb+qMQNcKgILKglJQUDB48GG5ubqirq0NERAQWLlzY6P3bRO0Zz1EQWZCzszPWrFkDo9EIvV6PoKAghgQ9cHhEQUREkngym4iIJDEoiIhIEoOCiIgkMSiIiEgSg4KIiCQxKIiISNL/A5ngmA6o1WgcAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 285 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "_Qkfth4ZZBQV" - }, - "outputs": [], - "source": [ - "# Save the predictions and report the disagregated metrics\r\n", - "# of the exponantiated gradient model\r\n", - "Y_expgrad_exercise = expgrad_exercise.predict(X_test)\r\n", - "mf_expgrad_exercise = MetricFrame(\r\n", - " metrics=metrics_dict,\r\n", - " y_true=Y_test,\r\n", - " y_pred=_______,\r\n", - " sensitive_features=________\r\n", - ")\r\n", - "mf_expgrad_exercise.______" - ] + "id": "nnI8lFHFZyBL", + "outputId": "e0596d0e-efb7-4fc0-d775-99fdae53c8bf" + } + }, + { + "cell_type": "markdown", + "source": [ + "As we see, that's indeed the case." + ], + "metadata": { + "id": "dBgmuoz8aXw4" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "## Exercise" + ], + "metadata": { + "id": "DdrS4By1dVk-" + } + }, + { + "cell_type": "markdown", + "source": [ + "Now, let's explore the relationship between paying with `medicare` and other demographic features. In the below sections, feel free to perform any analysis you would like to better understand the relationship between `medicare` and `race` and `gender` in this dataset." + ], + "metadata": { + "id": "V-sOl0rqblsb" + } + }, + { + "cell_type": "code", + "execution_count": 32, + "source": [ + "sns.pointplot(y=\"medicare\", x=\"race\", data=df, ci=95, join=False);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEJCAYAAACKWmBmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAk9ElEQVR4nO3de0CUZf7//+dwVORknDTLrDQ0wVMm5EclC9c0PNTHNtfKbA3X/Clla2bqGprmrhoey5VsNzdxdUtE0TJPaXwESctczSzNLI+cVUCRcWa+f/BzNgJux5UBxNfjH7zv675m3iPDvOa6D9dtstlsNkRERKrgUtsFiIhI3aagEBERQwoKERExpKAQERFDCgoRETGkoBAREUMKChERMeRW2wU4Q0FBMVarLg8REXGEi4uJxo0bVdleL4PCarUpKEREqol2PYmIiCEFhYiIGFJQiIiIIQWFiIgYUlCIiIghBYWIiBhSUIiIOMm+I7n8Jekr9h3Jre1Srku9vI5CRKQuSEn7kZ+yCikptdC+ZWBtl/Nf04hCRMRJSkovl/t5o1JQiIiIIQWFiIgYUlCIiIghBYWIiBhSUIiIiCEFhYiIGFJQiIiIIQWFiIgYUlCIiIghBYWIiBhSUIiIOEHhhVJKSi0AXLbYarma66OgEBGpRjabjZS0o/zx7Z2cKy4FIO98CfM+3EfRRXMtV/ffUVCIiFSjjV/8zLqdxyqMIv79Qx6Lkvdjs914owsFhYhINTFftvDJrp+rbP/++Fm+P3625gqqJgoKEZFq8nN20VV3L31zrKCGqqk+CgoRkWpiwuTANjceBYWISDW5PdgbHy93w23a3nlLDVVTfRQUIiLVxN3NhUcj76iyvc0djWl1m18NVlQ9dM9sEZFq1Ov+27l02cqGjGOUmq329R1bBTL80TaYTDfezieT7UY8V+sq8vKKsFrr3csSkRvIhRIzk97N5FxxKQF+DZj9QtfaLqlKLi4mAgK8q26vwVpERG4aXg3caeDhCoCby403ivglBYWIiBhSUIiIiCEFhYiIGFJQiIiIIQWFiIgYcmpQpKam0rdvX3r16kVSUlKF9s2bN9OvXz8effRRJkyYQGlp2ZS8KSkpdOvWjQEDBjBgwADmzp3rzDJFRMSA0y64y8rKYu7cuSQnJ+Ph4cHgwYOJiIigZcuWAFy4cIFp06axZs0aAgMDGTt2LGvWrOHJJ59k//79TJgwgZiYGGeVJyIiDnLaiCI9PZ3IyEj8/f3x8vKid+/ebNy40d7u5eXFtm3bCAwM5MKFC+Tl5eHr6wvA/v37SUlJoX///owbN45z5845q0wREbkKpwVFdnY2QUFB9uXg4GCysrLKbePu7s6OHTvo2bMnBQUFdOvWDYCgoCDGjBnD2rVradq0KdOmTXNWmSIichVO2/VU2cwglc1xEhUVRWZmJgkJCcTHx/PWW2/x9ttv29uff/55oqOjr+m5jS5Fl/rj5zPn+Tj9GEdPnqNhAze6t7+VqE634e7mWtuliQDg6upi/xkU5FPL1fz3nBYUISEh7Nmzx76cnZ1NcHCwffns2bMcOHDAPoro168fY8eOpbCwkNWrVzNs2DCgLHDc3K6tTM31VP/tOniGpeu/Lfd7/upQNuvTjvLykx1o6Kn5LqX2uf//QeHu6kJOTmEtV1O1WpvrqWvXrmRkZJCfn8/FixfZtGkTPXr0sLfbbDZeeeUVTp06BcAnn3xCp06d8PLyYunSpezbtw+A5cuX06tXL2eVKTeg/PMl/G3Dt5V+Gfjh1HlW7/ihFqoSqWhg9ztp3dyfgd3vrO1SrotTZ49NTU1lyZIlmM1mBg0aRGxsLLGxscTFxREeHs6WLVuYP38+JpOJli1bMnXqVHx8fNizZw8zZsygpKSEFi1aMGvWLHx8HB+2aURRv6Xu/JE1aT9W2e7p7sr8uG54uGsXlIgjrjai0DTjcsN5N/UbMr7JMtzmzyMfINi/YQ1VJHJj0zTjUu/4NvIwbHcxgXcDHaMQqS4KCrnhPNC2iWF7h1ZBeDUwvm+xiDhOQSE3nOYhPjzSpXmlbb6NPPhtz7truCKR+k3HKOSGZLPZyPjmDJv3nOCnM2WnHXo3dOf1YfcT4NeglqsTubHoYLbUe18fyWHTF8fp3aU57VsG1nY5IjccBYWIiBjSWU8iInJdFBQiImJIQSEiIoYUFCIiYkhBISIihhQUIiJiSEEhIiKGFBQiImJIQSEiIoYUFCIiYkhBISIihhQUIiJiSEEhIiKGFBQiImJIQSEiIoYUFCIiYkhBISIihhQUIiJiSEEhIiKGFBQiImJIQSEiIoYUFCIiYkhBISIihhQUIiJiSEEhIiKGFBQiImJIQSEiIoYUFCIiYkhBISIihpwaFKmpqfTt25devXqRlJRUoX3z5s3069ePRx99lAkTJlBaWgrAqVOneOqpp3jkkUd44YUXKC4udmaZIiJiwGlBkZWVxdy5c1mxYgVr165l1apVHDlyxN5+4cIFpk2bxt///nc2bNjApUuXWLNmDQBTp05lyJAhbNy4kbCwMN555x1nlSkiIlfhtKBIT08nMjISf39/vLy86N27Nxs3brS3e3l5sW3bNgIDA7lw4QJ5eXn4+vpiNpvZvXs3vXv3BuDxxx8v109ERGqW04IiOzuboKAg+3JwcDBZWVnltnF3d2fHjh307NmTgoICunXrRkFBAd7e3ri5uQEQFBRUoZ+IiNQcN2c9sM1mq7DOZDJVWBcVFUVmZiYJCQnEx8czfvx4h/oZCQjwvqbtRUSkag4FRXFxMXPmzOHo0aPMnz+fhIQEXn31VRo1alRln5CQEPbs2WNfzs7OJjg42L589uxZDhw4QLdu3QDo168fY8eO5ZZbbqGoqAiLxYKrqys5OTnl+jkiL68Iq7ViUImISEUuLibDL9gO7XqaPn06vr6+5OXl4enpSVFREVOmTDHs07VrVzIyMsjPz+fixYts2rSJHj162NttNhuvvPIKp06dAuCTTz6hU6dOuLu707lzZz7++GMAUlJSyvUTEZGaZbJVto/oVwYOHEhKSor9p9VqJSYmxv5hXpXU1FSWLFmC2Wxm0KBBxMbGEhsbS1xcHOHh4WzZsoX58+djMplo2bIlU6dOxcfHh5MnTzJhwgTy8vJo2rQpCQkJ+Pn5OfyiNKIQEXHc1UYUDu16cnEpP/CwWCwV1lWmX79+9OvXr9y6d9991/7v6OhooqOjK/Rr1qwZH3zwgSOliYiIkzkUFPfffz+zZ8+mpKSEtLQ0kpKSiIiIcHZtIiJSBzi068lsNpOYmMj27duxWCx0796dUaNG4enpWRM1XjPtehIRcdzVdj05FBRvvfUWf/zjH6u1MGdSUIiIOK5aznravn17ddUjIiI3GIeOUdx22238/ve/p1OnTuWunXjuueecVpiIiNQNDgWFv78/ACdPnnRmLSIiUgc5dIziRqNjFCIijquW6yj27t1LYmIiFy5cwGazYbVaOXHihI5diIjcBBw6mD158mQ6duxIUVER/fr1w9vbm9/85jfOrk1EROoAh0YUJpOJESNGUFBQwF133UX//v353e9+5+zaRESkDnBoRHHlTKfmzZtz+PBhPD09sVgsTi1MRETqBodGFOHh4bz00ku8+OKL/OEPf+DYsWO4uro6uzYREakDHDrryWazsW/fPjp06MCOHTtIT0/nySef5K677qqJGq+ZznoSEXFctVyZnZ2dTUpKCgC33347x48fx8fHp1oKFBGRus2hoHj11Vfto4dmzZrRpUsXJk6c6NTCRESkbnAoKAoKChg6dCgAnp6eDBs2jJycHKcWJiIidYNDQWGxWMjKyrIv5+bmUg8v6BYRkUo4dNbTsGHDGDhwIN27d8dkMpGens748eOdXZuIiNQBDs/1dOjQIXbt2oWrqysRERHcc889zq7tv6aznkREHHddNy764YcfuPvuu/nmm28qbW/btu31V+gECgoREcddV1CMGDGCxMREHnrooYodTSa2bt1aPVVWMwWFiIjjquVWqDcaBYWIiOOua5rxRYsWGT746NGj/7uqRETkhmEYFAUFBQAcPXqUH3/8kejoaNzc3Ni6dSuhoaE1UqCIiNQuh3Y9DR06lHnz5nHLLbcAcO7cOUaNGkVSUpLTC/xvaNeTiIjjqmWup5ycHHtIAPj6+pKXl3f91YmISJ3n0AV3oaGhvPbaawwYMACbzcZHH31E+/btnV2biIjUAQ7teioqKmLBggVkZGRgMpno3r07Y8aMoUGDBjVR4zXTricREcdV2+mxJSUlHDt2jHvuuYfS0tI6GxKgoBARuRbVcozi66+/Jjo6mpEjR5KdnU1UVBRfffVVtRUpIiJ1l0NBMWvWLN5//338/f1p0qQJs2bNYsaMGc6uTURE6gCHgqKkpISWLVval6OiorBYLE4rSkRE6g6HgsLNzY1z585hMpmAsgvwRETk5uDQ6bEjR47k6aefJicnh5dffpmdO3cybdo0Z9cmIiJ1gMNnPe3evZtDhw7h4uJC586dcXV1Lbc7qjKpqaksXrwYs9nMsGHDeOqpp8q1b9myhYULF2Kz2bjtttuYOXMmfn5+pKSkMGfOHAICAgB48MEHGTt2rMMvSmc9iYg4rlpOj505cyZJSUl4e//ngUwmExkZGVX2ycrK4ne/+x3Jycl4eHgwePBgEhIS7OFSVFTEI488wurVqwkJCWH+/PkUFhYyefJk3njjDTp27EhMTMy1vFY7BYWIiOOua/bYKzZv3kxaWhqNGzd2+InT09OJjIzE398fgN69e7Nx40b7jLNms5n4+HhCQkKAsqu/U1NTAdi/fz8//fQTiYmJ3HPPPfzpT3/Cz8/P4ecWEZHq49DB7BYtWuDr63tND5ydnU1QUJB9OTg4mKysLPty48aNiY6OBsrOqkpMTLQvBwUFMWbMGNauXUvTpk11PEREpBY5NKJ45plnePrpp4mIiMDN7T9djO5HUdkerStnTf1SYWEho0aNonXr1jz22GMAvP322/b2559/3h4gjjIaQtUFuw+eIXn7ER5/sCX339uktssRETHkUFAsXLiQgIAACgsLHX7gkJAQ9uzZY1/Ozs4mODi43DbZ2dkMHz6cyMhIJk6cCJQFx+rVqxk2bBhQFji/DCdH1PVjFMvWH+SnrEIKi0ppEdSotssRkZtctRyjuHjxIu++++41PXHXrl1ZuHAh+fn5NGzYkE2bNvHGG2/Y2y0WCyNHjqRPnz6MGjXKvt7Ly4ulS5fSsWNH2rdvz/Lly+nVq9c1PXddVVxiJm3faU7nFQOQe+4i54pL8WvkUcuViYhUzaGgaNWqFYcOHaJ169YOP3BISAhjx45l6NChmM1mBg0aRLt27YiNjSUuLo4zZ85w8OBBLBYLn376KQBhYWHMmDGDefPmER8fT0lJCS1atGDWrFn/3aurQ7IKLjD7n3vJP3/Jvq645DKT393FuMEduaOJTy1WJyJSNYdOjx08eDAHDhygWbNmeHj859vvlbOU6pq6uOtpxgd7+OHk+UrbgvwbMHPEA7i4VDyGIyLibNWy6+nll1+utoJuRj9nFVYZEgA5Z0v45lg+4XcF1GBVIiKOcSgounTp4uw66rWsgotX3eZM3gUFhYjUSQ5dRyHXx6eh+9W3aXT1bUREaoOCogbcc7s/Ab5V3xGwoYcrHVsGVdkuIlKbFBQ1wMXFxLOPhOLmWvFgtQl4+jeheHq41nxhIiIOcHj22BtJXTzrCeDoqfOsTz/G10dyAfBwcyFuUDvubXFLLVcmIjezarlntlSPu271JW5QO0IaNwSgsY+nQkJE6jwFRS1o4OFW7qeISF2moKgFA7vfSevm/gzsfmdtlyIiclU6RiEicpPTMQoREbkuCgoRETGkoBAREUMKChERMaSgEBERQwoKERExpKAQERFDCgoRETGkoBAREUMKChERMaSgEBERQwoKERExpKAQERFDCgoRETGkoBAREUMKChERMaSgEBERQwoKERExpKAQERFDCgoRETGkoBAREUMKChERMaSgEBERQwoKERExpKAQERFDTg2K1NRU+vbtS69evUhKSqrQvmXLFgYMGED//v0ZNWoU586dA+DUqVM89dRTPPLII7zwwgsUFxc7s0wRETHgtKDIyspi7ty5rFixgrVr17Jq1SqOHDliby8qKiI+Pp7ExETWrVtHaGgoCxcuBGDq1KkMGTKEjRs3EhYWxjvvvOOsMkVE5CqcFhTp6elERkbi7++Pl5cXvXv3ZuPGjfZ2s9lMfHw8ISEhAISGhnL69GnMZjO7d++md+/eADz++OPl+omISM1yWlBkZ2cTFBRkXw4ODiYrK8u+3LhxY6KjowEoKSkhMTGR6OhoCgoK8Pb2xs3NDYCgoKBy/UREpGa5OeuBbTZbhXUmk6nCusLCQkaNGkXr1q157LHHKg2FyvoZCQjwvqbtRUSkak4LipCQEPbs2WNfzs7OJjg4uNw22dnZDB8+nMjISCZOnAjALbfcQlFRERaLBVdXV3Jycir0u5q8vCKs1opBJSIiFbm4mAy/YDtt11PXrl3JyMggPz+fixcvsmnTJnr06GFvt1gsjBw5kj59+jBp0iT7qMHd3Z3OnTvz8ccfA5CSklKun4iI1CyTrbJ9RNUkNTWVJUuWYDabGTRoELGxscTGxhIXF8eZM2cYM2YMoaGh9u3DwsKYMWMGJ0+eZMKECeTl5dG0aVMSEhLw8/Nz+Hk1ohARcdzVRhRODYraoqAQEXFcre16EhGR+kFBISIihhQUIiJiSEEhIiKGFBQiImJIQSEiIoYUFCIiYkhBISIihhQUIiJiSEEhInb7juTyl6Sv2Hckt7ZLkTrEabPHisiNJyXtR37KKqSk1EL7loG1XY7UERpRiIhdSenlcj9FQEEhIiJXoaAQERFDCgoRETGkoBARLpktbMg4Ru65EgByz5ewec9xLlustVyZ1AW6cZHITe6S2cJbK7/myMlzFdo6tAzk/3s8DFcXfaesz3TjIhExtPXLE5WGBMDXR3LJPJhVwxVJXaOgELnJ7dx/+irtZ2qoEqmrFBQiN7lzRaXG7cXG7VL/KShEbnJBjRsat/s1qKFKpK5SUIjc5KI63Grc3rFZDVUidZWCQuQm16PdrXRpE1xpW/R9t9H+7oAarkjqGp0eKyJYbTa+/C6H9zYcpNRsxdPdlRcGhhF+1y2YTKbaLk+cTKfHishVuZhM3N86mMbengD4e3vQ7u4AhYQACgoREbkKBYWIiBhSUIiIiCEFhYiIGFJQiIhdAw+3cj9FQEEhIr8wsPudtG7uz8Dud9Z2KVKH6DoKEZGbnK6jEBGR66KgEBERQ04NitTUVPr27UuvXr1ISkqqcrtXX32V5ORk+3JKSgrdunVjwIABDBgwgLlz5zqzTBERMeC0UxuysrKYO3cuycnJeHh4MHjwYCIiImjZsmW5bV5//XUyMjKIiIiwr9+/fz8TJkwgJibGWeWJiIiDnDaiSE9PJzIyEn9/f7y8vOjduzcbN24st01qaioPP/wwffr0Kbd+//79pKSk0L9/f8aNG8e5c5XfplFERJzPaUGRnZ1NUFCQfTk4OJisrPL33n3++ed54oknKvQNCgpizJgxrF27lqZNmzJt2jRnlSkiIlfhtF1PlZ116+hMlG+//bb9388//zzR0dHX9NxGp3mJiMi1cdqIIiQkhNzcXPtydnY2wcGV3xzllwoLC3n//fftyzabDTc3XSUqIlJbnBYUXbt2JSMjg/z8fC5evMimTZvo0aPHVft5eXmxdOlS9u3bB8Dy5cvp1auXs8oUEZGrcNpX9ZCQEMaOHcvQoUMxm80MGjSIdu3aERsbS1xcHOHh4ZX2c3V1Zd68ecTHx1NSUkKLFi2YNWuWs8oUEZGrqJdTeIiISPXRldkiImJIQSEiIoYUFCIiYkhBISIihhQUIiJiSEEhIiKG6n1QfP/994SGhvLpp5/a1+3YsYOePXvyxz/+scL2AwYMcEodcXFx9OvXzymPDWUTKU6aNMlpj389ioqKmDp1KjExMQwYMIBnnnmGb775ptbqcdbvuK4qLi5m6tSp9OrVi/79+zNkyBAyMjIAWLVqFevXrwdgwoQJ5ab7v9mcOHGChx56qML60NDQKvtkZmbyzDPPOLOsOqHez42RnJxM7969WblyJb179wZg48aNjBw5kieffLLC9mvXrq32GgoKCjh48CCBgYF8+eWX3HfffdX+HOHh4VVexFibrFYrsbGxREREkJKSgpubG7t27SI2NpYNGzbQuHHjGq/JGb/juspmszFy5EjatGnDhg0b8PDw4ODBg4wYMYK33nqLvXv30qVLl9ouU+q4eh0Uly9fZt26dSQlJTF48GB+/vlnMjMz2bp1KxkZGbi4uLBu3Tr8/Pw4fPgw8+bNY+DAgXz33XecPXuWSZMmcfToUTw8PJgwYQIPPPAAy5cvZ+3atVy8eBGTycS8efO4++67eeihh+jfvz//93//x8WLF/nLX/5CWFgYUDadeufOnbnnnntYtWqVPSiSk5PZvn072dnZnDlzhmeffZZTp06xa9cu/P39Wbp0KZ6enqSkpLBs2TKsVitt27bl9ddfx9PTk8jISNq2bUtubi7jx4/nr3/9Kx988AHffvstU6ZMoaSkBD8/P+bMmUNgYCDx8fEcPnyY3Nxc7rzzThYtWkRubi6jR4+mVatWfPvttwQEBDB//nz8/f2r5XeQmZlJdnY2cXFxuLiUDWAjIyOZOXMmVquVyZMnV1rT0KFD2bZtGwALFy4EYMyYMaSmprJ48WJMJhPh4eG88cYb5OfnM3HiRAoLC8nJyeHRRx9l3LhxHDp0iClTpnD58mU8PT2ZOXMmLVq0IDQ0lO+++46srKxK+yUnJ5OWlsa5c+c4fvw4//M//0N8fHy1/H/UtC+++IJTp07xj3/8wz4p57333ssLL7zAsGHD8PHxYdeuXfaZnrdv386KFSvIy8uzf5kqLi5m2rRpHD58GIvFQmxsLDExMSQnJ7NmzRrOnj1Lz549efnll2vzpTqVI++JZcuWsWXLFhITExkxYgTh4eF8+eWX5OfnM3nyZKKiosjNzWXSpEmcOnUKNzc3xo4dS1hYmP2zA6B79+689tpr9O3bl8TEREwmEyUlJWRlZfHTTz9x8uRJnnjiCV544YUae/31etfT9u3bufXWW7nzzjuJjo5m5cqVPPHEEzz00EPExcXZpzi/smuqTZs29r7z58+nefPmfPLJJ8yaNYt58+ZRVFTEli1b+OCDD1i/fj3R0dGsWLHC3sff35+PPvqIwYMHs2TJEvv65ORk+vTpQ58+ffj00085e/asvW3//v0sXbqUpKQk/vznP9OjRw9SU1MBSEtL4/Dhw/zrX/9i5cqVrF27loCAAN577z2gbKQyYsQI1q5dW27ixHHjxjFq1Cj7HQaXLVvG3r17cXd3Z9WqVWzevJlLly6xY8cOAA4dOsRzzz3H+vXr8fX1tT9/dTh48CDh4eH2kLgiKiqKo0ePVllTZbKyspg5cyZ/+9vf2LBhAxaLhR07drB+/XpiYmL417/+xbp161ixYgX5+fksW7aM5557juTkZJ555hm+/vrrco9XVT+AvXv3smDBAtatW8dnn33Gd999V23/JzVp//79hIWFVZi5+f7776dhw4b2v4Xu3bsDUFpayocffsiSJUvsd5ZcvHgxbdu2JTk5maSkJP76179y/PhxoOx3smbNmnodElcYvSdWr17Npk2bWLJkCQ0bNgTAbDazatUqXnvtNebPnw/AG2+8QWRkJKmpqSxYsICJEyditVq59dZb+f777/nhhx+wWCx88cUXAHz++ef07NkTgO+++4733nuPDz/8kMTERM6fP19jr71ejyiSk5Ptd8nr27cv48aN46WXXqqwXbt27Sqs2717N3PmzAHKgmTVqlUAvPXWW2zYsIFjx46RlpZWLlyu/LG1atWKTZs2AfDtt99y+vRpunbtiru7O23atCElJYVhw4YB0KlTJ7y9vfH2Lpsa/YEHHgCgWbNmnD9/nszMTH766Sd++9vfAmVvvnvvvdf+nO3bty9Xd35+Pjk5OfY315AhQ+xt/v7+JCUlcfToUY4dO8aFCxcACAgIsD9mq1atqvVGUS4uLpVOOQ9lH1ZV1VSZvXv30qlTJ5o0aQLA7Nmz7W27du3ivffe4/Dhw5jNZi5evEhUVBTTpk0jLS2Nnj172nc9XjF8+PBK+wF07NjR/ju5/fbbb9ibZ5lMJiwWS4X1ZrO50u0ffvhhTCYTrVq1oqCgACi7CVlJSQmrV68G4MKFCxw+fBgoG53Ul9mdf/1lBsp23V0J2areE99//z1TpkwhISEBLy8ve99ffh5c+XK4a9cupk+fbn+M9u3bs2/fPqKiosjIyMDNzY2hQ4eyYcMG+0j3yl1BIyIi8PDwICAgAH9/fwoLC/H19XXOf8av1I/fcCXy8vL4/PPPOXDgAP/4xz+w2WycP3/e/gH+Sw0aNKiw7tdv/h9++IEGDRrw7LPP8vTTT9OjRw8CAwP59ttv7dt4enoC5e+7sXr1akpLS+0fUsXFxaxcudIeFO7u7obPa7FY6NOnD5MnT7b3/+Uf/q9r//XjXbp0iezsbL7//nsWLFjA0KFDefzxxykoKLB/gF+p+0rt1Tn9V1hYGCtWrCj3BweQkJBAu3btWLhwYYWafl3D5cuXcXNzq/B/c+Xbf2JiIsePHycmJobo6GjS09Ox2Ww88sgjdOzYkc8++4xly5axY8cO+x8pwJ///OdK+zn7/6QmtW/fng8++ACz2VzuvfH1119XekzL1dUVKP8etlqtzJ49m7Zt2wKQm5uLn58fqamplf7t3Kh8fX0pLCwsty4vLw8/Pz+g6vdEo0aNePPNN3nzzTfp3r27PSwq+zz49fvIZrNhsViIiopi0aJFeHh48OKLL/LJJ5+QmppqDxuj568J9XbX07p164iMjOTzzz9n27ZtfPbZZ4wcOdI+Mriazp078/HHHwNlIREbG8uBAwe44447GDZsGO3bt+fzzz+v9NvaFaWlpaSmpvL++++zbds2tm3bxtatW8nJySEzM9OhOiIiIti8eTN5eXnYbDbi4+NZtmxZldv7+PjQpEkTdu7cCZQduJ0/fz4ZGRn06dOH//3f/yUwMJDdu3cb1l5dOnfuTEBAAIsWLbI/X1pamn2fb2U1+fr6cu7cOfLz8yktLSUtLQ0oO2C/b98+cnJyAHjzzTfZunUrO3fuZPjw4fTp04fTp0+TlZWF1WrlpZde4t///jeDBw/mxRdf5ODBg+Vqq6pffdK5c2datmzJm2++aR9FHDhwgMWLFzNq1ChcXV2v+j6IjIzkn//8J1B2X5n+/ftz+vRpp9de07y9vbnjjjvKnSG5atUq+yi/Ks2aNePhhx+mS5cuLFiwwHDbyMhIPvroIwCOHz/OV199RYcOHWjbti0//vgjx44d4+677yYiIoLFixfz4IMPXvfrqg71dkSRnJzM2LFjy60bMmQIS5cuJTAw8Kr94+LimDx5Mv3798fNzY1Zs2bRpk0bVq5cSd++ffHw8KBdu3b2IXhlPvvsM5o1a1Zu95C3tzdPPPEEK1euLPdtoSqtW7dm9OjRPPvss1itVtq0acOIESMM+8yePZv4+HhmzZpF48aNmTVrFgUFBYwbN46NGzfi4eFBhw4dOHHixFWf/3qZTCbeeecdZs6cSUxMDG5ubjRu3JjExERcXV0rrcnHx4fhw4czaNAgmjRpYv/mGxISwqRJkxg+fDhWq5UOHTrw+OOP07BhQ8aPH4+vry8BAQGEhYVx4sQJRo4cyaRJk3jnnXdwdXVlwoQJ5Wr7wx/+UGm/+mbRokXMnTuXmJgYXF1d8fPzY/bs2URERJCbm0tCQgI+Pj5V9h89ejTx8fHExMRgsVh45ZVXaN68OXv27KnBV1EzrvztvP3225jNZkJDQ5kyZQrbt2+/at/x48cTExNjeBr8pEmTmDJliv005OnTp9tv6HbffffZd31GRkby4Ycf1pkz0jTNuIiIGKq3u55ERKR6KChERMSQgkJERAwpKERExJCCQkREDCkoRETEkIJCREQM1dsL7kRqSmZmJjNmzMDLy4vi4mI6derEwYMHKS4uxmazMX36dO677z6Ki4uZPn06X331Fa6urkRHRzN27FjMZjNz5syxX5l+7733MnnyZPu8QiK1TUEhUg0OHz7Mli1byM7O5u9//zurVq3CxcWFxMRE3n33Xe677z4WLFjApUuX+Pjjj7FYLPz+97/niy++YPfu3bi6upKcnIzJZCIhIYE5c+bcsFObS/2joBCpBk2bNqVZs2Y0a9YMPz8/Vq5cyfHjx8nMzKRRo0ZA2Sysr732Gq6urri6urJ8+XKgbNqIwsJC0tPTgbKZXQMCAmrttYj8moJCpBpcmTF0+/btzJgxg+eee46HH36Yu+66i3Xr1gFlMwP/cibR06dP06BBA6xWKxMnTiQqKgoomyH40qVLNf8iRKqgg9ki1Wjnzp307NmTIUOGEB4ezpYtW+yzsz7wwAOsWbMGq9VKaWkpcXFx7N69m27dupGUlERpaSlWq5U//elPJCQk1PIrEfkPBYVINRo8eDC7d++mX79+PPnkk9x+++2cOHECq9XK6NGjcXd3Z8CAAQwcOJCoqCh+85vfMGrUKJo1a8Zjjz1G3759sdlsFWa6FalNmj1WREQMaUQhIiKGFBQiImJIQSEiIoYUFCIiYkhBISIihhQUIiJiSEEhIiKGFBQiImLo/wEzi1/L0+HLWAAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 285 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Compare with the disaggregated metric values of the\r\n", - "# exponentiated gradient model based on logistic regression\r\n", - "metricframe_reductions.____" - ] + "id": "lp02oUbWZx54", + "outputId": "40c1cc55-625f-47d5-a21a-8950db45ab23" + } + }, + { + "cell_type": "code", + "execution_count": 33, + "source": [ + "sns.pointplot(y=\"medicare\", x=\"gender\", data=df, ci=95, join=False);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEJCAYAAAC61nFHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAk7klEQVR4nO3df1RVdb7/8efhl4lApHDwR2UzUXgn0ca8QY5KGV4VJPx5/VVo3UGt1ehQwcVKL6LlygQlnfyq617vrBlMGwcxUhCU60/QtLr+mGumU/bVdPgpehAUOOd8//DbmbgoHnccQH091nKx9v7s/TnvfQReZ3/23h9MdrvdjoiIyC1ya+sCRETk9qQAERERQxQgIiJiiAJEREQMUYCIiIghChARETFEASIiIoZ4tHUBrenChcvYbHrsRUTEGW5uJu67r9MN2++qALHZ7AoQEZEWoiEsERExRAEiIiKGKEBERMQQBYiIiBiiABEREUMUICIiYogC5A5w+FQ572V+weFT5W1diojcRe6q50DuVNl7vuW7EgtX6qz0DQ5o63JE5C6hM5A7wJW6hkZfRURagwJEREQMUYCIiIghChARETFEASIiIoYoQERExBAFiIiIGKIAERERQxQgIiJiiAJEREQMUYCIiIghChARETFEASIiIoYoQERExBAFiIiIGKIAERERQ1waIDk5OURFRTF06FAyMzObtBcUFBATE0N0dDTJycnU1dUBcOjQIcaMGUNMTAwzZ87k4sWLAFy6dInp06czYsQIpkyZQllZmSvLFxGRZrgsQEpKSli6dCnr1q1j8+bNbNiwgVOnTjnaa2pqSE1NZe3atWzZsoWrV6+yadMmAObMmcPixYvJyckhODiYf//3fwdg2bJl9O/fn9zcXMaPH88777zjqvJFROQmXBYgRUVFhIeH4+/vj7e3N8OGDSMvL8/R7u3tTWFhIQEBAdTU1FBRUYGfnx8AW7duJTg4mPr6ekpKShzrd+7cSUxMDAAjR45k9+7d1NfXu+oQRESkGS4LkNLSUgIDAx3LZrOZkpKSRtt4enqya9cunnnmGS5cuMDAgQMd60+cOEFERAQHDhwgOjq6SZ8eHh74+PhQWVnpqkMQEZFmeLiqY7vd3mSdyWRqsu6HkEhPTyclJYW0tDQAQkJCKCoqYv369SQkJLB+/frrvo6bm/MZ2KWLj9Pb3k7c3d0cXwMDfdu4GhG5W7gsQIKCgjh06JBjubS0FLPZ7Fiuqqri2LFjjrOOmJgYEhISuHr1Knv27CEyMhKA5557jvfeew+4dhZTXl5O165daWhooLq6Gn9/f6drqqioxmZrGmy3M5vdztU6KwANVhtlZZY2rkhE7hRubqZmP3i7bAhrwIABFBcXU1lZSW1tLfn5+QwePNjRbrfbSUxM5Ny5cwDk5ubSr18/PDw8mD9/PseOHWu0Hq6drWRnZwPXrpP0798fT09PVx1Cu3fwq1LeXL2fiktXACirqmXz3m/vuJAUkfbJZL/eWFMLycnJYdWqVdTX1zNu3Dji4+OJj49n1qxZhIaGsn37djIyMjCZTAQHBzN//nx8fX05dOgQ7777LlarlaCgIFJTU+natStVVVUkJydz5swZfH19WbJkCffff7/T9dxJZyD7/+dvrP7kf67b9vQvexA3LKSVKxKRO83NzkBcGiDtzZ0SIDabnaT/U0Tlpas33GbR9HCCOnu3YlUicqdpsyEscZ1v/3ap2fAA+OJrPWQpIq6lALkN1dXbbrrN1XprK1QiInczBcht6AGzDx7uTW+J/rGfd/drpWpE5G6lALkN+XT0ZGBotxu2dw/oRO+fdWnFikTkbqQAuU1NfPYRHg8OaLI+qLM3s8b1wc2t+TMUEZGfSndh3cbsdjt/PXeJZX86TM2VBu7t5MX7rwzAw12fC0Tkp9NdWHcwk8lEcI978e147WHKe7zcFR4i0mr020ZERAxRgIiIiCEKEBERMUQBIiIihihARETEEAWIiIgYogARERFDFCAiImKIAkRERAxRgIiIiCEKEBERMUQBIiIihihARETEEAWIiIgYogARERFDFCAiImKIAkRERAxxaYDk5OQQFRXF0KFDyczMbNJeUFBATEwM0dHRJCcnU1dXB8Dnn3/O2LFjiY2NZerUqXz//fcAHDx4kLCwMGJjY4mNjWXOnDmuLF9ERJrh4aqOS0pKWLp0KVlZWXh5eTFx4kTCwsIIDg4GoKamhtTUVDZt2kRAQAAJCQls2rSJCRMmkJiYyIcffkivXr3YuHEjCxcuZOXKlRw9epSXXnqJGTNmuKpsERFxksvOQIqKiggPD8ff3x9vb2+GDRtGXl6eo93b25vCwkICAgKoqamhoqICPz8/6urqmD17Nr169QIgJCSE8+fPA3D06FH27dvHqFGjmDlzpmO9iIi0PpedgZSWlhIYGOhYNpvNHDlypNE2np6e7Nq1i6SkJMxmMwMHDsTLy4vY2FgAbDYbK1asIDIyEgBfX1+io6OJjIzko48+IiEhgfXr1ztdU5cuPi1wZO2PTycvSi7U4tPJi8BA37YuR0TuEi4LELvd3mSdyWRqsi4iIoIDBw6Qnp5OSkoKaWlpANTV1ZGcnExDQ4NjyCo1NdWx36RJk0hLS8NiseDr69wvzYqKamy2pnXd7kaG92TbZ/+XYU8+SFmZpa3LEZE7hJubqdkP3i4bwgoKCqK8vNyxXFpaitlsdixXVVWxd+9ex3JMTAwnTpwA4PLly/z617+moaGBlStX4unpic1mY+XKlVit1kav4+Hhsgy8bfQNDiBpcj/6Bge0dSkichdxWYAMGDCA4uJiKisrqa2tJT8/n8GDBzva7XY7iYmJnDt3DoDc3Fz69esHQGJiIj179iQjIwMvL69rhbq5UVBQwLZt2wDIzs6mb9++dOzY0VWHICIizTDZrzfW1EJycnJYtWoV9fX1jBs3jvj4eOLj45k1axahoaFs376djIwMTCYTwcHBzJ8/nzNnzjB69GiCg4MdZxdms5k1a9Zw8uRJ5s6di8VioXPnzixevJhu3bo5Xc+dOoQlIuIKNxvCcmmAtDcKEBER57XZNRAREbmzKUBERMQQBYiIiBiiABEREUMUICIiYogCREREDFGAiIiIIQoQERExRAEiIiKGKEBERMQQBYiIiBiiABEREUMUICIiYogCREREDFGAiIiIIQoQERExRAEiIiKGKEBERMQQpwLk8uXLzJ8/n6lTp1JVVcW8efO4fPmyq2sTEZF2zKkAWbhwIX5+flRUVNChQweqq6uZN2+eq2sTEZF2zKkAOX78OAkJCXh4eNCxY0eWLFnC8ePHXV2biIi0Y04FiJtb482sVmuTdSIicnfxcGajf/zHf+T999/nypUr7Nmzh8zMTMLCwlxdm4iItGNOnUa88cYbeHt74+vry9KlSwkJCSEpKemm++Xk5BAVFcXQoUPJzMxs0l5QUEBMTAzR0dEkJydTV1cHwOeff87YsWOJjY1l6tSpfP/99wBcunSJ6dOnM2LECKZMmUJZWdmtHKuIiLQgk91ut99so7S0NF5//fVb6rikpIRJkyaRlZWFl5cXEydOJD09neDgYABqamoYNmwYmzZtIiAggISEBMLDw5kwYQJDhgzhww8/pFevXmzcuJEdO3awcuVKUlNT6dq1K9OnTyc7O5udO3eybNkyp2uqqKjGZrvp4YqICODmZqJLF58btzvTyc6dO2/5hYuKiggPD8ff3x9vb2+GDRtGXl6eo93b25vCwkICAgKoqamhoqICPz8/6urqmD17Nr169QIgJCSE8+fPO+qIiYkBYOTIkezevZv6+vpbrk1ERH46p66B3H///bz00kv069ePTp06Oda/+OKLN9yntLSUwMBAx7LZbObIkSONtvH09GTXrl0kJSVhNpsZOHAgXl5exMbGAmCz2VixYgWRkZFN+vTw8MDHx4fKykqCgoKcPFwREWkpTgWIv78/gONahDOuNzJmMpmarIuIiODAgQOkp6eTkpJCWloaAHV1dSQnJ9PQ0MCMGTNu+Dq3cjdYc6diIiJya5wKkEWLFt1yx0FBQRw6dMixXFpaitlsdixXVVVx7NgxBg4cCEBMTAwJCQnAtSffX375Zfz9/Vm5ciWenp7AtbOY8vJyunbtSkNDA9XV1Y5wc4augYiIOK9FroF8+eWXvPzyy0ydOpW4uDief/55nn766Wb3GTBgAMXFxVRWVlJbW0t+fj6DBw92tNvtdhITEzl37hwAubm59OvXD4DExER69uxJRkYGXl5ejn0iIiLIzs4GYOvWrfTv398RLiIi0rqcugsrOjqa2NhYtm3bxsSJE9mxYwcPPvggb775ZrP75eTksGrVKurr6xk3bhzx8fHEx8cza9YsQkND2b59OxkZGZhMJoKDg5k/fz5nzpxh9OjRBAcH4+Fx7QTJbDazZs0aqqqqSE5O5syZM/j6+rJkyRLuv/9+pw9WZyAiIs672RmIUwEycuRIPv30U9577z0iIyPp3bu34xbd24kCRETEeS0yhPXDnVcPPvggJ0+epEOHDlit1papUEREbktOXUQPDQ3lt7/9LbNnz2bGjBmcPn0ad3d3V9cmIiLtmFNDWHa7ncOHD/P444+za9cuioqKmDBhAj//+c9bo8YWoyEsERHntcgQVmlpqePupwceeMBxEVtERO5eTgXIv/7rvzrONnr06MGTTz550zuwRETkzuZUgFy4cIG4uDgAOnTowLRp0zQTrojIXc6pALFarZSUlDiWy8vLrztViYiI3D2cugtr2rRpjBo1ikGDBmEymSgqKnLq74GIiMidy6m7sAC++uor9u/fj7u7O2FhYTz66KOurq3F6S4sERHn/aQn0f/617/y8MMP85e//OW67Y899thPr7AVKUBERJz3kwJk+vTprF69miFDhjTd0WRix44dLVNlK1GAiIg4r0XmwrpTKEBERJx3swBp9iL6ihUrmu381VdfNVaViIjc9poNkAsXLgDwzTff8O233xIZGYmHhwc7duwgJCSkVQoUEZH2yakhrLi4OJYtW0bnzp0BuHjxIq+88gqZmZkuL7AlaQhLRMR5LTIXVllZmSM8APz8/KioqPjp1YmIyG3LqQcJQ0JCmDNnDrGxsdjtdjZu3Ejfvn1dXZuIiLRjTg1hVVdX88EHH1BcXIzJZGLQoEH85je/4Z577mmNGluMhrBERJzXYrfxXrlyhdOnT/Poo49SV1d324UHKEBERG5Fi1wD+e///m8iIyOZOXMmpaWlRERE8MUXX7RYkSIicvtxKkAWL17Mf/7nf+Lv70/Xrl1ZvHgx77zzjqtrExGRdsypALly5QrBwcGO5YiICKxWq8uKEhGR9s+pAPHw8ODixYuYTCbg2oOFIiJyd3PqNt6ZM2fy/PPPU1ZWxmuvvca+fftITU11dW0iItKOOX0X1sGDB/nqq69wc3Ojf//+uLu7NxrWup6cnBxWrlxJfX0906ZNY8qUKY3aCwoK+OCDD7DZbISGhpKamoqXl5ejPSMjAzc3N37zm984anj11Vfp2rUrAL/4xS9YtGiR0weru7BERJz3kyZT/MGiRYvIzMzEx+fvHZlMJoqLi2+4T0lJCUuXLiUrKwsvLy8mTpxIWFiYI3RqampITU1l06ZNBAQEkJCQwKZNm5gwYQIWi4VFixaxZcsWfv3rXzv6PHr0KC+99BIzZsxwpmwREXEhpwKkoKCAPXv2cN999zndcVFREeHh4fj7+wMwbNgw8vLyHDP4ent7U1hYiKenJzU1NVRUVODn5wfAjh07eOihh3jxxRcb9Xn06FEqKirIzc2la9eu/Nu//RvdunVzuiYREWk5TgXIQw895Pjl7qzS0lICAwMdy2azmSNHjjTaxtPTk127dpGUlITZbGbgwIEAjBo1CoDly5c32t7X15fo6GgiIyP56KOPSEhIYP369U7X1NypmIiI3BqnAuSFF17g+eefJywsDA+Pv+/S3N8Dud6llR/u4vqxiIgIDhw4QHp6OikpKaSlpd2wzx9fuJ80aRJpaWlYLBZ8fX2dOQxdAxERuQUt8iT68uXL8fHxwWKxcOHCBce/5gQFBVFeXu5YLi0txWw2O5arqqrYu3evYzkmJoYTJ07csD+bzcbKlSubPH/y40ATEZHW49Rv39raWtasWXNLHQ8YMIDly5dTWVlJx44dyc/PZ8GCBY52u91OYmIif/7zn+nevTu5ubn069fvhv25ublRUFBAz549iYqKIjs7m759+9KxY8dbqktERFqGUwHyyCOP8NVXX9GrVy+nOw4KCiIhIYG4uDjq6+sZN24cffr0IT4+nlmzZhEaGsqCBQuYMWMGJpOJ4OBg5s+f32yf7733HnPnzuV3v/sdnTt3ZvHixU7XIyIiLcup50AmTpzIsWPH6NGjR6PnNHJyclxaXEvTNRAREee1yHMgr732WosVJCIidwann0S/E+gMRETEeS1yF5aIiMj/pgARERFDFCAiImKIAkRERAxRgIiIiCEKEBERMUQBIiIihihARETEEAWIiIgYogARERFDFCAiImKIAkRERAxRgIiIiCEKEBERMUQBIiIihihARETEEAWIiIgYogARERFDFCAibezwqXLey/yCw6fK27oUkVvi0dYFiNztsvd8y3clFq7UWekbHNDW5Yg4TWcgIm3sSl1Do68itwuXBkhOTg5RUVEMHTqUzMzMJu0FBQXExMQQHR1NcnIydXV1jdozMjJYvny5Y/nSpUtMnz6dESNGMGXKFMrKylxZvoiINMNlAVJSUsLSpUtZt24dmzdvZsOGDZw6dcrRXlNTQ2pqKmvXrmXLli1cvXqVTZs2AWCxWHjzzTf5j//4j0Z9Llu2jP79+5Obm8v48eN55513XFW+iIjchMsCpKioiPDwcPz9/fH29mbYsGHk5eU52r29vSksLCQgIICamhoqKirw8/MDYMeOHTz00EO8+OKLjfrcuXMnMTExAIwcOZLdu3dTX1/vqkMQEZFmuCxASktLCQwMdCybzWZKSkoabePp6cmuXbt45plnuHDhAgMHDgRg1KhRTJ8+HXd39xv26eHhgY+PD5WVla46BBERaYbL7sKy2+1N1plMpibrIiIiOHDgAOnp6aSkpJCWlnZLr+Pm5nwGdunic0t9i7QGd3c3x9fAQN82rkbEeS4LkKCgIA4dOuRYLi0txWw2O5arqqo4duyY46wjJiaGhISEZvs0m82Ul5fTtWtXGhoaqK6uxt/f3+maKiqqsdmaBptIW7JabY6vZWWWNq5G5O/c3EzNfvB22RDWgAEDKC4uprKyktraWvLz8xk8eLCj3W63k5iYyLlz5wDIzc2lX79+zfYZERFBdnY2AFu3bqV///54enq66hBERKQZLj0DSUhIIC4ujvr6esaNG0efPn2Ij49n1qxZhIaGsmDBAmbMmIHJZCI4OJj58+c32+fs2bNJTk4mOjoaX19flixZ4qryRUTkJkz2612suENpCEvaozmriim5UEvQfR1ZNOOpti5HxKHNhrBEROTOpgARERFDFCAiImKIAkRERAxRgIiIiCEKEBERMUQBIiIihihARETEEAWIiIgYogARERFDFCAiImKIyyZTFJEbs9vtfPF1GdsPnaW0qhYAS209l2rq8PP2auPqRJyjyRRF2kDW7m/4tOh0k/WB/vcw5/kn8Pfp0PpFifwvmkxRpJ35vyWW64YHQFnVFTbu/GvrFiRikAJEpJXtO/q3Zts/O17C1XprK1UjYpwCRKSVVVVfbba9wWrncm19K1UjYpwCRKSVBfp3bLa9g6c7vrqQLrcBBYhIKxvUpxsm043bB4R2xdNDP5rS/um7VKSVBXX2ZnLko9dtezDIh7GDf97KFYkYo9t4RdrIqe8vUvj5WQ5+VYrVZsfX25PFLw+gg6d7W5cmAtz8Nl49SCjSRoJ73Etwj3v59vwlSi7U4t3BQ+EhtxUNYYmIiCEKEBERMUQBIiIihrg0QHJycoiKimLo0KFkZmY2aS8oKCAmJobo6GiSk5Opq6sD4Ny5c0yZMoXhw4fz8ssvc/nyZQAOHjxIWFgYsbGxxMbGMmfOHFeWLyIizXBZgJSUlLB06VLWrVvH5s2b2bBhA6dOnXK019TUkJqaytq1a9myZQtXr15l06ZNAMyfP5/JkyeTl5dH7969+fDDDwE4evQoL730Eps3b2bz5s0sWrTIVeWLiMhNuCxAioqKCA8Px9/fH29vb4YNG0ZeXp6j3dvbm8LCQgICAqipqaGiogI/Pz/q6+s5ePAgw4YNA2DMmDGO/Y4ePcq+ffsYNWoUM2fO5Pz5864qX0REbsJlt/GWlpYSGBjoWDabzRw5cqTRNp6enuzatYukpCTMZjMDBw7kwoUL+Pj44OFxrbTAwEBKSkoA8PX1JTo6msjISD766CMSEhJYv3690zU1dz+zSFtxd3dzfA0M9G3jakSc57IAud7ziabrzN8QERHBgQMHSE9PJyUlhaSkpBvul5qa6lg3adIk0tLSsFgs+Po690OnBwmlPbJabY6vZWWWNq5G5O/a7O+BBAUFUV5e7lguLS3FbDY7lquqqti7d69jOSYmhhMnTtC5c2eqq6uxWq9NZ11WVobZbMZms7Fy5UrH+h/8cKYiIiKty2UBMmDAAIqLi6msrKS2tpb8/HwGDx7saLfb7SQmJnLu3DkAcnNz6devH56envTv35+tW7cCkJ2dzeDBg3Fzc6OgoIBt27Y51vft25eOHZuf2VRERFzDpXNh5eTksGrVKurr6xk3bhzx8fHEx8cza9YsQkND2b59OxkZGZhMJoKDg5k/fz6+vr58//33JCcnU1FRQbdu3UhPT+fee+/l5MmTzJ07F4vFQufOnVm8eDHdunVzuh4NYUl7NGdVMSUXagm6ryOLZjzV1uWIONxsCEuTKYq0MQWItFf6m+giIuISChARETFEASIiIoYoQERExBAFiIiIGKIAERERQxQgIiJiiAJEREQMUYCItLF7vDwafRW5XShARNrYqEE/o9eD/owa9LO2LkXklmgqExERuS5NZSIiIi6hABEREUMUICIiYogCREREDFGAiIiIIQoQEREx5K56csnNzdTWJYiI3DZu9jvzrnoOREREWo6GsERExBAFiIiIGKIAERERQxQgIiJiiAJEREQMUYCIiIghChARETFEASIiIoYoQERExBAFSDtw9uxZevfuTWxsbKN/58+fd8lrDRkypMX7labOnj1LSEgI8+bNa7T++PHjhISEkJWVdcN9hwwZwtmzZ11d4m3jRt+3ISEhN9znwIEDvPDCC64s66bq6up44YUXXPJzl5ycTFZWFiUlJcTHx193m+ben5ZwV82F1Z6ZzWY2b97c1mVIC/P392fPnj1YrVbc3d0B2Lp1K507d27jyqQ1HDx4kP79+7v0NYKCglizZo1LX+NGFCDtWHl5OfPmzeNvf/sbJpOJ119/nQEDBrB8+XLOnTvHiRMnqKio4Le//S379+/n8OHD9OrVi6VLl2K1WklJSeHkyZOUl5fzs5/9jBUrVjjVv7ScTp060atXLw4ePEh4eDgA+/btc7zPf/zjH9m8eTO1tbWYTCaWLVvGww8/7NjfarWyePFiPvvsM6xWK2PGjGHatGltcSjtVlZWFnv27OHixYucOXOGX/3qV6SkpDTa5ve//z3bt29n9erVTJ8+ndDQUD7//HMqKyt5++23iYiIoLy8nLfeeotz587h4eFBQkICvXv35rnnnmPv3r0ADBo0iDlz5hAVFcXq1asxmUxcuXKFkpISvvvuO77//nvGjx/Pyy+/DMDu3bsZPny4U/W++uqrjBw50rH9mDFjWLBgAZcvX2bp0qVcuXKFixcvkpiYyIgRIxz9nT17lri4OAoLCzl79iyJiYnU1NTQt29fF77r1yhA2onS0lJiY2MdyzExMfzlL39h7NixPPvss5SWljJ58mSys7MB+Prrr/n444/54osvmDp1Kjk5OTz00ENERUVx4sQJLBYLnp6ebNiwAZvNxtSpU9m1axePPfaY4zXeeeed6/bv4+PT2od/RxsxYgTbtm0jPDycI0eOEBISgt1up7q6msLCQv7whz9wzz33kJGRwbp165g7d65j348//hiATZs2UVdXx7/8y7/Qu3dvl3+qvd18+eWXfPrpp7i7uzN8+HAmTZrkaPvzn/9Mfn4+a9asoWPHjgDU19ezYcMGCgsLycjIICIiggULFhAeHs6LL77ImTNnmDRpEtnZ2XTv3p2vv/4ad3d3rFYrn332GVFRUezevZuUlBRyc3M5ceIEmZmZWCwWIiMjmTJlCn5+fhw+fJikpKQmw9HXqzc2NpacnByGDx/O6dOnuXr1Ko899hizZs1i4cKFPPzwwxQXF/Puu+82CpAfW7BgAWPGjGH8+PFkZ2ezYcMG173pKEDajesNYYWFhfHNN9/wwQcfANDQ0MCZM2cA+NWvfoWHhwfdu3cnMDCQ4OBg4Nrp7MWLFwkLC8Pf35/MzEy++eYbTp8+TU1NTaP+i4qKrtv/P/zDP7j6cO8qzzzzDMuWLcNms5Gbm8uIESPYunUrPj4+pKWlsWXLFk6fPs2ePXuavPfFxcUcP36c/fv3A1BTU8OJEyfuugBxc2t6udZut2MyXZtu/Je//KXjg88DDzzAxYsXgWsftObNm0d6ejre3t6OfQcNGgTAI488QlVVFQD79+9n4cKFjj769u3L4cOHiYiIoLi4GA8PD+Li4tiyZQsWi4WysjLHz11YWBheXl506dIFf39/LBYLly5donv37o6hyx+7Xr0/hFh1dTWffvopMTExALz//vv813/9F3l5eRw+fJjLly/f8H367LPPSEtLA+C5557j7bffdvIdNkYB0o7ZbDZ+//vf4+/vD0BJSQkBAQFs374dT09Px3YeHk3/G3fs2MEHH3xAXFwcY8aM4cKFC/zvmftv1L+0LB8fH3r16sXnn3/O/v37ef3119m6dSvnz59nwoQJPP/88wwePJiAgACOHz/eaF+r1UpiYiL/9E//BEBlZWWjX4R3Cz8/PywWS6N1FRUV3HvvvQB06NDBsd5kMjm+1zt16sS7777Lu+++y6BBgxzv3Q/b/xBAQJOfD7vdjtVqJSIighUrVuDl5cXs2bPJzc0lJyfHEUI3ev3du3czePDg6x7P9bb38vLi6aefprCwkLy8PFatWgXA5MmTCQsLIywsjKeeeoo33nij2ffqh+MwmUyNjs8VdBdWOxYeHs66desAOHXqFM899xy1tbVO7VtcXMyIESMYO3YsAQEBHDx4EKvV2mL9y60ZMWIEaWlp9O7d2xH43t7e9OzZk2nTptG3b19279593f+jjz/+mPr6ei5fvszkyZM5fPhwWxxCm/Lx8aFnz55s27bNsW7Dhg089dRTze7Xo0cPnn32WZ588knHmfaNhIeHs3HjRgDOnDnDF198weOPP85jjz3Gt99+y+nTp3n44YcJCwtj5cqVPP300832t3fvXgYOHOjcAf5/sbGxrF27lnvvvZcePXpQVVXF6dOnmT17NhEREezbt6/J98iPDRgwgE8++QSA/Px86urqbun1b5XOQNqxt99+m3nz5jlOZRcvXuz09Ynx48fzxhtvkJeXh5eXF48//niT20J/Sv9ya5555hneeustZs+e7Vjn6emJzWYjKioKLy8v+vTpw8mTJxvtN3HiRL777jtGjx5NQ0MDY8aMISwsrLXLbxfef/99UlJS+N3vfkd9fb3jFumdO3fedN+kpCRGjhzp+F6/nrfeeot58+Y5bq9euHAhZrMZgCeeeMLx4So8PJw//elPPPnkkzfsq66uDovFcstn9E888QQWi4WJEycC1+7iGz9+PNHR0fj4+PD4449z5cqVJsPRP5g3bx6JiYmsX7+e0NBQOnXqdEuvf6v0FwlFRMQQDWGJiIghChARETFEASIiIoYoQERExBAFiIiIGKIAEWmH8vLy2nwmWZGbUYCIiIghepBQ5CdYvXo1GzdupFOnTvTv358dO3aQl5fHkiVLHE///+IXv+Dtt9/Gx8eHIUOGMHr0aIqLizl//jwjRowgKSkJgIyMDHJycvD396dnz56O16irq2u2vz59+nDixAlee+01hg4d2lZvhdyFdAYiYtCePXvIyspi48aNZGVlOSa5W716Ne7u7mRlZfHJJ59gNptZsmSJY7+amhrWrVvH+vXr+eMf/8iZM2fYvn07+fn5ZGdns379eqqrqx3b36y/Rx55hNzcXIWHtDqdgYgYtGvXLoYPH46fnx8AU6ZMYf/+/ezcuROLxUJRURFwberwLl26OPZ79tlngWszJ3fp0oWLFy9SXFzM0KFDHVPJjB07lj/84Q8AN+3vbpuZV9oPBYiIQR4eHo1mcP1h2m6bzcabb75JREQEAJcvX+bq1auO7a43E+uPZ5D9cV/O9Hc3zs4r7YOGsEQMioiIID8/3zHN+A8zuQ4cOJDMzEzq6uqw2WzMnTuX9PT0ZvsaNGgQeXl5XLp0CZvN1uhvwxjpT6Q1KEBEDHrqqaf453/+ZyZMmMCYMWOwWCx07NiRV155hR49ejB69GiioqKw2+0kJyc321dERARjx45l7NixjB8/Hl9fX0ebkf5EWoNm4xUx6OjRo3z55ZfExcUBsHbtWg4fPsyyZcvatjCRVqIAETGourqaN998k2+++QaTyUS3bt1YsGABQUFBbV2aSKtQgIiIiCG6BiIiIoYoQERExBAFiIiIGKIAERERQxQgIiJiiAJEREQM+X8FXS1M/UJU+QAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 285 }, - { - "cell_type": "markdown", - "metadata": { - "id": "Boo771yFUyJ7" - }, - "source": [ - "3.) Next, calculate the balanced error rate and false negative rate difference of each of the inner models learned by this new `ExponentiatedGradient` classifier." - ] + "id": "ONJcedCWULXN", + "outputId": "1f0e82df-b12a-459b-9b77-13285ae44008" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "sns.pointplot(y=\"medicare\", x=____, data=df, ci=95, join=False);" + ], + "outputs": [], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 285 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "jws7w2z6RWUM" - }, - "outputs": [], - "source": [ - "# Save the inner predictors of the new model\n", - "predictors_exercise = expgrad_exercise.predictors_" - ] + "id": "jXtY7_xdULLe", + "outputId": "67948de6-24eb-4d61-9116-1d465534ee53" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "## Datasheets for datasets" + ], + "metadata": { + "id": "ZVSerRqDG3we" + } + }, + { + "cell_type": "markdown", + "source": [ + "The _datasheets_ practice was proposed by [Gebru et al. (2018)](https://arxiv.org/abs/1803.09010). A datasheet of a given dataset documents the motivation behind the dataset creation, the dataset composition, collection process, recommended uses and many other characteristics. In the words of Gebru et al., the goal is to\n", + "> facilitate better communication between dataset creators\n", + "> and dataset consumers, and encourage the machine learning\n", + "> community to prioritize transparency and accountability.\n", + "\n", + "In this section, we show how to fill in some of the sections of the datasheet for the dataset that we are using. The information is obtained directly from [Strack et al. (2014)](https://www.hindawi.com/journals/bmri/2014/781670/)." + ], + "metadata": { + "id": "ACLMWSAwGaLc" + } + }, + { + "cell_type": "markdown", + "source": [ + "### Example sections of a datasheet [OPTIONAL SECTION]" + ], + "metadata": { + "id": "fcc4cUMhZnCb" + } + }, + { + "cell_type": "markdown", + "source": [ + "**For what purpose was the dataset created?** *Was there a specific task in mind? Was there a specific gap that needed to be filled?*" + ], + "metadata": { + "id": "0TIPJIhX1IKJ" + } + }, + { + "cell_type": "markdown", + "source": [ + "In the words of the dataset authors:\n", + "> [...] the management of hyperglycemia in the hospitalized patient has a significant bearing on outcome, in terms of both morbidity and mortality. This recognition has led to the development of formalized protocols in the intensive care unit (ICU) setting [...] However, the same cannot be said for most non-ICU inpatient admissions. [...] there are few national assessments of diabetes care in the hospitalized patient which could serve as a baseline for change [in the non-ICU protocols]. The present analysis of a large clinical database was undertaken to examine historical patterns of diabetes care in patients with diabetes admitted to a US hospital and to inform future directions which might lead to improvements in patient safety." + ], + "metadata": { + "id": "sBNKbKJQJbmf" + } + }, + { + "cell_type": "markdown", + "source": [ + "**Who created the dataset (e.g., which team) and on behalf of which entity?**" + ], + "metadata": { + "id": "iA-XRu0o1ErL" + } + }, + { + "cell_type": "markdown", + "source": [ + "The dataset was created by [Strack et al. (2014)](https://www.hindawi.com/journals/bmri/2014/781670/): a team of researchers from a variety of disciplines, ranging from computer science to public health, from three institutions (Virginia Commonwealth University, University of Cordoba, and Polish Academy of Sciences)." + ], + "metadata": { + "id": "zQSHxl26LQkt" + } + }, + { + "cell_type": "markdown", + "source": [ + "#### **Composition**" + ], + "metadata": { + "id": "0Qx32mSJG3zP" + } + }, + { + "cell_type": "markdown", + "source": [ + "**What do the instances that comprise the dataset represent?**\n", + "\n" + ], + "metadata": { + "id": "8RS2V8001F3E" + } + }, + { + "cell_type": "markdown", + "source": [ + "Each instance in this dataset represents a hospital admission for diabetic patient (diabetes was entered as a possible diagnosis for the patient) whose hospital stay lasted between one to fourteen days." + ], + "metadata": { + "id": "JPy_TXp_1Gub" + } + }, + { + "cell_type": "markdown", + "source": [ + "**Is any information missing from individual instances?**" + ], + "metadata": { + "id": "eOb0FPeOJqxm" + } + }, + { + "cell_type": "markdown", + "source": [ + "The features `Payer Code` and `Medical Specialty` have 40,255 and 49,947 missing values, respectively. For `Payer Code`, these missing values are reflected in the category *Unknown*. For `Medical Specialty`, these missing values are reflecting in the category *Missing*. \n", + "\n", + "For our demographic features, we are missing the `Gender` information for three patients in the dataset. These three records were dropped from our final dataset. Regarding `Race`, the 2,271 missing values were recoded into the `Unknown` race category. \n", + "\n" + ], + "metadata": { + "id": "4vlZWeQjJq8w" + } + }, + { + "cell_type": "markdown", + "source": [ + "**Does the dataset identify any subpopulations (e.g., by age, gender)?**" + ], + "metadata": { + "id": "Uh0lLV6mJrSp" + } + }, + { + "cell_type": "markdown", + "source": [ + "Patients are identified by gender, age group, and race. \n", + "\n", + "For gender, patients are identified as Male, Female, or Unknown. There were only three instances where the patient gender is *Unknown*, so these records were removed from our dataset.\n", + "\n", + "Gender | Count| Percentage\n", + "------ | ------|----------\n", + "Male | 47055 | 46.2%\n", + "Female | 54708 | 53.7% \n", + "\n", + "\n", + "\n", + "For age group, patients are binned into three age buckets: *30 years or younger*, *30-60 years*, *Older than 60 years*.\n", + "\n", + "Age Group |Count| Percentage\n", + "------ | ------|----------\n", + "30 years or younger | 2509 | 2.4%\n", + "30-60 years | 30716 | 30.2%\n", + "Older than 60 years | 68538 | 67.4% \n", + "\n", + "\n", + "For race, patients are identified as *AfricanAmerican*, *Caucasian*, and *Other*. For individuals whose race information was not collected during hospital admission, their race is listed as *Unknown*.\n", + "\n", + "Race | Count| Percentage\n", + "------ | ------|----------\n", + "Caucasian | 76099 | 74.8%\n", + "AfricanAmerican | 19210 | 18.9% \n", + "Other | 4183 | 4.1%\n", + "Unknown | 2271 | 2.2%" + ], + "metadata": { + "id": "U-ZnQfibJrcQ" + } + }, + { + "cell_type": "markdown", + "source": [ + "#### **Preprocessing**" + ], + "metadata": { + "id": "yzXw0egqG4J4" + } + }, + { + "cell_type": "markdown", + "source": [ + "**Was any preprocessing/cleaning/labeling of the data done?**" + ], + "metadata": { + "id": "rGfxGcI21Fyj" + } + }, + { + "cell_type": "markdown", + "source": [ + "For the `race` feature, the categories of *Asian* and *Hispanic* and *Other* were merged into the *Other* category. The `age` feature was bucketed into 30-year intervals (*30 years and below*, *30 to 60 years*, and *Over 60 years*). The `discharge_disposition_id` was binarized into a boolean outcome on whether an patient was discharged to home.\n", + "\n", + "The full preprocessing code is provided in the file `preprocess.py` of the tutorial [GitHub repository](https://github.com/fairlearn/talks/blob/main/2021_scipy_tutorial/).\n", + "\n", + "\n" + ], + "metadata": { + "id": "5jO4Pf911GrL" + } + }, + { + "cell_type": "markdown", + "source": [ + "#### **Uses**\n", + "\n" + ], + "metadata": { + "id": "C8b5nXfPIA7a" + } + }, + { + "cell_type": "markdown", + "source": [ + "**Has the dataset been used for any tasks already?** " + ], + "metadata": { + "id": "_YH8evvN1HX2" + } + }, + { + "cell_type": "markdown", + "source": [ + "This dataset has been used by [Strack et al. (2014)](https://www.hindawi.com/journals/bmri/2014/781670/) to model the relationship between patient readmission and HbA1c measurement during admission, based on primary medical diagnosis.\n", + "\n", + "The dataset is publicly available through the UCI Machine Learning Repository and, as of May 2021, has received over 350,000 views." + ], + "metadata": { + "id": "H8RW1LKW1Hbg" + } + }, + { + "cell_type": "markdown", + "source": [ + "# Training the initial model" + ], + "metadata": { + "id": "HB7zfhA1UKiW" + } + }, + { + "cell_type": "markdown", + "source": [ + "We next train a classification model to predict our target variable (readmission within 30 days) while optimizing balanced accuracy.\n", + "\n", + "What kind of model should we train? Deep neural nets? Random forests? Logistic regression?\n", + "\n", + "There are a variety of considerations. We highlight two:\n", + "\n", + "* **Interpretability.** Interpretability is tightly linked with questions of fairness. There are several reasons why it is important to have interpretable models that are open to the stakeholder scrutiny:\n", + " * It allows discovery of fairness issues that were not discovered by the data science team.\n", + " * It provides a path toward recourse for those that are affected by the model.\n", + " * It allows for a *face validity* check, a \"sniff test\", by experts to verify that the model \"makes sense\" (at the face value). While this step is subjective, it is really important when the model is applied to different populations than those on which the assessment was conducted.\n", + "\n", + "* **Model expressiveness.** How well can the model separate positive examples from negative examples? How well can it do so given the available dataset size? Can it do so across all groups or does it need to trade off performance on one group against performance on another group?\n", + "\n", + "Some additional considerations are training time (this impacts the ability to iterate), familiarity (this impacts the ability to fine tune and debug), and carbon footprint (this impacts the Earth climate both directly and indirectly by normalizing unnecessarily heavy workloads)." + ], + "metadata": { + "id": "0f7jZOzGX24Z" + } + }, + { + "cell_type": "markdown", + "source": [ + "### Decision point: Model type" + ], + "metadata": { + "id": "7hbsqXap9Mzp" + } + }, + { + "cell_type": "markdown", + "source": [ + "We will use a logistic regression model. Our reasoning:\n", + "\n", + "* **Interpretability**. Logistic models over a small number of variables (as used here) are highly interpretable in the sense that stakeholders can simulate them easily.\n", + "\n", + "* **Model expressiveness**. Logistic regression predictions are described by a linear weighting of the feature values. The concern might be that this is too simple. The previous work by [Strack et al. (2014)](https://www.hindawi.com/journals/bmri/2014/781670/), which also used a logistic model to predict readmission rates concluded that a slightly more expressive model might be useful (their analysis uncovered 8 pairwise interactions that were significant, see their Table 5)." + ], + "metadata": { + "id": "iftAwdfoVDM0" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Prepare training and test datasets" + ], + "metadata": { + "id": "52wWLOFoXkho" + } + }, + { + "cell_type": "markdown", + "source": [ + "As we mentioned in the task definition, our target variable is **readmission within 30 days**, and our sensitive feature for the purposes of fairness assessment is **race**.\n" + ], + "metadata": { + "id": "vCwtDyx8yqSG" + } + }, + { + "cell_type": "code", + "execution_count": 34, + "source": [ + "target_variable = \"readmit_30_days\"\n", + "demographic = [\"race\", \"gender\"]\n", + "sensitive = [\"race\"]\n", + "# If multiple sensitive features are chosen, the rest of the script considers intersectional groups." + ], + "outputs": [], + "metadata": { + "id": "WpiSRj2JyqSH" + } + }, + { + "cell_type": "code", + "execution_count": 35, + "source": [ + "Y, A = df.loc[:, target_variable], df.loc[:, sensitive]" + ], + "outputs": [], + "metadata": { + "id": "uaNqDyqvi1QE" + } + }, + { + "cell_type": "markdown", + "source": [ + "We next drop the features that we don't want to use in our model and expand the categorical features into 0/1 indicators (\"dummies\")." + ], + "metadata": { + "id": "niu49A9YXQmf" + } + }, + { + "cell_type": "code", + "execution_count": 36, + "source": [ + "X = pd.get_dummies(df.drop(columns=[\n", + " \"race\",\n", + " \"race_all\",\n", + " \"discharge_disposition_id\",\n", + " \"readmitted\",\n", + " \"readmit_binary\",\n", + " \"readmit_30_days\"\n", + "]))" + ], + "outputs": [], + "metadata": { + "id": "JXyRuCsri1cY" + } + }, + { + "cell_type": "code", + "execution_count": 37, + "source": [ + "X.head() # sanity check" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
time_in_hospitalnum_lab_proceduresnum_proceduresnum_medicationsnumber_diagnosesmedicaremedicaidhad_emergencyhad_inpatient_dayshad_outpatient_days...A1Cresult_NoneA1Cresult_Norminsulin_Downinsulin_Noinsulin_Steadyinsulin_Upchange_Chchange_NodiabetesMed_NodiabetesMed_Yes
0141011FalseFalseFalseFalseFalse...1001000110
13590189FalseFalseFalseFalseFalse...1000011001
22115136FalseFalseFalseTrueTrue...1001000101
32441167FalseFalseFalseFalseFalse...1000011001
4151085FalseFalseFalseFalseFalse...1000101001
\n", + "

5 rows × 46 columns

\n", + "
" + ], + "text/plain": [ + " time_in_hospital num_lab_procedures num_procedures num_medications \\\n", + "0 1 41 0 1 \n", + "1 3 59 0 18 \n", + "2 2 11 5 13 \n", + "3 2 44 1 16 \n", + "4 1 51 0 8 \n", + "\n", + " number_diagnoses medicare medicaid had_emergency had_inpatient_days \\\n", + "0 1 False False False False \n", + "1 9 False False False False \n", + "2 6 False False False True \n", + "3 7 False False False False \n", + "4 5 False False False False \n", + "\n", + " had_outpatient_days ... A1Cresult_None A1Cresult_Norm insulin_Down \\\n", + "0 False ... 1 0 0 \n", + "1 False ... 1 0 0 \n", + "2 True ... 1 0 0 \n", + "3 False ... 1 0 0 \n", + "4 False ... 1 0 0 \n", + "\n", + " insulin_No insulin_Steady insulin_Up change_Ch change_No \\\n", + "0 1 0 0 0 1 \n", + "1 0 0 1 1 0 \n", + "2 1 0 0 0 1 \n", + "3 0 0 1 1 0 \n", + "4 0 1 0 1 0 \n", + "\n", + " diabetesMed_No diabetesMed_Yes \n", + "0 1 0 \n", + "1 0 1 \n", + "2 0 1 \n", + "3 0 1 \n", + "4 0 1 \n", + "\n", + "[5 rows x 46 columns]" + ] + }, + "metadata": {}, + "execution_count": 37 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 274 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "aVTqrrSRAtGc" - }, - "outputs": [], - "source": [ - "# Compute the balanced error rate and false negative rate difference for each of the predictors on the test data.\r\n", - "balanced_error_exercise = [(1 - ______(Y_test, pred.predict(X_test))) for pred in predictors_exercise]\r\n", - "false_neg_exercise = [(______(Y_test, pred.predict(X_test), sensitive_features=_____)) for pred in predictors_exercise]" - ] + "id": "kAA0sIhQUWFa", + "outputId": "73d62224-bf2e-40d2-a29a-8909e26d3cf9" + } + }, + { + "cell_type": "markdown", + "source": [ + "We split our data into a training and test portion. The test portion will be used to evaluate our performance metric (i.e., balanced accuracy), but also for fairness assessment. The split is half/half for training and test to ensure that we have sufficient sample sizes for fairness assessment." + ], + "metadata": { + "id": "ATzi8cKCD7V3" + } + }, + { + "cell_type": "code", + "execution_count": 38, + "source": [ + "random_seed = 445\n", + "np.random.seed(random_seed)" + ], + "outputs": [], + "metadata": { + "id": "-wpnURazmJ4-" + } + }, + { + "cell_type": "code", + "execution_count": 39, + "source": [ + "X_train, X_test, Y_train, Y_test, A_train, A_test, df_train, df_test = train_test_split(\n", + " X,\n", + " Y,\n", + " A,\n", + " df,\n", + " test_size=0.50,\n", + " stratify=Y,\n", + " random_state=random_seed\n", + ")" + ], + "outputs": [], + "metadata": { + "id": "xgl_b-CUl7TW" + } + }, + { + "cell_type": "markdown", + "source": [ + "Our performance metric is **balanced accuracy**, so for the purposes of training (but not evaluation!) we will resample the data set, so that it has the same number of positive and negative examples. This means that we can use estimators that optimize standard accuracy (although some estimators allow the use us importance weights).\n" + ], + "metadata": { + "id": "eSMbXR9iVqr8" + } + }, + { + "cell_type": "markdown", + "source": [ + "Because we are downsampling the number of negative examples, we create a training dataset with a significantly lower number of data points. For more complex machine learning models, this lower number of training data points may affect the model's accuracy." + ], + "metadata": { + "id": "nPNQpb2ZN1ku" + } + }, + { + "cell_type": "code", + "execution_count": 40, + "source": [ + "def resample_dataset(X_train, Y_train, A_train):\n", + "\n", + " negative_ids = Y_train[Y_train == 0].index\n", + " positive_ids = Y_train[Y_train == 1].index\n", + " balanced_ids = positive_ids.union(np.random.choice(a=negative_ids, size=len(positive_ids)))\n", + "\n", + " X_train = X_train.loc[balanced_ids, :]\n", + " Y_train = Y_train.loc[balanced_ids]\n", + " A_train = A_train.loc[balanced_ids, :]\n", + " return X_train, Y_train, A_train" + ], + "outputs": [], + "metadata": { + "id": "L1aVgzyFNa4B" + } + }, + { + "cell_type": "code", + "execution_count": 41, + "source": [ + "X_train_bal, Y_train_bal, A_train_bal = resample_dataset(X_train, Y_train, A_train)" + ], + "outputs": [], + "metadata": { + "id": "6Ogw-r3DQsds" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Save descriptive statistics of training and test data" + ], + "metadata": { + "id": "fRddJS7XXv5n" + } + }, + { + "cell_type": "markdown", + "source": [ + "We next evaluate and save descriptive statistics of the training dataset. These will be provided as part of _model cards_ to document our training." + ], + "metadata": { + "id": "hZ-T4lGxX0IQ" + } + }, + { + "cell_type": "code", + "execution_count": 42, + "source": [ + "sns.countplot(x=\"race\", data=A_train_bal)\n", + "plt.title(\"Sensitive Attributes for Training Dataset\")\n", + "sensitive_train = figure_to_base64str(plt)" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEXCAYAAABRWhj0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA0n0lEQVR4nO3deViVdf7/8eeBg6gDaehBHb7mtKiJljQyKS2QaQEK7n4zTNJRSy0tmzREzCVLM8Ytg7KcmTLNJRGXCFtUMnEhpjS/aVqpuQ2rC6Bs59y/P/x5RhLxtjhi+npcl9fl+Zx7ed8397lf9/25z7lvi2EYBiIiIia41XQBIiLy+6HQEBER0xQaIiJimkJDRERMU2iIiIhpCg0RETFNoXEN++abbxgwYACRkZFEREQwZMgQ9u3b55J5DR06lB9++AGAv/71r+Tn51/QXh0WLlxIy5Yt+eabbyq0b9y4kTlz5gBQUFBAdHT0RafRvXt3Tp06RVJSEk8++eRl1zBv3jw+++yzyx6vMoWFhfTr14+uXbuybt26XzWNUaNG0b17d7p3707Lli2JjIyke/fuDBgw4LKmc269VGX8+PGkp6f/qjp/KSkpiXbt2jlrj4yMZMCAAezcudPU+OdvZ9Xp0KFDjBw5stqne80w5JpUUlJi3H333cauXbucbcnJyUZISIhRXl7u0nm3aNHCyMvLc8m0u3TpYvztb38znn322Qrtc+fONSZPnmwYhmEcOnTICAgIuOS0VqxYYTzxxBOXXcNjjz1mfPzxx5c9XmW2b99udO7cuVqmZRiuXffVrbL1v3nzZqN9+/bG4cOHLzm+q5Z169atRteuXat9utcKa02HlrjGmTNnKCgo4PTp0862bt264eXlhd1ux93dnfXr15OYmEhZWRm1a9fmhRde4K677uL111/nyJEj5OTkcOTIEXx8fJg1axaNGjVi8eLFLFmyBA8PDzw9PZkyZQq33XYbDz74IHPmzGHx4sUAPP7448yfP5/+/fszZ84c/vWvf+Hv78/gwYMB+OCDD9i2bRuzZ8++aB2/tG3bNk6ePMmYMWN46KGHOHbsGE2aNGHHjh0sWbIEu92Ot7c3//73vykuLqZ79+4kJSXRtm1bOnXqxJ49e4iPj6dPnz5s2bIFgJycHAYPHkx2djZ+fn689NJL2Gw2BgwYQP/+/QkLCwNwvs7Ly2PXrl3MmDEDd3d3QkJCiI+PJyMjA7vdjr+/P3FxcXh5eV10XZ3z008/ERsbS1ZWFt27d2fp0qV8+eWXzJs3D7vdjpeXF+PGjePOO+/k9ddf55tvviE7O5uWLVsSHx9vajuIiYnhxIkTHDp0iAceeIA+ffowZcoUTp8+TXZ2NrfffjuzZ8/G09OTli1bsmXLFjZu3Minn36Km5sbBw8exMPDg1dffZUWLVo410ObNm0YOHAgISEh7Nixg5MnTzJ69Gi6dOnCmTNnmDhxIjt27MDb29u5zNOnT79kvffccw8PPfQQH3zwAc8//zwbNmzgrbfeorS0lPz8fHr06MGzzz7LuHHjKmxne/bsqXS4oqIixo0bx8GDB3Fzc6N169ZMmTIFNze3Sre7O++8k7i4OLKyshg8eDALFiwwtZ6vKzWdWuI6//jHP4w777zTePDBB43nn3/eWL58uXH69GnDMAxj//79RkREhJGfn28YhmHs3bvXuPfee42ioiJj7ty5RqdOnYyCggLDMAzjySefNObMmWOUl5cbrVu3NrKysgzDMIyVK1caS5YsMQzDMDp27Gjs3LnTMIyKR4Dn2rds2WJEREQ4a+vTp4+xefPmKuv4pWeeecaYPn26YRiGMXToUGPGjBnO96o602jRooWxcuXKCq/z8vKMFStWGAEBAcaBAwcMwzCMv//978YzzzxjGMaFZxPnvz7//6+//roxffp0w+FwOKcxceLEKtfV+c4/qv3hhx+Me+65x/j5558NwzCM9PR049577zUKCgqMuXPnGqGhoUZZWdkF0zjfL4++X3jhBePxxx93vp4+fbqRnJxsGIZhlJaWGhEREUZqauoF66Vdu3bGsWPHDMMwjClTphhjx46tsOyHDh0yWrRoYaxfv94wDMNITU01HnjgAcMwDCM+Pt547rnnDLvdbhQUFBiRkZHGCy+8cEGtFzvTe//9942hQ4caDofDeOyxx4z9+/cbhmEY//nPf4xWrVo5l+9cvVUNt3LlSuOvf/2rYRiGUV5ebowfP944cOBAldudzjSqpjONa9igQYPo27cvGRkZZGRk8Pbbb/P222/z4YcfsnnzZrKzsxk4cKBzeIvFws8//wzA3XffjZeXFwD+/v6cPHkSd3d3wsLC6NevHw888AD33nsvkZGRpmpp3749JSUlfPvtt9SpU4f8/HyCgoJYvHjxReu4/fbbnW05OTl89tlnrFixAoAePXowadIknnrqKerWrXvJ+QcGBlbafs8999CsWTMA+vTpQ58+fUwtzzkbN26koKDA2c9fVlZGgwYNftW62rp1Kx06dKBp06YABAUF4ePjw65duwAICAjAar38j2y7du2c/x8zZgybN2/m7bff5sCBA2RnZ1c4Gz2ndevWNG7cGDj79//0008vGMbDw4OQkBDnMCdOnAAgLS2NcePG4ebmhpeXFz179uT777+/rJpr166NxWLhzTffZOPGjaxdu5Yff/wRwzA4c+ZMhWGrGq5du3bMmjWLAQMGcM899/D444/TrFkzFi1aVOX2Lxen0LhGZWZm8vXXXzNkyBA6duxIx44dee6554iMjGTz5s04HA6CgoKYPXu2c5xjx47h6+vLp59+Su3atZ3tFosF4//foiw+Pp69e/eSnp7uDKDExMRL1mOxWOjTpw+rVq3Cw8ODPn36YLFYqqzjfMuXLwdg+PDhADgcDgoLC1m5ciX9+/e/5PwvFizu7u7O/xuGUWGnbJx3W7aysrJKx3c4HMTGxjp3nkVFRZSUlACXv66MSm4DZxgG5eXlVS7DpZw/3nPPPYfdbic8PJwHHniAY8eOVTrfi/39z+fh4YGbm5tzmHOsVmuF4c8NY9auXbto0aIFp0+fpmfPnnTu3JnAwEB69+7NZ599dkEtVQ3XtGlTPv30U7Zt28bWrVsZNGgQcXFxVW53X3311WXVe73Rt6euUT4+PiQmJlb4AOTk5HDmzBlatGhBhw4d2Lx5Mz/++CNw9uiwW7duzh1eZfLz8wkJCaF+/foMHDiQZ599ttIjSHd3d+eO7nw9e/Zk/fr1rFu3jl69egGYqsNut7Ns2TImT57M+vXrWb9+PRs3buTJJ5/kvffewzCMCvO0Wq3Y7fZKd3S/tG3bNo4ePQqcvc4SHBzsXH/njvB//vnnCst5/rzuu+8+Fi1aRGlpKQ6HgwkTJjBz5kzT6+p859bFoUOHANiyZQvHjh2jbdu2l1wOs7788kueeuopunTpgsViYceOHdjt9mqbPkBISAgrVqzA4XBw5swZ1q5dWyFUqpKWlsbGjRt55JFHOHjwIIWFhTz77LM8+OCDbN++3bme4b9/h6qGW7x4MePGjeO+++5jzJgx3Hfffezbt6/K7c7d3f2iBwmiM41r1s0338wbb7zBrFmz+M9//oOnpyfe3t5MmTKFW265BYApU6bw3HPPOY+wExMTqzya9fHxYfjw4QwcOJDatWvj7u7O1KlTLxjuoYceIioqioSEhArtNpsNf39/ysvLadSoEQDNmze/ZB0bNmzA4XBc0L0zcOBA3nvvPdLS0ggKCmLkyJF4eHgQGxuLv78/4eHhfPDBB1WupxYtWhAbG0tubi633HILU6ZMAc6e0cTExJCWlsYtt9xSoXurY8eOvPrqq5SVlTFixAheffVVevbsid1up1WrVsTExODl5WVqXZ3vtttuY+LEiTz99NPY7XZq167Nm2++ibe3d5XjXY7Ro0fz1FNPUa9ePerUqcNf/vKXau+SefLJJ5kyZQqRkZF4e3vToEGDCmcu5/vqq6/o3r07cPZsxdfXlwULFmCz2WjQoAEPPPAA4eHh3HDDDdx0003cdtttHDx4kJtuusm5nc2bN++iw/Xo0YPt27fTpUsX6tSpwx//+Eeio6OpV6/eRbe75s2b4+7uTp8+fVi+fLnpwLteWAwzh2MiIiZ99NFHeHl5ERISgsPhYOTIkdx7771ERUXVdGlSDRQaIlKt9u7dy4svvsiZM2coKyujffv2xMbG4uHhUdOlSTVQaIiIiGm6EC4iIqYpNERExDSFhoiImKbQEBER067532kcP16Ew6Fr/SIiZri5Wbjxxj9c9P1rPjQcDkOhISJSTdQ9JSIipik0RETENIWGiIiYptAQERHTFBoiImKaQkNERExTaIiIiGnX/O805Mq5sV4trLU8a7qMq0J5aQnHT5bWdBki1U6hIdXGWsuTzBlDarqMq0K7se8ACg259qh7SkRETFNoiIiIaQoNERExTaEhIiKmKTRERMQ0hYaIiJim0BAREdMUGiIiYppLQ2PVqlV07dqVrl278uqrrwKwe/duevfuTWhoKOPHj6e8vByAo0eP0r9/f8LCwhg+fDhFRUUAnDp1iieeeILw8HD69+9PTk6OK0sWEZEquCw0zpw5w8svv8zChQtZtWoVX331Fenp6YwZM4YJEyawbt06DMNg2bJlAEyePJmoqChSU1Np06YNCQkJAMyePZvAwEA+/vhj+vbty8svv+yqkkVE5BJcFhp2ux2Hw8GZM2coLy+nvLwcq9VKcXExAQEBAPTq1YvU1FTKysrIyMggNDS0QjvAxo0biYyMBCAiIoIvvviCsrIyV5UtIiJVcNm9p7y8vHjmmWcIDw+ndu3a3H333Xh4eGCz2ZzD2Gw2srKyOH78OF5eXlit1grtANnZ2c5xrFYrXl5e5Ofn06hRI1eVLiIiF+Gy0NizZw8rVqxgw4YNeHt78/zzz7N58+YLhrNYLBiGUWn7xbi5mT9BatDAy/SwItXJZvOu6RJEqp3LQuPLL78kKCiIBg0aAGe7nBYsWEBubq5zmJycHHx9ffHx8aGwsBC73Y67u7uzHcDX15fc3FwaN25MeXk5hYWF1K9f33QdeXmFOBwXhpJUP+0kK8rJKajpEkQum5ubpcqDbZdd07j99ttJT0/n9OnTGIbB+vXrufvuu/H09CQzMxOA5ORkgoOD8fDwIDAwkJSUlArtACEhISQnJwOQkpJCYGAgHh4eripbRESqYDEq6xuqJvPnzycpKQkPDw/uuOMOJk6cyP79+4mLi6OoqAh/f3+mTZtGrVq1OHLkCDExMeTl5dGkSRNmzpxJvXr1OHHiBDExMRw6dAhvb2/i4+P5n//5H9M16EzjyrHZvPU8jf+v3dh3dKYhv0uXOtNwaWhcDRQaV45C478UGvJ7VWPdUyIicu1RaIiIiGkKDRERMU2hISIipik0RETENIWGiIiYptAQERHTFBoiImKaQkNERExTaIiIiGkKDRERMU2hISIipik0RETENIWGiIiYptAQERHTXPa41+XLl/P+++87Xx8+fJju3bvTuXNnpk2bRklJCeHh4YwePRqA3bt3ExcXR2FhIYGBgUyePBmr1crRo0cZM2YMeXl53HzzzcTHx/OHP/zBVWWLiEgVXHam0bdvX1atWsWqVauIj4+nQYMGDB06lNjYWBISEkhJSWHXrl2kpaUBMGbMGCZMmMC6deswDINly5YBMHnyZKKiokhNTaVNmzYkJCS4qmQREbmEK9I9NWnSJEaPHs2hQ4do1qwZTZs2xWq1EhkZSWpqKkeOHKG4uJiAgAAAevXqRWpqKmVlZWRkZBAaGlqhXUREaobLQyM9PZ3i4mLCw8PJzs7GZrM53/P19SUrK+uCdpvNRlZWFsePH8fLywur1VqhXUREaobLrmmcs2TJEgYNGgRAZY8jt1gsl91+Oap61q2IK9ls3jVdgki1c2lolJaWkpGRwfTp0wFo1KgRubm5zvezs7Px9fW9oD0nJwdfX198fHwoLCzEbrfj7u7ubL8ceXmFOBwXho9UP+0kK8rJKajpEkQum5ubpcqDbZd2T33//ff86U9/om7dugC0bduW/fv3c/DgQex2O2vXriU4OBg/Pz88PT3JzMwEIDk5meDgYDw8PAgMDCQlJaVCu4iI1AyXnmkcOnSIxo0bO197enoyffp0Ro4cSUlJCSEhIYSFhQEQHx9PXFwcRUVF+Pv7Ex0dDcDEiROJiYkhMTGRJk2aMHPmTFeWLCIiVbAYlV04uIaoe+rKsdm8yZwxpKbLuCq0G/uOuqfkd6lGu6dEROTaotAQERHTFBoiImKaQkNERExTaIiIiGkKDRERMU2hISIipik0RETENIWGiIiYptAQERHTFBoiImKaQkNERExTaIiIiGkKDRERMU2hISIipik0RETENJeGxvr16+nVqxdhYWFMnToVgPT0dCIjI3n44YeZNWuWc9jdu3fTu3dvQkNDGT9+POXl5QAcPXqU/v37ExYWxvDhwykqKnJlySIiUgWXhcahQ4eYOHEiCQkJrFmzhu+++460tDRiY2NJSEggJSWFXbt2kZaWBsCYMWOYMGEC69atwzAMli1bBsDkyZOJiooiNTWVNm3akJCQ4KqSRUTkElwWGp9++ildunShcePGeHh4MGvWLOrUqUOzZs1o2rQpVquVyMhIUlNTOXLkCMXFxQQEBADQq1cvUlNTKSsrIyMjg9DQ0ArtIiJSM6yumvDBgwfx8PBg8ODB5OTk0LFjR5o3b47NZnMO4+vrS1ZWFtnZ2RXabTYbWVlZHD9+HC8vL6xWa4X2y1HVs25FXMlm867pEkSqnctCw26389VXX7Fw4ULq1q3LiBEjqFOnzgXDWSwWDMO4rPbLkZdXiMNx4XSk+mknWVFOTkFNlyBy2dzcLFUebLssNBo2bEhQUBA+Pj4AdOrUidTUVNzd3Z3DZGdn4+vrS6NGjcjNzXW25+Tk4Ovri4+PD4WFhdjtdtzd3Z3tIiJSM1x2TaNjx458+eWXnDp1CrvdzqZNmwgLC2P//v0cPHgQu93O2rVrCQ4Oxs/PD09PTzIzMwFITk4mODgYDw8PAgMDSUlJqdAuIiI1w2VnGm3btmXIkCFERUVRVlbGvffey6OPPsott9zCyJEjKSkpISQkhLCwMADi4+OJi4ujqKgIf39/oqOjAZg4cSIxMTEkJibSpEkTZs6c6aqSRUTkEixGZRcOriG6pnHl2GzeZM4YUtNlXBXajX1H1zTkd+lS1zT0i3ARETFNoSEiIqYpNERExDSFhoiImKbQEBER0xQaIiJimkJDRERMU2iIiIhpCg0RETFNoSEiIqYpNERExDSFhoiImKbQEBER0xQaIiJimkJDRERMc9lDmACio6PJy8vDaj07mylTpvDzzz+TmJhIWVkZAwcOpH///gCkp6czbdo0SkpKCA8PZ/To0QDs3r2buLg4CgsLCQwMZPLkyc7piYjIleWyMw3DMPjpp59YtWqV81/jxo2ZNWsWixcvZtWqVSxdupQffviB4uJiYmNjSUhIICUlhV27dpGWlgbAmDFjmDBhAuvWrcMwDJYtW+aqkkVE5BJcFho//fQTFouFoUOH0q1bN95//33S09Pp0KED9evXp27duoSGhpKamsrOnTtp1qwZTZs2xWq1EhkZSWpqKkeOHKG4uJiAgAAAevXqRWpqqqtKFhGRSzDVz5OVlUWjRo0qtP3www/cdtttFx3n1KlTBAUFMWnSJIqLi4mOjiY8PBybzeYcxtfXl507d5KdnX1Be1ZW1gXtNpuNrKws0wsHVPnYQhFXstm8a7oEkWpXZWicOHECgKFDh7Jw4ULOPU68vLycESNG8Mknn1x03Lvuuou77roLgLp169KnTx+mTZvGsGHDKgxnsVio7DHlVbVfDj0j/MrRTrIiPSNcfo8u9YzwKkPjb3/7G5s3bwagffv2/x3JaqVz585Vzvirr76irKyMoKAg4Ow1Dj8/P3Jzc53DZGdn4+vrS6NGjUy15+Tk4OvrW+V8RUTEdaq8prFgwQL27NlDz5492bNnj/Pfrl27mD17dpUTLigoYMaMGZSUlFBYWMjKlSt57bXX2LJlC/n5+Zw5c4ZPPvmE4OBg2rZty/79+zl48CB2u521a9cSHByMn58fnp6eZGZmApCcnExwcHC1LbyIiFweU9c0pk2bxpEjRzh58mSFLqPWrVtfdJyOHTuyY8cOevTogcPhICoqinbt2jF69Giio6MpKyujT58+3HnnnQBMnz6dkSNHUlJSQkhICGFhYQDEx8cTFxdHUVER/v7+REdH/5blFRGR38BiVHbh4Bfi4+NZuHAhDRo0+O+IFguff/65S4urDrqmceXYbN5kzhhS02VcFdqNfUfXNOR36Tdd0zgnJSWFTz755IJvUImIyPXF1O80mjRposAQERFzZxpBQUHMmDGDTp06Ubt2bWd7Vdc0RETk2mMqNJKSkgAq/Br793JNQ0REqo+p0Fi/fr2r6xARkd8BU6Hxz3/+s9L2QYMGVWsxIiJydTMVGnv37nX+v7S0lMzMzAq/EBcRkeuD6R/3nS8/P5+xY8e6pCAREbl6/apbo/v4+HDkyJHqrkVERK5yl31NwzAMdu3aVeHX4SIicn247GsacPbHfuqeEhG5/lzWNY0jR45QXl5Os2bNXFqUiIhcnUyFxsGDBxkxYgTZ2dk4HA5uvPFG3nrrLW699VZX1yciIlcRUxfCp0yZwpAhQ8jIyCAzM5Phw4czefJkV9cmIiJXGVOhkZeXR8+ePZ2ve/fuzfHjx11WlIiIXJ1MhYbdbnc+LxzO/k7DrFdffZWYmBgAdu/eTe/evQkNDWX8+PGUl5cDcPToUfr3709YWBjDhw+nqKgIgFOnTvHEE08QHh5O//79ycnJMT1fERGpfqZC47HHHuORRx5h9uzZzJ49m0cffZRHH330kuNt2bKFlStXOl+PGTOGCRMmsG7dOgzDYNmyZQBMnjyZqKgoUlNTadOmDQkJCQDMnj2bwMBAPv74Y/r27cvLL7/8a5ZRRESqianQCAkJAaCsrIyffvqJrKwsHnrooSrHOXHiBLNmzWLYsGHA2W9eFRcXExAQAECvXr1ITU2lrKyMjIwMQkNDK7QDbNy4kcjISAAiIiL44osvKCsru/ylFBGRamHq21MxMTH079+f6OhoSkpK+OCDD4iNjeXtt9++6Dgvvvgio0eP5tixYwBkZ2djs9mc79tsNrKysjh+/DheXl5YrdYK7b8cx2q14uXlRX5+vh4IJSJSQ0yFxvHjx4mOjgbA09OTgQMHkpycfNHhly9fTpMmTQgKCnI+i6OyR5FbLJaLtl+Mm9vl3fmkqmfdiriSzeZd0yWIVDtToWG328nKynIe4efm5la6sz8nJSWFnJwcunfvzsmTJzl9+jQWi4Xc3FznMDk5Ofj6+uLj40NhYSF2ux13d3dnO4Cvry+5ubk0btyY8vJyCgsLqV+//mUtYF5eIQ7HxWuV6qOdZEU5OQU1XYLIZXNzs1R5sG0qNAYOHEiPHj24//77sVgspKenV3kbkfPvVZWUlMT27duZNm0aERERZGZm0q5dO5KTkwkODsbDw4PAwEBSUlKIjIx0tsPZaynJyckMGzaMlJQUAgMD8fDwMLvsIiJSzUyFRp8+fWjTpg1bt27F3d2dwYMH06JFi8ueWXx8PHFxcRQVFeHv7+/s8po4cSIxMTEkJibSpEkTZs6cCcAzzzxDTEwMXbt2xdvbm/j4+Muep4iIVB+LUVU/0zVA3VNXjs3mTeaMITVdxlWh3dh31D0lv0uX6p76Vc/TEBGR65NCQ0RETFNoiIiIaQoNERExTaEhIiKmKTRERMQ0hYaIiJim0BAREdMUGiIiYppCQ0RETFNoiIiIaQoNERExTaEhIiKmKTRERMQ0hYaIiJjm0tCYM2cOXbp0oWvXrs6n+aWnpxMZGcnDDz/MrFmznMPu3r2b3r17Exoayvjx4ykvLwfg6NGj9O/fn7CwMIYPH05RUZErSxYRkSq4LDS2b9/O1q1bWb16NStWrGDhwoXs2bOH2NhYEhISSElJYdeuXaSlpQEwZswYJkyYwLp16zAMg2XLlgEwefJkoqKiSE1NpU2bNiQkJLiqZBERuQSXhcbdd9/Ne++9h9VqJS8vD7vdzqlTp2jWrBlNmzbFarUSGRlJamoqR44cobi4mICAAAB69epFamoqZWVlZGRkEBoaWqFdRERqhku7pzw8PJg7dy5du3YlKCiI7OxsbDab831fX1+ysrIuaLfZbGRlZXH8+HG8vLywWq0V2kVEpGZYXT2DUaNGMXToUIYNG8aBAwcueN9isVDZY8qrar8cVT3rVsSVbDbvmi5BpNq5LDR+/PFHSktLadWqFXXq1OHhhx8mNTUVd3d35zDZ2dn4+vrSqFEjcnNzne05OTn4+vri4+NDYWEhdrsdd3d3Z/vlyMsrxOG4MHyk+mknWVFOTkFNlyBy2dzcLFUebLuse+rw4cPExcVRWlpKaWkpn3/+Of369WP//v0cPHgQu93O2rVrCQ4Oxs/PD09PTzIzMwFITk4mODgYDw8PAgMDSUlJqdAuIiI1w2VnGiEhIezYsYMePXrg7u7Oww8/TNeuXfHx8WHkyJGUlJQQEhJCWFgYAPHx8cTFxVFUVIS/vz/R0dEATJw4kZiYGBITE2nSpAkzZ850VckiInIJFqOyCwfXEHVPXTk2mzeZM4bUdBlXhXZj31H3lPwu1Vj3lIiIXHsUGiIiYppCQ0RETFNoiIiIaQoNERExTaEhIiKmKTRERMQ0hYaIiJim0BAREdMUGiIiYppCQ0RETFNoiIiIaQoNERExTaEhIiKmKTRERMQ0l4bGvHnz6Nq1K127dmXGjBkApKenExkZycMPP8ysWbOcw+7evZvevXsTGhrK+PHjKS8vB+Do0aP079+fsLAwhg8fTlFRkStLFhGRKrjsyX3p6el8+eWXrFy5EovFwpAhQ1i7di3x8fEsXLiQJk2a8OSTT5KWlkZISAhjxoxh6tSpBAQEEBsby7Jly4iKimLy5MlERUXRtWtX3njjDRISEhgzZky11el9Q21qe3pU2/R+z4pLyig4VVzTZYjIVcxloWGz2YiJiaFWrVoA3HrrrRw4cIBmzZrRtGlTACIjI0lNTeW2226juLiYgIAAAHr16sXcuXPp27cvGRkZvPHGG872xx57rFpDo7anB1FjF1Xb9H7PFs/oTwEKDRG5OJd1TzVv3twZAgcOHCAlJQWLxYLNZnMO4+vrS1ZWFtnZ2RXabTYbWVlZHD9+HC8vL6xWa4V2ERGpGS470zhn3759PPnkk7zwwgtYrVb2799f4X2LxUJljymvqv1yVPWsW7mQzeZd0yVcM7Qu5Vrk0tDIzMxk1KhRxMbG0rVrV7Zv305ubq7z/ezsbHx9fWnUqFGF9pycHHx9ffHx8aGwsBC73Y67u7uz/XLk5RXicFwYPufog11RTk7Brx5X67Ki37IuRWqKm5ulyoNtl3VPHTt2jKeeeor4+Hi6du0KQNu2bdm/fz8HDx7Ebrezdu1agoOD8fPzw9PTk8zMTACSk5MJDg7Gw8ODwMBAUlJSKrSLiEjNcNmZxoIFCygpKWH69OnOtn79+jF9+nRGjhxJSUkJISEhhIWFARAfH09cXBxFRUX4+/sTHR0NwMSJE4mJiSExMZEmTZowc+ZMV5UsIiKXYDEqu3BwDTHTPaVvT521eEb/39w9lTljSDVW9PvVbuw76p6S36Ua654SEZFrj0JDRERMU2iIiIhpCg0RETFNoSEiIqYpNERExDSFhoiImKbQEBER0xQaIiJimkJDRERMU2iIiIhpCg0RETFNoSEiIqYpNERExDSFhoiImKbQEBER01weGoWFhURERHD48GEA0tPTiYyM5OGHH2bWrFnO4Xbv3k3v3r0JDQ1l/PjxlJeXA3D06FH69+9PWFgYw4cPp6ioyNUli4jIRbg0NHbs2MGjjz7KgQMHACguLiY2NpaEhARSUlLYtWsXaWlpAIwZM4YJEyawbt06DMNg2bJlAEyePJmoqChSU1Np06YNCQkJrixZRESq4NLQWLZsGRMnTsTX1xeAnTt30qxZM5o2bYrVaiUyMpLU1FSOHDlCcXExAQEBAPTq1YvU1FTKysrIyMggNDS0QruIiNQMqysn/vLLL1d4nZ2djc1mc7729fUlKyvrgnabzUZWVhbHjx/Hy8sLq9Vaof1yVPWsW7mQzeZd0yVcM7Qu5Vrk0tD4JcMwLmizWCyX3X458vIKcTgunM45+mBXlJNT8KvH1bqs6LesS5Ga4uZmqfJg+4p+e6pRo0bk5uY6X2dnZ+Pr63tBe05ODr6+vvj4+FBYWIjdbq/QLiIiNeOKhkbbtm3Zv38/Bw8exG63s3btWoKDg/Hz88PT05PMzEwAkpOTCQ4OxsPDg8DAQFJSUiq0i4hIzbii3VOenp5Mnz6dkSNHUlJSQkhICGFhYQDEx8cTFxdHUVER/v7+REdHAzBx4kRiYmJITEykSZMmzJw580qWLCIi57kiobF+/Xrn/4OCgli9evUFw9x+++18+OGHF7T7+fmxcOFCl9YnIiLm6BfhIiJimkJDRERMU2iIiIhpCg0RETFNoSEiIqYpNERExDSFhoiImHZFf9wnIubdUM8Tz1q1arqMq0JJaSmnTpbUdBmCQkPkquVZqxYD//lMTZdxVfjXoDmAQuNqoO4pERExTaEhIiKmKTRERMQ0hYaIiJimC+Eicl2o710Lj9qeNV3GVaGsuIQTBaW/alyFhohcFzxqe5ISPaimy7gqdHnvn/ArQ+N30T21Zs0aunTpwkMPPcSiRYtquhwRkevWVX+mkZWVxaxZs0hKSqJWrVr069eP9u3bc9ttt9V0aSIi152rPjTS09Pp0KED9evXByA0NJTU1FSefvppU+O7uVkuOUzDG//wW0q8pphZX1WpdUODaqrk9++3rkuAhl4+1VDJtaE61medhto+z7nY+rzUerYYhmG4oqDq8tZbb3H69GlGjx4NwPLly9m5cycvvfRSDVcmInL9ueqvaVSWaRbLbz/iEBGRy3fVh0ajRo3Izc11vs7OzsbX17cGKxIRuX5d9aFxzz33sGXLFvLz8zlz5gyffPIJwcHBNV2WiMh16aq/EN6oUSNGjx5NdHQ0ZWVl9OnThzvvvLOmyxIRuS5d9RfCRUTk6nHVd0+JiMjVQ6EhIiKmKTRERMQ0hYaIiJh23YXG3r17admyJevWrXO2paWl0bFjR/72t79dMHz37t1dUseoUaOIjIx0ybQBvv32W8aPH++y6f9ahYWFTJ48mYiICLp3786AAQP4v//7vxqrx1V/36tVUVERkydP5qGHHqJbt25ERUWxZcsWAJYuXcratWsBiImJISkpqSZLrVGHDx/mwQcfvKC9ZcuWFx1n27ZtDBgwwJVlXRWu+q/cVrekpCRCQ0NZsmQJoaGhAKSmpjJs2DAeeeSRC4ZftWpVtddw/PhxvvvuOxo2bEhmZibt2rWr9nnccccd3HHHHdU+3d/C4XAwdOhQ2rdvT3JyMlarla1btzJ06FA++ugjbrzxxitekyv+vlcrwzAYNmwYrVq14qOPPqJWrVp89913PPHEE/z973/n66+/5u67767pMuUqd12FRnl5OatXr2bRokX069ePn3/+mW3btvH555+zZcsW3NzcWL16NfXq1WPfvn3Mnj2bHj168P3333PixAnGjx/PTz/9RK1atYiJiSEoKIj333+fVatWcebMGSwWC7Nnz+bWW2/lwQcfpFu3bnz55ZecOXOGV199lTZt2gBnb/UeGBhIixYtWLp0qTM0kpKS2LhxI9nZ2fznP//h8ccf5+jRo2zdupX69evzzjvv4OnpSXJyMu+++y4Oh4PWrVszceJEPD096dChA61btyY3N5exY8fy5ptvsnDhQnbv3s2LL75IcXEx9erVIz4+noYNGzJp0iT27dtHbm4uN998M/PmzSM3N5enn36a5s2bs3v3bho0aMCcOXOcN4z8LbZt20Z2djajRo3Cze3sSW6HDh2YNm0aDoeDuLi4SuuJjo5m/fr1ALz++usAjBw5kjVr1pCYmIjFYuGOO+7gpZdeIj8/n9jYWAoKCsjJyaFr1648//zz7NmzhxdffJHy8nI8PT2ZNm0af/rTn2jZsiXff/89WVlZlY6XlJTEpk2bOHnyJIcOHeLee+9l0qRJv3ld1ITt27dz9OhR3nvvPeetePz9/Rk+fDgDBw7E29ubrVu3YrPZANi4cSOLFy8mLy/PeVBVVFTElClT2LdvH3a7naFDhxIREUFSUhIrV67kxIkTdOzYkeeee64mF9WlzGwT7777Lp999hnz58/niSee4I477iAzM5P8/Hzi4uIICQkhNzeX8ePHc/ToUaxWK6NHj6ZNmzbO/QbA/fffz7hx4+jSpQvz58/HYrFQXFxMVlYWBw8e5MiRI/Tt25fhw4dfseW/rrqnNm7cyB//+EduvvlmOnfuzJIlS+jbty8PPvggo0aNom/fvgDO7qtWrVo5x50zZw433XQTH3/8MTNmzGD27NkUFhby2WefsXDhQtauXUvnzp1ZvHixc5z69evz4Ycf0q9fP9566y1ne1JSEuHh4YSHh7Nu3TpOnDjhfO/bb7/lnXfeYdGiRUyfPp3g4GDWrFkDwKZNm9i3bx/Lli1jyZIlrFq1igYNGrBgwQLg7BnME088wapVq7Ba/3s88PzzzzNixAjnc0neffddvv76azw8PFi6dCmffvopJSUlpKWlAbBnzx4GDRrE2rVrueGGG5zz/62+++477rjjDmdgnBMSEsJPP/100Xoqk5WVxbRp0/jHP/7BRx99hN1uJy0tjbVr1xIREcGyZctYvXo1ixcvJj8/n3fffZdBgwaRlJTEgAED+OabbypM72LjAXz99dfMnTuX1atXs2HDBr7//vtqWR9X2rfffkubNm0uuHfbX/7yF+rUqeP8HNx///0AlJaWsnz5ct566y1mzZoFQGJiIq1btyYpKYlFixbx5ptvcujQIeDs32TlypXXdGCcU9U2sWLFCj755BPeeust6tSpA0BZWRlLly5l3LhxzJkzB4CXXnqJDh06sGbNGubOnUtsbCwOh4M//vGP7N27lx9//BG73c727dsB+OKLL+jYsSMA33//PQsWLGD58uXMnz+fU6dOXbFlv67ONJKSkoiIiACgS5cuPP/88zz77LMXDFfZL84zMjKIj48HzobK0qVLAfj73//ORx99xIEDB9i0aVOFoDn34WvevDmffPIJALt37+bYsWPcc889eHh40KpVK5KTkxk4cCAAf/7zn/Hy8sLLywuAoKAgAPz8/Dh16hTbtm3j4MGD/O///i9wdmP09/d3zrNt27YV6s7PzycnJ8e5sUVFRTnfq1+/PosWLeKnn37iwIEDnD59GoAGDRo4p9m8eXNOnjxZ9Yo1yc3NrdIbUMLZHdfF6qnM119/zZ///GcaN24MwGuvveZ8b+vWrSxYsIB9+/ZRVlbGmTNnCAkJYcqUKWzatImOHTs6uybPGTx4cKXjAdx1113Ov0fTpk2rbX1caRaLBbvdfkF7WVlZpcN36tQJi8VC8+bNOX78OHD2UQXFxcWsWLECgNOnT7Nv3z7g7FnL+Qcrv2e/PLCBs9175wL3YtvE3r17efHFF5k5cyZ169Z1jnv+vuDcQeLWrVuZOnWqcxpt27Zlx44dhISEsGXLFqxWK9HR0Xz00UfOM+BzzxFq3749tWrVokGDBtSvX5+CggJuuOEG16yMX7g2/sIm5OXl8cUXX7Br1y7ee+89DMPg1KlTzp35+WrXrn1B2y8/DD/++CO1a9fm8ccf57HHHiM4OJiGDRuye/du5zCenmefR3z+kd2KFSsoLS117rSKiopYsmSJMzQ8PDyqnK/dbic8PJy4uDjn+OfvCH5Z+y+nV1JSQnZ2Nnv37mXu3LlER0fTq1cvjh8/7tyhn6v7XO3VddOANm3asHjx4gofPoCZM2dy55138vrrr19Qzy/nX15ejtVqvWC9nDsrmD9/PocOHSIiIoLOnTuTnp6OYRiEhYVx1113sWHDBt59913S0tKcH1iA6dOnVzqeK9fHlda2bVsWLlxIWVlZhe3im2++qfT6l7u7O1Bx+3U4HLz22mu0bt0agNzcXOrVq8eaNWsq/dz8Xt1www0UFBRUaMvLy6NevXrAxbeJP/zhD7zyyiu88sor3H///c7gqGxf8MvtyDAM7HY7ISEhzJs3j1q1avHMM8/w8ccfs2bNGmfwVDX/K+G66Z5avXo1HTp04IsvvmD9+vVs2LCBYcOGOc8YLiUwMJCUlBTgbGAMHTqUXbt20axZMwYOHEjbtm354osvKj2SO6e0tJQ1a9bwr3/9i/Xr17N+/Xo+//xzcnJy2LZtm6k62rdvz6effkpeXh6GYTBp0iTefffdiw7v7e1N48aN2bx5M3D2wu+cOXPYsmUL4eHh9O7dm4YNG5KRkVFl7dUhMDCQBg0aMG/ePOe8Nm3a5OwjrqyeG264gZMnT5Kfn09paSmbNm0Czl7o37FjBzk5OQC88sorfP7552zevJnBgwcTHh7OsWPHyMrKwuFw8Oyzz7Jz50769evHM888w3fffVehtouNdy0JDAzktttu45VXXnGeXezatYvExERGjBiBu7v7JbeBDh068MEHHwBn7zjdrVs3jh075vLarzQvLy+aNWtW4VuWS5cudZ75X4yfnx+dOnXi7rvvZu7cuVUO26FDBz788EMADh06xL///W8CAgJo3bo1+/fv58CBA9x66620b9+exMREHnjggd+8XNXhujnTSEpKcj7I6ZyoqCjeeecdGjZseMnxR40aRVxcHN26dcNqtTJjxgxatWrFkiVL6NKlC7Vq1eLOO+90nqpXZsOGDfj5+VXoQvLy8qJv374sWbKkwpHExdx+++08/fTTPP744zgcDlq1asUTTzxR5TivvfYakyZNYsaMGdx4443MmDGD48eP8/zzz5OamkqtWrUICAjg8OHDl5z/b2GxWEhISGDatGlERERgtVq58cYbmT9/Pu7u7pXW4+3tzeDBg+nTpw+NGzd2HhE3atSI8ePHM3jwYBwOBwEBAfTq1Ys6deowduxYbrjhBho0aECbNm04fPgww4YNY/z48SQkJODu7k5MTEyF2p588slKx7vWzJs3j1mzZhEREYG7uzv16tXjtddeo3379uTm5jJz5ky8vb0vOv7TTz/NpEmTiIiIwG63M2bMGG666Sa++uqrK7gUV8a5z80bb7xBWVkZLVu25MUXX2Tjxo2XHHfs2LFERERU+bX68ePH8+KLLzq/2jx16lTnYx/atWvn7B7t0KEDy5cvv2q+2aYbFoqIiGnXTfeUiIj8dgoNERExTaEhIiKmKTRERMQ0hYaIiJim0BAREdMUGiIiYtp18+M+kSth27ZtvPzyy9StW5eioiL+/Oc/891331FUVIRhGEydOpV27dpRVFTE1KlT+fe//427uzudO3dm9OjRlJWVER8f7/xFvL+/P3Fxcc77HInUNIWGSDXbt28fn332GdnZ2fzzn/9k6dKluLm5MX/+fN5++23atWvH3LlzKSkpISUlBbvdzl//+le2b99ORkYG7u7uJCUlYbFYmDlzJvHx8b/b27HLtUehIVLNmjRpgp+fH35+ftSrV48lS5Zw6NAhtm3bxh/+8Afg7N1ix40bh7u7O+7u7rz//vvA2VtXFBQUkJ6eDpy9A22DBg1qbFlEfkmhIVLNzt3ZdOPGjbz88ssMGjSITp06ccstt7B69Wrg7N2Lz7/j6bFjx6hduzYOh4PY2FhCQkKAs3cxLikpufILIXIRuhAu4iKbN2+mY8eOREVFcccdd/DZZ5857yIbFBTEypUrcTgclJaWMmrUKDIyMrjvvvtYtGgRpaWlOBwOJkyYwMyZM2t4SUT+S6Eh4iL9+vUjIyODyMhIHnnkEZo2bcrhw4dxOBw8/fTTeHh40L17d3r06EFISAgPP/wwI0aMwM/Pj549e9KlSxcMw7jgjrwiNUl3uRUREdN0piEiIqYpNERExDSFhoiImKbQEBER0xQaIiJimkJDRERMU2iIiIhpCg0RETHt/wGRTedxoNiPGAAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 301 }, - { - "cell_type": "markdown", - "metadata": { - "id": "Aos3gdHjDQEi" - }, - "source": [ - "4.) Finally, let's plot the performances of these individual inner models. In the below cells, plot the individual inner predictors against the performance of their corresponding exponentiated gradient model as well as the unmitigated logistic regression model, and the `ThresholdOptimizer`." - ] + "id": "n3GhcUCm2LjD", + "outputId": "9a3be0c7-99ae-4de6-b61b-cf0fd0bd6b0e" + } + }, + { + "cell_type": "code", + "execution_count": 43, + "source": [ + "sns.countplot(x=Y_train_bal)\n", + "plt.title(\"Target Label Histogram for Training Dataset\")\n", + "outcome_train = figure_to_base64str(plt)" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY0AAAEXCAYAAABRWhj0AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAn7klEQVR4nO3de1xU5b4/8M8wA3iBUvjNIIfMnSaZYmKSiRpklqAwSkipIIip5WVLcfLCTQ0v2ySO1wNkZbZza0aEWGwc3dnZthV1I3a8ZakpKGgzXBS5yG14fn/0co7ExQdlgvDzfr18vZhnPWvNd61Zzmet9cysUQghBIiIiCRYtHUBRET0x8HQICIiaQwNIiKSxtAgIiJpDA0iIpLG0CAiImkMjT+QlStXYsKECZgwYQJcXFzg5eVlelxZWWm25y0tLUVISEij0zZt2oTly5e3aHmpqal44403WlzHE088geLi4gbtwcHB0Ol09dry8vIwePBgAMD+/fuxcuXKZpf9z3/+Exs2bGhxTb+npKQkPP/884iMjLyn+dPS0kz7y9ChQ/Hcc8+ZHh87dkx6ORs2bEBaWlqzfWS2eUs88cQT0Gq1mDBhAsaPH4/x48djy5YtUvN+8cUX2L59e6vVcqeYmBicPn3aLMtur1RtXQDJi4mJMf39wgsvID4+HgMHDjT785aUlODUqVNmfx5zGT16NEaPHt1sn1OnTqGkpOR3qujepKSkID4+Hm5ubvc0v5+fH/z8/AAAERER6Nu3L2bMmNHi5bz55pt37SOzzVvqr3/9K+zs7AAAxcXFmD17NhQKBV577bVm58vOzkbfvn1btZbbMjMzMWnSJLMsu71iaHQAFRUVeOedd5CTk4OSkhJ07doV8fHx6N27N4KDg/Hwww/j4sWLmDJlCjw8PBAVFYWSkhKo1WoIITB+/Hj4+/vj+PHjiI+Px61bt6BQKDB//nyMGjUKkZGRqKysxIQJE5CamgqlUilVV0pKCj7//HPU1NSgpKQEs2bNQmBgIACgoKAAM2bMgMFggJOTE1asWAG1Wo3S0lKsWrUK586dQ01NDdzd3bFo0SKoVPe+q6ampmLv3r3YvHkz9u3bh6SkJCgUCiiVSixatAhWVlbYuXMnjEYjbG1tER4ejoSEBPz973+HUqnEY489hiVLlkCtViM3N7fR7Td06FAEBQWhT58+yM/Px7Zt25CamopvvvkGVVVVuHXrFhYvXoyXXnoJmzZtwuXLl3HlyhUYDAY89dRTGDFiBNLS0pCXl4eFCxfC19e33jq89dZb0Ov1iI6Oxptvvomnn34a77zzDvLz8yGEgJ+fH2bOnIm8vLwGdWg0mrtuo6NHj2LVqlXo0qULKioqkJKSgri4OJw4cQLl5eUQQmDlypUYMmRIvcAZOHAgXn/9dRw6dAgGgwEhISEIDQ2tt82Dg4Ph6uqK48eP49q1axgyZAjWrFkDCwsLpKam4oMPPkCnTp0wbNgwfPrpp/jhhx/uWq+dnR0iIiIQFhaG6dOno6ioCEuXLkVRUREKCgrg5OSE9evX4/jx4/j2229x6NAhdOrUCV5eXo32s7e3x44dO7Bz505YWlrC2toay5cvx+OPPw69Xo/ly5fj2rVrqKmpgY+PD2bPno1169bBYDBgwYIFiIuLw6BBg+55H/1DEfSHNGrUKHHy5EkhhBB79uwRK1asME1bsmSJWL58uRBCiKlTp4rIyEjTtFdffVVs375dCCHEhQsXxKBBg8SXX34pbty4IcaMGSOuXLkihBDil19+ER4eHiI/P19cuXJFuLq6NlrHxo0bRWxsbIP2srIy8eqrr4ri4mIhhBDff/+9aRlffvmlcHV1FTk5OUIIIf7rv/5LvPnmm0IIISIiIsSnn34qhBCitrZWLFiwQHzwwQdCCCGcnZ1FUVFRg+eaOnWqGDVqlBg/frzp39ixY+s93+uvvy6EEGL06NHi+++/F0II8a9//Uts2rSpwXqkpKSISZMmifLyctO01157rdntd+XKFeHs7CyysrKEEELk5eWJ4OBgcevWLSGEEOnp6cLX19e0vFGjRombN2+KW7duiWeeeUasXr1aCCHEP/7xDzFmzJhGt/Wdr3lQUJD4+OOPhRBC3Lx5U2i1WpGent6gjqYsXrxYfPTRR6bHR44cEf369RN5eXlCCCGOHz8u5s+fL4xGoxBCiM2bN4s33nijwbzOzs5i27ZtQgghTp06JVxcXERlZWW9bT516lQRFhYmjEajKC0tFSNHjhSHDx8W58+fF+7u7uLatWtCCCE2bdoknJ2dG623sde+vLzc1P7JJ5+IzZs3CyGEqKurEzNnzhRbtmxpUG9T/Wpra8WAAQOEXq8XQgixa9cusXPnTiGEEMHBwWL//v1CCCEqKytFcHCw+Pvf/97gNXlQ8EyjA/D29kbPnj2xbds25Obm4t///rfpej4A0+WMkpISnDx5En/7298AAH369MGwYcMAAP/7v/+LgoICzJs3zzSfQqHATz/9dE+n9l27dsX777+PAwcOICcnBz/++CMqKipM04cPH45evXoBAAICAhAQEADg17GFU6dOISUlBQCkx2oWLVoEb29v0+O8vDxotdoG/Xx8fPDnP/8Znp6eGDFiBGbNmtWgz3fffQd/f3906dIFABASEoL3338fhYWFTW4/AFCpVHB1dQUAODk5Yc2aNfj666+Rm5trOmK/c/1tbW0BABqNBs899xwA4NFHH8WNGzeaXdeKigocP34cH3/8MQDA1tYW/v7++O677zBo0KB6dbSEo6MjnJycAACDBw/Gww8/jJ07d+LKlSs4evQounbt2uh8ty9DDRgwANXV1fVe59tGjRoFCwsL2NjYoFevXigpKcGPP/6IESNGoEePHgCAqVOnYtOmTdL1KhQKAIC1tTWmTZuGY8eOYevWrcjJycH58+cbPfJvqp9SqYS3tzcmT56M559/HiNGjIBWq0VFRQWysrJQUlJiGvOqqKjAjz/+iHHjxknX2pEwNDqAHTt2IDk5GUFBQdBqtejWrRvy8vJM02+/+d2+rCTuuN3Y7Taj0Yg+ffrgiy++ME3T6/Wws7ODXq9vcU2//PILJk2ahFdffRVDhgyBt7c3/ud//qfB896u5/blp7q6OmzYsAF9+vQBANy8edP05tAawsPDERAQgIMHD5oujaSmptbrI35zO7a6ujrU1tbC2tq6wfQ718PKysq0HmfOnMHcuXMRGhqKESNG4JlnnkFsbGy9vndqyeW3urq6Jmv8bR0tcXs/AX4N71WrVmH69OkYPXo0evfuja+++qrR+W5vl9uv029rA4BOnTqZ/lYoFBBCQKlUNrktZZw6dQqPPPIIunbtivfeew8nT57ExIkT8eyzz6K2trbROprrFx8fj3PnziEzMxMffvghUlJS8N5770EIgZ07d6Jz584Afh1Pub3ODyJ+eqoDOHjwIF5++WW88soreOyxx/Dtt9/CaDQ26GdjY4Onn37a9CZ55coVHD58GAqFAq6ursjNzUVWVhYA4OzZs/Dy8oLBYIBKpYLRaGz0P2FTTp8+DTs7O8ydOxfPPfecKTBu13X06FFcvXoVAPDZZ5/Bw8MDADBy5Eh88sknEEKguroac+bMMR3Z36/a2lq88MILqKiowJQpU7Bs2TL8/PPPqK2thVKpNL3pjhw5EqmpqaYj5m3btuGZZ56Bra1tk9vvt7KysuDi4oLp06dj6NCh2L9/f6Ovyb2wsbHBoEGDTJ8IKi0tRVpaGoYPH94qyweAQ4cOYdSoUQgMDMTAgQPxzTfftFr9t40cORKHDx82HZTcecByN3q9HvHx8aZB8IMHD2LatGnw8/ODvb09MjMzTfXe+do21a+4uBienp7o1q0bQkND8dZbb+Gnn36CjY0NXF1dsXXrVgC/HsRMmTIF+/fvb7DsBwXPNDqA1157DUuXLjUNUg8YMADnzp1rtO+aNWsQHR2NHTt2wMHBAY888gg6deoEOzs7bNy4EXFxcaiqqoIQAnFxcXBycoLRaET//v0xduxYfPbZZ+jevXu9ZSYnJ2PXrl2mx0888QS2bt2KlJQUeHt7o3PnznjqqadgZ2eH3NxcAICzszOioqJQWFiI3r17mz62Gx0djVWrVkGr1aKmpgbDhw/HzJkzW2U7qVQqREVFYcGCBVCpVFAoFPjLX/4CKysruLu7Y/78+bC0tER0dDSuXbuGV155BXV1dejVqxfi4+Ob3X6/5evri3379mHcuHGwtLSEu7s7SkpKUFZW1irrEh8fj+XLlyM1NRXV1dXQarXw9/dHfn5+qyx/8uTJWLBgAbRaLZRKJdzc3LBv3z7U1dW1yvIB4LHHHkNkZCRmzJgBKysrPPnkk6aj+cZMmzYNFhYWpjOSiRMnIigoCAAwb948xMXFITExEUqlEk8//TQuX74MAPDw8MCKFSua7WdnZ4c5c+YgNDQUnTp1glKpNH1kOD4+HitWrIBWq0V1dTV8fX0xfvx4AMCLL76I8PBwrFy5EiNHjmy1bdOutcVACrWdxMREceHCBSHErwOozz//vDh//nwbV/XHwe3Xei5fviw2bdpkGmzfu3evCAgIaOOq6G54pvGA+dOf/oTw8HBYWFjAaDRi1qxZePzxx9u6rD8Mbr/W06NHDxgMBtPZjK2tLf7yl7+0dVl0Fwoh+CNMREQkhwPhREQkjaFBRETSGBpERCSNoUFERNI6/Kenrl8vR10dx/qJiGRYWCjQvXvjt4wBHoDQqKsTDA0iolbCy1NERCSNoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCStw39P437ZPtQJnawt27oMamcqq2pQelPu98vNpfvDVlBZPbg/O0qNq62uwvWSarMtn6FxF52sLRG4aHtbl0HtzI64IJSibUNDZWWN7LjW+VVD6jiGLPoIgPlCg5eniIhIGkODiIikMTSIiEgaQ4OIiKQxNIiISBpDg4iIpDE0iIhIGkODiIikMTSIiEgaQ4OIiKQxNIiISBpDg4iIpDE0iIhIGkODiIikMTSIiEgaQ4OIiKQxNIiISBpDg4iIpDE0iIhImll/IzwkJARFRUVQqX59muXLl+Py5ctISkpCTU0NQkNDERQUBADIzMzE6tWrUVVVhbFjxyI8PBwAcPbsWcTExKCsrAxubm6IjY01LY+IiH5fZjvTEELg4sWL2L17t+lfjx49sG7dOuzYsQO7d+/G559/jgsXLqCyshJRUVFITExERkYGTp8+jQMHDgAAFi5ciCVLlmDv3r0QQiA5OdlcJRMR0V2YLTQuXrwIhUKBWbNmYfz48fjb3/6GzMxMDBs2DN26dUOXLl3g5eUFnU6HkydPolevXujZsydUKhW0Wi10Oh3y8/NRWVkJV1dXAIC/vz90Op25SiYiorswW2jcvHkT7u7uSEhIwCeffIKdO3fi6tWrUKvVpj4ajQZ6vR4Gg0GqXa1WQ6/Xm6tkIiK6C7MNDgwePBiDBw8GAHTp0gUBAQFYvXo1Zs+eXa+fQqGAEKLB/M21t4S9vU2L+hPJUqtt27oEokaZc980W2gcO3YMNTU1cHd3B/DrGIeTkxMKCwtNfQwGAzQaDRwcHKTaCwoKoNFoWlRHUVEZ6uoaho8svjFQUwoKStv0+blvUlPuZ9+0sFA0e7BttstTpaWliIuLQ1VVFcrKyrBr1y689957OHz4MIqLi3Hr1i3s27cPHh4eGDRoEC5duoTc3FwYjUakp6fDw8MDTk5OsLa2RnZ2NgAgLS0NHh4e5iqZiIjuwmxnGqNGjcKJEyfg5+eHuro6BAYGYsiQIQgPD0dISAhqamoQEBCAp556CgDw7rvvYv78+aiqqoKnpye8vb0BAPHx8YiJiUF5eTn69++PkJAQc5VMRER3oRCNDRx0IK1xeSpw0fZWrIg6gh1xQe3i8lR23Mw2rYHanyGLPvpjXp4iIqKOh6FBRETSGBpERCSNoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCSNoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCSNoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCSNoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCSNoUFERNLMHhpr1qxBREQEAODs2bOYOHEivLy8EB0djdraWgDA1atXERQUBG9vb8yZMwfl5eUAgJs3b+L111/H2LFjERQUhIKCAnOXS0REzTBraBw+fBi7du0yPV64cCGWLFmCvXv3QgiB5ORkAEBsbCwCAwOh0+ng4uKCxMREAMD69evh5uaGPXv24JVXXsGqVavMWS4REd2F2ULjxo0bWLduHWbPng0AyM/PR2VlJVxdXQEA/v7+0Ol0qKmpQVZWFry8vOq1A8A///lPaLVaAICvry++++471NTUmKtkIiK6C5W5Frx06VKEh4fj2rVrAACDwQC1Wm2arlarodfrcf36ddjY2EClUtVr/+08KpUKNjY2KC4uhoODg3Qd9vY2rbVKRPWo1bZtXQJRo8y5b5olNL744gs4OjrC3d0dqampAAAhRIN+CoWiyfamWFi07OSoqKgMdXUNn0MW3xioKQUFpW36/Nw3qSn3s29aWCiaPdg2S2hkZGSgoKAAEyZMQElJCSoqKqBQKFBYWGjqU1BQAI1GAzs7O5SVlcFoNEKpVJraAUCj0aCwsBA9evRAbW0tysrK0K1bN3OUTEREEswyprF161akp6dj9+7dCAsLwwsvvIDVq1fD2toa2dnZAIC0tDR4eHjA0tISbm5uyMjIqNcOAJ6enkhLSwPwaxC5ubnB0tLSHCUTEZEEs41pNCY+Ph4xMTEoLy9H//79ERISAgBYtmwZIiIikJSUBEdHR6xduxYA8OabbyIiIgI+Pj6wtbVFfHz871kuERH9hkI0NqjQgbTGmEbgou2tWBF1BDvigtrFmEZ23Mw2rYHanyGLPjLrmAa/EU5ERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCSNoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCSNoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCSNoUFERNKkQkOv1zdou3DhQqsXQ0RE7VuzoXHjxg3cuHEDs2bNQklJielxYWEh5s6d+3vVSERE7YSquYlvv/02Dh06BAB49tln/28mlQovvviieSsjIqJ2p9nQ2LJlCwAgMjISq1ev/l0KIiKi9qvZ0Lht9erVyM/PR0lJCYQQpvYBAwaYrTAiImp/pEIjPj4e27Ztg729valNoVBg//79ZiuMiIjaH6nQyMjIwL59++Dg4GDueoiIqB2T+sito6MjA4OIiORCw93dHXFxccjOzsaZM2dM/+5mw4YNGDduHHx8fLB161YAQGZmJrRaLcaMGYN169aZ+p49exYTJ06El5cXoqOjUVtbCwC4evUqgoKC4O3tjTlz5qC8vPxe1pOIiFqB1OWp1NRUAIBOpzO13W1M49///jeOHDmCr776CrW1tRg3bhzc3d0RFRWFbdu2wdHREW+88QYOHDgAT09PLFy4ECtXroSrqyuioqKQnJyMwMBAxMbGIjAwED4+PkhISEBiYiIWLlx4n6tNRET3Qio0vv322xYveOjQofj000+hUqmg1+thNBpx8+ZN9OrVCz179gQAaLVa6HQ6PP7446isrISrqysAwN/fHxs3bsQrr7yCrKwsJCQkmNqnTp3K0CAiaiNSoXH70tJvTZ8+vdn5LC0tsXHjRnz88cfw9vaGwWCAWq02TddoNNDr9Q3a1Wo19Ho9rl+/DhsbG6hUqnrtRETUNqRC49y5c6a/q6urkZ2dXe8b4s0JCwvDrFmzMHv2bOTk5DSYrlAo6n33Q6a9JeztbVrUn0iWWm3b1iUQNcqc+6b0l/vuVFxcjEWLFjU7z88//4zq6mo8+eST6Ny5M8aMGQOdTgelUmnqYzAYoNFo4ODggMLCQlN7QUEBNBoN7OzsUFZWBqPRCKVSaWpviaKiMtTVNQwfWXxjoKYUFJS26fNz36Sm3M++aWGhaPZg+55ujW5nZ4f8/Pxm++Tl5SEmJgbV1dWorq7G/v37MXnyZFy6dAm5ubkwGo1IT0+Hh4cHnJycYG1tjezsbABAWloaPDw8YGlpCTc3N2RkZNRrJyKittHiMQ0hBE6fPl3v2+GN8fT0xIkTJ+Dn5welUokxY8bAx8cHdnZ2mD9/PqqqquDp6Qlvb28Av37rPCYmBuXl5ejfvz9CQkIAAMuWLUNERASSkpLg6OiItWvX3uu6EhHRfVKIxgYOfiMyMrLeYzs7OwQHB6NHjx5mK6y1tMblqcBF21uxIuoIdsQFtYvLU9lxM9u0Bmp/hiz6yKyXp1o0ppGfn4/a2lr06tXrngsiIqI/LqnQyM3Nxdy5c2EwGFBXV4fu3btj8+bN6NOnj7nrIyKidkRqIHz58uWYOXMmsrKykJ2djTlz5iA2NtbctRERUTsjFRpFRUV4+eWXTY8nTpyI69evm60oIiJqn6RCw2g04saNG6bHxcXF5qqHiIjaMakxjalTp2LSpEkYO3YsAGDPnj2YNm2aWQsjIqL2R+pMw9PTEwBQU1ODixcvQq/X46WXXjJrYURE1P5InWlEREQgKCgIISEhqKqqwmeffYaoqCh8+OGH5q6PiIjaEakzjevXr5u+oW1tbY3Q0FAUFBSYtTAiImp/pAfC77wleWFhYaN3oCUioo5N6vJUaGgo/Pz88Nxzz0GhUCAzM/Oud7klIqKORyo0AgIC4OLigiNHjkCpVGLGjBlwdnY2d21ERNTOSIUGAPTr1w/9+vUzZy1ERNTO3dPvaRAR0YOJoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCSNoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCSNoUFERNIYGkREJM2sofHf//3f8PHxgY+PD+Li4gAAmZmZ0Gq1GDNmDNatW2fqe/bsWUycOBFeXl6Ijo5GbW0tAODq1asICgqCt7c35syZg/LycnOWTEREzTBbaGRmZuLgwYPYtWsX0tLScObMGaSnpyMqKgqJiYnIyMjA6dOnceDAAQDAwoULsWTJEuzduxdCCCQnJwMAYmNjERgYCJ1OBxcXFyQmJpqrZCIiuguzhYZarUZERASsrKxgaWmJPn36ICcnB7169ULPnj2hUqmg1Wqh0+mQn5+PyspKuLq6AgD8/f2h0+lQU1ODrKwseHl51WsnIqK2If1zry3Vt29f0985OTnIyMhAcHAw1Gq1qV2j0UCv18NgMNRrV6vV0Ov1uH79OmxsbKBSqeq1t4S9vc19rglR49Rq27YugahR5tw3zRYat50/fx5vvPEGFi9eDJVKhUuXLtWbrlAoIIRoMF9z7S1RVFSGurqGy5HFNwZqSkFBaZs+P/dNasr97JsWFopmD7bNOhCenZ2N0NBQvP3223j55Zfh4OCAwsJC03SDwQCNRtOgvaCgABqNBnZ2digrK4PRaKzXTkREbcNsoXHt2jXMmzcP8fHx8PHxAQAMGjQIly5dQm5uLoxGI9LT0+Hh4QEnJydYW1sjOzsbAJCWlgYPDw9YWlrCzc0NGRkZ9dqJiKhtmO3y1JYtW1BVVYV3333X1DZ58mS8++67mD9/PqqqquDp6Qlvb28AQHx8PGJiYlBeXo7+/fsjJCQEALBs2TJEREQgKSkJjo6OWLt2rblKJiKiu1CIxgYOOpDWGNMIXLS9FSuijmBHXFC7GNPIjpvZpjVQ+zNk0Ud/3DENIiLqWBgaREQkjaFBRETSGBpERCSNoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCSNoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCSNoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSGBpERCSNoUFERNIYGkREJI2hQURE0hgaREQkjaFBRETSzB4aZWVl8PX1RV5eHgAgMzMTWq0WY8aMwbp160z9zp49i4kTJ8LLywvR0dGora0FAFy9ehVBQUHw9vbGnDlzUF5ebu6SiYioCWYNjRMnTmDKlCnIyckBAFRWViIqKgqJiYnIyMjA6dOnceDAAQDAwoULsWTJEuzduxdCCCQnJwMAYmNjERgYCJ1OBxcXFyQmJpqzZCIiaoZZQyM5ORnLli2DRqMBAJw8eRK9evVCz549oVKpoNVqodPpkJ+fj8rKSri6ugIA/P39odPpUFNTg6ysLHh5edVrJyKitqEy58JXrVpV77HBYIBarTY91mg00Ov1DdrVajX0ej2uX78OGxsbqFSqeu1ERNQ2zBoavyWEaNCmUCha3N4S9vY2LepPJEuttm3rEogaZc5983cNDQcHBxQWFpoeGwwGaDSaBu0FBQXQaDSws7NDWVkZjEYjlEqlqb0liorKUFfXMHxk8Y2BmlJQUNqmz899k5pyP/umhYWi2YPt3/Ujt4MGDcKlS5eQm5sLo9GI9PR0eHh4wMnJCdbW1sjOzgYApKWlwcPDA5aWlnBzc0NGRka9diIiahu/65mGtbU13n33XcyfPx9VVVXw9PSEt7c3ACA+Ph4xMTEoLy9H//79ERISAgBYtmwZIiIikJSUBEdHR6xdu/b3LJmIiO7wu4TGt99+a/rb3d0dX331VYM+/fr1Q0pKSoN2JycnbNu2zaz1ERGRHH4jnIiIpDE0iIhIGkODiIikMTSIiEgaQ4OIiKQxNIiISBpDg4iIpDE0iIhIGkODiIikMTSIiEgaQ4OIiKQxNIiISBpDg4iIpDE0iIhIGkODiIikMTSIiEgaQ4OIiKQxNIiISBpDg4iIpDE0iIhIGkODiIikMTSIiEgaQ4OIiKQxNIiISBpDg4iIpDE0iIhIGkODiIikMTSIiEjaHyI0vv76a4wbNw4vvfQStm/f3tblEBE9sFRtXcDd6PV6rFu3DqmpqbCyssLkyZPx7LPP4vHHH2/r0oiIHjjtPjQyMzMxbNgwdOvWDQDg5eUFnU6HP//5z1LzW1go7ruG/9e9630vgzqe1ti37pfVQ/ZtXQK1Q/ezb95t3nYfGgaDAWq12vRYo9Hg5MmT0vN3b4U3/I2Rfve9DOp47O1t2roEDJy9pq1LoHbInPtmux/TEEI0aFMo2v4Ij4joQdTuQ8PBwQGFhYWmxwaDARqNpg0rIiJ6cLX70Bg+fDgOHz6M4uJi3Lp1C/v27YOHh0dbl0VE9EBq92MaDg4OCA8PR0hICGpqahAQEICnnnqqrcsiInogKURjgwZERESNaPeXp4iIqP1gaBARkTSGBhERSWNoEBGRNIYG3RVvGEntXVlZGXx9fZGXl9fWpXR4DA1q1u0bRu7YsQO7d+/G559/jgsXLrR1WUQmJ06cwJQpU5CTk9PWpTwQGBrUrDtvGNmlSxfTDSOJ2ovk5GQsW7aMd4r4nbT7L/dR27rfG0YSmduqVavauoQHCs80qFm8YSQR3YmhQc3iDSOJ6E4MDWoWbxhJRHfimAY1izeMJKI78YaFREQkjZeniIhIGkODiIikMTSIiEgaQ4OIiKQxNIiISBpDg6gJgwcPbtFdU0+dOoWwsDAAwMmTJ7F06dK7zlNaWoqwsDD4+vpi3Lhx+OCDD0zTcnJyEBgYiHHjxiEgIAA///xzi9fB19cXR48ebfF8RE1haBC1koEDB2Ljxo0AgAsXLkCv1991ng0bNsDBwQHp6elISUnBzp078f333wMAFixYgClTpiAjIwPz589HWFhYo7d1Ifo98ct91CEcPXoUq1atQpcuXVBRUYGwsDBs3rwZNTU16NSpExYvXozBgwejsLAQS5cuRVFREQoKCuDk5IT169fD3t4ex44dw4oVK6BQKDBw4EDU1dWZlr127VpoNBqcP38enTt3xvz587Ft2zZcunQJY8aMQVRUFI4ePYoVK1bgww8/xMaNG1FaWorIyEisXr26ybqjo6NhNBoBAAUFBaiuroatrS30ej0uXrwIHx8fAICnpydiY2Pxww8/YMCAAU0u78KFC4iKisKtW7fQu3dvVFRUmKa9//77+Oabb1BVVYVbt25h8eLFePHFF+Ht7Y0lS5Zg5MiRAICYmBj07dsXI0eORHR0NKqrqyGEQEBAAIKCgu77taI/OEHUARw5ckT069dP5OXliUuXLglfX19RXFwshBDi3LlzYsSIEaK8vFx88sknYvPmzUIIIerq6sTMmTPFli1bRFVVlRg+fLjIzMwUQgjx9ddfC2dnZ3HlyhVx5MgR8eSTT4ozZ84IIYSYMWOGmDRpkqiqqhJFRUViwIAB4pdffhFHjhwRPj4+QgghvvzyS/H6669L1//2228LFxcXER4eLmpra8X3338vvLy86vWZPHmy+Oabb5pdzoQJE0RycrIQQohjx46JJ554Qhw5ckTk5eWJ4OBgcevWLSGEEOnp6cLX11cIIcTWrVtFWFiYEEKI0tJSMWzYMFFSUiIiIyNN28pgMIi33npLGI1G6XWijolnGtRhODo6wsnJCdu3b4fBYEBoaKhpmkKhwOXLlzFt2jQcO3YMW7duRU5ODs6fP49Bgwbh3LlzUKlUcHd3B/DrWMCdYxKPPPII+vfvDwB49NFHYWtrCysrK9jZ2aFr164oKSm5r9rj4+MRGxuLsLAwJCQkmI76f0upVDa5jOvXr+Onn36Cn58fAGDIkCHo27cvAMDJyQlr1qzB119/jdzcXJw4cQLl5eUAAH9/fyQkJKC4uBg6nQ7PP/88HnroIbz00ktYvHgxTp48CXd3d8TExMDCgle0H3TcA6jD6NKlCwCgrq4O7u7u2L17t+lfcnIy+vbti/feew8bNmxA9+7dMWnSJIwYMQJCCCgUigbjBSrV/x1TWVlZNTntfvzrX/8yjX107doVPj4++OGHH/Af//EfKCwsrFeTXq9Hjx49mlzW7VvW3znP7TrPnDmDyZMno6ysDCNGjMDMmTNNfR566CF4e3vjq6++wpdffokpU6YAAEaNGoW9e/di7NixOHv2LLRaLS5fvtwq601/XAwN6nCGDRuGQ4cOmT5tdODAAYwfPx5VVVU4ePAgpk2bBj8/P9jb2yMzMxNGoxHOzs4QQuDAgQMAgP3799/X2YNSqURtbe1d++3ZswcJCQkQQqC6uhp79uzBsGHD0KNHDzz66KPIyMgA8Gu4WFhYwNnZuclldevWDQMGDMAXX3wB4NegOHfuHAAgKysLLi4umD59OoYOHYr9+/ebxlIAICgoCJ9++imEEKYbUr799tvIyMiAj48Pli1bBhsbG1y7du2etwl1DLw8RR1O3759sXz5cvznf/4nhBBQqVRISkpCly5dMG/ePMTFxSExMRFKpRJPP/00Ll++DEtLSyQkJOCdd97B2rVr8eSTT8Le3v6eaxg8eDDWr1+PefPmISEhocl+ERERWLZsGbRaLRQKBUaPHo2QkBAAwNq1a7FkyRIkJSXBysoKGzZsuOvlobVr1yIyMhI7d+7Eo48+it69ewP49XLbvn37MG7cOFhaWsLd3R0lJSUoKyuDjY0N+vXrh4cffhiTJ082LWvu3LmIjo7G559/DqVSiRdffBFDhw69521CHQPvcktEuHz5MoKDg6HT6dC5c+e2LofaMZ5pEJnRxYsXER4e3ui0xx57DOvXr2/R8o4cOdLkR3ifffZZREVFtbREbNiwAcnJyYiOjmZg0F3xTIOIiKRxIJyIiKQxNIiISBpDg4iIpDE0iIhIGkODiIikMTSIiEja/wfAeg2RRuC9lwAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 301 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 260 - }, - "id": "IxTjb8CeZqPI", - "outputId": "8e06051c-4629-42f0-93f6-51931da2615b" - }, - "outputs": [], - "source": [ - "# Plot the individual predictors against the Unmitigated Model and the ThresholdOptimizer\n", - "plt.scatter(balanced_error_exercise, false_neg_exercise,\n", - " label=\"ExponentiatedGradient - Iterations - Exercise\")\n", - "for i in range(len(predictors_exercise)):\n", - " plt.annotate(str(i), xy=(balanced_error_exercise[i]+0.001, false_neg_exercise[i]+0.001))\n", - "\n", - "plt.scatter(1-balanced_accuracy_score(Y_test, Y_expgrad_exercise),\n", - " false_negative_rate_difference(Y_test, Y_expgrad_exercise, sensitive_features=A_test),\n", - " label=\"ExponentiatedGradient - Combined - Exercise\")\n", - "plt.scatter(1-balanced_accuracy_score(Y_test, Y_pred),\n", - " false_negative_rate_difference(Y_test, Y_pred, sensitive_features=A_test),\n", - " label=\"Unmitigated\")\n", - "plt.scatter(1-balanced_accuracy_score(Y_test, Y_pred_postprocess),\n", - " false_negative_rate_difference(Y_test, Y_pred_postprocess, sensitive_features=A_test),\n", - " label=\"ThresholdOptimizer\")\n", - "\n", - "plt.xlabel(\"Balanced Error Rate\")\n", - "plt.ylabel(\"False Negative Rate Difference\")\n", - "plt.legend(bbox_to_anchor=(1.9,1))\n", - "plt.show()" - ] + "id": "lIp3j8fD2LjE", + "outputId": "48b4cbdf-c0ad-4a5b-d5f2-987d6b75229b" + } + }, + { + "cell_type": "code", + "execution_count": 44, + "source": [ + "sns.countplot(x=\"race\", data=A_test)\n", + "plt.title(\"Sensitive Attributes for Testing Dataset\")\n", + "sensitive_test = figure_to_base64str(plt)" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEXCAYAAABoPamvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAA2B0lEQVR4nO3de1xUdf7H8dfAAGqQBgG6ZG4XM9GSkk3tAmsmqICYl800L633MstNDREzzdSM9UIGm9WWlqaWCGaImqaVqBGVyqZleUnFH1dFQbnNnN8frrOSqNhwkXw/Hw8fD+c755z5fL8M533O9wxnTIZhGIiIiNjBobYLEBGRuk9hIiIidlOYiIiI3RQmIiJiN4WJiIjYTWEiIiJ2U5hcw77//nsGDBhAWFgYoaGhDB06lH379lXLaw0bNoyff/4ZgL///e/k5eVd0F4V3n//fVq0aMH3339frn3z5s3Mnz8fgFOnTjFw4MCLbiM8PJyTJ08SHx/PiBEjrriGBQsW8Nlnn13xehUpKCigb9++hISEsG7dut+1jTFjxhAeHk54eDgtWrQgLCyM8PBwBgwYcMXbOn8cN27cyPTp039XTRU5v7bu3bvTvXt33nnnnUqt+9FHH7FkyZIqq+V8UVFRpKenV8u2/0jMtV2A1I6SkhJGjBjBv//9b1q1agVAYmIiw4YNY+PGjTg6Olbp67311lu2/2/durXC9qqwbNkywsLCWLRoEX5+frb23bt3k5+fD0B+fj67d+++6DYSExPtqmHHjh3cfvvtdm3jnD179pCbm8uGDRt+9zZiYmJs/2/RogWLFi3C3d39d23r/HHs1KkTnTp1+t11VeT82vLy8hg5ciQmk4m///3vl1wvLS2N5s2bV2kt56SkpPDYY49Vy7b/SBQm16gzZ85w6tQpTp8+bWvr3r07rq6uWCwWHB0d2bRpE3FxcZSWllKvXj1eeOEF7rnnHl5//XWOHj1KdnY2R48exd3dnblz5+Lt7c3SpUtZtmwZTk5OuLi4MG3aNG6//XYefvhh5s+fz9KlSwEYNGgQCxcupH///syfP5/33nsPX19fhgwZAsCHH37Ijh07mDdv3kXr+K0dO3aQn5/P+PHj6dy5M8eOHaNJkybs3LmTZcuWYbFYcHNz49tvv6WoqIjw8HDi4+Np06YNnTp1Yu/evURHR9O7d2+2bdsGQHZ2NkOGDCErKwsfHx9efvllPD09GTBgAP3796dLly4Atse5ubmkp6cze/ZsHB0dCQwMJDo6mtTUVCwWC76+vkRFReHq6nrRsTpn//79REZGkpmZSXh4OMuXL+err75iwYIFWCwWXF1dmThxInfffTevv/4633//PVlZWbRo0YLo6OhKvQ/i4uJYv349VqsVHx8fpkyZgre3N+vXrycuLg6TyYSjoyMTJkzA2dm53Dg2a9aMdevW8eabbzJgwAD8/Pz49ttvOXbsGG3btuXVV1/FwcGB+Ph4Fi5cSL169Wjfvj2LFy/mhx9+uGxt7u7uREREMGbMGJ588klyc3N58cUXyc3NJTs7Gx8fH+bNm8e3337Lpk2b2Lp1K/Xq1SM4OLjC5Tw8PC465pmZmUybNo1jx45RWlpKSEgII0eOZO7cuWRlZTFu3Dhmz55NmzZtKjWu1yRDrln//ve/jbvvvtt4+OGHjXHjxhkfffSRcfr0acMwDOPAgQNGaGiokZeXZxiGYfz000/GAw88YBQWFhoxMTFGp06djFOnThmGYRgjRoww5s+fb5SVlRmtWrUyMjMzDcMwjFWrVhnLli0zDMMwOnbsaOzatcswDMO44447jNzc3HLt27ZtM0JDQ2219e7d29i6desl6/itZ5991pg1a5ZhGIYxbNgwY/bs2bbnYmJijKlTpxqGYRiHDx82/Pz8bM/dcccdxqpVq8o9zs3NNVauXGn4+fkZBw8eNAzDMP75z38azz77rGEYhvHEE08Ya9euta1z/uPz///6668bs2bNMqxWq20bU6ZMueRYnW/79u1GSEiIYRiG8fPPPxv333+/8euvvxqGYRgpKSnGAw88YJw6dcqIiYkxgoODjdLS0gu2cb7zx37VqlXGc889Z1tn2bJlxtChQw3DMIxOnToZ3333nWEYhvHll18ar7/++gXjuHLlSmP48OG2Po8ZM8awWCzGqVOnjAcffNDYtm2bsW/fPqNDhw7GsWPHbONxxx13XLa2cwoLC23t7733nvHmm28ahmEYVqvVGDp0qPHOO+8YhmEYL7zwgvH2228bhmFcdLlLjfmAAQOMjRs3GoZhGEVFRcaAAQOMTz/91DCM8u9duTidmVzDnnzySfr06UNqaiqpqam89dZbvPXWW3z88cds3bqVrKwsBg8ebFveZDLx66+/AnDffffh6uoKgK+vL/n5+Tg6OtKlSxf69u3LX//6Vx544AHCwsIqVUu7du0oLi5m9+7d1K9fn7y8PDp06MDSpUsvWsedd95pa8vOzuazzz5j5cqVAPTo0YOXXnqJp59+mgYNGlz29f39/Stsv//++2nWrBkAvXv3pnfv3pXqzzmbN2/m1KlTpKSkAFBaWoqHh8fvGqvt27fTvn17mjZtCkCHDh1wd3e3zef7+flhNlf+V/rzzz9n9+7d9OrVCwCr1cqZM2cACAkJYfTo0QQGBvLAAw8wbNiwy26vY8eOODg44OrqSrNmzcjPz2fv3r088MADNG7cGIAnnniC119/vdI1mkwmAFxcXBg0aBDffPMN7777LgcPHmTfvn0VnilcbLmLjfnp06dJTU0lPz/fdj3o9OnT7N27l27dulW61mudwuQalZaWxnfffcfQoUPp2LEjHTt25B//+AdhYWFs3boVq9VKhw4dmDdvnm2dY8eO4eXlxYYNG6hXr56t3WQyYfz3Fm/R0dH89NNPpKSk2IIpLi7usvWYTCZ69+5NYmIiTk5O9O7dG5PJdMk6zvfRRx8BMGrUKODsjrGgoIBVq1bRv3//y77+xQLn/GtHhmGU21kb593WrrS0tML1rVYrkZGRBAYGAlBYWEhxcTFw5WNlVHAbPcMwKCsru2QfLsZqtTJ06FD69esHnL2Odu56yNixY+nduzdfffWVbZoqPj7+ktur6D3h6OhYru4rvRa3e/dubrrpJq677jpee+01du3aRa9evWjXrh1lZWUVjsmllqtozF977TUMw2DZsmXUr18fOHu9xsXF5Ypqvdbp01zXKHd3d+Li4vjmm29sbdnZ2Zw5c4Y77riD9u3bs3XrVn755RcAtmzZQvfu3W07work5eURGBhIo0aNGDx4MM899xw//vjjBcs5OjradoDne/TRR9m0aRPr1q2jZ8+eAJWqw2KxsGLFCqZOncqmTZvYtGkTmzdvZsSIESxevNi2Uzv3mmazGYvFUuGO6Ld27NhBRkYGcPY6TkBAgG38zp0R/Prrr+X6ef5rPfjggyxZsoSSkhKsViuTJ09mzpw5lR6r850bi8OHDwOwbds2jh079rvn8R988EE+/vhjCgoKAJg/fz4TJkygrKyMhx9+mNOnT/P4448zZcoUfvnlF8rKyi76s7vUa2zbto3MzEzgf6FfGZmZmURHR9suvn/11VcMGjSIHj164OHhQUpKChaLBSg/5hdb7mJj7urqip+fH++++y4AJ0+e5PHHH2fjxo0XbFsuTmcm16hbbrmFN954g7lz5/J///d/uLi44ObmxrRp07j11lsBmDZtGv/4xz9sR+RxcXGXPPp1d3dn1KhRDB48mHr16uHo6FjhR0c7d+5Mv379iI2NLdfu6emJr68vZWVleHt7A9C8efPL1vH5559jtVovmCYaPHgwixcvZsuWLXTo0IFnnnkGJycnIiMj8fX1pWvXrnz44YeXHKc77riDyMhIcnJyuPXWW5k2bRpw9gwoIiKCLVu2cOutt5abJuvYsSOvvvoqpaWlPPXUU7z66qs8+uijWCwWWrZsSUREBK6urpUaq/PdfvvtTJkyhdGjR2OxWKhXrx7/+te/cHNzu+R6F9OnTx8yMzP529/+hslkokmTJsyaNQuz2UxkZCTjxo3DbDZjMpmYMWMGzs7O5cbx3KcAL+WWW25h4sSJDBkyBGdnZ1q2bGk7+q/IoEGDcHBwsJ3B9OrVy3Zm+fTTTzN79mxiY2NxdHTk3nvvtU27BgQE8PLLL19yuUu9P6Ojo3n55ZcJCwujpKSE0NBQunfvDsAjjzzC2LFjmT59Og8++ODvGutrgcmozOGZiMjvcPjwYRITE3nqqadwcHBg/fr1vPXWW1d0hiJ1g85MRKTaNG7cmKysLMLCwnB0dMTNzY0ZM2bUdllSDXRmIiIidtMFeBERsZvCRERE7KYwERERuylMRETEbtfsp7mOHy/EatVnD0REKsPBwcQNN1x30eev2TCxWg2FiYhIFdE0l4iI2E1hIiIidlOYiIiI3RQmIiJiN4WJiIjYTWEiIiJ2U5iIiIjdrtm/M5Gac0NDZ8zO+gpUgLKSYo7nl9R2GSJVTmEi1c7s7ELa7KG1XcZVoe2EtwGFifzxaJpLRETspjARERG7KUxERMRuChMREbGbwkREROymMBEREbspTERExG4KExERsZvCRERE7KYwERERuylMRETEbtUaJvPnz6dbt26EhITw7rvvAjBx4kSCgoIIDw8nPDycDRs2AJCSkkJYWBhBQUHMnTvXto09e/bQq1cvgoODmTRpEmVlZQBkZGTQv39/unTpwqhRoygsLKzOroiIyCVUW5h8/fXXbN++ndWrV7Ny5Uref/999u/fT3p6Oh988AGJiYkkJibSuXNnioqKiIyMJDY2lqSkJNLT09myZQsA48ePZ/Lkyaxbtw7DMFixYgUAU6dOpV+/fiQnJ9O6dWtiY2OrqysiInIZ1RYm9913H4sXL8ZsNpObm4vFYsHFxYWMjAwmT55MWFgYMTExWK1Wdu3aRbNmzWjatClms5mwsDCSk5M5evQoRUVF+Pn5AdCzZ0+Sk5MpLS0lNTWV4ODgcu0iIlI7qnWay8nJiZiYGEJCQujQoQMWi4X27dszY8YMVqxYwTfffMPHH39MVlYWnp6etvW8vLzIzMy8oN3T05PMzEyOHz+Oq6srZrO5XLuIiNSOav8+kzFjxjBs2DBGjhzJtm3beOONN2zPDRgwgISEBLp06XLBeiaTCcMwrqj9Snh4uF7R8iJVxdPTrbZLEKly1RYmv/zyCyUlJbRs2ZL69esTFBREUlISjRo1sk1PGYaB2WzG29ubnJwc27pZWVl4eXld0J6dnY2Xlxfu7u4UFBRgsVhwdHS0tV+J3NwCrNYLQ0mqnnae5WVnn6rtEkSumIOD6ZIH4dU2zXXkyBGioqIoKSmhpKSEjRs38pe//IUZM2aQn59PaWkpy5cvp3PnzrRp04YDBw5w6NAhLBYLa9asISAgAB8fH1xcXEhLSwMgISGBgIAAnJyc8Pf3JykpqVy7iIjUjmo7MwkMDGTnzp306NEDR0dHgoKCGD16NDfccAOPP/44ZWVlBAUFERoaCsCsWbN45plnKC4uJjAw0Db1FR0dTVRUFIWFhfj6+jJw4EAApkyZQkREBHFxcTRp0oQ5c+ZUV1dEROQyTEZFFyCuAZrmqjmenm76Dvj/ajvhbU1zSZ1Ua9NcIiJy7VCYiIiI3RQmIiJiN4WJiIjYTWEiIiJ2U5iIiIjdFCYiImI3hYmIiNhNYSIiInZTmIiIiN0UJiIiYjeFiYiI2E1hIiIidlOYiIiI3RQmIiJiN4WJiIjYTWEiIiJ2U5iIiIjdqjVM5s+fT7du3QgJCeHdd98FICUlhbCwMIKCgpg7d65t2T179tCrVy+Cg4OZNGkSZWVlAGRkZNC/f3+6dOnCqFGjKCwsBODkyZMMHz6crl270r9/f7Kzs6uzKyIicgnVFiZff/0127dvZ/Xq1axcuZL333+fvXv3EhkZSWxsLElJSaSnp7NlyxYAxo8fz+TJk1m3bh2GYbBixQoApk6dSr9+/UhOTqZ169bExsYCMG/ePPz9/Vm7di19+vThlVdeqa6uiIjIZVRbmNx3330sXrwYs9lMbm4uFouFkydP0qxZM5o2bYrZbCYsLIzk5GSOHj1KUVERfn5+APTs2ZPk5GRKS0tJTU0lODi4XDvA5s2bCQsLAyA0NJQvvviC0tLS6uqOiIhcQrVOczk5ORETE0NISAgdOnQgKysLT09P2/NeXl5kZmZe0O7p6UlmZibHjx/H1dUVs9lcrh0ot47ZbMbV1ZW8vLzq7I6IiFyEubpfYMyYMQwbNoyRI0dy8ODBC543mUwYhnFF7Rfj4FD5bPTwcK30siJVydPTrbZLEKly1RYmv/zyCyUlJbRs2ZL69esTFBREcnIyjo6OtmWysrLw8vLC29ubnJwcW3t2djZeXl64u7tTUFCAxWLB0dHR1g5nz2pycnJo3LgxZWVlFBQU0KhRo0rXl5tbgNV6YVhJ1dPOs7zs7FO1XYLIFXNwMF3yILzaprmOHDlCVFQUJSUllJSUsHHjRvr27cuBAwc4dOgQFouFNWvWEBAQgI+PDy4uLqSlpQGQkJBAQEAATk5O+Pv7k5SUVK4dIDAwkISEBACSkpLw9/fHycmpurojIiKXUG1nJoGBgezcuZMePXrg6OhIUFAQISEhuLu788wzz1BcXExgYCBdunQBIDo6mqioKAoLC/H19WXgwIEATJkyhYiICOLi4mjSpAlz5swB4NlnnyUiIoKQkBDc3NyIjo6urq6IiMhlmIyKLkxcAzTNVXM8Pd1Imz20tsu4KrSd8LamuaROqrVpLhERuXYoTERExG4KExERsZvCRERE7KYwERERuylMRETEbgoTERGxm8JERETspjARERG7KUxERMRuChMREbGbwkREROymMBEREbspTERExG4KExERsZvCRERE7KYwERERuylMRETEbgoTERGxW7WGyYIFCwgJCSEkJITZs2cDMHHiRIKCgggPDyc8PJwNGzYAkJKSQlhYGEFBQcydO9e2jT179tCrVy+Cg4OZNGkSZWVlAGRkZNC/f3+6dOnCqFGjKCwsrM6uiIjIJVRbmKSkpPDVV1+xatUqEhIS+M9//sOGDRtIT0/ngw8+IDExkcTERDp37kxRURGRkZHExsaSlJREeno6W7ZsAWD8+PFMnjyZdevWYRgGK1asAGDq1Kn069eP5ORkWrduTWxsbHV1RURELqPawsTT05OIiAicnZ1xcnLitttuIyMjg4yMDCZPnkxYWBgxMTFYrVZ27dpFs2bNaNq0KWazmbCwMJKTkzl69ChFRUX4+fkB0LNnT5KTkyktLSU1NZXg4OBy7SIiUjvM1bXh5s2b2/5/8OBBkpKSWLp0KV9//TXTpk2jQYMGjBgxgo8//pgGDRrg6elpW97Ly4vMzEyysrLKtXt6epKZmcnx48dxdXXFbDaXa78SHh6udvZQ5Pfx9HSr7RJEqly1hck5+/btY8SIEbzwwgvceuutvPHGG7bnBgwYQEJCAl26dLlgPZPJhGEYV9R+JXJzC7BaL9yOVD3tPMvLzj5V2yWIXDEHB9MlD8Kr9QJ8WloagwcP5vnnn+fRRx/lxx9/ZN26dbbnDcPAbDbj7e1NTk6OrT0rKwsvL68L2rOzs/Hy8sLd3Z2CggIsFku5dhERqR3VFibHjh3j6aefJjo6mpCQEOBseMyYMYP8/HxKS0tZvnw5nTt3pk2bNhw4cIBDhw5hsVhYs2YNAQEB+Pj44OLiQlpaGgAJCQkEBATg5OSEv78/SUlJ5dpFRKR2mIyK5oyqwPTp01m5ciU333yzra1v375YrVaWLFlCWVkZQUFBjBs3DoBt27Yxc+ZMiouLCQwMZOLEiZhMJvbu3UtUVBSFhYX4+voyc+ZMnJ2dOXr0KBEREeTm5tKkSRPmzJlDw4YNK12fprlqjqenG2mzh9Z2GVeFthPe1jSX1EmXm+aqtjC52ilMao7C5H8UJlJX1eo1ExERuTYoTERExG4KExERsZvCRERE7KYwERERuylMRETEbgoTERGxm8JERETspjARERG7VSpMKrq9+88//1zlxYiISN10yTA5ceIEJ06cYNiwYeTn59se5+Tk8NRTT9VUjSIicpW75PeZPP/882zduhWAdu3a/W8ls5lHHnmkeisTEZE645Jh8s477wAwceJEZs6cWSMFiYhI3VOpb1qcOXMmR48eJT8/v9y3HLZq1araChMRkbqjUmESHR3N+++/j4eHh63NZDKxcePGaitMRETqjkqFSVJSEuvXr8fb27u66xERkTqoUh8NbtKkiYJEREQuqlJnJh06dGD27Nl06tSJevXq2dp1zURERKCSYRIfHw9AcnKyra0y10wWLFjA2rVrAQgMDGTChAmkpKTYvuu9a9eujB07FoA9e/YQFRVFQUEB/v7+TJ06FbPZTEZGBuPHjyc3N5dbbrmF6OhorrvuOk6ePMm4ceM4fPgw7u7uzJs3D09Pz981CCIiYp9KTXNt2rTpgn+XC5KUlBS++uorVq1aRUJCAv/5z39Ys2YNkZGRxMbGkpSURHp6Olu2bAFg/PjxTJ48mXXr1mEYBitWrABg6tSp9OvXj+TkZFq3bk1sbCwA8+bNw9/fn7Vr19KnTx9eeeUVe8ZBRETsUKkzk3fffbfC9ieffPKi63h6ehIREYGzszMAt912GwcPHqRZs2Y0bdoUgLCwMJKTk7n99tspKirCz88PgJ49exITE0OfPn1ITU3ljTfesLU/8cQTjB8/ns2bN7NkyRIAQkNDmTZtGqWlpTg5OVWu5yIiUmUqFSY//fST7f8lJSWkpaWV+4v4ijRv3tz2/4MHD5KUlMSAAQPKTUV5eXmRmZlJVlZWuXZPT08yMzM5fvw4rq6umM3mcu1AuXXMZjOurq7k5eVV+oMCHh6ulVpOpKp5errVdgkiVa7Sf7R4vry8PCZMmFCpF9i3bx8jRozghRdewGw2c+DAgXLPm0ymcn8IWZn2i3FwqPxNkHNzC7BaL9y+VD3tPMvLzj5V2yWIXDEHB9MlD8J/1y3o3d3dOXr06GWXS0tLY/DgwTz//PM8+uijeHt7k5OTY3s+KysLLy+vC9qzs7Px8vLC3d2dgoICLBZLuXY4e1Zzbp2ysjIKCgpo1KjR7+mOiIjY6YqvmRiGQXp6erm/hq/IsWPHePrpp5k7dy4dOnQAoE2bNhw4cIBDhw5x0003sWbNGnr16oWPjw8uLi6kpaXRtm1bEhISCAgIwMnJCX9/f5KSkggLC7O1w9lPhyUkJDBy5EiSkpLw9/fX9RIRkVpiMiqaS/qNiRMnlnvs7u7OgAEDaNy48UXXmT59OitXruTmm2+2tfXt25c///nPto8GBwYGMnHiREwmE3v37iUqKorCwkJ8fX2ZOXMmzs7OHD16lIiICHJzc2nSpAlz5syhYcOGnDhxgoiICA4fPoybmxvR0dHcdNNNle64prlqjqenG2mzh9Z2GVeFthPe1jSX1EmXm+aqVJicc/ToUcrKymjWrFmVFFebFCY1R2HyPwoTqasuFyaVmuY6dOgQTz31FFlZWVitVm644QbefPNNbrvttiorVERE6q5KXYCfNm0aQ4cOJTU1lbS0NEaNGsXUqVOruzYREakjKhUmubm5PProo7bHvXr14vjx49VWlIiI1C2VChOLxcKJEydsj/Py8qqrHhERqYMqdc3kiSee4LHHHqNr164ArF27lkGDBlVrYSIiUndU6swkMDAQgNLSUvbv309mZiadO3eu1sJERKTuqNSZSUREBP3792fgwIEUFxfz4YcfEhkZyVtvvVXd9YmISB1QqTOT48ePM3DgQABcXFwYPHgw2dnZ1VqYiIjUHZW+AH/ubr0AOTk5Fd6EUURErk2VmuYaPHgwPXr04KGHHsJkMpGSklLpuwaLiMgfX6XCpHfv3rRu3Zrt27fj6OjIkCFDuOOOO6q7NhERqSMqFSYAd955J3feeWd11iIiInXU7/o+ExERkfMpTERExG4KExERsZvCRERE7KYwERERuylMRETEbtUeJgUFBYSGhnLkyBHg7PfJBwUFER4eTnh4OBs2bAAgJSWFsLAwgoKCmDt3rm39PXv20KtXL4KDg5k0aRJlZWUAZGRk0L9/f7p06cKoUaMoLCys7q6IiMhFVGuY7Ny5k8cff5yDBw/a2tLT0/nggw9ITEwkMTGRzp07U1RURGRkJLGxsSQlJZGens6WLVsAGD9+PJMnT2bdunUYhsGKFSsAmDp1Kv369SM5OZnWrVsTGxtbnV0REZFLqNYwWbFiBVOmTMHLywuA06dPk5GRweTJkwkLCyMmJgar1cquXbto1qwZTZs2xWw2ExYWRnJyMkePHqWoqAg/Pz8AevbsSXJyMqWlpaSmphIcHFyuXUREakel/wL+93jllVfKPc7NzaV9+/ZMmzaNBg0aMGLECD7++GMaNGiAp6enbTkvLy8yMzPJysoq1+7p6UlmZibHjx/H1dUVs9lcrl1ERGpHtYbJbzVt2pQ33njD9njAgAEkJCTQpUuXC5Y1mUwV3pn4Uu1XwsPD9YqWF6kqnp5utV2CSJWr0TD58ccfOXjwoG16yjAMzGYz3t7e5OTk2JbLysrCy8vrgvbs7Gy8vLxwd3enoKAAi8WCo6Ojrf1K5OYWYLXqNvo1QTvP8rKzT9V2CSJXzMHBdMmD8Br9aLBhGMyYMYP8/HxKS0tZvnw5nTt3pk2bNhw4cIBDhw5hsVhYs2YNAQEB+Pj44OLiQlpaGgAJCQkEBATg5OSEv78/SUlJ5dpFRKR21OiZyZ133snw4cN5/PHHKSsrIygoiNDQUABmzZrFM888Q3FxMYGBgbapr+joaKKioigsLMTX19f2jY9TpkwhIiKCuLg4mjRpwpw5c2qyKyIich6TcY1+ZaKmuWqOp6cbabOH1nYZV4W2E97WNJfUSVfVNJeIiPwxKUxERMRuChMREbGbwkREROymMBEREbspTERExG4KExERsZvCRERE7KYwERERuylMRETEbgoTERGxm8JERETspjARERG7KUxERMRuChMREbGbwkREROymMBEREbspTERExG7VGiYFBQWEhoZy5MgRAFJSUggLCyMoKIi5c+faltuzZw+9evUiODiYSZMmUVZWBkBGRgb9+/enS5cujBo1isLCQgBOnjzJ8OHD6dq1K/379yc7O7s6uyEiIpdRbWGyc+dOHn/8cQ4ePAhAUVERkZGRxMbGkpSURHp6Olu2bAFg/PjxTJ48mXXr1mEYBitWrABg6tSp9OvXj+TkZFq3bk1sbCwA8+bNw9/fn7Vr19KnTx9eeeWV6uqGiIhUQrWFyYoVK5gyZQpeXl4A7Nq1i2bNmtG0aVPMZjNhYWEkJydz9OhRioqK8PPzA6Bnz54kJydTWlpKamoqwcHB5doBNm/eTFhYGAChoaF88cUXlJaWVldXRETkMszVteHfni1kZWXh6elpe+zl5UVmZuYF7Z6enmRmZnL8+HFcXV0xm83l2n+7LbPZjKurK3l5eXh7e1dXd0RE5BKqLUx+yzCMC9pMJtMVt1+Mg8OVnWR5eLhe0fIiVcXT0622SxCpcjUWJt7e3uTk5NgeZ2Vl4eXldUF7dnY2Xl5euLu7U1BQgMViwdHR0dYOZ89qcnJyaNy4MWVlZRQUFNCoUaMrqic3twCr9cLAkqqnnWd52dmnarsEkSvm4GC65EF4jYVJmzZtOHDgAIcOHeKmm25izZo19OrVCx8fH1xcXEhLS6Nt27YkJCQQEBCAk5MT/v7+JCUlERYWZmsHCAwMJCEhgZEjR5KUlIS/vz9OTk5VWq/b9fWo51K126yriopLOXWyqLbLEJGrWI2FiYuLC7NmzeKZZ56huLiYwMBAunTpAkB0dDRRUVEUFhbi6+vLwIEDAZgyZQoRERHExcXRpEkT5syZA8Czzz5LREQEISEhuLm5ER0dXeX11nNxot+EJVW+3bpo6ez+nEJhIiIXZzIqujhxDbjcNJenp5vC5L+Wzu5v19SMp6cbabOHVmFFdVfbCW9rmkvqpMtNc+kv4EVExG4KExERsZvCRERE7KYwERERuylMRETEbgoTERGxm8JERETspjARERG7KUxERMRuChMREbGbwkREROymMBEREbspTERExG4KExERsZvCRERE7KYwERERuylMRETEbgoTERGxW419B/z5Bg4cSG5uLmbz2ZefNm0av/76K3FxcZSWljJ48GD69+8PQEpKCjNnzqS4uJiuXbsyduxYAPbs2UNUVBQFBQX4+/szdepU2/ZERKRm1fiZiWEY7N+/n8TERNu/xo0bM3fuXJYuXUpiYiLLly/n559/pqioiMjISGJjY0lKSiI9PZ0tW7YAMH78eCZPnsy6deswDIMVK1bUdFdEROS/ajxM9u/fj8lkYtiwYXTv3p0PPviAlJQU2rdvT6NGjWjQoAHBwcEkJyeza9cumjVrRtOmTTGbzYSFhZGcnMzRo0cpKirCz88PgJ49e5KcnFzTXRERkf+q8TA5efIkHTp04I033uC9995j2bJlZGRk4OnpaVvGy8uLzMxMsrKyKtXu6elJZmZmjfZDRET+p8YvMtxzzz3cc889ADRo0IDevXszc+ZMRo4cWW45k8mEYRgXrH+p9ivh4eF6Rctf6zw93Wq7hD8MjaX8EdV4mHzzzTeUlpbSoUMH4Ow1FB8fH3JycmzLZGVl4eXlhbe3d6Xas7Oz8fLyuqI6cnMLsFovDKVz9AtfXnb2qd+9rsayPHvGUqS2ODiYLnkQXuPTXKdOnWL27NkUFxdTUFDAqlWreO2119i2bRt5eXmcOXOG9evXExAQQJs2bThw4ACHDh3CYrGwZs0aAgIC8PHxwcXFhbS0NAASEhIICAio6a6IiMh/1fiZSceOHdm5cyc9evTAarXSr18/2rZty9ixYxk4cCClpaX07t2bu+++G4BZs2bxzDPPUFxcTGBgIF26dAEgOjqaqKgoCgsL8fX1ZeDAgTXdFRER+S+TUdEFiGtAZaa5+k1YUoMVXb2Wzu5v9zRX2uyhVVhR3dV2wtua5pI66aqb5hIRkT8ehYmIiNhNYSIiInZTmIiIiN0UJiIiYjeFiYiI2E1hIiIidlOYiIiI3RQmIiJiN4WJiIjYTWEiIiJ205emi9Qx1zd0wcXZubbLuCoUl5RwMr+4tssQFCYidY6LszOD3322tsu4Krz35HxAYXI10DSXiIjYTWEiIiJ2U5iIiIjddM1ERK5pjdyccarnUttlXBVKi4o5carkd62rMBGRa5pTPReSBj5Z22VcFbotfhd+Z5homktEROxWp8Pkk08+oVu3bnTu3JklS/R97SIitaXOTnNlZmYyd+5c4uPjcXZ2pm/fvrRr147bb7+9tksTEbnm1NkwSUlJoX379jRq1AiA4OBgkpOTGT16dKXWd3AwXXaZG2+4zp4S/1AqM16X4ny9RxVVUvfZO5YAN7q6V0ElfwxVMZ71b9T785yLjeflxtlkGIZRHQVVtzfffJPTp08zduxYAD766CN27drFyy+/XMuViYhce+rsNZOKMtBksv8IRURErlydDRNvb29ycnJsj7OysvDy8qrFikRErl11Nkzuv/9+tm3bRl5eHmfOnGH9+vUEBATUdlkiItekOnsB3tvbm7FjxzJw4EBKS0vp3bs3d999d22XJSJyTaqzF+BFROTqUWenuURE5OqhMBEREbspTERExG4KExERsZvC5L9++uknWrRowbp162xtW7ZsoWPHjjz//PMXLB8eHl4tdYwZM4awsLBq2TbA7t27mTRpUrVt//cqKChg6tSphIaGEh4ezoABA/jPf/5Ta/VU18/3alVYWMjUqVPp3Lkz3bt3p1+/fmzbtg2A5cuXs2bNGgAiIiKIj4+vzVJr1ZEjR3j44YcvaG/RosVF19mxYwcDBgyozrKuCnX2o8FVLT4+nuDgYJYtW0ZwcDAAycnJjBw5kscee+yC5RMTE6u8huPHj/PDDz9w4403kpaWRtu2bav8Ne666y7uuuuuKt+uPaxWK8OGDaNdu3YkJCRgNpvZvn07w4YN49NPP+WGG26o8Zqq4+d7tTIMg5EjR9KyZUs+/fRTnJ2d+eGHHxg+fDj//Oc/+e6777jvvvtqu0y5yilMgLKyMlavXs2SJUvo27cvv/76Kzt27GDjxo1s27YNBwcHVq9eTcOGDdm3bx/z5s2jR48e/Pjjj5w4cYJJkyaxf/9+nJ2diYiIoEOHDnzwwQckJiZy5swZTCYT8+bN47bbbuPhhx+me/fufPXVV5w5c4ZXX32V1q1bA2dvqe/v788dd9zB8uXLbWESHx/P5s2bycrK4v/+7/8YNGgQGRkZbN++nUaNGvH222/j4uJCQkICixYtwmq10qpVK6ZMmYKLiwvt27enVatW5OTkMGHCBP71r3/x/vvvs2fPHl588UWKiopo2LAh0dHR3Hjjjbz00kvs27ePnJwcbrnlFhYsWEBOTg6jR4+mefPm7NmzBw8PD+bPn2+70aY9duzYQVZWFmPGjMHB4ezJcvv27Zk5cyZWq5WoqKgK6xk4cCCbNm0C4PXXXwfgmWee4ZNPPiEuLg6TycRdd93Fyy+/TF5eHpGRkZw6dYrs7GxCQkIYN24ce/fu5cUXX6SsrAwXFxdmzpzJn//8Z1q0aMGPP/5IZmZmhevFx8fz5Zdfkp+fz+HDh3nggQd46aWX7B6L2vD111+TkZHB4sWLbbck8vX1ZdSoUQwePBg3Nze2b9+Op6cnAJs3b2bp0qXk5ubaDrYKCwuZNm0a+/btw2KxMGzYMEJDQ4mPj2fVqlWcOHGCjh078o9//KM2u1qtKvOeWLRoEZ999hkLFy5k+PDh3HXXXaSlpZGXl0dUVBSBgYHk5OQwadIkMjIyMJvNjB07ltatW9v2GwAPPfQQEydOpFu3bixcuBCTyURRURGZmZkcOnSIo0eP0qdPH0aNGlVj/dc0F2d/Of70pz9xyy238Mgjj7Bs2TL69OnDww8/zJgxY+jTpw+AbRqsZcuWtnXnz5/PzTffzNq1a5k9ezbz5s2joKCAzz77jPfff581a9bwyCOPsHTpUts6jRo14uOPP6Zv3768+eabtvb4+Hi6du1K165dWbduHSdOnLA9t3v3bt5++22WLFnCrFmzCAgI4JNPPgHgyy+/ZN++faxYsYJly5aRmJiIh4cH77zzDnD2jGf48OEkJiZiNv/v+GHcuHE89dRTtu+FWbRoEd999x1OTk4sX76cDRs2UFxczJYtWwDYu3cvTz75JGvWrOH666+3vb69fvjhB+666y5bkJwTGBjI/v37L1pPRTIzM5k5cyb//ve/+fTTT7FYLGzZsoU1a9YQGhrKihUrWL16NUuXLiUvL49Fixbx5JNPEh8fz4ABA/j+++/Lbe9i6wF89913xMTEsHr1aj7//HN+/PHHKhmPmrZ7925at259wb3t/vKXv1C/fn3b78FDDz0EQElJCR999BFvvvkmc+fOBSAuLo5WrVoRHx/PkiVL+Ne//sXhw4eBsz+TVatW/aGD5JxLvSdWrlzJ+vXrefPNN6lfvz4ApaWlLF++nIkTJzJ//nwAXn75Zdq3b88nn3xCTEwMkZGRWK1W/vSnP/HTTz/xyy+/YLFY+PrrrwH44osv6NixIwA//vgj77zzDh999BELFy7k5MmTNdZ3nZlwdiceGhoKQLdu3Rg3bhzPPffcBctV9Bf2qampREdHA2fDZvny5QD885//5NNPP+XgwYN8+eWX5QLo3C9l8+bNWb9+PQB79uzh2LFj3H///Tg5OdGyZUsSEhIYPHgwAPfeey+urq64uroC0KFDBwB8fHw4efIkO3bs4NChQ/ztb38Dzr5JfX19ba/Zpk2bcnXn5eWRnZ1texP269fP9lyjRo1YsmQJ+/fv5+DBg5w+fRoADw8P2zabN29Ofn7+pQe2khwcHCq8cSec3aFdrJ6KfPfdd9x77700btwYgNdee8323Pbt23nnnXfYt28fpaWlnDlzhsDAQKZNm8aXX35Jx44dbVOc5wwZMqTC9QDuuece28+jadOmVTYeNc1kMmGxWC5oLy0trXD5Tp06YTKZaN68OcePHwfOfiVEUVERK1euBOD06dPs27cPOHuWc/5BTF322wMeODtNeC6IL/ae+Omnn3jxxReZM2cODRo0sK17/r7g3MHj9u3bmT59um0bbdq0YefOnQQGBrJt2zbMZjMDBw7k008/tZ0xn/sep3bt2uHs7IyHhweNGjXi1KlTXH/99dUzGL/xx/gJ2yE3N5cvvviC9PR0Fi9ejGEYnDx50raTP1+9evUuaPvtL8kvv/xCvXr1GDRoEE888QQBAQHceOON7Nmzx7aMi4sLUP4uxytXrqSkpMS2MyssLGTZsmW2MHFycrrk61osFrp27UpUVJRt/fN3EL+t/bfbKy4uJisri59++omYmBgGDhxIz549OX78uG1Hf67uc7VX1c0TWrduzdKlS8v9UgLMmTOHu+++m9dff/2Cen77+mVlZZjN5gvG5dxZxMKFCzl8+DChoaE88sgjpKSkYBgGXbp04Z577uHzzz9n0aJFbNmyxfaLDDBr1qwK16vO8ahpbdq04f3336e0tLTc++L777+v8Pqao6MjUP79a7Vaee2112jVqhUAOTk5NGzYkE8++aTC35u66vrrr+fUqVPl2nJzc2nYsCFw8ffEddddx4wZM5gxYwYPPfSQLVAq2hf89n1kGAYWi4XAwEAWLFiAs7Mzzz77LGvXruWTTz6xBdKlXr8mXPPTXKtXr6Z9+/Z88cUXbNq0ic8//5yRI0fazjAux9/fn6SkJOBskAwbNoz09HSaNWvG4MGDadOmDV988UWFR37nlJSU8Mknn/Dee++xadMmNm3axMaNG8nOzmbHjh2VqqNdu3Zs2LCB3NxcDMPgpZdeYtGiRRdd3s3NjcaNG7N161bg7AXn+fPns23bNrp27UqvXr248cYbSU1NvWTtVcHf3x8PDw8WLFhge60vv/zSNgddUT3XX389+fn55OXlUVJSwpdffgmc/YDBzp07yc7OBmDGjBls3LiRrVu3MmTIELp27cqxY8fIzMzEarXy3HPPsWvXLvr27cuzzz7LDz/8UK62i633R+Lv78/tt9/OjBkzbGcj6enpxMXF8dRTT+Ho6HjZ90D79u358MMPgbN38O7evTvHjh2r9tprmqurK82aNSv3qc/ly5fbZgouxsfHh06dOnHfffcRExNzyWXbt2/Pxx9/DMDhw4f59ttv8fPzo1WrVhw4cICDBw9y22230a5dO+Li4vjrX/9qd7+qwjV/ZhIfH2/7gq1z+vXrx9tvv82NN9542fXHjBlDVFQU3bt3x2w2M3v2bFq2bMmyZcvo1q0bzs7O3H333bZT/op8/vnn+Pj4lJuKcnV1pU+fPixbtqzckcfF3HnnnYwePZpBgwZhtVpp2bIlw4cPv+Q6r732Gi+99BKzZ8/mhhtuYPbs2Rw/fpxx48aRnJyMs7Mzfn5+HDly5LKvbw+TyURsbCwzZ84kNDQUs9nMDTfcwMKFC3F0dKywHjc3N4YMGULv3r1p3Lix7Qja29ubSZMmMWTIEKxWK35+fvTs2ZP69eszYcIErr/+ejw8PGjdujVHjhxh5MiRTJo0idjYWBwdHYmIiChX24gRIypc749mwYIFzJ07l9DQUBwdHWnYsCGvvfYa7dq1Iycnhzlz5uDm5nbR9UePHs1LL71EaGgoFouF8ePHc/PNN/PNN9/UYC9qxrnfmzfeeIPS0lJatGjBiy++yObNmy+77oQJEwgNDb3kx/8nTZrEiy++aPsI9vTp021fr9G2bVvbNGv79u356KOPrppP2ulGjyIiYrdrfppLRETspzARERG7KUxERMRuChMREbGbwkREROymMBEREbspTERExG7X/B8titSEHTt28Morr9CgQQMKCwu59957+eGHHygsLMQwDKZPn07btm0pLCxk+vTpfPvttzg6OvLII48wduxYSktLiY6Ott0BwNfXl6ioKNt9oERqm8JEpIbs27ePzz77jKysLN59912WL1+Og4MDCxcu5K233qJt27bExMRQXFxMUlISFouFv//973z99dekpqbi6OhIfHw8JpOJOXPmEB0dXWdvey9/PAoTkRrSpEkTfHx88PHxoWHDhixbtozDhw+zY8cOrrvuOuDs3XcnTpyIo6Mjjo6OfPDBB8DZW3icOnWKlJQU4OwdfT08PGqtLyK/pTARqSHn7hS7efNmXnnlFZ588kk6derErbfeyurVq4Gzd4M+/w6yx44do169elitViIjIwkMDATO3hW6uLi45jshchG6AC9Sw7Zu3UrHjh3p168fd911F5999pntrrwdOnRg1apVWK1WSkpKGDNmDKmpqTz44IMsWbKEkpISrFYrkydPZs6cObXcE5H/UZiI1LC+ffuSmppKWFgYjz32GE2bNuXIkSNYrVZGjx6Nk5MT4eHh9OjRg8DAQIKCgnjqqafw8fHh0UcfpVu3bhiGccEdjkVqk+4aLCIidtOZiYiI2E1hIiIidlOYiIiI3RQmIiJiN4WJiIjYTWEiIiJ2U5iIiIjdFCYiImK3/wez8tDvBLJLqwAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 301 }, - { - "cell_type": "markdown", - "metadata": { - "id": "iyknfgrsW1gi" - }, - "source": [ - "## Comparing performance of different techniques" - ] + "id": "czIGZYhk2LjF", + "outputId": "07490aa7-c22e-4614-8580-be8b8b6db229" + } + }, + { + "cell_type": "code", + "execution_count": 45, + "source": [ + "sns.countplot(x=Y_test)\n", + "plt.title(\"Target Label Histogram for Test Dataset\")\n", + "outcome_test = figure_to_base64str(plt)" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZMAAAEXCAYAAABoPamvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlFUlEQVR4nO3de1xVdb7/8deGDahBKf4AHUunTCu11IZMtGDMEhBQ85IX8lKpkzreTqkIoml5nDwcb6V2mcY5ebRCJUpD9OQcdRrT1Gm8NDZmCQoWF0EUVK7f3x/+3L9IQWC5RfD9fDx8PNzftdZ3f9ba67Hfe32/ey9sxhiDiIiIBS61XYCIiNR9ChMREbFMYSIiIpYpTERExDKFiYiIWKYwERERyxQmt5DXXnuNvn370rdvXzp06EBwcLDj8cWLF532vOfOnWPEiBFXXfbGG28wb968avWXkJDA7373u2rXcd9995GTk3NF+/Dhw0lOTi7XlpaWRufOnQHYtm0br732WqV9b9++naVLl1a7phtp5cqV/Pa3v2XmzJk12j4xMdFxvnTp0oXHH3/c8Xjfvn3V6uta50TXrl0dfYeFhfHiiy9y/PhxS/1adfDgQWbPnu2UvusDe20XIDfOrFmzHP9/4okniIuL48EHH3T68+bl5XHo0CGnP4+z9OzZk549e1a6zqFDh8jLy7tBFdXM+vXriYuLw9/fv0bb9+vXj379+gEQFRVFmzZteOGFF2rU17XOid69e5d7405MTGTUqFF89tlneHp61rhfK44dO0ZGRoZT+q4PFCbC+fPneeWVV0hJSSEvL4/bbruNuLg47rnnHoYPH84dd9zBDz/8wNChQwkMDCQ6Opq8vDx8fHwwxtCnTx/69+/P3//+d+Li4rhw4QI2m42JEyfSo0cPZs6cycWLF+nbty8JCQm4urpWqa7169fz0UcfUVxcTF5eHmPGjGHYsGEAZGVl8cILL5CZmUmLFi149dVX8fHx4dy5c8yfP5+jR49SXFxMQEAA06dPx26v+amekJDAli1bePvtt9m6dSsrV67EZrPh6urK9OnTcXd358MPP6S0tBQvLy+mTp3K8uXL+eyzz3B1deXuu+8mNjYWHx8fUlNTr3r8unTpQmRkJK1btyY9PZ3Vq1eTkJDA559/TmFhIRcuXGDGjBk89dRTvPHGG5w4cYKTJ0+SmZnJQw89RPfu3UlMTCQtLY1p06YRHh5ebh+mTJlCRkYGMTExTJ48mYcffphXXnmF9PR0jDH069eP0aNHk5aWdkUdvr6+1zxGFb32WVlZzJgxg9zcXACCgoKYMmVKtc+Jfv368emnn7Jx40aGDh1a4bnxy34//vjjq65XUV0A69at44MPPqCsrIzGjRsTGxtLo0aNWLZsGefOnWPmzJksWLCgBmdSPWfkltSjRw9z8OBBY4wxmzdvNq+++qpjWWxsrJk3b54xxphnn33WzJw507HsmWeeMWvWrDHGGHPs2DHTsWNHs2HDBnPmzBnTq1cvc/LkSWOMMT/99JMJDAw06enp5uTJk6ZTp05XrWPZsmVm7ty5V7Tn5+ebZ555xuTk5BhjjPn6668dfWzYsMF06tTJpKSkGGOM+c///E8zefJkY4wxUVFR5v333zfGGFNSUmJefvll88477xhjjGnbtq05ffr0Fc/17LPPmh49epg+ffo4/oWGhpZ7vrFjxxpjjOnZs6f5+uuvjTHG/PWvfzVvvPHGFfuxfv16M3jwYFNQUOBY9vzzz1d6/E6ePGnatm1r9u7da4wxJi0tzQwfPtxcuHDBGGPMpk2bTHh4uKO/Hj16mLNnz5oLFy6YRx55xCxYsMAYY8z//M//mF69el31WP/8NY+MjDR/+tOfjDHGnD171kRERJhNmzZdUUdFZsyYYf74xz8aY0ylr/2bb75pYmNjjTHGFBQUmClTppizZ8/W6Jz4wx/+YF555ZVKz42f91vZehXVtWfPHjNs2DBz/vx5Y8yl1zg0NNQYU/48kCvpykQICQnhrrvuYvXq1aSmpvLVV1855gsAx7BIXl4eBw8e5L//+78BaN26NV27dgXgH//4B1lZWUyYMMGxnc1m41//+hdt2rSpdk233XYbb731Fjt27CAlJYVvv/2W8+fPO5Z369aNVq1aATBw4EAGDhwIXJq7OHToEOvXrweo8lzQ9OnTCQkJcTxOS0sjIiLiivXCwsL4/e9/T1BQEN27d2fMmDFXrLNz50769+9Po0aNABgxYgRvvfUW2dnZFR4/ALvdTqdOnQBo0aIFr7/+Ohs3biQ1NZUDBw5QUFBQbv+9vLwA8PX15fHHHwegZcuWnDlzptJ9PX/+PH//+9/505/+BICXlxf9+/dn586ddOzYsVwdVVHZa//4448zduxYfvzxR7p168ZLL72El5dXjYYEbTYbDRo0uOa5cVll61VU1/bt20lNTWXIkCGOfvLy8q55TEXDXAKsXbuW+Ph4IiMjiYiIoHHjxqSlpTmWX35TvDwUYX52O7fLbaWlpbRu3Zp169Y5lmVkZODt7V2jceaffvqJwYMH88wzz/Cb3/yGkJAQ/vd///eK571cz+VhrLKyMpYuXUrr1q0BOHv2LDabrdrPX5GpU6cycOBAvvjiCxISEnjnnXdISEgot475xe3uysrKKCkpwcPD44rlP98Pd3d3x3588803jB8/nlGjRtG9e3ceeeQR5s6dW27dn6vOMF5ZWVmFNf6yjqqo7LV3c3Nj27ZtfPnll+zevZtBgwaxfPnyKg2d/dKhQ4cYMGDANc+Nyypb76GHHrpqXWVlZfTt25dp06Y5jktmZiZ33HFHteu91ejbXMIXX3zB008/zaBBg7j77rv5y1/+Qmlp6RXreXp68vDDDzvePE+ePMmXX36JzWajU6dOpKamsnfvXgCOHDlCcHAwmZmZ2O12SktLr3gDq8zhw4fx9vZm/PjxPP744443gct17dmzh1OnTgHwwQcfEBgYCMBjjz3Gn//8Z4wxFBUVMW7cOMeVgFUlJSU88cQTnD9/nqFDhzJnzhy+//57SkpKcHV1dbwZP/bYYyQkJDg+Ba9evZpHHnkELy+vCo/fL+3du5cOHTrw3HPP0aVLF7Zt23bV16QmPD096dixI2vWrAEufQMqMTGRbt261ai/yl77uLg4VqxYwZNPPklMTAz33nsvKSkp1T4n1q1bR1paGqGhoZWeGz/vt7L1Kqqre/fufPbZZ2RmZgKXzq2RI0cClHuN5Uq6MhGef/55Zs+e7ZgIbd++PUePHr3quq+//joxMTGsXbsWPz8/7rzzTho0aIC3tzfLli1j4cKFFBYWYoxh4cKFtGjRgtLSUtq1a0doaCgffPABTZo0KddnfHw8H3/8sePxfffdx6pVq1i/fj0hISE0bNiQhx56CG9vb1JTUwFo27Yt0dHRZGdnc8899zi+XhwTE8P8+fOJiIiguLiYbt26MXr06OtynOx2O9HR0bz88svY7XZsNhv//u//jru7OwEBAUycOBE3NzdiYmL48ccfGTRoEGVlZbRq1Yq4uLhKj98vhYeHs3XrVnr37o2bmxsBAQHk5eWRn59/XfYlLi6OefPmkZCQQFFREREREfTv35/09PRq91XZaz9y5EiioqIIDw/H3d2d++67j/DwcFxdXSs9J5KSkti/fz82m42ysjLuvvtu3n//fTw8POjevXuF50arVq0c/f7Xf/0Xfn5+V12vorrc3d0ZM2YMzz//PDabDU9PT958801sNhudO3dmyZIlTJgwgeXLl1+X16E+sZnqfFyUW97KlSvp1asXrVu35ty5c/Tp04d3332Xe++9t7ZLqxN0/KS+0pWJVMuvf/1rpk6diouLC6WlpYwZM0ZvhNWg4yf1la5MRETEMk3Ai4iIZQoTERGxTGEiIiKWKUxERMSyW/bbXLm5BZSV6bsHIiJV4eJio0mT2ypcfsuGSVmZUZiIiFwnGuYSERHLFCYiImKZwkRERCxTmIiIiGUKExERsUxhIiIililMRETEslv2dyZWed3egAYebrVdhtxkLhYWc+5s1f7uvEh9ojCpoQYebgybvqa2y5CbzNqFkZxDYSK3Hg1ziYiIZQoTERGxTGEiIiKWKUxERMQyhYmIiFimMBEREcsUJiIiYpnCRERELFOYiIiIZQoTERGxTGEiIiKWKUxERMQyhYmIiFimMBEREcsUJiIiYpnCRERELFOYiIiIZQoTERGxzOlh8vrrrxMVFQXAkSNHGDBgAMHBwcTExFBSUgLAqVOniIyMJCQkhHHjxlFQUADA2bNnGTt2LKGhoURGRpKVlQVAUVER06ZNIzQ0lKeffprvv//e2bshIiKVcGqYfPnll3z88ceOx9OmTSM2NpYtW7ZgjCE+Ph6AuXPnMmzYMJKTk+nQoQMrVqwAYMmSJfj7+7N582YGDRrE/PnzAVi9ejUNGzZk8+bNREdHO8JKRERqh9PC5MyZMyxevJgXX3wRgPT0dC5evEinTp0A6N+/P8nJyRQXF7N3716Cg4PLtQNs376diIgIAMLDw9m5cyfFxcVs376dPn36APDII4+Qm5vLqVOnnLUrIiJyDU4Lk9mzZzN16lRuv/12ADIzM/Hx8XEs9/HxISMjg9zcXDw9PbHb7eXaf7mN3W7H09OTnJycq/b1008/OWtXRETkGuzO6HTdunU0b96cgIAAEhISADDGXLGezWarsL0iLi5Xz7+K2ivStKlntdYXqSofH6/aLkHkhnNKmCQlJZGVlUXfvn3Jy8vj/Pnz2Gw2srOzHetkZWXh6+uLt7c3+fn5lJaW4urq6mgH8PX1JTs7m2bNmlFSUkJ+fj6NGzfG19eXrKwsWrVqVa6v6jh9Op+ysiuDrKr0hiEVyco6V9sliFx3Li62Sj+EO2WYa9WqVWzatIlPPvmESZMm8cQTT7BgwQI8PDzYv38/AImJiQQGBuLm5oa/vz9JSUnl2gGCgoJITEwELgWUv78/bm5uBAUF8cknnwCwb98+PDw8+NWvfuWMXRERkSq4ob8ziYuLY8GCBYSGhnLhwgVGjBgBwJw5c4iPj6d3797s27ePKVOmADB58mT+8Y9/EBYWxtq1a5k9ezYAw4cPp6ioiLCwMObPn8/ChQtv5G6IiMgv2MzVJi1uAddjmGvY9DXXsSKpD9YujNQwl9RLtTLMJSIitxaFiYiIWKYwERERyxQmIiJimcJEREQsU5iIiIhlChMREbFMYSIiIpYpTERExDKFiYiIWKYwERERyxQmIiJimcJEREQsU5iIiIhlChMREbFMYSIiIpYpTERExDKFiYiIWKYwERERyxQmIiJimcJEREQsU5iIiIhlChMREbFMYSIiIpYpTERExDKFiYiIWKYwERERyxQmIiJimcJEREQsU5iIiIhlChMREbFMYSIiIpYpTERExDKFiYiIWKYwERERyxQmIiJimcJEREQsc2qYLF26lN69exMWFsaqVasA2LVrFxEREfTq1YvFixc71j1y5AgDBgwgODiYmJgYSkpKADh16hSRkZGEhIQwbtw4CgoKADh79ixjx44lNDSUyMhIsrKynLkrIiJSCaeFyVdffcXu3bv59NNP2bBhA6tXr+bbb78lOjqaFStWkJSUxOHDh9mxYwcA06ZNIzY2li1btmCMIT4+HoC5c+cybNgwkpOT6dChAytWrABgyZIl+Pv7s3nzZgYNGsT8+fOdtSsiInINTguTLl268P7772O32zl9+jSlpaWcPXuWVq1acdddd2G324mIiCA5OZn09HQuXrxIp06dAOjfvz/JyckUFxezd+9egoODy7UDbN++nYiICADCw8PZuXMnxcXFztodERGphFOHudzc3Fi2bBlhYWEEBASQmZmJj4+PY7mvry8ZGRlXtPv4+JCRkUFubi6enp7Y7fZy7UC5bex2O56enuTk5Dhzd0REpAJ2Zz/BpEmTGDNmDC+++CIpKSlXLLfZbBhjqtVeEReXqmdj06aeVV5XpDp8fLxquwSRG85pYfL9999TVFTEAw88QMOGDenVqxfJycm4uro61snMzMTX1xc/Pz+ys7Md7VlZWfj6+uLt7U1+fj6lpaW4uro62uHSVU12djbNmjWjpKSE/Px8GjduXOX6Tp/Op6zsyrCqKr1hSEWyss7Vdgki152Li63SD+FOG+ZKS0tj1qxZFBUVUVRUxLZt2xgyZAjHjx8nNTWV0tJSNm3aRGBgIC1atMDDw4P9+/cDkJiYSGBgIG5ubvj7+5OUlFSuHSAoKIjExEQAkpKS8Pf3x83NzVm7IyIilXDalUlQUBAHDhygX79+uLq60qtXL8LCwvD29mbixIkUFhYSFBRESEgIAHFxccyaNYuCggLatWvHiBEjAJgzZw5RUVGsXLmS5s2bs2jRIgAmT55MVFQUYWFheHl5ERcX56xdERGRa7CZq01M3AKuxzDXsOlrrmNFUh+sXRipYS6pl2ptmEtERG4dChMREbFMYSIiIpYpTERExDKFiYiIWKYwERERyxQmIiJimcJEREQsU5iIiIhlChMREbFMYSIiIpZVKUwu/0Gqnzt27Nh1L0ZEROqmSsPkzJkznDlzhjFjxpCXl+d4nJ2dzfjx429UjSIicpOr9Bb0L730En/7298AePTRR///RnY7Tz75pHMrExGROqPSMHnvvfcAmDlzJgsWLLghBYmISN1TpT+OtWDBAtLT08nLyyv3d9nbt2/vtMJERKTuqFKYxMXFsXr1apo2bepos9lsbNu2zWmFiYhI3VGlMElKSmLr1q34+fk5ux4REamDqvTV4ObNmytIRESkQlW6MgkICGDhwoX07NmTBg0aONo1ZyIiIlDFMElISAAgOTnZ0aY5ExERuaxKYfKXv/zF2XWIiEgdVqUwWbVq1VXbn3vuuetajIiI1E1VCpOjR486/l9UVMT+/fvL/SJeRERubVX+0eLP5eTkMH36dKcUJCIidU+NbkHv7e1Nenr69a5FRETqqGrPmRhjOHz4cLlfw4uIyK2t2nMmcOlHjBrmEhGRy6o1Z5Kenk5JSQmtWrVyalEiIlK3VClMUlNTGT9+PJmZmZSVldGkSRPefvttWrdu7ez6RESkDqjSBPy8efMYPXo0e/fuZf/+/YwbN465c+c6uzYREakjqhQmp0+f5umnn3Y8HjBgALm5uU4rSkRE6pYqhUlpaSlnzpxxPM7JyXFWPSIiUgdVac7k2WefZfDgwYSGhgKwefNmRo4c6dTCRESk7qjSlUlQUBAAxcXF/PDDD2RkZPDUU085tTAREak7qnRlEhUVRWRkJCNGjKCwsJAPPviA6Oho3n33XWfXJyIidUCVrkxyc3MZMWIEAB4eHowaNYqsrCynFiYiInVHlSfgMzIyHI+zs7MxxlxzuzfffJOwsDDCwsJYuHAhALt27SIiIoJevXqxePFix7pHjhxhwIABBAcHExMTQ0lJCQCnTp0iMjKSkJAQxo0bR0FBAQBnz55l7NixhIaGEhkZqXATEalFVQqTUaNG0a9fP6ZPn86MGTN4+umnGT16dKXb7Nq1iy+++IKPP/6YxMREvvnmGzZt2kR0dDQrVqwgKSmJw4cPs2PHDgCmTZtGbGwsW7ZswRhDfHw8AHPnzmXYsGEkJyfToUMHVqxYAcCSJUvw9/dn8+bNDBo0iPnz51s5DiIiYkGVwmTgwIGsWrWKdu3a0aFDB9577z0iIiIq3cbHx4eoqCjc3d1xc3OjdevWpKSk0KpVK+666y7sdjsREREkJyeTnp7OxYsX6dSpEwD9+/cnOTmZ4uJi9u7dS3BwcLl2gO3btztqCA8PZ+fOnRQXF9f0OIiIiAVVmoAHuP/++7n//vur3HGbNm0c/09JSSEpKYnhw4fj4+PjaPf19SUjI4PMzMxy7T4+PmRkZJCbm4unpyd2u71cO1BuG7vdjqenJzk5Ofj5+VW5RhERuT6qHCY19d133/G73/2OGTNmYLfbOX78eLnlNpvtqvMvlbVXxMWl6n+epWlTzyqvK1IdPj5etV2CyA3n1DDZv38/kyZNIjo6mrCwML766iuys7MdyzMzM/H19cXPz69ce1ZWFr6+vnh7e5Ofn09paSmurq6Odrh0VZOdnU2zZs0oKSkhPz+fxo0bV7m206fzKSu79pcIKqI3DKlIVta52i5B5LpzcbFV+iG8Rn9psSp+/PFHJkyYQFxcHGFhYQB07NiR48ePk5qaSmlpKZs2bSIwMJAWLVrg4eHB/v37AUhMTCQwMBA3Nzf8/f1JSkoq1w6XfkiZmJgIQFJSEv7+/ri5uTlrd0REpBI2U5Xv+NbAa6+9xoYNG2jZsqWjbciQIfz6179mwYIFFBYWEhQUxMyZM7HZbHz77bfMmjWLgoIC2rVrx4IFC3B3dyc9PZ2oqChOnz5N8+bNWbRoEXfccQdnzpwhKiqKkydP4uXlRVxcHHfeeWeV67seVybDpq+p8fZSP61dGKkrE6mXrnVl4rQwudkpTMQZFCZSX9XaMJeIiNw6FCYiImKZwkRERCxTmIiIiGUKExERsUxhIiIililMRETEMoWJiIhYpjARERHLFCYiImKZwkRERCxTmIiIiGUKExERsUxhIiIililMRETEMoWJiIhYpjARERHLFCYiImKZwkRERCxTmIiIiGUKExERsUxhIiIililMRETEMoWJiIhYpjARERHLFCYiImKZwkRERCxTmIiIiGUKExERsUxhIiIililMRETEMoWJiIhYpjARERHLFCYiImKZwkRERCxTmIiIiGUKExERsczpYZKfn094eDhpaWkA7Nq1i4iICHr16sXixYsd6x05coQBAwYQHBxMTEwMJSUlAJw6dYrIyEhCQkIYN24cBQUFAJw9e5axY8cSGhpKZGQkWVlZzt4VERGpgFPD5MCBAwwdOpSUlBQALl68SHR0NCtWrCApKYnDhw+zY8cOAKZNm0ZsbCxbtmzBGEN8fDwAc+fOZdiwYSQnJ9OhQwdWrFgBwJIlS/D392fz5s0MGjSI+fPnO3NXRESkEk4Nk/j4eObMmYOvry8ABw8epFWrVtx1113Y7XYiIiJITk4mPT2dixcv0qlTJwD69+9PcnIyxcXF7N27l+Dg4HLtANu3byciIgKA8PBwdu7cSXFxsTN3R0REKmB3Zue/vFrIzMzEx8fH8djX15eMjIwr2n18fMjIyCA3NxdPT0/sdnu59l/2Zbfb8fT0JCcnBz8/vyrV1rSpp6V9E6mIj49XbZcgcsM5NUx+yRhzRZvNZqt2e0VcXKp+oXX6dD5lZVf2X1V6w5CKZGWdq+0SRK47FxdbpR/Cb+i3ufz8/MjOznY8zszMxNfX94r2rKwsfH198fb2Jj8/n9LS0nLtcOmq5vI2JSUl5Ofn07hx4xu3MyIi4nBDw6Rjx44cP36c1NRUSktL2bRpE4GBgbRo0QIPDw/2798PQGJiIoGBgbi5ueHv709SUlK5doCgoCASExMBSEpKwt/fHzc3txu5OyIi8v/c0GEuDw8P/vCHPzBx4kQKCwsJCgoiJCQEgLi4OGbNmkVBQQHt2rVjxIgRAMyZM4eoqChWrlxJ8+bNWbRoEQCTJ08mKiqKsLAwvLy8iIuLu5G7IiIiP2MzV5uYuAVcjzmTYdPXXMeKpD5YuzBScyZSL91UcyYiIlI/KUxERMQyhYmIiFimMBEREcsUJiIiYpnCRERELFOYiIiIZQoTERGxTGEiIiKWKUxERMQyhYmIiFimMBEREcsUJiIiYpnCRERELLuhf89ERG6MJne4Y3f3qO0y5CZTUlRIbl6RU/pWmIjUQ3Z3D/YvHF3bZchN5jfT/wg4J0w0zCUiIpYpTERExDKFiYiIWKYwERERyxQmIiJimcJEREQsU5iIiIhlChMREbFMYSIiIpYpTERExDKFiYiIWKYwERERyxQmIiJimcJEREQsU5iIiIhlChMREbFMYSIiIpYpTERExDKFiYiIWKYwERERyxQmIiJiWZ0Ok40bN9K7d2+eeuop1qxZU9vliIjcsuy1XUBNZWRksHjxYhISEnB3d2fIkCE8+uij3HvvvbVdmojILafOhsmuXbvo2rUrjRs3BiA4OJjk5GR+//vfV2l7Fxeb5Rr+T5PbLPch9c/1OLeuB/fbm9Z2CXITqun5ea3t6myYZGZm4uPj43js6+vLwYMHq7x9k+sQBMtm9rPch9Q/TZt61nYJADz44uu1XYLchJx1ftbZORNjzBVtNtvN8YlQRORWU2fDxM/Pj+zsbMfjzMxMfH19a7EiEZFbV50Nk27duvHll1+Sk5PDhQsX2Lp1K4GBgbVdlojILanOzpn4+fkxdepURowYQXFxMQMHDuShhx6q7bJERG5JNnO1yQcREZFqqLPDXCIicvNQmIiIiGUKExERsUxhIiIililMxBLdbFNuZvn5+YSHh5OWllbbpdR7ChOpscs321y7di2ffPIJH330EceOHavtskQAOHDgAEOHDiUlJaW2S7klKEykxn5+s81GjRo5brYpcjOIj49nzpw5ujPGDVJnf7Qotc/qzTZFnGn+/Pm1XcItRVcmUmO62aaIXKYwkRrTzTZF5DKFidSYbrYpIpdpzkRqTDfbFJHLdKNHERGxTMNcIiJimcJEREQsU5iIiIhlChMREbFMYSIiIpYpTESqqXPnztW6C+2hQ4eYNGkSAAcPHmT27NnX3ObcuXNMmjSJ8PBwevfuzTvvvONYlpKSwrBhw+jduzcDBw7k+++/r/Y+hIeHs2fPnmpvJ1IRhYmIkz344IMsW7YMgGPHjpGRkXHNbZYuXYqfnx+bNm1i/fr1fPjhh3z99dcAvPzyywwdOpSkpCQmTpzIpEmTrnprG5EbST9alHptz549zJ8/n0aNGnH+/HkmTZrE22+/TXFxMQ0aNGDGjBl07tyZ7OxsZs+ezenTp8nKyqJFixYsWbKEpk2bsm/fPl599VVsNhsPPvggZWVljr4XLVqEr68v3333HQ0bNmTixImsXr2a48eP06tXL6Kjo9mzZw+vvvoq7777LsuWLePcuXPMnDmTBQsWVFh3TEwMpaWlAGRlZVFUVISXlxcZGRn88MMPhIWFARAUFMTcuXP55z//Sfv27Svs79ixY0RHR3PhwgXuuecezp8/71j21ltv8fnnn1NYWMiFCxeYMWMGTz75JCEhIcTGxvLYY48BMGvWLNq0acNjjz1GTEwMRUVFGGMYOHAgkZGRll8rqeOMSD22e/duc//995u0tDRz/PhxEx4ebnJycowxxhw9etR0797dFBQUmD//+c/m7bffNsYYU1ZWZkaPHm3ee+89U1hYaLp162Z27dpljDFm48aNpm3btubkyZNm9+7d5oEHHjDffPONMcaYF154wQwePNgUFhaa06dPm/bt25uffvrJ7N6924SFhRljjNmwYYMZO3Zslet/6aWXTIcOHczUqVNNSUmJ+frrr01wcHC5dYYMGWI+//zzSvvp27eviY+PN8YYs2/fPnPfffeZ3bt3m7S0NDN8+HBz4cIFY4wxmzZtMuHh4cYYY1atWmUmTZpkjDHm3LlzpmvXriYvL8/MnDnTcawyMzPNlClTTGlpaZX3SeonXZlIvde8eXNatGjBmjVryMzMZNSoUY5lNpuNEydOMHLkSPbt28eqVatISUnhu+++o2PHjhw9ehS73U5AQABwaa7h53Med955J+3atQOgZcuWeHl54e7ujre3N7fddht5eXmWao+Li2Pu3LlMmjSJ5cuXO64SfsnV1bXCPnJzc/nXv/5Fv379APjNb35DmzZtAGjRogWvv/46GzduJDU1lQMHDlBQUABA//79Wb58OTk5OSQnJ/Pb3/6W22+/naeeeooZM2Zw8OBBAgICmDVrFi4uGjG/1ekMkHqvUaNGAJSVlREQEMAnn3zi+BcfH0+bNm34j//4D5YuXUqTJk0YPHgw3bt3xxiDzWa7Yj7Cbv//n8Hc3d0rXGbFX//6V8fcym233UZYWBj//Oc/+dWvfkV2dna5mjIyMmjWrFmFfV3+swA/3+Zynd988w1DhgwhPz+f7t27M3r0aMc6t99+OyEhIXz66ads2LCBoUOHAtCjRw+2bNlCaGgoR44cISIighMnTlyX/Za6S2Eit4yuXbvyt7/9zfHtpx07dtCnTx8KCwv54osvGDlyJP369aNp06bs2rWL0tJS2rZtizGGHTt2ALBt2zZLVxuurq6UlJRcc73NmzezfPlyjDEUFRWxefNmunbtSrNmzWjZsiVJSUnApdBxcXGhbdu2FfbVuHFj2rdvz7p164BLAXL06FEA9u7dS4cOHXjuuefo0qUL27Ztc8zVAERGRvL+++9jjHHcxPOll14iKSmJsLAw5syZg6enJz/++GONj4nUDxrmkltGmzZtmDdvHv/2b/+GMQa73c7KlStp1KgREyZMYOHChaxYsQJXV1cefvhhTpw4gZubG8uXL+eVV15h0aJFPPDAAzRt2rTGNXTu3JklS5YwYcIEli9fXuF6UVFRzJkzh4iICGw2Gz179mTEiBEALFq0iNjYWFauXIm7uztLly695jDTokWLmDlzJh9++CEtW7bknnvuAS4N223dupXevXvj5uZGQEAAeXl55Ofn4+npyf33388dd9zBkCFDHH2NHz+emJgYPvroI1xdXXnyySfp0qVLjY+J1A+6a7CIVOjEiRMMHz6c5ORkGjZsWNvlyE1MVyYiteCHH35g6tSpV1129913s2TJkmr1t3v37gq/avzoo48SHR1d3RJZunQp8fHxxMTEKEjkmnRlIiIilmkCXkRELFOYiIiIZQoTERGxTGEiIiKWKUxERMQyhYmIiFj2fwF9nO2Sz3IDbAAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 301 }, - { - "cell_type": "markdown", - "metadata": { - "id": "Vee-7c2Tw33O" - }, - "source": [ - "Now we have covered two different class of techniques for mitigating the fairness-related harms we found in our fairness-unaware model. In this section, we will compare the performance of the models we trained above across our key metrics." + "id": "YjhW0t9-2LjF", + "outputId": "0785b4e1-88b8-4ab3-cb0b-9bf1d3df5ffe" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Train the model" + ], + "metadata": { + "id": "4V523GQbYobT" + } + }, + { + "cell_type": "markdown", + "source": [ + "We train a logistic regression model and save its predictions on test data for analysis." + ], + "metadata": { + "id": "g7jwN2cVbO0g" + } + }, + { + "cell_type": "code", + "execution_count": 46, + "source": [ + "unmitigated_pipeline = Pipeline(steps=[\n", + " (\"preprocessing\", StandardScaler()),\n", + " (\"logistic_regression\", LogisticRegression(max_iter=1000))\n", + "])" + ], + "outputs": [], + "metadata": { + "id": "f6nKDzt164vw" + } + }, + { + "cell_type": "code", + "execution_count": 47, + "source": [ + "unmitigated_pipeline.fit(X_train_bal, Y_train_bal)" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
Pipeline(steps=[('preprocessing', StandardScaler()),\n",
+       "                ('logistic_regression', LogisticRegression(max_iter=1000))])
StandardScaler()
LogisticRegression(max_iter=1000)
" + ], + "text/plain": [ + "Pipeline(steps=[('preprocessing', StandardScaler()),\n", + " ('logistic_regression', LogisticRegression(max_iter=1000))])" ] + }, + "metadata": {}, + "execution_count": 47 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 111 }, - { - "cell_type": "markdown", - "metadata": { - "id": "XEXnEeLl7mgc" - }, - "source": [ - "#### Model performance - by group" - ] + "id": "ld9clGbHl7tv", + "outputId": "89fa7f0d-680f-424f-a7cd-96686987a943" + } + }, + { + "cell_type": "code", + "execution_count": 48, + "source": [ + "Y_pred_proba = unmitigated_pipeline.predict_proba(X_test)[:,1]\n", + "Y_pred = unmitigated_pipeline.predict(X_test)" + ], + "outputs": [], + "metadata": { + "id": "Ok-eREU0xbAD" + } + }, + { + "cell_type": "markdown", + "source": [ + "Check model performance on test data." + ], + "metadata": { + "id": "nkA0K8KV0HeD" + } + }, + { + "cell_type": "code", + "execution_count": 49, + "source": [ + "# Plot ROC curve of probabilistic predictions\r\n", + "plot_roc_curve(unmitigated_pipeline, X_test, Y_test);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEJCAYAAACUk1DVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABFz0lEQVR4nO3dd3xN9//A8VduIolIJEQSqyhB7E2EGkUJsRKb2quK71dLqdnWrGpV6aDV4SutnxE0HYTaRWvETqiZQbbsdcf5/ZG6lRI3ieTejPfz8fB49Jxzzznv973Nfd/zOZ/z+ZgpiqIghBCi1FOZOgAhhBBFgxQEIYQQgBQEIYQQf5OCIIQQApCCIIQQ4m9SEIQQQgBSEIQQQvzNwtQBPI+HD1PQ6fL+GIWjoy2xscmFEFHRJTmXDpJz6ZDfnFUqMypUKJfj9mJdEHQ6JV8F4dG+pY3kXDpIzqVDYeQsTUZCCCEAKQhCCCH+JgVBCCEEYISCkJycjJeXF2FhYU9sCwoKwsfHh549e7JgwQI0Gk1hhyOEECIHhVoQLl68yPDhw7l79+5Tt8+ZM4dFixaxf/9+FEVh+/bthRmOEEKIZyjUgrB9+3aWLFmCs7PzE9vCw8NJT0+nefPmAHh7e7Nv377CDEcIIYo1naKQodYW2vELtdvp8uXLc9wWFRWFk5OTftnJyYnIyMjCDEcIIYqNhOQMgkPiiY5P42Z4AgnJmdyLTAJg+uBmtKzjWODnNNlzCE+bl8fMzCxPx3B0tM33+Z2c7PK9b3ElOZcOknPxpNUp3A6PJ/juQ37+/Tbh0SnZtltaqPBoWgWXiuVo3cAFR/uyBR6DyQqCi4sLMTEx+uXo6OinNi09S2xscr4eznBysiM6OinP+xVnknPpIDkXL/cikrh6N47gkIdcuR2nX29va0nXFtVo5urIC8522NtaonrsB7Ojfdl85axSmT3zh7TJCkK1atWwsrLi3LlztGrVij179tCpUydThSOEEIUuKTWT09ciOXbhPuEx2a8AWtSthGt1exrUrECtyuVNEl+uCkJmZibh4eGYm5tTpUoVypQpk+8TTpo0iZkzZ9KkSRPWrFnDwoULSUlJoWHDhowePTrfxxVCiKJGURT+Ckvgp1N3CY1KJiE5E4Dy5Syp6WJH49oVaV63EjVd7LAwN/1jYWbK0xrz/xYcHMyGDRs4ceIEtra2aLVaMjIy6Nq1K1OmTKFevXrGjPUJ0mSUe5Jz6SA5Fw33IpI4eC6UU1ci0f39FVunanka1KrIi5XtaF63Up7vmT4uvznnu8no008/5dy5cwwaNIj333+fcuWyRshLTk7mxIkTLF++nDZt2jB9+vQ8ByWEECVNRqaWfX+G8Osf98hU6wCwLVsGj8aV6dSsKlUr5TzKaFGRY0Fwc3Pj9ddff2K9ra0tvXr1olevXhw8eLBQgxNCiKIs8K9ojgTe535MCrGJ6fr1nZpVoWuL6tSsXLx6P+VYELp162Zw5+7duxdoMEIIUZTpdAph0cmcCY7iwl8x+hvDVpbmeDSuTOv6zjSt44hKlf/mIFMq1vMhCCFEYUpNVxN07yF/XIvkr7AEElIy9dvMVWb4dK5NlxbVKGed/442RUmOBeGbb7555o7jxo0r8GCEEMLUQiKTuBmewOHAcP3DYRbmKqo42tDM1RGXijY0qe1IVcdyxfZKICc5FoQbN26wb98+evXqZcx4hBDC6BRF4dKtWPxP3uX2/UQg64awe0MXWtZzoplrJcpYmL5baGHLsSCsXLmS+/fv0759e/r162fMmIQQotClpqs5HBjOjdAErt6JQ6coWFma09ejFm3cnKnqVC7b08GlwTPvISxevJjvv/9eCoIQosRIy9Bw4Ewoe07c0a9r7eZMgxoOeDSpglUZcxNGZ1rPLAh16tRh0aJFxopFCCEKzV9h8fx2Loxz16PR6hRerFKe7q2r08bNuUg8JVwUSC8jIUSJdTcikR9P3OV66EPSMrSYq8xo08CZLs2rUe8FB1OHV+RIQRBClCiKonD6WiRngqK4cDNrRGUrS3MGdHyRHm1eoKyVfO3lRN4ZIUSJ8CA2Bf+Td7l4M4a0jKxZxVrVc8K7c22qOBb9YSOKAikIQohiKyE5g/O3YvntzxCC7j0EwMbKggEdX6RXuxpYluIbxPlhsCCsX7+eGTNmGFwnhBDGcvVOHD/+foe/whL067q1qk67Bi64Vrc3YWTFm8GC8LTRsZ8xYrYQQhQKRVEIDonnxKUHnLoaAUDHJlXw7PgiFW3KlOruogXFYEGYOXNmrtYJIURhSEjJ5LdzoZy+GklMQtaIoi3qVmKspxt2NpZFcj6E4krGMhJCFElxiemcuPyAX07dI1Ojo5pTOYa+7Ip7Qxfsba1MHV6J9MyxjIQQwtiu3onj0PkwAv/K6jJa3akcI3vUo36NCiaOrOR75lhGj0tMTKR8edNM/CyEKPnuRSThd+w2l2/HAtC4dkX6d3iROtXkJrGxGLyHcOfOHaZPn05iYiI7d+5k7NixbNiwgTp16hgjPiFECXc/JoV9f4Rw4vIDIKu3kJdHLezLWZo4stLHYEFYunQp8+fP54MPPsDFxYVRo0axePFifH19jRGfEKKECo9OZtfR2/qnid0bujCoSx0qlrc2cWSll8ERneLj4+nQoYN+eeTIkSQnJxdqUEKIkitDreWz3ZdZtPlPLtyMoUHNCqyc7M7kfo2kGJhYrp5UzsjIwOzvccGjo6PR6XSFGpQQouSJSUjjTFAUe07cQa3R0aq+E96dZFiJosRgQRgxYgQTJkwgNjaWDz/8kJ9//pmJEycaIzYhRAmQnKZmy/7rnA2OAsDOpgwT+jSgbQMXE0cm/s1gQRg0aBA1a9bkyJEjaDQa3nvvPTp27GiM2IQQxVjUw1R2Hr3Nhb+i0WgV6r/gQJ/2NWlc29HUoYkc5KrJyNXVleTkZCwsLGjatGlhxySEKKauhzzk1z9CCL73kExNVtNyg5oV8O5UW7qPFgMGC8KRI0eYO3cudevWRafTERISwtq1a2nTpo0x4hNCFANxielsDbih7zFU1sqcZjUdGdmjHpUcypo4OpFbBgvCunXr2Lp1K3Xr1gXg6tWrLFq0CD8/v0IPTghRdKVlaDhx6QGBf0UTHBIPQP0XHHhtQGPKyzMExZLBgmBmZqYvBgCNGjWS0U6FKMXUGh0Hz4Wy/48QElPVVCxvxcstq9GtVXXpMVTM5VgQ4uPjAWjcuDGbN29m2LBhqFQq/Pz8cHd3N1Z8QogiQqPVcTgwnN/OhRH1MI3qTrZM8GpI4xcr6ruli+Itx4Lg7u6OmZmZ/mrggw8+0G8zMzNj7ty5hR+dEKJI+P3yA7YG3CBDnTVR/ZR+jWjXULqNljQ5FoTg4ODnPri/vz+ff/45arWasWPHMnLkyGzbr169yuLFi1Gr1VSpUoUPPvhABtATogi5F5HEziM3uXr3IRXsrBjZox7tG7tgrjI4yIEohgzeQ8jMzOTo0aOkpKQAoNVqCQkJYdasWc/cLzIykrVr1+Ln54elpSXDhg2jXbt2uLq66l+zfPlyZs6cSefOnVm1ahWbN282eFwhROG7ERrPL6fvcelW1sijLepWYlLfhlhbyjTsJZnBT3fWrFmEhoYSHR1Nw4YNuXjxIm3btjV44JMnT+Lu7o6DgwMAPXv2ZN++fUyfPl3/Gp1Opy80aWlp2NtLP2UhTEWj1XHgbCiHzoURm5iBucqMXu1q0KP1C1SwkwlpSgODBSEoKIiAgADeeecdxo0bh6IovPvuuwYPHBUVhZOTk37Z2dmZS5cuZXvNvHnzGDduHCtWrKBs2bJs3749T8E7Otrm6fWPc3Kyy/e+xZXkXDrkJecMtZZTlx9w5FwoN8PiSUjOpEqlcgzs8gJDutfDtmyZQoy04MjnXDAMFgRnZ2csLCyoVasWN27cwNPTk7S0NIMHflrX1Md7IqSnp7NgwQK+++47mjZtyjfffMPcuXPZtGlTroOPjU1Gp8t7F9jSOAer5Fw65CXn4xfv43vwBpnqrCeKq1Uqx5hebjR3rQRAWnI6acnphRZrQZHPOfdUKrNn/pA2WBBsbGzw9/fHzc2N7du3U7t2bX2X1GdxcXHh7Nmz+uWoqCicnZ31yzdu3MDKyko/FMbQoUNZt26dweMKIfJPrdHy27lwTl+NICQqmQp2VoztVYc2DZzlRrEwPB/CokWLCAoKokOHDqhUKkaNGsX48eMNHtjDw4NTp04RFxdHWloaAQEBdOrUSb+9Zs2aREREcPv2bQB+++03mjRp8hypCCFyoigKv5y+x9QPj7L98E1CopIZ+NKLrJrSHvdGlaUYCADMlEJ87Njf35+NGzeiVqsZNGgQkyZNYtKkScycOZMmTZpw9OhRPvzwQxRFwdHRkaVLl/LCCy/k+vjSZJR7knPp8LScg+7GsfmXIOISM3BysMbLoxYdmlRBVUIeJpPPOfcMNRnlWBD69u37zAP7+/vnOZiCJgUh9yTn0uFRzjpF4eTlCA6eDSUkKhkrS3P6edSiR5sXsDAvWVcDpflzzqt830NYtGhRnk8mhDCt2IQ0vv01mFNXI1BrdFhbmuPlUYtX2rxQbHoMCdPJsSDk5lkDIUTR8DApg70n7nDs4n0A6lQtT8NaFenXsZbcHxC5Jo8dClGMaXU6dh25zb4/QwBoXs+Jbi2q0ejFiiaOTBRHUhCEKKYi4lJZt+MikQ/TqOFsy/g+DWjVuGqpa08XBSfXBSExMVEGnhOiCHiYlMGOwzf541okKpUZ4zzd6Ni0igxBLZ6bwcbF27dv06dPH/r06UNkZCSenp7cunXLGLEJIR6Tkq7G79gt3vr8JKevRVK7ankWjWnNS82qSjEQBcLgFcKyZcuYP38+H3zwAS4uLowaNYrFixfj6+trjPiEKPVS0zX8+sc9jl64T3KamsoVbXi1Z30a1Kxg6tBECWOwIMTHx9OhQwf9BDkjR47M8yB0Qoi8uxeRxE8n73LuRjQA9raWzB3Rgvo1pBCIwpGrewgZGRn6S9Lo6Gh0Ol2hBiVEaZaYkslne65wIzQeALcaDnRoUgWPxpWlaUgUKoMFYfjw4UyYMIHY2Fg+/PBDfv75ZyZOnGiM2IQoVRRF4dD5cHwP3ACghostrw1ojEsFGxNHJkoLgwVh8ODB1KpViyNHjqDRaHjvvffo2LGjMWITolTQ6RSOXrzPTyfv8jApg0r21ozxdKNRLXmWQBiXwYLw4YcfMmTIEObMmWOMeIQoVe5FJLFlfzB3HiRRyd6aV3vW56WmVUrceEOieMjVPYRRo0ZRs2ZNhgwZwiuvvIKlpWVhxyVEiZaWoWH9rksEh8QDMLJHPbq2qIZKJfcIhOnkavhrnU7H8ePH8fPz48KFC/Ts2ZP58+cbI75nktFOc09yLhpiE9I5FBjGkcD7pGVoaFLbkXG93XCwLZg5i4tizoVNcs69554xLesgKho3bsydO3e4c+dOtpnQhBCGRcensePwTc5ez+pCWquyHYO7usqzBKJIMVgQAgIC9FcGvXr1YuXKlTRq1MgYsQlR7F26FcvWgOvEJGTNTdy2gTPdWlWnbnUH0wYmxFMYLAhff/01Q4YMYe3atZQtW9YYMQlR7KWkq/nS/xqXbsUCUMnemqn9G1O7qowHJoquHAtCcnIytra2fPHFF0DWw2kZGRn67Q4ODoUenBDFTaZay9aAG5y4/ACAbi2r4925NmWtZGBhUfTl+H/pq6++yu7du3F3d8fMzIzH7z2bmZkRFBRklACFKA50OoU9J27z08l7AFRxtGFwV1eau1YycWRC5F6OBWH37t0AXLt2DdW/ZlyKj48v1KCEKE4u3Ypl55GbhEWn4FjeisFdXWnbwMXUYQmRZwavY318fPTF4ZERI0bwyy+/FFpQQhQH10Mesv3wLe48SATAp3NtervXlPGGRLGVY0EYM2YMly9fJj09nZYtW+rX63Q6GjRoYJTghCiK0jI0+B27zW/nwihnbUFv95p0bl4VJwfpdCGKtxwLwqeffkp8fDzz589n5cqV/+xgYYGTk5NRghOiKNEpCr+evseuo7eBrIns3xjaXG4YixLjmf8nV69enU8++eSJ9YmJidLLSJQqt8IT+PbXYMJjUqjhYkv/ji/S3LWSNA+JEkV6GQnxDLEJ6fz4+x2OX3qAtaV51phDLauhkkIgSiCDvYyCg4ONFowQRUVymprdx25zODAcgHYNXRjZox62ZcuYODIhCo/Bxs+YmBguXrxIt27dWLFiBcHBwcyfPx83NzdjxCeEUak1OnYfv82BM6FodQoNalZg6Muu1HCxM3VoQhQ6gwVh3rx5dOzYkVOnTnH69GnGjh3LsmXL2Lp1qzHiE8JoHiZlsGZbIA9iU2lax5G+HrWoU83e1GEJYTQGZ+GIj49n7NixHDt2DC8vL7y9vUlLSzNGbEIYTdTDVFZuPUd0fBrjezfgv4ObSTEQpY7BgqBWq1Gr1Rw/fhwPDw/S0tJITU01RmxCFDpFUTh1JYIl35whMTWT1wY0pmPTKqYOSwiTMFgQunXrRvv27alQoQKNGzdm8ODBeHl55erg/v7+9O7dmx49euDr6/vE9tu3b/Pqq6/Sr18/JkyYQEJCQt4zECKf0jI0rPjfOb786Rq21mWYM6wFLerKMzai9MrVjGkRERFUrlwZyOp1lJsbypGRkQwfPhw/Pz8sLS0ZNmwYH330Ea6urkDWL7NevXqxYMECOnXqxJo1a1AUJU9zN8uMabknOf9DURROXonA79htHiZlMLBTbTzb1SgR8xjL51w6mGzGNJ1Oh7+/P8eOHUOj0dChQwdcXV2xsHj2ridPnsTd3V3/AFvPnj3Zt28f06dPB+Dq1avY2NjQqVMnAKZOnUpiYmJu8xIizzLVWo5dvM+vf4TwMCmDMhYqxnm68VKzqqYOTYgiwWBB+PDDDwkODmbMmDHodDr+7//+j9WrVxucUzkqKirbEBfOzs5cunRJvxwSEkKlSpWYO3cu165do169eixatOg5UhEiZ5dvx/Ltr8E8TMqgkr01w7vX5eWW1TBXFf+rAiEKisGCcPz4cXbt2kWZMlkP5HTp0oV+/foZLAhPa4l6/DF/jUbDn3/+ydatW2nSpAkff/wxq1atYtWqVbkO/lmXPoY4OZW+fuWlLWdFUQi8HcfR82FcuhkDwNtj2tC+SZUSPeREafucQXIuKAYLgqIo+mIAYGlpmW05Jy4uLpw9e1a/HBUVhbOzs37ZycmJmjVr0qRJEwC8vLyYOXNmnoKXewi5V9pyDo9J4dt9wdwKy+qo0K1Vdbza18Te1oqYmGQTR1d4StvnDJJzXhi6h2DwetnNzY0VK1YQEhJCSEgIK1eupF69egZP7OHhwalTp4iLiyMtLY2AgAD9/QKAFi1aEBcXpx8a49ChQzRq1Cg3OQnxTDdC41n5v3OERibT16MWX83tysge9bC3tTJ1aEIUaQavEJYsWcLSpUsZNmwYiqLQsWPHXLX1u7i4MGvWLEaPHo1arWbQoEE0bdqUSZMmMXPmTJo0acKnn37KwoULSUtLo3LlyqxevbpAkhKlU3xyBtsP3+T01Ugq2VuzavpLqLRaU4clRLHxzG6nOp2uSA91LU1GuVfScz59LYJNP14DoGkdR8Z6ulH3xUolOuenKemf89NIzrmX726nFy5cYPr06cTGxlK3bl02bNhAjRo18hyAEIXpr7B4Vm09jwI42Fry2oDG1K3uYOqwhCiWcryHsHr1apYuXUpgYCDe3t6sWbPGmHEJ8Uyp6RrWbr/Iyq3nKVNGRefmVVn9mocUAyGeQ45XCKmpqXTt2hWAsWPHsnPnTqMFJUROFEXhTHAUvgdukJSqxr2hC8O718XOxtLUoQlR7OVYEFT/emAnN11NhSgsiqJw/kYM/ifvEBKZjGN5a94c2ohGL1Y0dWhClBg5FoRcDHEkhFE8TMrgi71X+CssAZWZGYO71qF7q+qUsTA3dWhClCg5FoTQ0FCmTp2a4/IXX3xRuJEJAdyNSGSD32USkjN5pc0LDHypNlaWUgiEKAw5FoQFCxZkW+7Zs2ehByPE4w6cDeWHg38B8J9BTWnmWsnEEQlRsuVYEDp37kzFis9un42NjcXR0bHAgxKlW4Zay9b91/n9SgTVnWwZ19uNF6uUN3VYQpR4ORaE+fPn065dO7y9vbG3zz6VYHJyMtu2beOPP/7gyy+/LPQgRemQnKbmm1+CCPwrayC66k7lmDeyJTbWBh+oF0IUgBz/0j777DO+/vprvLy8ePHFF6lZsyZarZbQ0FDu3LnD6NGj+eyzz4wZqyihdDqF3688YMu+62h1CmUsVAzqXIfurauX6FFJhShqntntdOLEiYwaNYrTp09z+/ZtVCoVr7zyCh4eHlhaSr9v8fwy1FrW77rEtbsPqWBnxbBudWnj5mx4RyFEgTN4LW5tbU2XLl3o0qWLEcIRpcnN8AQ+3Z3Vg6i5ayWm+zRBJVcEQpiMNM4Kk9h19BY/n7qHucqMyX0b4t6osqlDEqLUk4IgjO5IYDg/n7pH3er2TOnXiIrlrU0dkhACKQjCyPb9EcL2wzep4WzL7GEtKGMhcxoLUVQY/GtMSUnh3XffZcyYMcTHx7N48WJSUlKMEZsoQRRFYceRm2w/fJOale14a0RLKQZCFDEG/yKXLVtG+fLliY2NxcrKiuTkZBYvXmyM2EQJcft+Im98+ju/ng7BrYYDc0e0kGcLhCiCDBaEoKAgZs2ahYWFBWXLlmXNmjUEBQUZIzZRzCmKwvbDN1m25ax+LKLZw1tgbSnFQIiiyOBf5r+HwdZqtU+sE+LfMjK1fLbnCpdvx1K3uj3TBjSWSe6FKOIMFoQ2bdrwwQcfkJ6ezvHjx/H19aVt27bGiE0UU/4n7/LzybtkanR0b12dYd3qyvMFQhQDBn/qz549GxsbG+zs7Fi7di3169dn3rx5xohNFDNanY5dR2+x+9ht7G0teXNoc0Z0ryfFQIhiwuAVwtGjR3n99dd5/fXX9ev27NnDgAEDCjMuUczEJaazdsdFwqNTaFLbkeneTaQXkRDFTI4F4dChQ2g0GlavXo2iKPoZ1DQaDWvXrpWCIPQu3Yrhq5+CSE5TM7xbXbq1ri5XBUIUQzkWhKCgIE6fPk1sbCxbtmz5ZwcLCyZMmGCU4ETRd+zifb79NRj7cpbMG9mSei84mDokIUQ+5VgQHjUT+fr6MnLkSGPGJIoBrU7Hjyfu4n/yLjVcbJk1pDn25WQEXCGKM4P3EAYPHsyBAwf0TydrtVpCQkKYNWtWoQcniqbkNDXLt5wl8mEabjUceN27CeWsy5g6LCHEczJYEGbNmkVoaCjR0dE0bNiQixcvSrfTUiwmIY1VvueJS8zAy6Mm/Tu+iLk8lyJEiZCrJ5X9/Pzo1q0b8+fPZ9u2bSQlJRkjNlHEhEQmsXzLOZJS1bw1vAXenepIMRCiBDH41+zs7IyFhQW1atXixo0buLq6kpaWZozYRBESfO8ha7ZdIF2t5fWBTXCrWcHUIQkhCpjBJiMbGxv8/f1xc3Nj+/bt1K5dm/j4eCOEJooCtUbL7mN32P9nyN8Pm7WkZmU7U4clhCgEBq8QFi9eTFBQEB06dEClUjFq1CjpdlpK3AxLYPHmP9n3ZwhVncqxaEwbKQZClGAGC0KtWrV46623MDMz4+OPP+bMmTO0adMmVwf39/end+/e9OjRA19f3xxfd+TIEV5++eXcRy0K3eHAcFZsPUfkwzTG9KrPe+PbUsFOBqcToiTLsckoLCyMtWvX4uDgwOzZsylbtizJycmsX78eX19frly58swDR0ZGsnbtWvz8/LC0tGTYsGG0a9cOV1fXbK+LiYnh/fffL5hsxHPLVGvxPXCD45ceYG1pzjvj2+LsUNbUYQkhjCDHK4T58+dToUIFoqOj2bhxI2fPnsXT05MTJ07w1VdfGTzwyZMncXd3x8HBARsbG3r27Mm+ffueeN3ChQuZPn3682UhCoRWp+OTXZc4fukBHZpU5uMZHaUYCFGK5HiFEBERwZYtW0hPT8fb25vt27czduxYxo8fj4WF4QlOoqKicHJy0i87Oztz6dKlbK/ZsmULDRs2pFmzZvkK3tHRNl/7ATg5lb628GflrNbo+PiH81y7+5DB3eoyundDI0ZWeORzLh0k54KR4ze7jY0NANbW1iQkJLB69Wo6duyY6wM/GgzvcWaPDXh248YNAgIC+Pbbb4mIiMhLzHqxscnodE+exxAnJzuio0vXsxTPyllRFDb4XSbwrxheafMCnm1eKBHvj3zOpYPknHsqldkzf0jn6qkiR0fHPBUDABcXF2JiYvTLUVFRODs765f37dtHdHQ0Pj4+TJ48maioKEaMGJGnc4jn9+9iMKxbXVOHJIQwkRwLwuO/5s3NzfN8YA8PD06dOkVcXBxpaWkEBATQqVMn/faZM2eyf/9+9u7dy6ZNm3B2dub777/P83lE/ukUhY+2XyTwrxha1Xdi6MuuhncSQpRYOTYZXb9+nZYtWwKQnp6u/29FUTAzM+P8+fPPPLCLiwuzZs1i9OjRqNVqBg0aRNOmTZk0aRIzZ86kSZMmBZiGyI8t+4K5eieOV9q8wJCurtl+BAghSh8z5WmN/UB4ePgzd6xWrVqhBJQXcg8h9/6d8+5jt/E/eZeuLaox6pV6JbIYyOdcOkjOuWfoHkKOVwhF4QtfFI7D58PwP3mXWpXtGNmjZBYDIUTeyVCVpcy561H8L+AGTg7W/GdwM1QqKQZCiCyGHygQJca561F8uvsKTg7WvDehHVZl8t5ZQAhRcskVQilxNiiST3dfwc6mDHNHtJRiIIR4gsGCEB0dzeTJk+nZsycxMTFMmDCBqKgoY8QmCohGq2Pzj1ewMFfx3vi2VCxvbeqQhBBFkMGC8O6779K9e3esrKywt7fHzc2NhQsXGiM2UQAyMrV8vucKYVHJTPRqgL2tjFgqhHg6gwUhPDycIUOGoFKpKFOmDHPmzOHBgwfGiE08B52icPh8GG99cZLAv2IY3bsBbRu4mDosIUQRZvCmspmZGTqdTr+cnJycbVkUPYqisPmna5y6GonKzIwZPk14xaN2qeurLYTIG4MF4ZVXXmH27NkkJSWxbds2duzYgaenpzFiE/m0+ecgTl2NpGU9J6b2b4SFufQdEEIYZrAgTJ06lT179qDT6Th58iRDhw5l8ODBxohN5MOxi/c5eSWCTs2qMrpXfVTy0JkQIpcMFoRt27bh5eXFgAEDjBCOeB73IpLwPXCDF6vYMbJHXSkGQog8MdiW8Oeff9K9e3fefvttLly4YISQRH7EJKTx4f9dwKqMOVP7N6aMhTxnIITIG4NXCB999BEJCQn89NNPLFu2jIyMDAYNGsSYMWOMEZ/IhZiENFb5niclTc2ScW1wkmkvhRD5kKu7jfb29gwdOpQpU6ZQtmxZvvzyy8KOS+RSXGI6K7eeJzlVzRtDm1PDpfRNJSiEKBgGrxCuXr2Kn58f+/bto2HDhkycOJGXX37ZGLEJAyIfprJ8yzlS0tTMHt6CBjUrmDokIUQxZrAgvP766/j4+LBjxw6qVq1qjJhELpwNjuLLn66h0ylM7NtQioEQ4rkZLAhHjhwxQhgiL/4MiuSLvVcBWDK2DTUrSzOREOL55VgQhg8fzg8//ECLFi2yTaCS2yk0RcFTFIW9J+7w4+93ca5QlrdHtpSxiYQQBSbHgrBu3ToAfvrppye25TDrpihEiqLwf4duEnAmlMa1K/Ja/8aUtZLpLIQQBSfHXkbOzs4ALFmyhGrVqmX798YbbxgtQAGp6RrW7rhIwJlQ3Go48N/BzaQYCCEKXI7fKjNnzuTOnTuEhobSt29f/XqNRoNKJWPjGItao+Wj7Re4fT+RXu1qMLhLHZkDWQhRKHIsCG+99Rbh4eEsWrSIRYsW6debm5tTt25dowQnsgaqu30/kVdfqUfXltVNHY4QogTLsSBUr16d6tWrs3//fvlFaiK7jt7iz6AourWsLsVACFHoDPYyatmypfQyMoEff7/Dz6fuUbe6PcN7yBWZEKLw5auXkShcB86Esuf4HZq7VmJKv0YyaqkQwigM9jKqWLEi0dHRVKtWjYMHD7JhwwZpQipEfwZF8sNvf+FazZ5pAxtjZSmjlgohjMNgd6G3336b3377jUuXLrFlyxaqVq2a7SazKDg3wxPY+ONVnB3KMt2nicx0JoQwKoPfOKGhobz55pscPnyYgQMHMmPGDOLj440QWumSkJLJxr1XsC1bhvmvtqK8jaWpQxJClDIGC4JarQbgxIkTuLu7o9VqSU1NLfTAShO1RsvH2y/yMCmT1wc2oXw5KQZCCOMz+Lhry5Yt6d27N+bm5rRs2ZIxY8bg4eFhjNhKBZ2i8PGOS9yLTGKspxv1XnAwdUhCiFLKYEFYtGgRgYGBuLm5oVKpmDBhAp06dcrVwf39/fn8889Rq9WMHTuWkSNHZtt+8OBB1q9fj6IoVK9enZUrV2Jvb5+/TIqhpNRMPt19hRuh8Xi616BTMxleXAhhOgYLgrm5OVFRUezatQu1Wk2HDh1yNXRFZGQka9euxc/PD0tLS4YNG0a7du1wdXUFIDk5mXfeeYddu3bh4uLCunXrWL9+PQsXLnz+rIqBOw8SWfbdWRTAp3NtervXNHVIQohSzuA3++bNm9m4cSP169enUaNGfPvtt3z++ecGD3zy5Enc3d1xcHDAxsaGnj17sm/fPv12tVrNO++8g4uLCwD169fnwYMHz5FK8REVnzUHsrWVBf8Z1JQ+7WtJV14hhMkZvELYs2cPP/zwA7a2tgAMGjSIIUOG8Nprrz1zv6ioKJycnPTLzs7OXLp0Sb9coUIFunfvDkB6ejqbNm3i1VdfzVcSxUmmWsuGXZdRa3S8ObS53DMQQhQZuRpD+VExALCzs8PCwvBuT5sz4Wm/gpOSkpg2bRpubm4MHDgwN+HoOTraGn5RDpycjD/LmKIoLPziJGHRybw+qBkdWr5g1PObImdTk5xLB8m5YBj8Zq9WrRrfffcdI0aMAMDX1zdXcyu7uLhw9uxZ/XJUVJT+6efH102YMAF3d3fmz5+f19iJjU1Gp8v7ZD1OTnZERyfleb/noSgKvgducOlmDC+3rEYrV0ejxmCKnE1Nci4dJOfcU6nMnvlD2uA9hHfffZeDBw/SvHlzmjdvTkBAAEuWLDF4Yg8PD06dOkVcXBxpaWkEBARk652k1WqZOnUqnp6eLFiwoES3oesUhS/9r3HofDhNajsyokc9U4ckhBBPMHiF4OLiwv/+9z/S0tLQ6XSUK1cuVwd2cXFh1qxZjB49GrVazaBBg2jatCmTJk1i5syZREREcO3aNbRaLfv37wegcePGLF++/PkyKoL2HL/D6WuRdGtVnRHd65bo4ieEKL7MlBwmSL579y5vvvkmd+7cwd3dnaVLl+Lo6Gjs+J6pODQZ3QiN5/3vz9OynhPTBjQ2WTGQy+rSQXIuHYzeZPTee+8xcOBAduzYQc2aNVm9enWeT17aJaVmsvHHq1Sws+LVnvXlykAIUaTlWBBiYmIYNWoUderUYfbs2Vy9etWYcRV7cYnprNx6noTkTCZ5NZTB6oQQRV6O9xAe71pqbm6eq66mIsu1u3Gs3X4RnU5hhk9T6teoYOqQhBDCoBy/5f99a0GaO3InNiGdNdsuoDIz442hzWn0YkVThySEELmSY0GIiIhg2bJlOS6XljGH8iIxNZMPtgVirjJj7siWuFYrPQP1CSGKvxwLwr9HJv33snjSd78GE/UwjVlDmkkxEEIUOzkWhOnTpxszjmLvSGA4gX/F4Olegya1i1b3XCGEyA2ZtLcA3L6fyJb916lTtTw+neuYOhwhhMgXKQjPKSk1k8/3XKGctQVT+zdGJTffhRDFlBSE55CQksmSr/8kLjGdCX0a4mhvbeqQhBAi3wwWBJ1Ox1dffcXcuXNJTk5m48aNaLVaY8RWpOkUhU/9LpOQksmkvg1pXreSqUMSQojnYrAgrF69mhs3bugntzl+/DgrV64s9MCKup2Hb3EzPAGfznVwb1TZ1OEIIcRzM1gQTp06xapVq7CyssLW1pavv/6a33//3RixFVmXb8ey/88QWtV3wrNdDVOHI4QQBcJgQbCwsECl+udllpaWpXoYi7jEdDb/HISTQ1nGeTaQJ7iFECWGwW/2evXq4evri1ar5fbt23z77be4ubkZI7YiR6vTsenHqySmZDL91VbYWJfewiiEKHkMXiEsWLCAq1evEhsby/Dhw0lJScnXdJclwaYfr3EjLAGfzrXlSWQhRIlj8Ceura0tK1asMEYsRdqt8ATOBEfRpLYjfdrXMnU4QghR4AwWhMcHtHtcaRrc7n5MCl/9dA3bsmWY6NXA1OEIIUShMNhk5ODgoP9Xrlw5AgMDjRFXkZGWoWH19+eJTUxnSv9G2MlEN0KIEsrgFcK/B7mbMmUKU6ZMKbSAihKdorDxx6skpqr57+CmNKolcxsIIUquPHeTsbGxISoqqjBiKXJ2Hb3FpVux9Glfk6Z15ElkYRxarYaHD6PRaDLzvG9UlAqdTlcIURVdkvOTVCpzypa1xdbWPk9d4w0WhKVLl+oPqCgKV69epXbt2rk+QXF17noUv54OoUHNCgzsVPLzFUXHw4fRWFvbUK5c5Tw/52JhoUKjKV1fjpJzdoqioNVqSEqK5+HDaCpWdM79cQ29oEKF7PMB9+vXj379+uX6BMVRfHIGX/8SxItV7Hh9YBMZwVQYlUaTma9iIARkTXdsYVEGBwdHIiPD8rSvwYIQEhLC6tWr8x1ccZORqWXNtgukZWgZ59lAHj4TJiHFQDwvMzMVoORpH4O9jK5fv46i5O2gxZWiKKzbeZH7MSmM6F6X6s62pg5JCCGMxmBBqFSpEn369OHtt99m2bJl+n8l0bnr0QSHxDOwU226t37B1OEIYXIPHtynSxd3xo4dwbhxIxg1agj//e80oqIiAZg9eyYxMdH5Ou6gQX0B+OqrLzhx4miBxLtjx7Zsx7p9+yYdO7bmyJHfsr1u+vTJnD9/Ntu65cvf4Zdf/IGsH4fbtm1l7NgRjB07gvHjR3Lw4P58x6VWq1m6dBEjRw5i/PiR3Lt396mv+/HH3YwfP5Jhw7zx9f1Ov/6rr75g1KjBjBo1hG3btgKQmprK/PlzCnQ6ghzbQzIzM7G0tKRFixa0aNGiwE5YVN0IjefzvVeoWN6KV9pIMRDikUqVnPj22+/1y198sYG1az9g5co1rFnzyXMff+LEqc99DIC4uFhOnDjGunWf6df9/LM/Xbp0Y8+eXXTp0i3Xx9q06TNu3LjOhg2bsLW1JSoqkunTJ2Nv70CbNu3yHNuOHduwti6Lr+9OLlw4z7JlS/jyy++yvebixQts27aVTZu+Q6VSMX78SDw8XiI+/iHnzp3h229/QKvVMGrUEF56qRPVqtWgdeu27N3rh7f34DzH9DQ5FoShQ4eye/fuJ55DKIku3Yrlsz2XsSpjzlsjWmJVxtzUIQlRZDVr1oITJ44BMGhQX9av30hg4DmOHj1EYmIiDx/G0aHDS0yfPgszMzP+979vOXz4AFqtjnbt3HnttZnZjrd8+Tu0aNGKFi1aMX/+bGrXrsONG9epWNGRpUtXUb68PadPn2Tz5i/QaDRUqVKNuXMXYG/vkO04fn476Nr1Zf2yRqMhIOBXPv30S157bTzh4WFUq1bdYH6pqals3/49W7fuwNY2q9nY2dmFd99dgZVV9lkRIyMjmDv3jSeO8dlnX2JjU06/fOrUCX3ha968JQkJ8URERFC58j9zqRw6FMDAgYP151y79lPs7e158cXarF+/EQsLC2JiotFqtZQtWxaA7t1fYfLkcQwcOKhA7jvlWBBKy32Ds8FRfLbnCuVtyvDG0OY4O5Q1dUhCZPP75QecuPQgV681M4O8/Ol2bFqFDk2q5Pr1Go2GQ4cO0KRJsye2BQdf45tvvsfOrjwzZkzh2LHDWFlZc/16EF9+uQUzMzOWLl1MQMCvNG3a/KnHv3nzL95+ezH16rmxYMEcAgJ+pVu3nnzxxQY++eQLypcvz549u/j88/XMm7co274nThxjyZJ/mrNPnTpB5cqVqVGjJi+91IW9e3cxbdp/DOYYEnIXG5tyVKlSNdv6Bg0aPfFaF5fK2a6echITE42j4z/PMjk6ViI6OjJbQQgLC8PW1o7p0yeTnJyMl1c/Bg0aBmRNQ7B580Z++OF/dO3aHScnZ7RahfLl7bGxKcvNm39Rt249g3EYkmNByMjI4Nq1azkWhkaNnnxzipsHsSl8ty+Y8uUsWTaxHbZly5g6JCGKnJiYaMaOHQGAWp1JgwaNeO21J1sOOnToRMWKjgB06/YK586dwdLSimvXrjBhwqsAZGSk4+JSOceCUKFCRerVyxpev3ZtVxITE7l27QqRkRHMnJn1C1un01K+/JOjDYeFheDs/E+f+19+8ad7955/x9ODd99dxKRJ0yhTpky2OV4eURQFMzMzzMxUuf5BnNsrhKcdLqsX0D+0Wg2XL19k9eqP0Wg0zJgxmdq1XWnZsjUAEyZMYeTIMcydO4u9e/3w8hoIgItLFcLCQgq3IISGhjJjxoynvjFmZmb89ttvT9mr+IhLTGft9ouotTpm+DSVYiCKrA5Ncv8rvjAe0vr3PYScmJv/83WiKDrMzS3Q6bQMGTKcYcNGAZCUlIS5uTkJCfFPPYalZfaxwhRFQafT0rRpM95/fy2Q9WM1NTX1iX3NzFT6GB4+jOPUqd8JDg5ix45tKIpCUlIiR478Ro8evbCzsyM5OTnb/g8fPsTOrjy1atUiIyP9iSadgwf3ExcXx5Ahw/XrcnuF4OTkRGxsLNWrZ92fjI2NoVIlp2yvcXR0pF49N2xsbABo1649QUFXcXSsRGZmBnXr1sfa2ppOnbpy8+Zf+v3+PYnZ88jxKK6urvz2228cOnToiX+5LQb+/v707t2bHj164Ovr+8T2oKAgfHx86NmzJwsWLECj0eQ/kzxQFIUv/a8Rm5jOrMHNqPeCg1HOK0RJdvr0SZKTk8nIyODAgf24u3vQsmUb9u//hdTUVDQaDW+//eYTPX4MadiwMVevXiYk5B4A3377FZ99tu6J11WrVp2IiKymtf37f6FVq7bs3v0LO3f6s2vXT4wePZ69e/0AaNWqLfv2/az/zrl37y7Xr1+jceOmWFlZ4+09hA8/XElKSlbRePDgPhs3fkatWi/m671p374D+/b9DGTdPLa0tMpWbAA8PDpx7NgR1Go1GRnpnD17hvr1G3D/fhjvv7+czMxM1Go1J04cpVmzfzr6PHhwn2rVCqYjTKE9dRUZGcnatWvx8/PD0tKSYcOG0a5dO1xdXfWvmTNnDsuWLaN58+bMnz+f7du3M2LEiMIKSe/0lQdcD41nWLe61K9RwfAOQgiDKlSowJw5M4mPj6dnz960a9cegJs3bzB58lh0Oi3t2nng6eml/+LODUfHSsybt5jFi99Gp9Pi5OTC4sXvPfG6Dh1e4vz5s9Sq9SK//OLP5MmvZ9vu7T2Y77/fwr17d+nXbyDh4WGMHTsClcoMKysrlixZhoODAwCTJ0/jm2++ZMqUcZibW2BurmLq1Om0beuer/fGx2coH3ywglGjhmBpWYZFi7LiDw6+xldffcGaNZ/QrVsPwsNDGTduJBqNhl69etO6dVsArl27yvjxI1GpVHTu/DI9evREo9GRlJREcnIyrq518xXXv5kpOTSWLVu27LnmPNi9ezdnzpzRT67z6aefoiiKvtdSeHg4Y8aM4eDBgwCcPXuWTz75hC1btuT6HLGxyeh0ebv5naHWsuSbM6jVWt6f2h4L84K51CrqnJzsiI5OMnUYRlVcc46IuEflyjXzta+pxvX55Rd/AgPPsWDBO0Y/96OcY2NjWLz4bT799Eujx2Bsj3Levv0HzM3N8fEZ8tTX/fv/JZXKDEfHnB+4zfEK4XknwImKisLJ6Z82MmdnZy5dupTjdicnJyIjI/N0jmcllpPQyCQSkzN469XWVKlcuqbBdHKyM3UIRlccc46KUmFhkf8fKs+zb36pVGZ/j6Fjmh9YFhYqXFyc6dKlK7//fpTOnbuaJA5jysxM59y5P3n//Q9zfN9VKlWe/gYKrckop5vRud2eG/m5QrBWwfdLPYl/mFosfz3mV3H9tfw8imvOOp0u37/yTXWF0KuXF716eZnk3I/nPHhwVpNzSR/91MJChaWltf5Ge0756nS6bH8Dhq4QCq2cu7i4EBMTo1+OiorK1iXs39ujo6OzbS9MZSzkwTMhhPi3QisIHh4enDp1iri4ONLS0ggICKBTp0767dWqVcPKyopz584BsGfPnmzbhSjNSsuDoaLwKIoOyFurS6FeIcyaNYvRo0czYMAAvLy8aNq0KZMmTeLy5csArFmzhpUrV+Lp6UlaWhqjR48urHCEKDYsLCxJSUmUoiDyRVEUNBo18fExWFpaG97hMTn2MioO8nMPAYpv2/LzkJyLj+eZQlOlKn3TSUrOT9v+9Ck0893LSAhhGubmFlSqlPvxhR5XXIvg85CcC07p6IQvhBDCICkIQgghgGLeZKRS5X/87+fZt7iSnEsHybl0yE/OhvYp1jeVhRBCFBxpMhJCCAFIQRBCCPE3KQhCCCEAKQhCCCH+JgVBCCEEIAVBCCHE36QgCCGEAKQgCCGE+JsUBCGEEEAJLwj+/v707t2bHj164Ovr+8T2oKAgfHx86NmzJwsWLECj0ZggyoJlKOeDBw/Sv39/+vXrx7Rp00hISDBBlAXLUM6PHDlyhJdfftmIkRUeQznfvn2bV199lX79+jFhwoRS8TlfvXoVHx8f+vXrx5QpU0hMTDRBlAUrOTkZLy8vwsLCnthWKN9fSgkVERGhdO3aVXn48KGSkpKi9O3bV/nrr7+yvaZPnz5KYGCgoiiK8vbbbyu+vr4miLTgGMo5KSlJ6dChgxIREaEoiqJ8/PHHytKlS00VboHIzeesKIoSHR2t9OrVS+natasJoixYhnLW6XTKK6+8ohw9elRRFEX54IMPlNWrV5sq3AKRm895+PDhypEjRxRFUZSVK1cqH330kSlCLTAXLlxQvLy8lEaNGimhoaFPbC+M768Se4Vw8uRJ3N3dcXBwwMbGhp49e7Jv3z799vDwcNLT02nevDkA3t7e2bYXR4ZyVqvVvPPOO7i4uABQv359Hjx4YKpwC4ShnB9ZuHAh06dPN0GEBc9QzlevXsXGxkY/Je3UqVMZOXKkqcItELn5nHU6HSkpKQCkpaVhbZ232cKKmu3bt7NkyZKnzjVfWN9fJbYgREVF4eTkpF92dnYmMjIyx+1OTk7ZthdHhnKuUKEC3bt3ByA9PZ1Nmzbpl4srQzkDbNmyhYYNG9KsWTNjh1coDOUcEhJCpUqVmDt3Ln379mXJkiXY2NiYItQCk5vPed68eSxYsICOHTty8uRJhg0bZuwwC9Ty5ctp3br1U7cV1vdXiS0IylMGcX18KjlD24uj3OaUlJTEpEmTcHNzY+DAgcYIrdAYyvnGjRsEBAQwbdo0Y4ZVqAzlrNFo+PPPPxk1ahT+/v688MILrFq1ypghFjhDOaenp7NgwQK+++47Tpw4wYgRI5g7d64xQzSqwvr+KrEFwcXFhZiYGP1yVFRUtkuvf2+Pjo5+6qVZcWIo50frRowYgZubG8uXLzd2iAXOUM779u0jOjoaHx8fJk+erM+/ODOUs5OTEzVr1qRJkyYAeHl5cenSJaPHWZAM5Xzjxg2srKxo2rQpAEOHDuXPP/80epzGUljfXyW2IHh4eHDq1Cni4uJIS0sjICBA36YKUK1aNaysrDh37hwAe/bsyba9ODKUs1arZerUqXh6erJgwYJif0UEhnOeOXMm+/fvZ+/evWzatAlnZ2e+//57E0b8/Azl3KJFC+Li4ggODgbg0KFDNGrUyFThFghDOdesWZOIiAhu374NwG+//aYviCVRoX1/Pfdt6SLsxx9/VPr06aO88soryqZNmxRFUZSJEycqly5dUhRFUYKCghQfHx+lV69eyhtvvKFkZGSYMtwC8aycAwIClPr16yv9+vXT/5s/f76JI35+hj7nR0JDQ0tELyNFMZzzhQsXFB8fH6V3797K+PHjlZiYGFOGWyAM5XzkyBGlb9++ipeXlzJmzBglJCTElOEWmK5du+p7GRX295fMmCaEEAIowU1GQggh8kYKghBCCEAKghBCiL9JQRBCCAFIQRBCCPE3KQglSP369enbty/9+/fX/1uwYMEz9/Hz82PKlCkFcv7169fj7u5O//79GTBgAH379mXs2LHcuXMnX8eLjIzUDz8QGhrKjBkznlj/vMLCwmjQoEG296xHjx68+uqrhIaGGtx/w4YNHDx4MM/n1Wq1TJkyhZiYGPz8/GjVqpX+fevfvz/Dhg0jMDAwPykB0L9/fxITE0lKSmL06NFPrH9ef/zxB15eXgZfV79+feLi4vJ07Hnz5rF58+ZcvVZRlCdef/DgQTZs2JCnc4osFqYOQBSs7777jooVK5rs/L1792bx4sX65f/973+8+eab+Pn55flYLi4ubNu2DYD79+/rC8vj6wuCtbU1e/fu1S8risKyZctYu3YtH3300TP3/eOPP3B1dc3zOb/++mvatm1LpUqVAGjdujUbN27Ubz906BAzZszgyJEjWFjk/c/0UT5hYWFcvnz5ifUlwa1bt3j33Xe5ePEidevW1a/v3r07vr6+BAUF0aBBAxNGWPzIFUIpsXPnTgYPHsyAAQPo2rXrU5/WDQgIYODAgXh7ezN48GDOnDkDZI19NG/ePLy9venbty8rVqzI9djr7du313+RR0REMHXqVPr27YuXlxdfffUVkDX2zpIlS+jbty/e3t7MnDmTlJQUwsLCaNGiBVqtloULFxISEsKECRP063U6HZ07d872hTdr1ix9bp9//jkDBw6kf//+TJs2LdeDf2VkZBAdHY29vT0Ad+7cYdy4cQwdOpSuXbvy2muvkZGRga+vL1euXGH16tUcOHCAzMxMVqxYwcCBA+nXrx/z5s0jOTn5ieOnpaXx3Xff4e3t/cz3LTo6Wv8rf/bs2Xh5edG3b19Wr16tf/8/+eQT/fs2YcIEoqKigH9+mb/99tukp6fTv39/tFqtfv2wYcOyjY65Zs0aPvjgAwB27NiBt7c3AwYMYOzYsdy6deuZ71dO788jH3/8sf5zOHz4sH59bs6zbt061q1b99Tz+vr64u3tjaen5xPbBg0aJFcJ+fHcj7aJIqNevXqKl5dXtieRY2JilOTkZGXIkCFKXFycoiiKEhgYqDRv3lxRFEXZtWuXMnnyZEVRFKVbt2768dWPHz+urF+/XlEURZk3b56yZcsWRVEURaPRKLNnz9Y/Kfq4Tz75RHn33Xf1y2q1Wlm5cqUyZcoURVEUZeTIkcrXX3+tKIqiJCYmKn379lV++ukn5cyZM0qvXr0UnU6nKIqirF69Wjl37pwSGhqqj/P06dNKnz59FEVRsq1ft26d/pzx8fFK27ZtlcTERGX37t3Kf//7X0WtViuKoijbtm1TJk6c+ETMoaGhipubm9KvXz/Fy8tLad++vdKrVy/lo48+UpKTkxVFUZRVq1Ype/bsURRFUTIzMxUvLy9l3759iqIoyqhRo5Rff/1VURRFWb9+vbJq1Sp9Hh9++KGyZMmSJ8556NAhZdSoUfrlxz8DRcmaz+Cbb75RvLy8FEVRlLfeektZunSpotPplIyMDGX8+PHKxo0blfv37ystW7bUP6G6efNm5cCBA4qiZP2/EBsbm+29enz9zp079efUaDTKSy+9pNy5c0f5448/lBEjRiipqamKomT9f+Dp6flEDo9/Hs96f+rVq6ds3LhRURRFuX79utK2bVslNjb2meeZO3eu8tVXXz1xzpw87fVJSUlK06ZNlbS0tFwfRyiKNBmVMDk1GX3xxRccPXqUu3fvEhwcTGpq6hOv6dOnD9OnT6dz58506NCBSZMmAVkzjV2+fJmdO3cCWSNL5uSXX37Rj6+iVqtp1KgRS5cuJTU1lfPnz/P1118DYGdnh7e3N8eOHWPBggWYm5szePBgOnbsSM+ePWnatOlTZ4n6Nx8fHwYNGsS8efP46aef6Nq1K3Z2dhw+fJjLly/j4+MDZI2Vn5aW9tRjPN5kdPz4cebMmUOHDh0oV64cAHPmzOH333/nyy+/5O7du0RFRT31/Tty5AhJSUmcPHlSn7+jo+MTr7t9+zY1atTItu7s2bP0798fMzMzMjMzqV27Np988gkAx44d44cffsDMzAxLS0uGDRvGd999x8SJE/Uj1nbq1IlOnTrRvn17g+8ZgKenJ6tXryY6Oppr165Rs2ZNatWqxfbt27l37162ezQJCQnEx8fj4ODw1GMZen+GDx8OQL169ahTpw6BgYGcO3cux/MUBFtbW2xtbQkPD6dOnToFcszSQApCKRAREcHQoUMZMmQIrVq1olevXtku3R+ZNWsWgwYN4sSJE/j5+bFp0yb8/PzQ6XSsW7dO/4eVmJiY48B4/76H8EhycvITQ/bqdDo0Gg3ly5dn7969nD9/ntOnT/Pf//6X0aNH52quhmrVqtGwYUOOHDmCn58f8+fP1x974sSJ+pFNMzMzczWN5EsvvcS4ceN44403+PXXX7Gzs+ONN95Aq9Xi6elJly5dePDgwVOHH9bpdMyfP5/OnTsDkJKSkq3p5BGVSoVOp8u27t/3EP593H8vazQaVCoVW7du5fLly5w6dYoVK1bQrl07Fi5caDDPR5PM/PTTTwQGBjJ48GD9sfv378+cOXP0y1FRUfrms6cx9P6oVP+0TCuKgoWFRb7Ok1darRZzc/MCO15pIPcQSoErV65QsWJFpk2bxksvvaQvBlqtVv8ajUbDyy+/TGpqKsOHD2fJkiXcunULjUZDx44d+fbbb1EUhczMTF577TW2bt2apxhsbW1p1qyZfi7cpKQk9uzZg4eHB4cPH2bs2LG0aNGCGTNmMGDAAP1InY+Ym5ujVqufeuwhQ4bw5Zdfkp6eTqtWrQDo2LEjO3fu1Lfhr1u3jrfeeitXsY4fP57y5cvrf6GfOHGC119/nd69e2NmZsbFixf17525ubm+Pb9jx474+vqSmZmJTqdj0aJFT70pXatWrVz1YHrk0XEfvf/bt2/Hw8OD4OBgvLy8qFOnDlOmTGHs2LFcv349274WFhZotdqnFrAhQ4bg5+dHYGAgPXv2BKBDhw78/PPP+nsRP/zwA2PGjHlmfM96fwB2794NZM3kdu/ePZo1a5av8+RFUlISGRkZVK1atcCOWRrIFUIp0KFDB3bu3EmvXr0oW7YsTZs2pWLFity7d0//GgsLC+bPn8/s2bOxsLDAzMyMFStWYGlpyYIFC1i+fDl9+/ZFrVbj4eHBxIkT8xzHmjVreO+99/Dz8yMzM1N/M1Sn03Hs2DG8vLywsbHB3t6epUuXZtu3bt26mJubM2jQINauXZtt28svv8y7776rb+ICGDx4MJGRkQwZMgQzMzOqVKmS60liypQpw6JFi5g4cSKDBw9m1qxZvP7669jb21O2bFnatGlDSEgIAF27duX9999HrVYzbdo03n//fQYOHIhWq6VBgwbMmzfvieN7eHiwYMECEhMTKV++vMF4Fi5cyLJly/Tv/0svvcTUqVOxtLTE09MTHx8fbGxssLa2fuLqwMnJiYYNG+Lp6ckPP/yQbVvjxo2xsLCgZ8+eWFlZAVlXSJMmTWL8+PGYmZlha2vLhg0bnjlU+rPeH8jqMjxgwADMzMz46KOPcHBwyPV5Ht1Q/s9//mPwfXrciRMn6NKlC5aWlnnar7ST0U6FMIEvvvgCc3PzbEVMFJzRo0czf/583NzcTB1KsSJNRkKYwPjx4zl9+jTR0dGmDqXEOXDgAK1bt5ZikA9yhSCEEAKQKwQhhBB/k4IghBACkIIghBDib1IQhBBCAFIQhBBC/E0KghBCCAD+HwL8w2rovWvDAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 285 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "SNyCZxHJuXZV" - }, - "outputs": [], - "source": [ - "def plot_technique_comparison(mf_dict, metric):\n", - " \"\"\"\n", - " Plots a specified metric for a given dictionary of MetricFrames.\n", - " \"\"\"\n", - " mf_dict = {k:v.by_group[metric] for (k,v) in mf_dict.items()}\n", - " comparison_df = pd.DataFrame.from_dict(mf_dict)\n", - " comparison_df.plot.bar(figsize=(12, 6), legend=False)\n", - " plt.title(metric)\n", - " plt.xticks(rotation=0, ha='center');\n", - " plt.legend(bbox_to_anchor=(1.01,1), loc='upper left')" - ] + "id": "nz7QJOLx0RVH", + "outputId": "ef446998-8269-4e9b-c8ba-43a777b947d8" + } + }, + { + "cell_type": "code", + "execution_count": 50, + "source": [ + "# Show balanced accuracy rate of the 0/1 predictions\r\n", + "balanced_accuracy_score(Y_test, Y_pred)" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "0.5937900251950625" + ] + }, + "metadata": {}, + "execution_count": 50 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "9dNb-kI3uHzM" - }, - "outputs": [], - "source": [ - "test_dict = {\n", - " \"Reductions\": metricframe_reductions,\n", - " \"Unmitigated\": metricframe_unmitigated,\n", - " \"Postprocessing\": metricframe_postprocess,\n", - " \"Postprocessing (DET)\": mf_deterministic\n", - "}" - ] + "id": "pxYppCAy1owq", + "outputId": "b9febbe3-8016-43a7-c14a-98a94f05716a" + } + }, + { + "cell_type": "markdown", + "source": [ + "As we see, the performance of the model is well above the performance of a coin flip (whose performance would be 0.5 in both cases), albeit it is quite far from a perfect classifier (whose performance would be 1.0 in both cases).\n" + ], + "metadata": { + "id": "jmWrDs5N2HVD" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Inspect the coefficients of trained model\n", + "\n", + "We check the coefficients of the fitted model to make sure that they \"makes sense\". While subjective, this step is important and helps catch mistakes and might point out to some fairness issues. However, we will systematically assess the fairness of the model in the next section.\n", + "\n", + "*Note that coefficients are also a proxy for \"feature importance\", but this interpretation can be misleading when features are highly correlated.*" + ], + "metadata": { + "id": "AmhwS1Z9VnK9" + } + }, + { + "cell_type": "code", + "execution_count": 51, + "source": [ + "coef_series = pd.Series(data=unmitigated_pipeline.named_steps[\"logistic_regression\"].coef_[0], index=X.columns)\r\n", + "coef_series.sort_values().plot.barh(figsize=(4, 12), legend=False);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAccAAAKuCAYAAADU79e9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAAEAAElEQVR4nOzde1zP9///8VvnGClUlsMc5ziHWYStFEMqqUh85PA1bOSwTRNybk45ROY05tCQUClJzrJQGOJDZsSYZEQO6dzvj369PzqqZOrtcb1cXC72fr9ez9fz+WYevV7v1+txV8nKyspCCCGEEAqq73oCQgghRHkjxVEIIYTIQ4qjEEIIkYcURyGEECIPKY5CCCFEHlIchRBCiDykOAohhBB5qL/rCQgh4PHjF2RmVuxHjmvUqMKjR8/f9TTKjDKtR9aSn6qqCnp6HxT6vhRHIcqBzMysCl8cAaVYw6uUaT2ylpKRy6rvgcjISJydnUu1r7+/P25ubkVuM23aNC5dulSq8QuzYsUKzp49+8bjT5kyhb///rvY29+9excLC4tSHUsIoTzkzFG8sR9//LHMxzxz5gwdO3Z84/EjIyMZO3ZsWU1LFCE1LQN9/arvehplSpnW8z6tJTklnWdPX77RMaQ4vicSEhIYOXIkf/31Fw0aNGDFihX89NNPnDp1isTERPT09PD29kZfX5/AwEBWr15NlSpVqF27NpUrVy5ybGdnZ1xcXABYu3Yt2tra3Lhxg6ZNm7J48WIePHjAN998Q926dbl9+zZGRkZ4enqiq6vLr7/+yp49e3j58iUqKip4eXlx6dIlLl++jLu7OytXrsTDwwMXFxc6duzIunXrCA0NJSMjg88//xxXV1f+/vtvXFxcaNKkCVevXqVGjRosX74cPz8/Hjx4wKhRo9i6dSt6enoFzv/KlStMmzYNgGbNmile/+OPP5g7dy5JSUkkJCQwfPhwBg8eTPfu3dmwYQMNGjQgKSkJS0tLQkJCmD17NtevXwdg0KBBODo6lsUfXYWhqaGGzfd73vU0hCB4iS3P3nAMuaz6nrh37x4zZswgNDSUhw8fsn37dm7evImvry9hYWHUq1eP4OBg4uPjWbx4MVu3bmXHjh28ePGiRMc5f/684jj37t3jt99+A7ILzdChQwkJCaFRo0asXLmS58+fc+jQIXx8fNi7dy/du3dn27Zt9O3bl1atWuHh4UHTpk0VY4eHh3P58mV27dpFYGAg8fHxBAUFARATE8Pw4cPZu3cvOjo6BAcHM2rUKAwMDFi3bl2hhRFg8uTJuLq6EhAQQJ06dRSv79y5kzFjxrB79262bNnCsmXLUFVVpW/fvorjHjhwgK5du3LlyhUSExMJDAxk48aN/P777yX63IQQ5YucOb4nmjVrRt26dQFo1KgROjo6TJ48mZ07dxIbG8uFCxeoV68e58+fp127dtSsWRMAGxsbTp8+XezjNGnShFq1aimOk5iYCED9+vUVl0n79u3LpEmTqFKlCkuWLCEkJIRbt25x4sQJmjdvXujYp06dIjo6Gnt7ewCSk5MxMjKiffv21KhRgxYtWijmkHPc10lISODBgwd07twZAHt7e3bv3g2Am5sbJ06cYO3atVy7do2kpCTFNsOHD2fChAkEBATw3XffUa9ePWJjYxkxYgSmpqZMmjSp2J8ZZN+BJ4QoO296GVmK43tCXf1/f9QqKio8fvyYESNGMGzYMHr27ImqqipZWVmoqKiQmZlZ4H7FoaWlles4OYlor46TlZWFmpoacXFxODs7M3jwYExNTalZsyZXr14tdOyMjAyGDh3K8OHDAXj69Clqamo8fvy40OO+Tt5t1dTUFL+fOHEiOjo6mJub07t3b0JCQgCoU6cORkZGHDhwgEePHtGmTRsAQkJCiIiI4Pjx49jZ2RESEoKOjk6x5vHo0fMKfzehMn2nJSq+f/4p+sKqqqpKkT+UymXV95SKigodOnRg4MCBNG7cmIiICDIyMmjfvj0XL14kPj6ezMxM9u3bVybHi42NVRS+3bt3Y2pqyqVLl/joo48YNmwYbdq0ITw8nIyMDCC7SOX8PoeJiQl79uzhxYsXpKenM3bsWMLCwoo8bkHjvEpPTw8jIyOOHTsGwN69exXvRUREMH78eLp3786ZM2cAFGM5ODjg4eFBnz59ADh8+DCTJk2ia9euuLu7U7lyZeLi4krwCQkhyhMpju+p5ORkYmJisLGxYejQoTRt2pS7d+9Ss2ZN3N3dGTZsGP369aNKlbK53FetWjVWrFiBlZUVCQkJfPPNN3Tp0oXMzEx69+6No6MjtWvX5u7duwB88cUXzJw5M9d3dxYWFvTo0QNHR0esra1p1qwZdnZ2RR63a9eujBo1ijt37hS6jaenJytXrqRv37789ddfitfHjRvHoEGDsLOz47fffss1vx49epCYmIitrS0ApqamaGtrY2VlRf/+/enRo0eu70uFEBWLSlZxrz8JUUp3795lyJAhHDly5F1PpUxkZWURHh7O9u3bWbNmTZmMqQyXVavpVkZTQ+31GwrxlhXnUY7XXVaV7xxFsXz//ff8+eef+V63sLBgwoQJ72BGJVOW8583bx5Hjx7l559/LqvpKQVNDbXXfs9TkejrV1Wa9chaSk7OHIUoB5ThzFGZ/gEG5VqPrCU/uSFHCCGEKCEpjkIIIUQeUhyFEEKIPKQ4itd626keQghR3sjdqkKIMvG+pXKURfKDKL+kOIpieZupHrdv32bWrFk8efIEbW1tpk+fTosWLXBzc6NSpUqcO3eOZ8+eMXXqVPbs2UNMTAzdu3fHzc2NjIwMFi1aRFRUFBkZGdjb2zNs2DAiIyPx9PQkMzOTJk2a4O7uzg8//MBff/1F3bp1uX//PitXruTDDz8sdP+CEkY0NTXZtGkT27dvR01NDXNzc7755hu6devG4cOHqVKlCnfv3mX06NGKdnPvi/ctlaMskh9E+SXFURTLvXv3WLNmDbVr18bR0TFXqoeqqio//PADwcHBWFlZsXjxYgIDA9HV1WX06NGvLY6TJ09mxowZtGjRgj///DNXW7gHDx4QFBREQEAAU6ZMISwsDC0tLUxNTRk7dqyi3VtAQACpqamMGDGCVq1aAXDr1i2OHj1K1apVWbBgAQ0aNGD16tVcunRJESfl5+dX6P7nz58nNDQUAwMDHB0d+e2336hZsybbtm1j9+7dVKpUia+++opbt27RtWtX9u/fT79+/QgMDFR0zhFCVExSHEWxvK1UjxcvXnD58mWmTJmieC0pKYnHjx8D2W3ZAIyMjGjSpAk1atQAQFdXl8TERE6dOsXVq1cVx0hKSuLatWs0btyYBg0aULVq9mWxiIgIFi9eDMAnn3yiaO1W1P4FJYzExsZibm6uGHfTpk1Adq9Vb29v+vXrx969e9m8eXOJPl9J5aiYKtJl5Io019f5N9YixVEUy9tK9cjMzERTU5M9e/53Oe7+/fvo6uoCoKGhUeRYGRkZuLq60qNHDyD78m/lypW5ePEi2traiu3U1NQKTOooav+Ckj7yziE+Pp5KlSphbGzMgwcPOHDgAHXq1MHQ0LDIdeelLE0A3jcV5cF6aQKQnzQBEG9FWaV6VK1alfr16yuKY0REBP/5z3+KPQ8TExP8/PxIS0vjxYsXDBo0iIsXL+bbrnPnzgQHBwNw7do1rl+/joqKSrH3z/HZZ58RHh6uSAb5/vvvuXz5MioqKvTt2xcPDw9F3qQQouKSM0dRKq+memhoaBSY6lGpUiUaN2782rE8PT2ZNWsW69evR0NDg2XLlqGiolKseTg5OXH79m3s7OxIT0/H3t6ejh07EhkZmWu7MWPGMGXKFGxsbKhXrx41a9ZEW1u72PvnaNmyJYMHD8bJyYnMzEy+/PJLRVCylZUVGzdupHv37sWauxCi/JLequK9sGfPHurUqUP79u25d+8egwcP5tChQ6iqls3Fk8zMTLZv305sbCzu7u4l3l8ZLqu+b6kcFelRDrmsmp+kcohy4V2nejRs2JCZM2eSmZmJqqoqc+bMKbPCCODi4kJcXBwbNmwoszErGknlEMpEzhyFKAeU4cxR2YqJMq1H1pKf3JAjhBBClJAURyGEECIPKY5CCCFEHlIchRBCiDzkblUhRJkor6kcFemRC1F+SHEUFZqbmxsdOnSgS5cuuLu78/PPP5d4jClTpuDi4kLt2rUZOXIkHh4eJW7/JspvKoekZ4jSkMuqQikYGhqWqjBCdphzzhNNP//8sxRGIYScOYpsBeUXfvvtt4wYMYIjR44A4O3tDcC4cePo0qUL5ubmnD17Fn19fQYNGoSPjw/3799nwYIFdOjQodBjvUlOY1ZWFgsWLODYsWMYGBiQkZFBhw4duHv3LkOGDOHIkSP8/fffTJkyhYSEBLS1tfHw8KBZs2YsW7YsX/5kQEAADx48YNSoUWzduhUHBwe2bNmCkZER8+bN49SpU6ioqNCnTx9GjRpVaM5jamoq3333HQ8fPgRg7NixdOvWrdifv6RyvF2lvdxbHi8Tl5aspWSkOAqFgvILC/Pw4UO6du2Kh4cHzs7OHDp0iG3bthEQEMDmzZuLLI5Q+pzGhw8fcuXKFfbu3cuzZ8/o06dPvrFnz55Nz549+c9//sPx48dZvXo13333XYH5k6NGjcLX15d169ahp6enGGP79u3ExcURFBREamoqzs7OfPzxx1SqVKnAzykxMZHatWuzbt06bty4wa5du0pUHJWlCUB5VZqHxuXB+fJJ2seJf11B+YVFyclarF27Nu3btweycxefPn362mOVNqfxxo0b9OjRAw0NDapXr64Y51Vnzpxh6dKlAJiZmWFmZgZQYP5kYSIjI7Gzs0NNTY1KlSphY2PDqVOnsLCwKPBzateuHUuXLiU+Pp6uXbsyduzY134GQojyS4qjUMibXwjkykBMT0/PlWeoqamp+L2aWskaTpc2p9HT0/O1eZGvvpaVlcWNGzdITk7m+++/z5c/WZhXj5EzTkZGBpD/c8rKyqJ+/fqEhoZy4sQJjh49yi+//EJoaGix00WEEOWLFEdRqKpVq5KYmEhCQgJVqlThxIkTmJub/yvHzslZNDc3JzU1lUGDBjF79mw6derEhg0bGDhwIC9fvuTEiRO0bds2176fffYZISEhDBgwgJMnT7Jy5Up69OihyJ989uwZs2bNUqxFTU1NUfhePX5gYKDi+MHBwXz99deFzvfXX3/lzp07TJkyBVNTU8zNzXn27Bk6Ojpl/tmUV6lpGQQvsX3X08gnOSX9XU9BVEBSHEWhqlatyogRI+jXrx+1atXik08++deOXVjOIsClS5ewtramZs2aNGrUKN++M2bMwN3dnW3btlGpUiU8PDyoWrUqLi4u+fInAbp27cqoUaNYv369YowBAwZw69YtbG1tSUtLo0+fPnz55ZeF5jz27duX7777DhsbG9TV1XFxcXmvCiMoXyqHeL9JKocQ5YCy3JCjTMVRmdYja8lPbsgR78TChQs5efJkvtdbtWrFjz/++A5mJIQQxSfFUbwVkydPftdTEEKIUpMOOUIIIUQeUhyFEEKIPOSy6lty+PBhLl++zIQJE971VBRebf9ma2vLnj3vrkn08uXLadWqVYm6yED2Gnx9falZsyZZWVmkpaVhb2/PyJEjiz1u06ZNuXbtWrGPeeTIEW7fvs3w4cNLNNf3jaRyCGUixfEt6datW4n/4f83vcvCCLzRDw1OTk6MGzcOyG4OMHToUHR1denfv/9b+WHkv//9b5mPqYwklUMoEymOpRAZGYm3tzfq6urExcXRunVrvvnmG8aMGYOenh5aWlr06dOHqKgoFixYgIWFBZaWlhw7dgw1NTW+++47fvnlF27fvs3kyZPp3bs3f/zxB3PnziUpKYmEhASGDx/OkCFD8Pb25sKFC8TFxeHk5MQvv/zCkSNHUFVVJSoqinXr1uV6Pi+v9evX4+fnh56eHjo6OrRu3Rr439lTfHw8U6dO5dmzZ/zzzz9YWVkxadIk0tLSmDlzJufOncPQ0BAVFRXGjBkDUGDjbU1NTXbv3s3GjRtRUVGhZcuWTJ8+HU1NTaZOncr169cBGDRoEI6OjoqoqR49erxRw+7q1avzzTff8PPPP9O/f3/FuPb29gU2GtfX1wdg+vTpREdHo6enx7x58zAyMuL27dvMmjWLJ0+eoK2trZi/r68vkN3qrlevXsyZM4fr16+TkZHByJEjsba2JiYmhhkzZpCeno6Wlhbz58+nfv36Jf67JYQoH+Q7x1KKjo5mxowZ7N+/n5SUFI4fP05sbCyenp5s2rQp3/YGBgaEhITQsmVL1q1bxy+//IKnpyfr1q0DYOfOnYwZM4bdu3ezZcsWli1bptg3NTWVffv2MWTIEOrUqaN4ED0gIAB7e/tC53jp0iV2795NQEAAGzdu5P79+/m22bt3L9bW1vj5+REUFMS2bdtISEjA19eXly9fsn//fubPn8+lS5cU+5w/f54ZM2YQGhrKvXv3+O2337h27Rpr1qzBx8eH4OBgKlWqxMqVKzl//jyJiYkEBgayceNGfv/991zHP3jwILVr18bf3x9PT0/Onj1boj8HgI8//pibN2/meu327duKRuNhYWHUq1eP4OBgxfvGxsbs2bOHL7/8UvFoyeTJk3F1dSUgIIC5c+fy7bff0rhxY5ycnHBycsLBwYHVq1fTsmVL/P392bp1K2vWrOHOnTts3ryZ4cOH4+/vj7OzMxcuXCjxOoQQ5YecOZaSsbExDRs2BMDW1hY/Pz9q1KhBnTp1Ctz+1UbbBgYGqKur52rS7ebmxokTJ1i7di3Xrl0jKSlJsW/O2R6Ag4MDQUFBtG3bltOnTzN79uxC5xgVFYWZmRkffPABAL169crXM3TEiBGcPn2aDRs2cP36ddLS0nj58iURERE4OjqioqJC7dq16dSpk2Kfghpv37t3D3Nzc0WyxYABA5gyZQqjRo0iNjaWESNGYGpqyqRJk3IdvywadquoqKCtrZ3rtY8++qjQRuPa2tqKNA9bW1u8vLx48eIFly9fZsqUKYoxkpKSePz4ca5xT548SXJyMrt371Zsc/36dczMzJgzZ46ixV7Pnj1LtAaJrHq7JLJK1lJSUhxL6dVG21lZWaipqeX7B/pVr2u0PXHiRHR0dDA3N6d3796EhIQo3nt13F69erFs2TLCwsIwNTXN1fw7LxUVlXxNulNTU3Nts2DBAu7cuYO1tTXdu3fn5MmTivXkLaQ5Cmq8XVCj7vT0dPT09AgJCSEiIoLjx49jZ2eXa21l0bD72rVr+drIXb58udBG46qq/7tgkpWVhbq6OpmZmWhqaub6Lvb+/fvo6urmGjczMxNPT09atmwJZEd3VatWDQ0NDdq1a8fRo0fZvHkzx48fx8PDo9hrUJYOOeWVRFbJWvJ6XYccuaxaSufOnSM+Pp7MzEwCAwMLjE4qiYiICMaPH0/37t05c+YMQL5m2ACVKlXC1NSUpUuXFnlJFaBTp04cO3aMZ8+ekZKSwsGDBws87ogRI7C0tCQuLk6xps6dO7Nv3z6ysrKIj48nKiqqyILVoUMHjhw5wpMnTwDw8/OjY8eOHD58mEmTJtG1a1fc3d2pXLkycXFxiv1+/fVXvL29sbS0ZObMmSQkJPDsWfH/4j948IA1a9bwn//8J9frZ86cUTQab9y4MREREYrPMykpicOHDwOwe/duOnfuTNWqValfv76iOEZERCjGVFNTIz09u3m1iYkJ27dvVxy7T58+xMXFMXHiRKKjo3FycmLChAlcuXKl2GsQQpQ/cuZYSgYGBvzwww/Ex8fTpUsXOnfurPj+sDTGjRvHoEGD0NHRoUGDBtSuXVvRGDsvKysrfv/9d9q0aVPkmM2bN2fo0KH069cPHR0djIyM8m0zevRofvjhB3R0dKhRowatWrXi7t27ODo6EhMTg42NDfr6+hgZGaGtrc3LlwXfEt+sWTNGjx6Ns7MzaWlptGzZktmzZ6OlpUVYWBhWVlZoaWnRo0cPmjZtqtivNA27fX19OXTokOKsdcCAAVhZWeXapnfv3oU2GtfR0eHQoUMsX74cQ0ND5s+fD4CnpyezZs1i/fr1aGhosGzZMlRUVDA2Nmby5MnUrFkTFxcXZs2ahbW1tSJWq169enz99ddMmzaNVatWoaamhpubW5FrUEaSyiGUiTQeL4XIyEhWrlyJj4/Pv37sjIwMli1bRo0aNd7qc3fHjh0jKytLEb3Ut29fdu/ene8yoygbynJZVVku3YFyrUfWkp80HlcyDg4O6OnpsXr1agD++usvxTN/eXl4eJQ6ZqpRo0b88MMPeHl5ATB+/Ph/rTBK03IhxLsmZ45ClANy5lj+KNN6ZC35yQ05QgghRAlJcRRCCCHykOIohBBC5CHFURTp2bNnjBkzhvj4eEXyxdvwJuP7+/u/1Ucnli9frngu0tnZ+bXbW1hYFPoYjjLLSeUoT7+q6lR61x+LqKDkblVRpMTERGJiYjA0NOTnn39+a8d52+O/iVeTPqKiot7hTMq38pjKIYkcorSkOIoieXh48ODBA8aOHcvVq1c5cuQIbm5uVKpUiXPnzvHs2TOmTp3Knj17iImJoXv37ri5uZGRkcGiRYuIiooiIyMDe3t7hg0bVuhx7t69y5AhQxTjV6lShf/+97/Ex8czduxYHBwcipzn7du3cXZ25t69e3Tq1EnRum3NmjUEBQWhpqZGly5dcHV15eXLlwUmgTg7O9OwYUOio6NJSUlh6tSpfP7554qkj5yuN/3792fnzp38+uuv7Nmzh5cvX6KiooKXl1e+NnZCiIpJLquKIrm7u2NgYJCrITdkt04LCgpi/PjxTJkyhdmzZxMYGIifnx/Pnj3Dz88PyE4O2bVrF4cPHy5R4sb9+/fZtm0bq1evZtGiRa/dPi4uDm9vb0JDQwkPD+f69escP36cI0eO4O/vT0BAALdv38bX17fIJJDU1FQCAgJYsmQJbm5uuXrRuru7A9kJKs+fP+fQoUP4+Piwd+9eunfvzrZt24q9PiFE+SZnjqJUXk0ZadKkCTVq1ABAV1eXxMRETp06xdWrVzl9+jSQ3c/02rVrfPbZZ8Uav0uXLqioqPDxxx8r+rUW5bPPPlM0KahXrx6PHz/m9OnTWFlZKRq3Ozg4EBgYyKRJkwpNAnF0dASyW+/p6+tz7dq1Ao9XpUoVlixZQkhICLdu3eLEiRM0b968WGsriKRyvD1v0hC9PDdTLylZS8lIcRSl8rqUkZy+oz169AAgISGBypUrF3v8nOSP4qZzvDqHwpJCANLT0wtNAoHcaSuZmZkFrg2yz1SdnZ0ZPHgwpqam1KxZk6tXrxZ7fXkpSxOA8qi0D4zLg/PlkzQBEOWCurq6IpGiJExMTPDz8yMtLY0XL14waNAgLl68+BZmWPQcQkJCSE5OJj09nd27d2NiYlJkEsi+ffuA7KDop0+f8vHHH+caMyeh49KlS3z00UcMGzaMNm3aEB4eXmCKihCiYpIzR1GkGjVqYGRklO87x9dxcnLi9u3b2NnZkZ6ejr29PR07dnxLsyyYubk5V69excHBgfT0dL744gsGDx5McnJyoUkgd+7cwc7ODoBly5blOpME6NatmyLcevv27fTu3RtNTU1at27N9evX/9X1lTflMZVDEjlEaUlvVSH+P2dnZ1xcXP71Ig7Kc1lVWS7dgXKtR9aSn6RyiHLjTRJE9u3bx9q1awt8LyegWAghyoqcOQpRDsiZY/mjTOuRteQnN+QIIYQQJSTFUQghhMhDiqMQQgiRh9yQI97Iv3WHp4WFBVu2bOHatWtcvnw5VzNwUT7kpHKUB8kp6Tx7+vJdT0NUYFIcRYXSrVs3unXr9q6nIQpQnlI5JI1DvCkpju+RyMhI1q5di7a2Njdu3KBp06Z8++23jBgxgiNHjgDg7e0NwLhx4+jSpQvm5uacPXsWfX19Bg0ahI+PD/fv32fBggV06NABAD8/PxYsWEBWVhZTpkyhY8eOvHjxgjlz5nD9+nUyMjIYOXIk1tbWiibgT548wdzcnO+++67AuT558gRXV1fu379Po0aNSElJAbKzG6OioliwYAGhoaFs3LiR5ORkUlJS8PDwwNjYmD/++EORDPLZZ58RHh7OwYMHC037ePnyJe7u7ly7dg0VFRVGjBhB3759iYmJYcaMGaSnp6OlpcX8+fOpX78+4eHhrFixgvT0dOrUqcPcuXPR09Nj4cKFREREoKamRrdu3XBxcfkX/lSFEG+DFMf3zPnz5wkNDcXAwABHR0d+++23Qrd9+PAhXbt2xcPDA2dnZw4dOsS2bdsICAhg8+bNiuJYuXJlAgICiImJYfTo0Rw8eJDVq1fTsmVLFi5cyPPnz3FycqJNmzZAdrDxvn37Cu1bCrBixQpatGjBzz//zJkzZxS9T3NkZmbi6+vLmjVrqF69Ort27WLDhg0YGxvj5ubGhAkTMDMzY9OmTbnauuWkffzxxx8MGTIEBwcHvL290dPTY+/evSQkJNC/f3+aNWvG5s2bGT58OJaWluzbt48LFy6go6PDkiVL2LJlC9WqVcPX15fFixczZswYwsPDCQkJISUlhWnTppGSkqLoESuEqFikOL5nmjRpQq1atQBo1KgRiYmJRW6fk75Ru3Zt2rdvD2QncTx9+lSxTb9+/QBo1qwZ1atX5+bNm5w8eZLk5GR2794NZKdy5LRXa9GiRZGFEbJDhZcsWQKAsbExdevWzfW+qqoqP/30E0eOHCE2NpaoqChUVVV58uQJf//9N2ZmZkB2EseWLVsU+xWU9nH69GnmzZsHQPXq1enWrRtRUVGYmZkxZ84cTpw4gbm5OT179iQ8PJy4uDiGDBkCZBfpatWqYWhoiJaWFk5OTpibmzNx4sQSFUZJ5Sh7ZfH9Z3n5DrUsyFpKRorje+bVf7BzEi9e7QORnp6eq3Bpamoqfp+3z2hBr2dlZaGurk5mZiaenp60bNkSyD4LrVatGsHBwYoIqaLkJGsUduwXL17g4OCAra0txsbGNG3alK1bt6KmpkZRfS0KSvvIu31WVhYZGRn06tWLdu3acfToUTZv3szx48fp2rUrn376KWvWrAEgJSWFFy9eoK6uzs6dO4mKiiI8PBwnJyd8fHxo0KDBa9cKytMEoDx50wfF5cH58kmaAIh/RdWqVUlMTCQhIYHU1FROnDhR4jGCg4OB7CSL58+f89FHH2FiYsL27duB7GDkPn36EBcXV+wxO3XqpGgLFx0dzV9//ZXr/Vu3bqGqqsrXX3+NiYmJIhWjatWq1KtXj+PHj+eaW1FMTEzYtWsXkB2tdfjwYTp06MDEiROJjo7GycmJCRMmcOXKFdq0acOFCxeIjY0FYNWqVSxatIgrV64wePBgjI2NmTx5Mo0aNVJsI4SoeOTM8T1XtWpVRowYQb9+/ahVq1aR/U0Lk5SURN++fVFVVWXJkiVoaGjg4uLCrFmzsLa2VmQ71qtXj7NnzxZrzPHjx+Pm5oaVlRUNGzbMd1m1WbNmNG/eHEtLS7S1tTE2NubevXsALFy4kKlTp+Ll5UXTpk1fe6Y6duxYZs2ahY2NDRkZGXz99de0bNmSr7/+mmnTprFq1SrU1NRwc3NDX1+fefPmMXHiRDIzMzE0NMTT0xM9PT3atm2LtbU1lSpVonnz5opL0kKIikd6qwqls3LlShwdHTEwMODAgQMEBwcr7sItr5Thsmo13cpoahR86f3fVhbPOcqlyPJJUjmE0tu0aRMBAQH5XjcwMODnn38u9bhGRkb83//9H+rq6ujo6PDjjz++yTRFMWlqqCnNP8BCyJmjEOWAMpw5KtPZCSjXemQt+ckNOUIIIUQJSXEUQggh8pDiKIQQQuQhxVEIIYTIQ+5WFW/NpUuX8PX1LZO7Re/evcuQIUM4cuQIy5cvp1WrVqVK53B2dub+/ftUrlyZjIwMNDU1FX1YxZt5F5FVEk0l3hYpjuKt+eSTT0rVVOB13jTL0cPDQ5E/eenSJb766iu2bt1K48aNy2J67613EVkl0VTibZHLquKtiYyMxNnZGWdnZxYtWsSAAQP48ssvc7V2s7W1xd7envHjx5OSkqLYJ4ebmxv+/v65xs157e7du/Tt2xdXV1esra0ZOnSoopl4cX3yySdYWlqyc+dOAC5cuED//v3p06cPQ4cO5fbt24SFhTFx4kQgu21d06ZNefjwIQAjRowgOjq60DUKISomOXMU/4q0tDR27NihuCxqZmaGl5cXfn5+1KhRg2XLlnHz5s0SjxsTE8O8efNo0aIF48aNIzg4OFdxLY4mTZpw7NgxUlNT+e677/Dy8qJ169aEhoby3XffsXnzZjw8PMjKyuLUqVPUqFGDqKgoLCwsiI2NVZwdF7TG4pJUjtJ7m5dyy1sz9TchaykZKY7iX/HFF18A2YUo5+zO3NycgQMH0q1bN3r27Enz5s2JjIws0bg1atSgRYsWirFfF8FVEBUVFbS1tbl16xY6Ojq0bt0aAEtLS2bMmEFWVhYNGzbk2rVrnD59mqFDh3LmzBk++OADOnbsqEj4KGiNxaUsTQDehbf1cLs8OF8+SRMAoVQKiopyd3dnxYoV6Orq4urqyp49e/JFVaWlpRVr3JyxS9Pw6dq1azRq1IjMzMx87+XEV5mZmREREcHNmzdxdHTk7NmzhIeHY25uXuQahRAVkxRH8U6kp6fTo0cP9PT0GD16NLa2tly9ehU9PT3u3LlDSkoKT5484dy5c291HtHR0YSFhdGvXz8aNmzIkydPiI6OBmDfvn0YGRmhq6uLmZkZvr6+NG7cGD09PTQ0NDh69ChdunR5q/MTQrwbcllVvBPq6uqMHz+e4cOHo62tjY6ODgsXLsTQ0BAzMzOsrKyoXbs27du3L/Nju7u7U7lyZVRUVKhUqRLLli2jTp06ACxbtoy5c+fy8uVLqlWrxrJlywBo1KgRWVlZdOjQAYAOHTrwxx9/8MEHH5T5/Cqq1LQMgpfY/qvHTE5J/1ePJ94f0nhciHJAWb5zVJbvtUC51iNryU8iq8R7x9nZmadPn+Z73cnJiYEDB76DGQkhKhopjkLp+Pj4vOspCCEqOLkhRwghhMhDiqMQQgiRhxRHIYQQIg/5zlGUqRMnTrBixQqeP3+OqqoqXbp04dtvv6VSpUplepwHDx7g7u7OgwcP0NbWZvHixdSpU4enT58yadIk7ty5Q/Xq1fHy8kJfX79Mjy0K9rZTOSSBQ/yb5FEOUWZOnTrFtGnT8Pb2pmXLlqSmprJgwQJiY2P55ZdfyrRzzLBhw+jZsycDBw5k+/btREZG4uXlxZw5c6hVqxajRo0iMDCQY8eO4eXlVWbHfVuU5VGOt5nKEbzE9l99HEEefyifpH2cKFPp6em4u7szYMAAunXrxldffUVycjIAW7ZsoUePHjg4OODq6oq3tzcA4eHh9OvXj759++Li4sLjx4+LPMaqVatwcXGhZcuWAGhqajJlyhT+/PNPzp07h4uLC/v371dsb29vz3//+19u377N8OHDsbOzY+DAgVy5cgXITt/4+uuvsbS05MiRI4r9EhISiImJwcnJCQAHBwdFasaxY8ewsbEBwNramvDw8Fwt6J4/f07Hjh15/vw5kJ0TaWVlBUBgYCB2dnbY2toydepUUlJSAPj111/p378/1tbW2NjYcOPGDQAsLCyYOHEiPXv25N69e4waNQp7e3vs7e05fPhwSf+IhBDliFxWfU+cP38eDQ0NduzYQWZmJkOHDuX48eN89NFHbN26FX9/fzQ0NHB2dqZevXokJCSwZMkStmzZQrVq1fD19WXx4sVFBhdfunSJmTNn5npNQ0ODdu3acenSJWxtbQkODqZXr17cunWLlJQUWrZsiZOTEzNmzKBFixb8+eefjB07lrCwMAB0dXVZs2ZNrjHv3LmDkZER8+bNIzIyEiMjI6ZPnw5kX27NuYyqrq5OlSpVSEhIwNDQEIAqVarQtWtX9u/fT79+/QgMDMTW1pbr16/j5+eHr68vWlpaLFmyhA0bNjBkyBAOHTqEj48P2traLF++nG3btimOZ2pqipeXFwEBAdSuXZt169Zx48YNdu3aVaIwZknlKJ5/u7m5JFmUT5LKIcqMsbExurq6bN26lZs3b3Lr1i2SkpI4deoU5ubmVKmS/Y+zlZUVT58+5eLFi8TFxTFkyBAAMjMzqVatWpHHUFFRIT09fzuv1NRUAMzMzJg7dy7Pnz9n79692NjY8OLFCy5fvsyUKVMU2yclJSnOUnMSMl6Vnp7OlStXGDduHNOmTWPnzp24ubkV+nyjqmruCyQODg54e3vTr18/9u7dy+bNmzl48CC3b9/G0dERyG543qJFC6pUqcKSJUsICQnh1q1bnDhxgubNmyvGatOmDQDt2rVj6dKlxMfH07VrV8aOHVvkZ5WXslxWfdvksmrpyFrykw45AoDDhw+zYsUKhgwZgr29PY8fPyYrKwtVVdUC0ygyMjL49NNPFWdtKSkpvHjxoshjtG7dmgsXLtCsWTPFa6mpqVy5coWvvvoKTU1NunbtypEjR9i/fz9r164lMzMTTU1N9uz533dV9+/fR1dXFwBtbe18x9HX1+eDDz5QJGJYW1vj4eEBgIGBAQ8fPqRWrVqkp6fz/PlzxVg5jI2NefDgAQcOHKBOnToYGhqSkZGBpaUl7u7uALx48YKMjAzi4uJwdnZm8ODBmJqaUrNmTa5evaoYKyeJo379+oSGhnLixAmOHj3KL7/8QmhoqCR0CFFByXeO74lTp05haWmJg4MDNWvW5MyZM2RkZNCpUyeOHz/O8+fPSU1N5cCBA6ioqNCmTRsuXLhAbGwskP194qJFi4o8xrhx41i9ejX//e9/geyzLw8PDxo2bKhoIG5ra8vGjRupVq0atWvXpmrVqtSvX19RHCMiIvjPf/5T5HHq1auHoaEhx48fB+Do0aOK7znNzMwIDAwEslM1PvvsMzQ0NHLtr6KiQt++ffHw8MDe3h6Ajh07cvDgQR49ekRWVhazZs1i8+bNXLp0iY8++ohhw4bRpk0bwsPDycjIyDenX3/9FW9vbywtLZk5cyYJCQk8e6YcP6kL8T6SM8f3RP/+/Zk0aRL79+9HU1OTtm3bcvfuXfr378+QIUMYMGAAlStXRk9PDy0tLfT19Zk3bx4TJ04kMzMTQ0NDPD09izzGZ599xsKFC/nxxx9JTEwkPT0dU1NTVq1apTiDat++Pc+ePVPcTAPg6enJrFmzWL9+PRoaGixbtuy1Z1wrV65k5syZeHp6UqVKFRYsWADAhAkTcHNzw8rKiqpVq7J48eIC97eysmLjxo10794dgGbNmuHi4sLQoUPJzMykefPmjBo1ivT0dLZv307v3r3R1NSkdevWXL9+Pd94ffv25bvvvsPGxgZ1dXVcXFzQ0dEpcg3K5m2nckgCh/g3yaMc77nY2FiOHz/OsGHDAPjmm2/o378/FhYW73Zib1FmZibbt28nNjZWcRn1XVOW7xyV5XstUK71yFryk+8cRZFq167NpUuXsLa2RkVFhc8//zxXun1e33//PX/++We+1y0sLJgwYcLbnGqZcXFxIS4ujg0bNrzrqQghyik5cxSiHJAzx/JHmdYja8lPmgAIIYQQJSTFUQghhMhDiqMQQgiRhxRHJREZGYmzs3Op9vX398fNza1M5xMdHf3aRz+K8upabG1L93jAs2fPGDNmTIn2eRufxfsiJ5WjrH9V1SnbRBchikPuVhVvxZ9//smjR49KvX9UVJTi9692zymJxMREYmJiSj0HUTKaGmpvJZUjeIktynEriahIpDgqkYSEBEaOHMlff/1FgwYNWLFiBT/99BOnTp0iMTERPT09vL290dfXJzAwkNWrV1OlShVq165N5cqVixw7NjaWGTNm8OTJEypXrsy0adNo3bo1bm5udOjQQdFppmnTppw5c4YVK1aQlJTE6tWrMTQ05MCBAyQmJvLo0SPMzc1xc3MjIyODWbNmcf36dR4+fEiDBg1YuXKl4sH9/v37s3PnTpo2bcq1a9d48eIFc+bM4fr162RkZDBy5Eisra3x9/fnxIkTJCYmcufOHbp06cKsWbPw8PDgwYMHjB07lp9++qnQtRX2WYSGhrJx40aSk5NJSUnBw8MDAwMDhg4dypEjR1BVVSUqKop169bh4eHBpEmTSEpKQlVVFXd3d9q2bVs2f7BCiH+dXFZVIvfu3WPGjBmEhoby8OFDtm/fzs2bN/H19SUsLIx69eoRHBxMfHw8ixcvZuvWrezYseO1PVMBXF1dcXZ2Jjg4mClTpjBhwgRFQ/G8dHR0GD9+PBYWFnzzzTcAXL58GW9vb/bu3cvFixc5ePBgrqSQgwcPkpKSwvHjxxUP5u/cuTPXuKtXr6Zly5b4+/uzdetW1qxZw507d4Ds1JEVK1YQFBTE0aNHuXbtGu7u7hgYGBRZGAv7LDIzM/H19WXNmjUEBQUxcuRINmzYwEcffUSdOnWIjIwEICAgAHt7e3bt2kXXrl3x9/fH1dWVc+fOvfYzFUKUX3LmqESaNWtG3bp1AWjUqBE6OjpMnjyZnTt3Ehsby4ULF6hXrx7nz5+nXbt21KxZEwAbGxtOnz5d6LgvXrzgr7/+okePHgC0bduWatWqcfPmzWLPzcLCQnG83r17c/r0aWbMmFFgUkhhTp48SXJyMrt37way0ztyWrm1a9dOkSxSt25dEhMT+eCDD147r8I+C1VVVX766SeOHDlCbGwsUVFRinQPBwcHgoKCaNu2LadPn2b27NlcunSJcePGcfXqVczMzBg8eHCxPxuQyKrXeVdxSxLzVD5JZJUoEXX1//1xqqio8PjxY0aMGMGwYcPo2bMnqqqqZGVloaKikiuJ49X9CpKVlUXeXhFZWVlkZGSgoqKieO/VUOG81NTUFL/PzMxETU2t0KSQwmRmZuLp6aloMv7w4UOqVatGcHCwIh0jZ+3F7W1R2Gfx4sULHBwcsLW1xdjYmKZNm7J161YAevXqxbJlywgLC8PU1BRNTU3at29PSEgIx44dY9++fQQEBLBx48ZizQGUpwnA2/IuHmCXB+fLJ2kCIN6YiooKHTp0YODAgTRu3JiIiAgyMjJo3749Fy9eJD4+nszMTPbt21fkOFWqVKFu3bocOHAAgAsXLvDw4UOaNGmCrq6uop3coUOHFPuoqanlynYMDw/n2bNnpKSkEBISgqmpaaFJIQXtD2BiYsL27duB7FDjPn36EBcXV+i81dXVC8yXfFVhn8WtW7dQVVXl66+/xsTEJFcaR6VKlTA1NWXp0qWK71oXLVrEnj17sLOzY8aMGVy5cqXI4wohyjc5c1RiycnJxMTEYGNjg4aGBk2bNuXu3bvUrFkTd3d3hg0bRqVKlWjcuPFrx8pJzvD29kZDQwNvb280NTUZNGgQEydOxMbGBhMTE/T19YHsbMecm2saNmxIjRo1GDlyJI8fP8bW1pYvvvgCAwODApNCALp164atrS3+/v6KObi4uDBr1iysra3JyMjA1dWVevXqcfbs2QLnXKNGDYyMjHB2di40CLmwz6JZs2Y0b94cS0tLtLW1MTY25t69e4r9rKys+P333xVhx87Oznz//fcEBASgpqbGzJkzi/EnpFzeViqHpHGId0F6q4q3zt/fn6ioKEWsVEWXkZHBsmXLqFGjBsOHDy+TMZXlsqqyXLoD5VqPrCU/SeUQxaYMiRsFSU5OZsCAAQW+N378eLp161ai8RwcHNDT02P16tVlMT0hRDkkZ45ClANy5lj+KNN6ZC35yQ05QgghRAlJcRRCCCHykOIohBBC5CE35AghykROKkdZS05J59nTl2U+rhBFkeIo8nFzc6N+/fqcO3eOn3/+udDtvL29ARg3blyxxy7qmcOi+Pv7M2XKFJYsWYK1tbXi9U2bNjF//nwOHz5MnTp1ijXW6+Z98eJFRo4cSXBwMIaGhkD2Ha82NjZMnToVc3PzEs//fSCpHEKZyGVVUSADA4MiC2NpvRpFVVK1atUiLCws12sHDx5ER0fnTaeVS5s2bXBwcGDu3LmK15YvX85nn30mhVGI94ScOQqysrJYsGABx44dw8DAgIyMDDp06ICFhQVHjhzhjz/+YO7cuSQlJZGQkMDw4cMZMmQIkB1q3L9/f5KSknB0dGTo0KEArFu3jtDQUDIyMvj8889xdXXlxx9/BP4XRRUeHs6KFStIT0+nTp06zJ07Fz09PRYuXEhERARqamp069YNFxcXAIyNjTl37hxJSUlUrlyZv//+mw8++ICqVf93Ka+g46qoqLB+/Xr8/PzQ09NDR0eH1q1bF/mZTJgwAVtbWw4fPoyRkREHDx4kMDCw0NismJgYZsyYQXp6OlpaWsyfP5/69eu/hT8tIcS/QYqjICwsjCtXrrB3716ePXtGnz59cr2/c+dOxowZQ6dOnbhz5w59+vRRFMd//vmHbdu2kZmZib29PR06dOCff/7h8uXL7Nq1CxUVFVxdXQkKCsLd3R0fHx927txJQkICS5YsYcuWLVSrVg1fX18WL17MmDFjCA8PJyQkhJSUFKZNm0ZKSgqQ3Sv1888/5/jx41haWhIaGoqlpaXiMml4eHiBx23YsCG7d+8mICAAFRUVBgwY8NriqK2tzY8//siUKVOoUaMGc+fOpUqVKixevJiWLVuycOFCnj9/jpOTE23atGHz5s0MHz4cS0tL9u3bx4ULF0pUHCWVo2iSyvHmZC0lI8VREBUVRY8ePdDQ0KB69eqYmprmet/NzY0TJ06wdu1arl27litWqnfv3opwYHNzc6Kiorh//z7R0dGKptzJyckYGRnlGvPixYvExcUpimxmZibVqlXD0NAQLS0tnJycMDc3Z+LEibkSNywtLfHz88PS0pJDhw7x888/K4rjqVOnCjzuw4cPMTMzU0RY9erVK1cSR2E+++wzPvvsM7KysujUqRNQeGyWmZkZc+bM4cSJE5ibm9OzZ89ifvrZlKUJwNsiqRxvRtaSn7SPE6/1ugiriRMnoqOjg7m5Ob179yYkJKTAbbOyslBXVycjI4OhQ4cq+o4+ffo0V2QVZPcn/fTTT1mzZg0AKSkpvHjxAnV1dXbu3ElUVBTh4eE4OTnluoGnY8eOuLu788cff6Cnp5frkmphx92xY0e+9RUW1JxX3qJeWGyWhoYG7dq14+jRo2zevJnjx4/j4eFRrGMIIcofKY6CTp06sWHDBgYOHMjLly85ceIEbdu2VbwfERFBaGgohoaGipSMnPimsLAwBg8ezMuXLzl69Chr1qzhww8/ZMWKFTg6OqKlpcXYsWOxs7PD3t5eEUXVpk0b3N3diY2NpUGDBqxatYr4+HiGDBnC3Llz8fHxoVOnTly5coXY2FjFXNTU1Pj888+ZMWMG//nPf3Ktw8TEpMDjdurUiQkTJjBu3Dg0NTU5ePAgZmZmpfqscmKzPDw8ePDgAX379sXX15elS5diZWWFk5MTjRo1Yv78+aUavyKTVA6hTKQ4Crp3786lS5ewtramZs2aNGrUKNf748aNY9CgQejo6NCgQQNq166tiJYyMjLCycmJlJQURo8eTaNGjWjUqBExMTE4OjqSkZHBF198gZ2dHZA7imrevHlMnDiRzMxMDA0N8fT0RE9Pj7Zt22JtbU2lSpVo3rw5pqamBAUFKeZjaWnJnj17sLCwyDVPCwuLAo+roqLC0KFD6devHzo6OvnOBkuisNisr7/+mmnTprFq1SrU1NRwc3Mr9TEqKk0NNaW5dCeENB4XohxQlu8clak4KtN6ZC35yXeOQhTC2dmZp0+f5nvdycmJgQMHvoMZCSHKCymO4r1Vmk49Qoj3g3TIEUIIIfKQ4iiEEELkIcVRlDuXLl1i2rRpZTLW3bt3FXe1Ll++nMOHD5dqHGdnZyIjI3O95ubmpni0RfwvlaOsflXVqfSulyTeY/Kdoyh3PvnkEz755JMyH3fChAllPqb4n7JO5ZA0DvEuSXEU5U5kZCQrV64EsgvluXPnSEhIwN3dHTMzM4KDg1m/fj1qamrUqVMHT09PLly4wMqVKxU32bi5udGhQwc6dOigGPfV11xcXGjSpAlXr16lRo0aLF++HF1d3VLP2cTEBHNzcy5fvswHH3zA4sWLix2hJYQof+SyqijX0tLS2LFjB1OmTGH58uUAeHl58csvv+Dv70+DBg24efNmiceNiYlh+PDh7N27Fx0dHYKDg99ono8fP6ZDhw4EBwdjZWUlreOEqODkzFGUa1988QUATZo04cmTJ0B2g/OBAwfSrVs3evbsSfPmzfN9H/g6NWrUoEWLFoqxExMTi9xeRUUl32tZWVmoqmb/fKmlpUXfvn0BsLOzY+nSpSWcj6RyFORdJ0m86+OXJVlLyUhxFOVaTiLHq8XJ3d2dmJgYjh8/jqurKy4uLnz44Ye82uwpLS2tWOPmjP26RlHVqlXj2bPc34A9evRIEbSsqqqqmGNmZma+Ruuvoywdcsrau+zqIl1lyqd/q0OOXFYVFUp6ejo9evRAT0+P0aNHY2try9WrV9HT0+POnTukpKTw5MkTzp07V6bHNTExITAwkPT07CbYN27c4PLly4oG7S9fvuTIkSMA+Pv754v9EkJULHLmKCoUdXV1xo8fz/Dhw9HW1kZHR4eFCxdiaGiImZkZVlZW1K5dm/bt25fpcQcMGMCdO3ewtbVFVVUVLS0tlixZQvXq1RXb7N+/n2XLlmFgYMDChQvL9PgVQVmnckgah3iXpPG4EGWgadOmXLt2rdT7K8tlVWW5dAfKtR5ZS37SeFyIYpJG5EKIHFIchfj/3qQR+ZucNQohyh+5IUcIIYTIQ4qjEEIIkYcURyGEECIPKY7l1OHDhxXt0soLb29vvL29AbC1Lbtb9kvjTRI2oqOjGTJkCD179sTKyoopU6aQkJCgeN/Z2Vnx+6ZNm77xXIUQFY/ckFNOdevWjW7dur3raRRqz56yS18ojdImbPz555+MGTOGRYsW0blzZzIzM1m/fj1Dhgxh9+7daGlpERUVVcazfT/kRFaVheSUdJ49fVkmYwlRGlIc34HIyEi8vb1RV1cnLi6O1q1b88033zBmzBj09PTQ0tKiT58+REVFsWDBAiwsLLC0tOTYsWOoqanx3Xff8csvv3D79m0mT55M7969+eOPP5g7dy5JSUkkJCQwfPhwhgwZgre3NxcuXCAuLg4nJyd++eUXjhw5gqqqKlFRUaxbt47169cXOtf169fj5+eHnp4eOjo6tG7dGvjfc33x8fFMnTqVZ8+e8c8//2BlZcWkSZNIS0tj5syZnDt3DkNDQ1RUVBgzZgwAa9euRVtbmxs3btC0aVMWL16MpqYmu3fvZuPGjaioqNCyZUumT5+OpqYmU6dO5fr16wAMGjQIR0dHRcJGjx49+O6773j48CEAY8eOLfKHivXr1zNgwAA6d+4MZLd9GzVqFAcOHCA0NJTLly8D0L9/f3bu3AnAjBkzuHDhApB99vzRRx8RHR3N/PnzSU5ORk9Pj9mzZ1O3bl2cnZ2pVq0a169fx8vLi+bNm7/B35SKpSwjqySuSrxrcln1HYmOjmbGjBns37+flJQUjh8/TmxsLJ6enmzatCnf9gYGBoSEhNCyZUvWrVvHL7/8gqenJ+vWrQNg586djBkzht27d7NlyxaWLVum2Dc1NZV9+/YxZMgQ6tSpo2jSHRAQgL29faFzvHTpErt37yYgIICNGzdy//79fNvs3bsXa2tr/Pz8CAoKYtu2bSQkJODr68vLly/Zv38/8+fP59KlS4p9zp8/z4wZMwgNDeXevXv89ttvXLt2jTVr1uDj40NwcDCVKlVi5cqVnD9/nsTERAIDA9m4cSO///57ruMfPHiQ2rVr4+/vj6enJ2fPni3yc7906ZKiwL/K2NiYy5cv4+7urvg8c3Tu3JmgoCC6dOmCr68vqampuLu7s2TJEgICAhg+fDjTp09XbN+0aVPCwsLeq8IohLKRM8d3xNjYmIYNGwLZ39/5+flRo0aNQjMAc3p1GhkZYWBggLq6OkZGRoqH1t3c3Dhx4gRr167l2rVrJCUlKfZ9tRg4ODgQFBRE27ZtOX36NLNnzy50jlFRUZiZmfHBBx8A0KtXLzIzM3NtM2LECE6fPs2GDRu4fv06aWlpvHz5koiICBwdHVFRUaF27dp06tRJsU+TJk2oVasWAI0aNSIxMZF79+5hbm6Onp4ekN2ubcqUKYwaNYrY2FhGjBiBqakpkyZNynX8du3asXTpUuLj4+natStjx44t4lPPbjKe0x/1VUU1Ku/evTsAjRs35uzZs9y6dYs7d+7wzTffKLZ5/vy54vcFFd/XkVSO/MpDikR5mENZkbWUjBTHd+TV1IasrCzU1NTQ1tYudHsNDQ3F79XV8/+xTZw4ER0dHczNzenduzchISGK914dt1evXixbtoywsDBMTU3R1NQs9JgqKiq5iqG6ujqpqam5tlmwYAF37tzB2tqa7t27c/LkScV68hbSHAUlYuTdNisri/T0dPT09AgJCSEiIoLjx49jZ2eXa23169cnNDSUEydOcPToUX755RdCQ0MLjJiC7MJ14cKFfJdez58/n+tGnFflfN6vzrVOnTqK710zMjIUl3WBIv8cC6Ms7ePK0rtudyYt18onSeVQcufOnSM+Pp7MzEwCAwPfOMUhIiKC8ePH0717d86cOQNk/6OdV6VKlTA1NWXp0qVFXlIF6NSpE8eOHePZs2ekpKRw8ODBAo87YsQILC0tiYuLU6ypc+fO7Nu3j6ysLOLj44mKiiq0YAF06NCBI0eOKDIb/fz86NixI4cPH2bSpEl07doVd3d3KleuTFxcnGK/X3/9FW9vbywtLZk5cyYJCQn5oqVeNXr0aHbv3k1ERASQXYRXrVpFcnIylpaWQPYPLgWdXeZo2LAhiYmJiku4u3fvzndGK4So2OTM8R0xMDDghx9+ID4+ni5dutC5c2fF94elMW7cOAYNGoSOjg4NGjSgdu3a3L17t8Btrays+P3332nTpk2RYzZv3pyhQ4fSr18/dHR0MDIyyrfN6NGj+eGHH9DR0aFGjRq0atWKu3fv4ujoSExMDDY2Nujr62NkZIS2tjYvXxZ8B2KzZs0YPXo0zs7OpKWl0bJlS2bPno2WlhZhYWFYWVmhpaVFjx49cj1e0bdvX7777jtsbGxQV1fHxcVFkbFYkI8++ogNGzawePFiPDw8yMjIoH379vj4+CjOaLt164atrS3+/v4FjqGpqcny5cv58ccfSUlJoUqVKu9lCocQykxSOd6ByMhIVq5c+Ua9PEsrIyODZcuWUaNGDYYPH/7WjnPs2DGysrIwNzfn2bNn9O3bl927d6Orq/vWjlmRKcNl1Wq6ldHUKFnIc2HKw6MccimyfJJUDvFWODg4oKenx+rVqwH466+/GDduXIHbenh48Mknn5TqOI0aNeKHH37Ay8sLgPHjx/9rhXHhwoWcPHky3+utWrXixx9//Ffm8D7S1FBTmn+AhZAzRyHKAWU4c1SmsxNQrvXIWvKTG3KEEEKIEpLiKIQQQuQhxVEIIYTIQ4qjKLa7d+9iYWHxrqfxr1i+fDm9e/fGysqKjRs3Kl4/efIkNjY29OjRI1eLvqtXr+Lg4EDPnj2ZNm1akc9JCiHKP7lbVby3fHx8+Oyzz/L1QI2KiuL06dMEBQWRnp5O7969MTMzw8jIiKlTp+Lj48OHH37I6NGjOX78OGZmZri6uuLh4UHbtm2ZOnUqfn5+DBo06B2t7N2QVA6hTKQ4ViCRkZGsWbOGrKws/vrrL3r27EnVqlU5dOgQAOvWrWP//v3s2bOHly9foqKigpeXF5UrV8be3p5ff/2VunXr4uDgwPfff0/Xrl0LPdaSJUsICwtDT08PfX19LCws6NChg+L9nFSMnC47OSkdhYmJiWHGjBmkp6ejpaXF/PnzqV+/PuHh4axYsYL09HTq1KnD3Llz0dPTw8LCgtatW3P16lU8PT2ZOHEiR44cAVBkSo4bN44uXbpgbm7O2bNn0dfXZ9CgQfj4+HD//n0WLFiQa855GRkZMXPmTLS0tBg6dCgWFhaoqqrSoUMHtmzZgrq6OvHx8WRkZFC5cmWio6P56KOPqFu3LgA2Njbs37+fxo0bk5ycTNu2bQGwt7dnxYoV711xlFQOoUzksmoFc/HiRebPn09ISAi+vr5Ur14df39/mjZtSkhICIcOHcLHx4e9e/fSvXt3tm3bxocffsikSZOYNWsWP/30E+3atSuyMB45coRz586xd+9e1q1bx5UrV9543ps3b2b48OH4+/vj7OzMhQsXSEhIYMmSJWzYsIHAwEA+//xzFi9erNjH1NSUsLAwqlevXui4Dx8+pGvXruzfvx+AQ4cOsW3bNsaNG8fmzZuLnFO3bt3w8/PD1dWVsLAwrK2tFVmOGhoarFixAisrKzp16oShoSEPHjxAX19fsb+BgQHx8fH5XtfX1yc+Pr5Un5MQonyQM8cK5uOPP+bDDz8EQE9PT5F2kZPQsWTJEkJCQrh16xYnTpxQXDJ0cHAgNDSU4OBg9u7dW+QxTp48iaWlJZqammhqaipSKd6EmZkZc+bM4cSJE5ibm9OzZ0/Cw8OJi4tjyJAhAGRmZlKtWjXFPq9rb5cjpy9t7dq1ad++PUCuxJLXUVVVzfUrx/jx4xk5ciRff/01fn5+VK5cOd++Oc3IC3q9JCSVI7/ykCJRHuZQVmQtJSPFsYJ5NZ0Dcqd7xMXFMWDAAAYPHoypqSk1a9bk6tWrAKSkpHD//n0yMjK4f/++Ii6rIKqqqoUmauR4tSgUFfeUo1evXrRr146jR4+yefNmjh8/TteuXfn0009Zs2aNYo4vXrxQ7JPT6zRvAUpPT8+VTPJqssirn8frHDt2jLVr16Kurs7QoUOZP38+qqqq3Lhxg9TUVJo3b06lSpXo0aMH165do1evXrnSNx48eICBgQGGhoa5Xv/nn38wMDAo9jxAeZoAlKV3/dC6PDhfPkkTAFFily5d4qOPPmLYsGG0adOG8PBwRTKHl5cXJiYmTJkyhalTpxZZ/Lp06cKBAwdITU3l+fPnHDt2LN+ZkK6uLn/++SeA4jvPokycOJHo6GicnJyYMGECV65coU2bNly4cIHY2FgAVq1axaJFi/Ltq6OjQ2JiIgkJCaSmpnLixIlifyZFuXXrFu7u7vj4+NC9e3fFWePdu3dxd3cnNTWV1NRUDh8+TPv27WnTpg2xsbHcvn2bjIwM9u7di6mpKbVr10ZLS4tz584BlEnKihDi3ZIzRyXy+eefExMTQ+/evdHU1KR169Zcv36d8+fPExYWRlBQEFWqVCEgIIANGzYwcuTIAscxMzPj999/x87OjmrVqmFgYJArgxFg0KBBTJw4ERsbG0xMTHJ951aQr7/+mmnTprFq1SrU1NRwc3NDX1+fefPmMXHiRDIzMzE0NMTT0zPfvlWrVmXEiBH069ePWrVqlbrfa17Dhg0r8HUzMzMuXrxI3759UVNTo0ePHlhZWQHZ+ZXjxo0jJSUFMzMzevXqBcDixYtxd3fnxYsXtGjRQnGpWAhRMUlvVZHP+fPnuXXrFnZ2dqSlpTFgwADmzZtHs2bN3vXUlJYyXFaVVI7yS9aSn6RyiAKdPXuWuXPnFvjeunXrWLlyJRs3biQrK4u+ffsWqzDu27ePtWvXFvjenj1lc4t/SUlCx79HUjmEMpEzRyHKAWU4c1SmsxNQrvXIWvKTG3KEEEKIEpLiKIQQQuQhxVEIIYTIQ4qjEEIIkYfcrSrKHWdnZ1xcXOjYseO7noqChYUFNjY2fPvtt4rX8jZff9+VVSpHeXiMQwgpjkIU0+bNm/nyyy9p1arVu55KuVRWqRySyCHKAymO4p3Kyspi8eLFHDp0CDU1NQYMGADAzp07WbhwIYmJiUybNg0LCwv++OMP5s6dS1JSEgkJCQwfPpwhQ4bg7e1NfHw8t2/f5u+//6Z///588803pKWlMXPmTM6dO4ehoSEqKiqMGTOGjh07sm7dOkJDQ8nIyODzzz/H1dX1tc3CR48ezZQpU9i9e3eufq4AR48excvLi8zMTOrWrcucOXOoWbPmW/vchBBvlxRH8U7t37+f33//neDgYNLS0hg0aBApKSk0adIEf39/jh49ysqVK7GwsGDnzp2MGTOGTp06cefOHfr06aNo03bt2jW2bt3Ks2fP6N69O//5z38UuZb79+/n3r172NjYABAeHs7ly5fZtWsXKioquLq6EhQUhK2tbZFztbGx4dKlS/z000+5Lq8+evSIGTNmsH37durUqcP69euZM2cOK1asKPbnIKkcuZWXBInyMo+yIGspGSmO4p06c+ZMrnisPXv24OzsrIjJaty4MY8fPwayv+M7ceIEa9eu5dq1ayQlJSnG6dixI5qamtSoUQNdXV2ePXtGREQEjo6OqKioULt2bUW816lTp4iOjlZ8V5icnIyRkVGx5jt79mxsbW358ssvFa9FR0fTunVr6tSpA8CAAQNYt25diT4HZWkCUFbKwwPr8uB8+STt48R74dXoKchOxEhKSlJET716qXPixIno6Ohgbm5O7969CQkJUbz3amP0nIgrNTW1AtNHMjIyGDp0KMOHDwfg6dOnxY660tfXx83NjSlTpvDxxx8D5DtGVlYW6enpxRpPCFE+yaMc4p0yNjbm4MGDpKWl8fLlS7766ivi4+ML3DYiIoLx48fTvXt3zpw5A6CI5CpI586d2bdvH1lZWcTHxxMVFYWKigomJibs2bOHFy9ekJ6eztixYwkLCyv2nPv06UPdunUV+7Rp04aLFy9y9+5dAHbs2FGu7rQVQpScnDmKd+rLL7/k8uXL2Nvbk5mZyZAhQwgNDS1w23HjxjFo0CB0dHRo0KABtWvXVhSkgjg6OhITE4ONjQ36+voYGRmhra1Nhw4diImJwdHRkYyMDL744gvs7OxKNO/Zs2djbW0NQM2aNZkzZw4uLi6kpaVhZGT0XjY1T03LIHhJ0d/bFkdyipx1i3dPGo8LpXXs2DGysrIwNzfn2bNn9O3bl927d6Orq/uup5aPsnznqCzfa4FyrUfWkp985yjeW40aNeKHH37Ay8sLgPHjxxdaGJOTkxWPkeQ1fvx4unXr9pZmKYQoj+TMUYhyQM4cyx9lWo+sJT+JrBJCCCFKSIqjEEIIkYcURyGEECIPuSFHCFEmipvKIakboiKQ4iiK5Y8//sDGxoYVK1bQs2fPXO8tX74cVVVVxo0bp3jt2LFjrFmzhqSkJDIzM+nevTvjx49HVfXtXKzw9/cnKiqKBQsWsGLFCjp37sxnn31W6Pbe3t4EBwcTFBSEtrY2AJGRkaxcuRIfH5+3MkdlV9xUDkndEBWBXFYVxeLv70/Pnj3x9fVVvPbs2TOmTp3KL7/8kmvb8PBw5syZw/z58wkKCmLXrl3ExMSUqBH3mzhz5kyRnXNy3Lt3j6VLl/4LMxJCVDRSHMVrpaenExQUxLfffsuVK1f466+/ADh8+DD169dX9CjNsWbNGlxcXGjQoAEA2trazJo1iw4dOgD/CzPu2bMnV69eJTw8nH79+tG3b19cXFwUjcYXLlxInz59sLOzY+XKlUD2GZ+3t7fiWBYWFrm65AQGBnL58mXc3d25du1aketycnJi3759nD17Nt97Dx8+ZPTo0djY2GBnZ0d4eLji+O7u7jg7O2NhYcHq1auB7DZ28+fPx87Ojj59+rBp06Zif75CiPJHLquK1zp27BhGRkY0aNCA7t274+vryw8//EDfvn0BchUrgKtXr9KmTZtcr9WqVYtatWop/rtp06asXLmShIQE3Nzc2LJlC9WqVcPX15fFixczZswYwsPDCQkJISUlhWnTppGSkvLaueZ0wXFxcaFp06ZFblutWjVmzZrFtGnT2LMn9+XAuXPnYmJiwvDhw7lz5w4DBw4kMDAQKDgeK6cJekBAAKmpqYwYMYJWrVoVeWn3Ve9bZFVFiU+qKPMsDllLyUhxFK/l7++v6CPau3dvJk2axMSJE/MF/ubIScUoSuvWrQG4ePEicXFxilzGzMxMqlWrhqGhIVpaWjg5OWFubs7EiRNzJW+Ule7duxMaGsrSpUtzdcE5ffo0Hh4eANStW1fRXBwKjsc6deoUV69e5fTp0wAkJSVx7dq1YhdHZWkCUFwV4YF0eXC+fJL2caJcePTokSIceMuWLWRlZfH06VMOHDigKJh5tWrVisuXL9O4cWPFa7GxsaxevZpFixYBKG6CycjI4NNPP2XNmjUApKSk8OLFC9TV1dm5cydRUVGEh4fj5OSEj48PKioquSKi0tLS3niN06dPx9raOldrubzFPSsrS/E9ZkHxWBkZGbi6utKjRw8AEhISqFy58hvPTQjxbkhxFEUKCgrCxMSE9evXK17z9vZmx44dhRbHr776ijlz5tC2bVvq16/PixcvWLBgAc2aNcu3bZs2bXB3dyc2NpYGDRqwatUq4uPjGTJkCHPnzsXHx4dOnTpx5coVYmNj0dPTIzIyEsgOGf7nn3/yjammplasG3Jy6OrqMmvWLCZOnEi7du0AMDExYdeuXYrLqr///juzZs0q9HtMExMT/Pz8MDc3JzU1lUGDBjF79uz3KrqquKkckrohKgIpjqJI/v7+fPvtt7leGzRoEOvXr+fGjRs0atQo3z6mpqZ8++23fPvtt2RkZJCenk6vXr1wcXHJt62+vj7z5s1j4sSJZGZmYmhoiKenJ3p6erRt2xZra2sqVapE8+bNMTU15dmzZ4SFhdG7d29atmxJixYt8o35xRdfMHPmTBYuXMinn35arHV2796dnj178uDBAwCmTZvGjBkz8Pf3B8DDwwMDA4NC93dycuL27dvY2dmRnp6Ovb39e1UYIftRDmW5dCeENB4XohxQlu8clak4KtN6ZC35yXeO4r21cOFCTp48me/1Vq1avZdhxEKI4pPiKJTW5MmT3/UUhBAVlDQBEEIIIfKQ4iiEEELkIZdVhRBlojipHJLIISoKKY7irVG2JI/C7Ny5k19//VXx33fv3sXW1pYZM2aU5XTLveKkckgih6go5LKqeGuULcnDx8eHq1ev5nu9f//+7Nmzhz179rB48WJq1KhR4DOdQoiKQ4qjeCuUMcnDyMiImTNn4uzszKFDh3K1scsxa9Ysvv32W6pXr16aj00IUU7IZVXxVihjkke3bt3o1q0b0dHR+Pj4sHTp0lwF/OTJkyQnJ2NpaVncj0nhfUrlqEjpEBVprq8jaykZKY7irVDmJA9VVdVcv3L4+vrmOyMuLmXpkFMcFaVTi3SVKZ+kQ46osJQ1yePYsWOsXbsWdXV1hg4dyvz58xXFMTU1lTNnzrBgwYJSjS2EKF+kOIoyp6xJHrdu3cLd3Z2WLVvme+/atWvUr1//vY6pKk4qhyRyiIpCiqMoc8qa5DFs2LBC13znzp1c34++jySVQygTSeUQohxQlu8clak4KtN6ZC35yXeOQpSAJHkIIUCKoxC5SJKHEAKkCYAQQgiRjxRHIYQQIg8pjuVcZGQkzs7OAEybNo1Lly6VeAw3Nzf8/f3LemrFNnLkSOLj4wt9/9U17tixg71795bqOBYWFvkanKenp2NiYoKbmxtQus9w+/btbN++vVRzep/kpHIU9KuqTqV3PT0hSkS+c6xAKuoNIT///HOxtz1//ryiHVtpJCcnc+3aNUUbuFOnTqGioqJ4vzSf4cCBA0s9n/dJUakcksYhKhopjm9JZGQka9asISsri7/++ouePXtStWpVDh06BMC6deu4cuUKK1asID09nTp16jB37lz09PT47bffmD9/PlpaWopG3PC/5tsdOnRg8eLFHDp0CDU1NQYMGMDQoUOJiopi2bJlJCcnk5iYiKura7H6fKalpTF16lSuX78OZD+T6OjoiJubGyoqKvzxxx88f/6cb775hr59+/LixQvmzJnD9evXycjIYOTIkVhbW5OSksLs2bM5d+4cGhoajBkzht69e2NhYcGWLVvQ1dVl6tSpxMfH8+DBAz777DNF9xvI7k165MgRTp8+jY6ODtOmTePw4cNUqVKFu3fvMnr0aEJCQopcS48ePQgLC1MUx3379tGzZ0+Sk5NzfYYfffQRkyZNIikpCVVVVdzd3Wnbti0LFy4kIiICNTU1unXrhouLi6IP7Lhx4/j888/p2bMn586dQ01NDS8vL+rWrUtkZCQeHh6oqanRtm1bbty4gY+PTwn+xgghyhO5rPoWXbx4kfnz5xMSEoKvry/Vq1fH39+fpk2b4uvry5IlS9iwYQOBgYF8/vnnLF68mNTUVNzc3FixYgX+/v6Klmmv2r9/P7///jvBwcHs3LkTf39//vnnH3799Vc8PDwICAjgxx9/ZNWqVcWa5/nz50lMTCQwMJCNGzfy+++/K96Lj4/H19eXzZs3s2jRIv755x9Wr15Ny5Yt8ff3Z+vWraxZs4Y7d+7g4+NDUlISoaGhbNy4kZ9++onU1FTFWMeOHaN58+bs2LGDsLAwLly4wH//+1/F+507d8bCwoLx48fTvXt3unbtyv79+4Hs5Axb26K7rwD06tWLgwcPAtkt3WJiYhQ9WV+1a9cuunbtir+/P66urpw7d46///6b8PBwgoKC8PX15datW/kal//zzz906tSJwMBAjI2N2bp1K2lpafzwww94enoSGBiIurr8zClERSf/F79FH3/8MR9++CEAenp6dOrUCciOPjpy5EiBzbOvXbuGgYGBoouMnZ0dy5cvzzXumTNnsLS0RFNTE01NTfbsyb6U5enpydGjR9m/fz8XL17kxYsXxZpnkyZNiI2NZcSIEZiamjJp0iTFe/b29mhoaFCrVi0+/fRTzp07p0if2L17NwBJSUlcv36dM2fO4OjoiKqqKvr6+vnO8qytrYmOjmbTpk3cvHmTJ0+ekJSUVOi8HBwc8Pb2pl+/fuzdu5fNmze/di2GhoZUqVKFGzdu8Ndff9GlS5cCt+vUqRPjxo3j6tWrmJmZMXjwYNTU1IrVuPyLL75QfG5nz57ljz/+oEaNGopWd/369Svx5dv3IZWjIqZCVMQ5F0bWUjJSHN8iDQ2NXP+tpqam+H1mZmaBzbPv3buXq0n2q/vkyHtmcvfuXapXr46zszMdO3akY8eOdOrUKVeRK4qenh4hISFERERw/Phx7OzsFIUt75zV1dXJzMzE09NT0WP04cOHVKtWTVEsc9y+fVvxwwFkhwWHhYXh6OhI586d+eOPP4pM4jA2NubBgwccOHCAOnXqYGhoWKz19OrVi/3793P79m2GDRtGTExMvm3at29PSEgIx44dY9++fQQEBLBx48YCG5fnlVMwc5JE1NTUCsx2LAll6ZBTlIrWoUW6ypRP/1aHHLms+o60bt2aCxcuEBsbC8CqVatYtGgRTZs25dGjR4p/0Av6js3Y2JiDBw+SlpbGy5cv+eqrr/jzzz+5desWEyZMwMzMjIiIiNc20s5x+PBhJk2aRNeuXXF3d6dy5crExcUBEBoaSlZWFn///TfR0dG0b98eExMTxd2bDx48oE+fPsTFxWFsbKzY/tGjRwwePDjXZdWIiAgGDBhAnz59UFFRISYmJl9RebUBuIqKCn379sXDwwN7e/tif7Y5xfHGjRsF9lEFWLRoEXv27MHOzo4ZM2Zw5coVrly5wuDBgzE2Nmby5Mk0atRI8edTlIYNG/L06VNFUHJwcHCx5yqEKJ/kzPEdKax5toaGBkuXLsXV1RV1dfUC/3H/8ssvuXz5Mvb29mRmZjJkyBBat25N//79sbKyokqVKrRt25bk5OQiL1vmMDU1JSwsDCsrK7S0tOjRo4fihpbk5GQcHBxITU1lzpw56Onp4eLiwqxZs7C2tiYjIwNXV1fq1avHoEGD8PDwoE+fPgBMnz6dKlX+95PZ0KFDmTVrFr/88gsffPAB7dq14+7du9SrV0+xTefOnVm6dClVq1alV69eWFlZsXHjRrp3717sz9bQ0JCqVasWeders7Mz33//PQEBAaipqTFz5kxatGhRYOPyV78XLYimpiaLFi1i8uTJqKqq0qBBgwK/KxZCVBzSeFwUys3NjQ4dOpTorK0sZWZmsn37dmJjY3F3d38ncyiOzMxMFi9ejIuLC5UrV2bjxo3Ex8crnq0sDmW4rFpNtzKaGvm/BoDsqKpnT1/+yzN6M3IpsnySxuOiTCUnJzNgwIAC3xs/fjzdunX7l2f0ei4uLsTFxbFhwwag/K5BVVUVXV1d+vXrh4aGBrVr166wz6S+CYmsEspEzhyFKAeU4cxRmc5OQLnWI2vJT27IEUIIIUpIiqMQQgiRhxRHIYQQIo/3rjhKykXxOTs7ExkZWeQ2K1as4OzZs6Uav7he/bybNm3KiBEjcr2fkJBAy5YtFT1Qi+Pu3btYWFgAsHz5cg4fPlzotq97XwihfN7ru1Ur6h2F/2bKxeucOXOGjh07vrXxC3Lr1i0SExOpVq0aAAcOHEBHR6fU402YMOGN3hfZciKr8qqIj3EIUWGKo6RcvJuUixwFpVGcO3eOy5cv4+7uzsqVK9HW1mbWrFk8efIEbW1tpk+fTosWLXBzc+PJkyfcvn0bV1dXRaOA3377jZcvX7Jw4UJatWpV7M/bwsKCQ4cO4eDgAEBYWBhffvml4v3o6Gjmz59PcnIyenp6zJ49m7p163LlyhWmTZsGoOiDCrmf59y0aRPbt29HTU0Nc3NzXF1dFe936NABFxcXmjRpwtWrV6lRowbLly9HV1eX8PDwAv/uvU8Ki6ySuCpREVWoy6qScvHvp1zkKCiNom/fvrRq1QoPDw+aNm3K5MmTcXV1JSAggLlz5/Ltt98q9tfV1SU0NFRxKVNXV5ddu3bh5OTE2rVrAYr9eVtaWhIWFqaYV1ZWFvr6+kB2Eoe7uztLliwhICCA4cOHM336dIBc86tTp06+caOjo9m2bRu7du0iKCiI//73v1y+fDnXNjExMQwfPpy9e/eio6NDcHAwCQkJBf7dE0JUXBXmzBEk5eJdpFy8Km8axatevHjB5cuXmTJliuK1pKQkHj9+DJAvNurVsQ4cOAAU//Nu164dsbGxPHv2jLCwMHr27MnDhw+B7Euud+7c4ZtvvlFs//z5cxISEnjw4AGdO3cGsv8c8jZKP3PmDObm5lStmn1pcNOmTfmOXaNGDUVLvyZNmpCYmMjFixcL/LtXEsqeylFREyEq6rwLImspmQpVHCXl4t2kXOTIm0bxqszMzFw/WADcv38fXV1dgHxn7K+OlWPQoEHF+rxVVFQwNzfn8OHDHDhwAC8vL7Zu3aqYR506dRTzyMjI4OHDh/nmXJy/B/Hx8VSqVKnAeb/6OWRkZBT4d68klKUJQGEq4gPo8uB8+SRNAEpIUi7eXspFUXLGr1q1KvXr11cUpYiICP7zn/8Ue5wnT56U6PO2tLRk27ZtaGhoUL16dcXrDRs2JDExUXFmu3v3biZNmoSenh5GRkYcO3YMoMA7eD/77DPCw8N58eIF6enpfP/99/kuqxakTZs2Bf7dE0JUXBXqzLEoknLx9lIuivLFF18wc+ZMFi5ciKenJ7NmzWL9+vVoaGiwbNmyXGeGRdHV1S3R5922bVv++ecf+vfvn+t1TU1Nli9fzo8//khKSgpVqlRh4cKFQPZl2ylTpuDl5UXbtm3zjdmyZUsGDx6Mk5MTmZmZfPnll3Tu3JmgoKAi517Y3z0hRMUlvVX/RZJyIQqjDJdVC0vlqKiPcsilyPJJUjnKufKaEFGUipJyISomSeUQykTOHIUoB5ThzFGZzk5AudYja8nvvbkhRwghhCgrUhyFEEKIPKQ4CiGEEHlIcRRCCCHykOJYQt7e3iWKRipNLNb27dsVTQHeB4Wt19/fHzc3tyL3dXNzo2vXrtja2mJra0uPHj1wdHTkxo0bRe537949evXqhb29Pc+fP3+j+edVnKgvZZSTyvHqr6o6lV6/oxDlkDzK8ZaVJhZr4MCBb2Em5debrnf8+PG5nh398ccf8fb2xsvLq9B9oqKiaNmyJUuWLHmjY4v/KSiVQxI5REUlxfEV6enpzJo1i+vXr/Pw4UMaNGjAypUr+fXXX/Hz80NPTw8dHR1FE+0uXbpgbm7O2bNn0dfXZ9CgQfj4+HD//n0WLFhAhw4dFLFYH330EZMmTSIpKQlVVVXc3d1p27YtCxcuJCIiAjU1Nbp164aLi4vizHTcuHEcPXoULy8vMjMzqVu3LnPmzKFmzZpYWFgUGPtUmODgYNavX4+amhp16tTB09MTLS0t1qxZQ1BQEGpqanTp0gVXV1dFE+0jR44A5JqPiYkJLVu25OHDh+zatQsvL698UV+3b98uMLqqMK+OHxgYyOrVq6lSpQq1a9emcuXKJfozTE1N5Z9//lE0/i5oLioqKnh5eZGUlMSMGTOYPHlygZFh/v7+BAQE8OTJE8zNzXnw4EGu6K2UlBQ2btxIcnIyKSkpeHh4YGxsXKL5CiHKJymOrzh//jwaGhrs2LGDzMxMhg4dypYtWwgICCAgIAAVFRUGDBigKI4PHz6ka9eueHh44OzszKFDh9i2bRsBAQFs3rw5V8jwrl276Nq1K1999RWRkZGcO3cOfX19wsPDCQkJISUlhWnTppGSkqLY59GjR8yYMYPt27dTp04d1q9fz5w5c1ixYgXwv9gnHx8f1q5dW+TlXi8vL/z8/KhRowbLli3j5s2bPHjwgCNHjuDv74+6ujrjxo3D19cXMzOzQsd5/Pgxo0aNomPHjoSGhiqivtLS0hg0aBC9e/dm8uTJzJgxgxYtWvDnn38yduxYRcRUUeLj41m8eDGBgYHo6uoyevToYhXHFStWsGnTJp48eYKWlhbdu3dn7NixAIXOZfz48URFRTFnzhwWL15My5YtWbhwIc+fP8fJyYk2bdoo5rRv3z7U1dVxc3NDV1eXNWvWkJmZyfDhw1mzZg3Vq1dn165dbNiwodTFUZlTOSpyGkRFnntespaSkeL4CmNjY3R1ddm6dSs3b97k1q1bdOzYETMzMz744AMAevXqlau5t6mpKQC1a9emffv2QHaE1tOnT3ON3alTJ8aNG8fVq1cxMzNj8ODBqKmpoaWlhZOTE+bm5kycODFX6kN0dDStW7dWZA8OGDCAdevWKd4vKPapMObm5gwcOJBu3brRs2dPmjdvTlBQEFZWVorEDAcHBwIDA4ssjoCicBQU9VVUdNXrwn/Pnz9Pu3btqFmzJgA2NjacPn26yH3gf5dVb968yf/93//RsWNHqlSp8toYrRyFRYYBtGjRIldaR84PRqqqqvz0008cOXKE2NhYoqKiUFUt/Vf4ytIEoCAV9eFzeXC+fJL2ce/A4cOHWbFiBUOGDMHe3p7Hjx9TuXLlXIVOXV09VzKGpqam4vcFxSDlaN++PSEhIRw7dox9+/YREBDAxo0b2blzJ1FRUYSHh+Pk5ISPj49in7wJG1lZWaSnpyv+u6DYp8K4u7sTExPD8ePHcXV1xcXFJd/4kH1pOW+8U3p6eq4CkVNMC4r6qlatWpHRVUVRUVHJNae8479Ow4YNmTRpElOnTlWcqRZnLoVFhgUHB+eL2sr57xcvXuDg4ICtrS3GxsY0bdpUEZslhKj45G7VV5w6dQpLS0scHByoWbMmZ86cAeDYsWM8e/aMlJQUDh48WKqxFy1axJ49e7Czs2PGjBlcuXKFK1euMHjwYIyNjZk8eTKNGjVSxB5B9hnaxYsXuXv3LgA7duygY8eOJT52eno6PXr0QE9Pj9GjR2Nra8vVq1cxMTEhJCSE5ORk0tPT2b17NyYmJujo6JCYmEhCQgKpqamcOHGiwHELivp6+PBhqaOr2rdvz8WLF4mPjyczM5N9+/aVeK3W1tbUrVuXVatWFTtGq7DIsKLcunULVVVVvv76a0xMTAgPDy92pJkQovyTM8dX9O/fn0mTJrF//340NTVp27YtiYmJDB06lH79+qGjo4ORkVGpxnZ2dub7778nICAANTU1Zs6cSYsWLWjbti3W1tZUqlSJ5s2bY2pqyn//+18AatasyZw5c3BxcSEtLQ0jI6NS3f2qrq7O+PHjGT58ONra2ujo6LBw4UIMDQ25evUqDg4OpKen88UXXzB48GDU1dUZMWIE/fr1o1atWnzyyScFjltQ1FeDBg1KHV1Vs2ZN3N3dGTZsGJUqVaJx48YlXivADz/8wLBhwxg0aFCx5lJYZFhOJmRBmjVrRvPmzbG0tERbWxtjY2Pu3btXqvkqi9S0DIKX2OZ6LTklvZCthSjfpPG4EOWAsnznqCzfa4FyrUfWkp985/geWbhwISdPnsz3eqtWrUp1xlmWNm3aREBAQL7XDQwM+Pnnn4vc9/vvv+fPP//M97qFhQUTJkwoszkKIUQOOXMUohyQM8fyR5nWI2vJTyKrhBBCiBKS4iiEEELkIcVRCCGEyEOKoxKIjIzE2dkZKF0KCGSnW/j7+5f11Ipt5MiRxMfHF/r+q2vcsWMHe/fuLfWxNm3ahKWlJdbW1tja2uZ6eD86OhpPT0+geKkg4n/ypnJIIoeoyORuVSXzru9KLa3X3bH6qvPnz+fqW1sS3t7enDlzBh8fH2rWrElCQgJjxozhyZMnjB07lj///JNHjx6Vauz3Xd5UDknkEBWZFMd3KDIykjVr1pCVlcVff/1Fz549qVq1KocOHQJg3bp1XLlyhRUrVpCenk6dOnWYO3cuenp6/Pbbb8yfPx8tLS0aNGigGDMnBaRDhw4sXrw4X2JGVFQUy5YtIzk5mcTERFxdXbG0tHztXNPS0pg6daqi5+igQYNwdHTEzc0NFRUV/vjjD54/f84333xD3759efHiRYFJFykpKcyePZtz586hoaHBmDFj6N27NxYWFmzZsgVdXV2mTp1KfHw8Dx484LPPPmPRokWKeZw8eZIjR45w+vRpdHR0mDZtGocPH6ZKlSrcvXuX0aNHExISUuAaXr58yYYNG9i7d6+if2v16tXx8PCgf//+9OvXjxUrVpCUlMTq1asxNDTk9u3bODs7c+/ePTp16oSHh4fizyY0NJSMjAw+//xzXF1d+fvvv/nqq6/Q09NDS0uLTZs2lervhRDi3ZPi+I5dvHiRkJAQdHV16dy5M5MnT8bf358pU6bg6+vLwYMH2bJlC9WqVcPX15fFixczc+ZM3Nzc2Lx5M40aNWLatGn5xt2/f3+BiRm//vorHh4eNGrUiFOnTjFv3rxiFcfz58+TmJhIYGAgjx8/ZuHChTg6OgLZyRW+vr48evQIe3t7unTpwubNmwtMuggLCyMpKYnQ0FAePXrEsGHD6N69u+I4x44do3nz5qxYsYLU1FSsrKwUHYMAOnfujIWFBR06dKB79+4cPHiQ/fv3069fPwIDA7G1tc039xzXr1+nUqVKikbuORo3boympiYPHz5UpHV88803+Pv7ExcXR2BgIJUrV6Z79+5cv36duLg4Ll++zK5du1BRUcHV1ZWgoCDat29PbGws69evz3cMIUTFIsXxHfv444/58MMPAdDT06NTp05AdrLHkSNHFNmKkN0gu1q1aly7dg0DAwMaNWoEgJ2dHcuXL881bkGJGQCenp4cPXqU/fv3c/HiRV68eFGseTZp0oTY2FhGjBiBqakpkyZNUrxnb2+PhoYGtWrV4tNPP+XcuXOFJl2cOXMGR0dHVFVV0dfXz3eWZ21tTXR0NJs2beLmzZs8efKEpKSkQufl4OCAt7c3/fr1Y+/evWzevLnQbVVUVArtf/pqQ/dXffbZZ4pG5fXq1ePx48ecOnWK6OhoRcBycnIyRkZGtG/fnho1apSqMCprZFVFj0mq6PN/laylZKQ4vmMaGhq5/vvVZI/MzEw+/fRT1qxZA0BKSgovXrzg3r17udIrCkoDKSgxo3r16jg7O9OxY0c6duxIp06dchW5oujp6RESEkJERATHjx/Hzs5OUdjyzlldXb3QpIucYpnj9u3bih8OAHx8fAgLC8PR0ZHOnTvzxx9/UFSfCmNjYx48eMCBAweoU6cOhoaGhW7buHFj0tLSuHnzJg0bNlS8fv36dTIzM2nYsCHXrl3Ltc+rn2NOWklGRgZDhw5l+PDhADx9+hQ1NTUeP36cL8WjuJSlCUBeFfnBc3lwvnySJgCC1q1bc+HCBUVSx6pVq1i0aBFNmzbl0aNHxMTEABT4HVtBiRl//vknt27dYsKECZiZmREREVHsJInDhw8zadIkunbtiru7O5UrV1YkV4SGhpKVlcXff/9NdHQ07du3LzTpwtjYWLH9o0ePGDx4cK4IsIiICAYMGECfPn1QUVEhJiYmX7SWmpqaYt4qKir07dsXDw8PxZlcYSpVqsQ333zDtGnTFDfdPHr0iOnTp/PVV19RqVIl1NTUCj2LzGFiYqLIrkxPTy92mLMQouKQM8dyTF9fn3nz5jFx4kQyMzMxNDTE09MTDQ0Nli5diqurK+rq6rRo0SLfvgUlZrRu3Zr+/ftjZWVFlSpVaNu2LcnJyUVetsxhampKWFgYVlZWaGlp0aNHD5o2bQpkX1Z0cHAgNTWVOXPmoKenV2jSxaBBg/Dw8KBPnz4ATJ8+nSpV/vfT29ChQ5k1axa//PILH3zwAe3atePu3bvUq1dPsU3nzp1ZunQpVatWpVevXlhZWbFx48Zc310WZtSoUVStWpVhw4aRlZWFiooKTk5Oiiir1q1bs3LlShYvXpzr7PJVFhYWxMTE4OjoSEZGBl988QV2dnb8/fffrz2+MsubyiGJHKIik96q4o24ubnRoUOH1561vS2ZmZls376d2NhY3N3d38kcyoKyXFZVlkt3oFzrkbXkJ6kcotiSk5MZMGBAge+NHz+ebt26/cszej0XFxfi4uLYsGEDUDHXIIQof+TMUYhyQM4cyx9lWo+sJT+5IUcIIYQoISmOQgghRB5SHIUQQog83rsbcry9vQEYN25csbafNm0aTk5OfPLJJ8U+Rs7zfQMHDiz5BCugwtbr7+9PVFQUCxYsKHRfNzc3Tp8+TbVq1fLtW1BzA1F+5aRyQPZjHM+evnzHMxKi9N674lhSpUm5eF+KYo43Xe/48ePf2aMgouy8msohiRyiolOq4piens6sWbO4fv06Dx8+pEGDBqxcuZJff/0VPz8/9PT00NHRoXXr1gB06dIFc3Nzzp49i76+PoMGDcLHx4f79++zYMECOnTooEi5+Oijj5g0aRJJSUmoqqri7u5O27ZtWbhwIREREaipqdGtWzdcXFxynZ0ePXoULy8vMjMzqVu3LnPmzKFmzZpYWFjQp08ffvvtN16+fMnChQtp1apVoWsLDg5m/fr1qKmpUadOHTw9PdHS0mLNmjUEBQWhpqZGly5dcHV1VfRjPXLkCJD7bNnExISWLVvy8OFDdu3ahZeXV77kjtu3bzNr1iyePHmCtrY206dPL7DRQI5Xxw8MDGT16tVUqVKF2rVrU7ly5VL/eXp7e3Pv3j2uXbvGo0ePmDhxIqdPn+bixYs0a9aMZcuWoaKiUqyEjJ9//pmZM2dy7tw5DA0NUVFRYcyYMXTs2LHQ/V1cXGjSpAlXr16lRo0aLF++HF1dXYKDg1m9ejUqKip88sknzJkzh169erFhwwYaNGhAUlISlpaWHDhwAC0trVKvXwjx7ijVd47nz59HQ0ODHTt2cPDgQVJSUtiyZQu7d+8mICCAjRs3cv/+fcX2Dx8+pGvXruzfvx+AQ4cOsW3bNsaNG5evgfWuXbvo2rUr/v7+uLq6cu7cOf7++2/Cw8MJCgrC19eXW7dukZKSotjn0aNHzJgxg59++ong4GA+/fRT5syZo3hfV1eXXbt24eTkxNq1a4tcm5eXF7/88gv+/v40aNCAmzdvcvz4cY4cOYK/vz8BAQHcvn0bX1/fIsd5/Pgxo0aNYs+ePRw6dEiR3LFz5078/f35559/mDx5Mq6urgQEBDB37ly+/fbbYn3+8fHxLF68mK1bt7Jjx45iNzVfsWIFtra2il+zZ89WvPfHH3/g5+eHp6cnU6dOZeTIkezdu5crV65w7do1wsPDFQkZgYGBxMfHExQUBEBsbCyenp5s2rQJX19fXr58yf79+5k/f74iELqo/WNiYhg+fDh79+5FR0eH4OBg4uPjmT9/Pr/88gshISFkZGQQHh5O3759FfsdOHCArl27SmEUogJTqjNHY2NjdHV12bp1Kzdv3uTWrVt07NgRMzMzPvjgAwB69eqVq1enqakpALVr16Z9+/ZAdiLG06dPc43dqVMnxo0bx9WrVzEzM2Pw4MGoqamhpaWFk5MT5ubmTJw4Mdc/iNHR0bRu3VqR0jBgwADWrVuneP+LL74AshMvDhw4UOTazM3NGThwIN26daNnz540b96coKAgrKysFM2uHRwcCAwMxMzMrMix2rRpAxSc3PHixQsuX77MlClTFNsnJSXx+PFj9PT0ihz3/PnztGvXTpGVaGNjw+nTp4vcB4q+rNqlSxfU1dUxMjJCX1+fxo0bA2BoaEhiYmKxEzIiIiJwdHRERUWF2rVrK9JPXrd/zhlzkyZNSExM5Pz583z66afUqlULyE45AWjWrBnDhw9nwoQJBAQE8N1337123a9SxlQOZUiBUIY15JC1lIxSFcfDhw+zYsUKhgwZgr29PY8fP6Zy5cq5Cp26unquRteampqK3xd1A0j79u0JCQnh2LFj7Nu3T3EmunPnTqKioggPD8fJyQkfHx/FPnkbZmdlZeVqap1TSFVUVF67Nnd3d2JiYjh+/Diurq64uLjkGx+yLy3npEe8+tqr6RI5xbSg5I5q1arlirgCuH//viK2qSgqKiq55pR3/NJ4NbWkoPGKm5ChpqZW4OdV1P6v/qCT85nmnUNCQgIAderUwcjIiAMHDvDo0SPFDyDFpSxNAF5V0R86lwfnyydpAlAKp06dwtLSEgcHB2rWrMmZM2eA7ADdZ8+ekZKSwsGDB0s19qJFi9izZw92dnbMmDGDK1eucOXKFQYPHoyxsTGTJ0+mUaNGigQNyD5Du3jxInfv3gVgx44ddOzYscTHTk9Pp0ePHujp6TF69GhsbW25evUqJiYmhISEkJycTHp6Ort378bExAQdHR0SExNJSEggNTWVEydOFDhuQckdDx8+pH79+oriGBERoWjK/Trt27fn4sWLxMfHk5mZyb59+0q81pIqbkJG586d2bdvH1lZWcTHxxMVFYWKikqJEzY++eQTLl68yD///APAvHnzOHz4MJB95v5qU3UhRMWlVGeO/fv3Z9KkSezfvx9NTU3atm1LYmIiQ4cOpV+/fujo6GBkZFSqsZ2dnfn+++8JCAhATU2NmTNn0qJFC9q2bYu1tTWVKlWiefPmmJqaKpLra9asyZw5c3BxcSEtLQ0jI6NS3f2qrq7O+PHjGT58ONra2ujo6LBw4UIMDQ25evUqDg4OpKen88UXXzB48GDU1dUZMWIE/fr1o1atWoU+hlJQckeDBg3w9PRk1qxZrF+/Hg0NDcWNL69Ts2ZN3N3dGTZsGJUqVVJcAn2dFStW5PuOd8mSJcXat7gJGY6OjsTExGBjY4O+vj5GRkZoa2vToUOHEiVsGBoaMm3aNEaMGEFmZiZt27ZVXJLt0aMH06dPx9bWtsB9ld2rqRySyCEqOumtKt4Lx44dIysrC3Nzc549e0bfvn3ZvXt3sS4XF0dWVhbh4eFs375dEU5dEspyWVVZLt2Bcq1H1pKfpHJUIAsXLuTkyZP5Xm/VqlWpzjjL0qZNmwgICMj3uoGBAT///HOR+37//ff8+eef+V63sLBgwoQJZTbHojRq1IgffvgBLy8vIPsmoLIqjJB9efXo0aOv/SyEEBWDnDkKUQ7ImWP5o0zrkbXk917dkCOEEEKUBSmOQgghRB5SHIUQQog85IYc8U5FRkaycuXKXM0TSqughI+uXbsWu/1daY7XoUMHaZr+/0kqh1AmUhyFUpGEj3dHUjmEMpHiKEpkyZIlhIWFoaenh76+PhYWFqiqqrJ582YyMzNp2bIlM2fOREtLi88//5yePXty7tw51NTU8PLyom7duvz222/Mnz8fLS0tGjRooBi7sDQQNzc3njx5wu3bt3F1dcXCwqLE8w4MDCxwjsVJZomKimLZsmUkJyeTmJiIq6srlpaWxRpfCFExyXeOotiOHDnCuXPn2Lt3L+vWrePKlSu8fPkSPz8/fH192bNnDzVq1GDDhg0A/PPPP3Tq1InAwECMjY3ZunUrqampuLm5sWLFCvz9/XP1Py0qDURXV5fQ0NDXFsa8CR/Pnz/n+vXrhc6xOMksv/76Kx4eHgQEBPDjjz+yatWqXMcsanwhRMUkZ46i2E6ePJkrxaN79+5kZWVx+/ZtHB0dAUhLS8uV/fhq8sjZs2e5du0aBgYGNGrUCAA7OzuWL19eZBoIoMjgfJ2CLqtGRkYWOcfXJbN4enpy9OhR9u/fz8WLF/NFcb1u/OKQVI7ySRnWkEPWUjJSHEWxqaqq5ku2yMjIwNLSEnd3dwBevHhBRkaG4v1Xk0eysrLyJXfkJKFkZmYWmQby6hlmSb1ujq9LZhk0aBAdO3akY8eOdOrUiUmTJpVo/OJQliYAr6roD53Lg/PlkzQBEOVOly5dOHDgAKmpqTx//lyRdnLw4EEePXpEVlYWs2bNytdE/FVNmzbl0aNHxMTEABASEgJA1apVS50G8jodO3Ys0Rxf9eTJE27dusWECRMwMzMjIiIiX+F7k/GFEOWTnDmKYjMzM+P333/Hzs6OatWqYWBgQMOGDXFxcWHo0KFkZmbSvHlzRo0aVegYGhoaLF26FFdXV9TV1XNdfixtGsjrNGvWrERzfJWuri79+/fHysqKKlWq0LZtW5KTk0lKSiqT8ZWJpHIIZSK9VUWxnT9/nlu3bmFnZ0daWhoDBgxg3rx5NGvW7F1PrcJTlsuqynLpDpRrPbKW/CSVQ5SZBg0asHLlSjZu3EhWVhZ9+/b91wtjeUj4EEIoPzlzFKIckDPH8keZ1iNryU9uyBFCCCFKSIqjEEIIkYcURyGEECIPKY6i2O7evVuqvqYV0cqVK7GyssLKyopFixYpXj958iQ2Njb06NGDZcuWKV6/evUqDg4O9OzZk2nTppGeLo8yCFGRSXEU7y0fHx+uXr2a7/WTJ0/y22+/ERAQQGBgIP/97385ePAgycnJTJ06lVWrVrFv3z4uX77M8ePHAXB1dWX69OmEhYWRlZWFn5/fv72cdy4nsqqqTqV3PRUh3pgUxwokMjKS4cOHM2zYMCwsLFi4cCGrVq3C3t4ee3t7Hj58yK+//kr//v2xtrbGxsaGGzduEBcXR6dOnbhx4wapqanY2Nhw7NixIo+1ZMkSevTowYABA3BxccHf3z/X+25ubrlea9q0aZHjxcTE4OjoiL29PQMHDuTWrVsAhIeH069fP/r27YuLi4uil6qFhQUTJ06kZ8+eREdH5zpj9fb2xtvbG8ju2uPu7k6vXr1wdnYmNDSUQYMGYWFhQVRUVJFzMjIyYubMmTg7O3Po0CFFWzt9fX3c3NzQ1NREQ0ODRo0ace/ePaKjo/noo4+oW7cu6urq2NjYsH//fv7++2+Sk5Np27YtAPb29opG5u+TnMgqbS15QkxUfFIcK5iLFy8yf/58QkJC8PX1pXr16vj7+9O0aVNCQkI4dOgQPj4+7N27l+7du7Nt2zY+/PBDJk2axKxZs/jpp59o164dXbt2LfQYBaVvvKnNmzczfPhw/P39cXZ25sKFCyQkJLBkyRI2bNhAYGAgn3/+OYsXL1bsY2pqSlhYGNWrVy903OKkahSmW7du+Pn54erqSlhYGNbW1kRFRdGkSRNFobt16xb79u3DzMyMBw8eoK+vr9jfwMCA+Pj4fK/r6+sTHx9fmo9JCFFOyI94FczHH3/Mhx9+CICenh6dOnUC/pcisWTJEkJCQrh16xYnTpygefPmADg4OBAaGkpwcDB79+4t8hgFpW+8KTMzM+bMmcOJEycwNzenZ8+ehIeHExcXx5AhQ4Ds5uPVqlVT7NOmTZtijf26VI3XUVVVzfUrx/Xr1xk9ejSTJ0+mfv36XLp0Kd++OQ3VC3q9JJQtlUNZEiCUZR0gaykpKY4VjIaGRq7/fjVFIi4ujgEDBjB48GBMTU2pWbOm4ju1lJQU7t+/T0ZGBvfv36dhw4aFHqOg9I28Xi0KaWlpr513r169aNeuHUePHmXz5s0cP36crl278umnn7JmzRrFHF+Ng8qb6JEjPT0ddfX//dV9XapGYY4dO8batWtRV1dn6NChzJ8/X1Ecz507x/jx45k6dSpWVlYAGBoa8vDhQ8X+Dx48wMDAIN/r//zzDwYGBsWeByhPE4AcyvDAuTw4Xz5JEwBRYpcuXeKjjz5i2LBhtGnThvDwcEWChJeXFyYmJkyZMoWpU6cWWfwKSt/Ieyakq6uraON26NCh185t4sSJREdH4+TkxIQJE7hy5Qpt2rThwoULxMbGArBq1apcd4bm0NHRITExkYSEBFJTUzlx4kSxP5Oi3Lp1C3d3d3x8fOjevbuiMMbFxTF27FgWL16sKIyQfSYbGxvL7du3ycjIYO/evZiamlK7dm20tLQ4d+4cAIGBgYqzWSFExSRnjkrk888/JyYmht69e6OpqUnr1q25fv0658+fJywsjKCgIKpUqUJAQAAbNmxg5MiRBY5TUPpGzllcjkGDBjFx4kRsbGwwMTHJ9Z1bQb7++mumTZvGqlWrUFNTw83NDX19febNm8fEiRPJzMzE0NAQT0/PfPtWrVqVESNG0K9fP2rVqsUnn3xS+g/pFcOGDSvw9Q0bNpCSksKCBQsUrzk5OTFw4EAWLFjAuHHjSElJwczMjF69egGwePFi3N3defHiBS1atFBcKhZCVEzSW1XkI+kb/z5luKxaTbcymhpqJKek8+zpy3c9nTcmlyLLJ0nlEG/V2bNnmTt3boHvrVu3rlTpG/v27WPt2rUFvpcTYvxvW7hwISdPnsz3eqtWrfjxxx/fwYyUl6aGmtL8AyyEnDkKUQ4ow5mjMp2dgHKtR9aSn9yQI4QQQpSQFEchhBAiDymOQgghRB5SHJVMZGQkzs7OZTJW3v6pkLuvaWGcnZ2JjIwskzmU1qpVqzh27FiBa3gTryaTLF++nMOHD+fbpjifkRCifJPiKJRSZGQkHTt2fKvHmDBhAt26dXurxxBCvBvyKMc7tmTJEsLCwtDT00NfXx8LCwtUVVXZvHkzmZmZtGzZkpkzZ6KlpcXnn39Oz549OXfuHGpqanh5eVG3bl1+++035s+fj5aWFg0aNFCMffv2bWbNmsWTJ0/Q1tZm+vTptGjRAjc3N548ecLt27dxdXUtdUajhYUFffr04bfffuPly5csXLiQVq1aKd5/9OgRQ4cOZeLEiVStWpW1a9eira3NjRs3aNq0KYsXL0ZTU5Pdu3ezceNGVFRUaNmyJdOnT2fp0qU0atSIQYMG4efnx8aNGwkNDSUtLY3u3btz6NAhRY/WvJ9HYmIimpqaVKpU6bXzVVdXZ9KkSYp+s0ePHmXHjh2sXLmSWbNmcf36dR4+fEiDBg1YuXJlrvHc3Nzo0KED9vb2rF+/Hj8/P/T09NDR0aF169al+kyFEOWDnDm+QwWlX7x8+RI/Pz98fX3Zs2cPNWrUYMOGDUB2z85OnToRGBiIsbExW7duJTU1FTc3N1as+H/s3XdYFNf///0nHRuw9oAmUWOIMUGNgqIGBI1KERUsWBD9aTQxWCOCgohKbNghtliiRMUGKCJii2JQwV4+9lgiFiwoKkif+w9u5svCgmAJJedxXV5X2J05c84s2cO092sJISEh6Orqyu17eHjg7u5OaGgoM2bMYNy4cfJ7BgYGREZGvnN4sYGBAdu2bcPZ2VnpGceXL18yfPhw3Nzc5MLlZ86cwcfHh8jISO7fv89ff/3F1atXWb58OUFBQYSHh1OpUiUCAwOxtLTk+PHjABw7doykpCSePHnCqVOnaN68OVpaWir3B0BMTAzt2rUrVn+/+OIL1NXVuXbtGgC7du3CwcGBM2fOoKWlxebNm9m3bx9paWlydmN+Fy5cYPv27YSGhrJ27VoePnz4TvtUEITSJ44cS5Gq9AtJkrhz5w59+vQBcop6f/nll/I63377LQCNGzfm5MmTXL16ldq1a9OoUSMAevbsyeLFi0lOTubixYtMmjRJXjclJUXOSyzOkY2qZAlJkpSSK/L2Z+/evfLrU6dOpWbNmnTu3Fl+rXHjxtStWxeARo0akZSUxP3797GyskKhUADQt29fJk2axNixY/Hx8SErK4ubN29ia2vLiRMnuHDhAlZWVoXuD8jJiBw+fLjKManqb/fu3YmIiKB+/frExcUxc+ZMdHR0MDAwYMOGDdy8eZPbt2+TkpKiss24uDgsLS2pUqUKkFNk/U2F2/OrKKkcFSn5ASrWeMRYSkZMjqVIVfpFVlYWNjY2eHt7A5CcnCwXD4eCSRVqampKbeSmUmRnZ6Otra1Umebhw4cYGBgAKB1hFkZfX79A7NPTp0/57LPPVPYnr++//57Dhw+zadMmBgwYoLRs3v7nH78kSWRmZqKjo8MXX3xBeHg4DRs2pHXr1hw7doxTp04xbNiwQveHJEncvn270NQRVf21t7fH1dWVL774gvbt26Ojo8OBAwdYsmQJgwYNwtHRkWfPnqmMpsptK+84NDU1SU9PV7lsYUQRgLKnIo1HjKUgUQSgDFOVfvHy5Uv27dvH06dPkSQJX1/fIkN7jY2Nefr0KVeuXAEgIiICyCnW/emnn8qTY0xMjDxJFZe5uTm7d++Wj5geP37MoUOHaNOmzRvXbdKkCVOnTiUwMLDI4F8zMzMOHjzI8+fPAdiyZYt8I42lpSW//vorZmZmmJmZceDAASpVqlRk+PGlS5fkDMviqlOnDh999BErV67EwcEByDmVa2Njg5OTEzVr1uTEiRNKf6TkZW5uLn92aWlp7Nu3r0TbFwSh7BFHjqVIVfpFw4YNcXNzw9XVlezsbJo0aVLoKULIyXdcsGAB7u7uaGpqKp2C9ff3x9fXl1WrVqGlpcXChQtLFMJraWnJlStX6NOnD2pqaqirq+Pu7k7jxo2Ltf6nn37KgAEDmD59eqEpFV988QUjRozAxcWFjIwMmjZtyrRp0wDo0KEDvr6+mJmZoa+vT40aNejQoUOR24yOjn6ruKju3buzcOFCeWLu3bs3EyZMYM+ePWhra9O8eXPi4+NVrtukSRNcXV3p1asXenp6GBoalnj7giCULaK2aikS6RdCLnFateypSOMRYylIpHKUYbmPB5Q0/eJ9+vnnn+XQ4rysra0ZM2bMv9oXQRCEskIcOQpCGSCOHMueijQeMZaCxA05giAIglBCYnIUBEEQhHzE5CgIgiAI+YjJURAEQRDyEZOjUK6870iuL774okCRgpEjR76x5mxISAienp7vpR/lXTW9StSqVY30DNVFEgShPBKTo/CfVqdOHaWasK9eveLSpUul2KPyR1dHk24/70BbS6O0uyII7414zlH4oMp6JFfnzp2JioqSj0b3799Phw4diI6OBiAhIYHJkyfz8uVLHj9+jJ2dHRMmTFBq4/z588yaNYvU1FQUCgXTpk2jfv36H2BvCoLwr5EE4QM5cOCA1K9fPyktLU16/vy5ZGVlJf3xxx9Sv379pNTUVEmSJGnevHnSr7/+KkmSJH3++efSvn37JEmSpFmzZkmzZs2S0tLSpHbt2kk3btyQJEmSJk+eLA0cOFCSJEnq27ev9L///U+SJEm6fv261LlzZ0mSJMnDw0Py8PB4Y/88PDyk7du3S127dpUeP34sSZIkDR8+XDp+/LhkZWUlSZIkrVq1SgoJCZEkSZJevHghtWjRQnr69Km0fft2ycPDQ0pLS5O6desm3bt3T5IkSYqOjpZcXV3fed+VN/bjw0q7C4LwXokjR+GDKeuRXLk6d+7M3r17sbOz49WrVxgZGcnvDR06lOPHj7N69WquX79ORkYGr1+/lt+/ffs2d+/e5ccff5Rfe/XqVUl2E1C+iwDkjQ+qKA+ag3hwvqwS5eOEcq+sR3LlsrGxYdasWWhra/Pdd98pvTd79mzu3r2Lvb09nTp14ujRo0rRVdnZ2dSrV0/uR1ZWFk+ePCn2tgVBKJvEDTnCB1PWI7lyffHFFzx58oStW7fStWtXpfdiYmIYOnQoNjY2PHjwgISEBKXJumHDhiQlJclBy9u3by9wTVIQhPJHHDkKH0xZj+TK67vvviMuLo66desqRVONGDGCiRMnoqenR40aNfjqq6+U3tfW1mbx4sX88ssvpKWlUbVqVebMmfNWfSivUtMyCZ/fXTzKIVQoovC48MGISK7iK8/XHHNVpOtaULHGI8ZSkLjmKJQaEcklCEJ5JY4cBaEMEEeOZU9FGo8YS0EiskoQBEEQSkhMjoIgCIKQj5gcBUEQBCGfcnlDTmxsLIGBgQQFBeHl5YWzszNff/11idrw9PTEzMwMR0fHD9TLon3//ff4+flRp04dle/nHePmzZupUqUK9vb2Jd6OtbU1urq6aGlpya99+eWXzJo16637XtY4OzvTpEkTTp8+TUZGBv/8849cUWfQoEE4OTmVcg8rrmp6ldDVyfkaEY9yCBVJuZwc8/rll19Kuwtv5bfffiv2smfOnMHMzOytt7Vy5Urq1av31uuXZbdv3+bjjz9m6tSpAMTHxzNo0CClyjnCh5ObyAEQPr97KfdGEN6ff3VyjI2NZfny5UiSxD///EOXLl2oVq0a+/fvB3K+xC9dusSSJUvIzMykXr16zJgxA4VCUWgyg4uLC25ubpiZmTFv3jz279+PhoYGffv2xdXVlbi4OBYuXEhqaipJSUm4u7tjY2Pzxr5mZGQwefJkrl+/DkD//v3p06cPnp6eqKmpce3aNV69esWPP/5Ijx49SE5OZvr06Vy/fp2srCy+//577O3tSUtLY9q0aZw6dQotLS1GjhyJra0t1tbWrF+/HgMDAyZPnkxCQgKPHj2iVatWzJ07V+7H0aNHOXjwIMePH0dPTw8vLy8OHDhA1apViY+PZ8SIEXLVmJJycXGhSZMmHDt2jNTUVLy9vQkKCuLGjRsMHjyYwYMHFzqukJAQQkNDef78OVZWVvTv358JEyaQlJTE559/zokTJ4iOji5y/SNHjpCUlMTdu3dp164dvr6+SJJU4HPs0KEDrq6uHDx4EHV1deLi4li5ciWrVq0iOjoaCwuLIsdpbW2NiYkJly9fZuPGjaxfv55jx46RlJSEQqEgICCAWrVqYWxszNWrV4GcvMa4uDhmz56NtbU1NjY2HDp0CA0NDcaPH8+aNWu4c+cOHh4e2Nracu3aNWbMmEFKSgqJiYkMGTKEQYMGvdXnIghC6fvXjxzPnTtHREQEBgYGtG3bFg8PD0JCQpg0aRLBwcHs27eP9evXo6+vT3BwMPPmzWPq1Kl4enqybt06GjVqhJeXV4F29+zZw+nTpwkPDycjI4P+/ftja2vLH3/8gZ+fH40aNeLYsWPMnDmzWJPjmTNnSEpKIiwsjGfPnjFnzhy5WHZCQgLBwcE8ffoUR0dH2rVrx7p162jatClz5szh1atXODs706xZM6KiokhJSSEyMpKnT58yePBgOnXqJG/n0KFDNGnShCVLlpCeno6dnR3/+9//5Pfbtm2LtbU1ZmZmdOrUiX379rFnzx569epFWFgY3bu/+a/14cOHK51WzX+qMTw8nMDAQPz8/Ni5cyeJiYn06NGDwYMHs2zZMpXjyt0Pu3fvRlNTk1GjRmFjY8OAAQPYt28fu3btAihy/TNnzrBr1y40NDTo2rUr/fr14+bNmyo/x3r16hEbG4u5uTmhoaHy6fCYmBhmz579xn1gYWHBokWLuHPnDjdv3iQ4OBh1dXUmTpxIeHg4/+///b8i169duzYRERFMmjSJlStXsn79ek6fPs3MmTOxtbVl69atjBw5EnNzc+7evYuDg4OYHAWhHPvXJ8fPP/+cjz76CACFQoG5uTkAhoaGHDx4kAcPHshfKtnZ2ejr6xeazJDXiRMnlBIgck+r+fv78+eff7Jnzx7OnTtHcnJysfrZuHFjbt26xdChQ7GwsFCql+no6IiWlhZ169blm2++4dSpUxw9epTU1FS2b98O5CREXL9+nRMnTtCnTx/U1dWpVatWgaM8e3t7zp8/z++//87Nmzd5/vw5KSkphfbLycmJgIAAevXqxa5du4qsS5qrqNOquUddhoaGNGvWjEqVKmFkZMSLFy8ACh0X5Fy71NTM+RWKiYmRr2N+99136OnpvXH9Fi1aULVqznNG9evXJykpqdDP0cnJiZ07d9K8eXOOHz/OtGnTSE1N5fXr1ygUijfug9wJ+ZNPPsHDw4OtW7dy69Ytzp49y8cff/zG9fPup9q1a6OpqYmhoaG8nzw9PTly5AgrVqzg6tWrRX6GqhT1vFV5kjehoyKoSOMRYymZf31yzHsEA/+XsgA5k+E333zD8uXLAUhLSyM5OZn79++rTGbIK/dLOld8fDzVq1fHxcWF1q1b07p1a8zNzYtdFFqhUBAREUFMTAyHDx+mZ8+e8sSWv8+amppkZ2fj7+9P06ZNAXjy5An6+vrypJDrzp078h8HAEFBQURFRdGnTx/atm3LtWvXKKoug6mpKY8ePWLv3r3Uq1ev0Bt6iivv55F/H+aOT9W4wsPDlZIvNDQ0VPa7qPVzEzjg/1I4Cvscu3btysKFC4mKisLCwgJtbW0OHz5c7Guxudu6ePEiP//8M4MHD6ZLly6oq6sr9Ts3CSQzM7NE+2ns2LHo6elhZWWFra1tiU91l9ciAPm/pCrKg+YgHpwvq/6TRQBMTEw4e/Yst27dAmDp0qXMnTu30GSGvExNTdm3b5+ctzds2DBu3LjB7du3GTNmDJaWlsTExCjFIxXlwIEDTJgwgQ4dOuDt7U3lypV58OABAJGRkUiSxL179zh//jwtW7akTZs2bNq0CYBHjx7h4ODAgwcPMDU1lZd/+vQpAwcOJD09Xd5OTEwMffv2xcHBATU1Na5cuVIg5klDQ0Put5qaGj169MDPz+9fudO2sHHl17ZtW8LDwwE4fPiwfERV3PVzqfocExISqFSpEhYWFixYsEAe95EjR954vTG/EydOYGZmRr9+/fjss8+UficUCgXXr19HkiQOHjxYonZjYmIYPXo0nTp14sSJEwDF/l0TBKHsKVN3q9aqVYuZM2cyduxYsrOzqVOnDv7+/kUmM+T67rvvuHjxIo6OjmRnZzNo0CBMTEzo3bs3dnZ2VK1alebNm5OamlqsU14WFhZERUVhZ2eHjo4OnTt3xtjYGIDU1FScnJxIT09n+vTpKBQK3Nzc8PX1xd7enqysLNzd3fn444/p378/fn5+ODg4ADBlyhT5VCKAq6srvr6+rFmzhipVqtCiRQvi4+OVTvW1bduWBQsWUK1aNbp27YqdnR1r165VunZZlPzXHCtVqkRwcHCx1i1sXLkRTbkmT56Mh4cHW7Zs4YsvvpBPqxZ3/VyqPsfcG7Ds7Ow4ffq0fIr04sWLKq8/F8XW1hY3Nze6deuGlpYWxsbGcsrGzz//zA8//EDNmjVp2bKlHJxcHKNGjaJ///7o6enRoEEDjIyMiI+P55NPPilR/8qb3EQOEI9yCBWLqK1aQqX9fGR2djabNm3i1q1bcmBwWbB+/Xratm3LZ599xv/+9z+mTJlCSEjIe2s/KyuLhQsXUqNGDYYMGfLe2i0ryutp1bwq0qk7qFjjEWMpSKRyFCE1NZW+ffuqfG/06NF07NjxX+7Rm7m5ufHgwQNWr14NlJ0xfPLJJ4wfPx51dXV0dHSYMWPGe23fyckJhULBsmXL3mu7giAIqogjR0EoA8SRY9lTkcYjxlJQubohRxAEQRDKAjE5CoIgCEI+YnIUBEEQhHzE5Ci8UXx8PNbW1qXdjfcmICCAzp07k5qaKr8WGxuLi4tLKfaq/KmmV4latarJ/8SjHEJFIiZH4T/p/v37LFiwoLS7Ua7lJnLk/tPWKli5ShDKq//0oxzlRXHSTPbs2cOOHTt4/fo1ampqLFq0iMqVK+Po6Mgff/xB/fr1cXJy4ueff6ZDhw6Fbmv+/PlERUWhUCioVauWXPQ8V/7nPPMmWahy5coVfHx8yMzMREdHh1mzZvHpp58SHR2tMn0lb4KGv78/Y8eOlavVBAQEADkP3Ldr1w4rKytOnjxJrVq16N+/P0FBQTx8+JDZs2e/saycs7Mzu3fvpnPnzrRq1UrpvSdPnuDl5cX9+/fR1NRk3LhxWFhYEBAQQEJCAnfu3OHevXv07t2bH3/8kaysLObOnUtcXBxZWVk4OjoyePDgIrcvCELZJo4cy4lz584xa9YsIiIiCA4Opnr16oSEhGBsbExERAT79+8nKCiIXbt20alTJzZu3MhHH33EhAkT8PX15ddff6VFixZFTowHDx7k1KlT7Nq1S44Pe1fr1q1jyJAhhISE4OLiwtmzZ0lMTGT+/PmsXr2asLAw2rdvz7x58+R1cqsTVa9evdB2nzx5QocOHdizZw8A+/fvZ+PGjYwaNapYxdj19fXx9fXFy8tL6fQqwIwZM2jTpg3h4eEsWbKEyZMn8+TJEwCuXr3K6tWr2bp1KytXruTFixds2bIFgNDQULZt28aBAwcKrQAkCEL5II4cy4mi0kxevHjB/PnziYiI4Pbt2xw5coQmTZoAOQ/PR0ZGEh4eLsdIFebo0aNKiRjFLU9XFEtLS6ZPn86RI0ewsrKiS5cuREdHq0xfyZVbHu5NcuuqGhkZ0bJlSwClpIw36dSpE5GRkSxYsECpWMLx48fx8/MDctJCmjVrxrlz5wBo3bo12tra1KhRAwMDA16+fMmxY8e4fPkyx48fB3KSR65evVrgiLQoIpWjbKpI4xFjKRkxOZYTRaWZPHjwgL59+zJw4EAsLCyoWbMmly9fBnKSTR4+fEhWVhYPHz6kYcOGhW5DXV29QNHz/HLTMyAnEPpNunbtSosWLfjzzz9Zt24dhw8fpkOHDirTV3LlJmjk3RZAZmamUiKGtra2yv1RElOmTMHe3h4DAwP5tfx1MSRJkouIq0oSya0Z27lzZwASExOpXLlyifpRHosAqPqCqigPmoN4cL6sEkUAhGK7cOECn3zyCYMHD6ZZs2ZER0fLX+aLFi2iTZs2TJo0icmTJxc5+bVr1469e/eSnp7Oq1evOHToEGpqakrLGBgYcOPGDQD5mmdRxo4dy/nz53F2dmbMmDFcunSJZs2aqUxfyU9PT4+kpCQSExNJT0/nyJEjxd4nxWVgYICvry9Lly6VX2vTpg3btm0D4O7du5w+fZrmzZsX2kabNm3YsmULGRkZJCcn079/f/lIUxCE8kkcOVYA7du358qVK9ja2qKtrY2JiQnXr1/nzJkzREVFsXPnTqpWrUpoaCirV6/m+++/V9mOpaUlp0+fpmfPnujr61O7dm2lIyWA/v37M3bsWLp160abNm2oVatWkX374Ycf8PLyYunSpWhoaODp6Vlo+kp+1apVY+jQofTq1Yu6devy9ddfv/1OKkKnTp3o0qULjx49AsDLywsfHx+5cLqfnx+1a9cudH1nZ2fu3LlDz549yczMxNHRkdatW3+QvpYleRM5QKRyCBWLqK0qyM6cOcPt27fp2bMnGRkZ9O3bl5kzZ/LFF1+UdtcqvPJ4WjW/inTqDirWeMRYChKpHIKSkydPFpqYsXLlSgIDA1m7di2SJNGjR49iTYy7d+9mxYoVKt/bsWPHO/X3bc2ZM4ejR48WeP2rr77il19+KYUeCYJQnogjR0EoA8SRY9lTkcYjxlKQuCFHEARBEEpITI6CIAiCkI+YHAVBEAQhHzE5CiXi6ekpP+JQGGNj43+pN+8uJCQET0/P0u5GuZI/jUOkcggVkbhbVRCEEslN48gv7zOPglDeicmxAoiNjWXFihXo6ury999/Y2xszLhx4xg6dOgHSbTItXDhQo4dO0ZSUhIKhYKAgAC5KMCUKVM4f/48CoWCmTNnYmhoWGg7np6eqKmpce3aNV69esWPP/5Ijx49CAgI4OzZszx48IABAwbQtm1bfHx8eP78OZUrV8bLywsTExPu3bvHpEmTSExMRFdXFz8/P7744gvCwsJYt24d2dnZNG3alKlTp6Kjo0NYWBjLli2jatWqGBkZyaXerK2tWb9+PfXq1SM2NpbAwECCgoJwcXFBX1+f69evs2jRIh4/fqwyUWTOnDnExMSgoaFBx44dcXNze5ePVRCEUiROq1YQZ86cwcfHh8jISO7fv89ff/1V6LLvmmgBcOfOHW7evElwcDBRUVF8/PHHhIeHy++bmpqyY8cOvvvuu2I9V5iQkEBwcDDr1q1j7ty5PH78GID09HR2797NgAEDcHd3x8XFhfDwcCZNmsSYMWNIT09n2rRpdOnShV27djFq1CiWLVvG9evX2bJlC8HBwezYsYMaNWqwevVqEhISmDdvHhs2bGDz5s1KNV2LYmxsTFRUFHXq1FGZKHLv3j2io6PZuXMnwcHB3L59m7S0tGK1LQhC2SOOHCuIxo0bU7duXQAaNWpEUlJSkcu/a6LFJ598goeHB1u3buXWrVucPXuWjz/+GABdXV0cHBwA6N69O4sWLXpje46OjmhpaVG3bl2++eYbTp06BYCJiQkAycnJ/PPPP3Jx7+bNm6Ovr8/Nmzc5ceKEHFxsaWmJpaUlf/zxB3fu3KFPnz5ATpH0L7/8kjNnztCiRQtq1qwJQLdu3eQ0jaLk9uPcuXMqE0Xq1KmDjo4Ozs7OWFlZMXbs2AKl94oiUjnKpoo0HjGWkhGTYwWRPy0C+KCJFhcvXuTnn39m8ODBdOnSBXV1dXl76ur/d0JCkiSl7RYmbx+ys7PldXR1deV2CkvLyNu+JEn8/fffZGVlYWNjg7e3N5AzuWZlZXHs2DGl4uv5+5a7jczMTKXXc/uRlZWlMlFEU1OTrVu3EhcXR3R0NM7OzgQFBdGgQYM3jh3KVxGAor6YKsqD5iAenC+rRBEA4Z1Uq1btgyZanDhxAjMzM/r168dnn31GTEyMnASSkpLCgQMHANi+fTtt27Z9Y3uRkZFIksS9e/c4f/68fDSbq2rVqtSvX5+9e/cCcPbsWZ48eULjxo1p1aoVERERQE4m5ZQpU2jdujX79u3j6dOnSJKEr68v69ato2XLlpw7d46EhASys7PZvXu3vA2FQiEnjuT2P7/CEkUuXbrEwIEDMTU1xcPDg0aNGsnLCIJQ/ogjxwrqQyda2Nra4ubmRrdu3dDS0sLY2Jj4+HggJ2pq//79LF68mDp16jBr1qw3tpeamoqTkxPp6elMnz4dhUJRYBl/f398fX0JCAhAS0uLgIAAtLW18fHxwdvbm40bN1KpUiX8/Pz47LPPcHNzw9XVlezsbJo0acLw4cPR0dHB29ubwYMHU6lSJT777DO5/dGjRzNjxgwCAwNp3769yn4WliiiUCho3rw59vb2VKpUiSZNmsinriua/GkcucSjHEJFImqrCqXO09MTMzMzHB0dS7srpaY8nVYtTEU6dQcVazxiLAWJVA6hxD5EokVRbQqCIJQ14shREMoAceRY9lSk8YixFCRuyBEEQRCEEhKToyAIgiDkIyZHQRAEQchHTI7CG8XHx2NtbV3a3XhvAgIC6Ny5M6mpqfJrsbGxuLi4lGKvBEEoS8TkKPwn3b9/Xy45JxRfYXFVIrJKqGjEoxzlQGxsLMuXL0eSJP755x+6dOlCtWrV2L9/PwArV65kz5497Nixg9evX6OmpsaiRYuoXLkyjo6O/PHHH9SvXx8nJyd+/vlnOnToUOi25s+fT1RUFAqFglq1amFtba2U0pH/mURjY2OuXr1aaHtXrlzBx8eHzMxMdHR0mDVrFp9++inR0dEqky2sra0xMTHh8uXL+Pv7M3bs2A+SLOLs7Mzu3bvp3LkzrVq1UnrvyZMneHl5cf/+fTQ1NRk3bhwWFhYFUkL27NlDkyZNOHbsGKmpqXh7exMUFMSNGzcYPHgwgwcPLrIP5VFhcVUgIquEikUcOZYT586dY9asWURERBAcHEz16tUJCQnB2NiYiIgI9u/fT1BQELt27aJTp05s3LiRjz76iAkTJuDr68uvv/5KixYtipwYDx48yKlTp9i1axcrV67k0qVL79zvdevWMWTIEEJCQnBxceHs2bMkJiaqTLbIZWFhQVRUFNWrVy+03XdNFtHX18fX1xcvLy+l06sAM2bMoE2bNoSHh7NkyRImT57MkydPAOWUkFzh4eF0794dPz8/AgIC2LBhA7/++muJ9pMgCGWLOHIsJz7//HM++ugjIKcGqLm5OfB/SRrz588nIiKC27dvc+TIEZo0aQKAk5MTkZGRhIeHs2vXriK3cfToUWxsbNDW1kZbW5tOnTq9c78tLS2ZPn06R44cwcrKii5duhAdHa0y2SJXs2bNitX2uyaLdOrUicjISBYsWEDHjh3l148fP46fnx8A9evXp1mzZpw7dw74v3SO/H0wNDSkWbNmVKpUCSMjo2L3IZdI5SibKtJ4xFhKRkyO5YSWlpbSz3lTLB48eEDfvn0ZOHAgFhYW1KxZk8uXLwM5qREPHz4kKyuLhw8f0rBhw0K3oa6urpRYoYqampqcXJGRkfHGfnft2pUWLVrw559/sm7dOg4fPkyHDh1UJlvkyk0YybsteP/JIpATymxvb4+BgYH8WmHpH/B/6Ry58n4uxUkfKUx5KQLwpi+livKgOYgH58sqUQRAKLYLFy7wySefMHjwYJo1a0Z0dLT8Zb5o0SLatGnDpEmTmDx5cpGTX7t27di7dy/p6em8evWKQ4cOyfFXuQwMDOTkitxrnkUZO3Ys58+fx9nZmTFjxnDp0qVCky3y09PT+6DJIpAzHl9fX5YuXSq/1qZNG7Zt2wbA3bt3OX36NM2bN3/v2xYEoewSR44VQPv27bly5Qq2trZoa2tjYmLC9evXOXPmDFFRUezcuZOqVasSGhrK6tWr+f7771W2Y2lpyenTp+nZsyf6+vrUrl27QGBv//79GTt2LN26daNNmzbUqlWryL798MMPeHl5sXTpUjQ0NPD09Cw02SK/D50skqtTp0506dKFR48eAeDl5YWPjw8hISEA+Pn5Ubt27Q+ybUEQyiZRW1WQnTlzhtu3b9OzZ08yMjLo27cvM2fO5IsvvijtrlV45eW0ajW9SujqqP6bOj0ji6TnKf9yjz4ccSqybBKpHMIHcfLkSWbMmKHyvZUrVxIYGMjatWuRJIkePXoUa2LcvXs3K1asUPnejh2qb/v/0D5EsogAL1+8prCvpYp0w4cgiCNHQSgDysuRY1Eq0tEJVKzxiLEUJG7IEQRBEIQSEpOjIAiCIOQjJkdBEARByEdMjoIgCIKQz782OeaNBPLy8uLChQslbsPT01N+9qw0fP/99yQkJBT6ft4xbt68+Y3l2gpjbW2Nra0t3bt3l//t27fvrdrKq3v3nMLQAQEBchHvwjx8+JDRo0cDkJiYiJeXF9999x02Njb07NmTAwcOvHN/3lbe3wNjY2O6d+9Ojx496NatG87OzkUWQi+uu3fvMnnyZCCnyIKXl9c7t1kRiFQO4b+iVB7lKK+30v/222/FXvbMmTNvTIYoysqVK6lXr95br69KSR6riI6OxsLCgvT0dFxdXenSpQt79uxBQ0ODmzdvMnToUIyMjMrEM5B5xxUcHIyHhwdhYWHv1Ob9+/e5e/cuAF9//fUHK0BQ3ohUDuG/4o2TY3Hiki5duqQyfuivv/5i1qxZ6Ojo0KBBA7lNFxcX3NzcMDMzY968eezfvx8NDQ369u2Lq6srcXFxLFy4kNTUVJKSknB3d8fGxuaNg8nIyGDy5Mlcv34dyKnm0qdPHzw9PVFTU+PatWu8evWKH3/8kR49epCcnMz06dO5fv06WVlZfP/999jb25OWlsa0adM4deoUWlpajBw5EltbW6ytrVm/fj0GBgZMnjyZhIQEHj16RKtWrZTKnx09epSDBw9y/Phx9PT08PLy4sCBA1StWpX4+HhGjBhBREREiT+sV69eqdxuXFzcGz+jmjVrFoiX2rp1K8ePH2f+/PkABAYGoq2tzfDhwzly5Aje3t5ERUWho6ODm5ubvF7Dhg3x9fWVS9SdP3+eWbNmkZqaikKhYNq0adSvXx8XFxe+/vprTp06RWJiIt7e3lhaWvLkyRN8fHx4+PAhampq/Pzzz7Rt27ZAJFTjxo1L/HtgamrK7NmzgZzfM319fa5fv86iRYs4depUgVivRo0acfToUWbPno0kSRgaGjJ//nz8/PyIj49n2rRpdO3alcDAQIKCgrh8+TI+Pj6kpqair6/PvHnzqFu3LitXriQyMpKsrCzat2+Pu7t7gdJ7giCUI9IbHD9+XGrRooV0//59KSUlRWrevLm0adMmSZIkydPTUwoICJAcHByk58+fS5IkSZs2bZImT54spaWlSe3atZNu3LghSZIkTZ48WRo4cKAkSZI0cOBA6fjx49Lu3bslZ2dnKS0tTXr16pXk4OAgPXr0SBo1apS83tGjRyV7e3tJkiTJw8ND2r59e6F9jY2Nlb7//ntJkiQpMTFR8vDwkNcbMmSIlJ6eLj148EAyNzeXHj16JPn7+0vr1q2TJEmSXr58KdnZ2Un//POP9Ntvv0ljxoyRsrKypEePHkm2trZSWlqaZGVlJd29e1cKDw+Xli5dKkmSJKWlpUmdOnWSLly4IB0/flweY96+Tpw4Udq6daskSZIUEBAgrVixosh9bmVlJdnY2EgODg6Sg4ODNGbMGEmSpCK3W9Rn9Pvvv0uSJEmff/65JEmStGTJEmnJkiXSq1evJHNzc+nVq1dSdna21LlzZ+nhw4dSenq61L9/f0mSJGnGjBnS7NmzC+1rWlqa1K1bN+nevXuSJElSdHS05OrqKn/Ofn5+kiRJ0oEDB6SePXtKkiRJY8eOlfbv3y9JkiQlJCRIHTt2lF6+fCktWbJE3n+SJBXr9yB3TJIkSdnZ2dKCBQukIUOGyNtfsmSJ/Pm6urpKr1+/liRJkhYtWiRNnz5dSktLk8zNzaVLly5JkiRJ8+fPl9avX6/0Web9b1tbW+ngwYOSJEnShg0bpNmzZ0uHDx+WRo0aJWVmZkpZWVnS+PHjpbCwsKI+4nLNfnyYyn+CUJEU67RqUXFJBw8eVBk/dPXqVWrXrk2jRo0A6NmzJ4sXL1Zq98SJE0oRSbmnx/z9/fnzzz/Zs2cP586dU0psKErjxo25desWQ4cOxcLCggkTJsjvOTo6oqWlRd26dfnmm284deoUR48eJTU1le3btwOQkpLC9evXOXHiBH369EFdXZ1atWoVOMqzt7fn/Pnz/P7779y8eZPnz5+TklJ42SwnJycCAgLo1asXu3btKlbeoKrTqkVt902RVqpUqVIFS0tL9u7dS/369alfvz516tQhNjaWb775RuU68+bN48iRI6SmpvLtt9/Sp08f7t69y48//igv8+rVK/m/v/32WyDns3n+/DmQc2R98+ZNlixZAuSkbeSewswbCVXc34Pca6np6ek0atSI6dOny+/ltle1alWVsV5Xr16lTp06csTX+PHjgZwzJvklJiby+PFjrKysgJwzE5BTjef8+fNyAHRqaiqGhoYq+1qY8lIEQKRylE9iLAW9l/JxRcUlZWdnq4wfun//vlIChKpIofwRP/Hx8VSvXh0XFxdat25N69atMTc3V5rkiqJQKIiIiCAmJobDhw/Ts2dPeWLL32dNTU2ys7Px9/enadOmQE6Arr6+vjxZ5rpz54488QAEBQURFRVFnz59aNu2LdeuXSsQc5SXqakpjx49Yu/evdSrV486deoUazz5FbXdoj6jojg5ObFs2TLq1asnf7lHR0fLochfffUVwcHB8vITJkxgwoQJhISEEBcXR3Z2NvXq1ZP/sMnKypKDgUE5fipXdnY269atk2OiEhISqFmzJvv371eKhOrfv3+xfg+Kupaa296DBw9wcXEpEOuVf7+9fPmy0Ek4/7JpaWk8evSIrKwsXF1dGTJkCAAvXrx46wgtQRDKhne+W9XExERl/JCxsTFPnz7lypUrACqvsZmamrJv3z4yMjJ4/fo1w4YN48aNG9y+fZsxY8ZgaWlJTEyMfG3rTQ4cOMCECRPo0KED3t7eVK5cmQcPHgAQGRmJJEncu3eP8+fP07JlS9q0acOmTZsAePToEQ4ODjx48ABTU1N5+adPnzJw4EDS09Pl7cTExNC3b18cHBxQU1PjypUrBaKgNDQ05H6rqanRo0cP/Pz85AnobRRnuyXVqlUrHj58SGxsrBxufObMGVq0aAGAra0tr1+/ZtmyZXJ+46tXr4iNjUVdXZ2GDRuSlJTEyZMnAdi+ffsb/5hp06YNGzduBODGjRs4ODjw+vVrpWWeP3/+1r8HqhQW69WgQQMSExPlGK5Vq1axadMmNDQ0yMzMVGqjWrVq1K1bl5iYGCBnUl68eDFt2rRhx44dJCcnk5mZyU8//URUVNRb91UQhNL3znerFhY/pKWlxYIFC3B3d0dTU5Mvv/yywLrfffcdFy9exNHRkezsbAYNGoSJiQm9e/fGzs6OqlWr0rx5c1JTU4s8bZnLwsKCqKgo7Ozs0NHRoXPnzhgbGwM5p7qcnJxIT09n+vTpKBQK3Nzc8PX1xd7enqysLNzd3fn444/p378/fn5+ODg4ADmBuFWr/t/ht6urK76+vqxZs4YqVarQokUL4uPj+fjjj+Vl2rZty4IFC6hWrRpdu3bFzs6OtWvXyhPQ2yjOdt/Gd999x/Pnz9HW1ubhw4fUrFlTPqrX1tZm/fr1LFq0iB49eshH3NbW1gwbNgxtbW0WL17ML7/8QlpaGlWrVmXOnDlFbs/b2xsfHx+6desGwNy5c5X2L+TkLL7t74Eq7dq1Y9OmTQVivXR0dPD392fixIlkZGTw8ccfM3fuXNLT03n58iXu7u706tVLbsff3x9fX1/mzp2LQqFg7ty51K5dmytXrtCnTx+ysrL49ttv6dmz51v1s6xLTcss9K5U8SiHUJH8JwqPe3p6YmZm9k5Hbe8iOzubTZs2cevWLby9vUulD6pIkkRGRgZDhgxh8uTJ8ull4d9XXq45FqUiXdeCijUeMZaCKlxkVWpqKn379lX53ujRo+nYseO/3KM3c3Nz48GDB6xevRooO2N4/PgxdnZ29O7dW0yMgiAIefwnjhwFoawTR45lT0UajxhLQSKyShAEQRBKSEyOgiAIgpCPmBwFQRAEIZ9SmRwPHDhQoFpOacubVJFbcaW0LF68+K1TL+7evcuYMWPo1KkTXbp0oV+/fpw6deqt+5KboPLy5UtGjhz51u3klZCQwPfff/9e2ioOa2tr4uPj/7XtVVRFJXKIVA6hoimVu1U7duxYJu8qzVWS9IoPYcyYMW+13rNnz+jfvz9jxoyR//g4c+YMo0aNIiwsjJo1a5a4zdwElfj4eLmgw7uqU6dOiRJOhLKhqEQOEKkcQsXyXifH2NhYAgIC0NTU5MGDB5iYmPDLL7/w6NEjhg0bhkKhQEdHBwcHB+Li4pg9ezbW1tbY2Nhw6NAhNDQ0GD9+PGvWrOHOnTt4eHhga2vLtWvXmDFjBikpKSQmJjJkyBAGDRqklOLg7OzMmjVrOHjwIOrq6sTFxbFy5UpWrVpVaH9XrVrFli1bUCgU6OnpyXU4c9MrEhISmDx5Mi9fvpQfe5gwYQIZGRlMnTqVU6dOUadOHdTU1OSjqhUrVqCrq8vff/+NsbEx8+bNQ1tbm+3bt7N27VrU1NRo2rQpU6ZMQVtbu9AUETMzMzp37sz48ePlcmw//fRTkX9UbN68mW+++UbpofUWLVrg6ekpV6CJjo5WmaBibW2Ng4MDf/31F69fv2bOnDl89dVXcoLK2rVrefToET/99BO//vqryvFUqVJFKfkjt8Rc7udsYmLC5cuX8ff3Z+zYsRw8eBBPT0+qVq3K//73PxISEvjpp59wcnIqdN+HhIQQGhrK8+fPMTc3Z8eOHcVOPHn16pXK/bl27VpCQ0NRV1fHxMSE6dOnK/Ud/i9JpnXr1ioTOJKTk0v0WQmCULa99yPH8+fPExYWRoMGDRgzZgwbNmzgu+++49atW6xatYp69eoVCCyuXbs2ERERTJo0iZUrV7J+/XpOnz7NzJkzsbW1ZevWrYwcORJzc3Pu3r2Lg4ODXOg8PT2d3bt3A7B3715iY2MxNzcnNDS0yIf+L1y4wPbt2wkNDUVNTY2+ffsqFb0G2LVrF/b29vTs2ZOXL19iaWnJ//t//4+IiAhev37Nnj17uH//vlzpBXKO1CIjI6lduzZ9+vThr7/+wsjIiOXLl8sT8bRp0wgMDMTKyoqkpCTCwsJ49uwZc+bMoU+fPnJb+/btw8jIiJUrV/L333+zbdu2Ir9wz549S/v27Qu8bm9vD+QUzp4/fz7r169HX1+f4OBg5s2bJx8dGhgYsG3bNoKCglixYoVSILK3tzeDBg3i119/5erVqyrH4+HhUWjfIKeC0aJFiwqc4nz48CEbN27k2rVrDBo0CCcnp0L3PeSclt29ezeampokJSWxZ88eevXqRVhYWJGnxFXtT0tLS1asWMGRI0fQ0NBg2rRpRQZaR0dHc/HiRbZt24aamhru7u7s3LmT7OzsEn1W+RV1S3l58qbC5OVNRRqPGEvJvPfJ0dTUlIYNGwI51+62bNnCd999R40aNQoN77WwsAByEiRq166NpqamUpqEp6cnR44cYcWKFVy9elWphFjeCc3JyYmdO3fSvHlzjh8/zrRp0wrtZ1xcHJaWllSpUgWArl27FqhTOnToUI4fP87q1au5fv26XAM2JiaGPn36oKamhpGRkZyAATnpE3Xr1gWgUaNGJCUlcf/+faysrFAoFAD07duXSZMmMXz48EJTRCDnqG/BggUkJCTQoUMHfvrppyL2fI68Bb4nTpwo7y9nZ2caNmyoMkElV94Ejb179xa6jRMnTqgcz5s0a9ZM5evt2rVDTU2Nzz//XE7uKGzfA3z55ZdyebuSJJ6o2p+ampq0aNGCXr160bFjRwYMGFBkYfhjx46pTOBwcnIq8WeVV3l4zrE4X0gV5Vk6EM8GllXltkJO3jQCSZLkn/OmLeSXN+0gf1IHwNixY9HT08PKygpbW1ul02Z52+3atSsLFy4kKioKCwsLtLW1C92mmpqa0mSoqampVFwcYPbs2dy9exd7e3s6derE0aNH5TEVVvA7N4UidxuSJBVYVpIkMjMzi0wRAfj000+JjIzkyJEj/Pnnn6xZs4bIyMhCQ3S//vprTp8+zYABAwDkAOaAgABSUlLIyspSmaCSv+9vCuktbDx5f1ZTUytQuDvvvlH1et7tFrbvQfkzL0niSWH7c+nSpZw9e5bo6GiGDRvGvHnz5M8uV27R9cISOKpUqVKiz0oQhLLtvd+teurUKRISEsjOziYsLEw+KnwXMTExjB49mk6dOnHixAkAlQkNlSpVwsLCggULFryxjqq5uTmHDh3i5cuXpKWlsW/fPpXbHTp0KDY2Njx48EAeV9u2bdm9ezeSJJGQkEBcXFyRX4JmZmYcPHhQPirasmULrVu3LjJFBOCPP/4gICAAGxsbpk6dSmJiIi9fFv4XU+6dqSEhIfIX+9OnTzl79izq6uo0a9ZMZYJKcWhqasqTXWHjgZzYsOvXryNJEgcPHixW26oUtu/zK0niiar9+ezZM2xsbPj8888ZM2YM7dq14+rVqygUCv7++28kSeLu3bvyddTCEjhK+lkJglC2vfcjx9q1azNx4kQSEhJo164dvXv3VvrCfxujRo2if//+6Onp0aBBA4yMjAq9Nd/Ozo7Tp08XegovV5MmTXB1daVXr17o6empDKcdMWIEEydORE9Pjxo1avDVV18RHx9Pnz59uHLlCt26daNWrVoYGhqiq6tbIHYp1xdffMGIESNwcXEhIyODpk2bMm3aNHR0dApNEQHo0aMH48ePp1u3bmhqauLm5oaenl6hY6pevTrBwcHMnz+f1atXo6Ghgbq6OnZ2dgwaNAgdHR2VCSrFUaNGDQwNDXFxcSEoKEjleAB+/vlnfvjhB2rWrEnLli159uxZsdrPr7B9r0pxE09U7c/q1avj7OxMr169qFSpEh999BE9e/aUb6Lq2rUrDRo0oGXLlkDOYyGqEjhyb8gp7mdVHhWVyAEilUOoWN5rbdXY2FgCAwMJCgp6X02WSFZWFgsXLqRGjRryaa8P4dChQ0iShJWVFS9fvqRHjx5s375dDu8V/j1lNfGkpMrDNcc3qUjXtaBijUeMpaAKl8pRFCcnJxQKBcuWLQPgn3/+YdSoUSqX9fPz4+uvv36r7TRq1IiJEyeyaNEiICdJ49+aGOfMmcPRo0cLvP7VV1/Jd53+l+RPPBEEQXgfRCqHIJQB4six7KlI4xFjKUikcgiCIAhCCYnJURAEQRDyEZOjIAiCIORToW7IEQThw6mmVwldncK/MsSjHEJFIiZH4Y2uXbtGt27dWLJkCV26dFF6b/HixairqyvdFXzo0CGWL19OSkoK2dnZdOrUidGjR6Ou/mFOVOQtEr5kyRLatm1Lq1atCl0+ICCA8PBwdu7cKVfbKe3HkMoDkcoh/JeI06rCG4WEhNClSxeCg4Pl116+fMnkyZNZs2aN0rLR0dFMnz6dWbNmsXPnTrZt28aVK1dYsmTJv9LXEydOqKyelN/9+/dZsGDBv9AjQRDKI3HkKBQpMzOTnTt3smHDBpydnfnnn3/4+OOPOXDgAJ9++mmBYgvLly/Hzc2NBg0aADl1UH19fbl58yaQE/2kr6/P9evXWbRoEY8fP1YZoTVnzhxiYmLQ0NCgY8eOuLm5ySkhuUep1tbWrF+/Xt52WFgYFy9exNvbm8DAQKVqQ/k5Ozuze/duOnfuXOAo88mTJ3h5eXH//n00NTUZN24cFhYWShFpAwYMYM+ePTRp0oRjx46RmpqKt7c3QUFB3Lhxg8GDBzN48OB33v+CIJQOMTkKRTp06BCGhoY0aNCATp06ERwczMSJE+nRoweAUqwVwOXLlwuU7qtbt66cVAI5eZmBgYEkJibi6elZIEJr5MiRREdHExERQVpaGl5eXqSlpb2xr7mVitzc3IqcGAH09fXx9fXFy8urQLj1jBkzaNOmDUOGDOHu3bv069ePsLAwQDkibc+ePQCEh4cTGBiIn58fO3fuJDExkR49epRochSRVWVTRRqPGEvJiMlRKFJISIicB2lra8uECRMYO3ZsoYkn+dMsVMmNGTt37pzKCK06deqgo6ODs7MzVlZWjB07ttBEj3fRqVMnIiMjWbBggVL24vHjx/Hz8wOgfv36NGvWjHPnzin1PVfeuLVmzZpRqVIljIyM5Li14ioPRQBEZFX5JcZS0H+qfJzwfj19+lQO912/fj2SJPHixQv27t0rT5j5ffXVV1y8eJHPPvtMfu3WrVssW7ZMTgDJvQmmsAgtTU1Ntm7dSlxcHNHR0Tg7OxMUFFQgZiw3RupdTJkyBXt7e6Xyf/knd0mS5OuY+aPX3hS3JghC+ST+bxYKtXPnTtq0acOqVavk1wICAti8eXOhk+OwYcOYPn06zZs359NPPyU5OZnZs2fzxRdfFFi2WbNmeHt7c+vWLRo0aMDSpUtJSEhg0KBBzJgxg6CgIMzNzbl06RK3bt1CoVAQGxsLwPnz53n8+HGBNjU0NIp1Q04uAwMDfH19GTt2LC1atAByYqm2bdsmn1Y9ffo0vr6+cmzVf5VI5RD+S8TkKBQqJCSEcePGKb3Wv39/Vq1axd9//02jRo0KrGNhYcG4ceMYN24cWVlZZGZm0rVrV9zc3AosW6tWLZURWgqFgubNm2Nvb0+lSpVo0qQJFhYWvHz5kqioKGxtbWnatClffvllgTa//fZbpk6dypw5c/jmm2+KNc5OnTrRpUsXHj16BICXlxc+Pj6EhIQAOUXqa9euXay2KrKXL15T1MmsinRNSxBE4XFBKAPKwzXHN6lI17WgYo1HjKUgcc1R+E8S0V6CILwLMTkKFZKHh0dpd0EQhHJMVMgRBEEQhHzE5CgIgiAI+YjTqoIgvNGbEjlAPMohVCxicvzA8tcDfRMvLy+cnZ35+uuvi72NTZs2AdCvX7+Sd7AcKmy8edM5inLkyBGWLFnCq1evUFdXp127dowbN45KlSrx8uVLPDw8WLp0KfHx8QwaNIiDBw9+sLGUF29K5ACRyiFULGJyLGPe5k7K/8qkmOtdxnvs2DGmTp1KQEAATZs2JT09ndmzZzNy5EjWrFlDUlISV65ceY+9FQShPBKT4zvIzMzE19eX69ev8+TJExo0aEBgYCB//PEHW7ZsQaFQoKenJ9fjbNeuHVZWVpw8eZJatWrRv39/goKCePjwIbNnz8bMzAwXFxfc3Nz45JNPmDBhAikpKairq+Pt7U3z5s3fmFbx559/smjRIrKzs6lfvz7Tp0+nZs2aWFtb4+DgwF9//cXr16+ZM2cOX331VaFjCw8PZ9WqVWhoaFCvXj38/f3R0dFh+fLl7Ny5Ew0NDdq1a4e7u7tcHzX3CCtvf9q0aUPTpk158uQJ27ZtY9GiRezfvx8NDQ369u2Lq6srd+7cwdfXl+fPn6Orq8uUKVNUPuCfK2/7YWFhLFu2jKpVq2JkZETlypWL/MyWLl2Km5sbTZs2BUBbW5tJkyZhbW3NqVOnWLVqFY8ePeKnn35i0qRJpKamMm7cOK5fv46enh6//vorCoWC6OholWki1tbWmJiYcPnyZTZu3EiNGjWK+dskCEJZIibHd3DmzBm0tLTYvHkz2dnZuLq6sn79ekJDQwkNDUVNTY2+ffvKk+OTJ0/o0KEDfn5+uLi4sH//fjZu3EhoaCjr1q3DzMxMbnvbtm106NCBYcOGERsby6lTp6hVq1aRaRVPnz7Fx8eHTZs2Ua9ePVatWsX06dPlLEUDAwO2bdtGUFAQK1asKJCokdeiRYvYsmULNWrUYOHChdy8eZNHjx5x8OBBQkJC0NTUZNSoUQQHB2NpaVloO8+ePWP48OG0bt2ayMhITp8+TXh4OBkZGfTv3x9bW1s8PDzw8fHhyy+/5MaNG/z0009ERUW9cf8nJCQwb948wsLCMDAwYMSIEW+cHC9cuMDUqVOVXtPS0qJFixZcuHABb29vBg0axK+//kp8fDyJiYkMGTIEExMTRo8eze7du7GxsWH+/PkF0kRyj/otLCxYtGjRG/ufl0jlKJsq0njEWEpGTI7vwNTUFAMDAzZs2MDNmze5ffs2rVu3xtLSkipVqgDQtWtXpWLZuSkORkZGtGzZEshJdMif4mBubs6oUaO4fPkylpaWDBw4EA0NjSLTKs6fP4+JiQn16tUDoG/fvqxcuVJ+/9tvvwWgcePG7N27t8ixWVlZ0a9fPzp27EiXLl1o0qQJO3fuxM7OTi6+7eTkRFhYWJGTIyBHWJ04cQIbGxu0tbXR1tZmx44dJCcnc/HiRSZNmiQvn5KSwrNnz1AoFEW2e+bMGVq0aEHNmjUB6NatG8ePHy9yHTU1NTIzMwu8np6ernL52rVry3/cfPbZZzx79qzQNJH84y2Jsl4hp7hfRhWlCguIqjJllaiQUw4cOHCAJUuWMGjQIBwdHXn27BmVK1dWmug0NTWVvnjzRj1paGgU2nbLli2JiIjg0KFD7N69m9DQUNauXasyrSJX3kkYctIk8k4EuROpmpraG8fm7e3NlStXOHz4MO7u7ri5uRVoH3JOLeePqcrMzFRKqMidTPOnVsTHx6Ovry9PlLkePnyolJJRmPwpHcVJxTAxMeHs2bNKhdDT09O5dOkSw4YNK7B83jZzx1lYmkiuDxGvJQjCv0s85/gOjh07ho2NDU5OTtSsWZMTJ04AOQHBL1++JC0tjX379r1V23PnzmXHjh307NkTHx8fLl26xKVLlxg4cCCmpqZ4eHjQqFEjbt26Ja+TmzsYHx8PwObNm2ndunWJt52ZmUnnzp1RKBSMGDGC7t27c/nyZdq0aUNERASpqalkZmayfft22rRpg56eHklJSSQmJpKens6RI0dUtmtqasq+ffvIyMjg9evXDBs2jCdPnvDpp5/Kk2NMTAwDBgwoVj9btmzJuXPnSEhIIDs7Ww4hLsqoUaNYtmwZ//vf/4Cc2Cs/Pz8aNmxIy5Yt0dTUVHlkmVezZs04e/asvO+XLl0qx3FVVLmJHEX9E49yCBWJOHJ8B71792bChAns2bMHbW1tmjdvTlJSEq6urvTq1Qs9PT0MDQ3fqm0XFxd+/vlnQkND0dDQYOrUqXz55Zcq0ypyv+hr1qzJ9OnTcXNzIyMjA0NDw7e6+1VTU5PRo0czZMgQdHV10dPTY86cOdSpU4fLly/j5OREZmYm3377LQMHDkRTU5OhQ4fSq1cv6tatW+hjKN999x0XL17E0dGR7OxsBg0aRIMGDfD398fX15dVq1ahpaXFwoULi3V0W7NmTby9vRk8eDCVKlVSypAsTKtWrZgzZw6//PILSUlJZGZmYmFhwdKlS1FTU6NGjRoYGhri4uLCrFmzVLZRWJpIRfamRA6oWNe0BEGkcghCGVDWrzkWR0W6rgUVazxiLAWJa45CocpycsXvv/9OaGhogddr167Nb7/9VuS6P//8Mzdu3CjwurW1NWPGjHlvfRQEoeISR46CUAaII8eypyKNR4yloDcdOYobcgRBEAQhHzE5CoIgCEI+4ppjBXXjxg28vb1JSUlBX1+f2bNnY2RkRHp6Ol5eXly8eBFdXV3mzZtHo0aNSru7QhlVnDSOXOJRDqEiEZNjBTVt2jRGjhyJhYUFmzZtYsGCBcyfP5+goCAqVapEZGQkJ06cwNPTk61bt5Z2d4UyqjhpHLlEKodQkYjTqqUgMzMTb29v+vbtS8eOHRk2bBipqakArF+/ns6dO+Pk5IS7u7tc/zQ6OppevXrRo0cP3NzcePbsWZHbWLt2LRYWFmRnZ3P//n309PSAnAIFDg4OQM5D+c+ePeP+/fvyetnZ2VhbW8sPuKekpGBpaUlaWlqhfYiMjKRPnz44ODjQpUsXuRhCbhH1Ll26cP78edzd3enRowc9evRgy5YtBfr85MkTRowYQbdu3ejZsyfR0dFATqHxoUOHYmtry4YNG+Tl79y5Q4cOHeQqOXFxcXKVm+XLl2Nra0u3bt2YPXs2WVlZxMfHY21tLa8fEBAg79/27dszY8YMevTogZOTE3fv3gUgNjaWbt260aNHD3x9fXFxcZG3PWTIEHr27Em/fv24dOkSAJ6envzwww/Y2NiIqCtBKMfE5FgK8hYs37dvH2lpaRw+fJgrV66wYcMGQkJC2LhxI3fu3AEgMTGR+fPns3r1asLCwmjfvj3z5s0rchuampq8ePFCPnLs06cPAI8ePaJWrVrycrVq1eLhw4fyz+rq6vTo0YOdO3cCsHfvXjp06EBycrLKPmRnZxMcHCyndXz//fesXr1abs/Y2JioqChSU1NJSkoiLCyMtWvXcvr06QJ9njFjBm3atCE8PJwlS5YwefJknjx5AuSUeNu9e7dS9ZxPPvmEevXqERsbC0BoaCiOjo4cPnxYLpAeGhrKnTt3CA4OLnJ/PX78GHNzc8LCwjA1NWXDhg1kZGQwceJE/P39CQsLUyol5+Hhgbu7O6GhocyYMYNx48bJ7xkYGBAZGak0EQuCUL6I06qlQFXB8pSUFI4dO4aVlRVVq+bcXmxnZ8eLFy/eWOi6MHp6evz1119ER0fz448/cuDAAZXLqasr/43k6OjIkCFDGDNmDKGhoYwfP77QPqirq/Prr79y8OBBbt26RVxcnFJ7uUW7GzduzK1btxg6dCgWFhZMmDChQD+OHz+On58fAPXr15fL4eVtJz8nJyd27txJ8+bNOX78ONOmTWPhwoVvVSA9b2H2kydPcu3aNWrUqCHXYe3Vqxe//PJLkcXSi+prUUQqR9lUkcYjxlIyYnIsBaoKlkuShLq6usri3m8qdK1KbrSSmpoaFhYW8pFb7dq1efz4MZ988gmQc8RUu3ZtpXXr1auHoaEhe/fu5enTpzRr1oz9+/er7ENycjJOTk50794dU1NTjI2NlU595k5QCoWCiIgIYmJiOHz4MD179iQiIkI+3QuQ/5Hb3CLfedvJr2vXrixcuJCoqCgsLCzQ1tZ+6wLpeQuzS5KEhoaGyrays7OLLJZeWF+LUlafcyzpl1BFeZYOxLOBZZV4zrECU1WwPCsrC3Nzcw4fPsyrV69IT09n7969qKmpvVWh6zVr1shFz48fP45CoaB69epYWlrKX+onT55ER0dHZf1XJycn/Pz85OuThfXh9u3bqKur88MPP9CmTRuio6PlCS2vAwcOMGHCBDp06IC3tzeVK1fmwYMHSsu0adOGbdu2AXD37l1Onz5N8+bNixxnpUqVsLCwYMGCBTg6OsrtvEuB9FwNGzbkxYsXXL16FcgJgAaoVq3aWxdLFwShfBBHjqVAVcHy+Ph4evfuzaBBg+jbty+VK1dGoVCgo6PzVoWuZ8+ezZQpU/j111+pVq2aHHjs4uKCj48PdnZ2aGtrFzrJdu7cmSlTptC9e84diIX1QU9PjyZNmmBjY4Ouri6mpqZKN/jksrCwICoqCjs7O3R0dOjcuTPGxsZKy3h5eeHj40NISAgAfn5+BY5qVbGzs+P06dNyjqKVldU7FUjPlbt/PDw8UFdXp0GDBvJR4dsWSxcEoXwQ5ePKkFu3bnH48GEGDx4MwI8//kjv3r3/9Rs7JEkiOjqaTZs2yadRy6qsrCwWLlxIjRo1GDJkyHttOzs7m3nz5uHm5kblypVZu3YtCQkJeHp6vtftQNk9rVrS5xyTnqd84B79e8SpyLJJFB7/DzIyMuLChQvY29ujpqZG+/btsbKyKnT5D1Vge+bMmfz5559vLPBdFjg5OaFQKFi2bNl7b1tdXR0DAwN69eqFlpYWRkZGpV6Q/d9WnKiqXBXphg9BEEeOglAGlNUjx5KoSEcnULHGI8ZSkLghRxAEQRBKSEyOgiAIgpCPmBwFQRAEIR8xOQqCIAhCPmJyFD6Ya9euybVV81u8eLFc9DvXoUOHcHZ2xsHBAXt7exYtWqSyQs37EhISIj+WsWTJEk6ePPnWbe3YsQM7Ozvs7OyYM2fO++piqaumV4lataoV65+IrBIqEvEoh/DBhISE0KVLF4KDg+nSpQsAL1++ZNasWURERMgJGpCTOjJ9+nRWr15NgwYNSE1NZezYsSxZsoSxY8d+8L6eOHGC1q1bF7lMUFAQrVq1okmTJkqvv379ml9++YU9e/agp6dHv379OHr0KG3btv2QXf5XiMgq4b9KHDkKH0RmZiY7d+5k3LhxXLp0iX/++QfIKSP36aefFnhgf/ny5bi5udGgQQMgpz6pr68vZmZmgHL81eXLlwuNz5ozZw4ODg707NmTwMBAQDmaCnKeA42Pj5d/DgsL4+LFi3h7e8ul4lQxNDRk6tSpuLi4sH//fvmoNisri+zsbF6/fk1mZiaZmZlynVZBEMonceQofBCHDh3C0NCQBg0a0KlTJ4KDg5k4cSI9evQAKHBK9fLly3L5t1x169albt268s/GxsYEBgaSmJiIp6cn69evR19fn+DgYObNm8fIkSOJjo4mIiKCtLQ0vLy8SEtLe2Nfe/Towfbt23FzcytQ0i6vjh070rFjR86fP09QUBALFiyQJ/AxY8bIJfTMzMz45ptvSrC3RCpHWVWRxiPGUjJichQ+iJCQEOzt7QGwtbVlwoQJjB07Fm1tbZXL50/MUCU3Cqqw+Kw6deqgo6ODs7MzVlZWjB079oMcwamrqyv9u3LlCtu3b+fPP/+kWrVqTJgwgdWrVyudNn6TsloEQKRyVIzxiLEUJMrHCf+6p0+fEh0dzcWLF1m/fj2SJPHixQv27t0rT5j5ffXVV1y8eJHPPvtMfu3WrVssW7ZMLo6eW/S7sAgvTU1Ntm7dSlxcHNHR0Tg7OxMUFISamprSjT0ZGRlvNa5Dhw6xYsUKNDU1cXV1ZdasWairq7Nq1SrMzc2pUaMGkJOHuXHjxhJNjoIglC3imqPw3u3cuVOOrzp48CB//vknP/zwA5s3by50nWHDhhEYGMjt27cBSE5OZvbs2Xz00UcFli0sPuvSpUsMHDgQU1NTPDw8aNSoEbdu3UKhUMg1aM+fP8/jx48LtKmhoaEyaiuv27dv4+3tTVBQEJ06dZJDnb/44guOHj1KSkoKkiRx8ODBNyZ+CIJQtokjR+G9CwkJYdy4cUqv9e/fn1WrVvH333/TqFGjAutYWFgwbtw4xo0bR1ZWFpmZmXTt2hU3N7cCyxYWn6VQKGjevDn29vZUqlSJJk2aYGFhwcuXL4mKisLW1pamTZvy5ZdfFmjz22+/ZerUqcyZM6fQ64W5aSn5tW/fnkuXLuHo6IiWlhZff/01w4cPL8aeKvtS0zKLfReqeJRDqEhE4XFBKAPK6jXHkqhI17WgYo1HjKUgcc1REEpgzpw5HD16tMDrX3311X8urkoQ/svE5CgIeXh4eJR2FwRBKAPEDTmCIAiCkI+YHAVBEAQhHzE5CoIgCEI+YnIU/pNiY2NxcXEBwMvLiwsXLpS4jcWLF3PgwIECr+ev5VpelSSRQ6RyCBWNuCFH+M9727tQx4wZ8557UraUJJEDRCqHULGIyVEoV2JjY1m+fDmSJPHPP//QpUsXqlWrxv79+wFYuXIlly5dYsmSJWRmZlKvXj1mzJiBQqHgr7/+YtasWejo6MjpH/B/iR9mZmbMmzeP/fv3o6GhQd++fXF1dSUuLo6FCxeSmppKUlIS7u7u2NjY4OnpiZmZGY6OjqxatYotW7agUCjQ09OT68AKglA+iclRKHfOnTtHREQEBgYGtG3bFg8PD0JCQpg0aRLBwcHs27evQGLH1KlT8fT0ZN26dTRq1AgvL68C7e7Zs4fTp08THh5ORkYG/fv3x9bWlj/++AM/Pz8aNWrEsWPHmDlzJjY2NvJ6Fy5cYPv27YSGhqKmpkbfvn1LPDmKVI6yqSKNR4ylZMTkKJQ7n3/+uVxzVaFQYG5uDuTkLR48eFBlYsfVq1epXbu2XLquZ8+eLF68WKndEydOYGNjg7a2Ntra2uzYkXNK0d/fnz///JM9e/Zw7tw5kpOTldaLi4vD0tKSKlWqANC1a1elQufFURYr5LzNF1BFqcICoqpMWSUq5AhCIbS0tJR+1tDQkP87OztbZWLH/fv3lSasvOvk0tRU/t8hPj6e6tWr4+LiQuvWrWndujXm5uZMmDBBabn8qR+ampqkp6e//QAFQSh14m5VoUIxMTFRmdhhbGzM06dPuXLlCgAREREF1jU1NWXfvn1kZGTw+vVrhg0bxo0bN7h9+zZjxozB0tKSmJiYAukd5ubmHDp0iJcvX5KWlsa+ffs+/EAFQfigxJGjUKEUltihpaXFggULcHd3R1NTU2Uyx3fffcfFixdxdHQkOzubQYMGYWJiQu/evbGzs6Nq1ao0b96c1NRUUlJS5PWaNGmCq6srvXr1Qk9PD0NDw39zyB9MSRI5QKRyCBWLSOUQhDKgLF5zLKmKdF0LKtZ4xFgKetM1R3FaVRAEQRDyEZOjIAiCIOQjJkdBEARByEdMjoIgCIKQj5gcBUEQBCEfMTkK7+zChQsqy7G9jfj4eKytrYHCUy+K4/79+/zwww9069YNe3t7xowZw9OnTwE4f/48/v7+76W/FSWBI7+SJnKIVA6hohHPOQrv7Ouvv+brr79+7+2+S+qFj48PPXr0wN7eHoAVK1YwdepUAgMDuXHjhjxRCqqVNJEDRCqHULGIyVF4Z7GxsQQGBgI5E+WpU6dITEzE29sbS0tLwsPDWbVqFRoaGtSrVw9/f3/Onj1LYGAgQUFBAHLChZmZmdxu3tfc3Nxo3Lgxly9fpkaNGixevBgDA4NC+/TkyRNev34t/zxgwAAuXLjAixcvWLJkCSkpKSxbtozhw4czd+5c4uLiyMrKwtHRkcGDB5OZmYmvry/Xr1/nyZMnNGjQgMDAQHR1dVUmcGzdupXjx48zf/58AAIDA9HW1mb48OEfYI8LgvChidOqwnuVkZHB5s2bmTRpklzYe9GiRaxZs4aQkBAaNGjAzZs3S9zulStXGDJkCLt27UJPT4/w8PAilx8/fjzz5s3DwsICDw8PDh8+TOvWrdHT02P06NFYW1vz448/smXLFgBCQ0PZtm0bBw4c4OTJk5w5cwYtLS02b97Mvn37SEtL4/Dhw0oJHGvXruXhw4cA2NracuzYMZKTk5EkifDwcLp3F0dSglBeiSNH4b369ttvAWjcuDHPnz8HwMrKin79+tGxY0e6dOlCkyZNiI2NLVG7NWrUkEu+NW7cmKSkpCKXt7CwIDo6mtjYWI4dO4a/vz8REREsXbpUabljx45x+fJljh8/DkBKSgpXr15lwIABGBgYsGHDBm7evMnt27dJSUkpNIGjSpUqWFpasnfvXurXr0/9+vWpU6dOCcYnIqvKooo0HjGWkhGTo/Be6ejoADlJFbm8vb25cuUKhw8fxt3dHTc3Nz766CPyVi7MyMgoVru5bRdV9fD58+csXbqUyZMnY2FhgYWFBSNHjqR9+/YkJiYqLZuVlYW7uzudO3cGIDExkcqVK3PgwAGWLFnCoEGDcHR05NmzZ0iSVGQCh5OTE8uWLaNevXo4Ojq+aVcpKWvl4972y6eilCgDUXKtrBLl44QKITMzk86dO6NQKBgxYgTdu3fn8uXLKBQK7t69S1paGs+fP+fUqVPvbZvVqlXj4MGDhIWFya/9888/1KhRA319fTQ0NMjMzASgTZs2bNmyhYyMDJKTk+nfvz/nzp3j2LFj2NjY4OTkRM2aNTlx4gRZWVlFJnC0atWKhw8fEhsbS6dOnd7beARB+PeJI0fhg9LU1GT06NEMGTIEXV1d9PT0mDNnDnXq1MHS0hI7OzuMjIxo2bLle9umhoYGK1euZPbs2SxevBhdXV1q167N8uXL0dDQwMTEhMDAQObNm8eYMWO4c+cOPXv2JDMzE0dHR1q3bo2BgQETJkxgz549aGtr07x5c+Lj4+ndu3eRCRzfffcdz58/R1tb+72NpzSUNJEDRCqHULGIVA5BeA8kSSIjI4MhQ4YwefJkmjZtWqL1y9pp1bdRkU7dQcUajxhLQW86rSqOHIVyy8XFhRcvXhR43dnZmX79+v2rfXn8+DF2dnb07t27xBOjIAhlj5gchXIr9xnJsqB27dqcOHGitLshCMJ7Im7IEQRBEIR8xOQoCIIgCPmIyVEQBEEQ8hHXHIUyx8XFBTc3N1q3bl3aXZFJksTvv/8uPzuprq7OsGHDsLOzA8DY2JirV6+WYg/fTTW9SujqvNvXgXiUQ6hIxOQoCMWwcOFCLl26xB9//EG1atV4+PAhAwcORKFQ0LZt29Lu3jt7mxSO/EQqh1CRiMlRKFWSJDFv3jz279+PhoYGffv2BWDr1q3MmTOHpKQkvLy8sLa25tq1a8yYMYOUlBQSExMZMmQIgwYNIiAggISEBO7cucO9e/fo3bs3P/74IxkZGUydOpVTp05Rp04d1NTUGDlyJK1bt2blypVERkaSlZVF+/btcXd3Vyp5l1dycjLr1q0jIiKCatVyyqrVrVuXBQsWUKlSJXk5Hx8fzp49C+TkPH7yyScfducJgvDBiGuOQqnas2cPp0+fJjw8nK1btxISEsLjx4/R09MjJCQEb29vfv31VyBnwhw5ciTbt29n/fr1LFy4UG7n6tWrrF69mq1bt7Jy5UpevHhBcHAwr1+/Zs+ePcyaNYsLFy4AEB0dzcWLF9m2bRthYWEkJCSwc+fOQvt48+ZNqlSpQr169ZReNzExoXHjxvLPbdu2ZefOnbRr147g4OD3uZsEQfiXiSNHoVSdOHECGxsbtLW10dbWZseOHbi4uMi1ST/77DOePXsG5OQ7HjlyhBUrVnD16lVSUlLkdlq3bo22tjY1atTAwMCAly9fEhMTQ58+fVBTU8PIyAhzc3MgJ4nj/PnzcnHw1NTUAmXg8lJXVy+y0HmuvH0+efJkifaDSOUomyrSeMRYSkZMjkKp0tRU/hWMj48nJSUFDQ0NQDndY+zYsejp6WFlZYWtrS0RERHye6pSOzQ0NJQSNHJlZWXh6urKkCFDAHjx4oW8PVUaNWpEamoq9+/fV5pEIyIiePLkCa6urkpjeVNqiCqlXT7ufX3ZVJQSZSBKrpVVIpVD+E8wNTVl3759ZGRk8Pr1a4YNG0ZCQoLKZWNiYhg9ejSdOnWSq9FkZRV+h2Tbtm3ZvXs3kiSRkJBAXFwcampqtGnThh07dpCcnExmZiY//fQTUVFRhbajq6vLgAED8PX15dWrV0DOJL5gwQIaNWr0DqMXBKGsEkeOQqn67rvvuHjxIo6OjmRnZzNo0CAiIyNVLjtq1Cj69++Pnp4eDRo0wMjIiPj4+ELb7tOnD1euXKFbt27UqlULQ0NDdHV1MTMz48qVK/Tp04esrCy+/fZbevbsWWQ/x40bR2BgIH369EFTUxMNDQ1+/vln2rdv/07jLyveJoUjP/Eoh1CRiFQOocI6dOgQkiRhZWXFy5cv6dGjB9u3b8fAwKC0u1ZAaZ9WfR8q0qk7qFjjEWMpSKRyCP9ZjRo1YuLEiSxatAiA0aNHFzoxpqamyo+R5Dd69Gg6duz4gXopCEJZJI4cBaEMEEeOZU9FGo8YS0HihhxBEARBKCExOQqCIAhCPmJyFARBEIR8xOT4gcXGxuLi4gKAl5eXXMKsJDw9PQkJCXnfXSu277//vtBnD0F5jJs3b2bXrl1vtR1Jkli7di3du3ene/fu9OzZU+lB/7cREBBAQEAAAN27F/2oQkhICJ6enu+0vfKqml4latWq9k7/xKMcQkUi7lb9F/3yyy+l3YW38ttvvxV72TNnzmBmZvZW2/nQyRc7drxb6kRFJlI5BEGZmBwLERsby/Lly5EkiX/++YcuXbpQrVo19u/fD8DKlSu5dOkSS5YsITMzk3r16jFjxgwUCgV//fUXs2bNQkdHhwYNGsht5uYUmpmZFUiicHV1JS4ujoULF5KamkpSUhLu7u7Y2Ni8sa8ZGRlMnjyZ69evA9C/f3/69OmDp6cnampqXLt2jVevXvHjjz/So0cPkpOTmT59OtevXycrK4vvv/8ee3t70tLSmDZtGqdOnUJLS4uRI0dia2uLtbU169evx8DAgMmTJ5OQkMCjR49o1aoVc+fOlftx9OhRDh48yPHjx9HT08PLy4sDBw5QtWpV4uPjGTFiRKFHgsVJvvjjjz/YsWMHr1+/Rk1NjUWLFtGoUSOsra0xMTHh8uXLbNy4kdDQULZs2YJCoUBPTw8TExPg/zIXX79+jbe3N1evXkVNTY2hQ4fSo0cPpf6cPXuWX375hbS0NBQKBdOnT+eTTz7h2rVreHp6kpWVRatWrYiOjiY0NJSOHTsWe6yCIJR94rRqEc6dO8esWbOIiIggODiY6tWrExISgrGxMcHBwcyfP5/Vq1cTFhZG+/btmTdvHunp6Xh6erJkyRJCQkLQ1dUt0G5hSRR//PEHfn5+hIaG8ssvv7B06dJi9fPMmTMkJSURFhbG2rVrOX36tPxeQkICwcHBrFu3jrlz5/L48WOWLVtG06ZNCQkJYcOGDSxfvpy7d+8SFBRESkoKkZGRrF27ll9//ZX09HS5rUOHDtGkSRM2b95MVFQUZ8+e5X//+5/8ftu2bbG2tpZLvHXo0IE9e/YAEBYWVuRpzTclX7x69Yr9+/cTFBTErl276NSpExs3bpSXs7CwICoqivv377N9+3ZCQ0NZu3YtDx8+LLCtgIAAFAoFu3btYt26dQQEBHDlyhX5/fT0dMaPH8+UKVPYuXMnzs7OjB8/Hsg5xT1mzBh27NhB/fr1ycrKomrVqiUaqyAIZZ84cizC559/zkcffQSAQqGQUx0MDQ05ePAgDx48YNCgQQBkZ2ejr6/P1atXqV27tlxzs2fPnixevFipXVVJFAD+/v78+eef7Nmzh3PnzpGcnFysfjZu3Jhbt24xdOhQLCwsmDBhgvyeo6MjWlpa1K1bl2+++YZTp05x9OhRUlNT2b59OwApKSlcv36dEydO0KdPH9TV1alVq1aBIx97e3vOnz/P77//zs2bN3n+/LlSMkZ+Tk5OBAQE0KtXL3kiKsybki+qVq3K/PnziYiI4Pbt2xw5coQmTZrI7zdr1gyAuLg4LC0tqVKlCgBdu3YtUHz8+PHjzJw5E4Dq1avTsWNH4uLiqFo155mn27dvKx1x2tjY4OPjw71797h37x6Wlpby+NavX1/isaoiUjnKpoo0HjGWkhGTYxG0tLSUfs6b3JCdnc0333zD8uXLAUhLSyM5OZn79+8rfRmrSntQlURRvXp1XFxcaN26Na1bt8bc3FxpkiuKQqEgIiKCmJgYDh8+rHQjS/4+a2pqkp2djb+/P02bNgXgyZMn6Ovry5Nlrjt37sh/HAAEBQURFRVFnz59aNu2LdeuXStyQjM1NeXRo0fs3buXevXqUadOnUKXfVPyRefOnXFxcWHgwIFYWFhQs2ZNLl++LC+Xm8qhpqamtP81NTWVjn6BAn2WJEmpgLmqJI/cdQobb0nGqkppFwEQqRwFiQfnyyZRBKCMMzEx4ezZs9y6dQuApUuXMnfuXIyNjXn69Kl8mk7VdSdVSRQ3btzg9u3bjBkzBktLS2JiYopMnMjrwIEDTJgwgQ4dOuDt7U3lypV58OABAJGRkUiSxL179zh//jwtW7akTZs2bNq0CYBHjx7h4ODAgwcPMDU1lZd/+vQpAwcOVJpYYmJi6Nu3Lw4ODqipqXHlypUCE4mGhobcbzU1NXr06IGfn5+cnViYNyVfXLhwgU8++YTBgwfTrFkzoqOjVe4fc3NzDh06xMuXL0lLS2Pfvn0FlmnTpg3btm0DIDExkQMHDijdRNSwYUOeP3/O+fPnAdi9ezeGhoYYGRnx8ccfc/jwYQDCw8PldUoyVkEQyj5x5PiWatWqxcyZMxk7dizZ2dnUqVMHf39/tLS0WLBgAe7u7mhqavLll18WWFdVEoWJiQm9e/fGzs6OqlWr0rx5c1JTU4s8bZkr93qbnZ0dOjo6dO7cGWNjYyCnZqiTkxPp6elMnz4dhUKBm5sbvr6+2Nvbk5WVhbu7Ox9//DH9+/fHz88PBwcHAKZMmSKfagRwdXXF19eXNWvWUKVKFVq0aEF8fDwff/yxvEzbtm1ZsGAB1apVo2vXrtjZ2bF27Vo5CLgoRSVfJCcns2nTJmxtbdHW1sbExES+ASmvJk2a4OrqSq9evdDT01MZYvzTTz/h6+tLt27dyMrK4ocffqBp06ZcvXoVAG1tbRYuXMiMGTN4/fo1+vr6LFy4EIA5c+YwefJkFi1ahLGxsdI15ZKMtawRqRyCoEzUVq3APD09MTMzK7UjmezsbDZt2sStW7fw9vYulT68b7mTd+3atdm7dy/h4eEEBAS881hL+7Tq+1CRTt1BxRqPGEtBIpWjgiiPqRFubm48ePCA1atXA+VzDPkZGhry//7f/0NTUxM9PT352dX8YxUEoXwTR46CUAaII8eypyKNR4ylIHFDjiAIgiCUkJgcBUEQBCEfMTkKgiAIQj7vZXI8cOBAgSowpa0kaQwf2uLFizlw4ECJ1wsICMDY2JgzZ84ovf7LL7/Ij2p8SHnTNj7kesVJHZk0aRL37t0rchkXFxdiY2MLvP5v7CtBECqW93K3aseOHcv0nYalncYwZsyYt163bt26REVF0aJFCyDn8YgTJ068r66VG7Gxsfz000+l3Y0Kq5peJXR13u3rQDznKFQkRf7fEBsbS0BAAJqamjx48AATExN++eUXHj16xLBhw1AoFOjo6ODg4EBcXByzZ8/G2toaGxsbDh06hIaGBuPHj2fNmjXcuXMHDw8PbG1tuXbtGjNmzCAlJYXExESGDBnCoEGDCAgI4OzZszx48ABnZ2fWrFnDwYMHUVdXJy4ujpUrV7Jq1apC+7tq1aoi0xgSEhKYPHkyL1++5PHjx9jZ2TFhwgQyMjKYOnUqp06dok6dOqipqTFy5EgAVqxYga6uLn///TfGxsbMmzcPbW1ttm/fztq1a1FTU6Np06ZMmTIFbW3tQtMxzMzM6Ny5M+PHj+fJkydAzsPob/qjomPHjhw8eFDOGTx16hTNmzeXS6eFhITI+x7+L/njk08+YcKECaSkpKCuro63tzfNmzfn6NGjzJ49G0mSMDQ0ZP78+ezdu1dlG3ndunULHx8fnj9/TuXKlfHy8sLExITw8HBWrVqFhoYG9erVw9/fX2m9devWsX//flauXMmjR4/w9fXl+fPn6OrqMmXKlAJFEsLCwli3bh3Z2dk0bdqUqVOnsm7dOh49esTw4cPZsGEDx48fZ+3ataSmppKWloafnx+mpqZF7keAhw8fqtwnc+bMISYmBg0NDTp27Iibm5t81mHUqFEAcjLJRx99xNy5c4mLiyMrKwtHR0cGDx5caNvlhYisEgRlbzytev78eXx8fNizZw9paWls2LAByPmy9Pf35/fffy+wTu3atYmIiKBp06asXLmSNWvW4O/vz8qVKwHYunUrI0eOZPv27axfv16uPgI5iQi7d+9m0KBB1KtXTz5NFhoaWuTD7BcuXHhjGsOuXbuwt7dny5Yt7Ny5k40bN5KYmEhwcDCvX79mz549zJo1SymQ+MyZM/j4+BAZGcn9+/f566+/uHr1KsuXLycoKIjw8HAqVapEYGBgkekYAPv27cPIyIiQkBD8/f05efLkm3Y/CoWCevXqKZUys7W1feN627Zto0OHDoSEhODu7s6pU6dIT09nwoQJzJkzh/DwcIyNjQkNDX1jWwDu7u64uLgQHh7OpEmTGDNmDOnp6SxatIg1a9YQEhJCgwYNuHnzprzO9u3b2bt3LytWrKBSpUp4eHjg7u5OaGgoM2bMYNy4cUrbuH79Olu2bCE4OJgdO3ZQo0YNVq9ezfDhw6lduzYrV65EX1+f4OBgli9fzs6dO/n++++L/Wyhqn1y7949oqOj2blzJ8HBwdy+fZu0tLRC29iyZQuQ8/u4bds2Dhw4wMmTJ1W2LQhC+fXG8yimpqY0bNgQyLl2t2XLFr777jtq1KhRIF4ol4WFBZDzwHTt2rXR1NTE0NCQFy9eADnXmI4cOcKKFSu4evWqUom03KM9yEk62LlzJ82bN+f48eNMmzat0H4WJ41h6NChHD9+nNWrV3P9+nW5tmlMTAx9+vRBTU0NIyMjOX0DchIv6tatC+QUx05KSuL+/ftYWVmhUCgA6Nu3L5MmTWL48OGFpmMAtGjRggULFpCQkECHDh2KfZrQxsaGqKgomjZtypkzZ5gyZcob1zE3N2fUqFFcvnwZS0tLBg4cyNWrV6lTp46cZpEbw/Sm633Jycn8888/dO7cGYDmzZujr6/PzZs3sbKyol+/fnTs2JEuXbrQpEkTYmNjuXbtGj4+PixYsIDKlSuTnJzMxYsXmTRpktxuSkoKz549k3+OjY3lzp079OnTB8jJqcx/ZKmurs6vv/7KwYMHuXXrFnFxcairF+/Suap9oqGhgY6ODs7OzlhZWTF27Fi5iLkqx44d4/Llyxw/flwew9WrV1W2XRIilaNsqkjjEWMpmTdOjnlTHSRJkn9WlVOYK2+aRf4ECoCxY8eip6eHlZUVtra2SsW587bbtWtXFi5cSFRUFBYWFmhraxe6zeKkMcyePZu7d+9ib29Pp06dOHr0qDwmVUkMgNIXpZqaGpIkFVhWkiQyMzOLTMcA+PTTT4mMjOTIkSP8+eefrFmzhsjISNTU1AodF0CnTp3o168f7du3p1WrVkqTQW6fcmVkZADQsmVLIiIiOHToELt37yY0NBQPDw+ldl++fElycnKhbeQdX2FJFt7e3ly5coXDhw/j7u6Om5sbdevWpUqVKsycOZOZM2fy7bffkp2drRTPBTmnOQ0MDOSfs7KysLGxkcuvJScnFygunpycjJOTE927d8fU1BRjY2P5bMabqNona9euZevWrcTFxREdHY2zszNBQUEFfp9y90luLdrcPxQSExOpXLkyurq6KtsurtIuAiBSOQoSD86XTWWmCMCpU6dISEggOzubsLAw+ajwXcTExMiBuLk3l6hKWKhUqRIWFhYsWLDgjfVBi5PGEBMTw9ChQ7GxseHBgwfyuNq2bcvu3buRJImEhATi4uKKnLDMzMw4ePAgz58/B3JOtbVu3brIdAzISbIPCAjAxsaGqVOnkpiYyMuXb/6QFQoFRkZGLF68uMApVYVCwd9//40kSdy9e1cunj137lx27NhBz5498fHx4dKlSzRo0IDExERu3LgB5Fyj3bRpU6Ft5KpatSr169dn7969AJw9e5YnT57QuHFjOnfujEKhYMSIEXTv3l2+FmpkZETHjh0xMzNjyZIlVKtWjU8//VSeHGNiYhgwYIDSdlq3bs2+fft4+vQpkiTh6+sr5yLmpn3cvn0bdXV1fvjhB9q0aVNoOocqqvbJpUuXGDhwIKampnh4eNCoUSNu3bqFQqGQ99P58+d5/PgxkJPosWXLFjIyMkhOTqZ///6cO3dOZduCIJRfbzxyrF27NhMnTiQhIYF27drRu3dvpS/8tzFq1Cj69++Pnp4eDRo0wMjIiPj4eJXL2tnZcfr0aTnMtjDFSWMYMWIEEydORE9Pjxo1avDVV18RHx9Pnz59uHLlCt26daNWrVoYGhqiq6vL69evVW7riy++YMSIEbi4uJCRkUHTpk2ZNm0aOjo6haZjAPTo0YPx48fTrVs3NDU1cXNzQ09Pr1j7rGvXrvz666/yXau52rZty/bt2+natSsNGjSgZcuWQM5NNT///DOhoaFoaGgwdepUdHR08Pf3Z+LEiWRkZPDxxx8zd+5ctLS0VLaRl7+/P76+vgQEBKClpUVAQADa2tqMHj2aIUOGoKuri56eHnPmzOH27dvyehMnTsTe3p5u3brJbaxatQotLS0WLlyo9EfIF198gZubG66urmRnZ9OkSROGDx8OQIcOHRg+fDi//fYbTZo0wcbGBl1dXUxNTbl//36x9qGqffLll1/SvHlz7O3tqVSpEk2aNMHCwoKXL18SFRWFra0tTZs2lU/vOjs7c+fOHXr27ElmZiaOjo60bt2ajz/+uEDbgiCUX0XWVo2NjSUwMJCgoKB/s0+yrKwsFi5cSI0aNRgyZMgH286hQ4eQJAkrKytevnxJjx492L59u9IpP0H4kEr7tOr7epQj6fmbI9bKC3EqsmwSqRzk3JCjUChYtmwZAP/88498a31+fn5+fP3112+1nUaNGjFx4kQWLVoE5CRE/FsT45w5czh69GiB17/66is58UEQPrSXL17zrl83FemGD0EQqRyCUAaU9pHj+1CRjk6gYo1HjKUgkcohCIIgCCUkJkdBEARByEdMjoIgCIKQj5gcBUEQBCGfMn23qiB8CHkfUfLy8sLZ2fmt73Qu797HIxy5RCqHUJGIyVH4T/uvPy7zPtI4colUDqEiEZOjUG7ExsayfPlyJEnin3/+oUuXLlSrVo39+/cDsHLlSi5dusSSJUvIzMykXr16zJgxA4VCwV9//cWsWbPQ0dGhQYMGcpu58VxmZmbMmzeP/fv3o6GhQd++fXF1dSUuLo6FCxeSmppKUlIS7u7u2NjY4OnpyfPnz7lz5w7u7u7UrFmTWbNmkZqaikKhYNq0adSvX7+0dpUgCO9ITI5CuXLu3DkiIiIwMDCgbdu2eHh4EBISwqRJkwgODmbfvn2sX79ejraaN28eU6dOxdPTk3Xr1tGoUSO8vLwKtLtnzx5Onz5NeHg4GRkZ9O/fH1tbW/744w/8/Pxo1KgRx44dY+bMmdjY2ABgYGDA8uXLSU9Pp1evXixfvhxDQ0OOHDnClClTVMa5FUakcpRNFWk8YiwlIyZHoVz5/PPP+eijj4Ccouu58WKGhoYcPHiQBw8eMGjQIACys7PR19fn6tWr1K5dm0aNGgHQs2dPFi9erNTuiRMnsLGxQVtbWyk9xN/fnz///JM9e/Zw7tw5kpOT5XVy49Vu377N3bt3+fHHH+X3Xr16VaJxlVYRgPf9JVNRHjQH8eB8WSXKxwmCCnnj0EA5Ui07O5tvvvmG5cuXA5CWlkZycjL3799Xip/Ku06u/NFq8fHxVK9eHRcXF1q3bk3r1q0xNzdXyujMjVfLzs6mXr168oSalZXFkydP3nGkgiCUJvEoh1BhmJiYcPbsWW7dugXA0qVLmTt3LsbGxjx9+pQrV64AKGVs5jI1NWXfvn1yAPawYcO4ceMGt2/fZsyYMVhaWhITE6MyHqthw4YkJSVx8uRJALZv314g6FoQhPJFHDkKFUatWrWYOXMmY8eOJTs7mzp16uDv74+WlhYLFizA3d0dTU1NOX4qr++++46LFy/i6OhIdnY2gwYNwsTEhN69e2NnZ0fVqlVp3rw5qamppKQoJ09oa2uzePFifvnlF9LS0qhatSpz5sz5t4b9TlLTMt/bXabiUQ6hIhGFxwWhDBCFx8ueijQeMZaCROFxQRAEQSghMTkKgiAIQj5ichQEQRCEfMTkKAiCIAj5iMlREARBEPIRj3IIpeLGjRt4e3uTkpKCvr4+s2fPxsjIiPT0dLy8vLh48SK6urrMmzdPrmyT16tXr5g6dSp///03kFNAvGnTpsVeX8ghUjkEQTUxOQqlYtq0aYwcORILCws2bdrEggULmD9/PkFBQVSqVInIyEhOnDiBp6cnW7duLbD+rFmz+Oijj5g/fz7R0dH4+vqydevWYq8v5BCpHIKgmpgchQIyMzPx9fXl+vXrPHnyhAYNGhAYGIiuri7r16/njz/+oFq1ajRs2JCPP/6YUaNGER0drTINozBr165FU1OT7Oxs7t+/j56eHgCHDh1izJgxQE7VmmfPnnH//n0MDQ3ldSVJYu/evRw4cAAACwsLud7qm9bPzs6mU6dOrF69mgYNGpCSkoKNjQ179+4lNjZW5RgiIyNZu3YtqamppKWl4efnh6mpKS4uLujr63P9+nX8/f0JCgri+vXrAPTv358+ffq8509GEIR/i5gchQLOnDmDlpYWmzdvJjs7G1dXVw4fPswnn3zChg0bCAkJQUtLCxcXFz7++GMSExOZP39+gTSMorISNTU1efHiBba2tqSmphIUFATAo0ePqFWrlrxcrVq1ePjwodLk+PTpU7S1tfnjjz/Yu3cvenp6TJ48uVjrq6ur06NHD3bu3MmYMWPYu3cvHTp0IDk5WeUYZsyYQXBwMMuXL6d69eps27aN1atXY2pqCoCxsTGBgYHExcWRlJREWFgYz549Y86cOSWaHEUqR9lUkcYjxlIyYnIUCjA1NcXAwIANGzZw8+ZNbt++TUpKCseOHcPKyoqqVXO+yO3s7Hjx4gXnzp1TmYbxJnp6evz1119ER0fz448/ykeC+amrK983llvYW19fn7CwMGJiYvjpp5+Kvb6joyNDhgxhzJgxhIaGMn78+ELHoK6uzq+//srBgwe5desWcXFxSu3lJnM0btyYW7duMXToUCwsLEpcW1WkcpQ9oqpM2SRSOYRSc+DAAZYsWcKgQYNwdHTk2bNnSJKEurq6UrpFrqysLJVpGEXZvXs3NjY2qKmpYWFhIYcJ165dm8ePH/PJJ58A8PjxY2rXrs3333/Po0ePgJyC4pqamtjb2wPQrl07UlJSePr0aaHr51WvXj0MDQ3Zu3cvT58+pVmzZuzfv1/lGJKTk3FycqJ79+6YmppibGzMhg0b5LZykzkUCgURERHExMRw+PBhevbsSUREhHy6WBCE8kU8yiEUcOzYMWxsbHBycqJmzZqcOHGCrKwszM3NOXz4MK9evSI9PZ29e/eipqZGs2bNVKZhFGXNmjXs27cPgOPHj6NQKKhevTqWlpZy9NPJkyfR0dHB0NCQ3377jR07drBjxw6MjIxo27atnK5x9uxZKlWqhEKhKHT9/JycnPDz88PBwQGg0DHcvn0bdXV1fvjhB9q0aUN0dLTKZI4DBw4wYcIEOnTogLe3N5UrV+bBgwdvs/sFQSgDROFxoYCrV68yYcIENDQ00NbWpk6dOjRs2JBx48axYcMGNm7cSOXKlVEoFJiamvL9999z8OBBFi9erJSGUdQNOTdu3GDKlCmkpKRQrVo1pk6dSuPGjUlLS8PHx4eLFy+ira2Nn58fTZs2LbD+o0eP8PHxIT4+Hk1NTaZNm0azZs2KvX5qaiqtW7dm79691KlTB0DlGPT09HB3d+d///sfurq6mJqasn//fg4dOoSLiwtubm60bt2ajIwM+RESHR0d2rdvz88//1zsfV5ap1Xf96McSc9T3rxgOSFORZZN/9ZpVTE5CsV269YtDh8+zODBgwH48ccf6d27N9bW1qXbsRKSJIno6Gg2bdokn0YtbSKVo+ypSOMRYylIXHMU3hsjIyMuXLiAvb09ampqtG/fHisrq0KX//nnn7lx40aB162treXHLUrDzJkz+fPPP/ntt99KrQ+CIJRt4shREMoAceRY9lSk8YixFCTyHAVBEAShhMTkKAiCIAj5iMlREARBEPIRN+QIwn/U+3yMA0Qqh1CxiMlRKHMuXLhAcHBwkbVZiys+Pp5BgwbJzzB+9dVXdOzYscTtuLi4ULt2bebPny+/FhAQAMCoUaPeuZ+l4X0mcoBI5RAqFjE5CmXO119/zddff/3e233Xx0eioqKwsbGhU6dO76lHgiCUVWJyFMqc2NhYAgMDgZyJ8tSpUyQmJuLt7Y2lpSXh4eGsWrUKDQ0N6tWrh7+/P2fPniUwMFBO9/D09MTMzAwzMzO53byvubm50bhxYy5fvkyNGjVYvHgxBgYGRfbrxx9/ZNq0abRq1arAsmfPnuWXX34hLS0NhULB9OnT5fqugiCUP2JyFMq0jIwMNm/eLJ8WtbS0ZNGiRWzZsoUaNWqwcOFCbt68WeJ2r1y5wsyZM/nyyy8ZNWoU4eHhuLi4FLlOq1ateP78OTNmzFA6vZqens748eNZtGgRJiYmREZGMn78eLZv317s/ojIqrKpIo1HjKVkxOQolGnffvstkBMJ9fz5cwCsrKzo168fHTt2pEuXLjRp0oTY2NgStVujRg2+/PJLue2kpKRirTd+/Hi6d+/O/v375ddu376Nnp6eHF9lY2ODj48PL1++pFq14v1PXBpFAD7EF0xFedAcxIPzZZUoAiAIgI6ODgBqamrya97e3ixZsgQDAwPc3d3ZsWMHampq5C32lJGRUax2c9subqGoSpUqMXPmTKZNmyZPqKpivCRJUpneIQhC+SCOHIVyJTMzE1tbW4KCghgxYgQZGRlcvnyZL7/8krt375KWlsbr1685deoU7dq1+yB9aNWqFV27dmXTpk2MGDGChg0b8vz5c86fP4+JiQm7d+/G0NDwjdcwS1tqWuZ7vcNUPMohVCRichTKFU1NTUaPHs2QIUPQ1dVFT0+POXPmUKdOHSwtLbGzs8PIyIiWLVt+0H6MHz+ew4cPA6Ct2xZSmgAAGRNJREFUrc3ChQuZMWMGr1+/Rl9fn4ULF37Q7b8PL1+85n2eaKtI17QEQRQeF4QyQBQeL3sq0njEWAoSkVWCUEwuLi68ePGiwOvOzs7069evFHokCEJpEZOjIPz/cp+RFARBEHerCoIgCEI+YnIUBEEQhHzEaVVB+A9634kcIB7lECoWMTkKRfL09OTTTz/l1KlT/Pbbb4Uu9zYJFS4uLm91nS8kJIRJkyYxf/587O3t5dd///13Zs2axYEDB6hXr16x2ipOv0UiR/GIVA6hIhGnVYU3ql27dpET49uKi4t763Xr1q1LVFSU0mv79u1DT0/vXbulUlRUlFLJOEEQKjYxOQpKJEli1qxZdOnSBRcXF/755x8ArK2tAbh27RouLi44OTlhZWXF+vXr5XXPnz9P7969sbOzY926dfLrK1eupGfPnjg4ODB37lwkScLPzw+A3r17AxAdHU2vXr3o0aMHbm5uPHv2DIA5c+bg4OBAz5495aQOAFNTUy5evEhKSgoA9+7do0qVKkq1TFVtF2DVqlV07tyZvn37cv78+WLtl9xEjtz6rnmdPXuW3r174+DggKurK3fu3ClWm4IglF3itKqgJCoqikuXLrFr1y5evnyJg4OD0vtbt25l5MiRmJubc/fuXRwcHBg0aBAAjx8/ZuPGjWRnZ+Po6IiZmRmPHz/m4sWLbNu2DTU1Ndzd3dm5cyfe3t4EBQWxdetWEhMTmT9/PuvXr0dfX5/g4GDmzZvHyJEjiY6OJiIigrS0NLy8vEhLSwNyKuW0b9+ew4cPY2NjQ2RkJDY2NvLpzujoaJXbbdiwIdu3byc0NBQ1NTX69u0rFwwvyodM5ACRylFWVaTxiLGUjJgcBSVxcXF07twZLS0tqlevjoWFhdL7np6eHDlyhBUrVnD16lX5yA3A1taWypUrAznJGXFxcTx8+JDz58/j6OgIQGpqKoaGhkptnjt3jgcPHsiTbHZ2Nvr6+tSpUwcdHR2cnZ2xsrJi7NixSgXDbWxs2LJlCzY2Nuzfv5/ffvtNnhyPHTumcrtPnjzB0tKSKlWqANC1a1eVhcNV+VCJHPDvV8j5UF8uFaUKC4iqMmWVqJAjlAo1NTWlyUJTU/lXZOzYsejp6WFlZYWtrS0REREql5UkCU1NTbKysnB1dWXIkCEAvHjxAg0NDaU2s7Ky+Oabb1i+fDkAaWlpJCcno6mpydatW4mLiyM6OhpnZ2elG3hat26Nt7c3165dQ6FQKE1GhW138+bNBcaXnp5erH2Tm8gxbtw4unTpgr6+vkjkEIQKSkyOghJzc3NWr15Nv379eP36NUeOHKF58+by+zExMURGRlKnTh1CQkIA5IkgKiqKgQMH8vr1a/7880+WL1/ORx99xJIlS+jTpw86Ojr89NNP9OzZE0dHRzQ0NMjMzKRZs2Z4e3tz69YtGjRowNKlS0lISGDQoEHMmDGDoKAgzM3NuXTpErdu3ZL7oqGhQfv27fHx8WHAgAFK42jTpo3K7ZqbmzNmzBhGjRqFtrY2+/btw9LSstj7RyRyFE48yiFUJGJyFJR06tSJCxcuYG9vT82aNWnUqJHS+6NGjaJ///7o6enRoEEDjIyMiI+PB8DQ0BBnZ2fS0tIYMWIEjRo1olGjRly5coU+ffqQlZXFt99+S8+ePQHo2LEj3bt3JyQkhJkzZzJ27Fiys7OpU6cO/v7+KBQKmjdvjr29PZUqVaJJkyZYWFiwc+dOuT82Njbs2LFDvmEol7W1tcrtqqmp4erqSq9evdDT0ytwirc4RCKHahXpmpYgiFQOQSgDRCpH2VORxiPGUpC45igIxSASOQRByEtMjoKASOQQBEGZKAIgCIIgCPmIyVEQBEEQ8hGnVYX/tJCQEOLi4pg9e3Zpd6VEPkSqxrsSj3IIFUnZ+r9LEIRi+RCpGu9KpHIIFYmYHAWVYmNjWbFiBbq6uvz9998YGxszbtw4hg4dysGDBwHl2KZ27dphZWXFyZMnqVWrFv379ycoKIiHDx8ye/ZszMzMCt2Wp6cnampqXLt2jVevXvHjjz/So0cPAgICOHv2LA8ePGDAgAG0bdsWHx8fnj9/TuXKlfHy8sLExIR79+4xadIkEhMT0dXVxc/Pjy+++IKwsDDWrVtHdnY2TZs2ZerUqejo6BAWFsayZcuoWrUqRkZGcsk7a2tr1q9fT7169YiNjSUwMJCgoCBcXFzQ19fn+vXrLFq0iMePH7NkyRIyMzOpV68eM2bMQKFQMGfOHGJiYtDQ0KBjx464ubl9+A9KEIQPQlxzFAp15swZfHx8iIyM5P79+/z111+FLvvkyRM6dOjAnj17ANi/fz8bN25k1KhRSgkdhUlISCA4OJh169Yxd+5cHj9+DOQU9t69ezcDBgzA3d0dFxcXwsPDmTRpEmPGjCE9PZ1p06bRpUsXdu3axahRo1i2bBnXr19ny5YtBAcHs2PHDmrUqMHq1atJSEhg3rx5bNiwgc2bN5OcnFysfWFsbExUVBR16tRh/vz5rF69mrCwMNq3b8+8efO4d+8e0dHR7Ny5k+DgYG7fvi0XSRcEofwRR45CoRo3bkzdunUBaNSoEUlJSUUun1uk3MjIiJYtWwI5VXNUPT+Yn6OjI1paWtStW5dvvvmGU6dOAcgFvZOTk/nnn3/o3LkzAM2bN0dfX5+bN29y4sQJFixYAIClpSWWlpb88ccf3Llzhz59+gCQkZHBl19+yZkzZ2jRogU1a9YEoFu3bhw/fvyN/cvtx9sWSX8TkcpRNlWk8YixlIyYHIVC5f1yV1NTAyBvQaXMzEylYuPa2tryf+cvLv4meZfPzs6W29XV1ZW3m7+YU26B7/wFz//++2+ysrKwsbHB29sbyJlcs7KyOHbsWJGF1XO3kZmZqfR6bj9KWiS9QYMGxRp/SSvklNUvuopShQVEVZmy6t+qkCNOqwrFVq1aNZKSkkhMTCQ9PZ0jR468t7YjIyORJIl79+5x/vx5+cgzV9WqValfvz579+4FcgKGnzx5QuPGjWnVqpWcDnL06FGmTJlC69at2bdvH0+fPkWSJHx9fVm3bh0tW7bk3LlzJCQkkJ2dze7du+VtKBQKbty4AcCBAwdU9rNZs2acPXtWLoC+dOlS5s6dy6VLlxg4cCCmpqZ4eHjQqFEjpSLpgiCUL+LIUSi2atWqMXToUHr16kXdunX5+uuv31vbqampODk5kZ6ezvTp01EoFAWW8ff3x9fXl4CAALS0tAgICEBbWxsfHx+8vb3ZuHEjlSpVws/Pj88++ww3NzdcXV3Jzs6mSZMmDB8+HB0dHby9vRk8eDCVKlXis88+k9sfPXo0M2bMIDAwkPbt26vsZ61atUpUJF0QhPJJFB4XSp2npydmZmZyMPF/UUlPq5bV5xyTnqe8ecFyQpyKLJtE4XGhQpkzZw5Hjx4t8PpXX31VCr0p/z5E5NS7KqvXQQXhbYjJUfhXeHh4lHYXBEEQik3ckCMIgiAI+YjJURAEQRDyEZOjIAiCIORTpifHAwcOsHjx4tLuhpKAgAC5pmj37qVbaHnx4sWFPo9XlICAANq1a0f37t3p3r07NjY2dOvWTa5K876UdP8cPHiQtWvXvtc+QE7N1Pj4+PferiAIFVeZviGnY8eOdOzYsbS7UagdO0o3FWHMmDFvva6zszOjRo2Sf/7999+ZPXs2W7dufR9dA0q+f/73v/+9t21XRGXx8Y28RGSVUJGUyv9psbGxBAQEoKmpyYMHDzAxMeGXX37h0aNHDBs2DIVCgY6ODg4ODnLWnrW1NTb/X3t3HlRV/T5w/H0FJZVBUIEJdEYzA8VInVFEDLyCCyAGsejQiBoaJqZG4koKiY6JqaOOJeMyWZZjIrgCLrgiSqlljYpmpJCyGIvEUgjn9wfj+cFlu6YJ+n1ef3Hn3HPO83w+zH3uOffez+PuzsmTJzEwMCAsLIxt27Zx+/Zt5s+fj4eHBzdu3GDZsmWUlZVRUFDAlClTCAoKqtPdYcKECWzbto2UlBTatGlDeno6sbGxbNmypdF4t2zZwu7duzEzM8PExERdZ9PGxoaMjAxyc3NZtGgRJSUl5Ofn4+npydy5c6msrGTp0qVcvHgRS0tLNBoNM2bMAKjX8WL16tW0a9eOuLg4tm/fjkajwc7Ojo8//ph27dqxaNEibt68CUBgYCABAQHq7wNHjRpFWFgY9+/fByA0NPSx3lRUV1eTk5NDp06dgJpFxJcsWUJOTg4ajYaPPvqIoUOHkpaWRkxMDACdOnXis88+o6ysjPfff5/u3btz+/ZtrKysiImJwdTUVB0f3e4avXv3Zu3atVRUVFBcXEx4eDi9e/dm165dQM16rB4eHkRERJCRkYFGoyE4OBhvb2/27t1LfHw8RUVFODo6sm/fPo4fP46xsTHZ2dmEhISoq+Xo+uuvvxocp+3btxMfH0+bNm2wt7fnk08+qdfnceLEicycORMHBwdiY2NJTEykqqqKYcOGER4eTmlp6RPNgT5aY5uq2qRllXiRtNjb0CtXrpCQkEDPnj2ZPXs2O3fuZOTIkWRmZrJlyxa6devG3r176+xjYWHBoUOHWLhwIbGxsezYsYNLly6xYsUKPDw8+O6775gxYwaOjo5kZWUxbtw4dYHoR90dAI4cOcKFCxdwdHQkPj6+yR+f//zzz8TFxREfH49Go2H8+PFqcXzk4MGDjB07Fh8fH0pKSnBxceHdd9/l0KFDlJeXk5SUxN27d/Hy8lL3uXz5MomJiVhYWBAQEMDZs2extrbmiy++UAtxVFQUGzduRKvVUlxcTEJCAoWFhXz66afqgtoAR48exdramtjYWG7dusWePXuafWHetWsXx44d48GDB1RXVzN8+HBWrFgBwPLly/H19cXV1ZW8vDwCAwNJSEhg06ZNREZGYm9vz44dO7h69So9evTgxo0bRERE4ODgwMqVK9m4caO6pukjtcd/1qxZREdH06tXL9LS0lixYgUHDhxgwoQJAPj6+rJq1SrMzMw4ePAgBQUF+Pv7Y2trC9R08Dh8+DCGhoYUFxeTlJSEn58fCQkJTd7KbWicXFxc2Lx5M2fOnMHAwICoqChyc3MbPcbp06f55Zdf2LNnDxqNhvDwcPbv3091dfVjz4EQovVqseI4aNAgXnnlFaDms6ndu3czcuRIunTpQrdu3Rrc59FyXFZWVlhYWGBoaFin68OCBQs4c+YMmzdvJiMjg7Ky/1+to3ZB8/X1Zf/+/fTv35/z588TFRXVaJzp6em4uLjQsWNHAMaMGVNn4WqA4OBgzp8/z9atW7l58yaVlZWUl5eTmppKQEAAGo0Ga2trHB0d1X0a6nhx9+5dtFqtunTa+PHjWbhwIe+99x6ZmZkEBwfj7OzM3Llz65x/wIABrFmzhtzcXIYPH05oaGgTI1/j0W3V/Px8Jk2ahJ2dHRYWFkDN+qS//fYb69evB2oW4c7KylJ7FLq5ueHq6oqTkxPZ2dn06NEDBwcHALy9vevFpzv+MTExnDhxgqSkJH766acG20adP39eLdadO3fG1dWV9PR0jI2N6du3r7pguK+vLxs2bMDPz4+DBw822R6roXEyNDRkwIAB+Pn54erqyjvvvIOlpWWjx0hLS+PKlSvqG6qKigqsrKzw9fV97DmoTbpytE4vUj6Sy+NpseJYuwuDoijq40fdDxrStm1b9W/dbgoAc+bMwcTEBK1Wi4eHR53ba7WPO2bMGNauXUtycjLOzs51ukno0mg09bo4/PPPP3Wes3LlSrKyshg7dixubm6cO3dOzUm3kD6i2/FCUZR6z1UUhYcPH2JmZsahQ4dITU3l1KlT+Pj41MmtR48eJCYmcubMGU6cOMG2bdtITExUO2k0xdzcnOjoaKZMmYKjoyPdu3enurqaL7/8ElNTU6DmSq1r16706dMHrVbLiRMniImJ4cqVK3h5edXritFQR47a4x8YGIiDgwMODg44Ojo2WEwb68Che6xBgwaRl5fHkSNH6NatW5OFrbFx2rRpEz/++COnT59m6tSprF69Wp2TRyorK4GarhyTJk1iypQpADx48AADAwM6duz4r+cA9Fs+7nl4cXtRligDWXKttXrhu3JcvHhR7YyQkJDwVBZpTk1NZdasWbi5ufH9998DqC+otbVv3x5nZ2fWrFnT7Hqejo6OnDx5kpKSEv7++2+OHj3a4HmDg4Nxd3fn3r17al5Dhw7l8OHDKIpCbm4u6enpTb5YDh48mJSUFIqKigDYvXs3Dg4OHD9+nLlz5zJ8+HAiIiLo0KED9+7dU/f7+uuv2bBhA+7u7ixdupSCggJKSvT/5xk4cCAjRoxQP08cMmQI33zzDQC//vor48aNo7y8HH9/f0pLS5k8eTKTJ0/m6tWrAGRmZnLt2jUA4uLimpzLoqIifv/9d2bPno2LiwupqanqHBkYGKitooYMGcKePXsAKCgo4Pjx4wwePLje8TQaDd7e3kRHRzc7lw2NU2FhIe7u7rz22mvMnj0bJycnMjIyMDMz49atWyiKQlZWFhkZGWpc+/bto7S0lIcPHxIaGkpycvITz4EQonVpsStHCwsL5s2bR25uLk5OTvj7+9d5wf83PvjgAwIDAzExMaFnz55YW1s3+hV+T09PLl26xBtvvNHkMfv06cOkSZPw8/PDxMQEKyures8JCQlh3rx5mJiY0KVLF/r160d2djYBAQFcv34dLy8vzM3NsbKy4qWXXqK8vLzBc9na2hISEsLEiROprKzEzs6OqKgojIyMSE5OxtPTEyMjI0aNGoWNjY26n7e3N2FhYepV3MyZMzExMXmMkYOwsDA8PDz44YcfiIiIYMmSJepnpKtWrcLY2JiwsDAWLFiAoaEhRkZG6u3oTp06sX79eu7cuYONjQ3R0dGNnsfU1BR/f388PT0xNjamf//+VFRUUFZWprZ76tq1K6GhoURGRuLl5UVVVRXTp0/Hzs5OLVK1eXp6sn37dtzc3JrMsaFx6ty5MxMmTMDPz4/27dvz8ssv4+Pjo345asyYMfTs2VNtoTVixAiuX79OQEAAVVVVvPnmm/j4+KhfyHmSORBCtB4t0pXjwoULbNy4ka+++upZnxqouZpcu3YtXbp0UW+P/RdOnjyJoihotVpKSkrw9vYmLi5OvV35IsjOziYoKIiUlJQWOX91dTXffvstmZmZ9b4E9DzR57bq8/BTDunK0TpJLvVJV44G+Pr6YmZmxueffw7AnTt36vzmr7bo6Oh/3bewV69ezJs3j3Xr1gE139J8VoWxqS4Yy5cvfyYxPAszZ87k3r17bN26taVD+c+1xk4ctT0Pn4kKoS/p5yhEK1BYWPpY/Rxboy5djPnzz79aOoyn5kXKR3Kpr00bDWZmHRvdLsVRCCGE0NGq11YVQgghWoIURyGEEEKHFEchhBBChxRHIYQQQocURyGEEEKHFEchhBBChxRHIYQQQocURyGEEEKHFEchhBBChxRHIcRjOXDgAB4eHowcOZKdO3fW237t2jV8fX0ZPXo0ixcvVtuQtVbN5fPI/Pnz2bt37zOM7PE1l8uxY8d46623GDduHDNmzKC4uLgFotRPc7kcPXoULy8vPD09WbBgQb0+u09MEUIIPeXk5CharVYpLCxUSktLFS8vL+XmzZt1nuPp6alcvnxZURRFWbhwobJz584WiFQ/+uSTk5OjhISEKPb29kpcXFwLRdq85nIpKSlRnJyclJycHEVRFGXdunXKsmXLWircJjWXS2lpqTJs2DAlPz9fURRFmTNnjrJr166nGoNcOQoh9Hbu3DmGDBmCqakpHTp0YPTo0SQlJanb//jjDyoqKujfvz8Ab7/9dp3trU1z+UDNFYyrqyvu7u4tFKV+msulsrKSyMhILC0tAbCxsXniHrr/leZy6dChAykpKXTt2pWysjL+/PPPp94/VYqjEEJveXl5mJubq48tLCzIzc1tdLu5uXmd7a1Nc/kATJ06FX9//2cd2mNrLhczMzO1IXhFRQWxsbHNNghvKfrMS9u2bTl16hRarZbCwkKGDRv2VGOQ4iiE0JvSQBMfjUaj9/bW5nmLtyn65lJSUsK0adOwtbXFx8fnWYT22PTNxcXFhQsXLqDVaomMjHyqMUhxFELozdLSkvv376uP8/LysLCwaHR7fn5+ne2tTXP5PE/0ySUvL4/AwEBsbW1bddPz5nIpKiri7Nmz6mMvLy8yMjKeagxSHIUQehs6dChpaWkUFBRQXl7OkSNHcHZ2VrdbW1tjZGTExYsXAUhISKizvbVpLp/nSXO5VFVVMX36dNzd3Vm8eHGrvkJuLhdFUQgPD+fu3bsAJCYmMnDgwKcag+FTPZoQ4oVmaWnJhx9+SFBQEJWVlfj5+WFvb8+0adOYNWsWr7/+OqtXryYiIoLS0lL69u1LUFBQS4fdKH3yeV40l0tOTg5Xr16lqqqK5ORkAPr169cqryD1mZdly5YREhKCRqPh1VdfJSoq6qnGoFEaurkrhBBC/A+T26pCCCGEDimOQgghhA4pjkIIIYQOKY5CCCGEDimOQgghhA4pjkIIIYQOKY5CCCGEDimOQgghhI7/A+9XMNrbAqWKAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 703 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 297 - }, - "id": "SweXBa-vEWFM", - "outputId": "a230ed5d-2665-477d-adba-287c76020032" - }, - "outputs": [], - "source": [ - "plot_technique_comparison(test_dict, \"false_negative_rate\")" - ] + "id": "Owzkar8R9Cyy", + "outputId": "7c073a93-e3e5-4f22-e089-217f97967df7" + } + }, + { + "cell_type": "markdown", + "source": [ + "# **Fairness assessment**" + ], + "metadata": { + "id": "hX8CrWjhD7MB" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Measuring fairness-related harms\n", + "\n", + "\n", + "\n" + ], + "metadata": { + "id": "0CS9-jaxtxh2" + } + }, + { + "cell_type": "markdown", + "source": [ + "The goal of fairness assessment is to answer the question: *Which groups of people may be disproportionately negatively impacted by an AI system and in what ways?*\n", + "\n", + "The steps of the assesment are as follows:\n", + "1. Identify harms\n", + "2. Identify the groups that might be harmed\n", + "3. Quantify harms\n", + "4. Compare quantified harms across the groups\n", + "\n", + "We next examine these four steps in more detail." + ], + "metadata": { + "id": "s8TMm9w8duVY" + } + }, + { + "cell_type": "markdown", + "source": [ + "### 1. Identify harms\n", + "\n", + "For example, in a system for screening job applications, qualified candidates that are automatically rejected experience an allocation harm. In a speech-to-text transcription system, high error rates constitute harm in the quality of service.\n", + "\n", + "**In the health care scenario**, the patients that would benefit from a care management program, but are not recommended for it experience an allocation harm. In the context of the classification scenario these are **FALSE NEGATIVES**." + ], + "metadata": { + "id": "X6hMFmzmPbL6" + } + }, + { + "cell_type": "markdown", + "source": [ + "### 2. Identify the groups that might be harmed\n", + "\n", + "In most applications, we consider demographic groups including historically marginalized groups (e.g., based on gender, race, ethnicity). We should also consider groups that are relevant to a particular application. For example, for speech-to-text transcription, groups based on the regional dialect or being a native or a non-native speaker.\n", + "\n", + "It is also important to consider group intersections, for example, in addition to considering groups according to gender and groups according to race, it is also important to consider their intersections (e.g., Black women, Latinx nonbinary people, etc.).\n", + "\n", + "**In the health care scenario**, based on the previous work, we focus on groups defined by **RACE**." + ], + "metadata": { + "id": "aqqUk1mjPnpM" + } + }, + { + "cell_type": "markdown", + "source": [ + "### 3. Quantify harms\n", + "\n", + "Define metrics that quantify harms or benefits:\n", + "\n", + "* In job screening scenario, we need to quantify the number of candidates that are classified as \"negative\" (not recommended for the job), but whose true label is \"positive\" (they are qualified). One possible metric is the **false negative rate**: fraction of qualified candidates that are screened out.\n", + "\n", + "* In speech-to-text scenario, the harm could be measured by **word error rate**, number of mistakes in a transcript divided by the overall number of words.\n", + "\n", + "* **In the health care scenario**, we could consider two metrics for quantifying harms / benefits:\n", + " * **false negative rate**: fraction of patients that are readmitted within 30 days, but that are not recommended for the care management program; this quantifies harm\n", + " * **selection rate**: overall fraction of patients that are recommended for the care management program (regardless of whether they are readmittted with 30 days or no); this quantifies benefit; here the assumption is that all patients benefit similarly from the extra care.\n", + "\n", + "There are several reasons for including selection rate in addition to false negative rate. We would like to monitor how the benefits are allocated, focusing on groups that might be disadvantaged. Another reason is to get extra robustness in our assessement, because our measure (i.e., readmission within 30 days) is only an imperfect measure of our construct (who is most likely to benefit from the care management program). The auxiliary metrics, like selection rate, may alert us to large disparities in how the benefit is allocated, and allow us to catch issues that we might have missed.\n" + ], + "metadata": { + "id": "nmvSqI3dPrVk" + } + }, + { + "cell_type": "markdown", + "source": [ + "### 4. Compare quantified harms across the groups\n", + "\n", + "The workhorse of fairness assessment are _disaggregated metrics_, which are **metrics evaluated on slices of data**. For example, to measure harms due to errors, we would begin by evaluating the errors on each slice of the data that corresponds to a group we identified in Step 2.\n", + "If some of the groups are seeing much larger errors than other groups, we would flag this as a fairness harm.\n", + "\n", + "To summarize the disparities in errors (or other metrics), we may want to report quantities such as the **difference** or **ratio** of the metric values between the best and the worst slice. In settings where the goal is to guarantee certain minimum quality of service (such as speech recognition), it is also meaningful to report the **worst performance** across all considered groups.\n", + "\n" + ], + "metadata": { + "id": "fpJXt6miPvRX" + } + }, + { + "cell_type": "markdown", + "source": [ + "For example, when comparing false negative rate across groups defined by race, we may summarize our findings with a table like the following:\n", + "\n", + "| | false negative rate
(FNR) |\n", + "|---|---|\n", + "| AfricanAmerican | 0.43 |\n", + "| Caucasian | 0.44 |\n", + "| Other | 0.52 |\n", + "| Unknown | 0.67 |\n", + "| | |\n", + "|_largest difference_| 0.24   (best is 0.0)|\n", + "|_smallest ratio_| 0.64   (best is 1.0)|\n", + "|_maximum_
_(worst-case) FNR_|0.67|" + ], + "metadata": { + "id": "7Is_zdXvnW0s" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Fairness assessment with `MetricFrame`" + ], + "metadata": { + "id": "9CjHlopBDgSG" + } + }, + { + "cell_type": "markdown", + "source": [ + "Fairlearn provides the data structure called `MetricFrame` to enable evaluation of disaggregated metrics. We will show how to use a `MetricFrame` object to assess the trained `LogisticRegression` classifier for potential fairness-related harms.\n", + "\n" + ], + "metadata": { + "id": "epJO2baHV2Dy" + } + }, + { + "cell_type": "code", + "execution_count": 52, + "source": [ + "# In its simplest form MetricFrame takes four arguments:\r\n", + "# metric_function with signature metric_function(y_true, y_pred)\r\n", + "# y_true: array of labels\r\n", + "# y_pred: array of predictions\r\n", + "# sensitive_features: array of sensitive feature values\r\n", + "\r\n", + "mf1 = MetricFrame(metrics=false_negative_rate,\r\n", + " y_true=Y_test,\r\n", + " y_pred=Y_pred,\r\n", + " sensitive_features=df_test['race'])\r\n", + "\r\n", + "# The disaggregated metrics are stored in a pandas Series mf1.by_group:\r\n", + "\r\n", + "mf1.by_group" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "race\n", + "AfricanAmerican 0.428\n", + "Caucasian 0.442\n", + "Other 0.523\n", + "Unknown 0.670\n", + "Name: false_negative_rate, dtype: object" + ] + }, + "metadata": {}, + "execution_count": 52 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 297 - }, - "id": "FJOwW9db3wKe", - "outputId": "a34b7c03-6bfc-4843-c67e-bc5997ebf86c" - }, - "outputs": [], - "source": [ - "plot_technique_comparison(test_dict, \"balanced_accuracy\")" - ] + "id": "0iiAYRvoduPh", + "outputId": "3287b55e-3610-424d-a99d-a6e19c2b1693" + } + }, + { + "cell_type": "code", + "execution_count": 53, + "source": [ + "# The largest difference, smallest ratio and worst-case performance are accessed as\r\n", + "# mf1.difference(), mf1.ratio(), mf1.group_max()\r\n", + "\r\n", + "print(f\"difference: {mf1.difference():.3}\\n\"\r\n", + " f\"ratio: {mf1.ratio():.3}\\n\"\r\n", + " f\"max across groups: {mf1.group_max():.3}\")" + ], + "outputs": [ + { + "output_type": "stream", + "name": "stdout", + "text": [ + "difference: 0.242\n", + "ratio: 0.639\n", + "max across groups: 0.67\n" + ] + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 430 - }, - "id": "dACpOpm67K-m", - "outputId": "65a6de47-9ca4-49d8-e97a-16ef1db40c02" - }, - "outputs": [], - "source": [ - "plot_technique_comparison(test_dict, \"selection_rate\")" - ] + "id": "5tl1Qxt1fT2v", + "outputId": "e2af1585-7602-4664-f396-2f0d395d4ad2" + } + }, + { + "cell_type": "code", + "execution_count": 54, + "source": [ + "# You can also evaluate multiple metrics by providing a dictionary\r\n", + "\r\n", + "metrics_dict = {\r\n", + " \"selection_rate\": selection_rate,\r\n", + " \"false_negative_rate\": false_negative_rate,\r\n", + " \"balanced_accuracy\": balanced_accuracy_score,\r\n", + "}\r\n", + "\r\n", + "metricframe_unmitigated = MetricFrame(metrics=metrics_dict,\r\n", + " y_true=Y_test,\r\n", + " y_pred=Y_pred,\r\n", + " sensitive_features=df_test['race'])\r\n", + "\r\n", + "# The disaggregated metrics are then stored in a pandas DataFrame:\r\n", + "\r\n", + "metricframe_unmitigated.by_group" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
selection_ratefalse_negative_ratebalanced_accuracy
race
AfricanAmerican0.4000.4280.597
Caucasian0.3910.4420.594
Other0.3280.5230.584
Unknown0.2640.6700.536
\n", + "
" + ], + "text/plain": [ + " selection_rate false_negative_rate balanced_accuracy\n", + "race \n", + "AfricanAmerican 0.400 0.428 0.597\n", + "Caucasian 0.391 0.442 0.594\n", + "Other 0.328 0.523 0.584\n", + "Unknown 0.264 0.670 0.536" + ] + }, + "metadata": {}, + "execution_count": 54 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 203 }, - { - "cell_type": "markdown", - "metadata": { - "id": "OYkgAggpW11N" - }, - "source": [ - "\n", - "\n", - "#### Model performance - overall" - ] + "id": "R2zmBHo5gk-F", + "outputId": "e970f25b-7218-4430-8969-d8f891ddce27" + } + }, + { + "cell_type": "code", + "execution_count": 55, + "source": [ + "# The largest difference, smallest ratio, and the maximum and minimum values\r\n", + "# across the groups are then all pandas Series, for example:\r\n", + "\r\n", + "metricframe_unmitigated.difference()" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "selection_rate 0.137\n", + "false_negative_rate 0.242\n", + "balanced_accuracy 0.061\n", + "dtype: object" + ] + }, + "metadata": {}, + "execution_count": 55 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "w3Rwe98m6Pv2" - }, - "outputs": [], - "source": [ - "overall_df = pd.DataFrame.from_dict({\n", - " \"Unmitigated\": metricframe_unmitigated.overall,\n", - " \"Postprocessing\": metricframe_postprocess.overall,\n", - " \"Postprocessing (DET)\": mf_deterministic.overall,\n", - " \"Reductions\": metricframe_reductions.overall\n", - "})" - ] + "id": "Hc29jRJrhlSC", + "outputId": "e2ffee50-2764-434f-d322-2a8fdafe6a09" + } + }, + { + "cell_type": "code", + "execution_count": 56, + "source": [ + "# You'll probably want to view them transposed:\r\n", + "\r\n", + "pd.DataFrame({'difference': metricframe_unmitigated.difference(),\r\n", + " 'ratio': metricframe_unmitigated.ratio(),\r\n", + " 'group_min': metricframe_unmitigated.group_min(),\r\n", + " 'group_max': metricframe_unmitigated.group_max()}).T" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
selection_ratefalse_negative_ratebalanced_accuracy
difference0.1370.2420.061
ratio0.6590.6390.897
group_min0.2640.4280.536
group_max0.4000.6700.597
\n", + "
" + ], + "text/plain": [ + " selection_rate false_negative_rate balanced_accuracy\n", + "difference 0.137 0.242 0.061\n", + "ratio 0.659 0.639 0.897\n", + "group_min 0.264 0.428 0.536\n", + "group_max 0.400 0.670 0.597" + ] + }, + "metadata": {}, + "execution_count": 56 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 172 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 172 - }, - "id": "44Q-lYTa6PzX", - "outputId": "376ce1db-ead9-4984-83c6-7e59e815a1b2" - }, - "outputs": [], - "source": [ - "overall_df.T" - ] + "id": "bVbjFa4Aig9Y", + "outputId": "228f2de1-4de8-4059-b166-c75550e52de2" + } + }, + { + "cell_type": "code", + "execution_count": 57, + "source": [ + "# You can also easily plot all of the metrics using DataFrame plotting capabilities\r\n", + "\r\n", + "metricframe_unmitigated.by_group.plot.bar(subplots=True, layout= [1,3], figsize=(12, 4),\r\n", + " legend=False, rot=-45, position=1.5);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAt4AAAFNCAYAAAA3jmgvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABV1klEQVR4nO3dd1gUV/s38C8dEQXLgr0nIgo2jOhjNBZEkWZXsHeNoiYasUVETdQoWB4rSTQqJppEQYwilugTRVFiItglxq5URToLnPcPX/YXYtml7eyy38915bqc2RnmPtm9Z+6dPXOOnhBCgIiIiIiIypW+1AEQEREREekCFt5ERERERGrAwpuIiIiISA1YeBMRERERqQELbyIiIiIiNWDhTURERESkBiy8iUjnLVmyBD169EBgYOAbX4+KioKrq6uaoypbMTEx+PzzzwEAsbGx8PHx0Zh4iIqrJDnZvHlzpKSklFNEyn3zzTfw9fWV7PikGVh4a7mNGzfC39+/xPsvWrQIV69eBQAsXLgQkZGRZRVaifBiTFLYt28f9u7di9mzZ0sdSrmJi4tDfHw8AMDOzg4bNmzQmHiIiHSFodQBkLQiIyMxdOhQAMCKFSskjoYXY1I/Ly8vCCEwceJE9OnTB7/99htyc3ORkpICT09PzJo1q8j20dHRWLlyJQoKCgAAkydPhrOzM3Jzc7FmzRpcunQJ+fn5sLW1xaJFi2Bubv7WY0dFRSEwMBD169fHnTt3kJubi88//xyOjo7v/HsxMTHw8/ODXC5HgwYN8OTJE/j6+qJDhw744osvcOXKFWRkZEAIgeXLl6NOnTrYsGED0tLSMH/+fHh6emLZsmX4/vvv0a1bNxw7dgwymQwAMGTIEHz88cfo1KlTidqzYsUKmJmZITMzEz/99BNWr16tNJ4vv/wSp06dwpYtWyCXy2Fqaop58+ahbdu2pX+DqcLKzMyEj48P7t+/j6pVqypuQvn7+yMzMxMJCQmwsbHBunXrYGJiUmQ/Pz8/3Lt3D6mpqahcuTLWrFmDJk2aYOTIkWjTpg0uX76Mp0+fon379li1ahX09fXx66+/Yt26dSgoKICZmRmWLl0KGxsbXL58GWvWrEFWVhb09PQwY8YMdO/eHXK5HMuXL0dkZCRq1KiBGjVqoEqVKkrbtXXrVpw4cQI5OTnIysrCvHnz4OTkhLy8PHz11Vc4ffo0DAwM0LZtWyxZsgT6+vpvXL9t2zY8f/5ccTNr48aNiuWRI0fCwsICd+/exfDhw2FnZ4evvvoKubm5SExMROfOnfHFF18AwBvb/euvvyIuLg5r164FAPz+++9YtmwZQkJCyvhdroAEaZT09HQxY8YM4e7uLjw9PcXChQtFfn6+OHnypBg0aJDw8PAQQ4cOFZcvXxZCCLFhwwaxdOlSIYQQz549E9OmTRP9+/cXrq6uYsuWLYq/e+rUKeHu7i5cXV3FkCFDxI0bN0RAQIBo2bKl6N27t/jzzz/FiBEjxNGjR4UQQhw/flx4eHgIV1dXMWzYMHHlyhXF8ebNmyfGjRsnnJ2dxfDhw8WzZ8/e2aYLFy4INzc3MXToUOHm5iZycnLEsmXLxKBBg0Tfvn1Fnz59RHR0tHjy5Ino1q2baNeunfD19RVCiLe2m6gsvf/++yI5OVmMGDFC/P3330KIV/nUokULkZycLC5cuCD69esnhBBi1KhR4vDhw0IIIW7cuCH8/PyEEEJs3LhRrFy5UhQUFAghhFi7dq1YsmTJO4974cIF0aJFC3H9+nUhhBDffPON8Pb2fuffk8vlomvXruL06dNCCCHOnz8vmjdvLi5cuCAuX74sZsyYIfLz84UQQmzbtk1MnjxZCCHEzz//LCZNmqQ4bmF7PvvsM/H1118LIYSIi4sTH330kcjPzy9xe2xsbMSjR4+EEELleP7++2/h6uoqUlJShBBC3L59W/znP/8RGRkZ7zwe6a7Cz9rvv/8uhBDihx9+EIMGDRIrV64UISEhQgghcnNzhaurqwgPDxdC/F+eHz16VCxbtkzxtxYvXiz8/f2FEEKMGDFC+Pj4iPz8fJGWlia6dOkizp8/LxITE0X79u0VuXrs2DExfvx48eLFC9G7d2/x8OFDIcSr80bXrl3F48ePxc6dO8WoUaNETk6OyMjIEP379xfz5s17Z7sePXokRo4cKbKysoQQQhw+fFi4uroKIYT47rvvhLe3t8jKyhL5+fli5syZ4uDBg29d/8/6QIii9cKIESPE/PnzFa/Nnj1bXLhwQQjxqg7p2LGjiI2NfWu7k5KSRLt27cTz58+FEELMnTtXfP/998V6D3UV73hrmOPHjyMjIwOhoaHIz8/HkiVL8ODBAwQGBmLXrl2oVq0a7ty5g7FjxyIiIqLIvnPnzsWYMWPQo0cP5OTkYOLEiWjQoAE++OADzJ07F7t370aLFi0QERGBNWvW4Ouvv0ZYWBjWrFkDOzs7xd/566+/sGTJEvzwww+oX78+zp8/j2nTpiE8PBzAqzt+ISEhMDc3x5QpU7Bv3z6l/UXv3LmDEydOoG7duvjjjz+QkJCAffv2QV9fH9u3b0dQUBC2bt0KHx8fHDt2DF9++SXu3bv31nabmZmV/f980nlbt27F6dOncfjwYfz1118QQiArK6vINn379oW/vz9OnTqFzp0745NPPgEAnD59GmlpaYruWnK5HDVq1FB6zDp16qBFixYAAFtbWxw8ePCdf+/27dsAgG7dugEAHB0d8d577wEA2rZtCwsLC/zwww94+PAhoqKiULly5Xcef/DgwVi6dCnGjx+Pn3/+GQMGDIC+vn6J21O7dm3UrVu3WPGcO3cOCQkJGDNmjGKdnp4eHjx4ABsbG6XHJN3UvHlztGvXDgDQv39/+Pn54dtvv8Wff/6JoKAg3Lt3DwkJCcjMzCyyX58+fVC/fn3s3r0b9+/fx8WLF4v8utK9e3fo6+vD3NwcDRs2RGpqKi5fvoz33ntPkau9e/dG7969cebMGSQmJuLjjz9W7K+np4dbt27h/PnzcHV1hbGxMYyNjeHm5oZbt269s01169bFqlWrEBYWhvv37yt+LQJe/ULt4eEBU1NTAMC6desAAFOmTHnj+o0bN77zWA4ODop/r1y5Ev/73/+wdetW3L17F9nZ2cjMzHxruwHgo48+QmhoKDw9PXH27FksWbLkncejV1h4a5j27dsjMDAQI0eOROfOnTF69Oh3XpQKZWZm4tKlS0hNTcX69esV627evAlDQ8O3Js6bXLhwAY6Ojqhfvz4AoFOnTqhevbqiL/gHH3yg+LnZ1tYWqampStvFizFpuqysLAwbNgy9evWCg4MDBg4ciBMnTkAIUWS7YcOGoXv37jh37hx+++03/Pe//8WhQ4dQUFCABQsWKArijIwM5OTkKD1u4cUSePX5Ljze2/5eYmLiazEZGBgAeFWsr1ixAmPHjkXPnj3RpEkTHDp06J3Hd3BwQF5eHmJiYnD48GH88MMP7zy+Mv/8UqxqPAUFBejUqZOiYACAp0+fwsrKSunxSHfp6xd9TE1PTw8LFy6EEAJ9+/bFRx99hKdPn76WL3v37sX+/fvh7e0NNzc3WFpa4tGjR4rX35SThoaG0NPTU6wXQuDWrVvIz89H06ZN8eOPPypei4+PR/Xq1bFv374ixy3M03e5du0apk2bhjFjxuA///kPOnTogKVLlwIADA2LlmxJSUkoKCh46/p/nk+AV1+e/+mfuert7Q0bGxt8+OGH6Nu3L65cuQIhBAwMDN7YbhsbG3h7e8PPzw+Ghobo3bu30i/59AofrtQw9evXx/HjxzFp0iSkp6dj7NixePHiBTp16oTQ0FDFf/v371fc5QJeXbiEEPjhhx8U2+zbtw+TJ09+Y+LcvHnzrTH8+yRVuC4vLw/A2wuFd/n3xXjy5MkAgJ49e2L48OFv3KfwYvyudhOVlZSUFKSnp2PWrFno0aMHLl68iNzcXEVf7kLDhg3DjRs3MGDAACxbtgwvX75EamoqunTpguDgYMU+ixcvRkBAQInjedvfa9q0KYyNjfG///0PwKsHkm/fvg09PT2cO3cO3bt3h5eXF+zs7HDixAnk5+cDeHXRL8zhfxs8eDCWLVuG5s2bo06dOu88fnGoGo+joyPOnTuHv/76CwBw5swZuLu7q1Tok+66desWbty4AeDVA9Lt27dHZGQkPv74Y7i4uEBPTw9XrlxRfOYKnT17Fv3798fgwYPRuHFjnDp16rVt/q1169b466+/cOfOHQDAyZMnMXfuXLRp0wb379/HpUuXAAA3btyAs7MzEhIS8OGHHyIkJAQ5OTnIycnBkSNHlLbp0qVLaNWqFcaOHYsPPvgAJ0+eVMTWqVMnHD58WJGTfn5++OWXX966vlq1arh27RqEEMjMzMTZs2ffeMzU1FRcvXoVc+bMQe/evREfH48HDx6goKDgre0GgHbt2kFfXx/ffPPNW6/j9Dre8dYwe/fuxe+//441a9bgww8/RHJysuInq7/++gtNmzbFmTNnMGfOHJw5c0axn7m5Odq0aYMdO3Zg2rRpePnyJYYPH46PP/4YnTt3ViTOe++9h5MnT2L9+vUICwt748XY0dER//3vf/Hw4UNFV5OnT5+idevW+OOPP0rdxn9ejHNychAUFPTWi/GGDRve2G52NaGyVqdOHXz00Ufo27cvqlatigYNGqBZs2a4f/8+jI2NFdvNmTMHX3zxBdatWwd9fX1Mnz4d9erVw7Rp07Bq1Sr0798f+fn5aNGiRamGDnvb3zM0NMTGjRuxZMkSBAQEoFGjRqhZsyZMTU0xbNgwzJkzB25ubjAwMICDgwMiIiJQUFCAtm3bYt26dfj4448xatSoIsfy9PREQEBAkcK6LNqjajybNm2Cv78/PvnkE8XdxS1btjDP6Z2aNGmiuFbVqFEDK1euxOnTp/Hxxx/DwsIClSpVQocOHYr8OgwA48aNw+eff44DBw7AwMAALVu2VHThepuaNWtizZo1mDdvHvLz82Fubo7AwEBUr14dGzZswOrVq5GTkwMhBFavXo26deti2LBhePDgAVxdXWFpaYmGDRsqbZOrqysiIiLg4uICIyMjdOrUCampqUhPT8ewYcPw+PFjDBgwAEIIfPDBBxg5ciT09PTeuD4rKwu//fYbevfuDWtra7Rt2/aNN8osLCwwadIk9O/fH5aWlqhWrRratWuH+/fvKx6y/ne7Cw0YMABHjhxB8+bNVXzXSE+ocruS1CYzMxMLFizArVu3UKlSJdSpUwcrVqxAZGQktm7dqrgoLViwAA4ODkWeUn706BGWLVuGJ0+eIDc3F66urpgxYwYA4LfffkNgYKAicZYuXYpmzZph1apVOHr0KJYvX45t27bB29sbffr0wdGjR7F161bk5+fD1NQUvr6+rx0PwGvLbxIVFYVly5bh8OHDAF71IZ8zZw7y8vKKXIxPnz6NR48eYfTo0bC1tcWmTZsUcfy73US6bNWqVRg/fjxq1qyJp0+fwsPDAydOnEDVqlWlDo2IdEReXh6mT58Od3d3uLi4SB2O1mDhTURUjmbNmoW///77ja8FBgaiSZMmxf6be/bswQ8//ABDQ0MIIfDxxx+/87mNslQe7SHSJV988QWioqLe+Nr8+fPh6Oio5oiKLy4uDsOHD0fXrl3x1Vdfvdbfnt6OhTeVCV6MiYiIiN6NhTeRjggLC1NMUDJmzBh4e3srXrtx40aR/rspKSmwsLBQdA8iIiKi0mPhTaQD4uPjMXz4cBw4cADGxsYYNmwYAgIC0KxZs9e2zcrKwuDBg+Hn58f+9ERERGWInXKIdEBkZCQcHR1haWkJMzMzODs7KyZE+rdt27ahQ4cOLLqJiIjKmEYPJ/j8eQYKCrT7hnyNGuZITk6XOoxSYRukp6+vh2rVSj45QUJCAmQymWLZysoKMTExr2338uVL7N+/H2FhYSU+FhFpB22/xmr7eR2oGG0AtL8dpb3GFodGF94FBUKrTwqF2AbNUBHaUFJv6lH2z0mVCoWFhaFXr14qTQ3+JsnJ6Vr9/1kmq4LExDSpwygVtkEz6OvroUYNc6nDeKeKcI3V9viBitEGoOK0o7yxqwmRDrC2tkZSUpJiOSEh4Y3TcZ84cYLjsRIREZUTFt5EOqBz5844f/48UlJSkJWVhYiICHTt2rXINkIIXLt2DW3btpUoSiIiooqNhTeRDrC2tsbs2bMxatQoeHp6wtXVFfb29pg4cSJiY2MBvBpC0MjICCYmJhJHS0REVDFpdB9vIio7bm5ucHNzK7IuKChI8e8aNWrg3Llz6g6LiIhIZ/CONxERkQYKCwuDi4sLnJycEBwc/Nrrd+/exciRI+Hu7o7x48cjNTVVgiiJqDhUKryVJX+h06dPo0ePHorlly9fYtKkSejbty+8vb2RmJhY+oiJiIgquPj4eAQGBmLv3r0IDQ3Fvn37EBcXp3hdCIGpU6di4sSJOHToEFq0aIHt27dLGDERqUJp4a0s+QslJSVh1apVRdatW7cODg4OOHr0KAYPHowVK1aUXeREREQVlLJJr65duwYzMzPFQ9JTpkyBt7e3VOESkYqU9vH+Z/IDUCT/9OnTi2y3aNEiTJ8+HWvXrlWsO336tOIOuaurK/z9/SGXy2FkZFSGTSAiIqpYlE169eDBA9SsWRPz5s3D9evX8f7772Px4sXFOoamjzOuCpmsitQhlFpFaANQcdpR3pQW3qrMeLdr1y7Y2tqidevWb93X0NAQ5ubmSElJgbW1tUrBFfekkCvPh7GRQbH2UcffrwgfRraBiDRZNQtjGBqX34g8ebk5eJ6aW25//9+UTXqVl5eHixcvYs+ePbCzs8O6deuwcuVKrFy5UuVjFGfCqyqWJjA1Mlb5b5dEtjwXaS9yVN6+Iky0VBHaAGh/O9Q54ZXSwltZ8t++fRsRERHYuXMnnj17pvSA+vqqP89Z3FnwZLIqcPs0VOXtiytsrUexP1ja/mEE2AZNoA2z4BFJydDYBHdXDCy3v99k4c8A1Fd4W1tbIzo6WrH870mvZDIZGjZsCDs7OwCvflX28fEpt3hMjYwxZN/Ucvv7ALB/6BakQfXCuyTK+wtEcb88kO5RWngrS/7w8HAkJiZi4MCBkMvlSEhIgJeXF/bu3QsrKyskJSWhVq1ayMvLQ3p6uqLLChEREb1Z586dsXHjRqSkpKBSpUqIiIjAsmXLFK+3bdsWKSkpuHnzJmxsbHDq1Cm0bNlSwoi1Q3l/gVDHlwfSbkpvPyub8c7HxwfHjh1DaGgotm/fDisrK+zduxcA0K1bN4SEhAAAjhw5AgcHB/bvJiIiUkLZpFempqbYtGkTFi1ahH79+iEqKgq+vr5Sh01ESqh0x7sw+eVyOQYNGqRIfh8fH8XPXG8yc+ZM+Pr6ol+/fqhSpQrWrFlTpsETERFVVMomvWrdujV++ukndYdFRKWg0syVypK/UL169XDq1CnFsqWlJbZu3VrKEHVLlaqVYGpSfhOKZufkIe1lVrn9fSIiIio/7Keu3ThlvIYxNTEs9wdEtfcRQyIiIt3GfurajVPGExERERGpAQtvIiIiIiI1YOFNRERERKQGLLyJiIiIiNSAhTcRERERkRqw8CYiIiIiUgMW3kREREREasDCm4iIiIhIDVh4ExERERGpAQtvIiIiIiI1YOFNRERERKQGLLyJdERYWBhcXFzg5OSE4ODg116/e/cuRo4cCXd3d4wfPx6pqakSRElERFRxsfAm0gHx8fEIDAzE3r17ERoain379iEuLk7xuhACU6dOxcSJE3Ho0CG0aNEC27dvlzBiIiKiioeFN5EOiIyMhKOjIywtLWFmZgZnZ2eEh4crXr927RrMzMzQtWtXAMCUKVPg7e0tVbhEREQVkqHUARBR+UtISIBMJlMsW1lZISYmRrH84MED1KxZE/PmzcP169fx/vvvY/HixcU+To0a5mUSr5RksipSh1BqbEP50MSYiLRRFUsTmBoZl+sxsuW5SHuRU67HKAkW3kQ6QAjx2jo9PT3Fv/Py8nDx4kXs2bMHdnZ2WLduHVauXImVK1cW6zjJyekoKHj9WNpCJquCxMQ0qcMoFV1tgzqK4uLEpK+vVyG+iBKVB1MjYwzZN7Vcj7F/6BakQfMKb5W6mih7KOv48eNwc3NDv3794Ovri9zcXABASEgIunTpAg8PD3h4eCAwMLBsoycilVhbWyMpKUmxnJCQACsrK8WyTCZDw4YNYWdnBwBwdXUtckeciIiISk9p4a3soazMzEz4+/tjx44d+OWXX5CTk4ODBw8CAGJjY+Hr64vQ0FCEhoZi9uzZ5dcSInqrzp074/z580hJSUFWVhYiIiIU/bkBoG3btkhJScHNmzcBAKdOnULLli2lCpeIiKhCUlp4K3soy8zMDKdOnULNmjWRmZmJ5ORkVK1aFcCrwjskJATu7u6YM2cOhycjkoi1tTVmz56NUaNGwdPTE66urrC3t8fEiRMRGxsLU1NTbNq0CYsWLUK/fv0QFRUFX19fqcMmIiKqUJT28Vb2UBYAGBkZ4cyZM/jss89gZWWFLl26AHj18/WkSZNgb2+PgIAA+Pv7Y+3atSoHp4n940rSj1DTHsipCG0oiYrQhtJwc3ODm5tbkXVBQUGKf7du3Ro//fSTusMiIiLSGUoLb2UPZRXq1q0boqKiEBAQAD8/P6xduxabNm1SvD5hwgT06tWrWMEV90EtTXu4Bij+Q0IVoQ2aSNvbwAe1iIiItJ/SribKHsp68eIFzp49q1h2c3PDrVu3kJaWhp07dyrWCyFgaMhBVIiIiIhINyktvJU9lCWEwNy5c/HkyRMAwNGjR9GuXTuYmZnh66+/xpUrVwAAe/bsgZOTUzk1g4iIiIhIsym9Bf3Ph7LkcjkGDRqkeCjLx8cHdnZ2WLZsGSZPngw9PT00a9YMS5cuhYGBAdatWwc/Pz9kZ2ejUaNGWL16tTraREREpPXCwsKwZcsWyOVyjBkz5rXZZP/73//i559/VgxoMGTIEM44S6ThVOr7oeyhrF69er2x/7aDg4NiaEEiIiJSTeFQvgcOHICxsTGGDRuGjh07olmzZoptrl69ioCAALRt21bCSImoOFSaQIeIiIjUR9lQvsCrwjsoKAhubm7w9/dHTo7mzdJHREXxaUciIiINo2wo34yMDLRo0QLz5s1D3bp14evri82bNxdrojpNHCmpuCN7aeIwsbrYhpLuU940MSYW3kRERBpG2VC+lStXLtLlc9y4cViwYEGxCu/iDNmrrgKmuMPvlmR43PKma20ANHPoZED1dqhzyF52NSEiItIwyobyffLkSZEJrzhkL5F2YOFNRESkYZQN5WtqaoqvvvoKDx8+hBACwcHBHLKXSAvw6zGVuSpVK8HUpHw/Wtk5eUh7mVWuxyAikooqQ/n6+/tj6tSpkMvlaNeuHcaOHSt12ESkBAtvKnOmJoZw+zS0XI8RttYD2jsBPBGRcsqG8nV2doazs7O6wyKiUmBXEyIiIiIiNWDhTURERESkBiy8iYiIiIjUgIU3EREREZEasPAmIiIiIlIDFt5ERERERGrA4QSJiHRYNQtjGBqblOsx8nJz8Dw1t1yPQUSkDVh4ExHpMENjE9xdMbBcj9Fk4c8AWHgTEbGrCRERERGRGrDwJiIiIiJSA5UK77CwMLi4uMDJyQnBwcGvvX78+HG4ubmhX79+8PX1RW7uq58Unzx5Am9vb/Tp0wdTp05FRkZG2UZPRCpTlsf//e9/0b17d3h4eMDDw+ON2xAREVHJKS284+PjERgYiL179yI0NBT79u1DXFyc4vXMzEz4+/tjx44d+OWXX5CTk4ODBw8CAJYuXQovLy+Eh4ejVatW2Lx5c/m1hIjeSlkeA8DVq1cREBCA0NBQhIaGwtvbW6JoiYiIKialhXdkZCQcHR1haWkJMzMzODs7Izw8XPG6mZkZTp06hZo1ayIzMxPJycmoWrUq5HI5Ll26BGdnZwDAgAEDiuxHROqjLI+BV4V3UFAQ3Nzc4O/vj5ycHImiJSIiqpiUjmqSkJAAmUymWLayskJMTEyRbYyMjHDmzBl89tlnsLKyQpcuXfD8+XOYm5vD0PDVIWQyGeLj44sVXI0a5sXaXh1ksipq2ac8VYQ2AMWPSRPboC7K8jgjIwMtWrTAvHnzULduXfj6+mLz5s2YPXt2sY6jiTlbXBXhc6KJbagI+aqJMRGRdlFaeAshXlunp6f32rpu3bohKioKAQEB8PPzw2effabSfu+SnJyOgoLXj/826jgpJiamFWt7maxKsfZhG1RX3JiK225Noq+vV6qiVlkeV65cGUFBQYrlcePGYcGCBcUuvIubs5pG2z8ngO7mq6adO0ubs0RUMSntamJtbY2kpCTFckJCAqysrBTLL168wNmzZxXLbm5uuHXrFqpXr4709HTk5+cDABITE4vsR0TqoyyPnzx5gp9++kmxLIRQ/FpFREREZUNp4d25c2ecP38eKSkpyMrKQkREBLp27ap4XQiBuXPn4smTJwCAo0ePol27djAyMoKDgwOOHDkCAAgJCSmyHxGpj7I8NjU1xVdffYWHDx9CCIHg4GA4OTlJGDEREVHFo9Id79mzZ2PUqFHw9PSEq6sr7O3tMXHiRMTGxqJatWpYtmwZJk+eDHd3d9y7dw9z584FACxZsgT79++Hi4sLoqOjMWvWrPJuDxG9gbI8rl69Ovz9/TF16lT06dMHQgiMHTtW6rCJiIgqFJV+S3Zzc4Obm1uRdf/sD9qrVy/06tXrtf3q1q2L3bt3lzJEIioLyvLY2dlZMQoRERERlT3OXElEREREpAYsvImIiIiI1ICFNxERERGRGrDwJiIiIiJSAxbeRERERERqwMKbiIiIiEgNWHgTEREREakBC28iIiINExYWBhcXFzg5OSE4OPit250+fRo9evRQY2REVBoqTaBDRERE6hEfH4/AwEAcOHAAxsbGGDZsGDp27IhmzZoV2S4pKQmrVq2SKEoiKgne8SYiItIgkZGRcHR0hKWlJczMzODs7Izw8PDXtlu0aBGmT58uQYREVFK8401ERKRBEhISIJPJFMtWVlaIiYkpss2uXbtga2uL1q1bl/g4NWqYl3jf8iKTVSnX7dVBF9tQ0n3KmybGxMKbiIhIgwghXlunp6en+Pft27cRERGBnTt34tmzZyU+TnJyOgoKXj/Wm6irgElMTFN5W5msSrG2L9ynvOlaG4Dit0PTPk/6+npq+yLKwpvoDapUrQRTk/JNj+ycPKS9zCrXY1D5qmZhDENjk3I9Rl5uDp6n5pbrMUizWFtbIzo6WrGckJAAKysrxXJ4eDgSExMxcOBAyOVyJCQkwMvLC3v37pUiXCIqBhbeRG9gamIIt09Dy/UYYWs9ULx7CqRpDI1NcHfFwHI9RpOFPwNg4a1LOnfujI0bNyIlJQWVKlVCREQEli1bpnjdx8cHPj4+AIBHjx5h1KhRLLqJtAQfriQiItIg1tbWmD17NkaNGgVPT0+4urrC3t4eEydORGxsrNThEVEp8I43ERGRhnFzc4Obm1uRdUFBQa9tV69ePZw6dUpdYRFRKal0x1vZQP4nTpyAh4cH3N3dMW3aNKSmpgIAQkJC0KVLF3h4eMDDwwOBgYFlGz0RERERkZZQesdb2UD+6enp8PPzw88//wxra2usX78eGzduxKJFixAbGwtfX1+4urqWe0OISLvwwUQiItI1Sgvvfw7kD0AxkH/hoP1yuRx+fn6wtrYGADRv3hxhYWEAgNjYWNy/fx/bt2/H+++/j8WLF8PCwqKcmkJE2oQPJhIRka5R2tXkTQP5x8fHK5arVauGXr16AQCys7Oxfft2xbJMJsOMGTMQGhqK2rVrw9/fv6zjJyIiIiLSCkrveCsbyL9QWloapk2bBhsbG/Tv3x8AsGnTJsXrEyZMUBTkqqoIs2qVdJ/yVBHaAOju7GBERESknZQW3soG8i9cN378eDg6OmLBggUAXhXiP//8M8aMGQPgVQFvaFi8QVSKM6sWwNmcVFUR2gCU7+xgmtYGdc6qRUREROVDaVeTzp074/z580hJSUFWVhYiIiLQtWtXxev5+fmYMmUK+vbti4ULFyruhpuZmeHrr7/GlStXAAB79uyBk5NTOTWDiJRRNjpRodOnT6NHjx5qjIyIiEg3qHTHu3Agf7lcjkGDBikG8vfx8cGzZ89w/fp15Ofn49ixYwCAVq1aYcWKFVi3bh38/PyQnZ2NRo0aYfXq1eXeICJ6nbLRiQolJSVh1apVEkVJRERUsanU9+NdA/nb2dnh5s2bb9zPwcEBBw8eLGWIRFRaykYnKrRo0SJMnz4da9eulSBKIiKiio0zVxLpgDeNThQTE1Nkm127dsHW1hatW7cu8XE0sR+6Lj6EyzaUD02MiYi0CwtvIh2gbHSi27dvIyIiAjt37sSzZ89KfJziPBCtaQ+wAtr/EC6gm20o3Ke8FScmPhBNRG+i0pTxRKTdrK2tkZSUpFj+9+hE4eHhSExMxMCBAzFp0iQkJCTAy8tLilCJiIgqLBbeRDpA2ehEPj4+OHbsGEJDQ7F9+3ZYWVlh7969EkZMRERU8bDwJtIB/xydyNPTE66urorRiWJjY6UOj4iISCewjzeRjnjX6ET/VK9ePZw6dUpdYREREekM3vEmIiIiIlIDFt5ERERERGrAwpuIiIiISA1YeBMRERERqQELbyIiIiIiNWDhTURERESkBiy8iYiIiIjUgIU3EREREZEasPAmIiIiIlIDFt5ERERERGrAwpuIiIiISA1UKrzDwsLg4uICJycnBAcHv/b6iRMn4OHhAXd3d0ybNg2pqakAgCdPnsDb2xt9+vTB1KlTkZGRUbbRExERVVDKrr3Hjx+Hm5sb+vXrB19fX+Tm5koQJREVh9LCOz4+HoGBgdi7dy9CQ0Oxb98+xMXFKV5PT0+Hn58ftm/fjkOHDqF58+bYuHEjAGDp0qXw8vJCeHg4WrVqhc2bN5dfS4iIiCoIZdfezMxM+Pv7Y8eOHfjll1+Qk5ODgwcPShgxEalCaeEdGRkJR0dHWFpawszMDM7OzggPD1e8LpfL4efnB2trawBA8+bN8fTpU8jlcly6dAnOzs4AgAEDBhTZj4iIiN5M2bXXzMwMp06dQs2aNZGZmYnk5GRUrVpVwoiJSBWGyjZISEiATCZTLFtZWSEmJkaxXK1aNfTq1QsAkJ2dje3bt2PkyJF4/vw5zM3NYWj46hAymQzx8fHFCq5GDfNiba8OMlkVtexTnipCG4Dix1QR2kBEukHZtRcAjIyMcObMGXz22WewsrJCly5dinWMinCN1cRzqC62oaT7lDdNjElp4S2EeG2dnp7ea+vS0tIwbdo02NjYoH///m8sst+037skJ6ejoOD147+NOv4HJyamFWt7maxKsfZhG1RX3Ji0uQ36+noaeZEkovKh6rW3W7duiIqKQkBAAPz8/LB27VqVj1Gca6ymnROB4p/XC/cpb7rWBoDX2OJQ2tXE2toaSUlJiuWEhARYWVkV2SYhIQFeXl6wsbHBihUrAADVq1dHeno68vPzAQCJiYmv7UdERESvU3btffHiBc6ePatYdnNzw61bt9QaIxEVn9LCu3Pnzjh//jxSUlKQlZWFiIgIdO3aVfF6fn4+pkyZgr59+2LhwoWKb+RGRkZwcHDAkSNHAAAhISFF9iMiIqI3U3btFUJg7ty5ePLkCQDg6NGjaNeunVThEpGKlHY1sba2xuzZszFq1CjI5XIMGjQI9vb2mDhxInx8fPDs2TNcv34d+fn5OHbsGACgVatWWLFiBZYsWQJfX19s2bIFtWvXRkBAQLk3iIiISNspu/ba2dlh2bJlmDx5MvT09NCsWTMsXbpU6rCJSAmlhTfw6icsNze3IuuCgoIAAHZ2drh58+Yb96tbty52795dyhCJiIh0z7uuvQDQq1cvxeAGRKQdOHMlEREREZEasPAmIiIiIlIDFt5EOoLTTxMREUmLhTeRDuD000RERNJj4U2kAzj9NBERkfRUGtWEiLSbOqafBjgFdXlhGzSDJsZERNqFhTeRDlDH9NOA7k1BzTaoTtem0VbnFNREpD3Y1YRIB3D6aSIiIumx8CbSAZx+moiISHrsakKkAzj9NBERkfRYeBPpCE4/TUREJC12NSEiIiIiUgMW3kREREREasDCm4iIiIhIDVh4ExERERGpAQtvIiIiIiI1YOFNRERERKQGKhXeYWFhcHFxgZOTE4KDg9+63bx583DgwAHFckhICLp06QIPDw94eHggMDCw9BETEREREWkhpeN4x8fHIzAwEAcOHICxsTGGDRuGjh07olmzZkW2WbJkCc6fP4+OHTsq1sfGxsLX1xeurq7lEz0RERERkZZQesc7MjISjo6OsLS0hJmZGZydnREeHl5km7CwMPTs2RN9+/Ytsj42NhYhISFwd3fHnDlzkJqaWrbRExERERFpCaV3vBMSEiCTyRTLVlZWiImJKbLNhAkTAAC///57kfUymQyTJk2Cvb09AgIC4O/vj7Vr16ocXI0a5ipvqy4yWRW17FOeKkIbgOLHVBHaQERERNpLaeEthHhtnZ6enkp/fNOmTYp/T5gwodjTUScnp6Og4PXjv406ipjExLRibS+TVSnWPmyD6oobkza3QV9fTyO/iBIREZHqlHY1sba2RlJSkmI5ISEBVlZWSv9wWloadu7cqVgWQsDQUGmdT0RERERUISktvDt37ozz588jJSUFWVlZiIiIQNeuXZX+YTMzM3z99de4cuUKAGDPnj1wcnIqfcRERERERFpI6S1oa2trzJ49G6NGjYJcLsegQYNgb2+PiRMnwsfHB3Z2dm/cz8DAAOvWrYOfnx+ys7PRqFEjrF69uswbQERERESkDVTq++Hm5gY3N7ci64KCgl7bbuXKlUWWHRwccPDgwVKER0REpJvCwsKwZcsWyOVyjBkzBt7e3kVeP3HiBDZu3AghBOrVq4cvv/wSFhYWEkVLRKrgzJVEREQapnAOjb179yI0NBT79u1DXFyc4vX09HT4+flh+/btOHToEJo3b46NGzdKGDERqYKFNxERkYZRNoeGXC6Hn58frK2tAQDNmzfH06dPpQqXiFTEYUaIiIg0jLI5NKpVq6YYojc7Oxvbt2/HyJEji3UMTRyiVBfnZ6gIbSjpPuVNE2Ni4U1ERKRhVJ1DIy0tDdOmTYONjQ369+9frGMUZ64MTZvbACj+/AyF+5Q3XWsDwLkyioNdTYiIiDSMKnNoJCQkwMvLCzY2NlixYoW6QySiEmDhTUREpGGUzaGRn5+PKVOmoG/fvli4cKHKM0oTkbTY1YSIiEjDKJtD49mzZ7h+/Try8/Nx7NgxAECrVq1455tIw7HwJtIRHBOYSLu8aw4NOzs73Lx5U4qwiKgU2NWESAdwTGAiIiLpsfAm0gEcE5iIiEh67GpCpAPUMSYwwHGBywvboBk0MSYi0i4svIl0gDrGBAZ0b1xgtkF1uja2sTrHBSYi7cGuJkQ6gGMCExERSY+FN5EO4JjARERE0mNXEyIdwDGBiYiIpMfCm0hHcExgIiIiabGrCRERERGRGqhUeIeFhcHFxQVOTk4IDg5+63bz5s3DgQMHFMtPnjyBt7c3+vTpg6lTpyIjI6P0ERMRERERaSGlhbeyGe8Kt5kyZUqRCTkAYOnSpfDy8kJ4eDhatWqFzZs3l230RERERERaQmnhrWzGO+DVHfGePXuib9++inVyuRyXLl2Cs7MzAGDAgAGv7UdEREREpCuUPlypbMY7AJgwYQIA4Pfff1ese/78OczNzWFo+OoQMpkM8fHxxQpOEycfKMkkDZo221lFaAPAmfCIiIhIuygtvFWd8a6s9vun4syCB2jezGWAZs4iVxHaAOjWbH6cBY+IiEj7Ke1qosqMd29SvXp1pKenIz8/HwCQmJio0n5ERERERBWR0sJb2Yx3b2NkZAQHBwccOXIEABASEqLSfkREREREFZFKd7wLZ7zz9PSEq6urYsa72NjYd+67ZMkS7N+/Hy4uLoiOjsasWbPKKm4iIiIiIq2i0syV75rx7p9WrlxZZLlu3brYvXt3KcIjIiIiIqoYOHMlEREREZEasPAmIiIiIlIDFt5ERERERGrAwpuIiIiISA1YeBMRERERqQELbyIiIiIiNWDhTURERESkBiy8iYiIiIjUgIU3ERGRBgoLC4OLiwucnJwQHBz81u3mzZuHAwcOqDEyIiopFt5EREQaJj4+HoGBgdi7dy9CQ0Oxb98+xMXFvbbNlClTEB4eLlGURFRcLLyJiIg0TGRkJBwdHWFpaQkzMzM4Ozu/VmCHhYWhZ8+e6Nu3r0RRElFxGUodABERERWVkJAAmUymWLayskJMTEyRbSZMmAAA+P3330t0jBo1zEseYDmRyaqU6/bqoIttKOk+5U0TY2LhTaQjwsLCsGXLFsjlcowZMwbe3t5v3G7evHno2LEjBgwYoOYIiaiQEOK1dXp6emV6jOTkdBQUvH6cN1FXAZOYmKbytjJZlWJtX7hPedO1NgDFb4emfZ709fXU9kWUXU2IdAD7ixJpF2trayQlJSmWExISYGVlJWFERFQWWHgT6QD2FyXSLp07d8b58+eRkpKCrKwsREREoGvXrlKHRUSlxK4mRDpAHf1FAfYZLS9sg2ZQZ0zW1taYPXs2Ro0aBblcjkGDBsHe3h4TJ06Ej48P7Ozs1BYLEZUdFt5EOkAd/UUB3eszyjaoTtf6vZZFn1E3Nze4ubkVWRcUFPTaditXrizVcYhIfVTqaqJsEP8bN25g4MCBcHZ2xsKFC5GXlwcACAkJQZcuXeDh4QEPDw8EBgaWbfREpBL2FyUiIpKe0sJblYey5s6di8WLF+PYsWMQQmD//v0AgNjYWPj6+iI0NBShoaGYPXt2+bSCiN6J/UWJiIikp7TwVvZQ1uPHj5GdnY02bdoAAAYMGKB4PTY2FiEhIXB3d8ecOXOQmppaPq0gonf6Z39RT09PuLq6KvqLxsbGSh0eERGRTlDax1vZQ1n/fl0mkyE+Pl7x70mTJsHe3h4BAQHw9/fH2rVrVQ6uIjyoVdJ9ylNFaAPAh7WKi/1FiYiIpKW08Fb2UNa7Xt+0aZNi3YQJE9CrV69iBVecB7UAzXu4BtDMB50qQhsA3XrgTJ2D+xMREVH5UNrVRNlDWf9+PTExEVZWVkhLS8POnTsV64UQMDTkICpEREREpJuUFt7KHsqqW7cuTExMFGP/hoSEoGvXrjAzM8PXX3+NK1euAAD27NkDJyencmoGEREREZFmU3oLWpVB/NesWYNFixYhIyMDtra2GDVqFAwMDLBu3Tr4+fkhOzsbjRo1wurVq9XRJiIiIiIijaNS3w9lD2XZ2Njgp59+em0/BwcHHDx4sJQhEhERERFpP5Um0CEiIiIiotJh4U1EREREpAYsvImIiIiI1ICFNxERERGRGrDwJiIiIiJSAxbeRERERERqwMKbiIiIiEgNWHgTEREREakBC28iIiIiIjVg4U1EREREpAYsvImIiIiI1ICFNxERERGRGrDwJiIiIiJSAxbeRERERERqwMKbiIiIiEgNWHgTEREREakBC28iIiIiIjVQqfAOCwuDi4sLnJycEBwc/NrrN27cwMCBA+Hs7IyFCxciLy8PAPDkyRN4e3ujT58+mDp1KjIyMso2eiJSWUnzmIikwZwlqniUFt7x8fEIDAzE3r17ERoain379iEuLq7INnPnzsXixYtx7NgxCCGwf/9+AMDSpUvh5eWF8PBwtGrVCps3by6fVhDRO5Umj4lI/ZizRBWTobINIiMj4ejoCEtLSwCAs7MzwsPDMX36dADA48ePkZ2djTZt2gAABgwYgA0bNmDw4MG4dOkSNm3apFg/YsQIzJ07V+Xg9PX1itkcwKpapWLvUxwliam4+7ANqiluTNrchpK8Z/9U0jz28vIq1nGKG6ehhaxY25dEeX9O2AbVlHcbgPJvR3Fi0oacLW6MMrPqxdq+JNTxOSnvduhiG0qyjyZ9nkqbr8WhtPBOSEiATPZ/JzMrKyvExMS89XWZTIb4+Hg8f/4c5ubmMDQ0LLK+OKpVq1ys7QHgm0W9i71PcdSoYV7u+7ANqiluTBWhDSVV0jwuruLmbIPpW4t9jOIq788J26Ca8m4DUP7tUFe+AurJ2eLm6ya3FcXaviTU8Tkp73boYhtKso8mfp7UQWlXEyHEa+v09PSUvq5sPyJSn5LmMRFJgzlLVDEpLbytra2RlJSkWE5ISICVldVbX09MTISVlRWqV6+O9PR05OfnF1lPROpX0jwmImkwZ4kqJqWFd+fOnXH+/HmkpKQgKysLERER6Nq1q+L1unXrwsTEBL///jsAICQkBF27doWRkREcHBxw5MiRIuuJSP1KmsdEJA3mLFHFpCfe9HvVv4SFhWHbtm2Qy+UYNGgQJk6ciIkTJ8LHxwd2dna4efMmFi1ahIyMDNja2uLLL7+EsbExHj9+DF9fXyQnJ6N27doICAiAhYWFOtpFRP9S0jwmImkwZ4kqHpUKbyIiIiIiKh3OXElEREREpAYsvImIiIiI1ICFNxERERGRGrDwJiIiIiJSAxbeOkRXnqPVlXZSxaYrn2NdaSdVbLryOdaVdpYnFt46RFdmNdOmdp48eRKXLl2SOgzSQNr0OS4NbWon85XeRps+x6WhTe3U1Hw1lDoAKn9ZWVn4448/cPbsWTRt2hTNmjVD69atpQ6rzGVnZ+PPP//E+fPn0ahRIzRu3Bht2rSROqy3On36NBYtWoR58+ahQ4cOUodDGoL5qpmYr/QmzFfNpMn5qnN3vP/9M0lCQgJiY2Nx4cIFiSIqX/Hx8diwYQOWLl2K9PR0PH36FIsWLcKZM2ekDq1MPXv2DOvXr8eSJUuQmJiIuLg4fPbZZxrbzlOnTmHZsmVYsGABPD09Abz6bPJnvKKYr8xXTcB8VQ3zlfmqCTQ9X3XqjrcQQvEzSUpKCn799VeEhoYiKysLubm52L9/P0xMTCSOsuzk5uYiODgYiYmJWLFiBRwcHAAAXbt2xaJFi1C9enXY2dlJHGXpyeVy7NmzBwkJCUXa6eTkhM8//xyWlpYadQfizJkzWLlyJWbMmAE3NzcAr94rY2NjZGVloVKlShJHqBmYr8xXTcB8VQ3zlfmqCbQhX3Xqjnd+fj5yc3Oxdu1arF27Fjt37oSPjw8aNWqEbt26wdCwYn0PuXr1KsLDwzF06FA4ODhACIHc3FzY29uje/fuePnypdQhlokbN27gl19+wZAhQ4q0s02bNujXrx8eP34sdYgK169fx7x58/Dxxx8rvokXnhTi4uLQsWNHPHjwQGO+mUuJ+cp8lRrzVXXMV+ar1LQlX3Wm8L579y58fX0xdepUBAUFwcLCAmFhYTAwMEB0dDSGDRsGAwMDZGdnIz09Xepwy0R0dDTatWuHDh06oKCgAHp6ejA2NkZeXh6uXr2Khw8fAtD+p5SvX78OOzs7dOzYEQUFBQAAY2NjFBQU4NKlS8jOzkZKSgoSExMljhSIiIjA0KFD4eHhgYKCAsVJ4f79+/D29saECRPQoEEDqcOUHPOV+cp81R7MV+Yr81V1Fesr6DsYGBigVq1a6N27N0aPHo1ly5ahZ8+euHbtGvr06YPs7Gz8+OOPePLkCaKjozF27Fj06NFD6rBLRSaTISEhAQBQUFAAff1X37MMDQ2xYsUK1KhRA8Crp5T/+TOhtqlVqxYsLCwAvGpn4Z2VxMREpKam4vLly7hz5w6sra0xZMgQmJmZSRarkZGR4kRcUFAAY2Nj3Lt3D4MHD4aDgwNGjRql2Fab35PSYr4yX5mv2oP5ynxlvhaD0CEFBQWKf58+fVo4OjoKGxsbcenSJXHjxg3xn//8R4wbN04cPHhQLF++XGRlZUkYbendv39fuLq6iiNHjgghhHjx4oVISkoSQgixb98+8cknn4iDBw8qtv/n/x9t8ujRI9GvXz9x+PBhIYQQ2dnZ4t69e2LWrFmiZcuWolOnTmLEiBEiKSlJZGdni7///luyWOPi4kT//v1FeHi4SE9PF1euXBEffPCB+OSTT8S2bdvEuHHjxC+//KLYXlvfk7LAfGW+Ml+1B/OV+cp8VY2eEFr+O0gJyOVyGBkZ4dNPP0W1atUwevRofPvtt5DL5cjPz8esWbNgbW0N4FW/NQMDA4kjLrk7d+7gs88+Q/v27XHu3DlMnToV3bp1w40bN5Cfn4+tW7eiZ8+eGDNmjNShlsrt27cxZ84c/Oc//0F0dDQaNGiAc+fOwdnZGVWqVMHLly8xZswY7Nu3D1lZWZg9ezaqVasmWawLFixAkyZN8Pfff6Nx48bo0KEDDh48CDc3N4SHh8PJyQkjRoyQJD5Nw3xlvjJftQfzlfnKfH03nSy8C40bNw5169ZFXl4eLC0tMW3aNGRlZWH+/PkYPnw4evXqJXWIZeLJkydISUnBL7/8AiEELl68CFdXV/Tp0wdZWVmYNWsWtm3bhtq1a2v1z2KPHj3CvXv38PLlSzx//hxXr15Fo0aN4OnpCTMzM2zZsgVpaWnw8PBA+/btJW1jcnIy9PT0cP78ebx8+RIxMTEYPnw47O3tcenSJWzcuBGbN2+GiYkJrl+/DmNjY7Ro0UKyeDUB85X5KhXma/ExX5mvUtH0fNWZPt7/lpCQgGvXrsHU1BR2dnZwd3eHnp4eAgICkJGRge+++w6PHz/G6NGjERERARMTE3Tr1k3qsEukTp06qFOnDp49e4ZTp05h5cqVyMjIwNatW3H16lW0bdsWderUUTyIoI0nBQCoV68e6tWrh7y8PKxfvx6NGzdG7969UbVqVWzfvh0XLlyAlZUV7t69qxgSSSo1atRAfn4+Hjx4gBs3bmDSpElo1aoVUlJSsHHjRrRo0QLm5uaIjY3Fxo0b0bRpU52+kDNfma9SYr4WD/OV+SolTc9XnRnV5N+srKxw6NAhfPDBB3BxcYGlpSUWL14MExMT+Pr6wt/fX/FThY2NDWrWrCl1yKXWtGlTXLlyBUlJSWjYsCEAoHHjxujfvz+SkpIwcuRI/PbbbxJHWXpyuRzR0dGoVq0aGjVqhM2bNyM1NRVeXl6YP38+goKCcOTIEanDhIGBAQYMGIBp06ahVatWePHiBZYsWYIWLVpgxIgRiImJwZYtW9CyZUvMnz8fABRPlesa5ivzVWrMV9UxX5mvUtPkfNXpriYAkJeXh9zcXMycORMNGjTAyJEjUbduXRgZGeG7777Dl19+icjISFSvXl0xZJC2fmMFXvVJ8/f3R3x8PJo2bYpx48ahUaNGWLJkCQoKCrB161apQywTDx8+xK1bt3D27FkArwb779ChA4yNjeHv7694r/X19TXi/UxNTcXMmTNhY2ODoUOH4uXLl/juu+9QtWpVLFmyBDk5OTA1NZU6TMkxX5mvmvB+Ml9Vw3xlvmrC+6lp+aqzXU0KGRoaIjk5Gc2aNcOgQYPQsGFD6OnpITg4GIGBgdi/fz+qV6+u+JkI0O5ho9577z2sWbMGmzdvRvfu3dGyZUt88sknMDY2xoYNGwAAFy5cgJGREdq3by9xtCVXv359VKpUCTExMWjbti0cHR1hYGCAvXv3Yt++fTh8+DAMDAyQl5enGB5Jyvc1KysL9vb2cHNzQ1paGgICAhAVFQVbW1tMnToVGRkZGDFiBJydnSWPVUrMV+Yr81V7MF+Zr8zX1+n8He9CcrkchoaG0NPTww8//IDly5fj0KFDePbsGY4dO4bnz5+jS5cuGDJkiNShlonU1FRYWFhg+fLluH37Nnbt2gXg1aQAR48eBQD4+Pgoxu/U1pNhZmYmTExMYGBggO+//x4rVqxASEgIsrKycPLkSaSmpsLW1haDBw+WOlTI5XLcunULgYGBsLOzg42NjWIc3CZNmqCgoACPHj1C8+bNAWjve1IWmK/MV6kxX1XHfGW+Sk2T8lVn+3j/m5GREfT09HDgwAH4+fnh4MGDuHv3Lr788kvY2trC09MTu3fvxo8//ih1qGXCwsICeXl5SE9PV3zLi4qKwpkzZ2Bubo7Ro0fD2NgYSUlJAP5vEgBtY2ZmBgMDAxw4cABLly7FwYMHce/ePcyYMQN//vknBg8ejNDQUAQHB0sdKoyMjCCTyWBra4tZs2bh9OnTyMvLQ8uWLVGpUiXs2bMHc+bMwZUrVwBo73tSFpivzFepMV9Vx3xlvkpNk/KVd7z/5a+//sLz58/h4OCAcePGYdKkSXB0dAQAnDlzBleuXIGPj49ie239plro1q1bmD9/Ptq2bYtr166hY8eOaNq0KV6+fIkdO3agTp066NatGyZMmCB1qKXyz/d1/PjxGD16NL7//nv06NEDXbt2xc6dOzF37lzF7GOa8L6OHz8eM2fOhL29PdavX4/o6Gg0bdoUqampsLe3x9ixYyWNTxMwX5mvgGa8r8xX5ZivzFdAM95XKfOVd7z/pWnTpnBwcMDDhw9RpUoVtGrVSvFaZmYmKleujNzcXCQmJgLQ3m+qhZo3b45NmzbBzMwM7733Htzd3fHixQtcu3YNq1evxvbt23H06FGcOXNG6lBLpfB9ffLkCQDA0dERmzZtQkREBFxdXRXbPX78GIC072tBQQFSUlLw/PlzWFtb45tvvsH169cxc+ZM+Pn5YdasWThw4ICiLbqM+cp8BZiv2oL5ynwFmK86/3Dl25iYmCA+Ph7nzp3DRx99hJSUFPTt2xdyuRw+Pj7Q19eHo6MjRo4cqdWD4gNA7dq1MXbsWLx48QIZGRnYsWMHNm/erBjXskGDBsjOzi6yj7a218TEBNnZ2fj111/h7OyMVatW4eDBg+jQoQPOnz+P/fv3w9nZGS4uLpK1T19fH9WrV8eePXuwe/dunD59Gp9++ilat24NAHj69CkqV6782oxv2vqelAXmK/OV+ao9mK/MV13OV3Y1eYdbt27Bz88PJiYmaNasGQYOHAhjY2M8f/4cbdu2hbe3N1xdXSvUVMG//fYb/ve//2HhwoWKJ5Kjo6ORlZWFBg0aIDk5Ge3atQPw6ptj4U9H2uTmzZvw9fVFbm4u3NzcMGbMGJw7dw4hISF48eIFsrKyMG7cOPTr10/qUPH06VM8e/YMLVu2hLGxMaKjo7Fw4ULMmDEDPXr0QEpKCl68eKG4c6StJ+yywHxlvkqN+ao65ivzVWpS5SsLbyUyMzNhZmaGc+fOYcOGDahSpQratGmDrl274ubNm4iKisKaNWsAvPr5RFuTpdDDhw8xbtw4jBkzBkIIDBkyBMbGxrhz5w7Cw8Nx9uxZeHh4wMvLS+pQSyU1NVUxhFVMTAyOHTuGmjVrYtasWcjKysLFixc1bia1qKgo+Pn5YcaMGWjevDl8fHzQsmVLxMXFYeDAgfD29pY6RMkxX5mvmoL5qhzzlfmqKdSZryy8lSj8hrN06VIYGBhg0aJFiIqKwu7du5GcnIzu3btj0qRJuHr1aoW5ixEXF4egoCA0bdoUhoaGSEtLg4ODA6pVq4YGDRrA29sbkydPhouLi9ShllpkZCT279+P9957D1OmTHnt5yVNkZ2djREjRqBfv37o168fPDw8MG/ePLi5uSE5ORkTJ07E559/rhgbVts/gyXFfGW+agLmq2qYr8xXTaDufNXer45qUvg/d8SIEbhw4QKOHDmCe/fuoWbNmrC3t8e4ceOwZ88ejBkzBn/++adiH23+PtOsWTOsWLECkyZNwuXLl5Gbm4tjx47hxIkTuHXrFmxtbfHgwYMi+2hre5s1awZbW1vFSUFTp3g2NTVFUFAQxo4di+PHj8Pd3R2enp4QQsDKygrNmjWDEEIRvzZfmEqD+cp81QTMV9UwX5mvmkDd+crCW0VNmzZFYGAgoqOjERQUBDMzM8yZMwd79uzBrl27MHjwYPz000/YuXMnAO0/kRoYGCA3NxdyuRxNmzaFv78/vLy8sGjRIly4cAH9+/dHeno6Hj58CEB7T4ZWVlaYNGmS4qSgyT9jWlpaAgBevnypiNPQ0BBHjhzB48eP0bhxY0yaNAknT56UMErNwHxlvkqN+ao65ivzVWrqzFeOalIM7733HiZPnoykpCTMmjUL33//Pfbv349169bB1tYWN2/exKefforevXujTp06UodbKnp6ejA2Nsbs2bPh5+cHuVyOS5cuwcDAAFu2bEFcXBx++eUX/P333+jXrx9GjBihOBlq60+BmnxSAP7vYtOnTx/MmjULP/74I+7du4f79+9j2LBhOHLkCPLy8tCzZ0+JI9UMzFfmq5SYr8XDfGW+Skmd+arZ/yc0kLW1NdavX4/Q0FBs27YNa9asga2tLbKzs/Hs2TM0adIExsbGAP7v56GcnBwpQy4VGxsbrF69GkeOHEF0dDTWr18PGxsbxMfHIy0tDZs2bUJISAhOnDih2Of58+cSRlzxNW7cGKtXr8bdu3fx9OlTjB49Gvb29sjPz8fo0aMBQPG+6DrmK/NVasxX1TFfma9SU0e+8uHKEnrw4AEyMzNhY2OD3NxcREdH49tvv8UHH3yASZMmAQAyMjJw+/Zt7Ny5E97e3vjggw8kjrrknj17hps3byI1NRXJycno3bs3Fi5ciDlz5iAqKgrNmzfHhx9+iOzsbHh5eWHr1q2QyWRa+c1cWxSmrp6eHiIjI7Fr1y60aNECFhYWWLVqFaZNm4bp06fzPQDzlfkqPear6pivzFeplWe+sqtJCTVo0AAAkJubi/Pnz2PXrl3o3r27YszRzZs3IyUlBTExMbh69SoaNWqk1SeGWrVqoVatWjh+/DiuXLmCevXqoWHDhrh27Rp69OiBhg0bKr6V79mzB2ZmZhJHXPEVJvxff/2F5cuXIzk5GX379sXz589ha2uL4cOHIzExES9fvkSzZs0kjlZazFfmq9SYr6pjvjJfpVae+cquJqWUn5+PQ4cOwcHBAUOGDMF3332HIUOG4Ny5c7Czs8PAgQMxevRojBo1SupQy0SzZs1w+fJlnDhxAvXr18eFCxfQpEkTREZGYvr06UhPTy9yUuAPKuXPwsICjRs3xuzZs9G7d28cP34cXl5eSEpKwoIFC3D48GHExMQottfl94T5ynyVGvNVdcxX5qvUyiNf2dWkDCQmJuLChQv44osv0LJlS1y8eBFBQUHo2LEjAgMDcf/+fSxfvhzm5uZSh1ombt++jbVr18LU1BRDhw6FkZERRo0ahfXr1+Ojjz7CtWvXkJGRgS5dugDQ3hm4tEl6ejrMzc1x9OhRbN++HYMGDcKBAwcwc+ZMmJub4/jx4+jTp49iWlxdfk+Yr8xXqTFfVcd8Zb5KrczzVVCZePTokQgLCxNCCBERESF69+4tNm3aJHr37i3++OMPIYQQOTk5iu0LCgqkCLPMFLYlMjJStGrVSpw6dUoIIcS1a9fEhg0bxLBhw8Tu3bulDFHn5ObmihEjRohu3bqJAwcOiIcPH4oXL16IlStXCldXVzF16lQRHBwsdZgagfnKfJUa81V1zFfmq9TKMl95x7ucXLhwAdOmTUOXLl0QGBiIrKwsnDhxAo0aNUKbNm0AaPc3VSEEMjMz0aNHD3z55ZeoXr06Dh06hI4dO6Jhw4Z477334OnpiU8++QTdu3eXOlyd8ejRI1hbW8PIyAj379/H/v37IZfL0alTJzg6OmLAgAFYvHgxOnfuLHWoGoX5ynyVAvO1ZJivzFcplFW+8uHKMib+/xibHTp0gIGBAZycnGBgYIDIyEh88803sLCwgIeHBwYPHqy1JwXg1YMHlStXxsmTJ2Fubo7x48ejc+fO+PPPP3H79m08e/YMlpaWSE1NLbKf0NIxSLVFvXr1AAA3b97Ejz/+CGNjYzg5OaFDhw4AXvVXS05OljJEjcJ8Zb5KiflaPMxX5quUyipfWXiXscIZpn766SfY2Nige/fuOHbsGE6ePIkPP/wQw4cPx9ixY2FlZYVu3boB0O5kqVy5MjIzM5GRkYFatWph/PjxePLkCQYPHoyGDRuiT58+uHDhAszMzGBvb6/4/6Ot7dUWMpkM5ubmcHR0VJwUPv30UxQUFKBv374SR6c5mK/MV03AfFUN85X5qglKm6/salJOnj17hm+//RbNmzdHdHQ0mjVrhvHjxwMA5s6diwYNGqBLly5o27YtAO0+OQCvvgF+8cUX6Nq1KyIiImBpaYnly5cjPj4eAQEBEEJg8ODB6Nevn9Sh6ozs7GyYmpoCeHVSSElJwVdffYWaNWtq/eetrDFfma9SY76qjvnKfJVaafJVe3+L0XC1atXCjBkzcPfuXdSpU0dxUvjyyy8RFhaGqlWrIigoCMHBwQCg1ScF4NUMXCtWrMDp06cBvGrngwcPcODAAZiZmWH69OnYvXs3fvzxR2kD1SEmJiYAgAULFuDevXtYs2YNatasCUD7P29ljfnKfJUa81V1zFfmq9RKk6+8413Onj9/jmrVqgEAVq1ahZ9++gmHDx+GtbU1Tpw4gS1btmDHjh0wNzfX6j5phRITE3H//n1kZ2fj119/haGhIaZOnQpLS0t8//33uHjxIlatWqWY9pfK35MnT2Bqaorq1atLHYrGY74yX6XGfFUd85X5KrWS5Kv2fxI1XOFJITAwEDt27EBoaCisra0RGxuLw4cPw9nZGVWrVi1yUtDm70IymQwtWrRAVFQUcnNzMW3aNFhaWiI6Ohp79+5Fp06deFJQszp16vAiriLmK/NVasxX1TFfma9SK0m+8o63mty9exdmZmaoVasWbty4gV27dqFx48Zo164dzMzM8Pfff8Pc3LxCPBACAElJSTA2NkbVqlVx+fJlrFu3Dj169MCgQYMqzEQHVHExX5mvpD2Yr8xXbcI73mrSpEkT1KpVC7du3cKWLVtQv3591KtXDxcuXMDMmTORlJSErVu3YteuXQC0v09azZo1UbVqVVy5cgXLli1Djx494O7uzpMCaQXmK/OVtAfzlfmqTVh4q1m1atVgY2OD3r17AwASEhJgaWmJSpUq4bvvvsOZM2dw9+5drf457J9q1qyJQYMGwd3dnT+fktZhvhJpD+YraQN2NZGAXC6HkZERpkyZgl69emHQoEEYP348KleujJo1a2LevHmKJ2YB7f9ZTJtnECNivhJpD+YraTq+WxIwMjKCXC5HtWrVYGj4ag6jjRs3wsTEBFlZWTAxMcHGjRuxf/9+AK9+FsvPz5cy5FLhSYG0GfOVSHswX0nT8Y63hG7evIm5c+eiUaNGaNGiBcaOHYtKlSph06ZN2LhxIzp06IC+ffvCy8sLMTExAAB7e3uJoybSTcxXIu3BfCVNxSnjJWRjY4PNmzfj4sWLqFy5MipVqoTly5cjLCwMYWFhMDMzw7Zt25CSkgJra2ucPXuWJwYiiTBfibQH85U0Fe94a5AlS5bg5MmTCA0NRY0aNQAA586dQ1BQEBYvXoymTZsC0P4+aUQVAfOVSHswX0lT8I63BhBCIDExEefPn8eBAwdQo0YNZGZm4sqVK/j222/h5OSkOCkAr/qk8eRAJA3mK5H2YL6SpuEdbw1S+HRyRkYGYmNjsX37dvTs2RM9e/bE6tWrUblyZbRo0QJeXl5Sh0qk85ivRNqD+Uqago/DapDCb9rXr1/H2rVr4ebmhm7dusHb2xs1a9aEi4sLDh48qHgam4ikw3wl0h7MV9IU7GqiQQp/2qpXrx769euH/v37w9XVFQMHDsS0adMAAN7e3oiNjQXwf33R+LMYkfoxX4m0B/OVNAXveGug2rVrY8yYMcjIyEDz5s0VJwUAaNCgAapVqwbg1axcwP99kyci9WO+EmkP5itJjXe8NVhOTg7+/vtvhIWFoWPHjqhcuTLatWuHOnXqYPv27YiKikKfPn0wePBgfjMnkhjzlUh7MF9JKny4UsPFxcVh/vz5aN68OUxNTeHu7o4qVaogISEBdevWxdy5c+Hi4oKRI0dKHSqRzmO+EmkP5itJgYW3Fnjx4gUqV66M4OBgxMXF4c6dO/joo4/Qr18/XLt2Dfv27cPmzZtRqVIlfjMnkhjzlUh7MF9J3djVRAtYWloCADIzM2FiYoLvvvsO9+/fx6pVqxAXF4ehQ4fCzMwMiYmJkMlkPDkQSYj5SqQ9mK+kbny4Uov07t0bf/zxB6KiovD06VOYmJigT58+6NevHy5cuICxY8fixIkTAPhACJHUmK9E2oP5SurCriZaJi4uDitXrsTNmzfh6emJoUOHIj4+HjNmzICDgwPMzc3RqlUreHt7Sx0qkc5jvhJpD+YrqQMLby2UmJiIb775BgMGDEBaWhoWLlwILy8vjBo1Ck+fPsXIkSOxefNmvP/++1KHSqTzmK9E2oP5SuWNXU20kEwmw8yZM5GTk4Pp06fD29sbo0aNQkFBASIjI2FjY4PatWtLHSYRgflKpE2Yr1Te+HCllqpUqRIsLCzg4+OD4cOHAwDCwsJw7do1ODg4wNjYWOIIiagQ85VIezBfqTyxq0kF8eOPP+LGjRuoX78+Bg4ciKpVq0odEhG9BfOVSHswX6kssatJBZCeno6//voLMpmMJwUiDcd8JdIezFcqa7zjXUG8ePECBgYGqFKlitShEJESzFci7cF8pbLEwpuIiIiISA3Y1YSIiIiISA1YeBMRERERqQELbyIiIiIiNWDhTURERESkBiy8iYiIiIjUgIU3EREREZEacMp4HRUVFYUVK1bAzMwMGRkZaNeuHa5fv46MjAwIIbB8+XK0b98eGRkZWL58OS5fvgwDAwP06tULs2fPhlwux5o1a3Dp0iXk5+fD1tYWixYtgrm5udRNI6pwmK9E2oU5S2/DwluH3blzBydOnEBCQgJ27NiBffv2QV9fH9u3b0dQUBDat2+PDRs2ICcnB0eOHEF+fj7GjRuHixcv4tKlSzAwMMCBAwegp6eHgIAArFmzBn5+flI3i6hCYr4SaRfmLL0JC28dVrt2bdStWxd169aFhYUFfvjhBzx8+BBRUVGoXLkyACAyMhLz58+HgYEBDAwMsGfPHgDAV199hbS0NERGRgIA5HI5atSoIVlbiCo65iuRdmHO0puw8NZhZmZmAIDTp09jxYoVGDt2LHr27IkmTZrg0KFDAABDQ0Po6ekp9nn69ClMTU1RUFCABQsWoFu3bgCAjIwM5OTkqL8RRDqC+UqkXZiz9CZ8uJJw7tw5dO/eHV5eXrCzs8OJEyeQn58PAOjUqRMOHjyIgoIC5ObmwsfHB5cuXUKXLl0QHByM3NxcFBQUYPHixQgICJC4JUQVH/OVSLswZ+mfWHgThg0bhkuXLsHNzQ1Dhw5F/fr18ejRIxQUFGD69OkwMjKCh4cHPD090a1bN/Tu3RvTpk1D3bp10b9/f7i4uEAIAV9fX6mbQlThMV+JtAtzlv5JTwghpA6CiIiIiKii4x1vIiIiIiI1YOFNRERERKQGLLyJiIiIiNSAhTcRERERkRqw8CYiIiIiUgMW3kREREREasDCm4iIiIhIDf4fEZ977qnFS5UAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 311 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 359 - }, - "id": "yUaAejwa6P3F", - "outputId": "c6d84d35-fdde-47be-af88-69881afaa163" - }, - "outputs": [], - "source": [ - "overall_df.transpose().plot.bar(subplots=True, layout= [1,3], figsize=(12, 5), legend=False, rot=-45, position=1.5);" - ] + "id": "DvjRBIcjjSkl", + "outputId": "0aebd5a1-f287-4d75-cdd3-8af1daf1d190" + } + }, + { + "cell_type": "markdown", + "source": [ + "According to the above bar chart, it seems that the group *Unknown* is selected for the care management program less often than other groups as reflected by the selection rate. Also this group experiences the largest false negative rate, so a larger fraction of group members that are likely to benefit from the care management program are not selected. Finally, the balanced accuracy on this group is also the lowest.\n", + "\n" + ], + "metadata": { + "id": "b5C3SITjuPUm" + } + }, + { + "cell_type": "markdown", + "source": [ + "We observe disparity, even though we did not include race in our model. There's a variety of reasons why such disparities may occur. It could be due to representational issues (i.e., not enough instances per group), or because the feature distribution itself differs across groups (i.e., different relationship between features and target variable, obvious example would be people with darker skin in facial recognition systems, but can be much more subtle). Real-world applications often exhibit both kinds of issues at the same time." + ], + "metadata": { + "id": "c2Qs68rv2_Vg" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "## Exercise: Train other fairness-unaware models" + ], + "metadata": { + "id": "n_1Rm8PbPmmk" + } + }, + { + "cell_type": "markdown", + "source": [ + "In this section, you'll be training your own fairness-unaware model and evaluate the model using the `MetricFrame` for fairness-related harms." + ], + "metadata": { + "id": "oeQF5qT6Qs-C" + } + }, + { + "cell_type": "markdown", + "source": [ + "We encourage you to explore the model's performance across different sensitive features (such as `age` or `gender`) as well as different model performance metrics." + ], + "metadata": { + "id": "SsHy-Os0oVQU" + } + }, + { + "cell_type": "markdown", + "source": [ + "1.) First, let's train our machine learning model. We'll create a `HistGradientBoostingClassifier` and fit it to the balanced training data set." + ], + "metadata": { + "id": "61GU_zrFSC-6" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "from sklearn.experimental import enable_hist_gradient_boosting\r\n", + "from sklearn.ensemble import HistGradientBoostingClassifier\r\n", + "\r\n", + "# Create your model here\r\n", + "clf = HistGradientBoostingClassifier()\r\n", + "\r\n", + "# Fit the model to the training data\r\n", + "clf.fit(__________, ________)\r\n", + "exercise_pred = clf.predict(______)" + ], + "outputs": [], + "metadata": { + "id": "bSIDkuV4_Rou" + } + }, + { + "cell_type": "markdown", + "source": [ + "2.) Next, let's evaluate the fairness of the model using the `MetricFrame`. In the below cells, create a `MetricFrame` that looks at the following metrics:\r\n", + "\r\n", + "\r\n", + "* _Count_: The number of data points belonging to each sensitive feature category.\r\n", + "* _False Positive Rate_: $\\dfrac{FP}{FP+TN}$\r\n", + "* _Recall Score_: $\\dfrac{TP}{TP+FN}$\r\n", + "\r\n", + "As an extra challenge, you can use the prediction probabilities to compute the _ROC AUC Score_ for each sensitive group pair.\r\n", + "\r\n" + ], + "metadata": { + "id": "Fnnles-p6OXr" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "# Define exercise fairness metrics of interest here\r\n", + "exercise_metrics = {\r\n", + " \"count\": count,\r\n", + " \"false_positive_rate\": _______,\r\n", + " \"recall_score\": _______\r\n", + "}" + ], + "outputs": [], + "metadata": { + "id": "bcf-x1oA_jP5" + } + }, + { + "cell_type": "markdown", + "source": [ + "Now, let's create our `MetricFrame` using the metrics listed above with the sensitive groups of `race` and `gender`." + ], + "metadata": { + "id": "Bll-8GAWJF6p" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "metricframe_exercise = MetricFrame(\r\n", + " metrics=__________,\r\n", + " y_true=Y_test,\r\n", + " y_pred=__________,\r\n", + " sensitive_features=_____\r\n", + ")" + ], + "outputs": [], + "metadata": { + "id": "jAjzjCqh_fNx" + } + }, + { + "cell_type": "markdown", + "source": [ + "3.) Finally, play around with the plotting capabilities of the `MetricFrame` in the below section.\n", + "\n" + ], + "metadata": { + "id": "QeghVCbLZOf5" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "metricframe_exercise._______" + ], + "outputs": [], + "metadata": { + "id": "Nd4D17ME_hB2" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "# Plot some of the performance disparities here\r\n", + "metricframe_exercise.by_group.____.bar(subplots=_____, layout=[1,3], figsize=(12, 4),\r\n", + " legend=False, rot=-45, position=1.5)" + ], + "outputs": [], + "metadata": { + "id": "_xaLx6Br_hyc" + } + }, + { + "cell_type": "markdown", + "source": [ + "The charts above are based on test data, so without any uncertainty quantification (such as error bars or confidence intervals), we cannot reliably compare these data statistics. Next optional section shows how to augment MetricFrame with the report of error bars.\n", + "\n", + "## Adding error bars [OPTIONAL SECTION]" + ], + "metadata": { + "id": "Me1ocEi2kEgw" + } + }, + { + "cell_type": "markdown", + "source": [ + "In this section, we define new custom metrics that quantify errors in our estimates of selection rate, false negative rate and balanced accuracy, and then review our metrics again." + ], + "metadata": { + "id": "9l8YJ8qQdehm" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "# All of our error bar calculations are based on normal approximation to\r\n", + "# the binomial variables.\r\n", + "\r\n", + "def error_bar_normal(n_successes, n_trials, z=1.96):\r\n", + " \"\"\"\r\n", + " Computes the error bars for the parameter p of a binomial variable\r\n", + " using normal approximation. The default value z corresponds to the 95%\r\n", + " confidence interval.\r\n", + " \"\"\"\r\n", + " point_est = n_successes / n_trials\r\n", + " error_bar = z*np.sqrt(point_est*(1-point_est))/np.sqrt(n_trials)\r\n", + " return error_bar\r\n", + "\r\n", + "def fpr_error(Y_true, Y_pred):\r\n", + " \"\"\"\r\n", + " Compute the 95%-error bar for the false positive rate\r\n", + " \"\"\"\r\n", + " tn, fp, fn, tp = confusion_matrix(Y_true, Y_pred).ravel()\r\n", + " return error_bar_normal(fp, tn+fp)\r\n", + "\r\n", + "def fnr_error(Y_true, Y_pred):\r\n", + " \"\"\"\r\n", + " Compute the 95%-error bar for the false negative rate\r\n", + " \"\"\"\r\n", + " tn, fp, fn, tp = confusion_matrix(Y_true, Y_pred).ravel()\r\n", + " return error_bar_normal(fn, fn+tp)\r\n", + "\r\n", + "def selection_rate_error(Y_true, Y_pred):\r\n", + " \"\"\"\r\n", + " Compute the 95%-error bar for the selection rate\r\n", + " \"\"\"\r\n", + " tn, fp, fn, tp = confusion_matrix(Y_true, Y_pred).ravel()\r\n", + " return error_bar_normal(tp+fp, tn+fp+fn+tp)\r\n", + "\r\n", + "def balanced_accuracy_error(Y_true, Y_pred):\r\n", + " \"\"\"\r\n", + " Compute the 95%-error bar for the balanced accuracy\r\n", + " \"\"\"\r\n", + " fnr_err, fpr_err = fnr_error(Y_true, Y_pred), fpr_error(Y_true, Y_pred)\r\n", + " return np.sqrt(fnr_err**2 + fpr_err**2)/2" + ], + "outputs": [], + "metadata": { + "id": "OiP-uXr_FLtz" + } + }, + { + "cell_type": "markdown", + "source": [ + "We next create a metric frame that includes the sample sizes and error bar sizes in addition to the metrics that we have used previously." + ], + "metadata": { + "id": "qHaXfBYWp6ob" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "metrics_with_err_bars = {\r\n", + " \"count\": count,\r\n", + " \"selection_rate\": selection_rate,\r\n", + " \"selection_err_bar\": selection_rate_error,\r\n", + " \"false_negative_rate\": false_negative_rate,\r\n", + " \"fnr_err_bar\": fnr_error,\r\n", + " \"balanced_accuracy\": balanced_accuracy_score,\r\n", + " \"bal_acc_err_bar\": balanced_accuracy_error\r\n", + "}\r\n", + "\r\n", + "# sometimes we will only want to display metrics without error bars\r\n", + "metrics_to_display = [\r\n", + " \"count\",\r\n", + " \"selection_rate\",\r\n", + " \"false_negative_rate\",\r\n", + " \"balanced_accuracy\"\r\n", + "]\r\n", + "\r\n", + "# sometimes we will only want to show the difference values of the metrics other than count\r\n", + "differences_to_display = [\r\n", + " \"selection_rate\",\r\n", + " \"false_negative_rate\",\r\n", + " \"balanced_accuracy\"\r\n", + "]" + ], + "outputs": [], + "metadata": { + "id": "OlEq6ogfyHb6" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "metricframe_unmitigated_w_err = MetricFrame(\r\n", + " metrics=metrics_with_err_bars,\r\n", + " y_true=Y_test,\r\n", + " y_pred=Y_pred,\r\n", + " sensitive_features=A_test\r\n", + ")" + ], + "outputs": [], + "metadata": { + "id": "55GOqFTYxbCi" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "unmitigated_groups = metricframe_unmitigated_w_err.by_group\r\n", + "unmitigated_groups # show both the metrics as well as the error bars" + ], + "outputs": [], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 223 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "pIGS8f2M2Afu" - }, - "outputs": [], - "source": [ - "difference_df = pd.DataFrame.from_dict({\n", - " \"Unmitigated\": metricframe_unmitigated.difference(),\n", - " \"Postprocessing\": metricframe_postprocess.difference(),\n", - " \"Postprocessing (DET)\": mf_deterministic.difference(),\n", - " \"Reductions\": metricframe_reductions.difference()\n", - "}\n", - ")" - ] + "id": "MlZKPNkHxbFb", + "outputId": "9d6cb95c-538a-42e7-f3b4-349c74258902" + } + }, + { + "cell_type": "markdown", + "source": [ + "We see that for smaller sample sizes we have larger error bars. The problem is further exacerbated for false negative rate, which is estimated only over *positive examples* and so its sample sizes is further reduced due to label imbalance.\n", + "\n", + "We next visualize the metrics with the corresponding error bars using a custom plotting function." + ], + "metadata": { + "id": "hh5F-A5C5rSV" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "def plot_group_metrics_with_error_bars(metricframe, metric, error_name):\r\n", + " \"\"\"\r\n", + " Plots the disaggregated `metric` for each group with an associated\r\n", + " error bar. Both metric and the erro bar are provided as columns in the \r\n", + " provided metricframe.\r\n", + " \"\"\"\r\n", + " grouped_metrics = metricframe.by_group\r\n", + " point_estimates = grouped_metrics[metric]\r\n", + " error_bars = grouped_metrics[error_name]\r\n", + " lower_bounds = point_estimates - error_bars\r\n", + " upper_bounds = point_estimates + error_bars\r\n", + "\r\n", + " x_axis_names = [str(name) for name in error_bars.index.to_flat_index().tolist()]\r\n", + " plt.vlines(x_axis_names, lower_bounds, upper_bounds, linestyles=\"dashed\", alpha=0.45)\r\n", + " plt.scatter(x_axis_names, point_estimates, s=25)\r\n", + " plt.xticks(rotation=0)\r\n", + " y_start, y_end = np.round(min(lower_bounds), decimals=2), np.round(max(upper_bounds), decimals=2)\r\n", + " plt.yticks(np.arange(y_start, y_end, 0.05))\r\n", + " plt.ylabel(metric)" + ], + "outputs": [], + "metadata": { + "id": "rqFPLsATwROr" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "plot_group_metrics_with_error_bars(metricframe_unmitigated_w_err, \"selection_rate\", \"selection_err_bar\")" + ], + "outputs": [], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 268 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 172 - }, - "id": "3pBkP7QCDVs6", - "outputId": "8bec1174-4a94-4cf6-8cab-634e179c0ea7" - }, - "outputs": [], - "source": [ - "difference_df.T" - ] + "id": "dsRFpuXfzrUA", + "outputId": "0f4adb40-6c34-45a2-b8ca-b50d151c4b55" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "plot_group_metrics_with_error_bars(metricframe_unmitigated_w_err, \"false_negative_rate\", \"fnr_err_bar\")" + ], + "outputs": [], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 268 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 356 - }, - "id": "_qXEkByRDkK0", - "outputId": "36ac6599-3113-4862-b5a0-7fe90bc71a5b" - }, - "outputs": [], - "source": [ - "difference_df.T.plot.bar(subplots=True, layout= [1,3], figsize=(12, 5), legend=False, rot=-45, position=1.5);" - ] + "id": "B68Q2ZIgzrcE", + "outputId": "368fed7e-80c6-4fca-ebef-d61cdd09e800" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "plot_group_metrics_with_error_bars(metricframe_unmitigated_w_err, \"balanced_accuracy\", \"bal_acc_err_bar\")" + ], + "outputs": [], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 268 }, - { - "cell_type": "markdown", - "metadata": { - "id": "7S6CIoJ-W1jH" - }, - "source": [ - "### Randomized predictions" - ] + "id": "htoUhyirpqZt", + "outputId": "c83ac73e-ccbf-4a53-8002-c2dc012b8e4c" + } + }, + { + "cell_type": "markdown", + "source": [ + "As we see above, even accounting for the larger uncertainty in estimating the false negative rate for *Unknown*, this group is experiencing substantially larger false negative rate than other groups and thus experiences the harm of allocation." + ], + "metadata": { + "id": "SAMhqvycR7eE" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "\n", + "---\n" + ], + "metadata": { + "id": "8ZqVGZkam1eH" + } + }, + { + "cell_type": "markdown", + "source": [ + "# **Mitigating fairness-related harms in ML models**" + ], + "metadata": { + "id": "-dgITdRiD7Yu" + } + }, + { + "cell_type": "markdown", + "source": [ + "We have found that the logistic regression predictor leads to a large difference in false negative rates between the groups. We next look at **algorithmic mitigation strategies** of this fairness issue (and similar ones).\n", + "\n", + "*Note that while we currently focus on the training stage of the AI lifecycle mitigation should not be limited to this stage. In fact, we have already discussed mitigation strategies that are applicable at the task definition stage (e.g., checking for construct validity) and data collection stage (e.g., collecting more data).*\n", + "\n", + "Within the model training stage, mitigation may occur at different steps relative to model training:\n", + "\n", + "* **Preprocessing**: A mitigation algorithm is applied to transform the input data to the training algorithm; for example, some strategies seek to remove and dependence between the input features and sensitive features.\n", + "\n", + "* **At training time**: The model is trained by an (optimization) algorithm that seeks to satisfy fairness constraints.\n", + "\n", + "* **Postprocessing**: The output of a trained model is transformed to mitigate fairness issues; for example, the predicted probability of readmission is thresholded according to a group-specific threshold.\n", + "\n", + "We will now dive into two algorithms: a postprocessing approach and a reductions approach (which is a training-time algorithm). Both of them are in fact **meta-algorithms** in the sense that they act as wrappers around *any* standard (fairness-unaware) machine learning algorithms. This makes them quite versatile in practice.\n" + ], + "metadata": { + "id": "sbUSG1jVA06G" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Postprocessing with `ThresholdOptimizer`" + ], + "metadata": { + "id": "rX8QycCL0mJj" + } + }, + { + "cell_type": "markdown", + "source": [ + "**Postprocessing** techniques are a class of unfairness-mitigation algorithms that take an already trained model and a dataset as an input and seek to fit a transformation function to model's outputs to satisfy some (group) fairness constraint(s). They might be the only feasible unfairness mitigation approach when developers cannot influence training of the model, due to practical reasons or due to security or privacy.\n" + ], + "metadata": { + "id": "fRZfSFzcFaXP" + } + }, + { + "cell_type": "markdown", + "source": [ + "Here we use the `ThresholdOptimizer` algorithm from Fairlearn, which follows the approach of [Hardt, Price, and Srebro (2016)](https://arxiv.org/abs/1610.02413).\n", + "\n", + "`ThresholdOptimizer` takes in an exisiting (possibly pre-fit) machine learning model whose predictions act as a scoring function and identifies a separate thrceshold for each group in order to optimize some specified objective metric (such as **balanced accuracy**) subject to specified fairness constraints (such as **false negative rate parity**). Thus, the resulting classifier is just a suitably thresholded version of the underlying machinelearning model.\n", + "\n", + "The constraint **false negative rate parity** requires that all the groups have equal values of false negative rate.\n", + "\n" + ], + "metadata": { + "id": "6PgzZkK9Wbni" + } + }, + { + "cell_type": "markdown", + "source": [ + "To instatiate our `ThresholdOptimizer`, we pass in:\n", + "\n", + "* An existing `estimator` that we wish to threshold. \n", + "* The fairness `constraints` we want to satisfy.\n", + "* The `objective` metric we want to maximize.\n", + "\n" + ], + "metadata": { + "id": "OFOovaN7AwDr" + } + }, + { + "cell_type": "code", + "execution_count": 58, + "source": [ + "# Now we instantite ThresholdOptimizer with the logistic regression estimator\r\n", + "postprocess_est = ThresholdOptimizer(\r\n", + " estimator=unmitigated_pipeline,\r\n", + " constraints=\"false_negative_rate_parity\",\r\n", + " objective=\"balanced_accuracy_score\",\r\n", + " prefit=True,\r\n", + " predict_method='predict_proba'\r\n", + ")" + ], + "outputs": [], + "metadata": { + "id": "8je0grKPWHhy" + } + }, + { + "cell_type": "markdown", + "source": [ + "In order to use the `ThresholdOptimizer`, we need access to the sensitive features **both during training time and once it's deployed**." + ], + "metadata": { + "id": "VDD86L7eCSe0" + } + }, + { + "cell_type": "code", + "execution_count": 59, + "source": [ + "postprocess_est.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" + ], + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\postprocessing\\_threshold_optimizer.py:309: UserWarning: The value of `prefit` is `True`, but `check_is_fitted` raised `NotFittedError` on the base estimator.\n", + "\n", + "If the provided base estimator has been fitted, this could mean that (1) its implementation does not conform to the sklearn estimator API, or (2) the enclosing ThresholdOptimizer has been cloned (for instance by `sklearn.model_selection.cross_validate`).\n", + "\n", + "In case (1), please file an issue with the base estimator developers, but continue to use the enclosing ThresholdOptimizer with `prefit=True`. In case (2), please use `prefit=False`.\n", + " type(self).__name__\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\postprocessing\\_interpolated_thresholder.py:115: UserWarning: The value of `prefit` is `True`, but `check_is_fitted` raised `NotFittedError` on the base estimator.\n", + "\n", + "If the provided base estimator has been fitted, this could mean that (1) its implementation does not conform to the sklearn estimator API, or (2) the enclosing InterpolatedThresholder has been cloned (for instance by `sklearn.model_selection.cross_validate`).\n", + "\n", + "In case (1), please file an issue with the base estimator developers, but continue to use the enclosing InterpolatedThresholder with `prefit=True`. In case (2), please use `prefit=False`.\n", + " warn(BASE_ESTIMATOR_NOT_FITTED_WARNING.format(type(self).__name__))\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
ThresholdOptimizer(constraints='false_negative_rate_parity',\n",
+       "                   estimator=Pipeline(steps=[('preprocessing',\n",
+       "                                              StandardScaler()),\n",
+       "                                             ('logistic_regression',\n",
+       "                                              LogisticRegression(max_iter=1000))]),\n",
+       "                   objective='balanced_accuracy_score',\n",
+       "                   predict_method='predict_proba', prefit=True)
StandardScaler()
LogisticRegression(max_iter=1000)
" + ], + "text/plain": [ + "ThresholdOptimizer(constraints='false_negative_rate_parity',\n", + " estimator=Pipeline(steps=[('preprocessing',\n", + " StandardScaler()),\n", + " ('logistic_regression',\n", + " LogisticRegression(max_iter=1000))]),\n", + " objective='balanced_accuracy_score',\n", + " predict_method='predict_proba', prefit=True)" + ] + }, + "metadata": {}, + "execution_count": 59 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 344 }, - { - "cell_type": "markdown", - "metadata": { - "id": "MzdFLJ9BA15F" - }, - "source": [ - "Both the `ExponentiatedGradient` and the `ThresholdOptimizer` yield randomized predictions (may return different result given the same instance). Due to legal regulations or other concerns, a practitioner may not be able to deploy a randomized model. To address these restrictions:\n", - "\n", - "* We created a deterministic predictor based on the randomized thresholds learned by the `ThresholdOptimizer`. This deteministic predictor achieved similar performance as the `ThresholdOptimizer`.\n", - "* For the `ExponentiatedGradient` model, we can deploy one of the deterministic inner models rather than the overall `ExponentiatedGradient` model.\n", - "\n" - ] + "id": "VCHJBB7x1rAK", + "outputId": "dfc6338e-2be0-4007-bde2-fd8544cc4a89" + } + }, + { + "cell_type": "code", + "execution_count": 60, + "source": [ + "# Record and evaluate the output of the trained ThresholdOptimizer on test data\r\n", + "\r\n", + "Y_pred_postprocess = postprocess_est.predict(X_test, sensitive_features=A_test)\r\n", + "metricframe_postprocess = MetricFrame(\r\n", + " metrics=metrics_dict,\r\n", + " y_true=Y_test,\r\n", + " y_pred=Y_pred_postprocess,\r\n", + " sensitive_features=A_test\r\n", + ")" + ], + "outputs": [], + "metadata": { + "id": "YscNZsYU1rCY" + } + }, + { + "cell_type": "markdown", + "source": [ + "We can now inspect how the metric values differ between the postprocessed model and the unmitigated model:" + ], + "metadata": { + "id": "_izbGv6tQ1KD" + } + }, + { + "cell_type": "code", + "execution_count": 61, + "source": [ + "pd.concat([metricframe_unmitigated.by_group,\r\n", + " metricframe_postprocess.by_group],\r\n", + " keys=['Unmitigated', 'ThresholdOptimizer'],\r\n", + " axis=1)" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
UnmitigatedThresholdOptimizer
selection_ratefalse_negative_ratebalanced_accuracyselection_ratefalse_negative_ratebalanced_accuracy
race
AfricanAmerican0.4000.4280.5970.4270.4000.598
Caucasian0.3910.4420.5940.4510.3890.590
Other0.3280.5230.5840.4120.4550.574
Unknown0.2640.6700.5360.4590.4360.557
\n", + "
" + ], + "text/plain": [ + " Unmitigated \\\n", + " selection_rate false_negative_rate balanced_accuracy \n", + "race \n", + "AfricanAmerican 0.400 0.428 0.597 \n", + "Caucasian 0.391 0.442 0.594 \n", + "Other 0.328 0.523 0.584 \n", + "Unknown 0.264 0.670 0.536 \n", + "\n", + " ThresholdOptimizer \n", + " selection_rate false_negative_rate balanced_accuracy \n", + "race \n", + "AfricanAmerican 0.427 0.400 0.598 \n", + "Caucasian 0.451 0.389 0.590 \n", + "Other 0.412 0.455 0.574 \n", + "Unknown 0.459 0.436 0.557 " + ] + }, + "metadata": {}, + "execution_count": 61 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 254 }, - { - "cell_type": "markdown", - "metadata": { - "id": "P9dvkQvKW1lv" - }, - "source": [ - "### Access to sensitive features\n", - "\n" - ] + "id": "-9mtWyWc1rH5", + "outputId": "982dcb16-b82f-42bc-addb-1caaf2d0aa09" + } + }, + { + "cell_type": "markdown", + "source": [ + "We next zoom in on differences between the largest and the smallest metric values:" + ], + "metadata": { + "id": "mzPCUFsXPU_S" + } + }, + { + "cell_type": "code", + "execution_count": 62, + "source": [ + "pd.concat([metricframe_unmitigated.difference(),\r\n", + " metricframe_postprocess.difference()],\r\n", + " keys=['Unmitigated: difference', 'ThresholdOptimizer: difference'],\r\n", + " axis=1).T" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
selection_ratefalse_negative_ratebalanced_accuracy
Unmitigated: difference0.1370.2420.061
ThresholdOptimizer: difference0.0470.0660.041
\n", + "
" + ], + "text/plain": [ + " selection_rate false_negative_rate \\\n", + "Unmitigated: difference 0.137 0.242 \n", + "ThresholdOptimizer: difference 0.047 0.066 \n", + "\n", + " balanced_accuracy \n", + "Unmitigated: difference 0.061 \n", + "ThresholdOptimizer: difference 0.041 " + ] + }, + "metadata": {}, + "execution_count": 62 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 127 }, - { - "cell_type": "markdown", - "metadata": { - "id": "L56tawsAW14B" - }, - "source": [ - "\n", - "\n", - "* The `ThresholdOptimizer` model requires access to the sensitive features during BOTH training time and once deployed. If you do not have access to the sensitive features once the model is deployed, you will not be able to use the `ThresholdOptimizer`.\n", - "* The `ExponentiatedGradient` model requires access to the sensitive features ONLY during training time. \n", - "\n", - "\n" - ] + "id": "dsC4v8Ap1rKt", + "outputId": "ebf489b6-07ac-45d5-ae2b-848d6488dbb4" + } + }, + { + "cell_type": "markdown", + "source": [ + "As we see, `ThresholdOptimizer` was able to substantiallydecrease the difference between the values of false negative rate." + ], + "metadata": { + "id": "Hhi_RSxSRoyg" + } + }, + { + "cell_type": "markdown", + "source": [ + "Finally, we save the disagregated statistics:" + ], + "metadata": { + "id": "GarQvopkVN2S" + } + }, + { + "cell_type": "code", + "execution_count": 63, + "source": [ + "metricframe_postprocess.by_group.plot.bar(subplots=True, layout=[1,3], figsize=(12, 4), legend=False, rot=-45, position=1.5)\r\n", + "postprocess_performance = figure_to_base64str(plt)" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtgAAAFNCAYAAAA6kBhoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABJvElEQVR4nO3dd1hTZ/8/8HcAQREVVMC6Vx8QBbeitVoXKBLFraC4HkdRUVqtu+Jq1ao4HiffLitWbKsiVhHR4lNlKLUVt+K2CmEoyg7k/v3hjzxaB8MDJ4H367p6XSY54XzuJu/cn5ycoRBCCBARERERkSQM5C6AiIiIiKgsYYNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYRFRuLF68GD169ICfn99rH4+Ojoarq2spVyWt2NhYfP755wCACxcuwNvbW2fqISqq4mTSxsYGKSkpJVRRwb7++mvMnTtXtvWTbmCDrSc2bdqEpUuXFvv5CxcuxMWLFwEACxYsQEREhFSlFQsnXZJDYGAgdu/eDR8fH7lLKTFxcXFISEgAANjb22Pjxo06Uw8RUXlhJHcBVDoiIiIwfPhwAMCKFStkroaTLpU+d3d3CCEwceJE9OnTB7///jtycnKQkpICNzc3zJw586XlY2JisHLlSmg0GgDA5MmT4ezsjJycHKxZswZnz55FXl4e7OzssHDhQpiZmb1x3dHR0fDz80O9evVw48YN5OTk4PPPP4ejo+Nb/15sbCx8fX2hVqtRv359PHz4EHPnzkX79u3xxRdf4Pz580hPT4cQAsuXL0ft2rWxceNGPHv2DPPmzYObmxuWLVuGH3/8Ed26dcPRo0dhaWkJABg2bBimTp2KTp06FWs8K1asgKmpKTIyMvDzzz9j9erVBdbz5Zdf4sSJE9i6dSvUajUqVqyIOXPmoHXr1u/+AlOZlZGRAW9vb9y9exdVq1bVbmxaunQpMjIyoFKpYGtri/Xr18PExOSl5/n6+uLOnTtITU1F5cqVsWbNGjRu3BijR49Gq1atcO7cOTx69Aht27bFqlWrYGBggN9++w3r16+HRqOBqakplixZAltbW5w7dw5r1qxBZmYmFAoFpk+fju7du0OtVmP58uWIiIhAjRo1UKNGDVSpUqXAcW3btg1hYWHIzs5GZmYm5syZg969eyM3NxdfffUVwsPDYWhoiNatW2Px4sUwMDB47f3bt2/H48ePtRutNm3apL09evRoVKtWDbdu3cLIkSNhb2+Pr776Cjk5OUhMTETnzp3xxRdfAMBrx/3bb78hLi4Oa9euBQD88ccfWLZsGQ4cOCDxq1wGCZJFWlqamD59uujfv79wc3MTCxYsEHl5eeL48eNiyJAhYsCAAWL48OHi3LlzQgghNm7cKJYsWSKEECI+Pl54eXmJgQMHCldXV7F161bt3z1x4oTo37+/cHV1FcOGDRNXrlwR69atE82bNxdOTk7ir7/+EqNGjRJHjhwRQghx7NgxMWDAAOHq6ipGjBghzp8/r13fnDlzxPjx44Wzs7MYOXKkiI+Pf+uYoqKihFKpFMOHDxdKpVJkZ2eLZcuWiSFDhoi+ffuKPn36iJiYGPHw4UPRrVs30aZNGzF37lwhhHjjuImk9K9//UskJyeLUaNGidu3bwshnuepWbNmIjk5WURFRYl+/foJIYTw9PQUhw4dEkIIceXKFeHr6yuEEGLTpk1i5cqVQqPRCCGEWLt2rVi8ePFb1xsVFSWaNWsmLl++LIQQ4uuvvxYeHh5v/XtqtVp07dpVhIeHCyGEiIyMFDY2NiIqKkqcO3dOTJ8+XeTl5QkhhNi+fbuYPHmyEEKIX375RUyaNEm73vzxfPbZZ+L//u//hBBCxMXFiY8++kjk5eUVezy2trbiwYMHQghR6Hpu374tXF1dRUpKihBCiOvXr4sPPvhApKenv3V9VH7lv9f++OMPIYQQe/bsEUOGDBErV64UBw4cEEIIkZOTI1xdXUVISIgQ4n85P3LkiFi2bJn2by1atEgsXbpUCCHEqFGjhLe3t8jLyxPPnj0TXbp0EZGRkSIxMVG0bdtWm9WjR4+KCRMmiCdPnggnJydx//59IcTzz42uXbuKv//+W3z33XfC09NTZGdni/T0dDFw4EAxZ86ct47rwYMHYvTo0SIzM1MIIcShQ4eEq6urEEKI77//Xnh4eIjMzEyRl5cnZsyYIfbv3//G+1/sD4R4uV8YNWqUmDdvnvYxHx8fERUVJYR43od07NhRXLhw4Y3jTkpKEm3atBGPHz8WQggxe/Zs8eOPPxbpNSyvuAVbJseOHUN6ejqCgoKQl5eHxYsX4969e/Dz88POnTthYWGBGzduYNy4cQgNDX3pubNnz8bYsWPRo0cPZGdnY+LEiahfvz46dOiA2bNn44cffkCzZs0QGhqKNWvW4P/+7/8QHByMNWvWwN7eXvt3bt68icWLF2PPnj2oV68eIiMj4eXlhZCQEADPt+AdOHAAZmZmmDJlCgIDAwvcn/PGjRsICwtDnTp18Oeff0KlUiEwMBAGBgbYsWMH/P39sW3bNnh7e+Po0aP48ssvcefOnTeO29TUVPr/+VTubdu2DeHh4Th06BBu3rwJIQQyMzNfWqZv375YunQpTpw4gc6dO+OTTz4BAISHh+PZs2fa3azUajVq1KhR4Dpr166NZs2aAQDs7Oywf//+t/6969evAwC6desGAHB0dMT7778PAGjdujWqVauGPXv24P79+4iOjkblypXfuv6hQ4diyZIlmDBhAn755RcMGjQIBgYGxR7Pe++9hzp16hSpntOnT0OlUmHs2LHa+xQKBe7duwdbW9sC10nlk42NDdq0aQMAGDhwIHx9ffHNN9/gr7/+gr+/P+7cuQOVSoWMjIyXntenTx/Uq1cPP/zwA+7evYszZ8689GtJ9+7dYWBgADMzMzRo0ACpqak4d+4c3n//fW1WnZyc4OTkhJMnTyIxMRFTp07VPl+hUODatWuIjIyEq6srjI2NYWxsDKVSiWvXrr11THXq1MGqVasQHByMu3fvan/9AZ7/4jxgwABUrFgRALB+/XoAwJQpU157/6ZNm966rnbt2mn/vXLlSvz3v//Ftm3bcOvWLWRlZSEjI+ON4waAjz76CEFBQXBzc8OpU6ewePHit66PnmODLZO2bdvCz88Po0ePRufOnTFmzJi3Tj75MjIycPbsWaSmpmLDhg3a+65evQojI6M3BuR1oqKi4OjoiHr16gEAOnXqhOrVq2v31e7QoYP2Z2I7OzukpqYWOC5OuqTrMjMzMWLECPTq1Qvt2rXD4MGDERYWBiHES8uNGDEC3bt3x+nTp/H777/jP//5Dw4ePAiNRoP58+drG9/09HRkZ2cXuN78SRF4/v7OX9+b/l5iYuIrNRkaGgJ43pSvWLEC48aNQ8+ePdG4cWMcPHjwretv164dcnNzERsbi0OHDmHPnj1vXX9BXvzyW9h6NBoNOnXqpG0MAODRo0ewsrIqcH1UfhkYvHy4mEKhwIIFCyCEQN++ffHRRx/h0aNHr+Rl9+7d2Lt3Lzw8PKBUKmFubo4HDx5oH39dJo2MjKBQKLT3CyFw7do15OXloUmTJvjpp5+0jyUkJKB69eoIDAx8ab35OX2bS5cuwcvLC2PHjsUHH3yA9u3bY8mSJQAAI6OXW7OkpCRoNJo33v/i5wnw/Evyi17MqoeHB2xtbfHhhx+ib9++OH/+PIQQMDQ0fO24bW1t4eHhAV9fXxgZGcHJyanAL/P0HA9ylEm9evVw7NgxTJo0CWlpaRg3bhyePHmCTp06ISgoSPvf3r17tVutgOcTlBACe/bs0S4TGBiIyZMnvzYgV69efWMN//wwyr8vNzcXwJsbgrf556Q7efJkAEDPnj0xcuTI1z4nf9J927iJpJKSkoK0tDTMnDkTPXr0wJkzZ5CTk6Pd1zrfiBEjcOXKFQwaNAjLli3D06dPkZqaii5duiAgIED7nEWLFmHdunXFrudNf69JkyYwNjbGf//7XwDPDwy+fv06FAoFTp8+je7du8Pd3R329vYICwtDXl4egOeTe36G/2no0KFYtmwZbGxsULt27beuvygKW4+joyNOnz6NmzdvAgBOnjyJ/v37F6qhp/Lr2rVruHLlCoDnByq3bdsWERERmDp1KlxcXKBQKHD+/Hntey7fqVOnMHDgQAwdOhSNGjXCiRMnXlnmn1q2bImbN2/ixo0bAIDjx49j9uzZaNWqFe7evYuzZ88CAK5cuQJnZ2eoVCp8+OGHOHDgALKzs5GdnY3Dhw8XOKazZ8+iRYsWGDduHDp06IDjx49ra+vUqRMOHTqkzaSvry9+/fXXN95vYWGBS5cuQQiBjIwMnDp16rXrTE1NxcWLFzFr1iw4OTkhISEB9+7dg0ajeeO4AaBNmzYwMDDA119//cZ5nF7FLdgy2b17N/744w+sWbMGH374IZKTk7U/Nd28eRNNmjTByZMnMWvWLJw8eVL7PDMzM7Rq1QrffvstvLy88PTpU4wcORJTp05F586dtQF5//33cfz4cWzYsAHBwcGvnXQdHR3xn//8B/fv39fuIvLo0SO0bNkSf/755zuP8cVJNzs7G/7+/m+cdDdu3PjacXMXEZJa7dq18dFHH6Fv376oWrUq6tevj6ZNm+Lu3bswNjbWLjdr1ix88cUXWL9+PQwMDDBt2jTUrVsXXl5eWLVqFQYOHIi8vDw0a9bsnU7J9aa/Z2RkhE2bNmHx4sVYt24dGjZsiJo1a6JixYoYMWIEZs2aBaVSCUNDQ7Rr1w6hoaHQaDRo3bo11q9fj6lTp8LT0/Oldbm5uWHdunUvNdBSjKew9WzevBlLly7FJ598ot1auHXrVuac3qpx48bauapGjRpYuXIlwsPDMXXqVFSrVg2VKlVC+/btX/q1FwDGjx+Pzz//HPv27YOhoSGaN2+u3fXqTWrWrIk1a9Zgzpw5yMvLg5mZGfz8/FC9enVs3LgRq1evRnZ2NoQQWL16NerUqYMRI0bg3r17cHV1hbm5ORo0aFDgmFxdXREaGgoXFxdUqFABnTp1QmpqKtLS0jBixAj8/fffGDRoEIQQ6NChA0aPHg2FQvHa+zMzM/H777/DyckJ1tbWaN269Ws3iFWrVg2TJk3CwIEDYW5uDgsLC7Rp0wZ3797VHuz8z3HnGzRoEA4fPgwbG5tCvmqkEIXZLEmSy8jIwPz583Ht2jVUqlQJtWvXxooVKxAREYFt27ZpJ5/58+ejXbt2Lx0V/ODBAyxbtgwPHz5ETk4OXF1dMX36dADA77//Dj8/P21AlixZgqZNm2LVqlU4cuQIli9fju3bt8PDwwN9+vTBkSNHsG3bNuTl5aFixYqYO3fuK+sD8Mrt14mOjsayZctw6NAhAM/38Z41axZyc3NfmnTDw8Px4MEDjBkzBnZ2dti8ebO2jn+Om6g8W7VqFSZMmICaNWvi0aNHGDBgAMLCwlC1alW5SyOiciI3NxfTpk1D//794eLiInc5eoMNNhGRBGbOnInbt2+/9jE/Pz80bty4yH9z165d2LNnD4yMjCCEwNSpU996XIWUSmI8ROXJF198gejo6Nc+Nm/ePDg6OpZyRUUXFxeHkSNHomvXrvjqq69e2R+e3owNNhUJJ10iIiKit2ODTUREREQkIW7rJyIiIiKSEBtsIiIiIiIJ6cRp+h4/TodGo797qtSoYYbk5DS5y3hnZWEc+j4GAwMFLCx0/yT+zKz8OAb5Ma+lQ9/fJ0DZGAOg/+MozczqRIOt0Qi9Dj8Ava8/X1kYR1kYg65jZnUDx0CFwbzqhrIwBqDsjKOkcRcRIiIiIiIJscEmIiIiIpIQG2wiIiIiIgmxwSYiIiIikhAbbCIiIpkEBwfDxcUFvXv3RkBAwCuP37p1C6NHj0b//v0xYcIEpKamylAlERUVG2wiIiIZJCQkwM/PD7t370ZQUBACAwMRFxenfVwIgY8//hgTJ07EwYMH0axZM+zYsUPGiomosNhgExERySAiIgKOjo4wNzeHqakpnJ2dERISon380qVLMDU1RdeuXQEAU6ZMgYeHh1zlElER6MR5sImIiMoblUoFS0tL7W0rKyvExsZqb9+7dw81a9bEnDlzcPnyZfzrX//CokWLiryeGjXMJKlXTpaWVeQu4Z2VhTEAZWccJY0NNpVrVapWQkWTko1BVnYunj3NLNF1EBXEopoxjIxNSnQduTnZeJyaU6LrKEuEePWCHQqFQvvv3NxcnDlzBrt27YK9vT3Wr1+PlStXYuXKlUVaT3JyWqEvDlLF3AQVKxgX6e8XVZY6B8+eZBd6eUvLKkhMfFaCFZW8sjAGQP/HYWCgKLUvnGywqVyraGIE5adBJbqO4LUDoL8fR1RWGBmb4NaKwSW6jsYLfgHABruwrK2tERMTo72tUqlgZWWlvW1paYkGDRrA3t4eAODq6gpvb+8SraliBWMMC/y4RNexd/hWPEPhG2wifcR9sImIiGTQuXNnREZGIiUlBZmZmQgNDdXubw0ArVu3RkpKCq5evQoAOHHiBJo3by5XuURUBNyCLZOS3jWBuyUQEek2a2tr+Pj4wNPTE2q1GkOGDIGDgwMmTpwIb29v2NvbY/PmzVi4cCEyMzNRq1YtrF69Wu6ydZ4u7uZC5Q8bbJmU9K4J3C2BiEj3KZVKKJXKl+7z9/fX/rtly5b4+eefS7ssvcbdXEgXcBcRIiIiIiIJscEmIiIiIpIQG2wiIiIiIglxH2wiItIbJX0+b57Lm3RFSR+syQM1SxYbbCIi0hslfT5vnsubdEVJH6zJAzVLFncRISIiIiKSEBtsIiIiIiIJscEmIiIiIpIQG2wiIiIiIgnp5UGOvMw4ERERkW4rz2dC0csGm5cZJyIiItJt5flMKNxFhIiIiIhIQmywiYiIiIgkxAabiIiIiEhCbLCJiIiIiCTEBpuIiIiISEJ6eRYR0g0lfbpEgKdMJCIiIv3DBpuKraRPlwjwlInFERwcjK1bt0KtVmPs2LHw8PB47XLh4eFYunQpTpw4UcoVEhERlW2F2kUkODgYLi4u6N27NwICAt64XHh4OHr06CFZcURUNAkJCfDz88Pu3bsRFBSEwMBAxMXFvbJcUlISVq1aJUOFREREZV+BDTYnbCL9ERERAUdHR5ibm8PU1BTOzs4ICQl5ZbmFCxdi2rRpMlRIRERU9hW4i8iLEzYA7YT9z8k5f8Jeu3ZtiRRKRAVTqVSwtLTU3rayskJsbOxLy+zcuRN2dnZo2bJlsddTo4ZZoZfV5ObAwKjkLpVb3L9vaVmlBKopXbo4hqLWVBbGQET0TwU22KUxYRdlsi4t5XWSKAvjKAtjKC4hxCv3KRQK7b+vX7+O0NBQfPfdd4iPjy/2epKT06DRvLqu17G0rIJbKwYXe10FabzgFyQmFm1PfUvLKkV+jq4p6hhK6z1Y1JqK89qVtMLWZGCg0Mn5i4jkV2CDXRoTdlEma0C3PmAB/Z8k8pXHCVvXxvCuE7a1tTViYmK0t1UqFaysrLS3Q0JCkJiYiMGDB0OtVkOlUsHd3R27d+8u9jqJiIjoZQXug21tbY2kpCTt7bdN2JMmTdJO2ERU+jp37ozIyEikpKQgMzMToaGh6Nq1q/Zxb29vHD16FEFBQdixYwesrKzYXBMREUmswAabEzaR/rC2toaPjw88PT3h5uYGV1dXODg4YOLEibhw4YLc5REREZULBe4i8uKErVarMWTIEO2E7e3tDXt7+9Kok4gKSalUQqlUvnSfv7//K8vVrVuX58AuJItqxjAyNinRdeTmZONxak6JroOIiEpHoS40wwmbiMozI2OTEj1QE3h+sCbABrs8KujiUP/5z3/wyy+/oGrVqgCAYcOGvfECUkSkG3glRyIiIpnkX2ti3759MDY2xogRI9CxY0c0bdpUu8zFixexbt06tG7dWsZKiagoCnUlRyIiIpJeYS4OdfHiRfj7+0OpVGLp0qXIzs6WqVoiKixuwSYiIpJJQdeaSE9PR7NmzTBnzhzUqVMHc+fOxZYtW+Dj41PodejiubrL4/UNysIYivuckqRr9eRjg01ERCSTgq41Ubly5ZeOeRo/fjzmz59fpAa7qBeGKg3l6foGAK+XUZKKUk9pXhyKu4gQERHJpKBrTTx8+BA///yz9rYQAkZG3DZGpOvYYBMREcmkoGtNVKxYEV999RXu378PIQQCAgLQu3dvGSsmosJgg01ERCSTgi4OVb16dSxduhQff/wx+vTpAyEExo0bJ3fZRFQA/s5EREQko4KuNeHs7AxnZ+fSLouI3gG3YBMRERERSYgNNhERERGRhNhgExERERFJiA02EREREZGE2GATEREREUmIDTYRERERkYTYYBMRERERSYgNNhERERGRhNhgExERERFJiA02EREREZGE2GATEREREUmIDTYRERERkYTYYBMRERERSYgNNhERERGRhNhgExERERFJiA02EREREZGE2GATEREREUmIDTYRERERkYTYYBMRERERSYgNNhERERGRhNhgExERERFJiA02EREREZGE2GATEREREUmIDTYRERERkYTYYBMRERERSYgNNhERkUyCg4Ph4uKC3r17IyAg4I3LhYeHo0ePHqVYGRG9CyO5CyAiIiqPEhIS4Ofnh3379sHY2BgjRoxAx44d0bRp05eWS0pKwqpVq2SqkoiKg1uwicqYgraIHTt2DEqlEv369cPcuXORk5MjQ5VEFBERAUdHR5ibm8PU1BTOzs4ICQl5ZbmFCxdi2rRpMlRIRMVVqC3YwcHB2Lp1K9RqNcaOHQsPD4+XHj927Bg2btwIjUYDe3t7LF26FMbGxiVSMBG9WUFbxDIyMrB06VLs378fNWvWhI+PD/bv34/hw4fLXDlR+aNSqWBpaam9bWVlhdjY2JeW2blzJ+zs7NCyZctir6dGDbNiP7ekWFpWKdHlS0N5HENxn1OSdK2efAU22JywifTHi1vEAGi3iOVv/TI1NcWJEydQoUIFZGRkIDk5GVWrVpWxYqLySwjxyn0KhUL77+vXryM0NBTfffcd4uPji72e5OQ0aDSvrut1SqtZSUx8VuhlLS2rFHn50lCSY8h/TkkrTk269loUpR4DA0WpfeEssMHmhE2kPwqzRaxChQo4efIkPvvsM1hZWaFLly5FXo+ubRErC1thgPK5RawsjKG4rK2tERMTo72tUqlgZWWlvR0SEoLExEQMHjwYarUaKpUK7u7u2L17d6nUR0TFV2CDXRoTtq5N1kD5nSTKwjjKwhiKq6AtYvm6deuG6OhorFu3Dr6+vli7dm2R1qNrW8TKwlYYgFv1CvucklbYmt51a1jnzp2xadMmpKSkoFKlSggNDcWyZcu0j3t7e8Pb2xsA8ODBA3h6erK5JtITBTbYpTFhF2WyBnTrAxbQ/0kiX3mcsHVtDO86YRe0RezJkye4ePGi9kuwUqmEj49PsddHRMVnbW0NHx8feHp6Qq1WY8iQIXBwcMDEiRPh7e0Ne3t7uUskomIq8Cwi1tbWSEpK0t5+3YR96tQp7W2lUolr165JXCYRFUbnzp0RGRmJlJQUZGZmIjQ0FF27dtU+LoTA7Nmz8fDhQwDAkSNH0KZNG7nKJSr3lEolDh06hKNHj2LixIkAAH9//1ea67p16+LEiRNylEhExVBgg80Jm0h/vLhFzM3NDa6urtotYhcuXICFhQWWLVuGyZMno3///rhz5w5mz54td9lERERlSoG7iBTmJ6z8CVuhUKBp06ZYsmRJadRORK+hVCqhVCpfus/f31/77169eqFXr16lXRYREVG5UajzYHPCJiIiIiIqHF7JkYiIiIhIQmywiYiIiIgkxAabiIiIiEhCbLCJiIiIiCTEBpuIiIiISEJssImIiIiIJMQGm4iIiIhIQmywiYiIiIgkxAabiIiIiEhCbLCJiIiIiCTEBpuIiIiISEJssImIiIiIJMQGm4iIiIhIQmywiYiIiIgkxAabiIiIiEhCbLCJiIiIiCTEBpuIiIiISEJssImIiIiIJMQGm4iIiIhIQmywiYiIiIgkxAabiIiIiEhCbLCJiIiIiCTEBpuIiIiISEJssImIiGQSHBwMFxcX9O7dGwEBAa88fuzYMSiVSvTr1w9z585FTk6ODFUSUVGxwSYiIpJBQkIC/Pz8sHv3bgQFBSEwMBBxcXHaxzMyMrB06VJ8++23+PXXX5GdnY39+/fLWDERFRYbbCIiIhlERETA0dER5ubmMDU1hbOzM0JCQrSPm5qa4sSJE6hZsyYyMjKQnJyMqlWrylgxERWWkdwFEBERlUcqlQqWlpba21ZWVoiNjX1pmQoVKuDkyZP47LPPYGVlhS5duhR5PTVqmL1zrVKztKxSosuXhvI4huI+pyTpWj352GATERHJQAjxyn0KheKV+7p164bo6GisW7cOvr6+WLt2bZHWk5ycBo3m1XW9Tmk1K4mJzwq9rKVllSIvXxpKcgz5zylpxalJ116LotRjYKAotS+c3EWEiIhIBtbW1khKStLeVqlUsLKy0t5+8uQJTp06pb2tVCpx7dq1Uq2RiIqHDTYREZEMOnfujMjISKSkpCAzMxOhoaHo2rWr9nEhBGbPno2HDx8CAI4cOYI2bdrIVS4RFQF3ESEiIpKBtbU1fHx84OnpCbVajSFDhsDBwQETJ06Et7c37O3tsWzZMkyePBkKhQJNmzbFkiVL5C6biAqBDTYREZFMlEollErlS/f5+/tr/92rVy/06tWrtMsionfEXUSIiIiIiCTEBpuIiIiISEJssImIiIiIJMQGm4iIiIhIQoVqsIODg+Hi4oLevXsjICDglcfDwsIwYMAA9O/fH15eXkhNTZW8UCIqHOaViIhIXgU22AkJCfDz88Pu3bsRFBSEwMBAxMXFaR9PS0uDr68vduzYgYMHD8LGxgabNm0q0aKJ6PWYVyIiIvkV2GBHRETA0dER5ubmMDU1hbOzM0JCQrSPq9Vq+Pr6wtraGgBgY2ODR48elVzFRPRGzCsREZH8CjwPtkqlgqWlpfa2lZUVYmNjtbctLCy05+jMysrCjh07MHr06CIVUVrXhS8KS8sqJbp8aShOTWVhHGVhDMVVGnkFdC+zfK/rjvI4BiKifyqwwRZCvHKfQqF45b5nz57By8sLtra2GDhwYJGKSE5Og0bz6nrepDQ+/BITnxV6WUvLKkVaPv85Ja04NRV13KWhJF8LXRuDgYHinZrX0sgrULTM8r1eeOXpvQ7o/2fnu+aViMquAncRsba2RlJSkva2SqWClZXVS8uoVCq4u7vD1tYWK1askL5KIioU5pWIiEh+BTbYnTt3RmRkJFJSUpCZmYnQ0FB07dpV+3heXh6mTJmCvn37YsGCBa/dWkZEpYN5JSIikl+Bu4hYW1vDx8cHnp6eUKvVGDJkCBwcHDBx4kR4e3sjPj4ely9fRl5eHo4ePQoAaNGiBbeMEcmAeSUiIpJfgQ02ACiVSiiVypfu8/f3BwDY29vj6tWr0ldGRMXCvBIREcmLV3IkIiIiIpIQG2wiIiIiIgmxwSYiIiIikhAbbCIiIiIiCbHBJiIiIiKSEBtsIiIiIiIJscEmIiIiIpIQG2wiIiIiIgmxwSYiIiIikhAbbCIiIiIiCbHBJiIiIiKSEBtsIiIiIiIJscEmIiKSSXBwMFxcXNC7d28EBAS88nhYWBgGDBiA/v37w8vLC6mpqTJUSURFxQabiIhIBgkJCfDz88Pu3bsRFBSEwMBAxMXFaR9PS0uDr68vduzYgYMHD8LGxgabNm2SsWIiKiw22ERERDKIiIiAo6MjzM3NYWpqCmdnZ4SEhGgfV6vV8PX1hbW1NQDAxsYGjx49kqtcIioCNthEREQyUKlUsLS01N62srJCQkKC9raFhQV69eoFAMjKysKOHTu0t4lItxnJXQAREVF5JIR45T6FQvHKfc+ePYOXlxdsbW0xcODAIq+nRg2zYtVXkiwtq5To8qWhPI6huM8pSbpWTz422ERERDKwtrZGTEyM9rZKpYKVldVLy6hUKkyYMAGOjo6YP39+sdaTnJwGjebVZv51SqtZSUx8VuhlLS2rFHn50lCSY8h/TkkrTk269loUpR4DA0WpfeHkLiJEREQy6Ny5MyIjI5GSkoLMzEyEhoaia9eu2sfz8vIwZcoU9O3bFwsWLHjt1m0i0k3cgk1ERCQDa2tr+Pj4wNPTE2q1GkOGDIGDgwMmTpwIb29vxMfH4/Lly8jLy8PRo0cBAC1atMCKFStkrpyICsIGm4iISCZKpRJKpfKl+/z9/QEA9vb2uHr1qhxlEdE74i4iREREREQSYoNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhNthERERERBJig01UxgQHB8PFxQW9e/dGQEDAG5ebM2cO9u3bV4qVERERlQ+FarA5YRPph4SEBPj5+WH37t0ICgpCYGAg4uLiXllmypQpCAkJkalKIiKisq3ABpsTNpH+iIiIgKOjI8zNzWFqagpnZ+dXchkcHIyePXuib9++MlVJRERUthkVtMCLEzYA7YQ9bdo07TL5E3b+MkQkD5VKBUtLS+1tKysrxMbGvrTMv//9bwDAH3/8Uez11KhhVuznlgRLyyql8pySVtSaOIaSoYs1EZF+KbDBLo0JW9cma6D8ThJlYRxlYQzFJYR45T6FQiH5epKT06DRvLqu1ymNsScmPivS8paWVYr0nNJ6/YpaU3kbQ/5zSlphazIwUOjk/EVE8iuwwS6NCbsokzWgWx+wgP5PEvnK44Sta2N41wnb2toaMTEx2tsqlQpWVlbF/ntERERUdAXug21tbY2kpCTtbU7YRLqrc+fOiIyMREpKCjIzMxEaGoquXbvKXRYREVG5UmCDzQmbSH9YW1vDx8cHnp6ecHNzg6urKxwcHDBx4kRcuHBB7vKIiIjKhQJ3EXlxwlar1RgyZIh2wvb29oa9vX1p1ElEhaRUKqFUKl+6z9/f/5XlVq5cWVolERERlSsFNtgAJ2wiIiIiosLilRyJiIiIiCTEBpuIiEgmvFIyUdnEBpuIiEgGvFIyUdnFBpuIiEgGL14p2dTUVHul5BflXym5b9++MlVJRMVRqIMciYiISFqlcaVkgFdLLinlcQzFfU5J0rV68rHBJiIikkFpXCkZKNrVknXt6raA/l+hF+AVn0tSUep516slFwV3ESEiIpIBr5RMVHaxwSYiIpIBr5RMVHaxwSYiIpLBi1dKdnNzg6urq/ZKyRcuXJC7PCJ6B9wHm4iISCa8UjJR2cQt2EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkoUI12MHBwXBxcUHv3r0REBDwyuNXrlzB4MGD4ezsjAULFiA3N1fyQomocJhXIv3BvBKVTQU22AkJCfDz88Pu3bsRFBSEwMBAxMXFvbTM7NmzsWjRIhw9ehRCCOzdu7fECiaiN2NeifQH80pUdhkVtEBERAQcHR1hbm4OAHB2dkZISAimTZsGAPj777+RlZWFVq1aAQAGDRqEjRs3wt3dvdBFGBgoily4lUWlIj+nKIpaU1kYQ3GeU9JjAEr+tdClMRTnNXtRaeS1OHUaVbMs0vJFVRrv9ZIeA1Dy7/WyMAZAd95PZTWvlqbVi7R8cZT0+6QsjAEo+XGUxmenLo3hXTNbFAU22CqVCpaW//sws7KyQmxs7Bsft7S0REJCQpGKsLCoXKTlAeDrhU5Ffk5R1KhhVqLLA7o3huI8p6THAJT8a6GLYyiu0sgrUPTM1p+2rcjrKIrSeK+X9BiAkn+vl4UxALr5fioOXc3rZuWKIq+jqEr6fVIWxgCU/DhK47NTF8dQGgrcRUQI8cp9CoWi0I8TUelhXon0B/NKVHYV2GBbW1sjKSlJe1ulUsHKyuqNjycmJr70OBGVHuaVSH8wr0RlV4ENdufOnREZGYmUlBRkZmYiNDQUXbt21T5ep04dmJiY4I8//gAAHDhw4KXHiaj0MK9E+oN5JSq7FOJ1v0H9Q3BwMLZv3w61Wo0hQ4Zg4sSJmDhxIry9vWFvb4+rV69i4cKFSE9Ph52dHb788ksYGxuXRv1E9A/MK5H+YF6JyqZCNdhERERERFQ4vJIjEREREZGE2GATEREREUmIDTYRERERkYTYYBMRERERSYgNdhlUXo5bLS/jpLKtvLyPy8s4qWwrL+/j8jLOksQGuwwqL1f60qdxHj9+HGfPnpW7DNJB+vQ+fhf6NE7mld5En97H70KfxqmreTWSuwCSTmZmJv7880+cOnUKTZo0QdOmTdGyZUu5y5JcVlYW/vrrL0RGRqJhw4Zo1KgRWrVqJXdZbxQeHo6FCxdizpw5aN++vdzlkI5gXnUT80qvw7zqJl3Oa5ndgv3PnzdUKhUuXLiAqKgomSoqWQkJCdi4cSOWLFmCtLQ0PHr0CAsXLsTJkyflLk1S8fHx2LBhAxYvXozExETExcXhs88+09lxnjhxAsuWLcP8+fPh5uYG4Pl7kz+/vYx5ZV51AfNaOMwr86oLdD2vZXILthBC+/NGSkoKfvvtNwQFBSEzMxM5OTnYu3cvTExMZK5SOjk5OQgICEBiYiJWrFiBdu3aAQC6du2KhQsXonr16rC3t5e5ynenVquxa9cuqFSql8bZu3dvfP755zA3N9epLQonT57EypUrMX36dCiVSgDPXytjY2NkZmaiUqVKMleoG5hX5lUXMK+Fw7wyr7pAH/JaJrdg5+XlIScnB2vXrsXatWvx3XffwdvbGw0bNkS3bt1gZFS2vldcvHgRISEhGD58ONq1awchBHJycuDg4IDu3bvj6dOncpcoiStXruDXX3/FsGHDXhpnq1at0K9fP/z9999yl6h1+fJlzJkzB1OnTtV+s84Pf1xcHDp27Ih79+7pzDdtOTGvzKvcmNfCY16ZV7npS17LXIN969YtzJ07Fx9//DH8/f1RrVo1BAcHw9DQEDExMRgxYgQMDQ2RlZWFtLQ0ucuVRExMDNq0aYP27dtDo9FAoVDA2NgYubm5uHjxIu7fvw9A/48Kvnz5Muzt7dGxY0doNBoAgLGxMTQaDc6ePYusrCykpKQgMTFR5kqB0NBQDB8+HAMGDIBGo9GG/+7du/Dw8MC///1v1K9fX+4yZce8Mq/Mq/5gXplX5rXwytZXTQCGhoaoVasWnJycMGbMGCxbtgw9e/bEpUuX0KdPH2RlZeGnn37Cw4cPERMTg3HjxqFHjx5yl/1OLC0toVKpAAAajQYGBs+/NxkZGWHFihWoUaMGgOdHBb/4856+qVWrFqpVqwbg+Tjzt5QkJiYiNTUV586dw40bN2BtbY1hw4bB1NRUtlorVKig/cDVaDQwNjbGnTt3MHToULRr1w6enp7aZfX5NXlXzCvzyrzqD+aVeWVei0CUQRqNRvvv8PBw4ejoKGxtbcXZs2fFlStXxAcffCDGjx8v9u/fL5YvXy4yMzNlrPbd3b17V7i6uorDhw8LIYR48uSJSEpKEkIIERgYKD755BOxf/9+7fIv/v/RJw8ePBD9+vUThw4dEkIIkZWVJe7cuSNmzpwpmjdvLjp16iRGjRolkpKSRFZWlrh9+7ZstcbFxYmBAweKkJAQkZaWJs6fPy86dOggPvnkE7F9+3Yxfvx48euvv2qX19fXRArMK/PKvOoP5pV5ZV4LRyGEnv+u8RZqtRoVKlTAp59+CgsLC4wZMwbffPMN1Go18vLyMHPmTFhbWwN4vl+ZoaGhzBUX340bN/DZZ5+hbdu2OH36ND7++GN069YNV65cQV5eHrZt24aePXti7Nixcpf6Tq5fv45Zs2bhgw8+QExMDOrXr4/Tp0/D2dkZVapUwdOnTzF27FgEBgYiMzMTPj4+sLCwkK3W+fPno3Hjxrh9+zYaNWqE9u3bY//+/VAqlQgJCUHv3r0xatQoWerTNcwr88q86g/mlXllXt+uTDfY+caPH486deogNzcX5ubm8PLyQmZmJubNm4eRI0eiV69ecpcoiYcPHyIlJQW//vorhBA4c+YMXF1d0adPH2RmZmLmzJnYvn073nvvPb3+OevBgwe4c+cOnj59isePH+PixYto2LAh3NzcYGpqiq1bt+LZs2cYMGAA2rZtK+sYk5OToVAoEBkZiadPnyI2NhYjR46Eg4MDzp49i02bNmHLli0wMTHB5cuXYWxsjGbNmslWry5gXplXuTCvRce8Mq9y0fW8lrl9sP9JpVLh0qVLqFixIuzt7dG/f38oFAqsW7cO6enp+P777/H3339jzJgxCA0NhYmJCbp16yZ32cVSu3Zt1K5dG/Hx8Thx4gRWrlyJ9PR0bNu2DRcvXkTr1q1Ru3Zt7QEB+hh+AKhbty7q1q2L3NxcbNiwAY0aNYKTkxOqVq2KHTt2ICoqClZWVrh165b2VENyqVGjBvLy8nDv3j1cuXIFkyZNQosWLZCSkoJNmzahWbNmMDMzw4ULF7Bp0yY0adKkXE/YzCvzKifmtWiYV+ZVTrqe1zJ3FpF/srKywsGDB9GhQwe4uLjA3NwcixYtgomJCebOnYulS5dqf2KwtbVFzZo15S75nTVp0gTnz59HUlISGjRoAABo1KgRBg4ciKSkJIwePRq///67zFW+O7VajZiYGFhYWKBhw4bYsmULUlNT4e7ujnnz5sHf3x+HDx+Wu0wYGhpi0KBB8PLyQosWLfDkyRMsXrwYzZo1w6hRoxAbG4utW7eiefPmmDdvHgBoj+Iub5hX5lVuzGvhMa/Mq9x0Oa/lYhcRAMjNzUVOTg5mzJiB+vXrY/To0ahTpw4qVKiA77//Hl9++SUiIiJQvXp17al49PUbKPB8n7GlS5ciISEBTZo0wfjx49GwYUMsXrwYGo0G27Ztk7tESdy/fx/Xrl3DqVOnADw/KX779u1hbGyMpUuXal9rAwMDnXg9U1NTMWPGDNja2mL48OF4+vQpvv/+e1StWhWLFy9GdnY2KlasKHeZsmNemVddeD2Z18JhXplXXXg9dS6vshxaKZP4+HixcuVKERcXpz2qdNeuXaJly5bi/PnzQgghsrOztcvr69HA+eLj48Xnn38ufvvtN5Geni4mT54spk+frn08MjJSxMTEyFihNBITE8XatWvFiRMnRG5urhBCiICAAGFnZydu3bolhBBCrVZrl5fzdX306JFYu3atuH79ujh//rzw9PQUNjY2YuDAgWLy5Mli1KhRIiQkRCdqlRvzyrwKwbzqC+aVeRWCeX1RudmCnU+tVsPIyAgKhQJ79uzB8uXLcfDgQcTHx+Po0aN4/PgxunTpgmHDhsldqiRSU1NRrVo1LF++HNevX8fOnTsBPD95/pEjRwAA3t7e2vNfCj09MCMjIwMmJiYwNDTEjz/+iBUrVuDAgQPIzMzE8ePHkZqaCjs7OwwdOlTuUqFWq3Ht2jX4+fnB3t4etra22vPINm7cGBqNBg8ePICNjQ0A/X1NpMC8Mq9yY14Lj3llXuWmS3kt8/tg/1OFChWgUCiwb98++Pr6Yv/+/bh16xa+/PJL2NnZwc3NDT/88AN++uknuUuVRLVq1ZCbm4u0tDQ4OzsDAKKjo3Hy5EmYmZlhzJgxMDY2RlJSEoD/nSxf35iamsLQ0BD79u3DkiVLsH//fty5cwfTp0/HX3/9haFDhyIoKAgBAQFyl4oKFSrA0tISdnZ2mDlzJsLDw5Gbm4vmzZujUqVK2LVrF2bNmoXz588D0N/XRArMK/MqN+a18JhX5lVuupTXcrcFO9/Nmzfx+PFjtGvXDuPHj8ekSZPg6OgIADh58iTOnz8Pb29v7fL6+s0z37Vr1zBv3jy0bt0aly5dQseOHdGkSRM8ffoU3377LWrXro1u3brh3//+t9ylvpMXX9cJEyZgzJgx+PHHH9GjRw907doV3333HWbPnq29GpcuvK4TJkzAjBkz4ODggA0bNiAmJgZNmjRBamoqHBwcMG7cOFnr0wXMK/MK6MbryrwWjHllXgHdeF3lzGu524Kdr0mTJmjXrh3u37+PKlWqoEWLFtrHMjIyULlyZeTk5CAxMRGA/n7zzGdjY4PNmzfD1NQU77//Pvr3748nT57g0qVLWL16NXbs2IEjR47g5MmTcpf6TvJf14cPHwIAHB0dsXnzZoSGhsLV1VW73N9//w1A3tdVo9EgJSUFjx8/hrW1Nb7++mtcvnwZM2bMgK+vL2bOnIl9+/Zpx1KeMa/MK8C86gvmlXkFmNcyfx7sgpiYmCAhIQGnT5/GRx99hJSUFPTt2xdqtRre3t4wMDCAo6MjRo8erdcnjweA9957D+PGjcOTJ0+Qnp6Ob7/9Flu2bNGeF7J+/frIysp66Tn6Ol4TExNkZWXht99+g7OzM1atWoX9+/ejffv2iIyMxN69e+Hs7AwXFxfZxmdgYIDq1atj165d+OGHHxAeHo5PP/0ULVu2BAA8evQIlStXfuUKaPr6mkiBeWVemVf9wbwyr+U5r+V2F5EXXbt2Db6+vjAxMUHTpk0xePBgGBsb4/Hjx2jdujU8PDzg6upapi6R+/vvv+O///0vFixYgNzcXBgZGSEmJgaZmZmoX78+kpOT0aZNGwDPvwnm/+SjT65evYq5c+ciJycHSqUSY8eOxenTp3HgwAE8efIEmZmZGD9+PPr16yd3qXj06BHi4+PRvHlzGBsbIyYmBgsWLMD06dPRo0cPpKSk4MmTJ9otQfr6wSwF5pV5lRvzWnjMK/MqN7nyygb7/8vIyICpqSlOnz6NjRs3okqVKmjVqhW6du2Kq1evIjo6GmvWrAHw/GcPfQ1Fvvv372P8+PEYO3YshBAYNmwYjI2NcePGDYSEhODUqVMYMGAA3N3d5S71naSmpmqvrBUbG4ujR4+iZs2amDlzJjIzM3HmzBmdu7JYdHQ0fH19MX36dNjY2MDb2xvNmzdHXFwcBg8eDA8PD7lLlB3zyrzqCua1YMwr86orSjOvbLD/v/xvLEuWLIGhoSEWLlyI6Oho/PDDD0hOTkb37t0xadIkXLx4scxslYiLi4O/vz+aNGkCIyMjPHv2DO3atYOFhQXq168PDw8PTJ48GS4uLnKX+s4iIiKwd+9evP/++5gyZcorPwvpiqysLIwaNQr9+vVDv379MGDAAMyZMwdKpRLJycmYOHEiPv/8c7Rt2xaA/r8Hi4t5ZV51AfNaOMwr86oLSjuv+vsVUWL5/xNHjRqFqKgoHD58GHfu3EHNmjXh4OCA8ePHY9euXRg7diz++usv7XP0+ftJ06ZNsWLFCkyaNAnnzp1DTk4Ojh49irCwMFy7dg12dna4d+/eS8/R1/E2bdoUdnZ22vDr6qWNK1asCH9/f4wbNw7Hjh1D//794ebmBiEErKys0LRpUwghtPXr8wT0LphX5lUXMK+Fw7wyr7qgtPPKBvsfmjRpAj8/P8TExMDf3x+mpqaYNWsWdu3ahZ07d2Lo0KH4+eef8d133wHQ/w9MQ0ND5OTkQK1Wo0mTJli6dCnc3d2xcOFCREVFYeDAgUhLS8P9+/cB6O+HnpWVFSZNmqQNvy7//Ghubg4AePr0qbZOIyMjHD58GH///TcaNWqESZMm4fjx4zJWqRuYV+ZVbsxr4TGvzKvcSjOv5f4sIq/z/vvvY/LkyUhKSsLMmTPx448/Yu/evVi/fj3s7Oxw9epVfPrpp3ByckLt2rXlLvedKBQKGBsbw8fHB76+vlCr1Th79iwMDQ2xdetWxMXF4ddff8Xt27fRr18/jBo1Svuhp68/4ely+IH/TSp9+vTBzJkz8dNPP+HOnTu4e/cuRowYgcOHDyM3Nxc9e/aUuVLdwLwyr3JiXouGeWVe5VSaedXt/xMysra2xoYNGxAUFITt27djzZo1sLOzQ1ZWFuLj49G4cWMYGxsD+N/POtnZ2XKW/E5sbW2xevVqHD58GDExMdiwYQNsbW2RkJCAZ8+eYfPmzThw4ADCwsK0z3n8+LGMFZd9jRo1wurVq3Hr1i08evQIY8aMgYODA/Ly8jBmzBgA0L4u5R3zyrzKjXktPOaVeZVbaeSVBzkW4N69e8jIyICtrS1ycnIQExODb775Bh06dMCkSZMAAOnp6bh+/Tq+++47eHh4oEOHDjJXXXzx8fG4evUqUlNTkZycDCcnJyxYsACzZs1CdHQ0bGxs8OGHHyIrKwvu7u7Ytm0bLC0t9fKbtr7Ij6hCoUBERAR27tyJZs2aoVq1ali1ahW8vLwwbdo0vgZgXplX+TGvhce8Mq9yK8m8cheRAtSvXx8AkJOTg8jISOzcuRPdu3fXnrNzy5YtSElJQWxsLC5evIiGDRvq9QdArVq1UKtWLRw7dgznz59H3bp10aBBA1y6dAk9evRAgwYNtN+yd+3aBVNTU5krLvvyg33z5k0sX74cycnJ6Nu3Lx4/fgw7OzuMHDkSiYmJePr0KZo2bSpztfJiXplXuTGvhce8Mq9yK8m8cheRQsrLy8PBgwfRrl07DBs2DN9//z2GDRuG06dPw97eHoMHD8aYMWPg6ekpd6mSaNq0Kc6dO4ewsDDUq1cPUVFRaNy4MSIiIjBt2jSkpaW9FH7+EFLyqlWrhkaNGsHHxwdOTk44duwY3N3dkZSUhPnz5+PQoUOIjY3VLl+eXxPmlXmVG/NaeMwr8yq3ksgrdxEpgsTERERFReGLL75A8+bNcebMGfj7+6Njx47w8/PD3bt3sXz5cpiZmcldqiSuX7+OtWvXomLFihg+fDgqVKgAT09PbNiwAR999BEuXbqE9PR0dOnSBYD+XpFKn6SlpcHMzAxHjhzBjh07MGTIEOzbtw8zZsyAmZkZjh07hj59+mgvB1ueXxPmlXmVG/NaeMwr8yo3yfMqqEgePHgggoODhRBChIaGCicnJ7F582bh5OQk/vzzTyGEENnZ2drlNRqNHGVKJn8sERERokWLFuLEiRNCCCEuXbokNm7cKEaMGCF++OEHOUssd3JycsSoUaNEt27dxL59+8T9+/fFkydPxMqVK4Wrq6v4+OOPRUBAgNxl6gTmlXmVG/NaeMwr8yo3KfPKLdjvKCoqCl5eXujSpQv8/PyQmZmJsLAwNGzYEK1atQKg3988hRDIyMhAjx498OWXX6J69eo4ePAgOnbsiAYNGuD999+Hm5sbPvnkE3Tv3l3ucsuNBw8ewNraGhUqVMDdu3exd+9eqNVqdOrUCY6Ojhg0aBAWLVqEzp07y12qTmFemVc5MK/Fw7wyr3KQKq88yLGYxP8/R2X79u1haGiI3r17w9DQEBEREfj6669RrVo1DBgwAEOHDtXb8APPDwCoXLkyjh8/DjMzM0yYMAGdO3fGX3/9hevXryM+Ph7m5uZITU196XlCT8/hqS/q1q0LALh69Sp++uknGBsbo3fv3mjfvj2A5/uTJScny1miTmFemVc5Ma9Fw7wyr3KSKq9ssIsp/4pLP//8M2xtbdG9e3ccPXoUx48fx4cffoiRI0di3LhxsLKyQrdu3QDodygqV66MjIwMpKeno1atWpgwYQIePnyIoUOHokGDBujTpw+ioqJgamoKBwcH7f8ffR2vvrC0tISZmRkcHR214f/000+h0WjQt29fmavTHcwr86oLmNfCYV6ZV13wrnnlLiLvKD4+Ht988w1sbGwQExODpk2bYsKECQCA2bNno379+ujSpQtat24NQL8/BIDn3+i++OILdO3aFaGhoTA3N8fy5cuRkJCAdevWQQiBoUOHol+/fnKXWm5kZWWhYsWKAJ6HPyUlBV999RVq1qyp9+83qTGvzKvcmNfCY16ZV7m9S17197cVHVGrVi1Mnz4dt27dQu3atbXh//LLLxEcHIyqVavC398fAQEBAKDX4QeeX5FqxYoVCA8PB/B8nPfu3cO+fftgamqKadOm4YcffsBPP/0kb6HliImJCQBg/vz5uHPnDtasWYOaNWsC0P/3m9SYV+ZVbsxr4TGvzKvc3iWv3IItkcePH8PCwgIAsGrVKvz88884dOgQrK2tERYWhq1bt+Lbb7+FmZmZXu8zli8xMRF3795FVlYWfvvtNxgZGeHjjz+Gubk5fvzxR5w5cwarVq3SXu6WSt7Dhw9RsWJFVK9eXe5SdB7zyrzKjXktPOaVeZVbcfKq/+9EHZEffj8/P3z77bcICgqCtbU1Lly4gEOHDsHZ2RlVq1Z9Kfz6/N3G0tISzZo1Q3R0NHJycuDl5QVzc3PExMRg9+7d6NSpE8NfymrXrs3JupCYV+ZVbsxr4TGvzKvcipNXbsGW2K1bt2BqaopatWrhypUr2LlzJxo1aoQ2bdrA1NQUt2/fhpmZWZk4MAMAkpKSYGxsjKpVq+LcuXNYv349evTogSFDhpSZCwJQ2cW8Mq+kP5hX5lWfcAu2xBo3boxatWrh2rVr2Lp1K+rVq4e6desiKioKM2bMQFJSErZt24adO3cC0P99xmrWrImqVavi/PnzWLZsGXr06IH+/fsz/KQXmFfmlfQH88q86hM22CXEwsICtra2cHJyAgCoVCqYm5ujUqVK+P7773Hy5EncunVLr3/GelHNmjUxZMgQ9O/fnz97kt5hXon0B/NK+oC7iJQgtVqNChUqYMqUKejVqxeGDBmCCRMmoHLlyqhZsybmzJmjPUIV0P+fs/T5ilpEzCuR/mBeSdfx1SpBFSpUgFqthoWFBYyMnl/TZ9OmTTAxMUFmZiZMTEywadMm7N27F8Dzn7Py8vLkLPmdMPykz5hXIv3BvJKu4xbsUnD16lXMnj0bDRs2RLNmzTBu3DhUqlQJmzdvxqZNm9C+fXv07dsX7u7uiI2NBQA4ODjIXDVR+cS8EukP5pV0FS+VXgpsbW2xZcsWnDlzBpUrV0alSpWwfPlyBAcHIzg4GKampti+fTtSUlJgbW2NU6dO8QOASCbMK5H+YF5JV3ELtgwWL16M48ePIygoCDVq1AAAnD59Gv7+/li0aBGaNGkCQP/3GSMqC5hXIv3BvJKu4BbsUiSEQGJiIiIjI7Fv3z7UqFEDGRkZOH/+PL755hv07t1bG37g+T5j/BAgkgfzSqQ/mFfSNdyCLYP8o4HT09Nx4cIF7NixAz179kTPnj2xevVqVK5cGc2aNYO7u7vcpRKVe8wrkf5gXklX8LBUGeR/c758+TLWrl0LpVKJbt26wcPDAzVr1oSLiwv279+vPfqZiOTDvBLpD+aVdAV3EZFB/k9SdevWRb9+/TBw4EC4urpi8ODB8PLyAgB4eHjgwoULAP63rxh/ziIqfcwrkf5gXklXcAu2jN577z2MHTsW6enpsLGx0YYfAOrXrw8LCwsAz69SBfzvmzkRlT7mlUh/MK8kN27B1gHZ2dm4ffs2goOD0bFjR1SuXBlt2rRB7dq1sWPHDkRHR6NPnz4YOnQov2kTyYx5JdIfzCvJhQc56oi4uDjMmzcPNjY2qFixIvr3748qVapApVKhTp06mD17NlxcXDB69Gi5SyUq95hXIv3BvJIc2GDrkCdPnqBy5coICAhAXFwcbty4gY8++gj9+vXDpUuXEBgYiC1btqBSpUr8pk0kM+aVSH8wr1TauIuIDjE3NwcAZGRkwMTEBN9//z3u3r2LVatWIS4uDsOHD4epqSkSExNhaWnJDwEiGTGvRPqDeaXSxoMcdZCTkxP+/PNPREdH49GjRzAxMUGfPn3Qr18/REVFYdy4cQgLCwPAAzOI5Ma8EukP5pVKC3cR0VFxcXFYuXIlrl69Cjc3NwwfPhwJCQmYPn062rVrBzMzM7Ro0QIeHh5yl0pU7jGvRPqDeaXSwAZbhyUmJuLrr7/GoEGD8OzZMyxYsADu7u7w9PTEo0ePMHr0aGzZsgX/+te/5C6VqNxjXon0B/NKJY27iOgwS0tLzJgxA9nZ2Zg2bRo8PDzg6ekJjUaDiIgI2Nra4r333pO7TCIC80qkT5hXKmk8yFHHVapUCdWqVYO3tzdGjhwJAAgODsalS5fQrl07GBsby1whEeVjXon0B/NKJYm7iOiZn376CVeuXEG9evUwePBgVK1aVe6SiOgNmFci/cG8kpS4i4geSUtLw82bN2FpacnwE+k45pVIfzCvJDVuwdYzT548gaGhIapUqSJ3KURUAOaVSH8wryQlNthERERERBLiLiJERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhXiq9jIuOjsaKFStgamqK9PR0tGnTBpcvX0Z6ejqEEFi+fDnatm2L9PR0LF++HOfOnYOhoSF69eoFHx8fqNVqrFmzBmfPnkVeXh7s7OywcOFCmJmZyT00ojKHeSXSH8wrvQ0b7HLgxo0bCAsLg0qlwrfffovAwEAYGBhgx44d8Pf3R9u2bbFx40ZkZ2fj8OHDyMvLw/jx43HmzBmcPXsWhoaG2LdvHxQKBdatW4c1a9bA19dX7mERlUnMK5H+YF7pTdhglwPvvfce6tSpgzp16qBatWrYs2cP7t+/j+joaFSuXBkAEBERgXnz5sHQ0BCGhobYtWsXAOCrr77Cs2fPEBERAQBQq9WoUaOGbGMhKuuYVyL9wbzSm7DBLgdMTU0BAOHh4VixYgXGjRuHnj17onHjxjh48CAAwMjICAqFQvucR48eoWLFitBoNJg/fz66desGAEhPT0d2dnbpD4KonGBeifQH80pvwoMcy5HTp0+je/fucHd3h729PcLCwpCXlwcA6NSpE/bv3w+NRoOcnBx4e3vj7Nmz6NKlCwICApCTkwONRoNFixZh3bp1Mo+EqOxjXon0B/NK/8QGuxwZMWIEzp49C6VSieHDh6NevXp48OABNBoNpk2bhgoVKmDAgAFwc3NDt27d4OTkBC8vL9SpUwcDBw6Ei4sLhBCYO3eu3EMhKvOYVyL9wbzSPymEEELuIoiIiIiIygpuwSYiIiIikhAbbCIiIiIiCbHBJiIiIiKSEBtsIiIiIiIJscEmIiIiIpIQG2wiIiIiIgmxwSYiIiIiktD/Awadl22cjXJ2AAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 314 }, - { - "cell_type": "markdown", - "metadata": { - "id": "fQOR2R3XVa-U" - }, - "source": [ - "# Model cards for model reporting" - ] + "id": "EsTehBH2SW7f", + "outputId": "3cf05d43-1389-41a2-e2dc-7e7511833c2b" + } + }, + { + "cell_type": "markdown", + "source": [ + "Next optional section shows that `ThresholdOptimizer` more closely satisfies constraints on the training data than on the test data.\n", + "\n", + "### Postprocessing: Correctness check [OPTIONAL SECTION]" + ], + "metadata": { + "id": "umd3slsDmk0d" + } + }, + { + "cell_type": "markdown", + "source": [ + "We can verify that `ThresholdOptimizer` achieves false negative rate parity on the training dataset, meaning that the values of the false negative rate parity with respect to all groups are close on the training data." + ], + "metadata": { + "id": "Cs1kWzda8f0z" + } + }, + { + "cell_type": "code", + "execution_count": 64, + "source": [ + "# Record and evaluate the output of the ThresholdOptimizer on the training data\r\n", + "\r\n", + "Y_pred_postprocess_training = postprocess_est.predict(X_train_bal, sensitive_features=A_train_bal)\r\n", + "metricframe_postprocess_training = MetricFrame(\r\n", + " metrics=metrics_dict,\r\n", + " y_true=Y_train_bal,\r\n", + " y_pred=Y_pred_postprocess_training,\r\n", + " sensitive_features=A_train_bal\r\n", + ")\r\n", + "metricframe_postprocess_training.by_group" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
selection_ratefalse_negative_ratebalanced_accuracy
race
AfricanAmerican0.5050.3720.616
Caucasian0.5250.3790.599
Other0.4910.3850.612
Unknown0.4860.3830.613
\n", + "
" + ], + "text/plain": [ + " selection_rate false_negative_rate balanced_accuracy\n", + "race \n", + "AfricanAmerican 0.505 0.372 0.616\n", + "Caucasian 0.525 0.379 0.599\n", + "Other 0.491 0.385 0.612\n", + "Unknown 0.486 0.383 0.613" + ] + }, + "metadata": {}, + "execution_count": 64 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 203 }, - { - "cell_type": "markdown", - "metadata": { - "id": "MzFzSHrQBAgi" - }, - "source": [ - "_Note: The Python code in this section works in Google Colab, but it does not work on all local environments that we tested._" - ] + "id": "ttOZAVbgmplf", + "outputId": "c6bb375e-5e32-4508-cfd8-45f775618519" + } + }, + { + "cell_type": "code", + "execution_count": 65, + "source": [ + "# Evaluate the difference between the largest and smallest value of each metric\n", + "metricframe_postprocess_training.difference()" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "selection_rate 0.038\n", + "false_negative_rate 0.013\n", + "balanced_accuracy 0.018\n", + "dtype: object" + ] + }, + "metadata": {}, + "execution_count": 65 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "markdown", - "metadata": { - "id": "2RAXJDoyVbBT" - }, - "source": [ - "[Mitchell et al. (2019)](https://arxiv.org/abs/1810.03993) proposed the *model cards* framework for documenting and reporting model training details and deployment considerations. A _model card_ documents, for example, training and evaluation dataset summaries, ethical considerations, and quantitative performance results.\n", - "\n" - ] + "id": "Tty0cmJomp1h", + "outputId": "1d821a8d-3e0a-4082-8efe-740d09ceb1e0" + } + }, + { + "cell_type": "markdown", + "source": [ + "The value of `false_negative_rate_difference` on the training data is smaller than on the test data." + ], + "metadata": { + "id": "Kvdn7wBhCqXw" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "### Exercise: ThresholdOptimizer" + ], + "metadata": { + "id": "z2vLNUnK_P66" + } + }, + { + "cell_type": "markdown", + "source": [ + "In this exercise, we will create a `ThresholdOptimizer` by constraining the *true positive rate* (also known as the *recall score*). For any model, the *true positive rate* + *false negative rate* = 1. \n", + "\n", + "By trying to achieve the *true positive rate parity*, we should produce a `ThresholdOptimizer` with the same performance as our original `ThresholdOptimizer`.\n", + "\n" + ], + "metadata": { + "id": "0YUembo02yQ8" + } + }, + { + "cell_type": "markdown", + "source": [ + "#### 1.) Create a new ThresholdOptimizer with the constraint `true_positive_rate_parity` and objective `balanced_accuracy_score`." + ], + "metadata": { + "id": "ZJxE2eMuNF3_" + } + }, + { + "cell_type": "code", + "execution_count": 66, + "source": [ + "thresopt_exercise = ThresholdOptimizer(\n", + " estimator=unmitigated_pipeline,\n", + " constraints=\"true_positive_rate_parity\",\n", + " objective=\"balanced_accuracy_score\",\n", + " prefit=True,\n", + " predict_method='predict_proba'\n", + ")" + ], + "outputs": [], + "metadata": { + "id": "zD5kQu6gqyl6" + } + }, + { + "cell_type": "code", + "execution_count": 67, + "source": [ + "thresopt_exercise.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)\n", + "threshopt_pred = thresopt_exercise.predict(X_test, sensitive_features=A_test)" + ], + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\postprocessing\\_threshold_optimizer.py:309: UserWarning: The value of `prefit` is `True`, but `check_is_fitted` raised `NotFittedError` on the base estimator.\n", + "\n", + "If the provided base estimator has been fitted, this could mean that (1) its implementation does not conform to the sklearn estimator API, or (2) the enclosing ThresholdOptimizer has been cloned (for instance by `sklearn.model_selection.cross_validate`).\n", + "\n", + "In case (1), please file an issue with the base estimator developers, but continue to use the enclosing ThresholdOptimizer with `prefit=True`. In case (2), please use `prefit=False`.\n", + " type(self).__name__\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\postprocessing\\_interpolated_thresholder.py:115: UserWarning: The value of `prefit` is `True`, but `check_is_fitted` raised `NotFittedError` on the base estimator.\n", + "\n", + "If the provided base estimator has been fitted, this could mean that (1) its implementation does not conform to the sklearn estimator API, or (2) the enclosing InterpolatedThresholder has been cloned (for instance by `sklearn.model_selection.cross_validate`).\n", + "\n", + "In case (1), please file an issue with the base estimator developers, but continue to use the enclosing InterpolatedThresholder with `prefit=True`. In case (2), please use `prefit=False`.\n", + " warn(BASE_ESTIMATOR_NOT_FITTED_WARNING.format(type(self).__name__))\n" + ] + } + ], + "metadata": { + "id": "CxdAikCKqyuW" + } + }, + { + "cell_type": "markdown", + "source": [ + "#### 2.) Create a new `MetricFrame` object to process the results of this classifier." + ], + "metadata": { + "id": "xBLD77OENFN-" + } + }, + { + "cell_type": "code", + "execution_count": 68, + "source": [ + "thresop_metricframe = MetricFrame(\n", + " metrics=metrics_dict,\n", + " y_true=Y_test,\n", + " y_pred=threshopt_pred,\n", + " sensitive_features=A_test\n", + ")" + ], + "outputs": [], + "metadata": { + "id": "B12UNsZErImo" + } + }, + { + "cell_type": "markdown", + "source": [ + "#### 3.) Compare the performance of the two `ThresholdOptimizers`." + ], + "metadata": { + "id": "DEyEie-9NEew" + } + }, + { + "cell_type": "code", + "execution_count": 69, + "source": [ + "# Visualize the performance of the new ThresholdOptimizer\n", + "thresop_metricframe.by_group" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
selection_ratefalse_negative_ratebalanced_accuracy
race
AfricanAmerican0.4270.4000.598
Caucasian0.4500.3890.590
Other0.4120.4460.580
Unknown0.4690.4040.569
\n", + "
" + ], + "text/plain": [ + " selection_rate false_negative_rate balanced_accuracy\n", + "race \n", + "AfricanAmerican 0.427 0.400 0.598\n", + "Caucasian 0.450 0.389 0.590\n", + "Other 0.412 0.446 0.580\n", + "Unknown 0.469 0.404 0.569" + ] + }, + "metadata": {}, + "execution_count": 69 + } + ], + "metadata": { + "id": "uz9mS66YrWka" + } + }, + { + "cell_type": "code", + "execution_count": 70, + "source": [ + "# Compare the performance to the original ThresholdOptimizer\n", + "metricframe_postprocess.by_group" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
selection_ratefalse_negative_ratebalanced_accuracy
race
AfricanAmerican0.4270.4000.598
Caucasian0.4510.3890.590
Other0.4120.4550.574
Unknown0.4590.4360.557
\n", + "
" + ], + "text/plain": [ + " selection_rate false_negative_rate balanced_accuracy\n", + "race \n", + "AfricanAmerican 0.427 0.400 0.598\n", + "Caucasian 0.451 0.389 0.590\n", + "Other 0.412 0.455 0.574\n", + "Unknown 0.459 0.436 0.557" + ] + }, + "metadata": {}, + "execution_count": 70 + } + ], + "metadata": { + "id": "oByEABVGrXWo" + } + }, + { + "cell_type": "markdown", + "source": [ + "Similar to many unfairness mitigation approaches, `ThresholdOptimizer` produces randomized classifiers. Next optional section presents a heuristic strategy for converting a randomized `ThresholdOptimizer` into a deterministic one. In our scenario, this heursitic is quite effective and the resulting deterministic classifier has similar performance as the original `ThresholdOptimizer`.\n", + "\n", + "### Deployment considerations: Randomized predictions [OPTIONAL SECTION]" + ], + "metadata": { + "id": "wAUI3LdQbBCs" + } + }, + { + "cell_type": "markdown", + "source": [ + "When we were describing `ThresholdOptimizer` we said that it picks a separate threshold for each group. However, that is not quite correct. In fact,`ThresholdOptimizer`, for each group, picks two thresholds that are close to each other (say `threshold0` and `threshold1`) and then, at deployment time, randomizes between the two: choosing `threshold0` with some probability `p0` and `threshold1` with the remaining probability `p1=1-p0` (the specific probabilities are determined during training; for certain kinds of constraints, three thresholds are considered.)\n", + "\n", + "This means that the predictions are randomized. To achieve reproducible randomization, it is possible to provide an argument `random_state` to the `predict` method. However, in some settings, even such reproducible randomization is not acceptable and can be in fact viewed as a fairness issue, because of its arbitrariness.\n", + "\n", + "One derandomization heuristic is to replace the two thresholds by their weighted average, i.e., `threshold = p0*threshold0 + p1*threshold1`. That corresponds to the assumption that the values of the scores between the two thresholds are approximately uniformly distributed. Using this heuristic, we derandomize `ThresholdOptimizer`.\n", + "\n" + ], + "metadata": { + "id": "wlfeHwVC6dna" + } + }, + { + "cell_type": "markdown", + "source": [ + "The randomized model of the `ThresholdOptimizer` is stored as the field\n", + "`interpolated_thresholder_` in the fitted ThresholdOptimizer, which is itself a\n", + "valid estimator of type `InterpolatedThresholder`:" + ], + "metadata": { + "id": "nia26AD2BBKf" + } + }, + { + "cell_type": "code", + "execution_count": 71, + "source": [ + "interpolated = postprocess_est.interpolated_thresholder_\n", + "interpolated" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
InterpolatedThresholder(estimator=Pipeline(steps=[('preprocessing',\n",
+       "                                                   StandardScaler()),\n",
+       "                                                  ('logistic_regression',\n",
+       "                                                   LogisticRegression(max_iter=1000))]),\n",
+       "                        interpolation_dict={'AfricanAmerican': {'operation0': [>0.4807478640752806],\n",
+       "                                                                'operation1': [>0.529649951710353],\n",
+       "                                                                'p0': 0.8307815126050422,\n",
+       "                                                                'p1': 0.16921848739495782},\n",
+       "                                            'Caucasian': {'operation0': [>0.47659266634712427...\n",
+       "                                                          'p0': 0.6009999999998796,\n",
+       "                                                          'p1': 0.39900000000012037},\n",
+       "                                            'Other': {'operation0': [>0.45349385499174055],\n",
+       "                                                      'operation1': [>0.49548258291649594],\n",
+       "                                                      'p0': 0.6000000000000001,\n",
+       "                                                      'p1': 0.3999999999999999},\n",
+       "                                            'Unknown': {'operation0': [>0.42796506996906125],\n",
+       "                                                        'operation1': [>0.4498310411887575],\n",
+       "                                                        'p0': 0.7612727272727273,\n",
+       "                                                        'p1': 0.23872727272727268}},\n",
+       "                        predict_method='predict_proba', prefit=True)
StandardScaler()
LogisticRegression(max_iter=1000)
" + ], + "text/plain": [ + "InterpolatedThresholder(estimator=Pipeline(steps=[('preprocessing',\n", + " StandardScaler()),\n", + " ('logistic_regression',\n", + " LogisticRegression(max_iter=1000))]),\n", + " interpolation_dict={'AfricanAmerican': {'operation0': [>0.4807478640752806],\n", + " 'operation1': [>0.529649951710353],\n", + " 'p0': 0.8307815126050422,\n", + " 'p1': 0.16921848739495782},\n", + " 'Caucasian': {'operation0': [>0.47659266634712427...\n", + " 'p0': 0.6009999999998796,\n", + " 'p1': 0.39900000000012037},\n", + " 'Other': {'operation0': [>0.45349385499174055],\n", + " 'operation1': [>0.49548258291649594],\n", + " 'p0': 0.6000000000000001,\n", + " 'p1': 0.3999999999999999},\n", + " 'Unknown': {'operation0': [>0.42796506996906125],\n", + " 'operation1': [>0.4498310411887575],\n", + " 'p0': 0.7612727272727273,\n", + " 'p1': 0.23872727272727268}},\n", + " predict_method='predict_proba', prefit=True)" + ] + }, + "metadata": {}, + "execution_count": 71 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 112 }, - { - "cell_type": "markdown", - "metadata": { - "id": "oP_tklbmMoFH" - }, - "source": [ - "### Fill out the model card [OPTIONAL SECTION]" - ] + "id": "5w-c22tsZsvj", + "outputId": "48c9f6ef-9457-41d2-afe4-78965966e470" + } + }, + { + "cell_type": "markdown", + "source": [ + "The `interpolation_dict` is a dictionary which assign to each sensitive feature value two thresholds and two respective probabilities. Using our derandomization strategy, we can create a dictionary that represents a deterministic rule:\n" + ], + "metadata": { + "id": "8PkqiznTuL0l" + } + }, + { + "cell_type": "code", + "execution_count": 72, + "source": [ + "def create_deterministic(interpolate_dict):\n", + " \"\"\"\n", + " Creates a deterministic interpolation_dictionary from a randomized\n", + " interpolation_dictionary. The determinstic thresholds are created by taking\n", + " the weighted combinations of the two randomized thresholds for each sensitive\n", + " group.\n", + " \"\"\"\n", + " deterministic_dict = {}\n", + " for (race, operations) in interpolate_dict.items():\n", + " op0, op1 = operations[\"operation0\"]._threshold, operations[\"operation1\"]._threshold\n", + " p0, p1 = operations[\"p0\"], operations[\"p1\"]\n", + " deterministic_dict[race] = Bunch(\n", + " p0=0.0,\n", + " p1=1.0,\n", + " operation0=ThresholdOperation(operator=\">\",threshold=(p0*op0 + p1*op1)),\n", + " operation1=ThresholdOperation(operator=\">\",threshold=(p0*op0 + p1*op1))\n", + " )\n", + " return deterministic_dict" + ], + "outputs": [], + "metadata": { + "id": "F3a1Mw8koLUa" + } + }, + { + "cell_type": "code", + "execution_count": 73, + "source": [ + "deterministic_dict = create_deterministic(interpolated.interpolation_dict)\n", + "deterministic_dict" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "{'AfricanAmerican': {'p0': 0.0,\n", + " 'p1': 1.0,\n", + " 'operation0': [>0.4890230013753432],\n", + " 'operation1': [>0.4890230013753432]},\n", + " 'Caucasian': {'p0': 0.0,\n", + " 'p1': 1.0,\n", + " 'operation0': [>0.47664354593739394],\n", + " 'operation1': [>0.47664354593739394]},\n", + " 'Other': {'p0': 0.0,\n", + " 'p1': 1.0,\n", + " 'operation0': [>0.47028934616164275],\n", + " 'operation1': [>0.47028934616164275]},\n", + " 'Unknown': {'p0': 0.0,\n", + " 'p1': 1.0,\n", + " 'operation0': [>0.4331850736438724],\n", + " 'operation1': [>0.4331850736438724]}}" + ] + }, + "metadata": {}, + "execution_count": 73 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "markdown", - "metadata": { - "id": "QfI9oEwIkS6v" - }, - "source": [ - "In this section, we will create a model card for one of the models we trained." - ] + "id": "lbynyyy2CKxv", + "outputId": "09ac99a7-5338-4e31-9e02-eb4acbb9d5ec" + } + }, + { + "cell_type": "markdown", + "source": [ + "Now, we can create an `InterpolatedThresholder` that uses the same pre-fit estimator, but with derandomized thresholds." + ], + "metadata": { + "id": "euqycBW_3EB2" + } + }, + { + "cell_type": "code", + "execution_count": 74, + "source": [ + "deterministic_thresholder = InterpolatedThresholder(estimator=interpolated.estimator,\n", + " interpolation_dict=deterministic_dict,\n", + " prefit=True,\n", + " predict_method='predict_proba')" + ], + "outputs": [], + "metadata": { + "id": "R3qKiaMiPOaG" + } + }, + { + "cell_type": "code", + "execution_count": 75, + "source": [ + "deterministic_thresholder.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" + ], + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\postprocessing\\_interpolated_thresholder.py:115: UserWarning: The value of `prefit` is `True`, but `check_is_fitted` raised `NotFittedError` on the base estimator.\n", + "\n", + "If the provided base estimator has been fitted, this could mean that (1) its implementation does not conform to the sklearn estimator API, or (2) the enclosing InterpolatedThresholder has been cloned (for instance by `sklearn.model_selection.cross_validate`).\n", + "\n", + "In case (1), please file an issue with the base estimator developers, but continue to use the enclosing InterpolatedThresholder with `prefit=True`. In case (2), please use `prefit=False`.\n", + " warn(BASE_ESTIMATOR_NOT_FITTED_WARNING.format(type(self).__name__))\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
InterpolatedThresholder(estimator=Pipeline(steps=[('preprocessing',\n",
+       "                                                   StandardScaler()),\n",
+       "                                                  ('logistic_regression',\n",
+       "                                                   LogisticRegression(max_iter=1000))]),\n",
+       "                        interpolation_dict={'AfricanAmerican': {'operation0': [>0.4890230013753432],\n",
+       "                                                                'operation1': [>0.4890230013753432],\n",
+       "                                                                'p0': 0.0,\n",
+       "                                                                'p1': 1.0},\n",
+       "                                            'Caucasian': {'operation0': [>0.47664354593739394],\n",
+       "                                                          'operation1': [>0.47664354593739394],\n",
+       "                                                          'p0': 0.0,\n",
+       "                                                          'p1': 1.0},\n",
+       "                                            'Other': {'operation0': [>0.47028934616164275],\n",
+       "                                                      'operation1': [>0.47028934616164275],\n",
+       "                                                      'p0': 0.0, 'p1': 1.0},\n",
+       "                                            'Unknown': {'operation0': [>0.4331850736438724],\n",
+       "                                                        'operation1': [>0.4331850736438724],\n",
+       "                                                        'p0': 0.0, 'p1': 1.0}},\n",
+       "                        predict_method='predict_proba', prefit=True)
StandardScaler()
LogisticRegression(max_iter=1000)
" + ], + "text/plain": [ + "InterpolatedThresholder(estimator=Pipeline(steps=[('preprocessing',\n", + " StandardScaler()),\n", + " ('logistic_regression',\n", + " LogisticRegression(max_iter=1000))]),\n", + " interpolation_dict={'AfricanAmerican': {'operation0': [>0.4890230013753432],\n", + " 'operation1': [>0.4890230013753432],\n", + " 'p0': 0.0,\n", + " 'p1': 1.0},\n", + " 'Caucasian': {'operation0': [>0.47664354593739394],\n", + " 'operation1': [>0.47664354593739394],\n", + " 'p0': 0.0,\n", + " 'p1': 1.0},\n", + " 'Other': {'operation0': [>0.47028934616164275],\n", + " 'operation1': [>0.47028934616164275],\n", + " 'p0': 0.0, 'p1': 1.0},\n", + " 'Unknown': {'operation0': [>0.4331850736438724],\n", + " 'operation1': [>0.4331850736438724],\n", + " 'p0': 0.0, 'p1': 1.0}},\n", + " predict_method='predict_proba', prefit=True)" + ] + }, + "metadata": {}, + "execution_count": 75 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 273 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "OViYhOAHPftT" - }, - "outputs": [], - "source": [ - "mct = ModelCardToolkit()\n", - "model_card = mct.scaffold_assets()\n" - ] + "id": "XLNNadraaFMT", + "outputId": "b3a7edc0-dfe1-4746-f208-0526ad4019a9" + } + }, + { + "cell_type": "code", + "execution_count": 76, + "source": [ + "y_pred_postprocess_deterministic = deterministic_thresholder.predict(X_test, sensitive_features=A_test)" + ], + "outputs": [], + "metadata": { + "id": "TPRwWuuWY2Yy" + } + }, + { + "cell_type": "code", + "execution_count": 77, + "source": [ + "mf_deterministic = MetricFrame(\n", + " metrics=metrics_dict,\n", + " y_true=Y_test,\n", + " y_pred=y_pred_postprocess_deterministic,\n", + " sensitive_features=A_test\n", + ")" + ], + "outputs": [], + "metadata": { + "id": "q3834uLWeXWO" + } + }, + { + "cell_type": "markdown", + "source": [ + "Now compare the two models in terms of their disaggregated metrics:" + ], + "metadata": { + "id": "2tnGhMYzDloo" + } + }, + { + "cell_type": "code", + "execution_count": 78, + "source": [ + "mf_deterministic.by_group" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
selection_ratefalse_negative_ratebalanced_accuracy
race
AfricanAmerican0.4240.4050.597
Caucasian0.4510.3890.590
Other0.4050.4370.588
Unknown0.4660.4040.571
\n", + "
" + ], + "text/plain": [ + " selection_rate false_negative_rate balanced_accuracy\n", + "race \n", + "AfricanAmerican 0.424 0.405 0.597\n", + "Caucasian 0.451 0.389 0.590\n", + "Other 0.405 0.437 0.588\n", + "Unknown 0.466 0.404 0.571" + ] + }, + "metadata": {}, + "execution_count": 78 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 203 }, - { - "cell_type": "markdown", - "metadata": { - "id": "Lnr6aRjIF_t6" - }, - "source": [ - "The first section of the Model Card is the _model details_ section. In _model details_, we fill in some basic information for our model.\n", - "\n", - "\n", - "* _Name_: A name for the model\n", - "* _Overview_: A brief description of the model and its intended use case\n", - "* _Owners_: Name of individual(s) or group who created the model.\n", - "* _References_: Any external links or references\n", - "\n" - ] + "id": "NIkoljGTeXZ0", + "outputId": "353905d4-3954-421a-9bf1-adbd6a8febd1" + } + }, + { + "cell_type": "code", + "execution_count": 79, + "source": [ + "metricframe_postprocess.by_group" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
selection_ratefalse_negative_ratebalanced_accuracy
race
AfricanAmerican0.4270.4000.598
Caucasian0.4510.3890.590
Other0.4120.4550.574
Unknown0.4590.4360.557
\n", + "
" + ], + "text/plain": [ + " selection_rate false_negative_rate balanced_accuracy\n", + "race \n", + "AfricanAmerican 0.427 0.400 0.598\n", + "Caucasian 0.451 0.389 0.590\n", + "Other 0.412 0.455 0.574\n", + "Unknown 0.459 0.436 0.557" + ] + }, + "metadata": {}, + "execution_count": 79 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 203 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "YKvUW25cPewp" - }, - "outputs": [], - "source": [ - "model_card.model_details.name = \"Diabetes Re-Admission Risk model\"\n", - "model_card.model_details.overview = \"This model predicts whether a patient will be re-admitted into a hospital within 30 days.\"\n", - "model_card.model_details.owners = [{\n", - " \"name\": \"Fairlearn Team\",\n", - " \"contact\": \"https://fairlearn.org/\"\n", - "}]\n", - "model_card.model_details.reference = [\n", - " \"https://archive.ics.uci.edu/ml/datasets/Diabetes+130-US+hospitals+for+years+1999-2008\"\n", - "]\n", - "model_card.model_details.version.name = \"v1.0\"\n", - "model_card.model_details.version.date = str(date.today())\n", - "model_card.model_details.license = \"MIT License\"\n" - ] + "id": "mVimPa2nlFpv", + "outputId": "7c197ff9-4eed-4e1d-d00d-6b16ddd34bb6" + } + }, + { + "cell_type": "markdown", + "source": [ + "The differences are generally small except for the *Unknown* group, whose false negative rate goes down and balanced accuracy goes up." + ], + "metadata": { + "id": "JOi4dwLKECHS" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Reductions approach with `ExponentiatedGradient`" + ], + "metadata": { + "id": "NU_rncBQ0lab" + } + }, + { + "cell_type": "markdown", + "source": [ + "With the `ThresholdOptimizer`, we took a fairness-unaware model and transformed the model's decision boundary to satisfy our fairness constraints. One limitation of `ThresholdOptimizer` is that it needs access to the sensitive features at deployment time.\n", + "\n", + "In this section, we will show how to use the _reductions_ approach of [Agarwal et. al (2018)](https://arxiv.org/abs/1803.02453) to obtain a model that satisfies the fairness constraints, but does not need access to sensitive features at deployment time.\n", + "\n", + "Terminology \"reductions\" refers to another kind of a wrapper approach, which instead of wrapping an already trained model, wraps any standard classification or regression algorithm, such as \n", + "`LogisticRegression`. In other words, an input to a reduction algorithm is an object that supports training on any provided (weighted) dataset. In addition, a reduction algorithm receives a data set that includes sensitive features. The goal, like with post-processing, is to optimize a performance metric (such as classification accuracy) subject to fairness constraints (such as an upper bound on differences between false negative rates).\n", + "\n", + "The main reduction algorithm algorithm in Fairlearn is `ExponentiatedGradient`. It creates a sequence of reweighted datasets and retrains the wrapped model on each of them. The \n", + "retraining process is guaranteed to find a model that satisfies the fairness constraints while optimizing the performance metric.\n", + "\n", + "The model returned by `ExponentiatedGradient` consists of several inner models, returned by the wrapped estimator. At deployment time, `ExponentiatedGradient` randomizes among these models according to a specific probability weights." + ], + "metadata": { + "id": "bpyBnqNLZ-w_" + } + }, + { + "cell_type": "markdown", + "source": [ + "To instantiate an `ExponentiatedGradient` model, we pass in two parameters:\n", + "\n", + "* A base `estimator` (an object that supports training)\n", + "* Fairness `constraints` (an object of type `Moment`).\n", + "\n", + "The constraints supported by `ExponentiatedGradient` are more general than those supported by `ThresholdOptimizer`. For example, rather than requiring that false negative rates be equal, it is possible to specify the maxium allowed difference or ratio between the largest and the smallest value.\n" + ], + "metadata": { + "id": "ZvT_qeduHCn8" + } + }, + { + "cell_type": "code", + "execution_count": 80, + "source": [ + "expgrad_est = ExponentiatedGradient(\n", + " estimator=LogisticRegression(max_iter=1000, random_state=random_seed),\n", + " constraints=TruePositiveRateParity(difference_bound=0.02)\n", + ")" + ], + "outputs": [], + "metadata": { + "id": "ToZdrYen0tJZ" + } + }, + { + "cell_type": "markdown", + "source": [ + "The constraints above are expressed for the true positive parity, they require that the difference between the largest and the smallest true positive rate (TPR) across all groups be at most 0.02. Since false negative rate (FNR) is equal to 1-TPR, this is equivalent to requiring that the difference between the largest and smallest FNR be at most 0.02." + ], + "metadata": { + "id": "bLFk3SCxrskU" + } + }, + { + "cell_type": "code", + "execution_count": 81, + "source": [ + "# Fit the exponentiated gradient model\n", + "expgrad_est.fit(X_train_bal, Y_train_bal, sensitive_features=A_train_bal)" + ], + "outputs": [ + { + "output_type": "stream", + "name": "stderr", + "text": [ + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n", + "c:\\users\\v-manandi\\anaconda3\\envs\\scipy_tutorial\\lib\\site-packages\\fairlearn\\reductions\\_moments\\utility_parity.py:251: FutureWarning: Using the level keyword in DataFrame and Series aggregations is deprecated and will be removed in a future version. Use groupby instead. df.sum(level=1) should use df.groupby(level=1).sum().\n", + " lambda_event = (lambda_vec[\"+\"] - self.ratio * lambda_vec[\"-\"]).sum(level=_EVENT) / \\\n" + ] + }, + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
ExponentiatedGradient(constraints=,\n",
+       "                      estimator=LogisticRegression(max_iter=1000,\n",
+       "                                                   random_state=445),\n",
+       "                      nu=0.002303268819590091)
LogisticRegression(max_iter=1000, random_state=445)
" + ], + "text/plain": [ + "ExponentiatedGradient(constraints=,\n", + " estimator=LogisticRegression(max_iter=1000,\n", + " random_state=445),\n", + " nu=0.002303268819590091)" + ] + }, + "metadata": {}, + "execution_count": 81 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 81 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "avEdkdzAPe2S" - }, - "outputs": [], - "source": [ - "model_card.considerations.use_cases = [\"High-Risk Patient Care Management\"]\n", - "model_card.considerations.users = [\"Medical Professionals\", \"ML Researchers\"]\n", - "model_card.considerations.limitations = [\n", - " \"\"\"\n", - " This model will not generalize to hospitals outside of the United States. Features, such as those encoding insurance\n", - " information, are inherently tied to the U.S healthcare system.\n", - " In addition, this model is intended for patients who are admitted into U.S. hospitals for diabetes-related illnessess.\n", - " \"\"\"\n", - "]\n", - "model_card.considerations.ethical_considerations = [{\n", - " \"name\": (\"Low sample sizes of certain racial groups could lead to poorer performance on these groups\"),\n", - " \"mitigation_strategy\": \"Collect additional data points from more hospitals.\"\n", - "}]" - ] + "id": "UFSF5Wn-3M-H", + "outputId": "43f639cf-58ef-4425-b772-3daec9d9ff3d" + } + }, + { + "cell_type": "markdown", + "source": [ + "Similarly to `ThresholdOptimizer` the predictions of `ExponentiatedGradient` models are randomized. If we want to assure reproducible results, we can pass `random_state` to the `predict` function. " + ], + "metadata": { + "id": "VsCeOlKFDQZZ" + } + }, + { + "cell_type": "code", + "execution_count": 82, + "source": [ + "# Record and evaluate predictions on test data\n", + "\n", + "Y_pred_reductions = expgrad_est.predict(X_test, random_state=random_seed)\n", + "metricframe_reductions = MetricFrame(\n", + " metrics=metrics_dict,\n", + " y_true=Y_test,\n", + " y_pred=Y_pred_reductions,\n", + " sensitive_features=A_test\n", + ")\n", + "metricframe_reductions.by_group" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
selection_ratefalse_negative_ratebalanced_accuracy
race
AfricanAmerican0.4080.4240.595
Caucasian0.3980.4450.588
Other0.3380.5180.580
Unknown0.3440.5740.545
\n", + "
" + ], + "text/plain": [ + " selection_rate false_negative_rate balanced_accuracy\n", + "race \n", + "AfricanAmerican 0.408 0.424 0.595\n", + "Caucasian 0.398 0.445 0.588\n", + "Other 0.338 0.518 0.580\n", + "Unknown 0.344 0.574 0.545" + ] + }, + "metadata": {}, + "execution_count": 82 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 203 }, - { - "cell_type": "markdown", - "metadata": { - "id": "orPNYAZFwGVM" - }, - "source": [ - "The next two sections of the model card are meant to provide the reader with information about the data used to train and evaluate the model. For each of these sections, we provide a brief `description` of the data and then submit a `visualization` of the distribution of labels in the dataset." - ] + "id": "YYz7GAqf4cbp", + "outputId": "72957bdb-41c5-41c6-b9c5-7f0ba8fdf610" + } + }, + { + "cell_type": "code", + "execution_count": 83, + "source": [ + "# Evaluate the difference between the largest and smallest value of each metric\n", + "metricframe_reductions.difference()" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "selection_rate 0.070\n", + "false_negative_rate 0.150\n", + "balanced_accuracy 0.050\n", + "dtype: object" + ] + }, + "metadata": {}, + "execution_count": 83 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "khWJN_vqm35e" - }, - "outputs": [], - "source": [ - "model_card.model_parameters.data.train.graphics.description = (\n", - " f\"{X_train_bal.shape[0]} rows with {X_train_bal.shape[1]} features. \"\n", - " f\"The original training data set was undersampled to allow for an equal number of positive and negative labeled instances.\"\n", - ")\n", - "\n", - "model_card.model_parameters.data.train.graphics.collection = [\n", - " {\"name\": \"Sensitive Features\", \"image\": sensitive_train},\n", - " {\"name\": \"Target Label\", \"image\": outcome_train} \n", - "]" - ] + "id": "idYvm9lq4mh3", + "outputId": "0848b976-8651-45ae-ddb3-b906b9417ac5" + } + }, + { + "cell_type": "markdown", + "source": [ + "While there is a decrease in the false negative rate difference from the unmitigated model, this decrease is not as substantial as with `ThresholdOptimizer`. Note, however, that `ThresholdOptimizer` was able to use the sensitive feature (i.e., race) at deployment time." + ], + "metadata": { + "id": "qpxYOozouVx9" + } + }, + { + "cell_type": "markdown", + "source": [ + "### Explore individual predictors" + ], + "metadata": { + "id": "IzTeHhWG4nJJ" + } + }, + { + "cell_type": "markdown", + "source": [ + "During the training process, the `ExponentiatedGradient` algorithm iteratively trains multiple inner models on a reweighted training dataset. The algorithm stores each of these predictors and then randomizes among them at deployment time.\n", + "\n", + "In many applications, the randomization is undesirable, and also using multiple inner models can pose issues for interpretability. However, the inner models that `ExponentiatedGradient` relies on span a variety of fairness-accuracy trade-offs, and they could be considered for stand-alone deployment: addressing the randomization and interpretability issues, while possibly offering additional flexibility thanks to a variety of trade-offs. \n", + "\n", + "In this section explore the performance of the individual predictors learned by the `ExponentiatedGradient` algorithm. First, note that since the base estimator was `LogisticRegression` all these predictors are different logistic regression models:" + ], + "metadata": { + "id": "o7qCmHeYKWGp" + } + }, + { + "cell_type": "code", + "execution_count": 84, + "source": [ + "predictors = expgrad_est.predictors_\n", + "predictors" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "0 LogisticRegression(max_iter=1000, random_state...\n", + "1 LogisticRegression(max_iter=1000, random_state...\n", + "2 LogisticRegression(max_iter=1000, random_state...\n", + "3 LogisticRegression(max_iter=1000, random_state...\n", + "4 LogisticRegression(max_iter=1000, random_state...\n", + "5 LogisticRegression(max_iter=1000, random_state...\n", + "6 LogisticRegression(max_iter=1000, random_state...\n", + "7 LogisticRegression(max_iter=1000, random_state...\n", + "8 LogisticRegression(max_iter=1000, random_state...\n", + "dtype: object" + ] + }, + "metadata": {}, + "execution_count": 84 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "h3fiatJ6nGQf" - }, - "outputs": [], - "source": [ - "model_card.model_parameters.data.eval.graphics.description = (\n", - " f\"{X_test.shape[0]} rows with {X_test.shape[1]} columns\"\n", - ")\n", - "\n", - "model_card.model_parameters.data.eval.graphics.collection = [\n", - " {\"name\": \"Sensitive Features\", \"image\": sensitive_test},\n", - " {\"name\": \"Target Label\", \"image\": outcome_test}\n", - "]" - ] + "id": "II_YtUZg3Ue4", + "outputId": "76948908-4240-4d6b-d1db-38cbeda7d5be" + } + }, + { + "cell_type": "code", + "execution_count": 85, + "source": [ + "# Collect predictions by all predictors and calculate balanced error\n", + "# as well as the false negative difference for all of them\n", + "\n", + "sweep_preds = [clf.predict(X_test) for clf in predictors]\n", + "balanced_error_sweep = [1-balanced_accuracy_score(Y_test, Y_sweep) for Y_sweep in sweep_preds]\n", + "fnr_diff_sweep = [false_negative_rate_difference(Y_test, Y_sweep, sensitive_features=A_test) for Y_sweep in sweep_preds]" + ], + "outputs": [], + "metadata": { + "id": "K3Bh2IVm4Ynj" + } + }, + { + "cell_type": "code", + "execution_count": 86, + "source": [ + "# Show the balanced error / fnr difference values of all predictors on a raster plot \n", + "\n", + "plt.scatter(balanced_error_sweep, fnr_diff_sweep, label=\"ExponentiatedGradient - Iterations\")\n", + "for i in range(len(predictors)):\n", + " plt.annotate(str(i), xy=(balanced_error_sweep[i]+0.001, fnr_diff_sweep[i]+0.001))\n", + "\n", + "# Also include in the plot the combined ExponentiatedGradient model\n", + "# as well as the three previously fitted models\n", + "\n", + "plt.scatter(1-balanced_accuracy_score(Y_test, Y_pred_reductions),\n", + " false_negative_rate_difference(Y_test, Y_pred_reductions, sensitive_features=A_test),\n", + " label=\"ExponentiatedGradient - Combined model\")\n", + "plt.scatter(1-balanced_accuracy_score(Y_test, Y_pred),\n", + " false_negative_rate_difference(Y_test, Y_pred, sensitive_features=A_test),\n", + " label=\"Unmitigated\")\n", + "plt.scatter(1-balanced_accuracy_score(Y_test, Y_pred_postprocess),\n", + " false_negative_rate_difference(Y_test, Y_pred_postprocess, sensitive_features=A_test),\n", + " label=\"ThresholdOptimizer\")\n", + "plt.scatter(1-balanced_accuracy_score(Y_test, y_pred_postprocess_deterministic),\n", + " false_negative_rate_difference(Y_test, y_pred_postprocess_deterministic, sensitive_features=A_test),\n", + " label=\"ThresholdOptimizer (DET)\")\n", + "\n", + "plt.xlabel(\"Balanced Error Rate\")\n", + "plt.ylabel(\"False Negative Rate Difference\")\n", + "plt.legend(bbox_to_anchor=(1.9,1))\n", + "plt.show()" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAArIAAAEJCAYAAABonMSrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABhiUlEQVR4nO3dd1yV5fvA8c9huwBBcJuKG/fWUDM1tyjuRQ5wZRm5N+JGE/0qLsyZAwfO0sw0KrVylAKCA7fIEJS9z/n9wc+TCHgA2V7v18vXi+d+1nU9QF3c537uW6FSqVQIIYQQQghRyGjldwBCCCGEEEJkhxSyQgghhBCiUJJCVgghhBBCFEpSyAohhBBCiEJJClkhhBBCCFEoSSErhBBCCCEKJSlkhRBCCCFEoaST3wHkhpcvo1Eqi+b0uKamJQkNjcrvMHKV5Fg0fAg5QsHO88yZH9m3bzcKhQJ9fQMcHKZTt269NMdpaSkoXbpEPkQohBDvp0gWskqlqsgWskCRzu01ybFo+BByhIKZ5+PHD3F1Xct33+2lTJkyXL78B7NnT8PD44f8Dk0IIXKMDC0QQogiSFdXj5kz51OmTBkA6tSpR1hYKImJifkcmRBC5Jwi2SMrhBAfuvLlK1C+fAUAVCoV69e7YGXVHl1d3XyOTAghco4UskIIUYTFxsaydKkjwcFBfPvt+vwOJ9cplUqePHlCdHQ0qoI34kMIkQUKBZQoUYLKlSujpZX+IAIpZIUQoogKDAxk5kwHqlatyvr1m9HXN8jvkHLdixcvSEpSUq5cZRQKGT0nRGGmUikJC3vBixcvMDc3T/cYKWSFEKKQu+wTiIenP6ER8Zga6mPTwQLLysX48stxdO/eizFjxuV3iHkmLOwlpqZlpYgVoghQKLQwMipNWFiQFLJCCFEUXfYJZNdpPxKSlACERsSz67QflVQ3CAoK5LfffuW3335VH79u3UaMjIzzJ9g8kJycjLa2/K9NiKJCW1uHpKTkDPfLb7sQQhRiHp7+6iL2tYQkJeGGzfnttyn5FFX+UigU+R2CECKHaPp9ls9ehBCiEAuNiM9Su8g7AQEBWFm1ZOTIIan+HT7snt+hpcvNbRP//nsdgKVLnfD1vfXO4ydNytqQlWvXrjJxor16+/ZtPxwcvmTAAGsGDLBmypRJPHhwP+uBv8HJaSGnTp0gJCQEB4cvs3WNgIBnLF26KEvnvJmbj483Gzasy9a93xWLr+8tli51ypHrFiXSIyuEEIWYqaF+ukWrqaF+PkQj3lamjBl79hzI7zAy5fr16zRt2gKAuXMXZOL4q9m+1+PHj5kyZRILFjjRtq0VAL/99isODl9y8OBR9PT0sn1tADMzM1xcsjdLx/Pnz3n69Gm27/3gwX3CwsKyfX5GsdStWy9T35cPjRSyQghRiNl0sEg1RhZAT0cLmw4W+RhV4XLJ+zmHLvgTGh6HqZEBAzta0LZ++Vy9p5+fLw4OX7Jv3yG0tLT4/POhODu74Ol5gSdPHvP06VMiIl7Rt29/Roz4HKVSiYvLaq5e/RuFQkG3bj2xtR3FtWtX2bVrOwYGBjx8+AALixo4OS1DV1eXH388xYED+1CplNSpU5dp02ahr69Pz56f0bFjJ27e/BdtbW2WLl3Jv//+g5/fLZYvd2LFim/59ltn7OzG06hRY5ydl3P//j3CwsKoUuUjVqxYjavr/wAYM8aW7dt3c/nyRdzcNpOUlET58hWYM2c+RkbG/PXXZdau/RY9PT0++qiaOv+9e3fRs2cfdREL0L79J4SGviAmJpoXL17g4DAZIyNj9PT0WLlyNUuXOhEcHMSLFyE0btyUhQsXA7Bu3RouXvydMmXMUCqTadq0GQEBAUyaZM+xYz8QGhrKypVLCQoKREtLi4kTv6Rly1a4uW0mJCSEJ08eExj4nD59+jJ6tB1r1jgTEPCMVauWM3367Cx9XyMjI3Fz20xsbAw7dmzD1nY069ev5fr1ayiVyfTs2ZuhQ0dw7dpVXF3XkZycjIVFDSZMmMyyZYuIjIwkNPQFXbp044svvkoVy6efdmHbti1s2uTG48ePWL58CRER4RQrVoxvvplBvXqWODktpGTJkvj5+RIcHISd3Th69bLmypW/2LBhHQqFglKlDFm8eBnGxqVz4Cc5/0khK4QQhVgby3IAaWYteN0u3u2S93O2/+BLQuL/vywXHsf2H3wBcqSYffEihJEjh6RqW7hwMXXq1MXGpj/r168lKSkJG5uB1KpVG0/PC/j738PNbSdKZTKjRg2nefOWeHt7ERwcxPffu5OYmMCkSeOwsLDAwKAYXl43cHf3oEwZM+zsPufPPy9RsWIljh/3wM1tB/r6+mzcuJ69e/cwZowdoaEvaNGiJdOmzWTdujUcOuTOlCnfcPLkcezsxlOjRk11rF5eN9DV1WHbtl0olUq++GI8ly5dZOrUGRw6dIDt23fz8uVLNm5cj6vrVgwNDTl69DAbNvyP6dNn4eS0kA0bNlOtWvVUH4t7e3sxceLkNM+rX78BAMTExPLo0UM8PE5RoUIFzp49Q82atVi2zJnExESGDu3P7du+BAQEcOfObfbvP0RkZBQjRgxOc00Xl1X06mVN+/YdePEihPHjx7J7934A7t27y5Yt3xEZGcmAAX0YMGAw33wzg23btmS5iAUoVaoU9vYTuH79GqNH2+HhcRiA3bv3kZCQwJQpX1C3bj0AHj9+xLFjP1CyZCn27t1Nly7d6NmzN1FRkfTp04Phw0emiuXatf96wB0d5zFy5Cg6duyEt/dN5syZzsGDxwAICgpky5bv8Pe/x6RJKYXsjh3fMXPmXOrVs8TdfT+3b/vRqlWbLOdXEEkhK4QQhVwby3JSuGbToQv+6iL2tYREJYcu+OdIIfuuoQWjR9sxatQI9PX1cXRcrG7/7LNuFC9eHAArqw5cu3YFb28vevbsjba2NtraxejatTtXrvxNu3YdqF7dAnPzsgBUrVqNiIgIAgOv8OTJE+zsPgcgMTGR2rXrqO/RunVbACwsLPjnn+sZxt+kSTOMjIw5fNidhw8f8vTpY2JjY1Id4+PjRVBQIF98kTJmVqlUYmhoiL//PcqUKUO1atUB6NmzF1u2bFKf9+ZLPJMnTyA8/BVRUVFMmvQllpYNKF3ahAoVKqifiY+PNwcO7OXhwweEh4cTExPL9etX+eSTT9HR0aV06dK0bftxmhyuXPmbR48e4uaWcu+kpCSePUv5uL5Zs+bo6upiYmKCoaEhUVGRGT6L7Lhy5S/u3r3NtWtXAIiNjeHevXtUq1adKlWqUrJkKQCGD7fl2rUr7N27G39/f5KSEomNjU33mjExMTx9+oSOHTsBUL9+QwwNjXj06CEArVq1QaFQYGFRg4iIcADatWvPzJlT6dDhE9q1+4RWrVrnaJ75SQpZIYQQH6zQ8LgsteekyMgoYmJiiImJISIiXP1Rr7a2tvoYlUqJtrY2SmXqYlulUpGcnDIlkZ7em+OhFahUKpRKJZ06dWHq1BlASvHz+ngAfX39N47POMbffvPEzW0TgwcPpVevPoSHv0L11glKpZKGDRuzevVaAOLj44mJiSEo6HmqY9/Mq149S27e/JePP24HwIYNm4GUl7Xi4+PfihEOHjzAhQvnsLa2YeDAVvj7+///tRWoVMp07/FffMls2LAFIyMjAEJCQjAxMcHT88JbY3Hf/Sy2bt3E7797AmBvP5H27TtkfPAbz+aLL6aoi85Xr15iYFAMHx/vVPmtW7eGgIBnfPZZN9q3/4QrV/7KMBaVSpnme5D65yElpzf/UBg6dARWVu25ePF3XF3XceuWN6NH22mMvzCQWQsKmPXrXbCx6cmoUcMYNWoYCxZk/aMNIYQQmWNqlP5qZxm156TVq1cwYMBg+vcfyKpVK9Ttnp4XSExMJCIigj/++J1WrdrQvHkLfvjhJMnJycTFxfLTT6dp1qx5htdu2rQ5np4XCAsLQ6VS4ey8jAMH9r4zHm1t7VTFLqT0KHbq1IVevawxNS3Dv/9eVxfV2traJCUlYWlZH2/vmzx+/AiA7dvdWL9+LTVq1OTlyzDu3r0DwNmzP6mvO3LkaE6ePMGlS3+o2wICnnHv3h20tdOWJn///Sd9+/anW7ceKBQK7t69jVKZTMuWrfjll3MkJCQQERHBn39eSnNus2YtOHLkEJDyItaIEYOJi8v4D5X0ngPAuHET2bPnAHv2HHhnEZtyfpL63sePHyUpKZGYmBjGjx+Lj493uvkNH25Lp05dCA4OIiQkGKUyOd1YSpQoScWKlbhw4RcAvL1vEhoaioVFjQxjGjPGlpiYGIYMGc6QIcO5fdsvw2MLG+mRLWC8vW+yaNEyGjRolN+hCCFEkTewo0WqMbIAerpaDOyYMy/LpTdGtnHjpjRq1JinT5/g5LQMlUrF6NEjOHfuLJDSEzl+/Biio6OxtR1NtWrVqVy5Mo8fP2bkyCEkJSXRrVsPPvnk01TjJt9Us2Ytxo4dx+TJ41EqldSqVQdb29HvjLVNm7Y4Oy9lwYL/hjlYW/dj4cK5nD9/Dl1dXSwtGxAQ8AyAdu06MHLkEHbu3MvcuQuZO3cmSqUSMzNzFi1ago6OLk5Oy3B0nIe2tk6qoQ1VqlTB1XULmzZtUL+EpK9vgI3NAD77rDtBQUGpYhsyZBjOzsvZu3cPJUoUp0GDRgQEBGBt3Y9bt3wYNmwgpqamVK1aPU1eU6fOZMWKxQwfPghIGaNcokSJDJ9DtWrViIqKxNFxHo6OS975zNJjaVmf777bgqvr/xg/fiJPnjzG1nYYycnJ9OzZh2bNmqf5vn3++RgWLZpPyZIlMTExpW7degQEBFC7dm11LL1791Uf7+i4hJUrl7Ft2xZ0dXVZsWI1urq6GcY0ceIXLF68EG1tbfT1DZg5c06W8yqoFKq3+6eLgNDQKJTKwpdWQkIC3bp9QuvWH/Ps2VMqVarEl19OpVy5/8a+mZmVIiQkZ8fwFDSSY9HwIeQIRSNPLS0FpqYl8zuMHOHjc4sKFT7K0jn5MWtBRtzcUj5it7efkC/3F6IgCgh4hKVlvXT3SY9sAfLiRQhNmzZnwoQvqFz5I/bv38Ps2d+wffteWalGCCFySdv65fOtcBVCvB8pZAuQChUqsnr1/9TbQ4eOZOfO73j+PIAKFSrmY2RCCCHygvTECpE1Usjmg8s+genO+Xjv3l3u3btDt2491ceqVCp0dOTbJIQQQgjxNpm1II9d9glk12k/9ZKSoRHx7Drtx2WfQLS0FKxdu1o9kP7o0cPUqFFDPT+gEEIIIYT4j3T15TEPT/9US0kCJCQp8fD0Z9Wkj3FwmM7MmQ7qNz8XLlyWT5EKIYQQQhRsUsjmsdc9sRm1d+3ag65de+RlSEIIIYQQhVKuDi04efIkPXr0oEuXLuzdm3Yi5nPnzmFtbU2fPn2YNGkS4eEpS6kdO3YMKysrrK2tsba2xsXFJTfDzFOmhvpZahdCCCGEEOnLVCEbGBiIp6cnycnJPH/+PFMXDgoKwsXFhX379nH8+HHc3d25d++een9UVBSOjo5s3bqVEydOULt2bdavXw+Al5cXs2bN4vjx4xw/fhwHB4dspFYw2XSwQE8n9WPX09HCpkPOTL4thBCiYAgICMDKqiUjRw5J9e/wYff8Di1dbm6b+Pff6wAsXeqEr++tdx4/adK4LF3/2rWrTJxor96+fdsPB4cvGTDAmgEDrJkyZRIPHtzPeuBvcHJayKlTJwgJCcHB4ctsXSMg4BlLly7K8nmPHj1k2rSvGT58EMOHD2LBgjm8evUyWzGkjieAvn17prvPweFLQkJC3vsebm6b1XMY55bWrZu+c/+pUydwclqY5etqHFrw66+/4ujoiJaWFgcOHKBHjx6sWrWKzp07v/O8S5cu0bp1a4yNjQHo2rUrZ86cYfLkyQAkJibi6OhI2bIpLzLVrl2bkydPAimF7KNHj9i6dSu1atVi/vz56jWSC7s2limLG6Q3a4HIeb/99itLlizk7FnP/A5FCFFAxd+5RNyfh1BGhaJV0hSD1gPRr9U2R65dpowZe/YcyJFr5bbr16/TtGkLAObOXZCJ49NfVSwzHj9+zJQpk1iwwIm2ba2AlP9eOzh8ycGDR9HT08v2tQHMzMxwcVmfrXOfP3/O06dPs3ROSEgIX3wxjpkz59KuXQdUKhW7dm1n5sypbNmyPVtxZEZ2cyxKNBayGzZs4ODBg4wbNw5zc3P27dvHzJkzNRaywcHBmJmZqbfNzc25efOmert06dLqa8TFxbF161ZGjhwJpPwAjhs3joYNG7JmzRqcnJz49ttvs5Xg+7p06Q+2bNlAQkICFhY1mT17PiVKvN8KOG0sy0nhmgeePHmMq+taVCql5oOFEB+k+DuXiPl1OyQlAKCMCk3ZhhwrZtPj5+eLg8OX7Nt3CC0tLT7/fCjOzi54el7gyZPHPH36lIiIV/Tt258RIz5HqVTi4rKaq1f/RqFQ0K1bT2xtR3Ht2lV27dqOgYEBDx8+wMKiBk5Oy9DV1eXHH09x4MA+VColderUZdq0Wejr69Oz52d07NiJmzf/RVtbm6VLV/Lvv//g53eL5cudWLHiW7791hk7u/E0atQYZ+fl3L9/j7CwMKpU+YgVK1bj6poy5/mYMbZs376by5cv4ua2maSkJMqXr8CcOfMxMjLmr78us3btt+jp6fHRR9XU+e/du4uePfuoi1iA9u0/ITT0BTEx0bx48QIHh8kYGRmjp6fHypWrWbrUieDgIF68CKFx46YsXJiylO66dWu4ePF3ypQxQ6lMpmnTZgQEBDBpkj3Hjv1AaGgoK1cuJSgoEC0tLSZO/JKWLVvh5raZkJAQnjx5TGDgc/r06cvo0XasWeNMQMAzVq1azvTpszP1/fTwOETLlm1o164DAAqFgpEjR1GhQkWSkpJISkpk2bIl3Lt3B4VCwfDhtvTo0YtTp05w6dIfhIQEExwczODBwwgKes7Vq1cwMjJWF6oJCQnMnTuTR48eUrFiJebOXYihoSF9+/Zk40Y3rl+/yp9/XiIiIoKAgKe0bNmGGTNSYt+9ewfnzv2MUplMq1ZtmDx5CgqFgu+/38WxYx4YGxtTqpQh9epZpsmrR48uWFm158aNfzA1LUP//gM5eHA/wcHBzJ+/iKZNm/H48SOWL19CREQ4xYoV45tvZlCvniUBAQE4Os4jNjaG+vUbqK8ZExPD6tUr8Pe/h1KpZOTIUXz2Wbes/PqkorGQVSqVmJubq7fr1q2bqVWm0lv5Nr3zIiMjmTRpEnXq1KFfv34AuLq6qvfb2dlpLJrfllNLLYaFhbFihRP79++natWqrFq1ip07t+Do6Jgj188uM7NS+Xr/vPC+OcbGxrJ8uSNz585h2rRpBfKZFcSYctqHkCN8OHkWRXF/HlIXsWpJCcT9eShHCtkXL0IYOXJIqraFCxdTp05dbGz6s379WpKSkrCxGUitWrXx9LyAv/893Nx2olQmM2rUcJo3b4m3txfBwUF8/707iYkJTJo0DgsLCwwMiuHldQN3dw/KlDHDzu5z/vzzEhUrVuL4cQ/c3Hagr6/Pxo3r2bt3D2PG2BEa+oIWLVoybdpM1q1bw6FD7kyZ8g0nTx7Hzm48NWrUVMfq5XUDXV0dtm3bhVKp5IsvxnPp0kWmTp3BoUMH2L59Ny9fvmTjxvW4um7F0NCQo0cPs2HD/5g+fRZOTgvZsGEz1apVZ+lSJ/V1vb29mDhxcprn1a/fAABiYmJ59OghHh6nqFChAmfPnqFmzVosW+ZMYmIiQ4f25/ZtXwICArhz5zb79x8iMjKKESMGp7mmi8sqevWypn37Drx4EcL48WPZvXs/APfu3WXLlu+IjIxkwIA+DBgwmG++mcG2bVsyXcQC3LnjR9u27VK1aWtrqwu0TZs2YGRkxL59h3j16iVjxthSs2YtAG7d8mbv3oNERETQr18v1q7dgIPDdCZNGsdff12mRo1avHwZxsCBQ2jcuAnr169l+/atfP31tFT38/K6yb59h9DW1mLQIBtsbAYQEhKMn58vO3bsQaFQ4Og4nzNnfqRq1WqcPHmc3bv3o1AosLP7PN1CNiwsFCurdsyZM59Jk8bh6XmBLVu288MPJ3F330fTps1wdJzHyJGj6NixE97eN5kzZzoHDx7j229X0LNnb6yt+3H69CmOHj0CwI4d26hduy4LFjgRHR2Fvf1oLC3rZ/pZv01jIVusWDECAgLURejVq1fR19f8YlLZsmW5evW/jx2Cg4NTFcSv28aOHUvr1q2ZM2cOkFLYHjlyhFGjRgHZWxAgNDQKpTJtIZ1VZ8+eo3btupQoYUpISCRdu/Zh1KihTJr0Tb4tGVsU1nXXJCdyXLx4Pj16WGNmVgmVSlXgnpl8H4uOopCnlpYixzoAChtlVGiW2rPqXUMLRo+2Y9SoEejr6+PouFjd/tln3ShevDgAVlYduHbtCt7eXvTs2RttbW20tYvRtWt3rlz5m3btOlC9uoV6vvGqVasRERFBYOAVnjx5gp3d50DKcL7ateuo79G6dUqRbmFhwT//XM8w/iZNmmFkZMzhw+48fPiQp08fExsbk+oYHx8vgoIC+eKLlDGzSqUSQ0ND/P3vUaZMGapVqw5Az5692LJlk/q8N/8/OnnyBMLDXxEVFcWkSV9iadmA0qVNqFChgvqZ+Ph4c+DAXh4+fEB4eDgxMbFcv36VTz75FB0dXUqXLk3bth+nyeHKlb959Oghbm4p905KSuLZs5ShA82aNUdXVxcTExMMDQ2Jisre77JCoZVuB95r165dYe7clPGfxsalad++A9evX6NEiRI0bNiYEiVKqj/tbd68JQDlypUnIiICgCpVPqJx4yYAdOvWg8WL044lbdCgISVKlACgYsWKREREcOXK3/j4eDNq1HAA4uPjKVeuHGFhobRta6X+OevUqQvJycnpxt6mzcfqeBo1apwqtpiYGJ4+fULHjp0AqF+/IYaGRjx69JDr16/h5LQcSJmR6fUfMleu/EVcXBynTh0HUjqe7t/3f/cDfgeNFeLUqVMZM2YMISEhDB48mIcPH6pfynqXtm3bsn79esLCwihWrBhnz55l8eL/flGTk5OZMGEC3bt3Z9KkSer24sWLs23bNpo0aUKjRo34/vvv6dKlSzbTez9BQUGpFiMwMzMnOjqamJjo9x5eIHKPh8chtLV16NXLmufPA/I7HCFEAaZV0jTdolWrpGmu3zsyMoqYmBhiYmKIiAjH2Lg0kNKT95pKpURbWxulMvUQKZVKpS489PTe7FxSoFKpUCqVdOrUhalTZwApH+e+Waj81yGl4B31F7/95omb2yYGDx5Kr159CA9/laZgUyqVNGzYmNWr1wIpxVJMTAxBQc9THftmXvXqWXLz5r98/HFKL+aGDSkvGjk5LSQ+Pv6tGOHgwQNcuHAOa2sbBg5shb+///9fW5Fq+Nib9/gvvmQ2bNiiftcmJCQEExMTPD0vvDUW993PYuvWTfz+e8r7Fvb2E2nfvoN6X9269fDzS/1ynFKpZPbs6cyYMSdN59qb3z8dHd1U+9LrvHu7Lb1j0v85SGbw4GEMGzYCSOks1NbW5ujRI2meW0aFrK6ubqrjUuehTPPz8Do3heK/741CoUBLK+VFd6VSiaPjEurUqQtAaGgoRkaGnDlzOt37a6Jx1oKmTZty8OBB1qxZw4QJEzhz5gwtW7bUeOGyZcvi4OCAra0tffv2pVevXjRs2BB7e3u8vLw4f/48t27d4qefflJPszV37ly0tbVZu3Ytjo6OdO/eHR8fH6ZPn56t5LLi78DrzLu4jC/Oz2DexWX8HXg9w7GVWlppf1FE/rjsE8j0jRcZs+I80zde5LJPIKdPn8TX14dRo4YxffoU4uPjGTVqGC9evP+bnUKIosWg9UDQeevFIh29lPZctnr1CgYMGEz//gNZtWqFut3T8wKJiYlERETwxx+/06pVG5o3b8EPP5wkOTmZuLhYfvrpNM2aNc/w2k2bNsfT8wJhYWGoVCqcnZdx4EDaaTDflF4xc+XKX3Tq1IVevawxNS3Dv/9eVxfV2traJCUlYWlZH2/vmzx+/AiA7dvdWL9+LTVq1OTlyzDu3r0DwNmzP6mvO3LkaE6eTBkf+lpAwDPu3buDtnba0uTvv/+kb9/+dOvWA4VCwd27t1Eqk2nZshW//HKOhIQEIiIi+PPPS2nObdasBUeOHALgwYP7jBgxmLi4uCw9B4Bx4yayZ88B9uw5kKqIBejb14aLF/9Q56NSqdi+3Y2XL19iampK8+YtOHHiGACvXr3E0/NXmjZtlmEMb3v48AG3b/sBcPLkcVq00FyHQUruZ878QExMDElJScyY8Q3nz5+jRYuW/PHH70RFRRIfH4+n54VMx/KmEiVKUrFiJS5c+AUAb++bhIaGYmFRgxYtWnHmzI8AXLhwnoSEBHVMHh6HgZShNyNGDCYwMDBb94dM9Mj6+fmxZs0atm7dyu3btxk1ahQuLi5Ur15d48V79+5N7969U7W5ubkB0KBBA/z8/NI9r3nz5hw9ejQz8eeIfVcvcPHVWdBK+cF9Gf+KfX5HqGVQkdDQF+rjXrwIoVQpQ4oVK5ZnsYmMvV7u9/VKaa+X+x3ztbP6ZbrnzwOwtR3Mzp378jNUIUQB9XocbG7NWpDeGNnGjZvSqFFjnj59gpPTMlQqFaNHj+DcubMpMenrM378GKKjo7G1HU21atWpXLkyjx8/ZuTIISQlJdGtWw8++eRTrl1Lf+aAmjVrMXbsOCZPHo9SqaRWrTrY2o5+Z6xt2rTF2XkpCxb89+mptXU/Fi6cy/nz59DV1cXSsoF6GfV27TowcuQQdu7cy9y5C5k7d6Z6VcpFi5ago6OLk9MyHB3noa2tk2poQ5UqVXB13cKmTRvYsGEdCoUCfX0DbGwG8Nln3QkKCkoV25Ahw3B2Xs7evXsoUaI4DRo0IiAgAGvrfty65cOwYQMxNTWlatW0tcnUqTNZsWIxw4cPAlLGKL/+CD491apVIyoqEkfHeTg6LnnnM3vN1LQMLi7r2bBhLa6u/yM5OZnatevg7JzyovrYsfY4Oy9n+PBBJCcnM2rUWOrUqcu9e3czdf1KlSqzfbsbT58+wcKiBhMmzMvUee3adeDu3buMHWuLUqmkdes29OzZG4VCwZAhwxg9eiSlShlSrlz5TF0vPY6OS1i5chnbtm1BV1eXFStWo6ury9SpM1m0aB5Hjx6hXj1LihdPeeZ2duNwdl7OsGEDSU5OZvLkKVSqVJl///0nW/dXqN41qAMYMmQIY8eOVX+8f+7cOXbt2sWePXuydcO8kJUxspd9AtnzeCMK/bR/nZVINMB3w59s3LiNypWrsHnzBsLCQpkzJ+vznOWUojAeT5PM5jh948V0V0ozNdRn1aSUMT2vC9mff/49x+N8H/J9LDqKQp5FaYysj88tKlT4KL/DyLbXc3na20/I50iEKDgCAh5haVkv3X0ahxbExsamGqPauXNnoqKici66fObh6Q966X/EEK0bx5w5C5g3bybDhw/g/v17TJ78dd4GKDKkablfgPLlKxS4IlYIIYQQOUPj0AKFQsHt27epXbs2AP7+/uoBu0VBaEQ8+gkG6fbIltY3pk0bK9q0sUrnTJHfTA31M+yRFUKIwkh6YoXIGo2F7JQpUxgxYgS1aqXMd3b//n1Wr16d64HlFVNDfV49qYVuNW8U2m+83KXUpo9F9ifoFbnPpoNFqjGyIMv9CiGEEB8SjYVsx44dOXPmDNevX0dbW5tGjRphapr705LklZRiKJHEB6BT+Q4KvThIMMDKrCMty717XWCRv2S5XyGEEOLDlqmVBvz9UyaqTUpK4tq1awB89tlnuRdVHvqvGNIl9EYFKYYKGVnuVwghhPhwaSxkZ8yYwZ9//slHH/33FqhCoSgyhSxIMSSEEEIIURhpfGvr6tWrnD59mj179qj/7d69Oy9iE0IIIQq1gIAA+vbtmaa9dev3H7r2en5aHx9vNmxYB6SsxLV166Z3nZah33/3ZN++77N0zsSJ9hnOZStEXtDYI1uhQoV3ThwshBBCiLy3Z88BIGW1qrCwMADat++QZtWpzPLz882x2ITIKxoL2aZNm+Lg4EDHjh0xMDBQtxeloQVCCCE+XH8FXOPYvdOExb3CxMCYvjW606pC5pcPza5Tp07w55+XiIiIICDgKS1btmHGjNlcu3aVnTu/Q6VS8ezZUz79tBMlSpTit98uoFKpWLNmPaamprRu3ZSff/bEzW0zsbEx7NixDTMzc65fv8aCBYu4du0qa9Y4o62tTf36DXnw4D6bNrlx/fo1Nm92JS4ujsjICCZPnkL16hYcPXoEgPLly/Ppp11YvXoF/v73UCqVjBw5is8+60ZCQgLLljnh63uL8uUrEB7+KtefkxDvorGQ/eeflCXDDh06pG4ramNkhRBCfJj+CrjG97cOk6BMBCAs7hXf30pZBz4vilkvr5vs23cIbW0tBg2ywcZmAJAyXGD//kMYGRnRvXtnvvrKgZ0797J48ULOnfuJwYOHAVCqVCns7Sdw/fo1Ro+249SpEwAkJSWyaNF8vv12HTVr1mLNmlXqex46dIA5c+ZTtWo1rl79GxeX1ezde5B+/foD0KuXNa6u/6N27bosWOBEdHQU9vajsbSsz6+/ngfA3d3j/5fMHZzrz0iId9FYyBbkpWiFEEKI93Hs3ml1EftagjKRY/dO50ghq6WlSNOmUqlQKFLaGzRoqB6+V7FiRSIiIgCwsLCgbNmUl5CNjIxp3rwlAOXKlVcf8y737t2jdOnS1KyZMgd8797W6mLW0XEJFy/+zvnz5/D29iImJibN+Veu/EVcXBynTh0HUlb5vH/fn+vXr9K3b0rBW6VKFRo0aJj5hyFELtD4sldISAjjxo2ja9euhIaGMnbsWEJCQvIitiLB3/8ekyePY/ToYYwdO1LGIAkhRAESFvcqS+1ZVaqUIdHRqZd1DwsLo1QpQwD09N5ciVCBSqUCQEdHN9U52tqZmi3zjeO11dd624QJY/Hx8aZOnbqMGjU23eOUSiWOjkvYs+cAe/YcYNu2XbRp0zZVjK/vI0R+0ljILlq0iM6dO6Ovr4+hoSF16tRh7ty5eRFboRcXF8c333zB8OG27Nixj1GjxuLkNC+/wxJCCPH/TAyMs9SeVSVKlKBy5SqcP/+Luu34cQ9atGiZI9eHlGIyOTkpVVvVqtWIiIjg3r27APz002kUCgXh4eE8fvyYceMm0ratFX/9dRmlUvn/19EhOTkZgGbNWuDhkTLE4sWLEEaMGExgYCAtW7bip59Oo1Qqef48AC+vmzmWhxDZobGQffbsGYMGDUJLSwtdXV2mT5/O8+fP8yK2Qu/vv/+kQoVKtGljBYCVVQecnFbkc1RCCCFe61ujO3paqXs/9bR06Vuje47dw9FxCR4eBxkxYjCDB9vg73+P6dNn5dj1LS3r4+Pjhavr/9Rturq6ODouwclpAZ9/Pozg4CD09fUxMjKiT5++DBs2EFvbobx8+ZK4uDhiY2Np0qQJP/10moMHD2BnN474+DiGDRvIF1+MZ/LkKVSqVJn+/QdSokRJhgzpz/LlS6heXZYEF/lLocros4f/Z2Njw+HDh7GxseHYsWNERUUxePBgfvjhh7yKMctCQ6NQKt+ZVp7Yu3cXvr4+lChRknv37lKyZCkmTfqK2rXrZPuaZmalCAmJzMEoCx7JsWj4EHKEopGnlpYCU9OS+R1GjvDxuUWFCh9pPvAN+TVrQW5SKpW4uv4PO7vxFCtWjH37vickJJgpU77J79CEyLKAgEdYWtZLd5/GQTefffYZ06ZNIzIykgMHDnDo0CG6d8+5v1SLsqSkJC5fvsj//rcFS8v6/P77r0yfPoXDh0+ip6eX3+EJIYQgZXaCwl64vk1LSwsjIyNGjx6Brq4u5ctXYO7cBfkdlhA5TmMhO2HCBI4dO4ZSqeTSpUsMHjyYgQMH5kVshcpln0A8PP0JjYjH1FAfmw4WlCljxkcfVcXSsj4A7dp9wsqVSwgIeEbVqtXyOWIhhBBFma3taGxtR+d3GELkKo2F7IwZM3B2dqZv3755EE7hdNknkF2n/UhIShkwHxoRz67TfvRrW5Pnz5/j5+dLnTp1+fff64CC8uUr5G/AQgghhBBFgMZC1s/PL9WcdyItD09/dRH7WkKSknP/hrF8+Wq+/XYFcXGx6OrqsXTpKvT19TO4khBCCCGEyCyNhayZmRk9e/akUaNG6kmbAebNk2mkXguNiM+wvXHjj3Fz25XHEQkhhBBCFH0aC9kmTZrQpEmTvIil0DI11E+3mDU1lJ5XIYQQQojcorGQnTx5MnFxcTx69IiaNWuSkJCAgYFBXsRWaNh0sEg1RhZAT0cLmw4yv54QQgghRG7RWMjeuHGDL774Ah0dHQ4cOIC1tTWbNm2iadOmeRFfgXb69Cnc3fcBEB2XSNjLcOJjXtGs7yKGdGtCG8ty+RyhEEKI/LJq1XJu3rxBYmIiT58+oVq16gBERkbQs2cf7O0n5Ni9AgICmDTJnmPHMj/He9++Pdm40Y0KFVK/gDxxoj12duNp1qw5MTExuLqu46+/LmNgUIwSJUpgZzeeFi1avfPav//uyZMnTxg2bIR6hTAbmwGZimvpUidsbAZQt27684YK8SaNhezKlSvZuXMn06ZNo1y5cjg7O7N06VKOHDmSF/EVaN2796J7915AypyxX3xhT/fu4+nbV+bZFUKID9306bOB/4rMPXsOAODmtjk/w8o0lUrFtGlfU6tWLfbvP4Kuri63b/sxdepXLFq0jGbNmmd4rp+fr/rrzBawr8l8tyIrNBaycXFx1KhRQ73doUMHXFxccjWowuj773dSunRp+vbtn9+hCCGEyIJXly4RfOQQSaGh6JiaYt5/IMZt2+bqPW/d8sHefhQhIcHq3tlTp07w44+nCA9/hZVVOwYNGsbKlUsJCgpES0uLiRO/pGXLVly58hcbNqxDoVBQqpQhixcvAyA+Pp5582Zx/74/pUqVwtl5DUZGxvzxx29s2bIRpVJJxYqVmDlzLqampupYEhISWLbMCV/fW5QvX4Hw8FcAXL9+jcDA57i6blHPXFS7dh1GjbJjxw43mjVrzsSJ9lStWg0fH28SEuJxcJiGuXlZjh5N6ewqX768ell7e/sJ9OjRBSur9ty48Q+mpmXo338gBw/uJzg4mPnzF9G0aTN1j/C9e3c5deq4Ordnz55y4sRpYmJicHZeTnh4OAYGBkydOoPatevg5LSQ8PBXPH36hMmTp9CuXYdc/R6KgkFL0wE6OjqEh4erf4jv37+f60EVNq9eveLAgb189dXU/A5FCCFEFry6dInnO7eTFBoKQFJoKM93bufVpUu5et+wsFBcXbeyc+de9u7dTXR0NADBwUHs2rWPiRO/xMVlFb16WbNr1z5WrXJh5cqlREdHs2PHd8ycOZedO/diZdWe27f9UnJ59ZKhQ0ewb98hTExM+fnnnwgLC2PlyqWsXLmGvXsP0rBhI779dmWqWA4dSukpdnf34JtvZvDs2VMAfH19qFu3XprpN5s0acqtW7fU24mJCezevQ8np2U4OS2kUqXK9OvXn379+tOrl3WavK2s2uHu7gGAp+cFtmzZjp3dePVQvdcGDx7Knj0H2L17P9WqVWfixMmYmpbByWkhkydPYffufcyePY/582erzzEyMsbd3UOK2A9Iplb2GjFiBC9evOCbb77h4sWLODk55UVshcaJEym/NBUqVMzvUIQQQmRB8JFDqBISUrWpEhIIPnIoV3tl27T5GD09PfT09DA2NiYiIhxI6fHU0Un5X/OVK3/z6NFD3Nw2ASlD2J49e0q7du2ZOXMqHTp8Qrt2n9CqVWsCAgIoU8ZMvZJk9erVefXqFbdueVOvXn31ONi+fW3YvXtHqliuX7+q/jSxSpUqNGjQEACFQkFSUlKa2BMTE1NtW1v3A6BWrdqYmpbh3r27GnMHKFeuPI0aNVZ/HRERke7xW7duQldXlxEjPicmJgZfXx+WLHFU74+NjVH3Ir/OX3w4Mixkb9y4QaNGjfj000+xsLDg4sWLKJVKJk2alGqowYcgveVn33yR65dffubrr6flY4RCCCGy43VPbGbbc4q2tvYbWwpUqpSv9PX/mxVIqUxmw4YtGBkZARASEoKJiQm1atXGyqo9Fy/+jqvrOm7d8qZr1x7pXFOFUqlKdV+VCpKSkt+KJuXYt2OztKzPwYMHSEpKREdHV73f2/sm9erVe+P4/0oJlUr5Vhxp6er+dy1Nx/7yy8/88cdvuLmlFN9KZTJ6evrq8caQ0ottaJjyjGTBoQ9PhkMLFi5cCMDnn3/ORx99xLBhwxgxYsQHWcTuOu2nnif29fKzl30CAYiIiODZsyc0aNAoP8MUQgiRDTpvjBXNTHteatasBUeOHALgwYP7jBgxmLi4OMaMsSUmJoYhQ4YzZMhw9dCC9Fha1sfb24uAgAAAjh07kuYlrZYtW/HTT6dRKpU8fx6Al9dNABo3bkq1atVxcVlNUlJKL6yf3y127PiO0aPt1ef//PNPAPj63iIyMgILixpoa+uQnPx2wZw1d+7cZv16F1au/BYDg2IAlCxZisqVK3P6dMrsDH/99ScTJti9131E4ZZhj2xycjJjxozBx8eHCRPSThGyeXPheOvyfWW0/KyHpz9tLMvx7NkTTE3LqD8KEkIIUXiY9x/I853bUw0vUOjpYd5/YD5GlWLq1JmsWLGY4cMHAbBw4WJKlCjBxIlfsHjxQrS1tdHXN2DmzDkZXsPU1JRZs+Yya9ZUEhMTKVeufJpZAfr3H4i/vz9DhvSnXLnyVK/+3xzoK1asZvNmV4YNG4SWlhaGhkY4Oi5OVQwHBDzD1nYYAEuWrEBbW5smTZrg5LQQE5Ps/0GwYcM6kpOTmT17Okql8v+fyQwWLVrKypXL+P77Xejq6rJkyYo043jFh0OhevPzhDdcvXqVp0+fsm7dOr766qs0+/v165frwWVXaGhUmo9TsmvMivMZ7ts+69McuUdWmJmVIiQkMs/vm5ckx6LhQ8gRikaeWloKTE1L5ncYOcLH5xYVKnyUpXPyY9aCouLNOWeFyC0BAY+wtEx/XuEMuxGXLVuGh4cHa9asKdBFa26T5WeFEKJoM27bVgpXIQqpDAvZyMhI5s+fT1JSEkuWLEmzf968ebkaWEEhy88KIYQQ6du0yS2/QxAfuAwL2fXr1/PLL7+gpaWFsbFxHoZUsLyeneBdsxYIIYQQQoi8l2EhW6dOHerUqUOVKlXo3bt3ti5+8uRJNm3aRGJiIqNGjWL48OGp9p87d47169ejUqmoVKkSy5cvx8jIiICAAKZPn05oaCjVqlVj9erVlChRIlsx5IQ2luWkcBVCCCGEKGAynH5r6dKlAPzwww9MmDAhzT9NgoKCcHFxYd++fRw/fhx3d3fu3bun3h8VFYWjoyNbt27lxIkT1K5dm/Xr1wOwaNEihg0bxpkzZ6hfvz4bN2583zyFEEIIIUQRk2GPbJs2bQDo2rVrti586dIlWrdurR6W0LVrV86cOcPkyZOBlJVBHB0dKVu2LAC1a9fm5MmTJCYmcuXKFVxdXQGwsbFhxIgRTJ8+PVtxCCGEEEKIoinDQvbTT1Omlvrss8948OABxYoVo3Llyujp6WXqwsHBwZiZmam3zc3NuXnzpnq7dOnSdO7cGYC4uDi2bt3KyJEjefnyJSVLllTPy2pmZkZQUFDWMxNCCCHy0apVy7l58waJiYk8ffqEatWqAzB48FCeP38OgL295k84MysgIIBJk+w5duyHTJ/Tt29PNm50Uy9h+9qb02rFxMTg6rqOv/66jIFBMUqUKIGd3XhatGj1zmv//rsnT548YdiwEXh4HAbAxmZApuJautQJG5sB1K2b/pRL2eXuvp/y5StQo0ZNBg3qq/6exMfHU6NGTaZOnYmpqSkBAQGp9r9mbd0PAwMD3N33AykLVVSqVBldXV0aNmzE55+PYfNmVxYscMrRuEXGMixklUoly5Ytw93dnVKlSgEQGxvLiBEjmDp1qsYLpzc9bXoTFkdGRjJp0iTq1KlDv3790i1aszrRcVGZDzEjZmal8juEXCc5Fg0fQo7w4eQpsmb69NnAfwXmm8uqurkVjkWFVCoV06Z9Ta1atdi//wi6urrcvu3H1KlfsWjRsnfOH+vn56v+OrMF7GtvL9qQE0JDQ/n9d082bNhMQEAAZcqYqb8nKpWKTZs2MGfOdLZs2Q6Qav/bevWyBlL+EFizZn2qPwRKlzbh0qU/aNvWKsdzEGllWMiuXbuW58+fc+7cOfXH/8+ePWPJkiVs3rxZ4zjZsmXLcvXqVfV2cHAw5ubmqY4JDg5m7NixtG7dmjlzUlYmMTExISoqiuTkZLS1tQkJCUlzniY5uSBCQVMUJl/XRHIsGj6EHKFo5FmUFkTIjtvegVy+cJ/I8HhKGenTpmN1atfP/Rd8b93ywd5+FCEhwfTs2Qd7+wmcOnWCH388RXj4K6ys2jFo0DBWrlxKUFAgWlpaTJz4JS1btuLKlb/YsGEdCoWCUqUMWbx4GZDSszhv3izu3/enVKlSODuvwcjImD/++I0tWzaiVCqpWLESM2fOxfSNZXgTEhJYtswJX99blC9fgfDwVwBcv36NwMDnuLpuUXcq1a5dh1Gj7Nixw41mzZozcaI9VatWw8fHm4SEeBwcpmFuXpajR48AUL58+VQ90D16dMHKqj03bvyDqWkZ+vcfyMGD+wkODmb+/EU0bdpM3SN8795dTp06rs7t2bOnnDhxmpiYGJydlxMeHo6BgQFTp86gdu06ODktJDz8FU+fPmHy5Cm0a9dBneORIwf59NPO6X4vFArF/8fWmbt371CiRPZ/H3r06MXq1SukkM0jGb7sdeHCBb799lt1EQtQsWJFVq5cyU8//aTxwm3btuXy5cuEhYURGxvL2bNnad++vXp/cnIyEyZMoHv37sydO1f9C6Krq0vz5s358ccfATh27Fiq84QQQoiccts7kPM/3CYyPGXhm8jweM7/cJvb3oG5fu+wsFBcXbeyc+de9u7dTXR0NADBwUHs2rWPiRO/xMVlFb16WbNr1z5WrXJh5cqlREdHs2PHd8ycOZedO/diZdWe27f9AHj16iVDh45g375DmJiY8vPPPxEWFsbKlUtZuXINe/cepGHDRnz77cpUsRw6lNLz6O7uwTffzODZs6cA+Pr6ULduvTSfjDZp0pRbt26ptxMTE9i9ex9OTstwclpIpUqV6devP/369Vf3Xr6Zt5VVO9zdPQDw9LzAli3bsbMbj7v7vlTHDh48lD17DrB7936qVavOxImTMTUtg5PTQiZPnsLu3fuYPXse8+fPVp9jZGSMu7tHqiIWUoY6NGnSNMPvh66uLpUrV+HRo4cAvHgRwsiRQ1L9u3fvbobnv2ZhUYMHD+4TERGh8Vjx/jLskdXS0sLAwCBNu6GhYaY+6i9btiwODg7Y2tqSmJjIgAEDaNiwIfb29nz11VcEBgZy69YtkpOT1YVx/fr1Wbp0KQsXLmTWrFls2rSJ8uXLs2bNmvdIUQghhEjf5Qv3SUpUpmpLSlRy+cL9XO+VbdPmY/T09NDT08PY2JiIiHAgpcfz9XsiV678zaNHD3Fz25QSW1ISz549pV279sycOZUOHT6hXbtPaNWqtfrjckvL+gBUr16dV69eceuWN/Xq1Vd//N23rw27d+9IFcv161fp27c/AFWqVKFBg4ZASk9lUlJSmtgTExNTbVtbp6wAWqtWbUxNy2gs+Nq0+RiAcuXK06hRY/XXGRV/W7duQldXlxEjPicmJgZfXx+WLHFU74+NjVH3Ir/O/21PnjzOxCe8CvT1U1bufNfQAk3Mzcvy7NlTDA1zdoyvSOudhWxG0hv/mp7evXunmYPWzS1lFZAGDRrg5+eX7nkVK1Zkz549mbqHEEIIkV2ve2Iz256TtLW139hS8Pp/rfr6/3UiKZXJbNiwBSMjIwBCQkIwMTGhVq3aWFm15+LF33F1XcetW9507dojnWuq0gy1U6kgKSn5rWgUqf7f/vo6lpb1OXjwAElJiejo6Kr3e3vfpF69em8c/185oVIp34ojLV3d/66l6dhffvmZP/74DTe3lOJbqUxGT08/VZEZHByEoWHKM3pdiL5NS0vrnfdKTEzk8eOHaV7wyg4dHZ0sv98jsifDajUuLo5bt27h4+OT5l9cXFxexijykKfnBT7/fAijRg3jyy/Hqz9eEkKIoqiUUfpFT0btea1ZsxYcOXIISHlDfsSIwcTFxTFmjC0xMTEMGTKcIUOGq4cWpMfSsj7e3l4EBAQAcOzYkTQvabVs2YqffjqNUqnk+fMAvLxSZhlq3Lgp1apVx8VlNUlJKb2wfn632LHjO0aPtlef//PPKZ+s+vreIjIyAguLGmhr65Cc/HbBnDV37txm/XoXVq78FgODYgCULFmKypUrc/p0yuwMf/31JxMm2Gm8VsWKldRjdd+mVCpxc9tM/foNqFSp8nvFDClz6VeoUPG9ryM0y7BHNj4+Xj3n69vkr4yiKT4+jsWL57Nz534qVaqMu/te1q5dxapV6/I7NCGEyBVtOlbn/A+3Uw0v0NHVok3H9++VywlTp85kxYrFDB8+CICFCxdTokQJJk78gsWLF6KtrY2+vgEzZ87J8BqmpqbMmjWXWbOmkpiYSLly5dPMCtC//0D8/f0ZMqQ/5cqVp3p1C/W+FStWs3mzK8OGDUJLSwtDQyMcHRenKoYDAp5hazsMgCVLVqCtrU2TJk1wclqIiYkp2bVhwzqSk5OZPXs6SqXy/5/JDBYtWsrKlcv4/vtd6OrqsmTJCo21iZVVe65du6rucX09BhZSCtlatWqzaNEy9fFv7n+tceOmTJ0645338fe/x0cfVcXQ0DDL+YqsU6gyO06gEJFZC7InJiaG3r274Oq6jTp16rJz5zbu3r3N0qWrcuV+GSkKb4FrIjkWHUUhz6I0a4GPzy0qVPgoS+fk16wFRcWbc84WZKGhL5g7dyabN3+Xq/dZu3Y1LVq04uOP2+XqfT4kAQGPsLRMf7xxhj2y4sNTvHhxpk2bzcSJYzA0NEKpVLJpU+7+wgshRH6rXb+cFK4fAFPTMnTo0BFPzwt06NAxV+4RFBRIWFiYFLF5SApZoebvf4+dO7fx/feHqFixEocOHWDu3Bns3LlPhpMIIYRI16ZNbvkdQqYNHToiV69ftmw5nJyWaT5Q5BgpZD9Ql30C8fD0JzQiHlNDfWw6WPDgxmUaNGhExYqVALCxGcj69WsIDw/H2Ng4fwMWQgghhHhLxnNsvSEwMBBPT0+Sk5MzfONPFB6XfQLZddqP0IiU6WVCI+LZddqPZH1z/v33OmFhoQD8/vuvlC9fQYpYIUShUgRf/RDig6Xp91ljj+yvv/6Ko6MjWlpaHDhwgB49erBq1So6d05/mTdR8Hl4+pOQlHoC8IQkJTeDSjJ06Ei+/HI8Ojq6GBoasnz5t/kUpRBCZF2xYgZERoZTqpSRDIkSopBTqVRERoZTrFjaBbpe01jIbtiwgYMHDzJu3DjMzc3Zt28fM2fOlEK2EHvdE5tee/9Jg+jff1AeRySEEDmjcuXKPHnyhOfPH+d3KEKIHFCsmAGVK2c8t6/GQlapVKZa0q1u3bryV24hZ2qon24xa2pYMCYAF0KI7NLV1aV69YIxB6wQIvdpHCNbrFgxAgIC1MXr1atXM1z+TRQONh0s0NNJ/a3X09HCpoNFBmcIIYQQQhQ8Gntkp06dypgxYwgJCWHw4ME8fPiQ9evX50VsIpe0sUyZL/HtWQtetwshhBBCFAYaC9mmTZty8OBB/vnnH5RKJY0aNcLExCQvYhO5qI1lOSlchRBCCFGoaRxaYGdnh6GhIR06dKBjx46YmJgwaJC8DCSEEEIIIfJXhj2yX331FQ8ePODJkyf07t1b3Z6UlISWVqamnxVCCCGEECLXZFjIzpgxg2fPnjF//nzmz5+vbtfW1qZmzZp5EpwQQgghhBAZybCQrVSpEpUqVeLMmTNpemBjYmJyPTAhhBBCCCHeRePLXufPn+d///sfMTExqFQqlEolr1694p9//smL+IQQQgghhEiXxkLW2dmZr7/+mv3792Nvb8+5c+coUaJEXsQmhBBCCCFEhjK1IEKPHj1o3Lgx+vr6ODo68ueff+ZFbEIIIYQQQmRIYyGrp6dHQkICVapUwdfXFy0tLRISEvIiNiGEEEIIITKkcWhBp06dGDduHCtWrGDIkCFcu3YNY2PjPAhNCCGEEEKIjGksZCdMmECfPn0oV64crq6uXL16NdW8skIIIYQQQuSHdw4tePDgAcHBwVSoUAEAS0tLunXrxtKlS/MkOCGEEEIIITKSYSG7bds2bGxs6Nq1K1euXEGlUrF9+3a6d+9OSEhIXsYohBBCCCFEGhkOLXB3d+fHH3/k+fPnbN++nT179nD9+nUWLVokQwuEEEIIIUS+y7BHtlixYpQvX56mTZty9epVEhIS+PHHH6WIFUIIIYQQBUKGPbLa2trqr0uVKsXatWsxMDDIk6CEEEIIIYTQROM8sgAlS5aUIlYIIYQQQhQoGfbIhoaGsmPHjjRfvzZ69OjcjUwIIYQQQoh3yLCQ/fjjj7lz506ar4UQQgghhCgIMixkly9fnpdxCCGEEEIIkSWZGiMrhBBCCCFEQSOFrBBCCCGEKJSkkBVCCCGEEIVSpgrZmzdvcuDAARISEvjnn38yffGTJ0/So0cPunTpwt69ezM8bubMmXh4eKi3jx07hpWVFdbW1lhbW+Pi4pLpewohhBBCiA9Dhi97vebh4cF3331HfHw8Xbp0YdKkSTg4ODBo0KB3nhcUFISLiwseHh7o6ekxZMgQWrVqRY0aNVIds3DhQi5fvkyrVq3U7V5eXsyaNYtevXq9R2p5J+HuJRKuHEEVFYqipCl6LfqjV7NtfoclhBBCCFGkaeyR3bNnD+7u7pQsWRJTU1M8PDzYtWuXxgtfunSJ1q1bY2xsTPHixenatStnzpxJdczJkyfp1KkT3bt3T9Xu5eXFsWPH6NOnD9OmTSM8PDyLaeWdhLuXiP99J6qoUABUUaHE/76ThLuX8jkyIYQQQoiiTWMhq6WlRcmSJdXb5cuXT7V8bUaCg4MxMzNTb5ubmxMUFJTqGDs7OwYOHJjmXDMzM7788kuOHz9O+fLlcXJy0ni//JJw5QgkJaRuTEpIaRdCCCGEELlG49ACY2NjfH19USgUAJw4cQIjIyONF1apVGnaXl9DE1dXV/XXdnZ2dO7cOVPnvWZqWlLzQTkkMios3XZVVBhmZqVy5Z65dd2CRHIsGj6EHOHDyVMIIQoajYXsnDlzmDJlCo8fP8bKygp9fX02btyo8cJly5bl6tWr6u3g4GDMzc01nhcZGcmRI0cYNWoUkFIQ6+hoDDOV0NAolMq0hXRuUJQ0UQ8reLs9JCQyx+9nZlYqV65bkEiORcOHkCMUjTy1tBR52gEghBA5RePQgurVq3P8+HGOHj3K9u3bOXPmDLVr19Z44bZt23L58mXCwsKIjY3l7NmztG/fXuN5xYsXZ9u2bdy4cQOA77//ni5dumQilfyh16I/6OilbtTRS2kXQgghhBC5RmNXZ4cOHRgwYAD9+/enYsWKmb5w2bJlcXBwwNbWlsTERAYMGEDDhg2xt7fnq6++okGDBumep62tzdq1a3F0dCQuLo6qVavi7Oyc+Yzy2OvZCWTWAiGEEEKIvKVQpTeY9Q3379/Hw8ODEydOULNmTQYOHEjnzp2z/HF/XsrLoQV5rSh8jKmJ5Fg0fAg5QtHIU4YWCCEKq0wNLZg2bRq//vortra2bN++PVNDBIQQQgghhMhNmepWDQ0N5cSJExw9ehSVSsXEiRNzOy4hhBBCCCHeSWMhO2HCBK5fv07Xrl1ZvHgxjRo1you4ioQjR9w5evQICgVUrFiJmTPnUbq0SX6HJYQQQghRJGgcWvDpp59y4cIFKWKzyM/Pl/37v2fz5u3s2XOQSpWq4Oa2Kb/DEkIIIYQoMjLskT1+/DjW1tZERUVx8ODBNPtHjx6dq4EVdnXq1OXAgaPo6OgQHx9PSEgwFSpkftYHIYQQQgjxbhkWso8ePQLg7t27eRZMUaOjo8Nvv/3KypWL0dXVw85uQn6HJIQQQghRZGRYyH711VcAdOrUKc0SsceOHcvVoIqS9u0/oX37Tzhx4ijffPMl7u5H0dLSOKJDCCGEEEJokGEhe/78eZKSknB2dkalUvF6utmkpCRcXFzo27dvXsVY4F32CcTD05/QiHhMDfWx6WBBZaNEQkNDadSoMQA9e/Zh9erlREZGYGRknK/xCiGEEEIUBRkWsr6+vvz555+Ehoaye/fu/07Q0WHs2LF5ElxhcNknkF2n/UhIUgIQGhHPrtN+tKuRxOFda9ixYx/GxsacPXuaatUspIgVQgghhMghGlf22rt3L8OHD8+reHJEXq7sNX3jRUIj4tO0mxrq07b8czw8DqKtrUOZMmX45puZ7/3CV1FYRUgTybFo+BByhKKRp6zsJYQorDTOIztw4EB+/vlnoqOjAUhOTubx48c4ODjkenCFQXpF7Ov2fpMG0K/fgDyOSAghhBDiw6CxkHVwcODJkyeEhIRQr149bty4QcuWLfMitkLB1FA/wx5ZIYQQQgiRezS+Pu/r64uHhwedOnVizpw5HDhwgMjIwv0xWk6y6WCBnk7qx6ino4VNB4t8ikgIIYQQ4sOgsZA1NzdHR0eHqlWrcufOHWrUqEFsbGxexFYotLEsx+fd66h7YE0N9fm8ex3aWJbL58iEEEIIIYo2jUMLihcvzsmTJ6lTpw4HDx6kevXqvHr1Kg9CKzzaWJaTwlUIIYQQIo9p7JGdP38+vr6+fPzxx2hpaTFixAjGjBmTF7EJIYQQQgiRIY3TbxVGeTn9Vl4rClP9aCI5Fg0fQo5QNPKU6beEEIWVxqEFn376KQqFQr2tUCgoVqwYNWvWZNasWZibm+dqgB8KlUrFsmWLqFbNgmHDRuZ3OEIIIYQQBZ7GQrZz585ER0czfPhwtLS0OHz4MNHR0dSuXZsFCxawefPmvIizSHv48AFr1qzEx8eLsWNltgMhhBBCiMzQWMhevXoVDw8P9fa8efMYMGAAy5cv58iRI7ka3IfCw+MgPXr0pmxZeWFMCCGEECKzNBay0dHRREVFUbJkyvipqKgo4uLicj2wD8k338wE4Nq1K/kciRBCCCFE4aGxkO3fvz+DBg2iW7duqFQqzp49y8CBA9mzZw/Vq1fPixiFEEIIIYRIQ2MhO27cOOrVq4enpyc6OjrMnz+f1q1b4+3tTb9+/fIiRiGEEEIIIdLQWMgCGBoaYmFhgY2NDT4+PgDUr18/VwMrii77BOLh6U9oRDymhvrYdLCQhRSEEEIIIbJJ44IIHh4ezJ49m23bthEZGcmkSZM4ePBgXsRWpFz2CWTXaT9CI+IBCI2IZ9dpPy77BOZzZEIIIYQQhZPGQnbPnj24u7tTsmRJTE1N8fDwYNeuXXkRW5Hi4elPQpIyVVtCkhIPT3/19ty5jjKHrBBCCCFEJmksZLW0tNQzFgCUL18ebW3tXA2qKHrdE5vZdiGEEEII8W4aC1ljY2N8fX3Vq3udOHECIyOjXA+sqDE11M9SuxBCCCGEeDeNL3vNmTOHKVOm8PjxY6ysrNDX12fjxo15EVuRYtPBgl2n/VINL9DT0cKmg6zkJYQQQgiRHRoLWQsLC44fP87Dhw9JTk6mWrVq6Orq5kVsRcrr2Qlk1gIhhBBCiJyRYSF75Ur6q0z9+++/ALRo0SJXAirK2liWk8JVCCGEECKHZFjIOjk5pdv++PFjEhMTuXXrVq4FJYQQQgghhCYZFrInT55MtR0bG8vSpUt5+fIlK1euzPXAhBBCCCGEeBeNsxYA+Pr60r9/f168eMGJEyf4+OOPczsuIYQQQggh3kljIbt9+3ZGjhzJsGHD2Lx5MyYmJnkRlxBCCCGEEO+U4dCC0NBQZsyYQXBwMPv376dmzZp5GZcQQgghhBDvlGEh27t3b6Kjo+nTpw/u7u5p9s+bN0/jxU+ePMmmTZtITExk1KhRDB8+PN3jZs6cSatWrbCxsQEgICCA6dOnExoaSrVq1Vi9ejUlSpTIbE5CCCGEEOIDkOHQgqFDh2Jvb0/ZsmUxNjZO80+ToKAgXFxc2LdvH8ePH8fd3Z179+6lOWbChAmcOXMmVfuiRYsYNmwYZ86coX79+rIAgxBCCCGESCPDHtkvv/zyvS586dIlWrdurS56u3btypkzZ5g8ebL6mJMnT9KpU6dUhXFiYiJXrlzB1dUVABsbG0aMGMH06dPfKx4hhBBCCFG0aFzZK7uCg4MxMzNTb5ubm3Pz5s1Ux9jZ2QFw7do1ddvLly8pWbIkOjopoZmZmREUFJSle5ualsxu2IWCmVmp/A4h10mORcOHkCN8OHkKIURBk2uFrEqlStOmUChy7bw3hYZGoVSmvU5RYGZWipCQyPwOI1dJjkXDh5AjFI08tbQURb4DQAhRNGVqHtnsKFu2LC9evFBvBwcHY25urvE8ExMToqKiSE5OBiAkJCRT5wkhhBBCiA+LxkJWqVSybds2Zs6cSVRUFFu2bFEXme/Stm1bLl++TFhYGLGxsZw9e5b27dtrPE9XV5fmzZvz448/AnDs2LFMnSeEEEIIIT4sGgtZZ2dn7ty5ox7f+vvvv7N8+XKNFy5btiwODg7Y2trSt29fevXqRcOGDbG3t8fLy+ud5y5cuJCDBw/So0cPrl69ytdff525bIQQQgghxAdDoUpvUOobrK2tOXr0KDY2Nhw7doyEhASsra05ffp0XsWYZTJGtnCTHIuGDyFHKBp5yhhZIURhpbFHVkdHBy2t/w7T09NTzygghBBCCCFEftFYkdaqVYu9e/eSnJzM/fv32blzJ3Xq1MmL2IQQQgghhMiQxh7ZuXPn4uPjQ2hoKEOHDiU6Opo5c+bkRWxCCCGEEEJkSGOPbMmSJVm2bBmQMsdrUlISurq6uR6YEEIIIYQQ76KxR/bq1ats3LiRhIQEbGxsUk2NJYQQQgghRH7RWMiuWrWKxo0bc+7cOcqUKcMPP/zA9u3b8yI2IYQQQgghMqSxkE1OTqZt27ZcunSJzp07U6lSJZRKZV7EJoQQQgghRIYytbLXzZs3+fXXX/n444+5c+cOiYmJeRGbEEIIIYQQGdL4steECROYOnUqAwYMoFKlSnz66afMnTs3L2ITQgghhBAiQxoL2c8++4zPPvtMvf3zzz+jra2dq0EJIYQQQgihSYaF7IQJE9554ubNm3M8GCGEEEIIITIrw0K2a9eueRmHEEIIIYQQWZJhIduvX79021UqFY8ePcq1gIQQQgghhMgMjWNkDxw4gLOzM7Gxseo2ExMTLl68mKuBCSGEEEII8S4aC9mtW7eyY8cONm3axNdff82FCxcIDAzMi9iEEEIIIYTIkMZ5ZI2NjWnUqBF169YlNDSUiRMn4uXllRexCSGEEEIIkSGNhayOjg7h4eF89NFH3Lx5E4Do6OhcD0wIIYQQQoh30VjIDho0iPHjx/PJJ5/g7u6OjY0NFhYWeRGbEEIIIYQQGdI4RnbAgAH06NGD4sWL4+7ujpeXF1ZWVnkRmxBCCCGEEBnKsEd2/vz56q/j4uIAKFu2LJ07d8bAwCD3IxNCCCGEEOIdMixkvb291V+PHTs2T4IRQgghhBAiszIcWqBSqdL9+kMX8eclXngcISksFB0TU8rY9Mewddv8DksIIYQQ4oOjcYwsgEKhyO04CoWIPy8RtHsnqoQEAJLCQgnavRNAilkhhBBCiDyWYSGrVCoJDw9HpVKRnJys/vo1Y2PjvIivQHnhcURdxL6mSkjghccRKWSFEEIIIfJYhoXsnTt3aN26tbp4bdWqlXqfQqHA19c396MrYJLCQrPULoQQQgghck+Ghayfn19exlEo6JiYplu06piY5kM0QgghhBAfNo0LIoj/lLHpj0JPL1WbQk+PMjb98ykiIYQQQogPV6Ze9hIpXo+DlVkLhBBCCCHynxSyWWTYuq0UrkIIIYQQBYAMLRBCCCGEEIWSFLJCCCGEEKJQkkJWCCGEEEIUSlLICiGEEEKIQkkKWSGEEEIIUShJISuEEEIIIQqlXC1kT548SY8ePejSpQt79+5Ns9/X15f+/fvTtWtX5s6dS1JSEgDHjh3DysoKa2trrK2tcXFxyc0whRBCCCFEIZRr88gGBQXh4uKCh4cHenp6DBkyhFatWlGjRg31MdOnT2fJkiU0btyYOXPmcPDgQYYNG4aXlxezZs2iV69euRWeEEIIIYQo5HKtR/bSpUu0bt0aY2NjihcvTteuXTlz5ox6/7Nnz4iLi6Nx48YA2NjYqPd7eXlx7Ngx+vTpw7Rp0wgPD8+tMDPtjk8Qezb+yaYVnuzZ+Cd3fILyOyQhhBBCiA9arhWywcHBmJmZqbfNzc0JCgrKcL+ZmZl6v5mZGV9++SXHjx+nfPnyODk55VaYmXLHJwjP03eIiogHICoiHs/Td6SYFUIIIYTIR7k2tEClUqVpUygUmdrv6uqqbrOzs6Nz585ZurepacksHa/Jvj/+JilJmaotKUnJ1T8e8fEnNTI4K/eYmZXK83vmNcmxaPgQcoQPJ08hhChocq2QLVu2LFevXlVvBwcHY25unmr/ixcv1NshISGYm5sTGRnJkSNHGDVqFJBS8OroZC3M0NAolMq0hXJ2hb+MzbA9JCQyx+6TGWZmpfL8nnlNciwaPoQcoWjkqaWlyPEOACGEyAu5NrSgbdu2XL58mbCwMGJjYzl79izt27dX769YsSL6+vpcu3YNSJmpoH379hQvXpxt27Zx48YNAL7//nu6dOmSW2FmSklD/Sy1CyGEEEKI3JdrhWzZsmVxcHDA1taWvn370qtXLxo2bIi9vT1eXl4ArF69muXLl9O9e3diY2OxtbVFW1ubtWvX4ujoSPfu3fHx8WH69Om5FWamtOpQDR2d1I9KR0eLVh2q5VNEQgghhBBCoUpvsGohl9NDCyDlha+/PB8QFRFPSUN9WnWoRi3Lsjl6j8woCh9jaiI5Fg0fQo5QNPKUoQVCiMIq18bIFjW1LMvmS+EqhBBCCCHSJ0vUCiGEEEKIQkkKWSGEEEIIUShJISuEEEIIIQolKWSFEEIIIUShVCRf9tLSUmg+qBAr6vmB5FhUfAg5QuHPs7DHL4T4cBXJ6beEEEIIIUTRJ0MLhBBCCCFEoSSFrBBCCCGEKJSkkBVCCCGEEIWSFLJCCCGEEKJQkkJWCCGEEEIUSlLICiGEEEKIQkkKWSGEEEIIUShJISuEEEIIIQolKWSFEEIIIUShJIVsAXHy5El69OhBly5d2Lt3b4bH/frrr3z66afqbX9/f4YNG4a1tTWDBw/G19c3L8LNtuzm+VpgYCAtW7bk6dOnuRnme8lujlFRUUydOpW+ffvSt29ffHx88iLcbMlujuHh4djb29OnTx8GDBhQoH9eNeW4YcMGOnbsiLW1NdbW1upjfH196d+/P127dmXu3LkkJSXldehCCPHhUIl8FxgYqOrYsaPq5cuXqujoaFXv3r1Vd+/eTXNcSEiIqlu3bqqOHTuq24YMGaI6f/68SqVSqS5duqTq3bt3nsWdVe+Tp0qlUiUnJ6vGjBmjaty4serJkyd5FXaWvE+Oc+bMUa1atUqlUqlUnp6eqgEDBuRZ3FnxPjm6uLionJ2dVSqVSvXLL7+ohgwZkmdxZ0Vmchw/frzq+vXrac7t2bOn6p9//lGpVCrV7NmzVXv37s2LkIUQ4oMkPbIFwKVLl2jdujXGxsYUL16crl27cubMmTTHzZs3j8mTJ6dqGzhwIO3btwegdu3aPH/+PE9izo73yRNg27ZttG3bltKlS+dFuNmS3RxVKhVnz55l3LhxALRv355ly5blWdxZ8T7fR6VSSXR0NACxsbEYGBjkScxZlZkcvb29cXNzo3fv3jg5OREfH8+zZ8+Ii4ujcePGANjY2KT7bIQQQuQMKWQLgODgYMzMzNTb5ubmBAUFpTpm9+7d1KtXj0aNGqVqt7GxQVtbG4D//e9/dO7cOfcDzqb3ydPb25u//vqL0aNH50ms2ZXdHENDQ9HT0+P777+nb9++2NrakpycnGdxZ8X7fB/HjBnD5cuXsbKyYt68eXz11Vd5EnNWacoxOjqaunXrMnPmTI4ePUpERAQbN25Mc56ZmVmaZyOEECLnSCFbAKhUqjRtCoVC/fWdO3c4e/YskyZNyvD8lStXcuPGDebMmZNrcb6v7OYZGxuLk5MTixcvRkurYP/IZjfH5ORkXrx4gZGREceOHWP8+PF88cUXuR5vdrzPz+vixYsZPnw4f/zxB9u3b8fBwUHdQ1uQaMqxRIkSuLm58dFHH6Gjo8OYMWPw9PTUeJ4QQoicVbCrgg9E2bJlefHihXo7ODgYc3Nz9faZM2cICQmhf//+jBs3juDgYIYNGwZAUlIS06ZNw8vLi927d1OqVKk8jz+zspvn1atXefHiBRMnTsTa2prg4GDGjRvH/fv38yONd8pujqVLl0ZHR4devXoB8PHHHxMTE0NoaGie56DJ+/y8/vLLL/Tv3x+AJk2aYGpqir+/f94mkAmacgwICODw4cPqbZVKhY6OTprzQkJCUp0nhBAih+Xb6Fyh9vrFktDQUFVMTIyqT58+qhs3bqR77JMnT1K9PLNkyRLVuHHjVPHx8XkVbra9T55v6tixY4F/2Ss7OdrZ2alfDPrnn39UHTt2VCUnJ+dJ3FnxPjkOHjxYdezYMZVKpVI9ePBAZWVlpYqIiMiTuLNCU46hoaGqli1bqh4/fqxSKpWq2bNnqzZv3qxSqVJe9rp69apKpVKp5s6dq3Jzc8uXHIQQ4kOgk9+FtEjp/XFwcMDW1pbExEQGDBhAw4YNsbe356uvvqJBgwbpnhcWFsbevXupVKkSAwcOVLcfP348r0LPkuzmWZi8T45Lly5lwYIF7Nu3Dx0dHVxcXArkUIr3yXHFihUsWLAANzc39PT0WLlyZYH8FCEzOTo5OTFx4kQSExNp2rSpevz26tWrmTdvHtHR0dSrVw9bW9t8zkYIIYouhUqVzqAuIYQQQgghCriC190jhBBCCCFEJkghK4QQQgghCiUpZIUQQgghRKEkhawQQgghhCiUpJAVQgghhBCFkky/JQqcp0+f0qVLF2rVqgWAUqnEwMCAWbNm0axZs3eeO2vWLGrWrMnYsWPzItQ0wsLCaNOmDbdv306zb+TIkTx79izNdFMTJ06kW7duuRbT2/dVKpUkJCQwceJE+vbt+85zb968yeHDh3Fycsq1+IQQQojskkJWFEgGBgap5sP98ccfmT17NmfPns3HqN7fjBkzcrVozex9vby8GDp0KJ07d6ZkyZIZnnfv3j2CgoLyIkQhhBAiy6SQFYXCq1evMDMzA1J6FJctW8aNGzeIjo5GpVKxZMmSNL21hw8fxt3dncTERMLDw7G3t2fYsGF4eHjw888/o6WlxaNHj9DV1WXlypXUqlWLkJAQFi5cyP3799HS0mLIkCHY2toSGRnJ0qVLuXPnDomJibRp04YZM2ago6PD2bNncXFxoVixYtSvXz/bOdavX59OnTrh5+fH6tWrGTp0aKrt+Ph4nJ2diY2NRVdXl6+//pr27dvj4eHB4cOHiY2NpWTJkuzZs0fjvZ48eULx4sXR09PL8HlWqFCB//3vf0RGRjJ79myWL1/O+fPn2bRpE4mJiRgYGDBz5kyaNGmS7ZyFEEKI9yGFrCiQ4uLisLa2BiAiIoKQkBBcXV0BuHHjBsHBwbi7u6OlpcXWrVtxc3NLVchGR0dz6NAhtm7dSunSpfn3338ZPXo0w4YNA+DKlSucOnWKcuXKsXjxYr777jtWrlzJokWLqFq1Khs3biQyMpKhQ4fSoUMHNm/ejKWlJStWrCA5OZlZs2axY8cO+vXrx5w5czhw4AA1atRgy5Yt78zL2dmZTZs2pWrbuXMnpUuXJjExkY4dO7Ju3TqAVNsvX76kZ8+ebNq0iUaNGnH37l1GjBjB4cOHgZSe0/Pnz2fYu/r6vpGRkcTFxdG6dWt27tyJnp4e//zzT7rPc/PmzXz11Vf89NNPLF++nIcPH+Li4sLu3bspXbo0d+/eZfTo0Zw9e5bixYtn47sshBBCvB8pZEWB9PbQguvXr2Nvb8+xY8do0qQJRkZGHDhwgCdPnvDXX39RokSJVOeXKFGCzZs34+npycOHD/Hz8yMmJka939LSknLlygFQr149fv75ZwAuXbrE9OnTAShVqhSnTp0C4Ndff8XLy0tdOMbFxQFw7do1atWqRY0aNQAYPHgwa9asyTAvTUMLmjdvnu72zZs3qVKlCo0aNQKgZs2aNG3alL///huFQkHt2rXfOUTg9X3DwsKwt7fHxMSEevXqAWTqeQJcvHiR4OBgRo0apW5TKBQ8fvyYOnXqZHhvIYQQIrdIISsKhaZNm1KtWjW8vLzw9/dn6dKljB49mk6dOlG9enVOnDiR6vjAwEAGDx7MoEGDaNasGd26dePChQvq/QYGBuqvFQoFr1dq1tHRQaFQqPc9efKE0qVLo1QqWbduHRYWFkBKL7FCoeDy5cu8ucqzjs77/Uq93bP5elupVKY5VqVSkZSUhK6ubqZ7RE1MTFi7di29evWiWbNmdO/enV9//VXj83wdQ5s2bVi7dq267fnz55ibm2chQyGEECLnyPRbolB48OABDx8+pG7duly8eJGOHTsybNgwGjRowLlz50hOTk51vLe3NyYmJkyaNIl27dqpi9i3j3tbmzZtOHLkCACRkZF8/vnnPHz4ECsrK3bu3IlKpVK/8f/999/TvHlz7t27h5+fHwAeHh65kD00atSIBw8ecPPmTQDu3r3LlStXaNmyZZavVblyZSZMmMDy5cuJiYl55/PU1tYmKSkJgNatW3Px4kX8/f0B8PT0pE+fPsTHx+dQlkIIIUTWSI+sKJDeHCMLKb2BTk5OVKtWjSFDhjBt2jR69+6NtrY2zZs35+zZs6l6LT/++GMOHz5Mt27dKFasGA0bNsTExIRHjx69874LFizA0dGR3r17o1KpGD9+PPXr12fu3LksXbqU3r17k5iYSNu2bbGzs0NXV5fVq1czbdo0dHV1adGixTuvn94Y2S5dujB58uR3nmdiYsK6detYvHgxcXFxKBQKli9fTrVq1fjnn3/eeW56xo4dy7Fjx9i4ceM7n2eTJk1Yu3YtX3zxBa6urjg5OfHNN9+gUqnQ0dFh06ZNMj5WCCFEvlGo3vxcVAghhBBCiEJChhYIIYQQQohCSQpZIYQQQghRKEkhK4QQQgghCiUpZIUQQgghRKEkhawQQgghhCiUpJAVQgghhBCFkhSyQgghhBCiUJJCVgghhBBCFEr/B3aAqeeIzGkhAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 260 }, - { - "cell_type": "markdown", - "metadata": { - "id": "frGlrXHAyn2C" - }, - "source": [ - "In the last section, we fill out the `quantitative_analysis` section where we describe the model's performance metrics on the evaluation dataset. In particular, we want to report the model's disagregated performance with respect to our three metrics including false negative rate, which quantifies fairness-related harms." - ] + "id": "r9TKGsNY8Myu", + "outputId": "06146e22-ea39-4d70-9488-0dd9b03eddca" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "### Exercise: Train an `ExponentiatedGradient` model" + ], + "metadata": { + "id": "1MozjJKkZqz_" + } + }, + { + "cell_type": "markdown", + "source": [ + "In this section, we will explore how changing the base model for the `ExponentiatedGradient` affects the overall performance of the classifier. \n", + "\n", + "We will instatiate a new `ExponentiatedGradient` classifier with a base `HistGradientBoostingClassifer` estimator. We will use the same `difference_bound` as above." + ], + "metadata": { + "id": "lLeGnB4juJsM" + } + }, + { + "cell_type": "markdown", + "source": [ + "1.) First, let's create our new `ExponentiatedGradient` instance in the cells below and fit it to the training data." + ], + "metadata": { + "id": "ghjfKhtB3Kl9" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "# Create ExponentiatedGradient instance here\n", + "expgrad_exercise = ExponentiatedGradient(\n", + " estimator=_______,\n", + " constraints=TruePositiveRateParity(difference_bound=____)\n", + ")" + ], + "outputs": [], + "metadata": { + "id": "CNjDh4JUAc6i" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "# Fit the new instance to the balanced training dataset\n", + "expgrad_exercise.fit(________, _________, sensitive_features=________)" + ], + "outputs": [], + "metadata": { + "id": "MFIAEkEBAc9P" + } + }, + { + "cell_type": "markdown", + "source": [ + "2.) Now, let's compute the performance of the `ExponentiatedGradient` model and compare it with the performance of `ExponentiatedGradient` model with logistic regression as base estimator" + ], + "metadata": { + "id": "8H_UDlAs3M-D" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "# Save the predictions and report the disagregated metrics\n", + "# of the exponantiated gradient model\n", + "Y_expgrad_exercise = expgrad_exercise.predict(X_test)\n", + "mf_expgrad_exercise = MetricFrame(\n", + " metrics=metrics_dict,\n", + " y_true=Y_test,\n", + " y_pred=_______,\n", + " sensitive_features=________\n", + ")\n", + "mf_expgrad_exercise.______" + ], + "outputs": [], + "metadata": { + "id": "_Qkfth4ZZBQV" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "# Compare with the disaggregated metric values of the\n", + "# exponentiated gradient model based on logistic regression\n", + "metricframe_reductions.____" + ], + "outputs": [], + "metadata": {} + }, + { + "cell_type": "markdown", + "source": [ + "3.) Next, calculate the balanced error rate and false negative rate difference of each of the inner models learned by this new `ExponentiatedGradient` classifier." + ], + "metadata": { + "id": "Boo771yFUyJ7" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "# Save the inner predictors of the new model\n", + "predictors_exercise = expgrad_exercise.predictors_" + ], + "outputs": [], + "metadata": { + "id": "jws7w2z6RWUM" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "# Compute the balanced error rate and false negative rate difference for each of the predictors on the test data.\n", + "balanced_error_exercise = [(1 - ______(Y_test, pred.predict(X_test))) for pred in predictors_exercise]\n", + "false_neg_exercise = [(______(Y_test, pred.predict(X_test), sensitive_features=_____)) for pred in predictors_exercise]" + ], + "outputs": [], + "metadata": { + "id": "aVTqrrSRAtGc" + } + }, + { + "cell_type": "markdown", + "source": [ + "4.) Finally, let's plot the performances of these individual inner models. In the below cells, plot the individual inner predictors against the performance of their corresponding exponentiated gradient model as well as the unmitigated logistic regression model, and the `ThresholdOptimizer`." + ], + "metadata": { + "id": "Aos3gdHjDQEi" + } + }, + { + "cell_type": "code", + "execution_count": null, + "source": [ + "# Plot the individual predictors against the Unmitigated Model and the ThresholdOptimizer\n", + "plt.scatter(balanced_error_exercise, false_neg_exercise,\n", + " label=\"ExponentiatedGradient - Iterations - Exercise\")\n", + "for i in range(len(predictors_exercise)):\n", + " plt.annotate(str(i), xy=(balanced_error_exercise[i]+0.001, false_neg_exercise[i]+0.001))\n", + "\n", + "plt.scatter(1-balanced_accuracy_score(Y_test, Y_expgrad_exercise),\n", + " false_negative_rate_difference(Y_test, Y_expgrad_exercise, sensitive_features=A_test),\n", + " label=\"ExponentiatedGradient - Combined - Exercise\")\n", + "plt.scatter(1-balanced_accuracy_score(Y_test, Y_pred),\n", + " false_negative_rate_difference(Y_test, Y_pred, sensitive_features=A_test),\n", + " label=\"Unmitigated\")\n", + "plt.scatter(1-balanced_accuracy_score(Y_test, Y_pred_postprocess),\n", + " false_negative_rate_difference(Y_test, Y_pred_postprocess, sensitive_features=A_test),\n", + " label=\"ThresholdOptimizer\")\n", + "\n", + "plt.xlabel(\"Balanced Error Rate\")\n", + "plt.ylabel(\"False Negative Rate Difference\")\n", + "plt.legend(bbox_to_anchor=(1.9,1))\n", + "plt.show()" + ], + "outputs": [], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 260 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "1cyHIMlAJqod" - }, - "outputs": [], - "source": [ - "def metricframe_to_dictionary(mframe, feature_name):\n", - " \"\"\"\n", - " Converts a MetricFrame into a Dictionary object that can be accepted by the Model Card's\n", - " Quantitative Analysis section.\n", - " \"\"\"\n", - " group_metrics = mframe.by_group[feature_name].reset_index()\n", - " group_metrics = group_metrics.melt(id_vars=\"race\", var_name=\"type\", value_vars=feature_name).rename(columns={\"race\":\"slice\"})\n", - " return group_metrics.to_dict(orient=\"records\")" - ] + "id": "IxTjb8CeZqPI", + "outputId": "8e06051c-4629-42f0-93f6-51931da2615b" + } + }, + { + "cell_type": "markdown", + "source": [ + "## Comparing performance of different techniques" + ], + "metadata": { + "id": "iyknfgrsW1gi" + } + }, + { + "cell_type": "markdown", + "source": [ + "Now we have covered two different class of techniques for mitigating the fairness-related harms we found in our fairness-unaware model. In this section, we will compare the performance of the models we trained above across our key metrics." + ], + "metadata": { + "id": "Vee-7c2Tw33O" + } + }, + { + "cell_type": "markdown", + "source": [ + "#### Model performance - by group" + ], + "metadata": { + "id": "XEXnEeLl7mgc" + } + }, + { + "cell_type": "code", + "execution_count": 89, + "source": [ + "def plot_technique_comparison(mf_dict, metric):\n", + " \"\"\"\n", + " Plots a specified metric for a given dictionary of MetricFrames.\n", + " \"\"\"\n", + " mf_dict = {k:v.by_group[metric] for (k,v) in mf_dict.items()}\n", + " comparison_df = pd.DataFrame.from_dict(mf_dict)\n", + " comparison_df.plot.bar(figsize=(12, 6), legend=False)\n", + " plt.title(metric)\n", + " plt.xticks(rotation=0, ha='center');\n", + " plt.legend(bbox_to_anchor=(1.01,1), loc='upper left')" + ], + "outputs": [], + "metadata": { + "id": "SNyCZxHJuXZV" + } + }, + { + "cell_type": "code", + "execution_count": 88, + "source": [ + "test_dict = {\n", + " \"Reductions\": metricframe_reductions,\n", + " \"Unmitigated\": metricframe_unmitigated,\n", + " \"Postprocessing\": metricframe_postprocess,\n", + " \"Postprocessing (DET)\": mf_deterministic\n", + "}" + ], + "outputs": [], + "metadata": { + "id": "9dNb-kI3uHzM" + } + }, + { + "cell_type": "code", + "execution_count": 90, + "source": [ + "plot_technique_comparison(test_dict, \"false_negative_rate\")" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAAGECAYAAABZM+NjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABFpUlEQVR4nO3deXxMZ///8fdksglZNBJb0VqqliilFe7auRUJsafuUqpR3KqNLlSU2NXaWqqWLlpUlCihRVUtFbve9tbWWhpNQiqCRJaZ3x9+nW/ThIROnCRez8fD4+Gcc53rfGbmymTeOWeuY7JarVYBAAAAAO47B6MLAAAAAIAHFYEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiAD8MAYNWqUmjdvrhkzZmS7fffu3QoICLjPVdnXoUOHNHLkSEnS4cOHNXjw4HxTDwAAyMrR6AIA4H6JiIjQli1bVKpUKaNLyTOnTp1SbGysJMnPz08zZ87MN/UAAICsCGQAHgg9evSQ1WpVSEiInn32WW3fvl2pqalKSEhQUFCQXnvttUzt9+3bp0mTJslisUiSXn75ZbVu3VqpqamaOnWq9u7dq4yMDFWvXl0jRoxQsWLFbnvs3bt3a8aMGSpXrpxOnjyp1NRUjRw5Uv7+/nfs79ChQwoPD1daWprKly+vmJgYDRs2TE899ZQmTJiggwcP6vr167JarRo3bpzKlCmjmTNnKikpSW+//baCgoI0duxYffHFF2rSpIk2bNggHx8fSVK3bt303//+Vw0aNLinxzN+/Hi5ubnpxo0bWrFihSZPnpxjPRMnTtTmzZs1d+5cpaWlydXVVUOHDlWdOnX++QsMAEABxSWLAB4IS5culSQtWrRIu3bt0qRJkxQZGamIiAjNnz9fCQkJmdrPmjVLffr0UWRkpCZMmKBdu3ZJkubPny+z2azIyEitWbNGvr6+mjp1ao7HP3TokF588UV99dVX6tKli2bPnn3H/tLT0/XKK6/o1VdfVVRUlHr27Knjx49Lkg4ePKi4uDhFRETo66+/VseOHbVgwQKVLl1agwcPVr169TRx4kTbsd3d3dWqVSutWbNGknT69GnFx8erUaNG9/x4Tp48qWnTpmnNmjU6evRorur59ddfNWPGDM2fP19fffWVxo4dq1deeUU3btzIxSsIAEDhxBkyAA+cDz/8UFu2bNHatWt1+vRpWa1WJScnZ2rTpk0bjRkzRps3b1bDhg01ZMgQSdKWLVuUlJSk6OhoSVJaWpq8vb1zPGaZMmVUrVo1SVL16tW1atWqO/Z34sQJSVKTJk0kSf7+/qpSpYokqU6dOvL09NSyZct0/vx57d69W0WLFr3j8bt27arRo0erb9++WrlypTp16iQHB4d7fjylS5dW2bJl76qeHTt2KC4uTr1797atM5lMOnfunB5//PEcjwkAQGFEIAPwQElOTlZwcLBatmypevXqqXPnztq0aZOsVmumdsHBwWrWrJl27Nih7du3a/bs2VqzZo0sFouGDx9uC0rXr1/XzZs3czyuq6ur7f8mk8l2vNv1Fx8fn6Ums9ks6VaIGz9+vPr06aMWLVqoYsWKtrNft1OvXj2lp6fr0KFDWrt2rZYtW3bH4+fEzc3N9v/c1mOxWNSgQQO99957tnUXL16Ur69vjscDAKCw4pJFAA+UhIQEXbt2Ta+99pqaN2+uPXv2KDU11fZdsT8FBwfr+PHj6tSpk8aOHaurV68qMTFRzzzzjJYsWWLb55133tH06dPvuZ7b9VepUiU5Oztr27Ztkm5d8njixAmZTCbt2LFDzZo1U48ePeTn56dNmzYpIyND0q3Qlp6enu2xunbtqrFjx6pq1aoqU6bMHY9/N3Jbj7+/v3bs2KHTp09LkrZu3ar27dvnKgACAFBYEcgAPFDKlCmjpk2bqk2bNurYsaM2b96sypUr6+zZs5navfHGG5o5c6aCgoLUq1cvDRo0SA8//LAGDhyosmXLqmPHjmrbtq2sVquGDRt2z/Xcrj9HR0fNmjVLs2fPVlBQkD7++GOVKFFCrq6uCg4O1t69exUYGKju3burXLlyunDhgiwWi+rUqaMzZ87ov//9b5ZjBQUF6fjx4+ratWuOx78bua2nSpUqGjNmjIYMGaL27dvr/fff19y5czOdbQMA4EFjsv79mhgAQL7w7rvvqm/fvipRooQuXryoDh06aNOmTfLw8DC6NAAAYCd8hwwA7OC1117TL7/8ku22GTNmqGLFinfdZ9myZdW7d285OjrappK/X2EsLx4PAADIijNkAAAAAGAQvkMGAAAAAAbJ1SWLUVFRmjt3rtLS0tS7d2/95z//sW07fvx4pi+AJyQkyNPTU2vXrrV/tQAAAABQiOR4yWJsbKyee+45RUZGytnZWcHBwZo+fboqV66cpW1ycrK6du2q8PBw1atXL8+KBgAAAIDCIMdLFqOjo+Xv7y8vLy+5ubmpdevWWr9+fbZt582bp6eeeoowBgAAAAC5kOMli3FxcfLx8bEt+/r66tChQ1naXb16VcuXL1dUVNRdF/HHH9dlsTC3iD14exfT5cvXjC4DyIKxifyM8Yn8irFpPw4OJhUvXtToMoAscgxk2V3RaDKZsqyLiopSy5Yt5e3tfddF8MNhX97exYwuAcgWYxP5GeMT+RVjEyjccgxkJUuW1L59+2zLcXFx8vX1zdJu06ZNevnll++piMuXr3GGzE58fNwVH59kdBlAFoxN5GeMT+RXjE37cXAwEW6RL+X4HbKGDRtq586dSkhIUHJysjZu3KjGjRtnamO1WnX06FHVqVMnzwoFAAAAgMImx0BWsmRJhYaGqlevXgoKClJAQIBq1aqlkJAQHT58WNKtqe6dnJzk4uKS5wUDAAAAQGGR47T39wOXLNoPlzYgv2JsIj9jfCK/Ymzaz71cspiWlqbz588rOTklj6rCg6BIEVeVK1dOTk5O2W7P1Y2hAQAAgAfN+fPn5ejootKlfbOd1A7IidVqVVJSos6fP6+KFStm2ybHSxYBAACAB1FycoqKFfMkjOGemUwmubt73vEsK4EMAAAAuA3CGP6pnMYQlywCAAAAuVDEzUWuLvb/+JxyM13JN27m2C4mJkbdugXp0UdvXfpmsVh0/fp1tWsXoJCQAbk61oIFH0qSQkL631WNMTG/6ZNPFiosbJSOHz+myMgVCgsbeVd9IHsEMgAAACAXXF0cFfj6arv3GzWtQ64CmSSVKOGjzz9fZluOj49X164d1LJla1tQywsXL17UhQsXJEnVqlUnjNkRgQwAAAAooC5fjpfVKhUtWlSfffaJNm36VhZLhurXb6BBg16VyWTS4sWL9NVXkfLy8pK7u4eqV68hSfL3f1K7dh2QJK1du0YHDuzXyJGjtWfPbs2cOV1Wq1WlSpXSmDETNH36ZMXE/KYpUyaqefNWWrhwnubOXaBz585q4sRxuno1UUWKFNGQIW+pevUaGjNmlIoVK6affjquuLhYvfRSPwUEdNDevbs1e/b7//+7VR4aO3aCvLyKG/kUGo5ABgAAABQQly7Fq2fPYKWmpurKlSuqVq263n13qk6fPqWffjquTz75XCaTSeHh72j9+q/1yCOPKipqtT777AuZTCa99NILtkCWndTUVI0aFab335+jxx6rqrlzZ2ndurUaMuQtLVw4T2+++bb2799nax8ePkI9e/ZWs2YtdOTIIQ0f/qaWL/9KkhQb+7vmzftIp0+f0sCBtwLZJ598pKFDw1S9eg1FRHyhn3/+SfXrN8jrpy1fI5ABAAAABcSflyxaLBbNnDldp06dVL16T+mDD2br6NEj6t37P5KkmzdvqlSpUkpIuKyGDZ+Rm5ubJKlFi1bKyMi4bf+nT5+Sj4+vHnusqiRpwIBXJClTCPvTjRs3dOHCeTVr1kKSVLNmLXl4eOrs2V8lSfXrN5DJZFKlSpV19WqiJKlRo8YaOvR1NWnSVI0aNVX9+v72eWIKMAIZAAAAUMA4ODho0KDX1KvXc1qy5HNZLBnq3r2HevR4XpKUlJQks9msVatWymq12PYzm82ZApnVapXJZFJ6erokydExczy4di1JN27cyLYGq9Uiq9X6t3VWW//Ozs6SMs8y+Nxzz+uZZxprx47tmjPnfR07dkR9+rx0r09DocC09wAAAHZQ3NNZPj7udv1nSU81+mEhH3N0dNQrr7ymRYs+VtWqj2v9+nW6ceOG0tPT9dZbQ7R58yY99dTT+uGH7bp2LUk3b97U1q3f2/b38vLSmTOnZbVatX37VklS+fIVdOXKH/rllzOSpM8/X6TIyBVZgpwkFS1aTGXLPqzvv/9OknTkyCFdvnxZlSpVvm3NL77YSzdu3FBw8H8UHPwf/fzzT/Z+WgoczpABAADYgaOzi86M72zXPiuGrZSUu9n38GBq0OBfqlHDTwcOHFDTpi3Ut28vWSwW+fs3ULt2gTKZTAoO7qE+fXrK3d1DpUqVtu07cOBgvf76q/L29tYTT9TRlStX5OLiovDwcRo9+h2lpaXp4YfLadSosUpLS9W1a0kKDx+hwMAgWx/h4eP07rsTtHDhPDk5OWnSpKlycnK6bb0DBvxXY8eOktlslouLq4YOHZ6XT0+BYLL+/TyjAS5fviaLxfAyCgUfH3fFxycZXQaQBWMT+RnjE/bg4+OeJ4GMsWkfDg4meXsXu6t9jh49pjJlKtiWjb4PGQqumJizqlGjerbbOEMGAAAA5ELyjZsEJ9gd3yEDAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAACAXPIo6qnjxonb/51E0d3eiiomJUVBQuyzr/f2f/MePrWfPYEnS0aNHNHv2+5Kkbdu2av78uffU3/btW7V06eK72mfAgBDt37/vno5XkHEfMgAAACAXzM4udr/5t3TrBuC6nm73fu/G558vkyT98ssZJSQkSJIaN26ixo2b3FN/P/103G61FXYEMgAAAKCAW7t2jXbtitbVq1cVE3NBTz/dQG+99bb279+nTz/9SFarVb/9dkHNm7dQ0aLu2rbte1mtVk2fPkve3t7y939S3367VQsWfKjk5Bv65JOF8vHx1YED+zVy5Gjt379P06dPltlsVs2atfTLL2c0d+4CHTiwXx9+OEcpKSlKSrqqQYNeVcWKlbRq1UpJUunSpdW8eStNnTpJp0+fksViUc+evfXvfz+r1NRUTZgwRsePH1Pp0mWUmHjF2CfRIAQyAAAAoBA4fPiQli79Umazg7p166ROnbpIunUZ4hdffClPT0+1adNSgweH6tNPl2js2FHatGmDunfvIUlyd3dXSEh/HTiwX336vKS1a9dIktLT0zR69DuaNu19VanymKZPn2I75pdfLtPw4e/okUce1b59ezRjxlQtWbJcHTveOpMYENBBc+bMVNWq1TRy5Bhdv35NISF9VKNGTW3ZslmSFBERqXPnzqlnz+738+nKNwhkAAAAQAHg4GDKss5qtcpkurXez6+WihYtKkkqW7asrl69KkmqVKmSSpYsJUny9PRSvXpPS5JKlSpta3Mnp06dUvHixVWlymOSpMDADrZQFh4+Tjt2bNfmzZt05Mhh3bhxI8v+e/fuVkpKitauXS1JSk5O1pkzp3XgwD4FBd0KbuXLl5efX63cPxmFCIEMAAAAKADc3T10/fq1TOsSEhLk7u4hSXJ2dvnLFpOsVqskydHRKdM+ZvPdRQCz2Wzr6+/69++rJ5+sp7p166levac1cuTwLG0sFovCw8fp8cerSZIuX74sT08PffVVZKZ+zWbzXdVVWDDLIgAAAFAAFC1aVOXKldfmzd/Z1q1eHamnnnrabscwm83KyMg8wcgjjzyqq1ev6tSpk5KkDRu+kclkUmJios6dO6d+/QaoYcNntHv3Tlkslv/fj6MyMjIkSXXrPqXIyBWSpEuX4vX88931+++/6+mn62vDhm9ksVh08WKMDh8+ZLfHUZBwhgwAAAAoIMLDx2ny5An6+OP5SktLU+XKVfTmm8P0ww/b7dJ/jRo19dFH8zRnzkxVqPCIJMnJyUnh4eM0ZsxImUwmVajwiFxcXOTp6an27YPUo0dXFS1aVDVr1lJKSoqSk5NVp04djRkzSg895K2XXuqnyZMnqkePrsrIyNCgQa/q4YfLqXPnrjp9+rSCgzurVKnSqlixkl0eQ0Fjst7u/ON9dPnyNVkshpdRKPj4uCs+PsnoMoAsGJvIzxifsAcfH3e7T4leMWwlY9NOHBxM8vYudlf7HD16TGXKVLAtexR1lDnTZYH2kZF6U1cNnvb+TiwWi+bMmamXXnpZRYoU0dKlixUfH6dXXx1idGkFRkzMWdWoUT3bbZwhAwAAAHLh6vV0w+8XZgQHBwd5enqqT5/n5eTkpNKlyygsbKTRZRUaBDIAAAAAd9SrVx/16tXH6DIKJSb1AAAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAzCpB4AAABALrgVc5KLk7Pd+72Zlqob19JybBcTE6Nu3YL06KMVZTKZlJaWphIlfPTOO+Hy9S2Z6+PFxPymTz5ZqLCwUf+k7DwXGvqKhg8fKR8fH6NLyVMEMgAAACAXXJyc1S1igN37Xd59rm4o50AmSSVK+Ojzz5fZlj/4YJamTZusd9+dluvjXbx4URcuXLjrOu+3GTNmGV3CfUEgAwAAAAqo2rWf1PbtW3XkyCFNnz5Vqak35eXlpaFDw1SuXHktXbpYX38dJQcHB1WvXkPDho3Q9OmTFRPzm6ZMmajmzVtp4cIPZTY7Ki4uVtWr19Dw4SN16dIlhYYOkqenl5ydnTVz5geaMWOq9u3bI5PJpGefbadevXrLarVqzpyZ2rr1e5nNZnXs2Fndu/fQ+fPnNHnyRCUmJsrV1VWvv/6WqlZ9XBs2fKPFixfJwcFBZcqUVXj4OCUmXtGoUWFKTk6Wg4ODhgx5UzVr1lJQUDt98MECHTiwT7t2Revq1auKibmgp59uoLfeelvSrUC6efMmeXl5ydu7hBo1aqKAgPYGvyp3J1eBLCoqSnPnzlVaWpp69+6t//znP5m2nzlzRqNGjVJiYqJ8fHw0ffp0eXp65knBAAAAAKT09DRt2rRRNWrU1IgRb2vChMmqXr2GvvvuW40cOVwLFnyqzz77WGvXbpCDg1lTpkxSXFychgx5SwsXztObb76t/fv36dixo/rssy9UvnwFhYUN1YoVy9W0aXOdPfurIiPXqkyZMlqxYrni4mK1eHGE0tJSNXBgP1WqVEkpKSk6dOiglixZrvT0dL388otq2fLfGjNmlN54Y6iqVn1cv/xyRkOHvq7ly1dp3rwPtHDhIj300EP68MM5Onv2V23btkX/+lcjPf/8C9q/f58OHvyfataslemxHj58SEuXfimz2UHdunVSp05ddPFijA4e/FFffLFCycnJeuGFHmrUqIlBr8a9yzGQxcbGasaMGYqMjJSzs7OCg4NVv359Va5cWZJktVo1YMAAhYWFqXHjxpo6darmz5+vN998M8+LBwAAAB4kly7Fq2fPYElSamqqatSoqXbt2uvEiZ9VvXoNSVKLFq00adI4paQky8/vCfXp01ONGzdRly7d5Ovrq/Pnz2Xqs3btOqpQ4RFJ0rPPttPq1SvVtGlzFS/+kMqUKSNJ2r9/r9q1C5TZbJbZXEStW7fR3r17lJ6ephYtWsnZ2VnOzs76/PNlunHjho4fP6px48Jtx0hOvqHExCt65pnG6tevj5o0aapmzVrosceqKjk5WcOGvaETJ35Ww4bPqEuX7lket59fLRUtWlSSVLZsWV29elV79uxWixb/lpOTk5ycnNSkSVM7P9v3R46BLDo6Wv7+/vLy8pIktW7dWuvXr9egQYMkSUePHpWbm5saN24sSerfv7+uXr2adxUDAAAAD6i/f4dMkk6ePJGlndVqVUaGRZMnT9eRI4e0c2e0XnttkEaPHp+lrdns+Jf9LLZlFxcX23qLxZJN/xlydMwcJ2JiYuTh4S5nZ5dMdcbFxcrDw1NDhrypwMAOio7+QeHhI9S378tq06adli1boR9+2K5NmzZq3boozZo1N1O/zs4uf1kyyWq1ymx2kNWaua6CKMdp7+Pi4jLNbOLr66vY2Fjb8rlz51SiRAkNHTpUgYGBGjVqlNzc3PKmWgAAAACZVKjwiBITE3Xs2FFJ0qZNG1WqVGlZLBYFB3dWpUpV1K/fANWv769Tp07KbDYrIyPDtv/Bg/9TXFycLBaLvv56rRo0aJjlGPXqPaV166KUkZGhlJRkbdjwjerWrafatZ/Uli2blZ6eppSUZIWGDlJCQoLKlSunb75ZJ0navXuX+vd/SRkZ6erSpYO8vIrrhRdeVJs2ATpx4mfNmvWevvlmndq1C9QbbwzVzz//lKvH/fTT/vr++++Ulpam69ev6YcftstkMtnhGb2/cjxDZrVas6z76wNNT0/Xnj17tHjxYvn5+em9997TpEmTNGnSpFwX4e1dLNdtkTMfH3ejSwCyxdhEfsb4RH7F2EROnJ2dNW7cRE2d+q5SUpLl4eGhceMmqXjx4goK6qQ+fZ6Xq6urSpUqrXbtApWenqZr15IUHj5CgYFBKlGihMaMeUfx8fF66qn6at++Y6YTMJLUsWNnnTt3Tj17Bis9PV3PPttWTZs2lyQdP35ML7zwH1ksFnXr9pzKl6+g0aPH6913J2jx4kVycnLSuHGT5OjopH79BuiVV/rL1dVVxYq5a+TIMbJaLRo5crjWrbs1+cifE3bkpGHDZ3To0EH16vWcPDw85OPjk+msXkFhsmaXuP5i1apV2rdvn8aPv3V6c86cObJarbZLFnfu3KmJEydqzZo1kqRTp05p8ODB+vrrr3NdxOXL12Sx3LEM5JKPj7vi45OMLgPIgrGJ/IzxCXvw8XHXmfGd7dpnxbCVjE07cXAw3fVJgKNHj6lMmQq2ZaPvQ5YX9u/fp4UL52nu3AWGHP+fOHz4oM6dO2cLmS+91FthYaNUpcpjRpeWRUzMWdWoUT3bbTmeIWvYsKFmzZqlhIQEFSlSRBs3btTYsWNt2+vUqaOEhAT99NNPevzxx7V582bVqFHDftUDAAAA+cCNa2m5vl8Y8l758o/oo4/m64svFstqtapt24B8GcZykmMgK1mypEJDQ9WrVy+lpaWpS5cuqlWrlkJCQjR48GD5+flpzpw5GjFihJKTk1WqVClNnjz5ftQOAAAA4B+oW7ee6tatZ3QZ98TT01PvvTfH6DL+sVzdhywwMFCBgYGZ1i1Y8H+nNZ944gmtWLHCvpUBAAAAQCGX4yyLAAAAAIC8QSADAAAAAIMQyAAAAADAIAQyAAAAADBIrib1AAAAAB507kUc5ehq/xsPp6fcVFJy+h3bxMTEqFu3ID36aEWZTCalpaWpRAkfvfNOuHx9S97V8WJiftMnnyxUWNiof1J2ngsNfUXDh4+Uj4/PP+5rzJiRGjDgFe3evVMzZ05XyZKlZLValZqaqkaNGmvgwMEym81au3aNbftfDR0apm++WatDhw4qLS1NFy6c16OPVpQkde/+nIoVc1d8fJy6dg2+69oIZAAAAEAuOLq6aEcH+978W5L+tXqllEMgk6QSJXz0+efLbMsffDBL06ZN1rvvTrur4128eFEXLly46zrvtxkzZtmlnx9+2KYSJXxswe6ZZ5po5MjRkqQbN27orbeGaOHCeXr55YFZtv9VzZp+km6F44EDQzK9FpL03//2U4sW/9ZDDz10V/URyAAAAIACqHbtJ7V9+1ZJ0pEjhzR9+lSlpt6Ul5eXhg4NU7ly5bV06WJ9/XWUHBwcVL16DQ0bNkLTp09WTMxvmjJlopo3b6WFCz+U2eyouLhYVa9eQ8OHj9SlS5cUGjpInp5ecnZ21syZH2jGjKnat2+PTCaTnn22nXr16i2r1ao5c2Zq69bvZTab1bFjZ3Xv3kPnz5/T5MkTlZiYKFdXV73++luqWvVxbdjwjRYvXiQHBweVKVNW4eHjlJh4RaNGhSk5OVkODg4aMuRN1axZS0FB7fTBBwt04MA+7doVratXryom5oKefrqB3nrrbUm3QunmzZvk5eUlb+8SatSoiQIC2md6nhYvXqRhw0Zk+xy6ublpwIBBGjJksPr1G/CPXo+mTZtr5coIhYTcXT8EMgAA8MBx9ygiVxc+BqHgSk9P06ZNG1Wr1hNKS0vTiBFva8KEyapevYa+++5bjRw5XAsWfKrPPvtYa9dukIODWVOmTFJcXJyGDHlLCxfO05tvvq39+/fp2LGj+uyzL1S+fAWFhQ3VihXL1bRpc509+6siI9eqTJkyWrFiueLiYrV4cYTS0lI1cGA/VapUSSkpKTp06KCWLFmu9PR0vfzyi2rZ8t8aM2aU3nhjqKpWfVy//HJGQ4e+ruXLV2nevA+0cOEiPfTQQ/rwwzk6e/ZXbdu2Rf/6VyM9//wL2r9/nw4e/J9q1qyV6fEePnxIS5d+KbPZQd26dVKnTl108WKMDh78UV98sULJycl64YUeatSoSab9EhMTde7cOT3yyKO3fS4rVaqkxMQr+uOPPyRJP/ywVT17/t+lh05Ozvr4489yfE1q135SY8eOIpABAADkxNXFUYGvr7Zrn1HTOti1P+DvLl2KtwWF1NRU1ahRUwMHDta5c2fl4eGh6tVrSJJatGilSZPGKSUlWX5+T6hPn55q3LiJunTpJl9fX50/fy5Tv7Vr11GFCo9Ikp59tp1Wr16ppk2bq3jxh1SmTBlJ0v79e9WuXaDMZrPM5iJq3bqN9u7do/T0NLVo0UrOzs5ydnbW558v040bN3T8+FGNGxduO0Zy8g0lJl7RM880Vr9+fdSkSVM1a9ZCjz1WVcnJyRo27A2dOPGzGjZ8Rl26dM/y2P38aqlo0aKSpLJly+rq1avas2e3WrT4t5ycnOTk5KQmTZpm2e+33y6oRImcvoNmkiS5uNz6fuDtLlnMSenSpbM8t7lBIAMAAAAKgL9/h+xPcXGxWdZZrVZlZFg0efJ0HTlySDt3Ruu11wZp9OjxWdqazY5/2c9iW/4zoEiSxWLJpv8MOTpmjhMxMTHy8HCXs7NLplrj4mLl4eGpIUPeVGBgB0VH/6Dw8BHq2/dltWnTTsuWrdAPP2zXpk0btW5dlGbNmpupX2fnv06mYpLVapXZ7CCrNXNdf+fgYJKjo/mObU6dOilf35K2wHevHB0dZTLd/ST2THsPAAAAFGAVKjyixMREHTt2VJK0adNGlSpVWhaLRcHBnVWpUhX16zdA9ev769SpkzKbzcrIyLDtf/Dg/xQXFyeLxaKvv16rBg0aZjlGvXpPad26KGVkZCglJVkbNnyjunXrqXbtJ7Vly2alp6cpJSVZoaGDlJCQoHLlyumbb9ZJknbv3qX+/V9SRka6unTpIC+v4nrhhRfVpk2ATpz4WbNmvadvvlmndu0C9cYbQ/Xzzz/l6nE//bS/vv/+O6Wlpen69Wv64YftMplMmdqULl0228D6p2vXkjR//lx16dItV8e8k5iYGD38cLm73o8zZAAAAEAB5uzsrHHjJmrq1HeVkpIsDw8PjRs3ScWLF1dQUCf16fO8XF1dVapUabVrF6j09DRdu5ak8PARCgwMUokSJTRmzDuKj4/XU0/VV/v2HRUbmznEdOzYWefOnVPPnsFKT0/Xs8+2VdOmzSVJx48f0wsv/EcWi0Xduj2n8uUraPTo8Xr33QlavHiRnJycNG7cJDk6OqlfvwF65ZX+cnV1VbFi7ho5coysVotGjhyudetuTT7y54QdOWnY8BkdOnRQvXo9Jw8PD/n4+GQ6qydJnp6eKlv2Yf3yyxnbNPV/fkfMZDIpIyNDzZq10PPPv2Db5+/fIZOk5557Xm3bBtyxnv3796px4yZ3bJMdk9Vqtd71XnZ2+fI1WSyGl1Eo+Pi4Kz4+yegygCwYm8jPGJ8PHh8f9zz5DtmZ8fadEr1i2ErGpp04OJjk7V3srvY5evSYypSpYFs28j5keWX//n1auHCe5s5dYMjx/4nDhw/q3LlztpD50ku9FRY2SlWqPJap3bZtW/W//x3Q4MGheVpPv34vatKkqdlOex8Tc1Y1alTPdj/OkAEAAAC5kJScnqv7heH+KF/+EX300Xx98cViWa1WtW0bkCWMSVLjxk20adNGxcfH2+Um09nZvHmTmjdvcdf3IJMIZAAAAMADq27deqpbt57RZdwTT09PvffenFy1HTMm62Qm9tS8ect73pdJPQAAAADAIAQyAAAA4DbywXQLKOByGkMEMgAAACAbRYq4KikpkVCGe2a1WpWUlKgiRVxv24bvkAEAAADZKFeunM6fP6+LF88ZXQoKsCJFXFWu3O3vT0YgAwAAALLh5OSkihUrGl0GCjkuWQQAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwiKPRBQAAcDeKezrL0dnFrn1a0lPt2h8AALlFIAMAFCiOzi46M76zXfusGLZS0k279gkAQG4QyAAAecbdo4hcXfhVAwDA7fBbEgCQZ1xdHBX4+mq79hk1rYNd+wMAwEhM6gEAAAAABiGQAQAAAIBBCGQAAAAAYJBcBbKoqCi1bdtWrVq10pIlS7Jsnz17tpo1a6YOHTqoQ4cO2bYBAAAAAGSW46QesbGxmjFjhiIjI+Xs7Kzg4GDVr19flStXtrU5cuSIpk+frjp16uRpsQAAAABQmOR4hiw6Olr+/v7y8vKSm5ubWrdurfXr12dqc+TIES1YsECBgYEaM2aMbt7kXi4AAAAAkJMcA1lcXJx8fHxsy76+voqNjbUtX79+XdWqVdPQoUO1atUqXb16VR988EHeVAsAAAAAhUiOlyxardYs60wmk+3/RYsW1YIFC2zLL774ooYPH67Q0NBcF+HtXSzXbZEzHx93o0sAssXYRH7G+ER+xdgECrccA1nJkiW1b98+23JcXJx8fX1tyzExMYqOjlaXLl0k3Qpwjo53d7/py5evyWLJGvxw93x83BUfn2R0GUAWjM0HU0H6IMn4fLAwNh88Dg4mTgIgX8rxksWGDRtq586dSkhIUHJysjZu3KjGjRvbtru6umrKlCk6f/68rFarlixZolatWuVp0QAAAABQGOQYyEqWLKnQ0FD16tVLQUFBCggIUK1atRQSEqLDhw/roYce0pgxYzRgwAA9++yzslqt6tOnz/2oHQAAAAAKtFxdWxgYGKjAwMBM6/76vbHWrVurdevW9q0MAAAAAAq5XN0YGgAAAABgfwQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADOJodAEAABgtNSNNPj7udu0zJS1VSVdu2rVPAEDhQyADADzwnM1O6hYxwK59Lu8+V0kikAEA7oxLFgEAAADAIAQyAAAAADAIgQwAAAAADMJ3yIACzt2jiFxd7PujnHIzXUlXk+3aJwAAALIikAEFnKuLowJfX23XPle/28buM85Z0lPt2h8AAEBhQCADkIWDo7POjO9s1z4rhq2UmHEOAAAgE75DBgAAAAAG4QwZAAB5wJKaavdLf9NTbuqPJC7/fZBw03Kg8COQAQCQBxycnbWjg30v/f3X6pUSgeyBwk3LgcKPSxYBAAAAwCAEMgAAAAAwCJcsGqi4p7McnV3s2idTiwMAAAAFB4HMQI7OLkwtDgAAADzACGS55O5RRK4uPF0AAAAA7IeEkUuuLo4KfH21XfuMmtbBrv0BAAAAKFiY1AMAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAySq0AWFRWltm3bqlWrVlqyZMlt223ZskXNmze3W3EAAAAAUJjlOO19bGysZsyYocjISDk7Oys4OFj169dX5cqVM7W7dOmS3n333TwrFMYp7u4sR1cXu/aZnnJTfySl2rVPAAAAoKDJMZBFR0fL399fXl5ekqTWrVtr/fr1GjRoUKZ2I0aM0KBBgzRt2rQ8KRTGcXR10Y4One3a579Wr5QIZAAA3HeW1FT5+LjbtU/+0ArcuxwDWVxcnHx8fGzLvr6+OnToUKY2n332mapXr64nnnjC/hUCAADAbhycnflDK5CP5BjIrFZrlnUmk8n2/xMnTmjjxo369NNP9fvvv99TEd7exe5pP2SVmpFm97965ZWCUifsh9cc+Of4OUJ+xdgE7k2OgaxkyZLat2+fbTkuLk6+vr625fXr1ys+Pl6dO3dWWlqa4uLi1KNHDy1dujTXRVy+fE0WS9bgl58UlDcZZ7OTukUMsGufy7vPtWt/f4qPT8qTfh80BWVsSrzmD6KCND4LCn6O7IOxaX/5fWw6OJg4CYB8KcdZFhs2bKidO3cqISFBycnJ2rhxoxo3bmzbPnjwYG3YsEGrV6/W/Pnz5evre1dhDAAAAAAeVDkGspIlSyo0NFS9evVSUFCQAgICVKtWLYWEhOjw4cP3o0YAAAAAKJRyvGRRkgIDAxUYGJhp3YIFC7K0e/jhh7V582b7VAYAAAAAhVyuAhkA/FN5MeFMxs1UmV2c7donUzcDAID7iUAG4L7IqwlnmLoZAAAUZDl+hwwAAAAAkDcIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGCRXgSwqKkpt27ZVq1attGTJkizbv/32WwUGBqpdu3YaNmyYUlNT7V4oAAAAABQ2OQay2NhYzZgxQ0uXLtXq1asVERGhU6dO2bbfuHFDY8aM0SeffKJ169bp5s2bWrVqVZ4WDQAAAACFQY6BLDo6Wv7+/vLy8pKbm5tat26t9evX27a7ublp8+bNKlGihG7cuKHLly/Lw8MjT4sGAAAAgMIgx0AWFxcnHx8f27Kvr69iY2MztXFyctLWrVvVrFkz/fHHH3rmmWfsXykAAAAAFDKOOTWwWq1Z1plMpizrmjRpot27d2v69OkKDw/XtGnTcl2Et3exXLdF4eHj4250CUC2GJvIzxifyK8Ym8C9yTGQlSxZUvv27bMtx8XFydfX17Z85coVHTlyxHZWLDAwUKGhoXdVxOXL12SxZA1++QlvMvYXH59kdAmFAmPT/hib9sP4tD/Gp30wNu0vv49NBwcTJwGQL+V4yWLDhg21c+dOJSQkKDk5WRs3blTjxo1t261Wq958803FxMRIkr755hs9+eSTeVcxAAAAABQSuTpDFhoaql69eiktLU1dunRRrVq1FBISosGDB8vPz09jx47Vyy+/LJPJpMqVK2v06NH3o3YAAAAAKNByDGTSrcsQAwMDM61bsGCB7f8tW7ZUy5Yt7VsZAAAAABRyuboxNAAAAADA/ghkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGCQXAWyqKgotW3bVq1atdKSJUuybN+0aZM6dOig9u3ba+DAgUpMTLR7oQAAAABQ2OQYyGJjYzVjxgwtXbpUq1evVkREhE6dOmXbfu3aNYWHh2v+/Plas2aNqlatqlmzZuVp0QAAAABQGOQYyKKjo+Xv7y8vLy+5ubmpdevWWr9+vW17WlqawsPDVbJkSUlS1apVdfHixbyrGAAAAAAKiRwDWVxcnHx8fGzLvr6+io2NtS0XL15cLVu2lCSlpKRo/vz5tmUAAAAAwO055tTAarVmWWcymbKsS0pK0sCBA/X444+rY8eOd1WEt3exu2qPwsHHx93oEoBsMTaRnzE+kV8xNoF7k2MgK1mypPbt22dbjouLk6+vb6Y2cXFx6tu3r/z9/TV8+PC7LuLy5WuyWLIGv/yENxn7i49PMrqEQoGxaX+MTfthfNof49M+GJv2l9/HpoODiZMAyJdyvGSxYcOG2rlzpxISEpScnKyNGzeqcePGtu0ZGRnq37+/2rRpo7CwsGzPngEAAAAAssrVGbLQ0FD16tVLaWlp6tKli2rVqqWQkBANHjxYv//+u44dO6aMjAxt2LBBklSzZk2NHz8+z4sHAAAAgIIsx0AmSYGBgQoMDMy0bsGCBZIkPz8//fTTT/avDAAAAAAKuVzdGBoAAAAAYH8EMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCC5CmRRUVFq27atWrVqpSVLlty23dChQxUZGWm34gAAAACgMMsxkMXGxmrGjBlaunSpVq9erYiICJ06dSpLm/79+2v9+vV5VigAAAAAFDY5BrLo6Gj5+/vLy8tLbm5uat26dZbgFRUVpRYtWqhNmzZ5VigAAAAAFDaOOTWIi4uTj4+PbdnX11eHDh3K1Oall16SJO3fv9/O5QEAAABA4ZVjILNarVnWmUwmuxbh7V3Mrv2hYPDxcTe6BCBbjE3kZ4xP5FeMTeDe5BjISpYsqX379tmW4+Li5Ovra9ciLl++Josla/DLT3iTsb/4+CSjSygUGJv2x9i0H8an/TE+7YOxaX/5fWw6OJg4CYB8KcfvkDVs2FA7d+5UQkKCkpOTtXHjRjVu3Ph+1AYAAAAAhVqOgaxkyZIKDQ1Vr169FBQUpICAANWqVUshISE6fPjw/agRAAAAAAqlHC9ZlKTAwEAFBgZmWrdgwYIs7SZNmmSfqgAAAADgAZCrG0MDAAAAAOyPQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABgkV4EsKipKbdu2VatWrbRkyZIs248fP67OnTurdevWCgsLU3p6ut0LBQAAAIDCJsdAFhsbqxkzZmjp0qVavXq1IiIidOrUqUxt3nzzTb3zzjvasGGDrFarli9fnmcFAwAAAEBhkWMgi46Olr+/v7y8vOTm5qbWrVtr/fr1tu2//fabUlJSVLt2bUlSp06dMm0HAAAAAGTPMacGcXFx8vHxsS37+vrq0KFDt93u4+Oj2NjYuyrCwcF0V+2N4lu8iN37dPT0ybnRXfJxe8jufbr42r/OgvK6FwSMTftibNoX49O+GJ/2w9i0r/w+NvN7fXhwmaxWq/VODT788EMlJycrNDRUkvTll1/q8OHDGjNmjCTpwIEDmjJlir744gtJ0tmzZ/Xyyy9zlgwAAAAAcpDjJYslS5bUpUuXbMtxcXHy9fW97fb4+PhM2wEAAAAA2csxkDVs2FA7d+5UQkKCkpOTtXHjRjVu3Ni2vWzZsnJxcdH+/fslSV999VWm7QAAAACA7OV4yaJ0a9r7efPmKS0tTV26dFFISIhCQkI0ePBg+fn56aefftKIESN0/fp1Va9eXRMnTpSzs/P9qB8AAAAACqxcBTIAAAAAgP3l6sbQAAAAAAD7I5ABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQ3aMTJ06oatWq2rBhg23d1q1b1axZM73++utZ2nfo0CFP6hg8eLACAwPzpG9JOnz4sMLCwvKsf+Qf165d0+jRoxUQEKAOHTqoZ8+eOnr0qGH15NXPDAq/69eva/To0WrVqpXat2+vHj16aOfOnZKkiIgIrV27VpI0bNgwRUZGGlkqCpELFy6oefPmWdZXrVr1tvvs3r1bPXv2zMuyABQAjkYXUFBFRkaqdevWWrZsmVq3bi1JWr9+vfr376/u3btnab969Wq71/DHH3/o2LFjKlGihPbv36+6deva/Rh+fn7y8/Oze7/IXywWi0JCQlS/fn199dVXcnR01K5duxQSEqJ169apePHi972mvPiZQeFntVrVv39/VatWTevWrZOzs7OOHTumfv36adq0afrxxx/19NNPG10mAAA2BLJ7kJ6erjVr1mjJkiUKDg7WuXPntHv3bn333XfauXOnHBwctGbNGnl6eurkyZN67733FBQUpJ9//llXrlxRWFiYzpw5I2dnZw0bNkwNGjTQ4sWLtXr1aiUnJ8tkMum9995TpUqV1Lx5c7Vv314//PCDkpOT9e6776pmzZqSbt2wu169enrssccUERFhC2SRkZHasmWL4uLi9Pvvv+uFF15QTEyMdu3aJS8vLy1cuFAuLi766quvtGjRIlksFtWoUUOjRo2Si4uL/P39VaNGDV26dElvvfWWPvzwQ33++ec6fvy4Ro4cqZSUFHl6emrq1KkqUaKEwsPDdfLkSV26dEmPPvqoZs+erUuXLmnQoEGqUqWKjh8/Lm9vb73//vvy8vIy8JXD7ezevVtxcXEaPHiwHBxunTj39/fXxIkTZbFYNGLEiGxf4169emnz5s2SpFmzZkmSXnnlFUVFRWnu3LkymUzy8/PT2LFjlZCQoOHDhyspKUnx8fFq166d3njjDf30008aOXKk0tPT5eLiookTJ+qRRx5R1apV9fPPPys2Njbb/SIjI7V9+3YlJibq/Pnz+te//qXw8HCjnkLkE3v27FFMTIw+++wzmUwmSVL16tU1YMAA9e7dW+7u7tq1a5d8fHwkSVu2bNHSpUt1+fJl2x/Url+/rjFjxujkyZPKyMhQSEiIAgICFBkZqVWrVunKlStq1qyZhgwZYuRDRQGSm/erRYsWadOmTZo/f7769esnPz8/7d+/XwkJCRoxYoSaNGmiS5cuKSwsTDExMXJ0dFRoaKhq1qxp+5wgSY0aNdLbb7+ttm3bav78+TKZTEpJSVFsbKzOnj2r3377TV27dtWAAQMMeCYAZIdLFu/Bli1bVKZMGT366KNq2bKlli1bpq5du6p58+YaPHiwunbtKkm2SxqrVatm2/f9999X+fLl9c0332jy5Ml67733dO3aNW3atEmff/651q5dq5YtW2rp0qW2fby8vLRixQoFBwdr3rx5tvWRkZFq06aN2rRpow0bNujKlSu2bYcPH9bChQu1ZMkSTZo0SY0bN1ZUVJQkafv27Tp58qSWL1+uZcuWafXq1fL29tZHH30k6daZt379+mn16tVydPy/zP7GG29o4MCBioqKUtu2bbVo0SL9+OOPcnJyUkREhL799lvdvHlTW7dulST99NNP6tOnj9auXSsPDw/b8ZH/HDt2TH5+frYw9qcmTZrozJkzt32NsxMbG6uJEyfq448/1rp165SRkaGtW7dq7dq1CggI0PLly7VmzRotXbpUCQkJWrRokfr06aPIyEj17NlT//vf/zL1d7v9JOnHH3/UzJkztWbNGn3//ff6+eef7f7coGA5fPiwatasaQtjf3rqqadUpEgR2/t0o0aNJEmpqan68ssvNW/ePM2YMUOSNHfuXNWoUUORkZFasmSJPvzwQ50/f17SrfG9atUqwhju2p3er1auXKmNGzdq3rx5KlKkiCQpLS1NERERevvtt/X+++9LksaOHSt/f39FRUVp5syZGj58uCwWi8qUKaMTJ07o9OnTysjI0J49eyRJ27ZtU7NmzSRJP//8sz766CN9+eWXmj9/vq5evXqfnwEAt8MZsnsQGRmpgIAASVLbtm31xhtv6LXXXsvSrlatWlnW7d27V1OnTpV0K7BFRERIkqZNm6Z169bp119/1fbt2zOFuD8/OFSpUkUbN26UJB0/flwXL15Uw4YN5eTkpGrVqumrr75S7969JUlPPvmkihUrpmLFikmSGjRoIEkqW7asrl69qt27d+vs2bPq1q2bpFtv/NWrV7cd84knnshUd0JCguLj421v7D169LBt8/Ly0pIlS3TmzBn9+uuvunHjhiTJ29vb1meVKlWUmJh45ycWhnFwcJDVas1221NPPXXb1zg7P/74o5588kmVKlVKkjRlyhTbtl27dumjjz7SyZMnlZaWpuTkZDVp0kRjxozR9u3b1axZM9slwH/q27dvtvtJUp06dWxjvFy5cowxyGQyKSMjI8v6tLS0bNu3aNFCJpNJVapU0R9//CFJio6OVkpKilauXClJunHjhk6ePCnp1tm2v/6hCvjT3/+gJd26hPbPPw7c7v3qxIkTGjlypKZPny43Nzfbvn/93f/nH1x37dqlcePG2fp44okndPDgQTVp0kQ7d+6Uo6OjevXqpXXr1tmuKqhcubIkqX79+nJ2dpa3t7e8vLyUlJQkDw+PvHkyANwVfqvcpcuXL2vbtm06cuSIPvvsM1mtVl29etUWlP7K1dU1y7q//yI/ffq0XF1d9cILL+j5559X48aNVaJECR0/ftzWxsXFRZIy/cV35cqVSk1NtX14vX79upYtW2YLZE5OTnc8bkZGhtq0aaMRI0bY9v/rh5i/1/73/m7evKm4uDidOHFCM2fOVK9evdSpUyf98ccftg/2f9b9Z+23+8AP49WsWVNLly7N9OFBkqZPn65atWpp1qxZWV7jv7+m6enpcnR0zDLW/jybNX/+fJ0/f14BAQFq2bKloqOjZbVa9eyzz6pOnTr6/vvvtWjRIm3dutX2gUOSJk2alO1+EmMMWT3xxBP6/PPPlZaWlul963//+1+234c1m82SMr+/WiwWTZkyRTVq1JAkXbp0SZ6enoqKisr2fR2QJA8PDyUlJWVad/nyZXl6ekq6/ftV0aJFNWHCBE2YMEGNGjWyhbLsfvf//T3OarUqIyNDTZo00ezZs+Xs7KxXX31V33zzjaKiomyh7k7HB2A8Llm8S2vWrJG/v7+2bdumzZs36/vvv1f//v1tZ7pyUq9ePX399deSboWxkJAQHTlyRBUqVFDv3r31xBNPaNu2bdn+hfdPqampioqK0qeffqrNmzdr8+bN+u677xQfH6/du3fnqo769evr22+/1eXLl2W1WhUeHq5Fixbdtr27u7tKlSqlHTt2SLo14cL777+vnTt3qk2bNurcubNKlCihvXv33rF25E/16tWTt7e3Zs+ebXv9tm/fbvveQ3avsYeHhxITE5WQkKDU1FRt375d0q2JYA4ePKj4+HhJ0oQJE/Tdd99px44d6tu3r9q0aaOLFy8qNjZWFotFr732mg4dOqTg4GC9+uqrOnbsWKbabrcfkJ169eqpcuXKmjBhgu2s2JEjRzR37lwNHDhQZrM5x/cof39/ffHFF5KkuLg4tW/fXhcvXszz2lGwFStWTBUqVMg0+3JERITtCpXbKVu2rFq0aKGnn35aM2fOvGNbf39/rVixQpJ0/vx5HThwQLVr11aNGjX0yy+/6Ndff1WlSpVUv359zZ07V02bNv3HjwtA3uMM2V2KjIxUaGhopnU9evTQwoULVaJEiRz3Hzx4sEaMGKH27dvL0dFRkydPVrVq1bRs2TK1bdtWzs7OqlWrlu3ymOx8//33Klu2bKbLCosVK6auXbtq2bJlmf4idjuPP/64Bg0apBdeeEEWi0XVqlVTv3797rjPlClTFB4ersmTJ6t48eKaPHmy/vjjD73xxhtav369nJ2dVbt2bV24cCHH4yN/MZlM+uCDDzRx4kQFBATI0dFRxYsX1/z582U2m7N9jd3d3dW3b1916dJFpUqVsp19KFmypMLCwtS3b19ZLBbVrl1bnTp1UpEiRfTWW2/Jw8ND3t7eqlmzpi5cuKD+/fsrLCxMH3zwgcxms4YNG5aptpdffjnb/YDbmT17tmbMmKGAgACZzWZ5enpqypQpql+/vi5duqTp06fL3d39tvsPGjRI4eHhCggIUEZGht58802VL19e+/btu4+PAgXRn78n58yZo7S0NFWtWlUjR47Uli1bctz3rbfeUkBAwB1vZRMWFqaRI0fabtcwbtw4+fr6SpLq1q1ru5zb399fX375JTOKAgWEyco5awAAAAAwBJcsAgAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIQbQwNAPrF7926NHz9ebm5uun79up588kkdO3ZM169fl9Vq1bhx41S3bl1dv35d48aN04EDB2Q2m9WyZUuFhoYqLS1NU6dO1d69e5WRkaHq1atrxIgRKlasmNEPDQAA3AaBDADykZMnT2rTpk2Ki4vTJ598ooiICDk4OGj+/PlasGCB6tatq5kzZ+rmzZv6+uuvlZGRoRdffFF79uzR3r17ZTabFRkZKZPJpOnTp2vq1KkKDw83+mEBAIDbIJABQD5SunRplS1bVmXLlpWnp6eWLVum8+fPa/fu3SpatKgkKTo6Wm+//bbMZrPMZrMWL14sSZoyZYqSkpIUHR0tSUpLS5O3t7dhjwUAAOSMQAYA+Yibm5skacuWLRo/frz69OmjFi1aqGLFilqzZo0kydHRUSaTybbPxYsX5erqKovFouHDh6tJkyaSpOvXr+vmzZv3/0EAAIBcY1IPAMiHduzYoWbNmqlHjx7y8/PTpk2blJGRIUlq0KCBVq1aJYvFotTUVA0ePFh79+7VM888oyVLlig1NVUWi0XvvPOOpk+fbvAjAQAAd0IgA4B8KDg4WHv37lVgYKC6d++ucuXK6cKFC7JYLBo0aJCcnJzUoUMHBQUFqUmTJvr3v/+tgQMHqmzZsurYsaPatm0rq9WqYcOGGf1QAADAHZisVqvV6CIAAAAA4EHEGTIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCD/D0WoO4UuvvAVAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 297 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "kETWDtN68Gmq" - }, - "outputs": [], - "source": [ - "model_card.quantitative_analysis.graphics.description = (\n", - " f\"These graphs show the models performance on the test dataset for disagregated racial categories.\"\n", - ")\n", - "model_card.quantitative_analysis.performance_metrics = metricframe_to_dictionary(metricframe_postprocess, \"false_negative_rate\")\n", - "model_card.quantitative_analysis.graphics.collection = [\n", - " {\"name\": \"ThresholdOptimizer\", \"image\": postprocess_performance}\n", - "]" - ] + "id": "SweXBa-vEWFM", + "outputId": "a230ed5d-2665-477d-adba-287c76020032" + } + }, + { + "cell_type": "code", + "execution_count": 91, + "source": [ + "plot_technique_comparison(test_dict, \"balanced_accuracy\")" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAAGECAYAAABZM+NjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABFDUlEQVR4nO3dd0CVdf//8dfhAG5AETRNTc2cuAda7q3gykzNWWHqbXZrORJDHJlbU3Nbt6koZqSiOTJ3uC33zHKEAYriAGSc8/vDn+cbgYJ17EJ6Pv7yuq7P9bne1zkfD+d1rmWyWq1WAQAAAAD+cQ5GFwAAAAAA/1YEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDMC/xv79++Xj4/NE65QuXVrR0dFPqaL0LV68WMOHDzds+wAA4OkikAEAAACAQRyNLgAA/kmxsbEaOHCgLl26JBcXF40ZM0aSNGbMGMXGxioyMlJlypTRjBkzlC1bthTrBQYG6tdff1VMTIxy5cqlKVOmqESJEurevbsqV66sI0eO6Nq1a6pWrZomTpwoBwcHbd++XTNmzJDFYlHOnDk1evRolSlTRkeOHNGUKVMUFxcnk8mkd999Vw0bNlRiYqLGjRunsLAwubu7y93dXXny5El3v+bNm6etW7fq/v37iouL07Bhw9S0aVMlJSVp8uTJ2rFjh8xms6pUqaJRo0bJwcEhzfnz58/XzZs3FRAQIEmaNWuWbbp79+5ydXXVxYsX1aVLF3l5eWny5MlKSEhQVFSU6tSpo/Hjx0tSmvu9fft2XbhwQVOnTpUkHT58WGPHjtWaNWvs/C4DAPDsIJAB+Fe5du2apkyZoqpVqyo4OFhDhw5V9erV1a5dO7Vt21aJiYnq0KGDduzYoebNm9vW27Vrl1xcXLRq1SpJUkBAgJYvX66PPvpIknT58mUtXbpUsbGxatmypQ4cOKAXX3xRQ4YM0dKlS1W2bFlt2bJFU6ZM0dSpU/Xhhx9q8eLFev755xUREaFOnTqpdOnS+u677/Trr79qw4YNSkpKUrdu3dINZL/99pvCwsK0bNkyZc+eXRs2bNDMmTPVtGlTBQUF6eTJk1q7dq2cnZ01ePBgffvtt7p9+3aa89Pj4uJiazd48GANHDhQtWrV0r1799S4cWOdOHFCBQsWTHO/J06cqGbNmunWrVtyc3NTcHCwOnfu/FffSgAAsgQCGYB/ldKlS6tq1aqSpPbt2yswMFCff/65fvrpJy1cuFC//vqrIiMjFRsbm2K9Fi1aqEiRIlq6dKkuXbqkAwcOqEqVKrblDRs2lIODg3Lnzq1ixYopJiZGR44cUalSpVS2bFlJUrNmzdSsWTPt3LlTUVFR+s9//mNb32Qy6ezZs9q7d698fHzk7OwsZ2dn+fr66uzZs4/dp8KFC2vixIkKDQ3VpUuXdPToUd27d0+SFBYWprZt2yp79uySpBkzZkiS+vbtm+b8WbNmPXZb1atXt/17woQJ2rVrl+bNm6eLFy8qPj5esbGxj9xvSWrQoIHWrl2rdu3aac+ePRo1atRjtwcAQFZHIAPwr+LgkPLSWZPJJH9/f1mtVrVs2VINGjTQtWvXZLVaU7QLCgrSqlWr9MYbb8jX11dubm66evWqbfnDYPOwT6vVKkdHR5lMJtt8q9Wqs2fPKjk5WSVLltRXX31lWxYREaF8+fIpODg4xXbNZnO6+3Ty5En1799fvXr10ssvv6waNWpo9OjRkiRHx5Qf89evX5fFYnnk/Ie1P5SYmJiiXc6cOW3/fuONN1SmTBnVrVtXLVu21NGjR2W1WmU2m9Pc7zJlyuiNN95QYGCgHB0d1axZM+XKlSvd/QMAICvjph4A/lXOnj2r06dPS5KCg4NVrVo1hYWF6T//+Y9atWolk8mko0ePKjk5OcV6e/bsUfv27fXaa6+pePHi2rZtW6o2f1apUiX9/PPPOn/+vCTp+++/15AhQ1S5cmVdunRJBw8elCSdPn1azZs3V2RkpOrWras1a9bo/v37un//foZOIzx48KAqVKig3r17q2bNmvr+++9ttdWuXVvr169XQkKCLBaLAgMDtWHDhkfOz5s3r06ePCmr1arY2Fjt2bMnzW3GxMToxIkT+uCDD9SsWTNFRETo8uXLslgsj9xvSapataocHBy0ePFidenSJd19AwAgq+MIGYB/lRIlSmj27Nm6cuWK3N3dNWHCBO3YsUP/+c9/5Orqqhw5cqhGjRq6fPlyivXefPNNBQQEKCQkRGazWeXLl9e5c+ceu638+fNrypQpGjZsmJKTk5U7d25Nnz5d+fLl08yZMzVp0iTdv39fVqtVkyZNUuHChdW5c2ddvnxZPj4+cnNzU7FixdLdJx8fH23ZskWtWrWSk5OTateurZiYGN29e1edO3fWb7/9pg4dOshqtapmzZrq3r27TCZTmvPj4uK0e/duNWvWTAUKFFCVKlVSHS2UJFdXV/Xp00ft27eXm5ub8ubNq6pVq+rSpUuqXbt2mvv9UIcOHfTtt9+qdOnSGXzXAADIukzWtP7SAgDwFCQlJWnAgAFq06aNWrVqZXQ5AAAYjiNkAPAMGD9+vPbv35/msg8//FDe3t7/cEVP7sKFC+rSpYvq1aunFi1aGF0OAACZAkfIAAAAAMAg3NQDAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMEimuMvizZv3ZLFwbxF7cHfPrRs37hpdBpAKYxOZGeMTmRVj034cHEzKmzeX0WUAqWSKQGaxWAlkdsRricyKsYnMjPGJzIqxCWRtnLIIAAAAAAYhkAEAAACAQQhkAAAAAGCQTHENGQAAAJDZJCYm6sqVK4qLize6FDzDcuTIriJFisjJySnN5QQyAAAAIA1XrlyRo2M2Pfecp0wmk9Hl4BlktVp1506Mrly5ohIlSqTZhlMWAQAAgDTExcUrd25Xwhj+MpPJpDx5XB97lJVABgAAADwCYQx/V3pjiFMWAQAAgAzIkTObsmez/9fn+PtJiou9n2678PBwderUTsWLPzj1zWKx6N69e2rd2kd+fv0ytK2FC+dJkvz8+j5RjeHhv+mLLxbJ33+UTp8+pZCQ1fL3D3iiPpC2DI2o0NBQzZ07V4mJierVq5feeOONFMsvXryoUaNGKSYmRh4eHpo2bZpcXV2fSsEAAACAEbJnc5Tv+2vt3m/o1LYZCmSSlD+/h5YuXWmbjoqK0muvtVWTJs1tQe1puHbtmq5evSpJKlu2HGHMjtI9ZTEiIkLTp09XUFCQ1q5dq+DgYF24cMG23Gq1ql+/fvLz89O6detUtmxZLViw4KkWDQAAAEC6cSNKVquUK1cuffnlF+rRo6u6dXtds2bNkNVqlSQtW7ZEHTu21dtv99SpUydt63p7V7X9e/36dRozZpQk6cCB/erW7XW98UYnvf/+QN27d1fTpk3SmTOnNHnyJzp8+JD69fOTJF2+fEn9+vnpjTc6peh/zJhRmjZtsvr0eVPt2rXW+vUPguzBg/vVs2dX9er1ht59t59u3br5j7xOmVm6R8jCwsLk7e0tNzc3SVLz5s21adMmDRgwQJJ08uRJ5cyZU/Xq1ZMk9e3bV7dv3356FQMAAAD/UtevR6l7985KSEjQrVu3VLZsOU2cOEU//3xBZ86c1hdfLJXJZFJg4EfatOlbvfBCcYWGrtWXX66QyWTS22/3VLly5R/Zf0JCgkaN8tenn36ml14qrblzZ2nDhvUaPHioFi2aryFDPtThw4ds7QMDR6p7915q2LCxTpw4phEjhmjVqjWSpIiI3zV//mL9/PMF9e/fRz4+bfXFF4s1bJi/ypUrr+DgFTp79oxq1ar9tF+2TC3dQBYZGSkPDw/btKenp44dO2abvnz5svLnz69hw4bp1KlTeumll/TRRx89nWoBAACAf7GHpyxaLBbNnDlNFy6cV/XqNTRnzmydPHlCvXo9uLTo/v37KliwoKKjb6hOnVeUM2dOSVLjxk2VnJz8yP5//vmCPDw89dJLpSVJ/fq9K0kpQthDsbGxunr1iho2bCxJqlCholxcXHXp0q+SpFq1astkMqlkyRd1+3aMJKlu3XoaNux91a/fQHXrNlCtWt72eWGeYekGsoeHOv/oj3cKSUpK0oEDB7Rs2TJ5eXlpxowZmjBhgiZMmJDhItzdc2e4LdLn4ZHH6BKANDE2kZkxPpFZMTaRFgcHBw0Y8F/16NFFy5cvlcWSrNdf76quXbtJku7cuSOz2axvvvlaVqvFtp7ZbE4RyKxWq0wmk5KSkiRJjo4p48Hdu3cUGxubZg1WqyVVVrBarbb+nZ2dJaXMDl26dNMrr9TTDz/s1meffapTp06od++3/+rLkCWkG8gKFCigQ4f+LxFHRkbK09PTNu3h4aFixYrJy8tLkuTj46OBAwc+URE3btyVxZI6+GV1eV2d5eicza59JiQlyNnR2a59Jt9PkDmbfftMir+vm3cS7NonMjcPjzyKirpjdBlAmhifyKwYm/bj4GDKcgcBHB0d9e67/5W//zANGTJcK1YsV7t2HeTs7KyhQwerdWtf1ahRUyNGDNXbb78jJydn7dy5XbVrvyxJcnNz08WLP6tEiZLavXunXF3dVLRoMd26dVO//HJRxYuX0NKlS2QymeTtXSfVkbVcuXKrcOHntX3797ZTFm/cuKGSJV98ZM1vvtlDw4aNUOfOb8jFxVW7du14mi/RMyHdQFanTh3NmjVL0dHRypEjh7Zs2aKxY8fallepUkXR0dE6c+aMypQpo23btql8+Uefl/qsyuOS46nc5vTix6/atb8S/l+rU3DGbnuaUaten6sf2tq3zpfXfi0RyAAAAP6W2rVfVvnyXjpy5IgaNGist97qIYvFIm/v2mrd2lcmk0mdO3dV797dlSePiwoWfM62bv/+A/X+++/J3d1dlSpV0a1bt5QtWzYFBo7T6NEfKTExUc8/X0SjRo1VYmKC7t69o8DAkfL1bWfrIzBwnCZOHK9Fi+bLyclJEyZMkZOT0yPr7dfvPxo7dpTMZrOyZcuuYcNGPM2X55lgsqZ1TuKfhIaGav78+UpMTFTHjh3l5+cnPz8/DRw4UF5eXjp69KjGjh2ruLg4FSxYUJMmTZK7u3uGi3gWjpB5eOSx+21OQ6e2/VcHMn7xy7yextFbS1KCbtzM2C19gX8aRyGQWTE27eevHCE7efKUChUqZps2+jlkeHaFh19S+fLl0lyWoRHl6+srX1/fFPMWLlxo+3elSpW0evXqv1EigMzE0TnbU/mxQOKPDQDg2RUXe5/gBLuzf8QHgDQkJCfa/cJ0rm8EAADPOgIZ8Ix7Wtc32puz2emZOZ2W6xthD3nzOMsxu31P/eUHAwDIejL/tzgAj5U9m+NTub4RwN/jmD0bPxgAANJFIAMAPFOe1k1nAAAwAoEMAPBM4aYzAICshEAGAHhqnpVrHAEAMAp/JQEATw3XOALISlxyOcps51OmJSk54b5u30tKt114eLj69/fTmjUbUsz39q6qffuO/K0aunfvrKVLV+rkyRPavv17DRjwnnbt2qkzZ06pT58nvynX7t07deXKFXXt2i3D6/Tr56e3335H1apVf+LtPcsIZAAAAEAGmJ/CKdPS/z9tOgOB7GlaunSlJOmXXy4qOjpaklSvXn3Vq1f/L/V35sxpu9WW1RHIAAD/ek/jOXkA8E9av36d9u0L0+3btxUeflU1a9bW0KEf6vDhQ/rf/xbLarXqt9+uqlGjxsqVK4927douq9WqadNmyd3dXd7eVfXddzu1cOE8xcXF6osvFsnDw1NHjhxWQMBoHT58SNOmTZLZbFaFChX1yy8XNXfuQh05cljz5n2m+Ph43blzWwMGvKcSJUrqm2++liQ999xzatSoqaZMmaCff74gi8Wi7t17qVmzFkpISND48WN0+vQpPfdcIcXE3DL2RTQIgQwA8K/3tJ6TBwD/pOPHjyko6CuZzQ7q1KmDOnToKEk6efKEVqz4Sq6urmrZsokGDhyk//1vucaOHaWtWzfr9de7SpLy5MkjP7++OnLksHr3flvr16+TJCUlJWr06I80deqnKlXqJU2bNtm2za++WqkRIz7SCy8U16FDBzR9+hQtX75K7ds/OJLo49NWn302U6VLl1VAwBjdu3dXfn69Vb58Be3YsU2SFBwcosuXL6t799f/yZcr0yCQAQAA/Ivw0PJnl4ODKdU8q9Uqk+nBfC+visqVK5ckqXDhwrp9+7YkqWTJkipQoKAkydXVTdWr15QkFSz4nK3N41y4cEF58+ZVqVIvSZJ8fdvaQllg4Dj98MNubdu2VSdOHFdsbGyq9Q8e3K/4+HitX//gmuK4uDhdvPizjhw5pHbtHgS3okWLysurYsZfjCyEQAYAAPAvwkPLn1158rjo3r27KeZFR0crTx4XSZJzihuOmGS1WiVJjo5OKdYxm58sApjNZltff9a371uqWrW6qlWrrurVayogYESqNhaLRYGB41SmTFlJ0o0bN+Tq6qI1a0JS9Gs2m5+orqzCwegCAAAAAKQvV65cKlKkqLZt+942b+3aENWoUdNu2zCbzUpOTnmDkRdeKK7bt2/rwoXzkqTNmzfKZDIpJiZGly9fVp8+/VSnzivav3+vLBbL/+/HUcnJyZKkatVqKCRktSTp+vUodev2un7//XfVrFlLmzdvlMVi0bVr4Tp+/Jjd9uNZwhEyAAAA4BkRGDhOkyaN1+efL1BiYqJefLGUhgwZrj17dtul//LlK2jx4vn67LOZKlbsBUmSk5OTAgPHacyYAJlMJhUr9oKyZcsmV1dXtWnTTl27vqZcuXKpQoWKio+PV1xcnKpUqaIxY0YpXz53vf12H02a9Im6dn1NycnJGjDgPT3/fBG9+upr+vnnn9W586sqWPA5lShR0i778KwhkAEAAAAZkJxw/8Et6p9CvxlVtGgxzZ49P9V8H5828vFpY5ueO3eh7d9/fK7XH59h5ufX1/bvh88xK1q0mFavXpeiX4vFot27d2r+/M+VI0cOBQUtU1RUpCTpvfcG6733BtvaDx36oSSpSpVq+uab9bb5o0d/nKpmR0cnjRjxUQb2OmsjkAEAAAAZcPtekuHPCzOCg4ODXF1d1bt3Nzk5Oem55wrJ3z/A6LKyDAIZAAAAgMfq0aO3evTobXQZWRKBDAAAIJPioeVA1kcgAwAAyKR4aDmQ9XHbewAAAAAwCIEMAAAAAAzCKYsAAABABuTM7aRsTs527/d+YoJi7yam2y48PFydOrVT8eIlZDKZlJiYqPz5PfTRR4Hy9CyQ4e2Fh/+mL75YJH//UX+n7Kdu0KB3NWJEgDw8PIwu5akikAEAAAAZkM3J2e7X9EkPruuLVfqBTJLy5/fQ0qUrbdNz5szS1KmTNHHi1Axv79q1a7p69eoT1/lPmz59ltEl/CMIZAAAAMAzqnLlqtq9e6dOnDimadOmKCHhvtzc3DRsmL+KFCmqoKBl+vbbUDk4OKhcufIaPnykpk2bpPDw3zR58idq1KipFi2aJ7PZUZGRESpXrrxGjAjQ9evXNWjQALm6usnZ2VkzZ87R9OlTdOjQAZlMJrVo0Vo9evSS1WrVZ5/N1M6d22U2m9W+/at6/fWuunLlsiZN+kQxMTHKnj273n9/qEqXLqPNmzdq2bIlcnBwUKFChRUYOE4xMbc0apS/4uLi5ODgoMGDh6hChYpq16615sxZqCNHDmnfvjDdvn1b4eFXVbNmbdsDqOfMmaVt27bKzc1N7u75Vbdu/RQPyH4WEMgAAMC/Th6XHMqezb5fg+LvJ+nO7Ti79gk8TlJSorZu3aLy5Sto5MgPNX78JJUrV17ff/+dAgJGaOHC/+nLLz/X+vWb5eBg1uTJExQZGanBg4dq0aL5GjLkQx0+fEinTp3Ul1+uUNGixeTvP0yrV69SgwaNdOnSrwoJWa9ChQpp9epVioyM0LJlwUpMTFD//n1UsmRJxcfH69ixo1q+fJWSkpL0zjtvqkmTZhozZpQ++GCYSpcuo19+uahhw97XqlXfaP78OVq0aIny5cunefM+06VLv2rXrh16+eW66tatpw4fPqSjR39ShQoVU+zr8ePHFBT0lcxmB3Xq1EEdOnTUtWvhOnr0R61YsVpxcXHq2bOr6tatb9C78dcRyAAAwL9O9myO8n1/rV37DJ3aVnfs2iOQ2vXrUerevbMkKSEhQeXLV1Dr1m107txZlStXXpLUuHFTTZgwTvHxcfLyqqTevburXr366tixkzw9PXXlyuUUfVauXEXFir0gSWrRorXWrv1aDRo0Ut68+VSoUCFJ0uHDB9W6ta/MZrPM5hxq3rylDh48oKSkRDVu3FTOzs5ydnbW0qUrFRsbq9OnT2rcuEDbNuLiYhUTc0uvvFJPffr0Vv36DdSwYWO99FJpxcXFafjwD3Tu3FnVqfOKOnZ8PdV+e3lVVK5cuSRJhQsX1u3bt3XgwH41btxMTk5OcnJyUv36Dez8av8zCGQAAADAM+LP15BJ0vnz51K1s1qtSk62aNKkaTpx4pj27g3Tf/87QKNHf5yqrdns+If1LLbpbNmy2eZbLJY0+k+Wo2PKOBEeHi4Xlzxyds6Wos7IyAi5uLhq8OAh8vVtq7CwPQoMHKm33npHLVu21sqVq7Vnz25t3bpFGzaEataslM/Lc3bO9ocpk6xWq8xmB1mtKet6FnHbewAAAOAZVqzYC4qJidGpUyclSVu3blHBgs/JYrGoc+dXVbJkKfXp00+1annrwoXzMpvNSk5Otq1/9OhPioyMlMVi0bffrlft2nVSbaN69RrasCFUycnJio+P0+bNG1WtWnVVrlxVO3ZsU1JSouLj4zRo0ABFR0erSJEi2rhxgyRp//596tv3bSUnJ6ljx7Zyc8urnj3fVMuWPjp37qxmzZqhjRs3qHVrX33wwTCdPXsmQ/tds6a3tm//XomJibp376727Nktk8lkh1f0n8URMgAAADuwJCXIwyOP0WXgX8jZ2Vnjxn2iKVMmKj4+Ti4uLho3boLy5s2rdu06qHfvbsqePbsKFnxOrVv7KikpUXfv3lFg4Ej5+rZT/vz5NWbMR4qKilKNGrXUpk17RUREpNhG+/av6vLly+revbOSkpLUokUrNWjQSJJ0+vQp9ez5hiwWizp16qKiRYtp9OiPNXHieC1btkROTk4aN26CHB2d1KdPP737bl9lz55duXPnUUDAGFmtFgUEjNCGDQ9uPvLwhh3pqVPnFR07dlQ9enSRi4uLPDw8UhzVe1YQyAAAAOzAwdFZFz9+1a59lvD/2q794e+5n5igVa/PTb/hX+g3IwoVKqQ1azakuczLq5I+//zLVPO7dOmmLl26pZofFPSVJOnw4UPKl89ds2fPf+y2HB2d9P77Q9Pcdr9+A9Sv34AU8154objmzl2Yqm2zZi3UrFmLVPPnz/881byH2y9UqE2KOyc+7Pf48aMqUqSoVqxYraSkRL39di/btXDPEgIZAAAAkAGxdxMz/LwwPH1Fi76gxYsXaMWKZbJarWrVykelSr1kdFlPjEAGAAAA/EtVq1Zd1apVN7qMv8TV1VUzZnxmdBl/Gzf1AAAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAg3CXRQAAACAD8uRwlGN2+z94OCn+vu7EJT22TXh4uDp1aqfixUvIZDIpMTFR+fN76KOPAuXpWeCJthce/pu++GKR/P1H/Z2yn7pBg97ViBEB8vDw+Nt9jRkToH793tX+/Xs1c+Y0FShQUFarVQkJCapbt5769x8os9ms9evX2Zb/0bBh/tq4cb2OHTuqxMREXb16RcWLl5Akvf56F+XOnUdRUZF67bXOT1wbgQwAAADIAMfs2fRDW/s+/FuSXl77tZROIJOk/Pk9tHTpStv0nDmzNHXqJE2cOPWJtnft2jVdvXr1iev8p02fPssu/ezZs0v583vYgt0rr9RXQMBoSVJsbKyGDh2sRYvm6513+qda/kcVKnhJehCO+/f3S/FeSNJ//tNHjRs3U758+Z6oPgIZAAAA8AyqXLmqdu/eKUk6ceKYpk2booSE+3Jzc9OwYf4qUqSogoKW6dtvQ+Xg4KBy5cpr+PCRmjZtksLDf9PkyZ+oUaOmWrRonsxmR0VGRqhcufIaMSJA169f16BBA+Tq6iZnZ2fNnDlH06dP0aFDB2QymdSiRWv16NFLVqtVn302Uzt3bpfZbFb79q/q9de76sqVy5o06RPFxMQoe/bsev/9oSpduow2b96oZcuWyMHBQYUKFVZg4DjFxNzSqFH+iouLk4ODgwYPHqIKFSqqXbvWmjNnoY4cOaR9+8J0+/ZthYdfVc2atTV06IeSHoTSbdu2ys3NTe7u+VW3bn35+LRJ8TotW7ZEw4ePTPM1zJkzp/r1G6DBgweqT59+f+v9aNCgkb7+Olh+fk/WD4EMAAAAeMYkJSVq69YtqlixkhITEzVy5IcaP36SypUrr++//04BASO0cOH/9OWXn2v9+s1ycDBr8uQJioyM1ODBQ7Vo0XwNGfKhDh8+pFOnTurLL1eoaNFi8vcfptWrV6lBg0a6dOlXhYSsV6FChbR69SpFRkZo2bJgJSYmqH//PipZsqTi4+N17NhRLV++SklJSXrnnTfVpEkzjRkzSh98MEylS5fRL79c1LBh72vVqm80f/4cLVq0RPny5dO8eZ/p0qVftWvXDr38cl1169ZThw8f0tGjP6lChYop9vf48WMKCvpKZrODOnXqoA4dOuratXAdPfqjVqxYrbi4OPXs2VV169ZPsV5MTIwuX76sF14o/sjXsmTJkoqJuaWbN29Kkvbs2anu3f/v1EMnJ2d9/vmX6b4nlStX1dixowhkAAAAQFZ0/XqULSgkJCSofPkK6t9/oC5fviQXFxeVK1dektS4cVNNmDBO8fFx8vKqpN69u6tevfrq2LGTPD09deXK5RT9Vq5cRcWKvSBJatGitdau/VoNGjRS3rz5VKhQIUnS4cMH1bq1r8xms8zmHGrevKUOHjygpKRENW7cVM7OznJ2dtbSpSsVGxur06dPaty4QNs24uJiFRNzS6+8Uk99+vRW/foN1LBhY730UmnFxcVp+PAPdO7cWdWp84o6dnw91b57eVVUrly5JEmFCxfW7du3deDAfjVu3ExOTk5ycnJS/foNUq33229XlT9/etegmSRJ2bI9uD7wUacspue5555L9dpmBIEMAAAAeAb8+RqyhyIjI1LNs1qtSk62aNKkaTpx4pj27g3Tf/87QKNHf5yqrdns+If1LLbphwFFkiwWSxr9J8vRMWWcCA8Pl4tLHjk7Z0tRa2RkhFxcXDV48BD5+rZVWNgeBQaO1FtvvaOWLVtr5crV2rNnt7Zu3aING0I1a9bcFP06O//xZiomWa1Wmc0OslpT1vVnDg4mOTqaH9vmwoXz8vQsYAt8f5Wjo6NMpie/iX2G1ggNDVWrVq3UtGlTLV++PNXy2bNnq2HDhmrbtq3atm2bZhsAAAAA9les2AuKiYnRqVMnJUlbt25RwYLPyWKxqHPnV1WyZCn16dNPtWp568KF8zKbzUpOTratf/ToT4qMjJTFYtG3365X7dp1Um2jevUa2rAhVMnJyYqPj9PmzRtVrVp1Va5cVTt2bFNSUqLi4+M0aNAARUdHq0iRItq4cYMkaf/+ferb920lJyepY8e2cnPLq54931TLlj46d+6sZs2aoY0bN6h1a1998MEwnT17JkP7XbOmt7Zv/16JiYm6d++u9uzZLZPJlKLNc88VTjOwPnT37h0tWDBXHTt2ytA2Hyc8PFzPP1/kiddL9whZRESEpk+frpCQEDk7O6tz586qVauWXnzxRVubEydOaNq0aapSpcoTFwAAAADgr3N2dta4cZ9oypSJio+Pk4uLi8aNm6C8efOqXbsO6t27m7Jnz66CBZ9T69a+SkpK1N27dxQYOFK+vu2UP39+jRnzkaKiolSjRi21adNeEREpQ0z79q/q8uXL6t69s5KSktSiRSs1aNBIknT69Cn17PmGLBaLOnXqoqJFi2n06I81ceJ4LVu2RE5OTho3boIcHZ3Up08/vftuX2XPnl25c+dRQMAYWa0WBQSM0IYND24+8vCGHempU+cVHTt2VD16dJGLi4s8PDxSHNWTJFdXVxUu/Lx++eWi7Tb1D68RM5lMSk5OVsOGjdWtW0/bOn++hkySunTpplatfB5bz+HDB1WvXv3HtklLuoEsLCxM3t7ecnNzkyQ1b95cmzZt0oABA2xtTpw4oYULF+rKlSuqUaOGhg0blurFAAAAAJ5lSfH3H9yi/in0m55ChQppzZoNj1zu5VUpzRtPdOnSTV26dEs1PyjoK0nS4cOHlC+fu2bPnv/Y7Tk6Oun994emue1+/QaoX78BKea98EJxzZ27MFXbZs1aqFmzFqnmz5//eap5D7dfqFCbFHdOfNjv8eNHVaRIUa1YsVpJSYl6++1etmvh/uiNN3oqNHStBg4cJB+fNqnuwvhH6S1/UE/a78V3323RhAlTHrtuWtINZJGRkSkexubp6aljx47Zpu/du6eyZctq2LBhKly4sIYPH645c+Zo0KBBT1wMAAAAkFndiUvK0PPC8M8oWvQFLV68QCtWLJPValWrVj4qVeqlVO3q1auvrVu3KCoqyi4PmU7Ltm1b1ahR4yd+BpmUgUBmtVpTzfvjuZm5cuXSwoX/l37ffPNNjRgx4okCmbt77gy3Rdbh4ZHH6BKANDE2kZkxPpFZMTafTdWqVVe1atWNLuMvcXV11YwZn2Wo7ZgxqW9mYk+NGjX5y+umG8gKFCigQ4cO2aYjIyPl6elpmw4PD1dYWJg6duwo6UGA+/PdVtJz48ZdWSypg19mwoeM/UVF3TG6hCyBsWl/jE37YXzaH+PTPhib9pfZx6aDg4mDAMiU0r3LYp06dbR3715FR0crLi5OW7ZsUb169WzLs2fPrsmTJ+vKlSuyWq1avny5mjZt+lSLBgAAAP4JaZ0tBjyJ9MZQuoGsQIECGjRokHr06KF27drJx8dHFStWlJ+fn44fP658+fJpzJgx6tevn1q0aCGr1arevXvbbQcAAAAAI+TIkV137sQQyvCXWa1W3bkToxw5sj+yTYbOLfT19ZWvr2+KeX+8bqx58+Zq3rz5XywTAAAAyHyKFCmiK1eu6Nq1y0aXgmdYjhzZVaTIo59P9mQXewEAAAD/Ek5OTipRooTRZSCLS/eURQAAAADA00EgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAg2QokIWGhqpVq1Zq2rSpli9f/sh2O3bsUKNGjexWHAAAAABkZY7pNYiIiND06dMVEhIiZ2dnde7cWbVq1dKLL76Yot3169c1ceLEp1YoAAAAAGQ16R4hCwsLk7e3t9zc3JQzZ041b95cmzZtStVu5MiRGjBgwFMpEgAAAACyonQDWWRkpDw8PGzTnp6eioiISNHmyy+/VLly5VSpUiX7VwgAAAAAWVS6pyxardZU80wmk+3f586d05YtW/S///1Pv//++18qwt09919aD882D488RpcApImxicyM8YnMirEJ/DXpBrICBQro0KFDtunIyEh5enrapjdt2qSoqCi9+uqrSkxMVGRkpLp27aqgoKAMF3Hjxl1ZLKmDX2bCh4z9RUXdMbqELIGxaX+MTfthfNof49M+GJv2l9nHpoODiYMAyJTSPWWxTp062rt3r6KjoxUXF6ctW7aoXr16tuUDBw7U5s2btXbtWi1YsECenp5PFMYAAAAA4N8q3UBWoEABDRo0SD169FC7du3k4+OjihUrys/PT8ePH/8nagQAAACALCndUxYlydfXV76+vinmLVy4MFW7559/Xtu2bbNPZQAAAACQxWXowdAAAAAAAPsjkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgkAwFstDQULVq1UpNmzbV8uXLUy3/7rvv5Ovrq9atW2v48OFKSEiwe6EAAAAAkNWkG8giIiI0ffp0BQUFae3atQoODtaFCxdsy2NjYzVmzBh98cUX2rBhg+7fv69vvvnmqRYNAAAAAFlBuoEsLCxM3t7ecnNzU86cOdW8eXNt2rTJtjxnzpzatm2b8ufPr9jYWN24cUMuLi5PtWgAAAAAyArSDWSRkZHy8PCwTXt6eioiIiJFGycnJ+3cuVMNGzbUzZs39corr9i/UgAAAADIYhzTa2C1WlPNM5lMqebVr19f+/fv17Rp0xQYGKipU6dmuAh399wZbousw8Mjj9ElAGlibCIzY3wis2JsAn9NuoGsQIECOnTokG06MjJSnp6etulbt27pxIkTtqNivr6+GjRo0BMVcePGXVksqYNfZsKHjP1FRd0xuoQsgbFpf4xN+2F82h/j0z4Ym/aX2cemg4OJgwDIlNI9ZbFOnTrau3evoqOjFRcXpy1btqhevXq25VarVUOGDFF4eLgkaePGjapaterTqxgAAAAAsogMHSEbNGiQevToocTERHXs2FEVK1aUn5+fBg4cKC8vL40dO1bvvPOOTCaTXnzxRY0ePfqfqB0AAAAAnmnpBjLpwWmIvr6+KeYtXLjQ9u8mTZqoSZMm9q0MAAAAALK4DD0YGgAAAABgfwQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAySoUAWGhqqVq1aqWnTplq+fHmq5Vu3blXbtm3Vpk0b9e/fXzExMXYvFAAAAACymnQDWUREhKZPn66goCCtXbtWwcHBunDhgm353bt3FRgYqAULFmjdunUqXbq0Zs2a9VSLBgAAAICsIN1AFhYWJm9vb7m5uSlnzpxq3ry5Nm3aZFuemJiowMBAFShQQJJUunRpXbt27elVDAAAAABZhGN6DSIjI+Xh4WGb9vT01LFjx2zTefPmVZMmTSRJ8fHxWrBggbp37/5ERbi7536i9sgaPDzyGF0CkCbGJjIzxicyK8Ym8NekG8isVmuqeSaTKdW8O3fuqH///ipTpozat2//REXcuHFXFkvq7WQmfMjYX1TUHaNLyBIYm/bH2LQfxqf9MT7tg7Fpf5l9bDo4mDgIgEwp3VMWCxQooOvXr9umIyMj5enpmaJNZGSkunbtqjJlyujjjz+2f5UAAAAAkAWlG8jq1KmjvXv3Kjo6WnFxcdqyZYvq1atnW56cnKy+ffuqZcuW8vf3T/PoGQAAAAAgtXRPWSxQoIAGDRqkHj16KDExUR07dlTFihXl5+engQMH6vfff9epU6eUnJyszZs3S5IqVKjAkTIAAAAASEe6gUySfH195evrm2LewoULJUleXl46c+aM/SsDAAAAgCwuQw+GBgAAAADYH4EMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMkqFAFhoaqlatWqlp06Zavnz5I9sNGzZMISEhdisOAAAAALKydANZRESEpk+frqCgIK1du1bBwcG6cOFCqjZ9+/bVpk2bnlqhAAAAAJDVpBvIwsLC5O3tLTc3N+XMmVPNmzdPFbxCQ0PVuHFjtWzZ8qkVCgAAAABZjWN6DSIjI+Xh4WGb9vT01LFjx1K0efvttyVJhw8ftnN5AAAAAJB1pRvIrFZrqnkmk8muRbi757Zrf3g2eHjkMboEIE2MTWRmjE9kVoxN4K9JN5AVKFBAhw4dsk1HRkbK09PTrkXcuHFXFkvq4JeZ8CFjf1FRd4wuIUtgbNofY9N+GJ/2x/i0D8am/WX2sengYOIgADKldK8hq1Onjvbu3avo6GjFxcVpy5Ytqlev3j9RGwAAAABkaekGsgIFCmjQoEHq0aOH2rVrJx8fH1WsWFF+fn46fvz4P1EjAAAAAGRJ6Z6yKEm+vr7y9fVNMW/hwoWp2k2YMME+VQEAAADAv0CGHgwNAAAAALA/AhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABslQIAsNDVWrVq3UtGlTLV++PNXy06dP69VXX1Xz5s3l7++vpKQkuxcKAAAAAFlNuoEsIiJC06dPV1BQkNauXavg4GBduHAhRZshQ4boo48+0ubNm2W1WrVq1aqnVjAAAAAAZBXpBrKwsDB5e3vLzc1NOXPmVPPmzbVp0ybb8t9++03x8fGqXLmyJKlDhw4plgMAAAAA0uaYXoPIyEh5eHjYpj09PXXs2LFHLvfw8FBERMQTFeHgYHqi9kbxzJvD7n06unqk3+gJeeTMZ/c+s3nav85n5X1/FjA27YuxaV+MT/tifNoPY9O+MvvYzOz14d/LZLVarY9rMG/ePMXFxWnQoEGSpK+++krHjx/XmDFjJElHjhzR5MmTtWLFCknSpUuX9M4773CUDAAAAADSke4piwUKFND169dt05GRkfL09Hzk8qioqBTLAQAAAABpSzeQ1alTR3v37lV0dLTi4uK0ZcsW1atXz7a8cOHCypYtmw4fPixJWrNmTYrlAAAAAIC0pXvKovTgtvfz589XYmKiOnbsKD8/P/n5+WngwIHy8vLSmTNnNHLkSN27d0/lypXTJ598Imdn53+ifgAAAAB4ZmUokAEAAAAA7C9DD4YGAAAAANgfgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIHsLzp37pxKly6tzZs32+bt3LlTDRs21Pvvv5+qfdu2bZ9KHQMHDpSvr+9T6VuSjh8/Ln9//6fWPzKPu3fvavTo0fLx8VHbtm3VvXt3nTx50rB6ntb/GWR99+7d0+jRo9W0aVO1adNGXbt21d69eyVJwcHBWr9+vSRp+PDhCgkJMbJUZCFXr15Vo0aNUs0vXbr0I9fZv3+/unfv/jTLAvAMcDS6gGdVSEiImjdvrpUrV6p58+aSpE2bNqlv3756/fXXU7Vfu3at3Wu4efOmTp06pfz58+vw4cOqVq2a3bfh5eUlLy8vu/eLzMViscjPz0+1atXSmjVr5OjoqH379snPz08bNmxQ3rx5//Gansb/GWR9VqtVffv2VdmyZbVhwwY5Ozvr1KlT6tOnj6ZOnaoff/xRNWvWNLpMAABsCGR/QVJSktatW6fly5erc+fOunz5svbv36/vv/9ee/fulYODg9atWydXV1edP39eM2bMULt27XT27FndunVL/v7+unjxopydnTV8+HDVrl1by5Yt09q1axUXFyeTyaQZM2aoZMmSatSokdq0aaM9e/YoLi5OEydOVIUKFSQ9eGB39erV9dJLLyk4ONgWyEJCQrRjxw5FRkbq999/V8+ePRUeHq59+/bJzc1NixYtUrZs2bRmzRotWbJEFotF5cuX16hRo5QtWzZ5e3urfPnyun79uoYOHap58+Zp6dKlOn36tAICAhQfHy9XV1dNmTJF+fPnV2BgoM6fP6/r16+rePHimj17tq5fv64BAwaoVKlSOn36tNzd3fXpp5/Kzc3NwHcOj7J//35FRkZq4MCBcnB4cODc29tbn3zyiSwWi0aOHJnme9yjRw9t27ZNkjRr1ixJ0rvvvqvQ0FDNnTtXJpNJXl5eGjt2rKKjozVixAjduXNHUVFRat26tT744AOdOXNGAQEBSkpKUrZs2fTJJ5/ohRdeUOnSpXX27FlFRESkuV5ISIh2796tmJgYXblyRS+//LICAwONegmRSRw4cEDh4eH68ssvZTKZJEnlypVTv3791KtXL+XJk0f79u2Th4eHJGnHjh0KCgrSjRs3bD+o3bt3T2PGjNH58+eVnJwsPz8/+fj4KCQkRN98841u3bqlhg0bavDgwUbuKp4hGfm8WrJkibZu3aoFCxaoT58+8vLy0uHDhxUdHa2RI0eqfv36un79uvz9/RUeHi5HR0cNGjRIFSpUsH1PkKS6devqww8/VKtWrbRgwQKZTCbFx8crIiJCly5d0m+//abXXntN/fr1M+CVAJAWTln8C3bs2KFChQqpePHiatKkiVauXKnXXntNjRo10sCBA/Xaa69Jku2UxrJly9rW/fTTT1W0aFFt3LhRkyZN0owZM3T37l1t3bpVS5cu1fr169WkSRMFBQXZ1nFzc9Pq1avVuXNnzZ8/3zY/JCRELVu2VMuWLbV582bdunXLtuz48eNatGiRli9frgkTJqhevXoKDQ2VJO3evVvnz5/XqlWrtHLlSq1du1bu7u5avHixpAdH3vr06aO1a9fK0fH/MvsHH3yg/v37KzQ0VK1atdKSJUv0448/ysnJScHBwfruu+90//597dy5U5J05swZ9e7dW+vXr5eLi4tt+8h8Tp06JS8vL1sYe6h+/fq6ePHiI9/jtEREROiTTz7R559/rg0bNig5OVk7d+7U+vXr5ePjo1WrVmndunUKCgpSdHS0lixZot69eyskJETdu3fXTz/9lKK/R60nST/++KNmzpypdevWafv27Tp79qzdXxs8W44fP64KFSrYwthDNWrUUI4cOWyf03Xr1pUkJSQk6KuvvtL8+fM1ffp0SdLcuXNVvnx5hYSEaPny5Zo3b56uXLki6cH4/uabbwhjeGKP+7z6+uuvtWXLFs2fP185cuSQJCUmJio4OFgffvihPv30U0nS2LFj5e3trdDQUM2cOVMjRoyQxWJRoUKFdO7cOf38889KTk7WgQMHJEm7du1Sw4YNJUlnz57V4sWL9dVXX2nBggW6ffv2P/wKAHgUjpD9BSEhIfLx8ZEktWrVSh988IH++9//pmpXsWLFVPMOHjyoKVOmSHoQ2IKDgyVJU6dO1YYNG/Trr79q9+7dKULcwy8OpUqV0pYtWyRJp0+f1rVr11SnTh05OTmpbNmyWrNmjXr16iVJqlq1qnLnzq3cuXNLkmrXri1JKly4sG7fvq39+/fr0qVL6tSpk6QHH/zlypWzbbNSpUop6o6OjlZUVJTtg71r1662ZW5ublq+fLkuXryoX3/9VbGxsZIkd3d3W5+lSpVSTEzM419YGMbBwUFWqzXNZTVq1Hjke5yWH3/8UVWrVlXBggUlSZMnT7Yt27dvnxYvXqzz588rMTFRcXFxql+/vsaMGaPdu3erYcOGtlOAH3rrrbfSXE+SqlSpYhvjRYoUYYxBJpNJycnJqeYnJiam2b5x48YymUwqVaqUbt68KUkKCwtTfHy8vv76a0lSbGyszp8/L+nB0bY//lAFPPTnH7SkB6fQPvxx4FGfV+fOnVNAQICmTZumnDlz2tb949/+hz+47tu3T+PGjbP1UalSJR09elT169fX3r175ejoqB49emjDhg22swpefPFFSVKtWrXk7Owsd3d3ubm56c6dO3JxcXk6LwaAJ8JflSd048YN7dq1SydOnNCXX34pq9Wq27dv24LSH2XPnj3VvD//If/555+VPXt29ezZU926dVO9evWUP39+nT592tYmW7ZskpTiF9+vv/5aCQkJti+v9+7d08qVK22BzMnJ6bHbTU5OVsuWLTVy5Ejb+n/8EvPn2v/c3/379xUZGalz585p5syZ6tGjhzp06KCbN2/avtg/rPth7Y/6wg/jVahQQUFBQSm+PEjStGnTVLFiRc2aNSvVe/zn9zQpKUmOjo6pxtrDo1kLFizQlStX5OPjoyZNmigsLExWq1UtWrRQlSpVtH37di1ZskQ7d+60feGQpAkTJqS5nsQYQ2qVKlXS0qVLlZiYmOJz66effkrzeliz2Swp5eerxWLR5MmTVb58eUnS9evX5erqqtDQ0DQ/1wFJcnFx0Z07d1LMu3HjhlxdXSU9+vMqV65cGj9+vMaPH6+6devaQllaf/v//BlntVqVnJys+vXra/bs2XJ2dtZ7772njRs3KjQ01BbqHrd9AMbjlMUntG7dOnl7e2vXrl3atm2btm/frr59+9qOdKWnevXq+vbbbyU9CGN+fn46ceKEihUrpl69eqlSpUratWtXmr/wPpSQkKDQ0FD973//07Zt27Rt2zZ9//33ioqK0v79+zNUR61atfTdd9/pxo0bslqtCgwM1JIlSx7ZPk+ePCpYsKB++OEHSQ9uuPDpp59q7969atmypV599VXlz59fBw8efGztyJyqV68ud3d3zZ492/b+7d6923bdQ1rvsYuLi2JiYhQdHa2EhATt3r1b0oMbwRw9elRRUVGSpPHjx+v777/XDz/8oLfeekstW7bUtWvXFBERIYvFov/+9786duyYOnfurPfee0+nTp1KUduj1gPSUr16db344osaP3687ajYiRMnNHfuXPXv319mszndzyhvb2+tWLFCkhQZGak2bdro2rVrT712PNty586tYsWKpbj7cnBwsO0MlUcpXLiwGjdurJo1a2rmzJmPbevt7a3Vq1dLkq5cuaIjR46ocuXKKl++vH755Rf9+uuvKlmypGrVqqW5c+eqQYMGf3u/ADx9HCF7QiEhIRo0aFCKeV27dtWiRYuUP3/+dNcfOHCgRo4cqTZt2sjR0VGTJk1S2bJltXLlSrVq1UrOzs6qWLGi7fSYtGzfvl2FCxdOcVph7ty59dprr2nlypUpfhF7lDJlymjAgAHq2bOnLBaLypYtqz59+jx2ncmTJyswMFCTJk1S3rx5NWnSJN28eVMffPCBNm3aJGdnZ1WuXFlXr15Nd/vIXEwmk+bMmaNPPvlEPj4+cnR0VN68ebVgwQKZzeY03+M8efLorbfeUseOHVWwYEHb0YcCBQrI399fb731liwWiypXrqwOHTooR44cGjp0qFxcXOTu7q4KFSro6tWr6tu3r/z9/TVnzhyZzWYNHz48RW3vvPNOmusBjzJ79mxNnz5dPj4+MpvNcnV11eTJk1WrVi1dv35d06ZNU548eR65/oABAxQYGCgfHx8lJydryJAhKlq0qA4dOvQP7gWeRQ//Tn722WdKTExU6dKlFRAQoB07dqS77tChQ+Xj4/PYR9n4+/srICDA9riGcePGydPTU5JUrVo12+nc3t7e+uqrr7ijKPCMMFk5Zg0AAAAAhuCURQAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIPwYGgAyCT279+vjz/+WDlz5tS9e/dUtWpVnTp1Svfu3ZPVatW4ceNUrVo13bt3T+PGjdORI0dkNpvVpEkTDRo0SImJiZoyZYoOHjyo5ORklStXTiNHjlTu3LmN3jUAAPAIBDIAyETOnz+vrVu3KjIyUl988YWCg4Pl4OCgBQsWaOHChapWrZpmzpyp+/fv69tvv1VycrLefPNNHThwQAcPHpTZbFZISIhMJpOmTZumKVOmKDAw0OjdAgAAj0AgA4BM5LnnnlPhwoVVuHBhubq6auXKlbpy5Yr279+vXLlySZLCwsL04Ycfymw2y2w2a9myZZKkyZMn686dOwoLC5MkJSYmyt3d3bB9AQAA6SOQAUAmkjNnTknSjh079PHHH6t3795q3LixSpQooXXr1kmSHB0dZTKZbOtcu3ZN2bNnl8Vi0YgRI1S/fn1J0r1793T//v1/ficAAECGcVMPAMiEfvjhBzVs2FBdu3aVl5eXtm7dquTkZElS7dq19c0338hisSghIUEDBw7UwYMH9corr2j58uVKSEiQxWLRRx99pGnTphm8JwAA4HEIZACQCXXu3FkHDx6Ur6+vXn/9dRUpUkRXr16VxWLRgAED5OTkpLZt26pdu3aqX7++mjVrpv79+6tw4cJq3769WrVqJavVquHDhxu9KwAA4DFMVqvVanQRAAAAAPBvxBEyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAg/w8Gq383FJW/bAAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 297 }, - { - "cell_type": "markdown", - "metadata": { - "id": "WvL2mHlwp-l0" - }, - "source": [ - "Finally, we pass our filled-out `model_card` to the `mct` object to generate an HTML version of the `model_card` that can be rendered within a Jupyter notebook." - ] + "id": "FJOwW9db3wKe", + "outputId": "a34b7c03-6bfc-4843-c67e-bc5997ebf86c" + } + }, + { + "cell_type": "code", + "execution_count": 92, + "source": [ + "plot_technique_comparison(test_dict, \"selection_rate\")" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAA2QAAAGECAYAAABZM+NjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABAU0lEQVR4nO3dd3gU5f7+8XuzaUAahoR2AAURIQRQEAKHXkQgAVRURGlHQeAgCtKDIYQqICCIVAtVohiBgPQm0qR4KNJRKQaTQCCEFLLJ7u8PvuzPmEBAFyeJ79d1eV3uzDPPfGb2ybL3TjPZbDabAAAAAAB/OyejCwAAAACAfyoCGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABgIPNmDFDERERf3r5ESNG6OjRo5Kk0NBQ7dq1y1Gl/SmHDx9WWFiYoTUAAFBQEcgAII/ZtWuXbDabJGns2LGqV6+eofWcOXNGsbGxhtYAAEBB5Wx0AQCQ1yUnJ2vYsGE6d+6cnJycFBAQoIiICG3btk2zZs2SxWKRu7u7hgwZoieeeCLLsrGxsYqIiNClS5dksVjUpk0b9erVS5K0detWTZs2TVarVYULF9aoUaO0du1axcXFaeDAgZo4caImT56sV155Rc8884w2bdqkDz/8UJmZmfLw8NCwYcNUrVo1zZgxQ7/++qvi4+P166+/6qGHHtLUqVNVvHjxO27T3r17NXbsWBUuXFgpKSlavny5Jk6cqEOHDik5OVk2m01jxoxRqVKlNH36dCUlJWnYsGEaP368tmzZkut2AwCAe0MgA4BcbNy4UcnJyVq5cqUyMzM1cuRInT9/XlOnTtXChQtVtGhRnT59Wt27d9eGDRuyLDto0CB169ZNTZs21c2bN9WjRw+VLVtWtWvX1qBBg7Ro0SJVrlxZGzZs0OTJkzV//nxFR0dr8uTJCgwMtPdz9uxZjRw5UsuWLVOZMmW0e/du9enTR+vWrZMk7d+/XytWrJCHh4d69eqlyMhI9evX767bdfr0aW3atEmlS5fWDz/8oLi4OEVGRsrJyUlz587VvHnzNHv2bPXr10/r16/X+PHj9csvv9xxuwsXLuz4nQ8AQAFHIAOAXNSsWVNTp05V586dVa9ePXXt2lU7d+5UXFycunXrZm9nMpl0/vx5++uUlBTt27dPiYmJ+uCDD+zTTpw4IWdnZ1WsWFGVK1eWJD399NN6+umn71jDnj17FBQUpDJlykiS6tatq4ceesh+rVnt2rXl4eEhSapSpYoSExNz3a6SJUuqdOnSkqQnnnhC3t7eWrZsmS5cuKC9e/eqSJEi2Za523Y//vjjua4TAABkRSADgFyUKVNGGzdu1N69e7Vnzx51795dL7/8surWratp06bZ2126dEn+/v7auHGjJMlqtcpms2nZsmUqVKiQJCkhIUFubm7as2ePTCaTfVmbzaaTJ0/eMdTcvqbsj9MyMjIkSe7u7vbpJpMpx/Z/9PsjWtu2bdPYsWPVvXt3NWvWTOXLl9eqVauyLWO1Wu+43QAA4P5xUw8AyMXSpUs1bNgw1a9fX4MGDVL9+vV18uRJ7dy5U2fPnpUkbd++XW3bttXNmzfty3l4eKhGjRr69NNPJUnXr1/Xyy+/rM2bN6t69eo6e/asTp8+LUnavHmzBg0aJEkym832oHVbUFCQdu7cqQsXLkiSdu/erUuXLql69eoO2cadO3eqSZMm6tSpkwIDA7Vp0yZlZmZmq+d2HXfbbgAAcO84QgYAuWjfvr2+//57tW7dWoUKFVKpUqU0duxY7dq1SwMGDJDNZpOzs7NmzZqV7TqqyZMna/To0QoJCVF6erqCg4PVtm1b+7whQ4bYb9IxdepUSVLz5s3Vv39/jRkzxt7Po48+qpEjR6pv377KzMyUu7u7Zs+eLU9PT4dsY8eOHTVw4ECFhITIbDarVq1a2rBhg6xWq5544glNmzZN//3vfzVz5kxFRETkut0AAODemGz3cl4LAAAAAMDhOEIGAAXU22+/rZ9//jnHeVOnTlX58uX/5ooAAMAfcYQMAAAAAAzCTT0AAAAAwCAEMgAAAAAwCIEMAAAAAAySJ27qcfVqsqxWLmVzBF9fD125csPoMoBsGJvIyxifyKsYm47j5GRS0aJFjC4DyCZPBDKr1UYgcyD2JfIqxibyMsYn8irGJlCwccoiAAAAABiEQAYAAAAABiGQAQAAAIBB8sQ1ZAAAAEBeY7FYdOHCBaWmphldCvKxQoXcVaZMGbm4uOQ4n0AGAAAA5ODChQtydnZTyZL+MplMRpeDfMhmsykpKVEXLlxQ+fLlc2zDKYsAAABADlJT0+Th4U0Yw59mMpnk6el916OsBDIAAADgDghj+KtyG0OcsggAAADcg0KF3eTu5vivz2k3M5SacjPXdjExMXrxxfZ65JFbp75ZrVYlJyerTZtg9ejR+57WNW/ebElSjx697qvGmJhf9emn8xUaOlLHjx9TVNRyhYaG3VcfyBmBDAAAALgH7m7OCnlnpcP7jX6/3T0FMkkqVsxPixYts7+Oj4/XCy+0U/PmLe1B7UG4dOmSLl68KEmqXLkKYcyBCGQAAABAPnXlSrxsNqlIkSJauPBTbdq0UVZrpurUqau+fd+SyWTS4sULtGJFlHx8fOTp6aUqVQIkSUFBT2rPnoOSpNWrV+ngwQMKCxul77/fq+nTp8hms6lEiRKKiBinKVMmKibmV02aNF5Nm7bQ/PlzNGvWPJ0/f07jx4/R9euJKlSokAYMGKwqVQIUETFSHh4eOnHiuOLiYvX66z0VHNxO+/bt1YcffvB/11Z5afTocfLxKWrkLjQcgQwAAADIJy5fjlfnzh2Vnp6ua9euqXLlKnrvvck6e/aMTpw4rk8/XSSTyaTw8He1bt03evjhRxQdvVILF34uk8mk11/vag9kOUlPT9fIkaH64IOZeuyxSpo1a4bWrFmtAQMGa/78ORo0aJgOHNhvbx8ePkKdO3dTkybNdPToYQ0fPkhffLFCkhQb+5vmzPlYZ8+eUZ8+twLZp59+rCFDQlWlSoAiIz/XyZMnVKdO3Qe92/I0AhkAAACQT9w+ZdFqtWr69Ck6c+a0atV6Sh999KF+/PGounV7RZJ08+ZNlShRQgkJV1SvXn0VLlxYktSsWQtlZmbesf+zZ8/Iz89fjz1WSZLUu/ebkpQlhN2WkpKiixcvqEmTZpKkqlWrycvLW+fO/SJJqlOnrkwmkypUeFTXrydKkho0aKghQ95Ro0aN1aBBY9WpE+SYHZOPEcgAAACAfMbJyUl9+76tLl1e1pIli2S1ZuqllzqpU6dXJUlJSUkym836+uuvZLNZ7cuZzeYsgcxms8lkMikjI0OS5OycNR7cuJGklJSUHGuw2ayy2Wx/mGaz9+/q6iop610GX375VdWv31A7d+7QzJkf6Nixo+re/fU/uxsKBG57DwAA8A9S1NNVfn6eDv2vqKer0Zv1j+Ts7Kw333xbCxZ8okqVHte6dWuUkpKijIwMDR48QFu2bNJTT9XWd9/t0I0bSbp586a2b99qX97Hx0c//XRWNptNO3ZslySVLVtO165d1c8//yRJWrRogaKilmcLcpJUpIiHSpf+l7Zu3SxJOnr0sK5cuaIKFR69Y83/+U8XpaSkqGPHV9Sx4ys6efKEo3dLvsMRMgAAgH8QZ3c37Wz3vEP7/PfKr6SkdIf2iXtTt+6/FRAQqIMHD6px42Z67bUuslqtCgqqqzZtQmQymdSxYyd1795Znp5eKlGipH3ZPn366Z133pKvr6+qV39C165dk5ubm8LDx2jUqHdlsVj0r3+V0ciRo2WxpOvGjSSFh49QSEh7ex/h4WP03nvjNH/+HLm4uGjChMlycXG5Y729e/9Xo0ePlNlslpubu4YMGf4gd0++YLL98TijAa5cuSGr1fAyCgQ/P0/FxycZXQaQDWMTeRnjE3nVgxibfn6eDySQ5fW/IScnk3x9Pe5rmR9/PKZSpcrZXxv9HDLkXzEx5xQQUCXHeRwhAwAAAO5BaspNghMcjkAGAACQR6VnWuTn52l0GQAeIAIZAABAHuVqdtGLkb0d2ucXL81yaH8A/hrusggAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhJt6APhbPIg7hWXeTJfZzdWhfWak3dRVHm4KAMiBVxFnmV3dHN5vZvpNXU/OyLVdTEyM+vTpoRUr1mSZHhT0pPbsOfiXaujcuaMWLVqmH388qq1bN6tv37f07bfbdeLEMfXsef83ltmxY7suXLigTp1evedlevfuoddff0M1a9a67/XlZwQyAH+LB3WnsAfxcFMRyAAAOTC7uumnsY79d0eSyod+Jd1DIHuQFi1aJkn6+eeflJCQIElq2LCRGjZs9Kf6O3HiuMNqK+gIZAAAAEA+t3r1Ku3Zs0vXr19XTMxF1a5dV4MHD9OBA/v12Wcfy2az6ddfL6pp02YqUsRT3367VTabTVOmzJCvr6+Cgp7Uxo3bNW/ebKWmpujTT+fLz89fBw8eUFjYKB04sF9TpkyU2WxW1arV9PPPP2nWrHk6ePCAZs+eqbS0NCUlXVffvm+pfPkK+vrrryRJJUuWVNOmLTR58gSdPXtGVqtVnTt309NPP6P09HSNGxeh48ePqWTJUkpMvGbsTjQIgQwAAAAoAI4cOaylS7+U2eykF198Ts8910GS9OOPR/X551/K29tbrVo1V79+/fXZZ0s0evRIbdq0Xi+91EmS5OnpqR49eungwQPq3v11rV69SpKUkWHRqFHv6v33P1DFio9pypRJ9nV++eUyDR/+rh5++BHt3/+9pk6drCVLvtCzz946khgc3E4zZ05XpUqVFRYWoeTkG+rRo7sCAqpq27YtkqTIyCidP39enTu/9HfurjyDQAYAAADkA05OpmzTbDabTKZb0wMDq6lIkSKSpNKlS+v69euSpAoVKqh48RKSJG9vH9WqVVuSVKJESXubuzlz5oyKFi2qihUfkySFhLSzh7Lw8DHauXOHtmzZpKNHjyglJSXb8vv27VVaWppWr14pSUpNTdVPP53VwYP71b79reBWtmxZBQZWu/edUYAQyAAAAIB8wNPTS8nJN7JMS0hIkKenlyTJNcsNR0yy2WySJGdnlyzLmM33FwHMZrO9rz/q1es1PflkLdWsWUu1atVWWNjwbG2sVqvCw8fo8ccrS5KuXLkib28vrVgRlaVfs9l8X3UVFNz2HgAAAMgHihQpojJlymrLls32aStXRumpp2o7bB1ms1mZmVlvMPLww4/o+vXrOnPmtCRp/fq1MplMSkxM1Pnz59WzZ2/Vq1dfe/fultVq/b9+nJWZmSlJqlnzKUVFLZckXb4cr1dffUm//fabateuo/Xr18pqterSpRgdOXLYYduRn3CEDAAAAMgnwsPHaOLEcfrkk7myWCx69NGKGjRoqL77bodD+g8IqKqPP56jmTOnq1y5hyVJLi4uCg8fo4iIMJlMJpUr97Dc3Nzk7e2ttm3bq1OnF1SkSBFVrVpNaWlpSk1N1RNPPKGIiJF66CFfvf56T02cOF6dOr2gzMxM9e37lv71rzJ6/vkXdPbsWXXs+LxKlCip8uUrOGQb8huT7U7HH/9GV67ckNVqeBkFgp+fp+Ljk4wuA8jGz88z39z2nr+hfx4+O5FX8dnpOE5OJvn6etzXMj/+eEylSpWzvzb6OWRGsVqtmjlzul5//Q0VKlRIS5cuVnx8nN56a4DRpeUbMTHnFBBQJcd5HCEDAAAA7sH15AzDnxdmBCcnJ3l7e6t791fl4uKikiVLKTQ0zOiyCgwCGQAAAIC76tKlu7p06W50GQUSN/UAAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADMJNPQAAAIB7UNjDRW4urg7v96YlXSk3LLm2i4mJ0Ysvttcjj5SXyWSSxWJRsWJ+evfdcPn7F7/n9cXE/KpPP52v0NCRf6XsB65//zc1fHiY/Pz8jC7lgSKQFTDpmRb5+Xk6tM80S7qSrt10aJ8AAAD5jZuLq8OfCyfdejZcinIPZJJUrJifFi1aZn/90Ucz9P77E/Xee+/f8/ouXbqkixcv3nedf7epU2cYXcLfgkBWwLiaXR7IAySTRCADAADIa2rUeFI7dmzX0aOHNWXKZKWn35SPj4+GDAlVmTJltXTpYn3zTbScnJxUpUqAhg4doSlTJiom5ldNmjReTZu20Pz5s2U2OysuLlZVqgRo+PAwXb58Wf3795W3t49cXV01ffpHmjp1svbv/14mk0nPPNNGXbp0k81m08yZ07V9+1aZzWY9++zzeumlTrpw4bwmThyvxMREubu76513BqtSpce1fv1aLV68QE5OTipVqrTCw8coMfGaRo4MVWpqqpycnDRgwCBVrVpN7du30UcfzdPBg/u1Z88uXb9+XTExF1W7dl0NHjxM0q1AumXLJvn4+MjXt5gaNGik4OC2Br8r94dAhlxZ09MdftQtI+2mrialO7RPAACAf5KMDIs2bdqggICqGjFimMaNm6gqVQK0efNGhYUN17x5n2nhwk+0evV6OTmZNWnSBMXFxWnAgMGaP3+OBg0apgMH9uvYsR+1cOHnKlu2nEJDh2j58i/UuHFTnTv3i6KiVqtUqVJavvwLxcXFavHiSFks6erTp6cqVKigtLQ0HT58SEuWfKGMjAy98cZ/1Lz504qIGKmBA4eoUqXH9fPPP2nIkHf0xRdfa86cjzR//gI99NBDmj17ps6d+0XffrtN//53A736alcdOLBfhw79T1WrVsuyrUeOHNbSpV/KbHbSiy8+p+ee66BLl2J06NAP+vzz5UpNTVXXrp3UoEEjg96NP49Ahlw5ubpqZ7vnHdrnv1d+JRHIAAAA7svly/Hq3LmjJCk9PV0BAVXVpk1bnTp1UlWqBEiSmjVroQkTxigtLVWBgdXVvXtnNWzYSB06vCh/f39duHA+S581ajyhcuUeliQ980wbrVz5lRo3bqqiRR9SqVKlJEkHDuxTmzYhMpvNMpsLqWXLVtq373tlZFjUrFkLubq6ytXVVYsWLVNKSoqOH/9RY8aE29eRmpqixMRrql+/oXr27K5GjRqrSZNmeuyxSkpNTdXQoQN16tRJ1atXXx06vJRtuwMDq6lIkSKSpNKlS+v69ev6/vu9atbsabm4uMjFxUWNGjV28N7+exDIAAAAgHzij9eQSdLp06eytbPZbMrMtGrixCk6evSwdu/epbff7qtRo8Zma2s2O/9uOav9tZubm3261WrNof9MOTtnjRMxMTHy8vKUq6tbljrj4mLl5eWtAQMGKSSknXbt+k7h4SP02mtvqFWrNlq2bLm++26HNm3aoDVrojVjxqws/bq6uv3ulUk2m01ms5Nstqx15Ufc9h4AAADIx8qVe1iJiYk6duxHSdKmTRtUokRJWa1Wdez4vCpUqKiePXurTp0gnTlzWmazWZmZmfblDx36n+Li4mS1WvXNN6tVt269bOuoVesprVkTrczMTKWlpWr9+rWqWbOWatR4Utu2bVFGhkVpaanq37+vEhISVKZMGa1du0aStHfvHvXq9boyMzPUoUM7+fgUVdeu/1GrVsE6deqkZsyYprVr16hNmxANHDhEJ0+euKftrl07SFu3bpbFYlFy8g19990OmUwmB+zRvxdHyAAAAIB8zNXVVWPGjNfkye8pLS1VXl5eGjNmgooWLar27Z9T9+6vyt3dXSVKlFSbNiHKyLDoxo0khYePUEhIexUrVkwREe8qPj5eTz1VR23bPqvY2Ngs63j22ed1/vx5de7cURkZGXrmmdZq3LipJOn48WPq2vUVWa1WvfjiyypbtpxGjRqr994bp8WLF8jFxUVjxkyQs7OLevbsrTff7CV3d3d5eHgqLCxCNptVYWHDtWbNrZuP3L5hR27q1auvw4cPqUuXl+Xl5SU/P78sR/XyC5PNZrMZXcSVKzdktRpeRoHg5+f5QO6y+CCuIYuPT3Jon8jbGJvIy/z8PHnfkSfx2ek4Tk4m+fp63NcyP/54TKVKlbO/Nvo5ZA/CgQP7NX/+HM2aNc+Q9f8VR44c0vnz5+0h8/XXuyk0dKQqVnzM6NKyiYk5p4CAKjnO4wgZAAAAcA9Sblju+XlhePDKln1YH388V59/vlg2m02tWwfnyTCWGwIZAAAA8A9Vs2Yt1axZy+gy/hRvb29NmzbT6DL+MgIZAOAfLz3T4vDnLaZZ0pV07aZD+wQAFDwEMgDAP56r2eWBXKeTJAIZAODuCGT3yNOrkNzdHLu70m5mKOl6qkP7BAAAAJB/EMjukbubs0LeWenQPqPfb6e8fT8iAAAAAA/SPQWy6OhozZo1SxaLRd26ddMrr7ySY7tt27YpIiJCW7ZscWiRAADkN9b0dIdfl5aRdlNXk9Id2icAwFi5BrLY2FhNnTpVUVFRcnV1VceOHVWnTh09+uijWdpdvnxZ77333gMrFACA/MTJ1fWBPOtJBDLAMJ6FnOXs7vgHD2ek3VRSasZd28TExOjFF9vrkUfKy2QyyWKxqFgxP737brj8/Yvf1/piYn7Vp5/OV2joyL9S9gPXv/+bGj48TH5+fn+5r4iIMPXu/ab27t2t6dOnqHjxErLZbEpPT1eDBg3Vp08/mc1mrV69yj7/94YMCdXatat1+PAhWSwWXbx4QY88Ul6S9NJLL8vDw1Px8XF64YWO911broFs165dCgoKko+PjySpZcuWWrdunfr27Zul3YgRI9S3b1+9//77910EAAAAkNc5u7s5/IcW6f9+bMklkElSsWJ+WrRomf31Rx/N0PvvT9R7793f9+9Lly7p4sWL913n323q1BkO6ee7775VsWJ+9mBXv34jhYWNkiSlpKRo8OABmj9/jt54o0+2+b9XtWqgpFvhuE+fHlneC0n67397qlmzp/XQQw/dV325BrK4uLgsqdTf31+HDx/O0mbhwoWqUqWKqlevfl8r/6ezZjj+dBYAAAD8M9So8aR27NguSTp69LCmTJms9PSb8vHx0ZAhoSpTpqyWLl2sb76JlpOTk6pUCdDQoSM0ZcpExcT8qkmTxqtp0xaaP3+2zGZnxcXFqkqVAA0fHqbLly+rf/++8vb2kaurq6ZP/0hTp07W/v3fy2Qy6Zln2qhLl26y2WyaOXO6tm/fKrPZrGeffV4vvdRJFy6c18SJ45WYmCh3d3e9885gVar0uNavX6vFixfIyclJpUqVVnj4GCUmXtPIkaFKTU2Vk5OTBgwYpKpVq6l9+zb66KN5Onhwv/bs2aXr168rJuaiateuq8GDh0m6FUq3bNkkHx8f+foWU4MGjRQc3DbLflq8eIGGDh2R4z4sXLiwevfuqwED+qlnz792t93GjZvqq68i1aPH/fWTayCz2WzZpplMJvv/nzp1Shs2bNBnn32m33777b5Wfpuvr8efWi6/c3J21U9jHfsrS/nQrxza34NEGEVexdhEXsb4RF7F2Px7ZWRYtGnTBlWrVl0Wi0UjRgzTuHETVaVKgDZv3qiwsOGaN+8zLVz4iVavXi8nJ7MmTZqguLg4DRgwWPPnz9GgQcN04MB+HTv2oxYu/Fxly5ZTaOgQLV/+hRo3bqpz535RVNRqlSpVSsuXf6G4uFgtXhwpiyVdffr0VIUKFZSWlqbDhw9pyZIvlJGRoTfe+I+aN39aEREjNXDgEFWq9Lh+/vknDRnyjr744mvNmfOR5s9foIceekizZ8/UuXO/6Ntvt+nf/26gV1/tqgMH9uvQof+patVqWbb3yJHDWrr0S5nNTnrxxef03HMddOlSjA4d+kGff75cqamp6tq1kxo0aJRlucTERJ0/f14PP/zIHfdlhQoVlJh4TVevXpUkfffddnXu/P9PPXRxcdUnnyzM9T2pUeNJjR490vGBrHjx4tq/f7/9dVxcnPz9/e2v161bp/j4eD3//POyWCyKi4tTp06dtHTp0nsu4sqVG7Jaswe/vIQPGceLj+cek46QXx7JkJ/+hhib/zyMT+RVjE3HcXIy5fuDAJcvx9uDQnp6ugICqqpPn346f/6cvLy8VKVKgCSpWbMWmjBhjNLSUhUYWF3du3dWw4aN1KHDi/L399eFC+ez9FujxhMqV+5hSdIzz7TRypVfqXHjpipa9CGVKlVKknTgwD61aRMis9kss7mQWrZspX37vldGhkXNmrWQq6urXF1dtWjRMqWkpOj48R81Zky4fR2pqSlKTLym+vUbqmfP7mrUqLGaNGmmxx6rpNTUVA0dOlCnTp1UvXr11aHDS9m2PTCwmooUKSJJKl26tK5fv67vv9+rZs2elouLi1xcXNSoUeNsy/3660UVK5bbNWi3Dja5ud26PvBOpyzmpmTJktn27b3I9VtcvXr1NGPGDCUkJKhQoULasGGDRo8ebZ/fr18/9evXT5J08eJFdenS5b7CGIC/hkcyAADwz/DHa8hui4uLzTbNZrMpM9OqiROn6OjRw9q9e5fefruvRo0am62t2ez8u+Ws9te3A4okWa3WHPrPlLNz1jgRExMjLy9Pubq6Zak1Li5WXl7eGjBgkEJC2mnXru8UHj5Cr732hlq1aqNly5bru+92aNOmDVqzJlozZszK0q+r6+9vpmKSzWaT2ewkmy1rXX/k5GSSs7P5rm3OnDktf//i9sD3Zzk7O8tkcrrv5XJdonjx4urfv7+6dOmi9u3bKzg4WNWqVVOPHj105MiRP1UsAAAAAMcoV+5hJSYm6tixHyVJmzZtUIkSJWW1WtWx4/OqUKGievbsrTp1gnTmzGmZzWZlZmbalz906H+Ki4uT1WrVN9+sVt269bKto1atp7RmTbQyMzOVlpaq9evXqmbNWqpR40lt27ZFGRkWpaWlqn//vkpISFCZMmW0du0aSdLevXvUq9fryszMUIcO7eTjU1Rdu/5HrVoF69Spk5oxY5rWrl2jNm1CNHDgEJ08eeKetrt27SBt3bpZFotFyck39N13O7JcWiVJJUuWzjGw3nbjRpLmzp2lDh1evKd13k1MTIz+9a8y973cPZ3nFBISopCQkCzT5s2bl63dv/71L55BBgAAAPyNXF1dNWbMeE2e/J7S0lLl5eWlMWMmqGjRomrf/jl17/6q3N3dVaJESbVpE6KMDItu3EhSePgIhYS0V7FixRQR8a7i4+P11FN11Lbts4qNzRpinn32eZ0/f16dO3dURkaGnnmmtRo3bipJOn78mLp2fUVWq1UvvviyypYtp1Gjxuq998Zp8eIFcnFx0ZgxE+Ts7KKePXvrzTd7yd3dXR4engoLi5DNZlVY2HCtWXPr5iO3b9iRm3r16uvw4UPq0uVleXl5yc/PL8tRPUny9vZW6dL/0s8//2S/Tf3ta8RMJpMyMzPVpEkzvfpqV/syf7yGTJJefvlVtW4dfNd6DhzYp4YNG921TU4ce+EJAAAAUEBlpN28dYv6B9BvbkqVKqUVK9bccX5gYPUcbzzx8suv6uWXX802fenSLyVJBw7s10MP+erDD+fcdX3Ozi56553BOa67d+++6t076yOxHn74Ec2alf0AztNPP6Onn34m2/Q5cz7JNu32+kuVapvlzom3+z1y5JDKlCmrzz9frowMi15/vZv9Wrjfe+WVroqOXql+/forOLhttrsw/l5u82/Vk/N7sXHjBk2YMPmuy+aEQAYgGx7JAABAdkmpGff0vDD8PcqWfVgffzxXn3++WDabTa1bB6tixceytWvYsJE2bdqg+Ph4hzxkOidbtmxS06bN7vsZZBKBDEAO/umPZAAA4J+iZs1aqlmzltFl/Cne3t6aNm3mPbWNiMh+MxNHatq0+Z9e9v5vAwIAAAAAcAgCGQAAAHAHNlveflYu8r7cxhCBDAAAAMhBoULuSkpKJJThT7PZbEpKSlShQu53bMM1ZAAAAEAOypQpowsXLujSpfNGl4J8rFAhd5Upc+fnkxHIAAAAgBy4uLiofPnyRpeBAo5TFgEAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIM4G10AAADA383Tq5Dc3Rz7NSjtZoaSrqc6tE8ABR+BDAAA/OO4uzkr5J2VDu0z+v12SnJojwD+CThlEQAAAAAMQiADAAAAAINwyiIA4IHhOh0AAO6OQAYAeGC4TgcAgLvjlEUAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAId1kEAOQr1ox0+fl5Gl0GAAAOQSADAOQrTs6u+mns8w7ts3zoVw7tDwCAe8UpiwAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEHuKZBFR0erdevWatGihZYsWZJt/saNGxUSEqI2bdpo6NChSk9Pd3ihAAAAAFDQ5BrIYmNjNXXqVC1dulQrV65UZGSkzpw5Y5+fkpKiiIgIffrpp1qzZo1u3rypr7/++oEWDQAAAAAFQa6BbNeuXQoKCpKPj48KFy6sli1bat26dfb5hQsX1pYtW1SsWDGlpKToypUr8vLyeqBFAwAAAEBBkGsgi4uLk5+fn/21v7+/YmNjs7RxcXHR9u3b1aRJE129elX169d3fKUAAAAAUMA459bAZrNlm2YymbJNa9Sokfbu3aspU6YoPDxc77///j0X4evrcc9tUXD4+XkaXQKQI8Ym8jLGZ972T35//snbDvwVuQay4sWLa//+/fbXcXFx8vf3t7++du2ajh49aj8qFhISov79+99XEVeu3JDVmj345SV8yDhefHyS0SUUCIxNx2NsOg7j0/EYn47xoMamo9+f/PQ3lNfHppOTiYMAyJNyPWWxXr162r17txISEpSamqoNGzaoYcOG9vk2m02DBg1STEyMJGnt2rV68sknH1zFAAAAAFBA3NMRsv79+6tLly6yWCzq0KGDqlWrph49eqhfv34KDAzU6NGj9cYbb8hkMunRRx/VqFGj/o7aAQAAACBfyzWQSbdOQwwJCckybd68efb/b968uZo3b+7YygAAAACggLunB0MDAAAAAByPQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQQhkAAAAAGAQAhkAAAAAGIRABgAAAAAGIZABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkAEAAACAQZyNLgAAAKAgsGaky8/P0+gyAOQzBDIAAAAHcHJ21U9jn3don+VDv3JofwDyHk5ZBAAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACD3FMgi46OVuvWrdWiRQstWbIk2/xNmzapXbt2atu2rfr06aPExESHFwoAAAAABU2ugSw2NlZTp07V0qVLtXLlSkVGRurMmTP2+Tdu3FB4eLjmzp2rVatWqVKlSpoxY8YDLRoAAAAACoJcA9muXbsUFBQkHx8fFS5cWC1bttS6devs8y0Wi8LDw1W8eHFJUqVKlXTp0qUHVzEAAAAAFBC5BrK4uDj5+fnZX/v7+ys2Ntb+umjRomrevLkkKS0tTXPnzrW/BgAAAADcmXNuDWw2W7ZpJpMp27SkpCT16dNHjz/+uJ599tn7KsLX1+O+2qNg8PPzNLoEIEeMTeRljE/kVYxN4M/JNZAVL15c+/fvt7+Oi4uTv79/ljZxcXF67bXXFBQUpOHDh993EVeu3JDVmj345SV8yDhefHyS0SUUCIxNx2NsOg7j0/EYn47B2HS8vD42nZxMHARAnpTrKYv16tXT7t27lZCQoNTUVG3YsEENGza0z8/MzFSvXr3UqlUrhYaG5nj0DAAAAACQ3T0dIevfv7+6dOkii8WiDh06qFq1aurRo4f69eun3377TceOHVNmZqbWr18vSapatarGjh37wIsHAAAAgPws10AmSSEhIQoJCckybd68eZKkwMBAnThxwvGVAQAAAEABd08PhgYAAAAAOB6BDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxyT4EsOjparVu3VosWLbRkyZI7thsyZIiioqIcVhwAAAAAFGS5BrLY2FhNnTpVS5cu1cqVKxUZGakzZ85ka9OrVy+tW7fugRUKAAAAAAVNroFs165dCgoKko+PjwoXLqyWLVtmC17R0dFq1qyZWrVq9cAKBQAAAICCxjm3BnFxcfLz87O/9vf31+HDh7O0ef311yVJBw4ccHB5AAAAAFBw5RrIbDZbtmkmk8mhRfj6eji0P+QPfn6eRpcA5IixibyM8Ym8irEJ/Dm5BrLixYtr//799tdxcXHy9/d3aBFXrtyQ1Zo9+OUlfMg4Xnx8ktElFAiMTcdjbDoO49PxGJ+Owdh0vLw+Np2cTBwEQJ6U6zVk9erV0+7du5WQkKDU1FRt2LBBDRs2/DtqAwAAAIACLddAVrx4cfXv319dunRR+/btFRwcrGrVqqlHjx46cuTI31EjAAAAABRIuZ6yKEkhISEKCQnJMm3evHnZ2k2YMMExVQEAAADAP8A9PRgaAAAAAOB4BDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCAEMgAAAAAwCIEMAAAAAAxCIAMAAAAAgxDIAAAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADEIgAwAAAACDEMgAAAAAwCD3FMiio6PVunVrtWjRQkuWLMk2//jx43r++efVsmVLhYaGKiMjw+GFAgAAAEBBk2sgi42N1dSpU7V06VKtXLlSkZGROnPmTJY2gwYN0rvvvqv169fLZrPpiy++eGAFAwAAAEBBkWsg27Vrl4KCguTj46PChQurZcuWWrdunX3+r7/+qrS0NNWoUUOS9Nxzz2WZDwAAAADImXNuDeLi4uTn52d/7e/vr8OHD99xvp+fn2JjY++rCCcn0321N4p/0UIO79PZ2y/3RvfJr/BDDu/Tzd/xdeaX9z0/YGw6FmPTsRifjsX4dBzGpmPl9bGZ1+vDP5fJZrPZ7tZg9uzZSk1NVf/+/SVJX375pY4cOaKIiAhJ0sGDBzVp0iR9/vnnkqRz587pjTfe4CgZAAAAAOQi11MWixcvrsuXL9tfx8XFyd/f/47z4+Pjs8wHAAAAAOQs10BWr1497d69WwkJCUpNTdWGDRvUsGFD+/zSpUvLzc1NBw4ckCStWLEiy3wAAAAAQM5yPWVRunXb+zlz5shisahDhw7q0aOHevTooX79+ikwMFAnTpzQiBEjlJycrCpVqmj8+PFydXX9O+oHAAAAgHzrngIZAAAAAMDx7unB0AAAAAAAxyOQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAYhkP1Jp06dUqVKlbR+/Xr7tO3bt6tJkyZ65513srVv167dA6mjX79+CgkJeSB9S9KRI0cUGhr6wPpH3nHjxg2NGjVKwcHBateunTp37qwff/zRsHoe1N8MCr7k5GSNGjVKLVq0UNu2bdWpUyft3r1bkhQZGanVq1dLkoYOHaqoqCgjS0UBcvHiRTVt2jTb9EqVKt1xmb1796pz584PsiwA+YCz0QXkV1FRUWrZsqWWLVumli1bSpLWrVunXr166aWXXsrWfuXKlQ6v4erVqzp27JiKFSumAwcOqGbNmg5fR2BgoAIDAx3eL/IWq9WqHj16qE6dOlqxYoWcnZ21Z88e9ejRQ2vWrFHRokX/9poexN8MCj6bzaZevXqpcuXKWrNmjVxdXXXs2DH17NlT77//vn744QfVrl3b6DIBALAjkP0JGRkZWrVqlZYsWaKOHTvq/Pnz2rt3rzZv3qzdu3fLyclJq1atkre3t06fPq1p06apffv2OnnypK5du6bQ0FD99NNPcnV11dChQ1W3bl0tXrxYK1euVGpqqkwmk6ZNm6YKFSqoadOmatu2rb777julpqbqvffeU9WqVSXdemB3rVq19NhjjykyMtIeyKKiorRt2zbFxcXpt99+U9euXRUTE6M9e/bIx8dH8+fPl5ubm1asWKEFCxbIarUqICBAI0eOlJubm4KCghQQEKDLly9r8ODBmj17thYtWqTjx48rLCxMaWlp8vb21uTJk1WsWDGFh4fr9OnTunz5sh555BF9+OGHunz5svr27auKFSvq+PHj8vX11QcffCAfHx8D3zncyd69exUXF6d+/frJyenWgfOgoCCNHz9eVqtVI0aMyPE97tKli7Zs2SJJmjFjhiTpzTffVHR0tGbNmiWTyaTAwECNHj1aCQkJGj58uJKSkhQfH682bdpo4MCBOnHihMLCwpSRkSE3NzeNHz9eDz/8sCpVqqSTJ08qNjY2x+WioqK0Y8cOJSYm6sKFC/r3v/+t8PBwo3Yh8ojvv/9eMTExWrhwoUwmkySpSpUq6t27t7p16yZPT0/t2bNHfn5+kqRt27Zp6dKlunLliv0HteTkZEVEROj06dPKzMxUjx49FBwcrKioKH399de6du2amjRpogEDBhi5qchH7uXzasGCBdq0aZPmzp2rnj17KjAwUAcOHFBCQoJGjBihRo0a6fLlywoNDVVMTIycnZ3Vv39/Va1a1f49QZIaNGigYcOGqXXr1po7d65MJpPS0tIUGxurc+fO6ddff9ULL7yg3r17G7AnAOSEUxb/hG3btqlUqVJ65JFH1Lx5cy1btkwvvPCCmjZtqn79+umFF16QJPspjZUrV7Yv+8EHH6hs2bJau3atJk6cqGnTpunGjRvatGmTFi1apNWrV6t58+ZaunSpfRkfHx8tX75cHTt21Jw5c+zTo6Ki1KpVK7Vq1Urr16/XtWvX7POOHDmi+fPna8mSJZowYYIaNmyo6OhoSdKOHTt0+vRpffHFF1q2bJlWrlwpX19fffzxx5JuHXnr2bOnVq5cKWfn/5/ZBw4cqD59+ig6OlqtW7fWggUL9MMPP8jFxUWRkZHauHGjbt68qe3bt0uSTpw4oe7du2v16tXy8vKyrx95z7FjxxQYGGgPY7c1atRIP/300x3f45zExsZq/Pjx+uSTT7RmzRplZmZq+/btWr16tYKDg/XFF19o1apVWrp0qRISErRgwQJ1795dUVFR6ty5s/73v/9l6e9Oy0nSDz/8oOnTp2vVqlXaunWrTp486fB9g/zlyJEjqlq1qj2M3fbUU0+pUKFC9s/pBg0aSJLS09P15Zdfas6cOZo6daokadasWQoICFBUVJSWLFmi2bNn68KFC5Juje+vv/6aMIb7drfPq6+++kobNmzQnDlzVKhQIUmSxWJRZGSkhg0bpg8++ECSNHr0aAUFBSk6OlrTp0/X8OHDZbVaVapUKZ06dUpnz55VZmamvv/+e0nSt99+qyZNmkiSTp48qY8//lhffvml5s6dq+vXr//NewDAnXCE7E+IiopScHCwJKl169YaOHCg3n777WztqlWrlm3avn37NHnyZEm3AltkZKQk6f3339eaNWv0yy+/aMeOHVlC3O0vDhUrVtSGDRskScePH9elS5dUr149ubi4qHLlylqxYoW6desmSXryySfl4eEhDw8PSVLdunUlSaVLl9b169e1d+9enTt3Ti+++KKkWx/8VapUsa+zevXqWepOSEhQfHy8/YO9U6dO9nk+Pj5asmSJfvrpJ/3yyy9KSUmRJPn6+tr7rFixohITE+++Y2EYJycn2Wy2HOc99dRTd3yPc/LDDz/oySefVIkSJSRJkyZNss/bs2ePPv74Y50+fVoWi0Wpqalq1KiRIiIitGPHDjVp0sR+CvBtr732Wo7LSdITTzxhH+NlypRhjEEmk0mZmZnZplsslhzbN2vWTCaTSRUrVtTVq1clSbt27VJaWpq++uorSVJKSopOnz4t6dbRtt//UAXc9scftKRbp9De/nHgTp9Xp06dUlhYmKZMmaLChQvbl/39v/23f3Dds2ePxowZY++jevXqOnTokBo1aqTdu3fL2dlZXbp00Zo1a+xnFTz66KOSpDp16sjV1VW+vr7y8fFRUlKSvLy8HszOAHBf+FflPl25ckXffvutjh49qoULF8pms+n69ev2oPR77u7u2ab98R/ys2fPyt3dXV27dtWrr76qhg0bqlixYjp+/Li9jZubmyRl+cX3q6++Unp6uv3La3JyspYtW2YPZC4uLnddb2Zmplq1aqURI0bYl//9l5g/1v7H/m7evKm4uDidOnVK06dPV5cuXfTcc8/p6tWr9i/2t+u+XfudvvDDeFWrVtXSpUuzfHmQpClTpqhatWqaMWNGtvf4j+9pRkaGnJ2ds42120ez5s6dqwsXLig4OFjNmzfXrl27ZLPZ9Mwzz+iJJ57Q1q1btWDBAm3fvt3+hUOSJkyYkONyEmMM2VWvXl2LFi2SxWLJ8rn1v//9L8frYc1ms6Ssn69Wq1WTJk1SQECAJOny5cvy9vZWdHR0jp/rgCR5eXkpKSkpy7QrV67I29tb0p0/r4oUKaJx48Zp3LhxatCggT2U5fRv/x8/42w2mzIzM9WoUSN9+OGHcnV11VtvvaW1a9cqOjraHurutn4AxuOUxfu0atUqBQUF6dtvv9WWLVu0detW9erVy36kKze1atXSN998I+lWGOvRo4eOHj2qcuXKqVu3bqpevbq+/fbbHH/hvS09PV3R0dH67LPPtGXLFm3ZskWbN29WfHy89u7de0911KlTRxs3btSVK1dks9kUHh6uBQsW3LG9p6enSpQooZ07d0q6dcOFDz74QLt371arVq30/PPPq1ixYtq3b99da0feVKtWLfn6+urDDz+0v387duywX/eQ03vs5eWlxMREJSQkKD09XTt27JB060Ywhw4dUnx8vCRp3Lhx2rx5s3bu3KnXXntNrVq10qVLlxQbGyur1aq3335bhw8fVseOHfXWW2/p2LFjWWq703JATmrVqqVHH31U48aNsx8VO3r0qGbNmqU+ffrIbDbn+hkVFBSkzz//XJIUFxentm3b6tKlSw+8duRvHh4eKleuXJa7L0dGRtrPULmT0qVLq1mzZqpdu7amT59+17ZBQUFavny5JOnChQs6ePCgatSooYCAAP3888/65ZdfVKFCBdWpU0ezZs1S48aN//J2AXjwOEJ2n6KiotS/f/8s0zp16qT58+erWLFiuS7fr18/jRgxQm3btpWzs7MmTpyoypUra9myZWrdurVcXV1VrVo1++kxOdm6datKly6d5bRCDw8PvfDCC1q2bFmWX8Tu5PHHH1ffvn3VtWtXWa1WVa5cWT179rzrMpMmTVJ4eLgmTpyookWLauLEibp69aoGDhyodevWydXVVTVq1NDFixdzXT/yFpPJpI8++kjjx49XcHCwnJ2dVbRoUc2dO1dmsznH99jT01OvvfaaOnTooBIlStiPPhQvXlyhoaF67bXXZLVaVaNGDT333HMqVKiQBg8eLC8vL/n6+qpq1aq6ePGievXqpdDQUH300Ucym80aOnRoltreeOONHJcD7uTDDz/U1KlTFRwcLLPZLG9vb02aNEl16tTR5cuXNWXKFHl6et5x+b59+yo8PFzBwcHKzMzUoEGDVLZsWe3fv/9v3ArkR7f/nZw5c6YsFosqVaqksLAwbdu2LddlBw8erODg4Ls+yiY0NFRhYWH2xzWMGTNG/v7+kqSaNWvaT+cOCgrSl19+yR1FgXzCZOOYNQAAAAAYglMWAQAAAMAgBDIAAAAAMAiBDAAAAAAMQiADAAAAAIMQyAAAAADAIAQyAAAAADAIgQwAAAAADMKDoQEgj9i7d6/Gjh2rwoULKzk5WU8++aSOHTum5ORk2Ww2jRkzRjVr1lRycrLGjBmjgwcPymw2q3nz5urfv78sFosmT56sffv2KTMzU1WqVNGIESPk4eFh9KYBAIA7IJABQB5y+vRpbdq0SXFxcfr0008VGRkpJycnzZ07V/PmzVPNmjU1ffp03bx5U998840yMzP1n//8R99//7327dsns9msqKgomUwmTZkyRZMnT1Z4eLjRmwUAAO6AQAYAeUjJkiVVunRplS5dWt7e3lq2bJkuXLigvXv3qkiRIpKkXbt2adiwYTKbzTKbzVq8eLEkadKkSUpKStKuXbskSRaLRb6+voZtCwAAyB2BDADykMKFC0uStm3bprFjx6p79+5q1qyZypcvr1WrVkmSnJ2dZTKZ7MtcunRJ7u7uslqtGj58uBo1aiRJSk5O1s2bN//+jQAAAPeMm3oAQB60c+dONWnSRJ06dVJgYKA2bdqkzMxMSVLdunX19ddfy2q1Kj09Xf369dO+fftUv359LVmyROnp6bJarXr33Xc1ZcoUg7cEAADcDYEMAPKgjh07at++fQoJCdFLL72kMmXK6OLFi7Jarerbt69cXFzUrl07tW/fXo0aNdLTTz+tPn36qHTp0nr22WfVunVr2Ww2DR061OhNAQAAd2Gy2Ww2o4sAAAAAgH8ijpABAAAAgEEIZAAAAABgEAIZAAAAABiEQAYAAAAABiGQAQAAAIBBCGQAAAAAYBACGQAAAAAYhEAGAAAAAAb5fxWL7x8VcN3lAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 430 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "A-j_4mBaPe5z" - }, - "outputs": [], - "source": [ - "mct.update_model_card_json(model_card)\n", - "html_modelcard = mct.export_format()" - ] + "id": "dACpOpm67K-m", + "outputId": "65a6de47-9ca4-49d8-e97a-16ef1db40c02" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "\n", + "#### Model performance - overall" + ], + "metadata": { + "id": "OYkgAggpW11N" + } + }, + { + "cell_type": "code", + "execution_count": 93, + "source": [ + "overall_df = pd.DataFrame.from_dict({\n", + " \"Unmitigated\": metricframe_unmitigated.overall,\n", + " \"Postprocessing\": metricframe_postprocess.overall,\n", + " \"Postprocessing (DET)\": mf_deterministic.overall,\n", + " \"Reductions\": metricframe_reductions.overall\n", + "})" + ], + "outputs": [], + "metadata": { + "id": "w3Rwe98m6Pv2" + } + }, + { + "cell_type": "code", + "execution_count": 94, + "source": [ + "overall_df.T" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
selection_ratefalse_negative_ratebalanced_accuracy
Unmitigated0.3870.4460.594
Postprocessing0.4450.3950.591
Postprocessing (DET)0.4440.3940.591
Reductions0.3960.4460.589
\n", + "
" + ], + "text/plain": [ + " selection_rate false_negative_rate balanced_accuracy\n", + "Unmitigated 0.387 0.446 0.594\n", + "Postprocessing 0.445 0.395 0.591\n", + "Postprocessing (DET) 0.444 0.394 0.591\n", + "Reductions 0.396 0.446 0.589" + ] + }, + "metadata": {}, + "execution_count": 94 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 172 }, - { - "cell_type": "markdown", - "metadata": { - "id": "CkuNE_MvMy7P" - }, - "source": [ - "### Display the model card" - ] + "id": "44Q-lYTa6PzX", + "outputId": "376ce1db-ead9-4984-83c6-7e59e815a1b2" + } + }, + { + "cell_type": "code", + "execution_count": 95, + "source": [ + "overall_df.transpose().plot.bar(subplots=True, layout= [1,3], figsize=(12, 5), legend=False, rot=-45, position=1.5);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtwAAAGICAYAAAByXjcVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABXUUlEQVR4nO3deVwU9f8H8NdyqYgKAktJ5R1ogqKmSIonogjemkdSauSNUV6pKZ55Y1rfPH5WmqJYHoh5ICqa9x2mgqLiBQICKghy7H5+f/hlv5EagszODryej4ePhzszu/P+sPOafc/s7K5KCCFARERERESSMJK7ACIiIiKi0owNNxERERGRhNhwExERERFJiA03EREREZGE2HATEREREUmIDTcRERERkYTYcBNRmTF9+nS0a9cOQUFBL5x/8uRJeHt767mqkhUVFYVp06YBAC5evAh/f3+DqYeoqIqTSQcHB6SmpkpUUeHWrFmDSZMmybZ+MkxsuBVi+fLlmDlzZrHvP3XqVPz1118AgClTpuDYsWMlVVqx8EWY5BASEoLg4GAEBATIXYpkYmNjkZiYCABwcnLCsmXLDKYeIqKyykTuAkg/jh07hg8//BAAMGfOHJmr4Ysw6d+AAQMghICfnx86deqEP/74Azk5OUhNTUX37t3x+eefF1j+zJkzmDdvHrRaLQBg2LBh8PT0RE5ODhYtWoTTp09Do9Ggfv36mDp1KiwsLF667pMnTyIoKAhvv/02rl27hpycHEybNg2urq7/+nhRUVEIDAxEbm4u3nnnHcTHx2PSpEl4//33MXfuXPz555948uQJhBCYPXs2qlWrhmXLliE9PR1fffUVunfvjlmzZmHjxo1o3bo19u7dC1tbWwBA3759MWrUKLRo0aJY45kzZw7Mzc2RmZmJ3377DQsWLCi0nm+++QYHDhzADz/8gNzcXJQvXx4TJ06Ei4vL6z/BVGplZmbC398ft27dQuXKlXUnn2bOnInMzEwkJSXB0dERS5cuRbly5QrcLzAwEHFxcXj06BEqVqyIRYsWoVatWhg0aBAaNWqEc+fOISEhAU2aNMH8+fNhZGSEgwcPYunSpdBqtTA3N8eMGTPg6OiIc+fOYdGiRcjKyoJKpcKYMWPQtm1b5ObmYvbs2Th27Bisra1hbW2NSpUqFTquFStWICIiAtnZ2cjKysLEiRPh4eGBvLw8LFy4EJGRkTA2NoaLiwumT58OIyOjF05fuXIl0tLSdCexli9frrs9aNAgVKlSBTdu3ED//v3h5OSEhQsXIicnB8nJyXBzc8PcuXMB4IXjPnjwIGJjY7F48WIAwNmzZzFr1ixs3769hJ/lMkCQLDIyMsSYMWNE165dRffu3cWUKVOERqMR+/fvF7179xbdunUTH374oTh37pwQQohly5aJGTNmCCGEuH//vhg5cqTo0aOH8Pb2Fj/88IPucQ8cOCC6du0qvL29Rd++fcWVK1fEkiVLxHvvvSc6duwoLly4ID766COxe/duIYQQ+/btE926dRPe3t6iX79+4s8//9Stb+LEiWLIkCHC09NT9O/fX9y/f/9fx3TixAnh4+MjPvzwQ+Hj4yOys7PFrFmzRO/evUXnzp1Fp06dxJkzZ0R8fLxo3bq1aNy4sZg0aZIQQrx03EQl6d133xUpKSnio48+Ejdv3hRCPMtTvXr1REpKijhx4oTo0qWLEEIIX19fsXPnTiGEEFeuXBGBgYFCCCGWL18u5s2bJ7RarRBCiMWLF4vp06f/63pPnDgh6tWrJy5fviyEEGLNmjVi4MCB//p4ubm5wt3dXURGRgohhDh+/LhwcHAQJ06cEOfOnRNjxowRGo1GCCHEypUrxbBhw4QQQmzZskV89tlnuvXmj2fChAni//7v/4QQQsTGxoo2bdoIjUZT7PE4OjqKu3fvCiHEK9dz8+ZN4e3tLVJTU4UQQly9elV88MEH4smTJ/+6Piq78re1s2fPCiGE2LRpk+jdu7eYN2+e2L59uxBCiJycHOHt7S327NkjhPhfznfv3i1mzZqle6yvv/5azJw5UwghxEcffST8/f2FRqMR6enpomXLluL48eMiOTlZNGnSRJfVvXv3iqFDh4qHDx+Kjh07ijt37gghnu033N3dxb1798TPP/8sfH19RXZ2tnjy5Ino0aOHmDhx4r+O6+7du2LQoEEiKytLCCHEzp07hbe3txBCiLVr14qBAweKrKwsodFoxNixY8W2bdteOv3v/YEQBfuFjz76SHz11Ve6eQEBAeLEiRNCiGd9SPPmzcXFixdfOu4HDx6Ixo0bi7S0NCGEEOPHjxcbN24s0nNIz/AMt0z27duHJ0+eIDQ0FBqNBtOnT8ft27cRFBSEdevWwcrKCteuXcPgwYMRHh5e4L7jx4/HJ598gnbt2iE7Oxt+fn5455130KxZM4wfPx6//PIL6tWrh/DwcCxatAj/93//h7CwMCxatAhOTk66x7l+/TqmT5+OTZs24e2338bx48cxcuRI7NmzB8CzM3zbt2+HhYUFhg8fjpCQkEKvB7127RoiIiJgb2+P8+fPIykpCSEhITAyMsKqVauwevVqrFixAv7+/ti7dy+++eYbxMXFvXTc5ubmJf/HpzJvxYoViIyMxM6dO3H9+nUIIZCVlVVgmc6dO2PmzJk4cOAA3Nzc8MUXXwAAIiMjkZ6errssKzc3F9bW1oWus1q1aqhXrx4AoH79+ti2bdu/Pt7Vq1cBAK1btwYAuLq6om7dugAAFxcXVKlSBZs2bcKdO3dw8uRJVKxY8V/X36dPH8yYMQNDhw7Fli1b0LNnTxgZGRV7PG+++Sbs7e2LVM/Ro0eRlJSETz75RDdNpVLh9u3bcHR0LHSdVDY5ODigcePGAIAePXogMDAQP/74Iy5cuIDVq1cjLi4OSUlJyMzMLHC/Tp064e2338Yvv/yCW7du4dSpUwXeTWnbti2MjIxgYWGB6tWr49GjRzh37hzq1q2ry2rHjh3RsWNHHDp0CMnJyRg1apTu/iqVCjExMTh+/Di8vb1hZmYGMzMz+Pj4ICYm5l/HZG9vj/nz5yMsLAy3bt3SvTsEPHtHulu3bihfvjwAYOnSpQCA4cOHv3D68uXL/3VdTZs21f1/3rx5OHz4MFasWIEbN27g6dOnyMzMfOm4AaBNmzYIDQ1F9+7dceTIEUyfPv1f10cvxoZbJk2aNEFQUBAGDRoENzc3fPzxx//6YpQvMzMTp0+fxqNHj/Dtt9/qpkVHR8PExOSlgXmREydOwNXVFW+//TYAoEWLFqhataruWu9mzZrp3lauX78+Hj16VOi4+CJMhi4rKwv9+vVDhw4d0LRpU/Tq1QsREREQQhRYrl+/fmjbti2OHj2KP/74A9999x127NgBrVaLyZMn6xrhJ0+eIDs7u9D15r9IAs+27/z1vezxkpOTn6vJ2NgYwLMmfc6cORg8eDDat2+PWrVqYceOHf+6/qZNmyIvLw9RUVHYuXMnNm3a9K/rL8zfD4ZftR6tVosWLVroGgUASEhIgFqtLnR9VHYZGRX8uJlKpcKUKVMghEDnzp3Rpk0bJCQkPJeX4OBgbN68GQMHDoSPjw8sLS1x9+5d3fwXZdLExAQqlUo3XQiBmJgYaDQa1K5dG7/++qtuXmJiIqpWrYqQkJAC683P6b+5dOkSRo4ciU8++QQffPAB3n//fcyYMQMAYGJSsDV78OABtFrtS6f/fX8CPDto/ru/Z3XgwIFwdHREq1at0LlzZ/z5558QQsDY2PiF43Z0dMTAgQMRGBgIExMTdOzYsdCDe3oxfmhSJm+//Tb27duHzz77DBkZGRg8eDAePnyIFi1aIDQ0VPdv8+bNurNawLMXLCEENm3apFsmJCQEw4YNe2FgoqOjX1rDP3dO+dPy8vIAvLxB+Df/fBEeNmwYAKB9+/bo37//C++T/yL8b+MmKimpqanIyMjA559/jnbt2uHUqVPIycnRXaudr1+/frhy5Qp69uyJWbNm4fHjx3j06BFatmyJDRs26O7z9ddfY8mSJcWu52WPV7t2bZiZmeHw4cMAnn3Q+OrVq1CpVDh69Cjatm2LAQMGwMnJCREREdBoNACevdjnZ/if+vTpg1mzZsHBwQHVqlX71/UXxavW4+rqiqNHj+L69esAgEOHDqFr166v1OBT2RUTE4MrV64AePbB5yZNmuDYsWMYNWoUvLy8oFKp8Oeff+q2uXxHjhxBjx490KdPH9SsWRMHDhx4bpl/atiwIa5fv45r164BAPbv34/x48ejUaNGuHXrFk6fPg0AuHLlCjw9PZGUlIRWrVph+/btyM7ORnZ2Nnbt2lXomE6fPo0GDRpg8ODBaNasGfbv36+rrUWLFti5c6cuk4GBgfj9999fOt3KygqXLl2CEAKZmZk4cuTIC9f56NEj/PXXXxg3bhw6duyIxMRE3L59G1qt9qXjBoDGjRvDyMgIa9aseenrOBWOZ7hlEhwcjLNnz2LRokVo1aoVUlJSdG9NXb9+HbVr18ahQ4cwbtw4HDp0SHc/CwsLNGrUCD/99BNGjhyJx48fo3///hg1ahTc3Nx0galbty7279+Pb7/9FmFhYS98EXZ1dcV3332HO3fu6C4pSUhIQMOGDXH+/PnXHuPfX4Szs7OxevXql74IL1u27IXj5iUlVNKqVauGNm3aoHPnzqhcuTLeeecd1KlTB7du3YKZmZluuXHjxmHu3LlYunQpjIyMMHr0aLz11lsYOXIk5s+fjx49ekCj0aBevXqv9RVgL3s8ExMTLF++HNOnT8eSJUtQo0YN2NjYoHz58ujXrx/GjRsHHx8fGBsbo2nTpggPD4dWq4WLiwuWLl2KUaNGwdfXt8C6unfvjiVLlhRoqEtiPK9az/fff4+ZM2fiiy++0J1N/OGHH5hz+le1atXSvVZZW1tj3rx5iIyMxKhRo1ClShVUqFAB77//foF3gwFgyJAhmDZtGrZu3QpjY2O89957uku1XsbGxgaLFi3CxIkTodFoYGFhgaCgIFStWhXLli3DggULkJ2dDSEEFixYAHt7e/Tr1w+3b9+Gt7c3LC0tUb169ULH5O3tjfDwcHh5ecHU1BQtWrTAo0ePkJGRgX79+uHevXvo2bMnhBBo1qwZBg0aBJVK9cLpWVlZ+OOPP9CxY0fY2dnBxcXlhSfIqlSpgs8++ww9evSApaUlrKys0LhxY9y6dUv34el/jjtfz549sWvXLjg4OLzis0b/pBKvctqSSlxmZiYmT56MmJgYVKhQAdWqVcOcOXNw7NgxrFixQvdiNHnyZDRt2rTAp47v3r2LWbNmIT4+Hjk5OfD29saYMWMAAH/88QeCgoJ0gZkxYwbq1KmD+fPnY/fu3Zg9ezZWrlyJgQMHolOnTti9ezdWrFgBjUaD8uXLY9KkSc+tD8Bzt1/k5MmTmDVrFnbu3Ang2TXi48aNQ15eXoEX4cjISNy9excff/wx6tevj++//15Xxz/HTVSWzZ8/H0OHDoWNjQ0SEhLQrVs3REREoHLlynKXRkRlRF5eHkaPHo2uXbvCy8tL7nIUiw03EVEJ+Pzzz3Hz5s0XzgsKCkKtWrWK/Jjr16/Hpk2bYGJiAiEERo0a9a+fyyhJUoyHqCyZO3cuTp48+cJ5X331FVxdXfVcUdHFxsaif//+cHd3x8KFC5+7np5eHRtuKhK+CBMREREVDRtuIiIiIiIJ8b0BIiIiIiIJseEmIiIiIpKQQXwtYFraE2i1yr2yxdraAikpGXKX8dpKwziUPgYjIxWsrAz/RwWYWflxDPJjXvVD6dsJUDrGACh/HHJm1iAabq1WKHpnAEDx9ecrDeMoDWMwdMysYeAY6FUwr4ahNIwBKD3j0DdeUkJEREREJCE23EREREREEmLDTUREREQkITbcREREREQSYsNNREQkk7CwMHh5ecHDwwMbNmx4bv6NGzcwaNAgdO3aFUOHDsWjR49kqJKIXhcbbiIiIhkkJiYiKCgIwcHBCA0NRUhICGJjY3XzhRAYMWIE/Pz8sGPHDtSrVw+rVq2SsWIiKi423ERERDI4duwYXF1dYWlpCXNzc3h6emLPnj26+ZcuXYK5uTnc3d0BAMOHD8fAgQPlKpeIXgMbbiIiIhkkJSXB1tZWd1utViMxMVF3+/bt27CxscHEiRPh4+OD6dOnw9zcXI5Sieg1GcQP3xAREZU1Qjz/AyIqlUr3/7y8PJw6dQrr16+Hk5MTli5dinnz5mHevHlFWo+1tcVr1yo3W9tKcpfw2krDGIDSMw59Y8NNREQkAzs7O5w5c0Z3OykpCWq1Wnfb1tYW1atXh5OTEwDA29sb/v7+RV5PSkqGon8d0Na2EpKT0+Uu47WUhjEAyh+HkZFKtgNQNtwyqVS5AsqXk+7P/zQ7D+mPsyR7fKJXZVXFDCZm5SR7/LycbKQ9ypHs8cmwlKbtyc3NDcuXL0dqaioqVKiA8PBwzJo1SzffxcUFqampiI6OhqOjIw4cOID33ntP0poqWZZDeVMzSdfxNDcH6Q+zJV0HkaFhwy2T8uVM4PNlqGSPH7a4G6Q+BpX6oAHggUNpYGJWDjfm9JLs8WtN2QKADXdZUZq2Jzs7OwQEBMDX1xe5ubno3bs3nJ2d4efnB39/fzg5OeH777/H1KlTkZWVhTfeeAMLFiyQtKbypmboGzJC0nVs/vAHpIMNN5UtbLip2KQ+aAD0c+BARCQXHx8f+Pj4FJi2evVq3f8bNmyI3377Td9lKVppOUsv9Tj4ToN+seEmIiKiUqO0nKWXehz6GAMPGv6HDTcRERERlbjScNBQUvg93EREREREEmLDTUREREQkITbcREREREQSYsNNRERERCQhNtxERERERBJiw01EREREJCE23EREREREEmLDTUREREQkITbcREREREQSYsNNRERERCQhNtxERERERBJiw01EREREJCE23EREREREEmLDTUREREQkITbcREREREQSYsNNRERERCQhNtxERERERBJiw01EREREJKFXarjDwsLg5eUFDw8PbNiw4aXLRUZGol27diVWHBERERGR0hXacCcmJiIoKAjBwcEIDQ1FSEgIYmNjn1vuwYMHmD9/viRFEtGr4wEyERGRYSm04T527BhcXV1haWkJc3NzeHp6Ys+ePc8tN3XqVIwePVqSIono1fAAmYiIyPCYFLZAUlISbG1tdbfVajWioqIKLLNu3TrUr18fDRs2LFYR1tYWxbqfIbG1rSR3Cc8pTk2lYRyGOAZ9+fsBMgDdAfI/D4bzD5AXL14sQ5VERERlS6ENtxDiuWkqlUr3/6tXryI8PBw///wz7t+/X6wiUlIyoNU+vx6lsLWthOTk9CLfR2rFqako99FXY1vUmoo6bkNiZKR6rQNQfRwgA4Z3kFxaDi6LimOQhiHWRETKVmjDbWdnhzNnzuhuJyUlQa1W627v2bMHycnJ6NWrF3Jzc5GUlIQBAwYgODhYmoqJ6KX0cYAMFO0guTQcXBqisjoGQ9qeXvcAmYjKjkIbbjc3NyxfvhypqamoUKECwsPDMWvWLN18f39/+Pv7AwDu3r0LX19fNtukGJUqV0D5coXG4LU8zc5D+uMsSdeRjwfIREREhueVznAHBATA19cXubm56N27N5ydneHn5wd/f384OTnpo04iSZQvZwKfL0MlXUfY4m7Q13lIHiBLw6qKGUzMykm6jrycbKQ9ypHs8UvDGIiIlOqVTu35+PjAx8enwLTVq1c/t9xbb72FAwcOlExlRFRkPECWholZOdyY00vSddSasgWAdM1qaRgDEZFSSfteOhHpHQ+QiYiIDAt/2p2IiIiISEKKPMMt9Qfd9PkhNyIiIiIq3RTZcEv9QTd9fsiNiIiIiEo3XlJCRERERCQhNtxERERERBJiw01ERCSjsLAweHl5wcPDAxs2bHhu/nfffYe2bduiW7du6Nat2wuXISLDpshruImIiEqDxMREBAUFYevWrTAzM0O/fv3QvHlz1KlTR7fMX3/9hSVLlsDFxUXGSonodfAMNxERkUyOHTsGV1dXWFpawtzcHJ6entizZ0+BZf766y+sXr0aPj4+mDlzJrKzs2WqloiKi2e4iYiIZJKUlARbW1vdbbVajaioKN3tJ0+eoF69epg4cSLs7e0xadIk/Oc//0FAQMArr8Pa2qJEay4JtraVJF1eH8riGIp7HykZWj0vw4abiIhIJkKI56apVCrd/ytWrFjgl2KHDBmCyZMnF6nhTknJgFb7/HpeRF/NS3Lyq3/5rq1tpSIvrw9SjiH/PlIrTk2G9lwUpR4jI5VsB6C8pISIiEgmdnZ2ePDgge52UlIS1Gq17nZ8fDx+++033W0hBExMeK6MSGnYcBMREcnEzc0Nx48fR2pqKrKyshAeHg53d3fd/PLly2PhwoW4c+cOhBDYsGEDPDw8ZKyYiIqDDTcREZFM7OzsEBAQAF9fX3Tv3h3e3t5wdnaGn58fLl68iKpVq2LmzJkYMWIEOnXqBCEEBg8eLHfZRFREfF+KiIhIRj4+PvDx8Skw7e/XbXt6esLT01PfZRFRCeIZbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpLQKzXcYWFh8PLygoeHBzZs2PDc/H379sHHxwddunTBpEmTkJOTU+KFEhEREREpUaENd2JiIoKCghAcHIzQ0FCEhIQgNjZWNz8zMxMzZ87ETz/9hN9//x3Z2dnYtm2bpEUT0cvxAJlIOQrLa77IyEi0a9dOj5URUUkqtOE+duwYXF1dYWlpCXNzc3h6emLPnj26+ebm5jhw4ABsbGyQmZmJlJQUVK5cWdKiiejFeIBMpByF5TXfgwcPMH/+fBkqJKKSUmjDnZSUBFtbW91ttVqNxMTEAsuYmpri0KFDaNu2LdLS0tCyZcuSr5SICsUDZCLlKCyv+aZOnYrRo0fLUCERlRSTwhYQQjw3TaVSPTetdevWOHnyJJYsWYLAwEAsXrz4lYuwtrZ45WX1xda2kqTL60NxaioN4ygNYyiuFx0gR0VFFVgm/wB5woQJUKvVxTpANrTMcls3HGVxDMX1Knldt24d6tevj4YNGxZ7PYaWV6BsbielYQzFvY+UDK2elym04bazs8OZM2d0t5OSkqBWq3W3Hz58iL/++kv3ou3j44OAgIAiFZGSkgGt9vnG/mX08cdNTk5/5WVtbSsVafn8+0itODUVddz6IOVzYWhjMDJSvdaLoz4OkIGiZZbb+qsrS9s6oPx9p9R5vXr1KsLDw/Hzzz/j/v37xV6PoeUV4Lb+qveRWmnYdxalntfN7Oso9JISNzc3HD9+HKmpqcjKykJ4eDjc3d1184UQGD9+POLj4wEAu3fvRuPGjaWrmIheys7ODg8ePNDdftEB8pEjR3S3fXx8EBMTo9caieiZwvK6Z88eJCcno1evXvjss8+QlJSEAQMGyFEqEb2mQhtuOzs7BAQEwNfXF927d4e3tzecnZ3h5+eHixcvwsrKCrNmzcKwYcPQtWtXxMXFYfz48fqonYj+gQfIRMpRWF79/f2xd+9ehIaGYtWqVVCr1QgODpaxYiIqrkIvKQGenQXz8fEpMG316tW6/3fo0AEdOnQo2cqIqMj+foCcm5uL3r176w6Q/f394eTkpDtAVqlUqFOnDmbMmCF32URl0qvklYhKh1dquIlIOXiATKQcheU131tvvYUDBw7oqywiKmH8aXciIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIimYSFhcHLywseHh7YsGHDc/P37dsHHx8fdOnSBZMmTUJOTo4MVRLR62LDTUREJIPExEQEBQUhODgYoaGhCAkJQWxsrG5+ZmYmZs6ciZ9++gm///47srOzsW3bNhkrJqLiYsNNREQkg2PHjsHV1RWWlpYwNzeHp6cn9uzZo5tvbm6OAwcOwMbGBpmZmUhJSUHlypVlrJiIistE7gKIiIjKoqSkJNja2upuq9VqREVFFVjG1NQUhw4dwoQJE6BWq9GyZcsir8fa2uK1ay1ptraVJF1eH8riGIp7HykZWj0vw4abiIhIBkKI56apVKrnprVu3RonT57EkiVLEBgYiMWLFxdpPSkpGdBqn1/Xi+ireUlOTn/lZW1tKxV5eX2Qcgz595FacWoytOeiKPUYGalkOwDlJSVEREQysLOzw4MHD3S3k5KSoFardbcfPnyII0eO6G77+PggJiZGrzUSUclgw01ERCQDNzc3HD9+HKmpqcjKykJ4eDjc3d1184UQGD9+POLj4wEAu3fvRuPGjeUql4heAy8pISIikoGdnR0CAgLg6+uL3Nxc9O7dG87OzvDz84O/vz+cnJwwa9YsDBs2DCqVCnXq1MGMGTPkLpuIioENNxERkUx8fHzg4+NTYNrq1at1/+/QoQM6dOig77KIqITxkhIiIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpLQKzXcYWFh8PLygoeHBzZs2PDc/IiICHTr1g1du3bFyJEj8ejRoxIvlIheDfNKRERkWAptuBMTExEUFITg4GCEhoYiJCQEsbGxuvkZGRkIDAzEqlWrsGPHDjg4OGD58uWSFk1EL8a8EhERGZ5CG+5jx47B1dUVlpaWMDc3h6enJ/bs2aObn5ubi8DAQNjZ2QEAHBwckJCQIF3FRPRSzCsREZHhKfSXJpOSkmBra6u7rVarERUVpbttZWWl+xWsp0+fYtWqVRg0aFCRirC2tijS8vpga1tJ0uX1oTg1lYZxlIYxFJc+8goYXma5rRuOsjgGIqLCFNpwCyGem6ZSqZ6blp6ejpEjR8LR0RE9evQoUhEpKRnQap9fz8voY2eYnJz+ysva2lYq0vL595FacWoq6rj1QcrnwtDGYGSkeq1mVh95BYqWWW7rr64sbeuA8vedr5tXIio7Cr2kxM7ODg8ePNDdTkpKglqtLrBMUlISBgwYAEdHR8yZM6fkqySiV8K8EhERGZ5CG243NzccP34cqampyMrKQnh4ONzd3XXzNRoNhg8fjs6dO2PKlCkvPJtGRPrBvBIRERmeQi8psbOzQ0BAAHx9fZGbm4vevXvD2dkZfn5+8Pf3x/3793H58mVoNBrs3bsXANCgQQOeOSOSAfNKRERkeAptuAHAx8cHPj4+BaatXr0aAODk5ITo6OiSr4yIioV5JSIiMiz8pUkiIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiIiIpIQG24iIiIiIgmx4SYiIiIikhAbbiIiIpmEhYXBy8sLHh4e2LBhw3PzIyIi0K1bN3Tt2hUjR47Eo0ePZKiSiF4XG24iIiIZJCYmIigoCMHBwQgNDUVISAhiY2N18zMyMhAYGIhVq1Zhx44dcHBwwPLly2WsmIiKiw03ERGRDI4dOwZXV1dYWlrC3Nwcnp6e2LNnj25+bm4uAgMDYWdnBwBwcHBAQkKCXOUS0WswkbsAIiKisigpKQm2tra622q1GlFRUbrbVlZW6NChAwDg6dOnWLVqFQYNGlTk9VhbW7x+sSXM1raSpMvrQ1kcQ3HvIyVDq+dl2HATERHJQAjx3DSVSvXctPT0dIwcORKOjo7o0aNHkdeTkpIBrfb5db2IvpqX5OT0V17W1rZSkZfXBynHkH8fqRWnJkN7LopSj5GRSrYDUF5SQkREJAM7Ozs8ePBAdzspKQlqtbrAMklJSRgwYAAcHR0xZ84cfZdIRCWEDTcREZEM3NzccPz4caSmpiIrKwvh4eFwd3fXzddoNBg+fDg6d+6MKVOmvPDsNxEpAy8pISIikoGdnR0CAgLg6+uL3Nxc9O7dG87OzvDz84O/vz/u37+Py5cvQ6PRYO/evQCABg0a8Ew3kQKx4SYiIpKJj48PfHx8CkxbvXo1AMDJyQnR0dFylEVEJYyXlBARERERSYgNNxERERGRhNhwExERERFJiA03EREREZGE2HATEREREUmIDTcRERERkYTYcBMRERERSYgNNxERERGRhNhwExERERFJiA03EREREZGE2HATEREREUmIDTcRERERkYTYcBMRERERSYgNNxERERGRhNhwExERERFJiA03EREREZGE2HATEREREUmIDTcRERERkYTYcBMRERERSYgNNxERERGRhF6p4Q4LC4OXlxc8PDywYcOGly43ceJEbN26tcSKI6KiY16JiIgMS6ENd2JiIoKCghAcHIzQ0FCEhIQgNjb2uWWGDx+OPXv2SFYoERWOeSUiIjI8hTbcx44dg6urKywtLWFubg5PT8/nXqjDwsLQvn17dO7cWbJCiahwzCsREZHhMSlsgaSkJNja2upuq9VqREVFFVjm008/BQCcPXu2WEVYW1sU635SsrWtJOny+lCcmkrDOErDGIpLH3kFDC+z3NYNR1kcAxFRYQptuIUQz01TqVQlWkRKSga02ufX8zL62BkmJ6e/8rK2tpWKtHz+faRWnJqKOm59kPK5MLQxGBmpXquZ1UdegaJlltv6qytL2zqg/H3n6+aViMqOQi8psbOzw4MHD3S3k5KSoFarJS2KiIqHeSUiIjI8hTbcbm5uOH78OFJTU5GVlYXw8HC4u7vrozYiKiLmlYiIyPC80hnugIAA+Pr6onv37vD29oazszP8/Pxw8eJFfdRIRK+IeSUiIjI8hV7DDQA+Pj7w8fEpMG316tXPLTdv3rySqYqIio15JSIiMiz8pUkiIiIiIgmx4SYiIiIikhAbbiIiIiIiCbHhJiIiIiKSEBtuIiIiIiIJseEmIiKSSVhYGLy8vODh4YENGza8dLmJEydi69ateqyMiEoSG24iIiIZJCYmIigoCMHBwQgNDUVISAhiY2OfW2b48OHYs2ePTFUSUUlgw01ERCSDY8eOwdXVFZaWljA3N4enp+dzjXVYWBjat2+Pzp07y1QlEZWEV/rhGyIiIipZSUlJsLW11d1Wq9WIiooqsMynn34KADh79qxeayOiksWGm4iISAZCiOemqVSqEl+PtbVFiT/m67K1rSTp8vpQFsdQ3PtIydDqeRk23ERERDKws7PDmTNndLeTkpKgVqtLfD0pKRnQap9v7l9EX81LcnL6Ky9ra1upyMvrg5RjyL+P1IpTk6E9F0Wpx8hIJdsBKK/hJiIikoGbmxuOHz+O1NRUZGVlITw8HO7u7nKXRUQSYMNNREQkAzs7OwQEBMDX1xfdu3eHt7c3nJ2d4efnh4sXL8pdHhGVIF5SQkREJBMfHx/4+PgUmLZ69ernlps3b56+SiIiCfAMNxERERGRhNhwExERERFJiA03EREREZGE2HATEREREUmIDTcRERERkYTYcBMRERERSYgNNxERERGRhNhwExERERFJiA03EREREZGE2HATEREREUmIDTcRERERkYTYcBMRERERSYgNNxERERGRhNhwExERERFJiA03EREREZGE2HATEREREUmIDTcRERERkYTYcBMRERERSYgNNxERERGRhNhwExERERFJiA03EREREZGE2HATEREREUmIDTcRERERkYTYcBMRERERSYgNNxERERGRhNhwExERERFJiA03EREREZGE2HATEREREUmIDTcRERERkYTYcBMRERERSYgNNxERERGRhNhwExERERFJiA03EREREZGE2HATEREREUmIDTcRERERkYReqeEOCwuDl5cXPDw8sGHDhufmX7lyBb169YKnpyemTJmCvLy8Ei+UiF4N80qkHMwrUdlQaMOdmJiIoKAgBAcHIzQ0FCEhIYiNjS2wzPjx4/H1119j7969EEJg8+bNkhVMRC/HvBIpB/NKVHaYFLbAsWPH4OrqCktLSwCAp6cn9uzZg9GjRwMA7t27h6dPn6JRo0YAgJ49e2LZsmUYMGDAKxdhZKQqcuFqqwpFvk9RFLWm0jCG4txH6jEA0j8XhjSG4jxnf6ePvBanTpMqtkVavqj0sa1LPQZA+m29NIwBMJztqbTm1da8apGWLw6pt5PSMAZA+nHoY99pSGN43cy+jkIb7qSkJNja/m/nplarERUV9dL5tra2SExMLFIRVlYVi7Q8AKyZ2rHI9ykKa2sLSZcHDG8MxbmP1GMApH8uDHEMxaWPvAJFz+w7o1cUeR1FoY9tXeoxANJv66VhDIBhbk/FYah5/d5nTpHXUVRSbyelYQyA9OPQx77TEMcgh0IvKRFCPDdNpVK98nwi0h/mlUg5mFeisqPQhtvOzg4PHjzQ3U5KSoJarX7p/OTk5ALziUh/mFci5WBeicqOQhtuNzc3HD9+HKmpqcjKykJ4eDjc3d118+3t7VGuXDmcPXsWALB9+/YC84lIf5hXIuVgXonKDpV40XtW/xAWFoaVK1ciNzcXvXv3hp+fH/z8/ODv7w8nJydER0dj6tSpePLkCerXr49vvvkGZmZm+qifiP6BeSVSDuaVqGx4pYabiIiIiIiKh780SUREREQkITbcREREREQSYsNNRERERCQhNtxERERERBJiw01EREREJCE23P+VmZkpdwmvLCsrS+4SnmOINcnp0KFDOH78uNxllFpKyitgmPkwxJrkxMxKS0mZNcRsGGJNclJiXtlwA9i7dy8++ugjZGRkyF1KocLDw+Hj44PY2Fi5S9ExxJrkdPDgQXz55Zfw9/fHkSNH5C6n1FFSXgHDzIch1iQnZlZaSsqsIWbDEGuSk1LzWuYb7gMHDmDJkiXo378/LCws5C7nXx04cABBQUFwdnZGYGAgbty4IXdJBlmTnA4dOoRZs2Zh9erV2LhxIxYvXow//vhD7rJKDSXlFTDMfBhiTXJiZqWlpMwaYjYMsSY5KTqvogzbv3+/aNeundi+fbtuWk5OjowVvdzBgwdFhw4ddLVu27ZNdO/eXVy/fp01GYj9+/eLDz74QHzwwQciOztbCCHE0aNHRfv27cXhw4dlrk75lJRXIQwzH4ZYk5yYWWkpKbOGmA1DrElOSs9rmW24jx8/Ltq2bSu2bdumm5b/BMbFxYmbN2/KU9gLHDhwQDRr1kwEBAQUmL5w4ULx9ddfsyYDcPDgQdG+fXsREREhdu/eLbp37y7u3r0rhBBi3bp1IjAwUOTm5spcpXIpKa9CGGY+DLEmOTGz0lJSZg0xG4ZYk5xKQ17L5CUliYmJGD9+PEaPHo3u3bsDAHJycmBmZoabN2/C09MTERER8hb5X/v27cOCBQswfPhwpKWlITg4GACQnp4OU1NTVK1a1WBqysjIkK0mOR0+fBjjx4/HmDFj0L59e3Tq1Anly5fHF198gd9++w1z5syBjY0NTExM5C5VkZSUV4CZVQJmVlpKyizzavhKS14NuzqJ5ObmolWrVujZsyc0Gg00Gg3MzMwQFxeHvn37YsiQIfj000/lLhMpKSmYM2cORo0ahT59+qBjx44YOXIkrly5AlNTU5w+fRojR44EAAghoFKpZKlp1KhRiI6Ohkajwfnz5zFixAjJ6zAk1atXx4wZM+Dl5aV7HmxsbFChQgVkZ2dj4sSJGDhwoG55fT1XpYVS8gows0rBzEpLKZllXpWhtORVJYQQchehb3FxcRg7dixmzJiBRo0a6ab16dMHH3zwAdq0aQMnJyfUrl1b9icuMTERdnZ2yMvLg4mJCZKSkrB9+3ZUqFABe/fuhZ+fH8qXL4/mzZvrrdZ/1nT//n3s2LED5ubmcHZ2hrOzM7RaLYyMjGT/++lbbm4u7t27h5EjR2LVqlXIzs7GX3/9hdu3b+Ptt9/Wne0pa3+X16GkvALMrNIwsyVPSZllXpVFyXktkw03AERGRmLRokUYOHAgjIyM8J///AcNGzZE48aN8fPPP6Nu3boYNGgQ3N3dZa0zf6MRQiA3NxdmZma4d+8efvjhB8TFxcHLywu//PILJk+ejFatWum9Jq1WC2NjY928ffv2YePGjbCyskLXrl3RunVrg9zwpXTq1ClMnToVvXv3xrlz53Dz5k3MmDED3377LXx8fDBgwAC5S1QcpeQVYGaViJkteUrJLPOqPErNa5m8pAQA2rRpA2NjYxw6dAjly5dHp06dUKdOHZw5cwaLFy+GWq3Gp59+Cjs7Ozg4OMhWZ36IVCoVzMzMAADz58+HmZkZ1q9fDwBQq9XYuHEjmjdvDlNTU8mD9/eajI2NdWHfuHEjtm7dikGDBqFq1apYuHAhypUrB1dXV0nrMRT5f4edO3eievXqMDU1RfPmzZGVlYX79+9jyZIlWLx4Mbp06YJKlSrByKhMfoSiWJSSV4CZVRJmVjpKySzzqhxKz2uZbbiFEGjVqhVatWqFa9euYd++fbhw4QI++ugjODk5QaPRwNjYGOnp6XKXWkBqairS0tKwYsUK3TRLS0sYGRnBzMwMWq1W70e6KpUKUVFRWL58OZYtW4amTZsCAGJjYxEdHV1mdgb5f/fPPvsM27dvx969e7Fw4UIMHjwYQ4cOxY8//ojWrVujSpUqBe5X1s5OFIdS8wows4aMmZWOUjPLvBoupefVsNp/Pfr7H9/Y2BhxcXH48MMP4eTkhLS0NAQEBKBRo0ZwcnJCVFQUbt68KWO1/6NSqZCTk4Nr164BAB4/foxJkybBzc0Nt2/fxsaNG3Hu3DkAzzYyfcnNzUWXLl10OwIAOHv2LHJzcwH872dpy8IVTG+99RY+/vhjVK1aFVeuXAEALFmyBB07doS1tTXu3LmD3bt3636WVqVS4dGjR3KWbPCUmleAmVUCZrbkKTWzzKvhU2pey+w13P+UkZEBCwsLpKamYsaMGbCzs0O/fv2QmZmJqVOnokKFCggICECzZs3kLhX79+/H4sWL0bp1a/z+++/o0KEDxo0bh44dO8LX1xfbtm3T6/VmAHD58mV88cUXmD17Nho2bIiFCxfi+PHj2Lx5MxYsWICnT5+iY8eOaNu2rcEcbUrtypUrmDx5MoyNjdG5c2d06dIFO3bswOXLl5GdnY27d+/C19cXzs7O2LJlC/r374+aNWvKXbYiKCmvADOrFMysdJSUWeZVGZSW1zJ7Sck/mZubIy8vD1OmTIGtrS369++Px48fY82aNXjnnXfQp08fTJkyBQsXLtR96loOQgi0b98e1tbWyMjIgLu7O959912Ym5vD1tYWXl5eaNq0KVasWIHmzZvDzMwMDx8+BPDsbTGp1K9fHxMnTsTs2bNRrVo1xMfHY+XKlTh9+jSOHj2KNWvWYOTIkahYsaJB7FD1oV69eli5ciUSExNhYWGB27dv486dO+jYsSO8vLzw5MkTjBgxApcvX4aJiQmys7PlLlkxlJJXgJlVEmZWOkrJLPOqHErLKxvu/zIyMoKRkRG+/PJLqFQqpKWlYf369bCzs8OUKVMAAO+//z7++OMP2Nrawt7eXpajyPz15e+QHj9+jL59++I///kPevbsieDgYDRp0gStWrXC06dPsWPHDvz111945513MGTIEMnqEkKgbdu2qFevHszNzWFkZISTJ0+ifv36qFChAiwsLODp6YkLFy6gWbNm0Gq1OHLkCN577z1YW1tLVpfc1Go11Go1ACA4OBjGxsbw8vICAJw5cwbx8fGoV68efH19YW9vL2epiqKUvALMrNIws9JQSmaZV2VRUl7L7DXcL1OnTh2o1Wr89NNPsLCw0O0Ivv/+e+zevRtZWVlYtGgRDh06BJVKBY1GI2u9Go0Gtra2qFChAlq2bIk//vgDLi4u6N+/P44cOYKpU6ciKSlJ0h0B8L+d1BtvvIHKlSvj4sWLWLlyJd588020a9cO4eHhqFevHjp16oRffvkFEyZMwLp165CamippXYYiLy8PRkZGeO+99wAAUVFRuHjxItq2bYtx48YhOjoaGzZskLlK5VFaXgFmVimYWWkoLbPMqzIoIa88w/0ClSpVwtixY1GnTh0AwIoVK7Br1y6sXbsWzs7O2L59O7799ls0adIEFhYWSE5ORnp6OmrVqqX3Wq2srDBo0CCMGTMGjRs3hr29ve4rco4cOYKOHTtCrVbjyZMnqFixot7OGNja2iI9PR2JiYkwMzPD7t278eOPPyIvLw8xMTE4cOAAvvrqK9StW7dMXG9mYmKCnj17YuLEiYiKikJcXBwaNGiAMWPG4OTJkxg1ahQWL15c4D5l4e9SEpSUV4CZVQpmVjpKyizzqgxKyCs/NFmI7777Dr/++iuWLVuGhg0b4s6dOwgJCYFKpcKXX36JnJwcnD9/HrGxsQV+WlRf8jeY06dP4+bNm7C3t4eFhQXWrVuHypUrY/r06bh//z727t2Lxo0bw8nJSW8bWUREBFasWIFatWqhTp066N69O7Zv346TJ0+iW7duAIBWrVrBysqqTOwQAODu3btYtWoVNBoNpk2bhtOnT8PPzw/Lli2Dh4cHnj59ipSUFNnf+lIqQ88rwMwqDTMrLUPPLPOqLAadV0H/6tSpU+L8+fNCCCHu3Lkjli9fLr7++mtx9epVkZCQIBo2bCjS09OFEEJcu3ZNpKamylitEHl5eWLcuHFi3rx5QqvVitTUVDFp0iTh7OwsBg0aJA4fPqyXOrRarRBCiKtXr4r4+Hhx8+ZNsXbtWjFo0CDx559/CiGE2L59uxgxYoQ4d+5cgfuUdikpKSIuLk5cu3ZN1KtXT4SHhwshhMjKyhL79u0Tn3/+ue5vREWjtLwKwcwqATMrHaVllnk1fIaaV17D/S+EEHj//ffRqFEj3VF3SkoKBg4ciKpVq6JXr14YMGAAHj9+jIULF2L9+vU4deqUrDXHxcXh5MmT6NGjB3Jzc/Hzzz8jJycHW7duxbRp0zBnzhzExMRIXkf+kXTdunXx5ptv4saNG/j1118xZ84cODs7Y8+ePVi+fDlyc3OxZMkSREZGlomjbwCoWrUqqlevjqtXr8LLywseHh7IysrCiRMncOjQIdSrVw9OTk4F7iP4RlShlJhXgJlVAmZWGkrMLPNq+Aw1r7yG+1/8fePUaDRIS0vDRx99BGtra3h5eWHgwIHw9/fH999/jzVr1mDIkCHw9PTE5cuXYWtrC1tbW73WK4RA7dq1sXTpUqSkpGDt2rW4cuUKVqxYAbVajZycHNjb2yMnJ0evdQGAhYUFzM3NYWVlhdDQUISHhyMgIABdunTBmTNnsHDhQjg5OaFq1aplZqfw3nvvYcGCBdi2bRtu3bqFBw8ewMbGBj4+Ppg+fTrKlSuHmjVrYsCAAWXmb/I6lJZXgJlVGma2ZCkts8yrshhaXnmG+xXVqFEDX331FaytrdGmTRsMGjQI/v7+2LlzJ+Lj4zFq1CjdBydMTEywZ88evdeYv8HY29vj7NmzOHHiBHr37g21Wo2MjAxcuHAB5ubmsLGxQUxMjO4XmvRxZNesWTPMmjULZ8+exYIFC9CsWTN06dIFAPDo0SO88cYbsLa21o2hLJwdql69On7++Wfcvn0bkZGRqFevHjp37oxFixYhOTkZHh4e+O2337B582a5S1UcJeQVYGaVhpmVjhIyy7wqi6HllWe4i6BixYq4cOECBg0ahNGjRyMsLAz79+9HixYt8OGHH+LQoUMICgrCZ599JtsHsvLy8rBp0yacO3cO48ePx5o1a1C+fHncu3cPsbGxcHJyQmZmJoYMGYLy5ctj1qxZcHNzk7yurKwsXL58GWFhYbCysoKrqysA4NatW/jrr79QrVo1PHnyBKtXr0bLli3RtGnTMvEhjxo1amDAgAGoWLEiBg4cCD8/P9jb2+s+Te3r64uzZ8+ib9++yMnJwYEDB1C3bl3Url1b5soNnxLyCjCzSsPMSkcJmWVelcWQ8spvKSmCv2+c9+7dQ2BgIHr06AEvLy+cOnUKO3bsgKWlJcaMGYNy5co9dx99SUhIgJGREezs7BAZGYmIiAjY29ujdu3aqFu3Lr744gs0bNgQ3t7eWLx4MVatWoVKlSpJWlNGRgbGjRsHb29vvP322/jqq6/QsGFD5OTkIDc3F2PHjsWGDRsQEhICJycnjBgxAq1bt5a0JkOTnZ0Nf39/rFy5EsCzbefcuXO4ePEicnJycPToUZiammLKlCn8OelXoJS8AsysUjGzJUspmWVelUnuvPIMdxH8PdSpqamIj4+Hs7Mzzp8/j507d6JKlSoYMmQI1q9fD61Wi3r16qFly5Z6r/PNN98E8OxIvE2bNmjZsiVMTEwQHR2NSZMmwc3NDePHjwcATJ48Gbm5ubh58yZq1qwp2c7LwsICCxYsQOXKlQEAs2fPxt27d1GtWjW89dZb2LRpE06fPo0jR47g/v37OHTokG5nUBaOwoFnO4MHDx5gz549aNiwISpVqoQmTZqgWrVq2Lx5M65cuYJhw4ahZs2aOHToEKpUqSL7z5YbMqXkFWBmlYqZLVlKySzzqkxy55VnuF9DTEwMrl27hkOHDqFGjRpo27Yt1q9fj9u3b6NHjx5YsWIFFi9eDGdnZ1k3aK1Wi8zMTPTp0wdt2rTBxIkTATz7/tPY2Fikp6cjLS0NI0eORIcOHSSrI/9voNFoYGxsrJv+9ddfIyYmBitXroSVlRUSExNx6dIlpKWloWbNmmjcuHGZ2SFcvXoV06ZNg1qtRoMGDdClSxds374d58+fx9ChQ5Geng4XFxdoNBpcvXoV7u7ucpesGErJK8DMKgkzKx2lZJZ5VQ5Z8yrFdw2WJbGxsWLevHlCo9GIsWPHikmTJunmbdmyRUyaNMlgvp/15s2buv9v27ZNBAQEiBMnToiMjAxx48YN0atXL3Hnzh291hQdHS169Ogh0tLShBBCnDhxQowYMUJMmDBBzJo1S3h6eoqDBw/qtSa5PXjwQKSnp4tbt26JtWvXCl9fX3Hp0iUhxLPnrUePHuLu3bu65cvKd6uWBCXlVQhmVimYWekoKbPMqzLIlVdeUvIatFotateujYkTJyI7Oxvp6elYunQpgGdvNcXHx+PcuXNo166dvIX+V/Xq1XX/v3fvHmrUqAEnJyeYm5vD3t4e5cqVw5MnT/Rak4ODAzZv3gwTExM8fPgQe/fuhZubGzw8PGBnZ4c///wTP/74I5o2bQoLCwu91iYXa2trAMDZs2exYcMGrF27Fm+88QZCQ0OxZcsWfP755wV+JUulUpWZsxOvQ2l5BZhZpWBmpaG0zDKvyiBXXtlwvwYjo2ffqrhhwwaYmZkhOTkZUVFRqFmzJi5fvowzZ85gypQpqF+/Ps6cOQNzc3PUr19fth1t/kaj0WiQmJiIRo0awdzcHBkZGTh8+DDq1KmDqlWrAnj2NUJVqlTRS60mJs82wwoVKiAxMRGdO3eGnZ0dgGdv/2g0GlSsWFHSGgxRjRo1YGdnh9zcXISGhmLz5s0YMWIELC0tMWLECFSuXBkuLi7o168fX7hfgdLyCjCzSsPMliylZZZ5VRZ955UN92vKycnBtWvX0LZtWyxduhQLFiyAmZkZkpKSMGLECLi7u2P79u24e/cuIiIiEBAQgNatW0Or1ep2JvqkUqlgYmICT09PTJ8+HSYmJjhz5ozugxahoaH4888/kZ2djX79+un1zMHjx4/x5MkT3ae5o6KikJaWhubNmyMpKQmpqamoUKECatSoUSbODlWvXh3ffvstLl68iB9//BFTp06FSqXC0KFD4e/vj7p162LRokUAgH79+slcrTIoLa8AM6skzGzJU1pmmVfl0Hde+aHJEhATE4Nx48bho48+wr1793D+/HmMGTMGRkZG2LhxI3x8fNC4cWPEx8fDz88Pq1evhqOjIwAgNzcXpqamstR9+PBhHD58GJUqVcKbb76JmJgY2NjYwMbGBrVr18aECRPw3Xff6WrVh4iICMyfPx8uLi7IyMiAnZ0d8vLyoFKpEBsbiydPnmD06NHw8PDQW01yi4+Px4ULF9C+fXt07NgRU6ZMQceOHQEA586dw+7duzFlyhTd8mVhR/k6lJpXgJlVCma2ZCk1s8yrMugrrzzDXQIcHBywdOlSbNy4ETVr1sTIkSPRoEEDeHt7o1evXrhy5QrOnj2Lhg0bonHjxnj69ClycnJw8eJF3L59G+3bt9cd/eqTu7s7WrRoAVNTU0RFReHkyZPo0qULGjduDABo0KABbt++DUdHR728IAgh0KFDB9jZ2SEuLg5vvfUWYmNjcfv2bTg4OGDy5MlISEjAV199hUaNGsnyU9xyqFatGqpVq4bExEQ0a9ZMtyMAgLCwMKSmpgIAoqOj4ejoyOtDC6HUvALMrFIwsyVLqZllXpVBX3llw11CateujalTp+pu37t3DxUrVkTfvn1ha2uLCxcu4PPPP0etWrVQu3ZtfP3110hISEC7du2Qk5MjW935Xx0UFxcHU1NT3Y4AAJo0aQIXFxcIIfDw4UNYWVlJulPIf1wnJyc4OTkBAE6dOoWMjAy4u7ujfPnysLKyQnp6OvLy8iSpwZAZGRnh+vXr2LFjB7p27YoffvgBd+7cwbRp07Br1y5s2bIFY8aMQaNGjfgCXgil5hVgZpWEmS05Ss0s86ocUueVl5RIaNeuXVi3bh1at26NM2fOQKPRYNWqVbh27RoWLVqEmzdvYuHChXj//fflLhUXL17EpEmTsHjxYly/fh1dunQBAEyYMAEVK1bE9evXMXToUL39MpVWq0VeXh7mzZsHFxcX+Pj4ICcnB2ZmZrh27RqAZ0freXl5sn+wTZ+uXbuGL7/8EtWqVcO9e/cwe/ZsnD9/HidOnICZmRmEEHBxccGQIUPkLlVxlJRXgJlVCmZWOkrKLPOqDJLmtUS+XJCek/+9jdHR0cLT01N8+OGHIicnR5w+fVpMmjRJrF+/Xly5ckXcu3fvufvIJTw8XIwdO1YMGTJE3Lt3T3zyySfCwcFB/PzzzyIuLk54e3uLc+fO6bWmI0eOiLZt24r169eLwMBAIcSz78ls1qyZ2LRpk+jevbuIjIzUa01yS01NFY8fPxZCCLF9+3YxceJEsXPnTpGZmSkePnwounTpUuD7YKlwSsyrEMysUjCzJU+JmWVelUGqvPKSEonkHwk6ODhg9uzZqFWrFq5fv45t27ahRo0a6NOnDy5evIgTJ04gPj4ebdu2RYsWLWQ7ihRCwMPDAy1btoSZmRk2b96M3NxcnDx5EsOHD0eNGjUwdOhQPH36VK91ffDBB5g7dy4uXryIpk2bYtu2bZg9ezZsbGxgaWmJZcuW4fPPP4ejo6Pua45KOysrKwDAr7/+iv3796Nr165o0aIFKlSogLi4OGRmZqJcuXIF7iPXdqUUSssrwMwqCTNb8pSWWeZVOSTLa8kcD9CrWLJkiZg4caIQ4tkvZLVt21aMGTNGHDlyRHh7exvUUeSWLVvEkiVLhBBC3Lp1S/Ts2VM4ODiIixcvivT0dJGeni6E0P8Zg7Fjx4qIiAih0WhE7969xdatW8VXX32l+xWtsuT+/ftiz549IiUlRQghRExMjHB2dhYbNmwQCQkJYteuXWLfvn265eU+u6M0SsqrEMysEjCz0lJSZplXw1fSeZXni2XLqCZNmqBcuXK4fv06QkNDMX78eCQlJcHMzAwLFizAn3/+CY1GI3eZAIC33noLBw4cwJEjR1CuXDkEBQVh+fLlqFChAoYOHYqAgABERETo7UyBEAJarRampqZIT0+HkZERli9fjhUrVuD27duwtLREVFQUbty4oVu+tLOzs4OHhweqVq2Ky5cvo1+/fhg+fDgGDBiAuXPnIjExEatXr0ZwcDAA8GxZESkprwAzqwTMrLSUlFnm1fCVeF4lOjCgf8g/8snMzBTx8fHCz89PCCHE7du3RZ8+fUSrVq3E8uXLC9xHo9Hovc6/i4iIEIMHDxaenp7i0qVLQgghfvzxRzF9+nSRmpoqevXqJc6fP6/XmiIjI4W3t7dYv369CAkJEQkJCeLhw4di8uTJYuzYscLb21scPHhQrzXJSavViuzsbPHBBx+In376SXfd2fDhw8XatWtFQkKC6NmzJ68PLSIl5lUIZlYJmFlpKDGzzKvhK8m88hpuPck/8qlQoQISEhLw4MEDXL16Fe+++y4WL16MrVu3omXLljh37hzi4+Nhb28PFxcXAEBmZibMzc31Wq8QAu3bt0erVq1gYmKCa9euYcWKFWjVqhXCwsKQkZEBBwcHJCcn67Wm1q1bQ6VS4dKlS3jw4AH69u0L4NkvaHXv3h316tXD2LFjUaNGDdSoUUNvtclFpVLBzMwMO3bsQNWqVTFs2DB89tln+Pjjj7Fjxw40aNAAXbp0QbVq1eQuVVGUlleAmVUKZlYaSsss86oMJZlXNtwyqFWrFkaMGIHPP/8cWq0Ww4cPx4cffoi5c+fCzMwMAHD58mUsWrQI6enp+Omnn7Bo0SJYWFjorcb8nVd+PVeuXMHjx4/x3nvvwdnZGX/99Rc+/vhjGBkZITU1FWlpaahdu7bk3yEqhIC7uzvc3d0BAJMmTULr1q3h5eWFsLAwWFlZoVGjRrCxsdHdT8qaDIWVlRW0Wi0qVKiAO3fuwMPDA0uWLIGpqSn69u2rex6p6JSQV4CZVRpmVjpKyCzzqiwlkVc23DIQ//20cr169fDo0SPY29sjODgYb7zxBvr06YO6desiOjoagYGBKFeuHNq3by93yXjrrbfwww8/oFOnTsjLy8OZM2fQuXNnrF+/Hvfu3cORI0cwfvx4uLu7Q6PR6L7sv6T9PdQajQZmZmaoWrUqmjdvjo0bN6JcuXIYNmwYzp49i/T0dKjVajRr1qzU7xBUKhVUKhVGjBiBqVOnIikpCVWqVEF8fLzuBw6oeJSYV4CZNXTMrHSUmFnm1bCVRF75oUkZ5G+Ub731Ft577z1YWlri3r17qF+/vu4tmsuXL+Pq1avw8fFB9+7d9X627J+aNm2KyZMnY9asWbCystId3Z4/fx4mJiZYunQpFixYgBs3buh2BElJSZLWZGxsDHd3d8yfPx9bt25F+fLlkZCQgMqVK2Pfvn24ffs25s2bh0OHDumO3Es7BwcHzJs3DxUrVkT79u3Rrl07uUtSPCXmFWBmlYKZLXlKzCzzqgyvk1ee4ZaZEALp6em4f/8+7OzsYGpqimPHjmHXrl0ICAhAly5dUKFCBbnLBAC0bt0ajo6OsLKywqpVq3Dx4kX07NkTv/76K4yNjdGmTRvdTuv69ev46aefMGPGDMmOxIUQ6NChA8zMzBAeHo6mTZsiLS0NZmZmOHnyJKZOnYpmzZph5cqVaNKkCSwsLHD27FnY2NigevXqktRkCGrXro3atWvLXUappKS8AsysUjCz0lFSZplXZShuXnmGW2YqlQqVK1fGwIED8Z///AfTpk1DYGAgOnXqhG7duhnMjiCfWq2GmZkZHj58iKNHj6J58+bQarVITExEjx49oFarMWnSJJiZmWHatGmS7QiA/53FcHd3x+zZs/Hpp58iJiYGd+/ehZ+fH44ePQozMzM0bNgQFhYWyMjIgEqlQlxcnGQ1UemmtLwCzCyVbUrLLPNaerHhNgBCCLRr1w5jx47F1atX0b9/f3h5eaFy5cpyl/acvwdw7969uH37Nh48eICLFy+idu3amD59Oq5evQoTExNkZWUB0M/3dQoh8OjRI5w/fx43btyAp6cnYmJi8MYbb2D06NE4duwYvL298e6776J169aS10Oll5LyCjCzRErKLPNaeqlEWbjoRkHyv6Tf0tJS7lJeKv/DEREREYiIiEBGRgaWLFmCb775BnFxcZg3bx4SExOxd+9edOvWDe+++67ePlARGRmJxYsXw9fXFzNmzMD+/ftx6dIlzJ8/HyNHjoSrqyu0Wi3efPPNUv8hD5KeEvIKMLNE+ZSQWea1dGLDTcWSHyStVgsjIyMEBgbqdgT379/HTz/9hJiYGLzzzjsYOHCgXo5482s6efIkbty4AQcHB5ibm2PkyJGYMGECXF1dsX79ety6dQu+vr5wcnLiDoHKDGaWSDmY19KHl5RQseQHyMjICNHR0bh+/Tq++eYbJCYmYufOnbC2tsbmzZvx9ddfY/78+Xq5piu/pubNm6N///548uQJPvnkE4wfPx6NGzfGtm3bEB4ejsqVK2PevHmIjIzkjoDKDGaWSDmY19KH31JCr83R0RFr1qzBpUuXsGXLFlSsWBEjRoxA5cqVER8fj9zcXGg0Gr3XZWpqimnTpqFx48bYvXs3Ll26hLFjx6J9+/Y4evQoVq1aBRcXF1SpUkXvtRHJiZklUg7mtXRgw00lwszMDJUqVYKZmRn8/PxgbW2NP//8E0uXLsWAAQNQq1YtPH36FE+ePIG1tbXkbzMJIeDq6oqcnBysX78eZ8+eRZ8+fdCmTRsAwJYtW2BjY8MdAZVZzCyRcjCvysdruKlE5ebmwtTUFBcuXMCyZcvQsWNH2NjY4Nq1azhz5gxycnIwePBgvf64w5UrVxAfH6/7NbHp06dDq9ViwIABsLGxwc2bN2FlZYW6devyejMqc5hZIuVgXpWLDTeVKI1Gg8zMTIwYMQJubm5o2bIltmzZAnt7ezg4OECtVmP69On4+uuvZfn54q+++goA0KtXLxw5cgR37txBWloaEhISMGnSJH6dEZU5zCyRcjCvysVLSqhEGRsbo1KlSpg/fz4sLS0RExODlJQU9O/fH46OjgCAihUrYtOmTbCzs4NardZbbVevXkV6ejo+++wz3L17F5mZmfD29kbbtm1x/fp1BAYGolq1aqhbt67eaiKSGzNLpBzMq3LxW0qoxGm1Wtja2iI+Ph5Xr17Fm2++qdsRhIWFQQiB999/X++/8PXuu+/im2++gbOzM2JiYlCuXDm4ubkBANLS0hAXF4cDBw7gyZMneq2LSG7MLJFyMK/KxDPcVOKMjIzw6NEj7Ny5Ey4uLli/fj3OnTuHI0eO4MmTJ+jVqxfat28Pc3Nzvdf28OFDREVF4f79+/D09ES5cuUQExODyMhI9OzZEx07dkTFihV1y+dfb1aWrzuj0o+ZJVIO5lWZ2HCTJLRaLf744w94eHhg+fLlmDRpErRaLT755BO0atVKlh0BAJw9exY3b97EwIEDMXPmTERHR+P48eNo1aoVWrdujZo1awIA8vLyYGJiotsBZGdno3z58rLUTKQPzCyRcjCvysMPTZJkoqOjMX78eNSvXx8PHz5Enz594ObmJtuOAABu3ryJL7/8EnPnzsX58+exe/duODk5oU+fPqhRowYAFDjSjoiIQHJyMq5fv44JEybAzMxMttqJpMbMEikH86osbLhJUomJibh79y5q1aqFypUrw9jYWO6SEB0djcWLF8PIyAju7u5o1aoV3nnnHd18IQTi4+MxYcIEvPPOO3Bzc4OLiwuMjY1Rrlw5VK1aVcbqiaTFzBIpB/OqHGy4qUzKyMgAAFhYWOim5b/FdfHiRSxduhRvvvkmZs+eDQC4f/8+1q5di/379+OHH35A7dq1ZambqKxiZomUg3l9HhtuIjy7Hs7IyAjZ2dn49NNP0bBhQ4wbNw7As6862rdvH5KTk2Fubo4LFy5g1qxZpXKHQKQUzCyRcjCv/FpAIgDPPvUNAIGBgVCr1QV2BBEREYiLi0OPHj0wYcIE9OnTB2PGjMHdu3flLJmoTGNmiZSDeWXDTaTz9OlT2NjYYPLkyQD+tyO4efMmBg0ahIYNGwIAevTogYYNG+LEiRNylktU5jGzRMpR1vPKrwUk+q/z588jISEBFhYWuH37Nnbu3InHjx9jwIABcHZ2BgDk5ubCxMQEDx8+1P2oQP5bZUSkX8wskXKU9bwqfwREJaRWrVr45JNPUK5cOeTm5iIkJAR169aFi4sLNBoN8vLyYGpqivPnz+PcuXN4++23kZGRgV27duHOnTtyl09U5jCzRMpR1vPKhpvov+zs7NCgQQMAQO3atTF37lyEhYXh/PnzMDY2homJCS5cuIBPPvkEgwcPhrOzMyIiIrB27VoMGjQIN27ckHkERGULM0ukHGU9r/yWEqIXyP9i/sjISCxYsABt2rRBSkoKDh8+jE8//RSffPIJtm/fjiNHjqBv37548OABfvnlF8ybNw+1atWSu3yiMoeZJVKOsphXNtxEhTh//jyuXLkCU1NT1K5dG40bN8bWrVuxefNmpKamIjw8HADw66+/Yt26dVi6dGmp+zojIiVhZomUo6zklQ03URFkZGQgIiICf/zxB9q3b4/o6Ghcu3YNy5Ytg6mpKX799VdUqFABbdu2RcWKFeUul6jMY2aJlKM055UNN9ErEEIgLy8PoaGh+P333zFw4EB06NABADB//nxER0djzZo1MDIyQkJCAn744QcMHjwYNWvWlLlyorKJmSVSjrKQV35okugVqFQqmJqa4u2338alS5dgb2+vmzdx4kQ4Ojri0qVLAIA333wTTZo0wZdffqn4D3kQKRUzS6QcZSGvPMNN9IryP+Rx6NAhfPvtt5gyZQqaNGlSYJm/f1/ozJkzER4ejnXr1in2Qx5ESsbMEilHac8rz3ATFVHr1q0xduxYBAYG4ty5cwXm5e8IDh48iJMnT6Jly5aKOwonKm2YWSLlKK155S9NEr0ilUql+3/r1q0hhMCBAwfw7rvvwsLCQjfv4MGDmD17NsaMGYPu3btj+/btiIiIwIABAwosR0TSYmaJlKO055WXlBAVQ/5bX48fP0blypV10w8ePIg5c+Zg9OjR6N69u256WloaTp06hQ8++MCgdwhEpRUzS6QcpTGvvKSEqBjyj8T/viOIiIjAvHnzCuwINBoNAMDExAS///47EhMT9V4rETGzREpSGvPKhpuoBKSmpmLOnDn49NNPdTsCIQSMjY0hhEBkZCTq1aunyC/rJyqNmFki5SgNeeU13EQloGrVqti0aRPs7OwAPNsR5L8llpOTg2PHjj33aWsikg8zS6QcpSGvvIabqITkf12RRqOBsbGxbvro0aPx+PFjrF27tsCHQohIXswskXIoPa88w01UQvK/rsjY2BhZWVlITEzE8uXLce/ePWzZsgUqlarAd4gSkbyYWSLlUHpe2XATlaB79+4hPj4emzZtgq2tLczNzbF161aoVKrnjsqJSH7MLJFyKDmvvKSEqAQ9fPgQu3btQqVKlfDBBx+gatWqAGDQR91EZRkzS6QcSs4rG26iEpaXlwcTk/+9eZT/wQ4iMkzMLJFyKDWvbLiJiIiIiCRk2OffiYiIiIgUjg03EREREZGE2HATEREREUmIDTcRERERkYTYcBMRERERSej/AayUd3M2aDjFAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 359 }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "colab": { - "base_uri": "https://localhost:8080/", - "height": 1000 - }, - "id": "89kqC0Jj9D6O", - "outputId": "a64294f5-205a-4dc9-8e8a-fcd245a7182f" - }, - "outputs": [], - "source": [ - "display.HTML(html_modelcard)" - ] + "id": "yUaAejwa6P3F", + "outputId": "c6d84d35-fdde-47be-af88-69881afaa163" + } + }, + { + "cell_type": "code", + "execution_count": 96, + "source": [ + "difference_df = pd.DataFrame.from_dict({\n", + " \"Unmitigated\": metricframe_unmitigated.difference(),\n", + " \"Postprocessing\": metricframe_postprocess.difference(),\n", + " \"Postprocessing (DET)\": mf_deterministic.difference(),\n", + " \"Reductions\": metricframe_reductions.difference()\n", + "}\n", + ")" + ], + "outputs": [], + "metadata": { + "id": "pIGS8f2M2Afu" + } + }, + { + "cell_type": "code", + "execution_count": 97, + "source": [ + "difference_df.T" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
selection_ratefalse_negative_ratebalanced_accuracy
Unmitigated0.1370.2420.061
Postprocessing0.0470.0660.041
Postprocessing (DET)0.0610.0480.026
Reductions0.0700.1500.050
\n", + "
" + ], + "text/plain": [ + " selection_rate false_negative_rate balanced_accuracy\n", + "Unmitigated 0.137 0.242 0.061\n", + "Postprocessing 0.047 0.066 0.041\n", + "Postprocessing (DET) 0.061 0.048 0.026\n", + "Reductions 0.070 0.150 0.050" + ] + }, + "metadata": {}, + "execution_count": 97 + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 172 }, - { - "cell_type": "markdown", - "metadata": { - "id": "1j9WzWcCVbZV" - }, - "source": [ - "# Discussion and conclusion" - ] + "id": "3pBkP7QCDVs6", + "outputId": "8bec1174-4a94-4cf6-8cab-634e179c0ea7" + } + }, + { + "cell_type": "code", + "execution_count": 98, + "source": [ + "difference_df.T.plot.bar(subplots=True, layout= [1,3], figsize=(12, 5), legend=False, rot=-45, position=1.5);" + ], + "outputs": [ + { + "output_type": "display_data", + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAuIAAAGICAYAAAAEZIWeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABn1ElEQVR4nO3deVhU5f8+8HtYBkVUEGfmk1SaS6AJbqREiZoiiiCISypJVmpqLlGSpJi4horiUmHZnpqSC4RfRcKtzCVFC0pEcRcUUFABWYbh+f3hjylCBZSZMwP367q6LuYsc95Pzj2853DOMzIhhAAREREREemVidQFEBERERHVR2zEiYiIiIgkwEaciIiIiEgCbMSJiIiIiCTARpyIiIiISAJsxImIiIiIJMBGnIjqvblz5+Lll19GRETEfdcfPXoUXl5eeq6qdiUlJeHDDz8EACQnJ2PatGkGUw9RTT1KJu3t7ZGTk6Ojiqr25ZdfIjg4WLLjk2FiI27k1qxZg/nz5z/y/iEhIfjrr78AALNnz8ahQ4dqq7RHwl/OJIXNmzdj48aNCAwMlLoUnUlLS0NmZiYAwNHREatXrzaYeoiI6iszqQsgaR06dAivvPIKAGDRokUSV8NfzqR/o0ePhhAC48ePx4ABA/Drr7+ipKQEOTk58PX1xTvvvFNh++PHjyMsLAxlZWUAgLfeegseHh4oKSlBeHg4jh07Bo1Ggw4dOiAkJARWVlYPPPbRo0cRERGBp556CmfPnkVJSQk+/PBDuLi4PPT5kpKSEBoaCrVajaeffhoZGRkIDg7G888/j8WLF+PPP/9EQUEBhBBYuHAhWrRogdWrVyMvLw8ffPABfH19sWDBAvzwww/o1asXdu/eDYVCAQAYMWIE3n77bbzwwguPNJ5FixbB0tISd+/exZYtW7B06dIq6/noo4+wd+9eREZGQq1Wo0GDBpg5cya6dOny+P/AVGfdvXsX06ZNw6VLl9CkSRPtSan58+fj7t27yMrKgoODA1auXAkLC4sK+4WGhuLixYu4ffs2GjVqhPDwcLRu3RpjxoxB586dceLECVy7dg3dunXDkiVLYGJign379mHlypUoKyuDpaUl5s2bBwcHB5w4cQLh4eEoLCyETCbD1KlT0adPH6jVaixcuBCHDh2Cra0tbG1t0bhx4yrHtXbtWiQkJKC4uBiFhYWYOXMm3N3dUVpaimXLlmH//v0wNTVFly5dMHfuXJiYmNx3+WeffYbc3Fztya01a9ZoH48ZMwZNmzbF+fPnMWrUKDg6OmLZsmUoKSlBdnY2XF1dsXjxYgC477j37duHtLQ0LF++HACQmJiIBQsWIDo6upb/lesBQQYlPz9fTJ06VQwePFj4+vqK2bNnC41GI/bs2SOGDRsmfHx8xCuvvCJOnDghhBBi9erVYt68eUIIIa5fvy4mT54shgwZIry8vERkZKT2effu3SsGDx4svLy8xIgRI0RKSopYsWKFeO6550T//v3FH3/8IV599VWxa9cuIYQQP//8s/Dx8RFeXl5i5MiR4s8//9Qeb+bMmeKNN94QHh4eYtSoUeL69esPHdORI0eEt7e3eOWVV4S3t7coLi4WCxYsEMOGDRMDBw4UAwYMEMePHxcZGRmiV69eomvXriI4OFgIIR44bqLa9Oyzz4qbN2+KV199VVy4cEEIcS9P7du3Fzdv3hRHjhwRgwYNEkIIERAQIHbs2CGEECIlJUWEhoYKIYRYs2aNCAsLE2VlZUIIIZYvXy7mzp370OMeOXJEtG/fXpw6dUoIIcSXX34p/P39H/p8arVauLm5if379wshhDh8+LCwt7cXR44cESdOnBBTp04VGo1GCCHEZ599Jt566y0hhBBbt24VEyZM0B63fDzvv/+++OKLL4QQQqSlpYnevXsLjUbzyONxcHAQV69eFUKIatdz4cIF4eXlJXJycoQQQpw5c0a8+OKLoqCg4KHHo/qr/LWWmJgohBBi06ZNYtiwYSIsLExER0cLIYQoKSkRXl5eIi4uTgjxT8537dolFixYoH2uOXPmiPnz5wshhHj11VfFtGnThEajEXl5eeKll14Shw8fFtnZ2aJbt27arO7evVu8+eab4tatW6J///7iypUrQoh77xtubm4iPT1dfPPNNyIgIEAUFxeLgoICMWTIEDFz5syHjuvq1atizJgxorCwUAghxI4dO4SXl5cQQohvv/1W+Pv7i8LCQqHRaMT06dPF9u3bH7j83/2BEBX7hVdffVV88MEH2nWBgYHiyJEjQoh7fUiPHj1EcnLyA8d948YN0bVrV5GbmyuEECIoKEj88MMPNfo3pHt4RtzA/PzzzygoKEBMTAw0Gg3mzp2Ly5cvIyIiAt999x1sbGxw9uxZvP7664iPj6+wb1BQEMaOHYuXX34ZxcXFGD9+PJ5++ml0794dQUFB+P7779G+fXvEx8cjPDwcX3zxBWJjYxEeHg5HR0ft85w7dw5z587Fpk2b8NRTT+Hw4cOYPHky4uLiANw7IxgdHQ0rKytMnDgRmzdvrvJ607NnzyIhIQF2dnY4efIksrKysHnzZpiYmODzzz/HunXrsHbtWkybNg27d+/GRx99hIsXLz5w3JaWlrX/P5/qvbVr12L//v3YsWMHzp07ByEECgsLK2wzcOBAzJ8/H3v37oWrqyveffddAMD+/fuRl5envbxLrVbD1ta2ymO2aNEC7du3BwB06NAB27dvf+jznTlzBgDQq1cvAICLiwvatWsHAOjSpQuaNm2KTZs24cqVKzh69CgaNWr00OMPHz4c8+bNw5tvvomtW7fCz88PJiYmjzyeJ554AnZ2djWq57fffkNWVhbGjh2rXSaTyXD58mU4ODhUeUyqn+zt7dG1a1cAwJAhQxAaGoqvvvoKf/zxB9atW4eLFy8iKysLd+/erbDfgAED8NRTT+H777/HpUuX8Pvvv1f460ufPn1gYmICKysrtGzZErdv38aJEyfQrl07bVb79++P/v3748CBA8jOzsbbb7+t3V8mkyE1NRWHDx+Gl5cX5HI55HI5vL29kZqa+tAx2dnZYcmSJYiNjcWlS5e0f00C7v0F28fHBw0aNAAArFy5EgAwceLE+y5fs2bNQ4/l7Oys/TksLAy//PIL1q5di/Pnz6OoqAh379594LgBoHfv3oiJiYGvry8OHjyIuXPnPvR4dH9sxA1Mt27dEBERgTFjxsDV1RWvvfbaQ39Jlbt79y6OHTuG27dvY9WqVdplp0+fhpmZ2QODdD9HjhyBi4sLnnrqKQDACy+8gGbNmmmvJe/evbv2z9MdOnTA7du3qxwXfzmToSssLMTIkSPRr18/ODs7Y+jQoUhISIAQosJ2I0eORJ8+ffDbb7/h119/xccff4yffvoJZWVlmDVrlrZBLigoQHFxcZXHLf/lCdx7fZcf70HPl52dXakmU1NTAPea90WLFuH1119H37590bp1a/z0008PPb6zszNKS0uRlJSEHTt2YNOmTQ89flX+/SG5uvWUlZXhhRde0DYQAHDt2jUolcoqj0f1l4lJxdvcZDIZZs+eDSEEBg4ciN69e+PatWuV8rJx40ZERUXB398f3t7esLa2xtWrV7Xr75dJMzMzyGQy7XIhBFJTU6HRaNCmTRv8+OOP2nWZmZlo1qwZNm/eXOG45Tl9mL///huTJ0/G2LFj8eKLL+L555/HvHnzAABmZhVbths3bqCsrOyBy//9fgLc+zD9b//Oqr+/PxwcHNCzZ08MHDgQf/75J4QQMDU1ve+4HRwc4O/vj9DQUJiZmaF///5Vfuin++PNmgbmqaeews8//4wJEyYgPz8fr7/+Om7duoUXXngBMTEx2v+ioqK0Z8GAe7/IhBDYtGmTdpvNmzfjrbfeum+QTp8+/cAa/vumVb6stLQUwIMbh4f57y/nt956CwDQt29fjBo16r77lP9yfti4iWpLTk4O8vPz8c477+Dll1/G77//jpKSEu214OVGjhyJlJQU+Pn5YcGCBbhz5w5u376Nl156CRs2bNDuM2fOHKxYseKR63nQ87Vp0wZyuRy//PILgHs3OJ85cwYymQy//fYb+vTpg9GjR8PR0REJCQnQaDQA7jUB5Rn+r+HDh2PBggWwt7dHixYtHnr8mqhuPS4uLvjtt99w7tw5AMCBAwcwePDgajX+VH+lpqYiJSUFwL0brrt164ZDhw7h7bffhqenJ2QyGf7880/ta67cwYMHMWTIEAwfPhzPPPMM9u7dW2mb/+rUqRPOnTuHs2fPAgD27NmDoKAgdO7cGZcuXcKxY8cAACkpKfDw8EBWVhZ69uyJ6OhoFBcXo7i4GDt37qxyTMeOHUPHjh3x+uuvo3v37tizZ4+2thdeeAE7duzQZjI0NBT/93//98DlNjY2+PvvvyGEwN27d3Hw4MH7HvP27dv466+/MGPGDPTv3x+ZmZm4fPkyysrKHjhuAOjatStMTEzw5ZdfPvD3OFWNZ8QNzMaNG5GYmIjw8HD07NkTN2/e1P6J69y5c2jTpg0OHDiAGTNm4MCBA9r9rKys0LlzZ3z99deYPHky7ty5g1GjRuHtt9+Gq6urNkjt2rXDnj17sGrVKsTGxt73l7OLiws+/vhjXLlyRXtpyrVr19CpUyecPHnyscf471/OxcXFWLdu3QN/Oa9evfq+4+alKVTbWrRogd69e2PgwIFo0qQJnn76abRt2xaXLl2CXC7XbjdjxgwsXrwYK1euhImJCaZMmYInn3wSkydPxpIlSzBkyBBoNBq0b9/+saYqe9DzmZmZYc2aNZg7dy5WrFiBVq1aoXnz5mjQoAFGjhyJGTNmwNvbG6ampnB2dkZ8fDzKysrQpUsXrFy5Em+//TYCAgIqHMvX1xcrVqyo0GjXxniqW88nn3yC+fPn491339WefYyMjGTO6aFat26t/V1la2uLsLAw7N+/H2+//TaaNm2Khg0b4vnnn6/w12MAeOONN/Dhhx9i27ZtMDU1xXPPPae95OtBmjdvjvDwcMycORMajQZWVlaIiIhAs2bNsHr1aixduhTFxcUQQmDp0qWws7PDyJEjcfnyZXh5ecHa2hotW7asckxeXl6Ij4+Hp6cnzM3N8cILL+D27dvIz8/HyJEjkZ6eDj8/Pwgh0L17d4wZMwYymey+ywsLC/Hrr7+if//+UKlU6NKly31PnDVt2hQTJkzAkCFDYG1tDRsbG3Tt2hWXLl3S3rT933GX8/Pzw86dO2Fvb1/NfzX6L5mozulM0pu7d+9i1qxZSE1NRcOGDdGiRQssWrQIhw4dwtq1a7W/pGbNmgVnZ+cKd0FfvXoVCxYsQEZGBkpKSuDl5YWpU6cCAH799VdERERogzRv3jy0bdsWS5Yswa5du7Bw4UJ89tln8Pf3x4ABA7Br1y6sXbsWGo0GDRo0QHBwcKXjAaj0+H6OHj2KBQsWYMeOHQDuXYM+Y8YMlJaWVvjlvH//fly9ehWvvfYaOnTogE8++URbx3/HTVSfLVmyBG+++SaaN2+Oa9euwcfHBwkJCWjSpInUpRFRPVFaWoopU6Zg8ODB8PT0lLoco8VGnIhIh9555x1cuHDhvusiIiLQunXrGj/n+vXrsWnTJpiZmUEIgbfffvuh933UJl2Mh6g+Wbx4MY4ePXrfdR988AFcXFz0XFHNpaWlYdSoUXBzc8OyZcsqXa9P1cdGnGoFfzkTERER1QwbcSIiIiIiCfBvCUREREREEmAjTkREREQkAYOevjA3twBlZcZ95YytrRVu3syXuozHwjFIz8REBhsbw/+yBGPPrLG/TgCOwVAYQ2aZV+nVhTEAxj8OKfNq0I14WZkw6jeJchyDYagLY6iO2NhYREZGQq1WY+zYsfD396+wPiEhAWvWrIEQAk8++SQ++ugjNG3aFNHR0QgPD9d+jXnv3r0RGBhYo2PXhcwae/0Ax0DVw7wahrowBqDujEPfDLoRJ6KayczMREREBLZt2wa5XI6RI0eiR48eaNu2LQAgPz8foaGh2Lp1K1QqFVatWoU1a9YgJCQEycnJCA4OhpeXl8SjICIiqh+qdY14bGwsPD094e7ujg0bNjxwu5kzZ2Lbtm2Vlp86dQodO3Z89CqJqFoOHToEFxcXWFtbw9LSEh4eHoiLi9OuV6vVCA0NhUqlAgDY29vj2rVrAIDk5GRER0dj8ODBmDFjBm7fvi3JGIiIiOqLKs+IV3WGrXybuXPn4vDhw+jRo0eF/QsLCzF//nyo1erar56IKsjKyoJCodA+ViqVSEpK0j62sbFBv379AABFRUX4/PPPMWbMGACAQqHAhAkT4OTkhBUrVmD+/PlYvnx5jY5va2tVC6OQlkLRWOoSHhvHQERkHKpsxP99hg2A9gzblClTtNvExsaib9++2m3+LSwsDGPHjsXJkydrrWgiur/7fS2ATCartCwvLw+TJ0+Gg4MDhgwZAgD45JNPtOvHjRunbdhr4ubNfKO+TlChaIzs7Dypy3gsHINhMDGR1YkPpkSkW1U24lWdYQPu/dIGgMTExArL9+zZg6KiIgwYMOCRiqsrb2J14cwOx2AcVCoVjh8/rn2clZUFpVJZYZusrCy8+eabcHFxwaxZswDca8y3bt2KsWPHArjX0JuZ8RYSIiIiXaryN211z7D9V3Z2NiIjI/HNN988UmGA8Z9dA+rGmR2OQXrVPbvm6uqKNWvWICcnBw0bNkR8fDwWLFigXa/RaDBx4kQMHDgQkydP1i63tLTEF198gS5duqBTp05Yv3493N3ddTIWIiIiuqfKRrw6Z9juZ//+/bh161aFqdN8fHywYcMGWFnVjTPdRIZGpVIhMDAQAQEBUKvVGDZsGJycnDB+/HhMmzYN169fx6lTp6DRaLB7924AQMeOHbFo0SKsXLkSoaGhKCoqQqtWrbB06VKJR0NERFS3VdmIV3WG7UGGDx+O4cOHax/b29sjJibm8aoloip5e3vD29u7wrJ169YBABwdHXH69On77ufs7Izt27frvD4iIiK6p8rpC/99hs3X1xdeXl7aM2zJycn6qJGIiIiIqM6p1t1YDzvD9m9hYWEPfI7U1NQalkZEREREVHfVqWkRGjdpiAYWuhtSUXEp8u4U6uz5ieoTm6ZymMktdHqM0pJi5N4u0ekxiOqDxtYWaGAu1+kxitQlyLtVrNNjEBmaOtWIN7Awg/d7ursOPXa5D4x33g0iw2Imt8D5RUN1eozWs7cCYCNO9LgamMsxYvMknR4j6pVI5IGNONUv1fqKeyIiIiIiql1sxImIiCQSGxsLT09PuLu7Y8OGDZXWp6SkYOjQofDw8MDs2bNRWloK4N5UwhMmTICvry9GjhyJq1ev6rt0IqoFbMSJiIgkkJmZiYiICGzcuBExMTHYvHkz0tLSKmwTFBSEOXPmYPfu3RBCICoqCgDw/vvvo0+fPoiOjoaPjw/Cw8OlGAIRPSY24kRERBI4dOgQXFxcYG1tDUtLS3h4eCAuLk67Pj09HUVFRejcuTMAwM/PD3FxccjJycHp06cxcuRIAMDQoUPxzjvvSDACInpcbMSJiIgkkJWVBYVCoX2sVCqRmZn5wPUKhQKZmZm4cuUKWrRogcWLF2Pw4MGYNm0azM3N9Vo7EdWOOjVrChERkbEQQlRaJpPJqlxfWlqKU6dOYerUqZg9ezZ+/PFHBAcH4/vvv6/R8W1trWpetI4pFI11ur0hqgtjAOrOOPSNjTgREZEEVCoVjh8/rn2clZUFpVJZYf2NGze0j7Ozs6FUKqFQKNCoUSP06dMHAODl5YWFCxfW+Pg3b+ajrKxys38/+mqysrOrP0mwQtG4RtsborowBsD4x2FiIpPsgykvTSEiIpKAq6srDh8+jJycHBQWFiI+Ph5ubm7a9XZ2drCwsEBiYiIAIDo6Gm5ubnj66aehUqlw4MABAMC+ffvw3HPPSTIGIno8bMSJiIgkoFKpEBgYiICAAPj6+sLLywtOTk4YP348kpOTAQDh4eH46KOPMHDgQBQWFiIgIAAA8PHHH+OLL76Al5cXvvvuOyxevFjKoRDRI+KlKURERBLx9vaGt7d3hWXr1q3T/uzg4IAtW7ZU2q9169Y1viaciAwPz4gTEREREUmAjTgRERERkQTYiBMRERERSYCNOBERERGRBNiIExERERFJgI04EREREZEE2IgTEREREUmAjTgRERERkQTYiBMRERERSYCNOBERERGRBNiIExERERFJgI04EREREZEE2IgTEREREUmAjTgRERERkQTYiBMRERERSYCNOBERERGRBNiIExERERFJoFqNeGxsLDw9PeHu7o4NGzY8cLuZM2di27Zt2seJiYkYOnQofHx88NprryE9Pf3xKyYiIiIiqgOqbMQzMzMRERGBjRs3IiYmBps3b0ZaWlqlbSZOnIi4uLgKy4OCgrBo0SLExMTA29sbCxcurN3qiYiIiIiMVJWN+KFDh+Di4gJra2tYWlrCw8OjUsMdGxuLvn37YuDAgdplJSUlmD59OhwcHAAA9vb2uHbtWi2XT0RERERknMyq2iArKwsKhUL7WKlUIikpqcI248aNA3DvUpRycrkcPj4+AICysjJ8/PHH6NevX42Ks7W1qtH2+qBQNNbLPoaGYyAiIiKqXVU24kKISstkMlm1D1BSUoLg4GCUlpbirbfeqlFxN2/mo6ys8vEfRB+NVnZ2Xo22Vyga13gfQ8MxSM/ERGaQH0yJiIjo0VV5aYpKpcKNGze0j7OysqBUKqv15AUFBRg3bhxKS0sRGRkJc3PzR6+UiIiIiKgOqbIRd3V1xeHDh5GTk4PCwkLEx8fDzc2tWk8eFBSEli1bYtWqVZDL5Y9dLBERERFRXVHlpSkqlQqBgYEICAiAWq3GsGHD4OTkhPHjx2PatGlwdHS8736nTp3Cnj170LZtW/j6+gK4d335unXranUARERERETGqMpGHAC8vb3h7e1dYdn9GuqwsDDtzx06dEBqaupjlkdEREREVDfxmzWJiIiIiCTARpyIiIiISAJsxImIiIiIJMBGnIiIiIhIAmzEieqY2NhYeHp6wt3dHRs2bKi0PiEhAT4+Phg8eDAmT56M27dvAwAyMjLg7++PAQMGYNKkSSgoKNB36URERPUKG3GiOiQzMxMRERHYuHEjYmJisHnzZqSlpWnX5+fnIzQ0FJ9//jl++ukn2NvbY82aNQCAefPmYfTo0YiLi0PHjh3x6aefSjUMIiKieoGNOFEdcujQIbi4uMDa2hqWlpbw8PBAXFycdr1arUZoaChUKhUAwN7eHteuXYNarcaxY8fg4eEBAPDz86uwHxEREdU+NuJEdUhWVhYUCoX2sVKpRGZmpvaxjY0N+vXrBwAoKirC559/jn79+iE3NxdWVlYwM7v31QIKhaLCfkRERFT7qvWFPkRkHIQQlZbJZLJKy/Ly8jB58mQ4ODhgyJAh922677dfVWxtrWq8j64pFI11ur0h4hiMS2xsLCIjI6FWqzF27Fj4+/tXWJ+SkoKQkBDk5+fD2dkZ8+bNg5mZGaKjoxEeHg5bW1sAQO/evREYGCjFEIjoEbERJ6pDVCoVjh8/rn2clZUFpVJZYZusrCy8+eabcHFxwaxZswAAzZo1Q35+PjQaDUxNTZGdnV1pv+q4eTMfZWWVPwzcj74arezsvGpvq1A0rtH2hohjMAwmJrJqfTAtv69j27ZtkMvlGDlyJHr06IG2bdtqtwkKCsLChQvRuXNnzJo1C1FRURg9ejSSk5MRHBwMLy8vXQ6FqNY1trZAA3O5To9RpC5B3q1inR6jNrARJ6pDXF1dsWbNGuTk5KBhw4aIj4/HggULtOs1Gg0mTpyIgQMHYvLkydrl5ubmcHZ2xs6dO+Ht7Y3o6Gi4ublJMQSieuXf93UA0N7XMWXKFABAeno6ioqK0LlzZwD37t9YvXq1thG/dOkSPv/8czz77LOYM2cOmjZtKtFIiKqvgbkcIzZP0ukxol6JRB7YiBORHqlUKgQGBiIgIABqtRrDhg2Dk5MTxo8fj2nTpuH69es4deoUNBoNdu/eDQDo2LEjFi1ahLlz5yI4OBiRkZF44oknsGLFColHQ1T33e++jqSkpAeu//f9GwqFAhMmTICTkxNWrFiB+fPnY/ny5dU+Ni8lMwx1YQyAYY7DEGv6LzbiRHWMt7c3vL29Kyxbt24dAMDR0RGnT5++7352dnb4/vvvdV4fEf2jqvs6Hrb+k08+0S4bN26c9kbs6uKlZNKrC2MAaj4OQ3s9VfdSMl3grClEREQSUalUuHHjhvbxf+/r+O/68vs38vLy8M0332iXCyG0sx4RkfFgI05ERCQRV1dXHD58GDk5OSgsLER8fHyF+zPs7OxgYWGBxMREANDev2FpaYkvvvgCf/75JwBg/fr1cHd3l2QMRPTo+PGZiIhIIlXd1+Ho6Ijw8HCEhISgoKAAHTp0QEBAAExNTbFy5UqEhoaiqKgIrVq1wtKlS6UeDhHVEBtxIiIiCT3svg4AcHBwwJYtWyrt5+zsjO3bt+u8PiLSHV6aQkREREQkATbiREREREQSYCNORERERCQBNuJERERERBJgI05EREREJAE24kREREREEmAjTkREREQkATbiREREREQSYCNORERERCQBNuJERERERBJgI05EREREJIFqNeKxsbHw9PSEu7s7NmzY8MDtZs6ciW3btmkfZ2RkwN/fHwMGDMCkSZNQUFDw+BUTEREREdUBVTbimZmZiIiIwMaNGxETE4PNmzcjLS2t0jYTJ05EXFxcheXz5s3D6NGjERcXh44dO+LTTz+t3eqJiIiIiIxUlY34oUOH4OLiAmtra1haWsLDw6NSwx0bG4u+ffti4MCB2mVqtRrHjh2Dh4cHAMDPz6/SfkRERERE9ZVZVRtkZWVBoVBoHyuVSiQlJVXYZty4cQCAxMRE7bLc3FxYWVnBzOzeIRQKBTIzM2ulaCIiIiIiY1dlIy6EqLRMJpNV+cSPut+/2dpa1Wh7fVAoGutlH0PDMRARERHVriobcZVKhePHj2sfZ2VlQalUVvnEzZo1Q35+PjQaDUxNTZGdnV2t/f7t5s18lJVVbugfRB+NVnZ2Xo22Vyga13gfQ8MxSM/ERGaQH0yJiIjo0VV5jbirqysOHz6MnJwcFBYWIj4+Hm5ublU+sbm5OZydnbFz504AQHR0dLX2IyIiIiKqD6psxFUqFQIDAxEQEABfX194eXnByckJ48ePR3Jy8kP3nTt3LqKiouDp6Ynjx4/jnXfeqa26iYiIiIiMWpWXpgCAt7c3vL29Kyxbt25dpe3CwsIqPLazs8P333//GOUREREREdVN/GZNIiIiIiIJsBEnIiIiIpIAG3EiIiIiIgmwESciIiIikgAbcSIiIiIiCbARJyIiIiKSABtxIiIiIiIJsBEnIiIiIpIAG3EiIiIiIgmwESciIiIikgAbcSIiIiIiCbARJyIiIiKSgJnUBRARERHpQ2NrCzQwl+vs+YvUJci7Vayz56e6h404ERGRRGJjYxEZGQm1Wo2xY8fC39+/wvqUlBSEhIQgPz8fzs7OmDdvHszM/vnVferUKYwYMQJ//fWXvks3Sg3M5RixeZLOnj/qlUjkgY04VR8vTSEiIpJAZmYmIiIisHHjRsTExGDz5s1IS0ursE1QUBDmzJmD3bt3QwiBqKgo7brCwkLMnz8farVa36UTUS1hI05ERCSBQ4cOwcXFBdbW1rC0tISHhwfi4uK069PT01FUVITOnTsDAPz8/CqsDwsLw9ixY/VcNRHVJl6aQkREJIGsrCwoFArtY6VSiaSkpAeuVygUyMzMBADs2bMHRUVFGDBgwCMf39bW6pH31RWForFOt9eHujCGR2GI4zDEmv6LjTgREZEEhBCVlslksirXZ2dnIzIyEt98881jHf/mzXyUlVU+xv3oq6HJzs6r9rYKReMabV++j67pegyGqKbjMLTXk4mJTLIPprw0hYiISAIqlQo3btzQPs7KyoJSqXzg+uzsbCiVSuzfvx+3bt2Cv78/fHx8AAA+Pj7Iz8/XX/FEVCvYiBMREUnA1dUVhw8fRk5ODgoLCxEfHw83Nzftejs7O1hYWCAxMREAEB0dDTc3NwwfPhwJCQmIiYlBTEwMACAmJgZWVoZ3qQkRPRwbcSIiIgmoVCoEBgYiICAAvr6+8PLygpOTE8aPH4/k5GQAQHh4OD766CMMHDgQhYWFCAgIkLhqIqpNvEacqI6pal7icjNnzkSPHj3g5+cH4N7ZtvDwcNja2gIAevfujcDAQL3VTVQfeXt7w9vbu8KydevWaX92cHDAli1bHvocqampOqmNiHSPjThRHVI+L/G2bdsgl8sxcuRI9OjRA23btq2wzdy5c3H48GH06NFDuzw5ORnBwcHw8vKSonQiIqJ6h5emENUhVc1LDNw7Y963b18MHDiwwvLk5GRER0dj8ODBmDFjBm7fvq3P0omIiOodnhEnqkOqmpcYAMaNGwcA2hvAyikUCkyYMAFOTk5YsWIF5s+fj+XLl9fo+JyX2DDUxzGUlZbAxEyuo2p0//xEVD+xESeqQ6qal/hhPvnkE+3P48aNQ79+/Wp8/Po4L7Ghqa9jUCga4/yioTqqCGg9e2uNapJyXmIiMh68NIWoDqlqXuIHycvLq/DlIEIImJnxczoREZEusREnqkOqmpf4QSwtLfHFF1/gzz//BACsX78e7u7uui6XiIioXuMpL6I65N/zEqvVagwbNkw7L/G0adPg6Oh43/1MTU2xcuVKhIaGoqioCK1atcLSpUv1XD0REVH9Uq1GvKp5iVNSUhASEoL8/Hw4Oztj3rx5MDMzw9WrVzFz5kzk5+ejSZMmCAsLg52dnU4GQkT3VDUvcbmwsLAKj52dnbF9+3ad1kZERET/qPLSlPJ5iTdu3IiYmBhs3rwZaWlpFbYJCgrCnDlzsHv3bgghEBUVBQBYtWoVBg0ahJiYGPTv3x8RERG6GQURERERkZGpshGval7i9PR0FBUVoXPnzgAAPz8/7fqysjLk5+cDAAoLC9GgQQMdDIGIiIiIyPhUeWlKVfMS/3e9QqFAZmYmAGD69OkYOXIkvv/+e6jVamzevLk2ayciIiIiMlpVNuJVzUv8sPUzZ87E/Pnz0a9fP+zevRtTpkzBTz/9VO15jQ1xDtZHmfu4Pn65hiGqC2MgIiKiuqPKRlylUuH48ePax/+dl/i/8xZnZ2dDqVQiJycH58+f134piIeHB+bOnYvc3Fw0a9asWsXV5MtBAP00Wo/yJRP18cs1DI2xj4FfDkJERFT3VHmNeFXzEtvZ2cHCwkL7ddnR0dFwc3ODjY0NLCwstE18YmIiGjVqVO0mnIiIiIioLqvWGfGq5iUODw9HSEgICgoK0KFDBwQEBEAmk+Hjjz/GggULUFRUhEaNGmHNmjX6GBMRERERkcGr1jziVc1L7ODggC1btlTaz8nJCT/++ONjlkhEREREVPfwK+6JiIiIiCTARpyIiIiISAJsxImIiIiIJMBGnIiIiIhIAmzEiYiIiIgkwEaciIiIiEgCbMSJiIiIiCTARpyIiIiISAJsxImIiIiIJMBGnIiIiIhIAmzEiYiIiIgkwEaciIiIiEgCbMSJiIiIiCTARpyIiIiISAJmUhdAZIgaN2mIBha6jUdRcSny7hTq9BhERERkuNiIE91HAwszeL8Xo9NjxC73QZ5Oj0BERESGjJemEBERERFJgI04ERGRRGJjY+Hp6Ql3d3ds2LCh0vqUlBQMHToUHh4emD17NkpLSwEAx48fh5+fH7y9vTFx4kTcvn1b36UTUS1gI05ERCSBzMxMREREYOPGjYiJicHmzZuRlpZWYZugoCDMmTMHu3fvhhACUVFRAIAPPvgAS5cuRWxsLNq2bYsvv/xSiiEQ0WNiI05ERCSBQ4cOwcXFBdbW1rC0tISHhwfi4uK069PT01FUVITOnTsDAPz8/LTrd+7cibZt20KtViMzMxNNmjSRYghE9Jh4syYREZEEsrKyoFAotI+VSiWSkpIeuF6hUCAzMxMAYG5ujtTUVLz++uswMzPDu+++W+Pj29paPUb1uqFQNNbp9vpQF8bwKAxxHIZY03+xESciIpKAEKLSMplMVu319vb2OHToEDZt2oTAwEBs2rSpRse/eTMfZWWVj3E/+mposrOrP5eUQtG4RtuX76Nruh6DIarpOAzt9WRiIpPsgykvTSEiIpKASqXCjRs3tI+zsrKgVCofuD47OxtKpRLFxcVISEjQLh88eDBSU1P1UzQR1So24kRERBJwdXXF4cOHkZOTg8LCQsTHx8PNzU273s7ODhYWFkhMTAQAREdHw83NDWZmZpg3bx7++usvAMCuXbvQtWtXScZARI+Hl6YQERFJQKVSITAwEAEBAVCr1Rg2bBicnJwwfvx4TJs2DY6OjggPD0dISAgKCgrQoUMHBAQEwNTUFBEREfjwww+h0WigUqmwaNEiqYdDRI+AjTgREZFEvL294e3tXWHZunXrtD87ODhgy5YtlfZzdnbGtm3bdF4fEekWL00hIiIiIpIAG3EiIiIiIgmwESciIiIikkC1GvHY2Fh4enrC3d0dGzZsqLQ+JSUFQ4cOhYeHB2bPno3S0lIA96ZimjBhAnx9fTFy5EhcvXq1dqsnIiIiIjJSVTbimZmZiIiIwMaNGxETE4PNmzcjLS2twjZBQUGYM2cOdu/eDSEEoqKiAADvv/8++vTpg+joaPj4+CA8PFw3oyAiIiIiMjJVzppy6NAhuLi4wNraGgDg4eGBuLg4TJkyBQCQnp6OoqIidO7cGQDg5+eH1atXY8CAATh9+jS+/vprAMDQoUPxwgsv6GYUZFAaN2mIBha6nZCnqLgUeXcKdXoMIiIiIl2qslvKysqCQqHQPlYqlUhKSnrgeoVCgczMTFy5cgUtWrTA4sWLcfToUbRo0QJz5syp5fLJEDWwMIP3ezE6PUbsch8Y/5cCExERUX1WZSMuhKi0TCaTVbm+tLQUp06dwtSpUzF79mz8+OOPCA4Oxvfff1/t4mxtraq9rb4oFI31so+hMcQx1LSmujAGIiIiqjuqbMRVKhWOHz+ufZyVlQWlUllh/Y0bN7SPs7OzoVQqoVAo0KhRI/Tp0wcA4OXlhYULF9aouJs381FWVrnRfxB9NDXZ2TU7D6tQNK7xPoampmPQV3NZ05qMeQwmJjKD/GBKREREj67KmzVdXV1x+PBh5OTkoLCwEPHx8XBzc9Out7Ozg4WFBRITEwEA0dHRcHNzw9NPPw2VSoUDBw4AAPbt24fnnntOR8MgIiIiIjIu1TojHhgYiICAAKjVagwbNgxOTk4YP348pk2bBkdHR4SHhyMkJAQFBQXo0KEDAgICAAAff/wx5s6di2XLlsHKygphYWE6HxBRfRcbG4vIyEio1WqMHTsW/v7+991u5syZ6NGjB/z8/AAAGRkZCAoKws2bN/HMM88gPDwcjRo10mfpRERUhcbWFmhgLtfZ8xepS5B3q1hnz08VVWtqC29vb3h7e1dYtm7dOu3PDg4O2LJlS6X9WrduXaNrwono8ZRPN7pt2zbI5XKMHDkSPXr0QNu2bStsM3fuXBw+fBg9evTQLp83bx5Gjx6NQYMG4ZNPPsGnn36KoKAgKYZBREQP0MBcjhGbJ+ns+aNeiUQe2IjrC79Zk6gO+fd0o5aWltrpRv8tNjYWffv2xcCBA7XL1Go1jh07Bg8PDwD3piH9735ERERUu3Q72TMR6VVV040CwLhx4wBAe18HAOTm5sLKygpmZvfeEsqnIa0pQ7yhtC7MrlNTHINuGGJNRGTc2IgT1SFVTTda2/v9V01mOjK0mWmA+jnLkSF6lDEY2qxZnOmIiKqDl6YQ1SH/nU70v9ONPkizZs2Qn58PjUYD4J9pSImIiEh32IgT1SFVTTf6IObm5nB2dsbOnTsB/DMNKREREekOG3GiOuTf0436+vrCy8tLO91ocnLyQ/edO3cuoqKi4OnpiePHj+Odd97RT9FERET1FK8RJ6pjqpputNx/5/W3s7PjdKNERER6xDPiREREREQSYCNORERERCQBNuJERERERBJgI05EREREJAE24kREREREEmAjTkREREQkATbiREREREQSYCNORERERCQBNuJERERERBJgI05EREREJAE24kREREREEmAjTkREREQkATOpC6CKGjdpiAYWuvtnKSouRd6dQp09PxERERFVDxtxA9PAwgze78Xo7Pljl/sgT2fPTkRERETVxUtTiIiIJBIbGwtPT0+4u7tjw4YNldanpKRg6NCh8PDwwOzZs1FaWgoASExMxNChQ+Hj44PXXnsN6enp+i6diGoBG3EiIiIJZGZmIiIiAhs3bkRMTAw2b96MtLS0CtsEBQVhzpw52L17N4QQiIqK0i5ftGgRYmJi4O3tjYULF0oxBCJ6TGzEiYiIJHDo0CG4uLjA2toalpaW8PDwQFxcnHZ9eno6ioqK0LlzZwCAn58f4uLiUFJSgunTp8PBwQEAYG9vj2vXrkkxBCJ6TLxGnIiISAJZWVlQKBTax0qlEklJSQ9cr1AokJmZCblcDh8fHwBAWVkZPv74Y/Tr16/Gx7e1tXqM6nVDoWis0+31oT6O4VH30TVDrOm/2IgTERFJQAhRaZlMJqv2+pKSEgQHB6O0tBRvvfVWjY9/82Y+ysoqH+N+9NXQZGdXfzoBhaJxjbYv30fX6tsYgJqPw9BeTyYmMsk+mPLSFCIiIgmoVCrcuHFD+zgrKwtKpfKB67Ozs7XrCwoKMG7cOJSWliIyMhLm5ub6K5yIag0bcSIiIgm4urri8OHDyMnJQWFhIeLj4+Hm5qZdb2dnBwsLCyQmJgIAoqOjteuDgoLQsmVLrFq1CnK5XJL6iejx8dIUIiIiCahUKgQGBiIgIABqtRrDhg2Dk5MTxo8fj2nTpsHR0RHh4eEICQlBQUEBOnTogICAAJw6dQp79uxB27Zt4evrC+De9eXr1q2TdkBEVGPVasRjY2MRGRkJtVqNsWPHwt/fv8L6lJQUhISEID8/H87Ozpg3bx7MzP556lOnTmHEiBH466+/ard6IiIiI+bt7Q1vb+8Ky/7dUDs4OGDLli0V1nfo0AGpqal6qY+IdKvKS1MeZ55TACgsLMT8+fOhVqtrv3oiIiIiIiNVZSP+qPOclgsLC8PYsWNrvXAiIiIiImNW5aUpjzrPKQDs2bMHRUVFGDBgwCMVVxfmOH3UfXSpLowBqL9ztRIREVHdUGUj/qjznGZnZyMyMhLffPPNIxdXkzlOAc6tWV11YQyAbudqNbQxSDnHKREREelGlZemPOo8p/v378etW7fg7++v/QYwHx8f5Ofn12b9RERERERGqcpG/FHnOR0+fDgSEhIQExODmJgYAEBMTAysrHhWj4iIiIioWmfEy+c59fX1hZeXl3ae0+TkZABAeHg4PvroIwwcOBCFhYUICAjQeeFERERERMasWvOIP8o8p//FOU+JiIiIiP7Bb9YkInpENk3lMJNb6PQYpSXFyL1dotNjEBGRNNiIExE9IjO5Bc4vGqrTY7SevRUAG3EiorqoymvEiYiIiIio9rERJyIiIiKSABtxIiIiIiIJsBEnIiIiIpIAG3EiIiIiIgmwESciIiIikgAbcSIiIiIiCbARJyIiIiKSABtxIiIiIiIJsBEnIiIiIpIAG3EiIiIiIgmwESciIiIikgAbcaI6JjY2Fp6ennB3d8eGDRsqrU9JScHQoUPh4eGB2bNno7S0FAAQHR2Nl156CT4+PvDx8UFERIS+SyciIqpXzKQugIhqT2ZmJiIiIrBt2zbI5XKMHDkSPXr0QNu2bbXbBAUFYeHChejcuTNmzZqFqKgojB49GsnJyQgODoaXl5eEIyAiIqo/eEacqA45dOgQXFxcYG1tDUtLS3h4eCAuLk67Pj09HUVFRejcuTMAwM/PT7s+OTkZ0dHRGDx4MGbMmIHbt29LMQQiIqJ6g2fEieqQrKwsKBQK7WOlUomkpKQHrlcoFMjMzNT+PGHCBDg5OWHFihWYP38+li9fXqPj29paPeYIap9C0Vin2+tDXRhDTRniGAyxJiIybmzEieoQIUSlZTKZrFrrP/nkE+2ycePGoV+/fjU+/s2b+Sgrq3yM+9FXU5OdnVftbRWKxjXeXh90OQZD9Chj0Me/RU1qMjGRGeQHUyIyLGzEieoQlUqF48ePax9nZWVBqVRWWH/jxg3t4+zsbCiVSuTl5WHr1q0YO3YsgHsNu5kZ3x7qA5umcpjJLXR6jNKSYuTeLtHpMYiIjBF/0xLVIa6urlizZg1ycnLQsGFDxMfHY8GCBdr1dnZ2sLCwQGJiIrp164bo6Gi4ubnB0tISX3zxBbp06YJOnTph/fr1cHd3l3AkpC9mcgucXzRUp8doPXsrADbiRET/xUacqA5RqVQIDAxEQEAA1Go1hg0bBicnJ4wfPx7Tpk2Do6MjwsPDERISgoKCAnTo0AEBAQEwNTXFypUrERoaiqKiIrRq1QpLly6VejhERER1GhtxojrG29sb3t7eFZatW7dO+7ODgwO2bNlSaT9nZ2ds375d5/URERHRPZy+kIiIiIhIAmzEiYiIiIgkwEaciIhIIrGxsfD09IS7uzs2bNhQaX1KSgqGDh0KDw8PzJ49G6WlpRXWr1q1CmvWrNFXuURUy9iIExERSSAzMxMRERHYuHEjYmJisHnzZqSlpVXYJigoCHPmzMHu3bshhEBUVBQAIC8vD7NmzcJXX30lRelEVEvYiBMREUng0KFDcHFxgbW1NSwtLeHh4YG4uDjt+vT0dBQVFaFz584AAD8/P+36PXv2oFWrVnj99delKJ2IaglnTSEiIpJAVlYWFAqF9rFSqURSUtID1ysUCmRmZgIAfH19AeCxLksxxG/+rOk3pOrr221roj6O4VH30TVDrOm/qtWIx8bGIjIyEmq1GmPHjoW/v3+F9SkpKQgJCUF+fj6cnZ0xb948mJmZITExEYsXL0ZpaSmsra2xePFi2NnZ6WQgRERExkQIUWmZTCar9vrHdfNmPsrKKh/jfvTV0GRn51V7W4WicY22L99H1+rbGICaj8PQXk8mJjLJPphWeWnK41zDFhQUhEWLFiEmJgbe3t5YuHChbkZBRERkZFQqFW7cuKF9nJWVBaVS+cD12dnZFdYTkfGrshF/1GvYSkpKMH36dDg4OAAA7O3tce3aNd2MgoiIyMi4urri8OHDyMnJQWFhIeLj4+Hm5qZdb2dnBwsLCyQmJgIAoqOjK6wnIuNX5aUpj3oNm1wuh4+PDwCgrKwMH3/8Mfr161ej4urC9WuPuo8u1YUxAPX3OjwiqhtUKhUCAwMREBAAtVqNYcOGwcnJCePHj8e0adPg6OiI8PBwhISEoKCgAB06dEBAQIDUZRNRLaqyEX/ca9hKSkoQHByM0tJSvPXWWzUqribXrwG8bqq66sIYAN1eh2doY5Dy+jUi0h1vb294e3tXWLZu3Trtzw4ODtiyZcsD9586darOaiMi3avy0pTHuYatoKAA48aNQ2lpKSIjI2Fubl6btRMRERERGa0qG/HHuYYtKCgILVu2xKpVqyCXy3U0BCIiIiIi41PlpSmPeg3bqVOnsGfPHrRt21Y736lSqazwJzciIiIiovqqWvOIP8o1bB06dEBqamotlEhEREREVPfwK+6JiIiIiCTARpyIiIiISAJsxImIiIiIJMBGnIiIiIhIAmzEiYiIiIgkwEaciIiIiEgCbMSJiIiIiCTARpyIiIiISAJsxImIiIiIJMBGnIiIiIhIAmzEiYiIiIgkwEaciIiIiEgCbMSJiIiIiCTARpyIiIiISAJsxImIiIiIJMBGnIiIiIhIAmzEiYiIiIgkwEaciIiIiEgCbMSJiIiIiCTARpyIiIiISAJsxImIiIiIJMBGnIiIiIhIAmzEiYiIiIgkwEaciIiIiEgCbMSJiIiIiCTARpyIiIiISAJsxImIiIiIJMBGnIiIiIhIAmzEiYiIiIgkUK1GPDY2Fp6ennB3d8eGDRsqrU9JScHQoUPh4eGB2bNno7S0FACQkZEBf39/DBgwAJMmTUJBQUHtVk9ElTCvRMaDeSWq36psxDMzMxEREYGNGzciJiYGmzdvRlpaWoVtgoKCMGfOHOzevRtCCERFRQEA5s2bh9GjRyMuLg4dO3bEp59+qptREBEA5pXImDCvRGRW1QaHDh2Ci4sLrK2tAQAeHh6Ii4vDlClTAADp6ekoKipC586dAQB+fn5YvXo1hg8fjmPHjuGTTz7RLn/11VcRFBRU7eJMTGQ1HA6gtGlY431q4lFqquk+HEP11LQmYx5DdbeTMq81qbOcWVNFjbZ/FLp+nXAM1aPrMQC6H0dNaqrOtsaWV4Vlsxpt/yj08TrR9Tjq4xgeZR9Dej09ynhrS5WNeFZWFhSKf97clEolkpKSHrheoVAgMzMTubm5sLKygpmZWYXlNWFj06hG2wPAlyH9a7xPTdjaWul8H46hempaU10YQ1WkzCtQ88w+PWVtjY9RU7p+nXAM1aPrMQC6H0d9z+sn3otqfIya0sfrRNfjqI9jeJR9DPH1JIUqL00RQlRaJpPJqlxf1X5EVPuYVyLjwbwSUZWNuEqlwo0bN7SPs7KyoFQqH7g+OzsbSqUSzZo1Q35+PjQaTYXlRKQ7zCuR8WBeiajKRtzV1RWHDx9GTk4OCgsLER8fDzc3N+16Ozs7WFhYIDExEQAQHR0NNzc3mJubw9nZGTt37qywnIh0h3klMh7MKxHJxP3+xvUfsbGx+Oyzz6BWqzFs2DCMHz8e48ePx7Rp0+Do6IjTp08jJCQEBQUF6NChAz766CPI5XKkp6cjODgYN2/exBNPPIEVK1agadOm+hgXUb3FvBIZD+aVqH6rViNORERERES1i9+sSUREREQkATbiREREREQSYCNORERERCQBNuJERERERBJgI05EREREJAE24lW4e/eu1CXUSGFhodQlVGKINUnpwIEDOHz4sNRl1EnM6+MzxJqkxLzqljFl1hCzYYg1SckY88pG/CF2796NV199Ffn5+VKXUi3x8fHw9vZGWlqa1KVoGWJNUtq3bx/ee+89TJs2DQcPHpS6nDqFeX18hliTlJhX3TKmzBpiNgyxJikZa17ZiD/A3r17sWLFCowaNQpWVlZSl1OlvXv3IiIiAk5OTggNDcX58+elLskga5LSgQMHsGDBAqxbtw4//PADli9fjl9//VXqsuoE5rVu1iQl5lW3jCmzhpgNQ6xJSkadV0GV7NmzR7z88ssiOjpau6ykpETCih5u3759ol+/ftp6t2/fLnx9fcW5c+dYk4HYs2ePePHFF8WLL74oiouLhRBC/Pbbb6Jv377il19+kbg648a81s2apMS86pYxZdYQs2GINUnJ2PPKRvw/Dh8+LPr06SO2b9+uXVb+D3vx4kVx4cIFaQp7gL1794ru3buLwMDACsuXLVsm5syZw5oMwL59+0Tfvn1FQkKC2LVrl/D19RVXr14VQgjx3XffidDQUKFWqyWu0jgxr3WzJikxr7plTJk1xGwYYk1Sqgt55aUp/5KZmYmgoCBMmTIFvr6+AICSkhLI5XJcuHABHh4eSEhIkLbIf/n555+xdOlSTJw4Ebm5udi4cSMAIC8vD+bm5mjWrJnB1JSfny9ZTVL65ZdfEBQUhKlTp6Jv374YMGAAGjRogHfffRdbtmzBokWL0Lx5c5iZmUldqtFhXnVXE/PKvOqCMWWWeTV8dSWvhl2dnqnVavTs2RN+fn7QaDTQaDSQy+W4ePEiRowYgTfeeAPjxo2TukwAwM2bN7Fo0SK8/fbbGD58OPr374/JkycjJSUF5ubmOHbsGCZPngwAEEJAJpNJUtPbb7+N06dPQ6PR4OTJk5g0aZLO6zAkLVu2xLx58+Dp6an9d2jevDkaNmyI4uJizJw5E/7+/trt9fVvVRcwr7VfE/PKvOqSsWSWeTUOdSWvMiGEkLoIQ3Hx4kVMnz4d8+bNQ+fOnbXLhg8fjhdffBG9e/eGo6Mj2rRpYxD/oJmZmVCpVCgtLYWZmRmysrIQHR2Nhg0bYvfu3Rg/fjwaNGiAHj166K3e/9Z0/fp1/PTTT7C0tISTkxOcnJxQVlYGExMTg/h/qE9qtRrp6emYPHkyPv/8cxQXF+Ovv/7C5cuX8dRTT2nPENW3/y+Pinmt/ZqY138wr7XPmDLLvBoXY84rG/H/2L9/P8LDw+Hv7w8TExN8+umn6NSpE7p27YpvvvkG7dq1w5gxY+Dm5iZ1qdoXlBACarUacrkc6enpiIyMxMWLF+Hp6Ynvv/8es2bNQs+ePfVeU1lZGUxNTbXrfv75Z/zwww+wsbHB4MGD0atXL4MMhS79/vvvCAkJwbBhw3DixAlcuHAB8+bNw6pVq+Dt7Y3Ro0dLXaJRYV5rrybmtTLmtfYZS2aZV+NjrHnlpSn/0bt3b5iamuLAgQNo0KABBgwYgLZt2+L48eNYvnw5lEolxo0bB5VKBXt7e0lrLQ+YTCaDXC4HACxZsgRyuRzr168HACiVSvzwww/o0aMHzM3NdR7Kf9dkamqqfSP44YcfsG3bNowZMwbNmjXDsmXLYGFhARcXF53WYyjK/z/s2LEDLVu2hLm5OXr06IHCwkJcv34dK1aswPLlyzFo0CA0btwYJia8faM6mNfaq4l5/QfzqjvGklnm1XgYe17ZiP+HEAI9e/ZEz549cfbsWfz888/4448/8Oqrr8LR0REajQampqbIy8uTutRKcnJykJubi7Vr12qXWVtbw8TEBHK5HGVlZXr/dCyTyZCUlIQ1a9Zg9erVcHZ2BgCkpaXh9OnT9eaNovz/+4QJExAdHY3du3dj2bJleP311/Hmm2/iq6++Qq9evdC0adMK+9W3Mxo1xbzWLub1HuZVd4w1s8yr4TL2vBrWxwID8O9/FFNTU1y8eBGvvPIKHB0dkZubi8DAQHTu3BmOjo5ISkrChQsXJKy2IplMhpKSEpw9exYAcOfOHQQHB8PV1RWXL1/GDz/8gBMnTgC49wLUF7VajUGDBmnfJAAgMTERarUawD9f0VsfrpJ68skn8dprr6FZs2ZISUkBAKxYsQL9+/eHra0trly5gl27dmm/olcmk+H27dtSlmzQmNfax7z+g3mtfcaaWebV8BlrXnmNeBXy8/NhZWWFnJwczJs3DyqVCiNHjsTdu3cREhKChg0bIjAwEN27d5e6VADAnj17sHz5cvTq1Qv/93//h379+mHGjBno378/AgICsH37dr1e0wYAp06dwrvvvouFCxeiU6dOWLZsGQ4fPoyoqCgsXboURUVF6N+/P/r06WMwn1B1LSUlBbNmzYKpqSkGDhyIQYMG4aeffsKpU6dQXFyMq1evIiAgAE5OTti6dStGjRqFZ555RuqyDR7z+viY18qYV90xpswyr8bB2PLKS1OqYGlpidLSUsyePRsKhQKjRo3CnTt38OWXX+Lpp5/G8OHDMXv2bCxbtkx7F7hUhBDo27cvbG1tkZ+fDzc3Nzz77LOwtLSEQqGAp6cnnJ2dsXbtWvTo0QNyuRy3bt0CcO9PbLrSoUMHzJw5EwsXLkSLFi2QkZGBzz77DMeOHcNvv/2GL7/8EpMnT0ajRo0M4s1WH9q3b4/PPvsMmZmZsLKywuXLl3HlyhX0798fnp6eKCgowKRJk3Dq1CmYmZmhuLhY6pKNAvP6+JjXyphX3TGWzDKvxsPY8spGvAomJiYwMTHBe++9B5lMhtzcXKxfvx4qlQqzZ88GADz//PP49ddfoVAoYGdnJ9mnzvJjlr9Z3blzByNGjMCnn34KPz8/bNy4Ed26dUPPnj1RVFSEn376CX/99ReefvppvPHGGzqrSwiBPn36oH379rC0tISJiQmOHj2KDh06oGHDhrCysoKHhwf++OMPdO/eHWVlZTh48CCee+452Nra6qwuqSmVSiiVSgDAxo0bYWpqCk9PTwDA8ePHkZGRgfbt2yMgIAB2dnZSlmo0mNfHx7zeH/OqG8aSWebVuBhTXnmNeDW1bdsWSqUSX3/9NaysrLRvEJ988gl27dqFwsJChIeH48CBA5DJZNBoNBJXDGg0GigUCjRs2BAvvfQSfv31V3Tp0gWjRo3CwYMHERISgqysLJ2+SQD/vIH973//Q5MmTZCcnIzPPvsMTzzxBF5++WXEx8ejffv2GDBgAL7//nu8//77+O6775CTk6PTugxFaWkpTExM8NxzzwEAkpKSkJycjD59+mDGjBk4ffo0NmzYIHGVxoV5fXTM68Mxr7phbJllXo2DMeSVZ8RroHHjxpg+fTratm0LAFi7di127tyJb7/9Fk5OToiOjsaqVavQrVs3WFlZITs7G3l5eWjdurUk9drY2GDMmDGYOnUqunbtCjs7O+10PgcPHkT//v2hVCpRUFCARo0a6e0sg0KhQF5eHjIzMyGXy7Fr1y589dVXKC0tRWpqKvbu3YsPPvgA7dq1qxfXtJmZmcHPzw8zZ85EUlISLl68iI4dO2Lq1Kk4evQo3n77bSxfvrzCPvXh/8vjYl5rB/NaEfOqO8aUWebVOBhDXnmz5iP6+OOP8eOPP2L16tXo1KkTrly5gs2bN0Mmk+G9995DSUkJTp48ibS0tApfsapP5S+mY8eO4cKFC7Czs4OVlRW+++47NGnSBHPnzsX169exe/dudO3aFY6Ojnp7ASYkJGDt2rVo3bo12rZtC19fX0RHR+Po0aPw8fEBAPTs2RM2Njb14s0CAK5evYrPP/8cGo0GH374IY4dO4bx48dj9erVcHd3R1FREW7evCn5n9GMEfP6eJjXyphX3TL0zDKvxsWg8yrokfz+++/i5MmTQgghrly5ItasWSPmzJkjzpw5I65duyY6deok8vLyhBBCnD17VuTk5EhY7T2lpaVixowZIiwsTJSVlYmcnBwRHBwsnJycxJgxY8Qvv/yilzrKysqEEEKcOXNGZGRkiAsXLohvv/1WjBkzRvz5559CCCGio6PFpEmTxIkTJyrsU9fdvHlTXLx4UZw9e1a0b99exMfHCyGEKCwsFD///LN45513tP+PqPqY10fHvD4Y86o7xpZZ5tXwGWpeeY34IxBC4Pnnn0fnzp21n9Jv3rwJf39/NGvWDEOHDsXo0aNx584dLFu2DOvXr8fvv/8uddm4ePEijh49iiFDhkCtVuObb75BSUkJtm3bhg8//BCLFi1Camqqzuso//Tdrl07PPHEEzh//jx+/PFHLFq0CE5OToiLi8OaNWugVquxYsUK7N+/v158YgeAZs2aoWXLljhz5gw8PT3h7u6OwsJCHDlyBAcOHED79u3h6OhYYR/BP2o9FPP6eJjXB2NedcMYM8u8Gj5DzSuvEX8E/37RajQa5Obm4tVXX4WtrS08PT3h7++PadOm4ZNPPsGXX36JN954Ax4eHjh16hQUCgUUCoXeaxZCoE2bNli5ciVu3ryJb7/9FikpKVi7di2USiVKSkpgZ2eHkpISvddmZWUFS0tL2NjYICYmBvHx8QgMDMSgQYNw/PhxLFu2DI6OjmjWrFm9ecN47rnnsHTpUmzfvh2XLl3CjRs30Lx5c3h7e2Pu3LmwsLDAM888g9GjR9eb/yePinmtXcxrZcxr7TK2zDKvxsXQ8soz4o+pVatW+OCDD2Bra4vevXtjzJgxmDZtGnbs2IGMjAy8/fbb2hs2zMzMEBcXJ0md5S8mOzs7JCYm4siRIxg2bBiUSiXy8/Pxxx9/wNLSEs2bN0dqaqr2W6n08Wmwe/fuWLBgARITE7F06VJ0794dgwYNAgDcvn0b//vf/2Bra6sdQ304o9SyZUt88803uHz5Mvbv34/27dtj4MCBCA8PR3Z2Ntzd3bFlyxZERUVJXapRYV4fH/NaGfOqO8aQWebVuBhaXnlGvBY0atQIf/zxB8aMGYMpU6YgNjYWe/bswQsvvIBXXnkFBw4cQEREBCZMmCDZjWDAvWl8Nm3ahBMnTiAoKAhffvklGjRogPT0dKSlpcHR0RF3797FG2+8gQYNGmDBggVwdXXVeV2FhYU4deoUYmNjYWNjAxcXFwDApUuX8Ndff6FFixYoKCjAunXr8NJLL8HZ2ble3GDSqlUrjB49Go0aNYK/vz/Gjx8POzs77R3eAQEBSExMxIgRI1BSUoK9e/eiXbt2aNOmjcSVGzbm9fEwr/fHvOqOMWSWeTUuhpRXzppSC/79ok1PT0doaCiGDBkCT09P/P777/jpp59gbW2NqVOnwsLCotI++nTt2jWYmJhApVJh//79SEhIgJ2dHdq0aYN27drh3XffRadOneDl5YXly5fj888/R+PGjXVaU35+PmbMmAEvLy889dRT+OCDD9CpUyeUlJRArVZj+vTp2LBhAzZv3gxHR0dMmjQJvXr10mlNhqa4uBjTpk3DZ599BuDe6+fEiRNITk5GSUkJfvvtN5ibm2P27Nn8au0qMK+Ph3mtGvNau4wls8yrcZI6rzwjXgv+HfacnBxkZGTAyckJJ0+exI4dO9C0aVO88cYbWL9+PcrKytC+fXu89NJLktT6xBNPALj36b1379546aWXYGZmhtOnTyM4OBiurq4ICgoCAMyaNQtqtRoXLlzAM888o7M3NisrKyxduhRNmjQBACxcuBBXr15FixYt8OSTT2LTpk04duwYDh48iOvXr+PAgQPaN4r68MkduPdGcePGDcTFxaFTp05o3LgxunXrhhYtWiAqKgopKSl466238Mwzz+DAgQNo2rSp5F/hbqiY18fDvFaNea1dxpJZ5tU4SZ1XnhHXgdTUVJw9exYHDhxAq1at0KdPH6xfvx6XL1/GkCFDsHbtWixfvhxOTk6Sv9DLyspw9+5dDB8+HL1798bMmTMB3JvDNS0tDXl5ecjNzcXkyZPRr18/ndVR/v9Bo9HA1NRUu3zOnDlITU3FZ599BhsbG2RmZuLvv/9Gbm4unnnmGXTt2lXy/4f6cubMGXz44YdQKpXo2LEjBg0ahOjoaJw8eRJvvvkm8vLy0KVLF2g0Gpw5cwZubm5Sl2wUmNeaY16rxrzqjrFklnk1HpLmVRdzIpIQaWlpIiwsTGg0GjF9+nQRHBysXbd161YRHBxsUPPLXrhwQfvz9u3bRWBgoDhy5IjIz88X58+fF0OHDhVXrlzRa02nT58WQ4YMEbm5uUIIIY4cOSImTZok3n//fbFgwQLh4eEh9u3bp9eapHbjxg2Rl5cnLl26JL799lsREBAg/v77byHEvX+3IUOGiKtXr2q3ry/zwz4u5vXxMa+VMa+6Y0yZZV6Ng1R55aUpOlBWVoY2bdpg5syZKC4uRl5eHlauXAng3p+sMjIycOLECbz88svSFvovLVu21P6cnp6OVq1awdHREZaWlrCzs4OFhQUKCgr0WpO9vT2ioqJgZmaGW7duYffu3XB1dYW7uztUKhX+/PNPfPXVV3B2doaVlZVea5OKra0tACAxMREbNmzAt99+i//973+IiYnB1q1b8c4771T4ZjCZTFZvzmg8Kua1djCvlTGvumFsmWVejYNUeWUjrgMmJvdmhdywYQPkcjmys7ORlJSEZ555BqdOncLx48cxe/ZsdOjQAcePH4elpSU6dOgg6Rtw+QtKo9EgMzMTnTt3hqWlJfLz8/HLL7+gbdu2aNasGYB7Ux41bdpUL/Wamd17iTZs2BCZmZkYOHAgVCoVgHt/StJoNGjUqJFOazBErVq1gkqlglqtRkxMDKKiojBp0iRYW1tj0qRJaNKkCbp06YKRI0fyl3oVmNfaw7zeH/Nau4wts8yrcdF3XtmI60hJSQnOnj2LPn36YOXKlVi6dCnkcjmysrIwadIkuLm5ITo6GlevXkVCQgICAwPRq1cvlJWVad9k9E0mk8HMzAweHh6YO3cuzMzMcPz4ce1NHjExMfjzzz9RXFyMkSNH6vVsw507d1BQUKC9wzwpKQm5ubno0aMHsrKykJOTg4YNG6JVq1b14oxSy5YtsWrVKiQnJ+Orr75CSEgIZDIZ3nzzTUybNg3t2rVDeHg4AGDkyJESV2v4mNfaxbxWxLzWPmPLLPNqPPSdV96sqUOpqamYMWMGXn31VaSnp+PkyZOYOnUqTExM8MMPP8Db2xtdu3ZFRkYGxo8fj3Xr1sHBwQEAoFarYW5uLlntv/zyC3755Rc0btwYTzzxBFJTU9G8eXM0b94cbdq0wfvvv4+PP/5YW68+JCQkYMmSJejSpQvy8/OhUqlQWloKmUyGtLQ0FBQUYMqUKXB3d9dbTVLLyMjAH3/8gb59+6J///6YPXs2+vfvDwA4ceIEdu3ahdmzZ2u3rw9voo+Kea1dzGtlzGvtMtbMMq/GQV955RlxHbK3t8fKlSvxww8/4JlnnsHkyZPRsWNHeHl5YejQoUhJSUFiYiI6deqErl27oqioCCUlJUhOTsbly5fRt29f7adlfXNzc8MLL7wAc3NzJCUl4ejRoxg0aBC6du0KAOjYsSMuX74MBwcHvfyyEEKgX79+UKlUuHjxIp588kmkpaXh8uXLsLe3x6xZs3Dt2jV88MEH6Ny5syRfSy6FFi1aoEWLFsjMzET37t21bxIAEBsbi5ycHADA6dOn4eDgwGtQH4J5rT3M6/0xr7XLWDPLvBoHfeWVjbiOtWnTBiEhIdrH6enpaNSoEUaMGAGFQoE//vgD77zzDlq3bo02bdpgzpw5uHbtGl5++WWUlJRIWDm00xxdvHgR5ubm2jcJAOjWrRu6dOkCIQRu3boFGxsbnb5hlD+vo6MjHB0dAQC///478vPz4ebmhgYNGsDGxgZ5eXkoLS3VSQ2GzMTEBOfOncNPP/2EwYMHIzIyEleuXMGHH36InTt3YuvWrZg6dSo6d+7MX+4PwbzWDub14ZjX2mOsmWVejYeu88pLUySwc+dOfPfdd+jVqxeOHz8OjUaDzz//HGfPnkV4eDguXLiAZcuW4fnnn5e6VABAcnIygoODsXz5cpw7dw6DBg0CALz//vto1KgRzp07hzfffFNv38ZVVlaG0tJShIWFoUuXLvD29kZJSQnkcjnOnj0L4N4n/NLSUslvqtOns2fP4r333kOLFi2Qnp6OhQsX4uTJkzhy5AjkcjmEEOjSpQveeOMNqUs1Kszr42Fe74951R1jyizzahx0mtdamQSRqq183snTp08LDw8P8corr4iSkhJx7NgxERwcLNavXy9SUlJEenp6pX2kFB8fL6ZPny7eeOMNkZ6eLsaOHSvs7e3FN998Iy5evCi8vLzEiRMn9FrTwYMHRZ8+fcT69etFaGioEOLeXJ/du3cXmzZtEr6+vmL//v16rUlqOTk54s6dO0IIIaKjo8XMmTPFjh07xN27d8WtW7fEoEGDKsxpSw/HvNYe5rUy5rX2GWNmmVfjoKu88tIUPSv/5Ghvb4+FCxeidevWOHfuHLZv345WrVph+PDhSE5OxpEjR5CRkYE+ffrghRdekPRTpxAC7u7ueOmllyCXyxEVFQW1Wo2jR49i4sSJaNWqFd58800UFRXpta4XX3wRixcvRnJyMpydnbF9+3YsXLgQzZs3h7W1NVavXo133nkHDg4O2imZ6jobGxsAwI8//og9e/Zg8ODBeOGFF9CwYUNcvHgRd+/ehYWFRYV9pHxtGTrmtfYwr5Uxr7XP2DLLvBoPneW1dj4n0ONYsWKFmDlzphDi3jeC9enTR0ydOlUcPHhQeHl5Gdynzq1bt4oVK1YIIYS4dOmS8PPzE/b29iI5OVnk5eWJvLw8IYT+zzJMnz5dJCQkCI1GI4YNGya2bdsmPvjgA+03h9Un169fF3FxceLmzZtCCCFSU1OFk5OT2LBhg7h27ZrYuXOn+Pnnn7XbS31GyJgwr7WDef0H86pbxpRZ5tXw1XZepZkAlyro1q0bLCwscO7cOcTExCAoKAhZWVmQy+VYunQp/vzzT2g0GqnL1HryySexd+9eHDx4EBYWFoiIiMCaNWvQsGFDvPnmmwgMDERCQoLezi4IIVBWVgZzc3Pk5eXBxMQEa9aswdq1a3H58mVYW1sjKSkJ58+f125f16lUKri7u6NZs2Y4deoURo4ciYkTJ2L06NFYvHgxMjMzsW7dOmzcuBEAeIatBpjXx8O8Vsa86pYxZZZ5NXy1nlcdfWCgair/pHT37l2RkZEhxo8fL4QQ4vLly2L48OGiZ8+eYs2aNRX20Wg0eq/zvxISEsTrr78uPDw8xN9//y2EEOKrr74Sc+fOFTk5OWLo0KHi5MmTeq1p//79wsvLS6xfv15s3rxZXLt2Tdy6dUvMmjVLTJ8+XXh5eYl9+/bptSYplZWVieLiYvHiiy+Kr7/+Wntt28SJE8W3334rrl27Jvz8/HgNag0wr7WHea2IedUNY8ws82r4ajOvvEZcYuWflBo2bIhr167hxo0bOHPmDJ599lksX74c27Ztw0svvYQTJ04gIyMDdnZ26NKlCwDg7t27sLS01HvNQgj07dsXPXv2hJmZGc6ePYu1a9eiZ8+eiI2NRX5+Puzt7ZGdna3Xmnr16gWZTIa///4bN27cwIgRIwDc+9YwX19ftG/fHtOnT0erVq3QqlUrvdUmFZlMBrlcjp9++gnNmjXDW2+9hQkTJuC1117DTz/9hI4dO2LQoEFo0aKF1KUaDea19mpiXitiXnXD2DLLvBqH2swrG3ED0rp1a0yaNAnvvPMOysrKMHHiRLzyyitYvHgx5HI5AODUqVMIDw9HXl4evv76a4SHh8PKykqvdZa/sZXXlJKSgjt37uC5556Dk5MT/vrrL7z22mswMTFBTk4OcnNz0aZNG53PgyqEgJubG9zc3AAAwcHB6NWrFzw9PREbGwsbGxt07twZzZs31+6ny5oMhY2NDcrKytCwYUNcuXIF7u7uWLFiBczNzTFixAjtvyPVDPP6eDUxr/fHvOqOMWSWeTUutZFXNuIGRPz/u6fbt2+P27dvw87ODhs3bsT//vc/DB8+HO3atcPp06cRGhoKCwsL9O3bV+qSAdy7pi0yMhIDBgxAaWkpjh8/joEDB2L9+vVIT0/HwYMHERQUBDc3N2g0Gu0XGdS2fwdeo9FALpejWbNm6NGjB3744QdYWFjgrbfeQmJiIvLy8qBUKtG9e/c6/2Yhk8kgk8kwadIkhISEICsrC02bNkVGRob2yxuo5pjXx8O83h/zqjvGmFnm1bDVRl55s6YBKX+xPvnkk3juuedgbW2N9PR0dOjQQfunnlOnTuHMmTPw9vaGr6+v3s+u3Y+zszNmzZqFBQsWwMbGRvuJ+OTJkzAzM8PKlSuxdOlSnD9/XvsmkZWVpdOaTE1N4ebmhiVLlmDbtm1o0KABrl27hiZNmuDnn3/G5cuXERYWhgMHDmg/7dd19vb2CAsLQ6NGjdC3b1+8/PLLUpdk1JjX2sO8Vsa81j5jzCzzahweJ688I26ghBDIy8vD9evXoVKpYG5ujkOHDmHnzp0IDAzEoEGD0LBhQ6nL1OrVqxccHBxgY2ODzz//HMnJyfDz88OPP/4IU1NT9O7dW/uGdu7cOXz99deYN2+ezj69CyHQr18/yOVyxMfHw9nZGbm5uZDL5Th69ChCQkLQvXt3fPbZZ+jWrRusrKyQmJiI5s2bo2XLljqpyRC0adMGbdq0kbqMOod5fTzM6/0xr7pjTJllXo3Do+aVZ8QNlEwmQ5MmTeDv749PP/0UH374IUJDQzFgwAD4+PgYzBvEvymVSsjlcty6dQu//fYbevTogbKyMmRmZmLIkCFQKpUIDg6GXC7Hhx9+qLM3CeCfMx9ubm5YuHAhxo0bh9TUVFy9ehXjx4/Hb7/9Brlcjk6dOsHKygr5+fmQyWS4ePGizmqiuot5fTzMK+mbsWWWea272IgbMCEEXn75ZUyfPh1nzpzBqFGj4OnpiSZNmkhd2n39O5y7d+/G5cuXcePGDSQnJ6NNmzaYO3cuzpw5AzMzMxQWFgLQz5yjQgjcvn0bJ0+exPnz5+Hh4YHU1FT873//w5QpU3Do0CF4eXnh2WefRa9evXReD9VNzGvtYF5JX4wps8xr3SUT9eHinTqg/MsHrK2tpS7locpvzEhISEBCQgLy8/OxYsUKfPTRR7h48SLCwsKQmZmJ3bt3w8fHB88++6zebubYv38/li9fjoCAAMybNw979uzB33//jSVLlmDy5MlwcXFBWVkZnnjiiTp/gwnpFvP6+JhX0idjyCzzWjexEadaVx6ysrIymJiYIDQ0VPsmcf36dXz99ddITU3F008/DX9/f718Si6v6ejRozh//jzs7e1haWmJyZMn4/3334eLiwvWr1+PS5cuISAgAI6OjnyzoHqBeSUyHsxr3cNLU6jWlYfLxMQEp0+fxrlz5/DRRx8hMzMTO3bsgK2tLaKiojBnzhwsWbJEL9eNldfUo0cPjBo1CgUFBRg7diyCgoLQtWtXbN++HfHx8WjSpAnCwsKwf/9+vklQvcC8EhkP5rXu4awppFMODg748ssv8ffff2Pr1q1o1KgRJk2ahCZNmiAjIwNqtRoajUbvdZmbm+PDDz9E165dsWvXLvz999+YPn06+vbti99++w2ff/45unTpgqZNm+q9NiKpMK9ExoN5rRvYiJPOyeVyNG7cGHK5HOPHj4etrS3+/PNPrFy5EqNHj0br1q1RVFSEgoIC2Nra6vxPVkIIuLi4oKSkBOvXr0diYiKGDx+O3r17AwC2bt2K5s2b802C6iXmlch4MK/Gj9eIk96o1WqYm5vjjz/+wOrVq9G/f380b94cZ8+exfHjx1FSUoLXX39dr19ckZKSgoyMDO03qM2dOxdlZWUYPXo0mjdvjgsXLsDGxgbt2rXjNW1UrzCvRMaDeTVebMRJbzQaDe7evYtJkybB1dUVL730ErZu3Qo7OzvY29tDqVRi7ty5mDNnjiRf5fzBBx8AAIYOHYqDBw/iypUryM3NxbVr1xAcHMypl6heYV6JjAfzarx4aQrpjampKRo3bowlS5bA2toaqampuHnzJkaNGgUHBwcAQKNGjbBp0yaoVCoolUq91XbmzBnk5eVhwoQJuHr1Ku7evQsvLy/06dMH586dQ2hoKFq0aIF27drprSYiKTGvRMaDeTVenDWF9KqsrAwKhQIZGRk4c+YMnnjiCe2bRGxsLIQQeP755/X+rWbPPvssPvroIzg5OSE1NRUWFhZwdXUFAOTm5uLixYvYu3cvCgoK9FoXkZSYVyLjwbwaJ54RJ70yMTHB7du3sWPHDnTp0gXr16/HiRMncPDgQRQUFGDo0KHo27cvLC0t9V7brVu3kJSUhOvXr8PDwwMWFhZITU3F/v374efnh/79+6NRo0ba7cuvaavP17ZR3ca8EhkP5tU4sREnvSsrK8Ovv/4Kd3d3rFmzBsHBwSgrK8PYsWPRs2dPSd4kACAxMREXLlyAv78/5s+fj9OnT+Pw4cPo2bMnevXqhWeeeQYAUFpaCjMzM+2bQ3FxMRo0aCBJzUS6xrwSGQ/m1fjwZk2SxOnTpxEUFIQOHTrg1q1bGD58OFxdXSV7kwCACxcu4L333sPixYtx8uRJ7Nq1C46Ojhg+fDhatWoFABU+nSckJCA7Oxvnzp3D+++/D7lcLlntRLrEvBIZD+bVuLARJ8lkZmbi6tWraN26NZo0aQJTU1OpS8Lp06exfPlymJiYwM3NDT179sTTTz+tXS+EQEZGBt5//308/fTTcHV1RZcuXWBqagoLCws0a9ZMwuqJdId5JTIezKvxYCNO9B/5+fkAACsrK+2y8j+XJScnY+XKlXjiiSewcOFCAMD169fx7bffYs+ePYiMjESbNm0kqZuoPmJeiYwH81oZG3GiKpSVlcHExATFxcUYN24cOnXqhBkzZgC4Ny3Tzz//jOzsbFhaWuKPP/7AggUL6uSbBZExYF6JjAfzyukLiapkYnIvJqGhoVAqlRXeJBISEnDx4kUMGTIE77//PoYPH46pU6fi6tWrUpZMVG8xr0TGg3llI05ULUVFRWjevDlmzZoF4J83iQsXLmDMmDHo1KkTAGDIkCHo1KkTjhw5ImW5RPUa80pkPOp7Xjl9IVE1nDx5EteuXYOVlRUuX76MHTt24M6dOxg9ejScnJwAAGq1GmZmZrh165b2CxPK/+xGRPrDvBIZj/qeV+MfAZEetG7dGmPHjoWFhQXUajU2b96Mdu3aoUuXLtBoNCgtLYW5uTlOnjyJEydO4KmnnkJ+fj527tyJK1euSF0+Ub3CvBIZj/qeVzbiRNWgUqnQsWNHAECbNm2wePFixMbG4uTJkzA1NYWZmRn++OMPjB07Fq+//jqcnJyQkJCAb7/9FmPGjMH58+clHgFR/cG8EhmP+p5XzppCVEPlXzqwf/9+LF26FL1798bNmzfxyy+/YNy4cRg7diyio6Nx8OBBjBgxAjdu3MD333+PsLAwtG7dWuryieoV5pXIeNTHvLIRJ3oMJ0+eREpKCszNzdGmTRt07doV27ZtQ1RUFHJychAfHw8A+PHHH/Hdd99h5cqVdW7qJSJjwbwSGY/6klc24kS1JD8/HwkJCfj111/Rt29fnD59GmfPnsXq1athbm6OH3/8EQ0bNkSfPn3QqFEjqcslqteYVyLjUZfzykac6DEJIVBaWoqYmBj83//9H/z9/dGvXz8AwJIlS3D69Gl8+eWXMDExwbVr1xAZGYnXX38dzzzzjMSVE9U/zCuR8agPeeXNmkSPSSaTwdzcHE899RT+/vtv2NnZadfNnDkTDg4O+PvvvwEATzzxBLp164b33nvP6G8wITJGzCuR8agPeeUZcaJaUH6DyYEDB7Bq1SrMnj0b3bp1q7DNv+c8nT9/PuLj4/Hdd98Z7Q0mRMaKeSUyHnU9rzwjTlSLevXqhenTpyM0NBQnTpyosK78TWLfvn04evQoXnrpJaP75E5UlzCvRMajruaV36xJVAtkMpn25169ekEIgb179+LZZ5+FlZWVdt2+ffuwcOFCTJ06Fb6+voiOjkZCQgJGjx5dYTsi0h3mlch41PW88tIUolpW/me0O3fuoEmTJtrl+/btw6JFizBlyhT4+vpql+fm5uL333/Hiy++aNBvFkR1EfNKZDzqYl55aQpRLSv/9P7vN4mEhASEhYVVeJPQaDQAADMzM/zf//0fMjMz9V4rUX3HvBIZj7qYVzbiRDqWk5ODRYsWYdy4cdo3CSEETE1NIYTA/v370b59e6P8IgKiuoZ5JTIedSGvvEacSMeaNWuGTZs2QaVSAbj3JlH+57WSkhIcOnSo0h3gRCQN5pXIeNSFvPIacSI9KJ9aSaPRwNTUVLt8ypQpuHPnDr799tsKN6QQkXSYVyLjYex55RlxIj0on1rJ1NQUhYWFyMzMxJo1a5Ceno6tW7dCJpNVmAeViKTDvBIZD2PPKxtxIj1JT09HRkYGNm3aBIVCAUtLS2zbtg0ymazSJ3kikhbzSmQ8jDmvvDSFSE9u3bqFnTt3onHjxnjxxRfRrFkzADDoT+pE9RXzSmQ8jDmvbMSJ9Ki0tBRmZv/8Iar8phIiMjzMK5HxMNa8shEnIiIiIpKAYZ+vJyIiIiKqo9iIExERERFJgI04EREREZEE2IgTEREREUmAjTgRERERkQT+H61XiAWq0+qMAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {} + } + ], + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 356 }, - { - "cell_type": "markdown", - "metadata": { - "id": "xtvW54LtB8hp" - }, - "source": [ - "In this tutorial we have explored in depth a health care scenario through all stages of the AI lifecycle except the model deployment stage. We have seen how fairness-related harms can arise at the stage of task definition, data collection, model training, and model evaluation. We have also seen how to use a variety of tools and practices, such as datasheets for datasets, Fairlearn, and model cards.\n", - "\n", - "Once the model is deployed, it is important to continue monitoring the key metrics to assess any performance difference as well as the potential for fairness related harms. As you learn more about how the model is used, you may need to revise the fairness metrics, update the model, consider additional sensitive features, update the task definition, or collect new data.\n", - "\n", - "Although we used a variety of software tools, fairness is a sociotechnical challenge, so mitigations cannot be purely technical, and need to be supported by processes and practices, including government regulation and organizational incentives.\n", - "\n", - "If you would like to learn more about fairness of AI systems, or to contribute to Fairlearn, we welcome you to join our community. Fairlearn is built and maintained by contributors with a variety of backgrounds and expertise.\n", - "\n", - "Further resources can also be found [on our website](https://fairlearn.org/main/user_guide/further_resources.html)." + "id": "_qXEkByRDkK0", + "outputId": "36ac6599-3113-4862-b5a0-7fe90bc71a5b" + } + }, + { + "cell_type": "markdown", + "source": [ + "### Randomized predictions" + ], + "metadata": { + "id": "7S6CIoJ-W1jH" + } + }, + { + "cell_type": "markdown", + "source": [ + "Both the `ExponentiatedGradient` and the `ThresholdOptimizer` yield randomized predictions (may return different result given the same instance). Due to legal regulations or other concerns, a practitioner may not be able to deploy a randomized model. To address these restrictions:\n", + "\n", + "* We created a deterministic predictor based on the randomized thresholds learned by the `ThresholdOptimizer`. This deteministic predictor achieved similar performance as the `ThresholdOptimizer`.\n", + "* For the `ExponentiatedGradient` model, we can deploy one of the deterministic inner models rather than the overall `ExponentiatedGradient` model.\n", + "\n" + ], + "metadata": { + "id": "MzdFLJ9BA15F" + } + }, + { + "cell_type": "markdown", + "source": [ + "### Access to sensitive features\n", + "\n" + ], + "metadata": { + "id": "P9dvkQvKW1lv" + } + }, + { + "cell_type": "markdown", + "source": [ + "\n", + "\n", + "* The `ThresholdOptimizer` model requires access to the sensitive features during BOTH training time and once deployed. If you do not have access to the sensitive features once the model is deployed, you will not be able to use the `ThresholdOptimizer`.\n", + "* The `ExponentiatedGradient` model requires access to the sensitive features ONLY during training time. \n", + "\n", + "\n" + ], + "metadata": { + "id": "L56tawsAW14B" + } + }, + { + "cell_type": "markdown", + "source": [ + "# Model cards for model reporting" + ], + "metadata": { + "id": "fQOR2R3XVa-U" + } + }, + { + "cell_type": "markdown", + "source": [ + "_Note: The Python code in this section works in Google Colab, but it does not work on all local environments that we tested._" + ], + "metadata": { + "id": "MzFzSHrQBAgi" + } + }, + { + "cell_type": "markdown", + "source": [ + "[Mitchell et al. (2019)](https://arxiv.org/abs/1810.03993) proposed the *model cards* framework for documenting and reporting model training details and deployment considerations. A _model card_ documents, for example, training and evaluation dataset summaries, ethical considerations, and quantitative performance results.\n", + "\n" + ], + "metadata": { + "id": "2RAXJDoyVbBT" + } + }, + { + "cell_type": "markdown", + "source": [ + "### Fill out the model card [OPTIONAL SECTION]" + ], + "metadata": { + "id": "oP_tklbmMoFH" + } + }, + { + "cell_type": "markdown", + "source": [ + "In this section, we will create a model card for one of the models we trained." + ], + "metadata": { + "id": "QfI9oEwIkS6v" + } + }, + { + "cell_type": "code", + "execution_count": 99, + "source": [ + "mct = ModelCardToolkit()\n", + "model_card = mct.scaffold_assets()\n" + ], + "outputs": [], + "metadata": { + "id": "OViYhOAHPftT" + } + }, + { + "cell_type": "markdown", + "source": [ + "The first section of the Model Card is the _model details_ section. In _model details_, we fill in some basic information for our model.\n", + "\n", + "\n", + "* _Name_: A name for the model\n", + "* _Overview_: A brief description of the model and its intended use case\n", + "* _Owners_: Name of individual(s) or group who created the model.\n", + "* _References_: Any external links or references\n", + "\n" + ], + "metadata": { + "id": "Lnr6aRjIF_t6" + } + }, + { + "cell_type": "code", + "execution_count": 100, + "source": [ + "model_card.model_details.name = \"Diabetes Re-Admission Risk model\"\n", + "model_card.model_details.overview = \"This model predicts whether a patient will be re-admitted into a hospital within 30 days.\"\n", + "model_card.model_details.owners = [{\n", + " \"name\": \"Fairlearn Team\",\n", + " \"contact\": \"https://fairlearn.org/\"\n", + "}]\n", + "model_card.model_details.reference = [\n", + " \"https://archive.ics.uci.edu/ml/datasets/Diabetes+130-US+hospitals+for+years+1999-2008\"\n", + "]\n", + "model_card.model_details.version.name = \"v1.0\"\n", + "model_card.model_details.version.date = str(date.today())\n", + "model_card.model_details.license = \"MIT License\"\n" + ], + "outputs": [], + "metadata": { + "id": "YKvUW25cPewp" + } + }, + { + "cell_type": "code", + "execution_count": 101, + "source": [ + "model_card.considerations.use_cases = [\"High-Risk Patient Care Management\"]\n", + "model_card.considerations.users = [\"Medical Professionals\", \"ML Researchers\"]\n", + "model_card.considerations.limitations = [\n", + " \"\"\"\n", + " This model will not generalize to hospitals outside of the United States. Features, such as those encoding insurance\n", + " information, are inherently tied to the U.S healthcare system.\n", + " In addition, this model is intended for patients who are admitted into U.S. hospitals for diabetes-related illnessess.\n", + " \"\"\"\n", + "]\n", + "model_card.considerations.ethical_considerations = [{\n", + " \"name\": (\"Low sample sizes of certain racial groups could lead to poorer performance on these groups\"),\n", + " \"mitigation_strategy\": \"Collect additional data points from more hospitals.\"\n", + "}]" + ], + "outputs": [], + "metadata": { + "id": "avEdkdzAPe2S" + } + }, + { + "cell_type": "markdown", + "source": [ + "The next two sections of the model card are meant to provide the reader with information about the data used to train and evaluate the model. For each of these sections, we provide a brief `description` of the data and then submit a `visualization` of the distribution of labels in the dataset." + ], + "metadata": { + "id": "orPNYAZFwGVM" + } + }, + { + "cell_type": "code", + "execution_count": 102, + "source": [ + "model_card.model_parameters.data.train.graphics.description = (\n", + " f\"{X_train_bal.shape[0]} rows with {X_train_bal.shape[1]} features. \"\n", + " f\"The original training data set was undersampled to allow for an equal number of positive and negative labeled instances.\"\n", + ")\n", + "\n", + "model_card.model_parameters.data.train.graphics.collection = [\n", + " {\"name\": \"Sensitive Features\", \"image\": sensitive_train},\n", + " {\"name\": \"Target Label\", \"image\": outcome_train} \n", + "]" + ], + "outputs": [], + "metadata": { + "id": "khWJN_vqm35e" + } + }, + { + "cell_type": "code", + "execution_count": 103, + "source": [ + "model_card.model_parameters.data.eval.graphics.description = (\n", + " f\"{X_test.shape[0]} rows with {X_test.shape[1]} columns\"\n", + ")\n", + "\n", + "model_card.model_parameters.data.eval.graphics.collection = [\n", + " {\"name\": \"Sensitive Features\", \"image\": sensitive_test},\n", + " {\"name\": \"Target Label\", \"image\": outcome_test}\n", + "]" + ], + "outputs": [], + "metadata": { + "id": "h3fiatJ6nGQf" + } + }, + { + "cell_type": "markdown", + "source": [ + "In the last section, we fill out the `quantitative_analysis` section where we describe the model's performance metrics on the evaluation dataset. In particular, we want to report the model's disagregated performance with respect to our three metrics including false negative rate, which quantifies fairness-related harms." + ], + "metadata": { + "id": "frGlrXHAyn2C" + } + }, + { + "cell_type": "code", + "execution_count": 104, + "source": [ + "def metricframe_to_dictionary(mframe, feature_name):\n", + " \"\"\"\n", + " Converts a MetricFrame into a Dictionary object that can be accepted by the Model Card's\n", + " Quantitative Analysis section.\n", + " \"\"\"\n", + " group_metrics = mframe.by_group[feature_name].reset_index()\n", + " group_metrics = group_metrics.melt(id_vars=\"race\", var_name=\"type\", value_vars=feature_name).rename(columns={\"race\":\"slice\"})\n", + " return group_metrics.to_dict(orient=\"records\")" + ], + "outputs": [], + "metadata": { + "id": "1cyHIMlAJqod" + } + }, + { + "cell_type": "code", + "execution_count": 105, + "source": [ + "model_card.quantitative_analysis.graphics.description = (\n", + " f\"These graphs show the models performance on the test dataset for disagregated racial categories.\"\n", + ")\n", + "model_card.quantitative_analysis.performance_metrics = metricframe_to_dictionary(metricframe_postprocess, \"false_negative_rate\")\n", + "model_card.quantitative_analysis.graphics.collection = [\n", + " {\"name\": \"ThresholdOptimizer\", \"image\": postprocess_performance}\n", + "]" + ], + "outputs": [], + "metadata": { + "id": "kETWDtN68Gmq" + } + }, + { + "cell_type": "code", + "execution_count": 108, + "source": [ + "postprocess_performance" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/plain": [ + "'iVBORw0KGgoAAAANSUhEUgAAAtgAAAFNCAYAAAA6kBhoAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8rg+JYAAAACXBIWXMAAAsTAAALEwEAmpwYAABJvElEQVR4nO3dd1hTZ/8/8HcAQREVVMC6Vx8QBbeitVoXKBLFraC4HkdRUVqtu+Jq1ao4HiffLitWbKsiVhHR4lNlKLUVt+K2CmEoyg7k/v3hjzxaB8MDJ4H367p6XSY54XzuJu/cn5ycoRBCCBARERERkSQM5C6AiIiIiKgsYYNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYRFRuLF68GD169ICfn99rH4+Ojoarq2spVyWt2NhYfP755wCACxcuwNvbW2fqISqq4mTSxsYGKSkpJVRRwb7++mvMnTtXtvWTbmCDrSc2bdqEpUuXFvv5CxcuxMWLFwEACxYsQEREhFSlFQsnXZJDYGAgdu/eDR8fH7lLKTFxcXFISEgAANjb22Pjxo06Uw8RUXlhJHcBVDoiIiIwfPhwAMCKFStkroaTLpU+d3d3CCEwceJE9OnTB7///jtycnKQkpICNzc3zJw586XlY2JisHLlSmg0GgDA5MmT4ezsjJycHKxZswZnz55FXl4e7OzssHDhQpiZmb1x3dHR0fDz80O9evVw48YN5OTk4PPPP4ejo+Nb/15sbCx8fX2hVqtRv359PHz4EHPnzkX79u3xxRdf4Pz580hPT4cQAsuXL0ft2rWxceNGPHv2DPPmzYObmxuWLVuGH3/8Ed26dcPRo0dhaWkJABg2bBimTp2KTp06FWs8K1asgKmpKTIyMvDzzz9j9erVBdbz5Zdf4sSJE9i6dSvUajUqVqyIOXPmoHXr1u/+AlOZlZGRAW9vb9y9exdVq1bVbmxaunQpMjIyoFKpYGtri/Xr18PExOSl5/n6+uLOnTtITU1F5cqVsWbNGjRu3BijR49Gq1atcO7cOTx69Aht27bFqlWrYGBggN9++w3r16+HRqOBqakplixZAltbW5w7dw5r1qxBZmYmFAoFpk+fju7du0OtVmP58uWIiIhAjRo1UKNGDVSpUqXAcW3btg1hYWHIzs5GZmYm5syZg969eyM3NxdfffUVwsPDYWhoiNatW2Px4sUwMDB47f3bt2/H48ePtRutNm3apL09evRoVKtWDbdu3cLIkSNhb2+Pr776Cjk5OUhMTETnzp3xxRdfAMBrx/3bb78hLi4Oa9euBQD88ccfWLZsGQ4cOCDxq1wGCZJFWlqamD59uujfv79wc3MTCxYsEHl5eeL48eNiyJAhYsCAAWL48OHi3LlzQgghNm7cKJYsWSKEECI+Pl54eXmJgQMHCldXV7F161bt3z1x4oTo37+/cHV1FcOGDRNXrlwR69atE82bNxdOTk7ir7/+EqNGjRJHjhwRQghx7NgxMWDAAOHq6ipGjBghzp8/r13fnDlzxPjx44Wzs7MYOXKkiI+Pf+uYoqKihFKpFMOHDxdKpVJkZ2eLZcuWiSFDhoi+ffuKPn36iJiYGPHw4UPRrVs30aZNGzF37lwhhHjjuImk9K9//UskJyeLUaNGidu3bwshnuepWbNmIjk5WURFRYl+/foJIYTw9PQUhw4dEkIIceXKFeHr6yuEEGLTpk1i5cqVQqPRCCGEWLt2rVi8ePFb1xsVFSWaNWsmLl++LIQQ4uuvvxYeHh5v/XtqtVp07dpVhIeHCyGEiIyMFDY2NiIqKkqcO3dOTJ8+XeTl5QkhhNi+fbuYPHmyEEKIX375RUyaNEm73vzxfPbZZ+L//u//hBBCxMXFiY8++kjk5eUVezy2trbiwYMHQghR6Hpu374tXF1dRUpKihBCiOvXr4sPPvhApKenv3V9VH7lv9f++OMPIYQQe/bsEUOGDBErV64UBw4cEEIIkZOTI1xdXUVISIgQ4n85P3LkiFi2bJn2by1atEgsXbpUCCHEqFGjhLe3t8jLyxPPnj0TXbp0EZGRkSIxMVG0bdtWm9WjR4+KCRMmiCdPnggnJydx//59IcTzz42uXbuKv//+W3z33XfC09NTZGdni/T0dDFw4EAxZ86ct47rwYMHYvTo0SIzM1MIIcShQ4eEq6urEEKI77//Xnh4eIjMzEyRl5cnZsyYIfbv3//G+1/sD4R4uV8YNWqUmDdvnvYxHx8fERUVJYR43od07NhRXLhw4Y3jTkpKEm3atBGPHz8WQggxe/Zs8eOPPxbpNSyvuAVbJseOHUN6ejqCgoKQl5eHxYsX4969e/Dz88POnTthYWGBGzduYNy4cQgNDX3pubNnz8bYsWPRo0cPZGdnY+LEiahfvz46dOiA2bNn44cffkCzZs0QGhqKNWvW4P/+7/8QHByMNWvWwN7eXvt3bt68icWLF2PPnj2oV68eIiMj4eXlhZCQEADPt+AdOHAAZmZmmDJlCgIDAwvcn/PGjRsICwtDnTp18Oeff0KlUiEwMBAGBgbYsWMH/P39sW3bNnh7e+Po0aP48ssvcefOnTeO29TUVPr/+VTubdu2DeHh4Th06BBu3rwJIQQyMzNfWqZv375YunQpTpw4gc6dO+OTTz4BAISHh+PZs2fa3azUajVq1KhR4Dpr166NZs2aAQDs7Oywf//+t/6969evAwC6desGAHB0dMT7778PAGjdujWqVauGPXv24P79+4iOjkblypXfuv6hQ4diyZIlmDBhAn755RcMGjQIBgYGxR7Pe++9hzp16hSpntOnT0OlUmHs2LHa+xQKBe7duwdbW9sC10nlk42NDdq0aQMAGDhwIHx9ffHNN9/gr7/+gr+/P+7cuQOVSoWMjIyXntenTx/Uq1cPP/zwA+7evYszZ8689GtJ9+7dYWBgADMzMzRo0ACpqak4d+4c3n//fW1WnZyc4OTkhJMnTyIxMRFTp07VPl+hUODatWuIjIyEq6srjI2NYWxsDKVSiWvXrr11THXq1MGqVasQHByMu3fvan/9AZ7/4jxgwABUrFgRALB+/XoAwJQpU157/6ZNm966rnbt2mn/vXLlSvz3v//Ftm3bcOvWLWRlZSEjI+ON4waAjz76CEFBQXBzc8OpU6ewePHit66PnmODLZO2bdvCz88Po0ePRufOnTFmzJi3Tj75MjIycPbsWaSmpmLDhg3a+65evQojI6M3BuR1oqKi4OjoiHr16gEAOnXqhOrVq2v31e7QoYP2Z2I7OzukpqYWOC5OuqTrMjMzMWLECPTq1Qvt2rXD4MGDERYWBiHES8uNGDEC3bt3x+nTp/H777/jP//5Dw4ePAiNRoP58+drG9/09HRkZ2cXuN78SRF4/v7OX9+b/l5iYuIrNRkaGgJ43pSvWLEC48aNQ8+ePdG4cWMcPHjwretv164dcnNzERsbi0OHDmHPnj1vXX9BXvzyW9h6NBoNOnXqpG0MAODRo0ewsrIqcH1UfhkYvHy4mEKhwIIFCyCEQN++ffHRRx/h0aNHr+Rl9+7d2Lt3Lzw8PKBUKmFubo4HDx5oH39dJo2MjKBQKLT3CyFw7do15OXloUmTJvjpp5+0jyUkJKB69eoIDAx8ab35OX2bS5cuwcvLC2PHjsUHH3yA9u3bY8mSJQAAI6OXW7OkpCRoNJo33v/i5wnw/Evyi17MqoeHB2xtbfHhhx+ib9++OH/+PIQQMDQ0fO24bW1t4eHhAV9fXxgZGcHJyanAL/P0HA9ylEm9evVw7NgxTJo0CWlpaRg3bhyePHmCTp06ISgoSPvf3r17tVutgOcTlBACe/bs0S4TGBiIyZMnvzYgV69efWMN//wwyr8vNzcXwJsbgrf556Q7efJkAEDPnj0xcuTI1z4nf9J927iJpJKSkoK0tDTMnDkTPXr0wJkzZ5CTk6Pd1zrfiBEjcOXKFQwaNAjLli3D06dPkZqaii5duiAgIED7nEWLFmHdunXFrudNf69JkyYwNjbGf//7XwDPDwy+fv06FAoFTp8+je7du8Pd3R329vYICwtDXl4egOeTe36G/2no0KFYtmwZbGxsULt27beuvygKW4+joyNOnz6NmzdvAgBOnjyJ/v37F6qhp/Lr2rVruHLlCoDnByq3bdsWERERmDp1KlxcXKBQKHD+/Hntey7fqVOnMHDgQAwdOhSNGjXCiRMnXlnmn1q2bImbN2/ixo0bAIDjx49j9uzZaNWqFe7evYuzZ88CAK5cuQJnZ2eoVCp8+OGHOHDgALKzs5GdnY3Dhw8XOKazZ8+iRYsWGDduHDp06IDjx49ra+vUqRMOHTqkzaSvry9+/fXXN95vYWGBS5cuQQiBjIwMnDp16rXrTE1NxcWLFzFr1iw4OTkhISEB9+7dg0ajeeO4AaBNmzYwMDDA119//cZ5nF7FLdgy2b17N/744w+sWbMGH374IZKTk7U/Nd28eRNNmjTByZMnMWvWLJw8eVL7PDMzM7Rq1QrffvstvLy88PTpU4wcORJTp05F586dtQF5//33cfz4cWzYsAHBwcGvnXQdHR3xn//8B/fv39fuIvLo0SO0bNkSf/755zuP8cVJNzs7G/7+/m+cdDdu3PjacXMXEZJa7dq18dFHH6Fv376oWrUq6tevj6ZNm+Lu3bswNjbWLjdr1ix88cUXWL9+PQwMDDBt2jTUrVsXXl5eWLVqFQYOHIi8vDw0a9bsnU7J9aa/Z2RkhE2bNmHx4sVYt24dGjZsiJo1a6JixYoYMWIEZs2aBaVSCUNDQ7Rr1w6hoaHQaDRo3bo11q9fj6lTp8LT0/Oldbm5uWHdunUvNdBSjKew9WzevBlLly7FJ598ot1auHXrVuac3qpx48bauapGjRpYuXIlwsPDMXXqVFSrVg2VKlVC+/btX/q1FwDGjx+Pzz//HPv27YOhoSGaN2+u3fXqTWrWrIk1a9Zgzpw5yMvLg5mZGfz8/FC9enVs3LgRq1evRnZ2NoQQWL16NerUqYMRI0bg3r17cHV1hbm5ORo0aFDgmFxdXREaGgoXFxdUqFABnTp1QmpqKtLS0jBixAj8/fffGDRoEIQQ6NChA0aPHg2FQvHa+zMzM/H777/DyckJ1tbWaN269Ws3iFWrVg2TJk3CwIEDYW5uDgsLC7Rp0wZ3797VHuz8z3HnGzRoEA4fPgwbG5tCvmqkEIXZLEmSy8jIwPz583Ht2jVUqlQJtWvXxooVKxAREYFt27ZpJ5/58+ejXbt2Lx0V/ODBAyxbtgwPHz5ETk4OXF1dMX36dADA77//Dj8/P21AlixZgqZNm2LVqlU4cuQIli9fju3bt8PDwwN9+vTBkSNHsG3bNuTl5aFixYqYO3fuK+sD8Mrt14mOjsayZctw6NAhAM/38Z41axZyc3NfmnTDw8Px4MEDjBkzBnZ2dti8ebO2jn+Om6g8W7VqFSZMmICaNWvi0aNHGDBgAMLCwlC1alW5SyOiciI3NxfTpk1D//794eLiInc5eoMNNhGRBGbOnInbt2+/9jE/Pz80bty4yH9z165d2LNnD4yMjCCEwNSpU996XIWUSmI8ROXJF198gejo6Nc+Nm/ePDg6OpZyRUUXFxeHkSNHomvXrvjqq69e2R+e3owNNhUJJ10iIiKit2ODTUREREQkIW7rJyIiIiKSEBtsIiIiIiIJ6cRp+h4/TodGo797qtSoYYbk5DS5y3hnZWEc+j4GAwMFLCx0/yT+zKz8OAb5Ma+lQ9/fJ0DZGAOg/+MozczqRIOt0Qi9Dj8Ava8/X1kYR1kYg65jZnUDx0CFwbzqhrIwBqDsjKOkcRcRIiIiIiIJscEmIiIiIpIQG2wiIiIiIgmxwSYiIiIikhAbbCIiIpkEBwfDxcUFvXv3RkBAwCuP37p1C6NHj0b//v0xYcIEpKamylAlERUVG2wiIiIZJCQkwM/PD7t370ZQUBACAwMRFxenfVwIgY8//hgTJ07EwYMH0axZM+zYsUPGiomosNhgExERySAiIgKOjo4wNzeHqakpnJ2dERISon380qVLMDU1RdeuXQEAU6ZMgYeHh1zlElER6MR5sImIiMoblUoFS0tL7W0rKyvExsZqb9+7dw81a9bEnDlzcPnyZfzrX//CokWLiryeGjXMJKlXTpaWVeQu4Z2VhTEAZWccJY0NNpVrVapWQkWTko1BVnYunj3NLNF1EBXEopoxjIxNSnQduTnZeJyaU6LrKEuEePWCHQqFQvvv3NxcnDlzBrt27YK9vT3Wr1+PlStXYuXKlUVaT3JyWqEvDlLF3AQVKxgX6e8XVZY6B8+eZBd6eUvLKkhMfFaCFZW8sjAGQP/HYWCgKLUvnGywqVyraGIE5adBJbqO4LUDoL8fR1RWGBmb4NaKwSW6jsYLfgHABruwrK2tERMTo72tUqlgZWWlvW1paYkGDRrA3t4eAODq6gpvb+8SraliBWMMC/y4RNexd/hWPEPhG2wifcR9sImIiGTQuXNnREZGIiUlBZmZmQgNDdXubw0ArVu3RkpKCq5evQoAOHHiBJo3by5XuURUBNyCLZOS3jWBuyUQEek2a2tr+Pj4wNPTE2q1GkOGDIGDgwMmTpwIb29v2NvbY/PmzVi4cCEyMzNRq1YtrF69Wu6ydZ4u7uZC5Q8bbJmU9K4J3C2BiEj3KZVKKJXKl+7z9/fX/rtly5b4+eefS7ssvcbdXEgXcBcRIiIiIiIJscEmIiIiIpIQG2wiIiIiIglxH2wiItIbJX0+b57Lm3RFSR+syQM1SxYbbCIi0hslfT5vnsubdEVJH6zJAzVLFncRISIiIiKSEBtsIiIiIiIJscEmIiIiIpIQG2wiIiIiIgnp5UGOvMw4ERERkW4rz2dC0csGm5cZJyIiItJt5flMKNxFhIiIiIhIQmywiYiIiIgkxAabiIiIiEhCbLCJiIiIiCTEBpuIiIiISEJ6eRYR0g0lfbpEgKdMJCIiIv3DBpuKraRPlwjwlInFERwcjK1bt0KtVmPs2LHw8PB47XLh4eFYunQpTpw4UcoVEhERlW2F2kUkODgYLi4u6N27NwICAt64XHh4OHr06CFZcURUNAkJCfDz88Pu3bsRFBSEwMBAxMXFvbJcUlISVq1aJUOFREREZV+BDTYnbCL9ERERAUdHR5ibm8PU1BTOzs4ICQl5ZbmFCxdi2rRpMlRIRERU9hW4i8iLEzYA7YT9z8k5f8Jeu3ZtiRRKRAVTqVSwtLTU3rayskJsbOxLy+zcuRN2dnZo2bJlsddTo4ZZoZfV5ObAwKjkLpVb3L9vaVmlBKopXbo4hqLWVBbGQET0TwU22KUxYRdlsi4t5XWSKAvjKAtjKC4hxCv3KRQK7b+vX7+O0NBQfPfdd4iPjy/2epKT06DRvLqu17G0rIJbKwYXe10FabzgFyQmFm1PfUvLKkV+jq4p6hhK6z1Y1JqK89qVtMLWZGCg0Mn5i4jkV2CDXRoTdlEma0C3PmAB/Z8k8pXHCVvXxvCuE7a1tTViYmK0t1UqFaysrLS3Q0JCkJiYiMGDB0OtVkOlUsHd3R27d+8u9jqJiIjoZQXug21tbY2kpCTt7bdN2JMmTdJO2ERU+jp37ozIyEikpKQgMzMToaGh6Nq1q/Zxb29vHD16FEFBQdixYwesrKzYXBMREUmswAabEzaR/rC2toaPjw88PT3h5uYGV1dXODg4YOLEibhw4YLc5REREZULBe4i8uKErVarMWTIEO2E7e3tDXt7+9Kok4gKSalUQqlUvnSfv7//K8vVrVuX58AuJItqxjAyNinRdeTmZONxak6JroOIiEpHoS40wwmbiMozI2OTEj1QE3h+sCbABrs8KujiUP/5z3/wyy+/oGrVqgCAYcOGvfECUkSkG3glRyIiIpnkX2ti3759MDY2xogRI9CxY0c0bdpUu8zFixexbt06tG7dWsZKiagoCnUlRyIiIpJeYS4OdfHiRfj7+0OpVGLp0qXIzs6WqVoiKixuwSYiIpJJQdeaSE9PR7NmzTBnzhzUqVMHc+fOxZYtW+Dj41PodejiubrL4/UNysIYivuckqRr9eRjg01ERCSTgq41Ubly5ZeOeRo/fjzmz59fpAa7qBeGKg3l6foGAK+XUZKKUk9pXhyKu4gQERHJpKBrTTx8+BA///yz9rYQAkZG3DZGpOvYYBMREcmkoGtNVKxYEV999RXu378PIQQCAgLQu3dvGSsmosJgg01ERCSTgi4OVb16dSxduhQff/wx+vTpAyEExo0bJ3fZRFQA/s5EREQko4KuNeHs7AxnZ+fSLouI3gG3YBMRERERSYgNNhERERGRhNhgExERERFJiA02EREREZGE2GATEREREUmIDTYRERERkYTYYBMRERERSYgNNhERERGRhNhgExERERFJiA02EREREZGE2GATEREREUmIDTYRERERkYTYYBMRERERSYgNNhERERGRhNhgExERERFJiA02EREREZGE2GATEREREUmIDTYRERERkYTYYBMRERERSYgNNhERERGRhNhgExERERFJiA02EREREZGE2GATEREREUmIDTYRERERkYTYYBMRERERSYgNNhERkUyCg4Ph4uKC3r17IyAg4I3LhYeHo0ePHqVYGRG9CyO5CyAiIiqPEhIS4Ofnh3379sHY2BgjRoxAx44d0bRp05eWS0pKwqpVq2SqkoiKg1uwicqYgraIHTt2DEqlEv369cPcuXORk5MjQ5VEFBERAUdHR5ibm8PU1BTOzs4ICQl5ZbmFCxdi2rRpMlRIRMVVqC3YwcHB2Lp1K9RqNcaOHQsPD4+XHj927Bg2btwIjUYDe3t7LF26FMbGxiVSMBG9WUFbxDIyMrB06VLs378fNWvWhI+PD/bv34/hw4fLXDlR+aNSqWBpaam9bWVlhdjY2JeW2blzJ+zs7NCyZctir6dGDbNiP7ekWFpWKdHlS0N5HENxn1OSdK2efAU22JywifTHi1vEAGi3iOVv/TI1NcWJEydQoUIFZGRkIDk5GVWrVpWxYqLySwjxyn0KhUL77+vXryM0NBTfffcd4uPji72e5OQ0aDSvrut1SqtZSUx8VuhlLS2rFHn50lCSY8h/TkkrTk269loUpR4DA0WpfeEssMHmhE2kPwqzRaxChQo4efIkPvvsM1hZWaFLly5FXo+ubRErC1thgPK5RawsjKG4rK2tERMTo72tUqlgZWWlvR0SEoLExEQMHjwYarUaKpUK7u7u2L17d6nUR0TFV2CDXRoTtq5N1kD5nSTKwjjKwhiKq6AtYvm6deuG6OhorFu3Dr6+vli7dm2R1qNrW8TKwlYYgFv1CvucklbYmt51a1jnzp2xadMmpKSkoFKlSggNDcWyZcu0j3t7e8Pb2xsA8ODBA3h6erK5JtITBTbYpTFhF2WyBnTrAxbQ/0kiX3mcsHVtDO86YRe0RezJkye4ePGi9kuwUqmEj49PsddHRMVnbW0NHx8feHp6Qq1WY8iQIXBwcMDEiRPh7e0Ne3t7uUskomIq8Cwi1tbWSEpK0t5+3YR96tQp7W2lUolr165JXCYRFUbnzp0RGRmJlJQUZGZmIjQ0FF27dtU+LoTA7Nmz8fDhQwDAkSNH0KZNG7nKJSr3lEolDh06hKNHj2LixIkAAH9//1ea67p16+LEiRNylEhExVBgg80Jm0h/vLhFzM3NDa6urtotYhcuXICFhQWWLVuGyZMno3///rhz5w5mz54td9lERERlSoG7iBTmJ6z8CVuhUKBp06ZYsmRJadRORK+hVCqhVCpfus/f31/77169eqFXr16lXRYREVG5UajzYHPCJiIiIiIqHF7JkYiIiIhIQmywiYiIiIgkxAabiIiIiEhCbLCJiIiIiCTEBpuIiIiISEJssImIiIiIJMQGm4iIiIhIQmywiYiIiIgkxAabiIiIiEhCbLCJiIiIiCTEBpuIiIiISEJssImIiIiIJMQGm4iIiIhIQmywiYiIiIgkxAabiIiIiEhCbLCJiIiIiCTEBpuIiIiISEJssImIiIiIJMQGm4iIiIhIQmywiYiIiIgkxAabiIiIiEhCbLCJiIiIiCTEBpuIiIiISEJssImIiGQSHBwMFxcX9O7dGwEBAa88fuzYMSiVSvTr1w9z585FTk6ODFUSUVGxwSYiIpJBQkIC/Pz8sHv3bgQFBSEwMBBxcXHaxzMyMrB06VJ8++23+PXXX5GdnY39+/fLWDERFRYbbCIiIhlERETA0dER5ubmMDU1hbOzM0JCQrSPm5qa4sSJE6hZsyYyMjKQnJyMqlWrylgxERWWkdwFEBERlUcqlQqWlpba21ZWVoiNjX1pmQoVKuDkyZP47LPPYGVlhS5duhR5PTVqmL1zrVKztKxSosuXhvI4huI+pyTpWj352GATERHJQAjxyn0KheKV+7p164bo6GisW7cOvr6+WLt2bZHWk5ycBo3m1XW9Tmk1K4mJzwq9rKVllSIvXxpKcgz5zylpxalJ116LotRjYKAotS+c3EWEiIhIBtbW1khKStLeVqlUsLKy0t5+8uQJTp06pb2tVCpx7dq1Uq2RiIqHDTYREZEMOnfujMjISKSkpCAzMxOhoaHo2rWr9nEhBGbPno2HDx8CAI4cOYI2bdrIVS4RFQF3ESEiIpKBtbU1fHx84OnpCbVajSFDhsDBwQETJ06Et7c37O3tsWzZMkyePBkKhQJNmzbFkiVL5C6biAqBDTYREZFMlEollErlS/f5+/tr/92rVy/06tWrtMsionfEXUSIiIiIiCTEBpuIiIiISEJssImIiIiIJMQGm4iIiIhIQoVqsIODg+Hi4oLevXsjICDglcfDwsIwYMAA9O/fH15eXkhNTZW8UCIqHOaViIhIXgU22AkJCfDz88Pu3bsRFBSEwMBAxMXFaR9PS0uDr68vduzYgYMHD8LGxgabNm0q0aKJ6PWYVyIiIvkV2GBHRETA0dER5ubmMDU1hbOzM0JCQrSPq9Vq+Pr6wtraGgBgY2ODR48elVzFRPRGzCsREZH8CjwPtkqlgqWlpfa2lZUVYmNjtbctLCy05+jMysrCjh07MHr06CIVUVrXhS8KS8sqJbp8aShOTWVhHGVhDMVVGnkFdC+zfK/rjvI4BiKifyqwwRZCvHKfQqF45b5nz57By8sLtra2GDhwYJGKSE5Og0bz6nrepDQ+/BITnxV6WUvLKkVaPv85Ja04NRV13KWhJF8LXRuDgYHinZrX0sgrULTM8r1eeOXpvQ7o/2fnu+aViMquAncRsba2RlJSkva2SqWClZXVS8uoVCq4u7vD1tYWK1askL5KIioU5pWIiEh+BTbYnTt3RmRkJFJSUpCZmYnQ0FB07dpV+3heXh6mTJmCvn37YsGCBa/dWkZEpYN5JSIikl+Bu4hYW1vDx8cHnp6eUKvVGDJkCBwcHDBx4kR4e3sjPj4ely9fRl5eHo4ePQoAaNGiBbeMEcmAeSUiIpJfgQ02ACiVSiiVypfu8/f3BwDY29vj6tWr0ldGRMXCvBIREcmLV3IkIiIiIpIQG2wiIiIiIgmxwSYiIiIikhAbbCIiIiIiCbHBJiIiIiKSEBtsIiIiIiIJscEmIiIiIpIQG2wiIiIiIgmxwSYiIiIikhAbbCIiIiIiCbHBJiIiIiKSEBtsIiIiIiIJscEmIiKSSXBwMFxcXNC7d28EBAS88nhYWBgGDBiA/v37w8vLC6mpqTJUSURFxQabiIhIBgkJCfDz88Pu3bsRFBSEwMBAxMXFaR9PS0uDr68vduzYgYMHD8LGxgabNm2SsWIiKiw22ERERDKIiIiAo6MjzM3NYWpqCmdnZ4SEhGgfV6vV8PX1hbW1NQDAxsYGjx49kqtcIioCNthEREQyUKlUsLS01N62srJCQkKC9raFhQV69eoFAMjKysKOHTu0t4lItxnJXQAREVF5JIR45T6FQvHKfc+ePYOXlxdsbW0xcODAIq+nRg2zYtVXkiwtq5To8qWhPI6huM8pSbpWTz422ERERDKwtrZGTEyM9rZKpYKVldVLy6hUKkyYMAGOjo6YP39+sdaTnJwGjebVZv51SqtZSUx8VuhlLS2rFHn50lCSY8h/TkkrTk269loUpR4DA0WpfeHkLiJEREQy6Ny5MyIjI5GSkoLMzEyEhoaia9eu2sfz8vIwZcoU9O3bFwsWLHjt1m0i0k3cgk1ERCQDa2tr+Pj4wNPTE2q1GkOGDIGDgwMmTpwIb29vxMfH4/Lly8jLy8PRo0cBAC1atMCKFStkrpyICsIGm4iISCZKpRJKpfKl+/z9/QEA9vb2uHr1qhxlEdE74i4iREREREQSYoNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhNthERERERBJig01UxgQHB8PFxQW9e/dGQEDAG5ebM2cO9u3bV4qVERERlQ+FarA5YRPph4SEBPj5+WH37t0ICgpCYGAg4uLiXllmypQpCAkJkalKIiKisq3ABpsTNpH+iIiIgKOjI8zNzWFqagpnZ+dXchkcHIyePXuib9++MlVJRERUthkVtMCLEzYA7YQ9bdo07TL5E3b+MkQkD5VKBUtLS+1tKysrxMbGvrTMv//9bwDAH3/8Uez11KhhVuznlgRLyyql8pySVtSaOIaSoYs1EZF+KbDBLo0JW9cma6D8ThJlYRxlYQzFJYR45T6FQiH5epKT06DRvLqu1ymNsScmPivS8paWVYr0nNJ6/YpaU3kbQ/5zSlphazIwUOjk/EVE8iuwwS6NCbsokzWgWx+wgP5PEvnK44Sta2N41wnb2toaMTEx2tsqlQpWVlbF/ntERERUdAXug21tbY2kpCTtbU7YRLqrc+fOiIyMREpKCjIzMxEaGoquXbvKXRYREVG5UmCDzQmbSH9YW1vDx8cHnp6ecHNzg6urKxwcHDBx4kRcuHBB7vKIiIjKhQJ3EXlxwlar1RgyZIh2wvb29oa9vX1p1ElEhaRUKqFUKl+6z9/f/5XlVq5cWVolERERlSsFNtgAJ2wiIiIiosLilRyJiIiIiCTEBpuIiEgmvFIyUdnEBpuIiEgGvFIyUdnFBpuIiEgGL14p2dTUVHul5BflXym5b9++MlVJRMVRqIMciYiISFqlcaVkgFdLLinlcQzFfU5J0rV68rHBJiIikkFpXCkZKNrVknXt6raA/l+hF+AVn0tSUep516slFwV3ESEiIpIBr5RMVHaxwSYiIpIBr5RMVHaxwSYiIpLBi1dKdnNzg6urq/ZKyRcuXJC7PCJ6B9wHm4iISCa8UjJR2cQt2EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhNthERERERBJig01EREREJCE22EREREREEmKDTUREREQkoUI12MHBwXBxcUHv3r0REBDwyuNXrlzB4MGD4ezsjAULFiA3N1fyQomocJhXIv3BvBKVTQU22AkJCfDz88Pu3bsRFBSEwMBAxMXFvbTM7NmzsWjRIhw9ehRCCOzdu7fECiaiN2NeifQH80pUdhkVtEBERAQcHR1hbm4OAHB2dkZISAimTZsGAPj777+RlZWFVq1aAQAGDRqEjRs3wt3dvdBFGBgoily4lUWlIj+nKIpaU1kYQ3GeU9JjAEr+tdClMRTnNXtRaeS1OHUaVbMs0vJFVRrv9ZIeA1Dy7/WyMAZAd95PZTWvlqbVi7R8cZT0+6QsjAEo+XGUxmenLo3hXTNbFAU22CqVCpaW//sws7KyQmxs7Bsft7S0REJCQpGKsLCoXKTlAeDrhU5Ffk5R1KhhVqLLA7o3huI8p6THAJT8a6GLYyiu0sgrUPTM1p+2rcjrKIrSeK+X9BiAkn+vl4UxALr5fioOXc3rZuWKIq+jqEr6fVIWxgCU/DhK47NTF8dQGgrcRUQI8cp9CoWi0I8TUelhXon0B/NKVHYV2GBbW1sjKSlJe1ulUsHKyuqNjycmJr70OBGVHuaVSH8wr0RlV4ENdufOnREZGYmUlBRkZmYiNDQUXbt21T5ep04dmJiY4I8//gAAHDhw4KXHiaj0MK9E+oN5JSq7FOJ1v0H9Q3BwMLZv3w61Wo0hQ4Zg4sSJmDhxIry9vWFvb4+rV69i4cKFSE9Ph52dHb788ksYGxuXRv1E9A/MK5H+YF6JyqZCNdhERERERFQ4vJIjEREREZGE2GATEREREUmIDTYRERERkYTYYBMRERERSYgNdhlUXo5bLS/jpLKtvLyPy8s4qWwrL+/j8jLOksQGuwwqL1f60qdxHj9+HGfPnpW7DNJB+vQ+fhf6NE7mld5En97H70KfxqmreTWSuwCSTmZmJv7880+cOnUKTZo0QdOmTdGyZUu5y5JcVlYW/vrrL0RGRqJhw4Zo1KgRWrVqJXdZbxQeHo6FCxdizpw5aN++vdzlkI5gXnUT80qvw7zqJl3Oa5ndgv3PnzdUKhUuXLiAqKgomSoqWQkJCdi4cSOWLFmCtLQ0PHr0CAsXLsTJkyflLk1S8fHx2LBhAxYvXozExETExcXhs88+09lxnjhxAsuWLcP8+fPh5uYG4Pl7kz+/vYx5ZV51AfNaOMwr86oLdD2vZXILthBC+/NGSkoKfvvtNwQFBSEzMxM5OTnYu3cvTExMZK5SOjk5OQgICEBiYiJWrFiBdu3aAQC6du2KhQsXonr16rC3t5e5ynenVquxa9cuqFSql8bZu3dvfP755zA3N9epLQonT57EypUrMX36dCiVSgDPXytjY2NkZmaiUqVKMleoG5hX5lUXMK+Fw7wyr7pAH/JaJrdg5+XlIScnB2vXrsXatWvx3XffwdvbGw0bNkS3bt1gZFS2vldcvHgRISEhGD58ONq1awchBHJycuDg4IDu3bvj6dOncpcoiStXruDXX3/FsGHDXhpnq1at0K9fP/z9999yl6h1+fJlzJkzB1OnTtV+s84Pf1xcHDp27Ih79+7pzDdtOTGvzKvcmNfCY16ZV7npS17LXIN969YtzJ07Fx9//DH8/f1RrVo1BAcHw9DQEDExMRgxYgQMDQ2RlZWFtLQ0ucuVRExMDNq0aYP27dtDo9FAoVDA2NgYubm5uHjxIu7fvw9A/48Kvnz5Muzt7dGxY0doNBoAgLGxMTQaDc6ePYusrCykpKQgMTFR5kqB0NBQDB8+HAMGDIBGo9GG/+7du/Dw8MC///1v1K9fX+4yZce8Mq/Mq/5gXplX5rXwytZXTQCGhoaoVasWnJycMGbMGCxbtgw9e/bEpUuX0KdPH2RlZeGnn37Cw4cPERMTg3HjxqFHjx5yl/1OLC0toVKpAAAajQYGBs+/NxkZGWHFihWoUaMGgOdHBb/4856+qVWrFqpVqwbg+Tjzt5QkJiYiNTUV586dw40bN2BtbY1hw4bB1NRUtlorVKig/cDVaDQwNjbGnTt3MHToULRr1w6enp7aZfX5NXlXzCvzyrzqD+aVeWVei0CUQRqNRvvv8PBw4ejoKGxtbcXZs2fFlStXxAcffCDGjx8v9u/fL5YvXy4yMzNlrPbd3b17V7i6uorDhw8LIYR48uSJSEpKEkIIERgYKD755BOxf/9+7fIv/v/RJw8ePBD9+vUThw4dEkIIkZWVJe7cuSNmzpwpmjdvLjp16iRGjRolkpKSRFZWlrh9+7ZstcbFxYmBAweKkJAQkZaWJs6fPy86dOggPvnkE7F9+3Yxfvx48euvv2qX19fXRArMK/PKvOoP5pV5ZV4LRyGEnv+u8RZqtRoVKlTAp59+CgsLC4wZMwbffPMN1Go18vLyMHPmTFhbWwN4vl+ZoaGhzBUX340bN/DZZ5+hbdu2OH36ND7++GN069YNV65cQV5eHrZt24aePXti7Nixcpf6Tq5fv45Zs2bhgw8+QExMDOrXr4/Tp0/D2dkZVapUwdOnTzF27FgEBgYiMzMTPj4+sLCwkK3W+fPno3Hjxrh9+zYaNWqE9u3bY//+/VAqlQgJCUHv3r0xatQoWerTNcwr88q86g/mlXllXt+uTDfY+caPH486deogNzcX5ubm8PLyQmZmJubNm4eRI0eiV69ecpcoiYcPHyIlJQW//vorhBA4c+YMXF1d0adPH2RmZmLmzJnYvn073nvvPb3+OevBgwe4c+cOnj59isePH+PixYto2LAh3NzcYGpqiq1bt+LZs2cYMGAA2rZtK+sYk5OToVAoEBkZiadPnyI2NhYjR46Eg4MDzp49i02bNmHLli0wMTHB5cuXYWxsjGbNmslWry5gXplXuTCvRce8Mq9y0fW8lrl9sP9JpVLh0qVLqFixIuzt7dG/f38oFAqsW7cO6enp+P777/H3339jzJgxCA0NhYmJCbp16yZ32cVSu3Zt1K5dG/Hx8Thx4gRWrlyJ9PR0bNu2DRcvXkTr1q1Ru3Zt7QEB+hh+AKhbty7q1q2L3NxcbNiwAY0aNYKTkxOqVq2KHTt2ICoqClZWVrh165b2VENyqVGjBvLy8nDv3j1cuXIFkyZNQosWLZCSkoJNmzahWbNmMDMzw4ULF7Bp0yY0adKkXE/YzCvzKifmtWiYV+ZVTrqe1zJ3FpF/srKywsGDB9GhQwe4uLjA3NwcixYtgomJCebOnYulS5dqf2KwtbVFzZo15S75nTVp0gTnz59HUlISGjRoAABo1KgRBg4ciKSkJIwePRq///67zFW+O7VajZiYGFhYWKBhw4bYsmULUlNT4e7ujnnz5sHf3x+HDx+Wu0wYGhpi0KBB8PLyQosWLfDkyRMsXrwYzZo1w6hRoxAbG4utW7eiefPmmDdvHgBoj+Iub5hX5lVuzGvhMa/Mq9x0Oa/lYhcRAMjNzUVOTg5mzJiB+vXrY/To0ahTpw4qVKiA77//Hl9++SUiIiJQvXp17al49PUbKPB8n7GlS5ciISEBTZo0wfjx49GwYUMsXrwYGo0G27Ztk7tESdy/fx/Xrl3DqVOnADw/KX779u1hbGyMpUuXal9rAwMDnXg9U1NTMWPGDNja2mL48OF4+vQpvv/+e1StWhWLFy9GdnY2KlasKHeZsmNemVddeD2Z18JhXplXXXg9dS6vshxaKZP4+HixcuVKERcXpz2qdNeuXaJly5bi/PnzQgghsrOztcvr69HA+eLj48Xnn38ufvvtN5Geni4mT54spk+frn08MjJSxMTEyFihNBITE8XatWvFiRMnRG5urhBCiICAAGFnZydu3bolhBBCrVZrl5fzdX306JFYu3atuH79ujh//rzw9PQUNjY2YuDAgWLy5Mli1KhRIiQkRCdqlRvzyrwKwbzqC+aVeRWCeX1RudmCnU+tVsPIyAgKhQJ79uzB8uXLcfDgQcTHx+Po0aN4/PgxunTpgmHDhsldqiRSU1NRrVo1LF++HNevX8fOnTsBPD95/pEjRwAA3t7e2vNfCj09MCMjIwMmJiYwNDTEjz/+iBUrVuDAgQPIzMzE8ePHkZqaCjs7OwwdOlTuUqFWq3Ht2jX4+fnB3t4etra22vPINm7cGBqNBg8ePICNjQ0A/X1NpMC8Mq9yY14Lj3llXuWmS3kt8/tg/1OFChWgUCiwb98++Pr6Yv/+/bh16xa+/PJL2NnZwc3NDT/88AN++uknuUuVRLVq1ZCbm4u0tDQ4OzsDAKKjo3Hy5EmYmZlhzJgxMDY2RlJSEoD/nSxf35iamsLQ0BD79u3DkiVLsH//fty5cwfTp0/HX3/9haFDhyIoKAgBAQFyl4oKFSrA0tISdnZ2mDlzJsLDw5Gbm4vmzZujUqVK2LVrF2bNmoXz588D0N/XRArMK/MqN+a18JhX5lVuupTXcrcFO9/Nmzfx+PFjtGvXDuPHj8ekSZPg6OgIADh58iTOnz8Pb29v7fL6+s0z37Vr1zBv3jy0bt0aly5dQseOHdGkSRM8ffoU3377LWrXro1u3brh3//+t9ylvpMXX9cJEyZgzJgx+PHHH9GjRw907doV3333HWbPnq29GpcuvK4TJkzAjBkz4ODggA0bNiAmJgZNmjRBamoqHBwcMG7cOFnr0wXMK/MK6MbryrwWjHllXgHdeF3lzGu524Kdr0mTJmjXrh3u37+PKlWqoEWLFtrHMjIyULlyZeTk5CAxMRGA/n7zzGdjY4PNmzfD1NQU77//Pvr3748nT57g0qVLWL16NXbs2IEjR47g5MmTcpf6TvJf14cPHwIAHB0dsXnzZoSGhsLV1VW73N9//w1A3tdVo9EgJSUFjx8/hrW1Nb7++mtcvnwZM2bMgK+vL2bOnIl9+/Zpx1KeMa/MK8C86gvmlXkFmNcyfx7sgpiYmCAhIQGnT5/GRx99hJSUFPTt2xdqtRre3t4wMDCAo6MjRo8erdcnjweA9957D+PGjcOTJ0+Qnp6Ob7/9Flu2bNGeF7J+/frIysp66Tn6Ol4TExNkZWXht99+g7OzM1atWoX9+/ejffv2iIyMxN69e+Hs7AwXFxfZxmdgYIDq1atj165d+OGHHxAeHo5PP/0ULVu2BAA8evQIlStXfuUKaPr6mkiBeWVemVf9wbwyr+U5r+V2F5EXXbt2Db6+vjAxMUHTpk0xePBgGBsb4/Hjx2jdujU8PDzg6upapi6R+/vvv+O///0vFixYgNzcXBgZGSEmJgaZmZmoX78+kpOT0aZNGwDPvwnm/+SjT65evYq5c+ciJycHSqUSY8eOxenTp3HgwAE8efIEmZmZGD9+PPr16yd3qXj06BHi4+PRvHlzGBsbIyYmBgsWLMD06dPRo0cPpKSk4MmTJ9otQfr6wSwF5pV5lRvzWnjMK/MqN7nyygb7/8vIyICpqSlOnz6NjRs3okqVKmjVqhW6du2Kq1evIjo6GmvWrAHw/GcPfQ1Fvvv372P8+PEYO3YshBAYNmwYjI2NcePGDYSEhODUqVMYMGAA3N3d5S71naSmpmqvrBUbG4ujR4+iZs2amDlzJjIzM3HmzBmdu7JYdHQ0fH19MX36dNjY2MDb2xvNmzdHXFwcBg8eDA8PD7lLlB3zyrzqCua1YMwr86orSjOvbLD/v/xvLEuWLIGhoSEWLlyI6Oho/PDDD0hOTkb37t0xadIkXLx4scxslYiLi4O/vz+aNGkCIyMjPHv2DO3atYOFhQXq168PDw8PTJ48GS4uLnKX+s4iIiKwd+9evP/++5gyZcorPwvpiqysLIwaNQr9+vVDv379MGDAAMyZMwdKpRLJycmYOHEiPv/8c7Rt2xaA/r8Hi4t5ZV51AfNaOMwr86oLSjuv+vsVUWL5/xNHjRqFqKgoHD58GHfu3EHNmjXh4OCA8ePHY9euXRg7diz++usv7XP0+ftJ06ZNsWLFCkyaNAnnzp1DTk4Ojh49irCwMFy7dg12dna4d+/eS8/R1/E2bdoUdnZ22vDr6qWNK1asCH9/f4wbNw7Hjh1D//794ebmBiEErKys0LRpUwghtPXr8wT0LphX5lUXMK+Fw7wyr7qgtPPKBvsfmjRpAj8/P8TExMDf3x+mpqaYNWsWdu3ahZ07d2Lo0KH4+eef8d133wHQ/w9MQ0ND5OTkQK1Wo0mTJli6dCnc3d2xcOFCREVFYeDAgUhLS8P9+/cB6O+HnpWVFSZNmqQNvy7//Ghubg4AePr0qbZOIyMjHD58GH///TcaNWqESZMm4fjx4zJWqRuYV+ZVbsxr4TGvzKvcSjOv5f4sIq/z/vvvY/LkyUhKSsLMmTPx448/Yu/evVi/fj3s7Oxw9epVfPrpp3ByckLt2rXlLvedKBQKGBsbw8fHB76+vlCr1Th79iwMDQ2xdetWxMXF4ddff8Xt27fRr18/jBo1Svuhp68/4ely+IH/TSp9+vTBzJkz8dNPP+HOnTu4e/cuRowYgcOHDyM3Nxc9e/aUuVLdwLwyr3JiXouGeWVe5VSaedXt/xMysra2xoYNGxAUFITt27djzZo1sLOzQ1ZWFuLj49G4cWMYGxsD+N/POtnZ2XKW/E5sbW2xevVqHD58GDExMdiwYQNsbW2RkJCAZ8+eYfPmzThw4ADCwsK0z3n8+LGMFZd9jRo1wurVq3Hr1i08evQIY8aMgYODA/Ly8jBmzBgA0L4u5R3zyrzKjXktPOaVeZVbaeSVBzkW4N69e8jIyICtrS1ycnIQExODb775Bh06dMCkSZMAAOnp6bh+/Tq+++47eHh4oEOHDjJXXXzx8fG4evUqUlNTkZycDCcnJyxYsACzZs1CdHQ0bGxs8OGHHyIrKwvu7u7Ytm0bLC0t9fKbtr7Ij6hCoUBERAR27tyJZs2aoVq1ali1ahW8vLwwbdo0vgZgXplX+TGvhce8Mq9yK8m8cheRAtSvXx8AkJOTg8jISOzcuRPdu3fXnrNzy5YtSElJQWxsLC5evIiGDRvq9QdArVq1UKtWLRw7dgznz59H3bp10aBBA1y6dAk9evRAgwYNtN+yd+3aBVNTU5krLvvyg33z5k0sX74cycnJ6Nu3Lx4/fgw7OzuMHDkSiYmJePr0KZo2bSpztfJiXplXuTGvhce8Mq9yK8m8cheRQsrLy8PBgwfRrl07DBs2DN9//z2GDRuG06dPw97eHoMHD8aYMWPg6ekpd6mSaNq0Kc6dO4ewsDDUq1cPUVFRaNy4MSIiIjBt2jSkpaW9FH7+EFLyqlWrhkaNGsHHxwdOTk44duwY3N3dkZSUhPnz5+PQoUOIjY3VLl+eXxPmlXmVG/NaeMwr8yq3ksgrdxEpgsTERERFReGLL75A8+bNcebMGfj7+6Njx47w8/PD3bt3sXz5cpiZmcldqiSuX7+OtWvXomLFihg+fDgqVKgAT09PbNiwAR999BEuXbqE9PR0dOnSBYD+XpFKn6SlpcHMzAxHjhzBjh07MGTIEOzbtw8zZsyAmZkZjh07hj59+mgvB1ueXxPmlXmVG/NaeMwr8yo3yfMqqEgePHgggoODhRBChIaGCicnJ7F582bh5OQk/vzzTyGEENnZ2drlNRqNHGVKJn8sERERokWLFuLEiRNCCCEuXbokNm7cKEaMGCF++OEHOUssd3JycsSoUaNEt27dxL59+8T9+/fFkydPxMqVK4Wrq6v4+OOPRUBAgNxl6gTmlXmVG/NaeMwr8yo3KfPKLdjvKCoqCl5eXujSpQv8/PyQmZmJsLAwNGzYEK1atQKg3988hRDIyMhAjx498OWXX6J69eo4ePAgOnbsiAYNGuD999+Hm5sbPvnkE3Tv3l3ucsuNBw8ewNraGhUqVMDdu3exd+9eqNVqdOrUCY6Ojhg0aBAWLVqEzp07y12qTmFemVc5MK/Fw7wyr3KQKq88yLGYxP8/R2X79u1haGiI3r17w9DQEBEREfj6669RrVo1DBgwAEOHDtXb8APPDwCoXLkyjh8/DjMzM0yYMAGdO3fGX3/9hevXryM+Ph7m5uZITU196XlCT8/hqS/q1q0LALh69Sp++uknGBsbo3fv3mjfvj2A5/uTJScny1miTmFemVc5Ma9Fw7wyr3KSKq9ssIsp/4pLP//8M2xtbdG9e3ccPXoUx48fx4cffoiRI0di3LhxsLKyQrdu3QDodygqV66MjIwMpKeno1atWpgwYQIePnyIoUOHokGDBujTpw+ioqJgamoKBwcH7f8ffR2vvrC0tISZmRkcHR214f/000+h0WjQt29fmavTHcwr86oLmNfCYV6ZV13wrnnlLiLvKD4+Ht988w1sbGwQExODpk2bYsKECQCA2bNno379+ujSpQtat24NQL8/BIDn3+i++OILdO3aFaGhoTA3N8fy5cuRkJCAdevWQQiBoUOHol+/fnKXWm5kZWWhYsWKAJ6HPyUlBV999RVq1qyp9+83qTGvzKvcmNfCY16ZV7m9S17197cVHVGrVi1Mnz4dt27dQu3atbXh//LLLxEcHIyqVavC398fAQEBAKDX4QeeX5FqxYoVCA8PB/B8nPfu3cO+fftgamqKadOm4YcffsBPP/0kb6HliImJCQBg/vz5uHPnDtasWYOaNWsC0P/3m9SYV+ZVbsxr4TGvzKvc3iWv3IItkcePH8PCwgIAsGrVKvz88884dOgQrK2tERYWhq1bt+Lbb7+FmZmZXu8zli8xMRF3795FVlYWfvvtNxgZGeHjjz+Gubk5fvzxR5w5cwarVq3SXu6WSt7Dhw9RsWJFVK9eXe5SdB7zyrzKjXktPOaVeZVbcfKq/+9EHZEffj8/P3z77bcICgqCtbU1Lly4gEOHDsHZ2RlVq1Z9Kfz6/N3G0tISzZo1Q3R0NHJycuDl5QVzc3PExMRg9+7d6NSpE8NfymrXrs3JupCYV+ZVbsxr4TGvzKvcipNXbsGW2K1bt2BqaopatWrhypUr2LlzJxo1aoQ2bdrA1NQUt2/fhpmZWZk4MAMAkpKSYGxsjKpVq+LcuXNYv349evTogSFDhpSZCwJQ2cW8Mq+kP5hX5lWfcAu2xBo3boxatWrh2rVr2Lp1K+rVq4e6desiKioKM2bMQFJSErZt24adO3cC0P99xmrWrImqVavi/PnzWLZsGXr06IH+/fsz/KQXmFfmlfQH88q86hM22CXEwsICtra2cHJyAgCoVCqYm5ujUqVK+P7773Hy5EncunVLr3/GelHNmjUxZMgQ9O/fnz97kt5hXon0B/NK+oC7iJQgtVqNChUqYMqUKejVqxeGDBmCCRMmoHLlyqhZsybmzJmjPUIV0P+fs/T5ilpEzCuR/mBeSdfx1SpBFSpUgFqthoWFBYyMnl/TZ9OmTTAxMUFmZiZMTEywadMm7N27F8Dzn7Py8vLkLPmdMPykz5hXIv3BvJKu4xbsUnD16lXMnj0bDRs2RLNmzTBu3DhUqlQJmzdvxqZNm9C+fXv07dsX7u7uiI2NBQA4ODjIXDVR+cS8EukP5pV0FS+VXgpsbW2xZcsWnDlzBpUrV0alSpWwfPlyBAcHIzg4GKampti+fTtSUlJgbW2NU6dO8QOASCbMK5H+YF5JV3ELtgwWL16M48ePIygoCDVq1AAAnD59Gv7+/li0aBGaNGkCQP/3GSMqC5hXIv3BvJKu4BbsUiSEQGJiIiIjI7Fv3z7UqFEDGRkZOH/+PL755hv07t1bG37g+T5j/BAgkgfzSqQ/mFfSNdyCLYP8o4HT09Nx4cIF7NixAz179kTPnj2xevVqVK5cGc2aNYO7u7vcpRKVe8wrkf5gXklX8LBUGeR/c758+TLWrl0LpVKJbt26wcPDAzVr1oSLiwv279+vPfqZiOTDvBLpD+aVdAV3EZFB/k9SdevWRb9+/TBw4EC4urpi8ODB8PLyAgB4eHjgwoULAP63rxh/ziIqfcwrkf5gXklXcAu2jN577z2MHTsW6enpsLGx0YYfAOrXrw8LCwsAz69SBfzvmzkRlT7mlUh/MK8kN27B1gHZ2dm4ffs2goOD0bFjR1SuXBlt2rRB7dq1sWPHDkRHR6NPnz4YOnQov2kTyYx5JdIfzCvJhQc56oi4uDjMmzcPNjY2qFixIvr3748qVapApVKhTp06mD17NlxcXDB69Gi5SyUq95hXIv3BvJIc2GDrkCdPnqBy5coICAhAXFwcbty4gY8++gj9+vXDpUuXEBgYiC1btqBSpUr8pk0kM+aVSH8wr1TauIuIDjE3NwcAZGRkwMTEBN9//z3u3r2LVatWIS4uDsOHD4epqSkSExNhaWnJDwEiGTGvRPqDeaXSxoMcdZCTkxP+/PNPREdH49GjRzAxMUGfPn3Qr18/REVFYdy4cQgLCwPAAzOI5Ma8EukP5pVKC3cR0VFxcXFYuXIlrl69Cjc3NwwfPhwJCQmYPn062rVrBzMzM7Ro0QIeHh5yl0pU7jGvRPqDeaXSwAZbhyUmJuLrr7/GoEGD8OzZMyxYsADu7u7w9PTEo0ePMHr0aGzZsgX/+te/5C6VqNxjXon0B/NKJY27iOgwS0tLzJgxA9nZ2Zg2bRo8PDzg6ekJjUaDiIgI2Nra4r333pO7TCIC80qkT5hXKmk8yFHHVapUCdWqVYO3tzdGjhwJAAgODsalS5fQrl07GBsby1whEeVjXon0B/NKJYm7iOiZn376CVeuXEG9evUwePBgVK1aVe6SiOgNmFci/cG8kpS4i4geSUtLw82bN2FpacnwE+k45pVIfzCvJDVuwdYzT548gaGhIapUqSJ3KURUAOaVSH8wryQlNthERERERBLiLiJERERERBJig01EREREJCE22EREREREEmKDTUREREQkITbYREREREQSYoNNRERERCQhXiq9jIuOjsaKFStgamqK9PR0tGnTBpcvX0Z6ejqEEFi+fDnatm2L9PR0LF++HOfOnYOhoSF69eoFHx8fqNVqrFmzBmfPnkVeXh7s7OywcOFCmJmZyT00ojKHeSXSH8wrvQ0b7HLgxo0bCAsLg0qlwrfffovAwEAYGBhgx44d8Pf3R9u2bbFx40ZkZ2fj8OHDyMvLw/jx43HmzBmcPXsWhoaG2LdvHxQKBdatW4c1a9bA19dX7mERlUnMK5H+YF7pTdhglwPvvfce6tSpgzp16qBatWrYs2cP7t+/j+joaFSuXBkAEBERgXnz5sHQ0BCGhobYtWsXAOCrr77Cs2fPEBERAQBQq9WoUaOGbGMhKuuYVyL9wbzSm7DBLgdMTU0BAOHh4VixYgXGjRuHnj17onHjxjh48CAAwMjICAqFQvucR48eoWLFitBoNJg/fz66desGAEhPT0d2dnbpD4KonGBeifQH80pvwoMcy5HTp0+je/fucHd3h729PcLCwpCXlwcA6NSpE/bv3w+NRoOcnBx4e3vj7Nmz6NKlCwICApCTkwONRoNFixZh3bp1Mo+EqOxjXon0B/NK/8QGuxwZMWIEzp49C6VSieHDh6NevXp48OABNBoNpk2bhgoVKmDAgAFwc3NDt27d4OTkBC8vL9SpUwcDBw6Ei4sLhBCYO3eu3EMhKvOYVyL9wbzSPymEEELuIoiIiIiIygpuwSYiIiIikhAbbCIiIiIiCbHBJiIiIiKSEBtsIiIiIiIJscEmIiIiIpIQG2wiIiIiIgmxwSYiIiIiktD/Awadl22cjXJ2AAAAAElFTkSuQmCC'" + ] + }, + "metadata": {}, + "execution_count": 108 + } + ], + "metadata": {} + }, + { + "cell_type": "markdown", + "source": [ + "Finally, we pass our filled-out `model_card` to the `mct` object to generate an HTML version of the `model_card` that can be rendered within a Jupyter notebook." + ], + "metadata": { + "id": "WvL2mHlwp-l0" + } + }, + { + "cell_type": "code", + "execution_count": 106, + "source": [ + "mct.update_model_card_json(model_card)\n", + "html_modelcard = mct.export_format()" + ], + "outputs": [], + "metadata": { + "id": "A-j_4mBaPe5z" + } + }, + { + "cell_type": "markdown", + "source": [ + "### Display the model card" + ], + "metadata": { + "id": "CkuNE_MvMy7P" + } + }, + { + "cell_type": "code", + "execution_count": 107, + "source": [ + "display.HTML(html_modelcard)" + ], + "outputs": [ + { + "output_type": "execute_result", + "data": { + "text/html": [ + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + " Model Card for Diabetes Re-Admission Risk model\n", + "\n", + "\n", + "\n", + "

\n", + " Model Card for Diabetes Re-Admission Risk model\n", + "

\n", + "
\n", + "
\n", + "

Model Details

\n", + "

Overview

\n", + " This model predicts whether a patient will be re-admitted into a hospital within 30 days.\n", + "

Version

\n", + " \n", + " \n", + "
name: v1.0
\n", + "\n", + " \n", + " \n", + "
date: 2021-07-13
\n", + "\n", + " \n", + " \n", + "\n", + " \n", + "

Owners

\n", + "
    \n", + "
  • Fairlearn Team, https://fairlearn.org/
    • \n", + "
\n", + " \n", + "

License

\n", + " MIT License\n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
\n", + " \n", + "
\n", + "

Considerations

\n", + " \n", + "

Intended Users

\n", + " \n", + " \n", + "
    \n", + " \n", + "
  • Medical Professionals
    • \n", + " \n", + "
  • ML Researchers
    • \n", + " \n", + "
\n", + "\n", + " \n", + " \n", + "

Use Cases

\n", + " \n", + " \n", + "
    \n", + " \n", + "
  • High-Risk Patient Care Management
    • \n", + " \n", + "
\n", + "\n", + " \n", + " \n", + "

Limitations

\n", + " \n", + " \n", + "
    \n", + " \n", + "
  • \n", + " This model will not generalize to hospitals outside of the United States. Features, such as those encoding insurance\n", + " information, are inherently tied to the U.S healthcare system.\n", + " In addition, this model is intended for patients who are admitted into U.S. hospitals for diabetes-related illnessess.\n", + "
    • \n", + " \n", + "
\n", + "\n", + " \n", + " \n", + " \n", + "

Ethical Considerations

\n", + "
    \n", + "
  • \n", + "
    Risk: Low sample sizes of certain racial groups could lead to poorer performance on these groups
    \n", + "
    Mitigation Strategy: Collect additional data points from more hospitals.
    \n", + "
\n", + "
\n", + " \n", + "
\n", + " \n", + "
\n", + "
\n", + "

Train Set

\n", + " 11356 rows with 46 features. The original training data set was undersampled to allow for an equal number of positive and negative labeled instances.\n", + " \n", + "
\n", + " \n", + "
\n", + " Sensitive Features\n", + "
\n", + " \n", + "
\n", + " Target Label\n", + "
\n", + " \n", + "
\n", + "\n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + "
\n", + "

Eval Set

\n", + " 50882 rows with 46 columns\n", + " \n", + "
\n", + " \n", + "
\n", + " Sensitive Features\n", + "
\n", + " \n", + "
\n", + " Target Label\n", + "
\n", + " \n", + "
\n", + "\n", + "
\n", + "
\n", + " \n", + " \n", + "
\n", + "
\n", + "

Quantitative Analysis

\n", + " \n", + " These graphs show the models performance on the test dataset for disagregated racial categories.\n", + " \n", + "
\n", + " \n", + "
\n", + " ThresholdOptimizer\n", + "
\n", + " \n", + "
\n", + "\n", + " \n", + " \n", + "\n", + " \n", + " \n", + "\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "\n", + "
Performance Metrics
NameValue
\n", + "false_negative_rate, AfricanAmerican\n", + "\n", + "0.4001736111111111\n", + "
\n", + "false_negative_rate, Caucasian\n", + "\n", + "0.3889812396105438\n", + "
\n", + "false_negative_rate, Other\n", + "\n", + "0.45495495495495497\n", + "
\n", + "false_negative_rate, Unknown\n", + "\n", + "0.43617021276595747\n", + "
\n", + "\n", + " \n", + "
\n", + "
\n", + " \n", + "\n", + "" + ], + "text/plain": [ + "" ] + }, + "metadata": {}, + "execution_count": 107 } - ], - "metadata": { + ], + "metadata": { "colab": { - "collapsed_sections": [ - "4MMbl3u7-J-a", - "ZVSerRqDG3we" - ], - "name": "SciPy 2021 Tutorial.ipynb", - "provenance": [], - "toc_visible": true - }, - "interpreter": { - "hash": "a025db62d48a12d86b0e6b0cb53f59776c5e11a448915a1ba45134646da53519" + "base_uri": "https://localhost:8080/", + "height": 1000 }, - "kernelspec": { - "display_name": "Python 3.8.5 64-bit ('base': conda)", - "name": "python3" - }, - "language_info": { - "name": "python", - "version": "" - } + "id": "89kqC0Jj9D6O", + "outputId": "a64294f5-205a-4dc9-8e8a-fcd245a7182f" + } + }, + { + "cell_type": "markdown", + "source": [ + "# Discussion and conclusion" + ], + "metadata": { + "id": "1j9WzWcCVbZV" + } + }, + { + "cell_type": "markdown", + "source": [ + "In this tutorial we have explored in depth a health care scenario through all stages of the AI lifecycle except the model deployment stage. We have seen how fairness-related harms can arise at the stage of task definition, data collection, model training, and model evaluation. We have also seen how to use a variety of tools and practices, such as datasheets for datasets, Fairlearn, and model cards.\n", + "\n", + "Once the model is deployed, it is important to continue monitoring the key metrics to assess any performance difference as well as the potential for fairness related harms. As you learn more about how the model is used, you may need to revise the fairness metrics, update the model, consider additional sensitive features, update the task definition, or collect new data.\n", + "\n", + "Although we used a variety of software tools, fairness is a sociotechnical challenge, so mitigations cannot be purely technical, and need to be supported by processes and practices, including government regulation and organizational incentives.\n", + "\n", + "If you would like to learn more about fairness of AI systems, or to contribute to Fairlearn, we welcome you to join our community. Fairlearn is built and maintained by contributors with a variety of backgrounds and expertise.\n", + "\n", + "Further resources can also be found [on our website](https://fairlearn.org/main/user_guide/further_resources.html)." + ], + "metadata": { + "id": "xtvW54LtB8hp" + } + } + ], + "metadata": { + "colab": { + "collapsed_sections": [ + "4MMbl3u7-J-a", + "ZVSerRqDG3we" + ], + "name": "SciPy 2021 Tutorial.ipynb", + "provenance": [], + "toc_visible": true + }, + "interpreter": { + "hash": "a025db62d48a12d86b0e6b0cb53f59776c5e11a448915a1ba45134646da53519" + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" }, - "nbformat": 4, - "nbformat_minor": 0 -} + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.7.10" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} \ No newline at end of file