SKL-Advanced-PimaDiabetes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using Scikit-Learn\n",
    "\n",
    "\n",
    "*  Scikit-Learn has implemented all the basic algorithms of machine learning. \n",
    "*  Let’s take a look at some of them.\n",
    "*  First of all, the data should be loaded into memory, so that we could work with it. \n",
    "*  The Scikit-Learn library uses NumPy arrays in its implementation, so we will use NumPy to load `*.csv` files.\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://raw.githubusercontent.com/RWorkshop/workshopdatasets/master/pimadiabetes.csv\"\n",
    "\n",
    "myData = pd.read_csv(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### About the PIMA data set\n",
    "The Pima diabetes database, donated by Vincent Sigillito (John Hopkins University), is a collection of medical diagnostic reports of 768 examples from a population living near Phoenix, Arizona, USA. \n",
    "\n",
    "The samples consist of examples with 8 attribute values and one of the two possible outcomes, namely whether the patient is tested positive for diabetes (indicated by output one) or not (indicated by two). \n",
    "\n",
    "The database now available in the repository has 512 examples in the training set and 256 examples in the test set.\n",
    "\n",
    "#### Attribute Information\n",
    "There are nine variables in this data set\n",
    "\n",
    "\n",
    "1. Number of times pregnant \n",
    "2. Plasma glucose concentration a 2 hours in an oral glucose tolerance test \n",
    "3. Diastolic blood pressure (mm Hg) \n",
    "4. Triceps skin fold thickness (mm) \n",
    "5. 2-Hour serum insulin (mu U/ml) \n",
    "6. Body mass index (weight in kg/(height in m)$^2$) \n",
    "7. Diabetes pedigree function \n",
    "8. Age (years) \n",
    "9. Class variable (0 or 1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>npreg</th>\n",
       "      <th>glu</th>\n",
       "      <th>bp</th>\n",
       "      <th>skin</th>\n",
       "      <th>serum</th>\n",
       "      <th>bmi</th>\n",
       "      <th>ped</th>\n",
       "      <th>age</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   npreg  glu  bp  skin  serum   bmi    ped  age  type\n",
       "0      6  148  72    35      0  33.6  0.627   50     1\n",
       "1      1   85  66    29      0  26.6  0.351   31     0\n",
       "2      8  183  64     0      0  23.3  0.672   32     1\n",
       "3      1   89  66    23     94  28.1  0.167   21     0\n",
       "4      0  137  40    35    168  43.1  2.288   33     1"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "myData.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = myData.drop(\"type\",1)\n",
    "y = myData[\"type\"]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will work with this dataset in all examples, namely, with the X feature-object matrix and values of the y target variable.\n",
    "\n",
    "#### Data Normalization\n",
    "\n",
    "*  The majority of gradient methods (on which almost all machine learning algorithms are based) are highly sensitive to \\textit{data scaling}.\n",
    "*  Therefore, before running an algorithm, we should perform either normalization, or the so-called standardization. \n",
    "*  Normalization involves replacing nominal features, so that each of them would be in the range from 0 to 1.\n",
    "*   As for standardization, it involves data pre-processing, after which each feature has an average 0 and 1 dispersion. \n",
    "*  The Scikit-Learn library provides ready-made functions for this:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\n",
    "from sklearn import preprocessing\n",
    "\n",
    "# normalize the data attributes\n",
    "normalized_X = preprocessing.normalize(X)\n",
    "\n",
    "# standardize the data attributes\n",
    "standardized_X = preprocessing.scale(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### More on Data Normalization and Data Scaling\n",
    "Data scaling is a method used to standardize the range of independent variables or features of data. In data processing, it is also known as data normalization and is generally performed during the data preprocessing step.\n",
    "\n",
    "Since the range of values of raw data varies widely, in some machine learning algorithms, objective functions will not work properly without normalization. For example, the majority of classifiers calculate the distance between two points by the distance. \n",
    "\n",
    "If one of the features has a broad range of values, the distance will be governed by this particular feature[citation needed]. Therefore, the range of all features should be normalized so that each feature contributes approximately proportionately to the final distance.\n",
    "\n",
    "Another reason why feature scaling is applied is that gradient descent converges much faster with feature scaling than without it.\n",
    "\n",
    "#### Rescaling\n",
    "The simplest method is rescaling the range of features to scale the range in [0, 1] or [−1, 1]. Selecting the target range depends on the nature of the data. The general formula is given as:\n",
    "\n",
    "$$ x' = \\frac{x - \\text{min}(x)}{\\text{max}(x)-\\text{min}(x)}$$\n",
    "where x is an original value, x' is the normalized value. For example, suppose that we have the students' weight data, and the students' weights span [160 pounds, 200 pounds]. To rescale this data, we first subtract 160 from each student's weight and divide the result by 40 (the difference between the maximum and minimum weights).\n",
    "\n",
    "#### Standardization\n",
    "In machine learning, we can handle various types of data, e.g. audio signals and pixel values for image data, and this data can include multiple dimensions. Feature standardization makes the values of each feature in the data have zero-mean (when subtracting the mean in the enumerator) and unit-variance. This method is widely used for normalization in many machine learning algorithms (e.g., support vector machines, logistic regression, and neural networks). \n",
    "\n",
    "This is typically done by calculating standard scores. The general method of calculation is to determine the distribution mean and standard deviation for each feature. Next we subtract the mean from each feature. Then we divide the values (mean is already subtracted) of each feature by its standard deviation.\n",
    "\n",
    "The standard score of a raw score x is\n",
    "\n",
    "$$z = {x- \\mu \\over \\sigma}$$\n",
    "where:\n",
    "\n",
    "\n",
    "- [$\\mu$] is the mean of the population;\n",
    "- [$\\sigma$] is the standard deviation of the population.\n",
    "\n",
    "The absolute value of z represents the distance between the raw score and the population mean in units of the standard deviation. z is negative when the raw score is below the mean, positive when above.\n",
    "\n",
    "#### {Scaling to unit length}\n",
    "Another option that is widely used in machine-learning is to scale the components of a feature vector such that the complete vector has length one. This usually means dividing each component by the Euclidean length of the vector. In some applications (e.g. Histogram features) it can be more practical to use the L1 norm (i.e. Manhattan Distance, City-Block Length or Taxicab Geometry) of the feature vector:\n",
    "\n",
    "$$ x' = \\frac{x}{||x||} $$\n",
    "This is especially important if in the following learning steps the Scalar Metric is used as a distance measure.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feature Selection\n",
    "\n",
    "The most important thing in solving a task is the ability to properly choose or even create features. It’s called Feature Selection and Feature Engineering. While Future Engineering is quite a creative process and relies more on intuition and expert knowledge, there are plenty of ready-made algorithms for Feature Selection. Tree algorithms allow to compute the informativeness of features.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.1036469   0.22530292  0.10056835  0.07999164  0.07552998  0.15237788\n",
      "  0.11254108  0.15004124]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "from sklearn import metrics\n",
    "from sklearn.ensemble import ExtraTreesClassifier\n",
    "\n",
    "model = ExtraTreesClassifier()\n",
    "model.fit(X, y)\n",
    "\n",
    "# display the relative importance of each attribute\n",
    "print(model.feature_importances_)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All other methods are based on the effective search of subsets of features in order to find the best subset, on which the developed model gives the best quality. One of these search algorithms is the Recursive Feature Elimination Algorithm that is also available in the Scikit-Learn library.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ True False False False False  True  True False]\n",
      "[1 2 3 5 6 1 1 4]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_selection import RFE\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "model = LogisticRegression()\n",
    "\n",
    "# create the RFE model and select 3 attributes\n",
    "rfe = RFE(model, 3)\n",
    "rfe = rfe.fit(X, y)\n",
    "\n",
    "# summarize the selection of the attributes\n",
    "print(rfe.support_)\n",
    "print(rfe.ranking_)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Logistic Regression\n",
    "Most often used for solving tasks of classification (binary), but multiclass classification (the so-called one-vs-all method) is also allowed. The advantage of this algorithm is that there’s the probability of belonging to a class for each object at the output.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
      "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
      "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
      "          verbose=0, warm_start=False)\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.79      0.90      0.84       500\n",
      "          1       0.74      0.55      0.63       268\n",
      "\n",
      "avg / total       0.77      0.77      0.77       768\n",
      "\n",
      "[[448  52]\n",
      " [121 147]]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "from sklearn import metrics\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "model = LogisticRegression()\n",
    "model.fit(X, y)\n",
    "print(model)\n",
    "\n",
    "# make predictions\n",
    "expected = y\n",
    "predicted = model.predict(X)\n",
    "\n",
    "# summarize the fit of the model\n",
    "print(metrics.classification_report(expected, predicted))\n",
    "print(metrics.confusion_matrix(expected, predicted))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Naive Bayes\n",
    "This is also one of the most well-known machine learning algorithms, the main task of which is to \n",
    "restore the density of data distribution of the training sample. \n",
    "This method often provides good quality in multiclass classification problems.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GaussianNB(priors=None)\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.80      0.84      0.82       500\n",
      "          1       0.68      0.62      0.64       268\n",
      "\n",
      "avg / total       0.76      0.76      0.76       768\n",
      "\n"
     ]
    }
   ],
   "source": [
    "\n",
    "from sklearn import metrics\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "\n",
    "model = GaussianNB()\n",
    "model.fit(X, y)\n",
    "print(model)\n",
    "\n",
    "# make predictions\n",
    "expected = y\n",
    "predicted = model.predict(X)\n",
    "\n",
    "# summarize the fit of the model\n",
    "print(metrics.classification_report(expected, predicted))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[421  79]\n",
      " [103 165]]\n"
     ]
    }
   ],
   "source": [
    "print(metrics.confusion_matrix(expected, predicted))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### k-Nearest Neighbours\n",
    "The kNN (k-Nearest Neighbors) method is often used as part of a more complex classification algorithm. \n",
    "\n",
    "For instance, we can use its estimate as an object’s feature. Sometimes, a simple kNN provides great quality on well-chosen features. \n",
    "\n",
    "When parameters (metrics mostly) are set well, the algorithm often gives good quality in regression problems.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',\n",
      "           metric_params=None, n_jobs=1, n_neighbors=5, p=2,\n",
      "           weights='uniform')\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.83      0.88      0.85       500\n",
      "          1       0.75      0.65      0.70       268\n",
      "\n",
      "avg / total       0.80      0.80      0.80       768\n",
      "\n",
      "[[442  58]\n",
      " [ 93 175]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# fit a k-nearest neighbor model to the data\n",
    "model = KNeighborsClassifier()\n",
    "model.fit(X, y)\n",
    "print(model)\n",
    "\n",
    "# make predictions\n",
    "expected = y\n",
    "predicted = model.predict(X)\n",
    "\n",
    "# summarize the fit of the model\n",
    "print(metrics.classification_report(expected, predicted))\n",
    "print(metrics.confusion_matrix(expected, predicted))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "#### Decision Trees\n",
    "Classification and Regression Trees (CART) are often used in problems, in which objects have category features and used for regression and classification problems. The trees are very well suited for multiclass classification.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,\n",
      "            max_features=None, max_leaf_nodes=None, min_samples_leaf=1,\n",
      "            min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
      "            presort=False, random_state=None, splitter='best')\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      1.00      1.00       500\n",
      "          1       1.00      1.00      1.00       268\n",
      "\n",
      "avg / total       1.00      1.00      1.00       768\n",
      "\n",
      "[[500   0]\n",
      " [  0 268]]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "from sklearn import metrics\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "\n",
    "# fit a CART model to the data\n",
    "model = DecisionTreeClassifier()\n",
    "model.fit(X, y)\n",
    "print(model)\n",
    "\n",
    "# make predictions\n",
    "expected = y\n",
    "predicted = model.predict(X)\n",
    "\n",
    "# summarize the fit of the model\n",
    "print(metrics.classification_report(expected, predicted))\n",
    "print(metrics.confusion_matrix(expected, predicted))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Support Vector Machines\n",
    "SVM (Support Vector Machines) is one of the most popular machine learning algorithms used mainly for the classification problem. As well as logistic regression, SVM allows multi-class classification with the help of the one-vs-all method.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
      "  decision_function_shape=None, degree=3, gamma='auto', kernel='rbf',\n",
      "  max_iter=-1, probability=False, random_state=None, shrinking=True,\n",
      "  tol=0.001, verbose=False)\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       1.00      1.00      1.00       500\n",
      "          1       1.00      1.00      1.00       268\n",
      "\n",
      "avg / total       1.00      1.00      1.00       768\n",
      "\n",
      "[[500   0]\n",
      " [  0 268]]\n"
     ]
    }
   ],
   "source": [
    "\n",
    "from sklearn import metrics\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "# fit a SVM model to the data\n",
    "model = SVC()\n",
    "model.fit(X, y)\n",
    "print(model)\n",
    "\n",
    "# make predictions\n",
    "expected = y\n",
    "predicted = model.predict(X)\n",
    "\n",
    "# summarize the fit of the model\n",
    "print(metrics.classification_report(expected, predicted))\n",
    "print(metrics.confusion_matrix(expected, predicted))\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "In addition to classification and regression algorithms, Scikit-Learn has a huge number of more complex algorithms, including clustering, and also implemented techniques to create compositions of algorithms, including Bagging and Boosting.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "#### How to Optimize Algorithm Parameters\n",
    "One of the most difficult stages in creating really efficient algorithms is choosing correct parameters. It’s usually easier with experience, but one way or another, we have to do the search. Fortunately, Scikit-Learn provides many implemented functions for this purpose.\n",
    "\n",
    "As an example, let’s take a look at the selection of the regularization parameter, in which several values are searched in turn:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GridSearchCV(cv=None, error_score='raise',\n",
      "       estimator=Ridge(alpha=1.0, copy_X=True, fit_intercept=True, max_iter=None,\n",
      "   normalize=False, random_state=None, solver='auto', tol=0.001),\n",
      "       fit_params={}, iid=True, n_jobs=1,\n",
      "       param_grid={'alpha': array([  1.00000e+00,   1.00000e-01,   1.00000e-02,   1.00000e-03,\n",
      "         1.00000e-04,   0.00000e+00])},\n",
      "       pre_dispatch='2*n_jobs', refit=True, scoring=None, verbose=0)\n",
      "0.279617559313\n",
      "1.0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.linear_model import Ridge\n",
    "from sklearn.grid_search import GridSearchCV\n",
    "\n",
    "# prepare a range of alpha values to test\n",
    "alphas = np.array([1,0.1,0.01,0.001,0.0001,0])\n",
    "\n",
    "# create and fit a ridge regression model, testing each alpha\n",
    "model = Ridge()\n",
    "grid = GridSearchCV(estimator=model, param_grid=dict(alpha=alphas))\n",
    "grid.fit(X, y)\n",
    "print(grid)\n",
    "\n",
    "# summarize the results of the grid search\n",
    "print(grid.best_score_)\n",
    "print(grid.best_estimator_.alpha)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes it is more efficient to randomly select a parameter from the given range, estimate the algorithm quality for this parameter and choose the best one.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomizedSearchCV(cv=None, error_score='raise',\n",
      "          estimator=Ridge(alpha=1.0, copy_X=True, fit_intercept=True, max_iter=None,\n",
      "   normalize=False, random_state=None, solver='auto', tol=0.001),\n",
      "          fit_params={}, iid=True, n_iter=100, n_jobs=1,\n",
      "          param_distributions={'alpha': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f0b36423780>},\n",
      "          pre_dispatch='2*n_jobs', random_state=None, refit=True,\n",
      "          scoring=None, verbose=0)\n",
      "0.279617540491\n",
      "0.9990375546835027\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from scipy.stats import uniform as sp_rand\n",
    "from sklearn.linear_model import Ridge\n",
    "from sklearn.grid_search import RandomizedSearchCV\n",
    "\n",
    "# prepare a uniform distribution to sample for the alpha parameter\n",
    "param_grid = {'alpha': sp_rand()}\n",
    "\n",
    "# create and fit a ridge regression model, testing random alpha values\n",
    "model = Ridge()\n",
    "rsearch = RandomizedSearchCV(estimator=model, param_distributions=param_grid, n_iter=100)\n",
    "rsearch.fit(X, y)\n",
    "print(rsearch)\n",
    "\n",
    "# summarize the results of the random parameter search\n",
    "print(rsearch.best_score_)\n",
    "print(rsearch.best_estimator_.alpha)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}