notebooks/max_grpc_msg_size.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Increasing the Maximum Message Size for gRPC\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequistes\n",
    "You will need\n",
    " - [Git clone of Seldon Core](https://github.com/SeldonIO/seldon-core)\n",
    " - [Helm](https://github.com/kubernetes/helm)\n",
    " - [Minikube](https://github.com/kubernetes/minikube) version v0.24.0 or greater\n",
    " - [python grpc tools](https://grpc.io/docs/quickstart/python.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create Cluster\n",
    "\n",
    "Start minikube and ensure custom resource validation is activated and there is 5G of memory. \n",
    "\n",
    "**2018-06-13** : At present we find the most stable version of minikube across platforms is 0.25.2 as there are issues with 0.26 and 0.27 on some systems. We also find the default VirtualBox driver can be problematic on some systems to we suggest using the [KVM2 driver](https://github.com/kubernetes/minikube/blob/master/docs/drivers.md#kvm2-driver).\n",
    "\n",
    "Your start command would then look like:\n",
    "```\n",
    "minikube start --vm-driver kvm2 --memory 4096 --feature-gates=CustomResourceValidation=true --extra-config=apiserver.Authorization.Mode=RBAC\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "namespace \"seldon\" created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create namespace seldon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "clusterrolebinding \"kube-system-cluster-admin\" created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create clusterrolebinding kube-system-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Install Helm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "serviceaccount \"tiller\" created\n",
      "clusterrolebinding \"tiller\" created\n",
      "$HELM_HOME has been configured at /home/clive/.helm.\n",
      "\n",
      "Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster.\n",
      "\n",
      "Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy.\n",
      "To prevent this, run `helm init` with the --tiller-tls-verify flag.\n",
      "For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation\n",
      "Happy Helming!\n"
     ]
    }
   ],
   "source": [
    "!kubectl -n kube-system create sa tiller\n",
    "!kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller\n",
    "!helm init --service-account tiller"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Start seldon-core"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install the custom resource definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core-crd\n",
      "LAST DEPLOYED: Sat Sep  8 10:02:25 2018\n",
      "NAMESPACE: default\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1beta1/ClusterRole\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  0s\n",
      "\n",
      "==> v1beta1/ClusterRoleBinding\n",
      "NAME                        AGE\n",
      "seldon-spartakus-volunteer  0s\n",
      "\n",
      "==> v1/ConfigMap\n",
      "NAME                     DATA  AGE\n",
      "seldon-spartakus-config  3     0s\n",
      "\n",
      "==> v1beta1/CustomResourceDefinition\n",
      "NAME                                         AGE\n",
      "seldondeployments.machinelearning.seldon.io  0s\n",
      "\n",
      "==> v1beta1/Deployment\n",
      "NAME                        DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "seldon-spartakus-volunteer  1        0        0           0          0s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME                        SECRETS  AGE\n",
      "seldon-spartakus-volunteer  1        0s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "NOTES: TODO\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../helm-charts/seldon-core-crd --name seldon-core-crd --set usage_metrics.enabled=true"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core\n",
      "LAST DEPLOYED: Sat Sep  8 10:04:42 2018\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/ClusterRoleBinding\n",
      "NAME    AGE\n",
      "seldon  0s\n",
      "\n",
      "==> v1/Service\n",
      "NAME                          TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)                        AGE\n",
      "seldon-core-ambassador        NodePort   10.110.189.27   <none>       8080:32124/TCP                 0s\n",
      "seldon-core-ambassador-admin  NodePort   10.103.225.230  <none>       8877:32484/TCP                 0s\n",
      "seldon-core-seldon-apiserver  NodePort   10.99.72.60     <none>       8080:32479/TCP,5000:31269/TCP  0s\n",
      "seldon-core-redis             ClusterIP  10.101.46.175   <none>       6379/TCP                       0s\n",
      "\n",
      "==> v1beta1/Deployment\n",
      "NAME                                DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "seldon-core-ambassador              1        1        1           0          0s\n",
      "seldon-core-seldon-apiserver        1        1        1           0          0s\n",
      "seldon-core-seldon-cluster-manager  1        1        1           0          0s\n",
      "seldon-core-redis                   1        1        1           0          0s\n",
      "\n",
      "==> v1/Pod(related)\n",
      "NAME                                               READY  STATUS             RESTARTS  AGE\n",
      "seldon-core-ambassador-7fb4575f6b-r92bn            0/2    ContainerCreating  0         0s\n",
      "seldon-core-seldon-apiserver-68f5984b49-kpz8z      0/1    ContainerCreating  0         0s\n",
      "seldon-core-seldon-cluster-manager-cfdbbccf-9pmdh  0/1    ContainerCreating  0         0s\n",
      "seldon-core-redis-575979b79b-lr969                 0/1    ContainerCreating  0         0s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME    SECRETS  AGE\n",
      "seldon  1        0s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME        AGE\n",
      "seldon-crd  0s\n",
      "\n",
      "==> v1beta1/Role\n",
      "seldon-local  0s\n",
      "ambassador    0s\n",
      "\n",
      "==> v1/RoleBinding\n",
      "NAME    AGE\n",
      "seldon  0s\n",
      "\n",
      "==> v1beta1/RoleBinding\n",
      "ambassador  0s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "Thank you for installing Seldon Core.\n",
      "\n",
      "Documentation can be found at https://github.com/SeldonIO/seldon-core\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../helm-charts/seldon-core --name seldon-core --namespace seldon \\\n",
    "    --set ambassador.enabled=true"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check all services are running before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment \"seldon-core-seldon-cluster-manager\" successfully rolled out\n",
      "deployment \"seldon-core-seldon-apiserver\" successfully rolled out\n",
      "Waiting for rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"seldon-core-ambassador\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/seldon-core-seldon-cluster-manager\n",
    "!kubectl rollout status deploy/seldon-core-seldon-apiserver\n",
    "!kubectl rollout status deploy/seldon-core-ambassador "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set up REST and gRPC methods\n",
    "\n",
    "**Ensure you port forward to API Gateway**\n",
    "\n",
    "REST:\n",
    "\n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].metadata.name}') -n seldon 8003:8080\n",
    "```\n",
    "\n",
    "GRPC:\n",
    "\n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l app=seldon-apiserver-container-app -o jsonpath='{.items[0].metadata.name}') -n seldon 8004:5000\n",
    "```\n",
    "\n",
    "**Ensure you port forward ambassador**:\n",
    "\n",
    "```\n",
    "kubectl port-forward $(kubectl get pods -n seldon -l service=ambassador -o jsonpath='{.items[0].metadata.name}') -n seldon 8005:8080\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install gRPC modules for the prediction protos."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp ../proto/prediction.proto ./proto\n",
    "!python -m grpc.tools.protoc -I. --python_out=. --grpc_python_out=. ./proto/prediction.proto"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \u001b[34;01m\"apiVersion\"\u001b[39;49;00m: \u001b[33m\"machinelearning.seldon.io/v1alpha2\"\u001b[39;49;00m,\n",
      "    \u001b[34;01m\"kind\"\u001b[39;49;00m: \u001b[33m\"SeldonDeployment\"\u001b[39;49;00m,\n",
      "    \u001b[34;01m\"metadata\"\u001b[39;49;00m: {\n",
      "        \u001b[34;01m\"labels\"\u001b[39;49;00m: {\n",
      "            \u001b[34;01m\"app\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\n",
      "        },\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"seldon-model\"\u001b[39;49;00m\n",
      "    },\n",
      "    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"test-deployment\"\u001b[39;49;00m,\n",
      "        \u001b[34;01m\"oauth_key\"\u001b[39;49;00m: \u001b[33m\"oauth-key\"\u001b[39;49;00m,\n",
      "        \u001b[34;01m\"oauth_secret\"\u001b[39;49;00m: \u001b[33m\"oauth-secret\"\u001b[39;49;00m,\n",
      "        \u001b[34;01m\"predictors\"\u001b[39;49;00m: [\n",
      "            {\n",
      "                \u001b[34;01m\"componentSpecs\"\u001b[39;49;00m: [{\n",
      "                    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\n",
      "                        \u001b[34;01m\"containers\"\u001b[39;49;00m: [\n",
      "                            {\n",
      "                                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/mock_classifier:1.0\"\u001b[39;49;00m,\n",
      "                                \u001b[34;01m\"imagePullPolicy\"\u001b[39;49;00m: \u001b[33m\"IfNotPresent\"\u001b[39;49;00m,\n",
      "                                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\n",
      "                                \u001b[34;01m\"resources\"\u001b[39;49;00m: {\n",
      "                                    \u001b[34;01m\"requests\"\u001b[39;49;00m: {\n",
      "                                        \u001b[34;01m\"memory\"\u001b[39;49;00m: \u001b[33m\"1Mi\"\u001b[39;49;00m\n",
      "                                    }\n",
      "                                }\n",
      "                            }\n",
      "                        ],\n",
      "                        \u001b[34;01m\"terminationGracePeriodSeconds\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m\n",
      "                    }\n",
      "                }],\n",
      "                \u001b[34;01m\"graph\"\u001b[39;49;00m: {\n",
      "                    \u001b[34;01m\"children\"\u001b[39;49;00m: [],\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\n",
      "                    \u001b[34;01m\"endpoint\"\u001b[39;49;00m: {\n",
      "\t\t\t\u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"REST\"\u001b[39;49;00m\n",
      "\t\t    },\n",
      "                    \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m\n",
      "                },\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"example\"\u001b[39;49;00m,\n",
      "                \u001b[34;01m\"replicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\n",
      "\t\t\u001b[34;01m\"labels\"\u001b[39;49;00m: {\n",
      "\t\t    \u001b[34;01m\"version\"\u001b[39;49;00m : \u001b[33m\"v1\"\u001b[39;49;00m\n",
      "\t\t}\n",
      "            }\n",
      "        ]\n",
      "    }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "!pygmentize resources/model.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Seldon Deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deploy the runtime graph to kubernetes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment \"seldon-model\" created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl apply -f resources/model.json -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the status of the SeldonDeployment. **When ready the replicasAvailable should be 1**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "map[predictorStatus:[map[name:test-deployment-example-svc-orch replicas:1 replicasAvailable:1] map[name:test-deployment-example-classifier-0 replicas:1 replicasAvailable:1]]]"
     ]
    }
   ],
   "source": [
    "!kubectl get seldondeployments seldon-model -o jsonpath='{.status}' -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get predictions - no grpx max message size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from seldon_utils import *\n",
    "API_GATEWAY_REST=\"localhost:8003\"\n",
    "API_GATEWAY_GRPC=\"localhost:8004\"\n",
    "API_AMBASSADOR=\"localhost:8005\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send a small request which should suceed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"access_token\":\"522e6dd3-2232-44a7-a547-e3babc939ef0\",\"token_type\":\"bearer\",\"expires_in\":42264,\"scope\":\"read write\"}\n",
      "meta {\n",
      "  puid: \"fj2rftu976hrmde87v0dcmkhv7\"\n",
      "}\n",
      "data {\n",
      "  names: \"proba\"\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 1\n",
      "    values: 0.0787334522874084\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "grpc_request_api_gateway('oauth-key','oauth-secret',API_GATEWAY_REST,API_GATEWAY_GRPC,10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test via Ambassador"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "meta {\n",
      "  puid: \"qacuk4o0qk7dvvo963t8o1svo1\"\n",
      "}\n",
      "data {\n",
      "  names: \"proba\"\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 1\n",
      "    values: 0.0898729704574218\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "grpc_request_ambassador(\"seldon-model\",API_AMBASSADOR,data_size=10,rows=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send a large request which will be above the default gRPC message size and will fail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"access_token\":\"50094bf3-3141-48bf-958a-69a57c1896bd\",\"token_type\":\"bearer\",\"expires_in\":43181,\"scope\":\"read write\"}\n"
     ]
    },
    {
     "ename": "_Rendezvous",
     "evalue": "<_Rendezvous of RPC that terminated with (StatusCode.CANCELLED, Received RST_STREAM with error code 8)>",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31m_Rendezvous\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-17-cdf816bcf337>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mgrpc_request_api_gateway\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'oauth-key'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'oauth-secret'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mAPI_GATEWAY_REST\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mAPI_GATEWAY_GRPC\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m1000000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/work/seldon-core/fork-seldon-core/notebooks/seldon_utils.py\u001b[0m in \u001b[0;36mgrpc_request_api_gateway\u001b[0;34m(oauth_key, oauth_secret, rest_endpoint, grpc_endpoint, data_size)\u001b[0m\n\u001b[1;32m     46\u001b[0m     \u001b[0mstub\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mprediction_pb2_grpc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSeldonStub\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchannel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     47\u001b[0m     \u001b[0mmetadata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'oauth_token'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtoken\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 48\u001b[0;31m     \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstub\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mmetadata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmetadata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     49\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     50\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/grpc/_channel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, request, timeout, metadata, credentials)\u001b[0m\n\u001b[1;32m    485\u001b[0m         state, call, deadline = self._blocking(request, timeout, metadata,\n\u001b[1;32m    486\u001b[0m                                                credentials)\n\u001b[0;32m--> 487\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_end_unary_response_blocking\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcall\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdeadline\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    488\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    489\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mwith_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmetadata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcredentials\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/grpc/_channel.py\u001b[0m in \u001b[0;36m_end_unary_response_blocking\u001b[0;34m(state, call, with_call, deadline)\u001b[0m\n\u001b[1;32m    435\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mstate\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    436\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 437\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0m_Rendezvous\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdeadline\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    438\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    439\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31m_Rendezvous\u001b[0m: <_Rendezvous of RPC that terminated with (StatusCode.CANCELLED, Received RST_STREAM with error code 8)>"
     ]
    }
   ],
   "source": [
    "grpc_request_api_gateway('oauth-key','oauth-secret',API_GATEWAY_REST,API_GATEWAY_GRPC,1000000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "ename": "_Rendezvous",
     "evalue": "<_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, upstream connect error or disconnect/reset before headers)>",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31m_Rendezvous\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-12-dcff4a23459e>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mgrpc_request_ambassador\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"seldon-model\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mAPI_AMBASSADOR\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mdata_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1000000\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mrows\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/work/seldon-core/fork-seldon-core/notebooks/seldon_utils.py\u001b[0m in \u001b[0;36mgrpc_request_ambassador\u001b[0;34m(deploymentName, endpoint, data_size, rows)\u001b[0m\n\u001b[1;32m     81\u001b[0m     \u001b[0mstub\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mprediction_pb2_grpc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSeldonStub\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchannel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     82\u001b[0m     \u001b[0mmetadata\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'seldon'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mdeploymentName\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 83\u001b[0;31m     \u001b[0mresponse\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstub\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mmetadata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmetadata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     84\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     85\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/grpc/_channel.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, request, timeout, metadata, credentials)\u001b[0m\n\u001b[1;32m    485\u001b[0m         state, call, deadline = self._blocking(request, timeout, metadata,\n\u001b[1;32m    486\u001b[0m                                                credentials)\n\u001b[0;32m--> 487\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_end_unary_response_blocking\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcall\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdeadline\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    488\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    489\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mwith_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmetadata\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcredentials\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/grpc/_channel.py\u001b[0m in \u001b[0;36m_end_unary_response_blocking\u001b[0;34m(state, call, with_call, deadline)\u001b[0m\n\u001b[1;32m    435\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mstate\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresponse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    436\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 437\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0m_Rendezvous\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstate\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdeadline\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    438\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    439\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31m_Rendezvous\u001b[0m: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, upstream connect error or disconnect/reset before headers)>"
     ]
    }
   ],
   "source": [
    "grpc_request_ambassador(\"seldon-model\",API_AMBASSADOR,data_size=1000000,rows=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment \"seldon-model\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl delete -f resources/model.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "release \"seldon-core\" deleted\r\n"
     ]
    }
   ],
   "source": [
    "!helm delete seldon-core --purge"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Allowing larger gRPC messages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recreate seldon-core with extra annotation for the API Gateway"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NAME:   seldon-core\n",
      "LAST DEPLOYED: Sat Sep  8 10:31:47 2018\n",
      "NAMESPACE: seldon\n",
      "STATUS: DEPLOYED\n",
      "\n",
      "RESOURCES:\n",
      "==> v1/RoleBinding\n",
      "NAME    AGE\n",
      "seldon  0s\n",
      "\n",
      "==> v1/Service\n",
      "NAME                          TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)                        AGE\n",
      "seldon-core-ambassador        NodePort   10.108.32.98   <none>       8080:32409/TCP                 0s\n",
      "seldon-core-ambassador-admin  NodePort   10.106.70.248  <none>       8877:32597/TCP                 0s\n",
      "seldon-core-seldon-apiserver  NodePort   10.104.51.168  <none>       8080:32546/TCP,5000:30907/TCP  0s\n",
      "seldon-core-redis             ClusterIP  10.98.60.20    <none>       6379/TCP                       0s\n",
      "\n",
      "==> v1/ClusterRoleBinding\n",
      "NAME    AGE\n",
      "seldon  0s\n",
      "\n",
      "==> v1beta1/Role\n",
      "NAME          AGE\n",
      "ambassador    0s\n",
      "seldon-local  0s\n",
      "\n",
      "==> v1beta1/RoleBinding\n",
      "NAME        AGE\n",
      "ambassador  0s\n",
      "\n",
      "==> v1beta1/Deployment\n",
      "NAME                                DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE\n",
      "seldon-core-ambassador              1        1        1           0          0s\n",
      "seldon-core-seldon-apiserver        1        1        1           0          0s\n",
      "seldon-core-seldon-cluster-manager  1        1        1           0          0s\n",
      "seldon-core-redis                   1        1        1           0          0s\n",
      "\n",
      "==> v1/Pod(related)\n",
      "NAME                                               READY  STATUS             RESTARTS  AGE\n",
      "seldon-core-ambassador-7fb4575f6b-mb5m2            0/2    Terminating        0         19m\n",
      "seldon-core-ambassador-7fb4575f6b-t62r2            0/2    ContainerCreating  0         0s\n",
      "seldon-core-seldon-apiserver-6f4b96b5cc-8twkg      0/1    ContainerCreating  0         0s\n",
      "seldon-core-seldon-apiserver-6f4b96b5cc-mvqkb      1/1    Terminating        0         19m\n",
      "seldon-core-seldon-cluster-manager-cfdbbccf-2bvb8  0/1    ContainerCreating  0         0s\n",
      "seldon-core-seldon-cluster-manager-cfdbbccf-5l9sm  0/1    Terminating        0         19m\n",
      "seldon-core-redis-575979b79b-v57q4                 0/1    ContainerCreating  0         0s\n",
      "\n",
      "==> v1/ServiceAccount\n",
      "NAME    SECRETS  AGE\n",
      "seldon  1        0s\n",
      "\n",
      "==> v1beta1/ClusterRole\n",
      "NAME        AGE\n",
      "seldon-crd  0s\n",
      "\n",
      "\n",
      "NOTES:\n",
      "Thank you for installing Seldon Core.\n",
      "\n",
      "Documentation can be found at https://github.com/SeldonIO/seldon-core\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!helm install ../helm-charts/seldon-core --name seldon-core --namespace seldon \\\n",
    "    --set ambassador.enabled=true \\\n",
    "    -f grpc_size_values.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for seldon core deployment to be running"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deployment \"seldon-core-seldon-cluster-manager\" successfully rolled out\n",
      "deployment \"seldon-core-seldon-apiserver\" successfully rolled out\n",
      "Waiting for rollout to finish: 0 of 1 updated replicas are available...\n",
      "deployment \"seldon-core-ambassador\" successfully rolled out\n"
     ]
    }
   ],
   "source": [
    "!kubectl rollout status deploy/seldon-core-seldon-cluster-manager\n",
    "!kubectl rollout status deploy/seldon-core-seldon-apiserver\n",
    "!kubectl rollout status deploy/seldon-core-ambassador "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we change our SeldonDeployment to include a annotation for max grpx message size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\r\n",
      "    \u001b[34;01m\"apiVersion\"\u001b[39;49;00m: \u001b[33m\"machinelearning.seldon.io/v1alpha2\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"kind\"\u001b[39;49;00m: \u001b[33m\"SeldonDeployment\"\u001b[39;49;00m,\r\n",
      "    \u001b[34;01m\"metadata\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"labels\"\u001b[39;49;00m: {\r\n",
      "            \u001b[34;01m\"app\"\u001b[39;49;00m: \u001b[33m\"seldon\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"seldon-model\"\u001b[39;49;00m\r\n",
      "    },\r\n",
      "    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "        \u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "\t    \u001b[34;01m\"seldon.io/grpc-max-message-size\"\u001b[39;49;00m:\u001b[33m\"10000000\"\u001b[39;49;00m\r\n",
      "        },\r\n",
      "        \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"test-deployment\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_key\"\u001b[39;49;00m: \u001b[33m\"oauth-key\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"oauth_secret\"\u001b[39;49;00m: \u001b[33m\"oauth-secret\"\u001b[39;49;00m,\r\n",
      "        \u001b[34;01m\"predictors\"\u001b[39;49;00m: [\r\n",
      "            {\r\n",
      "                \u001b[34;01m\"componentSpecs\"\u001b[39;49;00m: [{\r\n",
      "                    \u001b[34;01m\"spec\"\u001b[39;49;00m: {\r\n",
      "                        \u001b[34;01m\"containers\"\u001b[39;49;00m: [\r\n",
      "                            {\r\n",
      "                                \u001b[34;01m\"image\"\u001b[39;49;00m: \u001b[33m\"seldonio/mock_classifier_grpc:1.0\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"imagePullPolicy\"\u001b[39;49;00m: \u001b[33m\"IfNotPresent\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                                \u001b[34;01m\"resources\"\u001b[39;49;00m: {\r\n",
      "                                    \u001b[34;01m\"requests\"\u001b[39;49;00m: {\r\n",
      "                                        \u001b[34;01m\"memory\"\u001b[39;49;00m: \u001b[33m\"1Mi\"\u001b[39;49;00m\r\n",
      "                                    }\r\n",
      "                                }\r\n",
      "                            }\r\n",
      "                        ],\r\n",
      "                        \u001b[34;01m\"terminationGracePeriodSeconds\"\u001b[39;49;00m: \u001b[34m20\u001b[39;49;00m\r\n",
      "                    }\r\n",
      "                }],\r\n",
      "                \u001b[34;01m\"graph\"\u001b[39;49;00m: {\r\n",
      "                    \u001b[34;01m\"children\"\u001b[39;49;00m: [],\r\n",
      "                    \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"classifier\"\u001b[39;49;00m,\r\n",
      "                    \u001b[34;01m\"endpoint\"\u001b[39;49;00m: {\r\n",
      "\t\t\t\u001b[34;01m\"type\"\u001b[39;49;00m : \u001b[33m\"GRPC\"\u001b[39;49;00m\r\n",
      "\t\t    },\r\n",
      "                    \u001b[34;01m\"type\"\u001b[39;49;00m: \u001b[33m\"MODEL\"\u001b[39;49;00m\r\n",
      "                },\r\n",
      "                \u001b[34;01m\"name\"\u001b[39;49;00m: \u001b[33m\"grpc-size\"\u001b[39;49;00m,\r\n",
      "                \u001b[34;01m\"replicas\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m,\r\n",
      "\t\t\u001b[34;01m\"annotations\"\u001b[39;49;00m: {\r\n",
      "\t\t    \u001b[34;01m\"predictor_version\"\u001b[39;49;00m : \u001b[33m\"v1\"\u001b[39;49;00m\r\n",
      "\t\t}\r\n",
      "            }\r\n",
      "        ]\r\n",
      "    }\r\n",
      "}\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize resources/model_grpc_size.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seldondeployment \"seldon-model\" created\r\n"
     ]
    }
   ],
   "source": [
    "!kubectl create -f resources/model_grpc_size.json -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Make sure you rerun the port forwards above as a new seldon core API gateway and Ambassador will have started**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "map[predictorStatus:[map[name:test-deployment-grpc-size-svc-orch replicas:1 replicasAvailable:1] map[name:test-deployment-grpc-size-classifier-0 replicas:1 replicasAvailable:1]]]"
     ]
    }
   ],
   "source": [
    "!kubectl get seldondeployments seldon-model -o jsonpath='{.status}' -n seldon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send a large message. This time it should succeed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"access_token\":\"a4929490-a3c9-4d1a-956d-9c173a22ba8a\",\"token_type\":\"bearer\",\"expires_in\":43199,\"scope\":\"read write\"}\n",
      "meta {\n",
      "  puid: \"uv6fgur0ggvjntduisrgdccvdn\"\n",
      "}\n",
      "data {\n",
      "  names: \"proba\"\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 1\n",
      "    values: 0.08190040853130452\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "grpc_request_api_gateway('oauth-key','oauth-secret',API_GATEWAY_REST,API_GATEWAY_GRPC,data_size=1000000,rows=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Send a request via ambassador. This should also succeed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "meta {\n",
      "  puid: \"a8lgbq67prn8huq4mmdjgur5js\"\n",
      "}\n",
      "data {\n",
      "  names: \"proba\"\n",
      "  tensor {\n",
      "    shape: 1\n",
      "    shape: 1\n",
      "    values: 0.08562220593812069\n",
      "  }\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "grpc_request_ambassador(\"seldon-model\",API_AMBASSADOR,data_size=10,rows=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}