400_deepracer_rl_reinvent2019.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dive under the hood of AWS DeepRacer \n",
    "\n",
    "---\n",
    "## Introduction\n",
    "\n",
    "[AWS DeepRacer](https://console.aws.amazon.com/deepracer/home?region=us-east-1#getStarted) is a 1/18th scale autonomous race car designed to get developers hands-on with reinforcement learning (RL). AWS DeepRacer leverages [AWS RoboMaker](https://console.aws.amazon.com/robomaker/home?region=us-east-1#) to create a simulation environment in which our reinforcement learning agent (i.e. our autonomous car) can learn how to drive autonomously. The learning part, or training, is facilitated using  [Amazon SageMaker](https://console.aws.amazon.com/sagemaker/home?region=us-east-1#/dashboard).\n",
    "\n",
    "The AWS DeepRacer console is optimized to provide a user-friendly introduction to reinforcement learning to developers new to machine learning. This lab-as-a-notebook is intended to provide developers with an deep dive on how they can use an Amazon SageMaker notebook to achieve the following in their AWS DeepRacer models:\n",
    "\n",
    "- Augment the simulation environment\n",
    "- Manipulate inputs to the neural network\n",
    "- Modify neural network architecture\n",
    "- Run distributed rollouts\n",
    "- Debug your model\n",
    "\n",
    "The following image gives some context to our agenda and to the training process that we will cover next.\n",
    "\n",
    "![Agenda in Action](img/DistributedN.png)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## How it works?  \n",
    "\n",
    "\n",
    "The reinforcement learning agent learns to drive by interacting with its environment, e.g., the track, by taking an action in a given state to maximize the cumulative expected reward across all possible future actions. The agent learns the optimal plan of actions in training by trial-and-error through repeated episodes.  \n",
    "  \n",
    "The figure above shows an example of distributed RL training across SageMaker and four RoboMaker simulation envrionments that perform the **rollouts** - execute a fixed number of episodes using the current model or policy. The rollouts collect agent experiences (state-action-reward-next state tuples) and share this data with SageMaker for training. SageMaker updates the model policy which is then used to execute the next sequence of rollouts. This training loop continues until the model converges, i.e. the car learns to drive and stops going off-track. More formally, we can define the problem in terms of the following:  \n",
    "\n",
    "1. **Objective**: Learn to drive autonomously by staying on the track and completing the lap as quick as possible.\n",
    "2. **Environment**: A 3D driving simulator hosted on AWS RoboMaker.\n",
    "3. **State**: The driving point of view (be it image or LIDAR or other sensor) captured by the car's sensors\n",
    "4. **Action**: Discrete action space that the car can choose from: driving angle and speed (configurable)\n",
    "5. **Reward**: Positive reward for staying close to the center line; High penalty for going off-track. This is configurable and can be made more complex (for e.g. steering penalty can be added).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "---\n",
    "## Architecture  \n",
    "\n",
    "In the background AWS DeepRacer uses Amazon SageMaker, AWS RoboMaker, Amazon S3, Amazon Kinesis Video Streams, and Amazon Cloudwatch to create, train, evaluate, and store models and logs, and display video streams during training and evaluation. Trained models can be downloaded and run on AWS DeepRacer, which uses Intel's OpenVINO to optimize and accelarate inference, allowing your car to drive faster.\n",
    "\n",
    "![Architecture](img/Architecture.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AWS RoboMaker Simulation Application\n",
    "\n",
    "AWS RoboMaker hosts the 3D racing simulator in the cloud. To start the simulation AWS RoboMaker requires a simulation application. While training is running you can navigate to AWS RoboMaker and observe the simulation. The simulation application contains the 3D assets (meshes, textures, 3D models, amongst other things) needed for the simulation to run. This simulation application is a pre-compiled tarball. However, it can be unpacked (carefully), edited, and repacked allowing you to add or edit race tracks. You have to make these changes before you start the simulation.\n",
    "\n",
    "We provide a pre-built simulation application in `build/output.tar.gz` and 3D assets for customizing your world environment in `build/simapp` folder. \n",
    "\n",
    "Download the pre-built simulation environment for customizing simulation assets and extract to the notebook folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Hint\n",
    "Jupyter notebook shortcut >> to uncomment a block of code, select the relevant lines and press Ctrl + /\n",
    "Press it again to comment the code again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !aws s3 cp s3://xxx/build.tar.gz .\n",
    "# !tar -xzvf build.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1: Customizing the simulation track and objects\n",
    "\n",
    "\n",
    "You may be interested in changing tracks (adding a new track, or editing an existing one), adding objects to your track, or adding bot cars to train against.\n",
    "\n",
    "![changetracks](img/Changetracks.png)\n",
    "\n",
    "\n",
    "### Adding new tracks to your simulation application\n",
    "\n",
    "Racing tracks with various shapes and textures are included among the 3D assets `build/simapp/bundle/opt/install/deepracer_simulation_environment/share/deepracer_simulation_environment`.\n",
    "\n",
    "To add a new track, you need to provide the following files:\n",
    "\n",
    "- 3D assets `meshes/mytrack/mytrack.dae` and textures `meshes/mytrack/textures`. Note that AWS RoboMaker currently uses Gazebo to create the simulation environment, so everything needs to be compatible with Gazebo. For the already existing tracks, we provide Blender files for modifications as needed.\n",
    "- `models/mytrack/mytrack.sdf` and `models/mytrack/model.config`, which point to the meshes and allows us to construct a completed model of the meshes where we specify which meshes should be visual and which should provide collision to our car \n",
    "- `worlds/myworld.world`, which takes our track model and wraps it in a world file, complete with sky and orientation.\n",
    "- `routes/mytrack.npy`, contains the waypoints of our track, outside border, center, and inside border, which our simulation will use to determine things such as is the agent on the track, and how much prrogress has been made etc.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding new objects (excluding bot car) on the track\n",
    "\n",
    "To add objects, you can use the following files:\n",
    "\n",
    "- `models/racecar_box/model.sdf` and `models/racecar_box/model.config`, contain model of a box object. Any other Gazebo compatiable object can be added by creating a folder with these files under `models` folder.\n",
    "- `worlds/myworld_with_objects.world`, which takes our prior track assets and our new object models. Any number of objects can be added by including the following lines in the world file: \n",
    "\n",
    "```\n",
    "   <include>\n",
    "    <pose>0 0 0 0 0 0</pose>\n",
    "    <static>true</static>\n",
    "    <uri>model://models/racecar_box</uri>\n",
    "    <name>box1</name>\n",
    "   </include>\n",
    "```\n",
    "- To replace any object during training, make a note of the object name, e.g., box1, and in your environment file, import `rospy` library and `set_model_state` function.\n",
    "\n",
    "\n",
    "### Adding new bot cars\n",
    "\n",
    "A bot car is simply another robocar without the neural network, hence, uses the same assets as the robocar in   `build/simapp/bundle/opt/install/deepracer_simulation_environment/share/deepracer_simulation_environment`. The bot car uses the same functionality such as reset in `build/simapp/bundle/opt/install/deepracer_simulation_environment/lib/deepracer_simulation_environment/car_node.py`\n",
    "\n",
    "To add, remove, or modify the speed and lane changing behavior bot cars, use function in `src/markov/environments/deepracer_racetrack_env.py`:\n",
    "- `class BotCarController` defines the functionality for the bot car behavior, including which segments on the track to spawn the bot car. Note that learning passing around corners is difficult due to partial observability, hence, initially, you can train a model to pass only on the straight segments to reduce the training time.\n",
    "- `class DeepRacerRacetrackEnv(gym.Env) def __init__(self)` is used to set the number of bot cars, their speed, and lane changning frequency.\n",
    "\n",
    "\n",
    "### Adding and configuring new sensors\n",
    "\n",
    "A new sensor such as camera or LIDAR with existing ROS plug-in can be added in directly in the Gazebo car asset files located in `build/simapp/bundle/opt/install/deepracer_simulation_environment/share/deepracer_simulation_environment/urdf/`\n",
    "\n",
    "To add another camera: \n",
    "\n",
    "- `racecar.gazebo` in the original time-trial already includes a camera, to add another camerara you can replicate that code snippet with a new name. Alternatively, you can add a less to the existing camera.\n",
    "\n",
    "- `racecar.xacro` contains the parameters to set the location of the cameras, see `_leftcam` and `_rightcam`.\n",
    "\n",
    "- `macro.xacro` contains the joints that the sensors are attached to including cameras and lenses for the cameras. The angle of the camera lenses are adjusted in this file.\n",
    "\n",
    "\n",
    "To add a LIDAR:\n",
    "\n",
    "- `racecar.gazebo` uses a LIDAR plugin compatible with the LIDAR on the device, similar to the camera module. There are two parameters that can be customized for the LIDAR, `scan` and `range`. These parameters determine the angle bracket and range to detect light reflections. The default settings below are for +/- 1.0472 radians (60 degrees) for the scanning angle and is between 0.15 and 0.5 meters for the observability range.\n",
    "```\n",
    "<scan>\n",
    "  <horizontal>\n",
    "    <samples>64</samples>\n",
    "    <resolution>1</resolution>\n",
    "    <min_angle>-1.0472</min_angle>\n",
    "    <max_angle>1.0472</max_angle>\n",
    "  </horizontal>\n",
    "</scan>\n",
    "<range>\n",
    "  <min>0.15</min>\n",
    "  <max>0.5</max>\n",
    "  <resolution>0.01</resolution>\n",
    "</range>\n",
    "```\n",
    "\n",
    "- `racecar.xacro` contains the parameters to set the location of the LIDAR, similar to the camera.\n",
    "\n",
    "- `macro.xacro` contains the joints that the sensors are attached to and its angle.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the AWS RoboMaker Bundle\n",
    "\n",
    "After making changes to the simulation application assets, re-bundle it using the Python file sim_app_bundler.py. We will upload the tar.gz file to the AWS RoboMaker arn later in the notebook.\n",
    "\n",
    "The compression may take longer depending on the instance type of your Amazon SageMaker notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bundling the SimApp output.tar.gz\n",
      "Removing previously build build/output.tar.gz\n",
      "Creating tar of build/simapp/bundle\n",
      "Removing the previously created tar files\n",
      "============ Archiving completed. Available at build/Output.tar.gz============ \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# #Only when you have changes to SimApp\n",
    "# !python3 sim_app_bundler.py --tar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now **run the following blocks of code until you get to Section 2**.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Starting our Amazon SageMaker Notebook DeepRacer training environment\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get started, we'll import the Python libraries we need, set up the environment with a few prerequisites for permissions and configurations.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import sys\n",
    "import os\n",
    "import re\n",
    "import numpy as np\n",
    "import subprocess\n",
    "sys.path.append(\"common\")\n",
    "from misc import get_execution_role, wait_for_s3_object\n",
    "from docker_utils import build_and_push_docker_image\n",
    "from sagemaker.rl import RLEstimator, RLToolkit, RLFramework\n",
    "from time import gmtime, strftime\n",
    "import time\n",
    "from IPython.display import Markdown\n",
    "from markdown_helper import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initializing basic parameters\n",
    "\n",
    "**Call-out, please check Amazon SageMaker instance type** - it has a cost implication\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select the instance type\n",
    "# Be aware of the costs of these training instances. They will run as long as the training duration\n",
    "# We use ml.c4.2xlarge or ml.c5.2xlarge in the AWS DeepRacer console\n",
    "\n",
    "\n",
    "# instance_type = \"ml.c4.2xlarge\"\n",
    "instance_type = \"ml.p2.xlarge\"\n",
    "\n",
    "\n",
    "# Starting SageMaker session\n",
    "sage_session = sagemaker.session.Session()\n",
    "\n",
    "# Create unique job name.\n",
    "job_name_prefix = 'deepracer-notebook'\n",
    "\n",
    "# Duration of job in seconds (1 hours)\n",
    "job_duration_in_seconds = 3600 * 1\n",
    "\n",
    "# AWS Region\n",
    "aws_region = sage_session.boto_region_name\n",
    "if aws_region not in [\"us-west-2\", \"us-east-1\", \"eu-west-1\"]:\n",
    "    raise Exception(\"This notebook uses RoboMaker which is available only in US East (N. Virginia),\"\n",
    "                    \"US West (Oregon) and EU (Ireland). Please switch to one of these regions.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup S3 bucket\n",
    "Set up the linkage and authentication to the S3 bucket that we want to use for checkpoint and metadata.\n",
    "\n",
    "#### **Note: This cell snapshots time to create folders** - training output will go to these folders. So running the code again means you are pointing to a different location. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using s3 bucket sagemaker-us-west-2-828753602420\n",
      "Model checkpoints and other metadata will be stored at: \n",
      "s3://sagemaker-us-west-2-828753602420/deepracer-notebook-sagemaker-191201-005405\n"
     ]
    }
   ],
   "source": [
    "# S3 bucket\n",
    "s3_bucket = sage_session.default_bucket()\n",
    "\n",
    "# SDK appends the job name and output folder\n",
    "s3_output_path = 's3://{}/'.format(s3_bucket)\n",
    "\n",
    "# Ensure that the S3 prefix contains the keyword 'sagemaker'\n",
    "s3_prefix = job_name_prefix + \"-sagemaker-\" + strftime(\"%y%m%d-%H%M%S\", gmtime())\n",
    "\n",
    "# Get the AWS account id of this account\n",
    "sts = boto3.client(\"sts\")\n",
    "account_id = sts.get_caller_identity()['Account']\n",
    "\n",
    "print(\"Using s3 bucket {}\".format(s3_bucket))\n",
    "print(\"Model checkpoints and other metadata will be stored at: \\ns3://{}/{}\".format(s3_bucket, s3_prefix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an IAM role\n",
    "Either get the execution role when running from a SageMaker notebook `role = sagemaker.get_execution_role()` or, when running from local machine, use utils method `role = get_execution_role('role_name')` to create an execution role."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using Sagemaker IAM role arn: \n",
      "arn:aws:iam::828753602420:role/service-role/AmazonSageMaker-ExecutionRole-20190604T142184\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    sagemaker_role = sagemaker.get_execution_role()\n",
    "except:\n",
    "    sagemaker_role = get_execution_role('sagemaker')\n",
    "\n",
    "print(\"Using Sagemaker IAM role arn: \\n{}\".format(sagemaker_role))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Please note that this notebook cannot be run in `SageMaker local mode` as the simulator is based on AWS RoboMaker service."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Permission setup for invoking AWS RoboMaker from this notebook\n",
    "**Please update the SageMaker role's trust**. In order to enable this notebook to be able to execute AWS RoboMaker jobs, we need to add one trust relationship to the default execution role of this notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "1. Go to IAM console to edit current SageMaker role: [AmazonSageMaker-ExecutionRole-20190604T142184](https://console.aws.amazon.com/iam/home#/roles/AmazonSageMaker-ExecutionRole-20190604T142184).\n",
       "2. Next, go to the `Trust relationships tab` and click on `Edit Trust Relationship.` \n",
       "3. Replace the JSON blob with the following:\n",
       "```json\n",
       "            {\n",
       "              \"Version\": \"2012-10-17\",\n",
       "              \"Statement\": [\n",
       "                {\n",
       "                  \"Effect\": \"Allow\",\n",
       "                  \"Principal\": {\n",
       "                    \"Service\": [\n",
       "                      \"sagemaker.amazonaws.com\",\n",
       "                      \"robomaker.amazonaws.com\"\n",
       "                    ]\n",
       "                  },\n",
       "                  \"Action\": \"sts:AssumeRole\"\n",
       "                }\n",
       "              ]\n",
       "            }```\n",
       "4. Once this is complete, click on Update Trust Policy and you are done."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(generate_help_for_robomaker_trust_relationship(sagemaker_role)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Permission setup for Sagemaker to S3 bucket\n",
    "\n",
    "**Please update the SageMaker role's permissions** in IAM. The sagemaker writes the Redis IP address, models to the S3 bucket. This requires PutObject permission on the bucket. Make sure the sagemaker role you are using as this permissions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "1. Go to IAM console to edit current SageMaker role: [AmazonSageMaker-ExecutionRole-20190604T142184](https://console.aws.amazon.com/iam/home#/roles/AmazonSageMaker-ExecutionRole-20190604T142184).\n",
       "2. Next, go to the `Permissions tab` and click on `Attach Policy.` \n",
       "3. Search and select `AmazonKinesisVideoStreamsFullAccess` policy\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(generate_s3_write_permission_for_sagemaker_role(sagemaker_role)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Permission setup for Sagemaker to create KinesisVideoStreams\n",
    "\n",
    "**Please update the SageMaker role permissions** in IAM. The sagemaker notebook has to create a kinesis video streamer. You can observer the car making epsiodes in the kinesis video streamer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "1. Go to IAM console to edit current SageMaker role: [AmazonSageMaker-ExecutionRole-20190604T142184](https://console.aws.amazon.com/iam/home#/roles/AmazonSageMaker-ExecutionRole-20190604T142184).\n",
       "2. Next, go to the `Permissions tab` and click on `Attach Policy.` \n",
       "3. Search and select `AmazonS3FullAccess` policy\n"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(generate_kinesis_create_permission_for_sagemaker_role(sagemaker_role)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build and push docker image\n",
    "\n",
    "The file ./Dockerfile contains all the packages that are installed into the docker. Instead of using the default sagemaker container. We will be using this docker container. \n",
    "\n",
    "If the docker file is not yet present, this takes about 8 minutes to complete. It takes a few seconds on subsequent runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!docker rm -f $(docker ps -a -q);\n",
    "#!docker rmi -f $(docker images -q);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying files from your notebook to existing sagemaker container\n",
      "docker images deepracer-notebook-gpu | sed -n 2,2p\n",
      "Sagemaker docker id : 72c99fd7c362\n",
      "docker run -d -t 72c99fd7c362\n",
      "docker exec -d ece5f02afd3b01b6935ac12682455637eec20f82028e42732c6abf747bf42a19 rm -rf /opt/amazon/markov\n",
      "docker cp ./src/markov ece5f02afd3b01b6935ac12682455637eec20f82028e42732c6abf747bf42a19:/opt/amazon/markov\n",
      "============ Copied Markov scripts to sagemaker docker ============ \n",
      " \n",
      "docker ps -l|sed -n 2,2p\n",
      "docker commit ece5f02afd3b deepracer-notebook-gpu\n",
      "============ Commited all the changes to docker ============ \n",
      " \n",
      "CPU times: user 37.1 ms, sys: 29.8 ms, total: 66.9 ms\n",
      "Wall time: 2.42 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from copy_to_sagemaker_container import get_sagemaker_docker, copy_to_sagemaker_container, get_custom_image_name\n",
    "cpu_or_gpu = 'gpu' if instance_type.startswith('ml.p') else 'cpu'\n",
    "# repo name\n",
    "repository_short_name = job_name_prefix + \"-%s\" % cpu_or_gpu\n",
    "custom_image_name = get_custom_image_name(repository_short_name)\n",
    "\n",
    "try:\n",
    "    print(\"Copying files from your notebook to existing sagemaker container\")\n",
    "    sagemaker_docker_id = get_sagemaker_docker(repository_short_name)\n",
    "    copy_to_sagemaker_container(sagemaker_docker_id, repository_short_name)\n",
    "except Exception as e:\n",
    "    print(\"Creating sagemaker container\")\n",
    "    docker_build_args = {\n",
    "        'CPU_OR_GPU': cpu_or_gpu, \n",
    "        'AWS_REGION': boto3.Session().region_name,\n",
    "    }\n",
    "    custom_image_name = build_and_push_docker_image(repository_short_name, build_args=docker_build_args)\n",
    "    print(\"Using ECR image %s\" % custom_image_name)\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure VPC\n",
    "\n",
    "Since SageMaker and RoboMaker have to communicate with each other over the network, both of these services need to run in VPC mode. This can be done by supplying subnets and security groups to the job launching scripts.  \n",
    "We will check if the deepracer-vpc stack is created and use it if present (This is present if the AWS Deepracer console is used atleast once to create a model). Else we will use the default VPC stack."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using the default VPC stacks\n",
      "Using VPC: vpc-b418facc\n",
      "Using security group: ['sg-fca8088c']\n",
      "Using subnets: ['subnet-57a5de2e', 'subnet-67cc983d', 'subnet-b63155fd']\n"
     ]
    }
   ],
   "source": [
    "ec2 = boto3.client('ec2')\n",
    "\n",
    "#\n",
    "# Check if the user has Deepracer-VPC and use that if its present. This will have all permission.\n",
    "# This VPC will be created when you have used the Deepracer console and created one model atleast\n",
    "# If this is not present. Use the default VPC connnection\n",
    "#\n",
    "deepracer_security_groups = [group[\"GroupId\"] for group in ec2.describe_security_groups()['SecurityGroups']\\\n",
    "                             if group['GroupName'].startswith(\"deepracer-vpc\")]\n",
    "if(deepracer_security_groups):\n",
    "    print(\"Using the DeepRacer VPC stacks\")\n",
    "    deepracer_vpc = [vpc['VpcId'] for vpc in ec2.describe_vpcs()['Vpcs'] \\\n",
    "                     if \"Tags\" in vpc for val in vpc['Tags'] \\\n",
    "                     if val['Value'] == 'deepracer-vpc'][0]\n",
    "    deepracer_subnets = [subnet[\"SubnetId\"] for subnet in ec2.describe_subnets()[\"Subnets\"] \\\n",
    "                         if subnet[\"VpcId\"] == deepracer_vpc]\n",
    "else:\n",
    "    print(\"Using the default VPC stacks\")\n",
    "    deepracer_vpc = [vpc['VpcId'] for vpc in ec2.describe_vpcs()['Vpcs'] if vpc[\"IsDefault\"] == True][0]\n",
    "\n",
    "    deepracer_security_groups = [group[\"GroupId\"] for group in ec2.describe_security_groups()['SecurityGroups'] \\\n",
    "                                 if 'VpcId' in group and group[\"GroupName\"] == \"default\" and group[\"VpcId\"] == deepracer_vpc]\n",
    "\n",
    "    deepracer_subnets = [subnet[\"SubnetId\"] for subnet in ec2.describe_subnets()[\"Subnets\"] \\\n",
    "                         if subnet[\"VpcId\"] == deepracer_vpc and subnet['DefaultForAz']==True]\n",
    "\n",
    "print(\"Using VPC:\", deepracer_vpc)\n",
    "print(\"Using security group:\", deepracer_security_groups)\n",
    "print(\"Using subnets:\", deepracer_subnets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Route Table\n",
    "A SageMaker job running in VPC mode cannot access S3 resourcs. So, we need to create a VPC S3 endpoint to allow S3 access from SageMaker container. To learn more about the VPC mode, please visit [this link.](https://docs.aws.amazon.com/sagemaker/latest/dg/train-vpc.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating \n",
      "Trying to attach S3 endpoints to the following route tables: ['rtb-0d549376']\n",
      "S3 endpoint already exists.\n"
     ]
    }
   ],
   "source": [
    "#TODO: Explain to customer what CREATE_ROUTE_TABLE is doing\n",
    "CREATE_ROUTE_TABLE = True\n",
    "\n",
    "def create_vpc_endpoint_table():\n",
    "    print(\"Creating \")\n",
    "    try:\n",
    "        route_tables = [route_table[\"RouteTableId\"] for route_table in ec2.describe_route_tables()['RouteTables']\\\n",
    "                        if route_table['VpcId'] == deepracer_vpc]\n",
    "    except Exception as e:\n",
    "        if \"UnauthorizedOperation\" in str(e):\n",
    "            display(Markdown(generate_help_for_s3_endpoint_permissions(sagemaker_role)))\n",
    "        else:\n",
    "            display(Markdown(create_s3_endpoint_manually(aws_region, deepracer_vpc)))\n",
    "        raise e\n",
    "\n",
    "    print(\"Trying to attach S3 endpoints to the following route tables:\", route_tables)\n",
    "    \n",
    "    if not route_tables:\n",
    "        raise Exception((\"No route tables were found. Please follow the VPC S3 endpoint creation \"\n",
    "                         \"guide by clicking the above link.\"))\n",
    "    try:\n",
    "        ec2.create_vpc_endpoint(DryRun=False,\n",
    "                                VpcEndpointType=\"Gateway\",\n",
    "                                VpcId=deepracer_vpc,\n",
    "                                ServiceName=\"com.amazonaws.{}.s3\".format(aws_region),\n",
    "                                RouteTableIds=route_tables)\n",
    "        print(\"S3 endpoint created successfully!\")\n",
    "    except Exception as e:\n",
    "        if \"RouteAlreadyExists\" in str(e):\n",
    "            print(\"S3 endpoint already exists.\")\n",
    "        elif \"UnauthorizedOperation\" in str(e):\n",
    "            display(Markdown(generate_help_for_s3_endpoint_permissions(role)))\n",
    "            raise e\n",
    "        else:\n",
    "            display(Markdown(create_s3_endpoint_manually(aws_region, default_vpc)))\n",
    "            raise e\n",
    "\n",
    "if CREATE_ROUTE_TABLE:\n",
    "    create_vpc_endpoint_table()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2: Go from single camera, to stereo camera and LIDAR\n",
    "\n",
    "\n",
    "Note that we have already added all the sensor combinations into the simulation application, and thus you only have to specify the preset that you want to use. Similarly, you can go add entirely new sensors into your simulation application, add them to the car model, and then use them during training. You will have to see what makes sense from a reward fucntion point of view.\n",
    "\n",
    "Possible sensors\n",
    " - Single camera\n",
    " - Stereo camera\n",
    " - Single camera + LIDAR\n",
    " - Stereo Camera + LIDAR\n",
    " \n",
    "When you specify a new sensor configuration this inpacts the state data that will feed into the neural network.\n",
    "\n",
    "![networkinput](img/networkinput.png)\n",
    "\n",
    "In specifying a single image, the process is straight forward, we take 160x120 pixel image and feed that as input.\n",
    "\n",
    "However, when you add a second sensor you need to decide if you are concatenating the inputs before feeding them into the neural network, or if you are creating a second input (double headed so to speak).\n",
    "\n",
    "Note that each input has its own embedder pipeline. For example, converting the color image to 8 bit grayscale. You could add more steps into this embedder process.\n",
    "\n",
    "![inputembedder](img/inputembedder.png)\n",
    "\n",
    "\n",
    "To add configure stereo camera and LIDAR, you need to update\n",
    "- the observations to include the new sensors added in use function in `src/markov/environments/deepracer_racetrack_env.py` and use the below code segment in `class DeepRacerRacetrackEnv(gym.Env) def __init__(self)` to define the observation space with dictionary keys corresponding to the different type of sensors:\n",
    "```\n",
    "self.observation_space = spaces.Dict({\n",
    "        'STEREO_CAMERAS': spaces.Box(low=0, high=255,\n",
    "                                shape=(TRAINING_IMAGE_SIZE[1], TRAINING_IMAGE_SIZE[0], 2),\n",
    "                                dtype=np.uint8),\n",
    "        'LIDAR': spaces.Box(low=0.15, high=1.0, shape=(64,))\n",
    "})\n",
    "```\n",
    "- the input header names for the neural network in `src/markov/presets/preset.py` and the neural network architecture which are described in the next section.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 3: Update the neural network architecture\n",
    "\n",
    "\n",
    "We have partnered with Intel and use [Coach](https://nervanasystems.github.io/coach/) as reinforcement learning framework. Furthermore, we are using Tensorflow as our deep learning framework. \n",
    "\n",
    "The neural network architecture typically includes an input embedder, middleware, and an output head, see descriptions [here](https://nervanasystems.github.io/coach/design/network.html). In this section we are interested in changing the middleware.\n",
    "\n",
    "Here are the Coach layer names\n",
    "\n",
    "![coachnames](img/coachnames.png)\n",
    "\n",
    "### Objective: Please update your middleware\n",
    "- Add another convolutional layer\n",
    "- Add mode convolutions\n",
    "- Test drop out\n",
    "\n",
    "To change the neural network architecture, edit the preset file in `src/markov/presets/`. We provide two example preset files:\n",
    "- The default neural network architecture, `src/markov/presets/default.py`, has an input embedder with a 3 layer Convolutional Neural Network (CNN).\n",
    "- The attention neural network architecture, `src/markov/presets/preset_attention_layer.py`, provides an example on how to use custom layers.\n",
    "\n",
    "Example:\n",
    "\n",
    "```\n",
    "agent_params.network_wrappers['main'].input_embedders_parameters['observation'].scheme = [Conv2d(32, 8, 4),Conv2d(64, 4, 2),Conv2dWithAttention(64, 3, 1, 256)]\n",
    "\n",
    "```\n",
    "\n",
    "In this example we have three convolutional layers on top of each other. The first one has 32 convolutional filters, a kernel size of 8x8, and a stride length of 4.\n",
    "\n",
    "The source code explanation of Conv2d is [here](https://github.com/NervanaSystems/coach/blob/19ad2d60a7022bb5125855c029f27d86aaa46d64/rl_coach/architectures/tensorflow_components/layers.py)\n",
    "\n",
    "Here are the default Coach layer presets\n",
    "\n",
    "![coachlayerpresets](img/coachlayerpresets.png)\n",
    "\n",
    "Be careful though, if you make a mistake here this will probably break your simulation.\n",
    "\n",
    "# Sahika please specify file locations and names and examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "\n",
    "### Configure the  RL algorithm hyperparameters\n",
    "\n",
    "We use Clipped PPO (as provided by [Coach](https://nervanasystems.github.io/coach/components/agents/policy_optimization/cppo.html)) as our reinformcent learning algorithm to train our network. To edit the hyperparameters of the Clipped PPO RL agent, edit the preset file in `src/markov/presets/`. The configurable hyperparameters include learning_rate, neural network structure, batch_size, discount factor. These really are vital to getting good convergence, or none at all."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other interesting manipulations to take note of\n",
    "\n",
    "### Changing the training algorithm\n",
    "To change the RL agent algorithm, refer to the DQN example for AWS DeepRacer. There are multiple files that need to be editted including the preset file in `src/markov/presets/`. \n",
    "\n",
    "### Configure the environment file with custom simulation variables\n",
    "\n",
    "We use an environment file, `src/markov/environment/deepracer_racetrack_env.py`, which contains \"step\" and \"reset\" functions and ability to exchange messages with the Gazebo based AWS RoboMaker simulator. This environment file is shared between Amazon Sagemaker and AWS RoboMaker jobs. The environment variable - `NODE_TYPE` defines which node the code is running on. So, the expressions that have `rospy` dependencies are executed on RoboMaker only. \n",
    "\n",
    "### How to add noise to observations?\n",
    "You can add noise to robocar camera observations by using OpenCV or other image editting packages available in Python. Note that these libraries need to be located in or copied to `build/simapp/bundle/usr/local/lib/python3.5/dist-packages` for the AWS RoboMaker to import them.\n",
    "\n",
    "As an example, we provide a modified environment `src/markov/environment/deepracer_racetrack_env_cv2.py`, to use `cv2` and add Gaussian noise to robocar observations in `def set_next_state():` (see Lines ~241-254)\n",
    "\n",
    "### How to add noise to actions, i.e., steering and speed, for robustness?\n",
    "Adding noise to your actions also increases the robustness of your model to steady-state or tracking errors of the robocar controllers for steering and speed in the real world. Since we use discrete action, we need to add noise to their associated mappings in `class DeepRacerRacetrackCustomActionSpaceEnv(DeepRacerRacetrackEnv):` (see Lines ~575-579)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # replace the environment file with your modified version\n",
    "# !cp src/markov/environments/deepracer_racetrack_env_cv2.py src/markov/environments/deepracer_racetrack_env.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Uncomment the pygmentize code lines to see the code\n",
    "\n",
    "# # Environmental File\n",
    "# !pygmentize src/markov/environments/deepracer_racetrack_env.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Preset File\n",
    "# !pygmentize src/markov/presets/default.py\n",
    "# !pygmentize src/markov/presets/preset_attention_layer.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the reward function\n",
    "\n",
    "To customize reward functions, modify `reward_function` in `src/markov/rewards/`. Note that the parameters exposed to the reward function are coded in the environment file. To create new variables for the reward function, edit the environment file, `src/markov/environment/deepracer_racetrack_env.py`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Reward function\n",
    "# !pygmentize src/markov/rewards/default.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure the action space\n",
    "\n",
    "Action space and steering angles can be changed by modifying `src/markov/actions/.json` file. The default action space for our RL agent is discrete, therefore, the number of actions correspond to the number of output nodes of the policy network.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Action space\n",
    "# !pygmentize src/markov/actions/model_metadata_10_state.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Copy custom files to S3 bucket so that Amazon SageMaker and AWS RoboMaker can pick them up\n",
    "\n",
    "**Very important**, remember to copy the edited files from ./src/ back into S3 where SageMaker and RoboMaker will pick them up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-west-2-828753602420/deepracer-notebook-sagemaker-191111-193206\n",
      "upload: src/markov/environments/deepracer_racetrack_env.py to s3://sagemaker-us-west-2-828753602420/deepracer-notebook-sagemaker-191111-193206/environments/deepracer_racetrack_env.py\n",
      "upload: src/markov/rewards/default.py to s3://sagemaker-us-west-2-828753602420/deepracer-notebook-sagemaker-191111-193206/rewards/reward_function.py\n",
      "upload: src/markov/actions/model_metadata_10_state.json to s3://sagemaker-us-west-2-828753602420/deepracer-notebook-sagemaker-191111-193206/model_metadata.json\n",
      "upload: src/markov/presets/default.py to s3://sagemaker-us-west-2-828753602420/deepracer-notebook-sagemaker-191111-193206/presets/preset.py\n"
     ]
    }
   ],
   "source": [
    "s3_location = \"s3://%s/%s\" % (s3_bucket, s3_prefix)\n",
    "print(s3_location)\n",
    "\n",
    "# Clean up the previously uploaded files\n",
    "!aws s3 rm --recursive {s3_location}\n",
    "\n",
    "# Make any changes to the environment and preset files below and upload these files\n",
    "!aws s3 cp src/markov/environments/deepracer_racetrack_env.py {s3_location}/environments/deepracer_racetrack_env.py\n",
    "\n",
    "!aws s3 cp src/markov/rewards/default.py {s3_location}/rewards/reward_function.py\n",
    "\n",
    "!aws s3 cp src/markov/actions/model_metadata_10_state.json {s3_location}/model_metadata.json\n",
    "\n",
    "!aws s3 cp src/markov/presets/default.py {s3_location}/presets/preset.py\n",
    "#!aws s3 cp src/markov/presets/preset_attention_layer.py {s3_location}/presets/preset.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train the RL model using the Python SDK Script mode\n",
    "\n",
    "Next, we define the following algorithm metrics that we want to capture from cloudwatch logs to monitor the training progress. These are algorithm specific parameters and might change for different algorithm. We use [Clipped PPO](https://coach.nervanasys.com/algorithms/policy_optimization/cppo/index.html) for this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric_definitions = [\n",
    "    # Training> Name=main_level/agent, Worker=0, Episode=19, Total reward=-102.88, Steps=19019, Training iteration=1\n",
    "    {'Name': 'reward-training',\n",
    "     'Regex': '^Training>.*Total reward=(.*?),'},\n",
    "    \n",
    "    # Policy training> Surrogate loss=-0.32664725184440613, KL divergence=7.255815035023261e-06, Entropy=2.83156156539917, training epoch=0, learning_rate=0.00025\n",
    "    {'Name': 'ppo-surrogate-loss',\n",
    "     'Regex': '^Policy training>.*Surrogate loss=(.*?),'},\n",
    "     {'Name': 'ppo-entropy',\n",
    "     'Regex': '^Policy training>.*Entropy=(.*?),'},\n",
    "   \n",
    "    # Testing> Name=main_level/agent, Worker=0, Episode=19, Total reward=1359.12, Steps=20015, Training iteration=2\n",
    "    {'Name': 'reward-testing',\n",
    "     'Regex': '^Testing>.*Total reward=(.*?),'},\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the RLEstimator for training RL jobs.\n",
    "\n",
    "1. Specify the source directory which has the environment file, preset and training code.\n",
    "2. Specify the entry point as the training code\n",
    "3. Specify the choice of RL toolkit and framework. This automatically resolves to the ECR path for the RL Container.\n",
    "4. Define the training parameters such as the instance count, instance type, job name, s3_bucket and s3_prefix for storing model checkpoints and metadata. **Only 1 training instance is supported for now.**\n",
    "4. Set the RLCOACH_PRESET as \"deepracer\" for this example.\n",
    "5. Define the metrics definitions that you are interested in capturing in your logs. These can also be visualized in CloudWatch and SageMaker Notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training job: deepracer-notebook-2019-11-11-19-32-12-595\n"
     ]
    }
   ],
   "source": [
    "estimator = RLEstimator(entry_point=\"training_worker.py\",\n",
    "                        source_dir='src',\n",
    "                        image_name=custom_image_name,\n",
    "                        dependencies=[\"common/\"],\n",
    "                        role=sagemaker_role,\n",
    "                        train_instance_type=instance_type,\n",
    "                        train_instance_count=1,\n",
    "                        output_path=s3_output_path,\n",
    "                        base_job_name=job_name_prefix,\n",
    "                        metric_definitions=metric_definitions,\n",
    "                        train_max_run=job_duration_in_seconds,\n",
    "                        hyperparameters={\n",
    "                            \"s3_bucket\": s3_bucket,\n",
    "                            \"s3_prefix\": s3_prefix,\n",
    "                            \"aws_region\": aws_region,\n",
    "                            \"preset_s3_key\": \"%s/presets/preset.py\"% s3_prefix,\n",
    "                            \"model_metadata_s3_key\": \"%s/model_metadata.json\" % s3_prefix,\n",
    "                            \"environment_s3_key\": \"%s/environments/deepracer_racetrack_env.py\" % s3_prefix,\n",
    "                        },\n",
    "                        subnets=deepracer_subnets,\n",
    "                        security_group_ids=deepracer_security_groups,\n",
    "                    )\n",
    "\n",
    "estimator.fit(wait=False)\n",
    "job_name = estimator.latest_training_job.job_name\n",
    "print(\"Training job: %s\" % job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Kinesis video stream (optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"StreamARN\": \"arn:aws:kinesisvideo:us-west-2:828753602420:stream/dr-kvs-deepracer-notebook-2019-11-11-19-32-12-595/1573500733492\"\n",
      "}\n",
      "Created kinesis video stream dr-kvs-deepracer-notebook-2019-11-11-19-32-12-595\n"
     ]
    }
   ],
   "source": [
    "kvs_stream_name = \"dr-kvs-{}\".format(job_name)\n",
    "\n",
    "!aws --region {aws_region} kinesisvideo create-stream --stream-name {kvs_stream_name} --media-type video/h264 --data-retention-in-hours 24\n",
    "print (\"Created kinesis video stream {}\".format(kvs_stream_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start the Robomaker job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "robomaker = boto3.client(\"robomaker\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Simulation Application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "robomaker_s3_key = 'robomaker/simulation_ws.tar.gz'\n",
    "robomaker_source = {'s3Bucket': s3_bucket,\n",
    "                    's3Key': robomaker_s3_key,\n",
    "                    'architecture': \"X86_64\"}\n",
    "simulation_software_suite={'name': 'Gazebo',\n",
    "                           'version': '7'}\n",
    "robot_software_suite={'name': 'ROS',\n",
    "                      'version': 'Kinetic'}\n",
    "rendering_engine={'name': 'OGRE',\n",
    "                  'version': '1.x'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload your customized simulation application to your s3 bucket\n",
    "\n",
    "The AWS DeepRacer bundle to be used by the AWS RoboMaker service is under `build/output.tar.gz`. Next, we need to upload the bundle to our S3 bucket and create an AWS RoboMaker Simulation Application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "upload: build/output.tar.gz to s3://sagemaker-us-west-2-828753602420/robomaker/simulation_ws.tar.gz\n"
     ]
    }
   ],
   "source": [
    "# Use your custom simApp\n",
    "!aws s3 cp ./build/output.tar.gz s3://{s3_bucket}/{robomaker_s3_key}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create arn for the AWS RoboMaker simulation application"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "deepracer-notebook-application191111-193223\n",
      "Created a new simulation app with ARN: arn:aws:robomaker:us-west-2:828753602420:simulation-application/deepracer-notebook-application191111-193223/1573500743944\n"
     ]
    }
   ],
   "source": [
    "# create arn for AWS RoboMaker\n",
    "app_name = \"deepracer-notebook-application\" + strftime(\"%y%m%d-%H%M%S\", gmtime())\n",
    "\n",
    "print(app_name)\n",
    "try:\n",
    "    response = robomaker.create_simulation_application(name=app_name,\n",
    "                                                       sources=[robomaker_source],\n",
    "                                                       simulationSoftwareSuite=simulation_software_suite,\n",
    "                                                       robotSoftwareSuite=robot_software_suite,\n",
    "                                                       renderingEngine=rendering_engine)\n",
    "    simulation_app_arn = response[\"arn\"]\n",
    "    print(\"Created a new simulation app with ARN:\", simulation_app_arn)\n",
    "except Exception as e:\n",
    "    if \"AccessDeniedException\" in str(e):\n",
    "        display(Markdown(generate_help_for_robomaker_all_permissions(role)))\n",
    "        raise e\n",
    "    else:\n",
    "        raise e"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 4: Using multiple rollouts during training\n",
    "\n",
    "\n",
    "We create AWS RoboMaker simulation jobs that simulates the environment and shares this data with SageMaker for training. Each roll-out uses a central model to independently collect experience in the form of episodes, where each episode consist of (state, action, next state, reward) tuples\n",
    "\n",
    "![distribrollout](img/fourrollouts.png)\n",
    "\n",
    "\n",
    "We use horizontal scaling where the neural network model files are synchronized between the Amazon Sagemaker training job and AWS RoboMaker simulation workers. Model sync behavior is coded in src/markov/training_worker.py \n",
    "\n",
    "\n",
    "![rollouts](./SageMaker_RoboMaker.png)\n",
    "\n",
    "\n",
    "```\n",
    "if graph_manager.agent_params.algorithm.distributed_coach_synchronization_type == \n",
    "            DistributedCoachSynchronizationType.SYNC:\n",
    "    graph_manager.save_checkpoint()\n",
    "else:\n",
    "    graph_manager.occasionally_save_checkpoint()\n",
    "```\n",
    "To enable sync models to the s3 bucket for multiple rollouts, set the parameter in src/markov/presets/preset.py\n",
    "\n",
    "```\n",
    "agent_params.algorithm.distributed_coach_synchronization_type = DistributedCoachSynchronizationType.SYNC\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "### Objective: Please increase number of workers to 2\n",
    "\n",
    "**Specify the number of roll-out workers** using the ***num_simulation_workers*** parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WORLD NAME =  reinvent_base\n"
     ]
    }
   ],
   "source": [
    "available_tracks = [\"reinvent_base\", # 0\n",
    "                    \"AWS_track\", # 1\n",
    "                    \"Tokyo_Training_track\", #2\n",
    "                    \"Virtual_May19_Train_track\", #3 (london)\n",
    "                    \"reInvent2018_36inch\", #4\n",
    "                    \"reInvent2018_mirror\", #5\n",
    "                    \"reInvent2019_track\"] #6\n",
    "\n",
    "training_tracks_indices = [6, 6, 5, 5]\n",
    "\n",
    "num_simulation_workers = len(training_tracks_indices)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created the following jobs:\n",
      "Job ARN arn:aws:robomaker:us-west-2:828753602420:simulation-job/sim-tlx5szzdd48g\n"
     ]
    }
   ],
   "source": [
    "\n",
    "envriron_vars = {\n",
    "    \"WORLD_NAME\": \"reinvent_base\",\n",
    "    \"KINESIS_VIDEO_STREAM_NAME\": \"SilverstoneStream\",\n",
    "    \"PYTHONPATHS\":\"wohooo this worked\",\n",
    "    \"ALTERNATE_DRIVING_DIRECTION\": \"false\",\n",
    "    \"SAGEMAKER_SHARED_S3_BUCKET\": s3_bucket,\n",
    "    \"SAGEMAKER_SHARED_S3_PREFIX\": s3_prefix,\n",
    "    \"TRAINING_JOB_ARN\": job_name,\n",
    "    \"APP_REGION\": aws_region,\n",
    "    \"METRIC_NAME\": \"TrainingRewardScore\",\n",
    "    \"METRIC_NAMESPACE\": \"AWSDeepRacer\",\n",
    "    \"REWARD_FILE_S3_KEY\": \"%s/rewards/reward_function.py\" % s3_prefix,\n",
    "    \"MODEL_METADATA_FILE_S3_KEY\": \"%s/model_metadata.json\" % s3_prefix,\n",
    "    \"METRICS_S3_BUCKET\": s3_bucket,\n",
    "    \"METRICS_S3_OBJECT_KEY\": s3_prefix + \"/training_metrics.json\",\n",
    "    \"TARGET_REWARD_SCORE\": \"None\",\n",
    "    \"ROBOMAKER_SIMULATION_JOB_ACCOUNT_ID\": account_id\n",
    "}\n",
    "\n",
    "simulation_application = {\"application\":simulation_app_arn,\n",
    "                          \"launchConfig\": {\"packageName\": \"deepracer_simulation_environment\",\n",
    "                                           \"launchFile\": \"distributed_training.launch\",\n",
    "                                           \"environmentVariables\": envriron_vars}\n",
    "                         }\n",
    "\n",
    "vpcConfig = {\"subnets\": deepracer_subnets,\n",
    "             \"securityGroups\": deepracer_security_groups,\n",
    "             \"assignPublicIp\": True}\n",
    "\n",
    "responses = []\n",
    "for job_no in range(num_simulation_workers):\n",
    "    simulation_application[\"launchConfig\"][\"environmentVariables\"][\"WORLD_NAME\"] = available_tracks[training_tracks_indices[job_no]]\n",
    "    client_request_token = strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime())\n",
    "    response =  robomaker.create_simulation_job(iamRole=sagemaker_role,\n",
    "                                            clientRequestToken=client_request_token,\n",
    "                                            maxJobDurationInSeconds=job_duration_in_seconds,\n",
    "                                            failureBehavior=\"Fail\",\n",
    "                                            simulationApplications=[simulation_application],\n",
    "                                            vpcConfig=vpcConfig\n",
    "                                            )\n",
    "    print(response)\n",
    "    responses.append(response)\n",
    "\n",
    "\n",
    "print(\"Created the following jobs:\")\n",
    "job_arns = [response[\"arn\"] for response in responses]\n",
    "for response in responses:\n",
    "    print(\"Job ARN\", response[\"arn\"]) \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy the below output for log analysis and to determine the top performing models on the real world track."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"S3_MODEL_SAVE_PREFIX = '%s'\" %experiment_name)\n",
    "\n",
    "arn_str = list()\n",
    "for response in responses:\n",
    "    arn_str.append(response[\"arn\"].split('/')[-1])\n",
    "print('stream_name_list =' + str(arn_str))\n",
    "print(\"sagemaker_simapp_name_list = ['\" + str(job_name) + \"']\")\n",
    "\n",
    "print(\"s3_bucket = '%s'\" %s3_bucket)\n",
    "print(\"s3_prefix = '%s'\" %s3_prefix)\n",
    "print('simulation_app_arn = \"%s\"' %simulation_app_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the simulations in RoboMaker\n",
    "You can visit the RoboMaker console to visualize the simulations or run the following cell to generate the hyperlinks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "> Click on the following links for visualization of simulation jobs on RoboMaker Console\n",
       "- [Simulation 1](https://us-west-2.console.aws.amazon.com/robomaker/home?region=us-west-2#simulationJobs/sim-tlx5szzdd48g)  \n",
       "\n",
       "You can click on Gazebo after you open the above link to start the simulator."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Markdown(generate_robomaker_links(job_arns, aws_region)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run evaluation in parallel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sys.path.append(\"./src\")\n",
    "\n",
    "num_simulation_workers = 1\n",
    "\n",
    "\n",
    "envriron_vars = {\n",
    "    \"WORLD_NAME\": \"reInvent2019_track\",\n",
    "    \"KINESIS_VIDEO_STREAM_NAME\": \"SilverstoneStream\",\n",
    "    \"SAGEMAKER_SHARED_S3_BUCKET\": s3_bucket,\n",
    "    \"SAGEMAKER_SHARED_S3_PREFIX\": s3_prefix,\n",
    "    \"MODEL_S3_BUCKET\": s3_bucket,\n",
    "    \"PYTHONPATHS\":\"wohooo this worked\",\n",
    "    \"MODEL_S3_PREFIX\": s3_prefix,\n",
    "    \"ALTERNATE_DRIVING_DIRECTION\": \"false\",\n",
    "    \"APP_REGION\": aws_region,\n",
    "    \"MODEL_METADATA_FILE_S3_KEY\": \"%s/model_metadata.json\" % s3_prefix,\n",
    "    \"METRICS_S3_BUCKET\": s3_bucket,\n",
    "    \"METRICS_S3_OBJECT_KEY\": s3_prefix + \"/evaluation_metrics.json\",\n",
    "    \"NUMBER_OF_TRIALS\": \"5\", # Doesnt matter\n",
    "    \"ROBOMAKER_SIMULATION_JOB_ACCOUNT_ID\": account_id\n",
    "}\n",
    "\n",
    "simulation_application = {\n",
    "    \"application\":simulation_app_arn,\n",
    "    \"launchConfig\": {\n",
    "         \"packageName\": \"deepracer_simulation_environment\",\n",
    "         \"launchFile\": \"evaluation.launch\",\n",
    "         \"environmentVariables\": envriron_vars\n",
    "    }\n",
    "}\n",
    "                            \n",
    "vpcConfig = {\"subnets\": deepracer_subnets,\n",
    "             \"securityGroups\": deepracer_security_groups,\n",
    "             \"assignPublicIp\": True}\n",
    "\n",
    "responses_eval = []\n",
    "for job_no in range(num_simulation_workers):\n",
    "    response =  robomaker.create_simulation_job(clientRequestToken=strftime(\"%Y-%m-%d-%H-%M-%S\", gmtime()),\n",
    "                                                outputLocation={ \n",
    "                                                  \"s3Bucket\": s3_bucket,\n",
    "                                                  \"s3Prefix\": s3_prefix\n",
    "                                                },\n",
    "                                                maxJobDurationInSeconds=job_duration_in_seconds*3,\n",
    "                                                iamRole=sagemaker_role,\n",
    "                                                failureBehavior=\"Continue\",\n",
    "                                                simulationApplications=[simulation_application],\n",
    "                                                vpcConfig=vpcConfig)\n",
    "    responses_eval.append(response)\n",
    "\n",
    "print('~~~ EVAL ~~~\\n')\n",
    "arn_str = list()\n",
    "for response in responses_eval:\n",
    "    arn_str.append(response[\"arn\"].split('/')[-1])\n",
    "print('eval_stream_name_list =' + str(arn_str))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 5: How to analyze your models and determine top 5 models to test on the real track"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Go to evaluation notebook?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean up RoboMaker and SageMaker training job\n",
    "\n",
    "Execute the cells below if you want to kill RoboMaker and SageMaker job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # Cancelling robomaker job\n",
    "# for job_arn in job_arns:\n",
    "#     robomaker.cancel_simulation_job(job=job_arn)\n",
    "\n",
    "# # Stopping sagemaker training job\n",
    "# sage_session.sagemaker_client.stop_training_job(TrainingJobName=estimator._current_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean Up Simulation Application Resource"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# robomaker.delete_simulation_application(application=simulation_app_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean your S3 bucket (Uncomment the awscli commands if you want to do it)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Uncomment if you only want to clean the s3 bucket\n",
    "# sagemaker_s3_folder = \"s3://{}/{}\".format(s3_bucket, s3_prefix)\n",
    "# !aws s3 rm --recursive {sagemaker_s3_folder}\n",
    "\n",
    "# robomaker_s3_folder = \"s3://{}/{}\".format(s3_bucket, job_name)\n",
    "# !aws s3 rm --recursive {robomaker_s3_folder}\n",
    "\n",
    "# robomaker_sim_app = \"s3://{}/{}\".format(s3_bucket, 'robomaker')\n",
    "# !aws s3 rm --recursive {robomaker_sim_app}\n",
    "\n",
    "# model_output = \"s3://{}/{}\".format(s3_bucket, s3_bucket)\n",
    "# !aws s3 rm --recursive {model_output}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Clean the docker images\n",
    "Remove this only when you want to completely remove the docker or clean up the space of the sagemaker instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !docker rmi -f $(docker images -q)"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "conda_amazonei_tensorflow_p36",
   "language": "python",
   "name": "conda_amazonei_tensorflow_p36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}