PyTorch/NoteBooks/GettingStarted/GettingStarted.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": "# Getting started with Clara Train SDK V4.0 PyTorch using MONAI \nClara Train SDK simply allows researcher to train AI models using configuration files. \nIt is simple to use, modular and flexible. Allowing researchers to focus on innovation, \nwhile leaving acceleration and performance issue for NVIDIA\u0027s engineers. \n\nClara Train SDK consists of different modules as shown below \n\u003cbr\u003e\u003cimg src\u003d\"screenShots/TrainBlock.png\" alt\u003d\"Drawing\" style\u003d\"height: 600px;\"/\u003e\u003cbr\u003e\n   \nBy the end of this notebook you will:\n1. Understand components of [Medical Model ARchive (MMAR)](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v4.0/nvmidl/mmar.html)\n2. Know how to configure train config json to train a CNN\n3. Train a CNN with single and multiple GPUs\n4. Fine tune a model\n5. Export a model \n6. Perform inference on testing dataset \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Prerequisites\n- Nvidia GPU with 8Gb of memory   \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Resources\nYou could watch the free GTC 2021 talks covering Clara Train SDK\n- [Clara Train 4.0 - 101 Getting Started [SE2688]](https://gtc21.event.nvidia.com/media/Clara%20Train%204.0%20-%20101%20Getting%20Started%20%5BSE2688%5D/1_0qgfrql2)\n- [Clara Train 4.0 - 201 Federated Learning [SE3208]](https://gtc21.event.nvidia.com/media/Clara%20Train%204.0%20-%20201%20Federated%20Learning%20%5BSE3208%5D/1_m48t6b3y)\n- [What’s New in Clara Train 4.0 [D3114]](https://gtc21.event.nvidia.com/media/What%E2%80%99s%20New%20in%20Clara%20Train%204.0%20%5BD3114%5D/1_umvjidt2)\n- [Take Medical AI from Concept to Production using Clara Imaging [S32482]](https://gtc21.event.nvidia.com/media/Take%20Medical%20AI%20from%20Concept%20to%20Production%20using%20Clara%20Imaging%20%20%5BS32482%5D/1_6bvnvyg7)\n- [Federated Learning for Medical AI [S32530]](https://gtc21.event.nvidia.com/media/Federated%20Learning%20for%20Medical%20AI%20%5BS32530%5D/1_z26u15uk)\n- [Get Started Now on Medical Imaging AI with Clara Train on Google Cloud Platform [S32518]](https://gtc21.event.nvidia.com/media/Get%20Started%20Now%20on%20Medical%20Imaging%20AI%20with%20Clara%20Train%20on%20Google%20Cloud%20Platform%20%5BS32518%5D/1_2yjdekmi)\n- [Automate 3D Medical Imaging Segmentation with AutoML and Neural Architecture Search [S32083]](https://gtc21.event.nvidia.com/media/Automate%203D%20Medical%20Imaging%20Segmentation%20with%20AutoML%20and%20Neural%20Architecture%20Search%20%5BS32083%5D/1_r5swh2jn)\n- [A Platform for Rapid Development and Clinical Translation of ML Models for Applications in Radiology at UCSF [S31619]](https://gtc21.event.nvidia.com/media/A%20Platform%20for%20Rapid%20Development%20and%20Clinical%20Translation%20of%20ML%20Models%20for%20Applications%20in%20Radiology%20at%20UCSF%20%5BS31619%5D/1_oz8qop5a)\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# 1. Background\n\nClara Train is built using a component-based architecture with using components from [MONAI](https://monai.io/) :\nMONAI’s [training workflows](https://docs.monai.io/en/latest/highlights.html#workflows) \nare based off of [PyTorch Ignite’s engine](https://pytorch.org/ignite/engine.html). \nBelow is a list of different components used:\n- Training Data Pipeline\n- Validation Data Pipeline\n- [Applications](https://docs.monai.io/en/latest/apps.html)\n- [Transforms](https://docs.monai.io/en/latest/transforms.html)\n- [Data](https://docs.monai.io/en/latest/data.html)\n- [Engines](https://docs.monai.io/en/latest/engines.html)\n- [Inference methods](https://docs.monai.io/en/latest/inferers.html)\n- [Event handlers](https://docs.monai.io/en/latest/handlers.html)\n- [Network architectures](https://docs.monai.io/en/latest/networks.html)\n- [Loss functions](https://docs.monai.io/en/latest/losses.html)\n- [Optimizers](https://docs.monai.io/en/latest/optimizers.html)\n- [Metrics](https://docs.monai.io/en/latest/metrics.html)\n- [Visualizations](https://docs.monai.io/en/latest/visualize.html)\n- [Utilities](https://docs.monai.io/en/latest/utils.html)   \n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# Lets get started\nBefore we get started lets check that we have an NVIDIA GPU available in the docker by running the cell below",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "# following command should show all gpus available \n!nvidia-smi",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Next cell defines functions that we will use throughout the notebook",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "MMAR_ROOT\u003d\"/claraDevDay/GettingStarted/\"\nprint (\"setting MMAR_ROOT\u003d\",MMAR_ROOT)\n%ls $MMAR_ROOT\n\n!chmod 777 $MMAR_ROOT/commands/*\ndef printFile(filePath,lnSt,lnEnd):\n    print (\"showing \",str(lnEnd-lnSt),\" lines from file \",filePath, \"starting at line\",str(lnSt))\n    !\u003c $filePath head -n \"$lnEnd\" | tail -n +\"$lnSt\"\n ",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# 2. Medical Model ARchive (MMAR)\nClara Train SDK uses the [Medical Model ARchive (MMAR)](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v4.0/nvmidl/mmar.html). \nThe MMAR defines a standard structure for organizing all artifacts produced during the model development life cycle. \nClara Train SDK simple basic idea is to train using config file\n ",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "**We recommend opening [config_train_Unet.json](config/config_train_Unet.json) and configuring your screen as shown below**\n\u003cbr\u003e\u003cimg src\u003d\"screenShots/MMAR.png\" alt\u003d\"Drawing\" style\u003d\"height: 400px;\"/\u003e\u003cbr\u003e\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "You can download sample models for different problems from [NGC](https://ngc.nvidia.com/catalog/models?orderBy\u003dmodifiedDESC\u0026pageNumber\u003d0\u0026query\u003dclara\u0026quickFilter\u003d\u0026filters\u003d) \u003cbr\u003e \nAll MMAR follow the structure provided in this Notebook. if you navigate to the parent folder structure it should contain the following subdirectories\n```\n./GettingStarted \n├── commands\n├── config\n├── docs\n├── eval\n├── models\n└── resources\n```\n\n* `commands` contains a number of ready-to-run scripts for:\n    - training\n    - training with multiple GPU\n    - fine tune\n    - fine tune with multiple GPU\n    - validation\n    - validation with multiple GPU\n    - inference (testing)\n    - exporting models in TensorRT Inference Server format\n* `config` contains configuration files (in JSON format) for eac training, \nvalidation, and deployment for [AI-assisted annotation](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v4.0/aiaa/index.html) \n(_Note:_ these configuration files are used in the scripts under the `commands` folder)\n* `docs` contains local documentation for the model, but for a more complete view it is recommended you visit the NGC model page\n* `eval` is used as the output directory for model evaluation (by default)\n* `models` is where the PyTorch checkpoint model is stored, and the corresponding graph definition files.\n* `resources` currently contains the logger configuration in `log.config` file\n\nSome of the most important files you will need to understand to configure and use Clara Train SDK are\n\n1. `environment.json` which has important common parameters to set the path for \n    * `DATA_ROOT` is the root folder where the data with which we would like to train, validate, or test resides in\n    * `DATASET_JSON` expects the path to a JSON-formatted file \n    * `MMAR_CKPT_DIR` the path to the where the PyTorch checkpoint files reside\n    * `MMAR_EVAL_OUTPUT_PATH` the path to output evaluation metrics for the neural network during training, validation, and inference\n    * `PROCESSING_TASK` the type of processing task the neural net is intended to perform (currently limited to `annotation`, `segmentation`, `classification`)\n    * `PRETRAIN_WEIGHTS_FILE` (_optional_) \tdetermines the location of the pre-trained weights file; if the file does not exist and is needed, \n    the training program will download it from predefined URL from the web\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "printFile(MMAR_ROOT+\"/config/environment.json\",0,30)\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# 3. Config.json Main Concepts \n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "`config_train.json` contains all the parameters necessary to define the neural net, \nhow is it trained (training hyper-parameters, loss, etc.), \npre- and post-transformation functions necessary to modify and/or augment the data before input to the neural net, etc. \nThe complete documentation on the training configuration is laid out \n[here](https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v4.0/nvmidl/appendix/configuration.html#training-configuration).\nConfiguration file defines all training related configurations. \nThis is were most the researcher would spent most of his time.\n\nLets take some time to examine each part of it.  \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 3.1. Global configurations ",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "pycharm": {}
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "    \"model\": {\n",
            "      \"name\": \"Unet\",\n",
            "      \"args\": {\n",
            "        \"num_classes\": 6,\n",
            "        \"nf_enc\":\"32,64,64,64\",\n",
            "        \"nf_dec\":\"64,64,64,64,64,32,32\"\n",
            "      }\n",
            "    },\n",
            "    \"pre_transforms\": [\n",
            "      {\n",
            "        \"name\": \"LoadNifti\",\n"
          ]
        }
      ],
      "source": "confFile\u003dMMAR_ROOT+\"/config/config_train_Unet.json\"\nprintFile(confFile,0,9)\n"
    },
    {
      "cell_type": "markdown",
      "source": "## 3.2. Training configurations section \nThis section includes:\n1. Loss functions:\n```\n    \"loss\": {\n      \"name\": \"DiceLoss\",\n      \"args\":{\n        \"to_onehot_y\": true,\n        \"softmax\": true\n      }\n    },\n```",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "2. Optimizer\n```\n    \"optimizer\": {\n      \"name\": \"Adam\",\n      \"args\": {\n        \"lr\": \"{learning_rate}\"\n      }\n    },\n```",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n3. Learning rate scheduler\n```\n    \"lr_scheduler\": {\n      \"name\": \"StepLR\",\n      \"args\": {\n        \"step_size\": 5000,\n        \"gamma\": 0.1\n      }\n    },\n```",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n4. Network architecture\n```\n    \"model\": {\n      \"name\": \"UNet\",\n      \"args\": {\n        \"dimensions\": 3,\n        \"in_channels\": 1,\n        \"out_channels\": 2,\n        \"channels\": [16, 32, 64, 128, 256],\n        \"strides\": [2, 2, 2, 2],\n        \"num_res_units\": 2,\n        \"norm\": \"batch\"\n      }\n    },\n```",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n5. Pre-transforms\n    1. Loading transformations\n    ```\n        {\n        \"name\": \"LoadImaged\",\n        \"args\": {\n          \"keys\": [\"image\", \"label\"]\n        }\n      },\n    ```\n    2. Ensure channel first Transformation\n    ```\n      {\n        \"name\": \"EnsureChannelFirstd\",\n        \"args\": {\n          \"keys\": [\"image\", \"label\"]\n        }\n      },    \n    ```\n    2. Resample Transformation\n    ```\n      {\n        \"name\": \"Spacingd\",\n        \"args\": {\n            \"keys\": [\"image\", \"label\"],\n            \"pixdim\": [1.0, 1.0, 1.0],\n            \"mode\":[\"bilinear\", \"nearest\"]\n        }\n      },    \n    ```\n    5. Intensity Transforms\n    ```\n      {\n        \"name\": \"ScaleIntensityRanged\",\n        \"args\": {\n          \"keys\": \"image\",\n          \"a_min\": -57,\n          \"a_max\": 164,\n          \"b_min\": 0.0,\n          \"b_max\": 1.0,\n          \"clip\": true\n        }\n      },    \n    ```\n    3. Cropping transformations\n    ```\n      {\n        \"name\": \"RandCropByPosNegLabeld\",\n        \"args\": {\n          \"keys\": [\"image\", \"label\"],\n          \"label_key\": \"label\",\n          \"spatial_size\": [96, 96, 96],\n          \"pos\": 1,\n          \"neg\": 1,\n          \"num_samples\": 4,\n          \"image_key\": \"image\",\n          \"image_threshold\": 0\n        }\n      },    \n    ```\n    4. Deformable transformations\n    ```\n    ```\n    6. Augmentation Transforms\n    ```\n      {\n        \"name\": \"RandShiftIntensityd\",\n        \"args\": {\n          \"keys\": \"image\",\n          \"offsets\": 0.1,\n          \"prob\": 0.5\n        }\n      },    \n    ```\n    7. Special transforms \n    ```\n      {\n        \"name\": \"ToTensord\",\n        \"args\": {\n          \"keys\": [\"image\", \"label\"]\n        }\n      }    \n    ```",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n6. DataSet to use \n```\n    \"dataset\": {\n      \"name\": \"CacheDataset\",\n      \"data_list_file_path\": \"{DATASET_JSON}\",\n      \"data_file_base_dir\": \"{DATA_ROOT}\",\n      \"data_list_key\": \"training\",\n      \"args\": {\n        \"cache_num\": 4,\n        \"cache_rate\": 1.0,\n        \"num_workers\": 2\n      }\n    },\n```",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n7. DataLoader\n```\n    \"dataloader\": {\n      \"name\": \"DataLoader\",\n      \"args\": {\n        \"batch_size\": 2,\n        \"shuffle\": true,\n        \"num_workers\": 4\n      }\n    },\n```",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n8. inferer\n```\n    \"inferer\": {\n      \"name\": \"SimpleInferer\"\n    },\n```",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n9. Handlers\nThere can be may handlers as:\n1. CheckpointLoader\n2. LrScheduleHandler\n3. ValidationHandler\n4. CheckpointSaver\n5. StatsHandler\n6. TensorBoardStatsHandler\n```\n    \"handlers\": [\n      {\n        \"name\": \"CheckpointLoader\",\n        \"disabled\": \"{dont_load_ckpt_model}\",\n        \"args\": {\n          \"load_path\": \"{MMAR_CKPT}\",\n          \"load_dict\": [\"model\"]\n        }\n      },\n      {\n        \"name\": \"LrScheduleHandler\",\n        \"args\": {\n          \"print_lr\": true\n        }\n      },\n      {\n        \"name\": \"ValidationHandler\",\n        \"args\": {\n          \"interval\": \"{num_interval_per_valid}\",\n          \"epoch_level\": true\n        }\n      },\n      {\n        \"name\": \"CheckpointSaver\",\n        \"rank\": 0,\n        \"args\": {\n          \"save_dir\": \"{MMAR_CKPT_DIR}\",\n          \"save_dict\": [\"model\", \"optimizer\", \"lr_scheduler\"],\n          \"save_final\": true,\n          \"save_interval\": 5\n        }\n      },\n      {\n        \"name\": \"StatsHandler\",\n        \"rank\": 0,\n        \"args\": {\n          \"tag_name\": \"train_loss\",\n          \"output_transform\": \"lambda x: x[\u0027loss\u0027]\"\n        }\n      },\n      {\n        \"name\": \"TensorBoardStatsHandler\",\n        \"rank\": 0,\n        \"args\": {\n          \"log_dir\": \"{MMAR_CKPT_DIR}\",\n          \"tag_name\": \"train_loss\",\n          \"output_transform\": \"lambda x: x[\u0027loss\u0027]\"\n        }\n      }\n    ],\n```",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n10. Post transforms\n    1. Activations \n    2. Change to oneHot \n```\n```",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "11. Metric\n```\n```",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n## 3.3. Validation config \nThis contains sub sections very similar to the ones in the training section including:\n1. Metric \n2. pre-transforms. Since these transforms are usually a subset from the pre-transforms in the training section, \nwe can use the alias to point to these transforms by name as ` \"ref\": \"LoadNifti\"`. \nIn case we use 2 transforms with the same name as `ScaleByResolution` \nwe can give each an alias to refer to as `\"name\": \"ScaleByResolution#ScaleImg\"` \nthen refer to it in the validation section as `ScaleImg` \n3. Image pipeline\n4. Inference",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "printFile(confFile,214,250)\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# 4. Training your first Network",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 4.1 Start TensorBoard \nBefore we start training or while the network is training, \nyou can monitor its accuracy using tensorboard in side jupyter lab as shown below \n \u003cbr\u003e\u003cimg src\u003d\"screenShots/TensorBoard.png\" alt\u003d\"Drawing\" style\u003d\"height: 300px;\"/\u003e\u003cbr\u003e\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 4.2 Training script \nWe have renamed `train.sh` to `train_W_Config` as we modified it to accept parameters with the config to use\n\nLet\u0027s take a look at `train_W_Config.sh` by executing the following cell.",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "printFile(MMAR_ROOT+\"/commands/train_W_Config.sh\",0,30)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 4.3 Start training\nNow that we have our training configuration, to start training simply run `train.sh` as below. \nPlease keep in mind that we have setup a dummy data with one file to train a dummy network fast (we only train for 2 epochs). \nPlease see exercises on how to easily switch data and train a real segmentation network.\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "! $MMAR_ROOT/commands/train_W_Config.sh config_train_Unet.json",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Now lets see the `models` directory, which would includes out models and the tensorboard files ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "! ls -la $MMAR_ROOT/models/config_train_Unet\n!echo ---------------------------------------\n!echo Display content of train_stats.json\n! cat $MMAR_ROOT/models/config_train_Unet/train_stats.json",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n# 5. Export Model\n\nTo export the model we simply run `export.sh` which will: \n- Create ts file\nThis optimized model will be used by TRITON server in AIAA and Clara Deploy.\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "! $MMAR_ROOT/commands/export.sh",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n\nlets check out what was created in the folder. \nafter running cell below you should see `model.ts`\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "!ls -la $MMAR_ROOT/models/config_train_Unet/*.ts",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n# 6. Validation \nNow that we have trained our model we would like to run evaluation to get some statistics and also do inference to see the resulted output\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 6.1 Validate with single GPU \nTo run evaluation on your validation dataset you should run `validate.sh`. \nThis will run evaluation on the validation dataset and place it in the `MMAR_EVAL_OUTPUT_PATH` as configured in the [environment.json](config/environment.json) \nfile (default is eval folder). \nThis evaluation would give min, max, mean of the metric as specified in the config_validation file\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "! $MMAR_ROOT/commands/validate.sh\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "You could also run `validate_ckpt.sh` which loads the model from the checkpoint instead of the ts file",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "! $MMAR_ROOT/commands/validate_ckpt.sh\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 6.2 Validate with multiple GPUs \nYou can also leverage multi-GPUs for validation using `validate_multi_gpu.sh` ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "!$MMAR_ROOT/commands/validate_multi_gpu.sh ",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Similarly you could also run `validate_multi_gpu_ckpt.sh` which loads the model from the checkpoint instead of the ts file",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "! $MMAR_ROOT/commands/validate_multi_gpu_ckpt.sh\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 6.3 Check Validation results \nNow lets see results in the folder by running cells below. ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "!ls -la $MMAR_ROOT/eval\nfor fName in [\"metrics.csv\",\"val_mean_dice_raw.csv\",\"val_mean_dice_summary.csv\"]:\n    print(\"---------------------------------------\")\n    print(\"Display content of \",fName)\n    ! cat $MMAR_ROOT/eval/$fName\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# 7. Inference  \n\nTo run inference on validation dataset or test dataset you should run `infer.sh`. \nThis will run prediction on the validation dataset and place it in the `MMAR_EVAL_OUTPUT_PATH` as configured in the \n[environment.json](config/environment.json) file (default is eval folder)\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "! $MMAR_ROOT/commands/infer.sh",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Now lets see results in the folder",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "! ls -la $MMAR_ROOT/eval/",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "! ls -la $MMAR_ROOT/eval/spleen_8\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# 8.Multi-GPU Training\nClara train aims to simplify scaling and utilizing all available GPUs. \nUsing the same config we already used for train we can simply invoke `train_multi_gpu.sh` to train on multiple gpus. \nMain difference between the `train.sh` and `train_multi_gpu.sh` is changing some parameters\n\ntrain.sh | train_multi_gpu.sh  \n --- | --- \npython3 -u -m medl.apps.train \\\\\u003cbr\u003e-m MMAR_ROOT \\\\\u003cbr\u003e-c CONFIG_FILE \\\\\u003cbr\u003e-e ENVIRONMENT_FILE \\\\\u003cbr\u003e--write_train_stats \\\\\u003cbr\u003e--set \\\\\u003cbr\u003e print_conf\u003dTrue | python -m torch.distributed.launch\\\\\u003cbr\u003e --nproc_per_node\u003d2 --nnodes\u003d1 --node_rank\u003d0 \\\\\u003cbr\u003e --master_addr\u003d\"localhost\" --master_port\u003d1234 \\\\\u003cbr\u003e-m medl.apps.train \\\\\u003cbr\u003e-m MMAR_ROOT \\\\\u003cbr\u003e-c CONFIG_FILE \\\\\u003cbr\u003e-e ENVIRONMENT_FILE \\\\\u003cbr\u003e --write_train_stats \\\\\u003cbr\u003e --set \\\\\u003cbr\u003e print_conf\u003dTrue \\\\\u003cbr\u003e multi_gpu\u003dTrue \\\\\u003cbr\u003e learning_rate\u003d 2e-4\n \nLets examine `train_multi_gpu.sh` script by running cell below. ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "printFile(MMAR_ROOT+\"/commands/train_multi_gpu.sh\",0,50)",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Lets give it a try and run cell below to train on 2 GPUs",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "! $MMAR_ROOT/commands/train_multi_gpu.sh",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n# 9. Training Vs FineTune\n`train.sh` and `finetune.sh` are identical and use the same config file. \nThe only difference is that `finetune.sh` enables the load of check point using the `disabled` as shown below \n\nexcept they train using different configurations files. \n\n_Note_: The only difference between the two configs `config_train_Unet.json` and `config_finetune.json` \nis that `config_finetune.json` specifies a `ckpt` file in section below \nwhile `config_train_Unet.json` does not since it is training from scratch.\n```\n      {\n        \"name\": \"CheckpointLoader\",\n        \"args\": {\n          \"disabled\": \"{dont_load_ckpt_model}\",\n          \"load_path\": \"{MMAR_CKPT}\",\n          \"load_dict\": [\"model\"]\n        }\n      },\n```\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# 10. Profiling\nNvidia provides multiple tools for profiling your training in order to eliminate bottlenecks. \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 10.1 Profiling with DLprof \nDLprof is a simple tool that does the analysis in of regular training then display results in tensorboard.\nMoreover, it provides recommendations guiding user on how to improve performance \n\nCell below uses DLprof tool. \nWe will use same config as we used above expect it have `AMP\u003dfalse` to see DLProf analysis  ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "!$MMAR_ROOT/commands/debug_dlprof.sh config_train_Unet_NoAMP.json",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% \n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "You then need to run tensor board manually (Not through jupyterlab) using \n```\ncd /claraDevDay/GettingStarted/models/config_train_Unet_NoAMP_debug\ntensorboard --logdir ./dlprof --port 5000\n```\nRecall we mounted port 3031 to 5000 by default for AIAA in the `docker-compose.yml` file, \nwe simply are using that mapping here for simplicity \nnow if you navigate to `\u003cyourip:3031\u003e` you should see DlProf tool as below. \nThis analysis shows you the GPUs you have along improvements that you can do to train faster. \nFor example this run shows multiple operations that would be accelerated from AMP.\nTo test this you can run cell below with AMP enabled in the configuration \n\n\u003cbr\u003e\u003cimg src\u003d\"screenShots/Dlprof.png\" alt\u003d\"Drawing\" style\u003d\"height: 400px;\"/\u003e\u003cbr\u003e\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 10.2 Profile your model with Nsight System\nNsight System is more advanced\n ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "In order to train faster you would need to analyze your training loop and check for bottlenecks.\n1. Download Nisght Systems locally on your machine from https://developer.nvidia.com/nsight-systems \n2. Open Nsight System locally (out side the docker)\n3. Load up files from \u003clocal path\u003e/GettingStarted/models/config_train_Unet_NoAMP_debug\nyou should see image analysis as below \n\u003cbr\u003e\u003cimg src\u003d\"screenShots/Nsight1FromDlprof.png\" alt\u003d\"Drawing\" style\u003d\"height: 400px;\"/\u003e\u003cbr\u003e\n\n# Next:",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "### 1. Load model into AIAA\nWe will show here how you can quickly load up the model we trained above into AIAA. \nFirst, you should run [AIAA Notebook](../AIAA/AIAA.ipynb) to start the server.\nSection 3.1 in the AIAA notebook shows how to load trained model into AIAA server. \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "### 2. Bring your own Components\nIn order to fully take advantage of clara train SDK you should write your own components. \nPlease go to [BYOC notebook](BYOC.ipynb) for examples  \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# Exercise:\nNow that you are familiar with clara train, you can try to: \n1. Explore different options of clara train by changing / creating a new config file and running training: \n    1. Model architecture: Ahnet, Unet, Segresnet \n    2. Losses\n    3. Transformation \n\nHint: you for training segresnet you can use the configuration `config_train_segresnet.json` that only changed the network section.\nyou can train by running cell below     ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "!$MMAR_ROOT/commands/train_W_Config.sh config_train_segresnet.json\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "2. Train on real spleen data for this you should:\n    1. Download spleen dataset by running the [download](../Data/DownloadDecathlonDataSet.ipynb) Notebook\n    2. Switch the dataset file in the [environment.json](config/environment.json)\n    3. rerun the `train.sh`\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "3. Experiment with multi-GPU training by changing number of gpus to train on from 2 to 3 or 4. \nYou should edit [train_multi_gpu.sh](commands/train_multi_gpu.sh) then rerun the script \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    },
    "stem_cell": {
      "cell_type": "raw",
      "source": "\u003c!--- SPDX-License-Identifier: Apache-2.0 --\u003e\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}