PyTorch/NoteBooks/FL/Admin_BYOC.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": "# FL Advanced Features \n\nThis notebook will walk you through advanced features of FL as:\n1. Configure privacy to clients \n1. Bring your own Aggregator to FL\n1. Bring your own privacy \n1. Running realistic FL experiment\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## Prerequisites\nBefore you continue make sure you are comfortable with FL and you have \n- Ran [Provisioning Notebook](Provisioning.ipynb) and started the server.\n- Ran [Client Notebook](Client.ipynb) checks for each client (at least 1 client).\n- Ran through [Admin Notebook](Admin.ipynb) \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "### Resources\nWe encourage you to watch the free GTC 2021 talks covering Clara Train SDK\n- [Clara Train 4.0 - 201 Federated Learning [SE3208]](https://gtc21.event.nvidia.com/media/Clara%20Train%204.0%20-%20201%20Federated%20Learning%20%5BSE3208%5D/1_m48t6b3y)\n- [Federated Learning for Medical AI [S32530]](https://gtc21.event.nvidia.com/media/Federated%20Learning%20for%20Medical%20AI%20%5BS32530%5D/1_z26u15uk)\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# Lets get started\n\nWe will continue using the project1 we created in previous notebooks. \nLets look into start by installing tree to look at directory structures ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "MMAR_DIR\u003d\"/claraDevDay/FL/project1/\"\n!apt-get install tree\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "#### Recommended JupyterLab setup \nThe Admin tool runs in an interactive shell therefore unfortunately we can\u0027t have cells to run the notebook. \nInstead we recommend you open a terminal to be as image below. \nYou could also open multiple terminals if you are interested in seeing what is the output for the client and server. \nBelow we have this notebook on the left, server and 2 clients on the top right and the admin shell on the bottom right\n\u003cbr\u003e![fl](screenShots/JLabLayout.png)\u003cbr\u003e",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 1- Make sure server and clients are connected \nin the terminal go to admin folder in side your project and run\n```\n./fl_admin.sh\n``` \ntype `admin@admin.com` \n\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Check server/client status\ntype \n```\n\u003e check_status server\n\u003e check_status client\n```\n\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "**Reminder**\u003cbr\u003e\nJust a reminder you should:\n\n1. `\u003e set_run_number \u003c123\u003e`\n1. `\u003e upload_folder ../../../adminMMAR_privacy`\n2. `\u003e deploy adminMMAR_privacy server`\n3. `\u003e deploy adminMMAR_privacy client` \n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "Then to start training you should:\n\n5. `\u003e start server`\n6. `\u003e start client`\n7. `\u003e check_status server` and/or `\u003e check_status client`   \n8. `\u003e cat server log.txt` to get logs from clients \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 2. Privacy and Model Protection \nTo mitigate the risk of recovering the training data from the trained model, \nwhich is also commonly known as reverse engineering or model inversion, \nwe provide a configurable client-side privacy control based on the differential-privacy (DP) technique. \nDuring training, each client could have their own privacy policy and could be updated by the admin client during training. \n\nThe DP protection consists of two major components: selective parameter update and sparse vector technique (SVT):\nFor selective parameter update, the client only sends a partial of the model weights/updates, \ninstead of the whole, to limit the amount of information shared. \nThis is achieved by:\n1. only uploading the fraction of the model weights/updates whose absolute values are greater than a predefined threshold or percentile of the absolute update values; \n2. further replacing the model weights by clipping its value to a fixed range. \nSparse vector technique operates on a random fraction of the weights/updates x by first adding a random noise to its absolute value abs(x)+Lap(s); \nthen share the clipped noisy value clip(x+Lap(s), ) iff the thresholding condition is satisfied. \nHere abs(x) represents absolute value, Lap(x) denotes a random valuable sampled from the Laplace distribution, \nis a pre-defined threshold, clip(x,) denotes clipping of x value to be in the range of [-,]. \n\nFor detail, please refer to [W, Li. et, al “Privacy-preserving Federated Brain Tumour Segmentation”,  arXiv preprint arXiv:1910.00962 (2019)](https://arxiv.org/abs/1910.00962)\nThe experimental results show that there is a tradeoff between model performance and privacy protection costs.\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "## 2.1 Use built in privacy Algorithms in Clara SDK\nYou can try enabling privacy on each client to drop percentage of the weights using the privacy section in the `client.json`. \nFor this check files in  \n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% md\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "You can do one or more of the following options:\n1. Use PercentileProtocol Algorithm by editing your client `config_fed_client.json`. \nYou should add section below to the `outbound_filters`\n    ```\n      {\n          \"name\": \"PercentilePrivacy\",\n          \"args\": {\n              \"gamma\": 0.1\n          }\n      },\n    ``` \nor you could copy `config_fed_client_w_PercentileProtocol.json` to your client config_fed_client.json by",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "clientNo\u003d\"client1\"\nrunNo\u003d\"1\"\n! cp $MMAR_DIR/../adminMMAR_privacy/config/config_fed_client_w_PercentileProtocol.json $MMAR_DIR/$clientNo/run_$runNo/mmar_$clientNo/config/config_fed_client.json",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% \n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "!tree $MMAR_DIR/$clientNo/run_$runNo",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "**You should now try re running an FL experiments with** \n```\n\u003e start server\n\u003e start client\n```\nYou should see the training go through successfully. \nTo should train a realistic model and see the accuracy effect as you change the privacy parameters  \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "2. SVTProtocol Algorithm by copying section below into your client `config_fed_client.json`\nYou should add section below to the `outbound_filters`\n\n```\n  {\n      \"name\": \"SVTPrivacy\",\n      \"args\": {\n        \"fraction\":0.1,\n        \"epsilon\":0.1,\n        \"noise_var\":0.1,\n        \"gamma\":1e-5,\n        \"tau\":1e-6\n      }\n  }\n```\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "clientNo\u003d\"client3\"\nrunNo\u003d\"1\"\n!cp $MMAR_DIR/../adminMMAR_privacy/config/config_fed_client_w_SVT.json \\\n  $MMAR_DIR/$clientNo/run_$runNo/mmar_$clientNo/config/config_fed_client.json",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%% \n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "!tree $MMAR_DIR/$clientNo/run_$runNo",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "**You should now try re running an FL experiments with** \n```\n\u003e start server\n\u003e start client\n```\nYou should see the training go through successfully. \nTo should train a realistic model and see the accuracy effect as you change the privacy parameters  \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# 3. Model Aggregator\nModel aggregation happens on the server as pacified in the `config_fed_server.json` file. Clara train comes with built in aggregator \n## 3.1 Built in Aggregator\nThis aggregator is based on algorithm in [Federated Learning for Breast Density Classification: A Real-World Implementation](https://arxiv.org/abs/2009.01871).\nThe ModelAggregator computes a weighted sum of the model gradients from each client, \nwhere the default weights are based on the number of training iterations that the client executed in this round of FL. \nThe user can further adjust the client’s weights by adding additional custom weights in the arguments of this component.\n\nyou can adjust this by changing arguments in the aggregation section of the `config_fed_server.json`\n```\n\"aggregator\":\n  {\n    \"name\": \"InTimeAccumulateWeightedAggregator\",\n    \"args\": {\n      \"aggregation_weights\":\n          {\n            \"client0\": 1,\n            \"client1\": 1.5,\n            \"client2\": 0.8\n          }\n    }\n  },\n``` \n## 3.2 Bring your own Aggregator to FL\nA sample of writing your own aggregator is shown in [custom_aggregator.py](adminMMAR/custom/custom_aggregator.py).\nThis file was already in the custom folder that we have been using. \nTherefore, in the `config_fed_server.json`  you simply need to change the component `name` tag into `path` and point to your code as below  \n\n```\n\"aggregator\":\n  {\n    \"path\": \"BYO_aggregator.MyJustInTimeAggregator\",\n    \"args\": {\n    }\n  },\n```\nThis change is already in the `config_fed_server.json` ",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "# runNo\u003d\"10\"\n# ! cp $MMAR_DIR/../adminMMAR_privacy/config/config_fed_server.json $MMAR_DIR/server/run_$runNo/mmar_server/config/config_fed_server.json",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "source": "# !tree $MMAR_DIR/server/run_$runNo\n",
      "metadata": {
        "pycharm": {
          "metadata": false,
          "name": "#%%\n"
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "**You should now try re running an FL experiments with** \n```\n\u003e start server\n\u003e start client\n```\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# 4. Security \nAs you might have noticed from this notebook there are couple of security issues with the current setup.\n\n# 4.1 Disallow BYOC  \nFirst issues is BYOC code is pushed by admin in the custom folder to all clients. \nThis is potential threat as admin could write any malicious code to run on the client.\n\nThe solution to this is to rule groups. \nThere you can set `BYOC` and/or `allow custom datalist` to false.\nThis will disallow uploading the custom folder of the mmar. \nIf research lead tries to do this he would get error \n```\nError: Authorization Error: the MMAR contains custom code, which is not allowed on site \"org1-a\" and reject any uploads\n``` \nYou can test this by starting client3 which has this strict rule and try to upload the mmar.\n\nFor clients how are more strict the workflow should be:\n1. Lead researched send custom folder to client via email.\n2. Client review the code and verify it is not doing any harm\n3. Client should place custom code in `/local/custom` \n\n# 4.2 Roles and Rights\nAll experiments till now, we have logged in as super user `admin@admin.com` who has all rights to de everything. \nYou should now try to log in with restricted user who have limited action.\n\nin the admin terminal you should: \n1. `cd project1/leadIT/startup`\n2. `./fl_admin.sh`\n3. log in as `leadIT@org1.com`\n   \nIn the admin terminal you should: \n1. `cd project1/leadIT/startup`\n2. `./fl_admin.sh`\n3. log in as `siteresearcher@org2.com`\n\n# 4.3 Disallow all Roles and Rights\nIn some extreme cases some clients would want to have the only access to their client that is lead researcher is not allowed to upload any mmar files. \nThis guaranties total security, however this would add the constant communication overhead with every run to move mmars into a run folder. \n\nNever the less this option is provided by steps below (see image):\n1. creating an new rights group (for Example: `only4secure`)\n2. Make sure other groups don\u0027t have any access\n3. Give access to  self deploy and train for this group\n\u003cbr\u003e\u003cimg src\u003d\"screenShots/Authorization4Sevure.png\" alt\u003d\"Drawing\" style\u003d\"height: 300px;\"/\u003e\u003cbr\u003e\n\nIn our Example, we have set up `client4` to be super secure with the only user allowed to upload is user `leadIT@secure.com`\n\nTo test this actually works:\n1. cd to client4 and start the client\n    1. `cd /claraDevDay/FL/project1/client4/startup`\n    2. `./start.sh`\n2. cd to leadIT and log in\n    In the admin terminal you should: \n    1. `cd project1/leadIT/startup`\n    2. `./fl_admin.sh`\n    3. log in as `leadIT@secure.com`\n3. try to upload mmar to client4 using `\u003e deploy adminMMAR client client4` \n4. You should get message below \n```\nError: Authorization Error: you are not authorized to deploy MMAR to \"client4\"\nDone [15666 usecs] 2020-10-26 18:55:39.807613\n```\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# Next steps:\nYou can now move to more advanced features of FL as:\n- [Bring Your Own Trainer Notebook](FLBYOTrainer.ipynb)\n- [Homomorphic Encryption Notebook](Homomorphic_Encryption.ipynb)\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# Exercise:\n### 1. Redo steps above the proper way\nIn the steps above we cut some corners as copying files / configs between admin and server or clients. \nAll of these steps should run from with in the admin console. \nAs stated in the Admin notebook, the correct way to do this is to \n1. set a run number \n1. Upload files from admin transfer folder to server staging \n2. run `\u003e deploy server` to copy files from staging to server run folder\n3. run `\u003e deploy client clientname` to copy files from staging to client \n4. start training on server and client \n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": "# Security:\nAs you might have noticed from this notebook there are couple of issues with the current setup. \nBoth issues below could be restricted when provisioning the project. \nCheck out [Provisioning](Provisioning.ipynb) Notebook Step 4   \n\n1. BYOC code is pushed by admin in the custom folder to all clients. \nThis is potential threat as admin could write any malicious code. \nIf this option is restricted then the BYOC code needs to be emailed or passed to clients, \nthen clients would check the code and place it in the custom folder so it is executed    \n2. Clients privacy: the method is set in the client config file. \nHowever, admin can push and override that the client does/want. \n\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.8"
    },
    "stem_cell": {
      "cell_type": "raw",
      "source": "\u003c!--- SPDX-License-Identifier: Apache-2.0 --\u003e\n",
      "metadata": {
        "pycharm": {
          "metadata": false
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}