Skip to content

Deployment

Jump to bottom
Shreyas Rao edited this page Oct 20, 2024 · 14 revisions

To deploy the Docker Image, we will use a Cloud Provider since it is free to host services on these platforms. We have previously used Google Cloud to deploy the server, however, you should be able to host on other platforms like AWS or Azure. Below are the instructions to host on Google Cloud

Before you start ensure the following:

  • Access to the MongoDB Atlas related to the DS3 Developer Email
  • Created a Google Cloud Platform Account related to the DS3 Developer Email
  • Review the cost on GCP Price Calculator

MongoDB Atlas

  1. Sign in to the DS3 Developers MongoDB Atlas, and check to see if the aws-ds3-datathon cluster is active, if it not activate it

  2. Go to Database Access and edit the password for user gcp

  3. Autogenerate and save the PASSWORD for later use

Setup GCP Project

  1. Once a GCP Account has been created, go here to Create a Project

  2. Enter the PROJECT_NAME as: DS3 Datathon <YEAR> Leaderboard

  3. Enter the PROJECT_ID as: ds3-datathon-<YEAR>-leaderboard

  4. Create the Project, and wait for the resources to be provisioned

  5. Navigate over to the Billing and complete the setup there

  6. Setup a Budget and Alert to ensure that an alert is sent if you are billed for resources

  7. Once Billing is setup, navigate to APIs & Services to initiate the following services:

    • Secret Manager API
    • Cloud Run API
    • Cloud Scheduler API

gcloud cli

Much of the below guide will make use of the cli, so start here and then proceed:

  1. Follow the step-by-step quick start guide on the official docs to install gcloud SDK

  2. After running gcloud init, select the above project from the list

Google IAM

Before you deploy any services you must create a GCP Service Account its role is to manage the deployment and ensure you are following the Least Privileges Principle.

  1. Create the Service Account with SERVICE_ACCOUNT_NAME as ds3-leaderboard-runner

    gcloud iam service-accounts create ds3-leaderboard-runner --display-name "DS3 Leaderboard Runner"

  2. After creating the Service Account, provide it with the permission for Secret Manager Secret Accessor

    gcloud projects add-iam-policy-binding ds3-datathon-<YEAR>-leaderboard \
  --member='serviceAccount:ds3-leaderboard-runner@ds3-datathon-<YEAR>-leaderboard.iam.gserviceaccount.com' \
  --role='roles/secretmanager.secretAccessor'

  3. After creating the Service Account, provide it with the permission for Cloud Run Invoker

    gcloud projects add-iam-policy-binding ds3-datathon-<YEAR>-leaderboard \
  --member='serviceAccount:ds3-leaderboard-runner@ds3-datathon-<YEAR>-leaderboard.iam.gserviceaccount.com' \
  --role='roles/run.invoker'

Google Secrets Manager

Before we can deploy the Docker Image, we configure secrets to provide the environment variables during runtime

  1. Create secret KAGGLE_KEY:

    echo -n <KAGGLE_KEY> | gcloud secrets create KAGGLE_KEY --data-file='-'

  2. Create secret KAGGLE_USERNAME:

    echo -n <KAGGLE_USERNAME> | gcloud secrets create KAGGLE_USERNAME --data-file='-'

  3. Create secret MONGO_DB:

    echo -n prod | gcloud secrets create MONGO_DB --data-file='-'

  4. Create secret MONGO_URI:

    echo -n 'mongodb+srv://gcp:<PASSWORD>@azure-datathon.qzzfe1n.mongodb.net/?retryWrites=true&w=majority' | 
gcloud secrets create MONGO_URI --data-file='-'

Google Cloud Run

Run the following command below to create an instance of the Leaderboard and store the URL for the next step

gcloud run deploy leaderboard \
 --max-instances=1 \
 --service-account='ds3-leaderboard-runner@ds3-datathon-<YEAR>-leaderboard.iam.gserviceaccount.com' \
 --timeout=300 \
 --memory=512Mi \
 --port=8000 \
 --set-secrets=KAGGLE_KEY=KAGGLE_KEY:latest,KAGGLE_USERNAME=KAGGLE_USERNAME:latest,MONGO_DB=MONGO_DB:latest,MONGO_URI=MONGO_URI:latest \
 --image='docker.io/devds3/leaderboard:<TAG_NAME>' \
 --cpu-boost \
 --allow-unauthenticated \
 --description='Updates the Datathon Public Leaderboard' \
 --region='us-central1'

Google Cloud Scheduler

To be able to constantly update the leaderboard, we make use of a service called Cloud Scheduler which send requests to an endpoint every so often

  1. Create a CRONTAB will be used as the schedule, we have previously used the following here

  2. Initialize App Engine

    gcloud app create --region='us/central1'

  3. Create the Cloud Scheduler Job

    gcloud scheduler jobs create http leaderboard-requester \
 --schedule='<CRONTAB>' \
 --uri=<URL>/public \
 --attempt-deadline='3m' \
 --description='Sends POST request to leaderboard server' \
 --headers=Content-Type=application/json,User-Agent=Google-Cloud-Scheduler \
 --http-method='post' \
 --time-zone="EST" \
 --message-body="{ \"competitions\": [ \"<COMPETITION_ID>\", \"<COMPETITION_ID>\", \"<COMPETITION_ID>\" ] }"

Clean Up

After the submissions are closed, the public leaderboard will no longer need to be updated and thus services can be removed from usage

  1. Delete the Cloud Scheduler Job: leaderboard-requester

    gcloud scheduler jobs delete leaderboard-requester

  2. Delete the Cloud Run Service: leaderboard

    gcloud run services delete leaderboard --region='us/central1'

  3. Delete all environmental variables stores in Secrets Manager:

    gcloud secrets delete KAGGLE_KEY
gcloud secrets delete KAGGLE_USERNAME
gcloud secrets delete MONGO_DB
gcloud secrets delete MONGO_URI

  4. Delete the Service Account called: ds3-leaderboard-runner@ds3-datathon--leaderboard.iam.gserviceaccount.com

    gcloud iam service-accounts delete ds3-leaderboard-runner@ds3-datathon-<YEAR>-leaderboard.iam.gserviceaccount.com

  5. Delete the Project with ds3-datathon--leaderboard

    gcloud projects delete ds3-datathon-<YEAR>-leaderboard

Clone this wiki locally