Skip to content

Commit e45af12

Browse files
hdahmehdahme
committed
+ terraform config and gcloud setup scripts
1 parent 98e46ef commit e45af12

16 files changed

+264
-8
lines changed

‎.gitignore

+12-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,14 @@
11
/venv/
22
/__pycache__/
3-
juno-arb/__pycache__/
3+
4+
# juno-arb
5+
juno-arb/__pycache__/
6+
juno-arb/.env
7+
juno-arb/skip_response.log
8+
juno-arb/ops/gcp/terraform/.terraform*
9+
10+
juno-arb/ops/gcp/terraform/provider.tf
11+
juno-arb/ops/gcp/terraform/variables.tf
12+
13+
juno-arb/Dockerfile
14+
juno-arb/.dockerignore

‎juno-arb/.env.copy

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
export MNEMONIC=""

‎juno-arb/README.md

+4-4
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,10 @@ Once you have python 3.10, install all the dependencies:
1515
pip install -r requirements.txt
1616
```
1717

18-
Then, edit the global variables in the main.py file to
19-
your liking. The most important being your mnemonic.
20-
```python
21-
MNEMONIC = "<your mnemonic>"
18+
Copy .env.copy and populate with your mnemonic. **Don't commit this**
19+
20+
```bash
21+
source .env
2222
```
2323

2424
Lastly, run the bot:

‎juno-arb/main.py

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
# General Imports
2+
import os
23
import json
34
import aiometer
45
import asyncio
@@ -34,7 +35,7 @@
3435

3536
# Mnenomic to generate private key
3637
# Replace with your own mnemonic
37-
MNEMONIC = "<your mnemonic>"
38+
MNEMONIC = os.environ.get("MNEMONIC")
3839

3940
# RPC URL to stream mempool and query contract state from
4041
RPC_URL = "https://rpc-juno-ia.cosmosia.notional.ventures/"

‎juno-arb/ops/gcp/cloudrun/.dockerignore

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
.env

‎juno-arb/ops/gcp/cloudrun/Dockerfile

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
FROM python:3.10-slim
2+
3+
RUN apt-get clean \
4+
&& apt-get -y update
5+
6+
RUN apt-get -y install \
7+
nginx \
8+
python3-dev \
9+
build-essential
10+
11+
WORKDIR /app
12+
13+
COPY requirements.txt /app/requirements.txt
14+
RUN pip install -r requirements.txt --src /usr/local/src
15+
16+
COPY . .
17+
18+
CMD [ "python3", "main.py" ]

‎juno-arb/ops/gcp/cloudrun/Readme.md

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
Cloudrun is if you want the easiest, lowest overhead way of running a bot. If you want more flexibility and customizability, check out the `terraform` config.
2+
3+
Since a job times out after a maximum of 1 hour, we get around that by scheduling this job to run every hour, at minute 0
4+
5+
Because these commands are fairly symmetric with Cloud Build, you should be able to plug them into your CI/CD process with little/no lift
6+
7+
Firstly, copy the `.dockerignore` and `Dockerfile` to the root of the bot. Then submit the build to cloudrun
8+
9+
```bash
10+
export REGION="us-central1"
11+
export PROJECT=$(gcloud config get-value project)
12+
export PROJECT_NUMBER=`gcloud projects list | grep $PROJECT | awk '{print $(NF)}'`
13+
export IMAGE="us-docker.pkg.dev/$PROJECT/gcr.io/skip-mev/juno-arb:latest"
14+
export JOB_NAME="skip-mev-juno-arb"
15+
source .env
16+
17+
# Build the image
18+
gcloud builds submit --tag $IMAGE
19+
20+
# Create the job
21+
gcloud beta run jobs create $JOB_NAME \
22+
--image $IMAGE \
23+
--region $REGION \
24+
--task-timeout 1h \
25+
--set-env-vars MNEMONIC="$MNEMONIC"
26+
27+
# Either run the job now
28+
gcloud beta run jobs execute $JOB_NAME --region $REGION
29+
30+
# Or schedule the job
31+
gcloud scheduler jobs create http $JOB_NAME \
32+
--location $REGION \
33+
--schedule "0 * * * *" \
34+
--uri="https://$REGION-run.googleapis.com/apis/run.googleapis.com/v1/namespaces/$PROJECT/jobs/$JOB_NAME:run" \
35+
--http-method POST \
36+
--oauth-service-account-email $PROJECT_NUMBER[email protected]
37+
38+
# To delete the job and cancel any scheduled executions
39+
gcloud beta run jobs delete $JOB_NAME
40+
```
41+
42+
You can monitor job executions at https://console.cloud.google.com/run and view logs at https://console.cloud.google.com/run/jobs/details/$REGION/$JOB_NAME/executions

‎juno-arb/ops/gcp/terraform/Readme.md

+72
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
If you want to use a persistent GCE instance to manage your bot, use this config. A persistent GCE instance offers more flexibility in line with more overhead, say if you want to run sidecar services and so on. Use the cloudrun config if you're just looking for the quick and dirty.
2+
3+
Firstly, set up your environment and create the VM
4+
5+
```bash
6+
export TF_VAR_zone="us-central1-c"
7+
export TF_VAR_instance_name="skip-mev-juno-arb"
8+
export PROJECT=$(gcloud config get-value project)
9+
10+
# Copy the .template files
11+
cp variables.template variables.tf
12+
cp provider.template provider.tf
13+
14+
# Dynamic substitution for the terraform config
15+
sed -i "" s/PROJECT_ID/$PROJECT/g variables.tf
16+
sed -i "" s/PROJECT_ID/$PROJECT/g provider.tf
17+
sed -i "" s/ZONE/$TF_VAR_zone/g variables.tf
18+
sed -i "" s/ZONE/$TF_VAR_zone/g provider.tf
19+
sed -i "" s/INSTANCE_NAME/$TF_VAR_instance_name/g variables.tf
20+
21+
# You can always reset to the template with the following
22+
# sed -i "" s/$PROJECT/PROJECT_ID/g variables.template
23+
# sed -i "" s/$PROJECT/PROJECT_ID/g provider.template
24+
# sed -i "" s/$TF_VAR_zone/ZONE/g variables.template
25+
# sed -i "" s/$TF_VAR_zone/ZONE/g provider.template
26+
# sed -i "" s/$TF_VAR_instance_name/INSTANCE_NAME/g variables.template
27+
28+
# Initialize terraform state in GCS
29+
terraform init \
30+
-backend-config="bucket=$PROJECT-tfstate" \
31+
-backend-config="prefix=$TF_VAR_instance_name"
32+
terraform apply
33+
```
34+
35+
Now, we can copy the bot to the instance. Make sure the mnemonic in your `.env` is popualted correctly.
36+
37+
```bash
38+
gcloud compute scp --recurse ../../../*.py ../../../*.json ../../../*.txt ../../../.env ubuntu@$TF_VAR_instance_name:/home/ubuntu --zone $TF_VAR_zone
39+
40+
# SSH onto the machine and follow setup instructions for the bot, namely
41+
gcloud compute ssh --zone $TF_VAR_zone ubuntu@$TF_VAR_instance_name --tunnel-through-iap --project $PROJECT
42+
43+
# Set up dependencies to run from the root, so that the startup script can function
44+
sudo apt update
45+
sudo apt-get install python3-pip -y
46+
sudo mkdir -p /app && sudo cp ~/.env ~/* /app
47+
cd /app
48+
sudo pip install -r requirements.txt
49+
exit
50+
```
51+
52+
[Optionally, but recommended] install the cloudwatch ops agent
53+
*You can also install via the console, by visiting https://console.cloud.google.com/monitoring/dashboards/resourceList/gce_instance*
54+
55+
```bash
56+
cd ~/ && curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
57+
sudo bash add-google-cloud-ops-agent-repo.sh --also-install
58+
```
59+
60+
Finally, restart the instance. The bot is initialized via the startup script specified in the terraform config
61+
62+
```bash
63+
gcloud compute instances reset $TF_VAR_instance_name --zone $TF_VAR_zone
64+
```
65+
66+
To verify the startup script worked, you can run `sudo journalctl -u google-startup-scripts.service` and can view logs by querying the following in the logs explorer
67+
68+
```bash
69+
resource.type="gce_instance"
70+
sourceLocation.function="main.setupAndRunScript"
71+
resource.labels.instance_id="THE_INSTANCE_ID_RETURNED_FROM_TERRAFORM"
72+
```

‎juno-arb/ops/gcp/terraform/backend.tf

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
terraform {
2+
backend "gcs" {}
3+
}

‎juno-arb/ops/gcp/terraform/gce.tf

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
resource "google_compute_instance" "default" {
2+
name = var.instance_name
3+
machine_type = var.machine_type
4+
zone = var.zone
5+
tags = [var.instance_name, "ssh"]
6+
project = var.project
7+
8+
boot_disk {
9+
initialize_params {
10+
image = "ubuntu-os-cloud/ubuntu-2204-lts"
11+
}
12+
}
13+
14+
metadata_startup_script = "cd /app && source .env && python3 main.py &"
15+
16+
network_interface {
17+
network = "default"
18+
19+
access_config {
20+
// Ephemeral public IP
21+
}
22+
}
23+
24+
service_account {
25+
email = google_service_account.default.email
26+
scopes = ["cloud-platform"]
27+
}
28+
29+
allow_stopping_for_update = true
30+
}

‎juno-arb/ops/gcp/terraform/iam.tf

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
resource "google_service_account" "default" {
2+
account_id = "${var.instance_name}-sa"
3+
display_name = "Service Account"
4+
}
5+
6+
resource "google_project_iam_binding" "logs-writer-iam" {
7+
role = "roles/logging.logWriter"
8+
project = var.project
9+
10+
members = [
11+
"serviceAccount:${google_service_account.default.email}",
12+
]
13+
}
14+
15+
resource "google_project_iam_binding" "metrics-writer-iam" {
16+
role = "roles/monitoring.metricWriter"
17+
project = var.project
18+
19+
members = [
20+
"serviceAccount:${google_service_account.default.email}",
21+
]
22+
}

‎juno-arb/ops/gcp/terraform/outputs.tf

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
output "instance_id" {
2+
value = "${google_compute_instance.default.instance_id}"
3+
}

‎juno-arb/ops/gcp/terraform/provider.template

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
provider "google" {
2+
project = "PROJECT_ID"
3+
zone = "ZONE"
4+
}

‎juno-arb/ops/gcp/terraform/variables.template

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
variable project {
2+
type = string
3+
default = "PROJECT_ID"
4+
description = "Project ID"
5+
}
6+
7+
variable "zone" {
8+
type = string
9+
default = "ZONE"
10+
description = "Zone"
11+
}
12+
13+
variable "region" {
14+
type = string
15+
default = "us-central1"
16+
description = "Region"
17+
}
18+
19+
variable "instance_name" {
20+
type = string
21+
default = "INSTANCE_NAME"
22+
description = "Instance Name"
23+
}
24+
25+
variable "machine_type" {
26+
type = string
27+
default = "e2-small"
28+
description = "Machine Type"
29+
}

‎juno-arb/ops/gcp/terraform/vpc.tf

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
resource "google_compute_network" "skip-mev" {
2+
name = "skip-mev"
3+
auto_create_subnetworks = false
4+
mtu = 1460
5+
project = var.project
6+
7+
}
8+
9+
resource "google_compute_firewall" "ssh" {
10+
name = "allow-ssh"
11+
project = var.project
12+
allow {
13+
ports = ["22"]
14+
protocol = "tcp"
15+
}
16+
direction = "INGRESS"
17+
network = google_compute_network.skip-mev.id
18+
priority = 1000
19+
source_ranges = ["0.0.0.0/0"]
20+
target_tags = ["ssh"]
21+
}

‎juno-arb/requirements.txt

-2
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,6 @@ google-api-python-client==2.65.0
2828
google-auth==2.14.1
2929
google-auth-httplib2==0.1.0
3030
googleapis-common-protos==1.56.4
31-
grpcio==1.47.0
3231
h11==0.12.0
3332
httpcore==0.15.0
3433
httplib2==0.21.0
@@ -48,7 +47,6 @@ pexpect==4.8.0
4847
pickleshare==0.7.5
4948
platformdirs==2.5.3
5049
prompt-toolkit==3.0.32
51-
protobuf==3.20.3
5250
psutil==5.9.4
5351
ptyprocess==0.7.0
5452
pure-eval==0.2.2

0 commit comments

Comments
 (0)