Skip to content

Commit 3ea3f5d

Browse files
committed
Updates to the GCP K8s installation documentation (#430)
* Updates to the GCP K8s installation documentation * Fix GKE/EKS mixup
1 parent 191180c commit 3ea3f5d

File tree

2 files changed

+38
-43
lines changed

2 files changed

+38
-43
lines changed

docs/setup_installation/aws/getting_started.md

+5-6
Original file line numberDiff line numberDiff line change
@@ -8,21 +8,20 @@ SageMaker and KubeFlow. This guide shows how to set up the Hopsworks platform in
88

99
To follow the instruction on this page you will need the following:
1010

11-
- Kubernetes Version: Hopsworks can be deployed on AKS clusters running Kubernetes >= 1.27.0.
11+
- Kubernetes Version: Hopsworks can be deployed on EKS clusters running Kubernetes >= 1.27.0.
1212
- [aws-cli](https://aws.amazon.com/cli/) to provision the AWS resources
1313
- [eksctl](https://eksctl.io/) to interact with the AWS APIs and provision the EKS cluster
1414
- [helm](https://helm.sh/) to deploy Hopsworks
1515

16-
## ECR Registry
16+
### ECR Registry
1717

1818
Hopsworks allows users to customize the images used by Python jobs, Jupyter Notebooks and (Py)Spark applications running in their projects. The images are stored in ECR. Hopsworks needs access to an ECR repository to push the project images.
1919

20-
## Permissions
20+
### Permissions
2121

22-
By default, the deployment requires cluster admin level access to be able to create a set of ClusterRoles, ServiceAccounts and ClusterRoleBindings. If you don’t have cluster admin level access, you can ask your administrator to provision the necessary ClusterRoles, ServiceAccounts and ClusterRoleBindings as described in the section below.
23-
24-
A namespace is required to deploy the Hopsworks stack. If you don’t have permissions to create a namespace you should ask your K8s administrator to provision one for you.
22+
- The deployment requires cluster admin access to create ClusterRoles, ServiceAccounts, and ClusterRoleBindings.
2523

24+
- A namespace is required to deploy the Hopsworks stack. If you don’t have permissions to create a namespace, ask your EKS administrator to provision one.
2625

2726
## EKS Deployment
2827

docs/setup_installation/gcp/getting_started.md

+33-37
Original file line numberDiff line numberDiff line change
@@ -9,15 +9,11 @@ SageMaker and KubeFlow. This guide shows how to set up the Hopsworks platform in
99

1010
To follow the instruction on this page you will need the following:
1111

12-
- Kubernetes Version: Hopsworks can be deployed on AKS clusters running Kubernetes >= 1.27.0.
13-
- The [gcloud CLI](https://cloud.google.com/sdk/gcloud)
14-
- The [gsutil tool](https://cloud.google.com/storage/docs/gsutil)
15-
- kubectl (to manage the AKS cluster)
16-
- helm (to deploy Hopsworks)
12+
- Kubernetes Version: Hopsworks can be deployed on GKE clusters running Kubernetes >= 1.27.0.
13+
- [gcloud CLI](https://cloud.google.com/sdk/gcloud) to provision the GCP resources
14+
- [gke-gcloud-auth-plugin](https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke) to manage authentication with the GKE cluster
15+
- [helm](https://helm.sh/) to deploy Hopsworks
1716

18-
## GCR Registry
19-
20-
Hopsworks allows users to customize images for Python jobs, Jupyter Notebooks, and (Py)Spark applications. These images should be stored in Google Container Registry (GCR). The GKE cluster needs access to a GCR repository to push project images.
2117

2218
### Permissions
2319

@@ -38,7 +34,7 @@ gsutil mb -l $region gs://$bucket_name
3834

3935
### Step 1.2: Create Service Account
4036

41-
Create a file named hopsworksai_role.yaml with the following content:
37+
Create a file named `hopsworksai_role.yaml` with the following content:
4238

4339
```bash
4440
title: Hopsworks AI Instances
@@ -67,56 +63,54 @@ includedPermissions:
6763
Execute the following gcloud command to create a custom role from the file. Replace $PROJECT_ID with your GCP project id:
6864

6965
```bash
70-
gcloud iam roles create hopsworksai_instances --project=$PROJECT_ID --file=hopsworksai_role.yaml
66+
gcloud iam roles create hopsworksai_instances \
67+
--project=$PROJECT_ID \
68+
--file=hopsworksai_role.yaml
7169
```
7270

73-
Create a service account:
74-
7571
Execute the following gcloud command to create a service account for Hopsworks AI instances. Replace $PROJECT_ID with your GCP project id:
7672

7773
```bash
78-
gcloud iam service-accounts create hopsworksai_instances --project=$PROJECT_ID --description="Service account for Hopsworks AI instances" --display-name="Hopsworks AI instances"
74+
gcloud iam service-accounts create hopsworksai_instances \
75+
--project=$PROJECT_ID \
76+
--description="Service account for Hopsworks AI instances" \
77+
--display-name="Hopsworks AI instances"
7978
```
8079

8180
Execute the following gcloud command to bind the custom role to the service account. Replace all occurrences $PROJECT_ID with your GCP project id:
8281

8382
```bash
84-
gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:hopsworks-ai-instances@$PROJECT_ID.iam.gserviceaccount.com" --role="projects/$PROJECT_ID/roles/hopsworksai_instances"
83+
gcloud projects add-iam-policy-binding $PROJECT_ID \
84+
--member="serviceAccount:hopsworks-ai-instances@$PROJECT_ID.iam.gserviceaccount.com" \
85+
--role="projects/$PROJECT_ID/roles/hopsworksai_instances"
8586
```
8687

87-
8888
### Step 1.3: Create a GKE Cluster
8989

9090
```bash
91-
gcloud container clusters create <cluster-name> --zone <zone> --machine-type n2-standard-8 --num-nodes 3 --enable-ip-alias --service-account [email protected]
91+
gcloud container clusters create <cluster-name> \
92+
--zone <zone> \
93+
--machine-type n2-standard-8 \
94+
--num-nodes 1 \
95+
--enable-ip-alias \
96+
--service-account [email protected]
9297
```
93-
94-
### Step 1.4: Create GCR repository
95-
96-
Enable Artifact Registry and create a GCR repository to store images:
98+
Once the creation process is completed, you should be able to access the cluster using the kubectl CLI tool:
9799

98100
```bash
99-
gcloud artifacts repositories create <repo-name> --repository-format=docker --location=<region>
101+
kubectl get nodes
100102
```
101103

102-
### Step 1.5: Link the GCS bucket and the GCR repository
103-
104-
```bash
105-
gsutil iam ch serviceAccount:[email protected]:objectViewer gs://YOUR_BUCKET_NAME
106-
gsutil iam ch serviceAccount:[email protected]:objectAdmin gs://YOUR_BUCKET_NAME
107-
108-
gsutil iam ch serviceAccount:YOUR_EMAIL_ADDRESS:objectViewer gs://YOUR_BUCKET_NAME
109-
gsutil iam ch serviceAccount:YOUR_EMAIL_ADDRESS:objectAdmin gs://YOUR_BUCKET_NAME
104+
### Step 1.4: Create GCR repository
110105

111-
gcloud projects add-iam-policy-binding $PROJECT_ID --member="serviceAccount:SERVICE_ACCOUNT_EMAIL" --role="roles/storage.objectViewer"
112-
```
106+
Hopsworks allows users to customize images for Python jobs, Jupyter Notebooks, and (Py)Spark applications. These images should be stored in Google Container Registry (GCR). The GKE cluster needs access to a GCR repository to push project images.
113107

114-
## Step 2: Configure kubectl
108+
Enable Artifact Registry and create a GCR repository to store images:
115109

116110
```bash
117-
gcloud auth configure-docker
118-
119-
kubectl get pods
111+
gcloud artifacts repositories create <repo-name> \
112+
--repository-format=docker \
113+
--location=<region>
120114
```
121115

122116
## Step 3: Setup Hopsworks for Deployment
@@ -177,7 +171,10 @@ global:
177171
Deploy Hopsworks in the created namespace.
178172

179173
```bash
180-
helm install hopsworks hopsworks/hopsworks --namespace hopsworks --values values.gcp.yaml --timeout=600s
174+
helm install hopsworks hopsworks/hopsworks \
175+
--namespace hopsworks \
176+
--values values.gcp.yaml \
177+
--timeout=600s
181178
```
182179

183180
Check that Hopsworks is installing on your provisioned AKS cluster.
@@ -194,7 +191,6 @@ Upon completion (circa 20 minutes), setup a load balancer to access Hopsworks:
194191
kubectl expose deployment hopsworks --type=LoadBalancer --name=hopsworks-service --namespace <namespace>
195192
```
196193

197-
198194
## Step 5: Next steps
199195

200196
Check out our other guides for how to get started with Hopsworks and the Feature Store:

0 commit comments

Comments
 (0)