re-work ref impl to use path based routing instead (#169)

Signed-off-by: Manabu McCloskey <[email protected]>

.devcontainer/devcontainer.json

@@ -8,5 +8,8 @@
   },
   "postCreateCommand": ".devcontainer/postCreateCommand.sh",
   "workspaceFolder": "/home/vscode/idpbuilder",
-  "workspaceMount": "source=${localWorkspaceFolder},target=/home/vscode/idpbuilder,type=bind"
+  "workspaceMount": "source=${localWorkspaceFolder},target=/home/vscode/idpbuilder,type=bind",
+  "hostRequirements": {
+    "cpus": 4
+  }
 }

examples/ref-implementation/README.md

@@ -18,15 +18,20 @@ Ensure you have the following tools installed on your computer.
 
 ## Installation
 
+**_NOTE:_**
+- If you'd like to run this in your web browser through Codespaces, please follow [the instructions here](./codespaces.md) to install instead. 
+
+- _This example assumes that you run the reference implementation with the default port configguration of 8443 for the idpBuilder.
+If you happen to configure a different host or port for the idpBuilder, the manifests in the reference example need to be updated
+and be configured with the new host and port. you can use the [replace.sh](replace.sh) to change the port as desired prior to applying the manifest as instructed in the command above._
+
 Run the following command from the root of this repository.
 
 ```bash
-idpbuilder create --package-dir examples/ref-implementation
+idpbuilder create --use-path-routing --package-dir examples/ref-implementation
 ```
 
-This will take ~6 minutes for everything to come up. To track the progress, you can go to the [ArgoCD UI](https://argocd.cnoe.localtest.me:8443/applications).
-
-**_NOTE:_**: _This example assumes that you run the reference implementation with the default port configguration of 8443 for the idpBuilder. If you happen to configure a different port for the idpBuilder, the manifests in the reference example need to be updated and be configured with the new port. you can use the [replace-port.sh](replace-port.sh) to change the port as desired prior to applying the manifest as instructed in the command above._
+This will take ~6 minutes for everything to come up. To track the progress, you can go to the [ArgoCD UI](https://cnoe.localtest.me:8443/argocd/applications).
 
 ### What was installed?
 
@@ -49,15 +54,15 @@ The only package that cannot be removed this way is Keycloak because other packa
 
 
 #### Accessing UIs
-- Argo CD: https://argocd.cnoe.localtest.me:8443
-- Argo Workflows: https://argo.cnoe.localtest.me:8443
-- Backstage: https://backstage.cnoe.localtest.me:8443
-- Gitea: https://gitea.cnoe.localtest.me:8443
-- Keycloak: https://keycloak.cnoe.localtest.me:8443/admin/master/console/
+- Argo CD: https://cnoe.localtest.me:8443/argocd
+- Argo Workflows: https://cnoe.localtest.me:8443/argo
+- Backstage: https://cnoe.localtest.me:8443/
+- Gitea: https://cnoe.localtest.me:8443/gitea
+- Keycloak: https://cnoe.localtest.me:8443/keycloak/admin/master/console/
 
 # Using it
 
-For this example, we will walk through a few demonstrations. Once applications are ready, go to the [backstage URL](https://backstage.cnoe.localtest.me:8443).
+For this example, we will walk through a few demonstrations. Once applications are ready, go to the [backstage URL](https://cnoe.localtest.me:8443).
 
 Click on the Sign-In button, you will be asked to log into the Keycloak instance. There are two users set up in this 
 configuration, and their password can be retrieved with the following command:
@@ -73,7 +78,7 @@ Both users use the same password retrieved above.
 
 If you want to create a new user or change existing users:
 
-1. Go to the [Keycloak UI](https://keycloak.cnoe.localtest.me:8443/admin/master/console/). 
+1. Go to the [Keycloak UI](https://cnoe.localtest.me:8443/keycloak/admin/master/console/). 
 Login with the username `cnoe-admin`. Password is the `KEYCLOAK_ADMIN_PASSWORD` field from the command above. 
 2. Select `cnoe` from the realms drop down menu.
 3. Select users tab.
@@ -100,9 +105,9 @@ You can click on the ArgoCD Application name to see more details.
 
 ### What just happened?
 
-1. Backstage created [a git repository](https://gitea.cnoe.localtest.me:8443/giteaAdmin/demo), then pushed templated contents to it.
-2. Backstage created [an ArgoCD Application](https://argocd.cnoe.localtest.me:8443/applications/argocd/demo?) and pointed it to the git repository.
-3. Backstage registered the application as [a component](https://gitea.cnoe.localtest.me:8443/giteaAdmin/demo/src/branch/main/catalog-info.yaml) in Backstage.
+1. Backstage created [a git repository](https://cnoe.localtest.me:8443/gitea/giteaAdmin/demo), then pushed templated contents to it.
+2. Backstage created [an ArgoCD Application](https://cnoe.localtest.me:8443/argocd/applications/argocd/demo?) and pointed it to the git repository.
+3. Backstage registered the application as [a component](https://cnoe.localtest.me:8443/gitea/giteaAdmin/demo/src/branch/main/catalog-info.yaml) in Backstage.
 4. ArgoCD deployed the manifests stored in the repo to the cluster.
 5. Backstage retrieved application health from ArgoCD API, then displayed it.
 

examples/ref-implementation/argo-workflows/manifests/dev/ingress.yaml

@@ -3,14 +3,27 @@ kind: Ingress
 metadata:
   name: argo-workflows-ingress
   namespace: argo
+  annotations:
+    nginx.ingress.kubernetes.io/use-regex: "true"
+    nginx.ingress.kubernetes.io/rewrite-target: /$2
 spec:
   ingressClassName: "nginx"
   rules:
-    - host: argo.cnoe.localtest.me
+    - host: localhost
       http:
         paths:
-          - path: /
-            pathType: Prefix
+          - path: /argo-workflows(/|$)(.*)
+            pathType: ImplementationSpecific
+            backend:
+              service:
+                name: argo-server
+                port:
+                  name: web
+    - host: cnoe.localtest.me
+      http:
+        paths:
+          - path: /argo-workflows(/|$)(.*)
+            pathType: ImplementationSpecific
             backend:
               service:
                 name: argo-server

examples/ref-implementation/argo-workflows/manifests/dev/patches/cm-argo-workflows.yaml

@@ -7,14 +7,14 @@ data:
   config: |
     sso:
       insecureSkipVerify: true 
-      issuer: https://keycloak.cnoe.localtest.me:8443/realms/cnoe
+      issuer: https://cnoe.localtest.me:8443/keycloak/realms/cnoe
       clientId:
         name: keycloak-oidc
         key: client-id
       clientSecret:
         name: keycloak-oidc
         key: secret-key
-      redirectUrl: https://argo.cnoe.localtest.me:8443/oauth2/callback
+      redirectUrl: https://cnoe.localtest.me:8443/argo-workflows/oauth2/callback
       rbac:
         enabled: true
       scopes:

examples/ref-implementation/argo-workflows/manifests/dev/patches/deployment-argo-server.yaml

@@ -13,6 +13,9 @@ spec:
               path: /
               port: 2746
               scheme: HTTP
+          env:
+            - name: BASE_HREF
+              value: "/argo-workflows/"
           args:
             - server
             - --configmap=workflow-controller-configmap

examples/ref-implementation/backstage-templates.yaml

@@ -21,4 +21,3 @@ spec:
       - CreateNamespace=true
     automated:
       selfHeal: true
-

examples/ref-implementation/backstage-templates/entities/app-with-bucket/skeleton/catalog-info.yaml

@@ -19,7 +19,7 @@ metadata:
     backstage.io/kubernetes-namespace: default
     argocd/app-name: ${{values.name | dump}}
   links:
-    - url: https://gitea.cnoe.localtest.me:8443
+    - url: https://cnoe.localtest.me:8443/gitea
       title: Repo URL
       icon: github
 spec:

examples/ref-implementation/backstage-templates/entities/app-with-bucket/template.yaml

@@ -100,7 +100,7 @@ spec:
       input:
         description: This is an example app
         # Hard coded value for this demo purposes only.
-        repoUrl: gitea.cnoe.localtest.me:8443?repo=${{parameters.name}}
+        repoUrl: cnoe.localtest.me:8443/gitea?repo=${{parameters.name}}
         defaultBranch: main
     - id: create-argocd-app
       name: Create ArgoCD App

examples/ref-implementation/backstage-templates/entities/argo-workflows/skeleton/catalog-info.yaml

@@ -13,7 +13,7 @@ metadata:
     apache-spark.cnoe.io/label-selector: env=dev,entity-id=${{values.name}}
     apache-spark.cnoe.io/cluster-name: local
   links:
-    - url: https://gitea.cnoe.localtest.me:8443
+    - url: https://cnoe.localtest.me:8443/gitea
       title: Repo URL
       icon: github
 spec:

examples/ref-implementation/backstage-templates/entities/argo-workflows/template.yaml

@@ -35,7 +35,7 @@ spec:
       input:
         description: This is an example app
         # Hard coded value for this demo purposes only.
-        repoUrl: gitea.cnoe.localtest.me:8443?repo=${{parameters.name}}
+        repoUrl: cnoe.localtest.me:8443/gitea?repo=${{parameters.name}}
         defaultBranch: main
     - id: create-argocd-app
       name: Create ArgoCD App

examples/ref-implementation/backstage-templates/entities/basic/skeleton/catalog-info.yaml

@@ -9,7 +9,7 @@ metadata:
     backstage.io/kubernetes-namespace: default
     argocd/app-name: ${{values.name | dump}}
   links:
-    - url: https://gitea.cnoe.localtest.me:8443
+    - url: https://cnoe.localtest.me:8443/gitea
       title: Repo URL
       icon: github
 spec:

examples/ref-implementation/backstage-templates/entities/basic/template.yaml

@@ -31,7 +31,7 @@ spec:
       input:
         description: This is an example app
         # Hard coded value for this demo purposes only.
-        repoUrl: gitea.cnoe.localtest.me:8443?repo=${{parameters.name}}
+        repoUrl: cnoe.localtest.me:8443/gitea?repo=${{parameters.name}}
         defaultBranch: main
     - id: create-argocd-app
       name: Create ArgoCD App

examples/ref-implementation/backstage/manifests/install.yaml

@@ -70,7 +70,7 @@ data:
   app-config.yaml: |
     app:
       title: CNOE Backstage
-      baseUrl: ${BACKSTAGE_FRONTEND_URL}
+      baseUrl: https://cnoe.localtest.me:8443
     organization:
       name: CNOE
     backend:
@@ -80,7 +80,7 @@ data:
       # auth:
       #   keys:
       #     - secret: ${BACKEND_SECRET}
-      baseUrl: ${BACKSTAGE_FRONTEND_URL}
+      baseUrl: https://cnoe.localtest.me:8443
       listen:
         port: 7007
         # Uncomment the following host directive to bind to specific interfaces
@@ -90,7 +90,7 @@ data:
         # Content-Security-Policy directives follow the Helmet format: https://helmetjs.github.io/#reference
         # Default Helmet Content-Security-Policy values can be removed by setting the key to false
       cors:
-        origin: ${BACKSTAGE_FRONTEND_URL}
+        origin: https://cnoe.localtest.me:8443
         methods: [GET, HEAD, PATCH, POST, PUT, DELETE]
         credentials: true
       database:
@@ -106,12 +106,12 @@ data:
 
     integrations:
       gitea:
-        - baseUrl: https://gitea.cnoe.localtest.me:8443
-          host: gitea.cnoe.localtest.me:8443
+        - baseUrl: https://cnoe.localtest.me:8443/gitea
+          host: cnoe.localtest.me:8443
           username: ${GITEA_USERNAME}
           password: ${GITEA_PASSWORD}
-        - baseUrl: https://gitea.cnoe.localtest.me
-          host: gitea.cnoe.localtest.me
+        - baseUrl: https://cnoe.localtest.me/gitea
+          host: cnoe.localtest.me
           username: ${GITEA_USERNAME}
           password: ${GITEA_PASSWORD}
     #  github:
@@ -166,7 +166,7 @@ data:
       locations:
         # Examples from a public GitHub repository.
         - type: url
-          target: https://gitea.cnoe.localtest.me:8443/giteaAdmin/idpbuilder-localdev-backstage-templates-entities/raw/branch/main/catalog-info.yaml
+          target: https://cnoe.localtest.me/gitea/giteaAdmin/idpbuilder-localdev-backstage-templates-entities/raw/branch/main/catalog-info.yaml
         ## Uncomment these lines to add an example org
         # - type: url
         #   target: https://github.com/backstage/backstage/blob/master/packages/catalog-model/examples/acme-corp.yaml
@@ -184,7 +184,7 @@ data:
         - type: 'config'
           instances:
             - name: in-cluster
-              url: https://argocd.cnoe.localtest.me:8443
+              url: https://cnoe.localtest.me:8443/argocd
               username: admin
               password: ${ARGOCD_ADMIN_PASSWORD}
     argoWorkflows:
@@ -234,6 +234,8 @@ kind: Deployment
 metadata:
   name: backstage
   namespace: backstage
+  annotations:
+    argocd.argoproj.io/sync-wave: "10"
 spec:
   replicas: 1
   selector:
@@ -362,14 +364,14 @@ spec:
     template:
       engineVersion: v2
       data:
-        BACKSTAGE_FRONTEND_URL: https://backstage.cnoe.localtest.me:8443
+        BACKSTAGE_FRONTEND_URL: https://cnoe.localtest.me:8443/backstage
         POSTGRES_HOST: postgresql.backstage.svc.cluster.local
         POSTGRES_PORT: '5432'
         POSTGRES_DB: backstage
         POSTGRES_USER: backstage
         POSTGRES_PASSWORD: "{{.POSTGRES_PASSWORD}}"
-        ARGO_WORKFLOWS_URL: https://argo.cnoe.localtest.me:8443
-        KEYCLOAK_NAME_METADATA: https://keycloak.cnoe.localtest.me:8443/realms/cnoe/.well-known/openid-configuration
+        ARGO_WORKFLOWS_URL: https://cnoe.localtest.me:8443/argo-workflows
+        KEYCLOAK_NAME_METADATA: https://cnoe.localtest.me:8443/keycloak/realms/cnoe/.well-known/openid-configuration
         KEYCLOAK_CLIENT_SECRET: "{{.BACKSTAGE_CLIENT_SECRET}}"
         ARGOCD_AUTH_TOKEN: "argocd.token={{.ARGOCD_SESSION_TOKEN}}"
         ARGO_CD_URL: 'https://argocd-server.argocd.svc.cluster.local/api/v1/'
@@ -422,7 +424,17 @@ metadata:
 spec:
   ingressClassName: "nginx"
   rules:
-    - host: backstage.cnoe.localtest.me
+    - host: localhost
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: backstage
+                port:
+                  name: http
+    - host: cnoe.localtest.me
       http:
         paths:
           - path: /

examples/ref-implementation/codespaces.md

@@ -0,0 +1,69 @@
+## Running idpbuilder in Codespaces in Browser
+
+**_NOTE:_**: __Steps described below applies to running this implementation in Codespaces in **web browsers** (e.g. Firefox and Chrome).
+If you are using Codespaces with GitHub CLI, steps described here do not apply to you.__ 
+
+
+Let's create an instance of Codespaces.
+
+![img.png](images/codespaces-create.png)
+
+It may take a few minutes for it to be ready. Once it's ready, you can either get the latest release of idpbuilder or build from the main branch.
+Build the idpbuilder binary. 
+
+
+- Get the latest release:
+  ```bash
+  version=$(curl -Ls -o /dev/null -w %{url_effective} https://github.com/cnoe-io/idpbuilder/releases/latest)
+  version=${version##*/}
+  wget https://github.com/cnoe-io/idpbuilder/releases/download/${version}/idpbuilder-linux-amd64.tar.gz
+  tar xzf idpbuilder-linux-amd64.tar.gz
+  sudo mv ./idpbuilder /usr/local/bin/
+  ```
+- Alternatively, build from the main branch
+  ```bash
+  make build
+  sudo mv ./idpbuilder /usr/local/bin/
+  ```
+
+Codespaces assigns random hostname to your specific instance. You need to make sure they are reflected correctly.
+Instance host name is available as an environment variable (`CODESPACE_NAME`). Let's use it to setup our host names. 
+Run the following commands to update host name and ports. Port is set to 443 because this is the port used by the browser to access your instance.
+
+
+```bash
+cd examples/ref-implementation 
+
+./replace.sh ${CODESPACE_NAME}-8080.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN} 443
+
+cd - 
+```
+
+Now you are ready to run idpbuilder with reference implementation.
+
+```bash
+idpbuilder create --protocol http  \
+  --host ${CODESPACE_NAME}-8080.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN} \
+  --port 8080 --use-path-routing --package-dir examples/ref-implementation
+```
+Once idpbuilder finishes bootstrapping, you should have port 8080 forward in the port tab within Codespaces.
+
+![](images/port.png)
+
+You may get a 404 page after clicking the port 8080 forwarded address. This is completely normal because Backstage may not be ready yet. 
+Give it a few more minutes and it should redirect you to a Backstage page.
+
+If you'd like to track progress of deployment, go to `/argocd` path and login with your ArgoCD credentials. 
+For example run this command to get the URL for Argo CD:
+```bash
+echo https://${CODESPACE_NAME}-8080.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN}/argocd
+```
+
+From here on, you can follow the instructions in the [README](./README.md) file. The only difference is that the URL to access UIs is given by:
+
+```echo
+echo https://${CODESPACE_NAME}-8080.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN}
+```
+
+For example, if you need to access Argo Workflows UI, instead of going to `https://cnoe.localtest.me:8443/argo`, 
+you go to `https://${CODESPACE_NAME}-8080.${GITHUB_CODESPACES_PORT_FORWARDING_DOMAIN}/argo`

examples/ref-implementation/coredns/manifests/cm-coredns.yaml

@@ -1,5 +1,6 @@
-# the only purpose of this is to resolve `keycloak.cnoe.localtest.me` to a cluster IP
-# normally, `keycloak.cnoe.localtest.me` resolves to 127.0.0.1 and thus oidc endpoint configurations cannot be obtained.
+# the only purpose of this is to resolve external DNS entries such as `redesigned-bassoon-r4jjwpvv99vhx9gp-8080.app.github.dev` to a cluster IP
+# normally, `redesigned-bassoon-r4jjwpvv99vhx9gp-8080.app.github.dev` resolves to 127.0.0.1 and thus oidc endpoint configurations cannot be obtained.
+# in addition, we need to ensure traffic do not go out of cluster when not necessary.
 apiVersion: v1
 kind: ConfigMap
 metadata:
@@ -13,10 +14,9 @@ data:
            lameduck 5s
         }
         ready
-        rewrite name keycloak.cnoe.localtest.me ingress-nginx-controller.ingress-nginx.svc.cluster.local
-        rewrite name gitea.cnoe.localtest.me ingress-nginx-controller.ingress-nginx.svc.cluster.local
-        rewrite name argocd.cnoe.localtest.me ingress-nginx-controller.ingress-nginx.svc.cluster.local
-        rewrite name argo.cnoe.localtest.me ingress-nginx-controller.ingress-nginx.svc.cluster.local
+    
+        rewrite name cnoe.localtest.me ingress-nginx-controller.ingress-nginx.svc.cluster.local
+    
         kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa