Installation and Trial notes (ongoing)

[agardnerIT]

Edit: This thread serves as a list of working notes and things I discover along the way. If things are mentioned here, they're not necessarily bugs or issues but more things I'm working around.

Here is my first attempt, as an Argo newbie so very much a trial and error exercise!

Initial state

• Keptn 0.12.7.
• Brand new ArgoCD installation

My shipyard:

apiVersion: "spec.keptn.sh/0.2.2"
kind: "Shipyard"
metadata:
  name: "argocd-demo-shipyard"
spec:
  stages:
    - name: "dev"
      sequences:
        - name: "artifact-delivery"
          tasks:
            - name: "deliver"

keptn create project argocd-podtato-head \
-s shipyard.yaml \
-r https://github.com/agardnerIT/my-repo \
-u agardnerit \
-t ghp_**********
keptn create service podtatohead --project argocd-podtato-head

# label the argo app
kubectl -n argocd get apps
kubectl -n argocd label app podtato-head keptnProject=argocd-podtato-head
kubectl -n argocd label app podtato-head keptnService=podtatohead
kubectl -n argocd label app podtato-head stage=dev

# Install notification service
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-notifications/release-1.0/catalog/install.yaml

# clone this repo
git clone https://github.com/keptn-sandbox/keptn-argocd-integration

# Install notifications templates from our repo
kubectl apply -n argocd -f ~/keptn-argocd-integration/argo-config/argocd-notifications-cm.yaml

# Don't know what this does but apply it
kubectl apply -n argocd -f ~/keptn-argocd-integration/argo-config/argocd-cm.yaml

export KEPTN_API_TOKEN=A*******9
export KEPTN_FQDN=mykeptn.com
kubectl apply -n argocd -f - << EOF
apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
stringData:
  keptn-x-token: $KEPTN_API_TOKEN
  keptn_api_fqdn: $KEPTN_FQDN
type: Opaque
EOF

# subscribe to something. dont really know what this does
kubectl patch app podtato-head -n argocd -p '{"metadata": {"annotations": {"notifications.argoproj.io/subscribe.on-out-of-sync-status":""}}}' --type merge

So far, no errors.

1. Make a change to my app
2. Sync Argo. ArgoCD goes OutOfSync in the UI

But I don't get the sequence triggered event.

[thschue]

Did you also create the proper notification on the ArgoCD side and label the application?

[agardnerIT]

Hi Thomas, what you see above is precisely what I did, no more, no less. I'm fully expecting that this is me missing things (because I don't know argo).
I'm following the docs here: https://argocd-notifications.readthedocs.io/en/stable/

Edit:
I'm a little bit further now. Rather than annotating with "notifications.argoproj.io/subscribe.on-out-of-sync-status":""
I used: "notifications.argoproj.io/subscribe.on-out-of-sync-status.keptn":"" (no idea why argocd doesn't like on-out-of-sync-status.webhook.keptn).

time="2022-05-06T06:27:32Z" level=info msg="Trigger on-out-of-sync-status result: [{[0].I5***M  [keptn-outofsync] true}]" app=argocd/podtato-head
time="2022-05-06T06:27:32Z" level=info msg="Sending notification about condition 'on-out-of-sync-status.[0].I54pp8hftvuoE3L8K7Rve4eD8zM' to '{keptn }'" app=argocd/podtato-head
time="2022-05-06T06:27:32Z" level=error msg="Failed to notify recipient {keptn } defined in app argocd/podtato-head: parse \"https://{{ keptn.mysite.com }}/api/v1/event\": invalid character \"{\" in host name" app=argocd/podtato-head

[thschue]

Created a branch with some updated config in a branch ... I found out that there were some whitespaces in the placeholders which have to be removed. Furthermore, you need the keptn-api-hostname and the keptn-api-token in the notification secret ...

[agardnerIT]

Notes:

If argo is installed on your own cluster, most likely a proper SSL certificate will not be available. In which case, when configuring the webhook (if done via the UI) you will need to manually edit the YAML in your upstream and add the --insecure flag:

curl --header 'Authorization: Bearer {{.env.secret_argodetails_argotoken}}' ....

Becomes:

curl --insecure --header 'Authorization: Bearer {{.env.secret_argodetails_argotoken}}' ....

Generating an Argo Bearer Token

I want a new role that is only allowed to sync my app. Following: https://argo-cd.readthedocs.io/en/stable/operator-manual/rbac/

kubectl apply -n argocd -f - << EOF
apiVersion: v1
kind: Secret
metadata:
  name: argocd-rbac-cm
stringData:
  policy.default: role:readonly
  policy.csv: |
    p, role:keptn, applications, sync, *, allow
type: Opaque
EOF

Now following: https://argoproj.github.io/argo-workflows/access-token/

kubectl -n argocd create role keptn --verb=sync --resource=workflows.argoproj.io
kubectl -n argocd create sa keptn
kubectl -n argocd create rolebinding keptn --role=keptn --serviceaccount=argo:keptn
SECRET=$(kubectl -n argocd get sa keptn -o=jsonpath='{.secrets[0].name}')
ARGO_TOKEN="$(kubectl -n argocd get secret $SECRET -o=jsonpath='{.data.token}')"
echo -n $ARGO_TOKEN | base64 --decode

# ARGO_TOKEN looks like: eyJh............vsg

Then I create a secret in Keptn and store the value as argo_token

Webhook has a header:

Authorization: Bearer {{.env.secret_argodetails_argotoken}}

Unfortunately the following response is received when issuing a curl:

{"error":"invalid session: SSO is not configured","code":16,"message":"invalid session: SSO is not configured"}

Going into Argo settings UI and generating a token for the keptn account via the UI results in a curl response:

{"error":"permission denied: applications, sync, default/podtato-head, sub: keptn, iat: 2022-05-09T00:38:00Z","code":7,"message":"permission denied: applications, sync, default/podtato-head, sub: keptn, iat: 2022-05-09T00:38:00Z"}

@bradmccoydev any ideas?

@thschue There's a regression on fixes_demo. argocd-notifications-cm.yaml pushes appname as a label whereas the webhook is looking for servicename

@thschue I received this invite to collaborate on this repo but Github shows "invalid invite". Could you please resend?

[agardnerIT]

First get the user session token (I don't know how we generate one for longer than a session):

curl -k https://argo-url/api/v1/session -d "{\"username\":\"admin\",\"password\":\"your-password\"}"

Returns a JSON document:

{"token":"ey*********"}

Use this token value below...

This curl works for argo 2.3.3:

export ARGOCD_TOKEN=ey*******
curl -v --insecure --request POST https://argo-url/api/v1/applications/podtato-head/sync \
-H "Authorization: Bearer $ARGOCD_TOKEN" \
-H "Content-Type: application/json" \
--data "{\"infos\": [{\"name\":\"triggeredid\",\"value\":\"d555f3f4-8650-4151-b0be-5ffb2108fe89\"},{\"name\":\"shkeptncontext\",\"value\":\"b40ff110-005c-46a1-83a0-b445e9a115c4\"}], \"revision\":\"HEAD\",\"prune\":true,\"dryRun\":false,\"strategy\":{\"hook\":{\"force\":false}},\"resources\":null,\"syncOptions\":{\"items\":[]}}"

[agardnerIT]

Working! 🎉

[agardnerIT]

Open question: The argo API token is a session token, which expires after 24hrs. How do we get a more permanent token?

[agardnerIT]

stage or keptnStage? Currently the JSON payload has keptnProject, keptnService and stage. I'm wondering whether that should really be keptnStage or is the use of stage deliberate?

[agardnerIT]

@thschue / @bradmccoydev strengths and weaknesses of using a pre / post sync job to trigger a keptn sequence? Discuss :)

https://github.com/argoproj/argocd-example-apps/blob/master/pre-post-sync/post-sync-job.yaml

[jkleinlercher]

Open question: The argo API token is a session token, which expires after 24hrs. How do we get a more permanent token?

You can create project JWT tokens for a specific role in this project. E.g. create a „sync“ role which only has permissions for syncing an application for this project and then create a JWT token which doesn’t expire for this role.

Some more details are mentioned in https://argo-cd.readthedocs.io/en/stable/operator-manual/security/#authentication in option 3.

[thschue]

stage or keptnStage? Currently the JSON payload has keptnProject, keptnService and stage. I'm wondering whether that should really be keptnStage or is the use of stage deliberate?

True, this is a bit inconsistent ... Think we should change this to keptnStage, or how do you think?

[agardnerIT]

keptnStage sounds logical to me.