500 Internal Error in Grafana and Prometheus UI with custom CA

[biloussi]

Hi,
I have deployed the application-monitoring-operator to a OpenShift 4.3 cluster built on-prem. All components seem to work fine, without errors. However, I get error "500 Internal Error" right after I try to login to Grafana UI or to Prometheus UI. We use internally signed certificates and custom Certification Authority in our OpenShift environment.

Steps to reproduce:

1. I install application-monitoring-operator
   git clone https://github.com/integr8ly/application-monitoring-operator.git
   make cluster/install
   and wait for completion. Components install without errors.
2. I go to routes and click a route to Grafana.
3. I click "Login with OpenShift" and get redirected to the OpenShift login, then I insert my username and password, I authorize the application to read my information, I get "500 Internal Error" message on the page.
4. Try steps 2 and 3 for Prometheus UI and also get "500 Internal Error".

Environment info:
oc get pods
NAME READY STATUS RESTARTS AGE
alertmanager-application-monitoring-0 3/3 Running 0 59m
application-monitoring-operator-5bc879f697-mcglx 1/1 Running 0 60m
grafana-deployment-58746b4f54-hr4xs 2/2 Running 0 9m32s
grafana-operator-66497b6fc6-q9lhc 1/1 Running 0 59m
prometheus-application-monitoring-0 5/5 Running 1 59m
prometheus-operator-76b4dfbb68-r7k95 1/1 Running 0 59m

Logs for grafana-proxy container:

2020/06/01 10:18:45 provider.go:117: Defaulting client-id to system:serviceaccount:application-monitoring:grafana-serviceaccount
2020/06/01 10:18:45 provider.go:122: Defaulting client-secret to service account token /var/run/secrets/kubernetes.io/serviceaccount/token
2020/06/01 10:18:45 provider.go:310: Delegation of authentication and authorization to OpenShift is enabled for bearer tokens and client certificates.
2020/06/01 10:18:45 oauthproxy.go:200: mapping path "/" => upstream "http://localhost:3000/"
2020/06/01 10:18:45 oauthproxy.go:221: compiled skip-auth-regex => "^/metrics"
2020/06/01 10:18:45 oauthproxy.go:227: OAuthProxy configured for Client ID: system:serviceaccount:application-monitoring:grafana-serviceaccount
2020/06/01 10:18:45 oauthproxy.go:237: Cookie settings: name:_oauth_proxy secure(https):true httponly:true expiry:168h0m0s domain: refresh:disabled
2020/06/01 10:18:45 http.go:106: HTTPS: listening on [::]:9091
2020/06/01 10:21:25 provider.go:392: authorizer reason:
2020/06/01 10:21:28 provider.go:573: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2020/06/01 10:21:28 provider.go:613: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
"issuer": "https://oauth-openshift.apps.os4-test.lab.local",
"authorization_endpoint": "https://oauth-openshift.apps.os4-test.lab.local/oauth/authorize",
"token_endpoint": "https://oauth-openshift.apps.os4-test.lab.local/oauth/token",
"scopes_supported": [
"user:check-access",
"user:full",
"user:info",
"user:list-projects",
"user:list-scoped-projects"
],
"response_types_supported": [
"code",
"token"
],
"grant_types_supported": [
"authorization_code",
"implicit"
],
"code_challenge_methods_supported": [
"plain",
"S256"
]
}
2020/06/01 10:21:38 provider.go:573: Performing OAuth discovery against https://172.30.0.1/.well-known/oauth-authorization-server
2020/06/01 10:21:38 provider.go:613: 200 GET https://172.30.0.1/.well-known/oauth-authorization-server {
"issuer": "https://oauth-openshift.apps.os4-test.lab.local",
"authorization_endpoint": "https://oauth-openshift.apps.os4-test.lab.local/oauth/authorize",
"token_endpoint": "https://oauth-openshift.apps.os4-test.lab.local/oauth/token",
"scopes_supported": [
"user:check-access",
"user:full",
"user:info",
"user:list-projects",
"user:list-scoped-projects"
],
"response_types_supported": [
"code",
"token"
],
"grant_types_supported": [
"authorization_code",
"implicit"
],
"code_challenge_methods_supported": [
"plain",
"S256"
]
}
2020/06/01 10:21:38 oauthproxy.go:645: error redeeming code (client:10.254.3.1:47476): Post https://oauth-openshift.apps.os4-test.lab.local/oauth/token: x509: certificate signed by unknown authority
2020/06/01 10:21:38 oauthproxy.go:438: ErrorPage 500 Internal Error Internal Error

So I understand that the issue is with internally signed certificates.

I added two configmaps with our root certificates and labels:
config.openshift.io/inject-trusted-cabundle: 'true'

I tried to add two sections to grafana-deployment:
volumeMounts:
- name: grafana-trusted-ca-bundle
readOnly: true
mountPath: /etc/pki/ca-trust/extracted/pem/
....
volumes:
- name: grafana-trusted-ca-bundle
configMap:
name: grafana-trusted-ca-bundle
items:
- key: ca-bundle.crt
path: tls-ca-bundle.pem
defaultMode: 420
optional: true

I also tried to edit Grafana and Prometheus instances of CRDs. However, in all cases the configuration is ovewritten by operator, which is expected behaviour, I believe.

Please advice, what is the correct flow of adding trusted-ca-bundle with this operator?
Thank you!
Sergiy