graphana operator cannot create events in other namespaces

[eguzki]

The grafana operator tries to create events and link to grafana objects, but the clusterrole does not have required roles to create events.

The error logged is:

E0310 17:13:32.235287       1 event.go:240] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"prometheus-exporter-redis.15fb00a73cfa01d8", GenerateName:"", Namespace:"test-app", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"GrafanaDashboard", Namespace:"test-app", Name:"prometheus-exporter-redis", UID:"77eabff8-62f2-11ea-aa7a-12d443f7f383", APIVersion:"integreatly.org/v1alpha1", ResourceVersion:"5335410", FieldPath:""}, Reason:"Success", Message:"dashboard test-app/prometheus-exporter-redis successfully submitted", Source:v1.EventSource{Component:"controller_grafanadashboard", Host:""}, FirstTimestamp:v1.Time{Time:time.Time{wall:0xbf92108f0de9a9d8, ext:538230446197802, loc:(*time.Location)(0x207ad60)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0xbf92108f0de9a9d8, ext:538230446197802, loc:(*time.Location)(0x207ad60)}}, Count:1, Type:"Normal", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'events is forbidden: User "system:serviceaccount:application-monitoring:grafana-operator" cannot create resource "events" in API group "" in the namespace "test-app"' (will not retry!)

[davidkirwan]

@eguzki can you describe how you deployed the AMO stack ?

The Role which should be bound to this account system:serviceaccount:application-monitoring:grafana-operator is https://github.com/integr8ly/application-monitoring-operator/blob/master/templates/grafana-operator-role.yaml#L14 which does have permissions "*" for events.

Do we need to allow a clusterrole (https://github.com/integr8ly/application-monitoring-operator/blob/master/deploy/cluster-roles/grafana-operator-clusterrole.yaml) to do create events in other namespaces @david-martin @pb82 ?

[eguzki]

The grafana operator tries to create events in another namespace, hence the permissions should be added in the clusterrole (clusterbindingroles cannot be created using local roles)

[pb82]

I think this is due to https://github.com/integr8ly/grafana-operator/blob/master/pkg/controller/grafanadashboard/dashboard_controller.go#L281

dashboard errors are recorded in the namespace of the dashboard CR (because we have one dashboard controller per namespace)

[davidkirwan]

If we give the clusterrole permission to create events, these will show in the event log right? So would be nice to have imo anyway.