Expose metrics about internal operations during a reconciler run

[aryan9600]

At the moment, we only expose the readiness of the object and the duration of the reconciler run. It'd be helpful for both users and developers if we expose metrics about internal ops such as garbage collection, remote operations (git fetch, helm pull, etc). This would enable us to analyze and optimize the operations and give users a clearer picture of how the controller is working.

[cwyl02]

hi @aryan9600, I am interested to work on this if ok

[cwyl02]

hi @aryan9600 finally get some time to look deeper into this. And I see we have a cacheRecorder. Should we try to reuse that pattern where it fits?
Also, I guess for every metric we come up with, namespace is always a desirable dimension?

[aryan9600]

Yes, using that pattern seems like a good idea for now, we can think about expanding the Recorder in fluxcd/pkg/runtime/metrics later.
I'd say the namespace and name labels are desirable for all metrics. Developers and users should be able to see stuff like the avg. duration of git checkout or a helm pull for each source object, which would enable them to have a clearer picture of the controller's behaviour.