Add backoff to watcher reconnection

[nightkr]

Currently watcher will always keep a reconnection attempt in-flight when polled (and there is no active connection already). That's a bit excessive, and pretty unkind to the apiserver.

This isn't too bad for watcher itself since you can add a delay on the consumer side instead, but we should at least add something for Controller...

[nightkr]

https://canary.discord.com/channels/500028886025895936/685161813675212899/861276713295675424

[clux]

Pasting comments here for people off discord:

pavelp: Out of pure curiosity, I noticed kube calls K8S API with high frequency when it's not available (e.g. local cluster down). Is this the usual frequency excercised by kuibe-rs to call REST API watches ? Is there any recommended ? I was fiddling with K8S API this weekend and did some throttling - I only ask twice a second maximum.
https://kubernetes.io/docs/reference/using-api/api-concepts/#efficient-detection-of-changes

teozkr: that said.. there are still some missing pieces (for example: exponential backoff, backoff for kube watchers, and so on)

Do we think the solution here is to try to bake an exponential backoff watcher or the the step_trampolined (inner state machinery of watcher) fn?

(btw one way I have gotten around on this in the controller-rs example is to try to get the CRD first before trying to watch it - as that causes the same error - but that only works for controllers who forgot to add the CRD)

[nightkr]

I think it would make sense to just have watcher (and applier/Controller in turn) take an arbitrary Backoff from the user.

[clux]

If that's the case then I think we should consider creating builders for watcher to pass it in (#276), it's already a bit strange to have to disambiguate the import paths runtime::watcher (module) vs runtime::watcher() (fn) or runtime::watcher::watcher (fn)

[clux]

rewording a bit; the watcher import problem is a separate issue, could be fixed by renaming the watcher module to watch,.

I do still think there's room for a Watcher<K>, there are a couple of other naming problems here that i would like to improve separately (module name + flatten helpers/naming (maybe something akin to #296)). I'll open an issue separately.

[nightkr]

To be honest, I don't really see the problem with the import, Rust has been smart enough to do the right thing in all the cases I've tried.

[clux]

Rustdoc needs explicit disambiguation when linking to either module or the fn.
And although that's more of a problem for us, people still have to do a double-take here on the name being re-used in imports:

Generally i just think we should organise it in the way that is less confusing:

-use runtime::{reflector, reflector::Store};
+use runtime::cache::{reflector, Store};

relatively minor point though.

[nightkr]

That makes sense to me.