Add backoff handling for watcher and Controller

[clux]

• backoff_watch helper from teo Reimplement watcher backoff as FSM #720
• ResetTimerBackoff helper from teo Add Backoff wrapper that implements client-go's reset timer behaviour #729
• Observer abstraction around watcher + backoff_watch + ResetTimerBackoff ignored for now
• default_backoff following client-go defaults
• requires teo's pr of backoff: Implement Backoff for Box<B: Backoff> ihrwein/backoff#49 (cargo deny blocks git pins)

introduces a dependency on backoff 0.4.0 for parts of this.

old notes, no longer relevant not hiding errors

initial experiment with watcher + backoff. injecting a backoff into the main `stream::unfold` of `watcher` and sleeping between steps. It works with `configmapgen_controller` (in say the example where the crd has not been installed); it keeps sleeping until a max time hits (figured we'd set this to something like 10-30minutes) after which the stream ends.

Problems:

• watcher bubbles up the error as it stands - so normal watcher users stop at the first error anyway
• Controller ignores error events from watcher so at some point the Controller becomes dysfunctional
• how to expose the parameter - imo, hide it behind a builder with sensible defaults - e.g. Cache: implement a builder wrapper for reflector #698 .

Solutions:
We could let the timeout grow almost indefinitely (to like an hour or more backoff) so that Controller can always deal with it.
We could also expose a new watcher error type that wraps the original - so that the user can ignore it easily, but not sure, what that interface would look like yet. Maybe this is something a builder can ignore by default in the way it presents the stream - i.e. ignore until the error has been going on for so lonng.

Also, should try to get the magic number baked into the backoff struct (it supports returning None). It didn't work with max_interval, but looks like it should work with https://docs.rs/backoff/0.3.0/backoff/exponential/struct.ExponentialBackoff.html
max_elapsed_time - yup works.

Anyway, will try more later.

[nightkr]

Sorry about the slow reviewing, I've been (and still am) home sick. I'll try to have a closer look when I'm back alive...

[clux]

Sorry about the slow reviewing, I've been (and still am) home sick. I'll try to have a closer look when I'm back alive...

No rush. Hope you get some rest and get to recover!

[codecov-commenter]

Codecov Report

Merging #703 (d02b1ec) into master (155859b) will increase coverage by 0.31%.
The diff coverage is 77.09%.

@@            Coverage Diff             @@
##           master     #703      +/-   ##
==========================================
+ Coverage   70.83%   71.15%   +0.31%     
==========================================
  Files          52       54       +2     
  Lines        3405     3529     +124     
==========================================
+ Hits         2412     2511      +99     
- Misses        993     1018      +25     

Impacted Files	Coverage Δ
kube-client/src/discovery/mod.rs	89.18% <ø> (ø)
kube-core/src/dynamic.rs	66.66% <ø> (ø)
kube-core/src/object.rs	80.00% <ø> (ø)
kube-core/src/params.rs	66.35% <ø> (ø)
kube-core/src/subresource.rs	75.72% <ø> (ø)
kube-runtime/src/controller/mod.rs	0.00% <0.00%> (ø)
kube-runtime/src/utils/mod.rs	17.64% <ø> (ø)
kube-runtime/src/watcher.rs	54.92% <0.00%> (-4.17%)	⬇️
kube/src/lib.rs	87.80% <ø> (ø)
kube-client/src/lib.rs	92.80% <25.00%> (-0.06%)	⬇️
... and 2 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 155859b...d02b1ec. Read the comment docs.

[clux]

One thought on StreamBackoff: because this is effectively a stream analogue of https://docs.rs/backoff/0.4.0/backoff/fn.retry.html , maybe we should allow bailing at bad errors in a simiar way to how backoff.retry defines its error type: https://docs.rs/backoff/0.4.0/backoff/enum.Error.html ?

[nightkr]

Hm, maybe. I'd lean towards KISS for now, and letting @ihrwein decide the future plan in the upstreaming PR (ihrwein/backoff#50).

[clux]

This appears unused now after the Observer cancel. I can remove this?

[nightkr]

Good catch. This was an attempt to be somewhat more ergonomic and point people who were already working in the watcher module towards it (rather than focusing on the implementation details of StreamBackoff itself). We can either drop this, or revert StreamBackoff::new to pub(crate) and fixing the examples to use backoff_watch instead.

I'm fine with either.

[clux]

I removed it for now. Want to focus on the future chain pipeline before teaching people too much new stuff.