Improve resilience when reconciling OIDC resources that require remote JWKS

[dimityrmirchev]

What would you like to be added:
The logic that immediately removes the authentication handler when a request to retrieve a remote JWKS fails can be improved to better handle intermittent failures.

oidc-webhook-authenticator/controllers/authentication/openidconnect_controller.go

Lines 113 to 120 in d272537

	keySet, err = remoteKeySet(ctx, config.Spec.IssuerURL, config.Spec.CABundle)
	if err != nil {
	log.Error(err, "Invalid remote JWKS KeySet")
	r.deleteHandler(req.Name)
	return reconcile.Result{RequeueAfter: 10 * time.Second}, nil
	}

It would be better to implement a retry mechanism that tries the request a couple of times before finally removing the auth handler from store. This will improve stability of the service. In addition to that the default resync period can be increased from its current default value of 10min.

oidc-webhook-authenticator/cmd/oidc-webhook-authenticator/app/options/options.go

Line 112 in d272537

fs.DurationVar(&s.Duration, "resync-period", time.Minute*10, "resync period")

Why is this needed:
An intermittent failure, i.e. DNS or network issue can cause an authentication handler to be removed from store causing request to the kube-apiserver to be unauthorized.

[dimityrmirchev]

/lifecycle frozen