Controller-manager restarting after machine reboot

[ejalberti]

Considering a Kubernetes Cluster, after the machine rebooting, the Controller Manager (only) restarts repeatedly until a new deploy of the manager is done. The logs file shows a problem on the leader election, as seen by the logs fragment below:

E0530 12:18:29.937141 1 leaderelection.go:367] Failed to update lock: client rate limiter Wait returned an error: context deadline exceeded {"level":"debug","ts":"2023-05-30T12:18:30Z","logger":"events","msg":"controller-manager-664469c876-9t8xf_31ee84ab-896f-4835-af8f-52598c6b719f stopped leading","type":"Normal","object":{"kind":"Lease","namespace":"intel-power","name":"power-operator-6846766c","uid":"9526a048-3cb0-41a3-96a4-d83320938624","apiVersion":"coordination.k8s.io/v1","resourceVersion":"4690743"},"reason":"LeaderElection"} I0530 12:18:31.031630 1 leaderelection.go:283] failed to renew lease intel-power/power-operator-6846766c: timed out waiting for the condition {"level":"info","ts":"2023-05-30T12:18:34Z","msg":"Stopping and waiting for non leader election runnables"} {"level":"info","ts":"2023-05-30T12:18:35Z","msg":"Stopping and waiting for leader election runnables"} {"level":"info","ts":"2023-05-30T12:18:36Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"powerconfig","controllerGroup":"power.intel.com","controllerKind":"PowerConfig"} {"level":"error","ts":"2023-05-30T12:18:34Z","logger":"setup","msg":"problem running manager","error":"leader election lost","stacktrace":"main.main\n\t/workspace/main.go:98\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

To reproduce:

1. Deploy Kubernetes Power Manager normally;
2. Observe if all pods are running correctly;
3. Reboot the machine using "sudo reboot";
4. After the reboot, the controller-manager will present problems with the leader election;
5. A new deployment of the controller-manager normalizes the operation.

[adorney99]

Hi @ejalberti, thanks for flagging this. Might take some time to replicate and get to the bottom of it but we'll look into this

[adorney99]

Hey @ejalberti sorry this took so long to get back to, testing host restarts can be a bit awkward on some of our setups. I've tried a couple scenarios with container restarts and full system restarts and haven't been able to replicate this. I've done some digging online and it looks like this issue stems from either constrained node resources or a high number of api calls causing a rate limit. Any chance you're working on a cluster with a large number of nodes? Since I can't replicate this I was wondering if you could try tweaking the following lines , rebuilding the images and deploying them to see if that resolves the issue.

	renewDeadline := time.Second * time.Duration(20)
	leaseDuration := time.Second * time.Duration(30)
	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		Scheme:             scheme,
		MetricsBindAddress: metricsAddr,
		Port:               9443,
		LeaderElection:     enableLeaderElection,
		LeaderElectionID:   "power-operator-6846766c",
		RenewDeadline: &renewDeadline,
		LeaseDuration: &leaseDuration,
	})

renewDeadline and leaseDuration determine how long the operator will wait to obtain or renew leader election so making them longer might resolve the issue. If it does it should be easy for us add an extra input parameter to the manager that will let you configure these values from the deployment yaml. If the issue persists we may need to try adding a longer RetryPeriod (default 2 seconds) to prevent throttling. The only other thing I can think of is that on restart for some reason your system has some rogue deployment running that's holding on to the leader election.