vSphere cluster upgrade failure : unrecognized feature gate: KubeletCredentialProviders

[saiteja313]

What happened:

Upgrading EKS Anywhere cluster from 1.27 to 1.28. One version at a time.

1. Took ETCD Cluster backup
2. EKSA command line version 0.20.x
3. executed eksa upgrade command. upgrade have scaled ETCD clusters from 3 to 6. Added two new controlplane (CP) nodes. Both CP nodes got stuck in NotReady status
4. Checking the kubelet logs on the CP node, kubelet shows following error,

 kubelet[280524]: E1107 17:25:55.770885  280524 run.go:74] "command failed" err="failed to set feature gates from initial flags-based config: unrecognized feature gate: KubeletCredentialProviders"

5. vi /etc/sysconfig/kubelet file, remove the --feature-gates=KubeletCredentialsProviders=true flag, save the file.
6. kubelet gets auto restarted and joins the Kubernetes cluster
7. Repeat the process for all CP and worker nodes to complete the upgrade.
8. Checking the package controller, packager controller version is outdated with old version.

What you expected to happen:

• EKSA should not add KubeletCredentialsProviders feature gate to kubelet configuration.

How to reproduce it (as minimally and precisely as possible):

• upgrade cluster from 1.27 to 1.28 with old version of packages

Anything else we need to know?:

Environment:

• EKS Anywhere Release: 0.20.8