Multus Network Plugin Role causes cluster-upgrade.yaml to fail if limited on non-control-plane nodes

[networkhell]

What happened?

ansible-playbook -ask-become-pass -i inventory/<redacted>.yml kubespray/upgrade-cluster.yml --limit=k8s-node01vt
fails with error:

2024-08-07 15:13:06,021 p=72979 u=*** n=ansible | fatal: [k8s-node01vt -> {{ groups['kube_control_plane'][0] }}]: FAILED! =>- msg: |- Error in jmespath.search in json_query filter plugin: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'multus_manifest_2'

What did you expect to happen?

Upgrade does not fail when limited to non-control-plane nodes

How can we reproduce it (as minimally and precisely as possible)?

Run a cluster upgrade with Branch release-2.25 limited to a worker node (after successfully upgrading the control plane)

OS

Linux 6.1.0-23-amd64 x86_64 PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/"

Version of Ansible

ansible [core 2.16.8]
...
jinja version = 3.1.4
libyaml = True

Version of Python

Python 3.11.2

Version of Kubespray (commit)

6ee9464

Network plugin used

calico

Full inventory with variables

I can provide this later but there is a lot of secret data in there so I first have to clean that out.

Command used to invoke ansible

ansible-playbook -ask-become-pass -i inventory/.yml kubespray/upgrade-cluster.yml --limit=k8s-node01vt

Output of ansible run

2024-08-07 15:13:06,021 p=72979 u=*** n=ansible | fatal: [k8s-node01vt -> {{ groups['kube_control_plane'][0] }}]: FAILED! =>- msg: |- Error in jmespath.search in json_query filter plugin: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'multus_manifest_2'

Anything else we need to know

I did not yet have time to dig into this issue but I guess the loop control var of the failing task is based on a variable that is constructed on a control_plane node and is just undefined in case that the play is limited to a worker node.

[aleMaqueda]

Try with --skip-tags=multus
Ref: #9790