fix(aws-eks): Controller CRD-read RBAC independent of chart ordering + pin k8s-firewall

[cmchenr]

Problem

On freshly deployed aws-eks-multicluster clusters, the Aviatrix Controller logged:

firewallpolicies.networking.aviatrix.com is forbidden: User "...aviatrix-role-app/..."
cannot list resource "firewallpolicies" in API group "networking.aviatrix.com" at the cluster scope
webgrouppolicies.networking.aviatrix.com is forbidden: ...

and the onboarded clusters showed partial status, even though the EKS access entry correctly maps the Controller role into the avx-controller group.

Root cause: layer ordering, not config

aviatrix_kubernetes_cluster (onboarding) and the access entry live in the cluster layer. The avx-controller ClusterRoleBinding that actually grants networking.aviatrix.com read is created by the k8s-firewall Helm chart in the later nodes layer. In a clean multi-layer deploy the Controller's first CRD-read poll fires ~1 min before that binding exists → forbidden, and the status sticks. This is structural, and is the only reason the previously-working avx-controller access-entry fix appeared to regress in multi-layer deploys.

Verified on the live clusters: access entry groups [avx-controller, view-nodes] ✅, chart binding present ✅, can-i list firewallpolicies --as-group=avx-controller → yes ✅ — i.e. RBAC was correct, just established too late for the first poll.

Fix

1. Decouple the grant from chart timing. The shared modules/aws-eks-cluster now creates a Terraform-managed avx-controller-crd-reader ClusterRole + binding in the cluster layer (mirroring the existing view-nodes pattern), granting the avx-controller group get/list/watch on networking.aviatrix.com/*. The grant is live at onboarding, independent of when/whether the chart lands. RBAC rules may reference CRD kinds before the CRDs are registered, so this is safe. Named distinctly from the chart's own avx-controller objects to avoid Helm ownership collisions; the duplicate grant to the same group is harmless. Fixes both aws-eks-singlecluster and aws-eks-multicluster (shared module).

2. Pin the chart. k8s-firewall was installed unpinned (silently tracking latest). Added a k8s_firewall_chart_version variable (default 9.0.0, the current release) matching the existing *_chart_version pattern, across both blueprints' nodes layers.

Validation

• Applied the module change live with -replace on aviatrix_kubernetes_cluster for both frontend and backend (re-onboards → fresh poll with RBAC already present). Both applied clean.
• Post-apply on both clusters: TF-managed avx-controller-crd-reader binding present; can-i list firewallpolicies/webgrouppolicies (group avx-controller) → yes.
• terraform validate passes on the nodes layer; terraform fmt clean.

Notes

• No DCF policy CRs are deployed in the lab (dcf-crd/ manifests are manual examples), so post-fix the Controller lists the CRDs successfully but returns empty — status clears with nothing to sync. Deploy a sample FirewallPolicy CR to see rulesets populate.
• Unrelated in-progress azure-aks-multicluster working-tree changes were intentionally left out of this PR.

🤖 Generated with Claude Code