[nvca] multi-node deployment can pin to NVLink partition without enough free nodes, follower pod stuck Pending

[estroz]

Describe the bug

A multi-node NVCF deployment using kai-scheduler can partially schedule into an NVLink partition that cannot satisfy the full worker group. One pod schedules, while the follower pod remains Pending because kai-scheduler reports no nodes satisfy the affinity constraints.

This makes multi-node NVLink-optimized deployments unreliable. Redeploy often works only because placement lands on a different partition, so the current workaround is nondeterministic.

Steps or code to reproduce bug

1. Partially saturate a GPU clique on an NVLink-optimized cluster with kai-scheduler enabled, with another clique being empty
2. Deploy multi-node function onto the cluster with dra.nvcf.nvidia.io/required-nvlink-domain-index: "0" annotations on all Pod metadata that should be in a single clique
3. Pods are scheduled via bin-packing into most saturated clique, then are stuck in Pending

Expected behavior

NVCF/NVCA should perform gang-aware placement at the NVLink partition/clique level. It should either atomically place all required nodes onto a partition with enough free nodes, or queue/fail cleanly before partially binding pods.

Additional context

---

By submitting this issue, you agree to follow our code of conduct and our contributing guidelines.

[estroz]

An interim fix specifically for Pods within a single StatefulSet is to create a cluster-wide kai scheduler Topology resource:

apiVersion: kai.scheduler/v1alpha1
kind: Topology
metadata:
  name: nvcf-nvlink-topology
spec:
  levels:
  - nodeLabel: "nvidia.com/gpu.clique"
  - nodeLabel: "kubernetes.io/hostname"

then have function owners set these annotations on their StatefulSet's:

metadata:
  annotations:
    kai.scheduler/topology: "nvcf-nvlink-topology"
    kai.scheduler/topology-required-placement: "nvidia.com/gpu.clique"

Those annotations propagate to the topology constraint of the PodGroup that kai-scheduler creates for the StatefulSet. Kai will not bind any pod in the StatefulSet to a node until all replicas can land within a single NVLink clique.

The above will not work for Deployments because kai forces each pod replica of a Deployment into its own PodGroup; there is no way to override this behavior. The long-term fix for Deployments and any other type for which a function needs gang-scheduling behavior, in addition to creating the Topology mentioned above is to:

1. Direct function owners to use a Dynamo Operator DynamoGraphDeployment, which automatically hooks into Grove/Kai for gang scheduling
2. Expose the Kai PodGroup type for use in Helm chart functions, for basic gang scheduling without Dynamo
3. Expose the Grove PodCliqueSet type for use in Helm chart functions, for more advanced and unified control of gang scheduling without Dynamo

[estroz]

NVCA needs Karta definitions for KAI/Grove top-level types too