Skip to content

test-plan.adoc

Latest commit

 

History

History
5792 lines (5426 loc) · 108 KB

test-plan.adoc

File metadata and controls

5792 lines (5426 loc) · 108 KB
Raw

AI Cloud Validation Test Suite

Test Domain Function Description Example Test ID Req ID GH Labels Actor Notes Priority Issues Dependencies Milestone Release Test Cases Status Justification

Foundational Services

Image Registry

Stores and distributes boot images, VM images, containers, and deployment artifacts. Acts as a trusted source for all deployable assets.

GCP Artifact Registry + Images

BOOT01-01

BOOT01

#37

image_registry, offtake

tenant

P0

LimitedEnv

M0

CRUD an custom OS image (iso file) (Upload an ISO file if supported, or remote URL if supported).

published

BOOT01-02

BOOT01

#39

image_registry, offtake

tenant

P0

#39

LimitedEnv

M1

CRUD an OS install configuration (e.g. iPXE config like Carbide)

published

BOOT03-01

BOOT03

#40

offtake

tenant

P0

#40

LimitedEnv

M0

CRUD a custom VM image

published

BOOT03-02

BOOT03

#104

image_registry, offtake

tenant

P0

M3

CRUD (get, list, create, delete) custom OS images for fast boot (raw, qcow2, img, etc) for BM/VM

published

BOOT01-03

BOOT01

#41

bare_metal, image_registry, offtake

tenant

P0

#41

LimitedEnv

M1

Check that an OS image can be installed on a BMaaS system

published

BOOT01-04

BOOT01

#43

image_registry, offtake

tenant

P0

#43

LimitedEnv

M1

Check that an OS install configuration can be installed on a BMaaS system

published

BOOT01-05

BOOT01

#44

image_registry, offtake

tenant

P0

#44

LimitedEnv

M0

Check that a VM image can be installed onto a VMaaS system

published

IMG01-01

IMG01

tenant

P1

M4

Verify that an OS image (iso file) with full NVIDIA support is readily available

approved

IMG02-01

IMG02

tenant

P1

M4

Verify that an OS install configuration (e.g. iPXE config like Carbide) with full NVIDIA support is readily available

approved

IMG03-01

IMG03

tenant

P1

M4

Verify that an VM image with full NVIDIA support is readily available

approved

Key Secret Mgmt

Securely stores and manages cryptographic keys, secrets, and certificates. Supports key rotation

AWS Secrets Manager / KMS

AUTH01-01

AUTH01

tenant

P1

LimitedEnv

M4

CRUD an SSH/Teleport compatible public key for system login

approved

AUTH02-01

AUTH02

#247

iam, vm

tenant

P1

M6

Spin up an instance which uses a specified key

published

AUTH03-01

AUTH03

#248

bare_metal, iam, security, vm

tenant

P1

M5

Access other components via a specified key as possible (SOL, Network devices)

approved

Identity and Asset Mgmt (IAM)

Manages users, service identities, roles, and permissions across tenants and services. Forms the security foundation of the platform.

AWS: IAM

IAM01-01

IAM01

#21

iam

tenant

P0

#21

LimitedEnv

M0

Create user and log in as that user

published

IAM02-01

IAM02

#23

iam

tenant

P0

#23

LimitedEnv

M0

Delete user

published

IAM03-01

IAM03

#341

iam

tenant

P1

LimitedEnv

M5

Validate that a user created can access the API and authorized resources

approved

DDI

core network services for name resolution, IP allocation, and address lifecycle management

GCP Cloud DNS + IPAM

IPAM01-01

IPAM01

#131

network, ssh

tenant

P0

MiniCloud

M4

Check that IP addresses are managed by the platform, and that DHCP is able to provide IP addresses

published

IPAM02-01

IPAM02

#132

network

tenant

P0

MiniCloud

M4

Check that IP addresses are managed sensibly as VPCs are configured (Move to SDN section?)

published

Network Underlay & Mgmt

Brings up the physical network fabric for Day 0, base configuration, and network OS installation. Manages switch OS versions, compliance, and safe upgrades over time. Provides base network that overlays and tenants are built upon.

Netris

NETMGMT01-01

NETMGMT01

operator

P0

MiniCloud

M4

Verify that all switches in the are reachable and healthy (SSH, API, etc)

pending

Resource Database

Resource Inventory for useable resources withing the data center

RESDB01-01

RESDB01

operator

P1

LimitedEnv

M4

Verify that all resources assigned to the cluster are accounted for

pending

Infrastructure and Data Services

Control Plane accessible

Control plane can be accessed

CP03-01

CP03

#26

control_plane

tenant

P0

LimitedEnv

M0

Make sure we can ping the control plane, or get a heart beat/status code

published

CP04-01

CP04

#261

operator

P2

LimitedEnv

M5

Control Plane can be upgraded (measure impact)

approved

CP05-01

CP05

#28

control_plane, iam

tenant

P0

#28

LimitedEnv

M0

Access keys can be created, received, used to log in

published

CP06-01

CP06

#29

control_plane, iam

tenant

P0

#29

LimitedEnv

M0

Access keys can expire or be disabled

published

CP07-01

CP07

#30

control_plane, iam

tenant

P0

#30

LimitedEnv

M0

Create tenants

published

CP08-01

CP08

#31

control_plane, iam

tenant

P0

#31

LimitedEnv

M0

Retrieve list of tenants

published

CP09-01

CP09

#32

control_plane, iam

tenant

P0

LimitedEnv

M0

Retrieve info about individual tenant

published

CP10-01

CP10

#34

control_plane, iam

tenant

P0

#34

LimitedEnv

M0

Delete Tenant

published

CP11-01

CP11

#35

tenant

P0

#35

LimitedEnv

M0

Add user to tenant

published

Hardware ingestion

Makes sure that all hardware under test has been ingested

HWING01-01

HWING01

#133

bare_metal

operator

P1 (check w/ ISVs)

MiniCloud

M6

Makes sure that all hardware under test has been ingested, and matches the provided hardware

published

HWING02-01

HWING02

operator

P1

MiniCloud

M4

Make sure that a device can be removed from the system

approved

Attestation

SEC22-01

SEC22

#315

attestation, bare_metal, offtake, security

operator

P1

MiniCloud

M6

Check that hardware passes nonce check

published

ATTEST01-01

ATTEST01

#243

operator

P1

MiniCloud

M5

Check that OS is approved

approved

SEC22-02

SEC22

#316

offtake

operator

P1

MiniCloud

M6

Check that TPM Configuration can be set per host

published

ATTEST02-01

ATTEST02

#319

operator

P1

MiniCloud

M5

Check that an updated BIOS is installed on all hardware

published

CNP09-01

CNP09

offtake

operator

P1

MiniCloud

M4

Check that the updated firmware is installed

approved

CNP09-02

CNP09

#259

attestation, bare_metal, firmware, offtake, security

operator

P1

M6

Verify all firmware is cryptographically signed and attested during boot

published

SEC22-03

SEC22

offtake

operator

REVIEW

P0

M9

TPM Secure Boot in BIOS

pending

Compute Services: BMaaS

Supports the basic lifecycle of Bare Metal nodes

AWS EC2/Forge

CNP01-01

CNP01

#49

bare_metal, offtake

tenant

P0

#49

LimitedEnv

M1

Create a Bare Metal node

published

CNP01-02

CNP01, CNP04

#51

offtake

tenant

P0

M1

Supports the basic lifecycle of Bare Metal nodes, following the standard CRUD pattern

published

CNP01-03

CNP01

#53

bare_metal, offtake

tenant

P0

#53

LimitedEnv

M1

Delete a Bare Metal node

published

SEC21-01

SEC21

#54

bare_metal, offtake, security

tenant

P0

LimitedEnv

M1

Confirm sanitization on delete - Tenant View

published

SEC21-02

SEC21

#134

bare_metal, offtake, sanitization, security

operator

P0

MiniCloud

M6

Confirm sanitization on delete - Low Level

published

CNP01-04

CNP01

#105

bare_metal, offtake

tenant

P0

MiniCloud

M3

Topology-based placement

published

BMAAS01-01

BMAAS01

tenant

REVIEW

P2

Node attached storage

pending

BMAAS02-01

BMAAS02

tenant

REVIEW

P2

Storage: Config/default/access to local NVME drives

pending

Once created, nodes can be accessed for usage, and have the high level support controls of reboot, reinstall.

AWS EC2/Forge

BMAAS03-01

BMAAS03

#48

bare_metal, ssh

tenant

P0

LimitedEnv

M1

A running node can be accessed via SSH/Teleport for further configuration (Bare Metal)

published

CNP01-05

CNP01

#50

bare_metal, offtake

tenant

P0

M1

A running node can be rebooted in case of issue (Bare Metal)

published

CNP01-06

CNP01

#135

bare_metal, offtake

tenant

P0

MiniCloud

M4

A running node can be power-cycled in case of issue

published

BMAAS04-01

BMAAS04

#58

tenant

P0

LimitedEnv

M1

DEPRECATED: A node can be reinstalled from its configured stock operating system

published

CNP01-07

CNP01

#106

bare_metal, offtake

tenant

MinAPI

P0

MiniCloud

M3

A running node can be powered off (Without being destroyed)

published

BOOT02-01

BOOT02

#107

bare_metal, offtake, ssh

tenant

MinAPI

P0

M3

Support for cloud-init and instance metadata discovery via link-local addresses (e.g. 169.254.169.254) or virtual devices (Bare Metal)

published

CNP01-08

CNP01

#108

bare_metal, offtake

tenant

MinAPI

P0

MiniCloud

M3

A powered off node can be powered back on

published

CNP05-01

CNP05

#112

bare_metal, offtake

tenant

MinAPI

P0

LimitedEnv

M3 .5

Support for user-defined tags/labels and cloud-init metadata on instances. (Bare Metal)

published

Nodes will be able to access expected resources on the local networks

AWS EC2/Forge

BMAAS05-01

BMAAS05

#136

bare_metal, dpu

operator

P0

MiniCloud

M6

Check the health and status of the DPU on instantiation

published

BMAAS06-01

BMAAS06

#60

bare_metal, gpu, network, ssh

tenant

P0

#60

LimitedEnv

M1

Nodes can communicate across ethernet, infiniband, and NVLink (Bare Metal)

published

Observability & Debug

AWS EC2/Forge

BMAAS07-01

BMAAS07

#250

bare_metal, observability

tenant

REVIEW

P1

M5

Check for any per-host status log over time.

published

CNP06-01

CNP06

#109

bare_metal, offtake

tenant

P0

LimitedEnv

M3

Serial console access is required (read-only sufficient, interactive preferred) (Bare Metal)

published

BMAAS08-01

BMAAS08

#61

bare_metal, gpu, ssh

tenant

P0

#61

LimitedEnv

M1

Verify NVIDIA hardware

published

BMAAS09-01

BMAAS09

#63

bare_metal, gpu, ssh, workload

tenant

P0

LimitedEnv

M1

GPU Stress workload (Bare Metal)

published

BMAAS10-01

BMAAS10

#64

bare_metal, gpu, slow, ssh, workload

tenant

P0

LimitedEnv

M1

Nim Inference jobs (Bare Metal)

published

BMAAS11-01

BMAAS11

#65

bare_metal, gpu, ssh, workload

tenant

P0

#65

LimitedEnv

M1

NCCL tests

published

BMAAS12-01

BMAAS12

#66

bare_metal, gpu, ssh, workload

tenant

P0

#66

LimitedEnv

M1

Training workload test

published

CNP06-02

CNP06

#258

bare_metal, offtake, observability

tenant

P0

M5

Verify that serial console output is logged and queryable for at least 1 month of history

published

Stable Identifiers

CNP08-01

CNP08

#197

bare_metal, offtake

tenant

P0

M5

Verify that a BM node’s ID does not change after a reboot

published

CNP08-02

CNP08

bare_metal, offtake

provider-dependent

P0

M5

Verify that a BM node’s ID does not change after a maintenance/reprovision event

pending

Compute Service: VMaaS

Supports the basic lifecycle of VM nodes

CNP01-09

CNP01

#42

offtake, vm

tenant

P0

#42

LimitedEnv

M0

Create a VM node

published

CNP01-10

CNP01, CNP04

#51

offtake, vm

tenant

P0

#51

LimitedEnv

M1

Get info about a specific VM node, including if it is ready for use

published

CNP01-11

CNP01

#45

offtake, vm

tenant

P0

#45

LimitedEnv

M1

Get a list of all VM nodes in a VPC

published

CNP01-12

CNP01

#46

offtake, vm

tenant

P0

#46

LimitedEnv

M0

Delete a VM node

published

SEC21-03

SEC21

#47

offtake

tenant

P0

#47

LimitedEnv

M0

Confirm sanitization on delete

published

Once created, VM nodes can be accessed for usage, and have the high level support controls of reboot, reinstall

VMAAS01-01

VMAAS01

ssh, vm

tenant

P0

LimitedEnv

M0

A running node can be accessed via SSH/Teleport for further configuration (VM)

approved

CNP01-13

CNP01

offtake, vm

tenant

P0

LimitedEnv

M0

A running node can be rebooted in case of issue (VM)

approved

VMAAS02-01

VMAAS02

tenant

P0

LimitedEnv

M1

DEPRECATED: A node can be reinstalled from its configured stock operating system

pending

CNP01-14

CNP01

#110

offtake, vm

tenant

MinAPI

P0

LimitedEnv

M3

A running VM can be stopped (Without being destroyed)

published

CNP01-15

CNP01

#111

offtake, vm

tenant

MinAPI

P0

LimitedEnv

M3

A powered off VM can be started

published

CNP05-02

CNP05

#112

offtake, vm

tenant

MinAPI

P0

LimitedEnv

M3

Support for user-defined tags/labels and cloud-init metadata on instances. (VM)

published

BOOT02-02

BOOT02

#113

offtake, ssh, vm

tenant

P0

M3

Support for cloud-init and instance metadata discovery via link-local addresses (e.g. 169.254.169.254) or virtual devices (VM)

published

Quota should be enforced

VMAAS03-01

VMAAS03

tenant

P2

Noisy Neighbor CPU/GPU tests

approved

VMAAS04-01

VMAAS04

tenant

P2

Tenant cannot allocate more VMs than their quota allows

approved

The underlying system allows VMs to access the GPU, with high performance, and are running with the correct parameters for optimal performance

VMAAS05-01

VMAAS05

#55

ssh, vm

tenant

P0

#55

LimitedEnv

M0

Check that the host OS is a known acceptable image (like Ubuntu/DGXOS)

published

VMAAS06-01

VMAAS06

#59

gpu, ssh, vm

tenant

P0

#59

LimitedEnv

M0

Check that the GPU is visible and accessible from the VM

published

VMAAS07-01

VMAAS07

#62

gpu, ssh, vm

tenant

P0

#62

LimitedEnv

M0

Check that the correct Linux Kernel, libvirt, sbios, and NVIDIA drivers are installed

published

VMAAS08-01

VMAAS08

tenant

P0

LimitedEnv

M1

Check that vEGM is configured correctly

approved

VMAAS09-01

VMAAS09

#67

gpu, ssh, vm

tenant

Does this include GPU Passthrough

P0

#67

LimitedEnv

M0

Check that vCPU pinning is set correctly, PCI bus is configured correctly

published

VMAAS10-01

VMAAS10

tenant

P1

LimitedEnv

M5

TODO: More things from the "NVIDIA Grace I/O Virtualization Guide"

approved

Can run AI/ML Jobs

VMAAS11-01

VMAAS11

tenant

P0

LimitedEnv

M0

GPU Stress workload (VM)

approved

VMAAS12-01

VMAAS12

gpu, slow, ssh, vm, workload

tenant

P0

LimitedEnv

M1

Nim Inference jobs (VM)

approved

VMAAS13-01

VMAAS13

tenant

P0

M5

Run NCCL tests

approved

nodes will be able to access expected resources on the local networks

AWS EC2/Forge

VMAAS14-01

VMAAS14

tenant

P0

LimitedEnv

M5

Nodes can communicate across ethernet, infiniband, and NVLink (VM)

approved

Observability & Debug

CNP06-03

CNP06

#114

offtake, vm

tenant

P0

M3

Serial console access is required (read-only sufficient, interactive preferred) (VM)

published

Stable Identifiers

CNP08-03

CNP08

#198

offtake, vm

tenant

P0

M5

Verify that a VM’s ID does not change after a stop/start cycle

published

CNP08-04

CNP08

offtake

provider-dependent

P0

M5

Verify that a VM’s ID does not change after a live migration or maintenance event

pending

VM Hardening

CNP01-16

CNP01

#256

iam, offtake, security, vm

tenant

P0

M5

Verify console access is restricted via RBAC

published

CNP01-17

CNP01

#257

offtake, security, vm

tenant

P1

M5

Verify USB, clipboard, and unnecessary virtual devices are disabled

published

SDN Controller

Controls software-defined networking overlays and fabric behavior to match the compute-on-demand requirements. Enables network segmentation, isolation, dynamic network configuration, and security groups.

SDN01-01

SDN01

#11

offtake, network

tenant

P0

#11

LimitedEnv

M0

Create a VPC

published

SDN01-02

SDN01

#12

offtake, network

tenant

P0

#12

LimitedEnv

M0

Retrieve a VPC

published

SDN01-03

SDN01

#13

offtake, network

tenant

P0

#13

LimitedEnv

M0

Update a VPC

published

SDN01-04

SDN01

#14

offtake, network

tenant

P0

#14

LimitedEnv

M0

Delete a VPC

published

SDN04-01

SDN04

#16

offtake

tenant

P0

#16

LimitedEnv

M0

Verify that a VPC has an exclusive subnet

published

SDN04-02

SDN04

#17

offtake, network, security

tenant

P0

LimitedEnv

M0

Verify that nodes in two VPCs cannot communicate over the N/S (TAN) network

published

SDN04-03

SDN04

#18

offtake

tenant

P0

#18

LimitedEnv

M0

Verify that nodes in two VPCs cannot communicate over the E/W (CIN) network

published

NET03-01

NET03, SDN01

#115

offtake, network

tenant

P0

M3

Support non-conflicting Bring-Your-Own-IP (including 7.0.0.0/8)

published

SDN11-01

SDN11

#116

network

tenant

P0

M3

Support Stable Private IP allocations, where if a VM crashes and restarts the same IP address remains pinned until the node is deleted

published

SDN05-01

SDN05

#117

offtake, network

tenant

P0

M3

Atomically switch a floating private IP between nodes via API within <10 seconds without requiring an instance reboot.

published

SDN06-01

SDN06

#118

offtake, network

tenant

P0

M3

Localized DNS: Support for custom, localized DNS settings to enable internal domain resolution to private endpoints (e.g. storage endpoints)

published

SDN07-01

SDN07

#119

offtake, network

tenant

P0

M3

Support for VPC peering with full bandwidth and no "hairpin" routing.

published

SDN02-01

SDN02, SDN03

#199

offtake, network, security

tenant

P0

M4

Create a Security Group

published

SDN02-02

SDN02, SDN03

#200

offtake, network, security

tenant

P0

Retrieve a Security Group/list Security Groups

published

SDN02-03

SDN02, SDN03

#201

offtake, network, security

tenant

P0

Update a Security Group

published

SDN02-04

SDN02, SDN03

#202

offtake, network, security

tenant

P0

Delete a Security Group

published

CNP03-01

CNP03

#19

offtake

tenant

P0

#19

LimitedEnv

M0

NVLink Partitioning

published

SDN12-01

SDN12

tenant

LimitedEnv

M4

NVLink: Create a new partition; verify success response with partition key.

approved

SDN13-01

SDN13

tenant

LimitedEnv

M4

NVLink: List partitions; retrieve a specific partition by partition ID

approved

SDN14-01

SDN14

tenant

LimitedEnv

M4

NVLink: Delete an empty partition (no instances); retrieve by ID and verify not found.

approved

SDN15-01

SDN15

tenant

LimitedEnv

M4

NVLink: Create a partition, create instance in that partition, make sure the instance becomes ready, delete instance, verify GPUs removed from partition.

approved

SDN16-01

SDN16

tenant

LimitedEnv

M4

NVLink: Create partition, create 2 instances, verify both share the same nvlink_partition_id and both GPUs are reported for that partition

approved

SDN17-01

SDN17

network

tenant

LimitedEnv

M4

IMEX: Assert nvidia-imex and nvidia-imex-ctl packages are installed and nvidia-imex.service unit is registered with systemd.

approved

SDN17-02

SDN17

#598

kubernetes, network

tenant

The Kubernetes half of SDN17, asserting cluster objects rather than packages on a host. Says nothing about any host IMEX daemon - the driver’s two ownership modes disagree on whether one should be running.

P0

LimitedEnv

M9

IMEX (Kubernetes DRA): On a cluster offering multi-node NVLink through the resource-allocation driver, assert the compute-domain device classes are registered and every GPU node in an NVLink clique publishes compute-domain resources through the driver’s per-node plugin, without tenant installation.

approved

SDN18-01

SDN18

network

tenant

LimitedEnv

M4

IMEX: Start nvidia-imex via systemctl start nvidia-imex and assert the service reaches active/ready state.

approved

SDN18-02

SDN18

#623

kubernetes, network, slow

tenant

The Kubernetes DRA reading of SDN18, asserting cluster objects rather than a host service unit. Destructive - allocates a compute domain, deletes a daemon inside it, and releases the domain afterwards, so it needs an idle cluster. Skips where the driver defers the daemon to an operator-run host service, read from the mode the driver declares rather than from the absence of a daemon.

P1

LimitedEnv

M9

IMEX (Kubernetes DRA): On a cluster where the resource-allocation driver owns the IMEX daemon, assert every node of an allocated compute domain runs a driver-started daemon and is a ready domain member, then terminate one daemon in place and assert the driver restores that node to ready membership unaided within a bounded time.

approved

SDN19-01

SDN19

network

tenant

LimitedEnv

M4

IMEX: Stop nvidia-imex via systemctl stop nvidia-imex and assert clean shutdown (other nodes show disabled node status).

approved

SDN19-02

SDN19

#624

kubernetes, network, slow

tenant

The Kubernetes DRA reading of SDN19, asserting cluster objects rather than a host service unit. The departure is a shrink of the domain - withdrawing one node’s channel claim - because a drain leaves the driver’s DaemonSet pods running and would report a departure that never happened. What peers observed is read from a surviving daemon’s own report, never from the domain’s record of the departed node, so stale controller bookkeeping is not mistaken for members noticing. Destructive - removes a node from a live domain, restores it, and releases the domain afterwards, so it needs an idle cluster. Skips where the driver defers the daemon to an operator-run host service.

P1

LimitedEnv

M9

IMEX (Kubernetes DRA): Remove one node from an allocated compute domain on purpose, and assert its driver-managed daemon exits cleanly, a surviving member reports the node unavailable within a bounded time, and the domain stays operational among the nodes that remain.

approved

SDN20-01

SDN20

network, slow

tenant

LimitedEnv

M4

IMEX: Enable IMEX at boot with systemctl enable nvidia-imex, reboot, and confirm IMEX starts automatically.

approved

SDN21-01

SDN21

network

tenant

LimitedEnv

M4

IMEX: After all nodes start IMEX, query domain state via nvidia-imex-ctl -N and assert state is UP, fully connected matrix

approved

SDN22-01

SDN22

tenant

LimitedEnv

M4

IMEX: With IMEX_ENABLE_AUTH_ENCRYPTION=0, form a multi-node domain and assert connectivity without any auth/encryption.

approved

SDN23-01

SDN23

tenant

LimitedEnv

M4

IMEX: Enable IMEX_ENABLE_AUTH_ENCRYPTION=1 with SSL_TLS mode, provide valid server/client certs, and assert mutual authentication and encrypted communication.

approved

SDN24-01

SDN24

tenant

LimitedEnv

M4

IMEX: Enable GSSAPI/Kerberos mode (GSS_AUTH_ENCRYPT), configure KDC/keytabs, and assert mutual auth + encryption between nodes.

approved

SDN25-01

SDN25

tenant

LimitedEnv

M4

IMEX: IMEX/K8s: Submit a workload requesting a ComputeDomain; assert ComputeDomain pods are created on each allocated node.

approved

SDN26-01

SDN26

tenant

LimitedEnv

M4

IMEX: From workload pods in a ComputeDomain run nvbandwidth-test-job

approved

SDN27-01

SDN27

tenant

LimitedEnv

M4

IMEX: After workload completes, assert ComputeDomain and associated IMEX resources are cleaned up.

approved

SDN28-01

SDN28

provider-dependent

REVIEW

Redundant Gateways

pending

Security Group Scope & Timing

SDN02-05

SDN02

#281

offtake, network, security

tenant

P0

M5

Verify security group rules can be scoped at workload level

published

SDN02-06

SDN02

#282

offtake, network, security

tenant

P0

M5

Verify security group rules can be scoped at node level

published

SDN02-07

SDN02

#283

offtake, network, security

tenant

P0

M5

Verify security group rules can be scoped at subnet/tenant level

published

SDN02-08

SDN02

#284

offtake, network, security

tenant

P1

M6

Measure and verify policy propagation timing is within acceptable bounds

published

SDN02-09

SDN02

#285

offtake, network, security

tenant

P0

M5

Verify security group rules can be scoped at K8s API service level

published

All-to-All Storage Routing

SDN08-01

SDN08

#289

offtake, network

tenant

P1

M6

Verify that storage hosts can route to each other with all-to-all L3 communication

published

Port Security

SDN02-10

SDN02

#286

offtake, network, security

tenant

P1

M6

Verify customizable port security policies can be applied to virtual interfaces

published

Network Infrastructure Observability

SDN09-01

SDN09

#290

offtake, network

tenant

P1

M5

Verify logging is available for network hardware faults

published

SDN09-02

SDN09

#291

offtake, network

tenant

P1

M5

Verify logging captures latency/performance fluctuations

published

SDN09-03

SDN09

#292

offtake, network, security

tenant

P1

M5

Verify a detailed audit trail exists for all configuration changes to network filtering rules

published

Network Operator validation through Kubernetes Launch Kit

SDN29-01

SDN29

kubernetes, network, network_operator, slow

operator

P0

LimitedEnv

Verify the Network Operator deployment matches the Launch Kit reference configuration

pending

SDN29-02

SDN29

kubernetes, network, network_operator, slow

operator

Ethernet and InfiniBand variants share this test ID

P0

LimitedEnv

Verify ICMP connectivity between secondary-network pods on every rail

pending

SDN29-03

SDN29

kubernetes, network, network_operator, slow

operator

Ethernet and InfiniBand variants share this test ID

P0

LimitedEnv

Verify pod-to-pod RDMA-CM (rping) connectivity on every rail

pending

SDN29-04

SDN29

kubernetes, network, network_operator, slow

operator

Ethernet and InfiniBand variants share this test ID

P0

LimitedEnv

Verify pod-to-pod RDMA bandwidth (ib_write_bw) meets the reference minimum

pending

SDN29-05

SDN29

gpu, kubernetes, network, network_operator, slow

operator

Ethernet and InfiniBand variants share this test ID

P0

LimitedEnv

Verify GPUDirect RDMA (DMA-BUF) bandwidth between GPU pods meets the reference minimum

pending

Metadata Service

shared service source of truth for resource metadata such as nodes, GPUs, topology, and ownership. Enables scheduling, automation, and lifecycle management.

GCP Resource Manager APIs

META01-01

META01

provider-dependent

P1

LimitedEnv

M4

TODO

approved

Cloud Control Plane

Primary access mechanism for IaaS, provding provision and manage of compute, networking, and storage.

AWS EC2 Control Plane

CTRL01-01

CTRL01

tenant

P0

M4

Needs to have an API we can access from our tests to do actions

approved

CAP02-01

CAP02

#203

bare_metal, governance, offtake

tenant

P1

M8

Verify that fleet management API provides full set of information requested below: Node ID (Unique identifier for a GPU node) Health State (Healthy/Unhealthy classification) Instance ID (Identifier for virtual workload) Creation Timestamp (Time workload/node was created) Hardware Type (Descriptor for the hardware model) GPU Count (Number of GPUs per node) Top-levelAccount/ID (Identifier for the top-level organization/account) Sub-LevelProject/ID (Identifier for the nested project/sub-account) In Use (True/False status indicating if the GPU Node is turned on and in use) Region (Region of the data center where nodes are deployed)

approved

CAP03-01

CAP03

#204

bare_metal, governance, offtake

tenant

P1

M8

Resource discovery API: newly delivered capacity must be discoverable. This "Resource Index" must provide a stable resource identifer.

approved

Stable Identifiers (Switches/Infrastructure)

CNP08-05

CNP08

#205

offtake

tenant

P1

M5

Verify that switch identifiers are stable across reboots and firmware updates

pending

Load Balancing

Distributes traffic across services (e.g. inference endpoints) for availability and scale.

AWS Network Load Balancer

LB01-01

LB01

tenant

P2

LimitedEnv

CRUD a load balancer

approved

LB02-01

LB02

tenant

LimitedEnv

Verify that traffic is distributed across multiple nodes

approved

LB03-01

LB03

tenant

Verify that nodes can be marked as down or up (for rolling deployments/etc)

approved

LB04-01

LB04

tenant

LimitedEnv

Verify that no traffic is dropped during outages

approved

Break-Fix

A break-fix service works with a variety of components to detect, triage, remediate, and validate nodes in an AI factory. The critical focus here is on the GPU nodes, but the system can scale beyond.

Lazarus, Shoreline

BFX04-01

BFX04

#555

bare_metal, breakfix

tenant

P1

LimitedEnv

Check that a GPU health monitoring process is running (e.g. NV Sentinel or GPUd as of Aug 2026)

approved

BFX05-01

BFX05

#556

bare_metal, breakfix

tenant

LimitedEnv

Verify that Tenants can be notified, by some communication system, of planned future node maintenance

approved

BFX06-01

BFX06

#557

bare_metal, breakfix

tenant

LimitedEnv

Verify that Tenants can be notified, by some communication system, of immediate node failure

approved

Breakfix API Lifecycle

BFX01-01

BFX01

#206

breakfix, gpu, kubernetes, offtake

both

P2

M5

Request a reset of GPUs on an operator-managed node

pending

BFX01-02

BFX01

#207

bare_metal, breakfix, offtake

both

P1

M5

Return an individual node to the provider for maintenance via the API

pending

BFX01-03

BFX01

#208

bare_metal, breakfix, offtake

both

P1

M5

Return a rack to the provider for maintenance via the API

pending

BFX01-04

BFX01

#209

breakfix, kubernetes, offtake

tenant

P1

M5

Cordon a node (mark unschedulable); verify no new workloads are placed; verify existing workloads continue

pending

BFX01-05

BFX01

#210

bare_metal, breakfix, offtake

both

P1

M5

Request a host replacement when health thresholds are breached; verify the node is removed from the pool

pending

BFX01-06

BFX01

#610

bare_metal, breakfix, offtake

both

The non-destructive counterpart to BFX01-02 - the tenant keeps the node and its workload while flagging that it needs repair eventually. Return relinquishes the node; report does not.

P1

M5

Report an individual node to the provider for maintenance via the API

pending

Breakfix Events Query

BFX02-01

BFX02

#211

bare_metal, breakfix, offtake

tenant

P1

M5

Query upcoming/current maintenance events for a node

pending

BFX02-02

BFX02

#212

bare_metal, breakfix, offtake

tenant

P1

M5

Query retirement notices for a node/rack

pending

BFX02-03

BFX02

#213

bare_metal, breakfix, offtake

tenant

P1

M5

Query historical repair status for a node

pending

Breakfix Diagnostics

BFX03-01

BFX03

#320

bare_metal, breakfix, offtake

tenant

P1

M5

Query serial numbers of installed hardware (chassis, baseboard, NICs, CPU, GPU) — obfuscated but stable IDs are OK

pending

BFX03-02

BFX03

#214

bare_metal, breakfix, offtake

tenant

P1

M5

Inspect firmware versions of NV switch trays

pending

BFX03-03

BFX03

#215

bare_metal, breakfix, offtake, observability

tenant

P1

M5

Query a node’s log history through a telemetry endpoint

pending

Data Services

SDS Controller

Manages software-defined storage services and policies. Orchestrates provisioning, tiers, and resiliency.

AWS: EBS + FSx backend control planes

HSS01-01

HSS01

#327

offtake, storage

tenant

P1

M7

Verify storage provisioning via vendor/NCP API

published

HSS01-02

HSS01

#505

offtake, storage

tenant

P1

M8

Verify authentication to the storage management API with the configured credential (basic auth or bearer token)

pending

HSS01-03

HSS01

#506

offtake, storage

tenant

P1

M8

Verify the volume provisioning API provisions a volume of the requested size within NVIDIA’s tenant

pending

HSS02-01

HSS02

#328

offtake, storage

tenant

P1

M7

Verify QoS — provisioned throughput meets requested minimum bandwidth and IOPS

published

HSS03-01

HSS03, K8S23

#329

offtake

tenant

P1

M7

Verify K8s CSI integration for storage

published

HSS03-02

HSS03

#486

offtake, storage

tenant

Implemented in #523 (K8sCsiDriverHealthCheck).

P1

M7

Verify the CSI controller Deployment and node DaemonSet pods are running and healthy

published

HSS03-03

HSS03

#487

offtake, storage

tenant

Implemented in #523 (K8sCsiDriverHealthCheck).

P1

M7

Verify the CSIDriver object is registered for the NCP provisioner

published

HSS03-04

HSS03

#488

offtake, storage

tenant

Implemented in #523 (K8sCsiDriverHealthCheck).

P1

M7

Verify at least one StorageClass references the NCP CSI provisioner

published

HSS03-05

HSS03, K8S23

#489

offtake, storage

tenant

P1

M7

Verify a PVC against the NCP StorageClass reaches Bound and creates a PV

published

HSS03-06

HSS03, K8S23

#490

offtake, storage

tenant

P1

M7

Verify the provisioned PV capacity is >= the PVC request

published

HSS03-07

HSS03, K8S23

#491

offtake, storage

tenant

P1

M7

Verify a pod can mount the PVC and read back data it wrote

published

HSS03-08

HSS03, K8S23

#492

kubernetes, offtake, storage

tenant

P1

M7

Verify multiple concurrent PVCs bind to distinct PVs

published

HSS04-01

HSS04

#330

tenant

P1

M7

Verify quota limits can be set on user workloads/volumes

published

HSS04-02

HSS04

#507

storage

tenant

P1

M8

Verify a functional API reports tenant-level storage utilization and quota

pending

HSS04-03

HSS04

#508

storage

tenant

P1

M8

Verify a functional API lists the quotas configured within the NVIDIA tenancy

pending

HSS04-04

HSS04

#509

storage

tenant

P1

M8

Verify a functional API returns utilization against a given quota by ID

pending

HSS04-05

HSS04

#510

storage

tenant

P1

M8

Verify reported quota consumption reflects data written to the filesystem

pending

HSS05-01

HSS05

#331

storage

both

P1

M7

Verify non-disruptive upgrades (NVIDIA can defer maintenance up to 2 weeks)

published

STOR01-01

STOR01

tenant

REVIEW

Storage: Config/default/access to network provided storage

pending

Home Directory Storage

DIR02-01

DIR02

#326

offtake, storage

tenant

P1

M5

Verify NFS protocol shared storage is available

pending

DIR02-02

DIR02

#494

kubernetes, offtake, slow, storage

tenant

P1

M7

Verify POSIX filesystem compliance via the applicable pjdfstest suites

published

DIR01-01

DIR01

#324

offtake, storage

tenant

P1

M5

Verify configurable filesystem-wide quota limits

pending

DIR01-02

DIR01

#325

offtake, storage

tenant

P1

M5

Verify usage accounting for uid/gids

pending

Data Movement

DMS01-01

DMS01

#263

offtake

provider-dependent

P1

M6

Verify a dedicated K8s cluster (or ability to create one) for the data mover stack

published

DMS02-01

DMS02

#264

offtake

tenant

P1

M6

Verify dedicated CPU nodes are available for data mover with high-performance networking

published

DMS03-01

DMS03

#265

offtake

tenant

P1

M6

Verify the same filesystem mounted on GPU nodes is accessible from data mover nodes

published

DMS05-01

DMS05

#266

offtake, network

tenant

P1

M6

Verify stable egress IP for allowlisting access to NVIDIA services

published

DGXC-Managed Storage Deployment

STG02-01

STG02

#358

bare_metal, offtake, sanitization, storage

tenant

P1

M5

Verify optional skip-sanitization flag during break/fix where tenancy doesn’t change

pending

STG03-01

STG03

#359

bare_metal, offtake, network, storage

tenant

P1

M5

Verify stable IP assignment for storage nodes across lifecycle operations

pending

STG04-01

STG04

#360

bare_metal, health, offtake, storage

tenant

P1

M5

Verify out-of-band failure detection (device, network, memory, drive)

pending

STG05-01

STG05

#361

bare_metal, offtake, storage

tenant

P1

M5

Verify topology observability — visibility into failure domains for physical diversity

pending

Storage Security

SEC04-01

SEC04

#296

iam, offtake, security

tenant

P1

M5

Verify least-privilege access policies (resource-based and user-based)

published

SEC04-03

SEC04

#511

offtake, security, storage

tenant

P1

M8

Verify multi-tenant isolation — quota listings contain only paths within the tenant’s namespace

pending

RDMA Memory Protection

HSS06-01

HSS06

#332

storage

tenant

P1

M7

Verify storage systems using RDMA enforce memory protection via authorization keys for both local and remote access

published

Parallel File System Services

high performance file system optimized for AI and HPC workloads

HSS07-01

HSS07

#333

offtake, storage

tenant

P1

M7

Verify that a parallel high-speed filesystem can be provisioned via API

published

HSS07-02

HSS07

#498

kubernetes, offtake, storage

tenant

P1

M7

Verify large-directory listing of 1,000,000 files and 500,000 subdirectories without error or truncation

published

HSS09-01

HSS09

#334

offtake, storage

tenant

P1

M7

Verify multiple filesystems can exist within total capacity (minimum FS size ⇐ 50 TiB)

published

HSS10-01

HSS10

#335

offtake, storage

tenant

P1

M7

Verify live filesystem expansion (capacity, inodes, IO, metadata)

published

HSS10-02

HSS10

#493

offtake, storage

tenant

Platform/context-sensitive (platform: kubernetes). Overlaps HSS10-01 (platform-agnostic whole-filesystem expansion). Candidate for consolidation/deprecation before v1.0; two implementations of one behavior must not both ship to customers.

P1

M7

Verify PVC expansion grows PV capacity and pod-visible size (if supported)

published

HSS12-01

HSS12

#336

storage

tenant

P1

M7

Verify uid/gid/project-id soft and hard quotas with enforcement

published

HSS13-01

HSS13

#337

offtake, storage

tenant

P1

M7

Verify root-squash can be enabled and disabled at any time

published

HSS14-01

HSS14

#338

storage

tenant

P1

M7

Verify the filesystem can be mounted with flock

published

HSS14-02

HSS14

#495

kubernetes, storage

tenant

Platform/context-sensitive (platform: kubernetes). Overlaps HSS14-01 (platform-agnostic provider-evidence flock check). Candidate for consolidation/deprecation before v1.0; two implementations of one behavior must not both ship to customers.

P1

M7

Verify flock() locking behavior is correct across pods on the same PVC

published

HSS15-01

HSS15

#339

offtake, storage

tenant

P1

M7

Verify changelog/audit data is accessible (tracking by uid/gid, file/dir operations)

published

HSS18-01

HSS18

#340

offtake, storage

tenant

P1

M7

Verify client multipathing to all storage servers

published

HSS17-01

HSS17

#496

kubernetes, offtake, storage

tenant

P1

M7

Verify a file written on one node is visible with correct content on another node within 1 second

published

HSS17-02

HSS17

#497

kubernetes, offtake, storage

tenant

P1

M7

Verify file size and mtime changes on one node are reflected on another within the attribute-cache window

published

HSS11-01

HSS11

#500

kubernetes, offtake, storage

tenant

P1

M7

Verify NFS mounts use the expected version (e.g. vers=4.1), nconnect value, transport protocol (proto=rdma or proto=tcp), and readahead tuning

published

HSS11-04

HSS11

#504

kubernetes, offtake, storage

tenant

P1

M7

Verify required vendor/DKMS kernel modules are installed on all GPU and CPU nodes

published

Object Storage Service

durable, scalable object storage for things like datasets, model artifacts, logs, backups.

S3

DATASVC01-01

DATASVC01

#262

control_plane, storage

tenant

P2

M6

Verify S3-compatible API access with authenticated endpoints

published

Block Storage Services

persistent block volumes for a wide range of uses

AWS EBS, GCP Persistent Disk

K8S23-01

K8S23

#272

offtake

tenant

P2

M5

Verify dynamic volume provisioning via CSI

published

DATASVC02-01

DATASVC02

#321

storage

tenant

P1

M7

Verify volume snapshots

published

DATASVC03-01

DATASVC03

#322

storage

tenant

P1

M7

Verify volume resizing

published

DATASVC04-01

DATASVC04

#323

storage

tenant

P1

M7

Verify persistent block volumes survive instance restarts

published

Cache Services

in-memory data store for frequently used data

ElastiCache

DATASVC05-01

DATASVC05

tenant

P2

LimitedEnv

Create a managed cache instance, write a key-value pair, read it back, verify data integrity

pending

Vector Store

stores and indexes embeddings for semantic search and RAG generation

Vertex AI Vector Search

DATASVC06-01

DATASVC06

tenant

P2

LimitedEnv

Create a vector store, insert an embedding, perform similarity search, verify result

pending

Relational Database

structured database (SQL) service

GCP Cloud SQL

DATASVC07-01

DATASVC07

tenant

P2

LimitedEnv

Create a managed SQL database, create a table, insert/query data, verify results

pending

Workload Orchestration

Backup and Recovery

Centralized managed service to automate and govern data backup across services

AWS backup

DATASVC08-01

DATASVC08

tenant

P2

M4

Create a backup of a block volume, delete original, restore from backup, verify data is the same as the original

pending

Model Registry

storage with versioning for models and AI artifacts

AWS Sagemaker Model Registry

DATASVC09-01

DATASVC09

tenant

P2

Push a model artifact with a version tag, retrieve by version, verify it matches (prefer a smaller reference model)

pending

Slurm Control Plane

control plane to schedule and manage large-scale GPU batch training with SLURM semantics.

AWS ParallelCluster

SLURM01-01

SLURM01

tenant

P0

LimitedEnv

M0

Create a SLURM instance using the software platform

published

SLURM02-01

SLURM02

slurm

tenant

P0

LimitedEnv

M0

SlurmInfoAvailable

published

SLURM03-01

SLURM03

slurm

tenant

P0

LimitedEnv

M0

SlurmPartition cpu and gpu

published

SLURM04-01

SLURM04

gpu, slurm

tenant

P0

LimitedEnv

M0

SlurmJobSubmission

published

SLURM05-01

SLURM05

gpu, slurm

tenant

P0

LimitedEnv

M0

SlurmGpuAllocation 1gpu and 2gpu

published

SLURM06-01

SLURM06

slurm

tenant

P0

LimitedEnv

M0

SlurmNodeJobExecution cpu and gpu

published

SLURM07-01

SLURM07

gpu, slow, slurm, workload

tenant

P0

LimitedEnv

M0

SlurmGpuStressWorkload

published

SLURM08-01

SLURM08

gpu, slow, slurm, workload

tenant

P0

LimitedEnv

M0

SlurmNcclMultiNodeWorkload

published

SLURM09-01

SLURM09

slow, slurm, workload

tenant

P0

LimitedEnv

M0

SlurmSbatchWorkload gpu and cpu

published

SLURM10-01

SLURM10

slow, slurm, workload

tenant

P0

LimitedEnv

M0

SlurmSbatchWorkload-inline

published

Managed Kubernetes Control Plane

Tenant-isolated Kubernetes control planes for managing k8s workloads. does placement, networking, lifecyle mgmt.

AWS EKS, GCP GKE

K8S05-01

K8S05

kubernetes, offtake

tenant

P0

LimitedEnv

M0

Create a K8s Instance using the software platform API/CLI

published

Full lifecycle management

K8S38-01

K8S38

both

Cross-role: the tenant requests lifecycle operations; the platform operator executes the managed control plane update. Application continuity is tested separately by K8S09-01.

M4

As a tenant, exercise the complete managed Kubernetes control plane lifecycle through the tenant-facing API/CLI, including requesting a supported control plane version update and observing provider-reported status through completion.

approved

K8S41-01

K8S41

both

P2

The control plane should automatically add more capacity when load increases

pending

K8S27-01

K8S27

#217

kubernetes, offtake

tenant

P2

Pin Control Plane instances to a specific count to handle a particular load-limit

pending

Control Plane Isolation

K8S14-01

K8S14

offtake

both

M4

per tenant k8s control plane outside of the tenant cluster/VPC

approved

K8S39-01

K8S39

tenant

P0

LimitedEnv

M0

Be able to run k8s workloads

published

K8S40-01

K8S40

gpu, kubernetes, slow, workload

tenant

P0

LimitedEnv

M0

K8s Nim Inference

published

K8S32-01

K8S32

gpu, kubernetes, slow, workload

tenant

P0

LimitedEnv

M0

K8s Nim Helm

published

Topology based placement

K8S33-01

K8S33

gpu, kubernetes, slow, workload

tenant

P0

K8s Nccl

approved

k8s upstream proxy

K8S34-01

K8S34

tenant

k8s05

P0

validate adherence to upstream proxy requirements (service to pod load balancing, acces to internal services)

approved

Dynamic Resource Allocaiton

K8S24-01

K8S24

#662

gpu, kubernetes, offtake

tenant

k8s06/22

P0

verify that the k8s cluster supports the DRA API (resources.k8s.io) to get GPUs on the same nvlink

approved

Run k8s conformance tests

K8S01-01

K8S01

#233

kubernetes, offtake, slow

tenant

k8s01

P0

get versions, topology, core APIs by: (a) pass CNCF k8s conformance: https://github.com/cncf/k8s-conformance (b) add AI conformance: [https://github.com/cncf/k8s-ai-conformance](https://github.com/cncf/k8s-ai-conformance)

published

k8s service account OIDC

K8S17-01

K8S17

offtake

tenant

test that the service account can be authorized with OIDC

approved

Test support for CRDs and Validating/Mutating Admission Controllers (webhooks)

K8S21-01

K8S21

#218

kubernetes

tenant

k8s19

P1

create a CRD, create webhook, and test validation/mutation requirement

published

TODO: L3 networking logging

K8S35-01

K8S35

tenant

K8s16

P1

Verify that pod-to-pod L3 traffic flow logs are available and queryable

approved

CNI Compliance

K8S22-01

K8S22

#219

kubernetes, offtake, network

tenant

K8s20

P0

Apply a NetworkPolicy; verify pod connectivity is restricted as expected; verify IPv4 and IPv6 addresses are assigned on dual-stack nodes

published

CSI installation, provisioning, snapshots, and resizing

K8S23-02

K8S23

#650

kubernetes, offtake, storage

tenant

K8S23 permits Helm or Kustomize; the unused alternative skips only after verified successful installation.

P0

Verify CSI driver installation through Helm, registration, and workload readiness

published

K8S23-03

K8S23

#650

kubernetes, offtake, storage

tenant

K8S23 permits Helm or Kustomize; the unused alternative skips only after verified successful installation.

P0

Verify CSI driver installation through Kustomize, registration, and workload readiness

published

K8S23-04

K8S23

#273

kubernetes, offtake, storage

tenant

P0

M5

Verify CSI supports block, shared filesystem, and NFS storage

published

K8S23-05

K8S23

#274

kubernetes, offtake, storage

tenant

P0

M5

Verify CSI supports both static and dynamic provisioning via PVs and PVCs

published

K8S23-06

K8S23

#275

kubernetes, offtake, storage

tenant

P0

M5

Verify CSI credentials are tenant cluster scoped (no cross-cluster access)

published

K8S23-07

K8S23

#276

kubernetes, offtake, storage

tenant

P0

M5

Verify APIs to query storage usage against overall cluster quota with per-PVC/Volume breakdown

published

K8S23-08

K8S23

#650

kubernetes, offtake, storage

tenant

P0

Verify a CSI snapshot restores point-in-time data into a distinct volume

published

K8S23-09

K8S23

#650

kubernetes, offtake, storage

tenant

P0

Verify CSI PVC expansion grows the mounted filesystem and preserves data

published

NVIDIA Operator support

K8S25-01

K8S25

gpu, kubernetes, offtake

tenant

K8S23

P0

TODO: Compatibility with the mainline NVIDIA GPU Operator

approved

K8S25-02

K8S25

#220

gpu, kubernetes, offtake

both

P0

M5

Verify provider-default accelerator operators and drivers can be replaced or overridden with tenant-required versions

pending

K8s Versioning & Compliance

K8S02-01

K8S02

#659

kubernetes, offtake

tenant

INFO

M5

Verify support for the three most recent minor releases (N-2)

pending

K8S02-02

K8S02

#659

kubernetes, offtake

operator

Downtime during control plane updates is covered by K8S09-01.

INFO

M5

Verify control plane security patching is automated and keeps the control plane on a current upstream patch

pending

Zero-Downtime Upgrades

K8S08-01

K8S08

#661

k8s_upgrades, kubernetes, offtake

operator

Read live managed-service configuration and supported version transitions; active continuity is covered by K8S09-01.

INFO

M5

Verify provider-managed control plane version policy and supported transitions

pending

K8S09-01

K8S09

#661

k8s_upgrades, kubernetes, offtake, slow

tenant

INFO

M5

Perform a minor version control plane upgrade and verify zero application downtime

pending

K8S10-01

K8S10

#661

k8s_upgrades, kubernetes, offtake, slow

tenant

INFO

M5

Perform a rolling node upgrade and verify disruption budgets are respected

pending

HA Control Plane

K8S11-01

K8S11

offtake

operator

INFO

M5

Verify the control plane recovers from a single-node failure

pending

Backup & Disaster Recovery

K8S12-01

K8S12

offtake

operator

INFO

M5

Verify backup and recovery within defined RPO/RTO

pending

K8s Access Controls

K8S15-01

K8S15

#271

kubernetes, offtake, network, security

tenant

P0

M5

Verify the K8s API endpoint has network access controls (firewall/private link)

published

K8s Encryption

K8S19-01

K8S19

offtake

operator

INFO

M5

Verify at-rest encryption is enabled for etcd

pending

Certificate Management

SEC09-01

SEC09

#301

offtake, security

operator

P1

M5

Verify certificates are rotated on a 60-day cycle

published

SEC09-02

SEC09

#302

offtake, security

both

P1

M5

Verify support for both provider-managed and customer-managed keys

published

Autoscaling

K8S36-01

K8S36

#267

kubernetes

tenant

P1

M6

Verify Cluster Autoscaler integration (upstream)

published

Kubernetes Security Response

K8S04-01

K8S04

offtake

operator

INFO

INFO

M5

Verify NCP participates in the Kubernetes Security Response Committee (SRC) process and can patch disclosed vulnerabilities during embargo

pending

Node Pool Lifecycle Management

K8S06-01

K8S06

#269

kubernetes, offtake, slow

tenant

P0

M5

Create a K8s node pool via API/CLI specifying node type (CPU or GPU instance type)

published

K8S06-02

K8S06

#270

kubernetes, offtake, slow

tenant

P0

M5

Update a K8s node pool (e.g., scale to a target count)

published

K8S06-03

K8S06

#223

kubernetes, offtake, slow

tenant

P0

M5

Delete a K8s node pool

published

K8S06-04

K8S06

#224

offtake

tenant

P0

M5

Verify default node labels and taints can be specified when a node joins the cluster via node pool

published

API Server Metrics

K8S07-01

K8S07

#225

kubernetes, offtake

both

P1

M5

Verify API Server metrics are available in a Prometheus-scrapable format for SLO measurement

published

Kubernetes ServiceAccount IAM Integration

K8S16-01

K8S16

#663

iam, kubernetes, offtake

tenant

Runs cloud operations from a pod and verifies the observed identity. The AWS reference uses IRSA and scoped S3 object reads.

P0

M5

Verify a workload assumes its ServiceAccount-bound platform IAM identity and enforces allowed and denied resource scopes

published

Public OIDC Endpoint

K8S17-02

K8S17

#226

iam, kubernetes, offtake

tenant

P0

M5

Verify the cluster exposes a cluster-specific OIDC Issuer endpoint for workload identity federation

published

K8s Control Plane Logging

K8S20-01

K8S20

#227

kubernetes, offtake

both

P1

M5

Verify ability to view or export Kubernetes control plane logs (apiserver, kcm)

published

K8s Performance

K8S28-01

K8S28

#268

offtake

tenant

Consolidates the duplicate reference requirement and former test K8S37-01 into the canonical offtake K8S28 requirement.

P0

M5

Verify the managed Kubernetes service meets standard Kubernetes performance and control plane SLO targets at up to 5,000 nodes or the provider’s declared maximum cluster size, whichever is smaller

pending

Multi-Cluster Support

K8S26-01

K8S26

#277

kubernetes, offtake, network

tenant

P0

M6

Support multiple clusters in the same tenancy and in the same VPC

published

LoadBalancer Service Support

K8S29-01

K8S29

#665

kubernetes, offtake, network, slow

operator

Creates three disposable Services and checks their backend responses through assigned ingress IPs; requires provider annotations and reserved static addresses.

P0

M5

Verify public and private LoadBalancer connectivity and exact static IP assignment

published

Conditional DNS Forwarding

K8S30-01

K8S30

#665

kubernetes, offtake, network, slow

operator

Adds and restores one unique CoreDNS zone; checks the answer and upstream query log.

P0

M5

Verify a temporary CoreDNS zone forwards to the designated resolver and cluster DNS still works

published

Configured Kubernetes CIDR Ranges

K8S31-01

K8S31

#665

kubernetes, offtake, network, slow

operator

Validates a preconfigured cluster; does not reprovision alternate ranges. Requires ServiceCIDR API and node podCIDRs; CNIs without those need a provider-specific probe.

P0

M5

Verify configured service, node and pod ranges against live control-plane ranges and fresh allocations

published

Power Policy Management

Control plane to manage power allocation (re-allocation) based on need and availability

Boost and Flex (DPS)

POWER01-01

POWER01

operator

P2

MiniCloud

TODO

approved

AI Platform & User Access

Host API Gateway

Service control plane "front door" that enforces authn/authz via IAM

AWS/GCP API Gateway

APIGW01-01

APIGW01

tenant

P2

TODO

approved

Al Platform 1..N Control Planes

Placeholder for all AI platform CPs you might run. E.g. Run:ai, NVCF, Redhat Inference Server

AICP01-01

AICP01

tenant

P2

TODO

approved

Web Management Dashboard

WEBUI01-01

WEBUI01

tenant

P2

LimitedEnv

M5

TODO

approved

Observability

Observability Collectors

OBS01-01

OBS01

both

P0

LimitedEnv

M0

Validate that an OTel endpoint is provided and some data can be pulled from it

published

OBS02-01

OBS02

both

P0

LimitedEnv

M0

Logging information about the control plane is available

published

OBS03-01

OBS03

both

P0

LimitedEnv

M0

Logging information from instances

published

OBS04-01

OBS04

both

P0

LimitedEnv

M0

Alerts can be issued by the system in case of problems

published

Telemetry Delivery

OBS05-01

OBS05

#342

observability

both

P1

M7

Verify telemetry latency is no longer than 120 seconds

published

Network Telemetry

OBS06-01

OBS06

#343

network, observability

both

P1

M7

Verify telemetry is available for North-South (front-end) network

published

OBS07-01

OBS07

#344

network, observability

both

P1

M7

Verify telemetry is available for East-West (back-end / GPU interconnect) network

published

OBS08-01

OBS08

#345

network, observability

both

P1

M7

Verify telemetry is available for Management network

published

OBS09-01

OBS09

#346

network, observability

both

P1

M7

Verify telemetry is available for NVSwitch Fabric (GB200+)

published

OBS10-01

OBS10

#347

network, observability

both

P1

M7

Verify telemetry is available for Host network (NIC-level)

published

Required Logs

OBS11-01

OBS11

#348

network, observability

both

P1

M7

Verify Fabric Manager logs are available (where applicable)

published

OBS12-01

OBS12

#349

network, observability

both

P1

M7

Verify Subnet Manager logs are available (where applicable)

published

OBS13-01

OBS13

#350

network, observability

both

P1

M7

Verify UFM Event logs are available

published

OBS14-01

OBS14

#351

network, observability

both

P1

M7

Verify general switch logs are available

published

OBS15-01

OBS15

#352

network, observability

both

P1

M7

Verify switch syslogs are available

published

OBS16-01

OBS16

#353

network, observability

both

P1

M7

Verify switch kernel logs are available

published

OBS17-01

OBS17

#354

observability

both

P1

M7

Verify BMC SEL logs are available

published

OBS18-01

OBS18

#355

bare_metal, observability

both

P1

M7

Verify host syslogs are available

published

OBS19-01

OBS19

#356

network, observability

both

P1

M7

Verify VPC Flow logs (all ingress/egress traffic) are available

published

Telemetry / Data Lakes

TELEM01-01

TELEM01

both

P1

LimitedEnv

TODO

approved

TELEM02-01

TELEM02

#120

tenant

MinAPI

P0

MiniCloud

M3

Read only access to a BMaaS system’s serial console

published

TELEM03-01

TELEM03

#121

tenant

MinAPI

P0

LimitedEnv

M3

Read only access to a VM serial console

published

BMC/Redfish Telemetry

TELEM04-01

TELEM04

#362

gpu, observability

both

P1

M7

Verify BMC/Redfish telemetry is accessible via API for GPU metrics not available from the host OS

published

Storage Telemetry

TELEM05-01

TELEM05

#363

observability, storage

both

P1

M5

Verify storage resource capacity metrics are available (used/free/total)

approved

TELEM06-01

TELEM06

#364

observability, storage

both

P1

M5

Verify storage performance metrics are available (bandwidth, IOPS, latency)

approved

NVLink Switch Telemetry

TELEM07-01

TELEM07

#365

gpu, network, observability

both

P1

M5

Verify NVLink metrics are available from the GPU perspective (per-link counters, bandwidth utilization)

approved

TELEM08-01

TELEM08

#366

network, observability

both

P1

M5

Verify NVLink metrics are available from the switch perspective (port-level counters, error rates)

approved

Security & Identity

Network Security

BMC Security

SEC12-01

SEC12

#306

offtake, network, security

operator

P0

M5

Verify BMC management is on a dedicated, restricted network (physically separate or VLAN/VRF-isolated)

published

SEC12-02

SEC12

#307

offtake, network, security

tenant

P0

M5

Verify BMC interfaces are not reachable from tenant networks

published

CNP10-01

CNP10

#260

offtake, network, security

operator

P0

M5

Verify IPMI is disabled; Redfish over TLS is used with AAA

published

SEC12-03

SEC12

#308

offtake, network, security

operator

P0

M5

Verify BMC is only accessible via a hardened bastion (jumphost) server; direct public/corporate network access is blocked

published

Network Traffic Encryption

SEC13-01

SEC13

#309

offtake, network, security

tenant

P1

M5

Verify mTLS (or equivalent) for all east-west and north-south traffic

pending

Edge Security

SEC14-01

SEC14

#311

offtake, network, security

both

P0

M5

Verify no public internet access to API endpoints by default

published

SEC13-02

SEC13

#310

offtake, network, security

tenant

P1

M6

Verify insecure protocols (HTTP, SSLv3, TLSv1) are disabled

published

SEC15-01

SEC15

#711

offtake, network, security

tenant

Only a dropped probe (timeout) counts as filtered; a refused probe reached the host. Source port is opt-in via required_dimensions, since common security group and ACL implementations cannot match on it.

P1

Verify traffic filtering enforces each five-tuple match dimension - an allowed baseline flow succeeds and flows differing only in protocol, source IP, destination IP, or destination port are blocked

pending

InfiniBand Security

SDN04-04

SDN04

#287

bare_metal, infiniband, offtake, network, security

operator

P0

M6

Verify IB tenant isolation — compute dedicated to NVIDIA is isolated from other customers

published

SDN04-05

SDN04

#288

bare_metal, infiniband, offtake, network, security

operator

P0

M6

Verify IB keys are configured: P_Key, Management Key, Aggregation Management Key, VendorSpecific Key, CongestionControl Key, Node2Node Key, Manager2Node Key

published

Tenancy Isolation

SEC11-01

SEC11

#305

iam, offtake, network, security

tenant

P0

M5

Verify hard physical or logical isolation between tenants for network, data, compute, and storage resources

published

SEC30-01

SEC30

#718

network, security

tenant

P0

Verify traffic across the tenant-to-provider boundary is default-deny, with an explicitly allowed positive-control flow succeeding

pending

Authentication

User Authentication

SEC01-01

SEC01

#293

iam, offtake, security

tenant

P0

M5

Verify user authentication via OIDC for platform services

published

In-Cluster Authentication

SEC06-01

SEC06

#294

iam, offtake, security

tenant

P0

M5

Verify workloads and nodes receive short-lived credentials/tokens

published

External Service Authentication

SEC03-01

SEC03

#295

iam, offtake, security

tenant

P0

M5

Verify out-of-cluster service accounts can authenticate with long-lived credentials

published

Admin Interface Security

SEC07-01

SEC07

#298

iam, offtake, security

operator

INFO

P0

M5

Verify all administrative interfaces (UI, CLI, API) are protected by Multi-Factor Authentication

published

Authorization

RBAC

SEC04-02

SEC04

#297

iam, offtake, security

tenant

P1

M5

Assign a minimal role to a user; verify they cannot perform actions outside that role on Compute, Storage, and Network APIs

published

Audit & Logging

Audit & Logging

SEC08-01

SEC08

#299

offtake, security

operator

P1

M5

Perform a management API call; verify an audit log entry is generated with correct metadata

published

SEC08-02

SEC08

#300

offtake, security

operator

P1

M5

Verify audit logs are retained for at least 30 days

published

Hardware Security & Compliance

Data Sanitization

SEC21-04

SEC21

#312

bare_metal, offtake, sanitization, security

operator

P0

M6

Verify memory sanitization between tenants

published

SEC21-05

SEC21

#313

bare_metal, gpu, offtake, sanitization, security

operator

P0

M6

Verify SRAM/GPU memory is sanitized between tenants

published

SEC21-06

SEC21, SEC22

#314

bare_metal, firmware, offtake, sanitization, security

operator

P0

M6

Verify TPM and BIOS are reset during tenant transitions or hardware replacement

published

At-Rest Encryption

SEC20-01

SEC20

offtake

operator

INFO

M5

Verify all data at rest is encrypted via SED on local NVMe/SSD and network-attached storage

pending

Key Management

SEC09-03

SEC09, SEC10

#303

offtake, security

operator

P1

M5

Verify a centralized KMS is used for all encryption keys and secrets

published

SEC09-04

SEC09

#304

offtake, security, slow, workload

tenant

P0

M5

Verify support for Customer Managed Keys (BYOK)

published

Network Transport

Backend Switch Fabric API

Backend Switch Fabric API

NET01-01

NET01

#278

offtake, network

tenant

P0

M5

Query the API for a compute node and verify it returns backend switch IDs (leaf, spine, core)

published

NVLink Domain API

NVLink Domain API

NET02-02

NET02

#279

offtake, network

tenant

P0

M5

Query the NVLink domain ID for a compute node supporting NVLink

published

Transport Connectivity

CorpIT Interconnect

NET04-01

NET04

#280

offtake

tenant

P1

M5

Verify Private Cloud Interconnect to NVIDIA CorpIT is operational

pending

DGXC Storage Interconnect

NET05-01

NET05

offtake

tenant

INFO

M5

Verify high-bandwidth Private Cloud Interconnect to DGXC on-prem object storage

pending

NET05-02

NET05

offtake

both

INFO

M5

Verify end-to-end MACsec encryption (fail-closed)

pending

Cluster Internet Access

NET06-01

NET06

offtake

tenant

INFO

M5

Verify egress NAT IPs are a static pool dedicated to NVIDIA

pending

NET06-02

NET06

offtake

operator

INFO

M5

Verify redundant upstream paths for internet connectivity

pending

Capacity & Fleet Management

Governance Metrics

Governance Metrics

CAP01-01

CAP01

#251

bare_metal, governance, offtake

tenant

P1

M6

Query and verify governance API returns Delivered, Healthy, Reserved, and Active metrics for nodes/GPUs

published

Capacity Reservations

Capacity Reservations

CAP04-01

CAP04

#252

bare_metal, capacity, offtake, security

tenant

P1

M6

Verify a set of resources can be logically grouped and pinned to an account

published

CAP04-02

CAP04

#253

bare_metal, capacity, offtake, security

tenant

P1

M6

Verify atomic allocation of a topology block as a single unit

published

Unified Health APIs

Unified Health APIs

CAP05-01

CAP05

#254

bare_metal, health, offtake

tenant

P1

M6

Verify per-host health API returns real-time GPU state, thermal status, memory health

published

CAP05-02

CAP05

#255

bare_metal, health, offtake

tenant

P1

M6

Verify primitive-level health aggregation (cluster, nodegroup, or reservation level)

published

Benchmarking

Exemplar

Run a mini-cloud configuration of Exemplar benchmarks

BM01-01

BM01

offtake

tenant

Simulate Exemplar across 512 GPU cluster using latest dgxc-benchmarking release

pending

BENCH01-01

BENCH01

provider-dependent

REVIEW

P2

NVMesh checks?

pending

BENCH02-01

BENCH02

provider-dependent

REVIEW

IPv4 and IPv6 checks?

pending

BM01-02

BM01

#249

offtake

tenant

P1

M6

Verify performance is within 5% of NVIDIA Reference

published