Kubernetes test infrastructure for NVIDIA GPU software — mock GPU environments, CI tooling, and testing utilities.
Turn any Kubernetes cluster into a multi-GPU environment for testing. No physical NVIDIA hardware required.
kind create cluster --name mokka
helm install nvml-mock oci://ghcr.io/nvidia/k8s-test-infra/chart/nvml-mock \
--namespace mokka --create-namespace \
--set gpu.profile=gb300Every node now reports 4 mock GB300 GPUs, one NVL72 compute tray. gb300 is the
chart default; swap in
a100, h100, b200, gb200, l40s, or t4 for other hardware.
After install, deploy a consumer to test:
| Consumer | Guide |
|---|---|
| NVIDIA Device Plugin | Quick Start |
| NVIDIA DRA Driver | Quick Start |
| NVIDIA GPU Operator | Quick Start |
Full documentation: Mokka documentation site
The nvml-mock Go E2E workflow gates standalone, DRA, GPU Operator, multi-node,
node-wide NRI, and NFD label-provenance coverage. Run manually via
workflow_dispatch or automatically on PRs.
| Test Suite | What It Validates | Profiles |
|---|---|---|
| Standalone Demo | nvml-mock chart install, nvidia-smi, NVLink/fabricmanager, InfiniBand, PCI sysfs, and cross-node checks |
Workflow-selected profiles |
| Failure Injection | Healthy, ECC, lost, and fallen-off-bus modes | Workflow-selected profiles |
| DRA Driver | Mock driver files, nvidia-smi, ResourceSlices, and DRA ResourceClaim scheduling |
Workflow-selected profiles |
| GPU Operator | GPU Operator install, validator pod startup, GFD labels, and allocatable GPUs | Workflow-selected profiles |
| Multi-Node Fleet | Heterogeneous A100/T4 workers, mock files, InfiniBand behavior, device plugin resources, and GPU workload scheduling | Fixed multi-node topology |
| Node-Wide NRI Injection | Ambient mock GPU injection into ordinary pods without GPU requests or hostPath mounts | Workflow-selected profiles |
| NFD Label Provenance | That NFD creates feature.node.kubernetes.io/pci-10de.present from the feature file nvml-mock writes, and that nvml-mock does not write the label itself |
Pinned to a100 — the label is vendor-only and byte-identical across profiles |
The underlying CGo-based mock libnvidia-ml.so that powers nvml-mock.
Use standalone for local development and CI pipelines.
| Document | Description |
|---|---|
| Overview | Project overview, components, GPU profiles |
| Quick Start | Build and run in 5 minutes |
| Configuration | YAML configuration reference |
| Architecture | System design and components |
| Development | Contributing and extending the library |
| Examples | Usage patterns and scenarios |
| Troubleshooting | Common issues and solutions |
| Integration | Description | Guide |
|---|---|---|
| fake-gpu-operator | Run:ai's K8s-level GPU simulation | Integration Guide |
| Demo | Description |
|---|---|
| Standalone | nvml-mock with FGO-style labels on Kind |
| With fake-gpu-operator | Full FGO + nvml-mock integration |
See docs/demo/ for the full list.
- Logo designed by Roman Hlushko with the assistance of OpenAI's ChatGPT.
Apache License 2.0 — see LICENSE.