Support and performance optimization for Hopper (H200) architecture?

[JimyMa]

Hi team,

Thanks for the great work on boltz-cp. I am currently evaluating the framework for long-sequence model training and inference on large-scale GPU clusters, and I would like to understand the current state of support and optimization for the Hopper (H200) architecture.

Could you share some insights on the following aspects?

1. Out-of-the-box Compatibility
   Are there any known issues, specific CUDA driver API requirements, or recommended baseline configurations when deploying boltz-cp on H200 nodes?

2. Hopper-Specific Hardware Utilization
   Given the high memory bandwidth and compute requirements of Context Parallelism, does the current implementation actively leverage Hopper-specific features?

Are the underlying kernels optimized using TMA (Tensor Memory Accelerator) for asynchronous data movement and global memory access?

Is there any integration with the Transformer Engine to accelerate core tensor computations using FP8 or mixed precision?

3. Communication and Cluster Scaling
   Efficient communication is critical for the frequent activation transfers in CP. How does the framework perform over 4th-Gen NVLink/NVSwitch topologies within an H200 node? Furthermore, for cross-node scaling, are there recommended parallelism strategies or known bottlenecks when operating over GPU Direct RDMA networks?

Any deployment best practices, performance benchmarks, or planned roadmap items regarding H200 support would be greatly appreciated.

Thank you!

[JimyMa]

Hi again,
I just went through the newly released Fold-CP preprint (arXiv:2603.14806) and realized that H100 was actually your primary testbed for the scaling benchmarks (up to 144 GPUs). This is fantastic work!
Given the extensive architectural details provided in the paper, I'd like to refine my initial questions and dive a bit deeper into the performance ceiling and topology constraints on Hopper clusters:

1. TMA and FP8 IntegrationThe paper mentions leveraging cuEquivariance fused kernels with BF16 precision to halve the memory footprint. For the extremely memory-bandwidth-bound distributed operations (like the asynchronous P2P ring communications and custom autograd paths), are there any plans to implement Hopper's TMA (Tensor Memory Accelerator) for more efficient data movement? Additionally, is there a roadmap for Transformer Engine (FP8) integration, or is the precision requirement for coordinate geometry too strict for FP8?
2. The Square Topology ($\sqrt{P} \times \sqrt{P}$) vs. Cross-Node RDMAI found the requirement for a strict square 2D device mesh (to maintain symmetry for Row-wise and Column-wise updates) very interesting. However, when scaling out to massive sizes (e.g., $8 \times 8$ for 64 GPUs) across multiple nodes, the network bounds transition from intra-node NVLink to cross-node GPU Direct RDMA.

• Does this strict square constraint introduce routing inefficiencies or specific load imbalances across the RDMA network boundaries?
• Are there specific network topologies or RDMA configurations you highly recommend to mitigate the latency of the ring-shift communications at this scale?

Looking forward to your insights!

[DejunL]

Hi, thank you for your interest and questions. Here are responses to your questions:

1. Most of math operations such as matmul involved are dispatched via pytorch code and will run with whatever pytorch kernel the user has in their runtime environment, presumably already has Hopper support with recent pytorch versions. Specifically to the TriAttn kernel from cuEq as you mentioned, we relies on cuEq implementation for kernel dispatch and as far as I know, cuEq currently doesn't have Hopper specific kernel but otherwise can run on Hopper GPUs with fallback kernels. On the other hand, the Blackwell TriAttn kernels are supported by cuEq
2. I don't think it the sqaure device counts per se is the limitation but rather the requirements to distribute the memory consumption to as many GPUs as possible, square or not square, will ultimately hit communication bandwith limitation simply because of the GPU count limit of a nvlink domain. As a lot of algorithms in Fold-CP requires communication across the device mesh along both CP axes, ultimately certain comm op would run outside of the nvlink domain and pytorch (via NCCL backend) will route traffic via network. We are actively working towards optimizing the individual CP algorithms to hide the latency. In general, and as mentioned in the paper, larger problem size tends to have more compute in flight to hide network latency but at this stage, we do not expose control for network topo in a way that would allow the user to easily toggle among algorithms, nor I think it would make a difference in the current implementation for the reason I just mentioned