NVIDIA guide: GPU-initiated NVSHMEM speeds GROMACS strong-scaling

NVIDIA demonstrates how replacing CPU-orchestrated MPI with GPU-native NVSHMEM removes the CPU from the halo-exchange critical path in GROMACS, eliminating up to 12 blocking CPU–GPU synchronizations per timestep and enabling stronger scaling, with benchmarks showing up to 2x improved intra- and inter-node strong-scaling versus GPU-aware MPI.

How does GPU-initiated communication replace MPI in GROMACS?

GPU-initiated remote memory access (RMA) using NVIDIA NVSHMEM moves packing, transfer and signaling into device-side kernels so the CPU no longer blocks the timestep. The previous MPI workflow required the CPU to synchronize with the GPU before and after every communication pulse; with a 3D decomposition that produced as many as 12 blocking CPU–GPU waits per timestep. The NVSHMEM approach fuses pack, put and completion-wait into a single device kernel and uses device-scoped barriers and nvshmem signaling calls (for example nvshmemx_float_put_signal_nbi_block) to transfer data and notify peers without host intervention.

NVIDIA describes starting from the stream-triggered API nvshmemx_put_signal_nbi_on_stream and then moving to fused device kernels that perform grid-stride packing, a device barrier, an NVSHMEM put that sets a peer signal, and a device-side wait on that signal. The host still launches kernels in pulse order, but it drops the blocking cudaStreamSynchronize and MPI calls that had serialized the original implementation.

What concrete performance and implementation changes did NVIDIA show?

The device-native design reduces synchronization points and kernel launches and improves overlap between compute and communication. GROMACS had reached 100–200 microseconds per timestep on modern heterogeneous clusters, where communication became the bottleneck. The NVSHMEM redesign reduced kernel launches per timestep (the AI summary notes a drop from six to one in the optimized fused design), allowed maximal overlap of communication and computation, and removed the host from the critical path. Benchmarking on the NVIDIA Eos supercomputer and NVIDIA GB200 NVL72 clusters demonstrated up to 2x improved intra- and inter-node strong-scaling performance versus GPU-aware MPI, particularly on latency-bound systems.

The post also quantifies CPU impact: prior MPI-based halo exchange consumed more than 50% of total CPU wall time at peak iteration rates because the CPU had to block to orchestrate transfers. The fused device kernels use nvshmem signaling so receivers wait on a device-side signal variable rather than rely on host-driven MPI completion. For NVLink-connected peers, NVSHMEM can use nvshmem_ptr and system-scope release stores for direct stores, while for non-NVLink peers the code defaults to NVSHMEM put operations over RDMA fabrics such as InfiniBand. NVIDIA also highlights Hopper’s TMA engine as an efficient path for bulk remote stores.

Why it matters

Removing the CPU from the communication critical path shifts GROMACS strong-scaling from a host-limited problem to a device-limited one. When iteration latencies reach the 100–200 microsecond range, the dozen blocking host synchronizations per timestep severely cap iteration rates and inflate CPU wall time. Device-initiated RMA reduces those synchronizations and enables finer-grained per-pulse signaling and kernel fusion that overlap communication and compute. That pattern generalizes to other halo-exchange HPC applications, though NVIDIA notes that wider standardization of GPU-native communication primitives remains an unresolved ecosystem challenge.

What to watch

Watch for broader NVSHMEM adoption across MD engines and other halo-exchange codes and for upstream or vendor libraries that expose the per-pulse, device-side signaling patterns NVIDIA describes. The next confirmatory signal will be external reproducible results on non-NVIDIA cluster topologies and integration of these device-initiated primitives into commonly used MPI or PGAS abstractions.

Authors and timing: the guide is dated Jul 09, 2026 and is authored by Mahesh Doijade, Andrey Alekseenko, Ania Brown, Alan Gray, Jan Ciesko and Szilárd Páll.