Inside NVIDIA Exemplar Cloud: Closing 8-12% AI Training Performance Gaps
How NVIDIA diagnosed 8-12% AI training performance gaps on H100 and GB200 NVL72 clusters using perf, Nsight Systems, SMMU tuning, and NUMA fixes.
NVIDIA's Exemplar Cloud validation program requires partner AI clusters to stay within 5% of reference architecture (RA) performance, yet real-world deployments built from identical H100, GB200 NVL72, or GB300 NVL72 hardware routinely lag 8-12% behind. NVIDIA engineers found these gaps rarely trace to one dramatic failure, but to compounding configuration issues across kernel, hypervisor, BIOS, and NCCL settings.
Two case studies illustrate the diagnostic approach. On a virtualized GB200 NVL72 cluster running DeepSeek-V3 MoE pretraining, Nsight Systems and Linux perf traces showed 24% of CPU cycles consumed by Arm SMMU command-queue invalidation calls, caused by serialized VM exits during frequent guest memory mapping. Enabling Virtual Command Queue (VCMDQ) support in the host kernel and hypervisor removed the serialization and closed a 12-14% iteration-time gap. On an H100 SXM5 cluster running Llama 3 70B, turbostat and numastat exposed BIOS C-states locked at C1, starving turbo frequency headroom, and hypervisor threads sharing physical cores with data-loader processes, triggering NUMA-remote memory stalls; correcting the C-state policy and isolating cpusets recovered most of a 12% deficit.
For infrastructure engineers, the takeaway is that closing these gaps demands layer-by-layer profiling—SMMU behavior, CPU power states, process affinity, and NCCL fabric tuning—rather than assuming identical hardware guarantees identical software performance.