NVIDIA NCP-AII - NVIDIA AI Infrastructure

The primary purpose of a NeMo burn-in is to stress test the hardware and software stack using representative NeMo workloads before releasing the AI infrastructure to production. NeMo workloads can exercise GPU compute, GPU memory, CUDA libraries, NCCL communication, storage access, checkpointing, container runtime, scheduler integration, and distributed training behavior. This makes NeMo burn-in more realistic than simply checking that GPUs are visible or that a small synthetic benchmark runs successfully. The goal is not to tune hyperparameters for model accuracy, because burn-in validates infrastructure reliability rather than model quality. It is also not mainly about ensuring all GPUs run at identical clock speeds; clock behavior can vary based on power, thermals, workload, and GPU boost behavior. What matters is that the workload runs reliably, without stalls, NCCL failures, GPU Xid errors, storage bottlenecks, memory faults, or unstable performance. In NVIDIA AI infrastructure validation, representative workload burn-in bridges the gap between low-level diagnostics and real production training, helping detect issues that synthetic tests alone may miss.

In an InfiniBand fabric, the Subnet Manager (SM) is the " brain " of the network, responsible for discovering the topology, assigning Local Identifiers (LIDs), and calculating routing tables. In a multi-node fabric, there is typically one Master SM and several Standby SMs for high availability. To identify the master, the sminfo command is first used; it queries the fabric and returns the LID of the current Master SM. Once the LID is obtained, the engineer must map that numerical LID to a physical server name or Node Description. The smpquery ND (Node Description) command is then executed, targeting that specific LID. This sequence is vital for troubleshooting fabric-wide issues, as logs on the Master SM server provide the definitive record of sweeps, traps, and topology changes. Using smpquery NI (Node Info) would provide hardware-level details like the GUID and device ID, but it does not return the human-readable string (server name) defined in the Node Description, which is necessary for rapid identification in a crowded data center.

The correct sequence is to first confirm switch and server firmware configuration, then validate link connectivity and port health, run network benchmarks, validate the software stack, and finally stage AI workload tests. This order follows a layered infrastructure validation approach. Firmware and configuration must be verified first because mismatched firmware or unsupported settings can cause link instability, incorrect RoCE behavior, or inconsistent congestion-control performance. Link and port checks come next to confirm that all physical and logical connections are operational. Network benchmarks then verify throughput, latency, packet behavior, and readiness for GPU-to-GPU communication. Only after the fabric is proven healthy should the software stack and AI workload staging be validated. Starting with application benchmarking before firmware and link validation makes troubleshooting harder because failures may be caused by lower-layer issues. Identity services such as LDAP and RBAC are important for operations, but they do not prove fabric readiness. In a Spectrum-X AI Ethernet deployment, scalable workload readiness depends on clean physical links, correct firmware, healthy port telemetry, validated network performance, and then application-level testing.

The correct solution is a high-performance parallel file system connected through InfiniBand or high-speed Ethernet. Large image training workloads can generate heavy concurrent reads from many GPU nodes, especially when datasets cannot be cached locally. If storage cannot deliver enough per-node and aggregate throughput, GPUs wait for input data and overall training efficiency drops. A parallel file system is designed to distribute metadata and data access across multiple storage servers, allowing many DGX nodes to read training data concurrently. Using InfiniBand or 40/100GbE provides the network bandwidth needed to avoid storage becoming the limiting factor. Expanding local RAID-0 cache may help some repeated reads, but it does not solve shared dataset access or scale well across many nodes. A single NFS server on 10GbE is likely to bottleneck under concurrent multi-node training. General object storage is useful for capacity and cloud-style workflows, but it is not always optimal for high-throughput POSIX-style training access. For DGX cluster optimization, storage must be sized for sustained throughput, metadata operations, and concurrent GPU consumption.

In a large-scale Spectrum-X Ethernet fabric, " East-West " traffic refers to the cross-rack communication between compute nodes. To validate the " Bisectional Bandwidth " (the throughput between two halves of the cluster), administrators use NCCL tests with specific environment variables to control traffic patterns. The NCCL_TESTS_SPLIT variable is used to partition the GPUs into distinct groups for the benchmark. Setting NCCL_TESTS_SPLIT= " DIV 8 " is a standard configuration for multi-node testing on 8-GPU systems. It effectively divides the total number of GPUs by the node count, creating a test environment where each GPU communicates with its corresponding rank on other nodes. By combining this with -g 1 (one GPU per process) across multiple nodes, the engineer can force data to travel across the leaf-and-spine switches rather than staying within the NVLink fabric of a single node. This isolates the physical network performance from the internal GPU-to-GPU bandwidth, providing a true measurement of the fabric ' s ability to handle high-speed AI traffic.

The NVIDIA Collective Communications Library (NCCL) " Tests " are used to verify the maximum achievable bandwidth of the interconnects. On a DGX H100, the GPUs are connected via a dedicated high-bandwidth NVLink Switch fabric (NVLink 4), which provides significantly higher throughput than PCIe. To validate the intra-node (within a single server) performance, the all_reduce_perf test is used. The command in Option D is specifically designed to stress all 8 GPUs (-g 8) across a wide range of message sizes (8 bytes to 16G). The use of the environment variable NCCL_TESTS_SPLIT with the bitwise " OR " or " AND " masks allows the engineer to isolate specific traffic patterns or groups of GPUs to ensure the NVLink switches are distributing the load evenly. For a standard 8-GPU H100 tray, achieving a " Bus Bandwidth " of ~450 GB/s to 900 GB/s (depending on the precision and message size) confirms that the NVLink fabric is operating at its theoretical peak. Using only 4 GPUs (Option B) or 1 GPU (Option C) would not provide a complete picture of the NVLink switch bisection bandwidth.

When a DGX system exhibits high fan speeds and performance degradation, it is typically engaging in Thermal Throttling. High-performance GPUs like the A100 or H100 will automatically reduce their clock speeds (and thus performance) if they exceed safe temperature thresholds. The first and most critical diagnostic step is to run nvidia-smi. This utility provides immediate, real-time telemetry on GPU temperatures, power draw, and " Clocks Throttle Reasons. " By reviewing the output, an administrator can see if " Thermal " is listed as the reason for reduced clocks. This identifies whether the issue is environmental (blocked airflow/hot aisle temperature) or hardware-specific (a failed GPU thermal interface or a dead internal fan). Running more workloads (Option A) would exacerbate the heat, while a power drain (Option C) is a " last resort " that doesn ' t provide diagnostic data. nvidia-smi provides the evidentiary data needed to determine if an RMA (Return Merchandise Authorization) is required for the GPU tray.

Uneven GPU utilization in a multi-node cluster is a classic symptom of PCIe/NUMA imbalance. In a DGX H100, each set of GPUs is physically closer to a specific CPU socket and a specific set of high-speed NICs. If the workload (HPL) is launched without strict affinity, an MPI rank running on a CPU core attached to Socket 0 might attempt to control a GPU attached to Socket 1. This forces data to cross the inter-processor links (UPI/QPI), which have significantly higher latency and lower bandwidth than a direct PCIe path. This " bottlenecking " causes some GPUs to wait for data longer than others, leading to the uneven utilization observed. The verified solution is to use an orchestration script or flags that enforce Affinity. By setting --gpu-affinity and --cpu-affinity, the administrator ensures that each GPU is managed by a CPU core on its local NUMA node. This alignment minimizes latency and ensures that every GPU in the cluster receives data at the same rate, resulting in the flat, high-utilization profile required for a successful HPL burn-in record.

In NVIDIA BlueField DPU architectures, the RShim (Remote-Shim) interface provides a vital communication channel between the DPU and the host or BMC, typically used for early-stage provisioning, console access, and firmware loading. While the DPU is usually managed via the host ' s PCIe bus, certain data center configurations require the DPU to be managed out-of-band via the server ' s Baseboard Management Controller (BMC). To enable this capability, a specific low-level command must be sent to the BMC to toggle the RShim functionality over the internal USB-to-BMC bridge. The command ipmitool raw 0x32 0x6a 1 is the verified raw IPMI hex code used in NVIDIA DGX and certified systems to enable the BMC-to-DPU RShim path. Once enabled, the BMC can " see " the DPU as a USB device, allowing the administrator to push a BlueField Boot (BFB) image to /dev/rshim0/boot for OS installation even if the host CPU is powered off or unresponsive. Option B and C are host-side service commands that assume the driver is already loaded and the hardware path is active, whereas the raw IPMI command is required to enable the hardware path itself.