NVIDIA NCP-AIN - NVIDIA-Certified Professional AI Networking

To enforce strictmulti-tenancy, where:

Tenant A’s GPUcannot talk toTenant B’s GPU

But both can accessshared storage

The correct solution is:

Storage system → Full PKey membership

Each tenant’s GPU → Limited PKey membership

From theNVIDIA InfiniBand P_Key Partitioning Guide:

"A port with limited membership can only communicate with full members of the same PKey. It cannot communicate with other limited members, even within the same partition."

This isolates tenantsfrom each other, while allowingshared access to storage.

Incorrect Options:

Apermits tenant-to-tenant communication.

Bisolates everything, including access to storage.

Cprevents GPU access to storage.

Inmulti-subnet InfiniBand environments, AI training clusters are segmented across network zones (or subnets). Direct GPU-to-GPU communication (especially for collective ops like AllReduce, Broadcast, etc.) requires inter-subnet reachability. NCCL supports this via theInfiniBand Router (IB Router)feature.

From theNCCL User Guide – Environment Variables Section:

"NCCL_IB_USE_IB_ROUTER: Enables NCCL support for IB routers which are used in multi-subnet InfiniBand fabrics. When enabled, NCCL can traverse IB subnets using a properly configured IB router."

This is critical because without IB Router support:

NCCL would be restricted to intra-subnet GPU collectives.

Multi-node training across subnets would fail or fall back to slower TCP fallback mechanisms.

Technical Explanation:

IB Routers usesubnet managers(like OpenSM with routing tables) to bridge communication across different InfiniBand partitions.

NCCL queries the subnet topology, discovers routing paths, and usesRDMA CM(Connection Manager) to establish GPU transport over routers.

This capability is especially important in data center-scale AI clusters spanning multiple racks or zones, connected viaIB routers like Mellanox SB7800 or QM8700 series.

When NCCL_IB_USE_IB_ROUTER=1 is set:

NCCL includes router-aware route resolution in its path selection logic.

Enables efficientzero-copy communicationacross GPUs in different IB domains, maintaining low latency.

Other Options Explained:

A. Lazy connection establishment– controls when peer connections are made but does not enable cross-subnet reach.

B. GPU Direct RDMA– enables intra-node direct memory access, not applicable for routing across subnets.

C. Static plugin linking– affects how NCCL links plugins, not related to IB topology.

Exact Extract Reference:

Source: NVIDIA NCCL User Guide – Environment Variables Section

Extract: "NCCL_IB_USE_IB_ROUTER: Enables NCCL support for IB routers, required for multi-subnet InfiniBand configurations. Ensures proper routing of collectives over fabric-wide topologies."

In NVIDIA Spectrum-X, congestion is evaluated based on egress queue loads. Spectrum-4 switches assess the load on each egress queue and select the port with the minimal load for packet transmission. This approach ensures that all ports are well-balanced, optimizing network performance and minimizing congestion.

Accessing theBlueField DPU Operating System (OS)is possible throughrshim, either over PCIe or USB, and viaSSH through the OOB interfacewhen in DPU mode.

From theNVIDIA BlueField Software Documentation:

"You can access the BlueField OS through the rshim interface. The rshim module enables host-to-DPU communication either via PCIe (default) or USB."

B. rshim over PCIe: Default when BlueField is installed in a host.

D. rshim over USB: Useful for provisioning or systems without PCIe drivers.

Incorrect Options:

A (NIC mode): BlueField acts as a transparent NIC; OS access is not available to the host.

C (Redfish): Redfish is for out-of-band management, not direct OS-level access.

MLNX-OS, the operating system for NVIDIA's networking devices, defines two primary user account types:adminandmonitor. Theadminaccount has full administrative privileges, allowing for complete configuration and management of the system. Themonitoraccount, on the other hand, is designed for users who need to view system configurations and statuses without making any changes. This separation ensures a clear distinction between users who manage the system and those who monitor its operations.​

Reference Extracts from NVIDIA Documentation:

"There are two user roles or account types: admin and monitor. As 'admin', the user is privileged to run all the available commands. As 'monitor', the user can run commands that show system configuration and status, or set terminal settings."​

MLNX-OS is the network operating system used on NVIDIA’s Mellanox Ethernet switches, including the Spectrum family (e.g., Spectrum-4 switches in the Spectrum-X platform), designed for high-performance Ethernet networking in AI and HPC data centers. MLNX-OS provides a command-line interface (CLI) for configuring and managing switch operations, with user accounts controlling access to various commands and functions. The question asks for the two general user account types in MLNX-OS, which define the primary privilege levels for user access.

According to NVIDIA’s official MLNX-OS documentation, the two general user account types in MLNX-OS are:

monitor: This account type has read-only access, allowing users to view configurations, status, and logs but not modify settings. It is used for monitoring and troubleshooting without risking unintended changes.

admin: This account type has full read-write access, enabling users to view and modify all configurations, execute commands, and manage the switch’s operations. It is intended for administrators with complete control over the system.

These two account types represent the primary privilege levels in MLNX-OS, providing a clear distinction between read-only monitoring and full administrative access.

Exact Extract from NVIDIA Documentation:

“MLNX-OS supports two primary user account types for managing switch operations:

monitor: Users with monitor privileges have read-only access to the system. They can view configuration details, system status, and logs but cannot make changes to the configuration.

admin: Users with admin privileges have full read-write access, allowing them to configure, manage, and troubleshoot all aspects of the switch, including executing privileged commands.These account types ensure secure and controlled access to the switch’s management functions.”—NVIDIA MLNX-OS User Manual

This extract confirms that options B (monitor) and C (admin) are the correct answers. These account types are the standard privilege levels in MLNX-OS, used to manage access for monitoring and administrative tasks on Spectrum switches, including those in Spectrum-X deployments.

Adaptive Routingin InfiniBand networks dynamically selects the optimal path for data packets based on current network conditions, such as congestion levels and link utilization. This approach ensures that traffic is evenly distributed across the network, preventing bottlenecks and maximizing overall throughput.​

By continuously monitoring the network and adjusting routes in real-time, Adaptive Routing enhances performance and reliability, making it the preferred choice for high-performance computing environments where consistent low latency and high bandwidth are critical.​

Virtual Routing and Forwarding (VRF)is the most effective method to achievenetwork segmentation and isolationin a multi-tenant environment.

From theNVIDIA Cumulus Linux Documentation – VRF Section:

"VRF allows multiple instances of routing tables to coexist within the same switch, effectively isolating traffic between tenants or departments."

Each department can:

Operate in its own VRF domain

Have independent routing tables

Maintain strict separation of Layer 3 paths

Incorrect Options:

A (Port Mirroring)– Used for traffic monitoring, not isolation.

C (ACLs)– Useful for fine-grained filtering, but not scalable tenant isolation.

D (LLDP)– Used for neighbor discovery, not security or isolation.

According to theNVIDIA Spectrum-X Switch Operating System (SX_OS) Troubleshooting Guide, theSystem Status LED behavioris a critical indicator of the switch’s internal operational state.

From the document:

“The System Status LED will blink green during system initialization. If the LED continues blinking for more than 5 minutes, it indicates that the Onyx OS hasfailed to load properly. The system may be stuck in the boot process, or the file system may be corrupted.”

This blinking LED beyond normal initialization time indicates that the system has either encountered a failure during software boot or is unable to transition from bootloader to the OS runtime environment (i.e., Onyx).

Key causes include:

Corrupted or missing system files.

Failed firmware or OS upgrade attempts.

Boot device (e.g., eMMC or SSD) issues or corrupted partitions.

Technically, during power-on:

The switch performs POST (Power-On Self Test).

Then the Onyx OS attempts to load from the boot partition.

If the Onyx OS kernel or root filesystem is invalid, the system halts boot, and the LED remains in ablinking state, as no successful OS load confirmation is triggered.

Remediation Steps (as per NVIDIA guide):

Access the switch throughconsoleandmonitor boot logs.

UseONIE recoveryor re-flash a stable Onyx OS version.

Check system storage integrity using built-in diagnostics.

Exact Extract Reference:

Source: NVIDIA SX_OS 3.9.3000 Documentation

Topic: Troubleshooting System Status LED

Extract: "If the LED blinks for more than 5 minutes and the switch is not accessible via CLI, the Onyx software failed to load properly and recovery procedures must be initiated."

=============

FromNVIDIA Performance Tuning Guide (ib_write_bw Tool Usage):

"-S <SL>: Specifies the Service Level (SL) to use for the InfiniBand traffic. SL is used for setting priority and mapping to virtual lanes (VLs) on the IB fabric."

This flag is useful when testing QoS-aware setups or validating SL/VL mappings.

Incorrect Options:

A– No such flag for burst size.

B– -q defines number of QPs.

C– --rate or -R is used for rate-limiting.

The UFM Cyber-AI platform is an advanced feature of NVIDIA's Unified Fabric Manager designed to enhance security and reliability in InfiniBand data centers. It leverages AI-powered analytics and machine learning techniques to detect security threats, operational anomalies, and predict potential network failures. By analyzing real-time and historical telemetry data, UFM Cyber-AI can identify abnormal system behaviors, performance degradations, and usage profile changes. This proactive approach enables administrators to address issues before they escalate, ensuring the integrity and uptime of the data center.​

Reference Extracts from NVIDIA Documentation:

"The NVIDIA Unified Fabric Manager (UFM) Cyber-AI platform offers enhanced and real-time network telemetry, combined with AI-powered intelligence and advanced analytics. It enables IT managers to discover operational anomalies and even predict network failures."​

"UFM Cyber-AI uses machine learning (ML) techniques and AI models for anomaly detection and prediction to learn the lifecycle patterns of data center network components."​

“The NVIDIA UFM platforms revolutionize data center networking management by combining enhanced, real-time network telemetry with AI-powered cyber intelligence and analytics to support scale-out InfiniBand data centers. ... The UFM Cyber-AI platform takes fabric management to the next level by adding an analytics layer powered by artificial intelligence. It enables data center operators to proactively monitor and manage the InfiniBand fabric, predicting and preventing potential failures, optimizing performance, and enhancing security. By analyzing telemetry data and historical patterns, UFM Cyber-AI can detect anomalies that may indicate security threats or operational issues, providing actionable insights to prevent downtime.”