NCP-AIN NVIDIA-Certified Professional AI Networking Questions and Answers

To perform Flow Analysis with NetQ, the following prerequisites must be met:​

Cumulus Linux Version: NetQ Flow Analysis requires Cumulus Linux 5.x or later.​

Switch Hardware: The feature is supported on Spectrum-2 and later switch models.​

Lifecycle Management (LCM): LCM must be enabled to utilize Flow Analysis capabilities.​

These requirements ensure compatibility and proper functioning of the Flow Analysis feature within NetQ.​

NVIDIA BlueField Data Processing Units (DPUs)are integral to securing AI infrastructures, especially in environments requiring stringent data privacy and security measures. BlueField DPUs offload and accelerate critical infrastructure tasks such as encryption, firewall enforcement, and intrusion detection, thereby isolating sensitive data paths from potential threats.​

In the context of AI workloads, BlueField DPUs enable secure and efficient data movement between GPUs and storage systems, ensuring that sensitive information, like financial data, is protected during both training and inference processes. Their integration into NVIDIA's reference architectures provides a hardware root of trust, essential for maintaining data integrity and compliance with security standards.​

NVIDIA NetQ is a highly scalable network operations toolset that provides visibility, troubleshooting, and validation of networks in real-time. It delivers actionable insights and operational intelligence about the health of data center networks—from the container or host all the way to the switch and port—enabling a NetDevOps approach.

NetQ can be used as the functional test platform for the network CI/CD in conjunction with NVIDIA Air. Customers benefit from testing the new configuration with NetQ in the NVIDIA Air environment (“digital twin”) and fix errors before deploying to their production. ​

The tool specifically designed for testingInfiniBand connectivityis **ibping**. It functions similarly to the traditional ping utility but is optimized forInfiniBand fabrics.

From theNVIDIA InfiniBand Diagnostic Utilities Documentation:

"ibping tests the connectivity of InfiniBand nodes by sending management datagrams (MADs) and verifying the response from the destination LID or GUID."

Tests basicnode-to-nodereachability

Supports testing viaLID, GUID, orport number

Helps verify subnet manager routing and fabric health

Incorrect Options:

pingandtracerouteare IP-based, not fabric-aware.

ibnetdiscovermaps topology but doesn't test live connectivity.

The UFM Cyber-AI platform is an advanced feature of NVIDIA's Unified Fabric Manager designed to enhance security and reliability in InfiniBand data centers. It leverages AI-powered analytics and machine learning techniques to detect security threats, operational anomalies, and predict potential network failures. By analyzing real-time and historical telemetry data, UFM Cyber-AI can identify abnormal system behaviors, performance degradations, and usage profile changes. This proactive approach enables administrators to address issues before they escalate, ensuring the integrity and uptime of the data center.​

Reference Extracts from NVIDIA Documentation:

"The NVIDIA Unified Fabric Manager (UFM) Cyber-AI platform offers enhanced and real-time network telemetry, combined with AI-powered intelligence and advanced analytics. It enables IT managers to discover operational anomalies and even predict network failures."​

"UFM Cyber-AI uses machine learning (ML) techniques and AI models for anomaly detection and prediction to learn the lifecycle patterns of data center network components."​

“The NVIDIA UFM platforms revolutionize data center networking management by combining enhanced, real-time network telemetry with AI-powered cyber intelligence and analytics to support scale-out InfiniBand data centers. ... The UFM Cyber-AI platform takes fabric management to the next level by adding an analytics layer powered by artificial intelligence. It enables data center operators to proactively monitor and manage the InfiniBand fabric, predicting and preventing potential failures, optimizing performance, and enhancing security. By analyzing telemetry data and historical patterns, UFM Cyber-AI can detect anomalies that may indicate security threats or operational issues, providing actionable insights to prevent downtime.”

Adaptive Routingin InfiniBand networks dynamically selects the optimal path for data packets based on current network conditions, such as congestion levels and link utilization. This approach ensures that traffic is evenly distributed across the network, preventing bottlenecks and maximizing overall throughput.​

By continuously monitoring the network and adjusting routes in real-time, Adaptive Routing enhances performance and reliability, making it the preferred choice for high-performance computing environments where consistent low latency and high bandwidth are critical.​

TheNVIDIA Network Operatoris designed to simplify and automate the deployment and management of networking components in Kubernetes clusters, particularly those utilizing NVIDIA Spectrum-X switches. It manages the installation and configuration of necessary drivers, plugins, and other networking resources to enable features like RDMA and GPUDirect RDMA, which are essential for high-performance AI workloads.​

By leveraging Kubernetes Custom Resource Definitions (CRDs) and the Operator Framework, the Network Operator ensures that networking components are consistently and correctly configured across the cluster, reducing manual intervention and potential configuration errors.​

The NVIDIA Spectrum-X Reference Architecture (RA) 1.0.1 is designed for Ethernet AI cloud deployments and includes the SN5600 Spectrum-4 switches and BlueField-3 SuperNICs. This architecture supports adaptive routing and DOCA programmable congestion control (PCC) for lossless RoCE traffic, optimizing performance for AI workloads.

The SN5600 switch offers 64 ports of 800GbE in a dense 2U form factor, providing high throughput and low latency essential for AI applications.

Inmulti-subnet InfiniBand environments, AI training clusters are segmented across network zones (or subnets). Direct GPU-to-GPU communication (especially for collective ops like AllReduce, Broadcast, etc.) requires inter-subnet reachability. NCCL supports this via theInfiniBand Router (IB Router)feature.

From theNCCL User Guide – Environment Variables Section:

"NCCL_IB_USE_IB_ROUTER: Enables NCCL support for IB routers which are used in multi-subnet InfiniBand fabrics. When enabled, NCCL can traverse IB subnets using a properly configured IB router."

This is critical because without IB Router support:

NCCL would be restricted to intra-subnet GPU collectives.

Multi-node training across subnets would fail or fall back to slower TCP fallback mechanisms.

Technical Explanation:

IB Routers usesubnet managers(like OpenSM with routing tables) to bridge communication across different InfiniBand partitions.

NCCL queries the subnet topology, discovers routing paths, and usesRDMA CM(Connection Manager) to establish GPU transport over routers.

This capability is especially important in data center-scale AI clusters spanning multiple racks or zones, connected viaIB routers like Mellanox SB7800 or QM8700 series.

When NCCL_IB_USE_IB_ROUTER=1 is set:

NCCL includes router-aware route resolution in its path selection logic.

Enables efficientzero-copy communicationacross GPUs in different IB domains, maintaining low latency.

Other Options Explained:

A. Lazy connection establishment– controls when peer connections are made but does not enable cross-subnet reach.

B. GPU Direct RDMA– enables intra-node direct memory access, not applicable for routing across subnets.

C. Static plugin linking– affects how NCCL links plugins, not related to IB topology.

Exact Extract Reference:

Source: NVIDIA NCCL User Guide – Environment Variables Section

Extract: "NCCL_IB_USE_IB_ROUTER: Enables NCCL support for IB routers, required for multi-subnet InfiniBand configurations. Ensures proper routing of collectives over fabric-wide topologies."

Before upgrading the DOCA SDK on aBlueField DPU, it ismandatory to uninstall the existing OFED driversto prevent compatibility conflicts.

From theNVIDIA DOCA Installation Guide:

"Before upgrading DOCA or BlueField-related software, you must remove existing OFED packages using: /usr/sbin/ofed_uninstall.sh -force."

This ensures:

Clean driver state

No residual kernel modules or userspace libraries

Proper registration of new DOCA/OFED versions

Incorrect Options:

AandCmay not resolve conflicts.

Dinstalls but doesn’t remove conflicting packages.

TheNVIDIA DGX SuperPODis a turnkey AI supercomputing infrastructure designed for large-scale deep learning and high-performance computing workloads. It integrates multiple DGX systems with high-speed networking and storage solutions, providing a scalable and efficient platform for AI research institutions. The architecture supports rapid deployment and is optimized for training complex models, making it the ideal choice for environments demanding top-tier AI performance.​

Spectrum-X achieves network isolation in multi-tenant environments by implementing Layer 3 Virtual Network Identifiers (L3VNIs) per Virtual Routing and Forwarding (VRF) instance. This approach allows each tenant to have a separate routing table and network segment, ensuring that traffic is isolated and secure between tenants.​

Reference Extracts from NVIDIA Documentation:

"Spectrum-X enhances multi-tenancy with performance isolation to ensure tenants' AI workloads perform optimally and consistently."​

Direct Memory Access (DMA) in InfiniBand networks allows data to be transferred directly between the memory of two devices without involving the CPU. This capability significantly reduces CPU overhead, lowers latency, and increases throughput, making it ideal for AI workloads that demand efficient data transfers.

Within the Spectrum-X platform, the NVIDIA BlueField-3 SuperNIC is responsible for reordering out-of-order packets. When RoCE adaptive routing is employed, packets may arrive at their destination out of order due to dynamic path selection. The BlueField-3 SuperNIC handles this by reassembling the packets in the correct order at the transport layer, ensuring that the application receives data seamlessly.​

Reference Extracts from NVIDIA Documentation:

"As different packets of the same flow travel through different paths of the network, they may arrive out of order to their destination. At the RoCE transport layer, the BlueField-3 DPU takes care of the out-of-order packets and forwards the data to the application in order."​

"The BlueField-3 SuperNIC offers adaptive routing, out-of-order packet handling and optimized congestion control."​

The NVIDIA Spectrum-X networking platform is an Ethernet-based solution optimized for AI workloads, combining Spectrum-4 switches, BlueField-3 SuperNICs, and software like DOCA and NetQ to deliver high performance, low latency, and efficient data transfer. A key feature of Spectrum-X is its adaptive routing, which dynamically selects the least-congested paths for packet transmission to maximize bandwidth and minimizelatency. However, this per-packet load balancing can result in packets arriving out of order at the destination, necessitating a mechanism to reorder them for seamless application performance. The question asks which Spectrum-X component is responsible for reordering these out-of-order packets.

According to NVIDIA’s official documentation, theBlueField-3 SuperNICis the component responsible for reordering out-of-order packets in the Spectrum-X platform. The SuperNIC, a network accelerator designed for hyperscale AI workloads, handles packet reordering at the RDMA over Converged Ethernet (RoCE) transport layer. It uses its processing capabilities to transparently reorder packets and place them in the correct sequence in the host memory, ensuring that adaptive routing’s out-of-order delivery is invisible to the application. This is critical for maintaining predictable performance in AI workloads, particularly for GPU-to-GPU communication in Spectrum-X networks.

Exact Extract from NVIDIA Documentation:

“The Spectrum-4 switches are responsible for selecting the least-congested port for data transmission on a per-packet basis. As different packets of the same flow travel through different paths of the network, they may arrive out of order to their destination. The BlueField-3 SuperNIC transforms any out-of-order data at the RoCE transport layer, transparently delivering in-order data to the application.”

—NVIDIA Technical Blog: Turbocharging Generative AI Workloads with NVIDIA Spectrum-X Networking Platform

This extract confirms that option A, the SuperNIC (specifically the BlueField-3 SuperNIC), is the correct answer. The SuperNIC’s role in reordering packets ensures that the adaptive routing implemented by Spectrum-4 switches does not compromise application performance, maintaining high effective bandwidth and low tail latency for AI workloads.

EVPN Multi-homingenablesactive-active redundancy without inter-switch linksby usingoverlay routingover VXLAN and distributed control plane using BGP EVPN.

From the officialNVIDIA Cumulus Linux EVPN Multihoming Documentation:

"EVPN multihoming allows multiple Top-of-Rack (ToR) switches to connect to the same server while maintaining full layer-2 redundancy without the need for inter-switch links or traditional MLAG configuration."

Key benefits:

Simplified topology (no ISL/peer-link needed)

BGP-based control plane

Fast convergence

Active-active links per host NIC

Incorrect Options:

MLAGrequires ISL between switches and peer-link configuration.

VSS(Virtual Switching System) is a Cisco term, not supported in NVIDIA networking.

Before upgrading Cumulus Linux, it's essential to back up configuration files to a different server. The /etc directory is the primary location for all configuration data in Cumulus Linux. Specifically, the following files and directories should be backed up:​

/etc/frr/ - Routing application (responsible for BGP and OSPF)

/etc/hostname - Configuration file for the hostname of the switch

/etc/network/ - Network configuration files, most notably /etc/network/interfaces and /etc/network/interfaces.d/

/etc/cumulus/acl - Access control list configurations​

Cumulus Linux is a network operating system used on NVIDIA Spectrum switches, including those in the Spectrum-X platform, to provide a Linux-based environment forEthernet networking in AI and HPC data centers. When upgrading Cumulus Linux to a new version, it’s critical to migrate specific configuration files to preserve network settings and ensure continuity. The question asks for the two configuration file locations that should be migrated from the old installation during an upgrade.

According to NVIDIA’s official Cumulus Linux documentation, the key directories containing configuration files that should be migrated during an upgrade are /etc/cumulus/acl (for access control list configurations) and /etc/network (for network interface configurations). These directories store critical network settings that define the switch’s behavior, such as ACL rules and interface settings, which must be preserved to maintain network functionality after the upgrade.

Exact Extract from NVIDIA Documentation:

“When upgrading Cumulus Linux, you must back up and migrate specific configuration files to ensure continuity of network settings. The following directories should be included in the backup:

/etc/cumulus/acl: Contains access control list (ACL) configuration files that define packet filtering and security policies.

/etc/network: Contains network interface configuration files, such as interfaces and ifupdown2 settings, which define the network interfaces and their properties.Back up these directories before upgrading and restore them after the new version is installed to maintain consistent network behavior.”—NVIDIA Cumulus Linux Upgrade Guide

This extract confirms that options A and B are the correct answers, as /etc/cumulus/acl and /etc/network contain essential configuration files that must be migrated during a Cumulus Linux upgrade. These files ensure that ACL policies and network interface settings are preserved, which are critical for Spectrum-X configurations in AI networking environments.

MLNX_OFED (Mellanox OpenFabrics Enterprise Distribution) is an NVIDIA-tested and packaged version of the OpenFabrics Enterprise Distribution (OFED) for Linux. It provides the necessary drivers and tools to support InfiniBand and Ethernet interconnects using the same RDMA (Remote Direct Memory Access) and kernel bypass APIs. MLNX_OFED enables high-performance networking capabilities essential for AI clusters, including support for up to 400Gb/s InfiniBand and RoCE (RDMA over Converged Ethernet).​

Reference Extracts from NVIDIA Documentation:

"MLNX_OFED is an NVIDIA tested and packaged version of OFED that supports two interconnect types using the same RDMA (remote DMA) and kernel bypass APIs called OFED verbs – InfiniBand and Ethernet."​

"Up to 400Gb/s InfiniBand and RoCE (based on the RDMA over Converged Ethernet standard) over 10/25/40/50/100/200/400GbE are supported."​

In NVIDIA Spectrum-X, congestion is evaluated based on egress queue loads. Spectrum-4 switches assess the load on each egress queue and select the port with the minimal load for packet transmission. This approach ensures that all ports are well-balanced, optimizing network performance and minimizing congestion.