For IT professionals, system administrators, DevOps engineers, and data-center operators moving into AI — not data scientists. NCA-AIIO proves you understand the physical and operational reality of AI computing: the GPUs, the networks, the power and cooling, and the software that schedules and monitors them.
The NCA-AIIO is NVIDIA's entry-level infrastructure certification, and it fills a gap no cloud-vendor exam covers: it tests whether you understand the physical and operational reality of AI computing — the GPUs, the networks that stitch them into clusters, the power and cooling that keep them alive, and the software that schedules, monitors, and carves them up for users. Where AWS and Azure certifications assume someone else built the data center, NCA-AIIO asks you to be that someone: to know why a training cluster needs InfiniBand or Spectrum-X while an inference fleet may not, why a rack of eight-GPU servers can draw more power than a small office building, and how tools like Base Command Manager, DCGM, Kubernetes, and Slurm turn a pile of hardware into a working AI factory.
You will not write code or train a model on this exam. You will be asked to compare architectures, match workloads to hardware, pick the right network fabric, and choose the right monitoring or virtualization approach for a scenario. It is a breadth exam: fifty questions in sixty minutes across three domains, so roughly seventy seconds per question. The questions are short; the trap is confusing NVIDIA's many similarly-named technologies (NVLink vs. InfiniBand, MIG vs. vGPU, BasePOD vs. SuperPOD, TensorRT vs. Triton). This guide is organized exactly around the three official domains and their weights, and it teaches every topic in the exam blueprint.
| Exam Fact | Detail |
|---|---|
| Exam code | NCA-AIIO |
| Certification earned | NVIDIA-Certified Associate: AI Infrastructure and Operations |
| Questions | 50 |
| Time limit | 60 minutes |
| Passing score | Not published (candidates commonly cite ~70%) |
| Question formats | Multiple choice (single and multiple response) |
| Delivery | Online, remote-proctored via Certiverse |
| Cost | $125 USD (as of July 2026) |
| Validity | 2 years |
| Language | English |
| Domains | Essential AI Knowledge 38% · AI Infrastructure 40% · AI Operations 22% |
Read the parts in order:
Because NVIDIA's product names overlap heavily, the comparison tables in Parts 2 and 3 are the highest-value revision material in this guide — rebuild them from memory before exam day.
Throughout the guide you'll see four kinds of callouts:
Tactical advice about how a topic is tested — question patterns, common traps, and wording to watch for.
The one thing you must remember from a section, even if you forget everything else.
A fact or distinction that candidates frequently mix up.
Useful context or clarification that rounds out your understanding.
A good self-test for this exam: for any NVIDIA product name, can you say in one sentence what layer it lives at (silicon, interconnect, system, software, service) and what problem it solves? If you can do that for everything in this guide, you are ready.
This domain tests whether you understand what AI workloads are and why they demand specialized infrastructure. It covers the AI/ML/DL hierarchy, GPU versus CPU architecture, the differences between training and inference, the full NVIDIA software stack, the AI lifecycle, and how AI is transforming industries. Expect roughly 19 of your 50 questions here.
The exam expects you to place these three terms in their correct nesting order and recognize examples of each.
Artificial intelligence (AI) is the broadest term: any technique that lets machines mimic human intelligence — including hand-written rule systems that involve no learning at all. Machine learning (ML) is the subset of AI where systems learn patterns from data instead of following explicit rules; classic examples are linear regression, decision trees, random forests, and gradient-boosted models like XGBoost, which typically work on structured, tabular data. Deep learning (DL) is the subset of ML that uses multi-layered neural networks. Depth lets these networks learn their own feature representations from raw, unstructured data — pixels, audio waveforms, text — instead of relying on human-engineered features. Convolutional neural networks (CNNs) dominate image tasks, recurrent networks (RNNs/LSTMs) handled sequences historically, and the transformer architecture now dominates language and increasingly everything else.
Generative AI sits atop deep learning: large transformer-based models (LLMs for text, diffusion models for images) trained on massive datasets to create new content rather than merely classify or predict. Generative AI is the workload driving today's infrastructure boom — and this exam exists because of it. Training a frontier LLM can occupy thousands of GPUs for months; serving one to millions of users is a permanent, latency-sensitive fleet operation.
Why does this taxonomy matter for infrastructure? Because compute intensity rises sharply as you descend the hierarchy. Classical ML often runs comfortably on CPUs. Deep learning is built on dense matrix multiplication — an operation GPUs accelerate by orders of magnitude — and modern generative models are so large they cannot fit, let alone train, on any single device.
AI ⊃ ML ⊃ DL, with generative AI as the deep-learning workload class reshaping infrastructure. Deep learning is the reason GPUs took over: its core math is massively parallel matrix multiplication.
This is bedrock material for the whole exam. A CPU is a latency-optimized generalist: a handful of powerful cores (typically 16–128 in servers), large caches, sophisticated branch prediction, and deep out-of-order pipelines, all designed to finish one thread of unpredictable, branching logic as fast as possible. A GPU is a throughput-optimized specialist: thousands of simpler cores (an H100 has 16,896 CUDA cores plus 528 Tensor Cores) designed to execute the same operation on huge blocks of data simultaneously — a model called SIMT, single instruction, multiple threads.
The second half of the story is memory. GPUs use high-bandwidth memory (HBM) stacked directly beside the processor die: an H100 SXM delivers about 3.35 TB/s from 80 GB of HBM3, and Blackwell-generation GPUs push past 8 TB/s with HBM3e — versus a few hundred GB/s for a CPU's DDR5 channels. Deep learning is often memory-bandwidth-bound, so this gap matters as much as core count.
The third piece is specialized silicon: Tensor Cores are dedicated units that execute matrix-multiply-accumulate operations in mixed precision (FP16, BF16, FP8, and FP4 on Blackwell) at rates far beyond general-purpose cores. Lower precision means more operations per second and less memory per value, which is why modern training uses mixed precision and modern inference is quantized.
| Characteristic | CPU | GPU |
|---|---|---|
| Design goal | Low latency for serial, branching tasks | High throughput for parallel tasks |
| Cores | Few (tens), complex, out-of-order | Thousands, simple, SIMT |
| Memory | Large-capacity DDR5, ~0.2–0.6 TB/s | HBM2e/HBM3/HBM3e, 2–8 TB/s |
| Cache/control silicon | Large share of the die | Small share; die spent on ALUs |
| Special units | Vector extensions (AVX) | Tensor Cores for mixed-precision matrix math |
| Best at | OS, databases, orchestration, preprocessing, branching logic | Matrix math: DL training/inference, HPC, rendering, analytics |
| In an AI server | Host: launches kernels, handles I/O, runs the OS | Accelerator: does the heavy math |
Note that the CPU never disappears — every DGX system pairs its eight GPUs with dual server CPUs that run the OS, feed data, and launch GPU work. The exam calls this the accelerated computing model: CPU as host, GPU as accelerator. NVIDIA's Grace CPU (Arm-based) extends this by pairing a CPU and GPU coherently in Grace Hopper (GH200) and Grace Blackwell (GB200) superchips, linked by NVLink-C2C at 900 GB/s so the GPU can use CPU memory at high speed.
When a question asks why GPUs suit deep learning, the expected answers are massive parallelism for matrix operations, Tensor Cores, and high memory bandwidth. When a scenario involves sequential, branching, or transactional work (databases, web servers, orchestration logic), the CPU is the right answer — don't reflexively pick the GPU.
Hardware alone accelerates nothing; software has to exploit it. CUDA is NVIDIA's parallel computing platform and programming model — the layer that lets developers write kernels that run across thousands of GPU cores, and the foundation every higher layer builds on. In practice, few AI practitioners write CUDA directly. They use frameworks (PyTorch, TensorFlow, JAX) that call CUDA-X libraries: cuDNN for deep-learning primitives, cuBLAS for linear algebra, NCCL (NVIDIA Collective Communications Library) for multi-GPU communication operations like all-reduce, and RAPIDS (cuDF, cuML) for GPU-accelerated data science that mirrors the pandas and scikit-learn APIs.
CUDA's twenty-year head start and ecosystem lock-in are themselves exam-relevant: they explain why "the NVIDIA platform" means hardware plus software, and why compatibility between driver versions, CUDA versions, and framework builds is an operational concern (one that containers, covered in Part 3, largely solve).
NCCL matters more than it looks. Distributed training is dominated by collective operations — all-reduce to average gradients across GPUs every step. NCCL automatically discovers the fastest paths (NVLink inside a node, InfiniBand or RoCE between nodes) and is the software glue that makes the fabrics in Part 2 actually pay off.
This distinction drives more exam questions than any other single concept, because every architecture choice downstream depends on it.
Training is the process of learning model weights: feed batches of data forward, compute a loss, backpropagate gradients, update weights, repeat for days to months. Its infrastructure profile: enormous sustained compute (the largest GPUs you can get), large GPU memory (weights plus gradients plus optimizer states plus activations), extremely fast GPU-to-GPU interconnects (because thousands of GPUs must exchange gradients every step, synchronously), high-throughput storage to keep data pipelines full, and checkpointing — periodically writing the full model state to storage so a hardware failure doesn't lose weeks of work. Training is throughput-oriented and batch-scheduled: jobs run for a long time, and a single slow link or flapping GPU stalls the entire synchronized job.
Inference is using the trained model to serve predictions. Its profile: latency-sensitive (users are waiting), elastic (traffic varies by time of day), memory-dominated for LLMs (weights plus the growing KV cache per active conversation), and cost-per-query-driven. Inference optimizations are their own discipline: quantization (FP8/INT8/INT4 to shrink memory and boost throughput), batching (grouping requests to keep the GPU busy), and model-level tricks. Inference can run on smaller or partitioned GPUs (L4, L40S, or MIG slices of larger GPUs), at the edge, and rarely needs the exotic inter-node fabrics training demands — though the largest LLMs still need multi-GPU tensor parallelism within a node.
Fine-tuning sits between: adapting a pretrained model on domain data. Full fine-tuning resembles a short training job; parameter-efficient methods (LoRA) shrink the hardware requirement dramatically, sometimes to a single GPU.
Training = throughput, biggest GPUs, fastest fabrics, checkpointing, batch schedulers. Inference = latency, elasticity, quantization, batching, right-sized or partitioned GPUs. Any scenario question that mentions "users waiting for responses" is an inference question; "weeks on thousands of GPUs" is training.
The exam expects you to identify the role of each layer of NVIDIA's software platform. Learn this table cold — the names are the single most-tested factual content in Domain 1.
| Layer | Product | What it does |
|---|---|---|
| Programming platform | CUDA | Parallel computing platform and API; the foundation everything else builds on |
| Core libraries | cuDNN, cuBLAS, NCCL, RAPIDS | DL primitives, linear algebra, multi-GPU collectives, GPU data science |
| Inference optimizer | TensorRT (and TensorRT-LLM) | Compiles trained models for fast inference: layer fusion, precision calibration (FP8/INT8), kernel auto-tuning |
| Inference server | Triton Inference Server | Serves models in production: multi-framework (PyTorch, ONNX, TensorRT), dynamic batching, concurrent models, HTTP/gRPC, metrics |
| Model/dev frameworks | NeMo | Framework and microservices for building, customizing (fine-tuning, RAG), and deploying generative AI models |
| Content hub | NGC Catalog | Registry of GPU-optimized containers, pretrained models, Helm charts, and SDKs — the standard way to obtain tested, versioned AI software |
| Enterprise platform | NVIDIA AI Enterprise | Licensed, supported suite of the entire AI stack (including NIM microservices) certified for VMware, mainstream servers, and clouds |
| Cluster management | Base Command Manager | Provisions, images, monitors, and manages AI clusters end to end (covered in Part 3) |
Two pairs cause most of the confusion. First, TensorRT vs. Triton: TensorRT optimizes a model (it's a compiler/runtime that makes one model faster); Triton serves models (it's the production server that handles requests, batching, and scaling, and can host TensorRT-optimized models among others). Second, NGC vs. NVIDIA AI Enterprise: NGC is the free catalog where you pull containers and models; AI Enterprise is the commercially licensed, enterprise-supported edition of the software stack with security patches, long-term support branches, and certified deployment on enterprise platforms — the thing a bank buys so its AI stack has an SLA. Within AI Enterprise, NIM (NVIDIA Inference Microservices) packages optimized models as ready-to-run containers with standard APIs.
TensorRT optimizes, Triton serves. NGC distributes, AI Enterprise supports. If a question says "reduce inference latency of a trained model," that's TensorRT; "deploy multiple models from different frameworks behind one production endpoint with dynamic batching" is Triton.
Triton's signature features are worth memorizing verbatim: multi-framework support, dynamic batching (transparently grouping incoming requests), concurrent model execution (multiple models or instances on one GPU), and Prometheus metrics for monitoring.
DGX systems are NVIDIA's flagship, fully integrated AI servers: eight flagship GPUs (A100, H100/H200, or B200 by generation) linked by NVLink and NVSwitch so all eight communicate at full bandwidth, dual host CPUs, high-speed NICs (one per GPU for east-west traffic), NVMe storage, and the full NVIDIA software stack preinstalled. A DGX H100 delivers 32 petaFLOPS of FP8 AI performance and draws up to ~10.2 kW; a DGX B200 raises that to 72 petaFLOPS FP8 training performance at ~14.3 kW. DGX is the building block of the BasePOD and SuperPOD reference architectures in Part 2. Alongside DGX, HGX is the board/platform partners use to build their own eight-GPU servers, and MGX is a modular reference design for varied server configurations.
Not every workload needs an eight-GPU flagship. The data-center GPU line is segmented: H100/H200/B200 for training and large-model inference; L40S for versatile compute-plus-graphics work; L4 for efficient, low-power inference and video; A100 as the previous-generation training workhorse that introduced MIG partitioning. At the far edge, the Jetson family embeds GPUs for robotics and edge AI.
The exam frames AI delivery as a lifecycle and asks which infrastructure and software supports each stage:
The whole loop — MLOps — is what turns one-off experiments into a production discipline, and NVIDIA positions its stack plus partner MLOps tools (Kubeflow, MLflow, Run:ai) across it.
The lifecycle also explains why AI infrastructure is never "just GPUs." Data prep stresses storage and CPUs; training stresses GPU compute and fabric; inference stresses latency and elasticity; monitoring stresses telemetry. Balanced systems, not maximum GPU counts, are the design goal — a theme Domain 2 tests hard.
Domain 1 closes with the business layer: what drives and blocks enterprise AI adoption, and where AI is transforming industries. Adoption drivers the exam recognizes: explosion of data, availability of accelerated computing, pretrained foundation models lowering the barrier to entry (you fine-tune instead of training from scratch), and competitive pressure. Common obstacles: shortage of skilled staff, data quality and governance, cost and energy footprint of infrastructure, integration with legacy systems, and trust/explainability/regulatory concerns.
Know one signature use case per major industry, because questions present a scenario and ask what class of AI (and therefore infrastructure) it implies:
Scenario-to-workload matching is quick marks. Recommendation engines and fraud detection = high-throughput, latency-bound inference. Drug discovery and autonomous-vehicle perception = massive training. Defect detection on a factory line = edge inference. Let the latency and location clues pick the infrastructure.
The heaviest domain — expect about 20 questions — covers the physical estate: sizing hardware for training, scaling to multi-node clusters, every network technology in the NVIDIA portfolio, DPUs, the BasePOD/SuperPOD reference architectures, power, cooling, facilities, energy efficiency, and the on-prem versus cloud decision.
The first infrastructure skill is estimating what a given model needs, and GPU memory is the binding constraint. The arithmetic is simple and exam-friendly: a model's parameters must live in GPU memory, and training multiplies the footprint because you also hold gradients, optimizer states, and activations.
| Item | Bytes per parameter | 7B model | 70B model |
|---|---|---|---|
| Inference, FP16/BF16 weights | 2 | 14 GB | 140 GB |
| Inference, INT8 quantized | 1 | 7 GB | 70 GB |
| Inference, INT4 quantized | 0.5 | 3.5 GB | 35 GB |
| Training, mixed precision + Adam (weights + gradients + optimizer states) | ~16 | ~112 GB | ~1.1 TB |
| Add activations, KV cache, CUDA overhead | varies | +tens of GB | +hundreds of GB |
Read the consequences off the table: a 7B model infers on a single 24 GB GPU (quantized, even smaller) but needs multiple 80 GB GPUs just to train. A 70B model doesn't even fit for FP16 inference on one 80 GB GPU — it needs tensor parallelism across at least two — and training it requires many nodes. The ~16 bytes/parameter training figure comes from FP16 weights (2) + FP16 gradients (2) + Adam optimizer states in FP32 (master weights 4 + momentum 4 + variance 4).
Beyond memory: match GPU generation and class to the task (H100/B200 for training; L4/L40S or MIG slices for inference), provision CPU and system RAM for data loading (a starved data pipeline idles the GPUs), and use NVMe local scratch plus high-throughput parallel network storage so I/O never becomes the bottleneck. Checkpoint writes for large models are bursty multi-terabyte events — storage must absorb them without stalling training.
Memorize "2 bytes per parameter for FP16 inference, roughly 16 for Adam mixed-precision training." Nearly every sizing question is one multiplication away from those two numbers.
When one GPU isn't enough, you scale in two directions. Scale up within a server: NVLink/NVSwitch makes eight GPUs behave like one large accelerator with fast peer-to-peer memory access. Scale out across servers: nodes connected by InfiniBand or high-performance Ethernet form a cluster, coordinated by NCCL and a scheduler.
Know the three parallelism strategies by name and their infrastructure implications:
Large LLM training combines all three (often called 3D parallelism): tensor parallel within nodes over NVLink, pipeline and data parallel across nodes over InfiniBand. This is why the fabric hierarchy exists — and why the exam cares that you know NVLink is intra-node and InfiniBand/Spectrum-X are inter-node.
Synchronous training makes the cluster only as fast as its slowest participant: one degraded GPU, one flaky cable, one congested switch port slows every GPU in the job. That fact motivates lossless fabrics, rail-optimized topologies, health checking (DCGM), and burn-in testing.
Tensor parallelism → NVLink → inside the node. Data/pipeline parallelism → InfiniBand or Spectrum-X → between nodes. If a question pairs a parallelism strategy with a fabric, this mapping is the answer.
A production AI cluster is four subsystems, and the exam expects you to name all four and resist the temptation to treat "cluster" as "GPUs only":
AI training traffic is unlike enterprise traffic: it's east-west (server-to-server), bursty, synchronized (all GPUs communicate at once during all-reduce), and composed of a few enormous "elephant flows" rather than many small ones. Standard data-center Ethernet handles such traffic poorly — hash-based load balancing collides elephant flows onto the same links, congestion causes packet loss, loss causes retransmission, and the whole synchronized job stalls. Hence NVIDIA's mantra that in AI, the network defines the data center.
RDMA (Remote Direct Memory Access) is the foundational protocol concept: it lets one server's NIC read/write another server's memory directly, bypassing both operating systems' kernels and CPUs — zero-copy, microsecond latency, no CPU overhead. GPUDirect RDMA extends this so the NIC moves data straight to and from GPU memory, skipping the host entirely; GPUDirect Storage does the same for NVMe/storage reads. RDMA is native to InfiniBand; on Ethernet it's delivered as RoCE (RDMA over Converged Ethernet) — RoCE v2 encapsulates RDMA in UDP/IP so it's routable, but it needs a well-engineered, effectively lossless Ethernet underneath (PFC priority flow control and ECN congestion notification), which is exactly the engineering Spectrum-X packages.
Now the fabric portfolio itself — the most heavily tested table in the exam:
| Fabric | What it is | Bandwidth (current gen) | Scope | Key features | Choose when |
|---|---|---|---|---|---|
| NVLink + NVSwitch | Proprietary GPU-to-GPU interconnect | 900 GB/s per H100 (gen 4); 1.8 TB/s per Blackwell GPU (gen 5) | Inside a node / NVL72 rack domain | Memory-semantic peer access; NVSwitch gives all-to-all full bandwidth; NVLink-C2C links Grace CPU to GPU | Tensor parallelism; making 8 (or 72) GPUs act as one |
| InfiniBand (Quantum-2 / Quantum-X800) | Purpose-built lossless HPC/AI fabric | NDR 400 Gb/s per port; XDR 800 Gb/s | Inter-node compute fabric | Native RDMA, credit-based lossless flow control, adaptive routing, SHARP in-network aggregation, managed by a subnet manager (UFM) | Maximum training performance; SuperPOD default |
| Spectrum-X Ethernet | AI-optimized Ethernet platform: Spectrum-4 switch + BlueField-3 SuperNIC, engineered end to end | 400–800 Gb/s per port | Inter-node compute fabric | RoCE with adaptive routing, per-packet spraying + reordering at the SuperNIC, telemetry-based congestion control; ~1.6× effective AI bandwidth vs. standard Ethernet | AI performance on Ethernet operational standards; multi-tenant AI clouds |
| Standard Ethernet | Conventional DC network | 10–100+ Gb/s typical | Storage, management, north-south | TCP/IP; loss-tolerant; ubiquitous tooling | Management networks, storage networks, general traffic — not the compute fabric of a large training cluster |
Two InfiniBand specifics recur on the exam. SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) performs reduction math inside the switches, so all-reduce data crosses the network once instead of twice — in-network computing. And InfiniBand is centrally managed by a subnet manager, unlike Ethernet's distributed protocols.
A topology note worth one mark: AI compute fabrics are built as non-blocking fat-tree (leaf-spine/Clos) networks, often rail-optimized — each of a node's eight NICs connects to its own "rail" of switches, mirroring how NCCL schedules traffic.
"Highest-performance fabric for a dedicated training cluster" → InfiniBand. "AI performance while standardizing on Ethernet" or "multi-tenant AI cloud" → Spectrum-X. "GPU-to-GPU within the server" → NVLink/NVSwitch. "RDMA on Ethernet" → RoCE. "Reduces CPU overhead by bypassing the kernel" → RDMA. Five phrasings, five instant answers.
Training is a synchronized, loss-intolerant, east-west workload. Everything NVIDIA builds in networking — RDMA, lossless fabrics, adaptive routing, SHARP, rail-optimized topologies — exists to keep thousands of GPUs from waiting on one congested link.
The DPU (data processing unit) — NVIDIA's BlueField line — is a NIC with an onboard Arm SoC that offloads, accelerates, and isolates infrastructure workloads: networking (overlays, RoCE acceleration), storage (NVMe-oF, elastic block storage), and security (firewalling, encryption, zero-trust microsegmentation). The pitch: infrastructure software that would burn 20–30% of host CPU cores runs on the DPU instead, and because it runs outside the host OS, the infrastructure layer stays secure even if a tenant workload is compromised — the basis of zero-trust, multi-tenant AI clouds. Developers program DPUs through the DOCA SDK (DPU analog of CUDA). In Spectrum-X, BlueField-3 based SuperNICs are the endpoint half of the platform, handling packet reordering and congestion control at 400 Gb/s per GPU. NVIDIA's framing: CPU for general compute, GPU for accelerated compute, DPU for infrastructure compute.
GPU accelerates the application's math; DPU accelerates and isolates the infrastructure (networking, storage, security). If the scenario says "offload networking/security from host CPUs" or "isolate tenants in a shared AI cloud," the answer is a DPU/BlueField — not a GPU, not a faster CPU.
NVIDIA publishes validated blueprints so organizations don't design clusters from scratch. DGX BasePOD is the entry reference architecture: roughly two to sixteen DGX systems plus specified networking (compute fabric, storage fabric, management), certified partner storage, and the software stack (Base Command Manager, AI Enterprise). It's the "department or single-project scale" design. DGX SuperPOD is the turnkey, data-center-scale AI supercomputer: built from scalable units (SUs) of 32 DGX systems, growing to hundreds or thousands of nodes, with a full InfiniBand (or, in newer variants, Spectrum-X) rail-optimized fabric, dedicated management plane, validated parallel storage, and delivery as an integrated, burn-in-tested product. SuperPOD designs based on GB200 NVL72 racks extend the NVLink domain from 8 GPUs to 72 GPUs per rack. The exam's angle: reference architectures de-risk deployment — validated performance, predictable scaling, faster time-to-value — versus a bespoke build.
Scale is the discriminator: BasePOD for smaller clusters built from the reference design; SuperPOD for extreme scale in 32-node scalable units, delivered as a full product. Both are DGX + networking + storage + software, validated together.
AI racks obliterate traditional data-center assumptions. A legacy enterprise rack draws 5–15 kW; a rack of four DGX H100s draws ~40 kW; a single GB200 NVL72 rack draws ~120 kW. Facilities questions on the exam revolve around the consequences:
Power. Individual GPUs draw 700 W (H100 SXM) to 1,000 W+ (Blackwell class); an 8-GPU server draws 10–14 kW. Facilities need high-capacity feeds, three-phase power distribution to racks, appropriately sized PDUs, redundant supplies (N+1 or 2N), and UPS/generator backup. Training clusters also create load dynamics — thousands of GPUs starting or finishing a synchronized job step swings megawatts — which power engineering must tolerate.
Cooling. Air cooling with hot-aisle/cold-aisle containment tops out around 30–40 kW per rack even when well engineered. Beyond that, liquid cooling becomes mandatory: direct-to-chip cold plates (the mainstream choice — coolant loops through plates on GPUs/CPUs, removing most of the heat, with rear-door heat exchangers or residual air handling the rest) or full immersion cooling. GB200 NVL72 is liquid-cooled by design. Liquid is also more efficient: it eliminates most fan energy and enables warm-water cooling with free-cooling chillers, and it enables heat reuse.
Facility planning covers floor loading (a loaded AI rack can exceed 1,500 kg — many raised floors can't take it), physical space and rack layout matched to fabric cable-length limits (optical vs. copper reach influences pod layout), structured cabling for thousands of fiber links, and site selection increasingly dominated by power availability.
Energy efficiency has its own blueprint bullet. Know PUE (Power Usage Effectiveness) = total facility power ÷ IT power; 1.0 is perfect, traditional DCs run ~1.5–1.8, modern liquid-cooled AI facilities target ~1.1. But NVIDIA's core efficiency argument is work per watt, not just facility overhead: accelerated computing does more computation per joule than CPUs, each GPU generation improves performance-per-watt (a major generational selling point), and software efficiency (mixed precision, TensorRT quantization, higher utilization via better scheduling) is an energy tool. DCGM exposes per-GPU power telemetry and enforcement of power caps.
Anchor three numbers: ~700 W per H100 GPU, ~10 kW per DGX H100 system, ~120 kW per GB200 NVL72 rack — and the rule of thumb that beyond roughly 30–40 kW per rack, you move from air to liquid (direct-to-chip) cooling. PUE closer to 1.0 is better.
The final Domain 2 decision area weighs deployment models. On-premises (own or colocate a cluster, e.g., BasePOD/SuperPOD): full control, data sovereignty and compliance, predictable performance from dedicated fabrics, and lower cost at sustained high utilization — but large capex, long lead times, facility demands, and the burden of operating it. Cloud (AWS/Azure/GCP/OCI GPU instances, or DGX Cloud, NVIDIA's managed AI platform hosted in partner clouds): instant elasticity, opex model, latest GPUs without procurement, managed operations — but at premium hourly rates that exceed on-prem TCO when utilization stays high, plus data-egress costs, data-residency questions, and periodic GPU scarcity. Specialized GPU clouds (CoreWeave, Lambda and peers) offer AI-tuned infrastructure between the two. Hybrid is the pragmatic norm the exam favors: steady-state training on owned infrastructure, burst and experimentation in the cloud, inference wherever latency and data gravity dictate — with containers, Kubernetes, and NGC-standardized software making workloads portable across all of it.
High sustained utilization and strict data control favor on-prem; bursty, elastic, or fast-start needs favor cloud; most enterprises land hybrid. TCO questions hinge on utilization — a GPU cluster busy 90% of the time is cheaper owned, one busy 20% of the time is cheaper rented.
The smallest domain — roughly 11 questions — covers running the cluster day to day: cluster management with Base Command Manager, orchestration and scheduling with Kubernetes and Slurm, GPU monitoring with DCGM, virtualization with vGPU and MIG, utilization analysis, and out-of-band management.
NVIDIA Base Command Manager (BCM) is the cluster-management layer for AI infrastructure (its lineage is Bright Cluster Manager, which NVIDIA acquired). It handles the full cluster lifecycle: bare-metal provisioning of nodes from head-node-managed software images (change an image once, redeploy consistently everywhere — no configuration drift), network and identity configuration, deployment of the AI software stack, installation and management of both Kubernetes and Slurm as workload managers, cluster-wide monitoring and health checking (nodes, GPUs, network, storage, with alerting), and ongoing operations — adding nodes, rolling updates, managing users. BCM is the standard management plane in BasePOD and SuperPOD deployments. Distinguish it from NVIDIA Mission Control, the newer full-stack operations suite for Blackwell-era DGX factories, and from the retired Base Command Platform SaaS — for this exam, BCM is the cluster manager to know.
BCM builds and manages the cluster (provisioning, images, monitoring, deploying schedulers). It is not itself the job scheduler — Slurm or Kubernetes schedules the work; BCM deploys and manages them.
Containers are the packaging foundation of AI operations. AI software stacks are notoriously version-sensitive (framework ↔ CUDA ↔ driver), and containers freeze a known-good combination into a portable image — which is exactly what NGC distributes. The NVIDIA Container Toolkit lets containers access GPUs; on Kubernetes, the GPU Operator automates everything a GPU node needs (driver, container runtime config, device plugin, DCGM exporter, MIG management) as cluster-managed software.
Two schedulers dominate, and comparing them is a guaranteed exam topic:
| Aspect | Slurm | Kubernetes |
|---|---|---|
| Origin | HPC batch scheduling | Cloud-native service orchestration |
| Unit of work | Batch job (script, queued) | Pod/deployment (often long-running) |
| Best for | Large multi-node training: gang-schedules all GPUs of a job to start together | Inference services and pipelines: autoscaling, rolling updates, self-healing |
| Scheduling model | Queues, priorities, fair-share, reservations, topology-aware placement | Declarative desired state; scheduler places pods; HPA autoscaling |
| GPU awareness | Native GRES (generic resource) GPU scheduling | Via device plugin / GPU Operator; fractional GPUs via MIG |
| Typical users | Researchers submitting sbatch jobs | Platform teams running production services |
The mapping to remember: Slurm for training clusters (its gang scheduling starts a 512-GPU job atomically and queues it fairly), Kubernetes for inference and production services (it scales replicas with traffic and restarts failures). Real estates run both — BCM can deploy the two side by side, and tools like Run:ai (acquired by NVIDIA) layer AI-aware scheduling — fair-share quotas, preemption, fractional GPUs — on Kubernetes.
"Gang scheduling," "batch queue," "multi-node training job" → Slurm. "Autoscaling," "microservices," "rolling update," "self-healing," "Triton deployment" → Kubernetes. "Why containers for AI?" → reproducible, portable environments that bundle the framework/CUDA dependency chain, distributed via NGC.
DCGM (Data Center GPU Manager) is NVIDIA's suite for monitoring and managing GPUs at fleet scale — the datacenter-grade successor to eyeballing nvidia-smi. Its capabilities: continuous low-overhead telemetry (utilization, memory use, temperature, power draw, clock speeds, NVLink/PCIe throughput, ECC memory errors), active health checks and diagnostics (dcgmi diag at increasing levels, used for burn-in and triage), policy-based automation (e.g., flag or drain a GPU on repeated ECC errors or thermal violations), and job-level statistics. The integration pattern to know: dcgm-exporter publishes GPU metrics to Prometheus, dashboards render in Grafana, and Kubernetes can consume the same data for scheduling and health decisions. This stack is how operators see the fleet.
Key metrics and what they diagnose: GPU utilization low while jobs run → input pipeline or CPU bottleneck (GPUs starved); memory utilization near capacity → OOM risk, consider a bigger GPU or smaller batches; temperature high → cooling problem, leads to thermal throttling (clocks drop, performance silently degrades); power draw far below TDP during "busy" periods → GPUs are actually waiting, not working; ECC errors climbing → failing memory, drain and replace the GPU; NVLink/PCIe throughput anomalies → communication bottlenecks.
Utilization analysis gets its own blueprint line because it's the economic heart of operations: GPUs are the most expensive asset in the building, and industry-average clusters often idle at 30–50%. The levers: better scheduling (fair-share, preemption, backfill), sharing idle capacity (MIG, time-slicing), fixing data-pipeline stalls, and right-sizing — moving small inference jobs off big training GPUs. Telemetry is what makes any of it visible.
DCGM = fleet-scale GPU health, diagnostics, policy, and telemetry; dcgm-exporter → Prometheus → Grafana is the canonical monitoring pipeline. And the most-tested inference: low GPU utilization during a running job means the GPUs are starved — look upstream at data loading, CPU, or I/O.
Two NVIDIA technologies let multiple users share GPUs, and telling them apart is among the most predictable questions on the exam.
MIG (Multi-Instance GPU) is hardware partitioning, available on A100/H100/H200/B200-class GPUs: one physical GPU is sliced into up to seven fully isolated instances, each with dedicated compute slices, memory, and cache. Isolation is at the silicon level — one instance's crash or memory pressure cannot touch another — giving predictable, guaranteed QoS per instance. Profiles are fixed fractions (e.g., on an 80 GB A100: 1g.10gb through 7g.80gb). MIG shines for serving many small inference workloads or giving multiple users guaranteed slices on bare metal or Kubernetes.
vGPU (virtual GPU) is software virtualization: the vGPU manager on a hypervisor (vSphere, KVM) presents virtual GPUs to VMs, time-slicing the physical GPU's compute among them (each vGPU gets a dedicated memory slice). It's licensed software, integrates with enterprise VM tooling (including live migration), and serves VDI/graphics and VM-based AI alike. The two compose: on supported GPUs, vGPU profiles can be backed by MIG instances for hardware-isolated VMs.
| Aspect | MIG | vGPU |
|---|---|---|
| Isolation | Hardware partitioning (dedicated compute, memory, cache) | Software time-slicing of compute; per-VM memory slice |
| Max sharing | 7 instances per GPU | Many VMs (profile-dependent) |
| QoS | Guaranteed, deterministic per instance | Scheduler-based; compute is shared in time |
| Requires | Ampere or newer data-center GPU | Hypervisor + NVIDIA vGPU software license |
| Environment | Bare metal or Kubernetes (and inside VMs) | Virtual machines |
| Sweet spot | Multi-tenant inference; guaranteed slices for small jobs | VDI, virtualized enterprises, VM-based AI with vMotion-style tooling |
MIG = hardware isolation, up to 7 instances, guaranteed QoS. vGPU = software time-slicing for VMs, needs a hypervisor and a license. "Strict isolation / predictable performance for containers" → MIG. "Share GPUs among virtual machines / VDI" → vGPU.
Out-of-band (OOB) management is the ability to manage servers independently of their operating system, through a dedicated management controller — the BMC (Baseboard Management Controller) — on its own network. Via Redfish (the modern REST API standard) or legacy IPMI, operators can power-cycle a hung node, mount virtual media and reinstall an OS, update firmware, watch hardware sensors (temperatures, fans, PSUs), and open a remote console — even when the host OS is dead. Every DGX has a BMC; BCM drives BMCs to do bare-metal provisioning; and best practice keeps the OOB network physically separate and locked down, since a BMC compromise means total hardware control. The contrast term is in-band management — SSH/agents through the host OS, which works only while the node is healthy. Modern platforms extend the story: BlueField DPUs provide an additional isolated management/control point independent of the host.
The exam phrase to match: "manage or recover a server when the OS is down or unreachable" → out-of-band management via the BMC (IPMI/Redfish) on a dedicated management network.
You don't need hands-on time with a DGX to pass — but you do need the mental model of one: eight GPUs lashed together with NVLink, fed by InfiniBand or Spectrum-X, provisioned by BCM, scheduled by Slurm or Kubernetes, watched by DCGM, sliced by MIG, and kept alive by serious power and liquid cooling. Master the comparison tables in this guide, and the exam becomes an exercise in recognition. Good luck.