← Course index Module 5 of 6
Module 5 · OS fundamentals for AI systems

Rack-scale: GB200 and GB300 NVL72

An NVL72 is 18 separate Linux hosts that share one NVLink memory fabric. Inside each host, the CPU and GPUs are cache-coherent, so the kernel can see GPU memory as NUMA nodes and the GPU walks the CPU's page tables. Across hosts, GPUs load and store into each other's memory with no shared kernel to arbitrate, so a userspace daemon (IMEX) does that job. Modules 1–4 give you every piece. This module puts them together at rack scale.

≈45 min reading3 interactive diagramsDrill ≈15 min (any box; full output needs a Grace node)Builds on M1 §3–5, M2 §5–9, M4 §4–7
This module is written from NVIDIA's public documentation and tuning guides, not from measurements on a GB200. Items that need a real tray to confirm are marked like this. Run drills/m5_rack.sh on one and replace them.

1One rack, 18 kernels, one NVLink domain

What it is. A GB200 NVL72 rack holds 18 compute trays. Each tray has 2 Grace CPUs and 4 Blackwell GPUs (two GB200 "superchips" of 1 Grace + 2 B200), so the rack has 36 Grace CPUs and 72 GPUs. It also holds 9 NVLink switch trays. Every GPU has 18 NVLink 5 links (1.8 TB/s total, 900 GB/s each way) running through a copper spine to the switch trays, so all 72 GPUs are one hop apart: 130 TB/s of NVLink in one rack. GB300 NVL72 keeps the same shape with Blackwell Ultra GPUs (288 GB HBM3e) and ConnectX-8 NICs.

Fig 1 · GB200 NVL72: what the hardware is vs. what the OS sees
the rack 18 Linux hosts · 1 NVLink domain one compute tray = one Linux host one kernel, 2 CPU sockets, 4 GPUs (2 superchips)
Power shelves and management switches are omitted. NVIDIA's layout puts the 9 switch trays in the middle of the rack, between banks of compute trays. Tray-internal CPU–CPU wiring isn't drawn.

How the OS sees it. Each tray boots its own kernel, with its own page tables, scheduler, cgroups and /dev/nvidia*. Nothing in Linux spans trays. But a GPU in tray 3 can execute st.global to an address that lands in a GPU's HBM in tray 11: the NVLink fabric routes it, and neither kernel is involved. So the "machine" is 72 GPUs for memory and collectives, and 18 hosts for everything the OS does. Several later sections come back to this mismatch: who authorizes cross-tray mappings (§6), what a tray failure takes down (§8), and how the scheduler treats a rack (§8).

Why it matters for AI workloads: the scale-up domain grew from 8 GPUs (HGX H100) to 72. Tensor and expert parallelism, which need all-to-all and all-reduce at every layer, can now span 72 GPUs at NVLink speed instead of 8. The price is that you're running a 72-GPU shared-memory machine on top of 18 independent operating systems.
Where this shows up in your stack torchrun --nnodes=18 --nproc-per-node=4: torch sees 18 nodes, just like an IB cluster. NCCL (2.25.2+) detects that all 72 ranks share a fabric (they report the same cluster UUID and clique ID), so it uses NVLink between trays instead of the network. That's MNNVL, multi-node NVLink. LOCAL_WORLD_SIZE is 4, but the "NVLink-local" world is 72: code that assumes "same node ⇔ NVLink" (e.g. picking intra-node process groups for TP) has to use the NVLink domain instead.

2The tray as the OS sees it

What it is. One host has two Grace CPUs (72 Arm Neoverse V2 cores each, up to 480 GB LPDDR5X at up to 512 GB/s each) and four B200 GPUs (186 GB HBM3e at ~8 TB/s each). Each Grace connects to its two GPUs with NVLink-C2C, 900 GB/s bidirectional, and the link is cache-coherent: a CPU load can hit a line cached by the GPU and vice versa. That's the difference from the PCIe in Module 1, where the GPU is a device behind a non-coherent bus with its own, separate memory.

One GB200 trayper unitper trayOS object
Grace CPU (Neoverse V2)72 cores144 CPUs2 NUMA nodes with CPUs
LPDDR5X≤480 GB · ≤512 GB/s≤960 GBmemory of those 2 nodes
B200 HBM3e186 GB · ~8 TB/s744 GBNUMA mode: 4 memory-only nodes · CDMM: driver-managed, invisible
NVLink-C2C (Grace ↔ its 2 GPUs)900 GB/s bidir×2coherent fabric; GPU uses CPU page tables (ATS)
NVLink 5 to the switch trays1.8 TB/s per GPU7.2 TB/snot visible to Linux at all
ConnectX-7 (scale-out, east-west)400 Gb/s4 (1 per GPU)RDMA devices, netdevs (M4 §4)
BlueField-3 (front-end, storage)DPU2netdevs; runs its own Arm OS

The 13.4 TB HBM and 17 TB LPDDR5X rack totals on NVIDIA's spec page are these numbers ×18. The 186 GB per GPU is what software sees; the product is often quoted as 192 GB.

Why it matters for AI workloads: every tray has more CPU memory (960 GB) than GPU memory (744 GB), and the GPU can read that CPU memory at C2C speed, which is ~20× the 4090's PCIe Gen4 link. So optimizer states, KV cache and activations offloaded to CPU memory are cheap enough to matter, in a way they weren't on PCIe boxes. See §4.
Where this shows up in your stack nvidia-smi topo -m on a tray shows NV18 between all four GPUs. Even the two GPUs on the same superchip talk through the switch trays, not directly. It also shows each GPU's CPU affinity, NUMA affinity and, new on coherent systems, a GPU NUMA ID column. That column read N/A on your 4090. Rank-to-CPU binding still matters (M1 §3): pin rank i's threads to the Grace attached to GPU i, or every host-side copy crosses sockets.
How the two Grace CPUs in one tray are linked, and the exact NUMA IDs the GPUs get (GH200 used node 0 for the CPU, node 1 for the GPU and 2–9 as empty MIG placeholders), aren't in NVIDIA's public docs that I could find. m5_rack.sh numa prints both.

3Who manages GPU memory: NUMA mode vs CDMM

What it is. Because HBM is coherent and the CPU can address it, the driver can hand it to the Linux kernel as ordinary memory: each GPU becomes a memory-only NUMA node (no CPUs). That's NUMA mode, the original behaviour on Grace Hopper and Grace Blackwell. Then malloc, numactl, mbind, the page allocator, reclaim, compaction and automatic NUMA balancing all apply to GPU memory too. Driver R580 (580.65.06) added CDMM (coherent driver-based memory management): the NVIDIA driver keeps HBM to itself, and the kernel never sees it as memory. The GPU NUMA nodes still exist, but they're empty. Check which mode you're in with grep Coherent /proc/driver/nvidia/params → "numa" or "driver".

Fig 2 · Where does this allocation land? One tray, two modes interactive
Bars are to scale: Grace LPDDR5X 480 GB per node, B200 HBM 186 GB per GPU. Grey = the OS and other processes, teal = your allocations, orange = the buffer x this step is about, amber = pages the kernel spilled where you didn't want them. Dashed outline = memory the kernel can't see.
Why it matters for AI workloads: NUMA mode turns HBM into a place where the kernel can put CPU data when LPDDR runs out. A data-loading or preprocessing job that over-allocates spills onto a GPU's node. Your next cudaMalloc then fails with plenty of "free" memory in nvidia-smi, and the CPU code runs slower because its pages now sit across C2C. Kubernetes in NUMA mode counts HBM as node memory and applies pod memory limits to it. NVIDIA recommends CDMM for Kubernetes, made it the default for GPU Operator deployments from 580.65.06, and says it becomes the default on coherent platforms with R615 / CUDA 13.4.
Where this shows up in your stack In CDMM, explicit cudaMalloc + cudaMemcpy code, which is all of PyTorch's caching allocator, is functionally unaffected. What changes: cudaMemPrefetchAsync and access-counter migration no longer move system-allocated (malloc/mmap) pages to the GPU. The GPU still reads them over C2C, so code that relies on "touch it from the GPU and it migrates" gets C2C bandwidth instead of HBM bandwidth. Managed memory (cudaMallocManaged) keeps its own migration. In NUMA mode, watch numastat -m for CPU pages on GPU nodes.
Exact behaviour of access-counter migration (thresholds, which hints enable it for system memory) is a driver implementation detail. NVIDIA's programming guide calls it that explicitly. Treat the "hot pages migrate" step as possible, not guaranteed.

4ATS: the GPU walks the CPU's page tables

What it is. On Grace systems, a GPU kernel can dereference any CPU pointer: malloc, mmap, a stack variable, a file mapping. The GPU's MMU sends an address translation service (ATS) request across C2C to the CPU's SMMU, which walks the process's own page table (the one from M2 §2). The CUDA attribute PAGEABLE_MEMORY_ACCESS_USES_HOST_PAGE_TABLES = 1 says this is happening. NVIDIA calls systems with a combined CPU–GPU page table hardware coherent. Compare the alternatives:

How the GPU reaches malloc'd memoryyour 4090 (PCIe)x86 + HMMDGX Spark (GB10)GB200 / GB300
PAGEABLE_MEMORY_ACCESS0 (measured)111
…USES_HOST_PAGE_TABLES (ATS)00: separate GPU page table, kept in sync by faults1 (expected)1
HOST_NATIVE_ATOMIC_SUPPORTED00expected 11
GPU access to a CPU pagefault → errorfault → migrate page to VRAMdirect (same DRAM)direct over C2C, or migrate (NUMA mode)
CPU↔GPU bandwidth20 GB/s pinned copy (measured)PCIeLPDDR5x 273 GB/s sharedC2C 900 GB/s bidir per superchip

Consequence 1: CPU page size is now GPU page size. For system memory the GPU uses the CPU's PTEs, and the GPU's TLB caches their translations. A 4 KiB kernel means GPU TLB entries cover 4 KiB, and NVIDIA's guide warns that GPU TLB misses are "significantly more expensive" than CPU ones. That's why Grace's tuning guide recommends a 64 KiB-page kernel (M2 §2, ARM64 64K granule). It also brings a THP trap: on a 64K kernel the PMD huge page is 512 MiB. The guide recommends disabling 512 MiB THP and using 2 MiB mTHP (kernel 6.9+) instead, because 512 MiB pages fragment memory and make first-touch placement coarse.

Consequence 2: the copy engine isn't the only path. SMs can stream CPU memory directly (zero-copy), and with C2C that's fast. The drill measures both paths: on your 4090 the SMs read pinned host memory at 12.5 GB/s, versus 955 GB/s from VRAM. On GB200, the expectation is hundreds of GB/s over C2C.

Why it matters for AI workloads: offload stops being a last resort. A 70B model's fp32 Adam states (≈840 GB) fit in one tray's LPDDR5X, and streaming them over C2C costs seconds per step where PCIe Gen4 would take tens of seconds. KV-cache offload for long-context inference gets the same benefit. Module 6 has a calculator for this.
Where this shows up in your stack Pinned memory still matters, for a different reason. On Grace, cudaHostAlloc memory sits in LPDDR5X, and pinning keeps the kernel from moving those pages during DMA, as on x86 (M2 §8). Whether pageable copies on Grace still go through a staging buffer the way they do on x86 (M1 §6) is what m5_coherent.cu section 2 measures. Extended GPU Memory (EGM) goes one step further: cuMemCreate with CU_MEM_LOCATION_TYPE_HOST_NUMA allocates "GPU memory" that physically lives in a Grace node's LPDDR and can be mapped by other GPUs. NVIDIA's guide describes it for single-node systems.
I haven't measured GB200 C2C read bandwidth, and NVIDIA's spec page doesn't say whether the 900 GB/s is shared by the superchip's two GPUs or per GPU. Assume shared until the drill says otherwise.

5The Grace OS tuning checklist, explained

NVIDIA's Grace Performance Tuning Guide has a short list of OS settings for Grace Hopper and Grace Blackwell. Each one follows from something in Modules 2–4. m5_rack.sh tune checks them all (read-only).

SettingWhy (the OS mechanism)Check
64 KiB kernel pagesGPU uses CPU PTEs via ATS. Larger pages mean 16× fewer TLB entries for the same memory and fewer faults. M2 §2, §5.getconf PAGESIZE → 65536
THP: 512 MiB never, 2 MiB madviseOn a 64K kernel the PMD THP is 512 MiB: too coarse for first-touch NUMA placement and prone to fragmentation. mTHP gives 2 MiB (6.9+). M2 §5.…/hugepages-524288kB/enabled
init_on_alloc=0Ubuntu's generic kernel zeroes every page as it's allocated (a hardening option). In NUMA mode HBM pages come from the kernel's allocator, so cudaMalloc pays for zeroing gigabytes through the CPU path. The guide says this "severely" slows it.grep init_on_alloc /proc/cmdline
kernel.numa_balancing=0AutoNUMA periodically unmaps pages to catch accesses with hinting faults (M2 §4), and GPU-heavy jobs suffer from the extra faults.cat /proc/sys/kernel/numa_balancing
CDMM (especially under Kubernetes)Keeps HBM out of the kernel's page allocator (§3), so there's no spill, no over-reported node memory, and no cgroup accounting of HBM.grep Coherent /proc/driver/nvidia/params
Swap ≥ ¼–½ of GPU memory (NUMA mode)Pages the kernel migrated into GPU nodes can only be reclaimed by swapping them out.swapon --show
vm.min_free_kbytes, compactionTHP makes khugepaged raise the free-memory reserve (17 GB observed in the guide), and that reserve applies to GPU nodes too. Proactive compaction migrates pages in the background, which "can heavily impact" cudaMalloc on coherent systems.sysctl vm.min_free_kbytes vm.compaction_proactiveness
iommu.passthrough=1DMA skips IOMMU translation (M2 §8, M4 §5): the same idea as iommu=pt on x86. Optional: measure first.cat /proc/cmdline
cpufreq performanceLaunch threads and NCCL proxies are latency-bound (M3 §6). Ubuntu's cpufrequtils service can reset this at boot.cpupower frequency-info
Why it matters for AI workloads: most of these were invisible on x86 + PCIe, because the kernel never touched GPU memory. On a coherent platform the kernel's memory-management policies (zeroing, balancing, compaction, THP sizing) act on GPU memory and on memory the GPU translates. Defaults written for CPU servers are wrong here.
Where this shows up in your stack If torch.empty(…, device="cuda") of a few hundred GB takes seconds on a fresh process but is instant on the next call, suspect init_on_alloc or compaction: the first call goes to the driver (which in NUMA mode goes to the kernel), and later calls hit PyTorch's caching allocator. The same check applies to the DGX Spark, which runs a Grace-family Arm CPU: m5_rack.sh tune there tells you whether DGX OS already sets these.

6Multi-node NVLink: who authorizes a cross-tray mapping?

What it is. Within one OS, sharing GPU memory between processes is a kernel-mediated capability: process A exports an allocation as a file descriptor (CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR) and passes it over a Unix socket with SCM_RIGHTS. The kernel duplicates the fd into process B, and the fd itself is the proof of permission. That's how torch.multiprocessing CUDA IPC and SymmetricMemory work on one node (M2 §9). Across trays there's no shared kernel, so a 64-byte blob has to stand in for the fd. CUDA 12.4 added CU_MEM_HANDLE_TYPE_FABRIC for this, and IMEX (the internode memory exchange service, a daemon on every tray) checks and brokers the imports.

The pieces. The switch trays run NVOS, which includes Fabric Manager and an NVLink subnet manager. They discover the topology, assign addresses and program routes, much like an InfiniBand SM. Each GPU registers with the fabric at boot; nvidia-smi -q shows its Fabric State/Status, ClusterUUID and CliqueId. GPUs with the same cluster UUID and clique can map each other. IMEX reads /etc/nvidia-imex/nodes_config.cfg (one management IP per tray), forms a quorum over TCP/gRPC (port 50000 by default), and serves import requests. IMEX channels are character devices, /dev/nvidia-caps-imex-channels/channelN: a process must be able to open the same channel as the exporter. That's the isolation boundary between users and jobs, and it's why containers need the channel device passed in.

Fig 3 · Sharing one GPU buffer between two processes interactive
Control path (export, send, import, map) runs once at setup. After that, the data path is GPU loads and stores over NVLink, with no CPU, kernel or daemon involved, exactly like RDMA after registration (M4 §5).
Why it matters for AI workloads: every NVL72 feature you care about rides on this path: NCCL's MNNVL transport, NVLS multicast across trays, NVSHMEM, and symmetric-memory kernels for all-to-all and all-reduce. If IMEX or the channel is wrong, those don't degrade gracefully. Since NCCL 2.27, a failed MNNVL fabric init is a hard error by default, rather than a silent fall-back to InfiniBand at a fraction of the bandwidth.
Where this shows up in your stack Check, in order: nvidia-smi -q | grep -A6 Fabric (State Completed, Status Success), systemctl status nvidia-imex, nvidia-imex-ctl -N (all peers connected), and ls -l /dev/nvidia-caps-imex-channels/ inside the container. The drill's fabric probe does cuMemCreate(…FABRIC…). On your 4090 the driver reports HANDLE_TYPE_FABRIC_SUPPORTED = 1, yet the create fails with operation not permitted: the attribute only means the driver knows the handle type. NCCL_MNNVL_ENABLE=0 forces NCCL back to IB/RoCE as a diagnostic, and NCCL_P2P_DISABLE=1 now implies it.
Whether PyTorch's SymmetricMemory uses fabric handles across trays depends on the PyTorch version and backend (CUDA vs NVSHMEM). Check your version's torch.distributed._symmetric_memory source before assuming it spans the rack.

7Collectives on the rack: two very different networks

What it is. An NVL72 GPU has two ways out. NVLink 5 gives 900 GB/s each way to any of the 71 others. Its ConnectX-7 NIC gives 400 Gb/s ≈ 50 GB/s to anything outside the rack (GB300's ConnectX-8: 800 Gb/s ≈ 100 GB/s). That's an 18× gap on GB200 and 9× on GB300. NCCL runs hierarchical collectives: reduce-scatter inside the NVLink domain, all-reduce of each GPU's 1/72 slice across racks over IB, then all-gather inside the domain. The bigger the domain, the smaller the slice that has to cross the slow network.

NVLS (NVLink SHARP) moves reduction into the switch. Using a multicast object (cuMulticastCreate, M2 Fig 5), each GPU sends its data to the switch once, the switch adds the contributions, and it multicasts the result back. A ring all-reduce moves 2(n−1)/n × S per GPU, almost twice the data. NVLS moves about S each way, in one switch round trip instead of 2(n−1) ring steps. NCCL enables NVLS by default where the switches support it (NCCL_NVLS_ENABLE), and it costs extra GPU memory for the multicast buffers.

Fig 4 · All-reduce time on different machines interactive
Bandwidth-only model with a fixed per-phase latency: in-domain ring 2(d−1)/d·S/BNVL (NVLS ≈ S/BNVL), cross-domain ring on S/d per GPU over that GPU's NIC, then in-domain all-gather (folded into the ring and NVLS terms). Link speeds are vendor peaks per direction. Real NCCL reaches roughly 70–90% of them, so use this for ratios, not predictions.
Why it matters for AI workloads: layout follows the gap. Keep tensor and expert parallelism (per-layer all-reduce and all-to-all, latency-critical) inside one NVLink domain, and put data, pipeline and FSDP-across-racks traffic on the scale-out network. On HGX H100, "inside the domain" meant TP ≤ 8. On NVL72 it means TP×EP up to 72, which is a large part of why big MoE models moved to rack-scale systems.
Where this shows up in your stack In DeviceMesh, the innermost (fastest-varying) dimensions should map to ranks in the same NVLink domain. With torchrun that means ranks 0–71 in rack 0, which depends on how your launcher orders hosts. NCCL_DEBUG=INFO logs MNNVL and NVLS when they're in use. An NVL72 can also be partitioned into smaller NVLink domains (clique IDs differ), and then "same rack" no longer means "same domain".

8Schedulers, failure domains, observability

Containers and Kubernetes. Everything in M4 §6–7 still applies (shm, memlock, /dev/infiniband, host networking), plus two things. (1) A multi-tray job needs an IMEX domain covering exactly its trays, and an IMEX channel in every pod. NVIDIA's DRA driver models this as a ComputeDomain: it starts per-domain nvidia-imex daemons and injects the channel device into the pods that claim it. Slurm sites typically write nodes_config.cfg and start IMEX in the job prolog. (2) Run in CDMM mode, or pod memory limits and the node's memory capacity will include HBM (§3).

Failure domains. A tray is a host: if it reboots, its 4 GPUs leave the NVLink domain, and every job with a rank there fails. Its peers' kernels and GPUs are fine, but a 72-way NCCL communicator can't lose members. A switch tray or NVLink fault affects bandwidth or connectivity for everyone in the domain. This is why NVL72 schedulers allocate in whole-rack or partition units, and why spare trays and fast checkpoint restart matter more than on 8-GPU boxes.

QuestionCommand on a tray
Coherent memory mode?grep Coherent /proc/driver/nvidia/params
Is HBM onlined, and is anything spilling onto it?numactl -H, numastat -m, grep -H . /sys/devices/system/node/node*/meminfo | grep MemUsed
GPU joined the fabric? Same clique?nvidia-smi -q | grep -A6 Fabric
IMEX healthy?systemctl status nvidia-imex, nvidia-imex-ctl -N
NVLink links up and at speed?nvidia-smi nvlink -s, nvidia-smi nvlink -e (error counters)
GPU or NVLink errors?dmesg | grep -i xid, dcgmi diag -r 2
Which transport did NCCL pick?NCCL_DEBUG=INFO: look for MNNVL / NVLS vs NET/IB
Tuning baselinebash drills/m5_rack.sh tune
Where this shows up in your stack A job that's fast on 4 GPUs (one tray, no IMEX needed) but fails or crawls at 8+ is the NVL72 equivalent of "works on one node, breaks on two". Check fabric state, IMEX and channels before suspecting NCCL. Single-tray jobs don't use IMEX at all.

DDrill: what kind of coherent machine is this?

# everything (read-only; builds /tmp/m5_coherent with nvcc)
bash drills/m5_rack.sh

# or parts: platform numa tune fabric bw place
bash drills/m5_rack.sh tune fabric
M5_MIB=4096 bash drills/m5_rack.sh bw        # bigger buffer for the bandwidth probe

Run it on every machine you have and compare. Section 1 of m5_coherent.cu is the most informative: the coherence attributes classify the machine. Section 3 compares copy-engine bandwidth with SMs reading CPU memory directly. Section 4 tells you whether multi-node NVLink sharing would work from this process.

Expected output

x86 + RTX 4090 measured on your box

PAGEABLE_MEMORY_ACCESS             0
..._USES_HOST_PAGE_TABLES          0
NUMA_CONFIG 0   NUMA_ID -1
HANDLE_TYPE_FABRIC_SUPPORTED       1
=> no pageable access
H2D pageable 16.1  pinned 20.3 GB/s
D2H pageable 15.2  pinned 19.3 GB/s
GPU memory (reference)  955.4 GB/s
cudaHostAlloc zero-copy  12.5 GB/s
plain malloc   skipped
cuMemCreate(FABRIC) failed:
  operation not permitted
coherent memory mode: n/a

GB200 tray expected, not yet measured

PAGEABLE_MEMORY_ACCESS             1
..._USES_HOST_PAGE_TABLES          1
HOST_NATIVE_ATOMIC_SUPPORTED       1
NUMA_CONFIG 1 (NUMA mode) / 0 (CDMM)
=> hardware-coherent (ATS)
H2D/D2H: pinned ~C2C speed (100s GB/s)
plain malloc: SMs read LPDDR over C2C
cuMemCreate(FABRIC): OK only with an
  IMEX channel; 64-byte handle
fabric: State Completed, same
  ClusterUUID/CliqueId on all 72 GPUs
tune: WARN on any 4K kernel,
  init_on_alloc=1, numa_balancing=1
GB200 values are predictions from NVIDIA's docs. The DGX Spark should show the same attributes as GB200 (ATS = 1) but INTEGRATED = 1, no fabric, and copy speeds bounded by its 273 GB/s LPDDR5x.

?Self-check

1. On a GB200 tray in NUMA mode, a CPU-only preprocessing job grows to 1.1 TB. What happens to it, and to the training job on the same tray?
The kernel fills both Grace LPDDR nodes (≈960 GB). Then, following its NUMA fallback order, it allocates from the memory-only GPU nodes: HBM. The preprocessing job keeps running, slower, since those pages are across C2C with higher CPU latency. The training job's next cudaMalloc on those GPUs fails even though nvidia-smi shows little memory used by its processes, and only swap can get those pages back. Under Kubernetes, the node's reported memory and the pod limits included HBM all along. CDMM avoids all of it: the kernel can't see HBM, so the CPU job hits reclaim, swap or the OOM killer within LPDDR, and the GPUs are untouched.
2. Why does the CPU's base page size affect GPU performance on Grace but not on your 4090?
On Grace the GPU accesses system memory through ATS, translating with the CPU's page tables, so its TLB entries cover CPU-sized pages: 4 KiB on a 4K kernel, 64 KiB on a 64K kernel. GPU TLB misses are expensive, and kernels touch huge ranges. On the 4090, the GPU uses its own page tables (2 MiB pages for cudaMalloc) and reaches host memory only through pinned buffers the driver maps itself. The CPU's page size never enters the GPU's translation path.
3. Why does NVIDIA recommend init_on_alloc=0 on coherent systems?
init_on_alloc=1 makes the kernel zero every page it hands out. In NUMA mode, GPU memory comes from the kernel's page allocator, so a large cudaMalloc pays for the kernel zeroing gigabytes of HBM, page by page, from the CPU side. It's a hardening default meant for CPU servers. Ubuntu's generic kernel enables it, and RHEL/SUSE don't.
4. Two processes on different trays want to share a cudaMalloc'd buffer. Why can't they use the fd-passing that works on one node, and what replaces each piece?
An fd only means something to the kernel that issued it, and there's no shared kernel across trays. What replaces each piece: the fd becomes a CU_MEM_HANDLE_TYPE_FABRIC handle (64 bytes of plain data, sent over any TCP channel). The kernel's permission check becomes IMEX validating the import against the exporter, with the IMEX channel device as the authorization boundary. The kernel's mapping becomes the importing driver mapping fabric addresses into the GPU's page table (cuMemMap + cuMemSetAccess). After setup, loads and stores go over NVLink with no software in the path.
5. A job that ran on an NVL72 under one container image now fails at NCCL init with an MNNVL error. Single-tray runs still work. What do you check?
Inside the new container: whether /dev/nvidia-caps-imex-channels/channel0 (or the job's channel) exists and is openable. It's the most likely thing a new image or runtime config drops. Then whether the host's IMEX service is up with all peers connected (nvidia-imex-ctl -N), and fabric state on each GPU. Single-tray runs work because they don't need IMEX. Since NCCL 2.27 a failed MNNVL init is fatal by default. NCCL_MNNVL_ENABLE=0 confirms the diagnosis (the job runs, but over IB at a fraction of the bandwidth), and isn't a fix.
6. You have 4 GB200 NVL72 racks and want TP × EP = 64 with DP = 4. Which dimension crosses racks, and why?
Put TP×EP (64 GPUs) inside one rack's NVLink domain and DP across the 4 racks. TP all-reduces and EP all-to-alls happen every layer, are latency-sensitive, and move large volumes, so they need the 900 GB/s-per-direction NVLink. DP gradient reduction happens once per step, overlaps with backward, and after the in-rack reduce-scatter only each GPU's slice crosses the 50 GB/s NIC. In DeviceMesh, order the dims as (dp, ep, tp), with TP and EP innermost and rank order following racks.
Saved in this browser and shown on the index page.

§Sources