Six modules on the host side of distributed training and inference. Part I covers one machine: how the box is wired, what virtual memory does to your bytes, how the kernel schedules the threads that feed the GPU, and how I/O, networks and containers fail. Part II scales that up to coherent CPU–GPU superchips and rack-sized NVLink domains: GB200/GB300 NVL72 and Vera Rubin. Each module takes ≈35–45 minutes to read, has interactive diagrams you can step through, and ends with a drill you can run on any NVIDIA machine, from an RTX 4090 workstation or a DGX Spark to an 8×H100 node or a GB200 tray.
Cores and cache lines, latency orders of magnitude, NUMA, PCIe topology, NVLink and NVLink-C2C, DMA and pinned memory, interrupts vs polling.
todoPage tables and TLBs, minor/major faults, THP, mmap and the page cache, COW, pinning and the IOMMU, CUDA VMM, expandable segments, SymmetricMemory, UVM/HMM/ATS.
todoSyscalls and context switches, futexes, fork vs spawn (and CUDA), EEVDF, affinity, isolcpus, IRQs, launch jitter, CUDA graphs.
todoPage cache vs O_DIRECT, io_uring, GPUDirect Storage, TCP vs RDMA, memory registration, cgroups and namespaces, container outages, perf/eBPF/nsys.
18 kernels and one NVLink domain, coherent NVLink-C2C, GPU memory as NUMA nodes vs CDMM, ATS and why page size matters to the GPU, the Grace tuning checklist, IMEX and fabric handles, NVLS, failure domains.
todoSix machines on the same four axes. Vera's 176-thread CPU, C2C at 1.8 TB/s and offload math, NVLink 6 and MoE all-to-all, ConnectX-9/BlueField-4 KV-cache tiers, Rubin CPX, and a day-one checklist.
The final self-check at the end of Module 4 asks all seven.
All drills are in drills/. They need Python 3 with PyTorch (CUDA), nvidia-smi, and optionally sudo apt install hwloc numactl sysstat. Run each one on both machines and keep the output: the pages compare the two.
| Module | Command | What it shows |
|---|---|---|
| 1 | bash drills/m1_machine.sh | topology, NUMA, PCIe links, GPU/NIC affinity; pinned vs pageable bandwidth; big vs small core (or cross-NUMA) |
| 2 | bash drills/m2_vm.sh | page size, THP, CUDA unified-memory attributes; read vs mmap faults, cold vs warm; THP fault counts |
| 3 | bash drills/m3_cpu.sh | strace of a training step; DataLoader COW growth (list vs numpy); launch jitter quiet / noisy / pinned / CUDA graph |
| 4 | bash drills/m4_io.shdocker run … sh /d/m4_container_check.shbash drills/m4_60s.sh PID | page cache and O_DIRECT; container limits that break training; the 60-second checklist |
| 5 | bash drills/m5_rack.sh | coherence class (ATS, NUMA config, fabric handles), copy engine vs SMs reading CPU memory, Grace tuning checklist, fabric/IMEX state, NUMA placement |
| 6 | bash drills/report.sh | one-page profile of any machine, written to results/<host>.md: the four questions plus headline numbers |
Get the drills with git clone https://github.com/kapilsh/ai-systems-notes, or download single files from the links in each module. m5_coherent.cu needs a CUDA toolkit (nvcc).
Module 1 + drill on both boxes (≈1 h). Module 2, the longest and most important, + drill (≈1.25 h). Re-do the page-walk and COW steppers until you can narrate them without looking.
Module 3 + drill (≈1 h). Module 4 + drill (≈1.25 h). Finish with the final self-check, out loud, without notes.
Module 5 (≈1 h), then Module 6 (≈45 min). Run drills/report.sh on every machine you can reach and compare the reports against Module 6's table.
Part of kapilsharma.dev · source and drills on GitHub.