← Course index Module 4 of 6
Module 4 · OS fundamentals for AI systems

I/O, networking, containers, observability

Bytes come in from disks and NICs and leave through them. Each path is either kernel-managed (convenient, copies, syscalls) or kernel-bypass (fast, but you must pin and register memory). Your jobs run inside containers that quietly limit memory, CPU, /dev/shm and locked pages. This module covers those paths and how to see what's happening on a live node.

≈40 min reading2 interactive diagramsDrill ≈25 minTeaches criteria #6 (RDMA) and #7 (containers); the final self-check covers all seven

1The page cache vs O_DIRECT

What it is. By default, file I/O is buffered. read() asks the page cache. On a miss the kernel reads from the device (with readahead), keeps the page, and copies it to your buffer. write() copies into the page cache, marks the page dirty, and returns. Background writeback flushes it later, or fsync forces it. The page cache uses all otherwise-free RAM and is reclaimed under memory pressure.

O_DIRECT bypasses it. The device DMAs straight to and from your (temporarily pinned) buffer. Buffer address, file offset and length must be aligned to the device's logical block size, typically 512 B or 4 KiB. There's no caching and no readahead: every read hits the device, warm or not.

Fig 1 · Three read paths for one checkpoint shard
Buffered read NVMe page cache user buffer pinned buffer GPU memory 2 CPU copies, 3 writes into DRAM; cached and shared for next time O_DIRECT NVMe page cache user buffer pinned buffer GPU memory device DMAs into an aligned (pinned) buffer: no CPU copy, no cache GPUDirect Storage NVMe page cache user buffer pinned buffer GPU memory NVMe DMAs straight into GPU memory over PCIe P2P (cuFile) teal = DMA by a device (no CPU), orange = CPU memcpy, faded = skipped
2 GiB checkpoint, your box (NVMe, ext4)timeGiB/s
buffered dd, cold (file evicted)0.81 s2.5
buffered dd, warm (from page cache)0.22 s8.9
O_DIRECT dd (always from device)0.34 s5.9
safetensors.load_file(device="cuda"), cold → warm0.99 → 0.37 s2.0 → 5.4
Why it matters for AI workloads: a single buffered reader is often limited by the CPU copy and by readahead depth, not by the drive. Here O_DIRECT with big requests read the cold file 2.4× faster than buffered. But warm loads from the page cache beat both. Rule of thumb: buffered (plus mmap) for checkpoints you'll reload or share across ranks, O_DIRECT (or GDS) for huge one-shot streams that would otherwise evict useful cache, such as dataset shards and checkpoint writes.
Where this shows up in your stack Checkpoint writes from all ranks at once fill the page cache with dirty pages. If dirty memory exceeds vm.dirty_ratio, writers get throttled in write(), which looks like a hung training loop. Async checkpointing (torch.distributed.checkpoint.async_save) moves the copy and write off the training thread. In containers, page cache counts against the cgroup's memory.max (it's reclaimable, but it's still charged).

2io_uring

What it is. An asynchronous I/O interface (Linux 5.1+) built from two ring buffers shared between your process and the kernel: a submission queue (SQ) where you write requests, and a completion queue (CQ) where the kernel writes results. One io_uring_enter syscall can submit hundreds of reads and wait for completions. In SQPOLL mode a kernel thread polls the SQ, so steady-state I/O needs no syscalls. "Fixed" (pre-registered) buffers are pinned once instead of on every I/O.

How it works. It's the same pattern as the GPU's command queues and RDMA's queue pairs: shared-memory rings plus a doorbell, with batching to amortize the kernel crossing. Compare read(): one syscall, one request, and a blocked thread per outstanding I/O.

Why it matters for AI workloads: NVMe drives need deep queues (dozens to hundreds of outstanding requests) to reach their rated bandwidth, and a thread pool doing blocking preads is a clumsy way to get there. io_uring lets one thread keep a whole RAID-0 of NVMe busy, which matters for streaming datasets and loading sharded checkpoints at full disk speed.
Where this shows up in your stack Some storage libraries for training data and checkpoints (and GDS's own I/O path) use io_uring or similar async I/O internally. Check what yours uses before hand-rolling it. Some container runtimes' default seccomp profiles block io_uring syscalls, so a library may silently fall back to a thread pool inside containers.
Which runtimes block io_uring by default, and since which versions, varies. Check your runtime's seccomp profile (docker info, or grep io_uring in the profile JSON) rather than relying on this note.

3GPUDirect Storage (GDS)

What it is. GDS lets the NVMe controller (or a NIC, for remote storage) DMA directly into GPU memory, skipping the host-memory bounce buffer. You use it through the cuFile API (cuFileRead(handle, devPtr, size, offset)), or Python wrappers such as KvikIO. It needs O_DIRECT-capable files, a supported filesystem, and kernel support for peer-to-peer DMA (NVIDIA's nvidia-fs module). When the direct path isn't available, it runs in compatibility mode: the same API, internally bouncing through pinned host memory.

Why it matters for AI workloads: host DRAM bandwidth and CPU memcpy are shared resources. A node loading 8 checkpoint shards at once pushes every byte through DRAM two or three times (page cache → user buffer → pinned → GPU, Module 1's data path). GDS cuts that to one DMA per byte, with no CPU involvement, so bandwidth scales with drives and PCIe switches instead of with host memory.
Where this shows up in your stack Check with gdscheck -p (from the GDS tools) and look for "compat mode". On the Spark, a direct NVMe→GPU path probably doesn't apply: NVIDIA says cudaMalloc memory isn't accessible to PCIe devices there (the same reason GPUDirect RDMA is unsupported), and "GPU memory" is the same LPDDR5x as host memory anyway. Reading into pinned host memory, or mmap plus ATS, is the natural path on the Spark.
I couldn't find NVIDIA documentation stating GDS support status on DGX Spark. The note above is an inference from the GPUDirect RDMA statement in the Spark porting guide.

4The kernel TCP stack vs RDMA kernel bypass

Kernel TCP. send() copies your bytes into kernel socket buffers. TCP/IP builds segments (the NIC usually offloads segmentation and checksums), the driver posts them to the NIC's ring, and the NIC DMAs them out. On the receiver, the NIC DMAs packets into kernel ring buffers and raises an interrupt (or the kernel polls under load, NAPI). TCP/IP processing runs in softirq context, data is queued on the socket, the receiving thread is woken, and recv() copies it into user memory. That's two CPU copies, syscalls on both sides, interrupts, wake-ups, and the CPU in the loop for every byte.

RDMA (InfiniBand, or RoCE v2 over Ethernet) moves the data path to the NIC. The application, through verbs, writes a work request into a queue pair that lives in its own memory and rings a doorbell on the NIC (an MMIO write, no syscall). The NIC DMAs the payload straight from the application's registered buffer, the remote NIC DMAs it straight into the remote application's registered buffer, and completions appear in a completion queue that the application polls. For an RDMA WRITE the remote CPU doesn't participate at all.

Fig 2 · Sending 1 MB to another node interactive
Simplified: TCP segmentation offload, GRO, NAPI polling and zero-copy socket options (MSG_ZEROCOPY, TCP devmem) reduce the TCP costs but don't change the overall shape.
Why it matters for AI workloads: at 400 Gb/s per NIC, about 50 GB/s, the kernel TCP path would need many cores per NIC just to copy data, and its per-message latency (tens of µs) makes small collectives slow. RDMA gets line rate with almost no CPU and ~2 µs latency. That's why NCCL uses the IB/RoCE transport when it can and treats TCP sockets as a fallback.
Where this shows up in your stack NCCL_DEBUG=INFO prints the transport: NET/IB (RDMA) vs NET/Socket (TCP), and GDRDMA when GPUDirect is used. NCCL's bootstrap (rank discovery) always uses TCP over NCCL_SOCKET_IFNAME, and so does torch.distributed's TCPStore. Two Sparks connected over their ConnectX-7 run RoCE, but host-memory only (no GDR), so NCCL stages through cudaHostAlloc buffers.

5Memory registration, pinned memory, and GPUDirect RDMA

What it is. Before the NIC can touch a buffer, the application calls ibv_reg_mr(pd, addr, len, access). Registration (1) pins every page so its physical frame can't change, (2) loads the virtual→physical translations into the NIC's own translation tables, and (3) returns keys: an lkey for local use and an rkey that a remote peer must present to read or write this region. Registration is slow (it faults in and pins every page, then programs the NIC), so applications register large buffers once and reuse them.

GPUDirect RDMA extends registration to GPU memory. With the nvidia-peermem module, or dma-buf on recent kernels with the open GPU driver, the NIC driver can ask the GPU driver for the physical (BAR) addresses of a cudaMalloc buffer. The NIC then DMAs straight to and from GPU memory over PCIe, ideally through a shared PCIe switch (topology PIX/PXB, Module 1).

Why it matters for AI workloads (completion criterion #6): RDMA hands the data path to the NIC: it reads and writes your memory by DMA, using translations it cached at registration time, with no kernel on the path to fix things up. If a page could move or be swapped, the NIC would silently DMA into a frame that now belongs to someone else. So the memory must be pinned (frames fixed for the registration's lifetime) and registered, so the NIC has the translations and the rkey/lkey checks enforce who may access it. Ordinary PCIe NICs can't take page faults mid-transfer. On-demand paging (ODP) NICs can, at a cost. Because pinning takes memory away from the kernel, it's bounded by RLIMIT_MEMLOCK (ulimit -l).
Where this shows up in your stack NCCL registers its own buffers and, with user-buffer registration (ncclCommRegister, or ncclMemAlloc / the PyTorch NCCL allocator), yours, so collectives can DMA from tensors in place. NCCL WARN Call to ibv_reg_mr failed almost always means the memlock limit, or on GPU memory, a missing nvidia-peermem/dma-buf path. On the Spark: no GDR, so NCCL registers pinned host buffers. NVIDIA's porting guide suggests applications do the same.

6cgroups and namespaces

What it is. A container is an ordinary process tree with two kinds of kernel fences around it:

Namespaces: what you can see

  • pid: your process is PID 1; host processes are invisible
  • net: own interfaces (usually just eth0 on a bridge), routes and ports. Host RDMA netdevs aren't visible unless you use --network=host
  • ipc: own SysV/POSIX IPC and its own /dev/shm tmpfs (64 MiB by default in Docker)
  • mnt: own filesystem view (image layers + volumes); devices appear only if passed in
  • uts, user, cgroup, time

cgroups v2: what you can use

  • memory.max: hard limit including page cache; above it, reclaim and then the OOM killer
  • cpu.max: quota per period. Over quota, all threads are throttled until the next period
  • cpuset.cpus / .mems: which CPUs and NUMA nodes
  • pids.max: processes plus threads
  • io.max: per-device bandwidth/IOPS
  • plus device access rules (which /dev nodes are usable)

GPUs get in through the NVIDIA Container Toolkit (--gpus or CDI), which bind-mounts /dev/nvidia* and the driver libraries. RDMA devices need /dev/infiniband/* passed in (or an RDMA device plugin), the right network namespace, and a memlock limit.

Why it matters for AI workloads: each fence has a default sized for web services, not for 8-GPU training: a 64 MiB /dev/shm, a small memlock limit (8 MiB on your Docker), a bridged network without RDMA devices, and CPU quotas that throttle the whole container. Distributed training trips over all of them.
Where this shows up in your stack Kubernetes: resources.limits.cpu becomes cpu.max (throttling), and a Guaranteed pod with integer CPUs plus the static CPU manager gets an exclusive cpuset instead (no throttling). Memory limits include the page cache, so a checkpoint load can push a pod over its limit even when the process's own memory looks fine. emptyDir: {medium: Memory} mounted at /dev/shm is the usual Kubernetes shm fix.

7What usually breaks distributed training in containers

Each scenario below is a real failure pattern. Read the symptom and the log, pick the cause, then check the explanation.

Fig 3 · Debug this container interactive
Why it matters for AI workloads (completion criterion #7): the usual culprits are (1) /dev/shm too small (DataLoader bus errors, NCCL SHM failures); (2) memlock limit (RDMA registration fails, or NCCL falls back to sockets); (3) RDMA devices or host network not visible (silent TCP fallback, 10× slower); (4) wrong bootstrap interface (hangs at init); (5) CPU quota throttling (sawtooth GPU utilization); (6) cgroup memory limit including page cache (exit 137 during checkpoint load); (7) missing GPU devices or a driver/library mismatch. Your Docker reproduced the first two by default: 64 MiB shm, 8 MiB memlock.
Where this shows up in your stack A "known good" docker run for multi-GPU training: --gpus all --ipc=host (or --shm-size=32g) --ulimit memlock=-1 --ulimit stack=67108864 --network=host --device=/dev/infiniband --cap-add=IPC_LOCK. The NCCL docs' own Docker example uses --shm-size=1g --ulimit memlock=-1 as a minimum.

8Observability: /proc, /sys, perf, eBPF, nsys

QuestionWhere to look
What is this process's memory really? (RSS/PSS/swap/THP)/proc/PID/smaps_rollup, /proc/PID/status
Is it being preempted?/proc/PID/task/TID/status (nonvoluntary_ctxt_switches), pidstat -wt 1
Is the machine short of CPU, memory or I/O right now?/proc/pressure/{cpu,memory,io} (PSI: % of time tasks stalled), vmstat 1
Where do interrupts land?/proc/interrupts, /proc/irq/N/smp_affinity_list
Container limits and throttling/sys/fs/cgroup/{memory.max,memory.events,cpu.max,cpu.stat}
Device topology and NUMA/sys/bus/pci/devices/*/numa_node, lstopo, nvidia-smi topo -m
Hot functions, user + kernelperf top -p PID, perf record -g + perf report
Syscall profile, low overheadperf trace -s -p PID (vs strace -c, which is much slower)
Anything else, live, in the kerneleBPF via bpftrace one-liners (below)
CPU ↔ GPU ↔ OS on one timelinensys profile --trace=cuda,nvtx,osrt --sample=cpu

eBPF runs small, verified programs inside the kernel at tracepoints and probes, so you can count and histogram events in production with low overhead. A few bpftrace one-liners worth keeping:

# who is making syscalls (by process)?
sudo bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
# page faults by process (minor + major)
sudo bpftrace -e 'software:page-faults:1 { @[comm] = count(); }'
# why does thread TID leave the CPU? (R = preempted, S = sleeping, D = I/O)
sudo bpftrace -e 'tracepoint:sched:sched_switch /args.prev_pid == TID/ { @[args.prev_state] = count(); }'
# disk I/O latency histogram
sudo bpftrace -e 'tracepoint:block:block_rq_issue { @s[args.dev, args.sector] = nsecs; }
  tracepoint:block:block_rq_complete /@s[args.dev, args.sector]/ { @us = hist((nsecs - @s[args.dev, args.sector]) / 1000); delete(@s[args.dev, args.sector]); }'

nsys with OS runtime tracing (--trace=osrt) adds the OS calls (pthread_mutex_lock, read, ioctl, poll, sem_wait…) to the same timeline as CUDA API calls and GPU kernels. That's how you connect a GPU gap to "the launch thread was in pthread_cond_wait for 3 ms waiting on the DataLoader".

Brendan Gregg's 60-second checklist, run on the host while the job runs: uptime → dmesg | tail → vmstat 1 → mpstat -P ALL 1 → pidstat 1 → iostat -xz 1 → free -m → sar -n DEV 1 → sar -n TCP,ETCP 1 → top. For AI nodes, add nvidia-smi dmon and the GPU's throttle reasons. In dmesg, look for OOM kills, GPU Xid errors and NIC link flaps.

Where this shows up in your stack The fastest triage for "training got slower": nsys a few steps (GPU gaps? which CPU call precedes them?), check cpu.stat throttling and nonvoluntary_ctxt_switches of the main thread, check the NCCL transport line in the init log, then mpstat for a single core pegged at 100% (a GIL-bound loader or launch thread).

DDrill: cache, containers, and the 60-second check

# 1. checkpoint load: cold vs warm page cache, O_DIRECT, safetensors -> GPU
bash drills/m4_io.sh              # uses sudo drop_caches if passwordless, else evicts just the file

# 2. container limits: default Docker vs fixed flags (any image with /bin/sh works)
docker run --rm -v "$PWD/drills:/d:ro" IMAGE sh /d/m4_container_check.sh
docker run --rm -v "$PWD/drills:/d:ro" --shm-size=16g --ulimit memlock=-1 IMAGE sh /d/m4_container_check.sh
#   with GPUs + RDMA:  --gpus all --ipc=host --ulimit memlock=-1 --network=host --device=/dev/infiniband

# 3. 60-second checklist while a job runs (sudo apt install sysstat first)
python3 drills/m3_train_step.py &   # or your real job
bash drills/m4_60s.sh $!

Expected output

x86 + RTX 4090 measured on your box

m4_io.sh (2 GiB, NVMe ext4)
 buffered cold  0.81 s  2.5 GiB/s
 buffered warm  0.22 s  8.9 GiB/s
 O_DIRECT       0.34 s  5.9 GiB/s
 load_file cuda 0.99 s -> 0.37 s

default docker:
 FAIL /dev/shm is 64 MiB
 FAIL ulimit -l = 8192 KiB
 WARN no /dev/infiniband
 WARN only lo/eth0 visible
 FAIL writing 128 MiB to /dev/shm
--shm-size=16g --ulimit memlock=-1 --cpus=4:
 OK   /dev/shm 16 GiB, ulimit -l unlimited
 WARN cpu.max = 400000 100000 -> 4 CPUs

DGX Spark expected, not yet measured

m4_io.sh: page cache and "GPU" are the
  same LPDDR5x; load_file(device="cuda")
  is DRAM->DRAM, expect warm loads well
  above the 4090's PCIe-bound 5.4 GiB/s

container check with --gpus all:
  /dev/infiniband present only if passed
  4 CX-7 netdevs visible only with
  --network=host
  GPU_DIRECT_RDMA_SUPPORTED = 0, so NCCL
  uses host-memory RDMA (needs memlock)
Spark load times and container behaviour are predictions. Docker's default memlock is inherited from the daemon's limits, so it can differ on DGX OS. Check the actual value.

?Final self-check: the seven completion criteria

These are the seven things you set out to explain without notes. Answer each aloud, then compare.

1. Why are pageable-memory copies slower, and why can't they be truly async? M1 §6, M2 §8
The GPU's copy engine is a DMA engine: it needs physical addresses that stay valid for the whole transfer. Pageable pages may not be allocated yet, and can be moved, swapped or COW-replaced by the kernel at any time. So the driver copies the data with the CPU (your calling thread) into its own pinned staging buffers, and DMAs from those. That's slower because of an extra pass through DRAM, a single core's memcpy throughput, and pipeline hand-offs (16 vs 20 GB/s on your 4090, and 8 GB/s from an E-core). It isn't async because the call can't return until the staging copy is done, so cudaMemcpyAsync from pageable memory holds the thread for about the whole transfer (≈120% measured), and nothing overlaps.
2. What happens step by step on a page fault, and what does a TLB miss cost? M2 §3–4
Fault: an access misses the TLB → the hardware walk finds the PTE not present (or write-protected) → the CPU traps to the kernel, recording the faulting address → the kernel finds the VMA and checks permissions (invalid → SIGSEGV) → it resolves the fault: zero a new frame (anonymous first touch), map a page-cache page (file, cached), start I/O and sleep (file, not cached: a major fault), or copy the page (COW) → it writes the PTE → it returns, and the instruction retries and hits. Minor ≈1 µs, major ≈ device latency (≈73 µs measured). A TLB miss is hardware only: up to 4 dependent page-table reads on x86-64, tens of cycles when they hit in cache and hundreds of ns when they go to DRAM. No kernel involvement. Huge pages cut both.
3. Why does GPU and NIC NUMA placement matter for NCCL? M1 §3–4
Inter-node traffic flows GPU ↔ PCIe ↔ NIC. If the GPU and its NIC share a PCIe switch (PIX/PXB), NCCL can use GPUDirect RDMA and the bytes never touch the CPU or host DRAM. If they hang off different sockets (SYS), the path crosses the socket interconnect, NCCL typically disables GDR, and it stages through host memory instead. That means more DRAM passes and a shared, slower link. NCCL's proxy thread and bounce buffers should also live on the GPU's NUMA node. Check nvidia-smi topo -m and the NCCL init log for which NIC each rank chose and whether GDRDMA is on.
4. Why do DataLoader workers seem to "leak" memory? M2 §7, M3 §5
Workers are forked, so they share the parent's dataset pages copy-on-write. CPython writes to an object's header (its refcount, and GC fields) whenever it's referenced, so merely reading dataset[i] dirties the page it lives on, and the kernel copies that page into the worker. Over an epoch each worker accumulates a private copy of every page it touched: USS and PSS grow while RSS barely moves (measured: 60 → 439 MiB per worker, flat for numpy storage). Fix: store data in numpy/Arrow/tensors or an mmap'd file, not millions of Python objects. Note that Python 3.14 defaults to forkserver, which changes the trade-off.
5. What does mmap buy you for checkpoint loading? M2 §6
No copy (tensors view page-cache pages directly instead of read() copying into private buffers, which also first-touch faults those buffers), lazy loading (only the tensors you touch are read), sharing (N ranks share one cached copy, and it survives restarts) and kernel-managed readahead and eviction. Measured warm: 0.14 s vs 0.80 s for read() on 2 GiB. It doesn't speed up cold I/O, cold-page touches are major faults that stall the thread, and the page cache counts against cgroup memory.
6. Why does RDMA need pinned, registered memory? M4 §4–5
The NIC moves data by DMA on its own, using virtual→physical translations it was given at registration, with no kernel on the data path. If the kernel could move or swap a page, the NIC would read or write the wrong frame, and ordinary NICs can't handle page faults mid-transfer. Registration pins the pages (frames fixed), loads the translations into the NIC, and issues keys (lkey/rkey) so only authorized local and remote operations can touch the region. Pinning is limited by ulimit -l, and GPU memory additionally needs nvidia-peermem/dma-buf for GPUDirect RDMA (not available on the Spark).
7. What usually breaks distributed training inside containers? M4 §6–7
(1) /dev/shm at Docker's 64 MiB default: DataLoader "bus error / insufficient shared memory", NCCL SHM segment errors. Fix: --ipc=host or --shm-size. (2) The memlock ulimit: ibv_reg_mr fails. Fix: --ulimit memlock=-1. (3) No RDMA devices or host netdevs: NCCL silently uses TCP sockets. Fix: --device=/dev/infiniband, --network=host, IPC_LOCK. (4) The wrong bootstrap interface (docker0, a bridge IP): init hangs. Fix: NCCL_SOCKET_IFNAME. (5) A CPU quota: throttled launch threads and a sawtooth GPU. Fix: exclusive cpusets. (6) A cgroup memory limit including page cache: exit 137 while loading checkpoints. (7) Missing GPU devices or a driver/library mismatch.
Bonus: why is O_DIRECT faster than buffered for a cold read here, but slower than a warm buffered read?
Cold buffered reads pay for the device and a CPU copy out of the page cache, with readahead that may not keep enough I/O in flight. O_DIRECT with 16 MiB requests DMAs straight into the user buffer with deep, large requests. Warm buffered reads don't touch the device at all, just a memory copy from the cache, so they win whenever data is reused.
Saved in this browser and shown on the index page.

§Sources