Module 2 · OS fundamentals for AI systems

Virtual memory

Every pointer in your process, including every data_ptr(), is a virtual address. Hardware and the kernel translate each one to a physical location on every access. This module covers how that translation works, what it costs, and why pinned memory, mmap'd checkpoints, expandable_segments, SymmetricMemory and Grace's coherent memory are all versions of the same idea.

≈45 min reading3 interactive diagramsDrill ≈20 minTeaches criteria #1 (completes it), #2, #5, and the memory side of #4 and #6

1Address spaces

What it is. Each process gets its own virtual address space: a private map from virtual addresses to physical memory. On x86-64 and ARM64 that map is usually 48 bits wide, giving each process 128 TiB of user-space addresses, almost all of them unmapped. The kernel keeps a list of the ranges that are valid, called VMAs (virtual memory areas): the code, heap, stacks, each mmap, each shared library. /proc/PID/maps prints that list.

How it works. Memory is managed in fixed-size pages, 4 KiB by default on x86 and on most ARM64 kernels. Physical memory is cut into page-sized frames. A per-process page table says which virtual page maps to which frame, with permission bits. Two processes can map the same frame: that's how shared libraries, shared memory and the page cache are shared without copying.

Fig 1 · Two address spaces, one physical memory

Why it matters for AI workloads: isolation (a crashing worker can't scribble on the trainer), sharing (eight ranks on a node mmap one checkpoint and the page cache holds it once), and laziness (allocating 100 GB of address space costs nothing until you touch it) all come from this one indirection. So does every cost in this module.

Where this shows up in your stack CUDA's Unified Virtual Addressing (UVA) puts host and device allocations in one 64-bit virtual address space, so a pointer value alone says where it lives (cudaPointerGetAttributes). The GPU has its own page tables and MMU. Its VA space is per CUDA context, and the driver keeps it consistent with host addresses for pinned and managed memory.

2Page tables and the multi-level walk

What it is. A flat table covering 2⁴⁸ bytes in 4 KiB pages would need 2³⁶ entries per process. Instead, the page table is a radix tree. On x86-64 with 4 KiB pages it has 4 levels (PML4 → PDPT → PD → PT). Each level is one 4 KiB page holding 512 eight-byte entries, so each level consumes 9 bits of the virtual address, and the last 12 bits are the offset within the page. Subtrees for unmapped regions simply don't exist.

How it works. A CPU register holds the physical address of the top table: CR3 on x86, TTBR0_EL1 on ARM64. The hardware page walker slices the virtual address into indices, reads one entry per level, and at the bottom finds the PTE: the frame number plus bits such as present, writable, user, accessed, dirty and no-execute. A 4-level walk costs four dependent memory reads before the real access can start. A "huge" entry higher up (2 MiB or 1 GiB) ends the walk early.

Fig 2 · Page-table walk interactive

Table contents are made up but consistent; the bit slicing is real. ARM64 with a 64 KiB granule and 48-bit VAs uses 3 levels (6 + 13 + 13 index bits + 16 offset bits). Some kernels use 42-bit VAs and only 2 levels.

Why it matters for AI workloads: the page walk is pure overhead on every TLB miss (next section). It's also the data structure the kernel edits for every mmap, munmap, fault, fork and pin, so page-table work shows up in allocation-heavy code and in multi-GB pin/unpin operations.

Where this shows up in your stack The GPU walks its own multi-level page tables, with large pages: cudaMalloc memory is mapped in 2 MiB pages (your 4090's cuMemCreate granularity is 2 MiB). That's one reason the CUDA caching allocator rounds large blocks to 2 MiB. On Grace and GB10 the GPU can also walk the CPU's page tables through the system MMU (ATS, §10).

3The TLB and what a miss costs

What it is. The translation lookaside buffer is a small cache of recent virtual→physical translations, per core. A modern x86 core has roughly 64–96 entries in its L1 data TLB and about 2,000 in a unified second-level TLB. A hit costs essentially nothing, because it runs in parallel with the L1 cache lookup.

How it works. On a miss, the page walker reads the 4 levels. The upper levels usually hit in dedicated paging-structure caches and in L1/L2, so a typical miss costs tens of cycles. A miss whose entries also miss the caches costs up to four DRAM round trips, several hundred ns, before the data access itself. The useful metric is TLB reach = entries × page size: 2,000 × 4 KiB ≈ 8 MiB, but 2,000 × 2 MiB ≈ 4 GiB. Random access over a working set much larger than the reach misses constantly.

The TLB also has to be kept correct. When the kernel changes or removes a mapping (munmap, migration, COW break), every core that might cache it must drop the entry. On x86 that means inter-processor interrupts, called a TLB shootdown. Context switches between processes change CR3; tagged TLBs (PCID on x86, ASID on ARM) avoid a full flush.

Event	Rough cost	When it happens
TLB hit	~0	hot working set within reach
TLB miss, walk hits cache	~10–40 cycles	sequential scans, moderate working sets
TLB miss, walk misses to DRAM	100s of ns	random access over many GB (embedding tables, hash maps)
TLB shootdown	µs, on every affected core	munmap / free of large blocks, page migration, mprotect

Why it matters for AI workloads (completion criterion #2, second half): a TLB miss costs a page walk: 1–4 memory reads, cheap if the page-table entries are cached and hundreds of ns if not. It isn't a fault, and the kernel isn't involved. Workloads that hop around many GB of host memory (CPU embedding lookups, tokenizer hash tables, giant pinned staging pools) pay it constantly, and huge pages are the main fix.

Where this shows up in your stack GPUs have TLBs too, and GPU TLB misses are expensive. That's why UVM and HMM workloads that touch scattered host pages from kernels run far below link bandwidth. Large GPU pages (2 MiB) make the GPU's reach enormous for cudaMalloc memory. Frequent cudaFree / cudaHostUnregister of large blocks triggers unmaps and shootdowns. One of the reasons PyTorch caches allocations is to avoid that churn.

4Page faults: minor and major

What it is. A page fault is a CPU exception raised when the walk finds no valid translation, or one whose permissions forbid the access. It's normal and constant. Linux allocates memory lazily: malloc(1 GB) just reserves address space, and each page gets a real frame at its first touch.

Minor fault: resolved without I/O. Examples: first touch of anonymous memory (allocate and zero a frame), a file page already in the page cache (just map it), a copy-on-write break. Measured on your box: ≈0.8 µs per 4 KiB first-touch fault.
Major fault: needs I/O. The page must be read from disk (or swap) first, and the thread sleeps until it arrives. Measured on your box: ≈73 µs per random 4 KiB checkpoint page from NVMe, about 100× a minor fault.
Invalid: no VMA covers the address, or the permissions forbid the access. The kernel delivers SIGSEGV.

Fig 3 · Inside the page-fault handler interactive

Simplified Linux path (do_user_addr_fault → handle_mm_fault). Since 6.1 VMAs are kept in a maple tree, and since 6.4 most faults take a per-VMA lock instead of the process-wide mmap_lock.

Why it matters for AI workloads (completion criterion #2): step by step: the access misses the TLB → the walk finds no present PTE → the CPU traps to the kernel with the faulting address → the kernel finds the VMA and checks permissions → it gets a frame (zero it, find it in the page cache, read it from disk, or copy it for COW) → it writes the PTE → it returns, and the instruction re-executes and now hits. First-touching a 100 GB host buffer costs 25 million minor faults (≈20 s of CPU at ≈0.8 µs each) unless huge pages cut the count by 512×. That's why "allocate then touch" in a hot path is a bug.

Where this shows up in your stack cudaHostAlloc / pin_memory() pre-fault and lock every page up front: slow to allocate, never faults afterwards. That's why PyTorch's pinned-memory allocator caches blocks. /usr/bin/time -v prints major and minor fault counts, perf stat -e page-faults,major-faults counts them, and the fault columns in ps -o min_flt,maj_flt show the running total. GPU page faults in UVM (§10) follow the same pattern, but the "handler" is the CUDA driver migrating pages over the link.

5Huge pages and THP

What it is. A huge page maps 2 MiB (or 1 GiB) with one entry higher in the tree: one TLB entry instead of 512, one fault instead of 512, and one level less to walk. Linux offers two mechanisms:

hugetlbfs: a pool reserved up front (vm.nr_hugepages), used explicitly via MAP_HUGETLB or a hugetlbfs mount. Guaranteed, but inflexible.
Transparent Huge Pages (THP): the kernel uses a 2 MiB page automatically when an aligned 2 MiB range of anonymous memory is faulted in, and the background thread khugepaged later collapses 4 KiB pages into huge ones. The mode in /sys/kernel/mm/transparent_hugepage/enabled is always, madvise (only regions marked with madvise(MADV_HUGEPAGE); the Ubuntu default and your box's setting) or never.

Costs. A huge page needs 2 MiB of physically contiguous free memory. On a fragmented machine the kernel must compact memory first (the defrag setting decides whether a faulting thread stalls for it), and that shows up as latency spikes. Huge pages also waste memory when only part of each 2 MiB is used. Since kernel 6.8, multi-size THP (mTHP) can also use intermediate sizes such as 16–64 KiB for anonymous memory. Your box exposes them under /sys/kernel/mm/transparent_hugepage/hugepages-*, all disabled except 2 MiB.

ARM64 is different. ARM64 kernels can use a 4 KiB, 16 KiB or 64 KiB base page (the "granule"). With 64 KiB pages a single TLB entry already covers 16× more, and the PMD-level huge page is 512 MiB instead of 2 MiB. ARM also has a "contiguous bit" that lets 16 adjacent 4 KiB PTEs share one TLB entry.

Which granule DGX OS uses on the Spark wasn't stated in any NVIDIA document I found. Community kernel projects offer both 4 KiB and 64 KiB builds for GB10. The drill prints getconf PAGESIZE: 4096 means 2 MiB THP, as on x86; 65536 means 512 MiB THP and very different fault counts.

Measured on your x86 box (1 GiB anonymous, first touch)	faults	first-touch time	random 8 B reads
`MADV_NOHUGEPAGE` (4 KiB pages)	262,144	208 ms	9.6 ns
`MADV_HUGEPAGE` (2 MiB THP)	6,644	130 ms	8.5 ns

The faults drop 40× (not the ideal 512×, because the region's unaligned edges and any range the kernel couldn't back with a huge page fall back to 4 KiB). The random-read gain is modest here because numpy issues many independent loads in parallel, which hides walk latency. A dependent pointer chase would show a larger difference.

Why it matters for AI workloads: THP helps large, long-lived, densely used CPU buffers: pinned staging pools, CPU-offloaded optimizer state, embedding tables. It can hurt latency-sensitive services through compaction stalls, and memory-tight hosts through bloat. That's why madvise mode, where the application opts in, is the common default.

Where this shows up in your stack PyTorch's CPU allocator can request THP for large allocations through the THP_MEM_ALLOC_ENABLE=1 environment variable (check that your version has it). jemalloc and tcmalloc, often LD_PRELOADed into data pipelines, have their own huge-page options. AnonHugePages in /proc/meminfo and /proc/PID/smaps_rollup shows how much memory THP is actually backing.

6mmap and the page cache

What it is. mmap creates a VMA. Either it's anonymous (zero-filled memory, which is how malloc gets big blocks) or it's file-backed: the range of addresses is the file. The kernel keeps file data in the page cache, a system-wide cache of file pages in otherwise-free RAM. A file-backed mapping points your page table straight at page-cache frames.

How it works. Nothing is read at mmap() time. Touching a page faults. If the page is in the page cache, the fault is minor and the kernel just maps it. If not, it's major: the kernel reads it, usually with readahead of the following pages, and maps several neighbours at once (fault-around, 64 KiB by default). MAP_SHARED writes go back to the file. MAP_PRIVATE writes get a private copy (COW). madvise hints change the policy: MADV_SEQUENTIAL / MADV_WILLNEED read ahead aggressively, MADV_RANDOM disables readahead, MADV_DONTNEED drops pages.

Load a 2 GiB checkpoint (your box, NVMe ext4)	cold cache	warm cache	faults (warm)
`read()` into a new buffer	1.33 s	0.80 s	524k minor
`mmap` + touch every page, sequential	0.87 s	0.14 s	33k minor
`mmap` + `MADV_RANDOM`, 64k random pages	4.80 s (65,536 major)	0.05 s	31k minor
safetensors `safe_open().get_tensor()`	1.47 s	0.73 s	104k minor

read() pays twice: the kernel copies every byte out of the page cache, and the destination buffer takes 524k first-touch faults. A warm mmap just maps page-cache frames, 16 pages per fault thanks to fault-around. Disabling readahead turns the same file into 65,536 synchronous 4 KiB reads.

Why it matters for AI workloads (completion criterion #5): mmap gives you (1) no copy: tensors can view page-cache memory directly; (2) laziness: you pay only for the pages you touch, so loading one shard's tensors, or inspecting metadata, is nearly free; (3) sharing: N ranks on a node mapping one file share one copy in RAM, and a restarted process finds it still cached; (4) kernel-managed I/O: readahead and eviction for free. The costs: the first touch of a cold page is a major fault that stalls the touching thread, page-cache memory counts against cgroup limits, and the data still has to be copied once into GPU memory (on discrete GPUs).

Where this shows up in your stack safetensors opens files via mmap. torch.load(path, mmap=True) (PyTorch ≥ 2.1) maps the checkpoint's storages instead of reading them. Combined with torch.device("meta") model init and load_state_dict(assign=True), the CPU never holds a second copy. On the Spark, page-cache pages and GPU allocations share the same 128 GB, which is why cudaMemGetInfo under-reports free memory there.

7fork and copy-on-write

What it is. fork() creates a child process with a copy of the parent's address space, without copying memory. Both processes' page tables point at the same frames, and every private writable page is marked read-only in both. When either process writes to such a page, the write faults, and the kernel copies that one page for the writer and makes it writable. That's copy-on-write (COW).

Fig 4 · fork + COW, page by page interactive

Parent (trainer) page table

Child (DataLoader worker 0) page table

PHYSICAL FRAMES (refcount = number of page tables pointing at it)

RSS counts every resident page a process maps, shared or not. PSS splits shared pages between sharers. USS counts only pages this process holds alone. Each page here stands in for many thousands.

Why it matters for AI workloads (completion criterion #4, memory half): in CPython, reading an object writes to it, because every reference bumps ob_refcnt in the object's header (and the cyclic GC writes its own headers). A DataLoader worker that merely indexes a Python list of a million samples writes to every page those objects live on, so each worker slowly gets a private copy of the dataset. Measured in the Module 3 drill on your box: each of 4 workers went from 60 MiB to 439 MiB of private memory, 1.76 GB in total, while RSS stayed flat. It isn't a leak. The memory is COW copies created by refcount writes.

Where this shows up in your stack Fixes: store sample metadata in numpy arrays, Arrow tables or torch tensors (one object header, no per-item refcounts), or in a file you mmap. gc.freeze() before creating workers stops the GC from walking the objects, but not refcount writes. Python 3.12's immortal objects only cover built-ins like None and small ints. Fork also has a CUDA problem: Module 3 covers why fork() after CUDA init is unsafe.

8Pinning, DMA, and the IOMMU

What it is. A device doing DMA needs addresses that stay valid for the whole transfer. Virtual memory works against that: the kernel may not have allocated a page yet (lazy), may move it (compaction for huge pages, NUMA balancing, memory tiering), may swap it out or drop it if it's a clean file page, or may replace it on a COW break. Any of these silently changes the physical frame behind a virtual address, and the device would read or write the wrong memory.

Pinning (pin_user_pages in the kernel) faults every page in and takes a reference that forbids migration, swap and reclaim until it's released. Then the driver can hand physical (bus) addresses to the device. cudaHostAlloc allocates pinned memory and also maps it into the GPU's address space. cudaHostRegister pins memory you already have. RDMA's ibv_reg_mr pins and gives the NIC a translation table (Module 4).

The IOMMU (Intel VT-d, AMD-Vi, ARM SMMU) is an MMU for devices. Devices issue I/O virtual addresses, and the IOMMU translates them through its own page tables and IOTLB. That confines each device to the memory it was granted, and it's what makes passing a GPU or NIC into a VM or container safe. Pinning is still required: IOMMU page tables are set up per mapping, and ordinary PCIe devices can't take a page fault mid-DMA. On GPU servers the IOMMU is often set to passthrough (iommu=pt) or tuned. NCCL's troubleshooting guide covers IOMMU and PCIe ACS settings, which can force peer-to-peer traffic up through the root complex.

Why it matters for AI workloads (completes criterion #1): a pageable copy is slow and synchronous because of virtual memory. The copy engine needs pinned frames, so the driver stages pageable data through its own pinned buffers with a CPU memcpy, and your thread does that memcpy. Pinning moves the work to allocation time, and every later copy is a pure DMA. The price is memory the kernel can't reclaim. Pin 200 GB on a 256 GB host and the page cache, and everything else, gets squeezed.

Where this shows up in your stack ulimit -l (RLIMIT_MEMLOCK) limits mlock and RDMA registration for non-root processes, and it's a classic container outage (Module 4). In practice CUDA's own pinning goes through the NVIDIA kernel driver and isn't bounded by it. PyTorch's CachingHostAllocator keeps pinned blocks around because cudaHostAlloc is expensive: it faults and pins every page and updates GPU mappings. Mlocked in /proc/meminfo doesn't include CUDA-pinned memory, so look at the process's RSS instead.

9GPU virtual memory: VMM APIs, expandable segments, SymmetricMemory

What it is. cudaMalloc hands you virtual addresses with physical memory already attached. The CUDA Virtual Memory Management (VMM) driver API splits that into the same steps the OS uses:

VMM call	OS analogue	What it does
`cuMemAddressReserve`	`mmap(PROT_NONE)`	reserve a range of GPU virtual addresses: no memory yet
`cuMemCreate`	allocating frames	create a physical allocation (a handle), in multiples of the granularity (2 MiB on your 4090)
`cuMemMap` / `cuMemUnmap`	writing PTEs	map a handle (or a slice of one) at a VA in the reserved range
`cuMemSetAccess`	`mprotect`	grant read/write to specific devices: this is also how peers get access
`cuMemExportToShareableHandle`	passing an fd over a socket	export the physical allocation as a POSIX fd (or a fabric handle across nodes) so another process can import and map it

Fig 5 · Same physical chunks, three mapping tricks

Why it matters for AI workloads: separating "address" from "memory" fixes two GPU problems. Fragmentation: a caching allocator with fixed segments strands free blocks that are too small to reuse when shapes change. Growing one segment by mapping more pages avoids that. Communication: mapping a peer's memory into your address space turns a collective into ordinary loads and stores from a kernel, with no copy engine, no proxy and no host involvement. It's the GPU version of shared memory between processes.

Where this shows up in your stack PYTORCH_ALLOC_CONF=expandable_segments:True (PYTORCH_CUDA_ALLOC_CONF is the legacy alias) uses the VMM path, per the PyTorch docs: "one segment (per stream) that grows as necessary". torch.distributed._symmetric_memory (alpha) gives each rank a handle with "pointers to symmetric buffer on peer ranks, multicast pointer (if supported), and signal pads". NCCL's own cuMem mode, NVLS and user-buffer registration (ncclMemAlloc) build on the same calls. Your 4090 reports MULTICAST_SUPPORTED = 0: multicast needs NVSwitch.

10UVM, HMM and ATS: one pointer for CPU and GPU

What it is. Three ways to let a GPU kernel dereference "host" pointers. Each one is a bigger step towards the GPU sharing the CPU's page tables:

Mechanism	Which memory	How a GPU access is resolved	Where you have it
UVM / managed memory `cudaMallocManaged`	only managed allocations	GPU page fault → the CUDA driver migrates the page (in chunks up to ~2 MiB) between host and device and updates both page tables. Hints: `cudaMemPrefetchAsync`, `cudaMemAdvise`	both machines
HMM (Heterogeneous Memory Management)	any `malloc` / `mmap` memory	a Linux kernel feature: the driver mirrors CPU page tables into the GPU's in software (and migrates pages), staying in sync via MMU notifiers. Needs CUDA 12.2+, open kernel modules, Linux 6.1.24+ / 6.2.11+	x86 + 4090 if using the open modules (your 4090 reports `PAGEABLE_MEMORY_ACCESS = 0` today)
ATS (Address Translation Services)	any system memory	hardware coherence over NVLink-C2C: the GPU's MMU asks the system MMU, which walks the CPU's own page tables. HMM is disabled when ATS is present	Grace Hopper, Grace Blackwell, and GB10 (Hot Chips lists ATS via GMMU + SMMU)

CUDA reports which one applies through device attributes. The drill prints them all. pageableMemoryAccess = 1 means kernels may use pageable memory directly (HMM or ATS). pageableMemoryAccessUsesHostPageTables = 1 means it's hardware (ATS), and 0 means software (HMM).

On the Spark specifically. There's no separate GPU memory. malloc, cudaMalloc, the page cache and pinned buffers are all LPDDR5x, and the GPU's L2 caches system physical addresses coherently. With ATS a kernel can read a tensor that came from np.load or an mmap'd checkpoint in place, with no copy and no migration. Whether that beats copying first depends on access pattern and TLB behaviour, so measure it.

I expect PAGEABLE_MEMORY_ACCESS = 1 and USES_HOST_PAGE_TABLES = 1 on the Spark (ATS per the GB10 Hot Chips deck), and have not confirmed it on hardware. How a GPU access to a never-touched malloc page gets serviced on GB10, and at what cost, is also not documented in what I found. Run python3 drills/m2_vm.py info on the Spark.

Why it matters for AI workloads: on a discrete GPU, "unified" memory is a convenience with a price: every first touch from the other side is a fault plus a migration over PCIe, and thrashing (CPU and GPU taking turns on the same pages) is brutal. It's fine for oversubscription experiments and prototyping, not for a hot path. On coherent systems (Grace, GB10) system memory is directly addressable at link speed, which is why GH200 and GB200 can offload KV cache, optimizer state or embeddings to CPU memory far more cheaply than an x86 + PCIe box.

Where this shows up in your stack PyTorch doesn't use managed memory by default. The caching allocator uses cudaMalloc or the VMM path. PyTorch can't pass a raw malloc pointer into a CUDA kernel today without a copy, but custom kernels, Triton and CuPy can on ATS/HMM systems. Watch nvidia-smi -q for the addressing mode (HMM / ATS / none), and treat cudaMemGetInfo on the Spark as a lower bound.

DDrill: faults, THP, and page size on both machines

# everything at once (creates a 2 GiB checkpoint in $HOME unless you pass a path)
bash drills/m2_vm.sh  [CKPT_PATH]  [GB]

# or piece by piece
python3 drills/m2_vm.py info                       # page size, THP, CUDA unified-memory attributes
python3 drills/m2_vm.py make  ~/osf_ckpt.safetensors 2
python3 drills/m2_vm.py evict ~/osf_ckpt.safetensors       # cold (no root needed)
/usr/bin/time -v python3 drills/m2_vm.py load ~/osf_ckpt.safetensors mmap
/usr/bin/time -v python3 drills/m2_vm.py load ~/osf_ckpt.safetensors mmap   # warm
python3 drills/m2_vm.py load ~/osf_ckpt.safetensors mmap-rand    # readahead off
python3 drills/m2_vm.py thp 1                      # MADV_NOHUGEPAGE vs MADV_HUGEPAGE

# whole-system cold cache instead of one file
sync; sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'

Expected output

x86 + RTX 4090 measured on your box

base page size: 4096 bytes
THP enabled: always [madvise] never
THP PMD size: 2 MiB
PAGEABLE_MEMORY_ACCESS            0
..._USES_HOST_PAGE_TABLES         0
CONCURRENT_MANAGED_ACCESS         1
VIRTUAL_MEMORY_MANAGEMENT         1
MULTICAST_SUPPORTED               0
cuMemCreate granularity: 2 MiB

read  cold 1.33s warm 0.80s  524k minor
mmap  cold 0.87s warm 0.14s   33k minor
mmap-rand cold 4.8s: 65536 major faults
NOHUGEPAGE 262144 faults 208 ms
HUGEPAGE     6644 faults 130 ms

DGX Spark expected, not yet measured

arch aarch64, page size 4096 or 65536 (check!)
  4 KiB -> THP PMD 2 MiB, like x86
  64 KiB -> THP PMD 512 MiB, and ~16x
           fewer faults for the same file
INTEGRATED                        1
PAGEABLE_MEMORY_ACCESS            1 (expected)
..._USES_HOST_PAGE_TABLES         1 (expected: ATS)
HOST_NATIVE_ATOMIC_SUPPORTED      1 (expected)
GPU_DIRECT_RDMA_SUPPORTED         0 (per NVIDIA docs)

mmap warm: faster than x86 per GB if the
  page size is 64 KiB (fewer faults)

The Spark column is reasoned from the GB10 Hot Chips deck and the DGX Spark porting guide, not measured. Replace it with your output.

What to notice: (1) warm vs cold is the page cache at work; (2) read's fault count equals the file's page count because it's faulting in a private copy, while mmap shares the cache; (3) random access with readahead off turns every page into a major fault; (4) THP divides first-touch faults by up to 512.

?Self-check

Walk through, step by step, what happens when a thread first writes to a page of a 1 GB buffer it just got from malloc. criterion #2

The store misses the TLB → the hardware walks the page table and finds no present PTE (malloc only created a VMA) → the CPU raises a page fault, saves the faulting address (CR2 / FAR_EL1) and enters the kernel → the kernel finds the VMA covering the address and checks that writes are allowed → it's anonymous memory with no page yet, so it allocates a frame, zeroes it, and writes a PTE (present, writable, user) → it returns from the exception → the store re-executes, the walk succeeds, the TLB caches the translation. Minor fault, ≈1 µs. Repeat 262,144 times for 1 GB of 4 KiB pages, or about 512 times with 2 MiB THP.

What does a TLB miss cost, and how does it differ from a page fault? criterion #2

A TLB miss is handled entirely by hardware: the page walker reads up to 4 page-table entries (x86-64, 4 KiB pages). If those entries are in the caches, it's tens of cycles. If they miss to DRAM, it's several hundred ns, before the actual data access. No kernel, no exception. A page fault happens when the walk finds no valid entry. It's an exception into the kernel: ≈1 µs for minor faults, tens to hundreds of µs for major faults that need disk I/O. Huge pages reduce both: fewer TLB entries needed (more reach, a shorter walk) and fewer faults.

Your colleague says "mmap'ing the checkpoint is pointless, it still has to read the file". What does mmap actually buy you? criterion #5

(1) No copy: read() copies every byte from page cache into a private buffer (and first-touch faults that buffer). mmap maps the page-cache frames directly, so a warm load was 0.14 s vs 0.80 s on your box. (2) Laziness: only touched pages are read, so loading a subset of tensors or a single shard costs only that. (3) Sharing: N processes on a node share one copy in the page cache, and it survives process restarts. (4) The kernel handles readahead and eviction. It doesn't make cold I/O faster, and a cold page touched from a hot thread is a major fault that stalls it.

Why must memory be pinned before a GPU copy engine or NIC can DMA from it, even with an IOMMU? criterion #1 / #6

The device works with physical (or IOMMU-translated) addresses, set up once for the transfer. Virtual memory lets the kernel change the frame behind a virtual page at any time: lazy allocation, migration and compaction, swap or reclaim, COW breaks. Ordinary PCIe devices can't take a page fault in the middle of a DMA. Pinning faults every page in and forbids moving it, so the address handed to the device stays valid. The IOMMU adds protection and remapping, but its own page tables still point at fixed frames.

Four DataLoader workers each show RSS 800 MB, but the node's free memory drops by 1.7 GB over the first epoch. Explain with PSS/USS. criterion #4

RSS counts shared COW pages in full in every process, so it barely changes. As workers read Python objects, refcount writes break COW and each worker gets private copies, so USS (and PSS) grow. The real memory cost is the sum of USS plus the shared pages once. Measured: USS went 60 → 439 MiB per worker. Fix with numpy/Arrow/tensor storage or an mmap'd file.

What problem does expandable_segments solve, and which CUDA APIs make it possible?

Fragmentation in the caching allocator: with fixed segments, a workload whose tensor sizes change (variable batch or sequence lengths) leaves free blocks stranded inside segments that can't be merged or returned, so you OOM while "reserved but unallocated" memory is high. With the VMM API, the allocator reserves a big virtual range (cuMemAddressReserve) and maps more physical chunks at its end as needed (cuMemCreate + cuMemMap), so one segment per stream grows and shrinks instead.

On a GH200 or a Spark, a CUDA kernel reads a NumPy array allocated with plain malloc. What makes that legal, and why is it a bad idea on your 4090 box?

ATS: the GPU's MMU translates the pointer by asking the system MMU, which walks the CPU process's page tables, and NVLink-C2C keeps caches coherent (pageableMemoryAccessUsesHostPageTables = 1). On the 4090 box, PAGEABLE_MEMORY_ACCESS = 0, so it would just fault. Even with HMM enabled (open modules), each GPU touch of a host page means a software-handled fault, mirroring or migration over PCIe at ≈20 GB/s with high per-page latency. For hot data, copy it to the GPU explicitly.

Saved in this browser and shown on the index page.

§Sources

Linux kernel docs: Transparent Hugepage Support (modes, defrag, khugepaged, multi-size THP), HugeTLB pages, x86-64 memory map (4- and 5-level paging), ARM64 memory layout (4K/16K/64K granules).
Linux kernel docs: pin_user_pages() and related calls (why DMA needs pinned pages, FOLL_LONGTERM).
LWN: The maple tree (VMA tree, 6.1); Per-VMA locks (6.4).
NVIDIA, CUDA Programming Guide: Unified and System Memory: pageableMemoryAccess, pageableMemoryAccessUsesHostPageTables, ATS on Grace Hopper/Blackwell disables HMM.
NVIDIA, CUDA Driver API: Virtual Memory Management and Multicast Object Management.
NVIDIA blog / Phoronix: CUDA 12.2 HMM support (open kernel modules, Linux 6.1.24+ / 6.2.11+).
GB10 at Hot Chips 2025: ATS through GMMU + SMMU, hardware CPU–GPU coherency, physically-tagged GPU L2.
NVIDIA, DGX Spark optimization (UMA, cudaMemGetInfo) and porting: CUDA (no GPUDirect RDMA).
PyTorch, CUDA semantics: expandable_segments, PYTORCH_ALLOC_CONF, pinned memory; SymmetricMemory (alpha); torch.load(mmap=True).
PyTorch dev-discuss: SymmetricMemory: harnessing NVLink programmability.
NCCL troubleshooting: IOMMU / PCI ACS and P2P.
Your measurements: drills/m2_vm.py on i9-13900K + RTX 4090, kernel 6.8, driver 580.178; drills/m3_cow_leak.py for the COW numbers.