The CPU side: processes, threads, scheduling
The GPU only runs what a CPU thread tells it to, when that thread gets to run. This module covers how the kernel decides who runs, what it costs to cross into the kernel and back, and why one descheduled Python thread can leave an H100 idle.
1The user/kernel boundary
What it is. The CPU runs in two privilege levels that matter here. Your code runs in user mode (ring 3 on x86, EL0 on ARM64). The kernel runs in kernel mode (ring 0 / EL1), where it can touch page tables, devices and other processes' memory. User code can't simply call into the kernel. It must use a controlled entry: a system call (deliberate), an exception such as a page fault (accidental), or an interrupt (a device or timer, unrelated to what's running).
libcuda writes commands into a ring buffer mapped into your process and rings a doorbell register on the GPU, also mapped in, so a kernel launch is usually zero syscalls. RDMA does the same for the NIC (Module 4). Anything that does enter the kernel (allocations, cudaHostAlloc, blocking syncs, file I/O, sockets) costs at least hundreds of ns and might put your thread to sleep.strace a training step is dominated by futex (threads waiting on each other), ioctl (the CUDA driver talking to /dev/nvidia* for allocations, events and some syncs), poll/read (DataLoader queues) and mmap/munmap (allocators). If you see thousands of ioctls per step, something is allocating or synchronizing in the loop.2Syscalls and context switches, step by step
How a syscall works. libc puts the syscall number and arguments in registers and executes syscall (x86-64) or svc #0 (ARM64). The CPU switches to kernel mode and jumps to a fixed entry point. The kernel saves user registers on the thread's kernel stack, runs the handler, and returns with sysret / eret. A trivial syscall costs ≈100–300 ns round trip, more with Spectre/Meltdown mitigations. Some calls never enter the kernel: clock_gettime reads a kernel-maintained page (the vDSO) mapped into every process.
How a context switch works. When a thread blocks (waits for data, a lock, I/O) or uses up its time slice, the kernel calls schedule(), which picks the next runnable thread on that CPU. It saves the old thread's registers and stack pointer and loads the new one's. If the new thread belongs to another process, it also switches the page-table root (CR3 / TTBR0), and the TLB entries tagged with the old address space become useless. The direct cost is ≈1–5 µs. The bigger cost is indirect: the new thread starts with cold caches and TLB, and the old thread finds them cold when it returns.
svc instead of syscall, x8 holds the syscall number, TTBR0_EL1 plays the role of CR3, and the return is eret. Thread T1 is your trainer's main thread and T2 is a DataLoader worker process./proc/PID/status has voluntary_ctxt_switches (blocked by choice: waiting on futex or I/O) and nonvoluntary_ctxt_switches (preempted: someone else needed the CPU). A climbing involuntary count on your launch thread means it's sharing a core. pidstat -wt 1 shows both per thread. cudaDeviceScheduleBlockingSync turns GPU waits into voluntary sleeps, trading CPU for wake-up latency.3Processes and threads
What it is. Linux schedules tasks. A process is a group of tasks that share one address space (page tables), file-descriptor table and signal handlers. A thread is another task in that group. Both are created by clone(): flags decide what's shared. Each thread has its own registers, stack and kernel stack, and is scheduled independently.
| Threads (one process) | Processes | |
|---|---|---|
| Memory | shared: same page tables | separate (COW after fork, or explicit shared memory) |
| Switch cost | no CR3 change, TLB stays warm | CR3 change, colder TLB/caches |
| Python | one GIL: one thread runs Python bytecode at a time (except free-threaded 3.13t/3.14t builds) | one interpreter each: true parallelism |
| Failure | a segfault kills all threads | isolated |
| In your stack | main thread, pin-memory thread, NCCL proxy/watchdog threads, CUDA driver threads, intra-op pools (OpenMP) | torchrun ranks, DataLoader workers, inference worker pools |
torchrun sets OMP_NUM_THREADS=1 per process by default when launching several ranks per node, precisely to avoid that oversubscription. Raise it deliberately (≈ cores per rank), not to the core count. ps -T -p PID or top -H -p PID lists the threads, and their names (pt_main_thread, pt_autograd_0, NCCL threads) tell you who is who.4Futexes: locks that only sometimes enter the kernel
What it is. A futex ("fast userspace mutex") is a 32-bit integer in shared memory plus a kernel wait queue keyed by its address. Uncontended lock/unlock is a single atomic instruction in user space, with no syscall. Only when a thread must wait does it call futex(FUTEX_WAIT) to sleep. The releasing thread calls futex(FUTEX_WAKE) only if someone is waiting. Every pthread mutex, condition variable and semaphore on Linux, and therefore Python's GIL and queue.Queue, is built on futexes.
strace -c of a training step, futex is usually #1 by time. That isn't the futex's cost. It's threads waiting: the main thread waiting for the next batch, the pin-memory thread waiting for a worker, the GIL being handed over. Heavy futex time tells you who's blocked, not what's slow. On your box, futex accounted for 58% of time spent in syscalls, summed across all threads.sys.getswitchinterval()): a thread holding the GIL can delay your launch thread by up to that long. nsys with --trace=osrt shows these as pthread_mutex_lock / pthread_cond_wait ranges.5fork vs spawn vs forkserver, and why fork after CUDA init breaks
What it is. Python's multiprocessing can start a child three ways:
| Start method | How | Startup | Gotchas |
|---|---|---|---|
fork | fork() the current process: the child is a COW clone of the parent, including all memory and handles, but only the calling thread | fast | other threads vanish mid-whatever; locks they held stay locked forever; CUDA/NCCL state is copied but meaningless |
spawn | start a fresh interpreter, then unpickle the target and its arguments | slow (re-imports torch: seconds) | dataset must be picklable and is copied per worker |
forkserver | at startup, spawn one clean, single-threaded server process; later children are forked from it | fast after the first | arguments must be picklable; the server must be started before CUDA init and before threads exist |
Why forking after CUDA init is unsafe. A CUDA context is far more than memory: driver threads, open file descriptors to /dev/nvidia*, GPU-side page tables keyed to the parent, pinned host mappings, and locks inside libcuda. fork() copies the parent's memory (COW) but only one thread, and none of the GPU-side state is duplicated. The child sees a context that claims to exist but whose helper threads are gone and whose locks may be held forever. CUDA doesn't support using it, and PyTorch refuses outright: RuntimeError: Cannot re-initialize CUDA in forked subprocess. More generally, POSIX only guarantees async-signal-safe calls in the child of a multithreaded parent until it calls exec. NCCL, OpenMP runtimes and logging libraries all have threads, so the same hazard applies.
fork on Linux for Python ≤ 3.13. That's fast, and it's the reason dataset memory is shared copy-on-write, and the reason it slowly un-shares as refcount writes copy pages (Module 2 §7). It works only because workers never touch CUDA. Python 3.14 changed the Linux default to forkserver. That's safer, but your Dataset now has to be picklable and is sent to each worker, so per-worker memory and startup behave differently. Check which interpreter your containers use.DataLoader(multiprocessing_context="fork" | "forkserver" | "spawn") pins the choice explicitly. torch.multiprocessing is a wrapper that shares tensors through shared memory (/dev/shm or fd passing; that's the accept4/recvmsg calls in the drill's strace) instead of pickling their bytes. torchrun starts ranks with a fresh exec, which is why ranks can each init CUDA safely.6The Linux scheduler: EEVDF
What it is. Each CPU has a runqueue of runnable tasks. Since Linux 6.6 the default policy for normal tasks (SCHED_OTHER) is EEVDF (Earliest Eligible Virtual Deadline First), which replaced CFS's picking logic. Your x86 box runs 6.8. Recent DGX OS kernels for the Spark are 6.17 and later, so both use EEVDF.
How it works, in the parts that matter:
- Fair share by weight. Each task accrues virtual runtime in proportion to its CPU use divided by its weight (from its nice value). Lag is how much CPU a task is owed versus its fair share. Tasks with lag ≥ 0 are eligible.
- Deadlines. Each task requests a slice (a base of 0.75 ms, scaled up on bigger machines to about 3 ms). Its virtual deadline is eligible time + slice / weight. The scheduler runs the eligible task with the earliest virtual deadline. Short requested slices give better latency, and that's the knob EEVDF adds over CFS (
sched_setattr'ssched_runtime). - Preemption. A running task is preempted at the end of its slice if someone else is eligible with an earlier deadline. A wakeup can preempt sooner. If 33 CPU-hungry tasks share 32 CPUs, some runnable task is always waiting a full slice.
- Load balancing moves tasks between CPUs' runqueues periodically and on idle. A migrated task lands on cold caches, possibly on an E-core, or on a CPU in another cluster or NUMA node.
- Other classes:
SCHED_FIFO/SCHED_RR(real-time: always beat normal tasks) andSCHED_IDLE. Since 6.12, sched_ext lets a BPF program implement the scheduling policy.
OMP_NUM_THREADS, num_workers), give the launch thread its own cores (affinity, §9), and reduce how often it needs to run (CUDA graphs, §8). Raising its priority (nice -n -5, or chrt for real-time) helps but can starve the helpers it depends on. Use it surgically.7Why a descheduled launch thread starves the GPU
The mechanism. The CPU enqueues kernels into a stream, and the GPU drains them. The queue is a buffer between the two. If each kernel runs longer than it takes to launch, the queue fills up, and a CPU hiccup is absorbed as long as the queue holds enough work to cover it. If kernels are shorter than the launch cost (small decode steps, small-batch inference, many elementwise ops in eager mode), the queue stays near empty. The GPU then runs at the CPU's pace, and every CPU stall becomes GPU idle time, one for one.
x.add_ launch vs ≈0.9 µs of GPU time per kernel. Stepping moves the cursor through time.cudaLaunchKernel row. OS runtime tracing (--trace=osrt) and CPU sampling show the thread blocked or descheduled. LLM decode with small batches is the classic case, which is why vLLM, SGLang and TensorRT-LLM capture decode steps as CUDA graphs. With large training kernels the queue absorbs most hiccups, until a collective's straggler effect turns one rank's hiccup into everyone's wait.8How CUDA graphs help
What it is. A CUDA graph records a sequence of kernel launches, copies and dependencies once (stream capture), then replays the whole thing with a single launch. The driver pre-validates and pre-builds the GPU work, so per-kernel CPU cost drops to near zero and the GPU walks the graph back to back.
Why it fixes jitter. The CPU's job becomes one launch per step instead of thousands. A deschedule in the middle of the step no longer matters: the whole step is already on the GPU. You still pay jitter between replays, so if steps are tiny, capture several.
Constraints. Graphs replay fixed memory addresses and shapes: inputs must be copied into static buffers, dynamic shapes need one graph per shape bucket, CPU-side logic and synchronizations inside the step aren't allowed, and allocations during capture come from a private memory pool.
torch.cuda.graph(g) / g.replay() directly; torch.compile(mode="reduce-overhead") captures automatically (CUDA graph trees); torch.cuda.make_graphed_callables for parts of a training step. NCCL collectives can be captured too. For the CPU side that remains, combine graphs with pinning the launch thread.9CPU affinity, isolcpus, IRQ affinity
CPU affinity restricts which CPUs a thread may run on (sched_setaffinity, taskset -c, numactl --cpunodebind, a cgroup cpuset). It keeps a thread on its GPU's NUMA node and away from noisy neighbours. Affinity only says where a thread may run. Other threads can still be scheduled onto the same CPUs unless you keep them out.
isolcpus= (kernel boot parameter) removes CPUs from the general scheduler: nothing runs there unless explicitly pinned. Usually combined with nohz_full= (stop the periodic timer tick when only one task runs) and rcu_nocbs= (move RCU callbacks off those CPUs). Isolated CPUs give a pinned thread a core that nobody else touches. Many clusters do the same thing more flexibly with cgroup cpusets.
IRQ affinity. Device interrupts run on some CPU, stealing it from whatever was there. /proc/interrupts shows counts per CPU, and /proc/irq/N/smp_affinity_list controls where IRQ N may land. irqbalance moves them around automatically. NVMe and many NIC drivers use managed per-queue IRQs spread over all CPUs. For latency-critical cores, steer NIC/GPU interrupts elsewhere: to a housekeeping core, on the device's own NUMA node.
--cpu-bind, --hint=nomultithread), Kubernetes' static CPU manager (exclusive cores for Guaranteed pods) and per-rank numactl wrappers are where you implement this. NCCL inherits the affinity it starts with. On the Spark, consider the X925 vs A725 split: the launch thread belongs on an X925 core (the drill's cpu_capacity list shows which CPUs those are).DDrill: syscalls, the COW "leak", launch jitter
bash drills/m3_cpu.sh syscalls # strace -f -c of a small training loop
bash drills/m3_cpu.sh cow # DataLoader workers: list-of-objects vs numpy storage
bash drills/m3_cpu.sh jitter # quiet / noisy / pinned launch threads, + CUDA graph
# lower-overhead syscall view (root):
sudo perf trace -s python3 drills/m3_train_step.py
# full timeline with OS runtime events:
nsys profile --trace=cuda,nvtx,osrt -o m3 python3 drills/m3_train_step.pyExpected output
x86 + RTX 4090 measured on your box
strace -c (top):
57.96% futex 9649 calls
28.74% poll 195
4.61% ioctl 3115
1.83% accept4 128 (fd passing)
cow --storage list (4 workers):
USS per worker 60 -> 439 MiB
sum USS 240 -> 1756 MiB
cow --storage numpy: 8 MiB, flat
jitter (2000 tiny kernels, GPU 1.77 ms):
quiet 4.74 ms (63% idle)
noise all CPUs 21.04 ms (92% idle)
pin 0, sib noisy 8.92 ms
pin 0, core quiet 5.35 ms
CUDA graph 1.67-1.77 ms alwaysDGX Spark expected, not yet measured
strace: same shape (futex/poll lead);
ioctl count similar (same driver model)
cow: same mechanism; if the kernel uses
64 KiB pages, each COW break copies 64 KiB,
so USS grows in bigger steps (and faster)
jitter: no SMT on Arm cores, so pin to
an X925 (cpu_capacity 1024);
A725 cores launch slower
CUDA graph still ~= GPU timecat /sys/devices/system/cpu/cpu0/cpu_capacity) and pass --pin explicitly if not.?Self-check
DataLoader workers' RSS looks stable, yet free memory falls steadily during the first epoch and the job gets OOM-killed at 6 workers. What's happening and what do you change? criterion #4
dataset[i] on a Python object increments its refcount (and GC touches headers), which is a write, so the kernel copies that page into the worker. Over an epoch each worker builds a private copy of every page it touched. RSS doesn't show it because those pages were already counted as resident. PSS and USS do (drill: 60 → 439 MiB USS per worker). Fixes: keep the data in numpy/Arrow/tensors or an mmap'd file (no per-item objects), reduce workers, or use per-worker lazy loading.Why does PyTorch refuse to use CUDA in a forked child, and what are the alternatives?
spawn or forkserver (started before CUDA init) for processes that need the GPU, or keep forked workers CPU-only, as DataLoader does.A small-batch decode loop gets 35% GPU utilization. nsys shows gaps between kernels, and the CPU thread shows cudaLaunchKernel calls of ~5 µs with occasional 2 ms holes. Explain, and give three fixes.
strace -c shows futex at 60% of syscall time. Is the futex implementation slow?
strace -c sums the time threads spend inside syscalls, and a futex wait is a thread sleeping until another thread releases a lock or signals a condition. A high futex share means threads are waiting for each other: the main thread for the next batch, the pin-memory thread for workers, threads for the GIL. Find out who waits on whom (nsys osrt, py-spy, perf trace). Don't optimize the futex.You pinned the launch thread with taskset -c 0, but jitter is still bad on a busy node. Name two likely reasons.
/proc/interrupts), and whether the Python GIL is held by a helper thread (that isn't a scheduler problem at all).Your team upgrades the training image from Python 3.12 to 3.14 and DataLoader startup gets slower and per-worker memory rises. Why?
fork to forkserver. Workers are no longer COW clones of the trainer: the Dataset object is pickled and sent to each worker, which rebuilds it in its own memory. That's slower startup and no sharing of the parent's pages. It's safer (no threads or locks inherited). Set multiprocessing_context explicitly, and make the dataset cheap to pickle (paths and mmap'd files, not in-memory lists).§Sources
- Linux kernel docs: EEVDF Scheduler (merged in 6.6), Extensible Scheduler Class (sched_ext) (6.12); LWN An EEVDF CPU scheduler for Linux;
kernel/sched/fair.cforsysctl_sched_base_slice(0.75 ms, scaled by CPU count). - Linux kernel docs: kernel parameters (
isolcpus,nohz_full,rcu_nocbs,irqaffinity), SMP IRQ affinity. - man pages: futex(2), fork(2) (only the calling thread is replicated; async-signal-safety), vdso(7), sched_setattr(2).
- Python docs: multiprocessing start methods (3.14: default changed to forkserver on POSIX); sys.setswitchinterval.
- PyTorch docs: DataLoader multi-process loading (COW and refcount note), multiprocessing best practices (CUDA in subprocesses), CUDA graphs, torchrun (OMP_NUM_THREADS default).
- NVIDIA, CUDA Programming Guide: CUDA graphs, device scheduling flags.
- Kernel releases: Linux 6.6 (EEVDF), 6.12 (sched_ext, EEVDF completion). Spark kernel versions (6.17-nvidia, 7.0-nvidia) are from NVIDIA developer-forum reports: check
uname -r. - Your measurements:
drills/m3_*.pyon i9-13900K (8P+16E, SMT on P-cores) + RTX 4090, torch 2.8.0, Python 3.12.