← Course index Module 1 of 6
Module 1 · OS fundamentals for AI systems

How the machine is built

Everything the kernel does later (page tables, scheduling, I/O) is a way of managing this hardware. This module covers what's physically in the box, how far apart things are in nanoseconds, and which wires your bytes cross on the way to the GPU.

≈35 min reading3 interactive diagramsDrill ≈20 minTeaches completion criteria #1 (pageable copies) and #3 (NUMA placement for NCCL)

0The map: your two machines

Your two boxes are the two basic designs. The x86 + RTX 4090 is the classic discrete design: the CPU and GPU each have their own memory, joined by a PCIe link that's slower than either memory. The DGX Spark is a unified design: one pool of LPDDR5x behind a coherent fabric. The GPU reaches it over NVLink-C2C, and there's no PCIe hop between CPU and GPU. Datacenter nodes (8 GPUs + NICs behind PCIe switches, NVLink between GPUs) mix both ideas, and GB200 is a big Spark: Grace and Blackwell joined by C2C.

Fig 0 · Discrete vs unified
x86 box + RTX 4090 (discrete) i9-13900K 8 P-cores + 16 E-cores private L1/L2, 36 MB L3 PCIe root complex DDR5, 2 channels ≈90 GB/s peak RTX 4090 AD102 GPU + copy engines 24 GB GDDR6X ≈1 TB/s PCIe 4 x16 32 GB/s NVMe 2.5 GbE The CPU↔GPU link is ~30× slower than either side's memory. DGX Spark · GB10 (unified) S-die (CPU) 10× X925 + 10× A725 2 clusters × 16 MB L3 16 MB SLC, mem ctrl, PCIe G-die (GPU) Blackwell 24 MB L2 no DRAM of its own NVLink-C2C coherent 128 GB LPDDR5x · one pool · 273 GB/s CPU and GPU read and write the same physical DRAM GPU's traffic goes via C2C → S-die NVMe ConnectX-72× PCIe Gen5 x4 200 Gb/s to a 2nd Spark
Spark numbers are from NVIDIA's DGX Spark hardware page and the GB10 Hot Chips 2025 talk (see Sources). Sizes not to scale.
Where this shows up in your stack On the 4090, tensor.cuda() crosses the slowest wire in the box. On Spark the same call copies DRAM to DRAM inside one memory pool. It can still be expensive, because every byte is read and written through the same 273 GB/s. Most of this course is about which of those two worlds a given API call lives in.

1Cores, cache lines, and coherence

What it is. A core runs one instruction stream (two with SMT/hyperthreading). It owns registers and small, fast private caches (L1, L2), and shares a larger L3 with its neighbours. Memory moves between DRAM and caches only in fixed blocks called cache lines: 64 bytes on both your x86 box and the Spark's Arm cores. Read one byte and the hardware fetches its whole 64-byte line.

How it works. Each line in a cache has a state. The common protocol is MESI: Modified, Exclusive, Shared, Invalid. Before a core writes a line, every other core's copy must be invalidated. If two cores keep writing different variables that happen to share a line, the line bounces between their caches on every write. That's false sharing, and it can make a "parallel" loop slower than a serial one.

Fig 1 · One core's view, and false sharing
Core 0 regs L1d 48 KB L2 2 MB Core 1 regs L1d 48 KB L2 2 MB shared L3 (36 MB on the 13900K) DRAM P-core sizes (Raptor Cove). Each step down: ≈3–10× more capacity, ≈3–5× more latency. False sharing: one 64-byte line, two writers ctr[0] ctr[1] … 6 more 8-byte slots in the same line … core 0 core 1 Each write invalidates the other core's copy, so the whole line ping-pongs between caches. ≈ tens of ns per bounce, worse across clusters or sockets. Fix: give each writer its own 64-byte line (padding).
Cache sizes are per-core for the i9-13900K P-cores (lscpu reports totals). Spark: each core has a private L2, each 10-core cluster shares a 16 MB L3, and a 16 MB system-level cache sits in front of DRAM.

Hybrid cores. Both of your CPUs mix big and small cores. The 13900K has 8 P-cores (CPUs 0–15 with hyperthreads) and 16 E-cores (CPUs 16–31). The Spark has 10 Cortex-X925 and 10 Cortex-A725 in two clusters. NVIDIA's Spark optimization guide specifically warns against concurrent writes to the same cache lines from different cores, "especially ones in different clusters."

Why it matters for AI workloads: your hot host-side code is single-threaded more often than you'd think: the Python launch loop, the NCCL proxy thread, a DataLoader collate. Which core it lands on (P vs E, near vs far cluster) and whether its data stays in cache can change step time measurably. The drill shows a 2× difference in one copy path just from picking an E-core.
Where this shows up in your stack Strided CPU tensor ops (t[:, ::2], a non-contiguous .permute()) waste most of every 64-byte line they fetch, which is why .contiguous() before CPU-side work pays off. Lock-free host-memory flags polled by both CPU and GPU (NCCL proxy FIFOs, cudaHostAlloc'd semaphores) are laid out one per cache line for exactly this reason.

2Latency and bandwidth, in orders of magnitude

What it is. Every layer, from register to network, trades distance for capacity. Two numbers describe each one: latency (how long until the first byte arrives) and bandwidth (bytes per second once data is flowing). You don't need exact values. You need the ratios, because they decide whether the fix is "batch it," "overlap it," or "keep it out of that layer."

How to read it. Each step down the hierarchy costs roughly 10× more latency. Hiding latency always works the same way: keep many requests in flight (out-of-order CPUs, GPU warps, NVMe queues, NCCL channels) so bandwidth, not latency, sets the speed.

Fig 2 · Latency & bandwidth explorer interactive
memory hierarchyOS / CUDA software costsdevices & network

Click any row

Each row has a note on where the number comes from and why it matters for your workloads.

Values are ballpark figures for current hardware (± 2× is normal). Treat them as orders of magnitude, and measure your own with the drills. Sources are at the bottom of the page.
Why it matters for AI workloads: a 4090 does ≈80 TFLOP/s of dense bf16 work, roughly 80,000 flops per nanosecond. During one 100 ns DRAM miss on the host, the GPU could have done eight million flops. Host-side latency is invisible only while the GPU has a deep queue of work. As soon as the queue drains, every microsecond of host latency becomes GPU idle time (Module 3 builds on this).
Where this shows up in your stack A cudaLaunchKernel costs a few µs of CPU time. That's why a step made of thousands of small kernels is launch-bound, and why CUDA graphs and torch.compile's kernel fusion help. Small NCCL all-reduces are latency-bound (µs per hop), and large ones are bandwidth-bound (GB/s per link). NCCL's algorithm choice (ring vs tree) and NCCL_ALGO tuning sit on that boundary.

3NUMA: not all memory is equally close

What it is. In a multi-socket server, each CPU socket has its own memory controllers and its own PCIe root complexes. Memory and devices attached to your socket are local, and those on the other socket are remote: you reach them across the socket interconnect (Intel UPI, AMD Infinity Fabric). That's Non-Uniform Memory Access. Linux exposes each socket (or sub-socket domain) as a NUMA node with its own CPUs, memory, and a numa_node attribute on every PCIe device.

How it works. By default Linux allocates a page on the node of the CPU that first touches it ("first-touch" policy). The scheduler tries to keep threads near their memory but doesn't guarantee it. A remote access adds roughly 50–100% latency, and the socket link has less bandwidth than local memory, shared by everyone crossing it.

Fig 3 · A typical 2-socket, 8-GPU training node
NUMA node 0 NUMA node 1 CPU socket 0cores, L3, root complex CPU socket 1cores, L3, root complex DRAM 0 DRAM 1 UPI / xGMI (socket link) PCIe switch A PCIe switch B PCIe switch C PCIe switch D G0 G1 NIC0 G2 G3 NIC1 G4 G5 NIC2 G6 G7 NIC3 G0↔NIC0: PIX (same switch). G0↔NIC1: NODE (via socket 0's root complex). G0↔NIC2: SYS (crosses UPI). NVLink/NVSwitch between the GPUs is omitted; it carries GPU↔GPU traffic but not GPU↔NIC traffic.
Labels are the nvidia-smi topo -m path types. Layouts differ by vendor, so always check your own node.

The GPU and NIC have NUMA nodes too. Each GPU and each NIC hangs off one socket's root complex. Inter-node NCCL traffic flows GPU memory → PCIe → NIC → wire. If the GPU and its NIC share a PCIe switch, that path never touches a CPU. If they sit on different sockets, every byte crosses the socket link or bounces through host memory, and it competes with everything else using that link.

Why it matters for AI workloads (completion criterion #3): NCCL reads the PCIe/NVLink topology at init and pairs each GPU with its closest NIC. It enables GPUDirect RDMA (NIC reads/writes GPU memory directly) only when the GPU-to-NIC distance is within NCCL_NET_GDR_LEVEL. When the distance is too far, it stages through host memory instead. The proxy thread that drives the NIC, and the pinned bounce buffers it uses, should also live on the GPU's NUMA node. Get placement wrong and you lose bandwidth twice: the transfer crosses the socket link, and the CPU-side helper runs on remote memory.
Where this shows up in your stack nvidia-smi topo -m prints each GPU's CPU and NUMA affinity and its distance to every NIC. NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,GRAPH logs which NIC each rank picked and whether GDR is on (look for GDRDMA in the channel lines). Launchers that bind each rank to its GPU's NUMA node (numactl --cpunodebind=N --membind=N, or Slurm's --cpu-bind) keep DataLoader workers and pinned buffers local. NCCL also honors the CPU affinity it inherits (NCCL_IGNORE_CPU_AFFINITY=0 by default), so a bad launcher binding becomes NCCL's binding.

Your machines: the i9-13900K and the Spark each expose one NUMA node, so true cross-NUMA effects only show up on multi-socket servers. The drill confirms this, then runs a near/far core comparison instead (P-core vs E-core on x86, X925 vs A725 on Spark). The point is the same: the CPU doing the copy matters.

4PCIe: lanes, generations, topology

What it is. PCIe is a packet-switched, point-to-point serial bus. A link is 1–16 lanes wide. Each generation doubles the per-lane rate. The CPU's root complex is the root of a tree. Switches fan one upstream link out to several devices, and devices under the same switch can talk peer-to-peer without going up to the CPU.

GenGT/s per lanex16, per directionTypical device
3.08≈16 GB/sV100-era GPUs, 100G NICs
4.016≈32 GB/sRTX 4090, A100, Gen4 NVMe (x4 ≈ 8 GB/s)
5.032≈63 GB/sH100/B200 host link, 400G NICs; Spark's CX-7 (as 2× x4)
6.064≈121 GB/semerging (800G NICs)

Real throughput is about 75–85% of these numbers after packet headers and flow control. Your 4090 measures ≈20 GB/s pinned H2D against a 32 GB/s peak (see the drill).

Links downtrain when idle. The drill on your 4090 reported pcie.link.gen.current = 1 at idle. That's power management (ASPM / dynamic link speed), not a fault. Check the link under load before blaming a riser or slot. A link that stays at x8 or Gen3 under load is a real problem, and it's a common one.

Reading nvidia-smi topo -m: each cell names the worst hop between two devices, nearest first:

CodePathImplication
NV## bonded NVLinksGPU↔GPU at NVLink speed, no PCIe involved
PIXat most one PCIe switchbest case for GPU↔NIC (GPUDirect RDMA)
PXBseveral PCIe switches, not through the CPUstill good for P2P and GDR
PHBthrough a CPU host bridgetraffic touches the root complex; GDR is often slower
NODEbetween host bridges in one NUMA nodecrosses on-die fabric
SYSacross the socket interconnectworst: NCCL usually won't use GDR here
Why it matters for AI workloads: PCIe is usually the narrowest pipe a tensor crosses. The H2D link sets a ceiling on how fast you can feed a discrete GPU from host memory, and on how fast you can offload optimizer state or KV cache to the CPU. In multi-node training, the GPU↔NIC PCIe path is part of every inter-node all-reduce.
Where this shows up in your stack ZeRO-Offload / FSDP CPU offload, KV-cache offload to host, and pin_memory'd DataLoader batches are all PCIe-bound on discrete GPUs. A rough rule: if per-step host↔device traffic divided by ≈20 GB/s (Gen4) or ≈50 GB/s (Gen5) is close to your compute time, the step is PCIe-bound, and overlap is the only fix. The NCCL_P2P_LEVEL and NCCL_NET_GDR_LEVEL settings take these same codes (PIX, PXB, PHB, SYS) as thresholds.

6DMA, copy engines, and why pinned memory matters

What it is. DMA (direct memory access) means a device reads or writes host memory itself, without a CPU loop doing the copying. NVMe controllers, NICs and the GPU's copy engines are all DMA engines. The CPU writes a descriptor ("copy N bytes from physical address A to B") into a queue, and the device does the rest.

How it works. A device can't use your process's virtual addresses on its own. It needs bus/physical addresses that will stay valid for the whole transfer. Ordinary malloc memory is pageable: the kernel can move a page, swap it out, or not have allocated it yet (Module 2 explains why). Pinned (page-locked) memory from cudaHostAlloc or cudaHostRegister is locked in place and pre-mapped for the device, so the driver can hand its addresses straight to the copy engine.

For pageable memory, the CUDA driver keeps its own small pinned staging buffers. A pageable H2D copy is really two copies: the CPU memcpy's chunks into staging, and the copy engine DMAs each chunk across PCIe, pipelined.

Fig 4 · Pageable vs pinned cudaMemcpyAsync interactive
CPU thread
copy engine
GPU SMs
user buffer
staging (pinned)
GPU memory
The staging-buffer pipeline is simplified: chunk sizes and the number of staging buffers are driver internals and aren't documented. The behaviour shown (when the call returns, and who moves the bytes) follows the CUDA runtime's documented sync rules, and you can see it in the drill.
Why it matters for AI workloads (completion criterion #1): a pageable copy is slower because (a) the bytes cross DRAM twice (user buffer → staging → PCIe), (b) throughput is capped by one CPU core's memcpy speed, and (c) the copy engine waits on the CPU between chunks. It can't be truly async because the driver can only DMA from pinned memory. Someone has to stage the pageable bytes, and that someone is your calling thread, so cudaMemcpyAsync doesn't return until the data has been staged. The CUDA docs say pageable async copies "might be synchronous with respect to host." Your 4090 shows it: the pageable call held the thread for ≈120% of the pinned transfer time, while the pinned call returned in ≈3 µs.
Where this shows up in your stack tensor.to("cuda", non_blocking=True) overlaps with compute only if the source is pinned. That's what DataLoader(pin_memory=True) buys you: a background thread copies each batch into pinned memory so the main thread's H2D copy becomes a real DMA. In nsys, a pageable copy shows up as Memcpy HtoD (Pageable) with the CPU stuck in the API call. Pinning has a cost: pinned pages can't be swapped or reclaimed, so pinning tens of GB starves the page cache (Module 2), and on Spark it comes out of the same 128 GB the GPU uses.

7Interrupts vs polling

What it is. When a device finishes work (a packet arrived, an NVMe read completed, a copy finished), it has two ways to tell the CPU. It can raise an interrupt: modern devices use MSI-X, a memory write that becomes an interrupt on a chosen core, with one vector per queue. Or the CPU can poll: spin on a completion flag in memory.

How it works. An interrupt stops whatever that core was running, enters the kernel's handler, and usually wakes a sleeping thread, which the scheduler then has to run. End to end that's microseconds, and more if the target core was busy or in a deep sleep state. Polling reacts in the time it takes to see a cache line change (≈100 ns), but it burns a whole core. High-rate devices compromise: interrupt coalescing (one interrupt per batch), NAPI in the network stack (switch to polling under load), and per-CPU NVMe queues (the interrupt lands on the core that asked).

Why it matters for AI workloads: the latency-critical paths in your stack poll. NCCL kernels spin on flags in GPU or host memory, the NCCL proxy thread busy-polls the NIC's completion queue, and RDMA (Module 4) is built on user-space polling instead of interrupts. The trade-off is that those spinning threads need a core to themselves. If the scheduler shares that core with a DataLoader worker, latency jumps (Module 3).
Where this shows up in your stack cudaStreamSynchronize spin-waits by default for a short time and then may yield. cudaSetDeviceFlags(cudaDeviceScheduleBlockingSync) makes the CPU sleep until the GPU interrupts instead: less CPU burn, more wake-up latency. /proc/interrupts shows which cores take the NIC's and GPU's interrupts, and IRQ affinity is set in /proc/irq/*/smp_affinity (Module 3).

8The end-to-end data path

Here is everything above combined: a checkpoint shard or a training batch travelling from disk to a GPU and then to another GPU or node. At each hop, notice who moves the bytes (a CPU core or a DMA engine), and how many times they're copied through DRAM.

Fig 5 · Disk → GPU → network, step by step interactive
Keyboard: focus the figure, then use ← / →. "DRAM passes" counts how many times the payload is written into host DRAM.
Why it matters for AI workloads: each extra copy through host DRAM costs memory bandwidth and usually a CPU core. GPUDirect Storage and GPUDirect RDMA (Module 4) exist to cut out exactly those copies. Faster checkpoint loading, input pipelines and inter-node communication all come down to removing hops in this diagram, or overlapping the ones you can't remove.
Where this shows up in your stack torch.load of a pickled checkpoint takes the full path: read into page cache, copy to Python-owned buffers, then H2D. safetensors with mmap skips the user-buffer copy (Module 2). On the network side, NCCL's NET/IB : GPU Direct RDMA enabled log line means the "bounce through host" step below has been removed.

DDrill: inspect and measure your machines

Two files in drills/. Run them on both boxes and keep the output, because Module 2 compares against it.

# one-time (both boxes): topology tools
sudo apt install hwloc numactl

# A. topology: CPUs, caches, NUMA, PCIe tree, GPU/NIC affinity
bash drills/m1_machine.sh topo

# B. pinned vs pageable bandwidth + "is it really async?"
bash drills/m1_machine.sh bw          # = python3 drills/m1_h2d_bandwidth.py

# C. CPU placement: cross-NUMA if you have >1 node, else big core vs small core
bash drills/m1_machine.sh place

# optional: a picture of the topology
lstopo --whole-io topo.png

What to look for

  1. Topology. Find the cache-line size, core types, NUMA node count, and the numa_node / local_cpus of the GPU and NIC. In nvidia-smi topo -m, read the CPU Affinity column.
  2. Bandwidth. Pinned should beat pageable, and the pinned call's "CPU blocked" time should be microseconds while the pageable one is about the whole transfer. That's criterion #1 on your own hardware.
  3. Placement. Pageable throughput follows the CPU core, and pinned throughput doesn't. On a multi-socket server with numactl, cross-node memory also cuts pinned bandwidth, because the DMA has to cross the socket link.

Expected output

x86 + RTX 4090 measured on your box

page size: 4096   cache line: 64
P-cores = 0-15   E-cores = 16-31
NUMA: node0 only
GPU0 CPU Affinity 0-31, NUMA 0
PCIe gen current 1 / max 4, x16  (idle)

  size | H2D pg  H2D pin | D2H pg  D2H pin
  256M |  16.0    20.2   |  14.9    19.2
CPU blocked: pageable 16.5 ms (124%)
             pinned   0.002 ms (0%)

place: P-core  pageable 16.4  pinned 20.3
       E-core  pageable  8.0  pinned 20.2

Pinned ≈20 GB/s is ~63% of the Gen4 x16 peak. The usual range is 20–26 GB/s depending on platform. The E-core halves pageable throughput and leaves pinned untouched, because only the pageable path runs a CPU memcpy.

DGX Spark expected, not yet measured

arch: aarch64   cache line: 64
capacity: 10 CPUs at 1024 (X925),
          10 lower (A725)
NUMA: expect a single node
GPU: integrated: True
  nvidia-smi PCIe gen/width: N/A
NICs: 4 CX-7 netdevs, e.g.
  enp1s0f0np0 + enP2p1s0f0np0 = port 0
  enp1s0f1np1 + enP2p1s0f1np1 = port 1

H2D/D2H: DRAM→DRAM inside one pool;
pinned should still beat pageable, and
pageable should still block the CPU

There's no PCIe ceiling here. A DRAM→DRAM copy reads and writes the same 273 GB/s, so expect pinned "H2D" well above the 4090's 20 GB/s but under ~135 GB/s.

I haven't found a published Spark cudaMemcpy bandwidth, a confirmed NUMA node count, or a statement of whether the driver skips staging for pageable memory on ATS-capable systems. Run the drill and compare. If pageable ≈ pinned and "CPU blocked" is small, the driver is using the direct path. The cpu_capacity values and netdev names come from community reports.
Where this shows up in your stack Keep the topology output from a multi-GPU node handy. When an NCCL job is slow, the first question is "did each rank get its local NIC, with GDR on?", and you answer it from topo -m plus the NCCL_DEBUG=INFO init log.

?Self-check

Answer out loud first, then open the answer.

You wrote x.to("cuda", non_blocking=True), but nsys shows the copy and the next kernel running back to back with no overlap, and the CPU stuck in cudaMemcpyAsync. What's happening, and what's the fix? criterion #1
x is in pageable memory. The copy engine can only DMA from pinned memory with stable physical addresses, so the driver has your calling thread memcpy the data, chunk by chunk, into its own pinned staging buffers. The call returns only after the last chunk is staged, so it's effectively synchronous, and bandwidth is capped by one core's memcpy plus a second pass through DRAM. Fix: put the source in pinned memory (pin_memory=True in the DataLoader, x.pin_memory() once for reused buffers, or allocate with torch.empty(..., pin_memory=True)). Then the call only enqueues a DMA descriptor and returns in microseconds.
Why can't the GPU's copy engine just DMA straight from a malloc'd buffer?
A DMA engine works on bus/physical addresses that must stay valid for the whole transfer. Pageable memory doesn't guarantee that: its pages might not be allocated yet, the kernel can migrate or swap them at any time, and the IOMMU mapping for the device might not exist. Pinning locks the pages in place and maps them for the device ahead of time. (Module 2 covers the page-table side of this.)
On an 8-GPU, 2-socket node, NCCL all-reduce across nodes is 40% slower than on an identical node. topo -m shows GPU4↔mlx5_2 as SYS on the slow node and PIX on the good one. Explain. criterion #3
On the slow node, GPU4's traffic to its NIC crosses the socket interconnect, so it isn't the NIC on the same PCIe switch. Either the NIC is cabled/slotted differently, or the rank-to-NIC mapping was forced (e.g. by NCCL_IB_HCA) to a far NIC. At SYS distance NCCL normally disables GPUDirect RDMA and stages through host memory instead. Every byte then crosses UPI, which is shared and slower than local PCIe, and bounces through DRAM. If the proxy thread and bounce buffers also sit on the far socket, you pay for remote memory too. Fixes: correct the hardware or NIC mapping, let NCCL auto-select, and bind each rank's CPU and memory to its GPU's NUMA node.
Two threads each increment their own counter, ctr[0] and ctr[1], in a shared int64 array. Adding the second thread makes it slower. Why?
False sharing. Both counters sit in the same 64-byte cache line, so every write by one core invalidates the other core's copy, and the line bounces between their caches (tens of ns per bounce, more across clusters or sockets). Pad each counter to its own 64-byte line, or keep per-thread counters in registers or locals and combine them at the end.
A 70B-parameter bf16 checkpoint is 140 GB. Roughly how long, at best, to get it from one Gen4 NVMe into GPUs, versus from a warm page cache over PCIe Gen5 x16? What does that suggest?
NVMe at ~7 GB/s: ~20 s just to read the disk, and that's the bottleneck. From page cache at ~50 GB/s achieved per Gen5 x16 link: ~3 s if one GPU had to take all of it, less when 8 GPUs load their own shards in parallel over separate links. So cold loads are disk-bound (use multiple drives, or keep checkpoints in page cache or tmpfs on the node), warm loads are PCIe-bound, and extra CPU copies (pickle → Python buffers) push both toward CPU-bound. Module 4 covers dropping caches to measure each case.
On DGX Spark, what does "H2D bandwidth" actually measure, and what else is competing for the same resource?
Both source and destination are in the same LPDDR5x pool, so an "H2D" copy reads and writes the same DRAM, and the 273 GB/s is shared between the read and the write. CPU threads (data prep, tokenization, the Python process) and the GPU's own kernels draw from that same 273 GB/s. On Spark the question isn't PCIe speed. It's whether a copy is needed at all, since the GPU can access system memory coherently over C2C (Module 2: ATS, HMM).
Why does NCCL's proxy thread busy-poll the NIC's completion queue instead of waiting for an interrupt, and what does that imply for how you place threads?
An interrupt plus a thread wake-up costs microseconds, and more if the core was asleep or busy. Spinning on a completion entry in memory reacts in about the time it takes a cache line to change. For latency-bound collectives, those microseconds are a big fraction of the operation. The price is a dedicated core: if the scheduler puts a DataLoader worker or another rank on that core, the proxy gets descheduled and latency spikes. Keep proxy and launch threads on reserved, NUMA-local cores (Module 3).
Progress is saved in this browser and shown on the index page.

§Sources

  • NVIDIA, DGX Spark Hardware Overview: 20 Arm cores (10× X925 + 10× A725), 128 GB LPDDR5x, 256-bit, 273 GB/s, 6,144 CUDA cores, two copy engines, ConnectX-7, 140 W SoC TDP.
  • NVIDIA / MediaTek, GB10 at Hot Chips 2025: S-die + G-die, 2×10-core clusters with 16 MB L3 each, 16 MB system-level cache, ~600 GB/s aggregate C2C, 24 MB GPU L2, hardware CPU↔GPU coherence, ATS through GMMU + SMMU, PCIe Gen5 x8 to the CX-7 NIC; GH200/GB200 C2C "up to 900 GB/s"; NVLink 5 at 1,800 GB/s.
  • NVIDIA, DGX Spark Porting Guide: CUDA: GPUDirect RDMA not supported; use cudaHostAlloc + ibv_reg_mr.
  • NVIDIA, DGX Spark Optimization: UMA, cudaMemGetInfo under-reporting, two clusters of P/E cores, avoiding cross-cluster cache-line sharing.
  • ServeTheHome, The NVIDIA GB10 ConnectX-7 200GbE networking is really different, and the NVIDIA developer forum thread ConnectX-7 NIC in DGX Spark: CX-7 runs as two PCIe Gen5 x4 links, each netdev pair maps to one port, and ~100 Gb/s per x4 link (community measurements).
  • NVIDIA, CUDA Runtime API: API synchronization behavior: pageable vs pinned memcpy semantics and staging buffers.
  • NVIDIA, NCCL environment variables: NCCL_P2P_LEVEL / NCCL_NET_GDR_LEVEL path types, NCCL_NET_GDR_READ, NCCL_IGNORE_CPU_AFFINITY.
  • Intel, i9-13900K (Raptor Lake) specs: 8P+16E cores, 36 MB L3, 2 MB L2 per P-core, PCIe 5.0/4.0. Cache and core layout confirmed with lscpu on your box.
  • PCI-SIG generation rates (8/16/32/64 GT/s per lane). x16 per-direction figures are computed after line encoding.
  • Latency ballparks: Colin Scott's interactive "Latency numbers every programmer should know" and Chips and Cheese microbenchmarks for recent cores and GPUs. They're rounded on purpose, so treat them as ±2×.
  • Your measurements: drills/m1_h2d_bandwidth.py on the i9-13900K + RTX 4090, driver 580.178, torch 2.8.0+cu128.