Post

NCCL Wire Protocols

How NCCL's wire protocols trade payload bytes against synchronization cost

Posted Updated
By Kapil Sharma
6 min read
NCCL Wire Protocols

NOTE: Mostly written by human in author’s voice with some Claude assistence on research and diagrams

I had not written a blog post in a while. Things have been busy. More recently, I have been working on getting a deeper technical understanding of NCCL. So, I thought I might share a nugget that I was looking into a couple of weeks ago.

I was reading a recent paper from NVIDIA on speed of light latency on GPU collectives. The main premise of the paper is focussed on using different wire protocols to establish low-latency NCCL collective performance baselines. I went down a rabbit hole of looking at NCCL kernel selection for collectives, especially for ones used with symmetric memory.

NCCL picks two things independently: an algorithm (Ring, Tree, CollNet, NVLS) and a protocol (Simple, LL, LL128).

At a high level,

  • algorithm decides who talks to whom
  • the protocol decides what goes out on the wire, i.e. the wire protocol

In this post, we will focus on the latter since choosing the right low-latency wire protocol is becoming increasingly critical for today’s inference and post-training workloads.

Protocols

Below we cover the 3 main protocols that are currently available in NCCL.

Simple

Data is written in large chunks such that 100% of the wire bytes are payload. Ordering between the payload stores and the synchronization signal is enforced by an explicit barrier using __threadfence_system() and a separate tail/flag store. This fence is expensive. It costs on the order of µs on some paths, which is irrelevant for medium to larger payloads but dominates smaller messages let’s say 4 KB.

NCCL Simple protocolA full send-buffer slot of pure payload, followed by a thread fence, then a separate tail flag store, which the receiver spins on before reading the payload. Send buffer slot (NCCL_BUFFSIZE / steps) Payload only — no inline metadata 100% of wire bytes are data __threadfence_system() tail / flag store Receiver spins on tail, then reads the payload

LL (Low Latency)

LL drops the fence entirely by exploiting naturally-aligned 8-byte atomic stores and replacing barrier fence with inlined synchronization flags. In essence, since bandwidth barely matters at small message sizes, it is worth trading wire efficiency for lower latency.

In LL, every 8-byte line is 4B of data plus a 4B flag, so the receiver polls the flag inline. Hence, there are no explicit barriers. Synchronization signal is embedded in the wire protocol. NCCL uses it below roughly 8 KB.

NCCL LL protocolRepeated 8-byte lines, each holding 4Bytes of data and a 4-byte flag, so the receiver polls a flag inline with the data it guards. Half of all bytes are flags. 8-byte line, repeated data4B flag4B data4B flag4B data4B flag4B data4B flag4B 50% payload · 50% flags · no fence, no separate signal Receiver polls the flag word data in that same 8B line is valid — 8B stores never tear

LL128

LL128 uses the same idea as LL but expands it to a coarser granularity. It uses a 128-byte line that carries 15 × 8B payload (120B) plus one 8B signal flag word. The main tradeoff relative to LL is bandwidth: for medium-sized messages, LL128’s overhead is only 6.25% of the wire, versus 50% for LL, so more of the wire carries actual payload.

One caveat to note is that 128B stores are not guaranteed to be atomic on all transports. For example, NVLink guarantees atomicity for 128B stores but PCIe does not.

However, LL128 comes with stricter hardware requirements. It depends on atomic 128-byte writes, which must not be split or reordered by the memory system or interconnect. In systems where such operations are not guaranteed, due to PCIe limitations or other architectural constraints, NCCL disables LL128 to avoid data corruption. Protocol selection is thus influenced not only by message size, but also by system-level capabilities.

Source: Demystifying NCCL

NCCL LL128 protocolA 128-byte line of sixteen 8-byte words: fifteen data words followed by one flag word, giving 6.25 percent overhead, valid only where 128-byte stores are atomic. 128-byte line — LL128_LINEELEMS = 16, DATAELEMS = 15 15 × 8B data words (120 B) flag 8B 1/16 overhead = 6.25% · ~93.75% of peak bandwidth Receiver polls the 16th word the preceding 120 B in that line are valid — still no fence Requires atomic 128B stores guaranteed on NVLink, not on PCIe — hence the topology gate

Summary

 Payload efficiencySync costTypical range
Simple100%__threadfence_system(), µs-scalelarge messages
LL≤ 50%none — inline 8B flagbelow ~8 KB
LL128~93.75%none — inline 8B flag per 128Bmid-size, NVLink only
  • Latency: LL < LL128 < Simple
  • Bandwidth: Simple > LL128 » LL

NCCL’s tuning model picks per (algorithm, protocol, message size, topology) from its internal latency/bandwidth tables.

Fenceless synchronization

LL and LL128 are the same idea. So let’s describe the mechanism in more detail.

Over a GPU-GPU connection, receiver needs to know that a producer’s data stores have landed before it reads them. Simplest answer to this is to order the stores with a fence, then publish a flag, and have the consumer spin on that flag. However, this barrier can account for a significant portion of latency when used with small messages.

Instead of using a barrier, the LL family makes the flag and the data part of the same store. If the hardware guarantees that a store of size N (8 in LL and 128 in LL128) is atomic, then a flag placed within those bytes can be observed atomically, with a guarantee against race conditions.

  • 8-byte stores are architecturally guaranteed to be atomic for naturally-aligned accesses
  • 128-byte stores are atomic only on NVLink, i.e. LL128

How are flags propagated?

The store is remote i.e. it crosses over fabric into the peer’s receive buffer. The poll is always local i.e. each rank spins on its own buffer.

The flag value that is exchanged is a step counter or epoch, and the receiver compares it against an expected value for that epoch, rather than using a plain boolean. A boolean flag would require zeroing the buffer between steps, which would reintroduce the ordering problem the protocol tries to eliminate. The diagram below shows this mechanism for a 4-rank NVLink connection.

LL128 flag polling across four NVLink-connected GPUsFour GPUs at the corners of an NVLink fabric, joined by matching right-angled connectors. Each holds a receive buffer of four data words plus a flag word. One GPU stores a 128-byte line directly into a peer's receive buffer over NVLink; the peer spins on its own local flag word and treats the preceding data as valid once the flag matches. NVLink peer-mapped stores GPU 0 stores into peer buffer GPU 1 ↻ flag hit → data valid GPU 2 ↻ spins on flag word GPU 3 ↻ spins on flag word remote store, one 128B line the flag rides in the same line as the data it guards — the poll never leaves local memory

Further Reading and Sources

Here are some papers and sources I enjoyed reading when I was researching this topic.

Blog
NCCL GPU Distributed Performance
This post is licensed under CC BY 4.0 by the author.