RoCE networking for AI data centers

Published 22 September 2026

RoCE lets one machine's network card place data directly into another machine's memory over Ethernet. That removes work from the CPUs and cuts latency: exactly what a cluster of expensive GPUs needs. It does not make Ethernet lossless by itself. The difficult part is building a fabric where congestion, pauses and packet loss do not leave thousands of accelerators waiting for one slow transfer.

The short answer

RoCE is RDMA over Converged Ethernet. RDMA (remote direct memory access) lets a capable NIC move data to or from registered memory without sending every byte through the remote machine's kernel and TCP/IP stack. RoCEv2 carries that RDMA transport inside UDP/IP, allowing it to cross a routed data center fabric and leave the layer 2 domain.

In an AI training cluster, RoCE normally runs on a dedicated back-end network between GPU servers. The ordinary front-end network still handles storage, checkpoints, management and user traffic. Meta describes exactly that split in its production clusters: front-end for ingestion and logging, back-end for the high-bandwidth collective traffic between accelerators.

Term	What it means
RDMA	Direct memory transfer performed by capable NICs, with far less CPU and kernel involvement
RoCE	The InfiniBand RDMA transport carried over Ethernet
RoCEv1	Layer 2 Ethernet only; it does not route between IP subnets
RoCEv2	RDMA inside UDP/IP; routable and the version used for large fabrics
PFC	Pauses one Ethernet priority before its receiving queue overflows
ECN	Marks packets as queues build, asking endpoints to reduce their rate
CNP	A congestion notification sent back to a RoCE sender after an ECN mark

Why AI training makes the network part of the computer

A GPU training job does not consist of independent machines doing useful work beside each other. Each accelerator calculates part of a result, then the group exchanges those partial results in collective operations such as all-reduce. The next step cannot begin until the exchange completes. One congested path therefore does not merely make one request slow: it can leave the rest of the cluster idle, waiting for the straggler.

The traffic is unusually hostile to an ordinary data center network. Meta's measurements describe it as low-entropy, bursty and dominated by elephant flows. There are relatively few flows for ECMP to spread, they start together, and each can run at line rate. A fabric that balances a thousand web requests beautifully can put three enormous training flows on one uplink and none on the next.

This is the reason for the dedicated back-end fabric. It can be built non-blocking, tuned for one traffic class, and changed without treating storage and management traffic as passengers in the experiment. It also lets the scheduler become part of the network design: placing communicating GPUs near each other can remove more congestion than another switch feature. Kubernetes GPU scheduling separates that placement problem from device allocation and gang admission.

What RDMA changes

With a conventional socket, an application hands data to the operating system, the kernel processes the transport and network stacks, and the receiving kernel places the result where the application can use it. Modern operating systems avoid many historical copies, but the CPU and kernel are still in the path.

RDMA applications register memory with an RDMA-capable NIC. The two endpoints establish queue pairs and exchange the information needed to access those buffers. The NICs then perform the transfer and notify the application when it completes. The operating system sets the operation up; it does not shepherd every packet through the data path.

Ordinary network path
application → kernel TCP/IP → NIC → network → NIC → kernel → application

RDMA data path
registered memory ↔ RDMA NIC → network → RDMA NIC ↔ registered memory

That is why the benefit is not merely bandwidth. The same link rate can consume less CPU and complete small transfers with lower latency. With GPU Direct RDMA, the useful buffer can be GPU memory, removing the staging area in host memory and another trip through the server.

RoCEv1 and RoCEv2 are different networks

RoCEv1 places the RDMA transport directly in an Ethernet frame using EtherType 0x8915. There is no IP header, so a router cannot forward it. That is workable for a small cluster inside one broadcast domain and a poor foundation for thousands of GPUs across a Clos fabric.

RoCEv2 adds IP and UDP. The destination port is 4791; the UDP source port carries entropy that switches can use for ECMP. The RDMA transport still provides the queue pairs, ordering and reliable-connection behaviour. UDP is the routable envelope, not a replacement for the RDMA transport and not evidence that the application has become an ordinary UDP service.

Packet-capture version. RoCEv1 looks like its own layer 2 protocol. RoCEv2 looks like IP and UDP port 4791, followed by an InfiniBand Base Transport Header. If the capture shows TCP, it is not RoCE data traffic.

Routing solves the scale problem and introduces a balancing problem. A conventional five-tuple hash sees one queue pair as one flow and pins it to one path. AI traffic has too few long-lived flows for statistical balance to rescue a bad hash. Implementations vary the UDP source port, use several queue pairs, or use more capable routing schemes to create enough path entropy. The principle is the same: eight equal-cost links are not useful if the traffic presents only one hash value.

"Lossless Ethernet" is a design target, not a mode

Ethernet switches are allowed to discard packets when a queue fills. A reliable RoCE connection can recover from loss, but recovery is expensive: a dropped packet stalls the queue pair and can trigger retransmission of later work. At AI-cluster scale, a low loss percentage can still mean a steady stream of delayed collectives. Operators therefore engineer the RoCE traffic class to run with very little or no congestion loss.

PFC stops the queue overflowing

Priority Flow Control, IEEE 802.1Qbb, lets a switch send a pause frame for a specific Ethernet priority. When the RoCE queue approaches its limit, the downstream switch pauses that priority on the upstream link long enough to drain the buffer. Other priorities can continue.

It prevents the drop and moves the problem upstream. If the congestion persists, the pause can propagate backwards through the fabric. Unrelated flows in the same priority wait behind it; a bad dependency can produce a pause storm or deadlock. PFC is a safety brake, not a congestion-control strategy.

ECN asks the sender to slow down

Explicit Congestion Notification is the earlier and gentler signal. As a queue crosses a configured threshold, the switch marks packets before it has to drop them. The receiver reflects that condition in a congestion notification packet, and the sender reduces its rate according to the configured congestion-control algorithm.

DCQCN is the best-known RoCEv2 example. Microsoft and Mellanox designed it to combine ECN feedback with endpoint rate control, specifically because PFC alone creates head-of-line blocking and fairness problems. The thresholds matter: mark too late and PFC fires before the sender reacts; mark too early and the fabric leaves bandwidth unused.

There is no universal set of numbers to copy. Port speed, cable delay, switch buffer, topology, NIC firmware and workload all change the safe thresholds. Meta's 400G training fabric is a useful warning against recipes: it found implementation problems while tuning DCQCN, disabled it for that deployment, and used PFC plus receiver-driven admission shaped by the collective library. That worked for Meta's traffic. It is evidence that the application and network must be co-designed, not evidence that everybody should turn congestion control off.

What the fabric has to get right

A separate back-end traffic class. RoCE, storage and management traffic should not accidentally share a queue whose pause state nobody intended.
Consistent priority mapping. DSCP, priority code point, switch queue and NIC priority must describe the same class at every hop. One default mapping in one server can turn a lossless path lossy.
Headroom before PFC. A switch must absorb the traffic already in flight after it sends a pause. Faster links and longer paths need more headroom.
ECN thresholds before the pause threshold. Rate control needs time to work while there is still buffer left.
A uniform MTU. RoCE is not the place to discover one server or one switch port remained at 1,500 bytes. Check the complete path, not the endpoint pair.
Enough path entropy. Watch link utilisation per member, not the aggregate. A fabric can show spare capacity while one ECMP path is saturated.
Topology-aware placement. Keep the heaviest collective traffic inside the least-oversubscribed part of the fabric whenever the scheduler has a choice.

The physical layer belongs on that list too. Meta identifies network cables among the recurring hardware failures seen while training at scale. A bad optic or marginal fibre can present as retransmissions and job instability without a clean link-down event. The fabric is not healthy merely because every interface is up.

How RoCE failures look

Symptom	First things to examine
One uplink full, peers quiet	ECMP hash, UDP source-port entropy, queue-pair placement
PFC counters climbing constantly	Persistent congestion, bad thresholds, slow receiver, oversubscription
CNP and ECN marks spike	Which queue and egress port is congested; whether sender rate actually falls
RoCE queue drops packets	Priority mapping, PFC state, buffer headroom, MTU and physical errors
Small jobs pass, large jobs collapse	Incast, collective pattern, pause propagation, hot ECMP paths
One host is always the straggler	NIC firmware, PCIe/GPU locality, cable and optic counters, NUMA placement

Counter names differ by switch and NIC, but the categories do not: PFC sent and received per priority, ECN marks, congestion notifications, queue occupancy or drops, link-level errors, retransmissions and per-path utilisation. Graph them together. A single counter without the event timeline usually tells you which mechanism reacted, not what caused it.

RoCE vs InfiniBand

Both expose RDMA to applications. The argument is about the network beneath it and the operational system around it, not whether one can move data directly between memories.

	RoCEv2	InfiniBand
Network	Ethernet with IP/UDP encapsulation	Purpose-built InfiniBand fabric
Loss handling	PFC, ECN and endpoint congestion control must be designed together	Credit-based link flow control is part of the architecture
Routing	Fits routed Clos fabrics and Ethernet operations	Uses InfiniBand fabric management and routing
Operational advantage	Reuses Ethernet hardware, tools and skills	The stack was designed as one high-performance system
Operational burden	More ways to build a valid-looking fabric with the wrong queue behaviour	A more specialised ecosystem with fewer transferable enterprise tools

A feature table cannot choose for you. Existing hardware, vendor diversity, staff experience, application libraries and the scale of the cluster all matter. The strongest public comparison comes from Meta, which built two 24,576-GPU H100 clusters (one RoCE, one NVIDIA Quantum2 InfiniBand) and reported equivalent performance after tuning. The largest Llama 3 model trained on the RoCE cluster. That proves RoCE can work at scale. The words after tuning are doing important work.

Does every AI data center need RoCE?

No. A server with accelerators connected inside one chassis does not need a data center RDMA fabric to move between them. Small inference services whose requests fit on one machine may care far more about ordinary front-end latency and storage. A training cluster can also use InfiniBand or an emerging transport built specifically for AI and HPC.

RoCE becomes relevant when the job crosses machines often enough that CPU overhead, transport latency and collective completion time are material. The question is not "does this building contain GPUs?" It is "does useful work stop while GPUs exchange data across the network?"

RoCE is not the last Ethernet answer. The Ultra Ethernet Consortium's 1.0 specification defines a new transport for AI and HPC with its own congestion, multipath and reliability work. That is a separate protocol, not a new name for RoCE, and deployments will coexist for years. For a network engineer walking into an existing AI cluster today, RoCEv2 remains the thing most likely to be waiting.

Questions people actually ask

Is RoCE TCP or UDP?

RoCEv2 is encapsulated in UDP/IP and normally uses destination port 4791. Its reliable transport behaviour comes from the RDMA transport carried inside that envelope, not from TCP. RoCEv1 uses neither TCP nor UDP; it is a layer 2 Ethernet protocol.

Does RoCE require lossless Ethernet?

RoCE reliable connections can recover from packet loss, but recovery is costly enough that traditional deployments engineer a near-lossless traffic class with PFC and congestion control. "Lossless" describes the intended behaviour of that class, not an inherent property switched on by RoCE.

What is the difference between RDMA and RoCE?

RDMA is the direct-memory-access capability and programming model. RoCE is one way to carry its transport over Ethernet. InfiniBand is another network that provides RDMA.

What is RoCE in AI?

It is usually the back-end scale-out network joining GPU servers for distributed training. It carries collective communication between accelerators while a separate front-end network handles storage, management, logging and ordinary application traffic.

Start one layer lower. RoCE makes more sense once the fabric beneath it does. Inside a data center network covers spine and leaf, ECMP, oversubscription and the front-end/back-end split from first principles.

The same AI infrastructure is also accelerating change on its front-end security path. X25519MLKEM768 covers the hybrid post-quantum TLS exchange now appearing between clients, edges and origins, and why its larger ClientHello matters to a network operator.

Sources

RoCE networks for distributed AI training at scale: Meta Engineering, August 2024: workload shape, front-end/back-end separation, routing and congestion-control experience
RDMA over Ethernet for Distributed AI Training at Meta Scale: Meta, SIGCOMM 2024
How Meta trains large language models at scale: the RoCE and InfiniBand 24K-GPU comparison
RDMA over Converged Ethernet: NVIDIA: RoCEv1 and RoCEv2 packet formats, routing and UDP port 4791
Congestion Control for Large-Scale RDMA Deployments: Microsoft and Mellanox, SIGCOMM 2015: PFC limitations and DCQCN
RFC 3168: Explicit Congestion Notification
Ultra Ethernet specification history: current and earlier public releases

Implementations and recommended thresholds change by NIC, switch, firmware and workload. The architecture here was checked against the sources above in September 2026; configuration values should come from the exact hardware and software release being deployed.