Writing

Inside a data center network: how it is actually built

Published 22 September 2026

A campus network and a data center fabric are both Ethernet, and almost nothing else about them is the same. The reason is a single change in where the traffic goes: it stopped travelling out to the internet and started travelling between servers in the same building. Spine and leaf, ECMP and VXLAN all follow from that change.

The traffic turned sideways

A campus network assumes a person at the edge wants something from a server, or the internet, at the middle. Traffic flows up through access switches to a distribution layer to a core, and back down. That is north-south traffic, and the three-tier design fits it almost perfectly.

Three things broke the assumption. Virtualisation turned one application on one box into a dozen services calling each other. Distributed storage made writing a block mean writing it to three machines across three racks. And AI training has thousands of accelerators exchanging results constantly. None of that traffic leaves the building.

Cisco's Global Cloud Index put the split at 76% east-west, 17% north-south and 7% between data centers, and that measurement is from 2014, before any of the current build-out. Cisco stopped publishing the index in 2019 and nothing as broad has replaced it publicly, so treat the figure as a historical marker. The direction has not reversed.

A note on spelling, because it sends people searching in circles: spine and leaf, spine-leaf and spine leaf architecture are the same design; top of rack switch, top-of-rack switch and ToR are the same box; east west traffic and east-west traffic are the same traffic. The hyphenated forms are used below.

Why three-tier had to go

The first problem is the path. Two servers on different access switches reach each other by going up to distribution, possibly up to core, and back down, and how many hops that takes depends on where the two machines happen to sit. Latency that varies with physical placement is tolerable when placement is fixed, and a real problem when a scheduler moves workloads every few minutes.

The second is worse: spanning tree. It exists for a good reason: an Ethernet frame has no hop count, so a loop in a bridged network is unsurvivable, and it solves that by reducing the topology to a tree. A tree has exactly one path between any two points, so every redundant link you bought sits blocking and carries nothing until something fails. With dual uplinks everywhere that can mean running half the capacity you paid for. A decade of workarounds (MLAG, port channels, stacking) narrowed the waste without removing the constraint.

Third and quieter: failure domain. A layer 2 domain spanning a distribution layer means one broadcast storm takes out everything inside it. Campus networks live with that. A hall full of racks cannot.

Spine and leaf

The replacement is a Clos network, named for Charles Clos, who published the mathematics for telephone switching in the Bell System Technical Journal in 1953. Al-Fares, Loukissas and Vahdat brought it into networking at SIGCOMM in 2008, showing a fat-tree of cheap identical switches could beat a few large expensive ones. The rules state completely:

• Every leaf connects to every spine.
• No leaf connects to another leaf.
• No spine connects to another spine.
• Servers connect only to leaves. Never to a spine.

The consequence is the whole point. Any server reaching any other server goes leaf, spine, leaf: two hops, always, whichever racks the two machines are in. Not "short". Predictable. The scheduler can place a workload anywhere and the network behaves identically, which is what makes the machines interchangeable.

Oversubscription, concretely

A leaf faces servers downward and spines upward, and the two numbers are rarely equal. Take a common configuration: 48 ports of 25GbE facing servers, 8 ports of 100GbE facing spines. That is 1,200 Gb/s down against 800 Gb/s up, an oversubscription ratio of 1.5:1. If every server transmitted at line rate at once, the uplinks would be a third short.

Choosing the ratio is the first real data center network design decision, and it is a purchasing decision as much as a technical one. Non-blocking (1:1) means buying uplink capacity most workloads will never use. Erring the other way gives you a fabric that is fine on average and congested exactly when everything is busy at once.

How it scales

Adding a spine gives every leaf more uplink bandwidth at once, lowering oversubscription across the whole fabric. Adding a leaf gives more server ports. The two dimensions are independent, which is the property three-tier never had. The ceiling is the spine's port count, since every leaf needs a port on every spine: a 64-port spine supports 64 leaves, and 64 leaves at 48 server ports each is a little over 3,000 servers in one fabric. Past that you build a second fabric and join them with a super-spine layer. And where three-tier growth meant a bigger core chassis and a forklift, this means another identical 1U box from whoever was cheapest that quarter.

ECMP, and why every link forwards

This is the single biggest practical difference between a fabric and a campus network. Every link between a leaf and a spine is routed: layer 3, its own tiny subnet, usually a /31 per point-to-point link. There is no bridged path between switches, so there is no loop for spanning tree to break, so spanning tree has nothing to do. It is still enabled facing servers, as a guard against somebody patching two ports together, and that is all.

Instead, the routing protocol installs one next hop per spine for the same destination:

leaf01# show ip route 10.20.3.0/24
10.20.3.0/24, ubest/mbest: 4/0
    *via 10.0.0.1,  Ethernet1/49, [20/0], 4d02h, bgp-65101
    *via 10.0.0.5,  Ethernet1/50, [20/0], 4d02h, bgp-65101
    *via 10.0.0.9,  Ethernet1/51, [20/0], 4d02h, bgp-65101
    *via 10.0.0.13, Ethernet1/52, [20/0], 4d02h, bgp-65101

Four equal-cost paths, one per spine, all of them live. Equal-cost multi-path forwarding hashes each packet's five-tuple: source and destination IP, protocol, source and destination port, and picks an interface from the result. Because those fields are constant for the life of a connection, every packet in a flow takes the same path and nothing arrives out of order, while different flows land on different spines.

The honest caveat: ECMP balances flows, not bytes. A single large, long-lived transfer is pinned to one link for its whole life, and eight flows across four links can easily land three, one, three, one: RFC 2992 set out the maths in 2000. With thousands of small flows the imbalance is invisible; with a handful of enormous ones it dominates, which is exactly the AI case below.

The protocol underneath is usually BGP, which surprises people arriving from enterprise networks where BGP is "the internet one". RFC 7938, Use of BGP for Routing in Large-Scale Data Centers (August 2016), makes the case for eBGP as the only protocol in the fabric, a private AS number per leaf, the AS path doing loop prevention for free.

The top of rack switch

There were two ways to cable a hall. End-of-row put one large chassis at the end of each row and home-ran every server to it: hundreds of copper runs of ten to twenty metres. Top-of-rack puts one or two small switches in each rack and cables the servers inside it.

Top-of-rack won on physics and money. At 25G and above, passive direct-attach copper (DAC) is good for roughly three metres, active copper for a little more. Inside a rack that is plenty, and a DAC costs a fraction of a pair of optical transceivers. Across a row you need optics on both ends of every server link, at which point the transceivers cost more than the switch.

So what a rack actually looks like: 42 to 48U of space, one or two switches at the top, a bundle of short DACs down one side to the servers, four to eight fibre uplinks leaving for the spine layer. The cables inside a rack are many and short; the cables leaving it are few and fixed. That asymmetry is the whole argument, and it is why a hall can be cabled to a repeatable pattern by a crew that never reads a bespoke diagram. Where there are two switches, the pair was traditionally bonded with MLAG; increasingly each server NIC sits in its own layer 3 domain and the host handles redundancy.

Overlays: VXLAN and EVPN

The fabric is routed end to end, which is what makes ECMP work. But plenty of software still expects two machines to share a layer 2 segment: clustering heartbeats, virtual machine mobility, appliances written in 2009 that are not going to change, and those machines sit in different racks, behind different leaves, separated by IP routing.

VXLAN (RFC 7348, 2014) resolves the contradiction by wrapping the original Ethernet frame in UDP and carrying it across the routed fabric as ordinary IP traffic. The device doing the wrapping is a VTEP, normally the leaf switch. The segment is identified by a VNI, a 24-bit number: 16,777,216 of them against 4,094 usable VLAN IDs, which is the other reason overlays exist anywhere with multiple tenants. EVPN (RFC 7432, 2015, adapted to VXLAN in RFC 8365) is the control plane on top: instead of VTEPs learning which MAC lives where by flooding, BGP advertises "this MAC is behind this VTEP", turning flood-and-learn into a routing protocol with a table you can query.

The honest note, and it matters if you are weighing up the work: the overlay is where the operational complexity now lives. The physical fabric is simple by construction and stays simple. The overlay is where a route target is mistyped and two tenants can see each other, where a MAC lands in the wrong VRF, and where a capture on a leaf shows you UDP port 4789 and nothing you can read. The design got simpler and the debugging got harder.

One concrete trap. VXLAN adds 50 bytes of encapsulation for IPv4, so a fabric left at a 1,500-byte underlay MTU will pass pings and fail real traffic in ways that look like an application fault. Fabrics are built at 9,000 bytes, or at minimum with headroom for the encapsulation. It is one of the commonest first-week mistakes.

What AI training changed

In synchronous training, every accelerator computes a partial result and then all of them exchange it in a collective operation (an all-reduce) and nothing proceeds until the exchange completes. That gives two awkward properties:

• Few flows, enormous, simultaneous. The precise worst case for flow-hashed ECMP, which needs many small flows to average out.
• The job runs at the speed of the slowest link. In a normal fabric one congested link makes one service slow. Here it leaves thousands of very expensive accelerators idle waiting for a straggler.

That is why AI clusters are built non-blocking, and why they usually have two networks: a dedicated back-end fabric carrying only accelerator-to-accelerator traffic, and a conventional front-end fabric for storage and management. Meta's 2024 clusters allocated 400 Gb/s per GPU on the back-end.

The other change is the demand for lossless behaviour. RDMA lets one machine's network card write directly into another's memory without involving either CPU. RoCE (RDMA over Converged Ethernet) carries it over an Ethernet fabric. RDMA assumes the network does not drop packets. Ethernet drops packets. Two mechanisms bridge the gap: PFC (IEEE 802.1Qbb), where a switch short of buffer sends a pause upstream for one traffic class, and ECN (RFC 3168), where a switch marks packets as its queues fill so the sender slows before anything is lost. ECN is the graceful mechanism and PFC is the blunt one: a pause propagates backwards hop by hop, so it can spread congestion to senders that were not causing it, and in pathological topologies deadlock. Tuning ECN thresholds so PFC almost never fires is a specialised job people do full-time. The dedicated guide to RoCE networking for AI data centers follows that packet path and the failure modes in detail.

InfiniBand or Ethernet: genuinely unsettled

This argument is live, and anyone presenting it as settled is selling something. InfiniBand is lossless by design rather than by configuration, uses credit-based flow control instead of retrofitted pause, and carries two decades of HPC operational experience, but is effectively a single-vendor ecosystem. Ethernet is cheaper per port, multi-vendor, and supported by an enormous pool of people who already know it; the Ultra Ethernet Consortium published its 1.0 specification in June 2025, aimed precisely at the transport and congestion behaviour where Ethernet had been losing.

The most useful data point: Meta built two 24,576-GPU H100 clusters in 2024, one on RoCE and one on NVIDIA Quantum2 InfiniBand, ran both for GenAI training, and reported comparable performance: Llama 3 was trained on the RoCE cluster. Building both and finding them equivalent says more than any benchmark.

Counting deployments does not settle it either. On the June 2026 TOP500 list, the InfiniBand Trade Association counts 293 systems using InfiniBand against 83 using Ethernet with RoCE: a figure published by the InfiniBand trade body, measuring scientific supercomputing rather than the commercial AI training where the build-out and the money currently are.

The job at the other end of this

Data center construction is creating a specific role in volume, and it is not the same role as enterprise network engineer. Four things define the work.

• You will not spend the day typing on switches. A fabric has forty leaves that are supposed to be identical, so nobody configures them individually. Config is generated from a source of truth and pushed, which makes the day Python, Jinja templates, a vendor API or Ansible, and a pipeline that validates before it deploys.
• You will design ahead. Oversubscription ratios, addressing plans, AS number plans, port and cable budgets: arithmetic done carefully, months before anything is racked.
• You will care about the physical layer. Transceiver part numbers, break-out cables, fibre polarity, DAC length limits. The commonest real fault in a new fabric is not a routing bug.
• You will troubleshoot statistically. When the report is "0.01% of flows between two pods see elevated latency", logging into a switch tells you nothing. You query telemetry across hundreds of devices and look for the shape of the problem.

What transfers, and what does not

What to learn first

1. Addressing, until it is automatic. A fabric consumes a remarkable number of small subnets: a /31 on every link, a loopback per switch, VTEP addresses, management. Practise with the subnet calculator, and read VLSM explained for why the allocation plan matters more than any individual mask.
2. BGP as a fabric protocol. Not internet policy: eBGP between leaves and spines, in the style RFC 7938 describes. The highest-value single topic on this list.
3. Build one. Containerlab, or any vendor's free virtual switch images, will run four leaves and two spines on a laptop. Watch ECMP install four next hops; unplug a virtual link and watch it become three. That half-hour teaches more than any diagram. Add VXLAN and EVPN once the underlay is boring.
4. Python and templating. Python for network engineers covers the parts that actually get used.
5. Optics and cabling. Least glamorous, most likely to be the real fault at three in the morning.

A CCNA is still the fastest route to the fundamentals, and a chunk of its layer 2 content describes a world you will not work in. Learn it anyway: understanding why spanning tree exists is how the fabric design makes sense, but do not mistake it for the destination. There is also a route that skips certification. Many data center network engineers arrive through operations such as cabling, commissioning and remote hands, then move across to the network team. The buildings are new and hiring in volume, so that internal move is available in a way it is not in established enterprise IT.

On pay, the honest version: the US Bureau of Labor Statistics puts the median for computer network architects at $134,050 as of May 2025, with 8% projected growth over 2025–2035. That category is broad and includes plenty of work that is not data center work, so read it as a regional signal. The neighbourhood is a good one, and the demand side of it is being poured in concrete right now.

Sources

• A Scalable, Commodity Data Center Network Architecture: Al-Fares, Loukissas and Vahdat, SIGCOMM 2008, the fat-tree paper
• Trends in Data Center Security: Traffic Trends: Cisco, 2014, citing the Global Cloud Index for the 76/17/7 split
• RFC 2992: Analysis of an Equal-Cost Multi-Path Algorithm, 2000
• RFC 7348: VXLAN, 2014: the 24-bit VNI and the 50-byte IPv4 overhead
• RFC 7432: BGP MPLS-Based Ethernet VPN (EVPN), 2015
• RFC 7938: Use of BGP for Routing in Large-Scale Data Centers, August 2016
• Building Meta's GenAI Infrastructure: Meta Engineering, March 2024: the two 24,576-GPU clusters and 400 Gb/s per GPU
• Ultra Ethernet Consortium Specification 1.0: June 2025
• RDMA Networking Powers the June 2026 TOP500: InfiniBand Trade Association, for the 293/83 split
• Computer Network Architects: US Bureau of Labor Statistics, Occupational Outlook Handbook, May 2025 data

Figures read in September 2026. Where a number comes from a party with an interest in it (the TOP500 interconnect counts most obviously), and that interest is stated in the text.