How Distributed AI Superfactories Stretch the Network Between Campuses

Power, land, water, and permitting—not just silicon—now decide how large an AI factory can grow inside one building. When a single site cannot host the next hundred thousand accelerators, operators do what electrical grids have always done: they distribute the load. The result is a new class of multi-campus AI network, where training and disaggregated inference jobs treat several facilities as one machine. That stretch from rack-to-rack scale-out to building-to-building and metro-to-metro connectivity is what the industry increasingly calls scale-across.

Scale-across is not a rebrand of yesterday’s backup pipe. Traditional AI data center interconnect (DCI) moved replicas, snapshots, and north–south cloud traffic. A modern scale-across fabric must carry GPU collectives, KV-cache and activation shards, checkpoint bursts, and control-plane telemetry with enough capacity, diversity, and congestion discipline that a synchronous job does not collapse into wait-for-network. This article explains how that works: the physics of distance, coherent and Coherent-Lite optics, line systems and C+L spectrum, peering and dark fiber, packet-layer scale-across (including NVIDIA Spectrum-XGS), software strategies for multi-site training, and what field trials actually measured.

From scale-up and scale-out to scale-across

Inside a hall, AI networking already splits into familiar layers. Scale-up links accelerators inside a node or pod (NVLink and similar). Scale-out ties pods together on a lossless or near-lossless back-end GPU fabric using InfiniBand or RoCE Ethernet. Front-end Ethernet still handles storage ingestion, APIs, and users. Cisco’s July 2026 scale-across briefing restates that separation and then asks the hard follow-up: which of those fabrics can you extend when the next rack of power sits in another building or another metro?

NVIDIA’s September 2025 engineering post defines scale-across as a third dimension of AI compute fabric connectivity, orthogonal to scale-up and scale-out. Spectrum-XGS Ethernet, announced 22 August 2025 at Hot Chips, is NVIDIA’s productization of that idea: algorithms that adapt congestion control and adaptive routing to the distance between facilities so multiple data centers can behave as one AI super-factory. NVIDIA positions the technology for links beyond about 500 meters—campus building spans as well as city- and region-scale paths. NVIDIA reports up to 1.9× higher NCCL all-reduce bandwidth versus off-the-shelf Ethernet in tests at 10 km, with the largest gains on the big messages typical of training.

Distance is not optional overhead; it is physics. NVIDIA notes that light in fiber propagates at roughly 5 nanoseconds per meter, so 1 km costs about 5 µs one way. That is invisible to a human and brutal to a tightly synchronized AllReduce. Deep-buffer “shock absorber” switches that telecoms use for long-haul drops can hide loss—but they add latency and jitter that synchronous AI jobs hate. Scale-across packet designs therefore try to remain shallow-buffer and telemetry-driven while still absorbing the bandwidth-delay product of the long path.

Why campuses and regions must share one job

Three forces push operators into multi-site designs.

Power and real estate. Lumentum’s September 2026 scale-across essay frames the slogan cleanly: it is often easier to move gigabytes than gigawatts. AI clusters concentrate enormous electrical and thermal loads; when grid interconnects, transformers, and land cannot keep up at one address, compute must fan out to wherever capacity comes online first.

Facility fungibility. NVIDIA argues that a unified scale-out / scale-across Ethernet architecture lets operators reallocate the same Spectrum-X switches and ConnectX-8 SuperNICs between intra-site and inter-site roles, improving ROI on stranded capacity. CoreWeave publicly committed early to Spectrum-XGS deployments for that reason.

Workload geography. STL Partners describes AI lifecycle traffic that trains in one region, fine-tunes in another, and serves inference from many metros, generating multi-directional, bursty DCI flows. Their research states that each megawatt of AI compute can generate up to twice the DCI throughput of traditional cloud capacity—an analyst figure, but directionally consistent with the industry’s rush toward denser wavelength counts and mesh routes.

None of this means every training job should straddle a continent. It means the network architecture must be ready when power maps and model size force the issue.

Optical networking AI: coherent, Coherent-Lite, and C+L

Once the path leaves the building, copper and short-reach IM-DD optics stop being the whole story. Optical networking AI at campus and metro scale increasingly means coherent pluggables plugged into routers and DCI switches, plus open line systems that light many wavelengths on shared fiber.

Standards: 400ZR, 800ZR, and the road to 1.6T

The Optical Internetworking Forum (OIF) made multi-vendor coherent DCI practical with 400ZR, first published in March 2020 and widely credited with enabling distributed data-center architectures via client-form-factor coherent modules. On 30 October 2024, OIF released the 800ZR Implementation Agreement for single-span, amplified 80–120 km DWDM links aimed at DCI, mapping Ethernet clients from 100GE up to 800G aggregate on one coherent line, with low-power pluggable modules suitable for high density. OIF also updated 400ZR (IA 3.0) and has continued work toward 1600ZR / Coherent-Lite variants for longer and shorter AI-relevant reaches, as summarized in OIF workshop materials through 2026.

Marvell’s OFC 2026 coherent panel framing is useful as a taxonomy (vendor presentation, but aligned with OIF reach classes): Coherent-Lite / campus for roughly 2–20 km low-latency dark-fiber O-band or optimized spans; ZR-class for roughly 20–120 km C/L DWDM; and ZR+/longer PCS modes for 1,000 km+ regional and backbone paths, with a commercial roadmap from 400G to 800G to 1.6T pluggables.

Near versus far scale-across

Ciena’s February 2026 scale-across blog, written by its chief product and technology officer, splits customer demand into near and far architectures.

Near scale-across targets metro/regional footprints with minimal intermediate amplifier sites. Hyperscalers asked for maximum reach without ILA huts “at all costs.” Ciena’s answer combines C- and L-band coherent pluggables (including WaveLogic 6 Nano 800G in L-band as well as C-band) with a compact RLS C+L line-system configuration—cited as fitting in about 6RU for an optically protected terminal—plus Raman amplification for stretched single spans and integrated optical protection switching across C or L spectrum. The commercial stakes are extreme: customers talking about tens of Pb/s of inter-site traffic and tens of thousands of connections turned up on compressed schedules.

Far scale-across accepts intermediate line amplifiers but demands radical densification. Traditional ILA shelves that terminate one fiber pair per chassis cannot economically support hundreds of fiber pairs for multi-region AI. Ciena’s RLS Hyper-Rail configuration is positioned as an order-of-magnitude fiber-density increase inside existing rack footprints so operators can avoid proliferating amplifier huts.

Ciena’s solutions page maps the continuum explicitly: campus DCI with Coherent-Lite; metro DCI with 400ZR/800ZR; dedicated scale-across with ultra-low-power C&L pluggables and diversely routed line systems; then backbone and submarine for longer hauls (including WaveLogic 6’s 1.6 Tb/s-class networking claims).

Peering, dark fiber, and mesh routes

Optics only matter if fiber paths exist with the right diversity and commercial terms. STL Partners argues that AI’s multi-directional bursts break classic hub-and-spoke DCI: traffic should take direct mesh paths between facilities rather than tromboning through a single core. Operators increasingly treat multi-route redundancy—same vendor or multi-vendor—as mandatory, and they look to dark fiber that can be lit or upgraded quickly when a training campaign spikes.

Nokia’s managed optical fiber network (MOFN) paper describes the commercial menu hyperscalers actually negotiate: managed bandwidth wavelengths from a CSP; private builds on leased or owned dark fiber where the hyperscaler owns the line system and coherent gear; and hybrid MOFN models that let the customer influence design while the CSP operates the plant. For continent-scale AI, paths often cross multiple CSP networks, with a lead CSP stitching partner segments—peering in the optical and business sense, not only BGP at an internet exchange.

Cisco’s design variables for scale-across are blunt: distance dominates RDMA degradation; backend GPU fabrics should keep lower oversubscription than frontend fabrics; deep buffering is situational, not dogma; and optics must match the span (FR/DR/LR versus coherent ZR/ZR+), verified by test rather than datasheet optimism. Their Zeus AI Lab methodology measures PerfTest RDMA throughput, NCCL collectives, and MLPerf job metrics on frontend- and backend-extension topologies from 100 meters to hundreds of kilometers.

In short, a multi-campus AI network is as much a fiber and peering strategy as a switch ASIC strategy: diverse routes, controllable optics, and enough spare spectrum that a failed conduit or a sudden AllReduce storm does not strand a billion-dollar training run.

Synchronous training across sites: what the evidence shows

Can you actually train one model across campuses without giving up most of your FLOPs to the WAN? Recent public evidence says yes—under the right parallelism and transport.

Software: hide the RTT

NVIDIA NeMo Framework 25.02 / Megatron-Core 0.11.0 documentation (May 2025 technical blog) describes four techniques for multi-data-center LLM training: adaptive resource orchestration that prefers latency-tolerant data and pipeline parallelism over chatty tensor/expert patterns across the WAN; hierarchical AllReduce that reduces inside a site, AllReduces across sites, then gathers locally; a distributed optimizer that shards state within each site and synchronizes gradients once across sites; and chunked inter-site transfers overlapped with compute. In a Nemotron-4 340B run across Oracle Cloud sites in Chicago and Ashburn—about 1,000 km apart, 21 ms measured RTT—NVIDIA reports that 3,072 GPUs (1,536 per site) reached over 96% of single-site throughput, with model FLOPS utilization moving only from 51% to 49%.

That result does not claim InfiniBand-inside-a-hall latency over a thousand kilometers. It shows that software topology can keep GPUs busy when the long haul carries carefully chosen collective phases.

Optics field trial: 120 km, 800G C+L

At OFC 2025, China Telecom Research Institute reported a field trial of 175-billion-parameter LLM training on 1,024 GPUs across two data centers 120 km apart on a high-reliability 800 Gbit/s C+L WDM optical transport network. Training efficiency reached up to 99.41% with pipeline parallelism and 98.95% with data parallelism. That is one of the clearest public proofs that metro-scale OTN plus the right parallel strategy can keep multi-site efficiency near single-site levels.

Packet scale-across: Spectrum-XGS at 10 km

NVIDIA’s 10 km Spectrum-XGS versus off-the-shelf Ethernet NCCL comparison (NVIDIA reports up to 1.9× all-reduce bandwidth) addresses a different slice of the problem: when the long path is still “Ethernet AI fabric” rather than a classic OTN circuit, congestion control and adaptive routing must become distance-aware or large training messages suffer. Spectrum-XGS shares Spectrum-X hardware so scale-out and scale-across are one operational domain with end-to-end telemetry—an important contrast to bolting a deep-buffer DCI box onto an otherwise shallow AI fabric.

Research stack: lossless RDMA over DC-OTN

Academic and industry research continues to attack the remaining failure modes of long-haul RDMA: stretched ACK loops, destination buffer pressure, and mixed intra-/inter-DC congestion timescales. GeoPipe (arXiv, October 2025) demonstrates enhanced pipeline parallelism over a lossless RDMA-enabled datacenter optical transport network across three sites, reporting large reductions in compute bubble ratio when cross-DC communication overlaps compute. Separate OTN-aware RDMA proposals (for example MatchRDMA-style rate matching) target the ACK and buffer pathologies of geo-distributed RoCE. These papers are early, but they underline a shared conclusion: optical networking AI for training is a full-stack problem—DSP, FEC, OTN buffering, NIC rate control, and model parallelism must be co-designed.

Designing the stretch: a practical blueprint

Operators stitching a distributed superfactory can treat the problem as layered contracts.

1. Classify the span. Campus (hundreds of meters to a few kilometers), metro (tens of kilometers), or regional (hundreds of kilometers), following Cisco’s tier language. Optics, amplification, and whether the backend fabric itself extends all change with the tier.

2. Decide what crosses the fiber. Extending the frontend for storage and orchestration is easier than extending the lossless backend. Many designs keep backend collectives mostly local and ship only pipeline stages, hierarchical reductions, or disaggregated inference shards across sites—exactly the NeMo orchestration philosophy.

3. Provision spectrum and diversity before GPUs. C+L doubling, Raman-assisted single spans, Hyper-Rail-class ILA density, and dual-conduit mesh are optical decisions that take longer than racking servers. STL Partners’ mesh-and-dark-fiber guidance belongs in the critical path of an AI campus plan, not as a follow-on ticket.

4. Instrument for jitter, not just bits. Scale-across success is predictable completion time. Measure NCCL and job-level metrics across the real fiber plant; vendor ZR reach numbers are necessary but not sufficient.

5. Keep commercial peering explicit. Whether you buy managed wavelengths, MOFN, or dark fiber you light yourself, write diversity, latency budgets, and burst upgrade rights into the contract. AI traffic’s peakiness makes static 95th-percentile pricing a false economy if a training wave needs a sudden Pb/s-class gulp between halls.

Inference, disaggregation, and the return path

Training gets the headlines, but multi-campus fabrics increasingly carry inference and disaggregated serving traffic as well. Prefill may run where power is cheapest; decode may sit closer to users; KV-cache and expert shards may bounce between halls on coherent wavelengths that were first lit for AllReduce. That return path has different traffic math—smaller messages, stricter tail latency, more continuous duty cycle—but it still depends on the same optical and peering substrate. Operators who over-optimize the interconnect solely for synchronous training risk under-serving the inference mesh that actually monetizes the GPUs between training campaigns.

Lumentum’s “gigabytes not gigawatts” framing applies equally here: moving activations and cache state between campuses is often cheaper than duplicating every accelerator pool in every metro. Combined with Spectrum-XGS-style distance-aware Ethernet for shorter metro stretches and ZR/ZR+ coherent for longer DCI, the factory floor becomes a programmable graph of halls rather than a single address. The engineering discipline is the same as for training: measure completion-time SLOs on the real fiber plant, keep diversity explicit in contracts, and refuse to treat the WAN as an afterthought once the racks are full.

Outlook: the network becomes the factory floor

By late 2026, the strategic picture is clear. Single-hall scale-out remains essential, but the ceiling on AI factories is increasingly set by AI data center interconnect—coherent pluggables, open line systems, dark fiber, and distance-aware Ethernet fabrics—plus training software that respects geography. OIF’s 800ZR and emerging 1.6T work industrialize the optical layer; Ciena, Marvell, Lumentum, and peers densify C+L plants; NVIDIA Spectrum-XGS and similar scale-across packet stacks try to keep collectives sane without deep-buffer jitter; NeMo-style hierarchical communication and OFC field trials show that synchronous multi-site training can retain most of single-site efficiency when parallelism matches the RTT.

The operators who win will not be those who merely lease the most wavelengths. They will be those who treat the multi-campus AI network as part of the accelerator complex itself: planned with power maps, lit with diverse coherent spectrum, peered for burst and failure, and scheduled so GPUs wait on algorithms—not on light still in flight between campuses.

Related reading

• InfiniBand vs Ethernet for AI Clusters: The Fabric War Inside the AI Factory
• Microsoft Fairwater and the Rise of the Distributed AI Superfactory
• Multi-Trillion-Parameter Training: What the Next Silicon Generation Must Deliver

Images: original illustrations generated for AndBeyond.Tech; no logos or real people depicted.