By Sagar Shankaran, Founder of CallSphere
Understand how high-bandwidth interconnects enable multi-GPU and multi-node AI training, the differences between interconnect technologies, and why network topology determines training efficiency.
Key takeaways
A common misconception about scaling AI training is that you simply add more accelerators and training goes proportionally faster. In reality, the relationship between accelerator count and training speed is governed by a deceptively simple equation: effective scaling efficiency equals useful compute time divided by total time (useful compute plus communication time).
When you double the number of accelerators in a distributed training setup, you double the available compute — but you also increase the volume of data that must be exchanged between accelerators. If the interconnect bandwidth cannot keep pace, communication overhead grows until adding more accelerators provides diminishing or even negative returns.
This is why interconnect technology has become the most critical differentiator in AI infrastructure design. The fastest accelerators in the world are worthless if they spend most of their time waiting for data from other accelerators.
The most common communication pattern in data-parallel training is all-reduce: every accelerator must send its gradient updates to every other accelerator, and every accelerator must receive the aggregated gradients before proceeding to the next training step.
flowchart LR
CORPUS[("Pre-training corpus<br/>trillions of tokens")]
FILTER["Quality filter and<br/>dedupe"]
TOK["BPE tokenizer"]
SHARD["Shard plus<br/>data parallel"]
GPU{"GPU cluster<br/>FSDP or DeepSpeed"}
CKPT[("Checkpoints<br/>every N steps")]
LOSS["Loss curve plus<br/>eval gates"]
SFT["SFT phase"]
DPO["DPO or RLHF"]
BASE([Base model])
INSTR([Instruct model])
CORPUS --> FILTER --> TOK --> SHARD --> GPU
GPU --> CKPT --> LOSS
LOSS --> BASE --> SFT --> DPO --> INSTR
style GPU fill:#4f46e5,stroke:#4338ca,color:#fff
style LOSS fill:#f59e0b,stroke:#d97706,color:#1f2937
style INSTR fill:#059669,stroke:#047857,color:#fff
For a model with B parameters trained across N accelerators, each all-reduce operation must transfer approximately 2B * (N-1)/N bytes of data. For a 70-billion parameter model in FP16 across 1,000 accelerators, that is roughly 280 GB of data that must flow through the network at every training step.
If a training step takes 100 milliseconds of compute time, the all-reduce must also complete within approximately 100 milliseconds to avoid being the bottleneck. Moving 280 GB in 100 ms requires an aggregate network bandwidth of 2.8 TB/s — far beyond what standard Ethernet can provide.
Mixture-of-experts architectures introduce all-to-all communication patterns where each accelerator sends different data to each other accelerator (routing tokens to their assigned experts). This pattern is more bandwidth-intensive per byte than all-reduce because it cannot exploit the tree-reduction shortcuts that all-reduce uses.
Pipeline-parallel training requires point-to-point transfers between adjacent pipeline stages. These transfers involve activation tensors (forward pass) and gradient tensors (backward pass) that can be hundreds of megabytes per micro-batch. Latency matters more than bandwidth here, because pipeline bubbles grow with transfer latency.
Within a single server containing multiple accelerators, proprietary high-bandwidth links provide the fastest communication:
Proprietary accelerator-to-accelerator links: Direct connections between accelerators within a node, typically providing 600-900 GB/s of bidirectional bandwidth per link. Each accelerator connects to its neighbors through multiple links, creating a mesh or fully-connected topology within the node.
The aggregate intra-node bandwidth can be extraordinary. An 8-accelerator server with full mesh connectivity might have:
Hear it before you finish reading
Talk to a live CallSphere AI voice agent in your browser — 60 seconds, no signup.
This bandwidth is critical for tensor parallelism, where individual matrix operations are split across accelerators within a node. The frequent, fine-grained communication of tensor parallelism requires the lowest possible latency and highest possible bandwidth.
Communication between servers uses network technologies that, while fast, cannot match intra-node bandwidth:
InfiniBand: The traditional high-performance computing interconnect, now offering 400 Gbps (NDR) and moving to 800 Gbps (XDR) per port. InfiniBand provides native RDMA, extremely low latency (under 1 microsecond), and hardware-based congestion management.
High-performance Ethernet: Modern 400/800 GbE with RDMA over Converged Ethernet (RoCE) provides competitive bandwidth to InfiniBand at lower cost. However, standard Ethernet was designed for web traffic, not the synchronized communication patterns of distributed training. Achieving InfiniBand-class performance on Ethernet requires sophisticated congestion control, priority flow control, and traffic engineering.
Proprietary scale-up fabrics: Some vendors offer proprietary network fabrics that extend the high-bandwidth, low-latency characteristics of intra-node interconnects across multiple servers. These fabrics treat a cluster of servers as a single logical system, providing 900 GB/s links between any two accelerators regardless of which server they reside in.
| Technology | Bandwidth per Port | Latency | Best Use |
|---|---|---|---|
| Proprietary intra-node | 600-900 GB/s | ~100 ns | Tensor parallelism within node |
| Proprietary scale-up | 900 GB/s | ~1 us | Extending intra-node bandwidth across nodes |
| InfiniBand NDR/XDR | 50-100 GB/s | ~1 us | General inter-node communication |
| 400/800 GbE + RoCE | 50-100 GB/s | ~2-5 us | Cost-effective inter-node communication |
The most common topology for AI clusters, a fat tree provides full bisection bandwidth — any half of the cluster can communicate with the other half at full aggregate bandwidth. This topology uses multiple tiers of switches:
The advantage is uniform performance: any-to-any communication patterns achieve consistent bandwidth. The disadvantage is cost — full bisection bandwidth requires a large number of switches and cables, particularly at the spine and core tiers.
Dragonfly topologies organize the network into groups:
Dragonfly is more cost-effective than fat tree for very large clusters because it requires fewer total switch ports. However, performance for adversarial traffic patterns (where many nodes in different groups communicate simultaneously) can be lower than fat tree.
A topology designed specifically for all-reduce communication patterns in AI training:
Each accelerator in a server connects to a different "rail" — an independent network fabric. For an 8-accelerator server, there are 8 rails, each carrying 1/8 of the all-reduce traffic. This spreads communication load across multiple independent networks and reduces the number of switch hops.
Rail-optimized topologies work exceptionally well for data-parallel training but may underperform for workloads with irregular communication patterns.
Accelerator compute performance has been growing faster than interconnect bandwidth:
This growing gap means that communication overhead increases with each hardware generation unless training algorithms are adapted to reduce communication requirements.
Still reading? Stop comparing — try CallSphere live.
CallSphere ships complete AI voice agents per industry — 14 tools for healthcare, 10 agents for real estate, 4 specialists for salons. See how it actually handles a call before you book a demo.
As cluster size grows, the minimum latency for global communication operations increases. An all-reduce across 10,000 accelerators requires data to traverse multiple network hops, with cumulative latency that can reach tens of microseconds. At training step durations of 50-100 milliseconds, this latency is a small fraction. But for very short training steps (small models, small batches), it becomes the dominant cost.
In a cluster of 10,000 accelerators connected by 100,000+ cables and thousands of switches, link failures are routine. The network must:
Adaptive routing, redundant paths, and automatic failover mechanisms are essential at this scale.
Hardware bandwidth is the ceiling, but software determines how close applications get to that ceiling.
Specialized communication libraries orchestrate collective operations across the network:
When interconnect bandwidth is the bottleneck, reducing the volume of data communicated can improve scaling:
Fully synchronous training (every accelerator waits for every other before proceeding) maximizes gradient quality but is most sensitive to communication bottlenecks. Asynchronous or semi-synchronous approaches relax this constraint:
Organizations deploying AI training infrastructure should:
Right-size the interconnect for the workload: Training frontier models across thousands of accelerators demands the highest-bandwidth interconnects available. Fine-tuning smaller models may work well with standard high-speed Ethernet.
Plan for topology: The physical arrangement of racks, switches, and cables constrains which training parallelism strategies are efficient. Work with network architects who understand both AI workloads and physical infrastructure constraints.
Budget for interconnect: Network infrastructure typically represents 15-25% of the total cost of an AI training cluster. Underinvesting in interconnect to save capital often results in poor accelerator utilization — a far more expensive outcome than the switch and cable costs saved.
The interconnect is the circulatory system of an AI training cluster. Without adequate bandwidth, latency, and reliability, even the most powerful accelerators cannot deliver their potential.
High-bandwidth interconnects are the specialized networking fabrics that connect accelerators within an AI training cluster, enabling them to exchange data at speeds ranging from 400 Gbps to over 1.8 terabytes per second for intra-node links. These interconnects are critical because distributed AI training requires accelerators to continuously synchronize model gradients, attention states, and activation data across thousands of devices. Network infrastructure typically represents 15-25% of the total cost of an AI training cluster, reflecting its importance to system performance.
Interconnect bandwidth is often the bottleneck in distributed AI training because scaling to thousands of accelerators generates enormous volumes of inter-device communication, particularly during gradient synchronization in data-parallel training. If the network cannot deliver data as fast as accelerators can consume it, expensive GPUs sit idle waiting for communication — a problem that worsens as cluster size increases. Modern training parallelism strategies like tensor parallelism, pipeline parallelism, and expert parallelism each have distinct communication patterns that place different demands on the interconnect topology.
Intra-node interconnects (like NVLink and NVSwitch) connect accelerators within a single server at bandwidths exceeding 900 GB/s per GPU with sub-microsecond latency, enabling accelerators to share memory almost as if they were a single device. Inter-node interconnects (like InfiniBand and high-speed Ethernet) connect servers across racks and data center floors at 400-800 Gbps per port with latencies in the low microseconds. The bandwidth gap between these two tiers — roughly 10x — fundamentally shapes how AI training workloads are partitioned across infrastructure.
Network topology determines which accelerators can communicate efficiently, directly constraining which parallelism strategies are practical for distributed AI training. Fat-tree topologies provide equal bandwidth between any pair of nodes but are expensive at scale, while rail-optimized designs reduce cost by matching topology to common communication patterns like all-reduce operations. Organizations should plan for topology during infrastructure design because the physical arrangement of racks, switches, and cables cannot be easily changed after deployment.
Written by
Sagar Shankaran· Founder, CallSphere
LinkedInSagar Shankaran is the founder of CallSphere, where he builds production AI voice and chat agents deployed across healthcare, hospitality, real estate, and home services. He writes about agentic AI, LLM engineering, and shipping voice agents that handle real calls in production.
See how AI voice agents work for your industry. Live demo available -- no signup required.
Three distributed-training options for PyTorch in 2026 compared on ergonomics, scaling, and where each one wins.
How generative AI produces verified dbt models for data migration — from scratch and incrementally — with SME validation and strict data governance.
See how Circini's automated incident management pipeline turns alert emails into triaged Jira tickets using Snowflake Cortex AI, GPT-4.1, Airflow & MS Teams.
Anthropic's Claude Fable 5 and Mythos 5 explained: pricing, availability, frontier benchmarks, the dual-model safeguard architecture, and what they mean for AI agents.
Where Claude Code, MCP, and multi-agent systems are taking GTM engineering next, and how to prepare your team now for standing and multi-agent workflows.
Where Claude Cowork and the Claude agent ecosystem are heading next — standing agents, MCP, skills as a moat — and the concrete moves to prepare your team now.
© 2026 CallSphere Inc. All rights reserved.
Made within San Francisco
Watch how CallSphere handles real customer calls, schedules appointments, and processes payments — live.
Try Live DemoBook a DemoCalculate Your ROI