Chapter 20

The Future of AI Compute: 2026–2029

Beyond the GPU Era — CPU renaissance · CXL memory disaggregation · In-network computing · Silicon photonics · HBM4/5 · The heterogeneous compute architecture that replaces brute-force GPU scaling
"The GPU won the first decade of deep learning because it was the right answer to a narrow question: how do you multiply large matrices fast? The next decade asks a different question: how do you move intelligence through systems at planetary scale? That question has a different answer."
— AI Infrastructure Architecture Perspective, 2026
Chapter Scope — What Changed Since 2024 Everything in this chapter is grounded in silicon already shipping, architectures formally announced, or physics that does not change. Timeline estimates reflect public roadmap commitments as of early 2026. The bottleneck shift from "GPU FLOPS shortage" to "memory bandwidth, networking, and power efficiency" is no longer a forecast — it is the present reality.

20.1 The Inflection Point — Why the Pure GPU Era Is Ending

For a decade, the answer to every AI scale question was: add more GPUs. That answer is hitting four simultaneous physical walls that more GPU FLOPS cannot solve.

The Four Physical Walls of GPU-Centric AI (2026) ⚡ WALL 1: POWER Density H100: 700W per GPU GB200: 1,400W per GPU 2028: ~2,000W+ Air cooling fails >500W. Liquid cooling mandatory. Data centers need 200–300 kW per rack. → Power IS the limit, not silicon area. 🧠 WALL 2: MEMORY Bandwidth Ceiling H100 HBM3: 3.35 TB/s B200 HBM3e: 8.0 TB/s HBM4: ~13+ TB/s (2027) Roofline ridge point shifts as BW grows. LLM decode is BW-bound at virtually all batch sizes. → Memory BW, not FLOPS, is #1 limit. 🔗 WALL 3: NETWORK Bandwidth Gap NVLink: 900 GB/s (intra) IB NDR: 400 Gbps (inter) Gap: 18× intra vs inter AllReduce is the primary training bottleneck at >1,024 GPUs. Photonics closes the gap by 2028–2029. → Inter-node BW is 🏭 WALL 4: SILICON Concentration Risk NVIDIA market share: ~80% TSMC N4: capacity ceiling Custom ASIC rush: 2025+ Google TPUs, AWS Trainium, Meta MTIA, MS Maia, Apple ANE, Groq LPU Structural diversification away from GPU monoculture. → Heterogeneous era
Figure 20.1 — The four physical walls forcing the shift from homogeneous GPU clusters to heterogeneous, disaggregated AI infrastructure.

20.1.1 The Architectural Response: Disaggregation

THE SHIFT IN SYSTEM DESIGN PHILOSOPHY: 2020–2025 (GPU-centric era): Server = CPU + 8 GPUs + fixed memory + fixed networking All resources co-located, statically allocated, vendor-locked One accelerator type for training AND inference Network: "just connectivity" between compute nodes 2026–2029 (Disaggregated era): Compute pool: GPUs/accelerators of multiple types Memory pool: CXL-attached DRAM/HBM, independently scalable Network: Active compute substrate (in-network reductions) CPU pool: Powerful host processors as first-class AI compute Workload → Scheduler → allocates from each pool dynamically No permanent binding of compute to memory to network ONE-LINE TAKEAWAY (from infographic): "AI is no longer a compute problem. It is a data movement and system efficiency problem."

20.2 The CPU Renaissance in AI Infrastructure

The narrative of "GPUs replaced CPUs for AI" was always an oversimplification. CPUs never left the critical path — they moved to the side. Now they are moving back to the center, driven by four structural forces.

CPU AI Renaissance: Four Structural Drivers Driver 1: CPU AI Performance Intel Xeon Granite Rapids: AMX: ~10 TFLOPS BF16 AMD EPYC Venice (2026): AI accelerator per CCX ARM Neoverse V3 (Grace): 900 GB/s to B200 via C2C Apple M4: 30–50 tok/s at 10W CPU FLOPS: no longer negligible vs GPU Driver 2: CPU-Native Loads Embedding lookup (RecSys) Tokenization at serving scale KV cache block allocator Request routing/batching Scheduling (Kubernetes) CPU is on the serving critical path regardless of GPU strength. Driver 3: Coherent Links Old: CPU ↔ PCIe (128 GB/s) → CPU memory isolated New: NVLink-C2C (900 GB/s) → Unified address space → KV cache spans LPDDR+HBM → 512K+ context feasible Grace Blackwell: 480 GB LPDDR5X + 192 GB HBM = 672 GB unified pool Driver 4: Edge Efficiency Small model inference (1B-7B) Apple M4: 30-50 tok/s @10W Snapdragon X: 20 tok/s @15W Cloud GPU: >100W per session For on-device inference: CPU/SoC wins on power. Margin vs GPU closing as model quantization enables INT4/INT8 on CPU.
Figure 20.2 — CPU AI renaissance: four structural drivers returning CPUs to the center of AI infrastructure design.

20.2.1 Grace Blackwell: The CPU/GPU Paradigm Shift SHIPPED

GRACE BLACKWELL SUPERCHIP ARCHITECTURE (GB200 NVL72): Grace CPU (ARM Neoverse V3) B200 GPU (Blackwell) ┌────────────────────────┐ ┌────────────────────────┐ │ 72 ARM Neoverse cores │ │ 192 GB HBM3e │ │ 480 GB LPDDR5X │◄─────►│ 8 TB/s memory BW │ │ 960 GB/s CPU BW │NVLink │ ~20 PFLOPS FP4 │ │ │ C2C │ │ │ Power: ~200W │900GB/s│ Power: ~1,400W │ └────────────────────────┘ └────────────────────────┘ ↕ ↕ Total coherent memory = 480 GB LPDDR5X + 192 GB HBM3e = 672 GB KEY IMPLICATIONS FOR INFERENCE: Long-context KV cache: overflows HBM → CPU LPDDR5X transparently Latency penalty: HBM (~100ns) → LPDDR5X from GPU (~300ns) Still 33× faster than NVMe → far better than disk-based KV tiering Context window feasibility on GB200 NVL72 (72 B200 + 36 Grace): HBM total: 72 × 192 GB = 13.8 TB LPDDR5X total: 36 × 480 GB = 17.3 TB Combined: = 31.1 TB unified Supports: multi-million token contexts on a single NVL72 rack

20.3 CXL — The Memory Disaggregation Standard

CXL (Compute Express Link) is the most significant memory architecture change in a decade. It enables memory to be disaggregated from processors — placed on independent devices, shared across multiple CPUs and accelerators, and dynamically allocated like compute resources.

CXL Memory Disaggregation Architecture (2026–2028) TRADITIONAL (Today) CPU Node A 2 TB DDR5 (local only) CPU Node B 2 TB DDR5 (local only) GPU Node A 80 GB HBM (local only) GPU Node B 80 GB HBM (local only) Memory is STRANDED — unused capacity cannot be shared Node A at 90% while Node B at 20% → waste → CXL DISAGGREGATED (2027) CPU Node A CPU Node B GPU Node A CXL 3.1 Switch Shared CXL Memory Pool 256 TB DRAM · Any node · Dynamic allocation Any node borrows from pool on demand — no waste KV cache migrates with request — no data movement
Figure 20.3 — CXL memory disaggregation: traditional stranded memory vs CXL pooled memory accessible by any compute node dynamically.
CXL ADOPTION TIMELINE AND SPECIFICATIONS: CXL Version Year Key Capability Status ─────────────────────────────────────────────────────────────────── CXL 1.1 2021-23 Type 1/2/3 devices, PCIe 5.0 base [SHIPPED] Memory extension for CPU-only servers Latency: ~250 ns overhead vs direct DRAM Shipping: Samsung, Micron, SK Hynix expanders CXL 2.0 2022-24 Switching + sharing between multiple CPUs [SHIPPED] One memory expander → shared by 4-8 CPU sockets Shipping: Xconn, Enfabrica CXL switches CXL 3.0 2024-26 Rack-scale memory disaggregation [SHIPPING] Multi-level switching, thousands of endpoints GPU support confirmed: NVIDIA Vera Rubin (2027) AMD: CXL support in EPYC Venice (2026+) CXL 3.1+ 2026-28 "Memory as a Service" at cluster scale [ROADMAP] AI cluster memory fabric replacing local DRAM Target: allocate memory like compute resources PROJECTED IMPACT ON AI INFERENCE (2027-2028): KV cache stored in CXL pool → served to any available GPU Request migration: session moves between GPUs without cache loss Memory efficiency: 2-3× effective KV capacity with same hardware vs per-GPU HBM siloed approach

20.4 In-Network Computing — The Network Becomes a Compute Layer

Traditional networks move data. Next-generation AI networks compute while they move data. This is the deepest architectural shift in cluster design since GPUDirect RDMA.

Evolution of Network Compute (2015 → 2029) Gen 1: Dumb Pipes Pre-2015 Network moves bytes A→B All aggregation at CPUs N endpoints = N-1 trips AllReduce: O(N) traffic CPU involved in every reduction step Bottleneck: CPU overhead + N-1 hops Gen 2: RDMA 2015–2022 GPUDirect RDMA Bypasses CPU entirely GPU → NIC → GPU direct Problem solved: CPU O/H Problem remaining: All compute still at endpoints (GPU) Bottleneck: endpoint GPU compute Gen 3: In-Network 2022–Present ✓ IB SHARP: AllReduce INSIDE IB switches NVLS: AllReduce inside NVSwitch ASICs Results: 30–50% less traffic 40% lower latency Bottleneck: network bandwidth Gen 4: Programmable 2026–2029 [ROADMAP] Smart NICs (BlueField-3): ARM cores in NIC P4 switches (Tofino 2): Custom algos in silicon Use cases: Gradient compression 10:1 Adaptive routing Token routing for MoE Bottleneck: software maturity
Figure 20.4 — In-network computing evolution from dumb pipes to programmable compute fabric inside the network switches themselves.

20.4.1 NVIDIA Spectrum-X: Ethernet for AI SHIPPED

THE ETHERNET vs INFINIBAND CONVERGENCE: Historical problem: InfiniBand: fast, reliable, lossless → good for AI AllReduce Ethernet: lossy, packet drops → bad for RDMA-based collectives NVIDIA Spectrum-X (2024): Ethernet that behaves like InfiniBand Congestion control: Advanced ECN + proactive congestion avoidance Adaptive routing: per-packet load balancing (IB ECMP equivalent) NIC pairing: ConnectX-7 with RoCEv2 optimized for ML workloads Claimed performance: RoCE AllReduce within 5% of InfiniBand COST COMPARISON AT 10,000 GPU CLUSTER: InfiniBand switching infrastructure: ~$500M Ethernet (Spectrum-X): ~$100M Savings: ~$400M per 10K GPU cluster If performance parity holds: Ethernet wins at hyperscaler scale. Meta 24K GPU cluster: RoCE over Ethernet (already deployed). TIMELINE FORECAST: 2024-2025: First Spectrum-X deployments; benchmarking underway 2026-2027: Broad hyperscaler adoption if performance confirmed 2028-2029: InfiniBand becomes niche (extreme latency requirements) Ethernet becomes the default AI cluster network

20.5 Photonic Interconnects — Light-Speed Data Movement

Electrical copper signals face a fundamental bandwidth-distance limit. At 400G per link, copper works for distances up to ~3 meters. As AI clusters scale to tens of thousands of nodes across large datacenter floors, photonics becomes structurally necessary.

Silicon Photonics Adoption Roadmap for AI Clusters Today 2025–2027 2027–2029 2029+ Stage 1: Pluggable Optics Digital → electrical → laser → fiber → photodetector BW: 400G per port Power: ~10W per module Cost: $1K–5K per port Status: [SHIPPED] Stage 2: Co-Packaged Optics Laser integrated INTO switch package Eliminates pluggable transceiver BW density: 5-10× improvement Power: ~50% reduction Shipping: Broadcom (2026) Status: [ANNOUNCED/SHIPPING] Stage 3: Silicon Photonics Photonic components on CMOS wafer All electrical long-haul eliminated BW: 1.6 Tbps per fiber link Per node: 51.2 Tbps (32 ports) Research: Intel Labs, MIT, Stanford Status: [ROADMAP 2028-2029] 2029 Impact Inter-node BW approaches NVLink speeds TP spans nodes w/o overhead
Figure 20.5 — Silicon photonics adoption roadmap. By 2029, inter-node bandwidth may approach NVLink intra-node speeds, fundamentally changing parallelism strategy.

20.6 New Memory Technologies — Beyond HBM3e

HBM EVOLUTION ROADMAP (2024–2030): HBM3e (2024) HBM4 (2027) HBM5 (2029) Current AMD MI450 [ROADMAP] standard NVIDIA Rubin ───────────────────────────────────────────────────────── Pin speed 9.6 Gbps 13-16 Gbps 20+ Gbps [INF] Bus width 1024-bit 2048-bit (!) 2048-bit Per stack ~1.2 TB/s ~3.2-4 TB/s ~5 TB/s [INF] Capacity 24-32 GB 64+ GB 128+ GB [INF] Key change — Wider bus, Logic die base die logic compute (PIM) IMPACT ON ROOFLINE: H100 (HBM3): Peak BW 3.35 TB/s → ridge point at 312 FLOPS/byte B200 (HBM3e): Peak BW 8.0 TB/s → ridge point at ~250 FLOPS/byte Rubin (HBM4): Peak BW ~12 TB/s → ridge point at ~200 FLOPS/byte Observation: As BW grows faster than compute, more operations become compute-bound. Attention (currently BW-bound at small batch) may become compute-bound at decode with HBM5 bandwidths. PROCESSING IN MEMORY (PIM) — Emerging [ROADMAP]: Samsung HBM-PIM: ARM cores embedded in HBM die SK Hynix AiM: DRAM with compute units between DRAM banks Use case: Embedding table lookup — computed INSIDE memory device Eliminates: the DRAM → GPU bus transfer entirely for this workload Benefit: ~7× power reduction for embedding lookup workloads vs conventional GPU-attached HBM approach

20.7 The Heterogeneous Compute Era — System Design Implications

The infographic's framework captures the structural shift precisely: we are moving from FLOPS-centric thinking to data movement and system efficiency thinking. For a Principal Performance Architect, this changes what you optimize.

Architecture Shift: What the Principal Engineer Optimizes OLD WORLD (2020–2024): FLOPS-Centric → FLOPS-Centric Thinking → Kernel Optimization (per-op) → Single GPU tuning focus → Compute scaling (add GPUs) → Today's throughput only → Homogeneous GPU cluster → Memory/network secondary "Add more GPUs" is the answer to everything ⟹ NEW WORLD (2026+): System Efficiency → Data Movement & System Efficiency Thinking → End-to-End Pipeline Optimization → Cluster-Level Efficiency (MFU at scale) → System Scaling (memory + network + compute) → Sustained Throughput at Scale → Heterogeneous: GPU+CPU+ASIC+CXL → Memory & network are first-class constraints "The system matters more than any single chip"
Figure 20.6 — The architecture shift in Principal Engineer thinking: from per-GPU FLOPS optimization to system-level data movement and efficiency engineering.

20.8 Five Invariants That Survive Every Hardware Generation

The hardware stack changes every 18–24 months. These five analytical invariants do not. They are the stable substrate beneath an unstable surface.

Invariant 1: Memory Bandwidth Is Always the Primary Constraint Whether the platform is HBM3e, HBM4, CXL-pooled DRAM, or a future medium, the ratio of available bandwidth to required bandwidth determines whether a workload is memory-bound. Roofline analysis on this ratio remains valid forever. Optimizing memory hierarchy, capacity, and bandwidth is always the primary job.
Invariant 2: Arithmetic Intensity Determines the Performance Regime FLOPs/byte is the universal classifier. At AI=1 (embedding lookup), you are always memory-bound. At AI=256 (large GEMM), you are always compute-bound. This ratio and what it implies do not change with hardware generation.
Invariant 3: Communication-to-Compute Ratio Governs Scalability The fraction of time spent in AllReduce vs computation is the fundamental scalability limit. As you add GPUs, this fraction grows. Reducing communication volume (gradient compression, better parallelism strategies, in-network reduction) remains the key lever at scale.
Invariant 4: Disaggregation Requires Programming for Latency As CXL, silicon photonics, and pooled memory increase the diversity of memory latencies, every algorithm must explicitly account for which memory tier it accesses. The roofline model expands from 2D (compute vs one memory) to N-D (compute vs HBM vs CXL vs NVMe vs remote). The principle remains the same.
Invariant 5: The Analytical Tools Are Generational, Not Generational Roofline analysis, arithmetic intensity, collective communication modeling, MFU calculation — these tools were invented for GPUs but apply to TPUs, CXL-attached memory, photonic fabrics, and every future substrate. Invest deeply in the tools, not just the current hardware they apply to.

Chapter 20 Review Questions

  1. The Grace Blackwell superchip achieves 900 GB/s coherent bandwidth between CPU and GPU via NVLink-C2C. Compare the economics of KV cache overflow to CPU DRAM vs to NVMe SSD for a 128K token context with LLaMA-3 70B in BF16. Calculate the effective access latency for each tier and the maximum fraction of KV cache that can reside in CPU LPDDR5X before P99 TPOT degrades by more than 20%.
  2. A CXL 3.1 memory pool provides 256 TB of shared DRAM across a 1,000-GPU inference cluster at 200 ns latency. Compared to per-node local DRAM (80 ns, 2 TB per node), what is the TCO advantage (fewer memory purchases) and what is the latency cost? At what request working-set size does the CXL pool outperform the local DRAM approach on cost-efficiency?
  3. IB SHARP reduces AllReduce latency by 40% for large collectives. If a 1,024-GPU training job spends 28% of step time in AllReduce, calculate: (a) the new AllReduce fraction with SHARP, (b) overall step time improvement, (c) what additional techniques are needed to reach AllReduce fraction below 10%?
  4. Silicon photonics provides 51.2 Tbps per node inter-node bandwidth by 2029, exceeding NVLink's 900 GB/s intra-node bandwidth. A startup claims this eliminates the need for tensor parallelism because inter-node communication becomes faster than intra-node. Evaluate this claim: what constraints remain, what new bottlenecks emerge, and how would you redesign the inference serving stack to exploit this topology?
  5. The chapter identifies five invariants. For each invariant, give a concrete example of how it applies to a system built on CXL-pooled memory and silicon photonic networking rather than traditional GPU-attached HBM and copper InfiniBand. What changes in the analysis, and what stays the same?
Closing Thought — The Field That Changes Faster Than Any Book

The specific hardware numbers in this chapter will be outdated within 18–24 months. The five invariants will not be. The engineer who builds and operates the next generation of AI systems is the one who understands the system as a whole — roofline analysis, communication-to-compute ratios, memory hierarchy reasoning — and applies those tools to every new platform as it appears. Build systems. Remove bottlenecks. Create the future.

— End of Chapter 20 · AI/ML Infrastructure from Silicon to Scale · Venkat Vinjam —