AI/ML Infrastructure
from Silicon to Scale
The complete technical reference for Principal Performance Architects — from GPU silicon physics through 10,000-GPU fleet orchestration. Every claim quantified. Every formula derivable from first principles.
21Chapters
5Appendices
29Files Complete
2026Edition
989TH100 BF16
3.35HBM3 TB/s
295Ridge F/byte
41.8ms70B TPOT min
5.37 GBKV/req 4K
10–30×Cont. batch
2–3×TRT-LLM gain
88%KV alert
Reader Paths
Choose Your Track
2 weeks · 2 hr/day
Interview Fast-Track
Ch 01 → Ch 02 → Ch 06→ Ch 11 → Ch 10 → Ch 14
→ Ch 18
Formulas first. Numbers beat prose in every interview. All review questions are your drill set.
Weeks 1–3 core
Inference Engineer
Ch 01 → Ch 06 → Ch 11→ Ch 13 → Ch 08 → Ch 17
vLLM · TRT-LLM · PagedAttn · Disaggregation · SLA engineering · Observability.
Weeks 1–3 systems
Training Engineer
Ch 01 → Ch 10 → Ch 14→ Ch 09 → Ch 07 → Ch 12
Megatron · ZeRO · FSDP · NCCL · 4D parallelism · Benchmarking pipeline.
Start broad
Infrastructure Architect
Ch 03B → Ch 05 → Ch 14→ Ch 16 → Ch 17 → Ch 15
TCO · Thermal · Cluster ops · Observability · Networking · Storage I/O at scale.
All Chapters
Ch 00 – Ch 21 · 📖 HTML ⇩ PDF — all available
Front Matter
Part I — Foundations
Ch 01
The AI/ML Performance Architecture Mindset ✓ Done
Seven-layer stack, three regimes, roofline model, MFU/HFU, optimization leverage hierarchy
Ch 02
Transformer Architecture Deep Dive ✓ Done
Attention FLOP budget, GQA/MQA/MLA, RoPE, SwiGLU — complete LLaMA-3 70B derivation
Ch 03
GPU Architecture: From Silicon to FLOPS ✓ Done
SM hierarchy, Tensor Cores, Transformer Engine FP8, memory subsystem, ridge point
Ch 3A
Accelerator Landscape: Beyond GPUs ✓ Done
TPU, Apple ANE, AWS Trainium/Inferentia, Intel Gaudi 3, Meta MTIA, Groq LPU — decision matrix
Part II — Physics of Compute
Ch 04
Memory Hierarchy, HBM & FlashAttention ✓ Done
Memory wall, five-tier hierarchy, HBM3 physics, FlashAttention v1/v2/v3 complete derivation
Part III — Inference & Kernel Engineering
Ch 06
Inference System Architecture ✓ Done
Prefill/decode physics, continuous batching, PagedAttention, disaggregation, fleet sizing
Ch 07
GPU Kernels & GEMM Optimization ✓ Done
Tiling hierarchy, Tensor Core alignment, Triton, CUTLASS, FlashDecoding, profiling
Ch 08
Quantization: FP8, INT4, MXFP, QAT ✓ Done
FP8 E4M3/E5M2 recipes, AWQ/GPTQ, SmoothQuant, KV cache quantization, NVFP4
Part IV — Training at Scale
Ch 10
Distributed Training Systems ✓ Done
4D parallelism (TP/PP/DP/SP), ZeRO stages 1/2/3, RLHF/PPO memory layout, MoE training
Ch 11
KV Cache: Heart of LLM Serving ✓ Done
PagedAttention deep dive, RadixAttention prefix caching, KV offloading, eviction policies
Part V — Infrastructure at Scale
Ch 13
Speculative Decoding & MoE Architecture ✓ Done
Draft-verify acceptance rate, Medusa, MoE expert routing, load balance, All-to-All
Ch 14
Networking & Collective Communication ✓ Done
NVLink topologies, InfiniBand NDR/XDR, NCCL algorithms, RDMA, RoCEv2 at scale
Ch 15
Storage, Data Pipelines & I/O ✓ Done
NVMe RAID, Lustre/GPFS/WekaFS, DALI, checkpoint I/O cadence, dataset streaming
Ch 16
Cluster Management & Scheduling ✓ Done
SLURM topology-aware placement, Kubernetes GPU, gang scheduling, MIG/MPS, SDC
Part VI — The Principal Architect
Ch 18
Career, Interviews & System Design ✓ Done
STAR+Impact framework, system design walkthroughs, ChatGPT design, 90-day onboarding
Ch 19
Production Resilience & Fault Tolerance ✓ Done
Failure taxonomy, checkpointing strategy, NCCL recovery, SDC detection, chaos engineering
Ch 20
The Future of AI Compute ✓ Done
CXL, photonic interconnects, new memory technologies, 2029 outlook, strategic positioning
Appendices
App A – App E
App A
✓ Done
Hardware Quick Reference
GPU specs (NVIDIA + AMD), HBM generations, interconnect table, 30 numbers cold, ridge points, MFU reference
App B
✓ Done
Profiling Tools Reference
Nsight/NCU command reference, DCGM field catalog, NCCL env vars, nccl-tests, AMD omniperf, vLLM metrics
App C
✓ Done
Glossary of Key Terms
A–Z glossary with chapter cross-references and formula pointers for every major term (~200 terms)
App D
✓ Done
Performance Formulas
Every formula a principal must derive — roofline, GEMM, KV cache, AllReduce, TCO, training throughput