"The engineer who can walk into an architecture review and say: 'this design is memory-bandwidth-bound at these shapes — here is the roofline, here is the gap, here are two approaches to close it with their TCO implications' — that is the principal. This book builds that engineer." — Book Mission Statement
This book began as a living preparation system for one of the most technically demanding roles in modern software: Principal Performance Architect for AI/ML infrastructure. It grew through real interview cycles, production architecture reviews, and hands-on performance engineering at GPU cluster scale into a comprehensive reference that covers the full technical stack — from the physics of DRAM cell refresh to the economics of a 10,000-GPU fleet.
Every other resource covers one layer. Deep learning textbooks cover model theory without hardware. CUDA programming guides cover kernel optimization without system context. MLOps books cover pipelines without silicon understanding. Interview prep books give patterns without first-principles math. This book is the only single reference that unifies all of these into a coherent analytical system.
The book serves three reader levels simultaneously:
| Level | Background | What You Get |
|---|---|---|
| Foundational | Strong engineer new to AI infra | First-principles mental models, "why does this matter" framing, intuitive analogies, no assumed GPU/ML background |
| Intermediate | Senior engineer, 3–5 years in cloud/HPC/ML | Quantitative depth, cross-system connections, profiling workflows, production numbers you can verify on your own cluster |
| Principal | Staff/principal engineer, architecture decisions | Cost-performance tradeoff math, system design patterns at scale, failure mode reasoning, ROI-driven decision frameworks, interview-ready depth |
| Path | Goal | Chapter Sequence |
|---|---|---|
| Hardware Track | Deep GPU/accelerator mastery | Ch 1 → 3A → 3B → 4 → 5 → 7 → App A |
| Inference Track | Production LLM serving expertise | Ch 1 → 2 → 6 → 11 → 8 → 13 → 17 |
| Training Track | Large-scale training systems | Ch 1 → 2 → 10 → 14 → 15 → 16 → 12 |
| Career Track | Principal-level interview readiness | Ch 1 → 18 → 12 → then any deep-dive chapter |
This book uses confidence labels on all hardware specifications. Technology moves faster than publication cycles — a label tells you whether to trust a number at face value or verify it against current documentation before using it in a design decision.
| Label | Meaning | Trust Level |
|---|---|---|
| [SHIPPED] | Measured on production hardware; verified against manufacturer datasheets | High — use in production calculations |
| [ANNOUNCED] | Vendor-disclosed, not yet generally available or shipping | Medium — verify before committing to design |
[ESTIMATED] | Calculated from first principles or extrapolated from known data | Medium — check the derivation, not just the value |
[REPRESENTATIVE] | Illustrative example; not a specific product claim | Conceptual — don't use in procurement decisions |
[ENV-SPECIFIC] | Highly cluster- and framework-dependent; will vary in your environment | Directional — always measure in your system |
| Part | Chapter | Title | Key Topics |
|---|---|---|---|
| I · Foundations | Ch 1 | Performance Mindset | Seven-layer stack, roofline, three regimes, MFU/HFU |
| Ch 2 | Transformer Architecture | Attention FLOPs, GQA/MQA/MLA, MoE, full forward pass budget | |
| Ch 3 | GPU Architecture | SMs, Tensor Cores, HBM, NVLink — the hardware model | |
| Ch 3A | Accelerator Landscape | TPU, Trainium, Gaudi 3, Groq, MTIA — when each wins | |
| Ch 3B | GPU Generation Roadmap | H100→GB300, MI300X→MI455X, TCO comparisons | |
| II · Physics | Ch 4 | Memory Hierarchy + FlashAttention | Memory wall, HBM architecture, FA v1/v2/v3 derivation |
| Ch 5 | Thermal, Power & TCO | Cooling, TDP, $/token/day, liquid cooling transition | |
| III · Inference | Ch 6 | Inference System Architecture | Prefill/decode, PagedAttention, disaggregation, SLA |
| Ch 7 | GPU Kernels & GEMM | Tiling, Tensor Core alignment, Triton, CUTLASS | |
| Ch 8 | Quantization | FP8 recipes, INT4/AWQ, KV quant, MXFP | |
| Ch 9 | Fusion & Compilers | torch.compile, CUDA Graphs, TensorRT, XLA | |
| IV · Training | Ch 10 | Distributed Training | 4D parallelism, ZeRO, RLHF/PPO, MoE training |
| Ch 11 | KV Cache | PagedAttention, radix attention, offloading, eviction | |
| Ch 12 | Benchmarking | MFU/HFU, MLPerf, TTFT/TPOT curves, regression CI | |
| V · Scale | Ch 13 | Speculative Decoding + MoE | Draft-verify loop, acceptance rate, expert routing |
| Ch 14 | Networking | NVLink, InfiniBand NDR, NCCL, AllReduce algorithms | |
| Ch 15 | Storage & I/O | NVMe, Lustre, DALI, checkpoint cadence | |
| Ch 16 | Cluster Management | SLURM, Kubernetes, gang scheduling, SDC, MIG | |
| Ch 17 | Telemetry & Tools | Nsight, DCGM, Prometheus/Grafana dashboards | |
| VI · Principal | Ch 18 | Career & System Design | STAR+Impact, ChatGPT system design walkthrough |
| Ch 19 | Production Resilience | Fault tolerance, SDC, recovery, fleet hardening | |
| Ch 20 | Future of AI Compute | Chiplets, optical interconnects, 2030+ roadmap | |
| Ch 21 | Real-World Case Studies | 70B, 405B, multimodal, MoE fleet deployments |