Back Matter · Appendix A
Hardware Quick Reference & Cheat Sheet
GPU specifications · HBM generations · Interconnect bandwidth · 30 numbers every principal knows cold · Key formulas · Roofline ridge points · Parallelism rules · Production thresholds
Book: AI/ML Infrastructure from Silicon to Scale | Author: Venkat Vinjam | Cross-ref: Ch 1: Roofline Ch 3B: GPU roadmap Ch 4: HBM deep dive
How to use this appendix
Every number here appears in at least one architecture review or interview. Know the starred ★ values cold — derive them from first principles, not memory. Specifications marked [SHIPPED] are from production hardware; [ANNOUNCED] are vendor-disclosed pre-GA; [ROADMAP] are directional estimates.
A.1 NVIDIA GPU Generation Comparison [SHIPPED]
| GPU | Arch | Node | BF16 TFLOPS | FP8 TFLOPS | FP4 TFLOPS | HBM Type | HBM Cap. | HBM BW | NVLink BW | TDP |
|---|
| V100 SXM2 | Volta | 12nm | 112 | N/A | N/A | HBM2 | 32 GB | 900 GB/s | 300 GB/s | 300W |
| A100 SXM4 | Ampere | 7nm | 312 | N/A | N/A | HBM2e | 80 GB | 2.0 TB/s | 600 GB/s | 400W |
| H100 SXM5 ★ | Hopper | 4N | 989 | 1,979 | N/A | HBM3 | 80 GB | 3.35 TB/s | 900 GB/s | 700W |
| H200 SXM5 | Hopper | 4N | 989 | 1,979 | N/A | HBM3e | 141 GB | 4.8 TB/s | 900 GB/s | 700W |
| B100 SXM | Blackwell | 4NP | 1,800 | 3,500 | 7,000 | HBM3e | 192 GB | 8.0 TB/s | 1,800 GB/s | 700W |
| B200 SXM | Blackwell | 4NP | 2,250 | 4,500 | 9,000 | HBM3e | 192 GB | 8.0 TB/s | 1,800 GB/s | 1,000W |
| GB200 NVL72 | Blackwell | 4NP | 2,250/GPU | 4,500/GPU | 9,000/GPU | HBM3e | 192 GB/GPU | 8.0 TB/s | 130 TB/s rack | 1,000W/GPU |
| GB300 NVL72 [ROADMAP] | Blk Ultra | 3nm est. | ~2,800 | ~6,000 | ~12,000 | HBM4 | 288 GB | 12.0 TB/s | 130+ TB/s | ~1,400W |
A.2 AMD GPU Comparison [SHIPPED]
| GPU | Arch | BF16 TFLOPS | FP8 TFLOPS | HBM Type | HBM Cap. | HBM BW | Fabric BW | TDP | Unique Advantage |
|---|
| MI250X | CDNA 2 | 383 | N/A | HBM2e | 128 GB | 3.28 TB/s | 800 GB/s | 500W | Dual-die, large memory |
| MI300X ★ | CDNA 3 | 1,307 | ~2,614 | HBM3 | 192 GB | 5.3 TB/s | 896 GB/s | 750W | Single-GPU 70B BF16 serving |
| MI325X [ANNOUNCED] | CDNA 3+ | 1,307 | ~2,614 | HBM3e | 256 GB | 6.0 TB/s | 896 GB/s | ~1,000W | 256 GB — largest available |
| MI355X [ANNOUNCED] | CDNA 4 | ~2,600 | ~5,200 | HBM3e | 288 GB | ~8.0 TB/s | ~1,500 GB/s | ~1,400W | CDNA 4 compute jump |
MI300X memory moat: 192 GB HBM3 fits LLaMA-3 70B (140 GB BF16) on a single GPU — H100/H200 require TP=2. At 100 serving instances that's 100 GPUs saved: $2.5M capex + $2.6M/year OpEx at $3/GPU-hr.
A.3 HBM Generation Reference
| Standard | BW / Stack | Cap / Stack | Pin Rate | Bus Width | GPU Examples |
|---|
| HBM2 | 256 GB/s | 8–16 GB | 2 Gbps | 1024-bit | V100, MI100 |
| HBM2e | 307 GB/s | 16 GB | 3.2 Gbps | 1024-bit | A100, MI250X |
| HBM3 ★ | 819 GB/s | 16 GB | 6.4 Gbps | 1024-bit | H100, MI300X |
| HBM3e | ~1.2 TB/s | 24 GB | 9.6 Gbps | 1024-bit | H200, B200, MI325X |
| HBM4 [ROADMAP] | ~2 TB/s | 32+ GB | ~16 Gbps | 1024-bit+ | GB300, MI400 |
A.4 Network Interconnect Reference [SHIPPED]
| Interconnect | BW (bidirectional) | Latency | Scope | AI Use Case | Constraint |
|---|
| NVLink 3.0 (A100) | 600 GB/s | <1 µs | 8-GPU intra-node | Tensor Parallelism | NVSwitch domain |
| NVLink 4.0 (H100) ★ | 900 GB/s | <1 µs | 8-GPU intra-node | TP, Sequence Parallelism | NVSwitch domain only |
| NVLink 5.0 (B200) | 1,800 GB/s | <1 µs | 8-GPU intra-node | TP, EP, SP | NVSwitch domain only |
| NVLink-C2C (GB200) | 10,000 GB/s | <1 µs | CPU↔GPU same chip | KV transfer, unified memory | Same superchip only |
| NVLink Switch (NVL72) | 130 TB/s (rack total) | <1 µs | 72-GPU rack | Full mesh TP/EP | One rack = one "compute" |
| InfiniBand HDR | 50 GB/s/port | ~1 µs | Inter-node | PP, DP, AllReduce | Not suitable for TP |
| InfiniBand NDR ★ | 100 GB/s/port | ~1 µs | Inter-node | PP, DP, KV transfer | Not suitable for TP |
| InfiniBand XDR | 200 GB/s/port | <1 µs | Inter-node | Next-gen AllReduce | Emerging 2025+ |
| PCIe Gen5 x16 | 128 GB/s | ~5 µs | CPU↔GPU | Model loading, checkpoints | 26× slower than HBM |
TP over InfiniBand is unusable for inference. TP=8 AllReduce, 70B model, 80 layers, 0.5 GB per AllReduce: NVLink (900 GB/s) = 44ms overhead; IB NDR (50 GB/s) = 800ms overhead. TP must stay within the NVSwitch domain — non-negotiable.
A.5 Key Capacity Planning Formulas ★
MODEL PARAMETER MEMORY:
model_bytes = N_params × bytes_per_param
LLaMA-3 70B BF16: 70×10⁹ × 2 = 140 GB
LLaMA-3 70B FP8: 70×10⁹ × 1 = 70 GB
LLaMA-3 70B INT4: 70×10⁹ × 0.5 = 35 GB
LLaMA-3 405B BF16: 405×10⁹ × 2 = 810 GB (requires TP=8 on H100, or 5× MI300X)
KV CACHE MEMORY (per request):
KV_bytes = 2 × L × H_kv × d_head × S × bytes_per_elem
LLaMA-3 70B (L=80, H_kv=8, d_head=128, S=4096, BF16):
= 2 × 80 × 8 × 128 × 4096 × 2 = 5.37 GB per concurrent request
Per token per layer: 2 × H_kv × d_head × 2 = 2×8×128×2 = 4 KB
TPOT MINIMUM (bandwidth-limited, batch=1):
TPOT_min = model_bytes / HBM_bandwidth
H100 BF16: 140 GB / 3.35 TB/s = 41.8 ms ★
H200 BF16: 140 GB / 4.8 TB/s = 29.2 ms
B200 BF16: 140 GB / 8.0 TB/s = 17.5 ms
MI300X BF16: 140 GB / 5.3 TB/s = 26.4 ms
H100 INT4: 35 GB / 3.35 TB/s = 10.4 ms (4× speedup from quantization)
PREFILL FLOPS (rule of thumb):
FLOPs_prefill ≈ 2 × N_params × S (sequence length)
LLaMA-3 70B, S=2048: 2 × 70×10⁹ × 2048 = 286 TFLOP
On H100 (989 TFLOPS, 50% MFU): 286 / (989 × 0.5) = 578 ms
MFU:
MFU = (FLOPs_per_step × steps/s) / Peak_TFLOPS
World-class: >55% | Good: 40–55% | Poor: <25% | Critical: <10%
ALLREDUCE TIME (ring algorithm):
t = 2 × (N−1)/N × M / BW [M = message bytes, N = GPUs, BW = link BW]
TP=8, M=256 MB, NVLink=900 GB/s: t = 2×(7/8)×256e6/900e9 = 0.50 ms
FLEET SIZING:
raw_GPUs = peak_tok/s / (1000/TPOT_ms × batch_at_SLA)
total_GPUs = raw_GPUs / 0.75 × 1.20 (75% util target, 20% burst headroom)
A.6 Parallelism Quick Reference
| Strategy | What is Sharded | Comm Pattern | BW Required | Scale Limit | Best For |
|---|
| Data Parallelism (DP) | Gradients only | AllReduce | IB acceptable | Model fits 1 GPU | High throughput training |
| ZeRO-1 | Optimizer states | AllGather before fwd | IB acceptable | Low comm overhead | Optimizer memory relief |
| ZeRO-3 / FSDP | Weights + optimizer + grads | AllGather + ReduceScatter | IB acceptable (3× comm) | Heaviest comm | Largest models, memory tight |
| Tensor Parallelism (TP) | Weight matrices | AllReduce per layer | NVLink mandatory | NVSwitch domain (≤8) | Inference latency, very large |
| Pipeline Parallelism (PP) | Layers across stages | P2P activations | IB acceptable | Bubble = (PP−1)/(PP+M−1) | Cross-node training scale |
| Sequence Parallelism (SP) | Sequence dim activations | AllGather + ReduceScatter | NVLink preferred | Requires TP | Long context training |
| Expert Parallelism (EP) | MoE expert weights | All-to-All tokens | IB+ acceptable | Load balance critical | MoE at scale |
A.7 Numerical Precision Reference
| Format | Bits | Exp | Mantissa | Range | GPU Support | Typical Use |
|---|
| FP32 | 32 | 8 | 23 | ±3.4×10³⁸ | All | Optimizer states, loss scaling accumulator |
| BF16 ★ | 16 | 8 | 7 | ±3.4×10³⁸ | A100+ | Training activations + weights (default) |
| FP16 | 16 | 5 | 10 | ±65,504 | V100+ | Legacy training; loss scaling required |
| FP8 E4M3 ★ | 8 | 4 | 3 | ±448 | H100+ | Forward pass weights + activations |
| FP8 E5M2 | 8 | 5 | 2 | ±57,344 | H100+ | Backward pass gradients |
| INT8 | 8 | — | — | −128 to 127 | All | LLM.int8(), SmoothQuant inference |
| INT4 / GPTQ | 4 | — | — | −8 to 7 | All | Memory-limited inference, AWQ |
| NVFP4 [SHIPPED] | 4 | 2 | 1 | block-scaled | B200+ | Blackwell-native training + inference |
A.8 The 30 Numbers Every Principal Knows Cold ★
COMPUTE:
H100 BF16: 989 TFLOPS H100 FP8: 1,979 TFLOPS
H200 BF16: 989 TFLOPS B200 FP8: 4,500 TFLOPS B200 FP4: 9,000 TFLOPS
MI300X BF16: 1,307 TFLOPS GB300 FP4: ~12,000 TFLOPS [ROADMAP]
HBM CAPACITY AND BANDWIDTH:
H100: 80 GB | 3.35 TB/s ★ H200: 141 GB | 4.8 TB/s
B200: 192 GB | 8.0 TB/s MI300X: 192 GB | 5.3 TB/s ★
GB300: 288 GB | 12.0 TB/s [ROADMAP]
RIDGE POINTS (FLOP/byte, the regime boundary):
A100: 156 H100: 295 ★ H200: 206 B200: ~281 MI300X: 247
INTERCONNECT:
NVLink 4.0 (H100): 900 GB/s ★ NVLink 5.0 (B200): 1,800 GB/s
NVLink-C2C (GB200): 10,000 GB/s IB NDR (per port): 50 GB/s ★
PCIe Gen5 x16: 64 GB/s (one-way GPU↔CPU)
MODEL MEMORY — LLAMA-3 70B:
BF16: 140 GB ★ FP8: 70 GB INT4: 35 GB
Single-GPU fit (BF16): MI300X only ★ (192 GB vs 140 GB needed)
LLaMA-3 405B BF16: 810 GB → min TP=8 on H100 (640 GB total)
TPOT MINIMUMS — LLAMA-3 70B, batch=1:
H100 BF16: 41.8 ms ★ H200 BF16: 29.2 ms
MI300X: 26.4 ms B200: 17.5 ms
H100 INT4: 10.4 ms ★ (4× speedup from quantization alone)
KV CACHE — LLAMA-3 70B, GQA, 4K context, BF16:
Per request: 5.37 GB ★ Per token per layer: 4 KB
PRODUCTION THRESHOLDS:
Max GPU utilization: 80% ★ Max KV cache: 80% ★
MFU world-class: >55% MFU poor: <25%
P99/P50 TTFT alarm: >5× ★ Queue time alarm: >200ms P99
SDC rate (10K H100): ~1/2 weeks Continuous batch gain: 10–30×
A.9 Roofline Ridge Points by GPU and Precision
| GPU | BF16 Ridge | FP8 Ridge | HBM BW | Key Implication |
|---|
| V100 | 124 FLOP/byte | N/A | 900 GB/s | Most transformers compute-bound (low ridge) |
| A100 | 156 FLOP/byte | N/A | 2.0 TB/s | Large GEMM still compute-bound |
| H100 ★ | 295 FLOP/byte | 590 FLOP/byte | 3.35 TB/s | Many ops now memory-bound vs V100 |
| H200 | 206 FLOP/byte | 412 FLOP/byte | 4.8 TB/s | Lower ridge — BW-rich, compute same |
| B200 | ~281 FLOP/byte | ~562 FLOP/byte | 8.0 TB/s | Higher compute + higher BW |
| MI300X ★ | 247 FLOP/byte | ~493 FLOP/byte | 5.3 TB/s | Lower ridge than H100 — bandwidth-rich |
H200 insight: H200 has a LOWER ridge (206) than H100 (295) despite identical TFLOPS — because HBM bandwidth increased without adding compute. Ops that are just barely compute-bound on H100 may become memory-bound on H200. Re-classify every kernel's regime at each GPU generation transition.
A.10 MFU Reference Table
| Workload | Model | GPU | MFU Range | Notes |
|---|
| LLM Pretraining (well-tuned) | LLaMA-3 70B | H100 | 45–52% | Megatron-LM, 4D parallel |
| LLM Pretraining | GPT-4 class TP=8 PP=16 | H100 | 40–48% | NVIDIA MLPerf Training |
| Inference Prefill (TRT-LLM) | LLaMA-3 70B, FP8, batch=32 | H100 | 50–65% | Compute-bound at large batch |
| Inference Decode (batch=1) | LLaMA-3 70B, BF16 | H100 | 3–8% | Bandwidth floor — expected and correct |
| Inference Decode (batch=64) | LLaMA-3 70B, BF16 | H100 | 30–45% | Approaching ridge point |
| RLHF / PPO | 70B policy + 70B reward | H100 | 15–30% | Memory constraints limit batch |
| Fine-tuning (LoRA r=64) | 7B | A100 | 25–40% | Small batch, frequent I/O |
Cross-References
Ch 1 Roofline model and three performance regimes — the ridge point numbers in §A.9 derive from Ch 1 §1.4 formulas.
Ch 3 / Ch 3B GPU silicon architecture and generation roadmap — specs in §A.1 match Ch 3B's detailed generation analysis.
Ch 4 HBM deep dive — §A.3 HBM generations connect to Ch 4's physical analysis of HBM3 TSV stacks and 1024-bit bus.
App D All formulas in §A.5 are reproduced with full derivations in Appendix D Performance Formulas.
Appendix A Complete · AI/ML Infrastructure from Silicon to Scale · Venkat Vinjam · 2026