Performance Formulas Quick Reference

ROOFLINE FUNDAMENTALS: Arithmetic Intensity (AI) = FLOPs executed / Bytes read+written from HBM [FLOP/byte] Memory Ceiling: P_mem(AI) = AI × HBM_bandwidth Compute Ceiling: P_compute = Peak_TFLOPS Achievable Perf: P_max = min(P_mem, P_compute) = min(AI × BW, Peak) Ridge Point ★: AI_ridge = Peak_TFLOPS / HBM_bandwidth H100 BF16: 989 TFLOPS / 3.35 TB/s = 295 FLOP/byte H200 BF16: 989 TFLOPS / 4.8 TB/s = 206 FLOP/byte B200 BF16: 2250 TFLOPS / 8.0 TB/s = 281 FLOP/byte MI300X: 1307 TFLOPS / 5.3 TB/s = 247 FLOP/byte REGIME CLASSIFICATION: AI > AI_ridge → COMPUTE-BOUND (add TFLOPS to improve) AI < AI_ridge → MEMORY-BOUND (reduce bytes or add bandwidth to improve) WORKED EXAMPLE — LLaMA-3 70B GEMV (batch=1, single decode step): Operation: [1 × 8192] × [8192 × 8192] (one linear layer) FLOPs: 2 × 1 × 8192 × 8192 = 134.2 M FLOPs Bytes (BF16): (1×8192 + 8192×8192 + 1×8192) × 2 = 134.2 MB AI: 134.2M / 134.2M = 1.0 FLOP/byte → deeply MEMORY-BOUND Achieved perf: 1.0 × 3.35 TB/s = 3.35 TFLOPS (vs 989 peak — only 0.34% MFU!)

GEMM FLOP COUNT: FLOPs = 2 × M × K × N (multiply-accumulate = 2 FLOPs each) With bias add: FLOPs += M × N GEMM ARITHMETIC INTENSITY: Bytes read: A[M×K] + W[K×N] + C[M×N] written AI = 2MKN / ((MK + KN + MN) × bytes) Dominated by K and N when M is small (batch size): At M=1 (batch=1): AI ≈ 2K / (K + N) × (1/bytes) → AI ≈ 2 / (1/K + 1/N) / bytes At M=batch: AI ≈ 2 × batch / (2 + batch/K + batch/N) / bytes Crossover batch size to compute-bound on H100 (BF16, K=N=8192): AI_ridge × bytes ≈ batch → batch_crossover ≈ 295 × 2 = 590 (For H100: need batch ≥ ~295 to be compute-bound on typical GEMM) TENSOR CORE ALIGNMENT: BF16/FP8 GEMMs require dimensions ≡ 0 (mod 16) If d_head = 130 (not aligned): padded to 144 → 9.6% wasted throughput At 1000 GPUs × $3/hr × 8760 hrs/yr × 9.6% = $252K/year from misalignment

TRANSFORMER MODEL MEMORY: model_bytes = N_params × bytes_per_param Training full precision: N × (4 + 4 + 8) = 16 bytes/param (weights + grads + Adam) Training BF16 mixed: N × (2 + 4 + 8) = 14 bytes/param Inference BF16: N × 2 bytes/param Inference FP8: N × 1 byte/param Inference INT4: N × 0.5 bytes/param ZeRO-3 / FSDP: model_bytes / N_GPUs (fully sharded, each GPU holds 1/N) KV CACHE: per_request = 2 × L × H_kv × d_head × S × dtype_bytes per_token_per_layer = 2 × H_kv × d_head × dtype_bytes LLaMA-3 70B (L=80, H_kv=8, d_head=128, BF16): per_token_per_layer = 2 × 8 × 128 × 2 = 4,096 bytes = 4 KB at S=4096 context: 80 × 4096 × 4096 = 5.37 GB per request ★ at S=128K context: 80 × 128000 × 4KB = 40.0 GB per request (single MI300X limit) ACTIVATION MEMORY (training, per layer): activations ≈ batch × seq_len × d_model × bytes (simplified) With recomputation (gradient checkpointing): reduce by 1/(checkpt_freq) Flash Attention: activations ∝ O(S) not O(S²) — critical for long context HBM CAPACITY PLANNING (inference): usable_KV = HBM_total − model_bytes − framework_overhead max_concurrent = usable_KV / KV_per_request H100, 70B BF16, TP=2: (80 − 70 − 5) = 5 GB → 5 / 5.37 ≈ 0 (can't fit!) H100, 70B INT4, TP=1: (80 − 35 − 5) = 40 GB → 40 / 2.68 = 14 requests

TRAINING FLOPS: Forward pass FLOPs ≈ 2 × N_params × S (per sample) Backward pass FLOPs ≈ 2 × Forward (gradients + activations) Total per step (without recomputation): ≈ 6 × N_params × S × batch_size With gradient checkpointing: ≈ 8 × N_params × S × batch_size TRAINING THROUGHPUT: tokens/sec = (batch_size × seq_len × steps_per_sec) GPU_efficiency = tokens/sec / (N_GPUs × peak_tokens/sec_per_GPU) MFU (Model FLOP Utilization) ★: MFU = (6 × N_params × S × batch/step × steps/sec) / (N_GPUs × Peak_TFLOPS) LLaMA-3 70B training, H100, 1024 GPUs, 2.5M tokens/sec: MFU = (6 × 70e9 × 4096 × 2.5e6/4096) / (1024 × 989e12) = (6 × 70e9 × 2.5e6) / (1024 × 989e12) = 1.05e18 / 1.013e18 = 104%? ← must divide by batch Correct: MFU = (6 × 70e9 × tokens_per_sec) / (N_GPU × 989e12) = (6 × 70e9 × 2.5e6) / (1024 × 989e12) = 1.05e18 / 1013e15 ≈ 1.03 → Check: 45% MFU → 45% × 989e12 × 1024 / (6 × 70e9) = ~1.08M tokens/sec PIPELINE BUBBLE FRACTION: bubble = (PP − 1) / (PP + M − 1) [M = microbatches per pipeline fill] PP=4, M=32: bubble = 3/35 = 8.6% (acceptable) PP=8, M=8: bubble = 7/15 = 46.7% (catastrophic — increase microbatches) GRADIENT CHECKPOINT TRADEOFF: Memory saved: (1 − 1/k) × activation_memory [k = checkpointing frequency] Compute added: ≈ 33% extra FLOPs (one extra forward pass per 3 layers) Cost: ~1/3 throughput penalty to enable 3× larger batch or model

TTFT (Time To First Token): TTFT = queue_wait + prefill_time prefill_time ≈ FLOPs_prefill / (N_GPUs × Peak_TFLOPS × MFU_prefill) ≈ (2 × N_params × S_input) / (N_GPUs × TFLOPS × 0.5) TPOT (Time Per Output Token): TPOT_min = model_bytes / (N_GPUs × HBM_bandwidth) ← physics floor ★ TPOT_actual ≥ TPOT_min (always bandwidth-limited at small batch) At large batch (batch > ridge_point): TPOT ≈ (2 × N_params × batch) / (N_GPUs × Peak_TFLOPS) END-TO-END LATENCY: E2EL = TTFT + TPOT × (output_tokens − 1) For chat (200 output tokens, TTFT=150ms, TPOT=25ms): E2EL = 150 + 25 × 199 = 150 + 4975 = 5,125 ms ≈ 5.1s THROUGHPUT vs LATENCY TRADEOFF: throughput (tok/s) = batch × (1000 / TPOT_ms) latency ↑ as batch ↑ (queuing delay, larger TPOT at higher AI) Operating point: max throughput subject to TPOT ≤ SLA_target SPECULATIVE DECODING SPEEDUP: E[accepted_tokens] = (1 − α^(K+1)) / (1 − α) [α = acceptance rate, K = draft tokens] speedup ≈ E[accepted_tokens] (relative to one target step per token) At α=0.8, K=5: E = (1 − 0.8^6) / 0.2 = (1 − 0.262) / 0.2 = 3.69× speedup ★ FLEET SIZING: raw_GPUs = peak_tok_per_s / (1000/TPOT_target_ms × batch_at_SLA) prod_GPUs = raw_GPUs / 0.75 × 1.20 [75% util, 20% burst headroom]

ATTENTION FLOPS (scaled dot-product, per head): QK^T: 2 × S × d × S = 2S²d FLOPs AV: 2 × S × S × d = 2S²d FLOPs Total per head: 4S²d FLOPs ATTENTION ARITHMETIC INTENSITY (standard attention): AI ≈ d/3 [independent of S! — always memory-bound for reasonable d] At d=128: AI ≈ 43 FLOP/byte (vs H100 ridge 295) → always memory-bound FlashAttention eliminates S² HBM writes → AI increases, latency decreases PAGEDATTENTION MEMORY EFFICIENCY: naive_waste = (max_seq − avg_seq) / max_seq At max_seq=4096, avg_seq=512: waste = (4096−512)/4096 = 87.5%! PagedAttention fragmentation: <5% (16-token blocks, on-demand allocation) Capacity gain: ~5−10× more concurrent requests KV QUANTIZATION SAVINGS: INT8 KV: 5.37 GB → 2.69 GB per request (2× more concurrent users) FP8 KV: 5.37 GB → 2.69 GB per request INT4 KV: 5.37 GB → 1.34 GB per request (4× more concurrent users) CROSSOVER — when attention FLOPs exceed FFN FLOPs: 4S²d_head × H_q > 3 × 2 × d_model × d_ffn (per layer, per token) LLaMA-3 70B: 4 × S² × 128 × 64 > 6 × 8192 × 28672 32768 S² > 1,409M → S > 6557 tokens (crossover at ~6.5K context)

ALLREDUCE (ring algorithm): t = 2 × (N−1)/N × M / BW [M = message bytes, N = GPUs, BW = link BW] TP=8, M=256 MB (one layer gradient), NVLink 900 GB/s: t = 2 × (7/8) × 256e6 / 900e9 = 0.50 ms per AllReduce TP=8 over IB NDR (50 GB/s): t = 2 × 7/8 × 256e6 / 50e9 = 8.96 ms → 80 layers × 8.96ms = 717ms per decode step ← unusable ★ ALLGATHER / REDUCESCATTER (ZeRO-3 / FSDP): t_allgather = (N−1)/N × M / BW (half of AllReduce) ZeRO-3 per step: 2× AllGather (fwd + bwd) + 1× ReduceScatter = 3 AllReduces BANDWIDTH UTILIZATION BUS BW (for benchmarking): busBW = (size / t) × (2×(N−1)/N) (what nccl-tests reports) Note: busBW > point-to-point BW — this is intentional, not a bug ALL-TO-ALL (MoE expert routing): t = M / BW × (N−1)/N × 2 (similar to AllReduce for balanced routing) At batch=32, d_model=7168, N_expert=8, IB NDR (50 GB/s): M = 32 × 7168 × 2 bytes × 7/8 = 401 KB → t ≈ 8µs (negligible)

TOKENS PER DOLLAR: cost_per_token = GPU_cost_per_hr / (throughput_tokens_per_hr) At TPOT=25ms, batch=32, 1 GPU: 32 tokens/25ms = 1280 tok/s At $3/GPU-hr: $3 / (1280 × 3600) = $6.5×10⁻⁷ per token = $0.65/million tokens TRAINING COST: GPU_hours = (6 × N_params × N_tokens) / (N_GPUs × TFLOPS × MFU × 3600) At 45% MFU, H100 989 TFLOPS, 1024 GPUs, 70B×1T tokens: = (6 × 70e9 × 1e12) / (1024 × 989e12 × 0.45 × 3600) = 261,800 GPU-hours At $3/hr: $785K per 1T token training run (70B BF16, 1024×H100, 45% MFU) TCO (3-year, per GPU): capex_amort = GPU_cost / (3 × 8760 × utilization) [per GPU-hour] H100 at $25K, 80% util: $25K / (3 × 8760 × 0.80) = $1.19/GPU-hr capex Total with $2/hr power+rack: ~$3.19/GPU-hr full TCO PUE IMPACT: actual_power_cost = IT_power × PUE × $/kWh At PUE=1.2, $0.05/kWh, 700W H100: 700W × 1.2 × $0.05 = $0.042/hr ≈ $0.04/GPU-hr Air cooling (PUE=1.4) vs liquid (PUE=1.15): saves 18% power per token