"Inference is where your model meets reality — and where you pay for every engineering mistake made during training infrastructure design. TTFT is the promise you make to the user. TPOT is the promise you keep. This chapter is about not breaking either."
— LLM Serving Engineering Principle
Learning Objectives
Level 1: Explain what TTFT and TPOT measure, why prefill and decode are fundamentally different, and why static batching fails
Level 2: Calculate TPOT minimum from model size and HBM bandwidth; derive fleet sizing requirements from traffic projections and SLA targets
Level 3: Design a production inference architecture with disaggregated prefill-decode, quantize the KV transfer budget, and defend every architectural decision with cost and latency math
6.1 Prefill vs. Decode: The Fundamental Dichotomy
LLM inference is not a single computation — it is two fundamentally different operations that happen to run sequentially. Every production serving system is, at its core, a system for managing the tension between these two phases.
THE TWO PHASES:
PHASE 1 — PREFILL:
Input: Full prompt (S tokens) processed IN PARALLEL (one forward pass)
Output: First output token + KV cache for all S positions populated
GEMM shape: [S × d_model] × [d_model × 4d_model] — large, square-ish
Arithmetic Intensity: high (S × d_model >> model_bytes)
Bottleneck: COMPUTE (Tensor Core throughput)
Metric: TTFT — Time To First Token
Target: <200ms (chat), <50ms (code completion), <5s (batch API)
PHASE 2 — DECODE:
Input: One token (the last generated) per autoregressive step
Output: Next token logits + one new KV row appended per layer
GEMM shape: [1 × d_model] × [d_model × 4d_model] — extremely thin (GEMV)
Arithmetic Intensity: ~1 FLOP/byte (deeply memory-bandwidth-bound)
Bottleneck: HBM BANDWIDTH (weight loading)
Metric: TPOT — Time Per Output Token
Target: <30ms (comfortable), <50ms (acceptable)
TPOT THEORETICAL MINIMUM (from first principles):
TPOT_min = model_bytes / HBM_bandwidth
LLaMA-3 70B [SHIPPED]:
BF16 (140 GB) / 3.35 TB/s = 41.8 ms/token ← H100 physics floor
FP8 (70 GB) / 3.35 TB/s = 20.9 ms/token ← 2× speedup from quantization
INT4 (35 GB) / 3.35 TB/s = 10.4 ms/token ← 4× speedup (bandwidth only!)
These are minimum possible TPOT at batch=1. No software optimization can
exceed them — they are limited by the speed of loading weights from HBM.
The ONLY escapes: more HBM bandwidth (H200/MI300X), quantization, batching.
Figure 6.1 — Prefill vs. decode phases in LLM inference. Prefill processes all input tokens in parallel (compute-bound, fast). Decode generates tokens one at a time (memory-bandwidth-bound, slow). The TTFT captures the prefill time; TPOT is the per-token decode time. KV cache grows by 4 KB/token/layer for LLaMA-3 70B with GQA.
6.2 Continuous / In-Flight Batching
STATIC BATCHING FAILURE MODES:
Problem 1 — Padding waste:
Batch of 4 requests, max output = 200 tokens:
Request A: finishes at token 20 → padded 180 steps (90% waste)
Request B: finishes at token 50 → padded 150 steps (75% waste)
Request C: finishes at token 150 → padded 50 steps (25% waste)
Request D: finishes at token 200 → no padding
Wasted compute: (180+150+50) / (4×200) = 47.5%
Problem 2 — Head-of-line blocking:
New request E arrives at t=10ms; D still running at t=200ms.
E's queue wait: 190ms — purely wasted.
Combined GPU utilization: 40–55% (typical static batch system)
CONTINUOUS BATCHING (iteration-level scheduling, Orca 2022):
At every decode step: evict finished requests, admit new ones immediately.
No padding — each request at its exact current sequence length.
No head-of-line blocking — new requests admitted as slots open.
Throughput gain vs static: 10–30× (measured on real workloads)
This is the #1 performance improvement available to any inference system.
If your serving stack is not using continuous batching, start there.
6.3 vLLM and PagedAttention
PRE-vLLM KV CACHE ALLOCATION:
Must reserve contiguous memory for max_sequence_length upfront:
LLaMA-3 70B, max_seq=4096, BF16:
Per-request KV = 2 × 80 × 8 × 128 × 4096 × 2 = 5.37 GB reserved upfront
On H100 (80 GB), with 10 GB weights (TP=1 fails — need TP=2, but say 10GB overhead):
Actually 70B BF16 = 140 GB, so need TP=2, 80 GB per GPU, 35 GB weights per GPU
Available for KV: ~40 GB → 40 / 5.37 = 7 max concurrent requests
Average actual seq len = 512 → actual KV used = 0.67 GB per request
Fragmentation: (5.37 - 0.67) / 5.37 = 87.5% KV memory wasted!
PAGEDATTENTION (Kwon et al., 2023):
Inspiration: OS virtual memory paging
Block size: 16 tokens per block
Allocation: on-demand as tokens arrive (not upfront for max_len)
Memory efficiency:
Fragmentation: <5% (vs 87% naive)
Concurrent requests: ~5-10× more per GPU vs naive allocation
Prefix sharing: requests with same system prompt share KV blocks
System prompt of 2048 tokens: stored ONCE, pointer-shared across all requests
TTFT improvement: 50-80% for system-prompt-heavy workloads
Preemption: when GPU KV runs full → swap low-priority request KV to CPU DRAM
Preemption cost: ~1-2s (KV transfer via PCIe 128 GB/s is slow)
Prevention: target <80% KV utilization at all times
6.4 Disaggregated Prefill-Decode Architecture
CO-LOCATED INTERFERENCE PROBLEM:
In a co-located system (prefill + decode on same GPUs):
Long prefill (2048 tokens, H100, 70B): occupies GPU for ~80ms
During that 80ms: ALL decode requests on those GPUs STALL
Observed effect (real workload [REPRESENTATIVE]):
P50 TPOT: 25ms ← when no prefill interference
P99 TPOT: 180ms ← dominated by prefill spikes
P99/P50 ratio: 7.2× ← indicator of prefill-decode interference
DISAGGREGATED SOLUTION:
Prefill cluster: optimized for compute throughput (H100 preferred)
large batch prefill → high TFLOPS utilization
Decode cluster: optimized for HBM bandwidth (H200 or MI300X preferred)
stable TPOT with no prefill interference
KV transfer: after prefill → RDMA/NVLink-C2C to decode GPU
KV TRANSFER BANDWIDTH BUDGET:
LLaMA-3 70B, 2048-token prompt, INT8 KV:
Transfer size = 2 × 80 × 8 × 128 × 2048 × 1 byte = 2.68 GB
Over 2× IB NDR (100 GB/s total): 2.68 / 100 = 26.8 ms transfer
Over NVLink-C2C (GB200, 10 TB/s): 2.68 / 10,000 = 0.27 ms (!!)
Break-even fleet size: >100 GPU-equivalents (engineering overhead is real)
Cost savings at scale: 20–35% per token
NIXL library (NVIDIA, 2025): achieves 85–95% of theoretical link BW
Production deployments: LLaMA.cpp+RDMA, Mooncake, Splitwise
6.5 TensorRT-LLM: Production Compilation
TRT-LLM OPTIMIZATION PIPELINE:
Input: HuggingFace / PyTorch checkpoint
→ Quantization (FP8/INT4 via modelopt / AMMO)
→ TensorRT compilation:
Kernel fusion (attention + norm + residual)
CUDA Graph recording (fixed decode pipeline)
Layer-by-layer optimization for target GPU + precision
→ Engine file (.plan) tied to {GPU, precision, max_batch, max_seq}
→ Triton Inference Server integration
PERFORMANCE GAINS vs PyTorch eager [REPRESENTATIVE]:
Kernel fusion: 30-50% latency reduction
CUDA Graphs: 20-40% decode overhead reduction
FP8 quantization: ~2× throughput
Combined: 2-3× total throughput improvement
OPERATIONAL COSTS:
Build time: 5-30 minutes per engine (offline)
Engine portability: NOT portable across {GPU, precision, max_batch, max_seq}
Re-build triggers: GPU type change, precision change, batch config change
Partial remedy: Use --strongly_typed=False for broader batch range
6.6 SLA Engineering: TTFT, TPOT, and the Throughput-Latency Curve
Metric
Definition
Chat SLA
Code Complete SLA
Batch API SLA
TTFT
Time from request to first output token (queue + prefill)
SPECULATIVE DECODING:
1. Draft model (7B) generates K=5 candidates in ~2ms
2. Target model (70B) verifies all 5 in ONE forward pass (same cost as 1 token verify)
3. Accept longest prefix that matches target distribution (rejection sampling)
ACCEPTANCE RATE MATH (α = per-token acceptance rate):
Expected accepted tokens = (1 - α^(K+1)) / (1 - α)
At α=0.8, K=5: E[tokens] = (1 - 0.8^6) / 0.2 = (1-0.262)/0.2 = 3.69 tokens/verify
Effective speedup: 3.69× decode throughput
PRODUCTION GUIDANCE:
WHEN speculative decoding helps:
✓ Batch=1 or small batch (single user sessions)
✓ Predictable text (code, structured output, templates)
✓ α > 0.70 (monitor in production via vllm:spec_decode_acceptance_rate)
WHEN it hurts:
✗ Large batch (draft overhead not amortized; draft verification competes)
✗ Creative/diverse generation (α typically 0.3-0.5)
✗ Already memory-bandwidth-bound at high batch (adds compute overhead)
6.8 Prefix Caching and KV Reuse
PREFIX CACHING ECONOMICS:
System prompt = 4,096 tokens. No caching at 10,000 req/s:
Wasted prefill: 4,096 × 10,000 = 40.96 billion tokens/s of repeated compute
At H100, 2048-token prefill ~= 80ms → prefill overhead per request = 160ms
With RadixAttention / prefix caching (vLLM ≥0.4, SGLang):
System prompt KV computed ONCE → stored in block pool → pointer-shared
TTFT improvement: 50-80% for system-prompt-heavy workloads
IMPORTANT: Cache keyed on EXACT input token IDs only.
Temperature ≠ 0 does NOT prevent cache hits (sampling is post-logits)
Cached: same system prompt → hit regardless of temperature
NOT cached: different user messages with same prompt → partial match
6.9 Distributed Inference: TP and PP
Strategy
What
Comm Type
Comm BW Needed
Constraint
Tensor Parallelism (TP)
Split weight matrices column/row
AllReduce per layer
NVLink required (>500 GB/s)
Must stay within NVSwitch domain (≤8 GPUs H100)
Pipeline Parallelism (PP)
Split layers across stages
P2P activations
IB acceptable (50 GB/s)
Pipeline bubble = 1/PP_degree overhead
Data Parallelism
Replicate model, split requests
None at inference
N/A
Model must fit per replica
TP over InfiniBand vs NVLink: TP=8 on 70B, 80 layers, 0.5 GB AllReduce each: NVLink: 0.5/900=0.56ms/layer × 80 = 44ms overhead. IB NDR: 0.5/50=10ms/layer × 80 = 800ms overhead. IB-based TP is unusable for latency-sensitive inference — always use NVLink for TP.
6.10 Inference Observability — The 5 Metrics That Matter
PRODUCTION MONITORING CHECKLIST (Prometheus metrics):
1. vllm:request_queue_time_seconds (P99)
Signal: >200ms → insufficient GPU capacity or batch admission problem
2. vllm:gpu_kv_cache_usage_perc
Signal: >80% → approaching preemption territory; add GPUs or reduce max_seq
3. vllm:request_prompt_throughput_tokens_per_s + generation_throughput
Signal: compare to theoretical (model_flops / gpu_peak_tflops → MFU proxy)
4. DCGM_FI_DEV_GPU_UTIL (per GPU)
Signal: <70% sustained during peak → scheduling or admission control issue
5. vllm:time_to_first_token_seconds (P50, P95, P99)
P99/P50 > 5× → prefill-decode interference (→ consider disaggregation)
🔧 Diagnostic Cookbook — When P99 TTFT Spikes
Symptom: P99 TTFT is 5–15× higher than P50 TTFT (normal is <3×)
Check queue depth first:vllm:request_queue_time_seconds P99 > 200ms → GPU is undersized for peak. Add capacity or reduce max_seq.
Check for prefill-decode interference: If queue time is low but TTFT still spikes, long prefill requests are stalling decode. Solution: chunked prefill (preempt prefill after N tokens) or disaggregation.
Check for GPU health issues:DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL below expected → NVLink degradation. DCGM_FI_DEV_ECC_SBE_VOL_TOTAL rising → memory errors.
Key Takeaways
Core Concepts
Prefill is compute-bound; decode is memory-bandwidth-bound. These two phases compete for the same GPU resources. Every inference system architecture decision — batching strategy, disaggregation, speculative decoding — is a response to this fundamental dichotomy.
TPOT minimum = model_bytes / HBM_bandwidth. For LLaMA-3 70B BF16 on H100: 41.8ms. This is a physics floor — no software optimization crosses it. Only quantization (reduces bytes) or batching (increases AI) can improve it.
PagedAttention reduces KV fragmentation from 87% to <5% by allocating memory in 16-token blocks on demand. This converts wasted HBM capacity into concurrent request capacity — typically 5–10× more requests per GPU.
Cost & Productivity Impact
Continuous batching delivers 10–30× throughput improvement over static batching. This is not a kernel optimization — it is a scheduling change. A team still running static batching is leaving $500K–$5M/year in GPU time on the table at typical inference fleet sizes.
TRT-LLM delivers 2–3× throughput over PyTorch eager via kernel fusion and CUDA Graphs. The tradeoff: 5–30 min engine build time per GPU+precision+batch config. A fleet of mixed H100/H200 nodes requires 2× the engine count — plan for this operational overhead.
Prefix caching eliminates system prompt computation. At 10K req/s with a 4K-token system prompt: saves ~80ms TTFT per request, equivalent to adding 22% GPU capacity at the same throughput. Zero additional hardware required.
Principal-Level Insight
Disaggregated prefill-decode is the correct architecture when P99/P50 TTFT ratio exceeds 5× and queue time is below 200ms — this pattern indicates prefill-decode interference rather than capacity shortage. The breakeven fleet size is ~100 GPU-equivalents; below that, CUDA stream priority is sufficient.
TP scope is a NVLink constraint, not a software preference. TP=8 over IB NDR (50 GB/s) for 70B inference adds 800ms per sequence in AllReduce overhead — unusable. TP must stay within a NVSwitch domain. Pipeline Parallelism crosses nodes because it communicates activations (P2P, infrequent) rather than weight gradients (AllReduce, per-layer).
Fleet sizing requires five steps, not one. The most common mistake is computing raw_GPUs without applying the utilization target (0.75) or burst headroom (1.20). Operating at 100% utilization means P99 queue time explodes — plan for 80% maximum utilization as a fleet health invariant.
Review Questions
FoundationalBuild the mental model — no math required
Explain in plain language: why is decode slower than prefill for the same model, even though each decode step processes fewer tokens? What physical resource is the limiting factor in each phase?
Your team's inference system has P99 TTFT of 800ms and P50 TTFT of 80ms. Before looking at any code, what do you suspect is happening? Name three possible root causes and how you'd distinguish between them using Prometheus metrics.
What is PagedAttention, and what problem does it solve? Use the analogy of OS virtual memory to explain how it works without using technical jargon.
IntermediateApply and calculate — production context
LLaMA-3 70B (140 GB BF16) is served on H100 SXM5 (80 GB, 3.35 TB/s HBM) with TP=2. (a) Calculate TPOT_min at batch=1. (b) The traffic is 200 req/s, avg 300 output tokens, P99 TPOT SLA=25ms. At batch=32 (the maximum at the SLA), how many H100 pairs (TP=2 units) are needed? Apply utilization=75% and burst headroom=20%. (c) Total capex at $25K/GPU?
Your vLLM deployment shows: P99 TTFT=1.8s, P50 TTFT=120ms, queue time P99=30ms, KV cache util=62%, GPU util=78%. (a) Is this a capacity problem or a scheduling problem? (b) What is causing the TTFT spike? (c) What is the first action you take?
Speculative decoding with 7B draft, K=5, α=0.75. (a) Expected accepted tokens per verify step. (b) Effective decode speedup. (c) At batch=64, verify cost per step = 80ms (draft + verify). Draft generates 5 tokens in 5ms. What is the break-even α where speculative decoding stops helping?
Principal LevelDesign, decide, and defend — architecture reviews
Design a production inference architecture for LLaMA-3 70B serving 10K req/s at P99 TPOT <25ms and P99 TTFT <200ms. Average input: 500 tokens, output: 300 tokens. Evaluate and quantify: (a) monolithic vs. disaggregated, (b) BF16 vs. INT4 weights, (c) TP=1 (MI300X) vs. TP=2 (H100), (d) prefix cache hit rate assumption and its impact on fleet size. Show all math, state all assumptions.
A competitor claims their inference system achieves 40% lower TPOT than yours on LLaMA-3 70B at batch=32 BF16. You are using TRT-LLM with FP8. Design a systematic investigation: what benchmarks do you run, what metrics do you compare, what architectural differences do you probe, and what is the most likely explanation for the gap?