Speculative Decoding & Mixture of Experts

Standard autoregressive decoding has a deep structural flaw: every token requires a full forward pass through the entire model, and these passes are sequential — token N cannot start until token N−1 is complete. At batch size 1, a 70B model on H100 runs at approximately 0.6% GPU compute utilization. The GPU is spending 99.4% of its time waiting for memory. This chapter covers the two most powerful techniques for attacking this problem from opposite directions.

Speculative decoding exploits the asymmetry between generation and verification. A small draft model proposes K tokens cheaply; the large target model verifies all K in a single parallel forward pass. If the draft is right often enough, you get K tokens per iteration instead of 1 — at a fraction of the cost increase. Mixture of Experts (MoE) restructures the model itself: instead of one dense FFN that all tokens pass through, MoE has N specialized experts, routing each token to only K of them. DeepSeek-V3 has 671B total parameters but processes each token through only ~37B — achieving frontier quality at LLaMA-3 70B active compute cost. Together, these techniques represent the production frontier for inference efficiency in 2024–2026.

13.1 Speculative Decoding — Exploiting Verification Asymmetry

13.1.1 Why LLM Decode Is Memory-Bound

Before understanding why speculative decoding works, you need to understand exactly why standard decoding is so inefficient. At batch size 1, each decode step requires loading the full model weights from HBM — once, to process exactly one token.

The conclusion is stark: at batch size 1, the H100 is 99.7% idle waiting for memory. Speculative decoding attacks this by ensuring that each time we load the full 140 GB model from memory, we extract more than one token of value from that load.

13.1.2 The Draft-Verify Loop

13.1.3 The Speedup Formula — Derived from First Principles

13.2 Draft Model Variants — Four Approaches

The core algorithm is the same for all variants; what differs is how draft tokens are generated. The tradeoff is memory overhead vs acceptance rate vs deployment complexity.

13.3 Mixture of Experts — Architecture and Routing

13.3.1 The Dense FFN Problem and MoE Solution

In a standard transformer, every token passes through the same Feed-Forward Network (FFN) at every layer. This is computationally straightforward but wasteful: a token about astrophysics and a token about cooking both activate the same FFN weights, even though those tasks require entirely different specialized knowledge. The MoE insight is to replace one monolithic FFN with N smaller specialized expert FFNs, routing each token to only the K most relevant ones.

13.3.2 Router Mechanics — How Tokens Are Assigned

13.3.3 Expert Collapse and Load Balancing

Without explicit balancing constraints, MoE training degenerates into expert collapse — a positive-feedback loop where the router concentrates tokens on a few experts, those experts improve fastest due to more gradient signal, and the router routes even more tokens to them. The end result: a model with 8 experts that behaves like a model with 1–2.

13.4 MoE Infrastructure — Expert Parallelism and All-to-All

13.4.1 Expert Parallelism: A New Dimension

Dense models use Tensor Parallelism (TP) and Pipeline Parallelism (PP) during inference. MoE introduces a third dimension: Expert Parallelism (EP), where different experts reside on different GPUs. This enables serving a 671B MoE model, but introduces a communication pattern that is fundamentally different from TP AllReduce — and in many ways more expensive.

13.4.2 The Small-Batch Expert Problem

Even with perfect NVLink placement, MoE inference at typical batch sizes faces a serious compute efficiency problem: each expert receives only a small fraction of the total batch.

13.5 Profiling Cookbook — Diagnosing in Production

# vLLM (0.5+): track via prometheus metrics
curl http://localhost:8000/metrics | grep speculative
# Look for: vllm:spec_decode_draft_acceptance_rate
# Healthy: ≥ 0.70 for chat, ≥ 0.85 for code
# Unhealthy: < 0.50 → spec decode likely hurting performance!

# Verify expected speedup:
alpha = 0.55; K = 5; c = 0.10
E_tokens = (1 - alpha**(K+1)) / (1 - alpha)  # = 1.87
speedup = E_tokens / (K*c + 1)               # = 1.87/1.5 = 1.25×
# If observed < formula → overhead in batch management or KV sync

# DCGM metrics for expert utilization:
dcgmi dmon -e 1001,1002,1006,1007 -d 1000 -c 10
# 1001: SM_ACTIVE, 1002: SM_OCCUPANCY, 1006: DRAM_ACTIVE, 1007: NVLink BW

# vLLM expert balance metrics:
curl http://localhost:8000/metrics | grep moe_expert
# vllm:moe_expert_load{expert_id="0"} 0.125  ← balanced (1/8)
# vllm:moe_expert_load{expert_id="4"} 0.450  ← HOT EXPERT!

# Python diagnostic script:
expert_loads = [0.10, 0.09, 0.11, 0.08, 0.45, 0.07, 0.05, 0.05]
mean_load = sum(expert_loads) / len(expert_loads)   # = 0.125
max_load  = max(expert_loads)                       # = 0.45
imbalance_ratio = max_load / mean_load              # = 3.6 → ALERT!
# Threshold: alert if imbalance_ratio > 2.0

13.6 Combining Speculative Decoding and MoE

Deploying spec decode on MoE targets (e.g., Mixtral 8×7B) introduces two complications: draft model selection and verification cost. The verification forward pass on the target now includes All-to-All routing for all K draft tokens simultaneously — but since they are batched as a single prefill, this is generally efficient.

13.7 Disaggregated Prefill-Decode — The Next Frontier

Speculative decoding optimizes decode throughput within a single GPU pool. A complementary technique — disaggregated prefill-decode — attacks TTFT and TPOT simultaneously by recognizing that prefill and decode have fundamentally different hardware requirements and should run on separate, specialized GPU pools.

13.8 Worked Example: Deploying Speculative Decoding for a Code Assistant

13.9 Expert Load Distribution — Visualizing Imbalance