Quantization & Precision Optimization

8.1 Floating Point Formats — The Precision Landscape

8.1.1 Number Formats from FP32 to FP4

Every floating point number uses three fields: a sign bit, an exponent (controls range), and a mantissa (controls precision within that range). The fundamental engineering tradeoff: bits given to the exponent widen representable range; bits given to the mantissa improve precision. Every AI format is a different point on this spectrum, chosen for a specific workload characteristic.

8.1.2 Why BF16 Replaced FP16 for Training

The replacement of FP16 by BF16 as the dominant training format was driven by one specific failure mode: overflow. Understanding this concretely is essential because the format choice affects every training run you will ever design.

8.1.3 The Numeric Format Decision Matrix

8.2 Quantization Mathematics

8.2.1 Uniform Affine Quantization — The Foundation

All practical quantization schemes build on a linear mapping from floating-point values to integers. Understanding this foundation precisely — including what fails and why — is the prerequisite for understanding every advanced technique in sections 8.3 and beyond.

8.2.2 Per-Tensor vs Per-Channel vs Per-Group Quantization

The granularity of scale factor computation is one of the most consequential quality-efficiency tradeoffs in quantization engineering. Each level of granularity costs more in metadata storage but recovers more quality.

8.2.3 The Outlier Problem — Why Transformers Break Naive Quantization

Transformer models contain a small but devastating minority of "outlier" activations — values 10–100× larger than typical. These outliers are the primary cause of quality degradation in naive quantization schemes, and every advanced PTQ technique (GPTQ, AWQ, SmoothQuant) exists specifically to address them.

8.3 Post-Training Quantization (PTQ) Techniques

PTQ applies quantization after training is complete, using a small calibration dataset (~128–512 samples) to compute scale factors. No model retraining is required — critical for fast deployment of large models. The three techniques below represent the state of the art for three distinct trade-offs between calibration cost, quality, and outlier handling.

8.3.2 GPTQ — Second-Order Weight Quantization

GPTQ (Frantar et al., 2022) is the breakthrough that made INT4 inference practical. It uses Hessian-based second-order information to compensate for each weight's quantization error in subsequent weights within the same row.

8.3.3 AWQ — Activation-Aware Weight Quantization

AWQ (Lin et al., 2023) protects the weights that matter most before quantization, rather than fixing errors after. It is generally faster to calibrate than GPTQ and performs comparably or better on models with pronounced outliers.

8.3.4 SmoothQuant — Enabling W8A8 Quantization

SmoothQuant (Xiao et al., 2022) enables W8A8 (both weights AND activations in INT8) by migrating the quantization difficulty from activations to weights. This matters when you want full INT8 Tensor Core utilization, not just weight compression.

8.4 FP8 and the NVIDIA Transformer Engine SHIPPED

The H100 Transformer Engine (TE) is NVIDIA's hardware+software co-design for automatic FP8 mixed precision. It is the simplest and most effective quantization approach on H100+ hardware: replace one import, wrap training in one context manager, achieve 2× GEMM throughput with less than 0.3% quality loss.

8.5 Weight-Only Quantization — W4A16 in Production

W4A16 quantizes weights to INT4 while keeping activations in BF16/FP16. This is the dominant format for memory-constrained inference — maximizing model size per GPU at acceptable quality loss. Understanding exactly when it helps (decode) and when it doesn't (prefill) is essential for production sizing decisions.

8.6 KV Cache Quantization

KV cache quantization is handled independently from weight quantization and deserves separate analysis. Unlike weights — which are static and can be calibrated offline — KV cache values are generated dynamically per request, requiring online quantization with per-head scale computation.

8.7 Quantization-Aware Training (QAT)

QAT trains the model WITH quantization simulation active, allowing weights to adapt to quantization error during training. It produces the highest quality quantized models but requires access to the training pipeline — not feasible for most deployments. It is primarily used by model developers before public release, and for NVFP4 on Blackwell where PTQ quality is insufficient.

8.8 Quantization Evaluation

Rigorous quantization evaluation requires four benchmark tiers. Many teams make the mistake of evaluating only MMLU, missing quality regressions in code generation or long-context coherence that automated benchmarks cannot catch.

8.9 Profiling Cookbook

# ———————————————————————————————————————————
# SYMPTOM: Quantization applied but MMLU dropped >2% — find the cause
# GOAL: Diagnose whether the issue is outliers, calibration, or granularity
# ———————————————————————————————————————————

# STEP 1: Baseline quality measurement
pip install lm-eval
lm_eval --model hf \
  --model_args pretrained=meta-llama/Meta-Llama-3-70B \
  --tasks mmlu,wikitext \
  --num_fewshot 5 \
  --output_path results_bf16.json

# STEP 2: Apply AWQ INT4 (recommended first choice)
pip install autoawq
python -c "
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model = AutoAWQForCausalLM.from_pretrained('meta-llama/Meta-Llama-3-70B')
tokenizer = AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-70B')

quant_config = { 'zero_point': True, 'q_group_size': 128, 'w_bit': 4 }
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized('./llama3-70b-awq-int4')
"

# STEP 3: Evaluate quantized model
lm_eval --model hf \
  --model_args pretrained=./llama3-70b-awq-int4 \
  --tasks mmlu,wikitext \
  --num_fewshot 5 \
  --output_path results_awq.json

# STEP 4: Compare results
python -c "
import json
bf16 = json.load(open('results_bf16.json'))
awq  = json.load(open('results_awq.json'))
mmlu_delta = awq['results']['mmlu']['acc'] - bf16['results']['mmlu']['acc']
ppl_bf16 = bf16['results']['wikitext']['word_perplexity']
ppl_awq  = awq['results']['wikitext']['word_perplexity']
print(f'MMLU delta: {mmlu_delta*100:.2f}%')
print(f'PPL: {ppl_bf16:.2f} (BF16) vs {ppl_awq:.2f} (AWQ)')
"
# Target: MMLU delta > -1.0%, PPL delta < 1.0
# If MMLU delta < -2.0%: check per-layer outlier analysis (below)
# If PPL delta > 2.0: increase group_size=64, or switch to GPTQ

# STEP 5: Outlier diagnosis per layer
python -c "
import torch
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained('meta-llama/Meta-Llama-3-70B',
          torch_dtype=torch.bfloat16, device_map='auto')
for name, param in model.named_parameters():
    if 'weight' in name:
        amax = param.abs().max().item()
        amean = param.abs().mean().item()
        ratio = amax / amean
        if ratio > 100:  # Extreme outliers
            print(f'HIGH OUTLIER RATIO: {name} | max={amax:.1f} mean={amean:.4f} ratio={ratio:.0f}x')
"
# Layers with ratio >100: need AWQ (not GPTQ) or larger group_size
# If many layers show high ratio: consider SmoothQuant + INT8 instead

# STEP 6: Throughput benchmarking with vLLM
python -m vllm.entrypoints.benchmark_throughput \
  --model ./llama3-70b-awq-int4 \
  --quantization awq \
  --kv-cache-dtype fp8 \
  --num-prompts 500 \
  --input-len 128 --output-len 512 \
  --max-num-seqs 32
# Expected: ~2,700-2,800 output tokens/sec on single H100
# Compare to BF16 baseline on 2 H100s: ~1,530 tokens/sec
# If throughput is LOWER after quantization: check INT8 GEMM kernel availability

Worked Example: LLaMA-3 70B from BF16 to Production — Full Cost Analysis