Profiling and Performance Tool Quick Reference

"Know which tool answers which question before you touch a keyboard. The wrong tool wastes hours. The right tool answers in minutes."

This appendix is a field guide, not a tutorial. Every section answers the question: given a specific symptom, what do I run, what do I look at, and what does the number mean? Detailed explanations of each tool live in Chapter 17 (Telemetry and Observability); use this appendix as your production-desk quick reference.

B.1 Tool Selection by Diagnostic Question

The single most important skill in production debugging is tool selection. Using ncu to diagnose an AllReduce hang, or nsys to investigate a kernel micro-optimization, costs 30–60 minutes per mistake. Use this table as your first lookup.

B.2 Nsight Systems (nsys) — System-Level Timeline

Core Commands

# ── STANDARD TRAINING PROFILE ─────────────────────────────────────────────────
nsys profile \
  --trace=cuda,nvtx,nccl,osrt \       # CUDA kernels + NVTX markers + NCCL + OS runtime
  --sample=cpu \                       # CPU call-stack sampling
  --output=/tmp/profile_%p \          # Separate file per rank (%p = PID)
  --duration=30 \                      # 30-second capture window
  python train.py

# ── MINIMAL (lowest overhead) ──────────────────────────────────────────────────
nsys profile --trace=cuda --output=/tmp/trace python script.py

# ── WITH NVTX MARKERS (requires torch.cuda.nvtx.mark() in code) ────────────────
nsys profile --trace=cuda,nvtx --output=/tmp/trace python train.py

# ── ATTACH TO RUNNING PROCESS ─────────────────────────────────────────────────
nsys profile -p $(pgrep python) --duration=30 --output=/tmp/attached

# ── OPEN IN GUI ───────────────────────────────────────────────────────────────
nsys-ui /tmp/profile_12345.nsys-rep

# ── CLI STATISTICS SUMMARY ────────────────────────────────────────────────────
nsys stats /tmp/profile_12345.nsys-rep --report gputrace
# Output: top kernels by duration (% of total GPU time)

What to Look for in the Timeline

NVTX Instrumentation (Add to Training Loop)

import torch

# Mark iteration boundaries — nsys shows step time breakdown
for step, batch in enumerate(dataloader):
    torch.cuda.nvtx.range_push(f"step_{step}")

    torch.cuda.nvtx.range_push("data_to_device")
    inputs = batch.to("cuda")
    torch.cuda.nvtx.range_pop()

    torch.cuda.nvtx.range_push("forward")
    loss = model(inputs)
    torch.cuda.nvtx.range_pop()

    torch.cuda.nvtx.range_push("backward")
    loss.backward()
    torch.cuda.nvtx.range_pop()

    torch.cuda.nvtx.range_push("optimizer")
    optimizer.step(); optimizer.zero_grad()
    torch.cuda.nvtx.range_pop()

    torch.cuda.nvtx.range_pop()   # step_N

B.3 Nsight Compute (ncu) — Kernel-Level Hardware Counters

Core Commands

# ── FULL KERNEL ANALYSIS ──────────────────────────────────────────────────────
ncu \
  --kernel-name "flash_attention_fwd" \  # Target kernel (regex match)
  --launch-skip 5 \                       # Skip 5 warmup launches
  --launch-count 3 \                      # Profile next 3 launches
  --set full \                            # All hardware counters
  --section SpeedOfLight_RooflineChart \  # Roofline analysis
  --section MemoryWorkloadAnalysis \      # Memory hierarchy breakdown
  -o /tmp/kernel_profile \               # Output file
  python infer.py

# ── QUICK ROOFLINE CHECK ──────────────────────────────────────────────────────
ncu --kernel-name X --set roofline --launch-count 1 python script.py

# ── TENSOR CORE UTILIZATION (key metric for GEMM health) ─────────────────────
ncu --metrics sm__pipe_tensor_cycles_active.avg.pct_of_peak_sustained_elapsed \
    --kernel-name matmul python script.py

# ── OPEN RESULT IN GUI ────────────────────────────────────────────────────────
ncu-ui /tmp/kernel_profile.ncu-rep

# ── COMPARE TWO PROFILES (regression detection) ───────────────────────────────
ncu-ui --diff baseline.ncu-rep current.ncu-rep
# Red = regression, Green = improvement

# ── CSV OUTPUT FOR CI PIPELINES ───────────────────────────────────────────────
ncu --kernel-name X --csv --metrics dram__bytes_read.avg,sm__throughput.avg \
    python script.py > /tmp/metrics.csv

Key Metrics and Interpretation

B.4 DCGM — Fleet GPU Health and the Critical Field IDs

Essential dcgmi dmon Commands

# ── PRODUCTION HEALTH MONITOR (run continuously) ──────────────────────────────
dcgmi dmon \
  -e 203,1002,1003,1004,1005,1009,100,155,1001 \
  -d 1000 \           # 1-second interval
  -c 3600             # 1-hour capture (drop -c for continuous)

# ── QUICK SPOT-CHECK (all GPUs on node) ───────────────────────────────────────
nvidia-smi dmon -s pucvmet -d 1 -c 30

# ── REMOTE FLEET MONITORING ──────────────────────────────────────────────────
dcgmi group -c training_group
dcgmi group -g 1 -a all
dcgmi dmon -g 1 -e 1003,1004,1005,1009 -d 1000

Complete DCGM Field ID Reference

Clock Throttle Reason Bitmask Decode

# Field 252 bitmask — decode to find WHY clocks dropped:
0x0000000000000001  HW_SLOWDOWN             GPU too hot — reduce workload or improve cooling
0x0000000000000002  SW_POWER_CAP            Power cap set below workload demand (check nvidia-smi -q -d POWER)
0x0000000000000004  HW_POWER_BRAKE_SLOWDOWN Power from upstream source (PSU/switch) limited
0x0000000000000008  SYNC_BOOST              Clock sync across multiple GPUs (normal in some configs)
0x0000000000000010  SW_THERMAL_SLOWDOWN     SW thermal limit reached (85°C default)
0x0000000000000020  HW_THERMAL_SLOWDOWN     HW thermal emergency — immediately reduce workload
0x0000000000000040  DISPLAY_CLOCKS_SETTING  Display active (remove from compute node)

# Prometheus query: find all throttled GPUs
DCGM_FI_DEV_CLOCK_THROTTLE_REASONS > 0

XID Error Reference (Most Common)

# Monitor XID errors in production:
dmesg -T | grep -i "NVRM.*Xid"
# or via DCGM:
dcgmi dmon -e 160 -d 1000 -c 60    # XID counter (field 160)

SM Activity Diagnostic Matrix

Prometheus Alert Rules for DCGM

# /etc/prometheus/rules/gpu_alerts.yml

groups:
  - name: gpu_health
    rules:
      - alert: GPUThermalThrottle
        expr: DCGM_FI_DEV_CLOCK_THROTTLE_REASONS > 0
        for: 5m
        labels: { severity: warning }
        annotations:
          summary: "GPU {{ $labels.gpu }} throttling on {{ $labels.Hostname }}"

      - alert: GPUXIDError
        expr: DCGM_FI_DEV_XID_ERRORS > 0
        for: 1m
        labels: { severity: critical }
        annotations:
          summary: "XID error on {{ $labels.Hostname }} GPU {{ $labels.gpu }}"

      - alert: GPUHighTemperature
        expr: DCGM_FI_DEV_GPU_TEMP > 83
        for: 5m
        labels: { severity: warning }

      - alert: GPUUncorrectableECC
        expr: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0
        for: 1m
        labels: { severity: critical }

      - alert: GPUSMUnderutilized
        expr: DCGM_FI_PROF_SM_ACTIVE < 50
        for: 10m
        labels: { severity: warning }
        annotations:
          summary: "SM utilization < 50% for 10min on {{ $labels.Hostname }}"

B.5 NCCL — Collective Communication Reference

Essential NCCL Environment Variables

nccl-tests — Cluster Health Benchmark

# ── BUILD (one time) ──────────────────────────────────────────────────────────
git clone https://github.com/NVIDIA/nccl-tests
cd nccl-tests
make MPI=1 MPI_HOME=/usr/local/mpi CUDA_HOME=/usr/local/cuda

# ── ALLREDUCE SWEEP (most important test) ─────────────────────────────────────
mpirun -np 8 --bind-to none \
  ./build/all_reduce_perf \
  -b 1M -e 4G -f 2 \     # message sizes 1MB → 4GB, 2× steps
  -g 1                    # 1 GPU per process

# ── MULTI-NODE ALLREDUCE (8 nodes × 8 GPUs = 64 GPUs) ─────────────────────────
mpirun -np 64 \
  -H node001:8,node002:8,...,node008:8 \
  --mca btl_openib_if_include mlx5_0:1 \
  ./build/all_reduce_perf -b 256M -e 4G -f 2 -g 1

# ── EXAMPLE HEALTHY OUTPUT ───────────────────────────────────────────────────
#   #                                                              out-of-place
#   #       size         count    type   redop    root    time   algbw   busbw
#   #        (B)    (elements)                             (us)  (GB/s)  (GB/s)
#    268435456      67108864   float     sum      -1    1423.7  188.5   188.5
# 
# H100 × 8, NVLink healthy baseline:
#   256MB  busbw ≥ 160 GB/s
#   1GB    busbw ≥ 180 GB/s
#   ALERT: busbw < 140 GB/s → NVLink degradation → check DCGM field 1011/1012
#
# H100 × 64, IB NDR 400G healthy baseline:
#   256MB  busbw ≥ 40 GB/s
#   1GB    busbw ≥ 45 GB/s
#   ALERT: busbw < 35 GB/s → IB degradation → check ibstatus

# ── TOPOLOGY DEBUGGING ────────────────────────────────────────────────────────
NCCL_TOPO_DUMP_FILE=/tmp/topo.xml python -c \
  "import torch.distributed as dist; dist.init_process_group('nccl')"
cat /tmp/topo.xml
# Verify each GPU-i connects to same leaf switch (rail-optimized topology)

# ── ISOLATE HANG ──────────────────────────────────────────────────────────────
# If training hangs, dump thread stacks to find stuck ranks:
py-spy dump --pid $(pgrep -f "train.py")
# Look for threads stuck in ncclAllReduce → rank is hanging

busbw vs algbw — Which Number Matters?

B.6 vLLM Inference Metrics Reference

Start vLLM with Metrics Endpoint

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-70b-instruct \
  --tensor-parallel-size 2 \
  --quantization fp8 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --port 8000

# Pull metrics:
curl http://localhost:8000/metrics | grep vllm:

vLLM Metric Reference with Alert Thresholds

B.7 PromQL Patterns for AI Cluster Monitoring

GPU Compute Queries

# ── CLUSTER-WIDE SM UTILIZATION AVERAGE ───────────────────────────────────────
avg(DCGM_FI_PROF_SM_ACTIVE) by (Hostname)

# ── FIND UNDERUTILIZED GPUs (potential stragglers) ────────────────────────────
DCGM_FI_PROF_SM_ACTIVE < 50

# ── TENSOR CORE UTILIZATION PER GPU ─────────────────────────────────────────
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE

# ── GPU POWER DRAW ACROSS FLEET ───────────────────────────────────────────────
DCGM_FI_DEV_POWER_USAGE

# ── MEMORY BANDWIDTH PRESSURE ────────────────────────────────────────────────
DCGM_FI_PROF_DRAM_ACTIVE > 0.95   # consistently near 100% = BW bottleneck

Thermal and Health Queries

# ── HOT GPUs ACROSS FLEET ─────────────────────────────────────────────────────
DCGM_FI_DEV_GPU_TEMP > 80

# ── CLOCK THROTTLE ACTIVE ─────────────────────────────────────────────────────
DCGM_FI_DEV_CLOCK_THROTTLE_REASONS > 0

# ── ECC CORRECTABLE ERROR RATE (rolling 1hr window) ──────────────────────────
rate(DCGM_FI_DEV_ECC_SBE_VOL_TOTAL[1h])

# ── ANY UNCORRECTABLE ECC (critical) ─────────────────────────────────────────
DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0

# ── ECC ERRORS PER NODE IN LAST 24H ──────────────────────────────────────────
sum by (Hostname) (increase(DCGM_FI_DEV_ECC_SBE_VOL_TOTAL[24h]))

# ── FREE GPU MEMORY (low = risk of OOM) ──────────────────────────────────────
DCGM_FI_DEV_FB_FREE < 5000    # in MiB

Inference Service Queries

# ── TTFT P99 LATENCY (5-minute window) ───────────────────────────────────────
histogram_quantile(0.99,
  rate(vllm:time_to_first_token_seconds_bucket[5m])
)

# ── TPOT P50 LATENCY ─────────────────────────────────────────────────────────
histogram_quantile(0.50,
  rate(vllm:time_per_output_token_seconds_bucket[5m])
)

# ── CURRENT TOKENS/SECOND THROUGHPUT ─────────────────────────────────────────
rate(vllm:generation_tokens_total[1m])

# ── KV CACHE UTILIZATION (alert if > 82%) ────────────────────────────────────
vllm:gpu_cache_usage_perc * 100 > 82

# ── REQUEST QUEUE DEPTH ──────────────────────────────────────────────────────
vllm:num_requests_waiting

# ── INFERENCE ERROR RATE ─────────────────────────────────────────────────────
rate(vllm:request_success_total[5m]) / rate(vllm:request_count_total[5m])

Training Queries

# ── MFU PER TRAINING JOB ─────────────────────────────────────────────────────
training_mfu

# ── MFU DROP DETECTION (alert if drops > 5 pp in 10 min) ────────────────────
training_mfu - training_mfu offset 10m > 0.05

# ── STEP TIME P99 (straggler detection) ───────────────────────────────────────
histogram_quantile(0.99, rate(training_step_seconds_bucket[5m]))

# ── ALLREDUCE LATENCY TREND ───────────────────────────────────────────────────
avg_over_time(allreduce_time_ms[5m])

# ── GRADIENT NORM SPIKE DETECTION ────────────────────────────────────────────
train_grad_norm > 10     # potential training instability

PromQL Pattern Reference

B.8 AMD Tool Equivalents (MI300X / MI355X)

AMD rocprof Quick Reference

# ── SYSTEM TIMELINE (equivalent to nsys) ──────────────────────────────────────
rocprof --sys-trace --output-dir /tmp/rocprof_out python train.py

# ── KERNEL STATISTICS ─────────────────────────────────────────────────────────
rocprof --stats python train.py
# Output: top kernels by wall time — equivalent to nsys stats gputrace

# ── HARDWARE COUNTERS ─────────────────────────────────────────────────────────
cat > /tmp/counters.txt << EOF
SQ_WAVES
GRBM_COUNT
TCC_HIT_sum
TCC_MISS_sum
EOF
rocprof --counters /tmp/counters.txt python train.py

# ── OMNIPERF ROOFLINE ANALYSIS ────────────────────────────────────────────────
omniperf profile -n my_run -- python train.py
omniperf analyze -p workloads/my_run/MI300_A1 --roof-only

# ── AMD-SMI FLEET HEALTH ──────────────────────────────────────────────────────
amd-smi monitor -p -t -u -m    # power, temp, utilization, mem
rocm-smi --showallinfo          # comprehensive single-GPU status

B.9 Storage and I/O Diagnostic Commands

Checkpoint I/O Diagnostics

# ── MEASURE ACTUAL CHECKPOINT WRITE SPEED ─────────────────────────────────────
# Time a checkpoint save directly:
time python -c "
import torch
model = ... # your model
torch.save(model.state_dict(), '/path/to/checkpoint/ckpt.pt')
"
# A 70B BF16 model checkpoint = ~140 GB
# Expected: NFS 10GbE = ~60 sec; Lustre 200 GB/s = ~0.7 sec

# ── FIO SEQUENTIAL BENCHMARK (measure storage peak throughput) ───────────────
# Sequential write (checkpoint scenario):
fio --name=seqwrite --ioengine=libaio --direct=1 \
    --rw=write --bs=4M --numjobs=16 \
    --size=10G --runtime=60 --group_reporting \
    --filename=/mnt/scratch/testfile
# Target for Lustre with 10+ OSS nodes: > 50 GB/s aggregate write

# Sequential read (dataset prefetch scenario):
fio --name=seqread --ioengine=libaio --direct=1 \
    --rw=read --bs=1M --numjobs=32 \
    --size=10G --runtime=60 --group_reporting \
    --filename=/mnt/scratch/testfile

# ── LUSTRE FILESYSTEM HEALTH ──────────────────────────────────────────────────
lfs df -h /mnt/scratch           # Capacity per OST
lfs df -i /mnt/scratch           # Inode usage per MDT
lfs check servers                 # OSS/MDS connectivity
lfs getstripe /path/to/file      # File stripe layout
lfs setstripe -c 64 /path/to/dir # Set 64-OST stripe for large files

# Lustre stats:
cat /proc/fs/lustre/llite/*/stats  # Client I/O statistics

# ── GPFS (IBM Spectrum Scale) HEALTH ─────────────────────────────────────────
mmlsfs gpfs0 -T         # Filesystem configuration
mmfsck gpfs0 -n         # Check without repair
mmpmon fs_io_s          # I/O statistics

# ── IDENTIFY DATA LOADING BOTTLENECKS IN PYTHON ───────────────────────────────
# Add instrumentation to DataLoader:
import time

class TimedDataLoader:
    def __init__(self, loader):
        self.loader = loader
        self.wait_times = []

    def __iter__(self):
        t0 = time.perf_counter()
        for batch in self.loader:
            wait = time.perf_counter() - t0
            self.wait_times.append(wait)
            yield batch
            t0 = time.perf_counter()

# If avg_wait_ms > 10ms → data loading is bottlenecking GPU

# ── NETWORK STORAGE (NFS) TUNING ──────────────────────────────────────────────
# Mount options for checkpoint performance:
# nfs4 rsize=1048576,wsize=1048576,hard,intr,rw,async
# Check current NFS stats:
nfsstat -c              # Client statistics
mountstats /mnt/nfs     # Detailed per-mount stats

Storage Health Checklist

B.10 Symptom-to-Command Quick Reference

This is the highest-frequency lookup table in this appendix — use it when you have a symptom but haven't yet identified the tool.

Tool Landscape Overview

Cross-Chapter References