How to Implement Scaled Dot-Product & Causal Multi-Head Attention in PyTorch
Write numerically stable causal self-attention with 1/sqrt(d_k) variance scaling and tensor head splitting.
Scaled Dot-Product Attention computes pairwise token affinity via Q @ K.transpose(-2, -1) / sqrt(d_k), applies an upper-triangular -inf causal mask so token i cannot attend to future tokens j > i, normalizes rows with softmax, and computes a convex combination of Value vectors V.
Interactive Runnable CodeSandbox
Causal Multi-Head Attention Tensor Forward Pass
Causal Scaled Dot-Product Attention
If Q and K have independent unit-variance entries, their dot product has variance d_k; dividing by sqrt(d_k) restores unit variance.
Step-by-Step Implementation Breakdown
Fuse Q, K, V Projections into a Single 3 * d_model GEMM
Instead of launching 3 separate matrix multiplications, project X with a single nn.Linear(d_model, 3 * d_model) and chunk into Q, K, V.
Apply Causal Mask Before Softmax, Never After
Masking with -inf before softmax ensures the remaining valid positions re-normalize to sum to 1.0.
Use F.scaled_dot_product_attention for FlashAttention-2 Kernel Fusion
In PyTorch 2.x, F.scaled_dot_product_attention avoids materializing the O(N^2) attention matrix in HBM by tiling in SRAM.
Production Edge Cases & Interview Traps
- When an entire row is masked with -inf (e.g., padding + causal mask), softmax(-inf) produces NaN; always keep at least the BOS token unmasked.
- Always call .contiguous() after .transpose(1, 2) before .view(B, N, d_model) to avoid stride errors.
Frequently Asked Questions
Why does attention scale by 1/sqrt(d_k) instead of 1/d_k?
The variance of the sum of d_k independent random variables with unit variance is d_k, so the standard deviation is sqrt(d_k). Dividing by the standard deviation normalizes variance to 1.
How does Grouped-Query Attention (GQA) change the K and V shapes?
In GQA, K and V have fewer heads (n_kv_heads < n_heads) and are repeated across query head groups, shrinking the KV-cache memory footprint by 4x-8x.