Attention Geometry: Softmax Kernel Smoothing & Query-Key Subspace Routing
Visualize Multi-Head Attention as differentiable dictionary lookup and low-rank bilinear form W_Q W_K^T.
Each attention head h parameterizes two coupled low-rank circuits: the QK circuit W_Q^h (W_K^h)^T in R^{d_model x d_model} that determines WHICH source tokens are attended to, and the OV circuit W_V^h W_O^h in R^{d_model x d_model} that determines WHAT information is moved to the residual stream.
Think of the residual stream x_i as a shared high-dimensional communication bus. Each attention head reads a low-dimensional subspace via W_Q and W_K, computes spherical inner-product similarity, and writes the weighted value projection back into the residual stream via W_V W_O.
Attention QKV Subspace Geometry (AttentionSubspaceCircuitScene)
Step-by-Step Symbolic LaTeX Derivation
Follow each algebraic transformation from first principles to computational implementation.
Factored QK & OV Circuit Decomposition
Both W_QK and W_OV are rank-at-most-d_k matrices operating on the d_model residual stream.
Geometrically, where must the output of a single attention head's weighted sum sum_j alpha_{ij} v_j lie?
Why do Transformers use multiple heads instead of a single full-rank attention layer?
Multiple heads allow the model to attend to multiple distinct token positions and syntactic/semantic relationships simultaneously across orthogonal subspaces.