How to Compute Batch Cosine Similarity & Top-K Vector Retrieval for RAG
L2-normalize embedding matrices so cosine similarity reduces to a single hardware-accelerated matrix multiplication.
Cosine similarity measures the cosine of the angle between two vectors: (u . v) / (||u||_2 ||v||_2). In production RAG systems, pre-normalizing all document embeddings to unit L2 norm (||d_i||_2 = 1) at ingestion time reduces cosine similarity to a pure inner product (q_hat @ D_hat.T) with zero division overhead.
Interactive Runnable CodeSandbox
Unit-Normalized Cosine Similarity & Top-K Retriever
Equivalence of Cosine Similarity & Squared L2 on Unit Sphere
On the unit hypersphere, maximizing cosine similarity is mathematically identical to minimizing Euclidean distance.
Step-by-Step Implementation Breakdown
Add Epsilon (1e-12) to L2 Norm Denominator
Prevent ZeroDivisionError / NaN propagation when an empty string or zero vector is passed.
Use HNSW (Hierarchical Navigable Small World) When N > 100,000
Exact flat scan takes O(N*D); HNSW graphs reduce query latency to O(D * log N) with >98% recall.
Production Edge Cases & Interview Traps
- When using Matryoshka embeddings (e.g., truncating 1536 dims to 256 dims), you MUST re-normalize the truncated slice to unit L2 norm.
Frequently Asked Questions
When should I use Dot Product instead of Cosine Similarity?
Use unnormalized Dot Product only when vector magnitude encodes document authority or popularity; for semantic similarity on text embeddings, use Cosine Similarity.