Spectral Theorem, Singular Value Decomposition (SVD), PCA, LoRA low-rank adaptation, and Jacobian/Hessian identities.
Essential mathematical reference connecting matrix factorizations (Eigen-decomposition, SVD, Cholesky) to Principal Component Analysis (PCA), Low-Rank Adaptation (LoRA), and gradient backpropagation.
Every m x n matrix A decomposes into orthogonal U (m x m), diagonal singular values Sigma >= 0, and orthogonal V (n x n).
Truncating SVD to top-k singular values minimizes Frobenius and spectral reconstruction error.
Freezes pretrained W_0 and trains rank-r matrices A and B, cutting trainable parameters by >99%.
For centered data matrix $X \in \mathbb{R}^{n \times d}$, the sample covariance matrix is $C = \frac{1}{n-1}X^\top X$. Computing PCA via the thin SVD $X = U \Sigma V^\top$ avoids forming $X^\top X$ explicitly, squaring the condition number $\kappa(X^\top X) = \kappa(X)^2$.
| Decomposition | Formula | Matrix Requirement | Primary ML Use Case |
|---|---|---|---|
| Spectral (Eigen) | A = Q Lambda Q^T | Real symmetric (A = A^T) | Covariance matrices, Graph Laplacians |
| SVD | A = U Sigma V^T | Any m x n rectangular matrix | PCA, Moore-Penrose Pseudoinverse, Recommender Systems |
| Cholesky | A = L L^T | Symmetric Positive Definite (SPD) | Gaussian Processes, Multivariate Normal sampling |
| QR Factorization | A = Q R | Full column rank m >= n | Numerically stable Ordinary Least Squares (OLS) |
The non-zero singular values sigma_i of X are the exact square roots of the non-zero eigenvalues lambda_i of X^T X (sigma_i = sqrt(lambda_i)), and the right singular vectors V are the eigenvectors of X^T X.