Linear algebra for deep learning

Every neural network is, at the matrix level, a stack of linear transformations interleaved with nonlinearities. A Linear(in=784, out=128) layer is literally a matrix multiply: y = x @ W + b. A convolution is a structured matrix multiply. Attention is three matrix multiplies and a softmax. Recurrence is a matrix multiply you apply in a loop. If you can manipulate matrices fluently, the math of deep learning stops looking like jargon and starts looking like Lego.

You do not need a semester of pure linear algebra. You need: vectors and their geometric meaning, matrix multiplication and its rules, the dot product as a similarity measure, broadcasting, eigenvalues at a working level, the SVD as a tool, norms, and a fluent feel for tensor shapes. That's the chapter. We'll build it in numpy because numpy is the lingua franca underneath every Python ML library — and because once you can do it in numpy, doing it in PyTorch is a one-line API translation.

A vector in deep learning is a 1-D array of numbers. Geometrically, picture an arrow from the origin; algebraically, a list. Both pictures are useful: when we talk about 'cosine similarity between embeddings' we mean the angle between two arrows; when we talk about 'shape mismatch in a forward pass' we mean lists of different lengths.

import numpy as np
x = np.array([1.0, 2.0, 3.0])    # shape (3,)
norm = np.linalg.norm(x)          # length: sqrt(1+4+9) ≈ 3.742
unit = x / norm                   # direction, length 1

The L2 norm ||x||_2 = sqrt(Σ x_i²) is the Euclidean length. The L1 norm ||x||_1 = Σ |x_i| measures total magnitude and is what weight_decay='l1' regularizes. The L∞ norm = max |x_i|. Each shows up in gradient-clipping, regularization, and adversarial-robustness work.

The dot product x·y = Σ x_i y_i measures how much two vectors agree — positive when they point the same way, zero when perpendicular, negative when opposed. Cosine similarity is (x·y)/(||x|| ||y||). That single formula is the heart of every retrieval-augmented system, every nearest-neighbor classifier, and the attention scores inside every transformer.

A matrix is a 2-D array. Matrix multiplication C = A @ B is defined when A.shape[1] == B.shape[0]. The result has shape (A.shape[0], B.shape[1]), and C[i, j] is the dot product of row i of A with column j of B.

A = np.random.randn(4, 3)     # 4 rows, 3 cols
B = np.random.randn(3, 5)     # 3 rows, 5 cols
C = A @ B                     # shape (4, 5)

The shape check (4,3) @ (3,5) → (4,5) is the single most-used debugging tool in deep learning. When a forward pass fails, 80% of the time the answer is in the shapes. Senior practitioners reflexively write the shape in a comment on every non-trivial line.

Key properties to internalize: matmul is associative but NOT commutative (A @ B ≠ B @ A in general). It distributes over addition. The transpose A.T swaps axes; (A @ B).T == B.T @ A.T. The identity matrix I is the matmul identity. The inverse A^(-1) undoes A when it exists (square, non-singular). We almost never compute inverses in deep learning — they're slow and unstable. We solve Ax = b with np.linalg.solve(A, b) instead.

Broadcasting lets numpy operate on arrays of different but compatible shapes by implicitly stretching the smaller one. It's how you add a bias vector to a batch of activations:

batch = np.random.randn(32, 128)   # 32 samples, 128 features
bias  = np.random.randn(128)        # one bias per feature
output = batch + bias               # works: bias broadcasts across batch dim

The rule: align shapes from the right; each dim must either match or be 1. The smaller array is treated as if repeated along the mismatched axes (no copy is made; the broadcast is virtual). Broadcasting is fast and lets you write code that reads like the math.

Misuse it and you'll silently train on broadcast garbage — the model 'runs' but never converges. Classic bug: subtracting a vector when you meant a matrix and getting a (32, 128) - (32,) shape mismatch with no error because the dims happened to be compatible by accident. The cure: write the shape next to every line until you don't have to. Treat assert x.shape == (B, T, C) as load-bearing documentation.

An eigenvector v of a matrix A satisfies A @ v = λ v — the matrix acts on it as pure scaling by λ (the eigenvalue). Eigenvectors are the 'natural axes' of a transformation. Why care? Three places:

1. PCA (Principal Component Analysis): the eigenvectors of the data's covariance matrix are the directions of maximum variance. The top-k of them give you a k-dimensional embedding. Still the right baseline for many dimensionality-reduction tasks. 2. Spectral norm: the largest singular value of a weight matrix bounds how much it can stretch any input. Spectral normalization (controlling this) stabilizes GAN training. 3. Eigenvalue analysis of the loss Hessian explains why training is stuck, saddle-point structure, sharpness-aware minimization, and second-order optimizers.

The Singular Value Decomposition A = U Σ V^T factors any matrix into rotation × scaling × rotation. np.linalg.svd(A) returns those three pieces. SVD is how LoRA fine-tuning works (we'll get there), how truncated SVD compresses embedding tables, and how solving least-squares numerically is implemented. You don't need to derive SVD; you need to know it exists and recognize when a problem fits it.

A tensor is just an n-D array. In PyTorch (and numpy) tensors come in standard shapes you should recognize on sight:

• Image batch: (B, C, H, W) — batch, channels, height, width. PyTorch convention.
• Sequence batch: (B, T, D) — batch, time, features. Most modern transformer code.
• Token logits: (B, T, V) — batch, time, vocabulary. Output of a language-model head.
• Attention weights: (B, H, T, T) — batch, heads, queries, keys.
• Conv kernel: (out_channels, in_channels, kH, kW).

Reshaping is constant: x.reshape(...), x.view(...), x.permute(...), x.transpose(...), x.flatten(2), x.unsqueeze(0), x.squeeze(-1). The two most-confused: reshape may copy if the memory isn't contiguous (view won't, and will error); permute reorders dims by name, transpose swaps exactly two. Read the docs once; you'll alias them in your head forever.

The einsum operator is the master tool: 'bhid,bhjd->bhij' expresses 'batch × heads × queries × dims dot batch × heads × keys × dims to produce attention scores' in one line. Write it down once and you've understood multi-head attention's tensor flow.