Probability and statistics, the parts that matter

A probability distribution assigns nonnegative weights to outcomes that sum (or integrate) to 1. The three you'll meet daily:

• Bernoulli(p) — single yes/no event. The output of a binary classifier through a sigmoid.
• Categorical(p₁,...,pₖ) — pick one of k options with given probs. The output of a softmax classifier.
• Gaussian(μ, σ²) — continuous, bell-shaped. The implicit assumption behind L2 / MSE loss (the maximum-likelihood estimator under Gaussian noise).

A neural network classifier doesn't predict 'cat' — it predicts a probability distribution over classes: [p(cat), p(dog), p(bird), ...], with the softmax ensuring entries sum to 1. Training pushes that distribution toward the one-hot truth. Inference picks the argmax. Every loss function has a probabilistic interpretation — knowing it lets you spot model bugs that compile but predict nonsense.

The expectation E[X] is the average of X weighted by its probability. For data, you estimate it with the empirical mean over your sample: (1/N) Σ x_i. Almost every loss function in deep learning is an expectation:

loss_per_sample = ...           # shape (batch,)
loss = loss_per_sample.mean()   # expectation under the empirical distribution

Variance E[(X - E[X])²] measures spread. In deep learning it appears every time you initialize weights (variance scaled to layer width keeps activations from exploding — that's the whole Xavier/Kaiming-init story) or normalize activations (BatchNorm/LayerNorm subtract the mean and divide by sqrt(var) + ε).

The law of large numbers says the empirical mean converges to the true expectation as N → ∞. The central limit theorem says the empirical mean is approximately Gaussian. These are why mini-batch SGD is unbiased — each batch is a noisy but correct estimate of the full-data gradient. The noise actually helps: it's why SGD generalizes better than full-batch GD.

Train a classifier by maximizing the likelihood of the observed labels under the model's predicted distribution. Equivalently — and this is the form that goes in the code — minimize the negative log-likelihood:

NLL = -Σ_i log p_model(y_i | x_i)

For a one-hot label, that's exactly the cross-entropy between the truth and the prediction:

H(p, q) = -Σ p(x) log q(x)

When p is one-hot at the true class y, this collapses to -log q(y). When you see nn.CrossEntropyLoss() in PyTorch, this is what's happening: softmax + log + negate, fused into one numerically-stable op. The reason it 'just works' is not arbitrary — it's the principled estimator for any model that outputs a probability distribution.

For regression with Gaussian noise: MLE → minimize squared error → nn.MSELoss. For regression with Laplace noise: minimize absolute error → nn.L1Loss. The choice of loss is the choice of noise model.

The Kullback-Leibler divergence D_KL(p || q) = Σ p(x) log(p(x)/q(x)) measures how 'far' q is from p (asymmetric — D_KL(p||q) ≠ D_KL(q||p)). It's always ≥ 0, and = 0 iff p == q.

Where you'll meet it:

• Variational autoencoders (VAEs): the loss has a KL term between the encoder's posterior and a Gaussian prior. We'll derive it in the generative-models chapter.
• Knowledge distillation: train a small student to match a big teacher's output distribution. Loss = KL between student and teacher logits.
• RLHF and DPO: keep the policy close to a reference model via a KL penalty.
• Label smoothing: replace the one-hot target with a softened distribution; this is equivalent to optimizing a KL with smoothing.

The identity D_KL(p || q) = H(p, q) - H(p). When p is fixed (the true labels), minimizing KL = minimizing cross-entropy. That's why cross-entropy *is* the right loss — it's KL with a constant subtracted.

Classical statistics worried about bias (model can't fit the truth) vs. variance (model overfits the sample). Deep learning shifted the story: massively over-parameterized networks have tiny bias and *should* have huge variance — yet they generalize. Why? Several reasons researchers are still arguing about:

• SGD's implicit bias toward 'simple' (flat-minimum) solutions.
• Massive data amortizing the variance.
• Architectural priors (CNNs assume locality; transformers assume permutation equivariance) that constrain the effective hypothesis class.
• Double descent: as model capacity grows past 'interpolation threshold,' test error sometimes *decreases* again. Counter-intuitive but real.

The operational rule of thumb hasn't changed: if training loss is still decreasing, you're underfitting — make the model bigger or train longer. If validation loss starts rising while training loss falls, you're overfitting — add regularization, more data, or stop earlier. Plot both curves on every training run; it's the single most useful diagnostic in the field.