Calculus and gradients — how networks learn

The derivative df/dx answers one question: *if I nudge x by a tiny amount, how much does f(x) change?* Positive → f increases with x. Negative → f decreases. Zero → f is locally flat (could be a minimum, maximum, or saddle).

Training a neural network is, mechanically, a search for parameter values that make a loss function small. The derivative of the loss with respect to each parameter is the answer to 'which way should I nudge this parameter to make the loss smaller?' Multiply by a small step size (the learning rate η) and step against the gradient — that's gradient descent.

θ_{t+1} = θ_t - η · ∇L(θ_t)

The simplicity of this equation hides everything interesting in optimization. The rest of this chapter is: how do we compute ∇L efficiently (backprop), and how do we make the step smarter than 'multiply by a constant' (momentum, Adam)?

Networks are compositions: loss = L(f(g(h(x; W3); W2); W1)). The chain rule says the derivative of a composition is the product of the local derivatives along the way:

d(loss)/d(W1) = d(loss)/d(f) · d(f)/d(g) · d(g)/d(h) · d(h)/d(W1)

Each factor depends only on what happens at one layer. Backpropagation is the algorithm that computes this product efficiently by walking backwards through the computation graph, multiplying local derivatives. You will never compute these by hand in production — autograd does it — but understanding that 'backward = chain rule applied to your forward code' is what lets you debug models that don't train.

A tangible example. Consider f(w) = (w * 3 - 5)². Let u = w*3 - 5, so f = u². Chain rule: df/dw = df/du · du/dw = 2u · 3 = 6u = 6(w*3 - 5). At w = 2: 6 * (6 - 5) = 6. PyTorch's autograd gives you exactly this number, with no calculus on your part:

import torch
w = torch.tensor(2.0, requires_grad=True)
f = (w * 3 - 5) ** 2
f.backward()
print(w.grad)   # tensor(6.)

When f takes a vector and returns a scalar (a loss), the gradient ∇f is a vector whose i-th entry is ∂f/∂x_i. It points in the direction of steepest *increase*. Gradient descent steps against it.

When f takes a vector and returns a vector (a layer's forward pass), the Jacobian J is a matrix whose (i,j) entry is ∂f_i/∂x_j. Backprop never builds the full Jacobian explicitly — it builds the Jacobian-vector product, which is what the chain rule actually needs. This is why autograd is cheap: an n×n matmul, not an n³ Jacobian construction.

The Hessian H is the matrix of second derivatives ∂²f/∂x_i ∂x_j. It tells you the *curvature* of the loss surface. Newton's method uses it directly (step = -H⁻¹ ∇f), which is too expensive for big networks. But Hessian analysis explains why some loss surfaces are easy (positive-definite, smooth bowls) and some are hard (saddle points, ill-conditioned ravines). Modern optimizers like Adam approximate Hessian information with per-parameter running averages.

Vanilla SGD: θ ← θ - η g, where g = ∇L. Simple, slow in ravines, sensitive to feature scale.

SGD + momentum: keep a running average of gradients. v ← βv + g; θ ← θ - η v. The 'heavy ball' picture — momentum lets you skip over noise and pick up speed down long valleys. Standard for vision CNNs trained from scratch.

RMSProp: per-parameter learning-rate adaptation. s ← βs + (1-β) g²; θ ← θ - η g / (√s + ε). Divides each parameter's step by its recent gradient magnitude — large-gradient params take smaller steps, small-gradient params take larger ones. Decouples training from feature scale.

Adam = SGD-momentum + RMSProp. Maintains running averages of both g (first moment) and g² (second moment). Default for almost every modern architecture. Bias-correction terms handle the early-training cold-start.

AdamW decouples weight decay from the gradient (the original Adam mixed them, which interacts badly with the moment estimates). For 2026 production, AdamW is the default for transformers and most non-vision work. Learning rate 3e-4 is the legendary starting point; for fine-tuning pretrained models, drop to 2e-5 to 5e-5.

Gradient descent is *iterative greedy descent on a complicated surface*. The loss landscape of a deep net is high-dimensional and full of saddle points and ravines, but two things make it tractable: (1) the surface is locally smooth almost everywhere, and (2) the dimensions are enormous, so escape directions from bad regions almost always exist.

What *can* go wrong:

• Vanishing gradients: gradient magnitudes shrink to zero through many layers, no learning signal at the bottom. Fix: ReLU, residuals, careful init, LayerNorm.
• Exploding gradients: gradients blow up, you get NaN losses. Fix: gradient clipping (clip_grad_norm_(..., 1.0)), smaller LR, better init.
• Saddle points: zero gradient but not a minimum. Modern optimizers (with momentum) typically escape; second-order info would help but is too expensive.
• Sharp minima vs. flat minima: flat minima generalize better empirically. SGD finds flatter ones than full-batch GD; this is partly why mini-batch SGD works so well.

Most architectural choices in deep learning — ReLU, batch norm, residual connections, careful initialization, dropout — exist to keep gradients well-behaved across many layers. We'll meet each one in context.