Generative models — VAEs, GANs, and Diffusion

A generative model learns a distribution p(x) over data so that you can either (a) compute likelihoods of new samples, (b) sample new examples that look like the training set, or (c) both. This is harder than classification — instead of mapping x → y, you're modeling all of x.

Three families dominate modern deep learning:

• VAEs (variational autoencoders): learn a compact, structured latent space. Sample by drawing from a prior and decoding. Strong likelihoods, slightly blurry samples.
• GANs (generative adversarial networks): train two networks to fight each other. Sharp, photorealistic samples but unstable training and no likelihood.
• Diffusion models: learn to reverse a gradual noising process. State of the art for images (Stable Diffusion, Midjourney, DALL·E 3), audio (AudioLDM), video (Sora, Veo).

Each trades off sample quality, training stability, sample diversity, and inference speed. We'll cover each at a working level — enough to read papers and fine-tune existing models, not enough to invent your own architecture from scratch.

A VAE learns a probabilistic encoder q(z|x) and decoder p(x|z), where z lives in a low-dim latent space. Encode an image into a Gaussian over z, sample z, decode back to image space.

The Evidence Lower Bound (ELBO) objective:

L = E_q[log p(x|z)]  -  KL(q(z|x) || p(z))
  = reconstruction loss  -  KL to prior (typically N(0, I))

The reconstruction term says 'decode z back to x'. The KL term regularizes the latent to be close to a unit Gaussian, so you can sample directly from N(0, I) at inference.

The reparameterization trick makes the encoder differentiable. Instead of sampling z ~ N(μ, σ²) (not differentiable wrt μ, σ), write z = μ + σ * ε where ε ~ N(0, I). Now z is a differentiable function of the encoder's outputs, and backprop works.

mu, logvar = encoder(x)
std = (0.5 * logvar).exp()
eps = torch.randn_like(std)
z = mu + std * eps
x_hat = decoder(z)
recon = F.mse_loss(x_hat, x, reduction='sum')
kl = -0.5 * (1 + logvar - mu.pow(2) - logvar.exp()).sum()
loss = recon + kl

VAEs are central to modern AI in disguise: the VAE in Stable Diffusion compresses 512×512 images into a 64×64×4 latent space where the diffusion model actually operates. That single design choice is what makes high-resolution diffusion tractable.

A GAN trains two networks at the same time:

• Generator G(z): maps a random vector z ~ N(0, I) to a fake sample.
• Discriminator D(x): predicts whether x is real (from data) or fake (from G).

The original minimax objective:

min_G max_D  E[log D(x)] + E[log(1 - D(G(z)))]

In practice, train them alternately: one step on D's loss, one step on G's loss, repeat for millions of steps. When training succeeds, G's samples become indistinguishable from data and D collapses to predicting 0.5 everywhere.

Training is famously unstable. Mode collapse, vanishing G gradients, oscillation. Solutions:

• Wasserstein GAN (WGAN-GP): replace the loss with the Wasserstein distance, add gradient penalty for Lipschitz constraint. Much more stable.
• Spectral normalization: divide weights by their spectral norm. Bounds the discriminator's Lipschitz constant.
• StyleGAN (StyleGAN2, StyleGAN3): the gold standard for face generation — adaptive instance normalization, progressive growth, then a refined fixed-resolution architecture.
• Conditional GANs (cGAN, Pix2Pix, CycleGAN): pass a condition (label, sketch, source image) into both G and D. Used for image-to-image translation.

In 2026, GANs are still excellent for super-resolution, image enhancement, and real-time generation (single forward pass vs. diffusion's many denoising steps). For general image generation, diffusion has won.

A diffusion model is trained to *reverse* a fixed process that gradually adds Gaussian noise to data:

Forward (no learned params):  x_0 → x_1 → ... → x_T = pure noise
Reverse (learned):             x_T → x_{T-1} → ... → x_0 = sample

The trick: instead of learning the full reverse distribution, train a network ε_θ(x_t, t) to predict the noise that was added at step t. The training loss is shockingly simple:

x_0 = batch
t = torch.randint(0, T, (batch_size,))
noise = torch.randn_like(x_0)
alpha_bar = scheduler.alpha_bars[t]
x_t = (alpha_bar.sqrt() * x_0) + ((1 - alpha_bar).sqrt() * noise)
pred_noise = model(x_t, t)
loss = F.mse_loss(pred_noise, noise)

That's it. MSE between predicted and actual noise, sampled at random timesteps. At inference, you start from x_T ~ N(0, I) and iteratively denoise:

for t = T..1:
    pred_noise = model(x_t, t)
    x_{t-1} = (1/sqrt(α_t)) * (x_t - (β_t / sqrt(1-α̅_t)) * pred_noise) + σ_t * z

With T=1000 steps (DDPM) or fewer (DDIM, DPM-Solver: 20-50 steps gives high quality).

Architectural anatomy:

• Backbone: usually a U-Net (Stable Diffusion 1/2/XL) or a Diffusion Transformer (DiT, Sora, Flux). DiT is taking over at scale.
• Time conditioning: the timestep t is embedded via sinusoidal encoding + MLP and added/injected into every block.
• Text conditioning: cross-attention from text embeddings (CLIP, T5) lets you guide generation.
• Latent diffusion (Stable Diffusion's trick): operate in VAE latent space, not pixel space. Compresses 512×512×3 to 64×64×4, making everything ~64× cheaper.

Diffusion is the state of the art for image, video, and audio generation. The same machinery, scaled up: SD3 / Flux for image; AudioLDM for audio; Sora / Veo for video; molecular generation in chemistry; protein structure (RFdiffusion).

Classifier-Free Guidance (CFG): at inference, run the model with and without the text condition, then push predictions away from the unconditional one: noise = noise_uncond + s * (noise_cond - noise_uncond). The single most impactful inference-time trick. s=7-12 is typical for SD.

Text generation works differently. The dominant approach is autoregressive language modeling: predict the next token given the previous ones. A causal-decoder transformer trained on loss = -log p(x_t | x_{<t}) learns the joint distribution p(x_1, ..., x_T) by chain rule.

Sampling at inference:

for i in range(max_new_tokens):
    logits = model(context)[:, -1, :]   # last position
    next_token = sample(logits, temperature, top_k, top_p)
    context = torch.cat([context, next_token], dim=1)

Sampling strategies:

• Greedy / argmax: deterministic, often repetitive.
• Temperature: divide logits by T before softmax. T=0.7 is typical; T→0 is greedy.
• Top-k: only sample from the k most probable tokens.
• Top-p (nucleus): only sample from the smallest set of tokens whose probabilities sum to ≥ p. p=0.9 is typical.
• Beam search: maintains k partial hypotheses, expands each, keeps top k. Better for translation and summarization than for free-form generation (which it makes too 'safe').

GPT-style models are the most successful generative architecture of 2026 by a wide margin. Diffusion handles continuous data well; autoregression handles discrete data (text, code, structured outputs) well. Diffusion-LM and discrete-diffusion research is trying to merge them but autoregression is still the practical default for text.