Reinforcement learning and RLHF

Reinforcement learning is the framework for an agent learning to act in an environment to maximize cumulative reward. Formalized as a Markov Decision Process (MDP): at each time step the agent observes a state s_t, picks an action a_t from a policy π(a|s), the environment returns a reward r_t and the next state s_{t+1}. Goal: maximize E[Σ γ^t r_t] (discounted sum of rewards) over trajectories.

Two quantities are central:

• State-value function V^π(s) = E_π[Σ γ^t r_t | s_0 = s] — expected return starting from state s under policy π.
• Action-value function Q^π(s, a) = E_π[Σ γ^t r_t | s_0 = s, a_0 = a] — expected return starting from (s, a) then following π.

The Bellman equation ties them recursively: V(s) = E[r + γ V(s')]. Almost every RL algorithm is a different way to approximate or use this equation.

RL is harder than supervised learning because (a) rewards are sparse and delayed, (b) the agent's actions change the data distribution it sees (exploration/exploitation tension), and (c) credit assignment over long trajectories is unstable. Deep RL = use a neural network to approximate V, Q, or π.

Deep Q-Networks (DQN) learns Q(s, a; θ) with a neural network. The training target is the Bellman equation, with the network used to estimate the future value:

target = r + γ * max_{a'} Q_target(s', a')
loss = (Q(s, a) - target)²

Three tricks made it work: 1. Replay buffer: store (s, a, r, s') transitions; train on random minibatches. Breaks temporal correlation. 2. Target network: a frozen copy of Q used for the target. Updated every N steps. Prevents the moving target from chasing itself. 3. ε-greedy exploration: pick the argmax action with probability 1-ε, random otherwise. Decay ε over training.

DQN solved Atari from raw pixels in 2015. Modern descendants (Rainbow DQN, distributional DQN) push the limits but the recipe is the same. Used for discrete action spaces — game playing, hardware-bound control.

Policy gradient methods learn π_θ(a|s) directly. The REINFORCE estimator:

∇_θ J(θ) = E[Σ_t ∇_θ log π_θ(a_t|s_t) * G_t]

where G_t is the cumulative return from step t. Intuition: increase the probability of actions that led to high returns, decrease those that led to low ones. Naively very high variance, so reduce variance with a baseline (subtract V(s_t) so you only get credit for outperforming average expectations).

Actor-Critic trains two networks together: an actor π_θ(a|s) and a critic V_φ(s) that provides the baseline / advantage A_t = G_t - V(s_t) (or the GAE estimator, generalized advantage estimation). Standard architecture for modern policy-gradient RL.

PPO (Proximal Policy Optimization) adds a clipping trick to keep the policy from changing too much per update:

L(θ) = E[ min(r_t(θ) A_t, clip(r_t(θ), 1-ε, 1+ε) A_t) ]
     where r_t(θ) = π_θ(a_t|s_t) / π_old(a_t|s_t)

PPO is the workhorse of modern RL — simple to implement, stable, the default for robotics, locomotion, game AI, and (until DPO arrived) LLM alignment. Read OpenAI's original PPO paper once; it's short and lucid.

Reinforcement Learning from Human Feedback (RLHF) is how ChatGPT, Claude, Gemini learn to be helpful and safe. The pipeline:

1. SFT (Supervised Fine-Tuning): fine-tune a base LLM on curated instruction-response pairs. This gives it the *format* of being a helpful assistant.

2. Reward Modeling: collect pairs (prompt, response_A, response_B) with human labels of 'which is better.' Train a reward model (a transformer with a scalar head) to predict the preferred response. Loss: pairwise ranking (Bradley-Terry): `` L = -log σ(R(prompt, chosen) - R(prompt, rejected)) ``

3. PPO on the LLM: treat the LLM as a policy, generate responses, score with the reward model, run PPO. Crucial detail: add a KL penalty to the reward to keep the policy near the SFT model — without it, the policy will hack the reward model and produce gibberish that scores high. `` R_total = R_reward_model - β * KL(policy || SFT_model) ``

RLHF is computationally heavy: you need the policy, a frozen reference policy (for KL), the reward model, and a value head — four model copies in memory. That's why labs invented simpler alternatives.

Direct Preference Optimization (DPO) (2023) showed that the RLHF objective can be reduced to a closed-form supervised loss on preference data — no reward model, no PPO, no RL machinery:

L_DPO = -E[ log σ( β log(π(yw|x) / π_ref(yw|x)) - β log(π(yl|x) / π_ref(yl|x)) ) ]

where yw is the preferred response, yl is the rejected one, π_ref is the SFT model. One forward pass through the trainable policy + one through the (frozen) reference, then a clean cross-entropy-style loss. Stable, fast, easy — almost every recent open LLM alignment uses DPO instead of full RLHF.

ORPO (Odds Ratio Preference Optimization, 2024) eliminates the reference model. Combines SFT loss with a preference penalty in one objective — train from a base model in a single stage.

KTO (Kahneman-Tversky Optimization, 2024) uses prospect-theory-style loss; only needs *binary* feedback (good / bad) per response, not paired preferences. Useful when you have implicit feedback signals (clicks, completions, satisfaction).

SimPO, IPO, DPO-variants keep arriving. The space is moving fast. The practical lesson: in 2026, you almost certainly don't need to run PPO on an LLM. DPO + a few thousand preference pairs gets you 90% of the alignment win at 10% of the engineering cost. We'll wire one up in the capstone.

Beyond LLMs:

• Game playing: AlphaGo, AlphaZero, AlphaStar — model-based RL with MCTS. OpenAI Five (Dota), DeepMind Agent (Minecraft, Quake).
• Robotics: locomotion (ANYmal, Boston Dynamics), dexterous manipulation, sim-to-real transfer. PPO + domain randomization.
• Resource scheduling: data center cooling, network packet routing, ad bidding. Real production deployments.
• Drug / molecule discovery: RL over discrete chemistry actions. Combined with diffusion / autoregressive priors.
• Architecture search: NAS — RL agent picks neural architectures, reward = validation accuracy.
• Compiler optimization: AlphaTensor, AlphaDev — RL finds faster matrix-multiplication and sorting algorithms.

RL is harder to train and trickier to debug than supervised learning, but for problems where the supervisory signal is 'cumulative outcome over a trajectory,' there's no alternative. Knowing the basic vocabulary (state, action, reward, policy, value, Bellman, PPO, KL constraint) is enough to read most modern papers.