Recurrent networks — RNNs, LSTMs, GRUs, and sequence modeling

A recurrent neural network (RNN) processes a sequence x_1, x_2, ..., x_T by maintaining a hidden state h_t that summarizes everything seen so far:

h_t = tanh(W_xh @ x_t + W_hh @ h_{t-1} + b)
y_t = W_hy @ h_t

The same weights W_xh, W_hh, W_hy are reused at every time step. Unfolded across time, an RNN is a very deep feed-forward net where the depth equals sequence length. This is what makes RNNs both powerful (variable-length input) and hard to train (gradients flow through every step).

In PyTorch:

rnn = nn.RNN(input_size=64, hidden_size=128, batch_first=True)
x = torch.randn(32, 100, 64)   # (B, T, D)
output, h_T = rnn(x)             # output: (B, T, 128), h_T: (1, B, 128)

Applications before transformers: language modeling (Karpathy's char-RNN), machine translation (seq2seq with attention), speech recognition, time-series forecasting, music generation. Some of these are now transformer-shaped; some are still RNN-shaped.

Backpropagating through time multiplies the same Jacobian W_hh^T · diag(tanh') at every step. Two failure modes:

• Vanishing gradients: if the dominant eigenvalue is < 1, after T steps it's near zero — long-range dependencies aren't learned.
• Exploding gradients: if > 1, gradients overflow — NaNs or wild updates.

Vanilla RNNs reliably handle ~10-20 step dependencies. For paragraphs, songs, sentences with complex grammar — they don't. Gradient clipping (clip_grad_norm_) tames explosion. Vanishing was harder; the solution was gated recurrence.

The Long Short-Term Memory (LSTM) cell adds a cell state c_t (a 'memory tape') that flows mostly unchanged across time, plus three sigmoid gates that decide what to forget, what to write, and what to read:

f_t = σ(W_f [h_{t-1}, x_t])       # forget gate
i_t = σ(W_i [h_{t-1}, x_t])       # input gate
g_t = tanh(W_g [h_{t-1}, x_t])    # candidate
o_t = σ(W_o [h_{t-1}, x_t])       # output gate

c_t = f_t * c_{t-1} + i_t * g_t   # update cell state
h_t = o_t * tanh(c_t)              # output hidden state

The key trick: gradient flow through c_t = f_t * c_{t-1} + .... When f_t ≈ 1, the cell state passes information forward almost losslessly — like a residual connection in time. LSTMs reliably handle 100-1000 step dependencies and dominated NLP from 2014 to 2017.

lstm = nn.LSTM(input_size=64, hidden_size=128, num_layers=2, dropout=0.2, batch_first=True, bidirectional=True)

The Gated Recurrent Unit (GRU) simplifies the LSTM into two gates and one state:

z_t = σ(W_z [h_{t-1}, x_t])           # update gate
r_t = σ(W_r [h_{t-1}, x_t])           # reset gate
h̃_t = tanh(W_h [r_t * h_{t-1}, x_t])   # candidate
h_t = (1 - z_t) * h_{t-1} + z_t * h̃_t

Fewer parameters, ~30% faster, often as good as LSTM. The default 'cheap RNN' choice. Empirically: LSTM marginally better for very long sequences, GRU marginally better for small data.

Bidirectional RNNs run one forward and one backward, concatenating their outputs. The result depends on past *and* future context — perfect for classification or labeling tasks where the whole sequence is available (POS tagging, NER, sentence classification). Useless for autoregressive generation, where the future doesn't exist yet.

The classic encoder-decoder for machine translation:

1. Encoder LSTM consumes the source sentence, ends with a final hidden state h_T. 2. Decoder LSTM is initialized from h_T, generates the target one token at a time.

Problem: h_T is a fixed-size bottleneck. Long source sentences lose information.

Attention (Bahdanau, 2014) let the decoder look at all encoder hidden states at each generation step, weighted by relevance:

α_t = softmax(score(h_dec_t, h_enc_*))    # attention weights
c_t = Σ α_t * h_enc_*                     # context vector

This directly inspired the Transformer's 'attention is all you need' insight: if attention is the heavy-lifting mechanism, why not skip the recurrence entirely? In 2026, attention-only architectures dominate, but RNN-style state-space models (Mamba, S6) are making a comeback for very long context because their compute is linear in sequence length, not quadratic.

Transformers replaced RNNs in most NLP work, but there are domains where recurrence is still preferred:

• Streaming inference: RNNs process token-by-token with constant memory; transformers need either a KV-cache (big) or chunked attention. Speech recognition systems (RNN-T) often still use LSTM/Conformer-RNN-T architectures.
• Very long context: linear-time recurrence beats quadratic-time attention when T > 100k. Modern revival: Mamba and other state-space models combine RNN linear-time with transformer-style quality.
• Time series with tight latency budgets: financial / IoT / control systems where you need to react to one new sample at a time.
• Tiny edge devices: a 1M-param GRU often beats a 1M-param transformer because attention's quadratic memory doesn't pay off at that scale.

Knowing RNNs isn't legacy — it's knowing the right tool when the constraint is sequential, streaming, or low-resource.