Recurrent Neural Networks and Memory

LESSON 6.1

Reuse one cell across time

A recurrent neural network updates a hidden state from the current input and previous state. The same parameters are used at every step. Parameter count does not grow with sequence length, although computation and saved activations do.

The hidden state is a learned summary, not an archive of every earlier input. A many-to-one model makes one decision about a sequence; a many-to-many model makes position-wise predictions. Whether the state retains useful information must be tested.

hₜ = tanh(Wxh xₜ + Whh hₜ₋₁ + bₕ)
logitsₜ = Why hₜ + bᵧ

LESSON 6.2

Backpropagation through the unrolled graph

Backpropagation through time is ordinary reverse-mode differentiation on the unrolled recurrent computation. Because each weight is reused, its gradient accumulates contributions from every use.

Long-range sensitivity contains a product of Jacobians. Saturated tanh derivatives and small gains can shrink a signal; large gains can amplify it. Matrix directions and singular values matter, so a scalar multiplier is an illustration rather than a complete stability diagnosis.

Jₜ = ∂hₜ/∂hₜ₋₁ = diag(1 − hₜ²) Whh
∂hT/∂hₖ = JT JT₋₁ … Jₖ₊₁

Keep this idea. A long sequence creates a deep computation graph even when the cell is small.

LESSON 6.3

Control explosions and bound the graph

Global norm clipping scales down a gradient when its norm exceeds a threshold. It preserves direction for that update but cannot recover a gradient that has already vanished. Clip after backward and before the optimizer step.

Truncated BPTT detaches a carried state at selected boundaries. Detaching preserves the numerical state while cutting its gradient history, reducing memory and limiting credit assignment. Resetting changes the value too; it is a different operation.

gclipped = g · min(1, c / ||g||₂)

loss.backward()
nn.utils.clip_grad_norm_(model.parameters(), 5.0)
optimizer.step()
# LSTM state at a truncation boundary:
h, c = h.detach(), c.detach()

LESSON 6.4

An additive memory with gates

An LSTM separates cell state c from exposed hidden state h. The forget gate controls retention, the input gate controls writing a candidate, and the output gate controls exposure. Sigmoid gates are learned functions of the current input and previous hidden state.

Holding the gate values fixed, the direct cell-state derivative is the forget gate. The total derivative also includes dependencies through gates and hidden states. Long retention is possible, not guaranteed: multiplying forget gates below one still attenuates the direct path.

fₜ, iₜ, oₜ = sigmoid(affine functions of [xₜ,hₜ₋₁])
gₜ = tanh(Wg[xₜ,hₜ₋₁] + bg)
cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ gₜ
hₜ = oₜ ⊙ tanh(cₜ)

LESSON 6.5

GRU: a compact gated alternative

A GRU has one state with reset and update gates. The reset gate changes how the past enters the candidate, while the update gate blends old state and candidate. It has no separate LSTM-style cell state.

Here z weights the new candidate, matching the lecture convention. PyTorch calls the old-state retention weight z and uses a particular reset placement. Translate conventions before comparing a hand calculation to library output.

Lecture convention:
h̃ₜ = tanh(Wx xₜ + Wh(rₜ ⊙ hₜ₋₁) + b)
hₜ = (1−zₜ) ⊙ hₜ₋₁ + zₜ ⊙ h̃ₜ
PyTorch update: hₜ = zₜ ⊙ hₜ₋₁ + (1−zₜ) ⊙ nₜ

LESSON 6.6

Do not invent observations through padding

With batch_first=True, inputs have shape [batch,time,features]. An ordinary RNN keeps updating through padded zeros because recurrent weights and biases still act. Both output[:,−1] and h_n from an unpacked padded input can therefore include nonexistent time steps.

Pack inputs with their true lengths or implement masked updates. With a packed, single-direction LSTM, the final layer's h_n corresponds to the last real step. In a bidirectional model, concatenate the final layer's forward and backward final states. h_n[-1] alone gives only one direction. Bidirectionality is appropriate only when future context is available at inference.

packed = nn.utils.rnn.pack_padded_sequence(
    x, lengths.cpu(), batch_first=True, enforce_sorted=False
)
_, (h_n, c_n) = lstm(packed)
logits = head(h_n[-1])  # final layer, one direction

LESSON 6.7

Teacher forcing and the context bottleneck

Teacher forcing feeds the correct previous output token during decoder training. At inference, a generator consumes its own earlier predictions. Errors then change later inputs, creating a mismatch between training and generation contexts.

An encoder–decoder can compress a source sequence into a final context state. Long inputs stress this fixed-size channel. Attention will allow a decoder to retrieve different information from multiple source positions.

Compare recurrent cells under fixed data splits and tuning budgets. Report results by sequence length, clipping frequency, and padding policy. Split related sequences by source or time when random rows would leak correlated information.

WHEN THE MODEL SURPRISES YOU

Mistakes worth catching early

FROM EQUATIONS TO A RUNNING MODEL

Remember the first observation

Compare RNN, GRU, and LSTM on variable-length sequences whose first observation determines the label. Later observations are distractors. Each model uses packed inputs, validation checkpoint selection, and a reserved test set.

Download the script, create a Python environment, and run the commands below in the folder containing the download. The example uses PyTorch on CPU and generates its own small dataset. No dataset or model download is required.

python -m venv .venv
# macOS / Linux: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install torch
python dl-week-6.py
# Fast execution check (not a convergence result):
python dl-week-6.py --smoke

"""Week 6: packed RNN/GRU/LSTM memory study. CPU, synthetic data only.
Install: python -m pip install torch
Run: python dl-week-6.py [--smoke]
"""
import argparse
import copy
import torch
from torch import nn
from torch.nn import functional as F
from torch.utils.data import DataLoader, TensorDataset

torch.set_num_threads(1)


def dataset(n, seed, longest=10):
    g = torch.Generator().manual_seed(seed)
    lengths = torch.randint(4, longest + 1, (n,), generator=g)
    y = torch.randint(0, 2, (n,), generator=g)
    x = .2 * torch.randn(n, longest, 1, generator=g)
    x[:, 0, 0] = 2 * y.float() - 1
    x[torch.arange(longest)[None, :] >= lengths[:, None]] = 0
    return TensorDataset(x, lengths, y)


class MemoryModel(nn.Module):
    def __init__(self, kind):
        super().__init__()
        self.cell = getattr(nn, kind)(1, 24, batch_first=True)
        self.head = nn.Linear(24, 2)

    def forward(self, x, lengths):
        packed = nn.utils.rnn.pack_padded_sequence(
            x, lengths.cpu(), batch_first=True, enforce_sorted=False)
        _, state = self.cell(packed)
        hidden = state[0] if isinstance(state, tuple) else state
        return self.head(hidden[-1])


@torch.no_grad()
def evaluate(model, data):
    model.eval()
    x, lengths, labels = data.tensors
    logits = model(x, lengths)
    return F.cross_entropy(logits, labels).item(), (logits.argmax(1) == labels).float().mean().item()


def main():
    p = argparse.ArgumentParser()
    p.add_argument('--smoke', action='store_true')
    args = p.parse_args()
    print('PyTorch', torch.__version__)
    train = dataset(128 if args.smoke else 640, 10)
    val, test, longer = dataset(200, 11), dataset(300, 12), dataset(300, 13, 24)
    for kind in ('RNN', 'GRU', 'LSTM'):
        torch.manual_seed(7)
        model = MemoryModel(kind)
        # Restart the same batch order for each architecture.
        loader = DataLoader(train, batch_size=64, shuffle=True,
                            generator=torch.Generator().manual_seed(8))
        optimizer = torch.optim.Adam(model.parameters(), lr=.008)
        best_loss, _ = evaluate(model, val)
        before = best_loss
        best = copy.deepcopy(model.state_dict())
        max_grad = 0.
        for _ in range(2 if args.smoke else 24):
            model.train()
            for x, lengths, labels in loader:
                optimizer.zero_grad()
                loss = F.cross_entropy(model(x, lengths), labels)
                assert torch.isfinite(loss)
                loss.backward()
                norm = nn.utils.clip_grad_norm_(model.parameters(), 5.)
                max_grad = max(max_grad, norm.item())
                optimizer.step()
            current, _ = evaluate(model, val)
            if current < best_loss:
                best_loss, best = current, copy.deepcopy(model.state_dict())
        model.load_state_dict(best)
        # Extra zero padding must not alter a packed sequence's final state.
        x, lengths, _ = test.tensors
        model.eval()
        with torch.no_grad():
            assert torch.allclose(model(x, lengths), model(F.pad(x, (0, 0, 0, 5)), lengths), atol=1e-6)
        _, acc = evaluate(model, test)
        _, stress = evaluate(model, longer)
        print(f'{kind:5} val {before:.4f}->{best_loss:.4f}; test acc={acc:.3f}; '
              f'longer acc={stress:.3f}; max pre-clip norm={max_grad:.3f}')
    print('Fixed toy split; longer set changes length distribution. Repeat seeds before ranking cells.')
    if args.smoke:
        print('Smoke run checks execution, not convergence.')


if __name__ == '__main__':
    main()

Read the output, then challenge it

• Verify true lengths exclude the padding from recurrent computation.
• Compare validation losses and test accuracy without assuming a universal winning cell.
• Increase sequence length in a separate stress study and inspect performance by length.
• Observe pre-clipping gradient norms to distinguish instability from weak long-range learning.

These are small mechanism studies, not benchmark claims. Seeds make runs easier to compare; versions and hardware can still change exact results. Decide your validation protocol before tuning and reserve final test results for reporting.

PREDICT → CALCULATE → EXPLAIN

Practice with worked answers

EXPLAIN IT WITHOUT THE CODE

Week 6 review questions

RESEARCH CONNECTIONS

Follow the original ideas

Course alignment: AI Algorithms 2, lecture Week 7. This site counts from Week 0. Explanations, browser labs, and runnable experiments are authored study companions to the supplied lecture material.