Deep Reinforcement Learning

LESSON 13.1

Learning changes the data collection process

In reinforcement learning, an agent observes a situation, chooses an action, and receives a reward and a new observation. Its policy changes which experiences it collects. This feedback loop distinguishes the problem from a fixed supervised dataset.

A Markov decision process specifies states, actions, transition probabilities, rewards, and a discount factor. The Markov assumption means the state contains the information needed to predict the next transition given the action. An observation may be incomplete; a recurrent policy can use history, but recurrence does not automatically make every hidden process fully observable.

sₜ → action aₜ ~ π(·|sₜ) → reward rₜ, next state sₜ₊₁
Return Gₜ = rₜ + γrₜ₊₁ + γ²rₜ₊₂ + …

LESSON 13.2

Predict future return under a policy

Vπ(s) is expected discounted return starting in state s and following policy π. Qπ(s,a) conditions additionally on the first action. Reward is immediate feedback; value estimates the longer-term consequences.

The Bellman expectation equation relates a value to immediate reward plus the value of the next state under the policy. The optimality equation instead maximizes over the next action. That maximum changes the learning target; it is not merely another notation for evaluating a fixed policy.

Vπ(s) = Eπ[r + γVπ(s′) | s]
Q*(s,a) = E[r + γ maxa′ Q*(s′,a′) | s,a]

LESSON 13.3

Learn from a temporal-difference error

Tabular Q-learning moves one state–action entry toward a bootstrapped target. The temporal-difference error compares that target with the current estimate. Bootstrapping uses a model's own estimate of future return, so targets can be biased or unstable.

Epsilon-greedy exploration chooses a random action with probability ε and otherwise selects a current maximizer. The random branch can also happen to choose the greedy action. Exploration changes what the agent learns about, not just how noisy its outputs look.

y = r + γ maxa′ Q(s′,a′) for a nonterminal transition
δ = y − Q(s,a)
Q(s,a) ← Q(s,a) + ηδ

LESSON 13.4

Replace the table with a neural value function

A DQN maps a state to one estimated Q-value per discrete action. These are return estimates, not class probabilities, so the output normally has no softmax. Training gathers the estimate for the action actually taken and regresses it toward a detached target.

Experience replay stores transitions and samples batches to reduce immediate temporal correlation and reuse experience. A target network changes more slowly than the online network, reducing how rapidly the regression target moves. Replay does not make experiences perfectly independent or eliminate distribution shift.

q_selected = online(states).gather(1, actions[:,None]).squeeze(1)
with torch.no_grad():
    next_value = target(next_states).max(dim=1).values
    y = rewards + gamma * (~terminated).float() * next_value
loss = F.smooth_l1_loss(q_selected, y)
# Periodically: target.load_state_dict(online.state_dict())

LESSON 13.5

Episode ending is not always task termination

A terminal state ends future return, so its target has no bootstrap term. A time-limit truncation of a continuing task stops data collection without necessarily making future value zero. Conflating them biases value estimates.

Bootstrap from the actual final observation when a rollout is truncated, not from the next episode's reset observation. If a finite horizon is intrinsic to the task, remaining time belongs in the state and horizon completion can be terminal. Define the task semantics before coding a generic done flag.

LESSON 13.6

Learn the policy directly

Policy-gradient methods adjust action probabilities in proportion to their estimated consequences. REINFORCE uses sampled returns to weight log-probability gradients. A baseline independent of the selected action can reduce variance without changing the expected score-function gradient under the usual assumptions.

Actor–critic methods learn a policy (actor) and a value estimate (critic). An advantage estimate compares an action's outcome with a baseline, often using a bootstrapped TD estimate. Bootstrapping can reduce variance while introducing estimation bias. Old trajectories cannot be reused indiscriminately in an on-policy objective; off-policy corrections or a suitable off-policy algorithm are needed.

∇θ J ≈ mean[∇θ log πθ(aₜ|sₜ) · (Gₜ − b(sₜ))]
Actor–critic TD advantage: Âₜ = rₜ + γV(sₜ₊₁) − V(sₜ)
At true termination, remove the next-value term.

LESSON 13.7

Evaluate behavior, not just the TD loss

A smaller TD loss can mean the network fits its current targets without solving the task well. Evaluate returns, success rates, episode lengths, and actual trajectories using a declared evaluation policy. Separate exploration during training from greedy or otherwise specified evaluation behavior.

Repeat independent training seeds and use held-out environment initializations or variations appropriate to the generalization claim. Record environment steps, not only optimizer updates. A reward proxy can be exploited in unintended ways, so inspect behavior against the real objective rather than celebrating reward alone.

LESSON 13.8

Connect the full deep learning course

Choose a problem where you can state the available observations, target or reward, inference constraints, and evaluation protocol. Compare two justified approaches against a simple baseline. Use a controlled ablation to test why a proposed mechanism helps.

For a vision problem, compare convolutional and residual features; for sequences, recurrence and attention; for generation, a VAE, GAN, or diffusion model; for control, a value-based or policy-based method. You do not need every architecture in one project. Match the mechanism to the problem.

Deliver runnable code, fixed splits or environment settings, validation decisions, final metrics, cost, failure cases, and a model card. Explain one result that challenged your expectation. The end of the course is the ability to defend a modeling decision with evidence.

WHEN THE MODEL SURPRISES YOU

Mistakes worth catching early

FROM EQUATIONS TO A RUNNING MODEL

A complete DQN in a small line world

Train a DQN agent to reach the right-hand goal in a seven-position world. Replay, target copies, epsilon-greedy exploration, detached targets, and separate termination/truncation handling are implemented directly. The script compares greedy evaluation with a random policy across all nonterminal starting positions.

Download the script, create a Python environment, and run the commands below in the folder containing the download. The example uses PyTorch on CPU and generates its own small dataset. No dataset or model download is required.

python -m venv .venv
# macOS / Linux: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install torch
python dl-week-13.py
# Fast execution check (not a convergence result):
python dl-week-13.py --smoke

"""Week 13: DQN with replay and correct terminal/truncation semantics.
python -m pip install torch
python dl-week-13.py [--smoke] [--seed 7]
Seven-state line world: move left/right, goal at 6, external cutoff at 20.
"""
import argparse
import copy
import random
from collections import deque
import torch
from torch import nn
from torch.nn import functional as F

torch.set_num_threads(1)
STATES, LIMIT, GAMMA = 7, 20, .95


def transition(state, action, elapsed):
    next_state = max(0, min(STATES-1, state + (1 if action == 1 else -1)))
    terminated = next_state == STATES-1
    truncated = elapsed >= LIMIT and not terminated
    reward = 1. if terminated else -.02
    return next_state, reward, terminated, truncated


def encode(states):
    return F.one_hot(torch.as_tensor(states, dtype=torch.long), STATES).float()


def network():
    return nn.Sequential(nn.Linear(STATES, 32), nn.ReLU(), nn.Linear(32, 2))


@torch.no_grad()
def evaluate(model, random_policy=False):
    rng = random.Random(9876)
    returns, successes, lengths = [], [], []
    # All possible nonterminal starts, repeated for random-policy variability.
    for start in list(range(STATES-1))*20:
        state, total, success = start, 0., False
        for step in range(1, LIMIT+1):
            action = rng.randrange(2) if random_policy else int(model(encode([state])).argmax(1))
            state, reward, terminal, truncated = transition(state, action, step)
            total += reward
            if terminal or truncated:
                success = terminal
                break
        returns.append(total)
        successes.append(success)
        lengths.append(step)
    n = len(returns)
    return sum(returns)/n, sum(successes)/n, sum(lengths)/n


def main():
    p = argparse.ArgumentParser()
    p.add_argument('--smoke', action='store_true')
    p.add_argument('--seed', type=int, default=7)
    args = p.parse_args()
    torch.manual_seed(args.seed)
    rng = random.Random(args.seed)
    print('PyTorch', torch.__version__, 'seed', args.seed)
    online, target = network(), network()
    target.load_state_dict(copy.deepcopy(online.state_dict()))
    target.eval()
    for parameter in target.parameters():
        parameter.requires_grad_(False)
    opt = torch.optim.Adam(online.parameters(), lr=.003)
    replay = deque(maxlen=4000)
    updates = env_steps = 0
    loss_value = float('nan')
    for episode in range(12 if args.smoke else 250):
        state = rng.randrange(STATES-1)
        epsilon = max(.08, 1.-episode/170)
        for elapsed in range(1, LIMIT+1):
            if rng.random() < epsilon:
                action = rng.randrange(2)
            else:
                with torch.no_grad():
                    action = int(online(encode([state])).argmax(1))
            next_state, reward, terminal, truncated = transition(state, action, elapsed)
            # Store actual next_state BEFORE resetting. Only terminal masks bootstrap.
            replay.append((state, action, reward, next_state, terminal, truncated))
            state = next_state
            env_steps += 1
            if len(replay) >= 32:
                batch = rng.sample(list(replay), 32)
                states, actions, rewards, next_states, terminals, _ = zip(*batch)
                rewards = torch.tensor(rewards)
                terminal_mask = torch.tensor(terminals, dtype=torch.bool)
                with torch.no_grad():
                    y = rewards + GAMMA*(~terminal_mask).float()*target(encode(next_states)).max(1).values
                assert torch.equal(y[terminal_mask], rewards[terminal_mask])
                selected = online(encode(states)).gather(1, torch.tensor(actions)[:, None]).squeeze(1)
                loss = F.smooth_l1_loss(selected, y)
                assert torch.isfinite(loss)
                opt.zero_grad()
                loss.backward()
                nn.utils.clip_grad_norm_(online.parameters(), 5.)
                opt.step()
                updates += 1
                loss_value = loss.item()
                if updates % 50 == 0:
                    target.load_state_dict(online.state_dict())
            if terminal or truncated:
                break
    online.eval()
    for name, random_policy in [('Random', True), ('DQN greedy', False)]:
        ret, success, length = evaluate(online, random_policy)
        print(f'{name}: mean undiscounted return={ret:.3f}; success={success:.3f}; mean length={length:.2f}')
    path, state = [0], 0
    with torch.no_grad():
        for step in range(1, LIMIT+1):
            action = int(online(encode([state])).argmax(1))
            state, _, terminal, truncated = transition(state, action, step)
            path.append(state)
            if terminal or truncated:
                break
    print('Greedy path from 0:', path)
    print(f'Environment steps={env_steps}; gradient updates={updates}; final TD loss={loss_value:.5f}')
    print('Evaluation covers this fixed toy world, not unseen dynamics. Repeat independent --seed values.')
    if args.smoke:
        print('Smoke run checks execution, not convergence.')


if __name__ == '__main__':
    main()

Read the output, then challenge it

• Verify the terminal target equals the immediate reward.
• Confirm time-limit transitions store their true final state and still bootstrap.
• Compare success rate and undiscounted return, not just TD loss.
• Repeat independent seeds before claiming reliable learning, and inspect the printed greedy path.

These are small mechanism studies, not benchmark claims. Seeds make runs easier to compare; versions and hardware can still change exact results. Decide your validation protocol before tuning and reserve final test results for reporting.

PREDICT → CALCULATE → EXPLAIN

Practice with worked answers

EXPLAIN IT WITHOUT THE CODE

Week 13 review questions

RESEARCH CONNECTIONS

Follow the original ideas

Course alignment: AI Algorithms 2, lecture Week 14. This site counts from Week 0. Explanations, browser labs, and runnable experiments are authored study companions to the supplied lecture material.