Diffusion Models

LESSON 11.1

A known process gradually removes structure

Choose a variance schedule β₁,…,βT and define αt=1−βt. Each forward step shrinks the previous signal and adds Gaussian noise. Multiplying α values gives ᾱt, the retained signal power relative to the original observation.

The Gaussian composition permits sampling a noisy observation directly at any t without simulating every earlier step. This is why training can sample independent time steps efficiently. A terminal distribution close to standard normal requires ᾱT to be sufficiently small for the data scale.

xₜ = √αₜ xₜ₋₁ + √βₜ εₜ
ᾱₜ = ∏s=1…t αs
xₜ = √ᾱₜ x₀ + √(1−ᾱₜ) ε, ε~N(0,I)

LESSON 11.2

Predict the noise using time as context

Sample a training observation, a time index, and fresh Gaussian noise. Construct x_t with the forward formula. A neural network receives x_t and t and predicts the added noise. The standard simple objective is squared prediction error.

Time conditioning tells the network how corrupted its input is. Without it, the same observed value could require different denoising behavior at different noise levels. Ensure schedule coefficients broadcast across features, not accidentally across batch members.

Lsimple = Ex₀,t,ε [ ||ε − εθ(xₜ,t)||² ]

t = torch.randint(0, T, (len(x0),))
a_bar = alpha_bar[t].view(-1, 1)  # [N,1] for vector data
noise = torch.randn_like(x0)
xt = a_bar.sqrt()*x0 + (1-a_bar).sqrt()*noise
loss = F.mse_loss(model(xt, t), noise)
# For [N,C,H,W] images, coefficients need shape [N,1,1,1].

LESSON 11.3

Denoising is a scheduled reverse transition

A DDPM sampler starts near the terminal noise distribution and runs backward. At each step it uses the noise predictor to form a reverse mean, then adds the prescribed reverse variance except at the final step. Simply subtracting the full predicted noise repeatedly is not the DDPM update.

Below, equations use mathematical times 1…T and ᾱ₀=1. The downloadable script stores those transitions at array indices 0…T−1; its previous cumulative product is therefore [1, ᾱ₁,…]. State this convention to avoid off-by-one errors.

μθ(xₜ,t) = [xₜ − βₜ εθ(xₜ,t)/√(1−ᾱₜ)] / √αₜ
β̃ₜ = βₜ (1−ᾱₜ₋₁)/(1−ᾱₜ)
xₜ₋₁ = μθ(xₜ,t) + √β̃ₜ z, z~N(0,I), for t>1
x₀ = μθ(x₁,1) at the final step

LESSON 11.4

What a predicted clean observation means

A predicted noise vector can be converted into a predicted clean observation. With oracle noise from the forward construction this exactly recovers the original, up to floating-point error. With a learned estimate it is a model prediction and may be wrong.

At high noise, many original observations could explain the same noisy input. The model learns a conditional denoising relationship rather than magically identifying which historical sample produced it. Under the noise-prediction setup, the optimal noise prediction is related to the noisy-data score by a negative scale factor.

x̂₀ = [xₜ − √(1−ᾱₜ) εθ(xₜ,t)] / √ᾱₜ
At the optimal predictor:
∇xₜ log pₜ(xₜ) = −E[ε | xₜ] / √(1−ᾱₜ)

Keep this idea. An oracle algebra check validates the schedule and broadcasting. It does not establish sample quality.

LESSON 11.5

Condition a network at several scales

For images, a U-Net combines downsampling, broad context, and upsampling, passing features between matching resolutions. Such skips are often concatenations; they are not automatically the same operation as a residual addition. Time embeddings modulate the model so it can adapt across noise levels.

The toy experiment uses a small MLP because two-dimensional points do not need an image hierarchy. This keeps the forward and reverse mechanisms inspectable without claiming to reproduce the quality or cost of an image diffusion system.

LESSON 11.6

Classifier-free guidance needs conditional training

A conditional model can receive a label, text representation, or other context. Classifier-free guidance combines conditional and unconditional predictions. The model must have learned an unconditional path, commonly through random condition dropping during training.

For the formula below, w=0 gives the unconditional prediction and w=1 gives the conditional one. Larger values extrapolate their difference. This can improve condition adherence while reducing diversity or introducing artifacts. The unconditional toy script does not claim to implement guided image generation.

εguided = εunconditional + w(εconditional − εunconditional)

LESSON 11.7

Separate regression quality from generation quality

A lower training noise loss does not guarantee uniformly better samples. Difficulty varies across noise levels, and the reverse trajectory repeatedly uses model estimates. Inspect loss by time range, sample diversity, distribution coverage, and sampling cost.

A VAE can decode one latent draw in one pass; a GAN can generate in one generator pass; ordinary DDPM sampling requires many denoiser calls. Compare quality and coverage together with that cost. Faster samplers are separate methods with their own assumptions, not simply arbitrary step removal from a DDPM loop.

WHEN THE MODEL SURPRISES YOU

Mistakes worth catching early

FROM EQUATIONS TO A RUNNING MODEL

Train and sample a complete toy DDPM

Train a time-conditioned noise predictor on an eight-mode point distribution and sample it with the full reverse mean and posterior-variance schedule. A direct oracle inversion assertion checks the forward math. Generated points are exported to CSV for comparison with the VAE and GAN examples.

Download the script, create a Python environment, and run the commands below in the folder containing the download. The example uses PyTorch on CPU and generates its own small dataset. No dataset or model download is required.

python -m venv .venv
# macOS / Linux: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install torch
python dl-week-11.py
# Fast execution check (not a convergence result):
python dl-week-11.py --smoke

"""Week 11: complete DDPM on an eight-mode point distribution. CPU only.
python -m pip install torch
python dl-week-11.py [--smoke]
Writes dl-week-11-samples.csv in the working directory.
"""
import argparse
import copy
import csv
import math
import torch
from torch import nn
from torch.nn import functional as F

torch.set_num_threads(1)
T = 64
# A deliberately short, strong schedule for two-dimensional toy data.
BETAS = torch.linspace(.02, .2, T)
ALPHAS = 1-BETAS
ALPHA_BAR = ALPHAS.cumprod(0)
PREVIOUS = torch.cat([torch.ones(1), ALPHA_BAR[:-1]])
POSTERIOR_VARIANCE = BETAS*(1-PREVIOUS)/(1-ALPHA_BAR)
ANGLES = torch.arange(8)*(2*math.pi/8)
CENTERS = 2*torch.stack([ANGLES.cos(), ANGLES.sin()], 1)


def data(n, seed):
    g = torch.Generator().manual_seed(seed)
    return CENTERS[torch.randint(8, (n,), generator=g)] + .12*torch.randn(n, 2, generator=g)


class Denoiser(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(6, 64), nn.SiLU(), nn.Linear(64, 64),
                                 nn.SiLU(), nn.Linear(64, 2))

    def forward(self, x, t):
        phase = t.float()[:, None]/(T-1)
        time = torch.cat([phase, phase.square(), (2*math.pi*phase).sin(), (2*math.pi*phase).cos()], 1)
        return self.net(torch.cat([x, time], 1))


def corrupt(x0, t, noise):
    a = ALPHA_BAR[t, None]
    return a.sqrt()*x0 + (1-a).sqrt()*noise


@torch.no_grad()
def sample(model, count):
    x = torch.randn(count, 2)
    # Array index 0 corresponds to mathematical forward transition t=1.
    for index in reversed(range(T)):
        t = torch.full((count,), index, dtype=torch.long)
        predicted_noise = model(x, t)
        mean = (x-BETAS[index]/(1-ALPHA_BAR[index]).sqrt()*predicted_noise)/ALPHAS[index].sqrt()
        x = mean + POSTERIOR_VARIANCE[index].sqrt()*torch.randn_like(x) if index > 0 else mean
    return x


def main():
    p = argparse.ArgumentParser()
    p.add_argument('--smoke', action='store_true')
    args = p.parse_args()
    torch.manual_seed(7)
    print('PyTorch', torch.__version__, 'terminal alpha_bar', ALPHA_BAR[-1].item())
    train, val = data(2048, 1), data(512, 2)
    val_t, val_noise = torch.randint(T, (len(val),)), torch.randn_like(val)
    val_noisy = corrupt(val, val_t, val_noise)
    a = ALPHA_BAR[val_t, None]
    recovered = (val_noisy-(1-a).sqrt()*val_noise)/a.sqrt()
    oracle_error = (recovered-val).abs().max().item()
    assert oracle_error < 1e-4
    assert POSTERIOR_VARIANCE[0] == 0
    print('Oracle inversion max error:', oracle_error)
    model = Denoiser()
    opt = torch.optim.Adam(model.parameters(), lr=.002)
    best_score, best = float('inf'), copy.deepcopy(model.state_dict())
    for step in range(15 if args.smoke else 2200):
        x = train[torch.randint(len(train), (128,))]
        t, noise = torch.randint(T, (len(x),)), torch.randn_like(x)
        loss = F.mse_loss(model(corrupt(x, t, noise), t), noise)
        assert torch.isfinite(loss)
        opt.zero_grad()
        loss.backward()
        opt.step()
        if step % 10 == 0:
            with torch.no_grad():
                score = F.mse_loss(model(val_noisy, val_t), val_noise).item()
            if score < best_score:
                best_score, best = score, copy.deepcopy(model.state_dict())
    model.load_state_dict(best)
    model.eval()
    points = sample(model, 1000)
    assert torch.isfinite(points).all()
    distance, nearest = torch.cdist(points, CENTERS).min(1)
    counts = torch.bincount(nearest[distance < .4], minlength=8)
    print(f'Validation noise MSE={best_score:.4f}; generated mean mode distance={distance.mean().item():.3f}')
    print(f'Modes with >=5 samples within radius .4: {int((counts>=5).sum())}/8; counts={counts.tolist()}')
    print(f'Sampling cost: {T} denoiser calls per sample (batched).')
    with open('dl-week-11-samples.csv', 'w', newline='') as f:
        writer = csv.writer(f)
        writer.writerow(['x', 'y'])
        writer.writerows(points.tolist())
    print('Compare coverage and costs with the other toy generators; this schedule is not an image-model recipe.')
    if args.smoke:
        print('Smoke run checks execution, not convergence.')


if __name__ == '__main__':
    main()

Read the output, then challenge it

• Read the reported terminal ᾱ and verify the oracle inversion error is tiny.
• Compare fixed validation noise-prediction loss with training progress.
• Plot generated points and inspect all modes, including missing or blurred ones.
• Report the number of denoiser evaluations per sample alongside sample quality.

These are small mechanism studies, not benchmark claims. Seeds make runs easier to compare; versions and hardware can still change exact results. Decide your validation protocol before tuning and reserve final test results for reporting.

PREDICT → CALCULATE → EXPLAIN

Practice with worked answers

EXPLAIN IT WITHOUT THE CODE

Week 11 review questions

RESEARCH CONNECTIONS

Follow the original ideas

Course alignment: AI Algorithms 2, lecture Week 12. This site counts from Week 0. Explanations, browser labs, and runnable experiments are authored study companions to the supplied lecture material.