Generative Adversarial Networks

LESSON 10.1

Two models with different jobs

A generator G maps random noise to a candidate observation. A discriminator D estimates whether an input came from the data or the generator under the training mixture. There is no paired target observation for a particular noise vector; the objective compares distributions through a learned critic.

Each update changes the problem faced by the other model. A discriminator loss that improves may mean it has learned a useful distinction, or that the generator has fallen behind. Neither loss is a direct measure of perceptual quality or full distribution coverage.

z ~ p(z), fake = Gθ(z)
Original game: minG maxD Edata[log D(x)] + Ez[log(1−D(G(z)))]

LESSON 10.2

Train the discriminator on real and detached fake samples

Using a discriminator that returns logits, binary cross-entropy with logits combines sigmoid and the log terms stably. Real examples receive target one and generated examples target zero. During the discriminator update, detach generated samples so the backward pass does not update the generator.

Batch balance, label conventions, preprocessing, and gradient resets must be explicit. If fake and real inputs have different scaling or trivial artifacts, D may solve an easier problem than the one you intended.

opt_d.zero_grad()
fake = generator(noise).detach()
loss_d = F.binary_cross_entropy_with_logits(discriminator(real), torch.ones(N,1))
loss_d += F.binary_cross_entropy_with_logits(discriminator(fake), torch.zeros(N,1))
loss_d.backward()
opt_d.step()

LESSON 10.3

Why the non-saturating objective helps

The original minimax generator minimizes log(1−D(fake)). When the discriminator confidently rejects fake samples, the derivative with respect to its fake logit is small. The common non-saturating alternative minimizes −log D(fake), giving a larger logit-level signal in that regime.

This improves one part of the gradient path, not every part. The discriminator's input Jacobian and generator Jacobian still determine the parameter gradient. A large logit derivative cannot rescue an uninformative or poorly conditioned mapping everywhere.

Let a be the fake logit, D(fake)=sigmoid(a).
Minimax LG = log(1−sigmoid(a)); dLG/da = −sigmoid(a)
Non-saturating LG = −log(sigmoid(a)); dLG/da = sigmoid(a)−1

LESSON 10.4

Freeze weights without cutting the input gradient

During a generator step, discriminator parameters can be frozen while gradients still flow through the discriminator to its input and then to G. Wrapping D in no_grad() would cut that required path. Detaching the fake during this step would also prevent G from learning.

requires_grad controls parameter gradients; train/eval controls mode-dependent behavior. BatchNorm buffers can still change while parameters are frozen. Decide whether D's running state should update during generator steps. The supplied toy MLP avoids BatchNorm so this issue is visible conceptually without hidden state changes.

for p in discriminator.parameters():
    p.requires_grad_(False)
opt_g.zero_grad()
fake = generator(torch.randn(N, noise_dim))
loss_g = F.binary_cross_entropy_with_logits(discriminator(fake), torch.ones(N,1))
loss_g.backward()  # gradient passes through D to G
opt_g.step()
for p in discriminator.parameters():
    p.requires_grad_(True)

LESSON 10.5

Convolutional and conditional generation

Image generators can progressively increase spatial resolution using upsampling plus convolution or transposed convolution. A transposed convolution is not generally the inverse of a convolution. Uneven overlap can produce checkerboard artifacts, so inspect the upsampling design and actual outputs.

A conditional GAN supplies context, such as a class label, to both G and D. The discriminator must judge whether an observation matches the condition as well as the data distribution. Conditioning does not by itself create a paired reconstruction objective for each noise draw.

LESSON 10.6

Good-looking samples can still miss most of the data

Mode collapse means the generator produces insufficient variety, perhaps mapping many noise inputs to similar outputs. A few convincing examples can conceal missing classes or modes. Compare fixed-latent snapshots, fresh samples, coverage measures, and nearest training neighbors.

Oscillating losses are not a unique diagnosis: they may reflect ordinary game dynamics, instability, or changing discriminator skill. Check gradients and sample evolution. Adjusting learning rates, update ratios, or regularization should follow a testable hypothesis rather than treating every curve fluctuation as failure.

Keep this idea. Quality and diversity are separate evaluation questions. A generator must address both.

LESSON 10.7

Choose metrics appropriate to the domain

FID compares feature-distribution summaries, so its interpretation depends on feature extractor, preprocessing, domain, and sample count. It is not a universal quality number that can be compared across incompatible pipelines. Small sample estimates can also be misleading.

For a two-dimensional toy mixture, directly measure distance to known modes and coverage of those modes. For images, combine compatible distribution metrics with diverse sample inspection and duplication checks. Keep the final evaluation protocol separate from repeated tuning on its results.

WHEN THE MODEL SURPRISES YOU

Mistakes worth catching early

FROM EQUATIONS TO A RUNNING MODEL

An adversarial game on eight modes

Train small MLP generator and discriminator networks on an eight-cluster ring. The script checks that generator gradients pass through a frozen discriminator, reports losses and coverage, and exports generated points to CSV. It uses a fixed training budget rather than selecting a snapshot from the final metrics.

Download the script, create a Python environment, and run the commands below in the folder containing the download. The example uses PyTorch on CPU and generates its own small dataset. No dataset or model download is required.

python -m venv .venv
# macOS / Linux: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install torch
python dl-week-10.py
# Fast execution check (not a convergence result):
python dl-week-10.py --smoke

"""Week 10: non-saturating GAN on eight Gaussian modes. CPU only.
python -m pip install torch
python dl-week-10.py [--smoke]
Writes dl-week-10-samples.csv in the working directory.
"""
import argparse
import csv
import math
import torch
from torch import nn
from torch.nn import functional as F

torch.set_num_threads(1)
ANGLES = torch.arange(8) * (2*math.pi/8)
CENTERS = 2*torch.stack([ANGLES.cos(), ANGLES.sin()], dim=1)


def real_samples(n):
    return CENTERS[torch.randint(8, (n,))] + .12*torch.randn(n, 2)


def network(input_dim, output_dim):
    return nn.Sequential(nn.Linear(input_dim, 64), nn.LeakyReLU(.2),
                         nn.Linear(64, 64), nn.LeakyReLU(.2), nn.Linear(64, output_dim))


def diagnostics(points):
    distance, mode = torch.cdist(points, CENTERS).min(dim=1)
    near = distance < .4  # declared diagnostic radius; not tuned on outputs
    counts = torch.bincount(mode[near], minlength=8)
    return int((counts >= 5).sum()), distance.mean().item(), counts.tolist()


def main():
    p = argparse.ArgumentParser()
    p.add_argument('--smoke', action='store_true')
    args = p.parse_args()
    torch.manual_seed(7)
    print('PyTorch', torch.__version__)
    generator, discriminator = network(4, 2), network(2, 1)
    opt_g = torch.optim.Adam(generator.parameters(), lr=.001, betas=(.5, .999))
    opt_d = torch.optim.Adam(discriminator.parameters(), lr=.001, betas=(.5, .999))
    fixed_z = torch.randn(128, 4)
    # Check routing independently of the training loop.
    for par in discriminator.parameters():
        par.requires_grad_(False)
    test_loss = F.softplus(-discriminator(generator(torch.randn(16, 4)))).mean()
    test_loss.backward()
    assert any(par.grad is not None and par.grad.abs().sum() > 0 for par in generator.parameters())
    assert all(par.grad is None for par in discriminator.parameters())
    generator.zero_grad(set_to_none=True)
    for par in discriminator.parameters():
        par.requires_grad_(True)
    print('Gradient routing: generator receives gradients through frozen discriminator.')
    steps = 12 if args.smoke else 1800
    for step in range(steps):
        real = real_samples(128)
        fake = generator(torch.randn(128, 4)).detach()
        opt_d.zero_grad(set_to_none=True)
        # softplus(-real_logit) + softplus(fake_logit) = BCE logits targets 1 and 0.
        loss_d = F.softplus(-discriminator(real)).mean() + F.softplus(discriminator(fake)).mean()
        loss_d.backward()
        opt_d.step()
        opt_d.zero_grad(set_to_none=True)
        for par in discriminator.parameters():
            par.requires_grad_(False)
        opt_g.zero_grad(set_to_none=True)
        loss_g = F.softplus(-discriminator(generator(torch.randn(128, 4)))).mean()
        loss_g.backward()
        opt_g.step()
        for par in discriminator.parameters():
            par.requires_grad_(True)
        assert torch.isfinite(loss_d) and torch.isfinite(loss_g)
        if step % 300 == 0:
            with torch.no_grad():
                _, dist, _ = diagnostics(generator(fixed_z))
            print(f'step={step} D={loss_d.item():.4f} G={loss_g.item():.4f} fixed-z mode distance={dist:.3f}')
    generator.eval()
    with torch.no_grad():
        samples = generator(torch.randn(1000, 4))
        coverage, distance, counts = diagnostics(samples)
        real_coverage, real_distance, _ = diagnostics(real_samples(1000))
    print(f'Generated: {coverage}/8 modes with >=5 samples within radius .4; mean mode distance={distance:.3f}')
    print('Counts within radius:', counts)
    print(f'Fresh real reference: {real_coverage}/8 modes; mean distance={real_distance:.3f}')
    with open('dl-week-10-samples.csv', 'w', newline='') as f:
        writer = csv.writer(f)
        writer.writerow(['x', 'y'])
        writer.writerows(samples.tolist())
    print('Fixed-budget result, no final-metric checkpoint selection. Repeat seeds; losses are not quality scores.')
    if args.smoke:
        print('Smoke run checks execution, not convergence.')


if __name__ == '__main__':
    main()

Read the output, then challenge it

• Read the gradient-routing assertion before interpreting training behavior.
• Compare the number of covered modes and distance to the nearest true mode.
• Plot the sample CSV; a reasonable mean and variance can still hide collapse.
• Repeat seeds and vary the D:G update budget without selecting only attractive runs.

These are small mechanism studies, not benchmark claims. Seeds make runs easier to compare; versions and hardware can still change exact results. Decide your validation protocol before tuning and reserve final test results for reporting.

PREDICT → CALCULATE → EXPLAIN

Practice with worked answers

EXPLAIN IT WITHOUT THE CODE

Week 10 review questions

RESEARCH CONNECTIONS

Follow the original ideas

Course alignment: AI Algorithms 2, lecture Week 11. This site counts from Week 0. Explanations, browser labs, and runnable experiments are authored study companions to the supplied lecture material.