Autoencoders and Variational Autoencoders

LESSON 9.1

Compress, then reconstruct

An autoencoder maps x to a code z and decodes that code to a reconstruction. A bottleneck, sparsity, corruption, or other constraint makes copying less trivial. A high-capacity unconstrained model can approximate identity without learning the representation we hoped for.

Reconstruction emphasizes information needed by its loss. A feature important for classification may contribute little to pixel error; nuisance detail may contribute a lot. A linear autoencoder with suitable squared-error conditions connects to PCA subspaces, but arbitrary nonlinear autoencoders do not inherit every PCA property.

z = fφ(x), x̂ = gθ(z)
Lreconstruction = mean over examples of a per-example distortion

LESSON 9.2

A loss is an assumption about observations

Squared error corresponds, up to constants and scaling, to a Gaussian observation model with fixed variance. Binary cross-entropy corresponds to a Bernoulli observation model. Rescaling arbitrary continuous pixels into [0,1] does not by itself make them Bernoulli observations.

A denoising autoencoder receives corrupted input but reconstructs a clean target. It learns a mapping tied to that corruption process. This can discourage literal copying, although success on one noise type does not establish robustness to every shift.

LESSON 9.3

Why an ordinary autoencoder is not enough for sampling

A deterministic encoder can place training codes in disconnected or irregular regions. Drawing a random point between or beyond those codes may visit areas the decoder never learned. Reconstruction alone supplies neither a useful sampling prior nor a complete probabilistic generative model.

A VAE defines a prior p(z), a decoder likelihood pθ(x|z), and an approximate posterior qφ(z|x). Commonly p(z) is standard normal and q is diagonal Gaussian, with the encoder producing a mean and log-variance for each latent coordinate.

p(z) = N(0,I)
qφ(z|x) = N(μ(x), diag(exp(logvar(x))))
pθ(x) = ∫ pθ(x|z) p(z) dz

LESSON 9.4

Move randomness outside the learned parameters

Sampling z directly from a parameterized Gaussian obscures the simple path from parameters to the sample. Reparameterization draws independent standard noise ε, then constructs z as a differentiable function of μ and σ. Randomness remains; gradients can now flow through the transformation.

The network predicts log-variance for numerical convenience. Standard deviation is exp(0.5·logvar), not exp(logvar). Large log-variances can still cause unstable exponentials, so inspect their values and the training objective.

ε ~ N(0,I)
σ = exp(0.5·logvar)
z = μ + σ ⊙ ε

mu, logvar = encoder(x).chunk(2, dim=-1)
std = torch.exp(0.5 * logvar)
z = mu + std * torch.randn_like(std)
reconstruction = decoder(z)

LESSON 9.5

Balance reconstruction and prior agreement

The evidence lower bound is expected log-likelihood minus the posterior-to-prior KL divergence. Maximizing it encourages plausible reconstruction while limiting how far each approximate posterior departs from the prior. Its negative is a reconstruction negative log-likelihood plus KL.

For diagonal Gaussian q and standard-normal p, KL is available analytically. Sum over latent dimensions, then average over examples. Similarly, state how reconstruction is reduced over input features. Changing one reduction silently changes the relative weight. β=1 gives the ordinary negative ELBO; other β values define a modified objective.

ELBO = Eq[log pθ(x|z)] − KL(qφ(z|x) || p(z))
KL = ½ Σj [μj² + exp(logvarj) − 1 − logvarj]
Lβ = reconstruction NLL + β·KL

LESSON 9.6

Separate three different questions

Reconstruction asks whether the model can recover an observation from its inferred code. Prior sampling asks whether z drawn from p(z) decodes into plausible new observations. Interpolation asks what the decoder does along a chosen latent path. These are different tests.

A decoder may output the mean of pθ(x|z). Displaying that mean is different from sampling the observation distribution itself. State which you show. Inspect distribution coverage as well as individual attractive examples; a smooth interpolation does not prove the learned marginal matches the data.

LESSON 9.7

Detect when the decoder ignores the latent

Posterior collapse occurs when the latent conveys little useful information and the decoder relies mostly on its own capacity. Small KL can be a symptom, but KL alone is not sufficient proof. Compare reconstructions after replacing or shuffling codes, inspect active dimensions, and check conditional variation.

A larger KL is not automatically better either. It may reflect poor prior agreement without improving useful information. Evaluate reconstruction, prior samples, and latent interventions together. β schedules, architecture changes, and capacity constraints are hypotheses to validate, not guaranteed fixes.

WHEN THE MODEL SURPRISES YOU

Mistakes worth catching early

FROM EQUATIONS TO A RUNNING MODEL

A VAE on a ring of small clusters

Train a two-dimensional latent VAE on a synthetic mixture. The script uses a fixed-variance Gaussian decoder, explicit feature-summed negative log-likelihood and latent-summed KL, validation selection, and held-out reconstruction. It exports decoder means from prior draws to CSV.

Download the script, create a Python environment, and run the commands below in the folder containing the download. The example uses PyTorch on CPU and generates its own small dataset. No dataset or model download is required.

python -m venv .venv
# macOS / Linux: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install torch
python dl-week-9.py
# Fast execution check (not a convergence result):
python dl-week-9.py --smoke

"""Week 9: VAE with explicit Gaussian likelihood and KL reductions.
python -m pip install torch
python dl-week-9.py [--smoke] [--beta 1.0]
Writes dl-week-9-samples.csv in the working directory (decoder means).
"""
import argparse
import copy
import csv
import math
import torch
from torch import nn

torch.set_num_threads(1)
OBS_VARIANCE = .1


def data(n, seed):
    g = torch.Generator().manual_seed(seed)
    k = torch.randint(8, (n,), generator=g)
    angles = k * (2*math.pi/8)
    return 2*torch.stack([angles.cos(), angles.sin()], 1) + .12*torch.randn(n, 2, generator=g)


class VAE(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(nn.Linear(2, 48), nn.Tanh(), nn.Linear(48, 4))
        self.decoder = nn.Sequential(nn.Linear(2, 48), nn.Tanh(), nn.Linear(48, 48), nn.Tanh(), nn.Linear(48, 2))

    def forward(self, x, eps):
        mu, logvar = self.encoder(x).chunk(2, dim=1)
        z = mu + (.5*logvar).exp()*eps
        return self.decoder(z), mu, logvar


def terms(model, x, eps):
    mean, mu, logvar = model(x, eps)
    # Sum observation features and latent coordinates, then average examples.
    nll = .5*((x-mean).square()/OBS_VARIANCE + math.log(2*math.pi*OBS_VARIANCE)).sum(1).mean()
    kl = .5*(mu.square() + logvar.exp() - 1 - logvar).sum(1).mean()
    return nll, kl


def main():
    p = argparse.ArgumentParser()
    p.add_argument('--smoke', action='store_true')
    p.add_argument('--beta', type=float, default=1.)
    args = p.parse_args()
    if args.beta < 0:
        p.error('--beta must be nonnegative')
    torch.manual_seed(7)
    print('PyTorch', torch.__version__)
    train, val, test = data(1024, 1), data(256, 2), data(512, 3)
    # Fixed validation noise makes checkpoint comparisons less noisy.
    eps_val = torch.randn(val.shape, generator=torch.Generator().manual_seed(4))
    model = VAE()
    opt = torch.optim.Adam(model.parameters(), lr=.003)
    best_value, best = float('inf'), copy.deepcopy(model.state_dict())
    for step in range(12 if args.smoke else 800):
        x = train[torch.randint(len(train), (128,))]
        nll, kl = terms(model, x, torch.randn_like(x))
        loss = nll + args.beta*kl
        assert torch.isfinite(loss)
        opt.zero_grad()
        loss.backward()
        opt.step()
        if step % 10 == 0:
            with torch.no_grad():
                vn, vk = terms(model, val, eps_val)
                score = (vn + args.beta*vk).item()
            if score < best_value:
                best_value, best = score, copy.deepcopy(model.state_dict())
    model.load_state_dict(best)
    model.eval()
    with torch.no_grad():
        nll, kl = terms(model, test, torch.randn_like(test))
        mu, _ = model.encoder(test).chunk(2, dim=1)
        ordinary = (model.decoder(mu)-test).square().mean().item()
        shuffled = (model.decoder(mu[torch.randperm(len(mu))])-test).square().mean().item()
        samples = model.decoder(torch.randn(1000, 2))
    assert torch.isfinite(samples).all()
    print(f'beta={args.beta:g}; test reconstruction NLL={nll.item():.4f}; KL={kl.item():.4f}')
    print(f'Decoder-at-posterior-mean MSE={ordinary:.4f}; shuffled-code MSE={shuffled:.4f}')
    with open('dl-week-9-samples.csv', 'w', newline='') as f:
        writer = csv.writer(f)
        writer.writerow(['decoder_mean_x', 'decoder_mean_y'])
        writer.writerows(samples.tolist())
    print('Saved decoder means from prior draws, not full Gaussian observation samples.')
    print('Inspect coverage, reconstructions, and latent interventions together; KL alone is not a collapse test.')
    if args.smoke:
        print('Smoke run checks execution, not convergence.')


if __name__ == '__main__':
    main()

Read the output, then challenge it

• Inspect reconstruction and KL separately; a lower combined objective can hide a tradeoff.
• Plot the exported prior decoder means and compare their mode coverage with the data.
• Compare reconstruction using inferred means versus shuffled latent means.
• Vary β while keeping reductions, splits, and training budget unchanged.

These are small mechanism studies, not benchmark claims. Seeds make runs easier to compare; versions and hardware can still change exact results. Decide your validation protocol before tuning and reserve final test results for reporting.

PREDICT → CALCULATE → EXPLAIN

Practice with worked answers

EXPLAIN IT WITHOUT THE CODE

Week 9 review questions

RESEARCH CONNECTIONS

Follow the original ideas

Course alignment: AI Algorithms 2, lecture Week 10. This site counts from Week 0. Explanations, browser labs, and runnable experiments are authored study companions to the supplied lecture material.