Representation Learning, Self-Supervision, and Pretraining

LESSON 8.1

Learn coordinates that make a task easier

A representation h=fθ(x) maps a complex input into features useful for later decisions. A good embedding can make related inputs nearby, expose predictive factors, or support a simple downstream classifier. Individual coordinates need not correspond to named concepts; useful structure can be distributed across many dimensions.

Training loss alone does not tell us whether a representation transfers. A model might solve its pretraining task using a shortcut. Evaluate it on declared downstream tasks, with held-out data and baselines that reveal what pretraining contributed.

Input x → encoder fθ → representation h → task head gψ → prediction

LESSON 8.2

Self-supervision creates targets from observations

Self-supervised learning uses supervision constructed from the data rather than requiring a person to label every example. Masked prediction hides content and asks the model to recover it. Autoregressive prediction uses earlier elements to predict a later one. Denoising predicts clean content from a corrupted observation.

Contrastive learning instead compares related and unrelated views. These objectives ask different questions and encourage different information to survive. Self-supervised does not mean target-free, bias-free, or automatically useful for every task.

LESSON 8.3

Augmentations define what may be ignored

Two views of the same example are treated as a positive pair. The model is encouraged to keep information shared by those views and to tolerate their differences. This means augmentation is part of the learning objective, not merely cosmetic preprocessing.

If a crop removes the labeled object, two views may no longer have the same semantic meaning. If color is the target signal, aggressive color jitter can erase it. Two different examples can also represent the same semantic class: treating them as negatives introduces false negatives. Inspect views, duplicates, and data provenance before tuning the loss.

Keep this idea. Every invariance is a decision about which information the model is allowed to discard.

LESSON 8.4

Pull a positive closer, compete with alternatives

A SimCLR-style pipeline maps inputs to h with an encoder and to z with a projection head. Normalize z before cosine similarity. For each anchor, the other view of the same example is positive; all other views except the anchor itself participate in the denominator.

Temperature controls score sharpness. A low value emphasizes difficult competitors and can create large, concentrated gradients. Both directions of every positive pair should contribute. The projection head serves the pretraining objective; downstream evaluation commonly uses the encoder representation h.

sᵢⱼ = cosine(zᵢ,zⱼ) / τ
ℓᵢ = −log[ exp(sᵢ,positive) / Σk≠i exp(sᵢk) ]
L = mean over all 2N anchors

LESSON 8.5

Get the pair indices and masks right

Concatenate first views followed by second views. Then anchor i's positive is (i+N) modulo 2N. Mask only the diagonal self-comparison, not the positive. Cross-entropy can treat the other-view index as the correct class.

A shape-correct implementation can still be semantically wrong if the two view batches are shuffled independently. Keep their correspondence explicit. Monitor representation variability and retrieval examples as well as the scalar loss.

h1, h2 = encoder(view1), encoder(view2)
z = F.normalize(projector(torch.cat([h1,h2])), dim=1)
N = len(view1)
logits = z @ z.T / temperature
logits.fill_diagonal_(float('-inf'))
positive = (torch.arange(2*N, device=z.device) + N) % (2*N)
loss = F.cross_entropy(logits, positive)

LESSON 8.6

Separate pretraining from downstream evidence

A linear probe freezes the encoder and fits a linear classifier using labeled training data. Fine-tuning updates some or all encoder weights too. These answer different questions: how accessible are the features already, and how well can they adapt?

Use the same downstream split, label budget, head training procedure, and validation budget when comparing pretrained and random encoders. Put a frozen encoder in evaluation mode when BatchNorm or dropout would otherwise change its behavior. Test labels must not select the pretraining checkpoint or hyperparameters.

Using unlabeled test inputs during pretraining is still access to the test distribution. It may be a valid transductive design if explicitly declared, but it differs from evaluating generalization to unseen examples. The supplied experiment excludes those inputs.

LESSON 8.7

From one task to broad reuse

Pretraining transfers parameters, not a ready-made answer to every downstream problem. Broader data, capacity, and compute can support broader reuse, but data quality, objective choice, and evaluation coverage remain decisive. A pretrained model is not automatically a foundation model simply because its weights were saved.

A two-dimensional embedding plot is a visualization of a projection, not proof of class separation in the original space. Report quantitative probe or retrieval results, label-efficiency curves, domain-shift tests, and collapse diagnostics such as per-dimension variation. Avoid interpreting a pleasing cluster plot as the entire evaluation.

WHEN THE MODEL SURPRISES YOU

Mistakes worth catching early

FROM EQUATIONS TO A RUNNING MODEL

Pretrain features, then freeze and probe

Contrastively pretrain an encoder on noisy views of unlabeled synthetic clusters. Then train linear probes on a small labeled training set for both the pretrained encoder and a random encoder. Validation selects each probe; a separate test set supplies the final comparison.

Download the script, create a Python environment, and run the commands below in the folder containing the download. The example uses PyTorch on CPU and generates its own small dataset. No dataset or model download is required.

python -m venv .venv
# macOS / Linux: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install torch
python dl-week-8.py
# Fast execution check (not a convergence result):
python dl-week-8.py --smoke

"""Week 8: contrastive pretraining plus frozen linear probes. CPU only.
python -m pip install torch
python dl-week-8.py [--smoke]
"""
import argparse
import copy
import math
import torch
from torch import nn
from torch.nn import functional as F

torch.set_num_threads(1)


def data(n, seed):
    g = torch.Generator().manual_seed(seed)
    labels = torch.arange(n) % 4
    angle = labels.float() * (math.pi / 2)
    centers = 2 * torch.stack([angle.cos(), angle.sin()], dim=1)
    return centers + .3 * torch.randn(n, 2, generator=g), labels


def encoder():
    return nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 16))


def contrastive_loss(z1, z2, temperature=.3):
    n = len(z1)
    z = F.normalize(torch.cat([z1, z2]), dim=1)
    scores = z @ z.T / temperature
    scores = scores.masked_fill(torch.eye(2*n, dtype=torch.bool), float('-inf'))
    targets = (torch.arange(2*n) + n) % (2*n)
    return F.cross_entropy(scores, targets)


def probe(backbone, train, val, test, steps):
    backbone.eval()
    for p in backbone.parameters():
        p.requires_grad_(False)
    with torch.no_grad():
        h_train, h_val, h_test = [backbone(d[0]) for d in (train, val, test)]
    torch.manual_seed(123)
    head = nn.Linear(16, 4)
    opt = torch.optim.Adam(head.parameters(), lr=.03)
    best_loss, best = float('inf'), copy.deepcopy(head.state_dict())
    for _ in range(steps):
        opt.zero_grad()
        loss = F.cross_entropy(head(h_train), train[1])
        loss.backward()
        opt.step()
        with torch.no_grad():
            v = F.cross_entropy(head(h_val), val[1]).item()
        if v < best_loss:
            best_loss, best = v, copy.deepcopy(head.state_dict())
    head.load_state_dict(best)
    with torch.no_grad():
        acc = (head(h_test).argmax(1) == test[1]).float().mean().item()
    return best_loss, acc


def main():
    p = argparse.ArgumentParser()
    p.add_argument('--smoke', action='store_true')
    args = p.parse_args()
    torch.manual_seed(7)
    print('PyTorch', torch.__version__)
    backbone = encoder()
    random_baseline = copy.deepcopy(backbone)
    projector = nn.Sequential(nn.Linear(16, 16), nn.ReLU(), nn.Linear(16, 16))
    unlabeled, _ = data(512, 10)  # labels are never used in pretraining
    # Downstream splits are generated independently of unlabeled pretraining inputs.
    train, val, test = data(32, 11), data(200, 12), data(300, 13)
    opt = torch.optim.Adam(list(backbone.parameters()) + list(projector.parameters()), lr=.003)
    uniform = contrastive_loss(torch.ones(4, 16), torch.ones(4, 16)).item()
    assert abs(uniform - math.log(7)) < 1e-5
    for step in range(8 if args.smoke else 300):
        x = unlabeled[torch.randint(len(unlabeled), (64,))]
        a, b = x + .15*torch.randn_like(x), x + .15*torch.randn_like(x)
        loss = contrastive_loss(projector(backbone(a)), projector(backbone(b)))
        assert torch.isfinite(loss)
        opt.zero_grad()
        loss.backward()
        opt.step()
    with torch.no_grad():
        variation = backbone(unlabeled).std(dim=0).mean().item()
    print(f'Final contrastive loss={loss.item():.4f}; mean feature std={variation:.4f}')
    for name, model in [('Pretrained', backbone), ('Random encoder', random_baseline)]:
        vl, acc = probe(model, train, val, test, 10 if args.smoke else 150)
        print(f'{name}: probe validation NLL={vl:.4f}; held-out accuracy={acc:.3f}')
    print('Easy clusters can also be linearly separable in random features. Do not assume pretraining must win.')
    if args.smoke:
        print('Smoke run checks execution, not convergence.')


if __name__ == '__main__':
    main()

Read the output, then challenge it

• Check positive indices and diagonal masking on a small batch before trusting loss.
• Compare feature standard deviations against a collapsed constant representation.
• Report both probes under the same label and optimization budget.
• Increase augmentation noise until it crosses cluster boundaries and explain the effect on semantics.

These are small mechanism studies, not benchmark claims. Seeds make runs easier to compare; versions and hardware can still change exact results. Decide your validation protocol before tuning and reserve final test results for reporting.

PREDICT → CALCULATE → EXPLAIN

Practice with worked answers

EXPLAIN IT WITHOUT THE CODE

Week 8 review questions

RESEARCH CONNECTIONS

Follow the original ideas

Course alignment: AI Algorithms 2, lecture Week 9. This site counts from Week 0. Explanations, browser labs, and runnable experiments are authored study companions to the supplied lecture material.