Interpreting and Stress-Testing Black Boxes

LESSON 12.1

A confident prediction can still be wrong

Accuracy measures the fraction of correct decisions. Calibration asks whether stated probabilities agree with observed frequencies. Among predictions assigned confidence near 0.8, roughly 80% should be correct for that confidence interpretation to hold in the evaluated population.

Softmax normalizes logits; it does not certify that an input is familiar or that the model understands its own uncertainty. A model can be highly confident on out-of-distribution inputs. Entropy summarizes one predictive distribution, but low entropy does not prove reliability.

Accuracy = mean[ predicted label = true label ]
Entropy(p) = −Σc pc log pc
NLL = −mean[log p(true class)]

LESSON 12.2

Measure confidence against outcomes

A reliability diagram bins predictions by confidence and compares each bin's mean confidence with its empirical accuracy. Expected calibration error weights each bin's absolute gap by its fraction of examples. Empty bins contribute zero.

ECE depends on binning and sample count and can conceal subgroup failures. Report bin counts and supplement it with a proper scoring rule such as negative log-likelihood or Brier score. Small differences on a tiny test set should not be interpreted as precise reliability improvements.

ECE = Σb (nb/N) · | accuracy(b) − confidence(b) |

LESSON 12.3

Adjust confidence without changing the winning class

Temperature scaling divides logits by a positive scalar T. Fit T by minimizing calibration-set NLL after freezing the trained model. T>1 softens probabilities; 0<T<1 sharpens them. A positive scalar preserves logit ordering and therefore argmax, apart from numerical or tie-handling effects.

Use separate data for fitting model parameters, selecting model settings, fitting temperature, and final testing when feasible. Fitting T on the final test set would contaminate the reported result. Calibration learned in one distribution may fail after a shift.

pT(y=c|x) = softmax(logits(x)/T)c, T>0
Fit log(T) on calibration data; keep the classifier fixed.

# logits_cal are detached outputs from the frozen model.
log_temperature = nn.Parameter(torch.zeros(()))
T = log_temperature.exp()
calibration_loss = F.cross_entropy(logits_cal / T, labels_cal)

LESSON 12.4

Different sources of uncertainty

Aleatoric uncertainty describes variability inherent in observations or labels under a model of the task. Epistemic uncertainty describes lack of knowledge about the model or its parameters. These distinctions are useful, but exact decompositions depend on assumptions and cannot be read directly from one softmax score.

Ensembles can reveal disagreement by averaging predicted probabilities from independently trained models. Monte Carlo dropout uses repeated stochastic predictions, but activating the whole model with train() also changes BatchNorm behavior. Enable the intended dropout layers while keeping BatchNorm in evaluation mode.

Keep this idea. Uncertainty estimates require their own validation under the conditions in which decisions will be made.

LESSON 12.5

Ask what an explanation actually measures

Input gradients measure local sensitivity of a chosen output to small input changes. Integrated gradients aggregate gradients along a path from a chosen baseline. Occlusion removes or replaces a region and measures the prediction change. Each answers a different intervention question.

Baselines, correlated features, saturation, and unrealistic perturbations can change conclusions. An occluded image may lie outside the training distribution. A saliency map is not proof of a causal mechanism in the world. Check whether explanations change appropriately when labels or model weights are randomized, and compare more than one view of the evidence.

Integrated gradient for coordinate i:
IGᵢ(x) = (xᵢ−x′ᵢ) ∫₀¹ ∂F(x′+α(x−x′))/∂xᵢ dα
x′ is the chosen baseline; F is a specified model output.

LESSON 12.6

Make failure conditions explicit

Test corruptions, acquisition changes, subgroup slices, sequence lengths, and other shifts relevant to the intended use. Record exactly which transformation was applied, its strength, and whether the label should remain valid. Report group counts and variation rather than averaging away the worst cases.

A bounded adversarial perturbation can test local sensitivity of your own model. FGSM takes one step in the sign of the input loss gradient, with a declared norm budget and valid input bounds. The budget must be expressed in the correct input units, especially when inputs are normalized. One attack is a local test, not a general robustness certificate.

xadv = clip_to_valid_domain(x + ε · sign(∇x L(model(x),y)))
Report ε, input scale, clipping, and the evaluated model.

LESSON 12.7

Turn observations into an honest model card

A model card records intended use, unsuitable uses, data provenance, preprocessing, training setup, evaluation splits, overall and sliced metrics, calibration, stress behavior, cost, and known limitations. It should make the evidence reviewable rather than merely describe the architecture.

A useful conclusion connects a claim to a result and its scope. For example: temperature scaling reduced NLL on this held-out distribution while preserving accuracy; under the declared corruption, both accuracy and calibration worsened. Such a statement is more informative than calling the model trustworthy without qualification.

WHEN THE MODEL SURPRISES YOU

Mistakes worth catching early

FROM EQUATIONS TO A RUNNING MODEL

Calibrate a classifier, then stress it

Train a small classifier on synthetic two-dimensional data using distinct training, validation, calibration, and test splits. Fit temperature only on calibration logits, compare clean test metrics before and after scaling, and evaluate controlled feature noise plus a bounded FGSM perturbation of this local model.

Download the script, create a Python environment, and run the commands below in the folder containing the download. The example uses PyTorch on CPU and generates its own small dataset. No dataset or model download is required.

python -m venv .venv
# macOS / Linux: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
python -m pip install torch
python dl-week-12.py
# Fast execution check (not a convergence result):
python dl-week-12.py --smoke

"""Week 12: calibration and local stress tests of a synthetic classifier.
python -m pip install torch
python dl-week-12.py [--smoke]
Separate train/validation/calibration/test sets. CPU, no external data.
"""
import argparse
import copy
import torch
from torch import nn
from torch.nn import functional as F

torch.set_num_threads(1)


def data(n, seed):
    g = torch.Generator().manual_seed(seed)
    x = 2*torch.rand(n, 2, generator=g)-1
    y = (x[:, 0]*x[:, 1] > 0).long()
    # Irreducible label noise in this synthetic observation model.
    flip = torch.rand(n, generator=g) < .1
    y[flip] = 1-y[flip]
    return x, y


@torch.no_grad()
def metrics(logits, labels, temperature, bins=10):
    p = (logits/temperature).softmax(1)
    confidence, prediction = p.max(1)
    correct = prediction.eq(labels).float()
    bin_index = (confidence*bins).long().clamp(max=bins-1)
    ece, counts = 0., []
    for i in range(bins):
        chosen = bin_index == i
        counts.append(int(chosen.sum()))
        if chosen.any():
            ece += chosen.float().mean().item() * abs(correct[chosen].mean().item()-confidence[chosen].mean().item())
    return correct.mean().item(), F.cross_entropy(logits/temperature, labels).item(), ece, counts


def report(name, logits, labels, temperature):
    acc, nll, ece, counts = metrics(logits, labels, temperature)
    print(f'{name}: accuracy={acc:.3f} NLL={nll:.4f} ECE={ece:.4f}; bins={counts}')


def main():
    p = argparse.ArgumentParser()
    p.add_argument('--smoke', action='store_true')
    args = p.parse_args()
    torch.manual_seed(7)
    print('PyTorch', torch.__version__)
    train, val, cal, test = [data(n, seed) for n, seed in [(800, 1), (300, 2), (400, 3), (600, 4)]]
    model = nn.Sequential(nn.Linear(2, 32), nn.ReLU(), nn.Linear(32, 32), nn.ReLU(), nn.Linear(32, 2))
    opt = torch.optim.Adam(model.parameters(), lr=.005)
    best_loss, best = float('inf'), copy.deepcopy(model.state_dict())
    for _ in range(5 if args.smoke else 90):
        model.train()
        for indices in torch.randperm(len(train[0])).split(64):
            opt.zero_grad()
            loss = F.cross_entropy(model(train[0][indices]), train[1][indices])
            loss.backward()
            opt.step()
        model.eval()
        with torch.no_grad():
            score = F.cross_entropy(model(val[0]), val[1]).item()
        if score < best_loss:
            best_loss, best = score, copy.deepcopy(model.state_dict())
    model.load_state_dict(best)
    model.eval()
    for parameter in model.parameters():
        parameter.requires_grad_(False)
    with torch.no_grad():
        logits_cal = model(cal[0])
    log_t = nn.Parameter(torch.zeros(()))
    opt_t = torch.optim.Adam([log_t], lr=.03)
    for _ in range(20 if args.smoke else 200):
        opt_t.zero_grad()
        loss = F.cross_entropy(logits_cal/log_t.exp(), cal[1])
        loss.backward()
        opt_t.step()
        with torch.no_grad():
            log_t.clamp_(-3, 3)  # predeclared positive temperature range
    temperature = log_t.exp().detach()
    with torch.no_grad():
        clean = model(test[0])
        noise = (test[0]+.2*torch.randn_like(test[0])).clamp(-1, 1)
        noisy_logits = model(noise)
    assert torch.equal(clean.argmax(1), (clean/temperature).argmax(1))
    print(f'T={temperature.item():.4f}; validation-selected classifier; temperature fitted on calibration only.')
    report('Clean, unscaled', clean, test[1], 1.)
    report('Clean, calibrated', clean, test[1], temperature)
    report('Feature noise std=.2, original labels', noisy_logits, test[1], temperature)
    # Local sensitivity test of this model. Inputs are in raw [-1,1] units.
    x = test[0].detach().clone().requires_grad_(True)
    loss = F.cross_entropy(model(x), test[1])
    gradient, = torch.autograd.grad(loss, x)
    attacked = (x.detach()+.1*gradient.sign()).clamp(-1, 1)
    assert (attacked-test[0]).abs().max() <= .100001
    with torch.no_grad():
        report('FGSM, L-infinity epsilon=.1', model(attacked), test[1], temperature)
    print('Stress tests retain original labels: near the true boundary, label-preservation is imperfect.')
    print('These are local sensitivity results, not a robustness certificate. Report this limitation in the model card.')
    if args.smoke:
        print('Smoke run checks execution, not convergence.')


if __name__ == '__main__':
    main()

Read the output, then challenge it

• Confirm positive temperature leaves test argmax predictions unchanged.
• Read accuracy, NLL, ECE, and bin counts together.
• Compare clean and stressed results; calibration is not guaranteed to transfer.
• Use the printed metrics to write a model card with the exact split and perturbation settings.

These are small mechanism studies, not benchmark claims. Seeds make runs easier to compare; versions and hardware can still change exact results. Decide your validation protocol before tuning and reserve final test results for reporting.

PREDICT → CALCULATE → EXPLAIN

Practice with worked answers

EXPLAIN IT WITHOUT THE CODE

Week 12 review questions

RESEARCH CONNECTIONS

Follow the original ideas

Course alignment: AI Algorithms 2, lecture Week 13. This site counts from Week 0. Explanations, browser labs, and runnable experiments are authored study companions to the supplied lecture material.