Week 2 — Depth, Activations, and Gradient Behaviour
Week 1 followed gradients through a tiny network. Now extend that calculation through more layers, and inspect the tools used to keep deeper networks trainable: initialization, normalization, dropout, and diagnostics.
Depth Needs Nonlinearity
Depth adds successive transformations. If every layer is affine, their composition is still one affine transformation. Nonlinear activations between layers allow the representation to change in ways a single affine map cannot express. Extra depth can be useful, but it does not guarantee better results.
Think it throughIf you stack ten linear layers without activations, have you expanded the set of affine functions you can represent?
# Two affine layers:
# h = W1 @ x + b1
# y = W2 @ h + b2
# Substitute h:
# y = (W2 @ W1) @ x + (W2 @ b1 + b2)
# With an activation between them:
# y = W2 @ tanh(W1 @ x + b1) + b2
# This generally cannot collapse the same way.Compare Activations and Their Slopes
Sigmoid maps values into (0, 1), while tanh maps them into (-1, 1). Both become nearly flat for large input magnitudes. ReLU is zero for negative inputs and has slope one for positive inputs. Its derivative is undefined at zero; implementations commonly choose zero there.
Think it throughWhich activations can have a very small slope even when their outputs are far from zero?
from math import exp, tanh
for z in [-6.0, -1.0, 0.0, 1.0, 6.0]:
sigmoid = 1 / (1 + exp(-z))
sigmoid_slope = sigmoid * (1 - sigmoid)
tanh_slope = 1 - tanh(z) ** 2
relu_slope = 1.0 if z > 0 else 0.0
print(z, round(sigmoid_slope, 4),
round(tanh_slope, 4), relu_slope)Inspect the output and its slope together.
At zero, ReLU has no mathematical derivative. This demo uses the conventional choice of zero; the derivative plot has a jump. Saturation concerns slope, not whether the output itself is near zero.
Gradients Can Shrink or Grow With Depth
Backpropagation repeatedly multiplies local derivatives. In a scalar chain, factors below one can shrink the gradient, while factors above one can amplify it. In a network, weight matrices and activation derivatives both matter; looking only at the activation does not predict the full behaviour.
Think it throughWhat happens to an early layer's update when its gradient is tiny? What can a very large gradient do?
# Illustrative scalar products:
for depth in [1, 5, 10, 20]:
shrinking = 0.25 ** depth
growing = 1.5 ** depth
print(depth, shrinking, growing)
# These are local-derivative products,
# not a prediction for every neural network.Initialization and Inactive ReLUs
Weight scale affects both activations and gradients. He initialization uses a variance of 2/fan_in as a useful starting point for ReLU layers. This is not a guarantee of stable training. A ReLU unit that remains negative for every training input gets no gradient through that activation and may stay inactive.
Think it throughWhy is 'ReLU solves vanishing gradients' too strong a claim?
from math import sqrt
from random import Random
rng = Random(0)
fan_in, fan_out = 16, 8
std = sqrt(2 / fan_in)
# Rows are output neurons; columns are inputs.
weights = [
[rng.gauss(0, std) for _ in range(fan_in)]
for _ in range(fan_out)
]Scale the starting weights to the layer.
| Method | Weight variance | Normal-distribution standard deviation |
|---|---|---|
| Glorot / Xavier (gain 1) | 2 / (fan_in + fan_out) = 0.0833 | 0.2887 |
| He (ReLU, fan_in mode) | 2 / fan_in = 0.1250 | 0.3536 |
These are target distribution scales, not measured sample variances. Xavier can use an activation-specific gain. He’s factor of two accounts for the loss of second moment after rectifying a symmetric distribution; the assumptions are approximate.
Inspect a Tiny Chain Yourself
This experiment computes the output and its derivative with respect to the input through a scalar chain. Each layer has the same weight and no bias. It isolates the effect of depth, weight scale, and activation; a full network has many interacting paths and parameters.
Think it throughWhy can the same weight scale produce different gradient behaviour with tanh and ReLU?
from math import tanh
def chain(depth, weight, activation, x=0.1):
gradient = 1.0
for _ in range(depth):
z = weight * x
if activation == "tanh":
x = tanh(z)
slope = 1 - x ** 2
else: # ReLU, with slope 0 at z = 0
x = max(0.0, z)
slope = 1.0 if z > 0 else 0.0
gradient *= weight * slope
return x, gradient
for activation in ["tanh", "relu"]:
for depth in [1, 5, 20]:
output, gradient = chain(depth, 1.5, activation)
print(activation, depth,
round(output, 6), round(gradient, 6))Follow a gradient through a deep scalar chain.
Every layer uses the selected activation, the same scalar weight, and zero bias. We calculate ∂hfinal/∂hᵢ, starting with a gradient of 1 at the output. This is an output derivative, not a loss gradient or a full network simulation.
Inspect every local derivative
| Layer i | zᵢ | hᵢ | g′(zᵢ) | w × g′(zᵢ) | ∂hfinal/∂hᵢ |
|---|---|---|---|---|---|
| 1 | 0.1500 | 0.1489 | 0.9778 | 1.4667 | 0.0841 |
| 2 | 0.2233 | 0.2197 | 0.9517 | 1.4276 | 0.0589 |
| 3 | 0.3295 | 0.3181 | 0.8988 | 1.3482 | 0.0437 |
| 4 | 0.4771 | 0.4440 | 0.8029 | 1.2044 | 0.0363 |
| 5 | 0.6659 | 0.5823 | 0.6609 | 0.9914 | 0.0366 |
| 6 | 0.8735 | 0.7031 | 0.5056 | 0.7584 | 0.0482 |
| 7 | 1.0547 | 0.7836 | 0.3859 | 0.5789 | 0.0833 |
| 8 | 1.1754 | 0.8260 | 0.3177 | 0.4766 | 0.1749 |
| 9 | 1.2390 | 0.8452 | 0.2857 | 0.4285 | 0.4081 |
| 10 | 1.2678 | 0.8532 | 0.2721 | 0.4081 | 1.0000 |
A zero ReLU slope anywhere in this single path blocks its input derivative. Negative weights may reverse the gradient sign. The plot shows magnitude; the readout preserves sign.
Batch Normalization
Batch normalization standardizes each layer's activations using the current mini-batch's mean and variance, then applies a learned scale (gamma) and shift (beta) so the network can undo the normalization if that turns out to be useful. With the default track_running_stats=True, evaluation uses running averages collected during training in place of mini-batch statistics — this is why model.eval() changes behavior, not just a formality.
Think it throughWhy does switching between model.train() and model.eval() change a BatchNorm layer's output for the exact same input?
# x_hat = (x - mean_B) / sqrt(var_B + eps)
# y = gamma * x_hat + beta
import torch.nn as nn
layer = nn.BatchNorm1d(num_features=64)
layer.train() # uses current batch statistics
layer.eval() # uses stored running statistics
# Defaults: affine=True, track_running_stats=TrueKeep one input fixed. Change its batch neighbors.
The highlighted input is always 1. Training uses statistics from the selected batch. Evaluation uses an illustrative, frozen running mean of 3 and running variance of 4.
μ = 2.5000 · variance = 1.2500
| Input in batch A | Output |
|---|---|
| 1 (fixed input) | -1.3416 |
| 2 | -0.4472 |
| 3 | 0.4472 |
| 4 | 1.3416 |
The demo does not update running statistics. It models the default track_running_stats=True behavior for one feature. Setting track_running_stats=False uses batch statistics in both modes. The training forward pass uses biased variance; PyTorch’s running variance update uses an unbiased estimate.
Dropout and Inverted Dropout
Dropout randomly zeroes a fraction of activations during training so the network can't rely on any fixed set of units firing together — this targets generalization, not training loss directly. Inverted dropout scales surviving activations by 1/(1-p) during training itself, so no rescaling is needed at evaluation time — dropout is simply turned off.
Think it throughIf training loss isn't decreasing, does adding dropout make sense as a fix? Why or why not?
# h_tilde = mask * h / (1 - p)
# mask_j ~ Bernoulli(1 - p)
import torch.nn as nn
dropout = nn.Dropout(p=0.5)
dropout.train() # dropout active, scaled
dropout.eval() # dropout disabled entirelyDrop units during training. Restore them for evaluation.
For each fixed activation, inverted dropout preserves its expected value across masks: E[mh/(1−p)] = h. A single mask does not have to preserve the mean or drop exactly p of the units.
Masks use a seeded pseudorandom generator for reproducibility. Evaluation leaves the values unchanged regardless of p. Dropout does not repair incorrect gradients or guarantee better validation performance.
The Jacobian View: Why Some Directions Vanish and Others Explode
Each layer contributes a Jacobian matrix to the gradient calculation. Directions aligned with the Jacobian's large singular values get amplified; directions aligned with small singular values get attenuated. Multiplying many layers' Jacobians together can produce a highly anisotropic result — some directions vanish while others explode, inside the very same network, at the very same time.
Think it throughWhy can a single deep network have some gradient directions vanishing and others exploding at the same time?
# Conceptual, not literal code:
# dL/dh_0 = (J_1^T J_2^T ... J_k^T) dL/dh_k
#
# Each J_i has its own singular values.
# A direction aligned with a small singular
# value in every J_i shrinks toward zero.
# A direction aligned with a large one grows.One network can shrink one direction and amplify another.
Consider the same diagonal Jacobian at each layer: J = diag(0.5, 1.5). Its axes stay aligned, so an upstream gradient (1, 1) becomes (0.5ᵈ, 1.5ᵈ) after d layers.
The bar length encodes log₁₀ magnitude on a fixed −7 to +7 scale; halfway is magnitude 1. Real Jacobians can rotate and mix directions. Multiplying individual singular values is generally insufficient without considering that alignment.
Diagnosing a Dead ReLU
A 'dead' ReLU is a unit that outputs zero — and has a derivative of zero — for every training example, so the loss sends no gradient through that unit to its incoming weights; a nearly always inactive unit receives sparse gradient signals. This is an optimization symptom, not a mysterious separate failure. Common causes include a learning rate that's too large, poor initialization, or unscaled inputs pushing pre-activations permanently negative.
Think it throughThis snippet doesn't tell you if the model is good — what does it actually tell you?
import torch
import torch.nn as nn
model = nn.Sequential(
nn.Linear(10, 32),
nn.ReLU(),
nn.Linear(32, 32),
nn.ReLU(),
nn.Linear(32, 1)
)
x = torch.randn(64, 10)
y = torch.randn(64, 1)
loss_fn = nn.MSELoss()
prediction = model(x)
loss = loss_fn(prediction, y)
model.zero_grad()
loss.backward()
for name, p in model.named_parameters():
if p.grad is not None:
print(name, p.grad.norm().item())# Run after defining model and x in the diagnostic example.
# Inspect activity for each hidden unit, not only a layer norm.
with torch.no_grad():
hidden = x
for i, layer in enumerate(model):
hidden = layer(hidden)
if isinstance(layer, nn.ReLU):
active_fraction = (hidden > 0).float().mean(dim=0)
print("ReLU layer", i, active_fraction)
# Repeat over representative batches and training steps.
# A zero fraction on one batch does not prove permanent death.Three Failure Modes That Look Similar From Outside
Poor final accuracy can come from three very different underlying problems, and each needs a different fix. High training and validation losses can suggest under-capacity, but can also reflect optimization, data, or objective problems. Flat or erratic training loss calls for optimization checks. Improving training loss with deteriorating validation loss suggests overfitting. Use gradient and activation measurements plus controlled experiments to distinguish causes; several can coexist.
Think it throughIf training loss falls while validation loss rises, what does that suggest, and why does it not rule out every other problem?
# Under-capacity:
# train_loss high, val_loss high
# -> test capacity after checking optimization/data
# Optimization failure:
# train_loss flat, erratic, or NaN
# -> inspect gradients, init, lr
# Overfitting:
# train_loss low, val_loss high
# -> regularize, dropout, more dataUse the curve to choose the next experiment.
These curves are synthetic illustrations, not measured training results. Select a pattern and decide what evidence you would collect next.
Training loss falls while validation loss rises after an early improvement. This is evidence consistent with overfitting; it does not establish a unique cause or rule out other problems.
Next experimentCheck the split and evaluation procedure. Then compare early stopping, regularization, or additional representative data using validation results.
Predict the signal before running the code.
- Run the scalar-chain experiment with weights 0.5, 1.0, and 1.5. Compare depths 1, 5, and 20, then repeat with a negative input. Predict the ReLU gradient before running each case.
- Run the PyTorch diagnostic example. Print each parameter’s gradient norm and per-unit active fractions across several batches. Explain what a near-zero early-layer norm does and does not establish.
- Switch the normalization and dropout demos between training and evaluation. Explain the different reasons their outputs change.
- Choose one failing baseline. Change one factor at a time, keeping the data split and seed policy fixed. Record training loss, validation loss, gradient norms, and compute cost.
Week 2 interview questions
Answer aloud before opening the explanation.
01Why do affine layers collapse without nonlinear activations?
Substitution combines their weight matrices and biases into one affine map. Ten affine layers still represent an affine function; intermediate bottlenecks may further restrict the available maps.
02What does saturation mean for sigmoid and tanh derivatives?
The function becomes nearly flat at large input magnitudes. Its derivative approaches zero, even when the output is near 1 or −1.
03How do weights and activation derivatives both influence gradients?
Each scalar layer contributes weight × activation slope. In a network, the corresponding Jacobian combines the weight matrix and activation derivatives. Products and direction alignment control the final signal.
04When can a ReLU unit stop receiving a useful gradient?
If its pre-activation is nonpositive for all examples, the conventional ReLU derivative is zero on those examples. Inspect per-unit activity over representative batches. One zero activation or a small layer gradient norm does not prove a unit is permanently dead.
05Why does adding depth sometimes make optimization harder?
It adds more Jacobian factors and interactions. Gradients can become tiny, large, or uneven across directions. Extra representational capacity alone does not guarantee effective optimization.
06Why does model.eval() change BatchNorm and Dropout?
With default running-statistics tracking, BatchNorm uses stored running statistics at evaluation. Dropout becomes the identity. eval() changes module behavior; it does not disable autograd. Use no_grad() or inference_mode() separately when appropriate.
07Why can gradient directions vanish and explode simultaneously?
The full Jacobian product can attenuate some directions and amplify others. A diagonal example makes this visible, but real networks also rotate and mix directions between layers.
08Can loss curves distinguish under-capacity from overfitting conclusively?
No. Low training loss and worsening validation loss suggest overfitting. High losses on both sets can reflect inadequate capacity, unsuccessful optimization, or data problems. Combine curves with diagnostics and controlled experiments.
Research milestones
| Year | Contributors | Contribution |
|---|---|---|
| 1989 | George Cybenko | Proved the universal approximation theorem for sigmoid networks — a formal reason why a single hidden layer with nonlinearity can approximate a wide class of functions. |
| 1991 | Kurt Hornik | Extended the universal approximation theorem to a broader class of activation functions, strengthening the theoretical case for nonlinear multilayer networks. |
| 1991 | Sepp Hochreiter | Identified the vanishing gradient problem in his diploma thesis, explaining why gradients can shrink to near-zero through long chains of multiplication — this directly motivated later architectures like LSTMs. |
| 2010 | Xavier Glorot & Yoshua Bengio | Introduced Xavier/Glorot initialization, choosing initial weight variance to keep activation and gradient variance stable across layers, primarily for tanh-like activations. |
| 2014 | Nitish Srivastava, Geoffrey Hinton, et al. | Coauthored the 2014 JMLR paper on dropout, randomly disabling units during training to reduce overfitting. Earlier dropout work appeared in 2012. |
| 2015 | Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun | Introduced He initialization, adjusting Xavier's approach specifically to account for how ReLU suppresses roughly half of its input distribution. |
| 2015 | Sergey Ioffe & Christian Szegedy | Introduced batch normalization, making it standard practice to normalize intermediate activations during training as a trainable part of the architecture. |
Universal approximation results assume suitable activations, function classes, and approximation domains; they do not guarantee successful training.