Week 1 — Neural Network from First Principles
Build a tiny neural network from arithmetic and derivatives. Start with a neuron, connect it to an output, measure the error, and work backward to update every parameter. The examples use Python's standard library.
Start With One Neuron
A neuron multiplies each input by a weight, adds a bias, and applies an activation. Weights control how inputs contribute; the bias shifts the result. The activation lets the neuron contribute a nonlinear transformation.
Think it throughWhich quantities come from the example, and which quantities can training change?
from math import tanh
x = 2.0
w, b = 0.3, 0.1
z = w * x + b
h = tanh(z)
print(round(h, 4)) # 0.6044Change the parameters. Watch tanh respond.
Input x stays at 2. Move w₁ or b₁ to change z. Near the ends of the curve, tanh saturates and its derivative becomes small.
h = tanh(z) = 0.6044
dh/dz = 1 − h² = 0.6347
Compose a Forward Pass
Feed the hidden activation into a second weighted sum. This tiny regression network has one input, one hidden neuron, and one linear output. A forward pass calculates a prediction using the current parameters; it does not update them.
Think it throughWhy does the output use a linear function here? What output would a binary classifier need instead?
# Same hidden neuron as above:
# h = tanh(w1 * x + b1)
w2, b2 = -0.5, 0.2
prediction = w2 * h + b2
# x -> weighted sum -> tanh
# -> weighted sum -> predictionMeasure Error With a Loss
For this regression example, use half the squared error. It is zero when the prediction equals the target, and its derivative with respect to the prediction is simply prediction minus target. The factor of one half makes that derivative easier to read.
Think it throughA loss is a number. A gradient tells us how that number changes. Why do we need both?
target = 1.0
error = prediction - target
loss = 0.5 * error ** 2
# d(loss) / d(prediction)
grad_prediction = errorBackpropagation Is the Chain Rule
Follow the calculation backward. The output depends on the hidden activation, and the hidden activation depends on its weighted input. Multiply those local derivatives to find how each parameter affects the loss. Compute every gradient using the same forward pass before changing any weights.
Think it throughWhy must grad_z use the old w2 rather than a weight that has already been updated?
# Gradients for the output parameters:
grad_w2 = error * h
grad_b2 = error
# tanh derivative: 1 - h**2
grad_z = error * w2 * (1 - h ** 2)
# Gradients for the hidden parameters:
grad_w1 = grad_z * x
grad_b1 = grad_zTrace one complete learning step.
Read left to right. Current parameters produce the intermediate values, prediction, and loss. No parameter has changed.
Inspect Update to enable the parameter update.
Train the Network Without Autograd
Run this complete example with Python. Gradient descent subtracts the learning rate times each gradient. Repeating the forward pass, backward pass, and update reduces the error on this one example. This demonstrates the training mechanics; one example cannot establish that a model generalizes.
Think it throughWhich lines compute gradients, and which lines actually learn by updating parameters?
from math import tanh
x, target = 2.0, 1.0
w1, b1 = 0.3, 0.1
w2, b2 = -0.5, 0.2
learning_rate = 0.1
for step in range(200):
h = tanh(w1 * x + b1)
prediction = w2 * h + b2
error = prediction - target
grad_w2 = error * h
grad_b2 = error
grad_z = error * w2 * (1 - h ** 2)
grad_w1 = grad_z * x
grad_b1 = grad_z
w1 -= learning_rate * grad_w1
b1 -= learning_rate * grad_b1
w2 -= learning_rate * grad_w2
b2 -= learning_rate * grad_b2
prediction = w2 * tanh(w1 * x + b1) + b2
print("Prediction:", round(prediction, 4))
print("Loss:", 0.5 * (prediction - target) ** 2)Same network. Different step sizes.
Every curve starts from the same four parameters and trains on the same example. Select a learning rate and scrub through its first 200 updates.
| Learning rate | Starting loss | Loss after 200 updates |
|---|---|---|
| 0.01 | 0.607405 | 0.000593 |
| 0.1 | 0.607405 | 2.47e-32 |
| 1 | 0.607405 | 9.86e-32 |
Near-zero values can differ slightly between Python and JavaScript due to floating-point arithmetic. Successful fitting here says nothing about performance on new inputs.
Does a small perturbation agree?
At the initial parameters, compare backpropagation with a central finite difference. Only the selected parameter changes for the two test evaluations; this does not train the model.
A large ε can hide local behavior; a very small ε can amplify rounding error. Numerical checking supports correctness, but it is too costly to replace backpropagation in large networks.
More neurons, the same operations.
A dense layer computes many weighted sums together. With observations stored as rows, the batch calculation is Z = XWᵀ + b, with the bias broadcast across observations.
| Object | Shape | Meaning |
|---|---|---|
| X | B × 3 | Input observations |
| W₁ / b₁ | 4 × 3 / 4 | Hidden weights / biases |
| H = g(XW₁ᵀ + b₁) | B × 4 | Hidden activations |
| W₂ / b₂ | 2 × 4 / 2 | Output weights / biases |
| HW₂ᵀ + b₂ | B × 2 | Output scores |
26 trainable parameters: (3 × 4 + 4) + (4 × 2 + 2). Changing batch size changes the amount of computation, not this parameter count.
Why not initialize every hidden unit identically?
If hidden units and their corresponding outgoing connections start symmetrically, they can receive identical gradients and keep learning the same feature. Random weight initialization breaks that symmetry. Our one-hidden-neuron demo uses fixed parameters so every calculation is reproducible.
Run it. Compare it. Explain it.
- Run the complete Python example, then reset the parameters and compare learning rates of 0.01, 0.1, and 1.0. Record the starting and final losses.
- Before training, perturb w1 slightly. Does the change in loss agree with the sign of its calculated gradient?
- Use the numerical gradient checker to compare the analytical and numerical gradients. Try an extremely small ε and explain the discrepancy.
Week 1 interview questions
Answer aloud, then open the explanation.
01What are weights, biases, and activations responsible for?
Weights scale incoming values, biases shift the weighted sum, and nonlinear activations change the kinds of functions the network can represent. Training adjusts weights and biases; this example fixes the activation to tanh.
02What changes during the forward pass, backward pass, and update?
Forward computes activations, a prediction, and a loss using current parameters. Backward computes gradients at those parameter values. The update changes the parameters using those gradients.
03How does the chain rule connect an early weight to the final loss?
For w1, multiply four local derivatives: (prediction − target) × w2 × (1 − h²) × x. If a quantity affects the loss through multiple paths, add the contributions from those paths.
04Why do we calculate all gradients before updating any parameter?
All components of the gradient must describe the same parameter state and forward pass. Updating w2 before calculating the hidden gradient mixes two different parameter states.
05Why does a low loss on one training example say little about new examples?
The network may fit that one pair without learning a useful rule for other inputs. Evaluate on held-out examples; use validation data for model choices and reserve test data for final evaluation.
Further reading: Deep Learning — Deep Feedforward Networks (opens in a new tab)
Back to the learning map ↑