You already read the symbols.
Now build the mathematics.

Two operations worth more than the rest

Exponents and logarithms carry more weight in this field than anything else at this level, and they are usually rushed. An exponent is repeated multiplication; a logarithm answers the reverse question, what power produces this number? Four rules follow, and each one earns its place later on this page.

The first rule is the one that matters most. The probability of a whole dataset is a product of millions of small numbers, which underflows to zero in any real arithmetic and has a useless derivative besides. Take the log and the product becomes a sum: numerically stable, and differentiable term by term. Almost every loss function you will meet is a log for that reason alone.

The equation in that lab is y = wx + b, which is the same equation the layer above translated as “take the input, multiply it by its importance, add an adjustment.” It is also, exactly, one neuron. Everything from here is that expression scaled up and composed with itself.

A claim needs its quantifiers

“For every input x, there exists a suitable model θ” permits a different model for each input. “There exists one model θ that works for every input x” asks for a single shared model. Switching ∀ and ∃ changes the problem.

A graph connects objects without requiring geometry

A graph G = (V, E) has vertices V and edges E. A directed computational graph records which operations depend on which earlier values. A connected undirected graph without cycles is a tree; choosing a root gives it a hierarchy. A graph adjacency matrix and a neural weight matrix are both arrays, but their entries have different meanings.

The three shapes worth recognising on sight

Linear · ax + b

A straight line. One neuron before its activation, and the whole of a layer in matrix form.

Quadratic · ax² + bx + c

A parabola, with exactly one turning point. Squared-error loss has this shape, which is why it is the example everyone uses.

Exponential · aᵇˣ

Growth proportional to current size. Softmax, learning-rate decay, and the scaling laws in the frontier chapter all live here.

Composition is the idea that makes depth mean something — and it comes with a warning. Compose two affine layers and you get another affine layer: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁ + b₂), which is one affine layer with different weights and bias. A hundred stacked linear layers are worth exactly one. Putting a nonlinear function between them is what buys you depth, and it is the only reason activation functions exist.

Four things a matrix is, all at once

A grid of numbers

The storage view. Shape (rows, columns), and the shapes must line up: (m × n) times (n × p) gives (m × p).

A stack of dot products waiting to happen

The computational view. Entry (i, j) of a product is row i of the first dotted with column j of the second.

A function that moves vectors

The geometric view. A matrix rotates, stretches, flattens or reflects space. This is the view that makes eigenvectors obvious.

A whole dataset

The practical view. One row per example, one column per feature. Shape (batch, features) is the first thing you will print when something breaks.

The vocabulary you will actually meet

These are the words a paper will use without explaining. None is difficult; each is a small fact about what a matrix does.

Length has more than one meaning

“How big is this vector” has several answers, and the choice is not cosmetic — it decides what a penalty does to your model.

Two uses of the same geometry

1. STEP 01A rate you already understand

   Drive 120 km in 2 hours and your average speed is 60 km/h. You divided a change in position by a change in time. Every derivative you will ever meet is that same division.

   average rate = change in output ÷ change in input
2. STEP 02The awkward question

   What is your speed at one instant? In an instant, no time passes and you travel no distance, so the division becomes 0 ÷ 0 — which is not a number. The question is good; the arithmetic is not ready for it.

   0 ÷ 0 is undefined
3. STEP 03Do not divide by zero. Shrink instead.

   Measure over an interval of width h, which is a perfectly ordinary division. Then ask what the answer approaches as h gets smaller. Not what it is at h = 0 — what it is heading towards.

   (f(x + h) − f(x)) ÷ h
4. STEP 04That destination is the derivative

   For f(x) = x², the quotient works out to exactly 2x + h. There is no mystery about where it is going: as h shrinks, it heads to 2x. That limit is the slope of the curve at that point.

   f′(x) = lim(h→0) (f(x + h) − f(x)) / h
5. STEP 05Read it as sensitivity

   f′(x) = 2x says: nudge the input near x, and the output moves about 2x times as much. At x = 3 the output moves six times as fast as the input. That is the reading that matters for AI.

   f′(x) ≈ (output nudge) ÷ (input nudge)
6. STEP 06Many inputs: hold the rest still

   A loss depends on millions of weights. Ask about one at a time: nudge w₇ and hold everything else fixed. That is a partial derivative. Collect one per weight and you have the gradient.

   ∇L = (∂L/∂w₁, ∂L/∂w₂, …, ∂L/∂wₙ)
7. STEP 07The gradient points uphill

   The gradient vector aims in the direction of steepest increase, and its length says how steep. To reduce the loss, step the other way. That single sentence is gradient descent, and the whole of training.

   θ ← θ − α∇L(θ)

The rules, and why you do not have to memorise them first

Every differentiation rule is the limit above, worked out once in general so nobody has to repeat it. Knowing that they are derived rather than decreed is worth more than knowing the list.

More than one input, more than one output

Three words get used constantly and are rarely distinguished. They are the same derivative, arranged differently as the number of inputs and outputs grows.

Curvature is worth dwelling on, because it explains a daily frustration. If the loss curves sharply in one direction and gently in another, a single learning rate cannot suit both: large enough to make progress along the flat direction, it oscillates along the sharp one. For a positive-definite quadratic, the ratio of the largest to smallest Hessian eigenvalue measures this conditioning. Neural losses need not have a positive-definite Hessian, and adaptive updates do not directly compute this curvature.

How derivatives are actually computed

Not by hand, not symbolically, and not by finite differences. A framework records every operation you perform into a computational graph, then applies the chain rule backwards through it — reverse-mode automatic differentiation. The results are exact to floating-point precision, and one backward sweep produces the gradient for every parameter at once.

And the other half: integration

The derivative breaks a total into rates. The integral puts rates back into a total, and the fundamental theorem of calculus says the two operations undo each other — which is genuinely surprising, because “slope of a curve” and “area under a curve” have no obvious reason to be related.

You need far less of it than of differentiation. It matters where it defines something: an expectation E[X] = ∫ x p(x) dx is an integral, a continuous probability is an area under a density, and the evidence term in Bayes' rule is an integral that is usually impossible to compute — which is precisely why variational methods and sampling exist. You will need to recognise integrals constantly and evaluate them almost never.

Two ideas from information theory follow immediately and are worth meeting by name. Entropy is the average surprise of a distribution — how uncertain it is. Cross-entropy is the average surprise your predicted distribution feels when reality is drawn from the true one. Minimising it is the same as maximising likelihood, which is why the standard classification loss is not an arbitrary choice but the statistically principled one.

Spread, and how two quantities move together

The mean says where a distribution sits. Variance says how widely it spreads, and its square root, the standard deviation, puts that back in the original units. Covariance measures how two centered quantities vary together. Collect the pairwise covariances into a matrix: PCA diagonalizes it, and whitening transforms it toward identity when the required directions are nondegenerate. A multivariate Gaussian requires both a mean vector and a covariance matrix.

The distributions worth knowing by name

A distribution is a shape of uncertainty. You do not need many.

Choosing parameters: maximum likelihood

Given data and a family of distributions, which member produced it? Maximum likelihood answers: pick the parameters maximizing the observed data's probability mass or density. Take the log — by that first rule from chapter one — and for independent observations the product over examples becomes a sum, and maximising it becomes minimising its negative. That negative log-likelihood is cross-entropy. The standard loss was not invented for neural networks; it is a century-old statistical principle arriving under a different name.

Add a prior belief about the parameters and you get MAP estimation instead — and a Gaussian prior on the weights turns out to be exactly an L2 penalty. This equivalence is between particular penalty terms and log priors; not every regularization method is simply MAP estimation.

Keep the target risk separate from the sample loss

The first is measured on particular rows. The second is an expectation under the target distribution. Fitting the first does not automatically minimize the second. A representative held-out sample estimates future performance; distribution shift can break that connection.

Learning theory formalizes when empirical performance transfers to population performance. PAC learning specifies error and confidence; VC dimension measures the capacity of binary hypothesis classes. These are useful next destinations in the map, with assumptions that deserve as much attention as the formulas.

The optimiser zoo, in four lines

Each rule turns a gradient into an update. Plain SGD has no optimizer memory; momentum and adaptive methods keep state across steps.

Second-order methods use curvature as well as slope. A Newton step solves HΔ = −g. Fast local convergence requires conditions on smoothness, curvature and the starting point; an indefinite Hessian may not produce a descent direction. Quasi-Newton and matrix-free methods reduce the cost. Adam instead tracks gradient moments: it is not a Hessian approximation.

Choose the computation as carefully as the formula

• Solve a system instead of explicitly inverting it. For least squares, QR or SVD often avoids the extra conditioning problems of forming XᵀX.
• Compute log probabilities directly. Log-sum-exp uses m + log Σ exp(zᵢ − m), with m = max zᵢ. Use fused log-loss operations when implementing a classifier.
• Check a derivative at several step sizes. Finite differences trade truncation error against roundoff. Reverse-mode autodiff uses vector–Jacobian products; it still operates with floating-point values.
• Fit scaling on training data. Reuse those statistics at evaluation. Scaling can improve conditioning but cannot guarantee a useful fit.

There is a genuine puzzle underneath this, and it is worth sitting with rather than resolving too quickly. Mathematics developed for one purpose keeps turning out to describe something else entirely — the physicist Eugene Wigner called it the unreasonable effectiveness of mathematics in the natural sciences. Complex numbers were a bookkeeping device for solving cubics before they were the language of quantum mechanics. Non-Euclidean geometry was a curiosity before it was general relativity. Whatever you learn here has a strong track record of being useful somewhere you did not expect.

Reviewed on September 19, 2026. These are directions of sustained work rather than a snapshot of this month, and each links the paper that framed the question rather than the latest result — deliberately, because the framing outlives the leaderboard.

1. STEP 01Get comfortable rearranging

   Before anything else: move terms across an equals sign without hesitating, handle fractions, and read powers and logs as operations rather than decoration. This is the layer everything else silently assumes.

   You are ready to move on when: You can rearrange y = wx + b to solve for w without thinking about it.

2. STEP 02Learn linear algebra before calculus

   This is the order people get wrong. The forward pass of every network is a matrix product; you will read shapes and dot products on day one and derivatives on day thirty. Vectors, dot products, matrix multiplication, and what a matrix does to space.

   You are ready to move on when: Given shapes (32, 784) and (784, 128), you can say what comes out and why.

3. STEP 03Then the derivative, slowly

   One idea, approached properly: the derivative as sensitivity of output to input. Then the chain rule, then partial derivatives, then the gradient. Do not detour into integration techniques — they are beautiful and you do not need them yet.

   You are ready to move on when: You can explain why backpropagation is the chain rule, out loud, to someone else.

4. STEP 04Add probability

   Random variables, distributions, expectation, independence, conditional probability and Bayes' rule, then likelihood and log-likelihood. This is what turns a score into a loss and a prediction into a claim about uncertainty.

   You are ready to move on when: You can say why cross-entropy loss is −log(p) and why the log belongs there.

5. STEP 05Finish with optimisation

   Gradient descent, learning rates, momentum, why convexity matters and why deep learning does not have it. At this point the four earlier layers combine and the training loop stops being a black box.

   You are ready to move on when: You can explain what a learning rate that is too large actually does to the update.

6. STEP 06Now go back and be rigorous

   Only once the machinery makes sense is it worth the formal treatment: proofs, epsilon–delta limits, measure-theoretic probability, convex analysis. Doing it in this order means the abstractions land on something you already understand.

   You are ready to move on when: You want the proof, rather than being told you need it.

Where to actually learn it