A neuron without an activation computes ŷ = wᵀx + b: exactly the model form used by linear regression. The weight controls each input’s contribution; the bias shifts the output.
Interview check: Distinguish the model’s equation from the loss and fitting procedure.
PYTHON / PYTORCH
# A linear model: one affine transformation
y = w * x + b
import torch.nn as nn
neuron = nn.Linear(in_features=1, out_features=1)
# nn.Linear includes a bias by default.
# Choose a loss and fit its parameters to train it.
CONCEPT 02
Logistic regression as a single neuron
Binary logistic regression applies sigmoid to the affine score: p = σ(wᵀx + b). The probability curve is nonlinear, but its 0.5 decision boundary is still linear in the supplied features.
Interview check: Why does sigmoid change the output range without making that boundary curved?
PYTHON / PYTORCH
import torch.nn as nn
model = nn.Sequential(
nn.Linear(10, 1),
nn.Sigmoid()
)
# Ten input features, one probability.
# For training with BCEWithLogitsLoss,
# omit Sigmoid: that loss handles it internally.
TRY IT
One neuron. Two familiar models.
Move the weight to change the slope and the bias to shift the output. Turn on sigmoid to map the score to a probability.
At x = 1: ŷ = 1.000
ŷ = wx + b · linear model
CONCEPT 03
The limitation: one straight boundary
XOR puts matching labels on opposite corners of a square. A single affine score followed by a sigmoid or threshold cannot separate the classes in the original input space.
Interview check: This is a statement about the model and its features. Engineered nonlinear features can also make XOR separable.
PYTHON / PYTORCH
# XOR: output 1 only when inputs differ
# x1 x2 target
# 0 0 0
# 0 1 1
# 1 0 1
# 1 1 0
# Logistic regression on raw x1, x2
# cannot separate these four points.
SEE THE REPRESENTATION CHANGE
Give XOR a new coordinate system.
The class-1 points sit on opposite corners. Any single straight boundary leaves at least one point on the wrong side.
Class 1 when score > 0.5. These are hand-picked weights that demonstrate a solution; this button does not train a network.
○ Class 0◆ Class 1
Original input space · no separating straight line
CONCEPT 04
Why nonlinear hidden layers matter
Two affine layers alone collapse into one: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + W₂b₁ + b₂. A nonlinear activation between them allows new kinds of functions.
ReLU networks form piecewise linear functions. Other activations can produce smooth curves. Depth can help represent some functions efficiently.
Theory check: Universal approximation can hold with one sufficiently wide hidden layer under suitable conditions. It guarantees representational capacity, not easy training or good generalization.
PYTHON / PYTORCH
import torch.nn as nn
model = nn.Sequential(
nn.Linear(2, 4),
nn.ReLU(),
nn.Linear(4, 1)
)
loss_fn = nn.BCEWithLogitsLoss()
# This architecture can represent XOR.
# It still needs training; success is not automatic.
# Use torch.sigmoid(model(x)) for probabilities.
CONCEPT 05
Feature engineering and feature learning
Feature engineering uses domain knowledge to construct inputs. Feature learning adjusts hidden transformations through training. Both can be useful in the same system.
You still select data, preprocessing, architecture, and objectives. A hidden unit is not guaranteed to discover a named concept such as BMI.
Interview check: Explain how the loss influences both the final prediction layer and the earlier feature transformations.
PYTHON / PYTORCH
# An engineered feature (height measured in meters)
df["bmi"] = df["weight"] / df["height"] ** 2
# A learned representation
encoder = nn.Sequential(
nn.Linear(10, 16),
nn.ReLU()
)
# During training, these weights can adapt
# to produce features useful for the objective.
CONCEPT 06
The learning loop stays familiar
Prediction, loss, gradient calculation, and parameter updates remain the basic loop. Backpropagation applies the chain rule through the computational graph. Automatic differentiation performs that calculation for you.
Weights and biases are parameters. Learning rate and layer widths are hyperparameters. Optimizers can also track state such as momentum.
Interview check: Calculating a gradient and changing a parameter are separate operations.
PYTHON / PYTORCH
for x, y in loader:
optimizer.zero_grad() # Clear old gradients
logits = model(x) # Forward pass
loss = loss_fn(logits, y)
loss.backward() # Compute gradients
optimizer.step() # Update parameters
# Use validation data for model selection.
# Reserve test data for final evaluation.
FOLLOW THE UPDATE
A gradient points. An optimizer steps.
Fixed input x = 3, target y = 8, bias b = 1, learning rate η = 0.1. Only the weight changes in this demo.
Try answering aloud before opening the explanation.
01How is linear regression related to a single neuron?
Both compute an affine function: ŷ = wᵀx + b. A neuron with an identity output has the same model form. Linear regression also specifies how those parameters are fitted, commonly using squared error.
02Why can’t logistic regression on the original inputs solve XOR?
At a probability threshold of 0.5, its boundary is wᵀx + b = 0. XOR places equal labels on opposite corners of a square, so no single straight line separates them.
03What does a hidden layer give you?
With a nonlinear activation, it can transform the inputs into features that make the final prediction easier. Without nonlinearity, stacked affine layers collapse to one affine transformation.
04What is feature learning?
Hidden layers learn intermediate representations through the training objective. You still choose inputs, preprocessing, architecture, and training data; useful features and generalization are not guaranteed.
05Does universal approximation require many hidden layers?
No. Under suitable conditions, one sufficiently wide hidden layer with an appropriate nonlinear activation can approximate continuous functions on a compact domain. The theorem does not guarantee that training will find the solution or that it will generalize.
06Does loss.backward() update the weights?
No. It computes and accumulates gradients. optimizer.step() uses those gradients to update parameters. Reset gradients between ordinary training steps with optimizer.zero_grad().