Weights, a loss, and many small steps.
A neural network is layers of simple units with adjustable weights. Training nudges every weight a little, again and again, to make one number, the loss, smaller. Here it is on small problems you can train in your browser.
- Explain neurons, layers and activations
- Explain gradient descent and the learning rate
- Say what backpropagation computes
You need: Lesson 3.
- 1Distributions and sampling
- 2Confidence intervals and tests
- 3Training, testing and overfitting
- 4Judging a classifier
- 5Trees and ensembles
- 6Clustering and PCA
- 7How a neural network learns
-
A neuron draws a line
Predict first
Blue points inside a circle, navy outside. Can a network with no hidden layer separate them?
Just show me
No: it gets 53% of 200 new points right, close to a coin toss. One hidden layer of 4 neurons gets 99.5%.
A neuron multiplies each input by a weight, adds them up with a bias, and passes the sum through an activation. With no hidden layer, the network can only draw a straight line, and no straight line splits a circle. A hidden layer combines several lines into a curve.
No hidden layer53%One hidden layer of 283%One hidden layer of 499.5%Share of 200 new points right after 1,500 steps, from the network trainer’s own code and seeds
Train it yourself in the network trainer
Quick check
What does a hidden layer add?
Several neurons’ lines, bent by activations and combined, make curves.
-
Harder shapes, more layers
Predict first
A spiral: one hidden layer of 8 neurons gets 85% of new points right. Two layers of 8?
Just show me
97.5% after 3,000 steps: a second layer builds curves out of curves.
Activations such as ReLU, max(0, x), and tanh bend each layer’s sums, so layers can build on each other; without them, any stack of layers is still one straight line. ReLU is the usual default for hidden layers. The output’s activation fits the task: a sigmoid for a yes-or-no chance, softmax for several classes.
Quick check
The output for “will this customer leave?”
A sigmoid squeezes the score into a probability.
-
The loss says how wrong
Predict first
A model says 0.9 that a customer leaves; they stay. Its log loss is…
Just show me
Large: −ln(1 − 0.9) = 2.3, against 0.11 had it said 0.1.
Training needs one number to shrink: the loss. Cross-entropy (log loss) punishes confident mistakes hardest, which pushes the network toward honest probabilities. For predicting numbers, the usual loss is the mean squared error.
Quick check
The customer stays. Which prediction costs more: 0.6 or 0.99?
The more confident the wrong answer, the larger the loss.
-
Small steps downhill
Predict first
The circle’s network starts at 50%, a coin toss. About how many steps until it gets 95% of new points right?
Just show me
About 100: 70% after 25 steps, 81% after 50, 98.5% after 100, and it settles at 99.5% by step 800.
The gradient says which way each weight should move to make the loss smaller; gradient descent takes a small step that way, over and over. The learning rate sets the step: too small and training crawls, too big and it overshoots and bounces. Adam, used here, adapts the step for each weight.
At the start50%After 25 steps70%After 50 steps81%After 100 steps98.5%After 200 steps98%After 400 steps98.5%After 800 steps99.5%After 1,500 steps99.5%The circle’s network (one hidden layer of 4): share of 200 new points right as it trains
Quick check
Training bounces around and never settles. First thing to try?
Steps too big overshoot the bottom; smaller ones settle.
-
Backpropagation
Predict first
A network has thousands of weights. How does it get each one’s gradient efficiently?
Just show me
Backpropagation: one backward pass applies the chain rule layer by layer and gives every weight’s gradient at once.
The forward pass computes the prediction and the loss; the backward pass carries the error back, each layer’s gradient built from the next one’s. That’s what
loss.backward()does in PyTorch. Deep networks add tricks to keep gradients healthy (ReLU, careful starting weights, normalisation, residual connections) and to stop overfitting (dropout, weight decay, early stopping).Quick check
Training loss keeps falling; validation loss starts rising. Now what?
That’s overfitting starting: stop at the best validation point, or regularise.
In the interview: how a neural network learns.
HardHow does a neural network learn?
EasyWhy do neural networks need non-linear activations, and why is ReLU the usual default for hidden layers?
EasyWhat does loss.backward() compute? Work through a one-weight example and check it with autograd.
MediumHow do you choose a batch size, and why does gradient descent step along the negative gradient?
MediumSGD, momentum or Adam: how do they differ?
MediumTraining loss keeps falling but validation loss has started to rise. What do you do?
MediumWhat causes vanishing and exploding gradients, and how do you fix each?
Ask Prism about this lesson
Sources
- Rumelhart, Hinton and Williams, “Learning representations by back-propagating errors”, Nature, 1986.
- Kingma and Ba, “Adam: A Method for Stochastic Optimization”, 2014: the optimiser used here.
- Goodfellow, Bengio and Courville, “Deep Learning”, MIT Press, 2016, chapter 6.
- Our network trainer, its own code and seeds for every run here.