Teach a network a shape, one step at a time.

A small neural network, written from scratch, learning to tell blue points from navy ones. Choose the shape and the network, press Train, and watch what it answers everywhere change as the loss falls.

Guess first

A network with no hidden layer can only draw a straight line between the two colours. The points make a circle, blue inside, navy outside. Can it learn to separate them?

Steps
0
Loss
–
New points right
–

The shape

Hidden layers

Neurons in each

Activation

Learning rate

2 inputs → 4 → 1 output

Try the spiral: one hidden layer of 8 gets 85% of new points right after 3,000 steps; two layers of 8 with ReLU, 97.5%. Then try a learning rate of 0.3, and of 0.003.

How it works here: 200 points to learn from and 200 new ones to check, the network’s weights started at random, and every step one pass of gradient descent (Adam) over all 200 points on the cross-entropy loss, written from scratch and computed in your browser. Training stops by itself after 20,000 steps, and pauses when you scroll away.

What the layers buy you.

  • Lines, then curves

    Each neuron draws a straight line through the points and bends it with its activation. A hidden layer combines those lines into curves; a second layer combines curves, which a spiral needs.

  • Steps down the loss

    Training is gradient descent: back-propagation works out how each weight moves the loss, and every step nudges them all downhill. Too big a learning rate overshoots; too small crawls.

  • Judged on new points

    The loss is measured on the points it learns from; the score that counts is on 200 points it never saw. A bigger network can fit its 200 points perfectly and still miss new ones.

Sources

  1. Rumelhart, Hinton and Williams, “Learning representations by back-propagating errors”, Nature, 1986.
  2. Kingma and Ba, “Adam: A Method for Stochastic Optimization”, 2014: the optimiser used here.
  3. Goodfellow, Bengio and Courville, “Deep Learning”, MIT Press, 2016, chapter 6: deep feedforward networks, and XOR.