A tiny neural network: forward pass and backpropagation

Weights & biases

The whole idea.
This network has one input x, two hidden neurons and one output y. It has seven numbers that training adjusts: each hidden neuron has a weight and a bias (w1, b1 and w2, b2), and the output has a weight for each neuron (w3, w4) and a bias b3.
• Forward pass: each neuron computes z = w·x + b, then ReLU keeps it if it is positive and makes it 0 if not. The output adds the two neurons up with its own weights: y = w3·h1 + w4·h2 + b3. The loss measures the miss: L = (y − t)², where t is the target.
• Backward pass: start at the loss and walk back, one box at a time. Each step multiplies by one small local slope, the chain rule. At the end, every one of the seven numbers has a gradient (purple): how much the loss changes if that number moves up a little.
• Update: move every number a small step against its gradient, new = old − η × gradient, as on the optimizers page.
Doing that over and over, on all five points at once, is training.

Things worth trying:
1. Click Next step eleven times, and check each line with a calculator. For the example x = 2, t = 3, the network says y = 1.50 and the loss is 2.25. Then click Update weights & biases and step through the forward pass again: with η = 0.05, the new y is 3.06 and the loss drops to 0.0032.

2. Choose the example x = −2. Now z1 = 0.50·(−2.00) + 0.00 = −1.00 is negative, ReLU outputs 0, and in the backward pass the gate closes: the gradients for w1, b1 and w3 are all exactly 0. For this example, neuron 1 cannot learn. If it is 0 for every example, it never will: that is a dead neuron, as on the activation page.

3. Raise η to 0.1 and update from the start. The step is too big: y overshoots the target, from 1.50 to 4.95, and the loss for this example rises from 2.25 to 3.80.