Dashed gray: the curve to match. Thin colored lines: what each neuron adds
(its output times its weight). Thick black line: the network's answer, which is their sum plus a constant.
Click or drag on the plot to choose an input x; the shaded columns of the table follow it
through the network. Click a neuron in the table to pick out its line.
The network itself. Every neuron gets the same x; each line's thickness shows the size
of its weight, and the neuron numbers match the colors in the plot and the table. Under x and
y are their values at the chosen x. Click a neuron here or in the table to pick it out.
The whole idea.
The network here is as small as it gets: one input x, one hidden layer of a few neurons, one output.
Each neuron computes a weighted sum, z = w·x + b, passes it through the
activation f, and the output adds up wout·f(z) over all neurons, plus a constant.
Take the activation away and every step is just “multiply, then add”. Stacking those
multiplies out to a single multiply and a single add, so the whole network is one straight line,
however many neurons it has. The readout shows that line, worked out from the weights. On the wave the
best it can do is an error of 0.4959, and pressing Fit it again changes nothing.
The activation is the only thing in the network that is not a straight line, so it is the only
thing that can bend the output. Each neuron adds one bent piece, and the black line is their sum.
ReLU pieces have one sharp corner each, while sigmoid and tanh pieces are smooth steps. More neurons
give more pieces, and more pieces can follow a more complicated curve.
Things worth trying:
1. None, then Reset a few times. The black line tilts and slides,
but it never bends.
2. ReLU on the wave with four neurons. After Fit it, the dots show where each neuron bends
the output: four neurons, four bends, and the error drops from 0.7104 to 0.0051. Add neurons one at a
time and fit again after each: 0.0042 with five, 0.0025 with six, 0.0008 with eight.
3. The same wave with Sigmoid or Tanh: four neurons reach 0.0002 and 0.0001. A smooth
curve is easier to build from smooth pieces. Then try the V shape: ReLU matches it exactly
(error 0.0000), because a V is just two straight pieces with one corner, while sigmoid stops
at 0.0010.
4. Look at the learning signal column. It is the average slope of f where the neuron
is being used, and learning can only nudge a neuron through that slope. Sigmoid's slope is never more
than 0.25, and after fitting the wave its neurons sit between 0.06 and 0.15. Tanh's neurons sit between
0.21 and 0.36. Both curves go flat at the ends, so a neuron pushed far out to one side barely
learns.
5. Dead neurons. If a ReLU neuron's input becomes negative for every x, its output
and its slope are both 0, so no learning signal reaches it and it never recovers. The table marks it
dead. With ReLU on the wave, press Reset seven times, then
Fit it: neuron 1 dies along the way and the error stalls at 0.2335. Over 50 random starts on
each of the three curves, about 1 ReLU neuron in 15 ended up dead after fitting. Leaky ReLU keeps a small
slope on the negative side (0.1 here; most libraries default to 0.01), so no neuron is ever completely
cut off. Switch to Leaky ReLU, which starts from the same weights, and press Fit it:
the error reaches 0.0047.