The whole idea.
A convolutional network turns a grid of pixels into a handful of class probabilities, one stage at a
time. Every stage on this page is one you have met on its own:
• Convolution: a 3×3 kernel slides over the image, as on the edge-detection page.
Here there are four, each looking for a short straight stroke in one direction. A run of three inked
pixels in the kernel's own direction scores 6, and the kernel's bias of −3 brings that down to 3.
The end of a stroke, a run of two, is left with 1; a single stray pixel ends up at −1.
• ReLU: every negative becomes 0, as on the activation page, so only real strokes survive.
• Max pooling: each 2×2 window keeps its largest value, as on the pooling page, and
the map shrinks from 26×26 to 13×13.
• Global max: each map becomes one number, its largest value anywhere: “is there a
stroke in this direction somewhere?” The whole image is now four numbers.
• Scores: each class adds up those four numbers with its own weights, plus a bias. A
+ wants vertical and horizontal and no diagonals; its bias of −3 means one direction
alone is not enough.
• Softmax: the scores become probabilities, as on the softmax page.
Every weight here was set by hand, so every number can be explained. A real network starts with random
weights and learns them with an optimizer.
Things worth trying:
1. Pick |. Only the vertical filter lights up, so the four numbers are 3, 0, 0, 0 and the
network says | with 95.0%. Now draw a horizontal stroke across it: the horizontal map lights
up and + takes over at 90.7%.
2. Click any pixel in a map to see its arithmetic: the 3×3 patch of the drawing, times the kernel,
plus the bias, then ReLU, then the 2×2 pooling window it falls in. Click one of the four numbers on
the right to jump to the spot where that maximum was found.
3. Pick *, a plus and a cross on top of each other. All four filters fire, exactly as for a
circle: both give 3, 3, 3, 3, and the network calls the star a ○ with 99.5%. Global max keeps
whether each direction appears and throws away where, and that is all this network
knows. Real CNNs stack many convolution layers, so that a second layer can combine strokes into corners
and curves, a third into parts, and so on. This page has one, to keep every number explainable.
4. Draw small. A circle with a radius under about 5 pixels can lose one of its directions, because its
curve has no straight run of three pixels that way, and it can come out as +. A 3×3
kernel only ever sees 3×3 pixels at a time.
5. In a network this short, pooling does not change the answer: the global max would find the same peak
in the 26×26 map. It is here because in a deeper network the next convolution works on the smaller
map, which is cheaper and lets each of its kernels see a wider part of the image.