the real dots (the smiley the network learned from)
new dots, starting as pure noise
The whole idea. Adding noise is easy. Mix a little random noise into a picture, then a little more, step
after step. After 100 steps there is only static. No learning is needed for this part:
noisy picture = a × picture + b × noise [where a = how much picture, b = how much noise, and a² + b² = 1]
Taking noise away has to be learned. To train, take a real example, add noise up to a
random step, and ask a network to guess the noise that was added. The loss is how far off the
guess is.
To make something new, start from pure noise. Ask the network for the noise, remove a
little of it, and repeat, 100 times. What is left looks like the real data, but it is new.
Part 2 trains on dots, not pictures, so it can learn in a few seconds here. Each dot is a tiny
“picture” with only 2 numbers, x and y. A picture model does the same thing with
every pixel, with a much bigger network and usually about 1,000 steps.
Diffusion and GANs.
A GAN’s forger makes a picture in one pass. Diffusion needs one pass per step — 100
here — so it is slower. But it trains one network on one simple job, guessing the noise,
with no tug-of-war between two networks. That makes it steadier to train.
Things worth trying:
1. Move the slider. At step 10, 90% of the picture’s strength is left. At step 25 it is
53% and the smiley is hard to see. At step 50 it is 7%, and by step 100 it is 0%: pure
noise.
2. Before training, press Run to the end. The network only makes random guesses, and in
10 runs out of 10, 0% of the dots ended up on the smiley.
3. Train the noise guesser (about 8 seconds), then Run to the end. Over 10
runs, 68–75% of the new dots landed on the smiley. Use Step ×10 to watch:
at step 10 only about 30% are on it. Most of the shape appears in the last 10 steps.
4. Train more, then make a new cloud. Over 10 runs, 73–81% of the dots landed on
the smiley.
5. The loss stops near 0.34 and never reaches 0. The guess is easy when there is a lot of noise,
because then the dot is almost all noise (miss 0.01 for steps 51–100). It is hard when
there is very little noise (miss about 1.45 for steps 1–10): is a dot off the smiley
because of noise, or is that where it really belongs?