Each class's probability at every temperature, for the scores above. The dashed line is the
current temperature. Far to the right every class heads for an even 25%; far to the left the top
score takes everything.
The whole idea.
An image classifier ends with one number per class, a score (also called a logit).
Scores can be any size and any sign, so they are not probabilities yet. Softmax turns them into
probabilities in two steps:
• raise e to each score, ez, which makes every number positive;
each point of score multiplies it by e, about 2.72;
• divide each one by the total, so they add up to 100%.
With the starting scores, cat 2.0, dog 1.0, car 0.1 and bird −1.0, that is 7.39, 2.72, 1.11 and
0.37, total 11.58, so cat 63.8%, dog 23.5%, car 9.5% and bird 3.2%. Every probability depends on every
score: drag one slider and all four bars move.
Temperature.
Temperature T divides every score before softmax: ez/T. Below 1
it stretches the gaps, so the top class gets more sure; above 1 it shrinks them, so everything evens out.
It never changes the order: cat stays first at every temperature.
Things worth trying:
1. Slide the temperature with the starting scores. Cat is 86.2% at 0.5, 63.8% at 1, 45.1% at 2
and 28.8% at 10, against an even split of 25%. At 0.2 it is already 99.3%.
2. Press Add 5 to every score. The bars do not move: still 63.8%, 23.5%, 9.5%, 3.2%. Softmax
only cares about the gaps between scores, not the scores themselves. Real code uses this: it subtracts
the largest score first, because e1000 is too big for a computer to store, while
e0 is just 1.
3. Press Reset, then Double every score. Now cat is 86.2%, dog 11.7%, car 1.9%, bird
0.2%: exactly what temperature 0.5 gave in step 1. Doubling the scores and halving the temperature are
the same thing.
4. Drag dog up to 2.0, level with cat. They split evenly, 45.5% each, and car and bird share what is
left. Equal scores always mean equal probabilities.
5. Draw 20 times picks a class at random, using the bars as the odds. At temperature 1 cat
comes up about 64% of the time, so roughly 13 of the 20. Drop the temperature to 0.2 and draw again:
nearly all cat. Raise it to 10: a near-even mix. An image classifier simply reports the top class, but
this is exactly how chatbots pick their next word, and why their temperature setting makes them
safer or more adventurous.