Each word's probability at every temperature, for the scores above. The dashed line is the
current temperature. Far to the right every word heads for an even share; far to the left the top
word takes everything.
The whole idea.
A language model writes one word at a time. For the next word it gives every word it knows a
score. Softmax turns the scores into probabilities in two steps: raise e to each
score, then divide each one by the total, so they add up to 100%. It is the same softmax as for the
image classes on the CNN page.
Then the model has to pick one word. Greedy always takes the top word.
Sampling draws one at random, with the probabilities as the odds, so the same sentence can end
differently each time. Temperature divides every score before softmax: below 1 the top word
gets even more likely, above 1 the unlikely words get a real chance.
Things worth trying:
1. Choose Greedy and press Write 10 endings: ten times “mat”, at any
temperature. Temperature never changes which word is on top.
2. Choose Sample at temperature 1. “mat” is 51.2%, so about 5 of the 10 endings
say “mat”, and the rest are spread over the other words.
3. Drop the temperature to 0.2: “mat” is 99.2%, and nearly every ending is the same. Raise
it to 5: “moon” goes from 0.6% to 8.3%, so it turns up in about 1 ending in 12.
4. Drag “moon” up to 3.5, above “mat”. Now “moon” is the top word
at 45.9%, and greedy writes “The cat sat on the moon” every time. Softmax only knows the
scores, not what makes sense.