Intersection over Union, box against box

The whole idea.
A detector says “there is a cat in this box.” To grade it, you need one number for how well its box matches the real one. IoU is that number: the area the two boxes share, divided by the area they cover between them.

IoU = shared ÷ (real + predicted − shared)

It is 1 when the boxes are identical and 0 when they do not touch. The common rule is that a prediction counts as finding the object when IoU is 0.5 or more — the same rule the object-detection page uses for its final count. NMS on that page uses IoU too, to decide whether two boxes are the same cat.

Why divide by the union?
The shared area alone would reward cheating: a box over the whole picture shares all of the cat. Dividing by everything either box covers punishes a box for being too big, just as much as for being in the wrong place or too small.

Things worth trying:
1. Shifted by a third. Two boxes the same size, one moved over by a third of its width. They still look like a decent match, but IoU is exactly 0.50 — right on the line. Nudge the blue box one pixel further with an arrow key and it drops under.

2. Small box, 5 px off, then Big box, 5 px off. The same 5-pixel mistake gives 0.60 on a 20-pixel box and 0.90 on a 100-pixel box. A few pixels is nothing for a big object and a lot for a small one, which is why small objects are the hardest to score well on.

3. Too big, then Too small. A box twice as wide and twice as tall, with the cat entirely inside it, scores 0.25 — a miss. A box half as wide and half as tall, entirely inside the cat, also scores 0.25. IoU does not care which way you got the size wrong.

4. Cover everything. The prediction contains the whole cat, and IoU is 0.03.

5. Some benchmarks find a 0.5 pass mark too generous. COCO, the most widely used one, grades detectors at every threshold from 0.50 to 0.95 and averages the results — so a box has to be tight, not just close, to score well.