R-CNN, region by region

The whole idea.
The object-detection page slides one fixed-size window over every position. R-CNN (2014) checks far fewer places. First it proposes regions: boxes that might hold an object. It finds them with selective search, which starts from small patches and keeps merging the two neighbors that look most alike; every merged group gives one box. Then it takes each region, cuts it out, squeezes it to the fixed size the CNN expects, and runs the CNN on it to get a class score. A squeezed region always has the right size, so a big cat and a small cat look the same to the CNN. A proposed region rarely fits the animal exactly, so R-CNN also nudges each box edge to fit better (box regression). Last, NMS removes the duplicates.

Fast R-CNN (2015) noticed that the regions overlap, so R-CNN runs the CNN over the same pixels again and again. It runs the CNN once on the whole picture, then cuts every region out of the resulting feature map and pools it to a fixed grid (RoI pooling). Same regions, one CNN run.

Faster R-CNN (2015) also replaced selective search. A small Region Proposal Network looks at the same feature map, puts a few anchor boxes of different sizes at every position, and scores each one: is there an object here? Only the best few dozen become regions. Proposals and checks now share one CNN run.

What is simulated here.
There is no trained network on this page. The “CNN” is the template score from the object-detection page: how closely the squeezed region matches a small cat or dog, from −1 to +1. The “feature map” is the picture shrunk 4 times; a real one has hundreds of channels. A real detector learns the box nudge and makes it in one step; this page finds it by trying small moves of each edge and keeping the best score. What the page shows as it really is: where the regions come from, what gets cut out, and how many times the CNN has to run.

Things worth trying:
1. On R-CNN, press Propose regions. Selective search starts from 900 small patches and ends with 111 regions; most of them are background. Press Check them all and watch the counter: 111 regions take 111 CNN runs. On a real photo, R-CNN checks about 2,000 regions — 2,000 CNN runs for one picture. Then Run NMS: all 5 animals are found.

2. Watch the panel while it checks: a region around an animal usually scores only 0.5 to 0.8 as proposed, because it does not fit the animal exactly. The nudge lifts it to 0.96–0.99. Before the nudge, only 2 regions clear the 0.70 cut-off; after it, 8 do.

3. Switch to Fast R-CNN and run it again. The same 111 regions, but the counter says 1 CNN run. Press The feature map to see what every region is cut from. The scores come out a little lower (0.86–0.97), because the feature map is coarser than the picture — the price of sharing one CNN run. All 5 animals are still found.

4. Switch to Faster R-CNN. The Region Proposal Network scores 4,900 anchors and keeps only 50 regions — less than half of what selective search gave — and everything still takes one CNN run. All 5 animals are found; the big dog is the weakest, at 0.77.

5. Pull the score cut-off down to 0.50 on R-CNN: one false alarm appears: a small patch of the big dog’s face, scoring 0.50 as a cat. A lower cut-off catches more, right and wrong.