The whole idea.
A z-score just re-expresses a reading as how many standard deviations it sits from the
mean:
z = (x − μ) / σ
That is the entire method. It has no notion of an anomaly — you supply one, by picking a
cut-off and calling anything past it odd. The usual choices are 2, 2.5 and 3, and they are
conventions, not discoveries.
Both μ and σ are measured from the very data you are screening, its weaknesses!
Cut-off
On data that really is normal, this flags
In 200 readings
|z| > 2
4.6% of perfectly ordinary points
about 9
|z| > 2.5
1.2%
about 2 or 3
|z| > 3
0.3%
about 1 in every 2 samples
Things worth trying:
1. None, |z| > 2, and press New random sample a few times. There is nothing
wrong with this data — it is one clean bell curve — and yet roughly 9 points are
flagged every single time. That is not the method failing. At |z| > 2 you have asked
for the outer 4.6%, so on clean data you get false alarms.
2. Now Many, |z| > 3. Twenty odd readings, and σ has more than doubled
— from about 15 to about 32. The outliers have stretched the very ruler you
are measuring them with, so only about 7 of the 20 are still caught; the other 13 hide
behind the σ they themselves inflated. That is masking. Drop the cut-off to
|z| > 2 and all 20 come back with almost no false alarms — but only because σ is
now so wide that an ordinary reading cannot reach 2σ either. The fix is not a different
cut-off. It is a statistic the outliers cannot drag: the median and the quartiles barely move,
which is why the IQR rule catches all 20 of these.