Northfield Analytics

Notes on measurement, sampling and data quality

Percentiles lie when the sample is small

A p99 computed over 40 events is not a p99. What to report instead when traffic is thin.

Counting unique visitors without storing them

HyperLogLog gets you within two percent and keeps the log free of identifiers.

Why your A/B test ended too early

Peeking at results daily inflates false positives. A short note on sequential testing.

Timezones in daily aggregates

The most common reporting bug we see, and a one-line fix.