A/B Testing & Experimentation
Lightweight experimentation for developers — A/B testing without heavy tooling, statistical significance, and sample-size questions.
- How to Calculate Sample Size for an A/B Test — Calculate A/B test sample size with n = 16 × p × (1−p) / δ² per variant, where p is the baseline rate and δ the absolute lift to detect. Includes a runnable function and worked example.
- How Many Visitors Do You Need for an A/B Test? — How many visitors an A/B test needs depends on your baseline rate and the smallest lift you want to detect. Small lifts need huge samples. Here's a quick reference table.
- Is My A/B Test Result Statistically Significant? — Your A/B test is significant if a two-proportion z-test gives |z| above 1.96 (p < 0.05) at a pre-decided sample size. Here's how to compute it and the mistakes that fake significance.
- How to A/B Test a Landing Page Without a Testing Platform — A/B test a landing page with no testing tool: split visitors with a cookie, tag the variant on every PostHog event, compare conversion in HogQL, and check significance by hand.