1- A/B Testing Tech Note: determining sample size by Noah of 37signals. In discussions on our posts about A/B testing the Highrise home page, a number of people asked about sample size and how long to run a test for.
It’s a good question, and one that’s important to understand. Running an A/B test without thinking about statistical confidence is worse than not running a test at all—it gives you false confidence that you know what works for your site, when the truth is that you don’t know any better than if you hadn’t run the test. There’s no simple answer or generic “rule of thumb” that you can use, but you can very easily determine the right sample size to use for your test.
What drives our needed sample size? There are a few concerns that drive the sample size required for a meaningful A/B test: 3- Statistical Analysis and A/B Testing. In this article we're going to talk about how hypothesis testing can tell you whether your A/B tests actually effect user behavior, or whether the variations you see are due to random chance.
First, if you haven't yet, read my previous introductory article on hypothesis testing. It explains the statistical principles behind hypothesis testing using the example of a biased coin. We're going to move quickly beyond that and dive right into A/B testing. Landing Page Conversion. 2- Split Testing Confidence Calculator - AB Tester.