Practical Guide to Controlled Experiments on the Web. Building an A/B Testing Framework for Web Applications. SPSS, an IBM Company: @cvahanian: Which software... Fonctions XL en 9 langues. Out of the cesspool and into the sewer: A/B testing trap. Your A/B tests are trapped in a cesspool when they should be in the sewer. Do you really care why A/B testing is analogous to unwanted liquids?
Not yet, so I’d better get right to the point. On the rare occasion that it rains in Austin we get these deep puddles in the backyard. Of course it would be better if the water would flow out into the street and into the sewer, but that’s not how gravity works. Water “seeks” the lowest point in the yard, but it’s narrow-minded.
It doesn’t survey the environment, locate the lowest area, and head there. In mathematical terms, water doesn’t “globally optimize” for getting to the lowest possible point, but rather “locally optimizes” at each step. Maybe your A/B tests are missing the forest for the trees too. A typical A/B test looks like this: You start with a baseline, then you make a change. And so on, inching your way through incremental improvements. Except, often it doesn’t. At this point you might be tempted to give up, but that’s wrong too. Play! Easy statistics for AdWords A/B testing, and hamsters. So you’ve got your AdWords test all set up: Will people go for the headline “Code Review Tools” or “Tools for Code Review?”
Gee they’re both so exciting! Who could choose! I know, I know, settle down. This is how these things go. Anyway, the next day you have 32 clicks on variant A (“Code Review Tools”) and 19 clicks on B (“Tools for Code Review”). The answer matters. Normally a formal statistical treatment would be too difficult, but I’m here to rescue you with a statistically sound yet incredibly simple formula that will tell you whether or not your A/B test results really are indicating a difference. I’ll get to it in a minute, but I can’t help but include a more entertaining example than AdWords. In the movie, Hammy chooses the organic produce 8 times and the conventional 4 times. If you’re like me, you probably think “organic” is the clear-cut winner — after all Hammy chose it twice as often as conventional veggies.
Okay okay, we suck at math. Define N as “the number of trials.” The Click-Through That May Be Hurting Your Brand - Advertising A.