20 lines of code that will beat A/B testing every time

Zwibbler.com is a drop-in solution that lets users draw on your web site. A/B testing is used far too often, for something that performs so badly. It is defective by design: Segment users into two groups. Show the A group the old, tried and true stuff. Show the B group the new whiz-bang design with the bigger buttons and slightly different copy. After a while, take a look at the stats and figure out which group presses the button more often. In recent years, hundreds of the brightest minds of modern civilization have been hard at work not curing cancer. With a simple 20-line change to how A/B testing works, that you can implement today, you can always do better than A/B testing -- sometimes, two or three times better. It can reasonably handle more than two options at once.. The Multi-armed bandit problem The multi-armed bandit problem takes its terminology from a casino. Like many techniques in machine learning, the simplest strategy is hard to beat. Why does this work? More blog entries

A Beginner's Guide to A/B Testing: Better Pay-Per-Click Ads Pay-per-click advertising is a key component of many online marketing campaigns. It can also be one of the most expensive ongoing costs in a campaign. Therefore, it’s key that you test your ads regularly, to make sure you aren’t letting conversions slip through the cracks. To an extent, PPC testing is simpler than many other kinds of A/B tests, partly because there are fewer things to test. But that doesn’t mean any less care and planning should go into preparing and executing these tests. This is the fourth installment in our A Beginner’s Guide to A/B Testing series. Deciding What to Test Pay-per-click ad testing is a bit more streamlined than the other topics we’ve covered in this series. The headlineThe body textThe linkThe keywords the ad displays for The headline is the part that’s going to show up as a link (in blue) in search results. The body text is the equivalent to your page’s description meta tag in organic search results. What Are You Testing For? Track and Analyze Your Results

A Beginner’s Guide To A/B Testing: Exceptional Web Copy Optimizing the copy on your website is at least as important as optimizing the design, especially if the primary goal of that site is to convert visitors. A pretty design can only get you so far. If you really want to gain new customers, you need to optimize the text on your site to instill trust in visitors and make them want to purchase from you. We often spend hours or days reading about the best techniques to use to sell products. But that doesn’t give us a complete picture, and what works great for one company or one product might not work at all for another. A/B testing is the simplest type of testing you can do to figure out which variations of copy, headline, and other factors are most effective in direct relation to your site and your offerings. Things to Test There are a ton of things you can test on your site to see what’s most effective. Here’s a brief list of the web copy elements on your site you might test: Tools for A/B Testing A/B Testing Best Practices

A Beginner's Guide To AB Testing: An Introduction A/B testing is a fantastic method for figuring out the best online promotional and marketing strategies for your business. It can be used to test everything from website copy to sales emails to search ads. And the advantages A/B testing provide are enough to offset the additional time it takes. Well-planned A/B testing can make a huge difference in the effectiveness of your marketing efforts. How Do You Plan an A/B Test? The first thing to do when planning an A/B test is to figure out what you want to test. With off-site tests, you’re probably testing either an ad, or a sales email. Once you know what you’ll test, make a list of all the variables you’ll test. the location of the call to actionthe exact text usedthe button color or surrounding space It’s a process, and it’s common for multiple A/B tests to be carried out prior to making a final decision or final change. Make sure that before you start testing you have a clear idea of the results you’re looking for. Things to Test

Most of your AB-tests will fail Jitbit Blog - please Subscribe if you like this post Jun 19 2013 A/B testing is overhyped. While A/B testing is a very powerful conversion optimization instrument, it requires lots of hard work but often it is just useless. Going from absolute total crap to something decent increases your conversions. No company have ever increased the revenue by changing an insignificant detail like a button color. Most of the tests produce no results A successful split test is a rare thing. From our experience, only one out of ten A/B tests produce results. Here is an example of a test we ran not so long ago. As you can see, the test had 85k participants and both alternatives have the exact same number of conversions. Another example: we sorted the plans in our pricing tables to go from the most expensive to the cheapest and vice versa, and tracked the most expensive plan orders. Again 30k participants and zero difference. We even radically redesigned our entire web-site from this: to this: Takeaways

A/B test case study: how two magical words increased conversion Posted in A/B Split Testing on May 26th, 2010 When we saw the results Soocial had got from their latest A/B test, we were astonished! They added just two words next to the Sign up button and the conversions shot up by 28%. If we say the phrase was one of these: “Sign up for Free”, “It’s Free” or “Free Signup”, can you guess which one did the trick? Background Soocial is an online address book that helps you keep your phone, computer and online services contacts sane. What was tested Their homepage features a large Sign up button. Results As you can read above, the changes on the page were extremely minor and, on the surface of it, look quite trivial. Control: 14.5% conversion rate Variation: 18.6% conversion rate The only difference between winning variation and the original design is presence of “It’s free!” Why “It’s Free” Worked When we asked Soocial why they thought the winning variation worked, this is what they had to say: How valuable was Visual Website Optimizer for the A/B Test? Tags

Start here: Statistics for A/B testing — The Product Management Coalition This article was first published internally at Freeletics. We sure take our decisions serious and don’t favour “hope-for-the-best” approaches. Also, we’re hiring Product Managers and Product Designers! What is an A/B test? An A/B test consists of taking two comparable groups of users and exposing them to two different versions (the control and a variation) of a software experience. Some relevant statistics concepts Overall Evaluation Criterion (OEC) Also called Primary Goal (in Optimizely and other tools) or Dependent variable, in statistics terminology. Null hypothesis (Ho) The hypothesis that the OECs for the variants are not different in fact, and that any observed differences during the experiment are due to random fluctuations. (Most of the statistics used in A/B testing come from the field of Statistical hypothesis testing.) P-value Also: Significance level (SL) Power The power of an experiment is influenced by a number of factors (such as sample size) and 80% is a typical desired value.