background preloader

20 lines of code that will beat A/B testing every time

20 lines of code that will beat A/B testing every time
Zwibbler.com is a drop-in solution that lets users draw on your web site. A/B testing is used far too often, for something that performs so badly. It is defective by design: Segment users into two groups. Show the A group the old, tried and true stuff. Show the B group the new whiz-bang design with the bigger buttons and slightly different copy. After a while, take a look at the stats and figure out which group presses the button more often. In recent years, hundreds of the brightest minds of modern civilization have been hard at work not curing cancer. With a simple 20-line change to how A/B testing works, that you can implement today, you can always do better than A/B testing -- sometimes, two or three times better. It can reasonably handle more than two options at once.. The Multi-armed bandit problem The multi-armed bandit problem takes its terminology from a casino. Like many techniques in machine learning, the simplest strategy is hard to beat. Why does this work? More blog entries

How do you review code? I'm hoping to find ways to improve the code review process at the company where I work. My team has a fairly has a fairly standard github PR-based process. When you have some code you want to merge into the master branch you open a PR, ask another developer or two to review it, address any comments they have, and then wait for one of the reviewers to give it an LGTM (looks good to me). The problem is that there can be a lot of lag between asking someone to review the PR and them actually doing it, or between addressing comments and them taking another look. Worst of all, you never really know how long things will take, so it's hard to know whether you should switch gears for the rest of the day or not. Over time we've gotten used to communicating a lot, and being shameless about pestering people who are less communicative. So, has anyone else run I to similar problems?

A Beginner's Guide to A/B Testing: Better Pay-Per-Click Ads Pay-per-click advertising is a key component of many online marketing campaigns. It can also be one of the most expensive ongoing costs in a campaign. Therefore, it’s key that you test your ads regularly, to make sure you aren’t letting conversions slip through the cracks. To an extent, PPC testing is simpler than many other kinds of A/B tests, partly because there are fewer things to test. But that doesn’t mean any less care and planning should go into preparing and executing these tests. This is the fourth installment in our A Beginner’s Guide to A/B Testing series. Deciding What to Test Pay-per-click ad testing is a bit more streamlined than the other topics we’ve covered in this series. The headlineThe body textThe linkThe keywords the ad displays for The headline is the part that’s going to show up as a link (in blue) in search results. The body text is the equivalent to your page’s description meta tag in organic search results. What Are You Testing For? Track and Analyze Your Results

Nick Craver - Stack Overflow: How We Do Deployment - 2016 Edition This is #3 in a very long series of posts on Stack Overflow’s architecture. Previous post (#2): Stack Overflow: The Hardware - 2016 Edition We’ve talked about Stack Overflow’s architecture and the hardware behind it. The next most requested topic was Deployment. How do we get code a developer (or some random stranger) writes into production? I’m going ahead and inserting a set of section links here because this post got a bit long with all of the bits that need an explanation: Source This is our starting point for this article. A Little Context We deploy roughly 25 times per day to development (our CI build) just for Stack Overflow Q&A. The Human Steps When we’re coding, if a database migration is involved then we have some extra steps. And here’s the local %Repo%\StackOverflow.Migrations\ folder: You can see both in chat and locally that 726 was the last migration number taken. Now let’s add some code — we’ll keep it simple here: A \StackOverflow\Models\User.cs diff: .. So far, so good. Tiers

A Beginner’s Guide To A/B Testing: Exceptional Web Copy Optimizing the copy on your website is at least as important as optimizing the design, especially if the primary goal of that site is to convert visitors. A pretty design can only get you so far. If you really want to gain new customers, you need to optimize the text on your site to instill trust in visitors and make them want to purchase from you. We often spend hours or days reading about the best techniques to use to sell products. But that doesn’t give us a complete picture, and what works great for one company or one product might not work at all for another. A/B testing is the simplest type of testing you can do to figure out which variations of copy, headline, and other factors are most effective in direct relation to your site and your offerings. Things to Test There are a ton of things you can test on your site to see what’s most effective. Here’s a brief list of the web copy elements on your site you might test: Tools for A/B Testing A/B Testing Best Practices

Rule of Silence definition by The Linux Information Project The rule of silence, also referred to as the silence is golden rule, is an important part of the Unix philosophy that states that when a program has nothing surprising, interesting or useful to say, it should say nothing. It means that well-behaved programs should treat their users' attention and concentration as being valuable and thus perform their tasks as unobtrusively as possible. That is, silence in itself is a virtue. There is no single, standardized statement of the Unix philosophy, but perhaps the simplest description would be: "Write programs that are small, simple and transparent. This philosophy has been fundamental to the the fact that Unix-like operating systems have been thriving for more than three decades, far longer than any other family of operating systems, and can be expected to see continued expansion of use in the years to come, particularly in the form of Linux. The rule of silence is one of the oldest and most persistent design rules of such operating systems.

A Beginner's Guide To AB Testing: An Introduction A/B testing is a fantastic method for figuring out the best online promotional and marketing strategies for your business. It can be used to test everything from website copy to sales emails to search ads. And the advantages A/B testing provide are enough to offset the additional time it takes. Well-planned A/B testing can make a huge difference in the effectiveness of your marketing efforts. How Do You Plan an A/B Test? The first thing to do when planning an A/B test is to figure out what you want to test. With off-site tests, you’re probably testing either an ad, or a sales email. Once you know what you’ll test, make a list of all the variables you’ll test. the location of the call to actionthe exact text usedthe button color or surrounding space It’s a process, and it’s common for multiple A/B tests to be carried out prior to making a final decision or final change. Make sure that before you start testing you have a clear idea of the results you’re looking for. Things to Test

The Wrong Abstraction — Sandi Metz I originally wrote the following for my Chainline Newsletter, but I continue to get tweets about this idea, so I'm re-publishing the article here on my blog. This version has been lightly edited. I've been thinking about the consequences of the "wrong abstraction." My RailsConf 2014 "all the little things" talk included a section where I asserted: duplication is far cheaper than the wrong abstraction And in the summary, I went on to advise: prefer duplication over the wrong abstraction This small section of a much bigger talk invoked a surprisingly strong reaction. The strength of the reaction made me realize just how widespread and intractable the "wrong abstraction" problem is. Programmer A sees duplication.Programmer A extracts duplication and gves it a name. Existing code exerts a powerful influence. When you appear in this story in step 8 above, this pressure may compel you to proceed forward, that is, to implement the new requirement by changing the existing code. News: 99Bottles Book

Most of your AB-tests will fail Jitbit Blog - please Subscribe if you like this post Jun 19 2013 A/B testing is overhyped. While A/B testing is a very powerful conversion optimization instrument, it requires lots of hard work but often it is just useless. Going from absolute total crap to something decent increases your conversions. No company have ever increased the revenue by changing an insignificant detail like a button color. Most of the tests produce no results A successful split test is a rare thing. From our experience, only one out of ten A/B tests produce results. Here is an example of a test we ran not so long ago. As you can see, the test had 85k participants and both alternatives have the exact same number of conversions. Another example: we sorted the plans in our pricing tables to go from the most expensive to the cheapest and vice versa, and tracked the most expensive plan orders. Again 30k participants and zero difference. We even radically redesigned our entire web-site from this: to this: Takeaways

Related: