5 quick tips to effective A/B and split testing

5 12 2007

New URL testingblog.widemile.com

While multivariate might be the hottest testing subject, you can’t beat a good split test in certain situations.

If you already know the difference between A/B and split tests, skip this part, otherwise it’s a quick read.

A/B testing is when you test one page then replace it with a new page, so the two versions are running concurrently, one after another. Split testing is when you test two pages at once, where some distribution of traffic is sent to either of the pages simultaneously.

A/B Test

A/B Test


Split Test (note you can split different %’s)

Split test

Now for the good stuff! I’ll try to keep this short so let’s start:

  1. A/B testing is out: Use split testing instead. Split testing is more accurate since it uses the same time period of traffic. Traffic during Halloween is different from traffic during Christmas, so testing one page at one time and one at another will skew your results making a relevant comparison impossible. Use A/B only if you don’t have permission/the capability to do split testing (Google Optimizer is free and allows split testing!)
  2. Template test = Split test: This is the sweet spot for split tests. Use one if you want to try a new layout/template against your old one or if you want to test two new pages. Here’s my template test primer if you need to brush up.
  3. One exception, one lesson: During these tests everything should be the same except for one thing on the page. If you try introduce 2 or more changes into a split or a/b test you won’t know which change improved your page. The only time I might have multiple changes is for template tests, where the new template can’t use the previous creative effectively. Still, emulate the creative as closely as possible for the new template.
  4. Be ready for your next test: Since these tests are easier to execute, you should also have an easier time getting the next test ready to go for when the first one finishes. Make tests ahead of time so that when the current test completes you can flip the switch and quickly get it measured and done with.
  5. Learn from the first test: You already completed a test, what does that tell you about what you should test for the next time? If a graphic heavy template beat the cleaner template, try testing against an even more graphics heavy template. Find where your customers lie and pinpoint it by seeing what each test tells you. This is a game of Marco Polo. You customers are shouting, “Polo!” with each test, follow them!

Split tests keep it simple and that is its strength. As long as you control anything that might confuse the test (like introducing new content), you can find winners and make a great page. However, after split testing, multivariate testing should be brought in to really pull out more from your page. But that’s a whole other blog….