3 steps to quickly make a good multivariate test

21 02 2008

Having great testing technology puts a lot of power in your hands. You can test anything and everything you want. However, like any other tool, to use it effectively you have to use it right. There’s a lot of best practices and thought that goes into test design, but following these three rules can get you a good test in most situations.

  1. Maximize your traffic: Pack as much as you can into a test for the amount of traffic you have to keep it a short test. Using Widemile’s platform that’s 2 weeks to be safe, with Google Optimizer you should do at least a month (explanation).
  2. Test opposites: If you test stuff that’s similar, they’ll perform about the same. So find out the general theme you should be following first by testing opposites (B2B vs B2C, podcast vs ebook, descriptive vs benefits).
  3. Learn from the previous test: Always make sure you line up your tests so that you learn something that can be used in the next one to either refine or to learn something new.

The goal of these three things are to maximize your time spent testing by testing as much as possible while also minimizing testing suboptimal content. For example, if I was selling iPods and I tested 2 images of people running with the iPod, one with a man and the other a woman, I might think that was a good test. However I could have totally missed out on an image that worked better, such as an iPod next to a PC. I could test that out after the initial test, but then I just wasted one test run. The right way would be to test one sport image versus one PC image and find out which direction to go. From there I could test against other opposing images or refine the PC image.

The only warning I’d throw in is that if you’re trying to test a lot of things at once, you might want to scale back. Pick a 2-4 themes depending on your test size and stick to testing them out. Don’t mix and match.

Follow these steps and you’re on your way to getting not quick tests, but efficient ones.

Test nothing and get results

18 01 2008
Test Nothing

Now this is a blog about testing right? So why test nothing? Because nothing is powerful.

I don’t mean don’t test at all, I mean test having less on your page. Those awesome sub headlines your copywriter created? Or those testimonials? Your audience might not care or even look at them.

Best practices say to use trust logos, reviews, awards and a whole lot of other content, but you never know if it really helps. In fact, one of the biggest lifts ever at Widemile, came from removing everything but the core material on the page. One hero shot, one headline, one description and one button.

It was plain and simple.

To me and everyone else, it was an empty page filled with white space, but it converted at an extremely high rate compared to the previous page. Everyone at my office and at our client’s couldn’t believe it was the optimal page, but we couldn’t argue with the data.

While this might be an extreme example, it also is an example of the opportunities you may be missing if you don’t test turning off (hiding) elements on your page. So don’t just test something, test nothing too.