How to get ideal test conditions (and results)

4 03 2008

A big mistake in testing is to overlook variables inside and outside of the test that impact results. In an ideal test, the only variables would be the ones you are testing on your page. That usually isn’t possible though, but as long as you account for them in your analysis, you will get correct and actionable information.

Sky image

If you test a seasonal page, then the optimal page you get for that season, probably won’t perform when the season ends. By not paying attention to those kind of variables, you are setting yourself up into thinking you’ve found the optimal page. The same type of mistake is made by grouping e-mail, print, SEM campaigns and event traffic, unless you know they react the same to your changes.

Even within segments, there might be more segments to uncover. Your only limitation should be traffic; don’t segment so granular that you can’t run a decent sized test in a decent amount of time.

One of my clients doesn’t get a lot of traffic, but the traffic he does get is very distinct. One converts in the single digits and the other converts in the teens. Although combining them would get me more data, it would be very confused data since they convert so differently.

A few things to look out for:

  • The ad or offers visitors see beforehand
  • Interactions between your factors (if you aren’t testing interactions)
  • Technical problems
  • Problems that occur before or after the tested page

A note about the last bullet, the problems can range from a technical problem to a problem with the overall funnel. If people get different experiences in the funnel that drastically impact whether they convert or not, it can add a noise to your test. Some examples are different checkout processes for registered and non-registered users or users being inelligible for service.

The purpose of testing is to find out if a certain element performs well under the conditions you provide. If you aren’t paying attention to all the conditions, then the results you derive will be incorrect without you knowing.

3 steps to quickly make a good multivariate test

21 02 2008

Having great testing technology puts a lot of power in your hands. You can test anything and everything you want. However, like any other tool, to use it effectively you have to use it right. There’s a lot of best practices and thought that goes into test design, but following these three rules can get you a good test in most situations.

  1. Maximize your traffic: Pack as much as you can into a test for the amount of traffic you have to keep it a short test. Using Widemile’s platform that’s 2 weeks to be safe, with Google Optimizer you should do at least a month (explanation).
  2. Test opposites: If you test stuff that’s similar, they’ll perform about the same. So find out the general theme you should be following first by testing opposites (B2B vs B2C, podcast vs ebook, descriptive vs benefits).
  3. Learn from the previous test: Always make sure you line up your tests so that you learn something that can be used in the next one to either refine or to learn something new.

The goal of these three things are to maximize your time spent testing by testing as much as possible while also minimizing testing suboptimal content. For example, if I was selling iPods and I tested 2 images of people running with the iPod, one with a man and the other a woman, I might think that was a good test. However I could have totally missed out on an image that worked better, such as an iPod next to a PC. I could test that out after the initial test, but then I just wasted one test run. The right way would be to test one sport image versus one PC image and find out which direction to go. From there I could test against other opposing images or refine the PC image.

The only warning I’d throw in is that if you’re trying to test a lot of things at once, you might want to scale back. Pick a 2-4 themes depending on your test size and stick to testing them out. Don’t mix and match.

Follow these steps and you’re on your way to getting not quick tests, but efficient ones.

What’s an average conversion rate? 40%!

1 02 2008


Would you believe that? And if it were true, would it really mean anything to you? It shouldn’t.

Conversion Rate Table

I get asked this question fairly often and at first glance it seems like a logical question to ask, but really the focus should be elsewhere.

From my experience, conversion rates range from less than a percent all the way up to 30% or more. Does knowing that help me optimize my clients’ pages? No. Every page has so many variables internally and externally that it is very difficult and nonsensical to worry about the average conversion rate.

The goal of your page, differences between your product/service against your competition, target you’re trying to reach, avenues you advertise and numerous other factors all effect your conversion rates. A competitor having a higher conversion rate than you, does not mean you’re doing something wrong. Set the baseline for yourself and keep improving it. That’s how marketers should approach their conversion rate.

If you’re testing, you’ll find out if you’re campaign is performing suboptimal and find out what the optimal is at the same time.

Pretty amazing huh?

I don’t tell clients I’m going to get their conversion rates above industry averages, I tell them that I’m going to make their campaign as successful as possible. Do that and you’ll be ahead of competition and ahead of where you were when you first started.

Google Optimizer is slow (or Not all Multivariate Testing is the same)

28 01 2008


Without knowing it, people might assume that there’s only one method to multivariate testing. That it has been long figured out by math and statistic wizards. I have learned otherwise from Widemile’s personal math wizard, Chief Scientist, Vladimir Brayman.

(Just as a side note, he does not have a typical office. Rather than papers and folders strewn about, he has statistic and math books. Lucky for me though, he has a great skill at distilling all the goodness in those books and teaching me what I need to know, in a way I understand.)

Most recently, we discussed why Widemile’s technology trumps Google Optimizer.

Widemile vs Google


Having a strong creative team and testing experts ensures better results than giving a marketer a tool like Google Optimizer, that’s easy for most people to understand. But explaining how Widemile’s technology can test more, faster, is a little more complicated.

Let’s explore how Google’s testing works versus Widemile’s. Google Optimizer uses full factorial test design, meaning it creates a page for every combination of your tested page elements. So if you wanted to test 4 different hero shots, 4 buttons and 4 headlines, that would require 4*4*4=64 page combinations. The disadvantage of this method is that you need significant traffic for each of the 64 pages. Meaning you either need a lot of traffic or a lot of time; for most companies, they’ll need both.

To solve this, Widemile’s optimization platform use fractional factorial test design. This method tests only a small fraction of the total possible page combinations and uses statistical analysis to derive almost all of the same information that would be found in a full factorial test. This works because marginal information is gained in testing all 64 page combinations, while testing a few important combinations tell us nearly everything we need to know.

Google actually criticizes fractional factorial test design (look here where it says “A note about ‘fractional factorial testing'”), saying that it requires the same number of impressions, but can not derive the depth of conclusions that a full factorial design can. While true that full factorial squeezes out the most information, that is at a sacrifice of extending the test many times longer than with a fractional factorial test, all to learn the smallest influences.

Doing successive tests to find high influence items with fractional factorial testing will get much higher gains than getting every ounce of information out of one extremely long full factorial test. In addition, with a carefully designed fractional factorial test you can learn all the major influences and the interactions between elements on the page.

Fractional factorial test design gets you a completed test in weeks rather than months or years even, and because of that, you can test more than you would normally be able to in the same time frame. You can either test more in one larger test, or do many smaller successive tests.

Not to say that Google Optimizer isn’t a great tool, especially since it is free, but any company that spends thousands of dollars on SEM has a lot to gain by using technology that gets rapid results.

If you got any questions about this, let me know and I’ll try to answer them or get you an answer.

SEO + Testing = Friends

11 12 2007

One common concern with optimization is that testing content will bring down SEO. In actuality, testing will not affect SEO.

Here’s what happens, a regular visitor goes to the web page and sees an experiment, which is a version of the tested page. Google’s spiders go and only see the default content, meaning what existed on the page previously (or whatever you want default content to be.)

What a visitor and Google spider see:
SEO Explanation

Don’t believe me? Here are some quotes from two industry folks that know their stuff:

Basically conversion testing is a legal form of cloaking. What you’re really doing is showing a bunch of different versions to people to find the best one, but you’re only showing one version to the search engines.
Rand Fishkin, CEO of SEOmoz

With Google, the spiders don’t look at the JavaScript content so the source code will look consistent. And even if the layout is different, the spirit of the content is similar.
Tom Leung, business product manager of Google Optimizer

Before you start testing, I’d make sure with your testing company that it works the same way with them. Generally though, if the content is coming in using javascript, it will work just like this.

You should worry about if what you’re testing will be good for SEO or not though. And what’s good for SEO is a slightly bigger topic… (understatement.)

Quotes from