Multivariate testing: a quick primer

26 12 2007


Want to quickly get up to speed on multivariate testing? This post is designed to help you grasp the basics of multivariate testing so that you can get started talking and even doing your own tests. While its hard to design great tests, it’s easy to get good results with only a little education. I will definitely be teaching more about multivariate testing soon, but get this stuff down first!

Note: Since the industry is new, there isn’t consistency in much of the vocabulary of multivariate testing, so I will try to use generic terms.

Billy’s Multivariate Testing Primer

What is Multivariate Testing?: Testing multiple versions of a page to determine a set of elements* providing the highest conversion rate.

*Elements can be any content (text or image) on a page, typically hero shots, buttons, button text, headlines and text blocks. Sometimes it can refer to position/layout also.

What happens:

Multivariate testing diagram

  • Visitors come to a page and are shown a random version of the page
  • Conversions are tracked based on which page they saw
  • Once a statistically significant number of conversions is reached, analysis determines the elements on a page that create the highest conversion rate


  • If used correctly, it hones in on correct messaging direction and then exact messaging can be found using further testing
  • Allows for quicker testing of multiple elements than split or A/B testing
  • Analysis derived from live visitor data
    • Proves winning page elements to be better than others


  • Medium learning curve
    • Test results easily ruined by mistakes or poor methodology
  • Requires a minimum number of conversions
  • Code must be added to web page
  • Cookies and JavaScript must be enabled by visitors

Keep in mind:

  • Increasing the number of elements being tested increases the time needed for the test.
  • Number of conversions over time determines test size and length
    • The faster a page gets conversions, the shorter the test will be and/or the more things you can test
    • More conversions is better
    • A shorter time frame is better than a long time frame, but don’t go shorter than 2 weeks
  • Don’t test things that are too similar, look for different segments or messaging to pursue

Other concepts:

  • Using it with split testing:
    • Use split tests to determine the best layout with a template test. Test layout against layout with the same content (see this article for more advice.)
    • Afterwards, use multivariate testing to try out different messaging and refine it with continual tests.
    • If you want to try new layouts later on, go back to split testing.
  • Advanced: There are two types of multivariate tests: Full factorial and fractional/partial factorial
    • Full factorial means every version is shown. Meaning if you have 4 headlines and 4 buttons, that is 4×4=16 combinations, so 16 pages are used in the test.
    • Partial factorial is when only a portion of the total possible combinations are shown to visitors. This relies on statistical formulas and algorithms to determine the influence of the various factors since not every page combination is shown.
    • Full factorial takes a much longer time, so partial factorial allows for more testing in a quicker time frame.

There’s still a lot more to teach, even about some of the things mentioned here, but I hope this is enough to get you off the ground and really digging into testing. Google has their free tool if you want to try out multivariate testing.