Multivariate testing: a quick primer

Want to quickly get up to speed on multivariate testing? This post is designed to help you grasp the basics of multivariate testing so that you can get started talking and even doing your own tests. While its hard to design great tests, it’s easy to get good results with only a little education. I will definitely be teaching more about multivariate testing soon, but get this stuff down first!

Note: Since the industry is new, there isn’t consistency in much of the vocabulary of multivariate testing, so I will try to use generic terms.

What is Multivariate Testing?: Testing multiple versions of a page to determine a set of elements* providing the highest conversion rate.

*Elements can be any content (text or image) on a page, typically hero shots, buttons, button text, headlines and text blocks. Sometimes it can refer to position/layout also.

What happens:

• Visitors come to a page and are shown a random version of the page
• Conversions are tracked based on which page they saw
• Once a statistically significant number of conversions is reached, analysis determines the elements on a page that create the highest conversion rate

Strengths:

• If used correctly, it hones in on correct messaging direction and then exact messaging can be found using further testing
• Allows for quicker testing of multiple elements than split or A/B testing
• Analysis derived from live visitor data
  • Proves winning page elements to be better than others

Weaknesses:

• Medium learning curve
  • Test results easily ruined by mistakes or poor methodology
• Requires a minimum number of conversions
• Code must be added to web page
• Cookies and JavaScript must be enabled by visitors

Keep in mind:

• Increasing the number of elements being tested increases the time needed for the test.
• Number of conversions over time determines test size and length
  • The faster a page gets conversions, the shorter the test will be and/or the more things you can test
  • More conversions is better
  • A shorter time frame is better than a long time frame, but don’t go shorter than 2 weeks
• Don’t test things that are too similar, look for different segments or messaging to pursue

Other concepts:

• Using it with split testing:
  • Use split tests to determine the best layout with a template test. Test layout against layout with the same content (see this article for more advice.)
  • Afterwards, use multivariate testing to try out different messaging and refine it with continual tests.
  • If you want to try new layouts later on, go back to split testing.
• Advanced: There are two types of multivariate tests: Full factorial and fractional/partial factorial
  • Full factorial means every version is shown. Meaning if you have 4 headlines and 4 buttons, that is 4×4=16 combinations, so 16 pages are used in the test.
  • Partial factorial is when only a portion of the total possible combinations are shown to visitors. This relies on statistical formulas and algorithms to determine the influence of the various factors since not every page combination is shown.
  • Full factorial takes a much longer time, so partial factorial allows for more testing in a quicker time frame.

There’s still a lot more to teach, even about some of the things mentioned here, but I hope this is enough to get you off the ground and really digging into testing. Google has their free tool if you want to try out multivariate testing.

Why always optimize landing pages?

Sometimes people ask me why landing pages are such popular targets for optimization and testing. Why not optimize a home page? A product page?

Actually, we can optimize those kinds of pages, but almost all businesses come to us with a landing page that needs help. Beyond the demand for landing page testing though, is the fact that landing pages inherently are fit for testing. Let me explain.

One of Widemile’s landing pages

The strength of multivariate and split testing is in pulling out the best page possible for what you want to do. With a homepage you have multiple things you want to do (e.g. show your products, company history, customer service, get people to spend X minutes browsing) and with a product page you are typically working with a CMS template for your whole site. Those factors complicate things a bit since you have to figure out what counts as improvement for the homepage and for the product page, you have to work with a CMS system and make changes that improve the majority of products pages using that template.

These things are not impossible to do or even difficult in many situations, but a landing page is usually totally independent of everything and has only one goal.

In a technical sense, a landing page is more simple to deal with. In a measurement sense you only need to improve one metric, the conversion. With a landing page, we don’t have to make copy and creative that works okay for all situations, we simply make copy and creative that is optimal for one situation.

This makes testing really fun and easy, since you can test and find out who your audience really is and what causes them to convert.

Now that I’ve answered this question, in the future I’ll move onto a more interesting post: Why you should optimize everything else too.

Case Study: Smartsheet.com

I try to keep this blog non-company specific but my boss Frans Keylard, Director of Optimization, helped write up an informative case study of a client he worked on, Smartsheet.com. The case study is 4 pages long, covering the process and analysis of a multivariate test we did on one of Smartsheet.com’s landing pages. You can download it from Widemile.com, it requires a name and an e-mail to get it (sorry!)

Here’s my favorite image from the whole thing…

5 tips to maximize your holiday campaign

Ready for the holiday season? My boss, Frans Keylard (Director of Optimization), wrote up a 3 pager on “5 short tips to maximize your holiday campaign” (download PDF). I summarized it in my own words below, but check the PDF for more detail on how to get your conversions up during the holidays or any season!

Monthly sales increase as much as 20% for many online retailers in December and so having a good site now is more important than any other time. Optimization can drive up your sales beyond the seasonal increase in the short term and, in the long term, helps to add even more lifetime customers to your business. With that in mind, here’s 5 short tips to ramp up your site.

1. Seasons Matter: Change your site to match the season, we call it “seasonal-tuning.” Don’t let your site look stale; you don’t want Halloween colors and images for Christmas time. Matching what your customers are actively seeking will really drive your campaigns to success. This may not always work though, so make sure to test what has worked previously against new seasonal images. Lastly, even seasonal images need testing, so try different messaging in each, e.g. product shots, giving the product to another, and people using the product.
2. Know your existing customers: Reach out to existing customers, they are the most qualified audience and you should know them better than anyone else. Create offers that have a lot of perceived value to them. Also, use offers for people who don’t follow through the whole purchase process and try to pull them back in. They are interested but just need a reason to bite. Remember to know both what customers buy and why too. If you’re best selling chairs are shown in an office setting, when your customers are really home business users, then you might lose conversions. Use a home office image instead and get more relevancy to your items.
3. Find a new audience: Seasonality doesn’t just go for your web pages, apply them to your PPC text and banners ads and you’ll get new visitors quickly. Think holidays, turkeys, gift-giving, family, anything that resonates with your audience and your products. Include some seasonal offers for good measure too. You can do SEO around these words too, but that takes future planning. A good way to setup seasonal SEO for next year is by looking at what PPC and SEO worked this year, so make sure you don’t lose that valuable information.
4. Offers over branding: Don’t get too caught up in branding, just sell the product and offer. People are under pressure to buy nowand if your focus isn’t on helping them to do that, they’ll find someone else. You can build your brand later and have a reason to if they bought from you!
5. Build seasonal landing pages and optimize them with multivariate testing: Building a seasonal landing page is easy to do and optimizing that page is a must during seasonal periods. We optimized an online recipe search over the summer and found the best summer imagery, but once winter came around we changed the type of food and got a 30% sustained lift over the best performing summer image.

Stores change their decorations every season, so make sure to do that with your online store front. Give people a reason to stay at your site, even if you think it is small, it could have a huge impact. Finally, never forget to test and tune your changes every season, every year. Good luck!

Test to success

Jupiter Research released a new report, Compelling Benefits of Multivariate Web Site Testing But Adoption Remains Low. I guess the title says it all but here are some charts I put together to summarize:

Note that this includes A/B testing, meaning only a fraction of marketers are doing multivariate testing.

 

 

The large majority of early adopters have received gains from testing, but only 68% of testing users have raised conversions. That number should be much higher!

But I guess if it was, I wouldn’t have as much to write about.

Why test?

I love web designers and creative people. They create beautiful things out of words and ideas we give them, yet still have to deal with people judging every little thing they put out into the world. Working on company websites is especially an ordeal. Everyone has an opinion, from CEO to marketer, and too often there is a struggle of who to listen to, but really who knows what will work?

When creative runs up against marketing, a gray area falls on who knows best. Do you go by experience, seniority or even… degree? In the worst case, the CEO is calling all the shots and everyone else is just nodding in agreement or can’t get the CEO to listen to their advice.

So why test? At its core, testing is about finding out what works best, without any of the politics. If I put up two versions of a page and one gets more sales than another, its hard to argue with that. My job ends up proving people right or wrong, judged by what real visitors are telling us.

Test to:

• Know the impact of redesigns (quantitative)
• Figure out what your customers are and are not looking for (qualitative)
• Find the truth (awesome!)
  

As a marketer, our goals are to speak to our audience and get them interested in our business. With testing, marketers have a way to directly listen to what visitors want, to find out their language. So start testing to start listening.

Make sure you come back for more, I will be writing more on reading messaging through testing. This is just the beginning!