Comments : 1 Comment »
Tags: a/b testing, always be testing, multivariate testing
Categories : Methodology, Why Test?
I stumbled upon a great post from Andy Edmonds at Always Be Testing about ways testing can be used. Here’s the list he came up with:
Business Acceptance Testing
Example: You want to add yet another shortcut on the homepage for a new sub-audience. Use testing to validate you didn’t mess up the other functions of the homepage.
That new search function is going to cost you X thousands of dollars. How long will it take to provide a positive return on investment.
Design Choice Determination
The boss thinks the logo should be purple. You don’t.
Multivariate methods are really valuable for this use case. Say you wanted to ask the question, “Is copy at the top of our product pages worthwhile? Or should we just drop it and get more products above the fold?” A multivariate test can let you vary the size & quality of the copy, along with other elements that push the product down the page, and assess the general impact of products higher on the page as well as the general impact of good copy.
Off the top of my head, I would change the definition for Customer Understanding though. You can learn more than something as specific as, “Should I put copy here or there or remove it?” Test different messaging by testing fear based versus lifestyle based content. It allows you to find out which messages really hit at home with your readers. Not only will that help you improve that page, but it will teach you why your users want your product or service in general.
In short, testing answers “Does this work or not?” Give the customer choices and let them do the hard work deciding what works best. Makes being a marketer easier doesn’t it?
Photo credit: e-magic via Flickr
Comments : 4 Comments »
Tags: a/b testing, split testing, tips
Categories : Methodology, Terminology
While multivariate might be the hottest testing subject, you can’t beat a good split test in certain situations.
If you already know the difference between A/B and split tests, skip this part, otherwise it’s a quick read.
A/B testing is when you test one page then replace it with a new page, so the two versions are running concurrently, one after another. Split testing is when you test two pages at once, where some distribution of traffic is sent to either of the pages simultaneously.
Split Test (note you can split different %’s)
Now for the good stuff! I’ll try to keep this short so let’s start:
- A/B testing is out: Use split testing instead. Split testing is more accurate since it uses the same time period of traffic. Traffic during Halloween is different from traffic during Christmas, so testing one page at one time and one at another will skew your results making a relevant comparison impossible. Use A/B only if you don’t have permission/the capability to do split testing (Google Optimizer is free and allows split testing!)
- Template test = Split test: This is the sweet spot for split tests. Use one if you want to try a new layout/template against your old one or if you want to test two new pages. Here’s my template test primer if you need to brush up.
- One exception, one lesson: During these tests everything should be the same except for one thing on the page. If you try introduce 2 or more changes into a split or a/b test you won’t know which change improved your page. The only time I might have multiple changes is for template tests, where the new template can’t use the previous creative effectively. Still, emulate the creative as closely as possible for the new template.
- Be ready for your next test: Since these tests are easier to execute, you should also have an easier time getting the next test ready to go for when the first one finishes. Make tests ahead of time so that when the current test completes you can flip the switch and quickly get it measured and done with.
- Learn from the first test: You already completed a test, what does that tell you about what you should test for the next time? If a graphic heavy template beat the cleaner template, try testing against an even more graphics heavy template. Find where your customers lie and pinpoint it by seeing what each test tells you. This is a game of Marco Polo. You customers are shouting, “Polo!” with each test, follow them!
Split tests keep it simple and that is its strength. As long as you control anything that might confuse the test (like introducing new content), you can find winners and make a great page. However, after split testing, multivariate testing should be brought in to really pull out more from your page. But that’s a whole other blog….
Comments : Leave a Comment »
Tags: a/b testing, Jupiter Research, multivariate testing, stats
Categories : Industry News
Jupiter Research released a new report, Compelling Benefits of Multivariate Web Site Testing But Adoption Remains Low. I guess the title says it all but here are some charts I put together to summarize:
Note that this includes A/B testing, meaning only a fraction of marketers are doing multivariate testing.
The large majority of early adopters have received gains from testing, but only 68% of testing users have raised conversions. That number should be much higher!
But I guess if it was, I wouldn’t have as much to write about.
Comments : Leave a Comment »
Tags: a/b testing, marketing, multivariate testing, politics, reasons to test
Categories : About, Methodology, Why Test?
I love web designers and creative people. They create beautiful things out of words and ideas we give them, yet still have to deal with people judging every little thing they put out into the world. Working on company websites is especially an ordeal. Everyone has an opinion, from CEO to marketer, and too often there is a struggle of who to listen to, but really who knows what will work?
When creative runs up against marketing, a gray area falls on who knows best. Do you go by experience, seniority or even… degree? In the worst case, the CEO is calling all the shots and everyone else is just nodding in agreement or can’t get the CEO to listen to their advice.
So why test? At its core, testing is about finding out what works best, without any of the politics. If I put up two versions of a page and one gets more sales than another, its hard to argue with that. My job ends up proving people right or wrong, judged by what real visitors are telling us.
- Know the impact of redesigns (quantitative)
- Figure out what your customers are and are not looking for (qualitative)
- Find the truth (awesome!)
As a marketer, our goals are to speak to our audience and get them interested in our business. With testing, marketers have a way to directly listen to what visitors want, to find out their language. So start testing to start listening.
Make sure you come back for more, I will be writing more on reading messaging through testing. This is just the beginning!