Multivariate testing: Let customer actions tell you what’s right

5 02 2008

This comic is both hilarious and insightful. Take a second and see if you’ve experienced something similar to this situation:


I came across this before starting my work at Widemile, but stumbled upon it again while browsing Marianina’s Web Analytics Princess blog. Surprisingly, I found it extremely relevant to the multivariate testing I do at Widemile now.

The comic is a joke about how hard it is to do what customers want. Even when you have the luxury of a customer telling you everything they want, they don’t always do it correctly and no two people interpret their instructions the same. I like this comic because it shows how difficult it is to create good experiences and products for customers, no matter the amount of help you have.

Now, imagine how this applies to your website. Visitors want your product presented to them in a certain way with certain offers. Before testing, you’d listen to visitors with analytics, surveys and usability studies even. From here everyone would make their own “panel” of the comic; the creative team would have a go, the marketing team would revise it and the CEO would change a few words around. Eventually a page is made and your business puts it online, not knowing whether the page is what your visitors truly want.

Take that same situation and add multivariate testing. Now we get the same perspective that we have when looking at the full comic strip. In the previous situation, all you could see was your individual panel and your customers could only choose the one you gave them. With a multivariate test, different customers see different versions and the trends that show up in your data, reveal the “whole comic” and point you to which one should be the last panel… what the customer really needs.

If you look at the first panel and the last panel, it’s immediately obvious that they are different. Your customers can tell what they like, even if they can’t articulate it. Start listening to their true feelings by testing your pages.

For more of these comics, check out the official Project Cartoon site.


Doubling conversion rates: MarketingSherpa case study

23 01 2008
MarketingSherpa Logo

MarketingSherpa is giving us a lot of love recently with The Weather Channel case study and now one on Smartsheet. If you were too shy to sign up for our case study on the Widemile website, then please check out the one at MarketingSherpa.

I especially like the last 4 points in the story:

#1. Don’t sit pat on your conversion rates. Colacurcio didn’t know that her original 5%-7% was above the industry average going in and is glad she didn’t. “If I would have said, ‘My conversion rate is pretty good’ and done nothing, I would have totally missed the opportunity to double it.”

#2. Conducting just a few multivariate test and applying them to a greater number of landing pages works. Simply put, you don’t have to test each landing page individually. “Sometimes it seems overwhelming when you think of multivariate testing, but you can cut corners. There are a lot of things that are low-hanging fruit — things that can be applied across landing pages.”

#3. Redesigning shouldn’t stop with your team’s new ideas. “You really have to get your organization into the mindset: ‘We are testing. We are not just going to spit out the next five Web pages.’ ”

#4. Even if your higher-ups are impressed with the initial results, Colacurcio says, marketers should expect to face organizational barriers when they start their second round of testing.

Don’t forget to check out the before and after in the creative samples.

If none of that catches your eye, Janet Meiners at MarketingPilgrim wrote a great summary on the case study.  She also makes a great point that, “I’ve been in heated debates about the best course of action but testing works best – assuming you’re humble enough to go with the data over your ego.

Trust your data and watch your numbers build your ego.

Increase SEM spend or optimize your page?

4 01 2008

SEM vs Multivariate Testing

Most decision makers start multivariate testing because it makes sense in terms of ROI. While I may talk about helping marketers make good decisions about their pages, my real job is to make my clients look good by getting them a result that looks good online (a quality web page) and on paper (quality returns.)

The value of multivariate testing and landing page optimization lies in the fact that SEM is rapidly becoming more expensive as competition rises.

How much would you have to spend in PPC to get a sustainable 30% raise in conversions? To be sustainable, you would have to spend that much money month after month.

  • Sustainable lifts even after optimization ends

With testing, the increased conversion rate will typically be sustainable over a significant period of time. Pay to optimize once and you’ll receive that raise in conversions for free for months and even possibly years after you finish your first test. Of course the best way to go is continual testing to squeeze even more out of your pages, but the results from any testing come pretty quickly and make good business sense.

  • More efficient SEM spend

A bonus is the natural cycle that occurs from optimizing. Once your page is optimized, your advertising spend will be more efficient. This allows you to either spend the excess money towards other campaigns or feed it back into the optimized campaign to further drive conversions.

  • Large increases in conversion rate

Lastly, while this only happens occasionally, conversion rates often shoot through the roof after testing. We have more than doubled conversion rates for some pages, which typically is not possible even with dumping money into advertising.

If you have a solid SEM campaign but aren’t seeing results you want or once had, or if you just want to stay ahead of the competition, multivariate and split testing are the necessary next steps.

I work in this field for two reasons: 1. I love marketing and testing, 2. I know that in a few years, every business will be testing.

SEM will be around for a long time and must be done right, but it is not getting cheaper anytime soon and most businesses already participate in it. Testing is still nascent and probably where SEM was 3-4 years ago. Don’t get left behind, the future is in testing.

4 reasons to test your web pages

2 01 2008
Why Test?

I stumbled upon a great post from Andy Edmonds at Always Be Testing about ways testing can be used. Here’s the list he came up with:

Business Acceptance Testing

Example: You want to add yet another shortcut on the homepage for a new sub-audience. Use testing to validate you didn’t mess up the other functions of the homepage.

Value Estimation

That new search function is going to cost you X thousands of dollars. How long will it take to provide a positive return on investment.

Design Choice Determination

The boss thinks the logo should be purple. You don’t.

Customer Understanding

Multivariate methods are really valuable for this use case. Say you wanted to ask the question, “Is copy at the top of our product pages worthwhile? Or should we just drop it and get more products above the fold?” A multivariate test can let you vary the size & quality of the copy, along with other elements that push the product down the page, and assess the general impact of products higher on the page as well as the general impact of good copy.

Off the top of my head, I would change the definition for Customer Understanding though. You can learn more than something as specific as, “Should I put copy here or there or remove it?” Test different messaging by testing fear based versus lifestyle based content. It allows you to find out which messages really hit at home with your readers. Not only will that help you improve that page, but it will teach you why your users want your product or service in general.

In short, testing answers “Does this work or not?” Give the customer choices and let them do the hard work deciding what works best. Makes being a marketer easier doesn’t it?

Photo credit: e-magic via Flickr

Every marketer’s New Year’s Resolution

27 12 2007

We marketers get to have a lot of fun. We have a variety of projects that we get to plan, develop and execute. Wrapping up every project is an accomplishment for us. Or is it?

New Years Fireworks

Unfortunately, many projects are started without tangible metrics to strive for other than finishing it on time and within the budget. One part of being a marketer, or any businessman or woman, is showing that what you do is valuable and has a solid number attached to it.

Because of that, every marketer should make a resolution to measure and test their campaigns rather than just finishing them.

Achievements like site redesigns and campaign launches should be celebrated, but they also should be effectively tracked and analyzed from beginning to end.

Jason Burby wrote a fantastic article at ClickZ about ringing in the new year by defining and executing based on goals for 2008. In it he describes how sometimes projects succeed in some measures but fail at others, so overall is it a success?

He even mentions multivariate testing as part of the solution: “One part of this new way of thinking is to ensure you have a simple, easy to use testing platform to try different things and to measure the impact of tests based on goals. If you haven’t invested in an A/B or multivariate tool, 2008 is the time.”

What I’ve found in doing optimization and working with marketers in web analytics is that defining success in every campaign is key to being a quality marketer. And the only way to know if you met those goals is to measure what you do and how much it drives those goals.

For web page testing, the goal typically is to raise conversion rates and so its easy to keep that in mind. However if I never changed conversion rate as my ultimate goal, I might miss out on other valuable goals. A great example is if there are two pages with different subscription levels for a product, then possibly the lower converting page has a higher lifetime ROI.

ROI Table

The great side benefit of having a goal and measuring what gets you there, is that you can learn from success and failure. Even when I test a page and get an improvement, I exam what factors were the worst, along with what did best. It tells me what scares away and attracts customers so that I can look out for those things in other parts of the page and overall funnel.

Never assume that changing or making a new version of anything is always better. Figure out what you want to improve, make the changes and see if you find that improvement. As long as you have a measurement for success and take the time to track your progress, you can learn from your campaign and in the end, get closer to reaching and beating your goals.

Why always optimize landing pages?

19 12 2007

Sometimes people ask me why landing pages are such popular targets for optimization and testing. Why not optimize a home page? A product page?

Actually, we can optimize those kinds of pages, but almost all businesses come to us with a landing page that needs help. Beyond the demand for landing page testing though, is the fact that landing pages inherently are fit for testing. Let me explain.

Widemile LPO Landing Page
One of Widemile’s landing pages

The strength of multivariate and split testing is in pulling out the best page possible for what you want to do. With a homepage you have multiple things you want to do (e.g. show your products, company history, customer service, get people to spend X minutes browsing) and with a product page you are typically working with a CMS template for your whole site. Those factors complicate things a bit since you have to figure out what counts as improvement for the homepage and for the product page, you have to work with a CMS system and make changes that improve the majority of products pages using that template.

These things are not impossible to do or even difficult in many situations, but a landing page is usually totally independent of everything and has only one goal.

In a technical sense, a landing page is more simple to deal with. In a measurement sense you only need to improve one metric, the conversion. With a landing page, we don’t have to make copy and creative that works okay for all situations, we simply make copy and creative that is optimal for one situation.

This makes testing really fun and easy, since you can test and find out who your audience really is and what causes them to convert.

Now that I’ve answered this question, in the future I’ll move onto a more interesting post: Why you should optimize everything else too.

What can your data really tell you?

6 12 2007

Online testing is a bit different from other marketing data. It uses live traffic to find out what works. Analytics is the same, measuring what’s occurring at the moment. So why is that important? Well you can infer all you want from surveys, usability studies and demographics, but in the end you can’t argue against what real users are doing.

Avinash Kaushik, a popular analytics blogger, summed up the juiciest bits of a presentation by Jim Novo at eMetrics. In it, Jim asked, “What data yields insights that can be actioned the most?” The answer:

Data pyramid

“[A]ctionability, relevance of insights that can be actioned decreased as you go down the slide”

He makes the point that the farther away you get from the top of the pyramid, the harder it is to accurately predict your users actions. Yet often times too much value is put into the bottom levels of the pyramid. Even when marketers do test and measure actual behavior, they go about it the wrong way because they stick to all this other data too much and end up testing things that are all alike, defeating the purpose of testing.

Think about it, can you really tell if a red button will work better than a blue button if all you have are demographics? There are places for all of these types of data, but there should not be a fear of actual behavioral data. Yes we are using live traffic, yes the data is driven by technology (online visitors, javascript, cookies, rather than people filling out a survey), but those numbers tell a story unlike any other data.

Make the most out of all types of your data, but don’t die by one or the other. Use what’s best for every situation, but realize that you will never know you are right until you test it out.