Google Optimizer is slow (or Not all Multivariate Testing is the same)

28 01 2008


Without knowing it, people might assume that there’s only one method to multivariate testing. That it has been long figured out by math and statistic wizards. I have learned otherwise from Widemile’s personal math wizard, Chief Scientist, Vladimir Brayman.

(Just as a side note, he does not have a typical office. Rather than papers and folders strewn about, he has statistic and math books. Lucky for me though, he has a great skill at distilling all the goodness in those books and teaching me what I need to know, in a way I understand.)

Most recently, we discussed why Widemile’s technology trumps Google Optimizer.

Widemile vs Google


Having a strong creative team and testing experts ensures better results than giving a marketer a tool like Google Optimizer, that’s easy for most people to understand. But explaining how Widemile’s technology can test more, faster, is a little more complicated.

Let’s explore how Google’s testing works versus Widemile’s. Google Optimizer uses full factorial test design, meaning it creates a page for every combination of your tested page elements. So if you wanted to test 4 different hero shots, 4 buttons and 4 headlines, that would require 4*4*4=64 page combinations. The disadvantage of this method is that you need significant traffic for each of the 64 pages. Meaning you either need a lot of traffic or a lot of time; for most companies, they’ll need both.

To solve this, Widemile’s optimization platform use fractional factorial test design. This method tests only a small fraction of the total possible page combinations and uses statistical analysis to derive almost all of the same information that would be found in a full factorial test. This works because marginal information is gained in testing all 64 page combinations, while testing a few important combinations tell us nearly everything we need to know.

Google actually criticizes fractional factorial test design (look here where it says “A note about ‘fractional factorial testing'”), saying that it requires the same number of impressions, but can not derive the depth of conclusions that a full factorial design can. While true that full factorial squeezes out the most information, that is at a sacrifice of extending the test many times longer than with a fractional factorial test, all to learn the smallest influences.

Doing successive tests to find high influence items with fractional factorial testing will get much higher gains than getting every ounce of information out of one extremely long full factorial test. In addition, with a carefully designed fractional factorial test you can learn all the major influences and the interactions between elements on the page.

Fractional factorial test design gets you a completed test in weeks rather than months or years even, and because of that, you can test more than you would normally be able to in the same time frame. You can either test more in one larger test, or do many smaller successive tests.

Not to say that Google Optimizer isn’t a great tool, especially since it is free, but any company that spends thousands of dollars on SEM has a lot to gain by using technology that gets rapid results.

If you got any questions about this, let me know and I’ll try to answer them or get you an answer.




13 responses

4 02 2008
Daniel Shields

The statement that you make about Google not being capable of Taguchi or ‘Fractional Factorial Design’ is, in my opinion, inaccurate, or, at best, not complete.

I’ve built and performed multiple sampled factorial tests and have instructed others on how to do so. The speed to results is hastened by the drastic reduction in variations which you described. Its a matter of how you structure the test, produce viable and sound Taguchi variations (as there is only one ‘calculator’ for this purpose), and the determination to correctly implement those variations.

Another note with regard to this is the proximity to conversion goal. If test completion timeframe is a concern, then, it might be advisable to develop formulae for proximation. I can illustrate some examples for this if you would like, please email me and we can discuss.

5 02 2008

Thanks for the comment. This is a fascinating subject and so I’d love to know more. I will get in touch with you shortly, Daniel.

Just from the tiny bit of information of your post, it seems like what you are doing is not the intended method by Google and so requires more manual work and analysis on your side, while it is all automated since it is the implied methodology in Widemile’s platform. I don’t want to say more until I know more about how you are testing though.

It excites me to see that people are skilled enough to truly use Google Optimizer beyond its out of the box capabilities!

Hopefully we can discuss and I can possibly post an updated version of the blog.

Talk to you soon,

14 02 2008
What is Taguchi? How does it relate to testing? « Billy’s Blog

[…] the term Taguchi is fractional factorial test design. (I discussed this at length in my post about the difference between Widemile’s technology and Google Optimizer.) The Taguchi method uses a fractional factorial test design and is under the umbrella of […]

21 02 2008
3 steps to quickly make a good multivariate test « Billy’s Blog

[…] Test the maximum minimum: Pack as much as you can into a test for the amount of traffic you have to keep it a short test. Using Widemile’s platform that’s 2 weeks to be safe, with Google Optimizer you should do at least a month (explanation). […]

11 03 2008
Marketing Mix 3-10-08

[…] Any online marketer running PPC campaigns should be actively running a several multivariate tests. This post from Billy’s Blog explores the use of Google Web Optimizer- Google Optimizer is slow (or Not all Multivariate Testing is the same) […]

19 06 2013

Hmm it looks like your blog ate my first comment (it was super long) so I guess I’ll just sum it up what I wrote and say, I’m
thoroughly enjoying your blog. I as well am an aspiring blog blogger but I’m still new to everything. Do you have any suggestions for beginner blog writers? I’d
certainly appreciate it. Cheers!

10 07 2013

Hi, I do not even know how I ended up here, but I thought this blog was
good. I do not know who you are but certainly you’re great content management. Thanks.

10 07 2013
solar power iphone charger

Admiring the dedication you put into your website and detailed information you present.
It’s great to come across a blog every once in a while that isn’t
the same unwanted rehashed material. Wonderful read!
I’ve saved your site and I’m including your RSS feeds to my
Google account. Very much appreciated information!

15 07 2013
Solar Cell Phone Battery Charger

Being a Newbie, I am always searching online for articles that can assist me.
Thank you so much, a wonderful job!

2 08 2013
solar 12v battery charger

Woah! I’m really loving the template/theme of this website. It’s simple, yet effective.
A lot of times it’s challenging to get that “perfect balance” between user friendliness and visual appeal. I must say that you’ve done a
very good job with this. Additionally, the blog loads extremely quick for me on Firefox.

4 08 2014

This is certainly truly wonderful- I must acknowledge, your website could be the gravy for
you to my mashed potatoes! Thank you so much plus sustain the great job!

1 10 2014
SEO Services Philadelphia

Hello, for all time i used to check webpage posts here in the early
hours in the dawn, bexause i enjoy to find out more and more.

30 03 2016
seo in bognor regis

Hello, its pleasant post regarding media print, we
all be ffamiliar with media is a impressive source of facts.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: