Without knowing it, people might assume that there’s only one method to multivariate testing. That it has been long figured out by math and statistic wizards. I have learned otherwise from Widemile’s personal math wizard, Chief Scientist, Vladimir Brayman.
(Just as a side note, he does not have a typical office. Rather than papers and folders strewn about, he has statistic and math books. Lucky for me though, he has a great skill at distilling all the goodness in those books and teaching me what I need to know, in a way I understand.)
Most recently, we discussed why Widemile’s technology trumps Google Optimizer.
Having a strong creative team and testing experts ensures better results than giving a marketer a tool like Google Optimizer, that’s easy for most people to understand. But explaining how Widemile’s technology can test more, faster, is a little more complicated.
Let’s explore how Google’s testing works versus Widemile’s. Google Optimizer uses full factorial test design, meaning it creates a page for every combination of your tested page elements. So if you wanted to test 4 different hero shots, 4 buttons and 4 headlines, that would require 4*4*4=64 page combinations. The disadvantage of this method is that you need significant traffic for each of the 64 pages. Meaning you either need a lot of traffic or a lot of time; for most companies, they’ll need both.
To solve this, Widemile’s optimization platform use fractional factorial test design. This method tests only a small fraction of the total possible page combinations and uses statistical analysis to derive almost all of the same information that would be found in a full factorial test. This works because marginal information is gained in testing all 64 page combinations, while testing a few important combinations tell us nearly everything we need to know.
Google actually criticizes fractional factorial test design (look here where it says “A note about ‘fractional factorial testing’”), saying that it requires the same number of impressions, but can not derive the depth of conclusions that a full factorial design can. While true that full factorial squeezes out the most information, that is at a sacrifice of extending the test many times longer than with a fractional factorial test, all to learn the smallest influences.
Doing successive tests to find high influence items with fractional factorial testing will get much higher gains than getting every ounce of information out of one extremely long full factorial test. In addition, with a carefully designed fractional factorial test you can learn all the major influences and the interactions between elements on the page.
Fractional factorial test design gets you a completed test in weeks rather than months or years even, and because of that, you can test more than you would normally be able to in the same time frame. You can either test more in one larger test, or do many smaller successive tests.
Not to say that Google Optimizer isn’t a great tool, especially since it is free, but any company that spends thousands of dollars on SEM has a lot to gain by using technology that gets rapid results.
If you got any questions about this, let me know and I’ll try to answer them or get you an answer.
The statement that you make about Google not being capable of Taguchi or ‘Fractional Factorial Design’ is, in my opinion, inaccurate, or, at best, not complete.
I’ve built and performed multiple sampled factorial tests and have instructed others on how to do so. The speed to results is hastened by the drastic reduction in variations which you described. Its a matter of how you structure the test, produce viable and sound Taguchi variations (as there is only one ‘calculator’ for this purpose), and the determination to correctly implement those variations.
Another note with regard to this is the proximity to conversion goal. If test completion timeframe is a concern, then, it might be advisable to develop formulae for proximation. I can illustrate some examples for this if you would like, please email me and we can discuss.
Thanks for the comment. This is a fascinating subject and so I’d love to know more. I will get in touch with you shortly, Daniel.
Just from the tiny bit of information of your post, it seems like what you are doing is not the intended method by Google and so requires more manual work and analysis on your side, while it is all automated since it is the implied methodology in Widemile’s platform. I don’t want to say more until I know more about how you are testing though.
It excites me to see that people are skilled enough to truly use Google Optimizer beyond its out of the box capabilities!
Hopefully we can discuss and I can possibly post an updated version of the blog.
Talk to you soon,
Billy
[...] the term Taguchi is fractional factorial test design. (I discussed this at length in my post about the difference between Widemile’s technology and Google Optimizer.) The Taguchi method uses a fractional factorial test design and is under the umbrella of [...]
[...] Test the maximum minimum: Pack as much as you can into a test for the amount of traffic you have to keep it a short test. Using Widemile’s platform that’s 2 weeks to be safe, with Google Optimizer you should do at least a month (explanation). [...]
[...] Any online marketer running PPC campaigns should be actively running a several multivariate tests. This post from Billy’s Blog explores the use of Google Web Optimizer- Google Optimizer is slow (or Not all Multivariate Testing is the same) [...]
Hmm it looks like your blog ate my first comment (it was super long) so I guess I’ll just sum it up what I wrote and say, I’m
thoroughly enjoying your blog. I as well am an aspiring blog blogger but I’m still new to everything. Do you have any suggestions for beginner blog writers? I’d
certainly appreciate it. Cheers!