What is Taguchi? How does it relate to testing?

14 02 2008
New URL testingblog.widemile.com
the Taguchi method

Multivariate testing is a buzz word these days, but the buzzword of multivariate testing seems to be Taguchi. However, that term is being abused. Do you know what Taguchi really means? I wasn’t even positive, so to get some background, I did some research and talked with Vladimir (Widemile’s Chief Scientist).

The name and method comes from Genichi Taguchi. His method, also known as Robust Design, attempted to improve product manufacturing quality. Therefore it falls into an area of engineering called Quality Engineering.

Does this sound aligned with website testing? Not really, and this is the problem of using the term Taguchi with web site testing. The goals of manufacturing and the goals of a website are not the same.

What most people are attempting to grasp when using the term Taguchi is fractional factorial test design. (I discussed this at length in my post about the difference between Widemile’s technology and Google Optimizer.) The Taguchi method uses a fractional factorial test design and is under the umbrella of fractional factorial testing but is not the only or best fractional factorial method. In fact, even within manufacturing, the Taguchi method was the inspiration for many new techniques but many statisticians find it flawed.*

It is important to differentiate the Taguchi method from fractional factorial test design since one is a basis for manufacturing while the other is purely related to design of experiments. You need to ensure that the math and science behind your testing is based on methods that have the end goal of optimizing your website only. So if your testing tool uses the Taguchi method for testing, you better ask what that really means.

So does Widemile use Taguchi? We don’t use the Taguchi method, however do use fractional factorial test design. I like to say that our platform goes beyond Taguchi because it was specifically made for optimizing web content.

Don’t get sucked into the Taguchi method, it is just a buzzword used by your fellow marketers. Just because the technology doesn’t use Taguchi, doesn’t mean you should count it out.

*Read more after the jump for Vladimir’s explanation of the Taguchi method and its criticisms
Read the rest of this entry »


Google Optimizer is slow (or Not all Multivariate Testing is the same)

28 01 2008

New URL testingblog.widemile.com

Without knowing it, people might assume that there’s only one method to multivariate testing. That it has been long figured out by math and statistic wizards. I have learned otherwise from Widemile’s personal math wizard, Chief Scientist, Vladimir Brayman.

(Just as a side note, he does not have a typical office. Rather than papers and folders strewn about, he has statistic and math books. Lucky for me though, he has a great skill at distilling all the goodness in those books and teaching me what I need to know, in a way I understand.)

Most recently, we discussed why Widemile’s technology trumps Google Optimizer.

Widemile vs Google


Having a strong creative team and testing experts ensures better results than giving a marketer a tool like Google Optimizer, that’s easy for most people to understand. But explaining how Widemile’s technology can test more, faster, is a little more complicated.

Let’s explore how Google’s testing works versus Widemile’s. Google Optimizer uses full factorial test design, meaning it creates a page for every combination of your tested page elements. So if you wanted to test 4 different hero shots, 4 buttons and 4 headlines, that would require 4*4*4=64 page combinations. The disadvantage of this method is that you need significant traffic for each of the 64 pages. Meaning you either need a lot of traffic or a lot of time; for most companies, they’ll need both.

To solve this, Widemile’s optimization platform use fractional factorial test design. This method tests only a small fraction of the total possible page combinations and uses statistical analysis to derive almost all of the same information that would be found in a full factorial test. This works because marginal information is gained in testing all 64 page combinations, while testing a few important combinations tell us nearly everything we need to know.

Google actually criticizes fractional factorial test design (look here where it says “A note about ‘fractional factorial testing'”), saying that it requires the same number of impressions, but can not derive the depth of conclusions that a full factorial design can. While true that full factorial squeezes out the most information, that is at a sacrifice of extending the test many times longer than with a fractional factorial test, all to learn the smallest influences.

Doing successive tests to find high influence items with fractional factorial testing will get much higher gains than getting every ounce of information out of one extremely long full factorial test. In addition, with a carefully designed fractional factorial test you can learn all the major influences and the interactions between elements on the page.

Fractional factorial test design gets you a completed test in weeks rather than months or years even, and because of that, you can test more than you would normally be able to in the same time frame. You can either test more in one larger test, or do many smaller successive tests.

Not to say that Google Optimizer isn’t a great tool, especially since it is free, but any company that spends thousands of dollars on SEM has a lot to gain by using technology that gets rapid results.

If you got any questions about this, let me know and I’ll try to answer them or get you an answer.

Multivariate testing: a quick primer

26 12 2007

New URL testingblog.widemile.com

Want to quickly get up to speed on multivariate testing? This post is designed to help you grasp the basics of multivariate testing so that you can get started talking and even doing your own tests. While its hard to design great tests, it’s easy to get good results with only a little education. I will definitely be teaching more about multivariate testing soon, but get this stuff down first!

Note: Since the industry is new, there isn’t consistency in much of the vocabulary of multivariate testing, so I will try to use generic terms.

Billy’s Multivariate Testing Primer

What is Multivariate Testing?: Testing multiple versions of a page to determine a set of elements* providing the highest conversion rate.

*Elements can be any content (text or image) on a page, typically hero shots, buttons, button text, headlines and text blocks. Sometimes it can refer to position/layout also.

What happens:

Multivariate testing diagram

  • Visitors come to a page and are shown a random version of the page
  • Conversions are tracked based on which page they saw
  • Once a statistically significant number of conversions is reached, analysis determines the elements on a page that create the highest conversion rate


  • If used correctly, it hones in on correct messaging direction and then exact messaging can be found using further testing
  • Allows for quicker testing of multiple elements than split or A/B testing
  • Analysis derived from live visitor data
    • Proves winning page elements to be better than others


  • Medium learning curve
    • Test results easily ruined by mistakes or poor methodology
  • Requires a minimum number of conversions
  • Code must be added to web page
  • Cookies and JavaScript must be enabled by visitors

Keep in mind:

  • Increasing the number of elements being tested increases the time needed for the test.
  • Number of conversions over time determines test size and length
    • The faster a page gets conversions, the shorter the test will be and/or the more things you can test
    • More conversions is better
    • A shorter time frame is better than a long time frame, but don’t go shorter than 2 weeks
  • Don’t test things that are too similar, look for different segments or messaging to pursue

Other concepts:

  • Using it with split testing:
    • Use split tests to determine the best layout with a template test. Test layout against layout with the same content (see this article for more advice.)
    • Afterwards, use multivariate testing to try out different messaging and refine it with continual tests.
    • If you want to try new layouts later on, go back to split testing.
  • Advanced: There are two types of multivariate tests: Full factorial and fractional/partial factorial
    • Full factorial means every version is shown. Meaning if you have 4 headlines and 4 buttons, that is 4×4=16 combinations, so 16 pages are used in the test.
    • Partial factorial is when only a portion of the total possible combinations are shown to visitors. This relies on statistical formulas and algorithms to determine the influence of the various factors since not every page combination is shown.
    • Full factorial takes a much longer time, so partial factorial allows for more testing in a quicker time frame.

There’s still a lot more to teach, even about some of the things mentioned here, but I hope this is enough to get you off the ground and really digging into testing. Google has their free tool if you want to try out multivariate testing.

5 quick tips to effective A/B and split testing

5 12 2007

New URL testingblog.widemile.com

While multivariate might be the hottest testing subject, you can’t beat a good split test in certain situations.

If you already know the difference between A/B and split tests, skip this part, otherwise it’s a quick read.

A/B testing is when you test one page then replace it with a new page, so the two versions are running concurrently, one after another. Split testing is when you test two pages at once, where some distribution of traffic is sent to either of the pages simultaneously.

A/B Test

A/B Test


Split Test (note you can split different %’s)

Split test

Now for the good stuff! I’ll try to keep this short so let’s start:

  1. A/B testing is out: Use split testing instead. Split testing is more accurate since it uses the same time period of traffic. Traffic during Halloween is different from traffic during Christmas, so testing one page at one time and one at another will skew your results making a relevant comparison impossible. Use A/B only if you don’t have permission/the capability to do split testing (Google Optimizer is free and allows split testing!)
  2. Template test = Split test: This is the sweet spot for split tests. Use one if you want to try a new layout/template against your old one or if you want to test two new pages. Here’s my template test primer if you need to brush up.
  3. One exception, one lesson: During these tests everything should be the same except for one thing on the page. If you try introduce 2 or more changes into a split or a/b test you won’t know which change improved your page. The only time I might have multiple changes is for template tests, where the new template can’t use the previous creative effectively. Still, emulate the creative as closely as possible for the new template.
  4. Be ready for your next test: Since these tests are easier to execute, you should also have an easier time getting the next test ready to go for when the first one finishes. Make tests ahead of time so that when the current test completes you can flip the switch and quickly get it measured and done with.
  5. Learn from the first test: You already completed a test, what does that tell you about what you should test for the next time? If a graphic heavy template beat the cleaner template, try testing against an even more graphics heavy template. Find where your customers lie and pinpoint it by seeing what each test tells you. This is a game of Marco Polo. You customers are shouting, “Polo!” with each test, follow them!

Split tests keep it simple and that is its strength. As long as you control anything that might confuse the test (like introducing new content), you can find winners and make a great page. However, after split testing, multivariate testing should be brought in to really pull out more from your page. But that’s a whole other blog….

Two types of tests (and I’m not talking MV or Split)

28 11 2007

New URL testingblog.widemile.com

There are a lot of things to know to be a pro at testing, so I’ll cover some of the basics every week. Hopefully I’ll have a nice inventory of posts that can be used as a resource for you or anyone else to quickly learn the testing language and methods.

A good starting point to learning about testing methodology is to wrap your head around the idea of template and in-place tests.

A template test is a test using a new template in competition with the original template and is best done as a split test. A template test basically means you have all the same content but in different positions on the page. This is an example template:

Template 1

If this is my original template and I am doing a template test, I would switch the items around, but have the same exact content (hero shot, headline, price, etc.) to make a new template like this:

Template 2

These two templates would compete in a split test. A note though, even if I only move one thing, say I bring the Call-to-Action Button above the First Party Validation, it is still considered a template test.

However, if all I am doing is swapping out the content in those positions then it is an in-place test. Usually a multivariate test is an in-place test, trying out new headlines, images and text within the positions of the original content.

Typically you want to do a template test first to find the best positioning and then an in-place test to find the best messaging.

Continue on for more in depth examples
Read the rest of this entry »