Multivariate testing is a buzz word these days, but the buzzword of multivariate testing seems to be Taguchi. However, that term is being abused. Do you know what Taguchi really means? I wasn’t even positive, so to get some background, I did some research and talked with Vladimir (Widemile’s Chief Scientist).

The name and method comes from Genichi Taguchi. His method, also known as Robust Design, attempted to improve product manufacturing quality. Therefore it falls into an area of engineering called Quality Engineering.

Does this sound aligned with website testing? Not really, and this is the problem of using the term Taguchi with web site testing. The goals of manufacturing and the goals of a website are not the same.

What most people are attempting to grasp when using the term Taguchi is *fractional factorial test design*. (I discussed this at length in my post about the difference between Widemile’s technology and Google Optimizer.) The Taguchi method uses a fractional factorial test design and is under the umbrella of fractional factorial testing but is not the only or best fractional factorial method. In fact, even within manufacturing, the Taguchi method was the inspiration for many new techniques but many statisticians find it flawed.*

It is important to differentiate the Taguchi method from fractional factorial test design since one is a basis for manufacturing while the other is purely related to design of experiments. You need to ensure that the math and science behind your testing is based on methods that have the end goal of optimizing your website only. So if your testing tool uses the Taguchi method for testing, you better ask what that really means.

So does Widemile use Taguchi? We don’t use the Taguchi method, however do use fractional factorial test design. I like to say that our platform goes beyond Taguchi because it was specifically made for optimizing web content.

Don’t get sucked into the Taguchi method, it is just a buzzword used by your fellow marketers. Just because the technology doesn’t use Taguchi, doesn’t mean you should count it out.

*Read more after the jump for Vladimir’s explanation of the Taguchi method and its criticisms

The following is written by Vladimir Brayman, Chief Scientist at Widemile. If you have any questions for him or I, leave a comment and I will try to get back to you ASAP.

Sometimes the term Taguchi method is used mistakenly to mean fractional factorial design. In fact, the Taguchi method is much narrower in both its scope and objectives. The Taguchi method (also known as robust design) belongs to an engineering discipline called Quality Engineering. The main objective of the Quality Engineering design is to minimize variability in the performance of a product under different environmental conditions. The main characteristics of the Taguchi method stem from that objective. Among the steps involved in the Taguchi method are:

- Defining two types of factors, control and noise. Control factors can be manipulated by a production team during the manufacturing process whereas noise factors model environmental impacts on the product and thus cannot be controlled precisely.
- Defining two orthogonal arrays – usually with mixed levels and of strength 2 (this implies that only main effects can be detected) – one array for the control factors and the other for the noise factors.
- Maximizing the signal-to-noise ratios, a logarithmic function of the ratio between the square of the average responses due to the control factors and the estimate of the variance due to the noise factors.

Statisticians criticized unjustified claims of almost limitless applicability of the Taguchi method by some of the researchers. Among the critiques are:

- There is no possibility of detecting interactions among the control factors.
- There are N1*N2 observations needed, where N1 is the number of level combinations of the control array and N2 is that of the noise array. However the confounding structure for the control factors is the same as that of an array of size N1. This implies that the same resolution can be obtained with much smaller number of runs.
- The influence of the noise factors on the response variables cannot be detected.

To conclude, some people mistakenly call fractional factorial design Taguchi method. Use of the genuine Taguchi method for Landing Page Optimization is not justified.

Terry Polyak(09:36:08) :Nice article. I always wondered how auto manufacturing QC and website Optimization through MVT were related……only that it was a process to arrive to an end with mathematics helping you get to a result quicker. Zoom zoom.

Dale Jelinek(12:37:59) :I am reading Ranjit Roy’s “Primer on the Taguchi Method” and I sense that his understanding differs from that of Vladimir in a few areas. Being a non-statistician and neophyte in this area I don’t claim to know the answers, but here is where I see the differences:

– “There is no possibility of detecting interactions among the control factors”

+ In Roy’s book he covers this topic in many sections, starting in Section 5-5-2 “Interaction Affects”. He also publishes Taguchi’s triangular tables and linear graphs which provide information on how to isolate interaction effects of the control factors in the orthogonal arrays.

– “However the confounding structure for the control factors is the same as that of an array of size N1. This implies that the same resolution can be obtained with much smaller number of runs. ”

+ The full factorial array is N1 * N2 in size, but isn’t that the point of the research done by Taguchi – that by using orthogonal fractional factorial arrays that you can use a fraction of the number of experimental runs to capture the essential information to determine main effects and interaction effects?

I have heard the comment in the past that the Taguchi approach does work in marketing, but there are “adjustments” that need to be made to make it useful in that context. Have you any idea what those “adjustments” might be?

Thank you in advance for your expanded comments in reply, I am looking forward to advancing my understanding of optimization techniques applied to marketing.

Billy(14:43:32) :Hi Dale,

It’s difficult to understand what you’re saying without having read Roy’s book. We are not saying that the Taguchi method has no standing or merit, just that by itself it is not the best method to decipher marketing data. The goals of the Taguchi method is far from what we, as marketers, are trying to achieve.

What you described as “adjustments” is close to what I am saying. The Taguchi method is a part of fractional factorial design, but is not all encompassing. Marketers that say they are using “the Taguchi method” probably are just doing fractional factorial design and incorrectly attributing it to the Taguchi method.

I just want to clarify that the true Taguchi method is not made for marketing and should not be sought as the answer to answer multivariate tests.

Juan Rivas(11:51:21) :This comments are erroneous in part, because Taguchi Methods are based on Plackett Burmann design, and Taguchi took this option for many interesting reasons.

The author of this article is an ignorant, because Taguchi Method is a philosophy , a little bit of Design of Experiments, in fact you can analyze a Design of Experiment from fractional factorial design classical through S/N ratios….in question to multivariate analysis you can work with MTGS also Taguchi Method.