Multivariate testing is a buzz word these days, but the buzzword of multivariate testing seems to be Taguchi. However, that term is being abused. Do you know what Taguchi really means? I wasn’t even positive, so to get some background, I did some research and talked with Vladimir (Widemile’s Chief Scientist).
The name and method comes from Genichi Taguchi. His method, also known as Robust Design, attempted to improve product manufacturing quality. Therefore it falls into an area of engineering called Quality Engineering.
Does this sound aligned with website testing? Not really, and this is the problem of using the term Taguchi with web site testing. The goals of manufacturing and the goals of a website are not the same.
What most people are attempting to grasp when using the term Taguchi is fractional factorial test design. (I discussed this at length in my post about the difference between Widemile’s technology and Google Optimizer.) The Taguchi method uses a fractional factorial test design and is under the umbrella of fractional factorial testing but is not the only or best fractional factorial method. In fact, even within manufacturing, the Taguchi method was the inspiration for many new techniques but many statisticians find it flawed.*
It is important to differentiate the Taguchi method from fractional factorial test design since one is a basis for manufacturing while the other is purely related to design of experiments. You need to ensure that the math and science behind your testing is based on methods that have the end goal of optimizing your website only. So if your testing tool uses the Taguchi method for testing, you better ask what that really means.
So does Widemile use Taguchi? We don’t use the Taguchi method, however do use fractional factorial test design. I like to say that our platform goes beyond Taguchi because it was specifically made for optimizing web content.
Don’t get sucked into the Taguchi method, it is just a buzzword used by your fellow marketers. Just because the technology doesn’t use Taguchi, doesn’t mean you should count it out.
*Read more after the jump for Vladimir’s explanation of the Taguchi method and its criticisms
The following is written by Vladimir Brayman, Chief Scientist at Widemile. If you have any questions for him or I, leave a comment and I will try to get back to you ASAP.
Sometimes the term Taguchi method is used mistakenly to mean fractional factorial design. In fact, the Taguchi method is much narrower in both its scope and objectives. The Taguchi method (also known as robust design) belongs to an engineering discipline called Quality Engineering. The main objective of the Quality Engineering design is to minimize variability in the performance of a product under different environmental conditions. The main characteristics of the Taguchi method stem from that objective. Among the steps involved in the Taguchi method are:
- Defining two types of factors, control and noise. Control factors can be manipulated by a production team during the manufacturing process whereas noise factors model environmental impacts on the product and thus cannot be controlled precisely.
- Defining two orthogonal arrays – usually with mixed levels and of strength 2 (this implies that only main effects can be detected) – one array for the control factors and the other for the noise factors.
- Maximizing the signal-to-noise ratios, a logarithmic function of the ratio between the square of the average responses due to the control factors and the estimate of the variance due to the noise factors.
Statisticians criticized unjustified claims of almost limitless applicability of the Taguchi method by some of the researchers. Among the critiques are:
- There is no possibility of detecting interactions among the control factors.
- There are N1*N2 observations needed, where N1 is the number of level combinations of the control array and N2 is that of the noise array. However the confounding structure for the control factors is the same as that of an array of size N1. This implies that the same resolution can be obtained with much smaller number of runs.
- The influence of the noise factors on the response variables cannot be detected.
To conclude, some people mistakenly call fractional factorial design Taguchi method. Use of the genuine Taguchi method for Landing Page Optimization is not justified.