Two types of tests (and I’m not talking MV or Split)

28 11 2007


There are a lot of things to know to be a pro at testing, so I’ll cover some of the basics every week. Hopefully I’ll have a nice inventory of posts that can be used as a resource for you or anyone else to quickly learn the testing language and methods.

A good starting point to learning about testing methodology is to wrap your head around the idea of template and in-place tests.

A template test is a test using a new template in competition with the original template and is best done as a split test. A template test basically means you have all the same content but in different positions on the page. This is an example template:

Template 1

If this is my original template and I am doing a template test, I would switch the items around, but have the same exact content (hero shot, headline, price, etc.) to make a new template like this:

Template 2

These two templates would compete in a split test. A note though, even if I only move one thing, say I bring the Call-to-Action Button above the First Party Validation, it is still considered a template test.

However, if all I am doing is swapping out the content in those positions then it is an in-place test. Usually a multivariate test is an in-place test, trying out new headlines, images and text within the positions of the original content.

Typically you want to do a template test first to find the best positioning and then an in-place test to find the best messaging.

Continue on for more in depth examples
For example, I took a look at Widemile’s own landing page and decided that I wanted to try out a new template. So here are the two landing pages. What kind of test is this?

Original (click for full size):

Widemile Landing Page 1

New (click for full size):

Widemile Landing Page 2

This is a little tricky. I would categorize this as a template test. Why? Because I have almost all of the original content, just in a new format. The new version utilizes a banner header template rather than a straight 2 column template. The trickiness is in the use of the new imagery. It is good practice to keep everything the content as similar as possible, but sometimes thats not possible either for client reasons or because the new template demands a new look. In these cases, just keep in mind that new content was used when looking at the test results.

Finally, template tests and in-place tests are normally completely separate, however there are situations where you will be doing both, such as trying out new headlines and hero shots while also moving the button position in a multivariate test. This is should usually be reserved only as a refinement step, after the page has been tested already. If you are going to make many or large changes, a full template test is the better way to go. This usually is not recommended, but you can move one item around during an in-place test without much harm.

This may seem like a small thing, but really this is an important distinction. Not only does it help you determine what type of test you want to do, it helps you remember to focus on not mixing them.

Did you find this confusing? Let me know in the comments!



2 responses

7 12 2007
5 quick tips to effective A/B and split testing « Billy’s Blog

[…] a new layout/template against your old one or if you want to test two new pages. Here’s my template test primer if you need to brush […]

19 03 2008
Multivariate testing: a quick primer « Billy’s Blog

[…] split tests to determine the best layout with a template test. Test layout against layout with the same content (see this article for more […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: