How to Get Real Value from Your A/B Test

Experienced marketers know offer and product testing should be conducted on a continual basis—what works today may or may not work tomorrow. The only real way to find out is to test alternative ideas and approaches.

The most basic of these testing scenarios is the tried-and-true A/B test. While multivariate testing has become a standard tool in the marketing toolbox, A/B testing remains one of the fastest and best ways to determine the success or failure of a new product or marketing approach.

A/B testing is a very old technique that originated long before the advent of the computer (to say nothing of Internet). Also referred to as split run testing, A/B testing has been used by marketers for years in direct mail and other direct response marketing efforts.

An A/B test is not difficult to conduct. In such tests, the “A” option is the control, or current champion. The “B” option is the challenger being tested in an attempt to provide better results than A. Visitors or respondents are randomly shown or offered the A or the B option. The difference between the two response rates is then evaluated for being statistically significant (or not).

Here are five things you should keep in mind to get the most value out of an A/B test:

1. Measure one thing at a time. Unintentional variables are easy to create. Even if you think you are only changing the offer, if you have altered the background color, the copy, or the graphics, you now have a multivariate test. Results are not as easily determined.

2. Make sure you are measuring the right data. If performing an e-commerce test, do you want to measure clicks? Path to transaction? Shopping cart abandonment? How you set up the test should be determined by what you want to know and how you are going to measure the data.

3. Be content with measuring only a few things well. Resist the temptation to interpret too many data sets. If you are testing one offer versus another, don’t try to interpret things that are less significant, such as page engagement time—the old-fashioned how many orders or requests for information are a better measure.

4. Allow the test to be conducted over a period long enough to get a true read. This will be different for a postal mail test versus an e-mail test, or even a broadcast test.

5. When you determine a test winner, go back to the drawing board immediately. You successfully beat the control. Now, don’t rest on your laurels. What can you do to create the next control?

Mark Kolier is president of the direct and digital marketing agency CGSM.