Email Optimization

The Situation

A market research firm had just completed a new report on the rapidly growing nanotechnology market, their first product in the category. Because of the wide ranging impact of nanotechnology, the firm expected that the market for the product would range well beyond their traditional set of customers and prospects.

The Challenge

The firm knew that selling a new product in a new market would require some experimentation and fine-tuning. They decided to employ a test and learn strategy in order to determine the best approach to selling this new report.

Email was chosen as the vehicle for testing. Email addresses of prospects were readily available, and promised a quick resolution of how best to sell the report.

The Solution

Two main approaches to testing were considered. The first was simple A/B testing, which involved side-by-side testing of two alternate approaches. The advantage of A/B testing was that it is very simple conceptually, and does not require much statistical know how to conduct and interpret the results. The disadvantage of A/B testing, however, was that the firm wanted to test many different approaches to selling the report via email. The firm wanted to test two subject lines, two offers, three salutations, three primary benefits, and two calls to action. This meant that to test all of these variations, they would have needed 72 test cells (2 x 2 x 3 x 3 x 2) plus a control cell. This would have required a much larger number of email addresses than they had available, and required additional addresses to be rented at a considerable cost.

The second approach is called multivariate testing, in which more than two variables are tested at a time. This approach requires more statistical know-how, but provides a way to test all of the alternatives more quickly and cost effectively.

Predictive marketing helped the firm structure the best way to test all of the alternatives. This structure, called an experimental design, was carefully constructed to minimize the amount of testing needed, while maximizing the amount of information that could be learned through the test. Instead of having to test all 72 combinations, an experimental design was developed so that only 16 combinations had to be tested.

The data from the test was analyzed using a statistical technique known as logistic regression. Using this technique, the performance of all of the 72 combinations could be predicted. The firm discovered that the optimal combination of the email elements generated more than four times the response of the worst combination. The details are summarized in the accompanying table.

The Results

The benefits from the test extended beyond the email campaign. The direct mail pieces and the telemarketing scripts were developed after the test and incorporated the information obtained to optimize those approaches as well. Overall, the test provided the foundation for the marketing campaign and helped the report become the most profitable in the firm’s history.