Predictive Marketing

Test Results: Are They Reliable?

Testing is one of the building blocks of predictive marketing. Sites such as Marketing Experiments and Marketing Sherpa frequently report the results of marketing tests, with the implication being that you can apply the results to your own business. As we saw in my last post, however, you can’t infer best practices from somebody else’s test. You have to run tests with your own target audience to know what really works with them.

There is another huge problem with the reporting of test results on these sites. Let’s take a look at this test result, as reported on the Marketing Experiments site, regarding good vs. bad email copy:

email-test-results-example-4

What is wrong with this? There is absolutely no way to tell if the results are statistically significant! If you are not familiar with the concept of statistical significance, it is absolutely essential to understand if you are going to be testing.

Here’s a fact that tends to get lost in the shuffle when these popular sites review test results: if you ran the test again, you are virtually guaranteed to get different results. The % difference above may be 70%, or 30%, or even -20%, instead of 49.5%. If you haven’t thought about this before, it might seem somewhat shocking. But it’s true.

To say that the results of a test are statistically significant means that you can be confident that the next time that you run the test, you can expect similar results. Let’s take a look at one of the results I reviewed in my last post:

subject-line-test-3At the bottom of the test results, you see a statement that “The difference in conversion rates is statistically significant at the 99% level.” What does this mean? It means that if you repeatedly conducted the test again, 99 times out of 100 the first subject line would once again beat the second subject line.

The reason tests are conducted is to determine how much a particular marketing variable, such as copy, affects response, on a small portion of your target audience, before you roll it out on a large scale to your entire target audience. And before you do that, you have to be extremely confident that at the very least you can show an improvement in conversions. That’s where statistical significance comes in. If you don’t know the level of statistical significance for your tests, you are playing with fire, and might get burned.

If you would like a free calculator to help you test the statistical significance of your tests, email me at rhodgson@predictive-marketing.com.


Always Be Testing

“It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.” – Mark Twain

In today’s rapidly evolving markets, you can never take anything for granted. The average lifetime of so-called “best practices” is shorter than ever.

I experienced this first hand recently on a couple of email tests designed to drive registrations for some upcoming conferences. Both conferences were targeted at highly technical IT audiences.

My past experience had always indicated that the best subject lines had offers and calls to action, especially when closing in on a limited time offer pricing deadline. So the following test results were as expected:

subject-line-test-3

I had seen this result dozens of times before; invariably, the subject line that emphasized the dollar savings and created a sense of urgency had always emerged triumphant. Imagine my surprise, then, when I saw the results of the following test for a different conference directed at a similar audience:

subject-line-test-2

Not only did the standard subject line offer no improvement on the alternative; it produced a result that was significantly worse. There is no doubt that, for this particular audience, the second subject line produced more conversions than the first.

The beauty of testing is that marketers don’t have to figuratively stumble around in the dark searching for the best way to communicate with customers. Test and learn strategies provide a way to find out directly, from prospects and customers, what they value and want most. There is no longer any excuse for a marketer to rely on hunches, anecdotes, and biased opinions in order to make marketing decisions. Even the seemingly most insignificant of decisions – the color of a registration button, for example – may have an effect on conversion rates which can be quantified.

However, when employing a test and learn approach to marketing, there is a trap to be avoided, which is illustrated by this example. Every business, and every customer set are different. There is no one set of best practices that apply in every situation. Think of the body of knowledge that you gain by testing to be a set of “best guidelines” rather than best practices. And know that no testing program ever arrives at a final best answer. Your customers are always changing. Always be testing.

Email Campaigns and Customer Engagement

Many marketers pay a lot of attention to email stats such as opens, click-throughs, unsubscribes, etc. When you have embarked on a campaign that involves multiple emails to the same list, one of the factors you should consider is how deeply you are engaging the universe of customers that you are trying to reach.

For example, it’s typical to see open rates that decline by a fairly constant percentage on each effort, such as 20% on the first email, 15% on the second, 11% on the third, and so forth. If you were to graph the percentage of opens as a function of the number of emails sent to this customer list, it would look something like this:

multiple-emails-to-same-list-graph-v2The question now is, how many unique names on this list have actually opened the email? Or, equivalently, what percentage of the list (i.e. market) has been engaged?

At one extreme, the minimum penetration would be 20%. In other words,after the first email, no one new is opening the email. The opens on the second through fifth efforts are coming from an ever decreasing proportion of the orignal 20% who opened the first email.

At the other extreme, the maximum penetration would be 61%. In this case, the prospects opening the email are different on every effort. No one opens any of the five emails twice.

As you might expect, the actual percentage of penetration is somewhere in between. Upon examination of the email addresses from the prospect list that opened up each email, the actual cumulative penetration is 33%. The penetration after each effort looks like this:

cumulative-market-penetrationSeveral facts are clear from this. First, the 33% penetration is a lot closer to the lower of the two extremes. Second, it is clearly approaching a limit. Finally, the limit in this case, judging from the trend of the graph, is surely less than 40%, even if additional efforts are made.

There are a number of ways to use this information:

  • To keep emailing this list with essentially the same subject lines, “from” field, etc. is like beating your head against a wall. Instead of continuing to beat your head against a wall, you can test new subject lines to try and motivate new prospects to open the emails.
  • If additional information about the prospects is available, by statistically analyzing the differences in the characteristics of those who opened the emails versus those who didn’t, a predictive model can be developed to devise a messaging strategy to motivate those who didn’t open the email to open subsequent emails with the new message.
  • This information can also be used in predicting which third party lists will be most responsive and in devising messaging strategies for other lists with similar characteristics.
  • Even if you find that you can’t penetrate this list (or market) any further, you can save time and money by not trying to do so.

These graphs may look very different for your lists. But the information you need to analyze a major list in this way is readily available, and you can use it to your advantage.

Predictive Marketing