Patience is a virtue – one that you should call upon when making decisions about whether or not the changes you made to your direct mail campaign worked.
In this article I’ll be focusing on the changes you might make to your control piece, but the basic principles presented here should apply to any change you might make to your direct mail campaign, including the mailing list, the graphic layout, and so on.
So, let’s start at the beginning. Your control piece is your old, reliable sales piece that you’ve been sending out, possibly for years, because it’s proven itself to work. It’s the backbone of your marketing program.
But the world of marketing can change quickly. And no matter how well a piece is doing, you can always do better. So, you should always be looking to improve on your control piece, whether it’s by making small changes to the piece itself, or by coming up with an entirely new piece. You should always be testing something new to see if you can outpace the results of the control piece. That’s how you keep growing your business.
And in case the point isn’t clear, you should always be performing scientific testing. Looking at a piece and thinking it’s more clever, or better written, and therefore it should work better, is no way to make a decision. We may think we know what works – but only testing can tell us what actually does work.
Why Patience Is Important
If you’ve been using the same control piece for a long time, and then you make changes to it or create a new piece, you may feel the urge to just ditch the old piece (which now looks so trite and worn out to you), and start using the fresh new one.
In your eagerness to bring on the new you may want to skip some steps. Even if you agree to do some split testing to compare the two pieces, you may jump on the initial results favoring the new piece, and immediately plan a large roll-out of the new piece to your complete list.
But let me caution you against this.
Good testing takes time – and patience. One test is rarely enough to give you definitive results. You have to keep testing your piece against the control until you have clear results. You should only roll out to the new piece to the whole universe of names after you’ve absolutely confirmed that it’s the better one.
It takes time because your testing has to consider all the variables. For example, the list you mail to, or the time of year can affect the results. It may be that one piece works better with one list, but the other works better with another. You want to compare the new piece to the control under a variety of conditions.
And there may be a combination of factors to look at. Maybe it was the teaser copy that made the difference between the two pieces – and it would work even better with the control copy as compared to the new copy. Or maybe with a different offer the control would work better compared to the new piece.
Only testing can tell you for sure.
Wait the Proper Amount of Time
You can’t really make a decision based on the results of a mailing until a critical amount of data has come in – and that takes time.
You may get great results the first week, and be ready to go with the new piece. But chances are, if you wait another week, you might see very different results.
You need to wait until you’ve received the majority of your responses. For a first class mailing that will take about three weeks. For a bulk mailing, it can be four weeks or more.
If you take a cake out of the oven before it’s properly baked, you won’t have a good product. In the same way, making a decision based on half-baked data could lead to a marketing program that falls flat.
About Testing
I’m a big proponent of always testing something. You never know when the results you get with your control piece will start thinning out, so you should always be working to develop the next great piece. Test some variable whenever you can. That’s the way to stay ahead of the curve.
But remember that you should never vary more than one variable at a time. Or if you do, properly segment your lists so that you can tease out all the variables.
And make sure that your test groups are as alike as possible. Plus, each group must be large enough to yield statistically reliable results – at least 5,000 names in each test group.
Have the Patience to Get the Big Picture
The entire life cycle of a direct mail campaign, all the way through an analysis of what happened later to the people who responded, can give you valuable information.
One factor to look at is conversion rates. For example, how many people who responded to your free trial offer actually went ahead and became paying customers? If one piece yielded more responses, but the other yielded more conversions, even with its fewer responses, then the critical issue is the number of conversions, not the number of initial responses. You’ll need to look at the overall ROI on each piece mailed.
Another important factor is lifetime value. One piece may appeal to buyers who go on to rack up a really high lifetime value. Another piece may have attracted more buyers initially, but they dropped out more quickly.
Getting information like this could take months or years. That would be much longer than you would be willing to wait to make a decision about a piece. If your new piece appears to be doing better within a reasonable period of time, then by all means, go ahead and roll out to a larger list.
But do keep tracking your results over time. If you find that you’re getting more customers with you’re new piece at first, but altogether you’re making less money, it’s time to start looking at your old piece again to see if you can freshen it up without losing whatever it was that made it work so well.