Content Marketing Experiment: A/B Testing

A Look at Content Marketing Metrics
CCO recently pointed out that the quants are taking over marketing in this quarter’s issue. We are on board at OpenView and track several key metrics on a weekly basis in order to effectively track our content marketing strategy. OpenView has been sending out a weekly email for a little over a year now. Since then I have shared some ideas and insight on how we have grown our program to nearly 5500 subscribers. In this post I wanted to touch base on our newsletter program. My latest experiment with our newsletter: A/B Testing.

First, what are A/B tests?
A/B tests are a method of split testing that allows for one control group and one test group. A/B testing is important because it allows marketers to more accurately hypothesize why a certain event or result occurred. Anything from subject lines to senders, content, and time sent can be tested in this A/B method.

The Experiment
Last week I ran an A/B test to gauge the difference an earlier time would have on our key newsletter statistics – open rate and click through rate. My colleague Corey O’Loughlin recently tuned into a Hubspot webinar on the Science of Timing and encouraged me to test the newsletter schedule (see Corey’s post about it here). Our newsletter typically is sent every Wednesday at 8:30 am. Last week we sent the newsletter to half of the database at 6:00 am to see what effect an earlier time could have.

Our Results
By the end of last week the results were in. The A/B test showed that the newsletter that was sent at 6:00 versus 8:30 had slightly higher click throughs and higher open rates.

Although the statistics for the A test are not statistically significant over the B results, the test yielded positive results. Going forward we plan on continuing to test earlier send times to see if we can get a result that is really significant.

Do you use A/B testing within your newsletter initiative?

Note: We use ExactTarget for our email marketing platform. About a month or so ago the folks at ExactTarget released the new version of the software which included A/B testing. I hadn’t used the new feature until last week. Needless to say, I am pretty impressed with the addition. Before it would take me about an hour to set up an A/B test within the system, and now it takes minutes. Let me just say, thank you ExactTarget!