A/B Testing: Why don’t companies track ROI of testing and optimization programs?
During our second annual MarketingSherpa and Marketing Experiments Optimization Summit 2012 two weeks ago in Denver, Dr. Flint McGlaughlin, Managing Director, MECLABS (the parent company of MarketingSherpa), presented some interesting data points on brand-new MECLABS research conducted by Meghan Lockwood, Senior Research Analyst, MECLABS, for the upcoming MarketingSherpa Website Optimization Benchmark Report.
I live blogged this material for MarketingExperiments, but I thought the research was worth sharing with our SherpaBlog readers as well.
One data point from this research from our Website Optimization Survey, which will be presented in an upcoming benchmark report, really stood out to me:
Click to enlarge
53% could not calculate ROI of website optimization
More than half of the surveyed marketers do not — or just could not — track the return on investment for their website optimization efforts.
Honestly, I was a little bit shocked. Unlike some marketing programs, optimization is by its very nature data driven. Done correctly, it is carefully tracked for validity, as well as for statistical soundness in the results represented by a confidence level above the 90th percentile. (Unfortunately, only 60% of marketers do validate tests.)
But, even for those who don’t validate tests, A/B testing is very numbers driven. After all, you’re seeing which treatment pulls in better results.
Given all these numbers and attributes being collected and analyzed, it would seem like putting some key performance indicators in place that can go from optimization all the way to the bottom line would be relatively painless.
38% reported positive ROI of website optimization
Let’s go back to the chart I threw out there. For marketers who do track ROI on optimization, the results are very favorable: 38% report a positive ROI, while only 3% report a negative ROI. Clearly, optimization overall has a positive effect on the balance sheet.
Meghan agrees with me, saying these results are surprising.
She says, “It is pretty shocking that marketers would not be tracking ROI, because once you get into the C-suite, you can’t be creative with your numbers.”
25% of marketing budget spent on the website
And Meghan adds, “The website accounts for one quarter of all marketing spending, so failing to measure your optimization efforts means that you have no idea how hard your dollars are working for you.”
That data point comes from the MarketingSherpa 2011 Search Marketing Benchmark Report.
An attendee and presenter from the Summit, Tim Kachuriak, Senior Vice President, Innovation and Optimization, Pursuant, says, “I was shocked to learn that most marketers who are testing are not able to measure the return on investment for their efforts. If you can’t measure the impact to the bottom line, then why waste your time? Optimization must always result in a learning that leads to impact. If not, then it isn’t testing — it isn’t optimization — it is just wasteful tinkering.”
Meghan did say that marketers are beginning to pay attention to how optimization, particularly on the website, can improve marketing efforts. She also gets the sense that practitioners understand the high-level aspect of testing and optimization.
Why measuring ROI of website optimization is difficult
Meghan also added two challenges that can make tracking optimization ROI somewhat “murky.”
One, when optimizing the homepage, the testing program is possibly dealing with multiple constituencies and calls-to-action, making it difficult to follow those results to the final sales conversion.
The second is a testing program will most likely involve several organizational groups within a company – Marketing, Information Technology and Design, to give one example – so the challenge becomes what group gets to claim the success for a positive ROI.
One issue in tracking website optimization stems from traffic being the most tracked website metric.
Meghan says, “While traffic is a contributing factor for the ultimate ROI of your website, there is not a direct line between the amount of people who visit your site and your ultimate conversion rate.”
She adds marketers need to understand what KPIs are most vital to the organization, and also know how website analytics inform those metrics. The challenge is fully understanding how interrelated, or not, these metrics are.
For example, a company with high-volume business, average order value might be the most relevant metric in regard to optimization, particularly when experimenting with an up-sell or cross-sell function on the site.
Companies with a lower volume, but higher dollar business, probably look at the lifetime value of a customer or total sales at the most relevant metric.
Another challenge Meghan mentions is that the sheer volume of data available online acts as both a blessing and a curse, and she cited a MECLABS A/B test on one of our webinar email sends. Treatment “B” dramatically outperformed “A” in clickthrough rate, but after drilling into the data, our researchers learned the majority of clicks on “B” were on the opt-out link.
“When you looked even one layer down on that onion, putting the second email into rotation would have been a killer for the webinar’s ROI,” explains Meghan.
“The website is also the hub of all marketing efforts, so all of your other online activities – including PPC and SEO, as well as any content marketing or LPO initiatives – are all going to circle back to the success and efficiency of your website,” says Meghan. “Where is your funnel? How are you measuring the success of your webpages? If you don’t know, how are you going to justify your budget next year?”
And you might ask why did I devote a second blog post to this topic?
Check out this chart from our research:
Click to enlarge
70% of marketers use lessons from website optimization to inform offline or other marketing initiatives
That’s right – 70% of our surveyed marketers reported that website optimization provides actionable ideas that can improve their overall strategy.
When you can show the value of optimization and its impact on the bottom line, you’ll be proving your worth to the C-suite and improving your entire marketing program at the same time.
Here is Dr. Flint McGlaughlin presenting the earlier chart on Meghan’s research at Optimization Summit 2012:
Related Resources:
Multivariate Testing: Can you radically improve marketing ROI by increasing variables you test?
Page Optimization: How to start optimization testing and get executive support
Page Testing and Optimization: How Intuit grew revenue per visitor 23%
Page Optimization: Radical redesign leads to 3,566% increase in conversion
Nonprofit Marketing: How a long, ugly page generated 274% more revenue
What to Test: 4 sample landing page treatments from Optimization Summit 2012
Categories: Research And Measurement A/B testing, Optimization Summit 2012, ROI, website optimization
We found the exact same lack of ROI tracking in our recent survey of B2B companies: just about 50% of our respondents who were testing social media for lead generation said they haven’t or can’t measure ROI. And we had the same reaction – shock.
If you’re not measuring ROI, why even bother testing? Just judging by ‘feel’ or ‘instinct’ isn’t going to give you results you can trust. And measuring some other metric – closes, say, or even revenue – doesn’t give you the complete picture.
Our study is at http://www.buyerzone.com/stateofB2B if you’d like to see more detail.