MarketingSherpa Blog

“The best research is when individuals pull out their wallet and vote with cold, hard cash.” – my first boss

My first experience in marketing was working with a specialized publishing company. I had the privilege to work on exciting products with sexy topics such as “human resource compliance regulations.” Trust me when I tell you there is no better ice-breaker at a party than talking about a ground-breaking court ruling that will change how your company meets compliance of the Fair Labor Standards Act (FLSA).

As a publisher, we used direct-response marketing to drive sales, with an aggressive program of direct-mail, email and telemarketing. And when it came to new product development, we were big believers in research. From customer surveys to industry research to focus groups, we used it all to make the best possible decision. At least, that was the general assumption…

Out of focus

You always have to test because many research tactics just help you achieve a best guess. And while a best guess is often closer to the truth than a random guess, it’s sometimes widely off the mark. In fact, I learned a valuable lesson one day when our company performed a focus group.

The members of this particular focus group were subscribers of a paid newsletter, and we knew that each person had subscribed by responding to a specific direct mail piece. That mail piece was extremely effective, with a powerful but somewhat provocative subject line and letter. Many people loved that direct-mail piece, but many hated it, so we wanted to get the opinion of the focus group members. When we showed the group the direct-mail piece and asked them if they would respond to that piece, 40 percent said they would never respond (if they only knew what we knew). Wow, we were shocked!

So, should we conclude that those 40% were bold-faced liars? Not necessarily. What we can conclude is that what people say they will do and what they actually do may be totally different. That is why research is only part of the equation, but if you want to sleep well at night, you have to take the next step…

Voting with their wallets

At the end of the day, the best research was when we tested the product and let the customers in the marketplace determine with their wallet if it was a viable product. We would test critical elements, like book title and price, and very quickly we would know if we had a winner or not.

Yes, all of the surveys and research were necessary to get started, but the most critical research was in our testing program. Testing is an amazing research tool. Regardless of the conversion you are trying to achieve, when your prospect takes (or doesn’t take) an action, you a have a valuable piece of information. Your conversion goal may be an event ticket sale, a white paper download, an email newsletter signup, or hundreds of other possible actions, but one thing never changes – the action you are seeking to drive can be tracked.

And if you’re ready to measure when your prospect engages with you, that is when the learning begins.

So, I’m thankful for that boss early in my career telling me repeatedly that the best research is when individuals pull out their wallet and vote with cold-hard cash. Over the years, I’ve had many experiences when individuals tell me they are going to do something but until they actually do it, I’m a little skeptical. (Editor’s Note: It’s true. Todd told me he was going to write a blog post for quite awhile. Now, I believe it.)

So gather as much research as possible, but always remember that cold, hard cash is a pretty sweet piece of research.

Related resources

Are Surveys Misleading? 7 Questions for Better Market Research (Members Library)

Marketing Research and Surveys: There are no secrets to online marketing success in this blog post

Focus Groups Vs. Reality: Would you buy a product that doesn’t exist with pretend money you don’t have?

Never Pull Sofa Duty Again: Stop guessing what your audience wants and start asking

“Would you like to hear a secret? Do you promise not to tell?” John, Paul, George and Ringo knew how powerful secrets are, as does every Internet marketing “expert” who has ever written a blog post.

Well, I’m sorry, but MarketingSherpa and MarketingExperiments don’t have any secrets to share with you. The only effective strategy I’ve ever seen is hard work and experimentation. Not only do we not have secrets for you, we don’t really even have any answers. But, we can help you ask the right questions.

Question everything

“My mother made me a scientist without ever intending to. Every other Jewish mother in Brooklyn would ask her child after school, ‘So, did you learn anything today?’ But not my mother. ‘Izzy,’ she would say, ‘did you ask a good question today?’ That difference, asking good questions, made me become a scientist.”
– Nobel laureate Isidor Isaac Rabi, discovered nuclear magnetic resonance

And do we ever raise those questions. Like a recent article by Senior Reporter Adam T. Sutton, Are Surveys Misleading? 7 Questions for Better Market Research. When Adam first showed me the article, I knew it would be a little controversial, so I pushed him a little harder than normal in the editing process. Look at the results, and I think you’ll agree that Adam delivered. (If not, I want to hear about it.)

I was a little surprised that the biggest challenge came from within my own company, though. MECLABS Director of Research, Sergio Balegno, questioned the article’s affront to online surveys. Sergio’s a smart guy, so when he says something I listen. And I think he’s right. Well, kinda…

When online surveys are effective

For the kind of surveys Sergio’s team conducts, I believe surveys to be very effective. I use his team’s research all the time in trying to decide what content would be the most helpful for  MarketingExperiments’ and MarketingSherpa’s audiences.

click image to enlarge

The above referenced article, from a recent Chart of the Week email newsletter, questions B2B marketers about the SEO tactics they are currently using. Sergio and his team are not asking about a vaguely potential and highly personal decision somewhere down the road; they are simply asking which SEO tactics B2B marketers use, which were the most effective and which required the greatest level of effort? And here’s where you can learn from Sergio.

I believe surveys can be effective for:

• Gaining insights into current actions
• Deciphering opinions on specific subjects that the audience has a high-level knowledge about
• Getting some new ideas (essentially, crowdsourcing)

When online surveys are not effective

“Would you buy a product that doesn’t exist with pretend money you don’t have?” Yeah, there’s the rub…

Online surveys do not accurately predict actual customer behavior. Or, do they? Frankly, it’s just a shot in the dark. Your goal should be to try to truly gain knowledge about real-world situations that require complex, often counterintuitive decision-making processes that your subject may not even understand. Would a few questions on a Web page really help you gain that knowledge?

Online surveys are not effective when you’re trying to decipher:

• Potential consumer actions (such as a purchase)
• Potential B2B marketer purchase decisions very early in a sales cycle (too many variables)
• Highly sensitive information (if you disagree with this statement, please share your past three sexual experiences in the comments section of this blog)
• True sentiment on a complex topic that the survey respondent does not have expertise in. For example, 58 percent of Americans favor repeal of the new health care law, according to a recent Rasmussen Reports survey. Meanwhile, in a CBS/New York Times Poll, 41 percent of Americans favor repeal (stop and think about that for a second); and when people were actually told what features would be given up if the law is repealed, that number dropped to 25 percent.

Let’s do a little thought experiment, shall we? Write the answer to this question down on a piece of paper and bury it in your backyard… “How likely are you to buy each of the following in the next 12 months: regular mayonnaise, light mayonnaise, mayonnaise with olive oil, canola mayonnaise, low-fat mayonnaise?”

Now go leave yourself a reminder on Outlook for November 23, 2011 that says, “Dig up mayonnaise survey.” So, how accurate were you Carnac the Magnificent?

Only you can discover the marketing tactics that work best for your company

OK, I was a little too fresh up there, sorry about that. But I’m trying to help you understand this simple point (to annotate MasterCard)…there are some things in marketing that can’t be observed, for everything else try an online survey.

If you can’t observe the information you seek to obtain and there is a strong likelihood that your subjects know the answer, then a survey could be very helpful. In the example chart above, you likely could not observe the SEO tactics of 935 marketers and see into their brains to determine the effectiveness and effort required. Those respondents also likely know what SEO tactics they used, how well they worked and how much effort they required.

However, when you’re looking at potential customer actions, don’t try to ask prospective customers to predict what they might do under fictional, hypothetical circumstances. From the number of times I’ve asked my wife why she bought those shoes, believe me when I say she likely doesn’t know the answer herself.

Instead, simply observe their actual actions. And you can do that with real-world, real-time online testing.

After all, that is the real goal of all the information we provide. Again, we don’t write about secrets to Internet marketing success on MarketingExperiments and MarketingSherpa, and very rarely even give you any answers.

But we do help you ask the right questions and then do the experimentation (and hard work) necessary to determine what works best for your organization.

Related resources

2011 Email Marketing Benchmark Report

2011 B2B Marketing Benchmark Report

Ask the Scientist: Price testing methods and practices

Anti-Crowdsourcing: On (not) getting marketing ideas from your customers

Many marketers have yet to explore mobile webpages as a marketing opportunity. If you are putting off tests because you think they are too complicated, take a look at this case study on mobile page testing we published today.

Mike Brown, VP, Internet Optimization, Vegas.com, and his team tested if catering to a mobile audience could improve site performance. The team showed mobile visitors a homepage and category pages designed for their devices. The test was a proof-of-concept on whether to invest in an end-to-end mobile experience for Vegas.com.

The tests successfully improved site metrics such as bounce rate and conversion rate (see the case study for details), and Brown’s team plans to rollout more mobile pages this fall.

Brown said these tests were easier than his team anticipated. From our conversation, I understood two criteria the team met to successfully run them:

– Mobile traffic present

Testing mobile pages only makes sense if your website receives mobile traffic. The team looked at its site analytics and noticed mobile visitors accounted for 7% of traffic, and growing rapidly.

– Testing and page design expertise

The team had extensive testing experience with Brown as the head of Internet Optimization. Also, the team used SiteSpect‘s multivariate and A/B testing tool to run tests. SiteSpect team members helped Brown’s team learn to create mobile pages, which Brown noted was surprisingly easy.

So if your team has capable developers, some mobile traffic and good testing experience, there should be little preventing you from running similar tests. And remember: you don’t have to dive in with a full investment. Brown’s team only dipped its toe and is now comfortably moving forward.

There is an array of webpage testing solutions available, helping marketers improve their landing pages, homepages and other online real estate. With so many options, it’s easy to get wrapped up in selecting tools, setting them up and testing them.

The problem is some marketers will spend months selecting and deploying a tool only to A/B test button colors, or different images on the same layout, says Lance Loveday, CEO, Closed Loop Marketing.

“To me, that feels like running 25 miles of a marathon and walking the last one.”

Time is much better invested in researching page data and designing tests that have the strongest likelihood of success, Loveday says.

“90% of your time should be in the planning and actual analysis and coming up with insights, and 10% should be in the technology.”

We spoke with Loveday for an upcoming MarketingSherpa article on how to select better landing page tests. One key to Loveday’s strategy is gathering thorough research, including:
o Analytics data
o Click-tracking analysis
o Qualitative usability studies
o Expert reviews

“We try to marry up quantitative analytics data with qualitative user experience and user profile information to develop some hypotheses for what the problem areas [on a page] might be,” Loveday says.

By digging through this information, your team can identify areas for improvement, attempt to diagnose problems and test solutions. Furthermore, you can estimate tests’ potential impact and prioritize those expected to bring the greatest benefit. Keep an eye on our Great Minds newsletter for more information.

Going through my mail a few days ago, I came across a letter from the U.S. Census Bureau. I read it and was baffled to find it alerting me to the census forms’ arrival next week.

“What a stupid letter,” I thought. “I’ll see the forms when they come.”

However, an editorial in my local paper the next day made me think twice.

In “Surprisingly Sensible,” the Spartanburg Herald-News’ editors report that while this letter cost about $50 million to send, it’s part of an alert and follow-up strategy. For the 2000 census, the strategy lifted the number of returned census forms 6 percentage points to 67%. How’s that for a response rate?

The letters arrived one week before the census forms. Ten days after the forms, a post card was sent reminding recipients to fill out the forms and return them, according the editorial.

The 6% lift reduced the number of houses the census workers had to visit, which translated into huge cost savings. So the $50 million may prove to be a sound investment.

The editorial goes into further detail on how much money the strategy could save. The important point for marketers is that even when a strategy does not make sense on its face — it still might lift performance.

MarketingSherpa’s recent Selling Online Subscriptions Summit 2008 in New York City gave scores of takeaways to participants. Here are a few of most consistent messages I picked up from the conference:

Read more…