Marketing Experimentation: How to get real-world answers to questions about a company’s marketing efforts
Whenever we have questions in our weekly LiveClass – ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel – we answer them here on this blog to help attendees, but to help any of our readers who didn’t attend but may have had a similar challenge as well. Read below and get ideas for powering the growth of your business with marketing experimentation. And feel free to join us for a Wednesday LiveClass using the link I just mentioned.
Has anyone tested video ads vs image? Preston’s question in the LinkedIn group got me thinking. Daniel Burstein, do you know of any?
Here’s an example. “The team discovered through testing these Facebook ads that medium-form copy received 33% more clickthrough than short- or long-form, and video drove more clickthroughs and had greater reach than static images. Because of this testing, the team realized that video was one of their most powerful tools” (from Email Marketing: List size increased 600% in one year through content, paid ad strategy).
The reason I start with this question is because it lays out the reason we should be conducting marketing experimentation – so our customers answer key questions we have about our company and our marketing with their real-world behavior, instead of just making decisions based on internal guesswork.
No matter what has worked for another company (including the above example I just used), it doesn’t mean it will work for your unique customers, your unique value proposition, your unique situation. So don’t just follow what others do, use it as fodder to come up with your own wildly creative ideas – and then test them.
Now that we know why we should test, let’s get into some of the mechanics of testing, starting with test planning…
What is the main purpose of the [pre-test planning] calculator? Is it to estimate how long you’ll need to run the campaign? If I have a set daily budget, set level of confidence, set conversion rate, set number of variants and an unknown variance [the questioner is referring to the relative conversion rate difference between variants of the ad], what is the main purpose of estimating the data? As soon as the ads start running the numbers will all change and need updating. What’s the main metric we’re aiming on understanding?
As I have mentioned in a previous blog post, I’m not the mathematician, I’m the storyteller. So I won’t get into the math behind these concepts. But I do think it helps for marketers to understand the concepts at a basic level, to inform their testing. And if you haven’t already, reading last week’s blog post will help get you up-to-speed on some basic marketing testing concepts – Factors Affecting Marketing Experimentation: Statistical significance in marketing, calculating sample size for marketing tests, and more.
In this question the person is outlining the different metrics in the pre-test estimation tool (that is included as part of membership in a MECLABS SuperFunnel Cohort).
The goal of the tool – as with any planning – is to inform your efforts. Things may not turn out exactly as you think they will, but when you see the levers that you can pull to shape results, it should help you make decisions on what you want to move forward with and actually execute.
One of those levers is budget, to affect how large of a sample size your treatments receive. In this question the “daily budget” can’t be changed, so if you run the pre-test calculation and realize it would take an inordinate amount of time to reach statistical validity based on the amount of impressions or traffic your budget can buy, you may have to get creative.
Here’s one example, using Bugs Bunny and Daffy Duck. The team behind Looney Tunes and Merrie Melodies had a set ‘daily budget’ for how long they could work on an episode. So if they wanted to do something a little more groundbreaking, they would ‘borrow’ from other episodes. Maybe they would take only four days instead of five for a few episodes, and then using that time they saved up they would dedicate eight days to an episode to really push the envelope (I recommend Duck Amuck and What’s Opera, Doc?).
Perhaps this planning tool might help you do the same thing. Invest less of your fixed daily budget in experiments where you know there will be a large conversion rate difference between the ad or landing page variants, so you save up some of that budget for experiments where there is a smaller difference.
Which brings us to another decision pre-test planning can help us make…
Ave Test Users = Impressions … Primary KPI Successes = Clicks (Or is it Opt-ins?) Variants = number of ads, Assumed Real Difference = Between the ads
The question is around what the primary KPI (key performance indicator) should be for an experiment – clicks on the ad or opt-ins on the landing page.
This is important to statistical significance because it will impact the sample size. The deeper you test in the funnel, the smaller your sample size will be. More people will click on an ad then will opt-in on the landing page. So the larger budget you will need to get a large enough sample size.
This is another reason why using the pre-test estimation calculator can be helpful – to inform where in the funnel you decide to test, based on your budget and/or other capabilities for getting people to see the conversion action you are trying to test.
What should your sample size be? There is no set sample size you need to reach. It is affected by how different the performance is in the different treatments. And we use the pre-test planning calculator to help us find that number. “An important factor in sample size determination is the difference in results between the treatments. If the treatments return very different results, it’s much easier to confidently say that you really do have two (or however many) emails that will perform differently. You don’t need as many samples to do that. However, if the treatments have very similar results, you want many more observations to see if there really is a difference.” (from Marketing Optimization: How to determine the proper sample size)
Why make impressions-to-ad-clicks as the primary KPI and then ad-clicks-to-appointments a secondary KPI? Why not make impressions-to-appointments the primary KPI? Is that just because the volume of data for the latter would be higher? So measuring more the message lever, rather the whole funnel effectiveness?
This question naturally follows from the previous one. As discussed, your KPI for a test is partially informed by the sample size and relative difference between the control and treatment.
But it is also partially informed by what you are trying to learn. Let’s not get too deep in the math and mechanics that we forget the goal of test – to learn how to better serve the customer and better communicate to the customer so we can improve our results (as discussed in the first question in this blog post).
And remember, you don’t have to learn everything in one big test. Your goal is to test and learn repeatedly. So your first test can be in the channel, and then your second test could be on the landing page, for example.
To inform future testing it helps to establish secondary KPIs when you set up your test. Of course, you could just look at every possible metric under the sun. But the reason we do pre-test planning is to run our experiments with intention.
One of our former data scientists explained it to me this way – an experiment is picking a specific tree in a forest, then throwing a rock and seeing if you hit that specific tree versus throwing a rock into a forest, seeing which tree it randomly hits, and then remarking, “oh yeah, I meant to hit that tree.”
Pre-test planning focuses our thinking and efforts on hitting that tree. And secondary KPIs can help here as well. “Don’t try to analyze it all; you’ll get lost in data and become discouraged and confused. Instead, narrow your focus to the metrics that will provide the most relevant insights. Having primary and secondary KPIs for your site will help you begin to narrow your focus” (from Marketing Analytics: 6 simple steps for interpreting your data).
So you could first test message levers in the channel using a primary KPI of clickthrough. Then if your secondary KPI is conversion rate on the landing page, and the conversion rate is low for the winning ad treatment, you have a next (and very interesting) question to test.
Did the ad message you used simply bring lower-quality traffic, less motivated people to your website? For example, an ad message of “Free iPaid” might win a test in the channel but isn’t necessarily doing your funnel any favors if you can’t pay off that promise.
Or, is it an effective ad message to attract your ideal customer, but you don’t pay off that message well with you landing page. There is poor continuity between the ad and the landing page, which was simply a previous page you already had and didn’t align with the treatment ad’s messaging. Which brings us to…
Why would we use a page separate from our webpage?
You may have an ad message you want to test that is very different from the message on any webpage you have now. In that case, it would likely make sense to create a new landing page for it.
We could use the experimentation process I just mentioned to determine that our current webpage does not deliver on the new ad message, that there is poor continuity from the new ad message to our webpage.
However, signing up for marketing experimentation is not a binding agreement to endure unnecessary agony. We can also use our common sense and marketer’s intuition to avoid changes that are very likely to have poor results – after all, we are testing with a real budget and real people – and focus our energies on the questions that are more difficult to answer and will have a big impact on our business.
Of course, this means you may have to build a new landing page. Which brings us to our next question…
Chris it looks like you followed the template exactly, Daniel are you saying feel free to move the layout around and put the hero image in place of the form?
Participants in the MECLABS SuperFunnel Cohort get access to MECLABS SuperFunnel builder software (a simple drag-and-drop landing page builder that is embedded with the MECLABS methodology). This includes templates to help you build a landing page based on an approach that has shown to be effective in previous experimentation.
I won’t go through the entire template, but it begins at the top of your landing page with Micro-Yes 1 – Yes, I will pay attention. This section should be a synopsis of your core offer. Followed by Micro-Yes 2 – Yes, I will engage deeper – where you address any friction or anxiety. Also, in this section you can add a video, image, or a form that supports your message.
The Cohort member I was providing optimization advice for in the LiveClass has a spokesperson that is well-known in his industry, and my point was invoking that spokesperson earlier on the landing page may help grab attention and be a core element of the offer that gets the ideal customer to say ‘yes.’ It may also be a way to reduce anxiety.
For example, I am a huge fan of Jerry Seinfeld. If he ever produced a show focused on marketing (let’s call it Copywriters In Cars Getting Conversions), his participation in the show would be a core part of the offer that would grab my attention. His participation in the show would also reduce my anxiety that it is a quality show and get me to engage deeper.
The SuperFunnel template can guide you to build your landing page but should also be spurring future hypotheses in you for further testing. For example, is our founder a core part of the offer and she should be included at the top of the page along with her image? Or is she a tertiary credibility indicator, and she should be used lower down on the page to help address any last-minute anxiety?
This is an example of a question you can test that has implications for the business. You can start by testing in the channel and conduct further testing on the page.
To get you thinking of possibilities for your own business, you can see three value categories we test through email for VolunteerMatch in A/B Testing: How to improve already effective marketing.
If you are conducting a marketing experiment in the channel, you will also need to create the different ads to test. Which bring us to…
Ask ChatGPT what colors to use: “I have an ad with #CFB82C as the primary text color, #384E6C as the background color, and #333333 and #FFFFFF as additional colors. What color should my call to action button be and what color should the text be on top of it. Give me html codes.” … “Is there a green color that will work?”
OK, this isn’t really a question. I found this in the Zoom chat, and I just thought it was some nifty advice for using artificial intelligence to help you build an ad.
Hi Flint, what does the cohort entail? 4-5 weekly meetings. 40 days total/ price? Also if I’m generating a sales funnel for the first time is this a good option for me or should I start on another level (finishing my Micro Yes’s now)
How do I join this group!?
How do I join the MEC200 group?
How do I register for the next cohort?
Is there another cohort scheduled yet after April or approx. start date?
Jane, Is this video zoom call a part of the first cohort?
What happens with the transition to cohort 300?
There were a lot of questions from attendees about the cohort itself, and we answered them with a Q&A session at the end of the cohort. If you would like to learn more about the cohort, and the five ways you can ‘pay’ if you choose to participate (there are monetary and non-monetary options), just join us on a Wednesday LiveClass of ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel.
Even if you don’t choose to join, by attending you should get a few ideas you can implement to your marketing funnel right away to improve conversion.
Here’s a quick excerpt from a recent LiveClass to give you an idea of what to expect – Hypothesis Articulation vs Essence.
