How To Improve Your Marketing Funnel: Answers to your questions about an early control peak, uncoupling value, and customer-first objectives
Every Wednesday, we hold a free Marketing LiveClass as part of ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel. Everyone is welcome to join and learn, as we build marketing funnels with members of the MECLABS SuperFunnel Research Cohort.
In the LiveClass, marketers and entrepreneurs can ask questions in the webinar chat. And we answer them right here…
What does it mean that the control peaked early and did not recover?
This question is in regard to A/B test interpretation and validity. In other words, what do the results of our marketing tests mean? And can we trust them?
I’m going to get to the actual question in a bit, but first let me address the chart that was presented in the LiveClass that sparked this question.
I think the question was in relation to this chart…
This chart does not show conversion rates over time. It doesn’t mean that the control got a lot of conversions, and then the treatment got some (and to be clear, it is actually the treatment – labeled Variation #1 – that is showing the peak on the left).
The chart shows the Bayesian probability density of the conversion rates. For this experiment, the treatment had a conversion rate of 0.24% and the control had a conversion rate of 1.01%. Without getting too deep, when you run an A/B test, you are getting a sample of the whole population.
Think of a drug trial as an example. You can’t give the drug to every possible person who would ever take it, that is the whole population. So you use a sample, and extrapolate the results. The same is true with a marketing experiment.
And as for that sample, you want a large enough sample size that you can confidently make decisions based on it (if getting a large enough sample size is a challenge for you, this might help – A/B Testing: Working with a very small sample size is difficult, but not impossible).
Looking at this chart, you could have a high degree of confidence that the control is better than the treatment because there is no overlap. And indeed, the tool gave the control a 100% probability of being a winner (probably rounding up). The treatment has a big peak around 0.24, and the true conversion rate is probably pretty close to that. The control’s true conversion rate is probably around 1, but the distribution is flatter so it may be between 0.8 to 1.2.
Those results are quite extreme though. Here is a more common distribution…
The probability of the treatment being a winner in this case is 90%. As you can see, most of the red indicates a likely higher true conversion rate than most of the blue, but there is some overlap (we’ll call that purple).
However, let’s get to the actual question at hand, because it is a good one – what does it mean if the control peaked earlier than the treatment and did not recover? That would probably look something like this…
As Egon Spengler said in “Ghostbusters,” “Don’t. Cross. The streams. It would be bad.”
When we see results cross like this, it should raise a red flag. Is there a validity threat?
Ideally, we would want our results to look like this…
This would give us more confidence that the Treatment is actually better than the results.
Which, again, brings us back to our question – but why did this happen? Here are a few validity threats you could check for:
- History Effects: Did anything happen due to the passage of time to influence results? Perhaps you launched on the weekend giving the control a huge boost, but the treatment performed better during the week.
- Instrumentation Effects: Did anything happen to the technical environment or the measurement tools that could significantly influence results? Perhaps that boost to the control was due to bot traffic.
- Selection Effects: Did anything happen to the testing environment that may have caused the nature of the incoming traffic to be different? For example, maybe someone else in the organization sent out an email promo on that day with an incentive that is only on the control landing page.
Now that we’ve discussed how to test, let’s get into a question that addresses what to test…
What does uncouple the value from the mechanism mean exactly? I’m not sure I understand.
Uncoupling the value from the mechanism means separating the value that a product or service provides from how it is delivered or produced. By focusing on the value that the product or service provides to the customer – and not on how it is delivered – we may discover new ways to deliver value to our ideal customer.
For example, let’s say you write a book filled with exercises that relieve knee pain. Is the value that is most compelling to potential customers the book itself, or the information in it? While some of your potential customers might enjoy a book, perhaps others would rather receive the information in a video. Or a podcast. Or through an AI tool or an app. You may have different groups of potential customers who would like to receive the value in different ways, so multiple formats might attract more business.
Now, sometimes the mechanism is in fact part of the value itself. For example, I love reading print newspapers and magazines. I’m on a digital device all… day… long. So reading a print newspaper with my morning coffee, or sitting out on a dock overlooking the river with a print magazine, these are customer experiences I crave. I could get the same information on a digital device, but that mechanism dilutes the value for me.
This topic is closely related to value decoupling. Is the way you have packaged and are selling the value your company delivers the most appealing way to do so?
An example I always liked was an insurance company that sells insurance for products only when you want it. The traditional option is – I buy a smartphone, and then I buy insurance for that smartphone with a monthly payment for as long as I own it. Instead, this insurance company decoupled that value into single days of insurance, and let you buy insurance just when you want it – just the two weeks you are going out of the country and worried about losing your phone. You can read more about that topic in – Mobile Marketing and Value Decoupling: Interview with Harvard professor about eight years of research into business disruption.
Uncoupling the value from the delivery mechanism, and scouring your company for value decoupling ideas, should produce some interesting marketing experiments. And I’ll end this question with a quote on this topic from a recent interview – “Be passionate about the challenge you are trying to solve and not stubborn about the product solution.”
That quote is from Andrew Beranbom. His episode of the How I Made It In Marketing podcast will be released Tuesday, May 16th.
CFO?
CFO stands for customer-first objective. You will be far more effective at value decoupling if you start with a customer-first objective. Learn more in Customer-First Objectives: Discover a 3-part formula for focusing your webpage message.
So the CFO should be focused on a funnel and specific offer in the funnel, not the overall business, correct?
While you should have a customer-first objective for everything you do, for ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel our focus is on creating CFOs for a funnel.
“That old conversion heuristic” There isn’t a new one, right?
When Flint said that in the LiveClass, he was referring to the patented MECLABS Conversion Sequence Heuristic. While there is not a new heuristic, there are many new thought tools as part of the MECLABS SuperFunnel Research Cohort.
How do we get feedback on our first assignment?
Members of the MECLABS SuperFunnel Research Cohort get feedback as they build their funnels in the Marketing LiveClasses. Cohort members can also get feedback in your small group meetings, in the LinkedIn Group, and in occasional one-on-one coaching. And of course, you can always email us.
You are welcome to join us on Wednesdays at 4 p.m. EDT to watch and learn from a Marketing LiveClass. You can RSVP now by clicking this link. Here are excerpts from recent LiveClasses to give you an idea of what you can experience…
We are trying way too hard to give away something for free
How do you think AI’s capability will change the thrust of our work as a marketer?
Categories: Value Proposition Customer-First Objective
