MarketingSherpa Blog

During AI Guild research briefings and LiveClasses, guild members ask questions using the Zoom chat feature. And we answer those questions right here…

Why Claude 2 vs ChatGPT? It’s important to pick a long-term tool to be able to teach it in depth. So looking to understand the pro’s / con’s.

I’ll challenge the idea of the need to pick a long-term tool (although we’ll get into that in the next question in this blog post).

The biggest difference I’ve seen between Claude and ChatGPT is that Claude can process much larger inputs than ChatGPT. You can even attach files when you message Claude.

This is a key differentiator if you want Claude to analyze a chat log, write a summary based on a transcript, or give you feedback on a long article.

ChatGPT gives you the ability to tune all of its responses to some extent using the custom instructions feature, although this is only available in the Plus subscription for $20 per month. The Plus subscription also gives you access to GPT-4.

The site explains the features as “Custom instructions let you share anything you’d like ChatGPT to consider in its response.” It is not available yet in the UK and EU or the free version of ChatGPT, which is currently using GPT-3.5, but there are plans to roll it out to all users soon, according to OpenAI’s website.

If you choose to upgrade to the Plus subscription, you can turn this feature on by clicking the three dots next to your email address in the lower left, going to the “Beta features” section and moving the slider next to “Custom instructions” to green. Then you will have access to two questions:

• What would you like ChatGPT to know about you to provide better responses?
• How would you like ChatGPT to respond?

You could, for example, include information about your ideal customer when answering these questions. If you would like ideas for how to get AI to build a customer persona, AI Guild members can go to the Briefing Notes for MEC200 [08-02-23], scroll down to the “Ken Ducey” group coaching and click the drop down carrot, and read the prompt under “Using GPT and the new Custom Instructions.”

Keep in mind both ChatGPT and Claude have a knowledge cutoff in 2021 (ChatGPT specifics September 2021 and Claude is less specific), so if more recent information is key to your prompts, you should consider AI connected to the internet, like Bing.

I also asked both Claude and ChatGPT about their pros and cons. Like any market leader might say about its challengers, ChatGPT ‘never heard of it.’ In fairness, this is probably literally true. As I mentioned, ChatGPT’s knowledge cutoff is currently September 2021, and Claude was first released in April 2022.

Claude said that ChatGPT has broader capabilities since it was trained on more data, but that Claude does a better job of staying on topic and remembering context from previous conversations. That has pretty much been my experience, and I have been using Claude more because of it. While ChatGPT can have some truly outstanding responses, it can also be wildly off base. I see that less with Claude but it does happen as well.

And in fairness, this has been a challenge with all generative AI tools. For example, a guild member mentioned in the chat that he had trouble with Bing giving fake info then getting upset.

If you’d like more insight about different AI tools, AI Guild members can go to Guild [07-26-23], scroll down to the Prompt School section, and read point #6 “Note the best uses for each of your primary AI platforms.”

Whichever tool(s) you use, the AI won’t do everything. The prompts and prompt stacks you input will significantly impact your results. Flint McGlaughlin walks through an example in Sales Funnel and ChatGPT: 6 ways to leverage AI now to develop a powerful value proposition.

Why do you say ‘no’ long-term use of AI tools? Just because of the rate of change?

Don’t build a house on sand.

Vendor lock in is a challenge with any industry, but the mass adoption of artificial intelligence is so new and moving so quickly that what is true today may already be dated in three weeks, let alone three months or three years.

For example, the ChatGPT feature I mention about for “custom instructions” is listed as a beta feature. And OpenAI states, “As a Plus user, enjoy early access to experimental new features, which may change during development” (my emphasis added).

Now there may be some artificial intelligence you have to lock into on some level. For example, MECLABS AI is built on ChatGPT, the best available option at the time we launched. You might train an AI chatbot and add it to your website or use AI for many different things like media buying. These choices will inherently necessitate placing a bet. But do everything you can to stay as fluid as possible and avoid vendor lock in.

However, when it comes to using generative AI chatbots, they are easily switchable (all you need to do is open a different tab in your browser). So keep your options open, stay curious, and keep playing.

Yes, you would need to teach a customer service or sales AI chatbot you put on your website in-depth information about your company. But I’m unaware of how much any generative AI LLM really learns from your interactions with it between sessions (other than the simple feature mentioned above in ChatGPT). So the downsides of brand loyalty may trump the upsides.

I want to mention another element of not thinking “long term” that goes beyond AI tool selection – designing your offer. Keep enough flexibility in your business model that you can bob and weave and pivot as technology changes. I’m not one to make predictions, but here’s a sure bet – things are going to get pretty interesting over the next few months and years.

To remix a Ferris Bueller quote, “AI moves pretty fast. If you don’t stop and look around once in a while, you could miss a really good business opportunity.”

Anyone have any ideas on how to use AI for lead gen? Specifically, analysis and recommendations to get in front of ideal prospects?

This question sets up the importance of AI prompts.

When I asked Bing AI this question for a wedding industry company with a very low-priced product, I got back some pretty rote answers. For example, “Share success secrets from thought leaders.” and “Create helpful videos to solve issues for prospects.” This is the type of basic advice you’d already find in an SEO-ed up blog post about lead gen.

But then I copied a bunch of reviews for this wedding industry company, pasted this into Claude, and gave it the prompt…

Analyze reviews. Who is the ideal customer for this company?

Claude gave me a series of attributes about the customer. Then I asked Claude…

How can I get in front of this ideal customer?

Some of the recommendations were pretty basic, like “Advertise on wedding planning websites and forums like TheKnot, WeddingWire, etc. Brides often turn to these sites when planning their wedding on a budget. You can sponsor content or run ads.”

But it also came up with this response as well, “Target ads in college towns, where many younger budget-conscious brides get married right after graduation.”

I thought that was a pretty good idea for a low-cost wedding service. An idea I hadn’t thought of when reading the question (although, in fairness, an entrepreneur who has worked in this field for several years might have already considered it).

To show how you can use multiple generative AI tools in tandem (in response to the two previous questions), I then pasted the ideal customer description from Claude into GPT-4 (remember, Claude is better at analyzing large amounts of information) along with the same prompt.

Again, most of the suggestions were pretty obvious to anyone with even a modicum of marketing experience. For example, “Optimize your website and blog content with keywords that this audience would likely use when searching for affordable wedding video solutions. Keywords could include phrases like ‘budget-friendly wedding video,’ ‘DIY wedding video,’ ‘guest-shot wedding video,’ and similar terms.”

So I pushed GPT-4 a little harder and said…

All these answers seem obvious to someone with marketing experience. Give me some truly breakthrough and out-of-the-box ideas.

It gave me seven more ideas, and I thought this one was something really unique that I hadn’t thought of: “Video Capsule Partnerships: Collaborate with a tech company that offers ‘time capsule’ services, where users can send messages (including videos) to their future selves. Couples could use the company to capture their wedding and then ‘send’ the video to be viewed on a future anniversary.”

This won’t generate instant leads, but it would give the company a reason to followup with happy customers down the road, at which point they could offer a referral bonus.

So I got two original ideas after only spending a few minutes with some AI tools. Not a bad ROI. If I really engaged deeply for an hour or more, if I was the entrepreneur and it was my business, I probably would have gotten much more from the generative AI tools.

The biggest point is – AI can feel like magic. And it is in a sense, but not the way we normally talk about magic as waving a wand and getting an instant, amazing result.

Magic in the real world is a carefully crafted illusion, powered by hard work and a keen methodology. And AI is no different.

You can’t just ask a general question and get an amazing result, usually. But feeding AI tools info about your ideal customer and the design of your offer will produce better results, especially for something as difficult as finding a new way to get in front of your customer.

Check out Marketing Funnel Optimization: A straight-forward guide to design your offer if you are unfamiliar with offer design. And if you’re an AI Guild member, you can go to the briefing notes for this week’s session at MEC200 [08-02-23] to see how Flint McGlaughlin used artificial intelligence to build customer profiles. Just scroll to the bottom of the page and click on the names of people we conducted group coaching for in this week’s LiveClass.

Related Resources

AI Marketing Prompts: 14 prompt examples marketers and entrepreneurs found most useful

MECLABS AI Guild

Wednesdays at 2 pm EDT we hold an AI Guild briefing. Chat with MECLABS AI if you would like to register to attend.

In the session, marketers and entrepreneurs can ask questions in the webinar chat. And we answer a few of them right here…

What is an example of a “what will you give” when you are looking to simply get them on a sales call? If you have no freemium model, is it typically a white paper?

This question came up while we were discussing the Customer-First Objective tool available to AI Guild members as they design their offer.

I hope the questioner doesn’t mind if I challenge the basis of the question. Because it gets to the very fundamental reason the Customer-First Objective tool can be helpful – no one wants to be sold, they want to be helped. If you start with helping people, if we all start there, it can help open all our minds to help more people, and in so doing, ultimately bring them towards the conversions we need for a profitable enterprise.

However, if we start with shoehorning people onto a sales call, then we are more likely to lose people who could potentially be ideal customers.

As Bill, an AI Guild member, said in the chat, “I’m wondering who actually desires a ‘sales call.’ What might actually be desirable that you can give them without pressure or obligation?”

This isn’t marketing morality, it’s a simple fact of life. And I guarantee you act the same way when you are a customer. You’re not excited to jump on a sales call. You want help so you can meet your goals and overcoming your pain points.

This gets to a fundamental challenge for why ads, landing pages, and other marketing can underperform. It isn’t necessarily that the design doesn’t pop enough, or that the wording isn’t exactly right in the body copy. It’s that the framework all that marketing is built on – the offer – just isn’t serving a potential customer.

It is really difficult for marketers and entrepreneurs to see this, though. After all, they built the thing. Or they are pouring their life energy into the company to market it. Flint McGlaughlin called while I was writing this, and he told me the following when we discussed the Customer-First Objective tool, “We’re not just trying to help people get an offer, we’re helping them face a hard truth before the marketplace does it for them, before they learn through their data cycles [of ad and landing page testing].”

If you want to build a SuperFunnel, you start with a customer-first objective. You craft a customer journey that begins with a beneficial offer to the customer. On this path, you will help them and build trust. And the ideal customer will get to the point where they want to get on a call with you.

Now this is easier said than done, of course. So here are some examples of engaging the customer earlier in their journey even if you don’t have a freemium model. Sometimes, as the questioner mentioned, it is a white paper. But it could also be:

• A price drop email, which engaged ‘lost leads’ better than a straight-up promo email for an apartment rental agency. See quick case study #3 in Effective Audience Engagement Strategies: Real-life examples from social media and email marketing
  
• A sweepstakes, which got more opt-ins for a synthetic ice company. See quick case study #3 in https://marketingsherpa.com/article/case-study/marketing-strategies
  
• A five-day free workshop for a business academy. See quick case study #3 in Online Business Strategy: Steps marketers took to create powerful marketing funnels with Google, Amazon, and Facebook

I hope you’ll see these as realistic examples. None of these three dropped business considerations entirely out the window and took a Pollyana-ish approach to helping customers. Nor were any of these as customer-first as they could be.

But they all moved closer to helping the customer and away from only focusing on the immediate promotional objective they had. Tiny steps, tiny steps. But tiny steps in the right direction.

To help you generate your own ideas, here are 54 elements that can help you guide your buyers’ journey through the marketing funnel.

And as I said, this is hard stuff. So here is a simple thought experiment you might try to help you reframe how you approach customers.

Pretend you’re out to dinner with a dear old friend and their spouse, catching up after many years apart. When they find out what you do for a living, it turns out they want to make a purchase in your field. However, they simply cannot purchase from you because you don’t serve their region. So you can’t sell them anything.

How would you help them? What would you do? If you sell bathtub refinishing, could you put together key trips for finding the right vendor? Give them guidance on how to check the installer’s license and insurance, how to understand if the warranty has value, mistakes to avoid in project planning, etc. Not to sell them and guide them only to your company, just to help them.

If you sell website hosting services, could you build a simple, free (or low-cost) tool that pings their cellphone the moment a denial-of-service attack is detected?

What could you do to help your friends?

And then – ok, super cheeseball alert here – don’t look at all those people in your ideal customer set as potential sales calls. Look at them as human beings, fellow wanderers on the journey, that you can help just like you helped your friends.

Fair warning though – these people will not know and trust you instantly, like your friends do. So once you make this internal shift and start offering something ‘free,’ you might get very excited. And assume just because it’s free you will get a lot of takers.

Sorry to be a Debbie Downer, but you likely won’t. Your free offering still needs a powerful value proposition. Too much to get into on that topic in this post, but feel free to read – Selling Free Content: Why Seth Godin never gives anything away for free.

Is the forum on LinkedIn? How do I find it?

You can join the MECLABS Super Funnel Research Cohort group on LinkedIn to discuss these kinds of topics. And you can join us Wednesdays at 2 pm EDT for an AI Guild briefing.

Every Wednesday, we hold a free Marketing LiveClass as part of ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel. Everyone is welcome to join and learn, as we build marketing funnels with members of the MECLABS SuperFunnel Research Cohort.

In the LiveClass, marketers and entrepreneurs can ask questions in the webinar chat. And we answer them right here…

Next steps – I have been testing big differences – how do we go deeper once we get some success?

This questioner is asking about testing big differences between a control and a treatment in a marketing experiment.

I’ll give you four options to decide what to test next. The option that works best for you will be based on the unique needs and strategy of your business. It should also be informed by what you learned in the original test. That is, after all, the main point of marketing experimentation. To learn about our customers so we can better serve them.

Continuously move through the optimization sequence

The MECLABS Optimization Sequence Heuristic is…

Opr = Optimize Product Factor
Oprn = Optimize Presentation Factor
Ocnn = Optimize Channel Factor

This means you should optimize your product before the presentation of the offer and optimize the channel last.

And if you think about it for a moment, it should make logical sense. If you optimize the channel first, you’re going to spend a lot of money sending traffic to an underperforming landing page presentation of your offer. And if you optimize the presentation first, people will be buying an inferior product that does not have a forceful value proposition.

This is the process we followed in the MECLABS SuperFunnel Research Cohort, without explicitly saying so. While working with the cohort and building their funnel, cohort members were challenged to identify how powerful their product’s value proposition actually is in the competitive marketplace, and how to present the offer. Then they tested in the channel.

I do want to note, this of course is not a stagnant process. Optimization is continual. So once you’ve gotten to the point of optimizing the channel factor and have a profitable flow going, you can circle back to the product and see how to further optimize it.

I also want to note that we break out these factors to help communicate information. In real-world testing, things can get messier and there can be overlap. For example, you can test in the channel to learn about your product.

Test a new place in your funnel

When I think of testing and optimizing a funnel, I think of a classic physical comedian like Charlie Chaplin or Lucille Ball. There is a long rope they need to keep straight, and wherever it droops they need to prop it up. Great, they’ve got it propped up the biggest drooping area. But oh no, now it’s drooping on the other end. Time to prop it up there. I can picture them comically running back and forth.

The same is true for your marketing funnel. You can look at your metrics to determine where to test in your funnel and try to prop up the most drooping area first. Where is your next biggest challenge?

Did you get more people from the landing page into your cart? That’s great. But maybe they are less qualified and now your cart isn’t converting as well. That is your next place to test.

Or maybe you started with the cart or form and improved conversion there. Now it’s time for a test to get more people to that optimized conversion point.

Move to another problem, solution, or tactic

When you first set up your experiment, you likely brainstormed many problems you might want to address, many solutions to those problems, and many tactics you can implement.

Based on what you learned in this test, what problem, solution, or tactic should you test next? There is a nice visual in Marketing Strategy: 4 steps to developing an effective and strategic test that illustrates this concept. When you look at that visual, you can see that you can ‘move over’ to the next problem, solution, or tactic with your next test.

Go deeper

If you tested big differences, you may not know what element of these big differences actually caused the change. For the next test you can go deeper to get an even better understanding of your customer. (Keep in mind though, the downside of this approach is why you tested big differences to begin with – if you get too deep, you may not find a statistically significant difference between the control and treatment).

For example, let’s say you have a shampoo bar. Maybe you tested the environmental friendliness of the shampoo bar versus the level of cleanliness the shampoo bar provides.

Let’s say, environmental friendliness won. But what aspect of environmental friendliness? Was it the lack of chemicals? Or less wasteful packaging?

Another way to go deeper is to test different expressions of the winning treatment. The winning treatment helped you identify an essence that was more effective with the customer – environmental friendliness. Now, you can test which expressions work best – which headlines about environmental friendliness, which images, which CTAs, etc?

Here’s one more way to go deeper – zoom into subsets of customers. You found the generally most appealing offer to the broadest set of potential customers. Now segment by, let’s say, US customers versus European customers. Start with your highest priority/largest segment and go from there. Which offer is most appealing to them? Which expression of the offer? And on and on.

Plan ahead for your testing

The above approaches don’t only have to happen after the test has run its course. In fact, you can consider many different tests when you begin, and then set a sequence for what to test next based on the results of your tests. ‘If the control wins, that means X about the customer, so next I will test…’ ‘If the treatment wins, that means Y about the customer, so next I will test…’

To help you prioritize your tests, you can use our free Test Planning Scenario Tool.

Are the MY’s constantly being optimized based on efficiency and conversion?

The MY’s are the micro-yeses, the sequence that your customers go through when making a decision about your conversion objective. You can see a discussion of the micro-yeses in Website Wireframes: 8 psychological elements that impact marketing conversion rates.

And to build on the previous question, yes, you should constantly optimize how you handle the micro-yes sequence throughout your funnel and on your landing page, based on what you are learning from your tests and other ways you are measuring customer behavior.

Essentially, you should ask – ‘am I serving potential customers, and the questions they will consider on their micro-yes journey?’ Your tests will help you answer that question and conversion rate performance will as well.

But also listen to their feedback directly to help you understand what you are seeing in the numbers. Talk to them. Give them an easy way for them to contact you. That qualitative information will help you make sense of your conversion and testing data.

This is where followup testing can help as well, and you can always bring something you learned somewhere else in the funnel back to your core – your landing page (I discuss the centrality of the landing page in Conversion Marketing and Landing Page Optimization: Don’t overlook the center of your marketing investment).

Learn something interesting in a channel? How does that affect your landing page, and what followup tests will you run on the landing page to get the most juice for the squeeze?

Am I a free guest here watching a course that these folks paid for? How did they get access to this Q/A database?

You can RSVP now to be a free guest and learn from a Marketing LiveClass on Wednesday at 4 pm EDT as well. Here are some excerpts from recent LiveClasses to give you an idea of what you can expect…

Probably the most frightening thing I have seen in AI development

3 things you can do if your test data is not actionable

What does a good offer look like?

Every Wednesday, we hold a Marketing LiveClass as part of the AI Guild. Everyone is welcome to sign up for a free trial to the AI Guild so they can join and learn from these sessions, as we build marketing funnels with members of the AI Guild.

In the LiveClass, marketers and entrepreneurs can ask questions in the webinar chat. And we answer them right here…

Are we supposed to be framing this like we are talking directly to the customer? Like a headline? Or are we speaking of our customers as a third party?

This question is about how to write a value proposition.

MECLABS defines a value proposition as being the answer to the question, “If I am your ideal customer, why should I buy from you instead of any of your competitors?”

So you are answering a question the customer is asking (in their minds), and the first word in your answer should be “Because…”

You are not speaking about the customer as a third-party. You are speaking directly to the customer.

But the value prop is not a headline. The headline is the articulation of the value proposition. This is directly answering the customer’s question.

To do that, you need to figure out how to increase your customer intimacy and your customer wisdom. If you need to answer the customer’s question, you better understand what the customer wants and what your product delivers. This will help you optimize the communication of the value proposition, or make you realize that you need to create more or different value in the product itself.

To get you thinking for how to find those answers, here’s an example from Radhika Duggal, Chief Marketing Officer, Super…

“We literally stood up for research discipline, doing professional-quality quantitative market research and maybe mediocre-quality qualitative research, in-house producing five to six studies a year simply because we wanted to make sure as we were innovating, we were creating the right products. And we did this with the agreement from the product team that understanding and hearing from our customers really did matter.” (from Consumer Financial Services Marketing: Your customer is your most important stakeholder – podcast episode #39).

Drill down to micro niche?

To have a forceful value prop, the value proposition needs an only factor – a combination of appeal and exclusivity. The less tightly your ideal customer is defined, the harder it is to have that only factor.

For example, a cohort member had an ad and landing page that focused on business consulting. It would be very hard (but not impossible) to have an only factor for such a broad customer set.

However, he conducted competitive research with Bing AI using a methodology taught in the cohort and discovered the lack of high-quality support available to medium-sized businesses in the construction industry.

That is a much more tightly focused ideal customer set (I supposed some people might call it a micro niche), and therefore, it is far easier to identify and communicate an only factor and have a forceful value proposition.

To get more ideas for identifying your ideal customer, you can download 7 Steps to Discovering Your Essential Value Proposition with Simple A/B Tests.

Is it too early to join if you have not determined who your target audience is?

No, it is not too early to join the MECLABS SuperFunnel Research Cohort if you have not identified your target audience. In fact, using the thought tools in the cohort can help you determine your target audience (as discussed in the example above).

Another example is determining if your target audience for a landing page is a prospect or previous customer. This can affect what value proposition you feature on a landing page or in an offer. (Get more ideas on that in Value Sequencing Decider Graphic: What do your customers need to know, and when do they need to know it?)

Container is the quiz?

Your offer will tend to have two parts – the container and the content. For example, I consulted with a brand at IBM to help with their sales enablement early in my career. We brought sales information and case studies to a globally distributed sales team in an email newsletter.

But we realized that this was a highly mobile audience, and they wanted to spend as much time as possible with clients and potential clients. They didn’t want to read an email newsletter.

We changed the format and delivered the content through a podcast. This way, they could listen during their downtime while they were traveling – in the car, an airplane (before there was Wi-Fi on planes, and when laptop use was limited), etc. So it wasn’t taking time away from clients (traveling was downtime anyway). And they had a lot of this travel time to use up since they were highly mobile.

The most appealing container for your ideal customer – like in this question, a quiz – can help pull them into your landing page and your offering.

Of course, the container is just part of it, For more ideas on pulling your ideal customer into your offering, watch Above-the-Fold Energy: How to engage the prospect’s mind with a carefully crafted opening.

Possible to get a link to the 54, uh, things?

To help cohort members come up with ideas for their offer container, in the LiveClass we shared this article – Funnel Strategy: 54 elements to help you guide your buyers’ journey through the marketing funnel.

In the LiveClass chat, a cohort member shared a brilliant way she was using AI to get the most out of the article. I share it here if you would like to use her idea as well. “I copied those 50 and asked ChatGPT for an idea for my audience for each of those … good brainstorming exercise. Even got it to output in a table so it pastes into a spreadsheet …”

How does the shorter landing layout relate to the layout diagram that we were following?

The layout diagram refers to the MECLABS Landing Page Blueprint. The blueprint features eight micro-yeses you should achieve from the customer.

But just because you need to win these eight yeses, does not necessarily mean you need to have a long landing page.

For example, one of the micro-yeses you have to secure is “Yes, I believe.” The amount of information you need to win this micro-yes will vary based on what you are asking them to believe. If you are going to send them a totally free sample that does not require a credit card, it will take less to get them to believe than if you are asking them to sign up for a high-dollar, recurring payment.

Another factor to consider is where they are in the customer journey when they reach your landing page. For this reason, you should also look at those micro-yeses across your entire funnel.

Here is an example from a classic experiment. A migraine treatment center had a lead form on its homepage. The MECLABS team hypothesized that the value exchange was occurring too early in the funnel. The customer had not said yes for all of the micro-yeses at that point.

So they created a treatment that communicated to the three major lead types on the homepage. It provided enough information to get the ideal customers to come to the conclusions “Yes, I will pay attention” and “Yes, I will engage deeper” and then taking customer type to a separate landing page that tied into their individual motivations so the page could answer the rest of their micro-yeses. The experiment produced a 331% increase in lead rate. You can see the experiment in this classic MarketingExperiments web clinic recap – Homepage Optimization Applied: Learn how to replicate a 331% lift on your own site.

If we need a long page for all 8 micro yes, which do we focus on for the short page?

Every landing page, in fact every communication you have, should address all 8 micro-yeses.

And to take it a step further, every landing page should have each of the four levels of value proposition (as discussed in Copywriting and Value Proposition: Unleashing the power of compelling copy).

The magnitude and nature of the decision, along with where they are in their customer journey (which your funnel should be modeled to), will determine how much you need to communicate about those elements to get those micro-yeses. Sometimes it doesn’t take much, and you better serve the customer by getting out of their way (removing friction).

To give an extreme example, let’s look at two purchase decisions and see how this plays out in the real world.

First, consider buying a stick of gum in the checkout line at a grocery store (an impulse buy). This is in a reputable store, and I have heard of the brand before, so I instantly understand and believe. I know I like original more than grape flavor, so the product-level value prop is instantly clear to me.

Now, consider the purchase of an automobile. Do I understand and believe that this car will be the right one for me? Is the company’s primary value prop forceful to me? And even if it is, which product-level value prop (car model from that company) is right for me?

To answer these micro-yeses and find these different levels of value proposition appealing, the customer may go on a long journey. They may read Consumer Reports magazine to understand the reliability and performance of different car companies as well as individual models. They may talk to a friend with an electric car to see if they are ready to make the switch from gas. They may test drive several cars at different dealerships. It will take a ‘long page’ (and funnel) to help them answer the micro-yeses and find the different levels of value prop appealing.

Theoretically, we could have all 8 MY’s (micro-yeses) in a single complex sentence, no?

Great observation. We might even call that an elevator pitch. At which point your goal isn’t to get the final sale (that would take a longer page), but just win the conversion of getting someone to want to learn more.

Hello, great to be here! I’m wondering how one gets access to this workspace?

The workspaces are for members of the MECLABS SuperFunnel Research Cohort. If you stick around after a Marketing LiveClass, we will tell you more about how to join. You can RSVP now for a LiveClass and join us on Wednesdays at 4 pm EDT.

Every Wednesday, we hold a free Marketing LiveClass as part of ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel. Everyone is welcome to join and learn, as we build marketing funnels with members of the MECLABS SuperFunnel Research Cohort.

In the LiveClass, marketers and entrepreneurs can ask questions in the webinar chat. And we answer them right here…

Hi Dan, I’m trying a new angle on copywriting for one of my products. If you have time to give feedback, I would really appreciate your wisdom. I also know you’re super busy, so no pressure. I also know you’re the master of copywriting, so your guidance is invaluable. Thanks for all your wisdom. I’ll load it into Notion later today. Here’s the link: [anonymized]

Here’s the CFO: To help motivated volunteers in the church boost their ministry effectiveness and gain spiritual confidence by giving them 4 theology/ministry courses (“Advanced Bootcamp”) in exchange for a cash payment and a significant investment of time.

Thanks for the vote of confidence. Let’s see if I can help.

First, let’s define copywriting. I define copywriting as ‘Helping the customer come to the best decision about a brand, product or conversion goal.’ (from Marketing 101: Copywriting vs. copy editing vs. content writing). And this person had taken a step in that direction, starting with a CFO (customer-first objective).

I don’t think it would be practical or fair to paste their entire page in here, but I also don’t want you to focus too much on what his copy says, but rather understand the principles of how you can optimize your own copy.

I also cannot definitively say whether the copywriting will be successful or not, because I am not the landing page’s ideal customer. But I think I can best help this person (and you) by calling out how the different levels of value proposition should appear on a landing page.

The landing page should primarily focus on one of the levels of value proposition. But for most landing pages (and certainly this landing page) all four levels should be present, usually with the other three levels supporting the main one you are focused on. (If you are unfamiliar with value proposition levels, you can read Customer Value: The 4 essential levels of value propositions).

So let’s look at those levels:

Product-level value proposition

This page is focused on the Advanced Bootcamp, which consists of the courses. The page clearly informs people what they are getting (“4 precisely selected courses”), has a section for each course that explains what it will enable the participant to achieve, discusses the elements of the course (video lecture, downloadable charts, etc.).

You’re probably doing this pretty well on your product-focused pages as well, so I won’t dwell here. You likely understand the basics of communicating what is involved with your product.

Prospect-level value proposition

According to the customer-first objective stated above, the prospect this page is focusing on is “motivated volunteers in the church.”

However, the tone of the copy is not what I would expect for that audience. “You’re a happy little Christian gerbil,” “It’s too hard. You don’t have enough time. Your brain will hurt. It costs too much. You’ll want your mommy,” “Fix Your Boring, Lame, Mediocre Spiritual Life.”

Again, I am not the ideal customer, but my best guess is this copy will turn off more of the ideal prospect than it will attract.

Remember, it’s not just what you say with your copy. But how you say it.

So one of two things has to happen. Either the Customer-First Objective has to change with a clear definition of exactly what subset of those motivated volunteers the copy will attract.

Or the copy needs to change so the tone speaks to the desired prospect.

If your prospect stays the same, what is the resulting experience of the tone of language like “G-d, , Jesus, Church, blah, blah, blah?” Does this emphasize the importance and relevance of spirituality in the ideal customer’s life? Would replacing phrases like “treadmill” and “happy little Christian gerbil trotting endlessly nowhere” with more positive and empowering language better speak to the ideal customer?

In the case of this page, I suspect much of this is intentional and the result of some deep thought. As the questioner mentioned, he is “trying a new angle.” But I would argue this is much more than a new angle. This is changing your ideal prospect. And so it is worth the time to change the Customer-First Objective, and force yourself to first clearly define who exactly the prospect is that you are trying to serve with this copy.

With that exercise, a few things may happen:

• It may help you better target your ads
• You may realize there is not a big enough total addressable market
• You may identify new ways to reach that audience
• You may find new avenues for messaging

I’ll give you an example. Right now, the product is called an Advanced Bootcamp. And the primary visual is a big black boot. However, that title and that visual connote to me the military. And the military is known more as a group that can strictly follow a rigorous process without dissent.

But the way the copy is written, I would define the ideal prospect as “demotivated Christians who haven’t found the right church to volunteer for, have a deep soulful connection to the faith but feel disconnected, cast out, and overlooked by church doctrine and/or communities.”

Now this might connote a different name for the product. How about the Faith Rebellion Experience. SoulFire Quest. Spiritual Reboot. I’m not sure any of these are the right names. But having a clearer definition of the prospect, we can better tailor the name – a key element of the copywriting – to connote that this is a product that is for people like them.

Process-level value proposition

The CTA buttons have a similar tone. “Let’s Kick the Devil in the Teeth. Clicking can be hazardous to your apathy.” “Get Off the Couch and Into the Battle. CAUTION: Clicking Here Will Wreck Your Excuses.”

These are evocative.

What they aren’t is clear.

You may get a curiosity click. But the challenge is, the next page is not something that would pay off clickbait. It’s a cart page (with supporting value in the right-hand side) that has a form for credit card info and a purchase.

It also has a line that says, “You have a free 24-hour inspection period. After that, your card will be charged.” So maybe you could change the CTA to “Get Free Preview” and change “inspection” on the cart to “preview.”

A few other thoughts to help this particular questioner as well as other readers:

• The buttons don’t look like buttons. Make sure your buttons look like something that can be clicked on. Compounding this problem, they have really long CTAs. For example, the first CTA is 15 words long. I was a little confused on where I can click. These looked like pull-quote boxes to me.
• Above one of the CTAs, we see four markdowns on the price from $16,500 to $387. This strains credulity. If the markdown is really this big, it needs a pretty compelling explanation for why. Otherwise, this will feel like a hype-y sale, not like help. Nobody wants to be sold, they want to be helped (look back to the definition of copywriting at the beginning of this blog post). Especially for an educational and faith-based product like this, trust and the feeling of not being sold to are essential.
• Is this even the right process-level value proposition? If you were selling socks or cameras, a process-level value prop focused on adding to cart and putting in a credit card number might make sense. But this is an online course. You do give a free 24-hour “inspection period.” But could you give the first full session for free? You have a full page of hype-y copy explaining how this is going to shift my paradigm, so let me actually see it!

As Anton Chekhov said, “Don’t tell me the moon is shining; show me the glint of light on broken glass.” The best copywriting shows, it doesn’t tell. And the best way to show the value is not with sales copy, but by giving them a taste of the actual experience (when possible).

Primary value proposition

What organization is standing behind this product? Why should I believe they will deliver on all these promises?

Especially in this case when the task is so big – to challenge one’s approach to their religion.

There are typical, good evidentials on this page – “40+ years, written over 27 books.” (written over 27 books? So… is that 28 books?)

But is there enough primary value prop there to get someone to act? If you are challenging their approach to their faith, shouldn’t we know more about why you are uniquely qualified to do this? Why the ideal customer should trust you?

This is a deep, and difficult, question to answer. And it will take more than simple evidentials. Frankly if you can crack that, you will likely unlock many insights that will help your organization better communicate its value to potential customers.

In addition, the footer of the page has no email address, phone number, physical mailing address, links to social media accounts, nothing that would help me build trust in the organization behind this offering and let me know it isn’t some sort of scam.

I appreciate today’s clarity, so let me expand a bit because I feel you’ll be able to provide a bit more depth on it. I also provided a super quick overview of exactly what they get out of the offer (60-minute consult). The rest is mainly a discussion on what I asked at the very end with my own struggles in intelligence, competency and meshing it all with AI.

This SuperFunnel cohort member is using some of the thought tools included as part of the program to help discover the most effective value proposition for his offer. By interviewing customers and reviewing competitors, he uncovered some elements of value he didn’t realize when he came up with the value proposition on his own, using his gut.

Getting this outside perspective can help us challenge our own assumptions and unlock value that truly matters to the ideal customer when communicated on our websites.

You can do this by interviewing potential customers. But also, what feedback are you already collecting in your organization and how can you systematize it? How can that better inform your primary value prop, but every level of value prop as well…including the process-level value proposition of your website’s usability. Here’s a quick example – “In our customer service group, we found that 50% of their calls at times were based on ‘I can’t find my order status,’” said Matt Clark, Global Head of eCommerce and Digital Marketing, Newark Element14 (from Customer-centric Marketing: How market research and listening to customers informs website optimization).

But when you get this outside research and go through this corporate soul searching, you end up with… a lot. And then when you add AI to the mix to help with competitive analysis, you have a lot more to work with.

At this point in the process the element of the value prop we need to focus on is clarity. We need to winnow, pare down, simplify, condense. There shouldn’t be one unnecessary or unclear word in our offer value proposition statement. As Antoine de Saint-Exupéry said, “Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.”

And this is the struggle facing this particular cohort member. He’s doing some smart and impressive stuff. And in his Customer-First Objective, he discusses what the customer can get out of the offer, but he doesn’t carry that through to his offer value proposition. For this reason, I question if his value prop is clear enough. For example, “streamlines processes through augmented intelligence systems.”

While the value prop isn’t the direct language in your marketing, I worry language like that isn’t the sharpened tip of the spear this business owner needs to clarify his marketing. What does that stuff do for the customer?

He opens the value prop by talking about “increases profits and reduces expenses.” But does that really clarify anything? Really, what B2B product or service doesn’t help a business owner increase profits or reduce expenses?

However, he knows how to winnow, simplify, and clarify for his customers. His main deliverable takes a lot of different data and puts it into a simple dashboard (it’s so much harder to simplify communication about our own work).

So how could he answer the offer value proposition question, “If I am your ideal customer, why should I act on your offer rather than offers from any of your competitors?”

That deliverable may be the tangible piece he leads with. “Because [ideal customer described] will get a dashboard that simplifies complex data needed to make key business decisions that affect costs and profitability. This dashboard is the only [ideal customer] tool powered by [a very high-level explanation of the business process here, supported underneath by evidentials]…”

This isn’t exactly right, of course. And may be way off. But it’s an example of how we can add the tangible to our value props and clarify the value the customer will receive…which is especially difficult for a complex offering.

Should the OVP include specific evidentials?

Yes, your offer value proposition should include specific evidentials. Credibility is one of the elements of a forceful value proposition. Every claim you make should be verified. Or else, why would your ideal customer believe it?

That said, to keep the value prop clear and simple (as discussed above) the best way to use evidentials is with footnotes. Put a superscript number by each claim that needs to be verified in your value prop, and then include that number below the value prop with the evidentials supporting it.

To give you ideas for evidentials for your value prop, you can read my former colleague’s attempt to create an evidential for a hypothetical car dealership I might own in Value Proposition: 3 techniques for standing out in a highly competitive market.

Question for Daniel’s next article: What role does SEO play in our current climate? We’re talking about paid ads… is that the best way forward? Thanks.

I’m sure by now you can see I love quotes. A great way to learn from the wisdom of the ages. So let me remixed Robert Frost who said, “The best way out is always through” to tell you that “the best way forward is always through…the customer’s eyes.”

Because the customer should be your focus. Yes, the current macro-climate is important. But the most important climate is the micro-climate. How do they want to receive information? Where do they look for answers to meet the pain point you are addressing or goal you are trying to help them achieve?

And don’t just put yourself in an SEO-or-paid-digital-ads box. Consider every conceivable option.

For example, our own research has shown that consumers trust print ads in newspapers and magazines more than any other advertising channel when making a purchase decision (see Marketing Chart: Which advertising channels consumers trust most and least when making purchases). Or as The Wall Street Journal reported, rising digital-advertising costs are leading many brands to open old-fashioned physical stores (see Digital Rebels Want Real Stores Now.)

So the real answer isn’t the current climate. The real answer is hammering out a go-to-market model for your business. For ideas on that, you can listen to Episode #58 of How I Made It In Marketing. Shruti Joshi, COO, Facet, describes her GTM approach at Verizon and at her current organization (listen to Marketing Operations: Process is the foundation for success).

You are welcome to join us on Wednesdays at 4 p.m. EDT to watch and learn from a Marketing LiveClass. You can RSVP now by clicking this link. Here are excerpts from recent LiveClasses to give you an idea of what you can experience…

Be Passionate about the Marketing Challenge You Are Trying to Solve

Chris Berkenkamp Talks about the Most Important Things He Learned from the MECLABS Cohort

Every Wednesday, we hold a free Marketing LiveClass as part of ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel. Everyone is welcome to join and learn, as we build marketing funnels with members of the MECLABS SuperFunnel Research Cohort.

In the LiveClass, marketers and entrepreneurs can ask questions in the webinar chat. And we answer them right here…

What does it mean that the control peaked early and did not recover?

This question is in regard to A/B test interpretation and validity. In other words, what do the results of our marketing tests mean? And can we trust them?

I’m going to get to the actual question in a bit, but first let me address the chart that was presented in the LiveClass that sparked this question.

I think the question was in relation to this chart…

This chart does not show conversion rates over time. It doesn’t mean that the control got a lot of conversions, and then the treatment got some (and to be clear, it is actually the treatment – labeled Variation #1 – that is showing the peak on the left).

The chart shows the Bayesian probability density of the conversion rates. For this experiment, the treatment had a conversion rate of 0.24% and the control had a conversion rate of 1.01%. Without getting too deep, when you run an A/B test, you are getting a sample of the whole population.

Think of a drug trial as an example. You can’t give the drug to every possible person who would ever take it, that is the whole population. So you use a sample, and extrapolate the results. The same is true with a marketing experiment.

And as for that sample, you want a large enough sample size that you can confidently make decisions based on it (if getting a large enough sample size is a challenge for you, this might help – A/B Testing: Working with a very small sample size is difficult, but not impossible).

Looking at this chart, you could have a high degree of confidence that the control is better than the treatment because there is no overlap. And indeed, the tool gave the control a 100% probability of being a winner (probably rounding up). The treatment has a big peak around 0.24, and the true conversion rate is probably pretty close to that. The control’s true conversion rate is probably around 1, but the distribution is flatter so it may be between 0.8 to 1.2.

Those results are quite extreme though. Here is a more common distribution…

The probability of the treatment being a winner in this case is 90%. As you can see, most of the red indicates a likely higher true conversion rate than most of the blue, but there is some overlap (we’ll call that purple).

However, let’s get to the actual question at hand, because it is a good one – what does it mean if the control peaked earlier than the treatment and did not recover? That would probably look something like this…

As Egon Spengler said in “Ghostbusters,” “Don’t. Cross. The streams. It would be bad.”

When we see results cross like this, it should raise a red flag. Is there a validity threat?

Ideally, we would want our results to look like this…

This would give us more confidence that the Treatment is actually better than the results.

Which, again, brings us back to our question – but why did this happen? Here are a few validity threats you could check for:

• History Effects: Did anything happen due to the passage of time to influence results? Perhaps you launched on the weekend giving the control a huge boost, but the treatment performed better during the week.
• Instrumentation Effects: Did anything happen to the technical environment or the measurement tools that could significantly influence results? Perhaps that boost to the control was due to bot traffic.
• Selection Effects: Did anything happen to the testing environment that may have caused the nature of the incoming traffic to be different? For example, maybe someone else in the organization sent out an email promo on that day with an incentive that is only on the control landing page.

Now that we’ve discussed how to test, let’s get into a question that addresses what to test…

What does uncouple the value from the mechanism mean exactly? I’m not sure I understand.

Uncoupling the value from the mechanism means separating the value that a product or service provides from how it is delivered or produced. By focusing on the value that the product or service provides to the customer – and not on how it is delivered – we may discover new ways to deliver value to our ideal customer.

For example, let’s say you write a book filled with exercises that relieve knee pain. Is the value that is most compelling to potential customers the book itself, or the information in it? While some of your potential customers might enjoy a book, perhaps others would rather receive the information in a video. Or a podcast. Or through an AI tool or an app. You may have different groups of potential customers who would like to receive the value in different ways, so multiple formats might attract more business.

Now, sometimes the mechanism is in fact part of the value itself. For example, I love reading print newspapers and magazines. I’m on a digital device all… day… long. So reading a print newspaper with my morning coffee, or sitting out on a dock overlooking the river with a print magazine, these are customer experiences I crave. I could get the same information on a digital device, but that mechanism dilutes the value for me.

This topic is closely related to value decoupling. Is the way you have packaged and are selling the value your company delivers the most appealing way to do so?

An example I always liked was an insurance company that sells insurance for products only when you want it. The traditional option is – I buy a smartphone, and then I buy insurance for that smartphone with a monthly payment for as long as I own it. Instead, this insurance company decoupled that value into single days of insurance, and let you buy insurance just when you want it – just the two weeks you are going out of the country and worried about losing your phone. You can read more about that topic in – Mobile Marketing and Value Decoupling: Interview with Harvard professor about eight years of research into business disruption.

Uncoupling the value from the delivery mechanism, and scouring your company for value decoupling ideas, should produce some interesting marketing experiments. And I’ll end this question with a quote on this topic from a recent interview – “Be passionate about the challenge you are trying to solve and not stubborn about the product solution.”

That quote is from Andrew Beranbom. His episode of the How I Made It In Marketing podcast will be released Tuesday, May 16^th.

CFO?

CFO stands for customer-first objective. You will be far more effective at value decoupling if you start with a customer-first objective. Learn more in Customer-First Objectives: Discover a 3-part formula for focusing your webpage message.

So the CFO should be focused on a funnel and specific offer in the funnel, not the overall business, correct?

While you should have a customer-first objective for everything you do, for ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel our focus is on creating CFOs for a funnel.

“That old conversion heuristic” There isn’t a new one, right?

When Flint said that in the LiveClass, he was referring to the patented MECLABS Conversion Sequence Heuristic. While there is not a new heuristic, there are many new thought tools as part of the MECLABS SuperFunnel Research Cohort.

How do we get feedback on our first assignment?

Members of the MECLABS SuperFunnel Research Cohort get feedback as they build their funnels in the Marketing LiveClasses. Cohort members can also get feedback in your small group meetings, in the LinkedIn Group, and in occasional one-on-one coaching. And of course, you can always email us.

You are welcome to join us on Wednesdays at 4 p.m. EDT to watch and learn from a Marketing LiveClass. You can RSVP now by clicking this link. Here are excerpts from recent LiveClasses to give you an idea of what you can experience…

We are trying way too hard to give away something for free

How do you think AI’s capability will change the thrust of our work as a marketer?

Every Wednesday we hold a free Marketing LiveClass as part of ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel. Marketers and entrepreneurs ask questions in the webinar chat. And we answer them right here…

Do you see it to be an issue that we’re all designing first for desktop and analyzing pages on desktop, but the majority of traffic will likely be on mobile. Is it a best practice to design for mobile first?

It is a best practice to put the ideal customer first. If the ideal customer is more likely to be using a mobile phone, then yes, start with a mobile landing page. Even better if you focus on the most popular device size (if you have a website already, you can find this info in your analytics).

But don’t stop there. How else can you put the customer first?

• Speed first – how fast is the connection where they are located? Some nations and areas have faster connections than others. If they are in an area with a slow-loading page, you should start by stripping the page down to its essence and only adding an element if it’s absolutely necessary.
• Contact-preference first – Phone? Email? Chat? Social media? How do they want to contact your company? You should likely provide several options, but for a landing page that revolves around an action, make sure you are starting with the action most preferable to the customer.
• Price first (or not) – Is the ideal customer more likely to want a deal? Or care less about price and more about service or durability? Lead with what is most valuable to them by, for example, including a coupon code front and center.

These are just some ideas to get you thinking about how to put the customer first. And if you do have an ideal customer that is likely to be on mobile, this article may help give you some mobile-specific ideas – Exploring the Mobile Customer Experience: Three discoveries for designing an effective mobile experience.

But a word of caution – don’t just design for mobile first because it’s a general best practice. Remember, it’s only a best practice if the majority of your brand’s ideal customers are accessing your landing pages with a mobile device. There are still likely many companies that would better serve their customer with desktop-first (or perhaps laptop-first) designs – for example, B2B companies or those serving older customers.

And lastly – a thank you to this questioner. One reason I hope Flint McGlaughlin and I have been able to add value to participants of the MECLABS SuperFunnel Research Cohort is by bringing the MECLABS methodology and conversion psychology to bear on their funnel, their ads, and their landing pages as we offer conversion optimization and marketing strategy suggestions.

But another reason I’ve brought value is because I’m simply not steeped in their business every day. I have an outside perspective. As Mark Twain said, an expert is just “an ordinary fellow from another town.” Being from another town, I can help point out their blind spots.

The SuperFunnel Cohort community has helped point out my blind spots as well. And this was one. I have provided optimization suggestions to many cohort participants’ landing pages, but I don’t think I ever challenged them to design for mobile first. Thanks to this questioner, I realize that this is a really obvious approach I overlooked.

But that’s how blind spots are, right? You don’t notice them at all. Until one day someone points it out to you, and then it seems breathtakingly obvious.

As an attendee, what’s the best way to get started with AI?

I like that this questioner didn’t ask – “what’s the easiest way to get started with artificial intelligence?” They asked for the best way to get started with artificial intelligence.

And I would say – take a look at the answer above. Don’t begin with AI, just like you shouldn’t necessarily begin with mobile. Begin with the customer:

1. What goals does the customer want to achieve in their life? What challenges do they want to overcome?
2. If you find many answers to Question #1 that have nothing to do with your brand…fantastic! You’re doing it right. You’re laser focused on your ideal customer, not your own self-interest. However, unless you have chosen the wrong addressable market, your brand should be able to help with some of those things. So how can your brand help?
3. Break down your answers to Question #2 into two buckets – ways that require a monetary payment from the customer (this is your “product”), and ways that do not require monetary payment (this is your “marketing”).
4. Now, how can you deliver the value identified in Question #3? This is where you discover what role artificial intelligence can play.

The rest is experimentation. Experimenting with AI tools to see how they can deliver that value. But also, conducting marketing experiments to see if the addition of artificial intelligence is helping you “move the needle” in your funnel. We answer questions about running tests in Marketing Experimentation: Answers to marketers’ and entrepreneurs’ questions about marketing experiments.

+1000 Daniel, when everyone has AI and a bot, who cares that you do?

In fairness, this isn’t a question, so to speak. The participant agreed with something I said in the LiveClass.

But I included it, because it is the reason for the four-step framework I gave to address the previous question.

When you approach your marketing with a technology-first mindset, someone else will always be breathing down your neck. Ready to replicate or outpace your success. Always ready with a better, faster, cheaper technology.

Which is why you should approach these types of decisions – yes, even technology decisions…especially technology decisions – with a value-proposition first mindset. This is how epic brands are built, and how you architect a sustainable competitive advantage.

Here’s a great example. Anyone can put an AI-powered chatbot on their site. But Medieval Times implemented a chatbot that helped communicate the dinner theater’s value proposition. You can read how they did it in – Artificial Intelligence and Machine Learning in Marketing: What marketers (even those who don’t care about tech) should know about AI and ML.

Incidentally enough, we published that article in the Middle Ages of AI – September 23^rd, 2022…more than two months before ChatGPT’s launch as a public prototype on November 30^th. However, in my (biased) opinion, the takeaways discussed in the article are just as relevant today because we focused (less) on the technology itself and more on the humans behind it – the ideal ‘customer’ for our content, marketers and entrepreneurs.

How can I join the next cohort?

How can I join a cohort? Is there info on that? I have purchased MarketingSherpa books and watched videos, but would love active feedback on my new project

At the end of the LiveClass, we answered questions about joining the MECLABS SuperFunnel Research Cohort. Feel free to join us for a Wednesday LiveClass to get ideas for your marketing funnel, and if you stick around to the end, we’ll answer your questions as well.

Here are some quick experts from previous LiveClasses:

Can we put all 8 micro-yes(es) on the landing page?

How do we measure the strategy?

How do we weigh the appeal or exclusivity of a claim?

Here are answers to some chat questions from last week’s ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel. I hope they help with your own marketing efforts. And feel free to join us on any Wednesday to get your marketing questions answered as well.

Am I understanding the message correctly that the main value at first isn’t in more conversions or $$$ but in deeper understanding of the customer mindset?

This questioner is asking about the value of marketing experimentation. And it reminds me of this great movie quote…

Alfred: Why do we fall, Master Wayne?
The (future) Batman: So we can learn to pick ourselves back up again.

Similarly, we might say…

Marketer: Why do we run experiments, Flint?
Flint (the real Batman) McGlaughlin: “The goal of a test is not to get a lift, but rather to get a learning.”

So when you see marketing experiments in articles or videos (and we are guilty of this as well), they usually focus on the conversion rate increase. It is a great way to get marketers’ attention. And of course we do want to get performance improvements from our tests.

But if you’re always getting performance improvements, you’re doing it wrong. Here’s why…

Marketing is essentially communicating value to the customer in the most efficient and effective way possible so they will want to take an action (from Customer Value: The 4 essential levels of value propositions).

So if you’re always getting performance improvements, you’re probably not pushing the envelope hard enough. You’re probably not finding the most efficient and effective way, you’re only fixing the major problems in your funnel. Which, of course, is helpful as well.

In other words, don’t feel bad about getting a loss in your marketing experiment. Little Bruce Wayne didn’t become Batman by always doing the obvious, always playing it safe. He had to try new things, fall down from time to time, so he could learn how to pick himself back up.

While that immediate lift feels good, and you should get many if you keep at it, the long-term, sustainable business improvement comes from actually learning from those lifts and losses to do one of the hardest thing any marketer, nay, any person can do – get into another human being’s head. We just happen to call those other human beings customers.

Which leads us to some questions about how to conduct marketing experiments…

Do we need a control number?

In the MEC300 LiveClass, we practiced using a calculator to determine if results from advertisement tests are statistically significant

The specific tool we practiced with for test analysis was the AB+ Test Calculator by CXL.

I’m guessing this questioner may think ‘control number’ comes from previous performance. And when we conducted pre-test analysis, we did use previous performance to help us plan (for more on pre-test planning and why you should calculate statistical significant with your advertising experiments, you can read Factors Affecting Marketing Experimentation: Statistical significance in marketing, calculating sample size for marketing tests, and more).

But once you’ve run an advertising experiment, your ‘control number’ – really two numbers, users or sessions and conversions – will be data from your ads’ or landing pages’ performance.

You may be testing your new ideas against an ad you have run previously by splitting your budget between the old and new ads. In this case, you would usually label the incumbent ad the control, and the new ad idea would be the treatment or variation.

If both ads are new, technically they would both be treatments or variations because you do not have any standard control that you are comparing against. For practical purposes in using the test analysis tool, it is usually easier to put the lower-performing ad’s numbers in the control, so you are dealing with a positive lift.

Remember, what you are doing with the test calculator is ensuring you have enough samples to provide a high likelihood that the difference in results you are seeing is not due to random chance. So for the sake of the calculator, it does not matter which results you put in the control section.

Labeling a version a ‘control’ is most helpful when actually analyzing the results, and realizing which ad you had originally been running, and what your hypothesis was for making a change.

Which brings us to what numbers you should put in the boxes in the test calculator…

Users or sessions, would that be landing page views? I’m running on Facebook.

In this specific test calculator, it asks for two numbers for the control and variation – ‘users or sessions’ and ‘conversions.’

What the calculator is basically asking for is – how many people saw it, and how many people acted on it – to get the conversion rate.

What you fill into these boxes will depend on the primary KPI for successes in the experiment (for more on primary KPI selection, you can read Marketing Experimentation: How to get real-world answers to questions about a company’s marketing efforts).

If your primary KPI is a conversion on a landing page, then yes, you could use landing page views or, even better, unique pageviews – conversion rate would be calculated by dividing the conversion actions (like a form fill or button click) by unique pageviews.

However, if your primary KPI is clickthrough on a Facebook ad, then the conversion rate would be calculated by dividing the ad’s clicks by its impressions.

Which brings us to the next question, since this tool allows you to add in a control and up to five variations of the control (so six total treatments)…

Can you confirm the definition of a variation really fast? Is it a change in copy/imagery or just different size of ad?

Remember what we are doing when we’re running a marketing experiment – we are trying to determine that there is a change in performance because of our change in messaging. For example, a headline about our convenient location works better than a headline about our user-friendly app, so we’ll focus our messaging about our convenient location.

When there are no validity threats and we just make that one change, we can be highly confident that the one change is the reason for the difference in results.

But when there are two changes – well, which change caused the difference in results?

For this reason, every change is a variation.

That said, testing in a business context with a necessarily limited budget and the need for reasonably quick action, it could make sense to group variations.

So in the question asked, each ad size should be a variation, but you can group those into Headline A ads, and Headline B ads.

Then you can see the difference in performance between the two headlines. But you also have the flexibility to get more granular and see if there are any differences among the sizes themselves. There shouldn’t be right? But by having the flexibility to zoom in and see what’s going on, you might discover that small space ads for Headline B are performing worst. Why? Maybe Headline B works better overall, but it is longer than Headline A, and that makes the small space ads too cluttered.

Ad size is a change unrelated to the hypothesis. But for other changes, this is where a hypothesis helps guide your testing. Changing two unrelated things would result in multiple variations (two headlines, and two images, would create four variations). However, if your experimentation is guided by a hypothesis, all of the changes you make should tie into that hypothesis.

So if you were testing what color car is most likely to attract customers, and you tested a headline of “See our really nice red car” versus “See our really nice blue car,” it would make no sense to have a picture of a red car in both ads. In this case, if you didn’t change the image, you wouldn’t really be testing the hypothesis.

For a real-world example see Experiment #1 (a classic test from MarketingExperiments) in No Unsupervised Thinking: How to increase conversions by guiding your audience. The team was testing a hypothesis that the original landing page had many objectives competing in an unorganized way that may have been creating friction. Testing this hypothesis necessitated making multiple changes, so they didn’t create a variation for each. However, when making a new variation would be informative (namely, how far should they go in reducing distractions) they created a new variation.

So there were three variations total. The control (original), treatment #1 which tested simplifying by making multiple changes, and treatment #2 which tested simplifying even further.

When we discuss testing, we usually talk about splitting traffic in half (or thirds, in the case above) and sending an equal amount of traffic to each variation to see how they perform. But what if your platform is fighting you on that…

One thing I’m noticing is that Google isn’t showing my ads evenly – very heavily skewed. If it continues, should I pause the high impression group to let the others have a go?

It really comes down to a business decision for you to make – how much transparency, risk, and reward are you after? Here are the factors to consider.

On the one hand, this could seem like a less risky approach. Google is probably using a statistical model (probably Bayesian) paired with artificial intelligence to skew towards your better performing ad to make you happy – i.e., to keep you buying Google ads because you see they are working. This is similar to multi-armed bandit testing, a methodology that emphasizes the higher performers while a test is running. You can see an example in case study #1 in Understanding Customer Experience: 3 quick marketing case studies (including test results).

So you could view trusting Google to do this well as less risky. After all, you are testing in a business context (not a perfect academic environment for a peer-reviewed paper). If one ad is performing better, why pay money to get a worse-performing ad in front of people?

And you can still reach statistical significance on uneven sample sizes. The downside I can see is that Google is doing this in a black box, and you essentially just have to trust Google. It’s up to you how comfortable you are doing that.

When you go to validate your test, you could get a Sample Ratio Mismatch warning in a test calculator, warning you that you don’t have a 50/50 split. But read the warning carefully (my emphasis added), “SRM-alert: if you intended to have a 50% / 50% split, we measured a possible Sample Ratio Mismatch (SRM). Please check your traffic distribution.”

This warning is likely meant to warn you of a difference that isn’t obvious to the naked eye if you intended to run a 50/50 split. This could be due to validity threats like instrumentation effect and selection effect. Let’s say your splitter wasn’t working properly, and some notable social media accounts shared a landing page link but it only went to one of the treatments. That could threaten the validity of the experiment. You are no longer randomly splitting the traffic.

On the flip side, if you want more control over things, you could evenly split the impressions or traffic and use a Frequentist (Z Test) statistical methodology and choose the winner after things have run. You aren’t trusting that Google is picking the right winner, and not giving up on an initially under-performing treatment too soon.

I can’t personally say that one approach is definitely right or wrong, it comes down to what you consider more risky, and how much control and transparency you would like to have.

And if you would like to get deeper into these different statistical models, you can read A/B Testing: Why do different sample size calculators and testing platforms produce different estimates of statistical significance?

Flint, do I remember right? 4 words to action in the headline?

This question is a nice reminder that we’ve been answering questions about testing methodology – the infrastructure that helps you get reliable data – but you still need to craft powerful treatments for your tests.

The teaching from Flint McGlaughlin that you are referring to is actually about four words to value in the headline, not action. Four words to value. To give you ideas for headline testing, you can watch Flint teach this concept in the free FastClass – Effective Headlines: How to write the first 4 words for maximum conversion.

Hi, how would one gain access to this? Looks fascinating.

How can I join a cohort?

How do you join a cohort on 4/26?

Are the classes in the cohort live or self-paced? I’m based in Australia so there’s a big-time difference.

Oh, I would be very interested in joining that cohort. Do I email and see if I can join it?

At the end of the LiveClass, we stayed on the Zoom and talked directly to the attendees who had questions about joining the MECLABS SuperFunnel Cohort. If you are thinking of joining, or just looking for a few good takeaways for your own marketing, RSVP now for a Wednesday LiveClass.

Here are some quick excerpt videos to give you an idea of what you can experience in a LiveClass:

Single variable testing vs variable cluster testing

What is an acceptable level of statistical confidence?

Paul Good talks about the need for the MECLABS cohort

Whenever we have questions in our weekly LiveClass – ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel – we answer them here on this blog to help attendees, but to help any of our readers who didn’t attend but may have had a similar challenge as well. Read below and get ideas for powering the growth of your business with marketing experimentation. And feel free to join us for a Wednesday LiveClass using the link I just mentioned.

Has anyone tested video ads vs image? Preston’s question in the LinkedIn group got me thinking. Daniel Burstein, do you know of any?

Here’s an example. “The team discovered through testing these Facebook ads that medium-form copy received 33% more clickthrough than short- or long-form, and video drove more clickthroughs and had greater reach than static images. Because of this testing, the team realized that video was one of their most powerful tools” (from Email Marketing: List size increased 600% in one year through content, paid ad strategy).

The reason I start with this question is because it lays out the reason we should be conducting marketing experimentation – so our customers answer key questions we have about our company and our marketing with their real-world behavior, instead of just making decisions based on internal guesswork.

No matter what has worked for another company (including the above example I just used), it doesn’t mean it will work for your unique customers, your unique value proposition, your unique situation. So don’t just follow what others do, use it as fodder to come up with your own wildly creative ideas – and then test them.

Now that we know why we should test, let’s get into some of the mechanics of testing, starting with test planning…

What is the main purpose of the [pre-test planning] calculator? Is it to estimate how long you’ll need to run the campaign? If I have a set daily budget, set level of confidence, set conversion rate, set number of variants and an unknown variance [the questioner is referring to the relative conversion rate difference between variants of the ad], what is the main purpose of estimating the data? As soon as the ads start running the numbers will all change and need updating. What’s the main metric we’re aiming on understanding?

As I have mentioned in a previous blog post, I’m not the mathematician, I’m the storyteller. So I won’t get into the math behind these concepts. But I do think it helps for marketers to understand the concepts at a basic level, to inform their testing. And if you haven’t already, reading last week’s blog post will help get you up-to-speed on some basic marketing testing concepts – Factors Affecting Marketing Experimentation: Statistical significance in marketing, calculating sample size for marketing tests, and more.

In this question the person is outlining the different metrics in the pre-test estimation tool (that is included as part of membership in a MECLABS SuperFunnel Cohort).

The goal of the tool – as with any planning – is to inform your efforts. Things may not turn out exactly as you think they will, but when you see the levers that you can pull to shape results, it should help you make decisions on what you want to move forward with and actually execute.

One of those levers is budget, to affect how large of a sample size your treatments receive. In this question the “daily budget” can’t be changed, so if you run the pre-test calculation and realize it would take an inordinate amount of time to reach statistical validity based on the amount of impressions or traffic your budget can buy, you may have to get creative.

Here’s one example, using Bugs Bunny and Daffy Duck. The team behind Looney Tunes and Merrie Melodies had a set ‘daily budget’ for how long they could work on an episode. So if they wanted to do something a little more groundbreaking, they would ‘borrow’ from other episodes. Maybe they would take only four days instead of five for a few episodes, and then using that time they saved up they would dedicate eight days to an episode to really push the envelope (I recommend Duck Amuck and What’s Opera, Doc?).

Perhaps this planning tool might help you do the same thing. Invest less of your fixed daily budget in experiments where you know there will be a large conversion rate difference between the ad or landing page variants, so you save up some of that budget for experiments where there is a smaller difference.

Which brings us to another decision pre-test planning can help us make…

Ave Test Users = Impressions … Primary KPI Successes = Clicks (Or is it Opt-ins?) Variants = number of ads, Assumed Real Difference = Between the ads

The question is around what the primary KPI (key performance indicator) should be for an experiment – clicks on the ad or opt-ins on the landing page.

This is important to statistical significance because it will impact the sample size. The deeper you test in the funnel, the smaller your sample size will be. More people will click on an ad then will opt-in on the landing page. So the larger budget you will need to get a large enough sample size.

This is another reason why using the pre-test estimation calculator can be helpful – to inform where in the funnel you decide to test, based on your budget and/or other capabilities for getting people to see the conversion action you are trying to test.

What should your sample size be? There is no set sample size you need to reach. It is affected by how different the performance is in the different treatments. And we use the pre-test planning calculator to help us find that number.  “An important factor in sample size determination is the difference in results between the treatments. If the treatments return very different results, it’s much easier to confidently say that you really do have two (or however many) emails that will perform differently. You don’t need as many samples to do that. However, if the treatments have very similar results, you want many more observations to see if there really is a difference.” (from Marketing Optimization: How to determine the proper sample size)

Why make impressions-to-ad-clicks as the primary KPI and then ad-clicks-to-appointments a secondary KPI? Why not make impressions-to-appointments the primary KPI? Is that just because the volume of data for the latter would be higher? So measuring more the message lever, rather the whole funnel effectiveness?

This question naturally follows from the previous one. As discussed, your KPI for a test is partially informed by the sample size and relative difference between the control and treatment.

But it is also partially informed by what you are trying to learn. Let’s not get too deep in the math and mechanics that we forget the goal of test – to learn how to better serve the customer and better communicate to the customer so we can improve our results (as discussed in the first question in this blog post).

And remember, you don’t have to learn everything in one big test. Your goal is to test and learn repeatedly. So your first test can be in the channel, and then your second test could be on the landing page, for example.

To inform future testing it helps to establish secondary KPIs when you set up your test. Of course, you could just look at every possible metric under the sun. But the reason we do pre-test planning is to run our experiments with intention.

One of our former data scientists explained it to me this way – an experiment is picking a specific tree in a forest, then throwing a rock and seeing if you hit that specific tree versus throwing a rock into a forest, seeing which tree it randomly hits, and then remarking, “oh yeah, I meant to hit that tree.”

Pre-test planning focuses our thinking and efforts on hitting that tree. And secondary KPIs can help here as well. “Don’t try to analyze it all; you’ll get lost in data and become discouraged and confused. Instead, narrow your focus to the metrics that will provide the most relevant insights. Having primary and secondary KPIs for your site will help you begin to narrow your focus” (from Marketing Analytics: 6 simple steps for interpreting your data).

So you could first test message levers in the channel using a primary KPI of clickthrough. Then if your secondary KPI is conversion rate on the landing page, and the conversion rate is low for the winning ad treatment, you have a next (and very interesting) question to test.

Did the ad message you used simply bring lower-quality traffic, less motivated people to your website? For example, an ad message of “Free iPaid” might win a test in the channel but isn’t necessarily doing your funnel any favors if you can’t pay off that promise.

Or, is it an effective ad message to attract your ideal customer, but you don’t pay off that message well with you landing page. There is poor continuity between the ad and the landing page, which was simply a previous page you already had and didn’t align with the treatment ad’s messaging. Which brings us to…

Why would we use a page separate from our webpage?

You may have an ad message you want to test that is very different from the message on any webpage you have now. In that case, it would likely make sense to create a new landing page for it.

We could use the experimentation process I just mentioned to determine that our current webpage does not deliver on the new ad message, that there is poor continuity from the new ad message to our webpage.

However, signing up for marketing experimentation is not a binding agreement to endure unnecessary agony. We can also use our common sense and marketer’s intuition to avoid changes that are very likely to have poor results – after all, we are testing with a real budget and real people – and focus our energies on the questions that are more difficult to answer and will have a big impact on our business.

Of course, this means you may have to build a new landing page. Which brings us to our next question…

Chris it looks like you followed the template exactly, Daniel are you saying feel free to move the layout around and put the hero image in place of the form?

Participants in the MECLABS SuperFunnel Cohort get access to MECLABS SuperFunnel builder software (a simple drag-and-drop landing page builder that is embedded with the MECLABS methodology). This includes templates to help you build a landing page based on an approach that has shown to be effective in previous experimentation.

I won’t go through the entire template, but it begins at the top of your landing page with Micro-Yes 1 – Yes, I will pay attention. This section should be a synopsis of your core offer. Followed by Micro-Yes 2 – Yes, I will engage deeper – where you address any friction or anxiety. Also, in this section you can add a video, image, or a form that supports your message.

The Cohort member I was providing optimization advice for in the LiveClass has a spokesperson that is well-known in his industry, and my point was invoking that spokesperson earlier on the landing page may help grab attention and be a core element of the offer that gets the ideal customer to say ‘yes.’ It may also be a way to reduce anxiety.

For example, I am a huge fan of Jerry Seinfeld. If he ever produced a show focused on marketing (let’s call it Copywriters In Cars Getting Conversions), his participation in the show would be a core part of the offer that would grab my attention. His participation in the show would also reduce my anxiety that it is a quality show and get me to engage deeper.

The SuperFunnel template can guide you to build your landing page but should also be spurring future hypotheses in you for further testing. For example, is our founder a core part of the offer and she should be included at the top of the page along with her image? Or is she a tertiary credibility indicator, and she should be used lower down on the page to help address any last-minute anxiety?

This is an example of a question you can test that has implications for the business. You can start by testing in the channel and conduct further testing on the page.

To get you thinking of possibilities for your own business, you can see three value categories we test through email for VolunteerMatch in A/B Testing: How to improve already effective marketing.

If you are conducting a marketing experiment in the channel, you will also need to create the different ads to test. Which bring us to…

Ask ChatGPT what colors to use: “I have an ad with #CFB82C as the primary text color, #384E6C as the background color, and #333333 and #FFFFFF as additional colors. What color should my call to action button be and what color should the text be on top of it. Give me html codes.” … “Is there a green color that will work?”

OK, this isn’t really a question. I found this in the Zoom chat, and I just thought it was some nifty advice for using artificial intelligence to help you build an ad.

Hi Flint, what does the cohort entail? 4-5 weekly meetings. 40 days total/ price? Also if I’m generating a sales funnel for the first time is this a good option for me or should I start on another level (finishing my Micro Yes’s now)

How do I join this group!?

How do I join the MEC200 group?

How do I register for the next cohort?

Is there another cohort scheduled yet after April or approx. start date?

Jane, Is this video zoom call a part of the first cohort?

What happens with the transition to cohort 300?

There were a lot of questions from attendees about the cohort itself, and we answered them with a Q&A session at the end of the cohort. If you would like to learn more about the cohort, and the five ways you can ‘pay’ if you choose to participate (there are monetary and non-monetary options), just join us on a Wednesday LiveClass of ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel.

Even if you don’t choose to join, by attending you should get a few ideas you can implement to your marketing funnel right away to improve conversion.

Here’s a quick excerpt from a recent LiveClass to give you an idea of what to expect – Hypothesis Articulation vs Essence.

Here are answers to questions SuperFunnel Cohort members put in the chat of recent MEC200 and MEC300 LiveClasses for ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel (feel free to register at that link to join us for an upcoming MECLABS LiveClass).

How many impressions or how much reach do we need for statistical significance?

I can’t give you a specific number, because the answer will vary based on several factors (described below). Also, MECLABS SuperFunnel Cohort members now have access to a Simplified Test Protocol in their Hub, and you can use that tool to calculate these numbers, as shown in Wednesday’s LiveClass.

But I included the question in this blog post because I thought it would be helpful to explain the factors that go into this calculation. And to be clear, I’m not the math guy here. So I won’t get into the formulas and calculations. However, a basic understanding of these factors has always helped me better understand marketing experimentation, and hopefully it will help you as well.

First of all, why do we even care about statistical significance in marketing experimentation? When we run a marketing test, essentially we are trying to measure a small group to learn lessons that would be applicable to all potential customers – take a lesson from this group, and apply it to everyone else.

Statistical significance helps us understand that our test results represent a real difference and aren’t just the result of random chance.

We want to feel like the change in results is because of our own hand. It’s human nature. A better headline on the treatment landing page, or a better offer. And we can see the results with our own eyes, so it is very hard to understand that a 10% conversion rate may not really be any different than an 8% conversion rate.

But it may just be randomness. “Why is the human need to be in control relevant to a discussion of random patterns? Because if events are random, we are not in control, and if we are in control of events, they are not random, there is therefore a fundamental clash between our need to feel we are in control and our ability to recognize randomness,” Dr. Leonard Mlodinow explains in The Drunkard’s Walk: How Randomness Rules Our Lives.

You can see the effect of randomness for yourself if you run a double control experiment – split traffic to two identical landing pages and even though they are exactly the same, they will likely get a different number of conversions.

We fight randomness with statistical significance. The key numbers we want to know to determine statistical significance are:

• Sample size – How many people see your message?
• Conversions – How many people act on your message?
• Number of treatments – For example, are you testing two different landing pages, or four?
• Level of confidence – Based on those numbers, how sure can you be that there really is a difference between your treatments?

And this is the reason I cannot give you a standard answer for the number of impressions you need to reach statistical significance – because of these multiple factors.

I’ll give you an (extreme) example. Let’s say your sample size is 100 and you have four treatments. That means, each landing page was visited by 25 people. Three of the landing pages each get three conversions, and the other landing page gets four conversions. Since so few people saw these pages and the difference in conversions is so small, how confident are you that they are different? Or perhaps you randomly had one more motivated person in that last group that gave you the extra conversion.

And this assumes an even traffic split, which you may not want to do based on how concerned you are about the change you are making. As we teach in How to Plan Landing Page Tests: 6 Steps to Guide Your Process, “Using an uneven traffic split is helpful when your team is testing major changes that could impact brand perception or another area of your business. Although the results will take longer to reach statistical significance, the test is less likely to have an immediate negative impact on business.”

Now, let’s take another extreme example. Say your sample size is 10,000,000 and you have just a control and a treatment. The control gets 11 conversions, but the treatment gets 842,957 conversions. In that case, you can be pretty confident that the control and treatment are different.

But there is another number at play here – Level of Confidence (LoC). When we say there is a statistically significant difference, it is at a specific Level of Confidence. How sure do you want to be that the control and treatment are different? For marketing experimentation, 95% is the gold standard. But 90%, or even 80% could be a enough if it is a change that likely isn’t going to be harmful, and doesn’t take too many resources to make. And the lower Level of Confidence you are OK with, the lower sample size you need and the less difference in conversions you need to be statistically significant at that LoC.

So is Estimated Minimum Relative Difference our desired/target lift if our test performs as expected?

Once you understand how statistical significance works (as I described in the previous question), the next natural question is – well, how does this affect my business decisions?

The first answer is, this understanding will help you run marketing experiments that are more likely to predict your potential customers’ real-world behavior.

But second answer is – this should impact how you plan and run tests.

This question refers to the Estimated Minimum Relative Difference in the Simplified Test Protocol that SuperFunnel Cohort members receive, specifically in the test planning section that helps you forecast how long to run a test to reach statistical significance. And yes, the Estimated Minimum Relative Difference is the difference in conversion rate you expect between the control and treatment.

As discussed above, the larger this number is, the less samples and time (to get those samples) it takes to run a test.

Which means that companies with a lot of traffic can run tests that reach statistical significance even if they make very small changes. For example, let’s say you’re running a test on the homepage of a major brand, like Google or YouTube, which get billions of visits per month. Even a very small change like button color may be able to reach statistical significance.

But if you have lower traffic and a smaller budget, you likely need to take a bigger swing with your test to find a big enough difference. This does not necessarily mean it has to require major dev work. For example, the headlines “Free courtside March Madness tickets, no credit card required” and “$12,000 upper level March Madness tickets, $400 application fee to see if you qualify” are very quick changes on a landing page. However, they are major changes in the mind of a potential customer and will likely receive very different results.

Which brings us to risk. When you run valid experiments, you decrease the risk in general. Instead of just making a change and hoping for the best, only part of your potential customer base sees the change. So if your change actually leads to a decrease, you learn before shifting your entire business. And you know what caused the decrease in results because you have isolated all the other variables.

But the results from your experiments will never guarantee a result. They will only tell you how likely there will be a difference when you roll out that change to all your customers for a longer period. So if you take that big swing you’ve always wanted to take, and the results aren’t what you expect, that may rein your team in from a major fail.

As we say in Quick Guide to Online Testing: 10 tactics to start or expand your testing process, “If a treatment has a significant increase over the control, it may be worth the risk for the possibility of high reward. However, if the relative difference between treatments is small and the LoC is low, you may decide you are not willing to take that risk.”

With a test running past 4 weeks, how concerned are you about audience contamination between the variants?

Up until now we’ve been talking about a validity threat called sampling distortion effect – failure to collect a sufficient sample size. As discussed, this could mean your marketing experiment results are due to random variability, and not a true difference between how your customers will react to your treatments when rolled out to your entire customer set.

But there are other validity threats as well. A validity threat simply means that a factor other than the change you made – say, different headlines or different CTAs – was the reason for the difference in performance you saw. You are necessarily testing with a small slice of your total addressable market, and you want to ensure that the results have a high probability of replicability – you will see an improvement when you roll out this change to all of your potential customers.

Other validity threats include instrumentation effect – your measurement instrument affecting the results, and selection effect – the mix of customers seeing the treatments do not represent the customers that you will ultimately try to sell to, or in this case, the same customer seeing multiple treatments.

These are the types of validity threats this questioner is referring to. However, I think there is a fairly low (but not zero) chance of these validity threats only coming from running the test (not too much) past four weeks. While we have seen this problem many years ago, most major platforms have gotten pretty good at assigning a visitor to a specific treatment and keeping them there on repeat visits.

That said, people can visit on multiple devices, so the split certainly isn’t perfect. And if your offer is something that calls for many repeat visits, especially from multiple devices (like at home and at work), this may become a bigger validity threat. If this is a concern, I suggest you ask your testing software provider how they mitigate against these validity threats.

However, when I see your question, the validity threat I would worry about most is history effect, an extraneous variable that occurs with the passage of time. And this one is all on you, friend, there is not much your testing software can do to mitigate against it.

As I said, you are trying to isolate your test so the only variables that affect the outcome are the ones you’ve purposefully changed and are intending to test based on your hypothesis. The longer a test runs, the harder this gets. For example, you (or someone else in your organization) may choose to run a promotion during that period. Maybe you can keep a tight lid on promotions for a seven-day test, but can you keep the promotion wolves at bay in your organization for a full two months?

Or you may work at an ecommerce company looking to get some customer wisdom to impact your holiday sales. If you have to test for two months before rolling anything out, you may test in September and October. However, customers may behave very differently earlier in the year than they would in December, when their motivation to purchase a gift near a looming deadline is a much bigger factor.

While a long test makes a history effect more likely, it can occur even during a shorter test. In fact, our most well-known history effect case study occurred during a seven-day experiment because of the NBC television program Dateline. You can read about it (along with info about other validity threats) in the classic MarketingExperiments article Optimization Testing Tested: Validity Threats Beyond Sample Size.

Join us for a Wednesday LiveClass

As I mentioned, these questions came from the chat of recent LiveClasses. You can RSVP now to join us for an upcoming LiveClass. Here are some short videos to give you an idea of what you can learn from a LiveClass…

“If there’s not a strong enough difference in these two Google ads…the difference isn’t going to be stark enough to probably produce a meaningful set of statistics [for a marketing test]…” – Flint McGlaughlin in this 27-second video.

“…but that’s what Daniel was really touching on a moment ago. OK, you’ve got a [marketing] test, you’ve got a hypothesis, but is this really where you want to invest your money? Is this really going to get the most dollars or the most impact for the energy you invest?…” – Flint McGlaughlin, from this 46-second video about finding the most important hypotheses to test.

How far do you have to go to with your marketing to take potential customers from the problem they think they have, to the problem they do have? I discuss this topic while coaching the co-founders of an eyebrow beauty salon training company on their marketing test hypothesis in this 54-second video.