Archive

Posts Tagged ‘key performance indicator’

AI In Marketing Strategies: Your questions answered

August 11th, 2023

Wednesdays at 2 pm EDT we hold an AI Guild briefing. Chat with MECLABS AI if you would like to register to attend.

In the session, marketers and entrepreneurs can ask questions in the webinar chat. And we answer a few of them right here…

How do you determine when to use Claude vs MECLABS AI?

Claude is a generative AI chatbot that doesn’t have a specific purpose. Although from my personal use, it is very good with large amounts of text, more so than any other generative AI I’ve tried, lending it especially well to tasks like analyzing webinar chat logs, selecting quotes from transcripts, giving feedback on long articles, and similar tasks.

MECLABS AI (available to MECLABS AI Guild members) does have a specific purpose – headlines, value propositions, and other marketing, which makes it better for marketing specific tasks.

MECLABS is built on ChatGPT, but trained with the MECLABS methodology. Since Claude is a more general AI, it is trained on all marketing-related information, not only successful marketing practices.

I also wanted to mention, that while the questions have lately focused on AI chatbots, there are other types of generative AI, like text-to-image models. Flint McGlaughlin teaches an example in Sales Funnel Design: How to combine an “old school” journal, a web page builder, and DALL·E.

How do you overcome the 2,000-character prompt limit on Bing AI?

Here’s a fun little trick. The 2,000-character prompt limit on Bing AI only applies to the ‘more balanced’ conversation style. If you choose the ‘more creative’ or ‘more precise’ conversation style, your prompt limit doubles to 4,000 characters.

Bing AI

Nice, but maybe still not enough for what you want. There are two options I can see.

One, you could use Claude. Again, Claude can consume some very long prompts.

However, on the downside, Claude is a self-contained chatbot and not connected to the internet like Bing. If you need internet connection, you could always treat Bing like a customer and take a Micro-Yes Sequence approach. Map out your information (like you would a customer journey) and spoon feed it along a path for each response.

Even that is not limitless though. I’ve seen data that show Bing has a limit of anywhere from five to 20 chat turns per session. I’m not sure the exact limit, but from my experience it varies based on the topic and length. And for what it’s worth, at some point Bing just acts like it is ‘just so over’ the string of conversation and wants to start a new one. I’m not sure if it’s a feature or a bug, but it reminds me of my teenage daughter.

When you go to Claude, is that automatically Claude 2?

Yes… and no.

When you go to the Claude website, you are automatically using Claude 2, which was released in July 2023. (And as I’ve mentioned before, the most exciting thing to me about Clause is the amount of information it can process. “Claude 2 has been trained to have a further expanded context window of 200K tokens, corresponding to roughly 150,000 words” according to Model Card and Evaluations for Claude Models).

Why you even have to ask this question, and what is interesting to me as a marketer – when you visit the Claude website, it doesn’t mention a model number. This is very different from ChatGPT, which clearly displays that you are using GPT-3.5 and even shows you the ability for a paid upgrade to GPT-4.

When I ask Claude itself the reason for this, it states, “The choice aligns with Anthropic’s overall philosophy of developing AI responsibly, with a focus on user benefits rather than technology marketing. But these are just my speculations on Anthropic’s possible motivations. The company may have other reasons as well.”

If this is the case, I think it’s a bit of company logic. While ChatGPT’s use of model numbers is a marketing tactic to try to show progress and encourage upsells (just like when I bought a Samsung S23 although I would have been totally fine with the S22), it’s also a little more transparent. And with the amount of hesitancy and outright fear around AI, a little transparency is a good thing.

Can you use custom instructions on the ChatGPT version that’s in Bing?

No. As of right now, custom instruction is only for ChatGPT Plus (the paid version), and not yet available in the UK and EU. You can read more about ChatGPT customer instructions in Choosing Your AI Partner: A critical comparison of ChatGPT and Claude 2 for digital marketing.

What is Priority One if we’re going to move into these methodologies? How do we figure out where to start?

This question refers to the AI prompt methodology Flint McGlaughlin presented in this week’s briefing and LiveClass.

Like any type of conversion optimization (which this still is, even if AI is assisting you), start where you will have the most impact.

Here is an example (from Web Optimization: VacationRoost implements 2 testing methodologies to boost total conversion rates by 12%.):

Upcoming tests are elected based on a prioritization spreadsheet.

While many testing teams may allocate importance to factors such as volume and bounce rates, VacationRoost’s spreadsheet is organized by which level in the funnel the page or test would impact.

“Really what you want [to ask] is, ‘What’s going to have the most dollar impact on the organization?'” Hutchings explained.

He added that the pages leading into the purchase funnel rise to the top of the testing docket because of the likelihood of increasing conversion for the whole site is far greater on those types of pages.

I used the suggested prompts to do the competitor analysis and as my website is new, it kept referring back to the content of a website that had a similar name, even when I provided my URL.  Does anyone have any other suggestions on what I can do in my prompts?

Then you may have to do this a little manually, so to speak. And here is where Claude can come in handy, since it can accept such a long prompt.

Copy and paste key information from your website (like your homepage, maybe other key pages) along with competitors’ sites into individual text files and attach them to your Claude prompt using the paper clip icon, telling Claude what you are including.

In general, answers that we get from ChatGPT, are text rich. Is that something we use as a reference or work to absorb it?

Everyone has a different working preference. But remember, the great thing about generative AI is that you don’t have to settle for the first output. If it is too long for you, tell ChatGPT what length you are looking for. Here’s an example…

I used the Prompt Kit (available to MECLABS AI Guild members) marketing prompt for audience research, and got this output from ChatGPT:

Certainly, as a marketing professional, I recognize the transformative potential of artificial intelligence (AI) in the marketing field. Yet, there are still substantial challenges in implementing AI effectively. Here’s an analysis of three major challenges, their causes, impacts, and potential solutions.

  1. Challenge: Data Privacy and Compliance
  • Why it Exists: AI relies heavily on data for personalization and targeting. The ever-changing regulations surrounding data privacy, such as GDPR and CCPA, create complexity.
  • Impact on My Life: Ensuring compliance with various regulations demands extensive legal oversight and continuous adaptation of AI algorithms, consuming both time and resources.
  • Potential Solutions: Implementing strict data governance policies and utilizing AI models that prioritize data anonymization can help. Collaborating with legal experts specializing in privacy law is also essential.
  1. Challenge: Integrating AI with Existing Systems
  • Why it Exists: Many organizations have legacy systems that aren’t built to support modern AI algorithms, leading to compatibility issues.
  • Impact on My Life: Integrating AI into these systems can be labor-intensive, time-consuming, and may create bottlenecks in marketing processes.
  • Potential Solutions: An incremental approach to integration, supported by a cross-functional team including IT experts, could streamline the process. Adopting flexible AI models that can be adapted to varying systems might also ease the integration.
  1. Challenge: Bias in AI Algorithms
  • Why it Exists: AI models are trained on historical data, and if that data contains biases (gender, racial, etc.), the model may perpetuate those biases.
  • Impact on My Life: Using biased AI models can lead to unethical targeting and representation in marketing, potentially damaging the brand’s reputation.
  • Potential Solutions: Regular audits of AI algorithms, involving diverse teams to understand and minimize biases, and utilizing fairness-aware algorithms can help overcome this issue.

Summary:

The integration of AI in marketing presents valuable opportunities but also notable challenges related to data privacy, system integration, and algorithm biases. By adopting robust legal compliance measures, embracing a cross-functional integration strategy, and rigorously auditing algorithms for bias, these challenges can be mitigated. This thoughtful approach to AI implementation aligns with the ethics and efficiency that today’s marketing professionals strive to uphold.

I then told ChatGPT, “What is the single most important thing I should know from your analysis?” To which it responded…

The single most important insight from the analysis is that while artificial intelligence presents immense opportunities for marketing, its responsible and effective implementation requires careful consideration of legal compliance, integration with existing systems, and the elimination of biases. The underlying theme is that success with AI in marketing demands a strategic approach that balances technological innovation with ethical considerations and compatibility with existing infrastructure.

Should we have already put the custom instructions into ChatGPT when we enter this prompt?

All else being equal, ChatGPT will serve you better with custom instructions because it will understand the context of your business and your customers.

But you can use any of the prompts in the Prompt Kit (available to MECLABS AI Guild members) without using custom instructions.

Again, if you want to learn more about custom instructions, you can read our last Q&A post – Choosing Your AI Partner: A critical comparison of ChatGPT and Claude 2 for digital marketing.

Would it be worth doing a few interviews with some of the people who have and haven’t purchased from our email sequence to understand the person behind the numbers?

Yes, I love that idea. That is a great example of increasing customer intimacy.

If you’re not able to go that far, you could send emails that people can reply to (in other words, not from a ‘no reply’ email address which some companies do) and make it clear in the email that replies are welcome.

Keep in mind, with this approach you will only hearing from the extremes, so take what they say with a grain of salt. But if something is truly outstanding or really horrible, you may hear about it. And you may also get a good understand of the words customer use to discuss your product or service.

For the pre-test success metric, is it important for this KPI to have large amounts of data? So for example if I wanted to use sales as my pre-test success metric but my volumes are not in thousands, is that a problem? Where is the biggest opportunity?

This is a question about A/B testing. To quickly explain a complex topic, if your results aren’t statistically significant, what you might be seeing is just random chance. An extreme example: 2 sales is a 100% increase over 1 sale, but that increase might just be random and not because of what you were testing (we explore this topic deeper in Factors Affecting Marketing Experimentation: Statistical significance in marketing, calculating sample size for marketing tests, and more).

Which begs the question – “so how many samples (sales, clicks, etc.) do I need?”

There is no one specific answer. The answer will vary based on the conversion rate difference between the control and treatment. The pre-test planning tab of the Simplified Test Protocol (available to MECLABS AI Guild members) will help you calculate that number.

So as the questioner rightly points out, it is difficult to make a number deep in your funnel, like sales, the success metric…unless you get a whole lot of sales.

As for the biggest priority, I would refer you to the above question about ‘where to start?’ However, your answer will be constrained by where you have the budget to get a statistically significant result.

We also have a free Test Planning Scenario Tool that helps you determine the optimal testing sequence.

Related Resources

Benefits of AI in Marketing: How do the views about artificial intelligence in marketing differ between leaders and practitioners? [chart]

AI Marketing Tools: How marketers are using artificial intelligence to help their campaigns right now

Artificial Intelligence and Machine Learning in Marketing: What marketers (even those who don’t care about tech) should know about AI and ML

Marketing Experimentation: How to get real-world answers to questions about a company’s marketing efforts

April 11th, 2023

Whenever we have questions in our weekly LiveClass – ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel – we answer them here on this blog to help attendees, but to help any of our readers who didn’t attend but may have had a similar challenge as well. Read below and get ideas for powering the growth of your business with marketing experimentation. And feel free to join us for a Wednesday LiveClass using the link I just mentioned.

Has anyone tested video ads vs image? Preston’s question in the LinkedIn group got me thinking. Daniel Burstein, do you know of any?

Here’s an example. “The team discovered through testing these Facebook ads that medium-form copy received 33% more clickthrough than short- or long-form, and video drove more clickthroughs and had greater reach than static images. Because of this testing, the team realized that video was one of their most powerful tools” (from Email Marketing: List size increased 600% in one year through content, paid ad strategy).

The reason I start with this question is because it lays out the reason we should be conducting marketing experimentation – so our customers answer key questions we have about our company and our marketing with their real-world behavior, instead of just making decisions based on internal guesswork.

No matter what has worked for another company (including the above example I just used), it doesn’t mean it will work for your unique customers, your unique value proposition, your unique situation. So don’t just follow what others do, use it as fodder to come up with your own wildly creative ideas – and then test them.

Now that we know why we should test, let’s get into some of the mechanics of testing, starting with test planning…

What is the main purpose of the [pre-test planning] calculator? Is it to estimate how long you’ll need to run the campaign? If I have a set daily budget, set level of confidence, set conversion rate, set number of variants and an unknown variance [the questioner is referring to the relative conversion rate difference between variants of the ad], what is the main purpose of estimating the data? As soon as the ads start running the numbers will all change and need updating. What’s the main metric we’re aiming on understanding?

As I have mentioned in a previous blog post, I’m not the mathematician, I’m the storyteller. So I won’t get into the math behind these concepts. But I do think it helps for marketers to understand the concepts at a basic level, to inform their testing. And if you haven’t already, reading last week’s blog post will help get you up-to-speed on some basic marketing testing concepts – Factors Affecting Marketing Experimentation: Statistical significance in marketing, calculating sample size for marketing tests, and more.

In this question the person is outlining the different metrics in the pre-test estimation tool (that is included as part of membership in a MECLABS SuperFunnel Cohort).

The goal of the tool – as with any planning – is to inform your efforts. Things may not turn out exactly as you think they will, but when you see the levers that you can pull to shape results, it should help you make decisions on what you want to move forward with and actually execute.

One of those levers is budget, to affect how large of a sample size your treatments receive. In this question the “daily budget” can’t be changed, so if you run the pre-test calculation and realize it would take an inordinate amount of time to reach statistical validity based on the amount of impressions or traffic your budget can buy, you may have to get creative.

Here’s one example, using Bugs Bunny and Daffy Duck. The team behind Looney Tunes and Merrie Melodies had a set ‘daily budget’ for how long they could work on an episode. So if they wanted to do something a little more groundbreaking, they would ‘borrow’ from other episodes. Maybe they would take only four days instead of five for a few episodes, and then using that time they saved up they would dedicate eight days to an episode to really push the envelope (I recommend Duck Amuck and What’s Opera, Doc?).

Perhaps this planning tool might help you do the same thing. Invest less of your fixed daily budget in experiments where you know there will be a large conversion rate difference between the ad or landing page variants, so you save up some of that budget for experiments where there is a smaller difference.

Which brings us to another decision pre-test planning can help us make…

Ave Test Users = Impressions … Primary KPI Successes = Clicks (Or is it Opt-ins?) Variants = number of ads, Assumed Real Difference = Between the ads

The question is around what the primary KPI (key performance indicator) should be for an experiment – clicks on the ad or opt-ins on the landing page.

This is important to statistical significance because it will impact the sample size. The deeper you test in the funnel, the smaller your sample size will be. More people will click on an ad then will opt-in on the landing page. So the larger budget you will need to get a large enough sample size.

This is another reason why using the pre-test estimation calculator can be helpful – to inform where in the funnel you decide to test, based on your budget and/or other capabilities for getting people to see the conversion action you are trying to test.

What should your sample size be? There is no set sample size you need to reach. It is affected by how different the performance is in the different treatments. And we use the pre-test planning calculator to help us find that number.  “An important factor in sample size determination is the difference in results between the treatments. If the treatments return very different results, it’s much easier to confidently say that you really do have two (or however many) emails that will perform differently. You don’t need as many samples to do that. However, if the treatments have very similar results, you want many more observations to see if there really is a difference.” (from Marketing Optimization: How to determine the proper sample size)

Why make impressions-to-ad-clicks as the primary KPI and then ad-clicks-to-appointments a secondary KPI? Why not make impressions-to-appointments the primary KPI? Is that just because the volume of data for the latter would be higher? So measuring more the message lever, rather the whole funnel effectiveness?

This question naturally follows from the previous one. As discussed, your KPI for a test is partially informed by the sample size and relative difference between the control and treatment.

But it is also partially informed by what you are trying to learn. Let’s not get too deep in the math and mechanics that we forget the goal of test – to learn how to better serve the customer and better communicate to the customer so we can improve our results (as discussed in the first question in this blog post).

And remember, you don’t have to learn everything in one big test. Your goal is to test and learn repeatedly. So your first test can be in the channel, and then your second test could be on the landing page, for example.

To inform future testing it helps to establish secondary KPIs when you set up your test. Of course, you could just look at every possible metric under the sun. But the reason we do pre-test planning is to run our experiments with intention.

One of our former data scientists explained it to me this way – an experiment is picking a specific tree in a forest, then throwing a rock and seeing if you hit that specific tree versus throwing a rock into a forest, seeing which tree it randomly hits, and then remarking, “oh yeah, I meant to hit that tree.”

Pre-test planning focuses our thinking and efforts on hitting that tree. And secondary KPIs can help here as well. “Don’t try to analyze it all; you’ll get lost in data and become discouraged and confused. Instead, narrow your focus to the metrics that will provide the most relevant insights. Having primary and secondary KPIs for your site will help you begin to narrow your focus” (from Marketing Analytics: 6 simple steps for interpreting your data).

So you could first test message levers in the channel using a primary KPI of clickthrough. Then if your secondary KPI is conversion rate on the landing page, and the conversion rate is low for the winning ad treatment, you have a next (and very interesting) question to test.

Did the ad message you used simply bring lower-quality traffic, less motivated people to your website? For example, an ad message of “Free iPaid” might win a test in the channel but isn’t necessarily doing your funnel any favors if you can’t pay off that promise.

Or, is it an effective ad message to attract your ideal customer, but you don’t pay off that message well with you landing page. There is poor continuity between the ad and the landing page, which was simply a previous page you already had and didn’t align with the treatment ad’s messaging. Which brings us to…

Why would we use a page separate from our webpage?

You may have an ad message you want to test that is very different from the message on any webpage you have now. In that case, it would likely make sense to create a new landing page for it.

We could use the experimentation process I just mentioned to determine that our current webpage does not deliver on the new ad message, that there is poor continuity from the new ad message to our webpage.

However, signing up for marketing experimentation is not a binding agreement to endure unnecessary agony. We can also use our common sense and marketer’s intuition to avoid changes that are very likely to have poor results – after all, we are testing with a real budget and real people – and focus our energies on the questions that are more difficult to answer and will have a big impact on our business.

Of course, this means you may have to build a new landing page. Which brings us to our next question…

Chris it looks like you followed the template exactly, Daniel are you saying feel free to move the layout around and put the hero image in place of the form?

Participants in the MECLABS SuperFunnel Cohort get access to MECLABS SuperFunnel builder software (a simple drag-and-drop landing page builder that is embedded with the MECLABS methodology). This includes templates to help you build a landing page based on an approach that has shown to be effective in previous experimentation.

I won’t go through the entire template, but it begins at the top of your landing page with Micro-Yes 1 – Yes, I will pay attention. This section should be a synopsis of your core offer. Followed by Micro-Yes 2 – Yes, I will engage deeper – where you address any friction or anxiety. Also, in this section you can add a video, image, or a form that supports your message.

The Cohort member I was providing optimization advice for in the LiveClass has a spokesperson that is well-known in his industry, and my point was invoking that spokesperson earlier on the landing page may help grab attention and be a core element of the offer that gets the ideal customer to say ‘yes.’ It may also be a way to reduce anxiety.

For example, I am a huge fan of Jerry Seinfeld. If he ever produced a show focused on marketing (let’s call it Copywriters In Cars Getting Conversions), his participation in the show would be a core part of the offer that would grab my attention. His participation in the show would also reduce my anxiety that it is a quality show and get me to engage deeper.

The SuperFunnel template can guide you to build your landing page but should also be spurring future hypotheses in you for further testing. For example, is our founder a core part of the offer and she should be included at the top of the page along with her image? Or is she a tertiary credibility indicator, and she should be used lower down on the page to help address any last-minute anxiety?

This is an example of a question you can test that has implications for the business. You can start by testing in the channel and conduct further testing on the page.

To get you thinking of possibilities for your own business, you can see three value categories we test through email for VolunteerMatch in A/B Testing: How to improve already effective marketing.

If you are conducting a marketing experiment in the channel, you will also need to create the different ads to test. Which bring us to…

Ask ChatGPT what colors to use: “I have an ad with #CFB82C as the primary text color, #384E6C as the background color, and #333333 and #FFFFFF as additional colors. What color should my call to action button be and what color should the text be on top of it. Give me html codes.” … “Is there a green color that will work?”

OK, this isn’t really a question. I found this in the Zoom chat, and I just thought it was some nifty advice for using artificial intelligence to help you build an ad.

Hi Flint, what does the cohort entail? 4-5 weekly meetings. 40 days total/ price? Also if I’m generating a sales funnel for the first time is this a good option for me or should I start on another level (finishing my Micro Yes’s now)

How do I join this group!?

How do I join the MEC200 group?

How do I register for the next cohort?

Is there another cohort scheduled yet after April or approx. start date?

Jane, Is this video zoom call a part of the first cohort?

What happens with the transition to cohort 300?

There were a lot of questions from attendees about the cohort itself, and we answered them with a Q&A session at the end of the cohort. If you would like to learn more about the cohort, and the five ways you can ‘pay’ if you choose to participate (there are monetary and non-monetary options), just join us on a Wednesday LiveClass of ChatGPT, CRO and AI: 40 Days to build a MECLABS SuperFunnel.

Even if you don’t choose to join, by attending you should get a few ideas you can implement to your marketing funnel right away to improve conversion.

Here’s a quick excerpt from a recent LiveClass to give you an idea of what to expect – Hypothesis Articulation vs Essence.