Marketing 101: What is an A/B split test?
Marketing has a language all its own. This is our latest in a series of posts aimed at helping new marketers learn that language. What term do you find yourself explaining most often to new hires during onboarding? Let us know.
An A/B split test refers to a test situation in which two randomized groups of users are sent different content at the same time to monitor the performance of specific campaign elements.
A/B split testing is a powerful way to improve marketing and messaging performance because it enables you to make decisions about the best headline, ad copy, landing page design, offer, etc., based on actual customer behavior and not merely a marketer’s opinion.
Let’s break down the process of A/B split testing.
Real People Enter the Test
This is part of the power of A/B split testing as compared to other forms of marketing research such as focus groups or surveys. A/B split testing is conducted with real people in a real-world purchase situation making real decisions, as opposed to a survey or focus group where you’re asking people who (hopefully) represent your customers what they might do in a hypothetical situation, or to remember what they have done in a past situation.
Not only can you inadvertently influence people in ways that change their answer (since the research gathering mechanism does not exactly mimic the real-world situation), but people may simply tell you what they think you want to hear.
Or, many times, customers misjudge how they would act in a situation or misremember how they have acted in the past.
That doesn’t mean you shouldn’t use surveys, focus groups and the like. Use this new information to create a hypothesis about your customers. And then run an A/B split test to learn from real customers if your hypothesis is correct.
Sample Split
People are randomly split into two (or more) groups. These groups are your sample. It’s known as a sample because it isn’t every customer who ever purchased from you. It is a sample of those customers. You need enough customers to make sure you have a statistically significant sample. This provides a high likelihood that the behavior you’re recording from this sample accurately represents customer behavior and isn’t merely random chance from too small of a sample size (more on that in a bit).
These groups will be shown different marketing elements so you can discover how these elements perform. There are a few essential components to this split to ensure that the test accurately represents customer behavior:
- Customers do not know they are being split — If they knew they were part of a test, it would influence their behavior (much like focus groups, surveys, etc.). They are simply going through their daily lives and making purchase decisions. In fact, you’ve probably been in a sample for many A/B split tests in your life without knowing it.
- Customers are randomly split — It’s important that the different groups exposed to the different marketing elements in your test are similar groups. You wouldn’t want to, for example, send all traffic from email to one headline you’re testing while sending all traffic from social media to a different headline. In that case, the difference in results you experience may be the result of the different type of customer seeing each one, not because one headline performs better than the other with all customers.
If customers are not randomly split, this represents a validity threat (i.e., a threat to the accuracy of your test results). This particular threat is known as a selection effect.
This random split is another reason why A/B split tests are more accurate than surveys or focus groups or similar types of marketing research. These types of research are inherently vulnerable to selection effect. In these cases, you are getting information from customers who self-select to take a survey, participate in a focus group or engage in other types of research and therefore may be more motivated — and otherwise different — from your customers.
Since customers do not know they are being split — they are simply going about their regular purchase routine and other behavior — A/B split tests happen with customers who are identical to your customer base because they are, after all, your actual customer base.
- There are enough customers in the sample — For example, if you only tested with two customers, those two customers are much less likely to represent all your customers then if you tested on one million.
Those two customers might just happen to dislike the color blue, and you have a blue car pictured on your landing page, so they bounce. However, that minor detail might not matter to most of your customers. If you take what you learned from only two customers as accurately reflecting all your customers, you would be severely misled.
If you tested on a million customers, there is much less likelihood of a random factor like this leading you astray.
That is an extreme example of course, but it’s important to understand testing sample size and how many interactions you need for a statistically significant result.
Behavior Recorded
After your customers are split, half will be shown Version A (usually the “control,” i.e., what you were doing before) and half will be shown Version B (usually the “treatment” or the new change you’re trying to test).
However, you could test more than simply one control and one treatment. You could test one control and two treatments. You could simply test four treatments. Just make sure you have enough traffic or other behaviors so that the sample size is statistically significant, as mentioned above.
Think about it this way. If you only have a control and a treatment, and you split 2,000 visitors per day in half, then 1,000 visitors would see each version. However, if you have 10 treatments, now only 200 visitors per day see each version.
Before the rise of the internet, recording these behaviors was quite a long project. For example, you might have to write, print, send and then wait for responses to two sales letters you were testing.
With digital A/B split testing, you’re able to garner real-time findings from your data. This allows you to quickly gain a new customer understanding and use that understanding to better serve customers and improve results. Just make sure you run the test long enough to accurately represent customer behavior.
Proceed to Next Step in Funnel
It is much more complex to test multiple steps in the funnel at the same time, and this can raise validity threats. So look at your analytics, and determine where in your funnel it would be most impactful to test (e.g., a landing page). And then send those conversions back into your regular funnel. For example, a shopping cart after the landing page.
You can follow Daniel Burstein, Senior Director, Content & Marketing, MarketingSherpa and MECLABS Institute, on Twitter @DanielBurstein.
If you’re reading this blog post, you are likely interested in increasing conversion through testing and optimization, so —
You Might Also Like …
Online Testing certification course — Learn a proven methodology for executing effective and valid experiments from MECLABS Institute, MarketingSherpa’s parent research organization
Online Testing: Microsite A/B Split Test Increases Lead Rate 155%
A/B Split Testing: How To Use The Right Test Progression To Get Customer Discoveries And Results
A/B Testing: Split Tests Are Meaningless Without The Proper Sample Size
Categories: Marketing, Research And Measurement A/B testing, Consumer Marketing, content marketing, Email Marketing, Marketing, marketing insights, testing
