Split tests are fun. They allow you to test your assumptions and make changes that you can be sure make sense for your audience. A split test is when you have two variations of a page on your site (or the entire site) running at the same time. You send half of your traffic to one version, then the other half to the other version. A split test tool tracks results and shows which variation produced more conversions. But, how much traffic do you need to run a split test? We’ll go over the details you need so you know when you’re ready to get started with split testing.
Any amount of traffic is technically enough. Split test tools don’t have limitations. The problem is, with few visitors it can take too long to get a statistical significance. For the results to be statistically significant, you want the results to be at least 90% confident. (And 95% would be even better.) What a confidence level of 90% means, is the declared winner will be the winner 90% of the time.
You also want the test to last no longer than 4-6 weeks. After that time people start to delete cookies and results begin to deteriorate. So, there is a certain amount of traffic required to have a large enough sample size to hit a statistical significance on time.
At a minimum, you want to have 500 conversions per month before you start running split tests. Some people say 500 conversions per week, or 5000 visitors per week. But, if you have 500 conversions per month you should have enough data to get started.
The more variations you test simultaneously, the more traffic you’ll need. So, if you have 500 conversions per month, that’s roughly 250 for each variation. If you want to run more than the original and one variation, you’ll need more conversions.
How to Run Split Tests with Less Traffic
Test Micro Conversions Instead of Macro
If your primary conversion point (macro conversion) doesn’t have enough conversions to run a split test, try a smaller (micro) conversion. So, instead of split testing your payment funnel, try split testing your lead magnet. Having people sign up for your newsletter may not be your end goal. But, if it’s getting more conversions that may be a good goal to try and improve with a split test. Improving that stage could have a trickle down effect and increase conversions lower in the funnel as well. So, you can get closer to the conversions needed to split test your primary conversion point.
Test Radical Changes
If you have low traffic and few conversions, don’t split test small changes. Skip testing button colors, button text, and headlines. Instead, make drastic changes. Completely redesign the page. It will be harder to pinpoint the exact change that improved or decreased conversions. But, since the performance change will be drastic, a conclusive result will be easier to reach.
Combine with Qualitative Data
If you have low conversions, you can combine the split test results with data from other conversion optimization tools. Surveys and qualitative data could provide insight into why the split test results look the way they do. Analyzing heatmaps of the variations could also provide insight to help solidify the results.
In Summary
Without conversions, split tests will be useless because you won’t have data to determine a winner. Once you have about 500 conversions a month, start running split tests. And while you’re getting there, it doesn’t hurt to start listing and prioritizing split test ideas.
Do you want to listen to this article? Here’s the podcast episode: