There’s nothing quite like the excitement of running a split test. In fact, the excitement always seems to generate a flurry of great ideas for the next test. How do you organize and prioritize all of these thoughts into actionable test ideas? You create a conversion test scoring system.
What is a Conversion Test Scoring System?
The important thing to know about a conversion test scoring system is that it organizes, clarifies, and prioritizes your split test ideas. The actual system can vary depending on your business. Several industry leaders have documented their systems for scoring conversion test ideas, including Optimizely and Chris Goward. The conversion test scoring system I’m going to outline is a combination of those systems. It’s meant to be easy to use with simple yes/no questions. With this system the hardest part is coming up with the hypothesis, determining the prioritization is quick and simple.
How to Create a Conversion Test Scoring System
Chris Goward’s system uses a simple formula where you put in a value for the potential (P), importance (I), and ease (E) for each idea. The result is a numeric value representing the average of those numbers, PIE. This is a nice system and a great way to think about your test ideas. However, the problem I’ve had with this system is it’s subjective. I’ve found that the PIE metric doesn’t accurately prioritize the test ideas unless I review and adjust the parameters for each test every time. The values seem to vary depending on my mood, data from recent split tests, and changes to business priorities. So, it’s important to review and update all metrics when any new data is added.
Optimizely has created a different system. Their system looks at ten different rules. If a test idea meets a rule, it gets one point, otherwise it’s assigned zero points. The points are added together and the idea with the largest sum is the highest priority.
I liked the idea behind Chris Goward’s system, and the simplicity of a yes/no system introduced by Optimizely, so I made a system that combines these two great tactics.
This combined system looks at a total of nine rules that are assigned one point if an idea meets the requirement, and zero points if it does not. Three of the rules identify the potential, four of the rules determine the importance, and two of the rules measure the ease. We then calculate a PIE metric by taking the sum of the average of the three categories, and dividing by 3.
| Category | Rule | 1 Point | 0 Points |
|---|---|---|---|
| Potential | Fold | Makes a change above the fold | Makes a change below the fold |
| New Information | Adds new information or a new element or removes an element from the page | Makes a change to the existing elements (copy, color, UI, etc.) | |
| Benchmark | Borrows from a success on one of our prior experiments | No benchmarking best practice | |
| Importance | Main Metric | Supports our main metric (such as payment) | Supports a secondary metric (such as subscribe or share) |
| Location | Tests a change at the end of the funnel process | Tests a change located on a landing page, or top of the funnel | |
| Targeting | Targets 100% of users | Targets a subset of users | |
| PPC | Could help reduce the CPA for PPC campaigns | Has no impact on PPC | |
| Ease | No Developer | Doesn’t require a developer to implement | Requires a developer |
| Length | Can be implemented in an hour or less | Would take more than an hour to implement |
The rules you follow could vary depending on your product. The important thing to remember is to determine the rules you want to follow to calculate the potential, importance, and ease of your test ideas. Creating concrete rules will allow you to determine the PIE metric without being subjective.
Additional Items to Include with the System
Creating and following the rules to score your test ideas is important. Of course, there is a little more to a conversion test scoring system than just the rules outlined above. Let’s discuss some of the additional fields you should include.
Submitted By
I always like to include a column that documents who submitted the test idea. This helps establish a data-driven culture in your workplace. It’s important to encourage everyone at the office to submit test ideas. If the test is implemented, share the results and findings with the person who submitted the idea. If a test is successful, you can share the results with the entire company and give some credit to the idea submitter.
Hypothesis
The test idea should be written in the form of a hypothesis. The idea should be structured in an If…,then… statement. Creating a hypothesis helps you identify the underlying reason you are performing your test.
Action
Once you have a hypothesis, you can identify the specific action you want to take to test the hypothesis. Structuring it in this manner provides an easy way to progress after a failed experiment. If a test fails, the action you took wasn’t the correct one. But, do you still believe in the hypothesis? If so, determine another action you can make to test the hypothesis.
Page / Location
Document the specific page you will be testing.
KPI
Identify the Key Performance Indicator (KPI). This will be the main metric you track to determine if the test was successful or not. It’s best practice to track multiple KPIs, but you need to understand the primary KPI this test will affect.
Keep in mind, you want to choose a KPI directly related to the test idea. If the idea tests an element on the homepage which is at the very top of the funnel, choosing payment or a KPI that triggers at the end of the funnel is likely not a good choice. Users will encounter many other variables in the middle of the funnel that could skew the data.
How to Calculate PIE
So, now you have documented the person who submitted the idea, created the hypothesis and action, determined the page and KPI, and determined if each rule receives one or zero points. Now, the PIE metric to determine the priority needs to be calculated.
PIE=((((Value of Fold + New Information + Benchmark)*10)/3)+(((Value of Main Metric + Location + Targeting + PPC)*10)/4)+(((Value of No Developer + Length)*10)/2))/3
The PIE calculation can vary depending on how you adjust the rules. You want to calculate it by taking the sum of all values within each category and multiplying the result by ten. Then you divide the result by the number of values within the category. You add the calculated value for each of the three categories, and then divide it by three. Once you have the final metric, you’ll want to sort the list from highest to lowest value. The higher the PIE metric, the higher the priority of the idea.