What are the best KPIs to measure a split test?

A/B split testing is the secret sauce that helps give successful e-commerce companies a healthy year on year growth. It’s fundamental to Conversion Rate Optimisation (CRO) too.

That’s why the metric often used to judge the success of a split test is conversions ie: how many visitors were turned into customers.

But there’s a catch. Conversion rate isn’t necessarily one of the best KPIs to measure a split test. Sometimes conversion rate falsely leads you to think you have a winning strategy when you don’t. Or it doesn’t help you spot sales-increasing winners. And that means it can be costing you money.

A better KPI to measure a split test

If you want to increase your overall turnover, there’s a much better metric to use. It focuses on one issue only – what’s really going on to your bottom line.

It’s a little-used yet highly valuable metric, called Revenue Per Visitor or RPV.

Why is RPV so much better than conversion rate?

Quite simply, conversion rate doesn’t give you the whole picture.

The way most people go about improving their e-commerce sites is to perform a simple A/B split test, launching two different configurations of the site and showing them to different visitors.

Then, they use the metric of conversion rate (ie: the percentage of visitors to the site who buy something) to determine which of the two branches of the split test produces more revenue. The higher the conversion rate, the better, right?

Not necessarily. You can have more visitors turn into customers, but if they’re all spending less, then you’re actually no better off (See example 3 below)

What is Revenue Per Visitor?

When you factor in Revenue Per Visitor, you have a much more accurate idea of which version is making more money, because RPV brings in more facets of e-commerce. And they’re also the facets that really matter.

Think of it this way: is it really the number of visitors who spend money that matters? Of course not. It’s how much they spend.

So why spend so much time simply measuring the number of visitors? Revenue Per Visitor lets you really increase sales because it factors in the bottom line – how much, on average, each visitor is spending.

---

---

Why RPV is a more powerful metric

Let’s look at some specific examples where RPV is better than just using conversion rate on its own. These hypothetical situations demonstrate how RPV can give you much more useful information than conversion rate.

Example 1: Revenue goes up but conversion stays the same

	Baseline	Split Test
Visitors	100,000	100,000
Conversion Rate	2.0%	2.0%
Orders	2,000	2,000
Average Order Value	£50.00	£62.50
Revenue	£100,000	£125,000