<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=1375589045796407&amp;ev=PageView&amp;noscript=1">
What are the best KPIs to measure a split test?

What are the best KPIs to measure a split test?

A/B split testing is the secret sauce that helps give successful e-commerce companies a healthy year on year growth. It’s fundamental to Conversion Rate Optimisation (CRO) too.

That’s why the metric often used to judge the success of a split test is conversions ie: how many visitors were turned into customers.

But there’s a catch. Conversion rate isn't necessarily one of the best KPIs to measure a split test. Sometimes conversion rate falsely leads you to think you have a winning strategy when you don’t. Or it doesn’t help you spot sales-increasing winners. And that means it can be costing you money.

A better KPI to measure a split test

If you want to increase your overall turnover, there’s a much better metric to use. It focuses on one issue only – what’s really going on to your bottom line.

It’s a little-used yet highly valuable metric, called Revenue Per Visitor or RPV.

Why is RPV so much better than conversion rate?

Quite simply, conversion rate doesn’t give you the whole picture.

The way most people go about improving their e-commerce sites is to perform a simple A/B split test, launching two different configurations of the site and showing them to different visitors.

Then, they use the metric of conversion rate (ie: the percentage of visitors to the site who buy something) to determine which of the two branches of the split test produces more revenue. The higher the conversion rate, the better, right?

Not necessarily. You can have more visitors turn into customers, but if they’re all spending less, then you’re actually no better off (See example 3 below)

What is Revenue Per Visitor?

When you factor in Revenue Per Visitor, you have a much more accurate idea of which version is making more money, because RPV brings in more facets of e-commerce. And they’re also the facets that really matter.

Think of it this way: is it really the number of visitors who spend money that matters? Of course not. It’s how much they spend.

So why spend so much time simply measuring the number of visitors? Revenue Per Visitor lets you really increase sales because it factors in the bottom line – how much, on average, each visitor is spending.

Free Guide: How to Make Split Testing Work For You

Why RPV is a more powerful metric

Let’s look at some specific examples where RPV is better than just using conversion rate on its own. These hypothetical situations demonstrate how RPV can give you much more useful information than conversion rate.

Example 1: Revenue goes up but conversion stays the same

  Baseline Split Test
Visitors 100,000 100,000
Conversion Rate 2.0% 2.0%
Orders 2,000 2,000
Average Order Value £50.00 £62.50
Revenue £100,000 £125,000
Revenue Per Visitor £1.00 £1.25
Overall Improvement -- 25%


In this first example, the split test didn’t show an improvement in Conversion Rate at all, so if that’s all you were measuring, you probably wouldn’t permanently implement the tested change.

The Revenue Per Visitor advantage:

Using RPV shows that the Average Order Value went up by 25%, resulting in a 25% increase in sales revenue. This is a change worth keeping. If you’d only relied on conversion, you would have lost £££s


Example 2: Revenue goes up even though conversion drops

  Baseline Split Test
Visitors 100,000 100,000
Conversion Rate 2.0% 1.6%
Orders 2,000 1,600
Average Order Value £50.00 £75.00
Revenue £100,000 £120,000
Revenue Per Visitor £1.00 £1.20
Overall Improvement -- 20%


In this case, conversion rate went down by 20%, which would cause many e-commerce professionals to panic and abandon the tested change. However, the RPV shows a 50% increase in Average Order Value, meaning that the change actually improves upon the baseline by 20%.

The Revenue Per Visitor advantage:

Not looking at RPV can cause you to reject a winning strategy without even realizing it. Not looking at RPV can also cause you to undervalue strategies, even if you recognize them as winners. Another example where ignoring RPV can lose you £££s


Example 3: Conversion goes up but revenue stays the same

  Baseline Split Test
Visitors 100,000 100,000
Conversion Rate 2.0% 2.5%
Orders 2,000 2,500
Average Order Value £50.00 £40.00
Revenue £100,000 £100,000
Revenue Per Visitor £1.00 £1.00
Overall Improvement -- 0%


Looking at conversion rate, this change gave a 25% increase. However, we can see that the total sales revenue never changed. This is because the Average Order Value decreased by 20%.

The Revenue Per Visitor advantage:

On balance, the overall improvement offered by this strategy that looked like a winner was actually 0%. Looking at RPV here would have saved you from investing in a dud.


Know the true impact of your A/B testing

In the end, the more information you can arm yourself with, the more successful you’ll be in e-commerce. In order to succeed, you need to know not only how to conduct analyses such as split-testing, but also the proper quantitative metrics you need to measure in order to know for sure whether or not your actions are really increasing value.

If you're looking for further help with split testing, download our free guide to learn how you can make it work for you.

New Call-to-action

Posted in: A/B Testing

If you enjoyed reading this, you might also like:

Request a free CRO consultation

If you’re serious about implementing website improvements that deliver massive sales increases we’d like to offer you a free, no-obligation, 60-minute CRO consultation.

Scott Liebenberg
"Having the free consultation with AWA was invaluable. They took the time to analyse my website and give me detailed feedback on opportunities. They were knowledgeable and easy to talk to. Couldn’t recommend highly enough."

Scott Liebenberg, Founder and Owner, Tapered Menswear.

Here’s what we’ll cover on your free 60-minute call:
  • Exactly how to start running an effective CRO programme running in your business
  • Or, if you’re already running an in-house CRO programme, we’ll talk about how to improve and troubleshoot your process
  • The top conversion mistakes most online businesses make, and how to avoid them
  • 5 recommendations tailored to your business that you can implement now for fast results

Get a Free Eyetrack

A visual map of your landing page

Yes Please show me what
my visitors look at

Thanks - but no thanks