Local maxima and the value of being bold

Split and MVT testing offer two powerful benefits to those wishing to optimise their online channel. Yet to get the most out of split-testing you need to understand about local maxima and the value of being bold.

These two benefits are:

• Irrefutable Proof – Since split testing concurrently serves different web pages to a random sample of your website audience and compares the results, it is proves beyond reasonable doubt that one version over the others produces superior results.

• A learning opportunity – The results from split-tests provide a valuable opportunity to learn about the behaviour and preferences of your website visitors. Depending on your split-testing tool, you will also be able to analyse how the tests performed with new vs. returning, branded vs non-branded, paid vs. organic, etc.

In this post, I examine the issue of local maxima and being bold in your tests for both those who run websites with large volumes of traffic as well as those who have a more modest number of visitors.

Websites with high numbers of visits

If your website receives hundreds of thousands of visits per day, then you have the opportunity to learn quickly and benefit from statistically significant trials. If this is you, then your challenge will be that of hitting what Andrew Chen has called “local maxima”.

This is the point where, using incremental improvements you have reached the maximum performance of the website in its current form. After this point there are diminishing returns in continuing to optimize in this way.

Your approach now is no longer to tweak the current design, but to take a new approach to your audience and website.

Smaller websites

For websites receiving a lot less visits, then your issue is not so much of hitting the point of local maxima, but the speed with which you can get the results from split-tests.

Let’s look at an example to show you what I mean.

This fictitious e-commerce website receives 9,000 visits and has a conversion rate of 1.82%.

Let’s suppose we do a split-test, with just two variations – one new version and the other being the control. How long will it take to get statistically significant results, in other words at a confidence level of 95%?

Conversion rate improvement	Maximum duration of split test	Average duration of split test (approx.)
5%	77 days	39 days
10%	19 days	10 days
20%	5 days	3 days

Note: This data is based on Visual Website Optimiser’s Split Test
Duration Calculator

You can see the table above that if your split-test only resulted in 5% improvement in conversion rate you might have to wait for as long as 77 days to call the result – remember, you are waiting for 95% confidence levels. In all likelihood you should be able to declare a winner in 39 days – but that’s still 8 weeks, and you are only testing one variation against your current page.

However, if you wanted to get your results far more quickly, you need a far higher level of improvement in conversion rate. We will look at what this would mean later on.

With 4 variations tested, that is 3 tests and 1 control, then the duration of split tests increases substantially.

Conversion rate improvement	Maximum duration of split test	Average duration of split test (approx.)
5%	153 days	77 days
10%	38 days	19 days
20%	10 days	5 days

With only a small improvement in conversion rate, then you are likely to be waiting for 11 weeks to declare which of your four variations is the ‘winner’. That’s a long time, for not much uplift. You are likely to have spent time and money setting up the test, waited a long time to release the results and then the results aren’t that startling. Disappointing.

Clearly, if you had hundreds of thousands of daily visitors, then a 5% uplift is certainly worth having, but in this example, you don’t.

The value of being bold

We all know that the bigger risk you take, the bigger the potential reward.
Conversion rate optimization is no different.

If you are larger website you can take small, incremental steps in improving your conversion rate, but over time this form of tweaking will reach a point of local maximum.

For smaller websites, given your traffic levels and the resources required to set up a split-test, you need substantial improvements in order to declare a winner is a reasonable time frame.

So for both types of website there is clear value in taking a new approach to split-tests. But let’s be clear that being bold, or taking a very different approach to your website visitors, has a greater chance of the control outperforming the test variation(s) – but you knew that, right?

Up until now, you have been changing a few elements of the existing design. To be bold you need to question your approach to both your audience and your offer.

Useful questions might include

• How well understood is my value proposition?
• How have my competitors approached their customers?
• Why do single purchasers not make more than one purchase?
• How profitable is my current customer base, are there other segments that could be more profitable?
• Why do visitors come to my website and what are they trying to achieve?
• What differentiates my high-spending customers from my low-spending ones?
• Does the way our merchandise is categorized represent the way in which our customers group our products in their mind?

Having answered these questions, ideally using a mixture of quantitative and qualitative methods, you are better able to develop new, bold designs.

This level of analysis is far more likely to help you to develop greater insights, interesting hypotheses and re-alignment of current thinking than changing the colour of your Add to Basket button.

As Joshua Porter says on his 52 Weeks of UX blog: –

And when the time comes to make the bigger changes, when you decide to jump from your local maximum to some other design possibility, make the decision with conviction. But don’t forget that the optimization has only just begun.

Please let us know what you think about this post by adding your comments below.