Why Amazon does ‘Experimentation’ not ‘CRO’ and why you should too

Experimentation, not CRO, is the strategy of the world’s leading companies to deliver massive growth. They are using experimentation to move from doubt to certainty – and you can too.

In this 45-minute webinar Dan Croxen-John shares with you:

The webinar was filmed on 3 December 2019.

Below is the transcript of the webinar.

Webinar Transcript

Good morning, good afternoon, good evening, wherever you are in the world.

We’re gonna wait a few more minutes for other attendees for this webinar to join, so just bear with us, thanks.

Just to let you know if you just joined, we’re waiting for a few more attendees to sign in and then we’ll be starting very shortly.

Every year, Jeff Bezos writes a letter to his shareholders. And these letters provide a really clear insight into the principles by which he runs his business.

One of these letters contained this statement:

“If you double the number of experiments you do per year, you’re going to…”

Can you guess what Jeff thinks you will double? Okay. Here’s for the reveal is you’re gonna double your inventiveness.

This is a major reason behind Amazon’s success: its ability to be continuously experimenting, both at the level of new products and services, but also with user experience. For Bezos, experimentation is not a way towards the strategy. It is the strategy.

Before I start, if you have any questions about what I’m saying, or something’s just not clear, just pop your questions into the chat box now, on Zoom, and then ask your questions. And if I can’t break for it there and then, I will come back to it by the end of the webinar. But just so I know that everyone’s found their chat box, could you type into it, “Hi, Dan”? Could you do that now please? Hi Eddie, Hi Lucas, Hi Simon, Hi Jim, hi Ian, hi Caroline, hi Craig, hi Haley. Thanks very much. You’ve got your chat box to work, that’s great.

Just to return to Jeff. You might be saying to yourself, “Look, I’m not Amazon. “I don’t have the same financial muscle. “My margins are much tighter. “We’re a multi-channel business,” etc. etc.

However, the principles of the Amazon story apply to you whatever the size of your business or the sector you find yourself in.

It is about moving from doubt to certainty, certainty that the decisions you’re making today will deliver the results you’re looking for a year from now. So if you’re a small business, then your strength is your agility. If you’re a larger business and follow my advice, you’ll be in a better position to avoid the fate of others who have not understood the power of experimentation.

Can anyone think of other companies, other than Blockbuster, that failed to innovate and respond to change in customer behavior that are no longer with us? Can you put the company name in the chat box now? Woolworths, HMV, Kodak, several votes for Kodak. Mothercare, still with us, maybe, just about. No they’re gone, sorry. Recent news. Kettler, Large German fitness brand. Blacks and Millets.

We all know businesses that haven’t adapted to customer, change in customer behavior. And so what I’m gonna be talking about today is why Amazon does experimentation, not conversion rate optimisation or CRO as I refer to it, and why you should do too.

The things that we’re going to cover:

Again, if you’ve got any questions or anything’s not clear, just pop your question in the chat box and I’ll come to it as soon as I can. Alternatively, it will be at the end of the webinar.

The problems with the conversion rate as a metric

We see every day an unhealthy, in my view, obsession with conversion rate. People quote figures at one another and refer to industry stats like this one. And yet this figure is an average, it’s a figure in aggregate. One number to describe the actions of thousands or even millions of website visitors. The metric is just way too narrow and difficult to action. Look below the surface of this figure and you’ll see a much more complex picture. Varying conversion rates by traffic source, by visitor, by visitor type, by visitor to certain products or services, categories, or landing page.

Describing your website visitors simply by their conversion rate is like describing a population by its average height. Many people have got it into their heads, this is the number two problem with CRO the metric, they’ve got it into their heads that they can increase their conversion rate from 1% to 2% then they’ve doubled their revenue.

What’s wrong with that mindset is there is an implicit assumption that your conversion rate is a fixed number. In this case, 1%, and you want to increase it to another fixed number, 2%.

The reality is that conversion rate varies widely.

This graph shows that conversion rate varies by about 12 times. So it’s not such a fixed number. And by the way, this isn’t a small website taking this data from. Third problem, so let’s look at this situation. You’ve run an AB Test, you’ve split the traffic down the middle. Variation A produces 25%, a 25% increase in conversion rate, right, so it’s a clear winner. No, it isn’t unfortunately. When you add in average order value, we see that Variation B produces more total sales and it has a higher revenue per visitor, or RPV. Had we declared A the winner, and not B, then actually we could be harming sales, rather than helping them.

RPV is a better metric for that reason that it includes average order value and conversion rate but it’s not perfect either.

The other thing we see repeatedly is people chasing conversion rates. And what this results in is them doing some things that often just don’t make sense. Too many times I’ve heard people talk about copying the check-out of a competitor because they like it. What they might be copying is the losing variation of an AB test, a variation that actually would hurt sales if it were rolled out.

Alternatively, rather than copying competitors, some CRO practitioners get obsessed with, let’s call it slavishly following, often without testing, these checklists of best practices. They’ve been entranced by promises of double digit conversion rate increases. And the fact is life’s not that simple. Copying things generally doesn’t work because it’s contextual. Your website visitors are different to your competitors and different again according to what season they might be in and so on. So there’s no quick fix unfortunately.

The fifth problem is this. It’s about goals. Our goal, your goal, my goal is to grow revenue, our client’s revenue or our own revenue. But looking at this goal tree, you can see that conversion rates, which is here, is not directly related to revenue, at the top of the tree here. It’s only one contributor amongst several. So what this goal tree shows you is that you can’t directly affect conversion rate, only the lower level metrics. And what that means particularly is you can’t take conversion rate to the bank, only revenue.

Five easy ways you can double your conversion rate

Here’s five ways that you can increase your conversion rate, you can probably double it at least.

All of these ways of increasing conversion rate are basically fiddling with the numbers. Your increased conversion rate won’t actually translate into more pounds and pence. So be cautious of people who quote conversion rate increases. Ask them to explain a number of factors like I’ve just raised.

Here is the problems with CRO, the practice. I’ve talked about the metric, now I’m going to talk about the practice.

So often, CRO is seen as a fix, a one off project, like a mechanic having a go at your broken car. I often hear the phrase, “Yeah, “we’ve done a bit of CRO and the results weren’t great.” These quick wins or low hanging fruit were supposed to double their conversion rate, we know the problems with that metric, and therefore double their turnover.

The issue is this, optimisation is a process by definition, it never stops. To stop optimising suggests that the entire world around you has stopped evolving and you just don’t have to adjust anymore.

A second problem is this. I don’t know if any of you remember a website called Which Test Won? What they did is publish case studies of AB tests and asked you, the visitor to the site, to guess which one of those, which one was the test that won, was it A or B before they revealed the answer. The danger to presenting testing in this way is that it encouraged users of the Which Test Won website to see optimisation as purely about pages, about moving things around on a page, and really paid scant attention to the view that whilst landing page optimisation is a good start, it’s not the complete picture.

Often the way that CRO is currently practiced is seen as a channel, similar to pay per click or SEO or email. It’s all competing for the same budget. The comparison with these acquisition channels results in a world being seen as a simple funnel, whether it’s a click from a paid ad to a landing page or progression from one step of the checkout to another. This simple funnel view ignores the impact of other important factors, such as customer service, merchandising, and competition.

And the fourth problem is this. A chap called Eric Ries, who wrote “Lean Startup”, he said the fundamental goal of entrepreneurship is to build an organisation under conditions of extreme uncertainty. So despite it’s power, it’s grounding in science, and access to cheap or even free tools, CRO, as it is too commonly being performed, focuses on lower level metrics.

In my view, it lacks ambition and risk-taking. Ultimately, we are trying to grow our business, so not simply focusing on reducing cart abandonment rate. Improving your checkout, would distract you from the broader question of, how do I grow my business?

This myopic approach, in my view, is delivering a generation of tweakers and optimisers rather than what business needs which are entrepreneurs, innovators, and experimenters.

So what is experimentation, the way that Amazon, Booking.com and some of our clients, Canon, Avis and Interflora practice it? And how can you do this too?

Well one of the key concepts is around innovation and failure. Firstly what these businesses understand is that to stand still is to die. We’re living in a time of extreme uncertainty and, that’s only set to increase. To stand still is a failure to innovate and learn, but, and here’s the dichotomy: you are gonna fail when you innovate.

One of the key areas of difference with a key concept within CRO is that of reversibility.

Because, within experimentation, you don’t know if your innovation’s gonna work out. So let me read to you what Jeff Bezos had to say about the concept of this reversibility. “Some decisions are consequential and irreversible “or nearly irreversible. “One-way doors. “And these decisions must be made methodically, “carefully, slowly, “with great deliberation and consultation. “But most decisions aren’t like that. “They are changeable, reversible. “They’re two-way doors, “with reversible decisions, “the difference is that you only keep the winners.” This applies to your website yes, but it also applies to other facets of your business. For example, product development.

Here’s some areas in which experimentation is different from CRO.

If you focus on growth and you develop experiments to test innovation, you need a wider set of metrics. So let me give some examples. So on the left hand side, we’ve got the number of metrics, on the y-axis, the scope of the business question that we wanted to answer. So if all we’re trying to do is understand what is the impact of changing a button color, and I’m using a rather extreme example to make my point, then conversion rate or RPV might be a sensible metric to use, and just those two would be fine. But if you’re asking yourself or asking the business, how can we retain more customers, what products should we develop, then you need a broader set of metrics. Customer lifetime value, net promoter score, gross profit, etc. So your, your metric base becomes far larger, broader, and richer than if you do this plain, old CRO. The other thing about experimentation is it has a much wider purview than simply converting more visitors.

We recently ran an experiment with one of our customers about reducing their product range. We had the hypothesis that too much choice can paralyze customers. In fact by running this slimmed-down product range, it increased revenue by about 6%. But more importantly, it allowed the business to simplify the logistics. They had less SKUs to manage after all. They had greater purchasing power with their suppliers and ultimately could reduce the complexity of their operation. So experimentation works across business functions. Conversion rate generally is just a tool of the eCommerce team.

So the literature around lean startup, I’m gonna give you a small reading list at the end, allies experimentation with product development, a product that is designed to meet customers’ needs. So your experiment is building up towards the next version of your product, one that will better meet the needs of the previous version because you’ve validated it each step of the way. CRO is often concerned with tweaking, the existing product, in this case, a website.

Experimentation is about co-creating products with your customers without asking them directly what they want. So Bezos wrote to shareholders in 2018, he said, “No customer was asking for the Echo. “This was definitely us wandering.” What he went on to say is, “Market research doesn’t help. “If you’d gone to a customer and said “‘Would you like “‘a black always-on cylinder that plays music, “‘sets alarms and timers, “‘you can talk to and ask questions of, “‘turns on the lights and so on,’ “our customers would have said, “‘No, no thank you.'”

Case Studies in Experimentation

Can you think of companies that have acquired a reputation for experimentation? ‘Cause I’m gonna talk about three but whilst I do have a think about others that you know of that have experimented and adapted to change in customer behaviour.
The one I want to talk to first is about Dropbox. And the purpose of this slide is to talk about minimum viable experiments, to give you an idea of three different types of companies and the way that they validated their core assumptions about their business, about their possible business, and tested it with customers.

Dropbox was founded by a guy called Drew Houston, and Drew was getting increasingly frustrated when he was at college that he would get to college, get off the bus, and find that he’d left his USB flash drive at home. This would happen many times. And he was thinking, “Well,” this is when he had spawned the idea of developing Dropbox, the file synchronisation software. But in order to get started, he developed a video of him using the software. And he posted this video on an early version of Reddit. It included lots of in-jokes aimed at the developer community because he believed that these were gonna be the early adopters of his software if it was gonna be successful. So this was his version of a minimum viable experiment. So he launched the video and by the end of that day, he had 75,000 sign-ups for his free version of Dropbox.

Netflix. Now before launching Netflix, Reed Hastings wanted to challenge a core assumption that underpinned the business model. But what they did is first of all consider and ultimately rejected VHS tapes. They were too expensive to stock, too delicate to ship. So when DVDs came along, the question was: have I got a business here? And one of the fundamental questions he needed to ask himself is: when I send this thing through the post, will it play? Because if it doesn’t, I don’t have a business. So that’s why they rejected VHS because it’s too delicate. So when DVDs came along, they actually sent compact discs to his house in Santa Cruise. When the disc arrived intact, and one of the core assumptions that they would be playable, was confirmed, they could then move from that place of doubt to one of certainty.

Zappos. A guy called Nick Swinmurn was the founder of Zappos. He felt, believed that there was an opportunity to exploit the 40 billion dollar market in shoes, especially since 5% of those sales were already going through mail order. But before he did, before he committed to the business in such a huge scale, he took every single order that they received on their ugly, early-days website, went down to the mall, bought the shoes for the customer, and posted them. So at that stage, the business wasn’t making any money. But again, he was moving from doubt to certainty, that this was an attractive market to enter. This was his minimum viable experiment.

Can you think of any other company aside from Amazon and the others I just mentioned, that have got a reputation for adapting to changes in their customer base? Just put the name of the company in the chat now if you can.

ASOS, Netflix, thanks. Secret Escapes from Caroline, thank you. Wall Street Journal.

A fifth important difference between CRO and experimentation is about how wide it operates within a business. So experimentation is a company-wide endeavor. It can’t be devolved to a single department. We know of one enterprise in the travel sector that has set up its own proposition team about 18 months ago. Almost all of its innovations have been killed by office politics, endless refinements. The net result, the rate of innovation is at a snail’s pace because what they’ve done is devolved innovation and experimentation to a single department.

And actually, I’ll talk about this later on, but Booking.com is a very fine example of a company that spreads out the responsibility for experimentation across its staff.

I’m going to give you some examples of businesses that have used experimentation to start the process of moving from doubt to certainty.

This is a business called Village Laundry Service. It was started by an Indian, ex-Proctor and Gamble brand manager. He was interested in whether he could launch his own village laundry service. At the time, only 7% of the Indian population had washing machines in their homes because of the high cost. The only other option they had was dhobis, who put their clothes into the nearest river, banged it against a rock, dried it, and then brought it back, often about 10 days later, and possibly not that clean. So what Mira did, the CEO of VLS he tested many of the assumptions that he had about the business. But the primary assumption he wanted to see whether it was correct, is whether people would be willing to have their laundry taken away, cleaned and returned to them, cleaned using washing machines that were on the back of these almost carts that he took around with him. The only way he discovered whether or not it would work, and it did, was he got out of the building. He interacted with paying customers. He built a prototype, putting a washing machine on a cart and taking it around villages in India.

Booking.com I mentioned just previously. Used to be a tiny business, now 17,000 employees. It’s built its own in-house experimentation platform similar to Optimize or Qubit or Test and Target. But it’s in-house. Over a thousand experiments are launched every single day. And of course, they’ve got the traffic to run these experiments. But the point is this, 75% of staff have been involved in running experiments, from all kinds of departments. And even more interestingly, 90% of these experiments emanate from teams rather than individuals, which says to me that actually some of the best ideas or hypotheses are developed collaboratively with colleagues and other team members.

See Amazon and Booking.com embrace failure as part of the process.

Amazon has had some high-profile failures such as the Fire Phone, A9 and Endless Shoes which was based on Zappos. But these unsuccessful ventures haven’t stopped them from growing.

Edison would not have invented the light bulb if he hadn’t followed the same approach. He actually slept in the lab. But if you contrast that with the fortunes of Blockbuster, Kodak, Blackberry, and so on, then you can see what happens to those that embrace experimentation. Bezos says that given a 10% chance of 100 times payoff you should take that bet every time. But you’re still gonna be wrong nine times out of 10.

Five actions you can take starting today.

The only way to win is to out-experiment your competition

Imagine a world where your boss or the board asked you not, “What’s our conversion rate?” But instead asked, “How many experiments did you run last month? “What did you learn? “What action are you gonna take?”

Let’s just have a think about this together. In your chat window, tell me in one word what is the single biggest challenge to get experimentation going, or at least working the way you want it to work in your business. Just put one word that describes that challenge in the chat box now. Exec buy-in, ROI, tools, time, sign off, from Gillian, thank you.

These are key obstacles that many businesses face. And, the one about exec buy-in is pretty key.

I know from listening to a presentation from a chap called, Guy at RBS who was a head of conversion there, is that they involved a lot of senior managers in adopting, they gave them a test to run and they allowed them access to the experimentation platform to see, so they could see the results that their tests were doing. So there’s lots of different ways you can get exec buy-in. In my view the tools are getting easier and easier, cheaper and cheaper, more and more powerful. So actually, in many cases the tools shouldn’t stop you. But we can come to that shortly.

Just we’re wrapping up towards the end now, leaving some time for questions. But, if you’re a CRO practitioner then all you need really to read is:

My reading list for you is this, is people that have written about business experiments. People like:

They come from the lean startup, product development. And product development is allying itself very closely with experimentation because you can get the tools and technology we’ve got now mean that we can get much, much faster results than we could 20 or 30 years ago.

And then our own book, “Ecommerce Website Optimization” which comes out in June of next year, which covers experimentation in quite some detail.

A summary. First of all, I think, focusing on conversion rate is going to distract you from growing your business. I think, as currently practiced, CRO is producing a generation of tweakers, not entrepreneurs or innovators. I think you have to realise that to innovate you have to fail and vice versa. But normally, as Jeff Bezos says, you have two-way doors to go through. If things don’t work out you can reverse your decision. And you only get to keep the wins.

Experimentation is much broader than CRO. It includes more metrics. It requires ambition, perseverance. It impacts many functions of the business, it involves new products or features, not tweaking existing customer journeys. And it’s also primarily about getting out of the building, not relying on market research. I don’t know whether Henry Ford actually said this or it’s more of a myth but, he said, “If I’d asked my customers what they wanted, “they would have said a faster horse.” They couldn’t conceive of a motorcar in the same way that people couldn’t necessarily conceive of the Amazon Echo before it arrived.

So start small, don’t waste time on huge, big prototypes that consume lots of time and resources. And show perseverance. And read about experimentation, because there’s lots of really interesting case studies about, not only online business, but offline businesses that have developed through experimentation they have adapted, that haven’t stood still, are still here with us today because that’s what they’ve done. And I wanted to share this quote with you.

This is Johann who is the co-author of the book, “At a time when the web is vital to almost all businesses, rigorous experimentation should be standard operating procedure.”

Webinar Q&A

So, I hope you found that useful. We’ve got some time left for questions. But these are the ways you can contact me, on LinkedIn, Twitter and email. So, I’m gonna throw it open to questions. I think some of you have submitted them already. And I’ll make sure I cover those.

So Jim asks, “We are familiar with the tools available for CRO, are there any upcoming experimentation-specific tools we should be watching?  It looks like some CRO tools are rebranding as experimentation platforms.”

Answer: You’re right, Jim, they are rebranding. But I think what, the underlying change that you’re seeing, particularly on a technology front, is that of a move toward server-side testing. Just to explain that to those that may not be familiar with it. So right now a lot of testing platforms like Google Optimize, it’s the browser that does the work in serving the variation. And what that means is, is that the things that you can test are often limited by the functionality you can get from within the browser. Server-side testing is actually when the variation is sent directly from your server and the client just receives it. What this means, server-side testing, and it’s now being offered by platforms like, Optimizely, and I think VWO as well has started doing it, is that you can easily roll out features far more complex programmatics and functionality than you could with client-side testing. And what that means is that you can run experiments which are far more complex, but also you can pull them back if they’re not working, very quickly. And the second thing I would say is, if they prove to be a winner, ie that you see an improvement with some of your metrics that you’ve chosen, then it’s easy to roll them out because you’ve done a lot of the programming up front, already. So I hope that hopefully that helps.

Question from Steven at Growth, “When looking at Google Analytics data what data range should you be checking e.g. last month’s, six months or a year?”

Answer: I think it depends on your business cycle. Some of our clients are businesses that do most of their revenue in quarter four of the year around Christmas. So I would always go for 12 months. If you’ve got a number of peaks within that, I would shorten that. But I think the important thing is is like-for-like samples. And I would say this, is that it’s probably more important to segment than it is to get the time horizon right. So start segmenting your Google Analytics data by traffic source, by device type, by visiting certain categories or key pages. That habit of segmentation will reveal a lot more underlying than sort of aggregate data giving you about conversation rate.

“Which UK businesses are really living and breathing experimentation?”

Answer: I think Skyscanner would be a really good example. Skyscanner is one of these unicorns that I think, worth over a billion pounds. I think, in many cases travel businesses like that fast-moving travel businesses, can experiment at some pace because people are reacting far more quickly, they’re comparing flights and holiday packages almost side-by-side. So I would say Skyscanner is one. I think Just Eat do very well. And I think those that adapt their, app optimisation is a new area that I think, some of the businesses who, where an app is part of you life, like delivery or Uber, where it’s part of the journey are doing very well at optimisation.

“Do I know any B2B businesses that do experimentation well?”

Answer: I think the answer is, companies like Cisco or ones where they’ve got large volumes of product aimed at businesses. RS Components would be another one. I think they do experimentation well. But it’s, I think in some ways, the scope of experimentation with B2B is often about the service rather than the exact product they’re asking for, and I think those two examples, are worth looking at. “Airbnb are great too,” thanks Hailey.

Terry asks, “It’s easy to answer or quantify the first question, ‘how many experiments did you run?’ But how are people answering the two other questions: “‘What did you learn, ” ‘what action “‘are you driving as a result of these experiments?”

Answer: I think that’s true. I think there is some times paucity of analysis in the test results. And that’s why I keep going back to this mantra of segments. So when you get your test results, if you just talk about, let’s say a revenue per visitor upgrade or increase of say a six percent well actually that might be is a 1% increase from your new visitors, of which they’re the majority, but a 6% decline by your returning visitors. So if you just looked at it in aggregate you’d not see the new visitors returning difference. And I think the other aspect of this, which is executive buy-in, is that people can still ignore the test results. They can still prefer their own judgment and hunch, and well-meaning opinions, rather than the test results. And I think when somebody talked about exec buy-in that’s the education that needs to go on in businesses just to explain experimentation and how it, you may well find things that as a business you didn’t expect to find. And that’s okay.

“Should you abandon CRO?”

Answer: That’s a good question. No, you shouldn’t abandon CRO. What you should do is broaden it out to stop focusing on just things like page layout and buttons if that’s what you’re still focused on. You should look at challenge yourself to think, “How do I grow this business, “and what hypothesis can I develop, “what data sources would I look to “to collect that data?” And then to design an experiment using the least amount of resources possible rather than saying, “How do I increase my conversion rate?” Because as soon as you answer that question, or try to answer that question, you’ll go down the rabbit hole. The broader question is: how do I grow my business?

Another question which hasn’t come up is: is AI going to be important in experimentation?

Answer: That’s a hot topic. I think in some cases it will. MVT tools often use AI to start picking out the, start killing the least successful variants. I still think that understanding the customers will be, is still a human, a human endeavor, albeit aided by AI backed testing tools for the time being.

Next question from Genoa Astudias Matshidiso, “How are you testing target markets for businesses you assist?”

Answer: When we work with Canon we help improve their 15 online stores selling cameras and printers and so on. And we run, some tests we run across all territories and then analyse the difference. And some territories we group together because we believe that they represent similar levels of culture. And the reason why we’re grouping them together is because together they have enough traffic to run statistically significant tests. So I think that that’s an important way to do it.

Last question, I’m conscious about time so I’m gonna make this the last question.

So question from Victoria, “Got any ideas on how to track specific conversions on third-party sites “that don’t cater for your own specific analytics or UTM code?”

Answer: Victoria that’s a very specific question which, unfortunately, we’re not gonna have time to answer, but, one of our analytics specialists will be able to help you out with that. So if you could email me the question I’ll put you in touch with one of our analytics guys and he can help you a bit further.

So, we’ve run out of time, apologies for over-running. Thank you very much for your time and finding space in your calendar to sit and listen to this. I hope you’ve enjoyed it. I should say that if you or your colleagues, if your colleagues missed it, we are gonna send out a recording of the webinar so you can have access to that, forward it to your colleagues if they missed it. And do keep in touch. So thank you very much. Thanks James. Thanks Caroline. Thanks Eric, thanks Jim.

We partner with the leading CRO technologies