Experimentation is the scientific process of trying out new ideas and using measurement to determine whether one approach or version produces more positive outcomes than the other.
It has a long history. Four centuries ago, Francis Bacon published “Novum Organum,” a treatise describing the use of experiments to build knowledge and “put Nature to the question”.
In an online setting digital experimentation is the reliable process of delivering winning digital experiences without guesswork and reducing risk.
Digital experimentation has been used to build some of the most successful businesses in the world; Amazon, Netflix and Booking.com. The practice has also been used by other organisations, perhaps not as well-known for a culture of experimentation, to transform their fortunes. Established businesses like the Wall Street Journal, Microsoft and IBM have had to battle harder than newer start-ups to adopt such a culture.
Behind these transformations are individuals. Individuals committed to driving growth and learning through experimentation; people like Ari Sheinkin of IBM, Ronny Kohavi of Microsoft and Peter Gray of the Wall Street Journal. Their and their team’s successes have inspired others, including the author, to look again at what companies can achieve through adoption of the ‘scientific method’. The work of Harvard Professor, Stefan Thomke, and this post celebrates these individuals and the impact they have made on businesses that have been in existence for an average of 95 years.
IBM’s VP of Marketing Analytics, Ari Sheinkin, admitted that in 2015 the company wasn’t much of an ‘experimentation organisation’. The IT function offered testing services but the whole process was costly, slow and bottlenecked. One person in IT would decide whether or not to run experiments brought to him by colleagues. If he didn’t think it would be a winner, that was that.
In 2015, across all its websites, IBM ran a total 97 experiments1. To put that figure into perspective in the same year Google ran over ten thousand.
An IBMer since 1999, Ari came to the role of VP of Marketing Analytics with a vision.
Ari wanted to put testing tools and resources in the hands of IBM’s 550 marketeers. So he did. He gave marketeers simple tools that were easy to use, introduced a process framework and made tests free for every department. No chargebacks.
Ari led a centralised team of 12 experts. His team were there to help their marketing colleagues design and run experiments. To really get the ball rolling, in 2018 he ran a ‘testing blitz’. The aim of the testing blitz? To run an experiment, one a day for thirty days.
To reinforce and reward behaviour that Ari and IBM wanted to see some policies had to change. Marketing units were told budgets would only be approved when they had an experimentation plan.
Think about this for a second.
What about if you had to have an experimentation plan before you got any budget?
Ari’s vision got support. He rolled out the new testing platform to 23 business units in 170 countries. In 2015 the number of personalized experiments was exactly zero. By 2018 it was 1505, surpassing the 1317 A/B tests the company had launched that year.
Most importantly the number of employees involved in experiments surged from 14 in 2015 to over 2,000 in 20181.
That’s the work of a hero.
The key aspects of Ari’s plan were:
The Wall Street Journal has been keeping its readers informed since July 8 1889, when its first issue was launched.
This conservative-leaning publication has won 37 Pulitzer prizes and now is the most widely read newspaper in the United States.
But they have a battle on their hands.
The New York Times, competing for a similar readership, has more digital subscribers, 3m compared to 1.8m2.
The WSJ has responded to the competition with digital experimentation. In the last three years, it has run over 500 experiments focusing on subscriber acquisition, engagement, product features and retention And the experimentation team exploded from 1 to 173.
As a result of this high-velocity experimentation programme, the WSJ has seen subscribers jump by 51%, app downloads by 400%, and newsletter signups by 300%. It has also used approaches taken from medicine to predict subscribers’ survival rates and model which behaviour has the highest and lowest ‘survival rate’ of their hard-won subscribers4.
The person behind this?
Peter Gray, WSJ’s Head of Product
The key aspects of Peter’s plan were to :
As Peter says on his LinkedIn Profile5,
For Peter, the promise of digital experimentation is that:
1. We can de-risk big bets, unlocking value that would otherwise remain hidden forever.
2. We can know our own products like never before, both by component and as a whole.
3. When you commit to testing, your product roadmap is propelled forward by one thing: accurate, unfiltered customer feedback.’
Well said, Peter.
Like WSJ, Microsoft isn’t the first company you might think of as committed to experimentation. But while at Microsoft, Ronny Kohavi, as Technical Fellow & VP of Analysis & Experimentation, built a hundred-strong experimentation team, with their own experimentation technology.
They ran over 20,000 experiments on Microsoft’s various products including Windows, MSN, Office 365, Skype, Xbox as well as Bing6.
The way he ran his team is democratic. A junior developer identifies a hypothesis he can code in hours, tests it and the gains seen can be profound. Through experimentation and relentless testing, though heavy use of experimentation, Bing’s US share of searches grew from 8% in 2009 to 23% in 20176.
Ronny wanted to put a value on every improvement to user experience. We know that speed improves user experience. But Ronny wanted to prove it.
Through experimentation – and inserting deliberate lags into webpage download speeds, Ronny proved that for every 100-millisecond cut in the time taken to display search results, Bing generates 0.6% more revenue6.
For a $3bn company at that time, this improvement was worth $18m.
As a result of this shift to experimentation and Ronny’s work, 80% of proposed changes at Microsoft are run first as controlled experiments6.
The key elements of Ronny’s plan were to
Ronny’s favourite quote is from the scientist, Richard Feynman.
Ronny continues his great work as Technical Fellow at AirBnB7.
Postscript: Ronny’s talks have inspired many businesses and individuals including developers at Booking.com. His new book Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing is out now.
Ari, Ronny and Peter and their counterparts in organisations many of you won’t have heard of are the real heroes of experimentation. Their work and passion have helped us all to think about what is possible in our businesses.
Think about the similarities in the approach that Ari, Ronny and Peter took in embedding a culture of experimentation:-
So if you’re motivated to embed a culture of experimentation in your business, don’t get overwhelmed by overwhelmed by constantly comparing yourself to Amazon, Netflix and Booking.com
In another post I detail the truth about Amazon and Booking.com’s culture of experimentation – you might be surprised.
Remind yourself instead of the results achieved by these unsung heroes of experimentation and write your own story
As ever, I am thankful for the work of Professor Stefan Thomke at Harvard Business School and the work he does in unearthing the work of these heroes, as well as Optimizely for also featuring organisations and individuals that have discovered the power of digital experimentation.
Posted in: Experimentation
Sign up to our newsletter and get all of the latest news straight to you.
If you’re serious about initiating change within your business, we’d like to offer you a 60-minute Initial Strategic Review.
“We’ll share what we’ve learned from decades of experience working with businesses using CRO, innovation and experimentation to achieve business goals like yours”