Heroes of Digital Experimentation

Experimentation is the scientific process of trying out new ideas and using measurement to determine whether one approach or version produces more positive outcomes than the other.

It has a long history. Four centuries ago, Francis Bacon published “Novum Organum,” a treatise describing the use of experiments to build knowledge and “put Nature to the question”.

In an online setting digital experimentation is the reliable process of delivering winning digital experiences without guesswork and reducing risk.

Digital experimentation has been used to build some of the most successful businesses in the world; Amazon, Netflix and Booking.com. The practice has also been used by other organisations, perhaps not as well-known for a culture of experimentation, to transform their fortunes. Established businesses like the Wall Street Journal, Microsoft and IBM have had to battle harder than newer start-ups to adopt such a culture.

Behind these transformations are individuals. Individuals committed to driving growth and learning through experimentation; people like Ari Sheinkin of IBM, Ronny Kohavi of Microsoft and Peter Gray of the Wall Street Journal. Their and their team’s successes have inspired others, including the author, to look again at what companies can achieve through adoption of the ‘scientific method’. The work of Harvard Professor, Stefan Thomke, and this post celebrates these individuals and the impact they have made on businesses that have been in existence for an average of 95 years.

 

Ari Sheinkin

IBM’s VP of Marketing Analytics, Ari Sheinkin, admitted that in 2015 the company wasn’t much of an ‘experimentation organisation’. The IT function offered testing services but the whole process was costly, slow and bottlenecked. One person in IT would decide whether or not to run experiments brought to him by colleagues. If he didn’t think it would be a winner, that was that.

In 2015, across all its websites, IBM ran a total 97 experiments1. To put that figure into perspective in the same year Google ran over ten thousand.

An IBMer since 1999, Ari came to the role of VP of Marketing Analytics with a vision.

“Making decisions through real-time feedback is my dream of how an organization should be run, and large-scale testing is at its core.”

Ari wanted to put testing tools and resources in the hands of IBM’s 550 marketeers. So he did. He gave marketeers simple tools that were easy to use, introduced a process framework and made tests free for every department. No chargebacks.

Ari led  a centralised team of 12 experts. His team were there to help their marketing colleagues design and run experiments. To really get the ball rolling, in 2018 he ran a ‘testing blitz’. The aim of the testing blitz? To run an experiment, one a day for thirty days.

To reinforce and reward behaviour that Ari and IBM wanted to see some policies had to change. Marketing units were told budgets would only be approved when they had an experimentation plan.

Think about this for a second.

What about if you had to have an experimentation plan before you got any budget?

Ari’s vision got support. He rolled out the new testing platform to 23 business units in 170 countries. In 2015 the number of personalized experiments was exactly zero. By 2018 it was 1505, surpassing the 1317 A/B tests the company had launched that year.

Most importantly the number of employees involved in experiments surged from 14 in 2015 to over 2,000 in 20181.

That’s the work of a hero.

The key aspects of Ari’s plan were:

  • He had a vision of how we wanted decision-making in IBM to be
  • He wrested control away from IT and put the tools in the hands of marketeers
  • He got important changes made to the way that budgets were approved
  • He got exponential increases in test velocity by campaigns like the ‘one-test-a-day’
  • He democratised the process, coaching teams of marketeers and other groups  to see that experimentation was the way to bring his vision to life

 

Peter Gray

The Wall Street Journal has been keeping its readers informed since July 8 1889, when its first issue was launched.

This conservative-leaning publication has won 37 Pulitzer prizes and now is the most widely read newspaper in the United States.

But they have a battle on their hands.

The New York Times, competing for a similar readership, has more digital subscribers, 3m compared to 1.8m2.

The WSJ has responded to the competition with digital experimentation. In the last three years, it has run over 500 experiments focusing on subscriber acquisition, engagement, product features and retention And the experimentation team exploded from 1 to 173.

As a result of this high-velocity experimentation programme, the WSJ has seen subscribers jump by 51%, app downloads by 400%, and newsletter signups by 300%. It has also used approaches taken from medicine to predict subscribers’ survival rates and model which behaviour has the highest and lowest ‘survival rate’ of their hard-won subscribers4.

The person behind this?

Peter Gray, WSJ’s Head of Product

The key aspects of Peter’s plan were to :

  • Recognise the competition (in this case, the New York Times), and define the battlefield – digital experience
  • Test at scale with a team of experts
  • Focus on the whole customer experience – from acquisition all the way to retention
  • Use models from different disciplines  to help predict behaviour
  • Do this with passion

As Peter says on his LinkedIn Profile5,

‘I’m a digital experimentation fanatic and I believe the world has yet to grasp its full promise’.

For Peter, the promise of digital experimentation is that:

1. We can de-risk big bets, unlocking value that would otherwise remain hidden forever.

2. We can know our own products like never before, both by component and as a whole.

3. When you commit to testing, your product roadmap is propelled forward by one thing: accurate, unfiltered customer feedback.’

Well said, Peter.

Ronny Kohavi

Like WSJ, Microsoft isn’t the first company you might think of as committed to experimentation. But while at Microsoft, Ronny Kohavi, as Technical Fellow & VP of Analysis & Experimentation, built a hundred-strong experimentation team, with their own experimentation technology.

They ran over 20,000 experiments on Microsoft’s various products including Windows, MSN, Office 365, Skype, Xbox as well as Bing6.

The way he ran his team is democratic. A junior developer identifies a hypothesis he can code in hours, tests it and the gains seen can be profound. Through experimentation and relentless testing, though heavy use of experimentation, Bing’s US share of searches grew from 8% in 2009 to 23% in 20176.

Ronny wanted to put a value on every improvement to user experience. We know that speed improves user experience. But Ronny wanted to prove it.

Through experimentation – and inserting deliberate lags into webpage download speeds, Ronny proved that for every 100-millisecond cut in the time taken to display search results, Bing generates 0.6% more revenue6.

For a $3bn company at that time, this improvement was worth $18m.

As a result of this shift to experimentation and Ronny’s work, 80% of proposed changes at Microsoft are run first as controlled experiments6.

The key elements of Ronny’s plan were to

  • Build a team and invest in his own experimentation platform
  • Let hypotheses be tested – whether from a junior or senior member of the team
  • Quantify, quantify, quantify – know the dollar-value of every improvement
  • Deploy the approach universally, across Microsoft’s multiple product groups
  • Believe in the ‘scientific method’

Ronny’s favourite quote is from the scientist, Richard Feynman.

‘It doesn’t matter how beautiful your theory is, it doesn’t matter how smart you are. If it doesn’t agree with the experiment, it’s wrong.’

Ronny continues his great work as Technical Fellow at AirBnB7.

Postscript: Ronny’s talks have inspired many businesses and individuals including developers at Booking.com. His new book Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing is out now.

Summary

Ari, Ronny and Peter and their counterparts in organisations many of you won’t have heard of are the real heroes of experimentation. Their work and passion have helped us all to think about what is possible in our businesses.

Think about the similarities in the approach that Ari, Ronny and Peter took in embedding a culture of experimentation:-

  • Have a vision of how you want decision-making to change in your organisation
  • Create a team of experts
  • Remove blockages to allow far more people in the organisation to run controlled experiments – and do it quickly
  • Deploy the test-and-learn on all products and stages of the customer lifecycle
  • Show passion, determination and believe in the scientific method

So if you’re motivated to embed a culture of experimentation in your business, don’t get overwhelmed by overwhelmed by constantly comparing yourself to Amazon, Netflix and Booking.com

In another post I detail the truth about Amazon and Booking.com’s culture of experimentation – you might be surprised.

Remind yourself instead of the results achieved by these unsung heroes of experimentation and write your own story

As ever, I am thankful for the work of Professor Stefan Thomke at Harvard Business School and the work he does in unearthing the work of these heroes, as well as Optimizely for also featuring organisations and individuals that have discovered the power of digital experimentation.

Sources

  1. THOMKE, S.., (2020) Experimentation Works – The Surprising Power of Business Experiments Boston: Harvard Business Review Press
  2. https://phys.org/news/2018-11-newscorp-higher-profits-subscriptions.html
  3. OPTIMIZELY (2019) The Big Book of Experimentation
  4. OPTIMIZELY (2019) Webinar – WSJ and Optimizely
  5. https://www.linkedin.com/in/peter-gray-04a5529/
  6. KOHAVI, R., THOMKE, S., The Surprising  Power of Online Experiments, Harvard Business Review September-October 2017
  7. https://www.linkedin.com/in/ronnyk/

FREE EBOOK

Discover how businesses have made the shift from CRO to experimentation – and you can too

Download your copy today >

Posted in: Experimentation

If you enjoyed reading this, you might also like

Request an Initial Strategic Review

If you’re serious about initiating change within your business, we’d like to offer you a 60-minute Initial Strategic Review.

“We’ll share what we’ve learned from decades of experience working with businesses using CRO, innovation and experimentation to achieve business goals like yours”

Johann Van Tonder, COO, AWA digital.

This is an opportunity for ambitious online businesses who are already generating significant revenue… and know they can bring in more… but are not sure exactly how.

BOOK YOUR INITIAL STRATEGIC REVIEW NOW >>

We partner with the leading CRO technologies