CRO brainstorming: generating A/B-test ideas
Ideas are the lifeblood of optimisation, innovation and growth. Without an idea, you have nothing to test or implement.
How do you come up with new ideas in your CRO programme?
When I’ve asked people this question, they tend to answer another question instead: how do you come up with insights? This is a different question. How do you move from insights (observations and inferences from data) to ideas about exactly what to test?
It turns out most organisations and teams don’t have an ideation process. Yet they have processes for research, analysis, writing hypotheses, running and stopping experiments, reporting results and just about everything else.
Some organisations leave it up to individual team members. Some run formal, structured ideation sessions. Others kick around ideas between people informally.
Guiding principles, distilled from research
I’ve been studying the science of ideation for four years. My initial objective was to find a better way of generating A/B-testing ideas for our clients. Our primary core value at AWA has always been “Evidence over opinion” so it was important to ground any methodology in scientific research, but test it in practice.
The more I discovered how much scientific knowledge there was on this topic, the more it became an obsession. Today we have a few different science-backed approaches, and continue to experiment and evolve in this area.
We were steered by a number of guiding principles distilled from the research. Here are some of them:
- Quantity breeds quality. You’ve heard this before. Start with a broad range of ideas before narrowing down. In most cases, we aim to start with around 30 and walk away with 3.
- Groups of people > individuals doing solitary ideation. You and I, when thinking of the same problem, will come up with different ideas. Even then, we probably won’t come close to tapping into the entire universe of ideas around the particular topic. Also, being exposed to other’s ideas spark new ones in yourself.
- Groups of people, using a formal ideation method, outperform unstructured group work (Gibson, 2015).
- Nominal groups – individuals coming up with ideas on their own, then aggregating them as a group – outperform teams generating ideas together as a group (Diehl & Stroebe, 1987). The father of brainstorming, Alex Osborn, made the point that “group brainstorming is recommended as a supplement to individual ideation” (Osborn, 1953).
- Team diversity is important, there is strong evidence in the literature that this improves quality. In practice, this means people from different functional areas (eg marketing, dev, UX etc).
- There are well-documented problems with brainstorming. I’m not using the term loosely, referring here to the official method. In designing a CRO ideation system, we want to be aware of those and overcome them.
- We face various cognitive biases that ultimately affect the quality of ideas. An interesting one is the Einstellung effect, our tendency to stick with familiar patterns and solutions instead of searching for better alternatives. Some fascinating studies in this area have been conducted with chess players.
- It’s better to sketch ideas than writing them down. For one, it’s easier to understand an abstract concept that exists in someone else’s mind when you have a visual representation of it. Sketching has been shown to reduce cognitive load when the brain is processing complex information. We’re talking about rough sketches, not pretty drawings.
A science-backed CRO ideation process
Here is one of our favourite ideation frameworks:
Gather 5-6 people, definitely no more than 7, including the facilitator. When we do this with clients, we aim for 2-3 from our agency and the balance from the client organisation.
Always try to have at least one person who is on the front lines, interacting with customers every day. If you’re doing it internally in your organisation, get at least one agency voice or external consultant to join the session.
- Define the problem or opportunity clearly. Don’t skip this step! Make sure everyone is clear on exactly what they are solving. What is going wrong in the journey? What is the customer or persona trying to do? What data do you have on it? What is the impact on the business?
- Round 1. Each participant generates 5 ideas in isolation, which they have to sketch. And they only get 7 minutes to do this. Time-boxing is important, as it helps to remove some of the filters that prevent us from “thinking outside the box”. I ask participants to include one wild or crazy idea. I found that those crazy ideas never survive, but trigger discussions that often lead to a breakthrough.
- Put all the sketches on a wall or a virtual whiteboard. Each individual gets up to 1 minute per sketch to explain the concept, and how it solves the problem identified in step 1.
- Structured group discussion. This is super important, but it has to be managed well. Give each person an opportunity to ask questions and comment on the ideas. Create space for positive and negative views. Assign someone to take notes. You’ll find that this discussion spawns lots of new ideas, and many a-ha moments. That’s the real value of it, and you ride on that momentum in the next step.
- Round 2. Each participant goes back to working on their own for another 5 minutes to come up with 2-3 new ideas. However, this time I ask them to steal another person’s idea and build on it and/or combine 2 different ideas into one and/or come up with someone completely different to what they’ve seen.
- Repeat steps 3 and 4, this time it will be faster because there are fewer concepts to review.
- Vote on the top 3 ideas as a group, using dot-voting.
- Optional, highly recommended. As a group, work together on a final sketch, considering everything you’ve seen and discussed.
The above framework is a combo of three methodologies in scientific literature:
- Gallery Method
- 635 Method
I recently discussed this with Lorenzo and Rishi on their YouTube channel, Understanding How Shoppers Think.
More on Step 1, problem framing, in this podcast with Convert.
Follow me on Twitter for more
Is your CRO programme delivering the impact you hoped for?
Benchmark your CRO now for immediate, free report packed with ACTIONABLE insights you and your team can implement today to increase conversion.
Takes only two minutes
Don’t miss the next event
Experimentation Works hosts some of the most interesting and knowledgeable speakers and practitioners on experimentation and advanced CRO, leave your email address to get advance notification of the next event
If your CRO programme is not delivering the highest ROI of all of your marketing spend, then we should talk.