CRO brainstorming: generating A/B-test ideas

Written by:

Johann Van Tonder

Last updated: 15 September, 2023

Ideas are the lifeblood of optimisation, innovation and growth. Without an idea, you have nothing to test or implement.

How do you come up with new ideas in your CRO programme?

When I’ve asked people this question, they tend to answer another question: how do you come up with insights? Instead, my question here is: How do you move from insights (observations and inferences from data) to ideas about exactly what to test?

It turns out most organisations and teams don’t have an ideation process. They have processes for research, analysis, writing hypotheses, running and stopping experiments, reporting results and just about everything else – but not generating ideas.

Some organisations leave it up to individual team members. Some run formal, structured ideation sessions. Others kick around ideas between people informally.

Guiding principles, distilled from research

I’ve been studying the science of ideation for four years. Our primary core value in the agency has always been “Evidence Over Opinion” so it was important to ground any methodology in scientific research, but test it in practice.

Today we have a few different science-backed approaches, and continue to experiment and evolve in this area.

We were steered by a number of guiding principles distilled from the research. Some of them:

Quantity breeds quality. You’ve probably heard this before expressed as “divergence”, “convergence” and the “double triangle”. Start with a broad range of ideas before narrowing down. We usually aim to start with around 30 and walk away with 3 at the end.
Groups of people > individuals doing solitary ideation. You and I, when thinking of the same problem, will come up with different ideas. Between us, we won’t come anywhere close to tapping into the entire universe of ideas around our topic. The more voices added into the mix, the more varied the ideas are likely to be. Also, being exposed to other’s ideas spark new ones in yourself.
Groups of people, using a formal ideation method, outperform unstructured group work (Gibson, 2015).
Nominal groups – individuals coming up with ideas on their own, then aggregating them as a group – outperform teams generating ideas together as a group (Diehl & Stroebe, 1987). The father of brainstorming himself, Alex Osborn, made the point that “group brainstorming is recommended as a supplement to individual ideation” (Osborn, 1953).
Team diversity leads to better ideas. There is strong evidence in the literature that this improves quality. In practice, this means people from different functional areas (eg marketing, dev, UX etc).
There are well-documented problems with brainstorming. (Most people use the term loosely and don’t actually follow the official process, but that’s another story). In designing a CRO ideation system, we wanted to be aware of those and overcome them.
We face various cognitive biases that ultimately affect the quality of our ideas. An interesting one is the Einstellung effect, our tendency to stick with familiar patterns and solutions instead of searching for better alternatives. Some fascinating studies in this area have been conducted with chess players.
It’s better to sketch ideas than write them down. For one, it’s easier to understand an abstract concept in someone else’s mind when you have a visual representation of it. Sketching has been shown to reduce cognitive load when the brain is processing complex information. We’re talking about rough sketches, not pretty drawings.

A science-backed CRO ideation process

Here is one of our favourite ideation frameworks:

Gather 5-6 people, definitely no more than 7, including the facilitator. When we do this with clients, we aim for 2-3 from our agency and the balance from the client organisation.

Always try to have at least one person who is on the front lines, interacting with customers every day. If you’re doing it internally in your organisation, get at least one agency voice or external consultant to join the session.

Define the problem or opportunity clearly. Don’t skip this step! Make sure everyone is clear on exactly what they are solving. What is going wrong in the journey? What is the customer or persona trying to do? What data do you have on it? What is the impact on the business?
Round 1. Each participant generates 5 ideas in isolation, which they have to sketch. And they only get 7 minutes to do this. Time-boxing is essential, as it helps to remove some of the filters that prevent us from “thinking outside the box”. I ask participants to include one wild or crazy idea. I found that those crazy ideas never survive, but trigger discussions that often lead to a breakthrough.
Put all the sketches on a wall or a virtual whiteboard. Each individual gets up to 1 minute per sketch to explain the concept, and how it solves the problem identified in step 1.
Structured group discussion. This is super important, but it has to be managed well. Give each person an opportunity to ask questions and comment on their ideas. Create space for positive and negative views. Assign someone to take notes. You’ll find that this discussion spawns many new ideas, and many a-ha moments. That’s the real value of it, and you ride on that momentum in the next step.
Round 2. Each participant goes back to working on their own for another 5 minutes to come up with 2-3 new ideas. However, this time I ask them to steal another person’s idea and build on it and/or combine 2 different ideas into one and/or come up with someone completely different to what they’ve seen.
Repeat steps 3 and 4, this time it will be faster because there are fewer concepts to review.
Vote on the top 3 ideas as a group, using dot-voting.
Optional, highly recommended. As a group, work together on a final sketch, considering everything you’ve seen and discussed.