Skip to main content

Our new CRO team member: ChatGPT

That moment arrived. We are now having conversations in real time with computers, in natural language.

“Helpful assistants that talk to you, answer questions, and give advice…” That’s how OpenAI CEO Sam Altman describes ChatGPT in his announcement (see below). Within days, over 1 million people were using it.

Not perfect. But pretty damn good.

From what I’ve seen on Twitter, many people are still having fun throwing simple questions at it. Like this one:

Not perfect. Pretty damn good though.

It waffled its way through a definition of statistical significance, but I don’t think it’s ready to assist Stats101 students with assignments. I duly submitted feedback with references, which is the objective of the current “free research preview”. At some point, there’ll be a charge. (Update: A paid tier was launched in January 2023 at $42 per month).

ChatGPT in Conversion Rate Optimisation

Of course I asked my new AI-assistant, now always open in a Chrome tab, to suggest a title for this post. “ChatGPT: The Future of Conversion Rate Optimisation“. Ambitious, I like it. But no.

It’s only been a week, so we’re seeing mere glimpses of potential. Here are three examples of how we’ve been experimenting with it.


VWO hosted a “friendly competition” between professional human copywriters and their bot counterparts. You already know where this is going… AI-generated copy performed at least as well as humans, and that’s being generous.

Out of 7 concluded A/B-tests:

  • AI won 3
  • Humans won 1
  • 3 were tied

This was a while ago, before the ChatGPT revolution. Not only that, but every attempt is an opportunity for the AI to improve itself. That is what it does, by definition

We’ve used it to help brainstorm copy ideas when creating wireframes. To be clear, we treat it as a member of a team – not a replacement. It’s one voice in ideation.

Does it replace copywriters? I don’t believe it does. If anything, it probably raises the bar and helps to separate the cream from the rest. If AI can do an average job, “average” just isn’t good enough anymore. Put the right tool in the right hands though, magic happens.

Text analysis

If you’ve worked with us, you know that Voice of Customer data is central in our methodology. We want to get into the hearts and minds of your customers. Understand what made them buy, not buy, buy from a competitor or do nothing.

We’re big fans of customer interviews, open-text surveys, helpline emails and customer reviews. That’s a lot of text to analyse and we use different tools for that, including actually reading through much of it!

My usual method is a script I wrote in R, a popular statistical programming language. It provides a good summary, word clouds, top likes and dislikes with context, sentiment analysis etc.

When we tested that script against ChatGPT, it was good enough for the level of effort required. The biggest problem though is “hallucination” – effectively, it makes up stuff and present it to you with confidence. In a series of experiments, it gave a slightly different response each time on the same data set.

Writing and debugging code

Some coders have been using ChatGPT for debugging and to ask questions they’d normally post on dev forums. Well-known CRO Lorenzo Carreri (@lorenzocarreri) made it build him an entire landing page.

Marketers and analysts often use languages like SQL, R and Python. We may use these for example to do advanced analysis, calculate statistical significance of A/B-tests etc.

I tested its ability to write Python code to run a simulation of A/B-test results. With minor edits, it worked perfectly. Five minutes max, if that.

The “threat” of AI

How worried should we be about this? Are computers finally coming after your job?

Perhaps we should take advice from the man behind ChatGPT. Here’s Sam Altman again:

It’s there to help you. Assign it tasks so you have more time to do what humans should be doing. What are those?

Well, who better to ask than ChatGPT itself:

  • Develop new ideas and innovations
  • Communicate effectively
  • Make ethical decisions
  • Strategic thinking
  • Understand cultural differences

I haven’t been this excited and buzzing about new possibilities since that sound of the dial-up modem back in the 90s.

If your CRO programme is not delivering the highest ROI of all of your marketing spend, then we should talk.