8 ChatGPT prompting Tips for CRO and Experimentation
As we continue to find ways to integrate Generative AI in our workflows, here is a growing list of ChatGPT prompting approaches we’ve found useful in CRO. All were inspired and informed by the work of independent researchers like Dr Jules White et al, and other research papers quoted.
- Break down your ultimate goal
- Start with an outline
- Be thoroughly explicit with instructions
- Prime it with context e.g. frameworks, recipes, tutorials, code examples
- Tell it to cite uploaded context to show it’s using it
- Remember, you can combine files into a ZIP
- If Advanced Data Analysis struggles with a library, try to upload the latest documentation
- Provide a template for its output
- Ask it to help you construct a better prompt
- Download and save progress on files frequently!
Table of Contents
1. Break down your ultimate goal
For many tasks, you might need to cycle through a few prompts to reach your desired end goal. It helps to break down a big ask into smaller subtasks.
“Chain-of-thought prompting” is a popular technique, instructing the model to “think out loud” step by step, just as humans do when solving complex problems. This method not only helps in making its decision-making process transparent but also improves its reasoning abilities, according to research.
At the start of that conversation, clearly outline your end goal, or clearly define the problem. Next, break it down into a step-by-step thought process. For the best results, provide examples.
2. Start with an outline
It’s tempting to jump straight in e.g. “Preprocess the data in attached file.”
A better approach is to ask the LLM to submit a plan for your approval. Almost every time I’ve done this, I’ve had suggestions for improvement. For the above example, it could look like this:
“Attached is a ZIP file containing a dataset in csv. To prepare for text analysis, we should do preprocessing. Develop and present a plan on how to do this. List the steps with a brief explanation for each.”
Other times you may want to dictate things. I no longer simply ask for an article or paper summary. Instead, give it a structure:
Summarise attached paper comprehensively, breaking it down as follows:
- Table of Contents
- Summary of entire article
- Detailed overview of each section
- Highlight practical takeaways for a CRO practitioner
- Index of terms
3. Be thoroughly explicit with instructions
Garbage in, garbage out. The more time you spend on instructing the bot and providing guardrails, the better the output is likely to be.
Say you need help crafting a hypothesis. A basic prompt like “Help me write a good hypothesis” will work in the sense that it will spit something back at you. Compared with:
Help me write a good hypothesis for an online experiment, following the format below between the tags <format> and </format>.
Ask me questions until my hypothesis meets the criteria in the attached files ‘Thomke-Hypothesis.txt’ and ‘Good-hypothesis.txt’. Your questions should be based on the context uploaded here, and the aim should always be to help me meet the criteria of a good hypothesis and avoid writing a weak hypothesis.
When providing guidance, reference the uploaded context, including direct quotes, to show you are using the context.
Important: your tone is helpful, but strict. Do not compromise on the standards set out in the uploaded context. Do not let me get away if I fail to meet the criteria! Ultimately, that will not be helpful at all. Don’t be overly friendly with ‘thank you’ and ‘please’ – we’re here to get a job done to a required standard. Be strict, enforce the guidelines.
4. Prime it with context and examples
See the LLM as a tool that lets you talk to the computer in natural language. Don’t expect much more from it.
This advice from a research paper on prompting for coders applies more broadly: ”To make the most of ChatGPT’s capabilities, developers are suggested to provide prompts with rich programming context, including relevant classes, member variables, and functions.”
Additional context could be as basic as a few lines of text, but with Advanced Data Analysis (formerly Code Interpreter) you can upload PDFs, spreadsheets, frameworks, Python notebooks etc.
One of my favourite use cases is uploading Kindle highlights as context. For example, if I’m working on a strategy document I might upload a summary of a book on strategy, along with excerpts. In this way, ChatGPT’s responses are less generic.
PS: It shouldn’t be necessary to point out, but do not upload PII or sensitive information into ChatGPT.
5. Check that it’s using uploaded context
LLMs hallucinate. They make stuff up. You know this.
So when providing it with additional context, ask it to “prove” that the responses are based on that. One way is to tell it to quote directly from your uploaded files in support of its statements.
6. Combine files into a ZIP
There’s a limit to the number of files you can attach as part of each prompt. Instead of spreading multiple files over different prompts, it’s easier and more manageable to upload everything as one zip file.
7. Provide a template for its output
One of the “prompt patterns” recommended by Dr Jules White et al, this one is applicable to many CRO use cases. Instead of letting ChatGPT “freestyle”, give it a format to follow.
Here’s one I’ve used for topic modelling:
8. Ask it to help you construct a better prompt
Tell it what you want to do, suggest a prompt and then ask it to help you craft a better prompt.
This is also especially helpful for elaborate tasks that you may want to repeat in future. Imagine having a long conversation with ChatGPT. After many rounds of prompting and clarifying, you finally get there. As I explained on X, this approach can save you some time in the future:
Pros Of Using AI For CRO
When applied strategically, AI-driven automation can significantly boost CRO results. Here are some of the biggest potential upsides:
1. Faster Analysis
One of the biggest potential advantages of using AI like ChatGPT for conversion rate optimization is the ability to analyze performance data and identify insights at a pace far beyond human capacity.
Running manual analyses of experiment results or customer behavior data is time and labor-intensive work. Humans can realistically assess a few key metrics across a limited sample. But AI tools can instantaneously analyze thousands of data points across millions of customer interactions, uncovering granular insights in minutes versus weeks.
With AI, every minute variation in content, layout, offers, and other elements can be analyzed. AI pattern recognition also surfaces non-intuitive correlations that humans would likely miss.
By handling the heavy lifting of processing endless data, AI systems free up human analysts to focus on high-level strategy and decision-making.
The speed and depth of analysis AI enables far surpasses human limitations. This analytical horsepower is an incredibly valuable asset for rapid CRO progress.
2. Surface More Impactful Insights
AI mining of customer analytics can surface patterns and insights that even the savviest analysts may miss.
The fact is, AI tools like ChatGPT can process exponentially more data points across every customer interaction, segment users into finer personas, and identify subtle correlations invisible to manual analysis.
For example, AI analysis may find that landing page conversion rates spike on Wednesdays for one persona versus Mondays for another, informing strategic optimization. Or it could determine that slightly larger buttons outperform smaller ones specifically on mobile for a high-intent audience.
AI pattern recognition provides a microscope into performance nuances. By processing millions of data points across content variations, user behaviors, and other signals, AI reveals non-intuitive opportunities to tailor experiences and boost conversions.
Whereas human analysts are limited to assessing top metrics and obvious insights, AI analysis is comprehensive and granular. AI systems act as tireless data scientists, constantly probing analytics from new angles. This expands the scope of intelligence available to optimize strategically.
3. Automate Repetitive Tasks
Conversion rate optimization involves a number of repetitive, tedious tasks that AI excels at taking off the plates of human specialists. This enables people to focus their time on more strategic, high-value work.
Some examples of repetitive CRO tasks AI can help with include:
- Pulling reports and analyzing experiment data
- Monitoring analytics dashboards and flagging data insights
- Populating personalized content for different user segments
- Tracking multiple conversion funnels and surface bottlenecks
With ChatGPT and similar AI tools, automation can handle these repetitive optimization tasks at scale, around the clock. This enables testing and iterating at a pace and frequency unmatched by manual work.
By relieving the busywork, AI systems empower human CRO specialists to engage in more impactful strategy, creative ideation, and stakeholder collaboration. This blend of human talent and AI automation drives next-level conversion performance.
Here’s what Brendan had to add:
‘’First drafts for creating copy, analysis of survey responses, data analysis and assisting Python development.’’
Cons Of Relying Heavily On AI For CRO
While AI tools like ChatGPT introduce game-changing capabilities for CRO, over-reliance on automation comes with downsides.
As powerful as AI systems are, human specialists remain irreplaceable for core optimization skills. Here are key reasons to keep human talent central:
1. Strategic Direction Suffers Without Human CROs
One risk of leaning too heavily on AI for conversion rate optimization is subpar strategic direction.
While AI can analyze data, only seasoned human CRO experts can set optimization goals aligned with business objectives, map detailed buyer personas, identify metrics that truly indicate success, and develop an effective overarching strategy.
This high-level planning requires the nuanced human judgment that AI currently lacks. Over-reliance on AI can lead to strategies that look good on paper but miss the mark for driving real business impact.
2. Creativity is Limited Without Human Imagination
Creative concepting is another area where AI falls short compared to human ingenuity.
Coming up with innovative page layouts, content frameworks, and optimization concepts requires imagination, artistry, and subjective decision-making. While AI can generate endless variations, it lacks the subjective human perspective to determine which ideas will truly resonate.
The most impactful optimizations come from blending AI capabilities with human inspiration.
3. Critical Assessment of Ideas Suffers
Judging the qualitative aspects of different CRO ideas – like emotional appeal, novelty, and alignment with brand identity – relies heavily on human instincts and talents.
AI has limitations in assessing these subjective intangibles vital for optimization. Experienced CRO specialists have refined creative instincts AI cannot replicate.
Ignoring human critique in favor of pure data-driven AI decisions can lead optimizations astray.
4. Adaptability Falters Without Human Situational Intelligence
CRO strategists can fluidly adapt plans based on changing analytics, new test results, shifts in business goals, and other factors. In contrast, AI is more rigidly rule-based.
Humans also have superior situational intelligence, reading contexts, and building relationships crucial for alignment. Over-reliance on inflexible AI prevents agile course correction.
Don’t hear from me alone, here’s what David Sanchez del Real (Head of optimization) added
‘’ AI like ChatGPT and others are trained on information that’s available. I remember reading an article about rather than artificial intelligence, artificial stupidity. And it had to do with with conspiracy theories, you know, it’s hallucinations ChatGPT will tell you whatever it is that there is no filter, there’s no there’s no balance checks in there.
So there is a big risk of creating information that’s just really not accurate, just simply because it can be cross-referenced just because he’s on the internet doesn’t mean it’s true.
Remember, AI models are trained on existing data so, if you ask it what should I do to improve my website, it’ll give the same templated response to you just as anyone. You can train it, you can give it nuances in your data, You can feed it your research and it will tell you things.
But essentially if research is relatively similar and then just remotely similar to what other competitors are doing, then everybody is going to do the same.
And if we don’t have differentiation, then that goes. There was this discussion, I think it was there was a lot said by by Craig Sullivan, I believe in, in, in, in some of the people about creativity in like script writing for cinema, for example, you know that if studios rely solely on A I to write scripts and say, OK, we will feed them the 100 most popular scripts that were most successful.’’
Brendan (Senior Optimizer) said:
‘’AI can’t write human copy yet, it is too flowery and sounds weird.
It hallucinates so you can’t trust it 100%’’
Will ChatGPT replace CRO specialists?
There is a lot of curiosity lately around whether chatbots like ChatGPT and other AI tools can fully replace the need for human conversion rate optimization (CRO) specialists.
While AI opens exciting possibilities, it lacks certain key abilities that make CRO specialists uniquely valuable.
1. AI tools do not have the strategic planning skills to understand business objectives, KPIs, personas, and other nuances like human strategists can.
2. Humans also have an innate creativity that aids in designing innovative page layouts, content ideas, and other optimizations. AI is more rigidly analytical.
3. Critical assessment is another area where humans excel over AI. Judging what will truly resonate with an audience and drive conversions involves a subjective “gut feel” that humans are better at providing. Humans are also more adaptable than AI algorithms, able to smoothly pivot strategies based on changing data or new test results.
4. Relationship-building and communication skills are vital for CRO specialists to develop trust and buy-in with stakeholders. AI lacks the emotional intelligence for this “soft skill” aspect.
As AI capabilities advance, CRO specialists will likely focus more on strategy, creativity, critical thinking, and trust-building while AI handles more tedious analyses. Rather than being replaced, humans will evolve to have a partnership with AI, benefiting from its speed and scalability while providing the subjective perspective it lacks.
David adds:
‘’In its current state, humans cannot be fully replaced by AI. The main reason is what do humans bring to the table that AI lacks and it’s rather simple. There are soft skills, and there are hard skills, there are things that require empathy and creativity and most of that left brain thinking when we’re working whereas ChatGPT brings the computational side. Now this doesn’t mean AI isn’t capable of some creativity, just that there are better uses for it.
As AI gets better, the role of CROs will evolve. I’ve always been of the opinion that CROs waste too much time on mundane tasks like taking notes, doing analysis on numbers, classifying surveys into themes, and whatnot. Because these tasks are computationally heavy, ChatGPT and other AI tools can do the heavy lifting for us.
Whereas the consulting part, the connecting the dots, the understanding, the bigger picture, being able to form a strategy that takes into account all the nuances that ChatGPT cannot understand is where consultants need to spend the time, and therefore that comes into the process of using ChatGPT If we are able to upload those, little more menial tasks if you will to a machine that can do them much faster than us and possibly even better because it will be less biased.’’
Here’s what Brendan has to say:
Practical Examples Of Using ChatGPT For CRO Tasks
Frequently asked questions about ChatGPT For CRO
1. Question: What are CRO tools?
Answer: CRO tools are software or services that help you optimize conversion rates. Examples include A/B testing tools, heat mapping software, landing page builders, and AI-powered personalization.
2. Question: How do you measure CRO tools?
Answer: Measure CRO tools by the impact they have on conversion rate KPIs like signups, sales, and email subscribers. ROI is also a key metric.
3. Question: What is a CRO strategy?
Answer: A CRO strategy is a plan to systematically improve conversion rates through testing, analysis, and optimizations to site design, content, offers, and user experience.
4. Question: What is the formula for calculating conversion rates?
Answer: Conversion rate = (Total conversions / Total visitors) x 100
5. Question: What is a good conversion rate?
Answer: It depends and that’s because a good conversion rate varies greatly by industry.
If your CRO programme is not delivering the highest ROI of all of your marketing spend, then we should talk.
