Vlad and Gus, forensic web analysts from the Web Detective Agency, are sitting in a canal-side café. Vlad is downcast after a tough meeting with a client.
Vlad: That was not a good meeting. We’re not going to get any more work from them, I am certain. This is the last thing we need. And they are going to tell everyone we are a complete waste of money.
Gus: Wait a minute before you jump into the canal, at least tell me why you think it went so badly. From where I sat, it went OK – not brilliant, but not terminal. How do you know this client?
Vlad: Dimitri put them onto us. I phoned Anna, the Marketing Director and we talked about what they were doing with analytics – not much at the time – and how they wanted to use analytics strategically throughout the business.
I suggested a workshop to agree their business goals and the most relevant KPI’s, and we had this workshop about three months ago. We also agreed that we would coach their in-house analyst to make sure that they were implementing a ‘world class approach to analytics’, as Anna calls it.
Gus: So why do you think the meeting went so badly?
Vlad: Because after seeing some immediate improvements in their conversion rate after the workshop, it has started to plateau, and we are getting the blame for it. And probably, Anna’s right, they are not ‘world class’ and we’ve let them down, and you know Anna’s well-connected and she’ll be tweeting about us right now – saying what a poor job we’ve done.
Gus: So, you didn’t say the workshop was actually a 'disaster'. Assuming that went well, something has clearly happened between the workshop and now, right ? Who came to the workshop?
Vlad: Lots of people were down to come but there were quite a few no-shows. I don’t think Anna had spelt out why they should come along and what role they were expected to play. Plus, by the end of the workshop there were just so many website KPIs that they were going to measure and optimize.
Gus: OK, I can see not having some key people at the workshop not helping matters right now, but why did you let them apply KPIs to absolutely everything ?
Vlad: Anna was insistent. She said that they had just invested in a new analytics tool that could produce these wonderful reports at the push of the button. She didn’t want to tell her boss, who had agreed to spend over 30,000 Euros on this system, that her team was only looking at less than 10% of these fancy reports. I kept saying that she was going to overwhelm her colleagues but she wasn’t listening.
Gus: The web analyst guy, Alex, it seemed Anna was pretty tough on him as well.
Vlad: I think he’s been put in a really difficult situation. He sits in the IT department, even though he is supposed to be working for Anna and her marketing team. He’s been told that he’s responsible for raising the conversion rate and yet because he’s got an IT background and sits with all the other IT people, he gets dragged into other projects.
Gus: It seemed to be that there was something else going on. When Alex presented the results on the last split test they’ve did on the product page I thought Anna’s head was going to explode
Vlad: Just like in ‘Scanners’.
Gus: What, what are you talking .. anyway. No, I meant that Anna’s was clearly not expecting a test to produce anything less than a fantastic result. When Alex showed her the figures from testing the original version of the product page against one we had recommended there was just silence.
Didn’t you explain that part of the testing process is that you sometimes find out what doesn’t work, as well as what does?
Vlad: I thought I did, during the workshop. But the way she spat out the word ‘failure’ I don’t think she has really taken it in.
Gus: But I thought there were some positives – Alex showed that some KPI’s had improved.
Vlad: That’s right, but because we had run out of the time during the workshop, partly having so many KPI’s to agree, we never got round to setting achievable targets for each of the measures. So, even though we have got some improvement, no-one is going to get a pat on the back to say ‘you’ve beaten your target’ because none were set.
Gus: Just thinking about the web analyst Alex, he spent most of the meeting going through all the reports that he was generating. He’d clearly no time to spend on investigating or playing the detective.
Vlad: Well, we’re supposed to be helping him with that area. But when I went to see him 6 weeks ago to help him delve a bit deeper into the data, all he was doing was getting the reports set up and scheduled to be delivered to just about everyone in the building.
Gus: And did you notice the reports?
Vlad: No, what do you mean?
Gus: What did you think about how the reports looked ?
Vlad: They looked alright, apart from there being far too many.
Gus: Vlad, they looked awful, all numbers in densely packed tables – not a graph to be seen. No series of traffic lights – you know, red, green and amber.
Vlad: He was probably too busy to worry about what they looked like. But I suppose you’re right – no-one is going to take much interest if they look like my old stats exam papers.
Gus: And what happened to bringing in results from the visitor surveys?
Vlad: Mmmm. I spoke to Alex about this. A fair number of survey responses were pretty negative about the product imagery on the site, and he was so worried that Anna was going to have a go at the art director, Johann, that he said he had 'forgotten' to download the data from SurveyMonkey.
Gus: But that type of insight is exactly what they need to understand what to improve.
Vlad: I know, I know. There’s no way we are going to get out of this with our reputations intact. The next email I get from Anna will be her terminating our retainer.
Gus: Hold on Vlad. Look I’m sure we could have done a better job in the workshop of trying to get Anna to understand that analytics is a process that sometimes gives results which aren’t what we would like, but I am sure the situation can be salvaged.
Let’s arrange another meeting to go through the issues we’ve just discussed. We’ll explain that the results from her analytics project could be *even* better if she understood that analytics is more than just a technical process or solely about conversion rates. She needs to stop ‘blaming’ members of her team and start seeing this as a journey – with its ups and downs.
Don’t forget, with our help, Anna did get a big boost in conversion in the early days – her boss will still be thinking that she's doing a great job – but we know that if we don’t get this process right now then six months down the line we will have a much harder job getting it back on track.
Vlad: Plus, we need to move Alex physically to sit with marketing team, and prioritize his reports with the marketing team, at least to give him some time to be explore what the data is telling him.
Gus: And don’t be so hard on yourself. Just because we’ve run a workshop with Anna and her team doesn’t mean that we can all just relax and assume it’s going to work. This is about making a process successful, not just analytics.
Vlad: You’re right. I just expected people to do what I recommended, and then follow the tips we gave them. I didn’t expect that we might have to remind them of what we’d told them.
Gus: I think if we forget to consider the people involved, then of course the process will suffer.
Vlad: You’re right. I’ll give Anna a call on Monday to set up a meeting.
Other posts you might like: