Interview with Head of Experimentation: Luís Trindade at Farfetch
We’re joined by Luís Trindade, Principal Product Manager for Experimentation at Farfetch.
Watch the video to view the full 38-minute interview, or read the transcript below.
Luís Trindade bio:
Luís is currently the Principal Product Manager for the Experimentation area at Farfetch, promoting a Test & Learn Center of Excellence across the company. And before was leading the Data Products area (recommendations, conversational commerce and a big sponsor of the computer vision initiatives). Luís started a few years back as a software engineer but quickly had the necessity to understand how users interacted with products and that led to being responsible for all the interaction and user experience. Then in a shift to the startup ecosystem accepted the challenge as Technology Lead at a company builder, developing new global distributable products and services in the digital area, helping the portfolio to get the necessary boost on tech, user experience and product expertise.
Dan Croxen-John (CEO of AWA digital): Welcome everybody to another one of our head of experimentation Interviews. Today I’m sat with, well not quite sat with, but talking to the Luís Trindade. Luís is the principal product manager at Farfetch, a principal product manager for experimentation for Farfetch. Luis, first of all, thanks very much for giving up your time to talk to us. I’ve got a few questions for you.
In your experience, what makes a really good head of experimentation? What are the qualities and attributes you think are essential?
Yeah, it’s a challenge. I cannot say that is easy work. I’m also not a head in the sense of the typical person that leads a group of people. I’m actually an individual contributor. That’s actually something that I’ve been seeing a lot on many other colleagues that have a similar role to mine, which is persons that are really committed to product development best practises. People that really believe that there are better ways on how to develop products and because that’s for me, the main quality. It’s really understanding how to do product development. That for me, I always say is the big priority.
But for me, product development equals experimentation. Which means if you are not doing experimentation, means that you are not doing product development. You are just delivering some solutions or doing project management. Sometimes that can be so out at the heart of what you do exactly and really understanding that doing further development is much more than simply delivering solutions on time. It’s about creating an actual experience for the users, really about learning how they directed your systems, really adapt your systems towards their need towards their success on and then of course, being a curious person. Being a person that reads some of the few books that actually exists in these areas.
It’s super interesting that a couple of years ago, when I started in this area, there was none. There were only a couple of articles or tweets or conversations that were happening offline between people that are interested in this area, but none of this knowledge was actually captured in a structured way. None of these opportunities of talking and sharing or some of the learning existed at all. Of course it exists a bit nowadays, but they got something that is mainstream, people can try to look at experimentations from another perspective.
But I think that it requires a person that really likes to understand and to go to undiscovered areas to explore new things, to actually apply the whole principle of experimentation to their own role in their way of doing things.
And how do you explain your job, your role to someone who is unfamiliar with the term experimentation? How would you explain in simple language that they get?
Yeah, no, definitely it’s a challenge especially when I need to explain to you what I’m actually doing, not easy, even product management. It’s tough – a challenge. How do I say this? My role here is actually to guide the company where I am, Farfetch, on actually elevating their way of delivering their solutions, evolving their products towards the actual needs of the users. It’s not about doing the quick wins.
It’s not about that, because there is sometimes that misconception that adopting a solution to your needs could create a trap. It’s all about creating that and when people ask me a bit more – but how do you do that? And what I do is, at Farfetch, I’m not even part of any structure. That’s why being an individual contributor is definitely a really interesting point here, being somebody that is like an internal consultant, let’s commit and make sure that we are able to enable the other teams to actually do experimentation.
So sometimes I use the word of being an evangelist, I’m almost like an evangelist for best practises for product development and, of course, somebody that helps and builds the tools to support all the other teams across the company to do better experimentation.
What is it that draws you Luís and your organisation Farfetch to experimentation?
I always say that because it’s a fact it’s perfect. I joined Farfetch four years ago and we had rapid growth in the past four years, but since forever we have been doing experimentations that came from the top level, as something that was essential on how to evolve our product. First of all, that’s what our top level and senior leadership really understanding the value of the doing experimentation and that that’s the first thing.
What has been driving at is that I can be a bit, I don’t know it feels a bit naïve, but for me, there is no other way in doing product development. Like I was saying on the other question was, if you are not actually not doing experimentation, you’re not doing product development, which means that that we wouldn’t be able to create the products that we are delivering, adapting them further the solutions towards the customer success as we have been doing and successfully doing and evolving on that. For me, that’s the only way to do a natural product development.
You’ve been very clear about the relationship between product development and experimentation. For some companies who are perhaps at an earlier stage of that realisation, what reasons would you give them to start the experimentation journey?
That’s definitely a great question. Lots of people ask me all the time. ‘Oh but I’m not doing it experimentation. What can I do?’ ‘But I don’t have the money to invest in doing experimentation’ and it’s completely wrong. It’s not the way that you should look at experimentation. Experimentation is, I always say that experimentation is a loss prevention tool. Because it’s actually a way for you to avoid making huge mistakes that you could get from simply rolling out and not on then starting to bleed your company’s business. Not knowing exactly where is the cause of the that not being able to learn from what’s happening? You would change from a proactive and the way to an actual reactive approach. Okay, I’m having problems now, but what can I do? It’s about changing that process. It’s about creating more control on what you are doing it. It’s about having a brief management because you will be able to control.
How do you expose your experiments ones that are more critical? Reduce the risk for that. The ones that you have more confidence with, you can be bolder and expose them more. It’s about getting statistical significance, than having a really true informed philosophy and mentality associated with how you are doing. So at the end, experimentation is not just having a bunch of guys that are actually there to try to do some quick wins and optimise and get some wins. It’s actually about changing and using those.
I always say it’s one of the missions of our area. These shared learnings to become the business compass, and that’s still true. It’s about using those learnings, really to drive even the strategy of the company. That’s why it’s super powerful. It’s not just the typical drop in get the result and move away and really about transformational. It’s about having a true experimentation culture.
Can you share with us an example where you’ve gained an insight or something, an unexpected outcome that just really blew your mind or really turned your understanding of something on its head?
I need to pick one. Actually, because we are always getting those and that has been, like I mentioned it, in that perspective, it was actually one of the main changes that that has happened since we started these new areas of the company two years ago. When I started the new area here, which was changing the mindset from the actual outputs and then the business inputs to actually focusing on the learning.
That’s why we have monthly learning sessions where we share those lessons with everybody at the company and focus them actually on the fight or flight for each of their tests, like on the bottom corner. And that’s why I think that every single month we are always having really great learnings about our users. I could highlight some.
They have not been completely out there. But it was like, for example, really confirming the different behaviours from some of our loyalty programme piers. Understanding that some of our users, something that we guessed, but only after starting tests. We actually start getting some really great insight, understanding that the user behaviours are different between how they engaged with our products or understanding how we have been building some solutions.
Another example is one of the longest tests, and by test I mean, experimentation around an area or was our dedication to change our recommendation engine? We decided a couple of years ago to actually start building our own recommendation engine. At the beginning, it was difficult, but we started interacting, trying to adapt, creating better algorithms for each of the touch points. Each of those steps have then proven better and better and all those learnings were included on the different situations that nowadays we have completely replaced another solution that we had the world class recommendation engine and now we have our own, supporting the entire platform and have all the flexibility and all the other gains actually that we get from that by creating a personalised experience.
So you thought through experimentation, you replaced your product recommendation engine with something that you proved that you could make it work as well if not better than the commercial option?
Interesting. What mistakes do you see people or organisations making in experimentation?
I think that the main mistakes I have already mentioned it before. It’s looking at experimentation as a tool. Something that is there that just can be simply adopted, and you will deliver it and you’ll have and say ‘ok I’ll sign two or three guys that are going to work on experimentation and that’s it’. Looking at experimentation as a tool as a way to actually drive fact results. That’s the biggest mistake because it’s very, very common misunderstanding, because that’s why I always try to change.
Many people talk about transformation out of zero conversion rate optimisation. For me, that’s wrong. I know that is the easiest way sometimes to get to somebody. But that is the wrong assumption, it is the wrong perspective, because experimentation is about change of culture, change of mindset. That’s how we should look at it.
And you were talking about your contribution, you are an individual rather than the head of a large team. What kind of skills can you pull upon within Farfetch to help you? What key skills have you got access to?
Let me present a bit about our structure of experimentation in our company. Of course I’m not alone. It would be impossible to do what we have achieved by being just one. But as I said I’m an individual contributor leading that area in terms of the evolution. But having support of my entire team. My team is basically a small team actually, we have 3 teams, software engineering teams, two focussing on actually building our own platform, we have been building our own experimentation platform. We also have a data science team, that have been building our own statistical engine.
And then we have what I like to call our extended team, which is the persons, that they are not 100% dedicated to the actual platform, but they’re crucial members on making sure that the work that we’re doing actually is scalable. So basically we have a network of what we call the experimentation ambassadors. People from different areas from product, analytic engineering, architecture, user research, marketing different areas of the business, sometimes even beside product, because we are a very big organisation.
We’re almost 5000 people and almost half of them engineering. All spread around different offices. I don’t even know a nowadays around the world, so this also leads to how can we have people that actually go in China, where we have a huge engineering team and product team operating there. So we should have somebody that’s closer there in terms of time zones and in terms of culture. And then creating regular one on one meetings with those people trying to understand what are the main challenges. Bring those challenges together trying to define a strategy on how to, overcome some of the opportunities that exist there.
So that’s how we have been doing these. Having a centralised team that actually doesn’t run the test, but enables all the others to do it and having part of the team dedicated to evolve and create the actual platform and then on the other team, that is more about making sure that all the training materials hold awareness of the challenges that exist whenever somebody there needs some guidance, help support there. And then we also created some tools that actually helped in promoting that again. For example, one of the main points in the success of the experimentation programme is making sure that the hypotheses are well crafted especially around their definition of success.
What was created was, to define all those all those hypotheses and the guideline especially for new joiners. Then we also created some regular ceremonies. We call them experimentation peer reviews, where every two weeks we have, like open office hours where it is not just me, not just even just the ecommerce team, the whole community that is interested in experimentation between product manager, family engineers, etc. They join on those sessions they hear from the others, those early hypothesis, and based on those early hypothesis, they feedback on how to make it first clear, the hypothesis of the definition of success is clear.
And so this way we are basically crowdsourcing the knowledge and making sure and of course in those situations, we always have members off the experimentation in attendance, making sure that whatever solution, whatever feedback given, it’s the most clear one, is the better one, and so it’s always a continued learning opportunity that we are sharing between everyone.
You know the story about booking.com, 80-85% of it’s staff are involved in experimentation of some kind. What’s the figure, do you think for Farfetch?
We are still far from there, actually it is one of our benchmarks in terms of the mindset of experimentation culture, but we are far from there, especially because we are a business that is quite fragmented in terms of business units.
We have our whole consumer product area, which, if we look at our marketplace, that sits on top of the whole platform of services that we are driving. The is the ones that I know for a fact, is not necessarily just the profits website or market. We are actually the luxury platform, and we are building the solution that any others can build their own businesses on top of the solution that they are providing. For example Heritage has their whole digital ecommerce solution built on top of us as does many other brands, and one of the challenges on such as our organisation is that, yes, we have the continued project area that is focused on the marketplace evolution, our apps etc.
That is why we started doing experimentation naturally for a long time. And that’s where most if not almost everything that we release is testable. But then we also have other areas that that we identify. That have a real potential and also applying these principles, like our back office products that work and build solutions and, some told to actually help our partners in really integrating and interacting with the platform that until now has been a bit more neglected.
I call it the poor cousins around experimentation. But there is a huge potential there, of course, and that’s another case of, it’s definitely not the hero strategy. It is not about conversion, it is about creating objectives that are completely different about speed, quality, errors, making sure that a frictionless solution for some of the users. We don’t have also the huge numbers of traffic with those tools that we would have on the public facing website. The challenge is completely different. But it’s a big part of our business. What we are entering, and we define these as one of the objectives for me and then we also have the other areas that are more challenging. In that sense, nothing is easy. My ambition is to really go to those levels of booking.com. But on the consumer facing side we are really well, on the others not that well. But we are working on it.
In numbers terms, how many experiments do you think you’re running per year or per month?
Not an easy number, because again I don’t know this off the top of my head. We run more or less simultaneously around 50. So then when we start looking at the outcomes and all the overlapping, I think that more than 200 to 300 per year.
Okay, that’s a healthy total. In terms of the technology, it’s interesting that you talked about developing your own product recommendation engine, your stats engine and your experimentation platform. Is the technology you are using full stack service side and client side?
Yeah, in that way. So yes, we have been building our own platform. It’s a full stack solution. But at the same time, we also have partnered with Optimizely as a way to actually, especially around the client-side experimentation as the tool, to also allow us to do rapid iteration. In some of the situations, some tests are more about copy. Something that is not necessarily about behaviour. That is much faster to reiterate and avoiding all the deployment process that even if it’s fast, that always creates more friction there to launch it. But at the same time, we also better help Optimizely for they have been a partner to us for two years from now. But we always had a full integration of their solution.
This means that, because we have built our own platform, we even have our own event system. We call it event tracking its one of our key assets because it’s the way that we tracked all the events across the entire platform. Imagine even the event on a one of our stores of the future could use exactly the same event system to correlate all these relationships, even those, for example. We are not Optimizely, we don’t rely on their own events or metrics. We actually use our own. We feed those events there, we use those, we catch back also the results. And we can even analyse the actual results on top of our own engine. So it’s a really interesting way that we build it. Taking the synergy is taking the advantages from the different solutions. But in a very consolidated way.
You mentioned one KPI for the success of the programme, what are some other KPIs that the organisation looks at when judging the success of the programme?
This is also a great question because it’s a very common one. Most of the people you were asking me for example, the number of tests. We don’t actively use that as one of our eight active focus of KPIs. Why? Because for example, for us, it’s much more important. The ratio between the actual number of tests per active teams, making sure that we have the team’s actually continuously doing experiments and making sure that we also increase that percentage of active teams that do experiment. That’s definitely one that we use, but much more than that, as a top metrics for KPI for success of our entire programme as a whole. We have been using the experimentation growth score so in terms of maturity. We have been using that super interesting concept based on a survey.
In terms of trying to identify more easily from the different dimensions there that the actual output, it’s not just the number. It’s actually a number for different dimensions, where we are and identifying the gaps that we have there or the gaps are different, even the different areas, so we could try to tackle that. It’s definitely, if we can call it that, our North Star experiment. Is that one.
And if you’re looking at a company from the outside Luís, how would you know whether or not it had a culture of experimentation?
First of all, it’s easy when you start looking at precedent. Imagine that there’s a website, or company that has a public facing website that’s the most important and focusing on that. If you see a website or an app that doesn’t change for the past six months. That means that you haven’t been targeted by any AB test at all that’s definitely a company that is not trying to adapt their product towards my need and normally, that’s the key indicator if that company’s doing experimentation or not, maybe they’re doing experimentation at another level, not that common. If they don’t do it on consumer facing product, they are not doing also on the other side.
Who do you admire in the world of experimentation and why?
Oh, like I mentioned at the beginning, there’s not that many people that are actually driving all these. Between Ronny Kohavi, Lukas Vermeer from booking.com, Stefan Thomke. So those are the guys that actually, I follow them on every single post or every single book that they have written. One of the few, like I even have their books here. For me it is one of the references from Ronny Kohavi which the physical matter or the latest book from Stefan Thomke Experimentation works. Basically, those are my bibles around experimentation to normally, anything that they do, I follow.
You are right, there is not many of them there, a small number. So if you were coaching a young person on and he or she was keen to get into this area. What advice would you give them?
It’s also funny, because even this morning, I was coaching the junior product developer that just joined Farfetch, and she came directly to me, saying ‘I’m here. I’m completely new, and I would love to hear more about the work that you have been doing’ and I did the one on one session with her.
What I always say and I said to her was, when I started, experimentation, it’s a way of thinking. It’s not a tool that you use. It’s how do you embrace the way that you look at the product that you are nurturing and really look at it and how to break down these whole product development cycles. Normally, I break it in to three main areas, like product discovery, build up stage and then the delivery stage, and not just use experimentation to actually measure the output, the delivery stage, like validating output drives more money or achieves the purpose of what we’re doing but actually use experimentation again, not just using AB testing that’s another misconception of experimentation. You have lots of different tools with an increased degree of certainty and, of course, also less risk. But use that in a way that actually applies scientific matters.
Nowadays, with all this Covid situation, everybody is almost an expert on clinical trial, but look at that as an example because the actual scientific matters applying basically using that as a way of always try to reduce it from the nature of the humans and use that on all the stages. What you’re doing when you are trying to identify an opportunity, when you are trying to identify a problem, after you validate that. What is the fastest way to really learn what could be the best next path in terms of when you are building, because it’s up to you, you have 10,000 ways that you can solve that problem and then always ask what is the fastest way to answer each and every one of those questions that you have in mind and from there create an experimentation strategy.
That’s the most important. When you are actually doing a product development, it’s all about creating a true experimentation strategy, not just the one shot that actually will drive money.
It’s very interesting that you said Farfetch isn’t particularly preoccupied with a number of tests that you’re running per year or per month. But what do you think separates those businesses that believe in experimentation versus those that are running tests? They are actually pushing out 5, 10, 15 a month, you know, but are they necessarily experimenters in the same way that you regard Farfetch?
Yeah, I don’t know if they are better or worse myself. I think it always depends on the context of their work and their maturity. Normally, the companies that are in the early stage of the maturity, they always run tests, sometimes by a dedicated team. They have a small team that’s working and one that has a problem in terms of the capacity. Of course, that small team cannot do many more tests than what they have resources. But that doesn’t mean that they’re doing the wrong test. Maybe they’re doing detail in the right way, but just in terms of the different capacity again. That’s why one of the reasons that the number of tests is not a good indicator of the quality of the actual experiment.
But at the same time it’s always when you start growing and if you are able to really run a lots of tests, I come back to the question or normally a comment that somebody asked me, should I do that everything ultimately? and my quick answer is yes, and of course, but there is a long answer. That means that of course not everything should be tested with the purpose of what you are thinking, which is driving so validating everything towards business impact. But if you use experimentation as part of every single step like I was mentioning before. Yes, that means a point that always having these are the way of doing further development. Doing AB tests on everything – No, doing experimentation on everything – Yes.
What development do you think we might see in the field of experimentation over the next 2 to 3 years?
I think that I’ve been seeing now the shift, like I mentioned between this year and the end of last year, some of the books have started to appear. Some people have started to consolidate some knowledge, creating ways that people are more aware of the potentials of having the true mindset of experimentation. I’m seeing actually a shift in the way that people look at this.
Hopefully 2, 3, 4 years, the main changes here that we would see is having more and more teams, more companies actually embracing this framework, this way of doing things. But I think it will be very good, especially for the users, for the consumer, for the customers, because that means that they will also be adapting their solutions, not with always the purpose of actually driving more from the customers but actually creating products that are more adapted, personalised, tailored towards their needs and towards their success.
As I was saying, I think that would be the ultimate goal. I am not saying that we will have every company doing this. It’s very difficult because we don’t have that many product companies in the world versus of course service companies. Now those will be more challenging to embrace the challenge, but not impossible and that’s actually something that I’ve been talking of, sometimes I’m also a mentor for many start-ups, and that’s one of the best types of advice that I always give them that, even as a start-up when they feel that ‘Oh, but we cannot do experimentation because we don’t have the traffic. We don’t have the users’.
Yes, they can, because they can apply the same principle and then they be doing it super successfully, and sometimes after a couple of months they reach out to me after the incubation programme and saying it is really giving value to what we are doing and actually changed the way that we actually develop our product.
So hopefully with more people looking at experimentation, sharing these interviews and these videos hopefully will also help on that. More people will start using it and so, we will be like a new evolution on that. Hopefully I’m doing my little bit on the part of the company. Also, you are also doing that. So hopefully in four years we are talking about having a huge conference with too many people that actually could join versus just having one or two books around experimentation.
That is a very succinct answer that you see it as an evolution of the number of companies doing it will increase. But also, non-product companies will start to hear the message. Thanks Luís.
Is your CRO programme delivering the impact you hoped for?
Benchmark your CRO now for immediate, free report packed with ACTIONABLE insights you and your team can implement today to increase conversion.
Takes only two minutes
Don’t miss the next event
Experimentation Works hosts some of the most interesting and knowledgeable speakers and practitioners on experimentation and advanced CRO, leave your email address to get advance notification of the next event
If your CRO programme is not delivering the highest ROI of all of your marketing spend, then we should talk.