Key Points
- Cross-functional workflow for managing qualitative feedback in a large-scale product
- Combining quantitative and qualitative insights for research at different stages of the product development cycle
- Examples of quantitative and qualitative research tools used at Pinterest
- The usefulness of getting to a “north star” KPI to help decision-making across the product and user segments
- How to prioritize features coming from all the feedback we receive
- The need to define success metrics both in terms of goals to hit, and KPIs that should not be impacted
Transcript
Daniel:
Can you tell me a little bit about yourself and your background in product management?
Lulu:
Sure. I’m a product manager on the discovery team at Pinterest where I focus primarily on search, and before that I was on the growth team at Dropbox where we worked a lot on new user activation, and before that I did marketing at Microsoft for a little bit and I started my career in media and telecommunications banking, actually, in New York.
Daniel:
Excellent. Okay, so can you tell me a bit more about your role in Pinterest right now? What kind of things are you working on?
Lulu:
Yeah. At Pinterest, I focus on search product, so essentially that’s all of the user facing features when it comes to search across our three platforms. So our mobile apps on iOS and Android, and then on web as well.
Daniel:
Right. That means that you have a lot of many different users, and can you tell me a bit more about how you try to manage all the unsolicited qualitative feedback?
Lulu:
Yeah. There are a couple of channels that we get qualitative feedback through, one is actually just other employees at Pinterest. I know we’re fortunate in that working on a consumer app that has a daily use case, that we get a lot of feedback and insight just from other employees at the company who are using product. So that’s one source, another source is users who write in to give feedback via our support or help channels. We get tickets from our pinner ops team, and then we also get feedback from qualitative research that we run, both through things like in-product surveys where we will intercept a user as they’re doing a search, and through in-person or remote usability studies where we get feedback from pinners on either current experiences in the product or prototypes for potential future experiences that we’re thinking about building.
Daniel:
Great. Let me unpack that a little bit, let’s start with when users are writing in, they’re sending tickets to support. How do you sift through all of that? How does the communication work between your support teams and the product team?
Lulu:
Yeah. Usually when people write in and… We actually just kind of made this change to our ticket submission flow recently, but you can tag your request as being search related and then when it comes in, our Pinner ops team will see that tag and they can filter it. And then also group tickets by the particular issue, is it related to spelling correction or is it related to search guides so they can kind of drill in that way. But yeah, we’re really fortunate in that we have a team of folks who are basically on the front lines of this feedback, and we have weekly syncs where… Cross functional syncs where folks from all these different teams are there, and if they see any sudden spikes in tickets coming in for a particular feature, or any changes in sentiment, they can relay that feedback to the product team and engineers and the designers. So we have that weekly touch point with the pinner ops team.
Daniel:
Right. For that kind of input, are you relying on that frontline user facing team and they will then maybe report back when something seems off? Do you ever actively read through that kind of feedback, even when they’re not telling you anything about something being out of the ordinary?
Lulu:
In general, I would say that this channel, it’s not one of our highest volume channels probably for feedback. Yeah, I’m not… It’s not something that I’m actively checking on a daily basis, it’s more kind of just monitoring long-term trends and then when… On the product side, when we know that we’re about to launch an experiment or ship something to a 100% that we think potentially might cause confusion or users might just have a lot of feedback on it, we will of course proactively bring that up to the pinner ops team so they know to expect it, and also just to keep an eye out specifically when those changes happen.
Daniel:
Right. That’s on the user facing team and you talked about a different type of research that you run with your users. You use surveys, you use remote testing. Can you tell me a bit more about how this works, and what kind of structure you use in those surveys, and what kind of things are you looking for?
Lulu:
Yeah. So stepping back a little bit, I think we use both… In terms of gathering feedback, we look at obviously both behavioral data, so quantitative data that we have, we have a ton of dashboards and key metrics that we’re monitoring all the time, and we pair that kind of quantitative analysis with qualitative research to understand the why because if you just see a number kind of go up or down week to week, that tells you what’s happening, but it doesn’t tell you why users are doing that particular action. So I think of research as both helping to answer the why, and then also qualitative research is really helpful when you’re also just trying to gather more foundational insights around a product, and you’re in the very early stages of kind of thinking through a redesign or a new feature. That is also… That’s another step of the product development cycle, where qualitative insights can be super helpful. So it really just depends on kind of what your goal is. So if it’s trying to understand why people use Pinterest search over Google search for instance, that’s like a foundational-type question and the research methodology that you would use in that case might be different than if you’re just trying to understand, “We shipped this feature and no one seems to be engaging with it, why?”
Lulu:
Those are two very different types of questions. Qualitative insights can help you answer both of them, but they would require different approaches. So I think you have to start always… You have to start with your question and your goal, and then at Pinterest we’re also very lucky to have a really talented insights team and so we work pretty closely with them to frame the research questions we’re trying to answer, and then they are kind of the domain experts there in terms of determining, “What’s the right methodology that we should be using?”
Daniel:
Alright. Can you walk me through an example of that? Is there anything that you could share, maybe, on a specific example of how you went about trying this kind of research?
Lulu:
Sure. So one example that could be helpful is to talk about… So this question that I just mentioned of trying to understand why would someone use Pinterest search versus Google image search, for instance, so that’s a question that, when I first joined the team, was pretty top of mind for me, just to understand what differentiates Pinterest search in our user’s minds. So that was kind of the research question that I went to our insights team with and, working with them, we kind of did actually a kind of a three-step process. So first, we did a series of short, pretty quick remote user interviews with searchers. Essentially we just recruited people using usertesting.com and we asked them to answer a set of five questions or so about their use cases for Pinterest search, why they liked it over other competitors, what they didn’t like about it, really just to draw out kind of the strengths and weaknesses of search on Pinterest versus other products. And then, based off of those qualitative insights, we kind of had a few hypotheses about why people would come to Pinterest search and kind of what our differentiators were, but we wanted to see if we could actually then kind of validate that through quantitative data as well. So we came out of the qualitative interviews thinking there were kind of like two… Not really personas, but more kind of just two scenarios for search on Pinterest.
So one is people who are trying to just find ideas; they come to Pinterest, they’re looking for inspiration, they’re looking to be inspired. They kind of have an idea of what they’re looking for, but it’s not like they have an exact thing. So that the metaphor is like… When you go to Google, you’re trying to find the needle in the haystack, and it’s all about, “How quickly can I get to this piece of information that I’m looking for?” Whereas on Pinterest, the hypothesis is that actually a lot of times people come and it’s not so much about getting from Point A to Point B as quickly as possible, but it’s about that discovery process, and being inspired, and the process of browsing through the hay is actually enjoyable and it’s fun and it’s what brings people back. So we kind of had that hypothesis that that was one use case for Pinterest search, and then we had another hypothesis that there are also situations where people are looking for a very specific thing. They’re trying to find that one recipe, or they’re trying to find that one party idea, and so we designed a survey to kind of basically try and get at the actual prevalence of both of these use cases among Pinterest searchers.
So we created this survey, we set it up as an in-product survey, so we basically were able to intercept people as they were searching on Pinterest and just ask them if they wanted to give us feedback, and coming out of that, we analyzed the results, we looked at how many people said they were looking for a specific answer, looking for that needle versus just browsing, just looking for inspiration, and then the next step, which we are still in the process of doing, is then to actually see if we can tie those self-reported responses back to behavioral data that we see in the product. So based on whether someone says they’re just looking to browse or to be inspired or they’re looking for a specific answer, can we actually look at their engagement in the product and predict which of those two modes that they’re in? So that’ll be the third step.
Daniel:
Right. Awesome. You mentioned that you guys have an insights team at Pinterest. How do you work with user experience and user research in terms of finding out about new things to build or to improve or as you try to build new things themselves? I mean there are multiple stages that you might resort to to this kind of cross-team work… Can you walk me through your process around user experience and research and the insights team?
Lulu:
Yeah. So as I mentioned before, I think it kind of all comes back to your research goal and the question you’re trying to answer. So in my experience, there are multiple… Throughout the product development cycle, the research insights team can bring value at multiple different stages of that cycle. So the very beginning of thinking about a new feature or a redesign, they can be super helpful and doing the foundational Blue Ocean, Blue Sky type research to just understand, “How do people currently solve this problem today? What are the other competing solutions?” Just very foundational research.
Daniel:
Right.
Lulu:
And then there’s kind of another flavor of research where it’s like, “Okay, we have this idea, we have a prototype, we have a proposed solution, let’s get feedback from users on what they think.” So… And that tends to lend itself to a more rapid iterative cycle where you might put a prototype in front of a few people, get some feedback, the designer might iterate on it, and then the next day you do it again with another few people. It’s this rapid iteration cycle. And then there’s a whole… Then there’s another flavor of research where it’s inspired by an interesting trend that you see in your behavioral data, like you’re trying to understand, “Oh, we launched something and no one seems to be engaging with it,” or, “Two months ago, engagement with this feature all of a sudden skyrocketed, what happened?” And there, it’s really more about providing these ad-hoc insights to understand trends that you’re seeing in the behavioral data. So I think… I’m a big proponent of research, and again, there’s multiple points throughout the product development cycle where research is super helpful. I think it’s up to the PM to define what the key research questions are and partner up with the insights team to figure out the best implementation.
Daniel:
Right. You mentioned that the user insights team is in charge of selecting which method to use in order to answer a particular research question. I’m curious what kind of methods besides the ones that you already mentioned are they using or are you using in your work?
Lulu:
Yeah, there’s so many. On the quantitative side, you have in-product surveys, you have surveys to non-users, so like more market research surveys where you’re probably trying to get maybe a pulse on brand perception, how that’s changing over time, how that’s… How it compares to competitors. Then, on a qualitative side, you have everything ranging from usability studies to in-person interviews at user’s homes to brainstorming sessions with… Not necessarily with users, but with other parts… Members of your team. Brainstorming exercises to think about new experiments that you could run or new features that you could build. There’s… User research is a very broad, deep field, but that’s just a sampling of some of the things that we do at Pinterest.
Daniel:
Great. Let me dive into another topic, which is user segmentation. You mentioned that you realize that you had two profiles and you’re trying to figure out if those profiles match what users are telling you. Do you have any other ways to segment your users in a way that makes sense for you to look at the feedback that you’re getting in?
Lulu:
Yeah. So the two kind of profiles that I mentioned before were specific to search, but across Pinterest as a product, we definitely think about our users in different segments. The segmentation is a constantly evolving thing so up until very recently, actually, the way that we were thinking about our user segments were just in terms of state. So, by state, “Are they a new user? Are they a casual user? Are they a core user? Are they dormant? Are they resurrected?” And all of these kind of labels tie to a certain… Basically engagement criteria. Like… So a casual user may be someone who’s engaged with our product twice in the past month, whereas a core user is someone maybe who’s engaged at least once a week. So you have these different user states and that kind of helps you think about the effects… So the user states are helpful in a few ways: One is to give a more nuanced understanding of engagement with your product and how it changes by different user states. So you might launch something, and if you look at the global numbers, everything looks great, but if you actually kind of do these more granular cuts, you might see that a particular feature is really good for core users, but actually is bad for new users because it hurts their comprehension of the product in some way. So being able to kind of get those granular cuts is really important.
So yeah that’s another way that we think about user states and, more recently, actually, we’ve been thinking about another way to define engagement using a metric… Or this idea of a one key activation metric. So you’ve probably heard of, in the early days, for Facebook, it was getting a new user to add seven friends within 10 days, that was kind of like their north star KPI that everyone was striving towards. So at Pinterest, we’ve been actually thinking about, “What is our similar seven friends in 10 days equivalent,” and trying to see if we can simplify our user segments a bit because when you have five or six different segments, the granularity is helpful but it also makes decisions more complicated because you’re trying to look at a bunch of different groups of people. So we’ve been thinking about, “What is our equivalent of the seven friends in 10 days?” Yep.
Daniel:
Cool. That’s really interesting. I was wondering, do you have any longterm relationships built with any group of customers or users? Some way to try to understand the qualitative feedback over a longer period of time?
Lulu:
So at Pinterest, we are very lucky, again, to have very devoted and passionate users. In the early days, actually, Pinterest was an invite only service and that’s kind of how it quickly spread, and so we actually have a lot of power users who… These are taste makers within their domain, so lifestyle bloggers, food bloggers, stylists, personal trainers, just people who are experts in their field who use Pinterest very heavily. And we have a really great community marketing and community management team that works very closely with these power pinners to understand their feedback and their thoughts on the product, so we have a very close, ongoing relationship with these power users. So that’s a really important touch point for us.
Daniel:
Cool. I wanted to ask a little bit more on the subject of how you prioritize what you learn through customer feedback and how does that play into the other things that you might need to be working on at the same time, so can you share maybe a little bit about your process on how you prioritize what you learn versus other things that might be on your roadmap? How does that all interplay and work out together?
Lulu:
Yeah. So I think it’s a balance of having a really clear vision and strategy for the overall product while also being mindful of, again, this ongoing feedback that you hear from users and figuring out at what point do you elevate a piece of feedback from, “Oh, that’s interesting,” to, “I think we should really invest in this.” Yeah, it… There’s definitely an element of… You just need to have a strong point of view on where you think you want the product to be in six months or a year or even five years. And when we do our roadmap prioritization, a lot of it just stems from a conviction that these are Pinterest’s strengths and this is how we’re going to… We believe this is the best way to demonstrate these strengths and to also bring… Show the value of Pinterest to more users. So I would say that, at a high level, you need to have kind of just this really strong sense of what those strategic priorities are, and, in terms of the kind of the tactical day-to-day feedback, there’s… We do a lot of our tracking in JIRA, so I know for search we have it broken down by the different feature areas in search and we have a running backlog of feedback that we’ve gotten from all different channels, and as a PM, it’s…
Yeah, it’s your job to kind of look at each of those pieces of feedback and decide, “Okay, what is the potential impact here,” versus the expected kind of investment and decide whether or not something is urgent enough to kind of be added or to be bumped up in the list or if it’s just not super high priority at this moment.
Daniel:
Right. Another thing that we’ve already touched upon but I wanted to try and understand a little bit better is what does success look like? After you’ve shipped something, what kind of thing tells you that a feature is both providing value to the user and providing value to the company and to the product?
Lulu:
Yeah, so I think actually one of the most important steps in launching any experiment or new feature is thinking through this question of how do you define and measure success, two equally important things to answer. So yeah, sometimes it’s very straight forward, like sometimes we have a bunch of metrics, obviously, that we track on the search side that we know are kind of proxies for whether or not someone has had a successful search session, and sometimes using… Measuring those existing metrics are exactly sufficient, but sometimes when you’re launching a new feature, it involves maybe a fundamental change to the product itself and it introduces a new behavior or a new metric that you weren’t tracking before that you have to now take into account. And so, in those cases, it’s really, really important to define how you’re gonna measure success, how will you know that you failed also. I think sometimes people don’t really think explicitly about that, but most experiments you run aren’t going to be successful. So kind of knowing what are the guardrails for… Even if we… Let’s say the way that we measure success is the number of people who click through, for instance, on a particular, let’s say, notification, even if you get 50% click-through rate on that notification, if you tank another key metric, that’s probably not great. So you also want to establish the guardrails for things that you… Metrics that you can’t degrade as well.
Daniel:
Right. So with that particular example, how would you know which potential metrics could be impacted by a given thing? It might be completely unrelated, and thus, unpredictable. Is there any way to try to systematize this and say how you create these guardrails in a way that is exhaustive and really is a guardrail and not something that might be maybe conditioned by our need to be successful at a given experiment? Know what I mean? I’m trying to find out if… How can you very completely determine which are the guardrails of things that you can not impact even if those things are not completely related to what you’re trying to do?
Lulu:
Yeah, so I think that kind of all comes back to your top level company metrics and we have… Our data science team at Pinterest has done a really good job of building out an A/B experiment dashboard that kind of lays out all of these core kind of guardrail metrics. So, daily active users, weekly active users, kind of just these very fundamental top line metrics that if you ship an experiment and conversion rate is 50%, but DAUs are down, Daily Active Users are down, then that’s never gonna be something that you ship. So I think having that core kind of dashboard of, “Here are the key metrics,” and everyone at the company knows that you launch any experiment, you need to come and look at these metrics first before you then look at kind of your experiment level metrics. That’s obviously super helpful, very clarifying, and actually pretty essential, to be honest, if you’re going to be in this experiment-heavy kind of culture. So I would say that’s the main thing, but to your point, there’s always gonna be sometimes unintended effects that you couldn’t… That you maybe somehow aren’t captured in the main dashboard 'cause it’s a new behavior or… Yeah, and in that case, that also happens to us. We will… We’ve definitely experienced occasions where we’ll launch an experiment and then just all of a sudden we’ll see a metrics drop and we’ll have to back.
We’ll have to roll back the launch and kind of debug and figure out. So it’s not a foolproof process for sure, but having kinda the key metrics centralized and be the default view for any experiment I think is definitely the best starting point.
Daniel:
Right, and I’m guessing that probably experiments are isolated from each other so you can know which ones are affecting which metrics, right?
Lulu:
Correct, yes. Yeah, there’s obviously a ton of best practices around running A/B tests. Yep.
Daniel:
Okay, we’ve gone through most of the topics that I wanted to talk to you about and I was wondering from your experience, what can you share with us in terms of bad experiences and things that you’ve learned of that you shouldn’t be doing when it comes to using customer feedback throughout the development cycle?
Lulu:
Well, I think we’ve kind of touched on some of these but I would say that… Being mindful. So yeah, a few things. First is I think just being mindful that any type of user research that you do, there’s a potential for bias in terms of the types of users that you’re hearing from. So when you launch a new product survey, for instance, you’re going to… Your responses are gonna be biased by your most vocal and engaged users. So as you’re kind of parsing through that feedback, you always have to keep that lens and that’s why you’ll often see as you’re going through, a lot of the future suggestions or future requests that we get from that type of research are… They tend to be power user features. And if we were to just base our roadmap off of those, we would only end up building things that were used by 2% of our users. So I think you have to be always mindful of selection bias when you are kind of analyzing qualitative feedback and then I think another thing to keep in mind as well is just… I think sometimes it’s harder to take feedback that you’ve been getting and to sit down and really try to spot broader trends or to try to think more deeply about what people are actually asking for because sometimes they…
People might say they want one thing or they feel XYZ about this other thing, but if you would just take that at face value and build the most obvious answer to that piece of feedback, that might not actually be what’s best in the long term. So I think taking the time to step back every now and then and really try to see if you could spot broader trends or themes from the feedback that you’re getting, that’s always a worthwhile exercise but it’s hard to do in the day-to-day. Like that kind of stuff always tends to get de-prioritized as you’re kind of tackling other near-term or more urgent things, but I think to be thoughtful about user feedback, that’s something that’s really important.
Daniel:
Right, well that’s it. It was really interesting talking to you and I wanted to thank you for your time today.
Lulu:
Thank you.