Product Management | 9 min read

Making sense of customer feedback

hero

Recently we told you there are some customer opinions you shouldn’t listen to. But after you’ve weeded out the aspirational, the hypothetical and the third party statements, what customer feedback should you listen to? And how do you go about making sense of it?

First, it’s important to understand that even after you’ve weeded out aspirational, hypothetical and third party statements, the feedback that remains is unlikely to be all of equal value. Here are some important additional filters you can apply to help you decide which feedback is most important:

1. Who’s giving the feedback matters

customer feedback segments

Do you pay equal attention to all the nuggets of wisdom people give you? Unlikely. Chances are the friends you’ve known the longest are the people whose opinions you’ll trust most. The stranger you just met on a bus who told you emphatically what you should do with your life? You’re probably not going to put as much weight on their views.

In a business situation, the customer’s relationship with your business influences how much weight you give their feedback. Customers who have been loyal the longest have a wealth of experience with your product that makes their opinions particularly valuable. Have some customers who only started using your product six months ago but use it heavily? They’re likely to have a lot of insights. Have some customers who pay significantly more than others? You may want to factor that in too.

2. Whether it’s prompted or unprompted feedback matters

customer feedback vocal minority

Unprompted feedback deserves special attention. Here’s one key reason why. The customer issues that aren’t on your radar, that you’re completely unaware of, can be the most important things you need to hear. You’re more likely to hear those left-field issues via unsolicited feedback or from open ended survey questions rather than, say, a survey with multiple choice answers. There’s a reason doctors ask if there’s “anything else you want to talk about?” at the end of your appointment. It often triggers the patient to talk about their most important issue.

3. Motivations matter

customer feedback J curve

Remember people are generally motivated to provide unsolicited feedback if they have an extreme experience. That’s why you see Yelp restaurant reviews clustered around the “Amazing” and “Appalling” end of the spectrum. People perceive they’ll gain social capital from telling others about the great restaurant they just went to or by warning others against a terrible restaurant. But the night your dinner was really average? You’re probably not going to bother writing a review because, well, what’s the point? It’s not a very interesting story is it?

The type of distribution that results in this kind of restaurant review data is often a J curve. The “J” shape refers to data where the the curve initially falls but then rises to a higher point than the start.

When it comes to customer feedback you receive about your business, you can expect there to be a similar pattern. Your customers are more motivated to tell you when they are very happy or unhappy about your product. However, this doesn’t mean that your customers only love/hate your product. You’ve probably got a large group in the middle who think your product is “fine”. These customers typically stay silent. Remember they could have also have useful feedback for you. If you’re smart, you’ll find ways to tease out their feedback.

4. Volume matters

customer feedback vocal minority

The volume of feedback about an issue matters. If 80% of your customer feedback in the last month is telling you that the “improvement” you made recently to your core product has broken people’s workflow, you should listen up. The overall volume of feedback about a single issue relative to other issues matters. It will also protect you from “fre-cently” bias, where people assume things they hear frequently or recently have the greatest importance.

5. Repetition matters

repetitive customer feedback

User issues are often dismissed on the grounds that “Oh we’ve heard that for years”. Maybe you’re planning to finally address that issue in a big redesign next year. Or more likely this request has become so repetitive that it’s become trite, a sort of dull whine that nobody listens to anymore. Either way, this kind of feedback is really worth listening to, especially when it relates to product quality, bugs, or difficulty achieving a core task in the product. It’s an indicator you haven’t got the basics right, and that’s something you have to address as a priority rather than ignore.

6. The stakes matter

high stakes customer feedback

Some feedback is worth listening to purely because of the severity of the problem the customer is experiencing. This is high stakes feedback. Perhaps you pushed a release that had a security loophole, or your product has accidentally put consumer’s privacy at risk. When reviewing customer feedback, you need a mechanism to alert you to this kind of very occasional but high stakes feedback so you can take action straightaway.

So, how can you analyse open ended customer feedback?

Here’s one effective technique if you have a bunch of open ended feedback that you want to make sense of. There is, unfortunately, no perfect tool that can automate this job for you. Analyzing open ended feedback is just pretty hard and time intensive. However, if you follow these steps, you’ll have a prioritised list of customer insights you can act upon with confidence.

1. Collate your data

First, collate all the open ended customer feedback you want to analyse, plus key metadata about each customer, into a spreadsheet. Ideally, the metadata will include attributes such as how long they’ve been a customer, how much they spend, date the feedback was submitted, and the source of the feedback (e.g. open ended survey question). Of course if you’re an Intercom user gathering this data on your customers is a cinch. Your column headings should look something like this:

View larger image

2. Get an overview

You want to get a feel for the data before starting to codify it. Scan through the feedback to get a sense of how diverse the responses are. Some themes should start to emerge the more you read.

3. Create a list of “feedback types”

You’re going to put each piece of customer feedback into a high level group. A high level list could be something like this:

  • Usability issue
  • New feature request
  • Regression (examples of where product appears to have gotten worse)
  • Bug (or potential bug)
  • Generic positive (e.g. “I love your product!”)
  • Generic negative (e.g. “I hate your product!”)
  • Junk (useful for nonsense feedback like “jambopasta!!!”)
  • Other (useful for hard to categorise feedback. You can recategorize it later as patterns emerge)

4. Draft a list of “Analysis Codes”

Based on your impressions of the feedback you’ve read through so far, draft a list of codes you can use to break down the feedback into actionable buckets. Examples will be very specific to your product but here are a few analysis codes for new feature requests to give you a flavour:

  • Ability to assign a task to multiple clients
  • Ability to add complex HTML to tasks
  • Ability to add or remove teammates from any screen
  • Ability to send emoji to clients

5. Code the feedback

Time to roll up your sleeves and focus. Find a place you won’t be disturbed and read through all the feedback, carefully coding each row. Add multiple codes per feedback statement in separate columns if necessary (sometimes customers provide multiple pieces of useful feedback in one paragraph). Try and restrict the number of codes you use overall.

6. Start with high level codes, then refine

It’s OK to start with higher level codes and break them down later. Pay attention to the exact language people use. Issues that sound similar upon first read through might actually be separate issues. For example, imagine you see a lot of feedback related to “email issues”. However, when you read the feedback carefully, you realise that these break down into separate issues: “Email composer bug” and “Email delivery bug”, which are quite different.

Sometimes, as you read more feedback you realise that you need to break one popular code down into a couple of more specific codes. That’s fine, go ahead and split it up into sub-codes. For example, “More control over visual design” could be broken down into “Ability to add fonts” and “Ability to control alignment of images”. Remember to go back and recode the earlier rows.

7. Calculate how popular each code is

Once you’ve coded everything, add them up. If you have a small data set, you can do this in your head. If you have a lot of feedback (i.e. more than 100 statements) copy the “Analysis code” column to a new sheet. Add up how many times each code appears. One way to do this is to sort alphabetically to group items with the same code and then highlight all cells that have the same code and a count will appear in the corner of your browser. Create a table to summarise the total issue counts.

8. Rank your customer issues

Now you can create a summary of top user feedback based on issue popularity and discuss it with your team. Want a more sophisticated ranking? You can also break the data down by the variables we discussed earlier. For example:

  • Break the feedback down into feature requests, usability issues, regressions, bugs, etc, and discuss next steps with the relevant team members
  • Extract the High Stakes feedback so you can action that as a priority
  • Use the customer feedback to make decisions on your product strategy: be explicit about what’s out of scope, as well as what’s in
  • Prioritise the feedback according to customer type
  • Compare patterns of feedback between different customer types
  • Start tracking persistent issues over time so you can see if any basics are being consistently neglected

Mining for valuable insights

So whilst there is some customer feedback you can ignore, there’s a huge range of customer feedback that you can mine for valuable insights. Hopefully this post has helped you understand some techniques you can use to make sense of a large amount of open ended customer feedback and turn it into an actionable list of issues your team can address. Closing that loop is the key: at Intercom we use this technique to turn customer feedback into a clear set of priorities that directly inform our product roadmap.


Further reading:
Intercom for your product research
More research articles from our blog