The ultimate or the last question? How NPS could compromise your understanding

ultimate questionThe number of top executives who is asking for NPS (Net Promoter Score) has grown permanently since the famous article by Bain & Company mentor Frederick F. Reichheld (The One Number You Need to Grow, Harvard Business Review, December 2003). Main reason of such success is not the NPS’s predictive ability but its own simplicity.

Top executives hasn’t enough time to read huge amounts of pages describing complex (although elegant) analysis or sophisticated indicators. NPS seems to provide a good recipe: it is simple to explain, it is short (“The one number…“) to show, it is clear to understand.

Like several examples we can catch from other contexts, NPS is a “postulate-driven” assumption: if I recommend, I am a promoter; if i don’t recommend, I am a detractor. Like other postulates, NPS has a corollary: company revenues are predicted by NPS.

I don’t agree. Simply. I don’t  mean I don’t care Top executives’ need to receive few, clear and self-standing results that could provide a compass for growth. I strongly suggest to understand that a postulate has to be weighted before to be accepted.

A careful observation of NPS could drive us to a healthy criticism. And I say healthy in an ontological meaning (every theory must be put to the test), but also in a restricted meaning (if you want a compass for growth, be sure it is calibrated).

I will describe main mistakes that may occur to you if you don’t accept a healthy criticism on NPS. Mistakes are divided between

A. NPS’s own lack of foundation

B. Self standing NPS risks

A1. Equivocal definition of detractors

We know the theory. NPS is based on a single question: How likely are you to recommend our company/product/service to your friends and colleagues? Answers are given by a 0-10 (normally) scale where 0 = Not at all likely and 10 = Extremely liking. People who answer 9-10 are considered promoters; 7-8 passive; 0-6 detractors. NPS = % Promoters – % of Detractors.

A customer of mine asked me recently why the detractors’ range is so wide if compared with other two. Probably is the most clever assumption in NPS: distributions of scores are often concentrated in the second half of a scale. This is not the real inconsistency of NPS assumption.

The real inconsistency is that… 0-6 respondents are not detractors necessarily. A low likelihood to recommend can rely on a whole set of causes. Some examples: I love unix-like operating systems. I used (happily) Linux machines for a long time (my preferred distro was Debian – probably the most “free software” militant distro). If I ought to answer to the question how likely am I to recommend Linux migration to my company now, I would answer 3: too much time to train the staff. If you are a domestic laptop user I would answer 10: Linux offers you a huge amount of software normally you have to pay for, Linux has a strong security architecture and it is virtually virus safe (if you are not a stupid user) and… a lot of further reasons. I am not a Linux detractor: in certain situations I am silent. A detractor is someone who discourage to purchase or use products/services.

For my clients I offer a version of NPS that I call NPS cleaned (against NPS official), it is sufficient to add a second questionto 0-6 respondents: How likely are you to advise against our company/product/service to your friends and colleagues?

An empirical example: NPS official = +18.1 – NPS cleaned = +26.9 (60.4% of “detractors” in NPS terms answered thy are not likely to advise against the Company).

Another example. In the case of a bank, NPS “detractors” was asked which reason drove them to such low vote and 30% said “I don’t recommend banks regardless”, 16% “I have no particular/specific reason”. NPS doesn’t ask if a customer could have the chance or willingness to recommend. NPS calls Detractors people who are merely silent.

A2. Mathematical aporias

You could have a NPS = +20 if you have 20% of 9-10 votes and 0% of 0-6 votes, but also with 40% of 9-10 votes and 20% of 0-6 votes. Also assuming NPS definition of Detractors is correct, if you were a Company CEO, which scenario would you prefer? Proportional changes in Promoters and Detractors percentages drive to the same NPS.

If you – my 25 readers – were statistical lovers, I could also mention the low reliability (of a single-item measure against multiple-items measures. I will steal a good quote: “Would you want your child’s Scholastic Aptitude Test score to be determined by a single question or the entire set of questions on the test?” (thanks to Bob Hayes). Okay… let’s think your Company’s Loyalty survey….

A3. A tricky claim

Brother Reichheld teaches us NPS is the “one number” (I think he means the “unique number”). He is sincere but… by half. one number and a second question, an open-ended question who asks detractors the reason of their vote. Okay: the one number plus the one open-ended question would have been more transparent as claim. Probably less persuasive.

B1. Factual aporias

A lot of examples could be brought to paint a contradictory scenario: NPS grows and… customers decrease. So revenues. If a point-of-sale is losing customers, the remaining ones are more likely to recommend it to the neighbours: “Oh, madam, I made shopping for all the family in 10 minutes. There were nobody”.

It happens also with Customer Satisfaction Index if measured with single item (i.e. overall satisfaction): ok guys, single numbers are quite dangerous.

B2. Cultural troubles

I don’t know where are you from. But in Italy an 8 is not a “neutral/passive” vote. Ask to whoever. Ask to a student or to a teacher.

 

Not the “ultimate” question, but the “last” question

In his famous book, Reichheld spoke of “the ultimate question”. He means the top question, the mother of every question, etc. I prefer to call it “The last question”. Likelihood to recommend has to be considered as last part of a complete set of questions exploring customer experience. Let’s assume the Bain  guys are right. If my NPS grows, my company registers more revenues. But how can I make my NPS growing? People are recommending me… and if I want to increase their recommendation likelihood?

Recently I met a down-hearted executive of a great financial multinational company (I will not name him). He told me “Do you know how the headquarter uses the NPS? Only to tell us we are worst than French branch”.

 

Few good resources:

  • Timothy L. Keiningham, Bruce Cooil, Tor Wallin Andreassen, & Lerzan Aksoy – A Longitudinal Examination of Net Promoter and Firm Revenue Growth, July 2007, Journal of Marketing 71 (3): 39–51. Authors questioned the assumed correlation between NPS and revenues using empirical data and demonstrate the higher predictivity ACSI (American Customer Satifsfaction Index) with respect of NPS. Timothy L. Keiningham is Global Chief Strategy Officer and EVP at Ipsos Loyalty, not a CFI Group officer…
  • Bob E. Hayes (2008), “The True Test of Loyalty,” Quality Progress, June 2008, 20–26. The Author stated that the “likelihood to recommend” question does not measure anything different from other conventional loyalty-related questions and in particular he questions the reliability of single-item measures
  • Schneider, Daniel; Berent, Matt; Thomas, Randall; Krosnick, Jon (2007): “Measuring Customer Satisfaction and Loyalty: Improving the ‘Net-Promoter’ Score“; paper presented at the Annual Conference of the World Association for Public Opinion Research (WAPOR); Berlin (Germany). Authors found that out of four scales tested, the 11-point scale advocated by Reichheld had the lowest predictive validity of the scales tested.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s