One of our current methodologies involves a survey distributed to a randomized sample of a program’s (i.e., community or online consumer panel) members asking questions related to the conversations they’ve had offline in the past month about that brand or product.  One question we recently added to the survey asks, “How likely are you to recommend [product/brand] to a friend or colleague”?

This question probably sounds familiar.  It’s the “Ultimate Question” as defined by Fred Reicheld in his book of the same name.  The Ultimate Question’s purpose is to determine your Net Promoter Score, that is, the ratio of promoters to detractors of your product or brand.  This score is designed to provide the measure of a company’s performance through the customer’s eyes.  It’s a widely accepted, yet controversial metric for determining an organization’s ability to grow.

Here’s how it works.  Consumers are asked, on a scale of 1-10 (with 1 being extremely unlikely and 10 being extremely likely), “How likely are you to recommend an organization to a friend or colleague?”  Those who answer 0-6 are “detractors” and those who answer 9-10 are “promoters”.  You subtract the percentage of detractors from the percentage of promoters to determine the Net Promoter Score (NPS).

One question we’ve always asked in our WOM surveys has been “How many conversations have you had about [product/brand] in the past month?” with the answer based on a 0-10+ scale.

We decided to dig up 6 months of historical WOM survey results and match the NPS results with our conversation metric to see how they fit together, and were surprised by the results.  Note: All three clients are of different verticals.

Basically, the more conversations a customer has, the more likely they are to recommend the product.  No surprises there.  But what’s most interesting is how conversations mimic the assumed detractor/promoter values of the Net Promoter Score.  According to the historical results of three of our existing clients, those customers who had 0-6 conversations are essentially detractors, unlikely to recommend the product or brand to a friend.  At seven and 8 conversations the NPS spikes, and 9-10 are significantly higher, extremely likely to recommend the  product.

Now admittedly this isn’t a conclusive correlation with only 6 months of historical data, but the striking resemblance of “conversations per month” to the Net Promoter Score begs the question, why are consumers who are having up to 6 conversations per month about a product or brand ‘unlikely’ to recommend the product to a friend or colleague?  Is this a flaw in either our or the NPS methodology?  Are customers over-reporting their conversations or under-reporting their likelihood of recommendation?  Are consumers aware of the difference between a product conversation and a product recommendation?  Share your thoughts with us!

The first, Is the Tipping Point Toast, discusses challenges being made to the “influencer” model made famous in WOM101 books The Tipping Point and The Influentials. In it, Duncan Watts, a researcher at Yahoo! with a background in network theory (who has conducted a series of experiments, both physical and computer-based, to model the spread of “trends”) argues that highly connected people are not crucial social hubs and a trend’s success is completely random.

“It just doesn’t work… A rare bunch of of cool people just don’t have that power. And when you test the way marketers say the world works, it falls apart. There’s no *there* there… If society is ready to embrace a trend, almost any one can start one – and if it isn’t, then almost no one can.”

The second, Customer advocacy metrics: the NPS theory in practice (published in Admap magazine), provides a critique of Net Promoter Score (NPS) (made famous in the book The Ultimate Question)and proposes ways the metric can be improved. NPS…

“is based on the question ‘Would you recommend [Brand/ Company X] to a friend or colleague’, answered on a scale between 0 (not at all likely) and 10 (extremely likely). The actual score is computed by subtracting the percentage of detractors (those giving 0–6 answers) from promoters (9–10s). The middle section, between 7 and 8, is so called passives.”

The article argues that while NPS tries to be a “one size fits all” metric, marketers need to understand its advantages and limitations for different types of uses, and that is it better suited as a loyalty metric than an indicator of future business growth. Among the critiques:

“Critics, however, do not like the fact that useful scaled information is later lost in conversion. Are people giving a 0 rating equally detracting as those with a 6? Intuitively: no. Similarly, once percentages are calculated, is an NPS of 40, consisting of 70% promoters and 30% detractors, the same as the same NPS consisting of 40% promoters and 0% detractors?”

So, where does this leave us? Regardless of which side of the fence you sit on regarding these issues, it’s great to see that people are challenging WOM-related theories and metrics (I personally find flaws in some of the methodologies used in some of Watts’ experiments and feel that NPS isn’t intended to be a hard, de-facto research metric versus a rallying cry for employees at organizations looking to make a commitment to customer satisfaction).

To be honest, there is still a lot of “hand-waving” that occurs in the industry in terms of metrics, and as our clients increasingly put us to the task of proving the value of our programs, it’s becoming even more important for us to be able to support and back up our findings.

Related links:
Is the Tipping Point Toast? (Fast Company)
Customer advocacy metrics: the NPS theory in practice (Admap)
The Tipping Point (via Amazon)
The Influentials (via Amazon)
The Ultimate Question (via Amazon)