Archive for category: Survey Research Practices

Summary: The Net Promoter Score® is widely adopted and wildly controversial. What exactly is NPS and what are the various areas of controversy for using this survey question as a customer insight metric? This article provides a summary of the NPS concept and the critical concerns.

~ ~ ~

In an earlier article, I discussed the need for having confidence that the customer insight metrics we use in our operation are valid indicators of underlying customer sentiment. Here I am specifically focusing on those summary metrics we capture in our feedback surveys. This article will focus its attention on the Net Promoter Score®.

The Net Promoter Score (NPS) has achieved a near other-worldly place in the arena of customer metrics. It is certainly the most hyped measurement seen in decades, yet the fundamental measurement has been used for decades. What is new is the claim of the metric to predicting customer behavior along with the particular statistical technique applied to the data. After examining these points, we’ll turn to issues with NPS as the indicator of customer sentiment.

What is NPS?

NPS is quite simply the self-reported measurement of a respondent’s likelihood to recommend a product or service to others. That’s it.

How likely is it that you would recommend [Company X] to a friend or colleague?

This is not a new concept. The recommendation question has been asked for decades in customer satisfaction research. I can recall hearing a keynote speaker at a conference 25 years ago present the big three of summary customer measurements: customer satisfaction, likelihood of future purchase, and likelihood of recommendation. But now the claim is that the response to this question provides unique insight to a company’s forward profitability.

How Do We Know It’s Hyped?

Why do I say that NPS is hyped? Just look and listen to how NPS is used in business discussions. Companies aren’t doing customer satisfaction or customer relationship surveys; they’re now doing “NPS surveys.” Some companies actually have job titles that include “NPS.” Frankly, this is every consultant’s dream to have their trade-marked phrase become part of the business lexicon. So, call me jealous.

While many companies have never heard of NPS, you will find companies that have drunk the Kool Aid lock, stock and barrel. In these companies, NPS has achieved a mystical aura and everyone tracks the company’s NPS on a near real-time basis. NPS measurements are pushed down in the company, even to an individual level, across all departments. It’s even used to measure service provided internally within a company. This can border on obsession. One company with which I’m familiar recognized the obsession was obstructing focus on core business practices and scaled back its use of NPS.

I have seen many company surveys where questions are posed predominantly on 5-, 6-, 7-, or 10-point scales, but when it comes to the NPS question, Reichheld’s 0-to-10, 11-point scale is somehow sacrosanct. Sadly, I know that part of the reason is to allow comparisons of the company’s NPS against published NPS data. Any professional surveyor will tell you that such comparisons are highly dubious, given the differences in survey instruments and administration methods. People seem to treat NPS scores as if they’re accounting data following standardized practices. They’re not.

The Background for the NPS Claims

The Net Promoter Score first came to prominence with an article by Fred Reichheld, a Bain consultant, in the December 2003 Harvard Business Review, “The One Number You Need to Grow.” The article opens with Reichheld recounting a conference talk by the CEO of Enterprise Rent-A-Car, Andy Taylor. He talked about his company’s “way to measure and manage customer loyalty without the complexity of traditional customer surveys.” Enterprise used a two-question survey instrument; the two questions were:

• What was the quality of their rental experience?
• Would they rent again from Enterprise?

This approach was simple and quick, and we can infer from other comments in the article the survey process had a high response rate, though none is stated. Enterprise also ranked its branch offices solely using the percentage of customers who rated their experience using the highest rating option. Why this approach? Encouraging branches to satisfy customers to the point where they would give top ratings was a “key driver of profitable growth” since those people had a high likelihood of repeat business and of recommendations.

Reichheld, thus intrigued, pursued a research agenda to see if this experience could be generalized across industries. The first stage of this research was to identify what survey question correlated best with a person’s future purchase behavior. In most cases — but not all — the willingness-to-recommend question was the best predictor. Reichheld conjectures that the more tangible question of making a recommendation resonated better with respondents than the more abstract questions about a company deserving a customer’s loyalty.

The second research stage was to validate the recommendation question as a predictor of company growth. Through some unspecified statistical procedure, they decided to group the responses on a 1-to-10 scale into three groups. 1-to-6 respondents were “detractors,” 7-to-8 respondents were “passively satisfied,” and 9-to-10 respondents were “promoters.” (Note that later the scale was changed to a 0-to-10 scale. The addition of the zero supposedly clearly identifies which end of the scale is positive.)

Satmetrix — Reichheld sits on their Board of Directors — then administered the recommendation survey to thousands of people from public lists and compared the responses for various companies against the companies’ three-year growth rates.

The study concluded “that a single survey question can, in fact, serve as a useful predictor of growth.” The question was: “willingness to recommend a product or service to someone else.” The scores on this question “correlated directly with differences in growth rates among competitors.” (emphasis added) This “evangelic customer loyalty is clearly one of the most important drivers of growth.”

That’s the research basis for NPS as a customer insight metric. In comparison to the Customer Effort Score, this is a very robust research program. But Reichheld added another element to the conversation beyond just setting the recommendation question upon the pinnacle of what surveyors call “attitudinal outcome” survey questions. He also added a new statistic: net scoring.

The Net Scoring Statistic

Net scoring adds a certain mystical aura to NPS. We’re not just measuring a survey score; we’re taking a net score. A statistics professor of mine talked about the “Ooo Factor.” If a client would utter “Ooo” when a statistic was presented, then it had cachet. Net scoring has that Ooo Factor. It sounds so sophisticated. (In my workshops, I include the statisticians’ term for averages: “measures of central tendency.” Talk about an Ooo Factor!)

Well, a net score is a just a statistic to describe a data set. The mean (or arithmetic average) is also a statistic to describe a data set. The mean and the net score of a data set are very likely to tell the same story.

Quite simply, net scoring takes the percentage of data at the top end of the distribution, in this case respondents’ survey scores, and subtracts from it the percentage of respondents who gave us scores at the low end. The top end and bottom end scores are frequently called “top box” and “bottom box” scores, respectively. The technique can be applied to any survey question that has ordinal data properties, not just the “promotion” question — but that would dilute its aura.

The earliest reference I have found for this net scoring statistic is an online article by Rajan Sambandam and George Hausser in 1998, though they recommend multiplying the bottom box score by a factor of two or three to give it more impact.

   % 9s + 10s)
– (% 1s to 6s)
————————
Net Promoter Score

The goal of any statistic is to summarize a data set into one or two numbers that we can comprehend, digest, and apply to some decision. Reichheld advocates the net scoring approach to provide visibility to the bottom end of the distribution. Improving the views of those who are unhappy is the best way to drive overall improvement, and Reichheld wanted NPS to be used by the front line managers to improve operational performance.

However, net scoring actually throws away data distinctions, which the mean does not. A survey score of 1 has the same weight as a 6, a 9 the same as a 10. Aren’t those distinctions important? Aren’t the behavioral characteristics of a customer scoring a 1 likely to be different from a customer scoring a 6 or a 9 different from a 10? While the simplicity of the approach is a virtue, some valuable information can be lost through its simplicity.

Regardless, the idea that we’re calculating a net score whose tracking will lead to promotion of our business certainly has grabbed the attention of senior managers. And managers love a “number” to run their businesses.

The Issues With NPS

But maybe the hype about NPS is justified. And maybe not. Here are some of the concerns that have been raised with NPS.

Cannot reproduce the results. The most telling issue with NPS is that researchers have tried unsuccessfully to replicate the findings that Reichheld and Satmetrix developed as the core argument for the role of NPS in companies. This plays to the point of the earlier article about the need for reliable research.

Keiningham et al. in the Journal of Marketing, July 2007 present research using available data sources, including the American Customer Satisfaction Index (ACSI) since the data Reichheld used was not available. As this was published in an academic journal, it went through a peer review process which Reichheld’s research did not. The authors did not find that the recommendation question was superior in predicting company future profitability. In fact, the satisfaction question was superior. The authors conclude:

The clear implication is that managers have adopted the Net Promoter metric for tracking growth on the basis of the belief that solid science underpins the findings and that it is superior to other metrics. However, our research suggests that such presumptions are erroneous. The consequences are the potential misallocation of resources as a function of erroneous strategies guided by Net Promoter on firm performance, company value, and shareholder wealth.

Additionally, Morgan and Rego published research in Marketing Science (2006), examining “The Value of Different Customer Satisfaction and Loyalty Metrics in Predicting Business Performance.” They found:

Our results indicate that average satisfaction scores have the greatest value in predicting future business performance and that Top 2 Box satisfaction scores also have good predictive value. We also find that while repurchase likelihood and proportion of customers complaining have some predictive value depending on the specific dimension of business performance, metrics based on recommendation intentions (net promoters) and behavior (average number of recommendations) have little or no predictive value. Our results clearly indicate that recent prescriptions to focus customer feedback systems and metrics solely on customers’ recommendation intentions and behaviors are misguided.

If the research by Reichheld et al. cannot be replicated by others, how sound is the blind devotion to this measure?

Is NPS for relationship or transactional surveys — or both? Reichheld says that the metric is best for annual, high-level relationship surveys. Yet, he promotes using NPS to identify at-risk customers and driving the results to the front line to fix the customer relationships. This sounds more like something that should be done in transactional surveys. This seeming contradiction leads to the next issue.

The question is potentially ambiguous. Many companies pose the NPS question in relationship surveys with no wording adjustments. But consider how differently you might react to these two questions on a transactional survey after the closure of an interaction with a company:

• How likely is it that you would recommend [Company X] to a friend or colleague?
• Based on your most recent experience, how likely is it that you would recommend [Company X] to a friend or colleague?

In the first version, respondents may use different benchmarks for their answer. Some would base their response on their most recent experience, while others would use their overall experience in some undefined time period. How would you interpret the results if different benchmarks are used? Quite simply, you cannot. The definition of an invalid question is where respondents can have drastically different interpretations of the question. If using the net promoter question on a relationship survey, the ambiguity must be removed.

Where’s the promotion? While called Net Promoter Score, the question asks the likelihood of making a recommendation. The behavioral difference between recommending and promoting is cavernous. Promotion is active.  Recommendation is in response to a request for information. Shouldn’t it be called Net Recommend Score™ (NRS)? But that doesn’t have the same Ooo Factor, does it?

What about so-called Detractors? The bottom end of the scale is typically anchored as “highly unlikely to recommend.” If you’re highly unlikely to recommend, does that make you a detractor, someone who is actively going to give bad word of mouth?

Not applicable to all situations. In many situations people cannot make recommendations. Ask any government employee if they can make product recommendations. They cannot. They would be fired for fear of kickback schemes. Reichheld recognized this shortcoming in his 2003 article where he noted that NPS works better for the consumer product world than in business-to-business situations. Yet, that critical distinction has been missed.

The solution is to phrase the question as a hypothetical.

If you were able to make recommendations, how likely is it that you would recommend [Company X] to a friend or colleague?

Some companies have made this adjustment, but now our question’s syntax is getting complicated.

Creates a focus on getting word-of-mouth recommendations. Did the research show that getting recommendations led to higher company profitability? No. It showed a correlation between the likelihood to recommend a company with its future profitability. While we certainly want good word of mouth, it should be the byproduct of improved operations and product value. NPS has made the focus on getting recommendations.

It’s become a measurement tool, not operational tool. As mentioned, Reichheld viewed NPS as a means to drive changes in the front line, especially in addressing customer concerns. But NPS has become a performance measurement tool. This can lead to perverse behavior that can actually hide problems. Organization Behavior 101 teaches us that people will improve a performance appraisal measure — whatever it takes.

We increasingly see a business transaction conclude with the agent telling us that we’ll be getting a survey and if we can’t give them top scores to call someone before filling out the survey. I have personally experienced some truly over-the-top examples of this. On one hand, you might say, “Great, this is getting front line attention.” But on the other hand this may also mean that a symptom is being treated while the existence of the core problem is hidden from senior management.

Net scoring threshold effects. Net scoring is susceptible to big threshold effects. While a change from 10 to 9 has no effect on the net score, a change from 9 to 8 does. Small changes in a survey instrument design or administration that have nothing to do with customers’ underlying feelings could lead to these shifts, amplified by the net score. Research that I’ll be publishing shortly shows how survey mode — phone vs. web-form — impacts survey scores, and the net scoring approach amplifies the differences.

Cross-company comparisons. As mentioned earlier, many people think that scores on specific survey questions can be compared across companies without consideration for differences in the wording of the survey question, scale lengths, or the placement of the survey question within the broader survey. Such comparisons are highly dubious at best, but with NPS as an industry best practice metric means the comparison is done – blind to the issues. The best, most legitimate benchmark is against your own company’s previous survey scores.

More than one number you need to know. The myopic fascination with NPS, along with the title of the seminal article, “The One Number You Need to Grow,” has led people to think you only need a one-question survey, devoid of any complementary diagnostic questions or research. Reichheld himself doesn’t recommend this; however, he buried this information in his article in a side bar in parentheses. What people have heard is “The One Number You Need to Know.” Not true.

~ ~ ~

Measuring whether customers would recommend your company and its products and services is a valuable indicator of customer feelings. But does this one score deserve the exalted place it has achieved in the business world? How likely would I be to recommend NPS as the only customer insight metric? I’d rate it as a 2 – not very likely.

Note: NPS and Net Promoter Score are trademarks of Satmetrix Systems, Inc., Bain & Company, and Fred Reichheld.

Summary: Political pollsters have a lot at stake in their polling predictions — their reputations. Many of their challenges are the same ones we confront in surveying our customers or employees or members. Recent newspaper articles discuss some of the challenges pollsters are confronting, and we confront similar changes. This articles presents some of these challenges to survey design and administration.

~ ~ ~

Deep into silly season with the US Fall 2010 elections right around the corner, we’re all sick of the ads and the polls. At least we’ll get a respite on November 2 from all this, but you can be sure the 2012 election starts in earnest on January 2.

That said, what can we learn professionally from all the political polls? We certainly see differences across polls. The differences can result from many factors, factors that are also in play when we design surveys to measure feelings of our customers, employees, members etc. The Wall Street Journal (WSJ) has had a couple of research articles examining the differences, which are enlightening. – and you don’t see this kind of public discussion about organizational surveying practices.

WSJ’s “The Numbers Guy,” Carl Bialik, had a recent article on the question rating presidential approval. Gallup presents the respondent with binary options, approve or disapprove. In contrast Rasmussen, presents a four-point scale adding “somewhat approve” and “somewhat disapprove” options. The Rasmussen poll finds much lower net approval ratings – those indicating some level of approval minus those indicating some level of disapproval. The difference is significant, about 5 % to 15%. Scott Rasmussen posits that those with mild disapproval are hesitant to say “disapprove” when presented with only the hard, binary options.

I’ve see binary options in many surveys. For example, hotel surveys may ask, “Will you stay at our hotel the next time traveling to our city? Yes or no?” Sorry, the world is not that clear cut. Many factors will be in play. If I see this on a paper survey, I add in a “maybe” option. If it’s a web form survey, I usually will skip the question or say, “no” and explain why in a comment box, hoping someone will read it.

Bialik’s column also points out that very complex questions will lead to more people choosing “Don’t know.” Political polls are typically done by telephone, which makes paramount the need for simple phrasing.

Of course, the rubber meets the road on election day. Which polling method better predicts the election? Rasmussen in a recent radio interview stated that after the election, his staff does a full debrief to see what they got right and what they got wrong. This will guide them in refining their polling procedures and in their statistical adjustment models. The key challenge is identifying those likely to vote.

In our organizational surveys we too should debrief after every survey to see what we have learned about the surveying process to refine future procedures. Even the pollster’s challenge of identifying those likely to vote has parallels for us. We have ongoing debates about what question(s) best measure the attitude customers feel toward companies, e.g., the Net Promoter Score®. Ours is a different domain but the same challenge.

In an earlier interview in the WSJ, Scott Rasmussen suggests a real flaw in many pollsters’ approach. Most pollsters fall into what Rasmussen would call the “Political Class” whom he feels view the world differently than the “Mainstream Public.” He differentiates the two groups through a series of questions. Pollsters tend to live in the power centers of the country and are disconnected from the Mainstream Public.

When pollsters write questions, they apply their world view, which may not be shared by most respondents. The response options offered in the survey will reflect the pollsters’ biases, and many respondents won’t know how to respond since none of the options reflect how they feel, even if they do understand the question. Thus, those who do provide a response are more likely to be ones who share the pollsters’ world view. We see this vibrantly in the non-scientific polls on advocacy media, but it is also present in the “scientific polls.”

When your organization designs its surveys, do you leave your safe confines and actually talk to those in your group of interest to find out what’s of concern to them or are you certain you know how they view their relationship with your organization? Is that confidence really justified? I see this practice frequently in surveys where the mental frame of the survey designer doesn’t align with the mental frame of the respondent group.

A Bloomberg National Poll, released on October 12, 2012 — but no longer available online, displays some other common survey mistakes, as well as some good practices. First, they open with the summary attitudinal question of right direction vs. wrong track. As it is the first question, the response is unbiased except by whatever opening the interviewer used. But I’ve always been intrigued by the wording. “In general, do you think things in the nation are headed in the right direction, or have they gotten off on the wrong track?” (Emphasis in original.)

Why isn’t the choice “right direction” versus “wrong direction” or “right track” versus “wrong track”? The wording makes it sound like an unintentional accident that we “got off on the wrong track” rather than the leaders’ choices that have taken us in the wrong direction. I can’t speak to the impact of the uneven wording, but I am sure it has an impact. Results reported on October 12: Right Direction = 31%, Wrong Track = 64%.

Also in the Bloomberg poll, questions that present a list of options, such as what are the most critical issues, are rotated in the order presented to responses to eliminate any bias toward choosing the first option. This is a practice that makes sense in all surveys, but it requires a fairly sophisticated survey tool.

While the poll had many interesting practices to analyze, probably most telling was the question about what changes to taxation and government programs should be adopted to reduce the deficit. When you look at the answers, you find an “I want my cake and eat it too” message. We want low taxes, except for the rich who should be taxed to pay for all the entitlements and programs we like. What a surprise!

If it was only that simple, but the fact is that all public policy decisions involve trade-offs. Binary choice and interval-rating questions for each item don’t force respondents to consider the range of trade-offs. Fixed-sum questions force consideration of trade-offs, and conjoint surveys also test trade-offs better. However, those approaches are very difficult to deliver through telephone surveys. So, we see the impact of the administration technique used upon the validity and value of the information garnered.

In our organizational surveys, we are also confronted with this challenge. The administration method constrains what we can do in a survey. But if your research objectives are to measure the trade-offs our “constituents” consider, then we have to use a method that provides valid results. Unless of course, the results that don’t force consideration of the trade-offs are the results you really want to argue some position.

Finally, we see another shortcoming in play in either the administration or analysis. According to the poll, Democrats will retain their majority in Congress, 42% to 40%. Mind you, the question was presented immediately after a question on the positive elements of the health care law, so a sequencing effect could well have been in play. I am writing this on October 15. Their opening “right direction” question makes one question the results of this generic ballot question, and every other poll shows Republicans leading the generic ballot by high single digits or more. Why does this one poll show the opposite?

Maybe they’re right, but it could also be due to very elements of sampling bias. Bloomberg polled 721 “likely voters,” giving it a margin of error or 3.7%, according to the authors. They contacted randomly selected landline and cell phone telephone numbers. The poll says they weighted the scores by gender and race to reflect the recent Census data, which means we can assume they did not weight the scores to reflect political background, which most polls do. They also do not indicate the randomly selected calls were made to insure a geographic spread nor that any weighting done to reflect geographic distinctions. We also do not know how they assessed those 721 people were likely to vote.

Perhaps these administrative and analytical shortcomings explain the difference that this poll shows in that summary question versus other contemporaneous polls. We, too, in our surveys have to be aware of biases we introduce into our data set from question sequencing, word choices, and administrative methods. We may not be predicting elections, but we may be basing significant organizational decisions upon the results.

Summary: A long understood, but seldom followed, truism of organization design is that reporting for operational control and management control should not be mixed. Tools designed to provide front-line managers information about operational performance will become compromised when used for performance measurement. This is true for customer satisfaction surveys used for operational control, including Reichheld’s Net Promoter Score®. It was intended to be an operational control tool, but when used for performance measurement, we can see the deleterious effects. Customer feedback surveys are one element in an organization’s measurement systems, and the survey program needs to be considered in full context to be sure the measurements are not corrupted, and — more importantly — that the results of the survey program don’t create customer dissatisfaction in the process of attempting to measure customer satisfaction.

Nearby you’ll see an “Important Customer Satisfaction Notice!!!” that was attached to my repair bill from a local Subaru dealer. Take a read of it. Even though I do surveys professionally, I try to listen to my gut when I take a survey or see something like the flyer in the nearby image. How am I reacting to it? How would a “normal” person — that is, someone for whom surveying is not the center of his work life — react to this? What’s your reaction to it at both an intellectual level and a gut level?

Your first reaction probably is that the writer’s English skills are woeful — and that’s being kind. With my professorial red pen, I count 16 editing corrections in one page. Three people had their names at the bottom of the page. (I blanked them out for privacy reasons.) Didn’t they proofread this? Maybe they did, which would be really sad. If I were the owner of this dealership, I would be embarrassed. The sloppiness sends a message about the dealer as a whole. It’s a window into the concern for quality at the dealership. One can only hope they are better, more careful mechanics than writers!

Your second reaction is probably something like, “Why are they telling me this now? Why didn’t they just do a better job in the first place? Do they expect me to now lie on the survey?” I’ve shown this sheet in my customer feedback workshop, and that is the type of reaction I’ve heard.

Another reaction I’ve heard is that this “Notice!!!” isn’t about customer satisfaction. It’s about the dealership’s scoring from those in headquarters. That led the readers to really question the sincerity of the motives.

Plus, they put the onus on me, the customer, to reach out to them to fix their problems to then get me to give them high scores.

Lastly, they sure are using a lot of loaded language to impress you after the fact!

As a teacher of over 23 years, I can relate to the feeling. I frequently get students approach me to say, “I’m concerned about my grade.” Many times it’s a part-time MBA student who adds, “I don’t get reimbursed in full by my employer unless I get at least a B+ in the course.” Want to guess when they approach me? Yes, in the last week of the semester just before the final exam. Basically, they’re asking me to not apply the same grading standards to them as to the rest of the class. My response is usually something akin to, “If this is so important, why didn’t you work harder during the semester? And why are you coming to me now when the semester is over?”

During the ’80s I developed management and operational reporting systems for Digital Equipment Corporation’s field service division in the US. One lesson that I learned over and over again is that a reporting tool can’t serve two masters. If it’s to be used for operational improvement, then it should be used solely for that purpose. Once those operational control reports get used for performance measurement purposes, the validity of the data will deteriorate. Those being measured could improve their performance to affect the reported numbers or they could manipulate the data collection and reporting systems to affect the reported numbers – or both. Trust me, the data collection systems will be manipulated. At DEC we called it “pencil whipping” the forms that the techs filled out.

Fred Reichheld, the father of Net Promoter Scoring®, lamented about how his child is being misused. An attendee at the NPS® conference in early 2009, posted a blog about Reichheld’s comments. Essentially, Reichheld had envisioned NPS® as a tool to drive change in the customer-facing operations. But it is increasingly used as a performance metric club to beat people up. Worse yet, many companies think their feedback surveys only need to contain the “one number they need to grow.” A one-question survey cannot serve as an operational control tool.

Colleagues of mine, Sam Klaidman and Dennis Gershowitz, point out in an article of theirs that this focus on getting good NPS® scores creates the impulse for front line managers to “fix” the survey score before the score is submitted, as shown above.  The unintended consequence is that it creates a black hole in knowledge of what’s really happening at the front line. They refer to these as “gratification surveys.”

A survey program needs to be one element in a broader customer feedback strategic program.  If you plug in just a survey without thinking more broadly, then you will get reactions such as seen in the example above. I am glad that the dealership is taking customer satisfaction seriously, but the approach they’re using just feels wrong. Why didn’t the service advisor ask me questions when he handed me my bill? Or someone could have called me the next day. However, make sure the person placing the call is a professional and has been trained. I have received those kinds of calls from car repair shops, and it was clear that the person placing the calls had not been trained. It was an awkward call that left me feeling more skeptical about the dealership.  (It may have been this same dealership!) And don’t mention the survey. Once you mention the survey, you’ve crossed the line into manipulation. Subaru’s failure at the headquarters level was to implement the survey program without the additional elements of training those measured of what to do with the results, and to use this customer feedback system as a key element in their dealership reward system.

How would you react if you were getting beat up on survey scores? Perhaps the way this Subaru dealership has reacted: work the system. The Law of Unintended Consequences rears its ugly head. Ugh.

Have you found your survey program get corrupted when it started to be used for performance appraisal purposes? Let us know.

~ ~ ~

I received more response on this article than perhaps on any other one I’ve posted. Here’s one response.

This was my story one year ago. I took my car in for service for first time at this dealer. Price was outrageous and they did stuff not approved.

There was a copy of the customer survey form at the checkup desk, with a note to it attached saying they would get bad ratings if any area was not a 9 or above, so please fill it out that way basically. (can’t recall the exact wording, but that was the gist of it.) Then the gal checking me out told me the exact same thing — that I would be getting a survey from JD Power and they really wanted us to rate them 9 or above for them to keep their level rating.

Then, on my receipt, there was a note that if I would think of rating them anything below a 9 on any category to please call the Customer Service Manager. Well, I promptly gave him a call and told him that if they needed a 9+ so badly, perhaps they should try focusing more on service than just instructions about how to fill out the survey with the desired numbers. He apologized and offered to fix any of my service problems from the day before.

When those issues were resolved he called me to follow up. They were not resolved to my satisfaction and I told him so, and also told him I would not be giving the 9 scores he was looking for.

I never received a survey.

Penny Reynolds
Senior Partner, The Call Center School

Common wisdom is that only large companies need customer-research programs. After all, small companies have their feet on the ground and know what customers are thinking, right? On the surface, that seems a reasonable attitude for small-business managers to take. But ask yourself if there is some percentage of customers defecting to competitors each year for part or all of their purchases.

If so, then some formal customer-research program, especially surveying, can yield a quick payback by shedding light on the reasons for the defections.

Let’s expose some other myths of customer surveying.

Surveys don’t provide data that can be used. There can be some truth here. Survey results are primarily numbers. The challenge — assuming the numbers accurately reflect respondents’ views — is to give voice to the data. This voice comes from designing a questionnaire with an understanding of how the data will be used, and then performing proper data analysis.

Sometimes the voices are a scream for help. In fact, identifying those customers needing “service recovery” should be a key goal of a survey program. Such actions may stop a defection before it occurs. Most often, the voices are pointedly identifying shortcomings in a business process. An ongoing survey program can play this quality control function. Sometimes the voices from the data are more subtle, requiring correlation to demographic data to uncover the phenomena that are in play.

Surveys should be viewed as part of a broader program of understanding customer needs and concerns.  In fact, a survey may pose more questions than it answers, but these are questions that likely need to be raised — and you can’t find answers until you know the right questions to ask.  Follow-up and in-depth research should yield the specific improvements that are needed.

Customer survey programs are too expensive for small businesses. While it’s true that large companies can spend eight figures on research programs, a small business can perform a very adequate job on a reasonable budget. Of course, this would require that much of the work is done in-house, but the advent of electronic-survey tools makes surveying far more accessible. Good desktop tools are available for less than $1,000, and there are even free, hosted survey services.

You might still consider outsourcing in areas of the program that require specific expertise or operational capacity that simply is not available. Outsourcers will also help drive a project schedule. A pure in-house survey project may fall prey to higher-priority concerns and languish undone.

No one on staff has the skills to do a survey. Much of sound research practice is applied common sense, and someone with a market-research background should have exposure to the survey research discipline. In fact, it’s possible that your staff has skills it doesn’t know it has. If you’re skilled at interviewing, then you have experience in eliciting information in a structured way. Don’t sell your skills short, but do know your shortcomings.

Surveys are just a bunch of questions strung together. A quote from Mark Twain has merit here: “It ain’t what you don’t know that gets you, it’s the things you know that ain’t so.” Survey instrument design is a craft and arguably the most challenging part of a survey program. If you think you have that craft and you don’t, then you may create an instrument that provides misleading information.

Look at the following survey question: “How satisfied were you with how quickly your call for information was answered?”  Show this question to 10 colleagues and ask them what they think is being asked.  You may get 12 different answers.

This question, apparently benign, is loaded with ambiguity.  What’s being “answered,” the phone or the question?  Does “quickly” refer to the number of phone rings before someone or something answers, how soon you were in contact with a person who could answer the question, how quickly you got the answer? Fleshing out ambiguous terminology is a real challenge.  By the way, if you do present the question to your colleagues for review, you’ve developed the skills for pilot testing your survey instrument.

As mentioned, don’t sell your skills short — but know where they are short — and a survey program can yield a very fast return on investment by listening to a now more loyal customer base and applying the findings to improvement programs.

From the print edition of Boston Business Journal