Archive for category: Survey Project Management

Summary: Incentives are part of many email requests to take a survey. Recently, I saw this taken to a shocking extreme. I was blatantly offered a bribe to give high scores. This article will cover the pros and cons of incentives. Incentives are known to be a two-edged sword. But this example highlights how incentives can corrupt survey data.  Are the data real or fabricated?

~ ~ ~

Ever had someone try to influence your willingness to answer some survey? Of course you have. Many, if not most, surveys come with some offer to get you to take the survey, but I recently had a truly blatant example that made my jaw drop. As I describe this “incentive,” I’ll place it within a framework that will allow you to think more deeply about the interaction between the drive for higher response rates to reduce non-participation (or non-response) bias and the potential to corrupt the data you intend to analyze.  Response rates and incentives are always a vibrant discussion topic in my survey workshops.

Everybody wants higher response rates perhaps because it’s one of the easiest survey factors to understand — even in the C suite. More responses means higher statistical accuracy. So, surveyors put considerable emphasis upon improving response rates. Perhaps so much that new problems are created.

In addition to better accuracy, getting a higher response rate reduces the non-participation (or non-response) bias. With a low response rate you are less sure that the data collected properly represents the feelings of the overall group of interest, i.e., your population. This bias or misrepresentation in the data set is created by those who chose to not respond. That’s why it’s called a non-response or non-participation bias. Consider elections where certain groups are less inclined to vote. That’s a participation bias.

Many factors influence response rates and thus the non-participation bias:

• Administration mode chosen: each modes affects demographic groups differently
• Quality of the survey administration process, especially the solicitation process:  effectiveness of the “sales pitch,” guarantee of confidentiality, frequency of survey administrations, reminder notes, sharing of survey findings with respondents
• Quality of the survey instrument design: length, layout, ease of execution, engaging nature of the questions
• Relationship of the respondent to the survey sponsor
• Incentives

While the relationship of the respondent audience to the surveying organization is without doubt the most important driver of response rates, my focus here will be on the impact of incentives. Why? Because in driving for more responses through incentives, we may actually corrupt the survey data by introducing erroneous invalid data — a response bias.

I recently bought a license from SideMicro.com for a Windows 7 license to run under Bootcamp on my new Mac laptop. The transaction was perfectly fine. Good price. Got the link to download the software. The product key worked as expected. I then got a follow up survey request by email. Here’s what it said: (See screenshot.)

Please Give us A 5 star Good Review ,And We Will Give You $25.00 Visa Gift Card Or Store Credit Immediately. We Appreciate Your Time .

Wow! Before I dissect this, let me point out a deception. Would you rather have a $25 gift card or a “store credit” — whatever that is? Cash is king. What did I get — and probably everyone else who fell for the hook? A coupon code good for 10% off additional purchases. I actually wrote to them to say that I didn’t appreciate the deception, but I got no response.

They got what they wanted — a high score for the ratings folks. But am I now more or less loyal to them? If I bought from them again, would I comply with their request for high scores? That answer is obvious.

Lesson: The survey process itself can affect loyalty.

Now let me turn to incentives. The goal of an incentive is to motivate people to take the survey who don’t feel strongly and wouldn’t otherwise take the survey. That reduces the non-participation bias.

Incentives fall into two categories depending upon when the incentive is provided to the respondent:

• Inducements. The incentive is provided with the survey invitation. Example: a survey by postal mail with some money inside. We have no guarantee that the person will take the survey.
• Rewards. The incentive is provided after you take the survey.

For the same budget we can offer a larger reward vs. inducement.

Here’s a key question. Is the incentive motivating people to:

1. Take the survey and provide legitimate, valid data? or
2. Just click on response boxes to get the prize, that is, provide bogus, invalid data? (I call this monkey typing.)

An incentive should be a token of appreciation. If the incentive increases response rates reducing non-participation bias, that’s good. But if it leads to garbage data being entered in the data set — a response bias — that’s bad. More, but bad, data is not the goal! If we offer incentives in our survey programs, we all know in our hearts that some people are just monkey typing, but we want to make that minimal. I have actually counseled clients to reduce the size of their incentives for fear it would promote unacceptable levels of monkey typing.

Consider the incentives you’ve been offered to take a survey. We’ve all had service people try to influence our responses on a survey. Some are subtle and unobtrusive, for example, by actually doing their job to a high degree of excellence, which is the behavior that a feedback program should create, or politely telling us that we might get a survey about the service received and hope that you’ve had a good experience. Then there are the blatant pitches, such as the car salesperson who hands you a copy of the JD Power survey you’ll be receiving — with the “correct” answers filled in.

On a scale of 1 to 10 where 10 is outrageously blatant, this pitch from SideMicro gets a 12. I have never seen such a overt bribe to get better survey scores. In an odd way, though, I respect their honesty. They weren’t coy or obsequious. They were direct. “We’ll pay you for a high score.”

Regardless of how we feel about this manipulation, the effect it creates is called a response bias.  A response bias occurs where the respondents’ scoring on a survey (or other research effort) is affected by the surveying process, either the questionnaire design or the administration process.  In other words, something triggers a reaction in the respondent that changes how they react to the questions and the scores they will give.

Think about the surveys you’ve taken. Did something in the solicition process or the questionnaire itself affect how you answered a question?  The result is that the data collected do not properly reflect the respondents’ true views.  The validity of the data set is therefore compromised, and we might draw erroneous conclusions and take incorrect action based on the data.

Response biases exist in many flavors and colors. Conformity, auspices, acquiescence, irrelevance, and concern for privacy are common types.  Incentives typically create an irrelevancy response bias, but it could also create an auspices bias.

• Irrelevancy means the respondent provides meaningless data just to get the prize. You  might see this in the pattern of clicks where the radio buttons are in the table to the right of the screen. You might see straight line answers — all 1s or 10s — or the famous “Christmas tree lights” diagonal pattern.
• Auspices is where the respondents gives the answer the respondent wants, which was the bias created from the payoff by SideMicro.

Lesson: A goal of the survey administration process is to generate a higher volume of valid data that reflect the views of the entire audience. Incentives are a two-edged sword. They may positively reduce the non-participation (non-response) bias, but that could be at the cost of introducing measurement error into the survey data set from bogus submissions.

That is, unless the goal of the survey program is purely for marketing hype, as is the case with SideMicro. They clearly don’t care about getting information for diagnostic purposes.

Lesson: Many survey “findings” we see from organizations are simply garbage due to flawed research processes.

Summary: Survey fatigue is genuine concern for surveyors. As more companies survey, the willingness of people to take all these surveys plummets. Applying the golden rule of surveying, can help you stand out, reduce the survey fatigue for your respondents, and increase survey response rates.

~ ~ ~

Tragedy of the Commons. Funny how some concepts learned in college have stickiness — and relevance years later in ways not envisioned. The fundamental idea of the Tragedy of the Commons is that a resource that is owned in common will be overexploited. Each individual, acting rationally from his or her point of view, will use the common resource to the point where collectively it will be seriously degraded.

The idea was described by Garrett Hardin in a Science article in 1968 using a town’s common grazing area for the town folks’ animals. One of today’s “commons” is customers’ mindshare. We all fight to get a piece of our customers’ – or prospects’ — attention to the point where it becomes a cacophony and no one gets these people’s full attention.

Consider customer feedback surveys. The New York Times printed an article in March 2012 about the “onslaught” of customer surveys. I couldn’t have used a more apt term — and perhaps because I train people on how to conduct customer survey programs, I am part of the problem! Survey fatigue — or survey burnout — is very real.

Remember when getting a survey invitation to take a web-form survey was novel and fun? (Heck, I can remember when email was novel and fun back in the ’80s.) I do survey design work for a living, so I have a natural incentive to take most any survey. Hey, I might learn something — or find grist for a future article. But even I have reached the burnout stage with the constant requests to give feedback on some website I’m using.

Why has this happened? As the cost of surveying has plummeted due to internet-based survey tools, more and more companies are conducting surveys. The “common” of people’s time, attention, and willingness to take surveys is being overgrazed. If the surveys were well-done, brief and to the point, the overgrazing might be tolerable. But so many surveys are so poorly done — for example, Home Depot’s way too long survey and Yahoo’s insulting survey — that the overgrazing has reached the point of eating the roots, leaving little for the rest of the organizations who might want to collect customer feedback.

Even worse, some surveys aren’t really surveys. They’re a lead generation marketing pitch — or a phishing attack — disguised as a survey.

You might say there’s lots of pastureland from which to graze for respondents, but the problem with survey fatigue is twofold, as I explained in a recent article on survey bias versus survey accuracy.

• Fatigue leads to lower response rates, reducing statistical accuracy.
• Fatigue means that those with extreme views are more likely to respond, leading to a serious survey bias, known as non-response bias. That is, those who don’t respond likely have different views from those who do. The survey data that winds up in the survey data base don’t properly reflect all customers’ views. The data are biased.

So what to do? Well, you cannot control other organizations surveying propensities, but by doing a survey program right, perhaps you can shine among the rest.

First, don’t over survey the same people. This is a particular concern for transactional survey programs. You should have some time window in which you would not survey the same person again. My default window for this is three months, which seems reasonable. And don’t survey people who say, “Stop sending me surveys!” That just creates bad word of mouth and could lose a customer.

Second, coordinate your organization’s survey programs. Are other groups in your organization doing surveys that you don’t know about? Two concurrent survey invitations to the same person from different survey programs in the same company presents a really bad image to the respondent. This is why many large companies invest in large (and expensive) customer feedback management systems and a central feedback program office which controls customer contact lists.

Third, keep the surveys short, to the point, and most importantly, engaging. Years ago I took a hotel survey where on the 8th or 9th screen — with many questions per screen — the progress indicator registered 35%. Resist the temptation to turn your transactional survey into a relationship survey. Take a Home Depot “brief” store visit survey and you’ll see what not to do. Market researchers and product managers — no offense if you are one — seem to have very unrealistic assumptions of respondents’ commitment to complete a survey. Know your audience and how bonded they are to you. That will guide their survey tolerance level.

Fourth, make the survey easy to take. Lots of extraneous verbiage increases the respondent burden, which is the work we ask the respondent to do. Checklist questions with lots of options with detailed explanations may generate precise data, but it won’t generate any valid data if no one completes it. If you need that precision, consider conducting interviews. “Think outside the survey box” when building your customer feedback program.

Fifth, consider offering an incentive. I’m not a big fan of incentives since they can promote what I call “monkey typing.” No, I don’t mean “Survey Monkey typing,” but rather, people clicking on any response choice just to get the prize. How valid are the data in that circumstance? It’s garbage. In fact, I have counseled clients to reduce the size of the incentive they were planning to offer. It’s supposed to be a token of appreciation, not payment.

Lastly, incent them to respond by sharing a summary of your findings. Show your respondents that you really mean it when you say their opinion is important. Close the loop and tell them what you’ve learned and what action you’re taking. That of course means that you actually take action. It’s such a simple, powerful idea. Yet, so very few organizations close the feedback loop with their customers.

While we can’t control all the survey noise and overgrazing, we can control our piece of the survey common. Basically, practice the Golden Rule of Surveying. Survey unto others as you would have them survey unto you.

Summary: Many survey project managers think that a survey project ends with the analysis, presentation of findings, and suggested courses of action. Companies that follow such practice will likely find their survey response rates falling over time as customers — or other group of interest — feel less engaged.  Two vital last steps in a survey project are: 1) take action on the findings and 2) communicate the findings and intended actions back to the respondent group. These two steps show the feedback program is for real, and they will help drive respondent engagement with future feedback requests.

You’ve written a beautiful questionnaire that meets your research objectives. You’ve sent it to your respondent group, getting a great response rate. The data have been analyzed, report written, and presentation made to management. You check “completed” on your job plan. You’re done, right? Wrong. Two more steps are vital to receiving full value from your survey project today and from your feedback program tomorrow.

• Take action on your findings
• Communicate your findings and your actions taken back to your respondent group.

To the first item, you may be — hopefully — thinking, “of course!” But to the second item, you may well be thinking, “Ohhhhh… Really???” Both are vital to getting full value from an ongoing survey program. Let’s look at each.

Take Action. This may seem so obvious you can’t believe I would bother mentioning it, but you would be surprised at the number of companies that do surveys “just because” or because the results are “nice to know.” I have actually done survey projects for clients where I know my spiral bound report is going on someone’s bookshelf (or “filed” elsewhere) and nothing will be done with the findings. I may get paid, but it’s not professionally fulfilling.

While I don’t think you would ever find this to be a corporate policy, some people want to gather data to support their position on an issue, and if they don’t get the “right” results, they just forget about the research. Others do surveys without any clear intent of taking action on the findings. Still others intend to take action but don’t design the questionnaire to generate actionable data.

This lack of action is less likely to occur where the research objectives have been well thought through, leading to a properly designed questionnaire. It is also less likely to occur where a company has a survey program office — or some such title — with control over customer contact data. One of my survey workshop attendees ran such an office. If someone came to her wanting to conduct a survey, her first question was, “What are you going to do with the data?” Without a good answer, you didn’t get access to customer contact data. She wasn’t going to let anyone take up customers’ time because something was “nice to know.” You needed to specify actions that would be taken from the findings. I suspect she’s not the most popular person in her company, but that gatekeeping action is needed to force people to think through their customer research programs.

Communicate Your Findings. I suspect this suggestion gave you cause to pause. Certainly you intend to communicate your survey’s findings to your management team and others to whom the findings would be relevant.

But why would you communicate the findings back to the respondent group? The answer can be found in your survey invitation. You likely told the person their opinion was of vital importance and you intended to use the data to improve the products and services you provide them. Great. Prove it! I know I have received such survey invitations from companies, and then I wonder what they’ve done with my feedback. All too often I am left having no idea if any responses were even tabulated.

If you are engaged in an ongoing feedback program from customers, suppliers, employees or some other group with whom you have an ongoing relationship, you are likely to go back to these same people repeatedly to get feedback. Why would they complete your next survey if they don’t see any tangible benefit from the last survey they completed? It’s your responsibility to show them how they will benefit from completing the survey beyond the vague sales pitch in the survey invitation. You promised them benefits; now show the benefits that have accrued from their feedback.

The point of the communication is to show respondents that you really did do something with the input from the survey. An ongoing survey program needs to be marketed, and communicating the findings is a critical marketing element.

Do be clear on one point. I am not suggesting that you should try to influence the responses provided by customers. You want honest, candid, forthright responses. Rather, I am suggesting you should try to influence their willingness to provide a response. Most companies consider offering an incentive to juice responses, but that can be a two-edged sword as I’ll explore in a future article. Communicating findings creates an incentive that costs little, and it supports the broader goals of the ongoing feedback program. It shows that as a company you’re serious about building long-term relationships rather than just engaging in sloganeering.

What would you communicate? Clearly, you are not going to reveal any proprietary information or personnel decisions derived from the survey. But you can certainly reveal the findings of the survey at a high level and say that you’re implementing steps to correct the problems identified and that you’ll be maintaining the practices that respondents liked.

How can you communicate the findings? My assumption in this article is that you are capturing feedback from a group with whom you have ongoing contact, be it customers, suppliers, members, or employees, and not an arms-length consumer-research project. Given that, how do you normally communicate with this group? Possible vehicles are newsletters, account manager visits, an article on the intranet, and user group meetings. Be pragmatic. Use these vehicles to communicate your feedback findings and actions. Newsletter editors are always looking for articles.

Hopefully, I have convinced you to communicate summarize findings and action plans to your respondent group. It’s all part of closing the feedback loop. You still have one more small task. Include the fact that you will communicate findings in your survey invitation!

One category of communication I consider absolutely not optional: responding to specific problems customers reveal in a survey.

Summary: Survey projects are frequently treated without the due diligence required to ensure success. They seem deceptively simple. Why the need to plan? That, as any project manager knows, is a recipe for failure. Some level of resource commitment is needed from an organization to provide a firm foundation for a successful survey project. This article examines those requirements from both monetary and personnel resource perspectives.

In the decade or more that I’ve been doing survey work I have one observation that I can’t quite explain. Company budgets for customer feedback programs are widely bifurcated. Many companies will write good-size, six-figure (in US dollars) checks to run a comprehensive, customer feedback office. These may be totally outsourced, or done entirely with internal resources, or a combination.

Then there are the companies that will barely spend a dime. I wish that were as big an exaggeration as it sounds, but it’s not. I find there’s little middle ground.  Companies either:

• “Get it” and are willing to invest in listening to the customer
• “Get it” but view it as an expense with a dubious return so they spend little, or
• “Don’t get it” but may go through the motions of a poorly funded program.

Far be it for me to say it’s a waste of money to hire one of the large, well-known professional firms to run a feedback program – though you may also be paying a lot for their “name.” They do a very good job, employ sound methodologies — maybe with a few exceptions — deliver a lot of analysis and information with considerable hand holding. (The exceptions I have seen are survey programs designed by market research firms, not the dedicated survey firms, that don’t have solid background in survey design and apparently think there’s nothing to it. I have written some articles about these types of surveys I have encountered.)

On the other extreme, many years ago I got a call from a gentleman in Florida who was put in charge of doing a customer satisfaction survey for his company. He asked me a bunch of questions, and it was readily apparent that he had no background in conducting surveys. In fact, he was very open about that. I gave him a lot of advice, and, yes, I did suggest to him to attend one of the workshop. Sorry, no budget for that. So, I suggested my book. Sorry, no budget for even a book! About a year later I was speaking at a conference in Florida, so I wrote to this person just on the chance he might be able to attend the conference. I was relieved to learn that the survey project had been axed. Given the level of commitment that management was showing, the project would have failed.

So, what budget do you need to run a decent survey project if you’re doing this internally?  Let me break the budget into personnel and dollars.

From a personnel standpoint survey programs have two key players. First, a project leader is needed. This person will likely be the person who drafts the survey instruments and thus needs a solid background in survey questionnaire design as well as knowledge of the range of tasks in a survey project. Running the survey program may be a full-time job for the project leader, but that depends on the size of the company and the scope of the program. What is absolutely necessary is that the survey project management role be part of the person’s job plan. I’ve encountered service managers who are told to do a survey “in their spare time.” Yea, right. Who has spare time in their work life, and why would you spend time doing something on which you’re not measured or rewarded?

Perhaps the more important player is the sponsor or champion. In my tale above about the Florida gentleman, it was clear that no one was championing the project. The sponsor needs to argue for the monetary budget, fight the political battles, and make sure that any people assigned to the project team don’t get pulled off for some other project.

The aforementioned project team is the third category of personnel needed on any survey project, especially if the survey program is to be executed primarily or exclusively with internal resources. A survey project has a number of tasks to be executed, more than one person can do handily. Most important, the project team needs to be the review and refinement body for the survey instrument that the project leader is likely writing. The worst survey instruments are those written by one person with no reviewers.

For the monetary side of the budgeting process, the most important and perhaps largest piece is the headcount dollars for the project leader and project team. This could be one half of a full-time equivalent (FTE) to several FTEs for a highly involved, customer feedback program done internally. But beyond headcount there are going to be out-of-pocket expenses. Let me address those through the time line of a survey project.

During the questionnaire design phase of a survey project, most expenses will be minor. If you’re running focus groups as part of an exploratory research project to build a better instrument, you will likely need to rent meeting room space and provide refreshments and some token gift to participants. You may also want to provide some token gift to participants in the pilot testing phase, and you may pay someone to transcribe recorded notes. All of these are minor, especially compared to the personnel costs. (Of course, we’d recommend you attend our survey workshop series to get a solid grounding in survey methodology.)

The survey administration stage will incur most of the out-of-pocket cost of a survey project since that’s where the logistical tasks lie. If you’re using web-form administration, you will need to buy software or rent one of the hosted survey tools. This cost could run from hundreds of dollars to tens of thousands of dollars. A broad range of capabilities exists in these tools, depending upon your requirements. Many companies start out with one of the inexpensive hosted tools, such as SurveyMonkey, QuestionPro, or Zoomerang. (The list of survey tools is very extensive.) These tools will do the job, but as your requirements or sophistication increase, you may well outgrow them. But as with most any product, you don’t know what you don’t know until you use them. Using one of these tools will make you a more informed purchaser of the next generation survey tool. Again, most of the cost here will be personnel-oriented. As with most software applications, the cost of learning to use the survey tool will be far greater than the cost of buying or renting the tool.

Phone surveys done internally on a small scale probably encounter little to no out-of-pocket expenses, but when you get to a certain scale, you need to invest in a software application to manage the surveying process.  Postal mail surveys entail considerable expense due to the mailing logistics.  Even if you use internal resources to stuff envelopes and key in the data from the returned forms, you still have costs for envelopes, paper, copying, and postage both outbound and inbound – you must provide return postage.

The data analysis and reporting stage will again have very low out-of-pocket expenses. The survey tool you use will likely have an analytical package, and your company likely has a statistical package that you can use. Excel will work fine for most survey analysis purposes. Personnel time is again the major cost here. And don’t forget that included in this stage is any service recovery program where you respond to customers who have voiced serious issues with your products or services along with implementing the findings of the survey program.

If you are budget constrained, you might be asking why you would consider outsourcing a survey project. As with any project, the two main reasons to outsource are:

• Capacity. Do you really have the bandwidth in your staff to do the job? Survey programs look simple on their face, but they are more involved then most people realize — until they are knee-deep into the administration process.
• Competency. The devil’s in the details.  Do you have the knowledge of surveying methodology to know all the details that need to be considered to correctly design and execute a survey project?  If you don’t, then you might compromise the whole value sought from the program.  Bad, invalid data is worse than no data at all.

Survey projects add a third reason for outsourcing:

• Credibility and Confidentiality. Your customers may not tell you things that you really need to know, but they may be willing to tell a third party “the bad news.” Because respondents feel more assured about the confidentiality of their responses, the findings of the program may be more credible. The most obvious example of these factors is personnel surveys, which almost all companies outsource.

It’s your call about whether to outsource your survey project or do it internally, but make sure the decision is made knowledgeably. A company can run its own survey program and do an admirable job, but don’t be penny-wise and pound foolish. You don’t need to spend hundreds of thousands of dollars, but you do need to establish a budget that properly recognizes the resources needed to do a credible job of generating valid feedback from your customers or your employees.

You would probably think that mistakes in survey design would be made by small companies with limited resources and knowledge. Yet, some of the best examples of bad survey design practice I find in big companies. In this article, I’ll illustrate the mistakes made in the IVR  survey to measure Sears customer satisfaction. (Home Depot’s store visit survey is another glaring example of bad survey design.) Read my related article regarding to learn the strengths and weaknesses of IVR survey administration method.

Note: If you have landed on this page because you ran a search on “Sears Survey,” please note that you are NOT on the Sears website. We get people contacting us thinking we’re Sears looking for all kinds of help. We’re not unsympathetic, but please contact Sears. By the way, it appears that entering the Sears survey gets you on all kinds of mailing lists. Beware.

In a nutshell, the Sears IVR customer satisfaction survey:

• Didn’t allow me, the customer, to fully express my feelings about my entire transaction with Sears
• Forced me to answer irrelevant (to me) survey questions
• Force fit a suvey question to a scale to the point where the data generated are not interpretable (that is, they generate garbage data)
• Asked me for comments — and then ignored me.
• Most importantly, lead to greater customer dissatisfaction and destroyed any customer loyalty I had to Sears

The last bullet should startle you. Customer surveys are typically meant to provide customer measurements to increase customer satisfaction. So, how could a survey program design actually lead to greater customer dissatisfaction? The answer lies in a poorly conceived survey project and program design that leads the customer to come away with a more negative attitude toward the company in addition to generating survey data of questionable value.

Before I explain all of these survey design problems in detail and how to avoid them — hint: the answer lies in having a good customer survey program design — some background on my interaction with Sears customer service is needed.

My Sears Service Interaction

I bought a hot water heater in 1997 from Sears. It had a lifetime warranty. In September of 2007 the heater failed; it cracked. My transaction with Sears evolved into the following series of interactions.

1. Claim Initialization. I called the Sears call center at 800-4MyHome to inquire how I would invoke my warranty protection for the failed hot water heater. This interaction was great. They actually had my purchase on record, so I didn’t even need my receipt, which I did have. They arranged for a repair tech to come to my house two days later to verify whether the tank failure was covered under warranty. I wasn’t thrilled with the 2-day wait, but I had backup hot water capabilities. The agent told me that should the technician verify that the failure was covered by my warranty, then I would not be levied the $69 service charge for the technician visit. The explanation was poorly worded, so I restated what I thought the agent said to get verification.
2. Technician Verification of My Claim. The technician came to my house, verified my claim, gave me the paperwork to get a new hot water heater at my local Sears store — and charged me the $69 service charge. Foolishly, I paid it (with my credit card). I figured getting my refund from a company of Sears’ stature would be no problem. Boy was I wrong.
3. Fulfillment of my claim. The field tech gave me a form to take to my local store to get my replacement heater. My day was shot so I went to the Marlboro Sears store to get my new heater, but they told me the paperwork would take 24 hours to get into the IT system. Why didn’t the technician tell me this? I was told I would get a call the next day. I didn’t. I had to call them. I did eventually get my new heater, got it installed, etc.
4. Resolving the Improper Charge. When questioning the charge with the field tech at my house, he told me to call the local office in Danvers, Mass. rather than call the service center. He said I could talk to a “customer relationship specialist” there. The person with whom I spoke was the most useless, aggravating person I have ever engaged. He cut me off in mid sentence, refused to let me explain the situation and basically told me whatever the tech said is right. He was completely unconcerned about the discrepancy between the service center and the tech.  He was so exasperating, I hung up on him. If he is a “customer relationship specialist,” then Sears is in big trouble.I called 1-800-4MyHome again and verified that I should not have been charged. They told me to get my refund by working through my local Sears store. I tried unsuccessfully for months. I wrote to the Sears headquarters about this likely illegal practice in the hopes I could avoid going through my state’s consumer affairs division.A month after my letter was delivered I got a call from a troubleshooter at Sears headquarters. She told me the charge was legitimate. The service agents at 800-4MyHome gave me wrong information and Sears is not responsible if its agents give a customer wrong information that is the basis for establishing a contractual arrangement between Sears and the customer. Tough luck. End of story. I was flabbergasted. My jaw dropped.

Soon after the technician’s visit, I received an automated phone message asking me to take an IVR survey. I should have transcribed the survey, but I can do it good justice. The 6 or 7 question survey — it was one question longer than stated in the introduction — posed all its questions on a satisfaction scale, where 1 represented Very Dissatisfied and 5 represented Very Satisfied. All of the questions pertained to the technician’s visit to my home, which, I’ll argue, was a glaring mistake. But the biggest mistake was what Sears did with my survey data.

What Lessons Can We Learn from the Sears IVR Survey?

Keep the Survey Design Focused. I teach in my Survey Design Workshop to keep a tight focus on the survey contents and not let other departments meddle their way into the survey design process. This I believe happened at Sears. The first 6 questions were about the technician’s performance, but the last question, asked about my “satisfaction with the technician making me aware of Sears’ products and services.” That’s not something about which I give two hoots, and I suspect the operational folks in Sears service don’t either. But the marketing folks do. Clearly, the technician is tasked to promote Sears products, which is one of those strange — and inappropriate roles — for service technicians. Survey focus is achieved by having a good Statement of Purpose or Research Objectives — and sticking to it.

Word Questions to the Scale. The question mentioned immediately above also displays a classic problem in survey question design: wording a question to fit the scale. Look at the question. It was posed on a 1 to 5 satisfaction scale. I had no idea how to respond. The technician didn’t make me aware of any Sears products. I didn’t know he was supposed to, and I was glad he didn’t. So, does that mean I should have scored it a 5 indicating Very Satisfied with the technician, but not because he nicely made me aware of Sears products but because he didn’t waste my time with a sales pitch that I would at best find annoying? Or should I score it a 1 indicating Very Dissatisfied because… well, I don’t know why I would. But I am sure the marketing folks who pushed for this question would want me to score it a 1.  I didn’t. I did the cop-out, middle-of-the-road, 3. Regardless, the marketing folks at Sears are interpreting the results from this survey question that is simply generating garbage data.

Develop Comprehensive Response Options. That last question really needed a Not Applicable option. I tried just not answering, but the system would not let me do that. So, as said, I entered a 3. I had a sense I would soon get the opportunity to answer an open-ended question, so I had to get beyond this question.

Develop Comprehensive Question Set. While we want to keep any survey, but especially an IVR transactional survey, short and sweet, you still need to pose all the needed questions. Usually, my criticism of surveys is that they go too far a field. The Sears survey was too narrow. The technician gave me factually incomplete information about how to get my replacement heater, and I was charged improperly. Those critical flaws in the transaction went unaddressed in the survey questions. Trust me; those factors have made the deep impressions on me about Sears on-site service.

Think Beyond the Interaction to the Transaction. A service transaction is a chain of interactions, and the weak link needs to be identified. The Sears survey only asked me about my experience with the technician. My transaction with Sears went well beyond the technician’s visit, including my interaction with the service center and the store. No one ever asked me about those interactions, and thus did not allow me to express my full feelings about my experience. (You’re probably thinking, “but wasn’t there an open-ended comment question?” Hold that thought!) Yet, I’m sure those who designed this awful survey think it’s great. I, as the customer, came away very frustrated since I had had a poor experience and did not get to say my piece.

Notice how this issue relates to the need for a good Statement of Purpose. While the typical customer (survey respondent) won’t think in these terms, they will know if a survey is comprehensive and well designed. The Sears IVR survey wasn’t. Why? My guess is that Sears has very stovepipe functional groups that do not interact.

Need for a Service Recovery Program. The above mistakes are bad but not egregious. The most disastrous error in the Sears survey is what they did NOT do with my data. The last question presented the opportunity to make a free-form comment about my experience. I was waiting for that opportunity.  I even developed notes to be sure I covered everything. I listed the litany of mistakes that Sears made. You would think that the person who analyzes that textual data would flag me for a follow-up call. A well-run company with a well-designed survey program would have done that. Sears did not. They ignored my cry for help, making me angrier than I already was. Yes, the survey — designed to measure customer satisfaction and identify needed operational improvements — actually made the situation worse!

Need for an Integrated Feedback Management Strategy. Even if someone had read and responded to my comments, it might not have mattered. I recounted in the background on my Sears transaction that Sears headquarters basically told me to get lost. Sears sees no ethical or legal obligation to honor the verbal contract established by its service agents. This is a pretty startling indictment of the strategic role of customer service at Sears. But why does Sears go to the bother and expense of having this one feedback survey that sits like an isolated island disconnected from its potential business value?

Will I ever shop at Sears again? You know the answer. That $69 savings is quite the Pyrrhic victory for Sears.

Addendum #1: Missed Value in a Service Recovery Act. After the headquarters’ troubleshooter told me to get lost, I started looking at my legal options, including initiating a grievance through my state’s attorney general office. Then, 6 weeks after phone call with Sears, a check for $69 shows up from Sears. No accompanying letter. No accompanying phone call.

Part of the purpose of a service recovery act is to win back the hearts and minds of the customer. The check partially won back my mind, but my heart? I’m still seething at the process and how I was treated. Compare this to how Subaru handled its service recovery of a flawed service recovery. Sears missed an opportunity here to win me back.

Addendum#2: Some People at Sears Do Take Ownership. Some Sears headquarters’ folks came across this article and one bought my Customer Survey Guidebook. And in Fall 2009, this person and a colleague came to my Voice of the Customer conference. We had some nice and interesting chats. I give them credit for not blaming the messenger. I actually have shopped in a Sears store with a little more confidence that customer service can be a trait at Sears.

~ ~ ~

Here’s some of the emails we’ve received from people thinking we’re Sears… unedited but anonymous.

• sears tool #9 43664 rachet wrench reversable angle 13mmx14mm, “NO ONE” can find this tool!!.. please call
• hello my name is XXX  we have been along time customer of sears but these days i really wonder if you really do think about taking care of your customers and i dont mean just a single store i have been in contact with the store  and every single phone number i can possibly find on or about sears  you people just seem to want to give lip service and nothing else i guess when sears changed owners you really have  a problem helping customers with their problems or you just run a big scam how your customers are really number one well im here to tell you the word of mouth is still a time tested and very effective way of comunicating even in this day of age  and believe me i know a lot of people  common and business men and women alike an old man once told a car dealer rather large that did  a friend of his  wrong  he said to them if you do people right  i will tell everyone i know and if you do people wrong i will tell everyone i know and he did the business went bank rupt after on
• I was in the sdtore shopping and on my slip is says to go to your web site and enter for a 4,000 sears gift card, i did not liket he site all it did was ask me question about college and buyin other things, this is faluse addvertising. now all these people are calling me. what about the gift cert you said about, never did get any information on it. I am a sears customer  not all this other junk.
• Today, I called the number for Sears repair and service in my local phone book. The automatic answering machine did NOT give me any option to speak with the people at the service center in Melrose Park, where I left my vacuum cleaner for repairs a week ago. And there is no contact number on the receipt I received when I dropped off the appliance. So can you help me reach this repair center???? The woman to whom I was transferred merely hung up on me when I told her what I need.