An Honest Survey Invitation?

Summary: A survey invitation may make the first impression of a survey program for those in your respondent pool. A good impression is critical to getting a good survey response rate, but the invitation may present other critical information to the potential respondent. Most importantly, the invitation should be honest. The elements of a good survey invitation are presented in this article in the context of reviewing a poor invitation.

~ ~ ~

Sometimes surveys just start off wrong, that is, with a misalignment between the survey invitation and the survey instrument itself. Usually this occurs due to sloppiness; the survey designer didn’t work the details. Maybe the survey instrument was revised after the introduction had been written. However, the misalignment may also be intentional in order to persuade the invitee to take the survey. I’ll present the misalignment with a real example.

Why should the invitation align with the survey instrument? Well, because it’s an invitation. (d’oh!) The primary purposes behind the invitation are to:

• Entice the recipient of the invitation to move along through the process and actually take the survey. In that sense it is a marketing document for the survey program. As the saying goes, you never get a second chance to make a good first impression. Later I’ll list the points that should be included in an invitation.
• Set the “mental frame” of the respondent. We tell the respondent, “This survey is about…” to get them thinking about the topical area.

What if the invitation doesn’t align? A person may incorrectly take the survey, or a recipient’s time may be wasted once he realizes that the survey is irrelevant – and may be turned off to any future invitations.

What brought this topic to mind was a survey invitation I got from HomeAway.com. My wife and I own a waterfront rental property in the state of Maine, and we have advertised it through HomeAway for slightly less than a year. HomeAway is one of the leading sites for rental home listings, and the parent company has bought up several other sites recently to expand its reach beyond the US to worldwide.

Above is the email invitation I received. Let’s analyze it. In the process, I’ll touch on the survey itself.

Right off, note the date. Who is their right mind launches a survey on December 28? We want to launch a survey when we’re most likely to get some of the respondent’s mindshare. I guess one could argue that the week between Christmas and New Years is a slow week, so people are more likely to see the invitation and have time to take the survey. However, a significant percentage of people are on holiday that week and will be doing only the minimal email checking. This is a business-to-business survey invitation, so I feel it should be launched when business is active.

In fact, in the US the whole time from mid-November (prior to our Thanksgiving holiday) to mid-January is a time when it’s tough to get people’s mindshare. I always recommend to clients to avoid this time period for launching surveys other than ongoing transactional surveys. Why launch a survey with an immediate handicap of a lower response rate?

We may also be introducing some type of a sample bias launching a survey in this time frame. A sample bias occurs when something about our survey administration could lead to some members of our target population being less likely to respond to the survey invitation. This bias could mean that our statistical results are misleading even though we have enough data points for reasonable accuracy.

Now let’s look at the wording of the invitation.

We would like to get your feedback about HomeAway.com in order to improve the value that we provide to you and other property owners… The survey should take about 20 minutes of your time and you will be entered into a drawing to win one of five $100 Amazon.com gift certificates if you qualify.

They identify the group doing the survey and end with

This survey is for research purposes only and is not a sales or marketing survey. That you very much for your feedback.

As a survey designer, I was impressed, though some critical elements are missing. They provided several good “hooks”. I benefit if their site is better, and I might win a $100 raffle. However, my guard did go up when I read the “if you qualify” phrase. After giving the survey some gravitas by indicating that they have contracted a research company to do the survey, they make assurances that the survey is not being used as a ruse for a sales pitch. This struck me positively.

Then I link to the survey. Each of the opening screens posed demographic questions:

• How long have I owned the vacation property?
• How long have I rented the vacation property?
• Who manages the property — the owner or a property manager?
• Who’s involved in marketing decisions?
• How do I market the property? They provided a checklist of marketing methods.
• Which of these online rental sites am I familiar with? They provided a checklist of rental sites.

As I went through each screen of probing demographic questions I became more and more suspicious — and ticked off. I answered “none” to that last checklist question even though I had heard of a few of them. The next screen said:

Those are all of the questions we have for you. Thank you for your participation!

Wow! Talk about a let-down and being left with a feeling of being unimportant!!

Let’s examine the contradiction between the invitation and the survey instrument. But first, when I take a survey I try to turn off my left-brain analytical side and turn on my right-brain gut-reaction side. I try to “experience” the survey before looking at it analytically. After all, this is how the typical respondent will come to the survey process.

First, as a rule, demographic questions should go at the end of the survey instrument. Why? Demographic questions are not engaging; they are off-putting. After getting me excited about the opportunity to “provide feedback,” I got hit with a bunch of questions that didn’t excite me at all — just the opposite.

Second, they never said why they needed all these demographic questions answered. Some explanation should always be provided with demographic questions to help allay the concerns with the personal questions. Sometimes we do need to pose one (or two) demographic questions at the beginning of a survey to qualify the respondent or to branch the respondent to the set of questions that are appropriate. Reichheld’s Net Promoter Score® methodology does this in fact, posing different questions to promoters than detractors.

However, if it’s not an anonymous survey, which this wasn’t, then they  should have most all the demographic data in their files to “pre-qualify” a respondent. Apparently, they don’t. Or… this wasn’t really a feedback survey. (More to that point in a minute.)

This gets to my third issue. Note that the invitation contains no assurance of confidentiality or anonymity with the information I will provide. I knew that the survey was not anonymous because of the URLs, but the typical invitee may not know this.

Clearly, the purpose of this battery of demographic questions was to qualify me. But did I qualify? They never told me! That’s my fourth point in the shortcomings of this survey design. I am a customer of HomeAway.  Don’t they owe me the professional courtesy — or common decency — to tell me if I “qualified”?

Instead, I got, “Those are all of the questions we have for you. Thank you for your participation!” While that may be an honest statement, it’s a blatant half truth. Do you really want to leave a customer with the feeling of being unimportant? That’s what this survey design did. A survey can be — and should be — a bonding opportunity with a customer, not an opportunity to weaken the bond.

Fifth, the barrage of demographic questions activated a response bias on my part. Response bias is the bias that the respondent brings to the process brought out by the questionnaire or administration procedure. It leads to untrue answers from the respondent.

Many types of response bias exist.  Here it’s what I call concern for privacy.  The number of questions about my business practices made me leery of the surveyor’s motives, combined with no promise of confidentiality.

Remember, they promised me in the invitation that the survey was not for sales or marketing purposes, yet look at the questions they asked. My guess is that had I qualified for the survey, I would have been asked to compare HomeAway to other home rental sites. We can have a long discussion on the nuanced difference between a market research survey and a marketing survey, but one thing I know for certain. This was NOT a feedback survey. A customer should not have to “qualify” to provide feedback.

That’s my sixth — and most important — point. The invitation was not truthful. We want the respondent to be honest, forthright, and candid with us. Shouldn’t we demonstrate those same principles to the respondent? Will I ever waste my time taking another survey from HomeAway? Would you?

This isn’t the only flaw in HomeAway’s survey “program.” About a week or two after adding HomeAway to my advertising program, I got a survey invitation. I was impressed. I thought the survey was part of an onboarding process that would ask my experiences as a new customer. (Constant Contact does a wonderful job of onboarding.)

Alas, it did not. I am sure the survey was sent to all others who advertised properties on their site. Since I was new, most of the questions were just plain irrelevant to me at that point. Worse, having just set up my site, I was loaded with constructive feedback — positive and negative — that could have improved the site. Their loss is my survey-article gain.

What should be addressed in the invitation?

• Benefit statement to the respondent. Why should the respondent give you their time? This is critical.
• The purpose of the survey. This helps set the respondent’s mental state.
• Who should be taking the survey.
• An estimate of the time to take the survey. It should be a real estimate, not a low-ball lie. I do not state the number of questions, unless it is quite low. Question counts are intimidating.
• Some statement about the anonymity — or lack thereof — for the person taking the survey along with a promise of the confidential handling of the information provided. This is especially important if using a third party for the survey process. If one is conducting “human factors research” for medical purposes, by law in the US all this must be disclosed. We shouldn’t need laws for this, and it should be part of all survey research.
• Who is conducting the survey? If you are using a third-party, the invitation should come from your organization’s email system. If the invitation is going to come from the research agency or through a survey tool’s mail system, then you need to send an email prior to the invitation to validate the third party — and ask that the invitee set the necessary permissions for the mail from the third party to get through email filters.
• Offer of an incentive if you choose to do so. If it’s a raffle, you should help show that the raffle is real by providing a link to a web page where the names of people who have won previous raffles are listed. Protect their privacy by listing just their name and their town. I checked the HomeAway site and found no such page. Perhaps I missed it. B&H Photo does a nice job of this as did United Airlines.
• Contact information for sometime who can clarify questions with the survey.
• An opt-out option for future surveys. The footer of HomeAway’s invitation has an Unsubscribe option, but is that for all of HomeAway’s emails or just for their survey invitations? Since I am a customer, I do want to receive relevant emails.

The real challenge is to cover these points succinctly. Since most of us view email using the preview pane, you want the “hooks” to be visible in the preview pane. Don’t fill the preview pane with the logo that marketing tells you that you must use to help brand the organization.

Writing a good invitation isn’t rocket science. It’s a combination of common sense and common decency with some marketing flair thrown in for good measure. But don’t let the marketing people use flair to cover the truth.