The Poetry of Surveys: A Respondent’s Survey Design Lessons

in

I never really thought being a survey designer would be a topic for the cocktail hour with friends and strangers. I’d be exaggerating if I said it were, but I am surprised how in first time encounters people want to talk about surveys they have taken. The last stage of a survey project is the pilot test with people from the actual respondent group, and these conversations serve as learning moments on a par with pilot tests, particularly in the area of respondent burden.

I was answering telephones for the pledge drive for my local NPR jazz and folk station, WICN, and I wound up in a conversation with another volunteer. When she learned what I did for a living, she immediately — and passionately — talked about her experiences with a survey from Poetry Magazine.  She is a subscriber and, as a poet, is very passionate about the periodical. “I want them to succeed.”  In particular, she described things she likes and dislikes in surveys. Here are her lessons:

  • One to two screens at maximum. “Three screens and I’m gone.” How many of you have “brief” transactional surveys that go on for five or more screens? Do so at your peril.
  • The survey should be easy to answer. She prefers yes/no questions, but 5-point rating scales are okay for her. Forget the elaborate rating scales with confusing anchors, for example, “somewhat this versus somewhat that…”  That’s a real turn-off for her. (I personally am not a fan of yes/no questions, unless they are truly binary in nature and not scalar, but I get her point.)
  • Don’t force her to write comments or ask for comments with every question. You should be nice in asking for follow-up comments, but beware of asking her for too many comments. You’ll likely get none.
  • Demographic questions should not be intrusive and should be few in number. In particular, the income question is a hot button. “I may go on with the survey, but I’m wary.” In my survey workshops I teach that demographic questions put up a wall beyond the survey designer and the respondent. Here was the vocalization of that.

She communicated these points to the magazine as suggestions on how to change their survey.

Then she told me a story that demonstrates the value of developing a meaningful rapport with your customers — or whoever your respondent group is. One of her Poetry Magazines literally fell apart at the binding. These are not magazines that you just throw away. They are meant to be saved. She wrote to the editor in cute, poetic verse to complain. She quickly got a response from the editor with a “care package”. “I love them even more. The rapid response made me feel fussed over.” This experience just reinforces the power of Service Recovery and the value in encouraging your customers to complain, hopefully as nicely as she did.

As a survey designer, what’s the main lesson here?  Many of those in your respondent group overtly think about your survey design and have strong, cogent feelings about the impact of the survey design upon them. They may not use the term “respondent burden”, but they know it when they experience it.

Listen to your respondent group about your survey. This should be done during the design stage, at minimum during the pilot testing. You may be shocked at how much you learn, which in turn will impact how much valid information you learn from your survey program. To paraphrase Yogi Berra, “You can learn a lot by listening.”