Summary: Proper survey question wording is essential to generate valid, meaningful data for organizational decisions.  A survey question working bias will lead to misleading interpretations and bad decisions.

Good question writing is an art form, and even the pros can make mistakes.  Here we’ll show a question wording bias example from a survey done by Pew Research Center. Ambiguity in question wording likely led to incorrect data and conclusions.  It provides some useful lessons for all survey designers.

~ ~ ~

Many people view how to write a good survey question as a trivial issue.  But it’s really an art form that requires a good nose for sniffing out ambiguities that could lead to collecting misleading data and bad decision making.

I saw an example recently at an IDC conference where the presenter included survey data from a Pew research study on the etiquette of using mobile devices.  Pew is a highly respected research organization. I’ve looked at their surveys and have been impressed by their neutral, professional approach to their research initiatives. But even pros like them (or me!) make mistakes.

Here’s one.

Social Acceptance Mobile Devices

See the slide nearby that piqued my interest, along with the wording from the survey instrument, provided by Pew. The survey question (PH8) was: “Do you think it is generally okay or not okay for people to use their cellphones in the following situations?”

Mobile etiquette is a topic in our culture, and even good for comedy skits, but here the study examined people’s views for various social situations.

The slide showed a breakdown by age group. For many of the scenarios there was little difference across age groups, but for three scenarios the differences were pretty stark:

  • At a restaurant
  • While waiting in line
  • On public transportation

The younger you are, the more you felt that “using your cellphone” was okay.  The older you are, the less you thought it was okay.

I immediately thought, “Why would older people – like me! – be more likely to think it’s wrong to use a phone while in queue? What’s going on here?”

Social Acceptability of Mobile Devices, Pew

And here’s where we find lessons for survey question wording

1) Challenge Your Findings

Whenever you find answers to survey questions that seem odd —what researchers call conclusion validity – challenge them! 

Why would someone in my age cohort agree with millennials about using a cellphone at a family dinner, but disagree strongly about using a phone while in queue or on public transportation?  I personally don’t care if someone is checking email while on the subway or at a Subway.

In fact, Impatient Fred does that.

Are we seeing:

  • A true difference in how age groups view societal norms? Or…
  • A difference in how the age groups interpreted the question?

Hmmm….

2) Look for Ambiguity that Could Compromise Validity

My guess – and it is but an educated guess – is that the question was interpreted differently by those in various age cohorts.  It was ambiguous.

The key phrase in the question – what academics call the “construct of interest” – is “use their cellphones.”

  • What does that mean?
  • What’s going through the respondents’ brains when they read that phrase? 
Tips for a Successful Survey

Request Chapter 1 of the new edition of our Survey Guidebook for key points in a more effective survey program.

Fill out my online form.

Unfortunately, the survey didn’t ask what functions of the “cellphone” people use, though they did ask in an earlier question the purposes for which the “cellphone” was used.

You’ll note that I put “cellphone” in quotes.  And that’s gets to the point.  To phrase it as a riddle: When isn’t a phone a phone?  When it’s in the hands of a millennial.

Here’s my guess about the interpretation of “use their cellphone”:

  • Millennials are more likely to think about texting or use of some app, such as Twitter or Facebook.  ⇒ Quiet typing activities.
  • Boomers and Gen Xers are more likely to think about using the phone to have a spoken conversation.  ⇒ Orally loud activities. 

    (Goodness, haven’t we all encountered people in airports who yell into their phones? And then give us a hairy eyeball because we’re  “listening” to their conversation?)

In other words, “use their cellphones” is probably ambiguous.  People are likely to interpret the phrase differently, and the different interpretations are likely associated with age.  If so, these different interpretations make the data invalid; the data don’t mean what they purport to mean.

If you look at the actual survey, you’ll see that the question was asked of people regardless of the type of cellphone they own – basic phone vs so-called smartphone. How different would the results have been if the basic phone owners had been eliminated?

Further, what if the nature of the phone use had been presented?  For example, here are two phrases that would have eliminated the ambiguity:

  • “use their cellphones to text, check email or browse” versus
  • “use their cellphone to call someone.”

My guess is that the difference across age groups would have been dramatically reduced.

Here’s the goal of good survey question writing:

Every respondent should have the identical interpretation of a question.  If multiple interpretations exist, then you’re asking different questions of different people and the findings can’t be interpreted.

That defines question validity.  Now the truth is that achieving 100% validity is near impossible.  Someone for some reason will have some bizarre interpretation of even the most clearly written question.  But this measurement error should be a 1% problem, not a 10% or 20% problem.

Survey Sample Size Calculator

Get our Excel-based calculator. It can also be used to gauge statistical accuracy after the survey has been completed.

Fill out my online form.

3) Test for Ambiguity

I always say in my Survey Design Workshops that the worst survey questionnaires are designed by one person in one day.  I doubt that’s what happened here, but this example shows the need to challenge the phrasing of every question and to get a project team to review the question wording.  Ask yourself, “Are there other ways that people could interpret the question?”

This example also highlights the value of pilot testing a survey before going live. A good pilot tester would have asked the test subjects, “Why do you feel it’s not okay for someone to use their phone while in queue.” They likely would have heard that being subjected to other people’s phone conversations in public is rude. Then, the problems with that survey question wording may have become evident.

And could we please stop using the poorly descriptive term “cellphones” or “smartphones”?  They’re mobile computing devices.