Summary: Each of the five survey question types has potential value in a survey research project. This article presents brief definitions of each question type, the analysis you can do, and some key concerns with each question type.
~ ~ ~
Survey question type determines the data type that is generated from a survey, and the data type constrains the type of data analysis that can be performed. In another article I presented the five survey question types and gave an example of incorrect data analysis when an ordinal question was assumed to have the analysis capabilities of interval data.
But when should each survey question type be used? What are the critical considerations with each question type?
Multiple choice question format
Also known as checklist questions, multiple choice questions generate categorical (aka, nominal) data that can only be analyzed with frequency distributions. We use these questions when we have a response set we want to present to the respondent and ask them to select the best response (single response) or all that apply (multiple response). Many demographic questions are checklist questions.
From a design standpoint, we want to have a comprehensive list of response options, but we want the options to be distinctly different. An “Other (please specify)” should also be offered, but those entries must be examined for reclassification, otherwise the frequency distributions generated won’t accurately reflect how the respondent group feels.
Ordinal scale question format
As in multiple choice questions, ordinal questions typically have a list of response choices, but what distinguishes the list here is that the response options are ordered in some way. Forced ranking questions, such as the one discussed regarding incorrect data analysis, are prone to respondent error. Resist the temptation to have too many items in any rank ordering question!
Another type of ordinal question is when the respondent is ask to choose the best response from an ordered set of statements, such as when Goldilocks was asked about the temperature of the porridge: too cold, just right, or too hot.We call this a positional scale. Other types of ordinal questions exist, such as the Guttman scale.
Many researchers believe that the typical 1-to-5 or 1-to-10 scalar questions are not truly interval but are in fact ordinal. Why? Because presenting a scale with truly equal intervals is a tall order, and many respondents don’t view the difference between a 9 and 10 the same as the difference between, say, a 3 and 4.
For example, consider the Net Promoter Score (NPS) 0-to-10 scale. Analytically, 0 to 6 — 7 total scale points! — are considered “detractors,” yet only 9 and 10 — 2 points — are considered “promoters.” Clearly, the intervals aren’t equal, which is why the data are analyzed using the net scoring statistic, which only assumes ordinal properties.
Ratio scale question format
When respondents are asked to respond regarding some physical measure, such as income, years of education, or how long their phone call was on hold, these are ratio scale questions. The data for the items being measured have a true zero. As a result, we can use division and multiplication on these data.
Frequently, we solicit ratio data with what appears to be an ordinal scale with response options presented in ranges, such as if we ask for the number of years as a customer with the following options:
Key Concern for Choice of Survey Question Type and Question Design
Remember the goal of a survey:
- You want to solicit information from the respondent that fulfills your research objectives
- But without creating undue respondent burden.
Respondent burden is the term surveyors used to describe the amount of effort we’re placing on the respondent to complete the questionnaire. Why are we concerned about respondent burden?
The higher the burden:
- The lower the survey response rate is likely to be, lowering statistical accuracy and increasing the likelihood of non-response bias.
- The greater the likelihood of confusion and errors by the respondent.
For example, while fixed sum questions are useful to force respondents to consider trade-offs, they are high in respondent burden. They should be used judiciously and only after you’ve engaged the respondent in the subject matter of the survey. I would never put one of these as the first question in a survey or use more than two of them in a survey!