Summary: Each of the five survey question types has potential value in a survey research project. This article presents brief definitions of each question type, the analysis you can do, and some key concerns with each question type.
~ ~ ~
Survey question type determines the data type that is generated from a survey, and the data type constrains the type of data analysis that can be performed. In another article I presented the five survey question types and gave an example of incorrect data analysis when an ordinal question was assumed to have the analysis capabilities of interval data.
But when should each survey question type be used? What are the critical considerations with each question type?
Open-ended text question format
Also known as comments or verbatims, open-ended questions generate a free-form text response. These are best used to generate detailed information needed to understand what led the respondent to feel as they do or to gather information hasn’t been addressed in any closed-ended questions. The text brings life to otherwise dry statistics.
However, they should be used judiciously since they are high in respondent burden, administrative burden, and analytical burden. If you have designed a questionnaire that is mostly open-ended questions, then you need to do some ground work to develop closed-ended questions that follow.
Multiple choice question format
Also known as checklist questions, multiple choice questions generate categorical (aka, nominal) data that can only be analyzed with frequency distributions. We use these questions when we have a response set we want to present to the respondent and ask them to select the best response (single response) or all that apply (multiple response). Many demographic questions are checklist questions.
From a design standpoint, we want to have a comprehensive list of response options, but we want the options to be distinctly different. An “Other (please specify)” should also be offered, but those entries must be examined for reclassification, otherwise the frequency distributions generated won’t accurately reflect how the respondent group feels.
Ordinal scale question format
As in multiple choice questions, ordinal questions typically have a list of response choices, but what distinguishes the list here is that the response options are ordered in some way. Forced ranking questions, such as the one discussed regarding incorrect data analysis, are prone to respondent error. Resist the temptation to have too many items in any rank ordering question!
Another type of ordinal question is when the respondent is ask to choose the best response from an ordered set of statements, such as when Goldilocks was asked about the temperature of the porridge: too cold, just right, or too hot.We call this a positional scale. Other types of ordinal questions exist, such as the Guttman scale.
Ordinal data analysis centers on calculating cumulative frequency distributions, but median and mode “measures of central tendency” can also be calculated.
Interval scale question format
Interval rating scale questions are the most common type of survey question, and we use them to capture the level of feelings the respondent has about the topic of interest. The level of feelings is captured by presenting a multiple point scale to the respondent and asking them where they fall on the scale range.
You have probably seen the following scales used: satisfaction, likelihood, strength of agreement, and frequency. But the survey questionnaire designer can create a scale to match the dimension she wishes to measure, and these can be anchored verbally, numerically, or ideographically.
A critical difference between ordinal and interval scale questions is that an interval scale has a consistent unit of measurement. This quantitative numerical data can be averaged, a key advantage over ordinal data, and many advanced statistical tests can be performed if we assume interval data properties.
Survey Training Classes
Running your own survey programs? Learn how to do it right from us — the premier worldwide trainers for survey design & analysis.
Featured Classes:
Many researchers believe that the typical 1-to-5 or 1-to-10 scalar questions are not truly interval but are in fact ordinal. Why? Because presenting a scale with truly equal intervals is a tall order, and many respondents don’t view the difference between a 9 and 10 the same as the difference between, say, a 3 and 4.
For example, consider the Net Promoter Score (NPS) 0-to-10 scale. Analytically, 0 to 6 — 7 total scale points! — are considered “detractors,” yet only 9 and 10 — 2 points — are considered “promoters.” Clearly, the intervals aren’t equal, which is why the data are analyzed using the net scoring statistic, which only assumes ordinal properties.
Ratio scale question format
When respondents are asked to respond regarding some physical measure, such as income, years of education, or how long their phone call was on hold, these are ratio scale questions. The data for the items being measured have a true zero. As a result, we can use division and multiplication on these data.
Frequently, we solicit ratio data with what appears to be an ordinal scale with response options presented in ranges, such as if we ask for the number of years as a customer with the following options:
- Less than 1 year
- 1 up to 3 years
- 3 up to 5 years
- 5 up to 10 years
- 10 or more years.
While the question looks ordinal — and it is! — the underlying data is truly ratio.
Why present ranges? First, it’s faster for the respondent to answer the question, lowering respondent burden. Second, it’s less invasive to ask someone to check an income range, for example, than to ask them their annual income. Would you tell a stranger your income level? Probably not, but you might be willing to check a box that says your income is $50,000 to $75,000 per year.
Key Concern for Choice of Survey Question Type and Question Design
Remember the goal of a survey:
- You want to solicit information from the respondent that fulfills your research objectives
- But without creating undue respondent burden.
Respondent burden is the term surveyors used to describe the amount of effort we’re placing on the respondent to complete the questionnaire. Why are we concerned about respondent burden?
The higher the burden:
- The lower the survey response rate is likely to be, lowering statistical accuracy and increasing the likelihood of non-response bias.
- The greater the likelihood of confusion and errors by the respondent.
For example, while fixed sum questions are useful to force respondents to consider trade-offs, they are high in respondent burden. They should be used judiciously and only after you’ve engaged the respondent in the subject matter of the survey. I would never put one of these as the first question in a survey or use more than two of them in a survey!
Survey Consulting
We assist clients with their survey projects ranging from a self-help guidebook, to targeted assistance, to full service solutions.
Featured Services:
Customer Experience Management
We evaluate clients’ current feedback processes and drive improvements for customer recovery and bonding.
Featured Services: