Imagine you were male, and an interviewer whose voice (and name) appeared female delivered the above question to you. Wouldn’t you be more likely to acquiesce to the statement? If you were a female respondent, would you want to tell a fellow female that you’re a cultural barbarian?
Unfortunately, the NY Times doesn’t disclose whether they tried to control for that effect. In their explanation of how they conducted the survey, no mention is made of the interviewer gender nor if they looked at splits based upon that gender. That’s too bad since it would be interesting to know the impact.
However, it’s also scary that they don’t recognize the importance of the gender-induced bias and other survey biases — or maybe they do.
~ ~ ~
In summary, as I tell my survey workshop students, some things are very difficult to measure with a survey. It can be a difficult challenge to phrase a question that measures what we want to measure – the definition of validity – not complicated by a swirl of measurement effects.
That’s likely what happened here. But so many surveyors, particularly in big polling organizations, refuse to recognize that their attempts at measurement have failed or take the time to pilot test their language.