CNN published a poll on June 10, 2020 (see nearby data table & graph) that asked whether people now felt “comfortable” returning to their “regular routine”. The data show an increase in comfort level, but the startling finding can be found in the graph.
Comfort Level Returning to Regular Routine
While both (self-described) democrats and republicans show increasing comfort in the past month, the disparity between the two groups is huge – more than 3 to 1.
Comfort Level w/Regular Routine by Party
In a companion article I examined possible reasons for this huge disparity between respondents from each political party:
Difference of Experience
Difference of Perception
But there are yet two other possibilities, :
The sampling methodology was performed incorrectly, leading to a sampling bias
As a survey trainer, I find people are skeptical about the impact of biases upon a survey’s results.
This poll provides indisputable proof of some bias distorting the data. The pollsters provides some detail about the polling methodology, and they report the sampling error – which is driven by the number of responses. But they provide no discussion about sample bias or conformity bias.
A moment’s reflection upon their findings shows something is in play.
Look at the nearby results taken from CNN’s detail report for Question 15, which asks if “you, personally, know someone who has been diagnosed with the novel coronavirus.”
Yet the percent of people knowing someone who tested positive stayed flat from May to June.How is that possible?
The percentage clearly should have tracked up! (Going down would be truly odd.) And as the number of positive cases increases, the probability of anyone knowing a positive person increases exponentially. (The process would parallel the logic of the virus spread.) It is highly unlikely that this can be explained only by sampling error.
Further, look at the nearby table.
For the April 6 polling, 22% of people reported knowing at least one of the 365,000 people who had tested positive, which is roughly 0.17% of the US population.
For the May 10 polling, 40% of people reported knowing at least one of the 1,325,000 people who had tested positive, which is roughly 0.63% of the US population.
For the June 5 polling, 40% of people reported knowing at least one of the 1,893,000 people who had tested positive, which is also roughly 0.90% of the US population.
Count Knowing Covid Positive Person
If the polling data are accurate, one helluva lot of people knew each person who was positive with Covid-19. Extrapolating to the US adult population for the June 5 polling, each Covid-19 positive person was known by 44 people on average. (For May it would be 63 people, and for April, 127 people.)
Even though the question emphasizes “personally” knowing someone with Covid-19, the numbers just don’t make sense.
Did Conformity Bias Cause this?
This strikes me as a classic example of conformity bias, which is a type of response bias where respondents don’t provide an accurate response due to something inherent in them. Conformity bias is where people give an answer to be “one of the crowd.” (I won’t about it here.)
Here’s a commonly used analogy (from my home city): Tens of thousands of people claim to have been at Fenway Park for Ted Williams last game in 1960 in which he hit a home run in his last at bat. But attendance was only 10,454. People want to say they were part of it. Same thing here?
But how could the percent not increase from May to June? Are people just making this up – that is lying to conform? And they didn’t lie consistently from May to June? Yup, that’s possible.
Another possibility is that the sampling done to conduct the survey had errors, inadvertent or not. The polling methodology (see nearby call-out) makes no mention how the 1,259 people sampled were selected. We know there was oversampling of smaller demographic groups, presumably to decrease the sampling error for those groups. But there’s no mention of a sampling frame to ensure the sample is representative of the entire US.
CNN Poll Methodology
If no sampling frame was used, that is, if people were just randomly called from the entire US rather than from a series of districts that were representative of the US, then it is possible that the June poll overweighted urban, democratic areas. We do know that they oversampled minorities, and minorities are most likely to live in urban areas.
Did they adjust all the data to remove the oversampling process? That’s not clear and could lead to a true sample bias!
From just reading the summary data and statements of methodology, it’s impossible to know what lead to the bizarre “findings” from the question asking about knowing someone who has tested positive. However, logic tells us that something is wrong.
What astonished me most is that no one involved in the polling and reporting noticed this oddity. That’s a real head scratcher.
Note that this casts doubt upon the rest of the findings from this survey. The effects that lead to the result for this question may have corrupted other questions, but the result isn’t so obvious.