Two Thirds Empty or One Third Full?
Surveys show that Americans do not know a lot of facts about politics (see, e.g., here). But why? A simple answer is that people do not know them.
But in a new paper (here or here, gated) Arthur Lupia and Markus Prior have a more complicated answer. They suggest that people may not give correct answers to knowledge questions because (1) they do not put forth much effort in answering the question or (2) cannot quickly recall the answer but perhaps could easily find it if given the chance. A short survey interview does not tend to encourage much effort; at the same time, it tests only immediate recall.
So Lupia and Prior designed an experiment. Each subject was asked to answer 14 knowledge questions. The experiment had four conditions:
1) A control group, which had 1 minute to answer each question.
2) The “money” group, which had 1 minute to answer each question and was paid $1 for each correct answer.
3) The “time” group, which was given 24 hours to answer these 14 questions.
4) The “money + time” group, which was given 24 hours and $1 per correct answer.
The results? Here is the mean number of correct answers in each group:
Control: 4.5 (out of 14)
Money: 5.0
Time: 5.4
Money + time: 5.6
The differences relative to the control group are statistically significant. Moreover, Lupia and Prior find that money and time are most effective upon those without a college degree and with only moderate interest in politics.
They conclude:
Our results provide a new and distinct reason for being skeptical when analysts use existing knowledge measures as the basis for sweeping generalizations about what citizens do not know…In particular, existing political knowledge measures, when used as measures of political competence, likely underestimate the public’s true abilities.
In thinking about this, it occurred to me that a skeptic (or, better put, a pessimist) might say: If the average respondent in every group answered about 5 or 6 out of 14 questions, is this “sweeping generalization” really that inaccurate? Is most of the variance in knowledge really explained by, well, knowledge, rather than by a lack of effort or ability to recall the answers correctly?
Those interested in this topic should also see this paper by Lupia, which questions the value of these knowledge questions in assessing the public’s competence as democratic citizens.
Comments
You are bang on. Making mountains out of molehills is an honored tradition in academic writing.
Of course, the same could be said about journalism. States that vote 53 percent Republican are strongholds of anti-liberalism. A five-point election win is a landslide (see your avalanche comic in the previous post).
Writers must believe that sensation sells. They are probably right, but it's an unproven position.
Could we be over-hyping hype as well?
Posted by: Chris Blattman | February 24, 2008 11:39 AM
That was my reaction. Although, it's not "any participant" it's the mean participant.
Posted by: jsalvati | February 24, 2008 11:40 AM
I wonder how the results of the survey would differ if the incentives were a bit better than a dollar a question: 14 questions only gets you 14 dollars. Now, if it was 100 dollars a question, there might be a bigger rise in the number who starting getting a substantial number of the questions correct.
Posted by: PRT | February 24, 2008 12:39 PM
I realize that the differences between the control group and the three treatment groups are statistically significant, but the fact remains that respondents are able to answer no more than 40% of the questions correctly. To be sure, the traditional question format used in most surveys probably underestimates "true" political knowledge, as is demonstrated here. But I am struck by the fact that creating a financial incentive and giving a lot of extra time generates only 40% correct responses (from about 33% in the control group). This strikes me as reinforcing the view that Americans' level of political knowledge is pretty low.
Posted by: Jim Garand | February 24, 2008 04:34 PM
Let me see if I've got this right -- most of the people answered five out of the 14 questions within a minute, but when given an additional 23 hours and 59 minutes, and a small financial incentive,they added on average another one or two correct answers to their score.
Could they not Google the answers? Go to a library? Phone their friends?
What this study proves to me, is that there is a lack here which goes beyond basic knowledge and off into the territory of knowing how to acquire answers to questions of fact in the public domain.
That's a polite way of saying that for practical purposes this study population was as dumb as a sack of hammers.
Noni
Posted by: Noni Mausa | February 24, 2008 09:13 PM
Do those results mean much in substantive terms, statistical significance notwithstanding?
Posted by: I love stars | February 25, 2008 04:36 AM
I agree with the previous commenters that the improvement is hardly staggering.
Moreover, even if the difference were staggering, the conclusions are hardly more optimistic as IMO when it really matters for political decision-making (i.e. in the voting booth) conditions are more comparable to those of the control group than to the time or money groups.
Posted by: eulogist | February 25, 2008 05:41 AM
Amazing, that this is in AJPS, and that some of our biggest names wrote this.
Then again, on a second thought, maybe not that amazing.
Posted by: I love stars | February 25, 2008 08:32 AM
For me, it kinda falls in the "who cares" category, and reminds me of the "deliberative polling" wave going on right now.
Knowing something and knowing how to find something are completely different things, with very different consequences. When a person receives new information, that has to contend against what they THINK they know, not what they might know if inclined to go look it up. Here, I'm largely agreeing with eulogist, and I think most of those here.
The "so what" question is pretty weakly answered here: the answer is "a small effect, and one that wouldn't necessarily mean anything if it was larger"
Posted by: Matt Jarvis | February 25, 2008 02:07 PM