The Monkey Cage

[With apologies for spending so much time in the weeds on this issue…]

In this piece, Andrew Kohut of the Pew Center gives his thoughts about why the NH polls got Hillary’s vote share wrong. Here, I think, is the crux of his argument:

Poorer, less well-educated white people refuse surveys more often than affluent, better-educated whites. Polls generally adjust their samples for this tendency. But here’s the problem: these whites who do not respond to surveys tend to have more unfavorable views of blacks than respondents who do the interviews.

I find this curious, because the Pew Center itself has done some excellent studies of the consequences for survey non-response for attitudes, and their findings suggest little difference between people who answer surveys and those who don’t. Here is the most recent report. (See also this scholarly publication based on the study.)

In this study, they conducted two polls, one “standard” poll that was in the field for 5 days (as per usual) and one “rigorous” poll that stayed in the field for 21 weeks. The former had a response rate of 25% and a cooperation rate of 34%; the latter had rates of 50% and 58%. The authors compare attitudes across the two polls, and then across a subsample of the rigorous poll that they call the “hardest to reach” (defined as “respondents refused the interview at least twice before complying and/or required 21 or more calls to complete”).

The key finding that bears on the NH polls is that there was very little difference in racial attitudes across the standard poll, the rigorous poll, and the hardest-to-reach. See the table entitled “Values and Attitudes” on this page.

Now, obviously this study is based on national samples, not NH samples. But Kohut’s explanation seems to entail an relatively unlikely series of events:

1) Non-respondents in the NH polls have significantly less positive affect towards blacks relative to respondents. In other words, the basic finding of the Pew study did not hold in this case.

2) A significant number of these non-respondents both voted and voted in the Democratic presidential primary. Even though the kinds of people who are less likely to answer surveys tend not to vote. Even though the kinds of people who vote in Democratic primaries tend not to have unfavorable attitudes toward blacks. (Of course, there are people who will vote but not answer surveys. And there are people who vote Democratic but have racial prejudices. I’m merely describing patterns of association here.) Moreover, there had to be a substantial number of these voting non-respondents to give Clinton 8 points more than she was anticipated to get.

3) Those voters were motivated to vote for Clinton, consciously or not, by their racial attitudes, rather than by other factors.

So I’m not convinced.

Interestingly, there is now at least one comparison of a pre-election poll to the exit poll, and this comparison shows significant movement to Clinton among women, but much less so among men — see Jon Cohen, pollster at the Washington Post, here. This suggests that perhaps there was a late trend, one that you cannot pick up by simply using the “when did you make your decision” question (as Kohut does in his op-ed, but Cohen suggests is problematic in his post).

So, there is still much to be learned. The American Association for Public Opinion Research has called on pollsters to release their NH data.

[Update: More from Mark Blumenthal. See also John Judis, who is skeptical of Kohut and has some data to back this up.]