The Monkey Cage

I find it very interesting that the writers of this blog are taking so much time to criticize this particular poll. As you have stated, many of these critiques can be directed toward almost any survey conducted by telephone. While I am sympathetic to some of the concerns regarding weighting and response options, it is still unclear why this blog is so skeptical of these research findings. I can imagine looking closely at any survey methodology and the results of that survey — of the many that are conducted across the country — and finding a number reasons for criticism.

Still, you chose these findings. Is this because the survey asks some unusual questions? Or is it because of the some of the sensationalism that is used to frame the results (mainly the use of the term “crazy”)? It seems to me that more time should be spent discussing the potential reasons and consequences of these striking results, with the caveat that (like most survey results) the findings may be somewhat different if certain aspects were modified.

About the leaners—i don’t see their exclusion as a concern. The poll is presented as a poll of self-identified Republicans. To me, the most natural interpretation of that would be people who would answer the party-affiliation question by saying they’re Republicans. I had, in fact, assumed this was the case, and really can’t see why anyone would assume that it did include leaners.

Also, another point. I believe it’s been hammered home in this very blog (unless I have you confused with someone else, but I don’t think so) that “Independent” doesn’t mean “moderate.” People call themselves Independents for a lot of reasons. Some of those people calling themselves Independents may be doing so because they don’t feel the Republican party pays enough attention to issues like Obama’s birth certificate, etc. I don’t think we can take it for granted that including the leaners would lead to lower percentages on these questions. It may seem reasonable, but it can’t be taken for granted.

The completion rate question is more concerning, I think. I can well imagine that if I were a Republican (I’m an Independent, and I’ll just leave you to guess what that means in my case (if you care, which you probably don’t)), that on being asked these questions, many of which seem ridiculous, I would suspect that I was being made fun of or something, gotten pissed off, and refused to continue. I would like to see that data.

I can try to answer CB’s question from my perspective, but it’s just my perspective.

Simply put, the results of this poll are pretty stark. Some might even use the term “shocking.” As such, they are interesting. There are tons of polls released every week, but both the questions asked in this one and the answers they got are, to me at least, inherently interesting and relatively novel.

In sum, my best guess as to why this much digging is being done by Monkey Cage into this poll is that it is, well, interesting, unusual, and has made a large splash. If this were my blog, I would be doing the same thing. As for discussing the reasons and consequences, well, that’s actually a REALLY big question. Huge. But perhaps it’s just the type of person that some political scientists are that when they hear something so stark the first question they ask is “really?” before they move on to “why?” and “so what?”

I don’t think all of those black people back in the day really thought OJ Simpson was innocent either. Sometimes polls become political and they don’t reflect what people really believe.

Which is more biased, this post or the Daily Kos/R2K poll?

I think that’s a perfectly fair question.

First, you use a completely made-up assumption (“that extreme 5% of voters on either the left or the right have some pretty crazy ideas”) to criticize the poll. If you’re going use that as a basis for criticism, you should back that up. My reading of polling of “crazy ideas” is that they are generally not associated with the leftmost Democrats or the rightmost Republicans, but rather folks that are alienated by the political system. Do you have any data to back up your claim?

Actually, by trafficking in stupid stereotypes, you’ve blinded yourself to one of the most interesting issues raised by this poll. Some of the questions polled by R2K are polled regularly by Pew (the only pollster I know of that discloses the type of information you’ve asked for from R2K). On gay marriage, the responses R2K got are slightly more extreme than conservative Republicans polled by Pew (in Oct. 2009). Pew’s numbers for Republicans in that survey don’t appear to include leaners either. Not including leaners is a perfectly acceptable (and respectable) technique, depending on the what one is trying to achieve.

Pew will give you their data if you ask nicely. Then you could do something more than bloviate and answer some important questions yourself (such as is R2K sample demographically different from other pollsters?)

As for the question wording, there is certainly room to criticise the wording of some of the questions (but others meet the AAPOR guidelines perfectly). It’s bit disingenuous to raise them about this survey without pointing out that many political surveys make the exact same mistake. Singling out this poll makes you sound like you are nitpicking.

Consistency bias is the strongest criticism you make. It might be a real problem with this survey. Or, maybe it’s not. Again, doing some real analysis would help answer the question you raise. Compare the results on questions that have been polled by others to the results R2K got. Figure out a fair way to ask these questions and persuade another pollster to ask them.

You make a big deal about post survey weighting. You pretty seriously misconstrue Ali’s remarks. You claim that that R2K doesn’t come close to living up to the standard he lays out, but your analysis is completely flawed. First, the two percent of the population with no telephone doesn’t affect the validity of phone surveys and you know it. That’s just silly. Second, you took his statements totally out of context. He’s explaining to lay people why they might never get called. You don’t seriously think that this all he would say about the topic, do you? Post survey weighting doesn’t really do anything to address the cell phone issue, either. If you take a look at his demographics, you will see that they match very well with those reported by other pollsters AND those reported by the 2008 exit polls for Republican voters. What’s your substantive beef with his procedures?

As far as I can tell, you have nothing more than a bunch of speculation and snide comments.

Interesting points regarding the survey. However, I think you are overestimating the impact of the screening process. This is because Independents are included in the screening. It seems then that only weak leaning republicans are excluded; those who tend to vote Republican but do not identify strongly enough with the party or their viewpoints to call themselves Republicans. These individuals are likely too tepid in their support for any given political party, such that they are not part of the population meant to be polled. In other words, if one wants a survey of Republicans, people who only tend to vote Republican, but are not active and sometimes vote for another party, seem to be outside of the population of interest. These voters could reasonably flip any given election and are more likely independents than republicans. Also, the examples you provided are fairly extreme. I have a hard time believing that eliminating “leans Republican” would skew the results as dramatically as you suggest. Your hypothetical examples suggest that “leaners” make up a substantial proportion of Republican candidate voters. Is there any evidence for this? I would suspect moderate and strong Republicans make up a far greater proportion of candidate votes, with but 5-10% coming from “leaners.” Either way, you need to provide evidence that excluding leaners does indeed provide a meaningful effect, rather than hypothetical examples.

Re: the response format, I completely agree. The poll should have used a format that allowed for strength of opinion to be determined. That having been said, I don’t think it negates the results, but it makes one wonder how many slightly agree to a question vs. strongly agree to a question.

Finally, you are correct that not having individual has an equal chance of being selected, and this would be a mistake by Research 2000. I would like to see if and how they weight their samples, as it would be unusual for a polling firm not to. Can you please find and verify this information?

without getting involved in the debate above, what struck me about this post is that it seemingly makes the argument that the results of a poll would look different if you include only self-identified republicans versus if you include republican leaners.

this mere days after another series of posts saying that there is no difference betwixt the two groups.

different authors, i know, and i also recognize that this juxtaposition can be spun both ways. but to some extent, it seems that either way, it’s spin.

what all this cries out for, imho, is a state (or even county?) level analysis of all these claims. are there more/fewer independents in massachusetts than california, and does the claim of being an independent have more/less meaning in those states? the newest post on the different spread between parties within each state legislature would seem to underscore this need. perhaps such a study is already out there?

This post by Nate Silver provides some evidence that the Daily Kos sample was not representative of Republicans:

http://www.fivethirtyeight.com/2010/02/republicans-are-conservative-but-are.html

William Ockham, your entire post is a Tu Quoque fallacy. You aren’t as rational as you think you are.

CB, are you one of those liberals that are the fair-weathered friends of science? Critical thinking runs both ways.

As progressives we SHOULD be criticizing our own. Otherwise we become a closed system of thought. CB and William are good examples of progressives that I’d like to see isolated from our movement.

Thanks for the interesting discussion and comments!

Just to be clear here, I’m speaking for myself now - Andrew can jump in with his own comments if he wants.

As to why interest in this particular poll, Matt Jarvis’s comments are right on. The poll results were a bit shocking and got a lot of publicity, so we figured they were worth taking a closer look at. Yes, many of the points we raised could be raised about other polls as well, but that doesn’t make them any less valid in terms of reasons to wonder whether the conclusions of this particular poll were warranted. Moreover, there seemed to be more questions than usual on this poll that weren’t replicated in lots of other polls, so we really had less information about our confidence in these findings than is usually the case with most polls.

On the question of the sample. Yes, as Passerby notes, you can interview anyone who you choose to interview. But I maintain that the more restrictive you make the sample in terms of “hard core” partisans, the more likely you are to get a higher proportion of responses that deviate from the median national respondent. As I have previously illustrated on The Monkey Cage, the median US citizen is a political moderate. Thus if we assume that the “hard-core” Democrats are more liberal than “weak” Democrats and “hard-core” Republicans are more conservative than “weak” Republicans, then it holds that the more you raise the bar to be considered a Republican (or Democrat), the higher the proportion of respondents you are going to find who whole views that deviate from the median voter.

So in response to William’s point, of course I made up the 5% of Republicans (or Democrats) are crazies figures, but it was merely to illustrate the point that making the sample more restrictive was going to drive up the proportion of crazies. But I also noted even in the original post that Ali could probably have made that category even more restrictive if he wanted (say limiting to strong Democrats; although as Nate Silver and passerby have noted it is worth wondering if the tenor of the questions didn’t have a similar effect, which makes the question of completion rate important.), so I wasn’t implying at all that they had picked this definition of Republicans to get the most sensational results - limiting to strong Republicans probably would have gone farther in this regard - but merely that the category “proportion of Republicans” who think X is a function of how we define “Republicans”.

I do agree that this opens up some interesting questions for future research. So it would be interesting to look at, for example, the ANES or some other survey that breaks Republicans down by leaners, weak Republicans, and Strong Republicans and see if the proportions holding these more “shocking” views do vary in the way I’ve suggested. (Although I’d guess that the ANES would be much less likely to include answers with only two possible responses…) Anyone who has relevant research or wants to poke around in the ANES please join the conversation!

Well, it seems the fervor about Obama has “changed” :D