Bias in Citation Networks
Design: A complete citation network was constructed from all PubMed indexed English literature papers addressing the belief that β amyloid, a protein accumulated in the brain in Alzheimer’s disease, is produced by and injures skeletal muscle of patients with inclusion body myositis. Social network theory and graph theory were used to analyse this network.
Results: The network contained 242 papers and 675 citations addressing the belief, with 220 553 citation paths supporting it. Unfounded authority was established by citation bias against papers that refuted or weakened the belief; amplification, the marked expansion of the belief system by papers presenting no data addressing it; and forms of invention such as the conversion of hypothesis into fact through citation alone. Extension of this network into text within grants funded by the National Institutes of Health and obtained through the Freedom of Information Act showed the same phenomena present and sometimes used to justify requests for funding.
Conclusion: Citation is both an impartial scholarly method and a powerful form of social communication. Through distortions in its social use that include bias, amplification, and invention, citation can be used to generate information cascades resulting in unfounded authority of claims. Construction and analysis of a claim specific citation network may clarify the nature of a published belief system and expose distorted methods of social citation.
From a fascinating piece by Steven Greenberg in the BMJ. I am often struck by how citation patterns — not just exclusion or inclusion, but misrepresentation of arguments, or what Greenberg calls “citation diversion” — develop within the literatures that I work in. For example, on a recent graduate student comprehensive exam, this student argued that Philip Converse believes that American voters are “stupid.”
We should probably train academics to read through literatures and synthesize them accurately, but few want to take the time. We focus on producing new original research and not on describing the extant research correctly. In fact, the misrepresentations of previous work often serve our own purposes: to make our work look more counterintuitive, path-breaking, or whatever. I am undoubtedly guilty of this.
When I first read the earliest studies of American presidential campaigns, I remarked to a colleague how well these books covered the field, and how much subsequent literature has been simply refinement.
My colleague was more pessimistic: “We’ve been un-learning what they learned,” he said.
Citation bias, and the resulting inefficiencies and distortions in the accumulation of knowledge, seem one reason for this.
[Hat tip to Julie Lynch.]
Comments
Professor Sides,
I found wikisum last summer. I started my PhD in the fall in comparative politics and political economy. I have never trusted Wikisum for that basic fact. If I can miss an argument in a paper, then why should I trust someone else to do it for me.
There are really bad incentives on this website. 1). Adverse selection. 2). Moral Hazard - in some pretty bad ways too. 3). Not to mention collective action problems of providing public information.
Needless to say, I think the good students should be concerned with summarizing correctly the past literature. I find that actually understanding the historical circumstances and surrounding literature gives great insight into why certain research projects came about. I hope this will come through in my Comparative politics exam, but it might not.
Thanks for the great post.
Posted by: Richard Pointer | July 22, 2009 07:26 PM
By the way, can you point out to me a good article summarizing the behavior revolution? As far as I can gather it was just the scientification of political research. Almond has a nice summary in The Handbook of Political Science (1998), but it is not as detailed as one would hope for comparative politics.
Posted by: Richard Pointer | July 22, 2009 07:32 PM
Robert: My use of wikisum is for a more prosaic reason: I need a link that gives a sense of what the work is about. Because the two things I link to are a chapter in an edited volume and a book, JSTOR et al. are of no help. I certainly don’t advise people to substitute wiki-anything for their own close reading.
Posted by: John Sides | July 22, 2009 07:33 PM
I liked reading this:
Dahl, Robert A. 1961. The Behavioral Approach in Political Science: Epitaph for a Monument to a Successful Protest. The American Political Science Review 55, no. 4 (December): 763-772. http://www.jstor.org/stable/1952525.
Posted by: William Hennessey | July 23, 2009 10:47 AM
Don’t you think the double-blind review process plays into this as well? If I’m reviewing a paper and someone cites my own work in a kind of off-hand manner that doesn’t really get the point of what I was arguing, I’m always hesitant to confront this head on out of fear that it will give up my anonymity. If we used a system where reviewers were identified, then we could at least have clearer communication in this regard.
Posted by: Joshua Tucker![[TypeKey Profile Page]](http://www.themonkeycage.org/nav-commenters.gif)
|
July 23, 2009 11:24 AM
I think Richard missed the point of why you linked to wikisum, but since he brings it up—
For what it’s worth, “wikisum” isn’t really a wiki (I’m the site’s owner). Back in graduate school seminars at UCSD, it was routine for professors to assign each article/book to a specific student; we all read the work, but that one student was tasked with bringing in a written summary and leading discussion on that work in class. I collected all those summaries and them posted them (mostly unedited) at wikisum.
Originally, it was an editable wiki like wikipedia (but only editable by folks I knew and gave a login to), but after a short time I turned off the editing capabilities since it wasn’t attracting many editors. So the usual problems that plague something like wikipedia don’t apply.
In other words, Richard is correct that there was a collective action problem (nobody worked on improving it), but I don’t see how there is a moral hazard or adverse selection problem—sort of a weird claim, actually.
Of course the summaries aren’t all perfect, but the point was never to replace reading the original texts—the point is just to jog your memory about what a text said.
Posted by: Adam | July 23, 2009 11:45 AM
Adverse selection because those who need such (anonymous) summaries are more likely to be weaker students who are trying to catch up on readings for which they don’t have time. If you allowed new additions, the summaries may not be of the best quality.
Moral hazard, I grant you would be less likely, mainly due to the large numbers of universities, but it is not unknown at my university library for key books to be trashed or stolen so as to harm other students’ ability to do important readings. Mind you, that is rare, but not unheard of. In this case, providing intentionally false or weak summaries. Not a huge problem and since it is a free service there is no compulsion to give summaries.
And yes, I was sure why he linked to wikisum. And yes we too did summaries for each other in my core phd courses. Often, I found them unhelpful and my time was better spent going over the text or my own notes.
Posted by: Richard Pointer | July 23, 2009 02:02 PM