The ID Report

by Denyse O'Leary
ARN correspondent

Peer review problems went "public" mainly as a result of recent high-profile scandals like Science's peer-reviewed Korean stem cell research paper that turned out be fraudulent. The reaction was interesting. Some sought to minimize the problem by treating the incident as highly unusual, disclaiming editorial responsibility for fraud, and placing their faith in peer review nonetheless. Soothing comparisons have frequently been made to Winston Churchill's characterization of democracy as the worst system - except for all the others.

But the convenient analogy to democracy fails. In the first place, the secrecy in which peer review operates make it a poor analogue to democracy. Second, democracy aims primarily to give every citizen a vote. The fact that some citizens vote for cranks or criminals does not mean that democracy has failed. But peer review's primary aim has been quality control, and it has been failing for decades. It squelches too many good ideas while failing to prevent too many frauds.

Flawed, yes! Fraud, no!

Some argue that the peer review system was designed to detect incompetence but not fraud. Flawed, yes, but fraud, no. Even if the system really worked that way, and - as we shall see - there are grounds for doubt, embarrassing frauds have created a demand for a system thatcan detect fraud.

A host of individual acts of sloppiness (or malice!) can get lost in the smoke generated by a really big fraud like the stem cell scam. Defenders of the system can then safely claim that the Big One is unrepresentative. That is usually not true. It would be more accurate to say that the ensuing uproar is unrepresentative. With scandals, as with rats, if you see one, there are probably a dozen, and the rat that caught your headlights was just unlucky. And a big one always gets more attention than a bunch of little ones.

A minor flurry can be more representative of the system's usual problems than a really big one. For example, according to Paul Greenberg, the prestigious New England Journal of Medicine (NEJM) took five years to correct information regarding the potentially harmful drug VIOXX, which it had first publicized in peer-reviewed research in 2000. In 2001, in response to a pharmacist's questions on a Seattle radio show, the editor explained, "We can't be in the business of policing every bit of data we put out." Then what, precisely, is peer review supposed to do?

Ironically, having helped diminish the value of peer review, NEJM sneered recently at the intelligent design theorists, quoting with approval a recent court decision that claimed that ID, among other things,"has not generated any peer-reviewed publications."

Now, that isn't true, as it happens. There is a modest but growing number of ID-friendly peer-reviewed publications. But - given the woeful state of peer review - papers that support or undermine ID hypotheses would probably be neither better nor worse recommended if they were never peer reviewed, just published, amid cheers and catcalls..

Interestingly, the ID journal, Progress in Complexity, Information, and Design (PCID) has opted to return to the early twentieth century approach, where a senior scholar recommends a junior scholar for publication. Time will tell if this old method can be revived successfully.

The grapes of froth?

Another classic case of questionable peer review was unearthed by mathematician Douglas J. Keenan: A research team published a paper in ther prestigious journal Nature (November 18, 2004), claiming to have pioneered a means of estimating the summer temperature in Burgundy (France) back to 1370, using the harvest dates of grapes. The authors asserted that 2003 featured the warmest Burgundy summer since 1370.

(No one should be surprised if considerable media publicity followed the publication of this paper, as Keenan asserts. During the summer of 2003, thousands of old people in France died of heat prostration, and some blamed the government and others blamed selfish adult children. Psychologically, all surely yearned for someone to tell them that it had indeed been an apocalypse.)

The trouble is, Keenan says, he had to go to a great deal of trouble to get the data to research the problem further,

To study the paper properly, I needed to have the authors' data. So I e-mailed Dr. Chuine, asking for this. The authors, though, were very reluctant to let me have the data. It took me eight months, tens of e-mails exchanged with the authors, and two formal complaints to Nature, before I got the data. (Some data was purchased from Météo France.) It is obviously inappropriate that such a large effort was necessary.

And when he did get a chance to study the data, he concluded that there were serious problems with the work:

In particular, the authors' estimate for the summer temperature of 2003 was higher than the actual temperature by 2.4 °C (about 4.3 °F). This is the primary reason that 2003 seemed, according to the authors, to be extremely warm.

(Note: Events would seem to suggest that that year was extremely warm , though not necessarily as warm as the Nature paper claimed - and not the prophesied global warming apocalypse either.)
Keenan's main issue was with the authors' statistical work. The errors he discovered should not, he argues, have required specialist scientific training to uncover. So why did the peer reviewers not notice them? But he need not have bothered wondering about that, because the lead author told him, "We never sent data to Nature."

A layperson might well be astounded to hear this. Researchers who could successfully estimate the summer temperature back to the 14th century could shed light on a variety of issues in European history, and might well be besieged for data. Unless ... can it be? Just any old nonsense that supports a global warming apocalypse gets a pass? Keenan later wrote about this problem in Theoretical and Applied Climatology (May 2006), and no doubt the controversy will go on.

This incident underlines a general problem with peer review: Findings that support a consensus are too easily accepted - that is the inevitable flip side to squelching new ideas.

So three things have become quite clear in recent years:

1) The Korean human stem cell fraud was certainly not the only peer review scandal, but it was the hottest media story.

2) Peer review in general has not been doing what it was supposed to do - improve the quality of science papers - and it has squelched new ideas while failing to challenge questionable support for accepted ones.

3) Internet-based technologies may enable a more open and dynamic system. In a way, it can be compared to the blogosphere. The blogosphere, for all its faults, has been a breath of fresh air in media. It has restored the original concept of news as what people want to hear about rather than what gatekeepers think they should want to hear about. As a result, "newsroom paternalism" is much harder to enforce than it used to be. Perhaps a similar benefit for science will result in the airing of genuinely new ideas - whether to rise to glory or be shot down in flames.

Next: Part Three: How the system is slowly becoming more open and dynamic, whether anyone wants it to or not

Toronto-based Canadian journalist Denyse O'Leary (www.designorchance.com) is the author of the multiple award-winning By Design or by Chance? (Augsburg Fortress 2004), an overview of the intelligent design controversy. She was named CBA Canada's Recommended Author of the Year in 2005 and is co-author, with Montreal neuroscientist Mario Beauregard, of the forthcoming The Spiritual Brain (Harper 2007).