The ID Report

Post details: Peer review: What happens when only a few reviewers act out? - No better than chance

09/25/10

by Denyse O'Leary 11:06:55 am, Categories: Commentary - OpEd, 333 words

Peer review: What happens when only a few reviewers act out? - No better than chance

by Denyse O'Leary
ARN correspondent

Here's a revealing PhysicsWorld paper on peer review:

After running the model with 1000 scientists over 500 time-steps, Thurner and Hanel find that even a small presence of rational or random referees can significantly reduce the quality of published papers. When just 10% of referees do not behave "correctly" the quality of accepted papers drops by one standard deviation. If the fractions of rational, random and correct referees are about 1/3 each, the quality selection aspect of peer review practically vanished altogether.

"Our message is clear: if it can not be guaranteed that the fraction of rational and random referees is confined to a very small number, the peer-review system will not perform much better than by accepting papers by throwing (an unbiased!) coin," explain the researchers.

Daniel Kennefick, a cosmologist at the University of Arkansas with a special interest in sociology, believes that the study exposes the vulnerability of peer review when referees are not accountable for their decisions. "The system provides an opportunity for referees to try to avoid embarrassment for themselves, which is not the goal at all," he says. (September 9, 2010)

The major problem, as I see it, is that peer review is sold to the public as a key determinant of quality, which it isn't and can't be, under the circumstances. Scientists know about and talk about this problem, but nothing much seems to get done about it.

(Note: "Rational" means self-serving enough to reject a paper that might draw attention away from one's own work - I guess they were looking for a polite way to put that ...)

I have written elsewhere about peer review.

Toronto-based Canadian journalist Denyse O'Leary (www.designorchance.com) is the author of the multiple award-winning By Design or by Chance? (Augsburg Fortress 2004), an overview of the intelligent design controversy. She was named CBA Canada's Recommended Author of the Year in 2005 and is co-author, with Montreal neuroscientist Mario Beauregard, of The Spiritual Brain: A neuroscientist's case for the existence of the soul (Harper 2007).

Pingbacks:

No Pingbacks for this post yet...

The ID Report

Recently (cached)
Recently (no cache)

August 2012
Mon	Tue	Wed	Thu	Fri	Sat	Sun
<< <		> >>
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31

Last comments

Search

Linkblog

Links - Groups and Organizations

Access Research Network
Discovery Institute
Evolution News Blog
Evolution has become a favorite topic of the news media recently, but for some reason, they never seem to get the story straight. The staff at Discovery Institute's Center for Science and Culture started this Blog to set the record straight and make sure you knew "the rest of the story".
Intelligent Design Network (IDNet)
Intelligent Design Undergraduate Research Center
Intelligent Reasoning
A blogger from New England offers his intelligent reasoning.
International Society for Complexity, Information, and Design (ISCID)
Telic Thoughts
We are a group of individuals, coming from diverse backgrounds and not speaking for any organization, who have found common ground around teleological concepts, including intelligent design. We think these concepts have real potential to generate insights about our reality that are being drowned out by political advocacy from both sides. We hope this blog will provide a small voice that helps rectify this situation.
The Design Inference - The Writings of William A. Dembski
The Intelligent Design and Evolution Awareness Center (IDEA)
www.themonkeytrial.com
Website dedicated to comparing scenes from the "Inherit the Wind" movie with factual information from actual Scopes Trial. View 37 clips from the movie and decide for yourself if this movie is more fact or fiction.

Links - Of General Interest

A Brief View of Time and Those That Live There
Don Cicchetti blogs on: Culture, Music, Faith, Intelligent Design, Guitar, Audio
A Quick Guide to Sequenced Genomes
ARN Related Web Links
Creation/Evolution Quotes
Australian biologist Stephen E. Jones maintains one of the best origins "quote" databases around. He is meticulous about accuracy and working from original sources.
CreationEvolutionDesign
Most guys going through midlife crisis buy a convertible. Austrialian Stephen E. Jones went back to college to get a biology degree and is now a proponent of ID and common ancestry.
Darwinian Fairytales by David Stove
Complete zipped downloadable pdf copy of David Stove's devastating, and yet hard-to-find, critique of neo-Darwinism entitled "Darwinian Fairytales"
ID The Future
Intelligent Design The Future is a multiple contributor weblog whose participants include the nation's leading design scientists and theorists: biochemist Michael Behe, mathematician William Dembski, astronomer Guillermo Gonzalez, philosophers of science Stephen Meyer, and Jay Richards, philosopher of biology Paul Nelson, molecular biologist Jonathan Wells, and science writer Jonathan Witt. Posts will focus primarily on the intellectual issues at stake in the debate over intelligent design, rather than its implications for education or public policy.
John Mark Reynolds Blog
A Philosopher's Journey: Political and cultural reflections of John Mark N. Reynolds. Dr. Reynolds is Director of the Torrey Honors Institute at
Biola University.
NASA Astronomy Picture of the Day