Peer review part 1

Is peer review over-rated?

http://www.samizdata.net/blog/archives/2009/12/peer_review.html

"Referees are supposedly anonymous. However, the author, the editor, and the referees often work in small fields where everybody knows one another, and people's beliefs, foibles and writing styles are often well known, so this anonymity is often more theoretical than real. The theoretical reason for anonymity - that the referee can say what he pleases without consequences - is not always entirely true. The anonymity is one sided: the referee receives a paper with the name of the author at the top. The name of a famous and influential scientist at the top has an impact. The editor is very powerful, as he gets to select the referees and by choosing referees carefully clearly has influence whether a paper will be published or not. A good editor will choose referees of mixed levels of seniority (referees include everybody from graduate students to senior professors), and (in areas of some dispute) of mixed positions in any argument."


http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/memo/climatedata/uc3602.htm
Peer review is inadequate to the task of assessing scientific findings for policymakers

"8 Academic studies on peer review to identify fraud and error have not painted a good picture of its ability to detect fraud and error. In the words of Richard Smith, the former editor of the British Medical Journal, "We have little evidence on the effectiveness of peer review, but we have considerable evidence of its defects. In addition to being poor at detecting gross defects and almost useless for detecting fraud, it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias and easily abused."

9 The CRU disclosures demonstrate that the peer review process can be subverted by a small but influential group of scientists. In the emails we see that there were at least four attempts to subvert journals[*] by putting pressure on editors to reject or delay submissions that were critical of mainstream climatology or to otherwise hinder sceptics. Editors who stood in the way of this group appear to have been forced from their posts. Articles by activist scientists were sent to sympathetic reviewers. Articles by sceptics were sent to hostile reviewers.

10 Policymakers need to be clear that peer review does not normally involve obtaining the scientific data and code used in a study and reproducing the findings. It is normally simply a read-through of a paper. This is adequate for finding glaring errors or non-original work. It is an absurdly inadequate process for justifying multi-billion pound decisions. As McCullough and McKitrick put it, "some government staff are surprised to find out that peer review does not involve checking data and calculations, while some academics are surprised that anyone thought it did".[3]

11 With scientists assessed on their productivity, in terms of numbers of papers published and citations achieved, there is little time for replication of the work of others. However, with peer review being such a weak check on scientific correctness, replication is the only way to ensure that decisions are taken on a sound scientific basis. Policymakers need to consider how they will ensure that scientific findings on which they base their decisions have been adequately replicated."


http://www.samizdata.net/blog/archives/2010/01/samizdata_quote_597.html
This Samizdata article mentions "peer-to-peer review" (with enough eyeballs all bugs are shallow). One problem is that scientists are not publishing enough data for their work to be properly evaluated or replicated. Though "with enough eyeballs all bugs are shallow" relies on enough people devoting their eyeballs, which didn't happen with Wikileaks. Commenters at http://www.bishop-hill.net/blog/2011/4/13/peer-to-peer-review.html mention problems with "open peer review". All data should still be published though. All papers too (ArXiv).


http://www.timeshighereducation.co.uk/story.asp?storycode=191225
"Lecturer plagiarised student's work
17 September 2004

A senior lecturer at Cardiff University has been suspended after an investigation panel found that he had plagiarised a former student's PhD thesis for articles published in two international journals.

Cardiff confirmed this week that an allegation of "misconduct in academic research" had been substantiated against Kamal Naser, a senior lecturer in accounting at the university's business school."