Comments on Computational Complexity: Public Referee Reports

I was thinking about the web-log format: Why not ...

2004-11-08T08:38:00.000-06:00

I was thinking about the web-log format:
Why not to publish our papers on-line, where people can comment on it / discuss it, as what we do with Lance's Posts?

I occasionally get sent papers to referee. I alway...

2004-11-08T03:00:00.000-06:00

I occasionally get sent papers to referee. I always decline, since I've never once been the primary author of an academic paper and have only a vague idea of how they're judged, although I do as a courtesy read the paper and give a few comments on it.

Since reviewers are anonymous, I could just write completely incompetent reviews and rely on the anonymity to avoid soiling my reputation, but that would appear to be a bad thing.

-Bram

Having referees sign their report is a terrible id...

2004-11-06T00:21:00.000-06:00

Having referees sign their report is a terrible idea. Just imagine a graduate student refereeing a paper by a senior person who might some day decide about her job application.

I don't like the idea of making the reports public either, but I think the main argument for this suggestion is that it helps the referees not waste time on papers that get submitted many times to different conferences until they find their place in the "food chain".

--Mohammad

I sometimes think that, if referee reports were pu...

2004-11-05T18:47:00.000-06:00

I sometimes think that, if referee reports were public and signed, people would be *more* willing to write them, because then their effort would be rewarded with what every academic craves the most: recognition. Also: if publicizing referee reports would cause referees to be too polite, then why is this not an issue in the wider intellectual world? For example, one often sees new books savaged in book reviews written (and signed) by the author's peers. And in that case, the damage to the author's ego is even greater.

I think the "moral arguments" for anonyminity that...

2004-11-05T15:14:00.000-06:00

I think the "moral arguments" for anonyminity that are being given are great. But let me suggest a more practical reason. How many journals can you name where the referee process is not already too long and too slow? Part of the reason for this is it is often very hard to find people willing to review papers.

"An argument for posting the reports goes as follows: If we make the reports public (though still anonymous) referees will feel a greater responsibility to make their report thorough and accurate."

Actually, it would make me feel less inspired to referee. Unless I tried to obfuscate my style, I imagine the public could conclude with high accuracy what referee reports were mine. Perhaps this would make me feel a greater responsibility to do a thorough job; more likely it would make me feel like the burden of refereeing was even greater, and I would say no more often.

People that sign their referee reports evidently r...

2004-11-05T12:38:00.000-06:00

People that sign their referee reports evidently reveal not only their own identities but also some information about the identities of other referees. Sometimes very important information.

I agree that reports should not be publicised (and...

2004-11-05T12:35:00.000-06:00

I agree that reports should not be publicised (and I've said as much). One nitpick: the back-and-forth you describe is for journal refereeing. In conference refereeing, there is no right of reply for the author. On the other hand, a referee who recommends "accept conditioned on revisions" may have no mechanism for ensuring that the author actually carries out the revisions. So conference refereeing is not the same sort of reasoned back-and-forth you describe in this entry. There may be discussions within the PC, but in the cases I've seen the author is not involved except in extraordinary circumstances. I don't think this makes posting conference referee reports a good idea, but it does highlight a source of concern.

-David Molnar

I thoroughly dislike the idea of making referee re...

2004-11-05T09:43:00.000-06:00

I thoroughly dislike the idea of making referee reports public. The proposal to make them public somehow stems from a basic mistrust of the exisiting authors-editors-referees path of communication. I have no such mistrust. For the purposes of evaluating relevance or importance of a paper to a researcher, as Lance has suggested, something like MathReviews is very helpful and we should encourage such venues. As a referee, publicizing my reports (even anonymously) would seriously hamper my objective evaluation of a paper. This is especially true if the report includes controversial comments about re-discoveries, earlier history (e.g. previously rejected from another journal or conference) of the same paper, etc. Of course, such things can be edited by editors, but how much more do we want to burden our already overloaded (mostly volunteer) editors?

Lance, what's your opinion about the usual anonymi...

2004-11-05T09:42:00.000-06:00

Lance, what's your opinion about the usual anonymity of referree reports? A few people in our community�most notably, Joe O'Rourke�always sign their referee reports. I can see arguments both ways.