thank you for sharing your thoughts - that are really very interesting points and ideas. Let me add some remarks, from my point of view, which is more from the side of obtaining and dealing with the referees' remarks and criticisms of my (not so many)submitted papers. I have found the referees' comments usually quite helpful to sharpen the points I would like to make and to avoid possible misinterpretation of what I want to say - but I suppose this stronlgy depends on two things: the field you are working on, and the care of the journal's editors to try to find a suitable and competent referee.
Of course, if you work, like me with heavy ion collisions, on a more mainstream subject with a large and well-connected community, getting a good and helpful referee is probably more easy than for more exotic topics. On the other hand, there are the ususal dangers of personal animosities, and more generally of uncritical group-think within the community, but I think that, in general, refereeing works more or less. This may be completely different if you are really working on some new ideas, which do not have a large community yet where they can be naturally embedded.
The other point, that the editor's choice of a referee is important, may be connected with your suggestion of paying - maybe more on a symbolical level - for refereeing. Of course, the possibilty or even the impression of the possibility of "buying" a favourite report has to be avoided at any case. But if there is some reward for the referee, the editor may have a chance to get more and more suitable candidates for refereeing a paper, and the referee may have the impression that his work and effort is not completely taken for granted and may be willing to invest more thoughts on his report.
I had once thought that it may be helpful to include parts of or all of a referee's report on a paper to the publication - at least for the electronic versions, this should not be possible without much extra cost. That may make a journals decision to publish a certain paper more transparent, anbd give some kind of justification why a paper is published in the end. I even thought that the referees name could be added, making the role of the referee more obvious. However, this may be not a good idea, since such a procedure may prevent doubtful and scrupulous referees from writing reports, because all could turn out to be wrong.
The idea to add reviews to papers on the arxive sounds really great - I think it may be really helpful if you try to get into some new field or subject, where usually the sheer number of papers is overwhelming. There is the Net Advance of Physics directory, which I found quite useful for some time, but it seems not to be updated any more since some time. Of course, I have no idea how big a deal it would be for Cornell to add such a review option to the arxive - that may require really lots of work. But it may be worth it.
Stefan, excuse me for saying that but it also depends on whether your name (or the name of your advisors, or the institution you are from) are known in the community.
Which it a) shouldn't and b) is probably the reason why you have mostly positive experience.
Go try to submit a paper as Imanuel Gohol Sanawajaran from eastern Iran to Nucl Phys B, with an email address at yahoo.com and read the report. Good luck,
B.
7:44 PM, March 29, 2006
Anonymous said...
Hi, my name is Carlos Leiva.
I totally agree with Sabine, usually is a matter of names, it is very hard to submit a paper with a name like mine and coming from Arica Chile, a totally unknowed university of a small country in South America. I usually submit to IJMPA or MPLA, but the whole process takes, at least, a year, and the referee answers in about 4 moths.
It is very sad to see papers as good as mines being published in 4 months, specially when I am the only one that works in my subject in my Univesity and have to read, learn, having good ideas, calculating and writing all alone. Impossible to challenge a professor with good ideas and many students doing the hard work.
thank you - I got your point. And Carlos, thank you for sharing this point of view from South America (I admit that I have never heard of Arica, but now I've learned that it is the northernmost city of Chile, and the city of eternal spring time).
It is indeed true that I had "names" of senior authors on these papers, and that my affiliation is also known within the community. One could argue that this does not help by default to get a paper published, but for sure, it speeds up the whole process.
But what could be done to avoid this? Would not, then, the double-blind process be the only option? Or are there other reasons involved, maybe even some bias against contributions of less known provenience already in the editorial offices of the big US or European journals?
Concerning the other point, about adding comments to papers on the arXive: this afternoon, I came across a site that at least once has tried to do something like that,
which has a little bit more traffic and is more concerned with hep-ph/gr-qc topics. Of course, it is a pitty that all these sites are not immediatly connected to the arxive, and must be searched separately.
Nice post. Points 1 and 2 are good suggestions. Giving a financial incentive and making the reviewer 'known' (through acknowledgments or with a separate tag) to the readers, certainly would help improve peer-review standards.
Hi Stefan and Carlos:
A related post of mine on peer-review discusses at least one alternative for the double blind peer-review process.
Let me know of what you think.
(I am an asst. prof. specializing in thermal sciences. from India.)
Thanks for redirecting me to this post. It was an interesting read. Two more comments on peer review:
1) My ideal moderation / ranking system would rate not just technical accuracy, but how readable it is, how innovative, and so on.
2) There is a lot of subjective preference in evaluating "good" papers. I think it would be nice if the ranking was based on my web of trust. It would be nice if I could assign weights to the opinion of this or that person to get a personalized ranking. If the default is a zero weight, then this also eliminates the impact of comments made by people you consider crackpots.
[PS: you told me you received e-mail notifications, but I don't. So chances are I may not see your reply to this if any.]
10:11 AM, September 24, 2007
Last week, I received a manuscript that I was asked to referee. Since already 3 other manuscripts are lying on my desk (somewhere) waiting to be refereed, and the topic was pretty outside my field, I decided to let the editors know that I am not suitable as a referee. Being blessed with one of these online services, I clicked on the wrong link in the email and got thanked for having agreed to referee the paper.
Now I actually have to read this thing. Though at second sight it turned out to be more interesting than I initially thought.
Anyway, I had an engaged discussion on the weekend on the publishing issue that I would like to share with you.
We all know that the pressure to publish in peer reviewed journals is not improving quality in research. On the contrary, it leads people to favor publishing more of less quality. The reasons why some papers get published, whereas others don't, are sometimes just mysterious, sometimes clearly due to the authors names. Peer review, it seems, does not work as it should. And the number of publications, and their citations are - at least in my opinion - not necessarily a way to single out good research. However, the question is, what can there be done about it.
Some points that we came up with.
The referee should have some advantage from refereeing a manuscript. In such a way that (s)he is motivated to think about the content and make reasonable suggestions. I am mostly thinking in terms of credibility. I suspect that most journals probably have an internal ranking for referees anyway, but what does the referee ever get out of writing good reports? One might also consider giving some kind of bonus for writing reports in time. I would happily pay $ 50 for actually receiving a report within 2 weeks!Being mentioned in acknowledgements should be rated higher. People who are frequently mentioned in acknowledgements show that they are engaged in discussions, are able to understand and criticize theories, and are active part of research. This is the more important, as those who don't want to be part of fashion waves often end up with less publications. As to my papers, the quality improves significantly with every person I can discuss its content. (Restrictions apply).The number of citations should be normalized to the number of active workers on the field. At least approximately.I would find it enormously helpful if the arxiv would allow reviews on the papers, maybe similar to those at amazon. You might argue that a good physicist should be able to judge on the quality on a paper by himself. Though that is in principle true, it is absolutely inapplicable if you are new in a field and try to get into it. Some kind of quality index, or references to basic papers on the field will help newcomers to get to the central questions much faster - and with less wasted toner. In addition, the possibility of having reviews on the arxiv would make it unnecessary to have follow up papers titled 'A note on gr-qc/...' and 'A remark on a note on ...' etc.Another point that I have argued against is the idea of double blind refereeing process. First, it does not work when the papers are already on the pre-print archive before submitted to the journal. You then would have to make sure to only accept manuscripts not on the pre-print server, which would make the pre-print idea completely absurd. Second, it would only lead authors to write their papers such that for everyone in the field it's clear who the author is. Third, I actually do think that the credibility of the author is an input the referee might want to consider. If you have further suggestions, let me know!Best,B.
posted by Sabine Hossenfelder at 3:34 PM on Mar 28, 2006
"Peer review II"
6 Comments -
Dear Bee,
thank you for sharing your thoughts - that are really very interesting points and ideas. Let me add some remarks, from my point of view, which is more from the side of obtaining and dealing with the referees' remarks and criticisms of my (not so many)submitted papers. I have found the referees' comments usually quite helpful to sharpen the points I would like to make and to avoid possible misinterpretation of what I want to say - but I suppose this stronlgy depends on two things: the field you are working on, and the care of the journal's editors to try to find a suitable and competent referee.
Of course, if you work, like me with heavy ion collisions, on a more mainstream subject with a large and well-connected community, getting a good and helpful referee is probably more easy than for more exotic topics. On the other hand, there are the ususal dangers of personal animosities, and more generally of uncritical group-think within the community, but I think that, in general, refereeing works more or less. This may be completely different if you are really working on some new ideas, which do not have a large community yet where they can be naturally embedded.
The other point, that the editor's choice of a referee is important, may be connected with your suggestion of paying - maybe more on a symbolical level - for refereeing. Of course, the possibilty or even the impression of the possibility of "buying" a favourite report has to be avoided at any case. But if there is some reward for the referee, the editor may have a chance to get more and more suitable candidates for refereeing a paper, and the referee may have the impression that his work and effort is not completely taken for granted and may be willing to invest more thoughts on his report.
I had once thought that it may be helpful to include parts of or all of a referee's report on a paper to the publication - at least for the electronic versions, this should not be possible without much extra cost. That may make a journals decision to publish a certain paper more transparent, anbd give some kind of justification why a paper is published in the end. I even thought that the referees name could be added, making the role of the referee more obvious. However, this may be not a good idea, since such a procedure may prevent doubtful and scrupulous referees from writing reports, because all could turn out to be wrong.
The idea to add reviews to papers on the arxive sounds really great - I think it may be really helpful if you try to get into some new field or subject, where usually the sheer number of papers is overwhelming. There is the Net Advance of Physics directory, which I found quite useful for some time, but it seems not to be updated any more since some time. Of course, I have no idea how big a deal it would be for Cornell to add such a review option to the arxive - that may require really lots of work. But it may be worth it.
All the best, Stefan
5:27 PM, March 29, 2006
Stefan, excuse me for saying that but it also depends on whether your name (or the name of your advisors, or the institution you are from) are known in the community.
Which it a) shouldn't and b) is probably the reason why you have mostly positive experience.
Go try to submit a paper as Imanuel Gohol Sanawajaran from eastern Iran to Nucl Phys B, with an email address at yahoo.com and read the report. Good luck,
B.
7:44 PM, March 29, 2006
Hi, my name is Carlos Leiva.
I totally agree with Sabine, usually is a matter of names, it is very hard to submit a paper with a name like mine and coming from Arica Chile, a totally unknowed university of a small country in South America. I usually submit to IJMPA or MPLA, but the whole process takes, at least, a year, and the referee answers in about 4 moths.
It is very sad to see papers as good as mines being published in 4 months, specially when I am the only one that works in my subject in my Univesity and have to read, learn, having good ideas, calculating and writing all alone. Impossible to challenge a professor with good ideas and many students doing the hard work.
Cheers
Carlos
3:11 PM, March 30, 2006
Dear Bee, dear Carlos,
thank you - I got your point. And Carlos, thank you for sharing this point of view from South America (I admit that I have never heard of Arica, but now I've learned that it is the northernmost city of Chile, and the city of eternal spring time).
It is indeed true that I had "names" of senior authors on these papers, and that my affiliation is also known within the community. One could argue that this does not help by default to get a paper published, but for sure, it speeds up the whole process.
But what could be done to avoid this? Would not, then, the double-blind process be the only option? Or are there other reasons involved, maybe even some bias against contributions of less known provenience already in the editorial offices of the big US or European journals?
Concerning the other point, about adding comments to papers on the arXive: this afternoon, I came across a site that at least once has tried to do something like that,
www.science-advisor.net
Unfortunately, it seems not to be completely up to date, and the few contributions I have checked are mainly about condensed matter topics.
But that reminded me also that there is still another similar site,
www.physcomments.org
which has a little bit more traffic and is more concerned with hep-ph/gr-qc topics. Of course, it is a pitty that all these sites are not immediatly connected to the arxive, and must be searched separately.
Cheers, Stefan
3:19 PM, April 04, 2006
Hi Bee,
Nice post. Points 1 and 2 are good suggestions. Giving a financial incentive and making the reviewer 'known' (through acknowledgments or with a separate tag) to the readers, certainly would help improve peer-review standards.
Hi Stefan and Carlos:
A related post of mine on peer-review discusses at least one alternative for the double blind peer-review process.
Let me know of what you think.
(I am an asst. prof. specializing in thermal sciences. from India.)
2:38 AM, August 05, 2006
Hi Bee,
Thanks for redirecting me to this post. It was an interesting read. Two more comments on peer review:
1) My ideal moderation / ranking system would rate not just technical accuracy, but how readable it is, how innovative, and so on.
2) There is a lot of subjective preference in evaluating "good" papers. I think it would be nice if the ranking was based on my web of trust. It would be nice if I could assign weights to the opinion of this or that person to get a personalized ranking. If the default is a zero weight, then this also eliminates the impact of comments made by people you consider crackpots.
[PS: you told me you received e-mail notifications, but I don't. So chances are I may not see your reply to this if any.]
10:11 AM, September 24, 2007