Comment: Publishing’s chinese whispers

A couple of days ago I checked Twitter and ended up reading a discomforting account of how things can go wrong when publishing a piece of research (or trying to do so) posted on the author’s blog.

To start, I should clarify that I do not work for the publisher that is part of the author’s story, and my post is written with no specific company or journal in mind. I shall express personal thoughts on what I read, and I’d like the accent to fall on “personal”. To reformulate this statement with words familiar to Twitter users, you will find “views [that are] my own” here.

One benefit of reading the author’s post is that it confirmed a feeling that I’ve had for a little while: there is often a (partly) broken link between authors, editors, publishers and readers. The connection can be broken at various points and to different levels of severity; as an unreliable communication between parties takes place, an effect that appears quickly is the propagation of corrupted information – following a phenomenon similar to what happens when playing chinese whispers.

I’m passionate about writing and publishing, and I have a genuine interest in open access. Particularly, I believe that it is important to keep the communication line open with this growing community, and that the worst move that ‘big traditional publishers’ can make is to break the line and leave the room. I read the author’s account in full, and there is little to argue on one point: his experience was dreadful and, one would hope, belongs to the set of rare events. As I read on, some comments on the peer review process caught my attention. The rest of this post focusses on my reactions to specific statements.

“One can only imagine how often this is the case for published papers.” This sentence referred to the peer review process not being rigorous, as it involved a single referee and an overly succinct review. As a general remark, I agree that the review process should provide more detailed feedback. However, to infer that what happened with this submission applies to many published papers that are out there seems far-fetched and, to be fair, not rigorous at all. Since when does one occurrence generate a rule?

“My apologies here to the referee who I imagine did their best” – this comment was a little jaw-dropping, I must say. Did I miss something along the way? I don’t understand why the author apologises to the reviewer here. Is this supposed to suggest that the referee was more or less surreptitiously ill-advised by the editor to write a short and superficial review of the article under consideration? I fail to see on which grounds it seems to be the case here that referees have no autonomous responsibility for the reviews that they provide. As a general observation, I would like to stress this point: the submission of a paper, and the peer review process that can follow, is a three-body interaction – the three ‘bodies’ being the author(s), the editor(s) and the referee(s). They interact with one another, yet each party has its independence. If this is not the case there is a problem, but that is a different question. Unless the author has evidence for this being the issue here, I have trouble figuring out why he feels compelled to apologise to the reviewer.

“this is a problem when you have a closed, anonymous system, in which editorial decisions are based often on incomplete information.” ‘Anonymous’ does not imply ‘closed’, a priori – perhaps “not transparent” should replace “closed”? This choice would make more sense to me. Indeed, one can argue that the peer review process is not always transparent. In fact, this important aspect has sparked stimulating discussions and valuable ‘experiments’ carried out by some journals – such as publishing the referee reports along with the original research papers, or acknowledging editor(s) and referee(s) explicitly when an article is published. This is all good and part of a (mostly) constructive debate. Yet I lose track of the logic behind the words when I read that “editorial decisions are based often on incomplete information” as if this was a shocking revelation. How do we define what complete and incomplete information with respect to the peer review process is? When is any decision – any! – based on a truly complete picture of the considered problem? I was under the impression that this was part of our evolution – learning to take decisions based on incomplete measurements. You collect a bunch of data, elaborate an interpretation and make up your mind – sometimes you’ll be right, sometimes you’ll be proved wrong. This is what life is about, and also what science is about. Is the discussion around a research paper – sometimes in the form of peer review – any different? I don’t think so.

I hope that my comments are not misinterpreted; the overall experience that the author reported must have been frustrating, to say the least, and none of what I wrote is meant to deny that. Rather, my post is a first and coarse attempt at restoring bits of a broken link, if possible. Of course, I’d like any replies to match the tone of my post: I’m happy for my views to be challenged, as long as the discussion remains civil.

 

Advertisements

2 thoughts on “Comment: Publishing’s chinese whispers

  1. Hey, thanks for writing this post! As the original author, I guess the appropriate thing to do would be to respond given the time you’ve taken to write this. I’ll try and do it point by point in sync with your text.
    Regarding the frequency of err, sub-standard peer review, I didn’t mean to suggest this is a rule. I simply meant to ask the question, how many times does this happen and go unchecked? We’ll simply never know because of the ‘black box’ of peer review, and the fact that many referee reports remain unpublished for one reason or another.
    My apology to the referee was for suggesting that they didn’t do a thorough job. I don’t like stating such things publicly, or privately, but thought it was important in this case to emphasise the strength/weakness of the review process. It wasn’t meant to imply anything about the editorial process. I just don’t like saying bad things about people..
    You are right that anonymous does not equal closed (see my writing on the ScienceOpen blog for more of this, but note my post and this response are purely personal). For me, there are three levels of openness in peer review: the identity, the report itself, and who is allowed to participate. Each has a ‘closed’ version. My point here was that to people outside of the process, they have no idea what actually transpired. They have no idea what the peer review process was like, what the editorial decision was based on, or how I responded to any of this. The incompleteness here refers to the fact that only a single report was ever received by the Editor. This could have been biased, too strong, too weak, or simply not an accurate or detailed response that my work warranted. What if it was just the same thing, three paragraphs, but then with a reject decision? My career could have been indefinitely harmed for that. The reason why two or more referees are used is to strike a semblance of balance (I’ve had five for one of my papers before!). So in this case, I have a problem with making important decisions on what I feel is insufficient information.
    I hope that helps to clarify a few points! I’ll make sure to write more clearly in the future to avoid any potential confusion, so thank you again for writing this post 🙂

    Jon

    1. Although I’m late by several months, I would still like to thank protohedgehog for responding to my blog post (luckily I did so on Twitter back then, so I was only partially rude… ?!). I believe that a general conclusion to draw is that the clearer we all write the better! As for the specific episode discussed in my post and in protohedgehog’s reply, I certainly agree on one significant point – a first post-review editorial decision based on a single referee report does not make sense at all. I look forward to further discussions about publishing and good editorial practice!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s