A guide to post publication peer review

Interesting things are happening in science publishing. In the recent months we witnessed how multiple post publication peer review platforms were born and gained popularity. In this post I will try to clarify what is post publication peer review, what are the differences between existing platforms and what those changes mean for the future of publishing. But before I begin let me say a few words about how most publishing works right now.

We were always joking that we should start a new journal that would accept everything and use post publication peer review. To give it enough gravitas to compete with Nature and Science we decided to call it Truth. In the  picture (on the right) - a long time supporter of Truth - +Jonathan Smallwood .

Peer review before the intertubes

Say you are a scientist (if you are reading this it is very likely that you are, but let me try to make this topic understandable for people outside of academia). You have just discovered some exciting properties of a chemical molecule, or came up with a new theory of attention. You would like to share your findings with everyone else. You write a paper and submit it to a journal of your choice. The journal assigns an editor who tries to find two or three reviewers that read your work and look for methodological mistakes (statistics, experimental procedures) and flaws of reasoning. They can suggest changes, additional experiments. When/if they are happy with your manuscript it’s gets accepted, you open a bottle of sparkly and celebrate. Otherwise the paper gets rejected. This usually means that you will have to substantially change it or try again with another journal and another set of reviewers. The purpose of this system is quality control. Only “good” science should be able to pass the test of peer review. However there are some problems with this approach:

1. The acceptance of a paper depends to a great extent on luck. Depending who are your reviewers your work will be assessed differently. It is not unheard of that papers get rejected due to reviewers working under tight deadline and outside of their strict expertise. This of course can work both ways (poor papers being accepted, good papers being rejected).
2. The content of reviews is not available publicly.
3. After publication there is no easy way to voice your opinion about the the quality of a paper or potential problems with the described science.
4. In most cases reviewers remain anonymous and get no credit for doing the reviews.

To put it in more relatable terms: imagine that Amazon would only sell books selected by their internal reviewers. There would not be any way of posting your ratings or opinions about the books nor to see what internal Amazon reviewers thought about them. Total blackout.

Post publication peer review to the rescue

A simple solution to this problem is to provide a platform for public reviewing of academic papers after their publication. There is nothing fancy about this idea. It is basically what IMDB and Amazon have been doing for ages. There are, however, some issues that make academic papers different from films and books. Below I will describe three in my opinion most prominent post publication peer review platforms.

PubPeer.com

PubPeer.com is currently probably the most popular platform for post publication peer review. It has been featured in several media outlets and reviews posted on it lead to several retractions. PubPeer is very simple to use: you just need to put a DOI of a paper, register an account (you need to be either first or last author on an existing paper to do this) and you are ready to review. The platform praises anonymity as a necessary mechanism for unbiased reviews and this is the default way of posting reviews (although no one prevents you from signing your review). The anonymity (and lack of consequences towards researchers reputation/carrier) is probably the reason why PubPeer is so popular. Unfortunately this also means that 90% of the reviews are negative. If anyone bothers to write a review they do it just to point out flaws and criticize papers.

PubPeer is hosted by anonymous (who would have guessed…) group of people and is funded through their private resources. The authors have turned down donation offers in the past to avoid any accusations of conflict of interest.

Publons.com

“A Publon is a fundamental unit of academic publishing” we can read on their website. The idea behind the website is 180 degrees different from PubPeer. What Publons is focusing is giving reviewers credit for their work. Reviews are therefore sign with the real name of their author and if they get voted up (or endorsed) by the community they receive a DOI and can be cited as publications. Publons are also allowing you to submit the reviews you did for a journal (pre publication reviews). You don’t even have to make them public - Publons promises to verify your review with the journal to give you credit anyway. Reviews are (or very soon will be) indexed by Google Scholar which should increase their impact.

Publons is a startup run by some very enthusiastic and dedicated people from New Zealand. Their business plan includes helping journals finding the right reviewers, by building a platform for pre and post publication reviews. As with all startups it’s future is unclear. I would not be shocked if when it gains momentum it will be bought by one of the big publishers (the same thing happened to Mendeley and Frontiers).

PubMed Commons

PubMed is an NHI run library of biomedical literature indexing publications from many, many journals. Recently they started a closed (invitation only - get in touch if you want one) trial for putting comments under the abstracts of the papers they index. Comments have to be signed by your own name (again - verified through your publications), and essentially are nothing else than reviews. There are two advantages of PubMed Commons: i) your reviews will appear on the website visited by thousands of researchers every day, ii) the platform is run supported by US government money. There is one caveat though - PubMed is only for biomedical research. Sorry physics!

On the horizon

An honourable mention goes to Libre. It’s yet another post publication peer review platform, that is still under construction. Little is known about their policies and business model, but they published a super cute video describing the ideas behind it:

Too many cooks?

One may think that we don’t need so many platforms, that they only cause confusion. I disagree. I think it’s great that we have competition. It’s very important, especially in this early stage, to try different ideas. Some people will want to get credit for their reviews, some will prefer to be anonymous. The only problematic aspect is how to aggregate reviews from all platforms. This should not be a problem if all of the platforms provide publicly available APIs. Publons has one, PubMed Commons is working on one. Unfortunately, PubPeer, even though it has an API, gives access to it only to selected partners. In their view having multiple platforms is a bad thing, and they are willing to use the control over the access to their API to fight the competition. When I approached them with the idea of a review aggregator they denied me access to their API. Let’s hope they will change their minds in the future and open their API. That is what open science is really about.

So which one to choose?

Considering available options the decision which platform to use is not that hard. If you want your review to be anonymous use PubPeer. If you are not afraid to put your name under your review and you would like to get academic credit for it go for Publons. If you see the appeal of the impact that PubMed exposure gives go with the Commons.

The bigger picture: the future of publishing

Post publication peer review is just the beginning of the revolution. Soon the “golden” days of researcher giving and curating their papers for free to commercial publishers so they could sell them to other researchers will be gone. Papers will not have to go through random and arbitrary peer review behind closed doors. Researchers will self publish their work (see excellent blog post by +Micah Allen about self publishing) and rely on post publication peer review. Welcome to the brave new world!