Why this project on peer review?
As researchers, we spend a significant amount of time writing papers and refereeing for peer-reviewed journals. These activities are closely interconnected: if we want to improve how we produce research, it is essential that we better understand how we evaluate research.
Despite the central role that peer review plays in the production and evaluation of research, systematic efforts to understand the current system and improve it have remained limited:
- There is very little community discussion about what should be expected from the peer review process and how each group of stakeholders (authors, referees, editors, readers) might best benefit from it.
- Institutions in peer review provide very little systematic guidance or training about how to write constructive referee reports.
- Finally, statistics about the current volume and distribution of peer review activity across the global research community are generally unavailable.
The aim of this project is to take steps towards filling the gaps in our knowledge, with the hope that this will be the beginning of a dynamic conversation about peer review that enables the community to:
- Identify and discuss potential deficiencies and frictions in our current system.
- Brainstorm and evaluate practical proposals aimed at improving it.
What did we do?
To create the foundation for productive conversations about peer review, we undertook a comprehensive effort at documenting the current state of our system:
1) Survey:
- We designed and disseminated a survey taken by over 1,400 economists in order to (i) understand their experiences with peer review and (ii) collect opinions about potential proposals to improve the system.
- We also offered respondents the opportunity to voice their concerns and suggest their own proposals for improving the system.
- The curated dataset, survey material, and codes can be accessed on our OSF page.
2) Literature review:
- We reviewed the existing literature about peer review, drawing on sources from inside and outside of economics.
- We then built a (non-comprehensive) themed bibliography, which we make available under the "Bibliography" tab of this website.
- The bibliography will be periodically updated as additional resources relating to peer review become known to us.
3) Report:
- Based on our survey data, a review of peer review literature, and research about current journal policies, we wrote a report to document what we learned about the system and to present a list of over 160 proposals that could improve it.
- The full report can be found here, while a summary of results can be found under the "Report" tab.
Where do we go next?
Our project is meant to be a launchpad for discussion. As a result, we do not take an official stance on any proposal in the report and instead present them all for community discussion. To facilitate this essential dialogue, we have taken the following steps:
1) Forum:
- We set up a discussion forum so researchers can engage with these important topics and debate the merits and limitations of various proposals.
- Please participate in the discussions! They can be accessed under the "Discussion" tab. Information about the forum rules can be found in our FAQ.
2) Future additions to the project:
- As these efforts continue to be expanded, we will be happy to assist with the setup of working groups or discussion panels as well as with the coordination of research projects, such as additional surveys or randomized trials. We will centralize this information.
- Please get in touch with us if you would like to contribute in some way.
Clearly, the most important part of this project is you! To implement change, we need engagement from the economics community at large. We strongly encourage everyone to participate, in one way or another, so that the full diversity of views present in the profession can be represented.