• Call for Physical Sciences Papers
  • Science Sessions: The PNAS Podcast Program

Reviewer bias in single- versus double-blind peer review

  1. William D. Heavlina
  1. aGoogle, Inc., Mountain View, CA 94043;
  2. bState Key Laboratory of Intelligent Technology and Systems, Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
  1. Edited by Susan T. Fiske, Princeton University, Princeton, NJ, and approved October 10, 2017 (received for review May 3, 2017)

Significance

Scientific peer review has been a cornerstone of the scientific method since the 1600s. Debate continues regarding the merits of single-blind review, in which anonymous reviewers know the authors of a paper and their affiliations, compared with double-blind review, in which this information is hidden. We present an experimental study of this question. In computer science, research often appears first or exclusively in peer-reviewed conferences rather than journals. Our study considers full-length submissions to the highly selective 2017 Web Search and Data Mining conference (15.6% acceptance rate). Each submission is simultaneously scored by two single-blind and two double-blind reviewers. Our analysis shows that single-blind reviewing confers a significant advantage to papers with famous authors and authors from high-prestige institutions.

Abstract

Peer review may be “single-blind,” in which reviewers are aware of the names and affiliations of paper authors, or “double-blind,” in which this information is hidden. Noting that computer science research often appears first or exclusively in peer-reviewed conferences rather than journals, we study these two reviewing models in the context of the 10th Association for Computing Machinery International Conference on Web Search and Data Mining, a highly selective venue (15.6% acceptance rate) in which expert committee members review full-length submissions for acceptance. We present a controlled experiment in which four committee members review each paper. Two of these four reviewers are drawn from a pool of committee members with access to author information; the other two are drawn from a disjoint pool without such access. This information asymmetry persists through the process of bidding for papers, reviewing papers, and entering scores. Reviewers in the single-blind condition typically bid for 22% fewer papers and preferentially bid for papers from top universities and companies. Once papers are allocated to reviewers, single-blind reviewers are significantly more likely than their double-blind counterparts to recommend for acceptance papers from famous authors, top universities, and top companies. The estimated odds multipliers are tangible, at 1.63, 1.58, and 2.10, respectively.

Footnotes

  • ?1To whom correspondence should be addressed. Email: atomkins{at}gmail.com.
  • An extended abstract of this work has been previously posted as a preprint (1).

  • Author contributions: A.T. and M.Z. designed research; A.T. and M.Z. performed research; W.D.H. analyzed data; and A.T., M.Z., and W.D.H. wrote the paper.

  • Conflict of interest statement: A.T. and W.D.H. are employed and paid by Google, Inc. Google often provides funding to conferences, including the WSDM conference studied in this work.

  • This article is a PNAS Direct Submission.

  • This article contains supporting information online at www.danielhellerman.com/lookup/suppl/doi:10.1073/pnas.1707323114/-/DCSupplemental.

Online Impact

                                      1. 613261309 2018-02-21
                                      2. 6972481308 2018-02-21
                                      3. 2758991307 2018-02-21
                                      4. 5213301306 2018-02-21
                                      5. 6402651305 2018-02-21
                                      6. 975701304 2018-02-20
                                      7. 619701303 2018-02-20
                                      8. 6291841302 2018-02-20
                                      9. 8182271301 2018-02-20
                                      10. 7717531300 2018-02-20
                                      11. 2811781299 2018-02-20
                                      12. 9132041298 2018-02-20
                                      13. 285331297 2018-02-20
                                      14. 2838721296 2018-02-20
                                      15. 274321295 2018-02-20
                                      16. 2027431294 2018-02-20
                                      17. 2738641293 2018-02-20
                                      18. 9584601292 2018-02-20
                                      19. 9002021291 2018-02-20
                                      20. 7995901290 2018-02-20