• PNAS Teaching Resources Portal
  • Science Sessions: The PNAS Podcast Program

Reviewer bias in single- versus double-blind peer review

  1. William D. Heavlina
  1. aGoogle, Inc., Mountain View, CA 94043;
  2. bState Key Laboratory of Intelligent Technology and Systems, Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
  1. Edited by Susan T. Fiske, Princeton University, Princeton, NJ, and approved October 10, 2017 (received for review May 3, 2017)

Significance

Scientific peer review has been a cornerstone of the scientific method since the 1600s. Debate continues regarding the merits of single-blind review, in which anonymous reviewers know the authors of a paper and their affiliations, compared with double-blind review, in which this information is hidden. We present an experimental study of this question. In computer science, research often appears first or exclusively in peer-reviewed conferences rather than journals. Our study considers full-length submissions to the highly selective 2017 Web Search and Data Mining conference (15.6% acceptance rate). Each submission is simultaneously scored by two single-blind and two double-blind reviewers. Our analysis shows that single-blind reviewing confers a significant advantage to papers with famous authors and authors from high-prestige institutions.

Abstract

Peer review may be “single-blind,” in which reviewers are aware of the names and affiliations of paper authors, or “double-blind,” in which this information is hidden. Noting that computer science research often appears first or exclusively in peer-reviewed conferences rather than journals, we study these two reviewing models in the context of the 10th Association for Computing Machinery International Conference on Web Search and Data Mining, a highly selective venue (15.6% acceptance rate) in which expert committee members review full-length submissions for acceptance. We present a controlled experiment in which four committee members review each paper. Two of these four reviewers are drawn from a pool of committee members with access to author information; the other two are drawn from a disjoint pool without such access. This information asymmetry persists through the process of bidding for papers, reviewing papers, and entering scores. Reviewers in the single-blind condition typically bid for 22% fewer papers and preferentially bid for papers from top universities and companies. Once papers are allocated to reviewers, single-blind reviewers are significantly more likely than their double-blind counterparts to recommend for acceptance papers from famous authors, top universities, and top companies. The estimated odds multipliers are tangible, at 1.63, 1.58, and 2.10, respectively.

Footnotes

  • ?1To whom correspondence should be addressed. Email: atomkins{at}gmail.com.
  • An extended abstract of this work has been previously posted as a preprint (1).

  • Author contributions: A.T. and M.Z. designed research; A.T. and M.Z. performed research; W.D.H. analyzed data; and A.T., M.Z., and W.D.H. wrote the paper.

  • Conflict of interest statement: A.T. and W.D.H. are employed and paid by Google, Inc. Google often provides funding to conferences, including the WSDM conference studied in this work.

  • This article is a PNAS Direct Submission.

  • This article contains supporting information online at www.danielhellerman.com/lookup/suppl/doi:10.1073/pnas.1707323114/-/DCSupplemental.

Online Impact

                                      1. 99132880 2018-01-23
                                      2. 802899879 2018-01-23
                                      3. 295573878 2018-01-23
                                      4. 352668877 2018-01-23
                                      5. 984633876 2018-01-23
                                      6. 545928875 2018-01-23
                                      7. 976569874 2018-01-23
                                      8. 871324873 2018-01-23
                                      9. 263462872 2018-01-23
                                      10. 577161871 2018-01-23
                                      11. 255603870 2018-01-23
                                      12. 117346869 2018-01-23
                                      13. 90982868 2018-01-23
                                      14. 663415867 2018-01-23
                                      15. 793874866 2018-01-23
                                      16. 843582865 2018-01-23
                                      17. 864971864 2018-01-22
                                      18. 258841863 2018-01-22
                                      19. 957295862 2018-01-22
                                      20. 553518861 2018-01-22