• Open Access Science Articles
  • Science Sessions: The PNAS Podcast Program

Reviewer bias in single- versus double-blind peer review

  1. William D. Heavlina
  1. aGoogle, Inc., Mountain View, CA 94043;
  2. bState Key Laboratory of Intelligent Technology and Systems, Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China
  1. Edited by Susan T. Fiske, Princeton University, Princeton, NJ, and approved October 10, 2017 (received for review May 3, 2017)

Significance

Scientific peer review has been a cornerstone of the scientific method since the 1600s. Debate continues regarding the merits of single-blind review, in which anonymous reviewers know the authors of a paper and their affiliations, compared with double-blind review, in which this information is hidden. We present an experimental study of this question. In computer science, research often appears first or exclusively in peer-reviewed conferences rather than journals. Our study considers full-length submissions to the highly selective 2017 Web Search and Data Mining conference (15.6% acceptance rate). Each submission is simultaneously scored by two single-blind and two double-blind reviewers. Our analysis shows that single-blind reviewing confers a significant advantage to papers with famous authors and authors from high-prestige institutions.

Abstract

Peer review may be “single-blind,” in which reviewers are aware of the names and affiliations of paper authors, or “double-blind,” in which this information is hidden. Noting that computer science research often appears first or exclusively in peer-reviewed conferences rather than journals, we study these two reviewing models in the context of the 10th Association for Computing Machinery International Conference on Web Search and Data Mining, a highly selective venue (15.6% acceptance rate) in which expert committee members review full-length submissions for acceptance. We present a controlled experiment in which four committee members review each paper. Two of these four reviewers are drawn from a pool of committee members with access to author information; the other two are drawn from a disjoint pool without such access. This information asymmetry persists through the process of bidding for papers, reviewing papers, and entering scores. Reviewers in the single-blind condition typically bid for 22% fewer papers and preferentially bid for papers from top universities and companies. Once papers are allocated to reviewers, single-blind reviewers are significantly more likely than their double-blind counterparts to recommend for acceptance papers from famous authors, top universities, and top companies. The estimated odds multipliers are tangible, at 1.63, 1.58, and 2.10, respectively.

Footnotes

  • ?1To whom correspondence should be addressed. Email: atomkins{at}gmail.com.
  • An extended abstract of this work has been previously posted as a preprint (1).

  • Author contributions: A.T. and M.Z. designed research; A.T. and M.Z. performed research; W.D.H. analyzed data; and A.T., M.Z., and W.D.H. wrote the paper.

  • Conflict of interest statement: A.T. and W.D.H. are employed and paid by Google, Inc. Google often provides funding to conferences, including the WSDM conference studied in this work.

  • This article is a PNAS Direct Submission.

  • This article contains supporting information online at www.danielhellerman.com/lookup/suppl/doi:10.1073/pnas.1707323114/-/DCSupplemental.

Online Impact

  • 864971864 2018-01-22
  • 258841863 2018-01-22
  • 957295862 2018-01-22
  • 553518861 2018-01-22
  • 983792860 2018-01-22
  • 539694859 2018-01-22
  • 956115858 2018-01-22
  • 730379857 2018-01-22
  • 346624856 2018-01-22
  • 201609855 2018-01-22
  • 72549854 2018-01-21
  • 795928853 2018-01-21
  • 752345852 2018-01-21
  • 566508851 2018-01-21
  • 615722850 2018-01-21
  • 689612849 2018-01-21
  • 846903848 2018-01-21
  • 674896847 2018-01-21
  • 11197846 2018-01-21
  • 986896845 2018-01-21