ACM DL

Database Systems (TODS)

Menu

Search TODS
enter search term and/or author name


Social Media

Frequently-Asked Questions About Double-Blind Reviewing


(written by Richard T. Snodgrass, March 2007)

As announced in an editorial, ACM Transactions on Database Systems has adopted double-blind reviewing, in which the identities of the author and the reviewer are not known to each other. That editorial provided a comprehensive analysis of the costs of double-blind reviewing (DBR) and a detailed examination of the decision.

The present document is an easy-to-read summary. For those readers who prefer TLAs (three-letter acronyms), the title may be restated as a "DBR FAQ."

 

TODS Policy

The following is now TODS policy.

  • TODS reaffirms the general ACM policy that "the quality of a refereed publication rests primarily on the impartial judgment of their volunteer reviewers."
  • TODS will continue to strive to ensure fairness in reviewing, even if that involves more work for the TODS editorial board.
  • Scientific studies have demonstrated opportunities for bias inherent in single-blind reviewing.
  • It is TODS policy that every submission should be judged on its own merits. The identity and affiliation of the authors should not influence, either positively or negatively, the evaluation of submissions to TODS.
  • In consideration of the above, TODS will utilize double-blind reviewing.
  • TODS will continue to strive to make the submission process for authors as simple as possible.
  • TODS will continue to strive to effect a comprehensive review of each submission.

This policy is not dependent on absolute or even relative blinding efficacy. The central and unambiguous message is that every submission should be judged solely on its own merits. This message applies even when reviewers know exactly who the authors are. The other important message is that TODS so values fairness that it is willing to undertake additional effort by AEs to make the process more fair.

 

Frequently-Asked Questions by Authors

During the long gestation of this policy, several questions and concerns were raised. The guidelines provided on the TODS web page were designed to address these concerns.

Here, we provide responses to these questions to explain the policy, starting with questions from authors and then turning to questions from reviewers.

Details may be found in the author guidelines, included in the editorial and on the TODS web site.

  • Why double-blind reviewing?

    Double-blind reviewing has been shown through numerous scientific studies to be more fair. TODS goes to great lengths to ensure fairness in the review process. DBR is just one component of this.

  • What must I as an author do to blind my manuscript?

    Authors need only apply six simple steps to blind their submission.

    1. Anonymize the title page.
    2. Remove mention of funding sources and personal acknowledgments.
    3. Anonymize references found in running prose that cite your papers.
    4. Anonymize citations of submitted work in the bibliography.
    5. Ambiguate statements on well-known or unique systems that identify an author.
    6. Name your files with care and ensure document properties are also anonymized.
  • Can I submit an extension of a published conference paper?

    Yes, certainly. TODS continues to encourage such submissions, subject to the existing novelty and disclosure requirements listed in the author instructions on the TODS web site. In summary, the paper must have 30% new material, must explicitly describe the differences between the submission and the paper(s) it was extended from, and reference those papers, as anonymous citations.

  • Can I distribute a technical report version of my manuscript prior to submission?

    Yes, you can. There are no restrictions on the dissemination of drafts or technical reports prior to or after submission.

  • What about papers that are descriptions of software artifacts or describe well-known projects?

    Such papers continue to be encouraged. The anonymizing steps listed above have been designed to allow such papers to be submitted.

  • Will consideration of DBR adversely affect the quality of my prose?

    The paper should be first written with no concern for DBR. Then, right before the paper is submitted, it can be blinded by following those six steps, which will generally affect only a few sentences of your manuscript.

  • Won't reviewers be able to guess my identity anyway?

    Blinding efficacy (the percentage of papers for which blinding was successful, that is, that a reviewer could not determine the identity of an author) varies greatly, depending on a mix of factors. Certainly if the submission is an extension of a paper published in a prevalent conference, blinding efficacy may be considerably reduced. Note however that in the worst case, the process merely devolves back to single-blind reviewing, which has been the practice for many years. And even in that case, the blinded cover page still emphasizes to the reviewer that the identity of the author should not be taken into account.

    Hence, when the identity is not known or is uncertain, DBR affords the benefit of increased fairness. When the identity is known, we're back to the previous process, with the added reminder to the reviewer.

    The reviewing process in place at TODS explicitly favors comprehensiveness of the review over blinding efficacy, via a set of stated principles.

  • I'm a prolific author. Why shouldn't the reviewer know my identity? Won't DBR penalize me?

    This in my view is a source of much of the concern about DBR.

    There is a prevalent feeling among prolific authors that their name on a submitted manuscript increases the chance of that paper being accepted. The scientific evidence does not support that view. A review of the scientific studies concludes that "These contradictory results render it impossible to say anything definitive about the impact of blinding on prolific authors."

    So where did this perception come from? My guess is that prolific authors are more likely to write high quality papers (there is scientific evidence to support this); it is then the higher quality of their average submission rather than the identity of the author that increases the acceptance rate.

    If you are a prolific author (or even if you are not), you should have nothing to fear from DBR, because such a process focuses the reviewer on the quality of your submission.

  • Can I cite my own work?

    Yes, you should retain these citations. Steps 3 through 4 require such citations to be anonymized, again, following the guidelines on the TODS web site.

  • Will a reviewer finding out my identity cause my paper to be rejected?

    No. If you follow the six stated steps your manuscript will meet the anonymity requirements for submission.

  • Does DBR affect anything else?

    Yes: you need to include in your cover letter details on any anonymous citations and lists conflicts of interest. See the editorial or the TODS web site for details.

    The entire process of blinding your manuscript and preparing the cover letter should take only an hour or two.

 

Frequently-Asked Questions by Reviewers

  • What if I as a reviewer discover the identity of an author?

    If you as a reviewer accidentally discover the identity of an author, just continue the review. (Of course, you should not strive to unmask the paper.)

    It is TODS policy that every submission should be judged on its own merits. Your task is to review the submission, not the author.

  • What if I know an author personally?

    In that case, you should check the conflict of interest rules in the editorial or the TODS web site; these rules are identical to those for the US National Science Foundation. If you have any concerns, please contact the Associate Editor handling this submission.

  • How can I check previous work?

    If this work is included as an anonymous citation in the submitted manuscript and you need to check it, just contact the Associate Editor and request the full citation. Additionally, you are permitted to search the web and other sources if you feel that there is related work that has not been cited.

     

Summary

Over the past few years, TODS has adopted many innovative policies: a call for short papers and directed surveys, a limit of one review per year per reviewer, a guaranteed turn-around time of five months, full implementation of the ACM Rights and Responsibilities policy, and reviewing statistics published on its web site. Most of these innovations were at the time unique to TODS; some other journals are now following up.

These policies were somewhat controversial when first considered. What if the reviewer pool dried up given this promised limit? What if reviewers weren't responsive, causing the Editorial Board to violate its five-month guarantee? Over time, the community responded and everything worked out.

These past policies were enacted to increase fairness and quality. The double-blind policy furthers both of these objectives. The expectation is that over time the database community will become accustomed to the process and benefits of double-blind reviewing, as has occurred in other scientific communities.

 
All ACM Journals | See Full Journal Index