Thoughts on my first double-blind peer review

Not too long ago I agreed to review a paper after skimming the abstract and looking up the journal. When I went to actually do the review, I saw that the journal has a double-blind policy, and so I couldn’t see the names or affiliations of the authors and they couldn’t see mine. (The latter part here is standard practice for all but the “open review” journals.)

I’ve read about double-blind review, but never actually participated as author or reviewer before, so I got a little thrill when I realized I’d be participating. In theory, I really like the concept, because there is reason to believe that unconscious bias affects how reviewers review papers. After all, unconscious bias affects pretty much everything. Unfortunately, there’s not a lot of good hard evidence for reviewer unconscious bias, mostly because this issue has only come into the general scientific awareness relatively recently, and – I’ve read – it’s hard to actually do a good study. Rather than feel like, “no need to get excited about reviewer implicit bias until there’s evidence,” I think we ought to be particularly motivated to get some really solid studies done. Because if it’s a problem, then it’s a very serious one.

My particular views on implicit bias come from realizing my own. Several years ago, I took this awareness test (choose “Gender-Science IAT” from the list to do the one I did). And I started paying attention to my own thoughts. And I realized something unsettling. In early grad school, I frequently skipped the authors section when first reading a paper, because I pretty much didn’t know who anyone was and the names were meaningless to me. Then, if it was a paper I liked, I’d note the names of the authors, so I could remember them. If the first author’s name was female, I’d be surprised – like, “oh wow! Hawkes is a woman!” Because the thing is, when I read a science paper, the default narrator’s voice in my head is male. It just is. These days, I note authors’ names before reading, and frequently enough I know the authors or at least know of them. But every once in a while, I’m still surprised by author gender – almost invariably I’ve read a paper that lists only first initials and then come to find out that the author is female later in some other way.

Okay, so this is horribly embarrassing. I mean, I consider myself to be pretty free of gender stereotypes. I’m a self-described tomboy. I’ve spent most of my life in male-dominated activities doing male gendered things. My husband and I are all in on equal parenting and householding. I know tons of accomplished female scientists and other highly respected women in male-dominated fields. So what gives? Culture. I am simply a product of my culture – just like everyone else. And so if I’ve got anti-female implicit bias, I figure pretty much everyone else does, too.

Back to double-blind review. Does implicit bias matter when reviewing? Very possibly. We don’t know for sure yet. [1] But I think that a good study will look in the math or physics or engineering fields first, where power to detect such a bias is likely higher. Even if it doesn’t matter or matter very much, the perception that it matters is still affecting where people send manuscripts. So, at the very least, it matters indirectly.

How is double blind actually conducted? I imagine there are variations on a theme at different journals. Here’s what the journal I reviewed for did:

• It specified that the author(s) were “blinded” and didn’t provide their names or affiliations
• Oddly, it also “blinded” the Date Submitted, but not the Total Time in Review. (shrug)
• Within the text itself, someone at the journal had redacted some bits of text here and there and replaced it with (Removed by [journal]).

I found this curious, as I am no stranger to reading and writing words that can’t be seen by most people, and I know from first-hand experience that redacting a document is awfully time-consuming, tedious, and error-prone. It’s a task that no one enjoys. It’s expensive. And to be honest, I’m not sure how useful it is in the case of double-blind review.

I don’t know the authors of the study I reviewed. But I can easily guess the nationality of their institution – and even what part of the country they’re in. If I wanted to do a little googling, I’m pretty sure I could figure out who exactly they are. And I’ve read critiques saying that because double-blind often doesn’t really blind the reviewer to the authors’ identities, double-blind peer review is an exercise in futility.

But I’m not so sure. Let’s divide the relationships between author and reviewer into three categories. First, we have authors and reviewers who know one another personally or know one another’s work well. They may have collaborated at some point, or more likely, they just study the same sorts of things and so read one another’s papers a lot. They may meet at conferences and workshops because of their mutual interests. When a reviewer reads a paper by someone whose work (or whose lab’s work) they know, they’re likely to figure out the authors if the review is double-blind. But I’d argue that this is okay. The reviewer already has an impression of the author based on other experiences, and so implicit bias may not be an issue. [2] Proof is left as an exercise for the reader

Now let’s consider reviewers who don’t know the authors in real life, and have never even heard of them. Let’s say that, like in my case, there’s enough information in the manuscript that the reviewer could figure out the authors’ likely name(s) if they tried. My question would be: who would bother? I mean, who has the time to go sleuthing for names? [3] If you do have that time, could you maybe sign up to do a bit more reviewing instead of sleuthing? So, here double-blind works to counter implicit bias in that the reviewer still doesn’t know who the authors are, even though the blind has technically failed.

Finally, we have the straightforward case of a reviewer who doesn’t know the authors and in which it isn’t possible to tell from the manuscript who they are. In this case the blinding works, and there’s not much more to say.

Assuming that reviewers aren’t willing to go the extra mile to uncover author identities and that reviewers who can figure out who the authors are just by reading the manuscript already have an impression of the authors, making double-blind really simple might be just as good as having it be complex. The strategy would be this: Just don’t provide author names and affiliations. That’s it. Really simple to do. Really fast. Really inexpensive.

As for my review, I signed it, as I always do. [4] I think the retaliation fear is way overblown, and I’m happy to be a guinea pig. But I’m glad I got to try a blind review, as it modified my thoughts on the double-blind process.