Analysis preregistration gets a big boost in the form of a million dollars

The Center for Open Science has recently issued The $1,000,000 Preregistration Challenge. What is preregistration? Briefly, the idea is to reduce “researcher degrees of freedom.” Researchers often explore their data and then make data-processing and analytical choices based on the data that will be analyzed. But this can lead to biased results and incorrect scientific inference. Recent meta-studies have found a surprisingly high level of non-reproducibility in psychology and biomed, and this has no doubt spurred increased interest in techniques that can potentially make studies more reproducible. And preregistration is one such technique.

When I first heard of preregistration, I thought, “huh, well, sure, I guess in theory that would be a good thing to do.” But I also thought, “why would I do that? It’s quite a bit of extra work and doing so won’t benefit me in any way.” Like all of us, my time is limited and very valuable. I do cost-benefit analyses all the time on whether or not to do things. And preregistration wasn’t even close to the break-even line.

But this Preregistration Challenge may tip the scales for me. $1,000 is not to be scoffed at. And 1,000 recipients isn’t a small number of awards. I have a major new project coming up that I haven’t yet started analyses on. And the timeline for awards distribution seems quite reasonable. (In other words, the awards won’t all be won by projects that are fast and in disciplines that have fast publishing turn-around times.)

It’s pretty clear that for preregistration to become the norm, academic culture will need to change. And cultural change is hard. This challenge is a neat way to get a lot of people to experiment with a new practice that they likely wouldn’t have bothered with otherwise.

Oh wait.

Maybe I’ll take it all back. There’s a list of ‘eligible’ journals one can publish results in. And I assumed this list was just to vet journal repute — i.e. you can’t win if you publish in a journal that doesn’t actually do peer-review. But the list is actually journals that are open science friendly. And ecology journals are poorly represented on the list. Which is a problem, of course. But nudging ecology journals towards more open practices is not something I have the time or energy for. Boo. Maybe I’ll preregister and hope that by the time I publish, there are more ecology journals on the list.