Sometimes when you say something on Twitter people respond. People don’t respond that much to what I have to say but, now and then, there’s enough of a reaction to help me realize that an idea was meaningful beyond the moment it popped into my mind and made its way onto the keyboard. So, thanks to those people for starting the conversation.
The idea I had today is that some scientific disciplines could benefit from more replication. And what better way to do it than to have Big Brother audit your science and see if they can replicate in their lab what you did in yours. The idea stemmed from my own feelings about my field. I’ve had serious thoughts lately about trying to replicate a couple findings that have had a lot of influence in the field. They’re important findings. They reveal key functions of new neurons that could be relevant for human health. For this reason the whole field is aware of them, cites them, uses them as justification for additional research. Sooo, then why haven’t they been replicated?
Pretty much since day one (of my science career) it’s been clear that fascinating studies appear and steer the entire field, usually in a good way. However, sometimes they’re such a perfect story that they go unquestioned or untested. And so, in thinking about contemporary examples, I’ve realized that I would really like to replicate them myself. Just to see, to know. Not because I don’t believe them, not to build on them (necessarily), but because I think it’s important to know whether it’s really the case, whether confirming or failing to confirm. I think science is ripe for this, given journals like PLoS ONE, whose aim is not to provide “interesting” science, but valid and useful science. And even if not in a traditional publication, there are new tools like Figshare that allow for datasets of all sizes to be archived, shared and cited with persistent identifiers.
But can you make a living by replicating studies? I thought about how lucky I am that I could pursue a project (mini-project?) whose aim is simply to replicate others’ work. This might not be as easy when I get my own lab. Do such grants exist?
That’s when NIRDY came to mind, the National Institute of Replicating Discoveries Y’all. This is a non-existent governmental research institution (NEGRI) that replicates a sampling of important scientific studies. Instead of only funding research, shouldn’t the government go back and doublecheck? Wouldn’t it be funny if the government couldn’t replicate any of the research they fund? Wouldn’t it be great if they could?
Factors to be considered when creating NIRDY
- How would they pick studies to replicate? Well, they’d be smart, first off. Maybe they could poll scientific communities, see which studies are getting cited the most, discussed the most on social networks, are immediately important for human health. Or go after everything published in Nature or Science.
- What effect would this have on future publications? Would scientists be more careful when publishing exciting results, to ensure validity?
- If NIRDY is ever born, can I get a job? Seriously, some of us get a kick out of systematically manipulating dozens of variables to figure out the exact conditions that are required to observe a scientific phenomenon. This relates to the question of how much replication must occur. One answer might be: until the original finding is replicated. Replication could happen on the first try. If not, maybe additional experiments are worthwhile. This was a problem with the role of neurogenesis in fear conditioning – very confusing until it was recently totally solved (though maybe I should replicate the entire study just to make sure).
- It can be very hard to replicate findings even with relatively standard techniques. Then what do we do about experiments where the technology is so advanced that very few labs are capable of replicating them? Sounds like a job for NIRDY.
- This could also all be good for reagent-sharing. I can think of a number of classic papers that used fancy mice and in the years, nay decades, since I can’t recall ever seeing those mice used by another group or in another paper.
And what happens if a study simply cannot be replicated?
Update: Alex Wiltschko had an interesting point, that replication could be a useful method for training scientists (or driving them batty):
In terms of career logistics, this makes a lot of sense for techs and postdocs, who get to thoroughly learn several state-of-the-art techniques during an experiment replication.