In search of big breakthroughs: why attempts to predict ‘significant’ research might backfire

Marc Kirschner, professor of systems biology at Harvard Medical School
Marc Kirschner, professor of systems biology at Harvard Medical School
photo courtesy of Steve Lipofsky

When scientists vie for limited federal funding to support their research, they know their proposals will be evaluated based on scientific merits, but other factors come into play, too: a few years ago, the National Institutes of Health asked reviewers to evaluate the overall impact and significance of the work being proposed.

Those seem like incredibly salient qualities to evaluate before handing out funding. No one likes the idea of taxpayer money supporting trivial research.

But Harvard Medical School systems biologist Marc Kirschner has written a thoughtful, critical editorial arguing that those two criteria are “misleading and dangerous,” and “invite exaggerated claims of the importance of predictable outcomes—which are unlikely to be the most important ones.”

Advertisement - Continue Reading Below

Science is, by its nature, hard to predict, Kirschner points out. People run experiments because they don’t know which parts of their hypotheses may hold up and which may be incorrect. Moreover, a research result that seems obviously significant and transformative in retrospect, mayhave seemed like a narrow, niche area before the big breakthrough occurred.

A good example is the discovery of short snippets of RNA that can be used to turn off genes, which were found in part through experiments on microscopic worms; the work won the Nobel Prize in 2006. In short order, it was adopted as a powerful tool by researchers who are probing questions about human health in laboratories all across the world. Selecting research that seems likely to be significant ahead of time might mean passing on profound projects in favor of ones that are more predictable but less important, Kirschner argues.

Kirschner also questions the pervasive idea that a failure to translate basic science discoveries into medical advances is at the root of our inability to cure many diseases—he’s not sure that slow progress is simply due to researchers’ insights getting trapped at the laboratory bench.

But given the need to show a return on research investment and the necessity of choosing some projects over others, how should funding be awarded?

Kirschner answered questions by phone about his editorial, called “A Perverted View of ‘Impact,’” which was published Thursday in the journal Science.

Q: You note that significance is often taken to mean medical potential. Why is that misguided? What do you lose sight of when you focus on human health?

A: People at the NIH have told me ... they never meant significance means it has to be done with human cells or human beings or mammals or anything like that. They meant it much more broadly. But for reasons that bother them as much as anyone else, it’s being interpreted over and over again to mean direct relevance to human disease. I don’t think there’s anything wrong with saying there’s relevance to human disease. But it’s [become] very, very narrow. So for example, people will say, “You’ve done this experiment on a mouse or frog or drosophila [fruit fly] and that doesn’t mean anything.” ... People are being forced out of fields for no good reason.

Q: There’s a big push for “translational” medicine, and many research institutions, including the one where you work, have benefited. What’s wrong with this idea?

A: Wrong might be a strong word—maybe not well-supported. First of all, there’s this assumption that we know a lot, and we all want to cure disease, of course, and the missing link is our ability to translate it. And that kind of assumption makes a lot of people feel very comfortable. ... So you ask the question, is it really true?

I don’t think the pharmaceutical industry actually believes this; they think we need to know more about what to target and we need to know more about how drugs act in complex systems and we have to know more about the differences between humans and mice and other things on a more fundamental level. No one’s really made the case or proven it by any means that translation is the … rate-limiting feature. So that’s where I feel the argument is sort of very flimsy.

Q: What can be done to change the situation?

A: I guess in some sense, my editorial is really addressed to scientists, and I think the scientific community has to really stand up and challenge some of the assumptions being made instead of being so passive.

We have to look to ourselves to some degree and to the leadership of our own institutions, who I think have sort of failed us, by feeling as they are so pressed for money and seeing this as a way maybe they can get more support without really sort of defending the principles they should be defending.