> This past January, the cancer reproducibility project published its protocol for replicating the experiments, and the waiting began for Young to see whether his work will hold up in their hands. He says that if the project does match his results, it will be unsurprising—the paper's findings have already been reproduced. If it doesn't, a lack of expertise in the replicating lab may be responsible. Either way, the project seems a waste of time, Young says. “I am a huge fan of reproducibility. But this mechanism is not the way to test it.”
One swallow does not make a spring. With a belief like 'one replication is enough', I'm not sure Young actually appreciates how large sampling error is under usual significance levels or how high heterogeneity between labs is.
As far as I can tell from the article, Young's stance is that the contract lab doing the reproduction doesn't seem competent enough to correctly reproduce the experiment.
I don't know enough to know if that's true or not. But it seems at least possible that in the rush to reproduce much, the project is cutting costs by using less-skilled contract labs.
> Young's stance is that the contract lab doing the reproduction doesn't seem competent enough to correctly reproduce the experiment.
Yeah, but the contract lab doesn't need to be filled with geniuses (like Young perhaps) just to reproduce a now-mundane result from a few years ago, right? We're not talking about coming up with cutting-edge new experiments, just reproducing an old result using lab techniques that have probably become far less "cutting edge" in the intervening years...
I am highly suspicious of a claim that some experiment from a few years ago can still only be replicated by a tiny number of the top experts.
I am highly suspicious of a claim that some experiment from a few years ago can still only be replicated by a tiny number of the top experts.
I don't think you can find any claim on the order of "only a few top experts can do this..." in the article, the objections claimed to be against specific contract shops. As I understand things, biology experiments are still a matter of craft, of doing things by hand. Like any such thing (say, wood working or computer programming) there are going to be levels of skill involved with the lowest level of qualified people often being rather bad (for example lowest level contract programming shop). An analogy that springs to mind is whether one would want one's video game ported by the cheapest porting operation the business.
Of course, all this is me talking about what I can glean from the article compared to what others imagine the article says - until and unless someone with specific knowledge of the complaints and their validity or lack-there-of appears.
But as the op mentions, lab techniques vary and not everything can be documented. It's reasonable for a scientist to ask for a certain level in a lab attempting reproduction. I don't know if these scientists are demanding too much and have something to hide or if there are problem with the lab involved. However, the article actually mentions numerous scientists expressing misgivings.
A scientist should be able to describe all assumptions and manipulable variables that go into an experiment.
>It's reasonable for a scientist to ask for a certain level
Maybe, but a scientist should be able to specifically describe what that 'level' is.
Instead, this article seems to present many individuals who think appealing to an abstract 'level' of their own choosing and to which they seem to not be able to describe sufficiently to anyone else (which is fishy, because all of this serves, intentionally or not, to prevent anyone from attempting to replicate their experiment).
In my view of science, there should never be a case where 'misgivings' about a replication attempt should ever be expressed. Either the experiment will be accurately replicated or it won't, in which case we can probably determine if the replication was not accurate or if the original experiment was flawed (either in design, execution, and/or analysis).
Science gives us the tools to remove things from the influence and design of opinion and to examine them from viewpoints that are as free from subjectivity as possible. Once you start moving into the realm of opinion (expressed misgivings) you are leaving the realm of science and moving into philosophy, religion, or worse.
I disagree here, because the inevitable result of a highly visible series of failed reproductions is a big media hit. The scientific community may be able to poke holes in the reproduction attempt and sort through the damage, but the media and the court of public opinion certainly can't. Not until long after the reputations of possibly faultless scientists have been ruined irreparably.
So it's important for the reproduction attempts to be as high quality and rigorous as we would hope the original studies were. And it behooves scientists to make sure that these attempts are legitimate, unbiased and equitable, and to investigate any experimental flaws and biases of the experimenters before the results.
Misgivings about a reproduction attempt don't indicate denial of the validity of the scientific process, but recognition of the volatility of scientific news media. The unfortunate reality is that both sides of this effort, both original researchers and reproduction attempts, are subject to a great number of biases and restrictions. Subjective opinion does deeply affect the lives of scientists, and it's not possible for even the best scientists to live in a bubble of scientific purity and assume things will work out.
>I disagree here, because the inevitable result of a highly visible series of failed reproductions is a big media hit.
Interesting proclamation.
If a result can't be replicated after many 'highly visible' attempts then the result should be called into serious question.
>The scientific community may be able to poke holes in the reproduction attempt and sort through the damage, but the media and the court of public opinion certainly can't. Not until long after the reputations of possibly faultless scientists have been ruined irreparably.
This sounds like unwarranted fatalism to me. If the result was not reproduced because the experiment was not actually reproduced...I don't see what the issue is here.
>So it's important for the reproduction attempts to be as high quality and rigorous as we would hope the original studies were. And it behooves scientists to make sure that these attempts are legitimate, unbiased and equitable, and to investigate any experimental flaws and biases of the experimenters before the results.
This is why it is critical for any researcher that desires credibility (and more importantly: explanatory power) to detail their work accurately enough that someone else can exactly replicate their experiments in order to provide independent verification of their claims.
The best way to ensure that replication attempts are 'legitimate, unbiased and equitable' is to ensure your work is good enough that someone can actually (as opposed to merely attempting to) reproduce it.
>Misgivings about a reproduction attempt don't indicate denial of the validity of the scientific process, but recognition of the volatility of scientific news media.
Maybe.
>The unfortunate reality is that both sides of this effort, both original researchers and reproduction attempts, are subject to a great number of biases and restrictions. Subjective opinion does deeply affect the lives of scientists, and it's not possible for even the best scientists to live in a bubble of scientific purity and assume things will work out.
This sounds like something scientists need to work towards resolving.
"As far as I can tell from the article, Young's stance is that the contract lab doing the reproduction doesn't seem competent enough to correctly reproduce the experiment."
I'm suddenly having flashbacks, something about cold fusion and how groups with the right competence were indeed able to reproduce the results, for some value of "reproduce".
Agreed. I would actually be surprised if only a single lab had tried to reproduce a key result.
There may well be other labs which tried, and failed, to reproduce the original author's findings. In any event, it is unlikely we would know, as failure is unfortunately rarely publicized in academia (especially so when such failure is commonly dismissed out of hand for reasons such as a "lack of expertise").
I personally think the work this replication work is incredibly important and should absolutely be a key priority for funding institutions.
>I'm not sure Young actually appreciates how large sampling error is under usual significance levels or how high heterogeneity between labs is.
Rick Young is an MIT Professor, has been in biology research for 3 decades, is in the National Academies of Sciences and has published hundreds of high-impact papers across a huge number of disciplines. I know this doesn't mean he is God but to say that he doesn't understand "sampling errors" or "heterogeneity between labs" is an absurd insult.
One might think it's absurd, but nevertheless, that is what he is quoted as saying, and there are a lot of prestigious names on record as criticizing things like evidence-based medicine or replication. (You may remember a particularly asinine editorial not that long ago by a prestigious Harvard researcher arguing against any attempt to replicate studies.) I would no more be surprised to learn that Young genuinely thinks what he is quoted as arguing, than I would be to see, say, yet another misinterpretation of p-values in a peer-reviewed published research article or yet another article where the power is closer to 0% than 100% & the results near-meaningless or another study with scores of described hypothesis tests and no consideration of false-positive issues or any of the dreary litany of problems we all know about.
Being a published biology researcher does not make one an instant meta-analyst, qualified to say that there is no need for any additional replications (to say that sort of thing, you should be drawing on something like a trial sequential analysis indicating futility of additional studies, not saying 'well it seems to me like one (successful) replication of my result is enough and this second replication is a waste').
You are misinterpreting him; he's not saying that there are no more replications needed, he's saying that this method is not necessary.
And true, if another lab already got it going, then some random contract researcher getting it going or not is in no way definitive; if they do get it going, fine, and if they don't replicate, then it's not evidence of anything.
So what's the point? Isn't it better to use that same time and money to build a new experiment that has a higher upside rather than one that has a very limited upside?
> You are misinterpreting him; he's not saying that there are no more replications needed, he's saying that this method is not necessary.
I am not misinterpreting him at all. Look at the disjunctive argument he offers; both forks imply a lack of appreciation for sampling error and heterogeneity. (If the experiment is replicated, then far from being useless, the specifics will add substantial precision in the effect size and/or existence and also help estimating heterogeneity across labs attempting the procedure; and if not, then it 'may' be due to incompetence, but may also not be, in which case one has learned even more...)
> if they do get it going, fine, and if they don't replicate, then it's not evidence of anything.
I disagree, it is evidence. If the lab has an 80% chance of replicating correctly yadda yadda bayes theorem yadda yadda. Unless the contract lab is so incompetent that all things it does are purely random, their success or failure is evidence.
> Isn't it better to use that same time and money to build a new experiment that has a higher upside rather than one that has a very limited upside?
No. I don't know why you would think all experiments are of equal value and that replication is worthless. The more important an experiment is, the more important it is that it be replicated. The less important an experiment, the less important that anyone spend the resources replicating it. It can very easily be much better to replicate an experiment to gain additional confidence and/or precision. (See Bayesian search & optimization, decision theory, value of information, etc.) Imagine a medical clinical trial for a cancer treatment, would you really argue 'well, the first experiment turned in p<0.05, let's roll this sucker out to all the patients in the world! We could run more trials to make sure it works in more than one place and is worth the money and side-effects, but after all, isn't it better to use that same time and money to study new drugs, which have higher upsides rather than one with a very limited upside?'
One swallow does not make a spring. With a belief like 'one replication is enough', I'm not sure Young actually appreciates how large sampling error is under usual significance levels or how high heterogeneity between labs is.