What have we learned?

As the final outputs of the Reproducibility Project: Cancer Biology are published, it is clear that preclinical research in cancer biology is not as reproducible as it should be.

Challenges for assessing replicability in preclinical cancer biology.

We conducted the Reproducibility Project: Cancer Biology to investigate the replicability of preclinical research in cancer biology. The initial aim of the project was to repeat 193 experiments from 53 high-impact papers, using an approach in which the experimental protocols and plans for data analysis had to be peer reviewed and accepted for publication before experimental work could begin. However, the various barriers and challenges we encountered while designing and conducting the experiments meant that we were only able to repeat 50 experiments from 23 papers. Here we report these barriers and challenges. First, many original papers failed to report key descriptive and inferential statistics: the data needed to compute effect sizes and conduct power analyses was publicly accessible for just 4 of 193 experiments. Moreover, despite contacting the authors of the original papers, we were unable to obtain these data for 68% of the experiments. Second, none of the 193 experiments were described in sufficient detail in the original paper to enable us to design protocols to repeat the experiments, so we had to seek clarifications from the original authors. While authors were extremely or very helpful for 41% of experiments, they were minimally helpful for 9% of experiments, and not at all helpful (or did not respond to us) for 32% of experiments. Third, once experimental work started, 67% of the peer-reviewed protocols required modifications to complete the research and just 41% of those modifications could be implemented. Cumulatively, these three factors limited the number of experiments that could be repeated. This experience draws attention to a basic and fundamental concern about replication - it is hard to assess whether reported findings are credible.

Investigating the replicability of preclinical cancer biology.

Replicability is an important feature of scientific research, but aspects of contemporary research culture, such as an emphasis on novelty, can make replicability seem less important than it should be. The Reproducibility Project: Cancer Biology was set up to provide evidence about the replicability of preclinical research in cancer biology by repeating selected experiments from high-impact papers. A total of 50 experiments from 23 papers were repeated, generating data about the replicability of a total of 158 effects. Most of the original effects were positive effects (136), with the rest being null effects (22). A majority of the original effect sizes were reported as numerical values (117), with the rest being reported as representative images (41). We employed seven methods to assess replicability, and some of these methods were not suitable for all the effects in our sample. One method compared effect sizes: for positive effects, the median effect size in the replications was 85% smaller than the median effect size in the original experiments, and 92% of replication effect sizes were smaller than the original. The other methods were binary - the replication was either a success or a failure - and five of these methods could be used to assess both positive and null effects when effect sizes were reported as numerical values. For positive effects, 40% of replications (39/97) succeeded according to three or more of these five methods, and for null effects 80% of replications (12/15) were successful on this basis; combining positive and null effects, the success rate was 46% (51/112). A successful replication does not definitively confirm an original finding or its theoretical interpretation. Equally, a failure to replicate does not disconfirm a finding, but it does suggest that additional investigation is needed to establish its reliability.

Is preclinical research in cancer biology reproducible enough?

The Reproducibility Project: Cancer Biology (RPCB) was established to provide evidence about reproducibility in basic and preclinical cancer research, and to identify the factors that influence reproducibility more generally. In this commentary we address some of the scientific, ethical and policy implications of the project. We liken the basic and preclinical cancer research enterprise to a vast 'diagnostic machine' that is used to determine which clinical hypotheses should be advanced for further development, including clinical trials. The results of the RPCB suggest that this diagnostic machine currently recommends advancing many findings that are not reproducible. While concerning, we believe that more work needs to be done to evaluate the performance of the diagnostic machine. Specifically, we believe three questions remain unanswered: how often does the diagnostic machine correctly recommend against advancing real effects to clinical testing?; what are the relative costs to society of false positive and false negatives?; and how well do scientists and others interpret the outputs of the machine?

Pseudogenes, RNAs and new reproducibility norms.

The partial success of a study to reproduce experiments that linked pseudogenes and cancer proves that understanding RNA networks is more complicated than expected.

Confounds in "Failed" Replications.

Reproducibility is essential to science, yet a distressingly large number of research findings do not seem to replicate. Here I discuss one underappreciated reason for this state of affairs. I make my case by noting that, due to artifacts, several of the replication failures of the vastly advertised Open Science Collaboration's Reproducibility Project: Psychology turned out to be invalid. Although these artifacts would have been obvious on perusal of the data, such perusal was deemed undesirable because of its post hoc nature and was left out. However, while data do not lie, unforeseen confounds can render them unable to speak to the question of interest. I look further into one unusual case in which a major artifact could be removed statistically-the nonreplication of the effect of fertility on partnered women's preference for single over attached men. I show that the "failed replication" datasets contain a gross bias in stimulus allocation which is absent in the original dataset; controlling for it replicates the original study's main finding. I conclude that, before being used to make a scientific point, all data should undergo a minimal quality control-a provision, it appears, not always required of those collected for purpose of replication. Because unexpected confounds and biases can be laid bare only after the fact, we must get over our understandable reluctance to engage in anything post hoc. The reproach attached to p-hacking cannot exempt us from the obligation to (openly) take a good look at our data.

Replication Study: The microRNA miR-34a inhibits prostate cancer stem cells and metastasis by directly repressing CD44.

As part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Li et al., 2015), that described how we intended to replicate selected experiments from the paper 'The microRNA miR-34a inhibits prostate cancer stem cells and metastasis by directly repressing CD44' (Liu et al., 2011). Here we report the results. We found the microRNA, miR-34a, was expressed at twice the level in CD44+ prostate cancer cells purified from xenograft tumors (LAPC4 cells) compared to CD44- LAPC4 cells, whereas the original study reported miR-34a was underexpressed in CD44+ LAPC4 cells (Figure 1B; Liu et al., 2011). When LAPC4 cells engineered to express miR-34a were injected into mice, we did not observe changes in tumor growth or CD44 expression; however, unexpectedly miR-34a expression was lost in vivo. In the original study, LAPC4 cells expressing miR-34a had a statistically significant reduction in tumor regeneration and reduced CD44 expression compared to control (Figure 4A and Supplemental Figures 4A,B and 5C; Liu et al., 2011). Furthermore, when we tested if miR-34a regulated CD44 through binding sites in the 3'UTR we did not find a statistically significant difference, whereas the original study reported miR-34a decreased CD44 expression that was partially abrogated by mutation of the binding sites in the CD44 3'UTR (Figure 4D; Liu et al., 2011). Finally, where possible, we report meta-analyses for each result.

Replication Study: Wnt activity defines colon cancer stem cells and is regulated by the microenvironment.

As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Evans et al., 2015), that described how we intended to replicate selected experiments from the paper 'Wnt activity defines colon cancer stem cells and is regulated by the microenvironment' (Vermeulen et al., 2010). Here, we report the results. Using three independent primary spheroidal colon cancer cultures that expressed a Wnt reporter construct we observed high Wnt activity was associated with the cell surface markers CD133, CD166, and CD29, but not CD24 and CD44, while the original study found all five markers were correlated with high Wnt activity (Figure 2F; Vermeulen et al., 2010). Clonogenicity was highest in cells with high Wnt activity and clonogenic potential of cells with low Wnt activity were increased by myofibroblast-secreted factors, including HGF. While the effects were in the same direction as the original study (Figure 6D; Vermeulen et al., 2010) whether statistical significance was reached among the different conditions varied. When tested in vivo, we did not find a difference in tumorigenicity between high and low Wnt activity, while the original study found cells with high Wnt activity were more effective in inducing tumors (Figure 7E; Vermeulen et al., 2010). Tumorigenicity, however, was increased with myofibroblast-secreted factors, which was in the same direction as the original study (Figure 7E; Vermeulen et al., 2010), but not statistically significant. Finally, we report meta-analyses for each results where possible.

Rethinking research into metastasis.

The partial success of an attempt to repeat findings in cancer biology highlights the need to improve study designs for preclinical research into metastasis and the targeting of cancer cells.

Replication Study: Biomechanical remodeling of the microenvironment by stromal caveolin-1 favors tumor invasion and metastasis.

As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Fiering et al., 2015) that described how we intended to replicate selected experiments from the paper 'Biomechanical remodeling of the microenvironment by stromal caveolin-1 favors tumor invasion and metastasis' (Goetz et al., 2011). Here we report the results. Primary mouse embryonic fibroblasts (pMEFs) expressing caveolin 1 (Cav1WT) demonstrated increased extracellular matrix remodeling in vitro compared to Cav1 deficient (Cav1KO) pMEFs, similar to the original study (Goetz et al., 2011). In vivo, we found higher levels of intratumoral stroma remodeling, determined by fibronectin fiber orientation, in tumors from cancer cells co-injected with Cav1WT pMEFs compared to cancer cells only or cancer cells plus Cav1KO pMEFs, which were in the same direction as the original study (Supplemental Figure S7C; Goetz et al., 2011), but not statistically significant. Primary tumor growth was similar between conditions, like the original study (Supplemental Figure S7Ca; Goetz et al., 2011). We found metastatic burden was similar between Cav1WT and Cav1KO pMEFs, while the original study found increased metastases with Cav1WT (Figure 7C; Goetz et al., 2011); however, the duration of our in vivo experiments (45 days) were much shorter than in the study by Goetz et al. (2011) (75 days). This makes it difficult to interpret the difference between the studies as it is possible that the cells required more time to manifest the difference between treatments observed by Goetz et al. We also found a statistically significant negative correlation of intratumoral remodeling with metastatic burden, while the original study found a statistically significant positive correlation (Figure 7Cd; Goetz et al., 2011), but again there were differences between the studies in terms of the duration of the metastasis studies and the imaging approaches that could have impacted the outcomes. Finally, we report meta-analyses for each result.

Replication study: Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET.

As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Lesnik et al., 2016) that described how we intended to replicate selected experiments from the paper 'Melanoma exosomes educate bone marrow progenitor cells toward a pro-metastatic phenotype through MET' (Peinado et al., 2012). Here we report the results. We regenerated tumor cells stably expressing a short hairpin to reduce Met expression (shMet) using the same highly metastatic mouse melanoma cell line (B16-F10) as the original study, which efficiently downregulated Met in B16F10 cells similar to the original study (Supplementary Figure 5A; Peinado et al., 2012). Exosomes from control cells expressed Met, which was reduced in exosomes from shMet cells; however, we were unable to reliably detect phosphorylated Met in exosomes. We tested the effect of exosome-dependent Met signaling on primary tumor growth and metastasis. Similar to the results in the original study, we did not find a statistically significant change in primary tumor growth. Measuring lung and femur metastases, we found a small increase in metastatic burden with exosomes from control cells that was diminished when Met expression was reduced; however, while the effects were in the same direction as the original study (Figure 4E; Peinado et al., 2012), they were not statistically significant. Differences between the original study and this replication attempt, such as level of knockdown efficiency, cell line genetic drift, sample sizes, study endpoints, and variability of observed metastatic burden, are factors that might have influenced the outcomes. Finally, we report meta-analyses for each result.

Replication Study: Intestinal inflammation targets cancer-inducing activity of the microbiota.

As part of the Reproducibility Project: Cancer Biology we published a Registered Report (Eaton et al., 2015) that described how we intended to replicate selected experiments from the paper "Intestinal Inflammation Targets Cancer-Inducing Activity of the Microbiota" (Arthur et al., 2012). Here we report the results. We observed no impact on bacterial growth or colonization capacity when the polyketide synthase (pks) genotoxic island was deleted from E. coli NC101, similar to the original study (Supplementary Figure 7; Arthur et al., 2012). However, for the experiment that compared inflammation, invasion, and neoplasia in azoxymethane (AOM)-treated interleukin-10-deficient mice mono-associated with NC101 or NC101[Formula: see text] pks the experimental timing of the replication attempt was longer than that of the original study. This difference was because in the original study the methodology was not clearly stated and likely led to the increased mortality and severity of inflammation observed in this replication attempt. Additionally, early death occurred during AOM treatment with higher mortality observed in NC101[Formula: see text] pks mono-associated mice compared to NC101, which was in the same direction, but more severe than the original study (Suppleme1ntal Figure 10; Arthur et al., 2012). A meta-analysis suggests that mice mono-associated with NC101[Formula: see text] pks have higher mortality compared to NC101. While these data were unable to address whether, under the conditions of the original study, NC101 and NC101[Formula: see text] pks differ in inflammation, invasion, and neoplasia this replication attempt demonstrates that clear description of experimental methods is essential to ensure accurate reproduction of experimental studies.

Getting to grips with c-Myc.

The transcription factor c-Myc amplifies the transcription of many growth-related genes in cancer cells, but its role as an oncogene is not fully understood.

Replication Study: Systematic identification of genomic markers of drug sensitivity in cancer cells.

In 2016, as part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Vanden Heuvel et al., 2016), that described how we intended to replicate selected experiments from the paper 'Systematic identification of genomic markers of drug sensitivity in cancer cells' (Garnett et al., 2012). Here we report the results. We found Ewing's sarcoma cell lines, overall, were more sensitive to the PARP inhibitor olaparib than osteosarcoma cell lines; however, while the effect was in the same direction as the original study (Figure 4C; Garnett et al., 2012), it was not statistically significant. Further, mouse mesenchymal cells transformed with either the EWS-FLI1 or FUS-CHOP rearrangement displayed similar sensitivities to olaparib, whereas the Ewing's sarcoma cell line SK-N-MC had increased olaparib sensitivity. In the original study, mouse mesenchymal cells transformed with the EWS-FLI1 rearrangement and SK-N-MC cells were found to have similar sensitivities to olaparib, whereas mesenchymal cells transformed with the FUS-CHOP rearrangement displayed a reduced sensitivity to olaparib (Figure 4E; Garnett et al., 2012). We also studied another Ewing's sarcoma cell line, A673: A673 cells depleted of EWS-FLI1 or a negative control both displayed similar sensitivities to olaparib, whereas the original study reported a decreased sensitivity to olaparib when EWS-FLI1 was depleted (Figure 4F; Garnett et al., 2012). Differences between the original study and this replication attempt, such as the use of different sarcoma cell lines and level of knockdown efficiency, are factors that might have influenced the outcomes. Finally, where possible, we report meta-analyses for each result.

Replication Study: Transcriptional amplification in tumor cells with elevated c-Myc.

As part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Blum et al., 2015), that described how we intended to replicate selected experiments from the paper 'Transcriptional amplification in tumor cells with elevated c-Myc' (Lin et al., 2012). Here we report the results. We found overexpression of c-Myc increased total levels of RNA in P493-6 Burkitt's lymphoma cells; however, while the effect was in the same direction as the original study (Figure 3E; Lin et al., 2012), statistical significance and the size of the effect varied between the original study and the two different lots of serum tested in this replication. Digital gene expression analysis for a set of genes was also performed on P493-6 cells before and after c-Myc overexpression. Transcripts from genes that were active before c-Myc induction increased in expression following c-Myc overexpression, similar to the original study (Figure 3F; Lin et al., 2012). Transcripts from genes that were silent before c-Myc induction also increased in expression following c-Myc overexpression, while the original study concluded elevated c-Myc had no effect on silent genes (Figure 3F; Lin et al., 2012). Treating the data as paired, we found a statistically significant increase in gene expression for both active and silent genes upon c-Myc induction, with the change in gene expression greater for active genes compared to silent genes. Finally, we report meta-analyses for each result.

Replication Study: Fusobacterium nucleatum infection is prevalent in human colorectal carcinoma.

As part of the Reproducibility Project: Cancer Biology, we published a Registered Report (Repass et al., 2016), that described how we intended to replicate an experiment from the paper 'Fusobacterium nucleatum infection is prevalent in human colorectal carcinoma' (Castellarin et al., 2012). Here we report the results. When measuring Fusobacterium nucleatum DNA by qPCR in colorectal carcinoma (CRC), adjacent normal tissue, and separate matched control tissue, we did not detect a signal for F. nucleatum in most samples: 25% of CRCs, 15% of adjacent normal, and 0% of matched control tissue were positive based on quantitative PCR (qPCR) and confirmed by sequencing of the qPCR products. When only samples with detectable F. nucleatum in CRC and adjacent normal tissue were compared, the difference was not statistically significant, while the original study reported a statistically significant increase in F. nucleatum expression in CRC compared to adjacent normal tissue (Figure 2; Castellarin et al., 2012). Finally, we report a meta-analysis of the result, which suggests F. nucleatum expression is increased in CRC, but is confounded by the inability to detect F. nucleatum in most samples. The difference in F. nucleatum expression between CRC and adjacent normal tissues was thus smaller than the original study, and not detected in most samples.

The who, where and how of fusobacteria and colon cancer.

The association between the bacterium Fusobacterium nucleatum and human colon cancer is more complicated than it first appeared.

Making sense of replications.

The first results from the Reproducibility Project: Cancer Biology suggest that there is scope for improving reproducibility in pre-clinical cancer research.

The challenges of replication.

Interpreting the first results from the Reproducibility Project: Cancer Biology requires a highly nuanced approach.

Melanoma mystery.

Biological variability has confounded efforts to confirm the role of PREX2 mutations in melanoma.