Research rigor and reproducibility in research education: A CTSA institutional survey.

We assessed the rigor and reproducibility (R&R) activities of institutions funded by the National Center for Advancing Translational Sciences (NCTSA) through a survey and website search (N = 61). Of 50 institutional responses, 84% reported incorporating some form of R&R training, 68% reported devoted R&R training, 30% monitored R&R practices, and 10% incentivized them. Website searches revealed 9 (15%) freely available training curricula, and 7 (11%) institutional programs specifically created to enhance R&R. NCATS should formally integrate R&R principles into its translational science models and institutional requirements.

A New Look at P Values for Randomized Clinical Trials.

A New Look at P Values for Randomized Clinical TrialsUsing the primary results of 23,551 randomized clinical trials from the Cochrane Database, van Zwet et al. provide an empirical guide for the interpretation of an observed P value from a "typical" clinical trial in terms of the degree of overestimation of the reported effect, the probability of the effect's sign being wrong, and the predictive power of the trial.

Race and Ethnicity Reporting and Representation in Pediatric Clinical Trials.

BACKGROUND AND OBJECTIVES: Representative enrollment of racial and ethnic minoritized populations in biomedical research ensures the generalizability of results and equitable access to novel therapies. Previous studies on pediatric clinical trial diversity are limited to subsets of journals or disciplines. We aimed to evaluate race and ethnicity reporting and representation in all US pediatric clinical trials on ClinicalTrials.gov. METHODS: We performed a cross-sectional study of US-based clinical trials registered on ClinicalTrials.gov that enrolled participants aged <18 years old between October 2007 and March 2020. We used descriptive statistics, compound annual growth rates, and multivariable logistic regression for data analysis. Estimates of US population statistics and disease burden were calculated with the US Census, Kids' Inpatient Database, and National Survey of Children's Health. RESULTS: Among 1183 trials encompassing 405 376 participants, race and ethnicity reporting significantly increased from 27% in 2007 to 87% in 2018 (P < .001). The median proportional enrollment of Asian American children was 0.6% (interquartile range [IQR], 0%-3.7%); American Indian, 0% (IQR, 0%-0%); Black, 12% (IQR, 2.9%-28.4%); Hispanic, 7.1% (IQR, 0%-18.6%); and white 66.4% (IQR, 41.5%-81.6%). Asian American, Black, and Hispanic participants were underrepresented relative to US population demographics. Compared with expected proportions based on disease prevalence and hospitalizations, Asian American and Hispanic participants were most consistently underrepresented across diagnoses. CONCLUSIONS: While race and ethnicity reporting in pediatric clinical trials has improved, the representative enrollment of minoritized participants remains an ongoing challenge. Evidence-based and policy solutions are needed to address these disparities to advance biomedical innovation for all children.

The Impact of Intermittent Fasting on Patients With Suspected Gastroesophageal Reflux Disease.

GOAL: The aim was to investigate the short-term impact of time restricted feeding on patients with suspected gastroesophageal reflux disease (GERD). BACKGROUND: Lifestyle modifications are often suggested, but the role of diet in GERD is unclear. Intermittent fasting is popular in the media and has demonstrated potential benefits with weight loss and inflammatory conditions as well as alterations in gastrointestinal hormones. STUDY: Patients who were referred for 96-hour ambulatory wireless pH monitoring off proton pump inhibitor to investigate GERD symptoms were screened for eligibility. Patients were instructed to maintain their baseline diet for the first 2 days of pH monitoring and switch to an intermittent fasting regimen (16 consecutive hour fast and 8 h eating window) for the second 2 days. Objective measures of reflux and GERD symptom severity were collected and analyzed. RESULTS: A total of 25 participants were analyzed. 9/25 (36%) fully adhered to the intermittent fasting regimen, with 21/25 (84%) demonstrating at least partial compliance. Mean acid exposure time on fasting days was 3.5% versus 4.3% on nonfasting days. Intermittent fasting was associated with a 0.64 reduction in acid exposure time (95% CI: -2.32, 1.05). There was a reduction in GERD symptom scores of heartburn and regurgitation during periods of intermittent fasting (14.3 vs. 9.9; difference of -4.46, 95% CI: -7.6,-1.32). CONCLUSIONS: Initial adherence to time restricted eating may be difficult for patients. There is weak statistical evidence to suggest that intermittent fasting mildly reduces acid exposure. Our data show that short-term intermittent fasting improves symptoms of both regurgitation and heartburn.

Robustness of reported postacute health outcomes in children with SARS-CoV-2 infection: a systematic review.

OBJECTIVE: To systematically assess the robustness of reported postacute SARS-CoV-2 infection health outcomes in children. METHODS: A search on PubMed and Web of Science was conducted to identify studies published up to 22 January 2022 that reported on postacute SARS-CoV-2 infection health outcomes in children (<18 years) with follow-up of ≥2 months since detection of infection or ≥1 month since recovery from acute illness. We assessed the consideration of confounding bias and causality, as well as the risk of bias. RESULTS: 21 studies including 81 896 children reported up to 97 symptoms with follow-up periods of 2.0-11.5 months. Fifteen studies had no control group. The reported proportion of children with post-COVID syndrome was between 0% and 66.5% in children with SARS-CoV-2 infection (n=16 986) and between 2.0% and 53.3% in children without SARS-CoV-2 infection (n=64 910). Only two studies made a clear causal interpretation of an association between SARS-CoV-2 infection and the main outcome of 'post-COVID syndrome' and provided recommendations regarding prevention measures. The robustness of all 21 studies was seriously limited due to an overall critical risk of bias. CONCLUSIONS: The robustness of reported postacute SARS-CoV-2 infection health outcomes in children is seriously limited, at least in all the published articles we could identify. None of the studies provided evidence with reasonable certainty on whether SARS-CoV-2 infection has an impact on postacute health outcomes, let alone to what extent. Children and their families urgently need much more reliable and methodologically robust evidence to address their concerns and improve care.

How large should the next study be? Predictive power and sample size requirements for replication studies.

We use information derived from over 40K trials in the Cochrane Collaboration database of systematic reviews (CDSR) to compute the replication probability, or predictive power of an experiment given its observed (two-sided) P$$ P $$ -value. We find that an exact replication of a marginally significant result with P=.05$$ P=.05 $$ has less than 30% chance of again reaching significance. Moreover, the replication of a result with P=.005$$ P=.005 $$ still has only 50% chance of significance. We also compute the probability that the direction (sign) of the estimated effect is correct, which is closely related to the type S error of Gelman and Tuerlinckx. We find that if an estimated effect has P=.05$$ P=.05 $$ , there is a 93% probability that its sign is correct. If P=.005$$ P=.005 $$ , then that probability is 99%. Finally, we compute the required sample size for a replication study to achieve some specified power conditional on the p$$ p $$ -value of the original study. We find that the replication of a result with P=.05$$ P=.05 $$ requires a sample size more than 16 times larger than the original study to achieve 80% power, while P=.005$$ P=.005 $$ requires at least 3.5 times larger sample size. These findings confirm that failure to replicate the statistical significance of a trial does not necessarily indicate that the original result was a fluke.

Early Discontinuation, Results Reporting, and Publication of Pediatric Clinical Trials.

BACKGROUND AND OBJECTIVES: Unique ethical, epidemiological, and economic factors are barriers to performing research in children. The landscape of pediatric clinical trials, including drivers of completion and timely dissemination of results, is not well understood. We aimed to characterize the prevalence of and factors associated with early discontinuation, results reporting, and publication of pediatric clinical trials registered at ClinicalTrials.gov. METHODS: Cross-sectional analysis of clinical trials enrolling participants <18 years old registered at ClinicalTrials.gov from October 2007 to March 2020. Multivariable logistic regressions were performed to assess the association between trial characteristics and primary outcomes. Publication data were obtained through PubMed, ClinicalTrials.gov, Embase, and Scopus. RESULTS: Overall, 11.1% trials were stopped early, with recruitment failure being the predominant reason for discontinuation. Only 23.5% of completed trials reported results, and 38.8% were published within 3 years of completion. Rates of discontinuation and publication significantly improved over the study period. Among funding sources, government-sponsored trials (adjusted odds ratio [aOR], 0.72; 95% CI, 0.47-0.97) and academic trials (aOR, 0.64; 95% CI, 0.50-0.82) had lower odds of discontinuation compared with industry trials and were more likely to be published (government: aOR, 1.94 [95% CI, 1.52-2.48] academic: aOR, 1.61 [95% CI, 1.35-1.92). Academic trial investigators were the least likely to report results (aOR, 0.34; 95% CI, 0.31-0.52). CONCLUSIONS: Early discontinuation and nonreporting/nonpublication of findings remain common in registered pediatric clinical trials and were associated with funding source and other trial features. Targeted efforts are needed to support trial completion and timely results dissemination toward strengthening evidence-based pediatric medicine.

U.S. Food and Drug Administration Reasoning in Approval Decisions When Efficacy Evidence Is Borderline, 2013-2018.

BACKGROUND: The U.S. Food and Drug Administration (FDA) has substantial flexibility in its approval criteria in the context of life-threatening disease and unmet therapeutic need. OBJECTIVE: To understand the FDA's evidentiary standards when flexible criteria are employed. DESIGN: Case series. SETTING: Applications submitted between 2013 and 2018 that went through multiple review cycles because the evidence for clinical efficacy was initially deemed insufficient. MEASUREMENTS: Information was obtained from the approval package (available on Drugs@FDA), including advisory committee minutes, FDA reviews, and complete response letters. RESULTS: Of 912 applications reviewed, 117 went through multiple review cycles; only 22 of these faced additional review primarily because of issues related to clinical efficacy. Concerns about the end point, the clinical meaningfulness of the observed effect, and inconsistent results were common bases for initial rejection. In 7 of the 22 cases, the approval did not require new evidence but rather new interpretations of the original evidence. No FDA decisions cited reasoning used in previous decisions. LIMITATION: The conclusions rely on the authors' interpretation of the FDA statements and on a series of "close calls." CONCLUSION: The FDA has no mechanism to find or tradition to cite similar cases when weighing evidence for approvals, resulting in standalone, bespoke decisions. These decisions show highly variable criteria for "substantial evidence" when flexible evidential criteria are used, highlighted by the recent approval of aducanumab. A precedential tradition and suitable information system are required for the FDA to improve institutional memory and build upon past decisions. These would increase the FDA's decisional transparency, consistency, and predictability, which are critical to preserving the FDA's most valuable asset, the public's trust. PRIMARY FUNDING SOURCE: U.S. Food and Drug Administration.

Assessment of the Frequency and Variety of Persistent Symptoms Among Patients With COVID-19: A Systematic Review.

Importance: Infection with COVID-19 has been associated with long-term symptoms, but the frequency, variety, and severity of these complications are not well understood. Many published commentaries have proposed plans for pandemic control that are primarily based on mortality rates among older individuals without considering long-term morbidity among individuals of all ages. Reliable estimates of such morbidity are important for patient care, prognosis, and development of public health policy. Objective: To conduct a systematic review of studies examining the frequency and variety of persistent symptoms after COVID-19 infection. Evidence Review: A search of PubMed and Web of Science was conducted to identify studies published from January 1, 2020, to March 11, 2021, that examined persistent symptoms after COVID-19 infection. Persistent symptoms were defined as those persisting for at least 60 days after diagnosis, symptom onset, or hospitalization or at least 30 days after recovery from the acute illness or hospital discharge. Search terms included COVID-19, SARS-CoV-2, coronavirus, 2019-nCoV, long-term, after recovery, long-haul, persistent, outcome, symptom, follow-up, and longitudinal. All English-language articles that presented primary data from cohort studies that reported the prevalence of persistent symptoms among individuals with SARS-CoV-2 infection and that had clearly defined and sufficient follow-up were included. Case reports, case series, and studies that described symptoms only at the time of infection and/or hospitalization were excluded. A structured framework was applied to appraise study quality. Findings: A total of 1974 records were identified; of those, 1247 article titles and abstracts were screened. After removal of duplicates and exclusions, 92 full-text articles were assessed for eligibility; 47 studies were deemed eligible, and 45 studies reporting 84 clinical signs or symptoms were included in the systematic review. Of 9751 total participants, 5266 (54.0%) were male; 30 of 45 studies reported mean or median ages younger than 60 years. Among 16 studies, most of which comprised participants who were previously hospitalized, the median proportion of individuals experiencing at least 1 persistent symptom was 72.5% (interquartile range [IQR], 55.0%-80.0%). Individual symptoms occurring most frequently included shortness of breath or dyspnea (26 studies; median frequency, 36.0%; IQR, 27.6%-50.0%), fatigue or exhaustion (25 studies; median frequency, 40.0%; IQR, 31.0%-57.0%), and sleep disorders or insomnia (8 studies; median 29.4%, IQR, 24.4%-33.0%). There were wide variations in the design and quality of the studies, which had implications for interpretation and often limited direct comparability and combinability. Major design differences included patient populations, definitions of time zero (ie, the beginning of the follow-up interval), follow-up lengths, and outcome definitions, including definitions of illness severity. Conclusions and Relevance: This systematic review found that COVID-19 symptoms commonly persisted beyond the acute phase of infection, with implications for health-associated functioning and quality of life. Current studies of symptom persistence are highly heterogeneous, and future studies need longer follow-up, improved quality, and more standardized designs to reliably quantify risks.

Randomized COVID-19 vaccination rollout can offer direct real-world evidence.

Vaccines are vital to control the Coronavirus disease 2019 (COVID-19) pandemic, but the pressure to quickly move from research to implementation in the context of a pandemic crisis raises concerns about benefits and harms, vaccine acceptance and fair access. Here we present a strategy for the COVID-19 vaccination rollout which can be rapidly embedded and would offer direct real-world evidence of vaccines on a large scale to generate otherwise unobtainable knowledge on the safety and perhaps efficacy of COVID-19 vaccines. Such strategic rollouts leveraging randomization can provide important evidence, for COVID-19 and in future occasions, for vaccines and beyond.

Association of Convalescent Plasma Treatment With Clinical Outcomes in Patients With COVID-19: A Systematic Review and Meta-analysis.

Importance: Convalescent plasma is a proposed treatment for COVID-19. Objective: To assess clinical outcomes with convalescent plasma treatment vs placebo or standard of care in peer-reviewed and preprint publications or press releases of randomized clinical trials (RCTs). Data Sources: PubMed, the Cochrane COVID-19 trial registry, and the Living Overview of Evidence platform were searched until January 29, 2021. Study Selection: The RCTs selected compared any type of convalescent plasma vs placebo or standard of care for patients with confirmed or suspected COVID-19 in any treatment setting. Data Extraction and Synthesis: Two reviewers independently extracted data on relevant clinical outcomes, trial characteristics, and patient characteristics and used the Cochrane Risk of Bias Assessment Tool. The primary analysis included peer-reviewed publications of RCTs only, whereas the secondary analysis included all publicly available RCT data (peer-reviewed publications, preprints, and press releases). Inverse variance-weighted meta-analyses were conducted to summarize the treatment effects. The certainty of the evidence was assessed using the Grading of Recommendations Assessment, Development, and Evaluation. Main Outcomes and Measures: All-cause mortality, length of hospital stay, clinical improvement, clinical deterioration, mechanical ventilation use, and serious adverse events. Results: A total of 1060 patients from 4 peer-reviewed RCTs and 10 722 patients from 6 other publicly available RCTs were included. The summary risk ratio (RR) for all-cause mortality with convalescent plasma in the 4 peer-reviewed RCTs was 0.93 (95% CI, 0.63 to 1.38), the absolute risk difference was -1.21% (95% CI, -5.29% to 2.88%), and there was low certainty of the evidence due to imprecision. Across all 10 RCTs, the summary RR was 1.02 (95% CI, 0.92 to 1.12) and there was moderate certainty of the evidence due to inclusion of unpublished data. Among the peer-reviewed RCTs, the summary hazard ratio was 1.17 (95% CI, 0.07 to 20.34) for length of hospital stay, the summary RR was 0.76 (95% CI, 0.20 to 2.87) for mechanical ventilation use (the absolute risk difference for mechanical ventilation use was -2.56% [95% CI, -13.16% to 8.05%]), and there was low certainty of the evidence due to imprecision for both outcomes. Limited data on clinical improvement, clinical deterioration, and serious adverse events showed no significant differences. Conclusions and Relevance: Treatment with convalescent plasma compared with placebo or standard of care was not significantly associated with a decrease in all-cause mortality or with any benefit for other clinical outcomes. The certainty of the evidence was low to moderate for all-cause mortality and low for other outcomes.

The worldwide clinical trial research response to the COVID-19 pandemic - the first 100 days.

Background: Never before have clinical trials drawn as much public attention as those testing interventions for COVID-19. We aimed to describe the worldwide COVID-19 clinical research response and its evolution over the first 100 days of the pandemic. Methods: Descriptive analysis of planned, ongoing or completed trials by April 9, 2020 testing any intervention to treat or prevent COVID-19, systematically identified in trial registries, preprint servers, and literature databases. A survey was conducted of all trials to assess their recruitment status up to July 6, 2020. Results: Most of the 689 trials (overall target sample size 396,366) were small (median sample size 120; interquartile range [IQR] 60-300) but randomized (75.8%; n=522) and were often conducted in China (51.1%; n=352) or the USA (11%; n=76). 525 trials (76.2%) planned to include 155,571 hospitalized patients, and 25 (3.6%) planned to include 96,821 health-care workers. Treatments were evaluated in 607 trials (88.1%), frequently antivirals (n=144) or antimalarials (n=112); 78 trials (11.3%) focused on prevention, including 14 vaccine trials. No trial investigated social distancing. Interventions tested in 11 trials with >5,000 participants were also tested in 169 smaller trials (median sample size 273; IQR 90-700). Hydroxychloroquine alone was investigated in 110 trials. While 414 trials (60.0%) expected completion in 2020, only 35 trials (4.1%; 3,071 participants) were completed by July 6. Of 112 trials with detailed recruitment information, 55 had recruited <20% of the targeted sample; 27 between 20-50%; and 30 over 50% (median 14.8% [IQR 2.0-62.0%]). Conclusions: The size and speed of the COVID-19 clinical trials agenda is unprecedented. However, most trials were small investigating a small fraction of treatment options. The feasibility of this research agenda is questionable, and many trials may end in futility, wasting research resources. Much better coordination is needed to respond to global health threats.

How often do leading biomedical journals use statistical experts to evaluate statistical methods? The results of a survey.

Scientific claims in biomedical research are typically derived from statistical analyses. However, misuse or misunderstanding of statistical procedures and results permeate the biomedical literature, affecting the validity of those claims. One approach journals have taken to address this issue is to enlist expert statistical reviewers. How many journals do this, how statistical review is incorporated, and how its value is perceived by editors is of interest. Here we report an expanded version of a survey conducted more than 20 years ago by Goodman and colleagues (1998) with the intention of characterizing contemporary statistical review policies at leading biomedical journals. We received eligible responses from 107 of 364 (28%) journals surveyed, across 57 fields, mostly from editors in chief. 34% (36/107) rarely or never use specialized statistical review, 34% (36/107) used it for 10-50% of their articles and 23% used it for all articles. These numbers have changed little since 1998 in spite of dramatically increased concern about research validity. The vast majority of editors regarded statistical review as having substantial incremental value beyond regular peer review and expressed comparatively little concern about the potential increase in reviewing time, cost, and difficulty identifying suitable statistical reviewers. Improved statistical education of researchers and different ways of employing statistical expertise are needed. Several proposals are discussed.

Association of Rapid Eye Movement Sleep With Mortality in Middle-aged and Older Adults.

Importance: Rapid eye movement (REM) sleep has been linked with health outcomes, but little is known about the relationship between REM sleep and mortality. Objective: To investigate whether REM sleep is associated with greater risk of mortality in 2 independent cohorts and to explore whether another sleep stage could be driving the findings. Design, Setting, and Participants: This multicenter population-based cross-sectional study used data from the Outcomes of Sleep Disorders in Older Men (MrOS) Sleep Study and Wisconsin Sleep Cohort (WSC). MrOS participants were recruited from December 2003 to March 2005, and WSC began in 1988. MrOS and WSC participants who had REM sleep and mortality data were included. Analysis began May 2018 and ended December 2019. Main Outcomes and Measures: All-cause and cause-specific mortality confirmed with death certificates. Results: The MrOS cohort included 2675 individuals (2675 men [100%]; mean [SD] age, 76.3 [5.5] years) and was followed up for a median (interquartile range) of 12.1 (7.8-13.2) years. The WSC cohort included 1386 individuals (753 men [54.3%]; mean [SD] age, 51.5 [8.5] years) and was followed up for a median (interquartile range) of 20.8 (17.9-22.4) years. MrOS participants had a 13% higher mortality rate for every 5% reduction in REM sleep (percentage REM sleep SD = 6.6%) after adjusting for multiple demographic, sleep, and health covariates (age-adjusted hazard ratio, 1.12; fully adjusted hazard ratio, 1.13; 95% CI, 1.08-1.19). Results were similar for cardiovascular and other causes of death. Possible threshold effects were seen on the Kaplan-Meier curves, particularly for cancer; individuals with less than 15% REM sleep had a higher mortality rate compared with individuals with 15% or more for each mortality outcome with odds ratios ranging from 1.20 to 1.35. Findings were replicated in the WSC cohort despite younger age, inclusion of women, and longer follow-up (hazard ratio, 1.17; 95% CI, 1.03-1.34). A random forest model identified REM sleep as the most important sleep stage associated with survival. Conclusions and Relevance: Decreased percentage REM sleep was associated with greater risk of all-cause, cardiovascular, and other noncancer-related mortality in 2 independent cohorts.

The high resource impact of reformatting requirements for scientific papers.

BACKGROUND: Most research manuscripts are not accepted for publication on first submission. A major part of the resubmission process is reformatting to another journal's specific requirements, a process separate from revising the scientific content. There has been little research to understand the magnitude of the burden imposed by the current resubmission process. METHODS: We analyzed original research article submission requirements from twelve randomly selected journals in each of eight scientific and clinical focus areas from the InCites Journal Citation Reports database. From the 96 journals selected, we randomly identified three recently published manuscripts and sent surveys to those first and/or corresponding authors (288 total) to solicit information on time spent reformatting resubmissions and opinions on the process. FINDINGS: There was significant variation in manuscript submission requirements for journals within the same scientific focus and only 4% of journals offered a fully format-free initial submission. Of 203 authors responding (71.5% response rate), only 11.8% expressed satisfaction with the resubmission process and 91% desired reforming the current system. Time spent on reformatting delays most publications by at least two weeks and by over three months in about 20% of manuscripts. The effort to comply with submission requirements has significant global economic burden, estimated at over $1.1 billion dollars annually when accounting for a research team's time. INTERPRETATION: We demonstrate that there is significant resource utilization associated with resubmitting manuscripts, heretofore not properly quantified. The vast majority of authors are not satisfied with the current process. Addressing these issues by reconciling reformatting requirements among journals or adopting a universal format-free initial submission policy would help resolve a major subject for the scientific research community and provide more efficient dissemination of findings.