Peer Review in a General Medical Research Journal Before and During the COVID-19 Pandemic.

Importance: Although peer review is an important component of publication for new research, the viability of this process has been questioned, particularly with the added stressors of the COVID-19 pandemic. Objective: To characterize rates of peer reviewer acceptance of invitations to review manuscripts, reviewer turnaround times, and editor-assessed quality of reviews before and after the start of the COVID-19 pandemic at a large, open-access general medical journal. Design, Setting, and Participants: This retrospective, pre-post cohort study examined all research manuscripts submitted to JAMA Network Open between January 1, 2019, and June 29, 2021, either directly or via transfer from other JAMA Network journals, for which at least 1 peer review of manuscript content was solicited. Measures were compared between the period before the World Health Organization declaration of a COVID-19 pandemic on March 11, 2020 (14.3 months), and the period during the pandemic (15.6 months) among all reviewed manuscripts and between pandemic-period manuscripts that did or did not address COVID-19. Main Outcomes and Measures: For each reviewed manuscript, the number of invitations sent to reviewers, proportions of reviewers accepting invitations, time in days to return reviews, and editor-assessed quality ratings of reviews were determined. Results: In total, the journal sought review for 5013 manuscripts, including 4295 Original Investigations (85.7%) and 718 Research Letters (14.3%); 1860 manuscripts were submitted during the prepandemic period and 3153 during the pandemic period. Comparing the prepandemic with the pandemic period, the mean (SD) number of reviews rated as high quality (very good or excellent) per manuscript increased slightly from 1.3 (0.7) to 1.5 (0.7) (P < .001), and the mean (SD) time for reviewers to return reviews was modestly shorter (from 15.8 [7.6] days to 14.4 [7.0] days; P < .001), a difference that persisted in linear regression models accounting for manuscript type, study design, and whether the manuscript addressed COVID-19. Conclusions and Relevance: In this cohort study, the speed and editor-reported quality of peer reviews in an open-access general medical journal improved modestly during the initial year of the pandemic. Additional study will be necessary to understand how the pandemic has affected reviewer burden and fatigue.

Real-World Trends in the Evaluation of Medical Products.

There is a compelling need to evaluate the real-world health effects of medical products outside of tightly controlled preapproval clinical trials. This is done through pharmacoepidemiology, which is the study of the health effects of medical products (including drugs, biologicals, and medical devices and diagnostics) in populations, often using nonrandomized designs. Recent developments in pharmacoepidemiology span changes in the focus of research questions, research designs, data used, and statistical analysis methods. Developments in these areas are thought to improve the value of the evidence produced by such studies, and are prompting greater use of real-world evidence to inform clinical, regulatory, and reimbursement decisions.

Advancing the Real-World Evidence for Medical Devices through Coordinated Registry Networks.

Objectives: Generating and using real-world evidence (RWE) is a pragmatic solution for evaluating health technologies. RWE is recognized by regulators, health technology assessors, clinicians, and manufacturers as a valid source of information to support their decision-making. Well-designed registries can provide RWE and become more powerful when linked with electronic health records and administrative databases in coordinated registry networks (CRNs). Our objective was to create a framework of maturity of CRNs and registries, so guiding their development and the prioritization of funding. Design setting and participants: We invited 52 stakeholders from diverse backgrounds including patient advocacy groups, academic, clinical, industry and regulatory experts to participate on a Delphi survey. Of those invited, 42 participated in the survey to provide feedback on the maturity framework for CRNs and registries. An expert panel reviewed the responses to refine the framework until the target consensus of 80% was reached. Two rounds of the Delphi were distributed via Qualtrics online platform from July to August 2020 and from October to November 2020. Main outcome measures: Consensus on the maturity framework for CRNs and registries consisted of seven domains (unique device identification, efficient data collection, data quality, product life cycle approach, governance and sustainability, quality improvement, and patient-reported outcomes), each presented with five levels of maturity. Results: Of 52 invited experts, 41 (79.9%) responded to round 1; all 41 responded to round 2; and consensus was reached for most domains. The expert panel resolved the disagreements and final consensus estimates ranged from 80.5% to 92.7% for seven domains. Conclusions: We have developed a robust framework to assess the maturity of any CRN (or registry) to provide reliable RWE. This framework will promote harmonization of approaches to RWE generation across different disciplines and health systems. The domains and their levels may evolve over time as new solutions become available.

Multi-institutional distributed data networks for real-world evidence about medical devices: building unique device identifiers into longitudinal data (BUILD).

Objectives: To support development of a robust postmarket device evaluation system using real-world data (RWD) from electronic health records (EHRs) and other sources, employing unique device identifiers (UDIs) to link to device information. Methods: To create consistent device-related EHR RWD across 3 institutions, we established a distributed data network and created UDI-enriched research databases (UDIRs) employing a common data model comprised of 24 tables and 472 fields. To test the system, patients receiving coronary stents between 2010 and 2019 were loaded into each institution's UDIR to support distributed queries without sharing identifiable patient information. The ability of the system to execute queries was tested with 3 quality assurance checks. To demonstrate face validity of the data, a retrospective survival study of patients receiving zotarolimus or everolimus stents from 2012 to 2017 was performed using distributed analysis. Propensity score matching was used to compare risk of 6 cardiovascular outcomes within 12 months postimplantation. Results: The test queries established network functionality. In the analysis, we identified 9141 patients (Mercy = 4905, Geisinger = 4109, Intermountain = 127); mean age 65 ± 12 years, 69% males, 23% zotarolimus. Separate matched analyses at the 3 institutions showed hazard ratio estimates (zotarolimus vs everolimus) of 0.85-1.59 for subsequent percutaneous coronary intervention (P = .14-.52), 1.06-2.03 for death (P = .16-.78) and 0.94-1.40 for the composite endpoint (P = .16-.62). Discussion: The analysis results are consistent with clinical studies comparing these devices. Conclusion: This project shows that multi-institutional data networks can provide clinically relevant real-world evidence via distributed analysis while maintaining data privacy.

Uncontrolled Extensions of Clinical Trials and the Use of External Controls-Scoping Opportunities and Methods.

Increased interest in real-world evidence (RWE) for clinical and regulatory decision making and the need to evaluate long-term benefits and risks of pharmaceutical products raise the importance of understanding the use of external controls (ECs) for uncontrolled extensions of randomized controlled trials (RCTs). We searched clinicaltrials.gov from 2009 to 2019 for uncontrolled extensions and assessed the use of ECs in the trial protocol registry and PubMed. We present characteristics of identified uncontrolled extensions, their adoption of ECs, and a qualitative appraisal of published uncontrolled extensions with ECs according to good pharmacoepidemiologic practice. The number of uncontrolled extensions increased slightly across the study period, resulting in a total of 1,115 studies. Most originated from phase III RCTs (62.2%) and specified safety outcomes (61.9% among those with specified outcomes). Most uncontrolled extensions incorporated no control group with only 7 out of 1,115 (0.6%) employing ECs. For those studies with ECs, all involved treatments for rare conditions and assessment of effectiveness. Attempts to balance comparison groups varied from none mentioned to propensity score matching. We noted consistent deficiencies in outcome ascertainment methods and approaches to address attrition bias. The contrast of the large and growing number of uncontrolled extensions with the small number of studies that utilized ECs showed clear opportunities for enhancement in design, measurement, and analysis of uncontrolled extensions to allow causal inferences on long-term treatment effects. As extensions continue to expand within RWE regulatory frameworks, development of guidelines for use of EC with uncontrolled extensions is needed.

Challenges of evaluating and modelling vaccination in emerging infectious diseases.

Outbreaks of emerging pathogens pose unique methodological and practical challenges for the design, implementation, and evaluation of vaccine efficacy trials. Lessons learned from COVID-19 highlight the need for innovative and flexible study design and application to quickly identify promising candidate vaccines. Trial design strategies should be tailored to the dynamics of the specific pathogen, location of the outbreak, and vaccine prototypes, within the regional socioeconomic constraints. Mathematical and statistical models can assist investigators in designing infectious disease clinical trials. We introduce key challenges for planning, evaluating, and modelling vaccine efficacy trials for emerging pathogens.

A systematic assessment of the epidemiologic literature regarding an association between acetaminophen exposure and cancer.

Introduced in the 1950s, acetaminophen is one of the most widely used antipyretics and analgesics worldwide. In 1999, the International Agency for Research on Cancer (IARC) reviewed the epidemiologic studies of acetaminophen and the data were judged to be "inadequate" to conclude that it is carcinogenic. In 2019 the California Office of Environmental Health Hazard Assessment initiated a review process on the carcinogenic hazard potential of acetaminophen. To inform this review process, the authors performed a comprehensive literature search and identified 136 epidemiologic studies, which for most cancer types suggest no alteration in risk associated with acetaminophen use. For 3 cancer types, renal cell, liver, and some forms of lymphohematopoietic, some studies suggest an increased risk; however, multiple factors unique to acetaminophen need to be considered to determine if these results are real and clinically meaningful. The objective of this publication is to analyze the results of these epidemiologic studies using a framework that accounts for the inherent challenge of evaluating acetaminophen, including, broad population-wide use in multiple disease states, challenges with exposure measurement, protopathic bias, channeling bias, and recall bias. When evaluated using this framework, the data do not support a causal association between acetaminophen use and cancer.

Signal Detection and Methodological Limitations in a Real-World Registry: Learnings from the Evaluation of Long-Term Safety Analyses in PSOLAR.

INTRODUCTION: Psoriasis Longitudinal Assessment and Registry (PSOLAR) was designed in 2007 as the first disease-based registry for patients with psoriasis. OBJECTIVE: The aim of this study was to discuss methodological limitations and post hoc analyses in long-term safety registries using learnings from analyses of a potential safety risk for major adverse cardiovascular events (MACE) in PSOLAR. METHODS: PSOLAR is an international observational study of over 12,000 psoriasis patients that was conducted to meet postmarketing safety commitments for infliximab and ustekinumab. A recent annual review of registry data indicated a potential MACE risk for ustekinumab vs. non-biologics based on prespecified COX model regression analyses, which yielded an adjusted hazard ratio (HR) of 1.533 (95% confidence interval [CI] 1.103-2.131). Therefore, we conducted a comprehensive review of key statistical methodology and implemented post hoc analytical methods to address specific limitations. RESULTS: The following limiting factors were identified: (1) inclusion of both prevalent and incident (new) users of biologics; (2) unanticipated imbalances in patient characteristics between treatment cohorts at baseline; (3) limited availability of relevant clinical data after enrollment; and (4) divergence of characteristics associated with outcomes among comparator groups over time. The analysis was modified to include only incident users, propensity scores were used to weight HRs, and adalimumab was deemed a more clinically appropriate comparator. The revised HR was 0.820 (95% CI 0.532-1.265), indicating no meaningful increase in MACE risk for ustekinumab. CONCLUSION: Our results, which do not support a causal association between ustekinumab exposure and MACE risk, underscore the need for ongoing assessment of analytical methods in long-term observational studies.

Comparative Effectiveness of Famotidine in Hospitalized COVID-19 Patients.

INTRODUCTION: Famotidine has been posited as a potential treatment for coronavirus disease 2019 (COVID-19). We compared the incidence of COVID-19 outcomes (i.e., death and death or intensive services use) among hospitalized famotidine users vs proton pump inhibitors (PPIs) users, hydroxychloroquine users, or famotidine nonusers separately. METHODS: We constructed a retrospective cohort study using data from COVID-19 Premier Hospital electronic health records. The study population was COVID-19 hospitalized patients aged 18 years or older. Famotidine, PPI, and hydroxychloroquine exposure groups were defined as patients dispensed any medication containing 1 of the 3 drugs on the day of admission. The famotidine nonuser group was derived from the same source population with no history of exposure to any drug with famotidine as an active ingredient before or on the day of admission. Time at risk was defined based on the intention-to-treat principle starting 1 day after admission to 30 days after admission. For each study comparison group, we fit a propensity score model through large-scale regularized logistic regression. The outcome was modeled using a survival model. RESULTS: We identified 2,193 users of PPI, 5,950 users of the hydroxychloroquine, 1,816 users of famotidine, and 26,820 nonfamotidine users. After propensity score stratification, the hazard ratios (HRs) for death were as follows: famotidine vs no famotidine HR 1.03 (0.89-1.18), vs PPIs: HR 1.14 (0.94-1.39), and vs hydroxychloroquine: 1.03 (0.85-1.24). Similar results were observed for the risk of death or intensive services use. DISCUSSION: We found no evidence of a reduced risk of COVID-19 outcomes among hospitalized COVID-19 patients who used famotidine compared with those who did not or compared with PPI or hydroxychloroquine users.

Quantifying bias in epidemiologic studies evaluating the association between acetaminophen use and cancer.

Many observational studies explore the association between acetaminophen and cancer, but known limitations such as vulnerability to channeling, protopathic bias, and uncontrolled confounding hamper the interpretability of results. To help understand the potential magnitude of bias, we identify key design choices in these observational studies and specify 10 study design variants that represent different combinations of these design choices. We evaluate these variants by applying them to 37 negative controls - outcome presumed not to be caused by acetaminophen - as well as 4 cancer outcomes in the Clinical Practice Research Datalink (CPRD) database. The estimated odds and hazards ratios for the negative controls show substantial bias in the evaluated design variants, with far fewer of the 95% confidence intervals containing 1 than the nominal 95% expected for negative controls. The effect-size estimates for the cancer outcomes are comparable to those observed for the negative controls. A comparison of exposed and unexposed reveals many differences at baseline for which most studies do not correct. We observe that the design choices made in many of the published observational studies can lead to substantial bias. Thus, caution in the interpretation of published studies of acetaminophen and cancer is recommended.

Methods for external control groups for single arm trials or long-term uncontrolled extensions to randomized clinical trials.

PURPOSE: Clinical trials compare outcomes among patients receiving study treatment with comparators drawn from the same source. These internal controls are missing in single arm trials and from long-term extensions (LTE) of trials including only the treatment arm. An external control group derived from a different setting is then required to assess safety or effectiveness. METHODS: We present examples of external control groups that demonstrate some of the issues that arise and make recommendations to address them through careful assessment of the data source fitness for use, design, and analysis steps. RESULTS: Inclusion and exclusion criteria and context that produce a trial population may result in trial patients with different clinical characteristics than are present in an external comparison group. If these differences affect the risk of outcomes, then a comparison of outcome occurrence will be confounded. Further, patients who continue into LTE may differ from those initially entering the trial due to treatment effects. Application of appropriate methods is needed to make valid inferences when such treatment or selection effects are present. Outcome measures in a trial may be ascertained and defined differently from what can be obtained in an external comparison group. Differences in sensitivity and specificity for identification or measurement of study outcomes leads to information bias that can also invalidate inferences. CONCLUSION: This review concentrates on threats to the valid use of external control groups both in the scenarios of single arm trials and LTE of randomized controlled trials, along with methodological approaches to mitigate them.

Learning from local to global: An efficient distributed algorithm for modeling time-to-event data.

OBJECTIVE: We developed and evaluated a privacy-preserving One-shot Distributed Algorithm to fit a multicenter Cox proportional hazards model (ODAC) without sharing patient-level information across sites. MATERIALS AND METHODS: Using patient-level data from a single site combined with only aggregated information from other sites, we constructed a surrogate likelihood function, approximating the Cox partial likelihood function obtained using patient-level data from all sites. By maximizing the surrogate likelihood function, each site obtained a local estimate of the model parameter, and the ODAC estimator was constructed as a weighted average of all the local estimates. We evaluated the performance of ODAC with (1) a simulation study and (2) a real-world use case study using 4 datasets from the Observational Health Data Sciences and Informatics network. RESULTS: On the one hand, our simulation study showed that ODAC provided estimates nearly the same as the estimator obtained by analyzing, in a single dataset, the combined patient-level data from all sites (ie, the pooled estimator). The relative bias was <0.1% across all scenarios. The accuracy of ODAC remained high across different sample sizes and event rates. On the other hand, the meta-analysis estimator, which was obtained by the inverse variance weighted average of the site-specific estimates, had substantial bias when the event rate is <5%, with the relative bias reaching 20% when the event rate is 1%. In the Observational Health Data Sciences and Informatics network application, the ODAC estimates have a relative bias <5% for 15 out of 16 log hazard ratios, whereas the meta-analysis estimates had substantially higher bias than ODAC. CONCLUSIONS: ODAC is a privacy-preserving and noniterative method for implementing time-to-event analyses across multiple sites. It provides estimates on par with the pooled estimator and substantially outperforms the meta-analysis estimator when the event is uncommon, making it extremely suitable for studying rare events and diseases in a distributed manner.

Designing, Conducting, Monitoring, and Analyzing Data from Pragmatic Randomized Clinical Trials: Proceedings from a Multi-stakeholder Think Tank Meeting.

In late 2018, the Food and Drug Administration (FDA) outlined a framework for evaluating the possible use of real-world evidence (RWE) to support regulatory decision-making. This framework was created to facilitate studies that would generate high-quality RWE, including pragmatic clinical trials (PCTs), which are randomized trials designed to inform clinical or policy decisions by assessing the real-world effectiveness of an intervention. There is general agreement among experts that the use of existing healthcare and patient-generated data holds promise for making randomized trials more efficient, less costly, and more generalizable. Yet the benefits of relying on real-world data sources must be weighed against difficulties with ensuring data integrity and completeness. Additionally, appropriately monitoring patient safety in randomized trials of new drugs using healthcare system data that might not be available in real time can be quite difficult. Recognizing that these and other concerns are critical to the development and acceptability of PCTs, a group of stakeholders from academia, industry, professional organizations, regulatory bodies, government agencies, and patient advocates discussed a path forward for PCT growth and sustainability at a think tank meeting entitled "Monitoring and Analyzing Data from Pragmatic Streamlined Randomized Clinical Trials," which took place in January 2019 (Washington, DC). The goals of this meeting were to: (1) evaluate study design and methodological options specific to PCTs that have the potential to yield high-quality evidence; (2) discuss best practices to ensure data quality in PCTs; and (3) identify appropriate methods for study monitoring. Proceedings from the think tank meeting are summarized in this manuscript.