Real-World Safety Analysis of Paclitaxel Devices Used for the Treatment of Peripheral Artery Disease.

PURPOSE: To investigate the real-world safety of paclitaxel (PTX)-coated devices for treating lower extremity peripheral artery disease using a commercial claims database. MATERIALS AND METHODS: Data from FAIR Health, the largest commercial claims data warehouse in the United States, were used for this study. The study consisted of patients who underwent femoropopliteal revascularization procedures between January 1, 2015, and December 31, 2019, with PTX and non-PTX devices. The primary outcome was 4-year survival following treatment. The secondary outcomes included 2-year survival, 2- and 4-year freedom from amputation, and repeat revascularization. Propensity score matching was used to minimize confounding, and the Kaplan-Meier methods were used to estimate survival. RESULTS: A total of 10,832 procedures were included in the analysis, including 4,962 involving PTX devices and 5,870 involving non-PTX devices. PTX devices were associated with a reduced hazard of death following treatment at 2 and 4 years (hazard ratio [HR], 0.74 [95% confidence interval {CI}, 0.69-0.79]; P <.05, and HR, 0.89 [95% CI, 0.77-1.02]; log-rank P =.018, respectively). The risk of amputation was also lower following treatment with PTX devices than with non-PTX devices at 2 and 4 years (HR, 0.82 [95% CI, 0.76-0.87]; P =.02, and HR, 0.77 [95% CI, 0.67-0.89]; log-rank P =.01, respectively). In addition, the odds of repeat revascularization were similar with PTX and non-PTX devices at 2 and 4 years. CONCLUSIONS: In the real-world commercial claims database, no short- or long-term signal for increased mortality or amputations was observed following treatment with PTX devices.

Simulating complex patient populations with hierarchical learning effects to support methods development for post-market surveillance.

BACKGROUND: Validating new algorithms, such as methods to disentangle intrinsic treatment risk from risk associated with experiential learning of novel treatments, often requires knowing the ground truth for data characteristics under investigation. Since the ground truth is inaccessible in real world data, simulation studies using synthetic datasets that mimic complex clinical environments are essential. We describe and evaluate a generalizable framework for injecting hierarchical learning effects within a robust data generation process that incorporates the magnitude of intrinsic risk and accounts for known critical elements in clinical data relationships. METHODS: We present a multi-step data generating process with customizable options and flexible modules to support a variety of simulation requirements. Synthetic patients with nonlinear and correlated features are assigned to provider and institution case series. The probability of treatment and outcome assignment are associated with patient features based on user definitions. Risk due to experiential learning by providers and/or institutions when novel treatments are introduced is injected at various speeds and magnitudes. To further reflect real-world complexity, users can request missing values and omitted variables. We illustrate an implementation of our method in a case study using MIMIC-III data for reference patient feature distributions. RESULTS: Realized data characteristics in the simulated data reflected specified values. Apparent deviations in treatment effects and feature distributions, though not statistically significant, were most common in small datasets (n < 3000) and attributable to random noise and variability in estimating realized values in small samples. When learning effects were specified, synthetic datasets exhibited changes in the probability of an adverse outcomes as cases accrued for the treatment group impacted by learning and stable probabilities as cases accrued for the treatment group not affected by learning. CONCLUSIONS: Our framework extends clinical data simulation techniques beyond generation of patient features to incorporate hierarchical learning effects. This enables the complex simulation studies required to develop and rigorously test algorithms developed to disentangle treatment safety signals from the effects of experiential learning. By supporting such efforts, this work can help identify training opportunities, avoid unwarranted restriction of access to medical advances, and hasten treatment improvements.

Registry-Based Prospective, Active Surveillance of Medical-Device Safety.

BACKGROUND: The process of assuring the safety of medical devices is constrained by reliance on voluntary reporting of adverse events. We evaluated a strategy of prospective, active surveillance of a national clinical registry to monitor the safety of an implantable vascular-closure device that had a suspected association with increased adverse events after percutaneous coronary intervention (PCI). METHODS: We used an integrated clinical-data surveillance system to conduct a prospective, propensity-matched analysis of the safety of the Mynx vascular-closure device, as compared with alternative approved vascular-closure devices, with data from the CathPCI Registry of the National Cardiovascular Data Registry. The primary outcome was any vascular complication, which was a composite of access-site bleeding, access-site hematoma, retroperitoneal bleeding, or any vascular complication requiring intervention. Secondary safety end points were access-site bleeding requiring treatment and postprocedural blood transfusion. RESULTS: We analyzed data from 73,124 patients who had received Mynx devices after PCI procedures with femoral access from January 1, 2011, to September 30, 2013. The Mynx device was associated with a significantly greater risk of any vascular complication than were alternative vascular-closure devices (absolute risk, 1.2% vs. 0.8%; relative risk, 1.59; 95% confidence interval [CI], 1.42 to 1.78; P<0.001); there was also a significantly greater risk of access-site bleeding (absolute risk, 0.4% vs. 0.3%; relative risk, 1.34; 95% CI, 1.10 to 1.62; P=0.001) and transfusion (absolute risk, 1.8% vs. 1.5%; relative risk, 1.23; 95% CI, 1.13 to 1.34; P<0.001). The initial alerts occurred within the first 12 months of monitoring. Relative risks were greater in three prespecified high-risk subgroups: patients with diabetes, those 70 years of age or older, and women. All safety alerts were confirmed in an independent sample of 48,992 patients from April 1, 2014, to September 30, 2015. CONCLUSIONS: A strategy of prospective, active surveillance of a clinical registry rapidly identified potential safety signals among recipients of an implantable vascular-closure device, with initial alerts occurring within the first 12 months of monitoring. (Funded by the Food and Drug Administration and others.).

Outcomes of COVID-19 in Patients With a History of Cancer and Comorbid Cardiovascular Disease.

BACKGROUND: Cancer and cardiovascular disease (CVD) are independently associated with adverse outcomes in patients with COVID-19. However, outcomes in patients with COVID-19 with both cancer and comorbid CVD are unknown. METHODS: This retrospective study included 2,476 patients who tested positive for SARS-CoV-2 at 4 Massachusetts hospitals between March 11 and May 21, 2020. Patients were stratified by a history of either cancer (n=195) or CVD (n=414) and subsequently by the presence of both cancer and CVD (n=82). We compared outcomes between patients with and without cancer and patients with both cancer and CVD compared with patients with either condition alone. The primary endpoint was COVID-19-associated severe disease, defined as a composite of the need for mechanical ventilation, shock, or death. Secondary endpoints included death, shock, need for mechanical ventilation, need for supplemental oxygen, arrhythmia, venous thromboembolism, encephalopathy, abnormal troponin level, and length of stay. RESULTS: Multivariable analysis identified cancer as an independent predictor of COVID-19-associated severe disease among all infected patients. Patients with cancer were more likely to develop COVID-19-associated severe disease than were those without cancer (hazard ratio [HR], 2.02; 95% CI, 1.53-2.68; P<.001). Furthermore, patients with both cancer and CVD had a higher likelihood of COVID-19-associated severe disease compared with those with either cancer (HR, 1.86; 95% CI, 1.11-3.10; P=.02) or CVD (HR, 1.79; 95% CI, 1.21-2.66; P=.004) alone. Patients died more frequently if they had both cancer and CVD compared with either cancer (35% vs 17%; P=.004) or CVD (35% vs 21%; P=.009) alone. Arrhythmias and encephalopathy were also more frequent in patients with both cancer and CVD compared with those with cancer alone. CONCLUSIONS: Patients with a history of both cancer and CVD are at significantly higher risk of experiencing COVID-19-associated adverse outcomes. Aggressive public health measures are needed to mitigate the risks of COVID-19 infection in this vulnerable patient population.

Survival analysis of hierarchical learning curves in assessment of cardiac device and procedural safety.

Many Americans rely on cardiac surgical procedures and devices such as pacemakers and thrombolytic catheters to treat or manage their cardiovascular diseases. However, the failure of these cardiac devices and procedures could have grave consequences. One reason cardiac devices tended to fail was due to physician error; there is a learning effect for the physician or operator to come up to speed in skillfully implanting devices and conducting procedures. In order to better understand these learning effects, we had previously modeled the resulting learning curve effects in simulations a hierarchical setting with physicians clustered within institutions using our unique methodology (see the work of Govindarajulu et al 2017). Previously, we had employed these in hierarchical linear modeling and also in generalized estimating equations. In this setting, we have demonstrated how to apply similar methodology but revised in a survival analytic framework or time-to-event analyses. Through simulations and real dataset applications, we found that, out of the three shapes modeled to fit the learning curve, the logarithmic shape tended to have the best fit, similar to previous work (see the work of Govindarajulu et al 2017). However, as seen before, modeling the learning rate can be dataset specific and one shape may be better than another. We learned that modeling the learning rate could also be applied in the survival analysis setting through this new methodology. The goal of this paper is to model cardiac device and procedure learning curve effects in a time-to-event setting so that this knowledge may allow for the improvement of both short and long-term patient survival.

Identifying patients for safe early hospital discharge following st elevation myocardial infarction.

OBJECTIVES: To examine whether the CADILLAC risk score is an effective method of patient stratification for early discharge following ST elevation myocardial infarction (STEMI). BACKGROUND: Patients with STEMI are typically hospitalized to monitor for serious complications such as arrhythmias, heart failure, and reinfarction. Optimal length of stay is unclear. Whether low risk patients can be safely discharged before 72 hr of hospitalization is unclear. METHODS: Patients with STEMI who underwent successful PCI were retrospectively stratified using CADILLAC risk score to low risk (n = 123) and intermediate to high risk (n = 105). The primary outcome was adverse clinical events at day 3 or later. Secondary outcomes were adverse clinical events on day 1 and mortality rates at 30 days and 31 to 365 days. RESULTS: Low risk patients had lower major adverse clinical events at day 3 or later (0 vs. 11.4%, P = 0.0002) and lower total mortality at 1 year (0 vs. 4.8%, P = 0.02) than patients with intermediate to high risk. Low risk patients were also less likely to have a cardiovascular event during the first 24 hr when compared to those with an intermediate to high risk score (3.3% vs. 13.3%, P = 0.006). CONCLUSION: Low risk patients identified using CADILLAC risk score with STEMI treated successfully with primary PCI have a low adverse event rate on the third day or later of hospitalization suggesting that an earlier discharge is safe in properly selected patients. Monitoring in a noncritical care setting following primary PCI for STEMI may be feasible for selected patients. © 2016 Wiley Periodicals, Inc.

Learning curve estimation in medical devices and procedures: hierarchical modeling.

In the use of medical device procedures, learning effects have been shown to be a critical component of medical device safety surveillance. To support their estimation of these effects, we evaluated multiple methods for modeling these rates within a complex simulated dataset representing patients treated by physicians clustered within institutions. We employed unique modeling for the learning curves to incorporate the learning hierarchy between institution and physicians and then modeled them within established methods that work with hierarchical data such as generalized estimating equations (GEE) and generalized linear mixed effect models. We found that both methods performed well, but that the GEE may have some advantages over the generalized linear mixed effect models for ease of modeling and a substantially lower rate of model convergence failures. We then focused more on using GEE and performed a separate simulation to vary the shape of the learning curve as well as employed various smoothing methods to the plots. We concluded that while both hierarchical methods can be used with our mathematical modeling of the learning curve, the GEE tended to perform better across multiple simulated scenarios in order to accurately model the learning effect as a function of physician and hospital hierarchical data in the use of a novel medical device. We found that the choice of shape used to produce the 'learning-free' dataset would be dataset specific, while the choice of smoothing method was negligibly different from one another. This was an important application to understand how best to fit this unique learning curve function for hierarchical physician and hospital data. Copyright © 2017 John Wiley & Sons, Ltd.

A comparison of in-hospital acute myocardial infarction management between Portugal and the United States: 2000-2010.

OBJECTIVE: To compare healthcare in acute myocardial infarction (AMI) treatment between contrasting health systems using comparable representative data from Europe and USA. DESIGN: Repeated cross-sectional retrospective cohort study. SETTING: Acute care hospitals in Portugal and USA during 2000-2010. PARTICIPANTS: Adults discharged with AMI. INTERVENTIONS: Coronary revascularizations procedures (percutaneous coronary intervention (PCI), coronary artery bypass graft (CABG) surgery). MAIN OUTCOME MEASURES: In-hospital mortality and length of stay. RESULTS: We identified 1 566 601 AMI hospitalizations. Relative to the USA, more hospitalizations in Portugal presented with elevated ST-segment, and fewer had documented comorbidities. Age-sex-adjusted AMI hospitalization rates decreased in USA but increased in Portugal. Crude procedure rates were generally lower in Portugal (PCI: 44% vs. 47%; CABG: 2% vs. 9%, 2010) but only CABG rates differed significantly after standardization. PCI use increased annually in both countries but CABG decreased only in the USA (USA: 0.95 [0.94, 0.95], Portugal: 1.04 [1.02, 1.07], odds ratios). Both countries observed annual decreases in risk-adjusted mortality (USA: 0.97 [0.965, 0.969]; Portugal: 0.99 [0.979, 0.991], hazard ratios). While between-hospital variability in procedure use was larger in USA, the risk of dying in a high relative to a low mortality hospital (hospitals in percentiles 95 and 5) was 2.65 in Portugal when in USA was only 1.03. CONCLUSIONS: Although in-hospital mortality due to an AMI improved in both countries, patient management in USA seems more effective and alarming disparities in quality of care across hospitals are more likely to exist in Portugal.

Multivessel vs culprit-only percutaneous coronary intervention among patients 65 years or older with acute myocardial infarction.

BACKGROUND: Older adults presenting with acute myocardial infarction (MI) often have multivessel coronary artery disease amenable to percutaneous coronary intervention (PCI), yet the risks of multivessel intervention may outweigh potential benefits in these patients. We sought to determine if nonculprit intervention during the index PCI is associated with better outcomes among older patients with acute MI and multivessel disease. METHODS: We examined 19,271 ST-segment elevation MI (STEMI) and 31,361 non-STEMI (NSTEMI) patients 65years or older with multivessel disease in a linked CathPCI Registry-Medicare database, excluding patients with prior coronary artery bypass grafting, left main disease, or cardiogenic shock. Using inverse probability-weighted propensity adjustment, we compared mortality between patients receiving culprit-only vs multivessel intervention during the index PCI procedure. RESULTS: Most older MI patients (91% STEMI and 74% NSTEMI) received culprit-only intervention during the index PCI. Among STEMI patients, multivessel intervention during the index PCI was associated with higher 30-day mortality (8.3% vs 6.3%, adjusted hazard ratio [HR] 1.36, 95% CI 1.14-1.62) than culprit-only intervention, and this trend persisted at 1year (13.8% vs 12.2%, adjusted HR 1.14, 95% CI 0.99-1.31). No significant mortality differences were observed among NSTEMI patients at 30days (3.4% vs 4.1%, adjusted HR 1.01, 95% CI 0.88-1.15) or at 1year (10.1% vs 10.8%, adjusted HR 0.99, 95% CI 0.91-1.08). CONCLUSIONS: Nonculprit intervention during the index PCI was associated with worse outcomes among STEMI patients, but not NSTEMI patients.

Comparison of Two Risk Models in Predicting the Incidence of Contrast-Induced Nephropathy after Percutaneous Coronary Intervention.

OBJECTIVES: We sought to compare 2 contrast-induced nephropathy (CIN) risk prediction models in a validation cohort using a consensus definition. BACKGROUND: Contrast-induced nephropathy (CIN) is independently associated with mortality following percutaneous coronary intervention (PCI). Multiple prediction models for the development of CIN have been published using heterogeneous outcome definitions. METHODS: We analyzed 5,540 patients who underwent PCI from January 2005 to June 2012 at a single academic medical center. The primary outcome was development of CIN, defined as an increase in serum creatinine of ≥0.5 mg/dl or a relative increase of ≥25% from baseline. Receiver operator characteristic (ROC) curves were used to evaluate the discriminatory power of Mehran and WBH prediction models. RESULTS: The mean age of our cohort was 68 ± 12 years. The mean baseline creatinine was 1.2 ± 0.53 mg/dl (eGFR 73 ± 27 ml/min). The mean contrast volume used was 212 ± 92 ml. CIN occurred in 436 patients (7.9%). The Mehran risk score demonstrated better discrimination than the William Beaumont Hospital (WBH) risk score to predict the occurrence of CIN (c statistic: 0.82 vs. 0.73, respectively). Mortality at 30 days was approximately 8 times higher among patients with CIN as compared to those without (14.7% vs. 1.8% P < 0.01). CONCLUSIONS: In an independent validation cohort, the Mehran risk model demonstrates greater discriminatory power than the WBH model in predicting the incidence of CIN. Mortality was significantly higher in patients who developed CIN after PCI.

Assessing the cost burden of United States FDA-mandated post-approval studies for medical devices.

Approved medical devices frequently undergo FDA mandated post-approval studies (PAS). However, there is uncertainty as to the value of PAS in assessing the safety of medical devices and the cost of these studies to the healthcare system is unknown. Since PAS costs are funded through device manufacturers who do not share the costs with regulators, we sought to estimate the total PAS costs through interviews with a panel of experts in medical device clinical trial design in order to design a general cost model for PAS which was then applied to the FDA PAS. A total of 277 PAS were initiated between 3/1/05 through 6/30/13 and demonstrated a median cost of $2.16 million per study and an overall cost of $1.22 billion over the 8.25 years of study. While these costs are funded through manufacturers, the ultimate cost is borne by the healthcare system through the medical device costs. Given concerns regarding the informational value of PAS, the resources used to support mandated PAS may be better allocated to other approaches to assure safety.

Reporting trends and outcomes in ST-segment-elevation myocardial infarction national hospital quality assessment programs.

BACKGROUND: For patients who undergo primary percutaneous coronary intervention (PCI) for ST-segment-elevation myocardial infarction, the door-to-balloon time is an important performance measure reported to the Centers for Medicare & Medicaid Services (CMS) and tied to hospital quality assessment and reimbursement. We sought to assess the use and impact of exclusion criteria associated with the CMS measure of door-to-balloon time in primary PCI. METHODS AND RESULTS: All primary PCI-eligible patients at 3 Massachusetts hospitals (Brigham and Women's, Massachusetts General, and North Shore Medical Center) were evaluated for CMS reporting status. Rates of CMS reporting exclusion were the primary end points of interest. Key secondary end points were between-group differences in patient characteristics, door-to-balloon times, and 1-year mortality rates. From 2005 to 2011, 26% (408) of the 1548 primary PCI cases were excluded from CMS reporting. This percentage increased over the study period from 13.9% in 2005 to 36.7% in the first 3 quarters of 2011 (P<0.001). The most frequent cause of exclusion was for a diagnostic dilemma such as a nondiagnostic initial ECG, accounting for 31.2% of excluded patients. Although 95% of CMS-reported cases met door-to-balloon time goals in 2011, this was true of only 61% of CMS-excluded cases and consequently 82.6% of all primary PCI cases performed that year. The 1-year mortality for CMS-excluded patients was double that of CMS-included patients (13.5% versus 6.6%; P<0.001). CONCLUSIONS: More than a quarter of patients who underwent primary PCI were excluded from hospital quality reports collected by CMS, and this percentage has grown substantially over time. These findings may have significant implications for our understanding of process improvement in primary PCI and mechanisms for reimbursement through Medicare.

Drug-eluting stents versus bare metal stents prior to noncardiac surgery.

BACKGROUND: The safety of drug-eluting stents (DES) vs. bare metal stents (BMS) in the perioperative setting, a heightened state of inflammation and thrombosis is not well defined. METHODS: All adults undergoing noncardiac surgical (NCS) procedures within 1 year following percutaneous coronary intervention (PCI) in Massachusetts between April 1, 2004, and September 30, 2007, were identified from an administrative claims database. Patients were divided into those who received BMS vs. DES at index PCI. Primary net clinical outcome was death, myocardial infarction (MI) or bleeding within 30 days of NCS. Primary clinical outcome was 30-day death or MI. RESULTS: Among 8,415 (22% BMS) patients that satisfied our inclusion criteria, 1,838 BMS patients were matched with 3,565 DES patients with similar propensity scores. In the DES cohort, the 30-day primary net clinical outcome rate was lower with longer time from PCI to NCS (P = 0.02) with lowest rates if NCS was performed after 90 days from PCI (event rate 8.57, 7.53, 5.21, and 5.75% for 1-30, 31-90, 91-180, and 181-365 days from PCI to NCS). However, in the BMS cohort, the event rate was uniformly high regardless of the time from PCI to NCS (P = 0.60) (event rate 8.20, 6.56, 8.05, and 8.82% for 1-30, 31-90, 91-180, and 181-365 days from PCI to NCS). There was no significant difference between DES and the BMS group for 30-day primary net clinical outcome (6.64 vs. 7.89%; P = 0.10), but there was a 26% lower odds of primary clinical outcome (OR = 0.74, 95% CI 0.58-0.94) with DES when compared with BMS, driven mainly by differences in event rates when NCS was performed >90 days post PCI. CONCLUSION: DES implantation was not associated with higher adverse events after NCS. Moreover, the incidence of adverse events following NCS was lower when NCS was performed >90 days post-DES implantation suggesting that it may not be necessary to wait until 12 months post PCI with DES before NCS.

Spontaneous Migration of Ventriculoperitoneal Shunt Into the Right Heart and Pulmonary Vasculature.

A 73-year-old woman with ventriculoperitoneal (VP) shunt presented for stress echocardiogram for evaluation of chest pain. Transthoracic echocardiogram revealed an incidental right heart mass representing a migrated VP shunt. This case highlights the role of multimodality cardiac imaging in diagnosing right heart masses and the multidisciplinary approach to management.

Testing a Cloud-Based Model for Active Surveillance of Medical Devices with Analyses of Coronary Stent Safety Using the Data Extraction and Longitudinal Trend Analysis (DELTA) System.

Objective: To demonstrate the use of the Data Extraction and Longitudinal Trend Analysis (DELTA) system in the National Evaluation System for health Technology's (NEST) medical device surveillance cloud environment by analyzing coronary stent safety using real world clinical data and comparing results to clinical trial findings. Design and Setting: Electronic health record (EHR) data from two health systems, the Social Security Death Master File, and device databases were ingested into the NEST cloud, and safety analyses of two stents were performed using DELTA. Participants and Interventions: This is an observational study of patients receiving zotarolimus drug-eluting coronary stents (ZES) or everolimus eluting coronary stents (EES) between July 1, 2015 and December 31, 2017. Results: After exclusions, 3334 patients receiving EES and 1002 receiving ZES were available for study. Analysis using inverse probability weighting showed no significant difference in one-year mortality or major adverse cardiac events (MACE) for EES compared to ZES [Mortality Odds Ratio 0.94 (95% CI 0.81-1.175); p = 0.780] [MACE Odds Ratio 1.04 (95% CI 0.92-1.16; p = 0.551]). Analysis using propensity matching showed no significant difference in EES one-year mortality (547 of 992 alive and available after censoring) compared to ZES (546 of 992) [Log-Rank statistic 0.3348 (p = 0.563)]. Conclusion: Automated cloud-based medical device safety surveillance using EHR data is feasible and was efficiently performed using DELTA. No statistically significant differences in 1-year safety outcomes between ZES and EES were identified using two statistical approaches, consistent with randomized trial findings.

A machine learning framework to adjust for learning effects in medical device safety evaluation.

OBJECTIVES: Traditional methods for medical device post-market surveillance often fail to accurately account for operator learning effects, leading to biased assessments of device safety. These methods struggle with non-linearity, complex learning curves, and time-varying covariates, such as physician experience. To address these limitations, we sought to develop a machine learning (ML) framework to detect and adjust for operator learning effects. MATERIALS AND METHODS: A gradient-boosted decision tree ML method was used to analyze synthetic datasets that replicate the complexity of clinical scenarios involving high-risk medical devices. We designed this process to detect learning effects using a risk-adjusted cumulative sum method, quantify the excess adverse event rate attributable to operator inexperience, and adjust for these alongside patient factors in evaluating device safety signals. To maintain integrity, we employed blinding between data generation and analysis teams. Synthetic data used underlying distributions and patient feature correlations based on clinical data from the Department of Veterans Affairs between 2005 and 2012. We generated 2494 synthetic datasets with widely varying characteristics including number of patient features, operators and institutions, and the operator learning form. Each dataset contained a hypothetical study device, Device B, and a reference device, Device A. We evaluated accuracy in identifying learning effects and identifying and estimating the strength of the device safety signal. Our approach also evaluated different clinically relevant thresholds for safety signal detection. RESULTS: Our framework accurately identified the presence or absence of learning effects in 93.6% of datasets and correctly determined device safety signals in 93.4% of cases. The estimated device odds ratios' 95% confidence intervals were accurately aligned with the specified ratios in 94.7% of datasets. In contrast, a comparative model excluding operator learning effects significantly underperformed in detecting device signals and in accuracy. Notably, our framework achieved 100% specificity for clinically relevant safety signal thresholds, although sensitivity varied with the threshold applied. DISCUSSION: A machine learning framework, tailored for the complexities of post-market device evaluation, may provide superior performance compared to standard parametric techniques when operator learning is present. CONCLUSION: Demonstrating the capacity of ML to overcome complex evaluative challenges, our framework addresses the limitations of traditional statistical methods in current post-market surveillance processes. By offering a reliable means to detect and adjust for learning effects, it may significantly improve medical device safety evaluation.

A Comprehensive Framework for Evaluating the Value Created by Real-World Evidence for Diverse Stakeholders: The Case for Coordinated Registry Networks.

OBJECTIVES: This manuscript presents a comprehensive framework for the assessment of the value of real-world evidence (RWE) in healthcare decision-making. While RWE has been proposed to overcome some limitations of traditional, one-off studies, no systematic framework exists to measure if RWE actually lowers the burden. This framework aims to fill that gap by providing conceptual approaches for evaluating the time and cost efficiencies of RWE, thus guiding strategic investments in RWE infrastructure. METHODS: The framework consists of four components: (114th Congress. 21st Century Cures Act.; 2015. https://www.congress.gov/114/plaws/publ255/PLAW-114publ255.pdf .) identification of stakeholders using and producing RWE, (National Health Council. Glossary of Patient Engagement Terms. Published 2019. Accessed May 18. 2021. https://nationalhealthcouncil.org/glossary-of-patient-engagement-terms/ .) understanding value propositions on how RWE can benefit stakeholders, (Center for Drug Evaluation and Research. CDER Patient-Focused Drug Development. U.S. Food & Drug Administration.) defining key performance indicators (KPIs), and (U.S. Department of Health and Human Services - Food and Drug Administration: Center for Devices and Radiological Health and Center for Biologics Evaluation and Research. Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices - Guidance for Industry and Food and Drug Administration Staff. 2017. http://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/Guida .) establishing metrics and case studies to assess value. KPIs are categorized as 'better, faster, or cheaper" as an indicator of value: better focusing on high-quality actionable evidence; 'faster,' denoting time-saving in evidence generation, and 'cheaper,' emphasizing cost-efficiency decision compared to methodologies that do not involve data routinely collected in clinical practice. Metrics and relevant case studies are tailored based on stakeholder value propositions and selected KPIs that can be used to assess what value has been created by using RWE compared to traditional evidence-generation approaches and comparing different RWE sources. RESULTS: Operationalized through metrics and case studies drawn from the literature, the value of RWE is documented as improving treatment effect heterogeneity evaluation, expanding medical product labels, and expediting post-market compliance. RWE is also shown to reduce the cost and time required to produce evidence compared to traditional one-off approaches. An original example of a metric that measures the time saved by RWE methods to detect a signal of a product failure was presented based on analysis of the National Cardiovascular Disease Registry. CONCLUSIONS: The framework presented in this manuscript offers a comprehensive approach for evaluating the value of RWE, applicable to all stakeholders engaged in leveraging RWE for healthcare decision-making. Through the proposed metrics and illustrated case studies, valuable insights are provided into the heightened efficiency, cost-effectiveness, and improved decision-making within clinical and regulatory domains facilitated by RWE. While this framework is primarily focused on medical devices, it could potentially inform the determination of RWE value in other medical products. By discerning the variations in cost, time, and data utility among various evidence-generation methods, stakeholders are empowered to invest strategically in RWE infrastructure and shape future research endeavors.