An Open-Source R Package for Detection of Adverse Events Under-Reporting in Clinical Trials: Implementation and Validation by the IMPALA (Inter coMPany quALity Analytics) Consortium.

Accurate and timely reporting of adverse events (AEs) in clinical trials is crucial to ensuring data integrity and patient safety. However, AE under-reporting remains a challenge, often highlighted in Good Clinical Practice (GCP) audits and inspections. Traditional detection methods, such as on-site investigator audits via manual source data verification (SDV), have limitations. Addressing this, the open-source R package {simaerep} was developed to facilitate rapid, comprehensive, and near-real-time detection of AE under-reporting at each clinical trial site. This package leverages patient-level AE and visit data for its analyses. To validate its efficacy, three member companies from the Inter coMPany quALity Analytics (IMPALA) consortium independently assessed the package. Results showed that {simaerep} consistently and effectively identified AE under-reporting across all three companies, particularly when there were significant differences in AE rates between compliant and non-compliant sites. Furthermore, {simaerep}'s detection rates surpassed heuristic methods, and it identified 50% of all detectable sites as early as 25% into the designated study duration. The open-source package can be embedded into audits to enable fast, holistic, and repeatable quality oversight of clinical trials.

Statistical Modeling for Quality Risk Assessment of Clinical Trials: Follow-Up at the Era of Remote Auditing.

BACKGROUND: As investigator site audits have largely been conducted remotely during the COVID-19 pandemic, remote quality monitoring has gained some momentum. To further facilitate the conduct of remote quality assurance (QA) activities for clinical trials, we developed new quality indicators, building on a previously published statistical modeling methodology. METHODS: We modeled the risk of having an audit or inspection finding using historical audits and inspections data from 2011 to 2019. We used logistic regression to model finding risk for 4 clinical impact factor (CIF) categories: Safety Reporting, Data Integrity, Consent and Protecting Endpoints. RESULTS: We could identify 15 interpretable factors influencing audit finding risk of 4 out of 5 CIF categories. They can be used to realistically predict differences in risk between 25 and 43% for different sites which suffice to rank sites by audit and inspection finding risk. CONCLUSION: Continuous surveillance of the identified risk factors and resulting risk estimates could be used to complement remote QA strategies for clinical trials and help to manage audit targets and audit focus also in post-pandemic times.

Clinical Quality in Cancer Research: Strategy to Assess Data Integrity of Germline Variants Inferred from Tumor-Only Testing Sequencing Data.

In the majority of cancers, pathogenic variants are only found at the level of the tumor; however, an unusual number of cancers and/or diagnoses at an early age in a single family may suggest a genetic predisposition. Predisposition plays a major role in about 5-10% of adult cancers and in certain childhood tumors. As access to genomic testing for cancer patients continues to expand, the identification of potential germline pathogenic variants (PGPVs) through tumor-DNA sequencing is also increasing. Statistical methods have been developed to infer the presence of a PGPV without the need of a matched normal sample. These methods are mainly used for exploratory research, for example in real-world clinico-genomic databases/platforms (CGDB). These databases are being developed to support many applications, such as targeted drug development, clinical trial optimization, and postmarketing studies. To ensure the integrity of data used for research, a quality management system should be established, and quality oversight activities should be conducted to assess and mitigate clinical quality risks (for patient safety and data integrity). As opposed to well-defined 'good practice' quality guidelines (GxP) areas such as good clinical practice, there are no comprehensive instructions on how to assess the clinical quality of statistically derived variables from sequencing data such as PGPVs. In this article, we aim to share our strategy and propose a possible set of tactics to assess the PGPV quality and to ensure data integrity in exploratory research.

Cross-company collaboration to leverage analytics for clinical quality and accelerate drug development: The IMPALA industry group.

Quality functions from pharmaceutical sponsor companies aim to increase the use of analytics in their oversight of Good Clinical Practices and Pharmacovigilance activities. To leverage and accelerate progress, several companies decided to establish a collaborative model. The goals of this collaboration span the sharing of knowledge and ideas, the sharing of analytics methods, discussion of talent upskilling and technology adoption strategies, and collaborative discussion on these potential changes with global Health Authorities.

Bayesian Modeling for the Detection of Adverse Events Underreporting in Clinical Trials.

INTRODUCTION: Safety underreporting is a recurrent issue in clinical trials that can impact patient safety and data integrity. Clinical quality assurance (QA) practices used to detect underreporting rely on on-site audits; however, adverse events (AEs) underreporting remains a recurrent issue. In a recent project, we developed a predictive model that enables oversight of AE reporting for clinical quality program leads (QPLs). However, there were limitations to using solely a machine learning model. OBJECTIVE: Our primary objective was to propose a robust method to compute the probability of AE underreporting that could complement our machine learning model. Our model was developed to enhance patients' safety while reducing the need for on-site and manual QA activities in clinical trials. METHODS: We used a Bayesian hierarchical model to estimate the site reporting rates and assess the risk of underreporting. We designed the model with Project Data Sphere clinical trial data that are public and anonymized. RESULTS: We built a model that infers the site reporting behavior from patient-level observations and compares them across a study to enable a robust detection of outliers between clinical sites. CONCLUSION: The new model will be integrated into the current dashboard designed for clinical QPLs. This approach reduces the need for on-site audits, shifting focus from source data verification to pre-identified, higher risk areas. It will enhance further QA activities for safety reporting from clinical trials and generate quality evidence during pre-approval inspections.

Clinical Quality Considerations when Using Next-Generation Sequencing (NGS) in Clinical Drug Development.

Next-generation sequencing (NGS) and decreased costs of genomic testing are changing the paradigm in precision medicine and continue to fuel innovation. Integration of NGS into clinical drug development has the potential to accelerate clinical trial conduct and ultimately will shape the landscape of clinical care by making it easier to identify patients who would benefit from particular therapy(ies) and to monitor treatment outcomes with less invasive tests. This has led to an increased use of NGS service providers by pharmaceutical sponsors: to screen patients for clinical trials eligibility and for patient stratification, expanded Companion Diagnostic (CDx) development for treatment recommendations and Comprehensive Genomic profiling (CGP). These changes are reshaping the face of clinical quality considerations for precision medicine. Although some clinical quality considerations do exist in Health Authorities (HA) guidances and regulations (e.g., International Conference of Harmonization Good Clinical Practices-GCP), there is currently no holistic GxP-like detailed framework for pharmaceutical sponsors using NGS service providers in clinical trials, or for the development of CDx and CGP. In this research, we identified existing and applicable regulations, guidelines and recommendations that could be translated into clinical quality considerations related to technology, data quality, patients and oversight. We propose these considerations as a basis for pharmaceutical sponsors using NGS service providers in clinical drug development to develop a set of guidelines for NGS clinical quality.

Using Statistical Modeling for Enhanced and Flexible Pharmacovigilance Audit Risk Assessment and Planning.

BACKGROUND: The European Medicines Agency Good Pharmacovigilance Practices (GVP) guidelines provide a framework for pharmacovigilance (PV) audits, including limited guidance on risk assessment methods. Quality assurance (QA) teams of large and medium sized pharmaceutical companies generally conduct annual risk assessments of the PV system, based on retrospective review of data and pre-defined impact factors to plan for PV audits which require a high volume of manual work and resources. In addition, for companies of this size, auditing the entire "universe" of individual entities on an annual basis is generally prohibitive due to sheer volume. A risk assessment approach that enables efficient, temporal, and targeted PV audits is not currently available. METHODS: In this project, we developed a statistical model to enable holistic and efficient risk assessment of certain aspects of the PV system. We used findings from a curated data set from Roche operational and quality assurance PV data, covering a span of over 8 years (2011-2019) and we modeled the risk with a logistic regression on quality PV risk indicators defined as data stream statistics over sliding windows. RESULTS: We produced a model for each PV impact factor (e.g. 'Compliance to Individual Case Safety Report') for which we had enough features. For PV impact factors where modeling was not feasible, we used descriptive statistics. All the outputs were consolidated and displayed in a QA dashboard built on Spotfire®. CONCLUSION: The model has been deployed as a quality decisioning tool available to Roche Quality professionals. It is used, for example, to inform the decision on which affiliates (i.e. pharmaceutical company commercial entities) undergo audit for PV activities. The model will be continuously monitored and fine-tuned to ensure its reliability.

Leveraging analytics to assure quality during the Covid-19 pandemic - The COVACTA clinical study example.

The world has seen a shift in the ways of working during the Covid-19 pandemic. Routine activities performed at the clinical investigator sites (e.g. on-site audits) that are a part of Quality Assurance (QA) have not been feasible at this time. Analytics has played a huge role in contributing to our continued efforts of ensuring quality during the conduct of a clinical trial. Decisions driven through data, now more than ever, heavily contribute to the efficiency of QA activities. In this report, we share the approach we took to conduct QA activities for the COVACTA study (to treat Covid-19 pneumonia) by leveraging analytics.

Harnessing the Power of Quality Assurance Data: Can We Use Statistical Modeling for Quality Risk Assessment of Clinical Trials?

BACKGROUND: The increasing number of clinical trials and their complexity make it challenging to detect and identify clinical quality issues timely. Despite extensive sponsor audit programs and monitoring activities, issues related to data integrity, safety, sponsor oversight and patient consent have recurring audit and inspection findings. Recent developments in data management and IT systems allow statistical modeling to provide insights to clinical Quality Assurance (QA) professionals to help mitigate some of the key clinical quality issues more holistically and efficiently. METHODS: We used findings from a curated data set from Roche/Genentech operational and quality assurance study data, covering a span of 8 years (2011-2018) and grouped them into 5 clinical impact factor categories, for which we modeled the risk with a logistic regression using hand crafted features. RESULTS: We were able to train 5 interpretable, cross-validated models with several distinguished risk factors, many of which confirmed field observations of our quality professionals. Our models were able to reliably predict a decrease in risk by 12-44%, with 2-8 coefficients each, despite a low signal-to-noise ratio in our data set. CONCLUSION: We proposed a modeling strategy that could provide insights to clinical QA professionals to help them mitigate key clinical quality issues (e.g., safety, consent, data integrity) in a more sustained data-driven way, thus turning the traditional reactive approach to a more proactive monitoring and alerting approach. Also, we are calling for cross-sponsors collaborations and data sharing to improve and further validate the use of statistical models in clinical QA.

Enabling Data-Driven Clinical Quality Assurance: Predicting Adverse Event Reporting in Clinical Trials Using Machine Learning.

INTRODUCTION: Adverse event (AE) under-reporting has been a recurrent issue raised during health authorities Good Clinical Practices (GCP) inspections and audits. Moreover, safety under-reporting poses a risk to patient safety and data integrity. The current clinical Quality Assurance (QA) practices used to detect AE under-reporting rely heavily on investigator site and study audits. Yet several sponsors and institutions have had repeated findings related to safety reporting, and this has led to delays in regulatory submissions. Recent developments in data management and IT systems allow data scientists to apply techniques such as machine learning to detect AE under-reporting in an automated fashion. OBJECTIVE: In this project, we developed a predictive model that enables Roche/Genentech Quality Program Leads oversight of AE reporting at the program, study, site, and patient level. This project was part of a broader effort at Roche/Genentech Product Development Quality to apply advanced analytics to augment and complement traditional clinical QA approaches. METHOD: We used a curated data set from 104 completed Roche/Genentech sponsored clinical studies to train a machine learning model to predict the expected number of AEs. Our final model used 54 features built on patient (e.g., demographics, vitals) and study attributes (e.g., molecule class, disease area). RESULTS: In order to evaluate model performance, we tested how well it would detect simulated test cases based on data not used for model training. For relevant simulation scenarios of 25%, 50%, and 75% under-reporting on the site level, our model scored an area under the curve (AUC) of the receiver operating characteristic (ROC) curve of 0.62, 0.79, and 0.92, respectively. CONCLUSION: The model has been deployed to evaluate safety reporting performance in a set of ongoing studies in the form of a QA/dashboard cockpit available to Roche Quality Program Leads. Applicability and production performance will be assessed over the next 12-24 months in which we will develop a validation strategy to fully integrate our model into Roche QA processes.