U.S. Food and Drug Administration Reasoning in Approval Decisions When Efficacy Evidence Is Borderline, 2013-2018.

BACKGROUND: The U.S. Food and Drug Administration (FDA) has substantial flexibility in its approval criteria in the context of life-threatening disease and unmet therapeutic need. OBJECTIVE: To understand the FDA's evidentiary standards when flexible criteria are employed. DESIGN: Case series. SETTING: Applications submitted between 2013 and 2018 that went through multiple review cycles because the evidence for clinical efficacy was initially deemed insufficient. MEASUREMENTS: Information was obtained from the approval package (available on Drugs@FDA), including advisory committee minutes, FDA reviews, and complete response letters. RESULTS: Of 912 applications reviewed, 117 went through multiple review cycles; only 22 of these faced additional review primarily because of issues related to clinical efficacy. Concerns about the end point, the clinical meaningfulness of the observed effect, and inconsistent results were common bases for initial rejection. In 7 of the 22 cases, the approval did not require new evidence but rather new interpretations of the original evidence. No FDA decisions cited reasoning used in previous decisions. LIMITATION: The conclusions rely on the authors' interpretation of the FDA statements and on a series of "close calls." CONCLUSION: The FDA has no mechanism to find or tradition to cite similar cases when weighing evidence for approvals, resulting in standalone, bespoke decisions. These decisions show highly variable criteria for "substantial evidence" when flexible evidential criteria are used, highlighted by the recent approval of aducanumab. A precedential tradition and suitable information system are required for the FDA to improve institutional memory and build upon past decisions. These would increase the FDA's decisional transparency, consistency, and predictability, which are critical to preserving the FDA's most valuable asset, the public's trust. PRIMARY FUNDING SOURCE: U.S. Food and Drug Administration.

Comparing clotting factors attributes across different methods of preference elicitation in haemophilia patients.

INTRODUCTION: Emerging, systematic approaches for capturing patient input, such as preference elicitation, can provide valuable information for the benefit-risk assessment of medical products for treating bleeding disorders, such as haemophilia. AIM: This study aims to identify existing and develop new methods to capture, rank and summarize preference scores for clotting factor therapies. METHODS: Haemophilia patient preference data were compiled from studies identified through literature review and publicly available US FDA patient-focused drug development meeting documents. Text mining was performed to identify major themes across studies. A standardized preference score was estimated and aggregated. RESULTS: Ten preference studies that employed qualitative (n = 3), and quantitative methods (n = 7) met the inclusion criteria. Text mining of qualitative and quantitative studies revealed similar themes as the standardized preference attribute importance. We found that seven quantitative studies employed discrete choice experiments (DCE)/conjoint analysis (CA) and examined a range of 5-12 attributes. For DCE/CA studies published prior to 2014 (n = 4), safety attributes (inhibitor and viral safety) were among the most important attributes, accounting for ~46% of the total utility measured. DCE/CA studies published after 2014 (n = 3) focused on frequency of infusion and reduction of bleeding risk, accounting for ~67% of the total utility. Interestingly, two studies that used different preference elicitation approaches (DCE and a monadic conjoint approach) both ranked infusion frequency as the most important attribute. CONCLUSIONS: Although there are few published patient preference studies for haemophilia, the results of this study can be viewed in the larger context of enhancing scientific methods of incorporating patient input in medical product development.

Incorporation of stochastic engineering models as prior information in Bayesian medical device trials.

Evaluation of medical devices via clinical trial is often a necessary step in the process of bringing a new product to market. In recent years, device manufacturers are increasingly using stochastic engineering models during the product development process. These models have the capability to simulate virtual patient outcomes. This article presents a novel method based on the power prior for augmenting a clinical trial using virtual patient data. To properly inform clinical evaluation, the virtual patient model must simulate the clinical outcome of interest, incorporating patient variability, as well as the uncertainty in the engineering model and in its input parameters. The number of virtual patients is controlled by a discount function which uses the similarity between modeled and observed data. This method is illustrated by a case study of cardiac lead fracture. Different discount functions are used to cover a wide range of scenarios in which the type I error rates and power vary for the same number of enrolled patients. Incorporation of engineering models as prior knowledge in a Bayesian clinical trial design can provide benefits of decreased sample size and trial length while still controlling type I error rate and power.

Incorporating patient-preference evidence into regulatory decision making.

BACKGROUND: Patients have a unique role in deciding what treatments should be available for them and regulatory agencies should take their preferences into account when making treatment approval decisions. This is the first study designed to obtain quantitative patient-preference evidence to inform regulatory approval decisions by the Food and Drug Administration Center for Devices and Radiological Health. METHODS: Five-hundred and forty United States adults with body mass index (BMI) ≥ 30 kg/m(2) evaluated tradeoffs among effectiveness, safety, and other attributes of weight-loss devices in a scientific survey. Discrete-choice experiments were used to quantify the importance of safety, effectiveness, and other attributes of weight-loss devices to obese respondents. A tool based on these measures is being used to inform benefit-risk assessments for premarket approval of medical devices. RESULTS: Respondent choices yielded preference scores indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate the minimum weight loss acceptable by a patient to receive a device with a given risk profile and the maximum mortality risk tolerable in exchange for a given weight loss. For example, to accept a device with 0.01 % mortality risk, a risk tolerant patient will require about 10 % total body weight loss lasting 5 years. CONCLUSIONS: Patient preference evidence was used make regulatory decision making more patient-centered. In addition, we captured the heterogeneity of patient preferences allowing market approval of effective devices for risk tolerant patients. CDRH is using the study tool to define minimum clinical effectiveness to evaluate new weight-loss devices. The methods presented can be applied to a wide variety of medical products. This study supports the ongoing development of a guidance document on incorporating patient preferences into medical-device premarket approval decisions.

Bayesian Statistics for Medical Devices: Progress Since 2010.

The use of Bayesian statistics to support regulatory evaluation of medical devices began in the late 1990s. We review the literature, focusing on recent developments of Bayesian methods, including hierarchical modeling of studies and subgroups, borrowing strength from prior data, effective sample size, Bayesian adaptive designs, pediatric extrapolation, benefit-risk decision analysis, use of real-world evidence, and diagnostic device evaluation. We illustrate how these developments were utilized in recent medical device evaluations. In Supplementary Material, we provide a list of medical devices for which Bayesian statistics were used to support approval by the US Food and Drug Administration (FDA), including those since 2010, the year the FDA published their guidance on Bayesian statistics for medical devices. We conclude with a discussion of current and future challenges and opportunities for Bayesian statistics, including artificial intelligence/machine learning (AI/ML) Bayesian modeling, uncertainty quantification, Bayesian approaches using propensity scores, and computational challenges for high dimensional data and models.

Application of Bayesian approaches in drug development: starting a virtuous cycle.

The pharmaceutical industry and its global regulators have routinely used frequentist statistical methods, such as null hypothesis significance testing and p values, for evaluation and approval of new treatments. The clinical drug development process, however, with its accumulation of data over time, can be well suited for the use of Bayesian statistical approaches that explicitly incorporate existing data into clinical trial design, analysis and decision-making. Such approaches, if used appropriately, have the potential to substantially reduce the time and cost of bringing innovative medicines to patients, as well as to reduce the exposure of patients in clinical trials to ineffective or unsafe treatment regimens. Nevertheless, despite advances in Bayesian methodology, the availability of the necessary computational power and growing amounts of relevant existing data that could be used, Bayesian methods remain underused in the clinical development and regulatory review of new therapies. Here, we highlight the value of Bayesian methods in drug development, discuss barriers to their application and recommend approaches to address them. Our aim is to engage stakeholders in the process of considering when the use of existing data is appropriate and how Bayesian methods can be implemented more routinely as an effective tool for doing so.

Adaptation of the WOMAC for Use in a Patient Preference Study.

OBJECTIVES: To adapt a patient-reported outcome (PRO) measure, the Western Ontario McMaster Universities Osteoarthritis Index (WOMAC), into efficacy attributes for a discrete choice experiment (DCE) survey designed to quantify the relative importance of endpoints commonly used in knee osteoarthritis (KOA) trials. METHODS: The adaptation comprised four steps: (1) selecting domains of interest; (2) determining presentation and framing of selected attributes; (3) determining attribute levels; and (4) developing choice tasks. This process involved input from multiple stakeholders, including regulators, health preference researchers, and patients. Pretesting was conducted to evaluate if patients comprehended the adapted survey attributes and could make trade-offs among them. RESULTS: The WOMAC pain and function domains were selected for adaption to two efficacy attributes. Two versions of the discrete choice experiment (DCE) instrument were created to compare efficacy using (1) total domain scores and (2) item scores for "walking on a flat surface." Both attributes were presented as improvement from baseline scores by levels of 0%, 30%, 50%, and 100%. Twenty-six participants were interviewed in a pretest of the instrument (average age 60 years; 58% female; 62% had KOA for ≥ 5 years). The participants found both versions of attributes meaningful and relevant for treatment decision-making. They demonstrated willingness and ability to tradeoff improvements in pain and function separately, though many perceived them as inter-related. CONCLUSIONS: This study adds to the growing literature regarding adapting PRO measures for patient preference studies. Such adaptation is important for designing a preference study that can incorporate a clinical trial's outcomes with PRO endpoints.

Preferences for potential benefits and risks for gene therapy in the treatment of sickle cell disease.

Objective of this study is to quantify benefit-risk tradeoffs pertaining to potential gene therapies among adults and parents/caregivers of children with sickle cell disease (SCD). A discrete-choice experiment survey was developed in which respondents selected their preferred treatment alternatives in a series of experimentally controlled pairs of hypothetical gene therapies and a "no gene therapy" option. Gene therapy alternatives were defined based on the chance of eliminating SCD symptoms, expected increases in life expectancy they could offer, treatment-related risk of death, and potential increases in lifetime cancer risk. Respondents made selections based on their current disease severity and in the context of expectations of worsened disease. Three clinical sites and 1 patient organization recruited 174 adult patients and 109 parents of children with SCD to complete the survey. Adult and parent respondents were generally willing to choose gene therapies, but the adults required higher expected levels of efficacy (ie, higher chance of eliminating symptoms) than parents to choose gene therapies that conferred mortality risks of ≥10%. When adults and parents of children with less severe symptoms were asked to consider scenarios of higher levels of disease severity, the increased risk tolerance, and the lowest acceptable level of efficacy for gene therapies with mortality risks dropped by >50%. Baseline SCD symptoms are a major driver of gene therapy acceptability. Adults and parents of patients with milder symptoms may prefer other treatment options; however, an expectation of symptoms deterioration triggers strong reassessment of the acceptable benefit-risk balance of this novel technology.

Preferences for Risks and Benefits of Islet Cell Transplantation for Persons With Type 1 Diabetes With History of Episodes of Severe Hypoglycemia: A Discrete-Choice Experiment to Inform Regulatory Decisions.

BACKGROUND: The advisory panel for US Food and Drug Administration (FDA) recently endorsed pancreatic islet cell transplantation (ICT) therapy for suboptimally controlled type 1 diabetes (T1D), and FDA approval is under consideration. An important part of regulatory approval includes the patient perspective, through discrete choice. We developed a discrete-choice instrument and used it to determine how 90 people with T1D weigh the risks and benefits of ICT to inform regulatory decisions. METHODS: Sawtooth software created a random, full-profile, balanced-overlap experimental design for a measure with 8 attributes of ICT risks/benefits, each with 3 to 5 levels. We asked 18 random task pairs, sociodemographics, diabetes management, and hypoglycemia questions. Analysis was performed using random parameters logistic regression technique. RESULTS: The strongest preference was for avoiding the highest chance (15%) of serious procedure-related complications (ß = -2.03, P < 0.001). The strongest positive preference was for gaining 5-y insulin independence (ß = 1.75, P < 0.001). The desire for 5-y HbA1C-defined clinical treatment success was also strong (ß = 1.39, P < 0.001). Subgroup analysis suggested strong gender differences with women showing much higher preferences for all benefits (68% higher for 5-y insulin independence), and men were generally more risk averse than women. Those with high versus low diabetes distress showed 3 times stronger preference for 5-y insulin independence but also twice preference to avoid risks of serious complications. CONCLUSIONS: Despite showing the most preference for avoiding serious ICT complications, people with T1D had a strong preference for achieving ICT benefits, especially insulin independence. We identified important attributes of ICT and demonstrated that patients are willing to make these trade-offs, showing support for the introduction of ICT.

Bayesian approaches in medical device clinical trials: a discussion with examples in the regulatory setting.

Challenging statistical issues often arise in the design and analysis of clinical trials to assess safety and effectiveness of medical devices in the regulatory setting. The use of Bayesian methods in the design and analysis of medical device clinical trials has been increasing significantly in the past decade, not only due to the availability of prior information, but mainly due to the appealing nature of Bayesian clinical trial designs. The Center for Devices and Radiological Health at the Food and Drug Administration (FDA) has gained extensive experience with the use of Bayesian statistical methods and has identified some important issues that need further exploration. In this article, we discuss several topics relating to the use of Bayesian statistical methods in medical device trials, based on our experience and real applications. We illustrate the benefits and challenges of Bayesian approaches when incorporating prior information to evaluate the effectiveness and safety of a medical device. We further present an example of a Bayesian adaptive clinical trial and compare it to a traditional frequentist design. Finally, we discuss the use of Bayesian hierarchical models for multiregional trials and highlight the advantages of the Bayesian approach when specifying clinically relevant study hypotheses.

Patient-centered clinical trials.

We apply Bayesian decision analysis (BDA) to incorporate patient preferences in the regulatory approval process for new therapies. By assigning weights to type I and type II errors based on patient preferences, the significance level (α) and power (1-ß) of a randomized clinical trial (RCT) for a new therapy can be optimized to maximize the value to current and future patients and, consequently, to public health. We find that for weight-loss devices, potentially effective low-risk treatments have optimal αs larger than the traditional one-sided significance level of 5%, whereas potentially less effective and riskier treatments have optimal αs below 5%. Moreover, the optimal RCT design, including trial size, varies with the risk aversion and time-to-access preferences and the medical need of the target population.

Advancing the science of patient input throughout the regulatory decision-making process.

The US Food and Drug Administration (FDA) understands the value of patient input in the regulatory decision-making process and has worked to enhance meaningful engagement. In recent years, there has been an increased scientific demand for more systematic and quantitative approaches to incorporate patient input throughout the medical product lifecycle, including to inform regulatory benefit-risk assessments. The use of patient preference information (PPI), elicited using established scientific methods, is a promising strategy for accomplishing this. Although much of the science behind PPI is not new, its application in a regulatory setting will require adapting and advancing the science of identifying, collecting, and evaluating patient input for informing regulatory decision making. Patient input and empowerment are foundational to a learning healthcare system. A learning healthcare system paradigm can also help us better understand and continuously improve the incorporation of the patient perspective in regulatory decision making. In this article, we highlight the Food and Drug Administration's Center for Biologics Evaluation and Research experience and current initiatives on advancing the science of patient input in a regulatory setting, in particular, PPI. We provide a use case that explores how the principles and benefits of PPI applied in shared clinical decision making can be realized and leveraged to enhance regulatory evaluation of innovative therapies. To further advance the application of the science of patient input in our regulatory framework, we compiled a list of example resources that support stakeholders in designing and conducting PPI studies. More collaborative research among stakeholders is needed to establish best practice approaches, ensure scientific validity, and continuously learn and improve the systematic incorporation of scientific patient input throughout the regulatory decision-making process.

Joint Estimation of Treatment and Placebo Effects in Clinical Trials with Longitudinal Blinding Assessments.

In some therapeutic areas, treatment evaluation is frequently complicated by a possible placebo effect (i.e., the psychobiological effect of a patient's knowledge or belief of being treated). When a substantial placebo effect is likely to exist, it is important to distinguish the treatment and placebo effects in quantifying the clinical benefit of a new treatment. These causal effects can be formally defined in a joint causal model that includes treatment (e.g., new versus placebo) and treatmentality (i.e., a patient's belief or mentality about which treatment she or he has received) as separate exposures. Information about the treatmentality exposure can be obtained from blinding assessments, which are increasingly common in clinical trials where blinding success is in question. Assuming that treatmentality has a lagged effect and is measured at multiple time points, this article is concerned with joint evaluation of treatment and placebo effects in clinical trials with longitudinal follow-up, possibly with monotone missing data. We describe and discuss several methods adapted from the longitudinal causal inference literature, apply them to a weight loss study, and compare them in simulation experiments that mimic the weight loss study.

Decision analysis of retrievable inferior vena cava filters in patients without pulmonary embolism.

BACKGROUND: Retrievable filters are increasingly implanted for prophylaxis in patients without pulmonary embolism (PE) but who may be at transient risk. These devices are often not removed after the risk of PE has diminished. This study employs decision analysis to weigh the risks and benefits of retrievable filter use as a function of the filter's time in situ. METHODS: Medical literature on patients with inferior vena cava (IVC) filters and a transient risk of PE were reviewed. Weights reflecting relative severity were assigned to each adverse event. The risk score was defined as weight × occurrence rate and combines the frequency and severity for each type of adverse event. The value function in the decision model combines the following risks: (1) risk in situ; (2) risk of removal, and (3) relative risk without filters. A decreasing net risk score represents a net expected benefit, and an increasing net risk score indicates the expected harm outweighs the expected benefit. RESULTS: The net risk score reaches its minimum between day 29 and 54 postimplantation. This is consistent with an increasing net risk associated with continued use of retrievable IVC filters in patients with transient, reversible risk of PE. The results were insensitive to reasonable variations in the assessed weights and adverse event occurrence rates. CONCLUSIONS: For patients with retrievable IVC filters in whom the transient risk of PE has passed, quantitative decision analysis suggests the benefit/risk profile begins to favor filter removal between 29 and 54 days after implantation.