A call to incorporate systems theory and human factors into the existing investigation of harm in clinical research involving healthcare products.

This is a joint statement from individual pharmacology and pharmaceutical professionals acting in their own capacity, including members of the Alliance for Clinical Research Excellence and Safety (ACRES) and the International Society of Pharmacovigilance (ISoP). By building on the extensive pharmacological and regulatory investigations that already take place, we are calling for a fuller and more robust systems-based approach to the independent investigation of clinical research when serious incidents of harm occur, starting with first-in-human clinical trials. To complement existing activities and regulations, we propose an additional approach blending evidence derived from both pharmacological and organizational science, which addresses human factors and transparency, to enhance organizational learning and continuous improvement. As happens with investigations in other sectors of society, such as the chemical and aviation sector, this systems approach should be seen as an additional way to understand how problems occur and how they might be prevented in the future. We believe that repetition of potentially preventable and adverse outcomes during clinical research, by failing to identify and act upon all systematic vulnerabilities, is a situation that needs urgent change. As we will discuss further on, approaches based on applying systems theory and human factors are much more likely to improve objectivity and transparency, leading to better system design.

Drug-versus-drug adverse event rate comparisons: a pilot study based on data from the US FDA Adverse Event Reporting System.

BACKGROUND: A number of published studies compare adverse event rates for drugs on the basis of reports in the US FDA Adverse Event Reporting System (AERS). While the AERS data have the advantage of timely availability and a large capture population, the database is subject to many significant biases, and lacks complete patient information that would allow for correction of those biases. The accuracy of comparative AERS-based data mining has been questioned, but has not been systematically studied. OBJECTIVE: To determine whether AERS could be used as a data source to accurately compare the adverse event rates for pairs of drugs, using pre-defined, stringent criteria to dictate whether a given pair of drugs was considered eligible for such a comparison. METHODS: The Fisher's Exact test was utilized to detect differences in adverse event rates between such pairs of drugs. Concordance was determined between statistically significant AERS-based adverse event rate differences, and adverse event rate differences published in the literature from clinical trials and case-control studies. The conditions for validity included (i) data that are free of 'extreme duplication' in AERS reports; (ii) drugs used in similar patient populations; (iii) drugs used for similar indications; (iv) drugs used with the same spectrum of concomitant medications; and (v) drugs not widely disparate in time on the market. RESULTS: For 19 drugs studied, a total of 36 evaluable adverse event rate comparisons were identified. Comparisons were classified as favouring 'drug A', favouring 'drug B' or detecting no difference. Concordance for the resulting 3x3 table (AERS vs literature) gave a kappa statistic of 0.654, indicating moderately good agreement. In only two cases was there absolute discordance, with AERS designating one drug as having a lower rate, while the published study designated the other drug as having a lower rate, with respect to a given adverse event. CONCLUSIONS: This pilot study encourages further research regarding the use of spontaneous report databases such as AERS, under stringently defined conditions, to compare adverse event rates for drugs. While not hypothesis proving, such estimates can be used for purposes such as generating hypotheses for controlled studies, and for designing those studies.

Systematic investigation of time windows for adverse event data mining for recently approved drugs.

The optimum timing of drug safety data mining for a new drug is uncertain. The objective of this study was to compare cumulative data mining versus mining with sliding time windows. Adverse Event Reporting System data (2001-2005) were studied for 27 drugs. A literature database was used to evaluate signals of disproportionate reporting (SDRs) from an urn model data-mining algorithm. Data mining was applied cumulatively and with sliding time windows from 1 to 4 years in width. Time from SDR generation to the appearance of a publication describing the corresponding adverse event was calculated. Cumulative data mining and 1- to 2-year sliding windows produced the most SDRs for recently approved drugs. In the first postmarketing year, data mining produced SDRs an average of 800 days in advance of publications regarding the corresponding drug-event combination. However, this timing advantage reduced to zero by year 4. The optimum window width for sliding windows should increase with time on the market. Data mining may be most useful for early signal detection during the first 3 years of a drug's postmarketing life. Beyond that, it may be most useful for supporting or weakening hypotheses.

An evaluation of three signal-detection algorithms using a highly inclusive reference event database.

BACKGROUND: Pharmacovigilance data-mining algorithms (DMAs) are known to generate significant numbers of false-positive signals of disproportionate reporting (SDRs), using various standards to define the terms 'true positive' and 'false positive'. OBJECTIVE: To construct a highly inclusive reference event database of reported adverse events for a limited set of drugs, and to utilize that database to evaluate three DMAs for their overall yield of scientifically supported adverse drug effects, with an emphasis on ascertaining false-positive rates as defined by matching to the database, and to assess the overlap among SDRs detected by various DMAs. METHODS: A sample of 35 drugs approved by the US FDA between 2000 and 2004 was selected, including three drugs added to cover therapeutic categories not included in the original sample. We compiled a reference event database of adverse event information for these drugs from historical and current US prescribing information, from peer-reviewed literature covering 1999 through March 2006, from regulatory actions announced by the FDA and from adverse event listings in the British National Formulary. Every adverse event mentioned in these sources was entered into the database, even those with minimal evidence for causality. To provide some selectivity regarding causality, each entry was assigned a level of evidence based on the source of the information, using rules developed by the authors. Using the FDA adverse event reporting system data for 2002 through 2005, SDRs were identified for each drug using three DMAs: an urn-model based algorithm, the Gamma Poisson Shrinker (GPS) and proportional reporting ratio (PRR), using previously published signalling thresholds. The absolute number and fraction of SDRs matching the reference event database at each level of evidence was determined for each report source and the data-mining method. Overlap of the SDR lists among the various methods and report sources was tabulated as well. RESULTS: The GPS algorithm had the lowest overall yield of SDRs (763), with the highest fraction of events matching the reference event database (89 SDRs, 11.7%), excluding events described in the prescribing information at the time of drug approval. The urn model yielded more SDRs (1562), with a non-significantly lower fraction matching (175 SDRs, 11.2%). PRR detected still more SDRs (3616), but with a lower fraction matching (296 SDRs, 8.2%). In terms of overlap of SDRs among algorithms, PRR uniquely detected the highest number of SDRs (2231, with 144, or 6.5%, matching), followed by the urn model (212, with 26, or 12.3%, matching) and then GPS (0 SDRs uniquely detected). CONCLUSIONS: The three DMAs studied offer significantly different tradeoffs between the number of SDRs detected and the degree to which those SDRs are supported by external evidence. Those differences may reflect choices of detection thresholds as well as features of the algorithms themselves. For all three algorithms, there is a substantial fraction of SDRs for which no external supporting evidence can be found, even when a highly inclusive search for such evidence is conducted.

Mycophenolate mofetil versus azathioprine therapy is associated with a significant protection against long-term renal allograft function deterioration.

BACKGROUND: To evaluate the association of long-term continuous mycophenolate mofetil (MMF) versus azathioprine (AZA) therapy and renal allograft function, as measured by the slope of reciprocal creatinine, we analyzed 49,666 primary renal allograft recipients reported to the United States Renal Data System between October 31, 1988 and June 30, 1998. METHODS: The primary study endpoint was defined as a greater than 20% decrease below a 6-month baseline of 1/serum creatinine (SCr) (slope of reciprocal creatinine) at or beyond 1 year after transplantation. A secondary endpoint was defined as reaching an SCr value greater than 1.6 mg/dL. Univariate Kaplan-Meier analysis and multivariate Cox proportional hazard models were used to investigate the risk of reaching the study endpoints. Multivariate analyses were corrected for potential confounding covariates. RESULTS: According to the Cox proportional hazard model, 12-month continued therapy of MMF versus AZA was associated with a protective effect against declining renal function, as measured by the slope of reciprocal creatinine (relative risk [RR]=0.84, confidence interval 0.78-0.91, P<0.001). For 24-month continued therapy of MMF versus AZA, MMF was associated with a further decreased risk for a decline in renal function (RR=0.66, confidence interval=0.57-0.77, P<0.001). Furthermore, MMF was associated with a protective effect against reaching the SCr threshold of 1.6 mg/dL (RR=0.80, P<0.001) beyond 12 months posttransplantation. CONCLUSIONS: Continuous use of MMF versus AZA was associated with a protective effect against declining renal function beyond 1 year after transplantation. Further study is needed to confirm that continued MMF therapy is protective against long-term deterioration in renal function.

Intensive lifestyle modification: impact on cardiovascular disease risk factors in subjects with and without clinical cardiovascular disease.

Intensive lifestyle modification programs are intended to stabilize or promote regression of coronary artery disease; however, clinical response is often nonuniform, complicating appropriate utilization of resources and prediction of outcome. This study assessed physiological and psychological benefits to 72 persons participating in a prospective, nonrandomized, four-component lifestyle change program and compared response between patients with clinical cardiovascular disease (CVD) and patients with elevated risk factors for CVD but without clinical manifestations of disease. Subjects entering the program due to elevated risk factor levels alone demonstrated equal or greater benefit, in terms of improvement in primary CVD risk factors and reduction in measures of coronary disease risk developed in the Framingham Heart Study, than those with clinical CVD. These findings suggest that intensive lifestyle change programs may be important for primary prevention in individuals at increased risk of CVD.

An experimental investigation of masking in the US FDA adverse event reporting system database.

BACKGROUND: A phenomenon of 'masking' or 'cloaking' in pharmacovigilance data mining has been described, which can potentially cause signals of disproportionate reporting (SDRs) to be missed, particularly in pharmaceutical company databases. Masking has been predicted theoretically, observed anecdotally or studied to a limited extent in both pharmaceutical company and health authority databases, but no previous publication systematically assesses its occurrence in a large health authority database. OBJECTIVE: To explore the nature, extent and possible consequences of masking in the US FDA Adverse Event Reporting System (AERS) database by applying various experimental unmasking protocols to a set of drugs and events representing realistic pharmacovigilance analysis conditions. METHODS: This study employed AERS data from 2001 through 2005. For a set of 63 Medical Dictionary for Regulatory Activities (MedDRA®) Preferred Terms (PTs), disproportionality analysis was carried out with respect to all drugs included in the AERS database, using a previously described urn-model-based algorithm. We specifically sought masking in which drug removal induced an increase in the statistical representation of a drug-event combination (DEC) that resulted in the emergence of a new SDR. We performed a series of unmasking experiments selecting drugs for removal using rational statistical decision rules based on the requirement of a reporting ratio (RR) >1, top-ranked statistical unexpectedness (SU) and relatedness as reflected in the WHO Anatomical Therapeutic Chemical level 4 (ATC4) grouping. In order to assess the possible extent of residual masking we performed two supplemental purely empirical analyses on a limited subset of data. This entailed testing every drug and drug group to determine which was most influential in uncovering masked SDRs. We assessed the strength of external evidence for a causal association for a small number of masked SDRs involving a subset of 29 drugs for which level of evidence adjudication was available from a previous study. RESULTS: The original disproportionality analysis identified 8719 SDRs for the 63 PTs. The SU-based unmasking protocols generated variable numbers of masked SDRs ranging from 38 to 156, representing a 0.43-1.8% increase over the number of baseline SDRs. A significant number of baseline SDRs were also lost in the course of our experiments. The trend in the number of gained SDRs per report removed was inversely related to the number of lost SDRs per protocol. Both the number and nature of the reports removed influenced the number of gained SDRs observed. The purely empirical protocols unmasked up to ten times as many SDRs. None of the masked SDRs had strong external evidence supporting a causal association. Most involved associations for which there was no external supporting evidence or were in the original product label. For two masked SDRs, there was external evidence of a possible causal association. CONCLUSIONS: We documented masking in the FDA AERS database. Attempts at unmasking SDRs using practically implementable protocols produced only small changes in the output of SDRs in our analysis. This is undoubtedly related to the large size and diversity of the database, but the complex interdependencies between drugs and events in authentic spontaneous reporting system (SRS) databases, and the impact of measures of statistical variability that are typically used in real-world disproportionality analysis, may be additional factors that constrain the discovery of masked SDRs and which may also operate in pharmaceutical company databases. Empirical determination of the most influential drugs may uncover significantly more SDRs than protocols based on predetermined statistical selection rules but are impractical except possibly for evaluating specific events. Routine global exercises to elicit masking, especially in large health authority databases are not justified based on results available to date. Exercises to elicit unmasking should be driven by prior knowledge or obvious data imbalances.

Influence of the MedDRA hierarchy on pharmacovigilance data mining results.

PURPOSE: To compare the results of drug safety data mining with three different algorithms, when adverse events are identified using MedDRA Preferred Terms (PT) vs. High Level Terms (HLT) vs. Standardised MedDRA Queries (SMQ). METHODS: For a representative set of 26 drugs, data from the FDA Adverse Event Reporting System (AERS) database from 2001 through 2005 was mined for signals of disproportionate reporting (SDRs) using three different data mining algorithms (DMAs): the Gamma Poisson Shrinker (GPS), the urn-model algorithm (URN), and the proportional reporting rate (PRR) algorithm. Results were evaluated using a previously described Reference Event Database (RED) which contains documented drug-event associations for the 26 drugs. Analysis emphasized the percentage of SDRs in the "unlabeled supported" category, corresponding to those adverse events that were not described in the U.S. prescribing information for the drug at the time of its approval, but which were supported by some published evidence for an association with the drug. RESULTS: Based on a logistic regression analysis, the percentage of unlabeled supported SDRs was smallest at the PT level, intermediate at the HLT level, and largest at the SMQ level, for all three algorithms. The GPS and URN methods detected comparable percentages of unlabeled supported SDRs while the PRR method detected a smaller percentage, at all three MedDRA levels. No evidence of a method/level interaction was seen. CONCLUSIONS: Use of HLT and SMQ groupings can improve the percentage of unlabeled supported SDRs in data mining results. The trade-off for this gain is the medically less-specific language of HLTs and SMQs compared to PTs, and the need for the added step in data mining of examining the component PTs of each HLT or SMQ that results in a signal of disproportionate reporting.

Predicting response of vitreous hemorrhage after intravitreous injection of highly purified ovine hyaluronidase (Vitrase) in patients with diabetes.

PURPOSE: To develop a predictive model for patients with diabetes who are most likely to have vitreous hemorrhage clearing by 3 months after a single, intravitreous injection of highly purified, preservative-free, ovine hyaluronidase (Vitrase; ISTA Pharmaceuticals, Inc., Irvine, CA). METHODS: Post hoc data analysis was performed on two randomized, double-masked, placebo-controlled, phase 3 clinical trials of a single intravitreous injection of Vitrase for severe vitreous hemorrhage. Vitreous hemorrhage density was scored using a 0 to 4 vitreous hemorrhage grading scale in 12 radial segments of the fundus ("clock hours"). Reduction in total hemorrhage point score (DeltaTHPS) between baseline and 1 month after injection was analyzed as a predictor of vitreous hemorrhage outcome at 3 months. RESULTS: A strong predictive model was demonstrated by receiver operating characteristic (ROC) curve analysis; area under the curve (AUC) = 0.845 (P < 0.0001). The DeltaTHPS was higher in hyaluronidase-treated subjects than in saline-treated control subjects. Median DeltaTHPS was 8.0 and 6.0 in subjects treated with 55 IU (68 USP) and 75 IU (93 USP) of hyaluronidase respectively, versus 2.0 in saline control subjects (P < 0.0001). discussion. The DeltaTHPS at 1 month provides quantitative guidance for predicting the outcome of a single intravitreous ovine hyaluronidase injection in patients with diabetes and severe vitreous hemorrhage (ClinicalTrials.gov numbers, NCT00198510 and NCT00198497).

Insulin-like growth factor type-I receptor internalization and recycling mediate the sustained phosphorylation of Akt.

Previously we demonstrated that insulin-like growth factor-I mediates the sustained phosphorylation of Akt, which is essential for long term survival and protection of glial progenitors from glutamate toxicity. These prosurvival effects correlated with prolonged activation and stability of the insulin-like growth factor type-I receptor. In the present study, we investigated the mechanisms whereby insulin-like growth factor-I signaling, through the insulin-like growth factor type-I receptor, mediates the sustained phosphorylation of Akt. We showed that insulin-like growth factor-I stimulation induced loss of receptors from the cell surface but that surface receptors recovered over time. Blocking receptor internalization inhibited Akt phosphorylation, whereas inhibition of receptor trafficking blocked receptor recovery at the cell surface and the sustained phosphorylation of Akt. Moreover the insulin-like growth factor type-I receptor localized with the transferrin receptor and Rab11-positive endosomes in a ligand-dependent manner, further supporting the conclusion that this receptor follows a recycling pathway. Our results provide evidence that ligand stimulation leads to internalization of the insulin-like growth factor type-I receptor, which mediates Akt phosphorylation, and that receptor recycling sustains Akt phosphorylation in glial progenitors. Mathematical modeling of receptor trafficking further supports these results and predicts an additional kinetic state of the receptor consistent with sustained Akt phosphorylation.

Long-term use of mycophenolate mofetil is associated with a reduction in the incidence and risk of late rejection.

To evaluate the association of long-term continuous (minimum 1 year) mycophenolate mofetil (MMF) vs. azathioprine (AZA) therapy with the incidence of late acute rejection, we analyzed 47 693 primary renal allograft recipients reported to the United States Renal Data System between 1988 and 1998. The primary study endpoint was acute rejection beyond 1 year after transplantation. Univariate Kaplan-Meier analysis and multivariate Cox proportional hazard models were used to investigate the risk of reaching the study endpoints. All multivariate analyses were corrected for potential confounding covariates. Mycophenolate mofetil was associated with a 65% decreased risk of developing late acute rejection as compared to AZA (RR = 0.35, CI 0.27-0.45, p < 0.001). The incidence of acute rejection episodes at 2 and 3 years post-transplantation was significantly lower in the MMF group (0.9% at 2 years, 1.1% at 3 years) than the AZA group (6.1% at 2 years, 9.3% at 3 years). In the primary vs. repeat late rejection analysis, MMF patients exhibited a decreased late acute rejection risk of 72% (RR = 0.28, p < 0.001) and 60%, respectively (RR = 0.40, p < 0.001). In African Americans, the late acute rejection risk was 70% lower in MMF patients than AZA patients (RR = 0.30, p < 0.001). Further study is indicated to determine the optimal duration of MMF therapy after renal allograft transplantation.