National validation of the Centers for Medicare & Medicaid Services strategy for identifying potential surgical-site infections following colon surgery and abdominal hysterectomy.

OBJECTIVE: National validation of claims-based surveillance for surgical-site infections (SSIs) following colon surgery and abdominal hysterectomy. DESIGN: Retrospective cohort study. SETTING: US hospitals selected for data validation by Centers for Medicare & Medicaid Services (CMS). PARTICIPANTS: The study included 550 hospitals performing colon surgery and 458 hospitals performing abdominal hysterectomy in federal fiscal year 2013. METHODS: We requested 1,200 medical records from hospitals selected for validation as part of the CMS Hospital Inpatient Quality Reporting program. For colon surgery, we sampled 60% with a billing code suggestive of SSI during their index admission and/or readmission within 30 days and 40% who were readmitted without one of these codes. For abdominal hysterectomy, we included all patients with an SSI code during their index admission, all patients readmitted within 30 days, and a sample of those with a prolonged surgical admission (length of stay > 7 days). We calculated sensitivity and positive predictive value for the different groups. RESULTS: We identified 142 colon-surgery SSIs (46 superficial SSIs and 96 deep and organ-space SSIs) and 127 abdominal-hysterectomy SSIs (58 superficial SSIs and 69 deep and organ-space SSIs). Extrapolating to the full CMS data validation cohort, we estimated an SSI rate of 8.3% for colon surgery and 3.0% for abdominal hysterectomy. Our colon-surgery surveillance codes identified 93% of SSIs, with 1 SSI identified for every 2.6 patients reviewed. Our abdominal-hysterectomy surveillance codes identified 73% of SSIs, with 1 SSI identified for every 1.6 patients reviewed. CONCLUSIONS: Using claims to target record review for SSI validation performed well in a national sample.

Pragmatic guidance for embedding pragmatic clinical trials in health plans: Large simple trials aren't so simple.

BACKGROUND: There are unique opportunities related to the design and conduct of pragmatic trials embedded in health insurance plans, which have longitudinal data on member/patient demographics, dates of coverage, and reimbursed medical care, including prescription drug dispensings, vaccine administrations, behavioral healthcare encounters, and some laboratory results. Such trials can be large and efficient, using these data to identify trial-eligible patients and to ascertain outcomes. METHODS: We use our experience primarily with the National Institutes of Health Pragmatic Trials Collaboratory Distributed Research Network, which comprises health plans that participate in the US Food & Drug Administration's Sentinel System, to describe lessons learned from the conduct and planning of embedded pragmatic trials. RESULTS: Information is available for research on more than 75 million people with commercial or Medicare Advantage health plans. We describe three studies that have used or plan to use the Network, as well as a single health plan study, from which we glean our lessons learned. CONCLUSIONS: Studies that are conducted in health plans provide much-needed evidence to drive clinically meaningful changes in care. However, there are many unique aspects of these trials that must be considered in the planning, implementation, and analytic phases. The type of trial best suited for studies embedded in health plans will be those that require large sample sizes, simple interventions that could be disseminated through health plans, and where data available to the health plan can be leveraged. These trials hold potential for substantial long-term impact on our ability to generate evidence to improve care and population health.

The impact of surgical volume on hospital ranking using the standardized infection ratio.

The Centers for Medicare and Medicaid Services require hospitals to report on quality metrics which are used to financially penalize those that perform in the lowest quartile. Surgical site infections (SSIs) are a critical component of the quality metrics that target healthcare-associated infections. However, the accuracy of such hospital profiling is highly affected by small surgical volumes which lead to a large amount of uncertainty in estimating standardized hospital-specific infection rates. Currently, hospitals with less than one expected SSI are excluded from rankings, but the effectiveness of this exclusion criterion is unknown. Tools that can quantify the classification accuracy and can determine the minimal surgical volume required for a desired level of accuracy are lacking. We investigate the effect of surgical volume on the accuracy of identifying poorly performing hospitals based on the standardized infection ratio and develop simulation-based algorithms for quantifying the classification accuracy. We apply our proposed method to data from HCA Healthcare (2014-2016) on SSIs in colon surgery patients. We estimate that for a procedure like colon surgery with an overall SSI rate of 3%, to rank hospitals in the HCA colon SSI dataset, hospitals that perform less than 200 procedures have a greater than 10% chance of being incorrectly assigned to the worst performing quartile. Minimum surgical volumes and predicted events criteria are required to make evaluating hospitals reliable, and these criteria vary by overall prevalence and between-hospital variability.

A COVID-19-ready public health surveillance system: The Food and Drug Administration's Sentinel System.

The US Food and Drug Administration's Sentinel System was established in 2009 to use routinely collected electronic health data for improving the national capability to assess post-market medical product safety. Over more than a decade, Sentinel has become an integral part of FDA's surveillance capabilities and has been used to conduct analyses that have contributed to regulatory decisions. FDA's role in the COVID-19 pandemic response has necessitated an expansion and enhancement of Sentinel. Here we describe how the Sentinel System has supported FDA's response to the COVID-19 pandemic. We highlight new capabilities developed, key data generated to date, and lessons learned, particularly with respect to working with inpatient electronic health record data. Early in the pandemic, Sentinel developed a multi-pronged approach to support FDA's anticipated data and analytic needs. It incorporated new data sources, created a rapidly refreshed database, developed protocols to assess the natural history of COVID-19, validated a diagnosis-code based algorithm for identifying patients with COVID-19 in administrative claims data, and coordinated with other national and international initiatives. Sentinel is poised to answer important questions about the natural history of COVID-19 and is positioned to use this information to study the use, safety, and potentially the effectiveness of medical products used for COVID-19 prevention and treatment.

The Limited Utility of Ranking Hospitals Based on Their Colon Surgery Infection Rates.

BACKGROUND: The Centers for Medicare and Medicaid Services (CMS) use colon surgical site infection (SSI) rates to rank hospitals and apply financial penalties. The CMS' risk-adjustment model omits potentially impactful variables that might disadvantage hospitals with complex surgical populations. METHODS: We analyzed adult patients who underwent colon surgery within facilities associated with HCA Healthcare from 2014 to 2016. SSIs were identified from National Health Safety Network (NHSN) reporting. We trained and validated 3 SSI prediction models, using (1) current CMS model variables, including hospital-specific random effects (HCA-adapted CMS model); (2) demographics and claims-based comorbidities (expanded-claims model); and (3) demographics, claims-based comorbidities, and NHSN variables (claims-plus-electronic health record [EHR] model). Discrimination, calibration, and resulting rankings were compared among all models and the current CMS model with published coefficient values. RESULTS: We identified 39 468 colon surgeries in 149 hospitals, resulting in 1216 (3.1%) SSIs. Compared to the HCA-adapted CMS model, the expanded-claims model had similar performance (c-statistic, 0.65 vs 0.67, respectively), while the claims-plus-EHR model was more accurate (c-statistic, 0.70; 95% confidence interval, .67-.73; P = .004). The sampling variation, due to the low surgical volume and small number of infections, contributed 74% of the total variation in observed SSI rates between hospitals. When CMS model rankings were compared to those from the expanded-claims and claims-plus-EHR models, 18 (15%) and 26 (22%) hospitals changed quartiles, respectively, and 10 (8.3%) and 12 (10%) hospitals changed into or out of the lowest-performing quartile, respectively. CONCLUSIONS: An expanded set of variables improved colon SSI risk predictions and quartile assignments, but low procedure volumes and SSI events remain a barrier to effectively comparing hospitals.

Underuse of oral anticoagulants in privately insured patients with atrial fibrillation: A population being targeted by the IMplementation of a randomized controlled trial to imProve treatment with oral AntiCoagulanTs in patients with Atrial Fibrillation (IMPACT-AFib).

BACKGROUND: Many studies showing underuse of oral anticoagulants (OACs) in patients with atrial fibrillation (AF) predated the advent of the non-vitamin K antagonist OACs. We retrospectively examined use of OACs in a large commercially insured population. METHODS: Administrative claims data from 4 research partners participating in FDA-Catalyst, a program of the Sentinel Initiative, were queried in September 2017. Patients were included if they were ≥30 years old with ≥365 days of medical/pharmacy coverage, and had ≥2 diagnosis codes for AF, a CHA2DS2-VASc score ≥2, absence of contraindications to OAC use, and no evidence of OAC use in the 365 days before the index AF diagnosis. The main outcome measures of the current analysis were rates of OAC use in the prior 12 months of cohort identification and factors associated with non-use. RESULTS: A total of 197,806 AF patients met the eligibility criteria prior to assessment of OAC treatment. Of these, 179,580 (91%) patients were ≥65 years old and 73,286 (37%) patients were ≥80 years old. Half of the patients (98,903) were randomized to the early intervention arm in the IMPACT-AFib trial and constitute the cohort for this analysis. Of these, 32,295 (33%) had no evidence of OAC use in the prior 12 months. Compared with patients with evidence of OAC use in the prior 12 months, patients without OAC use were more likely to be ≥80 years old, women, and have a history of anemia (51% vs 47%) and less likely to have diabetes (41% vs 44%), history of stroke or TIA (15% vs 19%), and history of heart failure (39% vs 48%). CONCLUSIONS: Despite a high risk of stroke, one-third of privately insured patients with AF and no obvious contraindications to an OAC were not treated with an OAC. There is an unmet need for evidence-based interventions that could lead to greater use of OACs in patients with AF at risk for stroke.

Practical challenges in the conduct of pragmatic trials embedded in health plans: Lessons of IMPACT-AFib, an FDA-Catalyst trial.

IMPACT-AFib was an 80,000-patient randomized clinical trial implemented by five US insurance companies (health plans) aimed at increasing the use of oral anticoagulants by individuals with atrial fibrillation who were at high risk of stroke and not on treatment. The underlying thesis was that patients could be change agents to initiate prescribing discussions with their providers. We tested the effect of mailing information to both patients and their providers. We used administrative medical claims and pharmacy dispensing data to identify eligible patients, to randomize them to an early or delayed intervention, and to assess clinical outcomes. The core data were analysis-ready datasets each site had created and curated for the FDA's Sentinel System, supplemented by updated "fresh" pharmacy and enrollment data to ensure eligibility at the time of intervention. Following mutually agreed upon procedures, sites linked to additional internal source data to implement the intervention-educational information mailed to patients and their providers in the early intervention arm, and to providers of patients in the delayed intervention arm approximately 12 months later. The primary analysis compares the early intervention arm to the delayed intervention arm, prior to the delayed intervention being conducted (i.e. compares intervention to non-intervention). The endpoints of interest were evidence of initiation of anticoagulation (primary) as well as clinical endpoints, including stroke and hospitalization for bleeding. Major challenges, some unanticipated, identified during the planning phase include convening multi-stakeholder investigator teams and advisors, addressing ethical concerns about not intervening in a usual care comparison group, and identifying and avoiding interference with sites' routine programs that were similar to the intervention. Needs and challenges during the implementation phase included the fact that even limited site-specific programming greatly increased time and effort, the need to refresh research data extracts immediately before outreach to patients and providers, potential difficulty identifying low-cost medications such as warfarin that may not be reimbursed by health plans and so not discoverable in dispensing data, the need to develop workarounds when "providers" in claims data were facilities, difficulty addressing clustering of patients by provider because providers can have multiple identifiers within and between health plans, and the need to anticipate loss to follow up because of health plan disenrollment or change in benefits. As pragmatic trials begin to shape evidence generation within clinical practice, investigators should anticipate issues inherent to claims data and working with multiple large sites. In IMPACT-AFib, we found that investing in collaboration and communication among all parties throughout all phases of the study helped ensure common understanding, early identification of challenges, and streamlined actual implementation.

Using Healthcare Data in Embedded Pragmatic Clinical Trials among People Living with Dementia and Their Caregivers: State of the Art.

Embedded pragmatic clinical trials (ePCTs) are embedded in healthcare systems as well as their data environments. For people living with dementia (PLWD), settings of care can be different from the general population and involve additional people whose information is also important. The ePCT designs have the opportunity to leverage data that becomes available through the normal delivery of care. They may be particularly valuable in Alzheimer's disease and Alzheimer's disease-related dementia (AD/ADRD), given the complexity of case identification and the diversity of care settings. Grounded in the objectives of the Data and Technical Core of the newly established National Institute on Aging Imbedded Pragmatic Alzheimer's Disease and AD-Related Dementias Clinical Trials Collaboratory (IMPACT Collaboratory), this article summarizes the state of the art in using existing data sources (eg, Medicare claims, electronic health records) in AD/ADRD ePCTs and approaches to integrating them in real-world settings. J Am Geriatr Soc 68:S49-S54, 2020.

Bystander Ethics and Good Samaritanism: A Paradox for Learning Health Organizations.

In 2012, a U.S. Institute of Medicine report called for a different approach to health care: "Left unchanged, health care will continue to underperform; cause unnecessary harm; and strain national, state, and family budgets." The answer, they suggested, would be a "continuously learning" health system. Ethicists and researchers urged the creation of "learning health organizations" that would integrate knowledge from patient-care data to continuously improve the quality of care. Our experience with an ongoing research study on atrial fibrillation-a trial known as IMPACT-AFib-gave us some insight into one of the challenges that will have to be dealt with in creating these organizations. Although the proposed educational intervention study placed no restrictions on what providers and health plans could do, the oversight team argued that the ethical principle of beneficence did not allow the researchers to be "bystanders" in relation to a control group receiving suboptimal care. In response, the researchers designed a "workaround" that allowed the project to go forward. We believe the experience suggests that what we call "bystander ethics" will create challenges for the kinds of quality improvement research that LHOs are designed to do.

The Emergence of Population Health in US Academic Medicine: A Qualitative Assessment.

Importance: In response to rapidly growing interest in population health, academic medical centers are launching department-level initiatives that focus on this evolving discipline. This trend, with its potential to extend the scope of academic medicine, has not been well characterized. Objective: To describe the emergence of departments of population health at academic medical centers in the United States, including shared areas of focus, opportunities, and challenges. Design, Setting, and Participants: This qualitative study was based on a structured in-person convening of a working group of chairs of population health-oriented departments on November 13 and 14, 2017, complemented by a survey of core characteristics of these and additional departments identified through web-based review of US academic medical centers. United States medical school departments with the word population in their name were included. Centers, institutes, and schools were not included. Main Outcomes and Measures: Departments were characterized by year of origin, areas of focus, organizational structure, faculty size, teaching programs, and service engagement. Opportunities and challenges faced by these emerging departments were grouped thematically and described. Results: Eight of 9 population health-oriented departments in the working group were launched in the last 6 years. The 9 departments had 5 to 97 full-time faculty. Despite varied organizational structures, all addressed essential areas of focus spanning the missions of research, education, and service. Departments varied significantly in their relationships with the delivery of clinical care, but all engaged in practice-based and/or community collaboration. Common attributes include core attention to population health-oriented research methods across disciplines, emphasis on applied research in frontline settings, strong commitment to partnership, interest in engaging other sectors, and focus on improving health equity. Tensions included defining boundaries with other academic units with overlapping areas of focus, identifying sources of sustainable extramural funding, and facilitating the interface between research and health system operations. Conclusions and Relevance: Departments addressing population health are emerging rapidly in academic medical centers. In supporting this new framing, academic medicine affirms and strengthens its commitment to advancing population health and health equity, to improving the quality and effectiveness of care, and to upholding the social mission of medicine.

FDA-Catalyst-Using FDA's Sentinel Initiative for large-scale pragmatic randomized trials: Approach and lessons learned during the planning phase of the first trial.

BACKGROUND: The US Food and Drug Administration's Sentinel Initiative is well positioned to support pragmatic clinical trials. FDA-Catalyst combines direct contact with health plan members and/or providers with data in the Sentinel infrastructure. Here, we describe the rationale, feasibility analyses, and lessons learned from the planning phase of the first large pragmatic trial conducted using the Sentinel Initiative's delivery system capabilities-IMplementation of a randomized controlled trial to imProve treatment with oral AntiCoagulanTs in patients with Atrial Fibrillation (the IMPACT-AFib trial). METHODS: During the planning phase, we convened representatives from five commercial health plans, FDA, study coordinating centers, and a patient representative for protocol development, institutional review board preparation, and other activities. Administrative claims data from the plans were included in a retrospective cohort analysis to assess sample size for the trial. Members ≥30 years old with ≥365 days of medical/pharmacy coverage, ≥2 diagnosis codes for atrial fibrillation, a guideline-based indication for oral anticoagulant use for stroke prevention, and no evidence of oral anticoagulant use in the 365 days prior to the index atrial fibrillation diagnosis in 2013 were included. Exclusions for the analysis included other conditions requiring anticoagulation, history of intracranial hemorrhage, and gastrointestinal bleed. We calculated rates of oral anticoagulant use, transient ischemic attack or stroke, and bleeding in the 365 days following the index atrial fibrillation diagnosis. RESULTS: A total of 44,786 members with atrial fibrillation with no evidence of recent oral anticoagulant use were identified. In total, 87% (n = 38,759) were classified as having a guideline-based indication for oral anticoagulants. Of those, 33% (n = 12,867) had a new oral anticoagulant dispensed during the following year, 15% (n = 5917) were hospitalized for stroke or transient ischemic attack, and 9% (n = 3469) for bleeding events. This information was used to develop the trial protocol including sample size, power calculations, and level of randomization. CONCLUSION: Sentinel infrastructure generated preliminary data that supported planning and implementation of a large pragmatic trial embedded in health plans. This planning identified unanticipated challenges that must be addressed in similar trials.

Pragmatic clinical trials embedded in healthcare systems: generalizable lessons from the NIH Collaboratory.

BACKGROUND: The clinical research enterprise is not producing the evidence decision makers arguably need in a timely and cost effective manner; research currently involves the use of labor-intensive parallel systems that are separate from clinical care. The emergence of pragmatic clinical trials (PCTs) poses a possible solution: these large-scale trials are embedded within routine clinical care and often involve cluster randomization of hospitals, clinics, primary care providers, etc. Interventions can be implemented by health system personnel through usual communication channels and quality improvement infrastructure, and data collected as part of routine clinical care. However, experience with these trials is nascent and best practices regarding design operational, analytic, and reporting methodologies are undeveloped. METHODS: To strengthen the national capacity to implement cost-effective, large-scale PCTs, the Common Fund of the National Institutes of Health created the Health Care Systems Research Collaboratory (Collaboratory) to support the design, execution, and dissemination of a series of demonstration projects using a pragmatic research design. RESULTS: In this article, we will describe the Collaboratory, highlight some of the challenges encountered and solutions developed thus far, and discuss remaining barriers and opportunities for large-scale evidence generation using PCTs. CONCLUSION: A planning phase is critical, and even with careful planning, new challenges arise during execution; comparisons between arms can be complicated by unanticipated changes. Early and ongoing engagement with both health care system leaders and front-line clinicians is critical for success. There is also marked uncertainty when applying existing ethical and regulatory frameworks to PCTS, and using existing electronic health records for data capture adds complexity.

Surgical Site Infections: Volume-Outcome Relationship and Year-to-Year Stability of Performance Rankings.

BACKGROUND: Surgical site infection (SSI) rates are publicly reported as quality metrics and increasingly used to determine financial reimbursement. OBJECTIVE: To evaluate the volume-outcome relationship as well as the year-to-year stability of performance rankings following coronary artery bypass graft (CABG) surgery and hip arthroplasty. RESEARCH DESIGN: We performed a retrospective cohort study of Medicare beneficiaries who underwent CABG surgery or hip arthroplasty at US hospitals from 2005 to 2011, with outcomes analyzed through March 2012. Nationally validated claims-based surveillance methods were used to assess for SSI within 90 days of surgery. The relationship between procedure volume and SSI rate was assessed using logistic regression and generalized additive modeling. Year-to-year stability of SSI rates was evaluated using logistic regression to assess hospitals' movement in and out of performance rankings linked to financial penalties. RESULTS: Case-mix adjusted SSI risk based on claims was highest in hospitals performing <50 CABG/year and <200 hip arthroplasty/year compared with hospitals performing ≥200 procedures/year. At that same time, hospitals in the worst quartile in a given year based on claims had a low probability of remaining in that quartile the following year. This probability increased with volume, and when using 2 years' experience, but the highest probabilities were only 0.59 for CABG (95% confidence interval, 0.52-0.66) and 0.48 for hip arthroplasty (95% confidence interval, 0.42-0.55). CONCLUSIONS: Aggregate SSI risk is highest in hospitals with low annual procedure volumes, yet these hospitals are currently excluded from quality reporting. Even for higher volume hospitals, year-to-year random variation makes past experience an unreliable estimator of current performance.

Impact of Policies on the Rise in Sepsis Incidence, 2000-2010.

BACKGROUND: Sepsis hospitalizations have increased dramatically in the last decade. It is unclear whether this represents an actual rise in sepsis illness or improved capture by coding. We evaluated the impact of Centers of Medicare and Medicaid Services (CMS) guidance after newly introduced sepsis codes and medical severity diagnosis-related group (MS-DRG) systems on sepsis trends. METHODS: In this retrospective cohort study of California hospitalizations from January 2000 to December 2010, sepsis was identified by International Classification of Diseases, Ninth Revision (ICD-9) coding (Dombrovskiy method). Sepsis-associated mortality rates were calculated. Logistic regression models evaluated variables associated with sepsis and mortality. Segmented regression time series analysis assessed changes in sepsis frequency for (1) baseline (January 2000 to September 2003); (2) post-CMS guidelines on sepsis coding (October 2003 to September 2007); and (3) after the introduction of MS-DRG (October 2007 to December 2010). RESULTS: Annual hospitalizations with sepsis diagnoses tripled within a decade, from 21.1 to 59.9 cases per 1000 admissions, with a 2.8- and 2.0-fold increase in severe and nonsevere sepsis, respectively, whereas annual admissions remained unchanged and sepsis-associated mortality decreased. Greatest increases were seen for severe sepsis present on admission (3.8-fold increase). Increases in sepsis were temporally correlated with CMS coding guidance and MS-DRG introduction after adjustment for comorbidity and other factors. CONCLUSIONS: Sepsis rate increases were associated with introduction of CMS-issued guidance for new sepsis ICD-9 coding and MS-DRGs. Coding artifact ("up-capture" of less severely ill septic patients) may be contributing to the apparent rise in sepsis incidence and decline in mortality. Epidemiologic trends based on administrative data should account for policy-related effects.

Objective Sepsis Surveillance Using Electronic Clinical Data.

OBJECTIVE: To compare the accuracy of surveillance of severe sepsis using electronic health record clinical data vs claims and to compare incidence and mortality trends using both methods. DESIGN: We created an electronic health record-based surveillance definition for severe sepsis using clinical indicators of infection (blood culture and antibiotic orders) and concurrent organ dysfunction (vasopressors, mechanical ventilation, and/or abnormal laboratory values). We reviewed 1,000 randomly selected medical charts to characterize the definition's accuracy and stability over time compared with a claims-based definition requiring infection and organ dysfunction codes. We compared incidence and mortality trends from 2003-2012 using both methods. SETTING: Two US academic hospitals. PATIENTS: Adult inpatients. RESULTS: The electronic health record-based clinical surveillance definition had stable and high sensitivity over time (77% in 2003-2009 vs 80% in 2012, P=.58) whereas the sensitivity of claims increased (52% in 2003-2009 vs 67% in 2012, P=.02). Positive predictive values for claims and clinical surveillance definitions were comparable (55% vs 53%, P=.65) and stable over time. From 2003 to 2012, severe sepsis incidence imputed from claims rose by 72% (95% CI, 57%-88%) and absolute mortality declined by 5.4% (95% CI, 4.6%-6.7%). In contrast, incidence using the clinical surveillance definition increased by 7.7% (95% CI, -1.1% to 17%) and mortality declined by 1.7% (95% CI, 1.1%-2.3%). CONCLUSIONS: Sepsis surveillance using clinical data is more sensitive and more stable over time compared with claims and can be done electronically. This may enable more reliable estimates of sepsis burden and trends.

Advanced Clinical Decision Support for Vaccine Adverse Event Detection and Reporting.

BACKGROUND: Reporting of adverse events (AEs) following vaccination can help identify rare or unexpected complications of immunizations and aid in characterizing potential vaccine safety signals. We developed an open-source, generalizable clinical decision support system called Electronic Support for Public Health-Vaccine Adverse Event Reporting System (ESP-VAERS) to assist clinicians with AE detection and reporting. METHODS: ESP-VAERS monitors patients' electronic health records for new diagnoses, changes in laboratory values, and new allergies following vaccinations. When suggestive events are found, ESP-VAERS sends the patient's clinician a secure electronic message with an invitation to affirm or refute the message, add comments, and submit an automated, prepopulated electronic report to VAERS. High-probability AEs are reported automatically if the clinician does not respond. We implemented ESP-VAERS in December 2012 throughout the MetroHealth System, an integrated healthcare system in Ohio. We queried the VAERS database to determine MetroHealth's baseline reporting rates from January 2009 to March 2012 and then assessed changes in reporting rates with ESP-VAERS. RESULTS: In the 8 months following implementation, 91 622 vaccinations were given. ESP-VAERS sent 1385 messages to responsible clinicians describing potential AEs. Clinicians opened 1304 (94.2%) messages, responded to 209 (15.1%), and confirmed 16 for transmission to VAERS. An additional 16 high-probability AEs were sent automatically. Reported events included seizure, pleural effusion, and lymphocytopenia. The odds of a VAERS report submission during the implementation period were 30.2 (95% confidence interval, 9.52-95.5) times greater than the odds during the comparable preimplementation period. CONCLUSIONS: An open-source, electronic health record-based clinical decision support system can increase AE detection and reporting rates in VAERS.