Medical subdomain classification of clinical notes using a machine learning-based natural language processing approach.

BACKGROUND: The medical subdomain of a clinical note, such as cardiology or neurology, is useful content-derived metadata for developing machine learning downstream applications. To classify the medical subdomain of a note accurately, we have constructed a machine learning-based natural language processing (NLP) pipeline and developed medical subdomain classifiers based on the content of the note. METHODS: We constructed the pipeline using the clinical NLP system, clinical Text Analysis and Knowledge Extraction System (cTAKES), the Unified Medical Language System (UMLS) Metathesaurus, Semantic Network, and learning algorithms to extract features from two datasets - clinical notes from Integrating Data for Analysis, Anonymization, and Sharing (iDASH) data repository (n = 431) and Massachusetts General Hospital (MGH) (n = 91,237), and built medical subdomain classifiers with different combinations of data representation methods and supervised learning algorithms. We evaluated the performance of classifiers and their portability across the two datasets. RESULTS: The convolutional recurrent neural network with neural word embeddings trained-medical subdomain classifier yielded the best performance measurement on iDASH and MGH datasets with area under receiver operating characteristic curve (AUC) of 0.975 and 0.991, and F1 scores of 0.845 and 0.870, respectively. Considering better clinical interpretability, linear support vector machine-trained medical subdomain classifier using hybrid bag-of-words and clinically relevant UMLS concepts as the feature representation, with term frequency-inverse document frequency (tf-idf)-weighting, outperformed other shallow learning classifiers on iDASH and MGH datasets with AUC of 0.957 and 0.964, and F1 scores of 0.932 and 0.934 respectively. We trained classifiers on one dataset, applied to the other dataset and yielded the threshold of F1 score of 0.7 in classifiers for half of the medical subdomains we studied. CONCLUSION: Our study shows that a supervised learning-based NLP approach is useful to develop medical subdomain classifiers. The deep learning algorithm with distributed word representation yields better performance yet shallow learning algorithms with the word and concept representation achieves comparable performance with better clinical interpretability. Portable classifiers may also be used across datasets from different institutions.

Efficacy of a clinical decision-support system in an HIV practice: a randomized trial.

BACKGROUND: Data to support improved patient outcomes from clinical decision-support systems (CDSSs) are lacking in HIV care. OBJECTIVE: To test the efficacy of a CDSS in improving HIV outcomes in an outpatient clinic. DESIGN: Randomized, controlled trial. (ClinicalTrials.gov registration number: NCT00678600) SETTING: Massachusetts General Hospital HIV Clinic. PARTICIPANTS: HIV care providers and their patients. INTERVENTION: Computer alerts were generated for virologic failure (HIV RNA level >400 copies/mL after a previous HIV RNA level ≤400 copies/mL), evidence of suboptimal follow-up, and 11 abnormal laboratory test results. Providers received interactive computer alerts, facilitating appointment rescheduling and repeated laboratory testing, for half of their patients and static alerts for the other half. MEASUREMENTS: The primary end point was change in CD4 cell count. Other end points included time to clinical event, 6-month suboptimal follow-up, and severe laboratory toxicity. RESULTS: Thirty-three HIV care providers followed 1011 patients with HIV. In the intervention group, the mean increase in CD4 cell count was greater (0.0053 vs. 0.0032 × 109 cells/L per month; difference, 0.0021 × 109 cells/L per month [95% CI, 0.0001 to 0.004]; P = 0.040) and the rate of 6-month suboptimal follow-up was lower (20.6 vs. 30.1 events per 100 patient-years; P = 0.022) than those in the control group. Median time to next scheduled appointment was shorter in the intervention group than in the control group after a suboptimal follow-up alert (1.71 vs. 3.48 months; P < 0.001) and after a toxicity alert (2.79 vs. >6 months; P = 0.072). More than 90% of providers supported adopting the CDSS as part of standard care. LIMITATION: This was a 1-year informatics study conducted at a single hospital subspecialty clinic. CONCLUSION: A CDSS using interactive provider alerts improved CD4 cell counts and clinic follow-up for patients with HIV. Wider implementation of such systems can provide important clinical benefits. PRIMARY FUNDING SOURCE: National Institute of Allergy and Infectious Diseases.

A cluster-randomized trial of a primary care informatics-based system for breast cancer screening.

BACKGROUND: Information technology offers the promise, as yet unfulfilled, of delivering efficient, evidence-based health care. OBJECTIVE: To evaluate whether a primary care network-based informatics intervention can improve breast cancer screening rates. DESIGN: Cluster-randomized controlled trial of 12 primary care practices conducted from March 20, 2007 to March 19, 2008. PATIENTS: Women 42-69 years old with no record of a mammogram in the prior 2 years. INTERVENTIONS: In intervention practices, a population-based informatics system was implemented that: connected overdue patients to appropriate care providers, presented providers with a Web-based list of their overdue patients in a non-visit-based setting, and enabled "one-click" mammography ordering or documented deferral reasons. Patients selected for mammography received automatically generated letters and follow-up phone calls. All practices had electronic health record reminders about breast cancer screening available during clinical encounters. MAIN MEASURES: The primary outcome was the proportion of overdue women undergoing mammography at 1-year follow-up. KEY RESULTS: Baseline mammography rates in intervention and control practices did not differ (79.5% vs 79.3%, p = 0.73). Among 3,054 women in intervention practices and 3,676 women in control practices overdue for mammograms, intervention patients were somewhat younger, more likely to be non-Hispanic white, and have health insurance. Most intervention providers used the system (65 of 70 providers, 92.9%). Action was taken for 2,652 (86.8%) intervention patients [2,274 (74.5%) contacted and 378 (12.4%) deferred]. After 1 year, mammography rates were significantly higher in the intervention arm (31.4% vs 23.3% in control arm, p < 0.001 after adjustment for baseline differences; 8.1% absolute difference, 95% CI 5.1-11.2%). All demographic subgroups benefited from the intervention. Intervention patients completed screening sooner than control patients (p < 0.001). CONCLUSIONS: A novel population-based informatics system functioning as part of a non-visit-based care model increased mammography screening rates in intervention practices. TRIAL REGISTRATION: ClinicalTrials.gov; NCT00462891.

Linking electronic health record-extracted psychosocial data in real-time to risk of readmission for heart failure.

BACKGROUND: Knowledge of psychosocial characteristics that helps to identify patients at increased risk for readmission for heart failure (HF) may facilitate timely and targeted care. OBJECTIVE: We hypothesized that certain psychosocial characteristics extracted from the electronic health record (EHR) would be associated with an increased risk for hospital readmission within the next 30 days. METHODS: We identified 15 psychosocial predictors of readmission. Eleven of these were extracted from the EHR (six from structured data sources and five from unstructured clinical notes). We then analyzed their association with the likelihood of hospital readmission within the next 30 days among 729 patients admitted for HF. Finally, we developed a multivariable predictive model to recognize individuals at high risk for readmission. RESULTS: We found five characteristics-dementia, depression, adherence, declining/refusal of services, and missed clinical appointments-that were associated with an increased risk for hospital readmission: the first four features were captured from unstructured clinical notes, while the last item was captured from a structured data source. CONCLUSIONS: Unstructured clinical notes contain important knowledge on the relationship between psychosocial risk factors and an increased risk of readmission for HF that would otherwise have been missed if only structured data were considered. Gathering this EHR-based knowledge can be automated, thus enabling timely and targeted care.

Queuing theory to guide the implementation of a heart failure inpatient registry program.

OBJECTIVE The authors previously implemented an electronic heart failure registry at a large academic hospital to identify heart failure patients and to connect these patients with appropriate discharge services. Despite significant improvements in patient identification and connection rates, time to connection remained high, with an average delay of 3.2 days from the time patients were admitted to the time connections were made. Our objective for this current study was to determine the most effective solution to minimize time to connection. DESIGN We used a queuing theory model to simulate 3 different potential solutions to decrease the delay from patient identification to connection with discharge services. MEASUREMENTS The measures included average rate at which patients were being connected to the post discharge heart failure services program, average number of patients in line, and average patient waiting time. RESULTS Using queuing theory model simulations, we were able to estimate for our current system the minimum rate at which patients need to be connected (262 patients/mo), the ideal patient arrival rate (174 patients/mo) and the maximal patient arrival rate that could be achieved by adding 1 extra nurse (348 patients/mo). CONCLUSIONS Our modeling approach was instrumental in helping us characterize key process parameters and estimate the impact of adding staff on the time between identifying patients with heart failure and connecting them with appropriate discharge services.

Mammography FastTrack: an intervention to facilitate reminders for breast cancer screening across a heterogeneous multi-clinic primary care network.

Health care information technology can be a means to improve quality and efficiency in the primary care setting. However, merely applying technology without addressing how it fits into provider workflow and existing systems is unlikely to achieve improvement goals. Improving quality of primary care, such as cancer screening rates, requires addressing barriers at system, provider, and patient levels. The authors report the development, implementation, and preliminary use of a new breast cancer screening outreach program in a large multicenter primary care network. This installation paired population-based surveillance with customized information delivery based on a validated model linking patients to providers and practices. In the first six months, 86% of physicians and all case managers voluntarily participated in the program. Providers intervened in 83% of the mammogram-overdue population by initiating mailed reminders or deferring contact. Overall, 63% of patients were successfully contacted. Systematic population-based efforts are promising tools to improve preventative care.

Lessons from implementing a combined workflow-informatics system for diabetes management.

Shortcomings surrounding the care of patients with diabetes have been attributed largely to a fragmented, disorganized, and duplicative health care system that focuses more on acute conditions and complications than on managing chronic disease. To address these shortcomings, we developed a diabetes registry population management application to change the way our staff manages patients with diabetes. Use of this new application has helped us coordinate the responsibilities for intervening and monitoring patients in the registry among different users. Our experiences using this combined workflow-informatics intervention system suggest that integrating a chronic disease registry into clinical workflow for the treatment of chronic conditions creates a useful and efficient tool for managing disease.

Designing healthcare information technology to catalyse change in clinical care.

The gap between best practice and actual patient care continues to be a pervasive problem in our healthcare system. Efforts to improve on this knowledge-performance gap have included computerised disease management programs designed to improve guideline adherence. However, current computerised reminder and decision support interventions directed at changing physician behaviour have had only a limited and variable effect on clinical outcomes. Further, immediate pay-for-performance financial pressures on institutions have created an environment where disease management systems are often created under duress, appended to existing clinical systems and poorly integrated into the existing workflow, potentially limiting their real-world effectiveness. The authors present a review of disease management as well as a conceptual framework to guide the development of more effective health information technology (HIT) tools for translating clinical information into clinical action.

A self-scaling, distributed information architecture for public health, research, and clinical care.

OBJECTIVE: This study sought to define a scalable architecture to support the National Health Information Network (NHIN). This architecture must concurrently support a wide range of public health, research, and clinical care activities. STUDY DESIGN: The architecture fulfils five desiderata: (1) adopt a distributed approach to data storage to protect privacy, (2) enable strong institutional autonomy to engender participation, (3) provide oversight and transparency to ensure patient trust, (4) allow variable levels of access according to investigator needs and institutional policies, (5) define a self-scaling architecture that encourages voluntary regional collaborations that coalesce to form a nationwide network. RESULTS: Our model has been validated by a large-scale, multi-institution study involving seven medical centers for cancer research. It is the basis of one of four open architectures developed under funding from the Office of the National Coordinator of Health Information Technology, fulfilling the biosurveillance use case defined by the American Health Information Community. The model supports broad applicability for regional and national clinical information exchanges. CONCLUSIONS: This model shows the feasibility of an architecture wherein the requirements of care providers, investigators, and public health authorities are served by a distributed model that grants autonomy, protects privacy, and promotes participation.

Randomized controlled trial of an informatics-based intervention to increase statin prescription for secondary prevention of coronary disease.

OBJECTIVE: Suboptimal treatment of hyperlipidemia in patients with coronary artery disease (CAD) is well documented. We report the impact of a computer-assisted physician-directed intervention to improve secondary prevention of hyperlipidemia. DESIGN AND SETTING: Two hundred thirty-five patients under the care of 14 primary care physicians in an academically affiliated practice with an electronic health record were enrolled in this proof-of-concept physician-blinded randomized, controlled trial. Each patient with CAD or risk equivalent above National Cholesterol Education Program-recommended low-density lipoprotein (LDL) treatment goal for greater than 6 months was randomized, stratified by physician and baseline LDL. Physicians received a single e-mail per intervention patient. E-mails were visit independent, provided decision support, and facilitated "one-click" order writing. MEASUREMENTS: The primary outcomes were changes in hyperlipidemia prescriptions, time to prescription change, and changes in LDL levels. The time spent using the system was assessed among intervention patients. RESULTS: A greater proportion of intervention patients had prescription changes at 1 month (15.3% vs 2%, P=.001) and 1 year (24.6% vs 17.1%, P=.14). The median interval to first medication adjustment occurred earlier among intervention patients (0 vs 7.1 months, P=.005). Among patients with baseline LDLs >130 mg/dL, the first postintervention LDLs were substantially lower in the intervention group (119.0 vs 138.0 mg/dL, P=.04). Physician processing time was under 60 seconds per e-mail. CONCLUSION: A visit-independent disease management tool resulted in significant improvement in secondary prevention of hyperlipidemia at 1-month postintervention and showed a trend toward improvement at 1 year.

Is this "my" patient? Development and validation of a predictive model to link patients to primary care providers.

BACKGROUND: Evaluating the quality of care provided by individual primary care physicians (PCPs) may be limited by failing to know which patients the PCP feels personally responsible for. OBJECTIVE: To develop and validate a model for linking patients to specific PCPs. DESIGN: Retrospective convenience sample. PARTICIPANTS: Eighteen PCPs from 10 practice sites within an academic adult primary care network. MEASUREMENTS: Each PCP reviewed the records for all outpatients seen over the preceding 3 years (16,435 patients reviewed) and designated each patient as "My Patient" or "Not My Patient." Using this reference standard, we developed an algorithm with logistic regression modeling to predict "My Patient" using development and validation subsets drawn from the same patient set. Quality of care was then assessed by "My Patient" or "Not My Patient" designation by analyzing cancer screening test rates. RESULTS: Overall, PCPs designated 11,226 patients (68.3%, range per provider 15% to 93%) to be "My Patient." The model accurately categorized patients in development and validation subsets (combined sensitivity 80.4%, specificity 93.7%, and positive predictive value 96.5%). To achieve positive predictive values of > 90% for individual PCPs, the model excluded 19.6% of PCP "My Patients" (range 5.5% to 75.3%). Cancer screening rates were higher among model-predicted "My Patients." CONCLUSIONS: Nearly one-third of patients seen were considered "Not My Patient" by the PCP, although this proportion varied widely. We developed and validated a simple model to link specific patients and PCPs. Such efforts may help effectively target interventions to improve primary care quality.

Automated identification of a physician's primary patients.

OBJECTIVE: To develop and validate an automated method for determining the set of patients for whom a given primary care physician holds overall clinical responsibility. DESIGN: The study included all adult patients (16,185) seen at least once in an ambulatory setting during a three-year period by 18 primary care physicians in ten practices. The physicians indicated whether they considered themselves to be the physician primarily responsible for the overall clinical care of each visiting patient. Statistical models were constructed to predict the physicians' designations using predictor variables derived from electronically available appointment schedules and demographic information. MEASUREMENTS: Predictive accuracy was assessed primarily using the area under the receiver-operating characteristic curve (AUC), and secondarily using positive predictive value (PPV) and sensitivity. RESULTS: A minimal set of six variables was identified as predictive of the physicians' designations. The constructed model had a median AUC for individual physicians of 0.92 (interquartile interval: 0.90-0.96), a PPV of 0.94 (interquartile interval: 0.87-0.95), and a sensitivity of 0.95 (interquartile interval: 0.87-0.97). CONCLUSION: A statistical model using a minimal set of commonly available electronic data can accurately predict the set of patients for whom a physician holds primary clinical responsibility. Further research examining the generalization of the model to other settings would be valuable.

Design and implementation of an application and associated services to support interdisciplinary medication reconciliation efforts at an integrated healthcare delivery network.

Confusion about patients' medication regimens during the hospital admission and discharge process accounts for many preventable and serious medication errors. Many organizations have begun to redesign their clinical processes to address this patient safety concern. Partners HealthCare, an integrated delivery network in Boston, Massachusetts, has answered this interdisciplinary challenge by leveraging its multiple outpatient electronic medical records (EMR) and inpatient computerized provider order entry (CPOE) systems to facilitate the process of medication reconciliation. This manuscript describes the design of a novel application and the associated services that aggregate medication data from EMR and CPOE systems so that clinicians can efficiently generate an accurate pre-admission medication list. Information collected with the use of this application subsequently supports the writing of admission and discharge orders by physicians, performance of admission assessment by nurses, and reconciliation of inpatient orders by pharmacists. Results from early pilot testing suggest that this new medication reconciliation process is well accepted by clinicians and has significant potential to prevent medication errors during transitions of care.

New models of population management for patients with diabetes--using informatics tools to support primary care.

Diabetes management continues to fall short of evidence-based goals of care. Population management represents a new approach to diabetes care for large numbers of patients with diabetes cared for within a single clinical system. This method is information intensive and generally requires an advanced informatics infrastructure. While Information Processing is a critical first step in population management, to have a significant impact on disease control population-based intervention must also employ potent Clinical Action tools that lower barriers to effective care. In this review we present two recent population management interventions within our health system that illustrate the principles of Information Processing and Clinical Action in diabetes care.

Internet use among primary care patients with type 2 diabetes: the generation and education gap.

BACKGROUND: The Internet represents a promising tool to improve diabetes care. OBJECTIVE: To assess differences in demographics, self-care behaviors, and diabetes-related risk factor control by frequency of Internet use. DESIGN AND PARTICIPANTS: We surveyed 909 patients with type 2 diabetes attending primary care clinics. MEASUREMENTS: Frequency of Internet use, socioeconomic status, and responses to the Problem Areas in Diabetes (PAID), Summary of Diabetes Self-care Activities (SDSCA), and Health Utilities Index (HUI) scales. Survey responses were linked to last measured hemoglobin A1c, cholesterol, and blood pressure results. Comorbidities and current medications were obtained from the medical record. RESULTS: Internet "never-users" (n=588, 66%) were significantly older (70.0+/-11.2 vs 59.0+/-11.3 years; P<.001) and less educated (26% vs 71% with>high school; P<.001) than Internet users (n=308, 34%). There were few significant differences in PAID or SDSCA scores or in diabetes metabolic control despite longer diabetes duration (10.3+/-8.2 vs 8.3+/-6.7 years; P<.001) and greater prevalence of coronary disease (40% vs 24%; P<.001) in nonusers. Less than 10% of current nonusers would use the Internet for secure health-related communication. CONCLUSIONS: Older and less educated diabetes patients are less likely to use the Internet. Despite greater comorbidity, nonusers engaged in primary care had equal or better risk factor control compared to users.

Impact of population management with direct physician feedback on care of patients with type 2 diabetes.

OBJECTIVE: Population-level strategies may improve primary care for diabetes. We designed a controlled study to assess the impact of population management versus usual care on metabolic risk factor testing and management in patients with type 2 diabetes. We also identified potential patient-related barriers to effective diabetes management. RESEARCH DESIGN AND METHODS: We used novel clinical software to rank 910 patients in a diabetes registry at a single primary care clinic and thereby identify the 149 patients with the highest HbA(1c) and cholesterol levels. After review of the medical records of these 149 patients, evidence-based guideline recommendations regarding metabolic testing and management were sent via e-mail to each intervention patient's primary care provider (PCP). Over a 3-month follow-up period, we assessed changes in the evidence-based management of intervention patients compared with a matched cohort of control patients receiving usual care at a second primary care clinic affiliated with the same academic medical center. RESULTS: In the intervention cohort, PCPs followed testing recommendations more often (78%) than therapeutic change recommendations (36%, P = 0.001). Compared with the usual care control cohort, population management resulted in a greater overall proportion of evidence-based guideline practices being followed (59 vs. 45%, P = 0.02). Most intervention patients (62%) had potential barriers to effective care, including depression (35%), substance abuse (26%), and prior nonadherence to care plans (18%). CONCLUSIONS: Population management with clinical recommendations sent to PCPs had a modest but statistically significant impact on the evidence-based management of diabetes compared with usual care. Depression and substance abuse are prevalent patient-level adherence barriers in patients with poor metabolic control.

A controlled trial of population management: diabetes mellitus: putting evidence into practice (DM-PEP).

OBJECTIVE: Population-level strategies to organize and deliver care may improve diabetes management. We conducted a multiclinic controlled trial of population management in patients with type 2 diabetes. RESEARCH DESIGN AND METHODS: We created diabetic patient registries (n = 3,079) for four primary care clinics within a single academic health center. In the intervention clinic (n = 898), a nurse practitioner used novel clinical software (PopMan) to identify patients on a weekly basis with outlying values for visit and testing intervals and last measured levels of HbA1c, LDL cholesterol, and blood pressure. For these patients, the nurse practitioner e-mailed a concise patient-specific summary of evidence-based management suggestions directly to primary care providers (PCPs). Population changes in risk factor testing, medication prescription, and risk factor levels from baseline (1 January 2000 to 31 August 2001) to follow-up (1 December 2001 to 31 July 2003) were compared with the three usual-care control clinics (n = 2,181). RESULTS: Patients had a mean age of 65 years, were mostly white (81%), and the majority were insured by Medicare/Medicaid (62%). From baseline to follow-up, the increase in proportion of patients tested for HbA1c (P = 0.004) and LDL cholesterol (P < 0.001) was greater in the intervention than control sites. Improvements in diabetes-related medication prescription and levels of HbA1c, LDL cholesterol, and blood pressure in the intervention clinic were balanced by similar improvements in the control sites. CONCLUSIONS: Population-level clinical registries combined with summarized recommendations to PCPs had a modest effect on management. The intervention was limited by good overall quality of care at baseline and temporal improvements in all control clinics. It is unknown whether this intervention would have had greater impact in clinical settings with lower overall quality. Further research into more effective methods of translating population registry information into action is required.

Automated coded ambulatory problem lists: evaluation of a vocabulary and a data entry tool.

BACKGROUND: Problem lists are fundamental to electronic medical records (EMRs). However, obtaining an appropriate problem list dictionary is difficult, and getting users to code their problems at the time of data entry can be challenging. OBJECTIVE: To develop a problem list dictionary and search algorithm for an EMR system and evaluate its use. METHODS: We developed a problem list dictionary and lookup tool and implemented it in several EMR systems. A sample of 10,000 problem entries was reviewed from each system to assess overall coding rates. We also performed a manual review of a subset of entries to determine the appropriateness of coded entries, and to assess the reasons other entries were left uncoded. RESULTS: The overall coding rate varied significantly between different EMR implementations (63-79%). Coded entries were virtually always appropriate (99%). The most frequent reasons for uncoded entries were due to user interface failures (44-45%), insufficient dictionary coverage (20-32%), and non-problem entries (10-12%). CONCLUSION: The problem list dictionary and search algorithm has achieved a good coding rate, but the rate is dependent on the specific user interface implementation. Problem coding is essential for providing clinical decision support, and improving usability should result in better coding rates.

Development and implementation of a real-time 30-day readmission predictive model.

Hospitals are under great pressure to reduce readmissions of patients. Being able to reliably predict patients at increased risk for rehospitalization would allow for tailored interventions to be offered to them. This requires the creation of a functional predictive model specifically designed to support real-time clinical operations. A predictive model for readmissions within 30 days of discharge was developed using retrospective data from 45,924 MGH admissions between 2/1/2012 and 1/31/2013 only including factors that would be available by the day after admission. It was then validated prospectively in a real-time implementation for 3,074 MGH admissions between 10/1/2013 and 10/31/2013. The model developed retrospectively had an AUC of 0.705 with good calibration. The real-time implementation had an AUC of 0.671 although the model was overestimating readmission risk. A moderately discriminative real-time 30-day readmission predictive model can be developed and implemented in a large academic hospital.

Applying operations research to optimize a novel population management system for cancer screening.

OBJECTIVE: To optimize a new visit-independent, population-based cancer screening system (TopCare) by using operations research techniques to simulate changes in patient outreach staffing levels (delegates, navigators), modifications to user workflow within the information technology (IT) system, and changes in cancer screening recommendations. MATERIALS AND METHODS: TopCare was modeled as a multiserver, multiphase queueing system. Simulation experiments implemented the queueing network model following a next-event time-advance mechanism, in which systematic adjustments were made to staffing levels, IT workflow settings, and cancer screening frequency in order to assess their impact on overdue screenings per patient. RESULTS: TopCare reduced the average number of overdue screenings per patient from 1.17 at inception to 0.86 during simulation to 0.23 at steady state. Increases in the workforce improved the effectiveness of TopCare. In particular, increasing the delegate or navigator staff level by one person improved screening completion rates by 1.3% or 12.2%, respectively. In contrast, changes in the amount of time a patient entry stays on delegate and navigator lists had little impact on overdue screenings. Finally, lengthening the screening interval increased efficiency within TopCare by decreasing overdue screenings at the patient level, resulting in a smaller number of overdue patients needing delegates for screening and a higher fraction of screenings completed by delegates. CONCLUSIONS: Simulating the impact of changes in staffing, system parameters, and clinical inputs on the effectiveness and efficiency of care can inform the allocation of limited resources in population management.