Structure and Funding of Clinical Informatics Fellowships: A National Survey of Program Directors.

BACKGROUND: In 2011, the American Board of Medical Specialties established clinical informatics (CI) as a subspecialty in medicine, jointly administered by the American Board of Pathology and the American Board of Preventive Medicine. Subsequently, many institutions created CI fellowship training programs to meet the growing need for informaticists. Although many programs share similar features, there is considerable variation in program funding and administrative structures. OBJECTIVES: The aim of our study was to characterize CI fellowship program features, including governance structures, funding sources, and expenses. METHODS: We created a cross-sectional online REDCap survey with 44 items requesting information on program administration, fellows, administrative support, funding sources, and expenses. We surveyed program directors of programs accredited by the Accreditation Council for Graduate Medical Education between 2014 and 2021. RESULTS: We invited 54 program directors, of which 41 (76%) completed the survey. The average administrative support received was $27,732/year. Most programs (85.4%) were accredited to have two or more fellows per year. Programs were administratively housed under six departments: Internal Medicine (17; 41.5%), Pediatrics (7; 17.1%), Pathology (6; 14.6%), Family Medicine (6; 14.6%), Emergency Medicine (4; 9.8%), and Anesthesiology (1; 2.4%). Funding sources for CI fellowship program directors included: hospital or health systems (28.3%), clinical departments (28.3%), graduate medical education office (13.2%), biomedical informatics department (9.4%), hospital information technology (9.4%), research and grants (7.5%), and other sources (3.8%) that included philanthropy and external entities. CONCLUSION: CI fellowships have been established in leading academic and community health care systems across the country. Due to their unique training requirements, these programs require significant resources for education, administration, and recruitment. There continues to be considerable heterogeneity in funding models between programs. Our survey findings reinforce the need for reformed federal funding models for informatics practice and training.

Evaluation of a Large Language Model to Identify Confidential Content in Adolescent Encounter Notes.

This cross-sectional study assesses the ability of a language learning model to classify whether a progress note contains confidential information and to identify the specific confidential content in the note.

Creation and Evaluation of a Clinical Informatics Match: Initial Findings.

BACKGROUND: Clinical Informatics (CI) fellowship programs utilize the Electronic Residency Application Service (ERAS) to gather applications but until recently used an American Medical Informatics Association (AMIA) member-developed, simultaneous offer-acceptance process to match fellowship applicants to programs. In 2021, program directors collaborated with the AMIA to develop a new match to improve the process. OBJECTIVE: Describe the results of the first 2 years of the match and address opportunities for improvement. METHODS: We obtained applicant data for fellowship applicants in 2021 and 2022 from the ERAS and match data for the same years from the AMIA. We analyzed our data using descriptive statistics. RESULTS: There were 159 unique applicants over the 2-year period. Applicants submitted 2,178 applications with a median of 10 per applicant (interquartile range [IQR] 3-20). One hundred and four applicants (65.4%) participated in the match and ranked a median of seven programs (2-12). Forty-two programs in 2021 and 47 programs in 2022 offered a combined total 153 positions in the match. Participating programs ranked a median of eight applicants per year (IQR 5-11). Of participating applicants, 95 (91.3%) successfully matched and of those 66 (69.5%) received their top choice. Thirty-two programs (76.2%) matched at least one candidate in 2021 and 33 programs (70.2%) matched at least one candidate in 2022. In both years, 24 programs filled all available slots (57.1% in 2021 and 51.1% in 2022). CONCLUSION: Applicants were extremely successful in the new match, which successfully addressed most of the challenges of the simultaneous offer-acceptance process identified by program directors. However, applicant attrition resulted in a quarter of programs going unmatched. Although many programs still filled slots outside the match, fellowship slots may remain unfilled while the CI practice pathway remains open.

The Prevalence of Confidential Content in Adolescent Progress Notes Prior to the 21st Century Cures Act Information Blocking Mandate.

BACKGROUND: The 21st Century Cures Act information blocking final rule mandated the immediate and electronic release of health care data in 2020. There is anecdotal concern that a significant amount of information is documented in notes that would breach adolescent confidentiality if released electronically to a guardian. OBJECTIVES: The purpose of this study was to quantify the prevalence of confidential information, based on California laws, within progress notes for adolescent patients that would be released electronically and assess differences in prevalence across patient demographics. METHODS: This is a single-center retrospective chart review of outpatient progress notes written between January 1, 2016, and December 31, 2019, at a large suburban academic pediatric network. Notes were labeled into one of three confidential domains by five expert reviewers trained on a rubric defining confidential information for adolescents derived from California state law. Participants included a random sampling of eligible patients aged 12 to 17 years old at the time of note creation. Secondary analysis included prevalence of confidentiality across age, gender, language spoken, and patient race. RESULTS: Of 1,200 manually reviewed notes, 255 notes (21.3%) (95% confidence interval: 19-24%) contained confidential information. There was a similar distribution among gender and age and a majority of English speaking (83.9%) and white or Caucasian patients (41.2%) in the cohort. Confidential information was more likely to be found in notes for females (p < 0.05) as well as for English-speaking patients (p < 0.05). Older patients had a higher probability of notes containing confidential information (p < 0.05). CONCLUSION: This study demonstrates that there is a significant risk to breach adolescent confidentiality if historical progress notes are released electronically to proxies without further review or redaction. With increased sharing of health care data, there is a need to protect the privacy of the adolescents and prevent potential breaches of confidentiality.

Association between Electronic Health Record Implementations and Hospital-Acquired Conditions in Pediatric Hospitals.

BACKGROUND: Implementing an electronic health record (EHR) is one of the most disruptive operational tasks a health system can undergo. Despite anecdotal reports of adverse events around the time of EHR implementations, there is limited corroborating research, particularly in pediatrics. We utilized data from Solutions for Patient Safety (SPS), a network of 145+ children's hospitals that share data and protocols to reduce harm in pediatric care delivery, to study the impact of EHR implementations on patient safety. OBJECTIVE: Determine if there is an association between the time immediately surrounding an EHR implementation and hospital-acquired conditions (HACs) rates in pediatrics. METHODS: A survey of information technology leaders at pediatric institutions identified EHR implementations occurring between 2012 and 2022. This list was cross-referenced with the SPS database to create an anonymized dataset of 27 sites comprising monthly HAC and care bundle compliance rates in the 7 months preceding and succeeding the transition. Six HACs were analyzed: central-line associated bloodstream infection (CLABSI), catheter-associated urinary tract infection (CAUTI), adverse drug events, surgical site infections (SSIs), pressure injuries (PIs), and falls, in addition to four associated care bundle compliance rates: CLABSI and CAUTI maintenance bundles, SSI bundle, and PI bundle. To determine if there was a statistically significant association with EHR implementation, the observation period was divided into three eras: "before" (months -7 to -3), "during" (months -2 to +2), and "after" go-live (months +3 to +7). Average monthly HAC and bundle compliance rates were calculated across eras. Paired t-tests were performed to compare rates between the eras. RESULTS: No statistically significant increase in HAC rates or decrease in bundle compliance rates was observed across the EHR implementation eras. CONCLUSION: This multisite study detected no significant increase in HACs and no decrease in preventive care bundle compliance in the months surrounding an EHR implementation.

Monitoring Approaches for a Pediatric Chronic Kidney Disease Machine Learning Model.

OBJECTIVE: The purpose of this study is to evaluate the ability of three metrics to monitor for a reduction in performance of a chronic kidney disease (CKD) model deployed at a pediatric hospital. METHODS: The CKD risk model estimates a patient's risk of developing CKD 3 to 12 months following an inpatient admission. The model was developed on a retrospective dataset of 4,879 admissions from 2014 to 2018, then run silently on 1,270 admissions from April to October, 2019. Three metrics were used to monitor its performance during the silent phase: (1) standardized mean differences (SMDs); (2) performance of a "membership model"; and (3) response distribution analysis. Observed patient outcomes for the 1,270 admissions were used to calculate prospective model performance and the ability of the three metrics to detect performance changes. RESULTS: The deployed model had an area under the receiver-operator curve (AUROC) of 0.63 in the prospective evaluation, which was a significant decrease from an AUROC of 0.76 on retrospective data (p = 0.033). Among the three metrics, SMDs were significantly different for 66/75 (88%) of the model's input variables (p <0.05) between retrospective and deployment data. The membership model was able to discriminate between the two settings (AUROC = 0.71, p <0.0001) and the response distributions were significantly different (p <0.0001) for the two settings. CONCLUSION: This study suggests that the three metrics examined could provide early indication of performance deterioration in deployed models' performance.

Pediatric subspecialty telemedicine use from the patient and provider perspective.

BACKGROUND: To characterize telemedicine use among pediatric subspecialties with respect to clinical uses of telemedicine, provider experience, and patient perceptions during the COVID-19 pandemic. METHODS: We performed a mixed-methods study of telemedicine visits across pediatric endocrinology, nephrology, orthopedic surgery, and rheumatology at a large children's hospital. We used deductive analysis to review observational data from 40 video visits. Providers and patients/caregivers were surveyed around areas of satisfaction and communication. RESULTS: We found adaptations of telemedicine including shared-screen use and provider-guided parent procedures among others. All providers felt that it was safest for their patients to conduct visits by video, and 72.7% reported completing some component of a clinical exam. Patients rated the areas of being respected by the clinical staff/provider and showing care and concern highly, and the mean overall satisfaction was 86.7 ± 19.3%. CONCLUSIONS: Telemedicine has been used to deliver care to pediatric patients during the pandemic, and we found that patients were satisfied with the telemedicine visits during this stressful time and that providers were able to innovate during visits. Telemedicine is a tool that can be successfully adapted to patient and provider needs, but further studies are needed to fully explore its integration in pediatric subspecialty care. IMPACT: This study describes telemedicine use at the height of the COVID-19 pandemic from both a provider and patient perspective, in four different pediatric subspecialties. Prior to COVID-19, pediatric telehealth landscape analysis suggested that many pediatric specialty practices had pilot telehealth programs, but there are few published studies evaluating telemedicine performance through the simultaneous patient and provider experience as part of standard care. We describe novel uses and adaptations of telemedicine during a time of rapid deployment in pediatric specialty care.

The Clinical Information Systems Response to the COVID-19 Pandemic.

OBJECTIVE: The year 2020 was predominated by the coronavirus disease 2019 (COVID-19) pandemic. The objective of this article is to review the areas in which clinical information systems (CIS) can be and have been utilized to support and enhance the response of healthcare systems to pandemics, focusing on COVID-19. METHODS: PubMed/MEDLINE, Google Scholar, the tables of contents of major informatics journals, and the bibliographies of articles were searched for studies pertaining to CIS, pandemics, and COVID-19 through October 2020. The most informative and detailed studies were highlighted, while many others were referenced. RESULTS: CIS were heavily relied upon by health systems and governmental agencies worldwide in response to COVID-19. Technology-based screening tools were developed to assist rapid case identification and appropriate triaging. Clinical care was supported by utilizing the electronic health record (EHR) to onboard frontline providers to new protocols, offer clinical decision support, and improve systems for diagnostic testing. Telehealth became the most rapidly adopted medical trend in recent history and an essential strategy for allowing safe and effective access to medical care. Artificial intelligence and machine learning algorithms were developed to enhance screening, diagnostic imaging, and predictive analytics - though evidence of improved outcomes remains limited. Geographic information systems and big data enabled real-time dashboards vital for epidemic monitoring, hospital preparedness strategies, and health policy decision making. Digital contact tracing systems were implemented to assist a labor-intensive task with the aim of curbing transmission. Large scale data sharing, effective health information exchange, and interoperability of EHRs remain challenges for the informatics community with immense clinical and academic potential. CIS must be used in combination with engaged stakeholders and operational change management in order to meaningfully improve patient outcomes. CONCLUSION: Managing a pandemic requires widespread, timely, and effective distribution of reliable information. In the past year, CIS and informaticists made prominent and influential contributions in the global response to the COVID-19 pandemic.

Measuring Electronic Health Record Use in the Pediatric ICU Using Audit-Logs and Screen Recordings.

BACKGROUND: Time spent in the electronic health record (EHR) has been identified as an important unit of measure for health care provider clinical activity. The lack of validation of audit-log based inpatient EHR time may have resulted in underuse of this data in studies focusing on inpatient patient outcomes, provider efficiency, provider satisfaction, etc. This has also led to a dearth of clinically relevant EHR usage metrics consistent with inpatient provider clinical activity. OBJECTIVE: The aim of our study was to validate audit-log based EHR times using observed EHR-times extracted from screen recordings of EHR usage in the inpatient setting. METHODS: This study was conducted in a 36-bed pediatric intensive care unit (PICU) at Lucile Packard Children's Hospital Stanford between June 11 and July 14, 2020. Attending physicians, fellow physicians, hospitalists, and advanced practice providers with ≥0.5 full-time equivalent (FTE) for the prior four consecutive weeks and at least one EHR session recording were included in the study. Citrix session recording player was used to retrospectively review EHR session recordings that were captured as the provider interacted with the EHR. RESULTS: EHR use patterns varied by provider type. Audit-log based total EHR time correlated strongly with both observed total EHR time (r = 0.98, p < 0.001) and observed active EHR time (r = 0.95, p < 0.001). Each minute of audit-log based total EHR time corresponded to 0.95 (0.87-1.02) minutes of observed total EHR time and 0.75 (0.67-0.83) minutes of observed active EHR time. Results were similar when stratified by provider role. CONCLUSION: Our study found inpatient audit-log based EHR time to correlate strongly with observed EHR time among pediatric critical care providers. These findings support the use of audit-log based EHR-time as a surrogate measure for inpatient provider EHR use, providing an opportunity for researchers and other stakeholders to leverage EHR audit-log data in measuring clinical activity and tracking outcomes of workflow improvement efforts longitudinally and across provider groups.

Quantifying Discharge Medication Reconciliation Errors at 2 Pediatric Hospitals.

INTRODUCTION: Medication reconciliation errors (MREs) are common and can lead to significant patient harm. Quality improvement efforts to identify and reduce these errors typically rely on resource-intensive chart reviews or adverse event reporting. Quantifying these errors hospital-wide is complicated and rarely done. The purpose of this study is to define a set of 6 MREs that can be easily identified across an entire healthcare organization and report their prevalence at 2 pediatric hospitals. METHODS: An algorithmic analysis of discharge medication lists and confirmation by clinician reviewers was used to find the prevalence of the 6 discharge MREs at 2 pediatric hospitals. These errors represent deviations from the standards for medication instruction completeness, clarity, and safety. The 6 error types are Duplication, Missing Route, Missing Dose, Missing Frequency, Unlisted Medication, and See Instructions errors. RESULTS: This study analyzed 67,339 discharge medications and detected MREs commonly at both hospitals. For Institution A, a total of 4,234 errors were identified, with 29.9% of discharges containing at least one error and an average of 0.7 errors per discharge. For Institution B, a total of 5,942 errors were identified, with 42.2% of discharges containing at least 1 error and an average of 1.6 errors per discharge. The most common error types were Duplication and See Instructions errors. CONCLUSION: The presented method shows these MREs to be a common finding in pediatric care. This work offers a tool to strengthen hospital-wide quality improvement efforts to reduce pediatric medication errors.

Improved Medical Student Engagement with EHR Documentation following the 2018 Centers for Medicare and Medicaid Billing Changes.

BACKGROUND: Medical student note writing is an important part of the training process but has suffered in the electronic health record (EHR) era as a result of student notes being excluded from the billable encounter. The 2018 CMS billing changes allow for medical student notes to be used for billable services provided that physical presence requirements are met, and attending physicians satisfy performance requirements and verify documentation. This has the potential to improve medical student engagement and decrease physician documentation burden. METHODS: Our institution implemented medical student notes as part of the billable encounter in August 2018 with support of our compliance department. Note characteristics including number, type, length, and time in note were analyzed before and after implementation. Rotating medical students were surveyed regarding their experience following implementation. RESULTS: There was a statistically significant increase in the number of student-authored notes following implementation. Attending physicians' interactions with student notes greatly increased following the change (4% of student notes reviewed vs. 84% of student notes). Surveyed students reported that having their notes as part of the billable record made their notes more meaningful and enhanced their learning. The majority of surveyed students also agreed that they received more feedback following the change. CONCLUSION: Medical students are interested in writing notes for education and feedback. Inclusion of their notes as part of the billable record can facilitate their learning and increase their participation in the note writing process.

A Clinical Informatics Program Directors' Proposal to the American Board of Preventive Medicine.

In 2013, the American Board of Preventive Medicine (ABPM) and the American Board of Pathology (ABPath) offered the first board certification examination in Clinical Informatics to eligible physicians in the United States. In 2022, the Practice Pathway will expire and in 2023 only candidates eligible through the Fellowship Pathway will be eligible for the board certification. To date, Clinical Informatics as a specialty has not had a regular match process and used a controlled offer-acceptance process that does not meet candidates' or programs' needs. Fellows may not be offered a position with their top choice program initially, and they may accept offers from other programs to avoid risk by ensuring that they have a fellowship position. Programs have to consider losing an applicant in the first round in the ranking of applicants. The process is open to manipulation including early agreements between program directors and candidates. In this open letter to the ABPM, program directors make the case for a third-party match and are calling on the ABPM to leverage its status as the Clinical Informatics certifying body and its existing infrastructure to implement a Clinical Informatics match.

Your Patient Has a New Health App? Start With Its Data Source.

Recent regulatory and technological advances have enabled a new era of health apps that are controlled by patients and contain valuable health information. These health apps will be numerous and use novel interfaces that appeal to patients but will likely be unfamiliar to practitioners. We posit that understanding the origin of the health data is the most meaningful and versatile way for physicians to understand and effectively use these apps in patient care. This will allow providers to better support patients and encourage patient engagement in their own care.

Electronic health record (EHR) training program identifies a new tool to quantify the EHR time burden and improves providers' perceived control over their workload in the EHR.

OBJECTIVE: To understand if providers who had additional electronic health record (EHR) training improved their satisfaction, decreased personal EHR-use time, and decreased turnaround time on tasks. MATERIALS AND METHODS: This pre-post study with no controls evaluated the impact of a supplemental EHR training program on a group of academic and community practice clinicians that previously had go-live group EHR training and 20 months experience using this EHR on self-reported data, calculated EHR time, and vendor-reported metrics. RESULTS: Providers self-reported significant improvements in their knowledge of efficiency tools in the EHR after training and doubled (significant) their preference list entries (mean pre = 38.1 [65.88], post = 63.5 [90.47], P < .01). Of the 7 EHR satisfaction variables, only 1 self-reported variable significantly improved after training: Control over my workload in the EHR (mean pre = 2.7 [0.96], post = 3.0 [1.04], P < .01). There was no significant decrease in their calculated EHR usage outside of clinic (mean pre = 0.39 [0.77] to post = 0.37 [0.48], P = .73). No significant difference was seen in turnaround time for patient calls (mean pre = 2.3 [2.06] days, post = 1.9 [1.76] days, P = .08) and results (mean before = 4.0 [2.79] days, after = 3.2 [2.33] days, P = .03). DISCUSSION: Multiple sources of data provide a holistic view of the provider experience in the EHR. This study suggests that individualized EHR training can improve the knowledge of EHR tools and satisfaction with their perceived control of EHR workload, however this did not translate into less Clinician Logged-In Outside Clinic (CLOC) time, a calculated metric, nor quicker turnaround on in box tasks. CLOC time emerged as a potential less-costly surrogate metric for provider satisfaction in EHR work than surveying clinicians. Further study is required to understand the cost-benefit of various interventions to decrease CLOC time. CONCLUSIONS: This supplemental EHR training session, 20 months post go-live, where most participants elected to receive 2 or fewer sessions did significantly improve provider satisfaction with perceived control over their workload in the EHR, but it was not effective in decreasing EHR-use time outside of clinic. CLOC time, a calculated metric, could be a practical trackable surrogate for provider satisfaction (inverse correlation) with after-hours time spent in the EHR. Further study into interventions that decrease CLOC time and improve turnaround time to respond to inbox tasks are suggested next steps.

Integration of Single-Center Data-Driven Vital Sign Parameters into a Modified Pediatric Early Warning System.

OBJECTIVES: Pediatric early warning systems using expert-derived vital sign parameters demonstrate limited sensitivity and specificity in identifying deterioration. We hypothesized that modified tools using data-driven vital sign parameters would improve the performance of a validated tool. DESIGN: Retrospective case control. SETTING: Quaternary-care children's hospital. PATIENTS: Hospitalized, noncritically ill patients less than 18 years old. Cases were defined as patients who experienced an emergent transfer to an ICU or out-of-ICU cardiac arrest. Controls were patients who never required intensive care. Cases and controls were split into training and testing groups. INTERVENTIONS: The Bedside Pediatric Early Warning System was modified by integrating data-driven heart rate and respiratory rate parameters (modified Bedside Pediatric Early Warning System 1 and 2). Modified Bedside Pediatric Early Warning System 1 used the 10th and 90th percentiles as normal parameters, whereas modified Bedside Pediatric Early Warning System 2 used fifth and 95th percentiles. MEASUREMENTS AND MAIN RESULTS: The training set consisted of 358 case events and 1,830 controls; the testing set had 331 case events and 1,215 controls. In the sensitivity analysis, 207 of the 331 testing set cases (62.5%) were predicted by the original tool versus 206 (62.2%; p = 0.54) with modified Bedside Pediatric Early Warning System 1 and 191 (57.7%; p < 0.001) with modified Bedside Pediatric Early Warning System 2. For specificity, 1,005 of the 1,215 testing set control patients (82.7%) were identified by original Bedside Pediatric Early Warning System versus 1,013 (83.1%; p = 0.54) with modified Bedside Pediatric Early Warning System 1 and 1,055 (86.8%; p < 0.001) with modified Bedside Pediatric Early Warning System 2. There was no net gain in sensitivity and specificity using either of the modified Bedside Pediatric Early Warning System tools. CONCLUSIONS: Integration of data-driven vital sign parameters into a validated pediatric early warning system did not significantly impact sensitivity or specificity, and all the tools showed lower than desired sensitivity and specificity at a single cutoff point. Future work is needed to develop an objective tool that can more accurately predict pediatric decompensation.

Improving Communication with Primary Care Physicians at the Time of Hospital Discharge.

BACKGROUND: Communication with primary care physicians (PCPs) at the time of a patient's hospital discharge is important to safely transition care to home. The goal of this quality improvement initiative was to increase discharge communication to PCPs at an academic children's hospital. METHODS: A multidisciplinary team at Lucile Packard Children's Hospital Stanford used Lean A3 problem solving methodology to address the problem of inadequate discharge communication with PCPs. Emphasis was placed on frontline provider (resident physicians) involvement in the improvement process, creating standards, and error proofing. Root cause analysis identified several key drivers of the problem, and successive countermeasures were implemented beginning in August 2013 aimed at achieving the target of 80% attempted verbal communication within seven days before or after (usually 24-48 hours) on the pediatric medical services. Run charts were generated tracking the outcome of PCP communication. RESULTS: On the pediatric medical services, the goal of 80% communication was met and sustained during a seven-month period starting October 2013, a statistically significant improvement. In the eight months prior to October 2013, hospitalwide PCP communication prior to discharge averaged 59.1% (n = 5,397) and improved to 76.7% (n = 4,870) in the seven months after (p <0.001). Fifteen of 19 specialty services had a significant increase in discharge communication after October 2013. CONCLUSION: Lean improvement methodology (including structured problem solving using A3 thinking), intensive frontline provider involvement, and process-oriented electronic health record work flow redesign led to increased verbal PCP communication at around the time of a patient's discharge.

Designing An Individualized EHR Learning Plan For Providers.

Electronic Health Records (EHRs) have been quickly implemented for meaningful use incentives; however these implementations have been associated with provider dissatisfaction and burnout. There are no previously reported instances of a comprehensive EHR educational program designed to engage providers and assist in improving efficiency and understanding of the EHR. Utilizing adult learning theory as a framework, Stanford Children's Health designed a tailored provider efficiency program with various inputs from: (1) provider specific EHR data; (2) provider survey data; and (3) structured observation sessions. This case report outlines the design of this individualized training program including team structure, resource requirements, and early provider response. CITATION: Stevens LA, DiAngi YT, Schremp JD, Martorana MJ, Miller RE, Lee TC, Pageler NM. Designing An Individualized EHR Learning Plan. Appl Clin Inform 2017; 8:924-935 https://doi.org/10.4338/040054.