Prediction of multiclass surgical outcomes in glaucoma using multimodal deep learning based on free-text operative notes and structured EHR data.

OBJECTIVE: Surgical outcome prediction is challenging but necessary for postoperative management. Current machine learning models utilize pre- and post-op data, excluding intraoperative information in surgical notes. Current models also usually predict binary outcomes even when surgeries have multiple outcomes that require different postoperative management. This study addresses these gaps by incorporating intraoperative information into multimodal models for multiclass glaucoma surgery outcome prediction. MATERIALS AND METHODS: We developed and evaluated multimodal deep learning models for multiclass glaucoma trabeculectomy surgery outcomes using both structured EHR data and free-text operative notes. We compare those to baseline models that use structured EHR data exclusively, or neural network models that leverage only operative notes. RESULTS: The multimodal neural network had the highest performance with a macro AUROC of 0.750 and F1 score of 0.583. It outperformed the baseline machine learning model with structured EHR data alone (macro AUROC of 0.712 and F1 score of 0.486). Additionally, the multimodal model achieved the highest recall (0.692) for hypotony surgical failure, while the surgical success group had the highest precision (0.884) and F1 score (0.775). DISCUSSION: This study shows that operative notes are an important source of predictive information. The multimodal predictive model combining perioperative notes and structured pre- and post-op EHR data outperformed other models. Multiclass surgical outcome prediction can provide valuable insights for clinical decision-making. CONCLUSIONS: Our results show the potential of deep learning models to enhance clinical decision-making for postoperative management. They can be applied to other specialties to improve surgical outcome predictions.

Guidance for reporting analyses of metadata on electronic health record use.

INTRODUCTION: Research on how people interact with electronic health records (EHRs) increasingly involves the analysis of metadata on EHR use. These metadata can be recorded unobtrusively and capture EHR use at a scale unattainable through direct observation or self-reports. However, there is substantial variation in how metadata on EHR use are recorded, analyzed and described, limiting understanding, replication, and synthesis across studies. RECOMMENDATIONS: In this perspective, we provide guidance to those working with EHR use metadata by describing 4 common types, how they are recorded, and how they can be aggregated into higher-level measures of EHR use. We also describe guidelines for reporting analyses of EHR use metadata-or measures of EHR use derived from them-to foster clarity, standardization, and reproducibility in this emerging and critical area of research.

Identifying and Addressing Barriers to Implementing Core Electronic Health Record Use Metrics for Ambulatory Care: Virtual Consensus Conference Proceedings.

Precise, reliable, valid metrics that are cost-effective and require reasonable implementation time and effort are needed to drive electronic health record (EHR) improvements and decrease EHR burden. Differences exist between research and vendor definitions of metrics. PROCESS: We convened three stakeholder groups (health system informatics leaders, EHR vendor representatives, and researchers) in a virtual workshop series to achieve consensus on barriers, solutions, and next steps to implementing the core EHR use metrics in ambulatory care. CONCLUSION: Actionable solutions identified to address core categories of EHR metric implementation challenges include: (1) maintaining broad stakeholder engagement, (2) reaching agreement on standardized measure definitions across vendors, (3) integrating clinician perspectives, and (4) addressing cognitive and EHR burden. Building upon the momentum of this workshop's outputs offers promise for overcoming barriers to implementing EHR use metrics.

Development of an Open-Source Annotated Glaucoma Medication Dataset From Clinical Notes in the Electronic Health Record.

Purpose: To describe the methods involved in processing and characteristics of an open dataset of annotated clinical notes from the electronic health record (EHR) annotated for glaucoma medications. Methods: In this study, 480 clinical notes from office visits, medical record numbers (MRNs), visit identification numbers, provider names, and billing codes were extracted for 480 patients seen for glaucoma by a comprehensive or glaucoma ophthalmologist from January 1, 2019, to August 31, 2020. MRNs and all visit data were de-identified using a hash function with salt from the deidentifyr package. All progress notes were annotated for glaucoma medication name, route, frequency, dosage, and drug use using an open-source annotation tool, Doccano. Annotations were saved separately. All protected health information (PHI) in progress notes and annotated files were de-identified using the published de-identifying algorithm Philter. All progress notes and annotations were manually validated by two ophthalmologists to ensure complete de-identification. Results: The final dataset contained 5520 annotated sentences, including those with and without medications, for 480 clinical notes. Manual validation revealed 10 instances of remaining PHI which were manually corrected. Conclusions: Annotated free-text clinical notes can be de-identified for upload as an open dataset. As data availability increases with the adoption of EHRs, free-text open datasets will become increasingly valuable for "big data" research and artificial intelligence development. This dataset is published online and publicly available at https://github.com/jche253/Glaucoma_Med_Dataset. Translational Relevance: This open access medication dataset may be a source of raw data for future research involving big data and artificial intelligence research using free-text.

Medical Scribes Have a Variable Impact on Documentation Workflows.

Physicians can reduce their documentation time by working with a scribe. However, what scribes document and how their actions affect existing documentation workflows is unclear. This study leverages electronic health record (EHR) audit logs to observe how scribes affected the documentation workflows of seven physicians and their staff across 13,000 outpatient ophthalmology visits. In addition to editing progress notes, scribes routinely edited exam findings and diagnoses. Scribes with clinical training also edited items such as vital signs that a scribe without clinical training did not. Every physician edited patient records later in the day when working with a scribe and those who deferred their editing the most had some of the largest reductions in EHR time. These results suggest that what scribes document, how physicians work with scribes, and scribe impact on documentation time are all highly variable, highlighting the need for evidence-based best practices.

Pediatric Ophthalmology Provider and Staff Attitudes and Patient Satisfaction in Telehealth Implementation During COVID-19.

Purpose:Describe a comprehensive overview of a telehealth implementation process that highlights attitudes and satisfaction scores toward telehealth from patients, providers, and staff in an academic pediatric ophthalmology practice during the early months of the coronavirus disease 2019 (COVID-19) pandemic.Methods:The electronic medical record data for telehealth and in-person visits, as well as a patient experience survey in pediatric ophthalmology were retrospectively reviewed for March 1 to July 31, 2020 and March 1 to July 31, 2019. Patient experience survey results were retrospectively reviewed. All current providers and staff were invited to participate in an anonymous and voluntary survey focused on attitudes at the time of telehealth implementation.Results:During March 1 to July 31, 2020, there was significant increase in telehealth visits (n = 1,006) compared with the same period in 2019 (n = 22). Evaluation and management (E & M) codes (n = 527) were the most commonly used billing codes, and strabismus, nystagmus, and irregular eye movement (n = 496) were the most common telehealth primary diagnoses. The telehealth attitudes survey showed more positive responses from providers than staff. The patient experience survey showed more favorable scores for telehealth visits compared with clinic visits. However, only about 50% of the respondents were satisfied with the technology in terms of ease and quality of connection during their telehealth visits.Conclusions:Telehealth was a satisfactory alternative to clinic visits in our academic pediatric ophthalmology practice during the early phase of the COVID-19 pandemic. Providers and staff had largely positive attitudes toward telehealth; however, future efforts should include strategies to increase staff buy in. Patients had high satisfaction scores with telehealth visits despite connection challenges.

Frequent but fragmented: use of note templates to document outpatient visits at an academic health center.

Recent changes to billing policy have reduced documentation requirements for outpatient notes, providing an opportunity to rethink documentation workflows. While many providers use templates to write notes-whether to insert short phrases or draft entire notes-we know surprisingly little about how these templates are used in practice. In this retrospective cross-sectional study, we observed the templates that primary providers and other members of the care team used to write the provider progress note for 2.5 million outpatient visits across 52 specialties at an academic health center between 2018 and 2020. Templates were used to document 89% of visits, with a median of 2 used per visit. Only 17% of the 100 230 unique templates were ever used by more than one person and most providers had their own full-note templates. These findings suggest template use is frequent but fragmented, complicating template revision and maintenance. Reframing template use as a form of computer programming suggests ways to maintain the benefits of personalization while leveraging standardization to reduce documentation burden.

Electronic health record note review in an outpatient specialty clinic: who is looking?

Note entry and review in electronic health records (EHRs) are time-consuming. While some clinics have adopted team-based models of note entry, how these models have impacted note review is unknown in outpatient specialty clinics such as ophthalmology. We hypothesized that ophthalmologists and ancillary staff review very few notes. Using audit log data from 9775 follow-up office visits in an academic ophthalmology clinic, we found ophthalmologists reviewed a median of 1 note per visit (2.6 ± 5.3% of available notes), while ancillary staff reviewed a median of 2 notes per visit (4.1 ± 6.2% of available notes). While prior ophthalmic office visit notes were the most frequently reviewed note type, ophthalmologists and staff reviewed no such notes in 51% and 31% of visits, respectively. These results highlight the collaborative nature of note review and raise concerns about how cumbersome EHR designs affect efficient note review and the utility of prior notes in ophthalmic clinical care.

Length and Redundancy of Outpatient Progress Notes Across a Decade at an Academic Medical Center.

Importance: There is widespread concern that clinical notes have grown longer and less informative over the past decade. Addressing these concerns requires a better understanding of the magnitude, scope, and potential causes of increased note length and redundancy. Objective: To measure changes between 2009 and 2018 in the length and redundancy of outpatient progress notes across multiple medical specialties and investigate how these measures associate with author experience and method of note entry. Design, Setting, and Participants: This cross-sectional study was conducted at Oregon Health & Science University, a large academic medical center. Participants included clinicians and staff who wrote outpatient progress notes between 2009 and 2018 for a random sample of 200 000 patients. Statistical analysis was performed from March to August 2020. Exposures: Use of a comprehensive electronic health record to document patient care. Main Outcomes and Measures: Note length, note redundancy (ie, the proportion of text identical to the patient's last note), and percentage of templated, copied, or directly typed note text. Results: A total of 2 704 800 notes written by 6228 primary authors across 46 specialties were included in this study. Median note length increased 60.1% (99% CI, 46.7%-75.2%) from a median of 401 words (interquartile range [IQR], 225-660 words) in 2009 to 642 words (IQR, 399-1007 words) in 2018. Median note redundancy increased 10.9 percentage points (99% CI, 7.5-14.3 percentage points) from 47.9% in 2009 to 58.8% in 2018. Notes written in 2018 had a mean value of just 29.4% (99% CI, 28.2%-30.7%) directly typed text with the remaining 70.6% of text being templated or copied. Mixed-effect linear models found that notes with higher proportions of templated or copied text were significantly longer and more redundant (eg, in the 2-year model, each 1% increase in the proportion of copied or templated note text was associated with 1.5% [95% CI, 1.5%-1.5%] and 1.6% [95% CI, 1.6%-1.6%] increases in note length, respectively). Residents and fellows also wrote significantly (26.3% [95% CI, 25.8%-26.7%]) longer notes than more senior authors, as did more recent hires (1.8% for each year later [95% CI, 1.3%-2.4%]). Conclusions and Relevance: In this study, outpatient progress notes grew longer and more redundant over time, potentially limiting their use in patient care. Interventions aimed at reducing outpatient progress note length and redundancy may need to simultaneously address multiple factors such as note template design and training for both new and established clinicians.

Clinical Documentation During Scribed and Non-scribed Ophthalmology Office Visits.

PURPOSE: Observe the impact of employing scribes on documentation efficiency in ophthalmology clinics. DESIGN: Single-center retrospective cohort study. PARTICIPANTS: A total of 29,997 outpatient visits conducted by seven attending ophthalmologists between 1/1/2018 and 12/31/2019 were included in the study; 18,483 with a scribe present during the encounter and 11,514 without a scribe present. INTERVENTION: Use of a scribe. MAIN OUTCOME MEASURES: Total physician documentation time, physician documentation time during and after the visit, visit length, time to chart closure, note length, and percent of note text edited by physician. RESULTS: Total physician documentation time was significantly less when working with a scribe (mean ± SD, 4.7 ± 2.9 vs. 7.6 ± 3.8 minutes/note, P<.001), as was documentation time during the visit (2.8 ± 2.2 vs. 5.9 ± 3.1 minutes/note, P<.001). Physicians also edited scribed notes less, deleting 1.9 ± 4.4% of scribes' draft note text and adding 14.8 ± 11.4% of the final note text, compared to deleting 6.0 ± 9.1%(P<.001) of draft note text and adding 21.2 ± 15.3%(P<.001) of final note text when not working with a scribe. However, physician after-visit documentation time was significantly higher with a scribe for 3 of 7 physicians (P<.001). Scribe use was also associated with an office visit length increase of 2.9 minutes (P<.001) per patient and time to chart closure of 3.0 hours (P<.001), according to mixed-effects linear models. CONCLUSIONS: Scribe use was associated with increased documentation efficiency through lower total documentation time and less note editing by physicians. However, the use of a scribe was also associated with longer office visit lengths and time to chart closure. The variability in the impact of scribe use on different measures of documentation efficiency leaves unanswered questions about best practices for the implementation of scribes, and warrants further study of effective scribe use.

Extraction of Active Medications and Adherence Using Natural Language Processing for Glaucoma Patients.

Accuracy of medication data in electronic health records (EHRs) is crucial for patient care and research, but many studies have shown that medication lists frequently contain errors. In contrast, physicians often pay more attention to the clinical notes and record medication information in them. The medication information in notes may be used for medication reconciliation to improve the medication lists' accuracy. However, accurately extracting patient's current medications from free-text narratives is challenging. In this study, we first explored the discrepancies between medication documentation in medication lists and progress notes for glaucoma patients by manually reviewing patients' charts. Next, we developed and validated a named entity recognition model to identify current medication and adherence from progress notes. Lastly, a prototype tool for medication reconciliation using the developed model was demonstrated. In the future, the model has the potential to be incorporated into the EHR system to help with realtime medication reconciliation.

Comparing Scribed and Non-scribed Outpatient Progress Notes.

Working with scribes can reduce provider documentation time, but few studies have examined how scribes affect clinical notes. In this retrospective cross-sectional study, we examine over 50,000 outpatient progress notes written with and without scribe assistance by 70 providers across 27 specialties in 2017-2018. We find scribed notes were consistently longer than those written without scribe assistance, with most additional text coming from note templates. Scribed notes were also more likely to contain certain templated lists, such as the patient's medications or past medical history. However, there was significant variation in how working with scribes affected a provider's mix of typed, templated, and copied note text, suggesting providers adapt their documentation workflows to varying degrees when working with scribes. These results suggest working with scribes may contribute to note bloat, but that providers' individual documentation workflows, including their note templates, may have a large impact on scribed note contents.

Measures of electronic health record use in outpatient settings across vendors.

Electronic health record (EHR) log data capture clinical workflows and are a rich source of information to understand variation in practice patterns. Variation in how EHRs are used to document and support care delivery is associated with clinical and operational outcomes, including measures of provider well-being and burnout. Standardized measures that describe EHR use would facilitate generalizability and cross-institution, cross-vendor research. Here, we describe the current state of outpatient EHR use measures offered by various EHR vendors, guided by our prior conceptual work that proposed seven core measures to describe EHR use. We evaluate these measures and other reporting options provided by vendors for maturity and similarity to previously proposed standardized measures. Working toward improved standardization of EHR use measures can enable and accelerate high-impact research on physician burnout and job satisfaction as well as organizational efficiency and patient health.

Registered Nurse Strain Detection Using Ambient Data: An Exploratory Study of Underutilized Operational Data Streams in the Hospital Workplace.

BACKGROUND: Registered nurses (RNs) regularly adapt their work to ever-changing situations but routine adaptation transforms into RN strain when service demand exceeds staff capacity and patients are at risk of missed or delayed care. Dynamic monitoring of RN strain could identify when intervention is needed, but comprehensive views of RN work demands are not readily available. Electronic care delivery tools such as nurse call systems produce ambient data that illuminate workplace activity, but little is known about the ability of these data to predict RN strain. OBJECTIVES: The purpose of this study was to assess the utility of ambient workplace data, defined as time-stamped transaction records and log file data produced by non-electronic health record care delivery tools (e.g., nurse call systems, communication devices), as an information channel for automated sensing of RN strain. METHODS: In this exploratory retrospective study, ambient data for a 1-year time period were exported from electronic nurse call, medication dispensing, time and attendance, and staff communication systems. Feature sets were derived from these data for supervised machine learning models that classified work shifts by unplanned overtime. Models for three timeframes -8, 10, and 12 hours-were created to assess each model's ability to predict unplanned overtime at various points across the work shift. RESULTS: Classification accuracy ranged from 57 to 64% across three analysis timeframes. Accuracy was lowest at 10 hours and highest at shift end. Features with the highest importance include minutes spent using a communication device and percent of medications delivered via a syringe. CONCLUSION: Ambient data streams can serve as information channels that contain signals related to unplanned overtime as a proxy indicator of RN strain as early as 8 hours into a work shift. This study represents an initial step toward enhanced detection of RN strain and proactive prevention of missed or delayed patient care.

Applications of Artificial Intelligence to Electronic Health Record Data in Ophthalmology.

Widespread adoption of electronic health records (EHRs) has resulted in the collection of massive amounts of clinical data. In ophthalmology in particular, the volume range of data captured in EHR systems has been growing rapidly. Yet making effective secondary use of this EHR data for improving patient care and facilitating clinical decision-making has remained challenging due to the complexity and heterogeneity of these data. Artificial intelligence (AI) techniques present a promising way to analyze these multimodal data sets. While AI techniques have been extensively applied to imaging data, there are a limited number of studies employing AI techniques with clinical data from the EHR. The objective of this review is to provide an overview of different AI methods applied to EHR data in the field of ophthalmology. This literature review highlights that the secondary use of EHR data has focused on glaucoma, diabetic retinopathy, age-related macular degeneration, and cataracts with the use of AI techniques. These techniques have been used to improve ocular disease diagnosis, risk assessment, and progression prediction. Techniques such as supervised machine learning, deep learning, and natural language processing were most commonly used in the articles reviewed.

Promoting Quality Face-to-Face Communication during Ophthalmology Encounters in the Electronic Health Record Era.

OBJECTIVE: To evaluate informatics-enabled quality improvement (QI) strategies for promoting time spent on face-to-face communication between ophthalmologists and patients. METHODS: This prospective study involved deploying QI strategies during implementation of an enterprise-wide vendor electronic health record (EHR) in an outpatient academic ophthalmology department. Strategies included developing single sign-on capabilities, activating mobile- and tablet-based applications, EHR personalization training, creating novel workflows for team-based orders, and promoting problem-based charting to reduce documentation burden. Timing data were collected during 648 outpatient encounters. Outcomes included total time spent by the attending ophthalmologist on the patient, time spent on documentation, time spent on examination, and time spent talking with the patient. Metrics related to documentation efficiency, use of personalization features, use of team-based orders, and note length were also measured from the EHR efficiency portal and compared with averages for ophthalmologists nationwide using the same EHR. RESULTS: Time spent on exclusive face-to-face communication with patients initially decreased with EHR implementation (2.9 to 2.3 minutes, p = 0.005) but returned to the paper baseline by 6 months (2.8 minutes, p = 0.99). Observed participants outperformed national averages of ophthalmologists using the same vendor system on documentation time per appointment, number of customized note templates, number of customized order lists, utilization of team-based orders, note length, and time spent after-hours on EHR use. CONCLUSION: Informatics-enabled QI interventions can promote patient-centeredness and face-to-face communication in high-volume outpatient ophthalmology encounters. By employing an array of interventions, time spent exclusively talking with the patient returned to levels equivalent to paper charts by 6 months after EHR implementation. This was achieved without requiring EHR redesign, use of scribes, or excessive after-hours work. Documentation efficiency can be achieved using interventions promoting personalization and team-based workflows. Given their efficacy in preserving face-to-face physician-patient interactions, these strategies may help alleviate risk of physician burnout.

Application of Machine Learning to Predict Patient No-Shows in an Academic Pediatric Ophthalmology Clinic.

Patient "no-shows" are missed appointments resulting in clinical inefficiencies, revenue loss, and discontinuity of care. Using secondary electronic health record (EHR) data, we used machine learning to predict patient no-shows in follow-up and new patient visits in pediatric ophthalmology and to evaluate features for importance. The best model, XGBoost, had an area under the receiver operating characteristics curve (AUC) score of 0.90 for predicting no-shows in follow-up visits. The key findings from this study are: (1) secondary use of EHR data can be used to build datasets for predictive modeling and successfully predict patient no-shows in pediatric ophthalmology, (2) models predicting no-shows for follow-up visits are more accurate than those for new patient visits, and (3) the performance of predictive models is more robust in predicting no-shows compared to individual important features. We hope these models will be used for more effective interventions to mitigate the impact ofpatient no-shows.

Methods for Large-Scale Quantitative Analysis of Scribe Impacts on Clinical Documentation.

Many medical providers employ scribes to manage electronic health record (EHR) documentation. Prior studies have shown the benefits of scribes, but no large-scale study has quantitively assessed scribe impact on documentation workflows. We propose methods that leverage EHR data for identifying scribe presence during an office visit, measuring provider documentation time, and determining how notes are edited and composed. In a case study, we found scribe use was associated with less provider documentation time overall (averaging 2.4 minutes or 39% less time, p < 0.001), fewer note edits by providers (8.4% less added and 4.2% less deleted text, p < 0.001), but significantly more documentation time after the visit for four out of seven providers (p < 0.001) and no change in the amount of copied and imported note text. Our methods could validate prior study results, identify variability for determining best practices, and determine that scribes do not improve all aspects of documentation.

Clinical Documentation as End-User Programming.

As healthcare providers have transitioned from paper to electronic health records they have gained access to increasingly sophisticated documentation aids such as custom note templates. However, little is known about how providers use these aids. To address this gap, we examine how 48 ophthalmologists and their staff create and use content-importing phrases - a customizable and composable form of note template - to document office visits across two years. In this case study, we find 1) content-importing phrases were used to document the vast majority of visits (95%), 2) most content imported by these phrases was structured data imported by data-links rather than boilerplate text, and 3) providers primarily used phrases they had created while staff largely used phrases created by other people. We conclude by discussing how framing clinical documentation as end-user programming can inform the design of electronic health records and other documentation systems mixing data and narrative text.