Pediatric Ophthalmology Provider and Staff Attitudes and Patient Satisfaction in Telehealth Implementation During COVID-19.

Purpose:Describe a comprehensive overview of a telehealth implementation process that highlights attitudes and satisfaction scores toward telehealth from patients, providers, and staff in an academic pediatric ophthalmology practice during the early months of the coronavirus disease 2019 (COVID-19) pandemic.Methods:The electronic medical record data for telehealth and in-person visits, as well as a patient experience survey in pediatric ophthalmology were retrospectively reviewed for March 1 to July 31, 2020 and March 1 to July 31, 2019. Patient experience survey results were retrospectively reviewed. All current providers and staff were invited to participate in an anonymous and voluntary survey focused on attitudes at the time of telehealth implementation.Results:During March 1 to July 31, 2020, there was significant increase in telehealth visits (n = 1,006) compared with the same period in 2019 (n = 22). Evaluation and management (E & M) codes (n = 527) were the most commonly used billing codes, and strabismus, nystagmus, and irregular eye movement (n = 496) were the most common telehealth primary diagnoses. The telehealth attitudes survey showed more positive responses from providers than staff. The patient experience survey showed more favorable scores for telehealth visits compared with clinic visits. However, only about 50% of the respondents were satisfied with the technology in terms of ease and quality of connection during their telehealth visits.Conclusions:Telehealth was a satisfactory alternative to clinic visits in our academic pediatric ophthalmology practice during the early phase of the COVID-19 pandemic. Providers and staff had largely positive attitudes toward telehealth; however, future efforts should include strategies to increase staff buy in. Patients had high satisfaction scores with telehealth visits despite connection challenges.

Changes in Electronic Health Record Use Time and Documentation over the Course of a Decade.

PURPOSE: With the current wide adoption of electronic health records (EHRs) by ophthalmologists, there are widespread concerns about the amount of time spent using the EHR. The goal of this study was to examine how the amount of time spent using EHRs as well as related documentation behaviors changed 1 decade after EHR adoption. DESIGN: Single-center cohort study. PARTICIPANTS: Six hundred eighty-five thousand three hundred sixty-one office visits with 70 ophthalmology providers. METHODS: We calculated time spent using the EHR associated with each individual office visit using EHR audit logs and determined chart closure times and progress note length from secondary EHR data. We tracked and modeled how these metrics changed from 2006 to 2016 with linear mixed models. MAIN OUTCOME MEASURES: Minutes spent using the EHR associated with an office visit, chart closure time in hours from the office visit check-in time, and progress note length in characters. RESULTS: Median EHR time per office visit in 2006 was 4.2 minutes (interquartile range [IQR], 3.5 minutes), and increased to 6.4 minutes (IQR, 4.5 minutes) in 2016. Median chart closure time was 2.8 hours (IQR, 21.3 hours) in 2006 and decreased to 2.3 hours (IQR, 18.5 hours) in 2016. In 2006, median note length was 1530 characters (IQR, 1435 characters) and increased to 3838 characters (IQR, 2668.3 characters) in 2016. Linear mixed models found EHR time per office visit was 31.9±0.2% (P < 0.001) greater from 2014 through 2016 than from 2006 through 2010, chart closure time was 6.7±0.3 hours (P < 0.001) shorter from 2014 through 2016 versus 2006 through 2010, and note length was 1807.4±6.5 characters (P < 0.001) longer from 2014 through 2016 versus 2006 through 2010. CONCLUSIONS: After 1 decade of use, providers spend more time using the EHR for an office visit, generate longer notes, and close the chart faster. These changes are likely to represent increased time and documentation pressure for providers. Electronic health record redesign and new documentation regulations may help to address these issues.

Data-Driven Scheduling for Improving Patient Efficiency in Ophthalmology Clinics.

PURPOSE: To improve clinic efficiency through development of an ophthalmology scheduling template developed using simulation models and electronic health record (EHR) data. DESIGN: We created a computer simulation model of 1 pediatric ophthalmologist's clinic using EHR timestamp data, which was used to develop a scheduling template based on appointment length (short, medium, or long). We assessed its impact on clinic efficiency after implementation in the practices of 5 different pediatric ophthalmologists. PARTICIPANTS: We observed and timed patient appointments in person (n = 120) and collected EHR timestamps for 2 years of appointments (n = 650). We calculated efficiency measures for 172 clinic sessions before implementation vs. 119 clinic sessions after implementation. METHODS: We validated clinic workflow timings calculated from EHR timestamps and the simulation models based on them with observed timings. From simulation tests, we developed a new scheduling template and evaluated it with efficiency metrics before vs. after implementation. MAIN OUTCOME MEASURES: Measurements of clinical efficiency (mean clinic volume, patient wait time, examination time, and clinic length). RESULTS: Mean physician examination time calculated from EHR timestamps was 13.8±8.2 minutes and was not statistically different from mean physician examination time from in-person observation (13.3±7.3 minutes; P = 0.7), suggesting that EHR timestamps are accurate. Mean patient wait time for the simulation model (31.2±10.9 minutes) was not statistically different from the observed mean patient wait times (32.6±25.3 minutes; P = 0.9), suggesting that simulation models are accurate. After implementation of the new scheduling template, all 5 pediatric ophthalmologists showed statistically significant improvements in clinic volume (mean increase of 1-3 patients/session; P ≤ 0.05 for 2 providers; P ≤ 0.008 for 3 providers), whereas 4 of 5 had improvements in mean patient wait time (average improvements of 3-4 minutes/patient; statistically significant for 2 providers, P ≤ 0.008). All of the ophthalmologists' examination times remained the same before and after implementation. CONCLUSIONS: Simulation models based on big data from EHRs can test clinic changes before real-life implementation. A scheduling template using predicted appointment length improves clinic efficiency and may generalize to other clinics. Electronic health records have potential to become tools for supporting clinic operations improvement.

Computerized patient-reported symptom assessment in radiotherapy: a pilot randomized, controlled trial.

PURPOSE: Computer-based, patient-reported symptom survey tools have been described for patients undergoing chemotherapy. We hypothesized that patients undergoing radiotherapy might also benefit, so we developed a computer application to acquire symptom ratings from patients and generate summaries for use at point of care office visits and conducted a randomized, controlled pilot trial to test its feasibility. METHODS: Subjects were randomized prior to beginning radiotherapy. Both control and intervention group subjects completed the computerized symptom assessment, but only for the intervention group were printed symptom summaries made available before each weekly office visit. Metrics compared included the Global Distress Index (GDI), concordance of patient-reported symptoms and symptoms discussed by the physician and numbers of new and/or adjusted symptom management medications prescribed. RESULTS: One hundred twelve patients completed the study: 54 in the control and 58 in the intervention arms. There were no differences in GDI over time between the control and intervention groups. In the intervention group, more patient-reported symptoms were actually discussed in radiotherapy office visits: 46/202 vs. 19/230. A sensitivity analysis to account for within-subjects correlation yielded 23.2 vs. 10.3 % (p = 0.03). Medications were started or adjusted at 15.4 % (43/280) of control visits compared to 20.4 % (65/319) of intervention visits (p = 0.07). CONCLUSIONS: This computer application is easy to use and makes extensive patient-reported outcome data available at the point of care. Although no differences were seen in symptom trajectory, patients who had printed symptom summaries had improved communication during office visits and a trend towards a more active symptom management during radiotherapy.

Prediction of multiclass surgical outcomes in glaucoma using multimodal deep learning based on free-text operative notes and structured EHR data.

OBJECTIVE: Surgical outcome prediction is challenging but necessary for postoperative management. Current machine learning models utilize pre- and post-op data, excluding intraoperative information in surgical notes. Current models also usually predict binary outcomes even when surgeries have multiple outcomes that require different postoperative management. This study addresses these gaps by incorporating intraoperative information into multimodal models for multiclass glaucoma surgery outcome prediction. MATERIALS AND METHODS: We developed and evaluated multimodal deep learning models for multiclass glaucoma trabeculectomy surgery outcomes using both structured EHR data and free-text operative notes. We compare those to baseline models that use structured EHR data exclusively, or neural network models that leverage only operative notes. RESULTS: The multimodal neural network had the highest performance with a macro AUROC of 0.750 and F1 score of 0.583. It outperformed the baseline machine learning model with structured EHR data alone (macro AUROC of 0.712 and F1 score of 0.486). Additionally, the multimodal model achieved the highest recall (0.692) for hypotony surgical failure, while the surgical success group had the highest precision (0.884) and F1 score (0.775). DISCUSSION: This study shows that operative notes are an important source of predictive information. The multimodal predictive model combining perioperative notes and structured pre- and post-op EHR data outperformed other models. Multiclass surgical outcome prediction can provide valuable insights for clinical decision-making. CONCLUSIONS: Our results show the potential of deep learning models to enhance clinical decision-making for postoperative management. They can be applied to other specialties to improve surgical outcome predictions.

Guidance for reporting analyses of metadata on electronic health record use.

INTRODUCTION: Research on how people interact with electronic health records (EHRs) increasingly involves the analysis of metadata on EHR use. These metadata can be recorded unobtrusively and capture EHR use at a scale unattainable through direct observation or self-reports. However, there is substantial variation in how metadata on EHR use are recorded, analyzed and described, limiting understanding, replication, and synthesis across studies. RECOMMENDATIONS: In this perspective, we provide guidance to those working with EHR use metadata by describing 4 common types, how they are recorded, and how they can be aggregated into higher-level measures of EHR use. We also describe guidelines for reporting analyses of EHR use metadata-or measures of EHR use derived from them-to foster clarity, standardization, and reproducibility in this emerging and critical area of research.

Identifying and Addressing Barriers to Implementing Core Electronic Health Record Use Metrics for Ambulatory Care: Virtual Consensus Conference Proceedings.

Precise, reliable, valid metrics that are cost-effective and require reasonable implementation time and effort are needed to drive electronic health record (EHR) improvements and decrease EHR burden. Differences exist between research and vendor definitions of metrics. PROCESS: We convened three stakeholder groups (health system informatics leaders, EHR vendor representatives, and researchers) in a virtual workshop series to achieve consensus on barriers, solutions, and next steps to implementing the core EHR use metrics in ambulatory care. CONCLUSION: Actionable solutions identified to address core categories of EHR metric implementation challenges include: (1) maintaining broad stakeholder engagement, (2) reaching agreement on standardized measure definitions across vendors, (3) integrating clinician perspectives, and (4) addressing cognitive and EHR burden. Building upon the momentum of this workshop's outputs offers promise for overcoming barriers to implementing EHR use metrics.

Medical Scribes Have a Variable Impact on Documentation Workflows.

Physicians can reduce their documentation time by working with a scribe. However, what scribes document and how their actions affect existing documentation workflows is unclear. This study leverages electronic health record (EHR) audit logs to observe how scribes affected the documentation workflows of seven physicians and their staff across 13,000 outpatient ophthalmology visits. In addition to editing progress notes, scribes routinely edited exam findings and diagnoses. Scribes with clinical training also edited items such as vital signs that a scribe without clinical training did not. Every physician edited patient records later in the day when working with a scribe and those who deferred their editing the most had some of the largest reductions in EHR time. These results suggest that what scribes document, how physicians work with scribes, and scribe impact on documentation time are all highly variable, highlighting the need for evidence-based best practices.

Development of an Open-Source Annotated Glaucoma Medication Dataset From Clinical Notes in the Electronic Health Record.

Purpose: To describe the methods involved in processing and characteristics of an open dataset of annotated clinical notes from the electronic health record (EHR) annotated for glaucoma medications. Methods: In this study, 480 clinical notes from office visits, medical record numbers (MRNs), visit identification numbers, provider names, and billing codes were extracted for 480 patients seen for glaucoma by a comprehensive or glaucoma ophthalmologist from January 1, 2019, to August 31, 2020. MRNs and all visit data were de-identified using a hash function with salt from the deidentifyr package. All progress notes were annotated for glaucoma medication name, route, frequency, dosage, and drug use using an open-source annotation tool, Doccano. Annotations were saved separately. All protected health information (PHI) in progress notes and annotated files were de-identified using the published de-identifying algorithm Philter. All progress notes and annotations were manually validated by two ophthalmologists to ensure complete de-identification. Results: The final dataset contained 5520 annotated sentences, including those with and without medications, for 480 clinical notes. Manual validation revealed 10 instances of remaining PHI which were manually corrected. Conclusions: Annotated free-text clinical notes can be de-identified for upload as an open dataset. As data availability increases with the adoption of EHRs, free-text open datasets will become increasingly valuable for "big data" research and artificial intelligence development. This dataset is published online and publicly available at https://github.com/jche253/Glaucoma_Med_Dataset. Translational Relevance: This open access medication dataset may be a source of raw data for future research involving big data and artificial intelligence research using free-text.

Frequent but fragmented: use of note templates to document outpatient visits at an academic health center.

Recent changes to billing policy have reduced documentation requirements for outpatient notes, providing an opportunity to rethink documentation workflows. While many providers use templates to write notes-whether to insert short phrases or draft entire notes-we know surprisingly little about how these templates are used in practice. In this retrospective cross-sectional study, we observed the templates that primary providers and other members of the care team used to write the provider progress note for 2.5 million outpatient visits across 52 specialties at an academic health center between 2018 and 2020. Templates were used to document 89% of visits, with a median of 2 used per visit. Only 17% of the 100 230 unique templates were ever used by more than one person and most providers had their own full-note templates. These findings suggest template use is frequent but fragmented, complicating template revision and maintenance. Reframing template use as a form of computer programming suggests ways to maintain the benefits of personalization while leveraging standardization to reduce documentation burden.

Length and Redundancy of Outpatient Progress Notes Across a Decade at an Academic Medical Center.

Importance: There is widespread concern that clinical notes have grown longer and less informative over the past decade. Addressing these concerns requires a better understanding of the magnitude, scope, and potential causes of increased note length and redundancy. Objective: To measure changes between 2009 and 2018 in the length and redundancy of outpatient progress notes across multiple medical specialties and investigate how these measures associate with author experience and method of note entry. Design, Setting, and Participants: This cross-sectional study was conducted at Oregon Health & Science University, a large academic medical center. Participants included clinicians and staff who wrote outpatient progress notes between 2009 and 2018 for a random sample of 200 000 patients. Statistical analysis was performed from March to August 2020. Exposures: Use of a comprehensive electronic health record to document patient care. Main Outcomes and Measures: Note length, note redundancy (ie, the proportion of text identical to the patient's last note), and percentage of templated, copied, or directly typed note text. Results: A total of 2 704 800 notes written by 6228 primary authors across 46 specialties were included in this study. Median note length increased 60.1% (99% CI, 46.7%-75.2%) from a median of 401 words (interquartile range [IQR], 225-660 words) in 2009 to 642 words (IQR, 399-1007 words) in 2018. Median note redundancy increased 10.9 percentage points (99% CI, 7.5-14.3 percentage points) from 47.9% in 2009 to 58.8% in 2018. Notes written in 2018 had a mean value of just 29.4% (99% CI, 28.2%-30.7%) directly typed text with the remaining 70.6% of text being templated or copied. Mixed-effect linear models found that notes with higher proportions of templated or copied text were significantly longer and more redundant (eg, in the 2-year model, each 1% increase in the proportion of copied or templated note text was associated with 1.5% [95% CI, 1.5%-1.5%] and 1.6% [95% CI, 1.6%-1.6%] increases in note length, respectively). Residents and fellows also wrote significantly (26.3% [95% CI, 25.8%-26.7%]) longer notes than more senior authors, as did more recent hires (1.8% for each year later [95% CI, 1.3%-2.4%]). Conclusions and Relevance: In this study, outpatient progress notes grew longer and more redundant over time, potentially limiting their use in patient care. Interventions aimed at reducing outpatient progress note length and redundancy may need to simultaneously address multiple factors such as note template design and training for both new and established clinicians.

Comparing Scribed and Non-scribed Outpatient Progress Notes.

Working with scribes can reduce provider documentation time, but few studies have examined how scribes affect clinical notes. In this retrospective cross-sectional study, we examine over 50,000 outpatient progress notes written with and without scribe assistance by 70 providers across 27 specialties in 2017-2018. We find scribed notes were consistently longer than those written without scribe assistance, with most additional text coming from note templates. Scribed notes were also more likely to contain certain templated lists, such as the patient's medications or past medical history. However, there was significant variation in how working with scribes affected a provider's mix of typed, templated, and copied note text, suggesting providers adapt their documentation workflows to varying degrees when working with scribes. These results suggest working with scribes may contribute to note bloat, but that providers' individual documentation workflows, including their note templates, may have a large impact on scribed note contents.

Clinical Documentation During Scribed and Non-scribed Ophthalmology Office Visits.

PURPOSE: Observe the impact of employing scribes on documentation efficiency in ophthalmology clinics. DESIGN: Single-center retrospective cohort study. PARTICIPANTS: A total of 29,997 outpatient visits conducted by seven attending ophthalmologists between 1/1/2018 and 12/31/2019 were included in the study; 18,483 with a scribe present during the encounter and 11,514 without a scribe present. INTERVENTION: Use of a scribe. MAIN OUTCOME MEASURES: Total physician documentation time, physician documentation time during and after the visit, visit length, time to chart closure, note length, and percent of note text edited by physician. RESULTS: Total physician documentation time was significantly less when working with a scribe (mean ± SD, 4.7 ± 2.9 vs. 7.6 ± 3.8 minutes/note, P<.001), as was documentation time during the visit (2.8 ± 2.2 vs. 5.9 ± 3.1 minutes/note, P<.001). Physicians also edited scribed notes less, deleting 1.9 ± 4.4% of scribes' draft note text and adding 14.8 ± 11.4% of the final note text, compared to deleting 6.0 ± 9.1%(P<.001) of draft note text and adding 21.2 ± 15.3%(P<.001) of final note text when not working with a scribe. However, physician after-visit documentation time was significantly higher with a scribe for 3 of 7 physicians (P<.001). Scribe use was also associated with an office visit length increase of 2.9 minutes (P<.001) per patient and time to chart closure of 3.0 hours (P<.001), according to mixed-effects linear models. CONCLUSIONS: Scribe use was associated with increased documentation efficiency through lower total documentation time and less note editing by physicians. However, the use of a scribe was also associated with longer office visit lengths and time to chart closure. The variability in the impact of scribe use on different measures of documentation efficiency leaves unanswered questions about best practices for the implementation of scribes, and warrants further study of effective scribe use.

Measures of electronic health record use in outpatient settings across vendors.

Electronic health record (EHR) log data capture clinical workflows and are a rich source of information to understand variation in practice patterns. Variation in how EHRs are used to document and support care delivery is associated with clinical and operational outcomes, including measures of provider well-being and burnout. Standardized measures that describe EHR use would facilitate generalizability and cross-institution, cross-vendor research. Here, we describe the current state of outpatient EHR use measures offered by various EHR vendors, guided by our prior conceptual work that proposed seven core measures to describe EHR use. We evaluate these measures and other reporting options provided by vendors for maturity and similarity to previously proposed standardized measures. Working toward improved standardization of EHR use measures can enable and accelerate high-impact research on physician burnout and job satisfaction as well as organizational efficiency and patient health.

Extraction of Active Medications and Adherence Using Natural Language Processing for Glaucoma Patients.

Accuracy of medication data in electronic health records (EHRs) is crucial for patient care and research, but many studies have shown that medication lists frequently contain errors. In contrast, physicians often pay more attention to the clinical notes and record medication information in them. The medication information in notes may be used for medication reconciliation to improve the medication lists' accuracy. However, accurately extracting patient's current medications from free-text narratives is challenging. In this study, we first explored the discrepancies between medication documentation in medication lists and progress notes for glaucoma patients by manually reviewing patients' charts. Next, we developed and validated a named entity recognition model to identify current medication and adherence from progress notes. Lastly, a prototype tool for medication reconciliation using the developed model was demonstrated. In the future, the model has the potential to be incorporated into the EHR system to help with realtime medication reconciliation.

Electronic health record note review in an outpatient specialty clinic: who is looking?

Note entry and review in electronic health records (EHRs) are time-consuming. While some clinics have adopted team-based models of note entry, how these models have impacted note review is unknown in outpatient specialty clinics such as ophthalmology. We hypothesized that ophthalmologists and ancillary staff review very few notes. Using audit log data from 9775 follow-up office visits in an academic ophthalmology clinic, we found ophthalmologists reviewed a median of 1 note per visit (2.6 ± 5.3% of available notes), while ancillary staff reviewed a median of 2 notes per visit (4.1 ± 6.2% of available notes). While prior ophthalmic office visit notes were the most frequently reviewed note type, ophthalmologists and staff reviewed no such notes in 51% and 31% of visits, respectively. These results highlight the collaborative nature of note review and raise concerns about how cumbersome EHR designs affect efficient note review and the utility of prior notes in ophthalmic clinical care.

Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods.

OBJECTIVE: To systematically review published literature and identify consistency and variation in the aims, measures, and methods of studies using electronic health record (EHR) audit logs to observe clinical activities. MATERIALS AND METHODS: In July 2019, we searched PubMed for articles using EHR audit logs to study clinical activities. We coded and clustered the aims, measures, and methods of each article into recurring categories. We likewise extracted and summarized the methods used to validate measures derived from audit logs and limitations discussed of using audit logs for research. RESULTS: Eighty-five articles met inclusion criteria. Study aims included examining EHR use, care team dynamics, and clinical workflows. Studies employed 6 key audit log measures: counts of actions captured by audit logs (eg, problem list viewed), counts of higher-level activities imputed by researchers (eg, chart review), activity durations, activity sequences, activity clusters, and EHR user networks. Methods used to preprocess audit logs varied, including how authors filtered extraneous actions, mapped actions to higher-level activities, and interpreted repeated actions or gaps in activity. Nineteen studies validated results (22%), but only 9 (11%) through direct observation, demonstrating varying levels of measure accuracy. DISCUSSION: While originally designed to aid access control, EHR audit logs have been used to observe diverse clinical activities. However, most studies lack sufficient discussion of measure definition, calculation, and validation to support replication, comparison, and cross-study synthesis. CONCLUSION: EHR audit logs have potential to scale observational research but the complexity of audit log measures necessitates greater methodological transparency and validated standards.

Applications of Artificial Intelligence to Electronic Health Record Data in Ophthalmology.

Widespread adoption of electronic health records (EHRs) has resulted in the collection of massive amounts of clinical data. In ophthalmology in particular, the volume range of data captured in EHR systems has been growing rapidly. Yet making effective secondary use of this EHR data for improving patient care and facilitating clinical decision-making has remained challenging due to the complexity and heterogeneity of these data. Artificial intelligence (AI) techniques present a promising way to analyze these multimodal data sets. While AI techniques have been extensively applied to imaging data, there are a limited number of studies employing AI techniques with clinical data from the EHR. The objective of this review is to provide an overview of different AI methods applied to EHR data in the field of ophthalmology. This literature review highlights that the secondary use of EHR data has focused on glaucoma, diabetic retinopathy, age-related macular degeneration, and cataracts with the use of AI techniques. These techniques have been used to improve ocular disease diagnosis, risk assessment, and progression prediction. Techniques such as supervised machine learning, deep learning, and natural language processing were most commonly used in the articles reviewed.

Application of Machine Learning to Predict Patient No-Shows in an Academic Pediatric Ophthalmology Clinic.

Patient "no-shows" are missed appointments resulting in clinical inefficiencies, revenue loss, and discontinuity of care. Using secondary electronic health record (EHR) data, we used machine learning to predict patient no-shows in follow-up and new patient visits in pediatric ophthalmology and to evaluate features for importance. The best model, XGBoost, had an area under the receiver operating characteristics curve (AUC) score of 0.90 for predicting no-shows in follow-up visits. The key findings from this study are: (1) secondary use of EHR data can be used to build datasets for predictive modeling and successfully predict patient no-shows in pediatric ophthalmology, (2) models predicting no-shows for follow-up visits are more accurate than those for new patient visits, and (3) the performance of predictive models is more robust in predicting no-shows compared to individual important features. We hope these models will be used for more effective interventions to mitigate the impact ofpatient no-shows.