Beyond order-based nursing workload: A retrospective cohort study in intensive care units.

INTRODUCTION: In order to be positioned to address the increasing strain of burnout and worsening nurse shortage, a better understanding of factors that contribute to nursing workload is required. This study aims to examine the difference between order-based and clinically perceived nursing workloads and to quantify factors that contribute to a higher clinically perceived workload. DESIGN: A retrospective cohort study was used on an observational dataset. METHODS: We combined patient flow, nurse staffing and assignment, and workload intensity data and used multivariate linear regression to analyze how various shift, patient, and nurse-level factors, beyond order-based workload, affect nurses' clinically perceived workload. RESULTS: Among 53% of our samples, the clinically perceived workload is higher than the order-based workload. Factors associated with a higher clinically perceived workload include weekend or night shifts, shifts with a higher census, patients within the first 24 h of admission, and male patients. CONCLUSIONS: The order-based workload measures tended to underestimate nurses' clinically perceived workload. We identified and quantified factors that contribute to a higher clinically perceived workload, discussed the potential mechanisms as to how these factors affect the clinically perceived workload, and proposed targeted interventions to better manage nursing workload. CLINICAL RELEVANCE: By identifying factors associated with a high clinically perceived workload, the nurse manager can provide appropriate interventions to lighten nursing workload, which may further reduce the risk of nurse burnout and shortage.

Identifying Barriers to The Implementation of Communicating Narrative Concerns Entered by Registered Nurses, An Early Warning System SmartApp.

BACKGROUND: Nurses are at the frontline of detecting patient deterioration. We developed Communicating Narrative Concerns Entered by Registered Nurses (CONCERN), an early warning system for clinical deterioration that generates a risk prediction score utilizing nursing data. CONCERN was implemented as a randomized clinical trial at two health systems in the Northeastern United States. Following the implementation of CONCERN, our team sought to develop the CONCERN Implementation Toolkit to enable other hospital systems to adopt CONCERN. OBJECTIVE: The aim of this study was to identify the optimal resources needed to implement CONCERN and package these resources into the CONCERN Implementation Toolkit to enable the spread of CONCERN to other hospital sites. METHODS: To accomplish this aim, we conducted qualitative interviews with nurses, prescribing providers, and information technology experts in two health systems. We recruited participants from July 2022 to January 2023. We conducted thematic analysis guided by the Donabedian model. Based on the results of the thematic analysis, we updated the α version of the CONCERN Implementation Toolkit. RESULTS: There was a total of 32 participants included in our study. In total, 12 themes were identified, with four themes mapping to each domain in Donabedian's model (i.e., structure, process, and outcome). Eight new resources were added to the CONCERN Implementation Toolkit. CONCLUSIONS: This study validated the α version of the CONCERN Implementation Toolkit. Future studies will focus on returning the results of the Toolkit to the hospital sites to validate the ß version of the CONCERN Implementation Toolkit. As the development of early warning systems continues to increase and clinician workflows evolve, the results of this study will provide considerations for research teams interested in implementing early warning systems in the acute care setting.

Examining the Generalizability of Pretrained De-identification Transformer Models on Narrative Nursing Notes.

BACKGROUND: Narrative nursing notes are a valuable resource in informatics research with unique predictive signals about patient care. The open sharing of these data, however, is appropriately constrained by rigorous regulations set by the Health Insurance Portability and Accountability Act (HIPAA) for the protection of privacy. Several models have been developed and evaluated on the open-source i2b2 dataset. A focus on the generalizability of these models with respect to nursing notes remains understudied. OBJECTIVES: The study aims to understand the generalizability of pretrained transformer models and investigate the variability of personal protected health information (PHI) distribution patterns between discharge summaries and nursing notes with a goal to inform the future design for model evaluation schema. METHODS: Two pretrained transformer models (RoBERTa, ClinicalBERT) fine-tuned on i2b2 2014 discharge summaries were evaluated on our data inpatient nursing notes and compared with the baseline performance. Statistical testing was deployed to assess differences in PHI distribution across discharge summaries and nursing notes. RESULTS: RoBERTa achieved the optimal performance when tested on an external source of data, with an F1 score of 0.887 across PHI categories and 0.932 in the PHI binary task. Overall, discharge summaries contained a higher number of PHI instances and categories of PHI compared with inpatient nursing notes. CONCLUSION: The study investigated the applicability of two pretrained transformers on inpatient nursing notes and examined the distinctions between nursing notes and discharge summaries concerning the utilization of personal PHI. Discharge summaries presented a greater quantity of PHI instances and types when compared with narrative nursing notes, but narrative nursing notes exhibited more diversity in the types of PHI present, with some pertaining to patient's personal life. The insights obtained from the research help improve the design and selection of algorithms, as well as contribute to the development of suitable performance thresholds for PHI.

Understanding the Technical Implementation of a Clinical Decision Support SmartApp: A Qualitative Analysis.

CONCERN is a SmartApp that identifies patients at risk for deterioration. This study aimed to understand the technical components and processes that should be included in our Implementation Toolkit. In focus groups with technical experts five themes emerged: 1) implementation challenges, 2) implementation facilitators, 3) project management, 4) stakeholder engagement, and 5) security assessments. Our results may aid other teams in implementing healthcare SmartApps.

Shifting Mindsets: The Impact of a Patient Portal on Functioning and Recovery in a Mental Health Setting.

OBJECTIVE: This study aims to understand whether higher use of a patient portal can have an impact on mental health functioning and recovery. METHOD: A mixed methods approach was used for this study. In 2019-2021, patients with mental health diagnoses at outpatient clinics in an academic centre were invited to complete World Health Organization Disability Assessment Scale 12 (WHODAS-12) and Mental Health Recovery Measure surveys at baseline, 3 months, and 6 months after signing up for the portal. At the 3-month time point, patients were invited to a semistructured interview with a member of the team to contextualize the findings obtained from the surveys. Analytics data was also collected from the platform to understand usage patterns on the portal. RESULTS: Overall, 113 participants were included in the analysis. There was no significant change in mental health functioning and recovery scores over the 6-month period. However, suboptimal usage was observed as 46% of participants did not complete any tasks within the portal. Thirty-five participants had low use of the portal (1-9 interactions) and 18 participants had high usage (10+ interactions). There were also no differences in mental health functioning and recovery scores between low and high users of the portal. Qualitative interviews highlighted many opportunities where the portal can support overall functioning and mental health recovery. CONCLUSIONS: Collectively, this study suggests that higher use of a portal had no impact, either positive or negative, on mental health outcomes. While it may offer convenience and improved patient satisfaction, adequate support is needed to fully enable these opportunities for patient care. As the type of interaction with the portal was not specifically addressed, future work should focus on looking at ways to support patient engagement and portal usage throughout their care journey.

Predicting emergency department visits and hospitalizations for patients with heart failure in home healthcare using a time series risk model.

OBJECTIVES: Little is known about proactive risk assessment concerning emergency department (ED) visits and hospitalizations in patients with heart failure (HF) who receive home healthcare (HHC) services. This study developed a time series risk model for predicting ED visits and hospitalizations in patients with HF using longitudinal electronic health record data. We also explored which data sources yield the best-performing models over various time windows. MATERIALS AND METHODS: We used data collected from 9362 patients from a large HHC agency. We iteratively developed risk models using both structured (eg, standard assessment tools, vital signs, visit characteristics) and unstructured data (eg, clinical notes). Seven specific sets of variables included: (1) the Outcome and Assessment Information Set, (2) vital signs, (3) visit characteristics, (4) rule-based natural language processing-derived variables, (5) term frequency-inverse document frequency variables, (6) Bio-Clinical Bidirectional Encoder Representations from Transformers variables, and (7) topic modeling. Risk models were developed for 18 time windows (1-15, 30, 45, and 60 days) before an ED visit or hospitalization. Risk prediction performances were compared using recall, precision, accuracy, F1, and area under the receiver operating curve (AUC). RESULTS: The best-performing model was built using a combination of all 7 sets of variables and the time window of 4 days before an ED visit or hospitalization (AUC = 0.89 and F1 = 0.69). DISCUSSION AND CONCLUSION: This prediction model suggests that HHC clinicians can identify patients with HF at risk for visiting the ED or hospitalization within 4 days before the event, allowing for earlier targeted interventions.

Eliciting Insights From Chat Logs of the 25X5 Symposium to Reduce Documentation Burden: Novel Application of Topic Modeling.

BACKGROUND: Addressing clinician documentation burden through "targeted solutions" is a growing priority for many organizations ranging from government and academia to industry. Between January and February 2021, the 25 by 5: Symposium to Reduce Documentation Burden on US Clinicians by 75% (25X5 Symposium) convened across 2 weekly 2-hour sessions among experts and stakeholders to generate actionable goals for reducing clinician documentation over the next 5 years. Throughout this web-based symposium, we passively collected attendees' contributions to a chat functionality-with their knowledge that the content would be deidentified and made publicly available. This presented a novel opportunity to synthesize and understand participants' perceptions and interests from chat messages. We performed a content analysis of 25X5 Symposium chat logs to identify themes about reducing clinician documentation burden. OBJECTIVE: The objective of this study was to explore unstructured chat log content from the web-based 25X5 Symposium to elicit latent insights on clinician documentation burden among clinicians, health care leaders, and other stakeholders using topic modeling. METHODS: Across the 6 sessions, we captured 1787 messages among 167 unique chat participants cumulatively; 14 were private messages not included in the analysis. We implemented a latent Dirichlet allocation (LDA) topic model on the aggregated dataset to identify clinician documentation burden topics mentioned in the chat logs. Coherence scores and manual examination informed optimal model selection. Next, 5 domain experts independently and qualitatively assigned descriptive labels to model-identified topics and classified them into higher-level categories, which were finalized through a panel consensus. RESULTS: We uncovered ten topics using the LDA model: (1) determining data and documentation needs (422/1773, 23.8%); (2) collectively reassessing documentation requirements in electronic health records (EHRs) (252/1773, 14.2%); (3) focusing documentation on patient narrative (162/1773, 9.1%); (4) documentation that adds value (147/1773, 8.3%); (5) regulatory impact on clinician burden (142/1773, 8%); (6) improved EHR user interface and design (128/1773, 7.2%); (7) addressing poor usability (122/1773, 6.9%); (8) sharing 25X5 Symposium resources (122/1773, 6.9%); (9) capturing data related to clinician practice (113/1773, 6.4%); and (10) the role of quality measures and technology in burnout (110/1773, 6.2%). Among these 10 topics, 5 high-level categories emerged: consensus building (821/1773, 46.3%), burden sources (365/1773, 20.6%), EHR design (250/1773, 14.1%), patient-centered care (162/1773, 9.1%), and symposium comments (122/1773, 6.9%). CONCLUSIONS: We conducted a topic modeling analysis on 25X5 Symposium multiparticipant chat logs to explore the feasibility of this novel application and elicit additional insights on clinician documentation burden among attendees. Based on the results of our LDA analysis, consensus building, burden sources, EHR design, and patient-centered care may be important themes to consider when addressing clinician documentation burden. Our findings demonstrate the value of topic modeling in discovering topics associated with clinician documentation burden using unstructured textual content. Topic modeling may be a suitable approach to examine latent themes presented in web-based symposium chat logs.

Assessing Usability and Ambulatory Clinical Staff Satisfaction with Two Electronic Health Records.

BACKGROUND: A growing body of literature has linked usability limitations within electronic health records (EHRs) to adverse outcomes which may in turn affect EHR system transitions. NewYork-Presbyterian Hospital, Columbia University College of Physicians and Surgeons (CU), and Weill Cornell Medical College (WC) are a tripartite organization with large academic medical centers that initiated a phased transition of their EHRs to one system, EpicCare. OBJECTIVES: This article characterizes usability perceptions stratified by provider roles by surveying WC ambulatory clinical staff already utilizing EpicCare and CU ambulatory clinical staff utilizing iterations of Allscripts before the implementation of EpicCare campus-wide. METHODS: A customized 19-question electronic survey utilizing usability constructs based on the Health Information Technology Usability Evaluation Scale was anonymously administered prior to EHR transition. Responses were recorded with self-reported demographics. RESULTS: A total of 1,666 CU and 1,065 WC staff with ambulatory self-identified work setting were chosen. Select demographic statistics between campus staff were generally similar with small differences in patterns of clinical and EHR experience. Results demonstrated significant differences in EHR usability perceptions among ambulatory staff based on role and EHR system. WC staff utilizing EpicCare accounted for more favorable usability metrics than CU across all constructs. Ordering providers (OPs) denoted less usability than non-OPs. The Perceived Usefulness and User Control constructs accounted for the largest differences in usability perceptions. The Cognitive Support and Situational Awareness construct was similarly low for both campuses. Prior EHR experience demonstrated limited associations. CONCLUSION: Usability perceptions can be affected by role and EHR system. OPs consistently denoted less usability overall and were more affected by EHR system than non-OPs. While there was greater perceived usability for EpicCare to perform tasks related to care coordination, documentation, and error prevention, there were persistent shortcomings regarding tab navigation and cognitive burden reduction, which have implications on provider efficiency and wellness.

Understanding the perceived role of electronic health records and workflow fragmentation on clinician documentation burden in emergency departments.

OBJECTIVE: Understand the perceived role of electronic health records (EHR) and workflow fragmentation on clinician documentation burden in the emergency department (ED). METHODS: From February to June 2022, we conducted semistructured interviews among a national sample of US prescribing providers and registered nurses who actively practice in the adult ED setting and use Epic Systems' EHR. We recruited participants through professional listservs, social media, and email invitations sent to healthcare professionals. We analyzed interview transcripts using inductive thematic analysis and interviewed participants until we achieved thematic saturation. We finalized themes through a consensus-building process. RESULTS: We conducted interviews with 12 prescribing providers and 12 registered nurses. Six themes were identified related to EHR factors perceived to contribute to documentation burden including lack of advanced EHR capabilities, absence of EHR optimization for clinicians, poor user interface design, hindered communication, increased manual work, and added workflow blockages, and five themes associated with cognitive load. Two themes emerged in the relationship between workflow fragmentation and EHR documentation burden: underlying sources and adverse consequences. DISCUSSION: Obtaining further stakeholder input and consensus is essential to determine whether these perceived burdensome EHR factors could be extended to broader contexts and addressed through optimizing existing EHR systems alone or through a broad overhaul of the EHR's architecture and primary purpose. CONCLUSION: While most clinicians perceived that the EHR added value to patient care and care quality, our findings underscore the importance of designing EHRs that are in harmony with ED clinical workflows to alleviate the clinician documentation burden.

Use of Real-Time Information to Predict Future Arrivals in the Emergency Department.

STUDY OBJECTIVE: We aimed to build prediction models for shift-level emergency department (ED) patient volume that could be used to facilitate prediction-driven staffing. We sought to evaluate the predictive power of rich real-time information and understand 1) which real-time information had predictive power and 2) what prediction techniques were appropriate for forecasting ED demand. METHODS: We conducted a retrospective study in an ED site in a large academic hospital in New York City. We examined various prediction techniques, including linear regression, regression trees, extreme gradient boosting, and time series models. By comparing models with and without real-time predictors, we assessed the potential gain in prediction accuracy from real-time information. RESULTS: Real-time predictors improved prediction accuracy on models without contemporary information from 5% to 11%. Among extensive real-time predictors examined, recent patient arrival counts, weather, Google trends, and concurrent patient comorbidity information had significant predictive power. Out of all the forecasting techniques explored, SARIMAX (Seasonal Autoregressive Integrated Moving Average with eXogenous factors) achieved the smallest out-of-sample the root mean square error (RMSE) of 14.656 and mean absolute prediction error (MAPE) of 8.703%. Linear regression was the second best, with out-of-sample RMSE and MAPE equal to 15.366 and 9.109%, respectively. CONCLUSION: Real-time information was effective in improving the prediction accuracy of ED demand. Practice and policy implications for designing staffing paradigms with real-time demand forecasts to reduce ED congestion were discussed.

The identification of clusters of risk factors and their association with hospitalizations or emergency department visits in home health care.

AIMS: To identify clusters of risk factors in home health care and determine if the clusters are associated with hospitalizations or emergency department visits. DESIGN: A retrospective cohort study. METHODS: This study included 61,454 patients pertaining to 79,079 episodes receiving home health care between 2015 and 2017 from one of the largest home health care organizations in the United States. Potential risk factors were extracted from structured data and unstructured clinical notes analysed by natural language processing. A K-means cluster analysis was conducted. Kaplan-Meier analysis was conducted to identify the association between clusters and hospitalizations or emergency department visits during home health care. RESULTS: A total of 11.6% of home health episodes resulted in hospitalizations or emergency department visits. Risk factors formed three clusters. Cluster 1 is characterized by a combination of risk factors related to "impaired physical comfort with pain," defined as situations where patients may experience increased pain. Cluster 2 is characterized by "high comorbidity burden" defined as multiple comorbidities or other risks for hospitalization (e.g., prior falls). Cluster 3 is characterized by "impaired cognitive/psychological and skin integrity" including dementia or skin ulcer. Compared to Cluster 1, the risk of hospitalizations or emergency department visits increased by 1.95 times for Cluster 2 and by 2.12 times for Cluster 3 (all p < .001). CONCLUSION: Risk factors were clustered into three types describing distinct characteristics for hospitalizations or emergency department visits. Different combinations of risk factors affected the likelihood of these negative outcomes. IMPACT: Cluster-based risk prediction models could be integrated into early warning systems to identify patients at risk for hospitalizations or emergency department visits leading to more timely, patient-centred care, ultimately preventing these events. PATIENT OR PUBLIC CONTRIBUTION: There was no involvement of patients in developing the research question, determining the outcome measures, or implementing the study.

A Delphi Process to Identify Relevant Outcomes That May Be Associated With a Predictive Analytic Tool to Detect Hemodynamic Deterioration in the Intensive Care Unit.

Background The critical care literature has seen an increase in the development and validation of tools using artificial intelligence for early detection of patient events or disease onset in the intensive care unit (ICU). The hemodynamic stability index (HSI) was found to have an AUC of 0.82 in predicting the need for hemodynamic intervention in the ICU. Future studies using this tool may benefit from targeting those outcomes that are more relevant to clinicians and most achievable. Methods A three-round Delphi study was conducted with a panel of 10 critical care physicians and three nurses in the United States to identify outcomes that may be most relevant and achievable with the HSI when evaluated for use in the ICU. To achieve criteria for relevance, at least 65% of panelists had to rate an outcome as a 4 or 5 on a 5-point scale. Results Nineteen of 24 outcomes that may be associated with the HSI achieved consensus for relevance. The Kemeny-Young approach was used to develop a matrix depicting the distribution of outcomes considering both relevance and achievability. "Reduces time spent in hemodynamic instability" and "reduces times to recognition of hemodynamic instability" were the highest-ranking outcomes considering both relevance and achievability. Conclusion This Delphi study was a feasible method to identify relevant outcomes that may be associated with an appropriate predictive analytic tool in the ICU. These findings can provide insight to researchers looking to study such tools to impact outcomes relevant to critical care practitioners. Future studies should test these tools in the ICU that target the most clinically relevant and achievable outcomes, such as time spent hemodynamically unstable or time until actionable nursing assessment or treatment.

A Practical Approach to Optimize Computerized Provider Order Entry Systems and Reduce Clinician Burden: Pre-Post Evaluation of Vendor-Derived "Order Friction" Data.

Computerized provider order entry (CPOE) systems have been cited as a significant contributor to clinician burden. Vendor-derived measures and data sets have been developed to help with optimization of CPOE systems. We describe how we analyzed vendor-derived Order Friction (OF) EHR log data at our health system and propose a practical approach for optimizing CPOE systems by reducing OF. We also conducted a pre-post intervention study using OF data to evaluate the impact of defaulting the frequency of urine, stool and nasal swab tests and found that all modified orders had significantly fewer changes required per order (p<0.01). Our proposed approach is a six-step process: 1) understand the ordering process, 2) understand OF data elements contextually, 3) explore ordering user-level factors, 4) evaluate order volume and friction from different order sources, 5) optimize order-level design, 6) identify high volume alerts to evaluate for appropriateness.

Identifying Reuse and Redundancies in Respiratory Flowsheet Documentation: Implications for Clinician Documentation Burden.

Documentation burden is experienced by clinical end-users of the electronic health record. Flowsheet measure reuse and clinical concept redundancy are two contributors to documentation burden. In this paper, we described nursing flowsheet documentation hierarchy and frequency of use for one month from two hospitals in our health system. We examined respiratory care management documentation in greater detail. We found 59 instances of reuse of respiratory care flowsheet measure fields over two or more templates and groups, and 5 instances of clinical concept redundancy. Flowsheet measure fields for physical assessment observations and measurements were the most frequently documented and most reused, whereas respiratory intervention documentation was less frequently reused. Further research should investigate the relationship between flowsheet measure reuse and redundancy and EHR information overload and documentation burden.

A Computational Framework to Evaluate Emergency Department Clinician Task Switching in the Electronic Health Record Using Event Logs.

Workflow fragmentation, defined as task switching, may be one proxy to quantify electronic health record (EHR) documentation burden in the emergency department (ED). Few measures have been operationalized to evaluate task switching at scale. Theoretically grounded in the time-based resource-sharing model (TBRSM) which conceives task switching as proportional to the cognitive load experienced, we describe the functional relationship between cognitive load and the time and effort constructs previously applied for measuring documentation burden. We present a computational framework, COMBINE, to evaluate multilevel task switching in the ED using EHR event logs. Based on this framework, we conducted a descriptive analysis on task switching among 63 full-time ED physicians from one ED site using EHR event logs extracted between April-June 2021 (n=2,068,605 events) which were matched to scheduled shifts (n=952). On average, we found a high volume of event-level (185.8±75.3/hr) and within-(6.6±1.7/chart) and between-patient chart (27.5±23.6/hr) switching per shift worked.

Variability in Nursing Documentation Patterns across Patients' Hospital Stays.

This study explores the variability in nursing documentation patterns in acute care and ICU settings, focusing on vital signs and note documentation, and examines how these patterns vary across patients' hospital stays, documentation types, and comorbidities. In both acute care and critical care settings, there was significant variability in nursing documentation patterns across hospital stays, by documentation type, and by patients' comorbidities. The results suggest that nurses adapt their documentation practices in response to their patients' fluctuating needs and conditions, highlighting the need to facilitate more individualized care and tailored documentation practices. The implications of these findings can inform decisions on nursing workload management, clinical decision support tools, and EHR optimizations.

Comparison of a Voluntary Safety Reporting System to a Global Trigger Tool for Identifying Adverse Events in an Oncology Population.

OBJECTIVE: There is a lack of research on adverse event (AE) detection in oncology patients, despite the propensity for iatrogenic harm. Two common methods include voluntary safety reporting (VSR) and chart review tools, such as the Institute for Healthcare Improvement's Global Trigger Tool (GTT). Our objective was to compare frequency and type of AEs detected by a modified GTT compared with VSR for identifying AEs in oncology patients in a larger clinical trial. METHODS: Patients across 6 oncology units (from July 1, 2013, through May 29, 2015) were randomly selected. Retrospective chart reviews were conducted by a team of nurses and physicians to identify AEs using the GTT. The VSR system was queried by the department of quality and safety of the hospital. Adverse event frequencies, type, and harm code for both methods were compared. RESULTS: The modified GTT detected 0.90 AEs per patient (79 AEs in 88 patients; 95% [0.71-1.12] AEs per patient) that were predominantly medication AEs (53/79); more than half of the AEs caused harm to the patients (41/79, 52%), but only one quarter were preventable (21/79; 27%). The VSR detected 0.24 AEs per patient (21 AEs in 88 patients; 95% [0.15-0.37] AEs per patient), a large plurality of which were medication/intravenous related (8/21); more than half did not cause harm (70%). Only 2% of the AEs (2/100) were detected by both methods. CONCLUSIONS: Neither the modified GTT nor the VSR system alone is sufficient for detecting AEs in oncology patient populations. Further studies exploring methods such as automated AE detection from electronic health records and leveraging patient-reported AEs are needed.

Factors Influencing Clinician Trust in Predictive Clinical Decision Support Systems for In-Hospital Deterioration: Qualitative Descriptive Study.

BACKGROUND: Clinician trust in machine learning-based clinical decision support systems (CDSSs) for predicting in-hospital deterioration (a type of predictive CDSS) is essential for adoption. Evidence shows that clinician trust in predictive CDSSs is influenced by perceived understandability and perceived accuracy. OBJECTIVE: The aim of this study was to explore the phenomenon of clinician trust in predictive CDSSs for in-hospital deterioration by confirming and characterizing factors known to influence trust (understandability and accuracy), uncovering and describing other influencing factors, and comparing nurses' and prescribing providers' trust in predictive CDSSs. METHODS: We followed a qualitative descriptive methodology conducting directed deductive and inductive content analysis of interview data. Directed deductive analyses were guided by the human-computer trust conceptual framework. Semistructured interviews were conducted with nurses and prescribing providers (physicians, physician assistants, or nurse practitioners) working with a predictive CDSS at 2 hospitals in Mass General Brigham. RESULTS: A total of 17 clinicians were interviewed. Concepts from the human-computer trust conceptual framework-perceived understandability and perceived technical competence (ie, perceived accuracy)-were found to influence clinician trust in predictive CDSSs for in-hospital deterioration. The concordance between clinicians' impressions of patients' clinical status and system predictions influenced clinicians' perceptions of system accuracy. Understandability was influenced by system explanations, both global and local, as well as training. In total, 3 additional themes emerged from the inductive analysis. The first, perceived actionability, captured the variation in clinicians' desires for predictive CDSSs to recommend a discrete action. The second, evidence, described the importance of both macro- (scientific) and micro- (anecdotal) evidence for fostering trust. The final theme, equitability, described fairness in system predictions. The findings were largely similar between nurses and prescribing providers. CONCLUSIONS: Although there is a perceived trade-off between machine learning-based CDSS accuracy and understandability, our findings confirm that both are important for fostering clinician trust in predictive CDSSs for in-hospital deterioration. We found that reliance on the predictive CDSS in the clinical workflow may influence clinicians' requirements for trust. Future research should explore the impact of reliance, the optimal explanation design for enhancing understandability, and the role of perceived actionability in driving trust.

25 × 5 Symposium to Reduce Documentation Burden: Report-out and Call for Action.

BACKGROUND: The widespread adoption of electronic health records and a simultaneous increase in regulatory demands have led to an acceleration of documentation requirements among clinicians. The corresponding burden from documentation requirements is a central contributor to clinician burnout and can lead to an increased risk of suboptimal patient care. OBJECTIVE: To address the problem of documentation burden, the 25 by 5: Symposium to Reduce Documentation Burden on United States Clinicians by 75% by 2025 (Symposium) was organized to provide a forum for experts to discuss the current state of documentation burden and to identify specific actions aimed at dramatically reducing documentation burden for clinicians. METHODS: The Symposium consisted of six weekly sessions with 33 presentations. The first four sessions included panel presentations discussing the challenges related to documentation burden. The final two sessions consisted of breakout groups aimed at engaging attendees in establishing interventions for reducing clinical documentation burden. Steering Committee members analyzed notes from each breakout group to develop a list of action items. RESULTS: The Steering Committee synthesized and prioritized 82 action items into Calls to Action among three stakeholder groups: Providers and Health Systems, Vendors, and Policy and Advocacy Groups. Action items were then categorized into as short-, medium-, or long-term goals. Themes that emerged from the breakout groups' notes include the following: accountability, evidence is critical, education and training, innovation of technology, and other miscellaneous goals (e.g., vendors will improve shared knowledge databases). CONCLUSION: The Symposium successfully generated a list of interventions for short-, medium-, and long-term timeframes as a launching point to address documentation burden in explicit action-oriented ways. Addressing interventions to reduce undue documentation burden placed on clinicians will necessitate collaboration among all stakeholders.

Clinical notes: An untapped opportunity for improving risk prediction for hospitalization and emergency department visit during home health care.

BACKGROUND/OBJECTIVE: Between 10 and 25% patients are hospitalized or visit emergency department (ED) during home healthcare (HHC). Given that up to 40% of these negative clinical outcomes are preventable, early and accurate prediction of hospitalization risk can be one strategy to prevent them. In recent years, machine learning-based predictive modeling has become widely used for building risk models. This study aimed to compare the predictive performance of four risk models built with various data sources for hospitalization and ED visits in HHC. METHODS: Four risk models were built using different variables from two data sources: structured data (i.e., Outcome and Assessment Information Set (OASIS) and other assessment items from the electronic health record (EHR)) and unstructured narrative-free text clinical notes for patients who received HHC services from the largest non-profit HHC organization in New York between 2015 and 2017. Then, five machine learning algorithms (logistic regression, Random Forest, Bayesian network, support vector machine (SVM), and Naïve Bayes) were used on each risk model. Risk model performance was evaluated using the F-score and Precision-Recall Curve (PRC) area metrics. RESULTS: During the study period, 8373/86,823 (9.6%) HHC episodes resulted in hospitalization or ED visits. Among five machine learning algorithms on each model, the SVM showed the highest F-score (0.82), while the Random Forest showed the highest PRC area (0.864). Adding information extracted from clinical notes significantly improved the risk prediction ability by up to 16.6% in F-score and 17.8% in PRC. CONCLUSION: All models showed relatively good hospitalization or ED visit risk predictive performance in HHC. Information from clinical notes integrated with the structured data improved the ability to identify patients at risk for these emergent care events.