Human-Centered Design to Address Biases in Artificial Intelligence.

The potential of artificial intelligence (AI) to reduce health care disparities and inequities is recognized, but it can also exacerbate these issues if not implemented in an equitable manner. This perspective identifies potential biases in each stage of the AI life cycle, including data collection, annotation, machine learning model development, evaluation, deployment, operationalization, monitoring, and feedback integration. To mitigate these biases, we suggest involving a diverse group of stakeholders, using human-centered AI principles. Human-centered AI can help ensure that AI systems are designed and used in a way that benefits patients and society, which can reduce health disparities and inequities. By recognizing and addressing biases at each stage of the AI life cycle, AI can achieve its potential in health care.

Understanding the Information Needs and Context of Trauma Handoffs to Design Automated Sensing Clinical Documentation Technologies: Qualitative Mixed-Method Study of Military and Civilian Cases.

BACKGROUND: Current methods of communication between the point of injury and receiving medical facilities rely on verbal communication, supported by brief notes and the memory of the field medic. This communication can be made more complete and reliable with technologies that automatically document the actions of field medics. However, designing state-of-the-art technology for military field personnel and civilian first responders is challenging due to the barriers researchers face in accessing the environment and understanding situated actions and cognitive models employed in the field. OBJECTIVE: To identify design insights for an automated sensing clinical documentation (ASCD) system, we sought to understand what information is transferred in trauma cases between prehospital and hospital personnel, and what contextual factors influence the collection, management, and handover of information in trauma cases, in both military and civilian cases. METHODS: Using a multi-method approach including video review and focus groups, we developed an understanding of the information needs of trauma handoffs and the context of field documentation to inform the design of an automated sensing documentation system that uses wearables, cameras, and environmental sensors to passively infer clinical activity and automatically produce documentation. RESULTS: Comparing military and civilian trauma documentation and handoff, we found similarities in the types of data collected and the prioritization of information. We found that military environments involved many more contextual factors that have implications for design, such as the physical environment (eg, heat, lack of lighting, lack of power) and the potential for active combat and triage, creating additional complexity. CONCLUSIONS: An ineffectiveness of communication is evident in both the civilian and military worlds. We used multiple methods of inquiry to study the information needs of trauma care and handoff, and the context of medical work in the field. Our findings informed the design and evaluation of an automated documentation tool. The data illustrated the need for more accurate recordkeeping, specifically temporal aspects, during transportation, and characterized the environment in which field testing of the developed tool will take place. The employment of a systems perspective in this project produced design insights that our team would not have identified otherwise. These insights created exciting and interesting challenges for the technical team to resolve.

Physicians' perspectives on receiving unsolicited genomic results.

PURPOSE: Physicians increasingly receive genomic test results they did not order, which we term "unsolicited genomic results" (UGRs). We asked physicians how they think such results will affect them and their patients. METHODS: Semistructured interviews were conducted with adult and pediatric primary care and subspecialty physicians at four sites affiliated with a large-scale return-of-results project led by the Electronic Medical Records and Genomics (eMERGE) Network. Twenty-five physicians addressed UGRs and (1) perceived need for actionability, (2) impact on patients, (3) health care workflow, (4) return of results process, and (5) responsibility for results. RESULTS: Physicians prioritize actionability of UGRs and the need for clear, evidence-based "paths" for action coupled with clinical decision support (CDS). They identified potential harms to patients including anxiety, false reassurance, and clinical disutility. Clinicians worried about anticipated workflow issues including responding to UGRs and unreimbursed time. They disagreed about who was responsible for responding to UGRs. CONCLUSION: The prospect of receiving UGRs for otherwise healthy patients raises important concerns for physicians. Their responses informed development of an in-depth survey for physicians following return of UGRs. Strategic workflow integration of UGRs will likely be necessary to empower physicians to serve their patients effectively.

A taxonomy of impacts on clinical and translational research from community stakeholder engagement.

BACKGROUND: Community engagement is increasingly recognized as a valuable tool in clinical and translational research; however, the impact of engagement is not fully understood. No standard nomenclature yet exists to clearly define how research changes when community stakeholders are engaged across the research spectrum. This severely limits our ability to assess the value of community engagement in research. To address this gap, we developed a taxonomy for characterizing and classifying changes in research due to community engagement. METHODS: Using an iterative process, we (a) identified areas of potential impact associated with community engagement from author experience, (b) categorized these in taxonomic bins based on research stages, (c) conducted semi-structured interviews with researchers and community stakeholders, (d) validated the codebook in a sample dataset and (e) refined the taxonomy based on the validation. Community stakeholders were involved in every step of the process including as members of the primary study team. RESULTS: The final taxonomy catalogues changes into eleven domains corresponding to research phases. Each domain includes 2-4 dimensions depicting concepts within the domain's scope and, within each dimension, 2-10 elements labelling activities through which community engagement could change research. CONCLUSIONS: Community engagement has great potential to enhance clinical and translational research. This taxonomy provides a common vocabulary and framework for understanding the impact of community engagement and suggests metrics for assessing the value of community engagement in research.

Randomized Controlled Comparative Effectiveness Trial of Risk Model-Guided Clinical Decision Support for Suicide Screening.

Suicide prevention requires risk identification, appropriate intervention, and follow-up. Traditional risk identification relies on patient self-reporting, support network reporting, or face-to-face screening with validated instruments or history and physical exam. In the last decade, statistical risk models have been studied and more recently deployed to augment clinical judgment. Models have generally been found to be low precision or problematic at scale due to low incidence. Few have been tested in clinical practice, and none have been tested in clinical trials to our knowledge. Methods: We report the results of a pragmatic randomized controlled trial (RCT) in three outpatient adult Neurology clinic settings. This two-arm trial compared the effectiveness of Interruptive and Non-Interruptive Clinical Decision Support (CDS) to prompt further screening of suicidal ideation for those predicted to be high risk using a real-time, validated statistical risk model of suicide attempt risk, with the decision to screen as the primary end point. Secondary outcomes included rates of suicidal ideation and attempts in both arms. Manual chart review of every trial encounter was used to determine if suicide risk assessment was subsequently documented. Results: From August 16, 2022, through February 16, 2023, our study randomized 596 patient encounters across 561 patients for providers to receive either Interruptive or Non-Interruptive CDS in a 1:1 ratio. Adjusting for provider cluster effects, Interruptive CDS led to significantly higher numbers of decisions to screen (42%=121/289 encounters) compared to Non-Interruptive CDS (4%=12/307) (odds ratio=17.7, p-value <0.001). Secondarily, no documented episodes of suicidal ideation or attempts occurred in either arm. While the proportion of documented assessments among those noting the decision to screen was higher for providers in the Non-Interruptive arm (92%=11/12) than in the Interruptive arm (52%=63/121), the interruptive CDS was associated with more frequent documentation of suicide risk assessment (63/289 encounters compared to 11/307, p-value<0.001). Conclusions: In this pragmatic RCT of real-time predictive CDS to guide suicide risk assessment, Interruptive CDS led to higher numbers of decisions to screen and documented suicide risk assessments. Well-powered large-scale trials randomizing this type of CDS compared to standard of care are indicated to measure effectiveness in reducing suicidal self-harm. ClinicalTrials.gov Identifier: NCT05312437.

Competencies for the Use of Artificial Intelligence-Based Tools by Health Care Professionals.

PURPOSE: The expanded use of clinical tools that incorporate artificial intelligence (AI) methods has generated calls for specific competencies for effective and ethical use. This qualitative study used expert interviews to define AI-related clinical competencies for health care professionals. METHOD: In 2021, a multidisciplinary team interviewed 15 experts in the use of AI-based tools in health care settings about the clinical competencies health care professionals need to work effectively with such tools. Transcripts of the semistructured interviews were coded and thematically analyzed. Draft competency statements were developed and provided to the experts for feedback. The competencies were finalized using a consensus process across the research team. RESULTS: Six competency domain statements and 25 subcompetencies were formulated from the thematic analysis. The competency domain statements are: (1) basic knowledge of AI: explain what AI is and describe its health care applications; (2) social and ethical implications of AI: explain how social, economic, and political systems influence AI-based tools and how these relationships impact justice, equity, and ethics; (3) AI-enhanced clinical encounters: carry out AI-enhanced clinical encounters that integrate diverse sources of information in creating patient-centered care plans; (4) evidence-based evaluation of AI-based tools: evaluate the quality, accuracy, safety, contextual appropriateness, and biases of AI-based tools and their underlying data sets in providing care to patients and populations; (5) workflow analysis for AI-based tools: analyze and adapt to changes in teams, roles, responsibilities, and workflows resulting from implementation of AI-based tools; and (6) practice-based learning and improvement regarding AI-based tools: participate in continuing professional development and practice-based improvement activities related to use of AI tools in health care. CONCLUSIONS: The 6 clinical competencies identified can be used to guide future teaching and learning programs to maximize the potential benefits of AI-based tools and diminish potential harms.

Screening Health-Related Social Needs in Hospitals: A Systematic Review of Health Care Professional and Patient Perspectives.

Health outcomes are markedly influenced by health-related social needs (HRSN) such as food insecurity and housing instability. Under new Joint Commission requirements, hospitals have recently increased attention to HRSN to reduce health disparities. To evaluate prevailing attitudes and guide hospital efforts, the authors conducted a systematic review to describe patients' and health care providers' perceptions related to screening for and addressing patients' HRSN in US hospitals. Articles were identified through PubMed and by expert recommendations, and synthesized by relevance of findings and basic study characteristics. The review included 22 articles, which showed that most health care providers believed that unmet social needs impact health and that screening for HRSN should be a standard part of hospital care. Notable differences existed between perceived importance of HRSN and actual screening rates, however. Patients reported high receptiveness to screening in hospital encounters, but cautioned to avoid stigmatization and protect privacy when screening. Limited knowledge of resources available, lack of time, and lack of actual resources were the most frequently reported barriers to screening for HRSN. Hospital efforts to screen and address HRSN will likely be facilitated by stakeholders' positive perceptions, but common barriers to screening and referral will need to be addressed to effectively scale up efforts and impact health disparities.

Clinical use of artificial intelligence requires AI-capable organizations.

Artificial intelligence-based algorithms are being widely implemented in health care, even as evidence is emerging of bias in their design, problems with implementation, and potential harm to patients. To achieve the promise of using of AI-based tools to improve health, healthcare organizations will need to be AI-capable, with internal and external systems functioning in tandem to ensure the safe, ethical, and effective use of AI-based tools. Ideas are starting to emerge about the organizational routines, competencies, resources, and infrastructures that will be required for safe and effective deployment of AI in health care, but there has been little empirical research. Infrastructures that provide legal and regulatory guidance for managers, clinician competencies for the safe and effective use of AI-based tools, and learner-centric resources such as clear AI documentation and local health ecosystem impact reviews can help drive continuous improvement.

External Validation and Updating of a Statistical Civilian-Based Suicide Risk Model in US Naval Primary Care.

Importance: Suicide remains an ongoing concern in the US military. Statistical models have not been broadly disseminated for US Navy service members. Objective: To externally validate and update a statistical suicide risk model initially developed in a civilian setting with an emphasis on primary care. Design, Setting, and Participants: This retrospective cohort study used data collected from 2007 through 2017 among active-duty US Navy service members. The external civilian model was applied to every visit at Naval Medical Center Portsmouth (NMCP), its NMCP Naval Branch Health Clinics (NBHCs), and TRICARE Prime Clinics (TPCs) that fall within the NMCP area. The model was retrained and recalibrated using visits to NBHCs and TPCs and updated using Department of Defense (DoD)-specific billing codes and demographic characteristics, including expanded race and ethnicity categories. Domain and temporal analyses were performed with bootstrap validation. Data analysis was performed from September 2020 to December 2022. Exposure: Visit to US NMCP. Main Outcomes and Measures: Recorded suicidal behavior on the day of or within 30 days of a visit. Performance was assessed using area under the receiver operating curve (AUROC), area under the precision recall curve (AUPRC), Brier score, and Spiegelhalter z-test statistic. Results: Of the 260 583 service members, 6529 (2.5%) had a recorded suicidal behavior, 206 412 (79.2%) were male; 104 835 (40.2%) were aged 20 to 24 years; and 9458 (3.6%) were Asian, 56 715 (21.8%) were Black or African American, and 158 277 (60.7%) were White. Applying the civilian-trained model resulted in an AUROC of 0.77 (95% CI, 0.74-0.79) and an AUPRC of 0.004 (95% CI, 0.003-0.005) at NBHCs with poor calibration (Spiegelhalter P < .001). Retraining the algorithm improved AUROC to 0.92 (95% CI, 0.91-0.93) and AUPRC to 0.66 (95% CI, 0.63-0.68). Number needed to screen in the top risk tiers was 366 for the external model and 200 for the retrained model; the lower number indicates better performance. Domain validation showed AUROC of 0.90 (95% CI, 0.90-0.91) and AUPRC of 0.01 (95% CI, 0.01-0.01), and temporal validation showed AUROC of 0.75 (95% CI, 0.72-0.78) and AUPRC of 0.003 (95% CI, 0.003-0.005). Conclusions and Relevance: In this cohort study of active-duty Navy service members, a civilian suicide attempt risk model was externally validated. Retraining and updating with DoD-specific variables improved performance. Domain and temporal validation results were similar to external validation, suggesting that implementing an external model in US Navy primary care clinics may bypass the need for costly internal development and expedite the automation of suicide prevention in these clinics.

Developing Ethics and Equity Principles, Terms, and Engagement Tools to Advance Health Equity and Researcher Diversity in AI and Machine Learning: Modified Delphi Approach.

BACKGROUND: Artificial intelligence (AI) and machine learning (ML) technology design and development continues to be rapid, despite major limitations in its current form as a practice and discipline to address all sociohumanitarian issues and complexities. From these limitations emerges an imperative to strengthen AI and ML literacy in underserved communities and build a more diverse AI and ML design and development workforce engaged in health research. OBJECTIVE: AI and ML has the potential to account for and assess a variety of factors that contribute to health and disease and to improve prevention, diagnosis, and therapy. Here, we describe recent activities within the Artificial Intelligence/Machine Learning Consortium to Advance Health Equity and Researcher Diversity (AIM-AHEAD) Ethics and Equity Workgroup (EEWG) that led to the development of deliverables that will help put ethics and fairness at the forefront of AI and ML applications to build equity in biomedical research, education, and health care. METHODS: The AIM-AHEAD EEWG was created in 2021 with 3 cochairs and 51 members in year 1 and 2 cochairs and ~40 members in year 2. Members in both years included AIM-AHEAD principal investigators, coinvestigators, leadership fellows, and research fellows. The EEWG used a modified Delphi approach using polling, ranking, and other exercises to facilitate discussions around tangible steps, key terms, and definitions needed to ensure that ethics and fairness are at the forefront of AI and ML applications to build equity in biomedical research, education, and health care. RESULTS: The EEWG developed a set of ethics and equity principles, a glossary, and an interview guide. The ethics and equity principles comprise 5 core principles, each with subparts, which articulate best practices for working with stakeholders from historically and presently underrepresented communities. The glossary contains 12 terms and definitions, with particular emphasis on optimal development, refinement, and implementation of AI and ML in health equity research. To accompany the glossary, the EEWG developed a concept relationship diagram that describes the logical flow of and relationship between the definitional concepts. Lastly, the interview guide provides questions that can be used or adapted to garner stakeholder and community perspectives on the principles and glossary. CONCLUSIONS: Ongoing engagement is needed around our principles and glossary to identify and predict potential limitations in their uses in AI and ML research settings, especially for institutions with limited resources. This requires time, careful consideration, and honest discussions around what classifies an engagement incentive as meaningful to support and sustain their full engagement. By slowing down to meet historically and presently underresourced institutions and communities where they are and where they are capable of engaging and competing, there is higher potential to achieve needed diversity, ethics, and equity in AI and ML implementation in health research.

Considering Clinician Competencies for the Implementation of Artificial Intelligence-Based Tools in Health Care: Findings From a Scoping Review.

BACKGROUND: The use of artificial intelligence (AI)-based tools in the care of individual patients and patient populations is rapidly expanding. OBJECTIVE: The aim of this paper is to systematically identify research on provider competencies needed for the use of AI in clinical settings. METHODS: A scoping review was conducted to identify articles published between January 1, 2009, and May 1, 2020, from MEDLINE, CINAHL, and the Cochrane Library databases, using search queries for terms related to health care professionals (eg, medical, nursing, and pharmacy) and their professional development in all phases of clinical education, AI-based tools in all settings of clinical practice, and professional education domains of competencies and performance. Limits were provided for English language, studies on humans with abstracts, and settings in the United States. RESULTS: The searches identified 3476 records, of which 4 met the inclusion criteria. These studies described the use of AI in clinical practice and measured at least one aspect of clinician competence. While many studies measured the performance of the AI-based tool, only 4 measured clinician performance in terms of the knowledge, skills, or attitudes needed to understand and effectively use the new tools being tested. These 4 articles primarily focused on the ability of AI to enhance patient care and clinical decision-making by improving information flow and display, specifically for physicians. CONCLUSIONS: While many research studies were identified that investigate the potential effectiveness of using AI technologies in health care, very few address specific competencies that are needed by clinicians to use them effectively. This highlights a critical gap.

Trust in AI: why we should be designing for APPROPRIATE reliance.

Use of artificial intelligence in healthcare, such as machine learning-based predictive algorithms, holds promise for advancing outcomes, but few systems are used in routine clinical practice. Trust has been cited as an important challenge to meaningful use of artificial intelligence in clinical practice. Artificial intelligence systems often involve automating cognitively challenging tasks. Therefore, previous literature on trust in automation may hold important lessons for artificial intelligence applications in healthcare. In this perspective, we argue that informatics should take lessons from literature on trust in automation such that the goal should be to foster appropriate trust in artificial intelligence based on the purpose of the tool, its process for making recommendations, and its performance in the given context. We adapt a conceptual model to support this argument and present recommendations for future work.

Disappearing expertise in clinical automation: Barcode medication administration and nurse autonomy.

OBJECTIVE: Using the case of barcode medication administration (BCMA), our objective is to describe the challenges nurses face when informatics tools are not designed to accommodate the full complexity of their work. MATERIALS AND METHODS: Autonomy is associated with nurse satisfaction and quality of care. BCMA organizes patient information and verifies medication administration. However, it presents challenges to nurse autonomy. Qualitative fieldwork, including observations of everyday work and interviews, was conducted during the implementation of BCMA in a large academic medical center. Fieldnotes and interview transcripts were coded and analyzed to describe nurses' perspectives on medication safety. RESULTS: Nurses adopt orienting frames to structure work routines and require autonomy to ensure safe task completion. Nurses exerted agency by trusting their own judgment over system information when the system did not consider workload complexity. Our results indicate that the system's rigidity clashed with adaptive needs embodied by nurses' orienting frames. DISCUSSION: Despite the fact that the concept of nurse as knowledge worker is foundational to informatics, nurses may be perceived as doers, rather than knowledge workers. In practice, nurses not only make decisions, but also engage in highly complex task-related work that is not well supported by process-oriented information technology tools. CONCLUSIONS: Information technology developers and healthcare organization managers should engage and better understand nursing work in order to develop technological and social systems to support it.

Design thinking in applied informatics: what can we learn from Project HealthDesign?

OBJECTIVE: The goals of this study are to describe the value and impact of Project HealthDesign (PHD), a program of the Robert Wood Johnson Foundation that applied design thinking to personal health records, and to explore the applicability of the PHD model to another challenging translational informatics problem: the integration of AI into the healthcare system. MATERIALS AND METHODS: We assessed PHD's impact and value in 2 ways. First, we analyzed publication impact by calculating a PHD h-index and characterizing the professional domains of citing journals. Next, we surveyed and interviewed PHD grantees, expert consultants, and codirectors to assess the program's components and the potential future application of design thinking to artificial intelligence (AI) integration into healthcare. RESULTS: There was a total of 1171 unique citations to PHD-funded work (collective h-index of 25). Studies citing PHD span medical, legal, and computational journals. Participants stated that this project transformed their thinking, altered their career trajectory, and resulted in technology transfer into the commercial sector. Participants felt, in general, that the approach would be valuable in solving contemporary challenges integrating AI in healthcare including complex social questions, integrating knowledge from multiple domains, implementation, and governance. CONCLUSION: Design thinking is a systematic approach to problem-solving characterized by cooperation and collaboration. PHD generated significant impacts as measured by citations, reach, and overall effect on participants. PHD's design thinking methods are potentially useful to other work on cyber-physical systems, such as the use of AI in healthcare, to propose structural or policy-related changes that may affect adoption, value, and improvement of the care delivery system.

Enabling adoption and use of new health information technology during implementation: Roles and strategies for internal and external support personnel.

OBJECTIVE: Successful technological implementations frequently involve individuals who serve as mediators between end users, management, and technology developers. The goal for this project was to evaluate the structure and activities of such mediators in a large-scale electronic health record implementation. MATERIALS AND METHODS: Field notes from observations taken during implementation beginning in November 2017 were analyzed qualitatively using a thematic analysis framework to examine the relationship between specific types of mediators and the type and level of support to end users. RESULTS: We found that support personnel possessing both contextual knowledge of the institution's workflow and training in the new technology were the most successful in mediation of adoption and use. Those that lacked context of either technology or institutional workflow often displayed barriers in communication, trust, and active problem solving. CONCLUSIONS: These findings suggest that institutional investment in technology training and explicit programs to foster skills in mediation, including roles for professionals with career development opportunities, prior to implementation can be beneficial in easing the pain of system transition.

Before-Visit Questionnaire: A Tool to Augment Communication and Decrease Provider Documentation Burden in Pediatric Diabetes.

OBJECTIVE: To develop and evaluate an electronic tool that collects interval history and incorporates it into a provider summary note. METHODS: A parent-facing online before-visit questionnaire (BVQ) collected information from parents and caregivers of pediatric diabetes patients prior to a clinic encounter. This information was related to interval history and perceived self-management barriers. The BVQ generated a summary note that providers could paste in their own documentation. Parents also completed postvisit experience questionnaires. We assessed the BVQs perceived usefulness to parents and providers and compared provider documentation content and length pre- and post-BVQ rollout. We interviewed providers regarding their experiences with the system-generated note. RESULTS: Seventy-three parents of diabetic children were recruited and completed the BVQ. A total of 79% of parents stated that the BVQ helped with visit preparation and 80% said it improved perceived quality of visits. All 16 participating providers reviewed BVQs prior to patient encounters and 100% considered the summary beneficial. Most providers (81%) desired summaries less than 1 week old. A total of 69% of providers preferred the prose version of the summary; however, 75% also viewed the bulleted version as preferable for provider review. Analysis of provider notes revealed that BVQs increased provider documentation of patients' adherence and barriers. We observed a 50% reduction in typing by providers to document interval histories. Providers not using summaries typed an average of 137 words (standard deviation [SD]: 74) to document interval history compared with 68 words [SD 47] typed with BVQ use. DISCUSSION: Providers and parents of children with diabetes appreciated the use of previsit, parent-completed BVQs that automatically produced provider documentation. Despite the BVQ redistributing work from providers to parents, its use was acceptable to both groups. CONCLUSION: Parent-completed questionnaires on the patient's behalf that generate provider documentation encourage communication between parents and providers regarding disease management and reduce provider workload.

Risky business: a scoping review for communicating results of predictive models between providers and patients.

OBJECTIVE: Given widespread excitement around predictive analytics and the proliferation of machine learning algorithms that predict outcomes, a key next step is understanding how this information is-or should be-communicated with patients. MATERIALS AND METHODS: We conducted a scoping review informed by PRISMA-ScR guidelines to identify current knowledge and gaps in this domain. RESULTS: Ten studies met inclusion criteria for full text review. The following topics were represented in the studies, some of which involved more than 1 topic: disease prevention (N = 5/10, 50%), treatment decisions (N = 5/10, 50%), medication harms reduction (N = 1/10, 10%), and presentation of cardiovascular risk information (N = 5/10, 50%). A single study included 6- and 12-month clinical outcome metrics. DISCUSSION: As predictive models are increasingly published, marketed by industry, and implemented, this paucity of relevant research poses important gaps. Published studies identified the importance of (1) identifying the most effective source of information for patient communications; (2) contextualizing risk information and associated design elements based on users' needs and problem areas; and (3) understanding potential impacts on risk factor modification and behavior change dependent on risk presentation. CONCLUSION: An opportunity remains for researchers and practitioners to share strategies for effective selection of predictive algorithms for clinical practice, approaches for educating clinicians and patients in effectively using predictive data, and new approaches for framing patient-provider communication in the era of artificial intelligence.

A Perioperative Care Display for Understanding High Acuity Patients.

BACKGROUND: The data visualization literature asserts that the details of the optimal data display must be tailored to the specific task, the background of the user, and the characteristics of the data. The general organizing principle of a concept-oriented display is known to be useful for many tasks and data types. OBJECTIVES: In this project, we used general principles of data visualization and a co-design process to produce a clinical display tailored to a specific cognitive task, chosen from the anesthesia domain, but with clear generalizability to other clinical tasks. To support the work of the anesthesia-in-charge (AIC) our task was, for a given day, to depict the acuity level and complexity of each patient in the collection of those that will be operated on the following day. The AIC uses this information to optimally allocate anesthesia staff and providers across operating rooms. METHODS: We used a co-design process to collaborate with participants who work in the AIC role. We conducted two in-depth interviews with AICs and engaged them in subsequent input on iterative design solutions. RESULTS: Through a co-design process, we found (1) the need to carefully match the level of detail in the display to the level required by the clinical task, (2) the impedance caused by irrelevant information on the screen such as icons relevant only to other tasks, and (3) the desire for a specific but optional trajectory of increasingly detailed textual summaries. CONCLUSION: This study reports a real-world clinical informatics development project that engaged users as co-designers. Our process led to the user-preferred design of a single binary flag to identify the subset of patients needing further investigation, and then a trajectory of increasingly detailed, text-based abstractions for each patient that can be displayed when more information is needed.

Mining tasks and task characteristics from electronic health record audit logs with unsupervised machine learning.

OBJECTIVE: The characteristics of clinician activities while interacting with electronic health record (EHR) systems can influence the time spent in EHRs and workload. This study aims to characterize EHR activities as tasks and define novel, data-driven metrics. MATERIALS AND METHODS: We leveraged unsupervised learning approaches to learn tasks from sequences of events in EHR audit logs. We developed metrics characterizing the prevalence of unique events and event repetition and applied them to categorize tasks into 4 complexity profiles. Between these profiles, Mann-Whitney U tests were applied to measure the differences in performance time, event type, and clinician prevalence, or the number of unique clinicians who were observed performing these tasks. In addition, we apply process mining frameworks paired with clinical annotations to support the validity of a sample of our identified tasks. We apply our approaches to learn tasks performed by nurses in the Vanderbilt University Medical Center neonatal intensive care unit. RESULTS: We examined EHR audit logs generated by 33 neonatal intensive care unit nurses resulting in 57 234 sessions and 81 tasks. Our results indicated significant differences in performance time for each observed task complexity profile. There were no significant differences in clinician prevalence or in the frequency of viewing and modifying event types between tasks of different complexities. We presented a sample of expert-reviewed, annotated task workflows supporting the interpretation of their clinical meaningfulness. CONCLUSIONS: The use of the audit log provides an opportunity to assist hospitals in further investigating clinician activities to optimize EHR workflows.