Development and preliminary testing of Health Equity Across the AI Lifecycle (HEAAL): A framework for healthcare delivery organizations to mitigate the risk of AI solutions worsening health inequities.

The use of data-driven technologies such as Artificial Intelligence (AI) and Machine Learning (ML) is growing in healthcare. However, the proliferation of healthcare AI tools has outpaced regulatory frameworks, accountability measures, and governance standards to ensure safe, effective, and equitable use. To address these gaps and tackle a common challenge faced by healthcare delivery organizations, a case-based workshop was organized, and a framework was developed to evaluate the potential impact of implementing an AI solution on health equity. The Health Equity Across the AI Lifecycle (HEAAL) is co-designed with extensive engagement of clinical, operational, technical, and regulatory leaders across healthcare delivery organizations and ecosystem partners in the US. It assesses 5 equity assessment domains-accountability, fairness, fitness for purpose, reliability and validity, and transparency-across the span of eight key decision points in the AI adoption lifecycle. It is a process-oriented framework containing 37 step-by-step procedures for evaluating an existing AI solution and 34 procedures for evaluating a new AI solution in total. Within each procedure, it identifies relevant key stakeholders and data sources used to conduct the procedure. HEAAL guides how healthcare delivery organizations may mitigate the potential risk of AI solutions worsening health inequities. It also informs how much resources and support are required to assess the potential impact of AI solutions on health inequities.

Development and Integration of Machine Learning Algorithm to Identify Peripheral Arterial Disease: Multistakeholder Qualitative Study.

BACKGROUND: Machine learning (ML)-driven clinical decision support (CDS) continues to draw wide interest and investment as a means of improving care quality and value, despite mixed real-world implementation outcomes. OBJECTIVE: This study aimed to explore the factors that influence the integration of a peripheral arterial disease (PAD) identification algorithm to implement timely guideline-based care. METHODS: A total of 12 semistructured interviews were conducted with individuals from 3 stakeholder groups during the first 4 weeks of integration of an ML-driven CDS. The stakeholder groups included technical, administrative, and clinical members of the team interacting with the ML-driven CDS. The ML-driven CDS identified patients with a high probability of having PAD, and these patients were then reviewed by an interdisciplinary team that developed a recommended action plan and sent recommendations to the patient's primary care provider. Pseudonymized transcripts were coded, and thematic analysis was conducted by a multidisciplinary research team. RESULTS: Three themes were identified: positive factors translating in silico performance to real-world efficacy, organizational factors and data structure factors affecting clinical impact, and potential challenges to advancing equity. Our study found that the factors that led to successful translation of in silico algorithm performance to real-world impact were largely nontechnical, given adequate efficacy in retrospective validation, including strong clinical leadership, trustworthy workflows, early consideration of end-user needs, and ensuring that the CDS addresses an actionable problem. Negative factors of integration included failure to incorporate the on-the-ground context, the lack of feedback loops, and data silos limiting the ML-driven CDS. The success criteria for each stakeholder group were also characterized to better understand how teams work together to integrate ML-driven CDS and to understand the varying needs across stakeholder groups. CONCLUSIONS: Longitudinal and multidisciplinary stakeholder engagement in the development and integration of ML-driven CDS underpins its effective translation into real-world care. Although previous studies have focused on the technical elements of ML-driven CDS, our study demonstrates the importance of including administrative and operational leaders as well as an early consideration of clinicians' needs. Seeing how different stakeholder groups have this more holistic perspective also permits more effective detection of context-driven health care inequities, which are uncovered or exacerbated via ML-driven CDS integration through structural and organizational challenges. Many of the solutions to these inequities lie outside the scope of ML and require coordinated systematic solutions for mitigation to help reduce disparities in the care of patients with PAD.

Pragmatic AI-augmentation in mental healthcare: Key technologies, potential benefits, and real-world challenges and solutions for frontline clinicians.

The integration of artificial intelligence (AI) technologies into mental health holds the promise of increasing patient access, engagement, and quality of care, and of improving clinician quality of work life. However, to date, studies of AI technologies in mental health have focused primarily on challenges that policymakers, clinical leaders, and data and computer scientists face, rather than on challenges that frontline mental health clinicians are likely to face as they attempt to integrate AI-based technologies into their everyday clinical practice. In this Perspective, we describe a framework for "pragmatic AI-augmentation" that addresses these issues by describing three categories of emerging AI-based mental health technologies which frontline clinicians can leverage in their clinical practice-automation, engagement, and clinical decision support technologies. We elaborate the potential benefits offered by these technologies, the likely day-to-day challenges they may raise for mental health clinicians, and some solutions that clinical leaders and technology developers can use to address these challenges, based on emerging experience with the integration of AI technologies into clinician daily practice in other healthcare disciplines.

Enhancing the value to users of machine learning-based clinical decision support tools: A framework for iterative, collaborative development and implementation.

BACKGROUND: Health care organizations are integrating a variety of machine learning (ML)-based clinical decision support (CDS) tools into their operations, but practitioners lack clear guidance regarding how to implement these tools so that they assist end users in their work. PURPOSE: We designed this study to identify how health care organizations can facilitate collaborative development of ML-based CDS tools to enhance their value for health care delivery in real-world settings. METHODOLOGY/APPROACH: We utilized qualitative methods, including 37 interviews in a large, multispecialty health system that developed and implemented two operational ML-based CDS tools in two of its hospital sites. We performed thematic analyses to inform presentation of an explanatory framework and recommendations. RESULTS: We found that ML-based CDS tool development and implementation into clinical workflows proceeded in four phases: iterative solution coidentification, iterative coengagement, iterative coapplication, and iterative corefinement. Each phase is characterized by a collaborative back-and-forth process between the technology's developers and users, through which both users' activities and the technology itself are transformed. CONCLUSION: Health care organizations that anticipate iterative collaboration to be an integral aspect of their ML-based CDS tools' development and implementation process may have more success in deploying ML-based CDS tools that assist end users in their work than organizations that expect a traditional technology innovation process. PRACTICE IMPLICATIONS: Managers developing and implementing ML-based CDS tools should frame the work as a collaborative learning opportunity for both users and the technology itself and should solicit constructive feedback from users on potential changes to the technology, in addition to potential changes to user workflows, in an ongoing, iterative manner.

An intraorganizational model for developing and spreading quality improvement innovations.

BACKGROUND: Recent policy reforms encourage quality improvement (QI) innovations in primary care, but practitioners lack clear guidance regarding spread inside organizations. PURPOSE: We designed this study to identify how large organizations can facilitate intraorganizational spread of QI innovations. METHODOLOGY/APPROACH: We conducted ethnographic observation and interviews in a large, multispecialty, community-based medical group that implemented three QI innovations across 10 primary care sites using a new method for intraorganizational process development and spread. We compared quantitative outcomes achieved through the group's traditional versus new method, created a process model describing the steps in the new method, and identified barriers and facilitators at each step. FINDINGS: The medical group achieved substantial improvement using its new method of intraorganizational process development and spread of QI innovations: standard work for rooming and depression screening, vaccine error rates and order compliance, and Pap smear error rates. Our model details nine critical steps for successful intraorganizational process development (set priorities, assess the current state, develop the new process, and measure and refine) and spread (develop support, disseminate information, facilitate peer-to-peer training, reinforce, and learn and adapt). Our results highlight the importance of utilizing preexisting organizational structures such as established communication channels, standardized roles, common workflows, formal authority, and performance measurement and feedback systems when developing and spreading QI processes inside an organization. In particular, we detail how formal process advocate positions in each site for each role can facilitate the spread of new processes. PRACTICE IMPLICATIONS: Successful intraorganizational spread is possible and sustainable. Developing and spreading new QI processes across sites inside an organization requires creating a shared understanding of the necessary process steps, considering the barriers that may arise at each step, and leveraging preexisting organizational structures to facilitate intraorganizational process development and spread.

The Radical Flank Effect and Cross-occupational Collaboration for Technology Development during a Power Shift.

This 12-month ethnographic study of an early entrant into the U.S. car-sharing industry demonstrates that when an organization shifts its focus from developing radical new technology to incrementally improving this technology, the shift may spark an internal power struggle between the dominant engineering group and a challenger occupational group such as the marketing group. Analyzing 42 projects in two time periods that required collaboration between engineering and marketing during such a shift, we show how cross-occupational collaboration under these conditions can be facilitated by a radical flank threat, through which the bargaining power of moderates is strengthened by the presence of a more-radical group. In the face of a strong threat by radical members of a challenger occupational group, moderate members of the dominant engineering group may change their perceptions of their power to resist challengers' demands and begin to distinguish between the goals of radical versus more-moderate challengers. To maintain as much power as possible and prevent the more-dramatic change in engineering occupational goals demanded by radical challengers, moderate engineers may build a coalition with moderate challengers and collaborate for incremental technology development.

Career satisfaction of women in surgery: perceptions, factors, and strategies.

BACKGROUND: With the current and projected shortages of general surgeons, more attention is being paid to the increasing pool of women physicians. This study seeks to understand the variables leading to career satisfaction for women surgeons to better recruit, retain, and support them. STUDY DESIGN: Eighteen semi-structured interviews of 12 female and 6 male surgeons 2 to 12 years into practice were qualitatively analyzed and converted to coded, categorized data. Significance was derived by Fisher's exact test. Participants were recruited by snowball sampling. RESULTS: Our sample represents a highly satisfied group of female and male surgeons. Although both women and men describe with equal frequency having made career tradeoffs for personal and family time, and vice versa, women far more frequently than men cite reasons related to their personal time, predictable time, and family relationships as why they are currently satisfied with their career (34.1% versus 8.7%; p < 0.05). Both cite being satisfied by career content equally. When describing strategies used in developing a successful surgical career, women most frequently cite social networks as a key to success (88% versus 12% by men; p < 0.05), and men more frequently cite reasons related to training (29% versus 0% by women; p < 0.05) and compensation (24% versus 0% by women; p < 0.05). CONCLUSIONS: Although both men and women make tradeoffs of career for family and family for career, women's perception of satisfaction comes from viewing their surgical career within the broader context of their lives. Women might be attracted to a career that acknowledges and values the whole person beyond the surgeon, and could benefit from work infrastructures that enhance networking.

Operating room: relational spaces and microinstitutional change in surgery.

One of the great paradoxes of institutional change is that even when top managers in organizations provide support for change in response to new regulation, the employees whom new programs are designed to benefit often do not use them. This 15-month ethnographic study of two hospitals responding to new regulation demonstrates that using these programs may require subordinate employees to challenge middle managers with opposing interests. The article argues that relational spaces--areas of isolation, interaction, and inclusion that allow middle-manager reformers and subordinate employees to develop a cross-position collective for change--are critical to the change process. These findings have implications for research on institutional change and social movements.

The impact of the 80-hour resident workweek on surgical residents and attending surgeons.

OBJECTIVE: To assess the impact of the 80-hour resident workweek restrictions on surgical residents and attending surgeons. SUMMARY BACKGROUND DATA: The ACGME mandated resident duty hour restrictions have required a major workforce restructuring. The impact of these changes needs to be critically evaluated for both the resident and attending surgeons, specifically with regards to the impact on motivation, job satisfaction, the quality of surgeon training, the quality of the surgeon's life, and the quality of patient care. METHODS: Four prospective studies were performed at a single academic surgical program with data collected both before the necessary workforce restructuring and 1 year after, including: 1) time cards to assess changes in components of daily activity; 2) Web-based surveys using validated instruments to assess burnout and motivation to work; 3) structured, taped, one-on-one interviews with an external PhD investigator; and 4) statistical analyses of objective, quantitative data. RESULTS: After the work-hour changes, surgical residents have decreased "burnout" scores, with significantly less "emotional exhaustion" (Maslach Burnout Inventory: 29.1 "high" vs. 23.1 "medium," P = 0.02). Residents have better quality of life both in and out of the hospital. They felt they got more sleep, have a lighter workload, and have increased motivation to work (Herzberg Motivation Dimensions). We found no measurable, statistically significant difference in the quality of patient care (NSQIP data). Resident training and education objectively were not statistically diminished (ACGME case logs, ABSITE scores). Attending surgeons perceived that their quality of their life inside and outside of the hospital was "somewhat worse" because of the work-hour changes, as they had anticipated. Many concerns were identified with regards to the professional development of future surgeons, including a change toward a shift-worker mentality that is not patient-focused, less continuity of care with a loss of critical information with each handoff, and a decrease in the patient/doctor relationship. CONCLUSION: Although the mandated restriction of resident duty hours has had no measurable impact on the quality of patient care and has led to improvements for the current quality of life of residents, there are many concerns with regards to the training of professional, responsible surgeons for the future.

Resistance to change in surgical residency: an ethnographic study of work hours reform.

BACKGROUND: Although the practical challenges to work hour restrictions have been the focus of much discussion, cultural resistance to such change has received less attention. Surgical residency has its own unique social structure, and we hypothesized that challenges to this would provide impediments to successful implementation of duty hours reform. STUDY DESIGN: We used ethnographic research methods to study the efforts at work hour restriction over a 15-month period before the introduction of the Accreditation Council for Graduate Medical Education regulations. These methods, validated for studying institutional change, build on intense periods of observation. Records of observations are then analyzed and coded to uncover cultural and political challenges. The frequency of successful hand-offs in sign-out situations between day and night float residents was measured as an objective index of success. RESULTS: Practical issues were addressed initially by scheduling adjustments including creating a night float system. The hand-offs that this system required, however, were successful only 14% of the time. Subsequent steps to address the challenge to resident identity by top-down support of a new definition of professionalism increased the number of successful hand-offs to 39%. Finally, a reduction in a noted hierarchy violation led to successful hand-offs 79% of the time. CONCLUSIONS: These results demonstrate that practical solutions alone may not be a sufficient basis for change in surgical residency. While we face other challenges to the traditional surgical culture, attention to social and political issues may enhance the success of our efforts.

Effect of work-hour reforms on operative case volume of surgical residents.

PURPOSE: There has been much concern regarding the impact of work-hour reforms on the operative case volume of surgical residents. Operative case volume by PGY year and clinical rotation were examined to determine if changes in work hours affected residents' operative case volume. METHODS: A careful but aggressive plan of work-hour reduction was devised for the residency of the authors' institution with the goal to decrease work hours while maintaining optimal patient care and resident education, including operative case volume. Changes made included hiring physician extenders (PEs), decreasing call schedules to every fourth night (with the next day free from clinical activities-Q4) or call from home (HC), and night float rotation coverage for services (NF). Case volume before (academic year 2002) and after (academic year 2003) changes were compared by PGY year, for all residents and for specific rotations-private general surgery, which had changes of PE, HC, and NF for PGY5; PE, Q4 and NF for PGY1 and 10% exemption for work hours; Churchill service (a resident-run ward teaching service), which had changes of PE and Q4 for PGY5 and PGY1 and 10% exemption for work hours; and vascular surgery, which had HC and NF for PGY5. Total case volume on these services was likewise compared. Statistical analysis was by student t test. Operative case volume was measured with data from the resident-entered information on the ACGME Surgical Operative Log (SOL). Case volume for PGY4 residents could not be compared over this time period because of lack of access to archived data by PGY year for graduated residents through the ACGME SOL. Work hours before and after rotation changes were measured with an intranet-based monitoring system. This article is a retrospective review of the affects of these work-hour changes on operative case volume. RESULTS: Total case volume for the general surgical services (both private and Churchill) was unchanged over this period (5905 in 02, 5930 in 03), and likewise for the vascular service (1101 vs 1196). Overall, there was no change in mean operative volume per year for surgical residents in this program (231 cases in 2002, 246 cases in 2003; p = 0.61). For PGY5 residents, the case volume increased; 339 cases 02, 390 in 03, and p = 0.05. Mean case volume for PGY5 residents increased on the private general surgery service (136 in 02, 160 in 03, p = 0.03), but it remained stable on the Churchill service (137 in 02, 158 in 03, p = 0.39) and vascular service (65 in 02, 73 in 03, p = 0.42). For PGY3 residents, case volume remained stable (171 in 02, 187 in 03, p = 0.62), as it did for PGY2 and PGY1 residents (PGY2: 148 in 02, 121 in 03, p = 0.12; PGY1: 265 in 02, 246 in 03, p = 0.23). However, operative case volume for PGY1 residents did decrease on the private general surgery service (mean 52 cases per month 02, 43 cases per month 03, p = 0.07), while remaining stable on the Churchill service (mean 23 cases per month 02, 25 cases per month 03, p = 0.66). Average hours worked per week decreased significantly over the time period. Before work-hour reforms, residents' average work hours were as follows: PGY1 105, PGY2 97, PGY3 78.7, PGY4 111, and PGY5 92. After the changes, average work hours were PGY1 81.5, PGY2 77.7, PGY3 78.7, PGY4 75.5, and PGY5 75.9. CONCLUSIONS: Work-hour limitation can be devised to maximize resident education, optimize patient care, and maintain resident operative volume. Although some changes (HC, PE, NF) seemed to increase the operative case volume for PGY5 residents, others had no effect (Q4, HC). There does not seem to be a clear relationship between types of changes and case volume. At the PGY1 level, Q4 and PE changes decreased operative experience on 1 rotation but not on another, although the difference in this decrease seems clinically insignificant. Individualization of changes to meet the needs of specific rotations seems more important than specific changes in coverage pattern. Perhaps the most important finding is that changes can be made to bring work hours into compliance without materially effecting operative case volume.