Translating ethical and quality principles for the effective, safe and fair development, deployment and use of artificial intelligence technologies in healthcare.

OBJECTIVE: The complexity and rapid pace of development of algorithmic technologies pose challenges for their regulation and oversight in healthcare settings. We sought to improve our institution's approach to evaluation and governance of algorithmic technologies used in clinical care and operations by creating an Implementation Guide that standardizes evaluation criteria so that local oversight is performed in an objective fashion. MATERIALS AND METHODS: Building on a framework that applies key ethical and quality principles (clinical value and safety, fairness and equity, usability and adoption, transparency and accountability, and regulatory compliance), we created concrete guidelines for evaluating algorithmic technologies at our institution. RESULTS: An Implementation Guide articulates evaluation criteria used during review of algorithmic technologies and details what evidence supports the implementation of ethical and quality principles for trustworthy health AI. Application of the processes described in the Implementation Guide can lead to algorithms that are safer as well as more effective, fair, and equitable upon implementation, as illustrated through 4 examples of technologies at different phases of the algorithmic lifecycle that underwent evaluation at our academic medical center. DISCUSSION: By providing clear descriptions/definitions of evaluation criteria and embedding them within standardized processes, we streamlined oversight processes and educated communities using and developing algorithmic technologies within our institution. CONCLUSIONS: We developed a scalable, adaptable framework for translating principles into evaluation criteria and specific requirements that support trustworthy implementation of algorithmic technologies in patient care and healthcare operations.

Incorporating Telemedicine in Rheumatology Fellowship Training Programs: Needs Assessment, Curricular Intervention, and Evaluation.

OBJECTIVE: To increase the confidence of rheumatology fellows in training (FITs) in delivering virtual care (VC) and prepare them for independent practice, we developed educational materials addressing gaps in their skills. METHODS: We identified gaps in telemedicine skills based on FIT performance in a virtual rheumatology objective structured clinical examination (vROSCE) station on VC delivery using video teleconference technology and survey (survey 1) responses. We created educational materials including videos of "mediocre" and "excellent" VC examples, discussion/reflection questions, and a document summarizing key practices. We measured change in the confidence levels of FITs for delivering VC with a post-intervention survey (survey 2). RESULTS: Thirty-seven FITs (19 first-year, 18 second- plus third-year fellows) from 7 rheumatology fellowship training programs participated in a vROSCE and demonstrated gaps in skills mapping to several Rheumatology Telehealth Competency domains. Confidence levels of FITs improved significantly from survey 1 to survey 2 for 22 of 34 (65%) questions. All participating FITs found the educational materials helpful for learning and reflecting on their own VC practice; 18 FITs (64%) qualified usefulness as "moderately" or "a lot." Through surveying, 17 FITs (61%) reported implementing skills from the instructional videos into VC visits. CONCLUSION: Continually assessing our learners' needs and creating educational materials addressing gaps in training are requisite. Using a vROSCE station, needs assessments, and targeted learning with videos and discussion-guidance materials enhanced the confidence level of FITs in VC delivery. It is imperative to incorporate VC delivery into fellowship training program curricula to ensure breadth in skills, attitudes, and knowledge of new entrants into the rheumatology workforce.

What do internal medicine residents know about rheumatology? A needs assessment for curriculum design.

We sought to determine if analyzing internal medicine (IM) resident performance on the in-training exam (ITE) might reveal curricular needs in rheumatology education beyond those intuitive to learners and educators from their perceptions and experience. We analyzed ITE scores of post-graduate year (PGY) 2 IM residents at our institution from 2010 to 2017. We sorted rheumatology-related educational objectives on the ITEs into 10 disease categories and calculated average scores. We then surveyed our IM residents' self-reported confidence and rheumatology educators' perceptions of IM resident proficiency in the 10 categories on a 10-point Likert scale. Finally, we tallied diagnoses in each category encountered by IM residents in rheumatology clinic. Pearson's correlation coefficients were calculated for the relationship between these data. The ITEs exposed 402 residents to 63 rheumatology-related educational objectives in the 10 categories; 24 objectives were categorized as "other." Survey respondents included 38 residents and 22 educators. There was no correlation between ITE scores in the 10 categories and resident confidence (r = - 0.226, p = 0.530), educator perception (r = - 0.274, p = 0.445), or diagnoses encountered (r = - 0.310, p = 0.383). There was a strong positive correlation between resident confidence and educator perception (r = 0.934, p < 0.001). ITE performance was low in crystalline arthritis and osteoarthritis despite high resident confidence, educator perception, and number of diagnoses seen. Our analysis of IM resident ITE performance identified curricular gaps not apparent in surveys of learner and educator perceptions or an assessment of learner experience. Key Points • In this study, we demonstrate how a systematic analysis of internal medicine resident performance on the in-training exam can identify important curricular gaps in rheumatology education that are not apparent in assessments of learners and educator perceptions or learner experience in rheumatology clinic. • In-training exam performance was low in crystalline arthritis and osteoarthritis despite high resident confidence, educator perception of proficiency, and number of diagnoses seen in these categories. • When rheumatology curricula for internal medicine residents are developed solely on the basis of learner and educator perceptions, common disorders traditionally considered "easy" may be overlooked.

Addition of price transparency to an education and feedback intervention reduces utilization of inpatient echocardiography by resident physicians.

Previous studies have demonstrated the impact of appropriate use criteria (AUC) education and feedback interventions in reducing unnecessary ordering of transthoracic echocardiography (TTE) by trainees. To our knowledge, no study has evaluated the impact of the addition of price transparency to this education and feedback model on TTE utilization by resident physicians. We performed an education and feedback quality improvement initiative combining charge transparency data with information on AUC. We hypothesized that the initiative would reduce the number of complete TTE ordered and increase the number of limited TTE ordered, anticipating there would be substitution of limited for complete studies. Residents rotating on inpatient teaching cardiology ward teams received education on AUC for TTE, indications for limited TTE, and hospital charges for TTE. Feedback was provided on the quantity and charges for complete and limited TTE ordered by each team. We analyzed the effects of the intervention using a linear mixed effects regression model to adjust for potential confounders. The post-intervention weeks showed a reduction of 4.6 complete TTE orders per 100 patients from previous weekly baseline of 31.3 complete TTE orders per 100 patients (p value = 0.012). Charges for complete TTE decreased $122 from baseline of $980 per patient (p value = 0.040) on a per-week basis. Secondarily, there was no statistically significant change in limited TTE ordering during the intervention period. This initiative shows the feasibility of a house staff-driven charge transparency and education/feedback initiative that decreased medical residents' ordering of inpatient TTE.