Twelve tips for creating a longitudinal quality improvement and safety education for early health professions students.

Despite the numerous calls for integrating quality improvement and patient safety (QIPS) curricula into health professions education, there are limited examples of effective implementation for early learners. Typically, pre-clinical QIPS experiences involve lectures or lessons that are disconnected from the practice of medicine. Consequently, students often prioritize other content they consider more important. As a result, they may enter clinical settings without essential QIPS skills and struggle to incorporate these concepts into their early professional identity formation. In this paper, we present twelve tips aimed at assisting educators in developing QIPS education early in the curricula of health professions students. These tips address various key issues, including aligning incentives, providing longitudinal experiences, incorporating real-world care outcomes, optimizing learning environments, communicating successes, and continually enhancing education and care delivery processes.

Secure Messaging Use Among Patients with Depression: An Analysis Using Real-World Data.

Background: Although depression is one of the most common mental health disorders outpacing other diseases and conditions, poor access to care and limited resources leave many untreated. Secure messaging (SM) offers patients an online means to bridge this gap by communicating nonurgent medical questions. We focused on self-care health management behaviors and delved into SM initiation as the initial act of engagement and SM exchanges as continuous engagement patterns. This study examined whether those with depression might be using SM more than those without depression. Methods: Patient portal data were obtained from a large academic medical center's electronic health records spanning 5 years, from January 2018 to December 2022. We organized and analyzed SM initiations and exchanges using the linear mixed-effects modeling technique. Results: Our predictors correlated with SM initiations, accounting for 25.1% of variance explained. In parallel, 24.9% of SM exchanges were attributable to these predictors. Overall, our predictors demonstrate stronger associations with SM exchanges. Discussion: We examined patients with and without depression across 2,629 zip codes over five years. Our findings reveal that the predictors affecting SM initiations and exchanges are multifaceted, with certain predictors enhancing its utilization and others impeding it. Conclusions: SM telehealth service provided support to patients with mental health needs to a greater extent than those without. By increasing access, fostering better communication, and efficiently allocating resources, telehealth services not only encourage patients to begin using SM but also promote sustained interaction through ongoing SM exchanges.

Developing the Expected Entrustment Score: Accounting for Variation in Resident Assessment.

BACKGROUND: Clinical competency committees (CCCs) and residency program leaders may find it difficult to interpret workplace-based assessment (WBA) ratings knowing that contextual factors and bias play a large role. OBJECTIVE: We describe the development of an expected entrustment score for resident performance within the context of our well-developed Observable Practice Activity (OPA) WBA system. DESIGN: Observational study PARTICIPANTS: Internal medicine residents MAIN MEASURE: Entrustment KEY RESULTS: Each individual resident had observed entrustment scores with a unique relationship to the expected entrustment scores. Many residents' observed scores oscillated closely around the expected scores. However, distinct performance patterns did emerge. CONCLUSIONS: We used regression modeling and leveraged large numbers of historical WBA data points to produce an expected entrustment score that served as a guidepost for performance interpretation.

See One, Do One, Forget One: Early Skill Decay After Paracentesis Training.

INTRODUCTION: Internal medicine residents perform paracentesis, but programs lack standard methods for assessing competence or maintenance of competence and instead rely on number of procedures completed. This study describes differences in resident competence in paracentesis over time. METHODS: From 2016 to 2017, internal medicine residents (n = 118) underwent paracentesis simulation training. Competence was assessed using the Paracentesis Competency Assessment Tool (PCAT), which combines a checklist, global scale, and entrustment score. The PCAT also delineates two categorical cut-point scores: the Minimum Passing Standard (MPS) and the Unsupervised Practice Standard (UPS). Residents were randomized to return to the simulation lab at 3 and 6 months (group A, n = 60) or only 6 months (group B, n = 58). At each session, faculty raters assessed resident performance. Data were analyzed to compare resident performance at each session compared with initial training scores, and performance between groups at 6 months. RESULTS: After initial training, all residents met the MPS. The number achieving UPS did not differ between groups: group A = 24 (40%), group B = 20 (34.5%), p = 0.67. When group A was retested at 3 months, performance on each PCAT component significantly declined, as did the proportion of residents meeting the MPS and UPS. At the 6-month test, residents in group A performed significantly better than residents in group B, with 52 (89.7%) and 20 (34.5%) achieving the MPS and UPS, respectively, in group A compared with 25 (46.3%) and 2 (3.70%) in group B (p < .001 for both comparison). DISCUSSION: Skill in paracentesis declines as early as 3 months after training. However, retraining may help interrupt skill decay. Only a small proportion of residents met the UPS 6 months after training. This suggests using the PCAT to objectively measure competence would reclassify residents from being permitted to perform paracentesis independently to needing further supervision.

Development of Resident-Sensitive Quality Measures for Inpatient General Internal Medicine.

BACKGROUND: Graduate medical education (GME) training has long-lasting effects on patient care quality. Despite this, few GME programs use clinical care measures as part of resident assessment. Furthermore, there is no gold standard to identify clinical care measures that are reflective of resident care. Resident-sensitive quality measures (RSQMs), defined as "measures that are meaningful in patient care and are most likely attributable to resident care," have been developed using consensus methodology and piloted in pediatric emergency medicine. However, this approach has not been tested in internal medicine (IM). OBJECTIVE: To develop RSQMs for a general internal medicine (GIM) inpatient residency rotation using previously described consensus methods. DESIGN: The authors used two consensus methods, nominal group technique (NGT) and a subsequent Delphi method, to generate RSQMs for a GIM inpatient rotation. RSQMs were generated for specific clinical conditions found on a GIM inpatient rotation, as well as for general care on a GIM ward. PARTICIPANTS: NGT participants included nine IM and medicine-pediatrics (MP) residents and six IM and MP faculty members. The Delphi group included seven IM and MP residents and seven IM and MP faculty members. MAIN MEASURES: The number and description of RSQMs generated during this process. KEY RESULTS: Consensus methods resulted in 89 RSQMs with the following breakdown by condition: GIM general care-21, diabetes mellitus-16, hyperkalemia-14, COPD-13, hypertension-11, pneumonia-10, and hypokalemia-4. All RSQMs were process measures, with 48% relating to documentation and 51% relating to orders. Fifty-eight percent of RSQMs were related to the primary admitting diagnosis, while 42% could also be related to chronic comorbidities that require management during an admission. CONCLUSIONS: Consensus methods resulted in 89 RSQMs for a GIM inpatient service. While all RSQMs were process measures, they may still hold value in learner assessment, formative feedback, and program evaluation.

Becoming a deliberately developmental organization: Using competency based assessment data for organizational development.

Medical education is situated within health care and educational organizations that frequently lag in their use of data to learn, develop, and improve performance. How might we leverage competency-based medical education (CBME) assessment data at the individual, program, and system levels, with the goal of redefining CBME from an initiative that supports the development of physicians to one that also fosters the development of the faculty, administrators, and programs within our organizations? In this paper we review the Deliberately Developmental Organization (DDO) framework proposed by Robert Kegan and Lisa Lahey, a theoretical framework that explains how organizations can foster the development of their people. We then describe the DDO's conceptual alignment with CBME and outline how CBME assessment data could be used to spur the transformation of health care and educational organizations into digitally integrated DDOs. A DDO-oriented use of CBME assessment data will require intentional investment into both the digitalization of assessment data and the development of the people within our organizations. By reframing CBME in this light, we hope that educational and health care leaders will see their investments in CBME as an opportunity to spur the evolution of a developmental culture.

On the validity of summative entrustment decisions.

Health care revolves around trust. Patients are often in a position that gives them no other choice than to trust the people taking care of them. Educational programs thus have the responsibility to develop physicians who can be trusted to deliver safe and effective care, ultimately making a final decision to entrust trainees to graduate to unsupervised practice. Such entrustment decisions deserve to be scrutinized for their validity. This end-of-training entrustment decision is arguably the most important one, although earlier entrustment decisions, for smaller units of professional practice, should also be scrutinized for their validity. Validity of entrustment decisions implies a defensible argument that can be analyzed in components that together support the decision. According to Kane, building a validity argument is a process designed to support inferences of scoring, generalization across observations, extrapolation to new instances, and implications of the decision. A lack of validity can be caused by inadequate evidence in terms of, according to Messick, content, response process, internal structure (coherence) and relationship to other variables, and in misinterpreted consequences. These two leading frameworks (Kane and Messick) in educational and psychological testing can be well applied to summative entrustment decision-making. The authors elaborate the types of questions that need to be answered to arrive at defensible, well-argued summative decisions regarding performance to provide a grounding for high-quality safe patient care.

Competency-based time-variable training internationally: Ensuring practical next steps in the wake of the COVID-19 pandemic.

Competency-based medical education has been advocated as the future of medical education for nearly a half-century. Inherent to this is the promise that advancement and transitions in training would be defined by readiness to practice rather than by time. Of the logistical problems facing competency-based, time-variable (CBTV) training, enacting time variability may be the largest hurdle to clear. Although it is true that an 'all or nothing' approach to CBTV training would require massive overhauls of both medical education and health care systems, the authors propose that training institutions should gradually evolve within their current environments to incrementally move toward the best version of CBTV training for learners, supervisors, and patients. In support of this evolution, the authors seek to demonstrate the feasibility of advancing toward the goal of realistic CBTV training by detailing examples of successful CBTV training and describing key features of initial steps toward CBTV training implementation.

Growth mindset in competency-based medical education.

The ongoing adoption of competency-based medical education (CBME) across health professions training draws focus to learner-centred educational design and the importance of fostering a growth mindset in learners, teachers, and educational programs. An emerging body of literature addresses the instructional practices and features of learning environments that foster the skills and strategies necessary for trainees to be partners in their own learning and progression to competence and to develop skills for lifelong learning. Aligned with this emerging area is an interest in Dweck's self theory and the concept of the growth mindset. The growth mindset is an implicit belief held by an individual that intelligence and abilities are changeable, rather than fixed and immutable. In this paper, we present an overview of the growth mindset and how it aligns with the goals of CBME. We describe the challenges associated with shifting away from the fixed mindset of most traditional medical education assumptions and practices and discuss potential solutions and strategies at the individual, relational, and systems levels. Finally, we present future directions for research to better understand the growth mindset in the context of CBME.

Linking Workplace-Based Assessment to ACGME Milestones: A Comparison of Mapping Strategies in Two Specialties.

Construct: The construct that is assessed is competency in Pediatrics and Internal Medicine residency training. Background: The Accreditation Council for Graduate Medical Education (ACGME) created milestones to measure learner progression toward competence over time but not as direct assessment tools. Ideal measurement of resident performance includes direct observation and assessment of patient care skills in the workplace. Residency programs have linked these concepts by mapping workplace-based assessments to the milestones of ACGME subcompetencies. Mapping is a subjective process, and little is known about specific techniques or the resulting consequences of mapping program-specific assessment data to larger frameworks of competency. Approach: In this article, the authors compare and contrast the techniques used to link workplace-based assessments called Observable Practice Activities (OPAs) to ACGME subcompetencies in two large academic residency programs from different specialties (Internal Medicine and Pediatrics). Descriptive analysis explored the similarities and differences in the assessment data generated by mapping assessment items to larger frameworks of competency. Results: Each program assessed the core competencies with similar frequencies. The largest discrepancy between the two subspecialties was the assessment of Medical Knowledge, which Internal Medicine assessed twice as often. Pediatrics also assessed the core competency Systems-based Practice almost twice as often as Internal Medicine. Both programs had several subcompetencies that were assessed more or less often than what appeared to be emphasized by the blueprint of mapping. Despite using independent mapping processes, both programs mapped each OPA to approximately three subcompetencies. Conclusions: Mapping workplace-based assessments to the ACGME subcompetencies allowed each program to see the whole of their curricula in ways that were not possible before and to identify existing curricular and assessment gaps. Although each program used similar assessment tools, the assessment data generated were different. The lessons learned in this work could inform other programs attempting to link their own workplace-based assessment elements to ACGME subcompetencies.

Twelve tips to maximize the value of a clinical competency committee in postgraduate medical education.

Medical education has shifted to a competency-based paradigm, leading to calls for improved learner assessment methods and validity evidence for how assessment data are interpreted. Clinical competency committees (CCCs) use the collective input of multiple people to improve the validity and reliability of decisions made and actions taken based on assessment data. Significant heterogeneity in CCC structure and function exists across postgraduate medical education programs and specialties, and while there is no "one-size-fits-all" approach, there are ways to maximize value for learners and programs. This paper collates available evidence and the authors' experiences to provide practical tips on CCC purpose, membership, processes, and outputs. These tips can benefit programs looking to start a CCC and those that are improving their current CCC processes.

Networking Matters: A Social Network Analysis of the Association of Program Directors of Internal Medicine.

Theory: Networking has positive effects on career development; however, personal characteristics of group members such as gender or diversity may foster or hinder member connectedness. Social network analysis explores interrelationships between people in groups by measuring the strength of connection between all possible pairs in a given network. Social network analysis has rarely been used to examine network connections among members in an academic medical society. This study seeks to ascertain the strength of connection between program directors in the Association of Program Directors in Internal Medicine (APDIM) and its Education Innovations Project subgroup and to examine possible associations between connectedness and characteristics of program directors and programs. Hypotheses: We hypothesize that connectedness will be measurable within a large academic medical society and will vary significantly for program directors with certain measurable characteristics (e.g., age, gender, rank, location, burnout levels, desire to resign). Method: APDIM program directors described levels of connectedness to one another on the 2012 APDIM survey. Using social network analysis, we ascertained program director connectedness by measuring out-degree centrality, in-degree centrality, and eigenvector centrality, all common measures of connectedness. Results: Higher centrality was associated with completion of the APDIM survey, being in a university-based program, Educational Innovations Project participation, and higher academic rank. Centrality did not vary by gender; international medical graduate status; previous chief resident status; program region; or levels of reported program director burnout, callousness, or desire to resign. Conclusions: In this social network analysis of program directors within a large academic medical society, we found that connectedness was related to higher academic rank and certain program characteristics but not to other program director characteristics like gender or international medical graduate status. Further research is needed to optimize our understanding of connection in organizations such as these and to determine which strategies promote valuable connections.

The Internal Medicine Reporting Milestones: Cross-sectional Description of Initial Implementation in U.S. Residency Programs.

BACKGROUND: High-quality assessment of resident performance is needed to guide individual residents' development and ensure their preparedness to provide patient care. To facilitate this aim, reporting milestones are now required across all internal medicine (IM) residency programs. OBJECTIVE: To describe initial milestone ratings for the population of IM residents by IM residency programs. DESIGN: Cross-sectional study. SETTING: IM residency programs. PARTICIPANTS: All IM residents whose residency program directors submitted milestone data at the end of the 2013-2014 academic year. MEASUREMENTS: Ratings addressed 6 competencies and 22 subcompetencies. A rating of "not assessable" indicated insufficient information to evaluate the given subcompetency. Descriptive statistics were calculated to describe ratings across competencies and training years. RESULTS: Data were available for all 21 774 U.S. IM residents from all 383 programs. Overall, 2889 residents (1621 in postgraduate year 1 [PGY-1], 902 in PGY-2, and 366 in PGY-3) had at least 1 subcompetency rated as not assessable. Summaries of average ratings by competency and training year showed higher ratings for PGY-3 residents in all competencies. Overall ratings for each of the 6 individual competencies showed that fewer than 1% of third-year residents were rated as "unsatisfactory" or "conditional on improvement." However, when subcompetency milestone ratings were used, 861 residents (12.8%) who successfully completed training had at least 1 competency with all corresponding subcompetencies graded below the threshold of "readiness for unsupervised practice." LIMITATION: Data were derived from a point in time in the first reporting period in which milestones were used. CONCLUSION: The initial milestone-based evaluations of IM residents nationally suggest that documenting developmental progression of competency is possible over training years. Subcompetencies may identify areas in which residents might benefit from additional feedback and experience. Future work is needed to explore how milestones are used to support residents' development and enhance residency curricula. PRIMARY FUNDING SOURCE: None.

Entrustment and mapping of observable practice activities for resident assessment.

Entrustable Professional Activities (EPAs) and the Next Accreditation System reporting milestones reduce general competencies into smaller evaluable parts. However, some EPAs and reporting milestones may be too broad to use as direct assessment tools. We describe our internal medicine residency curriculum and assessment system, which uses entrustment and mapping of observable practice activities (OPAs) for resident assessment. We created discrete OPAs for each resident rotation and learning experience. In combination, these serve as curricular foundation and tools for assessment. OPA performance is measured via a 5-point entrustment scale, and mapped to milestones and EPAs. Entrustment ratings of OPAs provide an opportunity for immediate structured feedback of specific clinical skills, and mapping OPAs to milestones and EPAs can be used for longitudinal assessment, promotion decisions, and reporting. Direct assessment and demonstration of progressive entrustment of trainee skill over time are important goals for all training programs. Systems that use OPAs mapped to milestones and EPAs provide the opportunity for achieving both, but require validation.

Secure messaging telehealth billing in the digital age: moving beyond time-based metrics.

OBJECTIVE: We proposed adopting billing models for secure messaging (SM) telehealth services that move beyond time-based metrics, focusing on the complexity and clinical expertise involved in patient care. MATERIALS AND METHODS: We trained 8 classification machine learning (ML) models using providers' electronic health record (EHR) audit log data for patient-initiated non-urgent messages. Mixed effect modeling (MEM) analyzed significance. RESULTS: Accuracy and area under the receiver operating characteristics curve scores generally exceeded 0.85, demonstrating robust performance. MEM showed that knowledge domains significantly influenced SM billing, explaining nearly 40% of the variance. DISCUSSION: This study demonstrates that ML models using EHR audit log data can improve and predict billing in SM telehealth services, supporting billing models that reflect clinical complexity and expertise rather than time-based metrics. CONCLUSION: Our research highlights the need for SM billing models beyond time-based metrics, using EHR audit log data to capture the true value of clinical work.

Using Theory-Informed Group Processes to Make TIMELESS Promotion Decisions.

ABSTRACT: Competency-based medical education (CBME) depends on effective programs of assessment to achieve the desired outcomes and goals of training. Residency programs must be able to defend clinical competency committee (CCC) group decisions about learner readiness for practice, including decisions about time-variable resident promotion and graduation. In this article, the authors describe why CCC group decision-making processes should be supported by theory and review 3 theories they used in designing their group processes: social decision scheme theory, functional theory, and wisdom of crowds. They describe how these theories were applied in a competency-based, time-variable training pilot-Transitioning in Internal Medicine Education Leveraging Entrustment Scores Synthesis (TIMELESS) at the University of Cincinnati internal medicine residency program in 2020-2022-to increase the defensibility of their CCC group decision-making. This work serves as an example of how use of theory can bolster validity arguments supporting group decisions about resident readiness for practice.