Racial, Ethnic, and Socioeconomic Differences in Primary Care No-Show Risk with Telemedicine During the COVID-19 Pandemic.

BACKGROUND: The coronavirus 2019 (COVID-19) pandemic resulted in rapid implementation of telemedicine. Little is known about the impact of telemedicine on both no-show rates and healthcare disparities on the general primary care population during the pandemic. OBJECTIVE: To compare no-show rates between telemedicine and office visits in the primary care setting, while controlling for the burden of COVID-19 cases, with focus on underserved populations. DESIGN: Retrospective cohort study. SETTING: Multi-center urban network of primary care clinics between April 2021 and December 2021. PARTICIPANTS: A total of 311,517 completed primary care physician visits across 164,647 patients. MAIN MEASURES: The primary outcome was risk ratio of no-show incidences (i.e., no-show rates) between telemedicine and office visits across demographic sub-groups including age, ethnicity, race, and payor type. RESULTS: Compared to in-office visits, the overall risk of no-showing favored telemedicine, adjusted risk ratio of 0.68 (95% CI 0.65 to 0.71), absolute risk reduction (ARR) 4.0%. This favorability was most profound in several cohorts with racial/ethnic and socioeconomic differences with risk ratios in Black/African American 0.47 (95% CI 0.41 to 0.53), ARR 9.0%; Hispanic/Latino 0.63 (95% CI 0.58 to 0.68), ARR 4.6%; Medicaid 0.58 (95% CI 0.54 to 0.62) ARR 7.3%; Self-Pay 0.64 (95% CI 0.58 to 0.70) ARR 11.3%. LIMITATION: The analysis was limited to physician-only visits in a single setting and did not examine the reasons for visits. CONCLUSION: As compared to office visits, patients using telemedicine have a lower risk of no-showing to primary care appointments. This is one step towards improved access to care.

Exploring the role of telehealth in providing equitable healthcare to the vulnerable patient population during COVID-19.

The COVID 19 pandemic resulted in widespread telehealth implementation. Existent health disparities were widened, with under-represented minorities (URM) disproportionately affected by COVID. In this study, we assess the role of telehealth in improving access to care in the URMs and vulnerable populations. We noted a statistically significant increase in the number of visits in Hispanic or Latino patients (15.2% increase, p < 0.01) and Black patients (19% increase, p < 0.01). Based on payer type, there was a statistically significant increase in the number of visits in the Medicare (10.2%, p = 0.0001) and Medicaid (16.2%, p < 0.01) groups. We also noted increased access to care with telehealth in patients who were 65 and older (10.6%, p = 0.004). This highlights the importance of telehealth in increasing access to care and promoting health equity in the URM and vulnerable patient populations.

Developing the Expected Entrustment Score: Accounting for Variation in Resident Assessment.

BACKGROUND: Clinical competency committees (CCCs) and residency program leaders may find it difficult to interpret workplace-based assessment (WBA) ratings knowing that contextual factors and bias play a large role. OBJECTIVE: We describe the development of an expected entrustment score for resident performance within the context of our well-developed Observable Practice Activity (OPA) WBA system. DESIGN: Observational study PARTICIPANTS: Internal medicine residents MAIN MEASURE: Entrustment KEY RESULTS: Each individual resident had observed entrustment scores with a unique relationship to the expected entrustment scores. Many residents' observed scores oscillated closely around the expected scores. However, distinct performance patterns did emerge. CONCLUSIONS: We used regression modeling and leveraged large numbers of historical WBA data points to produce an expected entrustment score that served as a guidepost for performance interpretation.

Warnings in early narrative assessment that might predict performance in residency: signal from an internal medicine residency program.

INTRODUCTION: Narrative assessment data are valuable in understanding struggles in resident performance. However, it remains unknown which themes in narrative data that occur early in training may indicate a higher likelihood of struggles later in training, allowing programs to intervene sooner. METHODS: Using learning analytics, we identified 26 internal medicine residents in three cohorts that were below expected entrustment during training. We compiled all narrative data in the first 6 months of training for these residents as well as 13 typically performing residents for comparison. Narrative data were blinded for all 39 residents during initial phases of an inductive thematic analysis for initial coding. RESULTS: Many similarities were identified between the two cohorts. Codes that differed between typical and lower entrusted residents were grouped into two types of themes: three explicit/manifest and three implicit/latent with six total themes. The explicit/manifest themes focused on specific aspects of resident performance with assessors describing 1) Gaps in attention to detail, 2) Communication deficits with patients, and 3) Difficulty recognizing the "big picture" in patient care. Three implicit/latent themes, focused on how narrative data were written, were also identified: 1) Feedback described as a deficiency rather than an opportunity to improve, 2) Normative comparisons to identify a resident as being behind their peers, and 3) Warning of possible risk to patient care. DISCUSSION: Clinical competency committees (CCCs) usually rely on accumulated data and trends. Using the themes in this paper while reviewing narrative comments may help CCCs with earlier recognition and better allocation of resources to support residents' development.

Constructing a Validity Map for a Workplace-Based Assessment System: Cross-Walking Messick and Kane.

PROBLEM: Health professions education has shifted to a competency-based paradigm in which many programs rely heavily on workplace-based assessment (WBA) to produce data for summative decisions about learners. However, WBAs are complex and require validity evidence beyond psychometric analysis. Here, the authors describe their use of a rhetorical argumentation process to develop a map of validity evidence for summative decisions in an entrustment-based WBA system. APPROACH: To organize evidence, the authors cross-walked 2 contemporary validity frameworks, one that emphasizes sources of evidence (Messick) and another that stresses inferences in an argument (Kane). They constructed a validity map using 4 steps: (1) Asking critical questions about the stated interpretation and use, (2) Seeking validity evidence as a response, (3) Categorizing evidence using both Messick's and Kane's frameworks, and (4) Building a visual representation of the collected and organized evidence. The authors used an iterative approach, adding new critical questions and evidence over time. OUTCOMES: The first map draft produced 25 boxes of evidence that included all 5 sources of evidence detailed by Messick and spread across all 4 inferences described by Kane. The rhetorical question-response process allowed for structured critical appraisal of the WBA system, leading to the identification of evidentiary gaps. NEXT STEPS: Future map iterations will integrate evidence quality indicators and allow for deeper dives into the evidence. The authors intend to share their map with graduate medical education stakeholders (e.g., accreditors, institutional leaders, learners, patients) to understand if it adds value for evaluating their WBA programs' validity arguments.

A Dynamic Risk Management Approach for Reducing Harm From Invasive Bedside Procedures Performed During Residency.

Internal medicine (IM) residents frequently perform invasive bedside procedures during residency training. Bedside procedure training in IM programs may compromise patient safety. Current evidence suggests that IM training programs rely heavily on the number of procedures completed during training as a proxy for resident competence instead of using objective postprocedure patient outcomes. The authors posit that the results of procedural training effectiveness should be reframed with outcome metrics rather than process measures alone. This article introduces the as low as reasonably achievable (ALARA) approach, which originated in the nuclear industry to increase safety margins, to help assess and reduce bedside procedural risks. Training program directors are encouraged to use ALARA calculations to define the risk trade-offs inherent in current procedural training and assess how best to reliably improve patient outcomes. The authors describe 5 options to consider: training all residents in bedside procedures, training only select residents in bedside procedures, training no residents in bedside procedures, deploying 24-hour procedure teams supervised by IM faculty, and deploying 24-hour procedure teams supervised by non-IM faculty. The authors explore how quality improvement approaches using process maps, fishbone diagrams, failure mode effects and analyses, and risk matrices can be effectively implemented to assess training resources, choices, and aims. Future research should address the drivers behind developing optimal training programs that support independent practice, correlations with patient outcomes, and methods that enable faculty to justify their supervisory decisions while adhering to ALARA risk management standards.

Variation in Entrustment When Sharing a Single Assessment System Between University- and Community-Based Residency Programs: A Comparison.

PURPOSE: Given resource constraints, many residency programs would consider adopting an entrustment-based assessment system from another program if given the opportunity. However, it is unclear if a system developed in one context would have similar or different results in another. This study sought to determine if entrustment varied between programs (community based and university based) when a single assessment system was deployed in different contexts. METHOD: The Good Samaritan Hospital (GSH) internal medicine residency program adopted the observable practice activity (OPA) workplace-based assessment system from the University of Cincinnati (UC). Comparisons for OPA-mapped subcompetency entrustment progression for programs and residents were made at specific timepoints over the course of 36 months of residency. Data collection occurred from August 2012 to June 2017 for UC and from September 2013 to June 2017 for GSH. RESULTS: GSH entrustment ratings were higher than UC for all but the 11th, 15th, and 36th months of residency (P < .0001) and were also higher for the majority of subcompetencies and competencies (P < .0001). The rate of change for average monthly entrustment was similar, with GSH having an increase of 0.041 each month versus 0.042 for UC (P = .73). Most residents progressed from lower to higher entrustment, but there was significant variation between residents in each program. CONCLUSIONS: Despite the deployment of a single entrustment-based assessment system, important outcomes may vary by context. Further research is needed to understand the contributions of tool, context, and other factors on the data these systems produce.

Transforming Resident Assessment: An Analysis Using Deming's System of Profound Knowledge.

W. Edwards Deming, in his System of Profound Knowledge, asserts that leaders who wish to transform a system should understand four essential elements: appreciation for a system, theory of knowledge, knowledge about variation, and psychology. The Accreditation Council for Graduate Medical Education (ACGME) introduced the milestones program as a part of the Next Accreditation System to create developmental language for the six core competencies and facilitate programmatic assessment within graduate medical education systems. Viewed through Deming's lens, the ACGME can be seen as the steward of a large system, with everyone who provides assessment data as workers in that system. The authors use Deming's framework to illustrate the working components of the assessment system of the University of Cincinnati College of Medicine's internal medicine residency program and draw parallels to the macrocosm of graduate medical education. Successes and failures in transforming resident assessment can be understood and predicted by identifying the system and its aims, turning information into knowledge, developing an understanding of variation, and appreciating the psychology of motivation of participants. The authors offer insights from their experience for educational leaders who wish to apply Deming's elements to their own assessment systems, with questions to explore, pitfalls to avoid, and practical approaches in doing this type of work.