The Impact of an Interactive Unconscious Bias Training on Perioperative Learners.

Background: Providers' unconscious biases reinforce health disparities through negative direct patient care and interactions with colleagues. Objective: We created a workshop grounded in Critical Race Theory and the importance of different intersectionalities to improve medical trainees' self-assessment of their implicit biases in curated facilitated spaces. Methods: A total of 44 UCSF first-year clinical anesthesiology residents (CA-1) (95% response rate) and 23 surgery residents in their research year (77% response rate) participated in this workshop over 4 separate sessions in September 2020 and 2021. Quantitative data from a pre-/post-workshop survey was analyzed via a paired t test to evaluate our workshop's effectiveness. Feedback on efficacy was obtained by coding themes from our survey's open-ended questions. Results: The workshop was evaluated positively by a total of 65 of 67 participants in the post-workshop survey. On a 5-point Likert scale, participants self-reported they agreed that their unconscious biases affect their clinical interactions from a pre-workshop mean of 3.3 (SD ± 1.32) to a post-workshop mean of 3.9 (SD ± 0.87, P = .008). Conclusion: Our findings suggest that this workshop was effective for perioperative residents and can be extrapolated to all residents by tailoring the workshop to their respective work environments.

The Consequences of Step 2 Clinical Skills Examination Discontinuation for Medical Schools and Sustainability Plans for Clinical Skills Assessment.

PURPOSE: Comprehensive clinical skills examinations using standardized patients are widely used to assess multiple physician competencies. However, these exams are resource intensive. With the discontinuation of the Step 2 Clinical Skills (CS) exam in 2021, how medical schools will change their approaches to comprehensive clinical skills exams is unknown. This study explores school responses to this change and future directions of comprehensive clinical skills exams using the program sustainability framework. METHOD: This cross-sectional, descriptive study surveyed medical school curriculum deans at 150 Liaison Committee on Medical Education-accredited U.S. medical schools from September to October 2021. The 30-question survey included questions about medical school and participant role, current comprehensive clinical skills exams, sustainability dimensions, and challenges and future directions. Descriptive statistics were used to characterize responses, and content analysis was used to identify themes in the open-ended responses. RESULTS: Educators at 75 of 150 institutions (50%) responded. Sixty-three respondents (84%) reported conducting a comprehensive clinical skills exam. The comprehensive clinical skills exam assessed readiness for graduation (51 [81%]), provided feedback for students (49 [78%]), evaluated curricula (38 [60%]), provided information for medical student performance evaluation or communication with residency (10 [16%]), and assessed other factors (6 [10%]), including preparation for Step 2 CS in the past and readiness for advancement to fourth year of medical school (multiple responses were allowed). Factors facilitating sustainability included sufficient funding to continue the exam (55 [87%]) and the belief that clinical skills assessment in medical school is now more important after discontinuation of the Step 2 CS exam (55 [87%]). Challenges to sustainability included organizational capacity and limited interinstitutional collaboration. CONCLUSIONS: Educators remain committed to the purpose of comprehensive clinical skills exams. Adapting to changed licensing requirements while sustaining clinical skills exams enables innovation and improvement in assessment of clinical competence.

A Call to Action From the California Consortium for the Assessment of Clinical Competence: Making the Case for Regional Collaboration.

The discontinuation of the United States Medical Licensing Examination Step 2 Clinical Skills Examination emphasizes the need for other reliable standardized assessments of medical student clinical skills. For 30 years, the California Consortium for the Assessment of Clinical Competence (CCACC) has collaborated in the development of clinical skills assessments and has become a valuable resource for clinicians, standardized patient educators, psychometricians, and medical educators. There are many merits to strong multi-institutional partnerships, including the integration of data across multiple schools to provide feedback to both students and curricula, pooled test performance statistics for analysis and quality assurance, shared best practices and resources, individual professional development, and opportunities for research and scholarship. The infrastructure of the CCACC allows member schools to adapt to a changing medical landscape, from emerging trends in clinical medicine to the limitations imposed by a global pandemic. In the absence of a national examination, there is now a greater need for all medical schools to develop a comprehensive, dynamic, and psychometrically sound assessment that accurately evaluates clinical skills. Medical schools working together in regional consortia have the opportunity to create and implement innovative and robust assessments that evaluate a wide range of clinical skills, ensure that medical students have met an expected level of clinical competency before graduation, and provide a framework that contributes to ongoing efforts for the development of new national clinical skills standards.

Development and validation of the safe transfusion assessment tool.

BACKGROUND: Given the prevalence and risks of blood transfusion, it is essential that trainees and practicing clinicians have a thorough understanding of relevant transfusion medicine competencies. The aim of this research was to develop and gather validity evidence for an instrument to assess knowledge of core transfusion-related competencies. METHODS: We developed the safe transfusion assessment tool (STAT) using a multistep process. Initially, 20 core competencies in transfusion medicine were identified through a consensus-driven Delphi process. Learning objectives and assessment items pertinent to each competency were created. Next, a 13-item assessment tool was piloted with multidisciplinary experts and trainees. Multiple iterative revisions were made based on feedback. Finally, the 12-item STAT was administered to 100 participants of varying training level and specialty to establish validity, difficulty and item discrimination indices, and perceived utility. RESULTS: Analysis of instrument item difficulty and item discrimination indices demonstrated the ability of the STAT to assess essential knowledge in transfusion medicine relevant to trainees and clinicians in multiple programs and practice settings. Eight of twelve items discriminated between learners with varying degrees of expertise. Hundred percent of students and trainees rated the STAT as Extremely Helpful or Somewhat Helpful and the majority planned to utilize the answer guide as a study aid. CONCLUSION: The STAT is a concise, valid, and reliable knowledge assessment tool that may be used by researchers and educators to augment transfusion medicine curricula (www.safetransfusion.ucsf.edu). Scores can help inform departments on areas in which trainees require additional support and areas of potential educational interventions.

Leadership Observation and Feedback Tool: A Novel Instrument for Assessment of Clinical Leadership Skills.

BACKGROUND: While leadership training is increasingly incorporated into residency education, existing assessment tools to provide feedback on leadership skills are only applicable in limited contexts. OBJECTIVE: We developed an instrument, the Leadership Observation and Feedback Tool (LOFT), for assessing clinical leadership. METHODS: We used an iterative process to develop the tool, beginning with adapting the Leadership Practices Inventory to create an open-ended survey for identification of clinical leadership behaviors. We presented these to leadership experts who defined essential behaviors through a modified Delphi approach. In May 2014 we tested the resulting 29-item tool among residents in the internal medicine and pediatrics departments at 2 academic medical centers. We analyzed instrument performance using Cronbach's alpha, interrater reliability using intraclass correlation coefficients (ICCs), and item performance using linear-by-linear test comparisons of responses by postgraduate year, site, and specialty. RESULTS: A total of 377 (of 526, 72%) team members completed the LOFT for 95 (of 519, 18%) residents. Overall ratings were high-only 14% scored at the novice level. Cronbach's alpha was 0.79, and the ICC ranged from 0.20 to 0.79. Linear-by-linear test comparisons revealed significant differences between postgraduate year groups for some items, but no significant differences by site or specialty. Acceptability and usefulness ratings by respondents were high. CONCLUSIONS: Despite a rigorous approach to instrument design, we were unable to collect convincing validity evidence for our instrument. The tool may still have some usefulness for providing formative feedback to residents on their clinical leadership skills.

Development of a Multi-Domain Assessment Tool for Quality Improvement Projects.

BACKGROUND: Improving the quality of health care and education has become a mandate at all levels within the medical profession. While several published quality improvement (QI) assessment tools exist, all have limitations in addressing the range of QI projects undertaken by learners in undergraduate medical education, graduate medical education, and continuing medical education. OBJECTIVE: We developed and validated a tool to assess QI projects with learner engagement across the educational continuum. METHODS: After reviewing existing tools, we interviewed local faculty who taught QI to understand how learners were engaged and what these faculty wanted in an ideal assessment tool. We then developed a list of competencies associated with QI, established items linked to these competencies, revised the items using an iterative process, and collected validity evidence for the tool. RESULTS: The resulting Multi-Domain Assessment of Quality Improvement Projects (MAQIP) rating tool contains 9 items, with criteria that may be completely fulfilled, partially fulfilled, or not fulfilled. Interrater reliability was 0.77. Untrained local faculty were able to use the tool with minimal guidance. CONCLUSIONS: The MAQIP is a 9-item, user-friendly tool that can be used to assess QI projects at various stages and to provide formative and summative feedback to learners at all levels.

Learner, Patient, and Supervisor Features Are Associated With Different Types of Cognitive Load During Procedural Skills Training: Implications for Teaching and Instructional Design.

PURPOSE: Cognitive load theory, focusing on limits of the working memory, is relevant to medical education; however, factors associated with cognitive load during procedural skills training are not well characterized. The authors sought to determine how features of learners, patients/tasks, settings, and supervisors were associated with three types of cognitive load among learners performing a specific procedure, colonoscopy, to identify implications for procedural teaching. METHOD: Data were collected through an electronically administered survey sent to 1,061 U.S. gastroenterology fellows during the 2014-2015 academic year; 477 (45.0%) participated. Participants completed the survey immediately following a colonoscopy. Using multivariable linear regression analyses, the authors identified sets of features associated with intrinsic, extraneous, and germane loads. RESULTS: Features associated with intrinsic load included learners (prior experience and year in training negatively associated, fatigue positively associated) and patient/tasks (procedural complexity positively associated, better patient tolerance negatively associated). Features associated with extraneous load included learners (fatigue positively associated), setting (queue order positively associated), and supervisors (supervisor engagement and confidence negatively associated). Only one feature, supervisor engagement, was (positively) associated with germane load. CONCLUSIONS: These data support practical recommendations for teaching procedural skills through the lens of cognitive load theory. To optimize intrinsic load, level of experience and competence of learners should be balanced with procedural complexity; part-task approaches and scaffolding may be beneficial. To reduce extraneous load, teachers should remain engaged, and factors within the procedural setting that may interfere with learning should be minimized. To optimize germane load, teachers should remain engaged.

Association Between Surgeon Scorecard Use and Operating Room Costs.

Importance: Despite the significant contribution of surgical spending to health care costs, most surgeons are unaware of their operating room costs. Objective: To examine the association between providing surgeons with individualized cost feedback and surgical supply costs in the operating room. Design, Setting, and Participants: The OR Surgical Cost Reduction (OR SCORE) project was a single-health system, multihospital, multidepartmental prospective controlled study in an urban academic setting. Intervention participants were attending surgeons in orthopedic surgery, otolaryngology-head and neck surgery, and neurological surgery (n = 63). Control participants were attending surgeons in cardiothoracic surgery, general surgery, vascular surgery, pediatric surgery, obstetrics/gynecology, ophthalmology, and urology (n = 186). Interventions: From January 1 to December 31, 2015, each surgeon in the intervention group received standardized monthly scorecards showing the median surgical supply direct cost for each procedure type performed in the prior month compared with the surgeon's baseline (July 1, 2012, to November 30, 2014) and compared with all surgeons at the institution performing the same procedure at baseline. All surgical departments were eligible for a financial incentive if they met a 5% cost reduction goal. Main Outcomes and Measures: The primary outcome was each group's median surgical supply cost per case. Secondary outcome measures included total departmental surgical supply costs, case mix index-adjusted median surgical supply costs, patient outcomes (30-day readmission, 30-day mortality, and discharge status), and surgeon responses to a postintervention study-specific health care value survey. Results: The median surgical supply direct costs per case decreased 6.54% in the intervention group, from $1398 (interquartile range [IQR], $316-$5181) (10 637 cases) in 2014 to $1307 (IQR, $319-$5037) (11 820 cases) in 2015. In contrast, the median surgical supply direct cost increased 7.42% in the control group, from $712 (IQR, $202-$1602) (16 441 cases) in 2014 to $765 (IQR, $233-$1719) (17 227 cases) in 2015. This decrease represents a total savings of $836 147 in the intervention group during the 1-year study. After controlling for surgeon, department, patient demographics, and clinical indicators in a mixed-effects model, there was a 9.95% (95% CI, 3.55%-15.93%; P = .003) surgical supply cost decrease in the intervention group over 1 year. Patient outcomes were equivalent or improved after the intervention, and surgeons who received scorecards reported higher levels of cost awareness on the health care value survey compared with controls. Conclusions and Relevance: Cost feedback to surgeons, combined with a small departmental financial incentive, was associated with significantly reduced surgical supply costs, without negatively affecting patient outcomes.

Predicting cost of care using self-reported health status data.

BACKGROUND: We examined whether self-reported employee health status data can improve the performance of administrative data-based models for predicting future high health costs, and develop a predictive model for predicting new high cost individuals. METHODS: This retrospective cohort study used data from 8,917 Safeway employees self-insured by Safeway during 2008 and 2009. We created models using step-wise multivariable logistic regression starting with health services use data, then socio-demographic data, and finally adding the self-reported health status data to the model. RESULTS: Adding self-reported health data to the baseline model that included only administrative data (health services use and demographic variables; c-statistic = 0.63) increased the model" predictive power (c-statistic = 0.70). Risk factors associated with being a new high cost individual in 2009 were: 1) had one or more ED visits in 2008 (adjusted OR: 1.87, 95 % CI: 1.52, 2.30), 2) had one or more hospitalizations in 2008 (adjusted OR: 1.95, 95 % CI: 1.38, 2.77), 3) being female (adjusted OR: 1.34, 95 % CI: 1.16, 1.55), 4) increasing age (compared with age 18-35, adjusted OR for 36-49 years: 1.28; 95 % CI: 1.03, 1.60; adjusted OR for 50-64 years: 1.92, 95 % CI: 1.55, 2.39; adjusted OR for 65+ years: 3.75, 95 % CI: 2.67, 2.23), 5) the presence of self-reported depression (adjusted OR: 1.53, 95 % CI: 1.29, 1.81), 6) chronic pain (adjusted OR: 2.22, 95 % CI: 1.81, 2.72), 7) diabetes (adjusted OR: 1.73, 95 % CI: 1.35, 2.23), 8) high blood pressure (adjusted OR: 1.42, 95 % CI: 1.21, 1.67), and 9) above average BMI (adjusted OR: 1.20, 95 % CI: 1.04, 1.38). DISCUSSION: The comparison of the models between the full sample and the sample without theprevious high cost members indicated significant differences in the predictors. This has importantimplications for models using only the health service use (administrative data) given that the past high costis significantly correlated with future high cost and often drive the predictive models. CONCLUSIONS: Self-reported health data improved the ability of our model to identify individuals at risk for being high cost beyond what was possible with administrative data alone.

Reviewing residents' competence: a qualitative study of the role of clinical competency committees in performance assessment.

PURPOSE: Clinical competency committees (CCCs) are now required in graduate medical education. This study examined how residency programs understand and operationalize this mandate for resident performance review. METHOD: In 2013, the investigators conducted semistructured interviews with 34 residency program directors at five public institutions in California, asking about each institution's CCCs and resident performance review processes. They used conventional content analysis to identify major themes from the verbatim interview transcripts. RESULTS: The purpose of resident performance review at all institutions was oriented toward one of two paradigms: a problem identification model, which predominated; or a developmental model. The problem identification model, which focused on identifying and addressing performance concerns, used performance data such as red-flag alerts and informal information shared with program directors to identify struggling residents.In the developmental model, the timely acquisition and synthesis of data to inform each resident's developmental trajectory was challenging. Participants highly valued CCC members' expertise as educators to corroborate the identification of struggling residents and to enhance credibility of the committee's outcomes. Training in applying the milestones to the CCC's work was minimal.Participants were highly committed to performance review and perceived the current process as adequate for struggling residents but potentially not for others. CONCLUSIONS: Institutions orient resident performance review toward problem identification; a developmental approach is uncommon. Clarifying the purpose of resident performance review and employing efficient information systems that synthesize performance data and engage residents and faculty in purposeful feedback discussions could enable the meaningful implementation of milestones-based assessment.

A comparison of two standard-setting approaches in high-stakes clinical performance assessment using generalizability theory.

PURPOSE: Scoring clinical assessments in a reliable and valid manner using criterion-referenced standards remains an important issue and directly affects decisions made regarding examinee proficiency. This generalizability study of students' clinical performance examination (CPX) scores examines the reliability of those scores and of their interpretation, particularly according to a newly introduced, "critical actions" criterion-referenced standard and scoring approach. METHOD: The authors applied a generalizability framework to the performance scores of 477 third-year students attending three different medical schools in 2008. The norm-referenced standard included all station checklist items. The criterion-referenced standard included only those items deemed critical to patient care by a faculty panel. The authors calculated and compared variance components and generalizability coefficients for each standard across six common stations. RESULTS: Norm-referenced scores had moderate generalizability (ρ = 0.51), whereas criterion-referenced scores showed low dependability (φ = 0.20). The estimated 63% of measurement error associated with the person-by-station interaction suggests case specificity. Increasing the number of stations on the CPX from 6 to 24, an impractical solution both for cost and time, would still yield only moderate dependability (φ = 0.50). CONCLUSIONS: Though the performance assessment of complex skills, like clinical competence, seems intrinsically valid, careful consideration of the scoring standard and approach is needed to avoid misinterpretation of proficiency. Further study is needed to determine how best to improve the reliability of criterion-referenced scores, by implementing changes to the examination structure, the process of standard-setting, or both.

Assessment of medical students' shared decision-making in standardized patient encounters.

BACKGROUND: Shared decision-making, in which physicians and patients openly explore beliefs, exchange information, and reach explicit closure, may represent optimal physician-patient communication. There are currently no universally accepted methods to assess medical students' competence in shared decision-making. OBJECTIVE: To characterize medical students' shared decision-making with standardized patients (SPs) and determine if students' use of shared decision-making correlates with SP ratings of their communication. DESIGN: Retrospective study of medical students' performance with four SPs. PARTICIPANTS: Sixty fourth-year medical students. MEASUREMENTS: Objective blinded coding of shared decision-making quantified as decision moments (exploration/articulation of perspective, information sharing, explicit closure for a particular decision); SP scoring of communication skills using a validated checklist. RESULTS: Of 779 decision moments generated in 240 encounters, 312 (40%) met criteria for shared decision-making. All students engaged in shared decision-making in at least two of the four cases, although in two cases 5% and 12% of students engaged in no shared decision-making. The most commonly discussed decision moment topics were medications (n = 98, 31%), follow-up visits (71, 23%), and diagnostic testing (44, 14%). Correlations between the number of decision moments in a case and students' communication scores were low (rho = 0.07 to 0.37). CONCLUSIONS: Although all students engaged in some shared decision-making, particularly regarding medical interventions, there was no correlation between shared decision-making and overall communication competence rated by the SPs. These findings suggest that SP ratings of students' communication skill cannot be used to infer students' use of shared decision-making. Tools to determine students' skill in shared decision-making are needed.

Commercial influence and learner-perceived bias in continuing medical education.

PURPOSE: To directly examine the relationship between commercial support of continuing medical education (CME) and perceived bias in the content of these activities. METHOD: Cross-sectional study of 213 accredited live educational programs organized by a university provider of CME from 2005 to 2007. A standard question from course evaluations was used to determine the degree to which attendees believed commercial bias was present. Binomial regression models were used to determine the association between course features that may introduce commercial bias and the extent of perceived bias at those CME activities. RESULTS: Mean response rate for attendee evaluations was 56% (SD 15%). Commercial support covered 20%-49% of costs for 45 (21%) educational activities, and > or = 50% of costs for 46 activities (22%). Few course participants perceived commercial bias, with a median of 97% (interquartile range 95%-99%) of respondents stating that the activity they attended was free of commercial bias. There was no association between extent of commercial support and the degree of perceived bias in CME activities. Similarly, perceived bias did not vary for 11 of 12 event characteristics evaluated as potential sources of commercial bias, or by score on a risk index designed to prospectively assess risk of commercial bias. CONCLUSIONS: Rates of perceived bias were low for the vast majority of CME activities in the sample and did not differ by the degree of industry support or other event characteristics. Further study is needed to determine whether commercial influence persisted in more subtle forms that were difficult for participants to detect.

Factors associated with medical students' career choices regarding internal medicine.

CONTEXT: Shortfalls in the US physician workforce are anticipated as the population ages and medical students' interest in careers in internal medicine (IM) has declined (particularly general IM, the primary specialty serving older adults). The factors influencing current students' career choices regarding IM are unclear. OBJECTIVES: To describe medical students' career decision making regarding IM and to identify modifiable factors related to this decision making. DESIGN, SETTING, AND PARTICIPANTS: Web-based cross-sectional survey of 1177 fourth-year medical students (82% response rate) at 11 US medical schools in spring 2007. MAIN OUTCOME MEASURES: Demographics, debt, educational experiences, and number who chose or considered IM careers were measured. Factor analysis was performed to assess influences on career chosen. Logistic regression analysis was conducted to assess independent association of variables with IM career choice. RESULTS: Of 1177 respondents, 274 (23.2%) planned careers in IM, including 24 (2.0%) in general IM. Only 228 (19.4%) responded that their core IM clerkship made a career in general IM seem more attractive, whereas 574 (48.8%) responded that it made a career in subspecialty IM more attractive. Three factors influenced career choice regarding IM: educational experiences in IM, the nature of patient care in IM, and lifestyle. Students were more likely to pursue careers in IM if they were male (odds ratio [OR] 1.75; 95% confidence interval [CI], 1.20-2.56), were attending a private school (OR, 1.88; 95% CI, 1.26-2.83), were favorably impressed with their educational experience in IM (OR, 4.57; 95% CI, 3.01-6.93), reported favorable feelings about caring for IM patients (OR, 8.72; 95% CI, 6.03-12.62), or reported a favorable impression of internists' lifestyle (OR, 2.00; 95% CI, 1.39-2.87). CONCLUSIONS: Medical students valued the teaching during IM clerkships but expressed serious reservations about IM as a career. Students who reported more favorable impressions of the patients cared for by internists, the IM practice environment, and internists' lifestyle were more likely to pursue a career in IM.