Assessing clinical competence: a multitrait-multimethod matrix construct validity study.

Education in Doctor of Medicine programs has moved towards an emphasis on clinical competency, with entrustable professional activities providing a framework of learning objectives and outcomes to be assessed within the clinical environment. While the identification and structured definition of objectives and outcomes have evolved, many methods employed to assess clerkship students' clinical skills remain relatively unchanged. There is a paucity of medical education research applying advanced statistical design and analytic techniques to investigate the validity of clinical skills assessment. One robust statistical method, multitrait-multimethod matrix analysis, can be applied to investigate construct validity across multiple assessment instruments and settings. Four traits were operationalized to represent the construct of critical clinical skills (professionalism, data gathering, data synthesis, and data delivery). The traits were assessed using three methods (direct observations by faculty coaches, clinical workplace-based evaluations, and objective structured clinical examination type clinical practice examinations). The four traits and three methods were intercorrelated for the multitrait-multimethod matrix analysis. The results indicated reliability values in the adequate to good range across the three methods with the majority of the validity coefficients demonstrating statistical significance. The clearest evidence for convergent and divergent validity was with the professionalism trait. The correlations on the same method/different traits analyses indicated substantial method effect; particularly on clinical workplace-based assessments. The multitrait-multimethod matrix approach, currently underutilized in medical education, could be employed to explore validity evidence of complex constructs such as clinical skills. These results can inform faculty development programs to improve the reliability and validity of assessments within the clinical environment.

Evaluation of Veterinary Student Surgical Skills Preparation for Ovariohysterectomy Using Simulators: A Pilot Study.

This paper describes the development and evaluation of training intended to enhance students' performance on their first live-animal ovariohysterectomy (OVH). Cognitive task analysis informed a seven-page lab manual, 30-minute video, and 46-item OVH checklist (categorized into nine surgery components and three phases of surgery). We compared two spay simulator models (higher-fidelity silicone versus lower-fidelity cloth and foam). Third-year veterinary students were randomly assigned to a training intervention: lab manual and video only; lab manual, video, and $675 silicone-based model; lab manual, video, and $64 cloth and foam model. We then assessed transfer of training to a live-animal OVH. Chi-square analyses determined statistically significant differences between the interventions on four of nine surgery components, all three phases of surgery, and overall score. Odds ratio analyses indicated that training with a spay model improved the odds of attaining an excellent or good rating on 25 of 46 checklist items, six of nine surgery components, all three phases of surgery, and the overall score. Odds ratio analyses comparing the spay models indicated an advantage for the $675 silicon-based model on only 6 of 46 checklist items, three of nine surgery components, and one phase of surgery. Training with a spay model improved performance when compared to training with a manual and video only. Results suggested that training with a lower-fidelity/cost model might be as effective when compared to a higher-fidelity/cost model. Further research is required to investigate simulator fidelity and costs on transfer of training to the operational environment.

Accuracy of portrayal by standardized patients: results from four OSCE stations conducted for high stakes examinations.

BACKGROUND: The reliability in Objective Structured Clinical Exams (OSCEs) is based on variance introduced due to examiners, stations, items, standardized patients (SP), and the interaction of one or more of these items with the candidates. The impact of SPs on the reliability has not been well studied. Accordingly, the main purpose of the present study was to assess the accuracy of portrayal by standardized patients. METHODS: Four stations from a ten station high-stakes OSCE were selected for video recording. Due to the large number of candidates to be evaluated, the OSCE was administered using four assessment tracks. Four SPs were trained for each case (n = 16). Two physician assessors were trained to assess the accuracy of SP portrayal using a station-specific instrument based on the station guidelines. For the items with disagreement a third physician was asked to review and the mode was used for analysis. Each instrument included case-specific items on verbal and physical portrayal using a 3-point rating scale ("yes", "yes, but" and "not done"). The physician assessors also scored each SP on their overall performance based on a 5-item anchored global rating scale ("very poor", "poor", "ok", "good", and "very good"). SPs at location 1 were trained by one trainer and SPs at location 2 had another trainer. All SPs were employed in a high-stakes OSCE for at least the second time. RESULTS: The reliability of rating scores ranged from Cronbach's alpha of .40 to .74. Verbal portrayal by SPs did not significantly differ for most items; however, the facial expressions of the SPs differed significantly (p < .05). An emergency management station that depended heavily on SPs physical presentation and facial expressions differed between all four SPs trained for that station. CONCLUSIONS: Variation of trained SP portrayal of the same station across different tracks and at different times in OSCE may contribute substantial error to OSCE assessments. The training of SPs should be strengthened and constantly monitored during the exam to ensure that the examinees' scores are a true reflection of their competency and devoid of exam errors.

Measuring the effectiveness of small-group and web-based training methods in teaching clinical communication: a case comparison study.

Current teaching approaches in human and veterinary medicine across North America, Europe, and Australia include lectures, group discussions, feedback, role-play, and web-based training. Increasing class sizes, changing learning preferences, and economic and logistical challenges are influencing the design and delivery of communication skills in veterinary undergraduate education. The study's objectives were to (1) assess the effectiveness of small-group and web-based methods for teaching communication skills and (2) identify which training method is more effective in helping students to develop communication skills. At the Ross University School of Veterinary Medicine (RUSVM), 96 students were randomly assigned to one of three groups (control, web, or small-group training) in a pre-intervention and post-intervention group design. An Objective Structured Clinical Examination (OSCE) was used to measure communication competence within and across the intervention and control groups. Reliability of the OSCEs was determined by generalizability theory to be 0.65 (pre-intervention OSCE) and 0.70 (post-intervention OSCE). Study results showed that (1) small-group training was the most effective teaching approach in enhancing communication skills and resulted in students scoring significantly higher on the post-intervention OSCE compared to the web-based and control groups, (2) web-based training resulted in significant though considerably smaller improvement in skills than small-group training, and (3) the control group demonstrated the lowest mean difference between the pre-intervention/post-intervention OSCE scores, reinforcing the need to teach communication skills. Furthermore, small-group training had a significant effect in improving skills derived from the initial phase of the consultation and skills related to giving information and planning.

Taking More Society for Academic Emergency Medicine Practice Tests Does Not Lead to Improved National EM-M4 Exam Scores.

INTRODUCTION: Emergency medicine (EM) is a required clerkship for third-year medical students, and an elective EM acting internship (AI) is available to fourth-year students at our institution. The Society for Academic Emergency Medicine's (SAEM) National Emergency Medicine M4 Examination (EM-M4) is administered to students at the end of the EM AI experience. To prepare for the exam, students gain access to 23 practice tests available from SAEM. In this study we investigate the correlation between the number of practice tests taken and EM-M4 performance. METHODS: We collected data for EM-M4 and the US Medical Licensing Exam (USMLE) Step 2 Clinical Knowledge (CK) from students completing a MS4 EM clerkship in consecutive medical school classes from 2014-2017 at a private medical school. In addition, we collected data during the clerkship on the number of practice exams taken and whether a comprehensive practice exam was taken. We analyzed the study population three ways to determine whether the number of practice tests impacted final exam results: a binary distribution (1-11 or 12-23 tests taken); quaternary distribution (1-6, 7-12, 13-18, or 19-23 tests taken); and individual test variability (1,2,3,…22,23 tests taken). Complete data for 147 students was used for data analysis. RESULTS: The EM-M4 showed moderate (r = 0.49) correlations with USMLE Step 2 CK. There was no significant difference in EM-M4 performance in the binary analysis (P ≤ 0.09), the quaternary analysis (P ≤ 0.09), or the continuous variable analysis (P ≤ 0.52). Inclusion of a comprehensive practice test also did not correlate with EM-M4 performance (P ≤ 0.78). CONCLUSION: Degree of utilization of SAEM practice tests did not seem to correlate with performance on the EM-M4 examination at our institution. This could be due to many factors including that the question bank is composed of items that had poor item discrimination, possible inadequate coverage of EM curriculum, and/or use of alternative study methods. While further investigation is needed, if our conclusions prove generalizable, then using the SAEM practice tests is an extraneous cognitive load from a modality without proven benefit.

A predictive and construct validity study of a high-stakes objective clinical examination for assessing the clinical competence of international medical graduates.

BACKGROUND: The purpose of this study was to investigate the predictive and construct validity of a high-stakes objective structured clinical examination (OSCE) used to select candidates for a 3-month clinical rotation to assess practice-readiness status. SUMMARY: Analyses were undertaken to establish the reliability and validity of the OSCE. The generalizability coefficient (Ep(2)) for the assessment scores (checklist, global, and total) were all high, ranging from 0.73 to 0.84. Two discriminant analyses (promotion to the 3-month rotation and pass/fail status on the rotation) provided evidence of predictive validity with a 100% correct classification rate in the pass/fail rotation results. Factor analysis results provided evidence of construct validity with four factors identified: Clinical Skills, Internal Medicine, General Medical Knowledge, and Counseling. The known group differences between licensing status and residency experience also provided evidence of construct validity. CONCLUSIONS: The results are encouraging for the predictive and construct validity of the OSCE as an assessment of clinical competence.

The Impact of Shortened Clinical Clerkships on Medical Student Performance and Clerkship Assessment.

INTRO: Medical schools sometimes need to adjust the length of third-year clinical clerkships. The literature surrounding the effects of shortened clerkships on student experience and performance is mixed. METHODS: Our medical school shortened the third year by an average of 20% per clerkship to accommodate a curricular re-design in 2018-2019. We examined test scores and measures of clinical performance as well as student experience in order to understand the impact of this change. RESULTS: Two hundred and eight students were included in the analysis, 104 in each cohort. No statistically significant differences were noted between cohorts on NBME subject examination results. There were no significant differences on Step 2 CK scores between the traditional curriculum cohort (M = 249.4, SD = 13.7) and shortened curriculum cohort (M = 248.7, SD = 15.8). Student performance on OSCE cases was similar. Similar percentages of students rated each clerkship either "good" or "excellent" in the traditional (77%) and shortened (78%) curriculum. CONCLUSION: There was no significant impact on student test scores after shortening the curriculum. Measures of student satisfaction and experience also remained stable, likely related to emphasis on retaining patient care experiences and streamlining of didactics. Curricular shortening during the third year of medical school was feasible and safe from the student perspective in our experience.

Effect of Perceived Level of Interaction on Faculty Evaluations of 3rd Year Medical Students.

INTRODUCTION: Several factors are known to affect the way clinical performance evaluations (CPEs) of medical students are completed by supervising physicians. We sought to explore the effect of faculty perceived "level of interaction" (LOI) on these evaluations. METHODS: Our third-year CPE requires evaluators to identify perceived LOI with each student as low, moderate, or high. We examined CPEs completed during the academic year 2018-2019 for differences in (1) clinical and professionalism ratings, (2) quality of narrative comments, (3) quantity of narrative comments, and (4) percentage of evaluation questions left unrated. RESULTS: A total of 3682 CPEs were included in the analysis. ANOVA revealed statistically significant differences between LOI and clinical ratings (p ≤ .001), with mean ratings from faculty with a high LOI significantly higher than from faculty with a moderate or low LOI (p ≤ .001). Chi-squared analysis demonstrated differences based on faculty LOI and whether questions were left unrated (p ≤ .001), quantity of narrative comments (p ≤ .001), and specificity of narrative comments (p ≤ .001). CONCLUSIONS: Faculty who perceive higher LOI were more likely to assign that student higher ratings, complete more of the clinical evaluation and were more likely to provide narrative feedback with more specific, higher-quality comments. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s40670-021-01307-w.

Association of a Health Equity Curriculum With Medical Students' Knowledge of Social Determinants of Health and Confidence in Working With Underserved Populations.

Importance: National organizations recommend that medical schools train students in the social determinants of health. Objective: To develop and evaluate a longitudinal health equity curriculum that was integrated into third-year clinical clerkships and provided experiential learning in partnership with community organizations. Design, Setting, and Participants: This longitudinal cohort study was conducted from June 2017 to October 2020 to evaluate the association of the curriculum with medical students' self-reported knowledge of social determinants of health and confidence working with underserved populations. Students from 1 large medical school in the southeastern US were included. Students in the class of 2019 and class of 2020 were surveyed at baseline (before the start of their third year), end of the third year, and graduation. The class of 2018 (No curriculum) was surveyed at graduation to serve as a control. Data analysis was conducted from June to September 2020. Exposures: The curriculum began with a health equity simulation followed by a series of modules. The class of 2019 participated in the simulation and piloted the initial 3 modules (pilot), and the class of 2020 participated in the simulation and the full 9 modules (full). Main Outcomes and Measures: A linear mixed-effects model was used to evaluate the change in the self-reported knowledge and confidence scores over time (potential scores ranged from 0 to 32, with higher scores indicating higher self-reported knowledge and confidence working with underserved populations). In secondary analyses, a Kruskal-Wallis test was conducted to compare graduation scores between the no, pilot, and full curriculum classes. Results: A total of 314 students (160 women [51.0%], 205 [65.3%] non-Hispanic White participants) completed at least 1 survey, including 125 students in the pilot, 121 in the full, and 68 in the no curriculum classes. One hundred forty-one students (44.9%) were interested in primary care. Total self-reported knowledge and confidence scores increased between baseline and end of clerkship (15.4 vs 23.7, P = .001) and baseline and graduation (15.4 vs 23.7, P = .001) for the pilot and full curriculum classes. Total scores at graduation were higher for the pilot curriculum (median, 24.0; interquartile range [IQR], 21.0-27.0; P = .001) and full curriculum classes (median, 23.0; IQR, 20.0-26.0; P = .01) compared with the no curriculum class (median, 20.5; IQR, 16.25-24.0). Conclusions and Relevance: In this cohort study of medical students, a dedicated health equity curriculum was associated with a significant improvement in students' self-reported knowledge of social determinants of health and confidence working with underserved populations.

Assessment of first-year veterinary students' clinical skills using objective structured clinical examinations.

The DVM program at the University of Calgary offers a Clinical Skills course each year for the first three years. The course is designed to teach students the procedural skills required for entry-level general veterinary practice. Objective Structured Clinical Examinations (OSCEs) were used to assess students' performance on these procedural skills. A series of three OSCEs were developed for the first year. Content was determined by an exam blueprint, exam scoring sheets were created, rater training was provided, a mock OSCE was performed with faculty and staff, and the criterion-referencing Ebel method was used to set cut scores for each station using two content experts. Each station and the overall exam were graded as pass or fail. Thirty first-year DVM students were assessed. Content validity was ensured by the exam blueprint and expert review. Reliability (coefficient α) of the stations from the three OSCE exams ranged from 0.0 to 0.71. The three exam reliabilities (Generalizability Theory) were, for OSCE 1, G=0.56; OSCE 2, G=0.37; and OSCE 3, G=0.32. Preliminary analysis has suggested that the OSCEs demonstrate face and content validity, and certain stations demonstrated adequate reliability. Overall exam reliability was low, which reflects issues with first-time exam delivery. Because this year was the first that this course was taught and this exam format was used, work continues in the program on the teaching of the procedural skills and the development and revision of OSCE stations and scoring checklists.

Does Clerkship Rotation Sequence Affect Performance on National Board of Medical Examiners (NBME) Clinical Subject Examinations and United States Medical Licensing Examination (USMLE) Step 2 Clinical Knowledge (CK) Examination?

PURPOSE: To investigate whether starting the clerkship year in family medicine (FM), internal medicine (IM), pediatrics, or surgery influences NBME shelf and USMLE Step 2 CK examination performance. METHODS: USMLE Step 1, Step 2 CK, and shelf examination scores for FM, IM, pediatrics, and surgery were collected. Sequences were selected on the following assignment criteria: rotation 1 (either FM or IM), rotation 5 (pediatrics), rotation 8 (surgery), rotation 1 (either pediatrics or surgery), and rotation 8 (IM). Multivariate analysis of covariance, with Step 1 as the covariate, was used to investigate rotation sequence on examination performance. RESULTS: Wilks's statistic found no statistically significant effect of rotation sequence (starting the clerkship year in FM or IM) on the pediatrics, surgery, and Step 2 CK examinations (Λ = .95, F[3,51] = .93, p ≤ .432). Wilk's statistic for the covariate (Step 1) was statistically significant (Λ = .488, F[3,51] = 17.827, p ≤ .001), indicating the two groups differ on Step 1 performance. Wilk's statistic found no statistically significant effect of rotation sequence (starting the clerkship year in pediatrics or surgery) on the IM and Step 2 CK results (Λ = .925, F[2,75] = 3.036, p ≤ .054). Wilk's statistic for the covariate (Step 1) was statistically significant (Λ = .309, F[2,75] = 83.915, p ≤ .001) indicating that the two groups differ on Step 1 performance. CONCLUSION: Starting the clerkship year in FM, IM, pediatrics, or surgery does not influence subsequent performance on shelf examinations or on Step 2 CK.

Development of content-valid technical skill assessment instruments for athletic taping skills.

BACKGROUND AND PURPOSE: The content validity of technical skill assessment instruments (TSAI) for the skills of athletic taping has not been reported. The purpose of this paper is to outline and present the process of content validation for nine TSAIs for athletic taping. Local and national validators were selected from Canadian Athletic Therapists' Association (CATA)-accredited athletic therapy (AT) programs to serve as content validators. METHODS: The process of content validation began with the creation of a detailed task analysis via mail and simple validation by local validators. Subsequently, the detailed task analysis was committee validated by a group of 10 validators from across Canada. Validators judged the importance and difficulty of each item, and a face-to-face committee-validator meeting established consensus on the majority of checklist items. Through a modified Ebel procedure, frequency distribution was used in the formation of the final TSAIs. RESULTS: Initial consensus for pre-taping assessment and technical skill performance items was low. Upon committee discussion and lack of agreement, the decision to remove pretaping assessment items was made. Initial results of importance and difficulty for athletic taping technical skills were low prior to the committee meeting. Results of importance and difficulty improved substantially following the face-to-face committee-validators meeting. Consensus on fail points improved from initial to final committee validation. CONCLUSION: The process of simple and committee validation can be seen as effective methods to establish the content validity of instruments used for the evaluation of athletic taping.