Response Process Validity Evidence for Video Commentary Assessment in Surgery: A Qualitative Study.

OBJECTIVES: Well-developed mental representations of a task are fundamental to proficient performance. 'Video Commentary' (VC) is a novel assessment intended to measure mental representations of surgical tasks that would reflect an important aspect of task proficiency. Whether examinees' actual response processes align with this intent remains unknown. As part of ongoing validation of the assessment, we sought to understand examinees' response processes in VC. DESIGN: Grounded theory qualitative study. In 2019, residents were interviewed about their understanding of and approach to VC. Using grounded theory, we created a theoretical model explaining relationships among factors that influence residents' response processes and performance. Residents' perceived purpose of VC was also explored using Likert-type questions. SETTING: Academic surgical residency program. PARTICIPANTS: Forty-eight surgical residents (PGY-1 to PGY-5). RESULTS: Analysis of narrative comments indicated that residents' perceived purposes of VC generally align with the educator's intent. Resident response processes are influenced by test characteristics, residents' perception and understanding of VC, and residents' personal characteristics. Four strategies seem to guide how residents respond, namely a focus on speed, points, logic, and relevance. Quantitative results indicated residents believe VC scores reflect their ability to speak quickly, ability to think quickly, and knowledge of anatomy (mean = 5.0, 4.5, and 4.4 respectively [1 = strongly disagree, 6 = strongly agree]). PGY-1 and PGY-2 residents tend to focus on naming facts whereas PGY-4 and PGY-5 residents focus on providing comprehensive descriptions. CONCLUSIONS: Residents generally have an accurate understanding of the purpose of VC. However, their use of different approaches could represent a threat to validity. The response strategies of speed, points, logic, and relevance may inform other clinical skills assessments.

LEGO®-based communication assessment in virtual general surgery residency interviews.

Purpose: Effective communication skills are a critical quality and skill that is highly sought after for surgeons which largely impacts patient outcomes. Residency programs design their interview processes to select the best candidates. LEGO®-based activities have been frequently used to enhance communication skills and team building. This study investigates the effectiveness and reliability of a novel LEGO®-based communication assessment in interviews for surgical residencies and the feasibility of implementing it in a virtual setting. Methods: This study conducted a retrospective analysis of a LEGO®-based communication assessment at the program's 2020/2021 residency interviews. Each applicant was assessed on a different model. The total scores were analyzed for consistency among raters and correlated to faculty interviews. Furthermore, the impact of the assessment structure, scoring criteria, and range of models' difficulties on the total scores were explored. Results: A total of 54 categorical and 55 preliminary applicants interviewed on 2 days. The assessment on different models and had no impact on applicants' total scores for either categorical and preliminary groups (p = 0.791 and 0.709, respectively). The communication components of the assessment showed high consistency between the raters. The two applicant groups displayed a statistically significant difference (p = 0.004) in the communication evaluation and model accuracy components. Total scores did not correlate with the faculty interviews of standardized questions in either group. Conclusion: This novel LEGO®-based communication assessment showed high reliability and promising results as a tool to assess communication and problem solving for residency interviews that can be readily implemented in a virtual setting. Supplementary Information: The online version contains supplementary material available at 10.1007/s44186-022-00021-4.

Video Commentary: Rapid-Fire Assessment of Surgical Cognitive Skills.

OBJECTIVE: We describe the use of "Video Commentary", an assessment using a set of operative videos, to assess trainees' surgical cognitive skills (operative knowledge, spatial awareness, and surgical insight). DESIGN & SETTING: The Video Commentary assessment has been routinely administered to Postgraduate Year (PGY) 1-5 general surgery residents since 2014 as part of a biannual multistation, OSCE-type exam at Mayo Clinic, Rochester, MN. Video Commentary is a rapid-fire, 6-minute assessment, where trainees watch a series of 20 to 30 second operative video clips and comment on them as they play. Each clip varies in procedure, approach, difficulty, and complexity. The combination of video clips differs according to trainees' PGY level except for a few videos that overlap among PGY groups. The name of the procedure is provided at the beginning of each clip with a countdown timer showing in the corner of the screen. A comprehensive checklist is used to score trainees' performance in real-time. DISCUSSION: Assessment of trainees of different levels and staff surgeons show a positive correlation with the experience level (p = 0.0001). The assessment provides a safe alternative to assess trainees in the operating room and encourages them to become more effective communicators. With the use of technology, large video databases can be created to provide just-in-time tailored feedback to the trainees. CONCLUSIONS: Video Commentary can serve as a time and resource-efficient assessment of trainees' surgical cognitive skills and insight. The use and demand of real-time commentary on operative videos may provide a viable approach to help surgeon educators determine trainees' baseline, progression, and readiness to advance.

Inspiring Young Learners Toward a Career in Healthcare Using Simulation.

STATEMENT: To inspire young learners toward a career in healthcare and provide them opportunities to learn lifesaving skills, a hospital-based simulation center collaborated with a local middle school to develop a 88-minute simulation-based educational activity. The activity consisted of eight 10-minute stations on lifesaving and basic medical skills. One hundred fifty students participated. Evaluation surveys showed students favored stations with extensive hands-on activities and valued the opportunity to interact with health professionals. Students also reported more interest in science careers after the visit. This collaborative effort is time-efficient and low-budget simulation-based learning experience that had an immediate impact on middle school students.