The Utility of Peers and Trained Raters in Technical Skill-based Assessments a Generalizability Theory Study.

OBJECTIVE: The gold standard for evaluation of resident procedural competence is that of validated assessments from faculty surgeons. A provision of adequate trainee assessments is challenged by a shortage of faculty due to increased clinical and administrative responsibilities. We hypothesized that with a well constructed assessment instrument and training, there would be minimal differences in procedural assessments made by near-peer resident raters (RR), faculty raters (FR), and trained raters (TR). DESIGN: Deidentified videos of residents performing hand-sewn (HA) and stapled (SA) anastomoses were distributed to blinded reviewers of 3 types. Intra-class correlation (ICC) of RR, FR and TR assessments was determined for each procedure. A fully-crossed design was used to examine the internal structure validity in a generalizability study. A Decision study was performed to make projections on the number of raters needed for a g-coefficient > 0.70. SETTING: This study was conducted within a private academic institution, using the creation of intestinal anastomoses as the procedural model. PARTICIPANTS: Raters consisted of residents who were untrained to the assessment (UTA) tool, UTA faculty surgeons, and individuals with training. RESULTS: Twenty nine videos were reviewed (15 HA and 14 SA) by a total of 9 video reviewers (4 RR, 2 FR, and 3 TR). HA ICC values were 0.84 (Confidence Interval [CI]:0.81-0.87) for RR, 0.89 (CI:0.86-0.92) for FR, and 0.88 (CI:0.86-0.90) for TR. SA ICC values were 0.77 (CI:0.72-0.80) for RR, 0.79 (CI:0.75-0.83) for FR, and 0.86 (CI:0.83-0.88) for TR. The g-coefficient was RR = 0.72, FR = 0.85, and TR = 0.77 for HA; and RR = 0.33, FR = 0.38, and TR = 0.4 for SA. The D-study indicated that at least 2 raters of any type were needed for HA and > 11 FR for SA. CONCLUSIONS: Faculty without training have high assessment agreement. Peers for surgical skills assessment is an option for formative evaluation without training. Training to assessment tools should be performed for any assessment, formative or summative, for the optimal evaluation of procedural competence.

A Novel Blended Curriculum for Communication of Informed Consent With Surgical Interns.

BACKGROUND: Interns often conduct procedural informed consent discussions (ICDs), identified as a core entrustable professional activity. Deficiencies in the training process for ICDs span across specialties. OBJECTIVE: We provide evidence for a curriculum and assessment designed to standardize the training process and ensure ICD competency in surgical interns. METHODS: In March 2019, PowerPoint educational materials were emailed to one academic institution's new surgical interns, who in June participated in an onsite 1-hour role-play "hot seat" group activity (GA) with an untrained simulated patient, and in October completed a single trained simulated patient (real-time raters) verification of proficiency (VOP) assessment. Curriculum evaluation was measured through intern pre-/post-confidence (5-point scale), and the VOP's Cronbach's alpha and test-retest were examined. Data were analyzed with descriptive statistics, paired t tests, and 2-way random effects models. RESULTS: Of 44 new interns, 40 (91%) participated in the remote teaching and live GA and were assessed by the VOP. Pre-/post-GA confidence increased a mean difference of 1.3 (SD = 0.63, P < .001). The VOP's Cronbach's alpha was 0.88 and test-retest was 0.84 (95% CI 0.67-0.93, P < .001), with a 95% pass rate. The 2 first-time fail students required remediation. Time commitment included 1 hour maximum for individual training and implementation and 30 minutes for assessment. The use of volunteers and donated space mitigated additional costs. CONCLUSIONS: Remote asynchronous and group skills teaching for new general surgical interns improved their confidence in conducting procedural ICDs. A patient-simulation verification process appeared feasible with preliminary evidence of retest and internal consistency.

A Generalizable Multimodal Scrub Training Curriculum in Surgical Sterile Technique.

Introduction: Recent endeavors from governing bodies such as the AAMC have formally recognized the importance of aseptic technique. AAMC guidelines include activities that all graduating physicians should be able to perform with minimum indirect supervision and were developed to recognize these needs. For example, the skills necessary for aseptic technique include daily safety habits and general physician procedures. Methods: We developed a scrub training curriculum and evaluated the program through a quasi-experimental study with a pre- and posttest design. Questions were developed to examine students' perceived knowledge and skills as related to the objectives of the course and to their anxieties, concerns, and future training needs. Results: Between February 2020 and March 2020, 44 students completed the curriculum. Students indicated that self-efficacy significantly increased in all aspects of the curricular goals following curriculum completion. Students identified understanding OR etiquette as the most anxiety-provoking element associated with scrub training. They felt that more time could be spent elucidating this etiquette. On the other hand, tasks such as surgical hand hygiene were the least anxiety-inducing. Discussion: We share this multimodal scrub training curriculum, mapped to the AAMC's guidelines, to reduce variability in teaching strategies and skills acquisition through a standardized curriculum. Also, we effectively imparted these skills and instilled a sense of confidence in learners as they worked to provide their best in patient care and safety.

Perception and confidence of medical students in informed consent: A core EPA.

BACKGROUND: Informed consent discussions have been identified as a core entrustable professional activity for medical students by the Association of American Medical Colleges. Medical students, however, rarely receive formal instruction on how to appropriately conduct informed consent discussions before residency, resulting in inconsistent levels of experience and deficiencies in performance. This study explores medical students' understanding of the elements of informed consent discussions and their readiness to perform a comprehensive informed consent discussion. METHODS: Using expert consensus, cognitive interviews, and piloting, we iteratively developed a 15-item survey aligned with entrustable professional activity guidelines concerning informed consent discussions consisting of multiple choice, free text, and 5-point Likert-type questions. The instrument covered domains of experience, confidence, medical-legal knowledge, and recall of informed consent discussion elements. The full survey was distributed anonymously to undergraduate medical students at our institution. An abbreviated survey was administered to postgraduate students who were new interns at our institution. Responses were analyzed quantitatively using descriptive statistics. The free text data were coded for inclusion in this analysis. RESULTS: A total of 75 undergraduate medical students across all years responded (response rate [RR] = 86%), and 34 (RR = 77%) of the postgraduate students who were new interns participated. A total of 45 (75%) undergraduate medical students reported no training on informed consent discussions, and 9 (15%) undergraduate medical students had never witnessed an informed consent discussion. The undergraduate medical students agreed that informed consent discussions could be legally performed by residents and advance practice providers but were unsure whether the same applied to medical students. On a 5-point scale (anchored to "Not at all," "Somewhat," and "Extremely"), they were "somewhat confident" in their ability to perform an informed consent discussion. When asked to list the 7 elements of an informed consent discussion, 2 undergraduate medical students (3%) were able to identify all the elements. Although 3 undergraduate medical students (9%) had experience leading an informed consent discussion and 11 (32%) reported formal instruction in informed consent, the ability (3.7 ± 0.9 standard deviation [SD]) of the postgraduate students who were new interns to recall the 7 elements was similar to that of the undergraduate medical students (3.4 ± 1.2 SD); P = .31. CONCLUSION: These findings suggest that undergraduate medical students and postgraduate students who are new interns are not confident or competent in their ability to perform an appropriate informed consent discussion. Our study findings support the creation of a needs-based, entrustable professional activity-aligned informed consent discussion teaching program and the need for an ongoing evaluation of the success of such a program.