Using virtual-reality simulation to ensure basic competence in hysteroscopy.

BACKGROUND: Hysteroscopy is a technically challenging procedure. Specialty curricula of obstetrics and gynaecology appraise hysteroscopy for trainees but there is no present evidence-based training program that certifies the fundamental technical skills before performance on patients. The objectives of this study were to develop and gather validity evidence for a simulation-based test that can ensure basic competence in hysteroscopy. METHODS: We used the virtual-reality simulator HystMentor™. Six experts evaluated the feasibility and clinical relevance of the simulator modules. Six modules were selected for the test and a pilot study was carried out. Subsequently, medical students, residents, and experienced gynaecologists were enrolled for testing. Outcomes were based on generated simulator metrics. Validity evidence was explored for all five sources of evidence (content, response process, internal structure, relations to other variables, consequences of testing). RESULTS: Inter-case reliability was high for four out of five metrics (Cronbach's alpha ≥ 0.80). Significant differences were identified when comparing the three groups' performances (p values < 0.05). Participants' clinical experience was significantly correlated to their simulator test score (Pearson's r = 0.49, p < 0.001). A single medical student managed to achieve the established pass/fail score (6.7% false positive) and three experienced gynaecologists failed the test (27.3% false negative). CONCLUSIONS: We developed a virtual-reality simulation-based test in hysteroscopy with supporting validity evidence. The test is intended to ensure competency in a mastery learning program where trainees practise on the simulator until they are able to pass before they proceed to supervised training on patients.

Development and validation of a theoretical test in endosonography for pulmonary diseases.

BACKGROUND: Theoretical testing provides the necessary foundation to perform technical skills. Additionally, testing improves the retention of knowledge. OBJECTIVES: The aims of this study were to develop a multiple-choice test in endosonography for pulmonary diseases and to gather validity evidence for this test. METHODS: Initially, 78 questions were constructed after informal conversational interviews with 4 international experts in endosonography. The clarity and content validity of the questions were tested using a Delphi-like approach. Construct validity was explored by administering the test to 3 groups with different levels of endosonography experience: 27 medical students, 18 respiratory physicians with limited endosonography experience, and 14 experts in endosonography. RESULTS: Two Delphi iterations reduced the test to 52 questions. After item analysis, the final test consisted of 46 questions with a mean item discrimination of 0.47 and a mean item difficulty of 0.63. The internal consistency reliability was calculated at 0.91. The 3 groups performed significantly differently (ANOVA: p < 0.001), and post hoc tests were significant. The experts performed significantly more consistently than the novices (p = 0.037) and the intermediates (p < 0.001). CONCLUSIONS: This study provides a theoretical test in endosonography consisting of multiple-choice questions. Validity evidence was gathered, and the test demonstrated content and construct validity.

Objective assessment of total laparoscopic hysterectomy: Development and validation of a feasible rating scale for formative and summative feedback.

OBJECTIVES: The aims of the study were to develop and gather validity evidence for a feasible rating scale for formative and summative assessment of total laparoscopic hysterectomy in the operating theatre. STUDY DESIGN: The study was a prospective observer-blinded cohort study. The rating scale was developed according to the generic format of Objective Structured Assessment of Technical Skills. We applied the contemporary framework of validity to examine validity evidence of the content, response process, internal structure, relationship to other variables, and consequences. Two experienced gynecologists constructed a preliminary version of the rating scale, which was reviewed by a multicentre team of experienced gynecologists in a modified Delphi process. The surgeons (beginners and experienced surgeons) were video recorded during live performance of total laparoscopic hysterectomies. Two blinded raters evaluated the performances independently using the rating scale. Internal consistency reliability and interrater reliability were calculated as measures of internal structure. The performances of the two groups were compared and a pass/fail score was set to show the consequences of the rating scale. RESULTS: The content of the rating scale was defined during three Delphi rounds and upon agreement comprised of 12 items. Sixteen participants including 8 beginners and 8 experienced surgeons performed total laparoscopic hysterectomies. The internal consistency reliability of the items was 0.95 (Cronbach's alpha), and the interrater reliabilities (Intraclass Correlation Coefficient, absolute agreement) were 0.996 for one rater and 0.998 for two raters (P < 0.001 for all correlations). The beginners' mean performance score was 19.2 (SD 7.1) and the experienced surgeons' score was 36.4 (SD 3.9); the groups performed statistically significantly different (P < 0.001). The pass/fail score was 29.3 with no false positives and no false negatives. CONCLUSION: With this study, a feasible rating scale for the objective assessment of total laparoscopic hysterectomy was developed with sound validity evidence. The rating scale is suitable for both formative and summative feedback in the commencement of surgical training in gynecology.