Development and validation of an objective virtual reality tool for assessing technical aptitude among potential candidates for surgical training.

BACKGROUND: Good technical skills are crucial for surgeons. Yet although surgical training programs strive to assess technical aptitude when selecting surgical residents, valid assessments of such aptitude are still lacking. Surgical simulators have been proposed as a potentially effective tool for this purpose. The current study aims to develop a technical aptitude test using a virtual reality surgical simulator, and to validate its use for the selection of surgical residents. METHODS: The study had three phases. In Phase 1, we developed an initial version of the technical aptitude test using the Lap-X-VR laparoscopic simulator. In Phases 2 and 3 we refined the test and collected empirical data to evaluate four main sources of validity evidence (content, response process, internal structure, and relationships with other variables), and to evaluate the feasibility and acceptability of the test. Specifically, Phase 2 comprised a review of the test by 30 senior surgeons, and in Phase 3 a revised version of the test was administered to 152 interns to determine its psychometric properties. RESULTS: Both the surgeons and interns rated the test as highly relevant for selecting surgical residents. Analyses of the data obtained from the trial administration of the test supported the appropriateness of the score calculation process and showed good psychometric properties, including reliability (α = 0.83) and task discrimination (mean discrimination = 0.5, SD = 0.1). The correlations between test scores and background variables revealed significant correlations with gender, surgical simulator experience, and video game experience (ps < 0.001). These variables, however, explained together only 10% of the variance in test scores. CONCLUSIONS: We describe the systematic development of an innovative virtual reality test for assessing technical aptitude in candidates for surgical training, and present evidence for its validity, feasibility and acceptability. Further validation is required to support the application of the test for selection, as well as to discern the impact of gender, surgical simulator experience, and video game experience on the fairness of test results. However, the test appears to be a promising tool that may help training programs assess the suitability of candidates for surgical training.

Using Job Analysis for Identifying the Desired Competencies of 21st-Century Surgeons for Improving Trainees Selection.

OBJECTIVE: The current selection for surgical training is based on ineffective methods. In order to identify or to develop more valid selection tools to improve the selection, it is necessary to first define what are the competencies that are most important for success in contemporary surgery. Therefore, the current study aims to identify what competencies are required for success as a surgeon in the 21st-century and to evaluate their relative importance for selection for surgical training. METHODS: Job analysis was conducted using a mixed-methods design. First, 104 senior surgeons from all surgical fields from various hospitals in Israel were interviewed in order to query their perceptions of competencies associated with success as a surgeon. Their answers were coded and analyzed to create a list of important competencies. Next, a larger sample of 1,102 surgeons and residents from all surgical fields completed a questionnaire in which they rated the importance of each competency in the list for success as a surgeon and for selection for surgical training in the 21st-century. RESULTS: Twenty-four competencies (five technical skills, six cognitive abilities, 13 personality characteristics) were identified in the interview analysis. Analysis of the questionnaire's data revealed that all 24 competencies were perceived as important for success as a surgeon in the 21st-century as well as for selection for surgical training. The perceived importance of personality characteristics was higher than both cognitive abilities (p < 0.001) and technical skills (p < 0.001). The results did not differ between different surgical fields. CONCLUSIONS: Twenty-four competencies were identified as important for 21st-century surgeons and for selection for surgical training. Although all competencies were perceived as important, personality characteristics were perceived as more important than technical skills and cognitive abilities. This updated definition of required competencies may aid in developing more valid selection methods of candidates for surgical training.