Development and evaluation of the General Surgery Objective Structured Assessment of Technical Skill (GOSATS).

BACKGROUND: Technical skill acquisition is important in surgery specialty training. Despite an emphasis on competency-based training, few tools are currently available for direct technical skills assessment at the completion of training. The aim of this study was to develop and validate a simulated technical skill examination for graduating (postgraduate year (PGY)5) general surgery trainees. METHODS: A simulated eight-station, procedure-based general surgery technical skills examination was developed. Board-certified general surgeons blinded to the level of training rated performance of PGY3 and PGY5 trainees by means of validated scoring. Cronbach's α was used to calculate reliability indices, and a conjunctive model to set a pass score with borderline regression methodology. Subkoviak methodology was employed to assess the reliability of the pass-fail decision. The relationship between passing the examination and PGY level was evaluated using χ2 analysis. RESULTS: Ten PGY3 and nine PGY5 trainees were included. Interstation reliability was 0·66, and inter-rater reliability for three stations was 0·92, 0·97 and 0·76. A pass score of 176·8 of 280 (63·1 per cent) was set. The pass rate for PGY5 trainees was 78 per cent (7 of 9), compared with 30 per cent (3 of 10) for PGY3 trainees. Reliability of the pass-fail decision had an agreement coefficient of 0·88. Graduating trainees were significantly more likely to pass the examination than PGY3 trainees (χ2  = 4·34, P = 0·037). CONCLUSION: A summative general surgery technical skills examination was developed with reliability indices within the range needed for high-stakes assessments. Further evaluation is required before the examination can be used in decisions regarding certification.

ANTECEDENTES: La adquisición de habilidades técnicas es importante en el entrenamiento especializado en cirugía. A pesar del énfasis en la capacitación basada en competencias, actualmente hay pocas herramientas disponibles para la evaluación directa de habilidades técnicas al finalizar el periodo de formación. El objetivo de este estudio fue desarrollar y validar un examen simulado de habilidades técnicas para postgraduados en formación en cirugía general (5º año postgraduado). MÉTODOS: Se desarrolló un examen simulado de habilidades técnicas de procedimientos de ocho estaciones. Cirujanos generales certificados y ciegos respecto al nivel de formación puntuaron la actuación de 10 postgraduados de tercer año (PGY3) y 9 postgraduados de quinto año (PGY5) usando una puntuación validada. Se utilizó el coeficiente alfa de Cochrane para calcular los índices de fiabilidad, y un modelo conjuntivo para establecer la nota de paso del examen mediante un método de regresión borderline. La fiabilidad de la decisión pasar/no pasar el examen se evaluó usando la metodología Subkoviak. La relación entre pasar el examen y el nivel PGY se analizó mediante la prueba de ji al cuadrado. RESULTADOS: La fiabilidad entre estaciones fue de 0,66 y la fiabilidad entre evaluadores de 0,92, 0,97 y 0,76. Se estableció una puntuación para pasar el examen de 176,8/280 (63,1%). La tasa de paso del examen para postgraduados PGY5 fue del 78% (7/9) en comparación con el 30% (3/10) para los posgraduados PGY3. La fiabilidad de la decisión pasar/no pasar fue pₒ= 0,88. Los postgraduados PGY5 presentaban una probabilidad significativamente superior de pasar el examen en comparación con los postgraduados PGY3 (χ2 = 4,34, P = 0,037). CONCLUSIÓN: Se desarrolló un examen sumatorio de habilidades técnicas de cirugía general con índices de fiabilidad dentro del rango necesario para este tipo de valoraciones. Se requiere una evaluación adicional antes de que el examen se pueda utilizar en las decisiones de certificación.

Setting pass scores for assessment of technical performance by surgical trainees.

BACKGROUND: One of the major challenges of competency-based training is defining a score representing a competent performance. The objective of this study was to set pass scores for the Objective Structured Assessment of Technical Skill. METHODS: Pass scores for the examination were set using three standard setting methods applied to data collected prospectively from first-year surgical residents (trainees). General surgery residents were then assigned an overall pass-fail status for each method. Using a compensatory model, residents passed the eight station examinations if they met the overall pass score; using a conjunctive model, residents passed if they met the overall pass score and passed at least 50 per cent of the stations. The consistency of the pass-fail decision across the three methods, and between a compensatory and conjunctive model, were compared. RESULTS: Pass scores were stable across all three methods using data from 513 residents, 133 of whom were general surgeons. Consistency of the pass-fail decision across the three methods was 95.5 and 93.2 per cent using compensatory and conjunctive models respectively. Consistency of the pass-fail status between compensatory and conjunctive models for all three methods was also very high (91.7, 95.5 and 96.2 per cent). CONCLUSION: Consistency in pass-fail status between the various methods builds evidence of validity for the set scores. These methods can be applied and studied across a variety of assessment platforms, helping to increase the use of standard setting for competency-based training.

A cryogenic ion trap for single molecule vibrational spectroscopy.

We report on an ion trapping system for performing a novel form of cryogenic messenger spectroscopy with single molecule sensitivity. The system features a cryogenic radio-frequency ion trap loaded with single analyte molecules via a quadrupole mass filter. We demonstrate the ability to controllably attach inert gas particles to buffer gas cooled, trapped molecular ions. Sympathetic cooling by co-trapped, laser cooled 88Sr+ further reduces the translational temperature of trapped molecules to the millikelvin regime. We verify the presence of cryogenic "tags" via non-destructive optical mass spectrometry and selectively remove these adducts by resonantly driving vibrational transitions in the tagged molecular ions. This enables us to derive the vibrational spectrum of a single analyte molecule from the frequency dependence of the tag detachment rate. We have demonstrated these capabilities by measuring transitions in the C-H stretching region for single cationic fragments of both indole (C8H7N) and 1,3-benzodioxole (C6H4O2CH2). These capabilities are not reliant on a specific molecular level structure and thus constitute a general, non-destructive method for vibrational spectroscopy of individual molecular ions.

Transatlantic comparison of the competence of surgeons at the start of their professional career.

BACKGROUND: Although the objective in European Union and North American surgical residency programmes is similar-to train competent surgeons-residents' working hours are different. It was hypothesized that practice-ready surgeons with more working hours would perform significantly better than those being educated within shorter working week curricula. METHODS: At each test site, 21 practice-ready candidate surgeons were recruited. Twenty qualified Canadian and 19 qualified Dutch surgeons served as examiners. At both sites, three validated outcome instruments assessing multiple aspects of surgical competency were used. RESULTS: No significant differences were found in performance on the integrative and cognitive examination (Comprehensive Integrative Puzzle) or the technical skills test (Objective Structured Assessment of Technical Skill; OSATS). A significant difference in outcome was observed only on the Patient Assessment and Management Examination, which focuses on skills needed to manage patients with complex problems (P < 0.001). A significant interaction was observed between examiner and candidate origins for both task-specific OSATS checklist (P = 0.001) and OSATS global rating scale (P < 0.001) scores. CONCLUSION: Canadian residents, serving many more working hours, perform equivalently to Dutch residents when assessed on technical skills and cognitive knowledge, but outperformed Dutch residents in skills for patient management. Secondary analyses suggested that cultural differences influence the assessment process significantly.