Development of a certification examination for orthopedic sports medicine fellows

Background: The purpose of this study was to develop a multifaceted examination to assess the competence of fellows following completion of a sports medicine fellowship. Methods: Orthopedic sports medicine fellows over 2 academic years were invited to participate in the study. Clinical skills were evaluated with objective structured clinical examinations, multiple-choice question examinations, an in-training evaluation report and a surgical logbook. Fellows' performance of 3 technical procedures was assessed both intraoperatively and on cadavers: anterior cruciate ligament reconstruction (ACLR), arthroscopic rotator cuff repair (RCR) and arthroscopic shoulder Bankart repair. Technical procedural skills were assessed using previously validated task-specific checklists and the Arthroscopic Surgical Skill Evaluation Tool (ASSET) global rating scale. Results: Over 2 years, 12 fellows were assessed. The Cronbach α for the technical assessments was greater than 0.8, and the interrater reliability for the cadaveric assessments was greater than 0.78, indicating satisfactory reliability. When assessed in the operating room, all fellows were determined to have achieved a minimal level of competence in the 3 surgical procedures, with the exception of 1 fellow who was not able achieve competence in ACLR. When their performance on cadaveric specimens was assessed, 2 of 12 (17%) fellows were not able to demonstrate a minimal level of competence in ACLR, 2 of 10 (20%) were not able to demonstrate a minimal level of competence for RCR and 3 of 10 (30%) were not able to demonstrate a minimal level of competence for Bankart repair. Conclusion: There was a disparity between fellows' performance in the operating room and their performance in the high-fidelity cadaveric setting, suggesting that technical performance in the operating room may not be the most appropriate measure for assessment of fellows' competence.

Contexte: Le but de cette étude était de concevoir un examen à plusieurs volets pour évaluer la compétence des moniteurs cliniques à la fin de leur formation en médecine sportive. Méthodes: Après leur formation de 2 ans pour devenir orthopédistes en médecine sportive, les moniteurs cliniques ont été invités à participer à l'étude. Leurs habiletés cliniques ont été évaluées au moyen d'examens cliniques objectifs structurés, de questionnaires à choix multiple, d'un rapport d'évaluation en cours de formation et d'un journal de bord chirurgical. Leur habileté à réaliser 3 techniques chirurgicales différentes a été évaluée au bloc opératoire et sur des cadavres : reconstruction du ligament croisé antérieur (RLCA), réparation arthroscopique de la coiffe des rotateurs (RACR) et intervention de Bankart sous endoscopie pour l'épaule. Les habiletés techniques ont été évaluées au moyen de listes de vérification spécifiques aux tâches validées et au moyen de l'outil d'évaluation globale ASSET (Arthroscopic Surgical Skill Evaluation). Résultats: Sur une période de 2 ans, 12 moniteurs ont été évalués. Le coefficient α de Cronbach pour les évaluations techniques a été supérieur à 0,8, et la fiabilité inter-examinateurs pour l'évaluation des interventions sur des cadavres a été supérieure à 0,78, soit une fiabilité jugée satisfaisante. Lors de l'évaluation au bloc opératoire, on a jugé que tous les moniteurs détenaient le niveau minimum de compétences pour exécuter les 3 techniques chirurgicales, à l'exception d'un seul qui n'a pas atteint le niveau de compétence pour la RLCA. À l'évaluation de leurs compétences pour les interventions sur des cadavres, 2 sur 12 (17 %) n'ont pas atteint le niveau minimum de compétence pour la RLCA, 2 sur 10 (20 %) pour la RACR et 3 sur 10 (30 %) pour l'intervention de Bankart. Conclusion: On a noté une disparité dans la compétence des moniteurs entre le bloc opératoire et le contexte cadavérique haute fidélité, ce qui donne à penser que le rendement technique au bloc opératoire pourrait ne pas être le moyen le plus approprié d'évaluer la compétence des moniteurs cliniques.

Use of an Objective Structured Assessment of Technical Skill After a Sports Medicine Rotation.

PURPOSE: The purpose of this study was to determine if the use of an Objective Structured Assessment of Technical skill (OSATS), using dry models, would be a valid method of assessing residents' ability to perform sports medicine procedures after training in a competency-based model. METHODS: Over 18 months, 27 residents (19 junior [postgraduate year (PGY) 1-3] and 8 senior [PGY 4-5]) sat the OSATS after their rotation, in addition to 14 sports medicine staff and fellows. Each resident was provided a list of 10 procedures in which they were expected to show competence. At the end of the rotation, each resident undertook an OSATS composed of 6 stations sampled from the 10 procedures using dry models-faculty used the Arthroscopic Surgical Skill Evaluation Tool (ASSET), task-specific checklists, as well as an overall 5-point global rating scale (GRS) to score each resident. Each procedure was videotaped for blinded review. RESULTS: The overall reliability of the OSATS (0.9) and the inter-rater reliability (0.9) were both high. A significant difference by year in training was seen for the overall GRS, the total ASSET score, and the total checklist score, as well as for each technical procedure (P < .001). Further analysis revealed a significant difference in the total ASSET score between junior (mean 18.4, 95% confidence interval [CI] 16.8 to 19.9) and senior residents (24.2, 95% CI 22.7 to 25.6), senior residents and fellows (30.1, 95% CI 28.2 to 31.9), as well as between fellows and faculty (37, 95% CI 36.1 to 27.8) (P < .05). CONCLUSIONS: The results of this study show that an OSATS using dry models shows evidence of validity when used to assess performance of technical procedures after a sports medicine rotation. However, junior residents were not able to perform as well as senior residents, suggesting that overall surgical experience is as important as intensive teaching. CLINICAL RELEVANCE: As postgraduate medical training shifts to a competency-based model, methods of assessing performance of technical procedures become necessary.

How to set the bar in competency-based medical education: standard setting after an Objective Structured Clinical Examination (OSCE).

BACKGROUND: The goal of the Objective Structured Clinical Examination (OSCE) in Competency-based Medical Education (CBME) is to establish a minimal level of competence. The purpose of this study was to 1) to determine the credibility and acceptability of the modified Angoff method of standard setting in the setting of CBME, using the Borderline Group (BG) method and the Borderline Regression (BLR) method as a reference standard; 2) to determine if it is feasible to set different standards for junior and senior residents, and 3) to determine the desired characteristics of the judges applying the modified Angoff method. METHODS: The results of a previous OSCE study (21 junior residents, 18 senior residents, and six fellows) were used. Three groups of judges performed the modified Angoff method for both junior and senior residents: 1) sports medicine surgeons, 2) non-sports medicine orthopedic surgeons, and 3) sports fellows. Judges defined a borderline resident as a resident performing at a level between competent and a novice at each station. For each checklist item, the judges answered yes or no for "will the borderline/advanced beginner examinee respond correctly to this item?" The pass mark was calculated by averaging the scores. This pass mark was compared to that created using both the BG and the BLR methods. RESULTS: A paired t-test showed that all examiner groups expected senior residents to get significantly higher percentage of checklist items correct compared to junior residents (all stations p < 0.001). There were no significant differences due to judge type. For senior residents, there were no significant differences between the cut scores determined by the modified Angoff method and the BG/BLR method. For junior residents, the cut scores determined by the modified Angoff method were lower than the cut scores determined by the BG/BLR Method (all p < 0.01). CONCLUSION: The results of this study show that the modified Angoff method is an acceptable method of setting different pass marks for senior and junior residents. The use of this method enables both senior and junior residents to sit the same OSCE, preferable in the regular assessment environment of CBME.

Current status of evidence-based sports medicine.

PURPOSE: The purpose of this investigation is to determine the proportion of sports medicine studies that are labeled as Level I Evidence in 5 journals and compare the quality of surgical and nonsurgical studies using simple quality assessment tools (Consolidated Standards of Reporting Trials [CONSORT] and Jadad). METHODS: By use of PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines over the prior 2 years in the top 5 (citation and impact factor based) sports medicine journals, only Level I Evidence studies were eligible for inclusion and were analyzed. All study types (therapeutic, prognostic, diagnostic, and economic) were analyzed. Study quality was assessed with the level of evidence, Jadad score, and CONSORT 2010 guidelines. Study demographic data were compared among journals and between surgical and nonsurgical studies by use of χ(2), 1-way analysis of variance, and 2-sample Z tests. RESULTS: We analyzed 190 Level I Evidence studies (10% of eligible studies) (119 randomized controlled trials [RCTs]). Therapeutic, nonsurgical, single-center studies from the United States were the most common studies published. Sixty-two percent of studies reported a financial conflict of interest. The knee was the most common body part studied, and track-and-field/endurance sports were the most common sports analyzed. Significant differences (P < .05) were shown in Jadad and CONSORT scores among the journals reviewed. Overall, the Jadad and CONSORT scores were 2.71 and 77%, respectively. No differences (P > .05) were shown among journals based on the proportion of Level I studies or appropriate randomization. Significant strengths and limitations of RCTs were identified. CONCLUSIONS: This study showed that Level I Evidence and RCTs comprise 10% and 6% of contemporary sports medicine literature, respectively. Therapeutic, nonsurgical, single-center studies are the most common publications with Level I Evidence. Significant differences across sports medicine journals were found in study quality. Surgical studies appropriately described randomization, blinding, and patient enrollment significantly more than nonsurgical studies. LEVEL OF EVIDENCE: Level I, systematic review of Level I studies.

Competency-Based Medical Education: Can Both Junior Residents and Senior Residents Achieve Competence After a Sports Medicine Training Module?

BACKGROUND: Competency-based medical education as a resident-training format will move postgraduate training away from time-based training, to a model based on observable outcomes. The purpose of this study was to determine whether junior residents and senior residents could demonstrate clinical skills to a similar level, after a sports medicine rotation. METHODS: All residents undertaking a three-month sports medicine rotation had to pass an Objective Structured Clinical Examination. The stations tested the fundamentals of history-taking, examination, image interpretation, differential diagnosis, informed consent, and clinical decision-making. Performance at each station was assessed with a binary station-specific checklist and an overall global rating scale, in which 1 indicated novice, 2 indicated advanced beginner, 3 indicated competent, 4 indicated proficient, and 5 indicated expert. A global rating scale was also given for each domain of knowledge. RESULTS: Over eighteen months, thirty-nine residents (twenty-one junior residents and eighteen senior residents) and six fellows (for a total of forty-five participants) completed the examination. With regard to junior residents and senior residents, analysis using a two-tailed t test demonstrated a significant difference (p < 0.01) in both total checklist score and overall global rating scale; the mean total checklist score (and standard deviation) was 56.15% ± 10.99% for junior residents and 71.87% ± 8.94% for senior residents, and the mean global rating scale was 2.44 ± 0.55 for junior residents and 3.79 ± 0.49 for senior residents. There was a significant difference between junior residents and senior residents for each knowledge domain, with a significance of p < 0.05 for history-taking and p < 0.01 for the remainder of the domains. CONCLUSIONS: Despite intensive teaching within a competency-based medical education model, junior residents were not able to demonstrate knowledge as well as senior residents, suggesting that overall clinical experience is critically important for achieving competency as measured by the Objective Structured Clinical Examination.

Publication of sports medicine-related randomized controlled trials registered in ClinicalTrials.gov.

BACKGROUND: There is increasing evidence that a significant proportion of randomized trials in medicine, and recently in orthopaedics, do not go on to publication. PURPOSE: The objectives of this study were (1) to determine publication rates of randomized controlled trials in sports medicine that have been registered with ClinicalTrials.gov (CTG) and (2) to compare the registration summaries of randomized trials on CTG with final published manuscripts on pertinent methodological variables. STUDY DESIGN: Systematic review. METHODS: Two independent investigators searched ClinicalTrials.gov for all closed and completed trials related to sports medicine until June 2009 using a text search strategy. The authors then searched for publications resulting from these registered trials in peer-reviewed journals that are indexed with MEDLINE and/or EMBASE as of February 2012 based on study authors and key words provided in the study protocol. Details of primary outcomes and secondary outcomes, study sponsors, and sample size were extracted and compared between registrations and publications. RESULTS: Of 34 closed and completed trials registered on CTG, there were 20 resultant publications in peer-reviewed journals (58.8%). There was no significant relationship between source of funding and rate of publication (P > .05). The authors found a discrepancy between the CTG registration summary and the manuscript in at least one methodological variable (primary/secondary outcomes, inclusion/exclusion criteria, sample size) in 16 of 20 (80.0%) articles and a discrepancy in the primary outcome in 8 of 20 (40.0%) published trials. CONCLUSION: Although registration of sports medicine trials in CTG does not consistently result in publication or disclosure of results at 32 months from the time of study completion, observed publication rates are higher than in other orthopaedic subspecialties. Changes are also frequently made to the final presentation of eligibility criteria and primary and secondary outcomes that are not reflected in the registered trial data.