An assessment of the Chilean National Examination of Orthopaedic Surgery.

BACKGROUND: The Chilean National Examination of Orthopaedic Surgery (EMNOT) has been administered since 2009. It was developed to determine whether individual residents are meeting minimal knowledge standards at the end of their training programs. METHODS: We performed a retrospective evaluation of the EMNOT for all years it has been administered (2009-2015). The test was analyzed for content, taxonomy of questions asked (1: direct recall; 2: diagnosis; 3: evaluation/decision-making), residents' performance, difficulty index and discrimination index. RESULTS: During the years of EMNOT administration, the most frequently tested areas have been pediatric orthopaedics (22.9 %), spine (13.8 %), general orthopaedics (13.8 %) and musculoskeletal trauma (9.9 %). A significant increase in questions with images was observed, as well as a significant decrease in the percentage of Type 1 and an increase in Type 3 questions. The Difficulty Index showed a medium level of difficulty for all years the examination has been administered. The Discrimination Index showed good discrimination in 2009, fair discrimination from 2010 through 2012, and excellent discrimination from 2013 through 2015. CONCLUSION: The EMNOT has evolved over several years to include better quality questions, better discrimination, and a more representative distribution of questions covering the different orthopaedic sub-specialties. This examination represents an effective instrument for quality assurance of orthopaedic residency programs in Chile.

Análise de um sistema de avaliação de aprendizagem para internato em pediatria baseado em exame clínico objetivo estruturado, observação de prática clínica e exame escrito / Analysis of a learning assessment system for pediatric internship based upon objective structured clinical examination, clinical practice observation and written examination

OBJETIVO: Descrever e analisar três ferramentas usadas no sistema de avaliação aplicado ao internato em pediatria por um período de 7 anos na Faculdade de Medicina, Pontificia Universidad Católica de Chile, Santiago, Chile. MÉTODOS: Pesquisa observacional retrospectiva projetada para as modalidades de avaliação usadas de 2001 a 2007 que foram implementadas no internato em pediatria. As ferramentas foram as seguintes: exame clínico objetivo estruturado (objective structured clinical examination, OSCE), exame escrito e diretrizes de observação de prática clínica diária (DOPCD). Os métodos de avaliação foram aplicados ao internato em pediatria do sexto ano, com um total de 697 alunos. A análise estatística incluiu uma avaliação descritiva, com correlação e regressões linear simples e múltipla (ANOVA), teste de Bonferroni e coeficiente alfa de Cronbach. O nível de significância foi de 5 por cento. RESULTADOS: Escores de sucesso com a OSCE foram atingidos em 75,7±8 por cento, com melhor média entre o sexo feminino (p < 0,001). Os escores de OSCE melhoraram após o terceiro ano de implementação. O coeficiente alfa de Cronbach foi de 0,11-0,78. O exame escrito teve escore médio de 79,8±10 por cento, e não houve diferenças quanto a gênero. O escore médio de DOPCD 97,1±3 por cento, com melhores resultados entre as mulheres (p < 0,005). A correlação entre os três métodos de avaliação mostrou uma relação positiva moderada, exceto no ano de 2007, quando a correlação foi maior (p < 0,001). CONCLUSÕES: Análise do sistema de avaliação da aprendizagem foi realizada usando OSCE, exame escrito e DOPCD, que são complementares entre si, gerando bons resultados.

OBJECTIVE: To describe and analyze three tools used in the assessment system applied to the pediatric internship over a 7-year period at the School of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile. METHODS: Retrospective observational research design for the assessment modalities implemented in the pediatric internship from 2001 through 2007. The tools were as follows: objective structured clinical examination (OSCE), written examination and daily clinical practice observation guidelines (DCPOG). The assessment methods were applied to the sixth-year pediatric internship with a total of 697 students. Statistical analysis included a descriptive assessment, with correlation and simple linear and multiple regressions (ANOVA), Bonferroni test and Cronbach's alpha coefficient. Significance level was set at p < 0.05. RESULTS: OSCE success scores were reached in 75.7±8 percent, with a better mean among females (p < 0.001). OSCE scores improved after the third year of implementation. Cronbach's alpha coefficient was 0.11-0.78. Written examination had a mean score of 79.8±10 percent and there were no sex differences. Mean DCPOG score was 97.1±3 percent and the results were better among females (p < 0.005). Correlation between the three assessment methods showed a moderate positive relationship except in the year of 2007, where the correlation was higher (p < 0.001). CONCLUSIONS: Analysis of the learning assessment system was performed using OSCE, written examination and DCPOG, which are complementary to each other, and yielded good results.

Analysis of a learning assessment system for pediatric internship based upon objective structured clinical examination, clinical practice observation and written examination.

OBJECTIVE: To describe and analyze three tools used in the assessment system applied to the pediatric internship over a 7-year period at the School of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile. METHODS: Retrospective observational research design for the assessment modalities implemented in the pediatric internship from 2001 through 2007. The tools were as follows: objective structured clinical examination (OSCE), written examination and daily clinical practice observation guidelines (DCPOG). The assessment methods were applied to the sixth-year pediatric internship with a total of 697 students. Statistical analysis included a descriptive assessment, with correlation and simple linear and multiple regressions (ANOVA), Bonferroni test and Cronbach's alpha coefficient. Significance level was set at p < 0.05. RESULTS: OSCE success scores were reached in 75.7+/-8%, with a better mean among females (p < 0.001). OSCE scores improved after the third year of implementation. Cronbach's alpha coefficient was 0.11-0.78. Written examination had a mean score of 79.8+/-10% and there were no sex differences. Mean DCPOG score was 97.1+/-3% and the results were better among females (p < 0.005). Correlation between the three assessment methods showed a moderate positive relationship except in the year of 2007, where the correlation was higher (p < 0.001). CONCLUSIONS: Analysis of the learning assessment system was performed using OSCE, written examination and DCPOG, which are complementary to each other, and yielded good results.