Evaluación del formato remoto en tiempos de pandemia / Evaluation of a remote format in times of pandemic

En el contexto de la pandemia de COVID 19 durante el año 2020 y hasta la fecha se diseñó un formato de evaluación para nuestros alumnos de pre y posgrado bajo el alero de los exámenes clínicos objetivos estructurados (ECOE) en un formato remoto. Se revisó la literatura al respecto y se diseñó un instrumento que diera cuenta en forma oportuna y estandarizada del desempeño de nuestros alumnos y residentes. Estas experiencias están siendo evaluadas en estudios con metodología mixta. Sin embargo, decidimos compartir esta experiencia, por su impacto en el desarrollo docente de las ciencias de la salud.

In the context of the COVID-19 pandemic during 2020 and to date, an evaluation format has been designed for our undergraduate and graduate students under the design of the objective structured clinical examinations (OSCE) in a remote format. The literature was reviewed, and an instrument was designed to account for the performance of our students and residents in a timely and standardized manner. These experiences are being evaluated in studies with mixed methodology. However, we wanted to convey this experience due to its impact on the educational development of health sciences.

Faculty perceptions on objective structured clinical exam in dental education / Percepción de los profesores sobre el examen clínico estructurado y su objetivo en la educación dental

Abstract Structured Clinical Exam (OSCE) uses standardized content and procedures to assess students across multiple domains of learning. The study is aimed to assess knowledge, attitudes, practices and observations of dental faculty on OSCE. The survey was distributed into dental faculty members in randomly selected government and private institutions in Saudi Arabia. The questionnaire was pre-tested and consisted of 4 categories including general characteristics of respondents, knowledge on utility of OSCE in curriculum and its reliability, attitudes regarding OSCE on a 5 point Likert scale, practices and observations on OSCE on Multiple choice questions (both single answer and multiple answer) and responses on a 5 point Likert scale. The sample size was determined to be 93 and the survey was sent electronically to 10 institutes. 101 complete responses from 7 institutions were considered from the 122 received. Faculty participation in OSCE was high within evaluators 94% (n=94) and administrators 61% (n=61). Majority of respondents (62%) believed that OSCE is most suited for competency based education, to assess cognitive skills (73%) and diagnostic interpretation (79%). Reliability of OSCE can be increased by standardization of evaluators (77%) with highest number believing that 6-8 stations (42%) are the minimum required in an OSCE. Institution guidelines (49%) coupled with workshops (47%) was the preferred method of preparation for OSCE. Majority felt that OSCE is most suitable for high stakes exams (mean=3.37) and it is an indispensable part of dental assessment (mean=3.78). Minimum number of stations for adequate reliability was reported to be lesser that in reported literature, specially so for high stakes assessments. Logistics required for arranging an OSCE and difficulty in standardized patients, may suggest that OSCE should be used in select situations.

Resumen El examen clínico estructurado (ECOE) utiliza contenido y procedimientos estandarizados para evaluar a los estudiantes en múltiples dominios de aprendizaje. Este estudio tiene como objetivo evaluar los conocimientos, las actitudes, las prácticas y las observaciones de los profesores de odontología sobre la ECOE. La encuesta se distribuyó a los miembros de la facultad de odontología en instituciones gubernamentales y privadas seleccionadas al azar en Arabia Saudita. El cuestionario se utilizó previamente y constaba de 4 categorías que incluían generalidades de los encuestados, conocimiento sobre la utilidad de la ECOE en el plan de estudios y su confiabilidad, actitudes con respecto a la ECOE en una escala Likert de 5 puntos, prácticas y observaciones sobre la ECOE en preguntas de opción múltiple (ambas respuesta y respuesta múltiple) y respuestas en una escala Likert de 5 puntos. Se determinó el tamaño de la muestra en 93 y la encuesta se envió electrónicamente a 10 institutos. Se consideraron 101 respuestas completas de 7 instituciones. La participación del profesorado en ECOE fue alta entre los evaluadores 94% (n=94) y los administradores 61% (n=61). La mayoría de los encuestados (62%) cree que la ECOE es más adecuada para la educación basada en competencias, para evaluar las habilidades cognitivas (73%) y la interpretación del diagnóstico (79%). La confiabilidad de la ECOE puede aumentarse mediante la estandarización de los evaluadores (77%) y el número más alto cree que 6-8 estaciones (42%) son el mínimo requerido en una ECOE. Las directrices de la institución (49%) junto con los talleres (47%) fue el método preferido de preparación para la ECOE. La mayoría consideró que la ECOE es más adecuada para exámenes de alto riesgo (media=3,37) y es una parte indispensable de la evaluación dental (media=3,78).Se informó que el número mínimo de estaciones para una confiabilidad adecuada es menor que en la literatura reportada, especialmente para evaluaciones de alto riesgo. La logística necesaria para organizar un ECOE y la dificultad en los pacientes estandarizados pueden sugerir que el ECOE se debe utilizar en situaciones seleccionadas.

Validation of an instrument for the evaluation of exchange transfusion (INEXTUS) via an OSCE.

INTRODUCTION: Exchange transfusion is the treatment of choice for patients with severe hyperbilirubinemia who do not respond to phototherapy. This procedure is highly complex and requires substantial expertise to perform, however it´s not done frequently enough to guarantee adequate training. Traditional learning scenarios do not have a space reserved for teaching this procedure or an instrument that fully and objectively evaluates the skills that a professional must acquire.  OBJECTIVES: The purpose was to construct and evaluate the INEXTUS instrument´s validity evidence relevant to internal structure, in a simulated scenario through the performance of an objective structured clinical exam (OSCE).  MATERIALS AND METHODS: The Delphi consensus methodology was utilized to design the instrument; six experts participated through three rounds using the Google Forms platform. The categories and items previously obtained were subjected to validation by nine experts through a dichotomous survey. Prior to data collection, the evaluators were trained through a pilot test with 10 medical students. Subsequently, all residents of a paediatric programme were evaluated through the OSCE methodology in a simulated scenario, with 6 stations, of a clinical case of a new-born with an explicit need for exchange transfusion. During their participation in the scenario, the residents were first evaluated with the instrument developed. Additionally, audio and video filming of all students who participated was performed with the aim of conducting a second evaluation two weeks after the first four evaluators participated. RESULTS: The final INEXTUS instrument consists of 46 subitems grouped into 23 items divided into 6 categories, demonstrating an inter-rater intraclass correlation coefficient of 0.96 (95% CI 0.94, 0.98 p-value < 0.001). For the Fleiss Kappa of the 23 items evaluated, concordance was evaluated for 14 items but could not be determined for the 9 remaining items because all the ratings were equal, either because the items were not performed or they were all performed adequately. Of the 14 items, 9 good scores were obtained (95% CI 0.61 to 0.8; p value < 0.001), and 5 very good scores were obtained (95% CI 0.81 to 1; p value < 0.001). CONCLUSIONS: The INEXTUS instrument evaluates exchange transfusion skills in medical personnel in training in simulated scenarios using the OSCE methodology; it has high validity and reliability and is a high-impact educational tool.

Associations between teacher self-efficacy and the use of Objective Structured Clinical Examination in medical education / Associações entre a autoeficácia docente e a utilização do Objective Structured Clinical Examination na educação médica

Abstract: Introduction: The inclusion of new methodologies and assessments in the medical education area has indicated the need to understand teachers' perception of their own ability to use them adequately. Objective: Thus, the present study aimed to investigate possible associations between teacher self-efficacy and the use of OSCE. Method: The Teacher Self-efficacy Scale and Self-Efficacy Sources Scale and a characterization questionnaire were used. Forty-seven medical teachers from a private university, of both genders, aged between 31 and 78 years, participated in the study. Results: The results indicated that the factors Social Persuasion and Vicarious Learning were the most endorsed, suggesting that these sources are the ones that interfere the most in the formation of the participants' beliefs. There was only one positive and statistically significant correlation, with a weak magnitude, established between the Intentionality of Action Efficacy and Vicarious Learning. The other identified correlations were statistically negative and of moderate magnitude. Conclusions: The teachers who agreed with some important characteristics about the OSCE method also showed higher levels of self-efficacy. Therefore, professionals with greater perseverance, resilience and confidence, have also been more committed to teaching, research and student assistance.

Resumo: Introdução: A inserção de novas metodologias e avaliações na área da educação médica tem indicado a necessidade de compreender a percepção dos docentes sobre sua própria capacidade de utilizá-las adequadamente. Objetivo: Com base nisso, este estudo buscou investigar as possíveis associações entre a autoeficácia docente e o uso do OSCE. Método: Utilizaram-se a Escala de Autoeficácia do Professor, a Escala sobre Fontes de Autoeficácia e um questionário de caracterização. Participaram 47 docentes de Medicina de uma universidade privada, de ambos os sexos, com idade entre 31 e 78 anos. Resultados: Os resultados indicaram que os fatores persuasão social e aprendizagem vicária foram os mais endossados, sugerindo que essas fontes são as de maior interferência na formação de crenças dos participantes. Houve apenas uma correlação positiva e com significância estatística, com magnitude fraca, estabelecida entre eficácia na intencionalidade da ação e aprendizagem vicária. As demais correlações encontradas se demonstraram estatisticamente em sentido negativo e com magnitudes moderadas. Conclusões: Os docentes concordantes com algumas características importantes sobre o método OSCE apresentaram maiores níveis de autoeficácia, e isso significa que os profissionais com alto nível de perseverança, superação, confiança e resiliência são mais comprometidos com o ensino, a pesquisa e a assistência estudantil.

Objetos contemporâneos para o processo de ensino e aprendizagem da ressuscitação cardiopulmonar neonatal / Contemporary objects for the teaching and learning process of neonatal cardiopulmonary resuscitation

INTRODUÇÃO: A ressuscitação cardiopulmonar é um conjunto de medidas aplicadas com a intencionalidade de reverter a parada cardiorrespiratória. Não é necessário que o profissional enfermeiro seja especialista para atender com eficiência o neonato em parada cardiorrespiratória, seja no ambiente pré-hospitalar móvel, pré-hospitalar fixo ou intra-hospitalar, porém é imprescindível a sua qualificação visando eficiência neste contexto. OBJETIVO: Desenvolver e validar o roteiro/script e storyboard de Tecnologias Educacionais Digitais - vídeos educativos -, questionário e OSCE sobre ressuscitação cardiopulmonar neonatal em ambiente intra-hospitalar para profissionais e estudantes de enfermagem e extensivo aos demais estudantes da área da saúde. MÉTODO: Trata-se de uma pesquisa aplicada de produção tecnológica, desenvolvida na Escola de Enfermagem de Ribeirão Preto da Universidade de São Paulo (EERP/USP). Realizou-se, primeiramente, uma revisão integrativa de literatura para contextualizar o processo de ensino e aprendizado da ressuscitação cardiopulmonar neonatal para estudantes e profissionais de enfermagem. Posteriormente desenvolveram-se os instrumentos e realizou-se a validação. A população foi composta por 13 experts na área neonatal, cuidados críticos neonatais e pediatria. Foi adotada a pontuação mínima de acordo com os critérios de Fehring (1987) para a seleção dos experts. Foram realizadas nesse estudo a fase I (construção do roteiro / script e storyboard da videoaula e do vídeo de simulação de atendimento) e as etapas II (validação dos roteiros / scripts) e III (validação dos storyboards) da fase II descritas na metodologia de Fleming, Reynolds e Wallace para a elaboração de vídeos; já, o questionário e o OSCE obedeceram, respectivamente, os critérios do manual do National Board of Medical Examiners e o Medical Council of Canadá, sendo posteriormente validados por meio de instrumentos adaptados segundo Ferreira e Bellan. A análise de dados foi realizada por meio de estatística descritiva simples e estatística AC1. A pesquisa foi aprovada no Comitê de Ética em Pesquisa (CEP), conforme Resolução 466/2012. Os roteiros/scripts e storyboard desenvolvidos foram validados de acordo com o objetivo, conteúdo, relevância, ambiente, linguagem verbal e inclusão de tópicos. O questionário e OSCE foram validados em relação à organização, objetividade e clareza. RESULTADOS: Obteve-se com a revisão integrativa de literatura um total de 21 artigos e verificou-se que as principais estratégias de ensino da ressuscitação cardiopulmonar neonatal ocorreram por meio do programa Helping Baby Breath, simulação em laboratório de habilidades, simulação in situ e por meio de associação de estratégias. Paticiparam desse estudo, médicos (4) e enfermeiros (9). Com relação a titulação máxima, 30,76% possuíam especialização, 30,77% possuíam título de mestrado e 23,07% doutorado. De acordo com a classificação de Fehring (1987), 46,15% dos experts apresentaram a pontuação mínima de cinco pontos, e a maior pontuação obtida foi de doze pontos e foi alcançada por um expert. Registrou-se, com relação à concordância interavaliador geral, uma "concordância quase perfeita" para o roteiro/script e storyboard da videoaula, com AC1=0,97777, " concordância quase perfeita" para o roteiro/script e storyboard do vídeo de simulação de atendimento, com AC1= 0,9844 e no questionário e no OSCE obteve-se "concordância quase perfeita" em ambos. CONCLUSÃO: Com relação aos instrumentos desenvolvidos, estes foram considerados validados e podem ser considerados como materiais de confiança metodológica para o ensino da RCP-N

INTRODUCTION: Cardiopulmonary resuscitation is a set of measures applied with the intention of reversing cardiorespiratory arrest. It is not necessary for the professional nurse to be a specialist to efficiently assist the neonate in cardiorespiratory arrest, whether in the mobile pre-hospital environment, fixed pre-hospital or intra-hospital, but it is essential to qualify for efficiency in this context. OBJECTIVE: Develop and validate the script / script and storyboard of Digital Educational Technologies - educational videos - questionnaire and OSCE on neonatal cardiopulmonary resuscitation in an in-hospital environment for nursing professionals and students and extended to other students in the health field.METHOD: This is an applied research of technological production, developed at the School of Nursing of Ribeirão Preto, University of São Paulo (EERP / USP). First, an integrative literature review was carried out to contextualize the teaching and learning process of neonatal cardiopulmonary resuscitation for students and nursing professionals. Subsequently, the instruments were developed and validation was carried out. The population consisted of 13 experts in the neonatal area, neonatal critical care and pediatrics. The minimum score was adopted according to the criteria of Fehring (1987) for the selection of experts. In this study, phase I (construction of the script / script and storyboard of the video lesson and the simulation of attendance video) and stages II (validation of the scripts / scripts) and III (validation of the storyboards) of phase II, described in the methodology of Fleming, Reynolds and Wallace for making videos; already, the questionnaire and the OSCE obeyed, respectively, the criteria of the manual of the National Board of Medical Examiners and the Medical Council of Canada, being subsequently validated through instruments adapted according to Ferreira and Bellan. The data analysis was performed through statistics simple descriptive and statistical AC1. The research was approved by the Research Ethics Committee (CEP), according to Resolution 466/2012. The scripts / scripts and storyboard developed were validated according to the objective, content, relevance, environment, verbal language and inclusion of topics. The questionnaire and OSCE were validated in relation to organization, objectivity and clarity. RESULTS: A total of 21 articles were obtained from the integrative literature review and it was found that the main strategies for teaching neonatal cardiopulmonary resuscitation occurred through the Helping Baby Breath program, simulation in the laboratory of skills, simulation in situ and by association of strategies. Doctors (4) and nurses (9) participated in this study. Regarding the maximum degree, 30.76% had a specialization, 30.77% had a master's degree and 23.07% had a doctorate. According to the classification by Fehring (1987), 46.15% of the experts had a minimum score of five points, and the highest score obtained was twelve points and was achieved by an expert. Regarding the general inter-rater agreement, an "almost perfect agreement" was recorded for the script / script and storyboard of the video lesson, with AC1 = 0.97777, "almost perfect agreement" for the script / script and storyboard of the simulation video attendance, with AC1 = 0.9844 and in the questionnaire and OSCE, "almost perfect agreement" was obtained in both.CONCLUSION: Regarding the instruments developed, these were considered validated and can be considered as methodological reliable materials for teaching CPR-N.

Stress, anxiety, self-efficacy, and the meanings that physical therapy students attribute to their experience with an objective structured clinical examination.

BACKGROUND: Excessive stress and anxiety can impair learning. The objective structured clinical examination (OSCE) is a valuable tool to assess and promote the acquisition of clinical skills. However, significant OSCE-related stress and anxiety are frequently reported. The aim of this study was to investigate the relationships between physiological stress, self-reported levels of anxiety due to an OSCE, self-efficacy, and the meanings that physical therapy students attribute to their experience with the exam. DESIGN: Concurrent mixed methods study. METHODS: A total of 32 students took part in this study. All were enrolled in the third semester of a 10-semester Physical Therapy Bachelor Program. Salivary cortisol levels, self-reported anxiety (State-Trait Anxiety Inventory, STAI) were measured before the OSCE. Exam scores and self-efficacy ratings were also recorded. Correlations between variables were tested with the Pearson correlation, with ɑ at 0.05. Semi-structured interviews were used to explore the personal perspectives of students. Thematic analysis was used to investigate emergent themes. RESULTS: Trait anxiety scores were significantly higher than normative values (p < 0.001). A high proportion of students showed high (STAI> 49) state anxiety (37.5%) and trait anxiety (65.6%). Salivary cortisol was not associated anxiety (p > 0.05). Neither stress nor anxiety correlated with OSCE scores. A moderate and significant direct correlation was found for self-efficacy scores and OSCE scores (r = 0.475, p = 0.007). Students reported that confidence had a calming effect and led to better self-perceived performance. They also reported that the OSCE can provide meaningful learning experiences despite being stressful. CONCLUSIONS: A high proportion of our students reported a stable/lingering negative affect. However, neither stress nor anxiety related to OSCE scores. Students' confidence in their capabilities was correlated with their performance. Their subjective reports suggest that self-confidence may have protected them from the negative effects of stress and anxiety on academic performance.

Diseño de intervención y estaciones ECOE sobre alimentación del menor de dos años / Design of intervention and oser stations on feeding children under two years of age

OBJETIVO: Probar una metodología de enseñanza-aprendizaje, instrumentos de medición y sistema de implementación de la ECOE en relación con lactancia materna, alimentación complementaria, crecimiento y consejería. MATERIAL Y MÉTODOS: Se estudió la adquisición de competencias sobre alimentación en menores de dos años en internos de pediatría aplicando la evaluación clínica objetiva estructurada (ECOE) antes y después del desarrollo de un proceso de enseñanza aprendizaje (PEA) estructurado. Se organizaron cuatro estaciones de evaluación de los aspectos centrales de alimentación y crecimiento, en un grupo de internos seleccionados al azar. RESULTADOS: Las cuatro estaciones de la ECOE se aplicaron sin dificultades antes y después del PEA. Los resultados mostraron una mejora en el rendimiento de los internos, de manera individual y de grupo; en este último las diferencias en la media fueron para alimentación complementaria pre 2,5 (DE 0,93) y post 5 (DE 2,39); consejería pre 5,75 (DE 1,49) y post 8,13 (DE 1,25); lactancia materna pre 12,63 (DE 2,5) y post 16,38 (DE 2) y velocidad de crecimiento pre 3,13 (DE 1,36) y post 3,38 (DE 0,92). Los resultados fueron estadísticamente significativos para los tres primeros rubros. CONCLUSIONES: En base a estos resultados se sugieren mejoras en el programa de enseñanza y se verifica la aplicabilidad de la ECOE en el internado del Hospital del Niño Dr. Ovidio Aliaga Uría.

OBJECTIVE: To test a teaching-learning methodology, measurement tools and OSCE implementation system in relation to breastfeeding, complementary feeding, growth and counseling. METHODOLOGY: The acquisition of competences was studied by applying objective structured clinical examination (OSCE) before and after the development of a structured teaching-learning process. Four assessment stations were organized considering central aspects on feeding and growth of children under two years of age, in a group of randomly selected students during medical internship. RESULTS: The four OSCE stations were applied without difficulties before and after the learning and teaching process. The results showed an improvement in the performance of interns, individually and in groups; in the latter, mean differences were: for complementary feeding pre 2.5 (SD 0.93) and post 5 (SD 2.39); counseling pre 5.75 (SD 1.49) and post 8.13 (SD 1.25); breastfeeding pre 12.63 (SD 2.5) and post 16.38 (SD 2) and growth velocity pre 3.13 (SD 1.36) and post 3.38 (SD 0.92). The results were statistically significant for the first three items. CONCLUSIONS: Based on these results, the authors suggest improvements in the teaching program, and verify the applicability of the OSCE for the evaluation of rotatory internship at the Hospital del Nino Dr. Ovidio Aliaga Uria.

Educational strategy for the development of skills in exchange transfusion: a randomized clinical trial protocol.

BACKGROUND: Exchange transfusion is a highly complex procedure that requires high levels of expertise. Trainee paediatricians do not have adequate training in exchange transfusion because opportunities to perform this procedure in practice are scarce. This protocol seeks to compare two educational interventions for exchange transfusion that allow students to develop competencies to perform the technique in an appropriate and safe way. METHODS/DESIGN: This is a randomized parallel single-blind clinical trial with allocation by simple randomization to the educational intervention (simulation or a digital didactic environment). Students from the paediatric specialization who volunteer to participate will be included. A practical evaluation of the procedure will be performed through a simulated scenario using a standardized clinical case. The main outcome is defined as the result of evaluation using the Objective Structured Clinical Examination; superior performance will be defined when the percentage is greater than or equal to 85%, and non-superior performance will be defined when the result is less than 84%. The chi-square independence test or the Fisher exact test will be used to evaluate the effect of the interventions. Multivariate analysis will be performed using a non-conditional logistic regression model. Stata 15® software will be used. DISCUSSION: Exchange transfusion is a procedure that requires expertise to achieve adequate outcomes. The inclusion of new educational strategies, such as simulation and digital didactic environments, is seen as a training option that can improve performance in clinical skills, reduce adverse events and increase the level of trust. TRIAL REGISTRATION: ClinicalTrials.gov: NCT04070066. Registered on 28 August 2019. https://clinicaltrials.gov.

An evaluative study of objective structured clinical examination (OSCE): students and examiners perspectives.

Background: The objective structured clinical examination (OSCE) is the gold standard and universal format to assess the clinical competence of medical students in a comprehensive, reliable and valid manner. The clinical competence is assessed by a team of many examiners on various stations of the examination. Therefore, it is found to be a more complex, resource- and time-intensive assessment exercise compared to the traditional examinations. Purpose: The objective of this study was to determine the final year MBBS students' and OSCE examiners' perception on the attributes, quality, validity, reliability and organization of the Medicine and Therapeutics exit OSCE held at the University of the West Indies (Cave Hill) in June 2017. Methods: At the end of the OSCE, students and examiners were provided with a questionnaire to obtain their views and comments about the OSCE. Due to the ordinal level of data produced by the Likert scale survey, statistical analysis was performed using the median, IQR and chi-square. Results: A total of 52 students and 22 examiners completed the questionnaire. The majority of the students provided positive views regarding the attributes (eg, fairness, administration, structure, sequence, and coverage of knowledge/clinical skills), quality (eg, awareness, instructions, tasks, and sequence of stations), validity and reliability (eg, true measure of essential clinical skills, standardized, practical and useful experiences), and organization (eg, orientation, timetable, announcements and quality of examination rooms) of the OSCE. Similarly, majority of the examiners expressed their satisfaction with organization, administration and process of OSCE. However, students expressed certain concerns such as stressful environment and difficulty level of OSCE. Conclusion: Overall, the OSCE was perceived very positively and welcomed by both the students and examiners. The concerns and challenges regarding OSCE can be overcome through better orientation of the faculty and preparation of the students for the OSCE.

Does an Additional Online Anatomy Course Improve Performance of Medical Students on Gross Anatomy Examinations?

AIM: An online learning course in anatomy was added to the regular academic anatomy course in the 2nd year of medicine at UNAN-Leon in Nicaragua, using the MOODLE platform. This study aims to determine the learning effect of this course. METHOD: Second-year medical students were randomly allocated to an experimental (N = 25) and control group (N = 50). Only the experimental group had access to the online learning module. We compared the performance of the experimental and the control group on both regular anatomy assessment and an objective structured practical exam (OSPE). Additionally, five focus groups were interviewed to learn about their experiences of the expanded course. RESULTS: Of students in the experimental group 94.1% and 81.6% of students in the control group took the OSPE. The experimental group significantly outperformed the control group (41.1 ± 19.3 points vs. 32.1 ± 23.1 points) on the OSPE. No differences between the two groups were found on the regular anatomy examination. Focus group interviews revealed students' opinions about the online course were generally positive. CONCLUSION: In general, the addition of an online course to the regular course was beneficial. The results of the qualitative evaluation of this intervention provides us with input about how to teach and evaluate the anatomy course and how to further improve the online course to enhance anatomy learning.

Percepção do estudante sobre a implantação do método OSCE no curso de Odontologia em uma universidade particular / Student's perception of the implementation of the OSCE method in the Dentistry course of a private university

O Objective Structured Clinical Examination (OSCE) é uma avaliação que observa e intervém nas inseguranças dos estudantes em uma condição clínica simulada. Essa forma de avaliar competências, habilidades clínicas, conhecimento, atitudes, comunicação e profissionalismo é considerada uma ferramenta importante no processo de ensino-aprendizagem. Nesse contexto, surgiu a necessidade em implementar, no curso de Odontologia, a avaliação de competência clínica OSCE. O objetivo do estudo foi avaliar a percepção do estudante de Odontologia sobre o método OSCE e seu impacto na formação acadêmica deles. Participaram 34 estudantes do último ano do curso. O instrumento de coleta de dados foi um questionário com opções de respostas ("discordo", "concordo", "concordo parcialmente", "indiferente") que foi respondido voluntariamente. Os dados foram submetidos ao teste Qui-quadrado que comparou as respostas das quatro aplicações da metodologia OSCE durante o ano, com nível de significância de 95% (p<0,05). Em relação à mesma pergunta, a diferença também foi significativa na última avaliação (93,1%; p=0,017). Quanto ao método ter contribuído para o futuro na prática profissional, 93,1% (p=0,042) dos estudantes concordaram, na última avaliação do ano. Pode-se concluir, com base na análise de resposta dos questionários nos quatro bimestres, que a percepção dos estudantes de Odontologia do 5á´¼ ano noturno da Universidade Positivo frente à introdução do método OSCE foi bem aceita e positiva, uma vez que a maioria das respostas indicaram que o método contribuiu significativamente para a formação e ofereceu oportunidade de aprendizagem (AU).

The Objective Structured Clinical Examination (OSCE) is used to detect and intervene in the students' lack of confidence in a simulated clinical condition. The OSCE assesses competences, clinical skills, knowledge, attitude, communication, and professionalism, being considered an important tool in the teachinglearning process. This context raised the need to implement the OSCE clinical competence assessment. The objective of the study was to assess the dentistry student's perception of the OSCE method and its impact on their academic training. This study included thirty-four sênior students. The data were collected using a multiple-choice questionnaire ("disagree", "agree", "partially agree", "indifferent") voluntarily answered. The chi-square test was used to analyze the data by comparing the answers of four OSCE methodology applications during the year, with a 95% (p <0.05) significance level. The difference was also significant regarding the same question in the last assessment (93.1%, p = 0.017). As for the method having contributed to their future professional practice, 93.1% (p = 0.042) of the students agreed in the last assessment of the year. Based on the analysis of questionnaire answers in the four bimesters, we concluded that the perception of the students attending the 5th year of dentistry at Positivo University is that the implementation of the OSCE method was well accepted and positive, since most of the answers indicated that the method significantly contributed to their training and provided learning opportunities (AU).

"Shadow" OSCE examiner. A cross-sectional study comparing the "shadow" examiner with the original OSCE examiner format

OBJECTIVES: Feedback is a powerful learning tool, but a lack of appropriate feedback is a very common complaint from learners to teachers. To improve opportunities for feedback on objective structured clinical examinations (OSCEs), a modified examiner role, termed the "shadow" examiner, was tested. This study aims to present and analyze comparisons between the "shadow" examiner and the original OSCE examiner format. METHODS: In 2011, experiments were carried out with modifications to the examiner's role to define the "shadow" examiner format. From February 2012 to May 2014, research was conducted with 415 6th-year medical students. Of these students, 316 were randomly assigned to assessments by both "shadow" and "fixed" examiners. Pearson correlation analysis with linear regression, Student's t-tests and Bland-Altman plots were the statistical methods used to compare the assessment modes. To strengthen the analysis, checklist items were classified by domain. RESULTS: High correlations between the "shadow" and "fixed" examiners' global scores were observed. The results of the analysis of specific domains demonstrated higher correlations for cognitive scores and lower correlations for affective scores. No statistically significant differences between the mean examiner global scores were found. The Bland-Altman analysis showed that the "shadow" examiners' affective scores were significantly higher than those of the "fixed" examiners, but the magnitude of this difference was small. CONCLUSION: The modified examiner role did not lead to any important bias in the students' scores compared with the original OSCE examiner format. This new strategy may provide important insights for formative assessments of clinical performance.

Actividades prácticas en anatomía: factor nivelador de los contrastes de origen entre los alumnos de medicina / Practical activities in anatomy: leveling factor for the origin related contrasts among medical students

Las mediciones de los logros educacionales y las pruebas de selección de ingreso a la educación superior están fuertemente correlacionadas con los antecedentes socioeconómicos y el tipo de establecimiento educacional del postulante, situación que representaría un factor predictor del rendimiento para los alumnos de primer año universitario. Considerando que los puntajes de ingreso para la carrera de medicina presentan una diferencia significativa entre los primeros y últimos seleccionados, nos preguntamos si existe algún factor nivelador que reduzca esa brecha de origen. Se analiza el rendimiento en las evaluaciones prácticas y pruebas teóricas, al comienzo y al final del año académico 2009 comparando los primeros y últimos seleccionados de acuerdo al puntaje de ingreso PSU a la carrera de medicina de la Pontificia Universidad Católica de Chile. Se observan diferencias significativas en el rendimiento de la primera evaluación Teórica de Anatomía y en la Nota Final obtenida en el curso, en cambio, no se encuentran diferencias significativas tanto en la primera Evaluación Práctica cómo en el examen final práctico del curso. Nuestros resultados sugieren que la única evaluación niveladora, que logra superar las diferencias de origen de los examinados, la constituyen nuestras pruebas prácticas; evaluaciones en las cuales el rendimiento de los primeros y de los últimos seleccionados de medicina es más homogéneo. Este tipo de evaluaciones prácticas, realizadas sobre preparaciones cadavéricas, demandan fundamentalmente experiencias vividas en nuestras actividades prácticas, colocándose por sobre la mayor o menor habilidad o experiencia que posea el estudiante relacionada con su establecimiento educacional de origen.

Level of educational achievement and screening tests for admission to higher education are strongly correlated with socioeconomic background and the type of school of the applicant, these situations would be a performance predictor for college freshmen students. Whereas the scores to be accepted into the medical career have a significant difference between the first and last selected, we wonder if there is a leveling factor that reduces the gap of origin. We analyze the performance in practical assessments and theory tests at the beginning and end of the academic year 2009 comparing the first and last selected according to the PSU score admission to medical studies at the Pontificia Universidad Catolica de Chile. Significant differences were observed in the performance of the first assessment of the theory of Anatomy and the final score obtained in the course, however, no significant differences in both the first practical test and practical final exam of the course. Our results suggest that the only evaluation level, which overcomes the differences of origin of the examined, are our practical tests; assessments in which the performance of the first and the last selected medical student is more homogenous. This type of practical assessments, performed on cadavers, primarily demanded experiences in our practical segment, standing by on the degree of skill or experience held by the students related to their educational institution of origin.

Reliability analysis of the objective structured clinical examination using generalizability theory.

BACKGROUND: The objective structured clinical examination (OSCE) is a widely used method for assessing clinical competence in health sciences education. Studies using this method have shown evidence of validity and reliability. There are no published studies of OSCE reliability measurement with generalizability theory (G-theory) in Latin America. The aims of this study were to assess the reliability of an OSCE in medical students using G-theory and explore its usefulness for quality improvement. METHODS: An observational cross-sectional study was conducted at National Autonomous University of Mexico (UNAM) Faculty of Medicine in Mexico City. A total of 278 fifth-year medical students were assessed with an 18-station OSCE in a summative end-of-career final examination. There were four exam versions. G-theory with a crossover random effects design was used to identify the main sources of variance. Examiners, standardized patients, and cases were considered as a single facet of analysis. RESULTS: The exam was applied to 278 medical students. The OSCE had a generalizability coefficient of 0.93. The major components of variance were stations, students, and residual error. The sites and the versions of the tests had minimum variance. CONCLUSIONS: Our study achieved a G coefficient similar to that found in other reports, which is acceptable for summative tests. G-theory allows the estimation of the magnitude of multiple sources of error and helps decision makers to determine the number of stations, test versions, and examiners needed to obtain reliable measurements.

Aplicación del Examen Clínico Objetivo Estructurado en la residencia de Pediatría de la Ciudad de Buenos Aires / The Use of the Objective Structured Clinical Examination for Paediatric residents in the City of Buenos Aires

Introducción: A pesar de considerarse el estándar de oro para la evaluación de competencias en posgrado, el Examen Clínico Objetivo Estructurado (ECOE) es escasamente aplicado en América Latina. El Gobierno de la Ciudad de Buenos Aires (GCBA) posee un sistema de residencia de Pediatría con cerca de 400 residentes, distribuidos en 13 hospitales, que comparten examen de ingreso y programa de formación. Nuestro objetivo es describir la experiencia de aplicación del ECOE a todos los residentes de Pediatría del GCBA, y comparar el desempeño según el tipo de hospital. Sujetos y método: Estudio descriptivo, incluyendo a todos los residentes de Pediatría del GCBA que finalizaban primer año, pertenecientes a 13 hospitales (2 pediátricos y 11 generales). El ECOE incluyó 10 estaciones. Resultados: Participaron 85 residentes; el 88,2% (IC 95% 79,7-93,5) aprobó la evaluación. No se encontraron diferencias significativas en la proporción de residentes que aprobó la evaluación entre los que provenían de hospitales pediátricos y los de hospitales generales (89,5 vs. 85,7%; OR = 1,4; IC 95% 0,4-5,5; p = 0,8). Conclusiones: En 2015 por primera vez se desarrolló un ECOE como evaluación unificada para todos los residentes de Pediatría del GCBA. La experiencia permitió identificar debilidades de cada evaluado y del sistema, estableciendo estrategias para superarlas.

Introduction: The Objective Structured Clinical Examination (OSCE) is considered the reference standard for competence evaluation, but its use in Latin America is limited. The City of Buenos Aires Government (CBAG) administers a Paediatric residency system that includes 400 residents distributed in 13 hospitals, sharing an admission system and education program. We aim to describe the experience of administering an OSCE for evaluating all the Paediatric residents of the CBAG. Subjects and method: Descriptive study, including all paediatric residents of the CBAG, belonging to 13 hospitals (2 paediatric and 11 general), ending their first year of training. The OSCE included 10 stations. Results: Eighty-five residents participated in the OSCE, and 88.2% (95% CI 79.7-93.5) passed the examination. There were no significant differences in the pass rate between residents from paediatric hospitals and from general hospitals (89.5 vs. 85.7%; OR = 1.4; 95% CI 0.4-5.5; P = .8). Conclusions: In 2015, the OSCE was administered to all paediatric residents of the CBAG for the first time. This experience allowed identifying weaknesses in the education system, in order to develop strategies to overcome them.

[The Use of the Objective Structured Clinical Examination for Paediatric residents in the City of Buenos Aires]. / Aplicación del Examen Clínico Objetivo Estructurado en la residencia de Pediatría de la Ciudad de Buenos Aires.

INTRODUCTION: The Objective Structured Clinical Examination (OSCE) is considered the reference standard for competence evaluation, but its use in Latin America is limited. The City of Buenos Aires Government (CBAG) administers a Paediatric residency system that includes 400 residents distributed in 13 hospitals, sharing an admission system and education program. We aim to describe the experience of administering an OSCE for evaluating all the Paediatric residents of the CBAG. SUBJECTS AND METHOD: Descriptive study, including all paediatric residents of the CBAG, belonging to 13 hospitals (2 paediatric and 11 general), ending their first year of training. The OSCE included 10 stations. RESULTS: Eighty-five residents participated in the OSCE, and 88.2% (95% CI 79.7-93.5) passed the examination. There were no significant differences in the pass rate between residents from paediatric hospitals and from general hospitals (89.5 vs. 85.7%; OR=1.4; 95% CI 0.4-5.5; P=.8). CONCLUSIONS: In 2015, the OSCE was administered to all paediatric residents of the CBAG for the first time. This experience allowed identifying weaknesses in the education system, in order to develop strategies to overcome them.

Examen Clínico Objetivo Estructurado como instrumento para evaluar la competencia clínica en Pediatría. Estudio piloto / Objective Structured Clinical Examination as an instrument for evaluation of clinical competence in pediatrics. A pilot study

Introducción. La evaluación es un componente esencial del proceso enseñanza-aprendizaje que motiva y dirige al alumno hacia sus objetivos y permite a los docentes conocer si los educandos poseen la destreza y el grado de conocimientos necesarios para una práctica profesional competente. En medicina no existe el método ideal de evaluación, por eso se recomienda el uso de diversos instrumentos como el Examen Clínico Objetivo Estructurado (ECOE) que ha mostrado su utilidad para evaluar las destrezas clínicas. El objetivo de este trabajo fue describir la experiencia de docentes y alumnos en la elaboración y la aplicación de un ECOE en un hospital pediátrico. Métodos. Se preparó un ECOE de 20 estaciones que se aplicó en un estudio piloto a 20 alumnos (2 residentes de primer año, 7 de segundo y 11 de tercero). Para cada estación se definieron los criterios de evaluación y el punto de corte. Con los datos obtenidos se calcularon los promedios por estación y por año de residencia. Resultados. El ECOE se llevó a cabo durante 2 horas con 20 minutos. Las estaciones aprobadas por los alumnos, de manera global, fueron doce. El valor promedio de las 20 estaciones fue 6.53, con una desviación estándar (DE) de 0.62 [R1 = 6.13 (DE 0.43), R2 = 6.26 (DE 0.60) y R3 = 6.76 (DE 0.59)]. Conclusiones. El ECOE es un método válido y confiable que permite evaluar la competencia clínica de manera integral. La experiencia de su uso en programas de posgrado ha sido limitada; sin embargo, el presente estudio muestra que es una herramienta factible que puede resultar de gran valor tanto para los pediatras en formación como para sus profesores.

Background. Assessment is an essential component of the teaching-learning process that stimulates and leads learners towards their goals and allows teachers to ascertain whether the students have acquired the necessary knowledge and clinical skills to be professionally competent. In medicine, the ideal assessment method does not exist; therefore, the use of several assessment instruments is advised; among them, the Objective Structured Clinical Examination (OSCE) has proven its advantage assessing clinical skills. The aim of this work is to describe the experience of teachers and learners developing and applying an OSCE in a children's hospital. Methods. Twenty OSCE stations were designed and applied in a pilot study. The assessment criteria for each station were defined. Individual marks were recorded and means for each station and year of residency were calculated (two first-year residents, seven second-year residents and 11 third-year residents). Results. The OSCE lasted 2 h and 20 min. Overall, 12 stations were accredited. The overall mean was 6.53, standard deviation (SD) 0.62; the mean for first-year residents was 6.13 (SD 0.43), for second-year residents 6.26 (SD 0.60) and 6.76 for third-year residents (SD 0.59). Conclusions. The OSCE isa valid and reliable method that permits an integral evaluation of clinical competence. The experience with this instrument has been limited to assessing postgraduate students. This study, however, shows that it is a useful tool that may be valuable for resident pediatricians and their professors.

Exame clínico objetivo estruturado (ECOE): uma experiência de ensino por meio de simulação do atendimento farmacêutico / Objective structured clinical examinations (OSCE): a teaching experience using simulation of pharmaceutical care / Examen clínico, objetivo estructurado (ECOE): una experiencia de enseñanza por medio simulación de la atención del farmacéutica

A simulação de atendimento farmacêutico tem por finalidade avaliar as competências clínicas dos acadêmicos. Constitui uma ferramenta pedagógica que possui três etapas: a) preparação do cenário e dos casos; b) simulação; e c) avaliação. Os casos simulados consistem de informações essenciais que são fornecidas espontaneamente ao acadêmico no início da simulação e outras complementares, que somente tornar-se-ão conhecidas se forem investigadas. Toda a simulação de atendimento é filmada, o que permite a análise da comunicação. O processo de avaliação é iniciado com a leitura do caso, a análise do vídeo e a aplicação do instrumento desenvolvido. O ECOE possibilita ao acadêmico uma oportunidade de melhorar as suas habilidades e atitudes na prestação do atendimento farmacêutico, além de aprofundar os conhecimentos a respeito das situações simuladas.

Simulation of pharmaceutical care aims to assess students' clinical skills. It comprises an educational tool that has three steps: (a) setting the stage and preparing the cases; (b) simulation; and (c) evaluation. The simulated cases consist of information that is provided spontaneously to students at the beginning of the simulation and other, complementary information that only becomes known if it is investigated. All of the simulated attendance is filmed, which enables analysis of the communication. The evaluation process starts with reading the case, analysis of the video and application of the tool that has been developed. This examination provides students with an opportunity to improve their skills and attitudes in relation to provision of pharmaceutical care, in addition to deepening their knowledge about the situations that were simulated.

La simulación de atención farmacéutica tiene por finalidad la de evaluar las capacidades clínicas de los académicos. Constituye en una herramienta pedagógica que posee tres etapas: a) preparación del lugar y de los casos, b) simulación; y c) evaluación. los casos simulados constan de informaciones esenciales proporcionadas espontáneamente al académico en el inicio de la simulación y otras complementarias que sólo se conocerán si son investigadas. Se filma toda la simulación de atención, lo que permite el análisis de la comunicación. El proceso de evaluación se inicia con la lectura del caso, el análisis del vídeo y la aplicación del instrumento desarrollado. El ECOE posibilita al académico una oportunidad de mejorar sus habilidades y actitudes en la prestación de la atención del farmacéutica, además de profundizar los conocimientos respecto a las situaciones simuladas.