Bridging Validity Frameworks in Assessment: Beyond Traditional Approaches in Health Professions Education.

Construct: High-stakes assessments measure several constructs, such as knowledge, competencies, and skills. In this case, validity evidence for test scores' uses and interpretations is of utmost importance, because of the consequences for everyone involved in their development and implementation. Background: Educational assessment requires an appropriate understanding and use of validity frameworks; however, health professions educators still struggle with the conceptual challenges of validity, and frequently validity analyses have a narrow focus. Important obstacles are the plurality of validity frameworks and the difficulty of grounding these abstract concepts in practice. Approach: We reviewed the validity frameworks literature to identify the main elements of frequently used models (Messick and Kane's) and proposed linking frameworks including Russell's recent overarching proposal. Examples are provided with commonly used assessment instruments in health professions education. Findings: Several elements in these frameworks can be integrated into a common approach, matching and aligning Messick's sources of validity with Kane's four inference types. Conclusions: This proposal to contribute evidence for assessment inferences may provide guidance to understanding the use of validity evidence in applied settings. The evolving field of validity research provides opportunities for its integration and practical use in health professions education.

Continuing professional development challenges in a rural setting: A mixed-methods study.

INTRODUCTION: Health professionals in rural settings encounter a wide range of medical conditions requiring broad knowledge for their clinical practice. This creates the need for ongoing continuing professional development (CPD). In this study, we explored the barriers that health professionals in a rural healthcare context faced participating in CPD activities and their preferences regarding educational strategies to overcome these challenges. METHODS: This mixed-methods (exploratory sequential) study in a community hospital in rural Mexico includes 22 interviews, 3 focus groups, 40 observational hours, and a questionnaire of healthcare staff. RESULTS: Despite low engagement with CPD activities (67% not motivated), all participants expressed interest and acknowledged the importance of learning for their practice. Barriers to participating include a disparity between strategies used (lecture-based) and their desire for practical learning, institutional barriers (poor leadership engagement, procedural flaws, and lack of resources), and collaboration barriers (adverse interprofessional education environment, ineffective teamwork, and poor communication). Additional barriers identified were inconvenient scheduling of sessions (75%), inadequate classrooms (65%), high workload (60%), ineffective speakers (60%), and boring sessions (55%). Participants' preferred learning strategies highlighted activities relevant to their daily clinical activities (practical workshops, simulations, and case analysis). The questionnaire had an 18% response rate. DISCUSSION: The barriers to CPD in this rural setting are multifactorial and diverse. A strong interest to engage in context-specific active learning strategies highlighted the need for leadership to prioritize interprofessional education, teamwork, and communication to enhance CPD and patient care. These results could inform efforts to strengthen CPD in other rural contexts.

MOOC Learning Assessment in Clinical Settings: Analysis from Quality Dimensions.

Online learning is becoming a fundamental modality of learning in medical education, and can be of great help during global crisis like the current COVID-19 pandemic. The MOOC (massive open online course) mode of e-learning is increasing its penetration worldwide, as a valid teaching approach to reach large populations. A major challenge in clinical education is the assessment of medical students and residents in clinical settings, and there is substantial evidence that the current situation requires improvement. The goal of this study was to evaluate the quality dimensions of a MOOC titled "Learning assessment in clinical settings," developed by three Mexican universities in the Coursera platform. A mixed-method study design was used to assess the quality dimensions of the MOOC in two phases: pilot and implementation. The best-rated aspects were learning resources and pedagogical perspective, and those with opportunity for improvement were collaboration and time management. Assessment might be learned through a MOOC format, especially for participants willing to engage with educational technology and self-direction.

[Diagnostic and formative assessment of competencies at the beginning of undergraduate medical internship]. / valuación diagnóstica y formativa de competencias en estudiantes de medicina a su ingreso al internado médico de pregrado.

INTRODUCTION: Research on diagnostic and formative assessment competencies during undergraduate medical training is scarce in Latin America. OBJECTIVE: To assess the level of clinical competence of students at the beginning of their medical internship in a new curriculum. METHODS: This was an observational cross-sectional study in UNAM Faculty of Medicine students in Mexico City: a formative assessment of the second class of Curriculum 2010 students as part of the integral evaluation of the program. The assessment had two components: theoretical and practical. RESULTS: We assessed 577 students (65.5%) of the 880 total population that finished the 9th semester of Curriculum 2010. The written exam consisted of 232 items, with a mean of 61.0 ± 19.6, a difficulty index of 0.61, and Cronbach's alpha of 0.89. The mean of the objective structured clinical examination (OSCE) was 62.2 ± 16.8, with a mean Cronbach's alpha of 0.51. Results were analyzed by knowledge area and exam stations. CONCLUSIONS: The overall results provide evidence that students achieve sufficiently the competencies established in the curriculum at the beginning of the internship, that they have the necessary foundation for learning new and more complex information, and integrate it with existing knowledge to achieve significant learning and continue their training.

[Teacher's perfomance assessment in Family Medicine specialization]. / Evaluación del desempeño docente en la especialización de Medicina Familiar.

BACKGROUND: In Mexico there is no systematic evaluation of teachers in medical specialties. It is difficult to identify appropriate teaching practices. The lack of evaluation has limited the recognition and improvement of teaching. The objective of this study was to analyze feedback from students about teaching activities of teachers-tutors responsible for the specialization course in family medicine, and evaluate the evidence of reliability and validity of the instrument applied online. METHODS: It was an observational and cross-sectional study. Seventy eight teachers of Family Medicine of medical residency were evaluated by 734 resident´s opinion. The anonymous questionnaire to assess teaching performance by resident's opinion and it is composed of 5 dimensions using a Likert scale. Descriptive and inferential statistics (t test, one-way ANOVA and factor analysis) were used. RESULTS: Residents stated that teaching performance is acceptable, with an average of 4.25 ± 0.93. The best valued dimension was "Methodology" with an average of 4.34 ± .92 in contrast to the "assessment" dimension with 4.16 ± 1.04. CONCLUSION: Teachers of specialization in family medicine have acceptable performance by resident's opinion. The online assessment tool meets the criteria of validity and reliability.

Introducción: en México no hay una evaluación sistemática del desempeño docente en las especializaciones médicas, lo que reduce la posibilidad de identificación de prácticas educativas adecuadas, su mejora y reconocimiento. El estudio pretende analizar la opinión de los residentes acerca de las actividades docentes de los profesores-tutores responsables del curso de especialización en medicina familiar, y valorar la fiabilidad y validez del instrumento aplicado en línea. Método: estudio observacional y transversal, la población de estudio fueron setenta y ocho profesores titulares de medicina familiar de la residencia médica de la Facultad de Medicina de la Universidad Nacional Autónoma de México evaluados por 734 residentes. Se utilizó un cuestionario anónimo de evaluación del desempeño docente por opinión del residente, integrado por 37 enunciados y 5 dimensiones con una escala de respuesta tipo Likert. Se utilizó estadística descriptiva e inferencial (t de Student, ANOVA de un factor y análisis factorial). Resultados: los residentes opinaron que el desempeño docente en general es aceptable, con una media de 4.25 ± 0.93, la dimensión con mayor puntaje fue la de metodología con 4.34 ± 0.92 en contraste con la dimensión de evaluación con la menor media de 4.16 ± 1.04. Conclusiones: los profesores de la especialización tienen un desempeño aceptable según la opinión de los residentes. El instrumento de evaluación en línea reúne los criterios de validez y confiabilidad.

Assessment of the Mexican Board of Pediatric Surgery Certification System.

OBJECTIVE: To evaluate the quality of the Mexican Board of Pediatric Surgery (MBPS) certifying system, using contemporary international guidelines. DESIGN: Retrospective assessment of evidence collected during the design and implementation processes of 2 consecutive applications of the MBPS certifying examination, using Cizek's checklist for evaluating credential-testing programs. It includes the relevant guidelines from the American Education Research Association, the American Psychological Association, the National Council on Measurement in Education, and the National Commission for Certifying Agencies. Four independent and previously trained raters used the checklist. They underwent a 2-week training using frame-of-reference and performance dimensions methodologies. SETTING: Certification examinations of MBPS. POPULATION: The 2013 and 2014 MBPS certification examinations, with 111 evidence items of the processes, followed for the assessment of 86 examinees. RESULTS: The checklist internal consistency was 0.89. Absolute interrater agreement was 0.34 for the 2013 and 0.66 for the 2014 editions of the examination. The 2013 examination complied with 55 (64%) of the checklist 86 items, in 2014 with 72 (84%). CONCLUSION: In 2014, the certifying system reached a quality of 84%. For optimal fulfillment of its social responsibility, the MBPS is required to maintain its level of quality and attempt to improve its performance.

[Diagnostic assessment of knowledge and competencies in medical students at the end of the second year of medical school: the challenge of building an airplane while it's flying]. / Evaluación diagnóstica de conocimientos y competencias en estudiantes de medicina al término del segundo año de la carrera: el reto de construir el avión mientras vuela.

INTRODUCTION: In Latin America there is almost no published information about knowledge retention and formative assessment of competencies in medical students, during medical school training and curricular changes. OBJECTIVE: To assess knowledge level and clinical competencies in medical students at the end of the second year in a new curriculum. METHODS: Observational, cross-sectional study in UNAM Faculty of Medicine students. A diagnostic evaluation was performed in the first class of the "Plan of Studies 2010" curriculum, as part of an integral program evaluation strategy. The assessment had two components: theoretical and practical. RESULTS: 456 (87%) of the 524 students that successfully completed the second year of Plan 2010 were assessed. The written test had 211 items, a mean score of 60 ± 14.5, mean difficulty index of 0.60, reliability with Cronbach's alpha of 0.85. The OSCE mean global score was 58 ± 9, Cronbach's alpha of 0.36, and G-coefficient of 0.48, and results were provided for each station. Results by area of knowledge, course, and station were reported. CONCLUSIONS: The results in general are acceptable, compared with previous written evaluations at the end of the second year, suggesting that the new program is achieving its educational goals. Competencies were formally assessed for the first time in our Institution, establishing a starting point for follow-up. The study provided useful information to the institution, teachers and students.

The resident-as-teacher educational challenge: a needs assessment survey at the National Autonomous University of Mexico Faculty of Medicine.

BACKGROUND: The role of residents as educators is increasingly recognized, since it impacts residents, interns, medical students and other healthcare professionals. A widespread implementation of resident-as-teacher courses in developed countries' medical schools has occurred, with variable results. There is a dearth of information about this theme in developing countries. The National Autonomous University of Mexico (UNAM) Faculty of Medicine has more than 50% of the residency programs' physician population in Mexico. This report describes a needs assessment survey for a resident as teacher program at our institution. METHODS: A cross-sectional descriptive survey was developed based on a review of the available literature and discussion by an expert multidisciplinary committee. The goal was to identify the residents' attitudes, academic needs and preferred educational strategies regarding resident-as-teacher activities throughout the residency. The survey was piloted and modified accordingly. The paper anonymous survey was sent to 7,685 residents, the total population of medical residents in UNAM programs in the country. RESULTS: There was a 65.7% return rate (5,186 questionnaires), a broad and representative sample of the student population. The residents felt they had knowledge and were competent in medical education, but the majority felt a need to improve their knowledge and skills in this discipline. Most residents (92.5%) felt that their role as educators of medical students, interns and other residents was important/very important. They estimated that 45.5% of their learning came from other residents. Ninety percent stated that it was necessary to be trained in teaching skills. The themes identified to include in the educational intervention were mostly clinically oriented. The educational strategies in order of preference were interactive lectures with a professor, small groups with a moderator, material available in a website for self-learning, printed material for self-study and homework, and small group web-based learning. CONCLUSIONS: There is a large unmet need to implement educational interventions to improve residents' educational skills in postgraduate educational programs in developing countries. Most perceived needs of residents are practical and clinically oriented, and they prefer traditional educational strategies. Resident as teachers educational interventions need to be designed taking into account local needs and resources.

[Treatment of hyaline membrane disease in the preterm newborn with exogenous lung surfactant: a controlled study]. / Estudio controlado del tratamiento de la enfermedad de membrana hialina del recién nacido pretérmino con surfactante pulmonar exógeno (porcino vs. bovino).

BACKGROUND: Hyaline membrane disease (HMD) due to lung surfactant deficiency in the preterm newborn is an important cause of neonatal morbidity and mortality. Exogenous lung surfactant has transformed HMD therapy in developed countries, but an equivalent benefit has not been accomplished in developing countries due to a variety of factors. Porcine surfactant developed in Cuba is an inexpensive alternative to other surfactants, and its use has not been studied in our settings. METHODS: A randomized, open, prospective and controlled trial was undertaken in 44 preterm newborns with HMD diagnosis. One group received bovine surfactant (BS) (Survanta) and the other Cuban porcine surfactant (PS) (Surfacen). The following clinical response variables were evaluated: oxygenation and ventilation indexes, days with supplementary oxygen, days with mechanical ventilation, incidence of complications, time of hospitalization, and mortality. RESULTS: 23 Patients received bovine surfactant and 21 the porcine type. The two groups were clinically similar, with patterns of oxygenation and ventilation response that were the same between groups, with a tendency to higher initial oxygenation increase in the PS group. The incidence of complications was similar between groups. 10 Patients (47.6%) died in the PS group, versus 12 (52.2%) in the BS group (p > 0.05). CONCLUSIONS: Porcine surfactant had similar clinical effects than bovine surfactant in the oxygenation and ventilation variables, with no significant differences in complications ormortality. Porcine surfactant is an effective and lower cost altenative to bovine surfactant in the treatment of HMD.

Estudio controlado del tratamiento de la enfermedad de membrana hialina del recién nacido pretérmino con surfactante pulmonar exógeno (porcino vs. bovino) / Treatment of hyaline membrane disease in the preterm newborn with exogenous lung surfanctant. A controlled study

Introducción: La enfermedad de membrana hialina (EMH) por deficiencia de surfactante pulmonar en el neonato prematuro es una causa importante de morbimortalidad. El surfactante pulmonar exógeno ha revolucionado el tratamiento de esta entidad en países desarrollados, aunque este beneficio ha sido menor en países en vías de desarrollo. El surfactante porcino de manufactura cubana es económico, y su uso comparado con otros surfactantes es desconocido. Material y métodos: Se llevó a cabo un estudio prospectivo, controlado, aleatorizado, abierto, en 44 recién nacidos prematuros con EMH. Un grupo recibió surfactante bovino (SB) (Survanta), y el otro surfactante porcino (SP) de fabricación cubana (Surfacen). Se evaluó la respuesta en variables de oxigenación y ventilación, días de oxígeno suplementario, ventilación mecánica, incidencia de complicaciones, tiempo de hospitalización y mortalidad. Resultados: 23 pacientes recibieron el surfactante bovino, y 21 el porcino. Los dos grupos fueron similares clínicamente y en sus patrones de respuesta de oxigenación y ventilación, con una tendencia a mayor incremento inicial en la oxigenación en el grupo tratado con SP. La incidencia de complicaciones fue similar en los dos grupos. Fallecieron 10 pacientes (47.6%) en el grupo SP, y 12 (52.2%) en el grupo SB (p>0.05). Conclusiones: El surfactante porcino tuvo efectos clínicos similares al bovino en las variables de oxigenación y ventilación estudiadas; no hubo diferencia significativa en complicaciones y mortalidad. El surfactante porcino es una alternativa efectiva y de menor costo que el surfactante bovino para el tratamiento de la EMH.

Background: Hyaline membrane disease (HMD) due to lung surfactant deficiency in the preterm newborn is an important cause of neonatal morbidity and mortality. Exogenous lung surfactant has transformed HMD therapy in developed countries, but an equivalent benefit has not been accomplished in developing countries due to a variety of factors. Porcine surfactant developed in Cuba is an inexpensive alternative to other surfactants, and its use has not been studied in our settings. Methods: A randomized, open, prospective and controlled trial was undertaken in 44 preterm newborns with HMD diagnosis. One group received bovine surfactant (BS) (Survanta) and the other Cuban porcine surfactant (PS) (Surfacen). The following clinical response variables were evaluated: oxygenation and ventilation indexes, days with supple mentary oxygen, days with mechanical ventilation, incidence of compli cations, time of hospitalization, and mortality. Results: 23 Patients received bovine surfactant and 21 the porcine type. The two groups were clinically similar, with patterns of oxygenation and ventilation response that were the same between groups, with a tendency to higher initial oxygenation increase in the PS group. The incidence of complications was similar between groups. Ten Patients (47.6%) died in the PS group, versus 12 (52.2%) in the BS group (p>0.05) Conclusions: Porcine surfactant had similar clinical effects than bovine surfactant in the oxygenation and ventilation variables, with no significant differences in complications or mortality. Porcine surfactant is an effective and lower cost alternative to bovine surfactant in the treatment of HMD.

La colaboración Cochrane en México / The Cochrane colaboration in Mexico

El proceso de toma de decisiones en la atención de la salud, tanto a nivel individual como colectivo, requiere actualmente de diversos elementos para asegurar la calidad en el mismo, uno de ellos es la disponibilidad de los productos de la investigación científica biomédica a nivel mundial, que ha alcanzado niveles máximos históricos. La elaboración de revisiones sistemáticas de las intervenciones en la atención de la salud es una de las soluciones más importantes al problema del exceso de publicaciones científicas en las ciencias médicas, ya que están diseñadas para disminuir el sesgo de la síntesis tradicional de la información médica. La Colaboración Cochrane es una organización mundial, llamada así en honor del epidemiólogo inglés Archie Cochrane, que utiliza fundamentalmente esta metodología para preparar, actualizar y divulgar revisiones sistemáticas permanentemente actualizadas sobre los efectos de las intervenciones más efectivas en la atención de la salud; su producto más importante es la Cochrane Library, disponible en CD-ROM e internet. Se hace una reseña histórica de los orígenes de la Colaboración, su misión y metodología de trabajo, así como de los retos a nivel mundial que enfrenta la organización, y el principio y las tareas fundamentales de la Colaboración Cochrane en México.