Validity Evidence for a Novel, Comprehensive Bag-Mask Ventilation Assessment Tool.

OBJECTIVE: To develop a comprehensive competency assessment tool for pediatric bag-mask ventilation (pBMV) and demonstrate multidimensional validity evidence for this tool. STUDY DESIGN: A novel pBMV assessment tool was developed consisting of 3 components: a 22-item-based checklist (trichotomized response), global rating scale (GRS, 5-point), and entrustment assessment (4-point). Participants' performance in a realistic simulation scenario was video-recorded and assessed by blinded raters. Multidimensional validity evidence for procedural assessment, including evidence for content, response-process, internal structure, and relation to other variables, was assessed. The scores of each scale were compared with training level. Item-based checklist scores also were correlated with GRS and entrustment scores. RESULTS: Fifty-eight participants (9 medical students, 10 pediatric residents, 18 critical care/neonatology fellows, 21 critical care/neonatology attendings) were evaluated. The pBMV tool was supported by high internal consistency (Cronbach α = 0.867). Inter-rater reliability for the item-based checklist component was acceptable (r = 0.65, P < .0001). The item-based checklist scores differentiated between medical students and other providers (P < .0001), but not by other trainee level. GRS and entrustment scores significantly differentiated between training levels (P < .001). Correlation between skill item-based checklist and GRS was r = 0.489 (P = .0001) and between item-based checklist and entrustment score was r = 0.52 (P < .001). This moderate correlation suggested each component measures pBMV skills differently. The GRS and entrustment scores demonstrated moderate inter-rater reliability (0.42 and 0.46). CONCLUSIONS: We established evidence of multidimensional validity for a novel entrustment-based pBMV competence assessment tool, incorporating global and entrustment-based assessments. This comprehensive tool can provide learner feedback and aid in entrustment decisions as learners progress through training.

The International Council of Ophthalmology Ophthalmic clinical evaluation exercise.

Purpose: Fifteen years after the publication of the Ophthalmic Clinical Evaluation Exercise (OCEX), it was deemed necessary to review and revise it, and to validate it for an international audience of ophthalmologists. This study to revise the OCEX and validate it for international use. Methods: The OCEX rubric was changed to a modified Dreyfus scale; a behavioral descriptor was created for each category. An international panel of ophthalmic educators reviewed the international applicability and appropriateness of the tool. Results: A tool for assessing and giving feedback on four aspects of clinical competence during the ophthalmic consultation (interview skills, examination, interpersonal and communication skills, and case presentation) was revised. The original scoring tool was improved to a new behavioral one, and relevant comments and suggestions from international reviewers were incorporated. The new tool has face and content validity for an international audience. Conclusion: The OCEX is the only tool for workplace assessment and feedback specifically for ophthalmology residents and the ophthalmic consultation. This improved and simplified version will facilitate its use and implementation to diverse programs around the world.

Adaptation, psychometric properties, and implementation of the Mini-CEX in dental clerkship.

BACKGROUND: Workplace-based assessment is a key component of dental-student clerkships, allowing students to demonstrate clinical proficiency. PURPOSE: This study adapts the Mini-Clinical Evaluation Exercise (Mini-CEX) to a dentistry-program clerkship, analyzing the results and examining the psychometric properties of Mini-CEX. METHODS: First, Delphi panel methodology was used to ensure content validity. Mini-CEX was then piloted in the dental-clerkship program, with each student assessed by at least 2 supervisors and a peer student. Subsequently, psychometric properties, acceptability, and observation time were analyzed. RESULTS: The study was conducted between July and November 2019. Overall, 140 Mini-CEX evaluation exercises were carried out on 30 students by 84 supervisors and 56 peers. The adapted instrument was found to be unidimensional, obtaining an acceptable internal consistency (α = 0.74). As the assessor type changed, there were differences in observation time; the medians (Q1-Q3) were 10 minutes (5-15) for supervisors and 30 minutes (20-45) for peer students (P < 0.001). This difference was also observed in assessor perceptions (P < 0.001), with supervisors scoring a median of 6 (6-6.75) and peer students scoring a median of 7 (6-7). No differences were found between supervisor and peer scores. CONCLUSION: The adapted version of Mini-CEX can objectively assess the clinical performance of dental students, achieving validity and reliability values similar to those obtained in the original instrument.

Evaluación para el aprendizaje: diseño e implementación de un mini-CEX en el internado profesional de la carrera de kinesiología / Assessment for learning: design and implementation of the Mini-CEX to the physiotherapy clerkship

Introducción: la evaluación kinesiológica contempla la realización de una rigurosa historia clínica y examen físico que orientan a establecer un diagnóstico, un pronóstico funcional, y es la base del plan de intervención. Esta competencia requiere de experiencia y razonamiento clínico para la toma de buenas decisiones terapéuticas. Los resultados históricos de las evaluaciones de aprendizajes del internado, concluyen que esta competencia tiene bajo nivel de desempeño, por lo que se deben mejorar las estrategias utilizadas para su aprendizaje durante el internado. El mini-CEX, es un instrumento de evaluación para el aprendizaje, que promueve la observación y feedback en el sitio de trabajo. Objetivo: diseñar e implementar un instrumento de evaluación para el aprendizaje de la evaluación kinesiológica durante el internado profesional de la Carrera de Kinesiología de la PUC. Método: estudio prospectivo, utilizando la me-todología de investigación acción, diseñado en 4 etapas. Resultados: se obtuvo un instrumento de evaluación para el aprendizaje, el mini-CEX, adaptado a las necesidades del internado profesional de kinesiología. El estudio piloto reportó buena aceptación entre estu-diantes y docentes. La percepción de los internos es que son más observados y que las destrezas asociadas a la evaluación kinesiológica mejoraron luego del estudio. Conclusiones: el instrumento reportó un tiempo de aplicación bajo (35 min.), lo que beneficiaría instancias de observación directa y feedback dentro de los internados. Es necesaria una aplicación a gran escala, que contribuya a determinar su validez y confiabilidad. Se requiere una capacitación formal a los docentes evaluadores y ajustar la escala.

Introduction: The physiotherapy clinical assessment competence includes the performance of a rigorous clinical history and physical examination that guides the establishment of a diagnosis and a functional prognosis, and is the basis of the intervention plan. This competence requires experience and clinical reasoning to make good therapeutic decisions. The historical global clerkship assessments conclude that this competency has a low level of performance, so the strategies used for its learning the clerkship must be improved. The mini-CEX is an assessment for learning, which promotes observation and feedback in the workplace. Objective: To design and im-plement an assessment for learning mini-CEX to the Physiotherapy PUC clerkship. Method: Prospective study, using an action research methodology, designed in 4 stages. Results: The main result was the mini-CEX, adapted to the Physiotherapy clerkship needs. The pilot study reported good acceptance among students and teachers. In addition, clerks' perception is that they feel more observed and that the skills associated with the physiotherapy assessment improved after its implementation.