Reliability, validity and acceptability of an online clinical reasoning simulator for medical students: An international pilot.

INTRODUCTION: Clinical reasoning skills are essential for decision-making. Current assessment methods are limited when testing clinical reasoning and management of uncertainty. This study evaluates the reliability, validity and acceptability of Practicum Script, an online simulation-based programme, for developing medical students' clinical reasoning skills using real-life cases. METHODS: In 2020, we conducted an international, multicentre pilot study using 20 clinical cases with 2457 final-year medical students from 21 schools worldwide. Psychometric analysis was performed (n = 1502 students completing at least 80% of cases). Classical estimates of reliability for three test domains (hypothesis generation, hypothesis argumentation and knowledge application) were calculated using Cronbach's alpha and McDonald's omega coefficients. Validity evidence was obtained by confirmatory factor analysis (CFA) and measurement alignment (MA). Items from the knowledge application domain were analysed using cognitive diagnostic modelling (CDM). Acceptability was evaluated by an anonymous student survey. RESULTS: Reliability estimates were high with narrow confidence intervals. CFA revealed acceptable goodness-of-fit indices for the proposed three-factor model. CDM analysis demonstrated good absolute test fit and high classification accuracy estimates. Student survey responses showed high levels of acceptability. CONCLUSION: Our findings suggest that Practicum Script is a useful resource for strengthening students' clinical reasoning skills and ability to manage uncertainty.

Longitudinal variation of correlations between different components of assessment within a medical school.

BACKGROUND: An assessment program should be inclusive and ensure that the various components of medical knowledge, clinical skills, and professionalism are assessed. The level and the variation over time in the strength of the correlation between these components of assessment is still a matter of study. Based on the meaningful learning theory and the integrated learning theory, we hypothesize that these components increase their connections during the medical school course. METHODS: This is a retrospective cohort study that analyzed data collected for a 10-year period in one medical school. We included students from the 3rd to 6th year of medical school from 2011 to 2021. Three assessment components were addressed: Knowledge, Clinical Skills, and Professionalism. For data analysis, Pearson correlation coefficients (R) and R2 were calculated to study the correlation between variables and a z-test on Fisher's r-to-z was used to determine the differences between correlation coefficients. RESULTS: 949 medical students were included in the study. The correlation between Clinical Skills and Professionalism showed a medium to strong association (Pearson's R ranging from 0.485 to 0.734), while the correlation between Knowledge and Professionalism was weaker but exhibited a steady evolution with Pearson's R fluctuating between 0.075 and 0.218. The Knowledge and Clinical Skills correlation became statistically significant from 2013 onwards and peaking at Pearson's R of 0.440 for the cohort spanning 2016-2019. We also revealed a strengthening of correlations between Professionalism and Clinical Skills from the beginning to the end of clinical training, but not with the knowledge component. CONCLUSIONS: This analysis contributes to our understanding of the dynamics of correlations of different assessment components within an institution and provides a framework for how they interact and influence each other. TRIAL REGISTRATION: This study was not a clinical trial, but a retrospective observational study, without health care interventions. Nevertheless, we provide herein the number of the study as submitted to the Ethics committee - CEICVS 146/2021.

The progress test as a structuring initiative for programmatic assessment.

BACKGROUND: The Progress Test is an individual assessment applied to all students at the same time and on a regular basis. The test was structured in the medical undergraduate education of a conglomerate of schools to structure a programmatic assessment integrated into teaching. This paper presents the results of four serial applications of the progress test and the feedback method to students. METHODS: This assessment comprises 120 items offered online by means of a personal password. Items are authored by faculty, peer-reviewed, and approved by a committee of experts. The items are classified by five major areas, by topics used by the National Board of Medical Examiners and by medical specialties related to a national Unified Health System. The correction uses the Item Response Theory with analysis by the "Rasch" model that considers the difficulty of the item. RESULTS: Student participation increased along the four editions of the tests, considering the number of enrollments. The median performances increased in the comparisons among the sequential years in all tests, except for test1 - the first test offered to schools. Between subsequent years of education, 2nd-1st; 4th-3rd and 5th-4th there was an increase in median scores from progress tests 2 through 4. The final year of undergraduate showed a limited increase compared to the 5th year. There is a consistent increase in the median, although with fluctuations between the observed intervals. CONCLUSION: The progress test promoted the establishment of regular feedback among students, teachers and coordinators and paved the road to engagement much needed to construct an institutional programmatic assessment.

Cognitive diagnostic modelling in healthcare professions education: an eye-opener.

Criticisms about psychometric paradigms currently used in healthcare professions education include claims of reductionism, objectification, and poor compliance with assumptions. Nevertheless, perhaps the most crucial criticism comes from learners' difficulty in interpreting and making meaningful use of summative scores and the potentially detrimental impact these scores have on learners. The term "post-psychometric era" has become popular, despite persisting calls for the sensible use of modern psychometrics. In recent years, cognitive diagnostic modelling has emerged as a new psychometric paradigm capable of providing meaningful diagnostic feedback. Cognitive diagnostic modelling allows the classification of examinees in multiple cognitive attributes. This measurement is obtained by modelling these attributes as categorical, discrete latent variables. Furthermore, items can reflect more than one latent variable simultaneously. The interactions between latent variables can be modelled with flexibility, allowing a unique perspective on complex cognitive processes. These characteristic features of cognitive diagnostic modelling enable diagnostic classification over a large number of constructs of interest, preventing the necessity of providing numerical scores as feedback to test takers. This paper provides an overview of cognitive diagnostic modelling, including an introduction to its foundations and illustrating potential applications, to help teachers be involved in developing and evaluating assessment tools used in healthcare professions education. Cognitive diagnosis may represent a revolutionary new psychometric paradigm, overcoming the known limitations found in frequently used psychometric approaches, offering the possibility of robust qualitative feedback and better alignment with competency-based curricula and modern programmatic assessment frameworks.

Cross-Cultural Adaptation of the Communication Assessment Tool for Use in a Simulated Clinical Setting.

Construct: The Communication Assessment Tool (CAT) is a 14-item instrument developed in English to assess medical trainees' interpersonal communication skills from the patient's perspective in clinical settings. Background: Using validated instruments and simulated patients constitutes good practice in assessing doctor-patient communication. The CAT was designed for use in real practice, but has not yet been applied to assessing OB-GYN residents' delivery of bad news in Objective Structured Clinical Examination (OSCE) stations. This study aims to provide validity evidence for using the CAT to assess residents' interpersonal communication skills under difficult circumstances in a simulated clinical setting in Brazil. Approach: Cross-cultural adaptation comprised translation into Portuguese, synthesis of translations, and back-translation. Next, a committee of 10 external and independent experts rated the items for linguistic equivalence and relevance to the overall scale. Researchers used the expert ratings to produce a preliminary Brazilian-Portuguese version. This version was applied by four simulated patients to assess 28 OB-GYN residents completing two, 10-minute OSCE stations focused on delivering bad news. Item and scale content validity indices and internal-consistency reliability were calculated. Simulated patients were interviewed to clarify any doubt regarding the content and usability of the tool and their response process. Findings: Thirteen of the 14 items in the Brazilian-Portuguese version were considered "equivalent" by at least 70% of the experts. All items were considered relevant by 100% of the experts. The Item Content Validity Index ranged from .9 to 1, and the Scale Content Validity Index was .99. The instrument showed good reliability for both scenarios (Cronbach's alpha > .90). Simulated patients considered the CAT easy to understand and complete. Conclusions: This study provides validity evidence for using the Brazilian-Portuguese CAT in a simulated clinical environment to assess OB-GYN residents' delivery of bad news. Based on this study's findings, the OB-GYN Department organized an annual formative assessment for residents to improve their interpersonal communication skills. This version of the CAT may also be applicable to other specialties.

Validity of the scan of postgraduate educational environment domains (SPEED) questionnaire in a rural general practice training setting.

BACKGROUND: The educational environment is critical to learning and is determined by social interactions. Trainee satisfaction translates to career commitment, retention and a positive professional attitude as well as being an important factor in assessing the impact of the training program. This study aimed to validate the Scan of Postgraduate Educational Environment Domain (SPEED) tool and assess its appropriateness in evaluating the quality of General Practice (GP) rural postgraduate educational environment. METHODS: A questionnaire containing the 15-item SPEED tool was administered to GP registrars to examine their perceptions of the educational environment. Principal component analysis (PCA) and exploratory factor analysis (EFA) were used to gather evidences of the validity of the instrument based on its internal structure. Additional validity evidence and reliability estimates were obtained using many-facet Rasch model analysis (MFRM). RESULTS: The survey was completed by 351 registrars with a response rate of 60%. Parallel analysis performed using principal component analysis and exploratory factor analysis suggests that the SPEED tool is unidimensional. The MFRM analysis demonstrated an excellent degree of infit and outfit for items and training sites, but not for persons. The MFRM analysis also estimated high reliability levels for items (0.98), training sites (0.95) and persons within training sites (ranging from 0.87 to 0.93 in each training sites). Overall, the registrars agreed that the educational environment had high quality, with most (13 out of 15) of the items rated above 4 out of 5. CONCLUSIONS: This study demonstrated a high degree of validity and reliability of the SPEED tool for the measurement of the quality of the educational environment in a rural postgraduate GP training context. However, when applied in a new setting, the tool may not function as a multidimensional tool consistent with its theoretical grounding.

Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis.

BACKGROUND: Progress testing is an assessment tool used to periodically assess all students at the end-of-curriculum level. Because students cannot know everything, it is important that they recognize their lack of knowledge. For that reason, the formula-scoring method has usually been used. However, where partial knowledge needs to be taken into account, the number-right scoring method is used. Research comparing both methods has yielded conflicting results. As far as we know, in all these studies, Classical Test Theory or Generalizability Theory was used to analyze the data. In contrast to these studies, we will explore the use of the Rasch model to compare both methods. METHODS: A 2 × 2 crossover design was used in a study where 298 students from four medical schools participated. A sample of 200 previously used questions from the progress tests was selected. The data were analyzed using the Rasch model, which provides fit parameters, reliability coefficients, and response option analysis. RESULTS: The fit parameters were in the optimal interval ranging from 0.50 to 1.50, and the means were around 1.00. The person and item reliability coefficients were higher in the number-right condition than in the formula-scoring condition. The response option analysis showed that the majority of dysfunctional items emerged in the formula-scoring condition. CONCLUSIONS: The findings of this study support the use of number-right scoring over formula scoring. Rasch model analyses showed that tests with number-right scoring have better psychometric properties than formula scoring. However, choosing the appropriate scoring method should depend not only on psychometric properties but also on self-directed test-taking strategies and metacognitive skills.

Knowledge attainment, learning approaches, and self-perceived study burnout among European veterinary students.

Introduction: This study investigates the relationship between approaches to learning, self-perceived study burnout, and the level of knowledge among veterinary students. Veterinary educational programs are under regular development and would benefit greatly from detailed feedback on students' knowledge, proficiency, influencing factors, and coping mechanisms. Methods: The VetRepos consortium developed and calibrated an item repository testing knowledge across the entire veterinary curriculum. Two hundred forty-eight students from seven European veterinary institutions took the VetRepos test, comprising a subset of the repository. They also responded to a questionnaire assessing deep and unreflective learning approaches and self-perceived study burnout, represented by exhaustion and cynicism. Structural equation modeling analyzed the relationship between these latent traits and the VetRepos test score. Results: The model failed the exact-fit test but was retained based on global fit indices, inter-item residual correlations, and standardized residual covariances. Root Mean Square Error of Approximation with robust standard errors and scaled test statistic was 0.049 (95% confidence interval 0.033-0.071), scaled and robust Comparative Fit Index 0.95 (0.90-0.98), and scaled Standardized Root Mean Square Residual 0.056 (0.049-0.071). Measurement invariance across study years was not violated (ΔCFI = 0.00, χ2 = 3.78, Δdf = 4, p = 0.44), but it could not be confirmed between genders or universities. The VetRepos test score regressed on the study year [standardized regression coefficient = 0.68 (0.62-0.73)], showed a negative regression on the unreflective learning approach [-0.25 (-0.47 to -0.03)], and a positive regression on the deep approach [0.16 (0.03-0.28)]. No direct association with perceived burnout was observed; however, a significant, medium-sized association was found between the unreflective approach and self-perceived study burnout. No significant differences in learning approaches or perceived burnout were found between study years. Discussion: The most important source of variance in VetRepos test scores, unrelated to the study year, was the learning approach. The association between the VetRepos test score and self-perceived burnout was indirect. Future research should complement this cross-sectional approach with longitudinal and person-oriented studies, further investigating the relationship between study burnout and learning approaches.

Measurement of uncertainty tolerance revisited.

INTRODUCTION: Uncertainty tolerance (UT) is attracting increasing attention in medical education due to the numerous challenges associated with uncertainty in professional life. Inconsistencies in analysing the relationship between UT and moderators may arise from inadequate measurement methods. Most instruments were formulated before the most widely accepted framework was published. Our aim was to investigate the validity of an UT scale using an actual framework to corroborate with better and accurate instruments. METHODS: A total of 1052 students were invited. Various psychometric methods were used to explore validity of the TAMSAD scale in light of actual framework. Classic exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) were performed. Secondly, content item classification was triangulated with exploratory graph analysis (EGA), and the new EFA, CFA, and cognitive diagnostic modelling (CDM) analysis were conducted. The reliability was calculated using Cronbach's alpha and McDonald's omega. RESULTS: A total of 694 students (65.9%) responded to the questionnaire. The reliability of the TAMSAD scale was 0.782. The initial EFA revealed no clear interpretable dimensions. The TAMSAD scale items can be classified into sources of uncertainty. The EGA has three dimensions, and the new EFA led to a 17-item TAMSAD scale with the following three dimensions: ambiguity, complexity, and probability. These dimensions lead to better adjustment fit indices in the new CFA and CDM analyses. CONCLUSION: We found evidence that the TAMSAD scale can be considered a multidimensional scale, organised in terms of sources of uncertainty.

Brazilian version of Calgary-Cambridge Observation Guide 28-item version: cross-cultural adaptation and psychometric properties.

OBJECTIVES: The search for appropriate tools to assess communicational skills remains an ongoing challenge. The Calgary-Cambridge Observation Guide (CCOG) 28-item version can measure and compare performance in communication skills training. Our goal was to adapt this version of the CCOG for the Brazilian cultural context and perform a psychometric quality analysis of the instrument. METHODS: Experienced preceptors (35) assessed videos of five medical residents with a simulated patient using the translated guide. For the cultural adaptation, we followed the methodological norms on synthesis, retro-translation, committee review, and testing. We obtained validity evidence for the CCOG 28-item version using confirmatory factor analysis and the Many-Facet Rasch Model (MFRM). RESULTS: Confirmatory factor analysis indicated an adequate level of goodness-of-fit. The MFRM reliability coefficient was high in all facets, namely assessors (0.90), stations (0.99), and items (0.98). The assessors had greater difficulty with attitudinal items, such as demonstration of respect, confidence, and empathy. CONCLUSIONS: The psychometric indicators of the tool were adequate, a good potential for reproducing its Brazilian version as well as acceptable reliability for its use.

The Setting Questionnaire for the Ayahuasca Experience: Questionnaire Development and Internal Structure.

The growing interest in research on psychedelic consumption in naturalistic contexts and their possible medical and therapeutic benefits requires assessment of the relationships between the substance and the individual who consumes it (set) and its context of use (setting). This study provides a novel measurement scale for the setting of Ayahuasca consumption, the Setting Questionnaire for the Ayahuasca Experience (SQAE), and examines its psychometric properties. Construction of the scale began with a literature review, followed by interviews on 19 Ayahuasca users from different backgrounds and different consumption experience, and an online survey for quantitative data collection (n = 2,994). Exploratory Graph Analysis (EGA) was used to investigate the questionnaire's dimensional structure with (n = 1,497, half of the sample), and multidimensional item response theory (MIRT) was used to compare the fit of the theoretical dimensions with the EGA proposed dimensions (n = 1,497, independent other half). EGA identified six dimensions, which corresponded partially to the theorized model (Leadership, Decoration, Infrastructure, Comfort, Instruction, and Social). The MIRT comparison found that the proposed theoretical model fit significantly better than the EGA model, providing support for the former (χ2/df = 1,967; CFI = 0,972; TLI = 0,969; RMSEA = 0,059; WRMR = 1,087). Our findings present evidence of validity of this instrument, justifying its use for future research on the influence of the setting during the ayahuasca experience. Its findings may provide a basis for expanding the settings investigated in the use of psychedelics in general.

Improving the Competency of Nurses.

To decrease infusion pump administration errors, time-consuming training is often initiated. The aims of this study were twofold: to develop minimum competency requirements for programming and operation of infusion pumps and to develop and validate a test for nurses based on those requirements. The test was completed by 226 nurses between May and December 2017. This study demonstrates that testing is a promising method to assess the competency of nurses in using medical devices. Moreover, test acceptability among nurses is high. Using competency requirements to develop a test offers the potential to tailor training needs and reduce training time.

Investigating possible causes of bias in a progress test translation: an one-edged sword.

PURPOSE: Assessment in different languages should measure the same construct. However, item characteristics, such as item flaws and content, may favor one test-taker group over another. This is known as item bias. Although some studies have focused on item bias, little is known about item bias and its association with items characteristics. Therefore, this study investigated the association between item characteristics and bias. METHODS: The University of Groningen offers both an international and a national bachelor's program in medicine. Students in both programs take the same progress test, but the international progress test is literally translated into English from the Dutch version. Differential item functioning was calculated to analyze item bias in four subsequent progress tests. Items were also classified by their categories, number of alternatives, item flaw, item length, and whether it was a case-based question. RESULTS: The proportion of items with bias ranged from 34% to 36% for the various tests. The number of items and the size of their bias was very similar in both programmes. We have identified that the more complex items with more alternatives favored the national students, whereas shorter items and fewer alternatives favored the international students. CONCLUSION: Although nearly 35% of all items contain bias, the distribution and the size of the bias were similar for both groups. The findings of this paper may be used to improve the writing process of the items, by avoiding some characteristics that may benefit one group whilst being a disadvantage for others.

Progress test utopia.

This paper discusses the advantages of progress testing. A utopia is described where medical schools would work together to develop and administer progress testing. This would lead to a significant reduction of cost, an increase in the quality of measurement and phenomenal feedback to learner and school. Progress testing would also provide more freedom and resources for more creative in-school assessment. It would be an educationally attractive alternative for the creation of cognitive licensing exams. A utopia is always far away in the future, but by formulating a vision for that future we may engage in discussions on how to get there.

Brazilian version of Calgary-Cambridge Observation Guide 28-item version: cross-cultural adaptation and psychometric properties

OBJECTIVES: The search for appropriate tools to assess communicational skills remains an ongoing challenge. The Calgary-Cambridge Observation Guide (CCOG) 28-item version can measure and compare performance in communication skills training. Our goal was to adapt this version of the CCOG for the Brazilian cultural context and perform a psychometric quality analysis of the instrument. METHODS: Experienced preceptors (35) assessed videos of five medical residents with a simulated patient using the translated guide. For the cultural adaptation, we followed the methodological norms on synthesis, retro-translation, committee review, and testing. We obtained validity evidence for the CCOG 28-item version using confirmatory factor analysis and the Many-Facet Rasch Model (MFRM). RESULTS: Confirmatory factor analysis indicated an adequate level of goodness-of-fit. The MFRM reliability coefficient was high in all facets, namely assessors (0.90), stations (0.99), and items (0.98). The assessors had greater difficulty with attitudinal items, such as demonstration of respect, confidence, and empathy. CONCLUSIONS: The psychometric indicators of the tool were adequate, a good potential for reproducing its Brazilian version as well as acceptable reliability for its use.

Antiepileptic drugs: a study of 1028 cases registered by the Sao Paulo Intoxication Control Center.

PURPOSE: To illustrate the profile of the cases of intoxications by antiepileptic drugs (AEDs) registered within a large urban area during the period of 1 year. METHODS: A cross-sectional observational study was performed in the Intoxication Control Center of the Sao Paulo City Hospital. We evaluated 6535 medical consults of intoxication cases involving prescription and over-the-counter drugs in 2001. We selected and analyzed the cases involving AEDs. Statistical analysis was performed in order to evaluate the frequency of different drug types and the circumstance of drug exposure. RESULTS: The most frequently observed AEDs were phenobarbital, carbamazepine, diazepam, and clonazepam. The frequency of intoxication cases among different age groups did not significantly differ according to the drug type. The circumstances involving AEDs intoxication were analogous to the circumstances of the cases of intoxication involving all other medications (p>0.05). CONCLUSIONS: There are a large number of intoxications involving AEDs. Imposing a restriction access to AEDs can be deleterious to patients that depend on AEDs, however, a strict program of AEDs distribution should be considered to reduce the cases of AED intoxication.

Avaliação de Habilidades de Comunicação em Ambiente Simulado na Formação Médica: Conceitos, Desafios e Possibilidades / Assessment of Communication Skills in the Simulated Environment of Medical Training: Concepts, Challenges and Possibilities in Medical Education

RESUMO Na comunidade acadêmica internacional, a comunicação eficaz entre profissionais de saúde, pacientes e seus familiares é reconhecida como condição indispensável para a qualidade dos cuidados em saúde. No Brasil, as Diretrizes Curriculares Nacionais do Curso de Graduação em Medicina estabelecem que o egresso deve ser preparado para se comunicar por meio de linguagem verbal e não verbal, com empatia, sensibilidade e interesse, promovendo o cuidado centrado na pessoa e uma relação horizontal e compartilhada com o paciente. Para alcançar esses objetivos, faz-se preciso a implementação de atividades curriculares que promovam o desenvolvimento dessa habilidade e avaliem a sua aquisição durante a graduação em Medicina. A habilidade em se comunicar adequadamente não pode ser aprendida apenas por observação e tende a declinar ao longo do curso. Recomenda-se que seu ensino se estenda de modo coerente a todos os níveis de formação, incluindo os internatos e programas de residência. Na avaliação de habilidades de comunicação, especialistas recomendam que seja instituída uma matriz que permita repetidas oportunidades de avaliação e feedback, reforçando o uso das habilidades mais básicas de entrevista até as mais complexas, como a comunicação de más notícias. Dessa maneira, para um ensino e avaliação eficientes, são necessários métodos e instrumentos com sólida fundamentação teórica. Atividades em ambiente simulado com a participação de pacientes padronizados têm sido amplamente utilizadas para o ensino e a avaliação dessa habilidade durante a consulta clínica. Nesse contexto, programas de desenvolvimento docente são fundamentais para que estratégias eficazes de ensino e avalição sejam implementadas e permitam ao futuro médico a aquisição de habilidades essenciais ao ético exercício profissional. Este artigo propôs-se a uma revisão narrativa sobre avaliação de habilidades de comunicação em ambiente simulado apresentando seus conceitos, desafios e possibilidades. Também aborda aspectos práticos para a organização desse tipo de avaliação.

ABSTRACT In the international academic community, effective communication between health professionals, patients, and their families are recognized as an indispensable condition for the quality of health care. In Brazil, the National Curriculum Guidelines for Undergraduate Medical Schools established that medical students should be prepared to communicate through verbal and non-verbal language, with empathy, sensitivity and interest, promoting patient-centered care, and a horizontal and shared relationship with the patient. To achieve these goals, it is necessary to implement curricular activities that promote the development of this skill and assess its acquisition during undergraduate medical school. Adequate communication skills cannot be learned by observation alone and tend to decline as medical students progress through their medical education. For the assessment of communication skills, experts recommend a framework that allows for repeated assessment and feedback opportunities, reinforcing the use of the most basic interviewing skills to the most complex ones, such as communicating bad news. Thus, for an efficient assessment, methods and instruments with a solid theoretical basis are necessary. Thus, activities in a simulated environment with the participation of standardized patients have been widely used for the teaching and assessment of these skills during the clinical consultation. In this context, faculty development programs are fundamental for effective teaching and assessment strategies to be implemented and allow the future doctors to acquire essential skills for their professional practice. This article proposes a narrative review of communication skills assessment in a simulated environment regarding its concepts, challenges, and possibilities. It also discusses practical aspects for the organization of this type of assessment.