Increasing Authenticity of Simulation-Based Assessment in Diagnostic Radiology.

INTRODUCTION: Clinical reasoning in diagnostic imaging professions is a complex skill that requires processing of visual information and image manipulation skills. We developed a digital simulation-based test method to increase authenticity of image interpretation skill assessment. METHODS: A digital application, allowing volumetric image viewing and manipulation, was used for three test administrations of the national Dutch Radiology Progress Test for residents. This study describes the development and implementation process in three phases. To assess authenticity of the digital tests, perceived image quality and correspondence to clinical practice were evaluated and compared with previous paper-based tests (PTs). Quantitative and qualitative evaluation results were used to improve subsequent tests. RESULTS: Authenticity of the first digital test was not rated higher than the PTs. Test characteristics and environmental conditions, such as image manipulation options and ambient lighting, were optimized based on participants' comments. After adjustments in the third digital test, participants favored the image quality and clinical correspondence of the digital image questions over paper-based image questions. CONCLUSIONS: Digital simulations can increase authenticity of diagnostic radiology assessments compared with paper-based testing. However, authenticity does not necessarily increase with higher fidelity. It can be challenging to simulate the image interpretation task of clinical practice in a large-scale assessment setting, because of technological limitations. Optimizing image manipulation options, the level of ambient light, time limits, and question types can help improve authenticity of simulation-based radiology assessments.

The Effect of Teaching Search Strategies on Perceptual Performance.

RATIONALE AND OBJECTIVES: Radiology expertise is dependent on the use of efficient search strategies. The aim of this study is to investigate the effect of teaching search strategies on trainee's accuracy in detecting lung nodules at computed tomography. MATERIALS AND METHODS: Two search strategies, "scanning" and "drilling," were tested with a randomized crossover design. Nineteen junior radiology residents were randomized into two groups. Both groups first completed a baseline lung nodule detection test allowing a free search strategy, followed by a test after scanning instruction and drilling instruction or vice versa. True positive (TP) and false positive (FP) scores and scroll behavior were registered. A mixed-design analysis of variance was applied to compare the three search conditions. RESULTS: Search strategy instruction had a significant effect on scroll behavior, F(1.3) = 54.2, P < 0.001; TP score, F(2) = 16.1, P < 0.001; and FP score, F(1.3) = 15.3, P < 0.001. Scanning instruction resulted in significantly lower TP scores than drilling instruction (M = 10.7, SD = 5.0 versus M = 16.3, SD = 5.3), t(18) = 4.78, P < 0.001; or free search (M = 15.3, SD = 4.6), t(18) = 4.44, P < 0.001. TP scores for drilling did not significantly differ from free search. FP scores for drilling (M = 7.3, SD = 5.6) were significantly lower than for free search (M = 12.5, SD = 7.8), t(18) = 4.86, P < 0.001. CONCLUSIONS: Teaching a drilling strategy is preferable to teaching a scanning strategy for finding lung nodules.

Tests, Quizzes, and Self-Assessments: How to Construct a High-Quality Examination.

OBJECTIVE: The purposes of this article are to highlight aspects of tests that increase or decrease their effectiveness and to provide guidelines for constructing high-quality tests in radiology. CONCLUSION: Many radiologists help construct tests for a variety of purposes. Only well-constructed tests can provide reliable and valuable information about the test taker.

Volumetric CT-images improve testing of radiological image interpretation skills.

RATIONALE AND OBJECTIVES: Current radiology practice increasingly involves interpretation of volumetric data sets. In contrast, most radiology tests still contain only 2D images. We introduced a new testing tool that allows for stack viewing of volumetric images in our undergraduate radiology program. We hypothesized that tests with volumetric CT-images enhance test quality, in comparison with traditional completely 2D image-based tests, because they might better reflect required skills for clinical practice. MATERIALS AND METHODS: Two groups of medical students (n=139; n=143), trained with 2D and volumetric CT-images, took a digital radiology test in two versions (A and B), each containing both 2D and volumetric CT-image questions. In a questionnaire, they were asked to comment on the representativeness for clinical practice, difficulty and user-friendliness of the test questions and testing program. Students' test scores and reliabilities, measured with Cronbach's alpha, of 2D and volumetric CT-image tests were compared. RESULTS: Estimated reliabilities (Cronbach's alphas) were higher for volumetric CT-image scores (version A: .51 and version B: .54), than for 2D CT-image scores (version A: .24 and version B: .37). Participants found volumetric CT-image tests more representative of clinical practice, and considered them to be less difficult than volumetric CT-image questions. However, in one version (A), volumetric CT-image scores (M 80.9, SD 14.8) were significantly lower than 2D CT-image scores (M 88.4, SD 10.4) (p<.001). The volumetric CT-image testing program was considered user-friendly. CONCLUSION: This study shows that volumetric image questions can be successfully integrated in students' radiology testing. Results suggests that the inclusion of volumetric CT-images might improve the quality of radiology tests by positively impacting perceived representativeness for clinical practice and increasing reliability of the test.

Volumetric and two-dimensional image interpretation show different cognitive processes in learners.

RATIONALE AND OBJECTIVES: In current practice, radiologists interpret digital images, including a substantial amount of volumetric images. We hypothesized that interpretation of a stack of a volumetric data set demands different skills than interpretation of two-dimensional (2D) cross-sectional images. This study aimed to investigate and compare knowledge and skills used for interpretation of volumetric versus 2D images. MATERIALS AND METHODS: Twenty radiology clerks were asked to think out loud while reading four or five volumetric computed tomography (CT) images in stack mode and four or five 2D CT images. Cases were presented in a digital testing program allowing stack viewing of volumetric data sets and changing views and window settings. Thoughts verbalized by the participants were registered and coded by a framework of knowledge and skills concerning three components: perception, analysis, and synthesis. The components were subdivided into 16 discrete knowledge and skill elements. A within-subject analysis was performed to compare cognitive processes during volumetric image readings versus 2D cross-sectional image readings. RESULTS: Most utterances contained knowledge and skills concerning perception (46%). A smaller part involved synthesis (31%) and analysis (23%). More utterances regarded perception in volumetric image interpretation than in 2D image interpretation (Median 48% vs 35%; z = -3.9; P < .001). Synthesis was less prominent in volumetric than in 2D image interpretation (Median 28% vs 42%; z = -3.9; P < .001). No differences were found in analysis utterances. CONCLUSIONS: Cognitive processes in volumetric and 2D cross-sectional image interpretation differ substantially. Volumetric image interpretation draws predominantly on perceptual processes, whereas 2D image interpretation is mainly characterized by synthesis. The results encourage the use of volumetric images for teaching and testing perceptual skills.

Feedback providers' credibility impacts students' satisfaction with feedback and delayed performance.

PURPOSE: Medical students receive feedback during clerkships from many different sources: attendings, residents, paramedics, other clerks and even patients. Not all feedback providers have similar impact on learning. One characteristic that is believed to have impact is their credibility to the recipient. This study investigates the effects of feedback provider credibility on medical student satisfaction, self-efficacy and performance with a trained skill. METHODS: A single-blind randomized controlled between-subjects design was used, with feedback provider credibility (high-low) as independent variable and examination of hearing abilities as the task. First year medical students' (n = 68) satisfaction, self-efficacy and performance were the dependent variables and were measured both directly after the intervention and after a three-week delay. RESULTS: Credibility did not significantly affect immediate or delayed self-efficacy. Students receiving feedback from a high-credibility source were more satisfied with the feedback. They did not perform significantly better immediately after the feedback intervention, but did so three weeks after the intervention. High credibility was associated with a perception of a negative feedback message and an unsocial feedback provider. CONCLUSIONS: Feedback provider credibility impacts satisfaction with feedback and delayed performance. If feedback is not effective in clinical settings, feedback providers may reconsider their credibility.

A case for competency-based anaesthesiology training with entrustable professional activities: an agenda for development and research.

Competency frameworks are based on what are considered to be the general essential qualities of a doctor. Competencies, being behavioural descriptors, need a strong link to clinical practice to allow trainers to observe and then use them in assessing trainees' performance. The emerging concept of entrustable professional activities (EPAs) may serve as such a link. An EPA is a description of an essential clinical task that frames competencies in the context of clinical practice. A full set of EPAs defines a specialty and constitutes the curriculum of specialty training. After observation of satisfactory performance on an EPA, the resident should be permitted to perform that activity without direct supervision. The terms of this should allow a trainer to provide justification for this decision. This makes graded assumption of responsibilities possible. We describe the potential benefits of working with EPAs in anaesthesiology training and set an agenda for curriculum development and research in this area.

Framing of feedback impacts student's satisfaction, self-efficacy and performance.

Feedback is considered important to acquire clinical skills. Research evidence shows that feedback does not always improve learning and its effects may be small. In many studies, a variety of variables involved in feedback provision may mask either one of their effects. E.g., there is reason to believe that the way oral feedback is framed may affect its effect if other variables are held constant. In a randomised controlled trial we investigated the effect of positively and negatively framed feedback messages on satisfaction, self-efficacy, and performance. A single blind randomised controlled between-subject design was used, with framing of the feedback message (positively-negatively) as independent variable and examination of hearing abilities as the task. First year medical students' (n = 59) satisfaction, self-efficacy, and performance were the dependent variables and were measured both directly after the intervention and after a 2 weeks delay. Students in the positively framed feedback condition were significantly more satisfied and showed significantly higher self-efficacy measured directly after the performance. Effect sizes found were large, i.e., partial η (2) = 0.43 and η (2) = 0.32 respectively. They showed a better performance throughout the whole study. Significant performance differences were found both at the initial performance and when measured 2 weeks after the intervention: effects were of medium size, respectively r = -.31 and r = -.32. Over time in both conditions performance and self-efficacy decreased. Framing the feedback message in either a positive or negative manner affects students' satisfaction and self-efficacy directly after the intervention be it that these effects seem to fade out over time. Performance may be enhanced by positive framing, but additional studies need to confirm this. We recommend using a positive frame when giving feedback on clinical skills.

A comparison of medical and pharmacy students' knowledge and skills of pharmacology and pharmacotherapy.

AIM: Pharmacotherapy might be improved if future pharmacists and physicians receive a joint educational programme in pharmacology and pharmacotherapeutics. This study investigated whether there are differences in the pharmacology and pharmacotherapy knowledge and skills of pharmacy and medical students after their undergraduate training. Differences could serve as a starting point from which to develop joint interdisciplinary educational programmes for better prescribing. METHODS: In a cross-sectional design, the knowledge and skills of advanced pharmacy and medical students were assessed, using a standardized test with three domains (basic pharmacology knowledge, clinical or applied pharmacology knowledge and pharmacotherapy skills) and eight subdomains (pharmacodynamics, pharmacokinetics, interactions and side-effects, Anatomical Therapeutic Chemical Classification groups, prescribing, prescribing for special groups, drug information, regulations and laws, prescription writing). RESULTS: Four hundred and fifty-one medical and 151 pharmacy students were included between August 2010 and July 2012. The response rate was 81%. Pharmacy students had better knowledge of basic pharmacology than medical students (77.0% vs. 68.2% correct answers; P < 0.001, δ = 0.88), whereas medical students had better skills than pharmacy students in writing prescriptions (68.6% vs. 50.7%; P < 0.001, δ = 0.57). The two groups of students had similar knowledge of applied pharmacology (73.8% vs. 72.2%, P = 0.124, δ = 0.15). CONCLUSIONS: Pharmacy students have better knowledge of basic pharmacology, but not of the application of pharmacology knowledge, than medical students, whereas medical students are better at writing prescriptions. Professional differences in knowledge and skills therefore might well stem from their undergraduate education. Knowledge of these differences could be harnessed to develop a joint interdisciplinary education for both students and professionals.

[Career preferences among medical students]. / Beroepsvoorkeuren van studenten geneeskunde.

OBJECTIVE: Research on the preference of medical specialty among medical students in the Netherlands and the attractiveness of aspects of the medical profession during the period 2009-2013. DESIGN: Retrospective, descriptive research. METHOD: Data from medical students in the Netherlands who participated in the computer programme Inventory Medical Professionals Choice (IMBK) were analyzed with respect to their preference of medical specialty and the attractiveness of various aspects of the medical profession. The IMBK programme was available free of charge through the Royal Dutch Medical Association (KNMG) website 'Arts in Spe' (Future Physician) during the period 2009-2013. The content of the IMBK programme was based on the questionnaire from the medical profile book developed by the pharmaceutical company GlaxoSmithKline (GSK). RESULTS: General practice was the most popular specialty, particularly among female medical students, with interest increasing during the undergraduate medical curriculum. Hardly any students were interested in insurance medicine, occupational medicine and elderly medicine. Direct patient care was the most attractive professional aspect for medical students. Female students were more attracted to direct and prolonged patient contact than their male counterparts. The number of hours students wished to work in future declined during the course of the undergraduate curriculum, and women were more inclined to prefer regular working hours with adequate leisure time than men. CONCLUSION: During the course of the undergraduate medical curriculum, medical students changed their preference for medical specialty. Major differences exist between male and female students in terms of preference of medical specialty and attractiveness of aspects of the medical profession.

Online formative tests linked to microlectures improving academic achievement.

BACKGROUND: Online formative tests (OFTs) are powerful tools to direct student learning behavior, especially when enriched with specific feedback. AIM: In the present study, we have investigated the effect of OFTs enriched with hyperlinks to microlectures on examination scores. METHODS: OFTs, available one week preceding each midterm and the final exams, could be used voluntarily. The use of OFTs was related to scores on midterm and final exams using ANOVA, with prior academic achievement as a covariate. RESULTS: On average, 74% of all students used the online formative tests (OFT+) while preparing for the summative midterm exam. OFT+ students obtained significantly higher grades compared to OFT-students, both without and with correction for previous academic achievement. Two out of three final exam scores did not significantly improve. CONCLUSION: Students using online formative tests linked to microlectures receive higher grades especially in highly aligned summative tests.

Why receiving feedback collides with self determination.

Providing feedback to trainees in clinical settings is considered important for development and acquisition of skill. Despite recommendations how to provide feedback that have appeared in the literature, research shows that its effectiveness is often disappointing. To understand why receiving feedback is more difficult than it appears, this paper views the feedback process through the lens of Self-Determination Theory (SDT). SDT claims that the development and maintenance of intrinsic motivation, associated with effective learning, requires feelings of competence, autonomy and relatedness. These three psychological needs are not likely to be satisfied in most feedback procedures. It explains why feedback is often less effective than one would expect. Suggestions to convey feedback in ways that may preserve the trainee's autonomy are provided.

Geriatric pharmacology and pharmacotherapy education for health professionals and students: a systematic review.

WHAT IS ALREADY KNOWN ABOUT THIS SUBJECT: The rate of medication errors is high, and these errors can cause adverse drug reactions. Elderly individuals are most vulnerable to adverse drug reactions. One cause of medication errors is the lack of drug knowledge on the part of different health professionals. Medical curricula have changed in recent years, resulting in less education in the basic sciences, such as pharmacology. WHAT THIS STUDY ADDS: Our study shows that little curricular time is devoted to geriatric pharmacology and that educational programmes in geriatric pharmacology have not been thoroughly evaluated. While interest in pharmacology education has increased recently, this is not the case for geriatric pharmacology education. Education on geriatric pharmacology should have more attention in the curricula of health professionals, given the often complex pharmacotherapy in elderly patients. Educational topics should be related to the known risk factors of medication errors, such as polypharmacy, dose adjustments in organ dysfunction and psychopharmacotherapeutics. AIMS: Given the reported high rates of medication errors, especially in elderly patients, we hypothesized that current curricula do not devote enough time to the teaching of geriatric pharmacology. This review explores the quantity and nature of geriatric pharmacology education in undergraduate and postgraduate curricula for health professionals. METHODS: Pubmed, Embase and PsycINFO databases were searched (from 1 January 2000 to 11 January 2011), using the terms 'pharmacology' and 'education' in combination. Articles describing content or evaluation of pharmacology education for health professionals were included. Education in general and geriatric pharmacology was compared. RESULTS: Articles on general pharmacology education (252) and geriatric pharmacology education (39) were included. The number of publications on education in general pharmacology, but not geriatric pharmacology, has increased over the last 10 years. Articles on undergraduate and postgraduate education for 12 different health disciplines were identified. A median of 24 h (from 15 min to 4956 h) devoted to pharmacology education and 2 h (1-935 h) devoted to geriatric pharmacology were reported. Of the articles on education in geriatric pharmacology, 61.5% evaluated the teaching provided, mostly student satisfaction with the course. The strength of findings was low. Similar educational interventions were not identified, and evaluation studies were not replicated. CONCLUSIONS: Recently, interest in pharmacology education has increased, possibly because of the high rate of medication errors and the recognized importance of evidence-based medical education. Nevertheless, courses on geriatric pharmacology have not been evaluated thoroughly and none can be recommended for use in training programmes. Suggestions for improvements in education in general and geriatric pharmacology are given.

Effects of two different instructional formats on scores and reliability of a script concordance test.

The script concordance test (SCT) is designed to assess clinical reasoning by adapting the likelihood of a case diagnosis, based on provided new information. In the standard instructions students are asked to exclude alternative diagnoses they have in mind when answering the questions, but it might be more authentic to include these. Fifty-nine final-year medical students completed an SCT. Twenty-nine were asked to take their differential diagnosis into account (adapted instructions). Thirty students were asked not to consider other diagnoses (standard instructions). All participants were asked to indicate for each question whether they were confused answering it with the given instructions ('confusion indication'). Mean score of the test with the adapted instructions was 81.5 (SD 3.8) and of the test with the standard instructions 82.9 (SD 5.0) (p = 0.220). Cronbach's alpha was 0.39 for the adapted instructions and 0.66 for the standard instructions. The mean number of confusion indications was 4.2 (SD 4.4) per student for the adapted instructions and 16.7 (SD 28.5) for the standard instructions (p = 0.139). Our attempt to improve SCTs reliability by modifying the instructions did not lead to a higher alpha; therefore we do not recommend this change in the instructional format.

Vertical integration in medical school: effect on the transition to postgraduate training.

OBJECTIVES: Recently, many medical schools' curricula have been revised so that they represent vertically integrated (VI) curricula. Important changes include: the provision of earlier clinical experience; longer clerkships, and the fostering of increasing levels of responsibility. One of the aims of vertical integration is to facilitate the transition to postgraduate training. The purpose of the present study is to determine whether a VI curriculum at medical school affects the transition to postgraduate training in a positive way. METHODS: We carried out a questionnaire study among graduates of six medical schools in the Netherlands, who had followed either a VI or a non-VI curriculum. Items in the questionnaire focused on preparedness for work and postgraduate training, the time and number of applications required to be admitted to residency, and the process of making career choices. RESULTS: In comparison with those who have followed non-VI programmes, graduates of VI curricula appear to make definitive career choices earlier, need less time and fewer applications to obtain residency positions and feel more prepared for work and postgraduate training. CONCLUSIONS: The curriculum at medical school affects the transition to postgraduate training. Additional research is required to determine which components of the curriculum cause this effect and to specify under which conditions this effect occurs.

The influence of early clinical experiences on career preference of male and female medical students.

BACKGROUND: Clinical experience is considered to affect medical students' career preferences. It is not known whether the sequence of the clinical rotations influences these preferences. AIM: To explore whether the first clinical clerkship has more impact on career preference than the second by examining the association between the first clinical clerkship and the choice of an elective sixth-year internship. METHOD: University Medical Center Utrecht students are assigned to either a surgical or a medical ward for the first third-year clerkship and to the other ward for the second clerkship. In a retrospective cohort study, internship data of 488 sixth-year students were related to their first clerkship 3 years earlier. RESULTS: For the group as a whole, no association was found between third-year clerkship and sixth-year internship. However, male students who had been assigned to surgery first more often chose a surgical internship than those who had been assigned to medical clerkship first and vice versa (p < 0.02). Within the female subgroup, no association was found. CONCLUSION: A positive association between the nature of the clerkship and the sixth-year internship preference among male students suggest that the first clinical experience can affect later specialty preference.

[A new, problem oriented medicine curriculum in Utrecht: less basic science knowledge]. / Een nieuw, probleemgeoriënteerd geneeskundecurriculum in Utrecht: minder kennis van de basisvakken.

OBJECTIVE: To investigate whether the transition from a conventional, discipline-based curriculum to a problem-orientated, integrated curriculum at the University Medical Center Utrecht, the Netherlands, has resulted in students having less knowledge of the basic medical sciences. DESIGN: Comparative. METHOD: The difference in the amount of basic science between the curricula was quantitatively assessed. 37 final-year students in each curriculum volunteered to complete a test specifically designed to measure knowledge of the basic sciences, a few weeks before their graduation. RESULTS: The transition from the old to the new curriculum resulted in a decline of almost half in the amount of time dedicated to the basic sciences, from 84 to 48 'fulltime week equivalents'. Students in the old curriculum performed significantly better on the test than students in the new curriculum, with 43.2% (SD: 9.56) correct answers versus 35.8% (SD: 8.19) correct answers respectively, which amounted to an effect size of 0.828 (Cohen-d). Yet, on the pathophysiology/pathology subscale, students in each curriculum showed similar performance: 36.1% (SD: 11.55) correct answers for students in the old curriculum, versus 37.2% (SD: 11.66) correct answers for students in the new curriculum. CONCLUSION: Students in the old curriculum had overall significantly more knowledge of the basic sciences than students in the new curriculum, except for pathophysiology/pathology, though the time devoted to this discipline in the new curriculum had also decreased considerably.

The effect of the APLS-course on self-efficacy and its relationship to behavioural decisions in paediatric resuscitation.

AIMS: Self-efficacy may predict performance following life-support training but may be negatively influenced by experiences during training. To investigate both this and the use of self-efficacy in self-assessment we investigated the relationship between self-efficacy and measured performance during a simulated resuscitation, and the effect of death of a simulated patient on self-efficacy. MATERIALS AND METHODS: Consultant and trainee paediatricians and anaesthesiologists scored their self-efficacy for paediatric resuscitation skills before taking an unannounced simulated resuscitation test and objective structured clinical examination (OSCE)-tests of chest compressions and bag- and mask-ventilation. Performance in the simulation was scored by three independent expert observers and the OSCE's using a modified Berden and ventilation penalty scores. RESULTS: Self-efficacy for the relevant skill was significantly higher in doctors choosing to give chest compressions, to intubate or insert an intraosseous device and in those who decided to intubate early. Self-efficacy correlated moderately with the quality of global performance on the simulation but not with the OSCE scores, nor was quality of individual skills during the simulation related to self-efficacy. Self-efficacy was higher in doctors who had taken the Advanced Paediatric Life Support (APLS)-course. Death of the simulated patient had a negative effect on self-efficacy. CONCLUSION: Self-efficacy seems to be predictive of certain actions during a simulated resuscitation but does not correlate with quality of performance of resuscitation skills. Self-efficacy might therefore be useful as a predictor of the application of learning, but cannot be recommended for self-assessment. There is evidence to support the unwritten rule during simulation training the patient should not be allowed to die.

Growth of analytical thinking skills over time as measured with the MATCH test.

CONTEXT: Ber's Comprehensive Integrative Puzzle aims to assess analytical clinical thinking in medical students. We developed a paediatric version, the MATCH test, in which we added two irrelevant options to each question in order to reduce guessing behaviour. We tested its construct validity and studied the development of integrative skills over time. METHODS: We administered a test (MATCH 1) to subjects from two universities, both with a 6-year medical training course. Subjects included 30 students from university 1 who had completed a paediatric clerkship in Year 4, 23 students from university 2 who had completed a paediatric clerkship in Year 5, 13 students from both universities who had completed an advanced paediatric clerkship in Year 6, 28 paediatric residents and 17 paediatricians. We repeated this procedure using a second test with different domains in a new, comparable group of subjects (MATCH 2). RESULTS: Mean MATCH 1 scores for the respective groups were: Year 4 students: 61.2% (standard deviation [SD] 1.3); Year 5 students: 71.3% (SD 1.6); Year 6 students: 76.2% (SD 1.5); paediatric residents: 88.5% (SD 0.7), and paediatricians: 92.2% (SD 1.1) (one-way ANOVA F = 104.00, P < 0.0001). Students of both universities had comparable scores. MATCH 1 and 2 scores were comparable. Cronbach's alpha-values in MATCH 1 and 2 were 0.92 and 0.91, respectively, for all subjects, and 0.82 and 0.87, respectively, for all students. CONCLUSIONS: Analytical clinical thinking develops over time, independently of the factual content of the course. This implies that shortened medical training programmes could produce less skilled graduates.