Tests, Quizzes, and Self-Assessments: How to Construct a High-Quality Examination.

OBJECTIVE: The purposes of this article are to highlight aspects of tests that increase or decrease their effectiveness and to provide guidelines for constructing high-quality tests in radiology. CONCLUSION: Many radiologists help construct tests for a variety of purposes. Only well-constructed tests can provide reliable and valuable information about the test taker.

Framing of feedback impacts student's satisfaction, self-efficacy and performance.

Feedback is considered important to acquire clinical skills. Research evidence shows that feedback does not always improve learning and its effects may be small. In many studies, a variety of variables involved in feedback provision may mask either one of their effects. E.g., there is reason to believe that the way oral feedback is framed may affect its effect if other variables are held constant. In a randomised controlled trial we investigated the effect of positively and negatively framed feedback messages on satisfaction, self-efficacy, and performance. A single blind randomised controlled between-subject design was used, with framing of the feedback message (positively-negatively) as independent variable and examination of hearing abilities as the task. First year medical students' (n = 59) satisfaction, self-efficacy, and performance were the dependent variables and were measured both directly after the intervention and after a 2 weeks delay. Students in the positively framed feedback condition were significantly more satisfied and showed significantly higher self-efficacy measured directly after the performance. Effect sizes found were large, i.e., partial η (2) = 0.43 and η (2) = 0.32 respectively. They showed a better performance throughout the whole study. Significant performance differences were found both at the initial performance and when measured 2 weeks after the intervention: effects were of medium size, respectively r = -.31 and r = -.32. Over time in both conditions performance and self-efficacy decreased. Framing the feedback message in either a positive or negative manner affects students' satisfaction and self-efficacy directly after the intervention be it that these effects seem to fade out over time. Performance may be enhanced by positive framing, but additional studies need to confirm this. We recommend using a positive frame when giving feedback on clinical skills.

A case for competency-based anaesthesiology training with entrustable professional activities: an agenda for development and research.

Competency frameworks are based on what are considered to be the general essential qualities of a doctor. Competencies, being behavioural descriptors, need a strong link to clinical practice to allow trainers to observe and then use them in assessing trainees' performance. The emerging concept of entrustable professional activities (EPAs) may serve as such a link. An EPA is a description of an essential clinical task that frames competencies in the context of clinical practice. A full set of EPAs defines a specialty and constitutes the curriculum of specialty training. After observation of satisfactory performance on an EPA, the resident should be permitted to perform that activity without direct supervision. The terms of this should allow a trainer to provide justification for this decision. This makes graded assumption of responsibilities possible. We describe the potential benefits of working with EPAs in anaesthesiology training and set an agenda for curriculum development and research in this area.

Feedback providers' credibility impacts students' satisfaction with feedback and delayed performance.

PURPOSE: Medical students receive feedback during clerkships from many different sources: attendings, residents, paramedics, other clerks and even patients. Not all feedback providers have similar impact on learning. One characteristic that is believed to have impact is their credibility to the recipient. This study investigates the effects of feedback provider credibility on medical student satisfaction, self-efficacy and performance with a trained skill. METHODS: A single-blind randomized controlled between-subjects design was used, with feedback provider credibility (high-low) as independent variable and examination of hearing abilities as the task. First year medical students' (n = 68) satisfaction, self-efficacy and performance were the dependent variables and were measured both directly after the intervention and after a three-week delay. RESULTS: Credibility did not significantly affect immediate or delayed self-efficacy. Students receiving feedback from a high-credibility source were more satisfied with the feedback. They did not perform significantly better immediately after the feedback intervention, but did so three weeks after the intervention. High credibility was associated with a perception of a negative feedback message and an unsocial feedback provider. CONCLUSIONS: Feedback provider credibility impacts satisfaction with feedback and delayed performance. If feedback is not effective in clinical settings, feedback providers may reconsider their credibility.

A comparison of medical and pharmacy students' knowledge and skills of pharmacology and pharmacotherapy.

AIM: Pharmacotherapy might be improved if future pharmacists and physicians receive a joint educational programme in pharmacology and pharmacotherapeutics. This study investigated whether there are differences in the pharmacology and pharmacotherapy knowledge and skills of pharmacy and medical students after their undergraduate training. Differences could serve as a starting point from which to develop joint interdisciplinary educational programmes for better prescribing. METHODS: In a cross-sectional design, the knowledge and skills of advanced pharmacy and medical students were assessed, using a standardized test with three domains (basic pharmacology knowledge, clinical or applied pharmacology knowledge and pharmacotherapy skills) and eight subdomains (pharmacodynamics, pharmacokinetics, interactions and side-effects, Anatomical Therapeutic Chemical Classification groups, prescribing, prescribing for special groups, drug information, regulations and laws, prescription writing). RESULTS: Four hundred and fifty-one medical and 151 pharmacy students were included between August 2010 and July 2012. The response rate was 81%. Pharmacy students had better knowledge of basic pharmacology than medical students (77.0% vs. 68.2% correct answers; P < 0.001, δ = 0.88), whereas medical students had better skills than pharmacy students in writing prescriptions (68.6% vs. 50.7%; P < 0.001, δ = 0.57). The two groups of students had similar knowledge of applied pharmacology (73.8% vs. 72.2%, P = 0.124, δ = 0.15). CONCLUSIONS: Pharmacy students have better knowledge of basic pharmacology, but not of the application of pharmacology knowledge, than medical students, whereas medical students are better at writing prescriptions. Professional differences in knowledge and skills therefore might well stem from their undergraduate education. Knowledge of these differences could be harnessed to develop a joint interdisciplinary education for both students and professionals.

Why receiving feedback collides with self determination.

Providing feedback to trainees in clinical settings is considered important for development and acquisition of skill. Despite recommendations how to provide feedback that have appeared in the literature, research shows that its effectiveness is often disappointing. To understand why receiving feedback is more difficult than it appears, this paper views the feedback process through the lens of Self-Determination Theory (SDT). SDT claims that the development and maintenance of intrinsic motivation, associated with effective learning, requires feelings of competence, autonomy and relatedness. These three psychological needs are not likely to be satisfied in most feedback procedures. It explains why feedback is often less effective than one would expect. Suggestions to convey feedback in ways that may preserve the trainee's autonomy are provided.

Online formative tests linked to microlectures improving academic achievement.

BACKGROUND: Online formative tests (OFTs) are powerful tools to direct student learning behavior, especially when enriched with specific feedback. AIM: In the present study, we have investigated the effect of OFTs enriched with hyperlinks to microlectures on examination scores. METHODS: OFTs, available one week preceding each midterm and the final exams, could be used voluntarily. The use of OFTs was related to scores on midterm and final exams using ANOVA, with prior academic achievement as a covariate. RESULTS: On average, 74% of all students used the online formative tests (OFT+) while preparing for the summative midterm exam. OFT+ students obtained significantly higher grades compared to OFT-students, both without and with correction for previous academic achievement. Two out of three final exam scores did not significantly improve. CONCLUSION: Students using online formative tests linked to microlectures receive higher grades especially in highly aligned summative tests.

Geriatric pharmacology and pharmacotherapy education for health professionals and students: a systematic review.

WHAT IS ALREADY KNOWN ABOUT THIS SUBJECT: The rate of medication errors is high, and these errors can cause adverse drug reactions. Elderly individuals are most vulnerable to adverse drug reactions. One cause of medication errors is the lack of drug knowledge on the part of different health professionals. Medical curricula have changed in recent years, resulting in less education in the basic sciences, such as pharmacology. WHAT THIS STUDY ADDS: Our study shows that little curricular time is devoted to geriatric pharmacology and that educational programmes in geriatric pharmacology have not been thoroughly evaluated. While interest in pharmacology education has increased recently, this is not the case for geriatric pharmacology education. Education on geriatric pharmacology should have more attention in the curricula of health professionals, given the often complex pharmacotherapy in elderly patients. Educational topics should be related to the known risk factors of medication errors, such as polypharmacy, dose adjustments in organ dysfunction and psychopharmacotherapeutics. AIMS: Given the reported high rates of medication errors, especially in elderly patients, we hypothesized that current curricula do not devote enough time to the teaching of geriatric pharmacology. This review explores the quantity and nature of geriatric pharmacology education in undergraduate and postgraduate curricula for health professionals. METHODS: Pubmed, Embase and PsycINFO databases were searched (from 1 January 2000 to 11 January 2011), using the terms 'pharmacology' and 'education' in combination. Articles describing content or evaluation of pharmacology education for health professionals were included. Education in general and geriatric pharmacology was compared. RESULTS: Articles on general pharmacology education (252) and geriatric pharmacology education (39) were included. The number of publications on education in general pharmacology, but not geriatric pharmacology, has increased over the last 10 years. Articles on undergraduate and postgraduate education for 12 different health disciplines were identified. A median of 24 h (from 15 min to 4956 h) devoted to pharmacology education and 2 h (1-935 h) devoted to geriatric pharmacology were reported. Of the articles on education in geriatric pharmacology, 61.5% evaluated the teaching provided, mostly student satisfaction with the course. The strength of findings was low. Similar educational interventions were not identified, and evaluation studies were not replicated. CONCLUSIONS: Recently, interest in pharmacology education has increased, possibly because of the high rate of medication errors and the recognized importance of evidence-based medical education. Nevertheless, courses on geriatric pharmacology have not been evaluated thoroughly and none can be recommended for use in training programmes. Suggestions for improvements in education in general and geriatric pharmacology are given.

Vertical integration in medical school: effect on the transition to postgraduate training.

OBJECTIVES: Recently, many medical schools' curricula have been revised so that they represent vertically integrated (VI) curricula. Important changes include: the provision of earlier clinical experience; longer clerkships, and the fostering of increasing levels of responsibility. One of the aims of vertical integration is to facilitate the transition to postgraduate training. The purpose of the present study is to determine whether a VI curriculum at medical school affects the transition to postgraduate training in a positive way. METHODS: We carried out a questionnaire study among graduates of six medical schools in the Netherlands, who had followed either a VI or a non-VI curriculum. Items in the questionnaire focused on preparedness for work and postgraduate training, the time and number of applications required to be admitted to residency, and the process of making career choices. RESULTS: In comparison with those who have followed non-VI programmes, graduates of VI curricula appear to make definitive career choices earlier, need less time and fewer applications to obtain residency positions and feel more prepared for work and postgraduate training. CONCLUSIONS: The curriculum at medical school affects the transition to postgraduate training. Additional research is required to determine which components of the curriculum cause this effect and to specify under which conditions this effect occurs.

The influence of early clinical experiences on career preference of male and female medical students.

BACKGROUND: Clinical experience is considered to affect medical students' career preferences. It is not known whether the sequence of the clinical rotations influences these preferences. AIM: To explore whether the first clinical clerkship has more impact on career preference than the second by examining the association between the first clinical clerkship and the choice of an elective sixth-year internship. METHOD: University Medical Center Utrecht students are assigned to either a surgical or a medical ward for the first third-year clerkship and to the other ward for the second clerkship. In a retrospective cohort study, internship data of 488 sixth-year students were related to their first clerkship 3 years earlier. RESULTS: For the group as a whole, no association was found between third-year clerkship and sixth-year internship. However, male students who had been assigned to surgery first more often chose a surgical internship than those who had been assigned to medical clerkship first and vice versa (p < 0.02). Within the female subgroup, no association was found. CONCLUSION: A positive association between the nature of the clerkship and the sixth-year internship preference among male students suggest that the first clinical experience can affect later specialty preference.

Validity of the visual analogue scale as an instrument to measure self-efficacy in resuscitation skills.

CONTEXT: Self-efficacy is an important factor in many areas of medical education, including self-assessment and self-directed learning, but has been little studied in resuscitation training, possibly because of the lack of a simple measurement instrument. OBJECTIVE: We aimed to assess the validity of a visual analogue scale (VAS) linked to a single question as an instrument to measure self-efficacy with respect to resuscitation skills by comparing the VAS with a questionnaire and using known-groups comparisons. METHODS: We developed questionnaires to measure self-efficacy for a number of resuscitation tasks and for computer skills. These were compared with VASs linked to a single question per task, using a multi-trait, multi-method matrix. We also used known-groups comparisons of self-efficacy in specific professional groups. RESULTS: There was good correlation between the questionnaires and the VASs for self-efficacy for specific resuscitation tasks. There was a less clear correlation for self-efficacy for paediatric resuscitation overall. There was no correlation between self-efficacy for resuscitation and computer tasks. In specific professional groups, measured self-efficacy accorded with theoretical predictions. CONCLUSIONS: A VAS linked to a single question appears to be a valid method of measuring self-efficacy with respect to specific well defined resuscitation tasks, but should be used with caution for multi-faceted tasks.

Growth of analytical thinking skills over time as measured with the MATCH test.

CONTEXT: Ber's Comprehensive Integrative Puzzle aims to assess analytical clinical thinking in medical students. We developed a paediatric version, the MATCH test, in which we added two irrelevant options to each question in order to reduce guessing behaviour. We tested its construct validity and studied the development of integrative skills over time. METHODS: We administered a test (MATCH 1) to subjects from two universities, both with a 6-year medical training course. Subjects included 30 students from university 1 who had completed a paediatric clerkship in Year 4, 23 students from university 2 who had completed a paediatric clerkship in Year 5, 13 students from both universities who had completed an advanced paediatric clerkship in Year 6, 28 paediatric residents and 17 paediatricians. We repeated this procedure using a second test with different domains in a new, comparable group of subjects (MATCH 2). RESULTS: Mean MATCH 1 scores for the respective groups were: Year 4 students: 61.2% (standard deviation [SD] 1.3); Year 5 students: 71.3% (SD 1.6); Year 6 students: 76.2% (SD 1.5); paediatric residents: 88.5% (SD 0.7), and paediatricians: 92.2% (SD 1.1) (one-way ANOVA F = 104.00, P < 0.0001). Students of both universities had comparable scores. MATCH 1 and 2 scores were comparable. Cronbach's alpha-values in MATCH 1 and 2 were 0.92 and 0.91, respectively, for all subjects, and 0.82 and 0.87, respectively, for all students. CONCLUSIONS: Analytical clinical thinking develops over time, independently of the factual content of the course. This implies that shortened medical training programmes could produce less skilled graduates.

The effect of the Advanced Paediatric Life Support course on perceived self-efficacy and use of resuscitation skills.

BACKGROUND: Perceived self-efficacy is a predictor of behaviour and therefore an important dimension of resuscitation training which may have consequences for patient care. The Advanced Paediatric Life Support (APLS) course makes use of techniques which would be expected to increase self-efficacy. We examined the effect of this course on perceived self-efficacy in respect of resuscitation skills and on the use of these skills. DESIGN: A prospective descriptive study using a questionnaire. METHODS: Questionnaires were sent to all doctor candidates following the course in the Netherlands over a period of 15 months and to a matched control group. Perceived self-efficacy in respect of paediatric resuscitation as a whole and six of its component skills was measured on a visual analogue scale. Questionnaires were sent out before the course and 3 and 6 months afterwards. FINDINGS: The candidate and control groups were not significantly different in terms of sex, specialty, grade or experience with children. Perceived self-efficacy increased significantly (p<0.01) in candidates following the course for all skills and remained increased for at least 6 months. There was no significant change in perceived self-efficacy in the control group. There was no change in the use of skills or in the number of critically ill children seen, in either group. CONCLUSIONS: The APLS course does have an important effect on perceived self-efficacy but this is not related to an increase in the frequency of use of the skills learned on the course, partly because the opportunity for increased use is lacking. Further work is needed to determine whether the quality of those skills is improved by the course.

Medical clerks' attitudes towards the basic sciences: a longitudinal and a cross-sectional comparison between students in a conventional and an innovative curriculum.

BACKGROUND: This study is the longitudinal sequel to a survey published by Custers and Ten Cate in December 2002 in which advanced medical students' attitudes towards the basic sciences were investigated. Students were enrolled in either a conventional or an innovative curriculum. AIMS AND METHODS: The aim of the present study was to assess longitudinal development of students' attitudes by recording beginning and advanced clerks responses to nine disagree-agree statements concerning the basic sciences. RESULTS: In general, most students in either curriculum acknowledge the importance of biomedical knowledge. Students in the conventional curriculum appear to assign a slightly more important role to the basic sciences than students in the innovative curriculum, and this difference is maintained over the clinical years. Surprisingly, advanced clerks in either curriculum are more likely than beginning clerks to support the view that many basic science facts should be learned before application in a clinical context. CONCLUSIONS: These findings suggest that students during their clerkships might experience regret having not paid more attention to factual basic science knowledge in their preclinical years. Finally, students in the innovative curriculum hold more favourable opinions about the way the basic sciences are taught in their curriculum.

The Effect of Teaching Search Strategies on Perceptual Performance.

RATIONALE AND OBJECTIVES: Radiology expertise is dependent on the use of efficient search strategies. The aim of this study is to investigate the effect of teaching search strategies on trainee's accuracy in detecting lung nodules at computed tomography. MATERIALS AND METHODS: Two search strategies, "scanning" and "drilling," were tested with a randomized crossover design. Nineteen junior radiology residents were randomized into two groups. Both groups first completed a baseline lung nodule detection test allowing a free search strategy, followed by a test after scanning instruction and drilling instruction or vice versa. True positive (TP) and false positive (FP) scores and scroll behavior were registered. A mixed-design analysis of variance was applied to compare the three search conditions. RESULTS: Search strategy instruction had a significant effect on scroll behavior, F(1.3) = 54.2, P < 0.001; TP score, F(2) = 16.1, P < 0.001; and FP score, F(1.3) = 15.3, P < 0.001. Scanning instruction resulted in significantly lower TP scores than drilling instruction (M = 10.7, SD = 5.0 versus M = 16.3, SD = 5.3), t(18) = 4.78, P < 0.001; or free search (M = 15.3, SD = 4.6), t(18) = 4.44, P < 0.001. TP scores for drilling did not significantly differ from free search. FP scores for drilling (M = 7.3, SD = 5.6) were significantly lower than for free search (M = 12.5, SD = 7.8), t(18) = 4.86, P < 0.001. CONCLUSIONS: Teaching a drilling strategy is preferable to teaching a scanning strategy for finding lung nodules.

Increasing Authenticity of Simulation-Based Assessment in Diagnostic Radiology.

INTRODUCTION: Clinical reasoning in diagnostic imaging professions is a complex skill that requires processing of visual information and image manipulation skills. We developed a digital simulation-based test method to increase authenticity of image interpretation skill assessment. METHODS: A digital application, allowing volumetric image viewing and manipulation, was used for three test administrations of the national Dutch Radiology Progress Test for residents. This study describes the development and implementation process in three phases. To assess authenticity of the digital tests, perceived image quality and correspondence to clinical practice were evaluated and compared with previous paper-based tests (PTs). Quantitative and qualitative evaluation results were used to improve subsequent tests. RESULTS: Authenticity of the first digital test was not rated higher than the PTs. Test characteristics and environmental conditions, such as image manipulation options and ambient lighting, were optimized based on participants' comments. After adjustments in the third digital test, participants favored the image quality and clinical correspondence of the digital image questions over paper-based image questions. CONCLUSIONS: Digital simulations can increase authenticity of diagnostic radiology assessments compared with paper-based testing. However, authenticity does not necessarily increase with higher fidelity. It can be challenging to simulate the image interpretation task of clinical practice in a large-scale assessment setting, because of technological limitations. Optimizing image manipulation options, the level of ambient light, time limits, and question types can help improve authenticity of simulation-based radiology assessments.

Volumetric CT-images improve testing of radiological image interpretation skills.

RATIONALE AND OBJECTIVES: Current radiology practice increasingly involves interpretation of volumetric data sets. In contrast, most radiology tests still contain only 2D images. We introduced a new testing tool that allows for stack viewing of volumetric images in our undergraduate radiology program. We hypothesized that tests with volumetric CT-images enhance test quality, in comparison with traditional completely 2D image-based tests, because they might better reflect required skills for clinical practice. MATERIALS AND METHODS: Two groups of medical students (n=139; n=143), trained with 2D and volumetric CT-images, took a digital radiology test in two versions (A and B), each containing both 2D and volumetric CT-image questions. In a questionnaire, they were asked to comment on the representativeness for clinical practice, difficulty and user-friendliness of the test questions and testing program. Students' test scores and reliabilities, measured with Cronbach's alpha, of 2D and volumetric CT-image tests were compared. RESULTS: Estimated reliabilities (Cronbach's alphas) were higher for volumetric CT-image scores (version A: .51 and version B: .54), than for 2D CT-image scores (version A: .24 and version B: .37). Participants found volumetric CT-image tests more representative of clinical practice, and considered them to be less difficult than volumetric CT-image questions. However, in one version (A), volumetric CT-image scores (M 80.9, SD 14.8) were significantly lower than 2D CT-image scores (M 88.4, SD 10.4) (p<.001). The volumetric CT-image testing program was considered user-friendly. CONCLUSION: This study shows that volumetric image questions can be successfully integrated in students' radiology testing. Results suggests that the inclusion of volumetric CT-images might improve the quality of radiology tests by positively impacting perceived representativeness for clinical practice and increasing reliability of the test.

Volumetric and two-dimensional image interpretation show different cognitive processes in learners.

RATIONALE AND OBJECTIVES: In current practice, radiologists interpret digital images, including a substantial amount of volumetric images. We hypothesized that interpretation of a stack of a volumetric data set demands different skills than interpretation of two-dimensional (2D) cross-sectional images. This study aimed to investigate and compare knowledge and skills used for interpretation of volumetric versus 2D images. MATERIALS AND METHODS: Twenty radiology clerks were asked to think out loud while reading four or five volumetric computed tomography (CT) images in stack mode and four or five 2D CT images. Cases were presented in a digital testing program allowing stack viewing of volumetric data sets and changing views and window settings. Thoughts verbalized by the participants were registered and coded by a framework of knowledge and skills concerning three components: perception, analysis, and synthesis. The components were subdivided into 16 discrete knowledge and skill elements. A within-subject analysis was performed to compare cognitive processes during volumetric image readings versus 2D cross-sectional image readings. RESULTS: Most utterances contained knowledge and skills concerning perception (46%). A smaller part involved synthesis (31%) and analysis (23%). More utterances regarded perception in volumetric image interpretation than in 2D image interpretation (Median 48% vs 35%; z = -3.9; P < .001). Synthesis was less prominent in volumetric than in 2D image interpretation (Median 28% vs 42%; z = -3.9; P < .001). No differences were found in analysis utterances. CONCLUSIONS: Cognitive processes in volumetric and 2D cross-sectional image interpretation differ substantially. Volumetric image interpretation draws predominantly on perceptual processes, whereas 2D image interpretation is mainly characterized by synthesis. The results encourage the use of volumetric images for teaching and testing perceptual skills.

"Clinical reasoning theater": a new approach to clinical reasoning education.

Recently the authors developed and implemented a new approach to clinical reasoning education called clinical reasoning theater (CRT). The purpose of CRT is to demonstrate the process of clinical reasoning to students through the conversation of a doctor with his or her patient. With students as the audience, the doctor's clinical reasoning skills are modeled in CRT when he or she thinks aloud during conversations with the patient, which makes clear why he or she asks particular questions. In CRT the interaction between doctor and patient is analyzed in such a way that the repeated cycles of questions and answers are revealed to the students. CRT's format and the activities of the three "actors,"(the doctor, the patient, and the audience of students) are described. Two "acts" are repeatedly performed. The first act (the communication between the doctor and the patient) deals with the history taking, physical examination, etc. The second act, the time-out period, is the most important component of CRT. In this act, the doctor explains his or her reflections and discusses suggestions from the audience for additional questions. The preliminary results of students' evaluations of the relevance of CRT reveal that they appreciate CRT. Further, the CRT format offers instructors with previous experience as lecturers only sufficient tools to apply the new approach successfully.