The Impact of Emotion on Learners' Application of Basic Science Principles to Novel Problems.

PURPOSE: Training to become a physician is an emotionally laden experience. Research in cognitive psychology indicates that emotions can influence learning and performance, but the materials used in such research (e.g., word lists) rarely reflect the complexity of material presented in medical school. The present study examined whether emotions influence learning of basic science principles. METHOD: Fifty-five undergraduate psychology students were randomly assigned to write about positive, negative, or neutral life events for nine minutes. Participants were then taught three physiological concepts, each in the context of a single organ system. Testing consisted of 13 clinical cases, 7 presented with the same concept/organ system pairing used during training ("near transfer") and 6 with novel pairings ("far transfer"). Testing was repeated after one week with 13 additional cases. RESULTS: Forty-nine students provided complete data. Higher test scores were found when the concept/organ system pairing was held constant (near transfer = 51% correct vs. far = 33%; P < .001). Emotion condition influenced participants' overall performance, with individuals in the neutral condition (50.1%) performing better than those in the positive (38.2%, P < .05) and negative (37.7%, P < .001) emotion conditions. CONCLUSIONS: These data suggest that regardless of whether the emotion is positive or negative, mild affective states can impair learning of basic science concepts by novices. Demands on working memory and subsequent cognitive load provide a potential explanation. Future work will examine the extent to which these findings generalize to medical trainees.

Assessing the assessment in emergency care training.

OBJECTIVE: Each year over 1.5 million health care professionals attend emergency care courses. Despite high stakes for patients and extensive resources involved, little evidence exists on the quality of assessment. The aim of this study was to evaluate the validity and reliability of commonly used formats in assessing emergency care skills. METHODS: Residents were assessed at the end of a 2-week emergency course; a subgroup was videotaped. Psychometric analyses were conducted to assess the validity and inter-rater reliability of the assessment instrument, which included a checklist, a 9-item competency scale and a global performance scale. RESULTS: A group of 144 residents and 12 raters participated in the study; 22 residents were videotaped and re-assessed by 8 raters. The checklists showed limited validity and poor inter-rater reliability for the dimensions "correct" and "timely" (ICC = .30 and.39 resp.). The competency scale had good construct validity, consisting of a clinical and a communication subscale. The internal consistency of the (sub)scales was high (α = .93/.91/.86). The inter-rater reliability was moderate for the clinical competency subscale (.49) and the global performance scale (.50), but poor for the communication subscale (.27). A generalizability study showed that for a reliable assessment 5-13 raters are needed when using checklists, and four when using the clinical competency scale or the global performance scale. CONCLUSIONS: This study shows poor validity and reliability for assessing emergency skills with checklists but good validity and moderate reliability with clinical competency or global performance scales. Involving more raters can improve the reliability substantially. Recommendations are made to improve this high stakes skill assessment.

Reexamining our bias against heuristics.

Using heuristics offers several cognitive advantages, such as increased speed and reduced effort when making decisions, in addition to allowing us to make decision in situations where missing data do not allow for formal reasoning. But the traditional view of heuristics is that they trade accuracy for efficiency. Here the authors discuss sources of bias in the literature implicating the use of heuristics in diagnostic error and highlight the fact that there are also data suggesting that under certain circumstances using heuristics may lead to better decisions that formal analysis. They suggest that diagnostic error is frequently misattributed to the use of heuristics and propose an alternative view whereby content knowledge is the root cause of diagnostic performance and heuristics lie on the causal pathway between knowledge and diagnostic error or success.

Multisource feedback in evaluating the performance of doctors: the example of the UK General Medical Council patient and colleague questionnaires.

PURPOSE: Internationally, there is increasing interest in monitoring and evaluating doctors' professional practice. Multisource feedback (MSF) offers one way of collecting information about doctors' performance. The authors investigated the psychometric properties of two questionnaires developed for this purpose and explored the biases that may exist within data collected via such instruments. METHOD: A cross-sectional study was conducted in 11 UK health care organizations during 2008-2011. Patients (n = 30,333) and colleagues (n = 17,012) rated the professional performance of 1,065 practicing doctors, using the General Medical Council Patient Questionnaire (PQ) and Colleague Questionnaire (CQ). The psychometric properties of the questionnaires were assessed, and regression modeling was used to explore factors that influenced patient and colleague responses on the core questionnaire items. RESULTS: Although the questionnaires demonstrated satisfactory internal consistency, test-retest reliability, and convergent validity, patient and colleague ratings were highly skewed toward favorable impressions of doctor performance. At least 34 PQs and 15 CQs were required to achieve acceptable reliability (G > 0.70). Item ratings were influenced by characteristics of the patient and colleague respondents and the context in which their feedback was provided. CONCLUSIONS: The PQ and CQ are acceptable for the provision of formative feedback on a doctor's professional practice within an appraisal process. However, biases identified in the questionnaire data suggest that caution is required when interpreting and acting on this type of information. MSF derived from these questionnaires should not be used in isolation to inform decisions about a doctor's fitness to practice medicine.

Evaluating agreement: conducting a reliability study.

Instruments that are useful in clinical or research practice will, when the object of measurement is stable, yield similar results when applied at different times, in different situations, or by different users. Studies that measure the relation of differences between patients or subjects and measurement error (reliability studies) are becoming increasingly common in the orthopaedic literature. In this paper, we identify common aspects of reliability studies and suggest features that improve the reader's confidence in the results. One concept serves as the foundation for all further consideration: in order for a reliability study to be relevant, the patients, raters, and test administration in the study must be similar to the clinical or research context in which the instrument will be used. We introduce the statistical measures that readers will most commonly encounter in reliability studies, and we suggest an approach to sample-size estimation. Readers interested in critically appraising reliability studies or in developing their own reliability studies may find this review helpful.

PBL in the undergraduate MD program at McMaster University: three iterations in three decades.

When the undergraduate MD program of McMaster University admitted its first cohort of 20 students in 1969, it heralded a major change in medical school pedagogy that has influenced the education of medical students around the world. The three-year PBL curriculum, which emphasized small-group tutorials, self-directed learning, a minimal number of didactic presentations, and student evaluation that was based almost entirely on performance in the tutorial, represented a radical departure from traditional curricula. Since the inception of the original curriculum in 1969, there have been two major curriculum revisions, the most recent of which was in 2005. The original curriculum attempted to integrate both basic science and clinical science into the biomedical problems. The second iteration of the curriculum focused on priority health problems and centered on a list of common medical problems as the foundation for curriculum organization, on the basis that an understanding of the management of common conditions included areas of knowledge that would be essential for clinical competence. Under the third, current curriculum, the COMPASS (concept-oriented, multidisciplinary, problem-based, practice for transfer, simulations in clerkship, streaming) model was adopted. Under this concept-based system, emphasis is placed on underscoring the underlying concepts in the curriculum with a logical sequencing of both the concepts and the body systems. This article briefly reviews the history of the development of the undergraduate MD program at McMaster and the three curricula that have been developed during the past three decades.

In-training evaluation using hand-held computerized clinical work sampling strategies in radiology residency.

PURPOSE: The in-training evaluation and final in-training evaluation are the mainstay format for evaluation summaries in Canadian residency training programs. This study investigates the feasibility of a clinical work sampling (CWS) approach to evaluation in radiology residency, with the aid of personal hand-held computing devices METHODS: This study was conducted over a 1-year period with 14 radiology residents spanning 4 postgraduate years. Residents were provided with a hand-held device to enter evaluation data, with entries assessing 9 categories of resident performance. Results from the CWS entries were compared with standard in-training evaluations completed at the end of the residents' rotations, as well as with an established annual objective evaluation tool. RESULTS: The overall reliability of the CWS approach according to the observed 7 forms per resident was 0.62, suggesting that a minimum of 20 forms would be required to achieve a reliability of 0.80. For the in-training evaluation report (ITER), internal consistency was 0.98, reflecting very high correlations between categories and indicating that the individual categories are not discriminating. Correlation across rotations was 0.36, which is low for summative evaluation. Correlation between the 2 measures was 0.47 (P = 0.09); neither measure was correlated with the American College of Radiology evaluation. CONCLUSION: The CWS strategy is feasible for adaptation to radiology residency, although compliance with voluntary entries was less than expected. It is not clear whether this reflects the additional burden of using the hand-held device, the fact that entries were voluntary rather than mandatory, or the many demands on both residents and evaluators. The added potential of this evaluation format includes the opportunity to discuss performance at the time of data entry, rather than resorting to the usual end-of-rotation evaluation. Nevertheless, the study has shown that the ITER remains only of marginal value for summative evaluation; the addition of the CWS would require at least 20 forms for acceptable reliability and might not justify the additional cost and complexity.

A web-based test of residents' skills in diagnostic radiology.

PURPOSE: To develop an objective, Web-based tool for evaluating residents' knowledge of diagnostic radiology. METHODS: We developed and tested a Web-based evaluation tool (the Diagnostic Radiology Skills Test) that consists of 3 tests, one in each of 3 domains of diagnostic radiology: chest, gastrointestinal, and musculoskeletal imaging. Each test comprises 30 cases representing a range of difficulty in the domain, including normal states, normal variants, typical cases of common diagnoses, and cases with more subtle findings. Cases are presented with a long menu of domain-specific possible diagnoses (response options), each coded for diagnostic appropriateness. Our subjects were 21 residents in postgraduate year (PGY) 2 to 5 and 11 experts in diagnostic radiology. Subjects accessed the tool via a Web site on our Web server. Residents' test results were compared for reliability and validity across domain, case, and training level. In addition, results were correlated with commonly used established and objective evaluation tools. RESULTS: The tool demonstrated consistent monotonic improvement in performance with training level. It showed acceptable reliability in discriminating between residents at different performance levels, both within and across training levels (r= 0.53 within level and 0.69 across levels). Test results also had concurrent validity against the American College of Radiology In-Training Examination, a widely accepted objective assessment tool (r = 0.65, P < 0.01), and 2 Objective Structured Clinical Examinations (OSCEs) focusing on diagnostic skills (r= 0.78 and r= 0.69, P < 0.01, respectively). CONCLUSIONS: Our study demonstrates the feasibility of a Web-based, standardized, objective assessment method for evaluating residents' performance.

The September epidemic of asthma hospitalization: school children as disease vectors.

BACKGROUND: Viral infections are associated with the majority of asthma exacerbations in children and adults. Increased asthma hospitalization rates of children and adults, particularly in the early fall, have been observed to follow school vacations. OBJECTIVE: We sought to determine the sequence of timing of September asthma hospitalization epidemics in children and adults and to determine whether school-age children are the primary source of transmission of agents that cause them. METHODS: By using Canadian asthma hospital admission data from 1990 to 2002, we examined geographic variation in the timing of fall asthma epidemics and applied mathematical modeling to estimate their exact timing and magnitude in school-age children, preschool children, and adults, and relation to school return. RESULTS: The September asthma hospitalization epidemic peak occurred in school-age children each year on average 17.7 (95% CI, 16.8-18.5) days after Labor Day. Similar epidemics of lesser magnitude were observed in preschool children peaking 1.7 (95% CI, 0.9-2.5; P<.001) days later, and in adults 6.3 (95% CI, 4.7-7.9; P<.001) days later than in school-age children. The epidemics peaked 4.2 (95% CI, 1.2-7.1; P<.001) days earlier in school-age children in northernmost compared with southernmost latitudes. CONCLUSION: September epidemics of asthma hospitalizations in Canada have a precise relationship to school return after the summer vacation. It may be speculated that school-age children transmit the agents responsible for the epidemic to adults. Measures to improve asthma control and reduce transmission of infections should be directed at children with asthma before school return.