Structured Smoking Cessation Training for Medical Students: A Prospective Study.

INTRODUCTION: Physician adherence to guideline recommendations regarding the provision of counseling and support for smokers willing to quit is low. A lack of training during undergraduate medical education has been identified as a potential cause. This prospective intervention study evaluated a novel teaching module for medical students. METHODS: As part of a 6-week cardiovascular course, 125 fourth-year undergraduate medical students received a multimodal and interactive teaching module on smoking cessation, including online learning material, lectures, seminars, and practical skills training. Short- and medium-term effects on knowledge, skills, attitudes, and self-reported practice were measured using written examinations and an objective structured clinical examination at the end of the module and 6 months later. Results were compared to data obtained from a historical control cohort (n = 70) unexposed to the intervention. RESULTS: At the 6-month follow-up, scores in the knowledge test were significantly higher in the intervention than the control group (61.1% vs. 51.7%; p < .001). A similar pattern was observed in the objective structured clinical examination (71.5% vs. 60.5%; p < .001). More students in the intervention than control group agreed that smoking was a chronic disease (83.1% vs. 68.1%; p = .045). The control group was more likely to report recording smoking status (p = .018), but no group difference was detected regarding the report of advising to quit (p = .154). CONCLUSIONS: A novel teaching module for undergraduate medical students produced a sustained learning outcome in terms of knowledge, skills, and attitudes but not self-reported practice. IMPLICATIONS: Studies across the world have identified considerable knowledge gaps and deficits in practical training with regard to smoking cessation counseling in undergraduate medical students. This paper describes a teaching intervention informed by current recommendations for the design of educational activities aimed at enabling medical students to deliver adequate behavior change counseling. The teaching module was tailored to the needs of a specific healthcare system. Given its effectiveness as demonstrated in this prospective study, a rollout of this intervention in medical schools might have the potential to substantially improve medical students' knowledge, skills, and attitudes in relation to smoking cessation counseling.

A systematic review of factors influencing student ratings in undergraduate medical education course evaluations.

BACKGROUND: Student ratings are a popular source of course evaluations in undergraduate medical education. Data on the reliability and validity of such ratings have mostly been derived from studies unrelated to medical education. Since medical education differs considerably from other higher education settings, an analysis of factors influencing overall student ratings with a specific focus on medical education was needed. METHODS: For the purpose of this systematic review, online databases (PubMed, PsycInfo and Web of Science) were searched up to August 1st, 2013. Original research articles on the use of student ratings in course evaluations in undergraduate medical education were eligible for inclusion. Included studies considered the format of evaluation tools and assessed the association of independent and dependent (i.e., overall course ratings) variables. Inclusion and exclusion criteria were checked by two independent reviewers, and results were synthesised in a narrative review. RESULTS: Twenty-five studies met the inclusion criteria. Qualitative research (2 studies) indicated that overall course ratings are mainly influenced by student satisfaction with teaching and exam difficulty rather than objective determinants of high quality teaching. Quantitative research (23 studies) yielded various influencing factors related to four categories: student characteristics, exposure to teaching, satisfaction with examinations and the evaluation process itself. Female gender, greater initial interest in course content, higher exam scores and higher satisfaction with exams were associated with more positive overall course ratings. CONCLUSIONS: Due to the heterogeneity and methodological limitations of included studies, results must be interpreted with caution. Medical educators need to be aware of various influences on student ratings when developing data collection instruments and interpreting evaluation results. More research into the reliability and validity of overall course ratings as typically used in the evaluation of undergraduate medical education is warranted.

Standardized and quality-assured video-recorded examination in undergraduate education: informed consent prior to surgery.

BACKGROUND: Communication skills combined with specialized knowledge are fundamental to the doctor-patient relationship in surgery. During a single-station video-recorded objective structured clinical examination (VOSCE), students were tasked with obtaining informed consent. Our aim was to develop a standardized and quality-assured assessment method in undergraduate education. METHODS: One hundred fifty-five students in their fifth year of medical school (78 videos) participated in a summative VOSCE within the framework of the teaching module "Operative Medicine." They prepared for three clinical scenarios and the surgical procedures involved. The examination comprised participants having to obtain informed consent from simulated patients, video recording their performance. Students were assessed by two independent raters, the background of one of whom was nonsurgical. Results were statistically tested using SPSS. RESULTS: Students' scores were all beyond the pass mark of 70%, averaging 91.0% (±4.0%), 88.4% (±4.4%), and 87.0% (±4.7%) for the appendectomy, cholecystectomy, and inguinal hernia repair checklist, respectively. Most items (68%-89% of the checklists) were found to have fair to excellent discrimination values. Cronbach's α values ranged between 0.565 and 0.605 for the individual checklists. Interrater agreement was strong (Pearson correlation coefficient = 0.80, P < 0.01; intraclass correlation coefficient 2.1 = 0.78). CONCLUSIONS: The VOSCE is both feasible and reliable as a method of assessing student communication skills and the application of clinical knowledge while obtaining informed consent in surgery. This method is efficient (flexible rating outside normal working hours possible with reductions in administrative load) and may be used for high-stakes evaluation of student performance.

Autopsy in undergraduate medical education--what do students really learn?

Forensic medicine can provide access to autopsies for students in undergraduate medical education. Previous qualitative studies reported that attending autopsies promotes learning and is helpful in organising information in the context of self-directed learning, covering cognitive, emotional and societal issues. In the present study, learning outcome regarding specific pathophysiological learning objectives of students who attended an autopsy elective (intervention group, n = 32) was estimated in comparison to peer students who signed up for the course but were not selected (control group A, n = 47) and students who had not signed up (control group B, n = 186). Learning outcome (expressed as a percent knowledge gain) was measured by means of comparative self-assessments (CSA). Furthermore, group interviews were conducted to evaluate the attending students' perceptions and experiences of the course. In the intervention group, the percent knowledge gain for learning objectives reflecting pathophysiological mechanisms was about twice as high than that in the control groups, while all three groups showed comparable knowledge gain for learning objectives covered by mandatory courses that were held during the study period. Results of mandatory post-course assessments showed no statistically significant differences between the intervention and control groups. Therefore, the knowledge gain regarding learning objectives not covered by mandatory courses that was observed in the intervention group reflected the additional benefit of attending the autopsy course. Group interviews with the participants revealed gain of integrated knowledge and understanding of pathophysiological relations as important issues, while negative effects were reported infrequently. Our findings suggest that attendance of a series of autopsies fosters learning and understanding of important issues in medical education and should therefore be part of undergraduate medical curricula wherever possible.

Assessment of two different types of bias affecting the results of outcome-based evaluation in undergraduate medical education.

BACKGROUND: Estimating learning outcome from comparative student self-ratings is a reliable and valid method to identify specific strengths and shortcomings in undergraduate medical curricula. However, requiring students to complete two evaluation forms (i.e. one before and one after teaching) might adversely affect response rates. Alternatively, students could be asked to rate their initial performance level retrospectively. This approach might threaten the validity of results due to response shift or effort justification bias. METHODS: Two consecutive cohorts of medical students enrolled in a six-week cardio-respiratory module were enrolled in this study. In both cohorts, performance gain was estimated for 33 specific learning objectives. In the first cohort, outcomes calculated from ratings provided before (pretest) and after (posttest) teaching were compared to outcomes derived from comparative self-ratings collected after teaching only (thentest and posttest). In the second cohort, only thentests and posttests were used to calculate outcomes, but data collection tools differed with regard to item presentation. In one group, thentest and posttest ratings were obtained sequentially on separate forms while in the other, both ratings were obtained simultaneously for each learning objective. RESULTS: Using thentest ratings to calculate performance gain produced slightly higher values than using true pretest ratings. Direct comparison of then- and posttest ratings also yielded slightly higher performance gain than sequential ratings, but this effect was negligibly small. CONCLUSIONS: Given the small effect sizes, using thentests appears to be equivalent to using true pretest ratings. Item presentation in the posttest does not significantly impact on results.

Student perceptions of evaluation in undergraduate medical education: A qualitative study from one medical school.

BACKGROUND: Evaluation is an integral part of medical education. Despite a wide use of various evaluation tools, little is known about student perceptions regarding the purpose and desired consequences of evaluation. Such knowledge is important to facilitate interpretation of evaluation results. The aims of this study were to elicit student views on the purpose of evaluation, indicators of teaching quality, evaluation tools and possible consequences drawn from evaluation data. METHODS: This qualitative study involved 17 undergraduate medical students in Years 3 and 4 participating in 3 focus group interviews. Content analysis was conducted by two different researchers. RESULTS: Evaluation was viewed as a means to facilitate improvements within medical education. Teaching quality was believed to be dependent on content, process, teacher and student characteristics as well as learning outcome, with an emphasis on the latter. Students preferred online evaluations over paper-and-pencil forms and suggested circulating results among all faculty and students. Students strongly favoured the allocation of rewards and incentives for good teaching to individual teachers. CONCLUSIONS: In addition to assessing structural aspects of teaching, evaluation tools need to adequately address learning outcome. The use of reliable and valid evaluation methods is a prerequisite for resource allocation to individual teachers based on evaluation results.

Evaluation in medical education: A topical review of target parameters, data collection tools and confounding factors.

BACKGROUND AND OBJECTIVE: Evaluation is an integral part of education in German medical schools. According to the quality standards set by the German Society for Evaluation, evaluation tools must provide an accurate and fair appraisal of teaching quality. Thus, data collection tools must be highly reliable and valid. This review summarises the current literature on evaluation of medical education with regard to the possible dimensions of teaching quality, the psychometric properties of survey instruments and potential confounding factors. METHODS: We searched Pubmed, PsycINFO and PSYNDEX for literature on evaluation in medical education and included studies published up until June 30, 2011 as well as articles identified in the "grey literature". RESULTS are presented as a narrative review. RESULTS: We identified four dimensions of teaching quality: structure, process, teacher characteristics, and outcome. Student ratings are predominantly used to address the first three dimensions, and a number of reliable tools are available for this purpose. However, potential confounders of student ratings pose a threat to the validity of these instruments. Outcome is usually operationalised in terms of student performance on examinations, but methodological problems may limit the usability of these data for evaluation purposes. In addition, not all examinations at German medical schools meet current quality standards. CONCLUSION: The choice of tools for evaluating medical education should be guided by the dimension that is targeted by the evaluation. Likewise, evaluation results can only be interpreted within the context of the construct addressed by the data collection tool that was used as well as its specific confounding factors.

Structured smoking cessation training for health professionals on cardiology wards: a prospective study.

BACKGROUND: Smoking is a major cardiovascular risk factor, and smoking cessation is imperative for patients hospitalized with a cardiovascular event. This study aimed to evaluate a systems-based approach to helping hospitalized smokers quit and to identify implementation barriers. DESIGN: Prospective intervention study followed by qualitative analysis of staff interviews. METHODS: The prospective intervention study assessed the effects of implementing standard operating procedures (SOPs) for the provision of counselling and pharmacotherapy to smokers admitted to cardiology wards on counselling frequency. In addition, a qualitative analysis of staff interviews was undertaken to examine determinants of physician and nurse behaviour; this sought to understand barriers in terms of motivation, capability, and/or opportunity. RESULTS: A total of 150 smoking patients were included in the study (75 before and 75 after SOP implementation). Before the implementation of SOPs, the proportion of patients reporting to have received cessation counselling from physicians and nurses was 6.7% and 1.3%, respectively. Following SOP implementation, these proportions increased to 38.7% (p < 0.001) and 2.7% (p = 0.56), respectively. Qualitative analysis revealed that lack of motivation, e.g. role incongruence, appeared to be a major barrier. CONCLUSIONS: Introduction of a set of standard operating procedures for smoking cessation advice was effective with physicians but not nurses. Analysis of barriers to implementation highlighted lack of motivation rather than capability or opportunity as a major factor that would need to be addressed.

Increase in medical knowledge during the final year of undergraduate medical education in Germany.

AIMS: In Germany, the final year of undergraduate medical education ('practice year') consists of three 16-week clinical attachments, two of which are internal medicine and surgery. Students can choose a specific specialty for their third 16-week attachment. Practice year students do not receive specific teaching to prepare them for the National Licensing Examination. It is unknown whether knowledge levels increase during this year. This study aimed at assessing knowledge at the beginning and the end of the final year of medical school. METHODS: Three hundred pre-selected United States Medical Licensing Examination type items from ten medical disciplines were reviewed by ten recent medical graduates from the Netherlands and Germany. The resulting test included 150 items and was taken by 77 and 79 final year medical students from Göttingen and Hamburg at the beginning and the end of their practice year, respectively. RESULTS: Cronbach's α of the pre- and post-test was 0.75 and 0.68, respectively. Mean percent scores in the pre- and post-test were 63.9±6.9 and 69.4±5.7, respectively (p<0.001; effect size calculated as Cohen's d: 0.87). In individual students, post-test scores were particularly high for items related to their specific chosen specialty. CONCLUSION: The knowledge test used in this study provides a suitable external tool to assess progress of undergraduate medical students in their knowledge during the practice year. The pre-test may be used to guide individual learning behaviour during this final year of undergraduate education.

Estimating learning outcomes from pre- and posttest student self-assessments: a longitudinal study.

PURPOSE: Learning outcome is an important measure for overall teaching quality and should be addressed by comprehensive evaluation tools. The authors evaluated the validity of a novel evaluation tool based on student self-assessments, which may help identify specific strengths and weaknesses of a particular course. METHOD: In 2011, the authors asked 145 fourth-year students at Göttingen Medical School to self-assess their knowledge on 33 specific learning objectives in a pretest and posttest as part of a cardiorespiratory module. The authors compared performance gain calculated from self-assessments with performance gain derived from formative examinations that were closely matched to these 33 learning objectives. RESULTS: Eighty-three students (57.2%) completed the assessment. There was good agreement between performance gain derived from subjective data and performance gain derived from objective examinations (Pearson r=0.78; P<.0001) on the group level. The association between the two measures was much weaker when data were analyzed on the individual level. Further analysis determined a quality cutoff for performance gain derived from aggregated student self-assessments. When using this cutoff, the evaluation tool was highly sensitive in identifying specific learning objectives with favorable or suboptimal objective performance gains. CONCLUSIONS: The tool is easy to implement, takes initial performance levels into account, and does not require extensive pre-post testing. By providing valid estimates of actual performance gain obtained during a teaching module, it may assist medical teachers in identifying strengths and weaknesses of a particular course on the level of specific learning objectives.

Piloting an outcome-based programme evaluation tool in undergraduate medical education.

AIMS: Different approaches to performance-oriented allocation of resources according to teaching quality are currently being discussed within German medical schools. The implementation of these programmes is impeded by a lack of valid criteria to measure teaching quality. An assessment of teaching quality should include structural and procedural aspects but focus on learning outcome itself. The aim of this study was to implement a novel, outcome-based evaluation tool within the clinical phase of a medical curriculum and address differences between the novel tool and traditional evaluation methods. METHODS: Student self-assessments before and after completion of a teaching module were used to compute performance gains for specific learning objectives. Mean performance gains in each module were compared to student expectations before the module and data derived from a traditional evaluation tool using overall course ratings at the end of the module. RESULTS: A ranking of the 21 modules according to computed performance gains yielded entirely different results than module rankings based on overall course ratings. There was no significant correlation between performance gain and overall ratings. However, the latter were significantly correlated to student expectations before entering the module as well as structural and procedural parameters (Pearson's r 0.7-0.9). CONCLUSION: Performance gain computed from comparative self-assessments adds an important new dimension to course evaluation in medical education. In contrast to overall course ratings, the novel tool is less heavily confounded by construct-irrelevant factors. Thus, it appears to be more appropriate than overall course ratings in determining teaching quality and developing algorithms to guide performance-oriented resource allocation in medical education.