Assessing Diagnostic Reasoning Using a Standardized Case-Based Discussion.

BACKGROUND: Development of diagnostic reasoning (DR) is fundamental to medical students' training, but assessing DR is challenging. Several written assessments focus on DR but lack the ability to dynamically assess DR. Oral assessment formats have strengths but have largely lost favour due to concerns about low reliability and lack of standardization. Medical schools and specialist medical colleges value many forms of oral assessment (eg, long case, Objective Structured Clinical Examination [OSCE], viva voce) but are increasingly searching for ways in which to standardize these formats. We sought to develop and trial a Standardized Case-Based Discussion (SCBD), a highly standardized and interactive oral assessment of DR. METHODS: Two initial cohorts of medical students (n = 319 and n = 342) participated in the SCBD as part of their assessments. All students watch a video trigger (based on an authentic clinical case) and discuss their DR with an examiner for 15 minutes. Examiners probe students' DR and assess how students respond to new standardized clinical information. An online examiner training module clearly articulates expected student performance standards. We used student achievement and student and examiner perceptions to gauge the performance of this new assessment form over 2 implementation years. RESULTS: The SCBD was feasible to implement for a large student cohort and was acceptable to students and examiners. Most students and all examiners agreed that the SCBD discussion provided useful information on students' DR. The assessment had acceptable internal consistency, and the associations with other assessment formats were small and positive, suggesting that the SCBD measures a related, yet novel construct. CONCLUSIONS: Rigorous, standardized oral assessments have a place in a programme of assessment in initial medical training because they provide opportunities to explore DR that are limited in other formats. We plan to incorporate an SCBD into our clinical assessments for the first year of clinical training, where teaching and assessing basic DR is emphasized. We will also explore further examiners' understanding of and approach to assessing DR.

Tensions in post-examination feedback: information for learning versus potential for harm.

OBJECTIVE: Self-regulation is recognised as being a requisite skill for professional practice This study is part of a programme of research designed to explore efficient methods of feedback that improve medical students' ability to self-regulate their learning. Our aim was to clarify how students respond to different forms and content of written feedback and to explore the impact on study behaviour and knowledge acquisition. METHODS: Year 2 students in a 4-year graduate entry medical programme completing four formative progress tests during the academic year were randomised into three groups receiving different feedback reports. All reports included proportion correct overall and by clinical rotation. One group received feedback reports including lists of clinical presentations relating to questions answered correctly and incorrectly; another group received reports containing this same information in combination with response certitude. The final group received reports involving normative comparisons. Baseline progress test performance quartile groupings (a proxy for academic ability) were determined by results on the first progress test. A mixed-method approach with triangulation of research findings was used to interpret results. Outcomes of interest included progress test scores, summative examination results and measures derived from study diaries, questionnaires and semi-structured interviews. RESULTS: Of the three types of feedback provided in this experiment, feedback containing normative comparisons resulted in inferior test performance for students in the lowest performance quartile group. This type of feedback appeared to stimulate general rather than examination-focused study. CONCLUSIONS: Medical students are often considered relatively homogenous and high achieving, yet the results of this study suggest caution when providing them with normative feedback indicating poorer performance relative to their peers. There is much need for further work to explore efficient methods of providing written feedback that improves medical students' ability to self-regulate their learning, particularly when giving feedback to those students who have the most room for improvement.

The effectiveness of the internship in meeting established learning objectives: A qualitative study.

OBJECTIVE: To identify and explore variations in intern (Postgraduate Year 1 doctors) exposure to clinical activities defined by an established early postgraduate curriculum within the contemporary health service environment. METHODS: A mixed methods study was undertaken in seven health services comprising (1) a questionnaire regarding interns' exposure to a range of clinical activities within a two-year early post-graduate curriculum and (2) group interviews with intern supervisors and managers to explore barriers and facilitators of interns' clinical exposure. RESULTS: One hundred and eleven interns completed the questionnaire and seventeen intern supervisors and managers participated in group interviews. Interns' exposure to early postgraduate curriculum activities varied significantly, with participation in patient emergency management and procedures being undertaken less frequently than interns wished. These findings were consistent across different regional settings. Of three mandatory terms, exposure to curriculum activities was greater in medicine and emergency medicine than in surgery. CONCLUSIONS: Junior doctors' participation within the complex contemporary health service environment is constrained by barriers relating to clinical governance frameworks and health system performance, particularly in high-risk clinical activities. Solutions will require structural approaches to internship reform to ensure that placements provide the necessary safe clinical experiences required for intern training.

Beyond Selection: The Use of Situational Judgement Tests in the Teaching and Assessment of Professionalism.

PROBLEM: Professionalism is a critical attribute of medical graduates. Its measurement is challenging. The authors sought to assess final-year medical students' knowledge of appropriate professional behavior across a broad range of workplace situations. APPROACH: Situational judgement tests (SJTs) are used widely in applicant selection to assess judgement or decision making in work-related settings as well as attributes such as empathy, integrity, and resilience. In 2014, the authors developed three 40-item SJTs with scenarios relevant to interns (first-year junior doctors) and delivered the tests to final-year medical students to assess aspects of professionalism. As preparation, students discussed SJT-style scenarios; after the tests they completed an evaluation. The authors applied the Angoff method for the standard-setting process, delivered electronic individualized feedback reports to students post test, and provided remediation for students failing to meet the cut score. OUTCOMES: Evaluation revealed that the tests positively affected students' learning and that students accepted them as an assessment tool. Validity and reliability were acceptable. Implementation costs were initially high but will be recouped over time. NEXT STEPS: Recent improvements include changes to pass requirements, question revision based on reliability testing, and provision of detailed item-level feedback. Work is currently under way to expand the item bank and to introduce tests earlier in the course. Future research will explore correlation of SJT performance with other measures of professionalism and focus on the impact of SJTs on professionalism and interns' ability to deal with challenging workplace situations.

Medical school benchmarking - from tools to programmes.

BACKGROUND: Benchmarking among medical schools is essential, but may result in unwanted effects. AIM: To apply a conceptual framework to selected benchmarking activities of medical schools. METHODS: We present an analogy between the effects of assessment on student learning and the effects of benchmarking on medical school educational activities. A framework by which benchmarking can be evaluated was developed and applied to key current benchmarking activities in Australia and New Zealand. RESULTS: The analogy generated a conceptual framework that tested five questions to be considered in relation to benchmarking: what is the purpose? what are the attributes of value? what are the best tools to assess the attributes of value? what happens to the results? and, what is the likely "institutional impact" of the results? If the activities were compared against a blueprint of desirable medical graduate outcomes, notable omissions would emerge. CONCLUSION: Medical schools should benchmark their performance on a range of educational activities to ensure quality improvement and to assure stakeholders that standards are being met. Although benchmarking potentially has positive benefits, it could also result in perverse incentives with unforeseen and detrimental effects on learning if it is undertaken using only a few selected assessment tools.

Conducting patient assessments as a medical student: frequency, barriers, and facilitators.

BACKGROUND: Opportunities for medical students to engage in deliberate practice through conducting patient assessments may be declining, but data on the numbers of patients assessed by students during training are lacking. PURPOSES: The study described relationships between the frequency of patient assessments, student confidence, belief they had seen sufficient patients, and their perceptions of barriers and facilitators of seeing patients. METHODS: We employed survey methodology to estimate the number of patient assessments conducted across 6 rotations in the 1st year of clinical training, gather ratings of confidence and student belief they had conducted sufficient patient assessments, and barriers and facilitators of seeing patients. RESULTS: Rotations focused on general medicine and surgery provided more opportunities for patient assessments than specialist rotations (all p < .001). Students conducting more than 10 patient assessments rated confidence in conducting patient assessments and belief they had seen enough patients for their clinical learning, higher than students who saw 10 or fewer patients (all p < .001). CONCLUSIONS: Our study demonstrated variation in the frequency of patient assessments, and weak relationships between numbers of assessments, student confidence, and barriers to seeing patients. Further investigation is warranted of the impact of fewer opportunities for deliberate practice of skills for expertise development.

Clinical teaching and learning: from theory and research to application.

Learning in the clinical setting is the cornerstone of medical school education, but there are strong imperatives to optimise the ways in which students acquire clinical expertise. Deliberate practice is characterised by attention, concentration, effort and repetition of skills; it is an important tool for developing and maintaining professional expertise. Research has led to a greater understanding of how medical students develop core clinical skills, especially in the areas of diagnostic reasoning, communication and physical examination. Advances in information technology and instructional design are helping to strengthen the links between formal educational activities and opportunistic learning in the clinical setting.

The requirement for bioscience knowledge in medical education.

The recent 100-year anniversary of the Flexner review and the release of the Australian Medical Education Study have stimulated vigorous debate about the role of bioscience knowledge in medical education. Two critical questions define debate in this area: does bioscience learning assist in educating medical students to become competent doctors, and, if so, what are the most effective teaching and learning methods to facilitate this outcome? There is tacit acceptance that specific bioscience knowledge is critical for the development of clinical expertise; however, there are few empirical data to support this notion. Two differing theories have been proposed to describe the role of bioscience learning in the development of clinical reasoning skills - the "two-worlds" model and the "encapsulation" model. A series of studies provides support for the encapsulation model. Some medical programs are now integrating bioscience teaching into the clinical years of the course. Evidence of the effectiveness of this on outcomes, such as improved clinical reasoning, is inconclusive.

Clinical assessment performance of graduate- and undergraduate-entry medical students.

BACKGROUND: Recent evidence suggests that graduate-entry medical students may have a marginal academic performance advantage over undergraduate entrants in a pre-clinical curriculum in both bioscience knowledge and clinical skills assessments. It is unclear whether this advantage is maintained in the clinical phase of medical training. AIM: The study aimed to compare graduate and undergraduate entrants undertaking an identical clinical curriculum on assessments undertaken during clinical training in the medical course. METHODS: Clinical assessment results for four cohorts of medical students (n = 713) were compared at the beginning and at the end of clinical training for graduate and undergraduate entrants. RESULTS: Results showed that graduate- and undergraduate-entry medical students performed similarly on clinical assessments. Female students performed consistently better than male students. CONCLUSION: The findings of this study suggest that any academic performance advantage held by graduate-entry medical students is limited to the early years of the medical course, and is not evident during clinical training in the later years of the course.

A mandatory intercalated degree programme: revitalising and enhancing academic and evidence-based medicine.

BACKGROUND: Recruitment of medical graduates to research careers is declining. Expansion of medical knowledge necessitates all graduates be equipped to critically evaluate new information. To address these challenges, a mandatory intercalated degree programme was introduced as part of curriculum reform. AIMS: To review the place on intercalated degrees, the methods available for learning about research and to analyse experience with a new university programme focusing on research. METHODS: A literature review followed by the analysis of experience with eight cohorts of students who had completed the new programme. RESULTS: A total of 1599 students completed the programme. Laboratory-based research was the most common choice followed by clinical research, population health, epidemiology, medical humanities and mental health. Also, 93% of students spent over 75% of their time undertaking research. Sixty-three students published their research, half as first authors. Students and coordinators support the programme. Learning about research during the postgraduate phase is variable and frequently left to individual choice. CONCLUSION: Intercalating an additional degree focusing on research can achieve a number of learning objectives but demands a level of maturity, autonomy and preparedness, not uniformly present in students undertaking a mandatory intercalated programme. A more realistic goal is the development of 'research-mindedness' amongst all students.