A realist evaluation of how, why and when objective structured clinical exams (OSCEs) are experienced as an authentic assessment of clinical preparedness.

INTRODUCTION: Whilst rarely researched, the authenticity with which Objective Structured Clinical Exams (OSCEs) simulate practice is arguably critical to making valid judgements about candidates' preparedness to progress in their training. We studied how and why an OSCE gave rise to different experiences of authenticity for different participants under different circumstances. METHODS: We used Realist evaluation, collecting data through interviews/focus groups from participants across four UK medical schools who participated in an OSCE which aimed to enhance authenticity. RESULTS: Several features of OSCE stations (realistic, complex, complete cases, sufficient time, autonomy, props, guidelines, limited examiner interaction etc) combined to enable students to project into their future roles, judge and integrate information, consider their actions and act naturally. When this occurred, their performances felt like an authentic representation of their clinical practice. This didn't work all the time: focusing on unavoidable differences with practice, incongruous features, anxiety and preoccupation with examiners' expectations sometimes disrupted immersion, producing inauthenticity. CONCLUSIONS: The perception of authenticity in OSCEs appears to originate from an interaction of station design with individual preferences and contextual expectations. Whilst tentatively suggesting ways to promote authenticity, more understanding is needed of candidates' interaction with simulation and scenario immersion in summative assessment.

Understanding and developing procedures for video-based assessment in medical education.

INTRODUCTION: Novel uses of video aim to enhance assessment in health-professionals education. Whilst these uses presume equivalence between video and live scoring, some research suggests that poorly understood variations could challenge validity. We aimed to understand examiners' and students' interaction with video whilst developing procedures to promote its optimal use. METHODS: Using design-based research we developed theory and procedures for video use in assessment, iteratively adapting conditions across simulated OSCE stations. We explored examiners' and students' perceptions using think-aloud, interviews and focus group. Data were analysed using constructivist grounded-theory methods. RESULTS: Video-based assessment produced detachment and reduced volitional control for examiners. Examiners ability to make valid video-based judgements was mediated by the interaction of station content and specifically selected filming parameters. Examiners displayed several judgemental tendencies which helped them manage videos' limitations but could also bias judgements in some circumstances. Students rarely found carefully-placed cameras intrusive and considered filming acceptable if adequately justified. DISCUSSION: Successful use of video-based assessment relies on balancing the need to ensure station-specific information adequacy; avoiding disruptive intrusion; and the degree of justification provided by video's educational purpose. Video has the potential to enhance assessment validity and students' learning when an appropriate balance is achieved.

The real costs of teaching medical students in general practice: a cost-collection survey of teaching practices across England.

BACKGROUND: Current funding arrangements for undergraduate medical student placements in general practice are widely regarded as outdated, inequitable, and in need of urgent review. AIM: To undertake a detailed costing exercise to inform the setting of a national English tariff for undergraduate medical student placements in general practice. DESIGN AND SETTING: A cost-collection survey in teaching practices across all regions of England between January 2017 and February 2017. METHOD: A cost-collection template was sent to 50 selected teaching practices across all 25 medical schools in England following the development of a cost-collection tool and an initial pilot study. Detailed guidance on completion was provided for practices. Data were analysed by the Department of Health and Social Care. RESULTS: A total of 49 practices submitted data. The mean cost per half-day student placement in general practice was 111 GBP, 95% confidence interval = 100 to 121 (146 USD), with small differences between students in different years of study. Based on 10 sessions per student per week this equated to around 1100 GBP (1460 USD) per student placement week. CONCLUSION: The costs of undergraduate placements in general practice are considerably greater than funding available at time of writing, and broadly comparable with secondary care funding in the same period. The actual cost of placing a medical student full time in general practice for a 37-week academic year is 40 700 GBP (53 640 USD) compared with the average payment rate of only 22 000 GBP (28 990 USD) per year at the time this study was undertaken.

Utility of an app-based system to improve feedback following workplace-based assessment.

OBJECTIVES: To determine whether an app-based software system to support production and storage of assessment feedback summaries makes workplace-based assessment easier for clinical tutors and enhances the educational impact on medical students. METHODS: We monitored our workplace assessor app's usage by Year 3 to 5 medical students in 2014-15 and conducted focus groups with Year 4 medical students and interviews with clinical tutors who had used the apps. Analysis was by constant comparison using a framework based on elements of van der Vleuten's utility index. RESULTS: The app may enhance the content of feedback for students. Using a screen may be distracting if the app is used during feedback discussions.    Educational impact was reduced by students' perceptions that an easy-to-produce feedback summary is less valuable than one requiring more tutor time and effort. Tutors' typing, dictation skills and their familiarity with mobile devices varied. This influenced their willingness to use the assessment and feedback mobile app rather than the equivalent web app. Electronic feedback summaries had more real and perceived uses than anticipated both for tutors and students including perceptions that they were for the school rather than the student. CONCLUSIONS: Electronic workplace-based assessment systems can be acceptable to tutors and can make giving detailed written feedback more practical but can interrupt the social interaction required for the feedback conversation. Tutor training and flexible systems will be required to minimise unwanted consequences. The educational impact on both tutors and students of providing pre-formulated advice within the app is worth further study.

Does formal workplace based assessment add value to informal feedback?

This article was migrated. The article was marked as recommended. Feedback is a key component of learning but effective feedback is a complex process with many aspects. One aspect may be a written summary which is passed to the learner but this may not be valued by learners. We examined the role of written feedback in the feedback process to determine whether it does more than provide a simple summary of the interaction. We conducted a secondary analysis of data gathered for a study of formative workplace based assessment. Interview data from 24 interviews with students and written summaries of workplace based assessments for 23 of them were reanalysed by two researchers who were already immersed in the data and examined all references to verbal, informal feedback and written, formal feedback or the assessment tool used. We found that students valued the verbal feedback discussion highly and that they often considered the written summaries superfluous. We also found that the act of preparing written feedback augmented the feedback discussion and tutors had adopted the language of the formal instrument in the verbal feedback and free text written feedback. What this study adds to existing research is evidence that there may be a secondary faculty development effect of requiring the preparation of written feedback which has served to enhance the educational content of feedback. Although this is not proof of causality (the requirement to provide written feedback alone producing the positive effects), we consider that the likelihood is sufficiently strong to continue the practice.

The epidemiology of teaching and training General Practices in England.

There is no national picture of teaching and training practices or the communities they serve. We aimed to describe the association between general practices' engagement with education and their characteristics, locality and patients' health-status and satisfaction. This data linkage study of all English practices calculated odds ratios for teaching and training status and practice, locality and patient variables. Teaching and training practices are larger than practices which do neither (mean list size (SD) 7074 (3736), 10112 (4934), and 5327 (3368) respectively, p < 0.001 and have fewer patients per GP (1932 (951), 1838 (544), and 2117 (1585) respectively, p < 0.001). Their localities have a higher proportion of White British residents (77.99% (24.17), 81.66% (20.81), 73.07% (26.91), p < 0.001). Practices with more GPs (OR 1.21 (95%CI 1.18-1.20)), fewer male GPs (0.45 (0.36-0.55)) and a higher proportion of White British people in their locality (1.30 (1.06-1.60)) were more likely to teach. Practices in rural areas (1.68 (1.43-1.98)), with more GPs (1.22 (1.27-1.39)), more full time equivalent GPs (2.68 (1.64-4.40)), fewer male GPs (0.17 (0.13-0.22)) and a higher proportion of White British people in their locality (1.34 (1.02-1.75)) were more likely to train. Teaching and training practices had higher patient satisfaction (0.293 (0.190, 0.397) and (0.563 (0.442, 0.685)) respectively and quality and outcomes framework scores (0.507 (0.211, 0.804)) and (0.996 (0.650, 1.342)) respectively than those which did not. Educationally engaged practices are unrepresentative in serving less ethnically diverse and (for training practices) less urban environments. Investment is needed to increase the proportion of educational practices in diverse urban localities.

The investment of a new medical school in its local primary care community.

BACKGROUND: Keele School of Medicine has an innovative new MBChB curriculum with a strong focus on primary care. Our students spend a minimum of 115 days in general practice. AIM: To describe the investment by the school in the local primary care community. METHOD: We collated data on list sizes of current undergraduate teaching practices, county populations and the investment in practices' teaching skills and physical premises. RESULTS: Between academic years 2010-11 and 2012-13 a total of 118 (annual mean of 99) practices contributed to teaching across six counties, providing care for 655 229 people. Forty-five per cent of 243 practices in Staffordshire and Shropshire teach. They serve 51% of the two counties' population. We have invested £1.62 million in the premises of 25 practices providing services for 18% of people in Staffordshire and Shropshire, £273 000 in a rural campus in Shropshire, and £99 387 in set-up grants. We have conducted 249 practice development visits. There were 540 attendances at tutor development sessions. We have had a total of 2300 hours of face-to-face contact with our practices. CONCLUSION: The school has made a major investment in general practice in its area, contributing to the development of general practitioners and investing in practice premises.

Grades in formative workplace-based assessment: a study of what works for whom and why.

CONTEXT: Grades are commonly used in formative workplace-based assessment (WBA) in medical education and training, but may draw attention away from feedback about the task. A dilemma arises because the self-regulatory focus of a trainee must include self-awareness relative to agreed standards, which implies grading. OBJECTIVES: In this study we aimed to understand the meaning which medical students construct from WBA feedback with and without grades, and what influences this. METHODS: Year 3 students were invited to take part in a randomised crossover study in which each student served as his or her own control. Each student undertook one WBA with and one without grades, and then chose whether or not to be given grades in a third WBA. These preferences were explored in semi-structured interviews. A realist approach to analysis was used to gain understanding of student preferences and the impact of feedback with and without grades. RESULTS: Of 83 students who were given feedback with and without grades, 65 (78%) then chose to have feedback with grades and 18 (22%) without grades in their third WBA. A total of 24 students were interviewed. Students described how grades locate their performance and calibrate their self-assessment. For some, low grades focused attention and effort. Satisfactory and high grades enhanced self-efficacy. CONCLUSIONS: Grades are concrete, powerful and blunt, can be harmful and need to be explained to help students create helpful meaning from them. Low grades risk reducing self-efficacy in some and may encourage others to focus on proving their ability rather than on improvement. A metaphor of the semi-permeable membrane is introduced to elucidate how students reduced potential negative effects and enhanced the positive effects of feedback with grades by selective filtering and pumping. This study illuminates the complexity of the processing of feedback by its recipients, and informs the use of grading in the provision of more effective, tailored feedback.

Development of a tool to support holistic generic assessment of clinical procedure skills.

CONTEXT: The challenges of maintaining comprehensive banks of valid checklists make context-specific checklists for assessment of clinical procedural skills problematic. OBJECTIVES: This paper reports the development of a tool which supports generic holistic assessment of clinical procedural skills. METHODS: We carried out a literature review, focus groups and non-participant observation of assessments with interview of participants, participant evaluation of a pilot objective structured clinical examination (OSCE), a national modified Delphi study with prior definitions of consensus and an OSCE. Participants were volunteers from a large acute teaching trust, a teaching primary care trust and a national sample of National Health Service staff. Results In total, 86 students, trainees and staff took part in the focus groups, observation of assessments and pilot OSCE, 252 in the Delphi study and 46 candidates and 50 assessors in the final OSCE. We developed a prototype tool with 5 broad categories amongst which were distributed 38 component competencies. There was > 70% agreement (our prior definition of consensus) at the first round of the Delphi study for inclusion of all categories and themes and no consensus for inclusion of additional categories or themes. Generalisability was 0.76. An OSCE based on the instrument has a predicted reliability of 0.79 with 12 stations and 1 assessor per station or 10 stations and 2 assessors per station. CONCLUSIONS: This clinical procedural skills assessment tool enables reliable assessment and has content and face validity for the assessment of clinical procedural skills. We have designated it the Leicester Clinical Procedure Assessment Tool (LCAT).

Checklists for assessment and certification of clinical procedural skills omit essential competencies: a systematic review.

OBJECTIVE: To develop generic criteria for the global assessment of clinical procedural competence and to quantify the extent to which existing checklists allow for holistic assessment of procedural competencies. METHODS: We carried out a systematic review and qualitative analysis of published clinical procedural skills assessment checklists and enumerated the contents of each. Source materials included all English-language papers published from 1990 to June 2005, identified from 18 databases, which described or referred to an assessment document for any clinical procedural skill. A pair of reviewers identified key generic themes and sub-themes through in-depth analysis of a subset of 20 checklists with iterative agreement and independent retesting of a coding framework. The resulting framework was independently applied to all checklists by pairs of reviewers checking for the emergence of new themes and sub-themes. Main outcome measures were identification of generic clinical procedural skills and the frequency of occurrence of each in the identified checklists. RESULTS: We identified 7 themes ('Procedural competence', represented in 85 [97%] checklists; 'Preparation', 65 [74%]; 'Safety', 45 [51%]; 'Communication and working with the patient', 32 [36%]; 'Infection control', 28 [32%]; 'Post-procedural care', 24 [27%]; 'Team working', 13 [15%]) and 37 sub-themes, which encapsulated all identified checklists. Of the sub-themes, 2 were identified after the initial coding framework had been finalised. CONCLUSIONS: It is possible to develop generic criteria for the global assessment of clinical procedural skills. A third and a half of checklists, respectively, do not enable explicit assessment of the key competencies 'Infection control' and 'Safety'. Their assessment may be inconsistent in assessments which use such checklists.

Regulatory end-point assessment of the consultation competence of family practice trainees in Kuwait.

BACKGROUND: No single approach to the regulatory assessment of global consultation competence has been shown to possess the required levels of validity, reliability and feasibility. OBJECTIVE: To evaluate the approach adopted in Kuwait to the regulatory end-point assessment of the global consultation competence of family practice trainees with particular reference to validity, reliability and feasibility. METHODS: Family practice trainees in Kuwait were individually and directly observed for 3 hours in consultation with a minimum of 10 patients by a pair of examiners. Performance was judged against the explicit criteria of consultation competence as contained in the Leicester Assessment Package (LAP). RESULTS: The marks independently allocated by the pairs of examiners to 126 trainees between 1994 and 2001 were within five percentage points on 91% of occasions. A reliability coefficient of 0.82 was achieved when two examiners independently marked candidates consulting with 10 real patients; this rose to 0.95 at the critical 50% pass-fail margin. The main sources of variance contributing to the reliability of marks allocated were candidate performance (42%) and the interaction of candidate performance across cases, i.e., case specificity (30%). The clinical challenges presented by the patients were judged by both examiners to be sufficient to enable performance to be assessed across the seven LAP consultation categories as follows: behaviour and relationship with patients (100% of consultations), interviewing/history taking (100%), record keeping (99%), patient management (99%), problem solving (98%), physical examination (95%), and anticipatory care (86%). Each assessment involved a pair of examiners and lasted approximately 3.5 hours. CONCLUSION: The Kuwait clinical examination achieves high content validity and authenticity as it uses direct observation of performance, validated and explicit criteria against which performance is judged, and real patient challenges. It can discriminate between different levels of consultation performance and satisfies the recognized reliability threshold for regulatory examinations (0.82 vs 0.80). Accordingly, we recommend the use of such an approach in the regulatory end-point assessment of the global consultation competence of trainees in family practice. Such an approach is more valid, and is likely to be more feasible, than simulated surgeries or the short-case OSCE format.