UK Prescribing Safety Assessment (PSA): The development, implementation and outcomes of a national online prescribing assessment.

AIMS: The United Kingdom (UK) Prescribing Safety Assessment (PSA) is a 2-h online assessment of basic competence to prescribe and supervise the use of medicines. It has been undertaken by students and doctors in UK medical and foundation schools for the past decade. This study describes the academic characteristics and performance of the assessment; longitudinal performance of candidates and schools; stakeholder feedback; and surrogate markers of prescribing safety in UK healthcare practice. METHODS: We reviewed the performance data generated by over 70 000 medical students and 3700 foundation doctors who have participated in the PSA since its inception in 2013. These data were supplemented by Likert scale and free text feedback from candidates and a variety of stakeholder groups. Further data on medication incidents, collected by national reporting systems and the regulatory body, are reported, with permission. RESULTS: We demonstrate the feasibility, high quality and reliability of an online prescribing assessment, uniquely providing a measure of prescribing competence against a national standard. Over 90% of candidates pass the PSA on their first attempt, while a minority are identified for further training and assessment. The pass rate shows some variation between different institutions and between undergraduate and foundation cohorts. Most responders to a national survey agreed that the PSA is a useful instrument for assessing prescribing competence, and an independent review has recommended adding the PSA to the Medical Licensing Assessment. Surrogate markers suggest there has been improvement in prescribing safety in practice, temporally associated with the introduction of the PSA but other factors could be influential too. CONCLUSIONS: The PSA is a practical and cost-effective way of delivering a reliable national assessment of prescribing competence that has educational impact and is supported by the majority of stakeholders. There is a need to develop national systems to identify and report prescribing errors and the harm they cause, enabling the impact of educational interventions to be measured.

The influence of candidates' race on examiners' ratings in standardised assessments of clinical practice.

PURPOSE: Delivering fair and reliable summative assessments in medical education assumes examiner decision making is devoid of bias. We investigated whether candidate racial appearances influenced examiner ratings in undergraduate clinical exams. METHODS: We used an internet-based design. Examiners watched a randomised set of six videos of three different white candidates and three different non-white (Asian, black and Chinese) candidates taking a clinical history at either fail, borderline or pass grades. We compared the median and interquartile range (IQR) of the paired difference between scores for the white and non-white candidates at each performance grade and tested for statistical significance. RESULTS: 160 Examiners participated. At the fail grade, the black and Chinese candidates scored lower than the white candidate, with median paired differences of -2.5 and -1 respectively (both p < 0.001). At the borderline grade, the black and Chinese candidates scored higher than the white candidate, with median paired differences of +2 and +3, respectively (both p < 0.001). At the passing grade, the Asian candidate scored lower than the white candidate (median paired difference -1, p < 0.001). CONCLUSION: The racial appearance of candidates appeared to influence the scores awarded by examiners, but not in a uniform manner.

The impact of patient skin colour on diagnostic ability and confidence of medical students.

Previous literature has explored unconscious racial biases in clinical education and medicine, finding that people with darker skin tones can be underrepresented in learning resources and managed differently in a clinical setting. This study aimed to examine whether patient skin colour can affect the diagnostic ability and confidence of medical students, and their cognitive reasoning processes. We presented students with 12 different clinical presentations on both white skin (WS) and non-white skin (NWS). A think aloud (TA) study was conducted to explore students' cognitive reasoning processes (n = 8). An online quiz was also conducted where students submitted a diagnosis and confidence level for each clinical presentation (n = 185). In the TA interviews, students used similar levels of information gathering and analytical reasoning for each skin type but appeared to display increased uncertainty and reduced non-analytical reasoning methods for the NWS images compared to the WS images. In the online quiz, students were significantly more likely to accurately diagnose five of the 12 clinical presentations (shingles, cellulitis, Lyme disease, eczema and meningococcal disease) on WS compared to NWS (p < 0.01). With regards to students' confidence, they were significantly more confident diagnosing eight of the 12 clinical presentations (shingles, cellulitis, Lyme disease, eczema, meningococcal disease, urticaria, chickenpox and Kawasaki disease) on WS when compared to NWS (p < 0.01). These findings highlight the need to improve teaching resources to include a greater diversity of skin colours exhibiting clinical signs, to improve students' knowledge and confidence, and ultimately, to avoid patients being misdiagnosed due to the colour of their skin.

Twelve tips for introducing very short answer questions (VSAQs) into your medical curriculum.

Most undergraduate written examinations use multiple-choice questions, such as single best answer questions (SBAQs) to assess medical knowledge. In recent years, a strong evidence base has emerged for the use of very short answer questions (VSAQs). VSAQs have been shown to be an acceptable, reliable, discriminatory, and cost-effective assessment tool in both formative and summative undergraduate assessments. VSAQs address many of the concerns raised by educators using SBAQs including inauthentic clinical scenarios, cueing and test-taking behaviours by students, as well as the limited feedback SBAQs provide for both students and teachers. The widespread use of VSAQs in medical assessment has yet to be adopted, possibly due to lack of familiarity and experience with this assessment method. The following twelve tips have been constructed using our own practical experience of VSAQs alongside supporting evidence from the literature to help medical educators successfully plan, construct and implement VSAQs within medical curricula.

Are remote clinical assessments a feasible and acceptable method of assessment? A systematic review.

The COVID-19 pandemic presented an enormous and immediate challenge to assessing clinical skills in healthcare professionals. Many institutions were unable to deliver established face-to-face assessment methods such as Objective Structured Clinical Examinations (OSCEs). Assessors needed to rapidly institute alternative assessment methods to ensure that candidates met the clinical competences required for progression. Using a systematic review, we aimed to evaluate the feasibility, and acceptability of remote methods of clinical skills assessment, including remote structured clinical assessments and the submission of video recordings. We searched for studies reporting on Remote Clinical Assessments or its variants in MEDLINE, Embase and The Cochrane library from 2000 to March 2021. Twenty eight studies were included in the review; 20 studies related to remote structured clinical examinations or OSCEs and 8 reported the use of video submissions. The participants of the different studies included medical students, nursing students, dental students and doctors in training. A variety of different online platforms were utilised including Zoom, Skype, webcams, and Adobe Connect online. The studies found that delivery of remote clinical assessments is possible and provides an alternative method of assessing many clinical skills, but most also acknowledge limitations and challenges. They are acceptable to both candidates and examiners, and where measured, show moderate agreement with on-site clinical assessments. Current evidence is based on studies with low methodological quality and for the most part, small sample sizes.

Standard setting Very Short Answer Questions (VSAQs) relative to Single Best Answer Questions (SBAQs): does having access to the answers make a difference?

BACKGROUND: We investigated whether question format and access to the correct answers affect the pass mark set by standard-setters on written examinations. METHODS: Trained educators used the Angoff method to standard set two 50-item tests with identical vignettes, one in a single best answer question (SBAQ) format (with five answer options) and the other in a very short answer question (VSAQ) format (requiring free text responses). Half the participants had access to the correct answers and half did not. The data for each group were analysed to determine if the question format or having access to the answers affected the pass mark set. RESULTS: A lower pass mark was set for the VSAQ test than the SBAQ test by the standard setters who had access to the answers (median difference of 13.85 percentage points, Z = -2.82, p = 0.002). Comparable pass marks were set for the SBAQ test by standard setters with and without access to the correct answers (60.65% and 60.90% respectively). A lower pass mark was set for the VSAQ test when participants had access to the correct answers (difference in medians -13.75 percentage points, Z = 2.46, p = 0.014). CONCLUSIONS: When given access to the potential correct answers, standard setters appear to appreciate the increased difficulty of VSAQs compared to SBAQs.

Thinking differently - Students' cognitive processes when answering two different formats of written question.

BACKGROUND: Single-best answer questions (SBAQs) are common but are susceptible to cueing. Very short answer questions (VSAQs) could be an alternative, and we sought to determine if students' cognitive processes varied across question types and whether students with different performance levels used different methods for answering questions. METHODS: We undertook a 'think aloud' study, interviewing 21 final year medical students at five UK medical schools. Each student described their thought processes and methods used for eight questions of each type. Responses were coded and quantified to determine the relative frequency with which each method was used, denominated on the number of times a method could have been used. RESULTS: Students were more likely to use analytical reasoning methods (specifically identifying key features) when answering VSAQs. The use of test-taking behaviours was more common for SBAQs; students frequently used the answer options to help them reach an answer. Students acknowledged uncertainty more frequently when answering VSAQs. Analytical reasoning was more commonly used by high-performing students compared with low-performing students. CONCLUSIONS: Our results suggest that VSAQs encourage more authentic clinical reasoning strategies. Differences in cognitive approaches used highlight the need for focused approaches to teaching clinical reasoning and dealing with uncertainty.

National inter-rater agreement of standardised simulated-patient-based assessments.

PURPOSE: The forthcoming UK Medical Licensing Assessment will require all medical schools in the UK to ensure that their students pass an appropriately designed Clinical and Professional Skills Assessment (CPSA) prior to graduation and registration with a licence to practice medicine. The requirements for the CPSA will be set by the General Medical Council, but individual medical schools will be responsible for implementing their own assessments. It is therefore important that assessors from different medical schools across the UK agree on what standard of performance constitutes a fail, pass or good grade. METHODS: We used an experimental video-based, single-blinded, randomised, internet-based design. We created videos of simulated student performances of a clinical examination at four scripted standards: clear fail (CF), borderline (BD), clear pass (CPX) and good (GD). Assessors from ten regions across the UK were randomly assigned to watch five videos in 12 different combinations and asked to give competence domain scores and an overall global grade for each simulated candidate. The inter-rater agreement as measured by the intraclass correlation coefficient (ICC) based on a two-way random-effects model for absolute agreement was calculated for the total domain scores. RESULTS: 120 assessors enrolled in the study, with 98 eligible for analysis. The ICC was 0.93 (95% CI 0.81-0.99). The mean percentage agreement with the scripted global grade was 74.4% (range 40.8-96.9%). CONCLUSIONS: The inter-rater agreement amongst assessors across the UK when rating simulated candidates performing at scripted levels is excellent. The level of agreement for the overall global performance level for simulated candidates is also high. These findings suggest that assessors from across the UK viewing the same simulated performances show high levels of agreement of the standards expected of students at a 'clear fail,' 'borderline,' 'clear pass' and 'good' level.

A collaboration on teaching and assessing triadic consultation skills.

Introduction: Teaching triadic consultation skills is becoming increasingly prevalent at medical schools but is included by few schools in summative assessments. We describe a collaboration between Leicester and Cambridge Medical Schools to share teaching practice and the development of an objective structure clinical examination (OSCE) station to assess these important skills. Methods: We agreed on the broad components of the process skills of a triadic consultation and wrote a framework. We used the framework to develop OSCE criteria and suitable case scenarios. The triadic consultation OSCEs were used in our summative assessments at Leicester and Cambridge. Results: Student feedback on teaching was largely positive. The OSCEs at both institutions performed effectively provided a fair and reliable test and had good face validity. Student performance was similar in both schools. Discussion and conclusion: Our collaboration provided peer support and enabled the production of a framework for teaching and assessing triadic consultations that is likely to be generalisable to other medical schools. We were able to reach a consensus on what skills should be included in the teaching of triadic consultations and to co-design an OSCE station to effectively assess those skills. Innovation: Collaboration between two medical schools using a constructive alignment principle allowed the efficient development of effective teaching and assessment of triadic consultations.

Using very short answer errors to guide teaching.

BACKGROUND: Student performance in examinations reflects on both teaching and student learning. Very short answer questions require students to provide a self-generated response to a question of between one and five words, which removes the cueing effects of single best answer format examinations while still enabling efficient machine marking. The aim of this study was to pilot a method of analysing student errors in an applied knowledge test consisting of very short answer questions, which would enable identification of common areas that could potentially guide future teaching. METHODS: We analysed the incorrect answers given by 1417 students from 20 UK medical schools in a formative very short answer question assessment delivered online. FINDINGS: The analysis identified four predominant types of error: inability to identify the most important abnormal value, over or unnecessary investigation, lack of specificity of radiology requesting and over-reliance on trigger words. CONCLUSIONS: We provide evidence that an additional benefit to the very short answer question format examination is that analysis of errors is possible. Further assessment is required to determine if altering teaching based on the error analysis can lead to improvements in student performance.

Is Academic Attainment or Situational Judgment Test Performance in Medical School Associated With the Likelihood of Disciplinary Action? A National Retrospective Cohort Study.

PURPOSE: Disciplinary action imposed on physicians indicates their fitness to practice medicine is impaired and patient safety is potentially at risk. This national retrospective cohort study sought to examine whether there was an association between academic attainment or performance on a situational judgment test (SJT) in medical school and the risk of receiving disciplinary action within the first 5 years of professional practice in the United Kingdom. METHOD: The authors included data from the UK Medical Education Database for 34,865 physicians from 33 U.K. medical schools that started the UK Foundation Programme (similar to internship) between 2014 and 2018. They analyzed data from 2 undergraduate medical assessments used in the United Kingdom: the Educational Performance Measure (EPM), which is based on academic attainment, and SJT, which is an assessment of professional attributes. The authors calculated hazard ratios (HRs) for EPM and SJT scores. RESULTS: The overall rate of disciplinary action was low (65/34,865, 0.19%) and the mean time to discipline was 810 days (standard deviation [SD] = 440). None of the physicians with fitness to practice concerns identified as students went on to receive disciplinary action after they qualified as physicians. The multivariate survival analysis demonstrated that a score increase of 1 SD (approximately 7.6 percentage points) on the EPM reduced the hazard of disciplinary action by approximately 50% (HR = 0.51; 95% confidence interval [CI]: 0.38, 0.69; P < .001). There was not a statistically significant association between the SJT score and the hazard of disciplinary action (HR = 0.84; 95% CI: 0.62, 1.13; P = .24). CONCLUSIONS: An increase in EPM score was significantly associated with a reduced hazard of disciplinary action, whereas performance on the SJT was not. Early identification of increased risk of disciplinary action may provide an opportunity for remediation and avoidance of patient harm.

Diabetic myonecrosis: A case series of two dialysis-dependent patients.

Diabetic myonecrosis (DMN) is a rare microangiopathic disorder that can present as an acutely painful and swollen limb in patients with established diabetes mellitus. The condition can be diagnosed noninvasively with magnetic resonance imaging and resolves with analgesia, bed rest, and glycemic control. Due to a relative lack of awareness regarding the condition, avoidable interventions such as muscle biopsies and even surgery are sometimes pursued, which have been associated with prolonged recovery times. The majority of patients with DMN have diabetic nephropathy, yet this condition is not widely recognized in the nephrology community, resulting in delayed diagnosis and patients undergoing unnecessary and potentially harmful investigations. There is therefore a need for increased awareness of the condition among renal physicians. Here, we report the cases of two patients on hemodialysis who were ultimately diagnosed with DMN, along with a review of the literature.

Comparing single-best-answer and very-short-answer questions for the assessment of applied medical knowledge in 20 UK medical schools: Cross-sectional study.

OBJECTIVES: The study aimed to compare candidate performance between traditional best-of-five single-best-answer (SBA) questions and very-short-answer (VSA) questions, in which candidates must generate their own answers of between one and five words. The primary objective was to determine if the mean positive cue rate for SBAs exceeded the null hypothesis guessing rate of 20%. DESIGN: This was a cross-sectional study undertaken in 2018. SETTING: 20 medical schools in the UK. PARTICIPANTS: 1417 volunteer medical students preparing for their final undergraduate medicine examinations (total eligible population across all UK medical schools approximately 7500). INTERVENTIONS: Students completed a 50-question VSA test, followed immediately by the same test in SBA format, using a novel digital exam delivery platform which also facilitated rapid marking of VSAs. MAIN OUTCOME MEASURES: The main outcome measure was the mean positive cue rate across SBAs: the percentage of students getting the SBA format of the question correct after getting the VSA format incorrect. Internal consistency, item discrimination and the pass rate using Cohen standard setting for VSAs and SBAs were also evaluated, and a cost analysis in terms of marking the VSA was performed. RESULTS: The study was completed by 1417 students. Mean student scores were 21 percentage points higher for SBAs. The mean positive cue rate was 42.7% (95% CI 36.8% to 48.6%), one-sample t-test against ≤20%: t=7.53, p<0.001. Internal consistency was higher for VSAs than SBAs and the median item discrimination equivalent. The estimated marking cost was £2655 ($3500), with 24.5 hours of clinician time required (1.25 s per student per question). CONCLUSIONS: SBA questions can give a false impression of students' competence. VSAs appear to have greater authenticity and can provide useful information regarding students' cognitive errors, helping to improve learning as well as assessment. Electronic delivery and marking of VSAs is feasible and cost-effective.

An educational approach to improve outcomes in acute kidney injury (AKI): report of a quality improvement project.

OBJECTIVE: To assess the impact of a quality improvement project that used a multifaceted educational intervention on how to improve clinician's knowledge, confidence and awareness of acute kidney injury (AKI). SETTING: 2 large acute teaching hospitals in England, serving a combined population of over 1.5 million people. PARTICIPANTS: All secondary care clinicians working in the clinical areas were targeted, with a specific focus on clinicians working in acute admission areas. INTERVENTIONS: A multifaceted educational intervention consisting of traditional didactic lectures, case-based teaching in small groups and an interactive web-based learning resource. OUTCOME MEASURES: We assessed clinicians' knowledge of AKI and their self-reported clinical behaviour using an interactive questionnaire before and after the educational intervention. Secondary outcome measures included clinical audit of patient notes before and after the intervention. RESULTS: 26% of clinicians reported that they were aware of local AKI guidelines in the preintervention questionnaire compared to 64% in the follow-up questionnaire (χ²=60.2, p<0.001). There was an improvement in the number of clinicians reporting satisfactory practice when diagnosing AKI, 50% vs 68% (χ²=12.1, p<0.001) and investigating patients with AKI, 48% vs 64% (χ²=9.5, p=0.002). Clinical audit makers showed a trend towards better clinical practice. CONCLUSIONS: This quality improvement project utilising a multifaceted educational intervention improved awareness of AKI as demonstrated by changes in the clinician's self-reported management of patients with AKI. Elements of the project have been sustained beyond the project period, and demonstrate the power of quality improvement projects to help initiate changes in practice. Our findings are limited by confounding factors and highlight the need to carry out formal randomised studies to determine the impact of educational initiatives in the clinical setting.

Osteoclastomas ('brown tumours') and spinal cord compression: a review.

Brown tumours are an uncommon manifestation of primary and secondary hyperparathyroidism. There are numerous case reports of brown tumours arising in various parts of the skeleton. They can therefore present a wide range of clinical manifestations. A recent case highlighted the need for heightened awareness of the diagnosis and prompted a literature review.