Knowledge Attainment and Engagement Among Medical Students: A Comparison of Three Forms of Online Learning.

Objective: This study compared knowledge attainment and student enjoyment and engagement between clinical case vignette, patient-testimony videos and mixed reality (MR) teaching via the Microsoft HoloLens 2, all delivered remotely to third year medical students. The feasibility of conducting MR teaching on a large scale was also assessed. Setting & Participants: Medical students in Year 3 at Imperial College London participated in three online teaching sessions, one in each format. All students were expected to attend these scheduled teaching sessions and to complete the formative assessment. Inclusion of their data used as part of the research trial was optional. Primary and Secondary Outcome Measures: The primary outcome measure was performance on a formative assessment, which served to compare knowledge attainment between three forms of online learning. Moreover, we aimed to explore student engagement with each form of learning via a questionnaire, and also feasibility of applying MR as a teaching tool on a large scale. Comparisons between performances on the formative assessment between the three groups were investigated using a repeated measures two-way ANOVA. Engagement and enjoyment were also analysed in the same manner. Results: A total of 252 students participated in the study. Knowledge attainment of students using MR was comparable with the other two methods. Participants reported higher enjoyment and engagement (p<0.001) for the case vignette method, compared with MR and video-based teaching. There was no difference in enjoyment or engagement ratings between MR and the video-based methods. Conclusion: This study demonstrated that the implementation of MR is an effective, acceptable, and feasible way of teaching clinical medicine to undergraduate students on a large scale. However, case-based tutorials were found to be favoured most by students. Future work could further explore the best uses for MR teaching within the medical curriculum.

The impact of patient skin colour on diagnostic ability and confidence of medical students.

Previous literature has explored unconscious racial biases in clinical education and medicine, finding that people with darker skin tones can be underrepresented in learning resources and managed differently in a clinical setting. This study aimed to examine whether patient skin colour can affect the diagnostic ability and confidence of medical students, and their cognitive reasoning processes. We presented students with 12 different clinical presentations on both white skin (WS) and non-white skin (NWS). A think aloud (TA) study was conducted to explore students' cognitive reasoning processes (n = 8). An online quiz was also conducted where students submitted a diagnosis and confidence level for each clinical presentation (n = 185). In the TA interviews, students used similar levels of information gathering and analytical reasoning for each skin type but appeared to display increased uncertainty and reduced non-analytical reasoning methods for the NWS images compared to the WS images. In the online quiz, students were significantly more likely to accurately diagnose five of the 12 clinical presentations (shingles, cellulitis, Lyme disease, eczema and meningococcal disease) on WS compared to NWS (p < 0.01). With regards to students' confidence, they were significantly more confident diagnosing eight of the 12 clinical presentations (shingles, cellulitis, Lyme disease, eczema, meningococcal disease, urticaria, chickenpox and Kawasaki disease) on WS when compared to NWS (p < 0.01). These findings highlight the need to improve teaching resources to include a greater diversity of skin colours exhibiting clinical signs, to improve students' knowledge and confidence, and ultimately, to avoid patients being misdiagnosed due to the colour of their skin.

Is the assumption of equal distances between global assessment categories used in borderline regression valid?

BACKGROUND: Standard setting for clinical examinations typically uses the borderline regression method to set the pass mark. An assumption made in using this method is that there are equal intervals between global ratings (GR) (e.g. Fail, Borderline Pass, Clear Pass, Good and Excellent). However, this assumption has never been tested in the medical literature to the best of our knowledge. We examine if the assumption of equal intervals between GR is met, and the potential implications for student outcomes. METHODS: Clinical finals examiners were recruited across two institutions to place the typical 'Borderline Pass', 'Clear Pass' and 'Good' candidate on a continuous slider scale between a typical 'Fail' candidate at point 0 and a typical 'Excellent' candidate at point 1. Results were analysed using one-sample t-testing of each interval to an equal interval size of 0.25. Secondary data analysis was performed on summative assessment scores for 94 clinical stations and 1191 medical student examination outcomes in the final 2 years of study at a single centre. RESULTS: On a scale from 0.00 (Fail) to 1.00 (Excellent), mean examiner GRs for 'Borderline Pass', 'Clear Pass' and 'Good' were 0.33, 0.55 and 0.77 respectively. All of the four intervals between GRs (Fail-Borderline Pass, Borderline Pass-Clear Pass, Clear Pass-Good, Good-Excellent) were statistically significantly different to the expected value of 0.25 (all p-values < 0.0125). An ordinal linear regression using mean examiner GRs was performed for each of the 94 stations, to determine pass marks out of 24. This increased pass marks for all 94 stations compared with the original GR locations (mean increase 0.21), and caused one additional fail by overall exam pass mark (out of 1191 students) and 92 additional station fails (out of 11,346 stations). CONCLUSIONS: Although the current assumption of equal intervals between GRs across the performance spectrum is not met, and an adjusted regression equation causes an increase in station pass marks, the effect on overall exam pass/fail outcomes is modest.

Standard setting Very Short Answer Questions (VSAQs) relative to Single Best Answer Questions (SBAQs): does having access to the answers make a difference?

BACKGROUND: We investigated whether question format and access to the correct answers affect the pass mark set by standard-setters on written examinations. METHODS: Trained educators used the Angoff method to standard set two 50-item tests with identical vignettes, one in a single best answer question (SBAQ) format (with five answer options) and the other in a very short answer question (VSAQ) format (requiring free text responses). Half the participants had access to the correct answers and half did not. The data for each group were analysed to determine if the question format or having access to the answers affected the pass mark set. RESULTS: A lower pass mark was set for the VSAQ test than the SBAQ test by the standard setters who had access to the answers (median difference of 13.85 percentage points, Z = -2.82, p = 0.002). Comparable pass marks were set for the SBAQ test by standard setters with and without access to the correct answers (60.65% and 60.90% respectively). A lower pass mark was set for the VSAQ test when participants had access to the correct answers (difference in medians -13.75 percentage points, Z = 2.46, p = 0.014). CONCLUSIONS: When given access to the potential correct answers, standard setters appear to appreciate the increased difficulty of VSAQs compared to SBAQs.

Using very short answer errors to guide teaching.

BACKGROUND: Student performance in examinations reflects on both teaching and student learning. Very short answer questions require students to provide a self-generated response to a question of between one and five words, which removes the cueing effects of single best answer format examinations while still enabling efficient machine marking. The aim of this study was to pilot a method of analysing student errors in an applied knowledge test consisting of very short answer questions, which would enable identification of common areas that could potentially guide future teaching. METHODS: We analysed the incorrect answers given by 1417 students from 20 UK medical schools in a formative very short answer question assessment delivered online. FINDINGS: The analysis identified four predominant types of error: inability to identify the most important abnormal value, over or unnecessary investigation, lack of specificity of radiology requesting and over-reliance on trigger words. CONCLUSIONS: We provide evidence that an additional benefit to the very short answer question format examination is that analysis of errors is possible. Further assessment is required to determine if altering teaching based on the error analysis can lead to improvements in student performance.

Thinking differently - Students' cognitive processes when answering two different formats of written question.

BACKGROUND: Single-best answer questions (SBAQs) are common but are susceptible to cueing. Very short answer questions (VSAQs) could be an alternative, and we sought to determine if students' cognitive processes varied across question types and whether students with different performance levels used different methods for answering questions. METHODS: We undertook a 'think aloud' study, interviewing 21 final year medical students at five UK medical schools. Each student described their thought processes and methods used for eight questions of each type. Responses were coded and quantified to determine the relative frequency with which each method was used, denominated on the number of times a method could have been used. RESULTS: Students were more likely to use analytical reasoning methods (specifically identifying key features) when answering VSAQs. The use of test-taking behaviours was more common for SBAQs; students frequently used the answer options to help them reach an answer. Students acknowledged uncertainty more frequently when answering VSAQs. Analytical reasoning was more commonly used by high-performing students compared with low-performing students. CONCLUSIONS: Our results suggest that VSAQs encourage more authentic clinical reasoning strategies. Differences in cognitive approaches used highlight the need for focused approaches to teaching clinical reasoning and dealing with uncertainty.

National inter-rater agreement of standardised simulated-patient-based assessments.

PURPOSE: The forthcoming UK Medical Licensing Assessment will require all medical schools in the UK to ensure that their students pass an appropriately designed Clinical and Professional Skills Assessment (CPSA) prior to graduation and registration with a licence to practice medicine. The requirements for the CPSA will be set by the General Medical Council, but individual medical schools will be responsible for implementing their own assessments. It is therefore important that assessors from different medical schools across the UK agree on what standard of performance constitutes a fail, pass or good grade. METHODS: We used an experimental video-based, single-blinded, randomised, internet-based design. We created videos of simulated student performances of a clinical examination at four scripted standards: clear fail (CF), borderline (BD), clear pass (CPX) and good (GD). Assessors from ten regions across the UK were randomly assigned to watch five videos in 12 different combinations and asked to give competence domain scores and an overall global grade for each simulated candidate. The inter-rater agreement as measured by the intraclass correlation coefficient (ICC) based on a two-way random-effects model for absolute agreement was calculated for the total domain scores. RESULTS: 120 assessors enrolled in the study, with 98 eligible for analysis. The ICC was 0.93 (95% CI 0.81-0.99). The mean percentage agreement with the scripted global grade was 74.4% (range 40.8-96.9%). CONCLUSIONS: The inter-rater agreement amongst assessors across the UK when rating simulated candidates performing at scripted levels is excellent. The level of agreement for the overall global performance level for simulated candidates is also high. These findings suggest that assessors from across the UK viewing the same simulated performances show high levels of agreement of the standards expected of students at a 'clear fail,' 'borderline,' 'clear pass' and 'good' level.

Systematic review of stepped wedge cluster randomized trials shows that design is particularly used to evaluate interventions during routine implementation.

OBJECTIVE: To describe the application of the stepped wedge cluster randomized controlled trial (CRCT) design. STUDY DESIGN AND SETTING: Systematic review. We searched Medline, Embase, PsycINFO, HMIC, CINAHL, Cochrane Library, Web of Knowledge, and Current Controlled Trials Register for articles published up to January 2010. Stepped wedge CRCTs from all fields of research were included. Two authors independently reviewed and extracted data from the studies. RESULTS: Twenty-five studies were included in the review. Motivations for using the design included ethical, logistical, financial, social, and political acceptability and methodological reasons. Most studies were evaluating an intervention during routine implementation. For most of the included studies, there was also a belief or empirical evidence suggesting that the intervention would do more good than harm. There was variation in data analysis methods and insufficient quality of reporting. CONCLUSIONS: The stepped wedge CRCT design has been mainly used for evaluating interventions during routine implementation, particularly for interventions that have been shown to be effective in more controlled research settings, or where there is lack of evidence of effectiveness but there is a strong belief that they will do more good than harm. There is need for consistent data analysis and reporting.

Antenatal care and perinatal outcomes in Kwale district, Kenya.

BACKGROUND: The importance of antenatal care (ANC) for improving perinatal outcomes is well established. However access to ANC in Kenya has hardly changed in the past 20 years. This study aims to identify the determinants of attending ANC and the association between attendance and behavioural and perinatal outcomes (live births and healthy birthweight) for women in the Kwale region of Kenya. METHOD: A Cohort survey of 1,562 perinatal outcomes (response rate 100%) during 2004-05 in the catchment areas for five Ministry of Health dispensaries in two divisions of the Kwale region. The associations between background and behavioural decisions on ANC attendance and perinatal outcomes were explored using univariate analysis and multivariate logistic regression models with backwards-stepwise elimination. The outputs from these analyses were reported as odds ratios (OR) with 95% confidence intervals (CI). RESULTS: Only 32% (506/1,562) of women reported having any ANC. Women with secondary education or above (adjusted OR 1.83; 95% CI 1.06-3.15) were more likely to attend for ANC, while those living further than 5 km from a dispensary were less likely to attend (OR 0.29; 95% CI 0.22-0.39). Paradoxically, however, the number of ANC visits increased with distance from the dispensary (OR 1.46; 95% CI 1.33-1.60). Women attending ANC at least twice were more likely to have a live birth (vs. stillbirth) in both multivariate models. Women attending for two ANC visits (but not more than two) were more likely to have a healthy weight baby (OR 4.39; 95% CI 1.36-14.15). CONCLUSION: The low attendance for ANC, combined with a positive relationship between attendance and perinatal outcomes for the women in the Kwale region highlight the need for further research to understand reasons for attendance and non-attendance and also for strategies to be put in place to improve attendance for ANC.

Cost-utility analysis when not everyone wants the treatment: modeling split-choice bias.

Not all clinically eligible patients will necessarily accept a new treatment. Cost-utility analysis recognizes this by multiplying the mean incremental expected utility (EU) by the participation rate to obtain the utility gain per head. However, the mean EU gain over all patients in a defined clinical category is traditionally used as a proxy for the mean EU gain over the subpopulation of acceptors. Even for clinically identical patients, this may lead to a biased assessment of total benefit because a patient motivated to accept the new treatment is likely to value its effects more favorably than a patient who declines. An analysis that ignores this tendency will be biased toward an underestimate of true benefits of a health technology (HT). The extent of this bias is described within a quality-adjusted life year-based utility model for a population of clinically indistinguishable patients who differ with respect to the values that they place on the possible health outcomes of an HT. The size of the bias is sensitive to the proportion of patients who accept the treatment, under both deterministic and probabilistic models of individual decision making. In all cases in which decision making is correlated with personal utility gain, the bias rises steeply as the proportion of acceptors declines.

The stepped wedge trial design: a systematic review.

BACKGROUND: Stepped wedge randomised trial designs involve sequential roll-out of an intervention to participants (individuals or clusters) over a number of time periods. By the end of the study, all participants will have received the intervention, although the order in which participants receive the intervention is determined at random. The design is particularly relevant where it is predicted that the intervention will do more good than harm (making a parallel design, in which certain participants do not receive the intervention unethical) and/or where, for logistical, practical or financial reasons, it is impossible to deliver the intervention simultaneously to all participants. Stepped wedge designs offer a number of opportunities for data analysis, particularly for modelling the effect of time on the effectiveness of an intervention. This paper presents a review of 12 studies (or protocols) that use (or plan to use) a stepped wedge design. One aim of the review is to highlight the potential for the stepped wedge design, given its infrequent use to date. METHODS: Comprehensive literature review of studies or protocols using a stepped wedge design. Data were extracted from the studies in three categories for subsequent consideration: study information (epidemiology, intervention, number of participants), reasons for using a stepped wedge design and methods of data analysis. RESULTS: The 12 studies included in this review describe evaluations of a wide range of interventions, across different diseases in different settings. However the stepped wedge design appears to have found a niche for evaluating interventions in developing countries, specifically those concerned with HIV. There were few consistent motivations for employing a stepped wedge design or methods of data analysis across studies. The methodological descriptions of stepped wedge studies, including methods of randomisation, sample size calculations and methods of analysis, are not always complete. CONCLUSION: While the stepped wedge design offers a number of opportunities for use in future evaluations, a more consistent approach to reporting and data analysis is required.