Are serious games seriously good at preparing students for clinical practice?: A randomized controlled trial.

PURPOSE: Serious games (SGs) have great potential for pediatric medical education. This study evaluated the efficacy of a SG in improving learner satisfaction, knowledge, and behavior. MATERIALS AND METHODS: This was an investigator-blinded randomized controlled trial (RCT) comparing a SG against two controls: (i) adaptive tutorial (AT), and (ii) low-stimulus control (LSC). SG is a highly immersive role-playing game in a virtual hospital. AT delivers interactive web-based lessons. LSC is paper-based clinical practice guidelines. Metropolitan senior medical students at UNSW were eligible. A total of 154 enrolled and were block randomized to one intervention. Participants had access to one intervention for 8 weeks which taught pediatric acute asthma and seizure assessment and management. Satisfaction was assessed with Likert-scale responses to 5 statements and 2 free-text comments. Knowledge was assessed with 10 multiple-choice questions (MCQs). Clinical behavior was assessed during a 30-point simulated clinical management scenario (CMS). Primary analysis was performed on a modified intention-to-treat basis and compared: (1) SG vs. AT; and (2) SG vs. LSC. RESULTS: A total of 118 participants were included in the primary analysis (modified intention-to-treat model). No significant differences in MCQ results between the SG and control groups. SG group outperformed the LSC group in the CMS, with a moderate effect (score out of 30: 20.8 (3.2) vs. 18.7 (3.2), respectively, d = 0.65 (0.2-1.1), p = 0.005). No statistically significant difference between SG and AT groups in the CMS (score: 20.8 (3.2) vs. 19.8 (3.1), respectively, d = 0.31 (-0.1 to 0.8), p = 0.18). A sensitivity analysis (per-protocol model) was performed with similar outcomes. CONCLUSIONS: This is the first investigator-blinded RCT assessing the efficacy of a highly immersive SG on learner attitudes, knowledge acquisition, and performance in simulated pediatric clinical scenarios. The SG demonstrated improved translation of knowledge to a simulated clinical environment, particularly compared to LSC. SGs show promise in pediatric medical education.

Empirical analysis comparing the tele-objective structured clinical examination (teleOSCE) and the in-person assessment in Australia.

PURPOSE: It aimed to compare the use of the tele objective structured clinical examination (teleOSCE) with in-person assessment in high-stakes clinical examination so as to determine the impact of the teleOSCE on the assessment undertaken. Discussion follows regarding what skills and domains can effectively be assessed in a teleOSCE. METHODS: This study is a retrospective observational analysis. It compares the results achieved by final year medical students in their clinical examination, assessed using the teleOSCE in 2020 (n=285), with those who were examined using the traditional in-person format in 2019 (n=280). The study was undertaken at the University of New South Wales, Australia. RESULTS: In the domain of physical examination, students in 2020 scored 0.277 points higher than those in 2019 (mean difference -0.277, P<0.001, effect size 0.332). Across all other domains, there was no significant difference in mean scores between 2019 and 2020. CONCLUSION: The teleOSCE does not negatively impact assessment in clinical examination in all domains except physical examination. If the teleOSCE is the future of clinical skills examination, assessment of physical examination will require concomitant workplace-based assessment.

Assessment methods and the validity and reliability of measurement tools in online objective structured clinical examinations: a systematic scoping review

The coronavirus disease 2019 (COVID-19) pandemic has required educators to adapt the in-person objective structured clinical examination (OSCE) to online settings in order for it to remain a critical component of the multifaceted assessment of a student's competency. This systematic scoping review aimed to summarize the assessment methods and validity and reliability of the measurement tools used in current online OSCE (hereafter, referred to as teleOSCE) approaches. A comprehensive literature review was undertaken following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. Articles were eligible if they reported any form of performance assessment, in any field of healthcare, delivered in an online format. Two reviewers independently screened the results and analyzed relevant studies. Eleven articles were included in the analysis. Pre-recorded videos were used in 3 studies, while observations by remote examiners through an online platform were used in 7 studies. Acceptability as perceived by students was reported in 2 studies. This systematic scoping review identified several insights garnered from implementing teleOSCEs, the components transferable from telemedicine, and the need for systemic research to establish the ideal teleOSCE framework. TeleOSCEs may be able to improve the accessibility and reproducibility of clinical assessments and equip students with the requisite skills to effectively practice telemedicine in the future.

Insights into student assessment outcomes in rural clinical campuses.

BACKGROUND: There is an ongoing debate about the impact of studying medicine in rural vs. metropolitan campuses on student assessment outcomes. The UNSW Medicine Rural Clinical School has five main campuses; Albury-Wodonga, Coffs Harbour, Griffith, Port Macquarie and Wagga Wagga. Historical data of student assessment outcomes at these campuses raised concerns regarding potential biases in assessment undertaken, as well as the availability and quality of learning resources. The current study aims to identify the extent to which the location of examination (rural versus metropolitan) has an impact on student marks in OSCEs. METHODS: Assessment data was employed for this study from 275 medical students who sat their final examinations in Years 3 and 6 of the undergraduate Medicine program at UNSW in 2018. The data consists of matched student assessment results from the Year 3 (Y3) MCQ examination and OSCE, and from the Year 6 (Y6) MCQ, OSCE and management viva examinations. The analysis used Univariate Analysis of Variance and linear regression models to identify the impact of site of learning and site of examination on assessment outcomes. RESULTS: The results demonstrate that neither site of learning nor site of examination had any significant impact on OSCE or Management Viva assessment outcomes while potential confounders are controlled. CONCLUSION: It is suggested that some of the supposed disadvantages inherent at rural campuses are effectively mitigated by perceived advantages; more intensive interaction with patients, the general and medical communities at those sites, as well as effective e-learning resources and moderation of assessment grades.

Definition of professionalism and tools for assessing professionalism in pharmacy practice: a systemic review.

PURPOSE: In contemporary pharmacy, the role of pharmacists has become more multifaceted as they now handle a wider range of tasks and take more responsibility for care provision for patients than 20 years ago. Such an evolution in the pharmacist's responsibilities is accompanied by the need for pharmacists to display high-quality patient-centred care and counselling, and to demonstrate professionalism, which now needs to be taught and assessed are part of pharmacy education and practice. This workaimed at identifying definition of professionalism in pharmacy practice and critically evaluatingpublished instruments for assessing professionalism in pharmacy practice. METHODS: We searched the medical literature listed in Scopus, Medline, andPsycINFOdatabasesfrom 1 January 2000 to 31 December 2018. All papers meeting our selection criteria, were reviewed and summarised into a clear review of professionalism requirements in pharmacy practice. Details of the instruments measuring professionalism were reviewed in detail. RESULTS: There is no accepted simple definition of professionalism, although we identified several theoretical and policy frameworks required for professional pharmaceutical practice. We identified 4 instruments (Behavioural Professionalism Assessment Instrument (BPAI), Lerkiatbundit's Instrument, Pharmacy Professionalism Instrument (PPI) and Professionalism Assessment Tool (PAT) that build on these frameworks and measure professional practice in pharmacy students. These were found to be reliable and valid but had only been used and tested in student populations. CONCLUSION: Given the increasing role of community pharmacisits there is a need for assessment of professionalism in practice. Professionalism is a complex concept that is challenging to measure because it has no standardised definition and existing literature related to the topic is limited. Current instruments available focus on measuring the development of the elements of professionalism among pharmacy students rather than pharmacists.

Comparison of effect between simulated patient clinical skill training and student role play on objective structured clinical examination performance outcomes for medical students in Australia.

PURPOSE: Optimal methods for communication skills training (CST) is an active area for research but the effect on communication performance in objective structured clinical examinations (OSCE) has not been so closely studied. Student role play (RP) for CST is common whilst volunteer simulated patient (SP) CST is cost-effective and provides authentic interactions. We assessed whether our volunteer SP CST improved OSCE performance compared to our previous RP strategy. METHODS: We performed a retrospective, quasi-experimental study of two second year medical student cohorts' OSCE data in Australia. The 2014 cohort received RP-only CST (N=182) while the 2016 cohort received SP-only CST (N=148). T-test and ANOVA were used to compare the total scores for three assessment domains: generic communication, clinical communication and physical examination/procedural skills. RESULTS: Baseline characteristics for groups (scores for Australian Tertiary Admission Rank (ATAR), Undergraduate Medicine and Health Sciences Admission Test (UMAT) and medicine program interview) showed no significantt difference between groups. For each domain, the SP-only CST group demonstrated superior outcomes in the OSCE and the difference between cohorts was significant (P < 0.01). Volunteer SP CST was superior to student RP CST when considering OSCE performance outcome., which was found across generic and clinical communication skills, and physical examination/procedural skills. CONCLUSION: Better performance of the SP cohort in physical examination/procedural skills might be explained by the requirement for patient compliance and cooperation, facilitated by good generic communication skills. We recommend a volunteer SP Program as an effective and efficient way to improve CST for junior medical students.

Borderline grades in high stakes clinical examinations: resolving examiner uncertainty.

BACKGROUND: Objective Structured Clinical Exams are used to increase reliability and validity, yet they only achieve a modest level of reliability. This low reliability is due in part to examiner variance which is greater than the variance of students. This variance often represents indecisiveness at the cut score with apparent confusion over terms such as "borderline pass". It is amplified by a well reported failure to fail. METHODS: A borderline grade (meaning performance is neither a clear pass nor a clear fail) was introduced in a high stakes undergraduate medical clinical skills exam to replace a borderline pass grade (which was historically resolved as 50%) in a 4 point scale (distinction, pass, borderline, fail). Each Borderline grade was then resolved into a Pass or Fail grade by a formula referencing the difficulty of the station and the performance in the same domain by the student in other stations. Raw pass or fail grades were unaltered. Mean scores and 95%CI were calculated per station and per domain for the unmodified and the modified scores/grades (results are presented on error bars). To estimate the defensibility of these modifications, similar analysis took place for the P and the F grades which resulted from the modification of the B grades. RESULTS: Of 14,634 observations 4.69% were Borderline. Application of the formula did not impact the mean scores in each domain but the failure rate for the exam increased from 0.7 to 4.1%. Examiners and students expressed satisfaction with the Borderline grade, resolution formula and outcomes. Mean scores (by stations and by domains respectively) of students whose B grades were modified to P were significantly higher than their counterparts whose B grades were modified to F. CONCLUSIONS: This study provides a feasible and defensible resolution to situations where the examinee's performance is neither a clear pass nor a clear fail, demonstrating the application of the resolution of borderline formula in a high stakes exam. It does not create a new performance standard but utilises real data to make judgements about these small number of candidates. This is perceived as a fair approach to Pass/Fail decisions.

Simulated patient and role play methodologies for communication skills training in an undergraduate medical program: Randomized, crossover trial.

Background: Educators utilize real patients, simulated patients (SP), and student role play (RP) in communication skills training (CST) in medical curricula. The chosen modality may depend more on resource availability than educational stage and student needs. In this study, we set out to determine whether an inexpensive volunteer SP program offered an educational advantage compared to RP for CST in preclinical medical students. Methods: Students and volunteer SPs participated in interactions across two courses. Students allocated to SP interactions in one course participated in RP in the other course and vice versa. Audio recordings of interactions were made, and these were rated against criterion descriptors in a modified Calgary-Cambridge Referenced Observation Guide. Results: Independent t-test scores comparing ratings of RP and SP groups revealed no significant differences between methodologies. Discussion: This study demonstrates that volunteer SPs are not superior to RP, when used in CST targeted at preclinical students. This finding is consistent with existing literature, yet we suggest that it is imperative to consider the broader purpose of CST and the needs of stakeholders. Consequently, it may be beneficial to use mixed methods of CST in medical programs.

Examiner seniority and experience are associated with bias when scoring communication, but not examination, skills in objective structured clinical examinations in Australia.

PURPOSE: The biases that may influence objective structured clinical examination (OSCE) scoring are well understood, and recent research has attempted to establish the magnitude of their impact. However, the influence of examiner experience, clinical seniority, and occupation on communication and physical examination scores in OSCEs has not yet been clearly established. METHODS: We compared the mean scores awarded for generic and clinical communication and physical examination skills in 2 undergraduate medicine OSCEs in relation to examiner characteristics (gender, examining experience, occupation, seniority, and speciality). The statistical significance of the differences was calculated using the 2-tailed independent t-test and analysis of variance. RESULTS: Five hundred and seventeen students were examined by 237 examiners at the University of New South Wales in 2014 and 2016. Examiner gender, occupation (academic, clinician, or clinical tutor), and job type (specialist or generalist) did not significantly impact scores. Junior doctors gave consistently higher scores than senior doctors in all domains, and this difference was statistically significant for generic and clinical communication scores. Examiner experience was significantly inversely correlated with generic communication scores. CONCLUSION: We suggest that the assessment of examination skills may be less susceptible to bias because this process is fairly prescriptive, affording greater scoring objectivity. We recommend training to define the marking criteria, teaching curriculum, and expected level of performance in communication skills to reduce bias in OSCE assessment.

Enhancing the defensibility of examiners' marks in high stake OSCEs.

BACKGROUND: Most assessments in health professions education consist of knowledge-based examinations as well as practical and clinical examinations. Among the most challenging aspects of clinical assessments is decision making related to borderline grades assigned by examiners. Borderline grades are commonly used by examiners when they do not have sufficient information to make clear pass/fail decisions. The interpretation of these borderline grades is rarely discussed in the literature. This study reports the application of the Objective Borderline Method (version 2, henceforth: OBM2) to a high stakes Objective Structured Clinical Examination undertaken at the end of the final year of a Medicine program in Australia. METHODS: The OBM2 uses all examination data to reclassify borderline grades as either pass or fail. Factor analysis was used to estimate the suitability of data for application of OBM2. Student's t-tests, utilising bootstrapping, were used to compare the OBM2 with 'traditional' results. Interclass correlations were used to estimate the association between the grade reclassification and all other grades in this examination. RESULTS: The correlations between scores for each station and pass/fail outcomes increased significantly after the mark reclassification, yet the reclassification did not significantly impact on students' total scores. Examiners, students and program leaders expressed high levels of satisfaction and the Faculty's Curriculum Development Committee has decided that the OBM2 will be used for all future clinical examinations. Implications of the OBM2 are discussed. CONCLUSIONS: The OBM2 provides a feasible, defensible and acceptable solution for classification of borderline grades as either pass or fail.

The sights and insights of examiners in objective structured clinical examinations.

PURPOSE: The objective structured clinical examination (OSCE) is considered to be one of the most robust methods of clinical assessment. One of its strengths lies in its ability to minimise the effects of examiner bias due to the standardisation of items and tasks for each candidate. However, OSCE examiners' assessment scores are influenced by several factors that may jeopardise the assumed objectivity of OSCEs. To better understand this phenomenon, the current review aims to determine and describe important sources of examiner bias and the factors affecting examiners' assessments. METHODS: We performed a narrative review of the medical literature using Medline. All articles meeting the selection criteria were reviewed, with salient points extracted and synthesised into a clear and comprehensive summary of the knowledge in this area. RESULTS: OSCE examiners' assessment scores are influenced by factors belonging to 4 different domains: examination context, examinee characteristics, examinee-examiner interactions, and examiner characteristics. These domains are composed of several factors including halo, hawk/dove and OSCE contrast effects; the examiner's gender and ethnicity; training; lifetime experience in assessing; leadership and familiarity with students; station type; and site effects. CONCLUSION: Several factors may influence the presumed objectivity of examiners' assessments, and these factors need to be addressed to ensure the objectivity of OSCEs. We offer insights into directions for future research to better understand and address the phenomenon of examiner bias.

Australian medical students have fewer opportunities to do physical examination of peers of the opposite gender.

PURPOSE: Peer physical examination (PPE), by which junior medical students learn physical examination skills before practicing on patients, is a widely implemented and accepted part of medical curricula. However, the ethical implications of PPE have been debated, since issues including student gender impact on its acceptability. Research has previously demonstrated the phenomenon of 'attitude-behavior inconsistency' showing that students' predictions about their participation in PPE differ from what they actually do in practice. This study asks whether gender and student self-ratings of outlook affect engagement in PPE. METHODS: This study gathered data from students who had completed PPE with the objective of determining what factors have the greatest impact on the actual practice of PPE by students. Data were used to derive the number of opportunities students had to examine a peer, for various body parts. Respondent gender and self-ratings of outlook were recorded. RESULTS: Responses from 130 students were analysed: 74 female (57%) and 56 male (43%). Students have fewer opportunities to examine peers of the opposite gender; this is statistically significant for all body parts when male students examine female peers. CONCLUSION: Gender is the factor of overriding importance on whether these peer interactions actually occur, such that students have fewer opportunities to examine peers of the opposite gender, particularly male students examining female peers. Student outlook has little impact. We speculate that the more acceptable PPE is to participants, paradoxically, the more complicated these interactions become, possibly with implications for future practice.

A Web-Based Telehealth Training Platform Incorporating Automated Nonverbal Behavior Feedback for Teaching Communication Skills to Medical Students: A Randomized Crossover Study.

BACKGROUND: In the interests of patient health outcomes, it is important for medical students to develop clinical communication skills. We previously proposed a telehealth communication skills training platform (EQClinic) with automated nonverbal behavior feedback for medical students, and it was able to improve medical students' awareness of their nonverbal communication. OBJECTIVE: This study aimed to evaluate the effectiveness of EQClinic to improve clinical communication skills of medical students. METHODS: We conducted a 2-group randomized crossover trial between February and June 2016. Participants were second-year medical students enrolled in a clinical communication skills course at an Australian university. Students were randomly allocated to complete online EQClinic training during weeks 1-5 (group A) or to complete EQClinic training during weeks 8-11 (group B). EQClinic delivered an automated visual presentation of students' nonverbal behavior coupled with human feedback from a standardized patient (SP). All students were offered two opportunities to complete face-to-face consultations with SPs. The two face-to-face consultations were conducted in weeks 6-7 and 12-13 for both groups, and were rated by tutors who were blinded to group allocation. Student-Patient Observed Communication Assessment (SOCA) was collected by blinded assessors (n=28) at 2 time points and also by an SP (n=83). Tutor-rated clinical communications skill in face-to-face consultations was the primary outcome and was assessed with the SOCA. We used t tests to examine the students' performance during face-to-face consultations pre- and postexposure to EQClinic. RESULTS: We randomly allocated 268 medical students to the 2 groups (group A: n=133; group B: n=135). SOCA communication skills measures (score range 4-16) from the first face-to-face consultation were significantly higher for students in group A who had completed EQClinic training and reviewed the nonverbal behavior feedback, compared with group B, who had completed only the course curriculum components (P=.04). Furthermore, at the second face-to-face assessment, the group that completed a teleconsultation between the two face-to-face consultations (group B) showed improved communication skills (P=.005), and the one that had teleconsultations before the first face-to-face consultation (group A) did not show improvement. CONCLUSIONS: The EQClinic is a useful tool for medical students' clinical communication skills training that can be applied to university settings to improve students clinical communication skills development.

EQClinic: a platform for learning communication skills in clinical consultations.

BACKGROUND: Doctors' verbal and non-verbal communication skills have an impact on patients' health outcomes, so it is important for medical students to develop these skills. Traditional, non-verbal communication skills training can involve a tutor manually annotating a student's non-verbal behaviour during patient-doctor consultations, but this is very time-consuming. Tele-conference systems have been used in verbal communication skills training. METHODS: We describe EQClinic, a system that enables verbal and non-verbal communication skills training during tele-consultations with simulated patients (SPs), with evaluation exercises promoting reflection. Students and SPs can have tele-consultations through the tele-consultation component. In this component, SPs can provide feedback to students through a thumbs-up/ thumbs-down tool and a comments box. EQClinic automatically analyses communication features in the recorded consultations, such as facial expressions, and provides graphical representations. Our 2015 pilot study investigated whether EQClinic helped students be aware of their non-verbal behaviour and improve their communication skills, and evaluated the usability of the platform. Students received automated feedback, and SP and tutor evaluations, and then completed self-assessment and reflection questionnaires. RESULTS: Eight medical students and three SPs conducted 13 tele-consultations using EQClinic. More students paid attention to their non-verbal communication and students who were engaged in two consultations felt more confident in their second consultation. Students rated the system positively, felt comfortable using it (5.9/7), and reported that the structure (5.4/7) and information (5.8/7) were clear. This pilot provides evidence that EQClinic helps, and positively influences, medical students practise their communication skills with SPs using a tele-conference platform. DISCUSSION: It is not easy to improve non-verbal communication skills in a short time period. Further evaluation of EQClinic with larger numbers will ascertain learning gains and application in health professional training. Developing a standard model for the assessment of non-verbal behaviour in tele-consultations and providing students with more valuable evaluation and suggestions are part of our future work.