Interrater reliability in neurology objective structured clinical examination across specialties.

PURPOSE: To assess interrater reliability and examiners' characteristics, especially specialty, associated with scoring of neurology objective structured clinical examination (OSCE). MATERIAL AND METHODS: During a neurology mock OSCE, five randomly chosen students volunteers were filmed while performing 1 of the 5 stations. Video recordings were scored by physicians from the Lyon and Clermont-Ferrand university teaching hospitals to assess students performance using both a checklist scoring and a global rating scale. Interrater reliability between examiners were assessed using intraclass coefficient correlation. Multivariable linear regression models including video recording as random effect dependent variable were performed to detect factors associated with scoring. RESULTS: Thirty examiners including 15 (50%) neurologists participated. The intraclass correlation coefficient of checklist scores and global ratings between examiners were 0.71 (CI95% [0.45-0.95]) and 0.54 (CI95% [0.28-0.91]), respectively. In multivariable analyses, no factor was associated with checklist scores, while male gender of examiner was associated with lower global rating (ß coefficient = -0.37; CI 95% [-0.62-0.11]). CONCLUSIONS: Our study demonstrated through a video-based scoring method that agreement among examiners was good using checklist scoring while moderate using global rating scale in neurology OSCE. Examiner's specialty did not affect scoring whereas gender was associated with global rating scale.

A proposed feedback model for the objective structured clinical examination (OSCE) in health and medical education.

WHAT WAS THE EDUCATIONAL CHALLENGE?: The objective structured clinical examination (OSCE) is used extensively in health and medical education to assess clinical competence. While OSCEs are not primarily designed for individual feedback, they provide an opportunity to generate meaningful and constructive feedback that students can use to reflect on areas of weakness (and strength). The most timely and effective way to incorporate individual feedback following a summative OSCE continues to be explored. WHAT WAS THE SOLUTION?: This paper describes a novel OSCE feedback model which uses readily available summative assessment data to calculate 10% index scores. This provides information about relative station difficulty and relative individual student performance. HOW WAS THE SOLUTION IMPLEMENTED?: An individualised feedback report was provided to every student after the OSCEs. This enables identification of stronger and weaker performance areas and composite skills, which can be utilised by all students to direct future learning. WHAT LESSONS WERE LEARNED THAT ARE RELEVANT TO A WIDER GLOBAL AUDIENCE?: Providing timely, individual actionable feedback to every student following a summative OSCE is possible without impacting the examination process or examiner burden. WHAT ARE THE NEXT STEPS?: A study is planned to determine the utilisation of this feedback by students and it's impact as a qualitative self-assessment exercise.

Can routine EPA-based assessments predict OSCE performances of undergraduate medical students?

BACKGROUND: Objective structured clinical examination (OSCE) is used worldwide. This study aims to explore potential alternatives to the OSCE by using entrustable professional activities (EPA)-based assessments in the workplace. METHODS: This study enrolled 265 six-year undergraduate medical students (UGY) from 2021 to 2023. During their rotations, students were assessed using 13 EPAs, with the grading methods modified to facilitate application. Before graduation, they participated in two mock OSCEs and a National OSCE. We used generalized estimating equations to analyze the associations between the EPA assessments and the OSCE scores, adjusting for age and sex, and developed a prediction model. EPA8 and EPA9, which represent advanced abilities that were not significant in the regression models, were removed from the prediction model. RESULTS: Most EPAs were significantly correlated with OSCE scores across the three cohorts. The prediction model for forecasting passing in the three OSCEs demonstrated fair predictive capacity (area under curve = 0.82, 0.66, and 0.71 for students graduated in 2021, 2022, and 2023, respectively all p < 0.05). CONCLUSIONS: The workplace-based assessments (EPA) showed a high correlation with competency-based assessments in simulated settings (OSCE). EPAs may serve as alternative tools to formal OSCE for medical students.

Developing an innovative national ACP-OSCE program in Taiwan: a mixed method study.

OBJECTIVES: To evaluate the process and the comprehensiveness of advance care planning (ACP), we designed a national ACP-OSCE (Objective Structured Clinical Examination) program. METHODS: The program was designed as a 40-minute OSCE test. Participants were categorized as different ACP team members to illustrate realistic scenarios. Preceptors were asked to observe ACP professionals' actions, responses, and communication skills during ACP with standardized patients (SP) through a one-way mirror. Participants' communication skills, medical expertise, legal knowledge, empathetic response and problem-solving skills of ACP were also self-evaluated before and after OSCE. Thematic analysis was used for qualitative analysis. RESULTS: In Nov 2019, a total of 18 ACP teams with 38 ACP professionals completed the ACP-OSCE program, including 15 physicians, 15 nurses, 5 social workers, and 3 psychologists. After the ACP-OSCE program, the average score of communication skills, medical expertise, legal knowledge, empathetic response, ACP problem-solving all increased. Nurses felt improved in medical expertise, legal knowledge, and problem-solving skills, psychologists and social workers felt improved in legal knowledge, while physicians felt no improved in all domain, statistically. Thematic analysis showed professional skills, doctoral-patient communication, benefit and difficulties of ACP were the topics which participants care about. Meanwhile, most participants agreed that ACP-OSCE program is an appropriate educational tool. CONCLUSION: This is the first national ACP-OSCE program in Asia. We believe that this ACP-OSCE program could be applied in other countries to improve the ACP process and quality.

Objective structured clinical examination to teach competency in planetary health care and management - a prospective observational study.

BACKGROUND: Health professionals are increasingly called upon and willing to engage in planetary health care and management. However, so far, this topic is rarely covered in medical curricula. As the need for professional communication is particularly high in this subject area, this study aimed to evaluate whether the objective structured clinical examination (OSCE) could be used as an accompanying teaching tool. METHODS: During the winter semester 2022/2023, 20 third- and fifth-year medical students voluntarily participated in a self-directed online course, three workshops, and a formal eight-station OSCE on planetary health care and management. Each examinee was also charged alternatingly as a shadower with the role of providing feedback. Experienced examiners rated students' performance using a scoring system supported by tablet computers. Examiners and shadowers provided timely feedback on candidates` performance in the OSCE. Immediately after the OSCE, students were asked about their experience using a nine-point Likert-scale survey and a videotaped group interview. Quantitative analysis included the presentation of the proportional distribution of student responses to the survey and of box plots showing percentages of maximum scores for the OSCE performance. The student group interview was analyzed qualitatively. RESULTS: Depending on the sub-theme, 60% -100% of students rated the subject of planetary health as likely to be useful in their professional lives. Similar proportions (57%-100%) were in favour of integrating planetary health into required courses. Students perceived learning success from OSCE experience and feedback as higher compared to that from online courses and workshops. Even shadowers learned from observation and feedback discussions. Examiners assessed students' OSCE performance at a median of 80% (interquartile range: 83%-77%) of the maximum score. CONCLUSIONS: OSCE can be used as an accompanying teaching tool for advanced students on the topic of planetary health care and management. It supports learning outcomes, particularly in terms of communication skills to sensitise and empower dialogue partners, and to initiate adaptation steps at the level of individual patients and local communities.

Situational simulation teaching effectively improves dental students' non-operational clinical competency and objective structured clinical examination performance.

BACKGROUND: Appropriate communication with dental patients enhances treatment outcomes and patient satisfaction. Implementing simulated patient interviews courses can improve patient-centered care and reduce conflict during clerkship training. Therefore, this study explored the relationship among student participation in a situational simulation course (SSC), academic performance, clerkship performance, and objective structured clinical examination (OSCE) performance. METHODS: This study was conducted with a sample of fifth-year dental students undergoing clerkship training. After implementing a situational simulation course to investigate the relationship among participation in SSC, academic performance, clerkship performance, and OSCE performance, a path analysis model was developed and tested. RESULTS: Eighty-seven fifth-year dental students were eligible for the SSC, and most (n = 70, 80.46%) volunteered to participate. The path analysis model revealed that academic performance had a direct effect on OSCE performance (ß = 0.281, P = 0.003) and clerkship performance (ß = 0.441, P < 0.001). In addition, SSC teaching had a direct effect on OSCE performance (ß = 0.356, P < 0.001). CONCLUSIONS: SSCs can enhance dental students' non-operational clinical competency and OSCE performance effectively. Simulated patient encounters with feedback, incorporated into the dental curricula, have led to improved communication. Based on our findings, we suggest implementing SSC teaching before the OSCE to improve communication and cognitive skills.

Breaking bad news: an active learning method for medical students.

BACKGROUND: Breaking bad news is one of the most difficult aspects of communication in medicine. The objective of this study was to assess the relevance of a novel active learning course on breaking bad news for fifth-year students. METHODS: Students were divided into two groups: Group 1, the intervention group, participated in a multidisciplinary formative discussion workshop on breaking bad news with videos, discussions with a pluri-professional team, and concluding with the development of a guide on good practice in breaking bad news through collective intelligence; Group 2, the control group, received no additional training besides conventional university course. The relevance of discussion-group-based active training was assessed in a summative objective structured clinical examination (OSCE) station particularly through the students' communication skills. RESULTS: Thirty-one students were included: 17 in Group 1 and 14 in Group 2. The mean (range) score in the OSCE was significantly higher in Group 1 than in Group 2 (10.49 out of 15 (7; 13) vs. 7.80 (4.75; 12.5), respectively; p = 0.0007). The proportion of students assessed by the evaluator to have received additional training in breaking bad news was 88.2% (15 of the 17) in Group 1 and 21.4% (3 of the 14) in Group 2 (p = 0.001). The intergroup differences in the Rosenberg Self-Esteem Scale and Jefferson Scale of Empathy scores were not significant, and both scores were not correlated with the students' self-assessed score for success in the OSCE. CONCLUSION: Compared to the conventional course, this new active learning method for breaking bad news was associated with a significantly higher score in a summative OSCE. A longer-term validation study is needed to confirm these exploratory data.

Analysis of satisfaction levels and perceptions of clinical competency: a mixed method study on objective structured clinical examinations in undergraduate dental students.

OBJECTIVE: To analyze the satisfaction levels, perceptions of developing clinical competencies through objective structured clinical examination and to explore the experiences, challenges, and suggestions of undergraduate dental students. METHODS: The study adopted a mixed-method convergent design. Quantitative data were collected from 303 participants through surveys, evaluating satisfaction levels with objective structured clinical examination (OSCE). Additionally, qualitative insights were gathered through student focus group interviews, fundamental themes were developed from diverse expressions on various aspects of OSCE assessments. The Chi-Square tests, was performed to assess associations between variables. Data integration involved comparing and contrasting quantitative and qualitative findings to derive comprehensive conclusions. RESULTS: The satisfaction rates include 69.4% for the organization of OSCE stations and 57.4% for overall effectiveness. However, a crucial challenge was identified, with only 36.7% of students receiving adequate post-OSCE feedback. Furthermore, a majority of students (50%) expressed concerns about the clinical relevance of OSCEs. The study showed a significant associations (p < 0.05) between satisfaction levels and years of study as well as previous OSCE experience. Student focus group interviews revealed diverse perspectives on OSCE assessments. While students appreciate the helpfulness of OSCEs, concerns were raised regarding time constraints, stress, examiner training, and the perceived lack of clinical relevance. CONCLUSION: The students anticipated concerns about the clinical relevance of OSCEs, highlighting the need for a more aligned assessment approach. Diverse perspectives on OSCE assessments reveal perceived helpfulness alongside challenges such as lack of feedback, examiner training, time constraints, and mental stress.

Administration of the American Board of Anesthesiology's virtual APPLIED Examination: successes, challenges, and lessons learned.

In response to the COVID-19 pandemic, the American Board of Anesthesiology transitioned from in-person to virtual administration of its APPLIED Examination, assessing more than 3000 candidates for certification purposes remotely in 2021. Four hundred examiners were involved in delivering and scoring Standardized Oral Examinations (SOEs) and Objective Structured Clinical Examinations (OSCEs). More than 80% of candidates started their exams on time and stayed connected throughout the exam without any problems. Only 74 (2.5%) SOE and 45 (1.5%) OSCE candidates required rescheduling due to technical difficulties. Of those who experienced "significant issues", concerns with OSCE technical stations (interpretation of monitors and interpretation of echocardiograms) were reported most frequently (6% of candidates). In contrast, 23% of examiners "sometimes" lost connectivity during their multiple exam sessions, on a continuum from minor inconvenience to inability to continue. 84% of SOE candidates and 89% of OSCE candidates described "smooth" interactions with examiners and standardized patients/standardized clinicians, respectively. However, only 71% of SOE candidates and 75% of OSCE candidates considered themselves to be able to demonstrate their knowledge and skills without obstacles. When compared with their in-person experiences, approximately 40% of SOE examiners considered virtual evaluation to be more difficult than in-person evaluation and believed the remote format negatively affected their development as an examiner. The virtual format was considered to be less secure by 56% and 40% of SOE and OSCE examiners, respectively. The retirement of exam materials used virtually due to concern for compromise had implications for subsequent exam development. The return to in-person exams in 2022 was prompted by multiple factors, especially concerns regarding standardization and security. The technology is not yet perfect, especially for testing in-person communication skills and displaying dynamic exam materials. Nevertheless, the American Board of Anesthesiology's experience demonstrated the feasibility of conducting large-scale, high-stakes oral and performance exams in a virtual format and highlighted the adaptability and dedication of candidates, examiners, and administering board staff.

The use of the objective structured clinical examination to evaluate paediatric cardiopulmonary resuscitation skills in medical students and measures to improve training.

BACKGROUND: The main objectives of the study were to analyse the use of the Objective Structured Clinical Examination (OSCE) to evaluate the skills of medical students in paediatric basic life support (PBLS), to compare two resuscitation training models and to evaluate the measures to improve the teaching program. METHODS: Comparative, prospective, observation study with intervention in two hospitals, one undergoing a PILS course (Paediatric Immediate Life Support) and another PBLS. The study was performed over three phases. 1º. PBLS OSCE in 2022 three months after the resuscitation training 2. Measures to improve the training program in 2023 3. PBLS OSCE in 2023. Overall results were analysed and comparison between both sites and those for 2022 and 2023 were made. RESULTS: A total of 210 and 182 students took part in the OSCE in 2022 and 2023, respectively. The overall mean score out of 100 was 83.2 (19), 77.8 (19.8) in 2022 and 89.5 (15.9) and 2023, P < 0.001. Overall cardiopulmonary resuscitation (CPR) effectiveness was adequate in 79.4% and 84.6% of students in 2022 and 2023, respectively. The results of hospital students undergoing a PILS course were better (86.4 (16.6) than those undergoing a PBLS. 80.2 (20.6) p < 0.001. The results from both hospitals improved significantly in 2023. CONCLUSIONS: The OSCE is a valid instrument to evaluate PBLS skills in medical students and to compare different training methods and program improvements. Medical students who receive a PILS attain better PBLS skills than those who undergo a PBLS course.

Raters and examinees training for objective structured clinical examination: comparing the effectiveness of three instructional methodologies.

BACKGROUND: Utilizing the objective structured clinical examination (OSCE) ensures objectivity when it comes to the assessment of nursing students' skills and competency. However, one challenge in OSCE integration is rater and examinee training and orientation. AIM: This study employed a quasi-experimental design to evaluate the effectiveness of different instructional methodologies in training and preparing raters and examinees for the OSCE. METHODS: Participants were divided into three group of training methodologies: online, simulation, and traditional lecture (six raters and 18 examinees were assigned to each group). A total of 18 raters and 54 examinees partook. RESULTS: The study found that raters trained through simulation exhibited a slight agreement with their rates, compared to those who were trained online and in traditional lectures. Moreover, examinees who were trained through the simulation methodology performed better compared to those trained via the other methodologies. CONCLUSIONS: The study findings indicate that using simulation by training raters and examinees in the OSCE is the most effective approach.

A level playing field? Evaluation of the virtual Objective Structured Clinical Examination in Psychiatry and Addiction Medicine: A mixed methods study.

OBJECTIVE: The Objective Structured Clinical Examination (OSCE) has been used for clinical assessment of a broad range of medical student competencies in Psychiatry and Addiction Medicine. However, there has been little research into online assessments. We investigated the virtual OSCE (v-OSCE) from the user perspective to better understand its acceptability, usefulness, benefits, challenges and potential improvements. METHODS: At the conclusion of the v-OSCE, all participants (medical students, examiners and simulated patients) were invited to participate in a brief online survey, based on the Technology and Acceptance Model. Freeform qualitative feedback was also obtained to explore participants' experiences and attitudes. RESULTS: Participants reported the v-OSCE was acceptable, efficient, convenient and easy to use. It was perceived as useful for demonstrating students' interviewing skills and interacting with the simulated patient. Benefits included greater convenience, reduced stress and travel time. Challenges were similar to those experienced in 'real world' telepsychiatry, primarily related to assessment of non-verbal cues and emotional prosody. CONCLUSIONS: Our findings inform recommendations for improving online examinations. These include increased preparation, practice and professionalism, to better simulate the in-person experience. Study credibility was strengthened by the triangulation of qualitative, quantitative and psychometric data.

Using Cognitive Task Analysis to Develop a Protocol for Teaching Ultrasound Pregnancy Diagnosis in the Bitch to Undergraduate Veterinary Students.

Pregnancy diagnosis in the bitch is routinely performed using ultrasound and is therefore an important skill for veterinarians to have been exposed to during undergraduate training. Proficiency of this skill is difficult to achieve, due to limited exposure to suitable live patients, and animal welfare considerations limiting repeated performance on the same bitch. Models have been beneficial in allowing undergraduates to perform a range of ultrasound techniques without the use of live animals. Using clinical veterinarians and a model created at the University of Surrey, cognitive task analysis (CTA) was used to construct a list of instructional steps required to perform ultrasound pregnancy diagnosis. Experts were asked to evaluate the existing model then video recorded while demonstrating the skill on the model as if teaching a novice student. Anonymized and muted video footage along with transcribed audio files were used to create a draft teaching protocol. A group consensus for the final teaching protocol was developed following a semi-structured interview. The final teaching protocol had 23 steps to guide a novice to perform this skill, broken down into three stages: setup and preparation, pregnancy identification, and estimation of gestational age. Not all steps were both performed and verbalized by all of the experts, hence the need for a panel discussion to confirm a final teaching protocol. This study demonstrated that CTA is useful in compiling a comprehensive list of steps, for a teaching protocol, including those which may have been missed if demonstrated through a lone subject matter expert.

ChatGPT outscored human candidates in a virtual objective structured clinical examination in obstetrics and gynecology.

BACKGROUND: Natural language processing is a form of artificial intelligence that allows human users to interface with a machine without using complex codes. The ability of natural language processing systems, such as ChatGPT, to successfully engage with healthcare systems requiring fluid reasoning, specialist data interpretation, and empathetic communication in an unfamiliar and evolving environment is poorly studied. This study investigated whether the ChatGPT interface could engage with and complete a mock objective structured clinical examination simulating assessment for membership of the Royal College of Obstetricians and Gynaecologists. OBJECTIVE: This study aimed to determine whether ChatGPT, without additional training, would achieve a score at least equivalent to that achieved by human candidates who sat for virtual objective structured clinical examinations in Singapore. STUDY DESIGN: This study was conducted in 2 phases. In the first phase, a total of 7 structured discussion questions were selected from 2 historical cohorts (cohorts A and B) of objective structured clinical examination questions. ChatGPT was examined using these questions and responses recorded in a script. Of note, 2 human candidates (acting as anonymizers) were examined on the same questions using videoconferencing, and their responses were transcribed verbatim into written scripts. The 3 sets of response scripts were mixed, and each set was allocated to 1 of 3 human actors. In the second phase, actors were used to presenting these scripts to examiners in response to the same examination questions. These responses were blind scored by 14 qualified examiners. ChatGPT scores were unblinded and compared with historical human candidate performance scores. RESULTS: The average score given to ChatGPT by 14 examiners was 77.2%. The average historical human score (n=26 candidates) was 73.7 %. ChatGPT demonstrated sizable performance improvements over the average human candidate in several subject domains. The median time taken for ChatGPT to complete each station was 2.54 minutes, well before the 10 minutes allowed. CONCLUSION: ChatGPT generated factually accurate and contextually relevant structured discussion answers to complex and evolving clinical questions based on unfamiliar settings within a very short period. ChatGPT outperformed human candidates in several knowledge areas. Not all examiners were able to discern between human and ChatGPT responses. Our data highlight the emergent ability of natural language processing models to demonstrate fluid reasoning in unfamiliar environments and successfully compete with human candidates that have undergone extensive specialist training.

Feasibility and reliability of the pandemic-adapted online-onsite hybrid graduation OSCE in Japan.

Objective structured clinical examination (OSCE) is widely used to assess medical students' clinical skills. Virtual OSCEs were used in place of in-person OSCEs during the COVID-19 pandemic; however, their reliability is yet to be robustly analyzed. By applying generalizability (G) theory, this study aimed to evaluate the reliability of a hybrid OSCE, which admixed in-person and online methods, and gain insights into improving OSCEs' reliability. During the 2020-2021 hybrid OSCEs, one examinee, one rater, and a vinyl mannequin for physical examination participated onsite, and a standardized simulated patient (SP) for medical interviewing and another rater joined online in one virtual breakout room on an audiovisual conferencing system. G-coefficients and 95% confidence intervals of the borderline score, namely border zone (BZ), under the standard 6-station, 2-rater, and 6-item setting were calculated. G-coefficients of in-person (2017-2019) and hybrid OSCEs (2020-2021) under the standard setting were estimated to be 0.624, 0.770, 0.782, 0.759, and 0.823, respectively. The BZ scores were estimated to be 2.43-3.57, 2.55-3.45, 2.59-3.41, 2.59-3.41, and 2.51-3.49, respectively, in the score range from 1 to 6. Although hybrid OSCEs showed reliability comparable to in-person OSCEs, they need further improvement as a very high-stakes examination. In addition to increasing clinical vignettes, having more proficient online/on-demand raters and/or online SPs for medical interviews could improve the reliability of OSCEs. Reliability can also be ensured through supplementary examination and by increasing the number of online raters for a small number of students within the BZs.

Entrustment versus performance scale in high-stakes OSCEs: Rater insights and psychometric properties.

BACKGROUND: Although entrustment scales are increasingly applied in workplace-based assessments, their role in OSCEs remains unclear. We investigated raters' perceptions using an entrustment scale and psychometric analyses. METHOD: A mixed-methods design was used. OSCE raters' (n = 162) perceptions were explored via questionnaire and four focus groups (n = 14). Psychometric OSCE properties were analyzed statistically. RESULTS: Raters (n = 53, response rate = 41%) considered the entrustment scale comprehensible (89%) and applicable (60%). A total of 43% preferred the entrustment scale, 21% preferred the global performance scale, and 36% were undecided. Raters' written comments indicated that while they appreciated the authenticity of entrustment levels, they considered them subjective. The focus groups highlighted three main themes: (1) recollections of the clinical workplace as a cognitive reference triggered by entrustment scales; (2) factors influencing entrustment decisions; and (3) cognitive load is reduced at the perceived cost of objectivity. Psychometric analyses (n = 480 students) revealed improvements in some OSCE metrics when entrustment and global performance scales were combined. CONCLUSION: Entrustment scales are beneficial for high-stakes OSCEs and have greater clinical relevance from the raters' perspective. Our findings support the use of entrustment and global performance scales in combination.

Twelve tips for improving the quality of assessor judgements in senior medical student clinical assessments.

Assessment of senior medical students is usually calibrated at the level of achieving expected learning outcomes for graduation. Recent research reveals that clinical assessors often balance two slightly different perspectives on this benchmark. The first is the formal learning outcomes at graduation, ideally as part of a systematic, program-wide assessment approach that measures learning achievement, while the second is consideration of the candidate's contribution to safe care and readiness for practice as a junior doctor. The second is more intuitive to the workplace, based on experience working with junior doctors. This perspective may enhance authenticity in assessment decisions made in OSCEs and work-based assessments to better align judgements and feedback with professional expectations that will guide senior medical students and junior doctors' future career development. Modern assessment practices should include consideration of qualitative as well as quantitative information, overtly including perspectives of patients, employers, and regulators. This article presents 12 tips for how medical education faculty might support clinical assessors by capturing workplace expectations of first year medical graduates and develop graduate assessments based on a shared heuristic of 'work-readiness'. Peer-to-peer assessor interaction should be facilitated to achieve correct calibration that 'merges' the differing perspectives to produce a shared construct of an acceptable candidate.

Assessing clinical reasoning in the OSCE: pilot-testing a novel oral debrief exercise.

INTRODUCTION: Clinical reasoning (CR) is a complex skill enabling transition from clinical novice to expert decision maker. The Objective Structured Clinical Examination (OSCE) is widely used to evaluate clinical competency, though there is limited literature exploring how this assessment is best used to assess CR skills. This proof-of-concept study explored the creation and pilot testing of a post-station CR assessment, named Oral Debrief (OD), in the context of undergraduate medical education. METHODS: A modified-Delphi technique was used to create a standardised domain-based OD marking rubric encapsulating the key skills of CR that drew upon existing literature and our existing placement-based CR tool. 16 OSCE examiners were recruited to score three simulated OD recordings that were scripted to portray differing levels of competency. Adopting a think-aloud approach, examiners vocalised their thought processes while utilising the rubric to assess each video. Thereafter, semi-structured interviews explored examiners' views on the OD approach. Recordings were transcribed, anonymised and analysed deductively and inductively for recurring themes. Additionally, inter-rater agreement of examiners' scoring was determined using the Fleiss Kappa statistic both within group and in comparison to a reference examiner group. RESULTS: The rubric achieved fair to good levels of inter-rater reliability metrics across its constituent domains and overall global judgement scales. Think-aloud scoring revealed that participating examiners considered several factors when scoring students' CR abilities. This included the adoption of a confident structured approach, discriminating between relevant and less-relevant information, and the ability to prioritise and justify decision making. Furthermore, students' CR skills were judged in light of potential risks to patient safety and examiners' own illness scripts. Feedback from examiners indicated that whilst additional training in rubric usage would be beneficial, OD offered a positive approach for examining CR ability. CONCLUSION: This pilot study has demonstrated promising results for the use of a novel post-station OD task to evaluate medical students' CR ability in the OSCE setting. Further work is now planned to evaluate how the OD approach can most effectively be implemented into routine assessment practice.

Scoring consistency of standard patients and examiners in the developed dental objective structured clinical examination system.

OBJECTIVE: To investigate the role of standard patients (SPs) and examiners as assessors for scoring in the dental objective structured clinical examination (OSCE) system and to evaluate the scoring differences between them. METHODS: We developed the doctor-patient communication and clinical examination station in the OSCE system. The examination time of this station was 10 min, and the examination institution wrote the script and recruited SPs. A total of 146 examinees who received standardized resident training at the Nanjing Stomatological Hospital, Medical School of Nanjing University between 2018 and 2021 were assessed. They were scored by SPs and examiners according to the same scoring rubrics. Subsequently, the SPSS software was used to analyze the examination results of different assessors and evaluate the consistency. RESULTS: The average score of all examinees provided by SPs and examiners was 90.45 ± 3.52 and 91.53 ± 4.13, respectively. The consistency analysis showed that the intraclass correlation coefficient was 0.718, which was indicative of medium consistency. CONCLUSION: Our findings showed that SPs could be used directly as assessors, as they could provide a simulated and realistic clinical setting and create favorable conditions for comprehensive competence training and improvement for medical students.

Progress testing of an objective structured clinical examination during undergraduate clinical clerkship: a mixed-methods pilot study.

BACKGROUND: Progress testing is an assessment method in which an examination reflecting competencies at graduation is regularly administered to students over multiple years, thereby facilitating self-directed learning. However, the significance of the objective structured clinical examination as a progress test in undergraduate education, needs to be determined. This study provides evidence of the role of the objective structured clinical examination for progress testing and optimal scoring methods for assessing students in different academic years. METHODS: We conducted a sequential explanatory mixed-methods pilot study. Participants were assessed using the Item Rating Scale, the year-adjusted Global Rating Scale, and the Training Level Rating Scale. The characteristics of each scale were compared quantitatively. In addition, the influence of the objective structured clinical examination as a progress test on learning attitudes was examined. Qualitative data from a post-examination questionnaire were analyzed, using content analysis to explore influences on self-directed learning. RESULTS: Sixth and fifth year clinical students (n = 235) took the objective structured clinical examination progress test. The total Item Rating Scales were recorded (%) as 59.03 ± 5.27 and 52.64 ± 5.08 (p < 0.01); Training Level Rating Scale was 3.94 ± 0.39 vs 3.22 ± 0.42 (p < 0.01); and the year-adjusted Global Rating Scale was 4.25 ± 0.44 vs 4.32 ± 0.52 (no significant difference), for the sixth and fifth year students, respectively. The correlations across stations and the reliability of each station were satisfactory. Four categories were identified in the qualitative analysis: "motivation to learn during the clinical clerkship was promoted," "dissatisfied with being asked about things they had not experienced," "confusion about being unable to use conventional test-taking strategies," and "insufficient understanding of competencies at graduation." The scores indicated significant differences in performance according to training year. CONCLUSIONS: This study provides evidence that the objective structured clinical examination can be used as a progress testing tool for undergraduate clinical clerkships. Further enhancement of training opportunities and dissemination of performance competency goals in clerkship curricula are required if we intend to promote self-directed learning through progress testing.