Exploring approaches to dietetic assessment of a common task across different universities through assessment moderation.

BACKGROUND: Assessment presents one of the greatest challenges to evaluating health professional trainee performance, as a result of the subjectivity of judgements and variability in assessor standards. The present study aimed to test a moderation procedure for assessment across four independent universities and explore approaches to assessment and the factors that influence assessment decisions. METHODS: Assessment tasks designed independently by each of the four universities to assess student readiness for placement were chosen for the present study. Each university provided four student performance recordings for moderation. Eight different academic assessors viewed the student performances and assessed them using the corresponding university assessment instrument. Assessment results were collated and presented back to the assessors, together with the original university assessment results. Results were discussed with assessors to explore variations. The discussion was recorded, transcribed, thematically analysed and presented back to all assessors to achieve consensus on the emerging major learnings. RESULTS: Although there were differences in absolute scores, there was consistency (12 out of 16 performances) in overall judgement decisions regarding placement readiness. Proficient communication skills were considered a key factor when determining placement readiness. The discussion revealed: (i) assessment instruments; (ii) assessor factors; and (iii) the subjectivity of judgement as the major factors influencing assessment. CONCLUSIONS: Assessment moderation is a useful method for improving the quality of assessment decisions by sharing understanding and aligning standards of performance.

Examiner workload comparison: three structured oral examination formats for the European diploma in anaesthesiology and intensive care.

The COVID-19 pandemic triggered transformations in academic medicine, rapidly adopting remote teaching and online assessments. Whilst virtual environments show promise in evaluating medical knowledge, their impact on examiner workload is unclear. This study explores examiner's workload during different European Diploma in Anaesthesiology and Intensive Care Part 2 Structured Oral Examinations formats. We hypothesise that online exams result in lower examiner's workload than traditional face-to-face methods. We also investigate workload structure and its correlation with examiner characteristics and marking performance. In 2023, examiner's workload for three examination formats (face-to-face, hybrid, online) using the NASA TLX instrument was prospectively evaluated. The impact of examiner demographics, candidate scoring agreement, and examination scores on workload was analysed. The overall NASA TLX score from 215 workload measurements in 142 examiners was high at 59.61 ± 14.13. The online examination had a statistically higher workload (61.65 ± 12.84) than hybrid but not face-to-face. Primary contributors to workload were mental and temporal demands, and effort. Online exams were associated with elevated frustration. Male examiners and those spending more time on exam preparation experienced a higher workload. Multiple diploma specialties and familiarity with European Diploma in Anaesthesiology and Intensive Care exams were protective against high workload. Perceived workload did not impact marking agreement or examination scores across all formats. Examiners experience high workload. Online exams are not systematically associated with decreased workload, likely due to frustration. Despite workload differences, no impact on examiner's performance or examination scores was found. The hybrid examination mode, combining face-to-face and online, was associated with a minor but statistically significant workload reduction. This hybrid approach may offer a more balanced and efficient examination process while maintaining integrity, cost savings, and increased accessibility for candidates.

Considering the Structured Oral Examinations Beyond Its Psychometrics Properties.

Decisions to set aside Structured Oral Examinations (SOE) are, almost invariably, based on their poor psychometric properties. However, considering the perspectives of the stakeholders might help us to understand its potential contribution. To explore this, we conducted focus groups and individual interviews with stakeholders: students, assessors, and administrators. Students and assessors perceived the SOE as a window on students' clinical reasoning, as an authentic assessment, but as a subjective and stressful method. Administrators emphasized the organizational consequences such as logistical challenges. Consequences must be considered when making decisions about SOE and our results support important positive consequences.

No Miracles in Two Minutes: A Randomized Controlled Study on the Impact of Preparatory Expansive Posing on Anesthesiology Residents' Performance in Mock Structured Oral Examinations.

Background: The objective of this study was to evaluate the impact of engaging in preparatory expansive posing on the performance of anesthesiology trainees during a mock structured oral examination. Methods: A total of 38 clinical residents at a single institution participated in this prospective randomized controlled study. Participants were stratified by clinical anesthesia year and randomly assigned to 1 of 2 orientation rooms to prepare for the examination. The preparatory expansive posing participants stood for 2 minutes with their hands and arms above their heads and with their feet approximately 1 ft apart. Conversely, the control participants sat quietly in a chair for 2 minutes. All participants then received the same orientation and examination. Faculty evaluation of resident performance, residents' self-assessment of performance, and anxiety score were collected. Results: There was no evidence to support our primary hypothesis that residents who engaged in preparatory expansive posing for 2 minutes prior to a mock structured oral examination would score higher than their control counterparts (P = .68). There was no evidence to support our secondary hypotheses that preparatory expansive posing increases self-assessment of one's performance (P = .31) or reduces perceived anxiety during a mock structured oral examination (P = .85). Conclusions: Preparatory expansive posing did not improve anesthesiology residents' mock structured oral examination performance or self-assessment of their performance, nor did it reduce their perceived anxiety. Preparatory expansive posing is likely not a useful technique in improving the performance of residents in structured oral examinations.

Promoting medical student's clinical reasoning during COVID-19 pandemic.

PURPOSE: The development of students' clinical reasoning skills should be a consideration in the design of instruction and evaluation in medical education. In response to the coronavirus disease 2019 (COVID-19) pandemic, several changes in the medical curriculum have been implemented in promoting clinical reasoning. This study aims to explore medical students' perceptions and experiences with the clinical reasoning curriculum during the COVID-19 pandemic and determine their skills development. METHODS: The study used a mixed-method design with a concurrent approach. A cross-sectional study was conducted to compare and examine the relationship between the outcomes of the structured oral examination (SOE) and the Diagnostic Thinking Inventory (DTI). Then, the qualitative method was used. A focus group discussion using a semi-structured interview guide with open-ended questions was conducted, then the verbatim transcript was subjected to thematic analysis. RESULTS: There is an increase in SOE and DTI scores between second-year to fourth-year students. The diagnostic thinking domains and SOE are significantly correlated (r=0.302, 0.313, and 0.241 with p<0.05). The three primary themes from the qualitative analysis are perceptions regarding clinical reasoning, clinical reasoning activities, and the learning component. CONCLUSION: Even if students are still studying throughout the COVID-19 pandemic, their clinical reasoning skills can improve. The clinical reasoning and diagnostic thinking skills of medical students increase as the length of the school year increases. Online case-based learning and assessment support the development of clinical reasoning skills. The skills are supported in their development by positive attitudes toward faculty, peers, case type, and prior knowledge.

Feasibility of radiology online structured oral examination for undergraduate medical students.

BACKGROUND: Online summative assessment has emerged during the COVID-19 pandemic as an alternative to traditional examinations, bringing opportunities and challenges. The study aims to evaluate the feasibility and effectiveness of online structured oral examination (SOE) in radiology clerkships. The study identifies measures taken to successfully implement online SOE and minimize chances of cheating. It also discusses the challenges encountered and how they were addressed. METHODS: SOE percent scores of fourth-year medical students from two institutions were correlated with students' grade point average (GPA). The scores were compared among different institutions, students' genders, students' batches, examination versions, and examiners with different experience levels. Students' perceived satisfaction and concerns were captured using anonymous self-administered questionnaire. Technical problems and success rate of SOE implementation were recorded. Results were analyzed using descriptive and inferential statistics. RESULTS: A total of 79 students participated in the study, out of which 81.0% (n = 64) responded to the survey. SOE scores showed poor positive correlation with the students' GPAs (r = 0.22, and p = .09). Scores showed no significant difference between the two institutions or genders. Scores were also not significantly different between students who were examined by junior or senior examiners. All but one version of examination showed no significant difference in students' scores. No significant difference was observed in students' scores between each two subsequent batches who were exposed to the same examination version. CONCLUSION: Online summative SOE is a feasible alternative whenever face-to-face SOE could not be implemented provided that appropriate measures are taken to ensure its successful execution.

Structured oral examination in pharmacology for undergraduate medical students: Factors influencing its implementation.

OBJECTIVES: The study aims to understand the process and factors influencing the implementation of structured oral examination (SOE) for undergraduate medical students; in comparison with conventional oral examination (COE) in pharmacology. METHODS: In a randomized, parallel group study, 123 students of pharmacology were divided into two groups, SOE (n = 63) and COE (n = 60). Students of each group were subdivided into two, and four examiners took viva voce individually. Three sets of questionnaires from autonomic nervous system were prepared, each having 15 items with increasing difficulty levels and were validated by subject experts and pretested. Ten minutes were allotted for each student for each viva. Feedback of students and faculty about the novel method was obtained. RESULTS: SOE yielded significantly lower marks as compared to COE. There were significant inter-examiner variations in marks awarded in SOE and COE. Other factors influencing implementation were difficulty in structuring viva, rigid time limits, lack of flexibility in knowledge content, monotony, and fatigue. The students perceived this format not different from COE but felt that it required in-depth preparation of topic. Faculty opined that SOE led to less drift from main topic and provided uniform coverage of topics in given time. CONCLUSION: Conducting SOE is a resource-intensive exercise. Despite structuring, inter-examiner variability was not completely eliminated. The students' performance was depended on factors related to examiners such as teaching experience, vernacular language used, and lack of training. Orientation and training of examiners in assessment strategies is necessary. Standardization of questionnaire is necessary before the implementation of SOE for summative assessment.