The reliability of the College of Intensive Care Medicine of Australia and New Zealand "Hot Case" examination.

BACKGROUND: High stakes examinations used to credential trainees for independent specialist practice should be evaluated periodically to ensure defensible decisions are made. This study aims to quantify the College of Intensive Care Medicine of Australia and New Zealand (CICM) Hot Case reliability coefficient and evaluate contributions to variance from candidates, cases and examiners. METHODS: This retrospective, de-identified analysis of CICM examination data used descriptive statistics and generalisability theory to evaluate the reliability of the Hot Case examination component. Decision studies were used to project generalisability coefficients for alternate examination designs. RESULTS: Examination results from 2019 to 2022 included 592 Hot Cases, totalling 1184 individual examiner scores. The mean examiner Hot Case score was 5.17 (standard deviation 1.65). The correlation between candidates' two Hot Case scores was low (0.30). The overall reliability coefficient for the Hot Case component consisting of two cases observed by two separate pairs of examiners was 0.42. Sources of variance included candidate proficiency (25%), case difficulty and case specificity (63.4%), examiner stringency (3.5%) and other error (8.2%). To achieve a reliability coefficient of > 0.8 a candidate would need to perform 11 Hot Cases observed by two examiners. CONCLUSION: The reliability coefficient for the Hot Case component of the CICM second part examination is below the generally accepted value for a high stakes examination. Modifications to case selection and introduction of a clear scoring rubric to mitigate the effects of variation in case difficulty may be helpful. Increasing the number of cases and overall assessment time appears to be the best way to increase the overall reliability. Further research is required to assess the combined reliability of the Hot Case and viva components.

Timing's not everything: Immediate and delayed feedback are equally beneficial for performance in formative multiple-choice testing.

INTRODUCTION: Test-enhanced learning (TEL) is an impactful teaching and learning strategy that prioritises active learner engagement through the process of regular testing and reviewing. While it is clear that meaningful feedback optimises the effects of TEL, the ideal timing of this feedback (i.e. immediate or delayed) in a medical education setting is unclear. METHOD: Forty-one second-year medical students were recruited from the University of Melbourne. Participants were given a multiple-choice question test with a mix of immediate (i.e. post-item) and delayed (i.e. post-item-block) conceptual feedback. Students were then tested on near and far transfer items during an immediate post-test, and at a one-week follow-up. RESULTS: A logistic mixed effects model was used to predict the probability of successful near and far transfer. As expected, participants in our study tended to score lower on far transfer items than they did on near transfer items. In addition, correct initial response on a parent question predicted subsequent correct responding. Contrary to our hypotheses, the feedback timing effect was non-significant-there was no discernible difference between feedback delivered immediately versus delayed feedback. DISCUSSION: The findings of this study suggest that the timing of feedback delivery (post-item versus post-item-block) does not influence the efficacy of TEL in this medical education setting. We therefore suggest that educators may consider practical factors when determining appropriate TEL feedback timing in their setting.

Diversity Audit of Medical School Examination Questions.

Phenomenon: This article reports the under-researched presentation of demographic, social, and economic diversity in medical school examination questions. Approach: The present study audited 3,566 pre-clinical and clinical multiple-choice and short answer examination questions in the same year (2018) from three medical schools in two continents to review the diversity of patients portrayed. The audit was based on an extension of Critical Race Theory beyond race and ethnicity to include pertinent social determinants of health. Findings: Patients were presented in 1,537 (43.1%) of the audited examination questions. Apart from age (89.4%) and binary genders (93.9%), other diversity characteristics were rarely portrayed (ethnicity 7.2%, relationship status 1.9%, sexual identity 1.1%, socio-economic status 0.5%, geographic residence 0.1%, disability 0.1%), or not at all (non-binary genders; residency status; religion/spirituality). Insights: While presenting excessive and unnecessary patient characteristics in examination questions should be avoided, the absence of many diversity aspects may reduce examination authenticity and defeat the teaching of diversity in medicine. Medical schools should consider a routine audit and reasonable improvement of the diversity features of patients in examination questions to support teaching and learning activities addressing patients' diversity.

Ottawa 2020 consensus statement for programmatic assessment - 1. Agreement on the principles.

INTRODUCTION: In the Ottawa 2018 Consensus framework for good assessment, a set of criteria was presented for systems of assessment. Currently, programmatic assessment is being established in an increasing number of programmes. In this Ottawa 2020 consensus statement for programmatic assessment insights from practice and research are used to define the principles of programmatic assessment. METHODS: For fifteen programmes in health professions education affiliated with members of an expert group (n = 20), an inventory was completed for the perceived components, rationale, and importance of a programmatic assessment design. Input from attendees of a programmatic assessment workshop and symposium at the 2020 Ottawa conference was included. The outcome is discussed in concurrence with current theory and research. RESULTS AND DISCUSSION: Twelve principles are presented that are considered as important and recognisable facets of programmatic assessment. Overall these principles were used in the curriculum and assessment design, albeit with a range of approaches and rigor, suggesting that programmatic assessment is an achievable education and assessment model, embedded both in practice and research. Knowledge on and sharing how programmatic assessment is being operationalized may help support educators charting their own implementation journey of programmatic assessment in their respective programmes.

Ottawa 2020 consensus statements for programmatic assessment - 2. Implementation and practice.

INTRODUCTION: Programmatic assessment is a longitudinal, developmental approach that fosters and harnesses the learning function of assessment. Yet the implementation, a critical step to translate theory into practice, can be challenging. As part of the Ottawa 2020 consensus statement on programmatic assessment, we sought to provide descriptions of the implementation of the 12 principles of programmatic assessment and to gain insight into enablers and barriers across different institutions and contexts. METHODS: After the 2020 Ottawa conference, we surveyed 15 Health Profession Education programmes from six different countries about the implementation of the 12 principles of programmatic assessment. Survey responses were analysed using a deductive thematic analysis. RESULTS AND DISCUSSION: A wide range of implementations were reported although the principles remained, for the most part, faithful to the original enunciation and rationale. Enablers included strong leadership support, ongoing faculty development, providing students with clear expectations about assessment, simultaneous curriculum renewal and organisational commitment to change. Most barriers were related to the need for a paradigm shift in the culture of assessment. Descriptions of implementations in relation to the theoretical principles, across multiple educational contexts, coupled with explanations of enablers and barriers, provided new insights and a clearer understanding of the strategic and operational considerations in the implementation of programmatic assessment. Future research is needed to further explore how contextual and cultural factors affect implementation.

Diagnostic Approaches to Neuroendocrine Neoplasms of Unknown Primary Site.

Neuroendocrine neoplasms (NENs) arise from cells of neuronal and endocrine differentiation. While they are a rare entity, an increasing proportion of patients with NEN present with metastatic disease and no evident primary site using routine imaging or histopathology. NENs of unknown primary site have a poorer prognosis, often due to the challenge of selecting appropriate evidence-based management. We review the available literature and guidelines for the management of NENs of unknown primary site including clinical features, biochemical tests, histopathology, imaging, surgical exploration and localised and systemic treatments. We also discuss novel molecular techniques currently under investigation to aid primary site identification.

Challenging feedback myths: Values, learner involvement and promoting effects beyond the immediate task.

CONTEXT: Research suggests that feedback in the health professions is less useful than we would like. In this paper, we argue that feedback has become reliant on myths that perpetuate unproductive rituals. Feedback often resembles a discrete episode of an educator "telling," rather than an active and iterative involvement of the learner in a future-facing process. With this orientation towards past events, it is not surprising that learners become defensive or disengaged when they are reminded of their deficits. METHODS: We tackle three myths of feedback: (a) feedback needs praise-criticism balancing rules; (b) feedback is a skill residing within the teacher; and (c) feedback is an input only. For each myth we provide a reframing with supporting examples from the literature. CONCLUSIONS: Equipping learners to engage in feedback processes may reduce the emotional burden on both parties, rendering techniques such as the feedback sandwich redundant. We also highlight the benefits for learners and teachers of conceptualising feedback as a relational activity, and of tracing the effects of information exchanges. These effects may be immediate or latent, and may manifest in different forms such as changes in learner evaluative judgement or professional identity.

Exploring Strategies for Using Social Media to Self-Manage Health Care When Living With and Beyond Breast Cancer: In-Depth Qualitative Study.

BACKGROUND: As breast cancer survival rates improve and structural health resources are increasingly being stretched, health providers require people living with and beyond breast cancer (LwBBC) to self-manage aspects of their care. OBJECTIVE: This study aimed to explore how women use and experience social media to self-manage their psychosocial needs and support self-management across the breast cancer continuum. METHODS: The experiences of 21 women (age range 27-64 years) were explored using an in-depth qualitative approach. The women varied in the duration of their experiences of LwBBC, which facilitated insights into how they evolve and change their self-management strategies over time. Semistructured interviews were analyzed inductively using a thematic analysis, a polytextual analysis, and voice-centered relational methods. RESULTS: The use of multiple social media platforms, such as YouTube, Facebook, WhatsApp, and Twitter, enabled women to self-manage aspects of their care by satisfying needs for timely, relevant, and appropriate support, by navigating identities disrupted by diagnosis and treatment and by allowing them to (re)gain a sense of control. Women described extending their everyday use of multiple platforms to self-manage their care. However, women experienced social media as both empowering and dislocating, as their engagement was impacted by their everyday experiences of LwBBC. CONCLUSIONS: Health care professionals (HCPs) need to be more aware, and open to the possibilities, of women using multiple social media resources as self-management tools. It is important for HCPs to initiate value-free discussions and create the space necessary for women to share how social media resources support a tailored and timely self-managed approach to their unique psychosocial needs.

Perceived educational impact of the medical student long case: a qualitative study.

BACKGROUND: The long case is a traditional method of clinical assessment which has fallen out of favour in certain contexts, primarily due to psychometric concerns. This study explored the long case's educational impact, an aspect which has been neglected in previous research. METHODS: Three focus groups of medical students (20 in total) and semi-structured interviews of six examiners were conducted. Cook and Lineberry's framework for exploring educational impact was used as a sensitising tool during thematic analysis of the data. RESULTS: Participants described the long case and its scoring as having influence on student learning. Engaging in the activity of a long case had an essential role in fostering students' clinical skills and served as a powerful driving force for them to spend time with patients. The long case was seen as authentic, and the only assessment to promote a holistic approach to patients. Students had concerns about inter-case variability, but there was general consensus that the long case was valuable, with allocation of marks being an important motivator for students. CONCLUSIONS: This study offers a unique focus on the traditional long case's educational consequences; the extent of its positive impact would support its place within a program of assessment.

Medical students as interpreters in health care situations: " it's a grey area".

OBJECTIVE: To examine the extent to which medical students in Australia are acting as interpreters in medical settings, and their perceptions of this activity. DESIGN, SETTING, PARTICIPANTS: Anonymous online survey of final year students in the graduate medical program of the University of Melbourne, undertaken in 2014. MAIN OUTCOME MEASURES: Numbers of students who had acted or who had been asked to act as ad hoc interpreters in health care situations during their clinical rotations and outside the medical education context. RESULTS: 146 of 319 final year medical students completed the survey (46% response). 106 students (73%) reported they could speak at least one language in addition to English; none had formal interpreting qualifications, but 40 (36%) had been asked to interpret during clinical rotations, and 36 (34%) had done so. The students described a diverse range of experiences, including complex interactions regarding informed consent and the breaking of bad news. CONCLUSION: Medical students frequently acted as interpreters during their clinical training. Most did not feel appropriately qualified to interpret in clinical situations, and some felt pressured to do so, but many found the experience positive. Our study highlights the lack of clear guidelines regarding medical student interpreters in Australian health care settings.

Validating the Sedentary Sphere method in children: Does wrist or accelerometer brand matter?

This study aimed to validate the Sedentary Sphere posture classification method from wrist-worn accelerometers in children. Twenty-seven 9-10-year-old children wore ActiGraph GT9X (AG) and GENEActiv (GA) accelerometers on both wrists, and activPAL on the thigh while completing prescribed activities: five sedentary activities, standing with a phone, walking (criterion for all 7: observation) and 10-min free-living play (criterion: activPAL). In an independent sample, 21 children wore AG and GA accelerometers on the non-dominant wrist and activPAL for two days of free-living. Per cent accuracy, pairwise 95% equivalence tests (±10% equivalence zone) and intra-class correlation coefficients (ICC) analyses were completed. Accuracy was similar, for prescribed activities irrespective of brand (non-dominant wrist: 77-78%; dominant wrist: 79%). Posture estimates were equivalent between wrists within brand (±6%, ICC > 0.81, lower 95% CI ≥ 0.75), between brands worn on the same wrist (±5%, ICC ≥ 0.84, lower 95% CI ≥ 0.80) and between brands worn on opposing wrists (±6%, ICC ≥ 0.78, lower 95% CI ≥ 0.72). Agreement with activPAL during free-living was 77%, but sedentary time was underestimated by 7% (GA) and 10% (AG). The Sedentary Sphere can be used to classify posture from wrist-worn AG and GA accelerometers for group-level estimates in children, but future work is needed to improve the algorithm for better individual-level results.

Demographics and study practices of psychiatry trainees sitting the RANZCP written examinations.

OBJECTIVES: This study explored trainee preparation for the Royal Australian and New Zealand College of Psychiatrists (RANZCP) written examinations. We examined the relationship between candidates' prior psychiatry experience, preparation techniques and expenses, perceptions of the exam and a pass result. We also examined perceptions of well-being. METHOD: A web-based survey was sent to RANZCP trainees. The association between responses and exam outcome was analysed using SPSS Statistics 23. RESULTS: A total of 23% ( n = 38) of the cohort responded to the survey. Respondents studied for an average of 7.8 months for the essay-style exam and 4.4 months for the multiple-choice questions. The mean months of total psychiatry experience prior to sitting was 36.1 months. Every additional 50 hours of study increased odds of passing by 23%. Respondents who believed that exam preparation would affect outcome had an almost 4-fold higher odds of passing than those who did not. Ratings of well-being before release of results were also positively associated with passing. Conclusions Trainees are sitting the written exams with clinical experience commensurate with RANZCP recommendations. Total study hours and favourable perceptions of exam processes appear to be predictors of success.

Tensions in post-examination feedback: information for learning versus potential for harm.

OBJECTIVE: Self-regulation is recognised as being a requisite skill for professional practice This study is part of a programme of research designed to explore efficient methods of feedback that improve medical students' ability to self-regulate their learning. Our aim was to clarify how students respond to different forms and content of written feedback and to explore the impact on study behaviour and knowledge acquisition. METHODS: Year 2 students in a 4-year graduate entry medical programme completing four formative progress tests during the academic year were randomised into three groups receiving different feedback reports. All reports included proportion correct overall and by clinical rotation. One group received feedback reports including lists of clinical presentations relating to questions answered correctly and incorrectly; another group received reports containing this same information in combination with response certitude. The final group received reports involving normative comparisons. Baseline progress test performance quartile groupings (a proxy for academic ability) were determined by results on the first progress test. A mixed-method approach with triangulation of research findings was used to interpret results. Outcomes of interest included progress test scores, summative examination results and measures derived from study diaries, questionnaires and semi-structured interviews. RESULTS: Of the three types of feedback provided in this experiment, feedback containing normative comparisons resulted in inferior test performance for students in the lowest performance quartile group. This type of feedback appeared to stimulate general rather than examination-focused study. CONCLUSIONS: Medical students are often considered relatively homogenous and high achieving, yet the results of this study suggest caution when providing them with normative feedback indicating poorer performance relative to their peers. There is much need for further work to explore efficient methods of providing written feedback that improves medical students' ability to self-regulate their learning, particularly when giving feedback to those students who have the most room for improvement.

Hospitalisation impacts on oral hygiene: an audit of oral hygiene in a metropolitan health service.

BACKGROUND: Poor oral health has been associated with systemic diseases, morbidity and mortality. Many patients in hospital environments are physically compromised and rely upon awareness and assistance from health professionals for the maintenance or improvement of their oral health. This study aimed to identify whether common individual and environment factors associated with hospitalisation impacted on oral hygiene. METHODS: Data were collected during point prevalence audits of patients in the acute and rehabilitation environments on three separate occasions. Data included demographic information, plaque score, presence of dental hygiene products, independence level and whether nurse assistance was documented in the health record. RESULTS: Data were collected for 199 patients. A higher plaque score was associated with not having a toothbrush (p = 0.002), being male (p = 0.007), being acutely unwell (p = 0.025) and requiring nursing assistance for oral hygiene (p = 0.002). There was fair agreement between the documentation of requiring assistance for oral care and the patient independently able to perform oral hygiene (ICC = 0.22). CONCLUSION: Oral hygiene was impacted by factors arising from hospitalisation, for those without a toothbrush and male patients of acute wards. Establishment of practices that increase awareness and promote good oral health should be prioritised.

Effectiveness of a national approach to prescribing education for multiple disciplines.

AIMS: To evaluate the effectiveness of a national approach to prescribing education on health professional students' prescribing and therapeutics knowledge, across multiple disciplines. METHODS: In a university examination setting, 83 medical, 40 pharmacy and 13 nurse practitioner students from three different universities completed a set of multiple choice questions (MCQs) before and after completing an online module from the National Prescribing Curriculum (NPC). To minimize overestimation of knowledge, students had to indicate the level of certainty for each answer on a three-point scale. MCQs were scored using a validated certainty-based marking scheme resulting in a composite score (maximum 30 and minimum -60). Students were asked to rate their perception of usefulness of the module. RESULTS: At the pre-module phase, there were no significant differences in the composite MCQ scores between the medical (9.0 ± 10.3), pharmacy (10.2 ± 10.6) and nurse practitioner (8.0 ± 10.7) students. The scores improved significantly for all groups at the post-module phase (P < 0.01 for all groups) by similar extents (post-module results: medical, 14.5 ± 9.6; pharmacy, 14.4 ± 9.9; nurse practitioner, 12.1 ± 9.6). 39.4% of the MCQs answered incorrectly with high level of certainty at the pre-module phase were still answered incorrectly with high level of certainty at the post-module phase. Almost all students (with no significant difference between the groups) found the NPC modules, post-module MCQs and feedback useful as a learning tool. CONCLUSIONS: A national online approach to prescribing education can improve therapeutics knowledge of students from multiple disciplines of health care and contribute towards streamlining interdisciplinary learning in medication management.

Too hot to handle? Assessing the validity and reliability of the College of Intensive Care Medicine "Hot Case" examination.

The College of Intensive Care Medicine of Australia and New Zealand is responsible for credentialling trainees for specialist practice in intensive care medicine for the safety of patients and the community. This involves defining trainees' performance standards and testing trainees against those standards to ensure safe practice. The second part examination performed towards the end of the training program is a high-stakes assessment. The two clinical "Hot Cases" performed in the examination have a low pass rate, with most candidates failing at least one of the cases. There is increasing expectation for medical specialist training colleges to provide fair and transparent assessment processes to enable defensible decisions regarding trainee progression. Examinations are a surrogate marker of clinical performance with advantages, disadvantages and inevitable compromises. This article evaluates the Hot Case examination using Kane's validity framework and van der Vleuten's utility equation, and identifies issues with validity and reliability which could be managed through an ongoing improvement process.

Football Fandom as a Platform for Digital Health Promotion and Behaviour Change: A Mobile App Case Study.

Background: The last decade has seen a dramatic shift toward the study of fitness surveillance, thanks in part to the emergence of mobile health (mHealth) apps that allow users to track their health through a variety of data-driven insights. This study examines the adoption trends and community mediation of the mobile fitness application 'FanFit', a platform aimed at promoting physical activity among sports fans by creating a fitness app branded to their favourite team for health promotion. Objective: Our study looked at the impact of a specially designed mobile app (FanFit) as a digital health intervention for initiating and maintaining physical activity as part of football club membership. Our analysis indicates that app users will adopt healthier behaviours as a result of the app's sense of fan community and behaviour change. Methods: The findings reported here are based on an implementation of the FanFit app and, in particular, on those who participated in a more in-depth study (n = 30). These participants were Rangers FC supporters with a mix of genders (n = 19 males and n = 11 females). Focus groups and interviews were conducted with participants to ascertain users' perspectives on the most effective methods for nudging users toward adopting and maintaining a pattern of fitness behaviours. Results: The findings show that the user community was interested in fitness and wanted to live a 'healthy lifestyle,' which was augmented and fuelled by the app's competitive architecture design. Furthermore, the data reveal a new fan-health discourse about a person's developing wants, talents, and identities as embodied beings. Conclusions: We have developed and presented valid links between the use of sports club apps and health programmes. The app could be useful for sports programmes and club providers looking for mHealth applications that provide community support through fan discourse with opportunities for both male and female fans.