Twelve tips for developing effective marking schemes for constructed-response examination questions.

Constructed-response questions (CRQs) require effective marking schemes to ensure that the intended learning objectives and/or professional competencies are appropriately addressed, and valid inferences regarding examinee competence are drawn from such assessments. While the educational literature on writing rubrics has proliferated in recent years, this is largely targeted at classroom use and formative purposes. There is comparatively little guidance on how to develop appropriate marking schemes for summative assessment contexts. The different purposes mean that different principles and practices apply to mark schemes for examinations. In this article, we draw on the educational literature as well as our own practical experience of working with medical and health professional educators on their questions and marking schemes to offer 12 key principles or tips for designing and implementing effective marking schemes.

Assessor discomfort and failure to fail in clinical performance assessments.

BACKGROUND: Assessment of trainee performance in the workplace is critical to ensuring high standards of clinical care. However, some supervisors find the task to be challenging, and may feel unable to deliver their true judgement on a trainee's performance. They may 'keep MUM' (that is, keep mum about undesirable messages) and fail to fail an underperforming trainee. In this study, we explore the effect of discomfort on assessors. METHODS: Using a survey method, supervisors of trainees in the Australasian College of Dermatologists were asked to self-report experiences of discomfort in various aspects of trainee workplace assessment and for their engagement in MUM behaviours including failure to fail. RESULTS: Sixty-one responses were received from 135 eligible assessors. 12.5% of assessors self-reported they had failed to fail a trainee and 18% admitted they had grade inflated a trainee's score on a clinical performance assessment in the previous 12-month period. Assessors who reported higher levels of discomfort in the clinical performance assessment context were significantly more likely to report previously failing to fail a trainee. The study did not reveal significant associations with assessor demographics and self-reports of discomfort or MUM behaviours. CONCLUSIONS: This study reveals the impact of assessor discomfort on the accuracy of assessment information and feedback to trainees, including as a contributing factor to the failure to fail phenomenon. Addressing assessor experience of discomfort offers one opportunity to impact on the complex and multifactorial issue that failure to fail represents.

Diversity Audit of Medical School Examination Questions.

Phenomenon: This article reports the under-researched presentation of demographic, social, and economic diversity in medical school examination questions. Approach: The present study audited 3,566 pre-clinical and clinical multiple-choice and short answer examination questions in the same year (2018) from three medical schools in two continents to review the diversity of patients portrayed. The audit was based on an extension of Critical Race Theory beyond race and ethnicity to include pertinent social determinants of health. Findings: Patients were presented in 1,537 (43.1%) of the audited examination questions. Apart from age (89.4%) and binary genders (93.9%), other diversity characteristics were rarely portrayed (ethnicity 7.2%, relationship status 1.9%, sexual identity 1.1%, socio-economic status 0.5%, geographic residence 0.1%, disability 0.1%), or not at all (non-binary genders; residency status; religion/spirituality). Insights: While presenting excessive and unnecessary patient characteristics in examination questions should be avoided, the absence of many diversity aspects may reduce examination authenticity and defeat the teaching of diversity in medicine. Medical schools should consider a routine audit and reasonable improvement of the diversity features of patients in examination questions to support teaching and learning activities addressing patients' diversity.

Accreditation Approaches for Professional Education Programs: Towards Best Practice.

Accreditation has a central role in the quality assurance of professional education programs, but research on the effectiveness of different models of accreditation is limited. The purpose of this study was to rapidly appraise the evidence for the effectiveness, impact, and feasibility of different accreditation approaches, in order to inform best practices for the accreditation of professional education programs. The study focused on accreditation for programs which produce practice-ready graduates, including for veterinary programs. The authors searched several databases for articles published from 2000 to 2020, using search terms identified during a scoping phase, and applied a "rapid review" methodology in line with contextual, time, and resource requirements. Relevant articles which were classed as empirical or conceptual were included in the study, while papers appraised as solely commentaries or descriptive were excluded from the evidence base. The full-text review included 32 articles. We identified a clear transition in the literature from input- and process-based models (pre- and early 2000s) to outcomes-based models (in the 2000s and early 2010s). Continuous quality improvement and targeted models (including risk-based and thematic) represent more recent approaches in accreditation practice. However, as noted by other scholars, we identified limited empirical evidence for the relative effectiveness of different accreditation approaches in professional education, although evidence for the more recent accreditation approaches is emerging. In terms of best practice in view of the current lack of definitive evidence for the adoption of any specific model of accreditation, we argue that accrediting authorities adopt a contextual approach to accreditation which includes clearly articulating the purpose and focus of their regulatory activities, and selecting and implementing accreditation methods that are consistent with their underlying principles.

On the use and abuse of metaphors in assessment.

This paper is motivated by a desire to advance assessment in the health professions through encouraging the judicious and productive use of metaphors. Through five specific examples (pixels, driving lesson/test, jury deliberations, signal processing, and assessment as a toolbox), we interrogate how metaphors are being used in assessment to consider what value they add to understanding and implementation of assessment practices. By unpacking these metaphors in action, we probe each metaphor's rationale and function, the gains each metaphor makes, and explore the unintended meanings they may carry. In summarizing common uses of metaphors, we elucidate how there may be both advantages and/or disadvantages. Metaphors can play important roles in simplifying, complexifying, communicating, translating, encouraging reflection, and convincing. They may be powerfully rhetorical, leading to intended consequences, actions, and other pragmatic outcomes. Although metaphors can be extremely helpful, they do not constitute thorough critique, justified evidence or argumentation. We argue that although metaphors have utility, they must be carefully considered if they are to serve assessment needs in intended ways. We should pay attention to how metaphors may be misinterpreted, what they ignore or unintentionally signal, and perhaps mitigate against this with anticipated corrections or nuanced qualifications. Failure to do so may lead to implementing practices that miss underlying and relevant complexities for assessment science and practice. Using metaphors requires careful attention with respect to their role, contributions, benefits and limitations. We highlight the value that comes from critiquing metaphors, and demonstrate the care required to ensure their continued utility.

Effects of an Online Education Program on Physical Therapists' Confidence in Weight Management for People With Osteoarthritis: A Randomized Controlled Trial.

OBJECTIVE: To evaluate effects of an online education program about weight management for osteoarthritis on physical therapists' self-reported confidence in knowledge and skills in weight management and attitudes toward obesity. METHODS: In a 2-group randomized controlled trial, 80 physical therapists (58 female physical therapists) were randomized to education or control groups. The theoretically informed and evidence-informed online self-directed training program covered biopsychosocial elements of obesity and weight management. The primary outcome was self-reported confidence in knowledge in weight management using a customized validated tool (scale 14-70, higher scores indicating higher confidence) assessed at baseline and 6 weeks. Secondary measures included confidence in nutrition care, clinical skills in weight management, and weight stigma. Process measures evaluated participant experience. Differences in change between groups were compared using linear regression models adjusted for baseline scores and stratifying variables (clinical setting; confidence in weight management). Moderation analysis was performed using an interaction approach in a linear regression model and multivariable fractional polynomial interaction approach. RESULTS: A total of 79 participants (99%) completed outcome measures at 6 weeks. The education group demonstrated greater improvement in confidence in knowledge than the control group (adjusted mean difference 22.6 units, 95% confidence interval 19.6, 25.5). Greater improvement in knowledge was associated with lower baseline values (interaction P = 0.002). Secondary outcomes showed greater improvements in confidence in skills and nutrition care and in weight stigma domains favoring the education group. Over 90% of participants would recommend the program to peers. CONCLUSION: An online education program improves physical therapists' short-term confidence in knowledge and skills in weight management for people with osteoarthritis and reduces weight-stigmatized attitudes.

Medical Humanities Education and Its Influence on Students' Outcomes in Taiwan: A Systematic Review.

Background: Medical education has emphasized the importance of integrating medical humanities training into the curriculum to benefit medical and nursing students' future practice, featuring in the list of national funding priorities for healthcare education research in Taiwan for many years. However, the extent to which this drive has resulted in medical humanities training, what rationales underpin its inclusion, and its efficacy is largely unknown. This study aims to address these issues across medical humanities programs within the Taiwanese context. Methods: We conducted a systematic review. Inclusion criteria included studies in English or Mandarin reporting outcomes of medical humanities courses in healthcare education settings in Taiwan between 2000 and 2019. We searched across five electronic databases (PubMed, Embase, ERIC, PsycInfo, Web of Science), following PRISMA guidelines. The Best Evidence Medical Education (BEME) Global Scale and Kirkpatrick Levels are used for identifying the strength of evidence. Results: 17 articles were extracted from the 134 identified. Intrinsic and instrumental rationales for the inclusion of medical humanities education were common, compared with epistemological-based and critical-based approaches. Several positive impacts were identified in relation to participation including modification of attitudes, knowledge, and skills. However, the highest level (i.e., unequivocal) of evidence characterized by effects on students' behaviors or ongoing interaction with colleagues and patients is lacking. Conclusion: Findings suggest that although medical humanities education is widely implemented in Taiwan, no clear consensus has been reached regarding the rationale for inclusion or how it is localized from Western to Asian contexts. Future research still needs to explore the long-term impact of medical humanities education for medical and nursing students and its impact on patient care. Systematic Review Registration: https://www.crd.york.ac.uk/PROSPERO, identifier: CRD42019123967.

In-verse reflection: structured creative writing exercises to promote reflective learning in medical students.

Medical educators recognize the value of reflection for medical students and the role creative writing can play in fostering this. However, direct creative writing tasks can be challenging for many students, particularly those with limited experience in the arts and humanities. An alternative strategy is to utilize an indirect approach, engaging students with structured tasks that obliquely encourage reflection. This paper reports one such approach. We refer to this approach as in-verse reflection, playing on both the structure of the writing and its novel approach to reflection. Students were invited to write, in verse-like structures, about their personal and clinical experiences as medical students. Thematic analysis of their creative outputs and reactions identified four principal themes: the challenges of life as a medical student, the emotional demands of the medical course, a sense of connectedness and solidarity with fellow students, and a sense of marginality within the hospital system. Students generally found the tasks highly engaging and conducive to reflection, producing texts representing significant insights into their experiences as medical students. The reported method offers a relatively simple, structured, and guided approach to reflective writing, adding to the repertoire of methods available to educators in the medical humanities.

Assessing a Capstone Research Project in Medical Training: Examiner Consistency Using Generic Versus Domain-Specific Rubrics.

Rubrics are utilized extensively in tertiary contexts to assess student performance on written tasks; however, their use for assessment of research projects has received little attention. In particular, there is little evidence on the reliability of examiner judgements according to rubric type (general or specific) in a research context. This research examines the concordance between pairs of examiners assessing a medical student research project during a two-year period employing a generic rubric followed by a subsequent two-year implementation of task-specific rubrics. Following examiner feedback, and with consideration to the available literature, we expected the task-specific rubrics would increase the consistency of examiner judgements and reduce the need for arbitration due to discrepant marks. However, in contrast, results showed that generic rubrics provided greater consistency of examiner judgements and fewer arbitrations compared with the task-specific rubrics. These findings have practical implications for educational practise in the assessment of research projects and contribute valuable empirical evidence to inform the development and use of rubrics in medical education.

The do's, don'ts and don't knows of redressing differential attainment related to race/ethnicity in medical schools.

INTRODUCTION: Systematic and structural inequities in power and privilege create differential attainment whereby differences in average levels of performance are observed between students from different socio-demographic groups. This paper reviews the international evidence on differential attainment related to ethnicity/race in medical school, drawing together the key messages from research to date to provide guidance for educators to operationalize and enact change and identify areas for further research. METHODS: Authors first identified areas of conceptual importance within differential attainment (learning, assessment, and systems/institutional factors) which were then the focus of a targeted review of the literature on differential attainment related to ethnicity/race in medical education and, where available and relevant, literature from higher education more generally. Each author then conducted a review of the literature and proposed guidelines based on their experience and research literature. The guidelines were iteratively reviewed and refined between all authors until we reached consensus on the Do's, Don'ts and Don't Knows. RESULTS: We present 13 guidelines with a summary of the research evidence for each. Guidelines address assessment practices (assessment design, assessment formats, use of assessments and post-hoc analysis) and educational systems and cultures (student experience, learning environment, faculty diversity and diversity practices). CONCLUSIONS: Differential attainment related to ethnicity/race is a complex, systemic problem reflective of unequal norms and practices within broader society and evident throughout assessment practices, the learning environment and student experiences at medical school. Currently, the strongest empirical evidence is around assessment processes themselves. There is emerging evidence of minoritized students facing discrimination and having different learning experiences in medical school, but more studies are needed. There is a pressing need for research on how to effectively redress systemic issues within our medical schools, particularly related to inequity in teaching and learning.

Interassessor agreement of portfolio-based competency assessment for orthotists/prosthetists in Australia: a mixed method study.

BACKGROUND: Internationally qualified orthotists/prosthetists who want to practice in Australia must pass a portfolio-based competency assessment. Testing the agreement between independent assessors is important to engender confidence in the assessment, and continually improve the processes. OBJECTIVES: To quantify interassessor agreement for all 68 performance indicators in the Australian Orthotic Prosthetic Association's Entry Level Competency Standards and where there was significant disagreement between assessors, to explore the reasons why. STUDY DESIGN: Mixed methods: explanatory sequential. METHOD: Fifteen portfolios were assigned to independent assessors. Assessors determined whether the evidence presented met the requirements of each performance indicator. Interassessor agreement was calculated using Gwet's Agreement Coefficient 1 (AC1), and these data informed semistructured interviews to explore the reasons for disagreement. RESULTS: Most performance indicators (87%) had moderate to substantial agreement (AC1 > 0.71), which could be attributed to a variety of factors including the use of a simple assessment rubric with supporting guidelines and assessor training to establish shared expectations. The remaining performance indicators (13%) had fair to slight agreement (AC1 ≤ 0.7). Interviews with assessors suggested that disagreement could be attributed to the complexity of some performance indicators, unconscious bias, and the appropriateness of the evidence presented. CONCLUSIONS: Although most performance indicators in Australian Orthotic Prosthetic Association's Entry Level Competency Standard were associated with moderate to substantial interassessor agreement, there are opportunities to improve agreement by simplifying the wording of some performance indicators and revising guidelines to help applicants curate the most appropriate evidence for each performance indicator.

Design and Delivery of the Clinical Integrative Puzzle as a Collaborative Learning Tool.

Well-designed collaborative learning tools can provide an opportunity for engaging student experiences that foster deep learning and act as a scaffold for enculturation to the profession through refinement of professional collaborative skills. The clinical integrative puzzle is a paper-and-pencil or computer-based teaching and learning activity that combines disciplinary knowledge with clinical reasoning and problem solving. Effective design and implementation of clinical integrative puzzles requires a multidisciplinary approach to design, a positive classroom climate, and a set of illness scripts (e.g., clinical cases or scenarios) that are similar yet have key differentiating features that provide students with the opportunity to exercise clinical reasoning skills. The tool allows students to co-construct knowledge and develop professional competencies and allows instructors to assess and respond to student learning in a safe and supportive environment, even with large student numbers. The tool can also be used in a summative fashion. This article provides a brief review of the use of this instructional tool and offers tips for design and implementation.

Incorporating aspects of programmatic assessment into examinations: Aggregating rich information to inform decision-making.

CONTEXT: A programmatic approach to assessment entails gathering and aggregating 'rich information' on candidates to inform progress decisions. However, there is little guidance on how such an approach might be implemented in practice. OBJECTIVE: We describe an approach to aggregating rich information across assessment formats to inform committee decision-making in a specialist medical college. METHODS: Each item (n = 272) for every examination was blueprinted to 15 curriculum modules and 7 proficiencies. We developed a six-point holistic rating scale with detailed rubrics outlining expected performance standards for every item. Examiners used this rating scale in making judgements for each item, generating rich performance data for each candidate. RESULTS: A colour-coded 'mosaic' of patterns of performance across modules and proficiencies was generated along with frequency distributions of ratings. These data allowed examiners to easily visualise candidate performance and to use these data to inform deliberations on borderline candidates. Committee decision-making was facilitated by maintaining the richness of assessment information throughout the process. Moreover, the data facilitated detailed and useful feedback to candidates. CONCLUSIONS: Our study demonstrates that incorporating aspects of programmatic thinking into high-stakes examinations by using a novel approach to aggregating information is a useful first step in reforming an assessment program.

Prompting Candidates in Oral Assessment Contexts: A Taxonomy and Guiding Principles.

Prompting is an aspect of oral assessment that deserves more attention. There appears to be considerable variation in how practitioners conceptualise prompting and how it is deployed in practice. In order to unpack the term and promote the validity of its use in performance assessments, we present a taxonomy of prompting as a continuum of types, namely: presenting the task; repeating information; clarifying questions; probing questions; and finally, leading questions. We offer general principles for consideration when using prompting in oral assessment: neutrality; consistency; transparency; and reflexivity. Whenever oral assessment is planned, assessors should be appropriately trained in the type and degree of prompting required, and candidates suitably briefed to know what to expect. Overall, we aim to raise awareness that quite different behaviours tend to be subsumed under the general term 'prompting'. This paper provides concrete guidelines for implementing the defensible and effective use of prompting in oral examinations, applicable to a wide range of assessment contexts.

Perceived educational impact of the medical student long case: a qualitative study.

BACKGROUND: The long case is a traditional method of clinical assessment which has fallen out of favour in certain contexts, primarily due to psychometric concerns. This study explored the long case's educational impact, an aspect which has been neglected in previous research. METHODS: Three focus groups of medical students (20 in total) and semi-structured interviews of six examiners were conducted. Cook and Lineberry's framework for exploring educational impact was used as a sensitising tool during thematic analysis of the data. RESULTS: Participants described the long case and its scoring as having influence on student learning. Engaging in the activity of a long case had an essential role in fostering students' clinical skills and served as a powerful driving force for them to spend time with patients. The long case was seen as authentic, and the only assessment to promote a holistic approach to patients. Students had concerns about inter-case variability, but there was general consensus that the long case was valuable, with allocation of marks being an important motivator for students. CONCLUSIONS: This study offers a unique focus on the traditional long case's educational consequences; the extent of its positive impact would support its place within a program of assessment.

Assessing Diagnostic Reasoning Using a Standardized Case-Based Discussion.

BACKGROUND: Development of diagnostic reasoning (DR) is fundamental to medical students' training, but assessing DR is challenging. Several written assessments focus on DR but lack the ability to dynamically assess DR. Oral assessment formats have strengths but have largely lost favour due to concerns about low reliability and lack of standardization. Medical schools and specialist medical colleges value many forms of oral assessment (eg, long case, Objective Structured Clinical Examination [OSCE], viva voce) but are increasingly searching for ways in which to standardize these formats. We sought to develop and trial a Standardized Case-Based Discussion (SCBD), a highly standardized and interactive oral assessment of DR. METHODS: Two initial cohorts of medical students (n = 319 and n = 342) participated in the SCBD as part of their assessments. All students watch a video trigger (based on an authentic clinical case) and discuss their DR with an examiner for 15 minutes. Examiners probe students' DR and assess how students respond to new standardized clinical information. An online examiner training module clearly articulates expected student performance standards. We used student achievement and student and examiner perceptions to gauge the performance of this new assessment form over 2 implementation years. RESULTS: The SCBD was feasible to implement for a large student cohort and was acceptable to students and examiners. Most students and all examiners agreed that the SCBD discussion provided useful information on students' DR. The assessment had acceptable internal consistency, and the associations with other assessment formats were small and positive, suggesting that the SCBD measures a related, yet novel construct. CONCLUSIONS: Rigorous, standardized oral assessments have a place in a programme of assessment in initial medical training because they provide opportunities to explore DR that are limited in other formats. We plan to incorporate an SCBD into our clinical assessments for the first year of clinical training, where teaching and assessing basic DR is emphasized. We will also explore further examiners' understanding of and approach to assessing DR.