A narrative review of adaptive testing and its application to medical education.

Adaptive testing has a long but largely unrecognized history. The advent of computer-based testing has created new opportunities to incorporate adaptive testing into conventional programmes of study. Relatively recently software has been developed that can automate the delivery of summative assessments that adapt by difficulty or content. Both types of adaptive testing require a large item bank that has been suitably quality assured. Adaptive testing by difficulty enables more reliable evaluation of individual candidate performance, although at the expense of transparency in decision making, and requiring unidirectional navigation. Adaptive testing by content enables reduction in compensation and targeted individual support to enable assurance of performance in all the required outcomes, although at the expense of discovery learning. With both types of adaptive testing, candidates are presented a different set of items to each other, and there is the potential for that to be perceived as unfair. However, when candidates of different abilities receive the same items, they may receive too many they can answer with ease, or too many that are too difficult to answer. Both situations may be considered unfair as neither provides the opportunity to demonstrate what they know. Adapting by difficulty addresses this. Similarly, when everyone is presented with the same items, but answer different items incorrectly, not providing individualized support and opportunity to demonstrate performance in all the required outcomes by revisiting content previously answered incorrectly could also be considered unfair; a point addressed when adapting by content. We review the educational rationale behind the evolution of adaptive testing and consider its inherent strengths and limitations. We explore the continuous pursuit of improvement of examination methodology and how software can facilitate personalized assessment. We highlight how this can serve as a catalyst for learning and refinement of curricula; fostering engagement of learner and educator alike.

'Everyone is trying to outcompete each other': a qualitative study of medical student attitudes to a novel peer-assessed undergraduate teamwork module.

The centrality of teamwork in ensuring the effective functioning of institutions across all sectors is undeniable. However, embedding teamwork into higher education has been hampered due to a range of deeply entrenched practices associated broadly with the foregrounding of knowledge, beliefs about the place of skill training and routines of assessment. As a result, despite an urgent need to address teamwork, little progress has been made with respect to progressing teamwork education. We have designed and evaluated a novel teamwork module delivered to fourth-year undergraduate medical students involving placements, a cocreated piece of work, reflection and summative peer assessment. This paper aimed to investigate whether the module increased students' insight into teamwork, including their own skill development, and whether their perceptions of teamwork changed. Throughout the evaluation, students played a key role, with four final-year medical students working alongside others in the multidisciplinary project team. Five distinct themes emerged from our in-depth, semi-structured interviews: (a) importance and meaning; (b) insight into skill development; (c) transferability; (d) peer assessment; and (e) resistance to teamwork education. Themes had positive and negative components, and student perceptions changed in multiple ways after experiencing a longitudinal educational opportunity to develop their teamwork skills. Before practice, students focused on superficial explanations and on where they might improve. In contrast, after practice, students conveyed deeper insights, contextualisation, focus on how they might improve, and shared structured reflection.

Standard setting anchor statements: a double cross-over trial of two different methods.

This article was migrated. The article was marked as recommended. Context: We challenge the philosophical acceptability of the Angoff method, and propose an alternative method of standard setting based on how important it is for candidates to know the material each test item assesses, and not how difficult it is for a subgroup of candidates to answer each item. Methods: The practicalities of an alternative method of standard setting are evaluated here, for the first time, with direct comparison to an Angoff method. To negate bias due to any leading effects, a prospective cross-over design was adopted involving two groups of judges (n=7 and n=8), both of which set the standards for the same two 100 item multiple choice question tests, by the two different methods. Results: Overall, we found that the two methods took a similar amount of time to complete. The alternative method produced a higher cut-score (by 12-14%), and had a higher degree of variability between judges' cut-scores (by 5%). When using the alternative method, judges reported a small, but statistically significant, increase in their confidence to decide accurately the standard (by 3%). Conclusion: This is a new approach to standard setting where the quantitative differences are slight, but there are clear qualitative advantages associated with use of the alternative method.

Timing is key to providing modified assessments for students with specific learning difficulties.

INTRODUCTION: Medical students who are diagnosed with a specific learning difficulty (SpLD) will typically receive a reasonable adjustment within examinations in the form of modified assessment provision (MAP). This study investigated whether the timing of SpLD diagnosis and subsequent implementation of MAP has an impact on performance in applied medical knowledge multiple choice question (MCQ) assessments. METHOD: The MCQ performance of 108 students diagnosed with SpLD who received a MAP was monitored and compared with 1960 students who received an unmodified assessment, over 5 years of a medical program. Students who received a SpLD diagnosis in the latter years of the program were identified as not receiving a MAP in assessments prior to diagnosis. RESULTS: Differences were found between declaration and diagnosis, with 44.4% of students who declared and 48.1% who did not declare subsequently receiving a diagnosis. Students with SpLD who receive a MAP increase their applied medical knowledge assessment performance, although there is a delay of up to a year for this impact to reach significance. CONCLUSION: Early diagnosis of SpLD is necessary to ensure the intended benefit is received from MAP.

Improving communication for learning with students: expectations, feedback and feedforward.

This article was migrated. The article was marked as recommended. The aim of this paper is to draw together numerous strands from within the literature and our own practice to provide advice for improving communication about learning with students in undergraduate medical education. There is an assumption within higher education that assessment drives learning and, as such, assessment forms the focal points for communication between teachers and students. However, a broader approach is required to avoid misunderstandings and maximise the successful engagement with learning of everyone involved. It is important to plan a clear communication strategy that incorporates and enables identification with the unique values of the particular school. Where communication about learning is overtly discussed there are three main areas to consider: (1) management of expectations (sometimes referred to as feed up or feed in) that needs to include not only the use of authentic formative assessments, but also the viewpoints of both teachers and students. (2) Feedback and (3) Feedforward, both need to be considered from the perspectives of student and teacher. All communication needs to be inclusive, it's structures must provide scaffolding for respectful exchanges of information, and this will have clear practical consequences for the activities within the school.

Analysis of question text properties for equality monitoring.

INTRODUCTION: Ongoing monitoring of cohort demographic variation is an essential part of quality assurance in medical education assessments, yet the methods employed to explore possible underlying causes of demographic variation in performance are limited. Focussing on properties of the vignette text in single-best-answer multiple-choice questions (MCQs), we explore here the viability of conducting analyses of text properties and their relationship to candidate performance. We suggest that such analyses could become routine parts of assessment evaluation and provide an additional, equality-based measure of an assessment's quality and fairness. METHODS: We describe how a corpus of vignettes can be compiled, followed by examples of using Microsoft Word's native readability statistics calculator and the koRpus text analysis package for the R statistical analysis environment for estimating the following properties of the question text: Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (Grade), word count, sentence count, and average words per sentence (WpS). We then provide examples of how these properties can be combined with equality and diversity variables, and the process automated to provide ongoing monitoring. CONCLUSIONS: Given the monitoring of demographic differences in assessment for assurance of equality, the ability to easily include textual analysis of question vignettes provides a useful tool for exploring possible causes of demographic variations in performance where they occur. It also provides another means of evaluating assessment quality and fairness with respect to demographic characteristics. Microsoft Word provides data comparable to the specialized koRpus package, suggesting routine use of word processing software for writing items and assessing their properties is viable with minimal burden, but that automation for ongoing monitoring also provides an additional means of standardizing MCQ assessment items, and eliminating or controlling textual variables as a possible contributor to differential attainment between subgroups.

Twelve tips for teaching twelve transferable skills.

This article was migrated. The article was marked as recommended. There is an increasing awareness of the importance of transferable skills. With the internet now allowing almost instant access to information, the recall of factual knowledge no longer has the primacy it has held for centuries. In contrast, transferable skills remain particularly important for medical students because they receive a generic education in preparation for a diverse range of careers. The twelve tips in this article are more relevant for certain skills than others. Transferable skills should be taught in context and students should be made aware of how the skills will play a part in their future. Learning outcomes must be made clear to both staff and students, and without clear assessment of transferable skills students are likely to focus exclusively on knowledge. Students and staff should proactively engage with transferable skills training, rather than taking a purely remedial approach.

When another assessment attempt is bad for progress.

This article was migrated. The article was marked as recommended. The history and current practice of resit tests are briefly reviewed. The evidence supporting resits and also the problems associated with resits are evaluated. In addition, the financial implications of resits for both students and institutions are explored, along with the need to ensure assessments establish currency of ability and are a reflection of the long-term capability of students. Although resit outcomes are typically capped at the passing score we argue that they still afford an unfair advantage. We conclude that where resit opportunities are provided then they should have higher pass marks than first sit attempts. However, we recommend that the stress of high stakes exams, and thus resits, should be avoided. Consequently, a case is made for an alternative to resits whereby multiple lower stakes assessment results are aggregated.

Research skills in medical education.

This article was migrated. The article was marked as recommended. The ability to find, interpret and use information is a key skill in any clinician's arsenal. During medical training, we aim to equip our students with the ability to navigate the potentially baffling amount of information available online, and come to valid conclusions. This article reflects on the nature of research skills and how they are delivered in medical education. It also explores whether these are the most efficient methods for enabling students to become able researchers. Comparisons with other types of university degrees are made, and consideration given to how research skills should best be integrated into our teaching.

Angoff anchor statements: setting a flawed gold standard?

This article was migrated. The article was marked as recommended. The Angoff standard setting method depends fundamentally on the conceptualisation of an anchor statement. The precise wording and consequent interpretation of anchor statements varies in practice. Emphasis is often placed on standard setting judges' perceptions of difficulty for a candidate subgroup. The current review focusses on the meaning of anchor statements and argues that when determining the required standard of performance it is more appropriate to consider: (1) what it is important to achieve, and not how difficult it is to achieve it; (2) what all candidates should achieve, and not what a subgroup of candidates would achieve. In summary, current practice should be refined by using an anchor statement which refers to estimating the 'minimum acceptable performance by every candidate' for each item being tested, and then requiring each judge to score the relevant aspects of importance which could then be combined to derive a cut-score.

Making progression and award decisions.

This article was migrated. The article was marked as recommended. It is incumbent on medical schools to show, both to regulatory bodies and to the public at large, that their graduating students are "fit for purpose" as doctors. Since students graduate by virtue of passing assessments, it is vital that schools quality assure their assessment procedures, standards and outcomes. An important part of this quality assurance process is how progression and award decisions are made. This begins with developing clear evidence-based policies and processes that ensure assessments are effective, relevant, fair, robust and secure. Assessment is a series of processes primarily designed to enable judgements to be made as to whether a student has, or has not, met the standard required for the outcomes at that stage. This article will provide a clear rationale and guidance for establishing robust processes for making progression and award decisions.

Evaluating your professionalism.

What does being professional look like? Does it mean that you do the 'right' thing, even when no-one is looking? How do you evaluate your professionalism knowledge, values and behaviour? How do you identify and address underperformance in professionalism? How can you transfer your professionalism to different circumstances?

Accounting for test reliability in student progression: the reliable change index.

CONTEXT: Developed by Jacobson and Truax, the reliable change index (RCI) provides a measure of whether the change in an individual's score over time is within or beyond that which might be accounted for by measurement variability. In combination with measures of whether an individual's final score is closer to those of one population or another, this provides useful individual-level information that can be used to supplement traditional analyses. OBJECTIVES: This article aims to highlight the potential of the RCI for use within medical education, particularly as a novel means of monitoring progress at the student level across successive test occasions or academic years. METHODS: We provide an example of how the RCI can be applied informatively to assessment evaluation, and discuss its wider usage. CONCLUSIONS: The RCI approach can be used to identify and support failing students, as well as to determine best teaching and learning practices by identifying high-performing students. Furthermore, the individual-level nature of the RCI makes it well suited for educational research with small cohorts, as well as for tracking individual profiles within a larger cohort or addressing questions about individual performance that may be unanswerable at group level.

Modifying Hofstee standard setting for assessments that vary in difficulty, and to determine boundaries for different levels of achievement.

BACKGROUND: Fixed mark grade boundaries for non-linear assessment scales fail to account for variations in assessment difficulty. Where assessment difficulty varies more than ability of successive cohorts or the quality of the teaching, anchoring grade boundaries to median cohort performance should provide an effective method for setting standards. METHODS: This study investigated the use of a modified Hofstee (MH) method for setting unsatisfactory/satisfactory and satisfactory/excellent grade boundaries for multiple choice question-style assessments, adjusted using the cohort median to obviate the effect of subjective judgements and provision of grade quotas. RESULTS: Outcomes for the MH method were compared with formula scoring/correction for guessing (FS/CFG) for 11 assessments, indicating that there were no significant differences between MH and FS/CFG in either the effective unsatisfactory/satisfactory grade boundary or the proportion of unsatisfactory graded candidates (p > 0.05). However the boundary for excellent performance was significantly higher for MH (p < 0.01), and the proportion of candidates returned as excellent was significantly lower (p < 0.01). MH also generated performance profiles and pass marks that were not significantly different from those given by the Ebel method of criterion-referenced standard setting. CONCLUSIONS: This supports MH as an objective model for calculating variable grade boundaries, adjusted for test difficulty. Furthermore, it easily creates boundaries for unsatisfactory/satisfactory and satisfactory/excellent performance that are protected against grade inflation. It could be implemented as a stand-alone method of standard setting, or as part of the post-examination analysis of results for assessments for which pre-examination criterion-referenced standard setting is employed.

Twelve tips for assessment psychometrics.

It is incumbent on medical schools to show, both to regulatory bodies and to the public at large, that their graduating students are "fit for purpose" as tomorrow's doctors. Since students graduate by virtue of passing assessments, it is vital that schools quality assure their assessment procedures, standards, and outcomes. An important part of this quality assurance process is the appropriate use of psychometric analyses. This begins with development of an empowering, evidence-based culture in which assessment validity can be demonstrated. Preparation prior to an assessment requires the establishment of appropriate rules, test blueprinting and standard setting. When an assessment has been completed, the reporting of test results should consider reliability, assessor, demographic, and long-term analyses across multiple levels, in an integrated way to ensure the information conveyed to all stakeholders is meaningful.

Promoting equality by assessing discrimination in clinical practice.

How well do you understand your impact on the opportunities of others, and their impact on your opportunities? Do you reflect on the equality of your values and whether your behaviour could be improved? This article gives an overview of these issues for hospital doctors.

Towards a better understanding of clinical leadership.

The effectiveness of leadership is important to everyone, yet leadership is often misunderstood and represented as undesirable. There are many theoretical models of the virtues which make a good leader, but comparatively little practical advice. How well do you understand your ability to influence others? How could you improve?

Group in-course assessment promotes cooperative learning and increases performance.

The authors describe and evaluate a method to motivate medical students to maximize the effectiveness of dissection opportunities by using In-Course-Assessments (ICAs) to encourage teamwork. A student's final mark was derived by combining the group dissection mark, group mark for questions, and their individual question mark. An analysis of the impact of the ICA was performed by comparing end of module practical summative marks in student cohorts who had, or had not, participated in the ICAs. Summative marks were compared by two-way ANOVA followed by Dunnets test, or by repeated measures ANOVA, as appropriate. A cohort of medical students was selected that had experienced both practical classes without (year one) and with the new ICA structure (year two). Comparison of summative year one and year two marks illustrated an increased improvement in year two performance in this cohort. A significant increase was also noted when comparing this cohort with five preceding year two cohorts who had not experienced the ICAs (P <0.0001). To ensure that variation in the practical summative examination was not impacting on the data, a comparison was made between three cohorts who had performed the same summative examination. Results show that students who had undertook weekly ICAs showed significantly improved summative marks, compared with those who did not (P <0.0001). This approach to ICA promotes engagement with learning resources in an active, team-based, cooperative learning environment.