What Stops Fairness from Emerging in Assessment? The Forces on a Complex Adaptive System.

Introduction: Workplace-based assessment occurs in authentic, dynamic clinical environments where reproducible, measurement-based assessments can often not be implemented. In these environments, research approaches that respect these multiple dynamic interactions, such as complexity perspectives, are encouraged. Previous research has shown that fairness in assessment is a nonlinear phenomenon that emerges from interactions between its components and behaves like a complex adaptative system. The aim of this study was to understand the external forces on the complex adaptive system which may disrupt fairness from emerging. Methods: We conducted online focus groups with a purposeful sample of nineteen academic leaders in the Netherlands. We used an iterative approach to collection, analysis and coding of the data and interpreted the results using a lens of complexity, focusing on how individual elements of fairness work in concert to create systems with complex behaviour. Results: We identified three themes of forces which can disrupt fairness: forces impairing interactivity, forces impairing adaption and forces impairing embeddedness. Within each of these themes, we identified subthemes: assessor and student forces, tool forces and system forces. Discussion: Consistent with complexity theory, this study suggests there are multiple forces which can hamper the emergence of fairness. Whilst complexity thinking does not reduce the scale of the challenge, viewing forces through this lens provides insight into why and how these forces are disrupting fairness. This allows for more purposeful, meaningful changes to support the use of fair judgement in assessment in dynamic authentic clinical workplaces.

Fairness in Assessment: Identifying a Complex Adaptive System.

Introduction: Assessment design in health professions education is continuously evolving. There is an increasing desire to better embrace human judgement in assessment. Thus, it is essential to understand what makes this judgement fair. This study builds upon existing literature by studying how assessment leaders conceptualise the characteristics of fair judgement. Methods: Sixteen assessment leaders from 15 medical schools in Australia and New Zealand participated in online focus groups. Data collection and analysis occurred concurrently and iteratively. We used the constant comparison method to identify themes and build on an existing conceptual model of fair judgement in assessment. Results: Fairness is a multi-dimensional construct with components at environment, system and individual levels. Components influencing fairness include articulated and agreed learning outcomes relating to the needs of society, a culture which allows for learner support, stakeholder agency and learning (environmental level), collection, interpretation and combination of evidence, procedural strategies (system level) and appropriate individual assessments and assessor expertise and agility (individual level). Discussion: We observed that within the data at fractal, that is an infinite pattern repeating at different scales, could be seen suggesting fair judgement should be considered a complex adaptive system. Within complex adaptive systems, it is primarily the interaction between the entities which influences the outcome it produces, not simply the components themselves. Viewing fairness in assessment through a lens of complexity rather than as a linear, causal model has significant implications for how we design assessment programs and seek to utilise human judgement in assessment.

The pursuit of fairness in assessment: Looking beyond the objective.

Health professions education has undergone significant changes over the last few decades, including the rise of competency-based medical education, a shift to authentic workplace-based assessments, and increased emphasis on programmes of assessment. Despite these changes, there is still a commonly held assumption that objectivity always leads to and is the only way to achieve fairness in assessment. However, there are well-documented limitations to using objectivity as the 'gold standard' to which assessments are judged. Fairness, on the other hand, is a fundamental quality of assessment and a principle that almost no one contests. Taking a step back and changing perspectives to focus on fairness in assessment may help re-set a traditional objective approach and identify an equal role for subjective human judgement in assessment alongside objective methods. This paper explores fairness as a fundamental quality of assessments. This approach legitimises human judgement and shared subjectivity in assessment decisions alongside objective methods. Widening the answer to the question: 'What is fair assessment' to include not only objectivity but also expert human judgement and shared subjectivity can add significant value in ensuring learners are better equipped to be the health professionals required of the 21st century.

Making it fair: Learners' and assessors' perspectives of the attributes of fair judgement.

INTRODUCTION: Optimising the use of subjective human judgement in assessment requires understanding what makes judgement fair. Whilst fairness cannot be simplistically defined, the underpinnings of fair judgement within the literature have been previously combined to create a theoretically-constructed conceptual model. However understanding assessors' and learners' perceptions of what is fair human judgement is also necessary. The aim of this study is to explore assessors' and learners' perceptions of fair human judgement, and to compare these to the conceptual model. METHODS: A thematic analysis approach was used. A purposive sample of twelve assessors and eight post-graduate trainees undertook semi-structured interviews using vignettes. Themes were identified using the process of constant comparison. Collection, analysis and coding of the data occurred simultaneously in an iterative manner until saturation was reached. RESULTS: This study supported the literature-derived conceptual model suggesting fairness is a multi-dimensional construct with components at individual, system and environmental levels. At an individual level, contextual, longitudinally-collected evidence, which is supported by narrative, and falls within ill-defined boundaries is essential for fair judgement. Assessor agility and expertise are needed to interpret and interrogate evidence, identify boundaries and provide narrative feedback to allow for improvement. At a system level, factors such as multiple opportunities to demonstrate competence and improvement, multiple assessors to allow for different perspectives to be triangulated, and documentation are needed for fair judgement. These system features can be optimized through procedural fairness. Finally, appropriate learning and working environments which considers patient needs and learners personal circumstances are needed for fair judgments. DISCUSSION: This study builds on the theory-derived conceptual model demonstrating the components of fair judgement can be explicitly articulated whilst embracing the complexity and contextual nature of health-professions assessment. Thus it provides a narrative to support dialogue between learner, assessor and institutions about ensuring fair judgements in assessment.

Fairness in human judgement in assessment: a hermeneutic literature review and conceptual framework.

Human judgement is widely used in workplace-based assessment despite criticism that it does not meet standards of objectivity. There is an ongoing push within the literature to better embrace subjective human judgement in assessment not as a 'problem' to be corrected psychometrically but as legitimate perceptions of performance. Taking a step back and changing perspectives to focus on the fundamental underlying value of fairness in assessment may help re-set the traditional objective approach and provide a more relevant way to determine the appropriateness of subjective human judgements. Changing focus to look at what is 'fair' human judgement in assessment, rather than what is 'objective' human judgement in assessment allows for the embracing of many different perspectives, and the legitimising of human judgement in assessment. However, this requires addressing the question: what makes human judgements fair in health professions assessment? This is not a straightforward question with a single unambiguously 'correct' answer. In this hermeneutic literature review we aimed to produce a scholarly knowledge synthesis and understanding of the factors, definitions and key questions associated with fairness in human judgement in assessment and a resulting conceptual framework, with a view to informing ongoing further research. The complex construct of fair human judgement could be conceptualised through values (credibility, fitness for purpose, transparency and defensibility) which are upheld at an individual level by characteristics of fair human judgement (narrative, boundaries, expertise, agility and evidence) and at a systems level by procedures (procedural fairness, documentation, multiple opportunities, multiple assessors, validity evidence) which help translate fairness in human judgement from concepts into practical components.

Entrustable professional activities for workplace assessment of general practice trainees.

OBJECTIVE: To assess whether entrustment levels for junior trainees with respect to entrustable professional activities (EPAs) increase over time; whether entrustment levels for senior trainees are higher than for junior trainees; and whether self-assessment of entrustment levels by senior trainees more closely matches supervisor assessment than self-assessment by junior trainees. DESIGN, SETTING, PARTICIPANTS: Observational study of 130 junior and 153 senior community-based general practice trainees in South Australia, 2017. MAIN OUTCOME MEASURES: Differences in entrustment levels between junior and senior trainees; change in entrustment levels for junior trainees over 9 months; concordance of supervisor and trainee assessment of entrustment level over 9 months. RESULTS: Senior trainees were 2.1 (95% CI, 1.66-2.58) to 3.7 times (95% CI, 2.60-5.28) as likely as junior trainees to be entrusted with performing clinical EPAs without supervision. The proportion of EPAs with which junior trainees were entrusted to perform unsupervised increased from 26% at 3 months to 35% at 6 months (rate ratio [RR], 1.37; 95% CI; 1.15-1.63), to 50% at 9 months (RR, 1.92; 95% CI, 1.64-2.26), and 69% at 12 months (RR, 2.68; 95% CI; 2.32-3.12). At 3 months, the mean differences in entrustment ratings between supervisors and trainees was 5.5 points (SD, 6.6 points) for junior trainees and 2.93 points (SD, 2.8 points) for senior trainees (P < 0.001). CONCLUSIONS: EPAs are valid assessment tools in a workplace-based training environment.

Identifying the narrative used by educators in articulating judgement of performance.

INTRODUCTION: Modern assessment in medical education is increasingly reliant on human judgement, as it is clear that quantitative scales have limitations in fully assessing registrars' development of competence and providing them with meaningful feedback to assist learning. For this, possession of an expert vocabulary is essential. AIM: This study aims to explore how medical education experts voice their subjective judgements about learners and to what extent they are using clear, information-rich terminology (high-level semantic qualifiers); and to gain a better understanding of the experts' language used in these subjective judgements. METHODS: Six experienced medical educators from urban and rural environments were purposefully selected. Each educator reviewed a registrar clinical case analysis in a think out loud manner. The transcribed data were analyzed, codes were identified and ordered into themes. Analysis continued until saturation was reached. RESULTS: Five themes with subthemes emerged. The main themes were: (1) Demonstration of expertise; (2) Personal credibility; (3) Professional credibility; (4) Using a predefined structure and (5) Relevance. DISCUSSION: Analogous to what experienced clinicians do in clinical reasoning, experienced medical educators verbalize their judgements using high-level semantic qualifiers. In this study, we were able to unpack these. Although there may be individual variability in the exact words used, clear themes emerged. These findings can be used to develop a helpful shared narrative for educators in observation-based assessment. The provision of a rich, detailed narrative will also assist in providing clarity to registrar feedback with areas of weakness clearly articulated to improve learning and remediation.

An application of programmatic assessment for learning (PAL) system for general practice training.

Aim: Programmatic assessment for learning (PAL) is becoming more and more popular as a concept but its implementation is not without problems. In this paper we describe the design principles behind a PAL program in a general practice training context. Design principles: The PAL program was designed to optimise the meaningfulness of assessment information for the registrar and to make him/her use that information to self regulate their learning. The main principles in the program were cognitivist and transformative. The main cognitive principles we used were fostering the understanding of deep structures and stimulating transfer by making registrars constantly connect practice experiences with background knowledge. Ericsson's deliberate practice approach was built in with regard to the provision of feedback combined with Pintrich's model of self regulation. Mezirow's transformative learning and insights from social network theory on collaborative learning were used to support the registrars in their development to become GP professionals. Finally the principal of test enhanced learning was optimised. Epilogue: We have provided this example explain the design decisions behind our program, but not want to present our program as the solution to any given situation.

Relative Hyperglycemia, a Marker of Critical Illness: Introducing the Stress Hyperglycemia Ratio.

CONTEXT: Hyperglycemia in hospitalized patients is associated with increased morbidity and mortality. OBJECTIVE: We examined whether critical illness is more strongly associated with relative or absolute hyperglycemia. DESIGN: The study was an observational cohort study. PATIENTS AND SETTING: A total of 2290 patients acutely admitted to a tertiary hospital. MAIN OUTCOME MEASURE: The relative hyperglycemia (stress hyperglycemia ratio [SHR]) was defined as admission glucose divided by estimated average glucose derived from glycosylated hemoglobin. The relationships between glucose and SHR with critical illness (in-hospital death or critical care) were examined. RESULTS: In univariable analyses, SHR (odds ratio, 1.23 per 0.1 increment [95% confidence interval, 1.18-1.28]; P < .001) and glucose (odds ratio, 1.18 per mmol/L [1.13-1.23]; P < .001) were associated with critical illness. In multivariable analysis, the association was maintained for SHR (odds ratio, 1.20 per 0.1 increment [1.13-1.28]; P < .001), but not glucose (odds ratio, 1.03 per mmol/L [0.97-1.11]; P = .31). Background hyperglycemia affected the relationship between glucose (P = .002) and critical illness, but not SHR (P = .35) and critical illness. In patients with admission glucose ≤ 10 mmol/L, the odds ratio for critical illness was higher in the fourth (2.4 [1.4-4.2]; P = .001) and fifth (3.9 [2.3-6.8]; P < .001) SHR quintiles than in the lowest SHR quintile. CONCLUSIONS: SHR controls for background glycemia and is a better biomarker of critical illness than absolute hyperglycemia. SHR identifies patients with relative hyperglycemia at risk of critical illness. Future studies should explore whether basing glucose-lowering therapy on relative, rather than absolute, hyperglycemia improves outcomes in hospitalized patients.

Clinical exposure of interns during a general practice placement.

BACKGROUND: For more than a decade, junior doctors have undertaken general practice rotations; however, little is known about the breadth of medical conditions seen. This study aims to determine the breadth of clinical presentations encountered by interns during a rotation. METHODS: Data were collected on all patients seen by interns at an RA-2 general practice during 2012-13. Each condition identified was compared with the Australian Curriculum Framework for Junior Doctors (ACFJD) and coded according to the International Classification of Diseases 10th revision (ICD-10). RESULTS: Interns saw an average of 482.2 (SD = 38) patients per rotation (10.7 patients per day), 150.3 (SD = 15.4) ICD-10 codes per rotation and 49.8 (SD = 4.9; 57.2%) of the listed clinical symptoms and conditions in the ACFJD.

Adenosine-diphosphate (ADP) receptor antagonists for the prevention of cardiovascular disease in type 2 diabetes mellitus.

BACKGROUND: Cardiovascular disease (CVD) is the most prevalent complication of type 2 diabetes with an estimated 65% of people with type 2 diabetes dying from a cause related to atherosclerosis. Adenosine-diphosphate (ADP) receptor antagonists like clopidogrel, ticlopidine, prasugrel and ticagrelor impair platelet aggregation and fibrinogen-mediated platelet cross-linking and may be effective in preventing CVD. OBJECTIVES: To assess the effects of adenosine-diphosphate (ADP) receptor antagonists for the prevention of cardiovascular disease in type 2 diabetes mellitus. SEARCH METHODS: We searched the Cochrane Central Register of Controlled Trials (CENTRAL) in The Cochrane Library (issue 2, 2011), MEDLINE (until April 2011) and EMBASE (until May 2011). We also performed a manual search, checking references of original articles and pertinent reviews to identify additional studies. SELECTION CRITERIA: Randomised controlled trials comparing an ADP receptor antagonist with another antiplatelet agent or placebo for a minimum of 12 months in patients with diabetes. In particular, we looked for trials assessing clinical cardiovascular outcomes. DATA COLLECTION AND ANALYSIS: Two review authors extracted data for studies which fulfilled the inclusion criteria, using standard data extraction templates. We sought additional unpublished information and data from the principal investigators of all included studies. MAIN RESULTS: Eight studies with a total of 21,379 patients with diabetes were included. Three included studies investigated ticlopidine compared to aspirin or placebo. Five included studies investigated clopidogrel compared to aspirin or a combination of aspirin and dipyridamole, or compared clopidogrel in combination with aspirin to aspirin alone. All trials included patients with previous CVD except the CHARISMA trial which included patients with multiple risk factors for coronary artery disease. Overall the risk of bias of the trials was low. The mean duration of follow-up ranged from 365 days to 913 days.Data for diabetes patients on all-cause mortality, vascular mortality and myocardial infarction were only available for one trial (355 patients). This trial compared ticlopidine to placebo and did not demonstrate any statistically significant differences for all-cause mortality, vascular mortality or myocardial infarction. Diabetes outcome data for stroke were available in three trials (31% of total diabetes participants). Overall pooling of two (statistically heterogeneous) studies showed no statistically significant reduction in the combination of fatal and non-fatal stroke (359/3194 (11.2%) versus 356/3146 (11.3%), random effects odds ratio (OR) 0.81; 95% confidence interval (CI) 0.44 to 1.49) for ADP receptor antagonists versus other antiplatelet drugs. There were no data available from any of the trials on peripheral vascular disease, health-related quality of life, adverse events specifically for patients with diabetes, or costs. AUTHORS' CONCLUSIONS: The available evidence for ADP receptor antagonists in patients with diabetes mellitus is limited and most trials do not report outcomes for patients with diabetes separately. Therefore, recommendations for the use of ADP receptor antagonists for the prevention of CVD in patients with diabetes are based on available evidence from trials including patients with and without diabetes. Trials with diabetes patients and subgroup analyses of patients with diabetes in trials with combined populations are needed to provide a more robust evidence base to guide clinical management in patients with diabetes.

Detecting undiagnosed diabetes using glycated haemoglobin: an automated screening test in hospitalised patients.

OBJECTIVE: To assess the utility of glycated haemoglobin (HbA(1c)) level as an automated screening test for undiagnosed diabetes among hospitalised patients and to estimate the prevalence of undiagnosed diabetes among hospitalised patients. DESIGN, PARTICIPANTS AND SETTING: A 3-month prospective study of all adult patients admitted to a tertiary hospital. An HbA(1c) test was automatically undertaken on admission for all patients with a random plasma glucose (RPG) level ≥ 5.5 mmol/L. Demographic, admission and biochemical data were obtained from hospital databases. A subset of patients was recruited for an oral glucose tolerance test (OGTT) after discharge. MAIN OUTCOME MEASURES: Prevalence of undiagnosed diabetes (defined as HbA(1c) ≥ 6.5% in accordance with International Expert Committee and American Diabetes Association recommendations) and utility of automated HbA(1c) testing. RESULTS: The prevalence of undiagnosed diabetes was 11% (95% CI, 9.8%-12.4%) (262/2360) during the study period. A further 312 patients with known diabetes were admitted. The prevalence of undiagnosed diabetes was highest in the 65-74-years age group. The HbA(1c) test cost was $152 per new diagnosis of diabetes. Conservatively assuming an annual incidence of undiagnosed diabetes of 0.8%, the ongoing cost of testing hospitalised patients would be $2100 per new diagnosis of diabetes. RPG testing was not sensitive or specific in diagnosing diabetes. Patients were poorly compliant with the post-discharge OGTT (27% completion rate). CONCLUSIONS: HbA(1c) is a simple, inexpensive screening test that can be automated using existing clinical blood samples. Hospital screening for diabetes needs to be coupled with resources for management in the community.