Factors influencing clinician-educators' assessment practice in varied Southern contexts: a health behaviour theory perspective.

In many contexts, responsibility for exit-level assessment design and implementation in undergraduate medical programmes lies with individuals who convene clinical clerkships. Their assessment practice has significant consequences for students' learning and the patients and communities that graduates will serve. Interventions to enhance assessment must involve these assessors, yet little is known about factors influencing their assessment practice. The purpose of this study was to explore factors that influence assessment practice of clerkship convenors in three varied low-and-middle income contexts in the global South. Taking assessment practice as a behaviour, Health Behaviour Theory (HBT) was deployed as a theoretical framework to explore, describe and explain assessor behaviour. Thirty-one clinician-educators responsible for designing and implementing high-stakes clerkship assessment were interviewed in South Africa and Mexico. Interacting personal and contextual factors influencing clinician-educator assessment intention and action were identified. These included attitude, influenced by impact and response appraisal, and perceived self-efficacy; along with interpersonal, physical and organisational, and distal contextual factors. Personal competencies and conducive environments supported intention to action transition. While previous research has typically explored factors in isolation, the HBT framing enabled a systematic and coherent account of assessor behaviour. These findings add a particular contextual perspective to understanding assessment practice, yet also resonate with and extend existing work that predominantly emanates from high-income contexts in the global North. These findings provide a foundation for the planning of assessment change initiatives, such as targeted, multi-factorial faculty development.

Performance assessment: Consensus statement and recommendations from the 2020 Ottawa Conference.

INTRODUCTION: In 2011 the Consensus Statement on Performance Assessment was published in Medical Teacher. That paper was commissioned by AMEE (Association for Medical Education in Europe) as part of the series of Consensus Statements following the 2010 Ottawa Conference. In 2019, it was recommended that a working group be reconvened to review and consider developments in performance assessment since the 2011 publication. METHODS: Following review of the original recommendations in the 2011 paper and shifts in the field across the past 10 years, the group identified areas of consensus and yet to be resolved issues for performance assessment. RESULTS AND DISCUSSION: This paper addresses developments in performance assessment since 2011, reiterates relevant aspects of the 2011 paper, and summarises contemporary best practice recommendations for OSCEs and WBAs, fit-for-purpose methods for performance assessment in the health professions.

Modelling the pre-assessment learning effects of assessment: evidence in the validity chain.

OBJECTIVES: We previously developed a model of the pre-assessment learning effects of consequential assessment and started to validate it. The model comprises assessment factors, mechanism factors and learning effects. The purpose of this study was to continue the validation process. For stringency, we focused on a subset of assessment factor-learning effect associations that featured least commonly in a baseline qualitative study. Our aims were to determine whether these uncommon associations were operational in a broader but similar population to that in which the model was initially derived. METHODS: A cross-sectional survey of 361 senior medical students at one medical school was undertaken using a purpose-made questionnaire based on a grounded theory and comprising pairs of written situational tests. In each pair, the manifestation of an assessment factor was varied. The frequencies at which learning effects were selected were compared for each item pair, using an adjusted alpha to assign significance. The frequencies at which mechanism factors were selected were calculated. RESULTS: There were significant differences in the learning effect selected between the two scenarios of an item pair for 13 of this subset of 21 uncommon associations, even when a p-value of < 0.00625 was considered to indicate significance. Three mechanism factors were operational in most scenarios: agency; response efficacy, and response value. CONCLUSIONS: For a subset of uncommon associations in the model, the role of most assessment factor-learning effect associations and the mechanism factors involved were supported in a broader but similar population to that in which the model was derived. Although model validation is an ongoing process, these results move the model one step closer to the stage of usefully informing interventions. Results illustrate how factors not typically included in studies of the learning effects of assessment could confound the results of interventions aimed at using assessment to influence learning.

A model of the pre-assessment learning effects of assessment is operational in an undergraduate clinical context.

BACKGROUND: No validated model exists to explain the learning effects of assessment, a problem when designing and researching assessment for learning. We recently developed a model explaining the pre-assessment learning effects of summative assessment in a theory teaching context. The challenge now is to validate this model. The purpose of this study was to explore whether the model was operational in a clinical context as a first step in this process. METHODS: Given the complexity of the model, we adopted a qualitative approach. Data from in-depth interviews with eighteen medical students were subject to content analysis. We utilised a code book developed previously using grounded theory. During analysis, we remained alert to data that might not conform to the coding framework and open to the possibility of deploying inductive coding. Ethical clearance and informed consent were obtained. RESULTS: The three components of the model i.e., assessment factors, mechanism factors and learning effects were all evident in the clinical context. Associations between these components could all be explained by the model. Interaction with preceptors was identified as a new subcomponent of assessment factors. The model could explain the interrelationships of the three facets of this subcomponent i.e., regular accountability, personal consequences and emotional valence of the learning environment, with previously described components of the model. CONCLUSIONS: The model could be utilized to analyse and explain observations in an assessment context different to that from which it was derived. In the clinical setting, the (negative) influence of preceptors on student learning was particularly prominent. In this setting, learning effects resulted not only from the high-stakes nature of summative assessment but also from personal stakes, e.g. for esteem and agency. The results suggest that to influence student learning, consequences should accrue from assessment that are immediate, concrete and substantial. The model could have utility as a planning or diagnostic tool in practice and research settings.

A model of the pre-assessment learning effects of summative assessment in medical education.

It has become axiomatic that assessment impacts powerfully on student learning. However, surprisingly little research has been published emanating from authentic higher education settings about the nature and mechanism of the pre-assessment learning effects of summative assessment. Less still emanates from health sciences education settings. This study explored the pre-assessment learning effects of summative assessment in theoretical modules by exploring the variables at play in a multifaceted assessment system and the relationships between them. Using a grounded theory strategy, in-depth interviews were conducted with individual medical students and analyzed qualitatively. Respondents' learning was influenced by task demands and system design. Assessment impacted on respondents' cognitive processing activities and metacognitive regulation activities. Individually, our findings confirm findings from other studies in disparate non-medical settings and identify some new factors at play in this setting. Taken together, findings from this study provide, for the first time, some insight into how a whole assessment system influences student learning over time in a medical education setting. The findings from this authentic and complex setting paint a nuanced picture of how intricate and multifaceted interactions between various factors in an assessment system interact to influence student learning. A model linking the sources, mechanism and consequences of the pre-assessment learning effects of summative assessment is proposed that could help enhance the use of summative assessment as a tool to augment learning.

Alienation and engagement in postgraduate training at a South African medical school.

BACKGROUND: The importance of contextual factors, such as the learning environment and sociocultural characteristics of the student, are becoming increasingly evident. Mann [2001. Alternative perspectives on the student experience: Alienation and engagement. Stud High Educ 26(1):7-19.] proposed that all learning experiences can be viewed as either alienating or engaging and Case expanded on this work. AIM: The purpose of this study was to explore perceptions of alienation or engagement as experienced by residents in anatomical pathology at one South African university. METHOD: A cross-sectional case study, with 16 semi-structured interviews was conducted. Residents were categorised as either alienated or engaged, based mainly on workplace experiences. RESULTS: Four relevant dimensions were identified; individual, home, workplace and institution. The personal attributes, strategies for coping and reasons for choosing pathology of alienated residents differed from those with engaged experiences. Poor socioeconomic background and schooling did not lead to predominantly alienating experiences, but this group still lacked some generic skills. In the workplace, two main factors resulting in alienated experiences were the interaction between residents and consultants and residents' comprehension of workplace-based learning. CONCLUSIONS: We present a simple model which may be used to identify factors that engage and alienate students in the learning experience in the workplace-based setting. Addressing these factors can contribute towards a more engaging experience for all residents.

The mechanism of impact of summative assessment on medical students' learning.

It has become axiomatic that assessment impacts powerfully on student learning, but there is a surprising dearth of research on how. This study explored the mechanism of impact of summative assessment on the process of learning of theory in higher education. Individual, in-depth interviews were conducted with medical students and analyzed qualitatively. The impact of assessment on learning was mediated through various determinants of action. Respondents' learning behaviour was influenced by: appraising the impact of assessment; appraising their learning response; their perceptions of agency; and contextual factors. This study adds to scant extant evidence and proposes a mechanism to explain this impact. It should help enhance the use of assessment as a tool to augment learning.

Initiating highly active antiretroviral therapy in sub-Saharan Africa: an assessment of the revised World Health Organization scaling-up guidelines.

OBJECTIVES: To assess the utility of the 2003 revised World Health Organization (WHO) criteria [initiating highly active antiretroviral therapy (HAART) in stage IV, in stage III plus CD4 cell count < 350 x 10(6) cells/l, or in stage I or II plus CD4 cell count < 200 x 10 cells/l] relative to other scenarios of HAART initiation. METHODS: Progression to AIDS and death in 292 patients taking HAART and 974 not taking HAART in a South African institution in 1992-2001, stratifying patients by baseline CD4 cell count and WHO stage. RESULTS: HAART was associated with decreased AIDS [adjusted rate ratio [ARR], 0.16; 95% confidence interval (CI), 0.08-0.31) and death (ARR, 0.10; 95% CI, 0.06-0.18). Benefit of HAART was significant across all WHO stages plus CD4 cell counts. The greatest number of deaths averted was in stages IV [74.0/100 patient-years (PY); 95% CI, 50.2-84.5) and III (32.8/100 PY; 95% CI, 22.4-40.9). AIDS cases averted in stage III (22.0/100 PY; 95% CI, 6.1-26.9) were higher than in stage I and II with CD4 cell count < 200 x 10(6) cells/l (8.9/100 PY 95% CI, 5.6-13.3). Treatment initiation for symptomatic disease resulted in greater benefits than using any CD4 cell thresholds. Application of WHO criteria increased the treatment-eligible proportion from 44.5% to 56.7% (P < 0.05) but did not prevent more death (P > 0.05) than treating symptomatic disease. CONCLUSION: Implementation of the revised WHO guidelines in sub-Saharan Africa may result in a significantly increased number of individuals eligible for treatment but would not be as effective a strategy for preventing death as treating symptomatic disease.