Where the rubber meets the road - An integrative review of programmatic assessment in health care professions education.

INTRODUCTION: Programmatic assessment was introduced as an approach to design assessment programmes with the aim to simultaneously optimize the decision-making and learning function of assessment. An integrative review was conducted to review and synthesize results from studies investigating programmatic assessment in health care professions education in practice. METHODS: The authors systematically searched PubMed, Web of Science, and ERIC to identify studies published since 2005 that reported empirical data on programmatic assessment. Characteristics of the included studies were extracted and synthesized, using descriptive statistics and thematic analysis. RESULTS: Twenty-seven studies were included, which used quantitative methods (n = 10), qualitative methods (n = 12) or mixed methods (n = 5). Most studies were conducted in clinical settings (77.8%). Programmatic assessment was found to enable meaningful triangulation for robust decision-making and used as a catalyst for learning. However, several problems were identified, including overload in assessment information and the associated workload, counterproductive impact of using strict requirements and summative signals, lack of a shared understanding of the nature and purpose of programmatic assessment, and lack of supportive interpersonal relationships. Thematic analysis revealed that the success and challenges of programmatic assessment were best understood by the interplay between quantity and quality of assessment information, and the influence of social and personal aspects on assessment perceptions. CONCLUSION: Although some of the evidence may seem compelling to support the effectiveness of programmatic assessment in practice, tensions will emerge when simultaneously stimulating the development of competencies and assessing its result. The identified factors and inferred strategies provide guidance for navigating these tensions.

Between trust and control: Teachers' assessment conceptualisations within programmatic assessment.

OBJECTIVES: Programmatic assessment attempts to facilitate learning through individual assessments designed to be of low-stakes and used only for high-stake decisions when aggregated. In practice, low-stake assessments have yet to reach their potential as catalysts for learning. We explored how teachers conceptualise assessments within programmatic assessment and how they engage with learners in assessment relationships. METHODS: We used a constructivist grounded theory approach to explore teachers' assessment conceptualisations and assessment relationships in the context of programmatic assessment. We conducted 23 semi-structured interviews at two different graduate-entry medical training programmes following a purposeful sampling approach. Data collection and analysis were conducted iteratively until we reached theoretical sufficiency. We identified themes using a process of constant comparison. RESULTS: Results showed that teachers conceptualise low-stake assessments in three different ways: to stimulate and facilitate learning; to prepare learners for the next step, and to use as feedback to gauge the teacher's own effectiveness. Teachers intended to engage in and preserve safe, yet professional and productive working relationships with learners to enable assessment for learning when securing high-quality performance and achievement of standards. When teachers' assessment conceptualisations were more focused on accounting conceptions, this risked creating tension in the teacher-learner assessment relationship. Teachers struggled between taking control and allowing learners' independence. CONCLUSIONS: Teachers believe programmatic assessment can have a positive impact on both teaching and student learning. However, teachers' conceptualisations of low-stake assessments are not focused solely on learning and also involve stakes for teachers. Sampling across different assessments and the introduction of progress committees were identified as important design features to support teachers and preserve the benefits of prolonged engagement in assessment relationships. These insights contribute to the design of effective implementations of programmatic assessment within the medical education context.

Understanding the influence of teacher-learner relationships on learners' assessment perception.

Low-stakes assessments are theorised to stimulate and support self-regulated learning. They are feedback-, not decision-oriented, and should hold little consequences to a learner based on their performance. The use of low-stakes assessment as a learning opportunity requires an environment in which continuous improvement is encouraged. This may be hindered by learners' perceptions of assessment as high-stakes. Teachers play a key role in learners' assessment perceptions. By investigating assessment perceptions through an interpersonal theory-based perspective of teacher-learner relationships, we aim to better understand the mechanisms explaining the relationship between assessment and learning within medical education. First, twenty-six purposefully selected learners, ranging from undergraduates to postgraduates in five different settings of programmatic assessment, were interviewed about their assessment task perception. Next, we conducted a focussed analysis using sensitising concepts from interpersonal theory to elucidate the influence of the teacher-learner relationship on learners' assessment perceptions. The study showed a strong relation between learners' perceptions of the teacher-learner relationship and their assessment task perception. Two important sources for the perception of teachers' agency emerged from the data: positional agency and expert agency. Together with teacher's communion level, both types of teachers' agency are important for understanding learners' assessment perceptions. High levels of teacher communion had a positive impact on the perception of assessment for learning, in particular in relations in which teachers' agency was less dominantly exercised. When teachers exercised these sources of agency dominantly, learners felt inferior to their teachers, which could hinder the learning opportunity. To utilise the learning potential of low-stakes assessment, teachers are required to stimulate learner agency in safe and trusting assessment relationships, while carefully considering the influence of their own agency on learners' assessment perceptions. Interpersonal theory offers a useful lens for understanding assessment relationships. The Interpersonal Circumplex provides opportunities for faculty development that help teachers develop positive and productive relationships with learners in which the potential of low-stakes assessments for self-regulated learning is realised.

Stakes in the eye of the beholder: an international study of learners' perceptions within programmatic assessment.

OBJECTIVES: Within programmatic assessment, the ambition is to simultaneously optimise the feedback and the decision-making function of assessment. In this approach, individual assessments are intended to be low stakes. In practice, however, learners often perceive assessments designed to be low stakes as high stakes. In this study, we explored how learners perceive assessment stakes within programmatic assessment and which factors influence these perceptions. METHODS: Twenty-six learners were interviewed from three different countries and five different programmes, ranging from undergraduate to postgraduate medical education. The interviews explored learners' experience with and perception of assessment stakes. An open and qualitative approach to data gathering and analyses inspired by the constructivist grounded theory approach was used to analyse the data and reveal underlying mechanisms influencing learners' perceptions. RESULTS: Learners' sense of control emerged from the analysis as key for understanding learners' perception of assessment stakes. Several design factors of the assessment programme provided or hindered learners' opportunities to exercise control over the assessment experience, mainly the opportunities to influence assessment outcomes, to collect evidence and to improve. Teacher-learner relationships that were characterised by learners' autonomy and in which learners feel safe were important for learners' believed ability to exercise control and to use assessment to support their learning. CONCLUSIONS: Knowledge of the factors that influence the perception of assessment stakes can help design effective assessment programmes in which assessment supports learning. Learners' opportunities for agency, a supportive programme structure and the role of the teacher are particularly powerful mechanisms to stimulate the learning value of programmatic assessment.

Embedding of the progress test in an assessment program designed according to the principles of programmatic assessment.

BACKGROUND: Progress tests (PT) are used to assess students on topics from all medical disciplines. Progress testing is usually one of the assessment methods of the cognitive domain. There is limited knowledge on how positioning of the PT in a program of assessment (PoA) influences students' PT scores, use of PT feedback and perceived learning value. METHODS: We compared PT total scores and use of a PT test feedback (ProF) system in two medical courses, where the PT is either used as a summative assessment or embedded in a comprehensive PoA and used formatively. In addition, an interview study was used to explore the students' perception on use of PT feedback and perceived learning value. RESULTS: PT total scores were higher, with considerable effect sizes (ESs) and students made more use of ProF when the PT was embedded in a comprehensive PoA. Analysis of feedback in the portfolio stimulated students to look for patterns in PT results, link the PT to other assessment results, follow-up on learning objectives, and integrate the PT in their learning for the entire PoA. CONCLUSIONS: Embedding the PT in an assessment program designed according to the principles of programmatic assessment positively affects PT total scores, use of PT feedback, and perceived learning value.