Theoretical considerations on programmatic assessment.

Introduction: Programmatic assessment (PA) is an approach to assessment aimed at optimizing learning which continues to gain educational momentum. However, the theoretical underpinnings of PA have not been clearly described. An explanation of the theoretical underpinnings of PA will allow educators to gain a better understanding of this approach and, perhaps, facilitate its use and effective implementation. The purpose of this article is twofold: first, to describe salient theoretical perspectives on PA; second to examine how theory may help educators to develop effective PA programs, helping to overcome challenges around PA.Results: We outline a number of learning theories that underpin key educational principles of PA: constructivist and social constructivist theory supporting meaning making, and longitudinality; cognitivist and cognitive development orientation scaffolding the practice of a continuous feedback process; theory of instructional design underpinning assessment as learning; self-determination theory (SDT), self-regulation learning theory (SRL), and principles of deliberate practice providing theoretical tenets for student agency and accountability.Conclusion: The construction of a plausible and coherent link between key educational principles of PA and learning theories should enable educators to pose new and important inquiries, reflect on their assessment practices and help overcome future challenges in the development and implementation of PA in their programs.

Twelve Tips for programmatic assessment.

Programmatic assessment is an integral approach to the design of an assessment program with the intent to optimise its learning function, its decision-making function and its curriculum quality-assurance function. Individual methods of assessment, purposefully chosen for their alignment with the curriculum outcomes and their information value for the learner, the teacher and the organisation, are seen as individual data points. The information value of these individual data points is maximised by giving feedback to the learner. There is a decoupling of assessment moment and decision moment. Intermediate and high-stakes decisions are based on multiple data points after a meaningful aggregation of information and supported by rigorous organisational procedures to ensure their dependability. Self-regulation of learning, through analysis of the assessment information and the attainment of the ensuing learning goals, is scaffolded by a mentoring system. Programmatic assessment-for-learning can be applied to any part of the training continuum, provided that the underlying learning conception is constructivist. This paper provides concrete recommendations for implementation of programmatic assessment.

Understanding the effects of time on collaborative learning processes in problem based learning: a mixed methods study.

Little is known how time influences collaborative learning groups in medical education. Therefore a thorough exploration of the development of learning processes over time was undertaken in an undergraduate PBL curriculum over 18 months. A mixed-methods triangulation design was used. First, the quantitative study measured how various learning processes developed within and over three periods in the first 1,5 study years of an undergraduate curriculum. Next, a qualitative study using semi-structured individual interviews focused on detailed development of group processes driving collaborative learning during one period in seven tutorial groups. The hierarchic multilevel analyses of the quantitative data showed that a varying combination of group processes developed within and over the three observed periods. The qualitative study illustrated development in psychological safety, interdependence, potency, group learning behaviour, social and task cohesion. Two new processes emerged: 'transactive memory' and 'convergence in mental models'. The results indicate that groups are dynamic social systems with numerous contextual influences. Future research should thus include time as an important influence on collaborative learning. Practical implications are discussed.

Workplace-based assessment: raters' performance theories and constructs.

Weaknesses in the nature of rater judgments are generally considered to compromise the utility of workplace-based assessment (WBA). In order to gain insight into the underpinnings of rater behaviours, we investigated how raters form impressions of and make judgments on trainee performance. Using theoretical frameworks of social cognition and person perception, we explored raters' implicit performance theories, use of task-specific performance schemas and the formation of person schemas during WBA. We used think-aloud procedures and verbal protocol analysis to investigate schema-based processing by experienced (N = 18) and inexperienced (N = 16) raters (supervisor-raters in general practice residency training). Qualitative data analysis was used to explore schema content and usage. We quantitatively assessed rater idiosyncrasy in the use of performance schemas and we investigated effects of rater expertise on the use of (task-specific) performance schemas. Raters used different schemas in judging trainee performance. We developed a normative performance theory comprising seventeen inter-related performance dimensions. Levels of rater idiosyncrasy were substantial and unrelated to rater expertise. Experienced raters made significantly more use of task-specific performance schemas compared to inexperienced raters, suggesting more differentiated performance schemas in experienced raters. Most raters started to develop person schemas the moment they began to observe trainee performance. The findings further our understanding of processes underpinning judgment and decision making in WBA. Raters make and justify judgments based on personal theories and performance constructs. Raters' information processing seems to be affected by differences in rater expertise. The results of this study can help to improve rater training, the design of assessment instruments and decision making in WBA.

Visualising the invisible: a network approach to reveal the informal social side of student learning.

World-wide, universities in health sciences have transformed their curriculum to include collaborative learning and facilitate the students' learning process. Interaction has been acknowledged to be the synergistic element in this learning context. However, students spend the majority of their time outside their classroom and interaction does not stop outside the classroom. Therefore we studied how informal social interaction influences student learning. Moreover, to explore what really matters in the students learning process, a model was tested how the generally known important constructs-prior performance, motivation and social integration-relate to informal social interaction and student learning. 301 undergraduate medical students participated in this cross-sectional quantitative study. Informal social interaction was assessed using self-reported surveys following the network approach. Students' individual motivation, social integration and prior performance were assessed by the Academic Motivation Scale, the College Adaption Questionnaire and students' GPA respectively. A factual knowledge test represented student' learning. All social networks were positively associated with student learning significantly: friendships (ß = 0.11), providing information to other students (ß = 0.16), receiving information from other students (ß = 0.25). Structural equation modelling revealed a model in which social networks increased student learning (r = 0.43), followed by prior performance (r = 0.31). In contrast to prior literature, students' academic motivation and social integration were not associated with students' learning. Students' informal social interaction is strongly associated with students' learning. These findings underline the need to change our focus from the formal context (classroom) to the informal context to optimize student learning and deliver modern medics.

A model for programmatic assessment fit for purpose.

We propose a model for programmatic assessment in action, which simultaneously optimises assessment for learning and assessment for decision making about learner progress. This model is based on a set of assessment principles that are interpreted from empirical research. It specifies cycles of training, assessment and learner support activities that are complemented by intermediate and final moments of evaluation on aggregated assessment data points. A key principle is that individual data points are maximised for learning and feedback value, whereas high-stake decisions are based on the aggregation of many data points. Expert judgement plays an important role in the programme. Fundamental is the notion of sampling and bias reduction to deal with the inevitable subjectivity of this type of judgement. Bias reduction is further sought in procedural assessment strategies derived from criteria for qualitative research. We discuss a number of challenges and opportunities around the proposed model. One of its prime virtues is that it enables assessment to move, beyond the dominant psychometric discourse with its focus on individual instruments, towards a systems approach to assessment design underpinned by empirically grounded theory.

Workplace-based assessment: effects of rater expertise.

Traditional psychometric approaches towards assessment tend to focus exclusively on quantitative properties of assessment outcomes. This may limit more meaningful educational approaches towards workplace-based assessment (WBA). Cognition-based models of WBA argue that assessment outcomes are determined by cognitive processes by raters which are very similar to reasoning, judgment and decision making in professional domains such as medicine. The present study explores cognitive processes that underlie judgment and decision making by raters when observing performance in the clinical workplace. It specifically focuses on how differences in rating experience influence information processing by raters. Verbal protocol analysis was used to investigate how experienced and non-experienced raters select and use observational data to arrive at judgments and decisions about trainees' performance in the clinical workplace. Differences between experienced and non-experienced raters were assessed with respect to time spent on information analysis and representation of trainee performance; performance scores; and information processing--using qualitative-based quantitative analysis of verbal data. Results showed expert-novice differences in time needed for representation of trainee performance, depending on complexity of the rating task. Experts paid more attention to situation-specific cues in the assessment context and they generated (significantly) more interpretations and fewer literal descriptions of observed behaviors. There were no significant differences in rating scores. Overall, our findings seemed to be consistent with other findings on expertise research, supporting theories underlying cognition-based models of assessment in the clinical workplace. Implications for WBA are discussed.

Measuring the development of insight by dental health professionals in training using workplace-based assessment.

INTRODUCTION: For health professionals, the development of insight into their performance is vital for safe practice, professional development and self-regulation. This study investigates whether the development of dental trainees' insight, when provided with external feedback on performance, can be assessed using a single criterion on a simple global ratings form such as the Longitudinal Evaluation of Performance or Mini Clinical Evaluation Exercise. METHODS: Postgraduate dental trainees (N = 139) were assessed using this tool on a weekly basis for 6 months. Regression analysis of the data was carried out using SPSS, and a short trainer questionnaire was implemented to investigate feasibility. RESULTS: Ratings for insight were shown to increase with time in a similar manner to the growth observed in other essential skills. The gradient of the slope for growth of insight was slightly less than that of the other observed skills. Trainers were mostly positive about the new criterion assessing trainees' insight, although the importance of training for trainers in this process was highlighted. DISCUSSION: Our data suggest that practitioners' insight into their performance can be developed with experience and regular feedback. However, this is most likely a complex skill dependent on a number of intrinsic and external factors. CONCLUSION: The development of trainees' insight into their performance can be assessed using a single criterion on a simple global ratings form. The process involves no additional burden on evaluators in terms of their time or cost, and promotes best practice in the provision of feedback for trainees.

A new framework for designing programmes of assessment.

Research on assessment in medical education has strongly focused on individual measurement instruments and their psychometric quality. Without detracting from the value of this research, such an approach is not sufficient to high quality assessment of competence as a whole. A programmatic approach is advocated which presupposes criteria for designing comprehensive assessment programmes and for assuring their quality. The paucity of research with relevance to programmatic assessment, and especially its development, prompted us to embark on a research project to develop design principles for programmes of assessment. We conducted focus group interviews to explore the experiences and views of nine assessment experts concerning good practices and new ideas about theoretical and practical issues in programmes of assessment. The discussion was analysed, mapping all aspects relevant for design onto a framework, which was iteratively adjusted to fit the data until saturation was reached. The overarching framework for designing programmes of assessment consists of six assessment programme dimensions: Goals, Programme in Action, Support, Documenting, Improving and Accounting. The model described in this paper can help to frame programmes of assessment; it not only provides a common language, but also a comprehensive picture of the dimensions to be covered when formulating design principles. It helps identifying areas concerning assessment in which ample research and development has been done. But, more importantly, it also helps to detect underserved areas. A guiding principle in design of assessment programmes is fitness for purpose. High quality assessment can only be defined in terms of its goals.

Progress testing in postgraduate medical education.

BACKGROUND: The role of knowledge in postgraduate medical education has often been discussed. However, recent insights from cognitive psychology and the study of deliberate practice recognize that expert problem solving requires a well-organized knowledge database. This implies that postgraduate assessment should include knowledge testing. Longitudinal assessment, like progress testing, seems a promising approach for postgraduate progress knowledge assessment. AIMS: To evaluate the validity and reliability of a national progress test in postgraduate Obstetrics and Gynaecology training. METHODS: Data of 10 years of postgraduate progress testing were analyzed on reliability with Cronbach's alpha and on construct validity using one-way ANOVA with a post hoc Scheffe test. RESULTS: Average reliability with true-false questions was 0.50, which is moderate at best. After the introduction of multiple-choice questions average reliability improved to 0.65. Construct validity or discriminative power could only be demonstrated with some certainty between training year 1 and training year 2 and higher training years. CONCLUSION: Validity and reliability of the current progress test in postgraduate Obstetrics and Gynaecology training is unsatisfactory. Suggestions for improvement of both test construct and test content are provided.

Building multidisciplinary health workforce capacity to support the implementation of integrated, people-centred Models of Care for musculoskeletal health.

To address the burden of musculoskeletal (MSK) conditions, a competent health workforce is required to support the implementation of MSK models of care. Funding is required to create employment positions with resources for service delivery and training a fit-for-purpose workforce. Training should be aligned to define "entrustable professional activities", and include collaborative skills appropriate to integrated and people-centred care and supported by shared education resources. Greater emphasis on educating MSK healthcare workers as effective trainers of peers, students and patients is required. For quality, efficiency and sustainability of service delivery, education and research capabilities must be integrated across disciplines and within the workforce, with funding models developed based on measured performance indicators from all three domains. Greater awareness of the societal and economic burden of MSK conditions is required to ensure that solutions are prioritised and integrated within healthcare policies from local to regional to international levels. These healthcare policies require consumer engagement and alignment to social, economic, educational and infrastructure policies to optimise effectiveness and efficiency of implementation.

[Assessment of medical competence in clinical education]. / Competentiegerichte toetsing in het klinisch onderwijs.

There has been considerable change in the field of assessment of medical competence. At the moment, competency-orientated assessment, 'mini-CEX' (brief clinical evaluation exercises) and portfolios are quite popular. These methods are based on research findings indicating that medical competence can better be described as a collection of the complex tasks (so-called competencies) that a doctor must be able to perform than as the sum of knowledge, skills, problem-solving ability and attitudes. Mini-CEX represents a method for the assessment of medical competence reliably and validly in a practical setting. Using a portfolio, information on the student's competence can be collated and evaluated from various sources, including mini-CEX. As such, a portfolio has much in common with a patient chart.

An Inter- and Intra-University Comparison With Short Case-Based Testing.

Comparisons between PBL and non-PBL medical schools on problem-solving ability often show no differences. This could be either due to the fact that no difference in problem-solving skills exists or that the instruments used are inadequate. In this study a key-feature approach case-based examination was used to compare two medical schools in the Netherlands, one of which has a PBL curriculum (Maastricht) and one which has a program half way a transition from a non-PBL towards a PBL curriculum (Groningen). Differences were found both in proficiency scores and in the pattern of response times, both supporting the assumption that a PBL approach would lead to a higher level of problem solving ability. The effect size, however, is not as large as originally assumed by the PBL proponents. Conclusions must be drawn with caution, but it seems likely that a test based on large numbers of short cases is the most sensitive in detecting differences in problem solving ability between students of different curricula.

Intensive care medicine trainees' perception of professionalism: a qualitative study.

The Competency-Based Training program in Intensive Care Medicine in Europe identified 12 competency domains. Professionalism was given a prominence equal to technical ability. However, little information pertaining to fellows' views on professionalism is available. A nationwide qualitative study was performed. The moderator asked participants to clarify the terms professionalism and professional behaviour, and to explore the questions "How do you learn the mentioned aspects?" and "What ways of learning do you find useful or superfluous?". Qualitative data analysis software (MAXQDA2007) facilitated analysis using an inductive coding approach. Thirty-five fellows across eight groups participated. The themes most frequently addressed were communication, keeping distance and boundaries, medical knowledge and expertise, respect, teamwork, leadership and organisation and management. Medical knowledge, expertise and technical skills seem to become more tacit when training progresses. Topics can be categorised into themes of workplace-based learning, by gathering practical experience, by following examples and receiving feedback on action, including learning from own and others' mistakes. Formal teaching courses (e.g. communication) and scheduled sessions addressing professionalism aspects were also valued. The emerging themes considered most relevant for intensivists were adequate communication skills and keeping boundaries with patients and relatives. Professionalism is mainly learned 'on the job' from role models in the intensive care unit. Formal teaching courses and sessions addressing professionalism aspects were nevertheless valued, and learning from own and others' mistakes was considered especially useful. Self-reflection as a starting point for learning professionalism was stressed.

Development of an e-learning portal for pediatric endocrinology: educational considerations.

BACKGROUND: Global accessibility and dissemination of developments in pediatric endocrinology prompted to examine how to develop an educational interactive e-SPE web portal. METHODS: A systematic approach was used to identify the relevant aspects of accessibility and dissemination. An orientation at the big idea was made, executed by an analysis of the needs of student and teacher pediatric endocrinologists, a definition of the learning objectives, a research in educational literature and an exploration of ICT design specifications. RESULTS: The intensive collaboration between medical, educational and information technology disciplines resulted in a portal design. The portal meets requirements of adult learning, stresses interaction between partners in learning and offers direct feedback during the learning process. The portal supports the development of not only knowledge but also competences both at junior and advanced levels. CONCLUSION: When the e-SPE portal is completed, the options for summative assessment will be examined as a medium for international certification in conjunction with local and national requirements (http://espe.elearning.nl).

The assessment of professional competence: building blocks for theory development.

This article presents lessons learnt from experiences with assessment of professional competence. Based on Miller's pyramid, a distinction is made between established assessment technology for assessing 'knows', 'knowing how' and 'showing how' and more recent developments in the assessment of (clinical) performance at the 'does' level. Some general lessons are derived from research of and experiences with the established assessment technology. Here, many paradoxes are revealed and empirical outcomes are often counterintuitive. Instruments for assessing the 'does' level are classified and described, and additional general lessons for this area of performance assessment are derived. These lessons can also be read as general principles of assessment (programmes) and may provide theoretical building blocks to underpin appropriate and state-of-the-art assessment practices.

Is an Angoff standard an indication of minimal competence of examinees or of judges?

BACKGROUND: To establish credible, defensible and acceptable passing scores for written tests is a challenge for health profession educators. Angoff procedures are often used to establish pass/fail decisions for written and performance tests. In an Angoff procedure judges' expertise and professional skills are assumed to influence their ratings of the items during standard-setting. The purpose of this study was to investigate the impact of judges' item-related knowledge on their judgement of the difficulty of items, and second, to determine the stability of differences between judges. METHOD: Thirteen judges were presented with two sets of 60 items on different occasions. They were asked to not only judge the difficulty of the items but also to answer them, without the benefit of the answer key. For each of the 120 items an Angoff estimate and an item score were obtained. The relationship between the Angoff estimate and the item score was examined by applying a regression analysis to the 60 items (Angoff estimate, score) for each judge at each occasion. RESULTS AND CONCLUSIONS: This study shows that in standard-setting the individual judgement of the individual item is not only a reflection of the difficulty of the item but also of the inherent stringency of the judge and his/her subject-related knowledge. Considerable variation between judges in their stringency was found, and Angoff estimates were significantly affected by a judge knowing or not knowing the answer to the item. These findings stress the importance of a careful selection process of the Angoff judges when making pass/fail decisions in health professions education. They imply that judges should be selected who are not only capable of conceptualising the 'minimally competent student', but who would also be capable of answering all the items.

The acquisition of tacit knowledge in medical education: learning by doing.

AIM: This study was designed to assess medical school teachers' tacit knowledge of basic pedagogic principles and to explore the specific character of the knowledge base. METHODS: We developed a 50-item, multiple-choice question test based on important pedagogic principles, and classified all questions as requiring either declarative or procedural knowledge. A total of 72 medical teachers representing 5 different groups of clinicians and educators agreed to sit the test. RESULTS: Teachers in all 5 groups performed well on the test of tacit pedagogic knowledge but those with advanced education degrees, or local recognition as experts, performed best. All test takers performed best on questions requiring procedural knowledge. CONCLUSION: Medical teachers possess tacit knowledge of basic pedagogic principles. Superior test performance on questions requiring procedural knowledge is consistent with their working in a clinical environment characterised by repeated procedural activities.