From aggregation to interpretation: how assessors judge complex data in a competency-based portfolio.

While portfolios are increasingly used to assess competence, the validity of such portfolio-based assessments has hitherto remained unconfirmed. The purpose of the present research is therefore to further our understanding of how assessors form judgments when interpreting the complex data included in a competency-based portfolio. Eighteen assessors appraised one of three competency-based mock portfolios while thinking aloud, before taking part in semi-structured interviews. A thematic analysis of the think-aloud protocols and interviews revealed that assessors reached judgments through a 3-phase cyclical cognitive process of acquiring, organizing, and integrating evidence. Upon conclusion of the first cycle, assessors reviewed the remaining portfolio evidence to look for confirming or disconfirming evidence. Assessors were inclined to stick to their initial judgments even when confronted with seemingly disconfirming evidence. Although assessors reached similar final (pass-fail) judgments of students' professional competence, they differed in their information-processing approaches and the reasoning behind their judgments. Differences sprung from assessors' divergent assessment beliefs, performance theories, and inferences about the student. Assessment beliefs refer to assessors' opinions about what kind of evidence gives the most valuable and trustworthy information about the student's competence, whereas assessors' performance theories concern their conceptualizations of what constitutes professional competence and competent performance. Even when using the same pieces of information, assessors furthermore differed with respect to inferences about the student as a person as well as a (future) professional. Our findings support the notion that assessors' reasoning in judgment and decision-making varies and is guided by their mental models of performance assessment, potentially impacting feedback and the credibility of decisions. Our findings also lend further credence to the assertion that portfolios should be judged by multiple assessors who should, moreover, thoroughly substantiate their judgments. Finally, it is suggested that portfolios be designed in such a way that they facilitate the selection of and navigation through the portfolio evidence.

An exploration of crossborder medical curriculum partnerships: Balancing curriculum equivalence and local adaptation.

CONTEXT: Worldwide, medical schools have entered into crossborder curriculum partnerships (CCPs) to provide equivalent curricula and learning experiences to groups of geographically separated students. Paradoxically, this process also involves adaptation of curricula to suit local contexts. This study has focused on challenges faced by medical Crossborder curriculum programme directors and strategies they employed to overcome these. METHODS: We conducted a qualitative study on six CCPs using document analysis and semi-structured interviews with 13 programme directors from 12 medical schools. Interview transcripts were coded iteratively, followed by cross-case analysis. RESULTS: The challenges faced by CCP programme directors are four-fold, springing from differences in health care systems, legislation and political interference, teaching and learning environments, and partnership. Deliberate strategies, such as intensifying interactions between partners in all academic echelons, can help to overcome these. Partnerships vary in their setup and collaboration strategy. CONCLUSION: Medical CCPs are challenging though seem feasible. Partnerships with more solid integration of academic operations appear robust in terms of ownership and provide, besides financial, also academic advantages to both institutions. However, more research is needed on the long-term effects on quality of graduates and impact on the host health care system.

The impact of programmatic assessment on student learning: theory versus practice.

CONTEXT: It is widely acknowledged that assessment can affect student learning. In recent years, attention has been called to 'programmatic assessment', which is intended to optimise both learning functions and decision functions at the programme level of assessment, rather than according to individual methods of assessment. Although the concept is attractive, little research into its intended effects on students and their learning has been conducted. OBJECTIVES: This study investigated the elements of programmatic assessment that students perceived as supporting or inhibiting learning, and the factors that influenced the active construction of their learning. METHODS: The study was conducted in a graduate-entry medical school that implemented programmatic assessment. Thus, all assessment information, feedback and reflective activities were combined into a comprehensive, holistic programme of assessment. We used a qualitative approach and interviewed students (n = 17) in the pre-clinical phase of the programme about their perceptions of programmatic assessment and learning approaches. Data were scrutinised using theory-based thematic analysis. RESULTS: Elements from the comprehensive programme of assessment, such as feedback, portfolios, assessments and assignments, were found to have both supporting and inhibiting effects on learning. These supporting and inhibiting elements influenced students' construction of learning. Findings showed that: (i) students perceived formative assessment as summative; (ii) programmatic assessment was an important trigger for learning, and (iii) the portfolio's reflective activities were appreciated for their generation of knowledge, the lessons drawn from feedback, and the opportunities for follow-up. Some students, however, were less appreciative of reflective activities. For these students, the elements perceived as inhibiting seemed to dominate the learning response. CONCLUSIONS: The active participation of learners in their own learning is possible when learning is supported by programmatic assessment. Certain features of the comprehensive programme of assessment were found to influence student learning, and this influence can either support or inhibit students' learning responses.

Training Physicians to Provide High-Value, Cost-Conscious Care: A Systematic Review.

IMPORTANCE: Increasing health care expenditures are taxing the sustainability of the health care system. Physicians should be prepared to deliver high-value, cost-conscious care. OBJECTIVE: To understand the circumstances in which the delivery of high-value, cost-conscious care is learned, with a goal of informing development of effective educational interventions. DATA SOURCES: PubMed, EMBASE, ERIC, and Cochrane databases were searched from inception until September 5, 2015, to identify learners and cost-related topics. STUDY SELECTION: Studies were included on the basis of topic relevance, implementation of intervention, evaluation of intervention, educational components in intervention, and appropriate target group. There was no restriction on study design. DATA EXTRACTION AND SYNTHESIS: Data extraction was guided by a merged and modified version of a Best Evidence in Medical Education abstraction form and a Cochrane data coding sheet. Articles were analyzed using the realist review method, a narrative review technique that focuses on understanding the underlying mechanisms in interventions. Recurrent patterns were identified in the data through thematic analyses. Resulting themes were discussed within the research team until consensus was reached. MAIN OUTCOMES AND MEASURES: Main outcomes were factors that promote education in delivering high-value, cost-conscious care. FINDINGS: The initial search identified 2650 articles; 79 met the inclusion criteria, of which 14 were randomized clinical trials. The majority of the studies were conducted in North America (78.5%) using a pre-post interventional design (58.2%; at least 1619 participants); they focused on practicing physicians (36.7%; at least 3448 participants), resident physicians (6.3%; n = 516), and medical students (15.2%; n = 275). Among the 14 randomized clinical trials, 12 addressed knowledge transmission, 7 reflective practice, and 1 supportive environment; 10 (71%) concluded that the intervention was effective. The data analysis suggested that 3 factors aid successful learning: (1) effective transmission of knowledge, related, for example, to general health economics and prices of health services, to scientific evidence regarding guidelines and the benefits and harms of health care, and to patient preferences and personal values (67 articles); (2) facilitation of reflective practice, such as providing feedback or asking reflective questions regarding decisions related to laboratory ordering or prescribing to give trainees insight into their past and current behavior (56 articles); and (3) creation of a supportive environment in which the organization of the health care system, the presence of role models of delivering high-value, cost-conscious care, and a culture of high-value, cost-conscious care reinforce the desired training goals (27 articles). CONCLUSIONS AND RELEVANCE: Research on educating physicians to deliver high-value, cost-conscious care suggests that learning by practicing physicians, resident physicians, and medical students is promoted by combining specific knowledge transmission, reflective practice, and a supportive environment. These factors should be considered when educational interventions are being developed.

Student perspectives on competency-based portfolios: Does a portfolio reflect their competence development?

INTRODUCTION: Portfolio-based assessments require that learners' competence development is adequately reflected in portfolio documentation. This study explored how students select and document performance data in their portfolios and how they perceive these data to be representative for their competence development. METHODS: Students uploaded performance data in a competency-based portfolio. During one clerkship period, twelve students also recorded an audio diary in which they reflected on experiences and feedback that they perceived to be indicants of their competence development. Afterwards, these students were interviewed to explore the extent to which the performance documentation in the portfolio corresponded with what they considered illustrative evidence of their development. The interviews were analyzed using thematic analysis. RESULTS: Portfolios provide an accurate but fragmented picture of student development. Portfolio documentation was influenced by tensions between learning and assessment, student beliefs about the goal of portfolios, student performance evaluation strategies, the learning environment and portfolio structure. DISCUSSION: This study confirms the importance of taking student perceptions into account when implementing a competency-based portfolio. Students would benefit from coaching on how to select meaningful experiences and performance data for documentation in their portfolios. Flexibility in portfolio structure and requirements is essential to ensure optimal fit between students' experienced competence development and portfolio content.