Challenges in engaging patients in feedback conversations for health care professionals' workplace learning.

INTRODUCTION: Patient feedback is relevant information for improvement of health care professionals' performance. Engaging patients in feedback conversations can help to harness patient feedback as a powerful tool for learning. However, health care settings may prevent patients and health care professionals to effectively engage in a feedback dialogue. To advance our understanding of how feedback conversations may support learning in and from practice, we sought to explore patients' and health care providers' perspectives on engaging patients in feedback conversations as informal learning opportunities. METHODS: For this qualitative study, we used a pragmatic approach and conducted semi-structured interviews with 12 health care providers and 10 patient consultants. We applied an inductive approach to thematic analysis to understand interviewees' perceptions regarding patient feedback for workplace learning. RESULTS: Participants attributed importance to patient feedback and described how the feedback may improve treatment relationships, professionals' performance and care processes on the team level and the organisational level. Participants experienced conflicting roles as patient and educator or expert and learner, respectively. Changing relationships, feelings of vulnerability and perceived power dynamics in treatment relationships would affect participants' engagement in feedback conversations. Patients and professionals alike saw a role for themselves in giving or inviting feedback but often missed the tools for engaging in feedback conversations. DISCUSSION: Patient feedback can contribute to professionals' practice-based learning but requires navigating tensions around conflicting roles and power dynamics in the treatment relationship. Both patients and health care professionals need to embrace vulnerability and may need facilitation and guidance to use patient feedback effectively. Attention to power dynamics, if not a shift towards collaborative relationships, is however crucial to engage patients in feedback conversations, thereby capitalising the power patients posses.

Putting self-regulated learning in context: Integrating self-, co-, and socially shared regulation of learning.

Processes involved in the regulation of learning have been researched for decades, because of its impact on academic and workplace performance. In fact, self-regulated learning is the focus of countless studies in health professions education and higher education in general. While we will always need competent individuals who are able to regulate their own learning, developments in healthcare require a shift from a focus on the individual to the collective: collaboration within and between healthcare teams is at the heart of high-quality patient care. Concepts of collaborative learning and collective competence challenge commonly held conceptualisations of regulatory learning and call for a focus on the social embeddedness of regulatory learning and processes regulating the learning of the collective. Therefore, this article questions the alignment of current conceptualisations of regulation of learning with demands for collaboration in current healthcare. We explore different conceptualisations of regulation of learning (self-, co-, and socially shared regulation of learning), and elaborate on how the integration of these conceptualisations adds to our understanding of regulatory learning in healthcare settings. Building on these insights, we furthermore suggest ways forward for research and educational practice.

How medical students co-regulate their learning in clinical clerkships: a social network study.

BACKGROUND: Self-regulated learning is a key competence to engage in lifelong learning. Research increasingly acknowledges that medical students in clerkships need others to regulate their learning. The concept of "co-regulated learning" captures this act of regulating one's learning by interacting with others. To effectively cultivate such skills in students, we need to increase our understanding of co-regulated learning. This study aimed to identify the purposes for which students in different phases of clinical training engage others in their networks to regulate their learning. METHODS: In this social network study, we administered a questionnaire to 403 medical students during clinical clerkships (65.5% response rate). The questionnaire probed into the composition of students' co-regulatory networks and the purpose for which they engaged others in specified self-regulated learning activities. We calculated the proportion of students that engaged others in their networks for each regulatory activity. Additionally, we conducted ANOVAs to examine whether first-, second-, and third-year students differed in how they used their networks to support self-regulation. RESULTS: Students used others within their co-regulatory networks to support a range of self-regulated learning activities. Whom students engaged, and the purpose of engagement, seemed to shift as students progressed through clinical training. Over time, the proportion of students engaging workplace supervisors to discuss learning goals, learning strategies, self-reflections and self-evaluations increased, whereas the proportion of students engaging peers to discuss learning strategies and how to work on learning goals in the workplace decreased. Of all purposes for which students engaged others measured, discussing self-reflections and self-evaluations were consistently among the ones most frequently mentioned. CONCLUSIONS: Results reinforce the notion that medical students' regulation of learning is grounded in social interactions within co-regulatory networks students construct during clerkships. Findings elucidate the extent to which students enact self-regulatory learning within their co-regulatory networks and how their co-regulatory learning behaviors develop over time. Explicating the relevance of interactions within co-regulatory networks might help students and supervisors to purposefully engage in meaningful co-regulatory interactions. Additionally, co-regulatory interactions may assist students in regulating their learning in clinical workplaces as well as in honing their self-regulated learning skills.

Supervisory dyads' communication and alignment regarding the use of workplace-based observations: a qualitative study in general practice residency.

BACKGROUND: In medical residency, performance observations are considered an important strategy to monitor competence development, provide feedback and warrant patient safety. The aim of this study was to gain insight into whether and how supervisor-resident dyads build a working repertoire regarding the use of observations, and how they discuss and align goals and approaches to observation in particular. METHODS: We used a qualitative, social constructivist approach to explore if and how supervisory dyads work towards alignment of goals and preferred approaches to performance observations. We conducted semi-structured interviews with supervisor-resident dyads, performing a template analysis of the data thus obtained. RESULTS: The supervisory dyads did not frequently communicate about the use of observations, except at the start of training and unless they were triggered by internal or external factors. Their working repertoire regarding the use of observations seemed to be primarily driven by patient safety goals and institutional assessment requirements rather than by providing developmental feedback. Although intended as formative, the institutional test was perceived as summative by supervisors and residents, and led to teaching to the test rather than educating for purposes of competence development. CONCLUSIONS: To unlock the full educational potential of performance observations, and to foster the development of an educational alliance, it is essential that supervisory dyads and the training institute communicate clearly about these observations and the role of assessment practices of- and for learning, in order to align their goals and respective approaches.

Beyond the self: The role of co-regulation in medical students' self-regulated learning.

CONTEXT: Medical students are expected to self-regulate their learning within complex and unpredictable clinical learning environments. Research increasingly focuses on the effects of social interactions on the development of self-regulation in workplace settings, a notion embodied within the concept of co-regulated learning (CRL). Creating workplace learning environments that effectively foster lifelong self-regulated learning (SRL) requires a deeper understanding of the relationship between CRL and SRL. The aim of this study was therefore to explore medical students' perceptions of CRL in clinical clerkships and its perceived impact on the development of their SRL. METHODS: We conducted semi-structured interviews with 11 purposively sampled medical students enrolled in clinical clerkships at one undergraduate competency-based medical school. Data collection and analysis were conducted iteratively, informed by principles of constructivist grounded theory. Data analysis followed stages of open, axial and selective coding, which enabled us to conceptualise how co-regulation influences the development of students' self-regulation. RESULTS: Data revealed three interrelated shifts in CRL and SRL as students progressed through clerkships. First, students' CRL shifted from a focus on peers to co-regulation with clinician role models. Second, self-regulated behaviour shifted from being externally driven to being internally driven. Last, self-regulation shifted from a task-oriented approach towards a more comprehensive approach focusing on professional competence and identity formation. Students indicated that if they felt able to confidently and proactively self-regulate their learning, the threshold for engaging others in meaningful CRL seemed to be lowered, enhancing further development of SRL skills. CONCLUSIONS: Findings from the current study emphasise the notion that SRL and its development are grounded in CRL in clinical settings. To optimally support the development of students' SRL, we need to focus on facilitating and organising learners' engagement in CRL from the start of the medical curriculum.

Learning in the workplace: Use of informal feedback cues in doctor-patient communication.

OBJECTIVES: We expect physicians to be lifelong learners. Participation in clinical practice is an important potential source of that learning. To support physicians in this process, a better understanding of how they learn in clinical practice is necessary. This study investigates how physicians recognise and use informal feedback from interactions with patients in outpatient settings as learning cues to adjust their communication behaviours in daily practice. METHODS: To understand physicians' use of informal feedback, we combined non-participant observations with semi-structured interviews. We enrolled 10 respiratory physicians and observed 100 physician-patient interactions at two teaching hospitals in the Netherlands. Data collection and analysis were performed iteratively according to the principles of constructivist grounded theory. RESULTS: Following stages of open, axial and selective coding, we were able to conceptualise how physicians use cues to reflect on and adjust their communication. In addition to vast variations within and across patient encounters, we observed recurring adjustments in physicians' communication behaviours in response to specific informal feedback cues. Physicians recognised and used these cues to self-monitor communication performance. They had established 'communication repertoires' based on multiple patient interactions, which many saw as learning opportunities contributing to the development of expertise. Our findings, however, show differences in physicians' individual levels of sensitivity in recognising and using learning opportunities in daily practice, which were further influenced by contextual, personal and interpersonal factors. Whereas some described themselves as having little inclination to change, others used critical incidents to fine-tune their communication repertoires, and yet others constantly reshaped them, seeking learning opportunities in their daily work. CONCLUSIONS: There is large variation in how physicians use learning cues from daily practice. To enhance learning in and from daily practice, we propose turning workplace learning into a collaborative effort with the aim of increasing awareness and the use of informal performance-relevant feedback.

Perceptions of people with respiratory problems on physician performance evaluation-A qualitative study.

BACKGROUND: Despite increasing calls for patient and public involvement in health-care quality improvement, the question of how patient evaluations can contribute to physician learning and performance assessment has received scant attention. OBJECTIVE: The objective of this study was to explore, amid calls for patient involvement in quality assurance, patients' perspectives on their role in the evaluation of physician performance and to support physicians' learning and decision making on professional competence. DESIGN: A qualitative study based on semi-structured interviews. SETTING AND PARTICIPANTS: The study took place in a secondary care setting in the Netherlands. The authors selected 25 patients from two Dutch hospitals and through the Dutch Lung Foundation, using purposive sampling. METHODS: Data were analysed according to the principles of template analysis, based on an a priori coding framework developed from the literature about patient empowerment, feedback and performance assessment. RESULTS: The analysis unearthed three predominant patient perspectives: the proactive perspective, the restrained perspective and the outsider perspective. These perspectives differed in terms of perceived power dynamics within the doctor-patient relationship, patients' perceived ability, and willingness to provide feedback and evaluate their physician's performance. Patients' perspectives thus affected the role patients envisaged for themselves in evaluating physician performance. DISCUSSION AND CONCLUSION: Although not all patients are equally suitable or willing to be involved, patients can play a role in evaluating physician performance and continuing training through formative approaches. To involve patients successfully, it is imperative to distinguish between different patient perspectives and empower patients by ensuring a safe environment for feedback.

Supervisors' intention to observe clinical task performance: an exploratory study using the theory of planned behaviour during postgraduate medical training.

BACKGROUND: Direct observation of clinical task performance plays a pivotal role in competency-based medical education. Although formal guidelines require supervisors to engage in direct observations, research demonstrates that trainees are infrequently observed. Supervisors may not only experience practical and socio-cultural barriers to direct observations in healthcare settings, they may also question usefulness or have low perceived self-efficacy in performing direct observations. A better understanding of how these multiple factors interact to influence supervisors' intention to perform direct observations may help us to more effectively implement the aforementioned guidelines and increase the frequency of direct observations. METHODS: We conducted an exploratory quantitative study, using the Theory of Planned Behaviour (TPB) as our theoretical framework. In applying the TPB, we transfer a psychological theory to medical education to get insight in the influence of cognitive and emotional processes on intentions to use direct observations in workplace based learning and assessment. We developed an instrument to investigate supervisors intention to perform direct observations. The relationships between the TPB measures of our questionnaire were explored by computing bivariate correlations using Pearson's R tests. Hierarchical regression analysis was performed in order to assess the impact of the respective TPB measures as predictors on the intention to perform direct observations. RESULTS: In our study 82 GP supervisors completed our TPB questionnaire. We found that supervisors had a positive attitude towards direct observations. Our TPB model explained 45% of the variance in supervisors' intentions to perform them. Normative beliefs and past behaviour were significant determinants of this intention. CONCLUSION: Our study suggests that supervisors use their past experiences to form intentions to perform direct observations in a careful, thoughtful manner and, in doing so, also take the preferences of the learner and other stakeholders potentially engaged in direct observations into consideration. These findings have potential implications for research into work-based assessments and the development of training interventions to foster a shared mental model on the use of direct observations.

Assessors' interpretations of narrative data on communication skills in a summative OSCE.

OBJECTIVES: Increasingly, narrative assessment data are used to substantiate and enhance the robustness of assessor judgements. However, the interpretation of written assessment comments is inherently complex and relies on human (expert) judgements. The purpose of this study was to explore how expert assessors process and construe or bring meaning to narrative data when interpreting narrative assessment comments written by others in the setting of standardised performance assessment. METHODS: Narrative assessment comments on student communication skills and communication scores across six objective structured clinical examination stations were obtained for 24 final-year pharmacy students. Aggregated narrative data across all stations were sampled for nine students (three good, three average and three poor performers, based on communication scores). A total of 10 expert assessors reviewed the aggregated set of narrative comments for each student. Cognitive (information) processing was captured through think-aloud procedures and verbal protocol analysis. RESULTS: Expert assessors primarily made use of two strategies to interpret the narratives, namely comparing and contrasting, and forming mental images of student performance. Assessors appeared to use three different perspectives when interpreting narrative comments, including those of: (i) the student (placing him- or herself in the shoes of the student); (ii) the examiner (adopting the role of examiner and reinterpreting comments according to his or her own standards or beliefs), and (iii) the professional (acting as the profession's gatekeeper by considering the assessment to be a representation of real-life practice). CONCLUSIONS: The present findings add to current understandings of assessors' interpretations of narrative performance data by identifying the strategies and different perspectives used by expert assessors to frame and bring meaning to written comments. Assessors' perspectives affect assessors' interpretations of assessment comments and are likely to be influenced by their beliefs, interpretations of the assessment setting and personal performance theories. These results call for the use of multiple assessors to account for variations in assessor perspectives in the interpretation of narrative assessment data.

Managing tensions in assessment: moving beyond either-or thinking.

CONTEXT: In health professions education, assessment systems are bound to be rife with tensions as they must fulfil formative and summative assessment purposes, be efficient and effective, and meet the needs of learners and education institutes, as well as those of patients and health care organisations. The way we respond to these tensions determines the fate of assessment practices and reform. In this study, we argue that traditional 'fix-the-problem' approaches (i.e. either-or solutions) are generally inadequate and that we need alternative strategies to help us further understand, accept and actually engage with the multiple recurring tensions in assessment programmes. METHODS: Drawing from research in organisation science and health care, we outline how the Polarity Thinking™ model and its 'both-and' approach offer ways to systematically leverage assessment tensions as opportunities to drive improvement, rather than as intractable problems. In reviewing the assessment literature, we highlight and discuss exemplars of specific assessment polarities and tensions in educational settings. Using key concepts and principles of the Polarity Thinking™ model, and two examples of common tensions in assessment design, we describe how the model can be applied in a stepwise approach to the management of key polarities in assessment. DISCUSSION: Assessment polarities and tensions are likely to surface with the continued rise of complexity and change in education and health care organisations. With increasing pressures of accountability in times of stretched resources, assessment tensions and dilemmas will become more pronounced. We propose to add to our repertoire of strategies for managing key dilemmas in education and assessment design through the adoption of the polarity framework. Its 'both-and' approach may advance our efforts to transform assessment systems to meet complex 21st century education, health and health care needs.

Exploring the effect of PAs on physician trainee learning: An interview study.

BACKGROUND: Physician assistants (PAs) often have been embedded in academic medical centers to help ensure an adequate patient care workforce while supporting compliance with work-hour restrictions for residents and fellows (also called trainees). Limited studies have explored the effect of PAs on trainee learning. This qualitative study explored, from the perspective of physician faculty and PAs, how PAs working in the clinical learning environment can enhance or hinder trainee learning. METHODS: Using purposive sampling, 12 PAs and 12 physician faculty members in one US teaching hospital were selected for semistructured interviews. Data collection and analysis were characterized by an iterative process. Data analysis was informed by principles of conventional content analysis. RESULTS: Participants identified various ways in which PAs may affect trainee learning, intrinsically linked to the roles PAs assume in the clinical learning environment: clinician, teammate, and clinical teacher. Trainee learning may be enhanced because learning time can be optimized by having PAs in the clinical learning environment. Trainees can learn about PAs and how to collaborate with them, and PAs can enculturate and provide clinical instruction to trainees. Trainee learning may be hindered if learning opportunities for trainees go to PAs, trainees feel intimidated by experienced PAs, or trainees become too dependent on PAs. CONCLUSIONS: Our findings demonstrate enhancements and hindrances to trainees' learning linked to three key roles PAs perform in the clinical learning environment. These findings can inform how PAs are integrated into teaching services. Further investigation is needed to understand how PAs can balance their professional roles to foster effective collaborative practice and learning.

From aggregation to interpretation: how assessors judge complex data in a competency-based portfolio.

While portfolios are increasingly used to assess competence, the validity of such portfolio-based assessments has hitherto remained unconfirmed. The purpose of the present research is therefore to further our understanding of how assessors form judgments when interpreting the complex data included in a competency-based portfolio. Eighteen assessors appraised one of three competency-based mock portfolios while thinking aloud, before taking part in semi-structured interviews. A thematic analysis of the think-aloud protocols and interviews revealed that assessors reached judgments through a 3-phase cyclical cognitive process of acquiring, organizing, and integrating evidence. Upon conclusion of the first cycle, assessors reviewed the remaining portfolio evidence to look for confirming or disconfirming evidence. Assessors were inclined to stick to their initial judgments even when confronted with seemingly disconfirming evidence. Although assessors reached similar final (pass-fail) judgments of students' professional competence, they differed in their information-processing approaches and the reasoning behind their judgments. Differences sprung from assessors' divergent assessment beliefs, performance theories, and inferences about the student. Assessment beliefs refer to assessors' opinions about what kind of evidence gives the most valuable and trustworthy information about the student's competence, whereas assessors' performance theories concern their conceptualizations of what constitutes professional competence and competent performance. Even when using the same pieces of information, assessors furthermore differed with respect to inferences about the student as a person as well as a (future) professional. Our findings support the notion that assessors' reasoning in judgment and decision-making varies and is guided by their mental models of performance assessment, potentially impacting feedback and the credibility of decisions. Our findings also lend further credence to the assertion that portfolios should be judged by multiple assessors who should, moreover, thoroughly substantiate their judgments. Finally, it is suggested that portfolios be designed in such a way that they facilitate the selection of and navigation through the portfolio evidence.

Box-ticking and Olympic high jumping - Physicians' perceptions and acceptance of national physician validation systems.

PURPOSE: National physician validation systems aim to ensure lifelong learning through periodic appraisals of physicians' competence. Their effectiveness is determined by physicians' acceptance of and commitment to the system. This study, therefore, sought to explore physicians' perceptions and self-reported acceptance of validation across three different physician validation systems in Europe. MATERIALS AND METHODS: Using a constructivist grounded-theory approach, we conducted semi-structured interviews with 32 respiratory specialists from three countries with markedly different validation systems: Germany, which has a mandatory, credit-based system oriented to continuing professional development; Denmark, with mandatory annual dialogs and ensuing, non-compulsory activities; and the UK, with a mandatory, portfolio-based revalidation system. We analyzed interview data with a view to identifying factors influencing physicians' perceptions and acceptance. RESULTS: Factors that influenced acceptance were the assessment's authenticity and alignment of its requirements with clinical practice, physicians' beliefs about learning, perceived autonomy, and organizational support. CONCLUSIONS: Users' acceptance levels determine any system's effectiveness. To support lifelong learning effectively, national physician validation systems must be carefully designed and integrated into daily practice. Involving physicians in their design may render systems more authentic and improve alignment between individual ambitions and the systems' goals, thereby promoting acceptance.

Stakeholder perspectives on workplace-based performance assessment: towards a better understanding of assessor behaviour.

Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments-rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically 'the locus of regulation of learning' (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use-and, consequently, the effectiveness-of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA.

Exploring the influence of cultural orientations on assessment of communication behaviours during patient-practitioner interactions.

BACKGROUND: Research has shown that patients' and practitioners' cultural orientations affect communication behaviors and interpretations in cross-cultural patient-practitioner interactions. Little is known about the effect of cultural orientations on assessment of communication behaviors in cross-cultural educational settings. The purpose of this study is to explore cultural orientation as a potential source of assessor idiosyncrasy or between-assessor variability in assessment of communication skills. More specifically, we explored if and how (expert) assessors' valuing of communication behaviours aligned with their cultural orientations (power-distance, masculinity-femininity, uncertainty avoidance, and individualism-collectivism). METHODS: Twenty-five pharmacist-assessors watched 3 videotaped scenarios (patient-pharmacist interactions) and ranked each on a 5-point global rating scale. Videotaped scenarios demonstrated combinations of well-portrayed and borderline examples of instrumental and affective communication behaviours. We used stimulated recall and verbal protocol analysis to investigate assessors' interpretations and evaluations of communication behaviours. Uttered assessments of communication behaviours were coded as instrumental (task-oriented) or affective (socioemotional) and either positive or negative. Cultural orientations were measured using the Individual Cultural Values Scale. Correlations between cultural orientations and global scores, and frequencies of positive, negative, and total utterances of instrumental and affective behaviours were determined. RESULTS: Correlations were found to be scenario specific. In videos with poor or good performance, no differences were found across cultural orientations. When borderline performance was demonstrated, high power-distance and masculinity were significantly associated with higher global ratings (r = .445, and .537 respectively, p < 0.05) as well as with fewer negative utterances regarding instrumental (task focused) behaviours (r = -.533 and - .529, respectively). Higher masculinity scores were furthermore associated with positive utterances of affective (socioemotional) behaviours (r = .441). CONCLUSIONS: Our findings thus confirm cultural orientation as a source of assessor idiosyncrasy and meaningful variations in interpretation of communication behaviours. Interestingly, expert assessors generally agreed on scenarios of good or poor performances but borderline performance was influenced by cultural orientation. Contrary to current practices of assessor and assessment instrument standardization, findings support the use of multiple assessors for patient-practitioner interactions and development of qualitative assessment tools to capture these varying, yet valid, interpretations of performance.

Relationships between medical students' co-regulatory network characteristics and self-regulated learning: a social network study.

INTRODUCTION: Recent conceptualizations of self-regulated learning acknowledge the importance of co-regulation, i.e., students' interactions with others in their networks to support self-regulation. Using a social network approach, the aim of this study is to explore relationships between characteristics of medical students' co-regulatory networks, perceived learning opportunities, and self-regulated learning. METHODS: The authors surveyed 403 undergraduate medical students during their clinical clerkships (response rate 65.5%). Using multiple regression analysis, structural equation modelling techniques, and analysis of variance, the authors explored relationships between co-regulatory network characteristics (network size, network diversity, and interaction frequency), students' perceptions of learning opportunities in the workplace setting, and self-reported self-regulated learning. RESULTS: Across all clerkships, data showed positive relationships between tie strength and self-regulated learning (ß = 0.095, p < 0.05) and between network size and tie strength (ß = 0.530, p < 0.001), and a negative relationship between network diversity and tie strength (ß = -0.474, p < 0.001). Students' perceptions of learning opportunities showed positive relationships with both self-regulated learning (ß = 0.295, p < 0.001) and co-regulatory network size (ß = 0.134, p < 0.01). Characteristics of clerkship contexts influenced both co-regulatory network characteristics (size and tie strength) and relationships between network characteristics, self-regulated learning, and students' perceptions of learning opportunities. DISCUSSION: The present study reinforces the importance of co-regulatory networks for medical students' self-regulated learning during clinical clerkships. Findings imply that supporting development of strong networks aimed at frequent co-regulatory interactions may enhance medical students' self-regulated learning in challenging clinical learning environments. Social network approaches offer promising ways of further understanding and conceptualising self- and co-regulated learning in clinical workplaces.

Student perspectives on competency-based portfolios: Does a portfolio reflect their competence development?

INTRODUCTION: Portfolio-based assessments require that learners' competence development is adequately reflected in portfolio documentation. This study explored how students select and document performance data in their portfolios and how they perceive these data to be representative for their competence development. METHODS: Students uploaded performance data in a competency-based portfolio. During one clerkship period, twelve students also recorded an audio diary in which they reflected on experiences and feedback that they perceived to be indicants of their competence development. Afterwards, these students were interviewed to explore the extent to which the performance documentation in the portfolio corresponded with what they considered illustrative evidence of their development. The interviews were analyzed using thematic analysis. RESULTS: Portfolios provide an accurate but fragmented picture of student development. Portfolio documentation was influenced by tensions between learning and assessment, student beliefs about the goal of portfolios, student performance evaluation strategies, the learning environment and portfolio structure. DISCUSSION: This study confirms the importance of taking student perceptions into account when implementing a competency-based portfolio. Students would benefit from coaching on how to select meaningful experiences and performance data for documentation in their portfolios. Flexibility in portfolio structure and requirements is essential to ensure optimal fit between students' experienced competence development and portfolio content.

Reliability of narrative assessment data on communication skills in a summative OSCE.

OBJECTIVE: To quantitatively estimate the reliability of narrative assessment data regarding student communication skills obtained from a summative OSCE and to compare reliability to that of communication scores obtained from direct observation. METHODS: Narrative comments and communication scores (scale 1-5) were obtained for 14 graduating pharmacy students across 6 summative OSCE stations with 2 assessors per station who directly observed student performance. Two assessors who had not observed the OSCE reviewed narratives and independently scored communication skills according to the same 5-point scale. Generalizability theory was used to estimate reliability. Correlation was used to evaluate the relationship between scores from each assessment method. RESULTS: A total of 168 narratives and communication scores were obtained. The G-coefficients were 0.571 for scores provided by assessors present during the OSCE and 0.612 for scores from assessors who provided scores based on narratives only. Correlation between the two sets of scores was 0.5. CONCLUSION: Reliability of communication scores is not dependent on whether assessors directly observe student performance or assess written narratives, yet both conditions appear to measure communication skills somewhat differently. PRACTICE IMPLICATIONS: Narratives may be useful for summative decision-making and help overcome the current limitations of using solely quantitative scores.