Multisource feedback: an overview of its use and application as a formative assessment.

Multisource feedback (MSF), often termed 360-degree feedback, is a formative performance assessment in which data about an individual's observable workplace behaviors are collected through questionnaires from those interacting with the individual; data are aggregated for anonymity and confidentiality; the aggregated data, along with self-assessment if available, are provided to the individual; and the recipient meets with a trusted individual to review the data and develop an action plan. It is used along the continuum of medical education. This article provides an overview of MSF's utility, its evidence base and cautions.

La rétroaction multisource (RMS), ou rétroaction 360 degrés, est une évaluation formative du rendement dans laquelle des informations sur les comportements observables d'un individu dans son lieu de travail sont recueillies par le biais de questionnaires auprès de ceux avec qui il interagit. Après avoir été agrégées pour garantir l'anonymat et la confidentialité, ces données, et l'auto-évaluation s'il y a lieu, sont remises à la personne évaluée. Accompagnée d'une personne de confiance, elle les examinera et élaborera un plan d'action, qui sera utilisé tout au long du continuum de sa formation médicale. Cet article présente un aperçu de l'utilité et des fondements de la RMS, ainsi que quelques mises en garde.

An Examination of Self-Reported Assessment Activities Documented by Specialist Physicians for Maintenance of Certification.

INTRODUCTION: Specialists in a Maintenance of Certification program are required to participate in assessment activities, such as chart audit, simulation, knowledge assessment, and multisource feedback. This study examined data from five different specialties to identify variation in participation in assessment activities, examine differences in the learning stimulated by assessment, assess the frequency and type of planned changes, and assess the association between learning, discussion, and planned changes. METHODS: E-portfolio data were categorized and analyzed descriptively. Chi-squared tests examined associations. RESULTS: A total of 2854 anatomical pathologists, cardiologists, gastroenterologists, ophthalmologists, and orthopedic surgeons provided data about 6063 assessment activities. Although there were differences in the role that learning played by discipline and assessment type, the most common activities documented across all specialties were self-assessment programs (n = 2122), feedback on teaching (n = 1078), personal practice assessments which the physician did themselves (n = 751), annual reviews (n = 682), and reviews by third parties (n = 661). Learning occurred for 93% of the activities and was associated with change. For 2126 activities, there were planned changes. Activities in which there was a discussion with a peer or supervisor were more likely to result in a change. CONCLUSIONS AND DISCUSSION: Although specialists engaged in many types of assessment activities to meet the Maintenance of Certification program requirements, there was variability in how assessment stimulated learning and planned changes. It seems that peer discussion may be an important component in fostering practice change and forming plans for improvement which bears further study.

Picking the Right Tool for the Job: A Reliability Study of 4 Assessment Tools for Central Venous Catheter Insertion.

BACKGROUND: Determining procedural competence requires psychometrically sound assessment tools. A variety of instruments are available to determine procedural performance for central venous catheter (CVC) insertion, but it is not clear which ones should be used in the context of competency-based medical education. OBJECTIVE: We compared several commonly used instruments to determine which should be preferentially used to assess competence in CVC insertion. METHODS: Junior residents completing their first intensive care unit rotation between July 31, 2006, and March 9, 2007, were video-recorded performing CVC insertion on task trainer mannequins. Between June 1, 2016, and September 30, 2016, 3 experienced raters judged procedural competence on the historical video recordings of resident performance using 4 separate tools, including an itemized checklist, Objective Structured Assessment of Technical Skills (OSATS), a critical error assessment tool, and the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE). Generalizability theory (G-theory) was used to compare the performance characteristics among the tools. A decision study predicted the optimal testing environment using the tools. RESULTS: At the time of the original recording, 127 residents rotated through intensive care units at the University of Calgary, Alberta, Canada. Seventy-seven of them (61%) met inclusion criteria, and 55 of those residents (71%) agreed to participate. Results from the generalizability study (G-study) demonstrated that scores from O-SCORE and OSATS were the most dependable. Dependability could be maintained for O-SCORE and OSATS with 2 raters. CONCLUSIONS: Our results suggest that global rating scales, such as the OSATS or the O-SCORE tools, should be preferentially utilized for assessment of competence in CVC insertion.

Assessment and Change: An Exploration of Documented Assessment Activities and Outcomes by Canadian Psychiatrists.

INTRODUCTION: Fellows of the Royal College of Physicians and Surgeons of Canada are required to participate in assessment activities for all new 5-year cycles beginning on or after January 2014 to meet the maintenance of certification program requirements. This study examined the assessment activities which psychiatrists reported in their maintenance of certification e-portfolios to determine the types and frequency of activities reported; the resultant learning, planned learning, and/or changes to the practice they planned or implemented; and the interrelationship between the types of assessment activities, learning that was affirmed or planned, and changes planned or implemented. METHODS: A total of 5000 entries from 2195 psychiatrists were examined. A thematic analysis drawing on the framework analysis was undertaken of the 2016 entries. RESULTS: There were 3841 entries for analysis; 1159 entries did not meet the criteria for assessment. The most commonly reported activities were self-assessment programs, feedback on teaching, regular performance reviews, and chart reviews. Less frequent were direct observation, peer supervision, and reviews by provincial medical regulatory authorities. In response to the data, psychiatrists affirmed that their practices were appropriate, identified gaps they intended to address, planned future learning, and/or planned or implemented changes. The assessment activities were internally or externally initiated and resulted in no or small changes (accommodations and adjustments) or redirections. DISCUSSION: Psychiatrists reported participating in a variety of assessment activities that resulted in variable impact on learning and change. The study underscores the need to ensure that assessments being undertaken are purposeful, relevant, and designed to enable identification of outcomes that impact practice.

Core principles of assessment in competency-based medical education.

The meaningful assessment of competence is critical for the implementation of effective competency-based medical education (CBME). Timely ongoing assessments are needed along with comprehensive periodic reviews to ensure that trainees continue to progress. New approaches are needed to optimize the use of multiple assessors and assessments; to synthesize the data collected from multiple assessors and multiple types of assessments; to develop faculty competence in assessment; and to ensure that relationships between the givers and receivers of feedback are appropriate. This paper describes the core principles of assessment for learning and assessment of learning. It addresses several ways to ensure the effectiveness of assessment programs, including using the right combination of assessment methods and conducting careful assessor selection and training. It provides a reconceptualization of the role of psychometrics and articulates the importance of a group process in determining trainees' progress. In addition, it notes that, to reach its potential as a driver in trainee development, quality care, and patient safety, CBME requires effective information management and documentation as well as ongoing consideration of ways to improve the assessment system.

More Than Reducing Complexity: Canadian Specialists' Views of the Royal College's Maintenance of Certification Framework and Program.

INTRODUCTION: Maintenance of Certification (MOC) programs are facing competing pressures from physicians and regulators to design and deliver systems that ensure physicians sustain or maintain competence postcertification. Adding to these pressures is an emerging discourse of dissatisfaction by physicians participating in MOC programs suggesting physicians themselves are seeking programs that are relevant to their scope of practice. The object was to determine Canadian specialists' views of a redesigned Royal College of Physician and Surgeons of Canada's MOC program in five priority areas: complexity, motivation, impact, credit sufficiency, and enhancement. METHODS: A national, cross-specialty, survey of Royal College of Physician and Surgeons of Canada's MOC program users who had participated for at least 1 year. RESULTS: Five thousand two hundred fifty nine respondents were included in the analysis. A majority of participates either "agreed" (2258, 43%) or "strongly agreed" (631, 12%) that the new framework was simpler to understand and the redesigned submission templates were simpler to use ("agree" 2297 [43.7%], "strongly agree" 701 [13.3%]). A total of 64.5% had a good understanding of the program purposes; however, some respondents believe the program does not yet deliver intended outcomes. Maintaining Fellowship designation, regulatory requirements, and a desire to remain up-to-date were indicated as the primary motivators for engaging in MOC. The program has had a modest influence on users' approach to lifelong learning ("neither agree nor disagree"). DISCUSSION: The program redesign reduced the perceived complexity of the MOC framework however it is described by some, as additional work (an add-on) by necessitating the documentation of participation in learning activities. An important "disconnect" between the program purposes and how physicians operationalize engagement was also illuminated by our study. Further attention needs to be paid to shifting the culture of continuing professional development to one that enables physicians to sufficiently demonstrate their continuing competence and enhanced performance meaningfully through participation in MOC programs.

Supporting Veterinary Preceptors in a Distributed Model of Education: A Faculty Development Needs Assessment.

Effective faculty development for veterinary preceptors requires knowledge about their learning needs and delivery preferences. Veterinary preceptors at community practice locations in Alberta, Canada, were surveyed to determine their confidence in teaching ability and interest in nine faculty development topics. The study included 101 veterinarians (48.5% female). Of these, 43 (42.6%) practiced veterinary medicine in a rural location and 54 (53.5%) worked in mixed-animal or food-animal practice. Participants reported they were more likely to attend an in-person faculty development event than to participate in an online presentation. The likelihood of attending an in-person event differed with the demographics of the respondent. Teaching clinical reasoning, assessing student performance, engaging and motivating students, and providing constructive feedback were topics in which preceptors had great interest and high confidence. Preceptors were least confident in the areas of student learning styles, balancing clinical workload with teaching, and resolving conflict involving the student. Disparities between preceptors' interest and confidence in faculty development topics exist, in that topics with the lowest confidence scores were not rated as those of greatest interest. While the content and format of clinical teaching faculty development events should be informed by the interests of preceptors, consideration of preceptors' confidence in teaching ability may be warranted when developing a faculty development curriculum.

Focused Critical Care Echocardiography: Development and Evaluation of an Image Acquisition Assessment Tool.

OBJECTIVES: Little attention has been placed on assessment tools to evaluate image acquisition quality for focused critical care echocardiography. We designed a novel assessment tool to objectively evaluate the image acquisition skills of critical care trainees learning focused critical care echocardiography and examined the tool for evidence of validity. DESIGN: Prospective observational study. SETTING: Medical-surgical ICUs at a tertiary care teaching hospital. SUBJECTS: Trainees in our critical care medicine fellowship program. INTERVENTIONS: Six trainees completed a focused critical care echocardiography training curriculum followed by performing 20 transthoracic echocardiograms on patients receiving invasive mechanical ventilation. At three assessment intervals (the 1st and 2nd examinations, 10th and 11th examinations, and 19th and 20th examinations), echocardiograms performed by trainees were compared with those of critical care physicians certified in echocardiography and scored according to the focused critical care echocardiography assessment tool. The primary outcome was an efficiency score (overall assessment tool score divided by examination time). Differences in mean efficiency scores between echocardiographers of differing skill levels and changes in trainees' mean efficiency scores with increasing focused critical care echocardiography experience were compared by using t tests. MEASUREMENTS AND MAIN RESULTS: On the initial assessment, mean efficiency scores (SD) for trainees and experienced physicians were 1.55 (0.95) versus 2.78 (1.38), respectively (p = 0.02), and for the second and third assessments, the corresponding efficiency ratings for trainees and experienced physicians were 2.48 (0.97) versus 4.55 (1.32) (p < 0.01) and 2.61 (1.37) versus 4.17 (2.12) (p = 0.04), respectively. CONCLUSIONS: Trainees' efficiency in focused critical care echocardiography image acquisition improved quickly in the first 10 studies, yet, it could not match with the performance of experienced physicians after 20 focused critical care echocardiography studies. The focused critical care echocardiography assessment tool demonstrated evidence of validity and could discern changes in trainees' image acquisition performance with increasing experience.

Exploring Perceptions of Early-Career Psychiatrists About Their Relationships With the Pharmaceutical Industry.

OBJECTIVES: The pharmaceutical industry has engaged physicians through medical education, patient care, and medical research. New conflict of interest policy has highlighted the challenges to these relationships. The objective of this study was to explore the perceptions that early career psychiatrists (e.g. those within 5 years of entering practice) have regarding their relationship with the pharmaceutical industry. METHODS: Data were collected through semi-structured interviews and were analysed using a grounded theory methodology. Interviews were conducted and analyzed in an iterative way using a constant comparison approach in which data were collected and open coded for themes and subthemes. As new interviews were conducted, the themes were applied to data along with emergent themes and previous interviews recoded until additional interviews failed to provide new themes and thematic saturation was achieved. Through axial coding, a process of relating codes (categories and concepts) to each other, the theory was generated to explain the core variable mediating perceptions participants had about the relationship with industry. RESULTS: The participants described increasing frequency of experiences with industry throughout training into practice. Their perceptions developed through training, physician culture, industry promotion, and their own practices. In managing the relationship with industry, participants would either avoid interactions or engage in behaviors aimed to reduce the risk of influence. Maintaining one's professional integrity was the underlying driver used to manage the relationship with industry. CONCLUSIONS: Psychiatrists develop perceptions about industry through experience and observation leading them to develop their own strategies to manage these relationships while maintaining their professional integrity.

Role for assessment in maintenance of certification: physician perceptions of assessment.

INTRODUCTION: The Royal College of Physicians and Surgeons of Canada modified its Maintenance of Certification (MOC) framework in 2011 to further incentivize assessment activities compared to group and self-learning. The purpose of this study was to explore physician's perceptions of their access to assessment activities, barriers to participation in assessment, and the need for the Royal College to further support its fellows in gaining access to assessment activities. METHODS: A questionnaire-based survey was sent to all participants of the MOC program as part of a program evaluation examining recent changes to the MOC program. RESULTS: 5259 respondents contributed responses. Most physicians were comfortable with the revised framework for assessment while approximately 40% were neutral regarding whether lack of access to self-assessment activities was a problem. Respondents expressed a need for more self-assessment programs particularly those developed outside of Canada. Neither a lack of feedback about performance or discomfort with recording performance gaps was perceived as a barrier to participation in assessment activities. Physician comments were consistent with the quantitative data and elaborated on the need to develop and recognize more assessment activities. DISCUSSION: Physicians accepted the revised MOC program framework but perceived difficulty in accessing assessment programs, activities, and tools. As the framework changed again January 2014, requiring all fellows and MOC program participants to completion of at least 25 credits in each section of the MOC program (including assessment) during their new 5-year MOC cycle, additional resources will be needed to support opportunities for physicians to engage in assessment.

Targeted needs assessment for a transitional "boot camp" curriculum for pediatric surgery residents.

INTRODUCTION: Transition periods in medical education are associated with increased risk for learners and patients. For pediatric surgery residents, the transition to training is especially difficult as learners must adjust to new patient populations. In this study we perform a targeted needs assessment to determine the ideal content and format of a pediatric surgery boot camp to facilitate the transition to residency. METHODS: A needs assessment survey was developed and distributed to pediatric surgery residents and staff across North America. The survey asked participants to rank 30 pediatric surgical diagnoses, 20 skills, and 11 physiological topics on "frequency" and "importance". Items were then ranked using empirical methods. The survey also evaluated the preferred boot camp format. RESULTS: In total, 12 residents and 23 staff completed the survey. No significant differences were identified between staff and residents in survey responses. The top 5 topics identified for inclusion in a boot camp were: (1) fluid and electrolyte management, (2) appendicitis, (3) pediatric hernias, (4) nutrition and (5) pain management. The preferred format for a boot camp was 3-4days in duration applying a blend of educational methods. CONCLUSIONS: Based on the results of the needs assessment survey, a novel pediatric surgery boot camp curriculum can be developed.

Toward a common understanding: supporting and promoting education scholarship for medical school faculty.

CONTEXT: Education scholarship (ES) is integral to the transformation of medical education. Faculty members who engage in ES need encouragement and recognition of this work. Beginning with the definition of ES as 'an umbrella term which can encompass both research and innovation in health professions education', and which as such represents an activity that is separate and distinct from teaching and leadership, the purpose of our study was to explore how promotion policies and processes are used in Canadian medical schools to support and promote ES. METHODS: We conducted an analysis of the promotion policies of 17 Canadian medical schools and interviews with a key informant at each institution. We drew on an interpretive approach to policy analysis to analyse the data and to understand explicit messages about how ES was represented and supported. RESULTS: Of the 17 schools' promotion documents, only nine contained specific reference to ES. There was wide variation in focus and level of detail. All key informants indicated that ES is recognised and considered for academic promotion. Barriers to the support and recognition of ES included a lack of understanding of ES and its relationship to teaching and leadership. This was manifest in the variability in promotion policies and processes, support systems, and career planning and pathways for ES. CONCLUSIONS: This lack of clarity may make it challenging for medical school faculty members to make sense of how they might successfully align ES within an academic career. There is a need therefore to better articulate ES in promotion policies and support systems. Creating a common understanding of ES, developing guidelines to assess the impact of all forms of ES, developing an informed leadership and system of mentors, and creating explicit role descriptions and guidelines are identified as potential strategies to ensure that ES is appropriately valued.

Multisource feedback: can it meet criteria for good assessment?

INTRODUCTION: High-quality instruments are required to assess and provide feedback to practicing physicians. Multisource feedback (MSF) uses questionnaires from colleagues, coworkers, and patients to provide data. It enables feedback in areas of increasing interest to the medical profession: communication, collaboration, professionalism, and interpersonal skills. The purpose of the study was to apply the 7 assessment criteria as a framework to examine the quality of MSF instruments used to assess practicing physicians. METHODS: The criteria for assessment (validity, reproducibility, equivalence, feasibility, educational effect, catalytic effect, and acceptability) were examined for 3 sets of instruments, drawing on published data. RESULTS: Three MSF instruments with a sufficient body of research for inclusion-the Canadian Physician Achievement Review instruments and the United Kingdom's GMC and CFEP360 instruments-were examined. There was evidence that MSF has been assessed against all criteria except educational effects, although variably for some of the instruments. The greatest emphasis was on validity, reproducibility, and feasibility for all of the instruments. Assessments of the catalytic effect were not available for 1 of the 2 UK instruments and minimally examined for the other. Data about acceptability are implicit in the UK instruments from their endorsement by the Royal College of General Practice and explicitly examined in the Canadian instruments. DISCUSSION: The 7 criteria provided a useful framework to assess the quality of MSF instruments and enable an approach to analyzing gaps in instrument assessment. These criteria are likely to be helpful in assessing other instruments used in medical education.

The development and assessment of an evaluation tool for pediatric resident competence in leading simulated pediatric resuscitations.

INTRODUCTION: It is critical that competency in pediatric resuscitation is achieved and assessed during residency or post graduate medical training. The purpose of this study was to create and evaluate a tool to measure all elements of pediatric resuscitation team leadership competence. METHODS: An initial set of items, derived from a literature review and a brainstorming session, were refined to a 26 item assessment tool through the use of Delphi methodology. The tool was tested using videos of standardized resuscitations. A psychometric assessment of the evidence for instrument validity and reliability was undertaken. RESULTS: The performance of 30 residents on two videotaped scenarios was assessed by 4 pediatricians using the tool, with 12 items assessing 'leadership and communication skills' (LCS) and 14 items assessing 'knowledge and clinical skills' (KCS). The instrument showed evidence of reliability; the Cronbach's alpha and generalizability co-efficients for the overall instrument were α=0.818 and Ep(2)=0.76, for LCS were α=0.827 and Ep(2)=0.844, and for KCS were α=0.673 and Ep(2)=0.482. While validity was initially established through literature review and brainstorming by the panel of experts, it was further built through the high strength of correlation between global scores and scores for overall performance (r=0.733), LCS (r=0.718) and KCS (r=0.662) as well as the factor analysis which accounted for 40.2% of the variance. CONCLUSION: The results of the study demonstrate that the instrument is a valid and reliable tool to evaluate pediatric resuscitation team leader competence.

Tensions in informed self-assessment: how the desire for feedback and reticence to collect and use it can conflict.

PURPOSE: Informed self-assessment describes the set of processes through which individuals use external and internal data to generate an appraisal of their own abilities. The purpose of this project was to explore the tensions described by learners and professionals when informing their self-assessments of clinical performance. METHOD: This 2008 qualitative study was guided by principles of grounded theory. Eight programs in five countries across undergraduate, postgraduate, and continuing medical education were purposively sampled. Seventeen focus groups were held (134 participants). Detailed analyses were conducted iteratively to understand themes and relationships. RESULTS: Participants experienced multiple tensions in informed self-assessment. Three categories of tensions emerged: within people (e.g., wanting feedback, yet fearing disconfirming feedback), between people (e.g., providing genuine feedback yet wanting to preserve relationships), and in the learning/practice environment (e.g., engaging in authentic self-assessment activities versus "playing the evaluation game"). Tensions were ongoing, contextual, and dynamic; they prevailed across participant groups, infusing all components of informed self-assessment. They also were present in varied contexts and at all levels of learners and practicing physicians. CONCLUSIONS: Multiple tensions, requiring ongoing negotiation and renegotiation, are inherent in informed self-assessment. Tensions are both intraindividual and interindividual and they are culturally situated, reflecting both professional and institutional influences. Social learning theories (social cognitive theory) and sociocultural theories of learning (situated learning and communities of practice) may inform our understanding and interpretation of the study findings. The findings suggest that educational interventions should be directed at individual, collective, and institutional cultural levels. Implications for practice are presented.

Controlling quality in CME/CPD by measuring and illuminating bias.

INTRODUCTION: There has been a surge of interest in the area of bias in industry-supported continuing medical education/continuing professional development (CME/CPD) activities. In 2007, we published our first study on measuring bias in CME, demonstrating that our assessment tool was valid and reliable. In light of the increasing interest in this area, and building on our experience, we wanted to further understand the application of this tool in different environments. We invited other CME/CPD providers from multiple sites in Canada to participate in a second CME bias study. METHODS: A new steering committee was established with representatives from 5 academic CME/CPD offices nationally, the Royal College of Physicians and Surgeons, and the College of Family Physicians of Canada to outline the project in terms of review of the literature, refining items on the tool, updating the training guide for implementation, and establishing a resource Web site for reviewers. Training involved a train-the-trainer session with the event coordinators at each of the 5 participating centers via videoconferencing. RESULTS: The content reviews from the study showed moderate inter-rater reliability (ICC = 0.54), and the live reviews showed poor overall inter-rater reliability; however, one center achieved substantial inter-rater reliability (ICC = 0.68). DISCUSSION: The analysis from this study suggests that the tool can be used as a part of a multistage process to introduce quality control mechanisms to help raise standards for CME/CPD. It is imperative to develop a cost-effective standardized training protocol that can be implemented at all sites to maximize the reliability of the tool.

Features of assessment learners use to make informed self-assessments of clinical performance.

CONTEXT: Conceptualisations of self-assessment are changing as its role in professional development comes to be viewed more broadly as needing to be both externally and internally informed through activities that enable access to and the interpretation and integration of data from external sources. Education programmes use various activities to promote learners' reflection and self-direction, yet we know little about how effective these activities are in 'informing' learners' self-assessments. OBJECTIVES: This study aimed to increase understanding of the specific ways in which undergraduate and postgraduate learners used learning and assessment activities to inform self-assessments of their clinical performance. METHODS: We conducted an international qualitative study using focus groups and drawing on principles of grounded theory. We recruited volunteer participants from three undergraduate and two postgraduate programmes using structured self-assessment activities (e.g. portfolios). We asked learners to describe their perceptions of and experiences with formal and informal activities intended to inform self-assessment. We conducted analysis as a team using a constant comparative process. RESULTS: Eighty-five learners (53 undergraduate, 32 postgraduate) participated in 10 focus groups. Two main findings emerged. Firstly, the perceived effectiveness of formal and informal assessment activities in informing self-assessment appeared to be both person- and context-specific. No curricular activities were considered to be generally effective or ineffective. However, the availability of high-quality performance data and standards was thought to increase the effectiveness of an activity in informing self-assessment. Secondly, the fostering and informing of self-assessment was believed to require credible and engaged supervisors. CONCLUSIONS: Several contextual and personal conditions consistently influenced learners' perceptions of the extent to which assessment activities were useful in informing self-assessments of performance. Although learners are not guaranteed to be accurate in their perceptions of which factors influence their efforts to improve performance, their perceptions must be taken into account; assessment strategies that are perceived as providing untrustworthy information can be anticipated to have negligible impact.

Feedback data sources that inform physician self-assessment.

BACKGROUND: Self-assessment is a process of interpreting data about one's performance and comparing it to explicit or implicit standards. AIM: To examine the external data sources physicians used to monitor themselves. METHODS: Focus groups were conducted with physicians who participated in three practice improvement activities: a multisource feedback program; a program providing patient and chart audit data; and practice-based learning groups. We used grounded theory strategies to understand the external sources that stimulated self-assessment and how they worked. RESULTS: Data from seven focus groups (49 physicians) were analyzed. Physicians used information from structured programs, other educational activities, professional colleagues, and patients. Data were of varying quality, often from non-formal sources with implicit (not explicit) standards. Mandatory programs elicited variable responses, whereas data and activities the physicians selected themselves were more likely to be accepted. Physicians used the information to create a reference point against which they could weigh their performance using it variably depending on their personal interpretation of its accuracy, application, and utility. CONCLUSIONS: Physicians use and interpret data and standards of varying quality to inform self-assessment. Physicians may benefit from regular and routine feedback and guidance on how to seek out data for self-assessment.

The processes and dimensions of informed self-assessment: a conceptual model.

PURPOSE: To determine how learners and physicians engaged in various structured interventions to inform self-assessment, how they perceived and used self-assessment in clinical learning and practice, and the components and processes comprising informed self-assessment and factors that influence these. METHOD: This was a qualitative study guided by principles of grounded theory. Using purposive sampling, eight programs were selected in Canada, the United States, the United Kingdom, the Netherlands, and Belgium, representing low, medium, and high degrees of structure/rigor in self-assessment activities. In 2008, 17 focus groups were conducted with 134 participants (53 undergraduate learners, 32 postgraduate learners, 49 physicians). Focus-group transcripts were analyzed interactively and iteratively by the research team to identify themes and compare and confirm findings. RESULTS: Informed self-assessment appeared as a flexible, dynamic process of accessing, interpreting, and responding to varied external and internal data. It was characterized by multiple tensions arising from complex interactions among competing internal and external data and multiple influencing conditions. The complex process was evident across the continuum of medical education and practice. A conceptual model of informed self-assessment emerged. CONCLUSIONS: Central challenges to informing self-assessment are the dynamic interrelationships and underlying tensions among the components comprising self-assessment. Realizing this increases understanding of why self-assessment accuracy seems frequently unreliable. Findings suggest the need for attention to the varied influencing conditions and inherent tensions to progress in understanding self-assessment, how it is informed, and its role in self-directed learning and professional self-regulation. Informed self-assessment is a multidimensional, complex construct requiring further research.

Learning to practice in Canada: the hidden curriculum of international medical graduates.

INTRODUCTION: There is movement of physicians internationally. In some cases, physicians are recruited from low-income countries to wealthier countries like Canada to provide medical services in underresourced communities. This needs assessment examined the clinical medicine learning challenges faced by international medical graduates (IMGs) from the perspective of both the IMGs and medical leaders (eg, Vice President-Medical for a Health Region). METHODS: Focus groups with 25 IMGs were held in 6 regional centers. Face-to-face interviews were held with 10 medical leaders. Participants were asked about the learning associated with patient management, patient referral, and investigation, for billing and insurance, and learning about new systems of care. Qualitative data were analyzed to determine how well the perspectives on learning were aligned. RESULTS: IMGs and medical leaders recognized that learning and support were needed by physicians without previous experience in Canada. They had similar lists of learning issues. Although medical leaders believed the new information was explicit, readily available, and could be learned from short explanations and lists; IMGs found that guidelines and expectations were implicit, confusing, and contradictory. There were mediating influences in the form of orientation programs, other IMGs, and "how to" lists in some cases, which helped the newcomer. DISCUSSION: There was concordance about aspects of the learning that was required between IMGs and medical leaders. There was little agreement about the approach to learning or a recognition that the learning tasks were complicated.