Data sharing and big data in health professions education: Ottawa consensus statement and recommendations for scholarship.

Changes in digital technology, increasing volume of data collection, and advances in methods have the potential to unleash the value of big data generated through the education of health professionals. Coupled with this potential are legitimate concerns about how data can be used or misused in ways that limit autonomy, equity, or harm stakeholders. This consensus statement is intended to address these issues by foregrounding the ethical imperatives for engaging with big data as well as the potential risks and challenges. Recognizing the wide and ever evolving scope of big data scholarship, we focus on foundational issues for framing and engaging in research. We ground our recommendations in the context of big data created through data sharing across and within the stages of the continuum of the education and training of health professionals. Ultimately, the goal of this statement is to support a culture of trust and quality for big data research to deliver on its promises for health professions education (HPE) and the health of society. Based on expert consensus and review of the literature, we report 19 recommendations in (1) framing scholarship and research through research, (2) considering unique ethical practices, (3) governance of data sharing collaborations that engage stakeholders, (4) data sharing processes best practices, (5) the importance of knowledge translation, and (6) advancing the quality of scholarship through multidisciplinary collaboration. The recommendations were modified and refined based on feedback from the 2022 Ottawa Conference attendees and subsequent public engagement. Adoption of these recommendations can help HPE scholars share data ethically and engage in high impact big data scholarship, which in turn can help the field meet the ultimate goal: high-quality education that leads to high-quality healthcare.

Validity evidence and psychometric evaluation of a socially accountable health index for health professions schools.

There is an expectation that health professions schools respond to priority societal health needs. This expectation is largely based on the underlying assumption that schools are aware of the priority needs in their communities. This paper demonstrates how open-access, pan-national health data can be used to create a reliable health index to assist schools in identifying societal needs and advance social accountability in health professions education. Using open-access data, a psychometric evaluation was conducted to examine the reliability and validity of the Canadian Health Indicators Framework (CHIF) conceptual model. A non-linear confirmatory factor analysis (CFA) on 67 health indicators, at the health-region level (n = 97) was used to assess the model fit of the hypothesized 10-factor model. Reliability analysis using McDonald's Omega were conducted, followed by Pearson's correlation coefficients. Findings from the non-linear CFA rejected the original conceptual model structure of the CHIF. Exploratory post hoc analyses were conducted using modification indices and parameter constraints to improve model fit. A final 5-factor multidimensional model demonstrated superior fit, reducing the number of indicators from 67 to 32. The 5-factors included: Health Conditions (8-indicators); Health Functions (6-indicators); Deaths (5-indicators); Non-Medical Health Determinants (7-indicators); and Community & Health System Characteristics (6-indicators). All factor loadings were statistically significant (p < 0.001) and demonstrated excellent internal consistency ( ω >0.95). Many schools struggle to identify and measure socially accountable outcomes. The process highlighted in this paper and the indices developed serve as starting points to allow schools to leverage open-access data as an initial step in identifying societal needs.

Global Perceptions on Social Accountability and Outcomes: A Survey of Medical Schools.

Phenomenon: Social accountability has become a universal component in medical education. However, medical schools have little guidance for operationalizing and applying this concept in practice. This study explored institutional practices and administrative perceptions of social accountability in medical education. Approach: An online survey was distributed to a purposeful sample of English-speaking undergraduate medical school deans and program directors/leads from 245 institutions in 14 countries. The survey comprised of 38-items related to program mission statements, admission processes, curricular content, and educational outcomes. Survey items were developed using previous literature and categorized using a context-input-process-products (CIPP) evaluation model. Exploratory Factor Analysis (EFA) was used to assess the inter-relationship among survey items. Reliability and internal consistency of items were evaluated using McDonald's Omega. Findings: Results from 81 medical schools in 14 countries collected between February and June 2020 are presented. Institutional commonalities of social accountability were observed. However, our findings suggest programs focus predominately on educational inputs and processes, and not necessarily on outcomes. Findings from our EFA demonstrated excellent internal consistency and reliability. Four-factors were extracted: (1) selection and recruitment; (2) institutional mandates; (3) institutional activities; and (4) community awareness, accounting for 71% of the variance. McDonald's Omega reliability estimates for subscales ranged from 0.80-0.87. Insights: This study identified common practices of social accountability. While many medical schools expressed an institutional commitment to social accountability, their effects on the community remain unknown and not evaluated. Overall, this paper offers programs and educators a psychometrically supported tool to aid in the operationalization and reliability of evaluating social accountability.

Using Resident-Sensitive Quality Measures Derived From Electronic Health Record Data to Assess Residents' Performance in Pediatric Emergency Medicine.

PURPOSE: Traditional quality metrics do not adequately represent the clinical work done by residents and, thus, cannot be used to link residency training to health care quality. This study aimed to determine whether electronic health record (EHR) data can be used to meaningfully assess residents' clinical performance in pediatric emergency medicine using resident-sensitive quality measures (RSQMs). METHOD: EHR data for asthma and bronchiolitis RSQMs from Cincinnati Children's Hospital Medical Center, a quaternary children's hospital, between July 1, 2017, and June 30, 2019, were analyzed by ranking residents based on composite scores calculated using raw, unadjusted, and case-mix adjusted latent score models, with lower percentiles indicating a lower quality of care and performance. Reliability and associations between the scores produced by the 3 scoring models were compared. Resident and patient characteristics associated with performance in the highest and lowest tertiles and changes in residents' rank after case-mix adjustments were also identified. RESULTS: 274 residents and 1,891 individual encounters of bronchiolitis patients aged 0-1 as well as 270 residents and 1,752 individual encounters of asthmatic patients aged 2-21 were included in the analysis. The minimum reliability requirement to create a composite score was met for asthma data (α = 0.77), but not bronchiolitis (α = 0.17). The asthma composite scores showed high correlations ( r = 0.90-0.99) between raw, latent, and adjusted composite scores. After case-mix adjustments, residents' absolute percentile rank shifted on average 10 percentiles. Residents who dropped by 10 or more percentiles were likely to be more junior, saw fewer patients, cared for less acute and younger patients, or had patients with a longer emergency department stay. CONCLUSIONS: For some clinical areas, it is possible to use EHR data, adjusted for patient complexity, to meaningfully assess residents' clinical performance and identify opportunities for quality improvement.

What is Transferred and How Much is Retained? A Simulation Study of Complex Surgical Skills.

INTRODUCTION: Studies indicate that learning surgical skills on low-fidelity models is equally beneficial to learning on high-fidelity models in terms of skills retention and transfer. However, it is unclear how low-fidelity simulation training impacts retention and transfer in novice learners, particularly on complex surgical tasks that incorporate multiple challenging skills. This study explores the capacity of complete novices to learn and transfer complex surgical skills from a low-fidelity model to a high-fidelity simulation after a delay. METHODS: Task-naïve medical and nonmedical undergraduate students (n = 62) participated in a three-phase prospective double-arm randomized (2:1) experimental study. Participants completed two skills training sessions (end-to-side anastomosis) on a low-fidelity bench model. After a 4-week delay, participants completed the task again either using the low-fidelity model or a high-fidelity model (cadaver) and were assessed using a validated checklist. RESULTS: There was a significant time × fidelity group interaction (P = 0.004). Simple effects analysis indicated the high-fidelity group (Mdiff = 4.18, P < 0.001) performed significantly worse (P = 0.003) in phase 3 relative to phase 2 compared to the low-fidelity group (Mdiff = 0.75, P = 0.39). Post hoc logistic regression analysis indicated that radial suturing technique and economy of motion skills were less likely to be completed correctly for those in the high-fidelity group. CONCLUSIONS: These findings suggest that for novice populations, relying on low-fidelity simulation training as a source of teaching complex skills may not provide a reliable transfer to high-fidelity models and in turn clinical settings.

The Interactive Relationship Between Instructor Perceptions and Learner Personality on Surgical Skills Performance.

OBJECTIVE: The objective of this study was to examine the association between learner personality and capacity to be trained (i.e., performance improvement) on a surgical task, and how instructor perceptions of the learners' capacity to be trained interact with learner personality and performance during training and feedback. There is meaningful heterogeneity in the degree of learner surgical skills acquisition despite receiving the same amount of training. While learner personality may independently contribute to skill acquisition, the instructor-learner feedback process is also important to consider. To better understand this interpersonal relationship, it is necessary to also consider instructor factors (i.e., perceptions), and how this may contribute to learner variability in skills training. DESIGN: This exploratory study employed a prospective two-phase design. Medical and non-medical undergraduate students (N = 62) completed measures of personality and participated in two 20-minute training sessions with expert feedback 2 weeks apart, performing an end-to-side anastomosis on a low-fidelity model. Learner performance and instructors' perceptions of a learner's capacity to be trained were assessed. PARTICIPANTS: Sixty-two medical and non-medical undergraduate students. RESULTS: There was a significant interaction between learner Extraversion and instructor's perceptions of learner capacity to be trained. Higher learner Extraversion was only associated with an increase in performance improvement for those who were considered trainable (OR = 4.83, p = 0.017). Post hoc analysis revealed a significant difference in the amount of feedback provided to participants who were considered trainable (M = 9.45) versus not trainable (M = 16.48). CONCLUSIONS: This study highlights the importance of both individual learner factors and instructor perceptions on surgical skill acquisition.

Chinese university students' conceptions of feedback and the relationships with self-regulated learning, self-efficacy, and English language achievement.

In China, under the influence of examination-driven culture and teacher-centered ways of learning, students' self-regulated learning (SRL) capabilities, self-efficacy, and actual English proficiency are greatly hindered. Given this situation, the Chinese Ministry of Education has promulgated the use of formative assessment in the College English curriculum at the tertiary level since 2004. Feedback, as an integrated part of formative assessment, takes up the largest proportion of the Chinese College English classroom assessment and intends to facilitate SRL and learning. However, whether feedback could facilitate students' SRL and learning has not been fully investigated in this context in China. Therefore, this study first explored how students self-reported their conceptions of feedback, SRL, and self-efficacy, and second, the relationships among these constructs and their English language achievement in the College English course. A questionnaire was used to collect data on students' conceptions of feedback, SRL, self-efficacy, and self-perceived English language proficiency. Their English test scores as an indicator of English language achievement were also collected. A total of 538 participants from a university in Northern China participated in this study. Data were analyzed using descriptive statistics, exploratory factor analyses, Pearson correlation analyses, and multiple regression analyses. The results found that Chinese students from the College English course reported a high level of conceptions of teacher and peer feedback, SRL, and self-efficacy, yet a low level of Teacher/Peer Feedback Ignored. For the relationships among these variables, students' conceptions of feedback contributed to SRL and self-efficacy. Besides, self-efficacy was found to be the strongest predictor for self-perceived English language proficiency and standardized English test scores, both indicators for English language achievement. From the theoretical perspective, this study addressed the research gap in the literature by examining four constructs together, that is, students' conceptions of feedback, SRL, self-efficacy, and English language achievement within a university context in China. From the pedagogical angle, the results can also support teachers in their feedback practices to facilitate students' SRL, self-efficacy, and learning.

Feedback Credibility in Healthcare Education: a Systematic Review and Synthesis.

PURPOSE: The purpose of this study was to systematically review and synthesize factors that influence learners' perceptions of credibility when feedback is provided by an authority figure in a healthcare environment. METHODS: This study reviewed literature from medicine, psychology, and education using systematic review and qualitative synthesis methods. In a multi-step process, major electronic bibliographic databases were searched for relevant studies until October 2020. RESULTS: The search identified 9216 articles. A total of 134 abstracts underwent full-text review. Of these, 22 articles met inclusion criteria. The studies were heterogenous and the majority utilized a qualitative design with interviews and focus groups. A few studies employed mixed methodology (n = 2) and two studies used a quantitative design. Four main themes were identified: feedback characteristics, context of feedback, source credibility, and recipient characteristics. CONCLUSION: As programs implement major educational change initiatives to create more formative assessment practices, feedback will become even more crucial. The four main themes identified are important factors that contribute to the perception of feedback credibility. While the factors are described independently, they may be viewed as interrelated and the association between these factors and feedback may be driven more by learning culture than each characteristic. SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1007/s40670-020-01167-w.

Working Towards Competence: A Novel Application of Borderline Regression to a Task-Specific Checklist for Technical Skills in Novices.

OBJECTIVE: Demonstrated competence through frequent assessment is an expected goal for progressive development in competency-based medical education curricula. The Objective Structured Assessment of Technical Skill (OSATS) is considered a valid method of formative assessment, but in few instances have standards been set for determining competence. The present study used borderline regression methods to examine standard setting of performance on a complex technical task with novices assessed using an OSATS checklist. METHODS: This was a single institution prospective single arm experimental design study. Participants were 58 non-medical undergraduate students with no previous surgical experience, who observed a computer-based training module on end-to-side vascular anastomosis. Subsequently, participants were provided two 20-minute training sessions, two weeks apart where they received expert feedback whilst performing the task on a low-fidelity model. After each training session, participants completed the task unaided. Sessions were recorded and assessed using an OSATS checklist retrospectively by experts. RESULTS: Paired t-test analyses indicate that for both the checklist total score (t(52) = 8.05, p < 0.001) and the global rating score (t(53) = 8.15, p < 0.001), individuals performed significantly better in Phase 2. Borderline regression analyses indicated that in Phase 1 (R2 = .60) and Phase 2 (R2 = .75), the OSATS checklist could adequately capture variation in performance in novices. Further, the checklist could reliably classify novices at three of the five global rating performance levels. Pass rates determined by regression equations improved from Phase 1 to Phase 2 on all global rating levels. CONCLUSIONS: With the increasing focus on competency-based medical education, it is imperative that training programs have the capacity to accurately assess outcomes and set minimum performance standards. Borderline regression methods can accurately differentiate novice learners of varying performance levels before and after training on a complex technical skill task using an OSATS checklist.

Factors Associated with a Label of Failure to Cope in Older Medical Inpatients: a Case-Control Study.

BACKGROUND: The term failure to cope (FTC) is often used to dismissively describe hospitalized older adults. The purpose of this study was to identify the factors associated with receiving a label of FTC. METHODS: Age-matched, case-control study with electronic and paper chart review identifying patient characteristics and admission details. RESULTS: One hundred eighty-five patients 70 years of age or older admitted to a general medicine team over two years: 99 patients with the label of FTC and 86 controls. No patients labelled with FTC came from long-term care. Characteristics associated with a label of FTC included living alone (aOR 3.8, 95% CI 1.9-7.8), falls (aOR 3.8, 95% CI 1.9-7.8), rehospitalization (aOR 3.6, 95% CI 1.7-8.0), and living in an independent dwelling (aOR 2.4, 95% CI 1.0-5.5). A higher number of chronic medications was associated with a lower likelihood of being labelled with FTC (aOR 0.9, 95% CI 0.8-1.0). CONCLUSIONS: The results suggest that FTC is a label based predominantly on social factors and has no role in a medical assessment. The patient's home setting was the key factor in being labelled with FTC, most medical factors did not play a significant role, and a pervasive language of blame was present.

"Doing it Right" Overnight: a Multi-perspective Qualitative Study Exploring Senior Medical Resident Overnight Call.

BACKGROUND: Competency-based medical education (CBME) requires the development of workplace-based assessment tools that are grounded in authentic clinical work. Developing such tools, however, requires a deep understanding of the underlying facets of the competencies being assessed. Gaining this understanding remains challenging in contexts where performance is not readily visible to supervisors such as the senior medical resident (SMR) on-call role in internal medicine. OBJECTIVE: This study draws on the perspectives of healthcare professionals with whom the SMR interacts with overnight to generate insights into the different components of on-call SMR practice and the range of ways SMRs effectively and less effectively enact these. APPROACH: We used a constructivist grounded theory (CGT) approach to examine variation in how on-call SMRs carry out their role overnight. PARTICIPANTS: Six medical students, five junior residents, five internal medicine attending physicians, five emergency physicians, and three emergency nurses conducted observations of their on-call interactions with SMRs. Participants were then interviewed and asked to elaborate on their observations as well as provide comparative reflections on the practices of past SMRs they worked with. KEY RESULTS: Strong collaboration and organizational skills were identified as critical components to effectively being the on-call SMR. Perceived weaker SMRs, while potentially also having issues with clinical skills, stood out more when they could not effectively manage the realities of collaboration in a busy workplace. CONCLUSION: What consistently differentiated a perceived effective SMR from a less effective SMR was someone who was equipped to manage the realities of interprofessional collaboration in a busy workplace. Our study invites medical educators to consider what residents, particularly those in more complex roles, need to receive feedback on to support their development as physicians. It is our intention that the findings be used to inform the ways programs approach teaching, assessment, and the provision of feedback.

Unusual suspects: Real-time physiological evaluation of stressors during laparoscopic donor nephrectomy.

INTRODUCTION: The purpose of this study was to document the variability of faculty surgeon electrodermal activity (EDA) peaks during laparoscopic donor nephrectomy (LDN) to determine the effect of case difficulty and learner expertise on the stress response. METHODS: EDA for a single faculty surgeon was captured over 15 LDN cases using an Empatica E4 wristband. During each case, one of three transplant fellows (novice, intermediate, or expert level LDN expertise) participated. Difficulty was rated preoperatively as "low/moderate/high" by the faculty. EDA peaks were collected and analyzed; the frequency and magnitude of EDA peaks, case difficulty, and fellow expertise were compared using a two-way factorial ANOVA. RESULTS: The main effects of learner expertise (F[2, 308]=11.27, p<0.001) and difficulty rating (F[2, 414]=15.13, p<0.001) were significant. The interaction between difficulty and expertise on faculty EDA peaks was also significant (F[3, 391]=14.29, p<0.001). The novice fellow resulted in higher faculty EDA levels compared to intermediate and expert fellows on low-difficulty cases, but not moderate- or high-difficulty cases. CONCLUSIONS: This is the first report examining faculty surgeon EDA across cases of varying difficulty and varying learner expertise during a high-stakes operation. EDA levels were inversely proportional to the expertise of the learner and case difficulty, suggestive of a significant impact of learner autonomy on faculty stress response.

The erosion of ambiguity tolerance and sustainment of perfectionism in undergraduate medical training: results from multiple samplings of a single cohort.

BACKGROUND: Medicine is a field that is simultaneously factual and ambiguous. Medical students have their first exposure to full time clinical practice during clerkship. While studies have examined medical trainees' tolerance of ambiguity (TOA), the extent to which TOA is affected by clinical experiences and its association with perfectionism is unknown. The aim of this study was to evaluate the effect of clerkship experience on TOA and perfectionism in medical students. METHODS: This was a multiple sampling, single cohort study of students in their first year of clinical clerkship which is comprised of 6 core rotations. Consenting students completed an online anonymous survey assessing their tolerance of ambiguity (TOA) and perfectionism in their first (pre) and last (post) 12 weeks of their clinical clerkship year. Tolerance of Ambiguity in Medical Students and Doctors (TAMSAD) and The Big Three perfectionism scale-short form (BTPS-SF) were used to assess TOA and perfectionism respectively. Pre-Post mean comparisons of TOA and perfectionism were assessed via t-tests. RESULTS: From a cohort of 174 clinical clerkship students, 51 students responded to pre-survey, 62 responded to post-survey. Clerkship was associated with a significant decrease in TOA (p < 0.00) with mean pre-TOA scores of 59.57 and post TOA of 43.8. Perfectionism scores were not significantly different over time (p > 0.05). There was a moderate inverse correlation between TOA and perfectionism before clerkship (r = 0.32) that increased slightly after clerkship (r = 0.39). Those preferring primary care specialties had significantly lower rigid and total perfectionism scores in pre-clerkship than those choosing other specialties, but this difference was not found post-clerkship. CONCLUSION: Exposure to clerkship decreased TOA while perfectionism remained stable in medical students. These results were not expected as exposure has been previously shown to increase TOA. The frequency of rotation changes maintaining a cycle of anxiety may be an underlying factor accounting for these results. Overall these results require further investigation to better characterize the role of clinical exposure on TOA.

Social Accountability Frameworks and Their Implications for Medical Education and Program Evaluation: A Narrative Review.

PURPOSE: Medical schools face growing pressures to produce stronger evidence of their social accountability, but measuring social accountability remains a global challenge. This narrative review aimed to identify and document common themes and indicators across large-scale social accountability frameworks to facilitate development of initial operational constructs to evaluate social accountability in medical education. METHOD: The authors searched 5 electronic databases and platforms and the World Wide Web to identify social accountability frameworks applicable to medical education, with a focus on medical schools. English-language, peer-reviewed documents published between 1990 and March 2019 were eligible for inclusion. Primary source social accountability frameworks that represented foundational values, principles, and parameters and were cited in subsequent papers to conceptualize social accountability were included in the analysis. Thematic synthesis was used to describe common elements across included frameworks. Descriptive themes were characterized using the context-input-process-product (CIPP) evaluation model as an organizational framework. RESULTS: From the initial sample of 33 documents, 4 key social accountability frameworks were selected and analyzed. Six themes (with subthemes) emerged across frameworks, including shared values (core social values of relevance, quality, effectiveness, and equity; professionalism; academic freedom and clinical autonomy) and 5 indicators related to the CIPP model: context (mission statements, community partnerships, active contributions to health care policy); inputs (diversity/equity in recruitment/selection, community population health profiles); processes (curricular activities, community-based clinical training opportunities/learning exposures); products (physician resource planning, quality assurance, program evaluation and accreditation); and impacts (overall improvement in community health outcomes, reduction/prevention of health risks, morbidity/mortality of community diseases). CONCLUSIONS: As more emphasis is placed on social accountability of medical schools, it is imperative to shift focus from educational inputs and processes to educational products and impacts. A way to begin to establish links between inputs, products, and impacts is by using the CIPP evaluation model.

A Comparison of Guided Video Reflection versus Self-Regulated Learning to Teach Knot Tying to Medical Students: A Pilot Randomized Controlled Trial.

OBJECTIVES: Self-regulated learning has been proposed as a resource saving alternative for learning knot tying. However, this may be hindered by the Dunning-Kruger effect. A potential alternative is guided video reflection. The objectives of this study are to compare the performance and self-assessment abilities amongst medical students learning knot tying using either a traditional self-regulated approach versus guided video reflection. DESIGN: This randomized, single-blinded, controlled trial used a pre-post-retention test design. All knot tying performances were video recorded and assessed nonsequentially by blinded evaluators using a modified Objective Structured Assessment of Technical Skills tool. PARTICIPANTS: This study recruited 31 first- and second-year medical students and 6 senior urology residents from Western University in Canada. RESULTS: At baseline, the performances of the experts were significantly higher than the experimental groups (F3,85 = 9.080, p < 0.001). After the intervention, there was a significant increase in the performance for both experimental groups compared to the pretest period (p < 0.001). The scores between the experimental groups were not significantly different (p = 0.338). The improved performances of both groups were sustained on retention testing (p < 0.001). The self-assessment abilities were accurate for both experimental groups at baseline. However, at the post-test period the accuracy was poor (interclass correlation 0.361) for the self-regulated group, while remaining moderately (interclass correlation 0.685) accurate for the reflection group. CONCLUSIONS: Students using guided video reflection were able to achieve competency and maintained their knot tying skills to the same degree as those who used the self-regulated approach. These results may be due to the positive effects of reflection on self-assessment abilities and subsequent improvement in goal setting for further practice.

Journey into uncertainty: Medical students' experiences and perceptions of failure.

CONTEXT: Having succeeded in being selected for medical school, medical students are not always familiar with failure and yet they are expected to graduate prepared to effectively function in the failure-burdened arena of clinical medicine. Lacking in the developing literature on learners and failure is an exploration of how this transformation is accomplished. The purpose of this study was to examine how medical students perceive and experience failure during their medical school training. METHOD: We used a qualitative description methodology to probe the failure experiences of medical students attending a Canadian medical school. Participants were provided with the broad definition of failure used in this research: 'deviation from expected and desired results.'In total, 12 students were sampled, three from each of the 4 years of study, and participated in individual, semi-structured interviews that were analysed using thematic analysis to identify and describe core themes. RESULTS: At the start of medical school, students admitted limited experience with failure; their early descriptions were self-centred and binary. Personal stories recounted by preceptors encouraged students and helped them understand that physicians are human and that failure is inevitable. Students felt relatively protected from failures that could impact patients. Both witnessing and participating in a failure event were distressing and sometimes at odds with their expectations. Students expressed a desire to talk about the experience. CONCLUSIONS: Medical students described examples of experiencing failure during medical school that transported them from the more certain black and white beginnings of their classroom into the uncertain shades of grey of clinical medicine. What the participants heard, saw and experienced suggests opportunities for classroom teachers to better prepare pre-clinical students for the inevitability of failure in clinical medicine and opportunities for clinical teachers to engage in open, inclusive conversations surrounding failures that occur on their watch.

Impressions on Reliability and Students' Perceptions of Learning in a Peer-Based OSCE.

BACKGROUND: Peer assessment of performance in the objective structured clinical examination (OSCE) is emerging as a learning instrument. While peers can provide reliable scores, there may be a trade-off with students' learning. The purpose of this study is to evaluate a peer-based OSCE as a viable assessment instrument and its potential to promote learning and explore the interplay between these two roles. METHODS: A total of 334 medical students completed an 11-station OSCE from 2015 to 2016. Each station had 1-2 peer examiners (PE) and one faculty examiner (FE). Examinees were rated on a 7-point scale across 5 dimensions: Look, Feel, Move, Special Tests and Global Impression. Students participated in voluntary focus groups in 2016 to provide qualitative feedback on the OSCE. Authors analysed assessment data and transcripts of focus group discussions. RESULTS: Overall, PE awarded higher ratings compared with FE, sources of variance were similar across 2 years with unique variance consistently being the largest source, and reliability (r φ ) was generally low. Focus group analysis revealed four themes: Conferring with Faculty Examiners, Difficulty Rating Peers, Insider Knowledge, and Observing and Scoring. CONCLUSIONS: While peer assessment was not reliable for evaluating OSCE performance, PE's perceived that it was beneficial for their learning. Insight gained into exam technique and self-appraisal of skills allows students to understand expectations in clinical situations and plan approaches to self-assessment of competence.

Do Factors from Admissions and Dental School Predict Performance on National Board Exams? A Multilevel Modeling Study.

The aim of this study was to assess the association among admissions variables, dental school performance, and licensing exam performance for six cohorts of graduates of one dental school. Data from all dental school graduates of Schulich School of Medicine & Dentistry, Western University, Canada, from 2009 to 2014 who had matching National Dental Examining Board of Canada (NDEB) data (N=298) were analyzed. In the results, significant differences between cohorts were found on both the NDEB objective structured clinical examination (OSCE) and written scores. Approximately 18% of the variation in OSCE scores was attributable to cohort differences and 82% to student differences. Approximately 10% of the variation in written scores was attributable to cohort differences and 90% to student differences. Several multilevel models were conducted. The final predictive model for NDEB OSCE scores consisted of age, Canadian Dental Aptitude Test (DAT) reading comprehension scores, year 2 average, and year 4 average. For predicting NDEB written exam scores, the final model consisted of DAT chemistry and year 1, 2, and 4 averages. The findings of this study showed that academic performance on admissions variables and in training predicted performance on dental licensing exams, whereas variables that captured noncognitive or interpersonal skills, such as interview scores, were not predictive. This difference may be due to construct mismatch, such that the outcome variables had no theoretical association with the predictors. Additional outcome measures (including noncognitive) are needed that have greater ecological validity in predicting potential for competence in practice.

Some assembly required: tracing the interpretative work of Clinical Competency Committees.

OBJECTIVES: This qualitative study describes the social processes of evidence interpretation employed by Clinical Competency Committees (CCCs), explicating how they interpret, grapple with and weigh assessment data. METHODS: Over 8 months, two researchers observed 10 CCC meetings across four postgraduate programmes at a Canadian medical school, spanning over 25 hours and 100 individual decisions. After each CCC meeting, a semi-structured interview was conducted with one member. Following constructivist grounded theory methodology, data collection and inductive analysis were conducted iteratively. RESULTS: Members of the CCCs held an assumption that they would be presented with high-quality assessment data that would enable them to make systematic and transparent decisions. This assumption was frequently challenged by the discovery of what we have termed 'problematic evidence' (evidence that CCC members struggled to meaningful interpret) within the catalogue of learner data. When CCCs were confronted with 'problematic evidence', they engaged in lengthy, effortful discussions aided by contextual data in order to make meaning of the evidence in question. This process of effortful discussion enabled CCCs to arrive at progression decisions that were informed by, rather than ignored, problematic evidence. CONCLUSIONS: Small groups involved in the review of trainee assessment data should be prepared to encounter evidence that is uncertain, absent, incomplete, or otherwise difficult to interpret, and should openly discuss strategies for addressing these challenges. The answer to the problem of effortful processes of data interpretation and problematic evidence is not as simple as generating more data with strong psychometric properties. Rather, it involves grappling with the discrepancies between our interpretive frameworks and the inescapably subjective nature of assessment data and judgement.

How Medical Error Shapes Physicians' Perceptions of Learning: An Exploratory Study.

PURPOSE: Error is inevitable in medicine, given its inherent uncertainty and complexity. Errors can teach powerful lessons; however, because of physicians' self-imposed silence and the intricacies of responsibility and blame, learning from medical error has been underexplored. The purpose of this study was to understand how physicians perceived learning from medical errors by exploring the tension between responsibility and blame and factors that affected physicians' learning. METHOD: Nineteen physicians participated in semistructured interviews, conducted in 2016-2017 at Western University in Canada, that probed their experiences in learning from medical errors. Data collection and analysis were conducted iteratively, with themes identified through constant comparative analysis. RESULTS: Participants felt personal responsibility and blame for their errors. Residency produced particularly salient memories of errors. Participants identified interconnecting cultural factors (normalizing error, peer support and mentorship, formal rounds) and individual factors (emotional response, confidence and experience), which either helped or hindered their perceived learning. CONCLUSIONS: Learning from medical error requires navigation through blame and responsibility. The keen responsibility felt by physicians must be acknowledged when enacting a system-based approach to medical error. Adopting a learning culture perspective suggests opportunities to enable and disable features of the learning environment to optimize learning from error as residents learn to become the most responsible physician for all outcomes. A better understanding of the factors that shape learning from error can help make the transition from error to learning more explicit, thereby increasing the opportunity to learn and teach from errors that permeate the practice of medicine.