Unintended consequences - Risk for bias in competency-based assessment.

As nursing education transitions to competency-based education, careful attention to the assessment methodology must be considered. As nursing education contends with structural implicit bias and supports social justice, any transition to the education delivery model must consider the underserved and marginalized. Multilingual and marginalized students have long had inequitable experiences when taking standardized assessments influenced by the perspectives and values of the dominant culture. Resourced and under-resourced nurse educators must be empowered to access information to inform the competency-based assessment design. This perspective article aims to uncover the risk of bias in competency-based assessment when not properly implemented and to present strategies to reduce bias in competency-based assessment. Considerations for schools/colleges of nursing as they implement competency-based assessment are explored.

Development of a competency-based assessment template for oncology nursing: A participatory action research study.

AIM: The main objective of this study was to collaboratively develop a competency-based assessment template for oncology nurses. BACKGROUND: The care of oncology patients and their families requires very specific knowledge and skills, as well as that nurses are trained not only in theoretical and practical knowledge but also in competencies. There is a lack of specific tools to assess competencies in oncology and it is important to include students in the learning process. DESIGN: This study has a Participatory Action Research. METHODS: A focus group was conducted for data collection. The process consisted of 4 phases: 1- Identification of the problem; 2- Planning of the change; 3- Realization of the change; 4- Effect of the change and reflection. A purposive sampling where all the students of the master's degree in oncology nursing who wished to participate voluntarily were selected. A total of 23 students participated. The information was recorded and fully transcribed. Triangulation of researchers was used during data analysis. RESULTS: One of the most important aspects included in the new tool was the need for a comprehensive approach to people with oncological diseases and their relatives. The students emphasized the need to primarily address the physical, psychological and social dimensions in this approach. CONCLUSION: This study developed a competency-based assessment template for oncology nurses, which includes the assessment of knowledge, skills and attitudes. The findings could help other oncology nurse educators or professionals to evaluate their training sessions and even their own knowledge in a healthcare setting.

Assessing the interplay of presentation and competency in online video examinations: a focus on health system science education.

BACKGROUND: The integration of Health System Science (HSS) in medical education emphasizes mastery of competencies beyond mere knowledge acquisition. With the shift to online platforms during the COVID-19 pandemic, there is an increased emphasis on Technology Enhanced Assessment (TEA) methods, such as video assessments, to evaluate these competencies. This study investigates the efficacy of online video assessments in evaluating medical students' competency in HSS. METHODS: A comprehensive assessment was conducted on first-year medical students (n = 10) enrolled in a newly developed curriculum integrating Health System Science (HSS) into the Bachelor of Medicine program in 2021. Students undertook three exams focusing on HSS competency. Their video responses were evaluated by a panel of seven expert assessors using a detailed rubric. Spearman rank correlation and the Interclass Correlation Coefficient (ICC) were utilized to determine correlations and reliability among assessor scores, while a mixed-effects model was employed to assess the relationship between foundational HSS competencies (C) and presentation skills (P). RESULTS: Positive correlations were observed in inter-rater reliability, with ICC values suggesting a range of reliability from poor to moderate. A positive correlation between C and P scores was identified in the mixed-effects model. The study also highlighted variations in reliability and correlation, which might be attributed to differences in content, grading criteria, and the nature of individual exams. CONCLUSION: Our findings indicate that effective presentation enhances the perceived competency of medical students, emphasizing the need for standardized assessment criteria and consistent assessor training in online environments. This study highlights the critical roles of comprehensive competency assessments and refined presentation skills in online medical education, ensuring accurate and reliable evaluations.

The development of Logix - An application for component-based case logging and surgical trainee assessment.

Background: The optimal training program to transform a new resident into a competent and capable surgeon is constantly evolving. Competency-based evaluation represents a change in mindset from quantitative or chronologic metrics for graduate readiness. As surgery becomes more specialized, more dependent on technology, and more public, we must continue to improve our ability to pass on technical skills. Approaching surgery in a component-based fashion enables even the most complex operation to be broken down into smaller sets of steps that range the entire spectrum of complexity. Treating an operation through the lens of its components, emphasizing stepwise forward progression in a trainee's experience, may provide a way to train competent surgeons more efficiently. Current case-logging products do not provide adequate granularity to apply this methodology. Methods: Application design relied on the involvement of local surgeons from all specialties and subspecialties related to general surgical training. Individual interviews with multiple experts in each field were used to generate a list of most commonly performed operations. Once a consensus was reached, the same surgeons were queried on what they felt were the core steps that make up each operation. This information was utilized to create a novel mobile application which enables the user to record cases by date, attending surgeon, specific operation, and which portions of the operation they were able/allowed to perform. Conclusion: Component-based case logging through the Logix application may be a useful adjunct as we continue to implement competency-based surgical training. Future investigation will assess user experience and compare subjective and objective metrics of training progression between the Logix application and currently utilized products. The information provided by the application stands to benefit not just trainees, but educators, training programs, and regulatory bodies. Key message: Component-based case logging via a novel mobile application stands to increase the efficiency of surgical training and more effectively assess trainee competency.

Assessment of Professionalism among Postgraduate Medical Students: A Cross-Sectional Study from Patna, Bihar.

BACKGROUND: Professionalism is a core quality which has to be developed and imbibed by medical students. However, training in professionalism was not a part of the medical curriculum until recently when the Competency-Based Medical Education system was introduced for undergraduate medical students in India, by the National Medical Commission. However, there has been no formal training on professionalism, in the curriculum of postgraduate (PG) medical students in India. Therefore, in this study, we have done a quantitative assessment of the attitude towards professionalism among PG medical students. METHODOLOGY: A cross-sectional study was conducted among 200 PG medical students of an autonomous institute in Patna, Bihar over a period of six months using the Learners' Attitudes on Medical Professionalism Scale (LAMPS) questionnaire. Data entry was done on Microsft Excel 2015 (Microsoft Corporation, Redmond, Washington, United States) and data analysis was done using IBM SPSS Statistics for Windows, Version 26.0 (Released 2019; IBM Corp., Armonk, New York, United States). RESULTS: Out of 200 study subjects, 55% (n=110) were PG students from clinical departments and 45% (n=90) were from pre-clinical and para-clinical departments; 25.5% (n=51) of them were from first year, 57% (n=114) of them were from second year, and 17.5% (n=35) were from third year PG training. The overall mean score for the LAMPS scale across all domains was found to be 2.78±0.178 (mean±S.D). Clinical PG students (2.99 ± 0.217) received significantly higher scores than pre/paraclinical PG students (2.68 ± 0.176) (T-test value = 10.9262, p < 0.0001). CONCLUSION: This study highlights the disparity in professionalism scores between clinical and pre/para-clinical students, suggesting a tailored approach might be beneficial. In order to improve the attitude towards professionalism, structured courses/workshops should be inculcated into the PG training curriculum. Continuous evaluation of professionalism throughout medical training can lead to better preparedness for future professional roles.

An exploration of Irish nutrition educators' experiences of competency-based assessment in nutrition science education.

BACKGROUND: Competency-based assessment (CBA) supports the development and attainment of skills required for the workforce. Little is known about educators' experience in developing or implementing CBA in nutrition science education or their opinions on how well it captures a student's preparedness for the workforce. The objective of this study was to explore educators' experience of CBA in nutrition education in Ireland. METHOD: Grounded in interpretivism, in-depth, semi-structured, audio-recorded interviews were conducted with 13 educators from five of the ten undergraduate honours degree nutrition programmes across Ireland. Interviews explored experiences of CBA and perception of students training to prepare for the workforce. A reflexive thematic analysis approach was implemented whereby the data were transcribed, inductively coded, and themes identified. RESULTS: A clear divide was evident between participants who were confident in their understanding of CBA and those who were unsure or had no knowledge of the term. Those with a clear understanding were more involved in programme development and evaluation. Three themes were identified: 'Assessment process' including intended learning outcomes, assessment design, and grading systems, 'Student-centred approach to assessment' focusing on work-based assessment and preparation for the workforce, and 'Upskilling educators' to equip educators with the skills and knowledge for professional development and to foster student success. CONCLUSION: The importance of CBA as a student-centred approach, supporting them to meet standards expected to practice as a nutrition professional, was the key experience of educators. Training in CBA and inclusion of more authentic assessment may better prepare students for the workforce.

Multi-modal recommender system for predicting project manager performance within a competency-based framework.

The evaluation of performance using competencies within a structured framework holds significant importance across various professional domains, particularly in roles like project manager. Typically, this assessment process, overseen by senior evaluators, involves scoring competencies based on data gathered from interviews, completed forms, and evaluation programs. However, this task is tedious and time-consuming, and requires the expertise of qualified professionals. Moreover, it is compounded by the inconsistent scoring biases introduced by different evaluators. In this paper, we propose a novel approach to automatically predict competency scores, thereby facilitating the assessment of project managers' performance. Initially, we performed data fusion to compile a comprehensive dataset from various sources and modalities, including demographic data, profile-related data, and historical competency assessments. Subsequently, NLP techniques were used to pre-process text data. Finally, recommender systems were explored to predict competency scores. We compared four different recommender system approaches: content-based filtering, demographic filtering, collaborative filtering, and hybrid filtering. Using assessment data collected from 38 project managers, encompassing scores across 67 different competencies, we evaluated the performance of each approach. Notably, the content-based approach yielded promising results, achieving a precision rate of 81.03%. Furthermore, we addressed the challenge of cold-starting, which in our context involves predicting scores for either a new project manager lacking competency data or a newly introduced competency without historical records. Our analysis revealed that demographic filtering achieved an average precision of 54.05% when dealing with new project managers. In contrast, content-based filtering exhibited remarkable performance, achieving a precision of 85.79% in predicting scores for new competencies. These findings underscore the potential of recommender systems in competency assessment, thereby facilitating more effective performance evaluation process.

Optimization of a standardized letter of recommendation for faculty who wish to support candidates applying to surgical training programs.

Letters of recommendation (LORs) play an important role in applicant selection for graduate medical education programs. LORs may be of increasing importance in the evaluation of applicants given the recent change of the USMLE Step 1 to pass/fail scoring and the relative lack of other objective measures by which to differentiate and stratify applicants. Narrative letters of recommendation (NLORs), although widely used, have certain limitations, namely variability in interpretation, introduction of gender/race bias, and performance inflation. Standardized letters of recommendation (SLOR) have been proposed as a potential corrective to these limitations. We conducted a series of semi-structured interviews and focus groups to gather perspectives from letter writers and readers to inform methods for improving information elicited by SLORs from which we collected and analyzed data using the constant comparative method and a process of iterative coding. We applied our findings to the development of a novel SLOR for use in surgical residency program applications and were subsequently invited to help revise existing SLORs for a surgical post-graduate training program.

Head and neck ultrasound training improves the diagnostic performance of otolaryngology residents.

Objective: Surgeon-performed head and neck ultrasound (US) is increasingly used among otolaryngologists in office-based and surgical settings. However, it is unknown how formal US training affects otolaryngology residents' diagnostic workup of patients with cervical pathology. This study examined how a formal US course for residents affected their outpatient clinic US performance and diagnostic accuracy. Methods: We conducted a randomized cross-over trial, where 13 otolaryngology residents participated in a 6-h formal US course. Participants were randomized to perform head and neck US on four patient cases before and after completing the course. Eight patients with and without neck pathology were invited to participate as test cases. The ultrasound examinations were video recorded and anonymized before two consultants rated the US performance using the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale. Otolaryngology residents wrote an ultrasound report with a diagnosis based on their US examination, which was used to calculate the specificity and sensitivity. Results: We found a statistically significant difference in the OSAUS score before compared to after the hands-on training (p = .035). The diagnostic accuracy also increased from 62% before the course to 75% after the course (p = .02). Specificity increased from 54% prior to the course to 62% following the course, and sensitivity increased from 64% prior to the course to 79% following the course. The intraclass correlation coefficient with "absolute agreement" was 0.63. Conclusion: This study demonstrates that short, formal ultrasound training can improve otolaryngology residents' ultrasound skills and diagnostic accuracy in an outpatient clinic setting. Lay summary: This study looks at the change of otolaryngology residents' diagnostic workup of patients after they take a formal ultrasound course and shows that they get better at using ultrasound and make more accurate diagnoses if they take a formal course. Level of Evidence: Level 2.

Measuring Competency: Improving the validity of your procedural performance assessments.

Objective: The objective of the study was to compare the use of ordinal scales and interval scales for capturing surgical competency information for general surgeons performing 3 complex trauma procedures. Background: Surgical performance assessment is typically captured using nonparametric data (eg, checklists) that do not support inferential analyses. Interval scales support parametric analyses that are essential for determining competency. We compared assessment outcomes for surgeons performing 3 complex trauma procedures using ordinal and interval scales. Methods: All participants were board-certified or eligible general surgeons. Each participant was assessed by an experienced trauma surgeon while performing 3 trauma procedures on cadavers. All assessors completed a rigorous assessment certification process. We calculated descriptive statistics to examine the differences between interval (parametric) and ordinal (nonparametric) outcomes. Results: Ordinal scales overestimated competence in up to 100% of the participants and did not identify specific performance gaps. Interval scales provided more granularity and identified specific capability gaps. Conclusions: Imprecise instrumentation conveys a false sense of competence and deprives surgeons of opportunities to close capability gaps. Measuring discrete procedural components with interval scales provides a more precise measurement of surgical competency.

Evaluating the Construct Validity of Competencies: A Retrospective Analysis.

Background: A competency-based framework focuses on alignment between professional standards and assessment design. This alignment implies improved measurement validity, yet it has not been established that competence in one context predicts performance in another context. High-stakes competence assessments offer insights into the relationship between assessment design and competencies. Methods/Analyses: The internationally educated nurses competency assessment program (IENCAP) was developed at Touchstone Institute in collaboration with the College of Nurses of Ontario (CNO) and includes a 12-station OSCE. Each station evaluated the same 10 competencies. We submitted competency scores to a multi-trait multi-method matrix analysis to evaluate the convergent and discriminant validity of competencies. Results/Observations: All correlations were significant and positive; however, we did not find evidence of convergent or discriminant validity. Correlations were higher between different competencies evaluated within the same station (mean correlation = 0.60) compared to identical competencies evaluated across different stations (mean correlation = 0.19). Discussion: The results do not provide evidence of construct validity for competencies. While competency-based approaches emphasize various generalized knowledge, skills, and attitudes, these findings indicate that the clinical context is a major determinant of performance. Conclusion: The context-dependent nature of competencies requires multiple assessments in varied contexts. Performance on a single competency cannot be determined in a single occasion. Supplementary Information: The online version contains supplementary material available at 10.1007/s40670-023-01794-z.

Competency Assessment of Pediatric Intensive Care Nurses.

One of the biggest challenges that clinical nurse educators at the unit level face today is measuring the ongoing competency of nursing staff members to ensure provision of high-quality, evidence-based care. Pediatric nursing leaders at an urban, level I trauma teaching institution in the southwestern United States used a shared governance approach to create a standardized competency assessment tool for pediatric intensive care unit nurses. Donna Wright's competency assessment model was used as a framework to guide the tool's development. The adoption of the standardized competency assessment tool was aligned with the organization's institutional goals and enabled clinical nurse educators to comprehensively evaluate staff members on a regular basis. This standardized competency assessment system for pediatric intensive care nurses is more effective than use of a practice-based, task-oriented assessment method and has improved nursing leaders' ability to staff the pediatric intensive care unit safely.

Development and preliminary testing of a virtual reality measurement for assessing intake assessment skills.

Objective structured clinical examinations (OSCEs) have been widely used in health care education to simultaneously assess knowledge, skill and attitude. Due to the high cost of running an OSCE, its application in professional psychology is still limited. To solve this problem, virtual standardised patient (VSP) implementations in creating psychology OSCEs can be a cost-effective method for administering a psychology OSCE regularly. This study aimed to develop and examine the psychometric properties of the VSP version of the Intake OSCE (VSP-Intake OSCE) in measuring psychologists' psychological assessment competencies (PACs) from entry to early practice. The initial development of the VSP-Intake OSCE contains a VSP station and a follow-up written station to measure PACs when conducting an intake assessment. To administer the VSP station, we built a new VSP system that allows psychologists to interact with a VSP verbally. A sample of 36 participants, including 27 graduate students and nine psychologists, were recruited to examine the psychometric properties of the VSP-Intake OSCE. As a newly developed instrument, the VSP-Intake OSCE revealed good inter-rater reliability and construct validity. We believe using VSP implementations to develop psychology OSCEs will be essential in promoting OSCE applications in professional psychology.

Teacher, Gatekeeper, or Team Member: supervisor positioning in programmatic assessment.

Competency-based assessment is undergoing an evolution with the popularisation of programmatic assessment. Fundamental to programmatic assessment are the attributes and buy-in of the people participating in the system. Our previous research revealed unspoken, yet influential, cultural and relationship dynamics that interact with programmatic assessment to influence success. Pulling at this thread, we conducted secondary analysis of focus groups and interviews (n = 44 supervisors) using the critical lens of Positioning Theory to explore how workplace supervisors experienced and perceived their positioning within programmatic assessment. We found that supervisors positioned themselves in two of three ways. First, supervisors universally positioned themselves as a Teacher, describing an inherent duty to educate students. Enactment of this position was dichotomous, with some supervisors ascribing a passive and disempowered position onto students while others empowered students by cultivating an egalitarian teaching relationship. Second, two mutually exclusive positions were described-either Gatekeeper or Team Member. Supervisors positioning themselves as Gatekeepers had a duty to protect the community and were vigilant to the detection of inadequate student performance. Programmatic assessment challenged this positioning by reorientating supervisor rights and duties which diminished their perceived authority and led to frustration and resistance. In contrast, Team Members enacted a right to make a valuable contribution to programmatic assessment and felt liberated from the burden of assessment, enabling them to assent power shifts towards students and the university. Identifying supervisor positions revealed how programmatic assessment challenged traditional structures and ideologies, impeding success, and provides insights into supporting supervisors in programmatic assessment.

Emergency medical services Milestones 2.0: What has changed?

Background: Since 2015, development of competencies by emergency medical services (EMS) fellows have been evaluated using the EMS Milestones 1.0 developed by a working group consisting of relevant stakeholders convened by the Accreditation Council for Graduate Medical Education (ACGME). Feedback from users and data collected from the milestones assessments in the interim indicated a need for revision of the original milestones. In May 2021, the Milestones 2.0 working group was convened for the purpose of revising this specialty-specific assessment tool. Methods: A working group consisting of representatives from American Board of Emergency Medicine, the Review Committee for Emergency Medicine, and volunteers selected by the ACGME Milestones Committee, chaired by the ACGME vice president for milestones development, was convened using a virtual platform to revise the milestones and develop a supplemental guide for use along with the Milestones 2.0. There were no in-person meetings of this working group due to the COVID-19 pandemic. Results: Data from milestones reporting, discussion within the working group, stakeholder input, and public commentary were used to revise the original milestones. A new supplemental guide to enhance milestone usability and provide recommended resource materials was also developed for use alongside the milestones. Discussion: The EMS Milestones 2.0 and accompanying supplemental guide provide an updated framework for fellowship programs to use as a guide for developing the competencies necessary for independent practice as EMS physicians and in the formal, competency-based evaluation of trainees as required by the ACGME.

Design guidelines for assessing students' interprofessional competencies in healthcare education: a consensus study.

INTRODUCTION: Healthcare systems require healthcare professionals and students educated in an interprofessional (IP) context. Well-designed assessments are needed to evaluate whether students have developed IP competencies, but we currently lack evidence-informed guidelines to create them. This study aims to provide guidelines for the assessment of IP competencies in healthcare education. METHODS: A qualitative consensus study was conducted to establish guidelines for the design of IP assessments using the nominal group technique. First, five expert groups (IP experts, patients, educational scientists, teachers, and students) were asked to discuss design guidelines for IP assessment and reach intra-group consensus. Second, one heterogeneous inter-group meeting was organized to reach a consensus among the expert groups on IP assessment guidelines. RESULTS: This study yielded a comprehensive set of 26 guidelines to help design performance assessments for IP education: ten guidelines for both the IP assessment tasks and the IP assessors and six guidelines for the IP assessment procedures. DISCUSSION: The results showed that IP assessment is complex and, compared to mono-professional assessment, high-quality IP assessments require additional elements such as multiple IP products and processes to be assessed, an IP pool of assessors, and assessment procedures in which standards are included for the IP collaboration process as well as individual contributions. The guidelines are based on expert knowledge and experience, but an important next step is to test these design guidelines in educational practice.

Development of a Bronchoscopy-Radiologic Skills and Task Assessment Tool (BRadSTAT): A Tool for Evaluating the Radiological Skills of Bronchoscopists with Different Experience.

BACKGROUND: Competency using radiologic images for bronchoscopic navigation is presumed during subspecialty training, but no assessments objectively measure combined knowledge of radiologic interpretation and ability to maneuver a bronchoscope into peripheral airways. OBJECTIVES: The objectives of this study were (i) to determine whether the Bronchoscopy-Radiology Skills and Tasks Assessment Tool (BRadSTAT) discriminates between bronchoscopists of various levels of experience and (ii) to improve construct validity using study findings. METHODS: BRadSTAT contains 10 questions that assess chest X-ray and CT scan interpretation using multiple images per question and 2 technical skill assessments. After administration to 33 bronchoscopists (5 Beginners, 9 Intermediates, 10 Experienced, and 9 Experts), discriminative power was strengthened using differential weighting on CT-related questions, producing the BRadSTAT-CT score. Cut points for both scores were determined via cross-validation. RESULTS: Mean BRadSTAT scores for Beginner, Intermediate, Experienced, and Expert were 74 (±13 SD), 78 (±14), 86 (±9), and 88 (±8), respectively. Statistically significant differences were noted between Expert and Beginner, Expert and Intermediate, and Experienced and Beginner (all p ≤ 0.05). Mean BRadSTAT-CT scores for Beginner, Intermediate, Experienced, and Expert were 63 (±14), 74 (±15), 82 (±13), and 90 (±9), respectively, all statistically significant (p ≤ 0.03). Cut points for BRadSTAT-CT had lower sensitivity but greater specificity and accuracy than for BRadSTAT. CONCLUSION: BRadSTAT represents the first validated assessment tool measuring knowledge and skills for bronchoscopic access to peripheral airways, which discriminates between bronchoscopists of various experience levels. Refining BRadSTAT produced the BRadSTAT-CT, which had higher discriminative power. Future studies should focus on their usefulness in competency-based bronchoscopy programs.

Competency-Based Assessment in Experiential Learning in Undergraduate Pharmacy Programmes: Qualitative Exploration of Facilitators' Views and Needs (ACTp Study).

Newly registered pharmacists will need to possess higher-level competencies and, in Great Britain, there is an expectation that assessments are undertaken during experiential learning (EL). The aim of this study was to explore the perceptions and educational needs of practice-based EL facilitators of student pharmacists, undertaking competency-based assessments during EL. Semi-structured one-on-one interviews were conducted with EL facilitators working in the community, hospital, and primary-care pharmacies. Data were thematically analysed. Fifteen facilitators were interviewed, and there were five from each site. There was general support for this role, but also anxiety due to the lack of knowledge about assessments and the repercussions on students. Benefits were that students would receive real-time feedback from workplace-based practitioners and facilitators would benefit from self-development. Challenges included additional workload and lack of consistency in marking. The majority agreed that clinical, professional, and communication skills could be assessed; however, a consensus was not reached regarding the tools, methods, and grading of assessments. The need for training and support were highlighted. A co-design method was proposed to ensure that the assessment methods and processes are accepted by all stakeholders. Training and resources should be tailored to the needs of facilitators.

Pre-clerkship EPA assessments: a thematic analysis of rater cognition.

BACKGROUND: Entrustable Professional Activities (EPAs) assessments measure learners' competence with an entrustment or supervisory scale. Designed for workplace-based assessment EPA assessments have also been proposed for undergraduate medical education (UME), where assessments frequently occur outside the workplace and may be less intuitive, raising validity concerns. This study explored how assessors make entrustment determinations in UME, with additional specific comparison based on familiarity with prior performance in the context of longitudinal student-assessor relationships. METHODS: A qualitative approach using think-alouds was employed. Assessors assessed two students (familiar and unfamiliar) completing a history and physical examination using a supervisory scale and then thought-aloud after each assessment. We conducted a thematic analysis of assessors' response processes and compared them based on their familiarity with a student. RESULTS: Four themes and fifteen subthemes were identified. The most prevalent theme related to "student performance." The other three themes included "frame of reference," "assessor uncertainty," and "the patient." "Previous student performance" and "affective reactions" were subthemes more likely to inform scoring when faculty were familiar with a student, while unfamiliar faculty were more likely to reference "self" and "lack confidence in their ability to assess." CONCLUSIONS: Student performance appears to be assessors' main consideration for all students, providing some validity evidence for the response process in EPA assessments. Several problematic themes could be addressed with faculty development while others appear to be inherent to entrustment and may be more challenging to mitigate. Differences based on assessor familiarity with student merits further research on how trust develops over time.