Usability of ENTRUST as an Assessment Tool for Entrustable Professional Activities (EPAs): A Mixed Methods Analysis.

OBJECTIVE: As the American Board of Surgery transitions to a competency-based model of surgical education centered upon entrustable professional activities (EPAs), there is a growing need for objective tools to determine readiness for entrustment. This study evaluates the usability of ENTRUST, an innovative virtual patient simulation platform to assess surgical trainees' decision-making skills in preoperative, intra-operative, and post-operative settings. DESIGN: This is a mixed-methods analysis of the usability of the ENTRUST platform. Quantitative data was collected using the system usability scale (SUS) and Likert responses. Analysis was performed with descriptive statistics, bivariate analysis, and multivariable linear regression. Qualitative analysis of open-ended responses was performed using the Nielsen-Shneiderman Heuristics framework. SETTING: This study was conducted at an academic institution in a proctored exam setting. PARTICIPANTS: The analysis includes n = 47 (PGY 1-5) surgical residents who completed an online usability survey following the ENTRUST Inguinal Hernia EPA Assessment. RESULTS: The ENTRUST platform had a median SUS score of 82.5. On bivariate and multivariate analyses, there were no significant differences between usability based on demographic characteristics (all p > 0.05), and SUS score was independent of ENTRUST performance (r = 0.198, p = 0.18). Most participants agreed that the clinical workup of the patient was engaging (91.5%) and felt realistic (85.1%). The most frequent heuristics represented in the qualitative analysis included feedback, visibility, match, and control. Additional themes of educational value, enjoyment, and ease-of-use highlighted participants' perspectives on the usability of ENTRUST. CONCLUSIONS: ENTRUST demonstrates high usability in this population. Usability was independent of ENTRUST score performance and there were no differences in usability identified in this analysis based on demographic subgroups. Qualitative analysis highlighted the acceptability of ENTRUST and will inform ongoing development of the platform. The ENTRUST platform holds potential as a tool for the assessment of EPAs in surgical residency programs.

Correlation of Performance on ENTRUST and Traditional Oral Objective Structured Clinical Examination for High-Stakes Assessment in the College of Surgeons of East, Central, and Southern Africa.

BACKGROUND: To address the global need for accessible evidence-based tools for competency-based education, we developed ENTRUST, an innovative online virtual patient simulation platform to author and securely deploy case scenarios to assess surgical decision-making competence. STUDY DESIGN: In partnership with the College of Surgeons of East, Central, and Southern Africa, ENTRUST was piloted during the Membership of the College of Surgeons (MCS) 2021 examination. Examinees (n = 110) completed the traditional 11-station oral objective structured clinical examinations (OSCEs), followed by 3 ENTRUST cases, authored to query similar clinical content of 3 corresponding OSCE cases. ENTRUST scores were analyzed for associations with MCS Examination outcome using independent sample t tests. Correlation of ENTRUST scores to MCS Examination Percentage and OSCE station scores was calculated with Pearson correlations. Bivariate and multivariate analyses were performed to evaluate predictors of performance. RESULTS: ENTRUST performance was significantly higher in examinees who passed the MCS examination compared with those who failed (p < 0.001). The ENTRUST score was positively correlated with MCS Examination Percentage (p < 0.001) and combined OSCE station scores (p < 0.001). On multivariate analysis, there was a strong association between MCS Examination Percentage and ENTRUST Grand Total Score (p < 0.001), Simulation Total Score (p = 0.018), and Question Total Score (p < 0.001). Age was a negative predictor for ENTRUST Grand Total and Simulation Total Score, but not for Question Total Score. Sex, native language status, and intended specialty were not associated with performance on ENTRUST. CONCLUSIONS: This study demonstrates feasibility and initial validity evidence for the use of ENTRUST in a high-stakes examination context for assessment of surgical decision-making. ENTRUST holds potential as an accessible learning and assessment platform for surgical trainees worldwide.

Risk of Permanent Hypoparathyroidism Requiring Calcitriol Therapy in a Population-Based Cohort of Adults Older Than 65 Undergoing Total Thyroidectomy for Graves' Disease.

Objective: Total thyroidectomy for Graves' disease (GD) is associated with rapid treatment of hyperthyroidism and low recurrence rates. However, it carries the risk of surgical complications including permanent hypoparathyroidism, which contributes to long-term impaired quality of life. The objective of this study was to determine the incidence of permanent hypoparathyroidism requiring calcitriol therapy among a population-based cohort of older adults undergoing total thyroidectomy for GD in the United States. Methods: We performed a population-based cohort study using 100% Medicare claims from beneficiaries older than 65 years with GD who underwent total thyroidectomy from 2007 to 2017. We required continuous enrollment in Medicare Parts A, B, and D for 12 months before and after surgery to ensure access to comprehensive claims data. Patients were excluded if they had a preoperative diagnosis of thyroid cancer or were on long-term preoperative calcitriol. Our primary outcome was permanent hypoparathyroidism, which was identified based on persistent use of calcitriol between 6 and 12 months following thyroidectomy. We used multivariable logistic regression to identify characteristics associated with permanent hypoparathyroidism, including patient age, sex, race/ethnicity, neighborhood disadvantage, Charlson-Deyo Comorbidity Index, urban or rural residence, and frailty. Results: We identified 4650 patients who underwent total thyroidectomy for GD during the study period and met the inclusion criteria (mean age = 72.8 years [standard deviation = 5.5], 86% female, and 79% white). Among this surgical cohort, 104 (2.2% [95% confidence interval, CI = 1.8-2.7%]) patients developed permanent hypoparathyroidism requiring calcitriol therapy. Patients who developed permanent hypoparathyroidism were on average older (mean age 74.1 vs. 72.8 years) than those who did not develop permanent hypoparathyroidism (p = 0.04). On multivariable regression, older age was the only patient characteristic associated with permanent hypoparathyroidism (odds ratio age ≥76 years = 1.68 [CI = 1.13-2.51] compared with age 66-75 years). Conclusions: The risk of permanent hypoparathyroidism requiring calcitriol therapy among this national, U.S. population-based cohort of older adults with GD treated with total thyroidectomy was low, even when considering operations performed by a heterogeneous group of surgeons. These findings suggest that the risk of hypoparathyroidism should not be a deterrent to operative management for GD in older adults who are appropriate surgical candidates.

Adverse Cardiovascular Outcomes Among Older Adults With Primary Hyperparathyroidism Treated With Parathyroidectomy Versus Nonoperative Management.

OBJECTIVE: The authors sought to compare the incidence of adverse cardiovascular (CV) events in older adults with primary hyperparathyroidism (PHPT) treated with parathyroidectomy versus nonoperative management. BACKGROUND: PHPT is a common endocrine disorder that is associated with increased CV mortality, but it is not known whether parathyroidectomy reduces the incidence of adverse CV events. METHODS: The authors conducted a population-based, longitudinal cohort study of Medicare beneficiaries diagnosed with PHPT (2006-2017). Multivariable, inverse probability weighted Cox proportional hazards regression was used to determine the associations of parathyroidectomy with major adverse cardiovascular events (MACEs), CV disease-related hospitalization, and CV hospitalization-associated mortality. RESULTS: The authors identified 210,206 beneficiaries diagnosed with PHPT from 2006 to 2017. Among 63,136 (30.0%) treated with parathyroidectomy and 147,070 (70.0%) managed nonoperatively within 1 year of diagnosis, the unadjusted incidence of MACE was 10.0% [mean follow-up 59.1 (SD 35.6) months] and 11.5% [mean follow-up 54.1 (SD 34.0) months], respectively. In multivariable analysis, parathyroidectomy was associated with a lower incidence of MACE [hazard ratio (HR): 0.92; 95% confidence interval (95% CI): 0.90-0.94], CV disease-related hospitalization (HR: 0.89; 95% CI: 0.87-0.91), and CV hospitalization-associated mortality (HR: 0.76; 95% CI: 0.71-0.81) compared to nonoperative management. At 10 years, parathyroidectomy was associated with adjusted absolute risk reduction for MACE of 1.7% (95% CI: 1.3%-2.1%), for CV disease-related hospitalization of 2.5% (95% CI: 2.1%-2.9%), and for CV hospitalization-associated mortality of 1.4% (95% CI: 1.2%-1.6%). CONCLUSIONS: In this large, population-based cohort study, parathyroidectomy was associated with a lower long-term incidence of adverse CV outcomes when compared with nonoperative management for older adults with PHPT, which is relevant to surgical decision making for patients with a long life expectancy.

Validity Evidence for ENTRUST as an Assessment of Surgical Decision-Making for the Inguinal Hernia Entrustable Professional Activity (EPA).

OBJECTIVE: As the American Board of Surgery (ABS) moves toward implementation of Entrustable Professional Activities (EPAs), there is a growing need for objective evaluation of readiness for entrustment of residents. This requires not only assessment of technical skills and knowledge, but also surgical decision-making in preoperative, intraoperative, and postoperative settings. We developed and piloted an Inguinal Hernia EPA Assessment on ENTRUST, a serious game-based online virtual patient simulation platform to assess trainees' decision-making competence. DESIGN: This is a prospective analysis of resident performance on the ENTRUST Inguinal Hernia EPA Assessment using bivariate analyses. SETTING: This study was conducted at an academic institution in a proctored exam setting. PARTICIPANTS: Forty-three surgical residents completed the ENTRUST Inguinal Hernia EPA Assessment. RESULTS: Four case scenarios for the Inguinal Hernia EPA and corresponding scoring algorithms were iteratively developed by expert consensus aligned with ABS EPA descriptions and functions. ENTRUST Inguinal Hernia Grand Total Score was positively correlated with PGY-level (p < 0.0001). Preoperative, Intraoperative, and Postoperative Total Scores were also positively correlated with PGY-level (p = 0.001, p = 0.006, and p = 0.038, respectively). Total Case Scores were positively correlated with PGY-level for cases representing elective unilateral inguinal hernia (p = 0.0004), strangulated inguinal hernia (p < 0.0001), and elective bilateral inguinal hernia (p = 0.0003). Preoperative Sub-Scores were positively correlated with PGY-level for all cases (p < 0.01). Intraoperative Sub-Scores were positively correlated with PGY-level for strangulated inguinal hernia and bilateral inguinal hernia (p = 0.0007 and p = 0.0002, respectively). Grand Total Score and Intraoperative Sub-Score were correlated with prior operative experience (p < 0.0001). Prior video game experience did not correlate with performance on ENTRUST (p = 0.56). CONCLUSIONS: Performance on the ENTRUST Inguinal Hernia EPA Assessment was positively correlated to PGY-level and prior inguinal hernia operative performance, providing initial validity evidence for its use as an objective assessment for surgical decision-making. The ENTRUST platform holds potential as tool for assessment of ABS EPAs in surgical residency programs.

Racial disparities in the utilization of parathyroidectomy among patients with primary hyperparathyroidism: Evidence from a nationwide analysis of Medicare claims.

BACKGROUND: Among patients with primary hyperparathyroidism, parathyroidectomy offers a chance of cure and mitigation of disease-related complications. The impact of race/ethnicity on referral and utilization of parathyroidectomy has not been fully explored. METHODS: Population-based, retrospective cohort study using 100% Medicare claims from beneficiaries with primary hyperparathyroidism from 2006 to 2016. Associations of race/ethnicity with disease severity, surgeon evaluation, and subsequent parathyroidectomy were analyzed using adjusted multivariable logistic regression models. RESULTS: Among 210,206 beneficiaries with primary hyperparathyroidism, 63,136 (30.0%) underwent parathyroidectomy within 1 year of diagnosis. Black patients were more likely than other races/ethnicities to have stage 3 chronic kidney disease (10.8%) but had lower prevalence of osteoporosis and nephrolithiasis compared to White patients, Black and Hispanic patients were more likely to have been hospitalized for primary hyperparathyroidism-associated conditions (White 4.8%, Black 8.1%, Hispanic 5.8%; P < .001). Patients who were White and met operative criteria were more likely to undergo parathyroidectomy than Black, Hispanic, or Asian patients (White 30.5%, Black 23.0%, Hispanic 21.4%, Asian 18.7%; P < .001). Black and Hispanic patients had lower adjusted odds of being evaluated by a surgeon (odds ratios 0.71 [95% confidence interval 0.69-0.74], 0.68 [95% confidence interval 0.61-0.74], respectively) and undergoing parathyroidectomy if evaluated by a surgeon (odds ratios 0.72 [95% confidence interval 0.68-0.77], 0.82 [95% confidence interval 0.67-0.99]). Asian race was associated with lower adjusted odds of being evaluated by a surgeon (odds ratio 0.64 [95% confidence interval 0.57-0.71]), but no difference in odds of parathyroidectomy. CONCLUSION: Racial/ethnic disparities exist in the management of primary hyperparathyroidism among older adults. Determining the factors that account for this disparity require urgent attention to achieve parity in the management of primary hyperparathyroidism.

Isolating steps instead of learners: Use of deliberate practice and validity evidence in coronavirus disease (COVID)-era procedural assessment.

BACKGROUND: In surgical training, assessment tools based on strong validity evidence allow for standardized evaluation despite changing external circumstances. At a large academic institution, surgical interns undergo a multimodal curriculum for central line placement that uses a 31-item binary assessment at the start of each academic year. This study evaluated this practice within increased in-person learning restrictions. We hypothesized that external constraints would not affect resident performance nor assessment due to a robust curriculum and assessment checklist. METHODS: From 2018 to 2020, 81 residents completed central line training and assessment. In 2020, this curriculum was modified to conform to in-person restrictions and social distancing guidelines. Resident score reports were analyzed using multivariate analyses to compare performance, objective scoring parameters, and subjective assessments among "precoronavirus disease" years (2018 and 2019) and 2020. RESULTS: There were no significant differences in average scores or objective pass rates over 3 years. Significant differences between 2020 and precoronavirus disease years occurred in subjective pass rates and in first-time success for 4 checklist items: patient positioning, draping, sterile ultrasound probe cover placement, and needle positioning before venipuncture. CONCLUSION: Modifications to procedural training within current restrictions did not adversely affect residents' overall performance. However, our data suggest that in 2020, expert trainers may not have ensured learner acquisition of automated procedural steps. Additionally, although 2020 raters could have been influenced by logistical barriers leading to more lenient grading, the assessment tool ensured training and assessment integrity.

Pilot evaluation of the Computer-Based Assessment for Sampling Personal Characteristics test.

BACKGROUND: High attrition rates hint at deficiencies in the resident selection process. The evaluation of personal characteristics representative of success is difficult. Here, we evaluate a novel tool for assessing personal characteristics. MATERIALS AND METHODS: To evaluate feasibility, we used an anonymous voluntary survey questionnaire offered to study participants before and after contact with the CASPer test. To evaluate the CASPer test as a predictor of success, we compared CASPer test assessments of personal characteristics versus traditional faculty assessment of personal characteristics with applicant rank list position. RESULTS: All applicants (n = 77) attending an in-person interview for general surgery residency, and all faculty interviewers (n = 34) who reviewed these applications were invited to participate. Among applicants, 84.4% of respondents (65 of 77) reported that a requirement to complete the CASPer test would have no bearing or would make them more likely to apply to the program (mean = 3.30, standard deviation = 0.96). Among the faculty, 62.5% respondents (10 of 16) reported that the same condition would have no bearing or would make applicants more likely to apply to the program (mean = 3.19, standard deviation = 1.33). The Spearman's rank-order correlation coefficients for the relationships between traditional faculty assessment of personal characteristics and applicant rank list position, and novel CASPer assessment of personal characteristics and applicant rank list position, were -0.45 (P = 0.033) and -0.41 (P = 0.055), respectively. CONCLUSIONS: The CASPer test may be feasibly implemented as component of the resident selection process, with the potential to predict applicant rank list position and improve the general surgery resident selection process.

The assessment of emotional intelligence among candidates interviewing for general surgery residency.

BACKGROUND: There is an increasing demand for physicians to possess strong personal and social qualities embodied in the concept of emotional intelligence (EI). However, the residency selection process emphasizes mainly academic accomplishments. In this system, the faculty interview is the primary means of evaluating the nontangible, nonacademic attributes of a candidate. OBJECTIVE: To determine whether the impressions derived from faculty interviews correlate with an applicant's actual EI as measured by a validated objective instrument. STUDY DESIGN: Participating applicants interviewing for a surgical residency position at Stanford completed an EI inventory Trait Emotional Intelligence Questionnaire (TEIQue). Faculty estimated the EI of the applicants they interviewed using a corresponding 360° evaluation form. Multivariate linear regression was performed to identify demographic and academic factors predictive of EI. Applicant TEIQue scores and faculty 360° impressions were correlated using Pearson coefficients. RESULTS: Mean EI of the cohort was higher than that of the average population (5.43 vs 4.89, p<0.001). Age was the only demographic variable that significantly informed EI (B = 0.07, p = 0.005). Among the academic factors considered, United States Medical Licensing Examination Step 1 score was a slight negative predictor of EI (B =-0.007, p = 0.04). Applicant global EI scores did not correlate with faculty impressions of overall EI (r = 0.27, p = 0.06). Of the 4 domains that comprise global EI, sociability and emotionality demonstrated a moderate correlation between applicant and faculty scores (r = 0.31, p = 0.03 and r = 0.27, p = 0.05, respectively). None of the fifteen individual facets of EI demonstrated any correlation between applicant and faculty ratings (r =-0.12 to 0.26, p = 0.06-0.91). No association was found between applicant TEIQue and traditional faculty interview evaluations (r = 0.18, p = 0.19). CONCLUSIONS: Applicant EI correlated poorly with academic parameters and was not accurately assessed by faculty interviews. Methods that better capture this dimension should be incorporated into the residency selection process.