Practical Guide to Assessment Tool Development for Surgical Education Research.

This Guide to Statistics and Methods describes the process of validation and gathering validity evidence for assessment tool development for surgical education research.

American College of Surgeons Objective Assessment of Skills in Surgery (ACS OASIS): A Formative Assessment of Junior Residents' Technical Skills.

OBJECTIVE: The objective assessment of technical skills of junior residents is essential in implementing competency-based training and providing specific feedback regarding areas for improvement. An innovative assessment that can be easily implemented by training programs nationwide has been developed by expert surgeon educators under the aegis of the American College of Surgeons (ACS) Division of Education. This assessment, ACS Objective Assessment of Skills in Surgery (ACS OASIS) uses eight stations to address technical skills important for junior residents within the domains of laparoscopic appendectomy, excision of lipoma, central line placement, laparoscopic cholecystectomy, trocar placement, exploratory laparotomy, repair of enterotomy, and tube thoracostomy. The purpose of this study was to implement ACS OASIS at a number of sites to study its psychometric rigor. DESIGN: The ACS OASIS was pre-piloted at two programs to establish feasibility and to gather information regarding implementation. Each skills station was 12 minutes long, and the faculty completed a checklist with 5 to 15 items, and a global assessment scale. The study was then repeated at three pilot sites and included 29 junior residents who were assessed by a total of 44 faculty. Psychometric data for the stations and checklists were collected and analyzed. SETTING: The pre-pilot sites were Geisinger and University of Tennessee Knoxville.Data were gathered from pilot sites that included Wellspan Health, Duke University, and University of California Los Angeles. RESULTS: The mean checklist score for all learners was 76% (IQR of 66%-85%). The average global rating was 3.36 on a 5-point scale with a standard deviation of 0.56. The overall cut score derived using the borderline group method was at 68% with 34% of performances requiring remediation. Using this criterion, the average number of stations that were completed by each learner without need for remediation was five.The station discrimination index ranged from 0.27 to 0.65 (all above the threshold of 0.25), demonstrating solid psychometric characteristics at the station level. The internal-consistency reliability was 0.76 with SEM of 5.8%. The inter-rater reliability (intraclass correlation) was high at 0.73 with general agreement of 79% between the two raters. The station discrimination was at 0.45 (range of 0.27 to 0.65) indicating a high level of differentiation between high and low performers. Using the generalizability theory, the G-coefficient reliability was at 0.72 with the reliability projection flattening after 8 stations. Overall, 75% to 82% the faculty and learners rated ACS OASIS as realistic and beneficial. CONCLUSIONS: ACS OASIS is a psychometrically sound technical skills assessment tool that can provide useful information for feedback to junior residents and support efforts to remediate gaps in performance.

Lived Experiences of Surgical Residents During the COVID-19 Pandemic: A Qualitative Assessment.

OBJECTIVE: As the COVID-19 pandemic dynamically changes our society, it is important to consider how the pandemic has affected the training and wellness of surgical residents. Using a qualitative study of national focus groups with general surgery residents, we aim to identify common themes surrounding their personal, clinical, and educational experiences that could be used to inform practice and policy for future pandemics and disasters. DESIGN: Six 90-minute focus groups were conducted by a trained qualitative researcher who elicited responses on six predetermined topics. De-identified transcripts and audio recordings were later analyzed by two independent researchers who organized responses to each topic into themes. SETTING: Focus groups were conducted virtually and anonymously. PARTICIPANTS: General surgery residents were recruited from across the country. Demographic information of potential participants was coded, and subjects were randomly selected to ensure a diverse group of participants. RESULTS: The impact of the COVID-19 pandemic on residents' clinical, educational, and personal experiences varied depending on the institutional response of the program and the burden of COVID-19 cases geographically. Many successes were identified: the use of telehealth and virtual didactics, an increased sense of camaraderie amongst residents, and flexibility in scheduling. Many challenges were also identified: uncertainty at work regarding personal protective equipment and scheduling, decreased case volume and educational opportunities, and emotional trauma and burnout associated with the pandemic. CONCLUSIONS: These data gathered from our qualitative study highlight a clear, urgent need for thoughtful institutional planning and policies for the remainder of this and future pandemics. Residency programs must ensure a balanced training program for surgical residents as they attempt to master the skills of their craft while also serving as employed health care providers in a pandemic. Furthermore, a focus on wellness, in addition to clinical competency and education, is vital to resident resilience and success in a pandemic setting.

Robotic Inguinal Hernia Repair Outcomes: Operative Time and Cost Analysis.

BACKGROUND: Robotic inguinal hernia repair is the latest iteration of minimally invasive herniorrhaphy. Previous studies have shown expedited learning curves compared to traditional laparoscopy, which may be offset by higher cost and longer operative time. We sought to compare operative time and direct cost across the evolving surgical practice of 10 surgeons in our healthcare system. METHODS: This is a retrospective review of all transabdominal preperitoneal robotic inguinal hernia repairs performed by 10 general surgeons from July 2015 to September 2018. Patients requiring conversion to an open procedure or undergoing simultaneous procedures were excluded. The data was divided to compare each surgeon's initial 20 cases to their subsequent cases. Direct operative cost was calculated based on the sum of supplies used intra-operatively. Multivariate analysis, using a generalized estimating equation, was adjusted for laterality and resident involvement to evaluate outcomes. RESULTS: Robotic inguinal hernia repairs were divided into two groups: early experience (n = 167) and late experience (n = 262). The late experience had a shorter mean operative time by 17.6 min (confidence interval: 4.06 - 31.13, p = 0.011), a lower mean direct operative cost by $538.17 (confidence interval: 307.14 - 769.20, p < 0.0001), and fewer postoperative complications (p = 0.030) on multivariate analysis. Thirty-day readmission rates were similar between both groups. CONCLUSION: Increasing surgeon experience with robotic inguinal hernia repair is associated with a predictable reduction in operative time, complication rates, and direct operative cost per case. Thirty-day readmission rates are not affected by the learning curve.

The Value of Surgical Graduate Medical Education (GME) Programs Within An Integrated Health Care System.

OBJECTIVE: Surgical graduate medical education (GME) programs add both significant cost and complexity to the mission of teaching hospitals. While expenses tied directly to surgical training programs are well tracked, overall cost-benefit accounting has not been performed. In this study, we attempt to better define the costs and benefits of maintaining surgical GME programs within a large integrated health system. DESIGN: We examined the costs, in 2018 US dollars, associated with the surgical training programs within a single health system. Total health system expenses were calculated using actual and estimated direct GME expenses (salary, benefits, supplies, overhead, and teaching expenses) as well as indirect medical education (IME) expenses. IME expenses for each training program were estimated by using both Medicare percentages and the Medicare Payment Advisor Commission study. The projected cost to replace surgical trainees with advanced practitioners or hospitalists was obtained through interviews with program directors and administrators and was validated by our system's business office. SETTING: A physician lead, integrated, rural health system consisting of 8 hospitals, a medical school and a health insurance company. PARTICIPANTS: GME surgical training programs within a single health system's department of surgery. RESULTS: Our health system's department of surgery supports 8 surgical GME programs (2 general surgery residencies along with residencies in otolaryngology, ophthalmology, oral-maxillofacial surgery, urology, pediatric dentistry, and vascular surgery), encompassing 89 trainees. Trainees work an average of 64.4 hours per week. Total health system cost per resident ranged from $249,657 to $516,783 based on specialty as well as method of calculating IME expenses. After averaging program costs and excluding IME and overhead expenses, we estimated the average annual cost per trainee to be $84,171. We projected that replacing our surgical trainees would require hiring 145 additional advanced practitioners at a cost of $166,500 each per year, or 97 hospitalists at a cost of $346,500 each per year. Excluding overhead, teaching and IME expenses, these replacements would cost the health system an estimated additional $16,651,281 or $26,119,281 per year, respectively. CONCLUSIONS: Surgical education is an integral part of our health system and ending surgical GME programs would require large expansion of human resources and significant additional fiscal capital.

Lymph Node Harvest as a Measure of Quality and Effect on Overall Survival in Esophageal Cancer: A National Cancer Database Assessment.

Surgical therapy for esophageal cancer is the cornerstone of treatment, and the highest quality operation should lead to the highest cure rate. Evaluated lymph node (ELN) count is one quality measure that has been championed. The objective of this study was to explore ELN in esophagectomy, examine predictors of harvesting ≥12 nodes, and determine whether higher ELN improves overall survival (OS). ELN was examined in patients with resected esophageal cancer using the National Cancer Database from 2004 to 2013. In this study, 41,746 patients met the inclusion criteria. Fifty-two per cent of patients had 12 or more nodes harvested. Academic programs were most likely to harvest ≥12 nodes (58% of cases) compared with other programs (43-56% of cases). Seventy per cent of cases with ≥12 nodes harvested were performed at high-volume centers. Preoperative radiation or preoperative chemoradiation led to lower ELN (46% and 48%) versus preoperative chemotherapy alone (66%). Multivariate analysis showed that patients who had ≥12 nodes removed had better OS (Hazard Ratio 0.843 [95 confidence interval 0.820-0.867]). In addition, care at a high-volume facility, care at an academic facility, private insurance, and income ≥$63,000 were all associated with improved OS. Higher ELN count is associated with OS in patients with esophageal cancer. Patients who receive care at high-volume centers and academic centers are more likely to undergo more extensive lymphadenectomy. All centers should strive to examine at least 12 nodes to provide a quality esophagectomy.

Laparoscopic versus robotic adrenalectomy: a review of the national inpatient sample.

BACKGROUND: Laparoscopic adrenalectomy (LA) has become the standard treatment of adrenal lesions. Recently, robotic-assisted adrenalectomy (RA) has become an option, however, short-term outcomes for RA have not been well studied and benefits over LA are debatable. The aim of this study was to explore differences in short-term outcomes between LA and RA using the national inpatient sample (NIS) database. METHODS: Patient data were collected from the NIS. All patients undergoing LA or RA from January 2009 to December 2012 were included. Univariate analysis and propensity matching were performed to look for differences between the groups. RESULTS: A total of 1006 patients (66.4% in LA group and 33.6% in RA group) were identified. Patient age group, gender, race, risk of mortality, severity of illness or indication for adrenalectomy did not differ significantly between the LA or RA cohorts. Insurance type predicted procedure type (45% of medicare patients underwent RA versus 29% of patients with private insurance, p < 0.0001). Patients living in the highest income areas were more likely to receive the laparoscopic approach (31.7 versus 17.4%, p < 0.0001). Hospital volume, bed size and teaching status of the hospital were not significant factors in the decision of RA versus LA. There was no difference in complication and conversion rates between RA versus LA. The mean length of stay was shorter in the RA group (2.2 versus 1.9 days, p = 0.03). Total charges were higher in the RA group ($42,659 versus $33,748, p < 0.0001). There was a significant trend towards more adrenalectomies being performed robotic assisted by year. Only 22% of adrenalectomies were performed robotic-assisted in 2009 compared with 48% in 2012. CONCLUSIONS: The overall benefit for RA remains small and higher total charges for RA may currently outweigh the benefits. These findings may change as more cases are performed robotically assisted and robotic technology improves.

Disparities in compliance with the Oncotype DX breast cancer test in the United States: A National Cancer Data Base assessment.

BACKGROUND: Oncotype DX (ODX) is a multi-gene tumor assay for breast cancer patients. Our objective is to assess whether eligible ODX patients received the test and whether recommendations were followed based on respective risk. METHODS: We retrospectively analyzed testing in patients deemed eligible for ODX using the National Cancer Data Base. RESULTS: A total of 158,235 patients met ODX eligibility criteria. Sixty-four percent of eligible patients did not receive the test. Non-testing rose with age. White patients were more likely to be tested (56%) versus black patients (46%, p < 0.0001). Testing was highest at academic facilities (40%). Privately insured patients were more likely to get the test compared to uninsured (45 versus 34%, p < 0.0001). Those in the highest income quartile were more likely to be tested (p < 0.001). CONCLUSIONS: ODX is under-utilized, with racial and socio-economic factors influencing testing. Further studies are necessary to identify ways to remove disparities and increase testing when appropriate.

Predictors of Utilization and Quality Assessment in Robotic Rectal Cancer Resection: A Review of the National Cancer Database.

Robotic surgery (RS) is a novel treatment for rectal cancer resection (RCR); however, this technology is not widely accessible. The objective of this study is to evaluate the utilization of RS in RCR compared with open and laparoscopic techniques and to assess the quality of resection. RCR from 2010 to 2012 were identified using the National Cancer Database and placed into categories: open, laparoscopic, and robotic. A total of 23,857 patients who received open, laparoscopic, and robotic RCR were included (n = 14,735 (61.8%); 7,185 (30.1%); 1,937 (8.1%), respectively). Patients over 70 had a lower likelihood of robotic RCR. Patients with insurance were 2 times more likely to have robotic RCR. Patients at an academic/research program were more likely to undergo RS compared with a community cancer program (OR 3.6, 95% CI [2.79, 4.78]; P < 0.0001). Length of stay (LOS) was longer in open (7.9 ± 7.1) versus laparoscopic (6.6 ± 6.3) or robotic (6.8 ± 6.4) RCR (P < 0.0001). Although there was an increased likelihood of positive surgical margins with open RCR (OR 1.3, 95% CI [1.09, 1.66]; P < 0.0001), there was no difference in robotic and laparoscopic techniques. Younger insured patients at academic/research affiliated hospitals have a higher likelihood of receiving robotic RCR. Compared with open RCR, robotic RCR have a lower likelihood of positive surgical margins and shorter LOS.

Same-day combined endoscopic retrograde cholangiopancreatography and cholecystectomy: Achievable and minimizes costs.

BACKGROUND: It is estimated that choledocholithiasis is present in 5% to 20% of patients at the time of laparoscopic cholecystectomy (LC). Several European studies have found decreased length of stay (LOS) when performing LC and intraoperative endoscopic retrograde cholangiopancreatography (ERCP) on the same day for choledocholithiasis. In the United States, common bile duct stones are usually managed preoperatively and typically on a day separate from the day LC was performed. Our aim was to evaluate LOS and total hospital cost for separate-day versus same-day ERCP/cholecystectomy. METHODS: This was a retrospective study of patients undergoing ERCP and cholecystectomy during the same admission for the management of choledocholithiasis from 2010 to 2014 at Geisinger Medical Center. The separate-day group underwent ERCP at least 1 day before cholecystectomy and often underwent two separate anesthesia events, while the same-day group had ERCP and cholecystectomy performed on the same day under one general anesthesia event. The primary outcome measured was LOS. RESULTS: The study population included 240 patients. There were 175 patients in the separate-day group and 65 patients in the same-day group. Median age was similar between the two groups. The separate-day group had a median of one minor comorbidity compared with zero within the same-day group using the Charlson Comorbidity Index. Overall, LOS for the separate-day group was 5 days compared with 3 days in the same-day group (p < 0.0001). There was no difference in conversion rates to open cholecystectomy between the two groups (14% in the separate-day vs. 12% in the same-day group). Total median hospital cost for the separate-day group was $102,537 compared with $90,269 in the same-day group (p < 0.0001). CONCLUSION: Same-day ERCP and cholecystectomy is feasible and minimizes costs. Same-day procedures decreased hospital LOS by 2 days and had approximately $12,000 in cost savings. Future goals include a multidisciplinary protocol to study outcomes in larger numbers. LEVEL OF EVIDENCE: Therapeutic study, level IV. Economic study, level III.

An evaluation of the usefulness of the standardized patient methodology in the assessment of surgery residents' communication skills.

OBJECTIVE: The overall objective of this study is to assess the usefulness of a standardized patient (SP) educational activity in the evaluation of surgery residents' communication skills. DESIGN: This is a pre/post observational study on surgery residents' communication skills, attitudes, and self-perceptions after an educational activity with standardized patients. SETTING: Scott & White Healthcare is a Central Texas-based non-profit integrated healthcare system with more than nine hospitals, 60 clinics, and a health plan. Scott & White Memorial Hospital in Temple is a Level-1 Trauma Center and the clinical site for the Texas A&M Health Science Center-College of Medicine in Temple, Texas. PARTICIPANTS: In all, 14 residents of the Texas A&M Health Science Center-College of Medicine/Scott & White General Surgery Residency Program participated in the SP education activity. RESULTS: After the activity, residents agreed more with the statements: "Communication skills are a learned behavior" and "Effective communication with patients is more difficult in high-stress situations." A significant increase in positive attitude toward physician-patient communication was measured in PGY3 and PGY-4 residents after the activity. However, there was no significant difference in residents' self-perceptions of communication skills after the activity. The residents believed the activity was somewhat useful, and feedback from the SPs was informative and helpful. CONCLUSIONS: Overall, in a formative setting, the SP methodology was a useful methodology to assess surgery residents' patient communication. The use of this methodology might require specific curriculum integration that is appropriate to the year of postgraduate training. This study demonstrates how SPs can be integrated into a Surgery Residency curriculum for teaching and assessing communication.

A multi-institutional study of the Family Conference Objective Structured Clinical Exam: a reliable assessment of professional communication.

BACKGROUND: To test the value of a simulated Family Conference Objective Structured Clinical Exam (OSCE) for resident assessment purposes, we examined the generalizability and construct validity of its scores in a multi-institutional study. METHODS: Thirty-four first-year (PG1) and 27 third-year (PG3) surgery residents (n = 61) from 6 training programs were tested. The OSCE consisted of 2 cases (End-of-Life [EOL] and Disclosure of Complications [DOC]). At each program, 2 clinicians and 2 standardized family members rated residents using case-specific tools. Performance was measured as the percentage of possible score obtained. We examined the generalizability of scores for each case separately. To assess construct validity, we compared PG1 with PG3 performance using repeated measures multivariate analysis of variance (MANOVA). RESULTS: The relative G-coefficient for EOL was .890. For DOC, the relative G-coefficient was .716. There were no significant performance differences between PG1 and PG3 residents. CONCLUSIONS: This OSCE provides reliable assessments suitable for formative evaluation of residents' interpersonal communication skills and professionalism.