The American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA): Validity Evidence From a Three-Year National Study.

OBJECTIVE: To gather validity evidence supporting the use and interpretation of scores from the American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA) Program. SUMMARY AND BACKGROUND DATA: ACS ERRA is an online formative assessment program developed to assess entering surgery residents' ability to make critical clinical decisions, and includes 12 clinical areas and 20 topics identified by a national panel of surgeon educators and residency program directors. METHODS: Data from 3 national testing administrations of ACS ERRA (2018-2020) were used to gather validity evidence regarding content, response process, internal structure (reliability), relations to other variables, and consequences. RESULTS: Over the 3 administrations, 1975 surgery residents participated from 125 distinct residency programs. Overall scores [Mean = 64% (SD = 7%)] remained consistent across the 3 years ( P = 0.670). There were no significant differences among resident characteristics (gender, age, international medical graduate status). The mean case discrimination index was 0.54 [SD = 0.15]. Kappa inter-rater reliability for scoring was 0.87; the overall test score reliability (G-coefficient) was 0.86 (Ф-coefficient = 0.83). Residents who completed residency readiness programs had higher ACS ERRA scores (66% versus 63%, Cohen's d = 0.23, P < 0.001). On average, 15% of decisions made (21/140 per test) involved potentially harmful actions. Variability in scores from graduating medical schools (7%) carried over twice as much weight than from matched residency programs (3%). CONCLUSIONS: ACS ERRA scores provide valuable information to entering surgery residents and surgery program directors to aid in development of individual and group learning plans.

The American College of Surgeons Entering Resident Readiness Assessment Program: Development and National Pilot Testing Results.

OBJECTIVE: To assess the readiness of entering residents for clinical responsibilities, the American College of Surgeons (ACS) Division of Education developed the "Entering Resident Readiness Assessment" (ACS-ERRA) Program. SUMMARY BACKGROUND: ACS-ERRA is an online formative assessment that uses a key features approach to measure clinical decision-making skills and focuses on cases encountered at the beginning of residency. Results can be used to develop learning plans to address areas that may need reinforcement. METHODS: A national panel of 16 content experts, 3 medical educators, and a psychometrician developed 98 short, key features cases. Each case required medical knowledge to be applied appropriately at challenging decision points during case management. Four pilot testing studies were conducted sequentially to gather validity evidence. RESULTS: Residents from programs across the United States participated in the studies (n = 58, 20, 87, 154, respectively). Results from the pilot studies enabled improvements after each pilot test. For the psychometric pilot (final pilot test), 2 parallel test forms of the ACS-ERRA were administered, each containing 40 cases, resulting in overall mean testing time of 2 hours 2 minutes (SD = 43 min). The mean test score was 61% (SD = 9%) and the G-coefficient reliability was 0.90. CONCLUSIONS: Results can be used to identify strengths and weaknesses in residents' decision-making skills and yield valuable information to create individualized learning plans. The data can also support efforts directed at the transition into residency training and inform discussions about levels of supervision. In addition, surgery program directors can use the aggregate test results to make curricular changes.

Resident hesitation in the operating room: does uncertainty equal incompetence?

CONTEXT: In postgraduate medical programmes, the progressive development of autonomy places residents in situations in which they must cope with uncertainty. We explored the phenomenon of hesitation, triggered by uncertainty, in the context of the operating room in order to understand the social behaviours surrounding supervision and progressive autonomy. METHODS: Nine surgical residents and their supervising surgeons at a Canadian medical school were selected. Each resident-supervisor pair was observed during a surgical procedure and subsequently participated in separate post-observation, semi-structured interviews. Constructivist grounded theory was used to guide the collection and analysis of data. RESULTS: Three hesitation-related themes were identified: the principle of progress; the meaning of hesitation, and the judgement of competence. Supervisors and residents understood hesitation in the context of a core surgical principle we termed the 'principle of progress'. This principle reflects the supervisors' and residents' shared norm that maintaining progress throughout a surgical procedure is of utmost importance. Resident hesitation was perceived as the first indication of a disruption to this principle and was therefore interpreted by supervisors and residents alike as a sign of incompetence. This interpretation influenced the teaching-learning process during these moments when residents were working at the edge of their abilities. CONCLUSIONS: The principle of progress influences the meaning of hesitation which, in turn, shapes judgements of competence. This has important implications for teaching and learning in direct supervision settings such as surgery. Without efforts to change the perception that hesitation represents incompetence, these potential teaching-learning moments will not fully support progressive autonomy.

Practice indicators of suboptimal care and avoidable adverse events: a content analysis of a national qualifying examination.

PURPOSE: To (1) compile an initial list of physician-related practice indicators (PRINDs) that contribute to causing or preventing suboptimal care (SOCR) and adverse events (AEs) and (2) determine the extent to which one national exam assessed these PRINDs. METHOD: In 2009-2010, the authors searched the literature and surveyed 17 physician experts to compile a list of PRINDs of SOCR and avoidable AEs. They then conducted a content analysis of the 2008 and 2009 Medical Council of Canada (MCC) Qualifying Examinations (QEs). RESULTS: The authors identified 92 unique PRINDs, of which 59 were behaviors or decisions expected of all physicians and suitable for assessment on a general medical examination. Of these, 36 (61%) were tested on the 2008 and 2009 MCC QEs. The mean number of PRINDs tested per exam was highest for Part I Knowledge (32.2), followed by Part I clinical decision making (CDM) (18.4) and Part II clinical performance (objective structured clinical examination [OSCE]) (9.8). The percentage of questions or cases per exam testing a PRIND (e.g., 14/36 [39%] for CDM and 5.26/12 [44%] for OSCE) differed from the percentage of the total test score attributed to PRINDs (e.g., 10.8/36 [30%] for CDM and 68.5/1,522.3 [5%] for OSCE). CONCLUSIONS: PRINDs represent candidates' abilities to avoid SOCR and AEs and constitute an important aspect of medical practice to be assessed on licensing or certifying examinations to best protect the public. The different scoring methods used to measure such knowledge and skills warrant further consideration.

Developing a unified list of physicians' reasoning tasks during clinical encounters.

PURPOSE: The clinical reasoning literature focuses on how physicians reason while making decisions, rather than on what they reason about while performing their clinical tasks. In an attempt to provide a common language for discussing, teaching, and researching clinical reasoning, the authors undertook the task of developing a unified list of physicians' reasoning tasks, or what they reason about, during clinical encounters. METHOD: The authors compiled an initial list of 20 reasoning tasks based on the literature from four content areas--clinical reasoning, communications, medical errors, and clinical guidelines. In the summer and fall of 2010, they surveyed a purposive sample of 46 international experts in clinical reasoning and communications. From the results of the first survey, the authors refined their list of reasoning tasks, then resurveyed 22 of the original participants. From the results of the second survey, they further refined their list and validated the inclusion of the reasoning tasks. RESULTS: Twenty-four of 46 (52%) and 15 of 22 (65%) participants completed the first- and second-round surveys, respectively. Following the second-round survey, the authors' list included 24 reasoning tasks, and a clinical example corresponding to each, that fell into four broad categories: framing the encounter (3), diagnosis (8), management (11), and self-reflection (2). CONCLUSIONS: The development of this unified list represents a first step in offering a vocabulary for discussing, reflecting on, teaching, and studying physicians' reasoning tasks during clinical encounters.

Reliability and validity of key feature cases for the self-assessment of colon and rectal surgeons.

OBJECTIVE: The purpose of this study was to determine the reliability and validity of the scores from "key feature" cases in the self-assessment of colon and rectal surgeons. BACKGROUND: Key feature (KF) cases specifically focus on the assessment of the unique challenges, critical decisions, and difficult aspects of the identification and management of clinical problems in practice. KF cases have been used to assess medical students and residents but rarely for specialists. METHODS: Responses from all 256 participants taking the American Society of Colon and Rectal Surgeons (ASCRS) Colon and Rectal Surgery Educational Program (CARSEP) V Self-assessment Examination (SAE) from 1997 to 2002 were scored and analyzed, including score reliability, item analysis for the factual (50 multiple-choice questions (MCQ)) and applied (9 KF cases) knowledge portions of the SAE, and the effect of examination preparation, examination setting, specialization, Board certification, and clinical experience on scores. RESULTS: The reliability (Cronbach alpha) of the scores for the MCQ and KF components was 0.97 and 0.95, respectively. The applied KF component of the SAE was more difficult than the factual MCQ component (0.52 versus 0.80, P < 0.001). Mean item discrimination (upper-lower groups) was 0.59 and 0.66 for the MCQ and KF components, respectively. Taking the test at the annual meeting was harder than at home (0.41 versus 0.81, P < 0.001). Content-related validity evidence for the KF cases was supported by mapping KF cases onto the examination blueprint and by judgments from expert colorectal surgeons about the challenging and critical nature of the KFs used. Construct validity of the KF cases was supported by incremental performance related to types of practice (general, anorectal, and colorectal), levels and types of Board certification, and years of clinical experience. CONCLUSIONS: The self-assessment of surgical specialists, in this case colorectal surgeons, using KF cases is possible and yielded reliable and valid scores.

Description, justification and clarification: a framework for classifying the purposes of research in medical education.

CONTEXT: Authors have questioned the degree to which medical education research informs practice and advances the science of medical education. OBJECTIVE: This study aims to propose a framework for classifying the purposes of education research and to quantify the frequencies of purposes among medical education experiments. METHODS: We looked at articles published in 2003 and 2004 in Academic Medicine, Advances in Health Sciences Education, American Journal of Surgery, Journal of General Internal Medicine, Medical Education and Teaching and Learning in Medicine (1459 articles). From the 185 articles describing education experiments, a random sample of 110 was selected. The purpose of each study was classified as description ('What was done?'), justification ('Did it work?') or clarification ('Why or how did it work?'). Educational topics were identified inductively and each study was classified accordingly. RESULTS: Of the 105 articles suitable for review, 75 (72%) were justification studies, 17 (16%) were description studies, and 13 (12%) were clarification studies. Experimental studies of assessment methods (5/6, 83%) and interventions aimed at knowledge and attitudes (5/28, 18%) were more likely to be clarification studies than were studies addressing other educational topics (< 8%). CONCLUSIONS: Clarification studies are uncommon in experimental studies in medical education. Studies with this purpose (i.e. studies asking: 'How and why does it work?') are needed to deepen our understanding and advance the art and science of medical education. We hope that this framework stimulates education scholars to reflect on the purpose of their inquiry and the research questions they ask, and to strive to ask more clarification questions.

A systematic review of titles and abstracts of experimental studies in medical education: many informative elements missing.

CONTEXT: Informative titles and abstracts facilitate reading and searching the literature. OBJECTIVE: To evaluate the quality of titles and abstracts of full-length reports of experimental studies in medical education. METHODS: We used a random sample of 110 articles (of 185 eligible articles) describing education experiments. Articles were published in 2003 and 2004 in Academic Medicine, Advances in Health Sciences Education, American Journal of Surgery, Journal of General Internal Medicine, Medical Education and Teaching and Learning in Medicine. Titles were categorised as informative, indicative, neither, or both. Abstracts were evaluated for the presence of a rationale, objective, descriptions of study design, setting, participants, study intervention and comparison group, main outcomes, results and conclusions. RESULTS: Of the 105 articles suitable for review, 86 (82%) had an indicative title and 10 (10%) had a title that was both indicative and informative. A rationale was present in 66 abstracts (63%), objectives were present in 84 (80%), descriptions of study design in 20 (19%), setting in 29 (28%), and number and stage of training of participants in 42 (40%). The study intervention was defined in 55 (52%) abstracts. Among the 48 studies with a control or comparison group, this group was defined in 21 abstracts (44%). Study outcomes were defined in 64 abstracts (61%). Data were presented in 48 (46%) abstracts. Conclusions were presented in 97 abstracts (92%). CONCLUSIONS: Reports of experimental studies in medical education frequently lack the essential elements of informative titles and abstracts. More informative reporting is needed.

Quality of reporting of experimental studies in medical education: a systematic review.

OBJECTIVE: Determine the prevalence of essential elements of reporting in experimental studies in medical education. DESIGN: Systematic review. DATA SOURCES: Articles published in 2003 and 2004 in Academic Medicine, Advances in Health Sciences Education, American Journal of Surgery, Journal of General Internal Medicine, Medical Education, and Teaching and Learning in Medicine. REVIEW METHODS: Articles describing education experiments, including evaluation studies with experimental designs, were identified (n = 185) by reviewing titles and abstracts. A random sample (n = 110) was selected for full review. The full text of each article was evaluated for the presence of guideline-based features of quality reporting: a critical literature review, conceptual framework, statement of study intent (e.g. aim, research question, or hypothesis), statement of study design, definition of main intervention and comparison intervention or control group, and consideration of human subject rights. RESULTS: Of the 105 articles suitable for review, 47 (45%) contained a critical literature review and 58 (55%) presented a conceptual framework. A statement of study intent was present in 80 articles (76%), among which the independent and dependent variables were operationally defined in 38 (47%) and 26 articles (32%), respectively. A total of 17 articles (16%) contained an explicit study design statement. Among the 48 studies with a comparison group, 35 (73%) clearly defined the comparison intervention or control group. Institutional review board approval or participant consent was reported in 44 articles (42%). CONCLUSIONS: The quality of reporting of experimental studies in medical education was generally poor. Criteria are proposed as a starting point for establishing reporting standards for medical education research.

Reliability and usefulness of clinical encounter cards for a third-year surgical clerkship.

BACKGROUND: Third-year medical students (MS3) were given clinical encounter cards (CECs) to stimulate feedback during their surgery clerkship. This study analyzes the feedback given on their clinical performance using CECs. METHODS: Two hundred one students enrolled in the 12-week surgery clerkship were given CECs. Each card contained the chief focus of the encounter, which was rated on a six-point Likert scale. The CECs were analyzed to determine if they provided reliable formative information, identified marginal performances, and identified differences between raters, settings, rotations, and aspects of clinical performance evaluated. RESULTS: Seven thousand three hundred eight CECs were submitted from 201 students. The CECs were completed most often (65%) in the inpatient setting. Technical skills were evaluated on 49% of CECs, history/physical examinations on 40%, and case presentations on 30%. There were comments written on half of the CECs and 89% of these were strictly positive. Women (52.8%) and faculty (63.3%) were more likely to provide written comments. The students were most likely to seek feedback from the interns and faculty who gave significantly higher ratings. The Cronbach-alpha reliability coefficient was 0.69, based on 12 raters per student. There was a significant positive correlation (P < 0.05) between the CEC composite rating and the clinical performance rating (r = 0.356), NBME score (r = 0.258), and the final grade (r = 0.250). CONCLUSIONS: The implementation of CECs in a surgery clerkship provided a large quantity of positive feedback. The quality was limited because there were minimal recommendations for improvement and they were a poor predictor of overall clinical performance.

Study habits of surgery residents and performance on American Board of Surgery In-Training examinations.

BACKGROUND: The purpose of this study was to evaluate the study habits (SHs) of surgery residents preparing for the annual American Board of Surgery In-Training Examination (ABSITE). METHODS: A validated instrument developed to assess SHs in college students, the Survey of Study Habits and Attitudes (SSHA), was modified slightly for use with residents. The modified SSHA contains 2 subscale scores, work methods and delay avoidance, and a combined overall study habit score. A total of 59 residents from 2 academic general surgical residency programs were administered the modified SSHA. The SSHA scores were correlated to performance on the in-training examination. RESULTS: There was a small but significant correlation between scores on the modified SSHA instrument and performance on the ABSITE overall (r = 0.29; P < .05; r2 = 0.0841). Linear-regression analysis showed that the clinical component and overall performance on the ABSITE were significantly predicted by the total SH scores. Overall total ABSITE percent correct scores were significantly predicted by residency levels of training and the overall SSHA scales (delay avoidance and work methods). Together they predicted 63% of the total variance in the overall performance scores. Residency level was the strongest predictor. SH performance accounted for 5.9% of the total variance beyond that contributed by residency level of training. CONCLUSIONS: The correlations of surgical resident ABSITE performance with SSHA scores were on the same order of magnitude as those of college students and academic performance with the original SSHA. Although SH in this study accounted for a measurable yet small contribution to ABSITE performance, this contribution was not enough to consider using the SSHA instrument in its current modified form as a diagnostic and counseling tool. Published instruments not specifically designed for residents may not be tailored enough to measure residents' unique SH.

Interactive voice response to assess residents' laparoscopic skills: an instrument validation study.

OBJECTIVE: The study was undertaken to evaluate the psychometric properties of the Interactive Voice Response (IVR) instrument to assess laparoscopic surgical skills of residents. STUDY DESIGN: Surgical competence of obstetric-gynecologic residents at the University of Ottawa (Canada) was evaluated after observed performance of skills. Three questions addressing general domains of surgical skill were rated per surgery using a 5-point Likert scale: overall, 993 surgical ratings were made by 29 residents and 13 faculty preceptors. RESULTS: With use of a generalizability study, a minimum of 12 preceptor ratings (G=0.80) are needed to obtain reliable measures of residents' surgical skills. Item analysis indicated that the three items on the instrument were highly correlated (Cronbach alpha=.92). Elements of construct validity were evident for diagnostic and simple laparoscopic procedures. CONCLUSION: The IVR instrument can be a valid and reliable measure of residents' surgical skills. The IVR ratings functioned as a global rating scale, capturing overall elements of surgical skills. The ratings on the IVR tool had construct validity with improved performance with increased surgical training and experience.

Patient outcomes for segmental colon resection according to surgeon's training, certification, and experience.

Background. We examined patient outcomes for colon resection to determine if they varied according to surgeon-specific factors including: (1) American Board of Surgery (ABS) certification, (2) colorectal surgery subspecialty certification, (3) site of residency training (university-based vs nonuniversity-based), and (4) years of experience since ABS certification. Methods. We performed a retrospective study of 15,427 admissions of northern Illinois residents who underwent segmental colon resection as their primary operation from 1994 to 1997 at 76 nonfederal Illinois hospitals. There were 514 surgeons. Main outcome measures were inpatient mortality, complications, and hospital length of stay. Regression analyses with mixed effects were used to assess the significance of surgeon-specific variables as a predictor of outcomes after risk adjustment for patient age, gender, emergency admission, surgeon volume, hospital site, colon pathology, and comorbid illnesses. Results. ABS-certification was associated with reduced mortality and morbidity. Increasing years of experience was associated with reduced mortality. Colorectal surgery certification and site of residency training did not significantly affect outcomes. Conclusion. We were able to link patient outcomes with surgeon's training. Certification was an important determinant of patient outcomes for colon resection. Increasing surgeon experience also had a favorable effect on outcomes, suggesting a continued learning curve subsequent to residency. (Surgery 2002;132:663-72).