Faculty evaluation of simulation-based modules for assessment of intraoperative decision making.

BACKGROUND: Previous studies using simulation-based curricula have focused largely on technical skills. We developed a set of simulation-based modules that focus on intraoperative decision making. The objective of this study was to conduct a faculty evaluation of: (1) the usefulness of 4 newly developed, simulation-based modules; (2) the curricular need to train and assess intraoperative decision making skills of the residents; and (3) potential for resident benefit. METHODS: Simulation-based modules were developed using a cognitive task analysis (CTA) framework. The CTA framework involved faculty interviews focusing on 4 operative tasks that span a range of complexity: (1) creation of small and large bowel stoma, (2) laparoscopic ventral hernia repair, (3) pancreaticojejunostomy, and (4) lymph node biopsy during a mediastinoscopy. An experienced psychologist conducted task-specific, one-on-one interviews with fellowship-trained specialists who perform these operations in their practice. Two faculty were interviewed for each procedure. The interviews lasted a minimum of 1 hour and focused on critical decisions, error prevention, error recognition, and error rescue strategies. The coded interview summaries were used as development guides for the simulation-based learning modules. Each module included locally developed physical models for the simulated operative tasks combined with oral and paper-based questions. The physical models were fabricated in such a way that simulated operative tasks could be performed using standard surgical instruments. To assess the newly developed simulation-based modules, 8 volunteer faculty (50% overlap with the interview pool) participated in a simulation-based exercise during a one-on-one session and then completed an 8-item survey cast on a 5-point Likert agreement scale (1 = strongly disagree, 5 = strongly agree). One of the items was worded negatively to ensure internal consistency. An independent observer recorded faculty session times and assessed faculty engagement in the task (1 = not engaged, 5 = extremely engaged). RESULTS: On average, faculty spent 60 minutes completing each simulation-based exercise. Over 80% of this time was spent performing the operative tasks as they would during a real-life procedure. Mean engagement rating was 4.9 (maximum 5.0, SD = 0.3). Survey results show strong agreement on the importance of training and assessing intraoperative decision making, and that residents would likely benefit from the simulation-based modules. CONCLUSION: We developed 4 high-fidelity simulation-based modules to assess intraoperative decision making. Faculty agree strongly on the importance and need for additional modules.

What defines a transplant surgeon? A needs assessment for curricular development in transplant surgery fellowship training.

This study compares the perceptions of transplant surgery program directors (PDs) and recent fellowship graduates (RFs) regarding the adequacy of training and relevancy to practice of specific curricular content items in fellowship training. Surveys were sent to all American Society of Transplant Surgery approved fellowship PDs and all RFs in practice <5 years. For operative procedures, the RFs considered the overall training to be less adequate than the PDs (p = 0.0117), while both groups considered the procedures listed to be relevant to practice (p = 0.8281). Regarding nonoperative patient care items, although RFs tended to rank many individual items lower, both groups generally agreed that the training was both adequate and relevant. For nonpatient care related items (i.e. transplant-related ethics, economics, research, etc.), both groups scored them low regarding their adequacy of training although RFs scored them significantly lower than PDs (p = 0.0006). Regarding their relevance to practice, while both groups considered these items relevant, RFs generally considered them more relevant than PDs. Therefore, although there is consensus on many items, significant differences exist between PDs and RFs regarding their perceptions of the adequacy of training and the relevance to practice of specific curriculum items in transplant surgery fellowship training.

Promoting collaborative teaching in clinical education.

BACKGROUND: Although the practice of medicine is increasingly a multidisciplinary effort, clinical teaching of medical students is accomplished primarily within a departmental structure. PURPOSE: The purpose of this study was to identify subject matter within the clinical curriculum that could serve as focus for multidisciplinary teaching. METHODS: A questionnaire was sent to 13 clerkship directors (representing required clerkships) at a large Midwestern medical school in which they were asked to rate a list of 631 patient problems as critical (primary), important (secondary), or "nice to know," relevant to their respective clerkship objectives. RESULTS: All clerkship directors completed the questionnaire. There were 523 items that were considered primary, and over 90% of these items were listed as either primary or secondary in more than 1 clerkship. Twelve topics were considered primary or secondary by at least 5 clerkship directors. Four clerkship directors identified 43 patient problems, and 3 clerkship directors identified 92 topics as primary or secondary clerkship objectives. CONCLUSIONS: In this study, listing of patient problems across clerkships demonstrated significant overlap of the clinical curriculum, suggesting multiple opportunities for faculty collaboration in clinical education.

Evaluating a clerkship curriculum: description and results.

BACKGROUND: A comprehensive and up-to-date curriculum requires periodic formal review to ensure it continues to meet learners' needs. PURPOSE: This study describes a model for evaluating a surgery clerkship curriculum designed to determine the appropriateness of its learning objectives to the general professional education of a physician. METHODS: A survey was mailed to graduates who pursued generalist residencies. Respondents estimated the number of patients encountered annually with specified presenting complaints or disease entities and the percentage of time these were referred to surgeons. For 23 technical procedures, respondents estimated the frequency done annually and whether remaining proficient in the skill was considered important. RESULTS: The majority of graduates reported the need to remain proficient in 19 technical procedures. Numerous patient problems were identified as requiring careful instruction so that learners know when and when not to refer for surgical intervention. CONCLUSIONS: The clerkship was modified to include skills and topics not previously included or appropriately emphasized.

A ten-year analysis of surgical education research.

BACKGROUND: Surgical education peer-reviewed publications have markedly increased over the last decade. The purpose of this study was to review the surgical education literature published over the last 10 years and address the following questions: What subjects in surgical education tend to be studied? What are the most to least commonly employed research designs and statistics? Has there been a change in how research data are collected? Where are these studies published? METHODS: A literature search encompassing surgical education papers published between January 1988 and August 1998 was performed. Four investigators coded qualifying abstracts on journal type, subject of research, data collection methods, research design, and statistics. Each investigator was asked to code 10 articles at the start of the study to assess interrater reliability. RESULTS: A total of 420 abstracts were evaluated. Interrater reliability yielded percent agreements ranging from 82% to 96%. Curriculum and teaching were the most frequent topics studied (40%), followed by assessment (23%) and program evaluation (18%). Most research designs used were descriptive (41%). Experimental design has progressively increased from 2% in 1988-89 to 16% in 1998. A total of 551 statistical methods were accounted for in the 420 abstracts. The most common statistical analyses used were descriptive statistics (32%). The predominant mode of data collection was through testing or direct observations (34%). Survey instruments followed closely as a popular data collection method at 27%. The majority of papers were published in peer-reviewed surgical journals (64%),followed by medical education journals (22%) and "other" journals (14%). CONCLUSIONS: An analysis of the surgical education literature demonstrates the growing emphasis on the use of educational research to explore relevant issues and problems. Descriptive research is most popular, with an increasing trend in experimental research. Publication of educational research in peer-reviewed surgical journals is becoming more popular. This study informs those interested in the surgical education research literature of current trends, and what they need to know for a more critical appraisal of this body of literature.

Content of thoracic residency training and its relevance to the practice of medicine.

BACKGROUND: The Thoracic Surgery Directors Association (TSDA) curriculum book provides learning objectives for a thoracic surgery residency. Our purpose was to evaluate the relevance of these objectives through feedback from recent graduates. METHODS: Graduates of multiple TSDA programs were mailed a 50-item questionnaire. Survey items were objectives from the TSDA curriculum book representing six areas of thoracic surgery. Graduates rated each objective for adequacy of instruction and relevance to their current practice on Likert-type scales. RESULTS: Two hundred twenty-eight surveys were included in the analysis. Despite excellent operating room education, graduates across subspecialty lines reported the need for improved education in "nonoperative" subjects. Graduates practicing cardiac surgery reported little relevance of their general thoracic educational experience. Conversely, graduates practicing general thoracic surgery expressed the need for more/better educational experiences in thoracic oncology and esophageal surgery. CONCLUSIONS: Contemporary thoracic surgical education can be improved. A strong need for improvement exists in the teaching of "nonoperative" subjects. As graduates elect careers in thoracic subspecialties, a need exists to align thoracic surgery educational experiences with ultimate career goals of residents.

Assessment of a surgery clerkship's performance evaluation system.

BACKGROUND: This study was designed to examine the validity of a surgery clerkship's performance evaluation system. The study also assessed faculty members' confidence in how clerks are evaluated and promotion decisions made. METHODS: Student files from five classes (n = 339) were analyzed. A 25-item published survey designed to study faculty perceptions of a student evaluation system was distributed. Chi-square tests of independence and descriptive statistics were used. RESULTS: Faculty survey results showed faculty perceptions of strengths and weaknesses in the evaluation system. Significant relationships were found with prior performance indicators and clerkship performance yielding evidence of concurrent and predictive validity. CONCLUSIONS: Findings provide guidance for enhancing how clerks are evaluated as well as specific profiles of students who may need special attention or additional challenges during the surgery clerkship. This study provides a model for other clerkships to assess their student evaluations systems.

Medical ethics curriculum for surgical residents: results of a pilot project.

INTRODUCTION: This study sought to develop and evaluate a medical ethics curriculum designed specifically for surgical residents. METHODS: The learning needs of surgical residents relevant to ethics were determined by using a structured literature review and synthesis strategy. We identified 5 primary areas of importance for ethics education for surgical residents: withdrawing and withholding treatment, advance directives, do-not-resuscitate orders, informed consent, and communicating bad news. Learning objectives were developed, and teaching plans were designed for four 90-minute interactive teaching episodes on the basis of adult learning principles. We surveyed residents using a published survey instrument modified for surgery to identify residents' beliefs about the usefulness of ethics training, confidence in addressing ethical issues, and factual knowledge of ethics questions. RESULTS: Twenty surgical residents at a single institution completed the pretest and posttest close-ended surveys. Results showed that although 88% had formal ethics exposure in medical school, 93% considered ethics education at the resident level to be a "very important" or "important" topic. Residents' confidence in addressing ethical issues showed statistically significant improvement between pretest and posttest surveys for 13 of 23 items. There were no statistically significant linear relationships between postgraduate year of residency and the pretest confidence items or the number of correct responses on the pretest multiple-choice items. CONCLUSIONS: Despite the prevalence of ethics education during medical school, surgical residents welcome formal instruction on numerous ethical issues pertinent to surgical practice. A focused curriculum can be developed that has a measurable impact on residents' confidence in addressing ethical issues.

Ambulatory teaching "lite": less clinic time, more educationally fulfilling.

Typically, the primary instructional method for ambulatory care education is direct interaction between a preceptor and a learner during a patient encounter. This paper describes instructional strategies teachers and learners can use in ambulatory care training that can occur before or after scheduled clinic hours, thus providing instruction without disrupting a preceptor's busy clinic. First, they describe how preceptors and clerkship or residency-program directors can orient learners prior to their arrival at assigned sites, so that learners are better prepared to assume their patient-care responsibilities. Then they discuss strategies for making use of various types of conferences and independent learning activities to enhance learners' clinical experiences. Conferences and independent study projects that occur before clinic hours can help learners bring a higher level of thinking and clinical sophistication to their role in the ambulatory care site; conferences and independent study activities that occur after clinic hours give learners an opportunity to reinforce and expand on what they have learned during clinic. In this way, learners' educational experiences are enhanced, the best use is made of preceptors' time and expertise, and clinic efficiency is not disrupted.

Description and evaluation of the Surgeons as Educators course.

BACKGROUND: Since 1993, the American College of Surgeons has sponsored an annual 6-day course entitled the Surgeons as Educators. The course was designed to provide academic surgeons with the knowledge and skills necessary to enhance the surgical education curriculum, teaching strategies, educational program administration, and performance evaluation. This article describes the development, implementation, and effect of the course on the classes graduating in 1993 and 1994. STUDY DESIGN: The effect of the course was studied by using a longitudinal survey approach. A survey was mailed to participants 3 to 6 months after they completed the course. Graduates were asked to describe any education-related actions taken attributable to attending the Surgeons as Educators course. The quality of course content and presentations were evaluated by using end-of-course evaluation forms and daily feedback forms and by an external reviewer. RESULTS: Within 6 months of returning from the course, more than one half of the graduates initiated actions related to curriculum development, teaching strategies, or educational administration. One third or more of the graduates modified their performance and program evaluation systems. Using a five-point scale, ratings of the course content ranged from 3.78 to 4.64 for "value of topic" and from 3.77 to 4.76 for "quality of presentation." Items evaluated by the graduates on the end-of-course evaluation forms ranged from 7.8 to 8.7 on a nine-point scale. CONCLUSIONS: The Surgeons as Educators course offered an opportunity for participants to interact among themselves and with course faculty about educational issues and to practice teaching skills. The course was highly rated for educational quality and value. The retreat environment and the length of the program helped attendees become immersed during this "protected time" to analyze strengths and weaknesses of their programs and devise achievable plans to improve their abilities as educators and the effectiveness of their programs.

The learning needs of first-year surgical residents in the intensive care unit.

BACKGROUND: Residents are more likely to learn from educational experiences when properly prepared from the start of instruction and expectations are clear and consistent. The purpose of this study was to examine what first year residents need to know and be able to perform before starting their rotation in the intensive care unit (ICU). METHODS: The sample included surgical faculty, first through fifth year general surgery residents, and ICU nurses. A survey was developed that listed 110 knowledge and skill items felt to be possible prerequisite learning objectives. Respondents were asked to evaluate the required level of knowledge or skill on a 5-point Likert-type scale. RESULTS: Items were divided into knowledge or skill categories and then ranked in descending order by their mean ratings. Four knowledge and 10 skill items were ranked highly by all 3 groups of respondents. Kruskall-Wallis analyses found 58 items with statistically significant differences in mean ratings between groups of respondents. CONCLUSION: With this information a self assessment tool was developed to determine the level of performance on these identified learning objectives. A course of instruction can now be developed to assist residents in meeting these objectives before they assume patient care responsibilities in the ICU.

Is test security a concern when OSCE stations are repeated across clerkship rotations?

PURPOSE: To examine the issue of test security when the same stations on an objective structured clinical examination (OSCE) are repeated across clerkship rotations. Specifically, is there a significant difference in students' scores on stations repeated in three or four rotations within a single academic year? METHOD: The sample consisted of 15 stations in the OSCE given at the end of the third-year surgery clerkship at the Southern Illinois University School of Medicine from 1989-90 through 1993-94. Each station was administered three or four times a year. One-way analyses of variance with contrast coding to test for linear trends were used. Results were considered significant at or below the .05 level. RESULTS: Only three of the 15 stations showed significant linear trends. A two-part couplet orthopedic station showed a significant decreasing linear trend (p=.0001). Two stations showed significant increasing linear trends: a general surgery couplet station (p=.0004) and a plastic surgery station with an essay question (p=.0253). CONCLUSION: There was no consistent evidence that students scored increasingly higher on OSCE stations repeated throughout the year. Thus, it would appear that a clerkship can repeat OSCE stations within an academic year without risk of a trend toward increasing scores.

Topics and skills in orthopaedics for the general practitioner.

We conducted a survey to identify topics and skills in orthopaedics that should be included in a curriculum for the education of general practitioners. Forty-one program chairpersons responded to a questionnaire that asked respondents to rate the importance of each of eighty topics or skills related to orthopaedic surgery. The questionnaires were analyzed with use of descriptive statistics. The stability of the results was assessed with a follow-up questionnaire five months after the first survey had been completed, and the results were analyzed with a correlation coefficient of the mean ratings of importance and percentage agreement on individual items. The results produced a rank-order listing of important topics and skills in orthopaedics for the general practitioner. The topics and skills that were rated highly by program chairpersons corresponded to the types of musculoskeletal problems most commonly seen by family practitioners. This survey provides concurrent validity for the rank-order lists and confirms the need to include these items in a core curriculum for general practitioners.

Description and results of a needs assessment in preparation for the "Surgeons as educators" course.

BACKGROUND: In August 1993 the American College of Surgeons sponsored a course entitled "Surgeons as Educators" (SAE) aimed at equipping academic surgeons with the knowledge and skills necessary to enhance surgical education administration, curriculum, teaching, and evaluation. METHODS: The instructional design model used to construct the course called for a formal needs assessment to determine the importance, current skill level, and priority of what needed to be learned to be an effective educator. The needs assessment was accomplished using a job analysis and questionnaire approach. The 68-item questionnaire was mailed to 320 academic surgeons representing eight medical schools. RESULTS: A 62% response rate was achieved. Results indicated the education-related tasks or activities that faculty felt were important to their careers, as well as their perceived level of development in each area. Descriptive statistics were used to summarize the responses that were critical to the SAE faculty in helping prioritize, sequence, and time ration course content. Collective results became the foundation for developing the SAE curriculum by the course's five faculty members. CONCLUSIONS: A well-done needs assessment does not necessarily guarantee course success; however, it is the first and critical step to planning an educationally sound faculty development course or program designed for adult learners.

Patients' attitudes toward the involvement of medical students in their care.

BACKGROUND: Research has shown that medical students are generally accepted by patients, but specific details that elucidate patient perceptions of the advantages and disadvantages to student involvement have not been documented. This study was designed to determine variables that influence patient satisfaction with students. PATIENTS AND METHODS: Patients were interviewed by one of two faculty members using a questionnaire-style format that covered 12 variables regarding patient care. Patients were asked to rate the extent to which medical students helped or hindered their hospital stay, with regard to the 12 variables. RESULTS: Patients' attitudes were favorable regardless of the students' extent of clinical experience or clinical abilities or the patients' age or length of hospital stay. Patients reported that students spent time with them and answered their questions. Most patients stated that they would allow students to participate in their future hospital care. CONCLUSION: Positive patient-student interactions can have important effects on patients' expectations and their acceptance of future encounters with students. This fact is becoming increasingly important due to the changes in health care and the decreasing incidence of inpatient surgical encounters.

Changing surgical education strategies in an environment of changing health care delivery systems.

Emerging changes in health care delivery will have a significant impact on the structure of surgical education in academic departments of surgery. Based on some assumptions as to the probable nature of the final product of this reform, this article encourages a proactive stance by surgical educators to anticipate changes and move toward restructuring in areas of curricular content, the teaching process, performance evaluation strategies, and faculty infrastructure of the academic department. Curriculum changes must bridge the gap between public health and medicine and continue the aggressive trend toward teaching in the outpatient setting. Surgical educators must adapt to evolving computer and instructional technology that will make multimedia presentations, distance education, teleconferencing, hypermedia, and virtual reality commonplace in the teaching setting. Increased emphasis on accountability and accreditation will require stringent criteria in performance and program evaluation methodology. The academic infrastructure will need to adapt to the changing goal of training more general surgeons and fewer specialists and yet maintain the fundamental responsibility of an academic surgeon for mentoring the medical student and surgical resident.

What does the American Board of Surgery In-Training/Surgical Basic Science Examination tell us about graduate surgical education?

BACKGROUND: This research sought to identify the strengths and weakness in residents' basic science knowledge and, second, to determine whether they progressively improve in their abilities to recall basic science information and clinical management facts, to analyze cause-effect relationships, and to solve clinical problems. METHODS AND RESULTS: Basic science knowledge was assessed by means of the results of the January 1990 American Board of Surgery's In-Training/Surgical Basic Science Exam (IT/SBSE). Postgraduate year (PGY) 1 residents' scores were compared with those of PGY5 residents. Content related to a question was considered "known" if 67% or more of the residents in each of the two groups answered it correctly. Findings showed 44% of the content tested by the basic science questions were unknown by new and graduating residents. The second research question required the 250 IT/SBSE questions to be classified into one of three levels of thinking abilities: recall, analysis, and inferential thinking. Profile analysis (split-plot analysis of variance) for each pair of resident levels indicated significant (P < 0.001) differences in performance on questions requiring factual recall, analysis, and inference between all levels except for PGY3s and PGY4s. CONCLUSIONS: The results of this research enable program directors to evaluate strengths and weaknesses in residency training curricula and the cognitive development of residents.