Applying the Bookmark method to medical education: standard setting for an aseptic technique station.

INTRODUCTION: The purpose was to evaluate the Bookmark standard-setting method for use on a performance-based assessment in medical education. METHODS: We compared cutscores for Aseptic performance assessment using the modified Angoff, Hofstee and modified Bookmark methods. RESULTS: The Angoff produced a cutscore of 62%, SD=18 and a percent passing (pp)= 64%. The Hofstee cutscore was 71%, SD=7 and pp=46%. Bookmark mean cutscores were 65.9% SD=10.7 and pp=42% for advanced beginners; 83.6%, SD=9.2 and pp=17% for competent and the proficient category resulted in a cutscore of 96.4% SD=3.9 and pp=1%. Faculty judges found the Bookmark method to be an easy and acceptable method. CONCLUSIONS: The Bookmark method was acceptable to faculty, has reasonable quality metrics when compared to other methods and can be a practical tool for establishing standards in performance-based examinations. The Bookmark method could be useful for establishing multiple levels of competency using the Dreyfus criteria.

Teaching of evidence-based medicine to medical students in Mexico: a randomized controlled trial.

BACKGROUND: Evidence-Based Medicine (EBM) is an important competency for the healthcare professional. Experimental evidence of EBM educational interventions from rigorous research studies is limited. The main objective of this study was to assess EBM learning (knowledge, attitudes and self-reported skills) in undergraduate medical students with a randomized controlled trial. METHODS: The educational intervention was a one-semester EBM course in the 5th year of a public medical school in Mexico. The study design was an experimental parallel group randomized controlled trial for the main outcome measures in the 5th year class (M5 EBM vs. M5 non-EBM groups), and quasi-experimental with static-groups comparisons for the 4th year (M4, not yet exposed) and 6th year (M6, exposed 6 months to a year earlier) groups. EBM attitudes, knowledge and self-reported skills were measured using Taylor's questionnaire and a summative exam which comprised of a 100-item multiple-choice question (MCQ) test. RESULTS: 289 Medical students were assessed: M5 EBM=48, M5 non-EBM=47, M4=87, and M6=107. There was a higher reported use of the Cochrane Library and secondary journals in the intervention group (M5 vs. M5 non-EBM). Critical appraisal skills and attitude scores were higher in the intervention group (M5) and in the group of students exposed to EBM instruction during the previous year (M6). The knowledge level was higher after the intervention in the M5 EBM group compared to the M5 non-EBM group (p<0.001, Cohen's d=0.88 with Taylor's instrument and 3.54 with the 100-item MCQ test). M6 Students that received the intervention in the previous year had a knowledge score higher than the M4 and M5 non-EBM groups, but lower than the M5 EBM group. CONCLUSIONS: Formal medical student training in EBM produced higher scores in attitudes, knowledge and self-reported critical appraisal skills compared with a randomized control group. Data from the concurrent groups add validity evidence to the study, but rigorous follow-up needs to be done to document retention of EBM abilities.

Development and preliminary psychometric properties of a well-being index for medical students.

BACKGROUND: Psychological distress is common among medical students but manifests in a variety of forms. Currently, no brief, practical tool exists to simultaneously evaluate these domains of distress among medical students. The authors describe the development of a subject-reported assessment (Medical Student Well-Being Index, MSWBI) intended to screen for medical student distress across a variety of domains and examine its preliminary psychometric properties. METHODS: Relevant domains of distress were identified, items generated, and a screening instrument formed using a process of literature review, nominal group technique, input from deans and medical students, and correlation analysis from previously administered assessments. Eleven experts judged the clarity, relevance, and representativeness of the items. A Content Validity Index (CVI) was calculated. Interrater agreement was assessed using pair-wise percent agreement adjusted for chance agreement. Data from 2248 medical students who completed the MSWBI along with validated full-length instruments assessing domains of interest was used to calculate reliability and explore internal structure validity. RESULTS: Burnout (emotional exhaustion and depersonalization), depression, mental quality of life (QOL), physical QOL, stress, and fatigue were domains identified for inclusion in the MSWBI. Six of 7 items received item CVI-relevance and CVI-representativeness of >or=0.82. Overall scale CVI-relevance and CVI-representativeness was 0.94 and 0.91. Overall pair-wise percent agreement between raters was >or=85% for clarity, relevance, and representativeness. Cronbach's alpha was 0.68. Item by item percent pair-wise agreements and Phi were low, suggesting little overlap between items. The majority of MSWBI items had a >or=74% sensitivity and specificity for detecting distress within the intended domain. CONCLUSIONS: The results of this study provide evidence of reliability and content-related validity of the MSWBI. Further research is needed to assess remaining psychometric properties and establish scores for which intervention is warranted.

Reliability and validity of key feature cases for the self-assessment of colon and rectal surgeons.

OBJECTIVE: The purpose of this study was to determine the reliability and validity of the scores from "key feature" cases in the self-assessment of colon and rectal surgeons. BACKGROUND: Key feature (KF) cases specifically focus on the assessment of the unique challenges, critical decisions, and difficult aspects of the identification and management of clinical problems in practice. KF cases have been used to assess medical students and residents but rarely for specialists. METHODS: Responses from all 256 participants taking the American Society of Colon and Rectal Surgeons (ASCRS) Colon and Rectal Surgery Educational Program (CARSEP) V Self-assessment Examination (SAE) from 1997 to 2002 were scored and analyzed, including score reliability, item analysis for the factual (50 multiple-choice questions (MCQ)) and applied (9 KF cases) knowledge portions of the SAE, and the effect of examination preparation, examination setting, specialization, Board certification, and clinical experience on scores. RESULTS: The reliability (Cronbach alpha) of the scores for the MCQ and KF components was 0.97 and 0.95, respectively. The applied KF component of the SAE was more difficult than the factual MCQ component (0.52 versus 0.80, P < 0.001). Mean item discrimination (upper-lower groups) was 0.59 and 0.66 for the MCQ and KF components, respectively. Taking the test at the annual meeting was harder than at home (0.41 versus 0.81, P < 0.001). Content-related validity evidence for the KF cases was supported by mapping KF cases onto the examination blueprint and by judgments from expert colorectal surgeons about the challenging and critical nature of the KFs used. Construct validity of the KF cases was supported by incremental performance related to types of practice (general, anorectal, and colorectal), levels and types of Board certification, and years of clinical experience. CONCLUSIONS: The self-assessment of surgical specialists, in this case colorectal surgeons, using KF cases is possible and yielded reliable and valid scores.

Quality control of an OSCE using generalizability theory and many-faceted Rasch measurement.

An Objective Structured Clinical Examination (OSCE) is an effective method for evaluating competencies. However, scores obtained from an OSCE are vulnerable to many potential measurement errors that cases, items, or standardized patients (SPs) can introduce. Monitoring these sources of errors is an important quality control mechanism to ensure valid interpretations of the scores. We describe how one can use generalizability theory (GT) and many-faceted Rasch measurement (MFRM) approaches in quality control monitoring of an OSCE. We examined the communication skills OSCE of 79 residents from one Midwestern university in the United States. Each resident performed six communication tasks with SPs, who rated the performance of each resident using 18 5-category rating scale items. We analyzed their ratings with generalizability and MFRM studies. The generalizability study revealed that the largest source of error variance besides the residual error variance was SPs/cases. The MFRM study identified specific SPs/cases and items that introduced measurement errors and suggested the nature of the errors. SPs/cases were significantly different in their levels of severity/difficulty. Two SPs gave inconsistent ratings, which suggested problems related to the ways they portrayed the case, their understanding of the rating scale, and/or the case content. SPs interpreted two of the items inconsistently, and the rating scales for two items did not function as 5-category scales. We concluded that generalizability and MFRM analyses provided useful complementary information for monitoring and improving the quality of an OSCE.

Simpler standards for local performance examinations: the Yes/No Angoff and whole-test Ebel.

BACKGROUND: The whole-test Ebel and Yes/No Angoff methods offer the possibility of faster and easier standard-setting exercises for local, medium-stakes performance exams. PURPOSE: We sought to establish if these less demanding methods might be substituted for the traditional but complex case-level Ebel. METHODS: Six faculty judges each used all three methods to set standards for six standardized-patient checklists used for 178 fourth-year medical students. We compared the cut scores, passing rates, reliability, ease of use, and credibility of the three methods. RESULTS: The three methods produced roughly equivalent cut scores. Generalizability of judgments was .94 and .96 for the case-level and whole-test Ebel, and .76 for the Yes/No Angoff. Judges found the simplified methods more credible than the case-level Ebel. The Yes/No Angoff method was preferred by five of six judges. CONCLUSIONS: The whole-test Ebel and the Yes/No Angoff may be simple and realistic options for setting standards for local performance exams.

Developing an institution-based assessment of resident communication and interpersonal skills.

PURPOSE: The authors describe the development and validation of an institution-wide, cross-specialty assessment of residents' communication and interpersonal skills, including related components of patient care and professionalism. METHOD: Residency program faculty, the department of medical education, and the Clinical Performance Center at the University of Illinois at Chicago College of Medicine collaborated to develop six standardized patient-based clinical simulations. The standardized patients rated the residents' performance. The assessment was piloted in 2003 for internal medicine and family medicine and was subsequently adapted for other specialties, including surgery, pediatrics, obstetrics-gynecology, and neurology. We present validity evidence based on the content, internal structure, relationship to other variables, feasibility, acceptability, and impact of the 2003 assessment. RESULTS: Seventy-nine internal medicine and family medicine residents participated in the initial administration of the assessment. A factor analysis of the 18 communication scale items resulted in two factors interpretable as "communication" and "interpersonal skills." Median internal consistency of the scale (coefficient alpha) was 0.91. Generalizability of the assessment ranged from 0.57 to 0.82 across specialties. Case-specific items provided information about group-level deficiencies. Cost of the assessment was about $250 per resident. Once the initial cases had been developed and piloted, they could be adapted for other specialties with minimal additional effort, at a cost saving of about $1,000 per program. CONCLUSION: Centrally developed, institution-wide competency assessment uses resources efficiently to relieve individual programs of the need to "reinvent the wheel" and provides program directors and residents with useful information for individual and programmatic review.

Prior experiences associated with residents' scores on a communication and interpersonal skill OSCE.

OBJECTIVE: This exploratory study investigated whether prior task experience and comfort correlate with scores on an assessment of patient-centered communication. METHODS: A six-station standardized patient exam assessed patient-centered communication of 79 PGY2-3 residents in Internal Medicine and Family Medicine. A survey provided information on prior experiences. t-tests, correlations, and multi-factorial ANOVA explored relationship between scores and experiences. RESULTS: Experience with a task predicted comfort but did not predict communication scores. Comfort was moderately correlated with communication scores for some tasks; residents who were less comfortable were indeed less skilled, but greater comfort did not predict higher scores. Female gender and medical school experiences with standardized patients along with training in patient-centered interviewing were associated with higher scores. Residents without standardized patient experiences in medical school were almost five times more likely to be rejected by patients. CONCLUSIONS: Task experience alone does not guarantee better communication, and may instill a false sense of confidence. Experiences with standardized patients during medical school, especially in combination with interviewing courses, may provide an element of "deliberate practice" and have a long-term impact on communication skills. PRACTICE IMPLICATIONS: The combination of didactic courses and practice with standardized patients may promote a patient-centered approach.

Use of flawed multiple-choice items by the New England Journal of Medicine for continuing medical education.

Physicians in the United States are required to complete a minimum number of continuing medical education (CME) credits annually. The goal of CME is to ensure that physicians maintain their knowledge and skills throughout their medical career. The New England Journal of Medicine (NEJM) provides its readers with the opportunity to obtain weekly CME credits. Deviation from established item-writing principles may result in a decrease in validity evidence for tests. This study evaluated the quality of 40 NEJM MCQs using the standard evidence-based principles of effective item writing. Each multiple-choice item reviewed had at least three item flaws, with a mean of 5.1 and a range of 3 to 7. The results of this study demonstrate that the NEJM uses flawed MCQs in its weekly CME program.

Establishing passing standards for classroom achievement tests in medical education: a comparative study of four methods.

PURPOSE: The purpose of this research was to evaluate the Direct Borderline standard-setting method, designed for classroom instructor use, and to compare the characteristics of this newer method to three well-established methods. Most standard-setting methods were designed for large-scale assessments, and most research has taken place in the context of high-stakes examinations. METHOD: Four absolute standard-setting methods (Nedelsky, Direct Borderline, Hofstee, and Ebel) were studied for year 1 and 2 basic science examinations. RESULTS: The Direct Borderline method produced passing scores similar to the Nedelsky method and was reproducible. The Hofstee and Ebel methods produced the lowest passing scores. Standard errors at the passing score were the same or lower for the Direct Borderline method compared with the Nedelsky method. CONCLUSIONS: The Direct Borderline method has reasonable psychometric characteristics and may be practical for faculty to use in establishing absolute passing standards for classroom achievement tests.

Toward meaningful evaluation of clinical competence: the role of direct observation in clerkship ratings.

PROBLEM STATEMENT AND PURPOSE: The lack of direct observation by faculty may affect meaningful judgments of clinical competence. The purpose of this study was to explore the influence of direct observation on reliability and validity evidence for family medicine clerkship ratings of clinical performance. METHOD: Preceptors rating family medicine clerks (n = 172) on a 16-item evaluation instrument noted the data-source for each rating: note review, case discussion, and/or direct observation. Mean data-source scores were computed and categorized as low, medium or high, with the high-score group including the most direct observation. Analyses examined the influence of data-source on interrater agreement and associations between clerkship clinical scores (CCS) and scores from the National Board of Medical Examiners (NBME(R)) subject examination as well as a fourth-year standardized patient-based clinical competence examination (M4CCE). RESULTS: Interrater reliability increased as a function of data-source; for the low, medium, and high groups, intraclass correlation coefficients were.29,.50, and.74, respectively. For the high-score group, there were significant positive correlations between CCS and NBME score (r =.311, p =.054); and between CCS and M4CCE (r =.423, p =.009). CONCLUSION: Reliability and validity evidence for clinical competence is enhanced when more direct observation is included as a basis for clerkship ratings.

Effective home laparoscopic simulation training: a preliminary evaluation of an improved training paradigm.

BACKGROUND: Laparoscopic simulation training has proven to be effective in developing skills but requires expensive equipment, is a challenge to integrate into a work-hour restricted surgical residency, and may use nonoptimal practice schedules. The purpose of this study was to evaluate the efficacy of laparoscopic skills training at home using inexpensive trainer boxes. METHODS: Residents (n = 20, postgraduate years 1-5) enrolled in an institutional review board-approved laparoscopic skills training protocol. An instructional video was reviewed, and baseline testing was performed using the fundamentals of laparoscopic surgery (FLS) peg transfer and suturing tasks. Participants were randomized to home training with inexpensive, self-contained trainer boxes or to simulation center training using standard video trainers. Discretionary, goal-directed training of at least 1 hour per week was encouraged. A posttest and retention test were performed. Intragroup and intergroup comparisons as well as the relationship between the suture score and the total training sessions, the time in training, and attempts were studied. RESULTS: Intragroup comparisons showed significant improvement from baseline to the posttest and the retention test. No differences were shown between the groups. The home-trained group practiced more, and the number of sessions correlated with suture retention score (r(2) = .54, P < .039). CONCLUSIONS: Home training results in laparoscopic skill acquisition and retention. Training is performed in a more distributed manner and trends toward improved skill retention.

Efficacy of a brief screening tool to identify medical students in distress.

PURPOSE: To determine whether the Medical Student Well-Being Index (MSWBI) can serve as a brief assessment tool to identify medical students in severe psychological distress. METHOD: The authors used data from 2,248 medical students at seven U.S. medical schools who responded to a 2007 survey to explore the accuracy of the MSWBI in identifying medical students with three outcomes: low mental quality of life (QOL; defined by having a Medical Outcomes Study Short-Form Health Survey mental component summary score ≥1/2 standard deviation below that of the age- and gender-matched population norm), suicidal ideation, or serious thoughts of dropping out. The authors confirmed their analyses using data from a separate sample of 2,682 students evaluated in 2009. RESULTS: Students with low mental QOL, suicidal ideation, or serious thoughts of dropping out were more likely to endorse each individual MSWBI item and a greater number of total items than were students without such distress (all P < .001). The likelihood ratio for low mental QOL among students with MSWBI scores <4 was 0.47 as compared with 4.79 for those with scores ≥4. At an MSWBI threshold score of ≥4, the MSWBI's sensitivity and specificity for identifying students with low mental QOL or recent suicidal ideation/serious thoughts of dropping out were both ≥90%. On multivariable logistic regression, all MSWBI items were independently associated with at least one outcome. CONCLUSIONS: The MSWBI is a useful brief screening tool to help identify students with severe distress.

A call for the utilization of consensus standards in the surgical education literature.

BACKGROUND: Assessment methods and theory continue to evolve in the general education literature. Nowhere is this more evident than in the framework of validity methods and concepts. The consensus standards of the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education have changed from "types of validity" (criterion, construct, and content) and "valid instruments," last used in 1974, to a concept of identifying evidence for the validity of results and the use of those results. The purpose of this study was to evaluate the surgical education literature for the adoption of the current consensus standards. METHODS: As a representative sample of the surgical educational literature, the validation effort in laparoscopic simulator education was chosen. A MEDLINE search using the terms validity.tw and laparoscop$.tw between 1996 and 2008 (September week 1) yielded 192 citations. All titles and abstracts were reviewed, resulting in 47 studies appropriate for in-depth analysis. RESULTS: Validation studies have evaluated 21 different simulators. Twenty-three percent of the studies adhere, in part, to the new consensus standards for validity. One hundred percent use the old framework of types of validity including 75% using construct validity, 38% using face validity, and 11% using content. CONCLUSION: The widespread use of the currently (after 1999) accepted framework for validity is lacking in the surgical education literature. Surgical educators must remain current and begin to investigate our assessments within the contemporary framework of validity to avoid improper judgments of performance.

Direct observation in medical education: a review of the literature and evidence for validity.

In 2000, the Accreditation Council for Medical Education introduced a new initiative that substantively changed the method by which residency programs are evaluated. In this new competency-based approach to residency education, assessment of performance became a main area of interest, and direct observation was offered as a tool to assess knowledge and skills. Despite being an inherent part of medical education as faculty and learners work together in clinical experiences, direct observation has traditionally been an informal and underused assessment method across all specialties. Residents and students report rarely being observed during their educational process, even though they value the experience. Reasons for this include a lack of faculty time, a lack of faculty skills, a potential stressful effect on the learner, and a perceived lack of validation of the assessment. This article examines the literature regarding the use of direct observation in medical education with a focus on validity evidence. We performed a PubMed search of articles pertaining to direct observation, using key words such as direct observation, performance observation, clinical observation, students, and residents. A subsequent search was conducted in known articles, focusing on variations of the term observation in the titles of articles and introducing the concept of clinical competence. In conclusion, direct observation is a unique and useful tool in the assessment of medical students and residents. Assessing learners in natural settings offers the opportunity to see beyond what they know and into what they actually do, which is fundamentally essential to training qualified physicians. Although the literature identifies several threats to its validity as an assessment, it also demonstrates methods to minimize those threats. Based on the current recommendations and need for performance assessment in education and with attention paid to the development and design, direct observation can and should be included in medical education curricula.

Composite score validity in clerkship grading.

BACKGROUND: Composite score validity depends on its reliability, content, and comparison with other outcomes measures. This study examined all three aspects of a composite score used in a third-year surgery clerkship. METHOD: Composite score reliability was calculated using stratified alpha; several other reasonable composite combinations were tested. Correlation coefficients between clerkship composite scores and grades were calculated with other achievement measures. Two consecutive years were studied (N = 162, N = 159). RESULTS: The clerkship composite score reliability was .76 (Year 1) and .81 (Year 2). Positive correlations (P

Setting standards for performance tests: a pilot study of a three-level Angoff method.

BACKGROUND: The Yes/No Angoff method is a standard-setting variant in which judges estimate whether a borderline student would or would not accomplish each item. However, when there is a 50% chance of success, systematically severe or lenient ratings may skew the cut score. This study piloted a three-level Angoff in which items were assigned to "Yes," "No," or "Maybe" categories. METHOD: Five judges provided three-level Angoff ratings for seven checklists used in a clinical skills exam for fourth-year students. Systematic severity and leniency were simulated by converting all "Maybe" ratings to "Yes" or "No" ratings, respectively. RESULTS: Forty-one percent of 121 items had at least one "Maybe" rating. Ten percent of all ratings were in the "Maybe" category. One judge accounted for 83% of all "Maybe" ratings; three did not use it at all. Case failure rates varied considerably depending on the simulated severity of ratings. Overall failure rates were not substantially impacted. CONCLUSIONS: The three-level Angoff retains the cognitive simplicity of the Yes/No Angoff while addressing the challenge of items midrange for the borderline candidate.

Validation of a colonoscopy simulation model for skills assessment.

OBJECTIVE: The purpose is to provide initial validation of a novel simulation model's fidelity and ability to assess competence in colonoscopy skills. METHODS: In a prospective, cross-sectional design, each of 39 endoscopists (13 staff, 13 second year fellows, and 13 novices) performed a colonoscopy on a novel bovine simulation model. Staff endoscopists also completed a survey examining different aspects of the model's realism as compared to human colonoscopy. The groups' simulation performances were compared. Additionally, individual performances were correlated to patient-based performance data. RESULTS: Median model realism evaluation scores were favorable for nearly all parameters evaluated with mucosa appearance, endoscopic view, and paradoxical motion parameters receiving the highest scores. During simulation procedures, each group outperformed the less experienced groups in all parameters evaluated. Specifically, median cecal intubation times were: staff 226 s (IQR [interquartile range] 179-273), fellows 340 s (282-568), and novices 1,027 s (970-1,122) (P < 0.05). Median total procedure times on the model were: staff 468 s (416-501), fellows 527 s (459-824), and novices 1,350 s (1,318-1,428) (P < 0.05). Finally, individual cecal intubation times on the simulation model had a very high correlation to their respective patient-based times (r = 0.764). CONCLUSIONS: Overall, this model possesses a favorable degree of realism and is able to easily differentiate users based on their level of colonoscopy experience. More impressive, however, is the strong correlation between individual's simulated intubation times and actual patient-based colonoscopy data. In light of these findings, we speculate that this model has potential to be an effective tool for assessment of colonoscopic competence.

Reliability and usefulness of clinical encounter cards for a third-year surgical clerkship.

BACKGROUND: Third-year medical students (MS3) were given clinical encounter cards (CECs) to stimulate feedback during their surgery clerkship. This study analyzes the feedback given on their clinical performance using CECs. METHODS: Two hundred one students enrolled in the 12-week surgery clerkship were given CECs. Each card contained the chief focus of the encounter, which was rated on a six-point Likert scale. The CECs were analyzed to determine if they provided reliable formative information, identified marginal performances, and identified differences between raters, settings, rotations, and aspects of clinical performance evaluated. RESULTS: Seven thousand three hundred eight CECs were submitted from 201 students. The CECs were completed most often (65%) in the inpatient setting. Technical skills were evaluated on 49% of CECs, history/physical examinations on 40%, and case presentations on 30%. There were comments written on half of the CECs and 89% of these were strictly positive. Women (52.8%) and faculty (63.3%) were more likely to provide written comments. The students were most likely to seek feedback from the interns and faculty who gave significantly higher ratings. The Cronbach-alpha reliability coefficient was 0.69, based on 12 raters per student. There was a significant positive correlation (P < 0.05) between the CEC composite rating and the clinical performance rating (r = 0.356), NBME score (r = 0.258), and the final grade (r = 0.250). CONCLUSIONS: The implementation of CECs in a surgery clerkship provided a large quantity of positive feedback. The quality was limited because there were minimal recommendations for improvement and they were a poor predictor of overall clinical performance.