The key-features approach to assess clinical decisions: validity evidence to date.

The key-features (KFs) approach to assessment was initially proposed during the First Cambridge Conference on Medical Education in 1984 as a more efficient and effective means of assessing clinical decision-making skills. Over three decades later, we conducted a comprehensive, systematic review of the validity evidence gathered since then. The evidence was compiled according to the Standards for Educational and Psychological Testing's five sources of validity evidence, namely, Content, Response process, Internal structure, Relations to other variables, and Consequences, to which we added two other types related to Cost-feasibility and Acceptability. Of the 457 publications that referred to the KFs approach between 1984 and October 2017, 164 are cited here; the remaining 293 were either redundant or the authors simply mentioned the KFs concept in relation to their work. While one set of articles reported meeting the validity standards, another set examined KFs test development choices and score interpretation. The accumulated validity evidence for the KFs approach since its inception supports the decision-making construct measured and its use to assess clinical decision-making skills at all levels of training and practice and with various types of exam formats. Recognizing that gathering validity evidence is an ongoing process, areas with limited evidence, such as item factor analyses or consequences of testing, are identified as well as new topics needing further clarification, such as the use of the KFs approach for formative assessment and its place within a program of assessment.

Reasons reviewers reject and accept manuscripts: the strengths and weaknesses in medical education reports.

PURPOSE: Scientific journals rely on peer review to maintain the high quality and standards of papers accepted for publication. The purpose of this study was to explore the strengths and weaknesses of medical education reports by analyzing the ratings and written comments given by external reviewers. METHOD: The author conducted a content analysis of reviewers' comments on 151 research manuscripts submitted to the 1997 and 1998 Research in Medical Education conference proceedings. The negative comments on 123 manuscripts that received "questionable, probably exclude" or "definitely exclude" overall ratings from at least one reviewer were evaluated. A similar analysis was performed on reviewers' positive comments for 28 manuscripts recommended unanimously for acceptance. RESULTS: On average, four peers (4.1, SD = 0.97, range = 2-6) reviewed each manuscript. Of those recommended for exclusion, a mean of 2.3 reviewers recommended exclusion and each reviewer wrote a mean of 8.1 (SD = 5.7) reasons. The top ten reasons for rejection were: inappropriate or incomplete statistics; overinterpretation of results; inappropriate or suboptimal instrumentation; sample too small or biased; text difficult to follow; insufficient problem statement; inaccurate or inconsistent data reported; incomplete, inaccurate, or outdated review of the literature; insufficient data presented; and defective tables or figures. The main strengths noted in accepted manuscripts were the importance or timeliness of the problem studies, excellence of writing, and soundness of study design. CONCLUSION: While overstating the results and applying the wrong statistics can be fixed, other problems that the reviewers identified (ignoring the literature, designing poor studies, choosing inappropriate instruments, and writing poor manuscripts) are likely to be fatal flaws warranting rejection.

An outcomes research perspective on medical education: the predominance of trainee assessment and satisfaction.

CONTEXT: A fundamental premise of medical education is that faculty should educate trainees, that is, students and residents, to provide high quality patient care. Yet, there is little research on the effect of medical education on patient outcomes. OBJECTIVE: A content analysis of leading medical education journals was performed to determine the primary foci of medical education research, using a three- dimensional outcomes research framework based on the paradigm of health services outcomes research. DATA SOURCES: All articles in three medical education journals (Academic Medicine, Medical Education, and Teaching and Learning in Medicine) from 1996 to 1998 were reviewed. Papers presented at the Research in Medical Education conference at the Association of American Medical Colleges annual meeting during the same period, and published as Academic Medicine supplements, were also analysed. STUDY SELECTION: Only data-driven articles were selected for analysis; thus editorials and abstracts were excluded. DATA EXTRACTION: Each article was categorized according to primary participant (i.e. trainee, faculty, provider and patient), outcome (performance, satisfaction, professionalism and cost), and level of analysis (geographic, system, institution and individual(s)). DATA SYNTHESIS: A total of 599 articles were analysed. Trainees were the most frequent participants studied (68.9%), followed by faculty (19.4%), providers (8.1%) and patients (3.5%). Performance was the most common outcome measured (49.4%), followed by satisfaction (34.1%). Cost was the focus of only 2.3% of articles and patient outcomes accounted for only 0.7% of articles. CONCLUSIONS: Medical education research is dominated by assessment of trainee performance followed by trainee satisfaction. Leading journals in medical education contain little information concerning the cost and products of medical education, that is, provider performance and patient outcomes. The study of these medical education outcomes represents an important challenge to medical education researchers.

Improving continuing medical education for surgical techniques: applying the lessons learned in the first decade of minimal access surgery.

OBJECTIVE: To examine the first decade of experience with minimal access surgery, with particular attention to issues of training surgeons already in practice, and to provide a set of recommendations to improve technical training for surgeons in practice. SUMMARY BACKGROUND DATA: Concerns about the adequacy of training in new techniques for practicing surgeons began almost immediately after the introduction of laparoscopic cholecystectomy. The concern was restated throughout the following decade with seemingly little progress in addressing it. METHODS: A preliminary search of the medical literature revealed no systematic review of continuing medical education for technical skills. The search was broadened to include educational, medical, and psychological databases in four general areas: surgical training curricula, continuing medical education, learning curve, and general motor skills theory. RESULTS: The introduction and the evolution of minimal access surgery have helped to focus attention on technical skills training. The experience in the first decade has provided evidence that surgical skills training shares many characteristics with general motor skills training, thus suggesting several ways of improving continuing medical education in technical skills. CONCLUSIONS: The educational effectiveness of the short-course type of continuing medical education currently offered for training in new surgical techniques should be established, or this type of training should be abandoned. At present, short courses offer a means of introducing technical innovation, and so recommendations for improving the educational effectiveness of the short-course format are offered. These recommendations are followed by suggestions for research.

Skills and attributes of directors of educational programmes.

OBJECTIVES: The purpose of this study was to obtain a list of desirable skills and attributes of prospective educational programme directors as judged by potential employers of future directors. DESIGN: An international mail survey was conducted worldwide, using a one-page, open-ended questionnaire. SETTING: University of Illinois at Chicago, USA. SUBJECTS: Deans, department heads and chief executive officers from the various health professions. RESULTS: A total of 139 respondents completed the questionnaire (22. 3% response rate). Responses were remarkably similar across health professions. The nine leading skills were: oral communication skills, interpersonal skills, competent practitioner, educational goal-definition skills, educational design skills, problem-solving and decision-making skills, team worker and building skills, written communication skills and fiscal manager and budgeting skills. The three leading personal attributes were: visionary, flexible and open-minded and trustworthy and value-driven individual. CONCLUSIONS: The complete list of skills and attributes can be used by employers or administrators to build checklist items when hiring or reviewing programme directors or to plan and evaluate training programmes. The importance of leadership compared to managerial responsibilities is discussed as well as the need to train future educational programme directors.

Education in ambulatory settings: developing valid measures of educational outcomes, and other research priorities.

Efforts to redesign education in ambulatory settings are hampered by a lack of rigorous and coherent research on the learning process in these settings and the desired outcomes of the educational experiences. The authors present 13 priority research topics and 51 important research questions concerning education in ambulatory settings that were defined by a distinguished group of medical educators, clinicians, and policymakers who attended an invitational conference on education in ambulatory settings in 1996. The need to establish valid and reliable measures of quality and outcomes of educational programs and instructional interventions stood out as the major prerequisite for conducting research on education in ambulatory settings. Issues of theory building, research priorities, and research design are discussed, and policy recommendations are made for the development of valid measures of educational outcomes. The creation of a "Medical Education Outcomes Commission" is proposed to act as a repository for measures and instruments, and to provide the field with mechanism to validate instruments and uniform recommendations to conduct studies of quality. The authors urge funding agencies with missions that support medical education to invest in basic research on the outcomes of education in ambulatory settings.

Sustaining curricular change at Sherbrooke through external, formative program evaluations.

In this case study, the authors discuss external, formative program evaluation as a means to monitor and sustain ongoing curricular change and to prepare for periodic accreditation reviews. The Faculty of Medicine at Sherbrooke (in Quebec, Canada), following a major curricular reform begun in the mid-1980s, held three external, formative program evaluations in 1988, 1991, and 1994, using expert judgment and "connoisseur" models of evaluation. The authors present the goals of the evaluations (e.g., to evaluate the implementation of intended curricular changes) and the 17-step process used (e.g., "involve as many faculty and students as possible before and during the visit"), and describe the preparation for the evaluations, the selection of the external evaluators (e.g., a chief medical officer from the World Health Organization, a high-profile basic sciences teacher, the chairman of the previous accreditation team, and others), and on-site activities and reporting. Recommendations from the evaluators and the subsequent actions taken (and in a few cases, not taken) by the school are presented and discussed in terms of program planning, curricular content, basic sciences, problem-based learning and tutors' training, assessment of students, resource constraints, clerkships, community orientation, students' self-directedness, and medical humanism. The three evaluations helped guide and support the major curricular undertakings and encouraged continued changed and refinements. They also had a significant effect on the curriculum itself, on the faculty, and on the socioeducational climate of the school.

When to recommend compulsory versus optional CME programs? A study to establish criteria.

When should remedial continuing medical education (CME) be compulsory for family physicians? When should it be optional? Should it be structured or not? In 1993-1994, the authors addressed this need for criteria by conducting a study that used reports on 14 physicians who had undergone a structured oral interview (SOI) at the College of Physicians of Quebec. (The SOI is a day-long encounter during which two specially trained physician-interviewers present a physician with standardized clinical cases that focus on ten specific aspects of a family physician's competence.) The 14 SOI reports were reviewed by 12 external physician-judges in an attempt to see how consistently they could link the ten aspects of competence, as shown in the reports, to five particular types of recommended remedial CME programs (the strictest being "compulsory program with suspended license" and the most lenient being "simple suggestions for improvement"). There was substantial agreement among the judges when choosing between compulsory and optional programs (kappa = 0.63, p < .05). The main criteria used when recommending an optional program were overall strengths and the quality of clinical reasoning. The same two criteria were also used for recommending a compulsory program, but the judges also considered three additional factors: the physician's ability to recognize his or her limits and how he or she handled referrals and prescribed medications. Many of the criteria used by the judges were based on unique information that came out of observations and interactions during the SOIs, such as quality of argumentation, sustaining a train of thought, sense for the case as a whole, or awareness of one's limits. Finally, the external judges corroborated the decisions previously made by the College of Physicians of Quebec concerning the appropriate CME programs for the 14 physicians.