The key-features approach to assess clinical decisions: validity evidence to date.

The key-features (KFs) approach to assessment was initially proposed during the First Cambridge Conference on Medical Education in 1984 as a more efficient and effective means of assessing clinical decision-making skills. Over three decades later, we conducted a comprehensive, systematic review of the validity evidence gathered since then. The evidence was compiled according to the Standards for Educational and Psychological Testing's five sources of validity evidence, namely, Content, Response process, Internal structure, Relations to other variables, and Consequences, to which we added two other types related to Cost-feasibility and Acceptability. Of the 457 publications that referred to the KFs approach between 1984 and October 2017, 164 are cited here; the remaining 293 were either redundant or the authors simply mentioned the KFs concept in relation to their work. While one set of articles reported meeting the validity standards, another set examined KFs test development choices and score interpretation. The accumulated validity evidence for the KFs approach since its inception supports the decision-making construct measured and its use to assess clinical decision-making skills at all levels of training and practice and with various types of exam formats. Recognizing that gathering validity evidence is an ongoing process, areas with limited evidence, such as item factor analyses or consequences of testing, are identified as well as new topics needing further clarification, such as the use of the KFs approach for formative assessment and its place within a program of assessment.

Reasons reviewers reject and accept manuscripts: the strengths and weaknesses in medical education reports.

PURPOSE: Scientific journals rely on peer review to maintain the high quality and standards of papers accepted for publication. The purpose of this study was to explore the strengths and weaknesses of medical education reports by analyzing the ratings and written comments given by external reviewers. METHOD: The author conducted a content analysis of reviewers' comments on 151 research manuscripts submitted to the 1997 and 1998 Research in Medical Education conference proceedings. The negative comments on 123 manuscripts that received "questionable, probably exclude" or "definitely exclude" overall ratings from at least one reviewer were evaluated. A similar analysis was performed on reviewers' positive comments for 28 manuscripts recommended unanimously for acceptance. RESULTS: On average, four peers (4.1, SD = 0.97, range = 2-6) reviewed each manuscript. Of those recommended for exclusion, a mean of 2.3 reviewers recommended exclusion and each reviewer wrote a mean of 8.1 (SD = 5.7) reasons. The top ten reasons for rejection were: inappropriate or incomplete statistics; overinterpretation of results; inappropriate or suboptimal instrumentation; sample too small or biased; text difficult to follow; insufficient problem statement; inaccurate or inconsistent data reported; incomplete, inaccurate, or outdated review of the literature; insufficient data presented; and defective tables or figures. The main strengths noted in accepted manuscripts were the importance or timeliness of the problem studies, excellence of writing, and soundness of study design. CONCLUSION: While overstating the results and applying the wrong statistics can be fixed, other problems that the reviewers identified (ignoring the literature, designing poor studies, choosing inappropriate instruments, and writing poor manuscripts) are likely to be fatal flaws warranting rejection.

The Medical Council of Canada's key features project: a more valid written examination of clinical decision-making skills.

In 1986 the Medical Council of Canada (MCC) commissioned a six-year research and development project to create a new, more valid written examination of clinical decision-making skills for the Canadian Qualifying Examination in Medicine. At that time, the qualifying examination consisted of three booklets of multiple-choice questions and one booklet of patient management problems administered over a two-day period. All graduates of Canadian and foreign medical schools must pass this examination before practicing medicine anywhere in Canada except Québec. The project was undertaken because (1) numerous studies do not support the use of patient management problems (PMPs) to assess clinical decision-making skills, and (2) research results on the characteristics of clinical decision-making skills offered guidance to develop new approaches to their assessment. In particular, research suggested that these skills are specific to the case or problem encountered and are contingent on the effective manipulation of a few elements of the problem that are crucial to its successful resolution--the problem's key features. The problems developed by this project focused only on the assessment of these key features. The project was implemented in three overlapping phases over a six-year period, 1986-1992, each containing a development component followed by a pilot test through which the research studies were carried out. The pilot tests were conducted by presenting sets of new key feature problems to classes of graduating students in medical schools across Canada.(ABSTRACT TRUNCATED AT 250 WORDS)

Developing key-feature problems and examinations to assess clinical decision-making skills.

This article introduces the concept of a key feature and describes its function as the cornerstone of key-feature problems, a new problem format for the written assessment of clinical decision-making skills of medical trainees and practitioners. The rationale for using this problem format and the steps in problem and examination development--including issues of scoring and standard setting--are described. A key feature is defined as a critical step in the resolution of a clinical problem, and a key-feature problem consists of a clinical case scenario followed by questions that focus on only those critical steps. The questions can be presented to require examinees either to write in their responses or to select them from a list of options. For each question, examines can be instructed to supply or select whatever number of responses is appropriate to the clinical task being tested, and answer keys can comprise one or several responses. This problem format, with its focus on only the critical steps in problem resolution, and with its flexibility in question format and scoring keys, effectively addresses the psychometric considerations of content validity and test score reliability, and accommodates the complexity and configurations of actions often required in the resolution of clinical problems.

Sustaining curricular change at Sherbrooke through external, formative program evaluations.

In this case study, the authors discuss external, formative program evaluation as a means to monitor and sustain ongoing curricular change and to prepare for periodic accreditation reviews. The Faculty of Medicine at Sherbrooke (in Quebec, Canada), following a major curricular reform begun in the mid-1980s, held three external, formative program evaluations in 1988, 1991, and 1994, using expert judgment and "connoisseur" models of evaluation. The authors present the goals of the evaluations (e.g., to evaluate the implementation of intended curricular changes) and the 17-step process used (e.g., "involve as many faculty and students as possible before and during the visit"), and describe the preparation for the evaluations, the selection of the external evaluators (e.g., a chief medical officer from the World Health Organization, a high-profile basic sciences teacher, the chairman of the previous accreditation team, and others), and on-site activities and reporting. Recommendations from the evaluators and the subsequent actions taken (and in a few cases, not taken) by the school are presented and discussed in terms of program planning, curricular content, basic sciences, problem-based learning and tutors' training, assessment of students, resource constraints, clerkships, community orientation, students' self-directedness, and medical humanism. The three evaluations helped guide and support the major curricular undertakings and encouraged continued changed and refinements. They also had a significant effect on the curriculum itself, on the faculty, and on the socioeducational climate of the school.

Comparing times and performances of French- and English-speaking candidates taking a national examination of clinical decision-making skills.

French-speaking candidates taking the translated version of the Medical Council of Canada's (MCC's) Qualifying Examination in Medicine often complain that poor performance could be due to translation defects. The purposes of this 1991 study were to determine (1) whether French-speaking candidates spend the same time as do English-speaking candidates in answering the questions in the fourth and last booklet of Part 1 of the MCC's Qualifying Exam, and (2) for items where the French-speaking candidates have more difficulty, whether these differences are within normal limits, and if not, could they be attributed to faults in the translation? Two versions of the fourth booklet, one in English and the other a translation into French, were administered to 229 randomly selected candidates (98 French speakers and 131 English speakers). The booklets contained 19 clinical problems and a total of 44 key features; performance was measured by the number of key features the examinees correctly responded to. (Key features are the critical or essential steps needed to resolve a clinical problem.) The French text was 16% longer, and the French candidates took longer to complete the two-hour examination (a mean of 116.31 minutes versus 107.84 minutes for the English speakers, p = .000). However, there was no direct relationship between the time spent on a section of the examination and the number of words it contained. The French candidates' overall scores did not differ from those of the English candidates (59.76% versus 61.33%, p = .11).(ABSTRACT TRUNCATED AT 250 WORDS)

Content validation of key features on a national examination of clinical decision-making skills.

Key features (KFs) represent the critical, or essential, steps in the identification and management of a clinical problem. KFs for 59 clinical problems were defined by members of a test committee for the Medical Council of Canada as part of their efforts to create a more valid written examination of clinical decision-making skills for the Canadian Qualifying Examination in Medicine. In order to evaluate the content validity of KFs that the test committee had defined for the examination, 99 physicians from outside the committee, who came from clerkship programs at all 16 of Canada's medical schools, participated in three studies conducted in 1991. The first study was retrospective and was designed to find the degree of agreement or disagreement that the outside physicians had with the KFs already defined for each problem by the committee members. The second study was prospective and was to compare the KFs generated de novo by the participants with those already defined by the committee members. The third study was to gather the outside physicians' opinions of the frequencies with which graduating students in Canada are exposed to the 59 problems used in the retrospective and prospective studies. Almost all the KFs defined by the test committee were corroborated by the outside physicians, 92% in the retrospective study and 94% in the prospective one.(ABSTRACT TRUNCATED AT 250 WORDS)

When to recommend compulsory versus optional CME programs? A study to establish criteria.

When should remedial continuing medical education (CME) be compulsory for family physicians? When should it be optional? Should it be structured or not? In 1993-1994, the authors addressed this need for criteria by conducting a study that used reports on 14 physicians who had undergone a structured oral interview (SOI) at the College of Physicians of Quebec. (The SOI is a day-long encounter during which two specially trained physician-interviewers present a physician with standardized clinical cases that focus on ten specific aspects of a family physician's competence.) The 14 SOI reports were reviewed by 12 external physician-judges in an attempt to see how consistently they could link the ten aspects of competence, as shown in the reports, to five particular types of recommended remedial CME programs (the strictest being "compulsory program with suspended license" and the most lenient being "simple suggestions for improvement"). There was substantial agreement among the judges when choosing between compulsory and optional programs (kappa = 0.63, p < .05). The main criteria used when recommending an optional program were overall strengths and the quality of clinical reasoning. The same two criteria were also used for recommending a compulsory program, but the judges also considered three additional factors: the physician's ability to recognize his or her limits and how he or she handled referrals and prescribed medications. Many of the criteria used by the judges were based on unique information that came out of observations and interactions during the SOIs, such as quality of argumentation, sustaining a train of thought, sense for the case as a whole, or awareness of one's limits. Finally, the external judges corroborated the decisions previously made by the College of Physicians of Quebec concerning the appropriate CME programs for the 14 physicians.

Education in ambulatory settings: developing valid measures of educational outcomes, and other research priorities.

Efforts to redesign education in ambulatory settings are hampered by a lack of rigorous and coherent research on the learning process in these settings and the desired outcomes of the educational experiences. The authors present 13 priority research topics and 51 important research questions concerning education in ambulatory settings that were defined by a distinguished group of medical educators, clinicians, and policymakers who attended an invitational conference on education in ambulatory settings in 1996. The need to establish valid and reliable measures of quality and outcomes of educational programs and instructional interventions stood out as the major prerequisite for conducting research on education in ambulatory settings. Issues of theory building, research priorities, and research design are discussed, and policy recommendations are made for the development of valid measures of educational outcomes. The creation of a "Medical Education Outcomes Commission" is proposed to act as a repository for measures and instruments, and to provide the field with mechanism to validate instruments and uniform recommendations to conduct studies of quality. The authors urge funding agencies with missions that support medical education to invest in basic research on the outcomes of education in ambulatory settings.

Physician-assessment and physician-enhancement programs in Canada.

In the mid-1980s, the licensing authorities in Quebec, Ontario and Manitoba have introduced programs to conduct in-depth assessments of the clinical skills and abilities of physicians with suspected deficiencies. These assessments are intended to supplement the provincial licensing authorities' existing peer review or patient-complaint mechanisms by confirming the physicians' overall level of competence and identifying specific clinical strengths and weaknesses. An "educational prescription", based on the results of the assessment, focuses on aspects of clinical practice in which the physicians need or wish to enhance their skills. In some situations, licensure decisions are based on the assessment information. This article describes the programs in Quebec, Ontario and Manitoba. Each program comprises a different process of personal assessment and individualized continuing medical education to help physicians improve their clinical competence, and each is built on sound principles of clinical competence assessment and educational planning.