Measuring the effects of an international health professions faculty development fellowship: the FAIMER Institute.

While there are many examples of evaluations of faculty development programs in resource rich countries, evaluation of transnational programs for faculty from developing countries is limited. We describe evaluation of the effects of the FAIMER Institute, an international health professions education fellowship that incorporates not only education content, but also leadership and management topics and, in addition, strives to develop a sustained community of educators. Data were obtained via retrospective pre/post surveys, as well as interviews. Results indicate that participating health professions faculty from developing countries are augmenting their knowledge and skills in education leadership, management, and methodology, and applying that knowledge at their home institutions. Fellows' perceptions of importance of, and their own competence in, all curriculum theme areas increased. Interviews confirmed a nearly universal gain of at least one leadership skill. Findings suggest that the high-engagement experience of the FAIMER model offering integration of education and leadership/management tools necessary to implement change, provides knowledge and skills which are useful across cultural and national contexts and results in the development of a supportive, global, professional network.

Capacity building in medical education and health outcomes in developing countries: the missing link.

BACKGROUND: Finding evidence for the link between capacity building in medical education and improved health outcomes in developing countries is an important challenge. We describe the Foundation for Advancement of International Medical Education and Research (FAIMER) Institute, a two year, part-time fellowship in medical education methodology and leadership and its evaluation as a model to bridge this gap by collecting quantitative and qualitative data on intermediary outcomes. METHODS: FAIMER has used the following framework of human capacity building programs: 1) identify young and talented individuals with potential to become agents for change; 2) organize and deliver an effective learning intervention that is relevant for the environment; 3) facilitate the opportunity for real-life application of acquired knowledge and skills with support; and 4) promote development of a sustainable career path with opportunities for growth and advancement. RESULTS: Twenty-three percent of curriculum innovation projects were directly related to community health. Of the 35 fellows in the first three classes of the Institute, there have been 11 promotions, 9 peer-reviewed publications and 14 international poster presentations, indicating development of the medical education field. Other qualitative and quantitative program evaluation data are presented. DISCUSSION: The link between capacity building in medical education and improved health can be demonstrated in several ways: align curriculum with local health needs, place learners in community clinical settings, teach basic healthcare workers, become involved in national policy development and develop the field of medical education. CONCLUSION: While experimental models may not be possible to evaluate the effect of capacity building, methods described may help support the connection between improved medical education and health.

The relationship between competence and performance: implications for assessing practice performance.

OBJECTIVE: This paper aims to describe current views of the relationship between competence and performance and to delineate some of the implications of the distinctions between the two areas for the purpose of assessing doctors in practice. METHODS: During a 2-day closed session, the authors, using their wide experiences in this domain, defined the problem and the context, discussed the content and set up a new model. This was developed further by e-mail correspondence over a 6-month period. RESULTS: Competency-based assessments were defined as measures of what doctors do in testing situations, while performance-based assessments were defined as measures of what doctors do in practice. The distinction between competency-based and performance-based methods leads to a three-stage model for assessing doctors in practice. The first component of the model proposed is a screening test that would identify doctors at risk. Practitioners who 'pass' the screen would move on to a continuous quality improvement process aimed at raising the general level of performance. Practitioners deemed to be at risk would undergo a more detailed assessment process focused on rigorous testing, with poor performers targeted for remediation or removal from practice. CONCLUSION: We propose a new model, designated the Cambridge Model, which extends and refines Miller's pyramid. It inverts his pyramid, focuses exclusively on the top two tiers, and identifies performance as a product of competence, the influences of the individual (e.g. health, relationships), and the influences of the system (e.g. facilities, practice time). The model provides a basis for understanding and designing assessments of practice performance.

Conceptual challenges in tailoring physician performance assessment to individual practice.

Practice inevitably narrows over time. Therefore, testing of established doctors requires that their assessment be tailored to a far narrower practice than is appropriate for testing of new doctors who have not yet differentiated. In this paper, we address the conceptual challenges of tailoring physician assessment to individual practice. Testing of established doctors needs to reflect that physicians specialise, often in idiosyncratic ways; otherwise, the testing will not be credible among established doctors and will not reflect the realities of their practice. Despite the importance of these goals, the conceptual and methodological challenges of creating tailored assessments remain daunting.

Facing the challenges of competency-based assessment of postgraduate dental training: Longitudinal Evaluation of Performance (LEP).

CONTEXT: The evaluation of competence in the health professions is of great importance to the public and professionals alike. Recent efforts to design dependable and accurate systems of assessment for demanding clinical roles are increasingly attempting to focus on all-round competence of practitioners. Many challenges are faced in this field as a balance between robust assessment methodology and feasibility in practice is crucial to implementation and adoption. OBJECTIVES: The authors discuss some of the challenges faced by educators and clinicians involved in the development of systems of assessment for the health professions, and describe a method which aims to address these issues in the assessment of postgraduate dental training in Scotland. DISCUSSION: Three of the major challenges facing educators and clinicians involved in the design of competency-based systems of assessment are considered: the requirement for evaluation in different areas of competence; the importance of association of assessment with the training objectives, and the types and focus of the assessment introduced. Issues around the use of formative and summative assessment, and the perception that these must always remain completely separate, are discussed in detail. SOLUTIONS: A proposal is made for the introduction of a method of assessment which has been designed keeping these challenges in mind. The rationale behind this assessment method is described.

Certification and specialization: do they matter in the outcome of acute myocardial infarction?

PURPOSE: To learn whether there are differences among certified and self-designated cardiologists, internists, and family practitioners in terms of the mortality of their patients with acute myocardial infarction (AMI). METHOD: Data on all patients admitted with AMI were collected for calendar year 1993 by the Pennsylvania Health Care Cost Containment Council and analyzed. Certified and self-designated family practitioners, internists, and cardiologists (n = 4,546) were compared with respect to the characteristics of their patients' illnesses. In addition, a regression model was fitted in which mortality was the dependent measure and the independent variables were the probability of death, hospital characteristics (location and the availability of advanced cardiac care), and physician characteristics (patient volume, years since graduation from medical school, specialty, and certification status). RESULTS: On average, cardiologists treated more patients than did generalists, and their patients were less severely ill. In the regression analysis, all variables were statistically significant except the availability of advanced cardiac care. Holding all other variables constant, treatment by a certified physician was associated with a 15% reduction in mortality among patients with AMI. CONCLUSIONS: Less patient mortality was associated with treatment by physicians who were cardiologists, cared for larger numbers of AMI patients, were closer to their graduation from medical school, and were certified.

The first decade of the American Board of Internal Medicine certification in critical care medicine: an overview of examinees and certificate holders from 1987 through 1996.

OVERVIEW: This study reviews the first decade of critical care medicine (CCM) certification by the American Board of Internal Medicine (1987-1996). Included are the characteristics of examinee and certificate-holder groups; examination performances from different underlying disciplines of internal medicine, with or without formal CCM training; and the influence of background and a training program as correlates of examination performance. DATA SOURCES: The CCM certification examination has been offered biennially since November 1987. Performance data on the American Board of Internal Medicine examinations in internal medicine and its subspecialties and added qualifications were available for candidates taking the CCM examinations. For examinees with formal CCM training, residency program director ratings, and information regarding the program characteristics of size and percentage of United States and Canadian medical graduates were also available. STUDY SELECTION: All examinees who ever attempted certification were included in this study. The study cohort for each of the five examination administrations consists of all first-time takers. CONCLUSIONS: Cohort sizes have decreased since formal training became an admission requirement in 1993. Percentages of International Medical Graduates and women attempting and achieving certification have increased steadily. Examination performance was positively associated with formal training, internal medicine examination performance, recent medical training, and pulmonary disease certification. For those with formal training, performance was also positively associated with training program director ratings of overall clinical competence and completion of a training program with a higher proportion of United States and Canadian medical graduates.

Standards and reliability in evaluation: when rules of thumb don't apply.

The purpose of this paper is to identify situations in which two rules of thumb in evaluation do not apply. The first rule is that all standards should be absolute. When selection decisions are being made or when classroom tests are given, however, relative standards may be better. The second rule of thumb is that every test should have a reliability of .80 or better. Depending on the circumstances, though, the standard error of measurement, the consistency of pass/fail classifications, and the domain-referenced reliability coefficients may be better indicators of reproducibility.

Examiner differences in the mini-CEX.

OBJECTIVE: The objective of this study was to analyze whether faculty ratings of residents, using the mini-CEX oral exam format, differed in stringency or were influenced by the clinical setting. It also sought to learn whether the examiners were satisfied with the format. METHOD: A mini-CEX encounter consisted of a single faculty member observing a resident conduct a focused history and physical examination in an inpatient, outpatient, or emergency room setting. After asking the resident for a diagnosis and treatment plan, the faculty member rated the resident and provided educational feedback. The encounters were intended to be short and occur as a routine part of the training, so each resident would be evaluated on many occasions by different faculty. SAMPLE: Sixty-four attending physicians evaluated residents from five internal medicine training programs; data were analyzed for 355 mini-CEX encounters involving 88 residents. RESULTS: There were not large differences among the examiners in their ratings. Moreover, there were not great differences among the ratings in terms of the training program with which the examiner was associated, the setting of the mini-CEX, or the nature of the patient. The examiners were generally satisfied with the format and their level of satisfaction was correlated with the residents' perceptions of the format. CONCLUSION: The mini-CEX adapts itself to a broad range of clinical situations, and these results show that it should produce roughly comparable scores over examiners and settings. This makes it a worthwhile device for evaluation at the local level.

The mini-CEX (clinical evaluation exercise): a preliminary investigation.

OBJECTIVE: To gather preliminary data on the mini-CEX (clinical evaluation exercise), a device for assessing the clinical skills of residents. DESIGN: Evaluation of residents by faculty members using the mini-CEX. SETTING: 5 internal medicine training programs in Pennsylvania. PARTICIPANTS: 388 mini-CEX encounters involving 88 residents and 97 evaluators. MEASUREMENTS: A mini-CEX encounter consists of a single faculty member observing a resident while that resident conducts a focused history and physical examination in any of several settings. After asking the resident for a diagnosis and treatment plan, the faculty member rates the resident and provides educational feedback. The encounters are intended to be short (about 20 minutes) and to occur as a routine part of training so that each resident can be evaluated on several occasions by different faculty members. RESULTS: The encounters occurred in both inpatient and ambulatory settings and were longer than anticipated (median duration, 25 minutes). Residents saw either new or follow-up patients who collectively presented with a broad range of clinical problems. The median evaluator assessed two residents and was generally satisfied with the mini-CEX format; residents were even more satisfied with the format. The reproducibility of the mini-CEX is higher than that of the traditional CEX, and its measurement characteristics are similar to those of other test formats, such as standardized patients and standardized oral examinations. CONCLUSIONS: The mini-CEX assesses residents in a much broader range of clinical situations than the traditional CEX, has better reproducibility, and offers residents greater opportunity for observation and feedback by more than one faculty member and with more than one patient. On the other hand, the mini-CEX may be more difficult to administer because multiple encounters must be scheduled for each resident. Exclusive use of the mini-CEX also prevents residents from being observed while doing a complete history and physical examination. Given the promising results and measurement characteristics of the mini-CEX, however, the American Board of Internal Medicine encourages the use of this method in conjunction with or as an alternative to the traditional CEX.

Indicators of the educational effectiveness of subspecialty training programs in internal medicine.

PURPOSE: To identify markers within the subspecialty educational process in internal medicine that are associated with greater improvements in knowledge and skill. METHOD: Candidates for the 1991 cardiovascular disease examination of the American Board of Internal Medicine (ABIM) were matched to the fellowship programs they attended, and their performances on the internal medicine examination of the ABIM, demographic data, and performances on the cardiovascular disease examination were averaged within each program. Information on the programs themselves was also available. The variables were divided according to whether they controlled for input to the training programs, were indicators of educational process, or served as the outcome measure. Analyses were restricted to the 140 programs for which complete information was available and that had four or more candidates who took the cardiovascular examination (97 programs were excluded). RESULTS: The multiple correlation between all measures and scores on the cardiovascular disease examination was .80. Forty-five percent of the explained variance is attributable to previous performance on the internal medicine examination alone. The remaining 55% is shared by the educational indicators and the input measure or explained by the indicators alone. Among the indicators, location of medical school, length of fellowship training, ratings of overall clinical competence, fellow-to-faculty ratio, and number of subspecialties in the same institution made small contributions on their own. Program size, university affiliation, and period of approval by the residency review committee contributed little. CONCLUSION: The data are clear that indicators of educational process such as fellow-to-faculty ratio, longer periods of training, and performance during fellowship can generate greater gains in knowledge and judgment than would be expected if fellows merely built on their abilities at the time of entry to training. Such indicators may be useful if changes in health care delivery require reducing the numbers of specialty and subspecialty training positions.

Guidelines for interpretation of some common indicators of residency program performance.

OBJECTIVE: To provide national norms for indicators of residency-training program quality and information on their reproducibility. PARTICIPANTS: The 364 residency-training programs that had 4 or more candidates take the 1989 to 1991 certifying examination in internal medicine for the first time. DESIGN: Within each residency, program directors' ratings of medical knowledge, certifying examination scores, and certification status (pass or fail) were available for each candidate from 1989 to 1991. Means of these data were calculated for each program for each year of the study. To provide a way of comparing an individual program with all other programs, percentiles are reported for each year. To assess the precision of the measures, generalizability theory was applied and confidence intervals for all data are reported for programs of various size (1 to 25 residents taking the examination) and over the years (1 to 3). RESULTS: Over the 3 years of the study, knowledge ratings, certification rates, and composite scores declined slightly. The correlations between program ratings of medical knowledge and the composite scores ranged from .47 to .60 and certification rates ranged from .44 to .55. The confidence intervals around all of the program performance indicators are relatively large and are affected most by the number of residents in the program. There is little variability across the years. CONCLUSIONS: In smaller programs the precision of the performance indicators is poor; in programs with only a few residents they are virtually meaningless. On the positive side, programs are relatively stable and aggregating indicators over years is a reasonable way to increase their precision in assessing program performance. If the goal of program evaluation is to identify programs at the extremes, especially those at the low end, then such data may help guide program directors and educators.

Recertification in the medical specialties.

To summarize, this paper presents three goals for an ideal rectification program. Based on these goals, two dimensions of an evaluation system for recertification are described. The first dimension focuses on what needs to be assessed and how it is measured, including assessments of practice, potential, and nontechnical aspects of practice. The second dimension is composed of three issues that cut across the assessment components: pathways to recertification, content of the assessments, and standards. It will take decades for the Boards work through these issues, but the result will be credibility in the eyes of the public and enhanced recognition in the eyes of the profession.