Medical Knowledge Assessment by Hematology and Medical Oncology In-Training Examinations Are Better Than Program Director Assessments at Predicting Subspecialty Certification Examination Performance.

The Accreditation Council for Graduate Medical Education's Next Accreditation System requires training programs to demonstrate that fellows are achieving competence in medical knowledge (MK), as part of a global assessment of clinical competency. Passing American Board of Internal Medicine (ABIM) certification examinations is recognized as a metric of MK competency. This study examines several in-training MK assessment approaches and their ability to predict performance on the ABIM Hematology or Medical Oncology Certification Examinations. Results of a Hematology In-Service Examination (ISE) and an Oncology In-Training Examination (ITE), program director (PD) ratings, demographic variables, United States Medical Licensing Examination (USMLE), and ABIM Internal Medicine (IM) Certification Examination were compared. Stepwise multiple regression and logistic regression analyses evaluated these assessment approaches as predictors of performance on the Hematology or Medical Oncology Certification Examinations. Hematology ISE scores were the strongest predictor of Hematology Certification Examination scores (ß = 0.41) (passing odds ratio [OR], 1.012; 95 % confidence interval [CI], 1.008-1.015), and the Oncology ITE scores were the strongest predictor of Medical Oncology Certification Examination scores (ß = 0.45) (passing OR, 1.013; 95 % CI, 1.011-1.016). PD rating of MK was the weakest predictor of Medical Oncology Certification Examination scores (ß = 0.07) and was not significantly predictive of Hematology Certification Examination scores. Hematology and Oncology ITEs are better predictors of certification examination performance than PD ratings of MK, reinforcing the effectiveness of ITEs for competency-based assessment of MK.

Association of Physician Certification in Interventional Cardiology With In-Hospital Outcomes of Percutaneous Coronary Intervention.

BACKGROUND: The value of American Board of Internal Medicine certification has been questioned. We evaluated the Association of Interventional Cardiology certification with in-hospital outcomes of patients undergoing percutaneous coronary intervention (PCI) in 2010. METHODS AND RESULTS: We identified physicians who performed ≥10 PCIs in 2010 in the CathPCI Registry and determined interventional cardiology (ICARD) certification status using American Board of Internal Medicine data. We compared in-hospital outcomes of patients treated by certified and noncertified physicians using hierarchical multivariable models adjusted for differences in patient characteristics and PCI volume. Primary end points were all-cause in-hospital mortality and bleeding complications. Secondary end points included emergency coronary artery bypass grafting, vascular complications, and a composite of any adverse outcome. With 510,708 PCI procedures performed by 5175 physicians, case mix and unadjusted outcomes were similar among certified and noncertified physicians. The adjusted risks of in-hospital mortality (odds ratio, 1.10; 95% confidence interval, 1.02-1.19) and emergency coronary artery bypass grafting (odds ratio, 1.32; 95% confidence interval, 1.12-1.56) were higher in the non-ICARD-certified group, but the risks of bleeding and vascular complications and the composite end point were not statistically significantly different between groups. CONCLUSIONS: We did not observe a consistent association between ICARD certification and the outcomes of PCI procedures. Although there was a significantly higher risk of mortality and emergency coronary artery bypass grafting in patients treated by non-ICARD-certified physicians, the risks of vascular complications and bleeding were similar. Our findings suggest that ICARD certification status alone is not a strong predictor of patient outcomes and indicate a need to enhance the value of subspecialty certification.

The relationship between performance on the Infectious Diseases In-Training and Certification Examinations.

BACKGROUND: The Infectious Diseases Society of America In-Training Examination (IDSA ITE) is a feedback tool used to help fellows track their knowledge acquisition during fellowship training. We determined whether the scores on the IDSA ITE and from other major medical knowledge assessments predict performance on the American Board of Internal Medicine (ABIM) Infectious Disease Certification Examination. METHODS: The sample was 1021 second-year fellows who took the IDSA ITE and ABIM Infectious Disease Certification Examination from 2008 to 2012. Multiple regression analysis was used to determine if ABIM Infectious Disease Certification Examination scores were predicted by IDSA ITE scores, prior United States Medical Licensing Examination (USMLE) scores, ABIM Internal Medicine Certification Examination scores, fellowship director ratings of medical knowledge, and demographic variables. Logistic regression was used to evaluate if these same assessments predicted a passing outcome on the certification examination. RESULTS: IDSA ITE scores were the strongest predictor of ABIM Infectious Disease Certification Examination scores (ß = .319), followed by prior ABIM Internal Medicine Certification Examination scores (ß = .258), USMLE Step 1 scores (ß = .202), USMLE Step 3 scores (ß = .130), and fellowship directors' medical knowledge ratings (ß = .063). IDSA ITE scores were also a significant predictor of passing the Infectious Disease Certification Examination (odds ratio, 1.017 [95% confidence interval, 1.013-1.021]). CONCLUSIONS: The significant relationship between the IDSA ITE score and performance on the ABIM Infectious Disease Certification Examination supports the use of the ITE as a valid feedback tool in fellowship training.

Assessing the Quality of Osteoporosis Care in Practice.

BACKGROUND: Patients with osteoporosis can sustain fractures following falls or other minimal trauma. This risk of fracture can be reduced through appropriate diagnostic testing, pharmacologic therapy, and other readily measured standards of care. OBJECTIVES: Our aim was to develop a credible clinical performance assessment to measure physicians' quality of osteoporosis care, and determine reasonable performance standards for both competent and excellent care. DESIGN: This was a retrospective cohort study. PARTICIPANTS: Three hundred and eighty one general internists and subspecialists with time-limited board certification were included in the study. MAIN MEASURES: Performance rates on eight evidence-based measures were obtained from the American Board of Internal Medicine (ABIM) Osteoporosis Practice Improvement Module® (PIM), a web-based tool that uses medical chart reviews to help physicians assess and improve care. We applied a patented methodology, using an adaptation of the Angoff standard-setting method and the Dunn-Rankin method, with an expert panel skilled in osteoporosis care to form a composite and establish standards for both competent and excellent care. Physician and practice characteristics, including a practice infrastructure score based on the Physician Practice Connections Readiness Survey (PPC-RS), were used to examine the validity of the inferences made from the composite scores. KEY RESULTS: The mean composite score was 67.54 out of 100 maximum points with a reliability of 0.92. The standard for competent care was 46.87, and for excellent care it was 83.58. Both standards had high classification accuracies (0.95). Sixteen percent of physicians performed below the competent care standard, while 22 % met the excellent care standard. Specialists scored higher than generalists, and better practice infrastructure was associated with higher composite scores, providing some validity evidence. CONCLUSIONS: We developed a rigorous methodology for assessing physicians' osteoporosis care. Clinical performance feedback relative to absolute standards of care provides physicians with a meaningful approach to self-evaluation to improve patient care.

Examining changes in certification/licensure requirements and the international medical graduate examinee pool.

Changes in certification requirements and examinee characteristics are likely to influence the validity of the evidence associated with interpretations made based on test data. We examined whether changes in Educational Commission for Foreign Medical Graduates (ECFMG) certification requirements over time were associated with changes in internal medicine (IM) residency program director ratings and certification examination scores. Comparisons were made between physicians who were ECFMG-certified before and after the Clinical Skills Assessment (CSA) requirement. A multivariate analysis of covariance was conducted to examine the differences in program director ratings based on CSA cohort and whether the examinees emigrated for undergraduate medical education (national vs. international students). A univariate analysis of covariance was conducted to examine differences in scores from the American Board of Internal Medicine (ABIM) Internal Medicine Certification Examination. For both analyses, United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores were used as covariates. Results indicate that, of those certified by ECFMG between 1993 and 1997, 17 % (n = 1,775) left their country of citizenship for undergraduate medical education. In contrast, 38 % (n = 1,874) of those certified between 1999 and 2003 were international students. After adjustment by covariates, the main effect of cohort membership on the program director ratings was statistically significant (Wilks' λ = 0.99, F 5, 15391 = 19.9, P < 0.001). However, the strength of the relationship between cohort group and the ratings was weak (η = 0.01). The main effect of migration status was statistically significant and weak (Wilks' λ = 0.98, F 5,15391 = 45.3, P < 0.01; η = 0.02). Differences in ABIM Internal Medicine Certification Examination scores based on whether or not CSA were required was statistically significant, although the magnitude of the association between these variables was very small. The findings suggest that the implementation of an additional evaluation of skills (e.g., history-taking, physical examination) as a prerequisite to postgraduate medical education (residency) provides some additional, relevant data to those who select ECFMG-certified residents.

Physician performance assessment: prevention of cardiovascular disease.

Given the rising burden of healthcare costs, both patients and healthcare purchasers are interested in discerning which physicians deliver quality care. We proposed a methodology to assess physician clinical performance in preventive cardiology care, and determined a benchmark for minimally acceptable performance. We used data on eight evidence-based clinical measures from 811 physicians that completed the American Board of Internal Medicine's Preventive Cardiology Practice Improvement Module(SM) to form an overall composite score for preventive cardiology care. An expert panel of nine internists/cardiologists skilled in preventive care for cardiovascular disease used an adaptation of the Angoff standard-setting method and the Dunn-Rankin method to create the composite and establish a standard. Physician characteristics were used to examine the validity of the inferences made from the composite scores. The mean composite score was 73.88 % (SD = 11.88 %). Reliability of the composite was high at 0.87. Specialized cardiologists had significantly lower composite scores (P = 0.04), while physicians who reported spending more time in primary, longitudinal, and preventive consultative care had significantly higher scores (P = 0.01), providing some evidence of score validity. The panel established a standard of 47.38 % on the composite measure with high classification accuracy (0.98). Only 2.7 % of the physicians performed below the standard for minimally acceptable preventive cardiovascular disease care. Of those, 64 % (N = 14) were not general cardiologists. Our study presents a psychometrically defensible methodology for assessing physician performance in preventive cardiology while also providing relative feedback with the hope of heightening physician awareness about deficits and improving patient care.

Development of an instrument to evaluate residents' confidence in quality improvement.

BACKGROUND: Practice-based learning and improvement is a core competency that all medical residents must demonstrate. Because confidence is important in translating competence into action, effective quality improvement (QI) curricula should evaluate trainees' knowledge and confidence to perform QI. Past efforts to assess educational outcomes in QI have not adequately evaluated trainees' confidence from a multidimensional perspective. METHODS: Participants--732 internal medicine and family medicine residents from 42 training programs in the United States--completed the 31-item Quality Improvement Confidence Instrument (QICI), which was developed to measure confidence in six QI skill domains based on the Institute for Healthcare Improvement model ofQI. Confirmatory factor analysis was performed to support construct validity. Multivariate analysis of covariance was used to examine associations between residents' QI experience and other characteristics with confidence scores. RESULTS: Confirmatory factor analysis supported the QICI's multidimensional structure. Individual items yielded adequate variability, and reliability estimates for all six domains were high (> 0.86). On average, residents rated their confidence lowest for skills pertaining to choosing a target for improvement (specifically, using methods to evaluate interventions and to identify sources of process errors) and for testing a change in practice using specific tools for data collection and analysis. After controlling for program year and other characteristics, residents with previous QI experience reported significantly greater QI confidence. CONCLUSION: The QICI offers a psychometrically rigorous approach to evaluating residents' confidence levels. It can be used to gauge the appropriateness of a trainee's confidence against actual QI performance.

Setting a fair performance standard for physicians' quality of patient care.

BACKGROUND: Assessing physicians' clinical performance using statistically sound, evidence-based measures is challenging. Little research has focused on methodological approaches to setting performance standards to which physicians are being held accountable. OBJECTIVE: Determine if a rigorous approach for setting an objective, credible standard of minimally-acceptable performance could be used for practicing physicians caring for diabetic patients. DESIGN: Retrospective cohort study. PARTICIPANTS: Nine hundred and fifty-seven physicians from the United States with time-limited certification in internal medicine or a subspecialty. MAIN MEASURES: The ABIM Diabetes Practice Improvement Module was used to collect data on ten clinical and two patient experience measures. A panel of eight internists/subspecialists representing essential perspectives of clinical practice applied an adaptation of the Angoff method to judge how physicians who provide minimally-acceptable care would perform on individual measures to establish performance thresholds. Panelists then rated each measure's relative importance and the Dunn-Rankin method was applied to establish scoring weights for the composite measure. Physician characteristics were used to support the standard-setting outcome. KEY RESULTS: Physicians abstracted 20,131 patient charts and 18,974 patient surveys were completed. The panel established reasonable performance thresholds and importance weights, yielding a standard of 48.51 (out of 100 possible points) on the composite measure with high classification accuracy (0.98). The 38 (4%) outlier physicians who did not meet the standard had lower ratings of overall clinical competence and professional behavior/attitude from former residency program directors (p = 0.01 and p = 0.006, respectively), lower Internal Medicine certification and maintenance of certification examination scores (p = 0.005 and p < 0.001, respectively), and primarily worked as solo practitioners (p = 0.02). CONCLUSIONS: The standard-setting method yielded a credible, defensible performance standard for diabetes care based on informed judgment that resulted in a reasonable, reproducible outcome. Our method represents one approach to identifying outlier physicians for intervention to protect patients.

Clinical protocols and trainee knowledge about mechanical ventilation.

CONTEXT: Clinical protocols are associated with improved patient outcomes; however, they may negatively affect medical education by removing trainees from clinical decision making. OBJECTIVE: To study the relationship between critical care training with mechanical ventilation protocols and subsequent knowledge about ventilator management. DESIGN, SETTING, AND PARTICIPANTS: A retrospective cohort equivalence study, linking a national survey of mechanical ventilation protocol availability in accredited US pulmonary and critical care fellowship programs with knowledge about mechanical ventilation among first-time examinees of the American Board of Internal Medicine (ABIM) Critical Care Medicine Certification Examination in 2008 and 2009. Exposure to protocols was defined as high intensity if an examinee's training intensive care unit had 2 or more protocols for at least 3 years and as low intensity if 0 or 1 protocol. MAIN OUTCOME MEASURES: Knowledge, measured by performance on examination questions specific to mechanical ventilation management, calculated as a mechanical ventilation score using item response theory. The score is standardized to a mean (SD) of 500 (100), and a clinically important difference is defined as 25. Variables included in adjusted analyses were birth country, residency training country, and overall first-attempt score on the ABIM Internal Medicine Certification Examination. RESULTS: Ninety of 129 programs (70%) responded to the survey. Seventy-seven programs (86%) had protocols for ventilation liberation, 66 (73%) for sedation management, and 54 (60%) for lung-protective ventilation at the time of the survey. Eighty-eight (98%) of these programs had trainees who completed the ABIM Critical Care Medicine Certification Examination, totaling 553 examinees. Of these 88 programs, 27 (31%) had 0 protocols, 19 (22%) had 1 protocol, 24 (27%) had 2 protocols, and 18 (20%) had 3 protocols for at least 3 years. Forty-two programs (48%) were classified as high intensity and 46 (52%) as low intensity, with 304 trainees (55%) and 249 trainees (45%), respectively. In bivariable analysis, no difference in mean scores was observed in high-intensity (497; 95% CI, 486-507) vs low-intensity programs (497; 95% CI, 485-509). Mean difference was 0 (95% CI, -16 to 16), with a positive value indicating a higher score in the high-intensity group. In multivariable analyses, no association of training was observed in a high-intensity program with mechanical ventilation score (adjusted mean difference, -5.36; 95% CI, -20.7 to 10.0). CONCLUSION: Among first-time ABIM Critical Care Medicine Certification Examination examinees, training in a high-intensity ventilator protocol environment compared with a low-intensity environment was not associated with worse performance on examination questions about mechanical ventilation management.

Cranial flap fixation in sheep using a resorbable bone adhesive.

OBJECTIVE: The authors' goal in this study was to investigate the use of a novel, bioresorbable, osteoconductive, wet-field mineral-organic bone adhesive composed of tetracalcium phosphate and phosphoserine (TTCP-PS) for cranial bone flap fixation and compare it with conventional low-profile titanium plates and self-drilling screws. METHODS: An ovine craniotomy surgical model was used to evaluate the safety and efficacy of TTCP-PS over 2 years. Bilateral cranial defects were created in 41 sheep and were replaced in their original position. The gaps (kerfs) were completely filled with TTCP-PS (T1 group), half-filled with TTCP-PS (T2 group), or left empty and the flaps fixated by plates and screws as a control (C group). At 12 weeks, 1 year, and 2 years following surgery, the extent of bone healing, local tissue effects, and remodeling of the TTCP-PS were analyzed using macroscopic observations and histopathological and histomorphometric analyses. Flap fixation strength was evaluated by biomechanical testing at 12 weeks and 1 year postoperatively. RESULTS: No adverse local tissue effects were observed in any group. At 12 weeks, the bone flap fixation strengths in test group 1 (1689 ± 574 N) and test group 2 (1611 ± 501 N) were both statistically greater (p = 0.01) than that in the control group (663 ± 385 N). From 12 weeks to 1 year, the bone flap fixation strengths increased significantly (p < 0.05) for all groups. At 1 year, the flap fixation strength in test group 1 (3240 ± 423 N) and test group 2 (3212 ± 662 N) were both statistically greater (p = 0.04 and p = 0.02, respectively) than that in the control group (2418 ± 1463 N); however, there was no statistically significant difference in the strengths when comparing the test groups at both timepoints. Test group 1 had the best overall performance based on histomorphometric evaluation and biomechanical testing. At 2 years postoperatively, the kerfs filled with TTCP-PS had histological evidence of osteoconduction and replacement of TTCP-PS by bone with nearly complete osteointegration. CONCLUSIONS: TTCP-PS was demonstrated to be safe and effective for cranial flap fixation in an ovine model. In this study, the bioresorbable, osteoconductive bone adhesive appeared to have multiple advantages over standard plate-and-screw bone flap fixation, including biomechanical superiority, more complete and faster bony healing across the flap kerfs without fibrosis, and the minimization of bone flap and/or hardware migration and loosening. These properties of TTCP-PS may improve human cranial bone flap fixation and cranioplasty.

Pulmonary and Critical Care In-Service Training Examination Score as a Predictor of Board Certification Examination Performance.

RATIONALE: Most trainees in combined pulmonary and critical care medicine fellowship programs complete in-service training examinations (ITEs) that test knowledge in both disciplines. Whether ITE scores predict performance on the American Board of Internal Medicine Pulmonary Disease Certification Examination and Critical Care Medicine Certification Examination is unknown. OBJECTIVES: To determine whether pulmonary and critical care medicine ITE scores predict performance on subspecialty board certification examinations independently of trainee demographics, program director competency ratings, fellowship program characteristics, and prior medical knowledge assessments. METHODS: First- and second-year fellows who were enrolled in the study between 2008 and 2012 completed a questionnaire encompassing demographics and fellowship training characteristics. These data and ITE scores were matched to fellows' subsequent scores on subspecialty certification examinations, program director ratings, and previous scores on their American Board of Internal Medicine Internal Medicine Certification Examination. Multiple linear regression and logistic regression were used to identify independent predictors of subspecialty certification examination scores and likelihood of passing the examinations, respectively. MEASUREMENTS AND MAIN RESULTS: Of eligible fellows, 82.4% enrolled in the study. The ITE score for second-year fellows was matched to their certification examination scores, which yielded 1,484 physicians for pulmonary disease and 1,331 for critical care medicine. Second-year fellows' ITE scores (ß = 0.24, P < 0.001) and Internal Medicine Certification Examination scores (ß = 0.49, P < 0.001) were the strongest predictors of Pulmonary Disease Certification Examination scores, and were the only significant predictors of passing the examination (ITE odds ratio, 1.12 [95% confidence interval, 1.07-1.16]; Internal Medicine Certification Examination odds ratio, 1.01 [95% confidence interval, 1.01-1.02]). Similar results were obtained for predicting Critical Care Medicine Certification Examination scores and for passing the examination. The predictive value of ITE scores among first-year fellows on the subspecialty certification examinations was comparable to second-year fellows' ITE scores. CONCLUSIONS: The Pulmonary and Critical Care Medicine ITE score is an independent, and stronger, predictor of subspecialty certification examination performance than fellow demographics, program director competency ratings, and fellowship characteristics. These findings support the use of the ITE to identify the learning needs of fellows as they work toward subspecialty board certification.

Blink or think: can further reflection improve initial diagnostic impressions?

PURPOSE: Experienced clinicians derive many diagnoses intuitively, because most new problems they see closely resemble problems they've seen before. The majority of these diagnoses, but not all, will be correct. This study determined whether further reflection regarding initial diagnoses improves diagnostic accuracy during a high-stakes board exam, a model for studying clinical decision making. METHOD: Keystroke response data were used from 500 residents who took the 2010 American Board of Internal Medicine (ABIM) Internal Medicine Certification Examination. Data included time to initial response on each question, whether the answer was correct, and whether or not the resident changed her or his initial response. The focus was on 80 diagnosis questions that comprised realistic clinical vignettes with multiple-choice single-best answers. Cognitive skill (ability) was measured using overall exam scores. Case complexity was determined using item difficulty (proportion of examinees that correctly answered the question). A hierarchical generalized linear model was used to assess the relationship between time spent on initial responses and the probability of correctly answering the questions. RESULTS: On average, residents changed their responses on 12% of all diagnosis questions (or 9.6 questions out of 80). Changing an answer from incorrect to correct was almost twice as likely as changing an answer from correct to incorrect. The relationship between response time and accuracy was complex. CONCLUSIONS: Further reflection appears to be beneficial to diagnostic accuracy, especially for more complex cases.

Performance on the adult rheumatology in-training examination and relationship to outcomes on the rheumatology certification examination.

OBJECTIVE: The American College of Rheumatology (ACR) Adult Rheumatology In-Training Examination (ITE) is a feedback tool designed to identify strengths and weaknesses in the content knowledge of individual fellows-in-training and the training program curricula. We determined whether scores on the ACR ITE, as well as scores on other major standardized medical examinations and competency-based ratings, could be used to predict performance on the American Board of Internal Medicine (ABIM) Rheumatology Certification Examination. METHODS: Between 2008 and 2012, 629 second-year fellows took the ACR ITE. Bivariate correlation analyses of assessment scores and multiple linear regression analyses were used to determine whether ABIM Rheumatology Certification Examination scores could be predicted on the basis of ACR ITE scores, United States Medical Licensing Examination scores, ABIM Internal Medicine Certification Examination scores, fellowship directors' ratings of overall clinical competency, and demographic variables. Logistic regression was used to evaluate whether these assessments were predictive of a passing outcome on the Rheumatology Certification Examination. RESULTS: In the initial linear model, the strongest predictors of the Rheumatology Certification Examination score were the second-year fellows' ACR ITE scores (ß = 0.438) and ABIM Internal Medicine Certification Examination scores (ß = 0.273). Using a stepwise model, the strongest predictors of higher scores on the Rheumatology Certification Examination were second-year fellows' ACR ITE scores (ß = 0.449) and ABIM Internal Medicine Certification Examination scores (ß = 0.276). Based on the findings of logistic regression analysis, ACR ITE performance was predictive of a pass/fail outcome on the Rheumatology Certification Examination (odds ratio 1.016 [95% confidence interval 1.011-1.021]). CONCLUSION: The predictive value of the ACR ITE score with regard to predicting performance on the Rheumatology Certification Examination supports use of the Adult Rheumatology ITE as a valid feedback tool during fellowship training.

Internists' views of maintenance of certification: a stages-of-change perspective.

INTRODUCTION: Board certification has evolved from a "point-in-time" event to a process of periodic learning and reevaluation of medical competence through maintenance of certification (MOC). To better understand MOC participation, the transtheoretical model (TTM) was used to describe physicians' perceptions of MOC as a sequence of attitudinal changes. METHOD: Data were from a survey of internal medicine (IM) physicians' attitudes toward periodic reevaluation through MOC. An overall importance or decisional balance score was computed for each physician by summing his or her ratings across the 10 quality measures. The decisional balance score was used to classify physicians according to their acceptance of MOC, aligned with the 3 early TTM stages-of-change groups-precontemplation (PC), contemplation (C), and preparation (P)-where PC was least accepting and P was most accepting. Effect sizes assessed whether differences in attitudes toward reevaluation via MOC were of sufficient magnitude to support the TTM principles. RESULTS: The difference in degree of acceptance of MOC between the P group and the PC and C groups was significant (p < 0.001), but the effect size was lower than predicted by the "strong" principle. Resistance to MOC for the PC and C groups was significantly greater than the P group (p < 0.001) and supported the "weak" principle. Physicians' beliefs about how often they should demonstrate performance on quality measures aligned well with the American Board of Internal Medicine's MOC requirements, with the P group believing in more frequent assessments than the PC and C groups (p < 0.001). CONCLUSIONS: Results show that physicians in the Preparation stage had overcome resistance to MOC as predicted by the "weak" principle of the TTM, but their attitude scores about the benefits of MOC were below what was expected by theory. This suggests that the structure of MOC may have made it easier for physicians to overcome barriers to MOC participation but may have lacked adequate resources to promote the benefits of participating in the process. More effort is needed to understand the specific benefits of MOC for reevaluating competencies, how to engage physicians and other stakeholders in the design of MOC, and how to communicate the rationale and evidence to those who are less accepting of MOC.

Specialty board certification in the United States: issues and evidence.

BACKGROUND: The American Board of Medical Specialties (ABMS) certification and maintenance of certification (MOC) programs strive to provide the public with guidance about a physician's competence. This study summarizes the literature on the effectiveness of these programs. METHOD: A literature search was conducted for studies published between 1986 and April 2013 and limited to ABMS certification. A modified version of Kirkpatrick's 4 levels of program evaluation included the reaction of stakeholders to certification, the extent to which physicians are encouraged to improve, the relationship between performance in the programs and nonclinical external measures of physician competence, and the relationship of performance in the programs with clinical quality measures. RESULTS: Patients' and hospitals' value of board certification and physician participation in MOC are high. Physicians are conflicted as to whether the effort involved is worth its value. Self-reported evidence shows improvement in knowledge, practice infrastructure, communication with patients and peers, and clinical care. Certification performance is generally related to nonclinical external measures such as types of training, practice characteristics, demographics, and disciplinary actions. In general, physicians who are board certified provide better patient care, albeit the results have modest effect sizes and are not unequivocal. CONCLUSIONS: Certification boards should continuously try to improve their programs in response to feedback from stakeholders, changes in the way physicians practice, as well as the growth in the fields of measurement and technology. Keeping pace with these changes in a responsible and evidence-based way is important.

Practice-based learning can improve osteoporosis care.

OBJECTIVES: To examine physician engagement in practice-based learning using a self-evaluation module to assess and improve their care of individuals with or at risk of osteoporosis. DESIGN: Retrospective cohort study. SETTING: Internal medicine and subspecialty clinics. PARTICIPANTS: Eight hundred fifty U.S. physicians with time-limited certification in general internal medicine or a subspecialty. MEASUREMENTS: Performance rates on 23 process measures and seven practice system domain scores were obtained from the American Board of Internal Medicine (ABIM) Osteoporosis Practice Improvement Module (PIM), an Internet-based self-assessment module that physicians use to improve performance on one targeted measure. Physicians remeasured performance on their targeted measures by conducting another medical chart review. RESULTS: Variability in performance on measures was found, with observed differences between general internists, geriatricians, and rheumatologists. Some practice system elements were modestly associated with measure performance; the largest association was between providing patient-centered self-care support and documentation of calcium intake and vitamin D estimation and counseling (correlation coefficients from 0.20 to 0.28, Ps < .002). For all practice types, the most commonly selected measure targeted for improvement was documentation of vitamin D level (38% of physicians). On average, physicians reported significant and large increases in performance on measures targeted for improvement. CONCLUSION: Gaps exist in the quality of osteoporosis care, and physicians can apply practice-based learning using the ABIM PIM to take action to improve the quality of care.