Modeling relationships between traditional preadmission measures and clinical skills performance on a medical licensure examination.

Medical schools employ a variety of preadmission measures to select students most likely to succeed in the program. The Medical College Admission Test (MCAT) and the undergraduate college grade point average (uGPA) are two academic measures typically used to select students in medical school. The assumption that presently used preadmission measures can predict clinical skill performance on a medical licensure examination was evaluated within a validity argument framework (Kane 1992). A hierarchical generalized linear model tested relationships between the log-odds of failing a high-stakes medical licensure performance examination and matriculant academic and non-academic preadmission measures, controlling for student-and school-variables. Data includes 3,189 matriculants from 22 osteopathic medical schools tested in 2009-2010. Unconditional unit-specific model expected average log-odds of failing the examination across medical schools is -3.05 (se = 0.11) or 5%. Student-level estimated coefficients for MCAT Verbal Reasoning scores (0.03), Physical Sciences scores (0.05), Biological Sciences scores (0.04), uGPA(science) (0.07), and uGPA(non-science) (0.26) lacked association with the log-odds of failing the COMLEX-USA Level 2-PE, controlling for all other predictors in the model. Evidence from this study shows that present preadmission measures of academic ability are not related to later clinical skill performance. Given that clinical skill performance is an important part of medical practice, selection measures should be developed to identify students who will be successful in communication and be able to demonstrate the ability to systematically collect a medical history, perform a physical examination, and synthesize this information to diagnose and manage patient conditions.

Competency-based classification of COMLEX-USA cognitive examination test items.

CONTEXT: The Comprehensive Osteopathic Medical Licensing Examination-USA (COMLEX-USA) currently assesses osteopathic medical knowledge via a series of 3 progressive cognitive examinations and 1 clinical skills assessment. In 2009, the National Board of Osteopathic Medical Examiners created the Fundamental Osteopathic Medical Competencies (FOMC) document to outline the essential competencies required for the practice of osteopathic medicine. OBJECTIVES: To measure the distribution and extent to which cognitive examination items of the current series of COMLEX-USA assess knowledge of each of the medical competencies included in the FOMC document. METHODS: Eight graduate medical education panelists with expertise in competency-based assessment reviewed 1046 multiple-choice examination items extracted from the 3 COMLEX-USA cognitive examinations (Level 1, Level 2-Cognitive Evaluation, and Level 3) used during the 2008-2009 testing cycle. The 8 panelists individually judged each item to classify it as 1 of the 6 fundamental osteopathic medical competencies described in the FOMC document. RESULTS: Panelists made 8368 judgments. The majority of the sample examination items were classified as either patient care (3343 [40%]) or medical knowledge (4236 [51%]). Panelists also reported these 2 competencies as being the easiest to define, teach, and assess. The frequency of medical knowledge examination items decreased throughout the COMLEX-USA series (69%, 43%, 40%); conversely, items classified as interpersonal and communication skills, systems-based practice, practice-based learning and improvement, and professionalism increased throughout the 3-examination series. CONCLUSION: Results indicate that knowledge of each of the 6 competencies is being assessed to some extent with the current COMLEX-USA format. These findings provide direction for the enhancement of existing examinations and development of new assessment tools.

Relationships between high-stakes clinical skills exam scores and program director global competency ratings of first-year pediatric residents.

BACKGROUND: Responding to mandates from the Accreditation Council for Graduate Medical Education (ACGME) and American Osteopathic Association (AOA), residency programs have developed competency-based assessment tools. One such tool is the American College of Osteopathic Pediatricians (ACOP) program directors' annual report. High-stakes clinical skills licensing examinations, such as the Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation (COMLEX-USA Level 2-PE), also assess competency in several clinical domains. OBJECTIVE: The purpose of this study is to investigate the relationships between program director competency ratings of first-year osteopathic residents in pediatrics and COMLEX-USA Level 2-PE scores from 2005 to 2009. METHODS: The sample included all 94 pediatric first-year residents who took COMLEX-USA Level 2-PE and whose training was reviewed by the ACOP for approval of training between 2005 and 2009. Program director competency ratings and COMLEX-USA Level 2-PE scores (domain and component) were merged and analyzed for relationships. RESULTS: Biomedical/biomechanical domain scores were positively correlated with overall program director competency ratings. Humanistic domain scores were not significantly correlated with overall program director competency ratings, but did show moderate correlation with ratings for interpersonal and communication skills. The six ACGME or seven AOA competencies assessed empirically by the ACOP program directors' annual report could not be recovered by principal component analysis; instead, three factors were identified, accounting for 86% of the variance between competency ratings. DISCUSSION: A few significant correlations were noted between COMLEX-USA Level 2-PE scores and program director competency ratings. Exploring relationships between different clinical skills assessments is inherently difficult because of the heterogeneity of tools used and overlap of constructs within the AOA and ACGME core competencies.