Identifying and Addressing Item Bias in the General Pediatrics Certifying Examination.

This article describes a 2-phase process implemented by the American Board of Pediatrics in 2021 to investigate and remove potential bias on its General Pediatrics Certifying Examination at the item (question) level based on gender or race and ethnicity. Phase 1 used a statistical technique known as differential item functioning (DIF) analysis to identify items in which 1 subgroup of the population outperformed another subgroup after controlling for overall knowledge level. Phase 2 involved a review of items flagged for statistical DIF by the American Board of Pediatrics' Bias and Sensitivity Review (BSR) panel, a diverse group of 12 voluntary subject matter experts tasked with identifying language or other characteristics of those items that may have contributed to the observed performance differences. Results indicated that no items on the 2021 examination were flagged for DIF by gender and 2.8% of the items were flagged for DIF by race and ethnicity. Of those items flagged for race and ethnicity, 14.3% (0.4% of total items administered) were judged by the BSR panel to contain biased language that may have undermined what the item was intending to measure and were therefore recommended to be removed from operational scoring. In addition to removing potentially biased items from the current pool of items, we hope that repeating the DIF/BSR process after each examination cycle will increase our understanding of how language nuances and other characteristics impact item performance so that we can improve our guidelines for developing future items.

Applying Practice Analysis to Develop a New Test Content Outline for the Pediatric Cardiology Certification Examination.

The evolving breadth and complexity of the contemporary pediatric cardiology specialty requires regular, systematic analysis of the practice to ensure that training and certification requirements address the demands of real-world clinical experience. We report the process of the American Board of Pediatrics (ABP) for conducting such a practice analysis and revising the test content outline (TCO) for the pediatric cardiology subspecialty certification exam. A panel of 15 pediatric cardiologists conducted seven 2-h virtual meetings, during which they identified 37 unique tasks that represent the work a pediatric cardiologist may reasonably expect to perform within the first 5 years after training. These tasks were grouped into nine performance domains, similar to the entrustable professional activities (EPA), previously endorsed by the ABP in collaboration with the pediatric cardiology education community, and which represent the critical activities of the profession. The panel then enumerated the knowledge, skills, and abilities necessary to perform each task. These deliberations resulted in two work products: a practice analysis document (PAD) and subspecialty board TCO based on testable knowledge, skills, and abilities. Survey assessments of the panel's work were then distributed to pediatric cardiology fellowship program directors and to practicing pediatric cardiologists for their input, which largely aligned with the panel's recommendations. Survey responses were considered in the final revisions of the PAD and TCO. This approach to practice analysis proved to be an efficient process for describing the work performed by today's pediatric cardiologists and the knowledge, skills, and abilities needed to competently perform that work.

Developing Content for Pediatric Hospital Medicine Certification Examination Using Practice Analysis.

OBJECTIVES: The American Board of Pediatrics (ABP) and the Pediatric Hospital Medicine (PHM) subboard developed a content outline to serve as a blueprint for the inaugural certification examination through practice analysis. The systematic approach of practice analyses process is described in the study. METHODS: A diverse, representative panel of 12 pediatric hospitalists developed the draft content outline using multiple resources (publications, textbooks, PHM Core Competencies, PHM fellow's curriculum, etc). The panel categorized practice knowledge into 13 domains and 202 subdomains. By using the ABP database self-defined practicing pediatric hospitalists were identified. Participants rated the frequency and criticality of content domains and subdomains along with providing open-ended comments. RESULTS: In total, 1449 (12.1%) generalists in the ABP database self-identified as pediatric hospitalists, and 800 full-time pediatric hospitalists responded. The content domains that were rated as highly critical and frequently required in practice were weighted more heavily (ie, the percentage of examination questions associated with a domain) than the less critical and less frequently rated. Both community and noncommunity pediatric hospitalists rated domains similarly (P = .943). Subdomain and preliminary weights were rated with similar means and SDs in the majority of topics. CONCLUSIONS: There was concordance in the rating of domain and universal tasks among both community and noncommunity hospitalists. The areas of significant differences, although minor, could be explained by difference in practice settings. The practice analysis approach was structured, engaged the PHM community, reflected the breadth and depth of knowledge required for PHM practice, and used an iterative process to refine the final product.