Applying Practice Analysis to Develop a New Test Content Outline for the Pediatric Cardiology Certification Examination.

The evolving breadth and complexity of the contemporary pediatric cardiology specialty requires regular, systematic analysis of the practice to ensure that training and certification requirements address the demands of real-world clinical experience. We report the process of the American Board of Pediatrics (ABP) for conducting such a practice analysis and revising the test content outline (TCO) for the pediatric cardiology subspecialty certification exam. A panel of 15 pediatric cardiologists conducted seven 2-h virtual meetings, during which they identified 37 unique tasks that represent the work a pediatric cardiologist may reasonably expect to perform within the first 5 years after training. These tasks were grouped into nine performance domains, similar to the entrustable professional activities (EPA), previously endorsed by the ABP in collaboration with the pediatric cardiology education community, and which represent the critical activities of the profession. The panel then enumerated the knowledge, skills, and abilities necessary to perform each task. These deliberations resulted in two work products: a practice analysis document (PAD) and subspecialty board TCO based on testable knowledge, skills, and abilities. Survey assessments of the panel's work were then distributed to pediatric cardiology fellowship program directors and to practicing pediatric cardiologists for their input, which largely aligned with the panel's recommendations. Survey responses were considered in the final revisions of the PAD and TCO. This approach to practice analysis proved to be an efficient process for describing the work performed by today's pediatric cardiologists and the knowledge, skills, and abilities needed to competently perform that work.

Practice Analysis and Determining the Knowledge and Skills Expected of a Pediatric Rheumatologist.

OBJECTIVE: The scope of clinical practice of pediatric rheumatology has been difficult to define. The lack of definition prevents an accurate understanding of the knowledge and skills required of practicing pediatric rheumatologists. A practice analysis process was used with the goal of establishing a precise definition of clinical pediatric rheumatology practice. The definition of practice will improve training and the creation of relevant certification examinations. METHODS: A practice analysis approach used meetings with a representative panel of pediatric rheumatologists to create a practice analysis document (PAD) and a test content outline (TCO). Panel experience, entrustable professional activities, and the current TCO were used to guide the process. Surveys were administered to fellowship program directors (PDs) and a broader group of practicing pediatric rheumatologists to revise and validate the content of the documents. RESULTS: A PAD was created, including 14 categories of conditions diagnosed or managed by pediatric rheumatologists and eight domains of practice, with the tasks, knowledge, and skills required to perform these tasks. The survey of PDs (n = 10) indicated that the PAD content is important and useful. A TCO was created and consists of 18 domains used to define content areas to be assessed on certifying examinations. The survey of practicing pediatric rheumatologists (n = 127) indicated that the TCO domains are relevant. CONCLUSION: A practice analysis process produced valuable resources for defining the clinical practice of pediatric rheumatology. The PAD and TCO can be used to develop more specific training curricula and to create relevant certification examinations.

Identifying and Addressing Item Bias in the General Pediatrics Certifying Examination.

This article describes a 2-phase process implemented by the American Board of Pediatrics in 2021 to investigate and remove potential bias on its General Pediatrics Certifying Examination at the item (question) level based on gender or race and ethnicity. Phase 1 used a statistical technique known as differential item functioning (DIF) analysis to identify items in which 1 subgroup of the population outperformed another subgroup after controlling for overall knowledge level. Phase 2 involved a review of items flagged for statistical DIF by the American Board of Pediatrics' Bias and Sensitivity Review (BSR) panel, a diverse group of 12 voluntary subject matter experts tasked with identifying language or other characteristics of those items that may have contributed to the observed performance differences. Results indicated that no items on the 2021 examination were flagged for DIF by gender and 2.8% of the items were flagged for DIF by race and ethnicity. Of those items flagged for race and ethnicity, 14.3% (0.4% of total items administered) were judged by the BSR panel to contain biased language that may have undermined what the item was intending to measure and were therefore recommended to be removed from operational scoring. In addition to removing potentially biased items from the current pool of items, we hope that repeating the DIF/BSR process after each examination cycle will increase our understanding of how language nuances and other characteristics impact item performance so that we can improve our guidelines for developing future items.