Validating Parallel-Forms Tests for Assessing Anesthesia Resident Knowledge.

We created a serious game to teach first year anesthesiology (CA-1) residents to perform general anesthesia for cesarean delivery. We aimed to investigate resident knowledge gains after playing the game and having received one of 2 modalities of debriefing. We report on the development and validation of scores from parallel test forms for criterion-referenced interpretations of resident knowledge. The test forms were intended for use as pre- and posttests for the experiment. Validation of instruments measuring the study's primary outcome was considered essential for adding rigor to the planned experiment, to be able to trust the study's results. Parallel, multiple-choice test forms development steps included: (1) assessment purpose and population specification; (2) content domain specification and writing/selection of items; (3) content validation by experts of paired items by topic and cognitive level; and (4) empirical validation of scores from the parallel test forms using Classical Test Theory (CTT) techniques. Field testing involved online administration of 52 shuffled items from both test forms to 24 CA-1's, 21 second-year anesthesiology (CA-2) residents, 2 fellows, 1 attending anesthesiologist, and 1 of unknown rank at 3 US institutions. Items from each form yielded near-normal score distributions, with similar medians, ranges, and standard deviations. Evaluations of CTT item difficulty (item p values) and discrimination (D) indices indicated that most items met assumptions of criterion-referenced test design, separating experienced from novice residents. Experienced residents performed better on overall domain scores than novices (P < .05). Kuder-Richardson Formula 20 (KR-20) reliability estimates of both test forms were above the acceptability cut of .70, and parallel forms reliability estimate was high at .86, indicating results were consistent with theoretical expectations. Total scores of parallel test forms demonstrated item-level validity, strong internal consistency and parallel forms reliability, suggesting sufficient robustness for knowledge outcomes assessments of CA-1 residents.

Survey Validation for Measuring Perceptions of Work-Related Factors That Influence Career Paths of Men and Women in Academic Anesthesiology.

BACKGROUND: Women's underrepresentation in positions of leadership in medicine has been attributed to environmental, structural, motivational, and situational factors. The purpose of this study was to design and validate a survey instrument based on these constructs, using a sample of men and women anesthesiologists from 3 urban academic medical centers. METHODS: Following institutional review board review, survey domains were defined based on a literature review. Items were developed, and content validation was performed by external experts. Anesthesiologists at 3 academic institutions were invited to complete the anonymous survey. Validation measures were performed on the collected responses, including reliability, convergent, and discriminant validity. In addition, differences between men and women respondents were evaluated. RESULTS: Content validation by external experts yielded 38 items with 5-point Likert scales, defining 3 constructs: environmental (14 items), structural (13 items), and motivational (11 items) factors, with single-item measures on situational factors. Content validity indices used Cohen's Kappa coefficients, with 0.85 as the acceptance cutoff. Two hundred seventy-four anesthesiologists in 3 academic institutions received the online survey. One hundred fifteen responses were received (42% response rate), yielding 103 complete survey responses, of which 86 cases included gender. Cronbach's α reliability estimates for the environmental, structural, and motivational scale scores were .88, .84, and .64, respectively, after scale revision. Evidence of convergent (Pearson's r = 0.68; P < .001) and discriminant validity (Pearson's r = 0.017; P = .84) confirmed theoretical expectations. Gender group differences showed statistically significant differences in perceptions toward environmental but not toward structural and motivational factors. CONCLUSIONS: The iterative design and validation processes yielded a 3-scale survey instrument with parsimonious item sets. The preliminary evidence of construct validity and reliability fills a gap in the instrumentation literature for assessing gender issues in medicine. Findings were consistent with theoretical expectations. Women are more likely than men to experience challenges in the work environment for career advancement. No differences were found between men and women on perceived resources and overall motivation factors. Investigations should continue with larger and more diverse samples and medical specialties.

Check the Blood Pressure!: An Educational Tool for Anesthesiology Trainees Converting Epidural Labor Analgesia to Cesarean Delivery Anesthesia.

Recognition and treatment of maternal hypotension during epidural anesthesia administration for intrapartum cesarean delivery preserves maternal-fetal perfusion. A case that required quality assurance review uncovered lapses in maternal hemodynamic monitoring during the transition to intrapartum cesarean delivery anesthesia. To address this, a practice outline was designed for trainee's education describing intrapartum epidural dosing for cesarean delivery and adequate blood pressure monitoring. The time-lapse between epidural dosing and subsequent blood pressure was evaluated before and after the introduction of our educational tool. The time-lapse between blood pressure measures decreased to <10 minutes (10.78-13.92 vs 8.8-9.76 minutes).

Causal inferences on the effectiveness of complex social programs: Navigating assumptions, sources of complexity and evaluation design challenges.

This paper explores avenues for navigating evaluation design challenges posed by complex social programs (CSPs) and their environments when conducting studies that call for generalizable, causal inferences on the intervention's effectiveness. A definition is provided of a CSP drawing on examples from different fields, and an evaluation case is analyzed in depth to derive seven (7) major sources of complexity that typify CSPs, threatening assumptions of textbook-recommended experimental designs for performing impact evaluations. Theoretically-supported, alternative methodological strategies are discussed to navigate assumptions and counter the design challenges posed by the complex configurations and ecology of CSPs. Specific recommendations include: sequential refinement of the evaluation design through systems thinking, systems-informed logic modeling; and use of extended term, mixed methods (ETMM) approaches with exploratory and confirmatory phases of the evaluation. In the proposed approach, logic models are refined through direct induction and interactions with stakeholders. To better guide assumption evaluation, question-framing, and selection of appropriate methodological strategies, a multiphase evaluation design is recommended.

L.E.A.D.: a framework for evidence gathering and use for the prevention of obesity and other complex public health problems.

This article summarizes a comprehensive, systems-oriented framework designed to improve the use of a wide variety of evidence sources to address population-wide obesity problems. The L.E.A.D. framework (for Locate the evidence, Evaluate the evidence, Assemble the evidence, and inform Decisions), developed by an expert consensus committee convened by the Institute of Medicine, is broadly applicable to complex, community-wide health problems. The article explains how to use the framework, presenting an evidence typology that helps specify relevant research questions and includes examples of how particular research methodologies and sources of evidence relate to questions that stem from decision-maker needs. The utility of a range of quantitative, qualitative, and mixed method designs and data sources for assembling a broad and credible evidence base is discussed, with a call for ongoing "evidence generation" to fill information gaps using the recommended systems perspective.

Systems-based practice defined: taxonomy development and role identification for competency assessment of residents.

PURPOSE: To demonstrate a methodology for coding and taxonomy development and to operationally define residents' competence in systems-based practice (SBP) in terms of observable roles, actions, and behaviors. METHODS: The Accreditation Council for Graduate Medical Education's (ACGME's) full-text definition of SBP and the 6 discrete expectations it contains were content analyzed. Structured interviews of 88 health care professionals using a variant of focus group interviews called nominal group processes were conducted and qualitatively analyzed to identify the key attributes of SBP. Themes obtained from these 2 procedures were conceptually matched and organized to create a taxonomy of observable SPB behaviors and the SBP domain. RESULTS: Six general resident roles emerged, under which 35 specific behavioral attributes were subsumed. From the SBP domain specified. Sample SBP items categorized by roles were derived that reflected "in-context" representations of ACGME SBP expectations. CONCLUSION: Our comprehensive analysis created an operational representation of the SBP competency. The taxonomy development model provides a framework for constructing assessment instrument(s) that could be applied to the other ACGME competencies or complex concepts in medical education.

Mapping cognitive overlaps between practice-based learning and improvement and evidence-based medicine: an operational definition for assessing resident physician competence.

PURPOSE: The complex competency labeled practice-based learning and improvement (PBLI) by the Accreditation Council for Graduate Medical Education (ACGME) incorporates core knowledge in evidence-based medicine (EBM). The purpose of this study was to operationally define a "PBLI-EBM" domain for assessing resident physician competence. METHOD: The authors used an iterative design process to first content analyze and map correspondences between ACGME and EBM literature sources. The project team, including content and measurement experts and residents/fellows, parsed, classified, and hierarchically organized embedded learning outcomes using a literature-supported cognitive taxonomy. A pool of 141 items was produced from the domain and assessment specifications. The PBLI-EBM domain and resulting items were content validated through formal reviews by a national panel of experts. RESULTS: The final domain represents overlapping PBLI and EBM cognitive dimensions measurable through written, multiple-choice assessments. It is organized as 4 subdomains of clinical action: Therapy, Prognosis, Diagnosis, and Harm. Four broad cognitive skill branches (Ask, Acquire, Appraise, and Apply) are subsumed under each subdomain. Each skill branch is defined by enabling skills that specify the cognitive processes, content, and conditions pertinent to demonstrable competence. Most items passed content validity screening criteria and were prepared for test form assembly and administration. CONCLUSIONS: The operational definition of PBLI-EBM competence is based on a rigorously developed and validated domain and item pool, and substantially expands conventional understandings of EBM. The domain, assessment specifications, and procedures outlined may be used to design written assessments to tap important cognitive dimensions of the overall PBLI competency, as given by ACGME. For more comprehensive coverage of the PBLI competency, such instruments need to be complemented with performance assessments.

Measuring leader perceptions of school readiness for reforms: use of an iterative model combining classical and Rasch methods.

This study examines validity of data generated by the School Readiness for Reforms: Leader Questionnaire (SRR-LQ) using an iterative procedure that combines classical and Rasch rating scale analysis. Following content-validation and pilot-testing, principal axis factor extraction and promax rotation of factors yielded a five factor structure consistent with the content-validated subscales of the original instrument. Factors were identified based on inspection of pattern and structure coefficients. The rotated factor pattern, inter-factor correlations, convergent validity coefficients, and Cronbach's alpha reliability estimates supported the hypothesized construct properties. To further examine unidimensionality and efficacy of the rating scale structures, item-level data from each factor-defined subscale were subjected to analysis with the Rasch rating scale model. Data-to-model fit statistics and separation reliability for items and persons met acceptable criteria. Rating scale results suggested consistency of expected and observed step difficulties in rating categories, and correspondence of step calibrations with increases in the underlying variables. The combined approach yielded more comprehensive diagnostic information on the quality of the five SRR-LQ subscales; further research is continuing.