Evaluating a National Biomedical Training Program Using QuantCrit: Revealing Disparities in Research Self-efficacy for Women of Color Undergraduates.

Program evaluation for interventions aimed at enhancing diversity can fall short when the evaluation unintentionally reifies the exclusion of multiple marginalized student experiences. The present study presents a Quantitative Critical Race Theory (QuantCrit) approach to program evaluation to understand outcomes for Women of Color undergraduates involved in a national biomedical training program called the Building Infrastructure Leading to Diversity (BUILD) initiative. Using longitudinal data, we examined the impact of participation in the BUILD Scholars programs and BUILD-developed novel biomedical curriculum on undergraduate's research self-efficacy. Employing propensity score matching and multiple regression models, we found that Black women who participated in the BUILD scholars program reported higher research self-efficacy, whereas Latine and White undergraduate BUILD scholars had lower research self-efficacy. Additionally, Latine women who participated in novel biomedical curricula reported significantly lower research self-efficacy. We contend that disaggregated and intersectional analyses of subpopulations are necessary for improving understanding of program interventions and identifying areas where systems of exclusion may continue to harm students from minoritized backgrounds. We provide recommendations for future quantitative program evaluation practices and research in science, technology, engineering, mathematics, and medicine (STEMM) equity efforts.

Investigating the relationship between participation in the building infrastructure leading to diversity (BUILD) initiative and intent to pursue a science career: A cross-sectional analysis.

This paper presents an analysis of survey data to examine the association between supervised structured mentoring and students' intent to pursue a career in science. Data were collected from students in the 10 Building Infrastructure Leading to Diversity (BUILD) research training programs, developed through grants from the National Institutes of Health. Propensity score matching and multinomial logistic regression demonstrated that exposure to BUILD programs-meaning participation in undergraduate research, receipt of mentoring from a primary mentor, and/or participation as a funded scholar and/or associate of each BUILD site's training program-was associated with increased intent to pursue a science career. These findings have implications for STEM program evaluation and practice in higher education.

The intent and influence of evaluation policies and strategies in the philanthropic sector.

Evaluation policies are being written across the philanthropic sector. These policies offer rules and principles that are intended to guide evaluation practice. However, it is unclear what has motivated the development of evaluation policies and what impact, if any, they have on evaluation practice. Through interviews with 10 evaluation directors at foundations that have written evaluation policies, we identify the intent of these policies and their perceived influence in the philanthropic sector. We conclude with suggestions for future research on evaluation policy.

A problem-bound evaluation approach.

Traditional modes of evaluation can be constraining when seeking to improve persistent societal problems within complex systems. Frameworks that assume stability and predictability may not match the system dynamics within which a program resides. This paper seeks to further evaluators' understanding of how to improve problems in complex systems by offering a "problem-bound" evaluation approach. This model is grounded in theoretical concepts from complexity science, complex adaptive systems, systems thinking, and improvement science. As such, it provides an agile and flexible framework for continually learning from emergent findings related to the problem. By offering this model, we seek to contribute to the limited knowledge base on how to apply systems change theoretical ideas in evaluation practice, along with practical implications regarding the evaluator's role.

Strengthening families and communities to prevent child abuse and neglect: lessons from the Los Angeles Prevention Initiative Demonstration Project.

The Prevention Initiative Demonstration Project, funded by the Los Angeles County Department of Children and Family Services (DCFS), is a community-specific strategy delivered through eight regional networks designed to address the full spectrum of community-based prevention. This article summarizes a strong and meaningful pattern of improvements found in the second year evaluation for three groups of families--those living in high-risk communities but not involved with DCFS, those being investigated by DCFS for possible child maltreatment, and those with open DCFS cases.

Theoretical and conceptual frameworks across local evaluation efforts in a nationwide consortium.

This paper describes the theoretical and conceptual frameworks used to guide the site-level evaluations of Building Infrastructure Leading to Diversity (BUILD) programs, part of the Diversity Program Consortium (DPC), funded by the National Institutes of Health. We aim to provide an understanding of which theories informed the evaluation work of the DPC and how the frameworks guiding BUILD site-level evaluations are conceptually aligned with one another and with the consortium-level evaluation.

Baseline Characteristics of the 2015-2019 First Year Student Cohorts of the NIH Building Infrastructure Leading to Diversity (BUILD) Program.

Objective: The biomedical/behavioral sciences lag in the recruitment and advancement of students from historically underrepresented backgrounds. In 2014 the NIH created the Diversity Program Consortium (DPC), a prospective, multi-site study comprising 10 Building Infrastructure Leading to Diversity (BUILD) institutional grantees, the National Research Mentoring Network (NRMN) and a Coordination and Evaluation Center (CEC). This article describes baseline characteristics of four incoming, first-year student cohorts at the primary BUILD institutions who completed the Higher Education Research Institute, The Freshmen Survey between 2015-2019. These freshmen are the primary student cohorts for longitudinal analyses comparing outcomes of BUILD program participants and non-participants. Design: Baseline description of first-year students entering college at BUILD institutions during 2015-2019. Setting: Ten colleges/universities that each received <$7.5mil/yr in NIH Research Project Grants and have high proportions of low-income students. Participants: First-year undergraduate students who participated in BUILD-sponsored activities and a sample of non-BUILD students at the same BUILD institutions. A total of 32,963 first-year students were enrolled in the project; 64% were female, 18% Hispanic/Latinx, 19% African American/Black, 2% American Indian/Alaska Native and Native Hawaiian/Pacific Islander, 17% Asian, and 29% White. Twenty-seven percent were from families with an income <$30,000/yr and 25% were their family's first generation in college. Planned Outcomes: Primary student outcomes to be evaluated over time include undergraduate biomedical degree completion, entry into/completion of a graduate biomedical degree program, and evidence of excelling in biomedical research and scholarship. Conclusions: The DPC national evaluation has identified a large, longitudinal cohort of students with many from groups historically underrepresented in the biomedical sciences that will inform institutional/national policy level initiatives to help diversify the biomedical workforce.