Gauging treatment impact: The development of exposure variables in a large-scale evaluation study.

While guidance on how to design rigorous evaluation studies abounds, prescriptive guidance on how to include critical process and context measures through the construction of exposure variables is lacking. Capturing nuanced intervention dosage information within a large-scale evaluation is particularly complex. The Building Infrastructure Leading to Diversity (BUILD) initiative is part of the Diversity Program Consortium, which is funded by the National Institutes of Health. It is designed to increase participation in biomedical research careers among individuals from underrepresented groups. This chapter articulates methods employed in defining BUILD student and faculty interventions, tracking nuanced participation in multiple programs and activities, and computing the intensity of exposure. Defining standardized exposure variables (beyond simple treatment group membership) is crucial for equity-focused impact evaluation. Both the process and resulting nuanced dosage variables can inform the design and implementation of large-scale, diversity training program, outcome-focused, evaluation studies.

Development of the Self-Efficacy as Faculty Change Agent for Diversity (SE-FCA-D) Scale.

Thispublicationprovides an overview of the development of the Self-Efficacy as Faculty Change Agent/or Diversity (SE-FCA-D) scale. The 5-item scale recently appeared as an additional module on the Higher Education and Research Institute (HERi) Faculty Survey 2019-2020 for the JO BUILD programs within the Diversity Program Consortium that areparticipating in the Enhance Diversity Study, supported by the National Institutes of Health (NIH U54GMI 19024). Thepiloted scale is meant to measure DPC Hallmark of Success FAC-16: "Strong self-efficacy to act as a change agent to enhance diversity in biomedical research and research training environments" (DPC, 2019). Once data are available, scale validation will movefonvard and.findings will be shared.

Baseline Characteristics of the 2015-2019 First Year Student Cohorts of the NIH Building Infrastructure Leading to Diversity (BUILD) Program.

Objective: The biomedical/behavioral sciences lag in the recruitment and advancement of students from historically underrepresented backgrounds. In 2014 the NIH created the Diversity Program Consortium (DPC), a prospective, multi-site study comprising 10 Building Infrastructure Leading to Diversity (BUILD) institutional grantees, the National Research Mentoring Network (NRMN) and a Coordination and Evaluation Center (CEC). This article describes baseline characteristics of four incoming, first-year student cohorts at the primary BUILD institutions who completed the Higher Education Research Institute, The Freshmen Survey between 2015-2019. These freshmen are the primary student cohorts for longitudinal analyses comparing outcomes of BUILD program participants and non-participants. Design: Baseline description of first-year students entering college at BUILD institutions during 2015-2019. Setting: Ten colleges/universities that each received <$7.5mil/yr in NIH Research Project Grants and have high proportions of low-income students. Participants: First-year undergraduate students who participated in BUILD-sponsored activities and a sample of non-BUILD students at the same BUILD institutions. A total of 32,963 first-year students were enrolled in the project; 64% were female, 18% Hispanic/Latinx, 19% African American/Black, 2% American Indian/Alaska Native and Native Hawaiian/Pacific Islander, 17% Asian, and 29% White. Twenty-seven percent were from families with an income <$30,000/yr and 25% were their family's first generation in college. Planned Outcomes: Primary student outcomes to be evaluated over time include undergraduate biomedical degree completion, entry into/completion of a graduate biomedical degree program, and evidence of excelling in biomedical research and scholarship. Conclusions: The DPC national evaluation has identified a large, longitudinal cohort of students with many from groups historically underrepresented in the biomedical sciences that will inform institutional/national policy level initiatives to help diversify the biomedical workforce.

Evaluating efforts to diversify the biomedical workforce: the role and function of the Coordination and Evaluation Center of the Diversity Program Consortium.

BACKGROUND: The National Institutes of Health (NIH)-funded Diversity Program Consortium (DPC) includes a Coordination and Evaluation Center (CEC) to conduct a longitudinal evaluation of the two signature, national NIH initiatives - the Building Infrastructure Leading to Diversity (BUILD) and the National Research Mentoring Network (NRMN) programs - designed to promote diversity in the NIH-funded biomedical, behavioral, clinical, and social sciences research workforce. Evaluation is central to understanding the impact of the consortium activities. This article reviews the role and function of the CEC and the collaborative processes and achievements critical to establishing empirical evidence regarding the efficacy of federally-funded, quasi-experimental interventions across multiple sites. The integrated DPC evaluation is particularly significant because it is a collaboratively developed Consortium Wide Evaluation Plan and the first hypothesis-driven, large-scale systemic national longitudinal evaluation of training programs in the history of NIH/National Institute of General Medical Sciences. KEY HIGHLIGHTS: To guide the longitudinal evaluation, the CEC-led literature review defined key indicators at critical training and career transition points - or Hallmarks of Success. The multidimensional, comprehensive evaluation of the impact of the DPC framed by these Hallmarks is described. This evaluation uses both established and newly developed common measures across sites, and rigorous quasi-experimental designs within novel multi-methods (qualitative and quantitative). The CEC also promotes shared learning among Consortium partners through working groups and provides technical assistance to support high-quality process and outcome evaluation internally of each program. Finally, the CEC is responsible for developing high-impact dissemination channels for best practices to inform peer institutions, NIH, and other key national and international stakeholders. IMPLICATIONS: A strong longitudinal evaluation across programs allows the summative assessment of outcomes, an understanding of factors common to interventions that do and do not lead to success, and elucidates the processes developed for data collection and management. This will provide a framework for the assessment of other training programs and have national implications in transforming biomedical research training.