Understanding the context and appreciating the complexity of evaluating the Diversity Program Consortium.

The National Institutes of Health (NIH) made a sizeable investment in developing a scientific approach to understanding how to best increase diversity in the NIH-funded workforce by fostering inclusive excellence at a national scale through the Diversity Program Consortium (DPC). This chapter provides an overview of the context in which the consortium-wide evaluation study has taken place to provide readers with an understanding of its level of complexity. This evaluation effort is the first large-scale, national, systemic, longitudinal evaluation of harmonized interventions focused on undergraduate biomedical research training programs in the history of the NIH and the National Institute of General Medical Sciences.

Gauging treatment impact: The development of exposure variables in a large-scale evaluation study.

While guidance on how to design rigorous evaluation studies abounds, prescriptive guidance on how to include critical process and context measures through the construction of exposure variables is lacking. Capturing nuanced intervention dosage information within a large-scale evaluation is particularly complex. The Building Infrastructure Leading to Diversity (BUILD) initiative is part of the Diversity Program Consortium, which is funded by the National Institutes of Health. It is designed to increase participation in biomedical research careers among individuals from underrepresented groups. This chapter articulates methods employed in defining BUILD student and faculty interventions, tracking nuanced participation in multiple programs and activities, and computing the intensity of exposure. Defining standardized exposure variables (beyond simple treatment group membership) is crucial for equity-focused impact evaluation. Both the process and resulting nuanced dosage variables can inform the design and implementation of large-scale, diversity training program, outcome-focused, evaluation studies.

BUILDing BLaST: promoting rural students' biomedical research careers using a culturally responsive, one health approach.

BACKGROUND AND PURPOSE: Most postsecondary institutions in the state of Alaska (USA) have a broad mission to serve diverse students, many of whom come from schools in rural villages that are accessible only by plane, boat, or snowmobile. The major research university, the University of Alaska in Fairbanks (UAF), serves a population whereby 40% are from groups recognized as underrepresented in the biomedical workforce. The purpose of this article is to describe the Building Infrastructure Leading to Diversity (BUILD)-supported program in the state of Alaska that seeks to engage students from rural areas with a culturally relevant approach that is centered on the One Health paradigm, integrating human, animal, and environmental health. PROGRAM AND KEY HIGHLIGHTS: The Biomedical Learning and Student Training (BLaST) program distinguished by broad themes that address recruitment, retention, and success of students in biomedical programs, especially for students from rural backgrounds. Targeted rural outreach emphasizes that biomedical research includes research on the integration of human, animal, and environmental health. This One Health perspective gives personal relevance and connection to biomedical research. This outreach is expected to benefit student recruitment, as well as foster family and community support for pursuit of college degrees. BLaST promotes integration of research into undergraduate curricula through curriculum development, and by creating a new class of instructors, laboratory research and teaching technicians, who provide research mentorship, course instruction, and comprehensive advising. Finally, BLaST facilitates early and sustained undergraduate research experiences in collaborations with graduate students and faculty. IMPLICATIONS: BLaST's approach is highly adapted to the Alaskan educational and physical environment, but components and concepts could be adapted to other rural areas as a means to engage students from rural backgrounds, who often have a closer relationship with the natural environment than urban students.

A participatory approach to evaluating a national training and institutional change initiative: the BUILD longitudinal evaluation.

BACKGROUND AND PURPOSE: The National Institutes of Health (NIH) funds training programs to increase the numbers and skills of scientists who obtain NIH research grants, but few programs have been rigorously evaluated. The sizeable recent NIH investment in developing programs to increase the diversity of the NIH-funded workforce, implemented through the Diversity Program Consortium (DPC), is unusual in that it also funds a Consortium-wide evaluation plan, which spans the activities of the 10 BUilding Infrastructure Leading to Diversity (BUILD) awardees and the National Research Mentoring Network (NRMN). The purpose of this article is to describe the evaluation design and innovations of the BUILD Program on students, faculty, and institutions of the 10 primarily undergraduate BUILD sites. KEY HIGHLIGHTS OF THE PROJECT: Our approach to this multi-methods quasi-experimental longitudinal evaluation emphasizes stakeholder participation and collaboration. The evaluation plan specifies the major evaluation questions and key short- to long-term outcome measures (or Hallmarks of Success). The Coordination and Evaluation Center (CEC) embarked on a comprehensive evaluation strategy by developing a set of logic models that incorporate the Hallmarks of Success and other outcomes that were collaboratively identified by the DPC. Data were collected from each BUILD site through national surveys from the Higher Education Research Institute at UCLA (HERI), annual followup surveys that align with the HERI instruments, site visits and case studies, program encounter data ("tracker" data), and institutional data. The analytic approach involves comparing changes in Hallmarks (key outcomes) within institutions for biomedical students who participated versus those who did not participate in the BUILD program at each institution, as well as between institution patterns of biomedical students at the BUILD sites, and matched institutions that were not BUILD grantees. Case studies provide insights into the institutionalization of these new programs and help to explain the processes that lead to the observed outcomes. IMPLICATIONS: Ultimately, the results of the consortium-wide evaluation will be used to inform national policy in higher education and will provide relevant examples of institutional and educational programmatic changes required to diversify the biomedical workforce in the USA.

Evaluating efforts to diversify the biomedical workforce: the role and function of the Coordination and Evaluation Center of the Diversity Program Consortium.

BACKGROUND: The National Institutes of Health (NIH)-funded Diversity Program Consortium (DPC) includes a Coordination and Evaluation Center (CEC) to conduct a longitudinal evaluation of the two signature, national NIH initiatives - the Building Infrastructure Leading to Diversity (BUILD) and the National Research Mentoring Network (NRMN) programs - designed to promote diversity in the NIH-funded biomedical, behavioral, clinical, and social sciences research workforce. Evaluation is central to understanding the impact of the consortium activities. This article reviews the role and function of the CEC and the collaborative processes and achievements critical to establishing empirical evidence regarding the efficacy of federally-funded, quasi-experimental interventions across multiple sites. The integrated DPC evaluation is particularly significant because it is a collaboratively developed Consortium Wide Evaluation Plan and the first hypothesis-driven, large-scale systemic national longitudinal evaluation of training programs in the history of NIH/National Institute of General Medical Sciences. KEY HIGHLIGHTS: To guide the longitudinal evaluation, the CEC-led literature review defined key indicators at critical training and career transition points - or Hallmarks of Success. The multidimensional, comprehensive evaluation of the impact of the DPC framed by these Hallmarks is described. This evaluation uses both established and newly developed common measures across sites, and rigorous quasi-experimental designs within novel multi-methods (qualitative and quantitative). The CEC also promotes shared learning among Consortium partners through working groups and provides technical assistance to support high-quality process and outcome evaluation internally of each program. Finally, the CEC is responsible for developing high-impact dissemination channels for best practices to inform peer institutions, NIH, and other key national and international stakeholders. IMPLICATIONS: A strong longitudinal evaluation across programs allows the summative assessment of outcomes, an understanding of factors common to interventions that do and do not lead to success, and elucidates the processes developed for data collection and management. This will provide a framework for the assessment of other training programs and have national implications in transforming biomedical research training.