Design and Analytic Methods to Evaluate Multilevel Interventions to Reduce Health Disparities: Rigorous Methods Are Available.

In June 2022, the NIH Office of Disease Prevention (ODP) issued a Call for Papers for a Supplemental Issue to Prevention Science on Design and Analytic Methods to Evaluate Multilevel Interventions to Reduce Health Disparities. ODP sought to bring together current thinking and new ideas about design and analytic methods for studies aimed at reducing health disparities, including strategies for balancing methodological rigor with design feasibility, acceptability, and ethical considerations. ODP was particularly interested in papers on design and analytic methods for parallel group- or cluster-randomized trials (GRTs), stepped-wedge GRTs, group-level regression discontinuity trials, and other methods appropriate for evaluating multilevel interventions. In this issue, we include 12 papers that report new methods, provide examples of strong applications of existing methods, or provide guidance on developing multilevel interventions to reduce health disparities. These papers provide examples showing that rigorous methods are available for the design and analysis of multilevel interventions to reduce health disparities.

The Science of Precision Prevention: Research Opportunities and Clinical Applications to Reduce Cardiovascular Health Disparities.

Precision prevention embraces personalized prevention but includes broader factors such as social determinants of health to improve cardiovascular health. The quality, quantity, precision, and diversity of data relatable to individuals and communities continue to expand. New analytical methods can be applied to these data to create tools to attribute risk, which may allow a better understanding of cardiovascular health disparities. Interventions using these analytic tools should be evaluated to establish feasibility and efficacy for addressing cardiovascular disease disparities in diverse individuals and communities. Training in these approaches is important to create the next generation of scientists and practitioners in precision prevention. This state-of-the-art review is based on a workshop convened to identify current gaps in knowledge and methods used in precision prevention intervention research, discuss opportunities to expand trials of implementation science to close the health equity gaps, and expand the education and training of a diverse precision prevention workforce.

Deciphering Disparities: The NHLBI Program on Disparities Elimination through Coordinated Interventions to Prevent and Control Heart and Lung Disease Risk (DECIPHeR).

Despite several ambitious national health initiatives to eliminate health disparities, spanning more than 4 decades, health disparities remain pervasive in the United States. In an attempt to bend the curve in disparities elimination, the National Heart, Lung, and Blood Institute (NHLBI) issued a funding opportunity on Disparities Elimination through Coordinated Interventions to Prevent and Control Heart and Lung Disease Risk (DECIPHeR) in March 2019. Seven implementation research centers and 1 research coordinating center were funded in September 2020 to plan, develop, and test effective implementation strategies for eliminating disparities in heart and lung disease risk. In the 16 articles presented in this issue of Ethnicity & Disease, the DECIPHeR Alliance investigators and their NHLBI program staff address the work accomplished in the first phase of this biphasic research endeavor. Included in the collection are an article on important lessons learned during technical assistance sessions designed to ensure scientific rigor in clinical study designs, and 2 examples of clinical study process articles. Several articles show the diversity of clinical and public health settings addressed including schools, faith-based settings, federally qualified health centers, and other safety net clinics. All strategies for eliminating disparities tackle a cardiovascular or pulmonary disease and related risk factors. In an additional article, NHLBI program staff address expectations in phase 2 of the DECIPHeR program, strategies to ensure feasibility of scaling and spreading promising strategies identified, and opportunities for translating the DECIPHeR research model to other chronic diseases for the elimination of related health disparities.

Implementation Research at NHLBI: Methodological and Design Challenges and Lessons Learned from the DECIPHeR Initiative.

NHLBI funded seven projects as part of the Disparities Elimination through Coordinated Interventions to Prevent and Control Heart and Lung Disease Risk (DECIPHeR) Initiative. They were expected to collaborate with community partners to (1) employ validated theoretical or conceptual implementation research frameworks, (2) include implementation research study designs, (3) include implementation measures as primary outcomes, and (4) inform our understanding of mediators and mechanisms of action of the implementation strategy. Several projects focused on late-stage implementation strategies that optimally and sustainably delivered two or more evidence-based multilevel interventions to reduce or eliminate cardiovascular and/or pulmonary health disparities and to improve population health in high-burden communities. Projects that were successful in the three-year planning phase transitioned to a 4-year execution phase. NHLBI formed a Technical Assistance Workgroup during the planning phase to help awardees refine study aims, strengthen research designs, detail analytic plans, and to use valid sample size methods. This paper highlights methodological and study design challenges encountered during this process. Important lessons learned included (1) the need for greater emphasis on implementation outcomes, (2) the need to clearly distinguish between intervention and implementation strategies in the protocol, (3) the need to address clustering due to randomization of groups or clusters, (4) the need to address the cross-classification that results when intervention agents work across multiple units of randomization in the same arm, (5) the need to accommodate time-varying intervention effects in stepped-wedge designs, and (6) the need for data-based estimates of the parameters required for sample size estimation.

Analysis of multiple-period group randomized trials: random coefficients model or repeated measures ANOVA?

BACKGROUND: Multiple-period parallel group randomized trials (GRTs) analyzed with linear mixed models can represent time in mean models as continuous or categorical. If time is continuous, random effects are traditionally group- and member-level deviations from condition-specific slopes and intercepts and are referred to as random coefficients (RC) analytic models. If time is categorical, random effects are traditionally group- and member-level deviations from time-specific condition means and are referred to as repeated measures ANOVA (RM-ANOVA) analytic models. Longstanding guidance recommends the use of RC over RM-ANOVA for parallel GRTs with more than two periods because RC exhibited nominal type I error rates for both time parameterizations while RM-ANOVA exhibited inflated type I error rates when applied to data generated using the RC model. However, this recommendation was developed assuming a variance components covariance matrix for the RM-ANOVA, using only cross-sectional data, and explicitly modeling time × group variation. Left unanswered were how well RM-ANOVA with an unstructured covariance would perform on data generated according to the RC mechanism, if similar patterns would be observed in cohort data, and the impact of not modeling time × group variation if such variation was present in the data-generating model. METHODS: Continuous outcomes for cohort and cross-sectional parallel GRT data were simulated according to RM-ANOVA and RC mechanisms at five total time periods. All simulations assumed time × group variation. We varied the number of groups, group size, and intra-cluster correlation. Analytic models using RC, RM-ANOVA, RM-ANOVA with unstructured covariance, and a Saturated random effects structure were applied to the data. All analytic models specified time × group random effects. The analytic models were then reapplied without specifying random effects for time × group. RESULTS: Results indicated the RC and saturated analytic models maintained the nominal type I error rate in all data sets, RM-ANOVA with an unstructured covariance did not avoid type I error rate inflation when applied to cohort RC data, and analytic models omitting time-varying group random effects when such variation exists in the data were prone to substantial type I error inflation unless the residual error variance is high relative to the time × group variance. CONCLUSION: The time × group RC and saturated analytic models are recommended as the default for multiple period parallel GRTs.

Two weights make a wrong: Cluster randomized trials with variable cluster sizes and heterogeneous treatment effects.

In cluster randomized trials (CRTs), the hierarchical nesting of participants (level 1) within clusters (level 2) leads to two conceptual populations: clusters and participants. When cluster sizes vary and the goal is to generalize to a hypothetical population of clusters, the unit average treatment effect (UATE), which averages equally at the cluster level rather than equally at the participant level, is a common estimand of interest. From an analytic perspective, when a generalized estimating equations (GEE) framework is used to obtain averaged treatment effect estimates for CRTs with variable cluster sizes, it is natural to specify an inverse cluster size weighted analysis so that each cluster contributes equally and to adopt an exchangeable working correlation matrix to account for within-cluster correlation. However, such an approach essentially uses two distinct weights in the analysis (i.e. both cluster size weights and covariance weights) and, in this article, we caution that it will lead to biased and/or inefficient treatment effect estimates for the UATE estimand. That is, two weights "make a wrong" or lead to poor estimation characteristics. These findings are based on theoretical derivations, corroborated via a simulation study, and illustrated using data from a CRT of a colorectal cancer screening program. We show that, an analysis with both an independence working correlation matrix and weighting by inverse cluster size is the only approach that always provides valid results for estimation of the UATE in CRTs with variable cluster sizes.

NIH Primary and Secondary Prevention Research in Humans: a Portfolio Analysis of Study Designs Used in 2012-2019.

We can learn a great deal about the research questions being addressed in a field by examining the study designs used in that field. This manuscript examines the research questions being addressed in prevention research by characterizing the distribution and trends of study designs included in primary and secondary prevention research supported by the National Institutes of Health through grants and cooperative agreements, together with the types of prevention research, populations, rationales, exposures, and outcomes associated with each type of design. The Office of Disease Prevention developed a taxonomy to classify new extramural NIH-funded research projects and created a database with a representative sample of 14,523 research projects for fiscal years 2012-2019. The data were weighted to represent the entirety of the extramural research portfolio. Leveraging this dataset, the Office of Disease Prevention characterized the study designs proposed in NIH-funded primary and secondary prevention research applications. The most common study designs proposed in new NIH-supported prevention research applications during FY12-19 were observational designs (63.3%, 95% CI 61.5%-65.0%), analysis of existing data (44.5%, 95% CI: 42.7-46.3), methods research (23.9%, 95% CI: 22.3-25.6), and randomized interventions (17.2%, 95% CI: 16.1%-18.4%). Observational study designs dominated primary prevention research, while intervention designs were more common in secondary prevention research. Observational designs were more common for exposures that would be difficult to manipulate (e.g., genetics, chemical toxin, and infectious disease (not pneumonia/influenza or HIV/AIDS)), while intervention designs were more common for exposures that would be easier to manipulate (e.g., education/counseling, medication/device, diet/nutrition, and healthcare delivery). Intervention designs were not common for outcomes that are rare or have a long latency (e.g., cancer, neurological disease, Alzheimer's disease) and more common for outcomes that are more common or where effects would be expected earlier (e.g., healthcare delivery, health related quality of life, substance use, and medication/device). Observational designs and analyses of existing data dominated, suggesting that much of the prevention research funded by NIH continues to focus on questions of association and on questions of identification of risk and protective factors. Randomized and non-randomized intervention designs were included far less often, suggesting that a much smaller fraction of the NIH prevention research portfolio is focused on questions of whether interventions can be used to modify risk or protective factors or to change some other health-related biomedical or behavioral outcome. The much heavier focus on observational studies is surprising given how much we know already about the leading risk factors for death and disability in the USA, because those risk factors account for 74% of the county-level mortality in the USA, and because they play such a vital role in the development of clinical and public health guidelines, whose developers often weigh results from randomized trials much more heavily than results from observational studies. Improvements in death and disability nationwide are more likely to derive from guidelines based on intervention research to address the leading risk factors than from additional observational studies.

Influential methods reports for group-randomized trials and related designs.

BACKGROUND: This article identifies the most influential methods reports for group-randomized trials and related designs published through 2020. Many interventions are delivered to participants in real or virtual groups or in groups defined by a shared interventionist so that there is an expectation for positive correlation among observations taken on participants in the same group. These interventions are typically evaluated using a group- or cluster-randomized trial, an individually randomized group treatment trial, or a stepped wedge group- or cluster-randomized trial. These trials face methodological issues beyond those encountered in the more familiar individually randomized controlled trial. METHODS: PubMed was searched to identify candidate methods reports; that search was supplemented by reports known to the author. Candidate reports were reviewed by the author to include only those focused on the designs of interest. Citation counts and the relative citation ratio, a new bibliometric tool developed at the National Institutes of Health, were used to identify influential reports. The relative citation ratio measures influence at the article level by comparing the citation rate of the reference article to the citation rates of the articles cited by other articles that also cite the reference article. RESULTS: In total, 1043 reports were identified that were published through 2020. However, 55 were deemed to be the most influential based on their relative citation ratio or their citation count using criteria specific to each of the three designs, with 32 group-randomized trial reports, 7 individually randomized group treatment trial reports, and 16 stepped wedge group-randomized trial reports. Many of the influential reports were early publications that drew attention to the issues that distinguish these designs from the more familiar individually randomized controlled trial. Others were textbooks that covered a wide range of issues for these designs. Others were "first reports" on analytic methods appropriate for a specific type of data (e.g. binary data, ordinal data), for features commonly encountered in these studies (e.g. unequal cluster size, attrition), or for important variations in study design (e.g. repeated measures, cohort versus cross-section). Many presented methods for sample size calculations. Others described how these designs could be applied to a new area (e.g. dissemination and implementation research). Among the reports with the highest relative citation ratios were the CONSORT statements for each design. CONCLUSIONS: Collectively, the influential reports address topics of great interest to investigators who might consider using one of these designs and need guidance on selecting the most appropriate design for their research question and on the best methods for design, analysis, and sample size.

New NIH Primary and Secondary Prevention Research During 2012-2019.

INTRODUCTION: This manuscript characterizes primary and secondary prevention research in humans and related methods research funded by NIH in 2012‒2019. METHODS: The NIH Office of Disease Prevention updated its prevention research taxonomy in 2019‒2020 and applied it to a sample of 14,523 new extramural projects awarded in 2012-2019. All projects were coded manually for rationale, exposures, outcomes, population focus, study design, and type of prevention research. All results are based on that manual coding. RESULTS: Taxonomy updates resulted in a slight increase, from an average of 16.7% to 17.6%, in the proportion of prevention research awards for 2012‒2017; there was a further increase to 20.7% in 2019. Most of the leading risk factors for death and disability in the U.S. were observed as an exposure or outcome in <5% of prevention research projects in 2019 (e.g., diet, 3.7%; tobacco, 3.9%; blood pressure, 2.8%; obesity, 4.4%). Analysis of existing data became more common (from 36% to 46.5%), whereas randomized interventions became less common (from 20.5% to 12.3%). Randomized interventions addressing a leading risk factor in a minority health or health disparities population were uncommon. CONCLUSIONS: The number of new NIH awards classified as prevention research increased to 20.7% in 2019. New projects continued to focus on observational studies and secondary data analysis in 2018 and 2019. Additional research is needed to develop and test new interventions or develop methods for the dissemination of existing interventions, which address the leading risk factors, particularly in minority health and health disparities populations.

Essential Ingredients and Innovations in the Design and Analysis of Group-Randomized Trials.

This article reviews the essential ingredients and innovations in the design and analysis of group-randomized trials. The methods literature for these trials has grown steadily since they were introduced to the biomedical research community in the late 1970s, and we summarize those developments. We review, in addition to the group-randomized trial, methods for two closely related designs, the individually randomized group treatment trial and the stepped-wedge group-randomized trial. After describing the essential ingredients for these designs, we review the most important developments in the evolution of their methods using a new bibliometric tool developed at the National Institutes of Health. We then discuss the questions to be considered when selecting from among these designs or selecting the traditional randomized controlled trial. We close with a review of current methods for the analysis of data from these designs, a case study to illustrate each design, and a brief summary.

Assessment of Prevention Research Measuring Leading Risk Factors and Causes of Mortality and Disability Supported by the US National Institutes of Health.

Importance: No studies to date have examined support by the National Institutes of Health (NIH) for primary and secondary prevention research in humans and related methods research that measures the leading risk factors or causes of death or disability as outcomes or exposures. Objective: To characterize NIH support for such research. Design and Setting: This serial cross-sectional study randomly sampled NIH grants and cooperative agreements funded during fiscal years 2012 through 2017. For awards with multiple subprojects, each was treated as a separate project. Study characteristics, outcomes, and exposures were coded from October 2015 through February 2019. Analyses weighted to reflect the sampling scheme were completed in March through June 2019. Using 2017 data from the Centers for Disease Control and Prevention and 2016 data from the Global Burden of Disease project, the leading risk factors and causes of death and disability in the United States were identified. Main Outcomes and Measures: The main outcome was the percentage of the NIH prevention research portfolio measuring a leading risk factor or cause of death or disability as an outcome or exposure. Results: A total of 11 082 research projects were coded. Only 25.9% (95% CI, 24.0%-27.8%) of prevention research projects measured a leading cause of death as an outcome or exposure, although these leading causes were associated with 74.0% of US mortality. Only 34.0% (95% CI, 32.2%-35.9%) measured a leading risk factor for death, although these risk factors were associated with 57.3% of mortality. Only 31.4% (95% CI, 29.6%-33.3%) measured a leading risk factor for disability-adjusted life-years lost, although these risk factors were associated with 42.1% of disability-adjusted life-years lost. Relatively few projects included a randomized clinical trial (24.6%; 95% CI, 22.5%-26.9%) or involved more than 1 leading cause (3.3%; 95% CI, 2.6%-4.1%) or risk factor (8.8%; 95% CI, 7.9%-9.8%). Conclusions and Relevance: In this cross-sectional study, the leading risk factors and causes of death and disability were underrepresented in the NIH prevention research portfolio relative to their burden. Because so much is already known about these risk factors and causes, and because randomized interventions play such a vital role in the development of clinical and public health guidelines, it appears that greater attention should be given to develop and test interventions that address these risk factors and causes, addressing multiple risk factors or causes when possible.

RGD-decorated cholesterol stabilized polyplexes for targeted siRNA delivery to glioblastoma cells.

The development of an effective and safe treatment for glioblastoma (GBM) represents a significant challenge in oncology today. Downregulation of key mediators of cell signal transduction by RNA interference is considered a promising treatment strategy but requires efficient, intracellular delivery of siRNA into GBM tumor cells. Here, we describe novel polymeric siRNA nanocarriers functionalized with cRGD peptide that mediates targeted and efficient reporter gene silencing in U87R invasive human GBM cells. The polymer was synthesized via RAFT copolymerization of N-(2-hydroxypropyl)-methacrylamide (HPMA) and N-acryloxysuccinimide (NAS), followed by post-polymerization modification with cholesterol for stabilization, cationic amines for siRNA complexation, and azides for copper-free click chemistry. The novel resultant cationic polymer harboring a terminal cholesterol group, self-assembled with siRNA to yield nanosized polyplexes (~ 40 nm) with good colloidal stability at physiological ionic strength. Post-modification of the preformed polyplexes with PEG-cRGD end-functionalized with bicyclo[6.1.0]nonyne (BCN) group resulted in enhanced cell uptake and increased luciferase gene silencing in U87R cells, compared to polyplexes lacking cRGD-targeting groups.

The Importance of Evaluating Health Disparities Research.

Health disparity populations are socially disadvantaged, and the multiple levels of discrimination they often experience mean that their characteristics and attributes differ from those of the mainstream. Programs and policies targeted at reducing health disparities or improving minority health must consider these differences. Despite the importance of evaluating health disparities research to produce high-quality data that can guide decision-making, it is not yet a customary practice. Although health disparities evaluations incorporate the same scientific methods as all evaluations, they have unique components such as population characteristics, sociocultural context, and the lack of health disparity common indicators and metrics that must be considered in every phase of the research. This article describes evaluation strategies grouped into 3 components: formative (needs assessments and process), design and methodology (multilevel designs used in real-world settings), and summative (outcomes, impacts, and cost). Each section will describe the standards for each component, discuss the unique health disparity aspects, and provide strategies from the National Institute on Minority Health and Health Disparities Metrics and Measures Visioning Workshop (April 2016) to advance the evaluation of health disparities research.

NIH Primary and Secondary Prevention Research in Humans During 2012-2017.

INTRODUCTION: This paper provides the first detailed analysis of the NIH prevention research portfolio for primary and secondary prevention research in humans and related methods research. METHODS: The Office of Disease Prevention developed a taxonomy of 128 topics and applied it to 11,082 projects representing 91.7% of all new projects and 84.1% of all dollars used for new projects awarded using grant and cooperative agreement activity codes that supported research in fiscal years 2012-2017. Projects were coded in 2016-2018 and analyzed in 2018. RESULTS: Only 16.7% of projects and 22.6% of dollars were used for primary and secondary prevention research in humans or related methods research. Most of the leading risk factors for death and disability in the U.S. were selected as an outcome in <5% of the projects. Many more projects included an observational study, or an analysis of existing data, than a randomized intervention. These patterns were consistent over time. CONCLUSIONS: The appropriate level of support for primary and secondary prevention research in humans from NIH will differ by field and stage of research. The estimates reported here may be overestimates, as credit was given for a project even if only a portion of that project addressed prevention research. Given that 74% of the variability in county-level life expectancy across the U.S. is explained by established risk factors, it seems appropriate to devote additional resources to developing and testing interventions to address those risk factors.

A Machine Learning Approach to Identify NIH-Funded Applied Prevention Research.

INTRODUCTION: To fulfill its mission, the NIH Office of Disease Prevention systematically monitors NIH investments in applied prevention research. Specifically, the Office focuses on research in humans involving primary and secondary prevention, and prevention-related methods. Currently, the NIH uses the Research, Condition, and Disease Categorization system to report agency funding in prevention research. However, this system defines prevention research broadly to include primary and secondary prevention, studies on prevention methods, and basic and preclinical studies for prevention. A new methodology was needed to quantify NIH funding in applied prevention research. METHODS: A novel machine learning approach was developed and evaluated for its ability to characterize NIH-funded applied prevention research during fiscal years 2012-2015. The sensitivity, specificity, positive predictive value, accuracy, and F1 score of the machine learning method; the Research, Condition, and Disease Categorization system; and a combined approach were estimated. Analyses were completed during June-August 2017. RESULTS: Because the machine learning method was trained to recognize applied prevention research, it more accurately identified applied prevention grants (F1 = 72.7%) than the Research, Condition, and Disease Categorization system (F1 = 54.4%) and a combined approach (F1 = 63.5%) with p<0.001. CONCLUSIONS: This analysis demonstrated the use of machine learning as an efficient method to classify NIH-funded research grants in disease prevention.

Design and analysis of group-randomized trials in cancer: A review of current practices.

The purpose of this paper is to summarize current practices for the design and analysis of group-randomized trials involving cancer-related risk factors or outcomes and to offer recommendations to improve future trials. We searched for group-randomized trials involving cancer-related risk factors or outcomes that were published or online in peer-reviewed journals in 2011-15. During 2016-17, in Bethesda MD, we reviewed 123 articles from 76 journals to characterize their design and their methods for sample size estimation and data analysis. Only 66 (53.7%) of the articles reported appropriate methods for sample size estimation. Only 63 (51.2%) reported exclusively appropriate methods for analysis. These findings suggest that many investigators do not adequately attend to the methodological challenges inherent in group-randomized trials. These practices can lead to underpowered studies, to an inflated type 1 error rate, and to inferences that mislead readers. Investigators should work with biostatisticians or other methodologists familiar with these issues. Funders and editors should ensure careful methodological review of applications and manuscripts. Reviewers should ensure that studies are properly planned and analyzed. These steps are needed to improve the rigor and reproducibility of group-randomized trials. The Office of Disease Prevention (ODP) at the National Institutes of Health (NIH) has taken several steps to address these issues. ODP offers an online course on the design and analysis of group-randomized trials. ODP is working to increase the number of methodologists who serve on grant review panels. ODP has developed standard language for the Application Guide and the Review Criteria to draw investigators' attention to these issues. Finally, ODP has created a new Research Methods Resources website to help investigators, reviewers, and NIH staff better understand these issues.

An evaluation of constrained randomization for the design and analysis of group-randomized trials with binary outcomes.

Group-randomized trials are randomized studies that allocate intact groups of individuals to different comparison arms. A frequent practical limitation to adopting such research designs is that only a limited number of groups may be available, and therefore, simple randomization is unable to adequately balance multiple group-level covariates between arms. Therefore, covariate-based constrained randomization was proposed as an allocation technique to achieve balance. Constrained randomization involves generating a large number of possible allocation schemes, calculating a balance score that assesses covariate imbalance, limiting the randomization space to a prespecified percentage of candidate allocations, and randomly selecting one scheme to implement. When the outcome is binary, a number of statistical issues arise regarding the potential advantages of such designs in making inference. In particular, properties found for continuous outcomes may not directly apply, and additional variations on statistical tests are available. Motivated by two recent trials, we conduct a series of Monte Carlo simulations to evaluate the statistical properties of model-based and randomization-based tests under both simple and constrained randomization designs, with varying degrees of analysis-based covariate adjustment. Our results indicate that constrained randomization improves the power of the linearization F-test, the KC-corrected GEE t-test (Kauermann and Carroll, 2001, Journal of the American Statistical Association 96, 1387-1396), and two permutation tests when the prognostic group-level variables are controlled for in the analysis and the size of randomization space is reasonably small. We also demonstrate that constrained randomization reduces power loss from redundant analysis-based adjustment for non-prognostic covariates. Design considerations such as the choice of the balance metric and the size of randomization space are discussed.