Reviewing Peer Review at the NIH.

Recent reports suggest that peer reviews of National Institutes of Health grant applications are at best imprecise predictors of research projects' scientific impact. But these findings may not mean that peer review is failing.

Predicting Productivity Returns on Investment: Thirty Years of Peer Review, Grant Funding, and Publication of Highly Cited Papers at the National Heart, Lung, and Blood Institute.

There are conflicting data about the ability of peer review percentile rankings to predict grant productivity, as measured through publications and citations. To understand the nature of these apparent conflicting findings, we analyzed bibliometric outcomes of 6873 de novo cardiovascular R01 grants funded by the National Heart, Lung, and Blood Institute (NHLBI) between 1980 and 2011. Our outcomes focus on top-10% articles, meaning articles that were cited more often than 90% of other articles on the same topic, of the same type (eg, article, editorial), and published in the same year. The 6873 grants yielded 62 468 articles, of which 13 507 (or 22%) were top-10% articles. There was a modest association between better grant percentile ranking and number of top-10% articles. However, discrimination was poor (area under receiver operating characteristic curve [ROC], 0.52; 95% confidence interval, 0.51-0.53). Furthermore, better percentile ranking was also associated with higher annual and total inflation-adjusted grant budgets. There was no association between grant percentile ranking and grant outcome as assessed by number of top-10% articles per $million spent. Hence, the seemingly conflicting findings on peer review percentile ranking of grants and subsequent productivity largely reflect differing questions and outcomes. Taken together, these findings raise questions about how best National Institutes of Health (NIH) should use peer review assessments to make complex funding decisions.

Citation impact of NHLBI R01 grants funded through the American Recovery and Reinvestment Act as compared to R01 grants funded through a standard payline.

RATIONALE: The American Recovery and Reinvestment Act (ARRA) allowed National Heart, Lung, and Blood Institute to fund R01 grants that fared less well on peer review than those funded by meeting a payline threshold. It is not clear whether the sudden availability of additional funding enabled research of similar or lesser citation impact than already funded work. OBJECTIVE: To compare the citation impact of ARRA-funded de novo National Heart, Lung, and Blood Institute R01 grants with concurrent de novo National Heart, Lung, and Blood Institute R01 grants funded by standard payline mechanisms. METHODS AND RESULTS: We identified de novo (type 1) R01 grants funded by National Heart, Lung, and Blood Institute in fiscal year 2009: these included 458 funded by meeting Institute's published payline and 165 funded only because of ARRA funding. Compared with payline grants, ARRA grants received fewer total funds (median values, $1.03 versus $1.87 million; P<0.001) for a shorter duration (median values including no-cost extensions, 3.0 versus 4.9 years; P<0.001). Through May 2014, the payline R01 grants generated 3895 publications, whereas the ARRA R01 grants generated 996. Using the InCites database from Thomson-Reuters, we calculated a normalized citation impact for each grant by weighting each article for the number of citations it received normalizing for subject, article type, and year of publication. The ARRA R01 grants had a similar normalized citation impact per $1 million spent as the payline grants (median values [interquartile range], 2.15 [0.73-4.68] versus 2.03 [0.75-4.10]; P=0.61). The similar impact of the ARRA grants persisted even after accounting for potential confounders. CONCLUSIONS: Despite shorter durations and lower budgets, ARRA R01 grants had comparable citation outcomes per $million spent to that of contemporaneously funded payline R01 grants.

Publication of trials funded by the National Heart, Lung, and Blood Institute.

BACKGROUND: Rapid publication of clinical trials is essential in order for the findings to yield maximal benefits for public health and scientific progress. Factors affecting the speed of publication of the main results of government-funded trials have not been well characterized. METHODS: We analyzed 244 extramural randomized clinical trials of cardiovascular interventions that were supported by the National Heart, Lung, and Blood Institute (NHLBI). We selected trials for which data collection had been completed between January 1, 2000, and December 31, 2011. Our primary outcome measure was the time between completion of the trial and publication of the main results in a peer-reviewed journal. RESULTS: As of March 31, 2012, the main results of 156 trials (64%) had been published (Kaplan-Meier median time to publication, 25 months, with 57% published within 30 months). Trials that focused on clinical events were published more rapidly than those that focused on surrogate measures (median, 9 months vs. 31 months; P<0.001). The only independent predictors of more rapid publication were a focus on clinical events rather than surrogate end points (adjusted publication rate ratio, 2.11; 95% confidence interval, 1.26 to 3.53; P=0.004) and higher costs of conducting the trial, up to a threshold of approximately $5 million (P<0.001). The 37 trials that focused on clinical events and cost at least $5 million accounted for 67% of the funds spent on clinical trials but received 82% of the citations. After adjustment of the analysis for a focus on clinical events and for cost, trial results that were classified as positive were published more quickly than those classified as negative. CONCLUSIONS: Results of less than two thirds of NHLBI-funded randomized clinical trials of cardiovascular interventions were published within 30 months after completion of the trial. Trials that focused on clinical events were published more quickly than those that focused on surrogate end points. (Funded by the National Heart, Lung, and Blood Institute.).

Prior publication productivity, grant percentile ranking, and topic-normalized citation impact of NHLBI cardiovascular R01 grants.

RATIONALE: We previously demonstrated absence of association between peer-review-derived percentile ranking and raw citation impact in a large cohort of National Heart, Lung, and Blood Institute cardiovascular R01 grants, but we did not consider pregrant investigator publication productivity. We also did not normalize citation counts for scientific field, type of article, and year of publication. OBJECTIVE: To determine whether measures of investigator prior productivity predict a grant's subsequent scientific impact as measured by normalized citation metrics. METHODS AND RESULTS: We identified 1492 investigator-initiated de novo National Heart, Lung, and Blood Institute R01 grant applications funded between 2001 and 2008 and linked the publications from these grants to their InCites (Thompson Reuters) citation record. InCites provides a normalized citation count for each publication stratifying by year of publication, type of publication, and field of science. The coprimary end points for this analysis were the normalized citation impact per million dollars allocated and the number of publications per grant that has normalized citation rate in the top decile per million dollars allocated (top 10% articles). Prior productivity measures included the number of National Heart, Lung, and Blood Institute-supported publications each principal investigator published in the 5 years before grant review and the corresponding prior normalized citation impact score. After accounting for potential confounders, there was no association between peer-review percentile ranking and bibliometric end points (all adjusted P>0.5). However, prior productivity was predictive (P<0.0001). CONCLUSIONS: Even after normalizing citation counts, we confirmed a lack of association between peer-review grant percentile ranking and grant citation impact. However, prior investigator publication productivity was predictive of grant-specific citation impact.

Rigorous science as the road to better public health.

In the current issue of Population Health Metrics, two reports paint a bleak picture of American public health. Both physical inactivity and obesity remain highly prevalent; yet, it is not clear that increased physical activity will reduce the burden of obesity. There continue to be widespread disparities in life expectancy across United States counties. These reports appear against a backdrop of debate regarding how we should allocate our scarce resources for improving health: should we focus more on improving access to high-quality medical care, or should we instead focus on more and better public health interventions? While optimal solutions remain obscure, a look at prior successes suggests that ultimately they will come from the conduct and implementation of rigorous science, and in particular event-driven trials.

National Institutes of Health research project grant inflation 1998 to 2021.

We analyzed changes in total costs for National Institutes of Health (NIH) awarded Research Project Grants (RPGs) issued from fiscal years (FYs) 1998 to 2021 . Costs are measured in 'nominal' terms, meaning exactly as stated, or in 'real' terms, meaning after adjustment for inflation. The NIH uses a data-driven price index - the Biomedical Research and Development Price Index (BRDPI) - to account for inflation, enabling assessment of changes in real (that is, BRDPI-adjusted) costs over time. The BRDPI was higher than the general inflation rate from FY1998 until FY2012; since then the BRDPI has been similar to the general inflation rate likely due to caps on senior faculty salary support. Despite increases in nominal costs, recent years have seen increases in the absolute numbers of RPG and R01 awards. Real average and median RPG costs increased during the NIH-doubling (FY1998 to FY2003), decreased after the doubling and have remained relatively stable since. Of note, though, the degree of variation of RPG costs has changed over time, with more marked extremes observed on both higher and lower levels of cost. On both ends of the cost spectrum, the agency is funding a greater proportion of solicited projects, with nearly half of RPG money going toward solicited projects. After adjusting for confounders, we find no independent association of time with BRDPI-adjusted costs; in other words, changes in real costs are largely explained by changes in the composition of the NIH-grant portfolio.

Cardiovascular epidemiology in a changing world--challenges to investigators and the National Heart, Lung, and Blood Institute.

Over the past 60 years, revolutionary discoveries made by epidemiologists have contributed to marked declines in cardiovascular disease morbidity and mortality. Now, in an era of increasingly constrained resources, researchers in cardiovascular epidemiology face a number of challenges that call for novel, paradigm-shifting approaches. In this paper, the authors pose to the community 4 critical questions: 1) How can we avoid wasting resources on studies that provide little incremental knowledge? 2) How can we assure that we direct our resources as economically as possible towards innovative science? 3) How can we be nimble, responding quickly to new opportunities? 4) How can we identify prospectively the most meritorious research questions? Senior program staff at the National Heart, Lung, and Blood Institute invite the epidemiology community to join them in an ongoing Web-based blog conversation so that together we might develop novel approaches that will facilitate the next generation of high-impact discoveries.

The role of comparative effectiveness research in transfusion medicine clinical trials: proceedings of a National Heart, Lung, and Blood Institute workshop.

Comparative effectiveness research (CER) is the study of existing treatments or ways to deliver health care to determine what intervention works best under specific circumstances. CER evaluates evidence from existing studies or generates new evidence, in different populations and under specific conditions in which the treatments are actually used. CER does not embrace one research design over another but compares treatments and variations in practice using methods that are most likely to yield widely generalizable results that are directly relevant to clinical practice. Treatments used in transfusion medicine (TM) are among the most widely used in clinical practice, but are among the least well studied. High-quality evidence is lacking for most transfusion practices, with research efforts hampered by regulatory restrictions and ethical barriers. To begin addressing these issues, the National Heart, Lung, and Blood Institute convened a workshop in June 2011 to address the potential role of CER in the generation of high-quality evidence for TM decision making. Workshop goals were to: 1) evaluate the current landscape of clinical research, 2) review the potential application of CER methods to clinical research, 3) assess potential barriers to the use of CER methodology, 4) determine whether pilot or vanguard studies can be used to facilitate planning of future CER research, and 5) consider the need for and delivery of training in CER methods for researchers.

Commentary: How the debate about comparative effectiveness research should impact the future of clinical trials.

Comparative effectiveness research represents the kind of research that arguably more directly affects clinical practice and policy. It includes observational studies, clinical trials, and systematic syntheses of existing literature. In this commentary, I argue for the ongoing and critical role of randomization in comparative effectiveness, noting the key differences between practical and explanatory trials.