Early administration of COVID-19 convalescent plasma with high titer antibody content by live viral neutralization assay is associated with modest clinical efficacy.

The efficacy of COVID-19 convalescent plasma (CCP) as a treatment for hospitalized patients with COVID-19 remains somewhat controversial; however, many studies have not evaluated CCP documented to have high neutralizing antibody titer by a highly accurate assay. To evaluate the correlation of the administration of CCP with titer determined by a live viral neutralization assay with 7- and 28-day death rates during hospitalization, a total of 23 118 patients receiving a single unit of CCP were stratified into two groups: those receiving high titer CCP (>250 50% inhibitory dilution, ID50; n = 13 636) or low titer CCP (≤250 ID50; n = 9482). Multivariable Cox regression was performed to assess risk factors. Non-intubated patients who were transfused with high titer CCP showed 1.1% and 1.7% absolute reductions in overall 7- and 28-day death rates, respectively, compared to those non-intubated patients receiving low titer CCP. No benefit of CCP was observed in intubated patients. The relative benefit of high titer CCP was confirmed in multivariable Cox regression. Administration of CCP with high titer antibody content determined by live viral neutralization assay to non-intubated patients is associated with modest clinical efficacy. Although shown to be only of modest clinical benefit, CCP may play a role in the future should viral variants develop that are not neutralized by other available therapeutics.

Reviewing Peer Review at the NIH.

Recent reports suggest that peer reviews of National Institutes of Health grant applications are at best imprecise predictors of research projects' scientific impact. But these findings may not mean that peer review is failing.

Unique Review Criteria and Patient and Stakeholder Reviewers: Analysis of PCORI's Approach to Research Funding.

OBJECTIVE: The Patient-Centered Outcomes Research Institute (PCORI) uses a unique approach to Merit Review that includes patients and stakeholders as reviewers with scientists, and includes unique review criteria (patient-centeredness and active engagement of end users in the research). This study assessed the extent to which different reviewer types influence review scores and funding outcomes, the emphasis placed on technical merit compared to other criteria by a multistakeholder panel, and the impact of the in-person discussion on agreement among different reviewer types. METHODS: Cross-sectional analysis of administrative data from PCORI online and in-person Merit Review (N = 1312 applications from the five funding cycles from November 2013 to August 2015). Linear and logistic regression models were used to analyze the data. RESULTS: For all reviewer types, final review scores were associated with at least one review criterion score from each of the three reviewer types. The strongest predictor of final overall scores for all reviewer types was scientists' prediscussion ratings of technical merit. All reviewers' prediscussion ratings of the potential to improve health care and outcomes, and scientists' ratings of technical merit and patient-centeredness, were associated with funding success. For each reviewer type, overall impact scores from the online scoring were changed on at least half of the applications at the in-person panel discussion. Score agreement across reviewer types was greater after panel discussion. CONCLUSIONS: Scientist, patient, and stakeholder views all contribute to PCORI Merit Review of applications for research funding. Technical merit is critical to funding success but patient and stakeholder ratings of other criteria also influence application disposition.

Researchers, Patients, and Stakeholders Evaluating Comparative-Effectiveness Research: A Mixed-Methods Study of the PCORI Reviewer Experience.

OBJECTIVES: The Patient-Centered Outcomes Research Institute (PCORI) includes patients and stakeholders alongside scientists in reviewing research applications using unique review criteria including patient-centeredness and patient and/or stakeholder engagement. To support extension of this unique collaborative model to other funders, information from the reviewers on the review process is needed to understand how scientists and nonscientists evaluate research proposals together. Thus, this study aimed to describe reviewers' perspectives of the interactions during the in-person review panel; to examine the value and challenges of including scientists, patients, and stakeholders together; and to understand the perceived importance of PCORI's review criteria. METHODS: This study utilized anonymous, cross-sectional surveys (N = 925 respondents from 5 funding cycles: 470 scientists, 217 patients, 238 stakeholders; survey completion rates by cycle: 70-89%) and group interviews (N = 18). RESULTS: Reviewers of all types describe PCORI Merit Review as respectful, balanced, and one of reciprocal influence among different reviewer types. Reviewers indicate strong support and value of input from all reviewer types, receptivity to input from others, and the panel chair's incorporation of all views. Patients and stakeholders provide real-world perspectives on importance to patients, research partnership plans, and study feasibility. Challenges included concerns about a lack of technical expertise of patient/stakeholder reviewers and about scientists dominating conversations. The most important criterion for assigning final review scores was technical merit-either alone or in conjunction with patient-centeredness or patient/ stakeholder engagement. CONCLUSIONS: PCORI Merit Reviewers' self-reports indicate that the perspectives of different reviewer types are influential in panel discussions and Merit Review outcomes.

Predicting Productivity Returns on Investment: Thirty Years of Peer Review, Grant Funding, and Publication of Highly Cited Papers at the National Heart, Lung, and Blood Institute.

There are conflicting data about the ability of peer review percentile rankings to predict grant productivity, as measured through publications and citations. To understand the nature of these apparent conflicting findings, we analyzed bibliometric outcomes of 6873 de novo cardiovascular R01 grants funded by the National Heart, Lung, and Blood Institute (NHLBI) between 1980 and 2011. Our outcomes focus on top-10% articles, meaning articles that were cited more often than 90% of other articles on the same topic, of the same type (eg, article, editorial), and published in the same year. The 6873 grants yielded 62 468 articles, of which 13 507 (or 22%) were top-10% articles. There was a modest association between better grant percentile ranking and number of top-10% articles. However, discrimination was poor (area under receiver operating characteristic curve [ROC], 0.52; 95% confidence interval, 0.51-0.53). Furthermore, better percentile ranking was also associated with higher annual and total inflation-adjusted grant budgets. There was no association between grant percentile ranking and grant outcome as assessed by number of top-10% articles per $million spent. Hence, the seemingly conflicting findings on peer review percentile ranking of grants and subsequent productivity largely reflect differing questions and outcomes. Taken together, these findings raise questions about how best National Institutes of Health (NIH) should use peer review assessments to make complex funding decisions.

Citation impact of NHLBI R01 grants funded through the American Recovery and Reinvestment Act as compared to R01 grants funded through a standard payline.

RATIONALE: The American Recovery and Reinvestment Act (ARRA) allowed National Heart, Lung, and Blood Institute to fund R01 grants that fared less well on peer review than those funded by meeting a payline threshold. It is not clear whether the sudden availability of additional funding enabled research of similar or lesser citation impact than already funded work. OBJECTIVE: To compare the citation impact of ARRA-funded de novo National Heart, Lung, and Blood Institute R01 grants with concurrent de novo National Heart, Lung, and Blood Institute R01 grants funded by standard payline mechanisms. METHODS AND RESULTS: We identified de novo (type 1) R01 grants funded by National Heart, Lung, and Blood Institute in fiscal year 2009: these included 458 funded by meeting Institute's published payline and 165 funded only because of ARRA funding. Compared with payline grants, ARRA grants received fewer total funds (median values, $1.03 versus $1.87 million; P<0.001) for a shorter duration (median values including no-cost extensions, 3.0 versus 4.9 years; P<0.001). Through May 2014, the payline R01 grants generated 3895 publications, whereas the ARRA R01 grants generated 996. Using the InCites database from Thomson-Reuters, we calculated a normalized citation impact for each grant by weighting each article for the number of citations it received normalizing for subject, article type, and year of publication. The ARRA R01 grants had a similar normalized citation impact per $1 million spent as the payline grants (median values [interquartile range], 2.15 [0.73-4.68] versus 2.03 [0.75-4.10]; P=0.61). The similar impact of the ARRA grants persisted even after accounting for potential confounders. CONCLUSIONS: Despite shorter durations and lower budgets, ARRA R01 grants had comparable citation outcomes per $million spent to that of contemporaneously funded payline R01 grants.

Publication of trials funded by the National Heart, Lung, and Blood Institute.

BACKGROUND: Rapid publication of clinical trials is essential in order for the findings to yield maximal benefits for public health and scientific progress. Factors affecting the speed of publication of the main results of government-funded trials have not been well characterized. METHODS: We analyzed 244 extramural randomized clinical trials of cardiovascular interventions that were supported by the National Heart, Lung, and Blood Institute (NHLBI). We selected trials for which data collection had been completed between January 1, 2000, and December 31, 2011. Our primary outcome measure was the time between completion of the trial and publication of the main results in a peer-reviewed journal. RESULTS: As of March 31, 2012, the main results of 156 trials (64%) had been published (Kaplan-Meier median time to publication, 25 months, with 57% published within 30 months). Trials that focused on clinical events were published more rapidly than those that focused on surrogate measures (median, 9 months vs. 31 months; P<0.001). The only independent predictors of more rapid publication were a focus on clinical events rather than surrogate end points (adjusted publication rate ratio, 2.11; 95% confidence interval, 1.26 to 3.53; P=0.004) and higher costs of conducting the trial, up to a threshold of approximately $5 million (P<0.001). The 37 trials that focused on clinical events and cost at least $5 million accounted for 67% of the funds spent on clinical trials but received 82% of the citations. After adjustment of the analysis for a focus on clinical events and for cost, trial results that were classified as positive were published more quickly than those classified as negative. CONCLUSIONS: Results of less than two thirds of NHLBI-funded randomized clinical trials of cardiovascular interventions were published within 30 months after completion of the trial. Trials that focused on clinical events were published more quickly than those that focused on surrogate end points. (Funded by the National Heart, Lung, and Blood Institute.).

Percentile ranking and citation impact of a large cohort of National Heart, Lung, and Blood Institute-funded cardiovascular R01 grants.

RATIONALE: Funding decisions for cardiovascular R01 grant applications at the National Heart, Lung, and Blood Institute (NHLBI) largely hinge on percentile rankings. It is not known whether this approach enables the highest impact science. OBJECTIVE: Our aim was to conduct an observational analysis of percentile rankings and bibliometric outcomes for a contemporary set of funded NHLBI cardiovascular R01 grants. METHODS AND RESULTS: We identified 1492 investigator-initiated de novo R01 grant applications that were funded between 2001 and 2008 and followed their progress for linked publications and citations to those publications. Our coprimary end points were citations received per million dollars of funding, citations obtained <2 years of publication, and 2-year citations for each grant's maximally cited paper. In 7654 grant-years of funding that generated $3004 million of total National Institutes of Health awards, the portfolio yielded 16 793 publications that appeared between 2001 and 2012 (median per grant, 8; 25th and 75th percentiles, 4 and 14; range, 0-123), which received 2 224 255 citations (median per grant, 1048; 25th and 75th percentiles, 492 and 1932; range, 0-16 295). We found no association between percentile rankings and citation metrics; the absence of association persisted even after accounting for calendar time, grant duration, number of grants acknowledged per paper, number of authors per paper, early investigator status, human versus nonhuman focus, and institutional funding. An exploratory machine learning analysis suggested that grants with the best percentile rankings did yield more maximally cited papers. CONCLUSIONS: In a large cohort of NHLBI-funded cardiovascular R01 grants, we were unable to find a monotonic association between better percentile ranking and higher scientific impact as assessed by citation metrics.

Prior publication productivity, grant percentile ranking, and topic-normalized citation impact of NHLBI cardiovascular R01 grants.

RATIONALE: We previously demonstrated absence of association between peer-review-derived percentile ranking and raw citation impact in a large cohort of National Heart, Lung, and Blood Institute cardiovascular R01 grants, but we did not consider pregrant investigator publication productivity. We also did not normalize citation counts for scientific field, type of article, and year of publication. OBJECTIVE: To determine whether measures of investigator prior productivity predict a grant's subsequent scientific impact as measured by normalized citation metrics. METHODS AND RESULTS: We identified 1492 investigator-initiated de novo National Heart, Lung, and Blood Institute R01 grant applications funded between 2001 and 2008 and linked the publications from these grants to their InCites (Thompson Reuters) citation record. InCites provides a normalized citation count for each publication stratifying by year of publication, type of publication, and field of science. The coprimary end points for this analysis were the normalized citation impact per million dollars allocated and the number of publications per grant that has normalized citation rate in the top decile per million dollars allocated (top 10% articles). Prior productivity measures included the number of National Heart, Lung, and Blood Institute-supported publications each principal investigator published in the 5 years before grant review and the corresponding prior normalized citation impact score. After accounting for potential confounders, there was no association between peer-review percentile ranking and bibliometric end points (all adjusted P>0.5). However, prior productivity was predictive (P<0.0001). CONCLUSIONS: Even after normalizing citation counts, we confirmed a lack of association between peer-review grant percentile ranking and grant citation impact. However, prior investigator publication productivity was predictive of grant-specific citation impact.

Engaging patients and stakeholders in research proposal review: the patient-centered outcomes research institute.

The inaugural round of merit review for the Patient-Centered Outcomes Research Institute (PCORI) in November 2012 included patients and other stakeholders, as well as scientists. This article examines relationships among scores of the 3 reviewer types, changes in scoring after in-person discussion, and the effect of inclusion of patient and stakeholder reviewers on the review process. In the first phase, 363 scientists scored 480 applications. In the second phase, 59 scientists, 21 patients, and 31 stakeholders provided a "prediscussion" score and a final "postdiscussion" score after an in-person meeting for applications. Bland-Altman plots were used to characterize levels of agreement among and within reviewer types before and after discussion. Before discussion, there was little agreement among average scores given by the 4 lead scientific reviewers and patient and stakeholder reviewers. After discussion, the 4 primary reviewers showed mild convergence in their scores, and the 21-member panel came to a much stronger agreement. Of the 25 awards with the best (and lowest) scores after phase 2, only 13 had ranked in the top 25 after the phase 1 review by scientists. Five percent of the 480 proposals submitted were funded. The authors conclude that patient and stakeholder reviewers brought different perspectives to the review process but that in-person discussion led to closer agreement among reviewer types. It is not yet known whether these conclusions are generalizable to future rounds of peer review. Future work would benefit from additional data collection for evaluation purposes and from long-term evaluation of the effect on the funded research.

Characterization of Mycobacteriophage Adephagia Cytotoxic Proteins.

Mycobacterium phage Adephagia is a Cluster K phage that infects Mycobacterium smegmatis and some strains of Mycobacterium pathogens. Adephagia has a siphoviral virion morphology and is temperate. Its genome is 59,646 bp long and codes for one tRNA gene and 94 predicted protein-coding genes; most genes not associated with virion structure and assembly are functionally ill-defined. Here, we determined the Adephagia gene expression patterns in lytic and lysogenic growth and used structural predictions to assign additional gene functions. We characterized 66 non-structural genes for their toxic phenotypes when expressed in M. smegmatis, and we show that 25 of these (38%) are either toxic or strongly inhibit growth, resulting in either reduced viability or small colony sizes. Some of these genes are predicted to be involved in DNA metabolism or regulation, but others are of unknown function. We also characterize the HicAB-like toxin-antitoxin system encoded by Adephagia (gp91 and gp90, respectively) and show that the gp90 antitoxin is lysogenically expressed, abrogates gp91 toxicity, and is required for normal lytic and lysogenic growth.

Rigorous science as the road to better public health.

In the current issue of Population Health Metrics, two reports paint a bleak picture of American public health. Both physical inactivity and obesity remain highly prevalent; yet, it is not clear that increased physical activity will reduce the burden of obesity. There continue to be widespread disparities in life expectancy across United States counties. These reports appear against a backdrop of debate regarding how we should allocate our scarce resources for improving health: should we focus more on improving access to high-quality medical care, or should we instead focus on more and better public health interventions? While optimal solutions remain obscure, a look at prior successes suggests that ultimately they will come from the conduct and implementation of rigorous science, and in particular event-driven trials.

National Institutes of Health research project grant inflation 1998 to 2021.

We analyzed changes in total costs for National Institutes of Health (NIH) awarded Research Project Grants (RPGs) issued from fiscal years (FYs) 1998 to 2021 . Costs are measured in 'nominal' terms, meaning exactly as stated, or in 'real' terms, meaning after adjustment for inflation. The NIH uses a data-driven price index - the Biomedical Research and Development Price Index (BRDPI) - to account for inflation, enabling assessment of changes in real (that is, BRDPI-adjusted) costs over time. The BRDPI was higher than the general inflation rate from FY1998 until FY2012; since then the BRDPI has been similar to the general inflation rate likely due to caps on senior faculty salary support. Despite increases in nominal costs, recent years have seen increases in the absolute numbers of RPG and R01 awards. Real average and median RPG costs increased during the NIH-doubling (FY1998 to FY2003), decreased after the doubling and have remained relatively stable since. Of note, though, the degree of variation of RPG costs has changed over time, with more marked extremes observed on both higher and lower levels of cost. On both ends of the cost spectrum, the agency is funding a greater proportion of solicited projects, with nearly half of RPG money going toward solicited projects. After adjusting for confounders, we find no independent association of time with BRDPI-adjusted costs; in other words, changes in real costs are largely explained by changes in the composition of the NIH-grant portfolio.

Virion glycosylation influences mycobacteriophage immune recognition.

Glycosylation of eukaryotic virus particles is common and influences their uptake, trafficking, and immune recognition. In contrast, glycosylation of bacteriophage particles has not been reported; phage virions typically do not enter the cytoplasm upon infection, and they do not generally inhabit eukaryotic systems. We show here that several genomically distinct phages of Mycobacteria are modified with glycans attached to the C terminus of capsid and tail tube protein subunits. These O-linked glycans influence antibody production and recognition, shielding viral particles from antibody binding and reducing production of neutralizing antibodies. Glycosylation is mediated by phage-encoded glycosyltransferases, and genomic analysis suggests that they are relatively common among mycobacteriophages. Putative glycosyltransferases are also encoded by some Gordonia and Streptomyces phages, but there is little evidence of glycosylation among the broader phage population. The immune response to glycosylated phage virions in mice suggests that glycosylation may be an advantageous property for phage therapy of Mycobacterium infections.

Cardiovascular epidemiology in a changing world--challenges to investigators and the National Heart, Lung, and Blood Institute.

Over the past 60 years, revolutionary discoveries made by epidemiologists have contributed to marked declines in cardiovascular disease morbidity and mortality. Now, in an era of increasingly constrained resources, researchers in cardiovascular epidemiology face a number of challenges that call for novel, paradigm-shifting approaches. In this paper, the authors pose to the community 4 critical questions: 1) How can we avoid wasting resources on studies that provide little incremental knowledge? 2) How can we assure that we direct our resources as economically as possible towards innovative science? 3) How can we be nimble, responding quickly to new opportunities? 4) How can we identify prospectively the most meritorious research questions? Senior program staff at the National Heart, Lung, and Blood Institute invite the epidemiology community to join them in an ongoing Web-based blog conversation so that together we might develop novel approaches that will facilitate the next generation of high-impact discoveries.