NIH Peer Review: Scored Review Criteria and Overall Impact.

The National Institutes of Health (NIH) is the largest source of funding for biomedical research in the world. Funding decisions are made largely based on the outcome of a peer review process that is intended to provide a fair, equitable, timely, and unbiased review of the quality, scientific merit, and potential impact of the research. There have been concerns about the criteria reviewers are using, and recent changes in review procedures at the NIH now make it possible to conduct an analysis of how reviewers evaluate applications for funding. This study examined the criteria and overall impact scores recorded by assigned reviewers for R01 grant applications. The results suggest that all the scored review criteria, including innovation, are related to the overall impact score. Further, good scores are necessary on all five scored review criteria, not just the score for research methodology, in order to achieve a good overall impact score.

An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes.

Background: Blinding reviewers to applicant identity has been proposed to reduce bias in peer review. Methods: This experimental test used 1200 NIH grant applications, 400 from Black investigators, 400 matched applications from White investigators, and 400 randomly selected applications from White investigators. Applications were reviewed by mail in standard and redacted formats. Results: Redaction reduced, but did not eliminate, reviewers' ability to correctly guess features of identity. The primary, preregistered analysis hypothesized a differential effect of redaction according to investigator race in the matched applications. A set of secondary analyses (not preregistered) used the randomly selected applications from White scientists and tested the same interaction. Both analyses revealed similar effects: Standard format applications from White investigators scored better than those from Black investigators. Redaction cut the size of the difference by about half (e.g. from a Cohen's d of 0.20-0.10 in matched applications); redaction caused applications from White scientists to score worse but had no effect on scores for Black applications. Conclusions: Grant-writing considerations and halo effects are discussed as competing explanations for this pattern. The findings support further evaluation of peer review models that diminish the influence of applicant identity. Funding: Funding was provided by the NIH.

NIH peer review: Criterion scores completely account for racial disparities in overall impact scores.

Previous research has found that funding disparities are driven by applications' final impact scores and that only a portion of the black/white funding gap can be explained by bibliometrics and topic choice. Using National Institutes of Health R01 applications for council years 2014-2016, we examine assigned reviewers' preliminary overall impact and criterion scores to evaluate whether racial disparities in impact scores can be explained by application and applicant characteristics. We hypothesize that differences in commensuration-the process of combining criterion scores into overall impact scores-disadvantage black applicants. Using multilevel models and matching on key variables including career stage, gender, and area of science, we find little evidence for racial disparities emerging in the process of combining preliminary criterion scores into preliminary overall impact scores. Instead, preliminary criterion scores fully account for racial disparities-yet do not explain all of the variability-in preliminary overall impact scores.

Scopolamine induced deficits in a battery of rat cognitive tests: comparisons of sensitivity and specificity.

Despite much research, the cognitive effects of scopolamine hydrobromide, a cholinergic antagonist, remain controversial. Scopolamine affects multiple systems each of which can impact behavior. One way to tease apart the effects of the drug is to determine the effects of low scopolamine doses on different abilities. The present experiments compared the effects of low doses of scopolamine on a single group of rats conducting a battery of behavioral tasks: Morris water maze, radial arm maze, delayed non-matching to position tasks, and fixed ratio 5 bar pressing. The behavioral battery ranged from tasks having little cognitive demand to those thought to be based more on attention and spatial-working memory. Control experiments using additional groups of rats assessing peripheral versus central effects were conducted with both liquid and dry reinforcement and with methyl scopolamine. Furthermore, the 5-choice serial reaction time test assessed scopolamine effects on attention. The data show a wide spectrum of central and peripheral cholinergic involvement. The central effects include attention and motor initiation, both of which impact and interact with the mnemonic function of acetylcholine. These results show that a limited disruption of the central cholinergic system can have profound effects on attention and/or psychomotor control before any measurable mnemonic disruption.

Clinical attrition due to biased preclinical assessments of potential efficacy.

Unless it is carefully controlled, bias often distorts the results of clinical trials, usually exaggerating the magnitude of true efficacy. For that reason, procedures to limit bias have been mandated by the FDA when assessing efficacy in clinical trials. The present review shows that the effects of bias in preclinical studies are at least as large as in clinical trials, and since bias is not usually controlled in preclinical proof of concept studies, compounds that actually have little or no therapeutic potential may often be advanced into clinical trials. This possibility is supported by the fact that lack of efficacy is the single biggest reason why compounds fail in the clinic. The shift to target-based discovery during the last 10-15 years may have further increased the effects of bias on preclinical assessments of potential efficacy, and contributed to the continuing decline in clinical success rates. Procedures are available to control for bias during preclinical assessments of potential efficacy, and their use could dramatically increase clinical success rates and substantially reduce the costs of drug discovery and development.

Scientific productivity: An exploratory study of metrics and incentives.

Competitive pressure to maximize the current bibliometric measures of productivity is jeopardizing the integrity of the scientific literature. Efforts are underway to address the 'reproducibility crisis' by encouraging the use of more rigorous, confirmatory methods. However, as long as productivity continues to be defined by the number of discoveries scientists publish, the impact factor of the journals they publish in and the number of times their papers are cited, they will be reluctant to accept high quality methods and consistently conduct and publish confirmatory/replication studies. This exploratory study examined a sample of rigorous Phase II-IV clinical trials, including unpublished studies, to determine if more appropriate metrics and incentives can be developed. The results suggest that rigorous procedures will help reduce false positives, but to the extent that higher quality methods are accepted as the standard of practice, the current bibliometric incentives will discourage innovative studies and encourage scientists to shift their research to less informative studies of subjects that are already being more actively investigated. However, the results also suggest that it is possible to develop a more appropriate system of rewards. In contrast to the current bibliometric incentives, evaluations of the quality of the methods and reproducibility of the results, innovation and diversity of thought, and amount of information produced may serve as measures and incentives that maintain the integrity of the scientific literature and maximize scientific progress.

Donepezil primarily attenuates scopolamine-induced deficits in psychomotor function, with moderate effects on simple conditioning and attention, and small effects on working memory and spatial mapping.

RATIONALE: Alzheimer's dementia (AD) patients have profound deficits in cognitive and social functions, mediated in part by a decline in cholinergic function. Acetylcholinesterase inhibitors (AChEI) are the most commonly prescribed treatment for the cognitive deficits in AD patients, but their therapeutic effects are small, and it is still not clear if they primarily affect attention, memory, or some other cognitive/behavioral functions. OBJECTIVES: The objective of the present experiments was to explore the effects of donepezil (Aricepttrade mark), an AChEI, on behavioral deficits related exclusively to cholinergic dysfunction. MATERIALS AND METHODS: The effects of donepezil were assessed in Sprague-Dawley rats with scopolamine-induced deficits in a battery of cognitive/behavioral tests. RESULTS: Scopolamine produced deficits in contextual and cued fear conditioning, the 5-choice serial reaction time test, delayed nonmatching to position, the radial arm maze, and the Morris water maze. Analyses of the pattern and size of the effects revealed that donepezil produced very large effects on scopolamine-induced deficits in psychomotor function (approximately 20-50% of the variance), moderate-sized effects on scopolamine-induced deficits in simple conditioning and attention (approximately 3-10% of the variance), but only small effects on scopolamine-induced deficits in higher cognitive functions of working memory and spatial mapping (approximately 1% of the variance). CONCLUSIONS: These results are consistent with the limited efficacy of donepezil on higher cognitive function in AD patients, and suggest that preclinical behavioral models could be used not only to determine if novel treatments have some therapeutic potential, but also to predict more precisely what the pattern and size of the effects might be.

Soluble Abeta and cognitive function in aged F-344 rats and Tg2576 mice.

Recent findings suggest that Alzheimer's dementia may be mediated by soluble beta amyloid (Abeta) more than the deposits of aggregated, insoluble Abeta, and vulnerability to cognitive deficits after scopolamine challenge may help identify AD even in patients that are still pre-symptomatic. The objectives of the present experiments were to determine if vulnerability to cognitive deficits after scopolamine challenge is related to levels of soluble Abeta, and if levels of soluble Abeta are more closely related to cognitive deficits than levels of insoluble Abeta, even in aged, transgenic mice, after they have developed very high levels of insoluble Abeta. Aged F-344 rats and young mice over-expressing the Swedish mutation in the human amyloid precursor protein (APPsw; Tg2576+) had elevated levels of soluble Abeta, and were more vulnerable to scopolamine challenge in the Morris water maze (MWM), relative to young rats and Tg2576- mice; but, among individual animals, higher levels of soluble Abeta were not correlated with vulnerability to scopolamine. On the other hand, in aged Tg2576+ mice, cognitive deficits were related to levels of soluble Abeta, not insoluble Abeta, despite the fact that the levels of insoluble Abeta were thousands of times higher than the levels of soluble Abeta. The results of the present experiments suggest that vulnerability to cognitive deficits after scopolamine challenge is not related to elevated levels of soluble Abeta, but that high levels of soluble Abeta are more closely correlated with cognitive deficits than the amount insoluble Abeta, even after large amounts of aggregated, insoluble Abeta have been deposited.

Adverse effects of gabapentin and lack of anti-allodynic efficacy of amitriptyline in the streptozotocin model of painful diabetic neuropathy.

Amitriptyline and gabapentin are the primary treatments for painful diabetic neuropathy (PDN), and it is clear that they produce beneficial effects, but there are questions about these treatments that have not been adequately addressed. For example, although there is a growing consensus that the therapeutic effects of amitriptyline in pain patients are independent of its effects on mood, it is not clear that amitriptyline has specific and direct effects on pain. There is also a fairly broad consensus that gabapentin is safe and well tolerated, but the side-effect profile of gabapentin has not been adequately assessed in pain populations. The rat streptozotocin (STZ) model of PDN was used (a) to assess the effects of amitriptyline on objective, quantitative measures of tactile allodynia, a common type of pain in PDN patients, and (b) to assess the side effects of gabapentin using measures of motor/ambulatory and cognitive function. Amitriptyline did not attenuate STZ-induced mechanical allodynia, even after chronic administration of high doses. Gabapentin produced robust anti-allodynic effects but also produced deficits in tests of motor/ambulatory and cognitive functions. The present experiments suggest that the beneficial effects of amitriptyline in PDN may not be a result of anti-allodynic efficacy and that gabapentin produces robust anti-allodynic effects but may also produce significant motor and cognitive deficits even at or near the lowest effective doses. These findings challenge the consensus opinions about these primary treatments for PDN and suggest that their therapeutic and adverse effects should be explored further in pain patients.

Long-lasting functional disabilities in middle-aged rats with small cerebral infarcts.

Recommendations from experts and recently established guidelines on how to improve the face and predictive validity of animal models of stroke have stressed the importance of using older animals and long-term behavioral-functional endpoints rather than relying almost exclusively on acute measures of infarct volume in young animals. The objective of the present study was to determine whether we could produce occlusions in older rats with an acceptable mortality rate and then detect reliable, long-lasting functional deficits. A reversible intraluminar suture middle cerebral artery occlusion (MCAO) procedure was used to produce small infarcts in middle-aged rats. This resulted in an acceptable mortality rate, and robust disabilities were detected in functional assays, although the degree of total tissue loss measured 90 d after MCAO was quite modest. Infarcted animals were functionally impaired relative to sham control animals even 90 d after the occlusions, and when animals were subgrouped based on amount of tissue loss, MCAO animals with only 4% tissue loss exhibited enduring neurological-behavioral impairments relative to sham-operated controls, and the functional impairments in the group with the largest infarcts (20% tissue loss) were more severe than the functional impairments in the rats with 4% tissue loss. These results suggest that this model, using reversible MCAO to produce small infarcts and long-lasting functional-behavioral deficits in older rats, may represent an advance in the relatively higher-throughput modeling of stroke and its recovery in rodents and may be useful in the development and characterization of future stroke therapies.

Effects of CRF1 receptor antagonists and benzodiazepines in the Morris water maze and delayed non-matching to position tests.

RATIONALE: Benzodiazepines continue to be widely used for the treatment of anxiety, but it is well known that benzodiazepines have undesirable side effects, including sedation, ataxia, cognitive deficits and the risk of addiction and abuse. CRF(1) receptor antagonists are being developed as potential novel anxiolytics, but while CRF(1) receptor antagonists seem to have a better side-effect profile than benzodiazepines with respect to sedation and ataxia, the effects of CRF(1) receptor antagonists on cognitive function have not been well characterized. It is somewhat surprising that the potential cognitive effects of CRF(1) receptor antagonists have not been more fully characterized since there is some evidence to suggest that these compounds may impair cognitive function. OBJECTIVE: The Morris water maze and the delayed non-matching to position test are sensitive tests of a range of cognitive functions, including spatial learning, attention and short-term memory, so the objective of the present experiments was to assess the effects of benzodiazepines and CRF(1) receptor antagonists in these tests. RESULTS: The benzodiazepines chlordiazepoxide and alprazolam disrupted performance in the Morris water maze and delayed non-matching to position at doses close to their therapeutic, anxiolytic doses. In contrast, the CRF(1) receptor antagonists DMP-904 and DMP-696 produced little or no impairment in the Morris water maze or delayed non-matching to position test even at doses 10-fold higher than were necessary to produce anxiolytic effects. CONCLUSIONS: The results of the present experiments suggest that, with respect to their effects on cognitive functions, CRF(1) receptor antagonists seem to have a wider therapeutic index than benzodiazepines.

Examining the Predictive Validity of NIH Peer Review Scores.

The predictive validity of peer review at the National Institutes of Health (NIH) has not yet been demonstrated empirically. It might be assumed that the most efficient and expedient test of the predictive validity of NIH peer review would be an examination of the correlation between percentile scores from peer review and bibliometric indices of the publications produced from funded projects. The present study used a large dataset to examine the rationale for such a study, to determine if it would satisfy the requirements for a test of predictive validity. The results show significant restriction of range in the applications selected for funding. Furthermore, those few applications that are funded with slightly worse peer review scores are not selected at random or representative of other applications in the same range. The funding institutes also negotiate with applicants to address issues identified during peer review. Therefore, the peer review scores assigned to the submitted applications, especially for those few funded applications with slightly worse peer review scores, do not reflect the changed and improved projects that are eventually funded. In addition, citation metrics by themselves are not valid or appropriate measures of scientific impact. The use of bibliometric indices on their own to measure scientific impact would likely increase the inefficiencies and problems with replicability already largely attributed to the current over-emphasis on bibliometric indices. Therefore, retrospective analyses of the correlation between percentile scores from peer review and bibliometric indices of the publications resulting from funded grant applications are not valid tests of the predictive validity of peer review at the NIH.

No detectable analgesic effects in the formalin test even with one million bovine adrenal chromaffin cells.

The present experiments were conducted to identify analgesic agents for transfection into immortalized adrenal chromaffin cell lines to maximize their analgesic potential. Analgesic agents known to be produced by adrenal chromaffin cells were infused intrathecally at a low dose (0.2 microg) which might conceivably be attained by adrenal chromaffin cell transplants. Numerous agents, administered individually and in two-factor combinations, produced significant analgesic effects in the formalin test. Before assessing the potential additive or synergistic effects of these analgesic agents with adrenal chromaffin cells, studies were conducted to demonstrate analgesic effects with adrenal chromaffin cells alone. Analgesic effects were previously reported in the literature with 80-100k intrathecal bovine adrenal chromaffin (BAC) cells; but in the present study 500k purified BAC cells failed to produce detectable analgesic effects. One million purified BAC cells also failed to produce analgesic effects in the formalin test. In a final study, even nicotine-stimulated release from one million purified BAC cells failed to produce analgesic effects in the formalin test. The fact that even one million nicotine-stimulated BAC cells failed to demonstrate therapeutic potential in these blinded experiments under conditions which were clearly sensitive to the analgesic agents produced by BAC cells, raises serious questions about the clinical utility of this experimental treatment.

Numerous adrenal chromaffin cell preparations fail to produce analgesic effects in the formalin test or in tests of acute pain even with nicotine stimulation.

Previous studies have reported that intrathecal implants of a variety of adrenal chromaffin cell preparations all produce analgesic effects in rodents. The major objective of the present study was to determine if any adrenal chromaffin cell preparations produce more robust analgesic effects than other cell preparations. The present study included adult rat adrenal chromaffin tissue allografts, purified adult bovine chromaffin cells, and polymer-encapsulated calf adrenal chromaffin cells, all prepared according to previously published procedures, as well as purified calf adrenal chromaffin cells. Previous studies have also suggested that immunosuppression may play a role in graft survival, and potentially increase the magnitude of analgesic effects, so the present study included both immunosuppressed and non-immunosuppressed groups (cyclosporin A, 10 mg/kg per day). Behavioral tests included the formalin test; and a dorsal tail-flick, hot-plate, and paw-pinch test, conducted sequentially 2 min after systemic nicotine (0.1 mg/kg) to evoke release from the chromaffin cells, as previously reported. Analgesic effects related to morphine and nicotine were detected, and consistent differences in performance could be detected between individual animals. Surprisingly, no analgesic effects were detectable with any of the four chromaffin cell preparations, with or without immunosuppression, in the formalin test or with nicotine stimulation in tests of acute pain.

Analgesic effects of adrenal chromaffin allografts: contingent on special procedures or due to experimenter bias?

Many articles have reported that adrenal chromaffin cell transplants produce analgesic effects. Surprisingly, studies conducted in our laboratory failed to detect analgesic effects of adrenal chromaffin cell transplants. Although we have attempted to replicate the procedures reported to produce analgesic effects with adrenal chromaffin transplants, many of the different cell preparation procedures we have examined are fairly complex, and it is possible that our transplants were not sufficiently viable because of some subtle difference in our cell preparation procedures. In the present study we attempted to replicate as precisely as possible, and with very large groups to maximize statistical power, the simplest and most straightforward procedures previously reported to produce analgesic effects, adrenal allografts in the formalin test. The first experiment, conducted in our laboratories, failed to detect analgesic effects of intrathecal adrenal allografts in the formalin test. Another study conducted at a different research facility confirmed the absence of analgesic effects in the formalin test but verified that analgesic effects of morphine were detectable under the same blinded conditions. In addition, graft viability was verified histologically, but there was no correlation in either experiment between adrenal chromaffin cell number and pain behaviors. These results demonstrate more clearly than any of our previous reports that the analgesic effects of intrathecal adrenal transplants are not reliable and should not be accepted as valid until they can be produced reliably under rigorously blinded conditions.

The pharmacology of DMP696 and DMP904, non-peptidergic CRF1 receptor antagonists.

CRF(1) antagonists DMP696 and DMP904 were designed as drug development candidates for the treatment of anxiety and depression. Both compounds display nanomolar affinity for human CRF(1) receptors, and exhibit >1000-fold selectivity for CRF(1) over CRF(2) receptors and over a broad panel of other proteins. DMP696 and DMP904 block CRF-stimulated adenylyl cyclase activity in cortical homogenates and cell-lines expressing CRF(1) receptors. Both compounds inhibit CRF-stimulated ACTH release from rat pituitary corticotropes. Binding and functional studies indicate that DMP696 and DMP904 behave as noncompetitive full antagonists. DMP696 and DMP904 exhibit anxiolytic-like efficacy in several rat anxiety models. In the defensive withdrawal test, both compounds reduce exit latency with lowest effective doses of 3 and 1 mg/kg, respectively. The anxiolytic-like effect is maintained over 14 days of repeated dosing. In the context of a novel environment used in this test, DMP696 and DMP904 reverse mild stress-induced increases in plasma CORT secretion but at doses 3-4-fold greater than those required for anxiolyticlike efficacy. DMP696 and DMP904 are ineffective in three depression models including the learned helplessness paradigm at doses up to 30 mg/kg. At lowest anxiolytic-like doses, DMP696 and DMP904 occupy >50% CRF(1) receptors in the brain. The in vivo IC(50) values (plasma concentrations required for occupying 50% CRF(1) receptors) estimated based upon free, but not total, plasma concentrations are an excellent correlation with the in vitro IC(50) values. Neither compound produces sedation, ataxia, chlordiazepoxide-like subjective effects or adverse effects on cognition at doses 10-fold higher than anxiolytic-like doses. Neither compound produces physiologically significant changes in cardiovascular, respiratory, gastrointestinal or renal functions at anxiolytic-like doses. DMP696 and DMP904 have favorable pharmacokinetic profiles with good oral bioavailabilities. The overall pharmacological properties suggest that both compounds may be effective anxiolytics with low behavioral side effect liabilities.

An assessment of the effects of serotonin 6 (5-HT6) receptor antagonists in rodent models of learning.

Antagonists of serotonin 6 (5-HT6) receptors have been reported to enhance cognition in animal models of learning, although this finding has not been universal. We have assessed the therapeutic potential of the specific 5-HT6 receptor antagonists 4-amino-N-(2,6-bis-methylamino-pyrimidin-4-yl)-benzenesulfonamide (Ro 04-6790) and 5-chloro-N-(4-methoxy-3-piperazin-1-yl-phenyl)-3-methyl-2-benzothiophenesulfonamide (SB-271046) in rodent models of cognitive function. Although mice express the 5-HT6 receptor and the function of this receptor has been investigated in mice, all reports of activity with 5-HT6 receptor antagonists have used rat models. In the present study, receptor binding revealed that the pharmacological properties of the mouse receptor are different from the rat and human receptor: Ro 04-6790 does not bind to the mouse 5-HT6 receptor, so all in vivo testing included in the present report was conducted in rats. We replicated previous reports that 5-HT6 receptor antagonists produce a stretching syndrome previously shown to be mediated through cholinergic mechanisms, but Ro 04-6790 and SB-271046 failed to attenuate scopolamine-induced deficits in a test of contextual fear conditioning. We also failed to replicate the significant effects reported previously in both an autoshaping task and in a version of the Morris water maze. The results of our experiments are not consistent with previous reports that suggested that 5-HT6 antagonists might have therapeutic potential for cognitive disorders.