Peer review: Risk and risk tolerance.

Peer review, commonly used in grant funding decisions, relies on scientists' ability to evaluate research proposals' quality. Such judgments are sometimes beyond reviewers' discriminatory power and could lead to a reliance on subjective biases, including preferences for lower risk, incremental projects. However, peer reviewers' risk tolerance has not been well studied. We conducted a cross-sectional experiment of peer reviewers' evaluations of mock primary reviewers' comments in which the level and sources of risks and weaknesses were manipulated. Here we show that proposal risks more strongly predicted reviewers' scores than proposal strengths based on mock proposal evaluations. Risk tolerance was not predictive of scores but reviewer scoring leniency was predictive of overall and criteria scores. The evaluation of risks dominates reviewers' evaluation of research proposals and is a source of inter-reviewer variability. These results suggest that reviewer scoring variability may be attributed to the interpretation of proposal risks, and could benefit from intervention to improve the reliability of reviews. Additionally, the valuation of risk drives proposal evaluations and may reduce the chances that risky, but highly impactful science, is supported.

Grant Review Feedback: Appropriateness and Usefulness.

The primary goal of the peer review of research grant proposals is to evaluate their quality for the funding agency. An important secondary goal is to provide constructive feedback to applicants for their resubmissions. However, little is known about whether review feedback achieves this goal. In this paper, we present a multi-methods analysis of responses from grant applicants regarding their perceptions of the effectiveness and appropriateness of peer review feedback they received from grant submissions. Overall, 56-60% of applicants determined the feedback to be appropriate (fair, well-written, and well-informed), although their judgments were more favorable if their recent application was funded. Importantly, independent of funding success, women found the feedback better written than men, and more white applicants found the feedback to be fair than non-white applicants. Also, perceptions of a variety of biases were specifically reported in respondents' feedback. Less than 40% of applicants found the feedback to be very useful in informing their research and improving grantsmanship and future submissions. Further, negative perceptions of the appropriateness of review feedback were positively correlated with more negative perceptions of feedback usefulness. Importantly, respondents suggested that highly competitive funding pay-lines and poor inter-panel reliability limited the usefulness of review feedback. Overall, these results suggest that more effort is needed to ensure that appropriate and useful feedback is provided to all applicants, bolstering the equity of the review process and likely improving the quality of resubmitted proposals.

The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey.

Scientific peer reviewers play an integral role in the grant selection process, yet very little has been reported on the levels of participation or the motivations of scientists to take part in peer review. The American Institute of Biological Sciences (AIBS) developed a comprehensive peer review survey that examined the motivations and levels of participation of grant reviewers. The survey was disseminated to 13,091 scientists in AIBS's proprietary database. Of the 874 respondents, 76% indicated they had reviewed grant applications in the last 3 years; however, the number of reviews was unevenly distributed across this sample. Higher review loads were associated with respondents who had submitted more grant proposals over this time period, some of whom were likely to be study section members for large funding agencies. The most prevalent reason to participate in a review was to give back to the scientific community (especially among frequent grant submitters) and the most common reason to decline an invitation to review was lack of time. Interestingly, few suggested that expectation from the funding agency was a motivation to review. Most felt that review participation positively influenced their careers through improving grantsmanship and exposure to new scientific ideas. Of those who reviewed, respondents reported dedicating 2-5% of their total annual work time to grant review and, based on their self-reported maximum review loads, it is estimated they are participating at 56-87% of their capacity, which may have important implications regarding the sustainability of the system. Overall, it is clear that participation in peer review is uneven and in some cases near capacity, and more needs to be done to create new motivations and incentives to increase the future pool of reviewers.

Moderators of Outcome in Problem-Solving Therapy for Depression in Primary Care.

OBJECTIVE: The purpose of this study was to conduct a secondary analysis of data from trials of problem-solving therapy in primary care (PST-PC) to examine differential outcomes by gender or race-ethnicity. METHODS: The participants were 352 patients with depression treated with PST-PC in multiple primary care sites in the United States. RESULTS: Women's depressive symptoms improved more over time than men's. Hopkins Symptom Checklist depression scale scores decreased significantly (t=-10.12, df=692, p<0.001) (estimate=-0.02, 95% confidence interval=-0.03, -0.02). Although patients from racial-ethnic minority groups had more depressive symptoms over time than patients from nonminority groups, there was no evidence of differential change by racial-ethnic group. CONCLUSIONS: These data add to the literature supporting the usefulness of PST-PC without evidence of differential effects for patients from racial-ethnic minority or nonminority groups.

The content of behavior therapy for depression demonstrates few associations with treatment outcome among low-income, medically ill older adults.

We reported previously the results of a randomized controlled trial of a home-based behavioral treatment for dysthymia or minor depression that emphasized problem solving and activity scheduling among low-income, medically ill older adults. This report focuses on the content of treatment sessions as predictors of depressive symptoms, social activity, and physical activity outcomes among 64 participants who completed 2 or more sessions and evaluations at 6 and 12 months after the baseline evaluation. Worksheets from the treatment sessions were coded for focus on 4 types of problems (functional, social, health/physical, emotional); the number of activities planned was counted. More activity scheduling was associated with increased physical activity at the 12-month evaluation relative to baseline. The limited findings suggest either that the study methodology did not reveal extant associations between treatment variables and outcomes or that the session content variables tested in this study are not the active ingredients of treatment.

Randomized trial of behavioral activation, cognitive therapy, and antidepressant medication in the prevention of relapse and recurrence in major depression.

This study followed treatment responders from a randomized controlled trial of adults with major depression. Patients treated with medication but withdrawn onto pill-placebo had more relapse through 1 year of follow-up compared to patients who received prior behavioral activation, prior cognitive therapy, or continued medication. Prior psychotherapy was also superior to medication withdrawal in the prevention of recurrence across the 2nd year of follow-up. Specific comparisons indicated that patients previously exposed to cognitive therapy were significantly less likely to relapse following treatment termination than patients withdrawn from medication, and patients previously exposed to behavioral activation did almost as well relative to patients withdrawn from medication, although the difference was not significantly different. Differences between behavioral activation and cognitive therapy were small in magnitude and not significantly different across the full 2-year follow-up, and each therapy was at least as efficacious as the continuation of medication. These findings suggest that behavioral activation may be nearly as enduring as cognitive therapy and that both psychotherapies are less expensive and longer lasting alternatives to medication in the treatment of depression.

Problem-solving treatment for depression among Mexican Americans in primary care.

The need to identify and improve access to evidence-based treatments for depressive disorders and other mental health problems is a public health priority, particularly in relation to racial/ethnic minorities and other underserved groups. One hundred forty-six Mexican American primary care patients who met diagnostic criteria for major depressive disorder, dysthymic disorder, or depressive disorder not otherwise specified (NOS) were offered eight sessions of an evidence-based behavioral treatment, problem-solving treatment for primary care (PST-PC). Among participants who agreed to treatment (N=117), depressive symptom scores decreased over time; a minority of participants completed four or more sessions of PST-PC (N=55), and those participants had greater decreases in depressive symptom severity than participants who completed three or fewer PST-PC sessions (N=62) (Hopkins Symptom Checklist-20 scores declined on average 0.86+/-0.97 and 0.40+/-0.66 points for these groups, respectively, p<.05). More work is needed to enhance the engagement in treatments for depression, especially among Latinos in primary care.

Detection of depression among low-income Mexican Americans in primary care.

Between one and two thirds of depressive disorders go undetected in primary care settings. Four hundred ninety-six Mexican American primary care patients from high-poverty areas were screened for depressive symptoms, and 41% endorsed depressive symptoms. Eighty percent of screened patients with depressive symptoms agreed to structured diagnostic interviews and 90% of those interviewed met diagnostic criteria for one or more depressive disorders. Cases of depression detected through this systematic process were compared with evidence of depression detected by providers in medical charts. Provider and study evaluation agreement was poor (kappa= 0.13); providers noted depression in 21% of patients with depressive disorders based on the systematic evaluation. More work is needed to enhance detection of depression in primary care, especially in minority populations.