Low agreement among reviewers evaluating the same NIH grant applications.

Obtaining grant funding from the National Institutes of Health (NIH) is increasingly competitive, as funding success rates have declined over the past decade. To allocate relatively scarce funds, scientific peer reviewers must differentiate the very best applications from comparatively weaker ones. Despite the importance of this determination, little research has explored how reviewers assign ratings to the applications they review and whether there is consistency in the reviewers' evaluation of the same application. Replicating all aspects of the NIH peer-review process, we examined 43 individual reviewers' ratings and written critiques of the same group of 25 NIH grant applications. Results showed no agreement among reviewers regarding the quality of the applications in either their qualitative or quantitative evaluations. Although all reviewers received the same instructions on how to rate applications and format their written critiques, we also found no agreement in how reviewers "translated" a given number of strengths and weaknesses into a numeric rating. It appeared that the outcome of the grant review depended more on the reviewer to whom the grant was assigned than the research proposed in the grant. This research replicates the NIH peer-review process to examine in detail the qualitative and quantitative judgments of different reviewers examining the same application, and our results have broad relevance for scientific grant peer review.

Laughter and the Management of Divergent Positions in Peer Review Interactions.

In this paper we focus on how participants in peer review interactions use laughter as a resource as they publicly report divergence of evaluative positions, divergence that is typical in the give and take of joint grant evaluation. Using the framework of conversation analysis, we examine the infusion of laughter and multimodal laugh-relevant practices into sequences of talk in meetings of grant reviewers deliberating on the evaluation and scoring of high-level scientific grant applications. We focus on a recurrent sequence in these meetings, what we call the score-reporting sequence, in which the assigned reviewers first announce the preliminary scores they have assigned to the grant. We demonstrate that such sequences are routine sites for the use of laugh practices to navigate the initial moments in which divergence of opinion is made explicit. In the context of meetings convened for the purposes of peer review, laughter thus serves as a valuable resource for managing the socially delicate but institutionally required reporting of divergence and disagreement that is endemic to meetings where these types of evaluative tasks are a focal activity.

'Your comments are meaner than your score': score calibration talk influences intra- and inter-panel variability during scientific grant peer review.

In scientific grant peer review, groups of expert scientists meet to engage in the collaborative decision-making task of evaluating and scoring grant applications. Prior research on grant peer review has established that inter-reviewer reliability is typically poor. In the current study, experienced reviewers for the National Institutes of Health (NIH) were recruited to participate in one of four constructed peer review panel meetings. Each panel discussed and scored the same pool of recently reviewed NIH grant applications. We examined the degree of intra-panel variability in panels' scores of the applications before versus after collaborative discussion, and the degree of inter-panel variability. We also analyzed videotapes of reviewers' interactions for instances of one particular form of discourse-Score Calibration Talk-as one factor influencing the variability we observe. Results suggest that although reviewers within a single panel agree more following collaborative discussion, different panels agree less after discussion, and Score Calibration Talk plays a pivotal role in scoring variability during peer review. We discuss implications of this variability for the scientific peer review process.

The effect of an intervention to break the gender bias habit for faculty at one institution: a cluster randomized, controlled trial.

PURPOSE: Despite sincere commitment to egalitarian, meritocratic principles, subtle gender bias persists, constraining women's opportunities for academic advancement. The authors implemented a pair-matched, single-blind, cluster randomized, controlled study of a gender-bias-habit-changing intervention at a large public university. METHOD: Participants were faculty in 92 departments or divisions at the University of Wisconsin-Madison. Between September 2010 and March 2012, experimental departments were offered a gender-bias-habit-changing intervention as a 2.5-hour workshop. Surveys measured gender bias awareness; motivation, self-efficacy, and outcome expectations to reduce bias; and gender equity action. A timed word categorization task measured implicit gender/leadership bias. Faculty completed a work-life survey before and after all experimental departments received the intervention. Control departments were offered workshops after data were collected. RESULTS: Linear mixed-effects models showed significantly greater changes post intervention for faculty in experimental versus control departments on several outcome measures, including self-efficacy to engage in gender-equity-promoting behaviors (P = .013). When ≥ 25% of a department's faculty attended the workshop (26 of 46 departments), significant increases in self-reported action to promote gender equity occurred at three months (P = .007). Post intervention, faculty in experimental departments expressed greater perceptions of fit (P = .024), valuing of their research (P = .019), and comfort in raising personal and professional conflicts (P = .025). CONCLUSIONS: An intervention that facilitates intentional behavioral change can help faculty break the gender bias habit and change department climate in ways that should support the career advancement of women in academic medicine, science, and engineering.

Securing recipiency in workplace meetings: Multimodal practices.

As multiparty interactions with single courses of coordinated action, workplace meetings place particular interactional demands on participants who are not primary speakers (e.g. not chairs) as they work to initiate turns and to interactively coordinate with displays of recipiency from co-participants. Drawing from a corpus of 26 hours of videotaped workplace meetings in a midsized US city, this article reports on multimodal practices - phonetic, prosodic, and bodily-visual - used for coordinating turn transition and for consolidating recipiency in these specialized speech exchange systems. Practices used by self-selecting non-primary speakers as they secure turns in meetings include displays of close monitoring of current speakers' emerging turn structure, displays of heightened interest as current turns approach possible completion, and turn initiation practices designed to pursue and, in a fine-tuned manner, coordinate with displays of recipiency on the parts of other participants as well as from reflexively constructed 'target' recipients. By attending to bodily-visual action, as well as phonetics and prosody, this study contributes to expanding accounts for turn taking beyond traditional word-based grammar (i.e. lexicon and syntax).

Bodily-visual practices and turn continuation.

This paper considers points in turn construction where conversation researchers have shown that talk routinely continues beyond possible turn completion, but where we find bodily-visual behavior doing such turn extension work. The bodily-visual behaviors we examine share many features with verbal turn extensions, but we argue that embodied movements have distinct properties that make them well-suited for specific kinds of social action, including stance display and by-play in sequences framed as subsidiary to a simultaneous and related verbal exchange. Our study is in line with a research agenda taking seriously the point made by Goodwin (2000a, b, 2003), Hayashi (2003, 2005), Iwasaki (2009), and others that scholars seeking to account for practices in language and social interaction do themselves a disservice if they privilege the verbal dimension; rather, as suggested in Stivers/Sidnell (2005), each semiotic system/modality, while coordinated with others, has its own organization. With the current exploration of bodily-visual turn extensions, we hope to contribute to a growing understanding of how these different modes of organization are managed concurrently and in concert by interactants in carrying out their everyday social actions.