Gender, Race, and Grant Reviews: Translating and Responding to Research Feedback.

Grant-writing and grant-getting are key to success in many academic disciplines, but research points to gender gaps in both, especially as careers progress. Using a sample of National Institutes of Health (NIH) K-Awardees-Principal Investigators of Mentored Career Development Awards-we examined gender and race effects in response to imagined negative grant reviews that emphasized either promise or inadequacy. Women translated both forms of feedback into worse NIH priority scores than did men and showed reduced motivation to reapply for funding following the review highlighting inadequacy. Translation of feedback mediated the effects of gender on motivation, changing one's research focus, and advice-seeking. Race effects were less consistent, and race did not moderate effects of gender. We suggest that gender bias in grant reviews (i.e., greater likelihood of highlighting inadequacy in reviews of women's grants), along with gender differences in responsiveness to feedback, may contribute to women's underrepresentation in academic medicine.

Low agreement among reviewers evaluating the same NIH grant applications.

Obtaining grant funding from the National Institutes of Health (NIH) is increasingly competitive, as funding success rates have declined over the past decade. To allocate relatively scarce funds, scientific peer reviewers must differentiate the very best applications from comparatively weaker ones. Despite the importance of this determination, little research has explored how reviewers assign ratings to the applications they review and whether there is consistency in the reviewers' evaluation of the same application. Replicating all aspects of the NIH peer-review process, we examined 43 individual reviewers' ratings and written critiques of the same group of 25 NIH grant applications. Results showed no agreement among reviewers regarding the quality of the applications in either their qualitative or quantitative evaluations. Although all reviewers received the same instructions on how to rate applications and format their written critiques, we also found no agreement in how reviewers "translated" a given number of strengths and weaknesses into a numeric rating. It appeared that the outcome of the grant review depended more on the reviewer to whom the grant was assigned than the research proposed in the grant. This research replicates the NIH peer-review process to examine in detail the qualitative and quantitative judgments of different reviewers examining the same application, and our results have broad relevance for scientific grant peer review.

A Gender Bias Habit-Breaking Intervention Led to Increased Hiring of Female Faculty in STEMM Departments.

Addressing the underrepresentation of women in science is a top priority for many institutions, but the majority of efforts to increase representation of women are neither evidence-based nor rigorously assessed. One exception is the gender bias habit-breaking intervention (Carnes et al., 2015), which, in a cluster-randomized trial involving all but two departmental clusters (N = 92) in the 6 STEMM focused schools/colleges at the University of Wisconsin - Madison, led to increases in gender bias awareness and self-efficacy to promote gender equity in academic science departments. Following this initial success, the present study compares, in a preregistered analysis, hiring rates of new female faculty pre- and post-manipulation. Whereas the proportion of women hired by control departments remained stable over time, the proportion of women hired by intervention departments increased by an estimated 18 percentage points (OR = 2.23, dOR = 0.34). Though the preregistered analysis did not achieve conventional levels of statistical significance (p < 0.07), our study has a hard upper limit on statistical power, as the cluster-randomized trial has a maximum sample size of 92 departmental clusters. These patterns have undeniable practical significance for the advancement of women in science, and provide promising evidence that psychological interventions can facilitate gender equity and diversity.

'Your comments are meaner than your score': score calibration talk influences intra- and inter-panel variability during scientific grant peer review.

In scientific grant peer review, groups of expert scientists meet to engage in the collaborative decision-making task of evaluating and scoring grant applications. Prior research on grant peer review has established that inter-reviewer reliability is typically poor. In the current study, experienced reviewers for the National Institutes of Health (NIH) were recruited to participate in one of four constructed peer review panel meetings. Each panel discussed and scored the same pool of recently reviewed NIH grant applications. We examined the degree of intra-panel variability in panels' scores of the applications before versus after collaborative discussion, and the degree of inter-panel variability. We also analyzed videotapes of reviewers' interactions for instances of one particular form of discourse-Score Calibration Talk-as one factor influencing the variability we observe. Results suggest that although reviewers within a single panel agree more following collaborative discussion, different panels agree less after discussion, and Score Calibration Talk plays a pivotal role in scoring variability during peer review. We discuss implications of this variability for the scientific peer review process.

Fair Play: A Study of Scientific Workforce Trainers' Experience Playing an Educational Video Game about Racial Bias.

Explicit racial bias has decreased in the United States, but racial stereotypes still exist and conspire in multiple ways to perpetuate the underparticipation of Blacks in science careers. Capitalizing on the potential effectiveness of role-playing video games to promote the type of active learning required to increase awareness of and reduce subtle racial bias, we developed the video game Fair Play, in which players take on the role of Jamal, a Black male graduate student in science, who experiences discrimination in his PhD program. We describe a mixed-methods evaluation of the experience of scientific workforce trainers who played Fair Play at the National Institutes of Health Division of Training Workforce Development and Diversity program directors' meeting in 2013 (n = 47; 76% female, n = 34; 53% nonwhite, n = 26). The evaluation findings suggest that Fair Play can promote perspective taking and increase bias literacy, which are steps toward reducing racial bias and affording Blacks equal opportunities to excel in science.

Write More Articles, Get More Grants: The Impact of Department Climate on Faculty Research Productivity.

BACKGROUND: Many studies find that female faculty in academic medicine, science, and engineering experience adverse workplace climates. This study longitudinally investigates whether department climate is associated with future research productivity and whether the associations are stronger for female than male faculty. METHOD: Two waves of a faculty climate survey, institutional grant records, and publication records were collected for 789 faculties in academic medicine, science, and engineering at the University of Wisconsin-Madison between 2000 and 2010. Research productivity was measured as Number of Publications and Number of Grants awarded, and department climate was measured with scales for professional interactions, department decision-making practices, climate for underrepresented groups, and work/life balance. Ordinary least squares and negative binomial regression methods were used to assess gender differences in productivity, influences of department climate on productivity, and gender differences in effects of climate on productivity. RESULTS: Female faculty published fewer articles and were awarded fewer grants in the baseline period, but their productivity did not differ from male faculty on these measures in subsequent years. Number of Publications was positively affected by professional interactions, but negatively affected by positive work/life balance. Number of Grants awarded was positively affected by climate for underrepresented groups. These main effects did not differ by gender; however, some three-way interactions illuminated how different aspects of department climate affected productivity differently for men and women in specific situations. CONCLUSIONS: In perhaps the first study to assess the longitudinal impact of department climate on faculty research productivity, positive department climate is associated with significantly greater productivity for all faculty-women and men. However, some positive aspects of climate (specifically, work/life balance) may be associated with lower productivity for some female faculty at specific career periods. These findings suggest that departments that wish to increase grants and publications would be wise to foster a positive workplace climate.

Are Female Applicants Disadvantaged in National Institutes of Health Peer Review? Combining Algorithmic Text Mining and Qualitative Methods to Detect Evaluative Differences in R01 Reviewers' Critiques.

BACKGROUND: Women are less successful than men in renewing R01 grants from the National Institutes of Health. Continuing to probe text mining as a tool to identify gender bias in peer review, we used algorithmic text mining and qualitative analysis to examine a sample of critiques from men's and women's R01 renewal applications previously analyzed by counting and comparing word categories. METHODS: We analyzed 241 critiques from 79 Summary Statements for 51 R01 renewals awarded to 45 investigators (64% male, 89% white, 80% PhD) at the University of Wisconsin-Madison between 2010 and 2014. We used latent Dirichlet allocation to discover evaluative "topics" (i.e., words that co-occur with high probability). We then qualitatively examined the context in which evaluative words occurred for male and female investigators. We also examined sex differences in assigned scores controlling for investigator productivity. RESULTS: Text analysis results showed that male investigators were described as "leaders" and "pioneers" in their "fields," with "highly innovative" and "highly significant research." By comparison, female investigators were characterized as having "expertise" and working in "excellent" environments. Applications from men received significantly better priority, approach, and significance scores, which could not be accounted for by differences in productivity. CONCLUSIONS: Results confirm our previous analyses suggesting that gender stereotypes operate in R01 grant peer review. Reviewers may more easily view male than female investigators as scientific leaders with significant and innovative research, and score their applications more competitively. Such implicit bias may contribute to sex differences in award rates for R01 renewals.

Analysis of National Institutes of Health R01 Application Critiques, Impact, and Criteria Scores: Does the Sex of the Principal Investigator Make a Difference?

PURPOSE: Prior text analysis of R01 critiques suggested that female applicants may be disadvantaged in National Institutes of Health (NIH) peer review, particularly for renewals. NIH altered its review format in 2009. The authors examined R01 critiques and scoring in the new format for differences due to principal investigator (PI) sex. METHOD: The authors analyzed 739 critiques-268 from 88 unfunded and 471 from 153 funded applications for grants awarded to 125 PIs (76 males, 49 females) at the University of Wisconsin-Madison between 2010 and 2014. The authors used seven word categories for text analysis: ability, achievement, agentic, negative evaluation, positive evaluation, research, and standout adjectives. The authors used regression models to compare priority and criteria scores, and results from text analysis for differences due to PI sex and whether the application was for a new (Type 1) or renewal (Type 2) R01. RESULTS: Approach scores predicted priority scores for all PIs' applications (P < .001), but scores and critiques differed significantly for male and female PIs' Type 2 applications. Reviewers assigned significantly worse priority, approach, and significance scores to female than male PIs' Type 2 applications, despite using standout adjectives (e.g., "outstanding," "excellent") and making references to ability in more critiques (P < .05 for all comparisons). CONCLUSIONS: The authors' analyses suggest that subtle gender bias may continue to operate in the post-2009 NIH review format in ways that could lead reviewers to implicitly hold male and female applicants to different standards of evaluation, particularly for R01 renewals.

Patterns of Feedback on the Bridge to Independence: A Qualitative Thematic Analysis of NIH Mentored Career Development Award Application Critiques.

BACKGROUND: NIH Mentored Career Development (K) Awards bridge investigators from mentored to independent research. A smaller proportion of women than men succeed in this transition. The aim of this qualitative study was to analyze reviewers' narrative critiques of K award applications and explore thematic content of feedback provided to male and female applicants. METHOD: We collected 88 critiques, 34 from 9 unfunded and 54 from 18 funded applications, from 70% (n = 26) of investigators at the University of Wisconsin-Madison with K awards funded between 2005 and 2009 on the first submission or after revision. We qualitatively analyzed text in the 5 critique sections: candidate, career development plan, research plan, mentors, and environment and institutional commitment. We explored thematic content within these sections for male and female applicants and for applicants who had received a subsequent independent research award by 2014. RESULTS: Themes revealed consistent areas of criticism for unfunded applications and praise for funded applications. Subtle variations in thematic content appeared for male and female applicants: For male applicants criticism was often followed by advice but for female applicants it was followed by questions about ability; praise recurrently characterized male but not female applicants' research as highly significant with optimism for future independence. Female K awardees that obtained subsequent independent awards stood out as having track records described as "outstanding." CONCLUSION: This exploratory study suggests that K award reviewer feedback, particularly for female applicants, should be investigated as a potential contributor to research persistence and success in crossing the bridge to independence.

Post-Code PTSD Symptoms in Internal Medicine Residents Who Participate in Cardiopulmonary Resuscitation Events: A Mixed Methods Study.

BACKGROUND: Internal medicine (IM) residents participate in cardiopulmonary resuscitation events (or "codes"), stressful events that involve the death or near death of patients. Experiencing traumatic stress can lead to posttraumatic stress disorder (PTSD). OBJECTIVE: We examined whether IM residents who participate in codes developed PTSD symptoms. METHODS: We conducted this research as a 2-phase, mixed methods study. In the first phase, we analyzed interview transcripts from 25 IM residents at 9 institutions for content relevant to PTSD. In the second phase, we surveyed 82 IM residents at 1 midwestern institution for symptoms reported post-code that are associated with PTSD (post-code PTSD symptoms). RESULTS: A total of 7 of 25 residents (28%) interviewed characterized codes as traumatic or described experiencing PTSD symptoms, including intrusive thoughts about the code, avoidance and emotional numbing, and hyperarousal, irritability, or hypervigilance. Of the 82 residents, 51 (62%) responded to the questionnaire. Of the 42 respondents who had participated in a code, 6 (14%) reported at least 1 PTSD symptom, and 3 (7%) women screened positive for potential PTSD with 2 or more PTSD symptoms. Endorsing any PTSD symptom was associated with residents' perception that code participation negatively affected their work performance. CONCLUSIONS: Our study found that some IM residents reported symptoms of PTSD after being involved in a code.

Why is John More Likely to Become Department Chair Than Jennifer?

This article reviews some of our research on how gender stereotypes and their accompanying assumptions and expectations can influence the careers of male and female physicians and scientists in a myriad of subtle ways. Although stereotype-based cognitive biases may be invisible and unintentional, they nevertheless shape the experiences of women in academic medicine in ways that frequently constrain their opportunities. We present research on the following: 1) subtle differences in the evaluation of male and female medical students as revealed through text analysis of written evaluations at a critical career juncture, 2) how cultural assumptions about the way men and women should and should not behave influence medical residents' experiences as leaders, and 3) how approaching gender bias among faculty in academic medicine, science, and engineering as a remedial habit can be successful in changing individual behaviors and in improving department climate.

A quantitative linguistic analysis of National Institutes of Health R01 application critiques from investigators at one institution.

PURPOSE: Career advancement in academic medicine often hinges on the ability to garner research funds. The National Institutes of Health's (NIH's) R01 award is the "gold standard" of an independent research program. Studies show inconsistencies in R01 reviewers' scoring and in award outcomes for certain applicant groups. Consistent with the NIH recommendation to examine potential bias in R01 peer review, the authors performed a text analysis of R01 reviewers' critiques. METHOD: The authors collected 454 critiques (262 from 91 unfunded and 192 from 67 funded applications) from 67 of 76 (88%) R01 investigators at the University of Wisconsin-Madison with initially unfunded applications subsequently funded between December 2007 and May 2009. To analyze critiques, the authors developed positive and negative grant application evaluation word categories and selected five existing categories relevant to grant review. They analyzed results with linear mixed-effects models for differences due to applicant and application characteristics. RESULTS: Critiques of funded applications contained more positive descriptors and superlatives and fewer negative evaluation words than critiques of unfunded applications. Experienced investigators' critiques contained more references to competence. Critiques showed differences due to applicant sex despite similar application scores or funding outcomes: more praise for applications from female investigators, greater reference to competence/ability for funded applications from female experienced investigators, and more negative evaluation words for applications from male investigators (all P<.05). CONCLUSIONS: Results suggest that text analysis is a promising tool for assessing consistency in R01 reviewers' judgments, and gender stereotypes may operate in R01 review.

The effect of an intervention to break the gender bias habit for faculty at one institution: a cluster randomized, controlled trial.

PURPOSE: Despite sincere commitment to egalitarian, meritocratic principles, subtle gender bias persists, constraining women's opportunities for academic advancement. The authors implemented a pair-matched, single-blind, cluster randomized, controlled study of a gender-bias-habit-changing intervention at a large public university. METHOD: Participants were faculty in 92 departments or divisions at the University of Wisconsin-Madison. Between September 2010 and March 2012, experimental departments were offered a gender-bias-habit-changing intervention as a 2.5-hour workshop. Surveys measured gender bias awareness; motivation, self-efficacy, and outcome expectations to reduce bias; and gender equity action. A timed word categorization task measured implicit gender/leadership bias. Faculty completed a work-life survey before and after all experimental departments received the intervention. Control departments were offered workshops after data were collected. RESULTS: Linear mixed-effects models showed significantly greater changes post intervention for faculty in experimental versus control departments on several outcome measures, including self-efficacy to engage in gender-equity-promoting behaviors (P = .013). When ≥ 25% of a department's faculty attended the workshop (26 of 46 departments), significant increases in self-reported action to promote gender equity occurred at three months (P = .007). Post intervention, faculty in experimental departments expressed greater perceptions of fit (P = .024), valuing of their research (P = .019), and comfort in raising personal and professional conflicts (P = .025). CONCLUSIONS: An intervention that facilitates intentional behavioral change can help faculty break the gender bias habit and change department climate in ways that should support the career advancement of women in academic medicine, science, and engineering.

Stuck in the out-group: Jennifer can't grow up, Jane's invisible, and Janet's over the hill.

Fifty years after Title IX, women remain sparsely represented in high ranks and leadership in academic medicine. Although men and women enter the career pipeline at similar rates, academic medicine does not equivalently advance them. Currently, women account for 32% of associate professors, 20% of full professors, 14% of department chairs, and 11% of deans at U.S. medical schools--far from the near sex parity seen in medical students since the 1990s. Over 30 years of research confirms that gender stereotypes can operate to disadvantage women in review processes and consequently bar their advancement in domains like science and medicine. The authors present three vignettes to illustrate how gender stereotypes can also operate to disadvantage women in social interactions by positioning them in the "out-group" for many career-advancing opportunities. The authors argue that policies alone will not achieve gender equity in the academic medicine workforce. Addressing stereotype-based gender bias is critical for the future of academic medicine. Interventions that treat gender bias as a remediable habit show promise in promoting gender equity and transforming institutional culture to achieve the full participation of women at all career stages. A critical step is to recognize when gender stereotyped assumptions are influencing judgments and decision making in ourselves and others, challenge them as unjust, and deliberately practice replacing them with accurate and objective data.

A qualitative study of work-life choices in academic internal medicine.

The high attrition rate of female physicians pursuing an academic medicine research career has not been examined in the context of career development theory. We explored how internal medicine residents and faculty experience their work within the context of their broader life domain in order to identify strategies for facilitating career advancement. Semi-structured interviews were conducted with a purposeful sample of 18 residents and 34 faculty members representing male and female physicians at different career stages. Using thematic analysis, three themes emerged: (1) the love of being a physician ("Raison d'être"), (2) family obligations ("2nd Shift"), and (3) balancing work demands with non-work life ("Negotiating Academic Medicine"). Female researchers and educators reported more strategies for multiple role planning and management than female practitioners. Interventions aimed at enhancing academic internists' planning and self-efficacy for multiple role management should be investigated as a potential means for increasing participation and facilitate advancement.

"Fair Play": A Videogame Designed to Address Implicit Race Bias Through Active Perspective Taking.

OBJECTIVE: Having diverse faculty in academic health centers will help diversify the healthcare workforce and reduce health disparities. Implicit race bias is one factor that contributes to the underrepresentation of Black faculty. We designed the videogame "Fair Play" in which players assume the role of a Black graduate student named Jamal Davis. As Jamal, players experience subtle race bias while completing "quests" to obtain a science degree. We hypothesized that participants randomly assigned to play the game would have greater empathy for Jamal and lower implicit race bias than participants randomized to read narrative text describing Jamal's experience. MATERIALS AND METHODS: University of Wisconsin-Madison graduate students were recruited via e-mail and randomly assigned to play "Fair Play" or read narrative text through an online link. Upon completion, participants took an Implicit Association Test to measure implicit bias and answered survey questions assessing empathy toward Jamal and awareness of bias. RESULTS: As hypothesized, gameplayers showed the least implicit bias but only when they also showed high empathy for Jamal (P=0.013). Gameplayers did not show greater empathy than text readers, and women in the text condition reported the greatest empathy for Jamal (P=0.008). However, high empathy only predicted lower levels of implicit bias among those who actively took Jamal's perspective through gameplay (P=0.014). CONCLUSIONS: A videogame in which players experience subtle race bias as a Black graduate student has the potential to reduce implicit bias, possibly because of a game's ability to foster empathy through active perspective taking.

Physicians and implicit bias: how doctors may unwittingly perpetuate health care disparities.

Although the medical profession strives for equal treatment of all patients, disparities in health care are prevalent. Cultural stereotypes may not be consciously endorsed, but their mere existence influences how information about an individual is processed and leads to unintended biases in decision-making, so called "implicit bias". All of society is susceptible to these biases, including physicians. Research suggests that implicit bias may contribute to health care disparities by shaping physician behavior and producing differences in medical treatment along the lines of race, ethnicity, gender or other characteristics. We review the origins of implicit bias, cite research documenting the existence of implicit bias among physicians, and describe studies that demonstrate implicit bias in clinical decision-making. We then present the bias-reducing strategies of consciously taking patients' perspectives and intentionally focusing on individual patients' information apart from their social group. We conclude that the contribution of implicit bias to health care disparities could decrease if all physicians acknowledged their susceptibility to it, and deliberately practiced perspective-taking and individuation when providing patient care. We further conclude that increasing the number of African American/Black physicians could reduce the impact of implicit bias on health care disparities because they exhibit significantly less implicit race bias.

Are men more likely than women to commit scientific misconduct? Maybe, maybe not.

In their study published in January 2013 in mBio, Fang et al. reviewed records from the Office of Research Integrity (ORI) and found more cases of scientific misconduct committed by men than women, particularly by faculty (F. C. Fang, J. W. Bennett, and A. Casadevall, mBio 4:1-3, 2013). Powerful social norms shape the way men and women behave, and implicit gender schemas can lead to different evaluation standards for men and women for tasks stereotypically linked to one gender. It is possible that norms for acceptable male and female behavior could lead to a lower threshold for men than women to engage in the risky behavior of scientific misconduct. It is also possible that women and men commit scientific fraud at the same rate but that, because crime is a male-gendered domain, evaluators require more proof of the criminal "competence" of women for an investigation to rise to the level of an ORI case or that female gender norms for likeability and a lower apology threshold more often prevent escalation of women's fraud beyond a local level. Male scientists also have more opportunity to commit fraud than female scientists because they receive more NIH research funding--a finding that may also be influenced by gender schemas. We cannot conclude from the ORI data that men are more likely than women to risk the consequences of committing scientific misconduct simply because risk taking aligns with male gender stereotypes. Neither can we conclude that because men are more likely than women to commit fraud in other contexts, men are also more likely than women to commit scientific fraud. We can conclude, however, that scientific misconduct, regardless of who commits it, diminishes all who contribute to the scientific enterprise.