Can Generative AI improve social science?

Generative AI that can produce realistic text, images, and other human-like outputs is currently transforming many different industries. Yet it is not yet known how such tools might influence social science research. I argue Generative AI has the potential to improve survey research, online experiments, automated content analyses, agent-based models, and other techniques commonly used to study human behavior. In the second section of this article, I discuss the many limitations of Generative. I examine how bias in the data used to train these tools can negatively impact social science research-as well as a range of other challenges related to ethics, replication, environmental impact, and the proliferation of low-quality research. I conclude by arguing that social scientists can address many of these limitations by creating open-source infrastructure for research on human behavior. Such infrastructure is not only necessary to ensure broad access to high-quality research tools, I argue, but also because the progress of AI will require deeper understanding of the social forces that guide human behavior.

Leveraging AI for democratic discourse: Chat interventions can improve online political conversations at scale.

Political discourse is the soul of democracy, but misunderstanding and conflict can fester in divisive conversations. The widespread shift to online discourse exacerbates many of these problems and corrodes the capacity of diverse societies to cooperate in solving social problems. Scholars and civil society groups promote interventions that make conversations less divisive or more productive, but scaling these efforts to online discourse is challenging. We conduct a large-scale experiment that demonstrates how online conversations about divisive topics can be improved with AI tools. Specifically, we employ a large language model to make real-time, evidence-based recommendations intended to improve participants' perception of feeling understood. These interventions improve reported conversation quality, promote democratic reciprocity, and improve the tone, without systematically changing the content of the conversation or moving people's policy attitudes.

Founder personality and entrepreneurial outcomes: A large-scale field study of technology startups.

Technology startups play an essential role in the economy-with seven of the ten largest companies rooted in technology, and venture capital investments totaling approximately $300B annually. Yet, important startup outcomes (e.g., whether a startup raises venture capital or gets acquired) remain difficult to forecast-particularly during the early stages of venture formation. Here, we examine the impact of an essential, yet underexplored, factor that can be observed from the moment of startup creation: founder personality. We predict psychological traits from digital footprints to explore how founder personality is associated with critical startup milestones. Observing 10,541 founder-startup dyads, we provide large-scale, ecologically valid evidence that founder personality is associated with outcomes across all phases of a venture's life (i.e., from raising the earliest funding round to exiting via acquisition or initial public offering). We find that openness and agreeableness are positively related to the likelihood of raising an initial round of funding (but unrelated to all subsequent conditional outcomes). Neuroticism is negatively related to all outcomes, highlighting the importance of founders' resilience. Finally, conscientiousness is positively related to early-stage investment, but negatively related to exit conditional on funding. While prior work has painted conscientiousness as a major benefactor of performance, our findings highlight a potential boundary condition: The fast-moving world of technology startups affords founders with lower or moderate levels of conscientiousness a competitive advantage when it comes to monetizing their business via acquisition or IPO.

Incentivizing free riders improves collective intelligence in social dilemmas.

Collective intelligence challenges are often entangled with collective action problems. For example, voting, rating, and social innovation are collective intelligence tasks that require costly individual contributions. As a result, members of a group often free ride on the information contributed by intrinsically motivated people. Are intrinsically motivated agents the best participants in collective decisions? We embedded a collective intelligence task in a large-scale, virtual world public good game and found that participants who joined the information system but were reluctant to contribute to the public good (free riders) provided more accurate evaluations, whereas participants who rated frequently underperformed. Testing the underlying mechanism revealed that a negative rating bias in free riders is associated with higher accuracy. Importantly, incentivizing evaluations amplifies the relative influence of participants who tend to free ride without altering the (higher) quality of their evaluations, thereby improving collective intelligence. These results suggest that many of the currently available information systems, which strongly select for intrinsically motivated participants, underperform and that collective intelligence can benefit from incentivizing free riding members to engage. More generally, enhancing the diversity of contributor motivations can improve collective intelligence in settings that are entangled with collective action problems.

Computational Social Psychology.

Social psychologists attempt to explain how we interact by appealing to basic principles of how we think. To make good on this ambition, they are increasingly relying on an interconnected set of formal tools that model inference, attribution, value-guided decision making, and multi-agent interactions. By reviewing progress in each of these areas and highlighting the connections between them, we can better appreciate the structure of social thought and behavior, while also coming to understand when, why, and how formal tools can be useful for social psychologists.

Gender-diverse teams produce more novel and higher-impact scientific ideas.

Science's changing demographics raise new questions about research team diversity and research outcomes. We study mixed-gender research teams, examining 6.6 million papers published across the medical sciences since 2000 and establishing several core findings. First, the fraction of publications by mixed-gender teams has grown rapidly, yet mixed-gender teams continue to be underrepresented compared to the expectations of a null model. Second, despite their underrepresentation, the publications of mixed-gender teams are substantially more novel and impactful than the publications of same-gender teams of equivalent size. Third, the greater the gender balance on a team, the better the team scores on these performance measures. Fourth, these patterns generalize across medical subfields. Finally, the novelty and impact advantages seen with mixed-gender teams persist when considering numerous controls and potential related features, including fixed effects for the individual researchers, team structures, and network positioning, suggesting that a team's gender balance is an underrecognized yet powerful correlate of novel and impactful scientific discoveries.

The consequences of online partisan media.

What role do ideologically extreme media play in the polarization of society? Here we report results from a randomized longitudinal field experiment embedded in a nationally representative online panel survey (N = 1,037) in which participants were incentivized to change their browser default settings and social media following patterns, boosting the likelihood of encountering news with either a left-leaning (HuffPost) or right-leaning (Fox News) slant during the 2018 US midterm election campaign. Data on ≈ 19 million web visits by respondents indicate that resulting changes in news consumption persisted for at least 8 wk. Greater exposure to partisan news can cause immediate but short-lived increases in website visits and knowledge of recent events. After adjusting for multiple comparisons, however, we find little evidence of a direct impact on opinions or affect. Still, results from later survey waves suggest that both treatments produce a lasting and meaningful decrease in trust in the mainstream media up to 1 y later. Consistent with the minimal-effects tradition, direct consequences of online partisan media are limited, although our findings raise questions about the possibility of subtle, cumulative dynamics. The combination of experimentation and computational social science techniques illustrates a powerful approach for studying the long-term consequences of exposure to partisan news.

Stewardship of global collective behavior.

Collective behavior provides a framework for understanding how the actions and properties of groups emerge from the way individuals generate and share information. In humans, information flows were initially shaped by natural selection yet are increasingly structured by emerging communication technologies. Our larger, more complex social networks now transfer high-fidelity information over vast distances at low cost. The digital age and the rise of social media have accelerated changes to our social systems, with poorly understood functional consequences. This gap in our knowledge represents a principal challenge to scientific progress, democracy, and actions to address global crises. We argue that the study of collective behavior must rise to a "crisis discipline" just as medicine, conservation, and climate science have, with a focus on providing actionable insight to policymakers and regulators for the stewardship of social systems.

Bots are less central than verified accounts during contentious political events.

Information manipulation is widespread in today's media environment. Online networks have disrupted the gatekeeping role of traditional media by allowing various actors to influence the public agenda; they have also allowed automated accounts (or bots) to blend with human activity in the flow of information. Here, we assess the impact that bots had on the dissemination of content during two contentious political events that evolved in real time on social media. We focus on events of heightened political tension because they are particularly susceptible to information campaigns designed to mislead or exacerbate conflict. We compare the visibility of bots with human accounts, verified accounts, and mainstream news outlets. Our analyses combine millions of posts from a popular microblogging platform with web-tracking data collected from two different countries and timeframes. We employ tools from network science, natural language processing, and machine learning to analyze the diffusion structure, the content of the messages diffused, and the actors behind those messages as the political events unfolded. We show that verified accounts are significantly more visible than unverified bots in the coverage of the events but also that bots attract more attention than human accounts. Our findings highlight that social media and the web are very different news ecosystems in terms of prevalent news sources and that both humans and bots contribute to generate discrepancy in news visibility with their activity.

Gender inequities in the online dissemination of scholars' work.

Unbiased science dissemination has the potential to alleviate some of the known gender disparities in academia by exposing female scholars' work to other scientists and the public. And yet, we lack comprehensive understanding of the relationship between gender and science dissemination online. Our large-scale analyses, encompassing half a million scholars, revealed that female scholars' work is mentioned less frequently than male scholars' work in all research areas. When exploring the characteristics associated with online success, we found that the impact of prior work, social capital, and gendered tie formation in coauthorship networks are linked with online success for men, but not for women-even in the areas with the highest female representation. These results suggest that while men's scientific impact and collaboration networks are associated with higher visibility online, there are no universally identifiable facets associated with success for women. Our comprehensive empirical evidence indicates that the gender gap in online science dissemination is coupled with a lack of understanding the characteristics that are linked with female scholars' success, which might hinder efforts to close the gender gap in visibility.

Postmortem memory of public figures in news and social media.

Deceased public figures are often said to live on in collective memory. We quantify this phenomenon by tracking mentions of 2,362 public figures in English-language online news and social media (Twitter) 1 y before and after death. We measure the sharp spike and rapid decay of attention following death and model collective memory as a composition of communicative and cultural memory. Clustering reveals four patterns of postmortem memory, and regression analysis shows that boosts in media attention are largest for premortem popular anglophones who died a young, unnatural death; that long-term boosts are smallest for leaders and largest for artists; and that, while both the news and Twitter are triggered by young and unnatural deaths, the news additionally curates collective memory when old persons or leaders die. Overall, we illuminate the age-old question of who is remembered by society, and the distinct roles of news and social media in collective memory formation.

Shadows of wisdom: Classifying meta-cognitive and morally grounded narrative content via large language models.

We investigated large language models' (LLMs) efficacy in classifying complex psychological constructs like intellectual humility, perspective-taking, open-mindedness, and search for a compromise in narratives of 347 Canadian and American adults reflecting on a workplace conflict. Using state-of-the-art models like GPT-4 across few-shot and zero-shot paradigms and RoB-ELoC (RoBERTa -fine-tuned-on-Emotion-with-Logistic-Regression-Classifier), we compared their performance with expert human coders. Results showed robust classification by LLMs, with over 80% agreement and F1 scores above 0.85, and high human-model reliability (Cohen's κ Md across top models = .80). RoB-ELoC and few-shot GPT-4 were standout classifiers, although somewhat less effective in categorizing intellectual humility. We offer example workflows for easy integration into research. Our proof-of-concept findings indicate the viability of both open-source and commercial LLMs in automating the coding of complex constructs, potentially transforming social science research.

Decoding the News Media Diet of Disinformation Spreaders.

In the digital era, information consumption is predominantly channeled through online news media and disseminated on social media platforms. Understanding the complex dynamics of the news media environment and users' habits within the digital ecosystem is a challenging task that requires, at the same time, large databases and accurate methodological approaches. This study contributes to this expanding research landscape by employing network science methodologies and entropic measures to analyze the behavioral patterns of social media users sharing news pieces and dig into the diverse news consumption habits within different online social media user groups. Our analyses reveal that users are more inclined to share news classified as fake when they have previously posted conspiracy or junk science content and vice versa, creating a series of "misinformation hot streaks". To better understand these dynamics, we used three different measures of entropy to gain insights into the news media habits of each user, finding that the patterns of news consumption significantly differ among users when focusing on disinformation spreaders as opposed to accounts sharing reliable or low-risk content. Thanks to these entropic measures, we quantify the variety and the regularity of the news media diet, finding that those disseminating unreliable content exhibit a more varied and, at the same time, a more regular choice of web-domains. This quantitative insight into the nuances of news consumption behaviors exhibited by disinformation spreaders holds the potential to significantly inform the strategic formulation of more robust and adaptive social media moderation policies.

Social media and social impact assessment: Evolving methods in a shifting context.

Among many by-products of Web 2.0 come the wide range of potential image and text datasets within social media and content sharing platforms that speak of how people live, what they do, and what they care about. These datasets are imperfect and biased in many ways, but those flaws make them complementary to data derived from conventional social science methods and thus potentially useful for triangulation in complex decision-making contexts. Yet the online environment is highly mutable, and so the datasets are less reliable than censuses or other standard data types leveraged in social impact assessment. Over the past decade, we have innovated numerous methods for deploying Instagram datasets in investigating management or development alternatives. This article synthesizes work from three Canadian decision contexts - hydroelectric dam construction or removal; dyke realignment or wetland restoration; and integrating renewable energy into vineyard landscapes - to illustrate some of the methods we have applied to social impact assessment questions using Instagram that may be transferrable to other social media platforms and contexts: thematic (manual coding, machine vision, natural language processing/sentiment analysis, statistical analysis), spatial (hotspot mapping, cultural ecosystem modeling), and visual (word clouds, saliency mapping, collage). We conclude with a set of cautions and next steps for the domain.

Parmi les nombreux sous-produits du Web 2.0 figure un large éventail de données provenant d'images et de textes, de contenus de médias sociaux et de plateformes numériques, qui révèlent comment les gens vivent, ce qu'ils font et les questions qui les préoccupent. Ces ensembles de données sont imparfaits et biaisés à bien des égards, mais nombre de leurs lacunes les rendent complémentaires des informations collectées par les sciences sociales à l'aide de méthodes conventionnelles. D'où leur utilité potentielle pour la triangulation dans des contextes décisionnels complexes. Cet article synthétise le travail de trois études de cas menées au Canada pour illustrer certaines des méthodes que nous avons développées et qui pourraient être utiles à d'autres chercheurs en EIS: thématiques (codage, apprentissage automatique, analyse sémantique, association statistique), spatiales (cartographie des points chauds, modélisation du transfert des bénéfices) et visuelles (cartes de saillance, collage).

Entre los muchos subproductos de la Web 2.0 se encuentra una amplia gama de datos de imágenes y texto, contenidos en redes sociales y plataformas digitales, que hablan de cómo vive, qué hace y por qué cuestiones se preocupa la gente. Estos conjuntos de datos son imperfectos y sesgados en muchos sentidos, pero muchos de sus defectos los hacen complementarios a la información recogida por las ciencias sociales con métodos convencionales. De ahí su potencial utilidad para la triangulación en contextos complejos de toma de decisiones. Este artículo sintetiza el trabajo de tres estudios de caso llevados a cabo en Canadá para ilustrar algunos de los métodos que hemos desarrollado y pueden resultar útiles para otros investigadores en EIS: temáticos (codificación, machine learning, análisis semántico, asociación estadística), espaciales (mapeo de puntos críticos, modelización de transferencia de beneficios) y visuales (mapas de saliencia, collage).

Estimating the deep replicability of scientific findings using human and artificial intelligence.

Replicability tests of scientific papers show that the majority of papers fail replication. Moreover, failed papers circulate through the literature as quickly as replicating papers. This dynamic weakens the literature, raises research costs, and demonstrates the need for new approaches for estimating a study's replicability. Here, we trained an artificial intelligence model to estimate a paper's replicability using ground truth data on studies that had passed or failed manual replication tests, and then tested the model's generalizability on an extensive set of out-of-sample studies. The model predicts replicability better than the base rate of reviewers and comparably as well as prediction markets, the best present-day method for predicting replicability. In out-of-sample tests on manually replicated papers from diverse disciplines and methods, the model had strong accuracy levels of 0.65 to 0.78. Exploring the reasons behind the model's predictions, we found no evidence for bias based on topics, journals, disciplines, base rates of failure, persuasion words, or novelty words like "remarkable" or "unexpected." We did find that the model's accuracy is higher when trained on a paper's text rather than its reported statistics and that n-grams, higher order word combinations that humans have difficulty processing, correlate with replication. We discuss how combining human and machine intelligence can raise confidence in research, provide research self-assessment techniques, and create methods that are scalable and efficient enough to review the ever-growing numbers of publications-a task that entails extensive human resources to accomplish with prediction markets and manual replication alone.

Assessing the Russian Internet Research Agency's impact on the political attitudes and behaviors of American Twitter users in late 2017.

There is widespread concern that Russia and other countries have launched social-media campaigns designed to increase political divisions in the United States. Though a growing number of studies analyze the strategy of such campaigns, it is not yet known how these efforts shaped the political attitudes and behaviors of Americans. We study this question using longitudinal data that describe the attitudes and online behaviors of 1,239 Republican and Democratic Twitter users from late 2017 merged with nonpublic data about the Russian Internet Research Agency (IRA) from Twitter. Using Bayesian regression tree models, we find no evidence that interaction with IRA accounts substantially impacted 6 distinctive measures of political attitudes and behaviors over a 1-mo period. We also find that interaction with IRA accounts were most common among respondents with strong ideological homophily within their Twitter network, high interest in politics, and high frequency of Twitter usage. Together, these findings suggest that Russian trolls might have failed to sow discord because they mostly interacted with those who were already highly polarized. We conclude by discussing several important limitations of our study-especially our inability to determine whether IRA accounts influenced the 2016 presidential election-as well as its implications for future research on social media influence campaigns, political polarization, and computational social science.

Mentorship and protégé success in STEM fields.

Einstein believed that mentors are especially influential in a protégé's intellectual development, yet the link between mentorship and protégé success remains a mystery. We marshaled genealogical data on nearly 40,000 scientists who published 1,167,518 papers in biomedicine, chemistry, math, or physics between 1960 and 2017 to investigate the relationship between mentorship and protégé achievement. In our data, we find groupings of mentors with similar records and reputations who attracted protégés of similar talents and expected levels of professional success. However, each grouping has an exception: One mentor has an additional hidden capability that can be mentored to their protégés. They display skill in creating and communicating prizewinning research. Because the mentor's ability for creating and communicating celebrated research existed before the prize's conferment, protégés of future prizewinning mentors can be uniquely exposed to mentorship for conducting celebrated research. Our models explain 34-44% of the variance in protégé success and reveals three main findings. First, mentorship strongly predicts protégé success across diverse disciplines. Mentorship is associated with a 2×-to-4× rise in a protégé's likelihood of prizewinning, National Academy of Science (NAS) induction, or superstardom relative to matched protégés. Second, mentorship is significantly associated with an increase in the probability of protégés pioneering their own research topics and being midcareer late bloomers. Third, contrary to conventional thought, protégés do not succeed most by following their mentors' research topics but by studying original topics and coauthoring no more than a small fraction of papers with their mentors.

Exposure to news grows less fragmented with an increase in mobile access.

The abundance of media options is a central feature of today's information environment. Many accounts, often based on analysis of desktop-only news use, suggest that this increased choice leads to audience fragmentation, ideological segregation, and echo chambers with no cross-cutting exposure. Contrary to many of those claims, this paper uses observational multiplatform data capturing both desktop and mobile use to demonstrate that coexposure to diverse news is on the rise, and that ideological self-selection does not explain most of that coexposure. We show that mainstream media outlets offer the common ground where ideologically diverse audiences converge online, though our analysis also reveals that more than half of the US online population consumes no online news, underlining the risk of increased information inequality driven by self-selection along lines of interest. For this study, we use an unprecedented combination of observed data from the United States comprising a 5-y time window and involving tens of thousands of panelists. Our dataset traces news consumption across different devices and unveils important differences in news diets when multiplatform or desktop-only access is used. We discuss the implications of our findings for how we think about the current communication environment, exposure to news, and ongoing attempts to limit the effects of misinformation.

Does big data serve policy? Not without context. An experiment with in silico social science.

The DARPA Ground Truth project sought to evaluate social science by constructing four varied simulated social worlds with hidden causality and unleashed teams of scientists to collect data, discover their causal structure, predict their future, and prescribe policies to create desired outcomes. This large-scale, long-term experiment of in silico social science, about which the ground truth of simulated worlds was known, but not by us, reveals the limits of contemporary quantitative social science methodology. First, problem solving without a shared ontology-in which many world characteristics remain existentially uncertain-poses strong limits to quantitative analysis even when scientists share a common task, and suggests how they could become insurmountable without it. Second, data labels biased the associations our analysts made and assumptions they employed, often away from the simulated causal processes those labels signified, suggesting limits on the degree to which analytic concepts developed in one domain may port to others. Third, the current standard for computational social science publication is a demonstration of novel causes, but this limits the relevance of models to solve problems and propose policies that benefit from the simpler and less surprising answers associated with most important causes, or the combination of all causes. Fourth, most singular quantitative methods applied on their own did not help to solve most analytical challenges, and we explored a range of established and emerging methods, including probabilistic programming, deep neural networks, systems of predictive probabilistic finite state machines, and more to achieve plausible solutions. However, despite these limitations common to the current practice of computational social science, we find on the positive side that even imperfect knowledge can be sufficient to identify robust prediction if a more pluralistic approach is applied. Applying competing approaches by distinct subteams, including at one point the vast TopCoder.com global community of problem solvers, enabled discovery of many aspects of the relevant structure underlying worlds that singular methods could not. Together, these lessons suggest how different a policy-oriented computational social science would be than the computational social science we have inherited. Computational social science that serves policy would need to endure more failure, sustain more diversity, maintain more uncertainty, and allow for more complexity than current institutions support.

A research agenda for the study of social norm change.

Social norms have been investigated across many disciplines for many years, but until recently, studies mainly provided indirect, implicit and correlational support for the role of social norms in driving behaviour. To understand how social norms, and in particular social norm change, can generate a large-scale behavioural change to deal with some of the most pressing challenges of our current societies, such as climate change and vaccine hesitancy, we discuss and review several recent advances in social norm research that enable a more precise underpinning of the role of social norms: how to identify their existence, how to establish their causal effect on behaviour and when norm change may pass tipping points. We advocate future research on social norms to study norm change through a mechanism-based approach that integrates experimental and computational methods in theory-driven, empirically calibrated agent-based models. As such, social norm research may move beyond unequivocal praising of social norms as the missing link between self-interested behaviour and observed cooperation or as the explanation for (the lack of) social tipping. It provides the toolkit to understand explicitly where, when and how social norms can be a solution to solve large-scale problems, but also to recognize their limits. This article is part of the theme issue 'Emergent phenomena in complex physical and socio-technical systems: from cells to societies'.