AI Psychometrics: Assessing the Psychological Profiles of Large Language Models Through Psychometric Inventories.

We illustrate how standard psychometric inventories originally designed for assessing noncognitive human traits can be repurposed as diagnostic tools to evaluate analogous traits in large language models (LLMs). We start from the assumption that LLMs, inadvertently yet inevitably, acquire psychological traits (metaphorically speaking) from the vast text corpora on which they are trained. Such corpora contain sediments of the personalities, values, beliefs, and biases of the countless human authors of these texts, which LLMs learn through a complex training process. The traits that LLMs acquire in such a way can potentially influence their behavior, that is, their outputs in downstream tasks and applications in which they are employed, which in turn may have real-world consequences for individuals and social groups. By eliciting LLMs' responses to language-based psychometric inventories, we can bring their traits to light. Psychometric profiling enables researchers to study and compare LLMs in terms of noncognitive characteristics, thereby providing a window into the personalities, values, beliefs, and biases these models exhibit (or mimic). We discuss the history of similar ideas and outline possible psychometric approaches for LLMs. We demonstrate one promising approach, zero-shot classification, for several LLMs and psychometric inventories. We conclude by highlighting open challenges and future avenues of research for AI Psychometrics.

Collective emotions during the COVID-19 outbreak.

The COVID-19 pandemic has exposed the world's population to unprecedented health threats and changes to social life. High uncertainty about the novel disease and its social and economic consequences, together with increasingly stringent governmental measures against the spread of the virus, likely elicited strong emotional responses. We analyzed the digital traces of emotional expressions in tweets during 5 weeks after the start of outbreaks in 18 countries and six different languages. We observed an early strong upsurge of anxiety-related terms in all countries, which was related to the growth in cases and increases in the stringency of governmental measures. Anxiety expression gradually relaxed once stringent measures were in place, possibly indicating that people were reassured. Sadness terms rose and anger terms decreased with or after an increase in the stringency of measures and remained stable as long as measures were in place. Positive emotion words only decreased slightly and briefly in a few countries. Our results reveal some of the most enduring changes in emotional expression observed in long periods of social media data. Such sustained emotional expression could indicate that interactions between users led to the emergence of collective emotions. Words that frequently occurred in tweets suggest a shift in topics of conversation across all emotions, from political ones in 2019, to pandemic related issues during the outbreak, including everyday life changes, other people, and health. This kind of time-sensitive analyses of large-scale samples of emotional expression have the potential to inform risk communication. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Validating daily social media macroscopes of emotions.

Measuring sentiment in social media text has become an important practice in studying emotions at the macroscopic level. However, this approach can suffer from methodological issues like sampling biases and measurement errors. To date, it has not been validated if social media sentiment can actually measure the temporal dynamics of mood and emotions aggregated at the level of communities. We ran a large-scale survey at an online newspaper to gather daily mood self-reports from its users, and compare these with aggregated results of sentiment analysis of user discussions. We find strong correlations between text analysis results and levels of self-reported mood, as well as between inter-day changes of both measurements. We replicate these results using sentiment data from Twitter. We show that a combination of supervised text analysis methods based on novel deep learning architectures and unsupervised dictionary-based methods have high agreement with the time series of aggregated mood measured with self-reports. Our findings indicate that macro level dynamics of mood expressed on an online platform can be tracked with social media text, especially in situations of high mood variability.

Colexification Networks Encode Affective Meaning.

Colexification is a linguistic phenomenon that occurs when multiple concepts are expressed in a language with the same word. Colexification patterns are frequently used to estimate the meaning similarity between words, but the hypothesis that these are related is still missing direct empirical validation at scale. Here, we show for the first time that words linked by colexification patterns capture similar affective meanings. Using pre-existing translation data, we extend colexification databases to cover much longer word lists. We achieve this with an unsupervised method of affective lexicon extension that uses colexification network data to interpolate the affective ratings of words that are not included in the original lexicon. We find positive correlations between network-based estimates and empirical affective ratings, which suggest that colexification networks contain information related to affective meanings. Finally, we compare our network method with state-of-the-art machine learning, trained on a large corpus, and show that our simple linguistics-informed unsupervised algorithm yields comparable performance with high explainability. These results show that it is possible to automatically expand affective norms lexica to cover exhaustive word lists when additional data are available, such as in colexification networks. Supplementary Information: The online version contains supplementary material available at 10.1007/s42761-021-00033-1.

Dashboard of Sentiment in Austrian Social Media During COVID-19.

To track online emotional expressions on social media platforms close to real-time during the COVID-19 pandemic, we built a self-updating monitor of emotion dynamics using digital traces from three different data sources in Austria. This allows decision makers and the interested public to assess dynamics of sentiment online during the pandemic. We used web scraping and API access to retrieve data from the news platform derstandard.at, Twitter, and a chat platform for students. We documented the technical details of our workflow to provide materials for other researchers interested in building a similar tool for different contexts. Automated text analysis allowed us to highlight changes of language use during COVID-19 in comparison to a neutral baseline. We used special word clouds to visualize that overall difference. Longitudinally, our time series showed spikes in anxiety that can be linked to several events and media reporting. Additionally, we found a marked decrease in anger. The changes lasted for remarkably long periods of time (up to 12 weeks). We have also discussed these and more patterns and connect them to the emergence of collective emotions. The interactive dashboard showcasing our data is available online at http://www.mpellert.at/covid19_monitor_austria/. Our work is part of a web archive of resources on COVID-19 collected by the Austrian National Library.