Testing theory of mind in large language models and humans.

At the core of what defines us as humans is the concept of theory of mind: the ability to track other people's mental states. The recent development of large language models (LLMs) such as ChatGPT has led to intense debate about the possibility that these models exhibit behaviour that is indistinguishable from human behaviour in theory of mind tasks. Here we compare human and LLM performance on a comprehensive battery of measurements that aim to measure different theory of mind abilities, from understanding false beliefs to interpreting indirect requests and recognizing irony and faux pas. We tested two families of LLMs (GPT and LLaMA2) repeatedly against these measures and compared their performance with those from a sample of 1,907 human participants. Across the battery of theory of mind tests, we found that GPT-4 models performed at, or even sometimes above, human levels at identifying indirect requests, false beliefs and misdirection, but struggled with detecting faux pas. Faux pas, however, was the only test where LLaMA2 outperformed humans. Follow-up manipulations of the belief likelihood revealed that the superiority of LLaMA2 was illusory, possibly reflecting a bias towards attributing ignorance. By contrast, the poor performance of GPT originated from a hyperconservative approach towards committing to conclusions rather than from a genuine failure of inference. These findings not only demonstrate that LLMs exhibit behaviour that is consistent with the outputs of mentalistic inference in humans but also highlight the importance of systematic testing to ensure a non-superficial comparison between human and artificial intelligences.

Kinematic priming of action predictions.

The ability to anticipate what others will do next is crucial for navigating social, interactive environments. Here, we develop an experimental and analytical framework to measure the implicit readout of prospective intention information from movement kinematics. Using a primed action categorization task, we first demonstrate implicit access to intention information by establishing a novel form of priming, which we term kinematic priming: subtle differences in movement kinematics prime action prediction. Next, using data collected from the same participants in a forced-choice intention discrimination task 1 h later, we quantify single-trial intention readout-the amount of intention information read by individual perceivers in individual kinematic primes-and assess whether it can be used to predict the amount of kinematic priming. We demonstrate that the amount of kinematic priming, as indexed by both response times (RTs) and initial fixations to a given probe, is directly proportional to the amount of intention information read by the individual perceiver at the single-trial level. These results demonstrate that human perceivers have rapid, implicit access to intention information encoded in movement kinematics and highlight the potential of our approach to reveal the computations that permit the readout of this information with single-subject, single-trial resolution.

Effects of acoustic and semantic cues on listening effort during native and non-native speech perception.

Relative to native listeners, non-native listeners who are immersed in a second language environment experience increased listening effort and a reduced ability to successfully perform an additional task while listening. Previous research demonstrated that listeners can exploit a variety of intelligibility-enhancing cues to cope with adverse listening conditions. However, little is known about the implications of those speech perception strategies for listening effort. The current research aims to investigate by means of pupillometry how listening effort is modulated in native and non-native listeners by the availability of semantic context and acoustic enhancements during the comprehension of spoken sentences. For this purpose, semantic plausibility and speaking style were manipulated both separately and in combination during a speech perception task in noise. The signal to noise ratio was individually adjusted for each participant in order to target 50% intelligibility level. Behavioural results indicated that native and non-native listeners were equally able to fruitfully exploit both semantic and acoustic cues to aid their comprehension. Pupil data indicated that listening effort was reduced for both groups of listeners when acoustic enhancements were available, while the presence of a plausible semantic context did not lead to a reduction in listening effort.

Alpha Oscillations Are Causally Linked to Inhibitory Abilities in Ageing.

Aging adults typically show reduced ability to ignore task-irrelevant information, an essential skill for optimal performance in many cognitive operations, including those requiring working memory (WM) resources. In a first experiment, young and elderly human participants of both genders performed an established WM paradigm probing inhibitory abilities by means of valid, invalid, and neutral retro-cues. Elderly participants showed an overall cost, especially in performing invalid trials, whereas younger participants' general performance was comparatively higher, as expected.Inhibitory abilities have been linked to alpha brain oscillations but it is yet unknown whether in aging these oscillations (also typically impoverished) and inhibitory abilities are causally linked. To probe this possible causal link in aging, we compared in a second experiment parietal alpha-transcranial alternating current stimulation (tACS) with either no stimulation (Sham) or with two control stimulation frequencies (theta- and gamma-tACS) in the elderly group while performing the same WM paradigm. Alpha- (but not theta- or gamma-) tACS selectively and significantly improved performance (now comparable to younger adults' performance in the first experiment), particularly for invalid cues where initially elderly showed the highest costs. Alpha oscillations are therefore causally linked to inhibitory abilities and frequency-tuned alpha-tACS interventions can selectively change these abilities in the elderly.SIGNIFICANCE STATEMENT Ignoring task-irrelevant information, an ability associated to rhythmic brain activity in the alpha frequency band, is fundamental for optimal performance. Indeed, impoverished inhibitory abilities contribute to age-related decline in cognitive functions like working memory (WM), the capacity to briefly hold information in mind. Whether in aging adults alpha oscillations and inhibitory abilities are causally linked is yet unknown. We experimentally manipulated frequency-tuned brain activity using transcranial alternating current stimulation (tACS), combined with a retro-cue paradigm assessing WM and inhibition. We found that alpha-tACS induced a significant improvement in target responses and misbinding errors, two indexes of inhibition. We concluded that in aging alpha oscillations are causally linked to inhibitory abilities, and that despite being impoverished, these abilities are still malleable.

Listening Effort During Sentence Processing Is Increased for Non-native Listeners: A Pupillometry Study.

Current evidence demonstrates that even though some non-native listeners can achieve native-like performance for speech perception tasks in quiet, the presence of a background noise is much more detrimental to speech intelligibility for non-native compared to native listeners. Even when performance is equated across groups, it is likely that greater listening effort is required for non-native listeners. Importantly, the added listening effort might result in increased fatigue and a reduced ability to successfully perform multiple tasks simultaneously. Task-evoked pupil responses have been demonstrated to be a reliable measure of cognitive effort and can be useful in clarifying those aspects. In this study we compared the pupil response for 23 native English speakers and 27 Italian speakers of English as a second language. Speech intelligibility was tested for sentences presented in quiet and in background noise at two performance levels that were matched across groups. Signal-to-noise levels corresponding to these sentence intelligibility levels were pre-determined using an adaptive intelligibility task. Pupil response was significantly greater in non-native compared to native participants across both intelligibility levels. Therefore, for a given intelligibility level, a greater listening effort is required when listening in a second language in order to understand speech in noise. Results also confirmed that pupil response is sensitive to speech intelligibility during language comprehension, in line with previous research. However, contrary to our predictions, pupil response was not differentially modulated by intelligibility levels for native and non-native listeners. The present study corroborates that pupillometry can be deemed as a valid measure to be used in speech perception investigation, because it is sensitive to differences both across participants, such as listener type, and across conditions, such as variations in the level of speech intelligibility. Importantly, pupillometry offers us the possibility to uncover differences in listening effort even when those do not emerge in the performance level of individuals.