Bayesian modeling of human-AI complementarity.

SignificanceWith the increase in artificial intelligence in real-world applications, there is interest in building hybrid systems that take both human and machine predictions into account. Previous work has shown the benefits of separately combining the predictions of diverse machine classifiers or groups of people. Using a Bayesian modeling framework, we extend these results by systematically investigating the factors that influence the performance of hybrid combinations of human and machine classifiers while taking into account the unique ways human and algorithmic confidence is expressed.

Building and benchmarking the motivated deception corpus: Improving the quality of deceptive text through gaming.

When one studies fake news or false reviews, the first step to take is to find a corpus of text samples to work with. However, most deceptive corpora suffer from an intrinsic problem: there is little incentive for the providers of the deception to put their best effort, which risks lowering the quality and realism of the deception. The corpus described in this project, the Motivated Deception Corpus, aims to rectify this problem by gamifying the process of deceptive text collection. By having subjects play the game Two Truths and a Lie, and by rewarding those subjects that successfully fool their peers, we collect samples in such a way that the process itself improves the quality of the text. We have amassed a large corpus of deceptive text that is strongly incentivized to be convincing, and thus more reflective of real deceptive text. We provide results from several configurations of neural network prediction models to establish machine learning benchmarks on the data. This new corpus is demonstratively more challenging to classify with the current state of the art than previous corpora.

A large-scale analysis of task switching practice effects across the lifespan.

An important feature of human cognition is the ability to flexibly and efficiently adapt behavior in response to continuously changing contextual demands. We leverage a large-scale dataset from Lumosity, an online cognitive-training platform, to investigate how cognitive processes involved in cued switching between tasks are affected by level of task practice across the adult lifespan. We develop a computational account of task switching that specifies the temporal dynamics of activating task-relevant representations and inhibiting task-irrelevant representations and how they vary with extended task practice across a number of age groups. Practice modulates the level of activation of the task-relevant representation and improves the rate at which this information becomes available, but has little effect on the task-irrelevant representation. While long-term practice improves performance across all age groups, it has a greater effect on older adults. Indeed, extensive task practice can make older individuals functionally similar to less-practiced younger individuals, especially for cognitive measures that focus on the rate at which task-relevant information becomes available.

Experimental design modulates variance in BOLD activation: The variance design general linear model.

Typical fMRI studies have focused on either the mean trend in the blood-oxygen-level-dependent (BOLD) time course or functional connectivity (FC). However, other statistics of the neuroimaging data may contain important information. Despite studies showing links between the variance in the BOLD time series (BV) and age and cognitive performance, a formal framework for testing these effects has not yet been developed. We introduce the variance design general linear model (VDGLM), a novel framework that facilitates the detection of variance effects. We designed the framework for general use in any fMRI study by modeling both mean and variance in BOLD activation as a function of experimental design. The flexibility of this approach allows the VDGLM to (a) simultaneously make inferences about a mean or variance effect while controlling for the other and (b) test for variance effects that could be associated with multiple conditions and/or noise regressors. We demonstrate the use of the VDGLM in a working memory application and show that engagement in a working memory task is associated with whole-brain decreases in BOLD variance.

Monitoring the ebb and flow of attention: Does controlling the onset of stimuli during encoding enhance memory?

Central to the operation of the Atkinson and Shiffrin's (Psychology of learning and motivation, 2, 89-195, 1968) model of human memory are a variety of control processes that manage information flow. Research on metacognition reveals that provision of control in laboratory learning tasks is generally beneficial to memory. In this paper, we investigate the novel domain of attentional fluctuations during study. If learners are able to monitor attention, then control over the onset of stimuli should also improve performance. Across four experiments, we found no evidence that control over the onset of stimuli enhances learning. This result stands in notable contrast to the fact that control over stimulus offset does enhance memory (Experiment 1; Tullis & Benjamin, Journal of memory and language, 64 (2), 109-118, 2011). This null finding was replicated across laboratory and online samples of subjects, and with both words and faces as study material. Taken together, the evidence suggests that people either cannot monitor fluctuations in attention effectively or cannot precisely time their study to those fluctuations.

The joint contribution of participation and performance to learning functions: Exploring the effects of age in large-scale data sets.

Large-scale data sets from online training and game platforms offer the opportunity for more extensive and more precise investigations of human learning than is typically achievable in the laboratory. However, because people make their own choices about participation, any investigation into learning using these data sets must simultaneously model performance-that is, the learning function-and participation. Using a data set of 54 million gameplays from the online brain training site Lumosity, we show that learning functions of participants are systematically biased by participation policies that vary with age. Older adults who are poorer performers are more likely to drop out than older adults who perform well. Younger adults show no such effect. Using this knowledge, we can extrapolate group learning functions that correct for these age-related differences in dropout.

Why more is better: Simultaneous modeling of EEG, fMRI, and behavioral data.

The need to test a growing number of theories in cognitive science has led to increased interest in inferential methods that integrate multiple data modalities. In this manuscript, we show how a method for integrating three data modalities within a single framework provides (1) more detailed descriptions of cognitive processes and (2) more accurate predictions of unobserved data than less integrative methods. Specifically, we show how combining either EEG and fMRI with a behavioral model can perform substantially better than a behavioral-data-only model in both generative and predictive modeling analyses. We then show how a trivariate model - a model including EEG, fMRI, and behavioral data - outperforms bivariate models in both generative and predictive modeling analyses. Together, these results suggest that within an appropriate modeling framework, more data can be used to better constrain cognitive theory, and to generate more accurate predictions for behavioral and neural data.

A hierarchical adaptive approach to optimal experimental design.

Experimentation is at the core of research in the behavioral and neural sciences, yet observations can be expensive and time-consuming to acquire (e.g., MRI scans, responses from infant participants). A major interest of researchers is designing experiments that lead to maximal accumulation of information about the phenomenon under study with the fewest possible number of observations. In addressing this challenge, statisticians have developed adaptive design optimization methods. This letter introduces a hierarchical Bayes extension of adaptive design optimization that provides a judicious way to exploit two complementary schemes of inference (with past and future data) to achieve even greater accuracy and efficiency in information gain. We demonstrate the method in a simulation experiment in the field of visual perception.

Evaluating probabilistic forecasts with Bayesian signal detection models.

We propose the use of signal detection theory (SDT) to evaluate the performance of both probabilistic forecasting systems and individual forecasters. The main advantage of SDT is that it provides a principled way to distinguish the response from system diagnosticity, which is defined as the ability to distinguish events that occur from those that do not. There are two challenges in applying SDT to probabilistic forecasts. First, the SDT model must handle judged probabilities rather than the conventional binary decisions. Second, the model must be able to operate in the presence of sparse data generated within the context of human forecasting systems. Our approach is to specify a model of how individual forecasts are generated from underlying representations and use Bayesian inference to estimate the underlying latent parameters. Given our estimate of the underlying representations, features of the classic SDT model, such as the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC), follow immediately. We show how our approach allows ROC curves and AUCs to be applied to individuals within a group of forecasters, estimated as a function of time, and extended to measure differences in forecastability across different domains. Among the advantages of this method is that it depends only on the ordinal properties of the probabilistic forecasts. We conclude with a brief discussion of how this approach might facilitate decision making.

How experts and novices judge other people's knowledgeability from language use.

How accurate are people in judging someone else's knowledge based on their language use, and do more knowledgeable people use different cues to make these judgments? We address this by recruiting a group of participants ("informants") to answer general knowledge questions and describe various images belonging to different categories (e.g., cartoons, basketball). A second group of participants ("evaluators") also answer general knowledge questions and decide who is more knowledgeable within pairs of informants, based on these descriptions. Evaluators perform above chance at identifying the most knowledgeable informants (65% with only one description available). The less knowledgeable evaluators base their decisions on the number of specific statements, regardless of whether the statements are true or false. The more knowledgeable evaluators treat true and false statements differently and penalize the knowledge they attribute to informants who produce specific yet false statements. Our findings demonstrate the power of a few words when assessing others' knowledge and have implications for how misinformation is processed differently between experts and novices.

A Bayesian framework for simultaneously modeling neural and behavioral data.

Scientists who study cognition infer underlying processes either by observing behavior (e.g., response times, percentage correct) or by observing neural activity (e.g., the BOLD response). These two types of observations have traditionally supported two separate lines of study. The first is led by cognitive modelers, who rely on behavior alone to support their computational theories. The second is led by cognitive neuroimagers, who rely on statistical models to link patterns of neural activity to experimental manipulations, often without any attempt to make a direct connection to an explicit computational theory. Here we present a flexible Bayesian framework for combining neural and cognitive models. Joining neuroimaging and computational modeling in a single hierarchical framework allows the neural data to influence the parameters of the cognitive model and allows behavioral data, even in the absence of neural data, to constrain the neural model. Critically, our Bayesian approach can reveal interactions between behavioral and neural parameters, and hence between neural activity and cognitive mechanisms. We demonstrate the utility of our approach with applications to simulated fMRI data with a recognition model and to diffusion-weighted imaging data with a response time model of perceptual choice.

Relating memory to functional performance in normal aging to dementia using hierarchical Bayesian cognitive processing models.

Determining how cognition affects functional abilities is important in Alzheimer disease and related disorders. A total of 280 patients (normal or Alzheimer disease and related disorders) received a total of 1514 assessments using the functional assessment staging test (FAST) procedure and the MCI Screen. A hierarchical Bayesian cognitive processing model was created by embedding a signal detection theory model of the MCI Screen-delayed recognition memory task into a hierarchical Bayesian framework. The signal detection theory model used latent parameters of discriminability (memory process) and response bias (executive function) to predict, simultaneously, recognition memory performance for each patient and each FAST severity group. The observed recognition memory data did not distinguish the 6 FAST severity stages, but the latent parameters completely separated them. The latent parameters were also used successfully to transform the ordinal FAST measure into a continuous measure reflecting the underlying continuum of functional severity. Hierarchical Bayesian cognitive processing models applied to recognition memory data from clinical practice settings accurately translated a latent measure of cognition into a continuous measure of functional severity for both individuals and FAST groups. Such a translation links 2 levels of brain information processing and may enable more accurate correlations with other levels, such as those characterized by biomarkers.

Collaborative memory in a serial combination procedure.

This article describes a new approach for studying collaborative memory that examines people's editing processes for naturally occurring memory errors. In this approach, memories of individuals are combined via a chaining method in which each participant indirectly receives information from the previous participant. Participants were asked to individually study word lists and recall as many words as possible in an online setting. Once a participant completed the recall task, his/her answers were provided for the next participant as suggested answers for their own recall. However, that participant was allowed to add or subtract words from the provided list of suggested answers. The final answer of the group was an aggregate of recalled words based on the answer given by the last participant in the chain. Results showed that participants displayed a very high accuracy of recall throughout the chain, although they were not able to replicate the entire study list or eliminate all errors by the end of the chain. This procedure has the advantage that it allows examination of the memory-editing processes individuals utilise when they communicate information indirectly, independent from social factors that arise in face-to-face group memory settings.

Three Challenges for AI-Assisted Decision-Making.

Artificial intelligence (AI) has the potential to improve human decision-making by providing decision recommendations and problem-relevant information to assist human decision-makers. However, the full realization of the potential of human-AI collaboration continues to face several challenges. First, the conditions that support complementarity (i.e., situations in which the performance of a human with AI assistance exceeds the performance of an unassisted human or the AI in isolation) must be understood. This task requires humans to be able to recognize situations in which the AI should be leveraged and to develop new AI systems that can learn to complement the human decision-maker. Second, human mental models of the AI, which contain both expectations of the AI and reliance strategies, must be accurately assessed. Third, the effects of different design choices for human-AI interaction must be understood, including both the timing of AI assistance and the amount of model information that should be presented to the human decision-maker to avoid cognitive overload and ineffective reliance strategies. In response to each of these three challenges, we present an interdisciplinary perspective based on recent empirical and theoretical findings and discuss new research directions.

Linking computational models of two core tasks of cognitive control.

Cognitive control refers to the ability to maintain goal-relevant information in the face of distraction, making it a core construct for understanding human thought and behavior. There is great theoretical and practical value in building theories that can be used to explain or to predict variations in cognitive control as a function of experimental manipulations or individual differences. A critical step toward building such theories is determining which latent constructs are shared between laboratory tasks that are designed to measure cognitive control. In the current work, we examine this question in a novel way by formally linking computational models of two canonical cognitive control tasks, the Eriksen flanker and task-switching task. Specifically, we examine whether model parameters that capture cognitive control processes in one task can be swapped across models to make predictions about individual differences in performance on another task. We apply our modeling and analysis to a large scale data set from an online cognitive training platform, which optimizes our ability to detect individual differences in the data. Our results suggest that the flanker and task-switching tasks probe common control processes. This finding supports the view that higher level cognitive control processes as opposed to solely strategies in speed and accuracy tradeoffs, or perceptual processing and motor response speed are shared across the two tasks. We discuss how our computational modeling substitution approach addresses limitations of prior efforts to relate performance across different cognitive control tasks, and how our findings inform current theories of cognitive control. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Differentiating mental models of self and others: A hierarchical framework for knowledge assessment.

Developing an accurate model of another agent's knowledge is central to communication and cooperation between agents. In this article, we propose a hierarchical framework of knowledge assessment that explains how people construct mental models of their own knowledge and the knowledge of others. Our framework posits that people integrate information about their own and others' knowledge via Bayesian inference. To evaluate this claim, we conduct an experiment in which participants repeatedly assess their own performance (a metacognitive task) and the performance of another person (a type of theory of mind task) on the same image classification tasks. We contrast the hierarchical framework with simpler alternatives that assume different degrees of differentiation between mental models of self and others. Our model accurately captures participants' assessment of their own performance and the performance of others in the task: Initially, people rely on their own self-assessment process to reason about the other person's performance, leading to similar self- and other-performance predictions. As more information about the other person's ability becomes available, the mental model for the other person becomes increasingly distinct from the mental model of self. Simulation studies also confirm that our framework explains a wide range of findings about human knowledge assessment of themselves and others. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Teaching categories via examples and explanations.

People often learn categories through interaction with knowledgeable others who may use verbal explanations, visual exemplars, or both, to share their knowledge. Verbal and nonverbal means of pedagogical communication are commonly used in conjunction, but their respective roles are not fully understood. In this work, we studied how well these modes of communication work with different category structures. We conducted two experiments to investigate the effect of perceptual confusability and stimulus dimensionality on the effectiveness of verbal, exemplar-based, and mixed communication. One group of participants - teachers - learned a categorization rule and prepared learning materials for the students. Students studied the materials prepared for them and then demonstrated their knowledge on test stimuli. All communication modes were generally successful, but not equivalent, with mixed communication consistently showing best results. When teachers were free to generate as many visual exemplars or words as they wish, verbal and exemplar-based communication showed similar performance, although the verbal channel was slightly less reliable in situations requiring high perceptual precision. At the same time, verbal communication was better suited to handling high-dimensional stimuli when communication volume was restricted. We believe that our work serves as an important step towards studying language as a means for pedagogical category leaning.

Jointly modeling behavioral and EEG measures of proactive control in task switching.

In this study, we implement joint modeling of behavioral and single-trial electroencephalography (EEG) data derived from a cued-trials task-switching paradigm to test the hypothesis that trial-by-trial adjustment of response criterion can be linked to changes in the event-related potentials (ERPs) elicited during the cue-target interval (CTI). Specifically, we assess whether ERP components associated with preparation to switch task and preparation of the relevant task are linked to a response criterion parameter derived from a simple diffusion decision model (DDM). Joint modeling frameworks characterize the brain-behavior link by simultaneously modeling behavioral and neural data and implementing a linking function to bind these two submodels. We examined three joint models: The first characterized the core link between EEG and criterion, the second added a switch preparation input parameter and the third also added a task preparation input parameter. The criterion-EEG link was strongest just before target onset. Inclusion of switch and task preparation parameters did not improve the performance of the criterion-EEG link but was necessary to accurately model the ERP waveform morphology. While we successfully jointly modeled latent model parameters and EEG data from a task-switching paradigm, these findings show that customized cognitive models are needed that are tailored to the multiple cognitive control processes underlying task-switching performance. This is the first paper to implement joint modeling of behavioral measures and single-trial electroencephalography (EEG) data derived from the cue-target interval in a cued-trials task-switching paradigm. Model hyperparameters showed a strong link between response criterion and the pre-target negativity amplitude. Additional parameters (switch preparation, task preparation) were necessary to model the cue-locked ERP waveform morphology. This is consistent with multiple cognitive control processes underlying proactive control and points to the need for more nuanced models of task-switching performance.

Comparing models of learning and relearning in large-scale cognitive training data sets.

Practice in real-world settings exhibits many idiosyncracies of scheduling and duration that can only be roughly approximated by laboratory research. Here we investigate 39,157 individuals' performance on two cognitive games on the Lumosity platform over a span of 5 years. The large-scale nature of the data allows us to observe highly varied lengths of uncontrolled interruptions to practice and offers a unique view of learning in naturalistic settings. We enlist a suite of models that grow in the complexity of the mechanisms they postulate and conclude that long-term naturalistic learning is best described with a combination of long-term skill and task-set preparedness. We focus additionally on the nature and speed of relearning after breaks in practice and conclude that those components must operate interactively to produce the rapid relearning that is evident even at exceptionally long delays (over 2 years). Naturalistic learning over long time spans provides a strong test for the robustness of theoretical accounts of learning, and should be more broadly used in the learning sciences.

A Critical Review of Network-Based and Distributional Approaches to Semantic Memory Structure and Processes.

Some of the earliest work on understanding how concepts are organized in memory used a network-based approach, where words or concepts are represented as nodes, and relationships between words are represented by links between nodes. Over the past two decades, advances in network science and graph theoretical methods have led to the development of computational semantic networks. This review provides a modern perspective on how computational semantic networks have proven to be useful tools to investigate the structure of semantic memory as well as search and retrieval processes within semantic memory, to ultimately model performance in a wide variety of cognitive tasks. Regarding representation, the review focuses on the distinctions and similarities between network-based (based on behavioral norms) approaches and more recent distributional (based on natural language corpora) semantic models, and the potential overlap between the two approaches. Capturing the type of relation between concepts appears to be particularly important in this modeling endeavor. Regarding processes, the review focuses on random walk models and the degree to which retrieval processes demand attention in pursuit of given task goals, which dovetails with the type of relation retrieved during tasks. Ultimately, this review provides a critical assessment of how the network perspective can be reconciled with distributional and machine-learning-based perspectives to meaning representation, and describes how cognitive network science provides a useful conceptual toolkit to probe both the structure and retrieval processes within semantic memory.