Evaluating language models for mathematics through interactions.

There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of evaluating LLMs relies on static pairs of inputs and outputs; this is insufficient for making an informed decision about which LLMs are best to use in an interactive setting, and how that varies by setting. Static assessment therefore limits how we understand language model capabilities. We introduce CheckMate, an adaptable prototype platform for humans to interact with and evaluate LLMs. We conduct a study with CheckMate to evaluate three language models (InstructGPT, ChatGPT, and GPT-4) as assistants in proving undergraduate-level mathematics, with a mixed cohort of participants from undergraduate students to professors of mathematics. We release the resulting interaction and rating dataset, MathConverse. By analyzing MathConverse, we derive a taxonomy of human query behaviors and uncover that despite a generally positive correlation, there are notable instances of divergence between correctness and perceived helpfulness in LLM generations, among other findings. Further, we garner a more granular understanding of GPT-4 mathematical problem-solving through a series of case studies, contributed by experienced mathematicians. We conclude with actionable takeaways for ML practitioners and mathematicians: models that communicate uncertainty, respond well to user corrections, and can provide a concise rationale for their recommendations, may constitute better assistants. Humans should inspect LLM output carefully given their current shortcomings and potential for surprising fallibility.

Neural Correlates of Analogical Reasoning on Syntactic Patterns.

Analogical reasoning is central to thought and learning. However, previous neuroscience studies have focused mainly on neural substrates for visuospatial and semantic analogies. There has not yet been research on the neural correlates of analogical reasoning on syntactic patterns generated by the syntactic rules, a key feature of human language faculty. The present investigation took an initial step to address this paucity. Twenty-four participants, whose brain activity was monitored by fMRI, engaged in first-order and second-order relational judgments of syntactic patterns as well as simple and complex working memory tasks. After scanning, participants rated the difficulty of each step during analogical reasoning; these ratings were related to signal intensities in activated regions of interest using Spearman correlation analyses. After prior research, differences in activation levels during second-order and first-order relational judgments were taken as evidence of analogical reasoning. These analyses showed that analogical reasoning on syntactic patterns recruited brain regions consistent with those supporting visuospatial and semantic analogies, including the anterior and posterior parts of the left middle frontal gyrus, anatomically corresponding to the left rostrolateral pFC and the left dorsolateral pFC. The correlation results further revealed that the posterior middle frontal gyrus might be involved in analogical access and mapping with syntactic patterns. Our study is the first to investigate the process of analogical reasoning on syntactic patterns at the neurobiological level and provide evidence of the specific functional roles of related regions during subprocesses of analogical reasoning.

Neural mechanisms distinguishing two types of cooperative problem-solving approaches: An fNIRS hyperscanning study.

Collaborative cooperation (CC) and division of labor cooperation (DLC) are two prevalent forms of cooperative problem-solving approaches in daily life. Despite extensive research on the neural mechanisms underlying cooperative problem-solving approaches, a notable gap exists between the neural processes that support CC and DLC. The present study utilized a functional near-infrared spectroscopy (fNIRS) hyperscanning technique along with a classic cooperative tangram puzzle task to investigate the neural mechanisms engaged by both friends and stranger dyads during CC versus DLC. The key findings of this study were as follows: (1) Dyads exhibited superior behavioral performance in the DLC task than in the CC task. The CC task bolstered intra-brain functional connectivity and inter-brain synchrony (IBS) in regions linked to the mirror neuron system (MNS), spatial perception (SP) and cognitive control. (2) Friend dyads showed stronger IBS in brain regions associated with the MNS than stranger dyads. (3) Perspective-taking predicted not only dyads' behavioral performance in the CC task but also their IBS in brain regions associated with SP during the DLC task. Taken together, these findings elucidate the divergent behavioral performance and neural connection patterns between the two cooperative problem-solving approaches. This study provides novel insights into the various neurocognitive processes underlying flexible coordination strategies in real-world cooperative contexts.

Time course of EEG power during creative problem-solving with insight or remote thinking.

Problem-solving often requires creativity and is critical in everyday life. However, the neurocognitive mechanisms underlying creative problem-solving remain poorly understood. Two mechanisms have been highlighted: the formation of new connections among problem elements and insight solving, characterized by sudden realization of a solution. In this study, we investigated EEG activity during a modified version of the remote associates test, a classical insight problem task that requires finding a word connecting three unrelated words. This allowed us to explore the brain correlates associated with the semantic remoteness of connections (by varying the remoteness of the solution word across trials) and with insight solving (identified as a Eurêka moment reported by the participants). Semantic remoteness was associated with power increase in the alpha band (8-12 Hz) in a left parieto-temporal cluster, the beta band (13-30 Hz) in a right fronto-temporal cluster in the early phase of the task, and the theta band (3-7 Hz) in a bilateral frontal cluster just prior to participants' responses. Insight solving was associated with power increase preceding participants' responses in the alpha and gamma (31-60 Hz) bands in a left temporal cluster and the theta band in a frontal cluster. Source reconstructions revealed the brain regions associated with these clusters. Overall, our findings shed new light on some of the mechanisms involved in creative problem-solving.

Aha! and D'oh! experiences enhance learning for incidental information-new evidence supports the insight memory advantage.

Research on creative problem-solving finds that solutions achieved via spontaneous insight (i.e., Aha! moment) are better remembered than solutions reached without this sense of epiphany, referred to as an "insight memory advantage." We hypothesized that the insight memory advantage can spread to incidental information encoded in the moments surrounding insight as well. Participants (N = 291) were first given Rebus puzzles. After they indicated that they had found a solution, but before they could submit this solution, they were presented with scholastic facts that were incidental and unrelated to the problem at hand. Participants indicated whether they reached the solution via either insight or a step-by-step analysis. Memory results showed better performance for incidental scholastic facts presented when problem solving was accompanied by a spontaneous (Aha! experience) and induced (D'oh! experience) insight compared with solutions reached with analysis. This finding suggests that the memory advantage for problems solved via insight spreads to other unrelated information encoded in close temporal proximity and has implications for novel techniques to enhance learning in educational settings.

Sex differences in visuospatial cognition- a female advantage in jigsaw puzzle solving.

Mentally visualizing objects, understanding relationships between two- or three- dimensional objects, and manipulating objects in space are some examples of visuospatial abilities. Numerous studies have shown that male participants outperform female participants in visuospatial tasks, particularly in mental rotation. One exception is solving jigsaw puzzles. Performance by seven- to eight-year-old girls was found to be superior to that of boys of the same age (Kocijan et al. 2017). No study, however, has confirmed this finding in an adult population, where sex differences are often detectable. Seventy-nine young adult participants were given four different jigsaw puzzles and the Shepard and Metzler mental rotation test (MRT) with two main goals: First, to investigate possible sex differences in jigsaw puzzle solving, and second, to explore a potential relationship between mental rotation and jigsaw puzzle solving. We hypothesized that female participants would outperform males in the jigsaw puzzles but males would outperform females in the MRT. The findings confirmed this hypothesis. Notably, the male performance in jigsaw puzzle solving was attributed to their sex and mediated by their higher MRT scores. These results yielded two key insights. First, they indicate a dissociation between these two visuospatial abilities, jigsaw puzzle solving and mental rotation; and second, female and male participants capitalize on their distinct cognitive strengths when solving visuospatial tasks.

Eye movements reveal that young school children shift attention when solving additions and subtractions.

Adults shift their attention to the right or to the left along a spatial continuum when solving additions and subtractions, respectively. Studies suggest that these shifts not only support the exact computation of the results but also anticipatively narrow down the range of plausible answers when processing the operands. However, little is known on when and how these attentional shifts arise in childhood during the acquisition of arithmetic. Here, an eye-tracker with high spatio-temporal resolution was used to measure spontaneous eye movements, used as a proxy for attentional shifts, while children of 2nd (8 y-o; N = 50) and 4th (10 y-o; N = 48) Grade solved simple additions (e.g., 4+3) and subtractions (e.g., 3-2). Gaze patterns revealed horizontal and vertical attentional shifts in both groups. Critically, horizontal eye movements were observed in 4th Graders as soon as the first operand and the operator were presented and thus before the beginning of the exact computation. In 2nd Graders, attentional shifts were only observed after the presentation of the second operand just before the response was made. This demonstrates that spatial attention is recruited when children solve arithmetic problems, even in the early stages of learning mathematics. The time course of these attentional shifts suggests that with practice in arithmetic children start to use spatial attention to anticipatively guide the search for the answer and facilitate the implementation of solving procedures. RESEARCH HIGHLIGHTS: Additions and subtractions are associated to right and left attentional shifts in adults, but it is unknown when these mechanisms arise in childhood. Children of 8-10 years old solved single-digit additions and subtractions while looking at a blank screen. Eye movements showed that children of 8 years old already show spatial biases possibly to represent the response when knowing both operands. Children of 10 years old shift attention before knowing the second operand to anticipatively guide the search for plausible answers.

The involvement of decomposition and composition processes in restructuring during problem solving.

Decomposition of chunks has been widely accepted as a critical proxy of restructuring, but the role of composition in forming new representations has been largely neglected. This study aims to investigate the roles of both decomposition and composition processes in chunk restructuring, as well as their relationships with "aha" experiences during problem-solving. Participants were asked to move a part of a character to another character to create two new characters. Across three experiments, the characters to be decomposed or composed were varied in terms of tight or loose chunks. The results showed that decomposition or composition of tight chunks led to lower success rates, longer response times, and significantly stronger "Aha!" emotional experiences (mainly in terms of surprise and suddenness). This study provides evidence for the contribution of both decomposition and composition processes to restructuring in creative insight.

Cognitive processes that underlie mathematically gifted emergent bilinguals.

The purpose of this study was to determine those cognitive measures that increase the likelihood of identifying mathematically gifted students who are emerging bilinguals. Elementary school children (Grades 1, 2, and 3) were administered a battery of math, vocabulary, reading, and cognitive measures (short-term memory, inhibition, and working memory in their first language (L1: Spanish) and second language (L2: English). Multilevel polytomous logistic modeling compared mathematically gifted children with children who were average math achievers or low math achievers. The results indicated that cognitive parameters that included estimation and working memory in the L2 and problem-solving in the L1 were unique predictors that significantly influenced whether a child was categorized as gifted relative to average achievers. Relative to average achievers, L2 parameters (magnitude judgment) and English reading were significantly related to the identification of children with low math computation. The results are discussed in terms of a multidimensional model that taps domain-specific skills and general cognitive processes that increase the ability to correctly identify children who score in the gifted range in both their L1 and L2.

Investigating low intelligence stereotype threat in adults with developmental dyslexia.

Stereotype threat (ST) is a phenomenon that leads to decreased test performance and occurs when one deals with added pressure of being judged on the basis of stereotyped group membership. The ST effect has been previously investigated in many contexts but not in individuals with dyslexia who are often stereotyped as less intelligent. Prevalent use of intelligence tests in job selection processes and employment gap between people with dyslexia and those without warrants this investigation. Sixty-three participants (30 with dyslexia and 33 without dyslexia; mean age = 33.7; SD = 13.7; 47 F, 13 M, three non-binary) were asked to complete intelligence test typically used in selection processes. All participants were randomly assigned to one of three test instruction conditions: (1) they were told the test was diagnostic of their intelligence (ST triggering instruction); (2) test was a measure of their problem-solving skills (reduced threat); (3) or they were simply asked to take the test (control). Results showed that participants with dyslexia in ST condition performed poorer than those in other conditions and those in the same condition who did not have dyslexia. This study provides preliminary evidence for diminishing effects of ST in individuals with dyslexia.

The impact of problem domain on Bayesian inferences: A systematic investigation.

Sparse (and occasionally contradictory) evidence exists regarding the impact of domain on probabilistic updating, some of which suggests that Bayesian word problems with medical content may be especially challenging. The present research aims to address this gap in knowledge through three pre-registered online studies, which involved a total of 2,238 participants. Bayesian word problems were related to one of three domains: medical, daily-life, and abstract. In the first two cases, problems presented realistic content and plausible numerical information, while in the latter, problems contained explicitly imaginary elements. Problems across domains were matched in terms of all relevant statistical values and, as much as possible, wording. Studies 1 and 2 utilized the same set of problems, but different response elicitation methods (i.e., an open-ended and a multiple-choice question, respectively). Study 3 involved a larger number of participants per condition and a smaller set of problems to more thoroughly investigate the magnitude of differences between the domains. There was a generally low rate of correct responses (17.2%, 17.4%, and 14.3% in Studies 1, 2, and 3, respectively), consistent with accuracy levels commonly observed in the literature for this specific task with online samples. Nonetheless, a small but significant difference between domains was observed: participants' accuracy did not differ between medical and daily-life problems, while it was significantly higher in corresponding abstract problems. These results suggest that medical problems are not inherently more difficult to solve, but rather that performance is improved with abstract problems for which participants cannot draw from their background knowledge.

Task complexity moderates group synergy.

Complexity-defined in terms of the number of components and the nature of the interdependencies between them-is clearly a relevant feature of all tasks that groups perform. Yet the role that task complexity plays in determining group performance remains poorly understood, in part because no clear language exists to express complexity in a way that allows for straightforward comparisons across tasks. Here we avoid this analytical difficulty by identifying a class of tasks for which complexity can be varied systematically while keeping all other elements of the task unchanged. We then test the effects of task complexity in a preregistered two-phase experiment in which 1,200 individuals were evaluated on a series of tasks of varying complexity (phase 1) and then randomly assigned to solve similar tasks either in interacting groups or as independent individuals (phase 2). We find that interacting groups are as fast as the fastest individual and more efficient than the most efficient individual for complex tasks but not for simpler ones. Leveraging our highly granular digital data, we define and precisely measure group process losses and synergistic gains and show that the balance between the two switches signs at intermediate values of task complexity. Finally, we find that interacting groups generate more solutions more rapidly and explore the solution space more broadly than independent problem solvers, finding higher-quality solutions than all but the highest-scoring individuals.

Relationship Between Prone Skills and Motor-Based Problem-Solving Abilities in Full-Term and Preterm Infants During the First 6 Months of Life.

Motor experiences shape cognitive development in infancy, with the prone position being one such crucial motor experience in the first 6 months of life. Although the motor benefits of the prone position are well-documented, its influence on early cognitive abilities remains insufficiently explored. This study quantified the relationship between prone motor skills and motor-based problem-solving abilities in 48 full-term and preterm infants aged 3-6 months. Prone skills were assessed using the Alberta Infant Motor Scale's prone domain. The Assessment of Problem-Solving in Play was utilized to measure motor-based problem-solving by observing how motor actions were used to solve toys. Advanced prone motor skills were correlated with an increase in sophisticated exploration skills and a concurrent decline in lower order exploration skills in all infants, with correlations being stronger in preterm infants. Notably, a 1-point increase in prone skills was associated with a 1.3-point increase in total motor-based problem-solving abilities in all infants. Our findings provide preliminary evidence for the contribution of prone play to cognitive development in infants, prompting considerations for assessment and intervention strategies. Further research is needed to ascertain if the delayed acquisition of prone motor skills is indicative of poor early problem-solving abilities in preterm infants.

Peripheral vision and crowding in mental maze-solving.

Solving a maze effectively relies on both perception and cognition. Studying maze-solving behavior contributes to our knowledge about these important processes. Through psychophysical experiments and modeling simulations, we examine the role of peripheral vision, specifically visual crowding in the periphery, in mental maze-solving. Experiment 1 measured gaze patterns while varying maze complexity, revealing a direct relationship between visual complexity and maze-solving efficiency. Simulations of the maze-solving task using a peripheral vision model confirmed the observed crowding effects while making an intriguing prediction that saccades provide a conservative measure of how far ahead observers can perceive the path. Experiment 2 confirms that observers can judge whether a point lies on the path at considerably greater distances than their average saccade. Taken together, our findings demonstrate that peripheral vision plays a key role in mental maze-solving.

Question-asking as a mechanism of information seeking.

Ivancovsky et al. explore the relationship between curiosity and creativity, by suggesting they align through novelty-seeking mechanisms. We argue that a general mechanism linking both capacities together is question-asking: Curiosity drives question-asking that leads to creative problem solving. Yet, current findings from our lab suggest that question complexity relates to creativity, but not necessarily to curiosity, warranting further investigation.

Form perception is a cognitive correlate of the relation between subitizing ability and math performance.

"Subitizing" defines a phenomenon whereby approximately four items can be quickly and accurately processed. Studies have shown the close association between subitizing and math performance, however, the mechanism for the association remains unclear. The present study was conducted to investigate whether form perception assessed on a serial figure matching task is a potential non-numerical mechanism between subitizing ability and math performance. Three-hundred and seventy-three Chinese primary school students completed four kinds of dot comparison tasks, serial figure matching task, math performance tasks (including three arithmetic computation tasks and math word problem task), and other cognitive tasks as their general cognitive abilities were observed as covariates. A series of hierarchical regression analyses showed that after controlling for age, gender, nonverbal matrix reasoning, and visual tracking, subitizing comparison (subitizing vs. subitizing, subitizing vs. estimation) still contributed to simple addition or simple subtraction but not to complex subtraction ability or math word problem. After taking form perception as an additional control variable, the predictive power of different dot comparison conditions disappeared. A path model also showed that form perception fully mediates the relation between numerosity comparison (within and beyond the subitizing range) and arithmetic performance. These findings support the claim that form perception is a non-numerical cognitive correlate of the relation between subitizing ability and math performance (especially arithmetic computation).

Striatal Cholinergic Interneurons Are Required for Contending Strategy Selection While Solving Spatial Navigation Problems.

How do animals adopt a given behavioral strategy to solve a recurrent problem when several effective strategies are available to reach the goal? Here we provide evidence that striatal cholinergic interneurons (SCINs) modulate their activity when mice must select between different strategies with similar goal-reaching effectiveness. Using a cell type-specific transgenic murine system, we show that adult SCIN ablation impairs strategy selection in navigational tasks where a goal can be independently achieved by adopting an allocentric or egocentric strategy. SCIN-depleted mice learn to achieve the goal in these tasks, regardless of their appetitive or aversive nature, in a similar way as controls. However, they cannot shift away from their initially adopted strategies, as control mice do, as training progresses. Our results indicate that SCINs are required for shaping the probability function used for strategy selection as experience accumulates throughout training. Thus, SCINs may be critical for the resolution of cognitive conflicts emerging when several strategies compete for behavioral control while adapting to environmental demands. Our findings may increase our understanding about the emergence of perseverative/compulsive traits in neuropsychiatric disorders with a reported SCIN reduction, such as Tourette and Williams syndromes.SIGNIFICANCE STATEMENT Selecting the best suited strategy to solve a problem is vital. Accordingly, available strategies must be compared across multiple dimensions, such as goal attainment effectiveness, cost-benefit trade-off, and cognitive load. The striatum is involved in strategy selection when strategies clearly diverge in their goal attainment capacity; however, its role whenever several strategies can be used for goal reaching-therefore making selection dependent on additional strategy dimensions-remains poorly understood. Here, we show that striatal cholinergic interneurons can signal strategy competition. Furthermore, they are required to adopt a given strategy whenever strategies with similar goal attainment capacity compete for behavioral control. Our study suggests that striatal cholinergic dysfunction may result in anomalous resolution of problems whenever complex cognitive valuations are required.

Quantitative modelling demonstrates format-invariant representations of mathematical problems in the brain.

Mathematical problems can be described in either symbolic form or natural language. Previous studies have reported that activation overlaps exist for these two types of mathematical problems, but it is unclear whether they are based on similar brain representations. Furthermore, quantitative modelling of mathematical problem solving has yet to be attempted. In the present study, subjects underwent 3 h of functional magnetic resonance experiments involving math word and math expression problems, and a read word condition without any calculations was used as a control. To evaluate the brain representations of mathematical problems quantitatively, we constructed voxel-wise encoding models. Both intra- and cross-format encoding modelling significantly predicted brain activity predominantly in the left intraparietal sulcus (IPS), even after subtraction of the control condition. Representational similarity analysis and principal component analysis revealed that mathematical problems with different formats had similar cortical organization in the IPS. These findings support the idea that mathematical problems are represented in the brain in a format-invariant manner.

Arithmetic problem size modulates brain activations in females but not in males.

Numerous empirical studies have reported that males and females perform equally well in mathematical achievement. However, still to date, very limited is understood about the brain response profiles that are particularly characteristic of males and females when solving mathematical problems. The present study aimed to tackle this issue by manipulating arithmetic problem size to investigate functional significance using functional magnetic resonance imaging (fMRI) in young adults. Participants were instructed to complete two runs of simple calculation tasks with either large or small problem sizes. Behavioural results suggested that the performance did not differ between females and males. Neuroimaging data revealed that sex/gender-related patterns of problem size effect were found in the brain regions that are conventionally associated with arithmetic, including the left middle frontal gyrus (MFG), left intraparietal sulcus (IPS) and insula. Specifically, females demonstrated substantial brain responses of problem size effect in these regions, whereas males showed marginal effects. Moreover, the machine learning method implemented over the brain signal levels within these regions demonstrated that sex/gender is discriminable. These results showed sex/gender effects in the activating patterns varying as a function of the distinct math problem size, even in a simple calculation task. Accordingly, our findings suggested that females and males use two complementary brain resources to achieve equally successful performance levels and highlight the pivotal role of neuroimaging facilities in uncovering neural mechanisms that may not be behaviourally salient.

The causal mechanisms underlying analogical reasoning performance improvement by executive attention intervention.

Analogical reasoning is important for human. We have found that a short executive attention intervention improved analogical reasoning performance in healthy young adults. Nevertheless, previous electrophysiological evidence was limited for comprehensively characterizing the neural mechanisms underlying the improvement. And although we hypothesized that the intervention improved active inhibitory control and attention shift first and then relation integration, it is still unclear whether there are two sequential cognitive neural activities were indeed changed during analogical reasoning. In the present study, we combined hypothesis with multivariate pattern analysis (MVPA) to explore the effects of the intervention on electrophysiology. Results showed that in the resting state after the intervention, alpha and high gamma power and the functional connectivity between the anterior and middle in the alpha band could discriminate the experimental group from the active control group, respectively. These indicated that the intervention influenced the activity of multiple bands and the interaction of frontal and parietal regions. In the analogical reasoning, alpha, theta, and gamma activities could also fulfill such discrimination, and furthermore, they were sequential (alpha first, theta, and gamma later). These results directly supported our previous hypothesis. The present study deepens our understanding about how executive attention contributes to higher-order cognition.