Don't let perfect be the enemy of better: In defense of unparameterized megastudies.

The target article argues researchers should be more ambitious, designing studies that systematically and comprehensively explore the space of possible experiments in one fell swoop. We argue that while "systematic" is rarely achievable, "comprehensive" is often enough. Critically, the recent popularization of massive online experiments shows that comprehensive studies are achievable for most cognitive and behavioral research questions.

Neither neural networks nor the language-of-thought alone make a complete game.

Cognitive science has evolved since early disputes between radical empiricism and radical nativism. The authors are reacting to the revival of radical empiricism spurred by recent successes in deep neural network (NN) models. We agree that language-like mental representations (language-of-thoughts [LoTs]) are part of the best game in town, but they cannot be understood independent of the other players.

Cognitive control across the lifespan: Congruency effects reveal divergent developmental trajectories.

The Simon, Stroop, and Eriksen flanker tasks are commonly used to assess cognitive control across the lifespan. However, it remains unclear whether these three tasks in fact measure the same cognitive abilities and in the same proportion. We take a developmental approach to this question: if the Simon, Stroop, and flanker tasks all roughly measure the same capacity, they should show similar patterns of age-related change. We present data from two massive online cross-sectional studies: Study 1 included 9,585 native English speakers between 10 and 80 years of age who completed the Simon and Stroop tasks, and Study 2 included 13,448 English speakers between 10 and 79 years of age who completed the flanker task. Of the three tasks, only the flanker task revealed an inverted U-shaped developmental trajectory, with performance improving until approximately 23 years of age and declining starting around 40 years of age. Performance on the Simon and Stroop tasks peaked around 34 and 26 years of age, respectively, and did not decline significantly in later life, though it is possible that age-related declines would be observed with more difficult versions of the tasks. Although the Simon and Stroop tasks are commonly interpreted to target similar underlying processes, we observed near zero correlations between the congruency effects observed in each task in terms of both accuracy and response time. We discuss these results in light of recent debates regarding the suitability of these tasks for assessing developmental and individual differences in cognitive control. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Contrastive machine learning reveals the structure of neuroanatomical variation within autism.

Autism spectrum disorder (ASD) is highly heterogeneous. Identifying systematic individual differences in neuroanatomy could inform diagnosis and personalized interventions. The challenge is that these differences are entangled with variation because of other causes: individual differences unrelated to ASD and measurement artifacts. We used contrastive deep learning to disentangle ASD-specific neuroanatomical variation from variation shared with typical control participants. ASD-specific variation correlated with individual differences in symptoms. The structure of this ASD-specific variation also addresses a long-standing debate about the nature of ASD: At least in terms of neuroanatomy, individuals do not cluster into distinct subtypes; instead, they are organized along continuous dimensions that affect distinct sets of regions.

More evidence from over 1.1 million subjects that the critical period for syntax closes in late adolescence.

The ability to attain native-like proficiency of a second language is heavily dependent on the age at which learning begins. However, the exact properties of this phenomenon remain unclear, and the literature is divided. Recently, Hartshorne, Tenenbaum, & Pinker presented a novel computational analysis of over 600,000 subjects, estimating that the ability to learn syntax drops at 17.4 years of age [Hartshorne, J. K., Tenenbaum, J. B., & Pinker, S. (2018). A critical period for second language acquisition: Evidence from 2/3 million English speakers. Cognition, 177, 263-277]. However, the novelty of the dataset and analyses raises questions and suggests caution [Frank, M. C. (2018). With great data comes great (theoretical) opportunity. Trends in cognitive sciences, 22(8), 669-671]. In the present paper, we address several such concerns by employing improved psychometric measurement, calculating confidence intervals, and considering alternative models. We also present data from an additional 466,607 subjects. The results support the prior report of a sharp decline in the ability to learn syntax, commencing at the tail end of adolescence.

Universality and diversity in human song.

What is universal about music, and what varies? We built a corpus of ethnographic text on musical behavior from a representative sample of the world's societies, as well as a discography of audio recordings. The ethnographic corpus reveals that music (including songs with words) appears in every society observed; that music varies along three dimensions (formality, arousal, religiosity), more within societies than across them; and that music is associated with certain behavioral contexts such as infant care, healing, dance, and love. The discography-analyzed through machine summaries, amateur and expert listener ratings, and manual transcriptions-reveals that acoustic features of songs predict their primary behavioral context; that tonality is widespread, perhaps universal; that music varies in rhythmic and melodic complexity; and that elements of melodies and rhythms found worldwide follow power laws.

The effect of working memory maintenance on long-term memory.

Initially inspired by the Atkinson and Shiffrin model, researchers have spent a half century investigating whether actively maintaining an item in working memory (WM) leads to improved subsequent long-term memory (LTM). Empirical results have been inconsistent, and thus the answer to the question remains unclear. We present evidence from 13 new experiments as well as a meta-analysis of 61 published experiments. Both the new experiments and meta-analysis show clear evidence that increased WM maintenance of a stimulus leads to superior recognition for that stimulus in subsequent LTM tests. This effect appears robust across a variety of experimental design parameters, suggesting that the variability in prior results in the literature is probably due to low power and random chance. The results support theories on which there is a close link between WM and LTM mechanisms, while challenging claims that this relationship is specific to verbal memory and evolved to support language acquisition.

A thousand studies for the price of one: Accelerating psychological science with Pushkin.

Half of the world's population has internet access. In principle, researchers are no longer limited to subjects they can recruit into the laboratory. Any study that can be run on a computer or mobile device can be run with nearly any demographic anywhere in the world, and in large numbers. This has allowed scientists to effectively run hundreds of experiments at once. Despite their transformative power, such studies remain rare for practical reasons: the need for sophisticated software, the difficulty of recruiting so many subjects, and a lack of research paradigms that make effective use of their large amounts of data, due to such realities as that they require sophisticated software in order to run effectively. We present Pushkin: an open-source platform for designing and conducting massive experiments over the internet. Pushkin allows for a wide range of behavioral paradigms, through integration with the intuitive and flexible jsPsych experiment engine. It also addresses the basic technical challenges associated with massive, worldwide studies, including auto-scaling, extensibility, machine-assisted experimental design, multisession studies, and data security.

Contrast and entailment: Abstract logical relations constrain how 2- and 3-year-old children interpret unknown numbers.

Do children understand how different numbers are related before they associate them with specific cardinalities? We explored how children rely on two abstract relations - contrast and entailment - to reason about the meanings of 'unknown' number words. Previous studies argue that, because children give variable amounts when asked to give an unknown number, all unknown numbers begin with an existential meaning akin to some. In Experiment 1, we tested an alternative hypothesis, that because numbers belong to a scale of contrasting alternatives, children assign them a meaning distinct from some. In the "Don't Give-a-Number task", children were shown three kinds of fruit (apples, bananas, strawberries), and asked to not give either some or a number of one kind (e.g. Give everything, but not [some/five] bananas). While children tended to give zero bananas when asked to not give some, they gave positive amounts when asked to not give numbers. This suggests that contrast - plus knowledge of a number's membership in a count list - enables children to differentiate the meanings of unknown number words from the meaning of some. Experiment 2 tested whether children's interpretation of unknown numbers is further constrained by understanding numerical entailment relations - that if someone, e.g. has three, they thereby also have two, but if they do not have three, they also do not have four. On critical trials, children saw two characters with different quantities of fish, two apart (e.g. 2 vs. 4), and were asked about the number in-between - who either has or doesn't have, e.g. three. Children picked the larger quantity for the affirmative, and the smaller for the negative prompts even when all the numbers were unknown, suggesting that they understood that, whatever three means, a larger quantity is more likely to contain that many, and a smaller quantity is more likely not to. We conclude by discussing how contrast and entailment could help children scaffold the exact meanings of unknown number words.

A critical period for second language acquisition: Evidence from 2/3 million English speakers.

Children learn language more easily than adults, though when and why this ability declines have been obscure for both empirical reasons (underpowered studies) and conceptual reasons (measuring the ultimate attainment of learners who started at different ages cannot by itself reveal changes in underlying learning ability). We address both limitations with a dataset of unprecedented size (669,498 native and non-native English speakers) and a computational model that estimates the trajectory of underlying learning ability by disentangling current age, age at first exposure, and years of experience. This allows us to provide the first direct estimate of how grammar-learning ability changes with age, finding that it is preserved almost to the crux of adulthood (17.4 years old) and then declines steadily. This finding held not only for "difficult" syntactic phenomena but also for "easy" syntactic phenomena that are normally mastered early in acquisition. The results support the existence of a sharply-defined critical period for language acquisition, but the age of offset is much later than previously speculated. The size of the dataset also provides novel insight into several other outstanding questions in language acquisition.

Psych verbs, the linking problem, and the acquisition of language.

In acquiring language, children must learn to appropriately place the different participants of an event (e.g., causal agent, affected entity) into the correct syntactic positions (e.g., subject, object) so that listeners will know who did what to whom. While many of these mappings can be characterized by broad generalizations, both within and across languages (e.g., semantic agents tend to be mapped onto syntactic subjects), not all verbs fit neatly into these generalizations. One particularly striking example is verbs of psychological state: The experiencer of the state can appear as either the subject (Agnes fears/hates/loves Bartholomew) or the direct object (Agnes frightens/angers/delights Bartholomew). The present studies explore whether this apparent variability in subject/object mapping may actually result from differences in these verbs' underlying meanings. Specifically, we suggest that verbs like fear describe a habitual attitude towards some entity whereas verbs like frighten describe an externally caused emotional episode. We find that this distinction systematically characterizes verbs in English, Mandarin, and Korean. This pattern is generalized to novel verbs by adults in English, Japanese, and Russian, and even by English-speaking children who are just beginning to acquire psych verbs. This results support a broad role for systematic mappings between semantics and syntax in language acquisition.

Comment: Acquiring metaphors.

Lakoff (current issue) describes an account of conceptual representation based in part on metaphor. Though promising, this account faces several challenges with respect to learning and development.

Response to Comment on "Estimating the reproducibility of psychological science".

Gilbert et al. conclude that evidence from the Open Science Collaboration's Reproducibility Project: Psychology indicates high reproducibility, given the study methodology. Their very optimistic assessment is limited by statistical misconceptions and by causal inferences from selectively interpreted, correlational data. Using the Reproducibility Project: Psychology data, both optimistic and pessimistic conclusions about reproducibility are possible, and neither are yet warranted.

The Causes and Consequences Explicit in Verbs.

Interpretation of a pronoun in one clause can be systematically affected by the verb in the previous clause. Compare Archibald angered Bartholomew because he… (he=Archibald) with Archibald criticized Bartholomew because he… (he=Bartholomew). While it is clear that meaning plays a critical role, it is unclear whether that meaning is directly encoded in the verb or, alternatively, inferred from world knowledge. We report evidence favoring the former account. We elicited pronoun biases for 502 verbs from seven Levin verb classes in two discourse contexts (implicit causality and implicit consequentiality), showing that in both contexts, verb class reliably predicts pronoun bias. These results confirm and extend recent findings about implicit causality and represent the first such study for implicit consequentiality. We discuss these findings in the context of recent work in semantics, and also develop a new, probabilistic generative account of pronoun interpretation.

The Neural Computation of Scalar Implicature.

Language comprehension involves not only constructing the literal meaning of a sentence but also going beyond the literal meaning to infer what was meant but not said. One widely-studied test case is scalar implicature: The inference that, e.g., Sally ate some of the cookies implies she did not eat all of them. Research is mixed on whether this is due to a rote, grammaticalized procedure or instead a complex, contextualized inference. We find that in sentences like If Sally ate some of the cookies, then the rest are on the counter, that the rest triggers a late, sustained positivity relative to Sally ate some of the cookies, and the rest are on the counter. This is consistent with behavioral results and linguistic theory suggesting that the former sentence does not trigger a scalar implicature. This motivates a view on which scalar implicature is contextualized but dependent on grammatical structure.

When does cognitive functioning peak? The asynchronous rise and fall of different cognitive abilities across the life span.

Understanding how and when cognitive change occurs over the life span is a prerequisite for understanding normal and abnormal development and aging. Most studies of cognitive change are constrained, however, in their ability to detect subtle, but theoretically informative life-span changes, as they rely on either comparing broad age groups or sparse sampling across the age range. Here, we present convergent evidence from 48,537 online participants and a comprehensive analysis of normative data from standardized IQ and memory tests. Our results reveal considerable heterogeneity in when cognitive abilities peak: Some abilities peak and begin to decline around high school graduation; some abilities plateau in early adulthood, beginning to decline in subjects' 30s; and still others do not peak until subjects reach their 40s or later. These findings motivate a nuanced theory of maturation and age-related decline, in which multiple, dissociable factors differentially affect different domains of cognition.

Development of the first-mention bias.

In many contexts, pronouns are interpreted as referring to the character mentioned first in the previous sentence, an effect called the 'first-mention bias'. While adults can rapidly use the first-mention bias to guide pronoun interpretation, it is unclear when this bias emerges during development. Curiously, experiments with children between two and three years old show successful use of order of mention, while experiments with older children (four to five years old) do not. While this could suggest U-shaped development, it could also reflect differences in the methodologies employed. We show that children can indeed use first-mention information, but do so too slowly to have been detected in previous work reporting null results. Comparison across the present and previously published studies suggests that the speed at which children deploy first-mention information increases greatly during the preschool years.

Love is hard to understand: the relationship between transitivity and caused events in the acquisition of emotion verbs.

Famously, dog bites man is trivia whereas man bites dog is news. This illustrates not just a fact about the world but about language: to know who did what to whom, we must correctly identify the mapping between semantic role and syntactic position. These mappings are typically predictable, and previous work demonstrates that young children are sensitive to these patterns and so could use them in acquisition. However, there is only limited and mixed evidence that children do use this information to guide acquisition outside of the laboratory. We find that children understand emotion verbs which follow the canonical CAUSE-VERB-PATIENT pattern (Mary frightened/delighted John) earlier than those which do not (Mary feared/liked John), despite the latter's higher frequency, suggesting children's generalization of the mapping between causativity and transitivity is broad and active in acquisition.

Are implicit causality pronoun resolution biases consistent across languages and cultures?

The referent of a nonreflexive pronoun depends on context, but the nature of these contextual restrictions is controversial. For instance, in causal dependent clauses, the preferred referent of a pronoun varies systematically with the verb in the main clause (Sally frightens Mary because she … vs. Sally loves Mary because she …). Several theories claim that verbs with similar meanings across languages should show similar pronoun resolution effects, but these claims run contrary to recent analyses on which much of linguistic and nonlinguistic cognition is susceptible to cross-cultural variation, and in fact there is little data in the literature to decide the question one way or another. Analysis of data in eight languages representing four historically unrelated language families reveals consistent pronoun resolution biases for emotion verbs, suggesting that the information upon which implicit causality pronoun resolution biases are derived is stable across languages and cultures.

Tracking replicability as a method of post-publication open evaluation.

Recent reports have suggested that many published results are unreliable. To increase the reliability and accuracy of published papers, multiple changes have been proposed, such as changes in statistical methods. We support such reforms. However, we believe that the incentive structure of scientific publishing must change for such reforms to be successful. Under the current system, the quality of individual scientists is judged on the basis of their number of publications and citations, with journals similarly judged via numbers of citations. Neither of these measures takes into account the replicability of the published findings, as false or controversial results are often particularly widely cited. We propose tracking replications as a means of post-publication evaluation, both to help researchers identify reliable findings and to incentivize the publication of reliable results. Tracking replications requires a database linking published studies that replicate one another. As any such database is limited by the number of replication attempts published, we propose establishing an open-access journal dedicated to publishing replication attempts. Data quality of both the database and the affiliated journal would be ensured through a combination of crowd-sourcing and peer review. As reports in the database are aggregated, ultimately it will be possible to calculate replicability scores, which may be used alongside citation counts to evaluate the quality of work published in individual journals. In this paper, we lay out a detailed description of how this system could be implemented, including mechanisms for compiling the information, ensuring data quality, and incentivizing the research community to participate.