Who is sensitive to selection biases in inductive reasoning?

The samples of evidence we use to make inferences in everyday and formal settings are often subject to selection biases. Two property induction experiments examined group and individual sensitivity to one type of selection bias: sampling frames - causal constraints that only allow certain types of instances to be sampled. Group data from both experiments indicated that people were sensitive to the effects of such frames, showing narrower generalization when sample instances were selected because they shared a target property (property sampling) than when instances were sampled because they belonged to a particular group (category sampling). Group generalization patterns conformed to the predictions of a Bayesian model of property induction that incorporates a selective sampling mechanism. In each experiment, however, there was considerable individual variation, with a nontrivial minority showing little sensitivity to sampling frames. Experiment 2 examined correlates of frames sensitivity. A composite measure of working memory capacity predicted individual sensitivity to sampling frames. These results have important implications for current debates about people's ability to factor sample selection mechanisms into their inferences and for the development of formal models of inductive inference. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The Pandemic as a Portal: Reimagining Psychological Science as Truly Open and Inclusive.

Psychological science is at an inflection point: The COVID-19 pandemic has exacerbated inequalities that stem from our historically closed and exclusive culture. Meanwhile, reform efforts to change the future of our science are too narrow in focus to fully succeed. In this article, we call on psychological scientists-focusing specifically on those who use quantitative methods in the United States as one context for such conversations-to begin reimagining our discipline as fundamentally open and inclusive. First, we discuss whom our discipline was designed to serve and how this history produced the inequitable reward and support systems we see today. Second, we highlight how current institutional responses to address worsening inequalities are inadequate, as well as how our disciplinary perspective may both help and hinder our ability to craft effective solutions. Third, we take a hard look in the mirror at the disconnect between what we ostensibly value as a field and what we actually practice. Fourth and finally, we lead readers through a roadmap for reimagining psychological science in whatever roles and spaces they occupy, from an informal discussion group in a department to a formal strategic planning retreat at a scientific society.

Protection from uncertainty in the exploration/exploitation trade-off.

The exploration/exploitation trade-off (EE trade-off) describes how, when faced with several competing alternatives, decision-makers must often choose between a known good alternative (exploitation) and one or more unknown but potentially more rewarding alternatives (exploration). Prevailing theory on how humans perform the EE trade-off states that uncertainty is a major motivator for exploration: the more uncertain the environment, the more exploration that will occur. The current article examines whether exploratory behavior in both choice and attention may be impacted differently depending on whether uncertainty is onset suddenly (unexpected uncertainty), or more slowly (expected uncertainty). It is shown that when uncertainty was expected, participants tended to explore less with their choices, but not their attention, than when it was unexpected. Crucially, the impact of this "protection from uncertainty" on exploration only occurred when participants had an opportunity to learn the structure of the task before experiencing uncertainty. This suggests that the interaction between uncertainty and exploration is more nuanced than simply more uncertainty leading to more exploration, and that attention and choice behavior may index separate aspects of the EE trade-off. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

The case for formal methodology in scientific reform.

Current attempts at methodological reform in sciences come in response to an overall lack of rigor in methodological and scientific practices in experimental sciences. However, most methodological reform attempts suffer from similar mistakes and over-generalizations to the ones they aim to address. We argue that this can be attributed in part to lack of formalism and first principles. Considering the costs of allowing false claims to become canonized, we argue for formal statistical rigor and scientific nuance in methodological reform. To attain this rigor and nuance, we propose a five-step formal approach for solving methodological problems. To illustrate the use and benefits of such formalism, we present a formal statistical analysis of three popular claims in the metascientific literature: (i) that reproducibility is the cornerstone of science; (ii) that data must not be used twice in any analysis; and (iii) that exploratory projects imply poor statistical practice. We show how our formal approach can inform and shape debates about such methodological claims.

If Mathematical Psychology Did Not Exist We Might Need to Invent It: A Comment on Theory Building in Psychology.

It is commonplace, when discussing the subject of psychological theory, to write articles from the assumption that psychology differs from the physical sciences in that we have no theories that would support cumulative, incremental science. In this brief article I discuss one counterexample: Shepard's law of generalization and the various Bayesian extensions that it inspired over the past 3 decades. Using Shepard's law as a running example, I argue that psychological theory building is not a statistical problem, mathematical formalism is beneficial to theory, measurement and theory have a complex relationship, rewriting old theory can yield new insights, and theory growth can drive empirical work. Although I generally suggest that the tools of mathematical psychology are valuable to psychological theorists, I also comment on some limitations to this approach.

Visual and Affective Multimodal Models of Word Meaning in Language and Mind.

One of the main limitations of natural language-based approaches to meaning is that they do not incorporate multimodal representations the way humans do. In this study, we evaluate how well different kinds of models account for people's representations of both concrete and abstract concepts. The models we compare include unimodal distributional linguistic models as well as multimodal models which combine linguistic with perceptual or affective information. There are two types of linguistic models: those based on text corpora and those derived from word association data. We present two new studies and a reanalysis of a series of previous studies. The studies demonstrate that both visual and affective multimodal models better capture behavior that reflects human representations than unimodal linguistic models. The size of the multimodal advantage depends on the nature of semantic representations involved, and it is especially pronounced for basic-level concepts that belong to the same superordinate category. Additional visual and affective features improve the accuracy of linguistic models based on text corpora more than those based on word associations; this suggests systematic qualitative differences between what information is encoded in natural language versus what information is reflected in word associations. Altogether, our work presents new evidence that multimodal information is important for capturing both abstract and concrete words and that fully representing word meaning requires more than purely linguistic information. Implications for both embodied and distributional views of semantic representation are discussed.

Skilled bandits: Learning to choose in a reactive world.

In uncertain environments we must balance our need to gather information with our desire to reap rewards by exploiting current knowledge. Achieving this balance is further complicated in reactive environments where actions produce long-lasting change to the system. In four experiments, we investigate how people learn to make effective decisions from experience in a dynamic multiarmed bandit task. In contrast to the typical exploitation-dependent diminishing rewards found in previous studies, options were framed as skills that developed greater rewards the more they were chosen. In Experiment 1, we provide a proof of concept, and in Experiments 2-4 we explore the boundaries of participants' sensitivity to reactive dynamics. Our results suggest that most individuals can learn effective strategies for coping with these reactive environments. A two-part comparison of several competing psychological models supports several conclusions: (a) a sizable minority of individuals learned that their environment was reactive, (b) evidence suggests several distinct groups of individuals employed unique decision strategies, and (c) testing models with the simulation method reveals qualitative misfits that motivate future theory development. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Adding Types, But Not Tokens, Affects Property Induction.

The extent to which we generalize a novel property from a sample of familiar instances to novel instances depends on the sample composition. Previous property induction experiments have only used samples consisting of novel types (unique entities). Because real-world evidence samples often contain redundant tokens (repetitions of the same entity), we studied the effects on property induction of adding types and tokens to an observed sample. In Experiments 1-3, we presented participants with a sample of birds or flowers known to have a novel property and probed whether this property generalized to novel items varying in similarity to the initial sample. Increasing the number of novel types (e.g., new birds with the target property) in a sample produced tightening, promoting property generalization to highly similar stimuli but decreasing generalization to less similar stimuli. On the other hand, increasing the number of tokens (e.g., repeated presentations of the same bird with the target property) had little effect on generalization. Experiment 4 showed that repeated tokens are encoded and can benefit recognition, but appear to be given little weight when inferring property generalization. We modified an existing Bayesian model of induction (Navarro, Dry, & Lee, 2012) to account for both the information added by new types and the discounting of information conveyed by tokens.

Selective sampling and inductive inference: Drawing inferences based on observed and missing evidence.

We propose and test a Bayesian model of property induction with evidence that has been selectively sampled leading to "censoring" or exclusion of potentially relevant data. A core model prediction is that identical evidence samples can lead to different patterns of inductive inference depending on the censoring mechanisms that cause some instances to be excluded. This prediction was confirmed in four experiments examining property induction following exposure to identical samples that were subject to different sampling frames. Each experiment found narrower generalization of a novel property when the sample instances were selected because they shared a common property (property sampling) than when they were selected because they belonged to the same category (category sampling). In line with model predictions, sampling frame effects were moderated by the addition of explicit negative evidence (Experiment 1), sample size (Experiment 2) and category base rates (Experiments 3-4). These data show that reasoners are sensitive to constraints on the sampling process when making property inferences; they consider both the observed evidence and the reasons why certain types of evidence has not been observed.

Sample size, number of categories and sampling assumptions: Exploring some differences between categorization and generalization.

Categorization and generalization are fundamentally related inference problems. Yet leading computational models of categorization (as exemplified by, e.g., Nosofsky, 1986) and generalization (as exemplified by, e.g., Tenenbaum and Griffiths, 2001) make qualitatively different predictions about how inference should change as a function of the number of items. Assuming all else is equal, categorization models predict that increasing the number of items in a category increases the chance of assigning a new item to that category; generalization models predict a decrease, or category tightening with additional exemplars. This paper investigates this discrepancy, showing that people do indeed perform qualitatively differently in categorization and generalization tasks even when all superficial elements of the task are kept constant. Furthermore, the effect of category frequency on generalization is moderated by assumptions about how the items are sampled. We show that neither model naturally accounts for the pattern of behavior across both categorization and generalization tasks, and discuss theoretical extensions of these frameworks to account for the importance of category frequency and sampling assumptions.

Do Additional Features Help or Hurt Category Learning? The Curse of Dimensionality in Human Learners.

The curse of dimensionality, which has been widely studied in statistics and machine learning, occurs when additional features cause the size of the feature space to grow so quickly that learning classification rules becomes increasingly difficult. How do people overcome the curse of dimensionality when acquiring real-world categories that have many different features? Here we investigate the possibility that the structure of categories can help. We show that when categories follow a family resemblance structure, people are unaffected by the presence of additional features in learning. However, when categories are based on a single feature, they fall prey to the curse, and having additional irrelevant features hurts performance. We compare and contrast these results to three different computational models to show that a model with limited computational capacity best captures human performance across almost all of the conditions in both experiments.

The diversity effect in inductive reasoning depends on sampling assumptions.

A key phenomenon in inductive reasoning is the diversity effect, whereby a novel property is more likely to be generalized when it is shared by an evidence sample composed of diverse instances than a sample composed of similar instances. We outline a Bayesian model and an experimental study that show that the diversity effect depends on the assumption that samples of evidence were selected by a helpful agent (strong sampling). Inductive arguments with premises containing either diverse or nondiverse evidence samples were presented under different sampling conditions, where instructions and filler items indicated that the samples were selected intentionally (strong sampling) or randomly (weak sampling). A robust diversity effect was found under strong sampling, but was attenuated under weak sampling. As predicted by our Bayesian model, the largest effect of sampling was on arguments with nondiverse evidence, where strong sampling led to more restricted generalization than weak sampling. These results show that the characteristics of evidence that are deemed relevant to an inductive reasoning problem depend on beliefs about how the evidence was generated.

The "Small World of Words" English word association norms for over 12,000 cue words.

Word associations have been used widely in psychology, but the validity of their application strongly depends on the number of cues included in the study and the extent to which they probe all associations known by an individual. In this work, we address both issues by introducing a new English word association dataset. We describe the collection of word associations for over 12,000 cue words, currently the largest such English-language resource in the world. Our procedure allowed subjects to provide multiple responses for each cue, which permits us to measure weak associations. We evaluate the utility of the dataset in several different contexts, including lexical decision and semantic categorization. We also show that measures based on a mechanism of spreading activation derived from this new resource are highly predictive of direct judgments of similarity. Finally, a comparison with existing English word association sets further highlights systematic improvements provided through these new norms.

Negative evidence and inductive reasoning in generalization of associative learning.

When generalizing properties from known to novel instances, both positive evidence (instances known to possess a property) and negative evidence (instances known not to possess a property) must be integrated. The current study compared generalization based on positive evidence alone against a mixture of positive evidence and perceptually dissimilar negative evidence in an interdimensional discrimination procedure. In 2 experiments, we compared generalization following training with a single positive stimulus (that predicted shock) against groups where an additional negative stimulus (that did not predict shock) was presented in a causal judgment (Experiment 1) and a fear conditioning (Experiment 2) procedure. In contrast to animal conditioning studies, we found that adding a "distant" negative stimulus resulted in an overall increase in generalization to stimuli varying on the dimension of the positive stimulus, consistent with the inductive reasoning literature. We show that this key qualitative result can be simulated by a Bayesian model that incorporates helpful sampling assumptions. Our results suggest that similar processes underlie generalization in inductive reasoning and associative learning tasks. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

When Extremists Win: Cultural Transmission Via Iterated Learning When Populations Are Heterogeneous.

How does the process of information transmission affect the cultural or linguistic products that emerge? This question is often studied experimentally and computationally via iterated learning, a procedure in which participants learn from previous participants in a chain. Iterated learning is a powerful tool because, when all participants share the same priors, the stationary distributions of the iterated learning chains reveal those priors. In many situations, however, it is unreasonable to assume that all participants share the same prior beliefs. We present four simulation studies and one experiment demonstrating that when the population of learners is heterogeneous, the behavior of an iterated learning chain can be unpredictable and is often systematically distorted by the learners with the most extreme biases. This results in group-level outcomes that reflect neither the behavior of any individuals within the population nor the overall population average. We discuss implications for the use of iterated learning as a methodological tool as well as for the processes that might have shaped cultural and linguistic evolution in the real world.

What do the experts know? Calibration, precision, and the wisdom of crowds among forensic handwriting experts.

Forensic handwriting examiners currently testify to the origin of questioned handwriting for legal purposes. However, forensic scientists are increasingly being encouraged to assign probabilities to their observations in the form of a likelihood ratio. This study is the first to examine whether handwriting experts are able to estimate the frequency of US handwriting features more accurately than novices. The results indicate that the absolute error for experts was lower than novices, but the size of the effect is modest, and the overall error rate even for experts is large enough as to raise questions about whether their estimates can be sufficiently trustworthy for presentation in courts. When errors are separated into effects caused by miscalibration and those caused by imprecision, we find systematic differences between individuals. Finally, we consider several ways of aggregating predictions from multiple experts, suggesting that quite substantial improvements in expert predictions are possible when a suitable aggregation method is used.