Strength of Belief Guides Information Foraging.

Theories of how people value and search for information share the assumption that beliefs give rise to the perceived value of information. However, few studies have directly addressed the pre-search processes that influence information-foraging behavior. This experiment examined the influence of pre-search belief updating on the perceived value of information sources. A sample of college students completed a hypothesis-testing, medical-diagnosis task. The experiment used medical tests with equal objective informative value before unveiling a presenting symptom intended to alter the strength of belief in different disease hypotheses. The observed patterns of test selection suggest that changes in beliefs about disease hypotheses result in systematic and predictable changes in test preference-a notion we refer to as the principle of hypothesis-guided search. We also present a simulation of how pre-search processes (e.g., hypothesis generation and working memory capacity) and task variables (e.g., time pressure) influence subsequent information search.

Does constraining memory maintenance reduce visual search efficiency?

We examine whether constraining memory retrieval processes affects performance in a cued recall visual search task. In the visual search task, participants are first presented with a memory prompt followed by a search array. The memory prompt provides diagnostic information regarding a critical aspect of the target (its colour). We assume that upon the presentation of the memory prompt, participants retrieve and maintain hypotheses (i.e., potential target characteristics) in working memory in order to improve their search efficiency. By constraining retrieval through the manipulation of time pressure (Experiments 1A and 1B) or a concurrent working memory task (Experiments 2A, 2B, and 2C), we directly test the involvement of working memory in visual search. We find some evidence that visual search is less efficient under conditions in which participants were likely to be maintaining fewer hypotheses in working memory (Experiments 1A, 2A, and 2C), suggesting that the retrieval of representations from long-term memory into working memory can improve visual search. However, these results should be interpreted with caution, as the data from two experiments (Experiments 1B and 2B) did not lend support for this conclusion.

Order-constrained linear optimization.

Despite the fact that data and theories in the social, behavioural, and health sciences are often represented on an ordinal scale, there has been relatively little emphasis on modelling ordinal properties. The most common analytic framework used in psychological science is the general linear model, whose variants include ANOVA, MANOVA, and ordinary linear regression. While these methods are designed to provide the best fit to the metric properties of the data, they are not designed to maximally model ordinal properties. In this paper, we develop an order-constrained linear least-squares (OCLO) optimization algorithm that maximizes the linear least-squares fit to the data conditional on maximizing the ordinal fit based on Kendall's τ. The algorithm builds on the maximum rank correlation estimator (Han, 1987, Journal of Econometrics, 35, 303) and the general monotone model (Dougherty & Thomas, 2012, Psychological Review, 119, 321). Analyses of simulated data indicate that when modelling data that adhere to the assumptions of ordinary least squares, OCLO shows minimal bias, little increase in variance, and almost no loss in out-of-sample predictive accuracy. In contrast, under conditions in which data include a small number of extreme scores (fat-tailed distributions), OCLO shows less bias and variance, and substantially better out-of-sample predictive accuracy, even when the outliers are removed. We show that the advantages of OCLO over ordinary least squares in predicting new observations hold across a variety of scenarios in which researchers must decide to retain or eliminate extreme scores when fitting data.

Using a model of hypothesis generation to predict eye movements in a visual search task.

We used a model of hypothesis generation (called HyGene; Thomas, Dougherty, Sprenger, & Harbison, 2008) to make predictions regarding the deployment of attention (as assessed via eye movements) afforded by the cued recall of target characteristics before the onset of a search array. On each trial, while being eyetracked, participants were first presented with a memory prompt that was diagnostic regarding the target's color in a subsequently presented search array. We assume that the memory prompts led to the generation of hypotheses (i.e., potential target characteristics) from long-term memory into working memory to guide attentional processes and ocular-motor behavior. However, given that multiple hypotheses might be generated in response to a prompt, it has been unclear how the focal hypothesis (i.e., the hypothesis that exerts the most influence on search) affects search behavior. We tested two possibilities using first fixation data, with the assumption that the first item fixated within a search array was the focal hypothesis. We found that a model assuming that the first item generated into working memory guides overt attentional processes was most consistent with the data at both the aggregate and single-participant levels of analysis.

Reply to Rouder (2014): good frequentist properties raise confidence.

Established psychological results have been called into question by demonstrations that statistical significance is easy to achieve, even in the absence of an effect. One often-warned-against practice, choosing when to stop the experiment on the basis of the results, is guaranteed to produce significant results. In response to these demonstrations, Bayes factors have been proposed as an antidote to this practice, because they are invariant with respect to how an experiment was stopped. Should researchers only care about the resulting Bayes factor, without concern for how it was produced? Yu, Sprenger, Thomas, and Dougherty (2014) and Sanborn and Hills (2014) demonstrated that Bayes factors are sometimes strongly influenced by the stopping rules used. However, Rouder (2014) has provided a compelling demonstration that despite this influence, the evidence supplied by Bayes factors remains correct. Here we address why the ability to influence Bayes factors should still matter to researchers, despite the correctness of the evidence. We argue that good frequentist properties mean that results will more often agree with researchers' statistical intuitions, and good frequentist properties control the number of studies that will later be refuted. Both help raise confidence in psychological results.

Explicit awareness supports conditional visual search in the retrieval guidance paradigm.

In four experiments we explored whether participants would be able to use probabilistic prompts to simplify perceptually demanding visual search in a task we call the retrieval guidance paradigm. On each trial a memory prompt appeared prior to (and during) the search task and the diagnosticity of the prompt(s) was manipulated to provide complete, partial, or non-diagnostic information regarding the target's color on each trial (Experiments 1-3). In Experiment 1 we found that the more diagnostic prompts was associated with faster visual search performance. However, similar visual search behavior was observed in Experiment 2 when the diagnosticity of the prompts was eliminated, suggesting that participants in Experiment 1 were merely relying on base rate information to guide search and were not utilizing the prompts. In Experiment 3 participants were informed of the relationship between the prompts and the color of the target and this was associated with faster search performance relative to Experiment 1, suggesting that the participants were using the prompts to guide search. Additionally, in Experiment 3 a knowledge test was implemented and performance in this task was associated with qualitative differences in search behavior such that participants that were able to name the color(s) most associated with the prompts were faster to find the target than participants who were unable to do so. However, in Experiments 1-3 diagnosticity of the memory prompt was manipulated via base rate information, making it possible that participants were merely relying on base rate information to inform search in Experiment 3. In Experiment 4 we manipulated diagnosticity of the prompts without manipulating base rate information and found a similar pattern of results as Experiment 3. Together, the results emphasize the importance of base rate and diagnosticity information in visual search behavior. In the General discussion section we explore how a recent computational model of hypothesis generation (HyGene; Thomas, Dougherty, Sprenger, & Harbison, 2008), linking attention with long-term and working memory, accounts for the present results and provides a useful framework of cued recall visual search.

What counts as evidence for working memory training? Problems with correlated gains and dichotomization.

The question of whether computerized cognitive training leads to generalized improvements of intellectual abilities has been a popular, yet contentious, topic within both the psychological and neurocognitive literatures. Evidence for the effective transfer of cognitive training to nontrained measures of cognitive abilities is mixed, with some studies showing apparent successful transfer, while others have failed to obtain this effect. At the same time, several authors have made claims about both successful and unsuccessful transfer effects on the basis of a form of responder analysis, an analysis technique that shows that those who gain the most on training show the greatest gains on transfer tasks. Through a series of Monte Carlo experiments and mathematical analyses, we demonstrate that the apparent transfer effects observed through responder analysis are illusory and are independent of the effectiveness of cognitive training. We argue that responder analysis can be used neither to support nor to refute hypotheses related to whether cognitive training is a useful intervention to obtain generalized cognitive benefits. We end by discussing several proposed alternative analysis techniques that incorporate training gain scores and argue that none of these methods are appropriate for testing hypotheses regarding the effectiveness of cognitive training.

When decision heuristics and science collide.

The ongoing discussion among scientists about null-hypothesis significance testing and Bayesian data analysis has led to speculation about the practices and consequences of "researcher degrees of freedom." This article advances this debate by asking the broader questions that we, as scientists, should be asking: How do scientists make decisions in the course of doing research, and what is the impact of these decisions on scientific conclusions? We asked practicing scientists to collect data in a simulated research environment, and our findings show that some scientists use data collection heuristics that deviate from prescribed methodology. Monte Carlo simulations show that data collection heuristics based on p values lead to biases in estimated effect sizes and Bayes factors and to increases in both false-positive and false-negative rates, depending on the specific heuristic. We also show that using Bayesian data collection methods does not eliminate these biases. Thus, our study highlights the little appreciated fact that the process of doing science is a behavioral endeavor that can bias statistical description and inference in a manner that transcends adherence to any particular statistical framework.

Using the memory activation capture (MAC) procedure to investigate the temporal dynamics of hypothesis generation.

Research investigating top-down capture has demonstrated a coupling of working memory content with attention and eye movements. By capitalizing on this relationship, we have developed a novel methodology, called the memory activation capture (MAC) procedure, for measuring the dynamics of working memory content supporting complex cognitive tasks (e.g., decision making, problem solving). The MAC procedure employs briefly presented visual arrays containing task-relevant information at critical points in a task. By observing which items are preferentially fixated, we gain a measure of working memory content as the task evolves through time. The efficacy of the MAC procedure was demonstrated in a dynamic hypothesis generation task in which some of its advantages over existing methods for measuring changes in the contents of working memory over time are highlighted. In two experiments, the MAC procedure was able to detect the hypothesis that was retrieved and placed into working memory. Moreover, the results from Experiment 2 suggest a two-stage process following hypothesis retrieval, whereby the hypothesis undergoes a brief period of heightened activation before entering a lower activation state in which it is maintained for output. The results of both experiments are of additional general interest, as they represent the first demonstrations of top-down capture driven by participant-established WM content retrieved from long-term memory.

Working memory dynamics bias the generation of beliefs: the influence of data presentation rate on hypothesis generation.

Although temporal dynamics are inherent aspects of diagnostic tasks, few studies have investigated how various aspects of time course influence hypothesis generation. An experiment is reported that demonstrates that working memory dynamics operating during serial data acquisition bias hypothesis generation. The presentation rate (and order) of a sequence of serially presented symptoms was manipulated to be either fast (180 ms per symptom) or slow (1,500 ms per symptom) in a simulated medical diagnosis task. When the presentation rate was slow, participants chose the disease hypothesis consistent with the symptoms appearing later in the sequence. When the presentation rate was fast, however, participants chose the disease hypothesis consistent with the symptoms appearing earlier in the sequence, therefore representing a novel primacy effect. We predicted and account for this effect through competitive working memory dynamics governing information acquisition and the contribution of maintained information to the retrieval of hypotheses from long-term memory.

Catching a glimpse of working memory: top-down capture as a tool for measuring the content of the mind.

This article outlines a methodology for probing working memory (WM) content in high-level cognitive tasks (e.g., decision making, problem solving, and memory retrieval) by capitalizing on attentional and oculomotor biases evidenced in top-down capture paradigms. This method would be of great use, as it could measure the information resident in WM at any point in a task and, hence, track information use over time as tasks dynamically evolve. Above and beyond providing a measure of information occupancy in WM, such a method would benefit from sensitivity to the specific activation levels of individual items in WM. This article additionally forwards a novel fusion of standard free recall and visual search paradigms in an effort to assess the sensitivity of eye movements in top-down capture, on which this new measurement technique relies, to item-specific memory activation (ISMA). The results demonstrate eye movement sensitivity to ISMA in some, but not all, cases.

Temporal dynamics of hypothesis generation: the influences of data serial order, data consistency, and elicitation timing.

The pre-decisional process of hypothesis generation is a ubiquitous cognitive faculty that we continually employ in an effort to understand our environment and thereby support appropriate judgments and decisions. Although we are beginning to understand the fundamental processes underlying hypothesis generation, little is known about how various temporal dynamics, inherent in real world generation tasks, influence the retrieval of hypotheses from long-term memory. This paper presents two experiments investigating three data acquisition dynamics in a simulated medical diagnosis task. The results indicate that the mere serial order of data, data consistency (with previously generated hypotheses), and mode of responding influence the hypothesis generation process. An extension of the HyGene computational model endowed with dynamic data acquisition processes is forwarded and explored to provide an account of the present data.

Robust decision making in a nonlinear world.

The authors propose a general modeling framework called the general monotone model (GeMM), which allows one to model psychological phenomena that manifest as nonlinear relations in behavior data without the need for making (overly) precise assumptions about functional form. Using both simulated and real data, the authors illustrate that GeMM performs as well as or better than standard statistical approaches (including ordinary least squares, robust, and Bayesian regression) in terms of power and predictive accuracy when the functional relations are strictly linear but outperforms these approaches under conditions in which the functional relations are monotone but nonlinear. Finally, the authors recast their framework within the context of contemporary models of behavioral decision making, including the lens model and the take-the-best heuristic, and use GeMM to highlight several important issues within the judgment and decision-making literature.

Implications of cognitive load for hypothesis generation and probability judgment.

We tested the predictions of HyGene (Thomas et al., 2008) that both divided attention at encoding and judgment should affect the degree to which participants' probability judgments violate the principle of additivity. In two experiments, we showed that divided attention during judgment leads to an increase in subadditivity, suggesting that the comparison process for probability judgments is capacity limited. Contrary to the predictions of HyGene, a third experiment revealed that divided attention during encoding leads to an increase in later probability judgment made under full attention. The effect of divided attention during encoding on judgment was completely mediated by the number of hypotheses participants generated, indicating that limitations in both encoding and recall can cascade into biases in judgments.

Contextual biases in the interpretation of auditory evidence.

Noisy recordings of dialogue often serve as evidence in criminal proceedings. The present article explores the ability of two types of contextual information, currently present in the legal system, to bias subjective interpretations of such evidence. The present experiments demonstrate that the general context of the legal system and the presence of transcripts of the recorded speech are both able to bias interpretations of degraded & benign recordings into interpretable & incriminating. Furthermore we demonstrate a curse of knowledge whereby people become miscalibrated to the true quality of degraded recordings when provided transcripts. Current methods of dealing with auditory evidence are insufficient to mollify the effects of biasing information within the criminal justice system.

Diagnostic hypothesis generation and human judgment.

Diagnostic hypothesis-generation processes are ubiquitous in human reasoning. For example, clinicians generate disease hypotheses to explain symptoms and help guide treatment, auditors generate hypotheses for identifying sources of accounting errors, and laypeople generate hypotheses to explain patterns of information (i.e., data) in the environment. The authors introduce a general model of human judgment aimed at describing how people generate hypotheses from memory and how these hypotheses serve as the basis of probability judgment and hypothesis testing. In 3 simulation studies, the authors illustrate the properties of the model, as well as its applicability to explaining several common findings in judgment and decision making, including how errors and biases in hypothesis generation can cascade into errors and biases in judgment.