Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences.

Citation data and journal impact factors are important components of faculty dossiers and figure prominently in both promotion decisions and assessments of a researcher's broader societal impact. Although these metrics play a large role in high-stakes decisions, the evidence is mixed about whether they are strongly correlated with indicators of research quality. We use data from a large-scale dataset comprising 45 144 journal articles with 667 208 statistical tests and data from 190 replication attempts to assess whether citation counts and impact factors predict three indicators of research quality: (i) the accuracy of statistical reporting, (ii) the evidential value of the reported data and (iii) the replicability of a given experimental result. Both citation counts and impact factors were weak and inconsistent predictors of research quality, so defined, and sometimes negatively related to quality. Our findings raise the possibility that citation data and impact factors may be of limited utility in evaluating scientists and their research. We discuss the implications of these findings in light of current incentive structures and discuss alternative approaches to evaluating research.

Making Research Evaluation More Transparent: Aligning Research Philosophy, Institutional Values, and Reporting.

There is a growing interest in changing the culture of psychology to improve the quality of our science. At the root of this interest is concern over the reproducibility of key findings. A variety of large-scale replication attempts have revealed that several previously published effects cannot be reproduced, whereas other analyses indicate that the published literature is rife with underpowered studies and publication bias. These revelations suggest that it is time to change how psychological science is carried out and increase the transparency of reporting. We argue that change will be slow until institutions adopt new procedures for evaluating scholarly activity. We consider three actions that individuals and departments can take to facilitate change throughout psychological science: the development of individualized research-philosophy statements, the creation of an annotated curriculum vitae to improve the transparency of scholarly reporting, and the use of a formal evaluative system that explicitly captures behaviors that support reproducibility. Our recommendations build on proposals for open science by enabling researchers to have a voice in articulating (and contextualizing) how they would like their work to be evaluated and by providing a mechanism for more detailed and transparent reporting of scholarly activities.

Do metacognitive judgments alter memory performance beyond the benefits of retrieval practice? A comment on and replication attempt of Dougherty, Scheck, Nelson, and Narens (2005).

A central question in the metacognitive literature concerns whether the act of making a metacognitive judgment alters one's memory for the information about which the judgment was made. Dougherty, Scheck, Nelson, and Narens (2005, Memory & Cognition, 33(6), 1096-1115) attempted to address this question by having participants make either retrospective confidence judgments (RCJs; i.e., evaluations of past retrieval success), judgments of learning (JOLs; i.e., predictions of future retrieval success), or no explicit judgments. When comparing final retrieval accuracy they found that accuracy was greater for items where participants had made JOLs compared with items that received RCJs or no judgment, suggesting that simply making a JOL can improve later memory performance. The present article presents results from four separate replication attempts that fail to duplicate this finding. Combined results provide compelling evidence that making a metacognitive judgment, regardless of the type, has no impact on later memory performance above and beyond retrieval practice.

Does constraining memory maintenance reduce visual search efficiency?

We examine whether constraining memory retrieval processes affects performance in a cued recall visual search task. In the visual search task, participants are first presented with a memory prompt followed by a search array. The memory prompt provides diagnostic information regarding a critical aspect of the target (its colour). We assume that upon the presentation of the memory prompt, participants retrieve and maintain hypotheses (i.e., potential target characteristics) in working memory in order to improve their search efficiency. By constraining retrieval through the manipulation of time pressure (Experiments 1A and 1B) or a concurrent working memory task (Experiments 2A, 2B, and 2C), we directly test the involvement of working memory in visual search. We find some evidence that visual search is less efficient under conditions in which participants were likely to be maintaining fewer hypotheses in working memory (Experiments 1A, 2A, and 2C), suggesting that the retrieval of representations from long-term memory into working memory can improve visual search. However, these results should be interpreted with caution, as the data from two experiments (Experiments 1B and 2B) did not lend support for this conclusion.

Crowdsourcing prior information to improve study design and data analysis.

Though Bayesian methods are being used more frequently, many still struggle with the best method for setting priors with novel measures or task environments. We propose a method for setting priors by eliciting continuous probability distributions from naive participants. This allows us to include any relevant information participants have for a given effect. Even when prior means are near-zero, this method provides a principle way to estimate dispersion and produce shrinkage, reducing the occurrence of overestimated effect sizes. We demonstrate this method with a number of published studies and compare the effect of different prior estimation and aggregation methods.

Making Retrospective Confidence Judgments Improves Learners' Ability to Decide What Not to Study.

Predictions about future retrieval success, known as judgments of learning (JOLs), are often viewed as important for effective control over learning. However, much less is known about how retrospective confidence judgments (RCJs), evaluations of past retrieval success, may affect control over learning. We compared participants' ability to identify items that would benefit from additional study after making either a JOL or an RCJ. Participants completed a cued-recall task in which they made a metacognitive judgment after an initial recall attempt and before making a restudy decision. Participants who made RCJs prior to their restudy decisions were more accurate at identifying items in need of being restudied, relative to participants who made JOLs. The results indicate that having participants assess their confidence in past retrieval success can nudge them toward better utilizing of valid information when deciding which items are in need of further study.

Order-constrained linear optimization.

Despite the fact that data and theories in the social, behavioural, and health sciences are often represented on an ordinal scale, there has been relatively little emphasis on modelling ordinal properties. The most common analytic framework used in psychological science is the general linear model, whose variants include ANOVA, MANOVA, and ordinary linear regression. While these methods are designed to provide the best fit to the metric properties of the data, they are not designed to maximally model ordinal properties. In this paper, we develop an order-constrained linear least-squares (OCLO) optimization algorithm that maximizes the linear least-squares fit to the data conditional on maximizing the ordinal fit based on Kendall's τ. The algorithm builds on the maximum rank correlation estimator (Han, 1987, Journal of Econometrics, 35, 303) and the general monotone model (Dougherty & Thomas, 2012, Psychological Review, 119, 321). Analyses of simulated data indicate that when modelling data that adhere to the assumptions of ordinary least squares, OCLO shows minimal bias, little increase in variance, and almost no loss in out-of-sample predictive accuracy. In contrast, under conditions in which data include a small number of extreme scores (fat-tailed distributions), OCLO shows less bias and variance, and substantially better out-of-sample predictive accuracy, even when the outliers are removed. We show that the advantages of OCLO over ordinary least squares in predicting new observations hold across a variety of scenarios in which researchers must decide to retain or eliminate extreme scores when fitting data.

Reevaluating the effectiveness of n-back training on transfer through the Bayesian lens: Support for the null.

A recent meta-analysis by Au et al. Psychonomic Bulletin & Review, 22, 366-377, (2015) reviewed the n-back training paradigm for working memory (WM) and evaluated whether (when aggregating across existing studies) there was evidence that gains obtained for training tasks transferred to gains in fluid intelligence (Gf). Their results revealed an overall effect size of g = 0.24 for the effect of n-back training on Gf. We reexamine the data through a Bayesian lens, to evaluate the relative strength of the evidence for the alternative versus null hypotheses, contingent on the type of control condition used. We find that studies using a noncontact (passive) control group strongly favor the alternative hypothesis that training leads to transfer but that studies using active-control groups show modest evidence in favor of the null. We discuss these findings in the context of placebo effects.

Using a model of hypothesis generation to predict eye movements in a visual search task.

We used a model of hypothesis generation (called HyGene; Thomas, Dougherty, Sprenger, & Harbison, 2008) to make predictions regarding the deployment of attention (as assessed via eye movements) afforded by the cued recall of target characteristics before the onset of a search array. On each trial, while being eyetracked, participants were first presented with a memory prompt that was diagnostic regarding the target's color in a subsequently presented search array. We assume that the memory prompts led to the generation of hypotheses (i.e., potential target characteristics) from long-term memory into working memory to guide attentional processes and ocular-motor behavior. However, given that multiple hypotheses might be generated in response to a prompt, it has been unclear how the focal hypothesis (i.e., the hypothesis that exerts the most influence on search) affects search behavior. We tested two possibilities using first fixation data, with the assumption that the first item fixated within a search array was the focal hypothesis. We found that a model assuming that the first item generated into working memory guides overt attentional processes was most consistent with the data at both the aggregate and single-participant levels of analysis.

Measuring working memory is all fun and games: a four-dimensional spatial game predicts cognitive task performance.

We developed a novel four-dimensional spatial task called Shapebuilder and used it to predict performance on a wide variety of cognitive tasks. In six experiments, we illustrate that Shapebuilder: (1) Loads on a common factor with complex working memory (WM) span tasks and that it predicts performance on quantitative reasoning tasks and Ravens Progressive Matrices (Experiment 1), (2) Correlates well with traditional complex WM span tasks (Experiment 2), predicts performance on the conditional go/no go task (Experiment 3) and N-back (Experiment 4), and showed weak or nonsignificant correlations with the Attention Networks Task (Experiment 5), and task switching (Experiment 6). Shapebuilder shows that it exhibits minimal skew and kurtosis, and shows good reliability. We argue that Shapebuilder has many advantages over existing measures of WM, including the fact that it is largely language independent, is not prone to ceiling effects, and take less than 6 min to complete on average.

Reply to Rouder (2014): good frequentist properties raise confidence.

Established psychological results have been called into question by demonstrations that statistical significance is easy to achieve, even in the absence of an effect. One often-warned-against practice, choosing when to stop the experiment on the basis of the results, is guaranteed to produce significant results. In response to these demonstrations, Bayes factors have been proposed as an antidote to this practice, because they are invariant with respect to how an experiment was stopped. Should researchers only care about the resulting Bayes factor, without concern for how it was produced? Yu, Sprenger, Thomas, and Dougherty (2014) and Sanborn and Hills (2014) demonstrated that Bayes factors are sometimes strongly influenced by the stopping rules used. However, Rouder (2014) has provided a compelling demonstration that despite this influence, the evidence supplied by Bayes factors remains correct. Here we address why the ability to influence Bayes factors should still matter to researchers, despite the correctness of the evidence. We argue that good frequentist properties mean that results will more often agree with researchers' statistical intuitions, and good frequentist properties control the number of studies that will later be refuted. Both help raise confidence in psychological results.

What counts as evidence for working memory training? Problems with correlated gains and dichotomization.

The question of whether computerized cognitive training leads to generalized improvements of intellectual abilities has been a popular, yet contentious, topic within both the psychological and neurocognitive literatures. Evidence for the effective transfer of cognitive training to nontrained measures of cognitive abilities is mixed, with some studies showing apparent successful transfer, while others have failed to obtain this effect. At the same time, several authors have made claims about both successful and unsuccessful transfer effects on the basis of a form of responder analysis, an analysis technique that shows that those who gain the most on training show the greatest gains on transfer tasks. Through a series of Monte Carlo experiments and mathematical analyses, we demonstrate that the apparent transfer effects observed through responder analysis are illusory and are independent of the effectiveness of cognitive training. We argue that responder analysis can be used neither to support nor to refute hypotheses related to whether cognitive training is a useful intervention to obtain generalized cognitive benefits. We end by discussing several proposed alternative analysis techniques that incorporate training gain scores and argue that none of these methods are appropriate for testing hypotheses regarding the effectiveness of cognitive training.

Retrieval dynamics in self-terminated memory search.

Most free-recall experiments employ a paradigm in which participants are given a preset amount of time to retrieve items from a list. While much has been learned using this paradigm, it ignores an important component of many real-world retrieval tasks: the decision to terminate memory search. The present study examines the temporal characteristics underlying memory search by comparing within subjects a standard retrieval paradigm with a finite, preset amount of time (closed interval) to a design that allows participants to terminate memory search on their own (open interval). Calling on the results of several presented simulations, we anticipated that the threshold for number of retrieval failures varied as a function of the nature of the recall paradigm, such that open intervals should result in lower thresholds than closed intervals. Moreover, this effect was expected to manifest in interretrieval times (IRTs). Although retrieval-interval type did not significantly impact the number of items recalled or error rates, IRTs were sensitive to the manipulation. Specifically, the final IRTs in the closed-interval paradigm were longer than those of the open-interval paradigm. This pattern suggests that providing participants with a preset retrieval interval not only masks an important component of the retrieval process (the memory search termination decision), but also alters temporal retrieval dynamics. Task demands may compel people to strategically control aspects of their retrieval by implementing different stopping rules.

When decision heuristics and science collide.

The ongoing discussion among scientists about null-hypothesis significance testing and Bayesian data analysis has led to speculation about the practices and consequences of "researcher degrees of freedom." This article advances this debate by asking the broader questions that we, as scientists, should be asking: How do scientists make decisions in the course of doing research, and what is the impact of these decisions on scientific conclusions? We asked practicing scientists to collect data in a simulated research environment, and our findings show that some scientists use data collection heuristics that deviate from prescribed methodology. Monte Carlo simulations show that data collection heuristics based on p values lead to biases in estimated effect sizes and Bayes factors and to increases in both false-positive and false-negative rates, depending on the specific heuristic. We also show that using Bayesian data collection methods does not eliminate these biases. Thus, our study highlights the little appreciated fact that the process of doing science is a behavioral endeavor that can bias statistical description and inference in a manner that transcends adherence to any particular statistical framework.

Robust decision making in a nonlinear world.

The authors propose a general modeling framework called the general monotone model (GeMM), which allows one to model psychological phenomena that manifest as nonlinear relations in behavior data without the need for making (overly) precise assumptions about functional form. Using both simulated and real data, the authors illustrate that GeMM performs as well as or better than standard statistical approaches (including ordinary least squares, robust, and Bayesian regression) in terms of power and predictive accuracy when the functional relations are strictly linear but outperforms these approaches under conditions in which the functional relations are monotone but nonlinear. Finally, the authors recast their framework within the context of contemporary models of behavioral decision making, including the lens model and the take-the-best heuristic, and use GeMM to highlight several important issues within the judgment and decision-making literature.

Generating and evaluating options for decision making: the impact of sequentially presented evidence.

We examined how decision makers generate and evaluate hypotheses when data are presented sequentially. In the first 2 experiments, participants learned the relationship between data and possible causes of the data in a virtual environment. Data were then presented iteratively, and participants either generated hypotheses they thought caused the data or rated the probability of possible causes of the data. In a 3rd experiment, participants generated hypotheses and made probability judgments on the basis of previously stored general knowledge. Findings suggest that both the hypotheses one generates and the judged probability of those hypotheses are heavily influenced by the most recent evidence observed and by the diagnosticity of the evidence. Specifically, participants generated a narrow set of possible explanations when the presented evidence was diagnostic compared with when it was nondiagnostic, suggesting that nondiagnostic evidence entices participants to cast a wider net when generating hypotheses.

Implications of cognitive load for hypothesis generation and probability judgment.

We tested the predictions of HyGene (Thomas et al., 2008) that both divided attention at encoding and judgment should affect the degree to which participants' probability judgments violate the principle of additivity. In two experiments, we showed that divided attention during judgment leads to an increase in subadditivity, suggesting that the comparison process for probability judgments is capacity limited. Contrary to the predictions of HyGene, a third experiment revealed that divided attention during encoding leads to an increase in later probability judgment made under full attention. The effect of divided attention during encoding on judgment was completely mediated by the number of hypotheses participants generated, indicating that limitations in both encoding and recall can cascade into biases in judgments.

On the lawfulness of the decision to terminate memory search.

Nearly every memory retrieval episode ends with a decision to terminate memory search. Yet, no research has investigated whether these search termination decisions are systematic, let alone whether they are made consistent with a particular rule. In the present paper, we used a modified free-recall paradigm to examine the decision to terminate search. Data from two experiments revealed that the total time engaged in retrieval was a monotonically increasing function of the total number of items retrieved, whereas the time from final retrieval to search termination (exit latency) was a monotonically decreasing function of the total number of items retrieved. These findings were compared to the predictions of previously proposed stopping rules, using the Search of Associative Memory framework. Of the four rules examined, only one predicts the obtained data pattern.

Diagnostic hypothesis generation and human judgment.

Diagnostic hypothesis-generation processes are ubiquitous in human reasoning. For example, clinicians generate disease hypotheses to explain symptoms and help guide treatment, auditors generate hypotheses for identifying sources of accounting errors, and laypeople generate hypotheses to explain patterns of information (i.e., data) in the environment. The authors introduce a general model of human judgment aimed at describing how people generate hypotheses from memory and how these hypotheses serve as the basis of probability judgment and hypothesis testing. In 3 simulation studies, the authors illustrate the properties of the model, as well as its applicability to explaining several common findings in judgment and decision making, including how errors and biases in hypothesis generation can cascade into errors and biases in judgment.

Psychological plausibility of the theory of probabilistic mental models and the fast and frugal heuristics.

The theory of probabilistic mental models (PMM; G. Gigerenzer, U. Hoffrage, & H. Kleinbölting, 1991) has had a major influence on the field of judgment and decision making, with the most recent important modifications to PMM theory being the identification of several fast and frugal heuristics (G. Gigerenzer & D. G. Goldstein, 1996). These heuristics were purported to provide psychologically plausible cognitive process models that describe a variety of judgment behavior. In this article, the authors evaluate the psychological plausibility of the assumptions upon which PMM were built and, consequently, the psychological plausibility of several of the fast and frugal heuristics. The authors argue that many of PMM theory's assumptions are questionable, given available data, and that fast and frugal heuristics are, in fact, psychologically implausible.