Do metacognitive judgments alter memory performance beyond the benefits of retrieval practice? A comment on and replication attempt of Dougherty, Scheck, Nelson, and Narens (2005).

A central question in the metacognitive literature concerns whether the act of making a metacognitive judgment alters one's memory for the information about which the judgment was made. Dougherty, Scheck, Nelson, and Narens (2005, Memory & Cognition, 33(6), 1096-1115) attempted to address this question by having participants make either retrospective confidence judgments (RCJs; i.e., evaluations of past retrieval success), judgments of learning (JOLs; i.e., predictions of future retrieval success), or no explicit judgments. When comparing final retrieval accuracy they found that accuracy was greater for items where participants had made JOLs compared with items that received RCJs or no judgment, suggesting that simply making a JOL can improve later memory performance. The present article presents results from four separate replication attempts that fail to duplicate this finding. Combined results provide compelling evidence that making a metacognitive judgment, regardless of the type, has no impact on later memory performance above and beyond retrieval practice.

Making Retrospective Confidence Judgments Improves Learners' Ability to Decide What Not to Study.

Predictions about future retrieval success, known as judgments of learning (JOLs), are often viewed as important for effective control over learning. However, much less is known about how retrospective confidence judgments (RCJs), evaluations of past retrieval success, may affect control over learning. We compared participants' ability to identify items that would benefit from additional study after making either a JOL or an RCJ. Participants completed a cued-recall task in which they made a metacognitive judgment after an initial recall attempt and before making a restudy decision. Participants who made RCJs prior to their restudy decisions were more accurate at identifying items in need of being restudied, relative to participants who made JOLs. The results indicate that having participants assess their confidence in past retrieval success can nudge them toward better utilizing of valid information when deciding which items are in need of further study.

Using a model of hypothesis generation to predict eye movements in a visual search task.

We used a model of hypothesis generation (called HyGene; Thomas, Dougherty, Sprenger, & Harbison, 2008) to make predictions regarding the deployment of attention (as assessed via eye movements) afforded by the cued recall of target characteristics before the onset of a search array. On each trial, while being eyetracked, participants were first presented with a memory prompt that was diagnostic regarding the target's color in a subsequently presented search array. We assume that the memory prompts led to the generation of hypotheses (i.e., potential target characteristics) from long-term memory into working memory to guide attentional processes and ocular-motor behavior. However, given that multiple hypotheses might be generated in response to a prompt, it has been unclear how the focal hypothesis (i.e., the hypothesis that exerts the most influence on search) affects search behavior. We tested two possibilities using first fixation data, with the assumption that the first item fixated within a search array was the focal hypothesis. We found that a model assuming that the first item generated into working memory guides overt attentional processes was most consistent with the data at both the aggregate and single-participant levels of analysis.

A memory-theoretic account of citation propagation.

Despite the common assumption that citations are indicative of an article's scientific merit, increasing evidence indicates that citation counts are largely driven by variables unrelated to quality. In this article, we treat people's decisions of what to cite as an instance of memory retrieval and show that observed citation patterns are well accounted for by a model of memory. The proposed exposure model anticipates that small alterations in factors that affect people's ability to retrieve to-be-cited articles from memory early in their life cycle are magnified over time and can lead to the emergence of highly cited papers. This effect occurs even when there is no variation in the starting point exposure probabilities (i.e. when assuming a level playing field where all articles are treated equally and of equal 'quality'), and is exacerbated by natural variation in retrievability of articles due to encoding. We discuss the implications of the model within the context of research evaluation and hiring, tenure and promotion decisions.

Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences.

Citation data and journal impact factors are important components of faculty dossiers and figure prominently in both promotion decisions and assessments of a researcher's broader societal impact. Although these metrics play a large role in high-stakes decisions, the evidence is mixed about whether they are strongly correlated with indicators of research quality. We use data from a large-scale dataset comprising 45 144 journal articles with 667 208 statistical tests and data from 190 replication attempts to assess whether citation counts and impact factors predict three indicators of research quality: (i) the accuracy of statistical reporting, (ii) the evidential value of the reported data and (iii) the replicability of a given experimental result. Both citation counts and impact factors were weak and inconsistent predictors of research quality, so defined, and sometimes negatively related to quality. Our findings raise the possibility that citation data and impact factors may be of limited utility in evaluating scientists and their research. We discuss the implications of these findings in light of current incentive structures and discuss alternative approaches to evaluating research.

Making Research Evaluation More Transparent: Aligning Research Philosophy, Institutional Values, and Reporting.

There is a growing interest in changing the culture of psychology to improve the quality of our science. At the root of this interest is concern over the reproducibility of key findings. A variety of large-scale replication attempts have revealed that several previously published effects cannot be reproduced, whereas other analyses indicate that the published literature is rife with underpowered studies and publication bias. These revelations suggest that it is time to change how psychological science is carried out and increase the transparency of reporting. We argue that change will be slow until institutions adopt new procedures for evaluating scholarly activity. We consider three actions that individuals and departments can take to facilitate change throughout psychological science: the development of individualized research-philosophy statements, the creation of an annotated curriculum vitae to improve the transparency of scholarly reporting, and the use of a formal evaluative system that explicitly captures behaviors that support reproducibility. Our recommendations build on proposals for open science by enabling researchers to have a voice in articulating (and contextualizing) how they would like their work to be evaluated and by providing a mechanism for more detailed and transparent reporting of scholarly activities.

Psychological plausibility of the theory of probabilistic mental models and the fast and frugal heuristics.

The theory of probabilistic mental models (PMM; G. Gigerenzer, U. Hoffrage, & H. Kleinbölting, 1991) has had a major influence on the field of judgment and decision making, with the most recent important modifications to PMM theory being the identification of several fast and frugal heuristics (G. Gigerenzer & D. G. Goldstein, 1996). These heuristics were purported to provide psychologically plausible cognitive process models that describe a variety of judgment behavior. In this article, the authors evaluate the psychological plausibility of the assumptions upon which PMM were built and, consequently, the psychological plausibility of several of the fast and frugal heuristics. The authors argue that many of PMM theory's assumptions are questionable, given available data, and that fast and frugal heuristics are, in fact, psychologically implausible.

Diagnostic hypothesis generation and human judgment.

Diagnostic hypothesis-generation processes are ubiquitous in human reasoning. For example, clinicians generate disease hypotheses to explain symptoms and help guide treatment, auditors generate hypotheses for identifying sources of accounting errors, and laypeople generate hypotheses to explain patterns of information (i.e., data) in the environment. The authors introduce a general model of human judgment aimed at describing how people generate hypotheses from memory and how these hypotheses serve as the basis of probability judgment and hypothesis testing. In 3 simulation studies, the authors illustrate the properties of the model, as well as its applicability to explaining several common findings in judgment and decision making, including how errors and biases in hypothesis generation can cascade into errors and biases in judgment.

Does constraining memory maintenance reduce visual search efficiency?

We examine whether constraining memory retrieval processes affects performance in a cued recall visual search task. In the visual search task, participants are first presented with a memory prompt followed by a search array. The memory prompt provides diagnostic information regarding a critical aspect of the target (its colour). We assume that upon the presentation of the memory prompt, participants retrieve and maintain hypotheses (i.e., potential target characteristics) in working memory in order to improve their search efficiency. By constraining retrieval through the manipulation of time pressure (Experiments 1A and 1B) or a concurrent working memory task (Experiments 2A, 2B, and 2C), we directly test the involvement of working memory in visual search. We find some evidence that visual search is less efficient under conditions in which participants were likely to be maintaining fewer hypotheses in working memory (Experiments 1A, 2A, and 2C), suggesting that the retrieval of representations from long-term memory into working memory can improve visual search. However, these results should be interpreted with caution, as the data from two experiments (Experiments 1B and 2B) did not lend support for this conclusion.

Motivated to retrieve: how often are you willing to go back to the well when the well is dry?

Despite the necessity of the decision to terminate memory search in many real-world memory tasks, little experimental work has investigated the underlying processes. In this study, the authors investigated termination decisions in free recall by providing participants an open-ended retrieval interval and requiring them to press a stop button when they had finished retrieving. Three variables important to assessing one's willingness to search memory were examined: (a) the time spent searching memory after the last successful retrieval before choosing to quit (the exit latency); (b) task difficulty; and (c) individual differences in motivation, as measured by Webster and Kruglanski's (1994) Need for Closure Scale. A strong negative correlation was found between individual differences in motivation and participants' exit latencies. This negative correlation was present only when the retrieval task started out as relatively difficult.

The effect of divided attention on global judgment of learning accuracy.

This study examined the effect of divided attention (DA) on global judgment of learning (JOL) accuracy in a multitrial list learning paradigm. A word monitoring task was used to divide attention. Participants were assigned to an attention condition (DA at encoding, DA at judgment, DA at retrieval, or focused attention) and completed 4 learning trials, each comprising a study, judgment, and recall phase. Participants showed greater overconfidence in the DA at encoding (Trial 2) and DA at retrieval (Trials 1 and 2) conditions than in the focused attention condition. DA atjudgment did not affect JOL accuracy, and there was no effect of DA in Trials 3 and 4 on JOL accuracy across all attention conditions. Results indicate that participants consider conditions of encoding and retrieval but do not engage in recall when forming global JOLs. These findings suggest that people rely on extrinsic cues when making repeated, global metamemoryjudgments.

Crowdsourcing prior information to improve study design and data analysis.

Though Bayesian methods are being used more frequently, many still struggle with the best method for setting priors with novel measures or task environments. We propose a method for setting priors by eliciting continuous probability distributions from naive participants. This allows us to include any relevant information participants have for a given effect. Even when prior means are near-zero, this method provides a principle way to estimate dispersion and produce shrinkage, reducing the occurrence of overestimated effect sizes. We demonstrate this method with a number of published studies and compare the effect of different prior estimation and aggregation methods.

Order-constrained linear optimization.

Despite the fact that data and theories in the social, behavioural, and health sciences are often represented on an ordinal scale, there has been relatively little emphasis on modelling ordinal properties. The most common analytic framework used in psychological science is the general linear model, whose variants include ANOVA, MANOVA, and ordinary linear regression. While these methods are designed to provide the best fit to the metric properties of the data, they are not designed to maximally model ordinal properties. In this paper, we develop an order-constrained linear least-squares (OCLO) optimization algorithm that maximizes the linear least-squares fit to the data conditional on maximizing the ordinal fit based on Kendall's τ. The algorithm builds on the maximum rank correlation estimator (Han, 1987, Journal of Econometrics, 35, 303) and the general monotone model (Dougherty & Thomas, 2012, Psychological Review, 119, 321). Analyses of simulated data indicate that when modelling data that adhere to the assumptions of ordinary least squares, OCLO shows minimal bias, little increase in variance, and almost no loss in out-of-sample predictive accuracy. In contrast, under conditions in which data include a small number of extreme scores (fat-tailed distributions), OCLO shows less bias and variance, and substantially better out-of-sample predictive accuracy, even when the outliers are removed. We show that the advantages of OCLO over ordinary least squares in predicting new observations hold across a variety of scenarios in which researchers must decide to retain or eliminate extreme scores when fitting data.

The influence of improper sets of information on judgment: how irrelevant information can bias judged probability.

This article introduces 2 new sources of bias in probability judgment, discrimination failure and inhibition failure, which are conceptualized as arising from an interaction between error prone memory processes and a support theory like comparison process. Both sources of bias stem from the influence of irrelevant information on participants' probability judgments, but they postulate different mechanisms for how irrelevant information affects judgment. The authors used an adaptation of the proactive interference (PI) and release from PI paradigm to test the effect of irrelevant information on judgment. The results of 2 experiments support the discrimination failure account of the effect of PI on probability judgment. In addition, the authors show that 2 commonly used measures of judgment accuracy, absolute and relative accuracy, can be dissociated. The results have broad implications for theories of judgment.

Reevaluating the effectiveness of n-back training on transfer through the Bayesian lens: Support for the null.

A recent meta-analysis by Au et al. Psychonomic Bulletin & Review, 22, 366-377, (2015) reviewed the n-back training paradigm for working memory (WM) and evaluated whether (when aggregating across existing studies) there was evidence that gains obtained for training tasks transferred to gains in fluid intelligence (Gf). Their results revealed an overall effect size of g = 0.24 for the effect of n-back training on Gf. We reexamine the data through a Bayesian lens, to evaluate the relative strength of the evidence for the alternative versus null hypotheses, contingent on the type of control condition used. We find that studies using a noncontact (passive) control group strongly favor the alternative hypothesis that training leads to transfer but that studies using active-control groups show modest evidence in favor of the null. We discuss these findings in the context of placebo effects.

Reducing bias in frequency judgment by improving source monitoring.

A common finding in judgment and decision making is that people's frequency judgments often fail to map onto objective frequencies. The present research examined the possibility that one source of bias in frequency judgment is attributable to people's inability to screen out irrelevant memory traces. We used a two-list source-monitoring paradigm to investigate whether frequency judgments are influenced by "extra-experimental" experiences and whether enhancing source monitoring improves judgment accuracy. Across four experiments we found: (1) frequency judgments regarding one list were biased by the second, (2) manipulating encoding between lists improved source monitoring and resulted in more accurate judgments, (3) manipulating item context between lists improved source monitoring and resulted in more accurate judgments, but only when the context was item specific, and (4) manipulating simple-background context between lists was ineffective at improving source monitoring.

Hypothesis generation, probability judgment, and individual differences in working memory capacity.

This research examined the role of working memory (WM) in probability judgment and hypothesis generation using a simulated task that involved estimating the likelihood that particular menu items would be ordered by customers at a dinner. Five main findings were observed. First, judgments of the likelihood of individual items were made relative to alternatives retrieved from long-term memory. Second, the number of alternatives retrieved was positively correlated with a measure of WM-capacity (the operation-span task). Third, participants' probability judgments were subadditive (summing to well over 100%). Fourth, the degree to which participants' judgments were subadditive was affected by the number and strength of the alternatives retrieved from long-term memory. Fifth, the degree to which participants were subadditive was negatively correlated with WM-capacity. The results suggest that individual differences in WM-capacity are fundamental to hypothesis generation and probability judgment.

What counts as evidence for working memory training? Problems with correlated gains and dichotomization.

The question of whether computerized cognitive training leads to generalized improvements of intellectual abilities has been a popular, yet contentious, topic within both the psychological and neurocognitive literatures. Evidence for the effective transfer of cognitive training to nontrained measures of cognitive abilities is mixed, with some studies showing apparent successful transfer, while others have failed to obtain this effect. At the same time, several authors have made claims about both successful and unsuccessful transfer effects on the basis of a form of responder analysis, an analysis technique that shows that those who gain the most on training show the greatest gains on transfer tasks. Through a series of Monte Carlo experiments and mathematical analyses, we demonstrate that the apparent transfer effects observed through responder analysis are illusory and are independent of the effectiveness of cognitive training. We argue that responder analysis can be used neither to support nor to refute hypotheses related to whether cognitive training is a useful intervention to obtain generalized cognitive benefits. We end by discussing several proposed alternative analysis techniques that incorporate training gain scores and argue that none of these methods are appropriate for testing hypotheses regarding the effectiveness of cognitive training.

Retrieval dynamics in self-terminated memory search.

Most free-recall experiments employ a paradigm in which participants are given a preset amount of time to retrieve items from a list. While much has been learned using this paradigm, it ignores an important component of many real-world retrieval tasks: the decision to terminate memory search. The present study examines the temporal characteristics underlying memory search by comparing within subjects a standard retrieval paradigm with a finite, preset amount of time (closed interval) to a design that allows participants to terminate memory search on their own (open interval). Calling on the results of several presented simulations, we anticipated that the threshold for number of retrieval failures varied as a function of the nature of the recall paradigm, such that open intervals should result in lower thresholds than closed intervals. Moreover, this effect was expected to manifest in interretrieval times (IRTs). Although retrieval-interval type did not significantly impact the number of items recalled or error rates, IRTs were sensitive to the manipulation. Specifically, the final IRTs in the closed-interval paradigm were longer than those of the open-interval paradigm. This pattern suggests that providing participants with a preset retrieval interval not only masks an important component of the retrieval process (the memory search termination decision), but also alters temporal retrieval dynamics. Task demands may compel people to strategically control aspects of their retrieval by implementing different stopping rules.

When decision heuristics and science collide.

The ongoing discussion among scientists about null-hypothesis significance testing and Bayesian data analysis has led to speculation about the practices and consequences of "researcher degrees of freedom." This article advances this debate by asking the broader questions that we, as scientists, should be asking: How do scientists make decisions in the course of doing research, and what is the impact of these decisions on scientific conclusions? We asked practicing scientists to collect data in a simulated research environment, and our findings show that some scientists use data collection heuristics that deviate from prescribed methodology. Monte Carlo simulations show that data collection heuristics based on p values lead to biases in estimated effect sizes and Bayes factors and to increases in both false-positive and false-negative rates, depending on the specific heuristic. We also show that using Bayesian data collection methods does not eliminate these biases. Thus, our study highlights the little appreciated fact that the process of doing science is a behavioral endeavor that can bias statistical description and inference in a manner that transcends adherence to any particular statistical framework.