Evaluating probabilistic forecasts with Bayesian signal detection models.

We propose the use of signal detection theory (SDT) to evaluate the performance of both probabilistic forecasting systems and individual forecasters. The main advantage of SDT is that it provides a principled way to distinguish the response from system diagnosticity, which is defined as the ability to distinguish events that occur from those that do not. There are two challenges in applying SDT to probabilistic forecasts. First, the SDT model must handle judged probabilities rather than the conventional binary decisions. Second, the model must be able to operate in the presence of sparse data generated within the context of human forecasting systems. Our approach is to specify a model of how individual forecasts are generated from underlying representations and use Bayesian inference to estimate the underlying latent parameters. Given our estimate of the underlying representations, features of the classic SDT model, such as the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC), follow immediately. We show how our approach allows ROC curves and AUCs to be applied to individuals within a group of forecasters, estimated as a function of time, and extended to measure differences in forecastability across different domains. Among the advantages of this method is that it depends only on the ordinal properties of the probabilistic forecasts. We conclude with a brief discussion of how this approach might facilitate decision making.

Predicting binary choices from probability phrase meanings.

The issues of how individuals decide which of two events is more likely and of how they understand probability phrases both involve judging relative likelihoods. In this study, we investigated whether derived scales representing probability phrase meanings could be used within a choice model to predict independently observed binary choices. If they can, this simultaneously provides support for our model and suggests that the phrase meanings are measured meaningfully. The model assumes that, when deciding which of two events is more likely, judges take a single sample from memory regarding each event and respond accordingly. The model predicts choice probabilities by using the scaled meanings of individually selected probability phrases as proxies for confidence distributions associated with sampling from memory. Predictions are sustained for 34 of 41 participants but, nevertheless, are biased slightly low. Sequential sampling models improve the fit. The results have both theoretical and applied implications.

Development of an automatic response mode to improve the clinical utility of sequential risk-taking tasks.

Sequential risk-taking tasks, especially the Balloon Analogue Risk Task (BART), have proven powerful and useful methods in studying and identifying real-world risk takers. A natural index in these tasks is the average number of risks the participant takes in a trial (e.g., pumps on the balloons), but this is difficult to estimate because some trials terminate early because of the consequences of those risks (e.g., when the desired number of balloon pumps exceeds the explosion point). The standard corrective strategy is to use an adjusted score that ignores such event-terminated trials. Although previous data supports the utility of this adjusted score, the authors show formally that it is biased. Therefore, the authors developed an automatic response procedure, in which respondents state at the beginning of each trial how many risks they wish to take and then observe the sequence of events unfold. A study comparing this new automatic and the original manual BART shows that the automatic procedure yields unbiased statistics whereas maintaining the BART's predictive validity of substance use. The authors also found that providing respondents with the expected-value-maximizing strategy and complete trial-by-trial feedback increased the number of risks they were willing to take during the BART. The authors interpret these results in terms of the potential utility of the automatic version including shorter administration time, unbiased behavioral measures, and minimizing motor involvement, which is important in neuroscientific investigations or with clinical populations with motor limitations.

Language performance under sustained work and sleep deprivation conditions.

INTRODUCTION: Although substantial research has been completed on the effects of sleep deprivation on performance, very little research has focused on language-based tasks. The purpose of the current study was two-fold: 1) to determine the extent to which short-term sleep deprivation affects language performance; and 2) to examine whether relatively short and easy-to-administer "probe" tasks could signal decrements in language performance under sleep deprivation conditions. METHODS: There were 38 non-native English-speaking students who were paid to complete a 28-h sleep deprivation study. The participants completed several potential cognitive and vigilance probe tasks and a variety of language-based tasks. Each task was administered four times, once in each testing session during the night (18:30-22:30, 23:00-03:00, 03: 30-07:30, and 08:00-12:00). All tasks were counterbalanced across the testing sessions. RESULTS: Repeated-measures ANOVAs indicated that language tasks that required sustained attention and higher level processing (e.g., reading comprehension) were negatively affected by sleep deprivation, whereas other tasks that relied primarily on more basic language processing (e.g., antonym identification) were not affected. Hierarchical linear modeling analyses assessed how well the probe tasks predicted language performance. These results indicated that performance accuracy and/or speed on many of the probe tasks predicted decrements in language performance. CONCLUSIONS: These findings suggest that sustained work conditions and sleep deprivation negatively affect some types of language performance. Moreover, the use of probe tasks indicates that easy-to-administer tasks may be useful to identify when detriments are likely to occur in language-based performance under sleep deprivation conditions.

Modeling behavior in a clinically diagnostic sequential risk-taking task.

This article models the cognitive processes underlying learning and sequential choice in a risk-taking task for the purposes of understanding how they occur in this moderately complex environment and how behavior in it relates to self-reported real-world risk taking. The best stochastic model assumes that participants incorrectly treat outcome probabilities as stationary, update probabilities in a Bayesian fashion, evaluate choice policies prior to rather than during responding, and maintain constant response sensitivity. The model parameter associated with subjective value of gains correlates well with external risk taking. Both the overall approach, which can be expanded as the basic paradigm is varied, and the specific results provide direction for theories of risky choice and for understanding risk taking as a public health problem.

A stochastic detection and retrieval model for the study of metacognition.

We present a signal detection-like model termed the stochastic detection and retrieval model (SDRM) for use in studying metacognition. Focusing on paradigms that relate retrieval (e.g., recall or recognition) and confidence judgments, the SDRM measures (1) variance in the retrieval process, (2) variance in the confidence process, (3) the extent to which different sources of information underlie each response, (4) simple bias (i.e., increasing or decreasing confidence criteria across conditions), and (5) metacognitive bias (i.e., contraction or expansion of the confidence criteria across conditions). In the metacognition literature, gamma correlations have been used to measure the accuracy of confidence judgments. However, gamma cannot distinguish between the first 3 attributes, and it cannot measure either form of bias. In contrast, the SDRM can distinguish among the attributes, and it can measure both forms of bias. In this way, the SDRM can be used to test competing process theories by determining the attribute that best accounts for a change across conditions. To demonstrate the SDRM's usefulness, we investigated judgments of learning (JOLs) followed by cued-recall. Through a series of nested and non-nested model comparisons applied to a new experiment, the SDRM determined that a reduction in variance during the confidence process is the most likely explanation of the delayed-JOL effect, and a stronger relation between information underlying JOLs and recall is the most likely explanation of the testing-JOL effect. Following a brief discussion of implications for JOL theories, we conclude with a broader discussion of how the SDRM can benefit metacognition research.

A query theory account of the effect of memory retrieval on the sunk cost bias.

The sunk cost bias occurs when individuals continue to invest in the same option when better alternatives are available. Many researchers believe that this bias is due to overemphasizing the past investment over the (missed) opportunities offered by alternatives. As an alternative or complement to this view, we show that memory retrieval and attention play important roles in the sunk cost bias. In two experiments, individuals generated more reasons for pursuing the invested option than for an alternative; they generated those reasons earlier in a sequence of reasons; and these effects increased as the individuals made progress toward attaining the reward yielded by the invested option. Associated with these effects, individuals perceived an increasingly wide gap in value between the invested and alternative options as they progressed toward the goal, thereby creating the sunk cost bias. Forcing individuals to reverse the order in which they generated reasons for the invested and alternative options reduced the bias. [corrected]

Subjective recalibration of advisors' probability estimates.

Are decision makers sensitive to the statistical properties (i.e., calibration) of probability estimates that they receive from advisors? After specifying the ideal use of such estimates, we derive the roughly ideal forecast consumer (RIFC) and generalize it to account for how humans might use the estimates. We report an experiment in which participants first experienced various advisors by seeing their probability estimates and the associated outcomes and then provided confidence judgments in the presence of the advisors' estimates. The generalized model described the data well and showed that the participants were appropriately sensitive to the statistical properties of the advisors. Models of the individuals were better calibrated than the participants themselves, but still inferior to the RIFC. A detailed description of our model-testing procedure can be found in an appendix to the article.

Interpersonal comparison of subjective probabilities: toward translating linguistic probabilities.

Interpersonal variability in understanding linguistic probabilities can adversely affect decision making. Using the fact that everyone judges canonical probability events similarly in a manner consistent with axiom systems that yield a probability measure, we developed and tested a method for comparing the meanings of probability phrases across individuals. An experiment demonstrated that despite extreme heterogeneity in participants' linguistic probability lexicons, interpersonal similarity in phrase meaning is well predicted by phrase rank order within the lexicons. Thus, equally ranked phrases have similar meanings, and individual differences in linguistic probabilities may simply be explained by the phrases people use at each rank.

On Setting Response Criteria for Calibrated Subjective Probability Estimates.

Framing the issue of subjective probability calibration in signal-detection-theory terms, this paper first proves a theorem regarding the placement of well-calibrated response criteria and then develops an algorithm guaranteed to find such criteria, should they exist. Application of this algorithm to tasks varying in difficulty and number of response categories shows that perfect calibration is easiest to attain under median difficulty levels (d' approximately 1.4) and is practically or theoretically impossible to attain when the task is either very hard (d' approximately 0.5) or very easy (d' approximately 10). Implications for calibration research, including the hard-easy effect, are discussed. Copyright 2001 Academic Press.

Assessing the Risk of Microbial Pathogens: Application of a Judgment-Encoding Methodology.

This paper reports the results of a risk assessment of the adverse health effects from ingesting foods contaminated with certain microbial pathogens. The risk assessment was performed as part of a larger project to develop a risk-based sampling methodology for imported foods inspected by the U.S. Food and Drug Administration (FDA). The sampling methodology was put into operation in a computer aid to assist FDA import inspectors in choosing samples to test for violative substances. The computer aid was designed to choose samples so as to maximize the benefit from the sampling plan. The expected benefit of sampling food depends on the probability that a violation exists, the probability that it will be detected by testing, and the risk of illness associated with the violation. While most of this information is available from data collected by the FDA, very little information has been published regarding the relationship between violative substances and human illness, particularly for microbial pathogens. To narrow this information gap, we recruited the assistance of several experts in the fields of microbiology and epidemiology to evaluate the uncertainty surrounding the intake-response relationships for several microbial pathogens. The information collected from the expert elicitation process and documented here cannot substitute for the scientific data needed to accurately estimate dose-response relationships and their variances. Our goal was simply to gather approximations of these relationships to use in the sampling aid. Our results differ from traditional dose-response curves in that we quantified the uncertainty associated with each probability judgment.