Performance Validity Test Failure in the Clinical Population: A Systematic Review and Meta-Analysis of Prevalence Rates.

Performance validity tests (PVTs) are used to measure the validity of the obtained neuropsychological test data. However, when an individual fails a PVT, the likelihood that failure truly reflects invalid performance (i.e., the positive predictive value) depends on the base rate in the context in which the assessment takes place. Therefore, accurate base rate information is needed to guide interpretation of PVT performance. This systematic review and meta-analysis examined the base rate of PVT failure in the clinical population (PROSPERO number: CRD42020164128). PubMed/MEDLINE, Web of Science, and PsychINFO were searched to identify articles published up to November 5, 2021. Main eligibility criteria were a clinical evaluation context and utilization of stand-alone and well-validated PVTs. Of the 457 articles scrutinized for eligibility, 47 were selected for systematic review and meta-analyses. Pooled base rate of PVT failure for all included studies was 16%, 95% CI [14, 19]. High heterogeneity existed among these studies (Cochran's Q = 697.97, p < .001; I2 = 91%; τ2 = 0.08). Subgroup analysis indicated that pooled PVT failure rates varied across clinical context, presence of external incentives, clinical diagnosis, and utilized PVT. Our findings can be used for calculating clinically applied statistics (i.e., positive and negative predictive values, and likelihood ratios) to increase the diagnostic accuracy of performance validity determination in clinical evaluation. Future research is necessary with more detailed recruitment procedures and sample descriptions to further improve the accuracy of the base rate of PVT failure in clinical practice.

Quantifying vulnerability to plant invasion across global ecosystems.

The widely referenced "tens rule" in invasion ecology suggests that approximately 10% of established, non-native species will become invasive. However, the accuracy of this estimate has been questioned, as the original analysis focused on small groups of plant species in Great Britain and Australia. Using a novel database of 9501 established plants and 2924 invasive plants, we provide a comprehensive evaluation of the tens rule and the first empirical analysis of how invasion rates vary across spatial scales, islands/mainlands, and climate zones. We found that invasion rates (the percentage of established species with negative impacts) are highly variable across the globe. Well-sampled environments (those with at least 2000 total non-native species recorded) had invasion rates that ranged from 7.2% to 33.8%. Invasion rates were strongly scale-dependent, averaging 17% at the country scale and 25% at the continental scale. We found significantly higher invasion rates on islands when compared with mainlands, regardless of scale. Tropical ecosystems are often considered to be resistant to invasion; however, our results showed significantly higher invasion rates on both tropical islands and mainlands, suggesting unexpectedly high vulnerability of these species-rich ecosystems. We conclude that the tens rule is a poor general estimate of invasion rates for plants, as calculated invasion rates vary widely and are frequently much higher than 10%. Most locations would be better served by using invasion rates that vary based on the recipient environment. Our updated estimates of invasion rates should be highly relevant for invasive species management strategies, including weed risk assessments, which can be adjusted to identify more species as high-risk in areas where invasion rates are higher. Assuming that 10% of established species will become invasive is likely to substantially underestimate invasion rates in most geographies.

Does context recollection depend on the base-rate of contextual features?

Episodic recollection is defined by the re-experiencing of contextual and target details of a past event. The base-rate dependency hypothesis assumes that the retrieval of one contextual feature from an integrated episodic trace cues the retrieval of another associated feature, and that the more often a particular configuration of features occurs, the more effective this mutual cueing will be. Alternatively, the conditional probability of one feature given another feature may be neglected in memory for contextual features since they are not directly bound to one another. Three conjoint recognition experiments investigated whether memory for context is sensitive to the base-rates of features. Participants studied frequent versus infrequent configurations of features and, during the test, they were asked to recognise one of these features with (vs. without) another feature reinstated. The results showed that the context recollection parameter, representing the re-experience of contextual features in the dual-recollection model, was higher for frequent than infrequent feature configurations only when the binding of feature information was made easier and the differences in the base-rates were extreme, otherwise no difference was found. Similarly, base-rates of features influenced response guessing only in the condition with salient differences in base-rates. The Bayes factor analyses showed that the evidence from two of our experiments favoured the base-rate neglect hypothesis over the base-rate dependency hypothesis; the opposite result was obtained in the third experiment, but only when high base-rate disproportion and facilitated feature binding conditions were used.

Sequential Decision Making in Beach Volleyball-A Mixed-Method Approach.

Which opponent player to sequentially serve to in beach volleyball is crucial given the advantage of the attacking team. The sequential choice theory was tested in three studies by analyzing allocation strategies based on the hot hand belief. Study 1 showed strong belief in the hot hand of national coaches. In Study 2, we analyzed Tokyo Olympics data to explore how base rates and sequential selection rates varied in an elite sample. When base rates of players differed by 0.25, low-performing players were frequently selected. In an experiment with elite athletes, Study 3A demonstrated accurate base-rate-difference recognition but low base-rate-change recognition. Study 3B found that the hot hand is believed to be important but is not often detected. We conclude that players and coaches follow predictions of the sequential choice theory and believe in the hot hand, but do not have a shared understanding of how to use it.

Empirical Bayes Methods, Evidentialism, and the Inferential Roles They Play.

Empirical Bayes-based Methods (EBM) is an increasingly popular form of Objective Bayesianism (OB). It is identified in particular with the statistician Bradley Efron. The main aims of this paper are, first, to describe and illustrate its main features and, second, to locate its role by comparing it with two other statistical paradigms, Subjective Bayesianism (SB) and Evidentialism. EBM's main formal features are illustrated in some detail by schematic examples. The comparison between what Efron calls their underlying "philosophies" is by way of a distinction made between confirmation and evidence. Although this distinction is sometimes made in the statistical literature, it is relatively rare and never to the same point as here. That is, the distinction is invariably spelled out intra- and not inter-paradigmatically solely in terms of one or the other accounts. The distinction made in this paper between confirmation and evidence is illustrated by two well-known statistical paradoxes: the base-rate fallacy and Popper's paradox of ideal evidence. The general conclusion reached is that each of the paradigms has a basic role to play and all are required by an adequate account of statistical inference from a technically informed and fine-grained philosophical perspective.

Base rate neglect and neural computations for subjective weight in decision under uncertainty.

Base rate neglect, an important bias in estimating probability of uncertain events, describes humans' tendency to underweight base rate (prior) relative to individuating information (likelihood). However, the neural mechanisms that give rise to this bias remain elusive. In this study, subjects chose between uncertain prospects where estimating reward probability was essential. We found that when the variability of prior and likelihood information about reward probability were systematically manipulated, prior variability significantly affected the degree to which subjects underweight the base rate of reward probability. Activity in the orbitofrontal cortex, medial prefrontal cortex, and putamen represented the relative subjective weight that reflected such bias. Further, sensitivity to likelihood relative to prior variability in the putamen correlated with individuals' overall tendency to underweight base rate. These findings suggest that in combining prior and likelihood, relative sensitivity to information variability and subjective-weight computations critically contribute to the individual heterogeneity in base rate neglect.

Neural correlates of the inverse base rate effect.

The inverse base rate effect (IBRE) is a nonrational behavioral phenomenon in predictive learning. Canonically, participants learn that the AB stimulus compound leads to one outcome and that AC leads to another outcome, with AB being presented three times as often as AC. When subsequently presented with BC, the outcome associated with AC is preferentially selected, in opposition to the underlying base rates of the outcomes. The current leading explanation is based on error-driven learning. A key component of this account is prediction error, a concept previously linked to a number of brain areas including the anterior cingulate, the striatum, and the dorsolateral prefrontal cortex. The present work is the first fMRI study to directly examine the IBRE. Activations were noted in brain areas linked to prediction error, including the caudate body, the anterior cingulate, the ventromedial prefrontal cortex, and the right dorsolateral prefrontal cortex. Analyzing the difference in activations for singular key stimuli (B and C), as well as frequency matched controls, supports the predictions made by the error-driven learning account.

The Value of Bayesian Methods for Accurate and Efficient Neuropsychological Assessment.

OBJECTIVE: Clinical neuropsychology has been slow in adopting novelties in psychometrics, statistics, and technology. Researchers have indicated that the stationary nature of clinical neuropsychology endangers its evidence-based character. In addition to a technological crisis, there may be a statistical crisis affecting clinical neuropsychology. That is, the frequentist null hypothesis significance testing framework remains the dominant approach in clinical practice, despite a recent surge in critique on this framework. While the Bayesian framework has been put forward as a viable alternative in psychology in general, the possibilities it offers to clinical neuropsychology have not received much attention. METHOD: In the current position paper, we discuss and reflect on the value of Bayesian methods for the advancement of evidence-based clinical neuropsychology. RESULTS: We aim to familiarize clinical neuropsychologists and neuropsychological researchers to Bayesian methods of inference and provide a clear rationale for why these methods are valuable for clinical neuropsychology. CONCLUSION: We argue that Bayesian methods allow for a more intuitive answer to our diagnostic questions and form a more solid foundation for sequential and adaptive diagnostic testing, representing uncertainty about patients' observed test scores and cognitive modeling of test results.

Can a fast thinker be a good thinker? The neural correlates of base-rate neglect measured using a two-response paradigm.

Traditionally, it has been assumed that logical thinking requires deliberation. However, people can also make logical responses quickly, exhibiting logical intuitions. We examined the neural correlates of logical intuitions by administering base rate problems during fMRI scanning using a two-response paradigm where participants first responded quickly and then reflectively to problems that did or did not pit a normative response against an intuitively-cued stereotypical response (i.e., conflict vs. non-conflict problems). As predicted, participants were less likely to make judgments in accordance with base rates on conflict problems. Critically, in only 4% of cases did longer deliberation change an initially biased response to a normatively correct response. The fMRI data revealed that intuitively-made initial biased judgments nevertheless activate regions typically involved in cognitive control, executive functions and attention, including anterior, inferior, middle and superior frontal cortex, suggesting that even when errors are made, there might be very early awareness of conflict.

On Ambiguity Reduction and the Role of Decision Analysis during the Pandemic.

The COVID-19 pandemic has created a multitude of decision problems for a variety of fields. Questions from the seriousness and breadth of the problem to the effectiveness of proposed mitigation measures have been raised. We assert that the decision sciences have a crucial role to play here, as the questions requiring answers involve complex decision making under both uncertainty and ambiguity. The collection, processing, and analysis of data is critical in providing a useful response-especially as information of fundamental importance to such decision making (base rates and transmission rates) is lacking. We propose that scarce testing resources should be diverted away from confirmatory analysis of symptomatic people, as laboratory diagnosis appears to have little decision value in treatment choice over clinical diagnosis in patients presenting with symptoms. In contrast, the exploratory use of testing resources to reduce ambiguity in estimates of the base rate of infection appears to have significant value and great practical import for public policy purposes. As these stances may be at odds with triage practices among medical practitioners, they highlight the important role the decision analyst can play in responding to the challenges of the COVID-19 pandemic.

Measures of effectiveness in medical research: Reporting both absolute and relative measures.

Biomedical research, especially pharmaceutical research, has been criticised for engaging in practices that lead to over-estimations of the effectiveness of medical treatments. A central issue concerns the reporting of absolute and relative measures of medical effectiveness. In this paper we critically examine proposals made by Jacob Stegenga to (a) give priority to the reporting of absolute measures over relative measures, and (b) downgrade the measures of effectiveness (effect sizes) of the treatments tested in clinical trials (Stegenga, 2015a). After exposing significant flaws in a central case study used by Stegenga to bolster his first proposal (a), we go on to argue that neither of these proposals is defensible (a or b). We defend the practice, in line with the New England Journal of Medicine, of reporting both absolute and relative measures whenever feasible.

Discrepancy-Based Evidence for Loss of Thinking Abilities (DELTA): Development and Validation of a Novel Approach to Identifying Cognitive Changes.

OBJECTIVE: To develop and validate the Discrepancy-based Evidence for Loss of Thinking Abilities (DELTA) score. The DELTA score characterizes the strength of evidence for cognitive decline on a continuous spectrum using well-established psychometric principles for improving detection of cognitive changes. METHODS: DELTA score development used neuropsychological test scores from the Alzheimer's Disease Neuroimaging Initiative (ADNI) cohort (two tests each from Memory, Executive Function, and Language domains). We derived regression-based normative reference scores using age, gender, years of education, and word-reading ability from robust cognitively normal ADNI participants. Discrepancies between predicted and observed scores were used for calculating the DELTA score (range 0-15). We validated DELTA scores primarily against longitudinal Clinical Dementia Rating-Sum of Boxes (CDR-SOB) and Functional Activities Questionnaire (FAQ) scores (baseline assessment through Year 3) using linear mixed models and secondarily against cross-sectional Alzheimer's biomarkers. RESULTS: There were 1359 ADNI participants with calculable baseline DELTA scores (age 73.7 ± 7.1 years, 55.4% female, 100% white/Caucasian). Higher baseline DELTA scores (stronger evidence of cognitive decline) predicted higher baseline CDR-SOB (ΔR2 = .318) and faster rates of CDR-SOB increase over time (ΔR2 = .209). Longitudinal changes in DELTA scores tracked closely and in the same direction as CDR-SOB scores (fixed and random effects of mean + mean-centered DELTA, ΔR2 > .7). Results were similar for FAQ scores. High DELTA scores predicted higher PET-Aß SUVr (ρ = 324), higher CSF-pTau/CSF-Aß ratio (ρ = .460), and demonstrated PPV > .9 for positive Alzheimer's disease biomarker classification. CONCLUSIONS: Data support initial development and validation of the DELTA score through its associations with longitudinal functional changes and Alzheimer's biomarkers. We provide several considerations for future research and include an automated scoring program for clinical use.

The base rate neglect in episodic memory.

The main objective of the present research was, for the first time, to assess a potential similarity in the representational bases for the base rate neglect in memory versus conditional probability judgment. Participants learned target and filler words, each of which was presented on a separate list (List 1 or List 2) and in a distinct colour (red or blue), with a manipulation of different base rates for these list and colour categories. During recognition tests, participants made prior and posterior episodic judgments (e.g., "What colour was the word?", "Given that the word was in red, on which list was the word?") on the target words, which respectively parallel independent and conditional probability assessments that figure in Bayes' theorem. The results implied that biased prior and posterior judgments presumably cause the base rate neglect, inasmuch as the prior cue of a low (high) base rate is likely to lead to a bias toward retrieving high (low) base rate posterior evidence. There was also a finding showing that memory analogues of probability estimates reflect the base rate neglect in both low and high base rate categories, but is presumably stronger with posterior judgment of Colour|List than List|Colour relative to the high base rate category.

Not All Information Is Equal: Effects of Disclosing Different Types of Likelihood Information on Trust, Compliance and Reliance, and Task Performance in Human-Automation Teaming.

OBJECTIVE: The study examines the effects of disclosing different types of likelihood information on human operators' trust in automation, their compliance and reliance behaviors, and the human-automation team performance. BACKGROUND: To facilitate appropriate trust in and dependence on automation, explicitly conveying the likelihood of automation success has been proposed as one solution. Empirical studies have been conducted to investigate the potential benefits of disclosing likelihood information in the form of automation reliability, (un)certainty, and confidence. Yet, results from these studies are rather mixed. METHOD: We conducted a human-in-the-loop experiment with 60 participants using a simulated surveillance task. Each participant performed a compensatory tracking task and a threat detection task with the help of an imperfect automated threat detector. Three types of likelihood information were presented: overall likelihood information, predictive values, and hit and correct rejection rates. Participants' trust in automation, compliance and reliance behaviors, and task performance were measured. RESULTS: Human operators informed of the predictive values or the overall likelihood value, rather than the hit and correct rejection rates, relied on the decision aid more appropriately and obtained higher task scores. CONCLUSION: Not all likelihood information is equal in aiding human-automation team performance. Directly presenting the hit and correct rejection rates of an automated decision aid should be avoided. APPLICATION: The findings can be applied to the design of automated decision aids.

The development of the representativeness heuristic in young children.

In classic examinations of the representativeness heuristic, Kahneman and Tversky (1973) presented adult participants with a description of an individual who fit their stereotype of a typical engineer. Importantly, even when participants were told that the individual was drawn from a sample of 70 lawyers and 30 engineers, they estimated that the individual was an engineer at very high levels, showing that they relied almost exclusively on the personality description. Relying on the representativeness heuristic can lead to base-rate neglect and, thus, biased judgments. Two experiments provide insight into the development of the representativeness heuristic in young children using an adaptation of the classic lawyer-engineer problem. Experiment 1 (N = 96) established that 3- to 5-year-olds can use base-rate information on its own, and 4- and 5-year-olds can use individuating information on its own, to make inferences. Experiment 2 (N = 192) varied the relevance of the individuating information across conditions to assess the pervasiveness of this bias early in development. Here 5- and 6-year-olds, much like adults, continue to attempt to rely on individuating information when making classifications even if that information is irrelevant. Together, these experiments reveal how the representativeness heuristic develops across the preschool years and suggest that the bias may strengthen between 4 and 6 years of age.

Effects of outcome and trial frequency on the inverse base-rate effect.

The inverse base-rate effect is a bias in contingency learning in which participants tend to predict a rare outcome for a conflicting set of perfectly predictive cues. Although the effect is often explained by attention biases during learning, inferential strategies at test may also contribute substantially to the effect. In three experiments, we manipulated the frequencies of outcomes and trial types to determine the critical conditions for the effect, thereby providing novel tests of the reasoning processes that could contribute to it. The rare bias was substantially reduced when the outcomes were experienced at equal rates in the presence of predictive-cue frequency differences (Exp. 2), and when the predictive cues were experienced at equal rates in the presence of outcome frequency differences (Exp. 3). We also found a consistent common-outcome bias for novel cue compounds. The results indicate the importance of both cue and outcome frequencies to the inverse base-rate effect, and reveal a combination of necessary conditions that are not well captured by appealing to inferential strategies at test. Although both attention-based and inferential theories explain some aspects of these data, no existing theory fully accounts for these effects of relative novelty.

Kuhnian theory-choice and virtue convergence: Facing the base rate fallacy.

Perhaps the strongest argument for scientific realism, the no-miracles-argument, has been said to commit the so-called base rate fallacy. The apparent elusiveness of the base rate of true theories has even been said to undermine the rationality of the entire realism debate. On the basis of the Kuhnian picture of theory choice, I confront this challenge by arguing that a theory is likely to be true if it possesses multiple theoretical virtues and is embraced by numerous scientists-even when the base rate converges to zero.

Screening for Mental Health Problems: Addressing the Base Rate Fallacy for a Sustainable Screening Program in Integrated Primary Care.

OBJECTIVE: The Affordable Care Act has stimulated interest in screening for psychological problems in primary care. Given the scale with which screening might occur, the implications of a problem known as the base rate fallacy need to be considered. METHODS: The concepts of sensitivity and specificity, positive and negative predictive value, and the base rate fallacy are discussed. The possibility that a screening program may not improve upon random selection is reviewed, as is the possibility that sequential screening might be useful. RESULTS AND DISCUSSION: Developing effective screening programs for pediatric mental health problems is highly desirable, and properly addressing the high rate of false positives may improve the likelihood that such programs can be sustained. Consideration needs to be given to the use of sequential screening, which has both advantages and disadvantages, depending upon the type of problem to be screened for and the availability of resources for follow-up evaluations.

The influence of base rates on correlations: An evaluation of proposed alternative effect sizes with real-world data.

Correlations are the simplest and most commonly understood effect size statistic in psychology. The purpose of the current paper was to use a large sample of real-world data (109 correlations with 60,415 participants) to illustrate the base rate dependence of correlations when applied to dichotomous or ordinal data. Specifically, we examined the influence of the base rate on different effect size metrics. Correlations decreased when the dichotomous variable did not have a 50 % base rate. The higher the deviation from a 50 % base rate, the smaller the observed Pearson's point-biserial and Kendall's tau correlation coefficients. In contrast, the relationship between base rate deviations and the more commonly proposed alternatives (i.e., polychoric correlation coefficients, AUCs, Pearson/Thorndike adjusted correlations, and Cohen's d) were less remarkable, with AUCs being most robust to attenuation due to base rates. In other words, the base rate makes a marked difference in the magnitude of the correlation. As such, when using dichotomous data, the correlation may be more sensitive to base rates than is optimal for the researcher's goals. Given the magnitude of the association between the base rate and point-biserial correlations (r = -.81) and Kendall's tau (r = -.80), we recommend that AUCs, Pearson/Thorndike adjusted correlations, Cohen's d, or polychoric correlations should be considered as alternate effect size statistics in many contexts.

What makes us think? A three-stage dual-process model of analytic engagement.

The distinction between intuitive and analytic thinking is common in psychology. However, while often being quite clear on the characteristics of the two processes ('Type 1' processes are fast, autonomous, intuitive, etc. and 'Type 2' processes are slow, deliberative, analytic, etc.), dual-process theorists have been heavily criticized for being unclear on the factors that determine when an individual will think analytically or rely on their intuition. We address this issue by introducing a three-stage model that elucidates the bottom-up factors that cause individuals to engage Type 2 processing. According to the model, multiple Type 1 processes may be cued by a stimulus (Stage 1), leading to the potential for conflict detection (Stage 2). If successful, conflict detection leads to Type 2 processing (Stage 3), which may take the form of rationalization (i.e., the Type 1 output is verified post hoc) or decoupling (i.e., the Type 1 output is falsified). We tested key aspects of the model using a novel base-rate task where stereotypes and base-rate probabilities cued the same (non-conflict problems) or different (conflict problems) responses about group membership. Our results support two key predictions derived from the model: (1) conflict detection and decoupling are dissociable sources of Type 2 processing and (2) conflict detection sometimes fails. We argue that considering the potential stages of reasoning allows us to distinguish early (conflict detection) and late (decoupling) sources of analytic thought. Errors may occur at both stages and, as a consequence, bias arises from both conflict monitoring and decoupling failures.