A complete method for assessing the effectiveness of eyewitness identification procedures: Expected information gain.

We present a method for measuring the efficacy of eyewitness identification procedures by applying fundamental principles of information theory. The resulting measure evaluates the expected information gain (EIG) for an identification attempt, a single value that summarizes an identification procedure's overall potential for reducing uncertainty about guilt or innocence across all possible witness responses. In a series of demonstrations, we show that EIG often disagrees with existing measures (e.g., diagnosticity ratios or area under the receiver operating characteristic) about the relative effectiveness of different identification procedures. Each demonstration is designed to highlight key distinctions between existing measures and EIG. An overarching theme is that EIG provides a complete measure of evidentiary value, in the sense that it factors in all aspects of identification performance. Collectively, these demonstrations show that EIG has substantial potential to inspire new discoveries in eyewitness research and provide a new perspective on policy recommendations for the use of identifications in real investigations. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

sdtlu: An R package for the signal detection analysis of eyewitness lineup data.

In a standard eyewitness lineup scenario, a witness observes a culprit commit a crime and is later asked to identify the culprit from a set of faces, the lineup. Signal detection theory (SDT), a powerful modeling framework for analyzing data, has recently become a common way to analyze lineup data. The goal of this paper is to introduce a new R package, sdtlu (Signal Detection Theory - LineUp), that streamlines and automates the SDT analysis of lineup data. sdtlu provides functions to process lineup data, determine the best-fitting SDT parameters, compute model-based performance measures such as area under the curve (AUC) and diagnosticity, use bootstrapping to determine uncertainty intervals around these parameters and measures, and compare parameters across two different data sets. The package incorporates closed-form solutions for both simultaneous and sequential lineups that allow for model-based analyses without Monte Carlo simulation. Show-ups are also supported. The package can estimate the base-rate of lineups that include a guilty suspect when the guilt or innocence of each suspect in the data set is unknown, as in "real-world" lineups. The package can also produce a full set of graphs, including data and model-based ROC curves and the underlying SDT model.

Estimating the proportion of guilty suspects and posterior probability of guilt in lineups using signal-detection models.

BACKGROUND: The majority of eyewitness lineup studies are laboratory-based. How well the conclusions of these studies, including the relationship between confidence and accuracy, generalize to real-world police lineups is an open question. Signal detection theory (SDT) has emerged as a powerful framework for analyzing lineups that allows comparison of witnesses' memory accuracy under different types of identification procedures. Because the guilt or innocence of a real-world suspect is generally not known, however, it is further unknown precisely how the identification of a suspect should change our belief in their guilt. The probability of guilt after the suspect has been identified, the posterior probability of guilt (PPG), can only be meaningfully estimated if we know the proportion of lineups that include a guilty suspect, P(guilty). Recent work used SDT to estimate P(guilty) on a single empirical data set that shared an important property with real-world data; that is, no information about the guilt or innocence of the suspects was provided. Here we test the ability of the SDT model to recover P(guilty) on a wide range of pre-existing empirical data from more than 10,000 identification decisions. We then use simulations of the SDT model to determine the conditions under which the model succeeds and, where applicable, why it fails. RESULTS: For both empirical and simulated studies, the model was able to accurately estimate P(guilty) when the lineups were fair (the guilty and innocent suspects did not stand out) and identifications of both suspects and fillers occurred with a range of confidence levels. Simulations showed that the model can accurately recover P(guilty) given data that matches the model assumptions. The model failed to accurately estimate P(guilty) under conditions that violated its assumptions; for example, when the effective size of the lineup was reduced, either because the fillers were selected to be poor matches to the suspect or because the innocent suspect was more familiar than the guilty suspect. The model also underestimated P(guilty) when a weapon was shown. CONCLUSIONS: Depending on lineup quality, estimation of P(guilty) and, relatedly, PPG, from the SDT model can range from poor to excellent. These results highlight the need to carefully consider how the similarity relations between fillers and suspects influence identifications.

Do modals identify better models? A comparison of signal detection and probabilistic models of inductive reasoning.

The nature of the relationship between deductive and inductive reasoning is a hotly debated topic. A key question is whether there is a single dimension of evidence underlying both deductive and inductive judgments. Following Rips (2001), Rotello and Heit (2009) and Heit and Rotello (2010) implemented one- and two-dimensional models grounded in signal detection theory to assess predictions for receiver operating characteristic data (ROCs), and concluded in favor of the two-dimensional model. Recently, Lassiter and Goodman (2015) proposed a different type of one-dimensional model, the Probability Threshold Model (PTM), that they concluded offered a good account of data collected over a range of decision modals (e.g., How likely, possible, or necessary is the argument conclusion?). Here, we apply the PTM and the signal detection models to ROCs from 3 large experiments in which participants made judgments about arguments varying in terms of modals introduced by Lassiter and Goodman (2015). Two independent variables that are theoretically important for the study of inductive reasoning, namely premise-conclusion similarity (as utilized in Heit & Rotello, 2010) and number of premises (as utilized in Rotello & Heit, 2009), are also varied in Experiments 1 and 2, respectively. In all cases, the PTM provides the poorest fit both quantitatively and qualitatively; the two-dimensional signal detection model is preferred.

A new argument for co-active parses during language comprehension.

One perennially important question for theories of sentence comprehension is whether the human sentence processing mechanism is parallel (i.e., it simultaneously represents multiple syntactic analyses of linguistic input) or serial (i.e., it constructs only a single analysis at a time). Despite its centrality, this question has proven difficult to address for both theoretical and methodological reasons (Gibson & Pearlmutter, 2000; Lewis, 2000). In the present study, we reassess this question from a novel perspective. We investigated the well-known ambiguity advantage effect (Traxler, Pickering, & Clifton, 1998) in a speeded acceptability judgment task. We adopted a signal detection theoretic approach to these data, with the goal of determining whether speeded judgment responses were conditioned on one or multiple syntactic analyses. To link these results to incremental parsing models, we developed formal models to quantitatively evaluate how serial and parallel parsing models should impact perceived sentence acceptability in our task. Our results suggest that speeded acceptability judgments are jointly conditioned on multiple parses of the input, a finding that is overall more consistent with parallel parsing models than serial models. Our study thus provides a new, psychophysical argument for coactive parses during language comprehension. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Putting replication in its place.

Direct replication is valuable but should not be elevated over other worthwhile research practices, including conceptual replication and checking of statistical assumptions. As noted by Rotello et al. (2015), replicating studies without checking the statistical assumptions can lead to increased confidence in incorrect conclusions. Finally, successful replications should not be elevated over failed replications, given that both are informative.

Fingerprinting antibiotics with PAE-based fluorescent sensor arrays.

We outline an evolution process for tongue elements composed of poly(p-aryleneethynylene)s (PAE) and detergents, resulting in a chemical tongue (24 elements) that discerns antibiotics. Cross-breeding of this new tongue with tongue elements that consist of simple poly(p-phenyleneethynylene)s (PPE) at different pH-values leads to an enlarged sensor array, composed of 30 elements. This tongue was pruned, employing principal component analysis. We find that a filial tongue featuring three elements from each original array (i.e. a six element tongue) is superior to either of the prior tongues and the composite tongue in the discrimination of structurally different antibiotics. Such a selection process should be general and give an idea how to successfully generate powerful low-selectivity sensor elements and configure them into discriminative chemical tongues.

Cancer Cell Discrimination Using Host-Guest "Doubled" Arrays.

We report a nanosensor that uses cell lysates to rapidly profile the tumorigenicity of cancer cells. This sensing platform uses host-guest interactions between cucurbit[7]uril and the cationic headgroup of a gold nanoparticle to non-covalently modify the binding of three fluorescent proteins of a multi-channel sensor in situ. This approach doubles the number of output channels to six, providing single-well identification of cell lysates with 100% accuracy. Significantly, this classification could be extended beyond the training set, determining the invasiveness of novel cell lines. The unique fingerprint of these cell lysates required minimal sample quantity (200 ng, ∼1000 cells), making the methodology compatible with microbiopsy technology.

ROC curve analyses of eyewitness identification decisions: An analysis of the recent debate.

How should the accuracy of eyewitness identification decisions be measured, so that best practices for identification can be determined? This fundamental question is under intense debate. One side advocates for continued use of a traditional measure of identification accuracy, known as the diagnosticity ratio, whereas the other side argues that receiver operating characteristic curves (ROCs) should be used instead because diagnosticity is confounded with response bias. Diagnosticity proponents have offered several criticisms of ROCs, which we show are either false or irrelevant to the assessment of eyewitness accuracy. We also show that, like diagnosticity, Bayesian measures of identification accuracy confound response bias with witnesses' ability to discriminate guilty from innocent suspects. ROCs are an essential tool for distinguishing memory-based processes from decisional aspects of a response; simulations of different possible identification tasks and response strategies show that they offer important constraints on theory development.

When more data steer us wrong: replications with the wrong dependent measure perpetuate erroneous conclusions.

There is a replication crisis in science, to which psychological research has not been immune: Many effects have proven uncomfortably difficult to reproduce. Although the reliability of data is a serious concern, we argue that there is a deeper and more insidious problem in the field: the persistent and dramatic misinterpretation of empirical results that replicate easily and consistently. Using a series of four highly studied "textbook" examples from different research domains (eyewitness memory, deductive reasoning, social psychology, and child welfare), we show how simple unrecognized incompatibilities among dependent measures, analysis tools, and the properties of data can lead to fundamental interpretive errors. These errors, which are not reduced by additional data collection, may lead to misguided research efforts and policy recommendations. We conclude with a set of recommended strategies and research tools to reduce the probability of these persistent and largely unrecognized errors. The use of receiver operating characteristic (ROC) curves is highlighted as one such recommendation.

A violation of the conditional independence assumption in the two-high-threshold model of recognition memory.

The 2-high-threshold (2HT) model of recognition memory assumes that test items result in distinct internal states: they are either detected or not, and the probability of responding at a particular confidence level that an item is "old" or "new" depends on the state-response mapping parameters. The mapping parameters are independent of the probability that an item yields a particular state (e.g., both strong and weak items that are detected as old have the same probability of producing a highest-confidence "old" response). We tested this conditional independence assumption by presenting nouns 1, 2, or 4 times. To maximize the strength of some items, "superstrong" items were repeated 4 times and encoded in conjunction with pleasantness, imageability, anagram, and survival processing tasks. The 2HT model failed to simultaneously capture the response rate data for all item classes, demonstrating that the data violated the conditional independence assumption. In contrast, a Gaussian signal detection model, which posits that the level of confidence that an item is "old" or "new" is a function of its continuous strength value, provided a good account of the data.

A multichannel nanosensor for instantaneous readout of cancer drug mechanisms.

Screening methods that use traditional genomic, transcriptional, proteomic and metabonomic signatures to characterize drug mechanisms are known. However, they are time consuming and require specialized equipment. Here, we present a high-throughput multichannel sensor platform that can profile the mechanisms of various chemotherapeutic drugs in minutes. The sensor consists of a gold nanoparticle complexed with three different fluorescent proteins that can sense drug-induced physicochemical changes on cell surfaces. In the presence of cells, fluorescent proteins are rapidly displaced from the gold nanoparticle surface and fluorescence is restored. Fluorescence 'turn on' of the fluorescent proteins depends on the drug-induced cell surface changes, generating patterns that identify specific mechanisms of cell death induced by drugs. The nanosensor is generalizable to different cell types and does not require processing steps before analysis, offering an effective way to expedite research in drug discovery, toxicology and cell-based sensing.

Memory, reasoning, and categorization: parallels and common mechanisms.

Traditionally, memory, reasoning, and categorization have been treated as separate components of human cognition. We challenge this distinction, arguing that there is broad scope for crossover between the methods and theories developed for each task. The links between memory and reasoning are illustrated in a review of two lines of research. The first takes theoretical ideas (two-process accounts) and methodological tools (signal detection analysis, receiver operating characteristic curves) from memory research and applies them to important issues in reasoning research: relations between induction and deduction, and the belief bias effect. The second line of research introduces a task in which subjects make either memory or reasoning judgments for the same set of stimuli. Other than broader generalization for reasoning than memory, the results were similar for the two tasks, across a variety of experimental stimuli and manipulations. It was possible to simultaneously explain performance on both tasks within a single cognitive architecture, based on exemplar-based comparisons of similarity. The final sections explore evidence for empirical and processing links between inductive reasoning and categorization and between categorization and recognition. An important implication is that progress in all three of these fields will be expedited by further investigation of the many commonalities between these tasks.

Recognition memory zROC slopes for items with correct versus incorrect source decisions discriminate the dual process and unequal variance signal detection models.

We tested the dual process and unequal variance signal detection models by jointly modeling recognition and source confidence ratings. The 2 approaches make unique predictions for the slope of the recognition memory zROC function for items with correct versus incorrect source decisions. The standard bivariate Gaussian version of the unequal variance model predicts little or no slope difference between the source-correct and source-incorrect functions. We also developed a "bounded" version of this model that did not permit below-chance source discrimination in any region of the evidence space. The bounded version predicts that the source-correct function should have a lower slope than the source-incorrect function. A bivariate version of the dual process signal detection model can predict slope differences in either direction, but it must predict a u-shaped source zROC function if the source-correct slope is lower than the source-incorrect slope. Across 4 experiments, results consistently showed that the recognition memory zROC function had a lower slope for items attributed to the correct source than items attributed to the incorrect source, and the source zROC function for words recognized with high confidence was linear. Only the bounded version of the unequal variance model successfully predicted the full pattern of results.

Traditional difference-score analyses of reasoning are flawed.

Studies of the belief bias effect in syllogistic reasoning have relied on three traditional difference score measures: the logic index, belief index, and interaction index. Dube, Rotello, and Heit (2010, 2011) argued that the interaction index incorrectly assumes a linear receiver operating characteristic (ROC). Here, all three measures are addressed. Simulations indicated that traditional analyses of reasoning experiments are likely to lead to incorrect conclusions. Two new experiments examined the role of instructional manipulations on the belief bias effect. The form of the ROCs violated assumptions of traditional measures. In comparison, signal detection theory (SDT) model-based analyses were a better match for the form of the ROCs, and implied that belief bias and instructional manipulations are predominantly response bias effects. Finally, reanalyses of previous studies of conditional reasoning also showed non-linear ROCs, violating assumptions of traditional analyses. Overall, reasoning research using traditional measures is at risk of drawing incorrect conclusions.

Memory bias for negative emotional words in recognition memory is driven by effects of category membership.

Recognition memory studies often find that emotional items are more likely than neutral items to be labelled as studied. Previous work suggests this bias is driven by increased memory strength/familiarity for emotional items. We explored strength and bias interpretations of this effect with the conjecture that emotional stimuli might seem more familiar because they share features with studied items from the same category. Categorical effects were manipulated in a recognition task by presenting lists with a small, medium or large proportion of emotional words. The liberal memory bias for emotional words was only observed when a medium or large proportion of categorised words were presented in the lists. Similar, though weaker, effects were observed with categorised words that were not emotional (animal names). These results suggest that liberal memory bias for emotional items may be largely driven by effects of category membership.

The statistical accuracy and theoretical status of discrete-state MPT models: reply to Batchelder and Alexander (2013).

In "A Critical Comparison of Discrete-State and Continuous Models of Recognition Memory: Implications for Recognition and Beyond," Pazzaglia, Dube, and Rotello (2013) explored the threshold multinomial processing tree (MPT) framework as applied to several domains of experimental psychology. Pazzaglia et al. concluded that threshold MPT analyses require assumptions at the representation and measurement levels that are contradicted by existing data in several domains. Furthermore, they showed that this flaw in the threshold MPT framework produces systematic errors in data interpretation. Pazzaglia et al. suggested measures derived from the empirically validated unequal-variance signal detection theory framework as a viable alternative and provided a simple tutorial for implementing such measures in an Excel spreadsheet. In their reply, Batchelder and Alexander (2013) disputed the conclusions advanced by Pazzaglia et al. Their arguments consisted of a small number of strong assertions, some of which were accompanied by references and/or data. In this reply, we demonstrate that both types of assertions--those with and without supporting references and/or data--are, at best, contradicted by several existing studies (many of which were already discussed in Pazzaglia et al., 2013) and, at worst, patently false. We conclude that the conclusions of Pazzaglia et al. are valid.

Paying attention to attention in recognition memory: insights from models and electrophysiology.

Reliance on remembered facts or events requires memory for their sources, that is, the contexts in which those facts or events were embedded. Understanding of source retrieval has been stymied by the fact that uncontrolled fluctuations of attention during encoding can cloud results of key importance to theoretical development. To address this issue, we combined electrophysiology (high-density electroencephalogram, EEG, recordings) with computational modeling of behavioral results. We manipulated subjects' attention to an auditory attribute, whether the source of individual study words was a male or female speaker. Posterior alpha-band (8-14 Hz) power in subjects' EEG increased after a cue to ignore the voice of the person who was about to speak. Receiver-operating-characteristic analysis validated our interpretation of oscillatory dynamics as a marker of attention to source information. With attention under experimental control, computational modeling showed unequivocally that memory for source (male or female speaker) reflected a continuous signal detection process rather than a threshold recollection process.