Boosting wisdom of the crowd for medical image annotation using training performance and task features.

A crucial bottleneck in medical artificial intelligence (AI) is high-quality labeled medical datasets. In this paper, we test a large variety of wisdom of the crowd algorithms to label medical images that were initially classified by individuals recruited through an app-based platform. Individuals classified skin lesions from the International Skin Lesion Challenge 2018 into 7 different categories. There was a large dispersion in the geographical location, experience, training, and performance of the recruited individuals. We tested several wisdom of the crowd algorithms of varying complexity from a simple unweighted average to more complex Bayesian models that account for individual patterns of errors. Using a switchboard analysis, we observe that the best-performing algorithms rely on selecting top performers, weighting decisions by training accuracy, and take into account the task environment. These algorithms far exceed expert performance. We conclude by discussing the implications of these approaches for the development of medical AI.

Can Invalid Information Be Ignored When It Is Detected?

With the rapid spread of information via social media, individuals are prone to misinformation exposure that they may utilize when forming beliefs. Over five experiments (total N = 815 adults, recruited through Amazon Mechanical Turk in the United States), we investigated whether people could ignore quantitative information when they judged for themselves that it was misreported. Participants recruited online viewed sets of values sampled from Gaussian distributions to estimate the underlying means. They attempted to ignore invalid information, which were outlier values inserted into the value sequences. Results indicated participants were able to detect outliers. Nevertheless, participants' estimates were still biased in the direction of the outlier, even when they were most certain that they detected invalid information. The addition of visual warning cues and different task scenarios did not fully eliminate systematic over- and underestimation. These findings suggest that individuals may incorporate invalid information they meant to ignore when forming beliefs.

Misinformation and Public Health Messaging in the Early Stages of the Mpox Outbreak: Mapping the Twitter Narrative With Deep Learning.

BACKGROUND: Shortly after the worst of the COVID-19 pandemic, an outbreak of mpox introduced another critical public health emergency. Like the COVID-19 pandemic, the mpox outbreak was characterized by a rising prevalence of public health misinformation on social media, through which many US adults receive and engage with news. Digital misinformation continues to challenge the efforts of public health officials in providing accurate and timely information to the public. We examine the evolving topic distributions of social media narratives during the mpox outbreak to map the tension between rapidly diffusing misinformation and public health communication. OBJECTIVE: This study aims to observe topical themes occurring in a large-scale collection of tweets about mpox using deep learning. METHODS: We leveraged a data set comprised of all mpox-related tweets that were posted between May 7, 2022, and July 23, 2022. We then applied Sentence Bidirectional Encoder Representations From Transformers (S-BERT) to the content of each tweet to generate a representation of its content in high-dimensional vector space, where semantically similar tweets will be located closely together. We projected the set of tweet embeddings to a 2D map by applying principal component analysis and Uniform Manifold Approximation Projection (UMAP). Finally, we group these data points into 7 topical clusters using k-means clustering and analyze each cluster to determine its dominant topics. We analyze the prevalence of each cluster over time to evaluate longitudinal thematic changes. RESULTS: Our deep-learning pipeline revealed 7 distinct clusters of content: (1) cynicism, (2) exasperation, (3) COVID-19, (4) men who have sex with men, (5) case reports, (6) vaccination, and (7) World Health Organization (WHO). Clusters that largely communicated erroneous or irrelevant information began earlier and grew faster, reaching a wider audience than later communications by official instances and health officials. CONCLUSIONS: Within a few weeks of the first reported mpox cases, an avalanche of mostly false, misleading, irrelevant, or damaging information started to circulate on social media. Official institutions, including the WHO, acted promptly, providing case reports and accurate information within weeks, but were overshadowed by rapidly spreading social media chatter. Our results point to the need for real-time monitoring of social media content to optimize responses to public health emergencies.

Financial constraint and perceptions of COVID-19.

In early March 2020, two crises emerged: the COVID-19 public health crisis and a corresponding economic crisis resulting from business closures and skyrocketing job losses. While the link between socioeconomic status and infectious disease is well-documented, the psychological relationships among economic considerations, such as financial constraint and economic anxiety, and health considerations, such as perceptions of disease spread and preventative actions, is not well understood. Despite past research illustrating the strong link between financial fragility and a wide range of behaviors, surprisingly little research has examined the psychological relationship between the economic crisis and beliefs and behaviors related to the co-occurring health crisis. We show that financial constraint predicts people's beliefs about both their personal risk of infection and the national spread of the virus as well as their social distancing behavior. In addition, we compare the predictive utility of financial constraint to two other commonly studied factors: political partisanship and local disease severity. We also show that negative affect partially mediates the relationship between financial constraint and COVID-19 beliefs and social distancing behaviors. These results suggest the economic crisis created by COVID-19 spilled over into people's beliefs about the health crisis and their behaviors.

The effect of preference learning on context effects in multi-alternative, multi-attribute choice.

Within the domain of preferential choice, it has long been thought that context effects, such as the attraction and compromise effects, arise due to the constructive nature of preferences and thus should not emerge when preferences are stable. We examined this hypothesis with a series of experiments where participants had the opportunity to experience selected alternatives and develop more enduring preferences. In our tasks, the options are presented in a description-based format so that participants need only learn their preferences for various options rather than the objective values of those options. Our results suggest that context effects can still emerge when stable preferences form through experience. This suggests that multi-alternative, multi-attribute decisions are likely influenced by relative evaluations, even when participants have the opportunity to experience options and learn their preferences. We hypothesize what was learned from experience in our tasks is the weights for various attributes. Through model simulations, we show that the observed choice patterns are well captured by a model with unequal attribute weights. A secondary finding is that the direction of observed context effects is opposite to standard effects and appears to be quite robust. Model simulations show that reserved effects can arise through various processes including representational noise and sensitivity to advantages and disadvantages when comparing options.

Advancing Research on Medical Image Perception by Strengthening Multidisciplinary Collaboration.

Medical image interpretation is central to detecting, diagnosing, and staging cancer and many other disorders. At a time when medical imaging is being transformed by digital technologies and artificial intelligence, understanding the basic perceptual and cognitive processes underlying medical image interpretation is vital for increasing diagnosticians' accuracy and performance, improving patient outcomes, and reducing diagnostician burnout. Medical image perception remains substantially understudied. In September 2019, the National Cancer Institute convened a multidisciplinary panel of radiologists and pathologists together with researchers working in medical image perception and adjacent fields of cognition and perception for the "Cognition and Medical Image Perception Think Tank." The Think Tank's key objectives were to identify critical unsolved problems related to visual perception in pathology and radiology from the perspective of diagnosticians, discuss how these clinically relevant questions could be addressed through cognitive and perception research, identify barriers and solutions for transdisciplinary collaborations, define ways to elevate the profile of cognition and perception research within the medical image community, determine the greatest needs to advance medical image perception, and outline future goals and strategies to evaluate progress. The Think Tank emphasized diagnosticians' perspectives as the crucial starting point for medical image perception research, with diagnosticians describing their interpretation process and identifying perceptual and cognitive problems that arise. This article reports the deliberations of the Think Tank participants to address these objectives and highlight opportunities to expand research on medical image perception.

Improving Medical Image Decision-Making by Leveraging Metacognitive Processes and Representational Similarity.

Improving the accuracy of medical image interpretation can improve the diagnosis of numerous diseases. We compared different approaches to aggregating repeated decisions about medical images to improve the accuracy of a single decision maker. We tested our algorithms on data from both novices (undergraduates) and experts (medical professionals). Participants viewed images of white blood cells and made decisions about whether the cells were cancerous or not. Each image was shown twice to the participants and their corresponding confidence judgments were collected. The maximum confidence slating (MCS) algorithm leverages metacognitive abilities to consider the more confident response in the pair of responses as the more accurate "final response" (Koriat, 2012), and it has previously been shown to improve accuracy on our task for both novices and experts (Hasan et al., 2021). We compared MCS to similarity-based aggregation (SBA) algorithms where the responses made by the same participant on similar images are pooled together to generate the "final response." We determined similarity by using two different neural networks where one of the networks had been trained on white blood cells and the other had not. We show that SBA improves performance for novices even when the neural network had no specific training on white blood cell images. Using an informative representation (i.e., network trained on white blood cells) allowed one to aggregate over more neighbors and further boosted the performance of novices. However, SBA failed to improve the performance for experts even with the informative representation. This difference in efficacy of the SBA suggests different decision mechanisms for novices and experts.

Context effects in similarity judgments.

Tversky's (1977) famous demonstration of a diagnosticity effect indicates that the similarity between the same two stimuli depends on the presence of contextual stimuli. In a forced choice task, the similarity between a target and a choice, appears to depend on the other choices. Specifically, introducing a distractor grouped with one of the options would reduce preference for the grouped option. However, the diagnosticity effect has been difficult to replicate, casting doubt on its robustness and our understanding of contextual effects in similarity generally. We propose that the apparent brittleness of the diagnosticity effect is because it is in competition with an opposite attraction effect. Even though in both the similarity and decision-making literatures there are indications for such a competition, we provide the first direct experimental demonstration of how an attraction effect can give way to a diagnosticity one, as a distractor option is manipulated. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

When a gain becomes a loss: The effect of wealth predictions on financial decisions.

When people make financial decisions, they need not only think about their current financial situation, but also about changes in future wealth. This work investigates people's beliefs about their future wealth and how these beliefs impact financial decisions. Using a joint experimental and computational cognitive modeling approach, we show that people's future beliefs serve as reference points when making investment decisions. These results are further supported by data from a large-scale cross-sectional survey (n = 4606) showing that people's beliefs about the future value of their assets are related to investment decisions between risky (i.e., stock market index) and safe (i.e., bond earning a fixed amount per year) options. In both the experiments and survey, we hypothesize that outcomes that are nominally stated as sure gains can become coded as losses due to belief-based reference points. This pattern leads to an increase in riskier choices across positive outcomes for individuals with optimistic beliefs about their future wealth.

Disentangling prevalence induced biases in medical image decision-making.

Many important real-world decision tasks involve the detection of rarely occurring targets (e.g., weapons in luggage, potentially cancerous abnormalities in radiographs). Over the past decade, it has been repeatedly demonstrated that extreme prevalence (both high and low) leads to an increase in errors. While this "prevalence effect" is well established, the cognitive and/or perceptual mechanisms responsible for it are not. One reason for this is that the most common tool for analyzing prevalence effects, Signal Detection Theory, cannot distinguish between different biases that might be present. Through an application to pathology image-based decision-making, we illustrate that an evidence accumulation modeling framework can be used to disentangle different types of biases. Importantly, our results show that prevalence influences both response expectancy and stimulus evaluation biases, with novices (students, N = 96) showing a more pronounced response expectancy bias and experts (medical laboratory professionals, N = 19) showing a more pronounced stimulus evaluation bias.

Urgency, leakage, and the relative nature of information processing in decision-making.

Over the last decade, there has been a robust debate in decision neuroscience and psychology about what mechanism governs the time course of decision-making. Historically, the most prominent hypothesis is that neural architectures accumulate information over time until some threshold is met, the so-called Evidence Accumulation hypothesis. However, most applications of this theory rely on simplifying assumptions, belying a number of potential complexities. Is changing stimulus information perceived and processed in an independent manner or is there a relative component? Does urgency play a role? What about evidence leakage? Although the latter questions have been the subject of recent investigations, most studies to date have been piecemeal in nature, addressing one aspect of the decision process or another. Here we develop a modeling framework, an extension of the Urgency Gating Model, in conjunction with a changing information experimental paradigm to simultaneously probe these aspects of the decision process. Using state-of-the-art Bayesian methods to perform parameter-based inference, we find that (a) information processing is relative with early information influencing the perception of late information, (b) time varying urgency and evidence accumulation are of roughly equal strength in the decision process, and (c) leakage is present with a time scale of ∼200-250 ms. We also show that these effects can only be identified in a changing information paradigm. To our knowledge, this is the first comprehensive study to utilize a changing information paradigm to jointly and quantitatively estimate the temporal dynamics of human decision-making. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

How do insects choose flowers? A review of multi-attribute flower choice and decoy effects in flower-visiting insects.

Understanding why animals (including humans) choose one thing over another is one of the key questions underlying the fields of behavioural ecology, behavioural economics and psychology. Most traditional studies of food choice in animals focus on simple, single-attribute decision tasks. However, animals in the wild are often faced with multi-attribute choice tasks where options in the choice set vary across multiple dimensions. Multi-attribute decision-making is particularly relevant for flower-visiting insects faced with deciding between flowers that may differ in reward attributes such as sugar concentration, nectar volume and pollen composition as well as non-rewarding attributes such as colour, symmetry and odour. How do flower-visiting insects deal with complex multi-attribute decision tasks? Here we review and synthesise research on the decision strategies used by flower-visiting insects when making multi-attribute decisions. In particular, we review how different types of foraging frameworks (classic optimal foraging theory, nutritional ecology, heuristics) conceptualise multi-attribute choice and we discuss how phenomena such as innate preferences, flower constancy and context dependence influence our understanding of flower choice. We find that multi-attribute decision-making is a complex process that can be influenced by innate preferences, flower constancy, the composition of the choice set and economic reward value. We argue that to understand and predict flower choice in flower-visiting insects, we need to move beyond simplified choice sets towards a view of multi-attribute choice which integrates the role of non-rewarding attributes and which includes flower constancy, innate preferences and context dependence. We further caution that behavioural experiments need to consider the possibility of context dependence in the design and interpretation of preference experiments. We conclude with a discussion of outstanding questions for future research. We also present a conceptual framework that incorporates the multiple dimensions of choice behaviour.

A parameter recovery assessment of time-variant models of decision-making.

Evidence accumulation models have been one of the most dominant modeling frameworks used to study rapid decision-making over the past several decades. These models propose that evidence accumulates from the environment until the evidence for one alternative reaches some threshold, typically associated with caution, triggering a response. However, researchers have recently begun to reconsider the fundamental assumptions of how caution varies with time. In the past, it was typically assumed that levels of caution are independent of time. Recent investigations have however suggested the possibility that levels of caution decrease over time and that this strategy provides more efficient performance under certain conditions. Our study provides the first comprehensive assessment of this newer class of models accounting for time-varying caution to determine how robustly their parameters can be estimated. We assess five overall variants of collapsing threshold/urgency signal models based on the diffusion decision model, linear ballistic accumulator model, and urgency gating model frameworks. We find that estimation of parameters, particularly those associated with caution/urgency modulation are most robust for the linearly collapsing threshold diffusion model followed by an urgency-gating model with a leakage process. All other models considered, particularly those with ballistic accumulation or nonlinear thresholds, are unable to recover their own parameters adequately, making their usage in parameter estimation contexts questionable.

Automatic biases in intertemporal choice.

Dual process theories of intertemporal decision making propose that decision makers automatically favor immediate rewards. In this paper, we use a drift diffusion model to implement these theories, and empirically investigate the role of their proposed automatic biases. Our model permits automatic biases in the response process, in the form of a shifted starting point, as well as automatic biases in the evaluation process, in the form of an additive drift rate intercept. We fit our model to individual-level choice and response time data, and find that automatic biases (as measured though the starting point and drift rate intercept in our model) are prevalent in intertemporal choice, but that the type, magnitude, and direction of these biases vary greatly across individuals. Our results pose new challenges for theories of intertemporal choice behavior.

Response-time data provide critical constraints on dynamic models of multi-alternative, multi-attribute choice.

Understanding the cognitive processes involved in multi-alternative, multi-attribute choice is of interest to a wide range of fields including psychology, neuroscience, and economics. Prior investigations in this domain have relied primarily on choice data to compare different theories. Despite numerous such studies, results have largely been inconclusive. Our study uses state-of-the-art response-time modeling and data from 12 different experiments appearing in six different published studies to compare four previously proposed theories/models of these effects: multi-alternative decision field theory (MDFT), the leaky-competing accumulator (LCA), the multi-attribute linear ballistic accumulator (MLBA), and the associative accumulation model (AAM). All four models are, by design, dynamic process models and thus a comprehensive evaluation of their theoretical properties requires quantitative evaluation with both choice and response-time data. Our results show that response-time data is critical at distinguishing among these models and that using choice data alone can lead to inconclusive results for some datasets. In conclusion, we encourage future research to include response-time data in the evaluation of these models.

The Quality of Response Time Data Inference: A Blinded, Collaborative Assessment of the Validity of Cognitive Models.

Most data analyses rely on models. To complement statistical models, psychologists have developed cognitive models, which translate observed variables into psychologically interesting constructs. Response time models, in particular, assume that response time and accuracy are the observed expression of latent variables including 1) ease of processing, 2) response caution, 3) response bias, and 4) non-decision time. Inferences about these psychological factors, hinge upon the validity of the models' parameters. Here, we use a blinded, collaborative approach to assess the validity of such model-based inferences. Seventeen teams of researchers analyzed the same 14 data sets. In each of these two-condition data sets, we manipulated properties of participants' behavior in a two-alternative forced choice task. The contributing teams were blind to the manipulations, and had to infer what aspect of behavior was changed using their method of choice. The contributors chose to employ a variety of models, estimation methods, and inference procedures. Our results show that, although conclusions were similar across different methods, these "modeler's degrees of freedom" did affect their inferences. Interestingly, many of the simpler approaches yielded as robust and accurate inferences as the more complex methods. We recommend that, in general, cognitive models become a typical analysis tool for response time data. In particular, we argue that the simpler models and procedures are sufficient for standard experimental designs. We finish by outlining situations in which more complicated models and methods may be necessary, and discuss potential pitfalls when interpreting the output from response time models.

A dynamic dual process model of risky decision making.

Many phenomena in judgment and decision making are often attributed to the interaction of 2 systems of reasoning. Although these so-called dual process theories can explain many types of behavior, they are rarely formalized as mathematical or computational models. Rather, dual process models are typically verbal theories, which are difficult to conclusively evaluate or test. In the cases in which formal (i.e., mathematical) dual process models have been proposed, they have not been quantitatively fit to experimental data and are often silent when it comes to the timing of the 2 systems. In the current article, we present a dynamic dual process model framework of risky decision making that provides an account of the timing and interaction of the 2 systems and can explain both choice and response-time data. We outline several predictions of the model, including how changes in the timing of the 2 systems as well as time pressure can influence behavior. The framework also allows us to explore different assumptions about how preferences are constructed by the 2 systems as well as the dynamic interaction of the 2 systems. In particular, we examine 3 different possible functional forms of the 2 systems and 2 possible ways the systems can interact (simultaneously or serially). We compare these dual process models with 2 single process models using risky decision making data from Guo, Trueblood, and Diederich (2017). Using this data, we find that 1 of the dual process models significantly outperforms the other models in accounting for both choices and response times. (PsycINFO Database Record

Metastudies for robust tests of theory.

We describe and demonstrate an empirical strategy useful for discovering and replicating empirical effects in psychological science. The method involves the design of a metastudy, in which many independent experimental variables-that may be moderators of an empirical effect-are indiscriminately randomized. Radical randomization yields rich datasets that can be used to test the robustness of an empirical claim to some of the vagaries and idiosyncrasies of experimental protocols and enhances the generalizability of these claims. The strategy is made feasible by advances in hierarchical Bayesian modeling that allow for the pooling of information across unlike experiments and designs and is proposed here as a gold standard for replication research and exploratory research. The practical feasibility of the strategy is demonstrated with a replication of a study on subliminal priming.

A quantum theory account of order effects and conjunction fallacies in political judgments.

Are our everyday judgments about the world around us normative? Decades of research in the judgment and decision-making literature suggest the answer is no. If people's judgments do not follow normative rules, then what rules if any do they follow? Quantum probability theory is a promising new approach to modeling human behavior that is at odds with normative, classical rules. One key advantage of using quantum theory is that it explains multiple types of judgment errors using the same basic machinery, unifying what have previously been thought of as disparate phenomena. In this article, we test predictions from quantum theory related to the co-occurrence of two classic judgment phenomena, order effects and conjunction fallacies, using judgments about real-world events (related to the U.S. presidential primaries). We also show that our data obeys two a priori and parameter free constraints derived from quantum theory. Further, we examine two factors that moderate the effects, cognitive thinking style (as measured by the Cognitive Reflection Test) and political ideology.

Bayesian analysis of the piecewise diffusion decision model.

Most past research on sequential sampling models of decision-making have assumed a time homogeneous process (i.e., parameters such as drift rates and boundaries are constant and do not change during the deliberation process). This has largely been due to the theoretical difficulty in testing and fitting more complex models. In recent years, the development of simulation-based modeling approaches matched with Bayesian fitting methodologies has opened the possibility of developing more complex models such as those with time-varying properties. In the present work, we discuss a piecewise variant of the well-studied diffusion decision model (termed pDDM) that allows evidence accumulation rates to change during the deliberation process. Given the complex, time-varying nature of this model, standard Bayesian parameter estimation methodologies cannot be used to fit the model. To overcome this, we apply a recently developed simulation-based, hierarchal Bayesian methodology called the probability density approximation (PDA) method. We provide an analysis of this methodology and present results of parameter recovery experiments to demonstrate the strengths and limitations of this approach. With those established, we fit pDDM to data from a perceptual experiment where information changes during the course of trials. This extensible modeling platform opens the possibility of applying sequential sampling models to a range of complex non-stationary decision tasks.