Knowing what to know: Implications of the choice of prior distribution on the behavior of adaptive design optimization.

Adaptive design optimization (ADO) is a state-of-the-art technique for experimental design (Cavagnaro et al., 2010). ADO dynamically identifies stimuli that, in expectation, yield the most information about a hypothetical construct of interest (e.g., parameters of a cognitive model). To calculate this expectation, ADO leverages the modeler's existing knowledge, specified in the form of a prior distribution. Informative priors align with the distribution of the focal construct in the participant population. This alignment is assumed by ADO's internal assessment of expected information gain. If the prior is instead misinformative, i.e., does not align with the participant population, ADO's estimates of expected information gain could be inaccurate. In many cases, the true distribution that characterizes the participant population is unknown, and experimenters rely on heuristics in their choice of prior and without an understanding of how this choice affects ADO's behavior. Our work introduces a mathematical framework that facilitates investigation of the consequences of the choice of prior distribution on the efficiency of experiments designed using ADO. Through theoretical and empirical results, we show that, in the context of prior misinformation, measures of expected information gain are distinct from the correctness of the corresponding inference. Through a series of simulation experiments, we show that, in the case of parameter estimation, ADO nevertheless outperforms other design methods. Conversely, in the case of model selection, misinformative priors can lead inference to favor the wrong model, and rather than mitigating this pitfall, ADO exacerbates it.

Capturing richer information: On establishing the validity of an interval-valued survey response mode.

Obtaining quantitative survey responses that are both accurate and informative is crucial to a wide range of fields. Traditional and ubiquitous response formats such as Likert and visual analogue scales require condensation of responses into discrete or point values-but sometimes a range of options may better represent the correct answer. In this paper, we propose an efficient interval-valued response mode, whereby responses are made by marking an ellipse along a continuous scale. We discuss its potential to capture and quantify valuable information that would be lost using conventional approaches, while preserving a high degree of response efficiency. The information captured by the response interval may represent a possible response range-i.e., a conjunctive set, such as the real numbers between 3 and 6. Alternatively, it may reflect uncertainty in respect to a distinct response-i.e., a disjunctive set, such as a confidence interval. We then report a validation study, utilizing our recently introduced open-source software (DECSYS), to explore how interval-valued survey responses reflect experimental manipulations of several factors hypothesised to influence interval width, across multiple contexts. Results consistently indicate that respondents used interval widths effectively, and subjective participant feedback was also positive. We present this as initial empirical evidence for the efficacy and value of interval-valued response capture. Interestingly, our results also provide insight into respondents' reasoning about the different aforementioned types of intervals-we replicate a tendency towards overconfidence for those representing epistemic uncertainty (i.e., disjunctive sets), but find intervals representing inherent range (i.e., conjunctive sets) to be well-calibrated.

Judgment and Decision Making.

The science of judgment and decision making involves three interrelated forms of research: analysis of the decisions people face, description of their natural responses, and interventions meant to help them do better. After briefly introducing the field's intellectual foundations, we review recent basic research into the three core elements of decision making: judgment, or how people predict the outcomes that will follow possible choices; preference, or how people weigh those outcomes; and choice, or how people combine judgments and preferences to reach a decision. We then review research into two potential sources of behavioral heterogeneity: individual differences in decision-making competence and developmental changes across the life span. Next, we illustrate applications intended to improve individual and organizational decision making in health, public policy, intelligence analysis, and risk management. We emphasize the potential value of coupling analytical and behavioral research and having basic and applied research inform one another.

The Strengths and Weaknesses of Crowds to Address Global Problems.

Global climate change, the COVID-19 pandemic, and the spread of misinformation on social media are just a handful of highly consequential problems affecting society. We argue that the rough contours of many societal problems can be framed within a "wisdom of crowds" perspective. Such a framing allows researchers to recast complex problems within a simple conceptual framework and leverage known results on crowd wisdom. To this end, we present a simple "toy" model of the strengths and weaknesses of crowd wisdom that easily maps to many societal problems. Our model treats the judgments of individuals as random draws from a distribution intended to represent a heterogeneous population. We use a weighted mean of these individuals to represent the crowd's collective judgment. Using this setup, we show that subgroups have the potential to produce substantively different judgments and we investigate their effect on a crowd's ability to generate accurate judgments about societal problems. We argue that future work on societal problems can benefit from more sophisticated, domain-specific theory and models based on the wisdom of crowds.

Compositional data analysis tutorial.

This article presents techniques for dealing with a form of dependency in data arising when numerical data sum to a constant for individual cases, that is, "compositional" or "ipsative" data. Examples are percentages that sum to 100, and hours in a day that sum to 24. Ipsative scales fell out of fashion in psychology during the 1960s and 1970s due to a lack of methods for analyzing them. However, ipsative scales have merits, and compositional data commonly occur in psychological research. Moreover, as we demonstrate, sometimes converting data to a compositional form yields insights not otherwise accessible. Fortunately, there are sound methods for analyzing compositional data. We seek to enable researchers to analyze compositional data by presenting appropriate techniques and illustrating their application to real data. First, we elaborate the technical details of compositional data and discuss both established and new approaches to their analysis. We then present applications of these methods to real social science data-sets (data and code using R are available in a supplementary document). We conclude with a discussion of the state of the art in compositional data analysis and remaining unsolved problems. A brief guide to available software resources is provided in the first section of the supplementary document. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Relationship between lay and expert perceptions of COVID-19 vaccine development timelines in Canada and USA.

OBJECTIVES: Compare lay expectations of medical development to those of experts in the context of SARS-CoV-2 vaccine development. METHODS: A short online survey of experts and lay people measuring when participants believe important vaccine milestones would occur and how likely potential setbacks were. Samples of US and Canadian lay people recruited through Qualtrics. The expert sample was created through a contact network in vaccine development and supplemented with corresponding authors of recent scholarly review articles on vaccine development. RESULTS: In aggregate, lay people gave responses that were within 3 months of experts, tending to be later than experts for early milestones and earlier for later milestones. Median lay best estimates for when a vaccine would be available to the public were 08/2021 and 09/2021 for the US and Canadian samples, compared with 09-10/2021 for the experts. However, many individual lay responses showed more substantial disagreement with expert opinions, with 54% of lay best estimates of when a vaccine would be available to the public being before the median expert soonest estimate or after the median expert latest estimate. Lay people were much more pessimistic about vaccine development encountering setbacks than experts (median probability 59% of boxed warning compared with only 30% for experts). Misalignment between layperson and expert expectations was not explained by any demographic variables collected in our survey. CONCLUSION: Median lay expectations were generally similar to experts. At the individual level, however, lay people showed substantial variation with many believing milestones would occur much sooner than experts. Lay people were in general much more pessimistic about the prospect of setbacks than were experts.

Global-Local Incompatibility: The Misperception of Reliability in Judgment Regarding Global Variables.

A number of important decision domains, including decisions about hiring, global warming, and weather hazards, are characterized by a global-local incompatibility. These domains involve variables that cannot be observed by a single decision maker (DM) and require the integration of observations from locally available information cues. This paper presents a new bifocal lens model that describes how the structure of the environment can lead to a unique form of overconfidence when generalizing the reliability of the local environment to a global scale. When the local environment does not reliably reflect the global environment, they are incompatible. While local perspectives vary across DMs, global-local incompatibility can be understood using the structure of classical test theory as the difference between (a) perceived estimates of the reliability derived from the local environment and (b) the true reliability of the local environment. I model global-local incompatibility as the difference between the true and estimated reliability when the assumptions of classical test theory are violated. Using a series of case studies and an empirical study, I demonstrate the widespread utility of this framework, and I conclude by discussing implications for cognitive-ecological theory, risk communication, and overconfidence.

Exclusion Criteria as Measurements I: Identifying Invalid Responses.

Background. In a systematic review, Engel et al. found large variation in the exclusion criteria used to remove responses held not to represent genuine preferences in health state valuation studies. We offer an empirical approach to characterizing the similarities and differences among such criteria. Setting. Our analyses use data from an online survey that elicited preferences for health states defined by domains from the Patient-Reported Outcomes Measurement Information System (PROMIS®), with a U.S. nationally representative sample (N = 1164). Methods. We use multidimensional scaling to investigate how 10 commonly used exclusion criteria classify participants and their responses. Results. We find that the effects of exclusion criteria do not always match the reasons advanced for applying them. For example, excluding very high and very low values has been justified as removing aberrant responses. However, people who give very high and very low values prove to be systematically different in ways suggesting that such responses may reflect different processes. Conclusions. Exclusion criteria intended to remove low-quality responses from health state valuation studies may actually remove deliberate but unusual ones. A companion article examines the effects of the exclusion criteria on societal utility estimates.

Exclusion Criteria as Measurements II: Effects on Utility Functions.

Background. Researchers often justify excluding some responses in studies eliciting valuations of health states as not representing respondents' true preferences. Here, we examine the effects of applying 8 common exclusion criteria on societal utility estimates. Setting. An online survey of a US nationally representative sample (N = 1164) used the standard gamble method to elicit preferences for health states defined by 7 health domains from the Patient-Reported Outcomes Measurement Information System (PROMIS®). Methods. We estimate the impacts of applying 8 commonly used exclusion criteria on mean utility values for each domain, using beta regression, a form of analysis suited to double-bounded scales, such as utility. Results. Exclusion criteria have varied effects on the utility functions for the different PROMIS health domains. As a result, applying those criteria would have varied effects on the value of treatments (and side effects) that change health status on those domains. Limitations. Although our method could be applied to any health utility judgments, the present estimates reflect the features of the study that produced them. Those features include the selected health domains, standard gamble method, and an online format that excluded some groups (e.g., visually impaired and illiterate individuals). We also examined only a subset of all possible exclusion criteria, selected to represent the space of possibilities, as characterized in a companion article. Conclusions. Exclusion criteria can affect estimates of the societal utility of health states. We use those effects, in conjunction with the results of the companion article, to make suggestions for selecting exclusion criteria in future studies.

Public perception and communication of scientific uncertainty.

Understanding how the public perceives uncertainty in scientific research is fundamental for effective communication about research and its inevitable uncertainty. Previous work found that scientific evidence differentially influenced beliefs from individuals with different political ideologies. Evidence that threatens an individual's political ideology is perceived as more uncertain than nonthreatening evidence. The authors present 3 studies examining perceptions of scientific uncertainty more broadly by including sciences that are not politically polarizing. Study 1 develops scales measuring perceptions of scientific uncertainty. It finds (a) 3 perceptual dimensions of scientific uncertainty, with the primary dimension representing a perception of precision; (b) the precision dimension of uncertainty is strongly associated with the perceived value of a research field; and (c) differences in perceived uncertainty across political affiliations. Study 2 manipulated these dimensions, finding that Republicans were more sensitive than Democrats to descriptions of uncertainty associated with a research field (e.g., psychology). Study 3 found that these views of a research field did not extend to the evaluation of individual results produced by the field. Together, these studies show that perceptions of scientific uncertainty associated with entire research fields are valid predictors of abstract perceptions of scientific quality, benefit, and allocation of funding. Yet, they do not inform judgments about individual results. Therefore, polarization in the acceptance of specific results is not likely due to individual differences in perceived scientific uncertainty. Further, the direction of influence potentially could be reversed, such that perceived quality of scientific results could be used to influence perceptions about scientific research fields. (PsycINFO Database Record