A strategy to improve expert technology forecasts.

Forecasts of the future cost and performance of technologies are often used to support decision-making. However, retrospective reviews find that many forecasts made by experts are not very accurate and are often seriously overconfident, with realized values too frequently falling outside of forecasted ranges. Here, we outline a hybrid approach to expert elicitation that we believe might improve forecasts of future technologies. The proposed approach iteratively combines the judgments of technical domain experts with those of experts who are knowledgeable about broader issues of technology adoption and public policy. We motivate the approach with results from a pilot study designed to help forecasters think systematically about factors beyond the technology itself that may shape its future, such as policy, economic, and social factors. Forecasters who received briefings on these topics provided wider forecast intervals than those receiving no assistance.

Cognitive Biases and Shared Decision Making in Acute Brain Injury.

Many patients hospitalized after severe acute brain injury are comatose and require life-sustaining therapies. Some of these patients make favorable recoveries with continued intensive care, while others do not. In addition to providing medical care, clinicians must guide surrogate decision makers through high-stakes, emotionally charged decisions about whether to continue life-sustaining therapies. These consultations require clinicians first to assess a patient's likelihood of recovery given continued life-sustaining therapies (i.e., prognosticate), then to communicate that prediction to surrogates, and, finally, to elicit and interpret the patient's preferences. At each step, both clinicians and surrogates are vulnerable to flawed decision making. Clinicians can be imprecise, biased, and overconfident when prognosticating after brain injury. Surrogates can misperceive the choice and misunderstand or misrepresent a patient's wishes, which may never have been communicated clearly. These biases can undermine the ability to reach choices congruent with patients' preferences through shared decision making (SDM). Decision science has extensively studied these biases. In this article, we apply that research to improving SDM for patients who are comatose after acute brain injury. After introducing SDM and the medical context, we describe principal decision science results as they relate to neurologic prognostication and end-of-life decisions, by both clinicians and surrogates. Based on research regarding general processes that can produce imprecise, biased, and overconfident prognoses, we propose interventions that could improve SDM, supporting clinicians and surrogates in making these challenging decisions.

Bereaved Parents: Insights for the Antenatal Consultation.

OBJECTIVE: The study aimed to explore experiences of extremely preterm infant loss in the delivery room and perspectives about antenatal consultation. STUDY DESIGN: Bereaved participants were interviewed, following a semi-structured protocol. Personal narratives were analyzed with a mixed-methods approach. RESULTS: In total, 13 participants, reflecting on 17 pregnancies, shared positive, healing and negative, harmful interactions with clinicians and institutions: feeling cared for or abandoned, doubted or believed, being treated rigidly or flexibly, and feeling that infant's life was valued or not. Participants stressed their need for personalized information, individualized approaches, and affective support. Their decision processes varied; some wanted different things for themselves than what they recommended for others. These interactions shaped their immediate experiences, long-term well-being, healing, and regrets. All had successful subsequent pregnancies; few returned to institutions where they felt poorly treated. CONCLUSION: Antenatal consultations can be strengthened by personalizing them, within a strong caregiver relationship and supportive institutional practices. KEY POINTS: · Personalized antenatal consultations should strive to balance cognitive and affective needs.. · Including perspectives from bereaved parents can strengthen antenatal consultations.. · Trusting provider-parent partnerships are pivotal for risk communication..

When Do Humans Heed AI Agents' Advice? When Should They?

OBJECTIVE: We manipulate the presence, skill, and display of artificial intelligence (AI) recommendations in a strategy game to measure their effect on users' performance. BACKGROUND: Many applications of AI require humans and AI agents to make decisions collaboratively. Success depends on how appropriately humans rely on the AI agent. We demonstrate an evaluation method for a platform that uses neural network agents of varying skill levels for the simple strategic game of Connect Four. METHODS: We report results from a 2 × 3 between-subjects factorial experiment that varies the format of AI recommendations (categorical or probabilistic) and the AI agent's amount of training (low, medium, or high). On each round of 10 games, participants proposed a move, saw the AI agent's recommendations, and then moved. RESULTS: Participants' performance improved with a highly skilled agent, but quickly plateaued, as they relied uncritically on the agent. Participants relied too little on lower skilled agents. The display format had no effect on users' skill or choices. CONCLUSIONS: The value of these AI agents depended on their skill level and users' ability to extract lessons from their advice. APPLICATION: Organizations employing AI decision support systems must consider behavioral aspects of the human-agent team. We demonstrate an approach to evaluating competing designs and assessing their performance.

Improving Social Bot Detection Through Aid and Training.

OBJECTIVE: We test the effects of three aids on individuals' ability to detect social bots among Twitter personas: a bot indicator score, a training video, and a warning. BACKGROUND: Detecting social bots can prevent online deception. We use a simulated social media task to evaluate three aids. METHOD: Lay participants judged whether each of 60 Twitter personas was a human or social bot in a simulated online environment, using agreement between three machine learning algorithms to estimate the probability of each persona being a bot. Experiment 1 compared a control group and two intervention groups, one provided a bot indicator score for each tweet; the other provided a warning about social bots. Experiment 2 compared a control group and two intervention groups, one receiving the bot indicator scores and the other a training video, focused on heuristics for identifying social bots. RESULTS: The bot indicator score intervention improved predictive performance and reduced overconfidence in both experiments. The training video was also effective, although somewhat less so. The warning had no effect. Participants rarely reported willingness to share content for a persona that they labeled as a bot, even when they agreed with it. CONCLUSIONS: Informative interventions improved social bot detection; warning alone did not. APPLICATION: We offer an experimental testbed and methodology that can be used to evaluate and refine interventions designed to reduce vulnerability to social bots. We show the value of two interventions that could be applied in many settings.

The value of adherence information during clinical pharmaceutical trials.

BACKGROUND/AIMS: The quality of the evidence used to evaluate a drug's safety and efficacy depends, in part, on how well participants adhere to the prescribed drug-taking regime. There are multiple approaches to measure adherence in clinical trials, varying in their cost and accuracy. We demonstrate a method for evaluating the cost-effectiveness of common adherence monitoring methods, considering the costs and data quality for drugs that differ in how forgiving they are of nonadherence. METHODS: We propose a simulation approach to estimate the value of evidence about adherence, considering both costs of collection and potential errors in interpreting clinical trial results. We demonstrate the approach with a simulated clinical trial of nitrendipine, a common calcium channel blocker. We consider two trial designs, one using pretrial adherence to "enrich" the trial sample and one without an enrichment strategy. We use scenarios combining high and low values of two key properties of a clinical trial: participant adherence and drug forgiveness. RESULTS: Under the conditions of these simulations, the most cost-effective adherence monitoring approach depends on both trial participant adherence and drug forgiveness. For example, the enrichment strategy is not cost-effective for the base scenario (high forgiveness and high adherence), but is for other scenarios. We also estimate the effects of evaluable patient analysis, a controversial procedure that excludes nonadherent participants from the analyses, after a trial is completed. CONCLUSIONS: Our proposed approach can guide drug regulators and developers in designing efficient clinical trials and assessing the impact of nonadherence on trial results. It can identify cost-effective adherence-monitoring methods, given available knowledge about the methods, drug, and patients' expected adherence.

Evaluating science communication.

Effective science communication requires assembling scientists with knowledge relevant to decision makers, translating that knowledge into useful terms, establishing trusted two-way communication channels, evaluating the process, and refining it as needed. Communicating Science Effectively: A Research Agenda [National Research Council (2017)] surveys the scientific foundations for accomplishing these tasks, the research agenda for improving them, and the essential collaborative relations with decision makers and communication professionals. Recognizing the complexity of the science, the decisions, and the communication processes, the report calls for a systems approach. This perspective offers an approach to creating such systems by adapting scientific methods to the practical constraints of science communication. It considers staffing (are the right people involved?), internal collaboration (are they talking to one another?), and external collaboration (are they talking to other stakeholders?). It focuses on contexts where the goal of science communication is helping people to make autonomous choices rather than promoting specific behaviors (e.g., voter turnout, vaccination rates, energy consumption). The approach is illustrated with research in two domains: decisions about preventing sexual assault and responding to pandemic disease.

Risk heatmaps as visual displays: Opening movie studios after the COVID-19 shutdown.

Upon shutting down operations in early 2020 due to the COVID-19 pandemic, the movie industry assembled teams of experts to help develop guidelines for returning to operation. It resulted in a joint report, The Safe Way Forward, which was created in consultation with union members and provided the basis for negotiations with the studios. A centerpiece of the report was a set of heatmaps displaying SARS-CoV-2 risks for a shoot, as a function of testing rate, community infection prevalence, community transmission rate (R0), and risk measure (either expected number of cases or probability of at least one case). We develop and demonstrate a methodology for evaluating such complex displays, in terms of how well they inform potential users, in this case, workers deciding whether the risks of a shoot are acceptable. We ask whether individuals making hypothetical return-to-work decisions can (a) read display entries, (b) compare display entries, and (c) make inferences based on display entries. Generally speaking, respondents recruited through the Amazon MTurk platform could interpret the display information accurately and make coherent decisions, suggesting that heatmaps can communicate complex risks to lay audiences. Although these heatmaps were created for practical, rather than theoretical, purposes, these results provide partial support for theoretical accounts of visual information processing and identify challenges in applying them to complex settings.

Duped by Bots: Why Some are Better than Others at Detecting Fake Social Media Personas.

OBJECTIVE: We examine individuals' ability to detect social bots among Twitter personas, along with participant and persona features associated with that ability. BACKGROUND: Social media users need to distinguish bots from human users. We develop and demonstrate a methodology for assessing those abilities, with a simulated social media task. METHOD: We analyze performance from a signal detection theory perspective, using a task that asked lay participants whether each of 50 Twitter personas was a human or social bot. We used the agreement of two machine learning models to estimate the probability of each persona being a bot. We estimated the probability of participants indicating that a persona was a bot with a generalized linear mixed-effects model using participant characteristics (social media experience, analytical reasoning, and political views) and stimulus characteristics (bot indicator score and political tone) as regressors. RESULTS: On average, participants had modest sensitivity (d') and a criterion that favored responding "human." Exploratory analyses found greater sensitivity for participants (a) with less self-reported social media experience, (b) greater analytical reasoning ability, and (c) who were evaluating personas with opposing political views. Some patterns varied with participants' political identity. CONCLUSIONS: Individuals have limited ability to detect social bots, with greater aversion to mistaking bots for humans than vice versa. Greater social media experience and myside bias appeared to reduce performance, as did less analytical reasoning ability. APPLICATION: These patterns suggest the need for interventions, especially when users feel most familiar with social media.

Outcomes after a Digital Behavior Change Intervention to Improve Trauma Triage: An Analysis of Medicare Claims.

BACKGROUND: Under-triage in trauma remains prevalent, in part because of decisions made by physicians at non-trauma centers. We developed two digital behavior change interventions to recalibrate physician heuristics (pattern recognition), and randomized 688 emergency medicine physicians to use the interventions or to a control. In this observational follow-up, we evaluated whether exposure to the interventions changed physician performance in practice. METHODS: We obtained 2016 - 2018 Medicare claims for severely injured patients, linked the names of trial participants to National Provider Identifiers (NPIs), and identified claims filed by trial participants for injured patients presenting to non-trauma centers in the year before and after their trial. The primary outcome measure was the triage status of severely injured patients. RESULTS: We linked 670 (97%) participants to NPIs, identified claims filed for severely injured patients by 520 (76%) participants, and claims filed at non-trauma centers by 228 (33%). Most participants were white (64%), male (67%), and had more than three years of experience (91%). Patients had a median Injury Severity Score of 16 (IQR 16 - 17), and primarily sustained neuro-trauma. After adjustment, patients treated by physicians randomized to the interventions experienced less under-triage in the year after the trial than before (41% versus 58% [-17%], P = 0.015); patients treated by physicians randomized to the control experienced no difference in under-triage (49% versus 56% [-7%], P = 0.35). The difference-in-the-difference was non-significant (10%, P = 0.18). CONCLUSIONS: It was feasible to track trial participants' performance in national claims. Sample size limitations constrained causal inference about the effect of the interventions.

Judgment and Decision Making.

The science of judgment and decision making involves three interrelated forms of research: analysis of the decisions people face, description of their natural responses, and interventions meant to help them do better. After briefly introducing the field's intellectual foundations, we review recent basic research into the three core elements of decision making: judgment, or how people predict the outcomes that will follow possible choices; preference, or how people weigh those outcomes; and choice, or how people combine judgments and preferences to reach a decision. We then review research into two potential sources of behavioral heterogeneity: individual differences in decision-making competence and developmental changes across the life span. Next, we illustrate applications intended to improve individual and organizational decision making in health, public policy, intelligence analysis, and risk management. We emphasize the potential value of coupling analytical and behavioral research and having basic and applied research inform one another.

Serious games may improve physician heuristics in trauma triage.

Trauma triage depends on fallible human judgment. We created two "serious" video game training interventions to improve that judgment. The interventions' central theoretical construct was the representativeness heuristic, which, in trauma triage, would mean judging the severity of an injury by how well it captures (or "represents") the key features of archetypes of cases requiring transfer to a trauma center. Drawing on clinical experience, medical records, and an expert panel, we identified features characteristic of representative and nonrepresentative cases. The two interventions instantiated both kinds of cases. One was an adventure game, seeking narrative engagement; the second was a puzzle-based game, emphasizing analogical reasoning. Both incorporated feedback on diagnostic errors, explaining their sources and consequences. In a four-arm study, they were compared with an intervention using traditional text-based continuing medical education materials (active control) and a no-intervention (passive control) condition. A sample of 320 physicians working at nontrauma centers in the United States was recruited and randomized to a study arm. The primary outcome was performance on a validated virtual simulation, measured as the proportion of undertriaged patients, defined as ones who had severe injuries (according to American College of Surgeons guidelines) but were not transferred. Compared with the control group, physicians exposed to either game undertriaged fewer such patients [difference = -18%, 95% CI: -30 to -6%, P = 0.002 (adventure game); -17%, 95% CI: -28 to -6%, P = 0.003 (puzzle game)]; those exposed to the text-based education undertriaged similar proportions (difference = +8%, 95% CI: -3 to +19%, P = 0.15).

Individuals with greater science literacy and education have more polarized beliefs on controversial science topics.

Although Americans generally hold science in high regard and respect its findings, for some contested issues, such as the existence of anthropogenic climate change, public opinion is polarized along religious and political lines. We ask whether individuals with more general education and greater science knowledge, measured in terms of science education and science literacy, display more (or less) polarized beliefs on several such issues. We report secondary analyses of a nationally representative dataset (the General Social Survey), examining the predictors of beliefs regarding six potentially controversial issues. We find that beliefs are correlated with both political and religious identity for stem cell research, the Big Bang, and human evolution, and with political identity alone on climate change. Individuals with greater education, science education, and science literacy display more polarized beliefs on these issues. We find little evidence of political or religious polarization regarding nanotechnology and genetically modified foods. On all six topics, people who trust the scientific enterprise more are also more likely to accept its findings. We discuss the causal mechanisms that might underlie the correlation between education and identity-based polarization.

Eliciting probabilistic expectations: Collaborations between psychologists and economists.

We describe two collaborations in which psychologists and economists provided essential support on foundational projects in major research programs. One project involved eliciting adolescents' expectations regarding significant future life events affecting their psychological and economic development. The second project involved eliciting consumers' expectations regarding inflation, a potentially vital input to their investment, saving, and purchasing decisions. In each project, we sought questions with the precision needed for economic modeling and the simplicity needed for lay respondents. We identify four conditions that, we believe, promoted our ability to sustain these transdisciplinary collaborations and coproduce the research: (i) having a shared research goal, which neither discipline could achieve on its own; (ii) finding common ground in shared methodology, which met each discipline's essential evidentiary conditions, but without insisting on its culturally acquired tastes; (iii) sharing the effort throughout, with common language and sense of ownership; and (iv) gaining mutual benefit from both the research process and its products.

Inferring Sexually Transmitted Infection Risk From Attractiveness in Online Dating Among Adolescents and Young Adults: Exploratory Study.

BACKGROUND: Sexually transmitted infection (STI) rates are on the rise among adolescents and young adults in the United States. With the popularity of online dating, adolescents and young adults must increasingly rely on limited cues to make initial judgments about potential sexual partners, including judgments about STI risk. OBJECTIVE: This study aimed to assess whether in the context of online dating, an attractiveness heuristic would be used for STI risk assessment. We hypothesized that consistent with research on halo effects, decision makers would judge more attractive people to be less likely to have STIs. METHODS: In a survey experiment, we asked participants to determine which individual in each of 20 sets of paired photographs was enrolled in a personals website for people with publicly disclosed STIs. RESULTS: Despite financial incentives for accuracy and high levels of self-confidence in their judgments, participants performed no better than chance at identifying individuals with self-reported STIs. Contrary to our hypothesis, however, more attractive people were judged as being more likely to have an STI. This relationship appears to be mediated by inferences regarding the target individual's sexual behavior, with more attractive individuals considered to have more partners. CONCLUSIONS: On showing adolescents and young adults photographs offering no diagnostic information about STIs, they appeared to use attractiveness as a cue for sexual risk, which was mediated by the belief that attractive individuals have more sexual opportunities. Health care providers may wish to address this heuristic process among their adolescent patients in discussions about sexual health.

Does Preexisting Practice Modify How Video Games Recalibrate Physician Heuristics in Trauma Triage?

BACKGROUND: A majority of severely injured patients fail to receive care at trauma centers (undertriage), in part, because of physician judgment. We previously developed two educational video games that reduced physicians' undertriage compared with control in two clinical trials. In this secondary analysis, we investigated heterogeneity of treatment effect of the interventions by assessing physicians' preexisting practice patterns in claims data. We hypothesized that physicians with high preexisting undertriage would benefit most from game-based training. METHODS: Using Medicare claims records from 2010 to 2015, we measured physicians' preexisting triage practices before their participation in one of two trials conducted in 2016 and 2017. We categorized physicians as having received game-based training versus control and noted their postintervention simulation triage performance in the trials. We used multivariable linear regression models to assess the heterogeneity of game-based training effect among physicians with high and low preexisting undertriage. RESULTS: Of the 394 eligible physicians from our trials, we identified 275 (70%) with claims for Medicare fee-for-service beneficiaries suffering severe injury between 2010 and 2015. On average, the physicians were 44 y old (SD 8.4) with 12 y (SD 8.2) of experience. We found significant interaction between preexisting practice and intervention efficacy (P = 0.04). Physicians with high undertriage before enrollment improved significantly with game-based training compared with the control (46% versus 63%, P < 0.001). Those with low preexisting undertriage did not (58% versus 56%, P = 0.76). CONCLUSIONS: Using claims-based data, we found heterogeneity of treatment effect of interventions designed to recalibrate physician heuristics. Physicians with high preexisting undertriage benefited most from game-based training.

Evaluation of options for presenting health-states from PROMIS® item banks for valuation exercises.

PURPOSE: Health status descriptive systems based on item response theory (IRT), such as the Patient-Reported Outcomes Measurement Information System (PROMIS®), have item banks to measure domains of health. We developed a method to present such banks for health-state valuation. METHODS: We evaluated four different presentation approaches: a single item (1S), 2 items presented separately (2S), 2 items presented together (2T), or 5 items presented together (5T). We evaluated these four approaches in three PROMIS item banks (depression, physical function, and sleep disturbance). Adult community members valued health-state descriptions using the visual analog scale and standard gamble methods. We compared the approaches by the range of item bank theta scores captured, participants' assessments of difficulty (1 = very easy to 7 = very hard), and exit interviews. RESULTS: Participants (n = 118) ranged in age from 18 to 71; 63% were female and 54% were white. The 1S approach captured the smallest range of theta scores. A monotonic relationship between theta score and mean standard gamble estimate was found with all approaches except 2S. Across all 3 item banks, mean difficulty assessments were 2.35 (1S), 2.69 (2T), 2.78 (5T), and 2.80 (2S). In exit interviews, participants generally found all four approaches similarly meaningful and realistic. CONCLUSIONS: Creating health descriptions by presenting 2 items maximized the range of theta while minimizing difficulty and maintaining a monotonic relationship with utility estimates. We recommend this approach for valuation of IRT-based descriptive systems such as PROMIS.

Setting Priorities in Behavioral Interventions: An Application to Reducing Phishing Risk.

Phishing risk is a growing area of concern for corporations, governments, and individuals. Given the evidence that users vary widely in their vulnerability to phishing attacks, we demonstrate an approach for assessing the benefits and costs of interventions that target the most vulnerable users. Our approach uses Monte Carlo simulation to (1) identify which users were most vulnerable, in signal detection theory terms; (2) assess the proportion of system-level risk attributable to the most vulnerable users; (3) estimate the monetary benefit and cost of behavioral interventions targeting different vulnerability levels; and (4) evaluate the sensitivity of these results to whether the attacks involve random or spear phishing. Using parameter estimates from previous research, we find that the most vulnerable users were less cautious and less able to distinguish between phishing and legitimate emails (positive response bias and low sensitivity, in signal detection theory terms). They also accounted for a large share of phishing risk for both random and spear phishing attacks. Under these conditions, our analysis estimates much greater net benefit for behavioral interventions that target these vulnerable users. Within the range of the model's assumptions, there was generally net benefit even for the least vulnerable users. However, the differences in the return on investment for interventions with users with different degrees of vulnerability indicate the importance of measuring that performance, and letting it guide interventions. This study suggests that interventions to reduce response bias, rather than to increase sensitivity, have greater net benefit.