Is generalization decay a fundamental law of psychology?

Generalizations strengthen in traditional sciences, but in psychology (and social and behavioral sciences, more generally) they decay. This is usually viewed as a problem requiring solution. It could be viewed instead as a law-like phenomenon. Generalization decay cannot be squelched because human behavior is metastable and all behavioral data collected thus far have resulted from a thin sliver of human time.

Predicting Clinical Trial Results: A Synthesis of Five Empirical Studies and Their Implications.

Expectations about future events underlie practically every decision we make, including those in medical research. This paper reviews five studies undertaken to assess how well medical experts could predict the outcomes of clinical trials. It explains why expert trial forecasting was the focus of study and argues that forecasting skill affords insights into the quality of expert judgment and might be harnessed to improve decision-making in care, policy, and research. The paper also addresses potential criticisms of the research agenda and summarizes key findings from the five studies of trial forecasting. Together, the studies suggest that trials frequently deliver surprising results to expert communities and that individual experts are often uninformative when it comes to forecasting trial outcome and recruitment. However, the findings also suggest that expert forecasts often contain a "signal" about whether a trial will be positive, especially when forecasts are aggregated. The paper concludes with needs for further research and tentative policy recommendations.

Communicating uncertainty in national security intelligence: Expert and nonexpert interpretations of and preferences for verbal and numeric formats.

Organizations in several domains including national security intelligence communicate judgments under uncertainty using verbal probabilities (e.g., likely) instead of numeric probabilities (e.g., 75% chance), despite research indicating that the former have variable meanings across individuals. In the intelligence domain, uncertainty is also communicated using terms such as low, moderate, or high to describe the analyst's confidence level. However, little research has examined how intelligence professionals interpret these terms and whether they prefer them to numeric uncertainty quantifiers. In two experiments (N = 481 and 624, respectively), uncertainty communication preferences of expert (n = 41 intelligence analysts in Experiment 1) and nonexpert intelligence consumers were elicited. We examined which format participants judged to be more informative and simpler to process. We further tested whether participants treated verbal probability and confidence terms as independent constructs and whether participants provided coherent numeric probability translations of verbal probabilities. Results showed that although most nonexperts favored the numeric format, experts were about equally split, and most participants in both samples regarded the numeric format as more informative. Experts and nonexperts consistently conflated probability and confidence. For instance, confidence intervals inferred from verbal confidence terms had a greater effect on the location of the estimate than the width of the estimate, contrary to normative expectation. Approximately one-fourth of experts and over one-half of nonexperts provided incoherent numeric probability translations for the terms likely and unlikely when the elicitation of best estimates and lower and upper bounds were briefly spaced by intervening tasks.

Framing, equivalence, and rational inference.

Bermúdez's case for rational framing effects, while original, is unconvincing and gives only parenthetical treatment to the problematic assumptions of extensional and semantic equivalence of alternative frames in framing experiments. If the assumptions are false, which they sometimes are, no valid inferences about "framing effects" follow and, then, neither do inferences about human rationality. This commentary recaps the central problem.

Can Oncologists Predict the Efficacy of Treatments in Randomized Trials?

BACKGROUND: Decisions about trial funding, ethical approval, or clinical practice guideline recommendations require expert judgments about the potential efficacy of new treatments. We tested whether individual and aggregated expert opinion of oncologists could predict reliably the efficacy of cancer treatments tested in randomized controlled trials. MATERIALS AND METHODS: An international sample of 137 oncologists specializing in genitourinary, lung, and colorectal cancer provided forecasts on primary outcome attainment for five active randomized cancer trials within their subspecialty; skill was assessed using Brier scores (BS), which measure the average squared deviation between forecasts and outcomes. RESULTS: A total of 40% of trials in our sample reported positive primary outcomes. Experts generally anticipated this overall frequency (mean forecast, 34%). Individual experts on average outperformed random predictions (mean BS = 0.29 [95% confidence interval (CI), 0.28-0.33] vs. 0.33) but underperformed prediction algorithms that always guessed 50% (BS = 0.25) or that were trained on base rates (BS = 0.19). Aggregating forecasts improved accuracy (BS = 0.25; 95% CI, 0.16-0.36]). Neither individual experts nor aggregated predictions showed appreciable discrimination between positive and nonpositive trials (area under the curve of a receiver operating characteristic curve, 0.52 and 0.43, respectively). CONCLUSION: These findings are based on a limited sample of trials. However, they reinforce the importance of basing research and policy decisions on the results of randomized trials rather than expert opinion or low-level evidence. IMPLICATIONS FOR PRACTICE: Predictions of oncologists, either individually or in the aggregate, did not anticipate reliably outcomes for randomized trials in cancer. These findings suggest that pooled expert opinion about treatment efficacy is no substitute for randomized trials. They also underscore the challenges of using expert opinion to prioritize interventions for clinical trials or to make recommendations in clinical practice guidelines.

Can cancer researchers accurately judge whether preclinical reports will reproduce?

There is vigorous debate about the reproducibility of research findings in cancer biology. Whether scientists can accurately assess which experiments will reproduce original findings is important to determining the pace at which science self-corrects. We collected forecasts from basic and preclinical cancer researchers on the first 6 replication studies conducted by the Reproducibility Project: Cancer Biology (RP:CB) to assess the accuracy of expert judgments on specific replication outcomes. On average, researchers forecasted a 75% probability of replicating the statistical significance and a 50% probability of replicating the effect size, yet none of these studies successfully replicated on either criterion (for the 5 studies with results reported). Accuracy was related to expertise: experts with higher h-indices were more accurate, whereas experts with more topic-specific expertise were less accurate. Our findings suggest that experts, especially those with specialized knowledge, were overconfident about the RP:CB replicating individual experiments within published reports; researcher optimism likely reflects a combination of overestimating the validity of original studies and underestimating the difficulties of repeating their methodologies.

Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation.

As in other areas of expert judgment, intelligence analysis often requires judging the probability that hypotheses are true. Intelligence organizations promote the use of structured methods such as "Analysis of Competing Hypotheses" (ACH) to improve judgment accuracy and analytic rigor, but these methods have received little empirical testing. In this experiment, we pitted ACH against a factorized Bayes's theorem (FBT) method, and we examined the value of recalibration (coherentization) and aggregation methods for improving the accuracy of probability judgment. Analytic techniques such as ACH and FBT were ineffective in improving accuracy and handling correlated evidence, and ACH in fact decreased the coherence of probability judgments. In contrast, statistical postanalytic methods (i.e., coherentization and aggregation) yielded large accuracy gains. A wide range of methods for instantiating these techniques were tested. The interactions among the factors considered suggest that prescriptive theorists and interventionists should examine the value of ensembles of judgment-support methods.

Accuracy of forecasts in strategic intelligence.

The accuracy of 1,514 strategic intelligence forecasts abstracted from intelligence reports was assessed. The results show that both discrimination and calibration of forecasts was very good. Discrimination was better for senior (versus junior) analysts and for easier (versus harder) forecasts. Miscalibration was mainly due to underconfidence such that analysts assigned more uncertainty than needed given their high level of discrimination. Underconfidence was more pronounced for harder (versus easier) forecasts and for forecasts deemed more (versus less) important for policy decision making. Despite the observed underconfidence, there was a paucity of forecasts in the least informative 0.4-0.6 probability range. Recalibrating the forecasts substantially reduced underconfidence. The findings offer cause for tempered optimism about the accuracy of strategic intelligence forecasts and indicate that intelligence producers aim to promote informativeness while avoiding overstatement.

Open-label tofacitinib and double-blind atorvastatin in rheumatoid arthritis patients: a randomised study.

OBJECTIVES: To evaluate the efficacy and safety of atorvastatin versus placebo in modifying lipids in patients with rheumatoid arthritis (RA) receiving the oral Janus kinase inhibitor, tofacitinib. METHODS: A randomised, placebo controlled, multicentre phase 2 study, open-label for tofacitinib and blinded for atorvastatin. Patients received tofacitinib 10 mg twice daily for 12 weeks; at week 6, patients were randomly assigned 1:1 to receive oral atorvastatin 10 mg once daily or placebo for 6 weeks. Main outcome measures were lipid moieties, American College of Rheumatology (ACR) response rates, disease activity score in 28 joint counts and safety. RESULTS: 111 patients meeting ACR 1987 RA criteria with active disease were enrolled. Tofacitinib-induced elevation of mean total, low-density lipoprotein (LDL) and high-density lipoprotein-cholesterol, triglycerides and apolipoprotein A-1 concentrations were sustained in placebo recipients to week 12; atorvastatin added at week 6 significantly reduced tofacitinib-associated increases in total and LDL-cholesterol, triglycerides and apolipoprotein B to below week 0 levels. Co-administration of atorvastatin resulted in a significant reduction of LDL-cholesterol versus placebo (primary endpoint; p<0.0001); from week 6 to week 12 the least squares mean reduction was 35.3% with atorvastatin, versus 5.8% increase with placebo. ACR responses were observed with tofacitinib; numerically greater rates were seen with atorvastatin versus placebo. Adverse events were consistent with phase 3 studies. CONCLUSIONS: Tofacitinib-associated elevated total and LDL-cholesterol and triglycerides were rapidly and significantly reduced by atorvastatin. Further investigation is required to explore the significance of reductions in RA disease activity in patients receiving tofacitinib and atorvastatin. (Pfizer protocol A3921109).

Suicide terrorism, moral relativism, and the situationist narrative.

Lankford challenges two popular views of suicide terrorists: first, that they are psychologically normal and, second, that they are heroic, not unlike professional soldiers. I augment Lankford's critique by tracing these views to a simplistic situationist narrative and to a careless form of moral relativism to which many scholars adhere. Conceptual weaknesses of these positions are briefly discussed.

When expert predictions fail.

We examine the opportunities and challenges of expert judgment in the social sciences, scrutinizing the way social scientists make predictions. While social scientists show above-chance accuracy in predicting laboratory-based phenomena, they often struggle to predict real-world societal changes. We argue that most causal models used in social sciences are oversimplified, confuse levels of analysis to which a model applies, misalign the nature of the model with the nature of the phenomena, and fail to consider factors beyond the scientist's pet theory. Taking cues from physical sciences and meteorology, we advocate an approach that integrates broad foundational models with context-specific time series data. We call for a shift in the social sciences towards more precise, daring predictions and greater intellectual humility.

Insights Into Learning Among Physical Therapy, Medical, and Nursing Students Following a Simulation-Based, Interprofessional Patient Safety Course.

As the need for a strong interdisciplinary approach in the delivery of healthcare services becomes increasingly vital, interprofessional education (IPE) is essential to equip healthcare professionals of the future to deliver better care. IPE encounters using simulation-based education can be a powerful tool in inculcating pre-professional students with foundational tools for successful interprofessional work. This qualitative study explores the learning that occurs during IPE encounters that include nursing, physical therapy, and medical students. The results of this work highlight how important IPE encounters are in uncovering and changing cross-disciplinary notions about knowledge, skills, role, and team contributions. Specifically, our analysis demonstrated that there are common misunderstandings about contemporary roles, knowledge, and skills of physical therapists among both nursing and medical students that can be corrected in IPE encounters. Results also demonstrated that careful planning can bolster opinions of the value of activities embedded as part of a larger course. Moreover, planning and attention to the specific educational needs of all students may prevent any group feeling that their educational needs were not fully met across all disciplines. The findings support the consideration that IPE can be an important method to instill foundational interprofessional knowledge, skills, and attitudes to promote a foundation from which to establish strong career-long interprofessional collaborations. It is important to lay foundational interprofessional skills and appreciation of the 'other' in pre-licensure curricula, but these efforts should not be limited to only pre-licensure programs and need also to be included as part of on-going professional development education, especially as healthcare education, roles, and responsibilities evolve.

Effect of exogenous testosterone in the context of energy deficit on risky choice: Behavioural and neural evidence from males.

Previous research has shown greater risk aversion when people make choices about lives than cash. We tested the hypothesis that compared to placebo, exogenous testosterone administration would lead to riskier choices about cash than lives, given testosterone's association with financial risk-taking and reward sensitivity. A double-blind, placebo-controlled, randomized trial was conducted to test this hypothesis (Clinical Trials Registry: NCT02734238, www.clinicaltrials.gov). We collected functional magnetic resonance imaging (fMRI) data from 50 non-obese males before and shortly after 28 days of severe exercise-and-diet-induced energy deficit, during which testosterone (200 mg testosterone enanthate per week in sesame oil) or placebo (sesame seed oil only) was administered. Because we expected circulating testosterone levels to be reduced due to severe energy deficit, testosterone administration served a restorative function to mitigate the impact of energy deficit on testosterone levels. The fMRI task involved making choices under uncertainty for lives and cash. We also manipulated whether the outcomes were presented as gains or losses. Consistent with prospect theory, we observed the reflection effect such that participants were more risk averse when outcomes were presented as gains than losses. Brain activation in the thalamus covaried with individual differences in exhibiting the reflection effect. Testosterone did not impact choice, but it increased sensitivity to negative feedback following risky choices. These results suggest that exogenous testosterone administration in the context of energy deficit can impact some aspects of risky choice, and that individual differences in the reflection effect engage a brain structure involved in processing emotion, reward and risk.

Predicting reliability through structured expert elicitation with the repliCATS (Collaborative Assessments for Trustworthy Science) process.

As replications of individual studies are resource intensive, techniques for predicting the replicability are required. We introduce the repliCATS (Collaborative Assessments for Trustworthy Science) process, a new method for eliciting expert predictions about the replicability of research. This process is a structured expert elicitation approach based on a modified Delphi technique applied to the evaluation of research claims in social and behavioural sciences. The utility of processes to predict replicability is their capacity to test scientific claims without the costs of full replication. Experimental data supports the validity of this process, with a validation study producing a classification accuracy of 84% and an Area Under the Curve of 0.94, meeting or exceeding the accuracy of other techniques used to predict replicability. The repliCATS process provides other benefits. It is highly scalable, able to be deployed for both rapid assessment of small numbers of claims, and assessment of high volumes of claims over an extended period through an online elicitation platform, having been used to assess 3000 research claims over an 18 month period. It is available to be implemented in a range of ways and we describe one such implementation. An important advantage of the repliCATS process is that it collects qualitative data that has the potential to provide insight in understanding the limits of generalizability of scientific claims. The primary limitation of the repliCATS process is its reliance on human-derived predictions with consequent costs in terms of participant fatigue although careful design can minimise these costs. The repliCATS process has potential applications in alternative peer review and in the allocation of effort for replication studies.

Communicating uncertainty using words and numbers.

Life in an increasingly information-rich but highly uncertain world calls for an effective means of communicating uncertainty to a range of audiences. Senders prefer to convey uncertainty using verbal (e.g., likely) rather than numeric (e.g., 75% chance) probabilities, even in consequential domains, such as climate science. However, verbal probabilities can convey something other than uncertainty, and senders may exploit this. For instance, senders can maintain credibility after making erroneous predictions. While verbal probabilities afford ease of expression, they can be easily misunderstood, and the potential for miscommunication is not effectively mitigated by assigning (imprecise) numeric probabilities to words. When making consequential decisions, recipients prefer (precise) numeric probabilities.

Principal investigators over-optimistically forecast scientific and operational outcomes for clinical trials.

OBJECTIVE: To assess the accuracy of principal investigators' (PIs) predictions about three events for their own clinical trials: positivity on trial primary outcomes, successful recruitment and timely trial completion. STUDY DESIGN AND SETTING: A short, electronic survey was used to elicit subjective probabilities within seven months of trial registration. When trial results became available, prediction skill was calculated using Brier scores (BS) and compared against uninformative prediction (i.e. predicting 50% all of the time). RESULTS: 740 PIs returned surveys (16.7% response rate). Predictions on all three events tended to exceed observed event frequency. Averaged PI skill did not surpass uninformative predictions (e.g., BS = 0.25) for primary outcomes (BS = 0.25, 95% CI 0.20, 0.30) and were significantly worse for recruitment and timeline predictions (BS 0.38, 95% CI 0.33, 0.42; BS = 0.52, 95% CI 0.50, 0.55, respectively). PIs showed poor calibration for primary outcome, recruitment, and timelines (calibration index = 0.064, 0.150 and 0.406, respectively), modest discrimination in primary outcome predictions (AUC = 0.76, 95% CI 0.65, 0.85) but minimal discrimination in the other two outcomes (AUC = 0.64, 95% CI 0.57, 0.70; and 0.55, 95% CI 0.47, 0.62, respectively). CONCLUSION: PIs showed overconfidence in favorable outcomes and exhibited limited skill in predicting scientific or operational outcomes for their own trials. They nevertheless showed modest ability to discriminate between positive and non-positive trial outcomes. Low survey response rates may limit generalizability.

Canadian and Spanish youths' risk perceptions of drinking and driving, and riding with a drunk driver.

The present research compared Canadian and Spanish youths' perceptions of the potential benefits and drawbacks of driving under the influence of alcohol (DUI) and riding with a drunk driver (RDD). Eighty (41 female) Canadian and 87 (71 female) Spanish undergraduates completed a survey asking about their past and forecasted engagement in DUI and RDD, and their perceptions of the benefits and drawbacks of DUI and RDD. A sizeable proportion of both samples reported DUI and RDD in the past year. Past risk takers forecasted significantly greater chances of engaging in these behaviors in the following year compared to those who had not engaged in DUI and RDD. Both samples provided significantly more drawbacks than benefits of DUI and RDD. Whereas the benefits of both behaviors tended to refer to personal effects (e.g., save money, arrive faster) that occurred before, during, or after driving, the drawbacks referred to a range of outcomes (e.g., accident, kill/injure, penal sanction) that mostly occurred during driving. Although Canada and Spain differ in important respects (e.g., potential penalty for DUI), there were similarities in the two samples' perceptions of DUI and RDD. Young people are aware of the costs of these risky behaviors but nevertheless engage in them. These findings can inform theories of the co-occurrence of risky driving behaviors, and the development of prevention programs that focus on perceived outcomes.

2011 recommendations for the diagnosis and management of gout and hyperuricemia.

Gout is a major health problem in the United States; it affects 8.3 million people, which is approximately 4% of the adult population. Gout is most often diagnosed and managed in primary care practices; thus, primary care physicians have a significant opportunity to improve patient outcomes. Following publication of the 2006 European League Against Rheumatism (EULAR) gout guidelines, significant new evidence has accumulated, and new treatments for patients with gout have become available. It is the objective of these 2011 recommendations to update the 2006 EULAR guidelines, paying special attention to the needs of primary care physicians. The revised 2011 recommendations are based on the Grading of Recommendations Assessment, Development, and Evaluation approach as an evidence-based strategy for rating quality of evidence and grading the strength of recommendation formulated for use in clinical practice. A total of 26 key recommendations, 10 for diagnosis and 16 for management, of patients with gout were evaluated, resulting in important updates for patient care. The presence of monosodium urate crystals and/or tophus and response to colchicine have the highest clinical diagnostic value. The key aspect of effective management of an acute gout attack is initiation of treatment within hours of symptom onset. Low-dose colchicine is better tolerated and is as effective as a high dose. When urate-lowering therapy (ULT) is indicated, the xanthine oxidase inhibitors allopurinol and febuxostat are the options of choice. Febuxostat can be prescribed at unchanged doses for patients with mild-to-moderate renal or hepatic impairment. The target of ULT should be a serum uric acid level that is ≤ 6 mg/dL. For patients with refractory and tophaceous gout, intravenous pegloticase is a new treatment option. This article is a summary of the 2011 clinical guidelines published in Postgraduate Medicine. This article provides a streamlined, accessible overview intended for quick review by primary care physicians, with the full guidelines being a resource for those seeking additional background information and expanded discussion.

On measuring agreement with numerically bounded linguistic probability schemes: A re-analysis of data from Wintle, Fraser, Wills, Nicholson, and Fidler (2019).

Across a wide range of domains, experts make probabilistic judgments under conditions of uncertainty to support decision-making. These judgments are often conveyed using linguistic expressions (e.g., x is likely). Seeking to foster shared understanding of these expressions between senders and receivers, the US intelligence community implemented a communication standard that prescribes a set of probability terms and assigns each term an equivalent numerical probability range. In an earlier PLOS ONE article, [1] tested whether access to the standard improves shared understanding and also explored the efficacy of various enhanced presentation formats. Notably, they found that embedding numeric equivalents in text (e.g., x is likely [55-80%]) substantially outperformed the status-quo approach in terms of the percentage overlap between participants' interpretations of linguistic probabilities (defined in terms of the numeric range equivalents they provided for each term) and the numeric ranges in the standard. These results have important prescriptive implications, yet Wintle et al.'s percentage overlap measure of agreement may be viewed as unfairly punitive because it penalizes individuals for being more precise than the stipulated guidelines even when the individuals' interpretations fall perfectly within the stipulated ranges. Arguably, subjects' within-range precision is a positive attribute and should not be penalized in scoring interpretive agreement. Accordingly, in the present article, we reanalyzed Wintle et al.'s data using an alternative measure of percentage overlap that does not penalize in-range precision. Using the alternative measure, we find that percentage overlap is substantially elevated across conditions. More importantly, however, the effects of presentation format and probability level are highly consistent with the original study. By removing the ambiguity caused by Wintle et al.'s unduly punitive measure of agreement, these findings buttress Wintle et al.'s original claim that the methods currently used by intelligence organizations are ineffective at coordinating the meaning of uncertainty expressions between intelligence producers and intelligence consumers. Future studies examining agreement between senders and receivers are also encouraged to reflect carefully on the most appropriate measures of agreement to employ in their experiments and to explicate the bases for their methodological choices.

Words or numbers? Communicating probability in intelligence analysis.

Intelligence analysis is fundamentally an exercise in expert judgment made under conditions of uncertainty. These judgments are used to inform consequential decisions. Following the major intelligence failure that led to the 2003 war in Iraq, intelligence organizations implemented policies for communicating probability in their assessments. Virtually all chose to convey probability using standardized linguistic lexicons in which an ordered set of select probability terms (e.g., highly likely) is associated with numeric ranges (e.g., 80-90%). We review the benefits and drawbacks of this approach, drawing on psychological research on probability communication and studies that have examined the effectiveness of standardized lexicons. We further discuss how numeric probabilities can overcome many of the shortcomings of linguistic probabilities. Numeric probabilities are not without drawbacks (e.g., they are more difficult to elicit and may be misunderstood by receivers with poor numeracy). However, these drawbacks can be ameliorated with training and practice, whereas the pitfalls of linguistic probabilities are endemic to the approach. We propose that, on balance, the benefits of using numeric probabilities outweigh their drawbacks. Given the enormous costs associated with intelligence failure, the intelligence community should reconsider its reliance on using linguistic probabilities to convey probability in intelligence assessments. Our discussion also has implications for probability communication in other domains such as climate science. (PsycInfo Database Record (c) 2021 APA, all rights reserved).