Standardizing to specific target populations in distributed networks and multisite pharmacoepidemiologic studies.

Distributed network studies and multisite studies assess drug safety and effectiveness in diverse populations by pooling information. Targeting groups of clinical or policy interest (including specific sites or site combinations) and applying weights based on effect measure modifiers (EMMs) prior to pooling estimates within multisite studies may increase interpretability and improve precision. We simulated a 4-site study, standardized each site using inverse odds weights (IOWs) to resemble the 3 smallest sites or the smallest site, estimated IOW-weighted risk differences (RDs), and combined estimates with inverse variance weights (IVWs). We also created an artificial distributed network in the Clinical Practice Research Datalink (CPRD) Aurum consisting of 1 site for each geographic region. We compared metformin and sulfonylurea initiators with respect to mortality, targeting the smallest region. In the simulation, IOWs reduced differences between estimates and increased precision when targeting the 3 smallest sites or the smallest site. In the CPRD Aurum study, the IOW + IVW estimate was also more precise (smallest region: RD = 5.41% [95% CI, 1.03-9.79]; IOW + IVW estimate: RD = 3.25% [95% CI, 3.07-3.43]). When performing pharmacoepidemiologic research in distributed networks or multisite studies in the presence of EMMs, designation of target populations has the potential to improve estimate precision and interpretability. This article is part of a Special Collection on Pharmacoepidemiology.

Variable selection when estimating effects in external target populations.

External validity is an important part of epidemiologic research. To validly estimate effects in specific external target populations using a chosen effect measure (ie, "transport"), some methods require that one account for all effect measure modifiers (EMMs). However, little is known about how including other variables that are not EMMs (ie, non-EMMs) in adjustment sets affects estimates. Using simulations, we evaluated how inclusion of non-EMMs affected estimation of the transported risk difference (RD) by assessing the impacts of covariates that (1) differ (or not) between the trial and the target, (2) are associated with the outcome (or not), and (3) modify the RD (or not). We assessed variation and bias when covariates with each possible combination of these factors were used to transport RDs using outcome modeling or inverse odds weighting. Inclusion of variables that differed in distribution between the populations but were non-EMMs reduced precision, regardless of whether they were associated with the outcome. However, non-EMMs associated with selection did not amplify bias resulting from omission of necessary EMMs. Including all variables associated with the outcome may result in unnecessarily imprecise estimates when estimating treatment effects in external target populations.

A critical analysis of stress-related gene regulation in isolated and connected individuals.

This letter responds to the article by Yvonne S. Yang et al. (2024) on the relationship between social isolation, loneliness, and stress-related gene expression, published in Brain, Behavior, and Immunity. The authors' recruitment methods and use of genome-wide transcriptional profiling to assess the Conserved Transcriptional Response to Adversity (CTRA) are highly commendable, offering a comprehensive understanding of how social isolation impacts immune gene expression. However, we raise concerns about the small sample size, demographic limitations, cross-sectional design, and lack of correction for multiple comparisons, which may affect the study's generalizability and validity. Despite these limitations, the study provides valuable insights, paving the way for future research to explore the complex interactions between social isolation, loneliness, and health. We recommend that future studies address these limitations to enhance the robustness of findings and the development of effective interventions.

Comparing clinical trial population representativeness to real-world users of 17 biologics approved for immune-mediated inflammatory diseases: An external validity analysis of 66,639 biologic users from the Italian VALORE project.

To date, no population-based studies have specifically explored the external validity of pivotal randomized clinical trials (RCTs) of biologics simultaneously for a broad spectrum of immuno-mediated inflammatory diseases (IMIDs). The aims of this study were, firstly, to compare the patients' characteristics and median treatment duration of biologics approved for IMIDs between RCTs' and real-world setting (RW); secondly, to assess the extent of biologic users treated for IMIDs in the real-world setting that would not have been eligible for inclusion into pivotal RCT for each indication of use. Using the Italian VALORE distributed database (66,639 incident biologic users), adult patients with IMIDs treated with biologics in the Italian real-world setting were substantially older (mean age ± SD: 50 ± 15 years) compared to those enrolled in pivotal RCTs (45 ± 15 years). In the real-world setting, certolizumab pegol was more commonly used by adult women with psoriasis/ankylosing spondylitis (F/M ratio: 1.8-1.9) compared to RCTs (F/M ratio: 0.5-0.6). The median treatment duration (weeks) of incident biologic users in RW was significantly higher than the duration of pivotal RCTs in almost all indications for use and most biologics (4-100 vs. 6-167). Furthermore, almost half (46.4%) of biologic users from RW settings would have been ineligible for inclusion in the respective indication-specific pivotal RCTs. The main reasons were: advanced age, recent history of cancer and presence of other concomitant IMIDs. These findings suggest that post-marketing surveillance of biologics should be prioritized for those patients.

Applying Trial-Derived Treatment Effects to Real-World Populations: Generalizing Cost-Effectiveness Estimates When Modeling Complex Hazards.

OBJECTIVES: Generalizability of trial-based cost-effectiveness estimates to real-world target populations is important for decision making. In the context of independent aggregate time-to-event baseline and relative effects data, complex hazards can make modeling of data for use in economic evaluation challenging. Our article provides an overview of methods that can be used to apply trial-derived relative treatment effects to external real-world baselines when faced with complex hazards and follows with a motivating example. METHODS: Approaches for applying trial-derived relative effects to real-world baselines are presented in the context of complex hazards. Appropriate methods are applied in a cost-effectiveness analysis using data from a previously published study assessing the real-world cost-effectiveness of a treatment for carcinoma of the head and neck as a motivating example. RESULTS: Lack of common hazards between the trial and target real-world population, a complex baseline hazard function, and nonproportional relative effects made the use of flexible models necessary to adequately estimate survival. Assuming common distributions between trial and real-world reference survival substantially affected survival and cost-effectiveness estimates. Modeling time-dependent vs proportional relative effects affected estimates to a lesser extent, dependent on assumptions used in cost-effectiveness modeling. CONCLUSIONS: Appropriately capturing reference treatment survival when attempting to generalize trial-derived relative treatment effects to real-world target populations can have important impacts on cost-effectiveness estimates. A balance between model complexity and adequacy for decision making should be considered where multiple data sources with complex hazards are being evaluated.

Evaluative Conditioning: Past, Present, and Future.

Evaluative conditioning (EC) research investigates changes in the evaluation of a stimulus after co-occurrence with an affective stimulus. To explain the motivation behind this research, this review begins with an overview of the history of EC research, followed by a summary of the state of the art with respect to three key questions. First, how should EC procedures be used to influence evaluation? We provide a guide based on evidence concerning the functional properties of EC effects. Second, how does the EC effect occur? We discuss the possible mediating cognitive processes and their automaticity. Third, are EC effects ubiquitous outside the lab? We discuss the evidence for the external validity of EC research. We conclude that the most important open questions pertain to the relevance of EC to everyday life and to the level of control that characterizes the processes that mediate the EC effect after people notice the stimulus co-occurrence.

Force-velocity-power profiling in flywheel squats: Differences between sports and association with countermovement jump and change of direction performance.

The objectives of our study were 1) to examine whether the force-velocity-power (FvP) outcomes in the concentric and eccentric phases of flywheel (FW) squats differ among sports disciplines and 2) to investigate the association between FvP outcome variables and two key sport-performance indicators: countermovement jump height (CMJ) and change of direction (CoD) time involving 90° or 180° turns. Tests were performed by 469 athletes from five different sport disciplines and physical education students. Our results showed that FvP outcomes, when measured during the eccentric phase of the FW squat, effectively differed between athletes from different sports. However, during the concentric phase of the squat, only for the slope variable differences between sports were found. Contrary to our hypothesis, there were no statistically significant correlations between FW squat FvP outcomes and CMJ or CoD test results. These results suggest that FvP outcomes derived from FW squats may lack external validity and cannot be reliably used as a predictor of athletes' jumping and agility performance. The paper discusses possible reasons for the larger differences between sports in eccentric FvP outcomes, as well as the absence of correlations between FvP outcomes and functional tests.

The Alzheimer's Disease Neuroimaging Initiative-4 (ADNI-4) Engagement Core: A culturally informed, community-engaged research (CI-CER) model to advance brain health equity.

INTRODUCTION: The Alzheimer's Disease Neuroimaging Initiative-4 (ADNI-4) Engagement Core was launched to advance Alzheimer's disease (AD) and AD-related dementia (ADRD) health equity research in underrepresented populations (URPs). We describe our evidence-based, scalable culturally informed, community-engaged research (CI-CER) model and demonstrate its preliminary success in increasing URP enrollment. METHODS: URPs include ethnoculturally minoritized, lower education (≤ 12 years), and rural populations. The CI-CER model includes: (1) culturally informed methodology (e.g., less restrictive inclusion/exclusion criteria, sociocultural measures, financial compensation, results disclosure, Spanish Language Capacity Workgroup) and (2) inclusive engagement methods (e.g., the Engagement Core team; Hub Sites; Community-Science Partnership Board). RESULTS: As of April 2024, 60% of ADNI-4 new in-clinic enrollees were from ethnoculturally or educationally URPs. This exceeds ADNI-4's ≥ 50% URP representation goal for new enrollees but may not represent final enrollment. DISCUSSION: Findings show a CI-CER model increases URP enrollment in AD/ADRD clinical research and has important implications for clinical trials to advance health equity. HIGHLIGHTS: The Alzheimer's Disease Neuroimaging Initiative-4 (ADNI-4) uses a culturally informed, community-engaged research (CI-CER) approach. The CI-CER approach is scalable and sustainable for broad, multisite implementation. ADNI-4 is currently exceeding its inclusion goals for underrepresented populations.

Eligibility for Substance Use Clinical Trials Among Emergency Psychiatry Patients: The Impact of Exclusion Criteria.

Objective: The first objective was to identify common exclusion criteria used in clinical trials. The second objective was to quantify the degree to which these criteria exclude emergency psychiatry patients. Methods: Qualitative Content Analysis was used for the first objective, identifying common exclusion criteria used in recent high-impact substance use clinical trials. A retrospective record review was used for the second objective, which examined the frequency of these exclusion criteria in a 1-month sample of adults receiving psychiatric evaluation in an emergency department. Results: Most trials had exclusions for co-occurring psychiatric problems (76.6%), medical problems (74.0%), prior or current treatment (72.7%), motivation for change (61.1%), pregnancy or lactation (57.1%), or using other specified substances of abuse (54.6%). In the clinical sample, exclusions for co-occurring psychiatric problems would make 94.7% of patients ineligible. Other exclusions had a combined effect of making 76% of patients ineligible. Conclusions: Clinical trials using typical exclusion criteria exclude nearly all emergency psychiatry patients with substance use problems.

Individual treatment selection for patients with post-traumatic stress disorder: External validation of a personalised advantage index.

OBJECTIVE: To test the predictive accuracy and generalisability of a personalised advantage index (PAI) model designed to support treatment selection for Post-Traumatic Stress Disorder (PTSD). METHOD: A PAI model developed by Deisenhofer et al. (2018) was used to predict treatment outcomes in a statistically independent dataset including archival records for N = 152 patients with PSTD who accessed either trauma-focussed cognitive behavioural therapy or eye movement desensitisation and reprocessing in routine care. Outcomes were compared between patients who received their PAI-indicated optimal treatment versus those who received their suboptimal treatment. RESULTS: The model did not yield treatment specific predictions and patients who had received their PAI-indicated optimal treatment did not have better treatment outcomes in this external validation sample. CONCLUSION: This PAI model did not generalise to an external validation sample.

The tricot approach: an agile framework for decentralized on-farm testing supported by citizen science. A retrospective.

Matching crop varieties to their target use context and user preferences is a challenge faced by many plant breeding programs serving smallholder agriculture. Numerous participatory approaches proposed by CGIAR and other research teams over the last four decades have attempted to capture farmers' priorities/preferences and crop variety field performance in representative growing environments through experimental trials with higher external validity. Yet none have overcome the challenges of scalability, data validity and reliability, and difficulties in capturing socio-economic and environmental heterogeneity. Building on the strengths of these attempts, we developed a new data-generation approach, called triadic comparison of technology options (tricot). Tricot is a decentralized experimental approach supported by crowdsourced citizen science. In this article, we review the development, validation, and evolution of the tricot approach, through our own research results and reviewing the literature in which tricot approaches have been successfully applied. The first results indicated that tricot-aggregated farmer-led assessments contained information with adequate validity and that reliability could be achieved with a large sample. Costs were lower than current participatory approaches. Scaling the tricot approach into a large on-farm testing network successfully registered specific climatic effects of crop variety performance in representative growing environments. Tricot's recent application in plant breeding networks in relation to decision-making has (i) advanced plant breeding lines recognizing socio-economic heterogeneity, and (ii) identified consumers' preferences and market demands, generating alternative breeding design priorities. We review lessons learned from tricot applications that have enabled a large scaling effort, which should lead to stronger decision-making in crop improvement and increased use of improved varieties in smallholder agriculture.

How Choice of Effect Measure Influences Minimally Sufficient Adjustment Sets for External Validity.

Epidemiologic researchers generalizing or transporting effect estimates from a study to a target population must account for effect-measure modifiers (EMMs) on the scale of interest. However, little attention is paid to how the EMMs required may vary depending on the mathematical nuances of each effect measure. We defined 2 types of EMMs: a marginal EMM, where the effect on the scale of interest differs across levels of a variable, and a conditional EMM, where the effect differs conditional on other variables associated with the outcome. These types define 3 classes of variables: class 1 (conditional EMM), class 2 (marginal but not conditional EMM), and class 3 (neither marginal nor conditional EMM). Class 1 variables are necessary to achieve a valid estimate of the RD in a target population, while an RR requires class 1 and class 2 and an OR requires classes 1, 2, and 3 (i.e., all variables associated with the outcome). This does not mean that fewer variables are required for an externally valid RD (because variables may not modify effects on all scales), but it does suggest that researchers should consider the scale of the effect measure when identifying an EMM necessary for an externally valid treatment effect estimate.

Proxy Variables and the Generalizability of Study Results.

When individuals self-select (or are selected) into a study based on factors that influence the outcome, conclusions may not generalize to the full population. To compensate for this, results may be adjusted, for example, by standardization on the set of common causes of participation and outcome. Although such standardization is useful in some contexts, the common causes of participation and outcome may in practice not be fully observed. Instead, the researcher may have access to one or several variables related to the common causes, that is, to proxies for the common causes. This article defines and examines different types of proxy variables and shows how these can be used to obtain generalizable study results. First of all, the researcher may exploit proxies that influence only participation or outcome but which still allow for perfect generalizability by rendering participation and outcome conditionally independent. Further, generalizability can be achieved by leveraging 2 proxies, one of which is allowed to influence participation and one of which is allowed to influence the outcome, even if participation and outcome do not become independent conditional on these. Finally, approximate generalizability may be obtained by exploiting a single proxy that does not itself influence participation or outcome.

Conditional cross-design synthesis estimators for generalizability in Medicaid.

While much of the causal inference literature has focused on addressing internal validity biases, both internal and external validity are necessary for unbiased estimates in a target population of interest. However, few generalizability approaches exist for estimating causal quantities in a target population that is not well-represented by a randomized study but is reflected when additionally incorporating observational data. To generalize to a target population represented by a union of these data, we propose a novel class of conditional cross-design synthesis estimators that combine randomized and observational data, while addressing their estimates' respective biases-lack of overlap and unmeasured confounding. These methods enable estimating the causal effect of managed care plans on health care spending among Medicaid beneficiaries in New York City, which requires obtaining estimates for the 7% of beneficiaries randomized to a plan and 93% who choose a plan, who do not resemble randomized beneficiaries. Our new estimators include outcome regression, propensity weighting, and double robust approaches. All use the covariate overlap between the randomized and observational data to remove potential unmeasured confounding bias. Applying these methods, we find substantial heterogeneity in spending effects across managed care plans. This has major implications for our understanding of Medicaid, where this heterogeneity has previously been hidden. Additionally, we demonstrate that unmeasured confounding rather than lack of overlap poses a larger concern in this setting.

Transporting monovalent rotavirus vaccine efficacy estimates to an external target population: a secondary analysis of data from a randomised controlled trial in Malawi.

Oral rotavirus vaccine efficacy estimates from randomised controlled trials are highly variable across settings. Although the randomised study design increases the likelihood of internal validity of findings, results from trials may not always apply outside the context of the study due to differences between trial participants and the target population. Here, we used a weight-based method to transport results from a monovalent rotavirus vaccine clinical trial conducted in Malawi between 2005 and 2008 to a target population of all trial-eligible children in Malawi, represented by data from the 2015-2016 Malawi Demographic and Health Survey (DHS). We reweighted trial participants to reflect the population characteristics described by the Malawi DHS. Vaccine efficacy was estimated for 1008 trial participants after applying these weights such that they represented trial-eligible children in Malawi. We also conducted subgroup analyses to examine the heterogeneous treatment effects by stunting and tuberculosis vaccination status at enrolment. In the original trial, the estimates of one-year vaccine efficacy against severe rotavirus gastroenteritis and any-severity rotavirus gastroenteritis in Malawi were 49.2% (95% CI 15.6%-70.3%) and 32.1% (95% CI 2.5%-53.1%), respectively. After weighting trial participants to represent all trial-eligible children in Malawi, vaccine efficacy increased to 62.2% (95% CI 35.5%-79.0%) against severe rotavirus gastroenteritis and 38.9% (95% CI 11.4%-58.5%) against any-severity rotavirus gastroenteritis. Rotavirus vaccine efficacy may differ between trial participants and target populations when these two populations differ. Differences in tuberculosis vaccination status between the trial sample and DHS population contributed to varying trial and target population vaccine efficacy estimates.

Revisiting the link between the sustained attention to response task (SART) and daily-life cognitive failures.

In this study, we examined the relationship betweenerrors of commissionon theSustained Attention to Response Task(SART)andscores on the Cognitive Failures Questionnaire (CFQ). The goal was to assess theecological validity of the SARTin a sample of people scoring high on fatigue complaints.SART errors of commission were positively associated with CFQ scores and this finding remained after controlling for fatigue level, age, and SART reaction times.Thus, our results generally supported the ecological validity of the SART. However, when examining subsamples separately, we found the association between SART and CFQ only in our subsample of employees, not in our subsample of university students. The three subscales of the CFQ showed the same pattern of findings. Our results imply that, when using the SART to draw conclusions about everyday life, it is crucial to consider the characteristics of one's sample and control for relevant confounding variables.

External Validity of a Randomized Controlled Trial on Duration of Antibiotics for the Treatment of Gram-Negative Bacteremia.

INTRODUCTION: Reports regarding the external validity of randomized controlled trials (RCTs) are scarce. We aimed to assess the population external validity of an investigator-initiated RCT on the duration of antibiotics for the treatment of Gram-negative bacteremia by comparing patients included in the RCT to patients that were not included in the trial. METHODS: Hospitalized patients with Gram-negative bacteremia were recruited into an RCT and randomized to receive 7 or 14 days of covering antibiotic therapy in Israel and Italy from 2013 to 2017. In a concomitant observational study, RCT participants were compared with patients who fulfilled the inclusion criteria but were not included in the trial due to participation in other trials, discharge before approached by researchers, refusal to participate, or unwillingness of the treating physician to allow participants' recruitment. RESULTS: Six hundred and four RCT patients were compared with 613 nonincluded patients. Almost 50% of nonincluded patients (288/613) were dependent on others for activities of daily living at baseline compared to 37.7% of RCT participants (228/604). Dementia was nearly 2-fold more frequent in nonincluded patients than those included (5.9% [36/613] versus 3.6% [22/604], p = 0.07). Patients who were not included in the RCT were more likely to acquire their infection in the hospital (53.3% [327/613] versus 29.1% [176/604], p < 0.001). The primary composite outcome of mortality, clinical failure, readmissions, or extended hospitalization at 90 days occurred in 353 of 613 nonincluded patients (57.6%) compared to 299 of 604 RCT participants (49.6%), p = 0.005. However, on multivariate analysis noninclusion in the RCT was not an independent risk factor for clinical failure and mortality. CONCLUSIONS: RCTs, even with broad eligibility criteria, do not represent the whole spectrum of patients and leave out a population with more severe illness for whom the evidence is lacking.

Keeping track of reality: embedding visual memory in natural behaviour.

Since immersive virtual reality (IVR) emerged as a research method in the 1980s, the focus has been on the similarities between IVR and actual reality. In this vein, it has been suggested that IVR methodology might fill the gap between laboratory studies and real life. IVR allows for high internal validity (i.e., a high degree of experimental control and experimental replicability), as well as high external validity by letting participants engage with the environment in an almost natural manner. Despite internal validity being crucial to experimental designs, external validity also matters in terms of the generalizability of results. In this paper, we first highlight and summarise the similarities and differences between IVR, desktop situations (both non-immersive VR and computer experiments), and reality. In the second step, we propose that IVR is a promising tool for visual memory research in terms of investigating the representation of visual information embedded in natural behaviour. We encourage researchers to carry out experiments on both two-dimensional computer screens and in immersive virtual environments to investigate visual memory and validate and replicate the findings. IVR is valuable because of its potential to improve theoretical understanding and increase the psychological relevance of the findings.

Smallholder farmers and contract farming in developing countries.

Poverty is prevalent in the small-farm sector of many developing countries. A large literature suggests that contract farming-a preharvest agreement between farmers and buyers-can facilitate smallholder market participation, improve household welfare, and promote rural development. These findings have influenced the development policy debate, but the external validity of the extant evidence is limited. Available studies typically focus on a single contract scheme or on a small geographical area in one country. We generate evidence that is generalizable beyond a particular contract scheme, crop, or country, using nationally representative survey data from 6 countries. We focus on the implications of contract farming for household income and labor demand, finding that contract farmers obtain higher incomes than their counterparts without contracts only in some countries. Contract farmers in most countries exhibit increased demand for hired labor, which suggests that contract farming stimulates employment, yet we do not find evidence of spillover effects at the community level. Our results challenge the notion that contract farming unambiguously improves welfare. We discuss why our results may diverge from previous findings and propose research designs that yield greater internal and external validity. Implications for policy and research are relevant beyond contract farming.

REPRESENT: REPresentativeness of RESearch data obtained through the 'General Informed ConsENT'.

BACKGROUND: We assessed potential consent bias in a cohort of > 40,000 adult patients asked by mail after hospitalization to consent to the use of past, present and future clinical and biological data in an ongoing 'general consent' program at a large tertiary hospital in Switzerland. METHODS: In this retrospective cohort study, all adult patients hospitalized between April 2019 and March 2020 were invited to participate to the general consent program. Demographic and clinical characteristics were extracted from patients' electronic health records (EHR). Data of those who provided written consent (signatories) and non-responders were compared and analyzed with R studio. RESULTS: Of 44,819 patients approached, 10,299 (23%) signed the form. Signatories were older (median age 54 [IQR 38-72] vs. 44 years [IQR 32-60], p < .0001), more comorbid (2614/10,299 [25.4%] vs. 4912/28,676 [17.1%] with Charlson comorbidity index ≤ 4, p < .0001), and more often of Swiss nationality (6592/10,299 [64%] vs. 13,813/28,676 [48.2%], p < .0001). CONCLUSIONS: Our results suggest that actively seeking consent creates a bias and compromises the external validity of data obtained via 'general consent' programs. Other options, such as opt-out consent procedures, should be further assessed.