Demographic diversity of participants in clinical trials conducted in Singapore.

Introduction: The under-representativeness of participants in clinical trials limits the generalisability of results. This review evaluates the representative-ness within pharmaceutical randomised controlled trials (RCTs) in Singapore. Method: Four bibliographic databases were searched for papers on pharmaceutical RCTs which included Singapore adults (≥18 years old), published between 2017 and 2022. The demographic characteristics of study participants were compared against the population in the 2020 Singapore census. Recruitment strategies and authors' comments on the generalisa-bility of their findings were reviewed. Results: Thirty-three publications were included (19 Singapore-only studies and 14 multiregional trials which included Singapore). Where data were available, we found that females and Indians were under-represented compared to the census (41.3% versus [vs] 51.1%, P<0.05; 7.3% vs 9.0%, P<0.05). Ethnic diversity varied between individual studies, and almost half (46.2%) of Singapore-only studies achieved census levels. However, more than one-third of the trials provided no data (31.6%) or partial data (5.3%) on ethnicity. Half of the multiregional publications stated the number of participants recruited from Singapore, but only 1 reported any detail beyond Asian participants. Recruitment strategies were mentioned in fewer than half (42.4%), and less than a quarter (24.2%) commented on sample representative-ness or the external validity of the evidence generated. Conclusion: There is room for improvement regarding the recruitment of RCT participants in Singapore, with particular attention to female gender and Indian ethnicity. Demographic data should also be presented in full. RCTs should be designed and reported such that clinicians can ascertain the generalisability to the Singapore population and the potential benefits from the studied interventions in clinical practice.

Testing a policy intervention in the lab: differences between students and non-students in switching bank accounts.

The reliance on student samples has long been a subject of debate in experimental approaches to studying behaviour. We contribute to this discussion by looking at differences in financial behaviour between a student and a non-student sample in three sets of lab experiments conducted in Spain, Germany and Poland (n=857). Participants from both samples switched more often and made better financial decisions after they received a message encouraging them to switch financial service providers. While the size of the effect on switching frequency was comparable between the two samples, the effect on switching quality was significantly stronger on non-students. Further analysis suggests this is due to a better performance of students before the prompt leaving less room for improvement by the reminder. Results suggest that experimental evidence derived from students should be generalized with caution.

Comparing the DSM-5 Dimensional Trait and Triarchic Model Conceptions of Psychopathy: An External Validity Analysis.

In the DSM-5 Alternative Model of Personality Disorders (AMPD), psychopathy is marked by the presence of attention seeking, low anxiousness, and lack of social withdrawal, along with traits from the domains of Antagonism and Disinhibition. The triarchic model of psychopathy (TriPM) posits three biobehaviorally based traits underlying it: disinhibition, meanness, and boldness. The current study directly compared relations for measures of the two models with the broad dimensions of externalizing, internalizing, and positive adjustment. Participants (1,678 adults) were surveyed regarding maladaptive personality traits, clinical symptoms, and positive adjustment features. The TriPM model explained more variance than the AMPD in substance use, positive adjustment, and empathy, whereas the AMPD model explained more variance in internalizing symptoms. In addition, AMPD Antagonism and the Psychopathy Specifier diverged from TriPM Meanness and Boldness in their associations with some specific outcomes. Overall, our study provides evidence for complementarity of the two models in characterizing the multifaceted nature of psychopathy.

Eligibility for Substance Use Clinical Trials Among Emergency Psychiatry Patients: The Impact of Exclusion Criteria.

Objective: The first objective was to identify common exclusion criteria used in clinical trials. The second objective was to quantify the degree to which these criteria exclude emergency psychiatry patients. Methods: Qualitative Content Analysis was used for the first objective, identifying common exclusion criteria used in recent high-impact substance use clinical trials. A retrospective record review was used for the second objective, which examined the frequency of these exclusion criteria in a 1-month sample of adults receiving psychiatric evaluation in an emergency department. Results: Most trials had exclusions for co-occurring psychiatric problems (76.6%), medical problems (74.0%), prior or current treatment (72.7%), motivation for change (61.1%), pregnancy or lactation (57.1%), or using other specified substances of abuse (54.6%). In the clinical sample, exclusions for co-occurring psychiatric problems would make 94.7% of patients ineligible. Other exclusions had a combined effect of making 76% of patients ineligible. Conclusions: Clinical trials using typical exclusion criteria exclude nearly all emergency psychiatry patients with substance use problems.

Simulated patient methodology as a "gold standard" in community pharmacy practice: Response to criticism.

The simulated patient methodology (SPM) is considered the "gold standard" as covert participatory observation. SPM is attracting increasing interest for the investigation of community pharmacy practice; however, there is criticism that SPM can only show a small picture of everyday pharmacy practice and therefore has limited external validity. On the one hand, a certain design and application of the SPM goes hand in hand with an increase in external validity. Even if, on the other hand, this occurs at the expense of internal validity due to the trade-off situation, the justified criticism of the SPM for investigating community pharmacy practice can be countered.

Individual treatment selection for patients with post-traumatic stress disorder: External validation of a personalised advantage index.

OBJECTIVE: To test the predictive accuracy and generalisability of a personalised advantage index (PAI) model designed to support treatment selection for Post-Traumatic Stress Disorder (PTSD). METHOD: A PAI model developed by Deisenhofer et al. (2018) was used to predict treatment outcomes in a statistically independent dataset including archival records for N = 152 patients with PSTD who accessed either trauma-focussed cognitive behavioural therapy or eye movement desensitisation and reprocessing in routine care. Outcomes were compared between patients who received their PAI-indicated optimal treatment versus those who received their suboptimal treatment. RESULTS: The model did not yield treatment specific predictions and patients who had received their PAI-indicated optimal treatment did not have better treatment outcomes in this external validation sample. CONCLUSION: This PAI model did not generalise to an external validation sample.

The Validation of the Japanese Version of the Scoliosis Research Society-30 Questionnaire for Adolescent Idiopathic Scoliosis Patients.

Introduction: The Scoliosis Research Society-30 (SRS-30) is a questionnaire originally developed from the SRS-22r questionnaire and is used to evaluate adolescent idiopathic scoliosis (AIS). It comprised questions on five domains: function, pain, self-image, mental health, and satisfaction, with seven additional questions related to postoperative aspects. In addition to the original English version, translations in multiple languages have been effectively applied. Herein, we evaluated the internal consistency and external validity of the Japanese version of the SRS-30 for AIS patients. Methods: Among the 30 questions in SRS-30, the eight additional questions from SRS-22r were translated and back-translated to create a Japanese version of the SRS-30. This translated questionnaire was then used to survey patients with AIS who underwent corrective fusion surgery one year postoperatively. The internal consistency of the responses was evaluated using the Cronbach α coefficient. Additionally, the Spearman correlation analyses were conducted to assess the correlation between the scores obtained from the SRS-30 Japanese version and SRS-22r and the Oswestry Disability Index (ODI) for the overall scale and the five domains. Results: A total of 81 cases (eight males and 73 females; mean age at surgery 14.4 years) were enrolled. The mean preoperative Cobb angle was 51.0°. The Cronbach α coefficient for the overall SRS-30 was 0.861, indicating high internal consistency, while the coefficients for each domain were as follows: function/activity, 0.697; pain, 0.405; self-image/appearance, 0.776; mental health, 0.845; and satisfaction, 0.559. The SRS-30 total score significantly correlated with the SRS-22r total (r=0.945, P<0.001) and the ODI (r=-0.511, P<0.001). The SRS-30 domains highly correlated with the corresponding SRS-22r domains, with correlations ranging from r=0.826 to 0.901 (all P<0.001). Conclusions: The Japanese version of the SRS-30 demonstrated good internal and external validity. The SRS-30 can be used as an assessment tool for health-related quality of life in AIS patients.

Racial, Ethnic, and Geographic Diversity in Population Neuroscience.

In this chapter, we consider lack of racial, ethnic, and geographic diversity in research studies from a public health perspective in which representation of a target population is critical. We review the state of the research field with respect to racial, ethnic, and geographic diversity in study participants. We next focus on key factors which can arise from the lack of diversity and can negatively impact external validity. Finally, we argue that the public's health, and future research, will ultimately be served by approaches from both recruitment and representation science and population neuroscience, and we close with recommendations from these two fields to improve diversity in studies.

Sample characteristics for quantitative analyses in Body Image: Issues of generalisability.

Psychological research frequently encounters criticism regarding the representativeness of the samples under study, highlighting concerns about the external validity of the obtained results. Here, we conducted a comprehensive survey of all the quantitative samples from the journal Body Image for 2021 (n = 149 samples). Our primary objective was to examine the extent to which the sampled populations deviated from the population at large, which could potentially compromise the generalizability of findings. We identified that a substantial number of these samples came from student populations (n = 44) and the majority were from the United States, United Kingdom, and Australia. Only a small number of samples (n = 9) employed direct measurements of body mass index (BMI), while the majority relied on self-reported data (n = 93). For a subset of samples in the journal, which were drawn from the general population, we compared whether these differed from population reference values in terms of age and BMI. Using Monte Carlo simulations, we found that samples tended to be younger and score lower on BMI than reference values obtained from the broader population. Samples drawn from female university students also tended to be lower on BMI than age-matched reference samples. We discuss the implications of our findings and make recommendations on sampling and inference. We conclude that a clearer specification of the parameters or conditions under which findings are expected to generalise has the potential to enhance the overall rigor and validity of this field of research.

The Myth of the Need for Diversity Among Subjects in Theory-Testing Research: Comments on "Racial Inequality in Psychological Research" by Roberts et al. (2020).

Roberts and colleagues focus on two aspects of racial inequality in psychological research, namely an alleged underrepresentation of racial minorities and the effects attributed to this state of affairs. My comment focuses only on one aspect, namely the assumed consequences of the lack of diversity in subject populations. Representativeness of samples is essential in survey research or applied research that examines whether a particular intervention will work for a particular population. Representativeness or diversity is not necessary in theory-testing research, where we attempt to establish laws of causality. Because theories typically apply to all of humanity, all members of humanity (even American undergraduates) are suitable for assessing the validity of theoretical hypotheses. Admittedly, the assumption that a theory applies to all of humanity is also a hypothesis that can be tested. However, to test it, we need theoretical hypotheses about specific moderating variables. Supporting a theory with a racially diverse sample does not make conclusions more valid than support from a nondiverse sample. In fact, cause-effect conclusions based on a diverse sample might not be valid for any member of that sample.

The Canadian Hypoglycemia During Hospitalization Score Is Externally Valid in the Australian Diabetes IN-hospital: Glucose & Outcomes (DINGO) Cohort.

OBJECTIVES: The Hypoglycemia During Hospitalization (HyDHo) score predicts hypoglycemia in a population of Canadian inpatients by assigning various weightings to 5 key clinical criteria known at the time of admission, in particular age, recent presentation to an emergency department, insulin use, use of oral hypoglycemic agents, and chronic kidney disease. Our aim in this study was to externally validate the HyDHo score by applying this risk calculator to an Australian population of inpatients with diabetes. METHODS: This study was a retrospective data analysis of a subset of the Diabetes IN-hospital: Glucose & Outcomes (DINGO) cohort. The HyDHo score was applied based on clinical information known at the time of admission to stratify risk of inpatient hypoglycemia. RESULTS: The HyDHo score was applied to 1,015 patients, generating a receiver-operating characteristic c-statistic of 0.607. A threshold of ≥9, as per the original study, generated a sensitivity of 83% and a specificity of 20%. A threshold of ≥10, to better suit this Australian population, generated a sensitivity of 90% and a specificity of 34%. The HyDHo score has been externally valid in a geographically different population; in fact, it outperformed the original study after accounting for local hypoglycemia rates. CONCLUSIONS: Our findings support the external validity of the HyDHo score in a geographically different population. Application of this simple and accessible tool can serve as an adjunct to predict an inpatient's risk of hypoglycemia and guide more appropriate glucose monitoring and diabetes management.

Variable selection when estimating effects in external target populations.

External validity is an important part of epidemiologic research. To validly estimate effects in specific external target populations using a chosen effect measure (ie, "transport"), some methods require that one account for all effect measure modifiers (EMMs). However, little is known about how including other variables that are not EMMs (ie, non-EMMs) in adjustment sets affects estimates. Using simulations, we evaluated how inclusion of non-EMMs affected estimation of the transported risk difference (RD) by assessing the impacts of covariates that (1) differ (or not) between the trial and the target, (2) are associated with the outcome (or not), and (3) modify the RD (or not). We assessed variation and bias when covariates with each possible combination of these factors were used to transport RDs using outcome modeling or inverse odds weighting. Inclusion of variables that differed in distribution between the populations but were non-EMMs reduced precision, regardless of whether they were associated with the outcome. However, non-EMMs associated with selection did not amplify bias resulting from omission of necessary EMMs. Including all variables associated with the outcome may result in unnecessarily imprecise estimates when estimating treatment effects in external target populations.

Using a Multi-Site RCT to Predict Impacts for a Single Site: Do Better Data and Methods Yield More Accurate Predictions?

Multi-site randomized controlled trials (RCTs) provide unbiased estimates of the average impact in the study sample. However, their ability to accurately predict the impact for individual sites outside the study sample, to inform local policy decisions, is largely unknown. To extend prior research on this question, we analyzed six multi-site RCTs and tested modern prediction methods-lasso regression and Bayesian Additive Regression Trees (BART)-using a wide range of moderator variables. The main study findings are that: (1) all of the methods yielded accurate impact predictions when the variation in impacts across sites was close to zero (as expected); (2) none of the methods yielded accurate impact predictions when the variation in impacts across sites was substantial; and (3) BART typically produced "less inaccurate" predictions than lasso regression or than the Sample Average Treatment Effect. These results raise concerns that when the impact of an intervention varies considerably across sites, statistical modelling using the data commonly collected by multi-site RCTs will be insufficient to explain the variation in impacts across sites and accurately predict impacts for individual sites.

Transporting results in an observational epidemiology setting: purposes, methods, and applied example.

In the medical domain, substantial effort has been invested in generating internally valid estimates in experimental as well as observational studies, but limited effort has been made in testing generalizability, or external validity. Testing the external validity of scientific findings is nevertheless crucial for the application of knowledge across populations. In particular, transporting estimates obtained from observational studies requires the combination of methods for causal inference and methods to transport the effect estimates in order to minimize biases inherent to observational studies and to account for differences between the study and target populations. In this paper, the conceptual framework and assumptions behind transporting results from a population-based study population to a target population is described in an observational setting. An applied example to life-course epidemiology, where internal validity was constructed for illustrative purposes, is shown by using the targeted maximum likelihood estimator.

How Mixed-Methods Research Can Improve the Policy Relevance of Impact Evaluations.

This paper describes how mixed methods can improve the value and policy relevance of impact evaluations, paying particular attention to how mixed methods can be used to address external validity and generalization issues. We briefly review the literature on the rationales for using mixed methods; provide documentation of the extent to which mixed methods have been used in impact evaluations in recent years; describe how we developed a list of recent impact evaluations using mixed methods and the process used to conduct full-text reviews of these articles; summarize the findings from our analysis of the articles; discuss three exemplars of using mixed methods in impact evaluations; and discuss how mixed methods have been used for studying and improving external validity and potential improvements that could be made in this area. We find that mixed methods are rarely used in impact evaluations, and we believe that increased use of mixed methods would be useful because they can reinforce findings from the quantitative analysis (triangulation), and they can also help us understand the mechanism by which programs have their impacts and the reasons why programs fail.

Standardizing to specific target populations in distributed networks and multisite pharmacoepidemiologic studies.

Distributed network studies and multisite studies assess drug safety and effectiveness in diverse populations by pooling information. Targeting groups of clinical or policy interest (including specific sites or site combinations) and applying weights based on effect measure modifiers (EMMs) prior to pooling estimates within multisite studies may increase interpretability and improve precision. We simulated a 4-site study, standardized each site using inverse odds weights (IOWs) to resemble the 3 smallest sites or the smallest site, estimated IOW-weighted risk differences (RDs), and combined estimates with inverse variance weights (IVWs). We also created an artificial distributed network in the Clinical Practice Research Datalink (CPRD) Aurum consisting of 1 site for each geographic region. We compared metformin and sulfonylurea initiators with respect to mortality, targeting the smallest region. In the simulation, IOWs reduced differences between estimates and increased precision when targeting the 3 smallest sites or the smallest site. In the CPRD Aurum study, the IOW + IVW estimate was also more precise (smallest region: RD = 5.41% [95% CI, 1.03-9.79]; IOW + IVW estimate: RD = 3.25% [95% CI, 3.07-3.43]). When performing pharmacoepidemiologic research in distributed networks or multisite studies in the presence of EMMs, designation of target populations has the potential to improve estimate precision and interpretability. This article is part of a Special Collection on Pharmacoepidemiology.

Binocular rivalry under naturalistic geometry: Evidence from worlds simulated in virtual reality.

Binocular rivalry is a fascinating, widely studied visual phenomenon in which perception alternates between two competing images. This experience, however, is generally restricted to laboratory settings where two irreconcilable images are presented separately to the two eyes, an implausible geometry where two objects occupy the same physical location. Such laboratory experiences are in stark contrast to everyday visual behavior, where rivalry is almost never encountered, casting doubt on whether rivalry is relevant to our understanding of everyday binocular vision. To investigate the external validity of binocular rivalry, we manipulated the geometric plausibility of rival images using a naturalistic, cue-rich, 3D-corridor model created in virtual reality. Rival stimuli were presented in geometrically implausible, semi-plausible, or plausible layouts. Participants tracked rivalry fluctuations in each of these three layouts and for both static and moving rival stimuli. Results revealed significant and canonical binocular rivalry alternations regardless of geometrical plausibility and stimulus type. Rivalry occurred for layouts that mirrored the unnatural geometry used in laboratory studies and for layouts that mimicked real-world occlusion geometry. In a complementary 3D modeling analysis, we show that interocular conflict caused by geometrically plausible occlusion is a common outcome in a visual scene containing multiple objects. Together, our findings demonstrate that binocular rivalry can reliably occur for both geometrically implausible interocular conflicts and conflicts caused by a common form of naturalistic occlusion. Thus, key features of binocular rivalry are not simply laboratory artifacts but generalize to conditions that match the geometry of everyday binocular vision.

Generalizability of predictive models for Clostridioides difficile infection, severity and recurrence at an urban safety-net hospital.

INTRODUCTION: Predictive models for Clostridioides difficile infection can identify high-risk patients and aid clinicians in preventing infection. Issues of generalizability regarding current predictive models have been acknowledged but, to the authors' knowledge, have never been quantified. METHODS: C. difficile infection, severity and recurrence predictive models were created using multi-variate logistic regression through case-control sampling from an urban safety-net hospital. Models were validated using five-fold cross-validation, and inverse probability weights (IPW) based on two different catchment area definitions were used to improve external validity. Akaike Information Criterion (AIC), area under the receiver operating characteristic curve (AUROC), and sensitivity and specificity with bootstrapped confidence intervals (CI) were used to assess and compare model fit and performance. RESULTS: Changes in performance before and after weighting were small across all models, although differences were more apparent after weighting the recurrence model (AUROC values of 0.78, 0.76 and 0.71 for the unweighted and two weighted models, respectively). Overall, the infection model performed the best (AUROC 0.82, 95% CI 0.78-0.85), followed by the recurrence model (AUROC 0.78, 95% CI 0.69-0.86) and then the severity model (AUROC 0.70, 95% CI 0.63-0.78). CONCLUSIONS: The performance of the models after weighting did not change drastically, suggesting that the models predicting C. difficile infection, severity and recurrence may not be impacted by patient selection factors. However, other researchers may wish to consider addressing these catchment forces using IPW.

Comparing clinical trial population representativeness to real-world users of 17 biologics approved for immune-mediated inflammatory diseases: An external validity analysis of 66,639 biologic users from the Italian VALORE project.

To date, no population-based studies have specifically explored the external validity of pivotal randomized clinical trials (RCTs) of biologics simultaneously for a broad spectrum of immuno-mediated inflammatory diseases (IMIDs). The aims of this study were, firstly, to compare the patients' characteristics and median treatment duration of biologics approved for IMIDs between RCTs' and real-world setting (RW); secondly, to assess the extent of biologic users treated for IMIDs in the real-world setting that would not have been eligible for inclusion into pivotal RCT for each indication of use. Using the Italian VALORE distributed database (66,639 incident biologic users), adult patients with IMIDs treated with biologics in the Italian real-world setting were substantially older (mean age ± SD: 50 ± 15 years) compared to those enrolled in pivotal RCTs (45 ± 15 years). In the real-world setting, certolizumab pegol was more commonly used by adult women with psoriasis/ankylosing spondylitis (F/M ratio: 1.8-1.9) compared to RCTs (F/M ratio: 0.5-0.6). The median treatment duration (weeks) of incident biologic users in RW was significantly higher than the duration of pivotal RCTs in almost all indications for use and most biologics (4-100 vs. 6-167). Furthermore, almost half (46.4%) of biologic users from RW settings would have been ineligible for inclusion in the respective indication-specific pivotal RCTs. The main reasons were: advanced age, recent history of cancer and presence of other concomitant IMIDs. These findings suggest that post-marketing surveillance of biologics should be prioritized for those patients.

The Logic of Generalization From Systematic Reviews and Meta-Analyses of Impact Evaluations.

Systematic reviews and meta-analyses are viewed as potent tools for generalized causal inference. These reviews are routinely used to inform decision makers about expected effects of interventions. However, the logic of generalization from research reviews to diverse policy and practice contexts is not well developed. Building on sampling theory, concerns about epistemic uncertainty, and principles of generalized causal inference, this article presents a pragmatic approach to generalizability assessment for use with systematic reviews and meta-analyses. This approach is applied to two systematic reviews and meta-analyses of effects of "evidence-based" psychosocial interventions for youth and families. Evaluations included in systematic reviews are not necessarily representative of populations and treatments of interest. Generalizability of results is limited by high risks of bias, uncertain estimates, and insufficient descriptive data from impact evaluations. Systematic reviews and meta-analyses can be used to test generalizability claims, explore heterogeneity, and identify potential moderators of effects. These reviews can also produce pooled estimates that are not representative of any larger sets of studies, programs, or people. Further work is needed to improve the conduct and reporting of impact evaluations and systematic reviews, and to develop practical approaches to generalizability assessment and guide applications of interventions in diverse policy and practice contexts.