Eligibility for Substance Use Clinical Trials Among Emergency Psychiatry Patients: The Impact of Exclusion Criteria.

Objective: The first objective was to identify common exclusion criteria used in clinical trials. The second objective was to quantify the degree to which these criteria exclude emergency psychiatry patients. Methods: Qualitative Content Analysis was used for the first objective, identifying common exclusion criteria used in recent high-impact substance use clinical trials. A retrospective record review was used for the second objective, which examined the frequency of these exclusion criteria in a 1-month sample of adults receiving psychiatric evaluation in an emergency department. Results: Most trials had exclusions for co-occurring psychiatric problems (76.6%), medical problems (74.0%), prior or current treatment (72.7%), motivation for change (61.1%), pregnancy or lactation (57.1%), or using other specified substances of abuse (54.6%). In the clinical sample, exclusions for co-occurring psychiatric problems would make 94.7% of patients ineligible. Other exclusions had a combined effect of making 76% of patients ineligible. Conclusions: Clinical trials using typical exclusion criteria exclude nearly all emergency psychiatry patients with substance use problems.

Validity and Characterization of Time to Symptom Resolution Outcome Measures in the ACTIV-2/A5401 Outpatient COVID-19 Treatment Trial.

BACKGROUND: Time to symptom resolution measures were used in outpatient coronavirus disease 2019 (COVID-19) treatment trials without prior validation. METHODS: ACTIV-2/A5401 trial participants completed a COVID-19 diary assessing 13 targeted symptoms and global experience (overall COVID-19 symptoms, return to pre-COVID-19 health) daily for 29 days. We evaluated concordance of time to sustained (2 days) resolution of all targeted symptoms (TSR) with resolution of overall symptoms and return to health in participants receiving placebo. RESULTS: The analysis included 77 high-risk and 81 standard-risk participants with overall median 6 days of symptoms at entry and median age 47 years, 50% female, 82% white, and 31% Hispanic/Latino. Correlation between TSR and resolution of overall symptoms was 0.80 and 0.68, and TSR and return to health, 0.66 and 0.57 for high- and standard-risk groups, respectively. Of the high- and standard-risk participants, 61% and 79%, respectively, achieved targeted symptom resolution, of which 47% and 43%, respectively, reported symptom recurrence. Requiring >2 days to define sustained resolution reduced the frequency of recurrences. CONCLUSIONS: There was good internal consistency between TSR and COVID-19-specific global outcomes, supporting TSR as a trial end point. Requiring >2 days of symptom resolution better addresses natural symptom fluctuations but must be balanced against the potential influence of non-COVID-19 symptoms. CLINICAL TRIALS REGISTRATION: NCT04518410.

Keeping track of reality: embedding visual memory in natural behaviour.

Since immersive virtual reality (IVR) emerged as a research method in the 1980s, the focus has been on the similarities between IVR and actual reality. In this vein, it has been suggested that IVR methodology might fill the gap between laboratory studies and real life. IVR allows for high internal validity (i.e., a high degree of experimental control and experimental replicability), as well as high external validity by letting participants engage with the environment in an almost natural manner. Despite internal validity being crucial to experimental designs, external validity also matters in terms of the generalizability of results. In this paper, we first highlight and summarise the similarities and differences between IVR, desktop situations (both non-immersive VR and computer experiments), and reality. In the second step, we propose that IVR is a promising tool for visual memory research in terms of investigating the representation of visual information embedded in natural behaviour. We encourage researchers to carry out experiments on both two-dimensional computer screens and in immersive virtual environments to investigate visual memory and validate and replicate the findings. IVR is valuable because of its potential to improve theoretical understanding and increase the psychological relevance of the findings.

The Sensitivity and Specificity of Repeated and Dominant Choice Tasks in Discrete Choice Experiments.

OBJECTIVES: This study aimed to identify the most commonly used internal validity tests in the discrete choice experiment (DCE) literature and establish their sensitivity and specificity. METHODS: A structured literature review of recent DCE articles (2018-2020Q1) published in the health, marketing, transport economics, and environmental science literature was used to identify commonly used internal validity tests. The 2 most frequently used internal validity tests were incorporated in 4 new data collections. Respondent preferences in each application were summarized using a mixed logit model, which served as the benchmark for the subsequent sensitivity and specificity calculations. The performance of the internal validity tests was also compared with that of the root likelihood (RLH) test, which is a likelihood-based statistical validity test that is commonly used in marketing applications. RESULTS: Dominant and repeated choice tasks were most commonly included in health-related DCE designs. Based on 4 applications, their specificity and sensitivity depend on the type of incorrect response pattern to be detected and on design characteristics such as the number of choice options per choice task and the number of internal validity tests as included in the experimental design. In all but one scenario, the performance of the dominant and repeated choice tasks was considerably worse than that of the RLH test. CONCLUSIONS: Dominant and repeated choice tasks are unreliable screening tests and costly in terms of statistical power. The RLH test, which is a statistical test that does not require additional choice tasks to be included in the DCE design, provides a more reliable alternative.

Measurement properties of the Swedish clinical outcomes in routine evaluation outcome measures (CORE-OM): Rasch analysis and short version for depressed and anxious out-patients in a multicultural area.

INTRODUCTION: The Swedish version of the patient-reported Clinical Outcomes in Routine Evaluation Outcome Measures (CORE-OM) has demonstrated high reliability and acceptable convergent validity in explanatory factor analyses. However, the fundamental scale properties have not yet been validated according to item response theory. The aim of this study was to analyze the measurement properties of the Swedish CORE-OM in a cohort of psychiatric out-patients with depression and anxiety in a multicultural area and to explore combinations of items based on shorter versions of the scale (CORE-10, CORE-6D) to improve measurement properties. METHODS: Data from CORE-OM assessments of 337 patients were analyzed using Rasch analysis. The patients had a mean age of 30 ± 14 years, the majority were women (72%). Requirements for measurement properties were checked: overall model fit, item fit residuals, targeting, internal consistency, differential item functioning and thresholds. Sensitivity to change was also analyzed. RESULTS: The CORE-OM showed high internal consistency (person separation index = 0.947) and adequate targeting, but there was overall model misfit (item trait interaction χ2 = 917.53, p < 0.001), indication of local dependency, and differential item functioning in 9 items. The risk items showed problems with disordered thresholds. The emotional component of the shorter CORE-6D showed the best fit for our sample. Adding 3 items to include depressive and trauma-related content resulted in a unidimensional 8-item set with acceptable reliability, model fit, targeting and sensitivity to change. CONCLUSION: For out-patients with diagnosed depression or anxiety in a multicultural area, the Swedish CORE-OM showed high internal consistency, but also validity problems. Based on the shorter CORE-6D version, a unidimensional 8-item set could be an alternative brief measure of psychological distress for this population, but further validity studies are required. Qualitative studies exploring the CORE-OM items in non-native speakers are also warranted.

Bias in Hand Surgical Randomized Controlled Trials: Systematic Review and Meta-Epidemiological Study.

PURPOSE: Inappropriately reported or conducted studies may decrease the quality of care due to under- or overestimation of the benefits or harms of interventions. Our aim was to evaluate how often hand surgical randomized controlled trials (RCTs) use and report adequate methods to ensure internal validity, and whether inadequate reporting or methods are associated with the magnitude of treatment effect estimates. METHODS: Data Sources were the Cochrane Central Register of Controlled Trials, MEDLINE, and Embase databases until November 2020. We included published RCTs investigating the effects of any surgical intervention in the hand and wrist region. We assessed internal validity using the Cochrane Risk of Bias (RoB) tool for 6 domains: selection, performance, detection, attrition, selective reporting, and "other" bias. We extracted the primary outcome and calculated the effect size for each study. We used mixed-effect meta-regression to assess whether the RoB modified the magnitude of the effects. RESULTS: For 207 assessed trials, the RoB was unclear or high for 72% in selection, 93% in performance, 88% in detection, 25% in attrition, 22% in selective reporting, and 34% in the "other" bias domain. Trials with a high or unclear risk of selection bias yielded 0.28 standardized mean difference (95% confidence interval, 0.02-0.55) larger effect sizes compared to studies with a low risk. Risks of bias for other domains did not modify the intervention effects. The risk for selection bias declined over time: the odds ratio for a high or unclear RoB was 0.90 (95% confidence interval, 0.85-0.95) per additional year of publication CONCLUSIONS: The internal validity and credibility of hand surgical RCTs can be improved by using established methods to achieve true randomization, blinding of the participants and study personnel, publishing the trial protocol and avoiding selective reporting of the outcomes, and reporting the trial as recommended in the Consolidated Standards of Reporting Trials statement. CLINICAL RELEVANCE: Clinicians should be aware that RCTs that do not use or report proper randomization and allocation concealment may overestimate the treatment effects.

How Within-Study Comparisons can Inform Research and Evaluation Design for the Third Sector.

Within-study comparisons (WSCs) compare quasi-experimental results to an experimental benchmark and assess the extent to which quasi-experiments (QEs) sacrifice internal validity for external validity. WSCs inform decisions about when to use experiments or QEs, as well as methodological decisions (e.g., covariate selection) about how to implement QEs when they are necessary or preferred. We review the methodological literature on WSCs and present the findings from a WSC of a school voucher program as an example of the use of WSCs to inform policy debates involving the third sector. Prior literature and our results suggest that QE evaluations of third-sector institutions can produce low levels of bias, but only if analytic models include key proxy variables for voluntary self-selection into programs.

Synthetic Control Methods for the Evaluation of Single-Unit Interventions in Epidemiology: A Tutorial.

Evaluating the impacts of population-level interventions (e.g., changes to state legislation) can be challenging as conducting randomized experiments is often impractical and inappropriate, especially in settings where the intervention is implemented in a single, aggregate unit (e.g., a country or state). A common nonrandomized alternative is to compare outcomes in the treated unit(s) with unexposed controls both before and after the intervention. However, the validity of these designs depends on the use of controls that closely resemble the treated unit on before-intervention characteristics and trends on the outcome, and suitable controls may be difficult to find because the number of potential control regions is typically limited. The synthetic control method provides a potential solution to these problems by using a data-driven algorithm to identify an optimal weighted control unit-a "synthetic control"-based on data from before the intervention from available control units. While popular in the social sciences, the method has not garnered as much attention in health research, perhaps due to a lack of accessible texts aimed at health researchers. We address this gap by providing a comprehensive, nontechnical tutorial on the synthetic control method, using a worked example evaluating Florida's "stand your ground" law to illustrate methodological and practical considerations.

Common Methodological Problems in Randomized Controlled Trials of Preventive Interventions.

Randomized controlled trials (RCTs) are often considered the gold standard in evaluating whether intervention results are in line with causal claims of beneficial effects. However, given that poor design and incorrect analysis may lead to biased outcomes, simply employing an RCT is not enough to say an intervention "works." This paper applies a subset of the Society for Prevention Research (SPR) Standards of Evidence for Efficacy, Effectiveness, and Scale-up Research, with a focus on internal validity (making causal inferences) to determine the degree to which RCTs of preventive interventions are well-designed and analyzed, and whether authors provide a clear description of the methods used to report their study findings. We conducted a descriptive analysis of 851 RCTs published from 2010 to 2020 and reviewed by the Blueprints for Healthy Youth Development web-based registry of scientifically proven and scalable interventions. We used Blueprints' evaluation criteria that correspond to a subset of SPR's standards of evidence. Only 22% of the sample satisfied important criteria for minimizing biases that threaten internal validity. Overall, we identified an average of 1-2 methodological weaknesses per RCT. The most frequent sources of bias were problems related to baseline non-equivalence (i.e., differences between conditions at randomization) or differential attrition (i.e., differences between completers versus attritors or differences between study conditions that may compromise the randomization). Additionally, over half the sample (51%) had missing or incomplete tests to rule out these potential sources of bias. Most preventive intervention RCTs need improvement in rigor to permit causal inference claims that an intervention is effective. Researchers also must improve reporting of methods and results to fully assess methodological quality. These advancements will increase the usefulness of preventive interventions by ensuring the credibility and usability of RCT findings.

Third time's the charm or three strikes you're out? An updated review of the efficacy of dolphin-assisted therapy for autism and developmental disabilities.

CONTEXT: Dolphin-assisted therapy (DAT) is a popular form of animal-assisted therapy for autism spectrum disorders and other psychological conditions. OBJECTIVE: In this review, our third, we analyze the most recent DAT studies in terms of construct and internal validity criteria to determine if there is empirical support for DAT. METHOD: To ensure a systematic review, we searched for peer-reviewed studies on DAT by submitting relevant search terms to Google Scholar from 2007 to 2020, conducted a further search of all DAT papers in several peer-reviewed journals, and reviewed reference sections of DAT articles to ensure a thorough review of the literature between 2007 and the present. RESULTS: The DAT literature continues to be marked by several weaknesses in both internal and construct validity that preclude confident inferences regarding the intervention's efficacy. CONCLUSION: There is still insufficient evidence that DAT has therapeutic value.

A Validity-Based Framework for Understanding Replication in Psychology.

In recent years, psychology has wrestled with the broader implications of disappointing rates of replication of previously demonstrated effects. This article proposes that many aspects of this pattern of results can be understood within the classic framework of four proposed forms of validity: statistical conclusion validity, internal validity, construct validity, and external validity. The article explains the conceptual logic for how differences in each type of validity across an original study and a subsequent replication attempt can lead to replication "failure." Existing themes in the replication literature related to each type of validity are also highlighted. Furthermore, empirical evidence is considered for the role of each type of validity in non-replication. The article concludes with a discussion of broader implications of this classic validity framework for improving replication rates in psychological research.

Using Bounds to Compare the Strength of Exchangeability Assumptions for Internal and External Validity.

In the absence of strong assumptions (e.g., exchangeability), only bounds for causal effects can be identified. Here we describe bounds for the risk difference for an effect of a binary exposure on a binary outcome in 4 common study settings: observational studies and randomized studies, each with and without simple random selection from the target population. Through these scenarios, we introduce randomizations for selection and treatment, and the widths of the bounds are narrowed from 2 (the width of the range of the risk difference) to 0 (point identification). We then assess the strength of the assumptions of exchangeability for internal and external validity by comparing their contributions to the widths of the bounds in the setting of an observational study without random selection from the target population. We find that when less than two-thirds of the target population is selected into the study, the assumption of exchangeability for external validity of the risk difference is stronger than that for internal validity. The relative strength of these assumptions should be considered when designing, analyzing, and interpreting observational studies and will aid in determining the best methods for estimating the causal effects of interest.

Software for designing rigorous and replicable preclinical research: The Experimental Design Accelerator.

In recent years, there have been concerns about research practices in basic and preclinical biomedical research. There have been problems with non-replicable results, and experimental designs lacking internal validity or external or translational validity. The Experimental Design Accelerator (XDA) is Internet-based, interactive software designed to help those trying to design, conduct and document rigorous, replicable and relevant experiments. It leads the investigator step-by-step through a series of decisions that will define the experimental design. It provides background regarding the significance of each decision and the advantages and disadvantages of each possible choice. For example, it leads the researcher to address issues such as choosing a research model, developing testable hypotheses, identifying extraneous variables, dealing with random and systematic error, picking appropriate sample size and picking appropriate statistical analyses. There are also sections to help conduct the experiment consistent with its design and to document the study to facilitate accurate replication. Helpful features include access to an online statistics book and provisions for rapid contact with consulting experts. A number of potential uses for such novel interactive software tools will be discussed.

Making Health Research Matter: A Call to Increase Attention to External Validity.

Most of the clinical research conducted with the goal of improving health is not generalizable to nonresearch settings. In addition, scientists often fail to replicate each other's findings due, in part, to lack of attention to contextual factors accounting for their relative effectiveness or failure. To address these problems, we review the literature on assessment of external validity and summarize approaches to designing for generalizability. When investigators conduct systematic reviews, a critical need is often unmet: to evaluate the pragmatism and context of interventions, as well as their effectiveness. Researchers, editors, and grant reviewers can implement key changes in how they consider and report on external validity issues. For example, the recently published expanded CONSORT figure may aid scientists and potential program adopters in summarizing participation in and representativeness of a program across different settings, staff, and patients. Greater attention to external validity is needed to increase reporting transparency, improve program dissemination, and reduce failures to replicate research.

Rural physical activity interventions in the United States: a systematic review and RE-AIM evaluation.

BACKGROUND: Previous reviews of rural physical activity interventions were focused on intervention effectiveness and had reported overall mixed findings. The purpose of this systematic review was to apply the Reach, Efficacy, Adoption, Implementation and Maintenance (RE-AIM) framework to evaluate the extent to which rural physical activity interventions in the U.S. have reported on dimensions of internal and external validity and to offer suggestions for future physical activity interventions for rural U.S. METHODS: Pubmed, PsychINFO, CINAHL, PAIS, and Web of Science were searched through February 2019 to identify physical activity intervention studies conducted in rural regions in the U.S. with adult populations. Titles, abstracts, and full texts of articles were reviewed against inclusion and exclusion criteria. Data extraction from included articles included a summary of study details, rural classification system used, and the presence or absence of a total 61 RE-AIM indicators, including reach (n = 13), efficacy/effectiveness (n = 10), adoption (n = 21), implementation (n = 9), and maintenance (n = 8). RESULTS: A total of 40 full-text articles representing 29 unique studies were included. Classifications of rurality included self-statements by authors (n = 19, 65.5%), population/census-based definitions (n = 3, 10.3%), Rural Urban Continuum Codes (n = 3, 10.3%), Rural Urban Commuting Area codes (n = 2, 6.9%), the 2014 Alabama Rural Health Association classification system (n = 1, 3.4%) and the U.S. Office of Management and Budget classification system (n = 1, 3.4%). Individual studies reported between 14.8 to 52.5% of total RE-AIM indicators. Studies reported 15.4 to 84.6% indicators for reach; 20.0 to 70.0% indicators for efficacy/effectiveness; 4.8 to 47.6% indicators for adoption; 11.1 to 88.9% indicators for implementation; and 0 to 25.0% indicators for maintenance. CONCLUSIONS: We found an overall poor reporting of components related to external validity, which hinders the generalizability of intervention findings, and a lack of consistency in the definition of rurality. Future research should focus on balancing factors of internal and external validity, and should aim to develop a greater understanding of how rurality influences health and behavior to provide contextual knowledge needed to advance the translation of physical activity interventions into practice in rural communities and reduce rural health disparities. TRIAL REGISTRATION: The review protocol was registered with PROSPERO: CRD42019116308.

Evidence-based medicine for every day, everyone, and every therapeutic study.

The rapid growth in published medical literature makes it difficult for clinicians to keep up with advances in their fields. This may result in a cursory scan of the abstract and conclusion of a study without critically evaluating study quality. The application of evidence-based medicine (EBM) is the process of converting the abstract task of reading the literature into a practical method of using the literature to inform care in a specific clinical context while simultaneously expanding one's knowledge. EBM involves 4 steps: (1) stating the clinical problem in a defined question; (2) searching the literature for the evidence; (3) critically appraising the evidence for its validity; and (4) applying the evidence in the context of the patient's situation, preferences, and values. In this review, we use the recently published trial of thymectomy in myasthenia gravis as an example and systematically go through the steps of assessing internal validity, precision, and external validity. Muscle Nerve 58: 486-496, 2018.

Assessing Rationality in Discrete Choice Experiments in Health: An Investigation into the Use of Dominance Tests.

BACKGROUND: Dominance tests are often applied to test for the rationality in the choice behavior of participants in discrete choice experiments (DCEs). OBJECTIVES: To examine how dominance tests have been implemented in recent DCE applications in health and discuss their theoretical and empirical interpretation. METHODS: Health-related DCEs published in 2015 were reviewed for the inclusion of tests on choice behavior. For studies that implemented a dominance test, information on application and interpretation of the test was extracted. Authors were contacted for test choice sets and observed proportions of subjects who chose the dominated option. Coefficients corresponding to the choice set were extracted to estimate the expected probability of choosing the dominated option with a logistic model and compared with the observed proportion. The theoretical range of expected probabilities of possible dominance tests was calculated. RESULTS: Of 112 health-related DCEs, 49% included at least one test for choice behavior; 28 studies (25%) included a dominance test. The proportion of subjects in each study who chose the dominated option ranged from 0% to 21%. In 46% of the studies, the dominance test led to the exclusion of participants. In the 15 choice sets that were analyzed, 2 had larger proportions of participants choosing the dominated option than expected (P < 0.05). CONCLUSIONS: Although dominance tests are frequently applied in DCEs, there is no consensus on how to account for them in data analysis and interpretation. Comparison of expected and observed proportions of participants failing the test might be indicative of DCE quality.

Pragmatism in practice: lessons learned during screening and enrollment for a randomised controlled trial in rural northern Ethiopia.

BACKGROUND: We use the example of the Gojjam Lymphoedema Best Practice Trial (GoLBeT), a pragmatic trial in a remote rural setting in northern Ethiopia, to extract lessons relevant to other investigators balancing the demands of practicality and community acceptability with internal and external validity in clinical trials. METHODS: We explain in detail the preparation for the trial, its setting in northern Ethiopia, the identification and selection of patients (inclusion and exclusion criterion, identifying and screening of patients at home, enrollment of patients at the health centres and health posts), and randomisation. RESULTS: We describe the challenges met, together with strategies employed to overcome them. CONCLUSIONS: Examples given in the previous section are contextualised and general principles extracted where possible. We conclude that it is possible to conduct a trial that balances approaches that support internal validity (e.g. careful design of proformas, accurate case identification, control over data quality and high retention rates) with those that favour generalisability (e.g. 'real world' setting and low rates of exclusion). Strategies, such as Rapid Ethical Assessment, that increase researchers' understanding of the study setting and inclusion of hard-to-reach participants are likely to have resource and time implications, but are vital in achieving an appropriate balance. TRIAL REGISTRATION: ISRCTN67805210, registered 24/01/2013.

Role of Risk of Bias in Systematic Review for Chemical Risk Assessment: A Case Study in Understanding the Relationship Between Congenital Heart Defects and Exposures to Trichloroethylene.

The National Academy of Science has recommended that a risk of bias (RoB; credibility of the link between exposure and outcome) assessment be conducted on studies that are used as primary data sources for hazard identification and dose-response assessment. Few applications of such have been conducted. Using trichloroethylene and congenital heart defects (CHDs) as a case study, we explore the role of RoB in chemical risk assessment using the National Toxicology Program's Office of Health Assessment and Translation RoB tool. Selected questions were tailored to evaluation of CHD and then applied to 12 experimental animal studies and 9 epidemiological studies. Results demonstrated that the inconsistent findings of a single animal study were likely explained by the limitations in study design assessed via RoB (eg, lack of concurrent controls, unvalidated method for assessing outcome, unreliable statistical methods, etc). Such limitations considered in the context of the body of evidence render the study not sufficiently reliable for the development of toxicity reference values. The case study highlights the utility of RoB as part of a robust risk assessment process and specifically demonstrates the role RoB can play in objectively selecting candidate data sets to develop toxicity values.