Reweighting estimators to extend the external validity of clinical trials: methodological considerations.

Methods to extend the strong internal validity of randomized controlled trials to reliably estimate treatment effects in target populations are gaining attention. This paper enumerates steps recommended for undertaking such extended inference, discusses currently viable choices for each one, and provides recommendations. We demonstrate a complete extended inference from a clinical trial studying a pharmaceutical treatment for Alzheimer's disease (AD) to a realistic target population of European residents diagnosed with AD. This case study highlights approaches to overcoming practical difficulties and demonstrates limitations of reliably extending inference from a trial to a real-world population.

The Impact of Preinjury and Secondary Attention-Deficit/Hyperactivity Disorder on Outcomes After Pediatric Traumatic Brain Injury.

OBJECTIVE: The objective of this study was to examine the impact of preinjury attention-deficit/hyperactivity disorder (PADHD) and secondary ADHD (SADHD) on outcomes after pediatric traumatic brain injury (TBI). METHODS: Two hundred eighty-four individuals aged 11 to 18 years hospitalized overnight for a moderate-to-severe TBI were included in this study. Parents completed measures of child behavior and functioning and their own functioning. Linear models examined the effect of ADHD status (PADHD vs SADHD vs no ADHD) on the child's executive functioning (EF), social competence, and functional impairment, and parental depression and distress. RESULTS: ADHD status had a significant effect on EF [F(2,269] = 9.19, p = 0.0001), social competence (F[2,263] = 32.28, p < 0.0001), functional impairment (F[2,269] = 16.82, p < 0.0001), parental depression (F[2,263] = 5.53, p = 0.005), and parental distress (F[2,259] = 3.57, p = 0.03). PADHD and SADHD groups had greater EF deficits, poorer social competence, and greater functional impairment than the no ADHD group. The SADHD group had greater levels of parental depression than the no ADHD and PADHD groups, and the SADHD group had higher parental distress than the no ADHD group. CONCLUSION: The results highlight the importance of early identification and management of ADHD symptoms after injury to mitigate downstream functional problems. Supporting parents managing new-onset ADHD symptoms may also be important.

Exploring the direct and indirect effects of elite influence on public opinion.

Political elites both respond to public opinion and influence it. Elite policy messages can shape individual policy attitudes, but the extent to which they do is difficult to measure in a dynamic information environment. Furthermore, policy messages are not absorbed in isolation, but spread through the social networks in which individuals are embedded, and their effects must be evaluated in light of how they spread across social environments. Using a sample of 358 participants across thirty student organizations at a large Midwestern research university, we experimentally investigate how real social groups consume and share elite information when evaluating a relatively unfamiliar policy area. We find a significant, direct effect of elite policy messages on individuals' policy attitudes. However, we find no evidence that policy attitudes are impacted indirectly by elite messages filtered through individuals' social networks. Results illustrate the power of elite influence over public opinion.

Understanding the Relationship Between Official and Social Information About Infectious Disease: Experimental Analysis.

BACKGROUND: Communicating official public health information about infectious diseases is complicated by the fact that individuals receive much of their information from their social contacts, either via interpersonal interaction or social media, which can be prone to bias and misconception. OBJECTIVE: This study aims to evaluate the effect of public health campaigns and the effect of socially communicated health information on learning about diseases simultaneously. Although extant literature addresses the effect of one source of information (official or social) or the other, it has not addressed the simultaneous interaction of official information (OI) and social information (SI) in an experimental setting. METHODS: We used a series of experiments that exposed participants to both OI and structured SI about the symptoms and spread of hepatitis C over a series of 10 rounds of computer-based interactions. Participants were randomly assigned to receive a high, low, or control intensity of OI and to receive accurate or inaccurate SI about the disease. RESULTS: A total of 195 participants consented to participate in the study. Of these respondents, 186 had complete responses across all ten experimental rounds, which corresponds to a 4.6% (9/195) nonresponse rate. The OI high intensity treatment increases learning over the control condition for all symptom and contagion questions when individuals have lower levels of baseline knowledge (all P values ≤.04). The accurate SI condition increased learning across experimental rounds over the inaccurate condition (all P values ≤.01). We find limited evidence of an interaction between official and SI about infectious diseases. CONCLUSIONS: This project demonstrates that exposure to official public health information increases individuals' knowledge of the spread and symptoms of a disease. Socially shared information also facilitates the learning of accurate and inaccurate information, though to a lesser extent than exposure to OI. Although the effect of OI persists, preliminary results suggest that it can be degraded by persistent contradictory SI over time.

Bayesian Generalized Linear Mixed-Model Analysis of Language Samples: Detecting Patterns in Expository and Narrative Discourse of Adolescents With Traumatic Brain Injury.

Purpose Generalized linear mixed-model (GLMM) and Bayesian methods together provide a framework capable of handling a wide variety of complex data commonly encountered across the communication sciences. Using language sample analysis, we demonstrate the utility of these methods in answering specific questions regarding the differences between discourse patterns of children who have experienced a traumatic brain injury (TBI), as compared to those with typical development. Method Language samples were collected from 55 adolescents ages 13-18 years, five of whom had experienced a TBI. We describe parameters relating to the productivity, syntactic complexity, and lexical diversity of language samples. A Bayesian GLMM is developed for each parameter of interest, relating these parameters to age, sex, prior history (TBI or typical development), and socioeconomic status, as well as the type of discourse sample (compare-contrast, cause-effect, or narrative). Statistical models are thoroughly described. Results Comparing the discourse of adolescents with TBI to those with typical development, substantial differences are detected in productivity and lexical diversity, while differences in syntactic complexity are more moderate. Female adolescents exhibited greater syntactic complexity, while male adolescents exhibited greater productivity and lexical diversity. Generally, our models suggest more advanced discourse among adolescents who are older or who have indicators of higher socioeconomic status. Differences relating to lecture type were also detected. Conclusions Bayesian and GLMM methods yield more informative and intuitive results than traditional statistical analyses, with a greater degree of confidence in model assumptions. We recommend that these methods be used more widely in language sample analysis. Supplemental Material https://doi.org/10.23641/asha.14226959.

A comparison of reweighting estimators of average treatment effects in real world populations.

Regulatory agencies typically evaluate the efficacy and safety of new interventions and grant commercial approval based on randomized controlled trials (RCTs). Other major healthcare stakeholders, such as insurance companies and health technology assessment agencies, while basing initial access and reimbursement decisions on RCT results, are also keenly interested in whether results observed in idealized trial settings will translate into comparable outcomes in real world settings-that is, into so-called "real world" effectiveness. Unfortunately, evidence of real world effectiveness for new interventions is not available at the time of initial approval. To bridge this gap, statistical methods are available to extend the estimated treatment effect observed in a RCT to a target population. The generalization is done by weighting the subjects who participated in a RCT so that the weighted trial population resembles a target population. We evaluate a variety of alternative estimation and weight construction procedures using both simulations and a real world data example using two clinical trials of an investigational intervention for Alzheimer's disease. Our results suggest an optimal approach to estimation depends on the characteristics of source and target populations, including degree of selection bias and treatment effect heterogeneity.

Behavior Problems Following Childhood TBI: The Role of Sex, Age, and Time Since Injury.

OBJECTIVE: To examine the frequency of behavioral problems after childhood traumatic brain injury (TBI) and their associations with injury severity, sex, and social environmental factors. SETTING: Children's hospitals in the Midwestern/Western United States. PARTICIPANTS: 381 boys and 210 girls with moderate (n = 359) and severe (n = 227) TBI, with an average age at injury of 11.7 years (range 0.3-18) who were injured ≤3 years ago. DESIGN: Secondary data analysis of a multistudy cohort. MAIN MEASURES: Child Behavior Checklist (CBCL) administered pretreatment. RESULTS: Thirty-seven percent had borderline/clinical elevations on the CBCL Total Problem Scale, with comparable rates of Internalizing and Externalizing problems (33% and 31%, respectively). Less parental education was associated with higher rates of internalizing, externalizing, and total problems. Time since injury had a linear association with internalizing symptoms, with greater symptoms at longer postinjury intervals. Younger boys had significantly higher levels of oppositional defiant symptoms than girls, whereas older girls had significantly greater attention-deficit hyperactivity disorder symptoms than boys. CONCLUSIONS: Pediatric TBI is associated with high rates of behavior problems, with lower socioeconomic status predicting substantially elevated risk. Associations of higher levels of internalizing symptoms with greater time since injury highlight the importance of tracking children over time.

Recovery Trajectories of Child and Family Outcomes Following Online Family Problem-Solving Therapy for Children and Adolescents after Traumatic Brain Injury.

OBJECTIVES: We conducted joint analyses from five randomized clinical trials (RCTs) of online family problem-solving therapy (OFPST) for children with traumatic brain injury (TBI) to identify child and parent outcomes most sensitive to OFPST and trajectories of recovery over time. METHODS: We examined data from 359 children with complicated mild to severe TBI, aged 5-18, randomized to OFPST or a control condition. Using profile analyses, we examined group differences on parent-reported child (internalizing and externalizing behavior problems, executive function behaviors, social competence) and family outcomes (parental depression, psychological distress, family functioning, parent-child conflict). RESULTS: We found a main effect for measure for both child and family outcomes [F(3, 731) = 7.35, p < .001; F(3, 532) = 4.79, p = .003, respectively], reflecting differing degrees of improvement across measures for both groups. Significant group-by-time interactions indicated that children and families in the OFPST group had fewer problems than controls at both 6 and 18 months post baseline [t(731) = -5.15, p < .001, and t(731) = -3.90, p = .002, respectively, for child outcomes; t(532) = -4.81, p < .001, and t(532) = -3.80, p < .001, respectively, for family outcomes]. CONCLUSIONS: The results suggest limited differences in the measures' responsiveness to treatment while highlighting OFPST's utility in improving both child behavior problems and parent/family functioning. Group differences were greatest at treatment completion and after extended time post treatment.

Examination of Injury, Host, and Social-Environmental Moderators of Online Family Problem Solving Treatment Efficacy for Pediatric Traumatic Brain Injury Using an Individual Participant Data Meta-Analytic Approach.

A series of five randomized controlled clinical trials (RCTs) conducted between 2002 and 2015 support the potential efficacy of online family problem-solving treatment (OFPST) in improving both child and parent/family outcomes after pediatric traumatic brain injury (TBI). However, small sample sizes and heterogeneity across individual studies have precluded examination of potentially important moderators. We jointly analyzed individual participant data (IPD) from these five RCTs, involving 359 children and adolescents between the ages of 5 and 18 years, to confirm the role of previously identified moderators (child's age and pre-treatment symptom levels, parental education) and to examine other potential moderators (race, sex, IQ), using IPD meta-analysis. This reanalysis revealed statistically strong evidence that parental education, child age at baseline, IQ, sex, and parental depression level pre-treatment moderated the effect of OFPST on various outcomes. In particular, children of parents with a less than high school education exhibited fewer internalizing problems and better social competence. Children injured at an older age exhibited fewer externalizing behaviors and less executive dysfunction following OFPST. Child IQ moderated the effect of OFPST on social competence, with significantly better competence for children with lower IQ who received OFPST. Lower levels of parental depression followed OFPST among subgroups with lower IQ, boys, and higher parental depression scores at baseline. Our findings indicate that the optimal application of OFPST is likely to involve older children, those with lower IQ scores, or those from families with lower socioeconomic status (SES).

Online Family Problem-solving Treatment for Pediatric Traumatic Brain Injury.

BACKGROUND AND OBJECTIVES: To determine whether online family problem-solving treatment (OFPST) is more effective in improving behavioral outcomes after pediatric traumatic brain injury with increasing time since injury. METHODS: This was an individual participant data meta-analysis of outcome data from 5 randomized controlled trials of OFPST conducted between 2003 and 2016. We included 359 children ages 5 to 18 years who were hospitalized for moderate-to-severe traumatic brain injury 1 to 24 months earlier. Outcomes, assessed pre- and posttreatment, included parent-reported measures of externalizing, internalizing, and executive function behaviors and social competence. RESULTS: Participants included 231 boys and 128 girls with an average age at injury of 13.6 years. Time since injury and age at injury moderated OFPST efficacy. For earlier ages and short time since injury, control participants demonstrated better externalizing problem scores than those receiving OFPST (Cohen's d = 0.44; P = .008; n = 295), whereas at older ages and longer time since injury, children receiving OFPST had better scores (Cohen's d = -0.60; P = .002). Children receiving OFPST were rated as having better executive functioning relative to control participants at a later age at injury, with greater effects seen at longer (Cohen's d = -0.66; P = .009; n = 298) than shorter (Cohen's d = -0. 28; P = .028) time since injury. CONCLUSIONS: OFPST may be more beneficial for older children and when begun after the initial months postinjury. With these findings, we shed light on the optimal application of family problem-solving treatments within the first 2 years after injury.

A comparison of existing methods for multiple imputation in individual participant data meta-analysis.

Multiple imputation is a popular method for addressing missing data, but its implementation is difficult when data have a multilevel structure and one or more variables are systematically missing. This systematic missing data pattern may commonly occur in meta-analysis of individual participant data, where some variables are never observed in some studies, but are present in other hierarchical data settings. In these cases, valid imputation must account for both relationships between variables and correlation within studies. Proposed methods for multilevel imputation include specifying a full joint model and multiple imputation with chained equations (MICE). While MICE is attractive for its ease of implementation, there is little existing work describing conditions under which this is a valid alternative to specifying the full joint model. We present results showing that for multilevel normal models, MICE is rarely exactly equivalent to joint model imputation. Through a simulation study and an example using data from a traumatic brain injury study, we found that in spite of theoretical differences, MICE imputations often produce results similar to those obtained using the joint model. We also assess the influence of prior distributions in MICE imputation methods and find that when missingness is high, prior choices in MICE models tend to affect estimation of across-study variability more than compatibility of conditional likelihoods. Copyright © 2017 John Wiley & Sons, Ltd.

Comparing multiple imputation methods for systematically missing subject-level data.

When conducting research synthesis, the collection of studies that will be combined often do not measure the same set of variables, which creates missing data. When the studies to combine are longitudinal, missing data can occur on the observation-level (time-varying) or the subject-level (non-time-varying). Traditionally, the focus of missing data methods for longitudinal data has been on missing observation-level variables. In this paper, we focus on missing subject-level variables and compare two multiple imputation approaches: a joint modeling approach and a sequential conditional modeling approach. We find the joint modeling approach to be preferable to the sequential conditional approach, except when the covariance structure of the repeated outcome for each individual has homogenous variance and exchangeable correlation. Specifically, the regression coefficient estimates from an analysis incorporating imputed values based on the sequential conditional method are attenuated and less efficient than those from the joint method. Remarkably, the estimates from the sequential conditional method are often less efficient than a complete case analysis, which, in the context of research synthesis, implies that we lose efficiency by combining studies. Copyright © 2015 John Wiley & Sons, Ltd.

Likelihood-Based Random-Effect Meta-Analysis of Binary Events.

Meta-analysis has been used extensively for evaluation of efficacy and safety of medical interventions. Its advantages and utilities are well known. However, recent studies have raised questions about the accuracy of the commonly used moment-based meta-analytic methods in general and for rare binary outcomes in particular. The issue is further complicated for studies with heterogeneous effect sizes. Likelihood-based mixed-effects modeling provides an alternative to moment-based methods such as inverse-variance weighted fixed- and random-effects estimators. In this article, we compare and contrast different mixed-effect modeling strategies in the context of meta-analysis. Their performance in estimation and testing of overall effect and heterogeneity are evaluated when combining results from studies with a binary outcome. Models that allow heterogeneity in both baseline rate and treatment effect across studies have low type I and type II error rates, and their estimates are the least biased among the models considered.

Systematic review and meta-analysis of cognitive interventions for children with central nervous system disorders and neurodevelopmental disorders.

OBJECTIVE: To assess the efficacy of cognitive interventions for children with neurological disorders, acquired brain injuries, and neurodevelopmental disorders. METHOD: We searched for randomized controlled trials of cognitive interventions; 13 studies met inclusion criteria. Risk of bias was rated for each study. Standardized effect size estimates were examined in 7 outcome domains. The overall quality of evidence was rated using the Grading of Recommendations Assessment, Development and Evaluation system. RESULTS: Significant positive treatment effects were found in all outcome domains aside from inhibitory control. Effects were large for attention, working memory, and memory tasks, and small for academic achievement and behavior rating scales. Results exhibited substantial heterogeneity in all domains. Overall quality of evidence was rated very low in all domains, suggesting substantial uncertainty about effect size estimates. DISCUSSION: The results provide some evidence of a positive benefit from cognitive interventions, but cannot be regarded as robust given the overall very low quality of the evidence.

The use of propensity scores and observational data to estimate randomized controlled trial generalizability bias.

Although randomized controlled trials are considered the 'gold standard' for clinical studies, the use of exclusion criteria may impact the external validity of the results. It is unknown whether estimators of effect size are biased by excluding a portion of the target population from enrollment. We propose to use observational data to estimate the bias due to enrollment restrictions, which we term generalizability bias. In this paper, we introduce a class of estimators for the generalizability bias and use simulation to study its properties in the presence of non-constant treatment effects. We find the surprising result that our estimators can be unbiased for the true generalizability bias even when all potentially confounding variables are not measured. In addition, our proposed doubly robust estimator performs well even for mis-specified models.

Reliable change in postconcussive symptoms and its functional consequences among children with mild traumatic brain injury.

OBJECTIVE To examine reliable change in postconcussive symptoms and its functional consequences among children with mild traumatic brain injury (TBI) over the first year postinjury as compared with children with orthopedic injuries. DESIGN Prospective, longitudinal cohort. SETTING Emergency departments at 2 children's hospitals. PARTICIPANTS Eight- to 15-year-old children with mild TBI (n = 186) or orthopedic injuries (n = 99). MAIN EXPOSURE Closed-head or orthopedic trauma. MAIN OUTCOME MEASURES Parents rated preinjury symptoms retrospectively shortly after injury and postconcussive symptoms at 2 weeks and 3 and 12 months postinjury. A regression-based approach was used to determine whether each child displayed reliable increases in postconcussive symptoms at each postinjury occasion. Health-related quality of life was assessed at 3 and 12 months postinjury. Information regarding children's educational programming was collected at the initial and 12-month assessments. RESULTS Children with mild TBI were significantly more likely than those with orthopedic injuries to show reliable increases in both cognitive and somatic symptoms. Group differences in the likelihood of reliable increases became less common with time for somatic symptoms but persisted to 12 months postinjury for cognitive symptoms. Among children with mild TBI, reliable increases in symptoms were more common among children with loss of consciousness or abnormalities on neuroimaging. Reliable increases in symptoms were associated with significant declines in health-related quality of life and an increased likelihood of educational intervention. CONCLUSION Many children with mild TBI show reliable increases in postconcussive symptoms that are associated with significant functional impairment in their daily lives.

Meta-Analysis of Rare Binary Adverse Event Data.

We examine the use of fixed-effects and random-effects moment-based meta-analytic methods for analysis of binary adverse event data. Special attention is paid to the case of rare adverse events which are commonly encountered in routine practice. We study estimation of model parameters and between-study heterogeneity. In addition, we examine traditional approaches to hypothesis testing of the average treatment effect and detection of the heterogeneity of treatment effect across studies. We derive three new methods, simple (unweighted) average treatment effect estimator, a new heterogeneity estimator, and a parametric bootstrapping test for heterogeneity. We then study the statistical properties of both the traditional and new methods via simulation. We find that in general, moment-based estimators of combined treatment effects and heterogeneity are biased and the degree of bias is proportional to the rarity of the event under study. The new methods eliminate much, but not all of this bias. The various estimators and hypothesis testing methods are then compared and contrasted using an example dataset on treatment of stable coronary artery disease.

Estimating treatment effect via simple cross design synthesis.

Randomized controlled trials (RCTs) are the traditional gold standard evidence for medical decision-making. However, protocols that limit enrollment eligibility introduce selection error that severely limits a RCT's applicability to a wide range of patients. Conversely, high quality observational data can be representative of entire populations, but freedom to choose treatment can bias estimators based on this data. Cross design synthesis (CDS) is an approach to combining both RCT and observational data in a single analysis that capitalizes on the RCT's strong internal validity and the observational study's strong external validity. We proposed and assessed a simple estimator of effect size based on the CDS approach. We evaluated its properties within a formal framework of causal estimation and compared our estimator with more traditional estimators based on single sources of evidence. We show that under ideal conditions the simple CDS estimator is unbiased whenever the observational data-based estimators' treatment selection error is constant across those who are and are not eligible for RCT participation. Whereas this assumption may not often hold in practice, assumptions required for the unbiasedness of usual single-source estimators may also be implausible. We show that, under some reasonable data assumptions, our simple CDS estimator has smaller bias and better coverage than commonly used estimates based on randomized or observational studies alone.

Root resorption associated with orthodontic tooth movement: a systematic review.

INTRODUCTION: This systematic review evaluated root resorption as an outcome for patients who had orthodontic tooth movement. The results could provide the best available evidence for clinical decisions to minimize the risks and severity of root resorption. METHODS: Electronic databases were searched, nonelectronic journals were hand searched, and experts in the field were consulted with no language restrictions. Study selection criteria included randomized clinical trials involving human subjects for orthodontic tooth movement, with fixed appliances, and root resorption recorded during or after treatment. Two authors independently reviewed and extracted data from the selected studies on a standardized form. RESULTS: The searches retrieved 921 unique citations. Titles and abstracts identified 144 full articles from which 13 remained after the inclusion criteria were applied. Differences in the methodologic approaches and reporting results made quantitative statistical comparisons impossible. Evidence suggests that comprehensive orthodontic treatment causes increased incidence and severity of root resorption, and heavy forces might be particularly harmful. Orthodontically induced inflammatory root resorption is unaffected by archwire sequencing, bracket prescription, and self-ligation. Previous trauma and tooth morphology are unlikely causative factors. There is some evidence that a 2 to 3 month pause in treatment decreases total root resorption. CONCLUSIONS: The results were inconclusive in the clinical management of root resorption, but there is evidence to support the use of light forces, especially with incisor intrusion.