Correspondence measures for assessing replication success.

Given recent evidence challenging the replicability of results in the social and behavioral sciences, critical questions have been raised about appropriate measures for determining replication success in comparing effect estimates across studies. At issue is the fact that conclusions about replication success often depend on the measure used for evaluating correspondence in results. Despite the importance of choosing an appropriate measure, there is still no widespread agreement about which measures should be used. This article addresses these questions by describing formally the most commonly used measures for assessing replication success, and by comparing their performance in different contexts according to their replication probabilities-that is, the probability of obtaining replication success given study-specific settings. The measures may be characterized broadly as conclusion-based approaches, which assess the congruence of two independent studies' conclusions about the presence of an effect, and distance-based approaches, which test for a significant difference or equivalence of two effect estimates. We also introduce a new measure for assessing replication success called the correspondence test, which combines a difference and equivalence test in the same framework. To help researchers plan prospective replication efforts, we provide closed formulas for power calculations that can be used to determine the minimum detectable effect size (and thus, sample sizes) for each study so that a predetermined minimum replication probability can be achieved. Finally, we use a replication data set from the Open Science Collaboration (2015) to demonstrate the extent to which conclusions about replication success depend on the correspondence measure selected. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

The validity of causal claims with repeated measures designs: A within-study comparison evaluation of differences-in-differences and the comparative interrupted time series.

Modern policies are commonly evaluated not with randomized experiments but with repeated measures designs like difference-in-differences (DID) and the comparative interrupted time series (CITS). The key benefit of these designs is that they control for unobserved confounders that are fixed over time. However, DID and CITS designs only result in unbiased impact estimates when the model assumptions are consistent with the data at hand. In this paper, we empirically test whether the assumptions of repeated measures designs are met in field settings. Using a within-study comparison design, we compare experimental estimates of the impact of patient-directed care on medical expenditures to non-experimental DID and CITS estimates for the same target population and outcome. Our data come from a multi-site experiment that includes participants receiving Medicaid in Arkansas, Florida, and New Jersey. We present summary measures of repeated measures bias across three states, four comparison groups, two model specifications, and two outcomes. We find that, on average, bias resulting from repeated measures designs are very close to zero (less than 0.01 standard deviations; SDs). Further, we find that comparison groups which have pre-treatment trends that are visibly parallel to the treatment group result in less bias than those with visibly divergent trends. However, CITS models that control for baseline trends produced slightly more bias and were less precise than DID models that only control for baseline means. Overall, we offer optimistic evidence in favor of repeated measures designs when randomization is not feasible.

Design-Based Approaches to Causal Replication Studies.

Recent interest in promoting replication efforts assumes that there is well-established methodological guidance for designing and implementing these studies. However, no such consensus exists in the methodology literature. This article addresses these challenges by describing design-based approaches for planning systematic replication studies. Our general approach is derived from the Causal Replication Framework (CRF), which formalizes the assumptions under which replication success can be expected. The assumptions may be understood broadly as replication design requirements and individual study design requirements. Replication failure occurs when one or more CRF assumptions are violated. In design-based approaches to replication, CRF assumptions are systematically tested to evaluate the replicability of effects, as well as to identify sources of effect variation when replication failure is observed. The paper describes research designs for replication and demonstrates how multiple designs may be combined in systematic replication efforts, as well as how diagnostic measures may be used to assess the extent to which CRF assumptions are met in field settings.

What Can Be Learned From Empirical Evaluations of Nonexperimental Methods?

Given the widespread use of nonexperimental (NE) methods for assessing program impacts, there is a strong need to know whether NE approaches yield causally valid results in field settings. In within-study comparison (WSC) designs, the researcher compares treatment effects from an NE with those obtained from a randomized experiment that shares the same target population. The goal is to assess whether the stringent assumptions required for NE methods are likely to be met in practice. This essay provides an overview of recent efforts to empirically evaluate NE method performance in field settings. We discuss a brief history of the design, highlighting methodological innovations along the way. We also describe papers that are included in this two-volume special issue on WSC approaches and suggest future areas for consideration in the design, implementation, and analysis of WSCs.

Designs of Empirical Evaluations of Nonexperimental Methods in Field Settings.

Over the last three decades, a research design has emerged to evaluate the performance of nonexperimental (NE) designs and design features in field settings. It is called the within-study comparison (WSC) approach or the design replication study. In the traditional WSC design, treatment effects from a randomized experiment are compared to those produced by an NE approach that shares the same target population. The nonexperiment may be a quasi-experimental design, such as a regression-discontinuity or an interrupted time-series design, or an observational study approach that includes matching methods, standard regression adjustments, and difference-in-differences methods. The goals of the WSC are to determine whether the nonexperiment can replicate results from a randomized experiment (which provides the causal benchmark estimate), and the contexts and conditions under which these methods work in practice. This article presents a coherent theory of the design and implementation of WSCs for evaluating NE methods. It introduces and identifies the multiple purposes of WSCs, required design components, common threats to validity, design variants, and causal estimands of interest in WSCs. It highlights two general approaches for empirical evaluations of methods in field settings, WSC designs with independent and dependent benchmark and NE arms. This article highlights advantages and disadvantages for each approach, and conditions and contexts under which each approach is optimal for addressing methodological questions.

Assessing Correspondence Between Experimental and Nonexperimental Estimates in Within-Study Comparisons.

In within-study comparison (WSC) designs, treatment effects from a nonexperimental design, such as an observational study or a regression-discontinuity design, are compared to results obtained from a well-designed randomized control trial with the same target population. The goal of the WSC is to assess whether nonexperimental and experimental designs yield the same results in field settings. A common analytic challenge with WSCs, however, is the choice of appropriate criteria for determining whether nonexperimental and experimental results replicate. This article examines different distance-based correspondence measures for assessing correspondence in experimental and nonexperimental estimates. Distance-based measures investigate whether the difference in estimates is small enough to claim equivalence of methods. We use a simulation study to examine the statistical properties of common correspondence measures and recommend a new and straightforward approach that combines traditional significance testing and equivalence testing in the same framework. The article concludes with practical advice on assessing and interpreting results in WSC contexts.

Psychometric properties of the Chalder Fatigue Scale revisited: an exploratory structural equation modeling approach.

OBJECTIVE: Previous validation studies of the Chalder Fatigue Scale (CFS) suffer methodological shortcomings. The present study aimed to re-evaluate its psychometric properties using exploratory structural equation modeling (ESEM). METHODS: A Chinese sample of 1259 community-dwelling residents completed the 11-item Chinese CFS and a variety of health measures (anxiety, depression, exhaustion, sleep disturbance, and quality of life). In addition to traditional confirmatory factor analysis, ESEM was performed to assess the fit of two- and three-factor models using robust maximum likelihood estimation and oblique geomin rotation. Convergent validity of the CFS was examined via associations with five covariates (gender, age, exercise, perceived health, and life event) and the health measures in the ESEM model. RESULTS: The ESEM models displayed a superior fit to confirmatory factor models. The three-factor ESEM model showed a satisfactory model fit to the data but not for the two-factor model. The three factors were physical fatigue (three items, α = .800), low energy (four items, α = .821), and mental fatigue (four items, α = .861). The factors exhibited convergent validity with the model covariates and health measures. CONCLUSION: The results demonstrate the satisfactory reliability and convergent validity for the three-factor structure of the CFS as a valid measure of fatigue symptoms in the general population. Future psychometric studies could adopt the ESEM approach as a practical alternative to traditional confirmatory factor analysis.

Add-on effect of chinese herbal medicine on mortality in myocardial infarction: systematic review and meta-analysis of randomized controlled trials.

In China, Chinese herbal medicine (CHM) is widely used as an adjunct to biomedicine (BM) in treating myocardial infarction (MI). This meta-analysis of RCTs evaluated the efficacy of combined CHM-BM in the treatment of MI, compared to BM alone. Sixty-five RCTs (12,022 patients) of moderate quality were identified. 6,036 patients were given CHM plus BM, and 5,986 patients used BM only. Combined results showed clear additional effect of CHM-BM treatment in reducing all-cause mortality (relative risk reduction (RRR) = 37%, 95% CI = 28%-45%, I(2) = 0.0%) and mortality of cardiac origin (RRR = 39%, 95% CI = 22%-52%, I(2) = 22.8). Benefits remained after random-effect trim and fill adjustment for publication bias (adjusted RRR for all-cause mortality = 29%, 95% CI = 16%-40%; adjusted RRR for cardiac death = 32%, 95% CI = 15%-46%). CHM is also found to be efficacious in lowering the risk of fatal and nonfatal cardiogenic shock, cardiac arrhythmia, myocardial reinfarction, heart failure, angina, and occurrence of total heart events. In conclusion, addition of CHM is very likely to be able to improve survival of MI patients who are already receiving BM. Further confirmatory evaluation via large blinded randomized trials is warranted.

The effect of qigong exercise on immunity and infections: a systematic review of controlled trials.

The objective of this review was to summarize and critically evaluate the clinical evidence of the effect of qigong exercise on immunity and its efficacy in the prevention or treatment of infectious diseases. Thirteen databases were searched from their respective inceptions through January 2011, and all controlled clinical trials of qigong exercise on immunity and infections were included. Quality and validity of the included studies were evaluated using standard scales. Seven studies including two randomized controlled trials (RCTs), two controlled clinical trials (CCTs) and three retrospective observational studies (ROSs) met the inclusion criteria. One study focused on functional measures of immunity (antigen-induced immunity) and six studies on enumerative parameters of immunity. No study on clinical symptoms relevant to infectious diseases could be identified. Overall, the included studies suggested favorable effects of qigong exercise on immunity, but the quality of research for most of the studies examined in this review was poor. Further rigorously designed studies are required, which should adhere to accepted standards of methodology for clinical trials.

A systematic review of the effectiveness of qigong exercise in supportive cancer care.

PURPOSE: Qigong as a complementary and alternative modality of traditional Chinese medicine is often used by cancer patients to manage their symptoms. The aim of this systematic review is to critically evaluate the effectiveness of qigong exercise in cancer care. METHODS: Thirteen databases were searched from their inceptions through November 2010. All controlled clinical trials of qigong exercise among cancer patients were included. The strength of the evidence was evaluated for all included studies using the Oxford Centre for Evidence-based Medicine Levels of Evidence. The validity of randomized controlled trials (RCTs) was also evaluated using the Jadad Scale. RESULTS: Twenty-three studies including eight RCTs and fifteen non-randomized controlled clinical trials (CCTs) were identified. The effects of qigong on physical and psychosocial outcomes were examined in 14 studies and the effects on biomedical outcomes were examined in 15 studies. For physical and psychosocial outcomes, it is difficult to draw a conclusion due to heterogeneity of outcome measures and variability of the results in the included studies. Among reviewed studies on biomedical outcomes, a consistent tendency appears to emerge which suggests that the patients treated with qigong exercise in combination with conventional methods had significant improvement in immune function than the patients treated with conventional methods alone. CONCLUSIONS: Due to high risk of bias and methodological problems in the majority of included studies, it is still too early to draw conclusive statements. Further vigorously designed large-scale RCTs with validated outcome measures are needed.

A randomized experiment comparing random and cutoff-based assignment.

In this article, we review past studies comparing randomized experiments to regression discontinuity designs, mostly finding similar results, but with significant exceptions. The latter might be due to potential confounds of study characteristics with assignment method or with failure to estimate the same parameter over methods. In this study, we correct the problems by randomly assigning 588 participants to be in a randomized experiment or a regression discontinuity design in which they are otherwise treated identically, comparing results estimating both the same and different parameters. Analysis includes parametric, semiparametric, and nonparametric methods of modeling nonlinearities. Results suggest that estimates from regression discontinuity designs approximate the results of randomized experiments reasonably well but also raise the issue of what constitutes agreement between the 2 estimates.

Using Chinese version of MYMOP in Chinese medicine evaluation: validity, responsiveness and minimally important change.

BACKGROUND: Measure Yourself Medical Outcome Profile (MYMOP) is a patient generated outcome instrument applicable in the evaluation of both allopathic and complementary medicine treatment. This study aims to adapt MYMOP into Chinese, and to assess its validity, responsiveness and minimally important change values in a sample of patients using Chinese medicine (CM) services. METHODS: A Chinese version of MYMOP (CMYMOP) is developed by forward-backward-forward translation strategy, expert panel assessment and pilot testing amongst patients. 272 patients aged 18 or above with subjective symptoms in the past 2 weeks were recruited at a CM clinic, and were invited to complete a set of questionnaire containing CMYMOP and SF-36. Follow ups were performed at 2nd and 4th week after consultation, using the same set of questionnaire plus a global rating of change question. Criterion validity of CMYMOP was assessed by its correlation with SF-36 at baseline, and responsiveness was evaluated by calculating the Cohen effect size (ES) of change at two follow ups. Minimally important difference (MID) values were estimated via anchor based method, while minimally detectable difference (MDC) figures were calculated by distribution based method. RESULTS: Criterion validity of CMYMOP was demonstrated by negative correlation between CMYMOP Profile scores and all SF-36 domain and summary scores at baseline. For responsiveness between baseline and 4th week follow up, ES of CMYMOP Symptom 1, Activity and Profile reached the moderate change threshold (ES>0.5), while Symptom 2 and Wellbeing reached the weak change threshold (ES>0.2). None of the SF-36 scores reached the moderate change threshold, implying CMYMOP's stronger responsiveness in CM setting. At 2nd week follow up, MID values for Symptom 1, Symptom 2, Wellbeing and Profile items were 0.894, 0.580, 0.263 and 0.516 respectively. For Activity item, MDC figure of 0.808 was adopted to estimate MID. CONCLUSIONS: The findings support the validity and responsiveness of CMYMOP for capturing patient centred clinical changes within 2 weeks in a CM clinical setting. Further researches are warranted (1) to estimate Activity item MID, (2) to assess the test-retest reliability of CMYMOP, and (3) to perform further MID evaluation using multiple, item specific anchor questions.

Psychological resilience and dysfunction among hospitalized survivors of the SARS epidemic in Hong Kong: a latent class approach.

OBJECTIVE: To examine trajectories of psychological functioning using latent class analysis on a sample of hospitalized survivors of the 2003 severe acute respiratory syndrome (SARS) epidemic in Hong Kong. DESIGN: A longitudinal study of 997 survivors, recruited from among 1,331 individuals hospitalized for SARS, were interviewed at 6, 12, and 18 months after hospitalization. MAIN OUTCOME MEASURES: Psychological and physical functioning at each time point was measured using the 12-item Medical Outcome Study Short-Form Health Survey (SF-12). RESULTS: Four latent classes were identified--chronic dysfunction, delayed dysfunction, recovery, and resilience. All groups had better physical health than the chronic group. Resilient and recovered individuals had greater social support and less SARS-related worry, and resilient individuals were more likely to be male. The resilient group also had greater social support than the delayed group and better physical functioning than the recovered group. CONCLUSION: This study demonstrated that longitudinal outcome trajectories following a major health-threat event in an Asian sample bear close resemblance to prototypical trajectories observed in trauma studies using Western samples. Unique predictors of the trajectories included factors observed in previous studies, such as social support, as well as factors of particular relevance to a major disease outbreak, such as SARS-related worry.

Development and validation of an all-cause mortality risk score in type 2 diabetes.

BACKGROUND: Diabetes reduces life expectancy by 10 to 12 years, but whether death can be predicted in type 2 diabetes mellitus remains uncertain. METHODS: A prospective cohort of 7583 type 2 diabetic patients enrolled since 1995 were censored on July 30, 2005, or after 6 years of follow-up, whichever came first. A restricted cubic spline model was used to check data linearity and to develop linear-transforming formulas. Data were randomly assigned to a training data set and to a test data set. A Cox model was used to develop risk scores in the test data set. Calibration and discrimination were assessed in the test data set. RESULTS: A total of 619 patients died during a median follow-up period of 5.51 years, resulting in a mortality rate of 18.69 per 1000 person-years. Age, sex, peripheral arterial disease, cancer history, insulin use, blood hemoglobin levels, linear-transformed body mass index, random spot urinary albumin-creatinine ratio, and estimated glomerular filtration rate at enrollment were predictors of all-cause death. A risk score for all-cause mortality was developed using these predictors. The predicted and observed death rates in the test data set were similar (P > .70). The area under the receiver operating characteristic curve was 0.85 for 5 years of follow-up. Using the risk score in ranking cause-specific deaths, the area under the receiver operating characteristic curve was 0.95 for genitourinary death, 0.85 for circulatory death, 0.85 for respiratory death, and 0.71 for neoplasm death. CONCLUSIONS: Death in type 2 diabetes mellitus can be predicted using a risk score consisting of commonly measured clinical and biochemical variables. Further validation is needed before clinical use.

Impacts of chronic kidney disease and albuminuria on associations between coronary heart disease and its traditional risk factors in type 2 diabetic patients - the Hong Kong diabetes registry.

BACKGROUND: Glycated haemoglobin (HbA1c), blood pressure and body mass index (BMI) are risk factors for albuminuria, the latter in turn can lead to hyperlipidaemia. We used novel statistical analyses to examine how albuminuria and chronic kidney disease (CKD) may influence the effects of other risk factors on coronary heart disease (CHD). METHODS: A prospective cohort of 7067 Chinese type 2 diabetic patients without history of CHD enrolled since 1995 were censored on July 30th, 2005. Cox proportional hazard regression with restricted cubic spline was used to auto-select predictors. Hazard ratio plots were used to examine the risk of CHD. Based on these plots, non-linear risk factors were categorised and the categorised variables were refitted into various Cox models in a stepwise manner to confirm the findings. RESULTS: Age, male gender, duration of diabetes, spot urinary albumin: creatinine ratio, estimated glomerular filtration rate, total cholesterol (TC), high density lipoprotein cholesterol (HDL-C) and current smoking status were risk factors of CHD. Linear association between TC and CHD was observed only in patients with albuminuria. Although in general, increased HDL-C was associated with decreased risk of CHD, full-range HDL-C was associated with CHD in an A-shaped manner with a zenith at 1.1 mmol/L. Albuminuria and CKD were the main contributors for the paradoxically positive association between HDL-C and CHD for HDL-C values less than 1.1 mmol/L. CONCLUSION: In type 2 diabetes, albuminuria plays a linking role between conventional risk factors and CHD. The onset of CKD changes risk associations between lipids and CHD.

Prognostication in severe acute respiratory syndrome: a retrospective time-course analysis of 1312 laboratory-confirmed patients in Hong Kong.

BACKGROUND AND OBJECTIVE: The temporal importance of prognostic indicators for severe acute respiratory syndrome (SARS) has not been studied. This study identified the various clinical prognostic factors for SARS and described the temporal evolution of these factors in the course of the SARS illness in Hong Kong in 2003. METHODS: A retrospective analysis of the entire Hong Kong cohort of 1312 laboratory-confirmed SARS patients aged 15-74 years was undertaken. Demographic, clinical and laboratory data at presentation and investigative data during the first 10 days of illness from the time of symptom onset were compiled. Two adverse outcomes were examined: hospital mortality and the development of oxygenation failure based on the estimated PaO(2)/FiO(2) ratio of <200 mm Hg. Logistic regression was used to identify the association between these prognostic factors and outcomes. RESULTS: Based on adjusted odds ratios with a P-value of <0.05, older age, male gender, elevated pulse rate and elevated neutrophil count were all predictive of oxygenation failure and death during the 10-day illness. Raised serum albumin and creatinine phosphokinase (CPK) levels were predictive of hospital mortality during this period. The presenting ALT and CPK level and the day 7 and day 10 platelet counts were predictive of oxygenation failure while the day 7 LDH was predictive of death. Contact exposure outside health-care institutions also appeared to carry higher risk of death. CONCLUSION: This large-scale analysis identified important discriminatory parameters related to the patients' demographic profile (age and gender), severity of illness (pulse rate and neutrophil count), and multisystem derangement (platelet count, CPK, ALT and LDH), all of which prognosticated adverse outcomes during the SARS episode. While age, pulse rate and neutrophil count consistently remained significant prognosticators during the first 10 days of illness, the prognostic impact of other derangements was more time-course dependent. Clinicians should be aware of the time-course evolution of these prognosticators.

Non-invasive versus invasive mechanical ventilation for respiratory failure in severe acute respiratory syndrome.

BACKGROUND: Severe acute respiratory syndrome is frequently complicated by respiratory failure requiring ventilatory support. We aimed to compare the efficacy of non-invasive ventilation against invasive mechanical ventilation treating respiratory failure in this disease. METHODS: Retrospective analysis was conducted on all respiratory failure patients identified from the Hong Kong Hospital Authority Severe Acute Respiratory Syndrome Database. Intubation rate, mortality and secondary outcome of a hospital utilizing non-invasive ventilation under standard infection control conditions (NIV Hospital) were compared against 13 hospitals using solely invasive ventilation (IMV Hospitals). Multiple logistic regression analyses with adjustments for confounding variables were performed to test for association between outcomes and hospital groups. RESULTS: Both hospital groups had comparable demographics and clinical profiles, but NIV Hospital (42 patients) had higher lactate dehydrogenase ratio and worse radiographic score on admission and ribavirin-corticosteroid commencement. Compared to IMV Hospitals (451 patients), NIV Hospital had lower adjusted odds ratios for intubation (0.36, 95% CI 0.164 - 0.791, P = 0.011) and death (0.235, 95% CI 0.077 - 0.716, P = 0.011), and improved earlier after pulsed steroid rescue. There were no instances of transmission of severe acute respiratory syndrome among health care workers due to the use of non-invasive ventilation. CONCLUSION: Compared to invasive mechanical ventilation, non-invasive ventilation as initial ventilatory support for acute respiratory failure in the presence of severe acute respiratory syndrome appeared to be associated with reduced intubation need and mortality.