Correspondence measures for assessing replication success.

Given recent evidence challenging the replicability of results in the social and behavioral sciences, critical questions have been raised about appropriate measures for determining replication success in comparing effect estimates across studies. At issue is the fact that conclusions about replication success often depend on the measure used for evaluating correspondence in results. Despite the importance of choosing an appropriate measure, there is still no widespread agreement about which measures should be used. This article addresses these questions by describing formally the most commonly used measures for assessing replication success, and by comparing their performance in different contexts according to their replication probabilities-that is, the probability of obtaining replication success given study-specific settings. The measures may be characterized broadly as conclusion-based approaches, which assess the congruence of two independent studies' conclusions about the presence of an effect, and distance-based approaches, which test for a significant difference or equivalence of two effect estimates. We also introduce a new measure for assessing replication success called the correspondence test, which combines a difference and equivalence test in the same framework. To help researchers plan prospective replication efforts, we provide closed formulas for power calculations that can be used to determine the minimum detectable effect size (and thus, sample sizes) for each study so that a predetermined minimum replication probability can be achieved. Finally, we use a replication data set from the Open Science Collaboration (2015) to demonstrate the extent to which conclusions about replication success depend on the correspondence measure selected. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Cluster-Based Input Weight Initialization for Echo State Networks.

Echo state networks (ESNs) are a special type of recurrent neural networks (RNNs), in which the input and recurrent connections are traditionally generated randomly, and only the output weights are trained. Despite the recent success of ESNs in various tasks of audio, image, and radar recognition, we postulate that a purely random initialization is not the ideal way of initializing ESNs. The aim of this work is to propose an unsupervised initialization of the input connections using the K -means algorithm on the training data. We show that for a large variety of datasets, this initialization performs equivalently or superior than a randomly initialized ESN while needing significantly less reservoir neurons. Furthermore, we discuss that this approach provides the opportunity to estimate a suitable size of the reservoir based on prior knowledge about the data.

Design-Based Approaches to Causal Replication Studies.

Recent interest in promoting replication efforts assumes that there is well-established methodological guidance for designing and implementing these studies. However, no such consensus exists in the methodology literature. This article addresses these challenges by describing design-based approaches for planning systematic replication studies. Our general approach is derived from the Causal Replication Framework (CRF), which formalizes the assumptions under which replication success can be expected. The assumptions may be understood broadly as replication design requirements and individual study design requirements. Replication failure occurs when one or more CRF assumptions are violated. In design-based approaches to replication, CRF assumptions are systematically tested to evaluate the replicability of effects, as well as to identify sources of effect variation when replication failure is observed. The paper describes research designs for replication and demonstrates how multiple designs may be combined in systematic replication efforts, as well as how diagnostic measures may be used to assess the extent to which CRF assumptions are met in field settings.

Evaluation of a Non-Personalized Optopalatographic Device for Prospective Use in Functional Post-Stroke Dysphagia Therapy.

OBJECTIVE: Stroke survivors commonly suffer from dysphagia, originating from oro-facial impairments which affect swallowing function. Functional therapy often employs tongue exercises that require the patient to perform short motion sequences. Evaluating the patient's performance on those exercises is difficult, because there is no reliable form of visual feedback. METHODS: We propose an optopalatographic device that does not require a personalized dental retainer and is capable of measuring tongue movement trajectories intraorally. The device features nine optical proximity sensors at 100 Hz and is fixated against the hard palate with a specifically developed palatal adhesive. The sensing capabilities of the device were evaluated on a tongue gesture corpus recorded from nine healthy individuals, containing eight different tongue exercises commonly used in functional dysphagia therapy. RESULTS: The measured tongue trajectories contained temporally and spatially resolved information about the tongue movement and location during each exercise. Furthermore, a simple DTW-kNN classifier was able to distinguish the exercises from one another with an average classification accuracy of 97.9 % and 61.4 % (cross-validation and inter-speaker test accuracy, respectively). CONCLUSION: the device can provide real-time feedback for tongue motion and we obtained promising gesture recognition results with relatively few sensors, even in the absence of a personalized dental retainer. SIGNIFICANCE: Non-personalized optopalatography is readily available and could aid in improving functional dysphagia therapy by providing visual feedback to both the physician and patient.

The Role of Sample Size to Attain Statistically Comparable Groups - A Required Data Preprocessing Step to Estimate Causal Effects With Observational Data.

Background: Propensity score methods provide data preprocessing tools to remove selection bias and attain statistically comparable groups - the first requirement when attempting to estimate causal effects with observational data. Although guidelines exist on how to remove selection bias when groups in comparison are large, not much is known on how to proceed when one of the groups in comparison, for example, a treated group, is particularly small, or when the study also includes lots of observed covariates (relative to the treated group's sample size). Objectives: This article investigates whether propensity score methods can help us to remove selection bias in studies with small treated groups and large amount of observed covariates. Measures: We perform a series of simulation studies to study factors such as sample size ratio of control to treated units, number of observed covariates and initial imbalances in observed covariates between the groups of units in comparison, that is, selection bias. Results: The results demonstrate that selection bias can be removed with small treated samples, but under different conditions than in studies with large treated samples. For example, a study design with 10 observed covariates and eight treated units will require the control group to be at least 10 times larger than the treated group, whereas a study with 500 treated units will require at least, only, two times bigger control group. Conclusions: To confirm the usefulness of simulation study results for practice, we carry out an empirical evaluation with real data. The study provides insights for practice and directions for future research.

Reservoir computing with biocompatible organic electrochemical networks for brain-inspired biosignal classification.

Early detection of malign patterns in patients' biological signals can save millions of lives. Despite the steady improvement of artificial intelligence-based techniques, the practical clinical application of these methods is mostly constrained to an offline evaluation of the patients' data. Previous studies have identified organic electrochemical devices as ideal candidates for biosignal monitoring. However, their use for pattern recognition in real time was never demonstrated. Here, we produce and characterize brain-inspired networks composed of organic electrochemical transistors and use them for time-series predictions and classification tasks using the reservoir computing approach. To show their potential use for biofluid monitoring and biosignal analysis, we classify four classes of arrhythmic heartbeats with an accuracy of 88%. The results of this study introduce a previously unexplored paradigm for biocompatible computational platforms and may enable development of ultralow-power consumption hardware-based artificial neural networks capable of interacting with body fluids and biological tissues.

Causal graphical views of fixed effects and random effects models.

Despite the long-standing discussion on fixed effects (FE) and random effects (RE) models, how and under what conditions both methods can eliminate unmeasured confounding bias has not yet been widely understood in practice. Using a simple pretest-posttest design in a linear setting, this paper translates the conventional algebraic formalization of FE and RE models into causal graphs and provides intuitively accessible graphical explanations about their data-generating and bias-removing processes. The proposed causal graphs highlight that FE and RE models consider different data-generating models. RE models presume a data-generating model that is identical to a randomized controlled trial, while FE models allow for unobserved time-invariant treatment-outcome confounding. Augmenting regular causal graphs that describe data-generating processes by adding the computational structures of FE and RE estimators, the paper visualizes how FE estimators (gain score and deviation score estimators) and RE estimators (quasi-deviation score estimators) offset unmeasured confounding bias. In contrast to standard regression or matching estimators that reduce confounding bias by blocking non-causal paths via conditioning, FE and RE estimators offset confounding bias by deliberately creating new non-causal paths and associations of opposite sign. Though FE and RE estimators are similar in their bias-offsetting mechanisms, the augmented graphs reveal their subtle differences that can result in different biases in observational studies.

When does measurement error in covariates impact causal effect estimates? Analytic derivations of different scenarios and an empirical illustration.

The average causal treatment effect (ATE) can be estimated from observational data based on covariate adjustment. Even if all confounding covariates are observed, they might not necessarily be reliably measured and may fail to obtain an unbiased ATE estimate. Instead of fallible covariates, the respective latent covariates can be used for covariate adjustment. But is it always necessary to use latent covariates? How well do analysis of covariance (ANCOVA) or propensity score (PS) methods estimate the ATE when latent covariates are used? We first analytically delineate the conditions under which latent instead of fallible covariates are necessary to obtain the ATE. Then we empirically examine the difference between ATE estimates when adjusting for fallible or latent covariates in an applied example. We discuss the issue of fallible covariates within a stochastic theory of causal effects and analyse data of a within-study comparison with recently developed ANCOVA and PS procedures that allow for latent covariates. We show that fallible covariates do not necessarily bias ATE estimates, but point out different scenarios in which adjusting for latent covariates is required. In our empirical application, we demonstrate how latent covariates can be incorporated for ATE estimation in ANCOVA and in PS analysis.

What Can Be Learned From Empirical Evaluations of Nonexperimental Methods?

Given the widespread use of nonexperimental (NE) methods for assessing program impacts, there is a strong need to know whether NE approaches yield causally valid results in field settings. In within-study comparison (WSC) designs, the researcher compares treatment effects from an NE with those obtained from a randomized experiment that shares the same target population. The goal is to assess whether the stringent assumptions required for NE methods are likely to be met in practice. This essay provides an overview of recent efforts to empirically evaluate NE method performance in field settings. We discuss a brief history of the design, highlighting methodological innovations along the way. We also describe papers that are included in this two-volume special issue on WSC approaches and suggest future areas for consideration in the design, implementation, and analysis of WSCs.

Designs of Empirical Evaluations of Nonexperimental Methods in Field Settings.

Over the last three decades, a research design has emerged to evaluate the performance of nonexperimental (NE) designs and design features in field settings. It is called the within-study comparison (WSC) approach or the design replication study. In the traditional WSC design, treatment effects from a randomized experiment are compared to those produced by an NE approach that shares the same target population. The nonexperiment may be a quasi-experimental design, such as a regression-discontinuity or an interrupted time-series design, or an observational study approach that includes matching methods, standard regression adjustments, and difference-in-differences methods. The goals of the WSC are to determine whether the nonexperiment can replicate results from a randomized experiment (which provides the causal benchmark estimate), and the contexts and conditions under which these methods work in practice. This article presents a coherent theory of the design and implementation of WSCs for evaluating NE methods. It introduces and identifies the multiple purposes of WSCs, required design components, common threats to validity, design variants, and causal estimands of interest in WSCs. It highlights two general approaches for empirical evaluations of methods in field settings, WSC designs with independent and dependent benchmark and NE arms. This article highlights advantages and disadvantages for each approach, and conditions and contexts under which each approach is optimal for addressing methodological questions.

Assessing Correspondence Between Experimental and Nonexperimental Estimates in Within-Study Comparisons.

In within-study comparison (WSC) designs, treatment effects from a nonexperimental design, such as an observational study or a regression-discontinuity design, are compared to results obtained from a well-designed randomized control trial with the same target population. The goal of the WSC is to assess whether nonexperimental and experimental designs yield the same results in field settings. A common analytic challenge with WSCs, however, is the choice of appropriate criteria for determining whether nonexperimental and experimental results replicate. This article examines different distance-based correspondence measures for assessing correspondence in experimental and nonexperimental estimates. Distance-based measures investigate whether the difference in estimates is small enough to claim equivalence of methods. We use a simulation study to examine the statistical properties of common correspondence measures and recommend a new and straightforward approach that combines traditional significance testing and equivalence testing in the same framework. The article concludes with practical advice on assessing and interpreting results in WSC contexts.

Identification and Sensitivity Analysis for Average Causal Mediation Effects with Time-Varying Treatments and Mediators: Investigating the Underlying Mechanisms of Kindergarten Retention Policy.

Considering that causal mechanisms unfold over time, it is important to investigate the mechanisms over time, taking into account the time-varying features of treatments and mediators. However, identification of the average causal mediation effect in the presence of time-varying treatments and mediators is often complicated by time-varying confounding. This article aims to provide a novel approach to uncovering causal mechanisms in time-varying treatments and mediators in the presence of time-varying confounding. We provide different strategies for identification and sensitivity analysis under homogeneous and heterogeneous effects. Homogeneous effects are those in which each individual experiences the same effect, and heterogeneous effects are those in which the effects vary over individuals. Most importantly, we provide an alternative definition of average causal mediation effects that evaluates a partial mediation effect; the effect that is mediated by paths other than through an intermediate confounding variable. We argue that this alternative definition allows us to better assess at least a part of the mediated effect and provides meaningful and unique interpretations. A case study using ECLS-K data that evaluates kindergarten retention policy is offered to illustrate our proposed approach.

Pretest Measures of the Study Outcome and the Elimination of Selection Bias: Evidence from Three Within Study Comparisons.

This paper examines how pretest measures of a study outcome reduce selection bias in observational studies in education. The theoretical rationale for privileging pretests in bias control is that they are often highly correlated with the outcome, and in many contexts, they are also highly correlated with the selection process. To examine the pretest's role in bias reduction, we use the data from two within study comparisons and an especially strong quasi-experiment, each with an educational intervention that seeks to improve achievement. In each study, the pretest measures are consistently highly correlated with post-intervention measures of themselves, but the studies vary the correlation between the pretest and the process of selection into treatment. Across the three datasets with two outcomes each, there are three cases where this correlation is low and three where it is high. A single wave of pretest always reduces bias across the six instances examined, and it eliminates bias in three of them. Adding a second pretest wave eliminates bias in two more instances. However, the pattern of bias elimination does not follow the predicted pattern-that more bias reduction ensues as a function of how highly the pretest is correlated with selection. The findings show that bias is more complexly related to the pretest's correlation with selection than we hypothesized, and we seek to explain why.

Attitudes towards assisted suicide and euthanasia among care-dependent older adults (50+) in Austria: the role of socio-demographics, religiosity, physical illness, psychological distress, and social isolation.

BACKGROUND: Care-dependency constitutes an important issue with regard to the approval of end-of-life decisions, yet attitudes towards assisted suicide and euthanasia are understudied among care-dependent older adults. We assessed attitudes towards assisted suicide and euthanasia and tested empirical correlates, including socio-demographics, religiosity, physical illness, psychological distress and social isolation. METHODS: A nationwide cross-sectional survey among older care allowance recipients (50+) in private households in Austria was conducted in 2016. In computer-assisted personal interviews, 493 respondents were asked whether or not they approved of the availability of assisted suicide and euthanasia in case of long-term care dependency and whether or not they would consider using assisted suicide or euthanasia for themselves. Multiple logistic regression analysis was used to assess the impact of potential determinants of attitudes towards assisted suicide and euthanasia. RESULTS: About a quarter (24.8-26.0%) of the sampled care-dependent older adults approved of the availability of assisted suicide and euthanasia respectively indicated the will to (hypothetically) make use of assisted suicide or euthanasia. Attitudes towards assisted suicide were most favourable among care-dependent older adults living in urban areas, those who did not trust physicians, those who reported active suicide ideation, and individuals with a strong fear of dying. With regard to euthanasia, living alone, religiosity and fear of dying were the central determinants of acceptance. CONCLUSIONS: Positive attitudes towards and will to (hypothetically) use assisted suicide and euthanasia were expressed by a substantial minority of care-dependent older adults in Austria and are driven by current psychological suffering and fear of the process of dying in the (near) future. Community-based psychosocial care should be expanded to address psychological distress and fears about end-of-life issues among care-dependent older adults.

Graphical Models for Quasi-experimental Designs.

Randomized controlled trials (RCTs) and quasi-experimental designs like regression discontinuity (RD) designs, instrumental variable (IV) designs, and matching and propensity score (PS) designs are frequently used for inferring causal effects. It is well known that the features of these designs facilitate the identification of a causal estimand and, thus, warrant a causal interpretation of the estimated effect. In this article, we discuss and compare the identifying assumptions of quasi-experiments using causal graphs. The increasing complexity of the causal graphs as one switches from an RCT to RD, IV, or PS designs reveals that the assumptions become stronger as the researcher's control over treatment selection diminishes. We introduce limiting graphs for the RD design and conditional graphs for the latent subgroups of com-pliers, always takers, and never takers of the IV design, and argue that the PS is a collider that offsets confounding bias via collider bias.

Identifying Causal Estimands for Time-Varying Treatments Measured with Time-Varying (Age or Grade-Based) Instruments.

This commentary discusses causal estimands of same-age and same-grade comparisons for assessing grade-retention effects on student ability and performance. Using potential outcomes notation, we show that same-age and same-grade comparisons refer to different retention-promotion contrasts and therefore assess different causal questions. We also comment on deleting versus censoring records of students who dropped out of the study or do not belong to the treatment regimes under investigation. Whereas deleting entire student records potentially induces collider bias, censoring circumvents bias if censoring is ignorable given the observed pretreatment covariates.

Quasi-Experimental Designs for Causal Inference.

When randomized experiments are infeasible, quasi-experimental designs can be exploited to evaluate causal treatment effects. The strongest quasi-experimental designs for causal inference are regression discontinuity designs, instrumental variable designs, matching and propensity score designs, and comparative interrupted time series designs. This article introduces for each design the basic rationale, discusses the assumptions required for identifying a causal effect, outlines methods for estimating the effect, and highlights potential validity threats and strategies for dealing with them. Causal estimands and identification results are formalized with the potential outcomes notations of the Rubin causal model.

The Mechanics of Omitted Variable Bias: Bias Amplification and Cancellation of Offsetting Biases.

Causal inference with observational data frequently requires researchers to estimate treatment effects conditional on a set of observed covariates, hoping that they remove or at least reduce the confounding bias. Using a simple linear (regression) setting with two confounders - one observed (X), the other unobserved (U) - we demonstrate that conditioning on the observed confounder X does not necessarily imply that the confounding bias decreases, even if X is highly correlated with U. That is, adjusting for X may increase instead of reduce the omitted variable bias (OVB). Two phenomena can cause an increasing OVB: (i) bias amplification and (ii) cancellation of offsetting biases. Bias amplification occurs because conditioning on X amplifies any remaining bias due to the omitted confounder U. Cancellation of offsetting biases is an issue whenever X and U induce biases in opposite directions such that they perfectly or partially offset each other, in which case adjusting for X inadvertently cancels the bias-offsetting effect. In this article we discuss the conditions under which adjusting for X increases OVB, and demonstrate that conditioning on X increases the imbalance in U, which turns U into an even stronger confounder. We also show that conditioning on an unreliably measured confounder can remove more bias than the corresponding reliable measure. Practical implications for causal inference will be discussed.

Practice of causal inference with the propensity of being zero or one: assessing the effect of arbitrary cutoffs of propensity scores.

Causal inference methodologies have been developed for the past decade to estimate the unconfounded effect of an exposure under several key assumptions. These assumptions include, but are not limited to, the stable unit treatment value assumption, the strong ignorability of treatment assignment assumption, and the assumption that propensity scores be bounded away from zero and one (the positivity assumption). Of these assumptions, the first two have received much attention in the literature. Yet the positivity assumption has been recently discussed in only a few papers. Propensity scores of zero or one are indicative of deterministic exposure so that causal effects cannot be defined for these subjects. Therefore, these subjects need to be removed because no comparable comparison groups can be found for such subjects. In this paper, using currently available causal inference methods, we evaluate the effect of arbitrary cutoffs in the distribution of propensity scores and the impact of those decisions on bias and efficiency. We propose a tree-based method that performs well in terms of bias reduction when the definition of positivity is based on a single confounder. This tree-based method can be easily implemented using the statistical software program, R. R code for the studies is available online.