Evaluating the internal consistency of subtraction-based and residualized difference scores: Considerations for psychometric reliability analyses of event-related potentials.

In studies of event-related brain potentials (ERPs), difference scores between conditions in a task are frequently used to isolate neural activity for use as a dependent or independent variable. Adequate score reliability is a prerequisite for studies examining relationships between ERPs and external correlates, but there is no extensive treatment on the suitability of the various available approaches to estimating difference score reliability that focus on ERP research. In the present study, we provide formulas from classical test theory and generalizability theory for estimating the internal consistency of subtraction-based and residualized difference scores. These formulas are then applied to error-related negativity (ERN) and reward positivity (RewP) difference scores from the same sample of 117 participants. Analyses demonstrate that ERN difference scores can be reliable, which supports their use in studies of individual differences. However, RewP difference scores yielded poor reliability due to the high correlation between the constituent reward and non-reward ERPs. Findings emphasize that difference score reliability largely depends on the internal consistency of constituent scores and the correlation between those scores. Furthermore, generalizability theory yields more suitable estimates of internal consistency for subtraction-based difference scores than classical test theory. We conclude that ERP difference scores can show adequate reliability and be useful for isolating neural activity in studies of individual differences.

The respiratory occlusion discrimination task: A new paradigm to measure respiratory interoceptive accuracy.

Interoception, or the sense of the internal state of the body, is hypothesized to be essential for a wide range of psychobiological processes and the development and perpetuation of several (mental) health problems. However, the study of interoceptive accuracy, the objectively measured capacity to detect or discriminate conscious bodily signals, has been hampered by the use of tasks with questionable construct validity and is often limited to studying interoception solely in the cardiac domain. We developed a novel task to measure interoceptive accuracy in the respiratory domain, the respiratory occlusion discrimination (ROD) task. In this task, interoceptive accuracy is defined as an individual's ability to detect small differences in lengths of short respiratory occlusions, assessed by means of an adaptive staircase procedure. This article describes a validation study (N = 97) aimed at investigating the internal consistency, test-retest reliability, and discriminant validity of the ROD task. The average just noticeable difference of lengths of respiratory occlusion was 74.22 ms, with large inter-individual variability (SD = 37.1 ms). The results of the validation study indicate acceptable internal consistency (Cronbach's alpha = 0.70), 1-week test-retest reliability (r = 0.53), and discriminant validity, as indicated by a lack of correlation between the ROD task and an auditory discrimination task with identical design (r = 0.18), and a weak correlation with breathing behavior (r = -0.27). The ROD task is a promising novel paradigm to study interoceptive accuracy and its role in various psychobiological processes and disorders.

Psychophysiological responses to treadwall and indoor wall climbing in adult female climbers.

The purpose of the study was to compare the psychophysiological response of climbers of a range of abilities (lower grade to advanced) when ascending identical climbing routes on a climbing wall and a rotating treadwall. Twenty-two female climbers (31.2 ± 9.4 years; 60.5 ± 6.5 kg; 168.6 ± 5.7 cm) completed two identical 18 m climbing trials (graded 4 on the French Sport scale) separated by 1 week, one on the treadwall (climbing low to the ground) and the other on the indoor wall (climbing in height). Indirect calorimetry, venous blood samples and video-analysis were used to assess energy cost, hormonal response and time-load characteristics. Energy costs were higher during indoor wall climbing comparing to those on the treadwall by 16% (P < 0.001, [Formula: see text] = 0.48). No interaction of climbing ability and climbing condition were found. However, there was an interaction for climbing ability and post-climbing catecholamine concentration (P < 0.01, [Formula: see text] = 0.28). Advanced climbers' catecholamine response increased by 238% and 166% with respect to pre-climb values on the treadwall and indoor wall, respectively; while lower grade climbers pre-climb concentrations were elevated by 281% and 376% on the treadwall and indoor wall, respectively. The video analysis showed no differences in any time-motion variables between treadwall and indoor wall climbing. The study demonstrated a greater metabolic response for indoor wall climbing, however, the exact mechanisms are not yet fully understood.

Inconsistent analytic strategies reduce robustness in fear extinction via skin conductance response.

Robustness of fear conditioning and extinction paradigms has become increasingly important for many researchers interested in improving the study of anxiety and trauma disorders. We recently illustrated the wide variability in data analysis techniques in this paradigm, which we argued may result in a lack of robustness. In the current study, we resampled data from six of our own fear acquisition and extinction data sets, with skin conductance as the outcome. In the resampled and original data sets, we found that effect sizes that were calculated using discrepant statistical strategies, sourced from a non-exhaustive search of high-impact articles, were often poorly correlated. The main contributors to poor correlations were the selection of trials from different stages of each experimental phase and the use of average compared to trial-by-trial analysis. These findings reinforce the importance of focusing on robustness in the psychophysiological measurement of fear acquisition and extinction in the laboratory and may guide prospective researchers in which decisions may most impact the robustness of their results.

Stress and cognition: A user's guide to designing and interpreting studies.

Fueling the rapid growth in our understanding of how stress influences cognition, the number of studies examining the effects of stress on various cognitive processes has grown substantially over the last two decades. Despite this growth, few published guidelines exist for designing these studies, and divergent paradigm designs can diminish typical effects of stress or even reverse them. The goal of this review, therefore, is to survey necessary considerations (e.g., validating a stress induction), important considerations (e.g., specifying the timing of the stressor and cognitive task), and best practices (e.g., using Bayesian analyses) when designing a study that aims at least in part to examine the effects of acute stress on some cognitive process or function. These guidelines will also serve to help readers of these studies interpret what may otherwise be very confusing, anomalous results. Designing and interpreting studies with these considerations and practices in mind will help to move the field of stress and cognition forward by clarifying how, exactly, stress influences performance on a given cognitive task in a population of interest.

Controlling the uncontrolled: Are there incidental experimenter effects on physiologic responding?

The degree to which experimenters shape participant behavior has long been of interest in experimental social science research. Here, we extend this question to the domain of peripheral psychophysiology, where experimenters often have direct, physical contact with participants, yet researchers do not consistently test for their influence. We describe analytic tools for examining experimenter effects in peripheral physiology. Using these tools, we investigate nine data sets totaling 1,341 participants and 160 experimenters across different roles (e.g., lead research assistants, evaluators, confederates) to demonstrate how researchers can test for experimenter effects in participant autonomic nervous system activity during baseline recordings and reactivity to study tasks. Our results showed (a) little to no significant variance in participants' physiological reactivity due to their experimenters, and (b) little to no evidence that three characteristics of experimenters that are well known to shape interpersonal interactions-status (using five studies with 682 total participants), gender (using two studies with 359 total participants), and race (in two studies with 554 total participants)-influenced participants' physiology. We highlight several reasons that experimenter effects in physiological data are still cause for concern, including the fact that experimenters in these studies were already restricted on a number of characteristics (e.g., age, education). We present recommendations for examining and reducing experimenter effects in physiological data and discuss implications for replication.

Reliability of Physiological Responses Induced by Basic Emotions: A Pilot Study.

BACKGROUND: Although emotion-specific autonomic responses based on the discrete theory of emotion have been widely studied, studies on the reliability of physiological responses to emotional stimuli are limited. In this study, we aimed to assess the reliability of physiological changes induced by the six basic emotions (happiness, sadness, anger, fear, disgust, and surprise) that were measured during 10 weekly repeated experiments. METHODS: Twelve college students participated, and in each experiment, physiological signals were collected before and while participants were watching emotion-provoking film clips. Additionally, the participants self-evaluated the emotions that they experienced during the film presentation at the end of each emotional stimulus. To avoid adaptation of participants to identical stimuli during repeated measurements, we used 10 different film clips for each emotion, and thus a total of 60 film clips over 10 weeks were used. Physiological features, such as skin conductance level (SCL), fingertip temperature (FT), heart rate (HR), and blood volume pulse (BVP), were extracted from the physiological signals. Two reliability indices, Cronbach's alpha and intraclass correlation coefficient, were calculated from the physiological features to assess internal consistency and interrater reliability, respectively. RESULTS: We found that SCL, HR, and BVP measured during the emotion-provoking phase over the 10 weekly sessions were more reliable than those assessed at baseline. Furthermore, SCL, HR, and BVP from the emotion-provoking phase exhibited excellent internal consistency and interrater reliability. CONCLUSIONS: Our findings suggest that these features can be used as reliable physiological indices in emotion studies. The results also support the significance of physiological signals as meaningful indicators for emotion recognition in HCI (human computer interface) area.

Methodological reporting behavior, sample sizes, and statistical power in studies of event-related potentials: Barriers to reproducibility and replicability.

Methodological reporting guidelines for studies of ERPs were updated in Psychophysiology in 2014. These guidelines facilitate the communication of key methodological parameters (e.g., preprocessing steps). Failing to report key parameters represents a barrier to replication efforts, and difficulty with replicability increases in the presence of small sample sizes and low statistical power. We assessed whether guidelines are followed and estimated the average sample size and power in recent research. Reporting behavior, sample sizes, and statistical designs were coded for 150 randomly sampled articles from five high-impact journals that frequently published ERP research from 2011 to 2017. An average of 63% of guidelines were reported, and reporting behavior was similar across journals, suggesting that gaps in reporting is a shortcoming of the field rather than any specific journal. Publication of the guidelines article had no impact on reporting behavior, suggesting that editors and peer reviewers are not enforcing these recommendations. The average sample size per group was 21. Statistical power was conservatively estimated as .72-.98 for a large effect size, .35-.73 for a medium effect, and .10-.18 for a small effect. These findings indicate that failing to report key guidelines is ubiquitous and that ERP studies are primarily powered to detect large effects. Such low power and insufficient following of reporting guidelines represent substantial barriers to replication efforts. The methodological transparency and replicability of studies can be improved by the open sharing of processing code and experimental tasks and by a priori sample size calculations to ensure adequately powered studies.

Critical evaluation of current data analysis strategies for psychophysiological measures of fear conditioning and extinction in humans.

Fear conditioning and extinction is a construct integral to understanding trauma-, stress- and anxiety-related disorders. In the laboratory, associative learning paradigms that pair aversive with neutral stimuli are used as analogues to real-life fear learning. These studies use physiological indices, such as skin conductance, to sensitively measure rates and intensity of learning and extinction. In this review, we discuss some of the potential limitations in interpreting and analysing physiological data during the acquisition or extinction of conditioned fear. We argue that the utmost attention should be paid to the development of modelling approaches of physiological data in associative learning paradigms, by illustrating the lack of replicability and interpretability of results in current methods. We also show that statistical significance may be easily achieved in this paradigm without more stringent data and data analysis reporting requirements, leaving this particular field vulnerable to misleading conclusions. This review is written so that issues and potential solutions are accessible to researchers without mathematical training. We conclude the review with some suggestions that all laboratories should be able to implement, including visualising the full data set in publications and adopting modelling, or at least regression-based, approaches.

Controlling for caffeine in cardiovascular research: A critical review.

Caffeine, the most widely consumed drug in the world, exerts numerous effects on cardiovascular activity. Thus, it is important and advisable to control for caffeine consumption in studies examining caffeine and/or cardiovascular activity and reactivity. This paper 1) reviews the literature concerning caffeine's effects on cardiovascular parameters; 2) summarizes the widely varying protocols used to control for the drug in extant cardiovascular literature, and 3) provide guidelines for caffeine control procedures to minimize potentially confounding acute and withdrawal effects of the drug. An abstention period equal to the average half-life of the drug is recommended for creation of methodological controls for caffeine. Additional methodological recommendations are described concerning factors that moderate the half-life of caffeine. When feasible, researchers should consider and aim to control for caffeine's acute and extended psychophysiological effects. This understudied issue has fundamental implications for caffeine-related investigations and research in psychophysiology and behavioral medicine.

Principles behind variance misallocation in temporal exploratory factor analysis for ERP data: Insights from an inter-factor covariance decomposition.

Temporal exploratory factor analysis (EFA) is commonly applied to ERP data sets to reduce their dimensionality and the ambiguity with respect to the underlying components. However, the risk of variance misallocation (i.e., the incorrect allocation of condition effects) has raised concerns with regard to EFA usage. Here, we show that variance misallocation occurs because of biased factor covariance estimates and the temporal overlap between the underlying components. We also highlight the consequences of our findings for the analysis of ERP data with EFA. For example, a direct consequence of our expositions is that researchers should use oblique rather than orthogonal rotations, especially when the factors have a substantial topographic overlap. A Monte Carlo simulation confirms our results by showing, for instance, that characteristic biases occur only for orthogonal Varimax rotation but not for oblique rotation methods such as Geomin or Promax. We discuss the practical implications of our results and outline some questions for future research.

Interpretation of physiological indicators of motivation: Caveats and recommendations.

Motivation scientists employing physiological measures to gather information about motivation-related states are at risk of committing two fundamental errors: overstating the inferences that can be drawn from their physiological measures and circular reasoning. We critically discuss two complementary approaches, Cacioppo and colleagues' model of psychophysiological relations and construct validation theory, to highlight the conditions under which these errors are committed and provide guidance on how to avoid them. In particular, we demonstrate that the direct inference from changes in a physiological measure to changes in a motivation-related state requires the demonstration that the measure is not related to other relevant psychological states. We also point out that circular reasoning can be avoided by separating the definition of the motivation-related state from the hypotheses that are empirically tested.

How many blinks are necessary for a reliable startle response? A test using the NPU-threat task.

Emotion-modulated startle is a frequently used method in affective science. Although there is a growing literature on the reliability of this measure, it is presently unclear how many startle responses are necessary to obtain a reliable signal. The present study therefore evaluated the reliability of startle responding as a function of number of startle responses (NoS) during a widely used threat-of-shock paradigm, the NPU-threat task, in a clinical (N=205) and non-clinical (N=92) sample. In the clinical sample, internal consistency was also examined independently for healthy controls vs. those with panic disorder and/or major depression and retest reliability was assessed as a function of NoS. Although results varied somewhat by diagnosis and for retest reliability, the overall pattern of results suggested that six startle responses per condition were necessary to obtain acceptable reliability in clinical and non-clinical samples during this threat-of-shock paradigm in the present study.

Introduction to the special issue on recentering science: Replication, robustness, and reproducibility in psychophysiology.

In recent years, the psychological and behavioral sciences have increased efforts to strengthen methodological practices and publication standards, with the ultimate goal of enhancing the value and reproducibility of published reports. These issues are especially important in the multidisciplinary field of psychophysiology, which yields rich and complex data sets with a large number of observations. In addition, the technological tools and analysis methods available in the field of psychophysiology are continually evolving, widening the array of techniques and approaches available to researchers. This special issue presents articles detailing rigorous and systematic evaluations of tasks, measures, materials, analysis approaches, and statistical practices in a variety of subdisciplines of psychophysiology. These articles highlight challenges in conducting and interpreting psychophysiological research and provide data-driven, evidence-based recommendations for overcoming those challenges to produce robust, reproducible results in the field of psychophysiology.

The science pendulum: From programmatic to incremental-and back?

The climate in which scientific research is conducted changes over time, and in recent years there has been a shift from a positive view of programmatic science to a more negative evaluation that its contribution to scientific progress is only incremental. In this special issue focusing on the tools of a programmatic approach-replication, reliability and reproducibility-I reflect on changes in scientific practice over my research career, considering some factors contributing to changes in emphasis and highlighting potential pitfalls, particularly in terms of the impact on scientific progress and future scientists. In concluding, I suggest that, as members of the scientific community, we can influence current scientific practices in our day-to-day roles as authors, reviewers, investigators, editors, employers, and educators.

Improving the rigor of psychophysiology research.

Psychology as a field is in the midst of what is sometimes called a "crisis" because false findings are prevalent. Although the focus of the methodological and substantive criticisms of psychology has focused on social psychology, psychophysiology research is not without its problems. The author discusses (a) researcher flexibility and its impact on the stability of conclusions and (b) the role power plays in the probability that a finding is true and the precision of estimates. The author uses examples and data from psychophysiological research to illustrate the problems. The author concludes with a discussion of ways to shift the practice of science to improve the reliability of findings. Suggestions for improvement include: increased power through collaboration, improved statistical and methodological training, pre-registration of studies, improved reporting standards, and shifting incentives surrounding hiring and promotion.

Psychometric considerations in the measurement of event-related brain potentials: Guidelines for measurement and reporting.

Failing to consider psychometric issues related to reliability and validity, differential deficits, and statistical power potentially undermines the conclusions of a study. In research using event-related brain potentials (ERPs), numerous contextual factors (population sampled, task, data recording, analysis pipeline, etc.) can impact the reliability of ERP scores. The present review considers the contextual factors that influence ERP score reliability and the downstream effects that reliability has on statistical analyses. Given the context-dependent nature of ERPs, it is recommended that ERP score reliability be formally assessed on a study-by-study basis. Recommended guidelines for ERP studies include 1) reporting the threshold of acceptable reliability and reliability estimates for observed scores, 2) specifying the approach used to estimate reliability, and 3) justifying how trial-count minima were chosen. A reliability threshold for internal consistency of at least 0.70 is recommended, and a threshold of 0.80 is preferred. The review also advocates the use of generalizability theory for estimating score dependability (the generalizability theory analog to reliability) as an improvement on classical test theory reliability estimates, suggesting that the latter is less well suited to ERP research. To facilitate the calculation and reporting of dependability estimates, an open-source Matlab program, the ERP Reliability Analysis Toolbox, is presented.

ERP Reliability Analysis (ERA) Toolbox: An open-source toolbox for analyzing the reliability of event-related brain potentials.

Generalizability theory (G theory) provides a flexible, multifaceted approach to estimating score reliability. G theory's approach to estimating score reliability has important advantages over classical test theory that are relevant for research using event-related brain potentials (ERPs). For example, G theory does not require parallel forms (i.e., equal means, variances, and covariances), can handle unbalanced designs, and provides a single reliability estimate for designs with multiple sources of error. This monograph provides a detailed description of the conceptual framework of G theory using examples relevant to ERP researchers, presents the algorithms needed to estimate ERP score reliability, and provides a detailed walkthrough of newly-developed software, the ERP Reliability Analysis (ERA) Toolbox, that calculates score reliability using G theory. The ERA Toolbox is open-source, Matlab software that uses G theory to estimate the contribution of the number of trials retained for averaging, group, and/or event types on ERP score reliability. The toolbox facilitates the rigorous evaluation of psychometric properties of ERP scores recommended elsewhere in this special issue.

Diversity and representation: Key issues for psychophysiological science.

This Special Issue is devoted to the illustration and discussion of three key demographic variables (sex/gender, race/ethnicity, and socioeconomic status) that have been shown to moderate associations between psychophysiological processes and behavior. The introduction to the issue discusses the role of phenotypic plasticity in the emergence of different neural processes that achieve the same behavioral outcome, with emphasis on how these relatively stable developmental contexts affect brain/behavior associations without necessarily resulting in difference in behavior. These findings have profound significance for the implications of generalization and call into question the presumption that diverse samples produce an average result that is appropriately reflective of the individuals themselves. Increasing diversity within psychophysiological research is critical in elucidating mechanisms by which the human brain can accomplish cognitive and affective behaviors. This article further examines the logistical and ethical challenges faced in achieving this goal.

The embodied brain.

In this editorial, I discuss the advantages of interpreting brain data in the context of other bodily systems. I also discuss current research challenges that may greatly benefit from a psychophysiological approach in which multiple methods--both peripheral and central--are used to improve our understanding of brain function, its underlying physiology, and its relationship to psychological constructs. In closing, I summarize the major changes and policy highlights regarding papers published in Psychophysiology.