In models we trust: preregistration, large samples, and replication may not suffice.

Despite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation process? In this article, we address both issues and show, in a first step, that in the face of selective samples and contrary to common practice, the validity of inferences, even when based on experimental designs, can be claimed without further justification and adaptation of standard methods only in very specific situations. We then broaden our perspective to discuss consequences of violated assumptions in linear models in the context of psychological research in general and in generalized linear mixed models as used in item response theory. These types of misspecification are oftentimes ignored in the psychological research literature. It is emphasized that the above problems cannot be overcome by strategies such as preregistration, large samples, replications, or a ban on testing null hypotheses. To avoid biased conclusions, we briefly discuss tools such as model diagnostics, statistical methods to compensate for selectivity and semi- or non-parametric estimation. At a more fundamental level, however, a twofold strategy seems indispensable: (1) iterative, cumulative theory development based on statistical methods with theoretically justified assumptions, and (2) empirical research on variables that affect (self-) selection into the observed part of the sample and the use of this information to compensate for selectivity.

On Reverse Shrinkage Effects and Shrinkage Overshoot.

Given a squared Euclidean norm penalty, we examine some less well-known properties of shrinkage estimates. In particular, we highlight that it is possible for some components of the shrinkage estimator to be placed further away from the prior mean than the original estimate. An analysis of this effect is provided within three different modeling settings-encompassing linear, logistic, and ordinal regression models. Additional simulations show that the outlined effect is not a mathematical artefact, but likely to occur in practice. As a byproduct, they also highlight the possibilities of sign reversals ("overshoots") for shrinkage estimates. We point out practical consequences and challenges, which might arise from the observed effects with special emphasis on psychometrics.

The counterintuitive impact of responses and response times on parameter estimates in the drift diffusion model.

Given a drift diffusion model with unknown drift and boundary parameters, we analyse the behaviour of maximum likelihood estimates with respect to changes of responses and response times. It is shown analytically that a single fast response time can dominate the estimation in that no matter how many correct answers a test taker provides, the estimate of the drift (ability) parameter decreases to zero. In addition, it is shown that although higher drift rates imply shorter response times, the reverse implication does not hold for the estimates: shorter response times can decrease the drift rate estimate. In the light of these analytical results, we illustrate the actual impact of the findings in a small simulation for a mental rotation test. The method of analysis outlined is applicable to a broader range of models, and we emphasize the need to further check currently used reaction time models within this framework.

Test-retest reliability of the emotional enhancement of memory.

Emotionally arousing stimuli are usually better remembered than neutral ones. This effect can be observed immediately after encoding and becomes more robust after a period of consolidation. The magnitude of this effect in an individual has been treated in various research contexts implicitly as reliable and temporally stable. However, we recently observed in 69 participants that an individual's memory advantage for negative over neutral stimuli, whether immediate or delayed, was very weakly correlated with the advantage measured after 3.5 years, albeit with slightly different memory paradigms. In the current study, we tested whether the test-retest reliability of these emotional memory effects might be larger if the temporal lapse between tests was shorter (10 weeks) and more similar memory tests were used. We observed that the better memory for emotional stimuli is highly replicable on the group level. However, the retest reliability on the individual level was very low. We replicated these findings by re-analysing data from a previous study where female participants took emotional memory tests at two different points of their menstrual cycle. We conclude, therefore, that the individual emotional enhancement of memory is not stable or that it cannot be measured reliably with the standard emotional memory paradigm.

Rethinking the Interpretation of Item Discrimination and Factor Loadings.

Factor loadings and item discrimination parameters play a key role in scale construction. A multitude of heuristics regarding their interpretation are hardwired into practice-for example, neglecting low loadings and assigning items to exactly one scale. We challenge the common sense interpretation of these parameters by providing counterexamples and general results which altogether cast doubt on our understanding of these parameters. In particular, we highlight the counterintuitive way in which the best prediction of a test taker's latent ability depends on the factor loadings. As a consequence, we emphasize that practitioners need to shift their focus from interpreting item discrimination parameters by their relative loading to an interpretation which incorporates the structure of the model-based latent ability estimate.

Simplified Estimation and Testing in Unbalanced Repeated Measures Designs.

In this paper we propose a simple estimator for unbalanced repeated measures design models where each unit is observed at least once in each cell of the experimental design. The estimator does not require a model of the error covariance structure. Thus, circularity of the error covariance matrix and estimation of correlation parameters and variances are not necessary. Together with a weak assumption about the reason for the varying number of observations, the proposed estimator and its variance estimator are unbiased. As an alternative to confidence intervals based on the normality assumption, a bias-corrected and accelerated bootstrap technique is considered. We also propose the naive percentile bootstrap for Wald-type tests where the standard Wald test may break down when the number of observations is small relative to the number of parameters to be estimated. In a simulation study we illustrate the properties of the estimator and the bootstrap techniques to calculate confidence intervals and conduct hypothesis tests in small and large samples under normality and non-normality of the errors. The results imply that the simple estimator is only slightly less efficient than an estimator that correctly assumes a block structure of the error correlation matrix, a special case of which is an equi-correlation matrix. Application of the estimator and the bootstrap technique is illustrated using data from a task switch experiment based on an experimental within design with 32 cells and 33 participants.

On Fair Person Classification Based on Efficient Factor Score Estimates in the Multidimensional Factor Analysis Model.

Since Hooker, Finkelman and Schwartzman (Psychometrika 74(3): 419-442, 2009) it is known that person parameter estimates from multidimensional latent variable models can induce unfair classifications via paradoxical scoring effects. The open question as to whether there is a fair and at the same time multidimensional scoring scheme with adequate statistical properties is addressed in this paper. We develop a theorem on the existence of a fair, multidimensional classification scheme in the context of the classical linear factor analysis model and show how the computation of the scoring scheme can be embedded in the context of linear programming. The procedure is illustrated in the framework of scoring the Wechsler Adult Intelligence Scale (WAIS-IV).

Duration of untreated illness in patients with somatoform disorders.

OBJECTIVE: A long duration of untreated mental illness (DUI) has been found to be associated with negative long-term outcomes. Although somatic symptom and related disorders are frequent in the general population and in primary care, data regarding the DUI of these disorders is scarce. The aim of this study was to investigate the DUI in patients with somatoform disorders. METHODS: In a cross-sectional study, primary care patients at high risk of having a somatoform disorder were identified using the Patient Health Questionnaire (PHQ). In a second step, life-time somatoform disorder diagnosis was established using the Composite International Diagnostic Interview (CIDI). Additionally, DUI was retrospectively assessed via self-reporting and sociodemographic information was collected. Survival analysis was used to estimate the DUI and to identify patient-related predictors of DUI. RESULTS: A total of 139 patients with somatoform disorders were included in the analyses. The mean DUI in these patients was 25.2years (median 23.1years). Higher education significantly predicted shorter DUI, whereas gender and age of onset were unrelated to DUI. CONCLUSIONS: The results reveal a substantial delay in adequate treatment of patients with somatoform disorders. The reported DUI emphasizes the importance of improvements in the management of patients with these disorders.

Predicting suicidal ideation in primary care: An approach to identify easily assessable key variables.

OBJECTIVE: To obtain predictors of suicidal ideation, which can also be used for an indirect assessment of suicidal ideation (SI). To create a classifier for SI based on variables of the Patient Health Questionnaire (PHQ) and sociodemographic variables, and to obtain an upper bound on the best possible performance of a predictor based on those variables. METHODS: From a consecutive sample of 9025 primary care patients, 6805 eligible patients (60% female; mean age = 51.5 years) participated. Advanced methods of machine learning were used to derive the prediction equation. Various classifiers were applied and the area under the curve (AUC) was computed as a performance measure. RESULTS: Classifiers based on methods of machine learning outperformed ordinary regression methods and achieved AUCs around 0.87. The key variables in the prediction equation comprised four items - namely feelings of depression/hopelessness, low self-esteem, worrying, and severe sleep disturbances. The generalized anxiety disorder scale (GAD-7) and the somatic symptom subscale (PHQ-15) did not enhance prediction substantially. CONCLUSIONS: In predicting suicidal ideation researchers should refrain from using ordinary regression tools. The relevant information is primarily captured by the depression subscale and should be incorporated in a nonlinear model. For clinical practice, a classification tree using only four items of the whole PHQ may be advocated.

A New Explanation and Proof of the Paradoxical Scoring Results in Multidimensional Item Response Models.

In multidimensional item response models, paradoxical scoring effects can arise, wherein correct answers are penalized and incorrect answers are rewarded. For the most prominent class of IRT models, the class of linearly compensatory models, a general derivation of paradoxical scoring effects based on the geometry of item discrimination vectors is given, which furthermore corrects an error in an established theorem on paradoxical results. This approach highlights the very counterintuitive way in which item discrimination parameters (and also factor loadings) have to be interpreted in terms of their influence on the latent ability estimate. It is proven that, despite the error in the original proof, the key result concerning the existence of paradoxical effects remains true-although the actual relation to the item parameters is shown to be a more complicated function than previous results suggested. The new proof enables further insights into the actual mathematical causation of the paradox and generalizes the findings within the class of linearly compensatory models.

Psychometric analysis of the Generalized Anxiety Disorder scale (GAD-7) in primary care using modern item response theory.

OBJECTIVE: The Generalized Anxiety Disorder scale (GAD-7) is one of the most frequently used diagnostic self-report scales for screening, diagnosis and severity assessment of anxiety disorder. Its psychometric properties from the view of the Item Response Theory paradigm have rarely been investigated. We aimed to close this gap by analyzing the GAD-7 within a large sample of primary care patients with respect to its psychometric properties and its implications for scoring using Item Response Theory. METHODS: Robust, nonparametric statistics were used to check unidimensionality of the GAD-7. A graded response model was fitted using a Bayesian approach. The model fit was evaluated using posterior predictive p-values, item information functions were derived and optimal predictions of anxiety were calculated. RESULTS: The sample included N = 3404 primary care patients (60% female; mean age, 52,2; standard deviation 19.2) The analysis indicated no deviations of the GAD-7 scale from unidimensionality and a decent fit of a graded response model. The commonly suggested ultra-brief measure consisting of the first two items, the GAD-2, was supported by item information analysis. The first four items discriminated better than the last three items with respect to latent anxiety. CONCLUSION: The information provided by the first four items should be weighted more heavily. Moreover, estimates corresponding to low to moderate levels of anxiety show greater variability. The psychometric validity of the GAD-2 was supported by our analysis.

The Somatic Symptom Disorder - B Criteria Scale (SSD-12): Factorial structure, validity and population-based norms.

PURPOSE: The Somatic Symptom Disorder - B Criteria Scale (SSD-12) assesses the psychological features of DSM-5 Somatic Symptom Disorder (SSD). The present study investigates the dimensionality and psychometric properties in a general population sample and provides norm values. METHOD: Test dimensionality was evaluated via confirmatory factor analysis and nonparametric item response theory. Correlational analyses and logistic regression models based on related measures (SSS 8, PHQ-2, GAD-2, Health Care Utilization) were used to derive predictive validity. Age and gender specific norms were derived via quantile regression. RESULTS: The SSD-12 has good item characteristics and excellent reliability (Cronbach's α=0.95). Confirmatory factor analyses revealed a high correlation between the three proposed psychological subscales interpreted as cognitive, affective and behavioral aspects, indicating a general factor model of the SSD-12 in the general population (n=2362, CFI=0.99, TLI=0.998, RMSEA=0.09, 90% CI: 0.09-0.1). SSD-12 total sum-score was significantly associated with somatic symptom burden (r=0.73, p<0.001), general anxiety (r=0.63, p<0.001) and depressive symptoms (r=0.64, p<0.001). Patients with a higher SSD-12 symptom burden reported higher general physical and mental health impairment and a significantly higher health care use. CONCLUSION: The SSD-12 is a reliable and valid self-report measure of the psychological characteristics of DSM-5 Somatic Symptom Disorder. The provided norms enable researchers and clinicians to compare SSD-12 scores with reference values of a general population sample.

Item response analysis of a shortened German version of the morningness-eveningness scale.

A shortened version of the German adaptation of the morningness-eveningness scale of Horne and Östberg is analysed within a large sample of 994 physicians with respect to dimensionality, reliability, gender differences and validity. The psychometric analysis - which incorporates a highly robust method to check for unidimensionality - shows discrepancies towards unidimensionality and highlights three misfitting items. In addition, hypothesis testing indicates the presence of differential item functioning (DIF) with respect to gender which could be caused by differences in response formats. Although, reliability estimates are satisfactory, an overall lack of adequate psychometric properties of the scale within the population of physicians has to be reported. We derive suggestions for improvement of the original morningness-eveningness questionnaire (MEQ)-scale and provide general comments on how to check for unidimensionality without imposing a restrictive response model.

Sleep apnea syndrome: improved detection of respiratory events and cortical arousals using oxymetry pulse wave amplitude during polysomnography.

Respiratory events (RE) during sleep induce cortical arousals (A) and marked changes in autonomic markers in sleep apnea syndrome (SAS). The aims of the study were double. First, we assessed whether pulse wave amplitude (PWA) added to polysomnography (PSG) could improve RE and A detection; second, we wanted to know whether the quality of detection of these two parameters could be improved using PWA. Respiratory disturbance index (RDI) and A were randomly scored twice by the same observer in 12 male patients with SAS. The first scoring was done using conventional PSG signals, the second scoring adding PWA to PSG. We also measured interobserver agreement by randomly selecting and reading 100 PSG sequences of 5 min with and without PWA by two observers. Adding PWA to PSG parameters allowed to detect significantly more RDI (53.9 +/- 21.6 h(-1) versus 48.3 +/- 22.3 h(-1), p < 0.001) and more A (68.0 +/- 14.4 versus 59.4 +/- 16.5, p < 0.001). Moreover, after using PWA, there was no significant disagreement between two observers for detecting RE, showing better quality of RE detection. PWA is a simple and cheap parameter that improves the diagnostic value of conventional PSG in sleep apnea syndrome by better detecting respiratory events and A.