The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses.

Active learning has become an increasingly popular method for screening large amounts of data in systematic reviews and meta-analyses. The active learning process continually improves its predictions on the remaining unlabeled records, with the goal of identifying all relevant records as early as possible. However, determining the optimal point at which to stop the active learning process is a challenge. The cost of additional labeling of records by the reviewer must be balanced against the cost of erroneous exclusions. This paper introduces the SAFE procedure, a practical and conservative set of stopping heuristics that offers a clear guideline for determining when to end the active learning process in screening software like ASReview. The eclectic mix of stopping heuristics helps to minimize the risk of missing relevant papers in the screening process. The proposed stopping heuristic balances the costs of continued screening with the risk of missing relevant records, providing a practical solution for reviewers to make informed decisions on when to stop screening. Although active learning can significantly enhance the quality and efficiency of screening, this method may be more applicable to certain types of datasets and problems. Ultimately, the decision to stop the active learning process depends on careful consideration of the trade-off between the costs of additional record labeling against the potential errors of the current model for the specific dataset and context.

Addressing the challenges of reconstructing systematic reviews datasets: a case study and a noisy label filter procedure.

Systematic reviews and meta-analyses typically require significant time and effort. Machine learning models have the potential to enhance screening efficiency in these processes. To effectively evaluate such models, fully labeled datasets-detailing all records screened by humans and their labeling decisions-are imperative. This paper presents the creation of a comprehensive dataset for a systematic review of treatments for Borderline Personality Disorder, as reported by Oud et al. (2018) for running a simulation study. The authors adhered to the PRISMA guidelines and published both the search query and the list of included records, but the complete dataset with all labels was not disclosed. We replicated their search and, facing the absence of initial screening data, introduced a Noisy Label Filter (NLF) procedure using active learning to validate noisy labels. Following the NLF application, no further relevant records were found. A simulation study employing the reconstructed dataset demonstrated that active learning could reduce screening time by 82.30% compared to random reading. The paper discusses potential causes for discrepancies, provides recommendations, and introduces a decision tree to assist in reconstructing datasets for the purpose of running simulation studies.

Which individual, social, and urban factors in early childhood predict psychopathology in later childhood, adolescence and young adulthood? A systematic review.

Background: A comprehensive picture is lacking of the impact of early childhood (age 0-5) risk factors on the subsequent development of mental health symptoms. Objective: In this systematic review, we investigated which individual, social and urban factors, experienced in early childhood, contribute to the development of later anxiety and depression, behavioural problems, and internalising and externalising symptoms in youth. Methods: Embase, MEDLINE, Scopus, and PsycInfo were searched on the 5th of January 2022. Three additional databases were retrieved from a mega-systematic review source that focused on the identification of both risk and protective indicators for the onset and maintenance of prospective depressive, anxiety and substance use disorders. A total of 46,450 records were identified and screened in ASReview, an AI-aided systematic review tool. We included studies with experimental, quasi-experimental, prospective and longitudinal study designs, while studies that focused on biological and genetical factors, were excluded. Results: Twenty studies were included. The majority of studies explored individual-level risk factors (N = 16). Eleven studies also explored social risk factors and three studied urban risk factors. We found evidence for early predictors relating to later psychopathology measures (i.e., anxiety and depression, behavioural problems, and internalising and externalising symptoms) in childhood, adolescence and early adulthood. These were: parental psychopathology, exposure to parental physical and verbal violence and social and neighbourhood disadvantage. Conclusions: Very young children are exposed to a complex mix of risk factors, which operate at different levels and influence children at different time points. The urban environment appears to have an effect on psychopathology but it is understudied compared to individual-level factors. Moreover, we need more research exploring the interaction between individual, social and urban factors.

Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records.

BACKGROUND: Conducting a systematic review demands a significant amount of effort in screening titles and abstracts. To accelerate this process, various tools that utilize active learning have been proposed. These tools allow the reviewer to interact with machine learning software to identify relevant publications as early as possible. The goal of this study is to gain a comprehensive understanding of active learning models for reducing the workload in systematic reviews through a simulation study. METHODS: The simulation study mimics the process of a human reviewer screening records while interacting with an active learning model. Different active learning models were compared based on four classification techniques (naive Bayes, logistic regression, support vector machines, and random forest) and two feature extraction strategies (TF-IDF and doc2vec). The performance of the models was compared for six systematic review datasets from different research areas. The evaluation of the models was based on the Work Saved over Sampling (WSS) and recall. Additionally, this study introduces two new statistics, Time to Discovery (TD) and Average Time to Discovery (ATD). RESULTS: The models reduce the number of publications needed to screen by 91.7 to 63.9% while still finding 95% of all relevant records (WSS@95). Recall of the models was defined as the proportion of relevant records found after screening 10% of of all records and ranges from 53.6 to 99.8%. The ATD values range from 1.4% till 11.7%, which indicate the average proportion of labeling decisions the researcher needs to make to detect a relevant record. The ATD values display a similar ranking across the simulations as the recall and WSS values. CONCLUSIONS: Active learning models for screening prioritization demonstrate significant potential for reducing the workload in systematic reviews. The Naive Bayes + TF-IDF model yielded the best results overall. The Average Time to Discovery (ATD) measures performance of active learning models throughout the entire screening process without the need for an arbitrary cut-off point. This makes the ATD a promising metric for comparing the performance of different models across different datasets.

Active learning-based systematic reviewing using switching classification models: the case of the onset, maintenance, and relapse of depressive disorders.

Introduction: This study examines the performance of active learning-aided systematic reviews using a deep learning-based model compared to traditional machine learning approaches, and explores the potential benefits of model-switching strategies. Methods: Comprising four parts, the study: 1) analyzes the performance and stability of active learning-aided systematic review; 2) implements a convolutional neural network classifier; 3) compares classifier and feature extractor performance; and 4) investigates the impact of model-switching strategies on review performance. Results: Lighter models perform well in early simulation stages, while other models show increased performance in later stages. Model-switching strategies generally improve performance compared to using the default classification model alone. Discussion: The study's findings support the use of model-switching strategies in active learning-based systematic review workflows. It is advised to begin the review with a light model, such as Naïve Bayes or logistic regression, and switch to a heavier classification model based on a heuristic rule when needed.

Measurement invariance in the social sciences: Historical development, methodological challenges, state of the art, and future perspectives.

This review summarizes the current state of the art of statistical and (survey) methodological research on measurement (non)invariance, which is considered a core challenge for the comparative social sciences. After outlining the historical roots, conceptual details, and standard procedures for measurement invariance testing, the paper focuses in particular on the statistical developments that have been achieved in the last 10 years. These include Bayesian approximate measurement invariance, the alignment method, measurement invariance testing within the multilevel modeling framework, mixture multigroup factor analysis, the measurement invariance explorer, and the response shift-true change decomposition approach. Furthermore, the contribution of survey methodological research to the construction of invariant measurement instruments is explicitly addressed and highlighted, including the issues of design decisions, pretesting, scale adoption, and translation. The paper ends with an outlook on future research perspectives.

Sex-differential PTSD symptom trajectories across one year following suspected serious injury.

Background: Recent years have shown an increased application of prospective trajectory-oriented approaches to posttraumatic stress disorder (PTSD). Although women are generally considered at increased PTSD risk, sex and gender differences in PTSD symptom trajectories have not yet been extensively studied. Objective: To perform an in-depth investigation of differences in PTSD symptom trajectories across one-year post-trauma between men and women, by interpreting the general trends of trajectories observed in sex-disaggregated samples, and comparing within-trajectory symptom course and prevalence rates. Method: We included N = 554 participants (62.5% men, 37.5% women) from a multi-centre prospective cohort of emergency department patients with suspected severe injury. PTSD symptom severity was assessed at 1, 3, 6, and 12 months post-trauma, using the Clinician-Administered PTSD Scale for DSM-IV. Latent growth mixture modelling on longitudinal PTSD symptoms was performed within the sex-disaggregated whole samples. Bayesian modelling with informative priors was applied for reliable model estimation, considering the imbalanced prevalence of the expected latent trajectories. Results: In terms of general trends, the same trajectories were observed for men and women, i.e. resilient, recovery, chronic symptoms and delayed onset. Within-trajectory symptom courses were largely comparable, but resilient women had higher symptoms than resilient men. Sex differences in prevalence rates were observed for the recovery (higher in women) and delayed onset (higher in men) trajectories. Model fit for the sex-disaggregated samples was better than for the whole sample, indicating preferred application of sex-disaggregation. Analyses within the whole sample led to biased estimates of overall and sex-specific trajectory prevalence rates. Conclusions: Sex-disaggregated trajectory analyses revealed limited sex differences in PTSD symptom trajectories within one-year post-trauma in terms of general trends, courses and prevalence rates. The observed biased trajectory prevalence rates in the whole sample emphasize the necessity to apply appropriate statistical techniques when conducting sex-sensitive research.

Antecedentes: Los últimos años han demostrado una mayor aplicación de enfoques prospectivos orientados a la trayectoria para el trastorno de estrés postraumático (TEPT). Aunque generalmente se considera que las mujeres tienen un mayor riesgo de TEPT, las diferencias de sexo y género en las trayectorias de los síntomas del TEPT aún no se han estudiado ampliamente.Objetivo: Realizar una investigación en profundidad de las diferencias en las trayectorias de los síntomas del TEPT a lo largo de un año después de un trauma entre hombres y mujeres, interpretando las tendencias generales de las trayectorias observadas en muestras desagregadas por sexo, así como comparar el curso y la evolución de los síntomas dentro de la trayectoria y las tasas de prevalencia.Método: Incluimos N = 554 participantes (62.5% hombres, 37.5% mujeres) de una cohorte prospectiva multicéntrica de pacientes del servicio de urgencias con sospecha de lesión grave. La gravedad de los síntomas del TEPT se evaluó 1, 3, 6 y 12 meses después del trauma, utilizando la Escala de TEPT administrada por un médico para el DSM-IV. Se realizó un modelo de mezcla de crecimiento latente sobre los síntomas longitudinales de TEPT en las muestras desagregadas por sexo y en la muestra completa. Se aplicó un modelo bayesiano con antecedentes informativos para una estimación confiable del modelo, considerando la prevalencia desequilibrada de las trayectorias latentes esperadas.Resultados: En términos de tendencias generales, se observaron las mismas trayectorias para hombres y mujeres, es decir, resiliente, recuperación, síntomas crónicos y aparición tardía. Los cursos de síntomas dentro de la trayectoria fueron en gran medida comparables, pero las mujeres resilientes tenían más síntomas másque los hombres resilientes. Se observaron diferencias por sexo en las tasas de prevalencia para las trayectorias de recuperación (mayor en mujeres) y de inicio tardío (mayor en hombres). El ajuste del modelo para las muestras desagregadas por sexo fue mejor que para la muestra completa, lo que indica la aplicación preferida de la desagregación por sexo. Los análisis de la muestra completa llevaron a estimaciones sesgadas de las tasas de prevalencia de trayectorias generales y específicas por sexo.Conclusiones: Los análisis de trayectoria desagregados por sexo revelaron diferencias limitadas entre los sexos en las trayectorias de los síntomas del TEPT durante el año posterior al trauma en términos de tendencias generales, cursos y tasas de prevalencia. Las tasas de prevalencia de trayectoria sesgada observadas en el conjunto de la muestra enfatizan la necesidad de aplicar técnicas estadísticas apropiadas al realizar investigaciones que tengan en cuenta el sexo.

Posttraumatic stress symptoms and interpersonal processes in burn survivors and their partners.

Background: A burn event can elicit symptoms of posttraumatic stress disorder (PTSD) in survivors and their partners and may impact the way these couple members interact with each other. They may try to protect each other from further emotional distress by avoiding talking about the burn event, but they may also show concern towards each other.Objective: The aim of this study was to investigate bidirectional relationships between survivor's and partner's PTSD symptoms and two interpersonal processes: partner-oriented 'self-regulation', which is avoidance-oriented, and 'expressed concern', which is approach-oriented.Method: In this longitudinal multi-centre study, 119 burn survivors and their partners participated. Measures of PTSD symptoms, self-regulation, and expressed concern were administered in the acute phase following the burns, and follow-ups took place up to 18 months postburn. Intra- and interpersonal effects were examined in a random intercept cross-lagged panel model. Exploratory effects of burn severity were also investigated.Results: Within individuals, survivor's expressed concern predicted later higher levels of survivor's PTSD symptoms. In their partners, self-regulation and PTSD symptoms reinforced each other in the early phase postburn. Between the two couple members, partner's expressed concern predicted later lower levels of survivor's PTSD symptoms. Exploratory regression analyses showed that burn severity moderated the effect of survivor's self-regulation on survivor's PTSD symptoms, indicating that self-regulation was continuously related to higher levels of PTSD symptoms over time within more severely burned survivors, but not in less severely burned survivors.Conclusion: PTSD symptoms and self-regulation reinforced each other in partners and possibly also in more severely burned survivors. Partner's expressed concern was related to lower levels of survivor's PTSD symptoms, whereas survivor's expressed concern was related to higher levels of survivor's PTSD symptoms. These findings emphasize the importance of screening for and monitoring PTSD symptoms in burn survivors and their partner and of encouraging couple's self-disclosure.

PTSD symptoms in burn survivors and their partners are related to both avoidance- and approach-oriented interpersonal processes.In partners, higher levels of self-regulation were bidirectionally related to higher levels of posttraumatic stress symptoms.Concern expressed by partners may mitigate posttraumatic stress symptoms in burn survivors.

The Use of Questionable Research Practices to Survive in Academia Examined With Expert Elicitation, Prior-Data Conflicts, Bayes Factors for Replication Effects, and the Bayes Truth Serum.

The popularity and use of Bayesian methods have increased across many research domains. The current article demonstrates how some less familiar Bayesian methods can be used. Specifically, we applied expert elicitation, testing for prior-data conflicts, the Bayesian Truth Serum, and testing for replication effects via Bayes Factors in a series of four studies investigating the use of questionable research practices (QRPs). Scientifically fraudulent or unethical research practices have caused quite a stir in academia and beyond. Improving science starts with educating Ph.D. candidates: the scholars of tomorrow. In four studies concerning 765 Ph.D. candidates, we investigate whether Ph.D. candidates can differentiate between ethical and unethical or even fraudulent research practices. We probed the Ph.D.s' willingness to publish research from such practices and tested whether this is influenced by (un)ethical behavior pressure from supervisors or peers. Furthermore, 36 academic leaders (deans, vice-deans, and heads of research) were interviewed and asked to predict what Ph.D.s would answer for different vignettes. Our study shows, and replicates, that some Ph.D. candidates are willing to publish results deriving from even blatant fraudulent behavior-data fabrication. Additionally, some academic leaders underestimated this behavior, which is alarming. Academic leaders have to keep in mind that Ph.D. candidates can be under more pressure than they realize and might be susceptible to using QRPs. As an inspiring example and to encourage others to make their Bayesian work reproducible, we published data, annotated scripts, and detailed output on the Open Science Framework (OSF).

Approximate Measurement Invariance of Willingness to Sacrifice for the Environment Across 30 Countries: The Importance of Prior Distributions and Their Visualization.

Nationwide opinions and international attitudes toward climate and environmental change are receiving increasing attention in both scientific and political communities. An often used way to measure these attitudes is by large-scale social surveys. However, the assumption for a valid country comparison, measurement invariance, is often not met, especially when a large number of countries are being compared. This makes a ranking of countries by the mean of a latent variable potentially unstable, and may lead to untrustworthy conclusions. Recently, more liberal approaches to assessing measurement invariance have been proposed, such as the alignment method in combination with Bayesian approximate measurement invariance. However, the effect of prior variances on the assessment procedure and substantive conclusions is often not well understood. In this article, we tested for measurement invariance of the latent variable "willingness to sacrifice for the environment" using Maximum Likelihood Multigroup Confirmatory Factor Analysis and Bayesian approximate measurement invariance, both with and without alignment optimization. For the Bayesian models, we used multiple priors to assess the impact on the rank order stability of countries. The results are visualized in such a way that the effect of different prior variances and models on group means and rankings becomes clear. We show that even when models appear to be a good fit to the data, there might still be an unwanted impact on the rank ordering of countries. From the results, we can conclude that people in Switzerland and South Korea are most motivated to sacrifice for the environment, while people in Latvia are less motivated to sacrifice for the environment.

Prevalence and course of posttraumatic stress disorder symptoms in partners of burn survivors.

Background: Partners of burn survivors may develop posttraumatic stress disorder (PTSD) symptoms in response to the potential life threatening nature of the burn event and the burn survivor's medical treatment. Objective: This longitudinal study examined the prevalence, course and potential predictors of partners' PTSD symptoms up to 18 months post-burn. Methods: Participants were 111 partners of adult burn survivors. In a multi-centre study, PTSD symptoms were assessed with the Impact of Event Scale-Revised during the acute phase and subsequently at 3, 6, 12 and 18 months post-burn. Partners' appraisal of life threat, anger, guilt and level of rumination were assessed as potential predictors of PTSD symptoms in an exploratory piecewise latent growth model. Results: Acute PTSD symptoms in the clinical range were reported by 30% of the partners, which decreased to 4% at 18 months post-burn. Higher acute PTSD symptoms were related to perceived life threat and higher levels of anger, guilt, and rumination. Over time, mean symptom levels decreased, especially in partners with high levels of acute PTSD symptoms, perceived life threat and rumination. From three months onward, PTSD symptoms decreased less in partners of more severely burned survivors. At 18 months post-burn, higher levels of PTSD symptoms were related to higher acute PTSD symptoms and more severe burns. Conclusions: One in three partners reported clinical levels of acute PTSD symptoms, of which the majority recovered over time. Perceived life threat, feelings of anger and guilt, and rumination may indicate the presence of acute PTSD symptoms, whereas more severe burns predict long-term PTSD symptom levels. The results highlight the need to screen for acute PTSD symptoms and offer psychological help to partners to alleviate acute elevated stress levels if indicated.

Antecedentes: Las parejas de sobrevivientes de quemaduras pueden desarrollar síntomas de trastorno de estrés postraumático (TEPT) en respuesta a la naturaleza potencialmente mortal de las quemaduras y al tratamiento médico del sobreviviente de quemaduras.Objetivo: Este estudio longitudinal examinó la prevalencia, el curso y predictores potenciales de los síntomas de TEPT de la pareja hasta 18 meses después de la quemadura.Métodos: Los participantes fueron 111 parejas de adultos sobrevivientes de quemaduras. En un estudio multicéntrico, los síntomas de TEPT se evaluaron con la Escala de Impacto de Eventos Revisada durante la fase aguda y posteriormente a los 3, 6, 12 y 18 meses de la quemadura. La apreciación de las parejas de la amenaza de vida, ira, culpa y nivel de rumiación fueron evaluados como posibles predictores de síntomas de TEPT en un modelo exploratorio de crecimiento latente por partes.Resultados: El 30% de las parejas informó síntomas de TEPT agudo en rango clínico, que disminuyó a un 4% a los 18 meses después de la quemadura. Los síntomas agudos más altos de TEPT se relacionaron con la percepción de una amenaza para la vida y mayores niveles de ira, culpa y rumiación. Con el tiempo, los niveles promedio de síntomas disminuyeron, especialmente en parejas con altos niveles de síntomas agudos de TEPT, amenaza vital percibida y rumiación. A partir de los 3 meses, los síntomas del TEPT disminuyeron menos en las parejas de los sobrevivientes con quemaduras más graves. Pasados 18 meses de la quemadura, los niveles más altos de síntomas de TEPT se relacionaron con síntomas de TEPT agudos más altos y quemaduras más graves.Conclusiones: Una de cada tres parejas informó niveles clínicos de síntomas agudos de TEPT, de los cuales la mayoría se recuperó con el tiempo. La percepción de amenaza a la vida, los sentimientos de ira y culpa y la rumiación pueden indicar la presencia de síntomas agudos de TEPT, mientras que quemaduras más graves predicen síntomas de TEPT a largo plazo. Los resultados resaltan la necesidad de realizar tamizaje para síntomas agudos de TEPT y ofrecer ayuda psicológica a las parejas para aliviar los niveles elevados de estrés agudo, si está indicado.

On Imbalance of Impulse Control and Sensation Seeking and Adolescent Risk: An Intra-individual Developmental Test of the Dual Systems and Maturational Imbalance Models.

Heterogeneity in development of imbalance between impulse control and sensation seeking has not been studied until now. The present study scrutinized this heterogeneity and the link between imbalance and adolescent risk. Seven-wave data of 7,558 youth (50.71% males; age range from 12/13 until 24/25) were used. Three developmental trajectories were identified. The first trajectory, "sensation seeking to balanced sensation seeking", included participants with a higher level of sensation seeking than impulse control across all ages. The second trajectory, "moderate dominant control", included participants showing moderate and increasing impulse control relative to sensation seeking across all ages. The third trajectory, "strong late dominant control", included participants showing the highest level of impulse control which was about as strong as sensation seeking from early to middle adolescence and became substantially stronger from late adolescence to early adulthood. Although the systematic increase of impulse control in all subgroups is in line with both models, neither of these combined trajectories of control and sensation seeking was predicted by the Dual Systems Model or the Maturational Imbalance Model. Consistent with both models the "sensation seeking to balanced sensation seeking" trajectory showed the highest level of substance use. It can be concluded that, even though both theories adequately predict the link between imbalance and risk, neither the Dual Systems Model nor the Maturational Imbalance Model correctly predict the heterogeneity in development of imbalance between impulse control and sensation seeking.

Bio-Psychological Predictors of Acute and Protracted Fatigue After Burns: A Longitudinal Study.

OBJECTIVE: Fatigue after burns is often attributed to the hyperinflammatory and hypermetabolic response, while it may be best understood from a bio-psychological perspective, also involving the neuro-endocrine system. This longitudinal multi-center study examined the course of fatigue up to 18 months postburn. The contribution of bio-psychological factors, including burn severity, pain, and acute PTSD symptoms, to the course and persistence of fatigue was studied in a multifactorial model. METHODS: Participants were 247 adult burn survivors. Fatigue symptoms were assessed with the Multidimensional Fatigue Inventory during the acute phase and subsequently at 3, 6, 12, and 18 months postburn, and were compared to population norms. Age, gender, burn severity, acute PTSD symptoms and pain were assessed as potential predictors of fatigue over time in a latent growth model. RESULTS: At 18 months postburn, 46% of the burn survivors reported fatigue, including 18% with severe fatigue. In the acute phase, higher levels of fatigue were related to multiple surgeries, presence of pain, and higher levels of acute PTSD symptoms. Fatigue gradually decreased over time with minor individual differences in rate of decrease. At 18 months, pain and acute PTSD symptoms remained significant predictors of fatigue levels. CONCLUSIONS: Protracted fatigue after burns was found in almost one out of five burn survivors and was associated with both pain and acute PTSD symptoms. Early detection of PTSD symptoms and early psychological interventions aimed at reducing PTSD symptoms and pain may be warranted to reduce later fatigue symptoms.

Individual recovery of health-related quality of life during 18 months post-burn using a retrospective pre-burn measurement: an exploratory study.

PURPOSE: This study explored the individual trajectories of health-related quality of life (HRQL) compared to recalled pre-burn level of HRQL and investigated whether burn severity and post-traumatic stress disorder (PTSD) symptoms increase the risk of not returning to pre-burn level of HRQL. METHODS: Data were obtained from 309 adult patients with burns in a multicenter study. Patients completed the EQ-5D-3L questionnaire with a Cognition bolt-on shortly after hospital admission, which included a recalled pre-injury measure, and, again, at 3, 6, 12 and 18 months post-burn. Burn severity was indicated by the number of surgeries, and PTSD symptoms were assessed with the IES-R at three months post-burn. Pre- and post-injury HRQL were compared to norm populations. RESULTS: Recalled pre-injury HRQL was higher than population norms and HRQL at 18 months post-burn was comparable to population norms. Compared to the pre-injury level of functioning, four HRQL patterns of change over time were established: Stable, Recovery, Deterioration, and Growth. In each HRQL domain, a subset of patients did not return to their recalled pre-injury levels, especially with regard to Pain, Anxiety/Depression, and Cognition. Patients with more severe burns or PTSD symptoms were less likely to return to pre-injury level of functioning within 18 months post-burn. CONCLUSION: This study identified four patterns of individual change. Patients with more severe injuries and PTSD symptoms were more at risk of not returning to their recalled pre-injury HRQL. This study supports the face validity of using a recalled pre-burn HRQL score as a reference point to monitor HRQL after burns.

The data representativeness criterion: Predicting the performance of supervised classification based on data set similarity.

In a broad range of fields it may be desirable to reuse a supervised classification algorithm and apply it to a new data set. However, generalization of such an algorithm and thus achieving a similar classification performance is only possible when the training data used to build the algorithm is similar to new unseen data one wishes to apply it to. It is often unknown in advance how an algorithm will perform on new unseen data, being a crucial reason for not deploying an algorithm at all. Therefore, tools are needed to measure the similarity of data sets. In this paper, we propose the Data Representativeness Criterion (DRC) to determine how representative a training data set is of a new unseen data set. We present a proof of principle, to see whether the DRC can quantify the similarity of data sets and whether the DRC relates to the performance of a supervised classification algorithm. We compared a number of magnetic resonance imaging (MRI) data sets, ranging from subtle to severe difference is acquisition parameters. Results indicate that, based on the similarity of data sets, the DRC is able to give an indication as to when the performance of a supervised classifier decreases. The strictness of the DRC can be set by the user, depending on what one considers to be an acceptable underperformance.

Expert Elicitation for Latent Growth Curve Models: The Case of Posttraumatic Stress Symptoms Development in Children With Burn Injuries.

Experts provide an alternative source of information to classical data collection methods such as surveys. They can provide additional insight into problems, supplement existing data, or provide insights when classical data collection is troublesome. In this paper, we explore the (dis)similarities between expert judgments and data collected by traditional data collection methods regarding the development of posttraumatic stress symptoms (PTSSs) in children with burn injuries. By means of an elicitation procedure, the experts' domain expertise is formalized and represented in the form of probability distributions. The method is used to obtain beliefs from 14 experts, including nurses and psychologists. Those beliefs are contrasted with questionnaire data collected on the same issue. The individual and aggregated expert judgments are contrasted with the questionnaire data by means of Kullback-Leibler divergences. The aggregated judgments of the group that mainly includes psychologists resemble the questionnaire data more than almost all of the individual expert judgments.

Classification of comorbidity in obsessive-compulsive disorder: A latent class analysis.

OBJECTIVE: Patients with OCD differ markedly from one another in both number and kind of comorbid disorders. In this study, we set out to identify and characterize homogeneous subgroups of OCD patients based on their comorbidity profile. METHODS: In a cohort of 419 adult subjects with OCD, the lifetime presence of fifteen comorbid disorders was assessed. Latent class analysis was used to identify comorbidity-based subgroups. Groups were compared with regard to core clinical characteristics: familiality, childhood trauma, age at onset, illness severity, OCD symptom dimensions, personality characteristics, and course of illness. RESULTS: The study sample could be divided in a large group (n = 311) with a low amount of comorbidity that could be further subdivided into two subgroups: OCD simplex (n = 147) and OCD with lifetime major depressive disorder (n = 186), and a group (n = 108) with a high amount of comorbidity that could be further subdivided into a general anxiety-related subgroup (n = 49), an autism/social phobia-related subgroup (n = 27), and a psychosis/bipolar-related subgroup (n = 10). Membership of the high-comorbid subgroup was associated with higher scores on childhood trauma, illness severity, and the aggression/checking symptom dimension and lower scores on several personality characteristics. CONCLUSION: Grouping OCD patients based on their comorbidity profile might provide more homogeneous, and therefore, more suitable categories for future studies aimed at unraveling the etiological mechanisms underlying this debilitating disorder.

Making eye movements during imaginal exposure leads to short-lived memory effects compared to imaginal exposure alone.

BACKGROUND AND OBJECTIVES: A plethora of eye movement desensitization and reprocessing (EMDR) analogue studies has shown that, in the short term, making eye movements (EM) during brief imaginal exposure ("recall + EM") blurs memories more than just imaginal exposure ("recall only"). Yet, results of the few studies that included a follow-up test are inconsistent. We improved this paradigm's ecological validity by including an extended intervention phase and multiple assessments per phase. We hypothesized that recall + EM results in larger immediate and 24 h reductions in memory vividness, negative valence, and distress than recall alone. We explored the persistence of the effects, as well as the predictive value of memory characteristics and individual differences. METHODS: Students (N = 100) selected a negative autobiographical memory and were randomized to recall + EM or recall alone; both interventions lasted 32 intervals of 24s. During the interventions they rated the memory after every four intervals. RESULTS: After 4 × 24s intervention, recall + EM resulted in memory deflation, while recall only caused memory inflation. After the full intervention (i.e., 32 × 24s), both conditions resulted in immediate and 24 h reductions on all outcome measures. Crucially, memory effects in the recall + EM condition partially relapsed 24 h later, while the effects in the recall only condition persisted. Change patterns were hardly explained by predictive variables. LIMITATIONS: We used a non-clinical sample; replication in clinical samples is warranted. CONCLUSION: Making EM during imaginal exposure leads to short-lived effects compared to imaginal exposure alone. However, EM may offer a response aid for those who avoid imaginal exposure.

Quantifying the informational value of classification images.

Reverse correlation is an influential psychophysical paradigm that uses a participant's responses to randomly varying images to build a classification image (CI), which is commonly interpreted as a visualization of the participant's mental representation. It is unclear, however, how to statistically quantify the amount of signal present in CIs, which limits the interpretability of these images. In this article, we propose a novel metric, infoVal, which assesses informational value relative to a resampled random distribution and can be interpreted like a z score. In the first part, we define the infoVal metric and show, through simulations, that it adheres to typical Type I error rates under various task conditions (internal validity). In the second part, we show that the metric correlates with markers of data quality in empirical reverse-correlation data, such as the subjective recognizability, objective discriminability, and test-retest reliability of the CIs (convergent validity). In the final part, we demonstrate how the infoVal metric can be used to compare the informational value of reverse-correlation datasets, by comparing data acquired online with data acquired in a controlled lab environment. We recommend a new standard of good practice in which researchers assess the infoVal scores of reverse-correlation data in order to ensure that they do not read signal in CIs where no signal is present. The infoVal metric is implemented in the open-source rcicr R package, to facilitate its adoption.