The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses.

Active learning has become an increasingly popular method for screening large amounts of data in systematic reviews and meta-analyses. The active learning process continually improves its predictions on the remaining unlabeled records, with the goal of identifying all relevant records as early as possible. However, determining the optimal point at which to stop the active learning process is a challenge. The cost of additional labeling of records by the reviewer must be balanced against the cost of erroneous exclusions. This paper introduces the SAFE procedure, a practical and conservative set of stopping heuristics that offers a clear guideline for determining when to end the active learning process in screening software like ASReview. The eclectic mix of stopping heuristics helps to minimize the risk of missing relevant papers in the screening process. The proposed stopping heuristic balances the costs of continued screening with the risk of missing relevant records, providing a practical solution for reviewers to make informed decisions on when to stop screening. Although active learning can significantly enhance the quality and efficiency of screening, this method may be more applicable to certain types of datasets and problems. Ultimately, the decision to stop the active learning process depends on careful consideration of the trade-off between the costs of additional record labeling against the potential errors of the current model for the specific dataset and context.

Impact of Active learning model and prior knowledge on discovery time of elusive relevant papers: a simulation study.

Software that employs screening prioritization through active learning (AL) has accelerated the screening process significantly by ranking an unordered set of records by their predicted relevance. However, failing to find a relevant paper might alter the findings of a systematic review, highlighting the importance of identifying elusive papers. The time to discovery (TD) measures how many records are needed to be screened to find a relevant paper, making it a helpful tool for detecting such papers. The main aim of this project was to investigate how the choice of the model and prior knowledge influence the TD values of the hard-to-find relevant papers and their rank orders. A simulation study was conducted, mimicking the screening process on a dataset containing titles, abstracts, and labels used for an already published systematic review. The results demonstrated that AL model choice, and mostly the choice of the feature extractor but not the choice of prior knowledge, significantly influenced the TD values and the rank order of the elusive relevant papers. Future research should examine the characteristics of elusive relevant papers to discover why they might take a long time to be found.

Addressing the challenges of reconstructing systematic reviews datasets: a case study and a noisy label filter procedure.

Systematic reviews and meta-analyses typically require significant time and effort. Machine learning models have the potential to enhance screening efficiency in these processes. To effectively evaluate such models, fully labeled datasets-detailing all records screened by humans and their labeling decisions-are imperative. This paper presents the creation of a comprehensive dataset for a systematic review of treatments for Borderline Personality Disorder, as reported by Oud et al. (2018) for running a simulation study. The authors adhered to the PRISMA guidelines and published both the search query and the list of included records, but the complete dataset with all labels was not disclosed. We replicated their search and, facing the absence of initial screening data, introduced a Noisy Label Filter (NLF) procedure using active learning to validate noisy labels. Following the NLF application, no further relevant records were found. A simulation study employing the reconstructed dataset demonstrated that active learning could reduce screening time by 82.30% compared to random reading. The paper discusses potential causes for discrepancies, provides recommendations, and introduces a decision tree to assist in reconstructing datasets for the purpose of running simulation studies.

Machine learning to optimize literature screening in medical guideline development.

OBJECTIVES: In a time of exponential growth of new evidence supporting clinical decision-making, combined with a labor-intensive process of selecting this evidence, methods are needed to speed up current processes to keep medical guidelines up-to-date. This study evaluated the performance and feasibility of active learning to support the selection of relevant publications within medical guideline development and to study the role of noisy labels. DESIGN: We used a mixed-methods design. Two independent clinicians' manual process of literature selection was evaluated for 14 searches. This was followed by a series of simulations investigating the performance of random reading versus using screening prioritization based on active learning. We identified hard-to-find papers and checked the labels in a reflective dialogue. MAIN OUTCOME MEASURES: Inter-rater reliability was assessed using Cohen's Kappa (ĸ). To evaluate the performance of active learning, we used the Work Saved over Sampling at 95% recall (WSS@95) and percentage Relevant Records Found at reading only 10% of the total number of records (RRF@10). We used the average time to discovery (ATD) to detect records with potentially noisy labels. Finally, the accuracy of labeling was discussed in a reflective dialogue with guideline developers. RESULTS: Mean ĸ for manual title-abstract selection by clinicians was 0.50 and varied between - 0.01 and 0.87 based on 5.021 abstracts. WSS@95 ranged from 50.15% (SD = 17.7) based on selection by clinicians to 69.24% (SD = 11.5) based on the selection by research methodologist up to 75.76% (SD = 12.2) based on the final full-text inclusion. A similar pattern was seen for RRF@10, ranging from 48.31% (SD = 23.3) to 62.8% (SD = 21.20) and 65.58% (SD = 23.25). The performance of active learning deteriorates with higher noise. Compared with the final full-text selection, the selection made by clinicians or research methodologists deteriorated WSS@95 by 25.61% and 6.25%, respectively. CONCLUSION: While active machine learning tools can accelerate the process of literature screening within guideline development, they can only work as well as the input given by human raters. Noisy labels make noisy machine learning.

Which individual, social, and urban factors in early childhood predict psychopathology in later childhood, adolescence and young adulthood? A systematic review.

Background: A comprehensive picture is lacking of the impact of early childhood (age 0-5) risk factors on the subsequent development of mental health symptoms. Objective: In this systematic review, we investigated which individual, social and urban factors, experienced in early childhood, contribute to the development of later anxiety and depression, behavioural problems, and internalising and externalising symptoms in youth. Methods: Embase, MEDLINE, Scopus, and PsycInfo were searched on the 5th of January 2022. Three additional databases were retrieved from a mega-systematic review source that focused on the identification of both risk and protective indicators for the onset and maintenance of prospective depressive, anxiety and substance use disorders. A total of 46,450 records were identified and screened in ASReview, an AI-aided systematic review tool. We included studies with experimental, quasi-experimental, prospective and longitudinal study designs, while studies that focused on biological and genetical factors, were excluded. Results: Twenty studies were included. The majority of studies explored individual-level risk factors (N = 16). Eleven studies also explored social risk factors and three studied urban risk factors. We found evidence for early predictors relating to later psychopathology measures (i.e., anxiety and depression, behavioural problems, and internalising and externalising symptoms) in childhood, adolescence and early adulthood. These were: parental psychopathology, exposure to parental physical and verbal violence and social and neighbourhood disadvantage. Conclusions: Very young children are exposed to a complex mix of risk factors, which operate at different levels and influence children at different time points. The urban environment appears to have an effect on psychopathology but it is understudied compared to individual-level factors. Moreover, we need more research exploring the interaction between individual, social and urban factors.