The transparency and reproducibility of systematic reviews in forensic science.

Systematic reviews are indispensable tools for both reliably informing decision-makers about the state of the field and for identifying areas that need further study. Their value, however, depends on their transparency and reproducibility. Readers should be able to determine what was searched for and when, where the authors searched, and whether that search was predetermined or evolved based on what was found. In this article, we measured the transparency and reproducibility of systematic reviews in forensic science, a field where courts, policymakers, and legislators count on systematic reviews to make informed decisions. In a sample of 100 systematic reviews published between 2018 and 2021, we found that completeness of reporting varied markedly. For instance, 50 % of reviews claimed to follow a reporting guideline and such statements were only modestly related to compliance with that reporting guideline. As to specific reporting items, 82 % reported all of the databases searched, 22 % reported the review's full Boolean search logic, and just 7 % reported the review was registered. Among meta-analyses (n = 23), only one stated data was available and none stated the analytic code was available. After considering the results, we end with recommendations for improved regulation of reporting practices, especially among journals. Our results may serve as a useful benchmark as the field evolves.

Beyond CSI: Calibrating public beliefs about the reliability of forensic science through openness and transparency.

What drives public beliefs about the credibility of a scientific field? This question is increasingly important, with recent discussion of a "reproducibility crisis" affecting many fields. Such discussions are vital in forensic science, a discipline that has experienced severe scrutiny from both the media and large oversight bodies. In this paper, we make three contributions to this discussion. First, we bring together and compare several studies in which laypeople were asked about the reliability of forensic science practices. This review suggests that forensic practices do not enjoy uniformly high reliability ratings from the public and these ratings are not calibrated with the scientific consensus. We then review three empirically-tested ways that other fields are dealing with their own crises, all centred around transparency and openness. Finally, we make recommendations for how forensic science can leverage transparency and openness to improve and maintain its long-term credibility. As part of these recommendations, we find that empirical research supports the Houston Forensic Science Center's recent claims that it has improved its credibility through openness and transparency.

Forensic science needs registered reports.

The registered report (RR) format is rapidly being adopted by scientific researchers and journals. RRs flip the peer review process, with reviewers evaluating proposed methods, rather than the data and findings. Editors then accept or reject articles largely based on the pre-data collection review. Accordingly, RRs reduce the incentive for researchers to exaggerate their findings, and they make any data-driven changes to the methods and analysis more conspicuous. They also reduce publication bias, ensuring studies with null or otherwise unfavorable results are published. RRs are being used in many fields to improve research practices and increase confidence in study findings. The authors suggest RRs ought to be the default way in which validation studies are conducted and reported in the forensic sciences. They produce more reliable findings, advance criminal justice values, and will lead to several efficiencies in the research process.

Open forensic science.

The mainstream sciences are experiencing a revolution of methodology. This revolution was inspired, in part, by the realization that a surprising number of findings in the bioscientific literature could not be replicated or reproduced by independent laboratories. In response, scientific norms and practices are rapidly moving towards openness. These reforms promise many enhancements to the scientific process, notably improved efficiency and reliability of findings. Changes are also underway in the forensic. After years of legal-scientific criticism and several reports from peak scientific bodies, efforts are underway to establish the validity of several forensic practices and ensure forensic scientists perform and present their work in a scientifically valid way. In this article, the authors suggest that open science reforms are distinctively suited to addressing the problems faced by forensic science. Openness comports with legal and criminal justice values, helping ensure expert forensic evidence is more reliable and susceptible to rational evaluation by the trier of fact. In short, open forensic science allows parties in legal proceedings to understand and assess the strength of the case against them, resulting in fairer outcomes. Moreover, several emerging open science initiatives allow for speedier and more collaborative research.

The role of mind-wandering in measurements of general aptitude.

Tests of working memory capacity (WMC) and fluid intelligence (gF) are thought to capture variability in a crucial cognitive capacity that is broadly predictive of success, yet pinpointing the exact nature of this capacity is an area of ongoing controversy. We propose that mind-wandering is associated with performance on tests of WMC and gF, thereby partially explaining both the reliable correlations between these tests and their broad predictive utility. Existing evidence indicates that both WMC and gF are correlated with performance on tasks of attention, yet more decisive evidence requires an assessment of the role of attention and, in particular, mind-wandering during performance of these tests. Four studies employing complementary methodological designs embedded thought sampling into tests of general aptitude and determined that mind-wandering was consistently associated with worse performance on these measures. Collectively, these studies implicate the capacity to avoid mind-wandering during demanding tasks as a potentially important source of success on measures of general aptitude, while also raising important questions about whether the previously documented relationship between WMC and mind-wandering can be exclusively attributed to executive failures preceding mind-wandering (McVay & Kane, 2010b).