A Practical Significance Bias in Laypeople's Evaluation of Scientific Findings.

People often rely on scientific findings to help them make decisions-however, failing to report effect magnitudes might lead to a potential bias in assuming findings are practically significant. Across two online studies (Prolific; N = 800), we measured U.S. adults' endorsements of expensive interventions described in media reports that led to effects that were small, large, or of unreported magnitude between groups. Participants who viewed interventions with unreported effect magnitudes were more likely to endorse interventions compared with those who viewed interventions with small effects and were just as likely to endorse interventions as those who viewed interventions with large effects, suggesting a practical significance bias. When effect magnitudes were reported, participants on average adjusted their evaluations accordingly. However, some individuals, such as those with low numeracy skills, were more likely than others to act on small effects, even when explicitly prompted to first consider the meaningfulness of the effect.

Evidence requirements of permanently listed digital health applications (DiGA) and their implementation in the German DiGA directory: an analysis.

BACKGROUND: With its digital health application (DiGA)-system, Germany is considered one of Europe's pioneers in the field of evidence-based digital health. Incorporating DiGA into standard medical care must be based on evidence-based success factors; however, a comprehensive overview of the evidence required of scientific studies for their approval is lacking. OBJECTIVE: The study aims to, (1) identify specific requirements defined by the Federal Institute for Drugs and Medical Devices (German: Bundesinstitut für Arzneimittel- und Medizinprodukte; BfArM) to design adequate studies, proving a positive healthcare effect, and (2) to assess the evidence given for applications permanently listed in the DiGA directory. METHODS: A multi-step approach was used: (1) identification of the evidence requirements for applications permanently listed in the DiGA directory, (2) identification of the evidence available supporting them. RESULTS: All DiGA permanently listed in the DiGA directory (13 applications) are included in the formal analysis. Most DiGA addressed mental health (n = 7), and can be prescribed for one or two indications (n = 10). All permanently listed DiGA have demonstrated their positive healthcare effect through a medical benefit, and most of them provide evidence for one defined primary endpoint. All DiGA manufacturers conducted a randomized controlled trial. DISCUSSION: It is striking that- although patient-relevant structural and procedural improvements show high potential for improving care, especially in terms of processes - all DiGA have provided a positive care effect via a medical benefit. Although BfArM accepts study designs with a lower level of evidence for the proof of a positive healthcare effect, all manufacturers conducted a study with a high level of evidence. CONCLUSION: The results of this analysis indicate that permanently listed DiGA meet higher standards than required by the guideline.

'Dopesick'; why a critical view of the literature may save your patient's life.

INTRODUCTION: Teaching critical literature appraisal is challenging. Providing a compelling clinical context using 'cinemeduation' stimulates interest in the topic. METHODS: After watching the first episode of the mini-series 'Dopesick', where the scope, timeline and extent of the problem of opioid abuse are clearly seen, abstracts of the period literature strongly supporting the use of Oxycontin for use in chronic pain are shown. All were published in highly ranked medical journals. A simple paper evaluation structure is suggested. It is PPICOREAD which stands for the following questions: Who Paid for the study? What was the Population studied? What was the Intervention given in the study? What was the Control group used? What was the Outcome and was it of clinical significance to you? Was the trial Registered? Was there an Educational element for you? Was there anything Applicable to your practice? What was the Duration of the trial? Is this duration sufficient to reassure you that the trial is relevant to your practice? RESULTS: The very poor quality and dishonest nature of the conclusions of these papers are quickly and easily uncovered in a supervised workshop. The causes for these clear discrepancies are discussed. The devastating consequences are described.

Digital health applications and the fast-track pathway to public health coverage in Germany: challenges and opportunities based on first results.

OBJECTIVES: Evidence-based decision-making is the sine qua non for safe and effective patient care and the long-term functioning of health systems. Since 2020 Digital Health Applications (DiHA) in Germany have been undergoing a systematic pathway to be reimbursed by statutory health insurance (SHI) which is attracting attention in other European countries. We therefore investigate coverage decisions on DiHA and the underlying evidence on health care effects, which legally include both medical outcomes and patient-centred structural and procedural outcomes. METHODS: Based on publicly available data of the Institute for Medicines and Medical Devices searched between 08/2021 and 02/2022, all DiHA listed in the corresponding registry and thus reimbursable by the SHI were systematically investigated and presented descriptively on the basis of predefined criteria, such as clinical condition, and costs. The clinical trials on DiHA permanently included in the registry were reviewed with regard to their study design, endpoints investigated, the survey instruments used, and whether an intention-to-treat analysis was performed. Risk of bias was assessed using the ROB II tool. RESULTS: By February 2022, 30 DiHA had been included in the DiHA registry, one third of them permanently and two thirds conditionally. Most DiHA were therapeutic applications for mental illness based on cognitive behavioural therapy. For all permanently included DiHA, randomised controlled trials were conducted to demonstrate the impact on health care effects. While medical outcomes were investigated for all of these DiHA, patient-centred structural and procedural outcomes were rarely investigated. The majority of clinical trials showed a high risk of bias, mainly due to insufficient reporting quality. Overall, the prices for DiHA covered by SHI are on average around € 150 per month (min. € 40; max. € 248). CONCLUSIONS: Evidence-based decision-making on coverage of DiHA leaves room for improvements both in terms of reporting-quality and the use of patient-centred structural and procedural outcomes in addition to medical outcomes. With appropriate evidence, DiHA can offer an opportunity as an adjunct to existing therapy while currently the high risk of bias of the trials raises doubts about the justification of its high costs.

Part 2: Evidence Evaluation and Guidelines Development: 2020 American Heart Association Guidelines for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care.

The 2020 American Heart Association (AHA) Guidelines for Cardiopulmonary Resuscitation and Emergency Cardiovascular Care is based on the extensive evidence evaluation performed in conjunction with the International Liaison Committee on Resuscitation. The Adult Basic and Advanced Life Support, Pediatric Basic and Advanced Life Support, Neonatal Life Support, Resuscitation Education Science, and Systems of Care Writing Groups drafted, reviewed, and approved recommendations, assigning to each recommendation a Class of Recommendation (ie, strength) and Level of Evidence (ie, quality). The 2020 Guidelines are organized in knowledge chunks that are grouped into discrete modules of information on specific topics or management issues. The 2020 Guidelines underwent blinded peer review by subject matter experts and were also reviewed and approved for publication by the AHA Science Advisory and Coordinating Committee and the AHA Executive Committee. The AHA has rigorous conflict-of-interest policies and procedures to minimize the risk of bias or improper influence during development of the guidelines. Anyone involved in any part of the guideline development process disclosed all commercial relationships and other potential conflicts of interest.

Assessment of the endocrine disrupting properties of bisphenol AF: a case study applying the European regulatory criteria and guidance.

BACKGROUND: Scientific criteria to identify endocrine disruptors (ED) was recently implemented for plant protection products (PPP) and biocidal products (BP). A guidance document has been published by ECHA and EFSA in the context of ED criteria for PPPs and BPs. METHODS: In the present work, a case study was performed on Bisphenol AF (BPAF) to explore the application of the EU criteria and EFSA/ECHA guidance document for the ED assessment of a non-pesticide chemical regulated under REACH. A data dossier was built by a systematic literature search (Web of Science, Pubmed, Embase; n = 511), title/abstract screening (n = 124) and full text examination (n = 88). All the information was extracted and systematically reported for 309 parameters (100 for adversity; 209 for endocrine activity). The reliability of studies was assessed (SciRAP tool). RESULTS: Data were synthesized into 96 lines of evidence for adversity (n = 57), and endocrine activity (n = 39); and assessed by weight of evidence methodology. The initial analysis of the evidence indicated EATS-mediated adversity in mammals, therefore a mode of action (MoA) was postulated for both male and female adult exposure. Female MoA included estrogen receptor activation and altered steroidogenesis leading to ovarian dysfunction, altered estrous cycling and impaired female fertility. Male MoA was initiated by androgen receptor inhibition and altered steroidogenesis leading to dysfunction of male reproductive organs and impaired male fertility. CONCLUSIONS: The overall conclusion of the ED assessment indicated that BPAF meets the ED criteria for human health. The steps described in the ED guidance document were successfully completed, resulting in a thorough, structured and transparent identification of BPAF as an ED. Advantages and limitations of applying the ED criteria and guidance for a REACH chemical are discussed.

Young children's metacognitive awareness of confounded evidence.

Young children selectively explore confounded evidence-when causality is ambiguous due to multiple candidate causes. This suggests that they have an implicit understanding that confounded evidence is uninformative. This study examined explicit understanding, or metacognitive awareness, of the informativeness of different qualities of evidence during early childhood. In two within-participants conditions, children (N = 60 5- and 6-year-olds) were presented with confounded and unconfounded evidence and were asked to evaluate and explain their knowledge of a causal relation. Children more frequently requested further information in the confounded condition than in the unconfounded condition. Nearly half of them referred to multiple candidate causes when explaining confounded evidence. Our data demonstrate that young children can reason explicitly about the informativeness of different kinds of evidence.

Using data to solve problems: Children reason flexibly in response to different kinds of evidence.

This study examined children's (5- to 9-year-olds, N = 363) abilities to use information seeking and explanation to solve problems using conclusive or inconclusive (i.e., consistent, inconsistent, or ambiguous) evidence. Results demonstrated that inconsistent and ambiguous evidence, not consistent evidence, motivate more requests for information than conclusive evidence. In addition, children's explanations were flexible in response to evidence; explanations based on transitive inference were more likely to be associated with an accurate conclusion than other explanation types. Children's requests for additional information in response to inconclusive evidence increased with age, as did their problem-solving accuracy. The data demonstrate that children's capacity to use information seeking and explanation develop in tandem as tools for problem solving.

Part 2: Evidence Evaluation and Management of Conflicts of Interest: 2015 International Consensus on Cardiopulmonary Resuscitation and Emergency Cardiovascular Care Science With Treatment Recommendations.

The process for evaluating the resuscitation science has evolved considerably over the past 2 decades. The current process, which incorporates the use of the GRADE methodology, culminated in the 2015 CoSTR publication, which in turn will inform the international resuscitation councils' guideline development processes. Over the next few years, the process will continue to evolve as ILCOR moves toward a more continuous evaluation of the resuscitation science.

Reframing the debate: A question of probability, not of likelihood ratio.

Evidential value is measured by a likelihood ratio. This ratio has two components, the probability, or probability density, of the evidence if the prosecution proposition is true and the probability (density) of the evidence if the defence proposition is true. It takes the form of a single value, even if these probabilities are subjective measures of belief of the reporting forensic scientist.

When 'neutral' evidence still has probative value (with implications from the Barry George Case).

The likelihood ratio (LR) is a probabilistic method that has been championed as a 'simple rule' for evaluating the probative value of forensic evidence in court. Intuitively, if the LR is greater than one then the evidence supports the prosecution hypothesis; if the LR is less than one it supports the defence hypothesis, and if the LR is equal to one then the evidence favours neither (and so is considered 'neutral'-having no probative value). It can be shown by Bayes' theorem that this simple relationship only applies to pairs of hypotheses for which one is the negation of the other (i.e. to mutually exclusive and exhaustive hypotheses) and is not applicable otherwise. We show how easy it can be - even for evidence experts - to use pairs of hypotheses that they assume are mutually exclusive and exhaustive but are not, and hence to arrive at erroneous conclusions about the value of evidence using the LR. Furthermore, even when mutually exclusive and exhaustive hypotheses are used there are extreme restrictions as to what can be concluded about the probative value of evidence just from a LR. Most importantly, while the distinction between source-level hypotheses (such as defendant was/was not at the crime scene) and offence-level hypotheses (defendant is/is not guilty) is well known, it is not widely understood that a LR for evidence about the former generally has no bearing on the LR of the latter. We show for the first time (using Bayesian networks) the full impact of this problem, and conclude that it is only the LR of the offence level hypotheses that genuinely determines the probative value of the evidence. We investigate common scenarios in which evidence has a LR of one but still has significant probative value (i.e. is not neutral as is commonly assumed). As illustration we consider the ramifications of these points for the case of Barry George. The successful appeal against his conviction for the murder of Jill Dando was based primarily on the argument that the firearm discharge residue (FDR) evidence, assumed to support the prosecution hypothesis at the original trial, actually had a LR equal to one and hence was 'neutral'. However, our review of the appeal transcript shows numerous examples of the problems with the use of hypotheses identified above. We show that if one were to follow the arguments recorded in the Appeal judgement verbatim, then contrary to the Appeal conclusion, the probative value of the FDR evidence may not have been neutral as was concluded.

From understanding to justifying: Computational reliabilism for AI-based forensic evidence evaluation.

Techniques from artificial intelligence (AI) can be used in forensic evidence evaluation and are currently applied in biometric fields. However, it is generally not possible to fully understand how and why these algorithms reach their conclusions. Whether and how we should include such 'black box' algorithms in this crucial part of the criminal law system is an open question that has not only scientific but also ethical, legal, and philosophical angles. Ideally, the question should be debated by people with diverse backgrounds. Here, we present a view on the question from the philosophy of science angle: computational reliabilism (CR). CR posits that we are justified in believing the output of an AI system, if we have grounds for believing its reliability. Under CR, these grounds are classified into 'reliability indicators' of three types: technical, scientific, and societal. This framework enables debates on the suitability of AI methods for forensic evidence evaluation that take a wider view than explainability and validation. We argue that we are justified in believing the AI's output for forensic comparison of voices and forensic comparison of faces. Technical indicators include the validation of the AI algorithm in itself, validation of its application in the forensic setting, and case-based validation. Scientific indicators include the simple notion that we know faces and voices contain identifying information along with operationalizing well-established metrics and forensic practices. Societal indicators are the emerging scientific consensus on the use of these methods, as well as their application and interpretation by well-educated and certified practitioners. We expect expert witnesses to rely more on technical indicators to be justified in believing AIsystems, and triers-of-fact to rely more on societal indicators to believe the expert witness supported by the AIsystem.

Research on likelihood ratio evaluation method of fingerprint evidence based on parameter estimation method.

Fingerprints with similar morphological characteristics but from different individuals can lead to errors in individual identification, especially when dealing with large databases containing millions of fingerprints. To address this issue and enhance the accuracy of similar fingerprint identification, the use of the likelihood ratio (LR) model for quantitative evaluation of fingerprint evidence has emerged as an effective research method. In this study, the LR fingerprint evidence evaluation model was established by using mathematical statistical methods, such as parameter estimation and hypothesis testing. This involved various steps, including database construction, scoring, fitting, calculation, and visual evaluation. Under the same-source conditions, the optimal parameter methods selected by different number of minutiae are gamma and Weibull distribution, while normal, Weibull, and lognormal distributions were the fitting parameters selected for minutiae configurations. The fitting parameters selected by different number of minutiae under different-source conditions are lognormal distribution, and the parameter methods selected for different minutiae configurations include Weibull, gamma, and lognormal distributions. The results of the LR model showed increased accuracy as the number of minutiae increased, indicating strong discriminative and corrective power. However, the accuracy of the LR evaluation based on different configurations was comparatively lower. In addition, the LR models with different numbers of minutiae outperformed those with different minutiae configurations. Our study shows that the use of LR models based on parametric methods is favoured in reducing the risk of fingerprint evidence misidentification, improving the quantitative assessment methods of fingerprint evidence, and promoting fingerprint identification from experience to science. Key points: Likelihood ratio (LR) method based on parameter estimation was applied to scientific evaluation of fingerprint evidence with excellent discriminatory and calibration capabilities.Both the number of minutiae and configuration of minutiae have significant effects on the score-based LR method.Fingerprints from the same source contain many different patterns of deformation.Databases containing 10 million fingerprints from different sources have been used for building the LR model.

The effect of following best practice reporting recommendations on legal and community evaluations of forensic examiners reports.

Commentators have recommended that forensic scientists' reports contain various disclosures to facilitate comprehension. However, little research has explored whether following best practice recommendations for disclosure impacts on receivers' impressions of the evidence. We examined whether forensic science reports that are more compliant with these best practice recommendations reduced overvaluing of the evidence and sensitized legal and community decision-makers to evidence quality. Across three experiments, 240 legal practitioners/trainees and 566 community decision-makers were presented with a fingerprint or footwear report that was either compliant or non-compliant with best practice recommendations. Participants were then asked to make evaluations and decisions based on the report. We found mixed effects of report compliance. Report compliance affected community participant's evaluations of the persuasiveness of the evidence but had limited impact on the judgments of legal practitioners/trainees. When presented with compliant reports, we found that community participants regarded unknown reliability evidence as less reliable and less persuasive than high reliability evidence, suggesting disclosures helped reduce overvaluing of the evidence and create sensitivity to differences in evidence quality. These results suggest compliance with reporting recommendations does affect community impressions, while only minimally influencing legal impressions of forensic science evidence. The costs and/or benefits of this outcome require further examination.

Childhood Screening for Familial Hypercholesterolemia: JACC Review Topic of the Week.

Screening for familial hypercholesterolemia (FH) in childhood remains controversial. Existing guidelines offer practitioners conflicting advice despite generally agreeing on the evidence and areas in which evidence is lacking, including a lack of long-term clinical trials demonstrating coronary event reduction as a result of screening and long-term data on statin side effects. A limitation of existing evidence-based frameworks is reliance on 1 evidence grading system to determine recommendations. However, rigorous evidence evaluation alternatives relevant to FH exist. FH is considered a tier 1 genetic condition, meaning that identification and treatment will improve health outcomes among those affected. Elevated low-density lipoprotein cholesterol, the primary consequence of FH, can be considered causal for atherosclerosis and coronary heart disease. Incorporating these concepts into existing evidence pathways allows the inclusion of surrogate clinical trial outcomes (low-density lipoprotein cholesterol reduction and atherosclerosis regression) and observational data on medication safety, strengthening the evidence for pediatric screening for FH.

Bloodstain pattern analysis & Bayes: A case report.

The findings from a bloodstain pattern analysis (BPA) may assist in formulating or falsifying scenarios that are considered in the investigative stages of a criminal investigation. When a case proceeds to trial the bloodstain pattern expert may be asked about the relevance of their findings given scenarios that are proposed by the prosecution and defense counsel. Such opinions provided by an expert are highly relevant to police investigation or legal proceedings, but the reasoning behind the opinion or implicit assumptions made by the expert may not be transparent. A proper framework for the evaluation of forensic findings has been developed since the late twentieth century, based on the hierarchy of propositions, Bayesian reasoning and a model for case assessment and interpretation. This framework, when implemented in casework, mitigates some of the risks of cognitive biases, and makes the reasoning and scientific basis for the opinion transparent. This framework is broadly used across forensic science disciplines. In this paper we describe its application to the field of BPA using a case example from the Netherlands Forensic Institute (NFI).

Direct and Secondary Transfer of Touch DNA on a Credit Card: Evidence Evaluation Given Activity Level Propositions and Application of Bayesian Networks.

In a judiciary setting, questions regarding the mechanisms of transfer, persistence, and recovery of DNA are increasingly more common. The forensic expert is now asked to evaluate the strength of DNA trace evidence at activity level, thus assessing if a trace, given its qualitative and quantitative features, could be the result of an alleged activity. The present study is the reproduction of a real-life casework scenario of illicit credit card use by a co-worker (POI) of its owner (O). After assessing the shedding propensity of the participants, differences in DNA traces' qualitative and quantitative characteristics, given scenarios of primary and secondary transfer of touch DNA on a credit card, a non-porous plastic support, were investigated. A case-specific Bayesian Network to aid statistical evaluation was created and discrete observations, meaning the presence/absence of POI as a major contributor in both traces from direct and secondary transfer, were used to inform the probabilities of disputed activity events. Likelihood Ratios at activity level (LRα) were calculated for each possible outcome resulting from the DNA analysis. In instances where only POI and POI plus an unknown individual are retrieved, the values obtained show moderate to low support in favour of the prosecution proposition.

Systematic evaluation of the evidence for identification of endocrine disrupting properties of Bisphenol F.

Identification of endocrine disruptors (EDs) is a highly prioritized issue in the EU. However, scientific criteria to identify EDs have so far only been implemented for biocidal and plant protection products. The European Commission is working on developing a horizontal approach to the identification of EDs across legislations, based on these scientific criteria. With this study, our aim was to evaluate evidence on endocrine disrupting properties of Bisphenol F (BPF) by applying the process set out for biocidal and plant protection products in Europe. BPF is not registered under REACH and therefore assumed not to be produced in the EU > 1 ton/year, yet the substance has been detected in urine, serum, and breast milk in populations from different countries in Europe. BPF is raising concern since it is an analogue of the known ED and reproductive toxicant Bisphenol A. Relevant evidence on endocrine disrupting properties of BPF from the open literature was collected using systematic review methodology. Pre-defined inclusion criteria were developed to select relevant studies, and data was extracted. The reliability of included studies was evaluated by the Science in Risk Assessment and Policy tool, and results were converted into Klimisch categories to allow for categorization of study reliability. A weight-of-evidence approach was used to analyze the evidence and draw conclusions on endocrine-related activity and/or endocrine adversity. We found that there is sufficient evidence to conclude on an endocrine mechanism, and while evidence for adversity was not considered sufficient, we still conclude that BPF could also cause endocrine-mediated adversity. Two modes of action were postulated based on the collected data for BPF. Challenges of performing the ED assessment for data-poor chemicals and the importance of adequate reporting of studies in the open literature, especially for new approach methods, are discussed.

Objectifying evidence evaluation for gunshot residue comparisons using machine learning on criminal case data.

Comparative gunshot residue analysis addresses relevant forensic questions such as 'did suspect X fire shot Y?'. More formally, it weighs the evidence for hypotheses of the form H1: gunshot residue particles found on suspect's hands are from the same source as the gunshot residue particles found on the crime scene and H2: two sets of particles are from different sources. Currently, experts perform this analysis by evaluating the elemental composition of the particles using their knowledge and experience. The aim of this study is to construct a likelihood-ratio (LR) system based on representative data. Such an LR system can support the expert by making the interpretation of the results of electron microscopy analysis more empirically grounded. In this study we chose statistical models from the machine learning literature as candidates to construct this system, as these models have been shown to work well for large and high-dimensional datasets. Using a subsequent calibration step ensured that the system outputs well-calibrated LRs. The system is developed and validated on casework data and an additional validation step is performed on an independent dataset of cartridge data. The results show that the system performs well on both datasets. We discuss future work needed before the method can be implemented in casework.