Benchmarking multi-ancestry prostate cancer polygenic risk scores in a real-world cohort.

Prostate cancer is a heritable disease with ancestry-biased incidence and mortality. Polygenic risk scores (PRSs) offer promising advancements in predicting disease risk, including prostate cancer. While their accuracy continues to improve, research aimed at enhancing their effectiveness within African and Asian populations remains key for equitable use. Recent algorithmic developments for PRS derivation have resulted in improved pan-ancestral risk prediction for several diseases. In this study, we benchmark the predictive power of six widely used PRS derivation algorithms, including four of which adjust for ancestry, against prostate cancer cases and controls from the UK Biobank and All of Us cohorts. We find modest improvement in discriminatory ability when compared with a simple method that prioritizes variants, clumping, and published polygenic risk scores. Our findings underscore the importance of improving upon risk prediction algorithms and the sampling of diverse cohorts.

Developing realistic benchmarks for glaucoma care delivery.

CLINICAL RELEVANCE: Realistic benchmarks can serve as comparators for optometrists wishing to engage in clinical practice audits of their glaucoma care. BACKGROUND: The iCareTrack study established the appropriateness of glaucoma care delivery through clinical record audits of Australian optometry practices. Benchmarks required for monitoring and improving glaucoma care delivery do not exist. This study developed realistic benchmarks for glaucoma care and then benchmarked the performance of practices from the iCareTrack study to establish aspects of care that warrant attention from quality improvement initiatives. METHODS: Benchmarks were developed from the pre-existing iCareTrack dataset using the Achievable Benchmarks of Care (ABC) method. The iCareTrack study had audited the appropriateness of glaucoma care delivery against 37 clinical indicators for 420 randomly sampled glaucoma patient records from 42 Australian optometry practices. The four-step ABC method calculates benchmarks based on the top 10% of best-performing practices adjusted for low patient encounter numbers. iCareTrack results were compared to the benchmarks to explore the distribution of practices that were at, above or below benchmark. RESULTS: Benchmarks were developed for 34 of 37 iCareTrack indicators. For 26 (of 34) indicators, the benchmarks were at or above 90% appropriateness. The benchmarks for 14 (of 34) iCareTrack indicators were met by more than 80% of eligible practices, indicating excellent performance. Some aspects of glaucoma care such as peripheral anterior angle assessment, applanation tonometry, and visual field assessment appeared to be delivered sub-optimally by optometrists when compared to the benchmarks. CONCLUSION: This study established benchmarks for glaucoma care delivery in optometry practices that reflect realistic and top achievable performance. The large number of indicators with benchmarks above 90% confirmed that glaucoma care can and should be delivered by optometrists at very high levels of appropriateness. Benchmarking identified pockets of sub-optimal performance that can now be targeted by quality improvement initiatives.

European value-based healthcare benchmarking: moving from theory to practice.

BACKGROUND: Value-based healthcare (VBHC) is a conceptual framework to improve the value of healthcare by health, care-process and economic outcomes. Benchmarking should provide useful information to identify best practices and therefore a good instrument to improve quality across healthcare organizations. This paper aims to provide a proof-of-concept of the feasibility of an international VBHC benchmarking in breast cancer, with the ultimate aim of being used to share best practices with a data-driven approach among healthcare organizations from different health systems. METHODS: In the VOICE community-a European healthcare centre cluster intending to address VBHC from theory to practice-information on patient-reported, clinical-related, care-process-related and economic-related outcomes were collected. Patient archetypes were identified using clustering techniques and an indicator set following a modified Delphi was defined. Benchmarking was performed using regression models controlling for patient archetypes and socio-demographic characteristics. RESULTS: Six hundred and ninety patients from six healthcare centres were included. A set of 50 health, care-process and economic indicators was distilled for benchmarking. Statistically significant differences across sites have been found in most health outcomes, half of the care-process indicators, and all economic indicators, allowing for identifying the best and worst performers. CONCLUSIONS: To the best of our knowledge, this is the first international experience providing evidence to be used with VBHC benchmarking intention. Differences in indicators across healthcare centres should be used to identify best practices and improve healthcare quality following further research. Applied methods might help to move forward with VBHC benchmarking in other medical conditions.

Benchmarking long-read RNA-sequencing analysis tools using in silico mixtures.

The lack of benchmark data sets with inbuilt ground-truth makes it challenging to compare the performance of existing long-read isoform detection and differential expression analysis workflows. Here, we present a benchmark experiment using two human lung adenocarcinoma cell lines that were each profiled in triplicate together with synthetic, spliced, spike-in RNAs (sequins). Samples were deeply sequenced on both Illumina short-read and Oxford Nanopore Technologies long-read platforms. Alongside the ground-truth available via the sequins, we created in silico mixture samples to allow performance assessment in the absence of true positives or true negatives. Our results show that StringTie2 and bambu outperformed other tools from the six isoform detection tools tested, DESeq2, edgeR and limma-voom were best among the five differential transcript expression tools tested and there was no clear front-runner for performing differential transcript usage analysis between the five tools compared, which suggests further methods development is needed for this application.

Aportaciones y avances de la genética forense en los sucesos con víctimas múltiples / Contributions and advances of forensic genetics in mass fatality incidents

La identificación de los afectados por un suceso con víctimas múltiples es una prioridad por razones humanitarias y legales. La genética forense juega un importante papel en estas situaciones que, por su complejidad, a menudo se convierten en un reto para los distintos profesionales implicados. El establecimiento de guías y recomendaciones facilita el seguimiento de protocolos estandarizados que permiten garantizar la fiabilidad del resultado final de la identificación. Así mismo, los avances en la genética forense contribuyen a agilizar la respuesta, aportando nuevas estrategias de análisis y herramientas de tipo bioinformático. Con este artículo, se pretende ofrecer una visión general de cómo la genética forense y sus avances pueden contribuir en estas situaciones, así como algunas claves para entender la labor de los laboratorios de genética forense en la identificación de cadáveres en sucesos con víctimas múltiples. (AU)

Disaster victim identification is crucial for humanitarian and legal reasons. Forensic genetics plays an important role in these situations which often become a challenge for the different professionals involved due to their complexity. The establishment of guidelines and recommendations makes it easier to follow standardized protocols that make it possible to guarantee the reliability of the identification final result. Likewise, advances in forensic genetics contribute to speeding up the response, providing new analysis strategies and bioinformatic tools. This article aims to provide an overview of how forensic genetics and its advances can contribute in these situations, as well as some keys to understanding the work of forensic genetics laboratories in the identification of corpses in events with multiple victims. (AU)

Benchmark dose calculations for PFAS exposure based on two data sets on immunotoxic effects.

BACKGROUND: Exposure to perfluorinated alkylate substances (PFAS) is associated with harmful effects on human health, including developmental immunotoxicity. This outcome was chosen as the critical effect by the European Food Safety Authority (EFSA), which calculated a new joint reference dose for four PFAS using a Benchmark Dose (BMD) analysis of a study of 1-year old children. However, the U.S. Environmental Protection Agency (EPA) recently proposed much lower exposure limits. METHODS: We explored the BMD methodology for summary and individual data and compared the results with and without grouping for two data sets available. We compared the performance of different dose-response models including a hockey-stick model and a piecewise linear model. We considered different ways of testing the assumption of equal weight-based toxicity of the four PFAS and evaluated more flexible models with exposure indices allowing for differences in toxicity. RESULTS: Results relying on full and decile-based data were in good accordance. However, BMD results for the larger study were lower than observed by EFSA for the smaller study. EFSA derived a lower confidence limit for the BMD of 17.5 ng/mL for the sum of serum-PFAS concentration, while similar calculations in the larger cohort yielded values of about 1.5 ng/mL. As the assumption of equal weight-based toxicity of the four PFAS seems questionable, we confirmed dose-dependencies that allowed potency differences between PFAS. We also found that models linear in the parameters for the BMD analysis showed superior coverage probabilities. In particular, we found the piecewise linear model to be useful for Benchmark analysis. CONCLUSIONS: Both data sets considered could be analyzed on a decile basis without important bias or loss of power. The larger study showed substantially lower BMD results, both for individual PFAS and for joint exposures. Overall, EFSA's proposed tolerable exposure limit appears too high, while the EPA proposal is in better accordance with the results.

Cross-Registry Benchmarking of Data Quality: Lessons Learned.

Feedback of data quality measures to study sites is an established procedure in the management of registries. Comparisons of data quality between registries as a whole are missing. We implemented a cross-registry benchmarking of data quality within the field of health services research for six projects. Five (2020) and six (2021) quality indicators were selected from a national recommendation. The calculation of the indicators was adjusted to the registries' specific settings. Nineteen (2020) and 29 results (2021) could be included in the yearly quality report. Seventy-four per cent (2020) and 79% (2021) of the results did not include the threshold in their 95%-confidence-limits. The benchmarking revealed several starting points for a weak-point analysis through a comparison of results with a predefined threshold as well as through comparisons among each other. In the future, a cross-registry benchmarking might be part of services provided through a health services research infrastructure.

Considerations for application of benchmark dose modeling in radiation research: workshop highlights.

BACKGROUND: Exposure to different forms of ionizing radiation occurs in diverse occupational, medical, and environmental settings. Improving the accuracy of the estimated health risks associated with exposure is therefore, essential for protecting the public, particularly as it relates to chronic low dose exposures. A key aspect to understanding health risks is precise and accurate modeling of the dose-response relationship. Toward this vision, benchmark dose (BMD) modeling may be a suitable approach for consideration in the radiation field. BMD modeling is already extensively used for chemical hazard assessments and is considered statistically preferable to identifying low and no observed adverse effects levels. BMD modeling involves fitting mathematical models to dose-response data for a relevant biological endpoint and identifying a point of departure (the BMD, or its lower bound). Recent examples in chemical toxicology show that when applied to molecular endpoints (e.g. genotoxic and transcriptional endpoints), BMDs correlate to points of departure for more apical endpoints such as phenotypic changes (e.g. adverse effects) of interest to regulatory decisions. This use of BMD modeling may be valuable to explore in the radiation field, specifically in combination with adverse outcome pathways, and may facilitate better interpretation of relevant in vivo and in vitro dose-response data. To advance this application, a workshop was organized on June 3rd, 2022, in Ottawa, Ontario that brought together BMD experts in chemical toxicology and the radiation scientific community of researchers, regulators, and policy-makers. The workshop's objective was to introduce radiation scientists to BMD modeling and its practical application using case examples from the chemical toxicity field and demonstrate the BMDExpress software using a radiation dataset. Discussions focused on the BMD approach, the importance of experimental design, regulatory applications, its use in supporting the development of adverse outcome pathways, and specific radiation-relevant examples. CONCLUSIONS: Although further deliberations are needed to advance the use of BMD modeling in the radiation field, these initial discussions and partnerships highlight some key steps to guide future undertakings related to new experimental work.

Initial recommendations for performing, benchmarking and reporting single-cell proteomics experiments.

Analyzing proteins from single cells by tandem mass spectrometry (MS) has recently become technically feasible. While such analysis has the potential to accurately quantify thousands of proteins across thousands of single cells, the accuracy and reproducibility of the results may be undermined by numerous factors affecting experimental design, sample preparation, data acquisition and data analysis. We expect that broadly accepted community guidelines and standardized metrics will enhance rigor, data quality and alignment between laboratories. Here we propose best practices, quality controls and data-reporting recommendations to assist in the broad adoption of reliable quantitative workflows for single-cell proteomics. Resources and discussion forums are available at https://single-cell.net/guidelines .

Benchmarking integration of single-cell differential expression.

Integration of single-cell RNA sequencing data between different samples has been a major challenge for analyzing cell populations. However, strategies to integrate differential expression analysis of single-cell data remain underinvestigated. Here, we benchmark 46 workflows for differential expression analysis of single-cell data with multiple batches. We show that batch effects, sequencing depth and data sparsity substantially impact their performances. Notably, we find that the use of batch-corrected data rarely improves the analysis for sparse data, whereas batch covariate modeling improves the analysis for substantial batch effects. We show that for low depth data, single-cell techniques based on zero-inflation model deteriorate the performance, whereas the analysis of uncorrected data using limmatrend, Wilcoxon test and fixed effects model performs well. We suggest several high-performance methods under different conditions based on various simulation and real data analyses. Additionally, we demonstrate that differential expression analysis for a specific cell type outperforms that of large-scale bulk sample data in prioritizing disease-related genes.

Benchmarking commonly used software suites and analysis workflows for DIA proteomics and phosphoproteomics.

A plethora of software suites and multiple classes of spectral libraries have been developed to enhance the depth and robustness of data-independent acquisition (DIA) data processing. However, how the combination of a DIA software tool and a spectral library impacts the outcome of DIA proteomics and phosphoproteomics data analysis has been rarely investigated using benchmark data that mimics biological complexity. In this study, we create DIA benchmark data sets simulating the regulation of thousands of proteins in a complex background, which are collected on both an Orbitrap and a timsTOF instruments. We evaluate four commonly used software suites (DIA-NN, Spectronaut, MaxDIA and Skyline) combined with seven different spectral libraries in global proteome analysis. Moreover, we assess their performances in analyzing phosphopeptide standards and TNF-α-induced phosphoproteome regulation. Our study provides a practical guidance on how to construct a robust data analysis pipeline for different proteomics studies implementing the DIA technique.

Systematic benchmarking of statistical methods to assess differential expression of circular RNAs.

Circular RNAs (circRNAs) are covalently closed transcripts involved in critical regulatory axes, cancer pathways and disease mechanisms. CircRNA expression measured with RNA-seq has particular characteristics that might hamper the performance of standard biostatistical differential expression assessment methods (DEMs). We compared 38 DEM pipelines configured to fit circRNA expression data's statistical properties, including bulk RNA-seq, single-cell RNA-seq (scRNA-seq) and metagenomics DEMs. The DEMs performed poorly on data sets of typical size. Widely used DEMs, such as DESeq2, edgeR and Limma-Voom, gave scarce results, unreliable predictions or even contravened the expected behaviour with some parameter configurations. Limma-Voom achieved the most consistent performance throughout different benchmark data sets and, as well as SAMseq, reasonably balanced false discovery rate (FDR) and recall rate. Interestingly, a few scRNA-seq DEMs obtained results comparable with the best-performing bulk RNA-seq tools. Almost all DEMs' performance improved when increasing the number of replicates. CircRNA expression studies require careful design, choice of DEM and DEM configuration. This analysis can guide scientists in selecting the appropriate tools to investigate circRNA differential expression with RNA-seq experiments.

The Use of Balanced Scorecards in Mental Health Services: an Integrative Review and Thematic Analysis.

Performance management of mental health services (MHS) through quality reporting of strategic indicators and goals is essential to improve efficiency and quality of care. One such method is the balanced scorecard (BSC). This integrative review of peer-reviewed and industry implemented BSCs in MHS aims to inform future development of a more comprehensive mental health-focused benchmarking tool. A two-part systematic literature search consisted of peer-reviewed published literature on MHS specific BSCs utilising the PRISMA guidelines in addition to industry published BSCs available online. A total of 17 unique BSCs were identified. A total of 434 indicators were subject to thematic analysis identifying 11 key themes: prevalence, accessibility, services provided, clinical outcomes, client satisfaction, client involvement, staff motivation, staffing levels, governance and compliance, development, and costs and revenue. These themes represented the measures that MHS believed measured key performance criteria in alignment with their organisational objectives.

Level of appropriate primary diabetic eyecare delivered and achievable in optometry practices in Australia.

CLINICAL RELEVANCE: Current levels of appropriateness for primary diabetic eyecare delivered by Australian optometrists are presented along with realistic targets (benchmarks) for quality improvement. The demonstrated methods can be used in practice evaluation and benchmarking of other clinical practice areas and settings. BACKGROUND: To examine the appropriateness of diabetic eye-care delivery and establish achievable benchmarks of care (ABCs) for optometry practices in Australia. METHOD: In a retrospective audit, clinical records of patients with type-II diabetes obtained from a randomly selected nationally representative sample of optometry practices were assessed against evidence-based clinical indicators. Appropriate care is defined as care delivered in compliance with the indicators. The ABC for each indicator was calculated as the average performance for the top 10% of optometry practices after Bayesian adjustment to account for a low number of eligible records. RESULTS: The audit of 420 randomly selected patient records from 42 practices against 12 clinical indicators showed an overall appropriateness of 69% (95% confidence interval (CI) 66%, 73%) for overall diabetic eye care. While a high level of appropriateness was identified for recall period (93%, 95% CI 85%, 100%) and referral (100%, 95% CI 38%, 100%), larger gaps existed in history taking (46%, 95% CI 44%, 52%), dilated fundus examination (80%, 95% CI 76%, 84%) and iris examination (0%, 95% CI 0%, 56%). The ABCs for 8 of 12 indicators were 100%, and the remaining three indicators had ABCs above 80%. An ABC for the iris examination indicator could not be calculated owing to the low number of eligible patient record cards. CONCLUSIONS: This study demonstrated a systematic process of practice evaluation and benchmarking in optometry practices. The diabetic eye care delivered by Australian optometrists was largely appropriate; however, improvement opportunities exist for history taking and physical examination. The ABCs demonstrate that excellence in primary diabetic eye care is attainable and will serve as an important tool in future initiatives to reduce the identified evidence-to-practice gaps.

Mapping global dynamics of benchmark creation and saturation in artificial intelligence.

Benchmarks are crucial to measuring and steering progress in artificial intelligence (AI). However, recent studies raised concerns over the state of AI benchmarking, reporting issues such as benchmark overfitting, benchmark saturation and increasing centralization of benchmark dataset creation. To facilitate monitoring of the health of the AI benchmarking ecosystem, we introduce methodologies for creating condensed maps of the global dynamics of benchmark creation and saturation. We curate data for 3765 benchmarks covering the entire domains of computer vision and natural language processing, and show that a large fraction of benchmarks quickly trends towards near-saturation, that many benchmarks fail to find widespread utilization, and that benchmark performance gains for different AI tasks are prone to unforeseen bursts. We analyze attributes associated with benchmark popularity, and conclude that future benchmarks should emphasize versatility, breadth and real-world utility.

Benchmarking taxonomic classifiers with Illumina and Nanopore sequence data for clinical metagenomic diagnostic applications.

Culture-independent metagenomic detection of microbial species has the potential to provide rapid and precise real-time diagnostic results. However, it is potentially limited by sequencing and taxonomic classification errors. We use simulated and real-world data to benchmark rates of species misclassification using 100 reference genomes for each of the ten common bloodstream pathogens and six frequent blood-culture contaminants (n=1568, only 68 genomes were available for Micrococcus luteus). Simulating both with and without sequencing error for both the Illumina and Oxford Nanopore platforms, we evaluated commonly used classification tools including Kraken2, Bracken and Centrifuge, utilizing mini (8 GB) and standard (30-50 GB) databases. Bracken with the standard database performed best, the median percentage of reads across both sequencing platforms identified correctly to the species level was 97.8% (IQR 92.7:99.0) [range 5:100]. For Kraken2 with a mini database, a commonly used combination, median species-level identification was 86.4% (IQR 50.5:93.7) [range 4.3:100]. Classification performance varied by species, with Escherichia coli being more challenging to classify correctly (probability of reads being assigned to the correct species: 56.1-96.0%, varying by tool used). Human read misclassification was negligible. By filtering out shorter Nanopore reads we found performance similar or superior to Illumina sequencing, despite higher sequencing error rates. Misclassification was more common when the misclassified species had a higher average nucleotide identity to the true species. Our findings highlight taxonomic misclassification of sequencing data occurs and varies by sequencing and analysis workflow. To account for 'bioinformatic contamination' we present a contamination catalogue that can be used in metagenomic pipelines to ensure accurate results that can support clinical decision making.

NSQIP Quality Benchmarking and Evaluation of Potential Bias Associated with Higher- or Lower-Risk Operation Case Mix.

BACKGROUND: To ensure validity and acceptance of NSQIP risk-adjusted benchmarking, it is important that adjustments adequately control for hospitals that vary in their proportions of lower- or higher-risk operations (combined risk for procedure and patient). This issue was addressed in separate empirical and simulation studies. STUDY DESIGN: For the empirical study, potential miscalibration bias favoring hospitals that do lower-risk operations or disfavoring hospitals that do higher-risk operations was evaluated for 14 modeled outcomes using NSQIP data. A determination was also made as to whether there was a relationship between mean hospital operation risk and benchmarking results (log odds ratio). In the simulation study of the same 14 outcomes, hospital benchmarked performance was evaluated when sampled cases were reconstituted to include either a larger proportion of lower-risk operations or a larger proportion of higher-risk operations. RESULTS: Miscalibration favoring either lower- or higher-risk operations was absent, as were important associations between operative risk and hospital log odds ratios (most model R 2 less than 0.01). In the simulation, there were no substantial changes in log odds ratios when greater percentages of either lower- or higher-risk operations were included in a hospital's sample (nonsignificant p values and effect sizes less than 0.1). CONCLUSIONS: These results should enhance NSQIP participants' confidence in the adequacy of NSQIP patient and procedure risk-adjustment methods. NSQIP participants may rely on benchmarking findings, and implement quality improvement efforts based on them, without concern that they are biased by a preponderance of lower or higher risk operations.

Facts and Fallacy of Benchmark Performance Indicators.

Efforts to improve quality in healthcare have arisen from the recognition that the quality of care delivered and resulting outcomes are highly variable. Performance benchmarking using high-quality data to compare risk-adjusted outcomes between hospitals and surgeons has been widely adopted as one means for addressing this problem. In this article we discuss the history, current state, methodologies, and potential pitfalls of benchmarking efforts to improve quality of healthcare in the United States.

Pros and Cons of Randomized Controlled Trials and Benchmarking Controlled Trials in Rehabilitation: An Academic Debate within the European Academy of Rehabilitation Medicine.

The European Academy of Rehabilitation Medicine (EARM) held a debate in Hannover, Germany, on 1st of September 2016 on the pros  and cons of randomized controlled trials (RCTs) and observational effectiveness studies (benchmarking controlled trials; BCTs). The  debate involved a chairperson, a person presenting the substance of the debate, an opponent, and a rapporteur. The academicians participated in the discussion. Eight propositions and proposed statements formed the substance of the debate. There was agreement that a study question should be the starting point of an effectiveness study, and not the study method, i.e. RCT or BCT. The term "benchmarking" was questioned: does it mean market-oriented medicine? It was clarified that benchmarking refers to the methodological features of this study design: there must always be a comparison between peers. It was agreed that BCTs might be better than RCTs for use in rehabilitation studies, in which one often needs multi-centred studies, such as in the assessment of the effectiveness of pathways when there is complexity of processes, health systems, organizational issues, structures and facilities; or where interactions between therapists, doctors and patients differ between centres; and when assessing the implementation of rehabilitation. In addition, BCTs may deal with ethical issues, e.g. the acceptability of interventions, more easily than RCTs. Recommendations regarding the different approaches (RCTs or BCTs) should be provided by the scientific rehabilitation societies. Concern over the validity of BCTs was considered justified, as the validity criteria of BCTs cover all those related to RCTs and include the risk of selection bias between treatment arms. Appropriate description of the essentials of the study object, including adequate description of how the interventions were actualized in comparison to the study plan, are essential features for a valid and generalizable study for both RCTs and BCTs. BCTs are necessary to widen the evidence-base of effectiveness in rehabilitation. It was suggested that the rehabilitation field should support the concept of BCTs. It was proposed that education regarding BCTs is indicated, and stakeholders need to be convinced that BCTs are a valid alternative to RCTs. EARM and other physical and rehabilitation medicine (PRM) bodies could advance the use of BCTs for clinical and health policy decision-making.