The potential of decision trees as a tool to simplify broiler chicken welfare assessments.

To simplify fast-growth broiler welfare assessments and use them as a benchmarking tool, decision trees were used to identify iceberg indicators discriminating flocks passing/failing welfare assessments as with the complete AWIN protocol. A dataset was constructed with data from 57 flocks and 3 previous projects. A final flock assessment score, previously not included in the dataset, was calculated and used as the benchmarking assessment classifier (pass/fail). A decision tree to classify flocks was built using the Chi-square Automatic Interaction Detection (CHAID) criterion. Cost-complexity pruning, and tenfold cross-validation were used. The final decision tree included cumulative mortality (%), immobile, lame birds (%), and birds with back wounds (%). Values were (mean ± se) 2.77 ± 0.14%, 0.16 ± 0.02%, 0.25 ± 0.02%, and 0.003 ± 0.001% for flocks passing the assessment; and 4.39 ± 0.49%, 0.24 ± 0.05%, 0.49 ± 0.09%, and 0.015 ± 0.006% for flocks failing. Cumulative mortality had the highest relative importance. The validated model correctly predicted 80.70% of benchmarking assessment outcomes. Model specificity was 0.8696; sensitivity was 0.5455. Decision trees can be useful to simplify welfare assessments. Model improvements will be possible as more information becomes available, and predictions are based on more samples.

Comparison and benchmark of deep learning methods for non-coding RNA classification.

The involvement of non-coding RNAs in biological processes and diseases has made the exploration of their functions crucial. Most non-coding RNAs have yet to be studied, creating the need for methods that can rapidly classify large sets of non-coding RNAs into functional groups, or classes. In recent years, the success of deep learning in various domains led to its application to non-coding RNA classification. Multiple novel architectures have been developed, but these advancements are not covered by current literature reviews. We present an exhaustive comparison of the different methods proposed in the state-of-the-art and describe their associated datasets. Moreover, the literature lacks objective benchmarks. We perform experiments to fairly evaluate the performance of various tools for non-coding RNA classification on popular datasets. The robustness of methods to non-functional sequences and sequence boundary noise is explored. We also measure computation time and CO2 emissions. With regard to these results, we assess the relevance of the different architectural choices and provide recommendations to consider in future methods.

Benchmarking transcriptome deconvolution methods for estimating tissue- and cell-type-specific extracellular vesicle abundances.

Extracellular vesicles (EVs) contain cell-derived lipids, proteins and RNAs; however, determining the tissue- and cell-type-specific EV abundances in body fluids remains a significant hurdle for our understanding of EV biology. While tissue- and cell-type-specific EV abundances can be estimated by matching the EV's transcriptome to a tissue's/cell type's expression signature using deconvolutional methods, a comparative assessment of deconvolution methods' performance on EV transcriptome data is currently lacking. We benchmarked 11 deconvolution methods using data from four cell lines and their EVs, in silico mixtures, 118 human plasma and 88 urine EVs. We identified deconvolution methods that estimated cell type-specific abundances of pure and in silico mixed cell line-derived EV samples with high accuracy. Using data from two urine EV cohorts with different EV isolation procedures, four deconvolution methods produced highly similar results. The three methods were also concordant in their tissue- and cell-type-specific plasma EV abundance estimates. We identified driving factors for deconvolution accuracy and highlighted the importance of implementing biological knowledge in creating the tissue/cell type signature. Overall, our analyses demonstrate that the deconvolution algorithms DWLS and CIBERSORTx produce highly similar and accurate estimates of tissue- and cell-type-specific EV abundances in biological fluids.

Embedding Water-Energy-Food nexus and circularity assessment for organization benchmarking: A case study for dairy farms.

The Water-Energy-Food (WEF) nexus approach is increasingly being used for supporting a transition to sustainable development, with initiatives involving the concept of circular economy (CE). In the agricultural sector in particular, assessing this nexus is crucial to ensure food security, control the consumption of key resources such as water and energy, as well as measure atmospheric emissions linked to climate change. This manuscript aims to propose a novel approach by coupling the WEF nexus with a circularity indicator, seeking to capture in a single index (the WEF+CEi) both performances in a sample of companies. The novel approach is applied to 30 dairy farms located in Galicia (NW Spain) to benchmark them in a holistic manner. To do this, the WEF nexus of each farm was represented through the following indicators: carbon footprint, water footprint, energy footprint, and food productivity. In addition, the percentage of circularity for each farm, and for the agro-industrial cooperative was measured thanks to the application of a circularity tool in percentage terms. Finally, the WEF+CEi indicator was obtained using the multicriteria mathematical tool of Data Envelopment Analysis (DEA). The results show that without considering the agro-industrial cooperative, the system is 51 % circular. On the other hand, considering the farms and the cooperative, the system goes up to 80 % of circularity. Finally, the proposed approach can support decision-making and provide insights for producers and stakeholders in the area.

Benchmarking of AlphaFold2 accuracy self-estimates as indicators of empirical model quality and ranking: a comparison with independent model quality assessment programmes.

MOTIVATION: Despite an increase in protein modelling accuracy following the development of AlphaFold2, there remains an accuracy gap between predicted and observed model quality assessment (MQA) scores. In CASP15, variations in AlphaFold2 model accuracy prediction were noticed for quaternary models of very similar observed quality. In this study, we compare plDDT and pTM to their observed counterparts the local distance difference test (lDDT) and TM-score for both tertiary and quaternary models to examine whether reliability is retained across the scoring range under normal modelling conditions and in situations where AlphaFold2 functionality is customized. We also explore plDDT and pTM ranking accuracy in comparison with the published independent MQA programmes ModFOLD9 and ModFOLDdock. RESULTS: plDDT was found to be an accurate descriptor of tertiary model quality compared to observed lDDT-Cα scores (Pearson r = 0.97), and achieved a ranking agreement true positive rate (TPR) of 0.34 with observed scores, which ModFOLD9 could not improve. However, quaternary structure accuracy was reduced (plDDT r = 0.67, pTM r = 0.70) and significant overprediction was seen with both scores for some lower quality models. Additionally, ModFOLDdock was able to improve upon AF2-Multimer model ranking compared to TM-score (TPR 0.34) and oligo-lDDT score (TPR 0.43). Finally, evidence is presented for increased variability in plDDT and pTM when using custom template recycling, which is more pronounced for quaternary structures. AVAILABILITY AND IMPLEMENTATION: The ModFOLD9 and ModFOLDdock quality assessment servers are available at https://www.reading.ac.uk/bioinf/ModFOLD/ and https://www.reading.ac.uk/bioinf/ModFOLDdock/, respectively. A docker image is available at https://hub.docker.com/r/mcguffin/multifold.

Navigation benchmarking for autonomous mobile robots in hospital environment.

The widespread adoption of robotic technologies in healthcare has opened up new perspectives for enhancing accuracy, effectiveness and quality of medical procedures and patients' care. Special attention has been given to the reliability of robots when operating in environments shared with humans and to the users' safety, especially in case of mobile platforms able to navigate autonomously. From the analysis of the literature, it emerges that navigation tests carried out in a hospital environment are preliminary and not standardized. This paper aims to overcome the limitations in the assessment of autonomous mobile robots navigating in hospital environments by proposing: (i) a structured benchmarking protocol composed of a set of standardized tests, taking into account conditions with increasing complexity, (ii) a set of quantitative performance metrics. The proposed approach has been used in a realistic setting to assess the performance of two robotic platforms, namely HOSBOT and TIAGo, with different technical features and developed for different applications in a clinical scenario.

Benchmarking Emergency Physician EHR Time per Encounter Based on Patient and Clinical Factors.

This cross-sectional study assesses the associations between patient and clinical factors and variations in time emergency department physicians spend using electronic health record (EHR) systems.

Benchmarking robustness of deep neural networks in semantic segmentation of fluorescence microscopy images.

BACKGROUND: Fluorescence microscopy (FM) is an important and widely adopted biological imaging technique. Segmentation is often the first step in quantitative analysis of FM images. Deep neural networks (DNNs) have become the state-of-the-art tools for image segmentation. However, their performance on natural images may collapse under certain image corruptions or adversarial attacks. This poses real risks to their deployment in real-world applications. Although the robustness of DNN models in segmenting natural images has been studied extensively, their robustness in segmenting FM images remains poorly understood RESULTS: To address this deficiency, we have developed an assay that benchmarks robustness of DNN segmentation models using datasets of realistic synthetic 2D FM images with precisely controlled corruptions or adversarial attacks. Using this assay, we have benchmarked robustness of ten representative models such as DeepLab and Vision Transformer. We find that models with good robustness on natural images may perform poorly on FM images. We also find new robustness properties of DNN models and new connections between their corruption robustness and adversarial robustness. To further assess the robustness of the selected models, we have also benchmarked them on real microscopy images of different modalities without using simulated degradation. The results are consistent with those obtained on the realistic synthetic images, confirming the fidelity and reliability of our image synthesis method as well as the effectiveness of our assay. CONCLUSIONS: Based on comprehensive benchmarking experiments, we have found distinct robustness properties of deep neural networks in semantic segmentation of FM images. Based on the findings, we have made specific recommendations on selection and design of robust models for FM image segmentation.

Benchmarking brain-computer interface algorithms: Riemannian approaches vs convolutional neural networks.

Objective.To date, a comprehensive comparison of Riemannian decoding methods with deep convolutional neural networks for EEG-based brain-computer interfaces remains absent from published work. We address this research gap by using MOABB, The Mother Of All BCI Benchmarks, to compare novel convolutional neural networks to state-of-the-art Riemannian approaches across a broad range of EEG datasets, including motor imagery, P300, and steady-state visual evoked potentials paradigms.Approach.We systematically evaluated the performance of convolutional neural networks, specifically EEGNet, shallow ConvNet, and deep ConvNet, against well-established Riemannian decoding methods using MOABB processing pipelines. This evaluation included within-session, cross-session, and cross-subject methods, to provide a practical analysis of model effectiveness and to find an overall solution that performs well across different experimental settings.Main results.We find no significant differences in decoding performance between convolutional neural networks and Riemannian methods for within-session, cross-session, and cross-subject analyses.Significance.The results show that, when using traditional Brain-Computer Interface paradigms, the choice between CNNs and Riemannian methods may not heavily impact decoding performances in many experimental settings. These findings provide researchers with flexibility in choosing decoding approaches based on factors such as ease of implementation, computational efficiency or individual preferences.

A Comparison of Structural Variant Calling from Short-Read and Nanopore-Based Whole-Genome Sequencing Using Optical Genome Mapping as a Benchmark.

The identification of structural variants (SVs) in genomic data represents an ongoing challenge because of difficulties in reliable SV calling leading to reduced sensitivity and specificity. We prepared high-quality DNA from 9 parent-child trios, who had previously undergone short-read whole-genome sequencing (Illumina platform) as part of the Genomics England 100,000 Genomes Project. We reanalysed the genomes using both Bionano optical genome mapping (OGM; 8 probands and one trio) and Nanopore long-read sequencing (Oxford Nanopore Technologies [ONT] platform; all samples). To establish a "truth" dataset, we asked whether rare proband SV calls (n = 234) made by the Bionano Access (version 1.6.1)/Solve software (version 3.6.1_11162020) could be verified by individual visualisation using the Integrative Genomics Viewer with either or both of the Illumina and ONT raw sequence. Of these, 222 calls were verified, indicating that Bionano OGM calls have high precision (positive predictive value 95%). We then asked what proportion of the 222 true Bionano SVs had been identified by SV callers in the other two datasets. In the Illumina dataset, sensitivity varied according to variant type, being high for deletions (115/134; 86%) but poor for insertions (13/58; 22%). In the ONT dataset, sensitivity was generally poor using the original Sniffles variant caller (48% overall) but improved substantially with use of Sniffles2 (36/40; 90% and 17/23; 74% for deletions and insertions, respectively). In summary, we show that the precision of OGM is very high. In addition, when applying the Sniffles2 caller, the sensitivity of SV calling using ONT long-read sequence data outperforms Illumina sequencing for most SV types.

A real-world multi-center RNA-seq benchmarking study using the Quartet and MAQC reference materials.

Translating RNA-seq into clinical diagnostics requires ensuring the reliability and cross-laboratory consistency of detecting clinically relevant subtle differential expressions, such as those between different disease subtypes or stages. As part of the Quartet project, we present an RNA-seq benchmarking study across 45 laboratories using the Quartet and MAQC reference samples spiked with ERCC controls. Based on multiple types of 'ground truth', we systematically assess the real-world RNA-seq performance and investigate the influencing factors involved in 26 experimental processes and 140 bioinformatics pipelines. Here we show greater inter-laboratory variations in detecting subtle differential expressions among the Quartet samples. Experimental factors including mRNA enrichment and strandedness, and each bioinformatics step, emerge as primary sources of variations in gene expression. We underscore the profound influence of experimental execution, and provide best practice recommendations for experimental designs, strategies for filtering low-expression genes, and the optimal gene annotation and analysis pipelines. In summary, this study lays the foundation for developing and quality control of RNA-seq for clinical diagnostic purposes.

BioCoder: a benchmark for bioinformatics code generation with large language models.

SUMMARY: Pretrained large language models (LLMs) have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Here, we target bioinformatics due to the amount of domain knowledge, algorithms, and data operations this discipline requires. We present BioCoder, a benchmark developed to evaluate LLMs in generating bioinformatics-specific code. BioCoder spans much of the field, covering cross-file dependencies, class declarations, and global variables. It incorporates 1026 Python functions and 1243 Java methods extracted from GitHub, along with 253 examples from the Rosalind Project, all pertaining to bioinformatics. Using topic modeling, we show that the overall coverage of the included code is representative of the full spectrum of bioinformatics calculations. BioCoder incorporates a fuzz-testing framework for evaluation. We have applied it to evaluate various models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT-4. Furthermore, we fine-tuned one model (StarCoder), demonstrating that our training dataset can enhance the performance on our testing benchmark (by >15% in terms of Pass@K under certain prompt configurations and always >3%). The results highlight two key aspects of successful models: (i) Successful models accommodate a long prompt (>2600 tokens) with full context, including functional dependencies. (ii) They contain domain-specific knowledge of bioinformatics, beyond just general coding capability. This is evident from the performance gain of GPT-3.5/4 compared to the smaller models on our benchmark (50% versus up to 25%). AVAILABILITY AND IMPLEMENTATION: All datasets, benchmark, Docker images, and scripts required for testing are available at: https://github.com/gersteinlab/biocoder and https://biocoder-benchmark.github.io/.

Associations Between Radiation Oncologist Demographic Factors and Segmentation Similarity Benchmarks: Insights From a Crowd-Sourced Challenge Using Bayesian Estimation.

PURPOSE: The quality of radiotherapy auto-segmentation training data, primarily derived from clinician observers, is of utmost importance. However, the factors influencing the quality of clinician-derived segmentations are poorly understood; our study aims to quantify these factors. METHODS: Organ at risk (OAR) and tumor-related segmentations provided by radiation oncologists from the Contouring Collaborative for Consensus in Radiation Oncology data set were used. Segmentations were derived from five disease sites: breast, sarcoma, head and neck (H&N), gynecologic (GYN), and GI. Segmentation quality was determined on a structure-by-structure basis by comparing the observer segmentations with an expert-derived consensus, which served as a reference standard benchmark. The Dice similarity coefficient (DSC) was primarily used as a metric for the comparisons. DSC was stratified into binary groups on the basis of structure-specific expert-derived interobserver variability (IOV) cutoffs. Generalized linear mixed-effects models using Bayesian estimation were used to investigate the association between demographic variables and the binarized DSC for each disease site. Variables with a highest density interval excluding zero were considered to substantially affect the outcome measure. RESULTS: Five hundred seventy-four, 110, 452, 112, and 48 segmentations were used for the breast, sarcoma, H&N, GYN, and GI cases, respectively. The median percentage of segmentations that crossed the expert DSC IOV cutoff when stratified by structure type was 55% and 31% for OARs and tumors, respectively. Regression analysis revealed that the structure being tumor-related had a substantial negative impact on binarized DSC for the breast, sarcoma, H&N, and GI cases. There were no recurring relationships between segmentation quality and demographic variables across the cases, with most variables demonstrating large standard deviations. CONCLUSION: Our study highlights substantial uncertainty surrounding conventionally presumed factors influencing segmentation quality relative to benchmarks.

Dyport: dynamic importance-based biomedical hypothesis generation benchmarking technique.

BACKGROUND: Automated hypothesis generation (HG) focuses on uncovering hidden connections within the extensive information that is publicly available. This domain has become increasingly popular, thanks to modern machine learning algorithms. However, the automated evaluation of HG systems is still an open problem, especially on a larger scale. RESULTS: This paper presents a novel benchmarking framework Dyport for evaluating biomedical hypothesis generation systems. Utilizing curated datasets, our approach tests these systems under realistic conditions, enhancing the relevance of our evaluations. We integrate knowledge from the curated databases into a dynamic graph, accompanied by a method to quantify discovery importance. This not only assesses hypotheses accuracy but also their potential impact in biomedical research which significantly extends traditional link prediction benchmarks. Applicability of our benchmarking process is demonstrated on several link prediction systems applied on biomedical semantic knowledge graphs. Being flexible, our benchmarking system is designed for broad application in hypothesis generation quality verification, aiming to expand the scope of scientific discovery within the biomedical research community. CONCLUSIONS: Dyport is an open-source benchmarking framework designed for biomedical hypothesis generation systems evaluation, which takes into account knowledge dynamics, semantics and impact. All code and datasets are available at: https://github.com/IlyaTyagin/Dyport .

Benchmarking NGS-Based HLA Typing Algorithms.

Knowledge of the expected accuracy of HLA typing algorithms is important when choosing between algorithms and when evaluating the HLA typing predictions of an algorithm. This chapter guides the reader through an example benchmarking study that evaluates the performances of four NGS-based HLA typing algorithms as well as outlining factors to consider, when designing and running such a benchmarking study. The code related to this benchmarking workflow can be found at https://github.com/nikolasthuesen/springers-hla-benchmark/ .

Leveraging the potential of the German operating room benchmarking initiative for planning: A ready-to-use surgical process data set.

We present a freely available data set of surgical case mixes and surgery process duration distributions based on processed data from the German Operating Room Benchmarking initiative. This initiative collects surgical process data from over 320 German, Austrian, and Swiss hospitals. The data exhibits high levels of quantity, quality, standardization, and multi-dimensionality, making it especially valuable for operating room planning in Operations Research. We consider detailed steps of the perioperative process and group the data with respect to the hospital's level of care, the surgery specialty, and the type of surgery patient. We compare case mixes for different subgroups and conclude that they differ significantly, demonstrating that it is necessary to test operating room planning methods in different settings, e.g., using data sets like ours. Further, we discuss limitations and future research directions. Finally, we encourage the extension and foundation of new operating room benchmarking initiatives and their usage for operating room planning.

The evolving regulatory system of advanced therapy medicinal products in China: a documentary analysis using the World Health Organization Global Benchmarking Tool standards.

Advanced therapy medicinal products (ATMPs) are rapidly evolving to offer new treatment options. The scientific, technical, and clinical complexities subject drug regulatory authorizes to regulatory challenges. To advance the regulatory capacity for ATMPs, the National Medical Products Administration in China made changes to the drug regulatory system and developed regulatory science with the goal of addressing patient needs and encouraging innovation. This study aimed to systematically identify the regulatory evidence on ATMPs in China under the guidance of an overarching framework from the World Health Organization Global Benchmarking Tool. It was found that China's administrative authorities at all levels have issued a number of policy documents to promote the development of ATMPs, covering biopharmaceutical products research and development (n = 14), biopharmaceutical industry development (n = 9), high-quality development of medical institutions (n = 1), specific development plans/projects (n = 6) and specific regional development (n = 4). The legal and regulatory framework of ATMPs in China has been established and is subject to continuous adjustment in various aspects including regulations (n = 3), departmental rules or administrative normative documents (n = 22), and technical guidance (n = 15). As the regulatory reform continues, the drug review processes have been revised, and various technical standards have been launched, which aim to establish a regulatory approach that oversees the full life-cycle development of ATMPs in the country. The limited number of investigational new drug applications and approved ATMPs suggests a lag remains between the translation of advanced therapeutic technologies into clinically available medical products. To accelerate the translational research of ATMP in countries such as China, developing and adopting real-world evidence generated from clinical use in designated healthcare facilities to support scientific decision-making in ATMP regulation is warranted. The enhancement of regulatory capacity building and multi-stakeholder collaborations should also be encouraged to facilitate the timely evaluation of promising ATMPs to meet more patient needs.

Creating and validating the Fine-Grained Question Subjectivity Dataset (FQSD): A new benchmark for enhanced automatic subjective question answering systems.

In the domain of question subjectivity classification, there exists a need for detailed datasets that can foster advancements in Automatic Subjective Question Answering (ASQA) systems. Addressing the prevailing research gaps, this paper introduces the Fine-Grained Question Subjectivity Dataset (FQSD), which comprises 10,000 questions. The dataset distinguishes between subjective and objective questions and offers additional categorizations such as Subjective-types (Target, Attitude, Reason, Yes/No, None) and Comparison-form (Single, Comparative). Annotation reliability was confirmed via robust evaluation techniques, yielding a Fleiss's Kappa score of 0.76 and Pearson correlation values up to 0.80 among three annotators. We benchmarked FQSD against existing datasets such as (Yu, Zha, and Chua 2012), SubjQA (Bjerva 2020), and ConvEx-DS (Hernandez-Bocanegra 2021). Our dataset excelled in scale, linguistic diversity, and syntactic complexity, establishing a new standard for future research. We employed visual methodologies to provide a nuanced understanding of the dataset and its classes. Utilizing transformer-based models like BERT, XLNET, and RoBERTa for validation, RoBERTa achieved an outstanding F1-score of 97%, confirming the dataset's efficacy for the advanced subjectivity classification task. Furthermore, we utilized Local Interpretable Model-agnostic Explanations (LIME) to elucidate model decision-making, ensuring transparent and reliable model predictions in subjectivity classification tasks.

NKUT: Dataset and Benchmark for Pediatric Mandibular Wisdom Teeth Segmentation.

Germectomy is a common surgery in pediatric dentistry to prevent the potential dangers caused by impacted mandibular wisdom teeth. Segmentation of mandibular wisdom teeth is a crucial step in surgery planning. However, manually segmenting teeth and bones from 3D volumes is time-consuming and may cause delays in treatment. Deep learning based medical image segmentation methods have demonstrated the potential to reduce the burden of manual annotations, but they still require a lot of well-annotated data for training. In this paper, we initially curated a Cone Beam Computed Tomography (CBCT) dataset, NKUT, for the segmentation of pediatric mandibular wisdom teeth. This marks the first publicly available dataset in this domain. Second, we propose a semantic separation scale-specific feature fusion network named WTNet, which introduces two branches to address the teeth and bones segmentation tasks. In WTNet, We design a Input Enhancement (IE) block and a Teeth-Bones Feature Separation (TBFS) block to solve the feature confusions and semantic-blur problems in our task. Experimental results suggest that WTNet performs better on NKUT compared to previous state-of-the-art segmentation methods (such as TransUnet), with a maximum DSC lead of nearly 16%.

Benchmarking kinship estimation tools for ancient genomes using pedigree simulations.

There is growing interest in uncovering genetic kinship patterns in past societies using low-coverage palaeogenomes. Here, we benchmark four tools for kinship estimation with such data: lcMLkin, NgsRelate, KIN, and READ, which differ in their input, IBD estimation methods, and statistical approaches. We used pedigree and ancient genome sequence simulations to evaluate these tools when only a limited number (1 to 50 K, with minor allele frequency ≥0.01) of shared SNPs are available. The performance of all four tools was comparable using ≥20 K SNPs. We found that first-degree related pairs can be accurately classified even with 1 K SNPs, with 85% F1 scores using READ and 96% using NgsRelate or lcMLkin. Distinguishing third-degree relatives from unrelated pairs or second-degree relatives was also possible with high accuracy (F1 > 90%) with 5 K SNPs using NgsRelate and lcMLkin, while READ and KIN showed lower success (69 and 79% respectively). Meanwhile, noise in population allele frequencies and inbreeding (first-cousin mating) led to deviations in kinship coefficients, with different sensitivities across tools. We conclude that using multiple tools in parallel might be an effective approach to achieve robust estimates on ultra-low-coverage genomes.