Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

The Gr64 cluster of gustatory receptors promotes survival and proteostasis of epithelial cells in Drosophila.

Gustatory Receptor 64 (Gr64) genes are a cluster of 6 neuronally expressed receptors involved in sweet taste sensation in Drosophila melanogaster. Gr64s modulate calcium signalling and excitatory responses to several different sugars. Here, we discover an unexpected nonneuronal function of Gr64 receptors and show that they promote proteostasis in epithelial cells affected by proteotoxic stress. Using heterozygous mutations in ribosome proteins (Rp), which have recently been shown to induce proteotoxic stress and protein aggregates in cells, we show that Rp/+ cells in Drosophila imaginal discs up-regulate expression of the entire Gr64 cluster and depend on these receptors for survival. We further show that loss of Gr64 in Rp/+ cells exacerbates stress pathway activation and proteotoxic stress by negatively affecting autophagy and proteasome function. This work identifies a noncanonical role in proteostasis maintenance for a family of gustatory receptors known for their function in neuronal sensation.

Systematic review and cumulative analysis of clinical properties of BRAF V600E mutations in PLNTY histological samples.

PURPOSE: Polymorphous low-grade neuroepithelial tumors of the young (PLNTY) represent a rare pediatric-type tumor that most commonly presents as medically refractory epilepsy. PLNTY has only recently been recognized as a distinct clinical entity, having been first described in 2016 and added to the World Health Organization classification of CNS tumors in 2021. Molecular studies have determined that PLNTY is uniformly driven by aberrant MAPK pathway activation, with most tumors carrying either a BRAF V600E mutation or activating FGFR2 or FGFR3 fusion protein. Although it is known that these driver mutations are mutually exclusive, little is known about differences in clinical presentation or treatment outcomes between PLNTY cases driven by these distinct mutations. METHODS: We performed a systematic review and cumulative analysis of PLNTY cases to assess whether or not PLNTY tumors carrying the BRAF V600E mutation exhibit different clinical behaviors. By searching the literature for all cases of PLNTY wherein BRAF V600E status was characterized, we compiled a dataset of 62 unique patient instances. Using a logistic regression-based approach, we assessed a primary outcome of what factors of a clinical presentation were associated with BRAF V600E mutations and a secondary outcome of what factors predicted total seizure freedom post-surgical resection. RESULTS: PLNTY cases carrying BRAF V600E mutations in the literature were strongly positively associated with adult patients (p = 0.0055, OR = 6.556; 95% Conf. Int. = 1.737-24.742). BRAF V600E status was also positively associated with tumor involvement of the temporal lobe (p = 0.0046, OR = 11.036; 95% Conf. Int. = 2.100-58.006). Male sex was also positively associated with BRAF V600E status, but the result did not quite achieve statistical significance (p = 0.0731). BRAF V600E status was not found to be associated with post-operative seizure freedom. CONCLUSIONS: These findings indicate that BRAF V600E-positive PLNTY exhibit characteristic clinical presentations but are not necessarily different in treatment responsiveness. Non-BRAF V600E tumors are more commonly associated with young patients.

Xrp1 and Irbp18 trigger a feed-forward loop of proteotoxic stress to induce the loser status.

Cell competition induces the elimination of less-fit "loser" cells by fitter "winner" cells. In Drosophila, cells heterozygous mutant in ribosome genes, Rp/+, known as Minutes, are outcompeted by wild-type cells. Rp/+ cells display proteotoxic stress and the oxidative stress response, which drive the loser status. Minute cell competition also requires the transcription factors Irbp18 and Xrp1, but how these contribute to the loser status is partially understood. Here we provide evidence that initial proteotoxic stress in RpS3/+ cells is Xrp1-independent. However, Xrp1 is sufficient to induce proteotoxic stress in otherwise wild-type cells and is necessary for the high levels of proteotoxic stress found in RpS3/+ cells. Surprisingly, Xrp1 is also induced downstream of proteotoxic stress, and is required for the competitive elimination of cells suffering from proteotoxic stress or overexpressing Nrf2. Our data suggests that a feed-forward loop between Xrp1, proteotoxic stress, and Nrf2 drives Minute cells to become losers.

Combining antibiotics with antivirulence compounds can have synergistic effects and reverse selection for antibiotic resistance in Pseudomonas aeruginosa.

Antibiotics are losing efficacy due to the rapid evolution and spread of resistance. Treatments targeting bacterial virulence factors have been considered as alternatives because they target virulence instead of pathogen viability, and should therefore exert weaker selection for resistance than conventional antibiotics. However, antivirulence treatments rarely clear infections, which compromises their clinical applications. Here, we explore the potential of combining antivirulence drugs with antibiotics against the opportunistic human pathogen Pseudomonas aeruginosa. We combined two antivirulence compounds (gallium, a siderophore quencher, and furanone C-30, a quorum sensing [QS] inhibitor) together with four clinically relevant antibiotics (ciprofloxacin, colistin, meropenem, tobramycin) in 9×9 drug concentration matrices. We found that drug-interaction patterns were concentration dependent, with promising levels of synergies occurring at intermediate drug concentrations for certain drug pairs. We then tested whether antivirulence compounds are potent adjuvants, especially when treating antibiotic resistant (AtbR) clones. We found that the addition of antivirulence compounds to antibiotics could restore growth inhibition for most AtbR clones, and even abrogate or reverse selection for resistance in five drug combination cases. Molecular analyses suggest that selection against resistant clones occurs when resistance mechanisms involve restoration of protein synthesis, but not when efflux pumps are up-regulated. Altogether, our work provides a first systematic analysis of antivirulence-antibiotic combinatorial treatments and suggests that such combinations have the potential to be both effective in treating infections and in limiting the spread of antibiotic resistance.

Resident microbial communities inhibit growth and antibiotic-resistance evolution of Escherichia coli in human gut microbiome samples.

Countering the rise of antibiotic-resistant pathogens requires improved understanding of how resistance emerges and spreads in individual species, which are often embedded in complex microbial communities such as the human gut microbiome. Interactions with other microorganisms in such communities might suppress growth and resistance evolution of individual species (e.g., via resource competition) but could also potentially accelerate resistance evolution via horizontal transfer of resistance genes. It remains unclear how these different effects balance out, partly because it is difficult to observe them directly. Here, we used a gut microcosm approach to quantify the effect of three human gut microbiome communities on growth and resistance evolution of a focal strain of Escherichia coli. We found the resident microbial communities not only suppressed growth and colonisation by focal E. coli but also prevented it from evolving antibiotic resistance upon exposure to a beta-lactam antibiotic. With samples from all three human donors, our focal E. coli strain only evolved antibiotic resistance in the absence of the resident microbial community, even though we found resistance genes, including a highly effective resistance plasmid, in resident microbial communities. We identified physical constraints on plasmid transfer that can explain why our focal strain failed to acquire some of these beneficial resistance genes, and we found some chromosomal resistance mutations were only beneficial in the absence of the resident microbiota. This suggests, depending on in situ gene transfer dynamics, interactions with resident microbiota can inhibit antibiotic-resistance evolution of individual species.

Bone marrow fibrosis is associated with non-response to CD19 CAR T-cell therapy in B-acute lymphoblastic leukemia.

CD19 directed CAR T-cell therapy is used to treat relapsed/refractory B-cell acute lymphoblastic leukemia. The role of the pre-CAR bone marrow (BM) stromal microenvironment in determining response to CAR T-cell therapy has been understudied. We performed whole transcriptome analysis, reticulin fibrosis assessment and CD3 T-cell infiltration on BM core biopsies from pre- and post-CAR timepoints for 61 patients, as well as on a cohort of 54 primary B-ALL samples. Pathways of fibrosis, extracellular matrix development, and associated transcription factors AP1 and TGF-ß3, were enriched and upregulated in nonresponders (NR) even prior to CAR T cell therapy. NR showed significantly higher levels of BM fibrosis compared to complete responders by both clinical reticulin assessment and AI-assisted digital image scoring. CD3+ T cells showed a trend toward lower infiltration in NR. NR had significantly higher levels of pre-CAR fibrosis compared to primary B-ALL. High levels of fibrosis were associated with lower overall survival after CAR T-cell therapy. In conclusion, BM fibrosis is a novel mechanism mediating nonresponse to CD19-directed CAR T-cell therapy in B-ALL. A widely used clinically assay for quantitating myelofibrosis can be repurposed to determine patients at high risk of non-response. Genes and pathways associated with BM fibrosis are a potential target to improve response.

Configurational Causal Modeling and Logic Regression.

Configurational comparative methods (CCMs) and logic regression methods (LRMs) are two families of exploratory methods that employ very different techniques to analyze data generated by causal structures featuring conjunctural causation and equifinality. Aiming for the same by different means carries a substantive synergy potential, which, however, remains untapped so far because representatives of the two frameworks know little of each other. The purpose of this article is to change that. We first level the field for readers from both backgrounds by providing brief introductions to the basic ideas behind CCMs and LRMs. Then, we carve out the strengths and weaknesses of the two method families by benchmarking their performance when applied to binary data under a variety of different discovery contexts. It turns out that CCMs and LRMs have complementary strengths and weaknesses. This creates various promising avenues for cross-validation.

Comparing methods of detecting and segmenting unruptured intracranial aneurysms on TOF-MRAS: The ADAM challenge.

Accurate detection and quantification of unruptured intracranial aneurysms (UIAs) is important for rupture risk assessment and to allow an informed treatment decision to be made. Currently, 2D manual measures used to assess UIAs on Time-of-Flight magnetic resonance angiographies (TOF-MRAs) lack 3D information and there is substantial inter-observer variability for both aneurysm detection and assessment of aneurysm size and growth. 3D measures could be helpful to improve aneurysm detection and quantification but are time-consuming and would therefore benefit from a reliable automatic UIA detection and segmentation method. The Aneurysm Detection and segMentation (ADAM) challenge was organised in which methods for automatic UIA detection and segmentation were developed and submitted to be evaluated on a diverse clinical TOF-MRA dataset. A training set (113 cases with a total of 129 UIAs) was released, each case including a TOF-MRA, a structural MR image (T1, T2 or FLAIR), annotation of any present UIA(s) and the centre voxel of the UIA(s). A test set of 141 cases (with 153 UIAs) was used for evaluation. Two tasks were proposed: (1) detection and (2) segmentation of UIAs on TOF-MRAs. Teams developed and submitted containerised methods to be evaluated on the test set. Task 1 was evaluated using metrics of sensitivity and false positive count. Task 2 was evaluated using dice similarity coefficient, modified hausdorff distance (95th percentile) and volumetric similarity. For each task, a ranking was made based on the average of the metrics. In total, eleven teams participated in task 1 and nine of those teams participated in task 2. Task 1 was won by a method specifically designed for the detection task (i.e. not participating in task 2). Based on segmentation metrics, the top two methods for task 2 performed statistically significantly better than all other methods. The detection performance of the top-ranking methods was comparable to visual inspection for larger aneurysms. Segmentation performance of the top ranking method, after selection of true UIAs, was similar to interobserver performance. The ADAM challenge remains open for future submissions and improved submissions, with a live leaderboard to provide benchmarking for method developments at https://adam.isi.uu.nl/.

Microbial community composition interacts with local abiotic conditions to drive colonization resistance in human gut microbiome samples.

Biological invasions can alter ecosystem stability and function, and predicting what happens when a new species or strain arrives remains a major challenge in ecology. In the mammalian gastrointestinal tract, susceptibility of the resident microbial community to invasion by pathogens has important implications for host health. However, at the community level, it is unclear whether susceptibility to invasion depends mostly on resident community composition (which microbes are present), or also on local abiotic conditions (such as nutrient status). Here, we used a gut microcosm system to disentangle some of the drivers of susceptibility to invasion in microbial communities sampled from humans. We found resident microbial communities inhibited an invading Escherichia coli strain, compared to community-free control treatments, sometimes excluding the invader completely (colonization resistance). These effects were stronger at later time points, when we also detected altered community composition and nutrient availability. By separating these two components (microbial community and abiotic environment), we found taxonomic composition played a crucial role in suppressing invasion, but this depended critically on local abiotic conditions (adapted communities were more suppressive in nutrient-depleted conditions). This helps predict when resident communities will be most susceptible to invasion, with implications for optimizing treatments based on microbiota management.

Associations between sensitivity to antibiotics, disinfectants and heavy metals in natural, clinical and laboratory isolates of Escherichia coli.

Bacteria in nature often encounter non-antibiotic antibacterials (NAAs), such as disinfectants and heavy metals, and they can evolve resistance via mechanisms that are also involved in antibiotic resistance. Understanding whether susceptibility to different types of antibacterials is non-randomly associated across natural and clinical bacteria is therefore important for predicting the spread of resistance, yet there is no consensus about the extent of such associations or underlying mechanisms. We tested for associations between susceptibility phenotypes of 93 natural and clinical Escherichia coli isolates to various NAAs and antibiotics. Across all compound combinations, we detected a small number of non-random associations, including a trio of positive associations among chloramphenicol, triclosan and benzalkonium chloride. We investigated genetic mechanisms that can explain such associations using genomic information, genetic knockouts and experimental evolution. This revealed some mutations that are selected for by experimental exposure to one compound and confer cross-resistance to other compounds. Surprisingly, these interactions were asymmetric: selection for chloramphenicol resistance conferred cross-resistance to triclosan and benzalkonium chloride, but selection for triclosan resistance did not confer cross-resistance to other compounds. These results identify genetic changes involved in variable cross-resistance across antibiotics and NAAs, potentially contributing to associations in natural and clinical bacteria.

Dual Oxygen and Temperature Luminescence Learning Sensor with Parallel Inference.

A well-known approach to the optical measure of oxygen is based on the quenching of luminescence by molecular oxygen. The main challenge for this measuring method is the determination of an accurate mathematical model for the sensor response. The reason is the dependence of the sensor signal from multiple parameters (like oxygen concentration and temperature), which are cross interfering in a sensor-specific way. The common solution is to measure the different parameters separately, for example, with different sensors. Then, an approximate model is developed where these effects are parametrized ad hoc. In this work, we describe a new approach for the development of a learning sensor with parallel inference that overcomes all these difficulties. With this approach we show how to generate automatically and autonomously a very large dataset of measurements and how to use it for the training of the proposed neural-network-based signal processing. Furthermore, we demonstrate how the sensor exploits the cross-sensitivity of multiple parameters to extract them from a single set of optical measurements without any a priori mathematical model with unprecedented accuracy. Finally, we propose a completely new metric to characterize the performance of neural-network-based sensors, the Error Limited Accuracy. In general, the methods described here are not limited to oxygen and temperature sensing. They can be similarly applied for the sensing with multiple luminophores, whenever the underlying mathematical model is not known or too complex.

Optical Oxygen Sensing with Artificial Intelligence.

Luminescence-based sensors for measuring oxygen concentration are widely used in both industry and research due to the practical advantages and sensitivity of this type of sensing. The measuring principle is the luminescence quenching by oxygen molecules, which results in a change of the luminescence decay time and intensity. In the classical approach, this change is related to an oxygen concentration using the Stern-Volmer equation. This equation, which in most cases is non-linear, is parameterized through device-specific constants. Therefore, to determine these parameters, every sensor needs to be precisely calibrated at one or more known concentrations. This study explored an entirely new artificial intelligence approach and demonstrated the feasibility of oxygen sensing through machine learning. The specifically developed neural network learns very efficiently to relate the input quantities to the oxygen concentration. The results show a mean deviation of the predicted from the measured concentration of 0.5% air, comparable to many commercial and low-cost sensors. Since the network was trained using synthetically generated data, the accuracy of the model predictions is limited by the ability of the generated data to describe the measured data, opening up future possibilities for significant improvement by using a large number of experimental measurements for training. The approach described in this work demonstrates the applicability of artificial intelligence to sensing technology and paves the road for the next generation of sensors.

Mn4+-Doped Magnesium Titanate-A Promising Phosphor for Self-Referenced Optical Temperature Sensing.

Phosphors based on magnesium titanate activated with Mn 4 + ions are of great interest because, when excited with blue light, they display a strong red-emitting luminescence. They are characterized by a luminescence decay which is strongly temperature dependent in the range from -50 ∘ C to 150 ∘ C, making these materials very promising for temperature sensing in the biochemical field. In this work, the optical and thermal properties of the luminescence of Mg 2 TiO 4 are investigated for different Mn 4 + doping concentrations. The potential of this material for temperature sensing is demonstrated by fabricating a fiber optic temperature microsensor and by comparing its performance against a standard resistance thermometer. The response of the fiber optic sensor is exceptionally fast, with a response time below 1 s in the liquid phase and below 1.1 s in the gas phase.

Advanced approaches in Pediatric Epilepsy surgery.

While recent technological advancements are reshaping the landscape of surgical epilepsy management, the established techniques of resective and disconnective surgeries guided by electrographic monitoring remain the workhorse interventions for the management of refractory seizures and have the highest likelihood of achieving complete seizure resolution. Here we discuss examples of recent developments in surgical approaches and techniques for resective and disconnective surgeries with discussion of their indications and potential advantages.

Burr Hole Hemispherotomy-Modification of Trans-Sylvian Peri-Insular Technique: 2-Dimensional Operative Video.

Trans-sylvian peri-insular hemispherotomy represents a functional hemispherectomy with minimal brain removal used for treatment of refractory hemispheric epilepsy.1 Exposure for this procedure is achieved by craniotomy. Refinement in the hemispherotomy technique, including trends toward minimizing cortical resection, has contributed to a substantial drop in complication rates.2 We present a refinement of this technique, allowing for complete hemispheric disconnection through a single burr hole. In this instance, this technique was applied in the case of a 4-year-old girl who presented with medically refractory epilepsy, which had developed on the first day of life due to a perinatal incomplete left middle cerebral artery stroke. Postoperatively, the patient experienced no worsening of her preexisting right-sided hemiparesis and remains seizure-free 18 months postoperatively, now off medication. While the trans-sylvian peri-insular hemispherotomy represents an established surgical technique, this is the first report of this procedure performed in a minimally invasive fashion through a single burr hole. Beyond the minimal incision and small aperture in the skull, seldom appreciated nuances of hemispheric disconnection are described and demonstrated, including amygdala disconnection, hippocampal tail disconnection directly into splenium disconnection, concomitant intermediate disconnection and callosotomy, and frontobasal disconnection landmarks. Consent was obtained from the patient's parents for the surgical procedure, use of outcome videos, and for publication of this video and associated materials. The participants and patient's parents consented to publication of their images and that of the patient.

Predictors of urine toxicology and other biologic specimen missingness in randomized trials of substance use disorders.

BACKGROUND: High levels of missing outcome data for biologically confirmed substance use (BCSU) threaten the validity of substance use disorder (SUD) clinical trials. Underlying attributes of clinical trials could explain BCSU missingness and identify targets for improved trial design. METHODS: We reviewed 21 clinical trials funded by the NIDA National Drug Abuse Treatment Clinical Trials Network (CTN) and published from 2005 to 2018 that examined pharmacologic and psychosocial interventions for SUD. We used configurational analysis-a Boolean algebra approach that identifies an attribute or combination of attributes predictive of an outcome-to identify trial design features and participant characteristics associated with high levels of BCSU missingness. Associations were identified by configuration complexity, consistency, coverage, and robustness. We limited results using a consistency threshold of 0.75 and summarized model fit using the product of consistency and coverage. RESULTS: For trial design features, the final solution consisted of two pathways: psychosocial treatment as a trial intervention OR larger trial arm size (complexity=2, consistency=0.79, coverage=0.93, robustness score=0.71). For participant characteristics, the final solution consisted of two pathways: interventions targeting individuals with poly- or nonspecific substance use OR younger age (complexity=2, consistency=0.75, coverage=0.86, robustness score=1.00). CONCLUSIONS: Psychosocial treatments, larger trial arm size, interventions targeting individuals with poly- or nonspecific substance use, and younger age among trial participants were predictive of missing BCSU data in SUD clinical trials. Interventions to mitigate missing data that focus on these attributes may reduce threats to validity and improve utility of SUD clinical trials.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Grazing resistant freshwater bacteria profit from chitin and cell-wall-derived organic carbon.

The rise of grazing resistant planktonic bacteria in freshwater lakes during vernal phytoplankton blooms is favoured by predation of heterotrophic nanoflagellates (HNF). The spring period is also characterized by increased availability of organic carbon species that are in parts derived from cellular debris generated during bacterivory or viral lysis, such as peptidoglycan, chitin and their subunit N-acetylglucosamine (NAG). We tested the hypothesis that two dominant grazing resistant bacterial taxa, the ac1 tribe of Actinobacteria (ac1) and filamentous bacteria from the LD2 lineage (Saprospiraceae), profit from such carbon sources during periods of intense HNF predation. The abundances of ac1 and LD2 rose in parallel with HNF, and disproportionally high fractions of cells from both lineages were involved in NAG uptake. Members of ac1 and LD2 were significantly more enriched after NAG addition to lake water. However, highest growth rates of both bacterial lineages were found on chitin and peptidoglycan. Moreover, the direct or indirect transfer of organic carbon from peptidoglycan to LD2 filaments could be demonstrated. We thus provide evidence that these taxa may benefit twofold from protistan predation: by removal of their competitors, and by specific physiological adaptations to utilize carbon sources that are released during grazing or viral lysis.