Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation.

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

Symptoms before and after COVID-19: a population and case-control study using prospective data.

BACKGROUND: Some individuals experience prolonged illness after acute COVID-19. We assessed whether pre-infection symptoms affected post-COVID illness duration. METHODS: Survival analysis was performed in adults (n=23 452) with community-managed SARC-CoV-2 infection prospectively self-logging data through the ZOE COVID Symptom Study app, at least weekly, from 8 weeks before to 12 weeks after COVID-19 onset, conditioned on presence versus absence of baseline symptoms (4-8 weeks before COVID-19). A case-control study was performed in 1350 individuals with long illness (≥8 weeks, 906 [67.1%] with illness ≥12 weeks), matched 1:1 (for age, sex, body mass index, testing week, prior infection, vaccination, smoking, index of multiple deprivation) with 1350 individuals with short illness (<4 weeks). Baseline symptoms were compared between the two groups; and against post-COVID symptoms. RESULTS: Individuals reporting baseline symptoms had longer post-COVID symptom duration (from 10 to 15 days) with baseline fatigue nearly doubling duration. Two-thirds (910 of 1350 [67.4%]) of individuals with long illness were asymptomatic beforehand. However, 440 (32.6%) had baseline symptoms, versus 255 (18.9%) of 1350 individuals with short illness (p<0.0001). Baseline symptoms increased the odds ratio for long illness (2.14 [CI: 1.78; 2.57]). Prior comorbidities were more common in individuals with long versus short illness. In individuals with long illness, baseline symptomatic (versus asymptomatic) individuals were more likely to be female, younger, and have prior comorbidities; and baseline and post-acute symptoms and symptom burden correlated strongly. CONCLUSIONS: Individuals experiencing symptoms before COVID-19 have longer illness duration and increased odds of long illness. However, many individuals with long illness are well before SARS-CoV-2 infection.

Symptom prevalence, duration, and risk of hospital admission in individuals infected with SARS-CoV-2 during periods of omicron and delta variant dominance: a prospective observational study from the ZOE COVID Study.

BACKGROUND: The SARS-CoV-2 variant of concern, omicron, appears to be less severe than delta. We aim to quantify the differences in symptom prevalence, risk of hospital admission, and symptom duration among the vaccinated population. METHODS: In this prospective longitudinal observational study, we collected data from participants who were self-reporting test results and symptoms in the ZOE COVID app (previously known as the COVID Symptoms Study App). Eligible participants were aged 16-99 years, based in the UK, with a body-mass index between 15 and 55 kg/m2, had received at least two doses of any SARS-CoV-2 vaccine, were symptomatic, and logged a positive symptomatic PCR or lateral flow result for SARS-CoV-2 during the study period. The primary outcome was the likelihood of developing a given symptom (of the 32 monitored in the app) or hospital admission within 7 days before or after the positive test in participants infected during omicron prevalence compared with those infected during delta prevalence. FINDINGS: Between June 1, 2021, and Jan 17, 2022, we identified 63 002 participants who tested positive for SARS-CoV-2 and reported symptoms in the ZOE app. These patients were matched 1:1 for age, sex, and vaccination dose, across two periods (June 1 to Nov 27, 2021, delta prevalent at >70%; n=4990, and Dec 20, 2021, to Jan 17, 2022, omicron prevalent at >70%; n=4990). Loss of smell was less common in participants infected during omicron prevalence than during delta prevalence (16·7% vs 52·7%, odds ratio [OR] 0·17; 95% CI 0·16-0·19, p<0·001). Sore throat was more common during omicron prevalence than during delta prevalence (70·5% vs 60·8%, 1·55; 1·43-1·69, p<0·001). There was a lower rate of hospital admission during omicron prevalence than during delta prevalence (1·9% vs 2·6%, OR 0·75; 95% CI 0·57-0·98, p=0·03). INTERPRETATION: The prevalence of symptoms that characterise an omicron infection differs from those of the delta SARS-CoV-2 variant, apparently with less involvement of the lower respiratory tract and reduced probability of hospital admission. Our data indicate a shorter period of illness and potentially of infectiousness which should impact work-health policies and public health advice. FUNDING: Wellcome Trust, ZOE, National Institute for Health Research, Chronic Disease Research Foundation, National Institutes of Health, and Medical Research Council.

Robust and high-throughput lipidomic quantitation of human blood samples using flow injection analysis with tandem mass spectrometry for clinical use.

Direct infusion of lipid extracts into the ion source of a mass spectrometer is a well-established method for lipid analysis. In most cases, nanofluidic devices are used for sample introduction. However, flow injection analysis (FIA) based on sample infusion from a chromatographic pump can offer a simple alternative to shotgun-based approaches. Here, we describe important modification of a method based on FIA and tandem mass spectrometry (MS/MS). We focus on minimizing contamination of the FIA/MS both to render the lipidomic platform more robust and to increase its capacity and applicability for long-sequence measurements required in clinical applications. Robust validation of the developed method confirms its suitability for lipid quantitation in human plasma analysis. Measurements of standard human plasma reference material (NIST SRM 1950) and a set of plasma samples collected from kidney cancer patients and from healthy volunteers yielded highly similar results between FIA-MS/MS and ultra-high-performance supercritical fluid chromatography (UHPSFC)/MS, thereby demonstrating that all modifications have practically no effect on the statistical output. Newly modified FIA-MS/MS allows for the quantitation of 141 lipid species in plasma (11 major lipid classes) within 5.7 min. Finally, we tested the method in a clinical laboratory of the General University Hospital in Prague. In the clinical setting, the method capacity reached 257 samples/day. We also show similar performance of the classification models trained based on the results obtained in clinical settings and the analytical laboratory at the University of Pardubice. Together, these findings demonstrate the high potential of the modified FIA-MS/MS for application in clinical laboratories to measure plasma and serum lipid profiles.

Hemodynamic matrix factorization for functional magnetic resonance imaging.

The General Linear Model (GLM) used in task-fMRI relates activated brain areas to extrinsic task conditions. The translation of resulting neural activation into a hemodynamic response is commonly approximated with a linear convolution model using a hemodynamic response function (HRF). There are two major limitations in GLM analysis. Firstly, the GLM assumes that neural activation is either on or off and matches the exact stimulus duration in the corresponding task timings. Secondly, brain networks observed in resting-state fMRI experiments present also during task experiments, but the GLM approach models these task-unrelated brain activity as noise. A novel kernel matrix factorization approach, called hemodynamic matrix factorization (HMF), is therefore proposed that addresses both limitations by assuming that task-related and task-unrelated brain activity can be modeled with the same convolution model as in GLM analysis. By contrast to the GLM, the proposed HMF is a blind source separation (BSS) technique, which decomposes fMRI data into modes. Each mode comprises of a neural activation time course and a spatial mapping. Two versions of HMF are proposed in which the neural activation time course of each mode is convolved with either the canonical HRF or predetermined subject-specific HRFs. Firstly, HMF with the canonical HRF is applied to two open-source cohorts. These cohorts comprise of several task experiments including motor, incidental memory, spatial coherence discrimination, verbal discrimination task and a very short localization task, engaging multiple parts of the eloquent cortex. HMF modes were obtained whose neural activation time course followed original task timings and whose corresponding spatial map matched cortical areas known to be involved in the respective task processing. Secondly, the alignment of these neural activation time courses to task timings were further improved by replacing the canonical HRF with subject-specific HRFs during HMF mode computation. In addition to task-related modes, HMF also produced seemingly task-unrelated modes whose spatial maps matched known resting-state networks. The validity of a fMRI task experiment relies on the assumption that the exposure to a stimulus for a given time causes an imminent increase in neural activation of equal duration. The proposed HMF is an attempt to falsify this assumption and allows to identify subject task participation that does not comply with the experiment instructions.

Anxiety and depression symptoms after COVID-19 infection: results from the COVID Symptom Study app.

BACKGROUND: Mental health issues have been reported after SARS-CoV-2 infection. However, comparison to prevalence in uninfected individuals and contribution from common risk factors (eg, obesity and comorbidities) have not been examined. We identified how COVID-19 relates to mental health in the large community-based COVID Symptom Study. METHODS: We assessed anxiety and depression symptoms using two validated questionnaires in 413148 individuals between February and April 2021; 26998 had tested positive for SARS-CoV-2. We adjusted for physical and mental prepandemic comorbidities, body mass index (BMI), age and sex. FINDINGS: Overall, 26.4% of participants met screening criteria for general anxiety and depression. Anxiety and depression were slightly more prevalent in previously SARS-CoV-2-positive (30.4%) vs SARS-CoV-2-negative (26.1%) individuals. This association was small compared with the effect of an unhealthy BMI and the presence of other comorbidities, and not evident in younger participants (≤40 years). Findings were robust to multiple sensitivity analyses. Association between SARS-CoV-2 infection and anxiety and depression was stronger in individuals with recent (<30 days) versus more distant (>120 days) infection, suggesting a short-term effect. INTERPRETATION: A small association was identified between SARS-CoV-2 infection and anxiety and depression symptoms. The proportion meeting criteria for self-reported anxiety and depression disorders is only slightly higher than prepandemic.

Correction to: Machine learning classifiers can predict Gleason pattern 4 prostate cancer with greater accuracy than experienced radiologists.

The original version of this article, published on 11 June 2019, unfortunately contained a mistake. The following correction has therefore been made in the original: In section "Multiparametric MRI review," the readers mentioned in the first sentence were partly incorrect.

New insights in hemp chemical composition: a comprehensive polar lipidome characterization by combining solid phase enrichment, high-resolution mass spectrometry, and cheminformatics.

The chemical composition of Cannabis sativa L. has been extensively investigated for several years; nevertheless, a detailed lipidome characterization is completely lacking in the literature. To achieve this goal, an extraction and enrichment procedure was developed for the characterization of phospholipids and sulfolipids. Firstly, a study on the solid-liquid extraction was performed, to maximize the recovery of the considered lipids; the best procedure consisted of a simple extraction with a mixture of methanol and chloroform (1:1, v/v). The hemp extracts were analyzed by ultra-high-performance liquid chromatography coupled to high-resolution mass spectrometry and lipids were tentatively identified by Lipostar. To improve the number of identifications, an enrichment method, based on graphitized carbon black solid phase extraction, was evaluated to fractionate phospholipids and sulfolipids into separate eluates. Recovery and matrix effects of the procedure were determined on a mixture of standard lipids, containing representative compounds for each considered lipid class. The optimized method allowed the tentative identification of 189 lipids, including 51 phospholipids and 80 sulfolipids, in the first and second fractions, respectively. The detection of only 6 sulfolipids in the first fraction and 9 phospholipids in the second fraction proved the efficacy of the fractionation method, which also allowed the number of lipid identifications to be increased compared to the same procedure without enrichment, which scored 100 lipids. Finally, a semi-quantitative analysis permitted the hemp polar lipidome to be characterized. The results of this study allow knowledge of the hemp chemical composition to be improved with a detailed description of its phospho- and sulfolipid profiles. Graphical abstract.

Machine learning classifiers can predict Gleason pattern 4 prostate cancer with greater accuracy than experienced radiologists.

OBJECTIVE: The purpose of this study was: To test whether machine learning classifiers for transition zone (TZ) and peripheral zone (PZ) can correctly classify prostate tumors into those with/without a Gleason 4 component, and to compare the performance of the best performing classifiers against the opinion of three board-certified radiologists. METHODS: A retrospective analysis of prospectively acquired data was performed at a single center between 2012 and 2015. Inclusion criteria were (i) 3-T mp-MRI compliant with international guidelines, (ii) Likert ≥ 3/5 lesion, (iii) transperineal template ± targeted index lesion biopsy confirming cancer ≥ Gleason 3 + 3. Index lesions from 164 men were analyzed (119 PZ, 45 TZ). Quantitative MRI and clinical features were used and zone-specific machine learning classifiers were constructed. Models were validated using a fivefold cross-validation and a temporally separated patient cohort. Classifier performance was compared against the opinion of three board-certified radiologists. RESULTS: The best PZ classifier trained with prostate-specific antigen density, apparent diffusion coefficient (ADC), and maximum enhancement (ME) on DCE-MRI obtained a ROC area under the curve (AUC) of 0.83 following fivefold cross-validation. Diagnostic sensitivity at 50% threshold of specificity was higher for the best PZ model (0.93) when compared with the mean sensitivity of the three radiologists (0.72). The best TZ model used ADC and ME to obtain an AUC of 0.75 following fivefold cross-validation. This achieved higher diagnostic sensitivity at 50% threshold of specificity (0.88) than the mean sensitivity of the three radiologists (0.82). CONCLUSIONS: Machine learning classifiers predict Gleason pattern 4 in prostate tumors better than radiologists. KEY POINTS: • Predictive models developed from quantitative multiparametric magnetic resonance imaging regarding the characterization of prostate cancer grade should be zone-specific. • Classifiers trained differently for peripheral and transition zone can predict a Gleason 4 component with a higher performance than the subjective opinion of experienced radiologists. • Classifiers would be particularly useful in the context of active surveillance, whereby decisions regarding whether to biopsy are necessitated.

Identification of bioactive short peptides in cow milk by high-performance liquid chromatography on C18 and porous graphitic carbon coupled to high-resolution mass spectrometry.

Short peptides are important compounds in a variety of fields, including food and nutraceutical applications, but also biomarker discovery, bioactive peptide discovery and peptide drug separation. Despite the importance of short peptides, they are currently less studied than other peptides because of the lack of dedicated methods for their characterization. The method described in this paper comprises a combination of strategies to tackle the main limitations in short peptide analysis. In particular, in this work an untargeted peptidomic approach based on ultrahigh-performance liquid chromatography coupled to high-resolution mass spectrometry was developed for the identification of short peptides in cow milk samples. After milk defatting and precipitation, the sample was purified by cotton-hydrophilic interaction liquid chromatography (HILIC) micro tip in order to avoid suppression phenomena due to contaminants present in milk, such as carbohydrates. The sample was then separated by means of two chromatographic columns, with a complementary selectivity mechanism, namely reversed-phase C18 column and porous graphitic carbon (PGC). By this approach, the method allowed the separation and characterization of di-, tri- and tetrapeptides. A total of 57 and 41 peptides were identified by using a C18 and a PGC column, respectively; in particular, 31 were exclusively identified by using the C18 column, 15 unique peptides were identified by using the PGC column, while 26 were in common between the two data sets, demonstrating that the two columns have a different selectivity mechanism. The results indicated that an integrated approach may be appropriate to improve the separation of different peptides and increase the number of identifications because of the wide range of polarity of short peptides. The method allowed the untargeted identification of short peptides in milk, a complex matrix chosen as a representative real sample for method application, and provides complementary information to that accessible by ordinary peptidomics. Graphical abstract.

Investigation of free and conjugated seleno-amino acids in wheat bran by hydrophilic interaction liquid chromatography with tandem mass spectrometry.

An analytical method for determining seleno-methionine, methyl-seleno-cysteine, and seleno-cystine in wheat bran was developed and validated. Four different extraction procedures were evaluated to simultaneously extract endogenous free and conjugated seleno-amino acids in wheat bran in order to select the best extraction protocol in terms of seleno amino acid quantitation. The extracted samples were subjected to a clean-up by a reversed phase/strong cation exchange solid-phase extraction and analyzed by chiral hydrophilic interaction liquid chromatography-tandem mass spectrometry. The optimized extraction protocol was employed to validate the methodology. Process efficiency ranged from 58 to 112% and trueness from 73 to 98%. Limit of detection and limit of quantification were lower than 1 ng/g. Four wheat bran samples were analyzed for both total Se and single seleno-amino acids determination. The results showed that Se- seleno-methyl-lselenocysteine was the major seleno-amino acid in wheat bran while seleno-methionine and seleno-cysteine were both minor species.

Peptidomic Approach for the Identification of Peptides with Potential Antioxidant and Anti-Hyperthensive Effects Derived From Asparagus By-Products.

Asparagus waste represents products of great interest since many compounds with high biological value are located in the lower portion of the spears. The extraction of bioactive compounds from asparagus by-products is therefore crucial for the purpose of adding value to these by-products. In this paper, bioactive peptides from asparagus waste were extracted, digested, purified and identified. In particular, Alcalase® was chosen as the enzyme to use to obtain protein hydrolysate due to its low cost and, consequently, the possibility of implementing the method on a large scale. In order to simplify the peptide extract to reach better identification, the hydrolysate was fractionated by reversed-phase chromatography in 10 fractions. Two tests were carried out for antioxidant activity (ABTS-DPPH) and one for antihypertensive activity (ACE). Fractions with a higher bioactivity score were identified by peptidomics technologies and screened for bioactivity with the use of bioinformatics. For ACE-inhibitor activity, two peptides were synthetized, PDWFLLL and ASQSIWLPGWL, which provided an EC50 value of 1.76 µmol L-1 and 4.02 µmol L-1, respectively. For the antioxidant activity, by DPPH assay, MLLFPM exhibited the lowest EC50 value at 4.14 µmol L-1, followed by FIARNFLLGW and FAPVPFDF with EC50 values of 6.76 µmol L-1 and 10.01 µmol L-1, respectively. A validation of the five identified peptides was also carried out. The obtained results showed that peptides obtained from asparagus by-products are of interest for their biological activity and are suitable for being used as functional ingredients.

Simultaneous Preconcentration, Identification, and Quantitation of Selenoamino Acids in Oils by Enantioselective High Performance Liquid Chromatography and Mass Spectrometry.

Selenium is an essential micronutrient for humans. In food, selenium can be present in both inorganic and organic forms, the latter mainly being selenomethionine, Se-methyl-selenocysteine, and selenocystine. Selenoamino acid speciation rarely involves the chirality of selenoamino acids. In this work, a 5 cm long CHIROBIOTIC TAG chromatographic column was used for enantioresolution of selenoamino acids (d- and l-selenomethionine, Se-methyl-l-selenocysteine, d-, l- and meso-selenocystine); in the optimized conditions, the complete resolution of the analytes was achieved within 15 min by using a very polar aqueous mobile phase (gradient elution by methanol/acetonitrile/H2O, 45:45:10 ( v/ v/ v) with 10 mmol L-1 of ammonium formate and 0.5% formic acid as the mobile phase A and acetonitrile/H2O, 20:80 ( v/ v) with 20 mmol L-1 of ammonium formate at apparent pH 4 as the mobile phase B). The affinity of the teicoplanin aglycone was further exploited to devise a preconcentration method for selenoamino acids in oils. In particular, the CHIROBIOTIC TAG precolumn was used to directly concentrate the selenoamino acids after simple dilution of oil samples with dichloromethane. An optimized procedure for selenoamino acid trapping and preconcentration under normal phase conditions was developed. The enrichment procedure also ensured band focusing during the subsequent separation. The target analytes were finally identified and quantified by triple quadrupole selected reaction monitoring. The method allowed obtainment of recovery values up to 73%, with limits of detection between 280 and 750 ng and limits of quantification between 375 and 960 ng for the different selenoamino acids. The method was applied to commercial oil samples, and only l-selenomethionine was detected.

Delving into the Polar Lipidome by Optimized Chromatographic Separation, High-Resolution Mass Spectrometry, and Comprehensive Identification with Lipostar: Microalgae as Case Study.

The work describes the chromatographic separation optimization of polar lipids on Kinetex-EVO, particularly focusing on sulfolipids in spirulina microalgae ( Arthrospira platensis). Gradient shape and mobile-phase modifiers (pH and buffer) were tested on lipid standards. Different conditions were evaluated, and resolution, peak capacity, and peak shape were calculated both in negative mode, for sulfolipids and phospholipids, and in positive mode, for glycolipids. A high-confidence lipid identification strategy was also applied. In collaboration with software creators and developers, Lipostar was implemented to improve the identification of phosphoglycerolipids and to allow the identification of glycosylmonoradyl- and glycosyldiradyl-glycerols classes, the last being the main focus of this work. By this approach, an untargeted screening also for searching lipids not yet reported in the literature could be accomplished. The optimized chromatographic conditions and database search were tested for lipid identification first on the standard mixture, then on the polar lipid extract of spirulina microalgae, for which 205 lipids were identified.

Whole-body MRI quantitative biomarkers are associated significantly with treatment response in patients with newly diagnosed symptomatic multiple myeloma following bortezomib induction.

OBJECTIVES: To evaluate whole-body MRI (WB-MRI) parameters significantly associated with treatment response in multiple myeloma (MM). METHODS: Twenty-one MM patients underwent WB-MRI at diagnosis and after two cycles of chemotherapy. Scans acquired at 3.0 T included T2, diffusion-weighted-imaging (DWI) and mDixon pre- and post-contrast. Twenty focal lesions (FLs) matched on DWI and post-contrast mDixon were selected for each time point. Estimated tumour volume (eTV), apparent diffusion coefficient (ADC), enhancement ratio (ER) and signal fat fraction (sFF) were derived. Clinical treatment response to chemotherapy was assessed using conventional criteria. Significance of temporal parameter change was assessed by the paired t test and receiver operating characteristics/area under the curve (AUC) analysis was performed. Parameter repeatability was assessed by interclass correlation (ICC) and Bland-Altman analysis of 10 healthy volunteers scanned at two time points. RESULTS: Fifteen of 21 patients responded to treatment. Of 254 FLs analysed, sFF (p < 0.0001) and ADC (p = 0.001) significantly increased in responders but not non-responders. eTV significantly decreased in 19/21 cases. Focal lesion sFF was the best discriminator of treatment response (AUC 1.0). Bone sFF repeatability was excellent (ICC 0.98) and better than bone ADC (ICC 0.47). CONCLUSION: WB-MRI derived focal lesion sFF shows promise as an imaging biomarker of treatment response in newly diagnosed MM. KEY POINTS: • Bone signal fat fraction using mDixon is a robust quantifiable parameter • Fat fraction and ADC significantly increase in myeloma lesions responding to treatment • Bone lesion fat fraction is the best discriminator of myeloma treatment response.

Recent advancements and future directions of superficially porous chiral stationary phases for ultrafast high-performance enantioseparations.

This review focuses on the use of superficially porous particles (SPPs) as chiral stationary phases for ultra-high performance liquid enantioseparations. In contrast to what happened in achiral separations where core-shell particles invaded the market, the introduction of SPPs in chiral liquid chromatography (LC) has been relatively recent. This is due in part to the technical difficulties in the preparation of these phases, and in part to scarce understanding of mass transfer phenomena in chiral chromatography. As a matter of fact, nowadays, the development of superficially porous CSPs is still in its infancy. This paper covers the most recent advancements in the field of core-shell technology applied to chiral separations. We review the kinds of chiral selectors that have been used for the preparation of these phases, by discussing the advantages of chiral SPPs over their fully-porous counterparts for high efficient high throughput enantioseparations. Notwithstanding the apparently obvious advantages in terms of the mass transfer of chiral SPPs, some critical aspects that could impact their development are presented.

Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Understanding the psychological experiences of loneliness in later life: qualitative protocol to inform technology development.

OBJECTIVES: Loneliness is a public health issue impacting the health and well-being of older adults. This protocol focuses on understanding the psychological experiences of loneliness in later life to inform technology development as part of the 'Design for health ageing: a smart system to detect loneliness in older people' (DELONELINESS) study. METHODS AND ANALYSIS: Data will be collected from semi-structured interviews with up to 60 people over the age of 65 on their experiences of loneliness and preferences for sensor-based technologies. The interviews will be audio-recorded, transcribed and analysed using a thematic codebook approach on NVivo software. ETHICS AND DISSEMINATION: This study has received ethical approval by Research Ethics Committee's at King's College London (reference number: LRS/DP-21/22-33376) and the University of Sussex (reference number: ER/JH878/1). All participants will be required to provide informed consent. Results will be used to inform technology development within the DELONELINESS study and will be disseminated in peer-reviewed publications and conferences.