Pitfalls in machine learning-based assessment of tumor-infiltrating lymphocytes in breast cancer: A report of the International Immuno-Oncology Biomarker Working Group on Breast Cancer

The clinical significance of the tumor-immune interaction in breast cancer is now established, and tumor-infiltrating lymphocytes (TILs) have emerged as predictive and prognostic biomarkers for patients with triple-negative (estrogen receptor, progesterone receptor, and HER2-negative) breast cancer and HER2-positive breast cancer. How computational assessments of TILs might complement manual TIL assessment in trial and daily practices is currently debated. Recent efforts to use machine learning (ML) to automatically evaluate TILs have shown promising results. We review state-of-the-art approaches and identify pitfalls and challenges of automated TIL evaluation by studying the root cause of ML discordances in comparison to manual TIL quantification. We categorize our findings into four main topics: (1) technical slide issues, (2) ML and image analysis aspects, (3) data challenges, and (4) validation issues. The main reason for discordant assessments is the inclusion of false-positive areas or cells identified by performance on certain tissue patterns or design choices in the computational implementation. To aid the adoption of ML for TIL assessment, we provide an in-depth discussion of ML and image analysis, including validation issues that need to be considered before reliable computational reporting of TILs can be incorporated into the trial and routine clinical management of patients with triple-negative breast cancer. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Review of Serial and Parallel Min-Cut/Max-Flow Algorithms for Computer Vision.

Minimum cut/maximum flow (min-cut/max-flow) algorithms solve a variety of problems in computer vision and thus significant effort has been put into developing fast min-cut/max-flow algorithms. As a result, it is difficult to choose an ideal algorithm for a given problem. Furthermore, parallel algorithms have not been thoroughly compared. In this paper, we evaluate the state-of-the-art serial and parallel min-cut/max-flow algorithms on the largest set of computer vision problems yet. We focus on generic algorithms, i.e., for unstructured graphs, but also compare with the specialized GridCut implementation. When applicable, GridCut performs best. Otherwise, the two pseudoflow algorithms, Hochbaum pseudoflow and excesses incremental breadth first search, achieves the overall best performance. The most memory efficient implementation tested is the Boykov-Kolmogorov algorithm. Amongst generic parallel algorithms, we find the bottom-up merging approach by Liu and Sun to be best, but no method is dominant. Of the generic parallel methods, only the parallel preflow push-relabel algorithm is able to efficiently scale with many processors across problem sizes, and no generic parallel method consistently outperforms serial algorithms. Finally, we provide and evaluate strategies for algorithm selection to obtain good expected performance. We make our dataset and implementations publicly available for further research.

Using virtual reality for anatomical landmark annotation in geometric morphometrics.

To study the shape of objects using geometric morphometrics, landmarks are oftentimes collected digitally from a 3D scanned model. The expert may annotate landmarks using software that visualizes the 3D model on a flat screen, and interaction is achieved with a mouse and a keyboard. However, landmark annotation of a 3D model on a 2D display is a tedious process and potentially introduces error due to the perception and interaction limitations of the flat interface. In addition, digital landmark placement can be more time-consuming than direct annotation on the physical object using a tactile digitizer arm. Since virtual reality (VR) is designed to more closely resemble the real world, we present a VR prototype for annotating landmarks on 3D models. We study the impact of VR on annotation performance by comparing our VR prototype to Stratovan Checkpoint, a commonly used commercial desktop software. We use an experimental setup, where four operators placed six landmarks on six grey seal (Halichoerus grypus) skulls in six trials for both systems. This enables us to investigate multiple sources of measurement error. We analyse both for the configuration and for single landmarks. Our analysis shows that annotation in VR is a promising alternative to desktop annotation. We find that annotation precision is comparable between the two systems, with VR being significantly more precise for one of the landmarks. We do not find evidence that annotation in VR is faster than on the desktop, but it is accurate.

Measurement error using a SeeMaLab structured light 3D scanner against a Microscribe 3D digitizer.

BACKGROUND: Geometric morphometrics is a powerful approach to capture and quantify morphological shape variation. Both 3D digitizer arms and structured light surface scanners are portable, easy to use, and relatively cheap, which makes these two capturing devices obvious choices for geometric morphometrics. While digitizer arms have been the "gold standard", benefits of having full 3D models are manifold. We assessed the measurement error and investigate bias associated with the use of an open-source, high-resolution structured light scanner called SeeMaLab against the popular Microscribe 3D digitizer arm. METHODOLOGY: The analyses were based on 22 grey seal (Halichoerus grypus) skulls. 31 fixed anatomical landmarks were annotated both directly using a Microscribe 3D digitizer and on reconstructed 3D digital models created from structured light surface scans. Each skull was scanned twice. Two operators annotated the landmarks, each twice on all the skulls and 3D models, allowing for the investigation of multiple sources of measurement error. We performed multiple Procrustes ANOVAs to compare the two devices in terms of within- and between-operator error, to quantify the measurement error induced by device, to compare between-device error with other sources of variation, and to assess the level of scanning-related error. We investigated the presence of general shape bias due to device and operator. RESULTS: Similar precision was obtained with both devices. If landmarks that were identified as less clearly defined and thus harder to place were omitted, the scanner pipeline would achieve higher precision than the digitizer. Between-operator error was biased and seemed to be smaller when using the scanner pipeline. There were systematic differences between devices, which was mainly driven by landmarks less clearly defined. The factors device, operator and landmark replica were all statistically significant and of similar size, but were minor sources of total shape variation, compared to the biological variation among grey seal skulls. The scanning-related error was small compared to all other error sources. CONCLUSIONS: As the scanner showed precision similar to the digitizer, a scanner should be used if the advantages of obtaining detailed 3D models of a specimen are desired. To obtain high precision, a pre-study should be conducted to identify difficult landmarks. Due to the observed bias, data from different devices and/or operators should not be combined when the expected biological variation is small, without testing the landmarks for repeatability across platforms and operators. For any study necessitating the combination of landmark measurements from different operators, the scanner pipeline will be better suited. The small scanning-related error indicates that by following the same scanning protocol, different operators can be involved in the scanning process without introducing significant error.

Shape from projections via differentiable forward projector for computed tomography.

In computed tomography, the reconstruction is typically obtained on a voxel grid. In this work, however, we propose a mesh-based reconstruction method. For tomographic problems, 3D meshes have mostly been studied to simulate data acquisition, but not for reconstruction, for which a 3D mesh means the inverse process of estimating shapes from projections. In this paper, we propose a differentiable forward model for 3D meshes that bridge the gap between the forward model for 3D surfaces and optimization. We view the forward projection as a rendering process, and make it differentiable by extending recent work in differentiable rendering. We use the proposed forward model to reconstruct 3D shapes directly from projections. Experimental results for single-object problems show that the proposed method outperforms traditional voxel-based methods on noisy simulated data. We also apply the proposed method on electron tomography images of nanoparticles to demonstrate the applicability of the method on real data.

Three-dimensional architecture of human diabetic peripheral nerves revealed by X-ray phase contrast holographic nanotomography.

A deeper knowledge of the architecture of the peripheral nerve with three-dimensional (3D) imaging of the nerve tissue at the sub-cellular scale may contribute to unravel the pathophysiology of neuropathy. Here we demonstrate the feasibility of X-ray phase contrast holographic nanotomography to enable 3D imaging of nerves at high resolution, while covering a relatively large tissue volume. We show various subcomponents of human peripheral nerves in biopsies from patients with type 1 and 2 diabetes and in a healthy subject. Together with well-organized, parallel myelinated nerve fibres we show regenerative clusters with twisted nerve fibres, a sprouted axon from a node of Ranvier and other specific details. A novel 3D construction (with movie created) of a node of Ranvier with end segment of a degenerated axon and sprout of a regenerated one is captured. Many of these architectural elements are not described in the literature. Thus, X-ray phase contrast holographic nanotomography enables identifying specific morphological structures in 3D in peripheral nerve biopsies from a healthy subject and from patients with type 1 and 2 diabetes.

Muscle fibre morphology and microarchitecture in cerebral palsy patients obtained by 3D synchrotron X-ray computed tomography.

BACKGROUND: Synchrotron X-ray computed tomography (SXCT) allows for three-dimensional imaging of objects at a very high resolution and in large field-of-view. PURPOSE: The aim of this study was to use SXCT imaging for morphological analysis of muscle tissue, in order to investigate whether the analysis reveals complementary information to two-dimensional microscopy. METHODS: Three-dimensional SXCT images of muscle biopsies were taken from participants with cerebral palsy and from healthy controls. We designed morphological measures from the two-dimensional slices and three-dimensional volumes of the images and measured the muscle fibre organization, which we term orientation consistency. RESULTS: The muscle fibre cross-sectional areas were significantly larger in healthy participants than in participants with cerebral palsy when carrying out the analysis in three dimensions. However, a similar analysis carried out in two dimensions revealed no patient group difference. The present study also showed that three-dimensional orientation consistency was significantly larger for healthy participants than for participants with cerebral palsy. CONCLUSION: Individuals with CP have smaller muscle fibres than healthy control individuals. We argue that morphometric measures of muscle fibres in two dimensions are generally trustworthy only if the fibres extend perpendicularly to the slice plane, and otherwise three-dimensional aspects should be considered. In addition, the muscle tissue of individuals with CP showed a decreased level of orientation consistency when compared to healthy control tissue. We suggest that the observed disorganization of the tissue may be induced by atrophy caused by physical inactivity and insufficient neural activation.

From Detection of Individual Metastases to Classification of Lymph Node Status at the Patient Level: The CAMELYON17 Challenge.

Automated detection of cancer metastases in lymph nodes has the potential to improve the assessment of prognosis for patients. To enable fair comparison between the algorithms for this purpose, we set up the CAMELYON17 challenge in conjunction with the IEEE International Symposium on Biomedical Imaging 2017 Conference in Melbourne. Over 300 participants registered on the challenge website, of which 23 teams submitted a total of 37 algorithms before the initial deadline. Participants were provided with 899 whole-slide images (WSIs) for developing their algorithms. The developed algorithms were evaluated based on the test set encompassing 100 patients and 500 WSIs. The evaluation metric used was a quadratic weighted Cohen's kappa. We discuss the algorithmic details of the 10 best pre-conference and two post-conference submissions. All these participants used convolutional neural networks in combination with pre- and postprocessing steps. Algorithms differed mostly in neural network architecture, training strategy, and pre- and postprocessing methodology. Overall, the kappa metric ranged from 0.89 to -0.13 across all submissions. The best results were obtained with pre-trained architectures such as ResNet. Confusion matrix analysis revealed that all participants struggled with reliably identifying isolated tumor cells, the smallest type of metastasis, with detection rates below 40%. Qualitative inspection of the results of the top participants showed categories of false positives, such as nerves or contamination, which could be targeted for further optimization. Last, we show that simple combinations of the top algorithms result in higher kappa metric values than any algorithm individually, with 0.93 for the best combination.

A multimodal data-set of a unidirectional glass fibre reinforced polymer composite.

A unidirectional (UD) glass fibre reinforced polymer (GFRP) composite was scanned at varying resolutions in the micro-scale with several imaging modalities. All six scans capture the same region of the sample, containing well-aligned fibres inside a UD load-carrying bundle. Two scans of the cross-sectional surface of the bundle were acquired at a high resolution, by means of scanning electron microscopy (SEM) and optical microscopy (OM), and four volumetric scans were acquired through X-ray computed tomography (CT) at different resolutions. Individual fibres can be resolved from these scans to investigate the micro-structure of the UD bundle. The data is hosted at https://doi.org/10.5281/zenodo.1195879 and it was used in Emerson et al. (2018) [1] to demonstrate that precise and representative characterisations of fibre geometry are possible with relatively low X-ray CT resolutions if the analysis method is robust to image quality.

Prediction of pork quality parameters by applying fractals and data mining on MRI.

This work firstly investigates the use of MRI, fractal algorithms and data mining techniques to determine pork quality parameters non-destructively. The main objective was to evaluate the capability of fractal algorithms (Classical Fractal algorithm, CFA; Fractal Texture Algorithm, FTA and One Point Fractal Texture Algorithm, OPFTA) to analyse MRI in order to predict quality parameters of loin. In addition, the effect of the sequence acquisition of MRI (Gradient echo, GE; Spin echo, SE and Turbo 3D, T3D) and the predictive technique of data mining (Isotonic regression, IR and Multiple linear regression, MLR) were analysed. Both fractal algorithm, FTA and OPFTA are appropriate to analyse MRI of loins. The sequence acquisition, the fractal algorithm and the data mining technique seems to influence on the prediction results. For most physico-chemical parameters, prediction equations with moderate to excellent correlation coefficients were achieved by using the following combinations of acquisition sequences of MRI, fractal algorithms and data mining techniques: SE-FTA-MLR, SE-OPFTA-IR, GE-OPFTA-MLR, SE-OPFTA-MLR, with the last one offering the best prediction results. Thus, SE-OPFTA-MLR could be proposed as an alternative technique to determine physico-chemical traits of fresh and dry-cured loins in a non-destructive way with high accuracy.

Non-invasive assessment of dairy products using spatially resolved diffuse reflectance spectroscopy.

The quality of a dairy product is largely determined by its microstructure which also affects its optical properties. Consequently, an assessment of the optical properties during production may be part of a feedback system for ensuring the quality of the production process. This paper presents a novel camera-based measurement technique that enables robust quantification of a wide range of reduced scattering coefficients and absorption coefficients. Measurements are based on hyperspectral images of diffuse reflectance in the wavelength range of 470 to 1020 nm. The optical properties of commercially available milk and yogurt products with three different levels of fat content are measured. These constitute a relevant range of products at a dairy plant. The measured reduced scattering properties of the samples are presented and show a clear discrimination between levels of fat contents as well as fermentation. The presented measurement technique and method of analysis is thus suitable for a rapid, non-contact, and non-invasive inspection that can deduce physically interpretable properties.

Evaluation of Yogurt Microstructure Using Confocal Laser Scanning Microscopy and Image Analysis.

The microstructure of protein networks in yogurts defines important physical properties of the yogurt and hereby partly its quality. Imaging this protein network using confocal scanning laser microscopy (CSLM) has shown good results, and CSLM has become a standard measuring technique for fermented dairy products. When studying such networks, hundreds of images can be obtained, and here image analysis methods are essential for using the images in statistical analysis. Previously, methods including gray level co-occurrence matrix analysis and fractal analysis have been used with success. However, a range of other image texture characterization methods exists. These methods describe an image by a frequency distribution of predefined image features (denoted textons). Our contribution is an investigation of the choice of image analysis methods by performing a comparative study of 7 major approaches to image texture description. Here, CSLM images from a yogurt fermentation study are investigated, where production factors including fat content, protein content, heat treatment, and incubation temperature are varied. The descriptors are evaluated through nearest neighbor classification, variance analysis, and cluster analysis. Our investigation suggests that the texton-based descriptors provide a fuller description of the images compared to gray-level co-occurrence matrix descriptors and fractal analysis, while still being as applicable and in some cases as easy to tune.

Assessment of algorithms for mitosis detection in breast cancer histopathology images.

The proliferative activity of breast tumors, which is routinely estimated by counting of mitotic figures in hematoxylin and eosin stained histology sections, is considered to be one of the most important prognostic markers. However, mitosis counting is laborious, subjective and may suffer from low inter-observer agreement. With the wider acceptance of whole slide images in pathology labs, automatic image analysis has been proposed as a potential solution for these issues. In this paper, the results from the Assessment of Mitosis Detection Algorithms 2013 (AMIDA13) challenge are described. The challenge was based on a data set consisting of 12 training and 11 testing subjects, with more than one thousand annotated mitotic figures by multiple observers. Short descriptions and results from the evaluation of eleven methods are presented. The top performing method has an error rate that is comparable to the inter-observer agreement among pathologists.

Structure identification in high-resolution transmission electron microscopic images: an example on graphene.

A connection between microscopic structure and macroscopic properties is expected for almost all material systems. High-resolution transmission electron microscopy is a technique offering insight into the atomic structure, but the analysis of large image series can be time consuming. The present work describes a method to automatically estimate the atomic structure in two-dimensional materials. As an example graphene is chosen, in which the positions of the carbon atoms are reconstructed. Lattice parameters are extracted in the frequency domain and an initial atom positioning is estimated. Next, a plausible neighborhood structure is estimated. Finally, atom positions are adjusted by simulation of a Markov random field model, integrating image evidence and the strong geometric prior. A pristine sample with high regularity and a sample with an induced hole are analyzed. False discovery rate-controlled large-scale simultaneous hypothesis testing is used as a statistical framework for interpretation of results. The first sample yields, as expected, a homogeneous distribution of carbon-carbon (C-C) bond lengths. The second sample exhibits regions of shorter C-C bond lengths with a preferred orientation, suggesting either strain in the structure or a buckling of the graphene sheet. The precision of the method is demonstrated on simulated model structures and by its application to multiple exposures of the two graphene samples.