Understanding metric-related pitfalls in image analysis validation.

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Removing non-nuclei information from histopathological images: A preprocessing step towards improving nuclei segmentation methods.

Disease interpretation by computer-aided diagnosis systems in digital pathology depends on reliable detection and segmentation of nuclei in hematoxylin and eosin (HE) images. These 2 tasks are challenging since appearance of both cell nuclei and background structures are very variable. This paper presents a method to improve nuclei detection and segmentation in HE images by removing tiles that only contain background information. The method divides each image into smaller patches and uses their projection to the noiselet space to capture different spatial features from non-nuclei background and nuclei structures. The noiselet features are clustered by a K-means algorithm and the resultant partition, defined by the cluster centroids, is herein named the noiselet code-book. A part of an image, a tile, is divided into patches and represented by the histogram of occurrences of the projected patches in the noiselet code-book. Finally, with these histograms, a classifier learns to differentiate between nuclei and non-nuclei tiles. By applying a conventional watershed-marked method to detect and segment nuclei, evaluation consisted in comparing pure watershed method against denoising-plus-watershed in an open database with 8 different types of tissues. The averaged F-score of nuclei detection improved from 0.830 to 0.86 and the dice score after segmentation increased from 0.701 to 0.723.

ROOD-MRI: Benchmarking the robustness of deep learning segmentation models to out-of-distribution and corrupted data in MRI.

Deep artificial neural networks (DNNs) have moved to the forefront of medical image analysis due to their success in classification, segmentation, and detection challenges. A principal challenge in large-scale deployment of DNNs in neuroimage analysis is the potential for shifts in signal-to-noise ratio, contrast, resolution, and presence of artifacts from site to site due to variances in scanners and acquisition protocols. DNNs are famously susceptible to these distribution shifts in computer vision. Currently, there are no benchmarking platforms or frameworks to assess the robustness of new and existing models to specific distribution shifts in MRI, and accessible multi-site benchmarking datasets are still scarce or task-specific. To address these limitations, we propose ROOD-MRI: a novel platform for benchmarking the Robustness of DNNs to Out-Of-Distribution (OOD) data, corruptions, and artifacts in MRI. This flexible platform provides modules for generating benchmarking datasets using transforms that model distribution shifts in MRI, implementations of newly derived benchmarking metrics for image segmentation, and examples for using the methodology with new models and tasks. We apply our methodology to hippocampus, ventricle, and white matter hyperintensity segmentation in several large studies, providing the hippocampus dataset as a publicly available benchmark. By evaluating modern DNNs on these datasets, we demonstrate that they are highly susceptible to distribution shifts and corruptions in MRI. We show that while data augmentation strategies can substantially improve robustness to OOD data for anatomical segmentation tasks, modern DNNs using augmentation still lack robustness in more challenging lesion-based segmentation tasks. We finally benchmark U-Nets and vision transformers, finding robustness susceptibility to particular classes of transforms across architectures. The presented open-source platform enables generating new benchmarking datasets and comparing across models to study model design that results in improved robustness to OOD data and corruptions in MRI.

Targeting tumour-associated macrophages in hodgkin lymphoma using engineered extracellular matrix-mimicking cryogels.

Tumour-associated macrophages are linked with poor prognosis and resistance to therapy in Hodgkin lymphoma; however, there are no suitable preclinical models to identify macrophage-targeting therapeutics. We used primary human tumours to guide the development of a mimetic cryogel, wherein Hodgkin (but not Non-Hodgkin) lymphoma cells promoted primary human macrophage invasion. In an invasion inhibitor screen, we identified five drug hits that significantly reduced tumour-associated macrophage invasion: marimastat, batimastat, AS1517499, ruxolitinib, and PD-169316. Importantly, ruxolitinib has demonstrated recent success in Hodgkin lymphoma clinical trials. Both ruxolitinib and PD-169316 (a p38 mitogen-activated protein kinase (p38 MAPK) inhibitor) decreased the percent of M2-like macrophages; however, only PD-169316 enhanced the percentage of M1-like macrophages. We validated p38 MAPK as an anti-invasion drug target with five additional drugs using a high-content imaging platform. With our biomimetic cryogel, we modeled macrophage invasion in Hodgkin lymphoma and then used it for target discovery and drug screening, ultimately identifying potential future therapeutics.

A radiomic biomarker for prognosis of resected colorectal cancer liver metastases generalizes across MRI contrast agents.

Introduction: Contrast-enhanced MRI is routinely performed as part of preoperative work-up for patients with Colorectal Cancer Liver Metastases (CRLM). Radiomic biomarkers depicting the characteristics of CRLMs in MRI have been associated with overall survival (OS) of patients, but the reproducibility and clinical applicability of these biomarkers are limited due to the variations in MRI protocols between hospitals. Methods: In this work, we propose a generalizable radiomic model for predicting OS of CRLM patients who received preoperative chemotherapy and delayed-phase contrast enhanced (DPCE) MRIs prior to hepatic resection. This retrospective two-center study included three DPCE MRI cohorts (n=221) collected between January 2006 and December 2012. A 10-minute delayed Gd-DO3A-butrol enhanced MRI discovery cohort was used to select features based on robustness across contrast agents, correlation with OS and pairwise Pearson correlation, and to train a logistic regression model that predicts 3-year OS. Results: The model was evaluated on a 10-minute delayed Gd-DO3A-butrol enhanced MRI validation cohort (n=121), a 20-minute delayed Gd-EOB-DTPA (n=72) cohort from the same institute, and a 5-minute delayed Gd-DTPA cohort (n=28) from an independent institute. Two features were selected: minor axis length and dependence variance. The radiomic signature model stratified high-risk and low-risk CRLM groups in the Gd-DO3Abutrol (HR = 6.29, p = .007), Gd-EOB-DTPA (HR = 3.54, p = .003) and Gd-DTPA (HR = 3.16, p = .04) validation cohorts. Discussion: While most existing MRI findings focus on a specific contrast agent, our study shows the potential of MRI features to be generalizable across main-stream contrast agents at delayed phase.

BI-RADS BERT and Using Section Segmentation to Understand Radiology Reports.

Radiology reports are one of the main forms of communication between radiologists and other clinicians, and contain important information for patient care. In order to use this information for research and automated patient care programs, it is necessary to convert the raw text into structured data suitable for analysis. State-of-the-art natural language processing (NLP) domain-specific contextual word embeddings have been shown to achieve impressive accuracy for these tasks in medicine, but have yet to be utilized for section structure segmentation. In this work, we pre-trained a contextual embedding BERT model using breast radiology reports and developed a classifier that incorporated the embedding with auxiliary global textual features in order to perform section segmentation. This model achieved 98% accuracy in segregating free-text reports, sentence by sentence, into sections of information outlined in the Breast Imaging Reporting and Data System (BI-RADS) lexicon, which is a significant improvement over the classic BERT model without auxiliary information. We then evaluated whether using section segmentation improved the downstream extraction of clinically relevant information such as modality/procedure, previous cancer, menopausal status, purpose of exam, breast density, and breast MRI background parenchymal enhancement. Using the BERT model pre-trained on breast radiology reports, combined with section segmentation, resulted in an overall accuracy of 95.9% in the field extraction tasks. This is a 17% improvement, compared to an overall accuracy of 78.9% for field extraction with models using classic BERT embeddings and not using section segmentation. Our work shows the strength of using BERT in the analysis of radiology reports and the advantages of section segmentation by identifying the key features of patient factors recorded in breast radiology reports.

Cell segmentation for immunofluorescence multiplexed images using two-stage domain adaptation and weakly labeled data for pre-training.

Cellular profiling with multiplexed immunofluorescence (MxIF) images can contribute to a more accurate patient stratification for immunotherapy. Accurate cell segmentation of the MxIF images is an essential step. We propose a deep learning pipeline to train a Mask R-CNN model (deep network) for cell segmentation using nuclear (DAPI) and membrane (Na+K+ATPase) stained images. We used two-stage domain adaptation by first using a weakly labeled dataset followed by fine-tuning with a manually annotated dataset. We validated our method against manual annotations on three different datasets. Our method yields comparable results to the multi-observer agreement on an ovarian cancer dataset and improves on state-of-the-art performance on a publicly available dataset of mouse pancreatic tissues. Our proposed method, using a weakly labeled dataset for pre-training, showed superior performance in all of our experiments. When using smaller training sample sizes for fine-tuning, the proposed method provided comparable performance to that obtained using much larger training sample sizes. Our results demonstrate that using two-stage domain adaptation with a weakly labeled dataset can effectively boost system performance, especially when using a small training sample size. We deployed the model as a plug-in to CellProfiler, a widely used software platform for cellular image analysis.

Missed Breast Cancers on MRI in High-Risk Patients: A Retrospective Case-Control Study.

Purpose: To determine if MRI features and molecular subtype influence the detectability of breast cancers on MRI in high-risk patients. Methods and Materials: Breast cancers in a high-risk population of 104 patients were diagnosed following MRI describing a BI-RADS 4-5 lesion. MRI characteristics at the time of diagnosis were compared with previous MRI, where a BI-RADS 1-2-3 lesion was described. Results: There were 77 false-negative MRIs. A total of 51 cancers were overlooked and 26 were misinterpreted. There was no association found between MRI characteristics, the receptor type and the frequency of missed cancers. The main factors for misinterpreted lesions were multiple breast lesions, prior biopsy/surgery and long-term stability. Lesions were mostly overlooked because of their small size and high background parenchymal enhancement. Among missed lesions, 50% of those with plateau kinetics on initial MRI changed for washout kinetics, and 65% of initially progressively enhancing lesions then showed plateau or washout kinetics. There were more basal-like tumours in BRCA1 carriers (50%) than in non-carriers (13%), p = 0.0001, OR = 6.714, 95% CI = [2.058-21.910]. The proportion of missed cancers was lower in BRCA carriers (59%) versus non-carriers (79%), p < 0.05, OR = 2.621, 95% CI = [1.02-6.74]. Conclusions: MRI characteristics or molecular subtype do not influence breast cancer detectability. Lesions in a post-surgical breast should be assessed with caution. Long-term stability does not rule out malignancy and multimodality evaluation is of added value. Lowering the biopsy threshold for lesions with an interval change in kinetics for a type 2 or 3 curve should be considered. There was a higher rate of interval cancers in BRCA 1 patients attributed to lesions more aggressive in nature.

Self-supervised driven consistency training for annotation efficient histopathology image analysis.

Training a neural network with a large labeled dataset is still a dominant paradigm in computational histopathology. However, obtaining such exhaustive manual annotations is often expensive, laborious, and prone to inter and intra-observer variability. While recent self-supervised and semi-supervised methods can alleviate this need by learning unsupervised feature representations, they still struggle to generalize well to downstream tasks when the number of labeled instances is small. In this work, we overcome this challenge by leveraging both task-agnostic and task-specific unlabeled data based on two novel strategies: (i) a self-supervised pretext task that harnesses the underlying multi-resolution contextual cues in histology whole-slide images to learn a powerful supervisory signal for unsupervised representation learning; (ii) a new teacher-student semi-supervised consistency paradigm that learns to effectively transfer the pretrained representations to downstream tasks based on prediction consistency with the task-specific unlabeled data. We carry out extensive validation experiments on three histopathology benchmark datasets across two classification and one regression based tasks, i.e., tumor metastasis detection, tissue type classification, and tumor cellularity quantification. Under limited-label data, the proposed method yields tangible improvements, which is close to or even outperforming other state-of-the-art self-supervised and supervised baselines. Furthermore, we empirically show that the idea of bootstrapping the self-supervised pretrained features is an effective way to improve the task-specific semi-supervised learning on standard benchmarks. Code and pretrained models are made available at: https://github.com/srinidhiPY/SSL_CR_Histo.

SPIE-AAPM-NCI BreastPathQ challenge: an image analysis challenge for quantitative tumor cellularity assessment in breast cancer histology images following neoadjuvant treatment.

Purpose: The breast pathology quantitative biomarkers (BreastPathQ) challenge was a grand challenge organized jointly by the International Society for Optics and Photonics (SPIE), the American Association of Physicists in Medicine (AAPM), the U.S. National Cancer Institute (NCI), and the U.S. Food and Drug Administration (FDA). The task of the BreastPathQ challenge was computerized estimation of tumor cellularity (TC) in breast cancer histology images following neoadjuvant treatment. Approach: A total of 39 teams developed, validated, and tested their TC estimation algorithms during the challenge. The training, validation, and testing sets consisted of 2394, 185, and 1119 image patches originating from 63, 6, and 27 scanned pathology slides from 33, 4, and 18 patients, respectively. The summary performance metric used for comparing and ranking algorithms was the average prediction probability concordance (PK) using scores from two pathologists as the TC reference standard. Results: Test PK performance ranged from 0.497 to 0.941 across the 100 submitted algorithms. The submitted algorithms generally performed well in estimating TC, with high-performing algorithms obtaining comparable results to the average interrater PK of 0.927 from the two pathologists providing the reference TC scores. Conclusions: The SPIE-AAPM-NCI BreastPathQ challenge was a success, indicating that artificial intelligence/machine learning algorithms may be able to approach human performance for cellularity assessment and may have some utility in clinical practice for improving efficiency and reducing reader variability. The BreastPathQ challenge can be accessed on the Grand Challenge website.

Overcoming the limitations of patch-based learning to detect cancer in whole slide images.

Whole slide images (WSIs) pose unique challenges when training deep learning models. They are very large which makes it necessary to break each image down into smaller patches for analysis, image features have to be extracted at multiple scales in order to capture both detail and context, and extreme class imbalances may exist. Significant progress has been made in the analysis of these images, thanks largely due to the availability of public annotated datasets. We postulate, however, that even if a method scores well on a challenge task, this success may not translate to good performance in a more clinically relevant workflow. Many datasets consist of image patches which may suffer from data curation bias; other datasets are only labelled at the whole slide level and the lack of annotations across an image may mask erroneous local predictions so long as the final decision is correct. In this paper, we outline the differences between patch or slide-level classification versus methods that need to localize or segment cancer accurately across the whole slide, and we experimentally verify that best practices differ in both cases. We apply a binary cancer detection network on post neoadjuvant therapy breast cancer WSIs to find the tumor bed outlining the extent of cancer, a task which requires sensitivity and precision across the whole slide. We extensively study multiple design choices and their effects on the outcome, including architectures and augmentations. We propose a negative data sampling strategy, which drastically reduces the false positive rate (25% of false positives versus 62.5%) and improves each metric pertinent to our problem, with a 53% reduction in the error of tumor extent. Our results indicate classification performances of image patches versus WSIs are inversely related when the same negative data sampling strategy is used. Specifically, injection of negatives into training data for image patch classification degrades the performance, whereas the performance is improved for slide and pixel-level WSI classification tasks. Furthermore, we find applying extensive augmentations helps more in WSI-based tasks compared to patch-level image classification.

Loss odyssey in medical image segmentation.

The loss function is an important component in deep learning-based segmentation methods. Over the past five years, many loss functions have been proposed for various segmentation tasks. However, a systematic study of the utility of these loss functions is missing. In this paper, we present a comprehensive review of segmentation loss functions in an organized manner. We also conduct the first large-scale analysis of 20 general loss functions on four typical 3D segmentation tasks involving six public datasets from 10+ medical centers. The results show that none of the losses can consistently achieve the best performance on the four segmentation tasks, but compound loss functions (e.g. Dice with TopK loss, focal loss, Hausdorff distance loss, and boundary loss) are the most robust losses. Our code and segmentation results are publicly available and can serve as a loss function benchmark. We hope this work will also provide insights on new loss function development for the community.

Learning to segment images with classification labels.

Two of the most common tasks in medical imaging are classification and segmentation. Either task requires labeled data annotated by experts, which is scarce and expensive to collect. Annotating data for segmentation is generally considered to be more laborious as the annotator has to draw around the boundaries of regions of interest, as opposed to assigning image patches a class label. Furthermore, in tasks such as breast cancer histopathology, any realistic clinical application often includes working with whole slide images, whereas most publicly available training data are in the form of image patches, which are given a class label. We propose an architecture that can alleviate the requirements for segmentation-level ground truth by making use of image-level labels to reduce the amount of time spent on data curation. In addition, this architecture can help unlock the potential of previously acquired image-level datasets on segmentation tasks by annotating a small number of regions of interest. In our experiments, we show using only one segmentation-level annotation per class, we can achieve performance comparable to a fully annotated dataset.

Deep neural network models for computational histopathology: A survey.

Histopathological images contain rich phenotypic information that can be used to monitor underlying mechanisms contributing to disease progression and patient survival outcomes. Recently, deep learning has become the mainstream methodological choice for analyzing and interpreting histology images. In this paper, we present a comprehensive review of state-of-the-art deep learning approaches that have been used in the context of histopathological image analysis. From the survey of over 130 papers, we review the field's progress based on the methodological aspect of different machine learning strategies such as supervised, weakly supervised, unsupervised, transfer learning and various other sub-variants of these methods. We also provide an overview of deep learning based survival models that are applicable for disease-specific prognosis tasks. Finally, we summarize several existing open datasets and highlight critical challenges and limitations with current deep learning approaches, along with possible avenues for future research.

Intensity Augmentation to Improve Generalizability of Breast Segmentation Across Different MRI Scan Protocols.

OBJECTIVE: The segmentation of the breast from the chest wall is an important first step in the analysis of breast magnetic resonance images. 3D U-Nets have been shown to obtain high segmentation accuracy and appear to generalize well when trained on one scanner type and tested on another scanner, provided that a very similar MR protocol is used. There has, however, been little work addressing the problem of domain adaptation when image intensities or patient orientation differ markedly between the training set and an unseen test set. In this work we aim to address this domain shift problem. METHOD: We propose to apply extensive intensity augmentation in addition to geometric augmentation during training. We explored both style transfer and a novel intensity remapping approach as intensity augmentation strategies. For our experiments, we trained a 3D U-Net on T1-weighted scans. We tested our network on T2-weighted scans from the same dataset as well as on an additional independent test set acquired with a T1-weighted TWIST sequence and a different coil configuration. RESULTS: By applying intensity augmentation we increased segmentation performance for the T2-weighted scans from a Dice of 0.71 to 0.88. This performance is very close to the baseline performance of training with T2-weighted scans (0.92). On the T1-weighted dataset we obtained a performance increase from 0.77 to 0.85. CONCLUSION: Our results show that the proposed intensity augmentation increases segmentation performance across different datasets. SIGNIFICANCE: The proposed method can improve whole breast segmentation of clinical MR scans acquired with different protocols.

BIAS: Transparent reporting of biomedical image analysis challenges.

The number of biomedical image analysis challenges organized per year is steadily increasing. These international competitions have the purpose of benchmarking algorithms on common data sets, typically to identify the best method for a given problem. Recent research, however, revealed that common practice related to challenge reporting does not allow for adequate interpretation and reproducibility of results. To address the discrepancy between the impact of challenges and the quality (control), the Biomedical Image Analysis ChallengeS (BIAS) initiative developed a set of recommendations for the reporting of challenges. The BIAS statement aims to improve the transparency of the reporting of a biomedical image analysis challenge regardless of field of application, image modality or task category assessed. This article describes how the BIAS statement was developed and presents a checklist which authors of biomedical image analysis challenges are encouraged to include in their submission when giving a paper on a challenge into review. The purpose of the checklist is to standardize and facilitate the review process and raise interpretability and reproducibility of challenge results by making relevant information explicit.

Convolutional neural networks versus radiologists in characterization of small hypoattenuating hepatic nodules on CT: a critical diagnostic challenge in staging of colorectal carcinoma.

Our objective was to compare the diagnostic performance and diagnostic confidence of convolutional neural networks (CNN) to radiologists in characterizing small hypoattenuating hepatic nodules (SHHN) in colorectal carcinoma (CRC) on CT scans. Retrospective review of CRC CT scans over 6-years yielded 199 patients (550 SHHN) defined as < 1 cm in diameter. The reference standard was established through 1-year stability/MRI for benign or nodule evolution for malignant nodules. Five CNNs underwent supervised training on 150 patients (412 SHHN). The remaining 49 patients (138 SHHN) were used as testing-set to compare performance of 3 radiologists to CNN, measured through ROC AUC analysis of confidence rating assigned to each nodule by the radiologists. Multivariable modeling was used to compensate for radiologist bias from visible findings other than SHHN. In characterizing SHHN as benign or malignant, the radiologists' mean AUC ROC (0.96) was significantly higher than CNN (0.84, p = 0.0004) but equivalent to CNN adjusted through multivariable modeling for presence of synchronous ≥ 1 cm liver metastases (0.95, p = 0.9). The diagnostic confidence of radiologists and CNN were analyzed. There were significantly lower number of nodules rated with low confidence by CNN (19.6%) and CNN with liver metastatic status (18.1%) than two (38.4%, 44.2%, p < 0.0001) but not a third radiologist (11.1%, p = 0.09). We conclude that in CRC, CNN in combination with liver metastatic status equaled expert radiologists in characterizing SHHN but with better diagnostic confidence.

Use of radiomics for the prediction of local control of brain metastases after stereotactic radiosurgery.

BACKGROUND: Local response prediction for brain metastases (BM) after stereotactic radiosurgery (SRS) is challenging, particularly for smaller BM, as existing criteria are based solely on unidimensional measurements. This investigation sought to determine whether radiomic features provide additional value to routinely available clinical and dosimetric variables to predict local recurrence following SRS. METHODS: Analyzed were 408 BM in 87 patients treated with SRS. A total of 440 radiomic features were extracted from the tumor core and the peritumoral regions, using the baseline pretreatment volumetric post-contrast T1 (T1c) and volumetric T2 fluid-attenuated inversion recovery (FLAIR) MRI sequences. Local tumor progression was determined based on Response Assessment in Neuro-Oncology‒BM criteria, with a maximum axial diameter growth of >20% on the follow-up T1c indicating local failure. The top radiomic features were determined based on resampled random forest (RF) feature importance. An RF classifier was trained using each set of features and evaluated using the area under the receiver operating characteristic curve (AUC). RESULTS: The addition of any one of the top 10 radiomic features to the set of clinical features resulted in a statistically significant (P < 0.001) increase in the AUC. An optimized combination of radiomic and clinical features resulted in a 19% higher resampled AUC (mean = 0.793; 95% CI = 0.792-0.795) than clinical features alone (0.669, 0.668-0.671). CONCLUSIONS: The increase in AUC of the RF classifier, after incorporating radiomic features, suggests that quantitative characterization of tumor appearance on pretreatment T1c and FLAIR adds value to known clinical and dosimetric variables for predicting local failure.

Automated and Manual Quantification of Tumour Cellularity in Digital Slides for Tumour Burden Assessment.

The residual cancer burden index is an important quantitative measure used for assessing treatment response following neoadjuvant therapy for breast cancer. It has shown to be predictive of overall survival and is composed of two key metrics: qualitative assessment of lymph nodes and the percentage of invasive or in situ tumour cellularity (TC) in the tumour bed (TB). Currently, TC is assessed through eye-balling of routine histopathology slides estimating the proportion of tumour cells within the TB. With the advances in production of digitized slides and increasing availability of slide scanners in pathology laboratories, there is potential to measure TC using automated algorithms with greater precision and accuracy. We describe two methods for automated TC scoring: 1) a traditional approach to image analysis development whereby we mimic the pathologists' workflow, and 2) a recent development in artificial intelligence in which features are learned automatically in deep neural networks using image data alone. We show strong agreements between automated and manual analysis of digital slides. Agreements between our trained deep neural networks and experts in this study (0.82) approach the inter-rater agreements between pathologists (0.89). We also reveal properties that are captured when we apply deep neural network to whole slide images, and discuss the potential of using such visualisations to improve upon TC assessment in the future.

Sample-Size Determination Methodologies for Machine Learning in Medical Imaging Research: A Systematic Review.

PURPOSE: The required training sample size for a particular machine learning (ML) model applied to medical imaging data is often unknown. The purpose of this study was to provide a descriptive review of current sample-size determination methodologies in ML applied to medical imaging and to propose recommendations for future work in the field. METHODS: We conducted a systematic literature search of articles using Medline and Embase with keywords including "machine learning," "image," and "sample size." The search included articles published between 1946 and 2018. Data regarding the ML task, sample size, and train-test pipeline were collected. RESULTS: A total of 167 articles were identified, of which 22 were included for qualitative analysis. There were only 4 studies that discussed sample-size determination methodologies, and 18 that tested the effect of sample size on model performance as part of an exploratory analysis. The observed methods could be categorized as pre hoc model-based approaches, which relied on features of the algorithm, or post hoc curve-fitting approaches requiring empirical testing to model and extrapolate algorithm performance as a function of sample size. Between studies, we observed great variability in performance testing procedures used for curve-fitting, model assessment methods, and reporting of confidence in sample sizes. CONCLUSIONS: Our study highlights the scarcity of research in training set size determination methodologies applied to ML in medical imaging, emphasizes the need to standardize current reporting practices, and guides future work in development and streamlining of pre hoc and post hoc sample size approaches.