A Review of the Metrics Used to Assess Auto-Contouring Systems in Radiotherapy.

Auto-contouring could revolutionise future planning of radiotherapy treatment. The lack of consensus on how to assess and validate auto-contouring systems currently limits clinical use. This review formally quantifies the assessment metrics used in studies published during one calendar year and assesses the need for standardised practice. A PubMed literature search was undertaken for papers evaluating radiotherapy auto-contouring published during 2021. Papers were assessed for types of metric and the methodology used to generate ground-truth comparators. Our PubMed search identified 212 studies, of which 117 met the criteria for clinical review. Geometric assessment metrics were used in 116 of 117 studies (99.1%). This includes the Dice Similarity Coefficient used in 113 (96.6%) studies. Clinically relevant metrics, such as qualitative, dosimetric and time-saving metrics, were less frequently used in 22 (18.8%), 27 (23.1%) and 18 (15.4%) of 117 studies, respectively. There was heterogeneity within each category of metric. Over 90 different names for geometric measures were used. Methods for qualitative assessment were different in all but two papers. Variation existed in the methods used to generate radiotherapy plans for dosimetric assessment. Consideration of editing time was only given in 11 (9.4%) papers. A single manual contour as a ground-truth comparator was used in 65 (55.6%) studies. Only 31 (26.5%) studies compared auto-contours to usual inter- and/or intra-observer variation. In conclusion, significant variation exists in how research papers currently assess the accuracy of automatically generated contours. Geometric measures are the most popular, however their clinical utility is unknown. There is heterogeneity in the methods used to perform clinical assessment. Considering the different stages of system implementation may provide a framework to decide the most appropriate metrics. This analysis supports the need for a consensus on the clinical implementation of auto-contouring.

Ensemble of Convolutional Neural Networks Improves Automated Segmentation of Acute Ischemic Lesions Using Multiparametric Diffusion-Weighted MRI.

BACKGROUND AND PURPOSE: Accurate automated infarct segmentation is needed for acute ischemic stroke studies relying on infarct volumes as an imaging phenotype or biomarker that require large numbers of subjects. This study investigated whether an ensemble of convolutional neural networks trained on multiparametric DWI maps outperforms single networks trained on solo DWI parametric maps. MATERIALS AND METHODS: Convolutional neural networks were trained on combinations of DWI, ADC, and low b-value-weighted images from 116 subjects. The performances of the networks (measured by the Dice score, sensitivity, and precision) were compared with one another and with ensembles of 5 networks. To assess the generalizability of the approach, we applied the best-performing model to an independent Evaluation Cohort of 151 subjects. Agreement between manual and automated segmentations for identifying patients with large lesion volumes was calculated across multiple thresholds (21, 31, 51, and 70 cm3). RESULTS: An ensemble of convolutional neural networks trained on DWI, ADC, and low b-value-weighted images produced the most accurate acute infarct segmentation over individual networks (P < .001). Automated volumes correlated with manually measured volumes (Spearman ρ = 0.91, P < .001) for the independent cohort. For the task of identifying patients with large lesion volumes, agreement between manual outlines and automated outlines was high (Cohen κ, 0.86-0.90; P < .001). CONCLUSIONS: Acute infarcts are more accurately segmented using ensembles of convolutional neural networks trained with multiparametric maps than by using a single model trained with a solo map. Automated lesion segmentation has high agreement with manual techniques for identifying patients with large lesion volumes.

Machine learning in whole-body MRI: experiences and challenges from an applied study using multicentre data.

Machine learning is now being increasingly employed in radiology to assist with tasks such as automatic lesion detection, segmentation, and characterisation. We are currently involved in an National Institute of Health Research (NIHR)-funded project, which aims to develop machine learning methods to improve the diagnostic performance and reduce the radiology reading time of whole-body magnetic resonance imaging (MRI) scans, in patients being staged for cancer (MALIBO study). We describe here the main challenges we have encountered during the course of this project. Data quality and uniformity are the two most important data traits to be considered in clinical trials incorporating machine learning. Robust data pre-processing and machine learning pipelines have been employed in MALIBO, a task facilitated by the now freely available machine learning libraries and toolboxes. Another important consideration for achieving the desired clinical outcome in MALIBO, was to effectively host the resulting machine learning output, along with the clinical images, for reading in a clinical environment. Finally, a range of legal, ethical, and clinical acceptance issues should be considered when attempting to incorporate computer-assisting tools into clinical practice. The road from translating computational methods into potentially useful clinical tools involves an analytical, stepwise adaptation approach, as well as engagement of a multidisciplinary team.

Encoding atlases by randomized classification forests for efficient multi-atlas label propagation.

We propose a method for multi-atlas label propagation (MALP) based on encoding the individual atlases by randomized classification forests. Most current approaches perform a non-linear registration between all atlases and the target image, followed by a sophisticated fusion scheme. While these approaches can achieve high accuracy, in general they do so at high computational cost. This might negatively affect the scalability to large databases and experimentation. To tackle this issue, we propose to use a small and deep classification forest to encode each atlas individually in reference to an aligned probabilistic atlas, resulting in an Atlas Forest (AF). Our classifier-based encoding differs from current MALP approaches, which represent each point in the atlas either directly as a single image/label value pair, or by a set of corresponding patches. At test time, each AF produces one probabilistic label estimate, and their fusion is done by averaging. Our scheme performs only one registration per target image, achieves good results with a simple fusion scheme, and allows for efficient experimentation. In contrast to standard forest schemes, in which each tree would be trained on all atlases, our approach retains the advantages of the standard MALP framework. The target-specific selection of atlases remains possible, and incorporation of new scans is straightforward without retraining. The evaluation on four different databases shows accuracy within the range of the state of the art at a significantly lower running time.

In vitro transcription from baculovirus late gene promoters: accurate mRNA initiation by nuclear extracts prepared from infected Spodoptera frugiperda cells.

Extracts prepared from nuclei of Autographa californica nuclear polyhedrosis virus-infected Spodoptera frugiperda cells were shown to support in vitro transcription from baculovirus late gene promoters. In vitro transcription was optimized for the late promoter of the 39K gene. The Mg2+ concentration was critical; concentrations higher than 1 to 2 mM did not support late transcription. Additional conditions included template (40 micrograms/ml), extract (2.5 mg/ml), and incubation time (25 min). Using a combination of runoff assays and high-resolution primer extension analyses, this system was shown to accurately initiate transcription from a variety of baculovirus late gene promoters, including those from the 39K and p39/capsid late genes and the hyperexpressed p10 and polyhedrin very late genes. In vitro transcription from the 39K late promoter was resistant to high concentrations of both alpha-amanitin (100 micrograms/ml) and tagetitoxin (4,000 U/ml), suggesting that neither RNA polymerase II nor III is responsible for the transcription of baculovirus late genes.

Characterization of the nucleotide sequence of the Lymantria dispar nuclear polyhedrosis virus DNA polymerase gene region.

The DNA polymerase gene of the Lymantria dispar multinucleocapsid nuclear polyhedrosis virus (LdMNPV) was cloned and sequenced. The predicted DNA polymerase protein (1113 amino acids, 115.9K) was found to have an amino acid identity of 48% with the corresponding gene of the Autographa californica MNPV (AcMNPV). It contains five domains associated with substrate binding, primase interaction, and pyrophosphate hydrolysis and three domains associated with 3'-->5' exonuclease activity common to other DNA polymerases. A region with a conserved TATA promoter and a CAGT mRNA start site sequence motif was identified and shown to be transcribed by RNA polymerase II, indicating that the LdMNPV DNA polymerase gene is expressed as an early gene. An open reading frame possibly expressed as a late gene, oriented in the opposite direction and overlapping the N-terminal coding region of the DNA polymerase gene was found in the LdMNPV sequence and was shown to be conserved in the same position in AcMNPV.

In vitro transactivation of baculovirus early genes by nuclear extracts from Autographa californica nuclear polyhedrosis virus-infected Spodoptera frugiperda cells.

Nuclear extracts, prepared from Autographa californica nuclear polyhedrosis virus-infected Spodoptera frugiperda cells during a time course of infection, were analyzed for activation of early gene transcription and for late gene transcription. The templates used in the in vitro transcription assays contained promoters for baculovirus genes that have been classified as immediate early, delayed early, and late. The promoters were derived from the baculovirus 39K, p26, gp64, and DNA polymerase genes. In addition, the adenovirus major late promoter was included in these studies. We found that transcription from promoters classified as immediate early or delayed early was accurately initiated by using extracts from uninfected cells. Furthermore, transcription from all early promoters tested was found to be transactivated by nuclear extracts prepared at 4 and 8 h postinfection. However, baculovirus enhancer-dependent transcriptional activation was not observed in tests with templates containing the hr5 enhancer sequence. Transcription from baculovirus late promoters was also not observed. A decline in transcription by nuclear extracts prepared from cells late in infection was associated with the presence of DNase activity.

Genetic analysis of the transfer region of the IncN plasmid N3.

Using lambda::Tn5 insertion mutagenesis and screening for conjugation, the boundaries of the IncN plasmid N3 transfer region were determined. Sensitivity to phage IKe infection was used to monitor that part of the N3 transfer region which harbours genes for pilus synthesis and assembly. We cloned this region, creating plasmid pBG21. Escherichia coli cells transformed with pBG21 became sensitive to phage IKe and produced pili, as shown by electron microscopy. Various plasmid constructions containing parts of the pilus-encoding region were used for expression in a minicell system and for expression in an in vitro translation system, thus characterizing for the first time some of the gene products of domain I (Winans and Walker, 1985a) of the transfer region.