Facilitating the use of routine data to evaluate artificial intelligence solutions: lessons from the NIHR/RCR data curation workshop.

Radiology currently stands at the forefront of artificial intelligence (AI) development and deployment over many other medical subspecialities within the scope of both research and clinical practice. Given this current leadership position, it is imperative that we foster collaboration and knowledge sharing to ensure the ethical, responsible and effective continued progress of AI technologies in our field, ultimately leading to enhanced patient care. To achieve this objective, three workshops have been planned through a coordinated effort by the NIHR/RCR committee. These workshops aim to convene key stakeholders including eminent academics, departmental leaders and industry partners to provide insights from their own experiences and strategies to overcome common challenges faced. In this article, we describe the outcomes from the first workshop, which addresses the topic of "facilitating the use of routine data to evaluate AI solutions". The main key insights uncovered include the need for ethical considerations, detailing of methods for data curation and storage depending on the need and requirements for de-identification. We provide resources for how to de-identify data and also a list of concerns to think about before curating your data. Finally, we address secure data-sharing methods and explore the need for quality assurances, the role of the data access committee and the patient perspectives in this task.

Beyond regulatory compliance: evaluating radiology artificial intelligence applications in deployment.

The implementation of artificial intelligence (AI) applications in routine practice, following regulatory approval, is currently limited by practical concerns around reliability, accountability, trust, safety, and governance, in addition to factors such as cost-effectiveness and institutional information technology support. When a technology is new and relatively untested in a field, professional confidence is lacking and there is a sense of the need to go above the baseline level of validation and compliance. In this article, we propose an approach that goes beyond standard regulatory compliance for AI apps that are approved for marketing, including independent benchmarking in the lab as well as clinical audit in practice, with the aims of increasing trust and preventing harm.

The time is now: making the case for a UK registry of deployment of radiology artificial intelligence applications.

Artificial intelligence (AI)-based healthcare applications (apps) are rapidly evolving, and radiology is a target specialty for their implementation. In this paper, we put the case for a national deployment registry to track the spread of AI apps into clinical use in radiology in the UK. By gathering data on the specific locations, purposes, and people associated with AI app deployment, such a registry would provide greater transparency on their spread in the radiology field. In combination with other regulatory and audit mechanisms, it would provide radiologists and patients with greater confidence and trust in AI apps. At the same time, coordination of this information would reduce costs for the National Health Service (NHS) by preventing duplication of piloting activities. This commentary discusses the need for a UK-wide registry for such apps, its benefits and risks, and critical success factors for its establishment. We conclude by noting that a critical window of opportunity has opened up for the development of a deployment registry, before the current pattern of localised clusters of activity turns into the widespread proliferation of AI apps across clinical practice.

Fully automated deep-learning section-based muscle segmentation from CT images for sarcopenia assessment.

AIM: To develop a fully automated deep-learning-based approach to measure muscle area for assessing sarcopenia on standard-of-care computed tomography (CT) of the abdomen without any case exclusion criteria, for opportunistic screening for frailty. MATERIALS AND METHODS: This ethically approved retrospective study used publicly available and institutional unselected abdominal CT images (n=1,070 training, n=31 testing). The method consisted of two sequential steps: section detection from CT volume followed by muscle segmentation on single-section. Both stages used fully convolutional neural networks (FCNN), based on a UNet-like architecture. Input data consisted of CT volumes with a variety of fields of view, section thicknesses, occlusions, artefacts, and anatomical variations. Output consisted of segmented muscle area on a CT section at the L3 vertebral level. The muscle was segmented into erector spinae, psoas, and rectus abdominus muscle groups. Output was tested against expert manual segmentation. RESULTS: Threefold cross-validation was used to evaluate the model. Section detection cross-validation error was 1.41 ± 5.02 (in sections). Segmentation cross-validation Dice overlaps were 0.97 ± 0.02, 0.95 ± 0.04, and 0.94 ± 0.04 for erector spinae, psoas, and rectus abdominus, respectively, and 0.96 ± 0.02 for the combined muscle area, with R2 = 0.95/0.98 for muscle attenuation/area in 28/31 hold-out test cases. No statistical difference was found between the automated output and a second annotator. Fully automated processing took <1 second per CT examination. CONCLUSIONS: A FCNN pipeline accurately and efficiently automates muscle segmentation at the L3 vertebral level from unselected abdominal CT volumes, with no manual processing step. This approach is promising as a generalisable tool for opportunistic screening for frailty on standard-of-care CT.

O-RADS MRI score: analysis of misclassified cases in a prospective multicentric European cohort.

OBJECTIVE: To retrospectively review the causes of categorization errors using O-RADS-MRI score and to determine the presumptive causes of these misclassifications. METHODS: EURAD database was retrospectively queried to identify misclassified lesions. In this cohort, 1194 evaluable patients with 1502 pelvic masses (277 malignant / 1225 benign lesions) underwent standardized MRI to characterize adnexal masses with histology or 2 years' follow-up as a reference standard. An expert radiologist reviewed cases with two junior radiologists and lesions termed misclassified if malignant lesion was scored ≤ 3, a benign lesion was scored ≥ 4, the site of origin was incorrect, or a non-adnexal mass was incorrectly categorized as benign or malignant. RESULTS: There were 139 / 1502 (9.2%) misclassified masses in 116 women including 109 adnexal and 30 non-adnexal masses. False-negative cases corresponded to 16 borderline or invasive malignant adnexal masses rated score ≤ 3 (16 / 139, 11.5%). False-positive cases corresponded to 88 benign masses were rated score 4 (67 / 139, 48.2%) or 5 (18 / 139,12.9%) or considered suspicious non-adnexal lesions (3 / 139, 2.2%). Misclassifications were only due to origin error in 12 adnexal masses (8 benign, 4 malignant) (8.6%, 12 / 139) and 23 non-adnexal masses (18 benign, 5 malignant,16.5%, 23 / 139) perceived respectively as non-adnexal and adnexal masses. Interpretive error (n = 104), failure to recognize technical insufficient exams (n = 9), and perceptual errors (n = 4) were found. Most interpretive was due to misinterpretation of solid tissue or incorrect assignment of mass origin. Eighty-four out of 139 cases were correctly reclassified by the readers with strict adherence to the score rules. CONCLUSION: Most errors were due to misinterpretation of solid tissue or incorrect assignment of mass origin. KEY POINTS: • Prospective assignment of O-RADS-MRI score resulted in misclassification of 9.25% of sonographically indeterminate pelvic masses. • Most errors were interpretive (74.8%) due to misinterpretation of solid tissue as defined by the lexicon or incorrect assignment of mass origin. • Pelvic inflammatory disease is a common source of misclassification (8.9%) (12 / 139).

An Inter-observer Study to Determine Radiotherapy Planning Target Volumes for Recurrent Gynaecological Cancer Comparing Magnetic Resonance Imaging Only With Computed Tomography-Magnetic Resonance Imaging.

AIMS: Target delineation uncertainty is arguably the largest source of geometric uncertainty in radiotherapy. Several factors can affect it, including the imaging modality used for delineation. It is accounted for by applying safety margins to the target to produce a planning target volume (PTV), to which treatments are designed. To determine the margin, the delineation uncertainty is measured as the delineation error, and then a margin recipe used. However, there is no published evidence of such analysis for recurrent gynaecological cancers (RGC). The aims of this study were first to quantify the delineation uncertainty for RGC gross tumour volumes (GTVs) and to calculate the associated PTV margins and then to quantify the difference in GTV, delineation uncertainty and PTV margin, between a computed tomography-magnetic resonance imaging (CT-MRI) and MRI workflow. MATERIALS AND METHODS: Seven clinicians delineated the GTV for 20 RGC tumours on co-registered CT and MRI datasets (CT-MRI) and on MRI alone. The delineation error, the standard deviation of distances from each clinician's outline to a reference, was measured and the required PTV margin determined. Differences between using CT-MRI and MRI alone were assessed. RESULTS: The overall delineation error and the resulting margin were 3.1 mm and 8.5 mm, respectively, for CT-MRI, reducing to 2.5 mm and 7.1 mm, respectively, for MRI alone. Delineation errors and therefore the theoretical margins, varied widely between patients. MRI tumour volumes were on average 15% smaller than CT-MRI tumour volumes. DISCUSSION: This study is the first to quantify delineation error for RGC tumours and to calculate the corresponding PTV margin. The determined margins were larger than those reported in the literature for similar patients, bringing into question both current margins and margin calculation methods. The wide variation in delineation error between these patients suggests that applying a single population-based margin may result in PTVs that are suboptimal for many. Finally, the reduced tumour volumes and safety margins suggest that patients with RGC may benefit from an MRI-only treatment workflow.

A systematic review and meta-analysis of the diagnostic accuracy of biparametric prostate MRI for prostate cancer in men at risk.

INTRODUCTION: Multiparametric magnetic resonance imaging (mpMRI), the use of three multiple imaging sequences, typically T2-weighted, diffusion weighted (DWI) and dynamic contrast enhanced (DCE) images, has a high sensitivity and specificity for detecting significant cancer. Current guidance now recommends its use prior to biopsy. However, the impact of DCE is currently under debate regarding test accuracy. Biparametric MRI (bpMRI), using only T2 and DWI has been proposed as a viable alternative. We conducted a contemporary systematic review and meta-analysis to further examine the diagnostic performance of bpMRI in the diagnosis of any and clinically significant prostate cancer. METHODS: A systematic review of the literature from 01/01/2017 to 06/07/2019 was performed by two independent reviewers using predefined search criteria. The index test was biparametric MRI and the reference standard whole-mount prostatectomy or prostate biopsy. Quality of included studies was assessed by the QUADAS-2 tool. Statistical analysis included pooled diagnostic performance (sensitivity; specificity; AUC), meta-regression of possible covariates and head-to-head comparisons of bpMRI and mpMRI where both were performed in the same study. RESULTS: Forty-four articles were included in the analysis. The pooled sensitivity for any cancer detection was 0.84 (95% CI, 0.80-0.88), specificity 0.75 (95% CI, 0.68-0.81) for bpMRI. The summary ROC curve yielded a high AUC value (AUC = 0.86). The pooled sensitivity for clinically significant prostate cancer was 0.87 (95% CI, 0.78-0.93), specificity 0.72 (95% CI, 0.56-0.84) and the AUC value was 0.87. Meta-regression analysis revealed no difference in the pooled diagnostic estimates between bpMRI and mpMRI. CONCLUSIONS: This meta-analysis on contemporary studies shows that bpMRI offers comparable test accuracies to mpMRI in detecting prostate cancer. These data are broadly supportive of the bpMRI approach but heterogeneity does not allow definitive recommendations to be made. There is a need for prospective multicentre studies of bpMRI in biopsy naïve men.

Machine learning in whole-body MRI: experiences and challenges from an applied study using multicentre data.

Machine learning is now being increasingly employed in radiology to assist with tasks such as automatic lesion detection, segmentation, and characterisation. We are currently involved in an National Institute of Health Research (NIHR)-funded project, which aims to develop machine learning methods to improve the diagnostic performance and reduce the radiology reading time of whole-body magnetic resonance imaging (MRI) scans, in patients being staged for cancer (MALIBO study). We describe here the main challenges we have encountered during the course of this project. Data quality and uniformity are the two most important data traits to be considered in clinical trials incorporating machine learning. Robust data pre-processing and machine learning pipelines have been employed in MALIBO, a task facilitated by the now freely available machine learning libraries and toolboxes. Another important consideration for achieving the desired clinical outcome in MALIBO, was to effectively host the resulting machine learning output, along with the clinical images, for reading in a clinical environment. Finally, a range of legal, ethical, and clinical acceptance issues should be considered when attempting to incorporate computer-assisting tools into clinical practice. The road from translating computational methods into potentially useful clinical tools involves an analytical, stepwise adaptation approach, as well as engagement of a multidisciplinary team.

Adnexal lesions: Imaging strategies for ultrasound and MR imaging.

Adnexal lesions are routinely encountered in general practice. Ultrasound is the first line of investigation in determining the benign or malignant potential of an adnexal lesion. In the cases of classic simple cysts, hemorrhagic cysts, endometriomas, dermoids and obviously malignant lesions, ultrasound may be sufficient for management recommendations. In cases where there is an isolated adnexal lesion, without peritoneal disease or serum CA-125 elevation, and in lesions considered indeterminate on ultrasound, MR imaging with incorporation of the ADNEx MR score can increase the specificity for the diagnosis of benignity or malignancy. This article will review the imaging evaluation of adnexal lesions and how to incorporate the ADNEx MR score to help guide clinical management.

MRI measurement of residual cervical length after radical trachelectomy for cervical cancer and the risk of adverse pregnancy outcomes: a blinded imaging analysis.

OBJECTIVE: To determine the association between the residual cervix measured on postoperative MRI after radical vaginal trachelectomy (RVT) and adverse obstetrical outcomes. DESIGN: Observational study. SETTING: Referral Cancer centre. POPULATION: Women who conceived after RVT for cervical cancer at the Royal Marsden Hospital, London, between 1995 and 2015. METHODS: Postoperative MRI scans were analysed by three researchers. The agreement between researchers was assessed by Pearson's correlation coefficient and Bland-Altman plot. Patients were divided into two groups (<10 and ≥10 mm residual cervix) for the analysis of adverse obstetrical outcomes. MAIN OUTCOME MEASURES: Late miscarriage, premature delivery, premature rupture of membranes (PROM) and chorioamnionitis. RESULTS: Thirty-one MRI scans were available; 29 of these women had a pregnancy that progressed beyond the first trimester. There was a strong reproducibility of the measurement of residual cervix (P < 0.001). Nineteen women (65.5%) had <10 mm residual cervix and 10 (34.5%) had ≥10 mm. Among women with <10 mm residual cervix, seven (36.8%) experienced PROM and ten (66.7%) had a preterm birth; No women with ≥10 mm residual cervix had PROM and two (22.2%) had a preterm birth (P = 0.028 and P = 0.035, respectively). Overall, there were nine (16.7%) first-trimester miscarriages, six (11.1%) late fetal losses, 12 (31.6%) preterm births and 36 (66.7%) live births. After a mean follow up of 78.1 months, 36 women were disease-free and one woman had died. CONCLUSIONS: MRI measurements of the residual cervix are reproducible between observers. The incidence of PROM and premature delivery is higher when the residual cervix after RVT is <10 mm. TWEETABLE ABSTRACT: The risk of prematurity after RVT can be predicted from measurements of residual cervical length on postoperative MRI scan.

Session 1: The evolution and development of the multidisciplinary team approach: USA, European and UK experiences - what can we do better?

The process of determining the best treatments that should be offered to patients with newly diagnosed colon and rectal cancer remains highly variable around the world. The aim of this expert review was to agree the key elements of good quality preoperative treatment decision making.

Pre-operative MRI staging of endometrial cancer in a multicentre cancer network: can we match single centre study results?

OBJECTIVES: To evaluate the staging accuracy of magnetic resonance imaging (MRI) for endometrial cancer in daily practice over a 3-year period at a tertiary referral centre receiving scans from a large number of hospitals with varying protocols. To compare these daily practice results to published data from single-centre studies. METHODS: After ethical approval, MRI staging records for 270 studies from nine network and three centre hospitals were retrospectively collected and compared with final operative histopathology. The International Federation of Gynaecology and Obstetrics (FIGO) stage, depth of invasion assessment and cervical stromal invasion were analysed and reasons for discrepancies reviewed. RESULTS: MRI-based complete FIGO stage was fully concordant with histopathology in 65.6%. MRI accuracy for depth of myometrial invasion and cervical stromal invasion was 73.3% and 89.3% respectively. Our results did not match the high accuracy previously reported in studies based on single centres. CONCLUSIONS: Published MRI staging accuracy from small single-centre studies were not replicated in a tertiary referral centre receiving scans with heterogeneous protocols over a 3-year period. These results highlight the challenges faced in daily practice and may reflect achievable and realistic MRI staging accuracies in large rapid throughput referral networks. Adherence to standardised high-quality protocols may help to improve future results. KEY POINTS: • Three-year MRI-staging accuracy for endometrial cancer in a multicentre cancer network • Daily practice MRI-staging accuracy did not meet results of single-centre studies • Large scale cancer network MRI-staging accuracies should be further evaluated • Treatment recommendations should be based on achievable MRI-staging accuracies.

Histogram analysis of apparent diffusion coefficient from whole-body diffusion-weighted MRI to predict early response to chemotherapy in patients with metastatic colorectal cancer: preliminary results.

AIM: To evaluate apparent diffusion coefficient (ADC) histogram analysis parameters, acquired from whole-body diffusion-weighted magnetic resonance imaging (DW-MRI), as very early predictors of response to chemotherapy in patients with metastatic colorectal cancer (mCRC). MATERIALS AND METHODS: This was a single-institution prospective study, approved by the West Midlands-South Birmingham research ethics committee. All patients gave fully informed consent prior to imaging. Sixteen patients with histologically confirmed mCRC were enrolled to the study and 11 were successfully scanned with whole-body DW-MRI before (baseline) and 10.8±2.7 days after commencing chemotherapy (follow-up). Therapy response was assessed by RECIST 1.1. Mean ADC and histogram parameters (skewness, kurtosis, 25th, 50th, and 75th percentiles) were compared between progressors and non-progressors at baseline and follow-up. Receiver operating characteristics (ROC) analysis was performed for the statistically significant parameters. Data from metastases were also compared to normative tissue data acquired from healthy volunteers. RESULTS: Three patients had progressive disease (progressors) and eight had partial response/stable disease (non-progressors). Mean, 25th, 50th, and 75th percentiles were significantly lower for progressors at baseline (p=0.012, 0.012, 0.012 and 0.025 respectively) with areas under the ROC curves (AUC)=0.58, 0.50, 0.58 and 0.63, respectively. Skewness and kurtosis were significantly lower for non-progressors at follow-up (p=0.001 and 0.003 respectively) with AUC=0.67 and 0.79 respectively. CONCLUSION: ADC histogram analysis shows potential in discriminating progressive from non-progressive disease in patients with mCRC, who underwent whole-body DW-MRI. The technique can potentially be tested as a response assessment methodology in larger trials.

European society of urogenital radiology (ESUR) guidelines: MR imaging of pelvic endometriosis.

Endometriosis is a common gynaecological condition of unknown aetiology that primarily affects women of reproductive age. The accepted first-line imaging modality is pelvic ultrasound. However, magnetic resonance imaging (MRI) is increasingly performed as an additional investigation in complex cases and for surgical planning. There is currently no international consensus regarding patient preparation, MRI protocols or reporting criteria. Our aim was to develop clinical guidelines for MRI evaluation of pelvic endometriosis based on literature evidence and consensus expert opinion. This work was performed by a group of radiologists from the European Society of Urogenital Radiology (ESUR), experts in gynaecological imaging and a gynaecologist expert in methodology. The group discussed indications for MRI, technical requirements, patient preparation, MRI protocols and criteria for the diagnosis of pelvic endometriosis on MRI. The expert panel proposed a final recommendation for each criterion using Oxford Centre for Evidence Based Medicine (OCEBM) 2011 levels of evidence. KEY POINTS: • This report provides guidelines for MRI in endometriosis. • Minimal and optimal MRI acquisition protocols are provided. • Recommendations are proposed for patient preparation, best MRI sequences and reporting criteria.

Correlation of pre-operative CT findings with surgical & histological tumor dissemination patterns at cytoreduction for primary advanced and relapsed epithelial ovarian cancer: A retrospective evaluation.

OBJECTIVES: Computed tomography (CT) is an essential part of preoperative planning prior to cytoreductive surgery for primary and relapsed epithelial ovarian cancer (EOC). Our aim is to correlate pre-operative CT results with intraoperative surgical and histopathological findings at debulking surgery. METHODS: We performed a systematic comparison of intraoperative tumor dissemination patterns and surgical resections with preoperative CT assessments of infiltrative disease at key resection sites, in women who underwent multivisceral debulking surgery due to EOC between January 2013 and December 2014 at a tertiary referral center. The key sites were defined as follows: diaphragmatic involvement(DI), splenic disease (SI), large (LBI) and small (SBI) bowel involvement, rectal involvement (RI), porta hepatis involvement (PHI), mesenteric disease (MI) and lymph node involvement (LNI). RESULTS: A total of 155 patients, mostly with FIGO stage IIIC disease (65%) were evaluated (primary=105, relapsed=50). Total macroscopic cytoreduction rates were: 89%. Pre-operative CT findings displayed high specificity across all tumor sites apart from the retroperitoneal lymph node status, with a specificity of 65%. The ability however of the CT to accurately identify sites affected by invasive disease was relatively low with the following sensitivities as relating to final histology: 32% (DI), 26% (SI), 46% (LBI), 44% (SBI), 39% (RI), 57% (PHI), 31% (MI), 63% (LNI). CONCLUSION: Pre-operative CT imaging shows high specificity but low sensitivity in detecting tumor involvement at key sites in ovarian cancer surgery. CT findings alone should not be used for surgical decision making.

Added Value of Assessing Adnexal Masses with Advanced MRI Techniques.

This review will present the added value of perfusion and diffusion MR sequences to characterize adnexal masses. These two functional MR techniques are readily available in routine clinical practice. We will describe the acquisition parameters and a method of analysis to optimize their added value compared with conventional images. We will then propose a model of interpretation that combines the anatomical and morphological information from conventional MRI sequences with the functional information provided by perfusion and diffusion weighted sequences.

Use of an internal reference in semi-quantitative dynamic contrast-enhanced MRI (DCE MRI) of indeterminate adnexal masses.

OBJECTIVE: Semi-quantitative dynamic contrast-enhanced MRI (DCE MRI) has proven useful in discriminating benign from borderline/malignant adnexal lesions. Our aim was to assess if the use of a lesion-to-internal-reference ratio improved the performance in characterizing adnexal masses and which internal reference was suitable. METHODS: Semi-quantitative DCE MRI images of 71 indeterminate adnexal lesions were retrospectively reviewed. A region of interest was manually drawn onto the enhancing solid component, psoas muscle and normal outer myometrium. The DCE parameters were evaluated, and the lesion-to-internal-reference ratios were calculated. RESULTS: When the wash in rate of the lesion was higher than that of the myometrium, 97% specificity and 12% sensitivity for borderline/malignancy was reached. When the maximum relative enhancement and maximum absolute enhancement (SImax) of the lesion was less than those of the psoas, 100% specificity for benignity was achieved. The highest area under the curve (AUC) (0.807) was achieved using a SImax lesion-myometrium ratio. A slightly lower AUC (0.799) was achieved using a SImax lesion-psoas ratio, but the psoas muscle was more frequently measurable in the same slice as the lesion ROI. Although the AUC was higher, when using ratios instead of individual DCE values, this was not significantly different. CONCLUSION: DCE MRI has added diagnostic value in the assessment of adnexal lesions, and the use of internal references enables high specificity for malignancy and benignity. Lesion-internal-reference ratios have no added diagnostic value over DCE values alone. ADVANCES IN KNOWLEDGE: Both psoas muscle and myometrium are suitable internal references in the DCE assessment of adnexal lesions enabling high specificity for malignancy and benignity.