Target Product Profile for a Machine Learning-Automated Retinal Imaging Analysis Software for Use in English Diabetic Eye Screening: Protocol for a Mixed Methods Study.

BACKGROUND: Diabetic eye screening (DES) represents a significant opportunity for the application of machine learning (ML) technologies, which may improve clinical and service outcomes. However, successful integration of ML into DES requires careful product development, evaluation, and implementation. Target product profiles (TPPs) summarize the requirements necessary for successful implementation so these can guide product development and evaluation. OBJECTIVE: This study aims to produce a TPP for an ML-automated retinal imaging analysis software (ML-ARIAS) system for use in DES in England. METHODS: This work will consist of 3 phases. Phase 1 will establish the characteristics to be addressed in the TPP. A list of candidate characteristics will be generated from the following sources: an overview of systematic reviews of diagnostic test TPPs; a systematic review of digital health TPPs; and the National Institute for Health and Care Excellence's Evidence Standards Framework for Digital Health Technologies. The list of characteristics will be refined and validated by a study advisory group (SAG) made up of representatives from key stakeholders in DES. This includes people with diabetes; health care professionals; health care managers and leaders; and regulators and policy makers. In phase 2, specifications for these characteristics will be drafted following a series of semistructured interviews with participants from these stakeholder groups. Data collected from these interviews will be analyzed using the shortlist of characteristics as a framework, after which specifications will be drafted to create a draft TPP. Following approval by the SAG, in phase 3, the draft will enter an internet-based Delphi consensus study with participants sought from the groups previously identified, as well as ML-ARIAS developers, to ensure feasibility. Participants will be invited to score characteristic and specification pairs on a scale from "definitely exclude" to "definitely include," and suggest edits. The document will be iterated between rounds based on participants' feedback. Feedback on the draft document will be sought from a group of ML-ARIAS developers before its final contents are agreed upon in an in-person consensus meeting. At this meeting, representatives from the stakeholder groups previously identified (minus ML-ARIAS developers, to avoid bias) will be presented with the Delphi results and feedback of the user group and asked to agree on the final contents by vote. RESULTS: Phase 1 was completed in November 2023. Phase 2 is underway and expected to finish in March 2024. Phase 3 is expected to be complete in July 2024. CONCLUSIONS: The multistakeholder development of a TPP for an ML-ARIAS for use in DES in England will help developers produce tools that serve the needs of patients, health care providers, and their staff. The TPP development process will also provide methods and a template to produce similar documents in other disease areas. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/50568.

Development and quality appraisal of a new English breast screening linked data set as part of the age, test threshold, and frequency of mammography screening (ATHENA-M) study.

OBJECTIVES: To build a data set capturing the whole breast cancer screening journey from individual breast cancer screening records to outcomes and assess data quality. METHODS: Routine screening records (invitation, attendance, test results) from all 79 English NHS breast screening centres between January 1, 1988 and March 31, 2018 were linked to cancer registry (cancer characteristics and treatment) and national mortality data. Data quality was assessed using comparability, validity, timeliness, and completeness. RESULTS: Screening records were extracted from 76/79 English breast screening centres, 3/79 were not possible due to software issues. Data linkage was successful from 1997 after introduction of a universal identifier for women (NHS number). Prior to 1997 outcome data are incomplete due to linkage issues, reducing validity. Between January 1, 1997 and March 31, 2018, a total of 11 262 730 women were offered screening of whom 9 371 973 attended at least one appointment, with 139 million person-years of follow-up (a median of 12.4 person years for each woman included) with 73 810 breast cancer deaths and 1 111 139 any-cause deaths. Comparability to reference data sets and internal validity were demonstrated. Data completeness was high for core screening variables (>99%) and main cancer outcomes (>95%). CONCLUSIONS: The ATHENA-M project has created a large high-quality and representative data set of individual women's screening trajectories and outcomes in England from 1997 to 2018, data before 1997 are lower quality. ADVANCES IN KNOWLEDGE: This is the most complete data set of English breast screening records and outcomes constructed to date, which can be used to evaluate and optimize screening.

Lessons learned from independent external validation of an AI tool to detect breast cancer using a representative UK data set.

OBJECTIVE: To pilot a process for the independent external validation of an artificial intelligence (AI) tool to detect breast cancer using data from the NHS breast screening programme (NHSBSP). METHODS: A representative data set of mammography images from 26,000 women attending 2 NHS screening centres, and an enriched data set of 2054 positive cases were used from the OPTIMAM image database. The use case of the AI tool was the replacement of the first or second human reader. The performance of the AI tool was compared to that of human readers in the NHSBSP. RESULTS: Recommendations for future external validations of AI tools to detect breast cancer are provided. The tool recalled different breast cancers to the human readers. This study showed the importance of testing AI tools on all types of cases (including non-standard) and the clarity of any warning messages. The acceptable difference in sensitivity and specificity between the AI tool and human readers should be determined. Any information vital for the clinical application should be a required output for the AI tool. It is recommended that the interaction of radiologists with the AI tool, and the effect of the AI tool on arbitration be investigated prior to clinical use. CONCLUSION: This pilot demonstrated several lessons for future independent external validation of AI tools for breast cancer detection. ADVANCES IN KNOWLEDGE: Knowledge has been gained towards best practice procedures for performing independent external validations of AI tools for the detection of breast cancer using data from the NHS Breast Screening Programme.

UK National Screening Committee's approach to reviewing evidence on artificial intelligence in breast cancer screening.

Artificial intelligence (AI) could have the potential to accurately classify mammograms according to the presence or absence of radiological signs of breast cancer, replacing or supplementing human readers (radiologists). The UK National Screening Committee's assessments of the use of AI systems to examine screening mammograms continues to focus on maximising benefits and minimising harms to women screened, when deciding whether to recommend the implementation of AI into the Breast Screening Programme in the UK. Maintaining or improving programme specificity is important to minimise anxiety from false positive results. When considering cancer detection, AI test sensitivity alone is not sufficiently informative, and additional information on the spectrum of disease detected and interval cancers is crucial to better understand the benefits and harms of screening. Although large retrospective studies might provide useful evidence by directly comparing test accuracy and spectrum of disease detected between different AI systems and by population subgroup, most retrospective studies are biased due to differential verification (ie, the use of different reference standards to verify the target condition among study participants). Enriched, multiple-reader, multiple-case, test set laboratory studies are also biased due to the laboratory effect (ie, radiologists' performance in retrospective, laboratory, observer studies is substantially different to their performance in a clinical environment). Therefore, assessment of the effect of incorporating any AI system into the breast screening pathway in prospective studies is required as it will provide key evidence for the effect of the interaction of medical staff with AI, and the impact on women's outcomes.

Virtual clinical trial to compare cancer detection using combinations of 2D mammography, digital breast tomosynthesis and synthetic 2D imaging.

OBJECTIVES: This study was designed to compare the detection of subtle lesions (calcification clusters or masses) when using the combination of digital breast tomosynthesis (DBT) and synthetic mammography (SM) with digital mammography (DM) alone or combined with DBT. METHODS: A set of 166 cases without cancer was acquired on a DBT mammography system. Realistic subtle calcification clusters and masses in the DM images and DBT planes were digitally inserted into 104 of the acquired cases. Three study arms were created: DM alone, DM with DBT and SM with DBT. Five mammographic readers located the centre of any lesion within the images that should be recalled for further investigation and graded their suspiciousness. A JAFROC figure of merit (FoM) and lesion detection fraction (LDF) were calculated for each study arm. The visibility of the lesions in the DBT images was compared with SM and DM images. RESULTS: For calcification clusters, there were no significant differences (p > 0.075) in FoM or LDF. For masses, the FoM and LDF were significantly improved in the arms using DBT compared to DM alone (p < 0.001). On average, both calcification clusters and masses were more visible on DBT than on DM and SM images. CONCLUSIONS: This study demonstrated that masses were detected better with DBT than with DM alone and there was no significant difference (p = 0.075) in LDF between DM&DBT and SM&DBT for calcifications clusters. Our results support previous studies that it may be acceptable to not acquire digital mammography alongside tomosynthesis for subtle calcification clusters and ill-defined masses. KEY POINTS: • The detection of masses was significantly better using DBT than with digital mammography alone. • The detection of calcification clusters was not significantly different between digital mammography and synthetic 2D images combined with tomosynthesis. • Our results support previous studies that it may be acceptable to not acquire digital mammography alongside tomosynthesis for subtle calcification clusters and ill-defined masses for the imaging technology used.

OPTIMAM Mammography Image Database: A Large-Scale Resource of Mammography Images and Clinical Data.

Supplemental material is available for this article.

Impact of Digital Mammography on Cancer Detection and Recall Rates: 11.3 Million Screening Episodes in the English National Health Service Breast Cancer Screening Program.

Purpose To report the impact of changing from screen-film mammography to digital mammography (DM) in a large organized national screening program. Materials and Methods A retrospective analysis of prospectively collected annual screening data from 2009-2010 to 2015-2016 for the 80 facilities of the English National Health Service Breast Cancer Screening Program, together with estimates of DM usage for three time periods, enabled the effect of DM to be measured in a study of 11.3 million screening episodes in women aged 45-70 years (mean age, 59 years). Regression models were used to estimate percentage and absolute change in detection rates due to DM. Results The overall cancer detection rate was 14% greater with DM (P < .001). There were higher rates of detection of grade 1 and 2 invasive cancers (both ductal and lobular), but no change in the detection of grade 3 invasive cancers. The recall rate was almost unchanged by the introduction of DM. At prevalent (first) screening episodes for women aged 45-52 years, DM increased the overall detection rate by 19% (P < .001) and for incident screening episodes in women aged 53-70 years by 13% (P < .001). Conclusion The overall cancer detection rate was 14% greater with digital mammography with no change in recall rates and without confounding by changes in other factors. There was a substantially higher detection of grade 1 and grade 2 invasive cancers, including both ductal and lobular cancers, but no change in the detection of grade 3 invasive cancers. © RSNA, 2018 Online supplemental material is available for this article. See also the editorial by C.I. Lee and J.M. Lee in this issue.

Commentary on duty of candour and cancer screening.

English law mandates a duty of candour (DOC) for all healthcare providers. They must be open and honest when something goes wrong with care causing harm. Providers must apologize to those affected and investigate what happened. Screening is not 100% accurate and false positive and false negative results are inevitable. Guidance on DOC assists providers to judge when something has gone wrong in screening and the DOC legislation applies. DOC guidance helps distinguish such incidents from harms that are an expected and inevitable consequence of the imperfections of screening tests. For breast cancer screening the classification of interval cancers has been updated to take account of DOC. This guidance on DOC and classification of prior films of those presenting with interval cancers has relevance to other areas of diagnostic imaging. Review of prior examinations after a significant diagnosis has been made may reveal a previously overlooked abnormality.

Design and validation of realistic breast models for use in multiple alternative forced choice virtual clinical trials.

A novel method has been developed for generating quasi-realistic voxel phantoms which simulate the compressed breast in mammography and digital breast tomosynthesis (DBT). The models are suitable for use in virtual clinical trials requiring realistic anatomy which use the multiple alternative forced choice (AFC) paradigm and patches from the complete breast image. The breast models are produced by extracting features of breast tissue components from DBT clinical images including skin, adipose and fibro-glandular tissue, blood vessels and Cooper's ligaments. A range of different breast models can then be generated by combining these components. Visual realism was validated using a receiver operating characteristic (ROC) study of patches from simulated images calculated using the breast models and from real patient images. Quantitative analysis was undertaken using fractal dimension and power spectrum analysis. The average areas under the ROC curves for 2D and DBT images were 0.51 ± 0.06 and 0.54 ± 0.09 demonstrating that simulated and real images were statistically indistinguishable by expert breast readers (7 observers); errors represented as one standard error of the mean. The average fractal dimensions (2D, DBT) for real and simulated images were (2.72 ± 0.01, 2.75 ± 0.01) and (2.77 ± 0.03, 2.82 ± 0.04) respectively; errors represented as one standard error of the mean. Excellent agreement was found between power spectrum curves of real and simulated images, with average ß values (2D, DBT) of (3.10 ± 0.17, 3.21 ± 0.11) and (3.01 ± 0.32, 3.19 ± 0.07) respectively; errors represented as one standard error of the mean. These results demonstrate that radiological images of these breast models realistically represent the complexity of real breast structures and can be used to simulate patches from mammograms and DBT images that are indistinguishable from patches from the corresponding real breast images. The method can generate about 500 radiological patches (~30 mm × 30 mm) per day for AFC experiments on a single workstation. This is the first study to quantitatively validate the realism of simulated radiological breast images using direct blinded comparison with real data via the ROC paradigm with expert breast readers.

The relationship between cancer detection in mammography and image quality measurements.

PURPOSE: To investigate the relationship between image quality measurements and the clinical performance of digital mammographic systems. METHODS: Mammograms containing subtle malignant non-calcification lesions and simulated malignant calcification clusters were adapted to appear as if acquired by four types of detector. Observers searched for suspicious lesions and gave these a malignancy score. Analysis was undertaken using jackknife alternative free-response receiver operating characteristics weighted figure of merit (FoM). Images of a CDMAM contrast-detail phantom were adapted to appear as if acquired using the same four detectors as the clinical images. The resultant threshold gold thicknesses were compared to the FoMs using a linear regression model and an F-test was used to find if the gradient of the relationship was significantly non-zero. RESULTS: The detectors with the best image quality measurement also had the highest FoM values. The gradient of the inverse relationship between FoMs and threshold gold thickness for the 0.25mm diameter disk was significantly different from zero for calcification clusters (p=0.027), but not for non-calcification lesions (p=0.11). Systems performing just above the minimum image quality level set in the European Guidelines for Quality Assurance in Breast Cancer Screening and Diagnosis resulted in reduced cancer detection rates compared to systems performing at the achievable level. CONCLUSIONS: The clinical effectiveness of mammography for the task of detecting calcification clusters was found to be linked to image quality assessment using the CDMAM phantom. The European Guidelines should be reviewed as the current minimum image quality standards may be too low.

Breast cancer detection rates using four different types of mammography detectors.

OBJECTIVE: To compare the performance of different types of detectors in breast cancer detection. METHODS: A mammography image set containing subtle malignant non-calcification lesions, biopsy-proven benign lesions, simulated malignant calcification clusters and normals was acquired using amorphous-selenium (a-Se) detectors. The images were adapted to simulate four types of detectors at the same radiation dose: digital radiography (DR) detectors with a-Se and caesium iodide (CsI) convertors, and computed radiography (CR) detectors with a powder phosphor (PIP) and a needle phosphor (NIP). Seven observers marked suspicious and benign lesions. Analysis was undertaken using jackknife alternative free-response receiver operating characteristics weighted figure of merit (FoM). The cancer detection fraction (CDF) was estimated for a representative image set from screening. RESULTS: No significant differences in the FoMs between the DR detectors were measured. For calcification clusters and non-calcification lesions, both CR detectors' FoMs were significantly lower than for DR detectors. The calcification cluster's FoM for CR NIP was significantly better than for CR PIP. The estimated CDFs with CR PIP and CR NIP detectors were up to 15% and 22% lower, respectively, than for DR detectors. CONCLUSION: Cancer detection is affected by detector type, and the use of CR in mammography should be reconsidered. KEY POINTS: The type of mammography detector can affect the cancer detection rates. CR detectors performed worse than DR detectors in mammography. Needle phosphor CR performed better than powder phosphor CR. Calcification clusters detection is more sensitive to detector type than other cancers.

The effect of image processing on the detection of cancers in digital mammography.

OBJECTIVE. The objective of our study was to investigate the effect of image processing on the detection of cancers in digital mammography images. MATERIALS AND METHODS. Two hundred seventy pairs of breast images (both breasts, one view) were collected from eight systems using Hologic amorphous selenium detectors: 80 image pairs showed breasts containing subtle malignant masses; 30 image pairs, biopsy-proven benign lesions; 80 image pairs, simulated calcification clusters; and 80 image pairs, no cancer (normal). The 270 image pairs were processed with three types of image processing: standard (full enhancement), low contrast (intermediate enhancement), and pseudo-film-screen (no enhancement). Seven experienced observers inspected the images, locating and rating regions they suspected to be cancer for likelihood of malignancy. The results were analyzed using a jackknife-alternative free-response receiver operating characteristic (JAFROC) analysis. RESULTS. The detection of calcification clusters was significantly affected by the type of image processing: The JAFROC figure of merit (FOM) decreased from 0.65 with standard image processing to 0.63 with low-contrast image processing (p = 0.04) and from 0.65 with standard image processing to 0.61 with film-screen image processing (p = 0.0005). The detection of noncalcification cancers was not significantly different among the image-processing types investigated (p > 0.40). CONCLUSION. These results suggest that image processing has a significant impact on the detection of calcification clusters in digital mammography. For the three image-processing versions and the system investigated, standard image processing was optimal for the detection of calcification clusters. The effect on cancer detection should be considered when selecting the type of image processing in the future.

Effect of image quality on calcification detection in digital mammography.

PURPOSE: This study aims to investigate if microcalcification detection varies significantly when mammographic images are acquired using different image qualities, including: different detectors, dose levels, and different image processing algorithms. An additional aim was to determine how the standard European method of measuring image quality using threshold gold thickness measured with a CDMAM phantom and the associated limits in current EU guidelines relate to calcification detection. METHODS: One hundred and sixty two normal breast images were acquired on an amorphous selenium direct digital (DR) system. Microcalcification clusters extracted from magnified images of slices of mastectomies were electronically inserted into half of the images. The calcification clusters had a subtle appearance. All images were adjusted using a validated mathematical method to simulate the appearance of images from a computed radiography (CR) imaging system at the same dose, from both systems at half this dose, and from the DR system at quarter this dose. The original 162 images were processed with both Hologic and Agfa (Musica-2) image processing. All other image qualities were processed with Agfa (Musica-2) image processing only. Seven experienced observers marked and rated any identified suspicious regions. Free response operating characteristic (FROC) and ROC analyses were performed on the data. The lesion sensitivity at a nonlesion localization fraction (NLF) of 0.1 was also calculated. Images of the CDMAM mammographic test phantom were acquired using the automatic setting on the DR system. These images were modified to the additional image qualities used in the observer study. The images were analyzed using automated software. In order to assess the relationship between threshold gold thickness and calcification detection a power law was fitted to the data. RESULTS: There was a significant reduction in calcification detection using CR compared with DR: the alternative FROC (AFROC) area decreased from 0.84 to 0.63 and the ROC area decreased from 0.91 to 0.79 (p < 0.0001). This corresponded to a 30% drop in lesion sensitivity at a NLF equal to 0.1. Detection was also sensitive to the dose used. There was no significant difference in detection between the two image processing algorithms used (p > 0.05). It was additionally found that lower threshold gold thickness from CDMAM analysis implied better cluster detection. The measured threshold gold thickness passed the acceptable limit set in the EU standards for all image qualities except half dose CR. However, calcification detection varied significantly between image qualities. This suggests that the current EU guidelines may need revising. CONCLUSIONS: Microcalcification detection was found to be sensitive to detector and dose used. Standard measurements of image quality were a good predictor of microcalcification cluster detection.

The number of women who would need to be screened regularly by mammography to prevent one death from breast cancer.

The number of women who would need to be screened regularly by mammography to prevent one death from breast cancer depends strongly on several factors, including the age at which regular screening starts, the period over which it continues, and the duration of follow-up after screening. Furthermore, more women would need to be INVITED for screening than would need to be SCREENED to prevent one death, since not all women invited attend for screening or are screened regularly. Failure to consider these important factors accounts for many of the major discrepancies between different published estimates. The randomised evidence indicates that, in high income countries, around one breast cancer death would be prevented in the long term for every 400 women aged 50-70 years regularly screened over a ten-year period.

Computer-aided detection in the United Kingdom National Breast Screening Programme: prospective study.

PURPOSE: To evaluate prospectively the recall and cancer detection rates with and without computer-aided detection (CAD) in the United Kingdom National Health Service Breast Screening Programme. MATERIALS AND METHODS: The study had appropriate ethics committee approval. Informed consent was not required; however, patients were informed that their mammograms might be used in research efforts, and all patients agreed to participate. Mammograms obtained in 6111 women (mean age, 58.4 years) undergoing routine screening every 3 years were analyzed with a CAD system. Mammograms were independently double read. Twelve readers participated. Readers recorded an initial evaluation, viewed the CAD prompts, and recorded a final evaluation. Recall to assessment was decided after arbitration. Sensitivities were calculated for single reading, single reading with CAD, and double reading, as a proportion of the total number of cancers detected by using double reading with CAD. RESULTS: A total of 62 cancers were detected in 61 women. CAD prompted 51 (84%) of 61 radiographically detected cancers. Of 12 cancers missed on single reading, nine were correctly prompted; however, seven prompts were overruled by the reader. Sensitivity of single reading was 90.2% (95% confidence interval [CI]: 83.0%, 95.0%), single reading with CAD was 91.5% (95% CI: 85.0%, 96.0%), and double reading without CAD was 98.4% (95% CI: 91.0%, 100%). Cancer detection rate was 1%. Recall to assessment rate was 6.1%, with an increase of 5.8% because of CAD. Average time required, per reader, to read a case was 25 seconds without CAD and 45 seconds with CAD. CONCLUSION: CAD increases sensitivity of single reading by 1.3%, whereas double reading increases sensitivity by 8.2%.