Improving the repeatability of deep learning models with Monte Carlo dropout.

The integration of artificial intelligence into clinical workflows requires reliable and robust models. Repeatability is a key attribute of model robustness. Ideal repeatable models output predictions without variation during independent tests carried out under similar conditions. However, slight variations, though not ideal, may be unavoidable and acceptable in practice. During model development and evaluation, much attention is given to classification performance while model repeatability is rarely assessed, leading to the development of models that are unusable in clinical practice. In this work, we evaluate the repeatability of four model types (binary classification, multi-class classification, ordinal classification, and regression) on images that were acquired from the same patient during the same visit. We study the each model's performance on four medical image classification tasks from public and private datasets: knee osteoarthritis, cervical cancer screening, breast density estimation, and retinopathy of prematurity. Repeatability is measured and compared on ResNet and DenseNet architectures. Moreover, we assess the impact of sampling Monte Carlo dropout predictions at test time on classification performance and repeatability. Leveraging Monte Carlo predictions significantly increases repeatability, in particular at the class boundaries, for all tasks on the binary, multi-class, and ordinal models leading to an average reduction of the 95% limits of agreement by 16% points and of the class disagreement rate by 7% points. The classification accuracy improves in most settings along with the repeatability. Our results suggest that beyond about 20 Monte Carlo iterations, there is no further gain in repeatability. In addition to the higher test-retest agreement, Monte Carlo predictions are better calibrated which leads to output probabilities reflecting more accurately the true likelihood of being correctly classified.

A demonstration of automated visual evaluation of cervical images taken with a smartphone camera.

We examined whether automated visual evaluation (AVE), a deep learning computer application for cervical cancer screening, can be used on cervix images taken by a contemporary smartphone camera. A large number of cervix images acquired by the commercial MobileODT EVA system were filtered for acceptable visual quality and then 7587 filtered images from 3221 women were annotated by a group of gynecologic oncologists (so the gold standard is an expert impression, not histopathology). We tested and analyzed on multiple random splits of the images using two deep learning, object detection networks. For all the receiver operating characteristics curves, the area under the curve values for the discrimination of the most likely precancer cases from least likely cases (most likely controls) were above 0.90. These results showed that AVE can classify cervix images with confidence scores that are strongly associated with expert evaluations of severity for the same images. The results on a small subset of images that have histopathologic diagnoses further supported the capability of AVE for predicting cervical precancer. We examined the associations of AVE severity score with gynecologic oncologist impression at all regions where we had a sufficient number of cases and controls, and the influence of a woman's age. The method was found generally resilient to regional variation in the appearance of the cervix. This work suggests that using AVE on smartphones could be a useful adjunct to health-worker visual assessment with acetic acid, a cervical cancer screening method commonly used in low- and middle-resource settings.

Risk Estimates Supporting the 2019 ASCCP Risk-Based Management Consensus Guidelines.

The 2019 American Society for Colposcopy and Cervical Pathology Risk-Based Management Consensus Guidelines for the management of cervical cancer screening abnormalities recommend 1 of 6 clinical actions (treatment, optional treatment or colposcopy/biopsy, colposcopy/biopsy, 1-year surveillance, 3-year surveillance, 5-year return to regular screening) based on the risk of cervical intraepithelial neoplasia grade 3, adenocarcinoma in situ, or cancer (CIN 3+) for the many different combinations of current and recent past screening results. This article supports the main guidelines presentation by presenting and explaining the risk estimates that supported the guidelines. METHODS: From 2003 to 2017 at Kaiser Permanente Northern California (KPNC), 1.5 million individuals aged 25 to 65 years were screened with human papillomavirus (HPV) and cytology cotesting scheduled every 3 years. We estimated immediate and 5-year risks of CIN 3+ for combinations of current test results paired with history of screening test and colposcopy/biopsy results. RESULTS: Risk tables are presented for different clinical scenarios. Examples of important results are highlighted; for example, the risk posed by most current abnormalities is greatly reduced if the prior screening round was HPV-negative. The immediate and 5-year risks of CIN 3+ used to decide clinical management are shown. CONCLUSIONS: The new risk-based guidelines present recommendations for the management of abnormal screening test and histology results; the key risk estimates supporting guidelines are presented in this article. Comprehensive risk estimates are freely available online at https://CervixCa.nlm.nih.gov/RiskTables.

A Study of Partial Human Papillomavirus Genotyping in Support of the 2019 ASCCP Risk-Based Management Consensus Guidelines.

INTRODUCTION: The 2019 ASCCP Risk-Based Management Consensus Guidelines include recommendations for partial human papillomavirus (HPV) genotyping in management of abnormal cervical cancer screening results. The guidelines are based on matching estimates of cervical intraepithelial neoplasia (CIN) 3+ risk to consensus clinical action thresholds. In support of the guidelines, this analysis addresses the risks predicted by individual identification of HPV 16 and HPV 18. METHODS: Risk estimates were drawn from a subset of women in the Kaiser Permanente Northern California screening program, whose residual cervical specimens were HPV typed as part of the HPV Persistence and Progression study. We calculated risk of CIN 3+ to assess how identification of HPV 16, HPV 18, or 12 other "high-risk" HPV types would influence recommended clinical management of new abnormal screening results, taking into account current cytologic results and recent screening history. Immediate and/or 5-year risks of CIN 3+ were matched to clinical actions identified in the guidelines. RESULTS: Identification of HPV 16 at the first visit including HPV testing elevated immediate risk of diagnosing CIN 3+ sufficiently to mandate colposcopic referral even when cytology was Negative for Intraepithelial Lesions or Malignancy and to support a preference for treatment of cytologic high-grade squamous intraepithelial lesion. HPV 18 less clearly elevated CIN 3+ risk. CONCLUSIONS: Identification of HPV 16 clearly mandated consideration in clinical management of new abnormal screening results. HPV 18 positivity must be considered as a special situation because of established disproportionate risk of invasive cancer. More detailed genotyping and use beyond initial management will be considered in guideline updates.

Assessment of a New Lower-Cost Real-Time PCR Assay for Detection of High-Risk Human Papillomavirus: Useful for Cervical Screening in Limited-Resource Settings?

Inexpensive and easy-to-perform human papillomavirus (HPV) tests are needed for primary cervical cancer screening in lower-resource regions. In a convenience sample of 516 residual exfoliative cervical specimens from the Kaiser Permanente Northern California and U.S. National Cancer Institute Persistence and Progression Study, we assessed the agreement and clinical performance of a simple, inexpensive real-time PCR assay for the detection of 13 carcinogenic HPV types (the H13 assay; Hybribio, Hong Kong) that is marketed in limited-resource settings compared to previous testing by the Hybrid Capture 2 assay (HC2; Qiagen, Germantown, MD) and the Onclarity assay (BD Diagnostics, Sparks, MD). The test set was chosen to include many HPV-positive specimens. The reference standard was a combination of HC2 and Onclarity results for HPV detection and histologic diagnosis of controls (less than cervical intraepithelial neoplasia grade 2 [