Successful Development of a Natural Language Processing Algorithm for Pancreatic Neoplasms and Associated Histologic Features.

OBJECTIVES: Natural language processing (NLP) algorithms can interpret unstructured text for commonly used terms and phrases. Pancreatic pathologies are diverse and include benign and malignant entities with associated histologic features. Creating a pancreas NLP algorithm can aid in electronic health record coding as well as large database creation and curation. METHODS: Text-based pancreatic anatomic and cytopathologic reports for pancreatic cancer, pancreatic ductal adenocarcinoma, neuroendocrine tumor, intraductal papillary neoplasm, tumor dysplasia, and suspicious findings were collected. This dataset was split 80/20 for model training and development. A separate set was held out for testing purposes. We trained using convolutional neural network to predict each heading. RESULTS: Over 14,000 reports were obtained from the Mass General Brigham Healthcare System electronic record. Of these, 1252 reports were used for algorithm development. Final accuracy and F1 scores relative to the test set ranged from 95% and 98% for each queried pathology. To understand the dependence of our results to training set size, we also generated learning curves. Scoring metrics improved as more reports were submitted for training; however, some queries had high index performance. CONCLUSIONS: Natural language processing algorithms can be used for pancreatic pathologies. Increased training volume, nonoverlapping terminology, and conserved text structure improve NLP algorithm performance.

Sybil: A Validated Deep Learning Model to Predict Future Lung Cancer Risk From a Single Low-Dose Chest Computed Tomography.

PURPOSE: Low-dose computed tomography (LDCT) for lung cancer screening is effective, although most eligible people are not being screened. Tools that provide personalized future cancer risk assessment could focus approaches toward those most likely to benefit. We hypothesized that a deep learning model assessing the entire volumetric LDCT data could be built to predict individual risk without requiring additional demographic or clinical data. METHODS: We developed a model called Sybil using LDCTs from the National Lung Screening Trial (NLST). Sybil requires only one LDCT and does not require clinical data or radiologist annotations; it can run in real time in the background on a radiology reading station. Sybil was validated on three independent data sets: a heldout set of 6,282 LDCTs from NLST participants, 8,821 LDCTs from Massachusetts General Hospital (MGH), and 12,280 LDCTs from Chang Gung Memorial Hospital (CGMH, which included people with a range of smoking history including nonsmokers). RESULTS: Sybil achieved area under the receiver-operator curves for lung cancer prediction at 1 year of 0.92 (95% CI, 0.88 to 0.95) on NLST, 0.86 (95% CI, 0.82 to 0.90) on MGH, and 0.94 (95% CI, 0.91 to 1.00) on CGMH external validation sets. Concordance indices over 6 years were 0.75 (95% CI, 0.72 to 0.78), 0.81 (95% CI, 0.77 to 0.85), and 0.80 (95% CI, 0.75 to 0.86) for NLST, MGH, and CGMH, respectively. CONCLUSION: Sybil can accurately predict an individual's future lung cancer risk from a single LDCT scan to further enable personalized screening. Future study is required to understand Sybil's clinical applications. Our model and annotations are publicly available.[Media: see text].

Optimizing risk-based breast cancer screening policies with reinforcement learning.

Screening programs must balance the benefit of early detection with the cost of overscreening. Here, we introduce a novel reinforcement learning-based framework for personalized screening, Tempo, and demonstrate its efficacy in the context of breast cancer. We trained our risk-based screening policies on a large screening mammography dataset from Massachusetts General Hospital (MGH; USA) and validated this dataset in held-out patients from MGH and external datasets from Emory University (Emory; USA), Karolinska Institute (Karolinska; Sweden) and Chang Gung Memorial Hospital (CGMH; Taiwan). Across all test sets, we find that the Tempo policy combined with an image-based artificial intelligence (AI) risk model is significantly more efficient than current regimens used in clinical practice in terms of simulated early detection per screen frequency. Moreover, we show that the same Tempo policy can be easily adapted to a wide range of possible screening preferences, allowing clinicians to select their desired trade-off between early detection and screening costs without training new policies. Finally, we demonstrate that Tempo policies based on AI-based risk models outperform Tempo policies based on less accurate clinical risk models. Altogether, our results show that pairing AI-based risk models with agile AI-designed screening policies has the potential to improve screening programs by advancing early detection while reducing overscreening.

Multi-Institutional Validation of a Mammography-Based Breast Cancer Risk Model.

PURPOSE: Accurate risk assessment is essential for the success of population screening programs in breast cancer. Models with high sensitivity and specificity would enable programs to target more elaborate screening efforts to high-risk populations, while minimizing overtreatment for the rest. Artificial intelligence (AI)-based risk models have demonstrated a significant advance over risk models used today in clinical practice. However, the responsible deployment of novel AI requires careful validation across diverse populations. To this end, we validate our AI-based model, Mirai, across globally diverse screening populations. METHODS: We collected screening mammograms and pathology-confirmed breast cancer outcomes from Massachusetts General Hospital, USA; Novant, USA; Emory, USA; Maccabi-Assuta, Israel; Karolinska, Sweden; Chang Gung Memorial Hospital, Taiwan; and Barretos, Brazil. We evaluated Uno's concordance index for Mirai in predicting risk of breast cancer at one to five years from the mammogram. RESULTS: A total of 128,793 mammograms from 62,185 patients were collected across the seven sites, of which 3,815 were followed by a cancer diagnosis within 5 years. Mirai obtained concordance indices of 0.75 (95% CI, 0.72 to 0.78), 0.75 (95% CI, 0.70 to 0.80), 0.77 (95% CI, 0.75 to 0.79), 0.77 (95% CI, 0.73 to 0.81), 0.81 (95% CI, 0.79 to 0.82), 0.79 (95% CI, 0.76 to 0.83), and 0.84 (95% CI, 0.81 to 0.88) at Massachusetts General Hospital, Novant, Emory, Maccabi-Assuta, Karolinska, Chang Gung Memorial Hospital, and Barretos, respectively. CONCLUSION: Mirai, a mammography-based risk model, maintained its accuracy across globally diverse test sets from seven hospitals across five countries. This is the broadest validation to date of an AI-based breast cancer model and suggests that the technology can offer broad and equitable improvements in care.

Artificial Intelligence and Early Detection of Pancreatic Cancer: 2020 Summative Review.

ABSTRACT: Despite considerable research efforts, pancreatic cancer is associated with a dire prognosis and a 5-year survival rate of only 10%. Early symptoms of the disease are mostly nonspecific. The premise of improved survival through early detection is that more individuals will benefit from potentially curative treatment. Artificial intelligence (AI) methodology has emerged as a successful tool for risk stratification and identification in general health care. In response to the maturity of AI, Kenner Family Research Fund conducted the 2020 AI and Early Detection of Pancreatic Cancer Virtual Summit (www.pdac-virtualsummit.org) in conjunction with the American Pancreatic Association, with a focus on the potential of AI to advance early detection efforts in this disease. This comprehensive presummit article was prepared based on information provided by each of the interdisciplinary participants on one of the 5 following topics: Progress, Problems, and Prospects for Early Detection; AI and Machine Learning; AI and Pancreatic Cancer-Current Efforts; Collaborative Opportunities; and Moving Forward-Reflections from Government, Industry, and Advocacy. The outcome from the robust Summit conversations, to be presented in a future white paper, indicate that significant progress must be the result of strategic collaboration among investigators and institutions from multidisciplinary backgrounds, supported by committed funders.

Toward robust mammography-based models for breast cancer risk.

Improved breast cancer risk models enable targeted screening strategies that achieve earlier detection and less screening harm than existing guidelines. To bring deep learning risk models to clinical practice, we need to further refine their accuracy, validate them across diverse populations, and demonstrate their potential to improve clinical workflows. We developed Mirai, a mammography-based deep learning model designed to predict risk at multiple timepoints, leverage potentially missing risk factor information, and produce predictions that are consistent across mammography machines. Mirai was trained on a large dataset from Massachusetts General Hospital (MGH) in the United States and tested on held-out test sets from MGH, Karolinska University Hospital in Sweden, and Chang Gung Memorial Hospital (CGMH) in Taiwan, obtaining C-indices of 0.76 (95% confidence interval, 0.74 to 0.80), 0.81 (0.79 to 0.82), and 0.79 (0.79 to 0.83), respectively. Mirai obtained significantly higher 5-year ROC AUCs than the Tyrer-Cuzick model ( P < 0.001) and prior deep learning models Hybrid DL ( P < 0.001) and Image-Only DL ( P < 0.001), trained on the same dataset. Mirai more accurately identified high-risk patients than prior methods across all datasets. On the MGH test set, 41.5% (34.4 to 48.5) of patients who would develop cancer within 5 years were identified as high risk, compared with 36.1% (29.1 to 42.9) by Hybrid DL ( P = 0.02) and 22.9% (15.9 to 29.6) by the Tyrer-Cuzick model ( P < 0.001).

External Validation of a Deep Learning Model for Predicting Mammographic Breast Density in Routine Clinical Practice.

RATIONALE AND OBJECTIVES: Federal legislation requires patient notification of dense mammographic breast tissue because increased density is a marker of breast cancer risk and can limit the sensitivity of mammography. As previously described, we clinically implemented our deep learning model at the academic breast imaging practice where the model was developed with high clinical acceptance. Our objective was to externally validate our deep learning model on radiologist breast density assessments in a community breast imaging practice. MATERIALS AND METHODS: Our deep learning model was implemented at a dedicated breast imaging practice staffed by both academic and community breast imaging radiologists in October 2018. Deep learning model assessment of mammographic breast density was presented to the radiologist during routine clinical practice at the time of mammogram interpretation. We identified 2174 consecutive screening mammograms after implementation of the deep learning model. Radiologist agreement with the model's assessment was measured and compared across radiologist groups. RESULTS: Both academic and community radiologists had high clinical acceptance of the deep learning model's density prediction, with 94.9% (academic) and 90.7% (community) acceptance for dense versus nondense categories (p < 0.001). The proportion of mammograms assessed as dense by all radiologists decreased from 47.0% before deep learning model implementation to 41.0% after deep learning model implementation (p < 0.001). CONCLUSION: Our deep learning model had a high clinical acceptance rate among both academic and community radiologists and reduced the proportion of mammograms assessed as dense. This is an important step to validating our deep learning model prior to potential widespread implementation.

Deep Learning to Estimate RECIST in Patients with NSCLC Treated with PD-1 Blockade.

Real-world evidence (RWE), conclusions derived from analysis of patients not treated in clinical trials, is increasingly recognized as an opportunity for discovery, to reduce disparities, and to contribute to regulatory approval. Maximal value of RWE may be facilitated through machine-learning techniques to integrate and interrogate large and otherwise underutilized datasets. In cancer research, an ongoing challenge for RWE is the lack of reliable, reproducible, scalable assessment of treatment-specific outcomes. We hypothesized a deep-learning model could be trained to use radiology text reports to estimate gold-standard RECIST-defined outcomes. Using text reports from patients with non-small cell lung cancer treated with PD-1 blockade in a training cohort and two test cohorts, we developed a deep-learning model to accurately estimate best overall response and progression-free survival. Our model may be a tool to determine outcomes at scale, enabling analyses of large clinical databases. SIGNIFICANCE: We developed and validated a deep-learning model trained on radiology text reports to estimate gold-standard objective response categories used in clinical trial assessments. This tool may facilitate analysis of large real-world oncology datasets using objective outcome metrics determined more reliably and at greater scale than currently possible.This article is highlighted in the In This Issue feature, p. 1.

Exploiting Rules to Enhance Machine Learning in Extracting Information From Multi-Institutional Prostate Pathology Reports.

PURPOSE: Literature on clinical note mining has highlighted the superiority of machine learning (ML) over hand-crafted rules. Nevertheless, most studies assume the availability of large training sets, which is rarely the case. For this reason, in the clinical setting, rules are still common. We suggest 2 methods to leverage the knowledge encoded in pre-existing rules to inform ML decisions and obtain high performance, even with scarce annotations. METHODS: We collected 501 prostate pathology reports from 6 American hospitals. Reports were split into 2,711 core segments, annotated with 20 attributes describing the histology, grade, extension, and location of tumors. The data set was split by institutions to generate a cross-institutional evaluation setting. We assessed 4 systems, namely a rule-based approach, an ML model, and 2 hybrid systems integrating the previous methods: a Rule as Feature model and a Classifier Confidence model. Several ML algorithms were tested, including logistic regression (LR), support vector machine (SVM), and eXtreme gradient boosting (XGB). RESULTS: When training on data from a single institution, LR lags behind the rules by 3.5% (F1 score: 92.2% v 95.7%). Hybrid models, instead, obtain competitive results, with Classifier Confidence outperforming the rules by +0.5% (96.2%). When a larger amount of data from multiple institutions is used, LR improves by +1.5% over the rules (97.2%), whereas hybrid systems obtain +2.2% for Rule as Feature (97.7%) and +2.6% for Classifier Confidence (98.3%). Replacing LR with SVM or XGB yielded similar performance gains. CONCLUSION: We developed methods to use pre-existing handcrafted rules to inform ML algorithms. These hybrid systems obtain better performance than either rules or ML models alone, even when training data are limited.

A Deep Learning Model to Triage Screening Mammograms: A Simulation Study.

Background Recent deep learning (DL) approaches have shown promise in improving sensitivity but have not addressed limitations in radiologist specificity or efficiency. Purpose To develop a DL model to triage a portion of mammograms as cancer free, improving performance and workflow efficiency. Materials and Methods In this retrospective study, 223 109 consecutive screening mammograms performed in 66 661 women from January 2009 to December 2016 were collected with cancer outcomes obtained through linkage to a regional tumor registry. This cohort was split by patient into 212 272, 25 999, and 26 540 mammograms from 56 831, 7021, and 7176 patients for training, validation, and testing, respectively. A DL model was developed to triage mammograms as cancer free and evaluated on the test set. A DL-triage workflow was simulated in which radiologists skipped mammograms triaged as cancer free (interpreting them as negative for cancer) and read mammograms not triaged as cancer free by using the original interpreting radiologists' assessments. Sensitivities, specificities, and percentage of mammograms read were calculated, with and without the DL-triage-simulated workflow. Statistics were computed across 5000 bootstrap samples to assess confidence intervals (CIs). Specificities were compared by using a two-tailed t test (P < .05) and sensitivities were compared by using a one-sided t test with a noninferiority margin of 5% (P < .05). Results The test set included 7176 women (mean age, 57.8 years ± 10.9 [standard deviation]). When reading all mammograms, radiologists obtained a sensitivity and specificity of 90.6% (173 of 191; 95% CI: 86.6%, 94.7%) and 93.5% (24 625 of 26 349; 95% CI: 93.3%, 93.9%). In the DL-simulated workflow, the radiologists obtained a sensitivity and specificity of 90.1% (172 of 191; 95% CI: 86.0%, 94.3%) and 94.2% (24 814 of 26 349; 95% CI: 94.0%, 94.6%) while reading 80.7% (21 420 of 26 540) of the mammograms. The simulated workflow improved specificity (P = .002) and obtained a noninferior sensitivity with a margin of 5% (P < .001). Conclusion This deep learning model has the potential to reduce radiologist workload and significantly improve specificity without harming sensitivity. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Kontos and Conant in this issue.

Incidental breast carcinoma: incidence, management, and outcomes in 4804 bilateral reduction mammoplasties.

INTRODUCTION: Bilateral reduction mammoplasty is one of the most common plastic surgery procedures performed in the U.S. This study examines the incidence, management, and prognosis of incidental breast cancer identified in reduction specimens from a large cohort of reduction mammoplasty patients. METHODS: Breast pathology reports were retrospectively reviewed for evidence of incidental cancers in bilateral reduction mammoplasty specimens from five institutions between 1990 and 2017. RESULTS: A total of 4804 women met the inclusion criteria of this study; incidental cancer was identified in 45 breasts of 39 (0.8%) patients. Six patients (15%) had bilateral cancer. Overall, the maximum diagnosis by breast was 16 invasive cancers and 29 ductal carcinomas in situs. Thirty-three patients had unilateral cancer, 15 (45.5%) of which had high-risk lesions in the contralateral breast. Twenty-one patients underwent mastectomy (12 bilateral and nine unilateral), residual cancer was found in 10 in 25 (40%) therapeutic mastectomies. Seven patients did not undergo mastectomy received breast radiation. The median follow-up was 92 months. No local recurrences were observed in the patients undergoing mastectomy or radiation. Three of 11 (27%) patients who did not undergo mastectomy or radiation developed a local recurrence. The overall survival rate was 87.2% and disease-free survival was 82.1%. CONCLUSIONS: Patients undergoing reduction mammoplasty for macromastia have a small but definite risk of incidental breast cancer. The high rate of bilateral cancer, contralateral high-risk lesions, and residual disease at mastectomy mandates thorough pathologic evaluation and careful follow-up of these patients. Mastectomy or breast radiation is recommended for local control given the high likelihood of local recurrence without either.

Do Neural Information Extraction Algorithms Generalize Across Institutions?

PURPOSE: Natural language processing (NLP) techniques have been adopted to reduce the curation costs of electronic health records. However, studies have questioned whether such techniques can be applied to data from previously unseen institutions. We investigated the performance of a common neural NLP algorithm on data from both known and heldout (ie, institutions whose data were withheld from the training set and only used for testing) hospitals. We also explored how diversity in the training data affects the system's generalization ability. METHODS: We collected 24,881 breast pathology reports from seven hospitals and manually annotated them with nine key attributes that describe types of atypia and cancer. We trained a convolutional neural network (CNN) on annotations from either only one (CNN1), only two (CNN2), or only four (CNN4) hospitals. The trained systems were tested on data from five organizations, including both known and heldout ones. For every setting, we provide the accuracy scores as well as the learning curves that show how much data are necessary to achieve good performance and generalizability. RESULTS: The system achieved a cross-institutional accuracy of 93.87% when trained on reports from only one hospital (CNN1). Performance improved to 95.7% and 96%, respectively, when the system was trained on reports from two (CNN2) and four (CNN4) hospitals. The introduction of diversity during training did not lead to improvements on the known institutions, but it boosted performance on the heldout institutions. When tested on reports from heldout hospitals, CNN4 outperformed CNN1 and CNN2 by 2.13% and 0.3%, respectively. CONCLUSION: Real-world scenarios require that neural NLP approaches scale to data from previously unseen institutions. We show that a common neural NLP algorithm for information extraction can achieve this goal, especially when diverse data are used during training.

A Deep Learning Mammography-based Model for Improved Breast Cancer Risk Prediction.

Background Mammographic density improves the accuracy of breast cancer risk models. However, the use of breast density is limited by subjective assessment, variation across radiologists, and restricted data. A mammography-based deep learning (DL) model may provide more accurate risk prediction. Purpose To develop a mammography-based DL breast cancer risk model that is more accurate than established clinical breast cancer risk models. Materials and Methods This retrospective study included 88 994 consecutive screening mammograms in 39 571 women between January 1, 2009, and December 31, 2012. For each patient, all examinations were assigned to either training, validation, or test sets, resulting in 71 689, 8554, and 8751 examinations, respectively. Cancer outcomes were obtained through linkage to a regional tumor registry. By using risk factor information from patient questionnaires and electronic medical records review, three models were developed to assess breast cancer risk within 5 years: a risk-factor-based logistic regression model (RF-LR) that used traditional risk factors, a DL model (image-only DL) that used mammograms alone, and a hybrid DL model that used both traditional risk factors and mammograms. Comparisons were made to an established breast cancer risk model that included breast density (Tyrer-Cuzick model, version 8 [TC]). Model performance was compared by using areas under the receiver operating characteristic curve (AUCs) with DeLong test (P < .05). Results The test set included 3937 women, aged 56.20 years ± 10.04. Hybrid DL and image-only DL showed AUCs of 0.70 (95% confidence interval [CI]: 0.66, 0.75) and 0.68 (95% CI: 0.64, 0.73), respectively. RF-LR and TC showed AUCs of 0.67 (95% CI: 0.62, 0.72) and 0.62 (95% CI: 0.57, 0.66), respectively. Hybrid DL showed a significantly higher AUC (0.70) than TC (0.62; P < .001) and RF-LR (0.67; P = .01). Conclusion Deep learning models that use full-field mammograms yield substantially improved risk discrimination compared with the Tyrer-Cuzick (version 8) model. © RSNA, 2019 Online supplemental material is available for this article. See also the editorial by Sitek and Wolfe in this issue.

Deep Learning Model to Assess Cancer Risk on the Basis of a Breast MR Image Alone.

OBJECTIVE. The purpose of this study is to develop an image-based deep learning (DL) model to predict the 5-year risk of breast cancer on the basis of a single breast MR image from a screening examination. MATERIALS AND METHODS. We collected 1656 consecutive breast MR images from screening examinations performed for 1183 high-risk women from January 2011 to June 2013, to predict the risk of cancer developing within 5 years of the screening. Women who lacked a 5-year screening follow-up examination and women who had cancer other than primary breast cancer develop in their breast were excluded from the study. We developed a logistic regression model based on traditional risk factors (the risk factor logistic regression [RF-LR] model) and a DL model based on the MR image alone (the Image-DL model). Examinations occurring within 6 months of a cancer diagnosis were excluded from the testing sets in each fold of cross-validation. We compared our models against the Tyrer-Cuzick (TC) model. All models were evaluated using mean (± SD) AUC values and observed-to-expected (OE) ratios across 10-fold cross-validation. RESULTS. The RF-LR and Image-DL models achieved mean AUC values of 0.558 ± 0.108 and 0.638 ± 0.094, respectively. In contrast, the TC model achieved an AUC value of 0.493 ± 0.092. The Image-DL and RF-LR models achieved mean OE ratios of 0.993 ± 0.658 and 0.828 ± 0.181, compared with the mean OE ratio of 1.091 ± 0.255 obtained using the TC model. CONCLUSION. Our DL model can assess the 5-year cancer risk on the basis of a breast MR image alone, and it showed improved individual risk discrimination when compared with a state-of-the-art risk assessment model. These results offer promising preliminary data regarding the potential of image-based risk assessment models to support more personalized care.

Atypical ductal hyperplasia in men with gynecomastia: what is their breast cancer risk?

PURPOSE: Atypical ductal hyperplasia (ADH) significantly increases the risk of breast cancer in women. However, little is known about the implications of ADH in men. METHODS: Review of 932 males with breast pathology was performed to identify cases of ADH. Patients were excluded if ADH was upgraded to cancer on excision, or if they had contralateral breast cancer. Cases were reviewed to determine whether any male with ADH developed breast cancer. RESULTS: Nineteen males were diagnosed with ADH from June 2003 to September 2018. All had gynecomastia. Surgical procedure was mastectomy in 8 patients and excision/reduction in 11. One patient had their nipple areola complex removed, and 1 required a free nipple graft. Median patient age at ADH diagnosis was 25 years (range 18-72 years). Of the 14 patients with bilateral gynecomastia, 10 had bilateral ADH and 4 had unilateral. Five cases of ADH were described as severe, bordering on ductal carcinoma in situ. No patient reported a family history of breast cancer. No patient took tamoxifen. At a mean follow-up of 75 months (range 4-185 months), no patient developed breast cancer. CONCLUSION: Our study is the first to provide follow-up information for males with ADH. With 6 years of mean follow-up, no male in our series has developed breast cancer. This suggests that either ADH in men does not pose the same risk as ADH in women or that surgical excision of symptomatic gynecomastia in men effectively reduces the risk of breast cancer.

Pathologic findings in reduction mammoplasty specimens: a surrogate for the population prevalence of breast cancer and high-risk lesions.

PURPOSE: Mammoplasty removes random samples of breast tissue from asymptomatic women providing a unique method for evaluating background prevalence of breast pathology in normal population. Our goal was to identify the rate of atypical breast lesions and cancers in women of various ages in the largest mammoplasty cohort reported to date. METHODS: We analyzed pathologic reports from patients undergoing bilateral mammoplasty, using natural language processing algorithm, verified by human review. Patients with a prior history of breast cancer or atypia were excluded. RESULTS: A total of 4775 patients were deemed eligible. Median age was 40 (range 13-86) and was higher in patients with any incidental finding compared to patients with normal reports (52 vs. 39 years, p = 0.0001). Pathological findings were detected in 7.06% (337) of procedures. Benign high-risk lesions were found in 299 patients (6.26%). Invasive carcinoma and ductal carcinoma in situ were detected in 15 (0.31%) and 23 (0.48%) patients, respectively. The rate of atypias and cancers increased with age. CONCLUSION: The overall rate of abnormal findings in asymptomatic patients undergoing mammoplasty was 7.06%, increasing with age. As these results are based on random sample of breast tissue, they likely underestimate the prevalence of abnormal findings in asymptomatic women.

Mammographic Breast Density Assessment Using Deep Learning: Clinical Implementation.

Purpose To develop a deep learning (DL) algorithm to assess mammographic breast density. Materials and Methods In this retrospective study, a deep convolutional neural network was trained to assess Breast Imaging Reporting and Data System (BI-RADS) breast density based on the original interpretation by an experienced radiologist of 41 479 digital screening mammograms obtained in 27 684 women from January 2009 to May 2011. The resulting algorithm was tested on a held-out test set of 8677 mammograms in 5741 women. In addition, five radiologists performed a reader study on 500 mammograms randomly selected from the test set. Finally, the algorithm was implemented in routine clinical practice, where eight radiologists reviewed 10 763 consecutive mammograms assessed with the model. Agreement on BI-RADS category for the DL model and for three sets of readings-(a) radiologists in the test set, (b) radiologists working in consensus in the reader study set, and (c) radiologists in the clinical implementation set-were estimated with linear-weighted κ statistics and were compared across 5000 bootstrap samples to assess significance. Results The DL model showed good agreement with radiologists in the test set (κ = 0.67; 95% confidence interval [CI]: 0.66, 0.68) and with radiologists in consensus in the reader study set (κ = 0.78; 95% CI: 0.73, 0.82). There was very good agreement (κ = 0.85; 95% CI: 0.84, 0.86) with radiologists in the clinical implementation set; for binary categorization of dense or nondense breasts, 10 149 of 10 763 (94%; 95% CI: 94%, 95%) DL assessments were accepted by the interpreting radiologist. Conclusion This DL model can be used to assess mammographic breast density at the level of an experienced mammographer. © RSNA, 2018 Online supplemental material is available for this article . See also the editorial by Chan and Helvie in this issue.

Machine learning to parse breast pathology reports in Chinese.

INTRODUCTION: Large structured databases of pathology findings are valuable in deriving new clinical insights. However, they are labor intensive to create and generally require manual annotation. There has been some work in the bioinformatics community to support automating this work via machine learning in English. Our contribution is to provide an automated approach to construct such structured databases in Chinese, and to set the stage for extraction from other languages. METHODS: We collected 2104 de-identified Chinese benign and malignant breast pathology reports from Hunan Cancer Hospital. Physicians with native Chinese proficiency reviewed the reports and annotated a variety of binary and numerical pathologic entities. After excluding 78 cases with a bilateral lesion in the same report, 1216 cases were used as a training set for the algorithm, which was then refined by 405 development cases. The Natural language processing algorithm was tested by using the remaining 405 cases to evaluate the machine learning outcome. The model was used to extract 13 binary entities and 8 numerical entities. RESULTS: When compared to physicians with native Chinese proficiency, the model showed a per-entity accuracy from 91 to 100% for all common diagnoses on the test set. The overall accuracy of binary entities was 98% and of numerical entities was 95%. In a per-report evaluation for binary entities with more than 100 training cases, 85% of all the testing reports were completely correct and 11% had an error in 1 out of 22 entities. CONCLUSION: We have demonstrated that Chinese breast pathology reports can be automatically parsed into structured data using standard machine learning approaches. The results of our study demonstrate that techniques effective in parsing English reports can be scaled to other languages.