OpenKBP-Opt: an international and reproducible evaluation of 76 knowledge-based planning pipelines.

Objective.To establish an open framework for developing plan optimization models for knowledge-based planning (KBP).Approach.Our framework includes radiotherapy treatment data (i.e. reference plans) for 100 patients with head-and-neck cancer who were treated with intensity-modulated radiotherapy. That data also includes high-quality dose predictions from 19 KBP models that were developed by different research groups using out-of-sample data during the OpenKBP Grand Challenge. The dose predictions were input to four fluence-based dose mimicking models to form 76 unique KBP pipelines that generated 7600 plans (76 pipelines × 100 patients). The predictions and KBP-generated plans were compared to the reference plans via: the dose score, which is the average mean absolute voxel-by-voxel difference in dose; the deviation in dose-volume histogram (DVH) points; and the frequency of clinical planning criteria satisfaction. We also performed a theoretical investigation to justify our dose mimicking models.Main results.The range in rank order correlation of the dose score between predictions and their KBP pipelines was 0.50-0.62, which indicates that the quality of the predictions was generally positively correlated with the quality of the plans. Additionally, compared to the input predictions, the KBP-generated plans performed significantly better (P< 0.05; one-sided Wilcoxon test) on 18 of 23 DVH points. Similarly, each optimization model generated plans that satisfied a higher percentage of criteria than the reference plans, which satisfied 3.5% more criteria than the set of all dose predictions. Lastly, our theoretical investigation demonstrated that the dose mimicking models generated plans that are also optimal for an inverse planning model.Significance.This was the largest international effort to date for evaluating the combination of KBP prediction and optimization models. We found that the best performing models significantly outperformed the reference dose and dose predictions. In the interest of reproducibility, our data and code is freely available.

OpenKBP: The open-access knowledge-based planning grand challenge and dataset.

PURPOSE: To advance fair and consistent comparisons of dose prediction methods for knowledge-based planning (KBP) in radiation therapy research. METHODS: We hosted OpenKBP, a 2020 AAPM Grand Challenge, and challenged participants to develop the best method for predicting the dose of contoured computed tomography (CT) images. The models were evaluated according to two separate scores: (a) dose score, which evaluates the full three-dimensional (3D) dose distributions, and (b) dose-volume histogram (DVH) score, which evaluates a set DVH metrics. We used these scores to quantify the quality of the models based on their out-of-sample predictions. To develop and test their models, participants were given the data of 340 patients who were treated for head-and-neck cancer with radiation therapy. The data were partitioned into training ( n = 200 ), validation ( n = 40 ), and testing ( n = 100 ) datasets. All participants performed training and validation with the corresponding datasets during the first (validation) phase of the Challenge. In the second (testing) phase, the participants used their model on the testing data to quantify the out-of-sample performance, which was hidden from participants and used to determine the final competition ranking. Participants also responded to a survey to summarize their models. RESULTS: The Challenge attracted 195 participants from 28 countries, and 73 of those participants formed 44 teams in the validation phase, which received a total of 1750 submissions. The testing phase garnered submissions from 28 of those teams, which represents 28 unique prediction methods. On average, over the course of the validation phase, participants improved the dose and DVH scores of their models by a factor of 2.7 and 5.7, respectively. In the testing phase one model achieved the best dose score (2.429) and DVH score (1.478), which were both significantly better than the dose score (2.564) and the DVH score (1.529) that was achieved by the runner-up models. Lastly, many of the top performing teams reported that they used generalizable techniques (e.g., ensembles) to achieve higher performance than their competition. CONCLUSION: OpenKBP is the first competition for knowledge-based planning research. The Challenge helped launch the first platform that enables researchers to compare KBP prediction methods fairly and consistently using a large open-source dataset and standardized metrics. OpenKBP has also democratized KBP research by making it accessible to everyone, which should help accelerate the progress of KBP research. The OpenKBP datasets are available publicly to help benchmark future KBP research.

Predicting Protein and Fat Content in Human Donor Milk Using Machine Learning.

BACKGROUND: Donor milk is the standard of care for hospitalized very low birth weight (VLBW) infants when mother's milk is unavailable; however, growth of donor milk-fed infants is frequently suboptimal. Variability in nutrient composition of donated milk complicates the production of a uniform pooled product and, subsequently, the provision of adequate nutrition to promote optimal growth and development of VLBW infants. We reasoned a machine learning approach to construct batches using characteristics of the milk donation might be an effective strategy in reducing the variability in donor milk product composition. OBJECTIVE: The objective of this study was to identify whether machine learning models can accurately predict donor milk macronutrient content. We focused on predicting fat and protein, given their well-established importance in VLBW infant growth outcomes. METHODS: Samples of donor milk, consisting of 272 individual donations and 61 pool samples, were collected from the Rogers Hixon Ontario Human Milk Bank and analyzed for macronutrient content. Four different machine learning models were constructed using independent variable groups associated with donations, donors, and donor-pumping practices. A baseline model was established using lactation stage and infant gestational status. Predictions were made for individual donations and resultant pools. RESULTS: Machine learning models predicted protein of individual donations and pools with a mean absolute error (MAE) of 0.16 g/dL and 0.10 g/dL, respectively. Individual donation and pooled fat predictions had an MAE of 0.91 g/dL and 0.42 g/dL, respectively. At both the individual donation and pool levels, protein predictions were significantly more accurate than baseline, whereas fat predictions were competitive with baseline. CONCLUSIONS: Machine learning models can provide accurate predictions of macronutrient content in donor milk. The macronutrient content of pooled milk had a lower prediction error, reinforcing the value of pooling practices. Future research should examine how macronutrient content predictions can be used to facilitate milk bank pooling strategies.

AutoAudio: Deep Learning for Automatic Audiogram Interpretation.

Hearing loss is the leading human sensory system loss, and one of the leading causes for years lived with disability with significant effects on quality of life, social isolation, and overall health. Coupled with a forecast of increased hearing loss burden worldwide, national and international health organizations have urgently recommended that access to hearing evaluation be expanded to meet demand. The objective of this study was to develop 'AutoAudio' - a novel deep learning proof-of-concept model that accurately and quickly interprets diagnostic audiograms. Adult audiogram reports representing normal, conductive, mixed and sensorineural morphologies were used to train different neural network architectures. Image augmentation techniques were used to increase the training image set size. Classification accuracy on a separate test set was used to assess model performance. The architecture with the highest out-of-training set accuracy was ResNet-101 at 97.5%. Neural network training time varied between 2 to 7 h depending on the depth of the neural network architecture. Each neural network architecture produced misclassifications that arose from failures of the model to correctly label the audiogram with the appropriate hearing loss type. The most commonly misclassified hearing loss type were mixed losses. Re-engineering the process of hearing testing with a machine learning innovation may help enhance access to the growing worldwide population that is expected to require audiologist services. Our results suggest that deep learning may be a transformative technology that enables automatic and accurate audiogram interpretation.

Predicting Postoperative Cochlear Implant Performance Using Supervised Machine Learning.

OBJECTIVES: To predict postoperative cochlear implant performance with heterogeneous text and numerical variables using supervised machine learning techniques. STUDY DESIGN: A supervised machine learning approach comprising neural networks and decision tree-based ensemble algorithms were used to predict 1-year postoperative cochlear implant performance based on retrospective data. SETTING: Tertiary referral center. PATIENTS: One thousand six hundred four adults who received one cochlear implant from 1989 to 2019. Two hundred eighty two text and numerical objective demographic, audiometric, and patient-reported outcome survey instrument variables were included. OUTCOME MEASURES: Outcomes for postoperative cochlear implant performance were discrete Hearing in Noise Test (HINT; %) performance and binned HINT performance classification ("High," "Mid," and "Low" performers). Algorithm performance was assessed using hold-out validation datasets and were compared using root mean square error (RMSE) in the units of the target variable and classification accuracy. RESULTS: The neural network 1-year HINT prediction RMSE and classification accuracy were 0.57 and 95.4%, respectively, with only numerical variable inputs. Using both text and numerical variables, neural networks predicted postoperative HINT with a RMSE of 25.0%, and classification accuracy of 73.3%. When applied to numerical variables only, the XGBoost algorithm produced a 1-year HINT score prediction performance RMSE of 25.3%. We identified over 20 influential variables including preoperative sentence-test performance, age at surgery, as well as specific tinnitus handicap inventory (THI), Short Form 36 (SF-36), and health utilities index (HUI) question responses as the highest influencers of postoperative HINT. CONCLUSION: Our results suggest that supervised machine learning can predict postoperative cochlear implant performance and identify preoperative factors that significantly influence that performance. These algorithms can help improve the understanding of the diverse factors that impact functional performance from heterogeneous data sources.

The importance of evaluating the complete automated knowledge-based planning pipeline.

We determine how prediction methods combine with optimization methods in two-stage knowledge-based planning (KBP) pipelines to produce radiation therapy treatment plans. We trained two dose prediction methods, a generative adversarial network (GAN) and a random forest (RF) with the same 130 treatment plans. The models were applied to 87 out-of-sample patients to create two sets of predicted dose distributions that were used as input to two optimization models. The first optimization model, inverse planning (IP), estimates weights for dose-objectives from a predicted dose distribution and generates new plans using conventional inverse planning. The second optimization model, dose mimicking (DM), minimizes the sum of one-sided quadratic penalties between the predictions and the generated plans using several dose-objectives. Altogether, four KBP pipelines (GAN-IP, GAN-DM, RF-IP, and RF-DM) were constructed and benchmarked against the corresponding clinical plans using clinical criteria; the error of both prediction methods was also evaluated. The best performing plans were GAN-IP plans, which satisfied the same criteria as their corresponding clinical plans (78%) more often than any other KBP pipeline. However, GAN did not necessarily provide the best prediction for the second-stage optimization models. Specifically, both the RF-IP and RF-DM plans satisfied the same criteria as the clinical plans 25% and 15% more often than GAN-DM plans (the worst performing plans), respectively. GAN predictions also had a higher mean absolute error (3.9 Gy) than those from RF (3.6 Gy). We find that state-of-the-art prediction methods when paired with different optimization algorithms, produce treatment plans with considerable variation in quality.

Knowledge-based automated planning with three-dimensional generative adversarial networks.

PURPOSE: To develop a knowledge-based automated planning pipeline that generates treatment plans without feature engineering, using deep neural network architectures for predicting three-dimensional (3D) dose. METHODS: Our knowledge-based automated planning (KBAP) pipeline consisted of a knowledge-based planning (KBP) method that predicts dose for a contoured computed tomography (CT) image followed by two optimization models that learn objective function weights and generate fluence-based plans, respectively. We developed a novel generative adversarial network (GAN)-based KBP approach, a 3D GAN model, which predicts dose for the full 3D CT image at once and accounts for correlations between adjacent CT slices. Baseline comparisons were made against two state-of-the-art deep learning-based KBP methods from the literature. We also developed an additional benchmark, a two-dimensional (2D) GAN model which predicts dose to each axial slice independently. For all models, we investigated the impact of multiplicatively scaling the predictions before optimization, such that the predicted dose distributions achieved all target clinical criteria. Each KBP model was trained on 130 previously delivered oropharyngeal treatment plans. Performance was tested on 87 out-of-sample previously delivered treatment plans. All KBAP plans were evaluated using clinical planning criteria and compared to their corresponding clinical plans. KBP prediction quality was assessed using dose-volume histogram (DVH) differences from the corresponding clinical plans. RESULTS: The best performing KBAP plans were generated using predictions from the 3D GAN model that were multiplicatively scaled. These plans satisfied 77% of all clinical criteria, compared to the clinical plans, which satisfied 67% of all criteria. In general, multiplicatively scaling predictions prior to optimization increased the fraction of clinical criteria satisfaction by 11% relative to the plans generated with nonscaled predictions. Additionally, these KBAP plans satisfied the same criteria as the clinical plans 84% and 8% more frequently as compared to the two benchmark methods, respectively. CONCLUSIONS: We developed the first knowledge-based automated planning framework using a 3D generative adversarial network for prediction. Our results, based on 217 oropharyngeal cancer treatment plans, demonstrated superior performance in satisfying clinical criteria and generated more realistic plans as compared to the previous state-of-the-art approaches.