Complementary and Integrative Health Information in the literature: its lexicon and named entity recognition.

OBJECTIVE: To construct an exhaustive Complementary and Integrative Health (CIH) Lexicon (CIHLex) to help better represent the often underrepresented physical and psychological CIH approaches in standard terminologies, and to also apply state-of-the-art natural language processing (NLP) techniques to help recognize them in the biomedical literature. MATERIALS AND METHODS: We constructed the CIHLex by integrating various resources, compiling and integrating data from biomedical literature and relevant sources of knowledge. The Lexicon encompasses 724 unique concepts with 885 corresponding unique terms. We matched these concepts to the Unified Medical Language System (UMLS), and we developed and utilized BERT models comparing their efficiency in CIH named entity recognition to well-established models including MetaMap and CLAMP, as well as the large language model GPT3.5-turbo. RESULTS: Of the 724 unique concepts in CIHLex, 27.2% could be matched to at least one term in the UMLS. About 74.9% of the mapped UMLS Concept Unique Identifiers were categorized as "Therapeutic or Preventive Procedure." Among the models applied to CIH named entity recognition, BLUEBERT delivered the highest macro-average F1-score of 0.91, surpassing other models. CONCLUSION: Our CIHLex significantly augments representation of CIH approaches in biomedical literature. Demonstrating the utility of advanced NLP models, BERT notably excelled in CIH entity recognition. These results highlight promising strategies for enhancing standardization and recognition of CIH terminology in biomedical contexts.

Ensuring fair, safe, and interpretable artificial intelligence-based prediction tools in a real-world oncological setting.

BACKGROUND: Cancer patients often experience treatment-related symptoms which, if uncontrolled, may require emergency department admission. We developed models identifying breast or genitourinary cancer patients at the risk of attending emergency department (ED) within 30-days and demonstrated the development, validation, and proactive approach to in-production monitoring of an artificial intelligence-based predictive model during a 3-month simulated deployment at a cancer hospital in the United States. METHODS: We used routinely-collected electronic health record data to develop our predictive models. We evaluated models including a variational autoencoder k-nearest neighbors algorithm (VAE-kNN) and model behaviors with a sample containing 84,138 observations from 28,369 patients. We assessed the model during a 77-day production period exposure to live data using a proactively monitoring process with predefined metrics. RESULTS: Performance of the VAE-kNN algorithm is exceptional (Area under the receiver-operating characteristics, AUC = 0.80) and remains stable across demographic and disease groups over the production period (AUC 0.74-0.82). We can detect issues in data feeds using our monitoring process to create immediate insights into future model performance. CONCLUSIONS: Our algorithm demonstrates exceptional performance at predicting risk of 30-day ED visits. We confirm that model outputs are equitable and stable over time using a proactive monitoring approach.

Patients with cancer often need to visit the hospital emergency department (ED), for example due to treatment side effects. Predicting these visits might help us to better manage the treatment of patients who are at risk. Here, we develop a computer-based tool to identify patients with cancer who are at risk of an unplanned ED visit within 30 days. We use health record data from over 28,000 patients who had visited a single cancer hospital in the US to create and test the model. The model performed well and was consistent across different demographic and disease groups. We monitor model behavior over time and show that it is stable. The approach we take to monitoring model performance may be a particularly useful contribution toward implementing similar predictive models in the clinic and checking that they are performing as intended.

Effectiveness of routine provision of feedback from patient-reported outcome measurements for cancer care improvement: a systematic review and meta-analysis.

BACKGROUND: Research shows that feeding back patient-reported outcome information to clinicians and/or patients could be associated with improved care processes and patient outcomes. Quantitative syntheses of intervention effects on oncology patient outcomes are lacking. OBJECTIVE: To determine the effects of patient-reported outcome measure (PROM) feedback intervention on oncology patient outcomes. DATA SOURCES: We identified relevant studies from 116 references included in our previous Cochrane review assessing the intervention for the general population. In May 2022, we conducted a systematic search in five bibliography databases using predefined keywords for additional studies published after the Cochrane review. STUDY SELECTION: We included randomized controlled trials evaluating the effects of PROM feedback intervention on processes and outcomes of care for oncology patients. DATA EXTRACTION AND SYNTHESIS: We used the meta-analytic approach to synthesize across studies measuring the same outcomes. We estimated pooled effects of the intervention on outcomes using Cohen's d for continuous data and risk ratio (RR) with a 95% confidence interval for dichotomous data. We used a descriptive approach to summarize studies which reported insufficient data for a meta-analysis. MAIN OUTCOME(S) AND MEASURES(S): Health-related quality of life (HRQL), symptoms, patient-healthcare provider communication, number of visits and hospitalizations, number of adverse events, and overall survival. RESULTS: We included 29 studies involving 7071 cancer participants. A small number of studies was available for each metanalysis (median = 3 studies, ranging from 2 to 9 studies) due to heterogeneity in the evaluation of the trials. We found that the intervention improved HRQL (Cohen's d = 0.23, 95% CI 0.11-0.34), mental functioning (Cohen's d = 0.14, 95% CI 0.02-0.26), patient-healthcare provider communication (Cohen's d = 0.41, 95% CI 0.20-0.62), and 1-year overall survival (OR = 0.64, 95% CI 0.48-0.86). The risk of bias across studies was considerable in the domains of allocation concealment, blinding, and intervention contamination. CONCLUSIONS AND RELEVANCE: Although we found evidence to support the intervention for highly relevant outcomes, our conclusions are tempered by the high risk of bias relating mainly to intervention design. PROM feedback for oncology patients may improve processes and outcomes for cancer patients but more high-quality evidence is required.

Machine Learning Models Using Routinely Collected Clinical Data Offer Robust and Interpretable Predictions of 90-Day Unplanned Acute Care Use for Cancer Immunotherapy Patients.

PURPOSE: Clinical management of patients receiving immune checkpoint inhibitors (ICIs) could be informed using accurate predictive tools to identify patients at risk of short-term acute care utilization (ACU). We used routinely collected data to develop and assess machine learning (ML) algorithms to predict unplanned ACU within 90 days of ICI treatment initiation. METHODS: We used aggregated electronic health record data from 7,960 patients receiving ICI treatments to train and assess eight ML algorithms. We developed the models using pre-SARS-COV-19 COVID-19 data generated between January 2016 and February 2020. We validated our algorithms using data collected between March 2020 and June 2022 (peri-COVID-19 sample). We assessed performance using area under the receiver operating characteristic curves (AUROC), sensitivity, specificity, and calibration plots. We derived intuitive explanations of predictions using variable importance and Shapley additive explanation analyses. We assessed the marginal performance of ML models compared with that of univariate and multivariate logistic regression (LR) models. RESULTS: Most algorithms significantly outperformed the univariate and multivariate LR models. The extreme gradient boosting trees (XGBT) algorithm demonstrated the best overall performance (AUROC, 0.70; sensitivity, 0.53; specificity, 0.74) on the peri-COVID-19 sample. The algorithm performance was stable across both pre- and peri-COVID-19 samples, as well as ICI regimen and cancer groups. Type of ICI agents, oxygen saturation, diastolic blood pressure, albumin level, platelet count, immature granulocytes, absolute monocyte, chloride level, red cell distribution width, and alcohol intake were the top 10 key predictors used by the XGBT algorithm. CONCLUSION: Machine learning algorithms trained using routinely collected data outperformed traditional statistical models when predicting 90-day ACU. The XGBT algorithm has the potential to identify high-ACU risk patients and enable preventive interventions to avoid ACU.

On the importance of interpretable machine learning predictions to inform clinical decision making in oncology.

Machine learning-based tools are capable of guiding individualized clinical management and decision-making by providing predictions of a patient's future health state. Through their ability to model complex nonlinear relationships, ML algorithms can often outperform traditional statistical prediction approaches, but the use of nonlinear functions can mean that ML techniques may also be less interpretable than traditional statistical methodologies. While there are benefits of intrinsic interpretability, many model-agnostic approaches now exist and can provide insight into the way in which ML systems make decisions. In this paper, we describe how different algorithms can be interpreted and introduce some techniques for interpreting complex nonlinear algorithms.

Documentation of Complementary and Integrative Health Therapies in the Electronic Health Record: A Scoping Review.

Introduction: Complementary and integrative health (CIH) therapies refers to massage therapy, acupuncture, aromatherapy, and guided imagery. These therapies have gained increased attention in recent years, particularly for their potential to help manage chronic pain and other conditions. National organizations not only recommend the use of CIH therapies but also the documentation of these therapies within electronic health records (EHRs). Yet, how CIH therapies are documented in the EHR is not well understood. The purpose of this scoping review of the literature was to examine and describe research that focused on CIH therapy clinical documentation in the EHR. Methods: The authors conducted a literature search using six electronic databases: Cumulative Index to Nursing and Allied Health Literature (CINAHL), Ovid MEDLINE, Scopus, Google Scholar, Embase, and PubMed. Predefined search terms included "informatics," "documentation," "complementary and integrative health therapies," "non-pharmacological approaches," and "electronic health records" using AND/OR statements. No restrictions were placed on publication date. The inclusion criteria were as follows: (1) Original peer-reviewed full article in English, (2) focus on CIH therapies, and (3) CIH therapy documentation practice used in the research. Results: The authors identified 1684 articles, of which 33 met the criteria for a full review. A majority of the studies were conducted in the United States (20) and hospitals (19). The most common study design was retrospective (9), and 26 studies used EHR data as a data source for analysis. Documentation practices varied widely across all studies, ranging from the feasibility of documenting integrative therapies (i.e., homeopathy) to create changes in the EHR to support documentation (i.e., flowsheet). Discussion: This scoping review identified varying EHR clinical documentation trends for CIH therapies. Pain was the most frequent reason for use of CIH therapies across all included studies and a broad range of CIH therapies were used. Data standards and templates were suggested as informatics methods to support CIH documentation. A systems approach is needed to enhance and support the current technology infrastructure that will enable consistent CIH therapy documentation in EHRs.

The Use of Machine Learning for Predicting Complications of Free-Flap Head and Neck Reconstruction.

BACKGROUND: Machine learning has been increasingly used for surgical outcome prediction, yet applications in head and neck reconstruction are not well-described. In this study, we developed and evaluated the performance of ML algorithms in predicting postoperative complications in head and neck free-flap reconstruction. METHODS: We conducted a comprehensive review of patients who underwent microvascular head and neck reconstruction between January 2005 and December 2018. Data were used to develop and evaluate nine supervised ML algorithms in predicting overall complications, major recipient-site complication, and total flap loss. RESULTS: We identified 4000 patients who met inclusion criteria. Overall, 33.7% of patients experienced a complication, 26.5% experienced a major recipient-site complication, and 1.7% suffered total flap loss. The k-nearest neighbors algorithm demonstrated the best overall performance for predicting any complication (AUROC = 0.61, sensitivity = 0.60). Regularized regression had the best performance for predicting major recipient-site complications (AUROC = 0.68, sensitivity = 0.66), and decision trees were the best predictors of total flap loss (AUROC = 0.66, sensitivity = 0.50). CONCLUSIONS: ML accurately identified patients at risk of experiencing postsurgical complications, including total flap loss. Predictions from ML models may provide insight in the perioperative setting and facilitate shared decision making.

Machine learning models for 180-day mortality prediction of patients with advanced cancer using patient-reported symptom data.

PURPOSE: The objective of the current study was to develop and test the performances of different ML algorithms which were trained using patient-reported symptom severity data to predict mortality within 180 days for patients with advanced cancer. METHODS: We randomly selected 630 of 689 patients with advanced cancer at our institution who completed symptom PRO measures as part of routine care between 2009 and 2020. Using clinical, demographic, and PRO data, we trained and tested four ML algorithms: generalized regression with elastic net regularization (GLM), extreme gradient boosting (XGBoost) trees, support vector machines (SVM), and a single hidden layer neural network (NNET). We assessed the performance of algorithms individually as well as part of an unweighted voting ensemble on the hold-out testing sample. Performance was assessed using area under the receiver-operating characteristic curve (AUROC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). RESULTS: The starting cohort of 630 patients was randomly partitioned into training (n = 504) and testing (n = 126) samples. Of the four ML models, the XGBoost algorithm demonstrated the best performance for 180-day mortality prediction in testing data (AUROC = 0.69, sensitivity = 0.68, specificity = 0.62, PPV = 0.66, NPV = 0.64). Ensemble of all algorithms performed worst (AUROC = 0.65, sensitivity = 0.65, specificity = 0.62, PPV = 0.65, NPV = 0.62). Of individual PRO symptoms, shortness of breath emerged as the variable of highest impact on the XGBoost 180-mortality prediction (1-AUROC = 0.30). CONCLUSION: Our findings support ML models driven by patient-reported symptom severity as accurate predictors of short-term mortality in patients with advanced cancer, highlighting the opportunity to integrate these models prospectively into future studies of goal-concordant care.

Predicting 180-day mortality for women with ovarian cancer using machine learning and patient-reported outcome data.

Contrary to national guidelines, women with ovarian cancer often receive treatment at the end of life, potentially due to the difficulty in accurately estimating prognosis. We trained machine learning algorithms to guide prognosis by predicting 180-day mortality for women with ovarian cancer using patient-reported outcomes (PRO) data. We collected data from a single academic cancer institution in the United States. Women completed biopsychosocial PRO measures every 90 days. We randomly partitioned our dataset into training and testing samples. We used synthetic minority oversampling to reduce class imbalance in the training dataset. We fitted training data to six machine learning algorithms and combined their classifications on the testing dataset into an unweighted voting ensemble. We assessed each algorithm's accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC) using testing data. We recruited 245 patients who completed 1319 PRO assessments. The final voting ensemble produced state-of-the-art results on the task of predicting 180-day mortality for ovarian cancer paitents (Accuracy = 0.79, Sensitivity = 0.71, Specificity = 0.80, AUROC = 0.76). The algorithm correctly identified 25 of the 35 women in the testing dataset who died within 180 days of assessment. Machine learning algorithms trained using PRO data offer encouraging performance in predicting whether a woman with ovarian cancer will die within 180 days. This model could be used to drive data-driven end-of-life care and address current shortcomings in care delivery. Our model demonstrates the potential of biopsychosocial PROM information to make substantial contributions to oncology prediction modeling. This model could inform clinical decision-making Future research is needed to validate these findings in a larger, more diverse sample.

Machine learning in medicine: a practical introduction to techniques for data pre-processing, hyperparameter tuning, and model comparison.

BACKGROUND: There is growing enthusiasm for the application of machine learning (ML) and artificial intelligence (AI) techniques to clinical research and practice. However, instructions on how to develop robust high-quality ML and AI in medicine are scarce. In this paper, we provide a practical example of techniques that facilitate the development of high-quality ML systems including data pre-processing, hyperparameter tuning, and model comparison using open-source software and data. METHODS: We used open-source software and a publicly available dataset to train and validate multiple ML models to classify breast masses into benign or malignant using mammography image features and patient age. We compared algorithm predictions to the ground truth of histopathologic evaluation. We provide step-by-step instructions with accompanying code lines. FINDINGS: Performance of the five algorithms at classifying breast masses as benign or malignant based on mammography image features and patient age was statistically equivalent (P > 0.05). Area under the receiver operating characteristics curve (AUROC) for the logistic regression with elastic net penalty was 0.89 (95% CI 0.85 - 0.94), for the Extreme Gradient Boosting Tree 0.88 (95% CI 0.83 - 0.93), for the Multivariate Adaptive Regression Spline algorithm 0.88 (95% CI 0.83 - 0.93), for the Support Vector Machine 0.89 (95% CI 0.84 - 0.93), and for the neural network 0.89 (95% CI 0.84 - 0.93). INTERPRETATION: Our paper allows clinicians and medical researchers who are interested in using ML algorithms to understand and recreate the elements of a comprehensive ML analysis. Following our instructions may help to improve model generalizability and reproducibility in medical ML studies.

Intelligent multi-modal shear wave elastography to reduce unnecessary biopsies in breast cancer diagnosis (INSPiRED 002): a retrospective, international, multicentre analysis.

BACKGROUND: Breast ultrasound identifies additional carcinomas not detected in mammography but has a higher rate of false-positive findings. We evaluated whether use of intelligent multi-modal shear wave elastography (SWE) can reduce the number of unnecessary biopsies without impairing the breast cancer detection rate. METHODS: We trained, tested, and validated machine learning algorithms using SWE, clinical, and patient information to classify breast masses. We used data from 857 women who underwent B-mode breast ultrasound, SWE, and subsequent histopathologic evaluation at 12 study sites in seven countries from 2016 to 2019. Algorithms were trained and tested on data from 11 of the 12 sites and externally validated using the additional site's data. We compared findings to the histopathologic evaluation and compared the diagnostic performance between B-mode breast ultrasound, traditional SWE, and intelligent multi-modal SWE. RESULTS: In the external validation set (n = 285), intelligent multi-modal SWE showed a sensitivity of 100% (95% CI, 97.1-100%, 126 of 126), a specificity of 50.3% (95% CI, 42.3-58.3%, 80 of 159), and an area under the curve of 0.93 (95% CI, 0.90-0.96). Diagnostic performance was significantly higher compared to traditional SWE and B-mode breast ultrasound (P < 0.001). Unlike traditional SWE, positive-predictive values of intelligent multi-modal SWE were significantly higher compared to B-mode breast ultrasound. Unnecessary biopsies were reduced by 50.3% (79 versus 159, P < 0.001) without missing cancer compared to B-mode ultrasound. CONCLUSION: The majority of unnecessary breast biopsies might be safely avoided by using intelligent multi-modal SWE. These results may be helpful to reduce diagnostic burden for patients, providers, and healthcare systems.

Effectiveness and Minimum Effective Dose of App-Based Mobile Health Interventions for Anxiety and Depression Symptom Reduction: Systematic Review and Meta-Analysis.

BACKGROUND: Mobile health (mHealth) apps offer new opportunities to deliver psychological treatments for mental illness in an accessible, private format. The results of several previous systematic reviews support the use of app-based mHealth interventions for anxiety and depression symptom management. However, it remains unclear how much or how long the minimum treatment "dose" is for an mHealth intervention to be effective. Just-in-time adaptive intervention (JITAI) has been introduced in the mHealth domain to facilitate behavior changes and is positioned to guide the design of mHealth interventions with enhanced adherence and effectiveness. OBJECTIVE: Inspired by the JITAI framework, we conducted a systematic review and meta-analysis to evaluate the dose effectiveness of app-based mHealth interventions for anxiety and depression symptom reduction. METHODS: We conducted a literature search on 7 databases (ie, Ovid MEDLINE, Embase, PsycInfo, Scopus, Cochrane Library (eg, CENTRAL), ScienceDirect, and ClinicalTrials, for publications from January 2012 to April 2020. We included randomized controlled trials (RCTs) evaluating app-based mHealth interventions for anxiety and depression. The study selection and data extraction process followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We estimated the pooled effect size using Hedge g and appraised study quality using the revised Cochrane risk-of-bias tool for RCTs. RESULTS: We included 15 studies involving 2627 participants for 18 app-based mHealth interventions. Participants in the intervention groups showed a significant effect on anxiety (Hedge g=-.10, 95% CI -0.14 to -0.06, I2=0%) but not on depression (Hedge g=-.08, 95% CI -0.23 to 0.07, I2=4%). Interventions of at least 7 weeks' duration had larger effect sizes on anxiety symptom reduction. CONCLUSIONS: There is inconclusive evidence for clinical use of app-based mHealth interventions for anxiety and depression at the current stage due to the small to nonsignificant effects of the interventions and study quality concerns. The recommended dose of mHealth interventions and the sustainability of intervention effectiveness remain unclear and require further investigation.

Public Health Nurse Tailored Home Visiting and Parenting Behavior for Families at Risk for Referral to Child Welfare Services, Colorado: 2018-2019.

Objectives. To examine public health nurse (PHN) intervention tailoring through the Colorado Nurse Support Program (NSP). Our 2 specific aims were to describe the NSP program and its outcomes and to determine the effects of modifying interventions on short- and long-term outcomes among NSP clients. Methods. In our retrospective causal investigation of 150 families in Colorado in 2018-2019, intervention effects were modeled via longitudinal modified treatment policy analyses. Results. Families served by PHNs improved in terms of knowledge, behavior, and status outcomes after receiving multidimensional, tailored home visiting interventions. Case management interventions provided in the first month of PHN home visits had lasting effects on behavior outcomes, and 2 additional case management interventions in the first month were estimated to have even more of an impact. Conclusions. Modern causal inference methods and real-world PHN data revealed a nuanced, fine-grained understanding of the real impact of tailored PHN interventions. Public Health Implications PHN programs such as the NSP and use of the Omaha System should be supported and extended to advance evaluations of intervention effectiveness and knowledge discovery and improve population health. (Am J Public Health. 2022;112(S3):S306-S313. https://doi.org/10.2105/AJPH.2022.306792).

Novel Machine Learning Approach for the Prediction of Hernia Recurrence, Surgical Complication, and 30-Day Readmission after Abdominal Wall Reconstruction.

BACKGROUND: Despite advancements in abdominal wall reconstruction (AWR) techniques, hernia recurrences (HRs), surgical site occurrences (SSOs), and unplanned hospital readmissions persist. We sought to develop, validate, and evaluate machine learning (ML) algorithms for predicting complications after AWR. METHODS: We conducted a comprehensive review of patients who underwent AWR from March 2005 to June 2019. Nine supervised ML algorithms were developed to preoperatively predict HR, SSOs, and 30-day readmission. Patient data were partitioned into training (80%) and testing (20%) sets. RESULTS: We identified 725 patients (52% women), with a mean age of 60 ± 11.5 years, mean body mass index of 31 ± 7 kg/m2, and mean follow-up time of 42 ± 29 months. The HR rate was 12.8%, SSO rate was 30%, and 30-day readmission rate was 10.9%. ML models demonstrated good discriminatory performance for predicting HR (area under the receiver operating characteristic curve [AUC] 0.71), SSOs (AUC 0.75), and 30-day readmission (AUC 0.74). ML models achieved mean accuracy rates of 85% (95% CI 80% to 90%), 72% (95% CI 64% to 80%), and 84% (95% CI 77% to 90%) for predicting HR, SSOs, and 30-day readmission, respectively. ML identified and characterized 4 unique significant predictors of HR, 12 of SSOs, and 3 of 30-day readmission. Decision curve analysis demonstrated that ML models have a superior net benefit regardless of the probability threshold. CONCLUSIONS: ML algorithms trained on readily available preoperative clinical data accurately predicted complications of AWR. Our findings support incorporating ML models into the preoperative assessment of patients undergoing AWR to provide data-driven, patient-specific risk assessment.

Frailty and Social and Behavioral Determinants of Health: Algorithm Refinement and Pattern Validation.

Existing frailty and social and behavioral determinants of health (SBDH) algorithms were refined and used to examine SBDH and frailty groups, revealing patterns in interventions and outcomes of older adults in a large community-based care data-set. The dataset was randomly split into training (n = 2,881) and testing (n = 1,441) sets. The training set was used to visually identify patterns in associations among SBDH, frailty, intervention doses, and outcomes, and the testing set was used to validate the patterns. Seven valid patterns were identified, showing increases in SBDH and frailty were associated with poorer health outcomes and more interventions (all p < 0.01). Findings suggest that the refined SBDH and frailty algorithms facilitate the identification of older adults with SBDH and frailty for intervention tailoring. [Journal of Gerontological Nursing, 48(4), 41-48.].

Machine Learning-Based Short-Term Mortality Prediction Models for Patients With Cancer Using Electronic Health Record Data: Systematic Review and Critical Appraisal.

BACKGROUND: In the United States, national guidelines suggest that aggressive cancer care should be avoided in the final months of life. However, guideline compliance currently requires clinicians to make judgments based on their experience as to when a patient is nearing the end of their life. Machine learning (ML) algorithms may facilitate improved end-of-life care provision for patients with cancer by identifying patients at risk of short-term mortality. OBJECTIVE: This study aims to summarize the evidence for applying ML in ≤1-year cancer mortality prediction to assist with the transition to end-of-life care for patients with cancer. METHODS: We searched MEDLINE, Embase, Scopus, Web of Science, and IEEE to identify relevant articles. We included studies describing ML algorithms predicting ≤1-year mortality in patients of oncology. We used the prediction model risk of bias assessment tool to assess the quality of the included studies. RESULTS: We included 15 articles involving 110,058 patients in the final synthesis. Of the 15 studies, 12 (80%) had a high or unclear risk of bias. The model performance was good: the area under the receiver operating characteristic curve ranged from 0.72 to 0.92. We identified common issues leading to biased models, including using a single performance metric, incomplete reporting of or inappropriate modeling practice, and small sample size. CONCLUSIONS: We found encouraging signs of ML performance in predicting short-term cancer mortality. Nevertheless, no included ML algorithms are suitable for clinical practice at the current stage because of the high risk of bias and uncertainty regarding real-world performance. Further research is needed to develop ML models using the modern standards of algorithm development and reporting.

The importance of multi-modal imaging and clinical information for humans and AI-based algorithms to classify breast masses (INSPiRED 003): an international, multicenter analysis.

OBJECTIVES: AI-based algorithms for medical image analysis showed comparable performance to human image readers. However, in practice, diagnoses are made using multiple imaging modalities alongside other data sources. We determined the importance of this multi-modal information and compared the diagnostic performance of routine breast cancer diagnosis to breast ultrasound interpretations by humans or AI-based algorithms. METHODS: Patients were recruited as part of a multicenter trial (NCT02638935). The trial enrolled 1288 women undergoing routine breast cancer diagnosis (multi-modal imaging, demographic, and clinical information). Three physicians specialized in ultrasound diagnosis performed a second read of all ultrasound images. We used data from 11 of 12 study sites to develop two machine learning (ML) algorithms using unimodal information (ultrasound features generated by the ultrasound experts) to classify breast masses which were validated on the remaining study site. The same ML algorithms were subsequently developed and validated on multi-modal information (clinical and demographic information plus ultrasound features). We assessed performance using area under the curve (AUC). RESULTS: Of 1288 breast masses, 368 (28.6%) were histopathologically malignant. In the external validation set (n = 373), the performance of the two unimodal ultrasound ML algorithms (AUC 0.83 and 0.82) was commensurate with performance of the human ultrasound experts (AUC 0.82 to 0.84; p for all comparisons > 0.05). The multi-modal ultrasound ML algorithms performed significantly better (AUC 0.90 and 0.89) but were statistically inferior to routine breast cancer diagnosis (AUC 0.95, p for all comparisons ≤ 0.05). CONCLUSIONS: The performance of humans and AI-based algorithms improves with multi-modal information. KEY POINTS: • The performance of humans and AI-based algorithms improves with multi-modal information. • Multimodal AI-based algorithms do not necessarily outperform expert humans. • Unimodal AI-based algorithms do not represent optimal performance to classify breast masses.

Contrast of Digital and Health Literacy Between IT and Health Care Specialists Highlights the Importance of Multidisciplinary Teams for Digital Health-A Pilot Study.

PURPOSE: Despite their promises, digital innovations have scarcely translated to technologies used in routine clinical practice, making the identification of barriers to successful implementation a research priority. Low levels of transdisciplinary skills represent such a barrier but so far, this has not been evaluated and compared between information technology (IT) and health care specialists. In this study, we evaluated the level of digital health literacy among IT and health care specialists. MATERIALS AND METHODS: An anonymous questionnaire was distributed to staff at a breast cancer unit and an IT department of two German universities in December 2020. The survey questionnaire consisted of the previously validated eHealth Literacy Assessment Toolkit and additional questions with respect to age, profession, and career stage. Mann-Whitney or Wilcoxon rank-sum tests and two-sample chi-square tests were used for the analysis. RESULTS: The survey was completed by 113 individuals: 70 (61.9%) IT specialists and 43 (38.1%) health care specialists. Health care specialists scored significantly higher on the health-related scales and IT specialists scored significantly higher on the digitally related scales. No single participant identified themselves to have the highest level of literacy on all survey questions (n = 0 of 113; 0%). Only one person (n = 1 of 113; 0.9%) consistently reported a high or the highest level of literacy. CONCLUSION: Although IT and health care specialists showed great literacy in their respective disciplines, only few individuals combined both digital and health care literacy. Multidisciplinary teams and transdisciplinary curricula are crucial to bridge skill gaps between disciplines and to drive the implementation of digital health initiatives.