A Computed Tomography-Based Fracture Prediction Model With Images of Vertebral Bones and Muscles by Employing Deep Learning: Development and Validation Study.

BACKGROUND: With the progressive increase in aging populations, the use of opportunistic computed tomography (CT) scanning is increasing, which could be a valuable method for acquiring information on both muscles and bones of aging populations. OBJECTIVE: The aim of this study was to develop and externally validate opportunistic CT-based fracture prediction models by using images of vertebral bones and paravertebral muscles. METHODS: The models were developed based on a retrospective longitudinal cohort study of 1214 patients with abdominal CT images between 2010 and 2019. The models were externally validated in 495 patients. The primary outcome of this study was defined as the predictive accuracy for identifying vertebral fracture events within a 5-year follow-up. The image models were developed using an attention convolutional neural network-recurrent neural network model from images of the vertebral bone and paravertebral muscles. RESULTS: The mean ages of the patients in the development and validation sets were 73 years and 68 years, and 69.1% (839/1214) and 78.8% (390/495) of them were females, respectively. The areas under the receiver operator curve (AUROCs) for predicting vertebral fractures were superior in images of the vertebral bone and paravertebral muscles than those in the bone-only images in the external validation cohort (0.827, 95% CI 0.821-0.833 vs 0.815, 95% CI 0.806-0.824, respectively; P<.001). The AUROCs of these image models were higher than those of the fracture risk assessment models (0.810 for major osteoporotic risk, 0.780 for hip fracture risk). For the clinical model using age, sex, BMI, use of steroids, smoking, possible secondary osteoporosis, type 2 diabetes mellitus, HIV, hepatitis C, and renal failure, the AUROC value in the external validation cohort was 0.749 (95% CI 0.736-0.762), which was lower than that of the image model using vertebral bones and muscles (P<.001). CONCLUSIONS: The model using the images of the vertebral bone and paravertebral muscle showed better performance than that using the images of the bone-only or clinical variables. Opportunistic CT screening may contribute to identifying patients with a high fracture risk in the future.

Innovative AI methods for monitoring front-of-package information: A case study on infant foods.

Front-of-package (FOP) is one of the most direct communication channels connecting manufacturers and consumers, as it displays crucial information such as certification, nutrition, and health. Traditional methods for obtaining information from FOPs often involved manual collection and analysis. To overcome these labor-intensive characteristics, new methods using two artificial intelligence (AI) approaches were applied for information monitoring of FOPs. In order to provide practical implementations, a case study was conducted on infant food products. First, FOP images were collected from Amazon.com. Then, from the FOP images, 1) the certification usage status of the infant food group was obtained by recognizing the certification marks using object detection. Moreover, 2) the nutrition and health-related texts written on the images were automatically extracted based on optical character recognition (OCR), and the associations between health-related texts were identified by network analysis. The model attained a 94.9% accuracy in identifying certification marks, unveiling prevalent certifications like Kosher. Frequency and network analysis revealed common nutrients and health associations, providing valuable insights into consumer perception. These methods enable fast and efficient monitoring capabilities, which can significantly benefit various food industries. Moreover, the AI-based approaches used in the study are believed to offer insights for related industries regarding the swift transformations in product information status.

Prediction of treatment response after stereotactic radiosurgery of brain metastasis using deep learning and radiomics on longitudinal MRI data.

We developed artificial intelligence models to predict the brain metastasis (BM) treatment response after stereotactic radiosurgery (SRS) using longitudinal magnetic resonance imaging (MRI) data and evaluated prediction accuracy changes according to the number of sequential MRI scans. We included four sequential MRI scans for 194 patients with BM and 369 target lesions for the Developmental dataset. The data were randomly split (8:2 ratio) for training and testing. For external validation, 172 MRI scans from 43 patients with BM and 62 target lesions were additionally enrolled. The maximum axial diameter (Dmax), radiomics, and deep learning (DL) models were generated for comparison. We evaluated the simple convolutional neural network (CNN) model and a gated recurrent unit (Conv-GRU)-based CNN model in the DL arm. The Conv-GRU model performed superior to the simple CNN models. For both datasets, the area under the curve (AUC) was significantly higher for the two-dimensional (2D) Conv-GRU model than for the 3D Conv-GRU, Dmax, and radiomics models. The accuracy of the 2D Conv-GRU model increased with the number of follow-up studies. In conclusion, using longitudinal MRI data, the 2D Conv-GRU model outperformed all other models in predicting the treatment response after SRS of BM.

A Visual Analytics System for Improving Attention-based Traffic Forecasting Models.

With deep learning (DL) outperforming conventional methods for different tasks, much effort has been devoted to utilizing DL in various domains. Researchers and developers in the traffic domain have also designed and improved DL models for forecasting tasks such as estimation of traffic speed and time of arrival. However, there exist many challenges in analyzing DL models due to the black-box property of DL models and complexity of traffic data (i.e., spatio-temporal dependencies). Collaborating with domain experts, we design a visual analytics system, AttnAnalyzer, that enables users to explore how DL models make predictions by allowing effective spatio-temporal dependency analysis. The system incorporates dynamic time warping (DTW) and Granger causality tests for computational spatio-temporal dependency analysis while providing map, table, line chart, and pixel views to assist user to perform dependency and model behavior analysis. For the evaluation, we present three case studies showing how AttnAnalyzer can effectively explore model behaviors and improve model performance in two different road networks. We also provide domain expert feedback.

Quality of laparoscopic camera navigation in robot-assisted versus conventional laparoscopic surgery for rectal cancer: An analysis of surgical videos through a video processing computer software.

BACKGROUND: To compare laparoscopic camera navigation (LCN) quality between robot-assisted laparoscopic surgery (RALS) and conventional laparoscopic surgery (CLS). METHODS: 20 recordings were selected by propensity score matching and subjected to Python® software to generate single frames at one second intervals. For each frame, the pixel where the camera should be centred, based on instrument position, current action (dissection/haemostasis/traction) in the frame, was detected. LCN quality was reviewed by two independent surgeons to evaluate erroneous LCN. RESULTS: RALS had higher incidence of centred views (83.1 ± 4.02% vs. 76.0 ± 2.38%, p < 0.05) and a shorter distance between actual and optimal frame centres (123.3 ± 9.8 vs. 144.8 ± 13.9, p < 0.05) compared to CLS. Erroneous camera navigations were more frequent in CLS regarding total time of horizontal alignment failure (2.1 ± 2.2 vs. 6.0 ± 5.4 min, p = 0.063) and number of excessive zoom-in visualization (0.1 ± 0.3 vs. 1.9 ± 1.4, p = 0.003). CONCLUSIONS: RALS provided higher LCN quality than did CLS, emphasising the benefits of a surgeon-controlled view.

Development of artificial intelligence system for quality control of photo documentation in esophagogastroduodenoscopy.

BACKGROUND: Esophagogastroduodenoscopy (EGD) is generally a safe procedure, but adverse events often occur. This highlights the necessity of the quality control of EGD. Complete visualization and photo documentation of upper gastrointestinal (UGI) tracts are important measures in quality control of EGD. To evaluate these measures in large scale, we developed an AI-driven quality control system for EGD through convolutional neural networks (CNNs) using archived endoscopic images. METHODS: We retrospectively collected and labeled images from 250 EGD procedures, a total of 2599 images from eight locations of the UGI tract, using the European Society of Gastrointestinal Endoscopy (ESGE) photo documentation methods. The label confirmed by five experts was considered the gold standard. We developed a CNN model for multi-class classification of EGD images to one of the eight locations and binary classification of each EGD procedure based on its completeness. RESULTS: Our CNN model successfully classified the EGD images into one of the eight regions of UGI tracts with 97.58% accuracy, 97.42% sensitivity, 99.66% specificity, 97.50% positive predictive value (PPV), and 99.66% negative predictive value (NPV). Our model classified the completeness of EGD with 89.20% accuracy, 89.20% sensitivity, 100.00% specificity, 100.00% PPV, and 64.94% NPV. We analyzed the credibility of our model using a probability heatmap. CONCLUSIONS: We constructed a CNN model that could be used in the quality control of photo documentation in EGD. Our model needs further validation with a large dataset, and we expect our model to help both endoscopists and patients by improving the quality of EGD procedures.

Missing Value Imputation of Time-Series Air-Quality Data via Deep Neural Networks.

To prevent severe air pollution, it is important to analyze time-series air quality data, but this is often challenging as the time-series data is usually partially missing, especially when it is collected from multiple locations simultaneously. To solve this problem, various deep-learning-based missing value imputation models have been proposed. However, often they are barely interpretable, which makes it difficult to analyze the imputed data. Thus, we propose a novel deep learning-based imputation model that achieves high interpretability as well as shows great performance in missing value imputation for spatio-temporal data. We verify the effectiveness of our method through quantitative and qualitative results on a publicly available air-quality dataset.

Prediction of hand-wrist maturation stages based on cervical vertebrae images using artificial intelligence.

OBJECTIVE: To predict the hand-wrist maturation stages based on the cervical vertebrae (CV) images, and to analyse the accuracy of the proposed algorithms. SETTINGS AND POPULATION: A total of 499 pairs of hand-wrist radiographs and lateral cephalograms of 455 orthodontic patients aged 6-18 years were used for developing the prediction model for hand-wrist skeletal maturation stages. MATERIALS AND METHODS: The hand-wrist radiographs and the lateral cephalograms were collected from two university hospitals and a paediatric dental clinic. After identifying the 13 anatomic landmarks of the CV, the width-height ratio, width-perpendicular height ratio and concavity ratio of the CV were used as the morphometric features of the CV. Patients' chronological age and sex were also included as input data. The ground truth data were the Fishman SMI based on the hand-wrist radiographs. Three specialists determined the ground truth SMI. An ensemble machine learning methods were used to predict the Fishman SMI. Five-fold cross-validation was performed. The mean absolute error (MAE), round MAE and root mean square error (RMSE) values were used to assess the performance of the final ensemble model. RESULTS: The final ensemble model consisted of eight machine learning models. The MAE, round MAE and RMSE were 0.90, 0.87 and 1.20, respectively. CONCLUSION: Prediction of hand-wrist SMI based on CV images is possible using machine learning methods. Chronological age and sex increased the prediction accuracy. An automated diagnosis of the skeletal maturation may aid as a decision-supporting tool for evaluating the optimal treatment timing for growing patients.

Artificial intelligence-assisted analysis of endoscopic retrograde cholangiopancreatography image for identifying ampulla and difficulty of selective cannulation.

The advancement of artificial intelligence (AI) has facilitated its application in medical fields. However, there has been little research for AI-assisted endoscopy, despite the clinical significance of the efficiency and safety of cannulation in the endoscopic retrograde cholangiopancreatography (ERCP). In this study, we aim to assist endoscopists performing ERCP through automatic detection of the ampulla and the identification of cannulation difficulty. We developed a novel AI-assisted system based on convolutional neural networks that predict the location of the ampulla and the difficulty of cannulation to the ampulla. ERCP data of 531 and 451 patients were utilized in the evaluation of our model for each task. Our model detected the ampulla with mean intersection-over-union 64.1%, precision 76.2%, recall 78.4%, and centroid distance 0.021. In classifying the cannulation difficulty, it achieved the recall of 71.9% for the class of easy cases and that of 61.1% for that of difficult cases. Remarkably, our model accurately detected AOV with varying morphological shape, size, and texture on par with the level of a human expert and showed promising results for recognizing cannulation difficulty. It demonstrated its potential to improve the quality of ERCP by assisting endoscopists.

HyperTendril: Visual Analytics for User-Driven Hyperparameter Optimization of Deep Neural Networks.

To mitigate the pain of manually tuning hyperparameters of deep neural networks, automated machine learning (AutoML) methods have been developed to search for an optimal set of hyperparameters in large combinatorial search spaces. However, the search results of AutoML methods significantly depend on initial configurations, making it a non-trivial task to find a proper configuration. Therefore, human intervention via a visual analytic approach bears huge potential in this task. In response, we propose HyperTendril, a web-based visual analytics system that supports user-driven hyperparameter tuning processes in a model-agnostic environment. HyperTendril takes a novel approach to effectively steering hyperparameter optimization through an iterative, interactive tuning procedure that allows users to refine the search spaces and the configuration of the AutoML method based on their own insights from given results. Using HyperTendril, users can obtain insights into the complex behaviors of various hyperparameter search algorithms and diagnose their configurations. In addition, HyperTendril supports variable importance analysis to help the users refine their search spaces based on the analysis of relative importance of different hyperparameters and their interaction effects. We present the evaluation demonstrating how HyperTendril helps users steer their tuning processes via a longitudinal user study based on the analysis of interaction logs and in-depth interviews while we deploy our system in a professional industrial environment.

Automatic detection of tympanic membrane and middle ear infection from oto-endoscopic images via convolutional neural networks.

Convolutional neural networks (CNNs), a popular type of deep neural network, have been actively applied to image recognition, object detection, object localization, semantic segmentation, and object instance segmentation. Accordingly, the applicability of deep learning to the analysis of medical images has increased. This paper presents a novel application of state-of-the-art CNN models, such as DenseNet, to the automatic detection of the tympanic membrane (TM) and middle ear (ME) infection. We collected 2,484 oto-endoscopic images (OEIs) and classified them into one of three categories: normal, chronic otitis media (COM) with TM perforation, and otitis media with effusion (OME). Our results indicate that CNN models have significant potential for the automatic recognition of TM and ME infections, demonstrating a competitive accuracy of 95% in classifying TM and middle ear effusion (MEE) from OEIs. In addition to accuracy measurement, our approach achieves nearly perfect measures of 0.99 in terms of the average area under the receiver operating characteristics curve (AUROC). All these results indicate robust performance when recognizing TM and ME effusions in OEIs. Visualization through a class activation mapping (CAM) heatmap demonstrates that our proposed model performs prediction based on the correct region of OEIs. All these outcomes ensure the reliability of our method; hence, the study can aid otolaryngologists and primary care physicians in real-world scenarios.

RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records.

We have recently seen many successful applications of recurrent neural networks (RNNs) on electronic medical records (EMRs), which contain histories of patients' diagnoses, medications, and other various events, in order to predict the current and future states of patients. Despite the strong performance of RNNs, it is often challenging for users to understand why the model makes a particular prediction. Such black-box nature of RNNs can impede its wide adoption in clinical practice. Furthermore, we have no established methods to interactively leverage users' domain expertise and prior knowledge as inputs for steering the model. Therefore, our design study aims to provide a visual analytics solution to increase interpretability and interactivity of RNNs via a joint effort of medical experts, artificial intelligence scientists, and visual analytics researchers. Following the iterative design process between the experts, we design, implement, and evaluate a visual analytics tool called RetainVis, which couples a newly improved, interpretable, and interactive RNN-based model called RetainEX and visualizations for users' exploration of EMR data in the context of prediction tasks. Our study shows the effective use of RetainVis for gaining insights into how individual medical codes contribute to making risk predictions, using EMRs of patients with heart failure and cataract symptoms. Our study also demonstrates how we made substantial changes to the state-of-the-art RNN model called RETAIN in order to make use of temporal information and increase interactivity. This study will provide a useful guideline for researchers that aim to design an interpretable and interactive visual analytics tool for RNNs.

Visual Analytics for Explainable Deep Learning.

Recently, deep learning has been advancing the state of the art in artificial intelligence to a new level, and humans rely on artificial intelligence techniques more than ever. However, even with such unprecedented advancements, the lack of explanation regarding the decisions made by deep learning models and absence of control over their internal processes act as major drawbacks in critical decision-making processes, such as precision medicine and law enforcement. In response, efforts are being made to make deep learning interpretable and controllable by humans. This article reviews visual analytics, information visualization, and machine learning perspectives relevant to this aim, and discusses potential challenges and future research directions.

ConceptVector: Text Visual Analytics via Interactive Lexicon Building Using Word Embedding.

Central to many text analysis methods is the notion of a concept: a set of semantically related keywords characterizing a specific object, phenomenon, or theme. Advances in word embedding allow building a concept from a small set of seed terms. However, naive application of such techniques may result in false positive errors because of the polysemy of natural language. To mitigate this problem, we present a visual analytics system called ConceptVector that guides a user in building such concepts and then using them to analyze documents. Document-analysis case studies with real-world datasets demonstrate the fine-grained analysis provided by ConceptVector. To support the elaborate modeling of concepts, we introduce a bipolar concept model and support for specifying irrelevant words. We validate the interactive lexicon building interface by a user study and expert reviews. Quantitative evaluation shows that the bipolar lexicon generated with our methods is comparable to human-generated ones.

Toward Predicting Social Support Needs in Online Health Social Networks.

BACKGROUND: While online health social networks (OHSNs) serve as an effective platform for patients to fulfill their various social support needs, predicting the needs of users and providing tailored information remains a challenge. OBJECTIVE: The objective of this study was to discriminate important features for identifying users' social support needs based on knowledge gathered from survey data. This study also provides guidelines for a technical framework, which can be used to predict users' social support needs based on raw data collected from OHSNs. METHODS: We initially conducted a Web-based survey with 184 OHSN users. From this survey data, we extracted 34 features based on 5 categories: (1) demographics, (2) reading behavior, (3) posting behavior, (4) perceived roles in OHSNs, and (5) values sought in OHSNs. Features from the first 4 categories were used as variables for binary classification. For the prediction outcomes, we used features from the last category: the needs for emotional support, experience-based information, unconventional information, and medical facts. We compared 5 binary classifier algorithms: gradient boosting tree, random forest, decision tree, support vector machines, and logistic regression. We then calculated the scores of the area under the receiver operating characteristic (ROC) curve (AUC) to understand the comparative effectiveness of the used features. RESULTS: The best performance was AUC scores of 0.89 for predicting users seeking emotional support, 0.86 for experience-based information, 0.80 for unconventional information, and 0.83 for medical facts. With the gradient boosting tree as our best performing model, we analyzed the strength of individual features in predicting one's social support need. Among other discoveries, we found that users seeking emotional support tend to post more in OHSNs compared with others. CONCLUSIONS: We developed an initial framework for automatically predicting social support needs in OHSNs using survey data. Future work should involve nonsurvey data to evaluate the feasibility of the framework. Our study contributes to providing personalized social support in OHSNs.

When Bitcoin encounters information in an online forum: Using text mining to analyse user opinions and predict value fluctuation.

Bitcoin is an online currency that is used worldwide to make online payments. It has consequently become an investment vehicle in itself and is traded in a way similar to other open currencies. The ability to predict the price fluctuation of Bitcoin would therefore facilitate future investment and payment decisions. In order to predict the price fluctuation of Bitcoin, we analyse the comments posted in the Bitcoin online forum. Unlike most research on Bitcoin-related online forums, which is limited to simple sentiment analysis and does not pay sufficient attention to note-worthy user comments, our approach involved extracting keywords from Bitcoin-related user comments posted on the online forum with the aim of analytically predicting the price and extent of transaction fluctuation of the currency. The effectiveness of the proposed method is validated based on Bitcoin online forum data ranging over a period of 2.8 years from December 2013 to September 2016.

TopicLens: Efficient Multi-Level Visual Topic Exploration of Large-Scale Document Collections.

Topic modeling, which reveals underlying topics of a document corpus, has been actively adopted in visual analytics for large-scale document collections. However, due to its significant processing time and non-interactive nature, topic modeling has so far not been tightly integrated into a visual analytics workflow. Instead, most such systems are limited to utilizing a fixed, initial set of topics. Motivated by this gap in the literature, we propose a novel interaction technique called TopicLens that allows a user to dynamically explore data through a lens interface where topic modeling and the corresponding 2D embedding are efficiently computed on the fly. To support this interaction in real time while maintaining view consistency, we propose a novel efficient topic modeling method and a semi-supervised 2D embedding algorithm. Our work is based on improving state-of-the-art methods such as nonnegative matrix factorization and t-distributed stochastic neighbor embedding. Furthermore, we have built a web-based visual analytics system integrated with TopicLens. We use this system to measure the performance and the visualization quality of our proposed methods. We provide several scenarios showcasing the capability of TopicLens using real-world datasets.

AxiSketcher: Interactive Nonlinear Axis Mapping of Visualizations through User Drawings.

Visual analytics techniques help users explore high-dimensional data. However, it is often challenging for users to express their domain knowledge in order to steer the underlying data model, especially when they have little attribute-level knowledge. Furthermore, users' complex, high-level domain knowledge, compared to low-level attributes, posits even greater challenges. To overcome these challenges, we introduce a technique to interpret a user's drawings with an interactive, nonlinear axis mapping approach called AxiSketcher. This technique enables users to impose their domain knowledge on a visualization by allowing interaction with data entries rather than with data attributes. The proposed interaction is performed through directly sketching lines over the visualization. Using this technique, users can draw lines over selected data points, and the system forms the axes that represent a nonlinear, weighted combination of multidimensional attributes. In this paper, we describe our techniques in three areas: 1) the design space of sketching methods for eliciting users' nonlinear domain knowledge; 2) the underlying model that translates users' input, extracts patterns behind the selected data points, and results in nonlinear axes reflecting users' complex intent; and 3) the interactive visualization for viewing, assessing, and reconstructing the newly formed, nonlinear axes.

Personas in online health communities.

Many researchers and practitioners use online health communities (OHCs) to influence health behavior and provide patients with social support. One of the biggest challenges in this approach, however, is the rate of attrition. OHCs face similar problems as other social media platforms where user migration happens unless tailored content and appropriate socialization is supported. To provide tailored support for each OHC user, we developed personas in OHCs illustrating users' needs and requirements in OHC use. To develop OHC personas, we first interviewed 16 OHC users and administrators to qualitatively understand varying user needs in OHC. Based on their responses, we developed an online survey to systematically investigate OHC personas. We received 184 survey responses from OHC users, which informed their values and their OHC use patterns. We performed open coding analysis with the interview data and cluster analysis with the survey data and consolidated the analyses of the two datasets. Four personas emerged-Caretakers, Opportunists, Scientists, and Adventurers. The results inform users' interaction behavior and attitude patterns with OHCs. We discuss implications for how these personas inform OHCs in delivering personalized informational and emotional support.

VisOHC: Designing Visual Analytics for Online Health Communities.

Through online health communities (OHCs), patients and caregivers exchange their illness experiences and strategies for overcoming the illness, and provide emotional support. To facilitate healthy and lively conversations in these communities, their members should be continuously monitored and nurtured by OHC administrators. The main challenge of OHC administrators' tasks lies in understanding the diverse dimensions of conversation threads that lead to productive discussions in their communities. In this paper, we present a design study in which three domain expert groups participated, an OHC researcher and two OHC administrators of online health communities, which was conducted to find with a visual analytic solution. Through our design study, we characterized the domain goals of OHC administrators and derived tasks to achieve these goals. As a result of this study, we propose a system called VisOHC, which visualizes individual OHC conversation threads as collapsed boxes-a visual metaphor of conversation threads. In addition, we augmented the posters' reply authorship network with marks and/or beams to show conversation dynamics within threads. We also developed unique measures tailored to the characteristics of OHCs, which can be encoded for thread visualizations at the users' requests. Our observation of the two administrators while using VisOHC showed that it supports their tasks and reveals interesting insights into online health communities. Finally, we share our methodological lessons on probing visual designs together with domain experts by allowing them to freely encode measurements into visual variables.