Graph attention-based fusion of pathology images and gene expression for prediction of cancer survival.

Multimodal machine learning models are being developed to analyze pathology images and other modalities, such as gene expression, to gain clinical and biological insights. However, most frameworks for multimodal data fusion do not fully account for the interactions between different modalities. Here, we present an attention-based fusion architecture that integrates a graph representation of pathology images with gene expression data and concomitantly learns from the fused information to predict patient-specific survival. In our approach, pathology images are represented as undirected graphs, and their embeddings are combined with embeddings of gene expression signatures using an attention mechanism to stratify tumors by patient survival. We show that our framework improves the survival prediction of human non-small cell lung cancers, outperforming existing state-of-the-art approaches that leverage multimodal data. Our framework can facilitate spatial molecular profiling to identify tumor heterogeneity using pathology images and gene expression data, complementing results obtained from more expensive spatial transcriptomic and proteomic technologies.

Curated and Annotated Dataset of Lung US Images in Zambian Children with Clinical Pneumonia.

See also the commentary by Sitek in this issue. Supplemental material is available for this article.

Consistent long-term practice leads to consistent improvement: Benefits of self-managed therapy for language and cognitive deficits using a digital therapeutic.

Background: Although speech-language therapy (SLT) is proven to be beneficial to recovery of post-stroke aphasia, delivering sufficiently high amounts of dosage remains a problem in real-world clinical practice. Self-managed SLT was introduced to solve the problem. Previous research showed in a 10-week period, increased dosage frequency could lead to better performance, however, it is uncertain if dosage still affects performance over a longer period of practice time and whether gains can be seen following practice over several months. Objective: This study aims to evaluate data from a health app (Constant Therapy) to investigate the relationship between dosage amount and improvements following a 30-week treatment period. Two cohorts of users were analyzed. One was comprised of patients with a consistent average weekly dosage amount and the other cohort was comprised of users whose practice had higher variability. Methods: We conducted two analyses with two cohorts of post-stroke patients who used Constant Therapy. The first cohort contains 537 "consistent" users, while the second cohort contains 2,159. The 30-week practice period was split into three consecutive 10-week practice windows to calculate average dosage amount. In each 10-week practice period, patients were grouped by their average dosage into low (0-15 min/week), medium (15-40 min/week) and moderate dosage (greater than 40 min/week) groups. Linear mixed-effects models were employed to evaluate if dosage amount was a significant factor affecting performance. Pairwise comparison was also applied to evaluate the slope difference between groups. Results: For the consistent cohort, medium (ß = .002, t 17,700 = 7.64, P < .001) and moderate (ß = .003, t 9,297 = 7.94, P < .001) dosage groups showed significant improvement compared to the low dosage group. The moderate group also showed greater improvement compared to the medium group. For the variable cohort in analysis 2, the same trend was shown in the first two 10-week windows, however, in weeks 21-30, the difference was insignificant between low and medium groups (ß = .001, t = 1.76, P = .078). Conclusions: This study showed a higher dosage amount is related to greater therapy outcomes in over 6 months of digital self-managed therapy. It also showed that regardless of the exact pattern of practice, self-managed SLT leads to significant and sustained performance gains.

A Graph-Transformer for Whole Slide Image Classification.

Deep learning is a powerful tool for whole slide image (WSI) analysis. Typically, when performing supervised deep learning, a WSI is divided into small patches, trained and the outcomes are aggregated to estimate disease grade. However, patch-based methods introduce label noise during training by assuming that each patch is independent with the same label as the WSI and neglect overall WSI-level information that is significant in disease grading. Here we present a Graph-Transformer (GT) that fuses a graph-based representation of an WSI and a vision transformer for processing pathology images, called GTP, to predict disease grade. We selected 4,818 WSIs from the Clinical Proteomic Tumor Analysis Consortium (CPTAC), the National Lung Screening Trial (NLST), and The Cancer Genome Atlas (TCGA), and used GTP to distinguish adenocarcinoma (LUAD) and squamous cell carcinoma (LSCC) from adjacent non-cancerous tissue (normal). First, using NLST data, we developed a contrastive learning framework to generate a feature extractor. This allowed us to compute feature vectors of individual WSI patches, which were used to represent the nodes of the graph followed by construction of the GTP framework. Our model trained on the CPTAC data achieved consistently high performance on three-label classification (normal versus LUAD versus LSCC: mean accuracy = 91.2 ± 2.5%) based on five-fold cross-validation, and mean accuracy = 82.3 ± 1.0% on external test data (TCGA). We also introduced a graph-based saliency mapping technique, called GraphCAM, that can identify regions that are highly associated with the class label. Our findings demonstrate GTP as an interpretable and effective deep learning framework for WSI-level classification.

Multimodal Neural and Behavioral Data Predict Response to Rehabilitation in Chronic Poststroke Aphasia.

BACKGROUND: Poststroke recovery depends on multiple factors and varies greatly across individuals. Using machine learning models, this study investigated the independent and complementary prognostic role of different patient-related factors in predicting response to language rehabilitation after a stroke. METHODS: Fifty-five individuals with chronic poststroke aphasia underwent a battery of standardized assessments and structural and functional magnetic resonance imaging scans, and received 12 weeks of language treatment. Support vector machine and random forest models were constructed to predict responsiveness to treatment using pretreatment behavioral, demographic, and structural and functional neuroimaging data. RESULTS: The best prediction performance was achieved by a support vector machine model trained on aphasia severity, demographics, measures of anatomic integrity and resting-state functional connectivity (F1=0.94). This model resulted in a significantly superior prediction performance compared with support vector machine models trained on all feature sets (F1=0.82, P<0.001) or a single feature set (F1 range=0.68-0.84, P<0.001). Across random forest models, training on resting-state functional magnetic resonance imaging connectivity data yielded the best F1 score (F1=0.87). CONCLUSIONS: While behavioral, multimodal neuroimaging data and demographic information carry complementary information in predicting response to rehabilitation in chronic poststroke aphasia, functional connectivity of the brain at rest after stroke is a particularly important predictor of responsiveness to treatment, both alone and combined with other patient-related factors.

Deep-Learning-Driven Quantification of Interstitial Fibrosis in Digitized Kidney Biopsies.

Interstitial fibrosis and tubular atrophy (IFTA) on a renal biopsy are strong indicators of disease chronicity and prognosis. Techniques that are typically used for IFTA grading remain manual, leading to variability among pathologists. Accurate IFTA estimation using computational techniques can reduce this variability and provide quantitative assessment. Using trichrome-stained whole-slide images (WSIs) processed from human renal biopsies, we developed a deep-learning framework that captured finer pathologic structures at high resolution and overall context at the WSI level to predict IFTA grade. WSIs (n = 67) were obtained from The Ohio State University Wexner Medical Center. Five nephropathologists independently reviewed them and provided fibrosis scores that were converted to IFTA grades: ≤10% (none or minimal), 11% to 25% (mild), 26% to 50% (moderate), and >50% (severe). The model was developed by associating the WSIs with the IFTA grade determined by majority voting (reference estimate). Model performance was evaluated on WSIs (n = 28) obtained from the Kidney Precision Medicine Project. There was good agreement on the IFTA grading between the pathologists and the reference estimate (κ = 0.622 ± 0.071). The accuracy of the deep-learning model was 71.8% ± 5.3% on The Ohio State University Wexner Medical Center and 65.0% ± 4.2% on Kidney Precision Medicine Project data sets. Our approach to analyzing microscopic- and WSI-level changes in renal biopsies attempts to mimic the pathologist and provides a regional and contextual estimation of IFTA. Such methods can assist clinicopathologic diagnosis.

Project SEARCH (Scanning EARs for Child Health): validating an ear biometric tool for patient identification in Zambia.

Patient identification in low- to middle-income countries is one of the most pressing public health challenges of our day. Given the ubiquity of mobile phones, their use for health-care coupled with a biometric identification method, present a unique opportunity to address this challenge. Our research proposes an Android-based solution of an ear biometric tool for reliable identification. Unlike many popular biometric approaches (e.g., fingerprints, irises, facial recognition), ears are noninvasive and easily accessible on individuals across a lifespan. Our ear biometric tool uses a combination of hardware and software to identify a person using an image of their ear. The hardware supports an image capturing process that reduces undesired variability. The software uses a pattern recognition algorithm to transform an image of the ear into a unique identifier. We created three cross-sectional datasets of ear images, each increasing in complexity, with the final dataset representing our target use-case population of Zambian infants (N=224, aged 6days-6months). Using these datasets, we conducted a series of validation experiments, which informed iterative improvements to the system. Results of the improved system, which yielded high recognition rates across the three datasets, demonstrate the feasibility of an Android ear biometric tool as a solution to the persisting patient identification challenge.

Using collision cones to assess biological deconfliction methods.

Biological systems consistently outperform autonomous systems governed by engineered algorithms in their ability to reactively avoid collisions. To better understand this discrepancy, a collision avoidance algorithm was applied to frames of digitized video trajectory data from bats, swallows and fish (Myotis velifer, Petrochelidon pyrrhonota and Danio aequipinnatus). Information available from visual cues, specifically relative position and velocity, was provided to the algorithm which used this information to define collision cones that allowed the algorithm to find a safe velocity requiring minimal deviation from the original velocity. The subset of obstacles provided to the algorithm was determined by the animal's sensing range in terms of metric and topological distance. The algorithmic calculated velocities showed good agreement with observed biological velocities, indicating that the algorithm was an informative basis for comparison with the three species and could potentially be improved for engineered applications with further study.

Perceptual Modalities Guiding Bat Flight in a Native Habitat.

Flying animals accomplish high-speed navigation through fields of obstacles using a suite of sensory modalities that blend spatial memory with input from vision, tactile sensing, and, in the case of most bats and some other animals, echolocation. Although a good deal of previous research has been focused on the role of individual modes of sensing in animal locomotion, our understanding of sensory integration and the interplay among modalities is still meager. To understand how bats integrate sensory input from echolocation, vision, and spatial memory, we conducted an experiment in which bats flying in their natural habitat were challenged over the course of several evening emergences with a novel obstacle placed in their flight path. Our analysis of reconstructed flight data suggests that vision, echolocation, and spatial memory together with the possible exercise of an ability in using predictive navigation are mutually reinforcing aspects of a composite perceptual system that guides flight. Together with the recent development in robotics, our paper points to the possible interpretation that while each stream of sensory information plays an important role in bat navigation, it is the emergent effects of combining modalities that enable bats to fly through complex spaces.

A protocol and calibration method for accurate multi-camera field videography.

Stereo videography is a powerful technique for quantifying the kinematics and behavior of animals, but it can be challenging to use in an outdoor field setting. We here present a workflow and associated software for performing calibration of cameras placed in a field setting and estimating the accuracy of the resulting stereoscopic reconstructions. We demonstrate the workflow through example stereoscopic reconstructions of bat and bird flight. We provide software tools for planning experiments and processing the resulting calibrations that other researchers may use to calibrate their own cameras. Our field protocol can be deployed in a single afternoon, requiring only short video clips of light, portable calibration objects.

The kernel semi-least squares method for sparse distance approximation.

We extend the semi-least squares problem defined by Rao and Mitra ( 1971 ) to the kernel semi-least squares problem. We introduce subset projection, a technique that produces a solution to this problem. We show how the results of subset projection can be used to approximate a computationally expensive distance metric.

Hierarchical partial matching and segmentation of interacting cells.

We propose a method that automatically tracks and segments living cells in phase-contrast image sequences, especially for cells that deform and interact with each other or clutter. We formulate the problem as a many-to-one elastic partial matching problem between closed curves. We introduce Double Cyclic Dynamic Time Warping for the scenario where a collision event yields a single boundary that encloses multiple touching cells and that needs to be cut into separate cell boundaries. The resulting individual boundaries may consist of segments to be connected to produce closed curves that match well with the individual cell boundaries before the collision event. We show how to convert this partial-curve matching problem into a shortest path problem that we then solve efficiently by reusing the computed shortest path tree. We also use our shortest path algorithm to fill the gaps between the segments of the target curves. Quantitative results demonstrate the benefit of our method by showing maintained accurate recognition of individual cell boundaries across 8068 images containing multiple cell interactions.

Explaining clustered ventilation defects via a minimal number of airway closure locations.

Computational models of the human lung have been developed to study lung physiology and have been used to identify the airways responsible for mechanical dysfunction in asthmatics. Tgavalekos et al. used models anatomically consistent with the human lung to link ventilation defects to the heterogeneous closure of small airways. Their approach implicitly assumed a high degree of independence between airway closures as indicated by the low compactness of the airway structures mapped to individual ventilation defects. Venegas et al. however, have found that significant mutual dependence of airways may play a role in patchy ventilation of asthmatics. This led us to explore the question to what extent anatomically consistent models can be built which do not implicitly assume high independence of airways but instead allow for the mutual dependence of airways responsible for ventilation defects. We propose an algorithm for generating subject-specific airway-tree models that minimize the number of airways that must be closed or severely constricted to cause observed ventilation defects. We also propose novel approaches for measuring the compactness of airway structures. Our approach shows that anatomically consistent models which link compact airway structures to ventilation defects can be built. Our model also shows that some ventilation defects may be caused by closures of larger airways than previously reported.

Brazilian free-tailed bats as insect pest regulators in transgenic and conventional cotton crops.

During the past 12000 years agricultural systems have transitioned from natural habitats to conventional agricultural regions and recently to large areas of genetically engineered (GE) croplands. This GE revolution occurred for cotton in a span of slightly more than a decade during which a switch occurred in major cotton production areas from growing 100% conventional cotton to an environment in which 95% transgenics are grown. Ecological interactions between GE targeted insects and other insectivorous insects have been investigated. However, the relationships between ecological functions (such as herbivory and ecosystem transport) and agronomic benefits of avian or mammalian insectivores in the transgenic environment generally remain unclear, although the importance of some agricultural pest management services provided by insectivorous species such as the Brazilian free-tailed bat, Tadarida brasiliensis, have been recognized. We developed a dynamic model to predict regional-scale ecological functions in agricultural food webs by using the indicators of insect pest herbivory measured by cotton boll damage and insect emigration from cotton. In the south-central Texas Winter Garden agricultural region we find that the process of insectivory by bats has a considerable impact on both the ecology and valuation of harvest in Bacillus thuringiensis (Bt) transgenic and nontransgenic cotton crops. Predation on agricultural pests by insectivorous bats may enhance the economic value of agricultural systems by reducing the frequency of required spraying and delaying the ultimate need for new pesticides. In the Winter Garden region, the presence of large numbers of insectivorous bats yields a regional summer dispersion of adult pest insects from Bt cotton that is considerably reduced from the moth emigration when bats are absent in either transgenic or non-transgenic crops. This regional decrease of pest numbers impacts insect herbivory on a transcontinental scale. With a few exceptions, we find that the agronomics of both Bt and conventional cotton production is more profitable when large numbers of insectivorous bats are present.

Detecting objects of variable shape structure with hidden state shape models.

This paper proposes a method for detecting object classes that exhibit variable shape structure in heavily cluttered images. The term "variable shape structure" is used to characterize object classes in which some shape parts can be repeated an arbitrary number of times, some parts can be optional, and some parts can have several alternative appearances. Hidden State Shape Models (HSSMs), a generalization of Hidden Markov Models (HMMs), are introduced to model object classes of variable shape structure using a probabilistic framework. A polynomial inference algorithm automatically determines object location, orientation, scale and structure by finding the globally optimal registration of model states with the image features, even in the presence of clutter. Experiments with real images demonstrate that the proposed method can localize objects of variable shape structure with high accuracy. For the task of hand shape localization and structure identification, the proposed method is significantly more accurate than previously proposed methods based on chamfer-distance matching. Furthermore, by integrating simple temporal constraints, the proposed method gains speed-ups of more than an order of magnitude, and produces highly accurate results in experiments on non-rigid hand motion tracking.

Applications of thermal infrared imaging for research in aeroecology.

The night sky remains a largely unexplored frontier for biologists studying the behavior and physiology of free-ranging, nocturnal organisms. Conventional imaging tools and techniques such as night-vision scopes, infrared-reflectance cameras, flash cameras, and radar provide insufficient detail for the scale and resolution demanded by field researchers. A new tool is needed that is capable of imaging noninvasively in the dark at high-temporal and spatial resolution. Thermal infrared imaging represents the most promising such technology that is poised to revolutionize our ability to observe and document the behavior of free-ranging organisms in the dark. Herein we present several examples from our research on free-ranging bats that highlight the power and potential of thermal infrared imaging for the study of animal behavior, energetics and censusing of large colonies, among others. Using never-before-seen video footage and data, we have begun to answer questions that have puzzled biologists for decades, as well as to generate new hypotheses and insight. As we begin to appreciate the functional significance of the aerosphere as a dynamic environment that affects organisms at different spatial and temporal scales, thermal infrared imaging can be at the forefront of the effort to explore this next frontier.

Pulmonary fissure segmentation on CT.

A pulmonary fissure is a boundary between the lobes in the lungs. Its segmentation is of clinical interest as it facilitates the assessment of lung disease on a lobar level. This paper describes a new approach for segmenting the major fissures in both lungs on thin-section computed tomography (CT). An image transformation called "ridge map" is proposed for enhancing the appearance of fissures on CT. A curve-growing process, modeled by a Bayesian network, is described that is influenced by both the features of the ridge map and prior knowledge of the shape of the fissure. The process is implemented in an adaptive regularization framework that balances these influences and reflects the causal dependencies in the Bayesian network using an entropy measure. The method effectively alleviates the problem of inappropriate weights of regularization terms, an effect that can occur with static regularization methods. The method was applied to segment and visualize the lobes of the lungs on chest CT of 10 patients with pulmonary nodules. Only 78 out of 3286 left or right lung regions with fissures (2.4%) required manual correction. The average distance between the automatically segmented and the manually delineated "ground-truth" fissures was 1.01 mm, which was similar to the average distance of 1.03 mm between two sets of manually segmented fissures. The method has a linear-time worst-case complexity and segments the upper lung from the lower lung on a standard computer in less than 5 min.

The correlation between internal and external markers for abdominal tumors: implications for respiratory gating.

PURPOSE: The correlation of the respiratory motion of external patient markers and abdominal tumors was examined. Data of this type are important for image-guided therapy techniques, such as respiratory gating, that monitor the movement of external fiducials. METHODS AND MATERIALS: Fluoroscopy sessions for 4 patients with internal, radiopaque tumor fiducial clips were analyzed by computer vision techniques. The motion of the internal clips and the external markers placed on the patient's abdominal skin surface were quantified and correlated. RESULTS: In general, the motion of the tumor and external markers were well correlated. The maximum amount of peak-to-peak craniocaudal tumor motion was 2.5 cm. The ratio of tumor motion to external-marker motion ranged from 0.85 to 7.1. The variation in tumor position for a given external-marker position ranged from 2 to 9 mm. The period of the breathing cycle ranged from 2.7 to 4.5 seconds, and the frequency patterns for both the tumor and the external markers were similar. CONCLUSIONS: Although tumor motion generally correlated well with external fiducial marker motion, relatively large underlying tumor motion can occur compared with external-marker motion and variations in the tumor position for a given marker position. Treatment margins should be determined on the basis of a detailed understanding of tumor motion, as opposed to relying only on external-marker information.

Segmentation of nodules on chest computed tomography for growth assessment.

Several segmentation methods to evaluate growth of small isolated pulmonary nodules on chest computed tomography (CT) are presented. The segmentation methods are based on adaptively thresholding attenuation levels and use measures of nodule shape. The segmentation methods were first tested on a realistic chest phantom to evaluate their performance with respect to specific nodule characteristics. The segmentation methods were also tested on sequential CT scans of patients. The methods' estimation of nodule growth were compared to the volume change calculated by a chest radiologist. The best method segmented nodules on average 43% smaller or larger than the actual nodule when errors were computed across all nodule variations on the phantom. Some methods achieved smaller errors when examined with respect to certain nodule properties. In particular, on the phantom individual methods segmented solid nodules to within 23% of their actual size and nodules with 60.7 mm3 volumes to within 14%. On the clinical data, none of the methods examined showed a statistically significant difference in growth estimation from the radiologist.

Quantification of respiration-induced abdominal tumor motion and its impact on IMRT dose distributions.

PURPOSE: The treatment of moving targets with intensity-modulated radiotherapy may introduce errors in dose delivery. The motion of tumors in the abdomen was studied using quantitative fluoroscopic analysis, and the effect on dose delivery to the target was studied. METHODS AND MATERIALS: Fluoroscopy sessions for 7 patients with pancreas or liver tumors and fiducial clips were recorded, converted to digital format, and analyzed to quantify the characteristics of tumor motion. Intensity-modulated radiotherapy plans were generated for 3 patients (a total of five plans), and the dose-volume histograms for the target volume were compared between plans with and without tumor motion. RESULTS: The average magnitude of the peak-to-peak motion for the 7 patients in the craniocaudal and AP directions was 7.4 mm and 3.8 mm, respectively. The clip motion varied widely, because the maximal clip excursions were about 47% greater than the average clip excursions for each patient. The inclusion of tumor motion did not lead to a significant degradation in the target dose-volume histogram for four of five treatment plans studied. CONCLUSION: The amount of tumor motion for most patients in this study was not large but could, in some instances, significantly degrade the planned target dose-volume histogram. For some patients, therefore, motion mitigation or intervention during treatment may be necessary.