High Frankia abundance and low diversity of microbial community are associated with nodulation specificity and stability of sea buckthorn root nodule.

Introduction: Actinorhizal symbioses are gaining attention due to the importance of symbiotic nitrogen fixation in sustainable agriculture. Sea buckthorn (Hippophae L.) is an important actinorhizal plant, yet research on the microbial community and nitrogen cycling in its nodules is limited. In addition, the influence of environmental differences on the microbial community of sea buckthorn nodules and whether there is a single nitrogen-fixing actinomycete species in the nodules are still unknown. Methods: We investigated the diversity, community composition, network associations and nitrogen cycling pathways of the microbial communities in the root nodule (RN), nodule surface soil (NS), and bulk soil (BS) of Mongolian sea buckthorn distributed under three distinct ecological conditions in northern China using 16S rRNA gene and metagenomic sequencing. Combined with the data of environmental factors, the effects of environmental differences on different sample types were analyzed. Results: The results showed that plants exerted a clear selective filtering effect on microbiota, resulting in a significant reduction in microbial community diversity and network complexity from BS to NS to RN. Proteobacteria was the most abundant phylum in the microbiomes of BS and NS. While RN was primarily dominated by Actinobacteria, with Frankia sp. EAN1pec serving as the most dominant species. Correlation analysis indicated that the host determined the microbial community composition in RN, independent of the ecological and geographical environmental changes of the sea buckthorn plantations. Nitrogen cycle pathway analyses showed that RN microbial community primarily functions in nitrogen fixation, and Frankia sp. EAN1pec was a major contributor to nitrogen fixation genes in RN. Discussion: This study provides valuable insights into the effects of eco-geographical environment on the microbial communities of sea buckthorn RN. These findings further prove that the nodulation specificity and stability of sea buckthorn root and Frankia sp. EAN1pec may be the result of their long-term co-evolution.

A chromosome-scale genome of Rhus chinensis Mill. provides new insights into plant-insect interaction and gallotannins biosynthesis.

Rhus chinensis Mill., an economically valuable Anacardiaceae species, is parasitized by the galling aphid Schlechtendalia chinensis, resulting in the formation of the Chinese gallnut (CG). Here, we report a chromosomal-level genome assembly of R. chinensis, with a total size of 389.40 Mb and scaffold N50 of 23.02 Mb. Comparative genomic and transcriptome analysis revealed that the enhanced structure of CG and nutritional metabolism contribute to improving the adaptability of R. chinensis to S. chinensis by supporting CG and galling aphid growth. CG was observed to be abundant in hydrolysable tannins (HT), particularly gallotannin and its isomers. Tandem repeat clusters of dehydroquinate dehydratase/shikimate dehydrogenase (DQD/SDH) and serine carboxypeptidase-like (SCPL) and their homologs involved in HT production were determined as specific to HT-rich species. The functional differentiation of DQD/SDH tandem duplicate genes and the significant contraction in the phenylalanine ammonia-lyase (PAL) gene family contributed to the accumulation of gallic acid and HT while minimizing the production of shikimic acid, flavonoids, and condensed tannins in CG. Furthermore, we identified one UDP glucosyltransferase (UGT84A), three carboxylesterase (CXE), and six SCPL genes from conserved tandem repeat clusters that are involved in gallotannin biosynthesis and hydrolysis in CG. We then constructed a regulatory network of these genes based on co-expression and transcription factor motif analysis. Our findings provide a genomic resource for the exploration of the underlying mechanisms of plant-galling insect interaction and highlight the importance of the functional divergence of tandem duplicate genes in the accumulation of secondary metabolites.

Variational Adversarial Defense: A Bayes Perspective for Adversarial Training.

Various methods have been proposed to defend against adversarial attacks. However, there is a lack of enough theoretical guarantee of the performance, thus leading to two problems: First, deficiency of necessary adversarial training samples might attenuate the normal gradient's back-propagation, which leads to overfitting and gradient masking potentially. Second, point-wise adversarial sampling offers an insufficient support region for adversarial data and thus cannot form a robust decision-boundary. To solve these issues, we provide a theoretical analysis to reveal the relationship between robust accuracy and the complexity of the training set in adversarial training. As a result, we propose a novel training scheme called Variational Adversarial Defense. Based on the distribution of adversarial samples, this novel construction upgrades the defend scheme from local point-wise to distribution-wise, yielding an enlarged support region for safeguarding robust training, thus possessing a higher promising to defense attacks. The proposed method features the following advantages: 1) Instead of seeking adversarial examples point-by-point (in a sequential way), we draw diverse adversarial examples from the inferred distribution; and 2) Augmenting the training set by a larger support region consolidates the smoothness of the decision boundary. Finally, the proposed method is analyzed via the Taylor expansion technique, which casts our solution with natural interpretability.

Change in three-dimensional choroidal vessel network after AR device assisted 1-hour visual task in 2D/3D mode in young healthy subjects.

PURPOSE: The purpose of the study was to investigate the changes of choroidal blood perfusion in different layers and quadrants and its possible related factors after 1 h visual task by augmented reality (AR) device in two-dimensional (2D) and three-dimensional (3D) mode, respectively. METHODS: Thirty healthy subjects aged 22-37 years watched the same video source in 2D and 3D mode separately using AR glasses for 1 h with a one-week interval. Swept-source optical coherence tomography angiography (SS-OCTA) was performed before and immediately after watching to acquire choroidal thickness (ChT), three-dimensional choroidal vascularity index (CVI) of large- and middle-sized choroidal vessels and choriocapillaris flow voids (FV%) at macular and peripapillary area. Near point of accommodation (NPA) and accommodative facility (AF) were examined to evaluate the accommodative ability. Pupil diameters by infrared-automated pupillometer under scotopic, mesopic and photopic condition were also obtained. RESULTS: Compared with pre-visual task, the subfoveal CVI decreased from 0.406 ± 0.097 to 0.360 ± 0.102 after 2D watching (p < 0.001) and to 0.368 ± 0.102 after 3D watching (p = 0.002). Pupil sizes under different illuminance conditions became smaller after both 2D and 3D watching (all p < 0.001). AF increased after both 2D and 3D watching (both p < 0.05). NPA receded in post-3D watching (p = 0.017) while a not significant tendency was observed in post-2D. CONCLUSION: A reduction in subfoveal choroidal blood flow accompanied with pupil constriction was observed immediately after 1 h visual task using AR glasses in 2D and 3D mode. Accommodative facility improved after 2D and 3D watching with AR glasses, whereas decrease in the maximum accommodation power was only found in 3D mode.

RibSeg v2: A Large-scale Benchmark for Rib Labeling and Anatomical Centerline Extraction.

Automatic rib labeling and anatomical centerline extraction are common prerequisites for various clinical applications. Prior studies either use in-house datasets that are inaccessible to communities, or focus on rib segmentation that neglects the clinical significance of rib labeling. To address these issues, we extend our prior dataset (RibSeg) on the binary rib segmentation task to a comprehensive benchmark, named RibSeg v2, with 660 CT scans (15,466 individual ribs in total) and annotations manually inspected by experts for rib labeling and anatomical centerline extraction. Based on the RibSeg v2, we develop a pipeline including deep learning-based methods for rib labeling, and a skeletonization-based method for centerline extraction. To improve computational efficiency, we propose a sparse point cloud representation of CT scans and compare it with standard dense voxel grids. Moreover, we design and analyze evaluation metrics to address the key challenges of each task. Our dataset, code, and model are available online to facilitate open research at https://github.com/M3DV/RibSeg.

CIPS-3D++: End-to-End Real-Time High-Resolution 3D-Aware GANs for GAN Inversion and Stylization.

Style-based GANs achieve state-of-the-art results for generating high-quality images, but lack explicit and precise control over camera poses. Recently proposed NeRF-based GANs have made great progress towards 3D-aware image generation. However, the methods either rely on convolution operators which are not rotationally invariant, or utilize complex yet suboptimal training procedures to integrate both NeRF and CNN sub-structures, yielding un-robust, low-quality images with a large computational burden. This article presents an upgraded version called CIPS-3D++, aiming at high-robust, high-resolution and high-efficiency 3D-aware GANs. On the one hand, our basic model CIPS-3D, encapsulated in a style-based architecture, features a shallow NeRF-based 3D shape encoder as well as a deep MLP-based 2D image decoder, achieving robust image generation/editing with rotation-invariance. On the other hand, our proposed CIPS-3D++, inheriting the rotational invariance of CIPS-3D, together with geometric regularization and upsampling operations, encourages high-resolution high-quality image generation/editing with great computational efficiency. Trained on raw single-view images, without any bells and whistles, CIPS-3D++ sets new records for 3D-aware image synthesis, with an impressive FID of 3.2 on FFHQ at the 1024×1024 resolution. In the meantime, CIPS-3D++ runs efficiently and enjoys a low GPU memory footprint so that it can be trained end-to-end on high-resolution images directly, in contrast to previous alternate/progressive methods. Based on the infrastructure of CIPS-3D++, we propose a 3D-aware GAN inversion algorithm named FlipInversion, which can reconstruct the 3D object from a single-view image. We also provide a 3D-aware stylization method for real images based on CIPS-3D++ and FlipInversion. In addition, we analyze the problem of mirror symmetry suffered in training, and solve it by introducing an auxiliary discriminator for the NeRF network. Overall, CIPS-3D++ provides a strong base model that can serve as a testbed for transferring GAN-based image editing methods from 2D to 3D.

Histone H3K9 acetylation modulates gene expression of key enzymes in the flavonoid and abscisic acid pathways and enhances drought resistance of sea buckthorn.

The effect of histone H3K9 acetylation modification on gene expression and drought resistance in drought-resistant tree species is not clear. Using the chromatin immunoprecipitation (ChIP) method, this study obtained nine H3K9 acetylated protein-interacting DNAs from sea buckthorn seedlings, and the ChIP sequencing result predicted about 56,591, 2217 and 5119 enriched region peaks in the control, drought and rehydration comparative groups, respectively. Gene functional analysis of differential peaks from three comparison groups revealed that 105 pathways were involved in the drought resistance process, and 474 genes were enriched in the plant hormone signaling transduction pathways. Combined ChIP-seq and transcriptome analysis revealed that six genes related to abscisic acid synthesis and signaling pathways, 17 genes involved in flavonoid biosynthesis, and 15 genes involved in carotenoid biosynthesis were positively regulated by H3K9 acetylation modification under drought stress. Under drought stress conditions, the content of abscisic acid and the expression of related genes were significantly up-regulated, while the content of flavonoids and the expression of key enzymes involved in their synthesis were largely down-regulated. Meanwhile, after exposure to histone deacetylase inhibitors (trichostatin A), the change of abscisic acid and flavonoids content and their related gene expression were slowed down under drought stress. This study will provide an important theoretical basis for understanding the regulatory mechanisms of histone acetylation modifications in sea buckthorn drought resistance.

Learning by Restoring Broken 3D Geometry.

The key point for an experienced craftsman to repair broken objects effectively is that he must know about them deeply. Similarly, we believe that a model can capture rich geometry information from a shape/scene and generate discriminative representations if it is able to find distorted parts of shapes/scenes and restore them. Inspired by this observation, we propose a novel self-supervised 3D learning paradigm named learning by restoring broken shapes/scenes (collectively called 3D geometry). We first develop a destroy-method cluster, from which we sample methods to break some local parts of an object. Then the destroyed object and the normal object are both sent into a point cloud network to get representations, which are employed to segment points that belong to distorted parts and further reconstruct/restore them to normal. To perform better in these two associated pretext tasks, the model is constrained to capture useful object features, such as rich geometric and contextual information. The object representations learned by this self-supervised paradigm transfer well to different datasets and perform well on downstream classification, segmentation and detection tasks. Experimental results on shape datasets and scene datasets demonstrate that our method achieves state-of-the-art performance among unsupervised methods. We also show experimentally that pre-training with our framework significantly boosts the performance of supervised models.

MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification.

We introduce MedMNIST v2, a large-scale MNIST-like dataset collection of standardized biomedical images, including 12 datasets for 2D and 6 datasets for 3D. All images are pre-processed into a small size of 28 × 28 (2D) or 28 × 28 × 28 (3D) with the corresponding classification labels so that no background knowledge is required for users. Covering primary data modalities in biomedical images, MedMNIST v2 is designed to perform classification on lightweight 2D and 3D images with various dataset scales (from 100 to 100,000) and diverse tasks (binary/multi-class, ordinal regression, and multi-label). The resulting dataset, consisting of 708,069 2D images and 9,998 3D images in total, could support numerous research/educational purposes in biomedical image analysis, computer vision, and machine learning. We benchmark several baseline methods on MedMNIST v2, including 2D/3D neural networks and open-source/commercial AutoML tools. The data and code are publicly available at https://medmnist.com/ .

Out-of-Domain Human Mesh Reconstruction via Dynamic Bilevel Online Adaptation.

We consider a new problem of adapting a human mesh reconstruction model to out-of-domain streaming videos, where the performance of existing SMPL-based models is significantly affected by the distribution shift represented by different camera parameters, bone lengths, backgrounds, and occlusions. We tackle this problem through online adaptation, gradually correcting the model bias during testing. There are two main challenges: First, the lack of 3D annotations increases the training difficulty and results in 3D ambiguities. Second, non-stationary data distribution makes it difficult to strike a balance between fitting regular frames and hard samples with severe occlusions or dramatic changes. To this end, we propose the Dynamic Bilevel Online Adaptation algorithm (DynaBOA). It first introduces the temporal constraints to compensate for the unavailable 3D annotations and leverages a bilevel optimization procedure to address the conflicts between multi-objectives. DynaBOA provides additional 3D guidance by co-training with similar source examples retrieved efficiently despite the distribution shift. Furthermore, it can adaptively adjust the number of optimization steps on individual frames to fully fit hard samples and avoid overfitting regular frames. DynaBOA achieves state-of-the-art results on three out-of-domain human mesh reconstruction benchmarks.

Learning Multi-Attention Context Graph for Group-Based Re-Identification.

Learning to re-identify or retrieve a group of people across non-overlapped camera systems has important applications in video surveillance. However, most existing methods focus on (single) person re-identification (re-id), ignoring the fact that people often walk in groups in real scenarios. In this work, we take a step further and consider employing context information for identifying groups of people, i.e., group re-id. On the one hand, group re-id is more challenging than single person re-id, since it requires both a robust modeling of local individual person appearance (with different illumination conditions, pose/viewpoint variations, and occlusions), as well as full awareness of global group structures (with group layout and group member variations). On the other hand, we believe that person re-id can be greatly enhanced by incorporating additional visual context from neighboring group members, a task which we formulate as group-aware (single) person re-id. In this paper, we propose a novel unified framework based on graph neural networks to simultaneously address the above two group-based re-id tasks, i.e., group re-id and group-aware person re-id. Specifically, we construct a context graph with group members as its nodes to exploit dependencies among different people. A multi-level attention mechanism is developed to formulate both intra-group and inter-group context, with an additional self-attention module for robust graph-level representations by attentively aggregating node-level features. The proposed model can be directly generalized to tackle group-aware person re-id using node-level representations. Meanwhile, to facilitate the deployment of deep learning models on these tasks, we build a new group re-id dataset which contains more than 3.8K images with 1.5K annotated groups, an order of magnitude larger than existing group re-id datasets. Extensive experiments on the novel dataset as well as three existing datasets clearly demonstrate the effectiveness of the proposed framework for both group-based re-id tasks.

Learning Multi-View Interactional Skeleton Graph for Action Recognition.

Capturing the interactions of human articulations lies in the center of skeleton-based action recognition. Recent graph-based methods are inherently limited in the weak spatial context modeling capability due to fixed interaction pattern and inflexible shared weights of GCN. To address above problems, we propose the multi-view interactional graph network (MV-IGNet) which can construct, learn and infer multi-level spatial skeleton context, including view-level (global), group-level, joint-level (local) context, in a unified way. MV-IGNet leverages different skeleton topologies as multi-views to cooperatively generate complementary action features. For each view, separable parametric graph convolution (SPG-Conv) enables multiple parameterized graphs to enrich local interaction patterns, which provides strong graph-adaption ability to handle irregular skeleton topologies. We also partition the skeleton into several groups and then the higher-level group contexts including inter-group and intra-group, are hierarchically captured by above SPG-Conv layers. A simple yet effective global context adaption (GCA) module facilitates representative feature extraction by learning the input-dependent skeleton topologies. Compared to the mainstream works, MV-IGNet can be readily implemented while with smaller model size and faster inference. Experimental results show the proposed MV-IGNet achieves impressive performance on large-scale benchmarks: NTU-RGB+D and NTU-RGB+D 120.

Genopathomic profiling identifies signatures for immunotherapy response of lung adenocarcinoma via confounder-aware representation learning.

Immunotherapy shows durable response but only in a subset of patients, and test for predictive biomarkers requires procedures in addition to routine workflow. We proposed a confounder-aware representation learning-based system, genopathomic biomarker for immunotherapy response (PITER), that uses only diagnosis-acquired hematoxylin-eosin (H&E)-stained pathological slides by leveraging histopathological and genetic characteristics to identify candidates for immunotherapy. PITER was generated and tested with three datasets containing 1944 slides of 1239 patients. PITER was found to be a useful biomarker to identify patients of lung adenocarcinoma with both favorable progression-free and overall survival in the immunotherapy cohort (p < 0.05). PITER was significantly associated with pathways involved in active cell division and a more immune activating microenvironment, which indicated the biological basis in identifying patients with favorable outcome of immunotherapy. Thus, PITER may be a potential biomarker to identify patients of lung adenocarcinoma with a good response to immunotherapy, and potentially provide precise treatment.

ViSTA: A Novel Network Improving Lung Adenocarcinoma Invasiveness Prediction from Follow-Up CT Series.

To investigate the value of the deep learning method in predicting the invasiveness of early lung adenocarcinoma based on irregularly sampled follow-up computed tomography (CT) scans. In total, 351 nodules were enrolled in the study. A new deep learning network based on temporal attention, named Visual Simple Temporal Attention (ViSTA), was proposed to process irregularly sampled follow-up CT scans. We conducted substantial experiments to investigate the supplemental value in predicting the invasiveness using serial CTs. A test set composed of 69 lung nodules was reviewed by three radiologists. The performance of the model and radiologists were compared and analyzed. We also performed a visual investigation to explore the inherent growth pattern of the early adenocarcinomas. Among counterpart models, ViSTA showed the best performance (AUC: 86.4% vs. 60.6%, 75.9%, 66.9%, 73.9%, 76.5%, 78.3%). ViSTA also outperformed the model based on Volume Doubling Time (AUC: 60.6%). ViSTA scored higher than two junior radiologists (accuracy of 81.2% vs. 75.4% and 71.0%) and came close to the senior radiologist (85.5%). Our proposed model using irregularly sampled follow-up CT scans achieved promising accuracy in evaluating the invasiveness of the early stage lung adenocarcinoma. Its performance is comparable with senior experts and better than junior experts and traditional deep learning models. With further validation, it can potentially be applied in clinical practice.

GMILT: A Novel Transformer Network That Can Noninvasively Predict EGFR Mutation Status.

Noninvasively and accurately predicting the epidermal growth factor receptor (EGFR) mutation status is a clinically vital problem. Moreover, further identifying the most suspicious area related to the EGFR mutation status can guide the biopsy to avoid false negatives. Deep learning methods based on computed tomography (CT) images may improve the noninvasive prediction of EGFR mutation status and potentially help clinicians guide biopsies by visual methods. Inspired by the potential inherent links between EGFR mutation status and invasiveness information, we hypothesized that the predictive performance of a deep learning network can be improved through extra utilization of the invasiveness information. Here, we created a novel explainable transformer network for EGFR classification named gated multiple instance learning transformer (GMILT) by integrating multi-instance learning and discriminative weakly supervised feature learning. Pathological invasiveness information was first introduced into the multitask model as embeddings. GMILT was trained and validated on a total of 512 patients with adenocarcinoma and tested on three datasets (the internal test dataset, the external test dataset, and The Cancer Imaging Archive (TCIA) public dataset). The performance (area under the curve (AUC) = 0.772 on the internal test dataset) of GMILT exceeded that of previously published methods and radiomics-based methods (i.e., random forest and support vector machine) and attained a preferable generalization ability (AUC = 0.856 in the TCIA test dataset and AUC = 0.756 in the external dataset). A diameter-based subgroup analysis further verified the efficiency of our model (most of the AUCs exceeded 0.772) to noninvasively predict EGFR mutation status from computed tomography (CT) images. In addition, because our method also identified the "core area" of the most suspicious area related to the EGFR mutation status, it has the potential ability to guide biopsies.

Graphical Modeling for Multi-Source Domain Adaptation.

Multi-Source Domain Adaptation (MSDA) focuses on transferring the knowledge from multiple source domains to the target domain, which is a more practical and challenging problem compared to the conventional single-source domain adaptation. In this problem, it is essential to model multiple source domains and target domain jointly, and an effective domain combination scheme is also highly required. The graphical structure among different domains is useful to tackle these challenges, in which the interdependency among various instances/categories can be effectively modeled. In this work, we propose two types of graphical models, i.e. Conditional Random Field for MSDA (CRF-MSDA) and Markov Random Field for MSDA (MRF-MSDA), for cross-domain joint modeling and learnable domain combination. In a nutshell, given an observation set composed of a query sample and the semantic prototypes (i.e. representative category embeddings) on various domains, the CRF-MSDA model seeks to learn the joint distribution of labels conditioned on the observations. We attain this goal by constructing a relational graph with all observations and conducting local message passing on it.

Prediction of future imagery of lung nodule as growth modeling with follow-up computed tomography scans using deep learning: a retrospective cohort study.

Background: Risk prediction models of lung nodules have been built to alleviate the heavy interpretative burden on clinicians. However, the malignancy scores output by those models can be difficult to interpret in a clinically meaningful manner. In contrast, the modeling of lung nodule growth may be more readily useful. This study developed a CT-based visual forecasting system that can visualize and quantify a nodule in three dimensions (3D) in any future time point using follow-up CT scans. Methods: We retrospectively included 246 patients with 313 lung nodules with at least 1 follow-up CT scan. For the manually segmented nodules, we calculated geometric properties including CT value, diameter, volume, and mass, as well as growth properties including volume doubling time (VDT), and consolidation-to-tumor ratio (CTR) at follow-ups. These nodules were divided into growth and non-growth groups by thresholding their VDTs. We then developed a convolutional neural network (CNN) to model the imagery change of the nodules from baseline CT image (combined with the nodule mask) to follow-up CT image with a particular time interval. The model was evaluated on the geometric and radiological properties using either logistic regression or receiver operating characteristic (ROC) curve. Results: The lung nodules consisted of 115 ground glass nodules (GGN) and 198 solid nodules and were followed up for an average of 354 days with 2 to 11 scans. The 2 groups differed significantly in most properties. The prediction of our forecasting system was highly correlated with the ground truth with small relative errors regarding the four geometric properties. The prediction-derived VDTs had an area under the curve (AUC) of 0.857 and 0.843 in differentiating growth and non-growth nodules for GGN and solid nodules, respectively. The prediction-derived CTRs had an AUC of 0.892 in classifying high- and low-risk nodules. Conclusions: This proof-of-concept study demonstrated that the deep learning-based model can accurately forecast the imagery of a nodule in a given future for both GGNs and solid nodules and is worthy of further investigation. With a larger dataset and more validation, such a system has the potential to become a prognostication tool for assessing lung nodules.

Fine-Grained Video Captioning via Graph-based Multi-Granularity Interaction Learning.

Learning to generate continuous linguistic descriptions for multi-subject interactive videos in great details has particular applications in team sports auto-narrative. In contrast to traditional video caption, this task is more challenging as it requires simultaneous modeling of fine-grained individual actions, uncovering of spatio-temporal dependency structures of frequent group interactions, and then accurate mapping of these complex interaction details into long and detailed commentary. To explicitly address these challenges, we propose a novel framework Graph-based Learning for Multi-Granularity Interaction Representation (GLMGIR) for fine-grained team sports auto-narrative task. A multi-granular interaction modeling module is proposed to extract among-subjects' interactive actions in a progressive way for encoding both intra- and inter-team interactions. Based on the above multi-granular representations, a multi-granular attention module is developed to consider action/event descriptions of multiple spatio-temporal resolutions. Both modules are integrated seamlessly and work in a collaborative way to generate the final narrative. In the meantime, to facilitate reproducible research, we collect a new video dataset from YouTube.com called Sports Video Narrative dataset (SVN). It is a novel direction as it contains 6K team sports videos (i.e., NBA basketball games) with 10K ground-truth narratives(e.g., sentences). Furthermore, as previous metrics such as METEOR (i.e., used in coarse-grained video caption task) DO NOT cope with fine-grained sports narrative task well, we hence develop a novel evaluation metric named Fine-grained Captioning Evaluation (FCE), which measures how accurate the generated linguistic description reflects fine-grained action details as well as the overall spatio-temporal interactional structure. Extensive experiments on our SVN dataset have demonstrated the effectiveness of the proposed framework for fine-grained team sports video auto-narrative.

A multi-omics-based serial deep learning approach to predict clinical outcomes of single-agent anti-PD-1/PD-L1 immunotherapy in advanced stage non-small-cell lung cancer.

Only 20% NSCLC patients benefit from immunotherapy with a durable response. Current biomarkers are limited by the availability of samples and do not accurately predict who will benefit from immunotherapy. To develop a unified deep learning model to integrate multimodal serial information from CT with laboratory and baseline clinical information. We retrospectively analyzed 1633 CT scans and 3414 blood samples from 200 advanced stage NSCLC patients who received single anti-PD-1/PD-L1 agent between April 2016 and December 2019. Multidimensional information, including serial radiomics, laboratory data and baseline clinical data, was used to develop and validate deep learning models to identify immunotherapy responders and nonresponders. A Simple Temporal Attention (SimTA) module was developed to process asynchronous time-series imaging and laboratory data. Using cross-validation, the 90-day deep learning-based predicting model showed a good performance in distinguishing responders from nonresponders, with an area under the curve (AUC) of 0.80 (95% CI: 0.74-0.86). Before immunotherapy, we stratified the patients into high- and low-risk nonresponders using the model. The low-risk group had significantly longer progression-free survival (PFS) (8.4 months, 95% CI: 5.49-11.31 vs. 1.5 months, 95% CI: 1.29-1.71; HR 3.14, 95% CI: 2.27-4.33; log-rank test, P<0.01) and overall survival (OS) (26.7 months, 95% CI: 18.76-34.64 vs. 8.6 months, 95% CI: 4.55-12.65; HR 2.46, 95% CI: 1.73-3.51; log-rank test, P<0.01) than the high-risk group. An exploratory analysis of 93 patients with stable disease (SD) [after first efficacy assessment according to the Response Evaluation Criteria in Solid Tumors (RECIST) 1.1] also showed that the 90-day model had a good prediction of survival and low-risk patients had significantly longer PFS (11.1 months, 95% CI: 10.24-11.96 vs. 3.3 months, 95% CI: 0.34-6.26; HR 2.93, 95% CI: 1.69-5.10; log-rank test, P<0.01) and OS (31.7 months, 95% CI: 23.64-39.76 vs. 17.2 months, 95% CI: 7.22-27.18; HR 2.22, 95% CI: 1.17-4.20; log-rank test, P=0.01) than high-risk patients. In conclusion, the SimTA-based multi-omics serial deep learning provides a promising methodology for predicting response of advanced NSCLC patients to anti-PD-1/PD-L1 monotherapy. Moreover, our model could better differentiate survival benefit among SD patients than the traditional RECIST evaluation method.

Reinventing 2D Convolutions for 3D Images.

There have been considerable debates over 2D and 3D representation learning on 3D medical images. 2D approaches could benefit from large-scale 2D pretraining, whereas they are generally weak in capturing large 3D contexts. 3D approaches are natively strong in 3D contexts, however few publicly available 3D medical dataset is large and diverse enough for universal 3D pretraining. Even for hybrid (2D + 3D) approaches, the intrinsic disadvantages within the 2D/3D parts still exist. In this study, we bridge the gap between 2D and 3D convolutions by reinventing the 2D convolutions. We propose ACS (axial-coronal-sagittal) convolutions to perform natively 3D representation learning, while utilizing the pretrained weights on 2D datasets. In ACS convolutions, 2D convolution kernels are split by channel into three parts, and convoluted separately on the three views (axial, coronal and sagittal) of 3D representations. Theoretically, ANY 2D CNN (ResNet, DenseNet, or DeepLab) is able to be converted into a 3D ACS CNN, with pretrained weight of a same parameter size. Extensive experiments validate the consistent superiority of the pretrained ACS CNNs, over the 2D/3D CNN counterparts with/without pretraining. Even without pretraining, the ACS convolution can be used as a plug-and-play replacement of standard 3D convolution, with smaller model size and less computation.