Bilateral Attention Network for RGB-D Salient Object Detection.

RGB-D salient object detection (SOD) aims to segment the most attractive objects in a pair of cross-modal RGB and depth images. Currently, most existing RGB-D SOD methods focus on the foreground region when utilizing the depth images. However, the background also provides important information in traditional SOD methods for promising performance. To better explore salient information in both foreground and background regions, this paper proposes a Bilateral Attention Network (BiANet) for the RGB-D SOD task. Specifically, we introduce a Bilateral Attention Module (BAM) with a complementary attention mechanism: foreground-first (FF) attention and background-first (BF) attention. The FF attention focuses on the foreground region with a gradual refinement style, while the BF one recovers potentially useful salient information in the background region. Benefited from the proposed BAM module, our BiANet can capture more meaningful foreground and background cues, and shift more attention to refining the uncertain details between foreground and background regions. Additionally, we extend our BAM by leveraging the multi-scale techniques for better SOD performance. Extensive experiments on six benchmark datasets demonstrate that our BiANet outperforms other state-of-the-art RGB-D SOD methods in terms of objective metrics and subjective visual comparison. Our BiANet can run up to 80 fps on 224×224 RGB-D images, with an NVIDIA GeForce RTX 2080Ti GPU. Comprehensive ablation studies also validate our contributions.

Low-Rank Constrained Super-Resolution for Mixed-Resolution Multiview Video.

Multiview video allows for simultaneously presenting dynamic imaging from multiple viewpoints, enabling a broad range of immersive applications. This paper proposes a novel super-resolution (SR) approach to mixed-resolution (MR) multiview video, whereby the low-resolution (LR) videos produced by MR camera setups are up-sampled based on the neighboring HR videos. Our solution analyzes the statistical correlation of different resolutions between multiple views, and introduces a low-rank prior based SR optimization framework using local linear embedding and weighted nuclear norm minimization. The target HR patch is reconstructed by learning texture details from the neighboring HR camera views using local linear embedding. A low-rank constrained patch optimization solution is introduced to effectively restrain visual artifacts and the ADMM framework is used to solve the resulting optimization problem. Comprehensive experiments including objective and subjective test metrics demonstrate that the proposed method outperforms the state-of-the-art SR methods for MR multiview video.

Structure-Preserving Neural Style Transfer.

State-of-the-art neural style transfer methods have demonstrated amazing results by training feed-forward convolutional neural networks or using an iterative optimization strategy. The image representation used in these methods, which contains two components: style representation and content representation, is typically based on high-level features extracted from pretrained classification networks. Because the classification networks are originally designed for object recognition, the extracted features often focus on the central object and neglect other details. As a result, the style textures tend to scatter over the stylized outputs and disrupt the content structures. To address this issue, we present a novel image stylization method that involves an additional structure representation. Our structure representation, which considers two factors: i) the global structure represented by the depth map and ii) the local structure details represented by the image edges, effectively reflects the spatial distribution of all the components in an image as well as the structure of dominant objects respectively. Experimental results demonstrate that our method achieves an impressive visual effectiveness, which is particularly significant when processing images sensitive to structure distortion, e.g. images containing multiple objects potentially at different depths, or dominant objects with clear structures.

Deep Online Video Stabilization with Multi-Grid Warping Transformation Learning.

Video stabilization techniques are essential for most hand-held captured videos due to high-frequency shakes. Several 2D, 2.5D and 3D-based stabilization techniques have been presented previously, but to our knowledge, no solutions based on deep neural networks had been proposed to date. The main reason for this omission is shortage in training data as well as the challenge of modeling the problem using neural networks. In this paper, we present a video stabilization technique using a convolutional neural network. Previous works usually propose an offline algorithm that smoothes a holistic camera path based on feature matching. Instead, we focus on low-latency, real-time camera path smoothing, that does not explicitly represent the camera path, and does not use future frames. Our neural network model, called StabNet, learns a set of mesh-grid transformations progressively for each input frame from the previous set of stabalized camera frames, and creates stable corresponding latent camera paths implicitly. To train the network, we collect a dataset of synchronized steady and unsteady video pairs via a specially designed hand-held hardware. Experimental results show that our proposed online method performs comparatively to traditional offline video stabilization methods without using future frames, while running about 10× faster. More importantly, our proposed StabNet is able to handle low-quality videos such as night-scene videos, watermarked videos, blurry videos and noisy videos, where existing methods fail in feature extraction or matching.

BiggerSelfie: Selfie Video Expansion With Hand-Held Camera.

Selfie photography from the hand-held camera is becoming a popular media type. Although being convenient and flexible, it suffers from low camera motion stability, small field of view, and limited background content. These limitations can annoy users, especially, when touring a place of interest and taking selfie videos. In this paper, we present a novel method to create what we call a BiggerSelfie that deals with these shortcomings. Using a video of the environment that has partial content overlap with the selfie video, we stitch plausible frames selected from the environment video to the original selfie frames and stabilize the composed video content with a portrait-preserving constraint. Using the proposed method, one can easily obtain a stable selfie video with expanded background content by merely capturing some background shots. We show various results and several evaluations to demonstrate the applicability of our method.

Hyper-Lapse From Multiple Spatially-Overlapping Videos.

Hyper-lapse video with high speed-up rate is an efficient way to overview long videos, such as a human activity in first-person view. Existing hyper-lapse video creation methods produce a fast-forward video effect using only one video source. In this paper, we present a novel hyper-lapse video creation approach based on multiple spatially-overlapping videos. We assume the videos share a common view or location, and find transition points where jumps from one video to another may occur. We represent the collection of videos using a hyper-lapse transition graph; the edges between nodes represent possible hyper-lapse frame transitions. To create a hyper-lapse video, a shortest path search is performed on this digraph to optimize frame sampling and assembly simultaneously. Finally, we render the hyper-lapse results using video stabilization and appearance smoothing techniques on the selected frames. Our technique can synthesize novel virtual hyper-lapse routes, which may not exist originally. We show various application results on both indoor and outdoor video collections with static scenes, moving objects, and crowds.

Timeline editing of objects in video.

We present a video editing technique based on changing the timelines of individual objects in video, which leaves them in their original places but puts them at different times. This allows the production of object-level slow motion effects, fast motion effects, or even time reversal. This is more flexible than simply applying such effects to whole frames, as new relationships between objects can be created. As we restrict object interactions to the same spatial locations as in the original video, our approach can produce highquality results using only coarse matting of video objects. Coarse matting can be done efficiently using automatic video object segmentation, avoiding tedious manual matting. To design the output, the user interactively indicates the desired new life spans of objects, and may also change the overall running time of the video. Our method rearranges the timelines of objects in the video whilst applying appropriate object interaction constraints. We demonstrate that, while this editing technique is somewhat restrictive, it still allows many interesting results.

High sensitivity plasmonic index sensor using slablike gold nanoring arrays.

We investigate the index sensing characteristics of plasmonic arrays based on square lattice slablike gold nanorings (NRs) with different ring widths. The gold NR arrays exhibit two extinction peaks in the visible and near-infrared corresponding to antibonding and bonding modes. Redshift and blueshift in antibonding and bonding modes when broadening the average ring width are observed. We experimentally demonstrate the sensitivity of bonding mode can be tuned by varying the average ring width. High sensitivity of 691 nm per refractive index unit is obtained for NRs with 199 nm average ring width.

Square lattice photonic crystal surface mode lasers.

In this report, we propose a square lattice photonic crystal hetero-slab-edge microcavity design. In numerical simulations, three surface modes in this microcavity are investigated and optimized by tuning the slab-edge termination τ and gradual mirror layer. High simulated quality (Q) factor of 2.3 × 10(5) and small mode volume of 0.105 µm(3) are obtained from microcavity with τ = 0.80. In experiments, we obtain and identify different surface modes lasing. The surface mode in the second photonic band gap shows a very-low threshold of 140 µW and high Q factor of 5,500, which could be an avenue to low-threshold optical lasers and highly sensitive sensor applications with efficient light-matter interactions.

Chemical characterization of Lycium barbarum polysaccharides and their reducing myocardial injury in ischemia/reperfusion of rat heart.

Polysaccharides were extracted from Lycium barbarum fruits in this work. Fourier transform infrared spectroscopy (FT-IR) and high-performance liquid chromatography (HPLC) have been employed to characterize this polysaccharides in the present study. The results of chemical composition indicated that the L. barbarum polysaccharides were composed of two kinds of monosaccharides, namely glucose and fructose in molar ratios of 1:2.1. The results indicated that the glucose and fructose were the predominant monosaccharides. IR spectrum of L. barbarum polysaccharides revealed a typical peaks of polysaccharides. The results still showed that L. barbarum polysaccharides significantly decreased the myocardium LD level, increased Na(+)-K(+)-ATPase and Ca(2+)-ATPase activities in heart ischemia reperfusion (IR) rats. In addition, L. barbarum polysaccharides still markedly decreased myocardium Bax positive rate and myocardial cell apoptosis and increased Bcl-2 positive rate in a dose-dependent manner. It may be concluded that administration of L. barbarum polysaccharides can prevented the development of cardiovascular diseases.