Deep Learning Methods for Calibrated Photometric Stereo and Beyond.

Photometric stereo recovers the surface normals of an object from multiple images with varying shading cues, i.e., modeling the relationship between surface orientation and intensity at each pixel. Photometric stereo prevails in superior per-pixel resolution and fine reconstruction details. However, it is a complicated problem because of the non-linear relationship caused by non-Lambertian surface reflectance. Recently, various deep learning methods have shown a powerful ability in the context of photometric stereo against non-Lambertian surfaces. This paper provides a comprehensive review of existing deep learning-based calibrated photometric stereo methods utilizing orthographic cameras and directional light sources. We first analyze these methods from different perspectives, including input processing, supervision, and network architecture. We summarize the performance of deep learning photometric stereo models on the most widely-used benchmark data set. This demonstrates the advanced performance of deep learning-based photometric stereo methods. Finally, we give suggestions and propose future research trends based on the limitations of existing models.

Manipulation of photosynthetic energy transfer by vibrational strong coupling.

Uncovering the mystery of efficient and directional energy transfer in photosynthetic organisms remains a critical challenge in quantum biology. Recent experimental evidence and quantum theory developments indicate the significance of quantum features of molecular vibrations in assisting photosynthetic energy transfer, which provides the possibility of manipulating the process by controlling molecular vibrations. Here, we propose and theoretically demonstrate efficient manipulation of photosynthetic energy transfer by using vibrational strong coupling between the vibrational state of a Fenna-Matthews-Olson (FMO) complex and the vacuum state of an optical cavity. Specifically, based on a full-quantum analytical model to describe the strong coupling effect between the optical cavity and molecular vibration, we realize efficient manipulation of energy transfer efficiency (from 58% to 92%) and energy transfer time (from 20 to 500 ps) in one branch of FMO complex by actively controlling the coupling strength and the quality factor of the optical cavity under both near-resonant and off-resonant conditions, respectively. Our work provides a practical scenario to manipulate photosynthetic energy transfer by externally interfering molecular vibrations via an optical cavity and a comprehensible conceptual framework for researching other similar systems.

YTHDF1 enhances stemness and chemoresistance in triple-negative breast cancer cells by upregulating SIAH2.

Triple-negative breast cancer (TNBC) is the most lethal and aggressive subtype of breast cancer, and chemoresistance is the major determinant of TNBC treatment failure. This study explores the molecular mechanism of TNBC chemoresistance. The Cancer Genome Atlas, breast cancer integrative platform, and GEPIA databases were used to analyze the expression and correlation of YTHDF1 and seven in absentia homology 2 (SIAH2) in breast cancer. Knockdown of YTHDF1 and SIAH2, or overexpression of SIAH2 in vitro and in vivo, was conducted to evaluate the impact of changes in YTHDF1 and SIAH2 expression on TNBC cell proliferation, apoptosis, stemness, drug resistance, and Hippo pathway gene expression. YTHDF1 and SIAH2 were highly expressed in breast cancer patients and TNBC cells. Knockdown of YTHDF1 and SIAH2 significantly inhibited proliferation and stemness and promoted apoptosis and chemosensitivity of TNBC cells. Mechanistically, the knockdown of YTHDF1 inhibited the expression of SIAH2, thereby downregulating the Hippo pathway, which inhibited proliferation and stemness and promoted apoptosis and chemosensitivity of TNBC cells. The current findings revealed the regulatory mechanism of YTHDF1 in TNBC and clarified the role of the YTHDF1/SIAH2 axis in TNBC drug resistance and stemness. This could provide new insights into the vital role of targeting YTHDF1/SIAH2 to suppress drug resistance and stemness in TNBC cells.

GR-PSN: Learning to Estimate Surface Normal and Reconstruct Photometric Stereo Images.

In this paper, we propose a novel method, namely GR-PSN, which learns surface normals from photometric stereo images and generates the photometric images under distant illumination from different lighting directions and surface materials. The framework is composed of two subnetworks, named GeometryNet and ReconstructNet, which are cascaded to perform shape reconstruction and image rendering in an end-to-end manner. ReconstructNet introduces additional supervision for surface-normal recovery, forming a closed-loop structure with GeometryNet. We also encode lighting and surface reflectance in ReconstructNet, to achieve arbitrary rendering. In training, we set up a parallel framework to simultaneously learn two arbitrary materials for an object, providing an additional transform loss. Therefore, our method is trained based on the supervision by three different loss functions, namely the surface-normal loss, reconstruction loss, and transform loss. We alternately input the predicted surface-normal map and the ground-truth into ReconstructNet, to achieve stable training for ReconstructNet. Experiments show that our method can accurately recover the surface normals of an object with an arbitrary number of inputs, and can re-render images of the object with arbitrary surface materials. Extensive experimental results show that our proposed method outperforms those methods based on a single surface recovery network and shows realistic rendering results on 100 different materials. Our code can be found in https://github.com/Kelvin-Ju/GR-PSN.

Different bacterial and fungal community patterns in restored habitats in coal-mining subsidence areas.

Soil microbiota, which plays a fundamental role in ecosystem functioning, is sensitive to environmental changes. Studying soil microbial ecological patterns can help to understand the consequences of environmental disturbances on soil microbiota and hence ecosystem services. The different habitats with critical environmental gradients generated through the restoration of coal-mining subsidence areas provide an ideal area to explore the response of soil microbiota to environmental changes. Here, based on high-throughput sequencing, we revealed the patterns of soil bacterial and fungal communities in habitats with different land-use types (wetland, farmland, and grassland) and with different restored times which were generated during the ecological restoration of a typical coal-mining subsidence area in Jining City, China. The α-diversity of bacterial was higher in wetland than in farmland and grassland, while that of fungi had no discrepancy among the three habitats. The ß-diversity of bacterial community in the grassland was lower than in the farmland, and fungal community was significant different in all three habitats, showing wetland, grassland, and farmland from high to low. The ß-diversity of the bacterial community decreased with restoration time while that of the fungal community had no significant change in the longer-restoration-time area. Furthermore, soil electrical conductivity was the most important driver for both bacterial and fungal communities. Based on the taxonomic difference among different habitats, we identified a group of biomarkers for each habitat. The study contributes to understand the microbial patterns during the ecological restoration of coal-mining subsidence areas, which has implications for the efficient ecological restoration of subsidence areas.

Double-Glued Multi-Focal Bionic Compound Eye Camera.

Compound eye cameras are a vital component of bionics. Compound eye lenses are currently used in light field cameras, monitoring imaging, medical endoscopes, and other fields. However, the resolution of the compound eye lens is still low at the moment, which has an impact on the application scene. Photolithography and negative pressure molding were used to create a double-glued multi-focal bionic compound eye camera in this study. The compound eye camera has 83 microlenses, with ommatidium diameters ranging from 400 µm to 660 µm, and a 92.3 degree field-of-view angle. The double-gluing structure significantly improves the optical performance of the compound eye lens, and the spatial resolution of the ommatidium is 57.00 lp mm-1. Additionally, the measurement of speed is investigated. This double-glue compound eye camera has numerous potential applications in the military, machine vision, and other fields.

HGR-Net: Hierarchical Graph Reasoning Network for Arbitrary Shape Scene Text Detection.

As a prerequisite step of scene text reading, scene text detection is known as a challenging task due to natural scene text diversity and variability. Most existing methods either adopt bottom-up sub-text component extraction or focus on top-down text contour regression. From a hybrid perspective, we explore hierarchical text instance-level and component-level representation for arbitrarily-shaped scene text detection. In this work, we propose a novel Hierarchical Graph Reasoning Network (HGR-Net), which consists of a Text Feature Extraction Network (TFEN) and a Text Relation Learner Network (TRLN). TFEN adaptively learns multi-grained text candidates based on shared convolutional feature maps, including instance-level text contours and component-level quadrangles. In TRLN, an inter-text graph is constructed to explore global contextual information with position-awareness between text instances, and an intra-text graph is designed to estimate geometric attributes for establishing component-level linkages. Next, we bridge the cross-feed interaction between instance-level and component-level, and it further achieves hierarchical relational reasoning by learning complementary graph embeddings across levels. Experiments conducted on three publicly available benchmarks SCUT-CTW1500, Total-Text, and ICDAR15 have demonstrated that HGR-Net achieves state-of-the-art performance on arbitrary orientation and arbitrary shape scene text detection.

MobileSky: Real-Time Sky Replacement for Mobile AR.

We present MobileSky, the first automatic method for real-time high-quality sky replacement for mobile AR applications. The primary challenge of this task is how to extract sky regions in camera feed both quickly and accurately. While the problem of sky replacement is not new, previous methods mainly concern extraction quality rather than efficiency, limiting their application to our task. We aim to provide higher quality, both spatially and temporally consistent sky mask maps for all camera frames in real time. To this end, we develop a novel framework that combines a new deep semantic network called FSNet with novel post-processing refinement steps. By leveraging IMU data, we also propose new sky-aware constraints such as temporal consistency, position consistency, and color consistency to help refine the weakly classified part of the segmentation output. Experiments show that our method achieves an average of around 30 FPS on off-the-shelf smartphones and outperforms the state-of-the-art sky replacement methods in terms of execution speed and quality. In the meantime, our mask maps appear to be visually more stable across frames. Our fast sky replacement method enables several applications, such as AR advertising, art making, generating fantasy celestial objects, visually learning about weather phenomena, and advanced video-based visual effects. To facilitate future research, we also create a new video dataset containing annotated sky regions with IMU data.

A Meniscus Multifocusing Compound Eye Camera Based on Negative Pressure Forming Technology.

To meet the challenge of preparing a high-resolution compound eye, this paper proposes a multi-focal-length meniscus compound eye based on MEMS negative pressure molding technology. The aperture is increased, a large field of view angle of 101.14° is obtained, and the ommatidia radius of each stage is gradually increased from 250 µm to 440 µm. A meniscus structure is used to improve the imaging quality of the marginal compound eye so that its resolution can reach 36.00 lp/mm. The prepared microlenses have a uniform shape and a smooth surface, and both panoramic image stitching and moving object tracking are achieved. This technology has great potential for application in many fields, including automatic driving, machine vision, and medical endoscopy.

Habitats generated by the restoration of coal mining subsidence land differentially alter the content and composition of soil organic carbon.

The content and composition of soil organic carbon (SOC) can characterize soil carbon storage capacity, which varies significantly between habitats. Ecological restoration in coal mining subsidence land forms a variety of habitats, which are ideal to study the effects of habitats on SOC storage capacity. Based on the analysis of the content and composition of SOC in three habitats (farmland, wetland and lakeside grassland) generated by different restoration time of the farmland which was destroyed by coal mining subsidence, we found that farmland had the highest SOC storage capacity among the three habitats. Both dissolved organic carbon (DOC) and heavy fraction organic carbon (HFOC) exhibited higher concentrations in the farmland (20.29 mg/kg, 6.96 mg/g) than in the wetland (19.62 mg/kg, 2.47 mg/g) or lakeside grassland (5.68 mg/kg, 2.31 mg/g), and the concentrations increased significantly over time, owing to the higher content of nitrogen in the farmland. The wetland and lakeside grassland needed more time than the farmland to recover the SOC storage capacity. The findings illustrate that the SOC storage capacity of farmland destroyed by coal mining subsidence could be restored through ecological restoration and indicate that the recovery rate depends on the reconstructed habitat types, among which farmland shows great advantages mainly due to the nitrogen addition.

Physics-Guided Generative Adversarial Networks for Sea Subsurface Temperature Prediction.

Sea subsurface temperature, an essential component of aquatic wildlife, underwater dynamics, and heat transfer with the sea surface, is affected by global warming in climate change. Existing research is commonly based on either physics-based numerical models or data-based models. Physical modeling and machine learning are traditionally considered as two unrelated fields for the sea subsurface temperature prediction task, with very different scientific paradigms (physics-driven and data-driven). However, we believe that both methods are complementary to each other. Physical modeling methods can offer the potential for extrapolation beyond observational conditions, while data-driven methods are flexible in adapting to data and are capable of detecting unexpected patterns. The combination of both approaches is very attractive and offers potential performance improvement. In this article, we propose a novel framework based on a generative adversarial network (GAN) combined with a numerical model to predict sea subsurface temperature. First, a GAN-based model is used to learn the simplified physics between the surface temperature and the target subsurface temperature in the numerical model. Then, observation data are used to calibrate the GAN-based model parameters to obtain a better prediction. We evaluate the proposed framework by predicting daily sea subsurface temperature in the South China Sea. Extensive experiments demonstrate the effectiveness of the proposed framework compared to existing state-of-the-art methods.

Transformer for Image Harmonization and Beyond.

Image harmonization, aiming to make composite images look more realistic, is an important and challenging task. The composite, synthesized by combining foreground from one image with background from another image, inevitably suffers from the issue of inharmonious appearance caused by distinct imaging conditions, i.e., lights. Current solutions mainly adopt an encoder-decoder architecture with convolutional neural network (CNN) to capture the context of composite images, trying to understand what it should look like in the foreground referring to surrounding background. In this work, we seek to solve image harmonization with Transformer, by leveraging its powerful ability of modeling long-range context dependencies, for adjusting foreground light to make it compatible with background light while keeping structure and semantics unchanged. We present the design of our two vision Transformer frameworks and corresponding methods, as well as comprehensive experiments and empirical study, demonstrating the power of Transformer and investigating the Transformer for vision. Our methods achieve state-of-the-art performance on the image harmonization as well as four additional vision and graphics tasks, i.e., image enhancement, image inpainting, white-balance editing, and portrait relighting, indicating the superiority of our work. Code, models, more results and details can be found at the project website http://ouc.ai/project/HarmonyTransformer.

Arbitrary-Scale Texture Generation From Coarse-Grained Control.

Existing deep-network based texture synthesis approaches all focus on fine-grained control of texture generation by synthesizing images from exemplars. Since the networks employed by most of these methods are always tied to individual exemplar textures, a large number of individual networks have to be trained when modeling various textures. In this paper, we propose to generate textures directly from coarse-grained control or high-level guidance, such as texture categories, perceptual attributes and semantic descriptions. We fulfill the task by parsing the generation process of a texture into the three-level Bayesian hierarchical model. A coarse-grained signal first determines a distribution over Markov random fields. Then a Markov random field is used to model the distribution of the final output textures. Finally, an output texture is generated from the sampled Markov random field distribution. At the bottom level of the Bayesian hierarchy, the isotropic and ergodic characteristics of the textures favor a construction that consists of a fully convolutional network. The proposed method integrates texture creation and texture synthesis into one pipeline for real-time texture generation, and enables users to readily obtain diverse textures with arbitrary scales from high-level guidance only. Extensive experiments demonstrate that the proposed method is capable of generating plausible textures that are faithful to user-defined control, and achieving impressive texture metamorphosis by interpolation in the learned texture manifold.

A global assessment of the mixed layer in coastal sediments and implications for carbon storage.

The sediment-water interface in the coastal ocean is a highly dynamic zone controlling biogeochemical fluxes of greenhouse gases, nutrients, and metals. Processes in the sediment mixed layer (SML) control the transfer and reactivity of both particulate and dissolved matter in coastal interfaces. Here we map the global distribution of the coastal SML based on excess 210Pb (210Pbex) profiles and then use a neural network model to upscale these observations. We show that highly dynamic regions such as large estuaries have thicker SMLs than most oceanic sediments. Organic carbon preservation and SMLs are inversely related as mixing stimulates oxidation in sediments which enhances organic matter decomposition. Sites with SML thickness >60 cm usually have lower organic carbon accumulation rates (<50 g C m-2 yr-1) and total organic carbon/specific surface area ratios (<0.4 mg m-2). Our global scale observations reveal that reworking can accelerate organic matter degradation and reduce carbon storage in coastal sediments.

Face image-sketch synthesis via generative adversarial fusion.

Face image-sketch synthesis is widely applied in law enforcement and digital entertainment fields. Despite the extensive progression in face image-sketch synthesis, there are few methods focusing on generating a color face image from a sketch. The existing methods pay less attention to learning the illumination or highlight distribution on the face region. However, the illumination is the key factor that makes the generated color face image looks vivid and realistic. Moreover, existing methods tend to employ some image preprocessing technologies and facial region patching approaches to generate high-quality face images, which results in the high complexity and memory consumption in practice. In this paper, we propose a novel end-to-end generative adversarial fusion model, called GAF, which fuses two U-Net generators and a discriminator by jointly learning the content and adversarial loss functions. In particular, we propose a parametric tanh activation function to learn and control illumination highlight distribution over faces, which is integrated between the two U-Net generators by an illumination distribution layer. Additionally, we fuse the attention mechanism into the second U-Net generator of GAF to keep the identity consistency and refine the generated facial details. The qualitative and quantitative experiments on the public benchmark datasets show that the proposed GAF has better performance than existing image-sketch synthesis methods in synthesized face image quality (FSIM) and face recognition accuracy (NLDA). Meanwhile, the good generalization ability of GAF has also been verified. To further demonstrate the reliability and authenticity of face images generated using GAF, we use the generated face image to attack the well-known face recognition system. The result shows that the face images generated by GAF can maintain identity consistency and well maintain everyone's unique facial characteristics, which can be further used in the benchmark of facial spoofing. Moreover, the experiments are implemented to verify the effectiveness and rationality of the proposed parametric tanh activation function and attention mechanism in GAF.

Factors controlling soil organic carbon content in wetlands at multiple scales and assessment of the universality of estimation equations: A mega-data study.

The factors controlling soil organic carbon (SOC) content in wetlands need to be identified to estimate the global stores of SOC. Although there have been a large number of small-scale studies of the local patterns of SOC content, global studies are still required. We used a random forest algorithm and other statistical approaches to determine the controls on the SOC content in wetlands at global, continental, and national scales based on the Harmonized World Soil Database and field data. The results showed that, at the three scales explored, the soil cation exchange capacity and bulk density were the main controls on the SOC content in wetlands. Moreover, equations for estimating global SOC content were established. To assess the universality of SOC content estimation equations, the soil properties were considered as a "community" and the normalized stochasticity ratio (NST) was used to assess the stochasticity in the assembly of soil "communities". The results showed that, globally, the interaction of these factors was stochastic in the "community" composed of the controllers and SOC. The reason for this result might be that microbes were not considered in the equation. Therefore, the weighted abundance of related microbes (WARM) was therefore recommended in the estimation of SOC. With NST and WARM factors, we found that microbes play a key role in increasing the determinacy of SOC estimation equations in wetlands with less anthropogenic contamination. Our findings show that when microbial impacts are taken into account, the patterns of SOC content in pristine wetlands are more universal. Our newly established equations for estimating global SOC content are crucial in projecting changes in wetland SOC, and the two factors indicated in this study favor the universality for SOC content estimation.

Catalytic Polymerization of Phthalonitrile Resins by Carborane with Enhanced Thermal Oxidation Resistance: Experimental and Molecular Simulation.

Biphenyl phthalonitrile (BPh) resins with good thermal and thermo-oxidative stability demonstrate great application potential in aerospace and national defense industries. However, BPh monomer has a high melting point, poor solubility, slow curing speed and high curing temperature. It is difficult to control the polymerization process to obtain the resins with high performance. Here, a BPh prepolymer (BPh-Q) was prepared by reacting 1,7-bis(hydroxymethyl)-m-carborane (QCB) with BPh monomers. The BPh-Q exhibited much better solubility, faster curing speed and lower curing temperature compared with pure BPh and BPh modified with bisphenol A (BPh-B, a common prepolymer of BPh). Thus, the polymerization process of BPh was greatly accelerated at a low temperature, resulting in a BPh resin with enhanced thermostability and oxidation resistance. The experimental and theoretical models revealed the promotion effect of B-H bond on the curing reaction of phthalonitrile via Markovnikov addition reaction due to the special steric structure of carborane. This study provided an efficient method to obtain low-temperature curing phthalonitrile resins with high thermal and thermo-oxidative resistance, which would be potentially useful for the preparation of high-performance cyanide resin-based composites.

Random Shapley Forests: Cooperative Game-Based Random Forests With Consistency.

The original random forests (RFs) algorithm has been widely used and has achieved excellent performance for the classification and regression tasks. However, the research on the theory of RFs lags far behind its applications. In this article, to narrow the gap between the applications and the theory of RFs, we propose a new RFs algorithm, called random Shapley forests (RSFs), based on the Shapley value. The Shapley value is one of the well-known solutions in the cooperative game, which can fairly assess the power of each player in a game. In the construction of RSFs, RSFs use the Shapley value to evaluate the importance of each feature at each tree node by computing the dependency among the possible feature coalitions. In particular, inspired by the existing consistency theory, we have proved the consistency of the proposed RFs algorithm. Moreover, to verify the effectiveness of the proposed algorithm, experiments on eight UCI benchmark datasets and four real-world datasets have been conducted. The results show that RSFs perform better than or at least comparable with the existing consistent RFs, the original RFs, and a classic classifier, support vector machines.