Infrared simulation of large-scale urban scene through LOD.

The growing use of infrared (IR) imaging systems places increasing demands for simulating infrared images of real scenes. Utilizing images captured from unmanned aerial vehicles (UAV), we propose a semi-automatic pipeline to generate large-scale IR urban scenes in the form of levels of detail (LODs). It significantly reduces the cost of labor and time while providing detailed IR structures. Starting from the surface meshes generated by multi-view stereo (MVS) systems, we produce watertight LODs via semantic segmentation and structure-aware approximation. For each LOD, we divide the surfaces into triangle facets of specific scales. For each facet, one material attribute is attached, and the heat balance equations are solved to obtain the temperature. Three strategies are proposed to accelerate the thermal distribution calculation. Finally, by synthesizing the radiance distribution, the whole IR scenes are generated and rendered. Experiments on real urban scenes show that the proposed pipeline could effectively simulate IR scenes of large-scale urban scenes.

Region Based CNN for Foreign Object Debris Detection on Airfield Pavement.

In this paper, a novel algorithm based on convolutional neural network (CNN) is proposed to detect foreign object debris (FOD) based on optical imaging sensors. It contains two modules, the improved region proposal network (RPN) and spatial transformer network (STN) based CNN classifier. In the improved RPN, some extra select rules are designed and deployed to generate high quality candidates with fewer numbers. Moreover, the efficiency of CNN detector is significantly improved by introducing STN layer. Compared to faster R-CNN and single shot multiBox detector (SSD), the proposed algorithm achieves better result for FOD detection on airfield pavement in the experiment.

Semi-automated infrared simulation on real urban scenes based on multi-view images.

The development of modern infrared applications require simulating thermal representations for targets of interest. However, generating geometric models for simulation has been a laborious, time-consuming work, which greatly limits the practical applications in real-world. In order to reduce the man-in-the-loop requirements, we devise a method that directly and semi-automatically simulates infrared signatures of real urban scenes. From raw meshes generated by multi-view stereo, we automatically produce a simplified watertight model through piecewise-planar 3D reconstruction. Model surface is subdivided into quality mesh elements to attach material attributes. For each element, heat balance equation is solved so as to render the whole scene by synthesizing the radiance distribution in infrared waveband. The credibility and effectiveness of our method are confirmed by comparing simulation results to the measured data in real-world. Our experiments on various types of buildings and large scale scene show that the proposed pipeline simulates meaningful infrared scenes while being robust and scalable.

Video stabilization in atmosphere turbulent conditions based on the Laplacian-Riesz pyramid.

Video stabilization in atmosphere turbulent conditions is aimed at removing spatiotemporally varying distortions from video recordings. Conventional shaky video stabilization approaches do not perform effectively under turbulent circumstances due to the erratic motion common to those conditions. Using complex-valued image pyramids, we propose a method to mitigate this erratic motion in videos. First, each frame of a video is decomposed into different spatial frequencies using the Laplacian pyramid. Second, a Riesz transform is adopted to extract the local amplitude and the local phase of each sub-band. Next, low-pass filters are designed to attenuate the local amplitude and phase variations to remove turbulence-induced distortions. Experimental results show that the proposed approach is efficient and provides stabilizing video in atmosphere turbulent conditions.

Microscopy mineral image enhancement through center operator construction.

Enhancing a microscopy mineral image to produce clear and rich details is important for mineral analysis. To enhance effectively, an algorithm utilizing the constructed center operator is proposed in this paper. First, the center operator is constructed from the opening and closing based toggle operator. Second, the mineral image features are extracted by using the constructed center operator. Third, the multiscale mineral features are extracted through multiscale morphological theory using the multiscale structuring elements. Fourth, the final features for mineral image enhancement are calculated from the extracted multiscale features based on the weight strategy. Finally, the mineral image is effectively enhanced through importing the final features into the original mineral image. Experimental results on various microscopy mineral images verified that the proposed algorithm performed well for enhancement and had competing performance compared with some existing algorithms.

Infrared and Visual Image Fusion through Fuzzy Measure and Alternating Operators.

The crucial problem of infrared and visual image fusion is how to effectively extract the image features, including the image regions and details and combine these features into the final fusion result to produce a clear fused image. To obtain an effective fusion result with clear image details, an algorithm for infrared and visual image fusion through the fuzzy measure and alternating operators is proposed in this paper. Firstly, the alternating operators constructed using the opening and closing based toggle operator are analyzed. Secondly, two types of the constructed alternating operators are used to extract the multi-scale features of the original infrared and visual images for fusion. Thirdly, the extracted multi-scale features are combined through the fuzzy measure-based weight strategy to form the final fusion features. Finally, the final fusion features are incorporated with the original infrared and visual images using the contrast enlargement strategy. All the experimental results indicate that the proposed algorithm is effective for infrared and visual image fusion.

Geometry-Augmented Molecular Representation Learning for Property Prediction.

Accurate molecular representation plays a crucial role in expediting the process of drug discovery. Graph neural networks (GNNs) have demonstrated robust capabilities in molecular representation learning, adept at capturing structural and spatial information in molecular graphs. For molecular representation learning, most previous GNN methods are specialized in dealing with 2D or 3D molecular data formats. By further fusing the geometric attributes and structural features of molecules, we can elevate the performance of molecular representation. To realize this, we present a novel geometryaugmented molecular representation learning model, designed to effectively encode both the 2D structural and 3D spatial information inherent in molecular graphs. By incorporating structural and spatial information as attention biases in the graph Transformer framework, our model offers a comprehensive architecture that introduces molecular structural details at both atom and bond levels. We further propose a geometry information fusion module to encode the geometry information within 3D molecular graphs. The experimental results show the efficacy of our model, demonstrating its ability to achieve competitive performance when compared to state-ofthe-art (SOTA) models in various property prediction tasks.

Teacher-student collaborated multiple instance learning for pan-cancer PDL1 expression prediction from histopathology slides.

Programmed cell death ligand 1 (PDL1), as an important biomarker, is quantified by immunohistochemistry (IHC) with few established histopathological patterns. Deep learning aids in histopathological assessment, yet heterogeneity and lacking spatially resolved annotations challenge precise analysis. Here, we present a weakly supervised learning approach using bulk RNA sequencing for PDL1 expression prediction from hematoxylin and eosin (H&E) slides. Our method extends the multiple instance learning paradigm with the teacher-student framework, which assigns dynamic pseudo-labels for intra-slide heterogeneity and retrieves unlabeled instances using temporal ensemble model distillation. The approach, evaluated on 12,299 slides across 20 solid tumor types, achieves a weighted average area under the curve of 0.83 on fresh-frozen and 0.74 on formalin-fixed specimens for 9 tumors with PDL1 as an established biomarker. Our method predicts PDL1 expression patterns, validated by IHC on 20 slides, offering insights into histologies relevant to PDL1. This demonstrates the potential of deep learning in identifying diverse histological patterns for molecular changes from H&E images.

Experimental validation of the accuracy of robotic-assisted radioactive seed implantation for tumor treatment.

An experimental validation of a robotic system for radioactive iodine-125 seed implantation (RISI) in tumor treatment was conducted using customized phantom models and animal models simulating liver and lung lesions. The robotic system, consisting of planning, navigation, and implantation modules, was employed to implant dummy radioactive seeds into the models. Fiducial markers were used for target localization. In phantom experiments across 40 cases, the mean errors between planned and actual seed positions were 0.98 ± 1.05 mm, 1.14 ± 0.62 mm, and 0.90 ± 1.05 mm in the x, y, and z directions, respectively. The x, y, and z directions correspond to the left-right, anterior-posterior, and superior-inferior anatomical planes. Silicone phantoms exhibiting significantly smaller x-axis errors compared to liver and lung phantoms (p < 0.05). Template assistance significantly reduced errors in all axes (p < 0.05). No significant dosimetric deviations were observed in parameters such as D90, V100, and V150 between plans and post-implant doses (p > 0.05). In animal experiments across 23 liver and lung cases, the mean implantation errors were 1.28 ± 0.77 mm, 1.66 ± 0.69 mm, and 1.86 ± 0.93 mm in the x, y, and z directions, slightly higher than in phantoms (p < 0.05), with no significant differences between liver and lung models. The dosimetric results closely matched planned values, confirming the accuracy of the robotic system for RISI, offering new possibilities in clinical tumor treatment.

Image analysis through feature extraction by using top-hat transform-based morphological contrast operator.

Image decomposition and reconstruction is an important way for image analysis. To be effective for image decomposition and reconstruction, a method using extracted features through top-hat transform-based morphological contrast operator (MCOTH) is proposed in this paper. First, the morphological contrast operator constructed using the top-hat transforms is discussed. Then, extracting the bright and dark image features in the result of MCOTH is given. Based on the extracted bright and dark image features, the original images are decomposed into multiscale complete decompositions using multiscale structuring elements. After processing the decomposed images following different application purposes, the final result image can be reconstructed from the processed decomposition images. To verify the effectiveness of the proposed image analysis method through image decomposition and reconstruction, the application of image enhancement and fusion are discussed. The experimental results show that because the proposed image decomposition and reconstruction method reasonably decomposes the original image into complete decomposition with useful image features at different scales, the useful image features could be easily used for different applications. After the useful image features are processed, the final result image could be reconstructed. Moreover, different types of images are used in the experiments of image enhancement and fusion, and the results are effective. Therefore, the proposed image decomposition and reconstruction method in this paper are effective methods for image analysis and could be widely used in different applications.

Fuzzy Sparse Subspace Clustering for Infrared Image Segmentation.

Infrared image segmentation is a challenging task, due to interference of complex background and appearance inhomogeneity of foreground objects. A critical defect of fuzzy clustering for infrared image segmentation is that the method treats image pixels or fragments in isolation. In this paper, we propose to adopt self-representation from sparse subspace clustering in fuzzy clustering, aiming to introduce global correlation information into fuzzy clustering. Meanwhile, to apply sparse subspace clustering for non-linear samples from an infrared image, we leverage membership from fuzzy clustering to improve conventional sparse subspace clustering. The contributions of this paper are fourfold. First, by introducing self-representation coefficients modeled in sparse subspace clustering based on high-dimensional features, fuzzy clustering is capable of utilizing global information to resist complex background as well as intensity inhomogeneity of objects, so as to improve clustering accuracy. Second, fuzzy membership is tactfully exploited in the sparse subspace clustering framework. Thereby, the bottleneck of conventional sparse subspace clustering methods, that they could be barely applied to nonlinear samples, can be surmounted. Third, as we integrate fuzzy clustering and subspace clustering in a unified framework, features from two different aspects are employed, contributing to precise clustering results. Finally, we further incorporate neighbor information into clustering, thus effectively solving the uneven intensity problem in infrared image segmentation. Experiments examine the feasibility of proposed methods on various infrared images. Segmentation results demonstrate the effectiveness and efficiency of the proposed methods, which proves the superiority compared to other fuzzy clustering methods and sparse space clustering methods.

Revelation of hidden 2D atmospheric turbulence strength fields from turbulence effects in infrared imaging.

Turbulence exists widely in the natural atmosphere and in industrial fluids. Strong randomness, anisotropy and mixing of multiple-scale eddies complicate the analysis and measurement of atmospheric turbulence. Although the spatially integrated strength of atmospheric turbulence can be roughly measured indirectly by Doppler radar or laser, direct measurement of two-dimensional (2D) strength fields of atmospheric turbulence is challenging. Here we attempt to solve this problem through infrared imaging. Specifically, we propose a physically boosted cooperative learning framework, termed the PBCL, to quantify 2D turbulence strength from infrared images. To demonstrate the capability of the PBCL, we constructed a dataset with 137,336 infrared images and corresponding 2D turbulence strength fields. The experimental results show that cooperative learning brings performance improvements, enabling the PBCL to simultaneously learn turbulence strength fields and inhibit adverse turbulence effects in images. Our work demonstrates the potential of imaging in measuring physical quantity fields.

Contour-aware network with class-wise convolutions for 3D abdominal multi-organ segmentation.

Accurate delineation of multiple organs is a critical process for various medical procedures, which could be operator-dependent and time-consuming. Existing organ segmentation methods, which were mainly inspired by natural image analysis techniques, might not fully exploit the traits of the multi-organ segmentation task and could not accurately segment the organs with various shapes and sizes simultaneously. In this work, the characteristics of multi-organ segmentation are considered: the global count, position and scale of organs are generally predictable, while their local shape and appearance are volatile. Thus, we supplement the region segmentation backbone with a contour localization task to increase the certainty along delicate boundaries. Meantime, each organ has exclusive anatomical traits, which motivates us to deal with class variability with class-wise convolutions to highlight organ-specific features and suppress irrelevant responses at different field-of-views. To validate our method with adequate amounts of patients and organs, we constructed a multi-center dataset, which contains 110 3D CT scans with 24,528 axial slices, and provided voxel-level manual segmentations of 14 abdominal organs, which adds up to 1,532 3D structures in total. Extensive ablation and visualization studies on it validate the effectiveness of the proposed method. Quantitative analysis shows that we achieve state-of-the-art performance for most abdominal organs, and obtain 3.63 mm 95% Hausdorff Distance and 83.32% Dice Similarity Coefficient on an average.

Exploring the influence of nasal vestibule structure on nasal obstruction using CFD and Machine Learning method.

Motivated by clinical findings about the nasal vestibule, this study analyzes the aerodynamic characteristics of the nasal vestibule and attempt to determine anatomical features which have a large influence on airflow through a combination of Computational Fluid Dynamics (CFD) and machine learning method. Firstly, the aerodynamic characteristics of the nasal vestibule are detailedly analyzed using the CFD method. Based on CFD simulation results, we divide the nasal vestibule into two types with distinctly different airflow patterns, which is consistent with clinical findings. Secondly, we explore the relationship between anatomical features and aerodynamic characteristics by developing a novel machine learning model which could predict airflow patterns based on several anatomical features. Feature mining is performed to determine the anatomical feature which has the greatest impact on respiratory function. The method is developed and validated on 41 unilateral nasal vestibules from 26 patients with nasal obstruction. The correctness of the CFD analysis and the developed model is verified by comparing them with clinical findings.

Electron transfer rules of minerals under pressure informed by machine learning.

Electron transfer is the most elementary process in nature, but the existing electron transfer rules are seldom applied to high-pressure situations, such as in the deep Earth. Here we show a deep learning model to obtain the electronegativity of 96 elements under arbitrary pressure, and a regressed unified formula to quantify its relationship with pressure and electronic configuration. The relative work function of minerals is further predicted by electronegativity, presenting a decreasing trend with pressure because of pressure-induced electron delocalization. Using the work function as the case study of electronegativity, it reveals that the driving force behind directional electron transfer results from the enlarged work function difference between compounds with pressure. This well explains the deep high-conductivity anomalies, and helps discover the redox reactivity between widespread Fe(II)-bearing minerals and water during ongoing subduction. Our results give an insight into the fundamental physicochemical properties of elements and their compounds under pressure.

Transport of intensity phase imaging from multiple noisy intensities measured in unequally-spaced planes.

The noise problem is generally inevitable for phase retrieval by solving the transport of intensity equation (TIE). The noise effect can be alleviated by using multiple intensities to estimate the axial intensity derivative in the TIE. In this study, a method is proposed for estimating the intensity derivative by using multiple unevenly-spaced noisy measurements. The noise-minimized intensity derivative is approximated by a linear combination of the intensity data, in which the coefficients are obtained by solving a constrained optimization problem. The performance of the method is investigated by both the error analysis and the numerical simulations, and the results show that the method can reduce the noise effect on the retrieved phase. In addition, guidelines for the choice of the number of the intensity planes are given.

Atmospheric spectral model and theoretical expressions of irradiance scintillation index for optical wave propagating through moderate-to-strong non-Kolmogorov turbulence.

A new atmospheric spectral model and expressions of irradiance scintillation index are derived theoretically for optical wave propagating through moderate-to-strong non-Kolmogorov turbulence. They are developed under Andrews' assumption that small-scale irradiance fluctuations are modulated by large-scale irradiance fluctuations of the wave, and the geometrical optics approximation is adopted for mathematical development. A wide range of turbulence strength is considered instead of a limited range for weak turbulence. The atmospheric spectral model has a spectral power law value in the range of 3 to 4 instead of the standard power law value of 11/3. Numerical calculations are conducted to analyze the influences of spectral power law and turbulence strength.

Image fusion through feature extraction by using sequentially combined toggle and top-hat based contrast operator.

Combing the useful information of multisensor or multifocus images is important for producing effective optical images. To extract and combine the image features of the original images for image fusion well, an algorithm through feature extraction by using the sequentially combined toggle and top-hat based contrast operator is proposed in this paper. Sequentially combining toggle contrast operator and top-hat based contrast operator could be used to identify well the effective bright and dark image features. Furthermore, through multiscale extension, the effective bright and dark image features at multiscales of an image are extracted. After the final bright and dark fusion features are constructed by using the pixel-wise maximum operation on the multiscale image features from different images, the final fusion result is obtained by importing the final bright and dark fusion features into the base image. Experimental results on different types of images show that the proposed algorithm performs well for image fusion, which may be widely used in different applications, such as security surveillance, object recognition, and so on.

Noise-suppressed image enhancement using multiscale top-hat selection transform through region extraction.

Enhancing an image through increasing the contrast of the image is one effective way of image enhancement. To well enhance an image and suppress the produced noises in the resulting image, a multiscale top-hat selection transform-based algorithm through extracting bright and dark image regions and increasing the contrast between them is proposed. First, the multiscale top-hat selection transform is discussed and then is used to extract the bright and dark image regions of each scale. Second, the final extracted bright and dark image regions are obtained through a maximum operation on all the extracted multiscale bright and dark image regions at all scales. Finally, by using a weight strategy, the image is enhanced through increasing the contrast of the image by adding the final bright regions on and subtracting the final dark regions from the original image. The weight parameters are used to adjust the effect of image enhancement. Because the multiscale top-hat selection transform is used to effectively extract the final image regions and discriminate the possible noise regions, the image is well enhanced and some noises are suppressed. Experimental results on different types of images show that our algorithm performs well for noise-suppressed image enhancement and is useful for different applications.

Multiple linear feature detection based on multiple-structuring-element center-surround top-hat transform.

Linear feature detection is an important technique in different applications of image processing. To detect linear features in different types of images, a simple but effective algorithm based on a multiple-structuring-element center-surround top-hat transform is proposed. The center-surround top-hat transform is discussed and analyzed. Based on the properties of this transform for image feature detection, multiple structuring elements are constructed corresponding to the possible linear features at different directions. The whole algorithm is divided into four parts. First, the algorithm uses the center-surround top-hat transform to detect all the possible linear features at different directions through constructing multiple structuring elements. Second, the detected linear feature regions at each direction are processed by a closing operation to remove the possible holes or unconnected regions. Third, the processed results of the detected linear feature regions at all directions are combined to form all the possible detected linear feature regions. Fourth, the combined result is refined by using some simple operations to form the final result. Experimental results on different types of images from different applications verified the effective performance of the proposed algorithm. Moreover, the experimental results indicate that the proposed algorithm could be used in different applications.