Just Noticeable Difference Model for Images with Color Sensitivity.

The just noticeable difference (JND) model reflects the visibility limitations of the human visual system (HVS), which plays an important role in perceptual image/video processing and is commonly applied to perceptual redundancy removal. However, existing JND models are usually constructed by treating the color components of three channels equally, and their estimation of the masking effect is inadequate. In this paper, we introduce visual saliency and color sensitivity modulation to improve the JND model. Firstly, we comprehensively combined contrast masking, pattern masking, and edge protection to estimate the masking effect. Then, the visual saliency of HVS was taken into account to adaptively modulate the masking effect. Finally, we built color sensitivity modulation according to the perceptual sensitivities of HVS, to adjust the sub-JND thresholds of Y, Cb, and Cr components. Thus, the color-sensitivity-based JND model (CSJND) was constructed. Extensive experiments and subjective tests were conducted to verify the effectiveness of the CSJND model. We found that consistency between the CSJND model and HVS was better than existing state-of-the-art JND models.

Where to look at the movies: Analyzing visual attention to understand movie editing.

In the process of making a movie, directors constantly care about where the spectator will look on the screen. Shot composition, framing, camera movements, or editing are tools commonly used to direct attention. In order to provide a quantitative analysis of the relationship between those tools and gaze patterns, we propose a new eye-tracking database, containing gaze-pattern information on movie sequences, as well as editing annotations, and we show how state-of-the-art computational saliency techniques behave on this dataset. In this work, we expose strong links between movie editing and spectators gaze distributions, and open several leads on how the knowledge of editing information could improve human visual attention modeling for cinematic content. The dataset generated and analyzed for this study is available at https://github.com/abruckert/eye_tracking_filmmaking.

BiconNet: An Edge-preserved Connectivity-based Approach for Salient Object Detection.

Salient object detection (SOD) is viewed as a pixel-wise saliency modeling task by traditional deep learning-based methods. A limitation of current SOD models is insufficient utilization of inter-pixel information, which usually results in imperfect segmentation near edge regions and low spatial coherence. As we demonstrate, using a saliency mask as the only label is suboptimal. To address this limitation, we propose a connectivity-based approach called bilateral connectivity network (BiconNet), which uses connectivity masks together with saliency masks as labels for effective modeling of inter-pixel relationships and object saliency. Moreover, we propose a bilateral voting module to enhance the output connectivity map, and a novel edge feature enhancement method that efficiently utilizes edge-specific features. Through comprehensive experiments on five benchmark datasets, we demonstrate that our proposed method can be plugged into any existing state-of-the-art saliency-based SOD framework to improve its performance with negligible parameter increase.

Infrared and Visible Image Fusion Based on Visual Saliency Map and Image Contrast Enhancement.

The purpose of infrared and visible image fusion is to generate images with prominent targets and rich information which provides the basis for target detection and recognition. Among the existing image fusion methods, the traditional method is easy to produce artifacts, and the information of the visible target and texture details are not fully preserved, especially for the image fusion under dark scenes and smoke conditions. Therefore, an infrared and visible image fusion method is proposed based on visual saliency image and image contrast enhancement processing. Aiming at the problem that low image contrast brings difficulty to fusion, an improved gamma correction and local mean method is used to enhance the input image contrast. To suppress artifacts that are prone to occur in the process of image fusion, a differential rolling guidance filter (DRGF) method is adopted to decompose the input image into the basic layer and the detail layer. Compared with the traditional multi-scale decomposition method, this method can retain specific edge information and reduce the occurrence of artifacts. In order to solve the problem that the salient object of the fused image is not prominent and the texture detail information is not fully preserved, the salient map extraction method is used to extract the infrared image salient map to guide the fusion image target weight, and on the other hand, it is used to control the fusion weight of the basic layer to improve the shortcomings of the traditional 'average' fusion method to weaken the contrast information. In addition, a method based on pixel intensity and gradient is proposed to fuse the detail layer and retain the edge and detail information to the greatest extent. Experimental results show that the proposed method is superior to other fusion algorithms in both subjective and objective aspects.

Multi-Scale Mixed Attention Network for CT and MRI Image Fusion.

Recently, the rapid development of the Internet of Things has contributed to the generation of telemedicine. However, online diagnoses by doctors require the analyses of multiple multi-modal medical images, which are inconvenient and inefficient. Multi-modal medical image fusion is proposed to solve this problem. Due to its outstanding feature extraction and representation capabilities, convolutional neural networks (CNNs) have been widely used in medical image fusion. However, most existing CNN-based medical image fusion methods calculate their weight maps by a simple weighted average strategy, which weakens the quality of fused images due to the effect of inessential information. In this paper, we propose a CNN-based CT and MRI image fusion method (MMAN), which adopts a visual saliency-based strategy to preserve more useful information. Firstly, a multi-scale mixed attention block is designed to extract features. This block can gather more helpful information and refine the extracted features both in the channel and spatial levels. Then, a visual saliency-based fusion strategy is used to fuse the feature maps. Finally, the fused image can be obtained via reconstruction blocks. The experimental results of our method preserve more textual details, clearer edge information and higher contrast when compared to other state-of-the-art methods.

A Moving Ship Detection and Tracking Method Based on Optical Remote Sensing Images from the Geostationary Satellite.

Geostationary optical remote sensing satellites, such as the GF-4, have a high temporal resolution and wide coverage, which enables the continuous tracking and observation of ship targets over a large range. However, the ship targets in the images are usually small and dim and the images are easily affected by clouds, islands and other factors, which make it difficult to detect the ship targets. This paper proposes a new method for detecting ships moving on the sea surface using GF-4 satellite images. First, the adaptive nonlinear gray stretch (ANGS) method was used to enhance the image and highlight small and dim ship targets. Second, a multi-scale dual-neighbor difference contrast measure (MDDCM) method was designed to enable detection of the position of the candidate ship target. The shape characteristics of each candidate area were analyzed to remove false ship targets. Finally, the joint probability data association (JPDA) method was used for multi-frame data association and tracking. Our results suggest that the proposed method can effectively detect and track moving ship targets in GF-4 satellite optical remote sensing images, with better detection performance than other classical methods.

Fusion of Infrared and Visible Images Using Fast Global Smoothing Decomposition and Target-Enhanced Parallel Gaussian Fuzzy Logic.

As a powerful technique to merge complementary information of original images, infrared (IR) and visible image fusion approaches are widely used in surveillance, target detecting, tracking, and biological recognition, etc. In this paper, an efficient IR and visible image fusion method is proposed to simultaneously enhance the significant targets/regions in all source images and preserve rich background details in visible images. The multi-scale representation based on the fast global smoother is firstly used to decompose source images into the base and detail layers, aiming to extract the salient structure information and suppress the halos around the edges. Then, a target-enhanced parallel Gaussian fuzzy logic-based fusion rule is proposed to merge the base layers, which can avoid the brightness loss and highlight significant targets/regions. In addition, the visual saliency map-based fusion rule is designed to merge the detail layers with the purpose of obtaining rich details. Finally, the fused image is reconstructed. Extensive experiments are conducted on 21 image pairs and a Nato-camp sequence (32 image pairs) to verify the effectiveness and superiority of the proposed method. Compared with several state-of-the-art methods, experimental results demonstrate that the proposed method can achieve more competitive or superior performances according to both the visual results and objective evaluation.

Multi-Scale Global Contrast CNN for Salient Object Detection.

Salient object detection (SOD) is a fundamental task in computer vision, which attempts to mimic human visual systems that rapidly respond to visual stimuli and locate visually salient objects in various scenes. Perceptual studies have revealed that visual contrast is the most important factor in bottom-up visual attention process. Many of the proposed models predict saliency maps based on the computation of visual contrast between salient regions and backgrounds. In this paper, we design an end-to-end multi-scale global contrast convolutional neural network (CNN) that explicitly learns hierarchical contrast information among global and local features of an image to infer its salient object regions. In contrast to many previous CNN based saliency methods that apply super-pixel segmentation to obtain homogeneous regions and then extract their CNN features before producing saliency maps region-wise, our network is pre-processing free without any additional stages, yet it predicts accurate pixel-wise saliency maps. Extensive experiments demonstrate that the proposed network generates high quality saliency maps that are comparable or even superior to those of state-of-the-art salient object detection architectures.

Increasing consumers' attention capture and food choice through bottom-up effects.

To guide consumers in their decision process, especially food products often carry labels indicating production method or nutritional content. However, past research shows that many labels are rarely attended to in the consumer's decision process. In order to enhance the effectiveness of such labels and to increase choice likelihood of labeled products, the label must capture attention. We address the question of how a single label on the product packaging can capture attention through bottom-up effects and increase choice through increased attention capture. To this end, we conducted a combined eye tracking and choice experiment manipulating the surface size and visual saliency - the two most important bottom-up effects on attention - of the Danish organic label across three food product categories. Results show a strong and significant increase in attention capture towards a larger and more visually salient label. Most importantly, the effect of attention capture carried over into increased choice likelihood. Both marketers and policy makers might benefit from this approach, which provides directions for designing product labels that can influence attention capture and product choice.

Time course of selective attention to face regions in social anxiety: eye-tracking and computational modelling.

We investigated the time course of selective attention to face regions during judgment of dis/approval by low (LSA) and high (HSA) social anxiety undergraduates (with clinical levels on questionnaire measures). The viewers' gaze direction was assessed and the stimulus visual saliency of face regions was computed, for video-clips displaying dynamic facial expressions. Social anxiety was related to perception of disapproval from faces with an ambiguous smile (i.e. with non-happy eyes), but not those with congruent happy eyes and a smile. HSA observers selectively looked earlier at the eye region, whereas LSA ones preferentially looked at the smiling mouth. Consistently, gaze allocation was less related to visual saliency of the smile for HSA than for LSA viewers. The attentional bias towards the less salient eye region - thus opposing the automatic capture by the smile - suggests that it is strategically driven in HSA individuals, possibly aimed at detecting negative evaluators.

PSDSD-A Superpixel Generating Method Based on Pixel Saliency Difference and Spatial Distance for SAR Images.

Superpixel methods are widely used in the processing of synthetic aperture radar (SAR) images. In recent years, a number of superpixel algorithms for SAR images have been proposed, and have achieved acceptable results despite the inherent speckle noise of SAR images. However, it is still difficult for existing algorithms to obtain satisfactory results in the inhomogeneous edge and texture areas. To overcome those problems, we propose a superpixel generating method based on pixel saliency difference and spatial distance for SAR images in this article. Firstly, a saliency map is calculated based on the Gaussian kernel function weighted local contrast measure, which can not only effectively suppress the speckle noise, but also enhance the fuzzy edges and regions with intensity inhomogeneity. Secondly, superpixels are generated by the local k-means clustering method based on the proposed distance measure, which can efficiently sort pixels to different clusters. In this step, the distance measure is calculated by combining the saliency difference and spatial distance with a proposed adaptive local compactness parameter. Thirdly, post-processing is utilized to clean up small segments. The evaluation experiments on the simulated SAR image demonstrate that our proposed method dramatically outperforms four state-of-the-art methods in terms of boundary recall, under-segmentation error, and achievable segmentation accuracy under almost all of the experimental parameters at a moderate segment speed. The experiments on real-world SAR images of different sceneries validate the superiority of our method. The superpixel results of the proposed method adhere well to the contour of targets, and correctly reflect the boundaries of texture details for the inhomogeneous regions.

Sea-Sky Line and its Nearby Ships Detection Based on the Motion Attitude of Visible Light Sensors.

In the maritime scene, visible light sensors installed on ships have difficulty accurately detecting the sea-sky line (SSL) and its nearby ships due to complex environments and six-degrees-of-freedom movement. Aimed at this problem, this paper combines the camera and inertial sensor data, and proposes a novel maritime target detection algorithm based on camera motion attitude. The algorithm mainly includes three steps, namely, SSL estimation, SSL detection, and target saliency detection. Firstly, we constructed the camera motion attitude model by analyzing the camera's six-degrees-of-freedom motion at sea, estimated the candidate region (CR) of the SSL, then applied the improved edge detection algorithm and the straight-line fitting algorithm to extract the optimal SSL in the CR. Finally, in the region of ship detection (ROSD), an improved visual saliency detection algorithm was applied to extract the target ships. In the experiment, we constructed SSL and its nearby ship detection dataset that matches the camera's motion attitude data by real ship shooting, and verified the effectiveness of each model in the algorithm through comparative experiments. Experimental results show that compared with the other maritime target detection algorithm, the proposed algorithm achieves a higher detection accuracy in the detection of the SSL and its nearby ships, and provides reliable technical support for the visual development of unmanned ships.

Inshore Ship Detection Based on Level Set Method and Visual Saliency for SAR Images.

Inshore ship detection is an important research direction of synthetic aperture radar (SAR) images. Due to the effects of speckle noise, land clutters and low signal-to-noise ratio, it is still challenging to achieve effective detection of inshore ships. To solve these issues, an inshore ship detection method based on the level set method and visual saliency is proposed in this paper. First, the image is fast initialized through down-sampling. Second, saliency map is calculated by improved local contrast measure (ILCM). Third, an improved level set method based on saliency map is proposed. The saliency map has a higher signal-to-noise ratio and the local level set method can effectively segment images with intensity inhomogeneity. In this way, the improved level set method has a better segmentation result. Then, candidate targets are obtained after the adaptive threshold. Finally, discrimination is employed to get the final result of ship targets. The experiments on a number of SAR images demonstrate that the proposed method can detect ship targets with reasonable accuracy and integrity.

Annotating Early Esophageal Cancers Based on Two Saliency Levels of Gastroscopic Images.

Early diagnoses of esophageal cancer can greatly improve the survival rate of patients. At present, the lesion annotation of early esophageal cancers (EEC) in gastroscopic images is generally performed by medical personnel in a clinic. To reduce the effect of subjectivity and fatigue in manual annotation, computer-aided annotation is required. However, automated annotation of EEC lesions using images is a challenging task owing to the fine-grained variability in the appearance of EEC lesions. This study modifies the traditional EEC annotation framework and utilizes visual salient information to develop a two saliency levels-based lesion annotation (TSL-BLA) for EEC annotations on gastroscopic images. Unlike existing methods, the proposed framework has a strong ability of constraining false positive outputs. What is more, TSL-BLA is also placed an additional emphasis on the annotation of small EEC lesions. A total of 871 gastroscopic images from 231 patients were used to validate TSL-BLA. 365 of those images contain 434 EEC lesions and 506 images do not contain any lesions. 101 small lesion regions are extracted from the 434 lesions to further validate the performance of TSL-BLA. The experimental results show that the mean detection rate and Dice similarity coefficients of TSL-BLA were 97.24 and 75.15%, respectively. Compared with other state-of-the-art methods, TSL-BLA shows better performance. Moreover, it shows strong superiority when annotating small EEC lesions. It also produces fewer false positive outputs and has a fast running speed. Therefore, The proposed method has good application prospects in aiding clinical EEC diagnoses.

[A novel method of optic disk segmentation based on visual saliency and rotary scanning].

Fast optic disk localization and boundary segmentation is an important research topic in computer aided diagnosis. This paper proposes a novel method to effectively segment optic disk by using human visual characteristics in analyzing and processing fundus image. After a general analysis of optic disk features in fundus images, the target of interest could be located quickly, and intensity, color and spatial distribution of the disc are used to generate saliency map based on pixel distance. Then the adaptive threshold is used to segment optic disk. Moreover, to reduce the influence of vascular, a rotary scanning method is devised to achieve complete and continuous contour of optic disk boundary. Tests in the public fundus images database Drishti-GS have good performances, which mean that the proposed method is simple and rapid, and it meets the standard of the eye specialists. It is hoped that the method could be conducive to the computer aided diagnosis of eye diseases in the future.

Image Processing Strategies Based on a Visual Saliency Model for Object Recognition Under Simulated Prosthetic Vision.

Retinal prostheses have the potential to restore partial vision. Object recognition in scenes of daily life is one of the essential tasks for implant wearers. Still limited by the low-resolution visual percepts provided by retinal prostheses, it is important to investigate and apply image processing methods to convey more useful visual information to the wearers. We proposed two image processing strategies based on Itti's visual saliency map, region of interest (ROI) extraction, and image segmentation. Itti's saliency model generated a saliency map from the original image, in which salient regions were grouped into ROI by the fuzzy c-means clustering. Then Grabcut generated a proto-object from the ROI labeled image which was recombined with background and enhanced in two ways--8-4 separated pixelization (8-4 SP) and background edge extraction (BEE). Results showed that both 8-4 SP and BEE had significantly higher recognition accuracy in comparison with direct pixelization (DP). Each saliency-based image processing strategy was subject to the performance of image segmentation. Under good and perfect segmentation conditions, BEE and 8-4 SP obtained noticeably higher recognition accuracy than DP, and under bad segmentation condition, only BEE boosted the performance. The application of saliency-based image processing strategies was verified to be beneficial to object recognition in daily scenes under simulated prosthetic vision. They are hoped to help the development of the image processing module for future retinal prostheses, and thus provide more benefit for the patients.

Brain lateralization of holistic versus analytic processing of emotional facial expressions.

This study investigated the neurocognitive mechanisms underlying the role of the eye and the mouth regions in the recognition of facial happiness, anger, and surprise. To this end, face stimuli were shown in three formats (whole face, upper half visible, and lower half visible) and behavioral categorization, computational modeling, and ERP (event-related potentials) measures were combined. N170 (150-180 ms post-stimulus; right hemisphere) and EPN (early posterior negativity; 200-300 ms; mainly, right hemisphere) were modulated by expression of whole faces, but not by separate halves. This suggests that expression encoding (N170) and emotional assessment (EPN) require holistic processing, mainly in the right hemisphere. In contrast, the mouth region of happy faces enhanced left temporo-occipital activity (150-180 ms), and also the LPC (late positive complex; centro-parietal) activity (350-450 ms) earlier than the angry eyes (450-600 ms) or other face regions. Relatedly, computational modeling revealed that the mouth region of happy faces was also visually salient by 150 ms following stimulus onset. This suggests that analytical or part-based processing of the salient smile occurs early (150-180 ms) and lateralized (left), and is subsequently used as a shortcut to identify the expression of happiness (350-450 ms). This would account for the happy face advantage in behavioral recognition tasks when the smile is visible.

Emotional content of an image attracts attention more than visually salient features in various signal-to-noise ratio conditions.

Emotional images are processed in a prioritized manner, attracting attention almost immediately. In the present study we used eye tracking to reveal what type of features within neutral, positive, and negative images attract early visual attention: semantics, visual saliency, or their interaction. Semantic regions of interest were selected by observers, while visual saliency was determined using the Graph-Based Visual Saliency model. Images were transformed by adding pink noise in several proportions to be presented in a sequence of increasing and decreasing clarity. Locations of the first two fixations were analyzed. The results showed dominance of semantic features over visual saliency in attracting attention. This dominance was linearly related to the signal-to-noise ratio. Semantic regions were fixated more often in emotional images than in neutral ones, if signal-to-noise ratio was high enough to allow participants to comprehend the gist of a scene. Visual saliency on its own did not attract attention above chance, even in the case of pure noise images. Regions both visually salient and semantically relevant attracted a similar amount of fixation compared to semantic regions alone, or even more in the case of neutral pictures. Results provide evidence for fast and robust detection of semantically relevant features.

Predicting human gaze beyond pixels.

A large body of previous models to predict where people look in natural scenes focused on pixel-level image attributes. To bridge the semantic gap between the predictive power of computational saliency models and human behavior, we propose a new saliency architecture that incorporates information at three layers: pixel-level image attributes, object-level attributes, and semantic-level attributes. Object- and semantic-level information is frequently ignored, or only a few sample object categories are discussed where scaling to a large number of object categories is not feasible nor neurally plausible. To address this problem, this work constructs a principled vocabulary of basic attributes to describe object- and semantic-level information thus not restricting to a limited number of object categories. We build a new dataset of 700 images with eye-tracking data of 15 viewers and annotation data of 5,551 segmented objects with fine contours and 12 semantic attributes (publicly available with the paper). Experimental results demonstrate the importance of the object- and semantic-level information in the prediction of visual attention.

A computational visual saliency model based on statistics and machine learning.

Identifying the type of stimuli that attracts human visual attention has been an appealing topic for scientists for many years. In particular, marking the salient regions in images is useful for both psychologists and many computer vision applications. In this paper, we propose a computational approach for producing saliency maps using statistics and machine learning methods. Based on four assumptions, three properties (Feature-Prior, Position-Prior, and Feature-Distribution) can be derived and combined by a simple intersection operation to obtain a saliency map. These properties are implemented by a similarity computation, support vector regression (SVR) technique, statistical analysis of training samples, and information theory using low-level features. This technique is able to learn the preferences of human visual behavior while simultaneously considering feature uniqueness. Experimental results show that our approach performs better in predicting human visual attention regions than 12 other models in two test databases.