castleCSF - A contrast sensitivity function of color, area, spatiotemporal frequency, luminance and eccentricity.

The contrast sensitivity function (CSF) is a fundamental visual model explaining our ability to detect small contrast patterns. CSFs found many applications in engineering, where they can be used to optimize a design for perceptual limits. To serve such a purpose, CSFs must explain possibly a complete set of stimulus parameters, such as spatial and temporal frequency, luminance, and others. Although numerous contrast sensitivity measurements can be found in the literature, none fully explains the complete space of stimulus parameters. Therefore, in this work, we first collect and consolidate contrast sensitivity measurements from 18 studies, which explain the sensitivity variation across the parameters of interest. Then, we build an analytical contrast sensitivity model that explains the data from all those studies. The proposed castleCSF model explains the sensitivity as the function of spatial and temporal frequencies, an arbitrary contrast modulation direction in the color space, mean luminance, and chromaticity of the background, eccentricity, and stimulus area. The proposed model uses the same set of parameters to explain the data from 18 studies with an error of 3.59 dB. The consolidated contrast sensitivity data and the code for the model are publicly available at https://github.com/gfxdisp/castleCSF/.

Impact of focus cue presentation on perceived realism of 3-D scene structure: Implications for scene perception and for display technology.

Stereoscopic imagery often aims to evoke three-dimensional (3-D) percepts that are accurate and realistic-looking. The "gap" between 3-D imagery and real scenes is small, but focus cues typically remain incorrect because images are displayed on a single focal plane. Research has concentrated on the resulting vergence-accommodation conflicts. Yet, incorrect focus cues may also affect the appearance of 3-D imagery. We investigated whether incorrect focus cues reduce perceived realism of 3-D structure ("depth realism"). Experiment 1 used a multiple-focal-planes display to compare depth realism with correct focus cues vs. conventional stereo presentation. The stimuli were random-dot stereograms, which isolated the role of focus cues. Depth realism was consistently lower with incorrect focus cues, providing proof-of-principle evidence that they contribute to perceptual realism. Experiments 2 and 3 examined whether focus cues play a similar role with realistic objects, presented with an almost complete set of visual cues using a high-resolution, high-dynamic-range multiple-focal-planes display. We also examined the efficacy of approximating correct focus cues via gaze-contingent depth-of-field rendering. Improvements in depth realism with correct focus cues were less clear in more realistic scenes, indicating that the role of focus cues in depth realism depends on scene content. Rendering-based approaches, if anything, reduced depth realism, which we attribute to their inability to present higher-order aspects of blur correctly. Our findings suggest future general 3-D display solutions may need to present focus cues correctly to maximise perceptual realism.

Demonstrating a multi-primary high dynamic range display system for vision experiments.

We describe the design, construction, calibration, and characterization of a multi-primary high dynamic range (MPHDR) display system for use in vision research. The MPHDR display is the first system to our knowledge to allowfor spatially controllable, high dynamic range stimulus generation using multiple primaries.We demonstrate the high luminance, high dynamic range, and wide color gamut output of the MPHDR display. During characterization, the MPHDR display achieved a maximum luminance of 3200 cd=m2, a maximum contrast range of 3; 240; 000 V 1, and an expanded color gamut tailored to dedicated vision research tasks that spans beyond traditional sRGB displays. We discuss how the MPHDR display could be optimized for psychophysical experiments with photoreceptor isolating stimuli achieved through the method of silent substitution. We present an example case of a range of metameric pairs of melanopsin isolating stimuli across different luminance levels, from an available melanopsin contrast of117%at 75 cd=m2 to a melanopsin contrast of23%at 2000 cd=m2.

Spatio-chromatic contrast sensitivity under mesopic and photopic light levels.

Contrast sensitivity functions (CSFs) characterize the sensitivity of the human visual system at different spatial scales, but little is known as to how contrast sensitivity for achromatic and chromatic stimuli changes from a mesopic to a highly photopic range reflecting outdoor illumination levels. The purpose of our study was to further characterize the CSF by measuring both achromatic and chromatic sensitivities for background luminance levels from 0.02 cd/m2 to 7,000 cd/m2. Stimuli consisted of Gabor patches of different spatial frequencies and angular sizes, varying from 0.125 to 6 cpd, which were displayed on a custom high dynamic range (HDR) display with luminance levels up to 15,000 cd/m2. Contrast sensitivity was measured in three directions in color space, an achromatic direction, an isoluminant "red-green" direction, and an S-cone isolating "yellow-violet" direction, selected to isolate the luminance, L/M-cone opponent, and S-cone opponent pathways, respectively, of the early postreceptoral processing stages. Within each session, observers were fully adapted to the fixed background luminance (0.02, 2, 20, 200, 2,000, or 7,000 cd/m2). Our main finding is that the background luminance has a differential effect on achromatic contrast sensitivity compared to chromatic contrast sensitivity. The achromatic contrast sensitivity increases with higher background luminance up to 200 cd/m2 and then shows a sharp decline when background luminance is increased further. In contrast, the chromatic sensitivity curves do not show a significant sensitivity drop at higher luminance levels. We present a computational luminance-dependent model that predicts the CSF for achromatic and chromatic stimuli of arbitrary size.

Naturally glossy: Gloss perception, illumination statistics, and tone mapping.

Recognizing materials and understanding their properties is very useful-perhaps critical-in daily life as we encounter objects and plan our interactions with them. Visually derived estimates of material properties guide where and with what force we grasp an object. However, the estimation of material properties, such as glossiness, is a classic ill-posed problem. Image cues that we rely on to estimate gloss are also affected by shape, illumination and, in visual displays, tone-mapping. Here, we focus on the latter two. We define some commonalities present in the structure of natural illumination, and determine whether manipulation of these natural "signatures" impedes gloss constancy. We manipulate the illumination field to violate statistical regularities of natural illumination, such that light comes from below, or the luminance distribution is no longer skewed. These manipulations result in errors in perceived gloss. Similarly, tone mapping has a dramatic effect on perceived gloss. However, when objects are viewed against an informative (rather than plain gray) background that reflects these manipulations, there are some improvements to gloss constancy: in particular, observers are far less susceptible to the effects of tone mapping when judging gloss. We suggest that observers are sensitive to some very simple statistics of the environment when judging gloss.

Stereoscopic depth perception through foliage.

Both humans and computational methods struggle to discriminate the depths of objects hidden beneath foliage. However, such discrimination becomes feasible when we combine computational optical synthetic aperture sensing with the human ability to fuse stereoscopic images. For object identification tasks, as required in search and rescue, wildlife observation, surveillance, and early wildfire detection, depth assists in differentiating true from false findings, such as people, animals, or vehicles vs. sun-heated patches at the ground level or in the tree crowns, or ground fires vs. tree trunks. We used video captured by a drone above dense woodland to test users' ability to discriminate depth. We found that this is impossible when viewing monoscopic video and relying on motion parallax. The same was true with stereoscopic video because of the occlusions caused by foliage. However, when synthetic aperture sensing was used to reduce occlusions and disparity-scaled stereoscopic video was presented, whereas computational (stereoscopic matching) methods were unsuccessful, human observers successfully discriminated depth. This shows the potential of systems which exploit the synergy between computational methods and human vision to perform tasks that neither can perform alone.

Use of image features in predicting visually lossless thresholds of JPEG2000 compressed body CT images: initial trial.

PURPOSE: To test the image features that may be useful in predicting the visually lossless thresholds (VLTs) of body computed tomographic (CT) images for Joint Photographic Experts Group 2000 (JPEG2000) compression. MATERIALS AND METHODS: The institutional review board approved this study, with a waiver of informed patient consent. One hundred body CT studies obtained in different patients by using five scanning protocols were obtained, and 100 images, each of which was selected from each of the 100 studies, were collected. Five radiologists independently determined the VLT of each image for JPEG2000 compression by using the QUEST algorithm. The 100 images were randomly divided into two data sets-the training set (50 images) and the testing set (50 images)-and the division was repeated 200 times. For each of the 200 divisions, a multiple linear regression model was constructed on a training set and tested on a testing set regarding each of five image features-standard deviation of image intensity, image entropy, relative percentage of low-frequency (LF) energy, variation in high-frequency (HF) energy, and visual complexity-as independent variables and considering the VLTs determined with the median value of the radiologists' responses as a dependent variable. The root mean square residual and intraclass correlation coefficient (ICC) for the 200 divisions between the VLTs predicted by the models and those determined by radiologists were compared between the models by using repeated-measures analysis of variance with post-hoc comparisons. RESULTS: Mean root-mean-square residuals for multiple linear regression models constructed with variation in HF energy (1.20 ± 0.10 [standard deviation]) and visual complexity (1.09 ± 0.07) were significantly lower than those for standard deviation of image intensity (1.65 ± 0.13), image entropy (1.63 ± 0.14), and relative percentage of LF energy (1.58 ± 0.12) (P < .01). ICCs for variation in HF energy (0.64 ± 0.05) and visual complexity (0.71 ± 0.04) were significantly higher than those for standard deviation of image intensity (0.04 ± 0.02), image entropy (0.05 ± 0.02), and relative percentage of LF energy (0.20 ± 0.04) (P < .01). CONCLUSION: Among the five tested image features, variation in HF energy and visual complexity were the most promising in predicting the VLTs of body CT images for JPEG2000 compression.

Performance Comparison of Classical Methods and Neural Networks for Colour Correction.

Colour correction is the process of converting RAW RGB pixel values of digital cameras to a standard colour space such as CIE XYZ. A range of regression methods including linear, polynomial and root-polynomial least-squares have been deployed. However, in recent years, various neural network (NN) models have also started to appear in the literature as an alternative to classical methods. In the first part of this paper, a leading neural network approach is compared and contrasted with regression methods. We find that, although the neural network model supports improved colour correction compared with simple least-squares regression, it performs less well than the more advanced root-polynomial regression. Moreover, the relative improvement afforded by NNs, compared to linear least-squares, is diminished when the regression methods are adapted to minimise a perceptual colour error. Problematically, unlike linear and root-polynomial regressions, the NN approach is tied to a fixed exposure (and when exposure changes, the afforded colour correction can be quite poor). We explore two solutions that make NNs more exposure-invariant. First, we use data augmentation to train the NN for a range of typical exposures and second, we propose a new NN architecture which, by construction, is exposure-invariant. Finally, we look into how the performance of these algorithms is influenced when models are trained and tested on different datasets. As expected, the performance of all methods drops when tested with completely different datasets. However, we noticed that the regression methods still outperform the NNs in terms of colour correction, even though the relative performance of the regression methods does change based on the train and test datasets.

Objective index of image fidelity for JPEG2000 compressed body CT images.

Compression ratio (CR) has been the de facto standard index of compression level for medical images. The aim of the study is to evaluate the CR, peak signal-to-noise ratio (PSNR), and a perceptual quality metric (high-dynamic range visual difference predictor HDR-VDP) as objective indices of image fidelity for Joint Photographic Experts Group (JPEG) 2000 compressed body computed tomography (CT) images, from the viewpoint of visually lossless compression approach. A total of 250 body CT images obtained with five different scan protocols (5-mm-thick abdomen, 0.67-mm-thick abdomen, 5-mm-thick lung, 0.67-mm-thick lung, and 5-mm-thick low-dose lung) were compressed to one of five CRs (reversible, 6:1, 8:1, 10:1, and 15:1). The PSNR and HDR-VDP values were calculated for the 250 pairs of the original and compressed images. By alternately displaying an original and its compressed image on the same monitor, five radiologists independently determined if the pair was distinguishable or indistinguishable. The kappa statistic for the interobserver agreement among the five radiologists' responses was 0.70. According to the radiologists' responses, the number of distinguishable image pairs tended to significantly differ among the five scan protocols at 6:1-10:1 compressions (Fisher-Freeman-Halton exact tests). Spearman's correlation coefficients between each of the CR, PSNR, and HDR-VDP and the number of radiologists who responded as distinguishable were 0.72, -0.77, and 0.85, respectively. Using the radiologists' pooled responses as the reference standards, the areas under the receiver-operating-characteristic curves for the CR, PSNR, and HDR-VDP were 0.87, 0.93, and 0.97, respectively, showing significant differences between the CR and PSNR (p = 0.04), or HDR-VDP (p < 0.001), and between the PSNR and HDR-VDP (p < 0.001). In conclusion, the CR is less suitable than the PSNR or HDR-VDP as an objective index of image fidelity for JPEG2000 compressed body CT images. The HDR-VDP is more promising than the PSNR as such an index.

Temporal Resolution Multiplexing: Exploiting the limitations of spatio-temporal vision for more efficient VR rendering.

Rendering in virtual reality (VR) requires substantial computational power to generate 90 frames per second at high resolution with good-quality antialiasing. The video data sent to a VR headset requires high bandwidth, achievable only on dedicated links. In this paper we explain how rendering requirements and transmission bandwidth can be reduced using a conceptually simple technique that integrates well with existing rendering pipelines. Every even-numbered frame is rendered at a lower resolution, and every odd-numbered frame is kept at high resolution but is modified in order to compensate for the previous loss of high spatial frequencies. When the frames are seen at a high frame rate, they are fused and perceived as high-resolution and high-frame-rate animation. The technique relies on the limited ability of the visual system to perceive high spatio-temporal frequencies. Despite its conceptual simplicity, correct execution of the technique requires a number of non-trivial steps: display photometric temporal response must be modeled, flicker and motion artifacts must be avoided, and the generated signal must not exceed the dynamic range of the display. Our experiments, performed on a high-frame-rate LCD monitor and OLED-based VR headsets, explore the parameter space of the proposed technique and demonstrate that its perceived quality is indistinguishable from full-resolution rendering. The technique is an attractive alternative to reprojection and resolution reduction of all frames.

From pairwise comparisons and rating to a unified quality scale.

The goal of psychometric scaling is the quantification of perceptual experiences, understanding the relationship between an external stimulus, the internal representation and the response. In this paper, we propose a probabilistic framework to fuse the outcome of different psychophysical experimental protocols, namely rating and pairwise comparisons experiments. Such a method can be used for merging existing datasets of subjective nature and for experiments in which both measurements are collected. We analyze and compare the outcomes of both types of experimental protocols in terms of time and accuracy in a set of simulations and experiments with benchmark and real-world image quality assessment datasets, showing the necessity of scaling and the advantages of each protocol and mixing. Although most of our examples focus on image quality assessment, our findings generalize to any other subjective quality-of-experience task.

Artifacts in slab average-intensity-projection images reformatted from JPEG 2000 compressed thin-section abdominal CT data sets.

OBJECTIVE: The objective of our study was to assess the effects of compressing source thin-section abdominal CT images on final transverse average-intensity-projection (AIP) images. MATERIALS AND METHODS: At reversible, 4:1, 6:1, 8:1, 10:1, and 15:1 Joint Photographic Experts Group (JPEG) 2000 compressions, we compared the artifacts in 20 matching compressed thin sections (0.67 mm), compressed thick sections (5 mm), and AIP images (5 mm) reformatted from the compressed thin sections. The artifacts were quantitatively measured with peak signal-to-noise ratio (PSNR) and a perceptual quality metric (High Dynamic Range Visual Difference Predictor [HDR-VDP]). By comparing the compressed and original images, three radiologists independently graded the artifacts as 0 (none, indistinguishable), 1 (barely perceptible), 2 (subtle), or 3 (significant). Friedman tests and exact tests for paired proportions were used. RESULTS: At irreversible compressions, the artifacts tended to increase in the order of AIP, thick-section, and thin-section images in terms of PSNR (p < 0.0001), HDR-VDP (p < 0.0001), and the readers' grading (p < 0.01 at 6:1 or higher compressions). At 6:1 and 8:1, distinguishable pairs (grades 1-3) tended to increase in the order of AIP, thick-section, and thin-section images. Visually lossless threshold for the compression varied between images but decreased in the order of AIP, thick-section, and thin-section images (p < 0.0001). CONCLUSION: Compression artifacts in thin sections are significantly attenuated in AIP images. On the premise that thin sections are typically reviewed using an AIP technique, it is justifiable to compress them to a compression level currently accepted for thick sections.

Regional difference in compression artifacts in low-dose chest CT images: effects of mathematical and perceptual factors.

OBJECTIVE: The objective of our study was to investigate the difference of perceptible artifacts between the lungs and the chest wall and mediastinum in Joint Photographic Experts Group (JPEG) 2000-compressed low-dose chest CT images and to show that a perceptual image quality metric-the High-Dynamic Range Visual Difference Predictor (HDR-VDP)-can reproduce this regional difference. MATERIALS AND METHODS: Twenty images were compressed reversibly and irreversibly to 6:1-30:1. To analyze the two regions separately (lungs; and chest wall and mediastinum), the compressed pixels outside each tested region were replaced with the original pixels. By comparing the compressed and original images, three radiologists independently rated the compression artifacts as grade 0, none, indistinguishable; 1, barely perceptible; 2, subtle; or 3, significant. At each compression level, the two regions were compared for the readers' responses, peak signal-to-noise ratio (PSNR), and HDR-VDP results. Wilcoxon's signed rank tests and exact tests for paired proportions were used with a p value threshold of 0.05. RESULTS: Artifacts were rated as lower grades in the lungs than in the chest wall and mediastinum, showing statistical significances at 10:1-20:1 for reader 1, 8:1-15:1 for reader 2, and 8:1-20:1 for reader 3. Grade 0 was more frequent in the lungs, showing statistical significances at 10:1 for reader 1 and at 8:1-10:1 for readers 2 and 3. The results of PSNR indicated greater artifacts in the lungs (p < 0.001), whereas HDR-VDP results indicated fewer artifacts in the lungs (p < 0.001). CONCLUSION: Although compression artifacts are mathematically greater in the lungs than in the chest wall and mediastinum, radiologists' artifact perceptions are the opposite, which can be successfully reproduced by HDR-VDP.

Prediction of perceptible artifacts in JPEG 2000-compressed chest CT images using mathematical and perceptual quality metrics.

OBJECTIVE: The objective of our study was to determine whether peak signal-to-noise ratio (PSNR) and a perceptual quality metric (High-Dynamic Range Visual Difference Predictor [HDR-VDP]) can predict the presence of perceptible artifacts in Joint Photographic Experts Group (JPEG) 2000-compressed chest CT images. MATERIALS AND METHODS: One hundred chest CT images were compressed to 5:1, 8:1, 10:1, and 15:1. Five radiologists determined if the original and compressed images were identical (negative response) or different (positive response). The correlation between the results for each metric and the number of readers with positive responses was evaluated using Spearman's rank correlation test. Using the pooled readers' responses as the reference standard, we performed receiver operating characteristic (ROC) analysis to determine the cutoff values balancing sensitivity and specificity and yielding 100% sensitivity in each metric. These cutoff values were then used to estimate the visually lossless thresholds for the compressions for the 100 original images, and the accuracy of the estimates of two metrics was compared (McNemar test). RESULTS: The correlation coefficients were -0.918 and 0.925 for PSNR and the HDR-VDP, respectively. The areas under the ROC curves for the two metrics were 0.983 and 0.984, respectively (p = 0.11). The PSNR and HDR-VDP accurately predicted the visually lossless threshold for 69% and 72% of the 100 images (p = 0.68), respectively, at the cutoff values balancing sensitivity and specificity and for 43% and 47% (p = 0.22), respectively, at the cutoff values reaching 100% sensitivity. CONCLUSION: Both metrics are promising in predicting the perceptible compression artifacts and therefore can potentially be used to estimate the visually lossless threshold.

Prediction of perceptible artifacts in JPEG2000 compressed abdomen CT images using a perceptual image quality metric.

RATIONALE AND OBJECTIVES: To test a perceptual quality metric (high-dynamic range visual difference predictor, HDR-VDP) in predicting perceptible artifacts in Joint Photographic Experts Group 2000 compressed thin- and thick-section abdomen computed tomography images. MATERIALS AND METHODS: A total of 120 thin (0.67 mm) and corresponding thick (5 mm) sections were compressed to ratios from 4:1 to 15:1. Peak signal-to-noise ratio (PSNR), HDR-VDP results (paired t-tests), and five radiologists' pooled responses for the presence of artifacts (exact tests for paired proportions) were compared between the thin and thick sections. For three subsets of 120 thin- (subset A), 120 thick- (subset B), and 60 thin- and 60 thick-section compressed images (subset C), receiver operating curve analysis was performed to compare PSNR and HDR-VDP in predicting the radiologists' responses. Using the cutoff values where the sum of sensitivity and specificity was the maximum in subset C, visually lossless thresholds (VLTs) were estimated for the 240 original images and the estimation accuracy was compared (McNemar test). RESULTS: Thin sections showed more artifacts in terms of PSNR, HDR-VDP, and radiologists' responses (p < .0001). HDR-VDP outperformed PSNR for subset C (area under the curve: 0.97 versus 0.93, p = 0.03), whereas they did not differ significantly for subset A or B. Using the cutoff values, PSNR and HDR-VDP predicted the VLT accurately for 124 (51.7%) and 183 (76.3%) images, respectively (p < .0001). CONCLUSIONS: HDR-VDP can predict the perceptible compression artifacts, and therefore can be potentially used to estimate the VLT for such compressions.

Analysis of reported error in Monte Carlo rendered images.

Evaluating image quality in Monte Carlo rendered images is an important aspect of the rendering process as we often need to determine the relative quality between images computed using different algorithms and with varying amounts of computation. The use of a gold-standard, reference image, or ground truth is a common method to provide a baseline with which to compare experimental results. We show that if not chosen carefully, the quality of reference images used for image quality assessment can skew results leading to significant misreporting of error. We present an analysis of error in Monte Carlo rendered images and discuss practices to avoid or be aware of when designing an experiment.

Optimizing a tone curve for backward-compatible high dynamic range image and video compression.

For backward compatible high dynamic range (HDR) video compression, the HDR sequence is reconstructed by inverse tone-mapping a compressed low dynamic range (LDR) version of the original HDR content. In this paper, we show that the appropriate choice of a tone-mapping operator (TMO) can significantly improve the reconstructed HDR quality. We develop a statistical model that approximates the distortion resulting from the combined processes of tone-mapping and compression. Using this model, we formulate a numerical optimization problem to find the tone-curve that minimizes the expected mean square error (MSE) in the reconstructed HDR sequence. We also develop a simplified model that reduces the computational complexity of the optimization problem to a closed-form solution. Performance evaluations show that the proposed methods provide superior performance in terms of HDR MSE and SSIM compared to existing tone-mapping schemes. It is also shown that the LDR image quality resulting from the proposed methods matches that produced by perceptually-based TMOs.

A comparison of three image fidelity metrics of different computational principles for JPEG2000 compressed abdomen CT images.

This study aimed to evaluate three image fidelity metrics of different computational principles--peak signal-to-noise ratio (PSNR), high-dynamic range visual difference predictor (HDR-VDP), and multiscale structural similarity (MS-SSIM)--in measuring the fidelity of JPEG2000 compressed abdomen computed tomography images from a viewpoint of visually lossless compression. Three hundred images with 0.67- or 5-mm section thickness were compressed to one of five compression ratios ranging from reversible compression to 15:1. The fidelity of each compressed image was measured by five radiologists' visual analyses (distinguishable or indistinguishable from the original) and the three metrics. The Spearman rank correlation coefficients of the PSNR, HDR-VDP, and MS-SSIM values with the number of readers responding as indistinguishable were 0.86, 0.94, and 0.86, respectively. Using the pooled readers' responses as the reference standard, the area under the receiver-operating-characteristic curve for the HDR-VDP (0.99) was significantly greater than that for the PSNR (0.95) (p < 0.001) and for the MS-SSIM (0.96) (p = 0.003), and there was no significant difference between the PSNR and MS-SSIM (p = 0.70). In measuring the image fidelity, the HDR-VDP outperforms the PSNR and MS-SSIM, and the MS-SSIM and PSNR are comparable.