Study of Subjective and Objective Quality Assessment of Audio-Visual Signals.

The topics of visual and audio quality assessment (QA) have been widely researched for decades, yet nearly all of this prior work has focused only on single-mode visual or audio signals. However, visual signals rarely are presented without accompanying audio, including heavy-bandwidth video streaming applications. Moreover, the distortions that may separately (or conjointly) afflict the visual and audio signals collectively shape user-perceived quality of experience (QoE). This motivated us to conduct a subjective study of audio and video (A/V) quality, which we then used to compare and develop A/V quality measurement models and algorithms. The new LIVE-SJTU Audio and Video Quality Assessment (A/V-QA) Database includes 336 A/V sequences that were generated from 14 original source contents by applying 24 different A/V distortion combinations on them. We then conducted a subjective A/V quality perception study on the database towards attaining a better understanding of how humans perceive the overall combined quality of A/V signals. We also designed four different families of objective A/V quality prediction models, using a multimodal fusion strategy. The different types of A/V quality models differ in both the unimodal audio and video quality prediction models comprising the direct signal measurements and in the way that the two perceptual signal modes are combined. The objective models are built using both existing state-of-the-art audio and video quality prediction models and some new prediction models, as well as quality-predictive features delivered by a deep neural network. The methods of fusing audio and video quality predictions that are considered include simple product combinations as well as learned mappings. Using the new subjective A/V database as a tool, we validated and tested all of the objective A/V quality prediction models. We will make the database publicly available to facilitate further research.

Detection of Gabor patterns of different sizes, shapes, phases and eccentricities.

Contrast thresholds of vertical Gabor patterns were measured as a function of their eccentricity, size, shape, and phase using a 2AFC method. The patterns were 4 c/deg and they were presented for 90 or 240 ms. Log thresholds increase linearly with eccentricity at a mean rate of 0.47 dB/wavelength. For patterns centered on the fovea, thresholds decrease as the area of the pattern increases over the entire standard deviation range of 12 wavelengths. The TvA functions are concave up on log-log coordinates. For small patterns there is an interaction between shape and size that depends on phase. Threshold contrast energy is a U-shaped function of area with a minimum in the vicinity of 0.4 wavelength indicating detection by small receptive fields. Observers can discriminate among patterns of different sizes when the patterns are at threshold indicating that more than one mechanism is involved. The results are accounted for by a model in which patterns excite an array of slightly elongated receptive fields that are identical except that their sensitivity decreases exponentially with eccentricity. Excitation is raised to a power and then summed linearly across receptive fields to determine the threshold. The results are equally well described by an internal-noise-limited model. The TvA functions are insufficient to separately estimate the noise and the exponent of the power function. However, an experiment that shows that mixing sizes within the trial sequence has no effect on thresholds, suggests that the limiting noise does not increase with the number of mechanisms monitored.