Deep learning models for webcam eye tracking in online experiments.

Eye tracking is prevalent in scientific and commercial applications. Recent computer vision and deep learning methods enable eye tracking with off-the-shelf webcams and reduce dependence on expensive, restrictive hardware. However, such deep learning methods have not yet been applied and evaluated for remote, online psychological experiments. In this study, we tackle critical challenges faced in remote eye tracking setups and systematically evaluate appearance-based deep learning methods of gaze tracking and blink detection. From their own homes and laptops, 65 participants performed a battery of eye tracking tasks including (i) fixation, (ii) zone classification, (iii) free viewing, (iv) smooth pursuit, and (v) blink detection. Webcam recordings of the participants performing these tasks were processed offline through appearance-based models of gaze and blink detection. The task battery required different eye movements that characterized gaze and blink prediction accuracy over a comprehensive list of measures. We find the best gaze accuracy to be 2.4° and precision of 0.47°, which outperforms previous online eye tracking studies and reduces the gap between laboratory-based and online eye tracking performance. We release the experiment template, recorded data, and analysis code with the motivation to escalate affordable, accessible, and scalable eye tracking that has the potential to accelerate research in the fields of psychological science, cognitive neuroscience, user experience design, and human-computer interfaces.

Pleasantness of nonlinear distortion in isolated triads of synthetic timbre.

Distortion of sound is an important tool to increase the variety of timbres in musical compositions, but perceived pleasantness of distortion is understudied, and studies are limited to guitar practices in rock and metal music. This study applied a more systematic approach, using synthetic timbre and creating an audio-plugin that realized nonlinear symmetric and asymmetric distortion. Participants evaluated the perceived pleasantness of isolated triads differing in distortion (undistorted, symmetric, asymmetric), tonality (minor, major), and position (low, high, wide), taking baseline differences of tonality and position into account. Perceived pleasantness decreased by distortion, and the decrease was stronger for minor than major triads and stronger for asymmetric than symmetric distortion. Position played only a minor role in the evaluations, except for stimuli in high positions. Stimulus-based analyses showed a relation between pleasantness and the variability of roughness, mean spectral centroid, and mean sound intensity. Subject-based analyses revealed a smaller decrease in pleasantness with a preference for electronic music. Importantly, some distorted triads were rated as pleasant in absolute terms: major triads with symmetric distortion in low or wide position. That is, indeed, distortion is not always categorized as unpleasant but can be perceived as pleasant.

Eye blinking, musical processing, and subjective states-A methods account.

Affective sciences often make use of self-reports to assess subjective states. Seeking a more implicit measure for states and emotions, our study explored spontaneous eye blinking during music listening. However, blinking is understudied in the context of research on subjective states. Therefore, a second goal was to explore different ways of analyzing blink activity recorded from infra-red eye trackers, using two additional data sets from earlier studies differing in blinking and viewing instructions. We first replicate the effect of increased blink rates during music listening in comparison with silence and show that the effect is not related to changes in self-reported valence, arousal, or to specific musical features. Interestingly, but in contrast, felt absorption reduced participants' blinking. The instruction to inhibit blinking did not change results. From a methodological perspective, we make suggestions about how to define blinks from data loss periods recorded by eye trackers and report a data-driven outlier rejection procedure and its efficiency for subject-mean analyses, as well as trial-based analyses. We ran a variety of mixed effects models that differed in how trials without blinking were treated. The main results largely converged across accounts. The broad consistency of results across different experiments, outlier treatments, and statistical models demonstrates the reliability of the reported effects. As recordings of data loss periods come for free when interested in eye movements or pupillometry, we encourage researchers to pay attention to blink activity and contribute to the further understanding of the relation between blinking, subjective states, and cognitive processing.

In touch: Cardiac and respiratory patterns synchronize during ensemble singing with physical contact.

Musical ensemble performances provide an ideal environment to gain knowledge about complex human interactions. Network structures of synchronization can reflect specific roles of individual performers on the one hand and a higher level of organization of all performers as a superordinate system on the other. This study builds on research on joint singing, using hyperscanning of respiration and heart rate variability (HRV) from eight professional singers. Singers performed polyphonic music, distributing their breathing within the same voice and singing without and with physical contact: that is touching each other's shoulder or waist. The idea of singing with touch was motivated by historical depictions of ensemble performances that showed singers touching each other. It raises the question of the potential benefit of touch for group performances. From a psycho-physiological point of view, physical contact should increase the synchronization of singing coordination. The results confirm previous findings on synchronization of respiration and HRV during choir singing and extend those findings to a non-homophonic musical repertoire while also revealing an increase in synchronization in respiration during physical contact. These effects were significant across different frequency ranges. The effect of physical contact was stronger when all singers were singing in comparison to the partial ensemble. Importantly, the synchronization could not be fully explained by the singing action (i.e., singing the same voice, or singing vs. listening) or by the standing position or touch. This finding suggests a higher level of organization of all singers, forming a superordinate system.

Multisensory integration of musical emotion perception in singing.

We investigated how visual and auditory information contributes to emotion communication during singing. Classically trained singers applied two different facial expressions (expressive/suppressed) to pieces from their song and opera repertoire. Recordings of the singers were evaluated by laypersons or experts, presented to them in three different modes: auditory, visual, and audio-visual. A manipulation check confirmed that the singers succeeded in manipulating the face while keeping the sound highly expressive. Analyses focused on whether the visual difference or the auditory concordance between the two versions determined perception of the audio-visual stimuli. When evaluating expressive intensity or emotional content a clear effect of visual dominance showed. Experts made more use of the visual cues than laypersons. Consistency measures between uni-modal and multimodal presentations did not explain the visual dominance. The evaluation of seriousness was applied as a control. The uni-modal stimuli were rated as expected, but multisensory evaluations converged without visual dominance. Our study demonstrates that long-term knowledge and task context affect multisensory integration. Even though singers' orofacial movements are dominated by sound production, their facial expressions can communicate emotions composed into the music, and observes do not rely on audio information instead. Studies such as ours are important to understand multisensory integration in applied settings.

Manual motor reaction while being absorbed into popular music.

In three experiments, we investigated the behavioral consequences of being absorbed into music on performance in a concurrent task. We tested two competing hypotheses: Based on a cognitive load account, captivation of attention by the music and state absorption might slow down reactions in the decisional task. Alternatively, music could induce spontaneous motor activity, and being absorbed in music might result in a more autonomous, flow-driven behavior with quicker motor reactions. Participants performed a simple, visual, two-alternative forced-choice task while listening to popular musical excerpts. Subsequently, they rated their subjective experience using a short questionnaire. We presented music in four tempo categories (between 80 and 140 BPM) to account for a potential effect of tempo and an interaction between tempo and absorption. In Experiment 1, absorption was related to decreased reaction times (RTs) in the visual task. This effect was small, as expected in this setting, but replicable in Experiment 2. There was no effect of the music's tempo on RTs but a tendency of mind wandering to relate to task performance. After slightly changing the study setting in Experiment 3, flow predicted decreased RTs, but absorption alone - as part of the flow construct - did not predict RTs. To sum up, we demonstrated that being absorbed in music can have the behavioral consequence of speeded manual reactions in specific task contexts, and people seem to integrate the music into an active, flow-driven and therefore enhanced performance. However, shown relations depend on task settings, and a systematic study of context is necessary to understand how induced states and their measurement contribute to the findings.

Gaze-based and attention-based rehearsal in spatial working memory.

How do we maintain information about spatial configurations in mind? Many working memory (WM) models assume that rehearsal processes are used to counteract forgetting in WM. Here, we investigated the contributions of gaze-based and attention-based rehearsal for protecting spatial representations from time-based forgetting. Participants memorized 6 locations selected from a grid of 30 scattered dots. Memory was tested after 1.5 or 4.5 s, and this interval was either blank or the grid remained onscreen (which is assumed to provide rehearsal support). In 2 experiments, we monitored eye movements during the retention phase, or asked participants to fixate the screen center. In 3 subsequent experiments, we tested spatial WM under dual-task conditions inhibiting shifts of visuospatial attention or central attention to the memoranda. Memory was better and more resistant to time-based forgetting in the grid than blank condition. Recording of fixations showed more frequent and efficient gaze-based rehearsal in the presence of the grid. Fixations toward distractor locations occurred at a similar frequency in the blank and grid conditions, and it did not predict incorrect recalls. Inhibition of eye-movements or shifts of visuospatial attention impaired memory overall, but it did not change the grid benefit nor the rate of time-based forgetting. In contrast, distracting central attention increased time-based forgetting regardless of grid presence. These results indicate that (a) the grid benefit is only partially explained by rehearsal; (b) gaze-errors (i.e., distractor fixations) do not lead to more forgetting; and (c) the maintenance of spatial representations over time depends on central processing. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

On-item fixations during serial encoding do not affect spatial working memory.

Ample evidence suggests that there is overlap between the eye-movement system and spatial working memory. Such overlapping structures or capacities may result in interference on the one hand and beneficial support on the other. We investigated eye-movement control during encoding of verbal or spatial information, keeping the display the same between tasks. Saccades to to-be-encoded items were scarce during spatial encoding in comparison with verbal encoding. However, despite replicating this difference across different tasks (serial, free recall) and presentation modalities (simultaneous, sequential presentation), we found no relation between item fixations and memory performance-that is, no costs or benefits. Inducing a change from covert to overt encoding did not affect spatial memory performance as well. In contrast, regressive fixations on prior items, that were no longer on the screen, were associated with increased spatial memory performance. Regressions occurred mainly at the end of the encoding period and were targeted at the first presented item. Our results suggest a dissociation between two types of fixations that accompany serial spatial memory: On-item fixations are epiphenomenal; regressions indicate rehearsal or output preparation.

The application of eye-tracking in music research.

Though eye-tracking is typically a methodology applied in the visual research domain, recent studies suggest its relevance in the context of music research. There exists a communityof researchers interested in this kind of research from varied disciplinary backgrounds scattered across the globe. Therefore, in August 2017, an international conference was held at the Max Planck Institute for Empirical Aesthetics in Frankfurt, Germany,to bring this research community together. The conference was dedicated to the topic of music and eye-tracking, asking the question: what do eye movements, pupil dilation, and blinking activity tell us about musical processing? This special issue is constituted of top-scoring research from the conference and spans a range of music-related topics. From tracking the gaze of performers in musical trios to basic research on how eye movements are affected by background music, the contents of this special issue highlight a variety of experimental approaches and possible applications of eye-tracking in music research.

The rhythm of cognition - Effects of an auditory beat on oculomotor control in reading and sequential scanning.

Eye-movement behavior is inherently rhythmic. Even without cognitive input, the eyes never rest, as saccades are generated 3 to 4 times per second. Based on an embodied view of cognition, we asked whether mental processing in visual cognitive tasks is also rhythmic in nature by studying the effects of an external auditory beat (rhythmic background music) on saccade generation in exemplary cognitive tasks (reading and sequential scanning). While in applied settings background music has been demonstrated to impair reading comprehension, the effect of musical tempo on eye-movement control during reading or scanning has not been investigated so far. We implemented a tempo manipulation in four steps as well as a silent baseline condition, while participants completed a text reading or a sequential scanning task that differed from each other in terms of underlying cognitive processing requirements. The results revealed that increased tempo of the musical beat sped up fixations in text reading, while the presence (vs. absence) of the auditory stimulus generally reduced overall reading time. In contrast, sequential scanning was unaffected by the auditory pacemaker. These results were supported by additionally applying Bayesian inference statistics. Our study provides evidence against a cognitive load account (i.e., that spare resources during low-demand sequential scanning allow for enhanced processing of the external beat). Instead, the data suggest an interpretation in favor of a modulation of the oculomotor saccade timer by irrelevant background music in cases involving highly automatized oculomotor control routines (here: in text reading).

Microsaccade-rate indicates absorption by music listening.

The power of music is a literary topos, which can be attributed to intense and personally significant experiences, one of them being the state of absorption. Such phenomenal states are difficult to grasp objectively. We investigated the state of musical absorption by using eye tracking. We utilized a load related definition of state absorption: multimodal resources are committed to create a unified representation of music. Resource allocation was measured indirectly by microsaccade rate, known to indicate cognitive processing load. We showed in Exp. 1 that microsaccade rate also indicates state absorption. Hence, there is cross-modal coupling between an auditory aesthetic experience and fixational eye movements. When removing the fixational stimulus in Exp. 2, saccades are no longer generated upon visual input and the cross-modal coupling disappeared. Results are interpreted in favor of the load hypothesis of microsaccade rate and against the assumption of general slowing by state absorption.

From pre-attentive processes to durable representation: an ERP index of visual distraction.

Visual search and oddball paradigms were combined to investigate memory for to-be-ignored color changes in a group of 12 healthy participants. The onset of unexpected color change of an irrelevant stimulus evoked two reliable ERP effects: a component of the event-related potential (ERP), similar to the visual mismatch negativity response (vMMN), with a latency of 120-160 ms and a posterior distribution over the left hemisphere and Late Fronto-Central Negativity (LFCN) with a latency of 320-400 ms, apparent at fronto-central electrodes and some posterior sites. Color change of that irrelevant stimulus also slowed identification of a visual target, indicating distraction. The amplitude of this color-change vMMN, but not LFCN, indexed this distraction effect. That is, electrophysiological and behavioral measures were correlated. The interval between visual scenes approximated 1s (611-1629 ms), indicating that the brain's sensory memory for the color of the preceding visual scenes must persist for at least 600 ms. Therefore, in the case of the neural code for color, durable memory representations are formed in an obligatory manner.

Differentiating between verbal and spatial encoding using eye-movement recordings.

Visual information processing is guided by an active mechanism generating saccadic eye movements to salient stimuli. Here we investigate the specific contribution of saccades to memory encoding of verbal and spatial properties in a serial recall task. In the first experiment, participants moved their eyes freely without specific instruction. We demonstrate the existence of qualitative differences in eye-movement strategies during verbal and spatial memory encoding. While verbal memory encoding was characterized by shifting the gaze to the to-be-encoded stimuli, saccadic activity was suppressed during spatial encoding. In the second experiment, participants were required to suppress saccades by fixating centrally during encoding or to make precise saccades onto the memory items. Active suppression of saccades had no effect on memory performance, but tracking the upcoming stimuli decreased memory performance dramatically in both tasks, indicating a resource bottleneck between display-controlled saccadic control and memory encoding. We conclude that optimized encoding strategies for verbal and spatial features are underlying memory performance in serial recall, but such strategies work on an involuntary level only and do not support memory encoding when they are explicitly required by the task.

Capture of the gaze does not capture the mind.

Sudden visual changes attract our gaze, and related eye movement control requires attentional resources. Attention is a limited resource that is also involved in working memory--for instance, memory encoding. As a consequence, theory suggests that gaze capture could impair the buildup of memory respresentations due to an attentional resource bottleneck. Here we developed an experimental design combining a serial memory task (verbal or spatial) and concurrent gaze capture by a distractor (of high or low similarity to the relevant item). The results cannot be explained by a general resource bottleneck. Specifically, we observed that capture by the low-similar distractor resulted in delayed and reduced saccade rates to relevant items in both memory tasks. However, while spatial memory performance decreased, verbal memory remained unaffected. In contrast, the high-similar distractor led to capture and memory loss for both tasks. Our results lend support to the view that gaze capture leads to activation of irrelevant representations in working memory that compete for selection at recall. Activation of irrelevant spatial representations distracts spatial recall, whereas activation of irrelevant verbal features impairs verbal memory performance.

Ease of access to list items in short-term memory depends on the order of the recognition probes.

We report data from 4 experiments using a recognition design with multiple probes to be matched to specific study positions. Items could be accessed rapidly, independent of set size, when the test order matched the study order (forward condition). When the order of testing was random, backward, or in a prelearned irregular sequence (reordered conditions), the classic Sternberg result was obtained: Response times were slow and increased linearly with set size. A number of explanations for forward-condition facilitation were ruled out, such as the predictability of the study order (Experiment 2), the predictability of the probe order (Experiment 1), the covariation of study and test orders (Experiments 1, 2, and 4), processes of probe encoding and perception that did not rely on STM access (Experiments 1, 2, and 4), specific support of the forward condition by articulatory processes (Experiment 3), or condition-dependent strategic differences (Experiment 4). More detailed analyses demonstrated that fast forward responses could not be accounted for by the effects of input position and output position that modulated random responses, effects that did account for the slower responses of the reordered conditions (Experiments 1, 3, and 4). A final analysis of probe-to-probe transitions as a function of encoding distance revealed a sizeable single-step benefit in the random condition. We concluded that STM representations were serial rather than spatial and that forward probes benefited from their serial adjacency.

Dual representation of item positions in verbal short-term memory: Evidence for two access modes.

Memory sets of N = 1~5 digits were exposed sequentially from left-to-right across the screen, followed by N recognition probes. Probes had to be compared to memory list items on identity only (Sternberg task) or conditional on list position. Positions were probed randomly or in left-to-right order. Search functions related probe response times to set size. Random probing led to ramped, "Sternbergian" functions whose intercepts were elevated by the location requirement. Sequential probing led to flat search functions-fast responses unaffected by set size. These results suggested that items in STM could be accessed either by a slow search-on-identity followed by recovery of an associated location tag, or in a single step by following item-to-item links in study order. It is argued that this dual coding of location information occurs spontaneously at study, and that either code can be utilised at retrieval depending on test demands.

No age differences in complex memory search: older adults search as efficiently as younger adults.

In 2 experiments, the authors investigated age differences in memory search under 4 conditions: forward search, backward search, random search, and fixed irregular search. Both search slopes and serial position curves were investigated. Mixing conditions led to smaller age differences than blocking conditions, suggesting that younger adults have an advantage over older adults when strategies can be applied to memory scanning. All age differences in scanning rates, however, disappeared when age differences in a magnitude-judgment control task were controlled for, showing that age differences in memory scanning tasks are not because of the scanning process per se, but because of attention, sensorimotor speed, and decision processes. In both experiments, the serial position curves of older adults echoed those of younger adults closely, demonstrating that younger and older adults use the same scanning processes.

Activation and binding in verbal working memory: a dual-process model for the recognition of nonwords.

The article presents a mathematical model of short-term recognition based on dual-process models and the three-component theory of working memory [Oberauer, K. (2002). Access to information in working memory: Exploring the focus of attention. Journal of Experimental Psychology: Learning, Memory, and Cognition, 28, 411-421]. Familiarity arises from activated representations in long-term memory, ignoring their relations; recollection retrieves bindings in the capacity-limited component of working memory. In three experiments participants encoded two short lists of nonwords for immediate recognition, one of which was then cued as irrelevant. Probes from the irrelevant list were rejected more slowly than new probes; this was also found with probes recombining letters of irrelevant nonwords, suggesting that familiarity arises from individual letters independent of their relations. When asked to accept probes whose letters were all in the relevant list, regardless of their conjunction, participants accepted probes preserving the original conjunctions faster than recombinations, showing that recollection accessed feature bindings automatically. The model fit the data best when familiarity depended only on matching letters, whereas recollection used binding information.

Overwriting of phonemic features in serial recall.

We tested two explanations of the phonological similarity effect in verbal short-term memory: The confusion hypothesis assumes that serial positions of similar items are confused. The overwriting hypothesis states that similar items share feature representations, which are overwritten. Participants memorised a phonologically dissimilar list of CVC-trigrams (Experiment 1) or words (Experiment 2 and 3) for serial recall. In the retention interval they read aloud other items. The material of the distractor task jointly overlapped one item of the memory list. The recall of this item was impaired, and the effect was not based on intrusions from the distractor task alone. The results provide evidence for feature overwriting as one potential mechanism contributing to the phonological similarity effect.