Deep learning models for webcam eye tracking in online experiments.

Eye tracking is prevalent in scientific and commercial applications. Recent computer vision and deep learning methods enable eye tracking with off-the-shelf webcams and reduce dependence on expensive, restrictive hardware. However, such deep learning methods have not yet been applied and evaluated for remote, online psychological experiments. In this study, we tackle critical challenges faced in remote eye tracking setups and systematically evaluate appearance-based deep learning methods of gaze tracking and blink detection. From their own homes and laptops, 65 participants performed a battery of eye tracking tasks including (i) fixation, (ii) zone classification, (iii) free viewing, (iv) smooth pursuit, and (v) blink detection. Webcam recordings of the participants performing these tasks were processed offline through appearance-based models of gaze and blink detection. The task battery required different eye movements that characterized gaze and blink prediction accuracy over a comprehensive list of measures. We find the best gaze accuracy to be 2.4° and precision of 0.47°, which outperforms previous online eye tracking studies and reduces the gap between laboratory-based and online eye tracking performance. We release the experiment template, recorded data, and analysis code with the motivation to escalate affordable, accessible, and scalable eye tracking that has the potential to accelerate research in the fields of psychological science, cognitive neuroscience, user experience design, and human-computer interfaces.

Pleasantness of nonlinear distortion in isolated triads of synthetic timbre.

Distortion of sound is an important tool to increase the variety of timbres in musical compositions, but perceived pleasantness of distortion is understudied, and studies are limited to guitar practices in rock and metal music. This study applied a more systematic approach, using synthetic timbre and creating an audio-plugin that realized nonlinear symmetric and asymmetric distortion. Participants evaluated the perceived pleasantness of isolated triads differing in distortion (undistorted, symmetric, asymmetric), tonality (minor, major), and position (low, high, wide), taking baseline differences of tonality and position into account. Perceived pleasantness decreased by distortion, and the decrease was stronger for minor than major triads and stronger for asymmetric than symmetric distortion. Position played only a minor role in the evaluations, except for stimuli in high positions. Stimulus-based analyses showed a relation between pleasantness and the variability of roughness, mean spectral centroid, and mean sound intensity. Subject-based analyses revealed a smaller decrease in pleasantness with a preference for electronic music. Importantly, some distorted triads were rated as pleasant in absolute terms: major triads with symmetric distortion in low or wide position. That is, indeed, distortion is not always categorized as unpleasant but can be perceived as pleasant.

Eye blinking, musical processing, and subjective states-A methods account.

Affective sciences often make use of self-reports to assess subjective states. Seeking a more implicit measure for states and emotions, our study explored spontaneous eye blinking during music listening. However, blinking is understudied in the context of research on subjective states. Therefore, a second goal was to explore different ways of analyzing blink activity recorded from infra-red eye trackers, using two additional data sets from earlier studies differing in blinking and viewing instructions. We first replicate the effect of increased blink rates during music listening in comparison with silence and show that the effect is not related to changes in self-reported valence, arousal, or to specific musical features. Interestingly, but in contrast, felt absorption reduced participants' blinking. The instruction to inhibit blinking did not change results. From a methodological perspective, we make suggestions about how to define blinks from data loss periods recorded by eye trackers and report a data-driven outlier rejection procedure and its efficiency for subject-mean analyses, as well as trial-based analyses. We ran a variety of mixed effects models that differed in how trials without blinking were treated. The main results largely converged across accounts. The broad consistency of results across different experiments, outlier treatments, and statistical models demonstrates the reliability of the reported effects. As recordings of data loss periods come for free when interested in eye movements or pupillometry, we encourage researchers to pay attention to blink activity and contribute to the further understanding of the relation between blinking, subjective states, and cognitive processing.

In touch: Cardiac and respiratory patterns synchronize during ensemble singing with physical contact.

Musical ensemble performances provide an ideal environment to gain knowledge about complex human interactions. Network structures of synchronization can reflect specific roles of individual performers on the one hand and a higher level of organization of all performers as a superordinate system on the other. This study builds on research on joint singing, using hyperscanning of respiration and heart rate variability (HRV) from eight professional singers. Singers performed polyphonic music, distributing their breathing within the same voice and singing without and with physical contact: that is touching each other's shoulder or waist. The idea of singing with touch was motivated by historical depictions of ensemble performances that showed singers touching each other. It raises the question of the potential benefit of touch for group performances. From a psycho-physiological point of view, physical contact should increase the synchronization of singing coordination. The results confirm previous findings on synchronization of respiration and HRV during choir singing and extend those findings to a non-homophonic musical repertoire while also revealing an increase in synchronization in respiration during physical contact. These effects were significant across different frequency ranges. The effect of physical contact was stronger when all singers were singing in comparison to the partial ensemble. Importantly, the synchronization could not be fully explained by the singing action (i.e., singing the same voice, or singing vs. listening) or by the standing position or touch. This finding suggests a higher level of organization of all singers, forming a superordinate system.

Multisensory integration of musical emotion perception in singing.

We investigated how visual and auditory information contributes to emotion communication during singing. Classically trained singers applied two different facial expressions (expressive/suppressed) to pieces from their song and opera repertoire. Recordings of the singers were evaluated by laypersons or experts, presented to them in three different modes: auditory, visual, and audio-visual. A manipulation check confirmed that the singers succeeded in manipulating the face while keeping the sound highly expressive. Analyses focused on whether the visual difference or the auditory concordance between the two versions determined perception of the audio-visual stimuli. When evaluating expressive intensity or emotional content a clear effect of visual dominance showed. Experts made more use of the visual cues than laypersons. Consistency measures between uni-modal and multimodal presentations did not explain the visual dominance. The evaluation of seriousness was applied as a control. The uni-modal stimuli were rated as expected, but multisensory evaluations converged without visual dominance. Our study demonstrates that long-term knowledge and task context affect multisensory integration. Even though singers' orofacial movements are dominated by sound production, their facial expressions can communicate emotions composed into the music, and observes do not rely on audio information instead. Studies such as ours are important to understand multisensory integration in applied settings.

Manual motor reaction while being absorbed into popular music.

In three experiments, we investigated the behavioral consequences of being absorbed into music on performance in a concurrent task. We tested two competing hypotheses: Based on a cognitive load account, captivation of attention by the music and state absorption might slow down reactions in the decisional task. Alternatively, music could induce spontaneous motor activity, and being absorbed in music might result in a more autonomous, flow-driven behavior with quicker motor reactions. Participants performed a simple, visual, two-alternative forced-choice task while listening to popular musical excerpts. Subsequently, they rated their subjective experience using a short questionnaire. We presented music in four tempo categories (between 80 and 140 BPM) to account for a potential effect of tempo and an interaction between tempo and absorption. In Experiment 1, absorption was related to decreased reaction times (RTs) in the visual task. This effect was small, as expected in this setting, but replicable in Experiment 2. There was no effect of the music's tempo on RTs but a tendency of mind wandering to relate to task performance. After slightly changing the study setting in Experiment 3, flow predicted decreased RTs, but absorption alone - as part of the flow construct - did not predict RTs. To sum up, we demonstrated that being absorbed in music can have the behavioral consequence of speeded manual reactions in specific task contexts, and people seem to integrate the music into an active, flow-driven and therefore enhanced performance. However, shown relations depend on task settings, and a systematic study of context is necessary to understand how induced states and their measurement contribute to the findings.

Gaze-based and attention-based rehearsal in spatial working memory.

How do we maintain information about spatial configurations in mind? Many working memory (WM) models assume that rehearsal processes are used to counteract forgetting in WM. Here, we investigated the contributions of gaze-based and attention-based rehearsal for protecting spatial representations from time-based forgetting. Participants memorized 6 locations selected from a grid of 30 scattered dots. Memory was tested after 1.5 or 4.5 s, and this interval was either blank or the grid remained onscreen (which is assumed to provide rehearsal support). In 2 experiments, we monitored eye movements during the retention phase, or asked participants to fixate the screen center. In 3 subsequent experiments, we tested spatial WM under dual-task conditions inhibiting shifts of visuospatial attention or central attention to the memoranda. Memory was better and more resistant to time-based forgetting in the grid than blank condition. Recording of fixations showed more frequent and efficient gaze-based rehearsal in the presence of the grid. Fixations toward distractor locations occurred at a similar frequency in the blank and grid conditions, and it did not predict incorrect recalls. Inhibition of eye-movements or shifts of visuospatial attention impaired memory overall, but it did not change the grid benefit nor the rate of time-based forgetting. In contrast, distracting central attention increased time-based forgetting regardless of grid presence. These results indicate that (a) the grid benefit is only partially explained by rehearsal; (b) gaze-errors (i.e., distractor fixations) do not lead to more forgetting; and (c) the maintenance of spatial representations over time depends on central processing. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

On-item fixations during serial encoding do not affect spatial working memory.

Ample evidence suggests that there is overlap between the eye-movement system and spatial working memory. Such overlapping structures or capacities may result in interference on the one hand and beneficial support on the other. We investigated eye-movement control during encoding of verbal or spatial information, keeping the display the same between tasks. Saccades to to-be-encoded items were scarce during spatial encoding in comparison with verbal encoding. However, despite replicating this difference across different tasks (serial, free recall) and presentation modalities (simultaneous, sequential presentation), we found no relation between item fixations and memory performance-that is, no costs or benefits. Inducing a change from covert to overt encoding did not affect spatial memory performance as well. In contrast, regressive fixations on prior items, that were no longer on the screen, were associated with increased spatial memory performance. Regressions occurred mainly at the end of the encoding period and were targeted at the first presented item. Our results suggest a dissociation between two types of fixations that accompany serial spatial memory: On-item fixations are epiphenomenal; regressions indicate rehearsal or output preparation.

The application of eye-tracking in music research.

Though eye-tracking is typically a methodology applied in the visual research domain, recent studies suggest its relevance in the context of music research. There exists a communityof researchers interested in this kind of research from varied disciplinary backgrounds scattered across the globe. Therefore, in August 2017, an international conference was held at the Max Planck Institute for Empirical Aesthetics in Frankfurt, Germany,to bring this research community together. The conference was dedicated to the topic of music and eye-tracking, asking the question: what do eye movements, pupil dilation, and blinking activity tell us about musical processing? This special issue is constituted of top-scoring research from the conference and spans a range of music-related topics. From tracking the gaze of performers in musical trios to basic research on how eye movements are affected by background music, the contents of this special issue highlight a variety of experimental approaches and possible applications of eye-tracking in music research.

The rhythm of cognition - Effects of an auditory beat on oculomotor control in reading and sequential scanning.

Eye-movement behavior is inherently rhythmic. Even without cognitive input, the eyes never rest, as saccades are generated 3 to 4 times per second. Based on an embodied view of cognition, we asked whether mental processing in visual cognitive tasks is also rhythmic in nature by studying the effects of an external auditory beat (rhythmic background music) on saccade generation in exemplary cognitive tasks (reading and sequential scanning). While in applied settings background music has been demonstrated to impair reading comprehension, the effect of musical tempo on eye-movement control during reading or scanning has not been investigated so far. We implemented a tempo manipulation in four steps as well as a silent baseline condition, while participants completed a text reading or a sequential scanning task that differed from each other in terms of underlying cognitive processing requirements. The results revealed that increased tempo of the musical beat sped up fixations in text reading, while the presence (vs. absence) of the auditory stimulus generally reduced overall reading time. In contrast, sequential scanning was unaffected by the auditory pacemaker. These results were supported by additionally applying Bayesian inference statistics. Our study provides evidence against a cognitive load account (i.e., that spare resources during low-demand sequential scanning allow for enhanced processing of the external beat). Instead, the data suggest an interpretation in favor of a modulation of the oculomotor saccade timer by irrelevant background music in cases involving highly automatized oculomotor control routines (here: in text reading).

Microsaccade-rate indicates absorption by music listening.

The power of music is a literary topos, which can be attributed to intense and personally significant experiences, one of them being the state of absorption. Such phenomenal states are difficult to grasp objectively. We investigated the state of musical absorption by using eye tracking. We utilized a load related definition of state absorption: multimodal resources are committed to create a unified representation of music. Resource allocation was measured indirectly by microsaccade rate, known to indicate cognitive processing load. We showed in Exp. 1 that microsaccade rate also indicates state absorption. Hence, there is cross-modal coupling between an auditory aesthetic experience and fixational eye movements. When removing the fixational stimulus in Exp. 2, saccades are no longer generated upon visual input and the cross-modal coupling disappeared. Results are interpreted in favor of the load hypothesis of microsaccade rate and against the assumption of general slowing by state absorption.

From pre-attentive processes to durable representation: an ERP index of visual distraction.

Visual search and oddball paradigms were combined to investigate memory for to-be-ignored color changes in a group of 12 healthy participants. The onset of unexpected color change of an irrelevant stimulus evoked two reliable ERP effects: a component of the event-related potential (ERP), similar to the visual mismatch negativity response (vMMN), with a latency of 120-160 ms and a posterior distribution over the left hemisphere and Late Fronto-Central Negativity (LFCN) with a latency of 320-400 ms, apparent at fronto-central electrodes and some posterior sites. Color change of that irrelevant stimulus also slowed identification of a visual target, indicating distraction. The amplitude of this color-change vMMN, but not LFCN, indexed this distraction effect. That is, electrophysiological and behavioral measures were correlated. The interval between visual scenes approximated 1s (611-1629 ms), indicating that the brain's sensory memory for the color of the preceding visual scenes must persist for at least 600 ms. Therefore, in the case of the neural code for color, durable memory representations are formed in an obligatory manner.

Differentiating between verbal and spatial encoding using eye-movement recordings.

Visual information processing is guided by an active mechanism generating saccadic eye movements to salient stimuli. Here we investigate the specific contribution of saccades to memory encoding of verbal and spatial properties in a serial recall task. In the first experiment, participants moved their eyes freely without specific instruction. We demonstrate the existence of qualitative differences in eye-movement strategies during verbal and spatial memory encoding. While verbal memory encoding was characterized by shifting the gaze to the to-be-encoded stimuli, saccadic activity was suppressed during spatial encoding. In the second experiment, participants were required to suppress saccades by fixating centrally during encoding or to make precise saccades onto the memory items. Active suppression of saccades had no effect on memory performance, but tracking the upcoming stimuli decreased memory performance dramatically in both tasks, indicating a resource bottleneck between display-controlled saccadic control and memory encoding. We conclude that optimized encoding strategies for verbal and spatial features are underlying memory performance in serial recall, but such strategies work on an involuntary level only and do not support memory encoding when they are explicitly required by the task.

Factors associated with self-assessed increase in tobacco consumption among over-indebted individuals in Germany: a cross-sectional study.

BACKGROUND: Over-indebtedness is an increasing phenomenon in industrialised nations causing individual hardship and societal problems. Nonetheless, few studies have explored smoking among over-indebted individuals. METHODS: A cross-sectional survey (n=949) on retrospectively assessed changes in tobacco consumption was carried out in 2006 and 2007 among clients of 84 officially approved debt and insolvency counselling centres in Germany (response rate 39.7%). Logistic regressions were performed to explore factors associated with reports of increased smoking after onset of over-indebtedness. RESULTS: 63% of all respondents stated daily or occasional tobacco consumption. Almost one fifth reported an increase in smoking after becoming over-indebted. Females were less likely to report increased smoking than men (aOR 0.66, 95% CI 0.44-0.99) whereas respondents who had been over-indebted for more than 10 years were more likely to report increased smoking than those who had been over-indebted for less than five years (aOR 1.66; 95%-CI 1.00-2.76). The odds of increased smoking were also elevated among those who reported that their families and friends had withdrawn from them as a consequence of their over-indebtedness (aOR 1.82; 95%-CI 1.06-3.14). CONCLUSIONS: The study identifies over-indebted individuals and particularly over-indebted men as a high-risk group of smokers. Low levels of social embeddedness/support were associated with a further increase in smoking after becoming over-indebted. Given recent increases of over-indebtedness, the findings highlight the need to develop appropriate public health policies.

Capture of the gaze does not capture the mind.

Sudden visual changes attract our gaze, and related eye movement control requires attentional resources. Attention is a limited resource that is also involved in working memory--for instance, memory encoding. As a consequence, theory suggests that gaze capture could impair the buildup of memory respresentations due to an attentional resource bottleneck. Here we developed an experimental design combining a serial memory task (verbal or spatial) and concurrent gaze capture by a distractor (of high or low similarity to the relevant item). The results cannot be explained by a general resource bottleneck. Specifically, we observed that capture by the low-similar distractor resulted in delayed and reduced saccade rates to relevant items in both memory tasks. However, while spatial memory performance decreased, verbal memory remained unaffected. In contrast, the high-similar distractor led to capture and memory loss for both tasks. Our results lend support to the view that gaze capture leads to activation of irrelevant representations in working memory that compete for selection at recall. Activation of irrelevant spatial representations distracts spatial recall, whereas activation of irrelevant verbal features impairs verbal memory performance.

Thinking about giving up clinical practice? A gender-stratified approach to understanding junior doctors' choices.

PURPOSE: This study examined the association between workplace factors and thinking about leaving clinical care by junior doctors (i.e., doctors in postgraduate training) in Germany, and used a gender-stratified approach to address the so-called feminization of medicine. METHOD: In this cross-sectional questionnaire study, 1,494 German junior doctors working in patient care in Bavaria in 2006-2007 were sent a questionnaire about considering leaving patient care (outcome variable). Logistic regression analyses were used to calculate the predictive influence of workplace factors on the outcome variable (adjusted odds ratio [aOR], 95% confidence interval [95% CI]). RESULTS: Of the 637 responding junior doctors, altogether 52% (men 49.3%, women 54.2%) had thought about leaving clinical practice. Support of colleagues and superiors (men aOR 2.70, 95% CI 1.20-6.04; women aOR 3.46, 95% CI 1.67-7.17), performance feedback (women aOR 1.88, 95% CI 1.23-3.14), the availability of postgraduate training possibilities (men aOR 4.74, 95% CI 1.53-14.69), overtime (women aOR 1.82, 95% CI 1.02-3.25), weekend duty (men aOR: 3.58, 95% CI 1.86-6.88), and the medical specialty were found to be significant predictors for thinking about leaving clinical practice for both men and women. CONCLUSIONS: Workplace factors predict the wish for leaving clinical care for junior doctors. Male and female junior doctors seem to have different priorities in the workplace, which should be addressed in order to retain them in patient care.

Well prepared for work? Junior doctors' self-assessment after medical education.

BACKGROUND: Apart from objective exam results, the overall feeling of preparedness is important for a successful transition process from being a student to becoming a qualified doctor. This study examines the association between self-assessed deficits in medical skills and knowledge and the feeling of preparedness of junior doctors in order to determine which aspects of medical education need to be addressed in more detail in order to improve the quality of this transition phase and in order to increase patient safety. METHODS: A cohort of 637 doctors with up to two years of clinical work experience was included in this analysis and was asked about the overall feeling of preparedness and self-assessed deficits with regard to clinical knowledge and skills. Three logistic regression models were used to identify medical skills which predict the feeling of preparedness. RESULTS: All in all, about 60% of the participating doctors felt poorly prepared for post-graduate training. Self-assessed deficits in ECG interpretation (aOR: 4.39; 95% CI: 2.012-9.578), treatment and therapy planning (aOR: 3.42; 95% CI: 1.366-8.555), and intubation (aOR: 2.10; 95% CI: 1.092-4.049) were found to be independently associated with the overall feeling of preparedness in the final regression model. CONCLUSIONS: Many junior doctors in Germany felt inadequately prepared for being a doctor. With regard to the contents of medical curricula, our results show that more emphasis on ECG-interpretation, treatment and therapy planning and intubation is required to improve the feeling of preparedness of medical graduates.

Ease of access to list items in short-term memory depends on the order of the recognition probes.

We report data from 4 experiments using a recognition design with multiple probes to be matched to specific study positions. Items could be accessed rapidly, independent of set size, when the test order matched the study order (forward condition). When the order of testing was random, backward, or in a prelearned irregular sequence (reordered conditions), the classic Sternberg result was obtained: Response times were slow and increased linearly with set size. A number of explanations for forward-condition facilitation were ruled out, such as the predictability of the study order (Experiment 2), the predictability of the probe order (Experiment 1), the covariation of study and test orders (Experiments 1, 2, and 4), processes of probe encoding and perception that did not rely on STM access (Experiments 1, 2, and 4), specific support of the forward condition by articulatory processes (Experiment 3), or condition-dependent strategic differences (Experiment 4). More detailed analyses demonstrated that fast forward responses could not be accounted for by the effects of input position and output position that modulated random responses, effects that did account for the slower responses of the reordered conditions (Experiments 1, 3, and 4). A final analysis of probe-to-probe transitions as a function of encoding distance revealed a sizeable single-step benefit in the random condition. We concluded that STM representations were serial rather than spatial and that forward probes benefited from their serial adjacency.

Prevalence of diagnosis and direct treatment costs of back disorders in 644,773 children and youths in Germany.

BACKGROUND: Many authors have reported about the high prevalence rates of self-reported back pain in children. Nevertheless, little is known about the diagnosis of back disorders--regardless of whether the diagnosis is associated with back pain or not. Therefore, the aim of this study was to analyse the prevalence rates and costs of diagnosis of back disorders in childhood and youth. METHODS: We conducted a secondary data analysis of a large, population based German data set (2,300,980 insurants of statutory health insurance funds) which allowed for identification of prevalence rates of diagnoses of back disorders in children (age group 0-14 years) and youths (age group 15-24 years) using three digit ICD-10 codes for dorsopathies (M40-M54: kyphosis and lordosis; scoliosis; spinal osteochondrosis; other deforming dorsopathies; ankylosing spondylitis; other inflammatory spondylopathies; spondylosis; other spondylopathies; spondylopathies in diseases classified elsewhere; cervical disc disorders; other intervertebral disc disorders; other dorsopathies, not elsewhere classified; dorsalgia). Direct treatment costs were calculated based on the real incurred costs for cases with a singular diagnosis of a back disorder. Wherever possible, the results of the random sample were extrapolated to all insurants of statutory health insurance funds (i. e., about 90% of the German population). RESULTS: We found prevalence rates for the diagnosis of back disorders to range between 0.01 - 12.5%. "Scoliosis" (M41) and "dorsalgia" (M54) were the most frequent diagnoses in both age groups. Based on these results, it was calculated that in 2002 alone, approximately 1.4 million children/youths in Germany were diagnosed with "dorsalgia" (M54), and that the direct costs for back disorders in childhood and youth accounted for at least 100 million Euros. CONCLUSIONS: Instead of focusing on the individual, and self-reported disorder or disability, this analysis allowed for the detailed evaluation of medical experts' opinion on back disorders in childhood and youth and for a more objective or public health oriented insight in the topic of diagnosis of back pain and other back disorders. However, due to the methodological limitations by using ICD-10 coding, standardized random validity checks of population based data sets should be mandatory.