RESUMO
A visual cortical prosthesis (VCP) has long been proposed as a strategy for restoring useful vision to the blind, under the assumption that visual percepts of small spots of light produced with electrical stimulation of visual cortex (phosphenes) will combine into coherent percepts of visual forms, like pixels on a video screen. We tested an alternative strategy in which shapes were traced on the surface of visual cortex by stimulating electrodes in dynamic sequence. In both sighted and blind participants, dynamic stimulation enabled accurate recognition of letter shapes predicted by the brain's spatial map of the visual world. Forms were presented and recognized rapidly by blind participants, up to 86 forms per minute. These findings demonstrate that a brain prosthetic can produce coherent percepts of visual forms.
Assuntos
Cegueira/fisiopatologia , Visão Ocular/fisiologia , Percepção Visual/fisiologia , Adulto , Estimulação Elétrica/métodos , Eletrodos , Feminino , Humanos , Masculino , Pessoa de Meia-Idade , Fosfenos , Córtex Visual/metabolismo , Córtex Visual/fisiologia , Próteses VisuaisRESUMO
Emotion is represented in limbic and prefrontal brain areas, herein termed the affective salience network (ASN). Within the ASN, there are substantial unknowns about how valence and emotional intensity are processed-specifically, which nodes are associated with affective bias (a phenomenon in which participants interpret emotions in a manner consistent with their own mood). A recently developed feature detection approach ('specparam') was used to select dominant spectral features from human intracranial electrophysiological data, revealing affective specialization within specific nodes of the ASN. Spectral analysis of dominant features at the channel level suggests that dorsal anterior cingulate (dACC), anterior insula and ventral-medial prefrontal cortex (vmPFC) are sensitive to valence and intensity, while the amygdala is primarily sensitive to intensity. Akaike information criterion model comparisons corroborated the spectral analysis findings, suggesting all four nodes are more sensitive to intensity compared to valence. The data also revealed that activity in dACC and vmPFC were predictive of the extent of affective bias in the ratings of facial expressions-a proxy measure of instantaneous mood. To examine causality of the dACC in affective experience, 130 Hz continuous stimulation was applied to dACC while patients viewed and rated emotional faces. Faces were rated significantly happier during stimulation, even after accounting for differences in baseline ratings. Together the data suggest a causal role for dACC during the processing of external affective stimuli.
Assuntos
Mapeamento Encefálico , Encéfalo , Humanos , Encéfalo/fisiologia , Emoções/fisiologia , Afeto , Eletroencefalografia , Imageamento por Ressonância MagnéticaRESUMO
Humans have the unique ability to decode the rapid stream of language elements that constitute speech, even when it is contaminated by noise. Two reliable observations about noisy speech perception are that seeing the face of the talker improves intelligibility and the existence of individual differences in the ability to perceive noisy speech. We introduce a multivariate BOLD fMRI measure that explains both observations. In two independent fMRI studies, clear and noisy speech was presented in visual, auditory and audiovisual formats to thirty-seven participants who rated intelligibility. An event-related design was used to sort noisy speech trials by their intelligibility. Individual-differences multidimensional scaling was applied to fMRI response patterns in superior temporal cortex and the dissimilarity between responses to clear speech and noisy (but intelligible) speech was measured. Neural dissimilarity was less for audiovisual speech than auditory-only speech, corresponding to the greater intelligibility of noisy audiovisual speech. Dissimilarity was less in participants with better noisy speech perception, corresponding to individual differences. These relationships held for both single word and entire sentence stimuli, suggesting that they were driven by intelligibility rather than the specific stimuli tested. A neural measure of perceptual intelligibility may aid in the development of strategies for helping those with impaired speech perception.
Assuntos
Percepção da Fala , Fala , Humanos , Imageamento por Ressonância Magnética , Individualidade , Percepção Visual/fisiologia , Percepção da Fala/fisiologia , Lobo Temporal/diagnóstico por imagem , Lobo Temporal/fisiologia , Inteligibilidade da Fala , Estimulação Acústica/métodosRESUMO
Lesion-behavior mapping (LBM) provides a statistical map of the association between voxel-wise brain damage and individual differences in behavior. To understand whether two behaviors are mediated by damage to distinct regions, researchers often compare LBM weight outputs by either the Overlap method or the Correlation method. However, these methods lack statistical criteria to determine whether two LBM are distinct versus the same and are disconnected from a major goal of LBMs: predicting behavior from brain damage. Without such criteria, researchers may draw conclusions from numeric differences between LBMs that are irrelevant to predicting behavior. We developed and validated a predictive validity comparison method (PVC) that establishes a statistical criterion for comparing two LBMs using predictive accuracy: two LBMs are distinct if and only if they provide unique predictive power for the behaviors being assessed. We applied PVC to two lesion-behavior stroke data sets, demonstrating its utility for determining when behaviors arise from the same versus different lesion patterns. Using region-of-interest-based simulations derived from proportion damage from a large data set (n = 131), PVC accurately detected when behaviors were mediated by different regions (high sensitivity) versus the same region (high specificity). Both the Overlap method and Correlation method performed poorly on the simulated data. By objectively determining whether two behavioral deficits can be explained by single versus distinct patterns of brain damage, PVC provides a critical advance in establishing the brain bases of behavior. We have developed and released a GUI-driven web app to encourage widespread adoption.
Assuntos
Lesões Encefálicas , Acidente Vascular Cerebral , Humanos , Mapeamento Encefálico , Encéfalo/diagnóstico por imagem , Encéfalo/patologia , Acidente Vascular Cerebral/diagnóstico por imagem , Acidente Vascular Cerebral/patologia , Lesões Encefálicas/patologia , Cabeça , Imageamento por Ressonância MagnéticaRESUMO
This paper is motivated by studying differential brain activities to multiple experimental condition presentations in intracranial electroencephalography (iEEG) experiments. Contrasting effects of experimental conditions are often zero in most regions and nonzero in some local regions, yielding locally sparse functions. Such studies are essentially a function-on-scalar regression problem, with interest being focused not only on estimating nonparametric functions but also on recovering the function supports. We propose a weighted group bridge approach for simultaneous function estimation and support recovery in function-on-scalar mixed effect models, while accounting for heterogeneity present in functional data. We use B-splines to transform sparsity of functions to its sparse vector counterpart of increasing dimension, and propose a fast nonconvex optimization algorithm using nested alternative direction method of multipliers (ADMM) for estimation. Large sample properties are established. In particular, we show that the estimated coefficient functions are rate optimal in the minimax sense under the L2 norm and resemble a phase transition phenomenon. For support estimation, we derive a convergence rate under the L ∞ $L_{\infty }$ norm that leads to a selection consistency property under δ-sparsity, and obtain a result under strict sparsity using a simple sufficient regularity condition. An adjusted extended Bayesian information criterion is proposed for parameter tuning. The developed method is illustrated through simulations and an application to a novel iEEG data set to study multisensory integration.
Assuntos
Algoritmos , Encéfalo , Teorema de BayesRESUMO
Tests of visuospatial memory following short (<1 s) and medium (1 to 30 s) delays have revealed characteristically different patterns of behavior in humans. These data have been interpreted as evidence for different memory systems operating during short (iconic memory) and long delays (working memory). Leising et al. (2019, Behavioural Processes, 169, Article 103957 ) found evidence for both systems in pigeons and humans completing a location change-detection task using a visual mask that disrupted accuracy following a short (100 ms), but not a long (1,000 ms) delay. Another common finding is that adding to-be-remembered items should disrupt accuracy after a long, but not short, delay. Experiments 1a and 1b reported this memory system crossover effect in pigeons and people, respectively, tested on location change detection with delays of 0, 100, and 1,000 ms and displays of two to 16 items. Experiments 2a and 2b reported that the color of the items had little (pigeons) or no (humans) effect on change-detection accuracy. Pigeons tested in Experiment 3 with longer delays (2,000, 4,000, and 8,000 ms) and large set sizes demonstrated the crossover effect with most displays but did not demonstrate an abrupt drop in accuracy characteristic of iconic memory. In Experiment 4, accuracy with novel types of change (color, shape, and size) was better after a 0-ms delay and above-chance levels on color and shape trials. These data demonstrate the memory system crossover effect in both humans and pigeons and expand our knowledge of the properties of memory systems across species.
Assuntos
Columbidae , Memória de Curto Prazo , Humanos , Animais , Rememoração Mental , ProbabilidadeRESUMO
Meningiomas account for one-third of all primary brain tumors. Although typically benign, about 20% of meningiomas are aggressive, and despite the rigor of the current histopathological classification system there remains considerable uncertainty in predicting tumor behavior. Here, we analyzed 160 tumors from all 3 World Health Organization (WHO) grades (I through III) using clinical, gene expression, and sequencing data. Unsupervised clustering analysis identified 3 molecular types (A, B, and C) that reliably predicted recurrence. These groups did not directly correlate with the WHO grading system, which classifies more than half of the tumors in the most aggressive molecular type as benign. Transcriptional and biochemical analyses revealed that aggressive meningiomas involve loss of the repressor function of the DREAM complex, which results in cell-cycle activation; only tumors in this category tend to recur after full resection. These findings should improve our ability to predict recurrence and develop targeted treatments for these clinically challenging tumors.
Assuntos
Proteínas Interatuantes com Canais de Kv/genética , Neoplasias Meníngeas/genética , Meningioma/genética , Recidiva Local de Neoplasia/genética , Proteínas Repressoras/genética , Adulto , Idoso , Idoso de 80 Anos ou mais , Ciclo Celular/genética , Ciclo Celular/fisiologia , Linhagem Celular , Variações do Número de Cópias de DNA/genética , Progressão da Doença , Feminino , Perfilação da Expressão Gênica , Humanos , Masculino , Neoplasias Meníngeas/patologia , Meningioma/patologia , Pessoa de Meia-Idade , Prognóstico , Adulto JovemRESUMO
Experimentalists studying multisensory integration compare neural responses to multisensory stimuli with responses to the component modalities presented in isolation. This procedure is problematic for multisensory speech perception since audiovisual speech and auditory-only speech are easily intelligible but visual-only speech is not. To overcome this confound, we developed intracranial encephalography (iEEG) deconvolution. Individual stimuli always contained both auditory and visual speech, but jittering the onset asynchrony between modalities allowed for the time course of the unisensory responses and the interaction between them to be independently estimated. We applied this procedure to electrodes implanted in human epilepsy patients (both male and female) over the posterior superior temporal gyrus (pSTG), a brain area known to be important for speech perception. iEEG deconvolution revealed sustained positive responses to visual-only speech and larger, phasic responses to auditory-only speech. Confirming results from scalp EEG, responses to audiovisual speech were weaker than responses to auditory-only speech, demonstrating a subadditive multisensory neural computation. Leveraging the spatial resolution of iEEG, we extended these results to show that subadditivity is most pronounced in more posterior aspects of the pSTG. Across electrodes, subadditivity correlated with visual responsiveness, supporting a model in which visual speech enhances the efficiency of auditory speech processing in pSTG. The ability to separate neural processes may make iEEG deconvolution useful for studying a variety of complex cognitive and perceptual tasks.SIGNIFICANCE STATEMENT Understanding speech is one of the most important human abilities. Speech perception uses information from both the auditory and visual modalities. It has been difficult to study neural responses to visual speech because visual-only speech is difficult or impossible to comprehend, unlike auditory-only and audiovisual speech. We used intracranial encephalography deconvolution to overcome this obstacle. We found that visual speech evokes a positive response in the human posterior superior temporal gyrus, enhancing the efficiency of auditory speech processing.
Assuntos
Potenciais Evocados , Percepção da Fala , Lobo Temporal/fisiologia , Percepção Visual , Adulto , Eletrodos Implantados , Eletroencefalografia/instrumentação , Eletroencefalografia/métodos , Feminino , Humanos , MasculinoRESUMO
Direct recording of neural activity from the human brain using implanted electrodes (iEEG, intracranial electroencephalography) is a fast-growing technique in human neuroscience. While the ability to record from the human brain with high spatial and temporal resolution has advanced our understanding, it generates staggering amounts of data: a single patient can be implanted with hundreds of electrodes, each sampled thousands of times a second for hours or days. The difficulty of exploring these vast datasets is the rate-limiting step in discovery. To overcome this obstacle, we created RAVE ("R Analysis and Visualization of iEEG"). All components of RAVE, including the underlying "R" language, are free and open source. User interactions occur through a web browser, making it transparent to the user whether the back-end data storage and computation are occurring locally, on a lab server, or in the cloud. Without writing a single line of computer code, users can create custom analyses, apply them to data from hundreds of iEEG electrodes, and instantly visualize the results on cortical surface models. Multiple types of plots are used to display analysis results, each of which can be downloaded as publication-ready graphics with a single click. RAVE consists of nearly 50,000 lines of code designed to prioritize an interactive user experience, reliability and reproducibility.
Assuntos
Encéfalo/fisiologia , Visualização de Dados , Eletroencefalografia , Processamento de Imagem Assistida por Computador/métodos , Eletrodos Implantados , Humanos , Reprodutibilidade dos Testes , SoftwareRESUMO
Although we routinely experience complex tactile patterns over our entire body, how we selectively experience multisite touch over our bodies remains poorly understood. Here, we characterized tactile search behavior over the full body using a tactile analog of the classic visual search task. On each trial, participants judged whether a target stimulus (e.g., 10-Hz vibration) was present or absent anywhere on the body. When present, the target stimulus could occur alone or simultaneously with distractor stimuli (e.g., 30-Hz vibrations) on other body locations. We systematically varied the number and spatial configurations of the distractors as well as the target and distractor frequencies and measured the impact of these factors on tactile search response times. First, we found that response times were faster on target-present trials compared with target-absent trials. Second, response times increased with the number of stimulated sites, suggesting a serial search process. Third, search performance differed depending on stimulus frequencies. This frequency-dependent behavior may be related to perceptual grouping effects based on timing cues. We constructed linear models to explore how the locations of the target and distractor cues influenced tactile search behavior. Our modeling results reveal that, in isolation, cues on the index fingers make relatively greater contributions to search performance compared with stimulation experienced on other body sites. Additionally, costimulation of sites within the same limb or simply on the same body side preferentially influence search behavior. Our collective findings identify some principles of attentional search that are common to vision and touch, but others that highlight key differences that may be unique to body-based spatial perception.NEW & NOTEWORTHY Little is known about how we selectively experience multisite touch patterns over the body. Using a tactile analog of the classic visual target search paradigm, we show that tactile search behavior for flutter cues is generally consistent with a serial search process. Modeling results reveal the preferential contributions of index finger stimulation and two-site stimulus interactions involving ipsilateral patterns and within-limb patterns. Our results offer initial evidence for spatial and temporal principles underlying tactile search behavior over the body.
Assuntos
Atenção/fisiologia , Extremidades/fisiologia , Percepção do Tato/fisiologia , Adulto , Feminino , Dedos/fisiologia , Humanos , Masculino , Tempo de Reação/fisiologia , Adulto JovemRESUMO
Human faces contain dozens of visual features, but viewers preferentially fixate just two of them: the eyes and the mouth. Face-viewing behavior is usually studied by manually drawing regions of interest (ROIs) on the eyes, mouth, and other facial features. ROI analyses are problematic as they require arbitrary experimenter decisions about the location and number of ROIs, and they discard data because all fixations within each ROI are treated identically and fixations outside of any ROI are ignored. We introduce a data-driven method that uses principal component analysis (PCA) to characterize human face-viewing behavior. All fixations are entered into a PCA, and the resulting eigenimages provide a quantitative measure of variability in face-viewing behavior. In fixation data from 41 participants viewing four face exemplars under three stimulus and task conditions, the first principal component (PC1) separated the eye and mouth regions of the face. PC1 scores varied widely across participants, revealing large individual differences in preference for eye or mouth fixation, and PC1 scores varied by condition, revealing the importance of behavioral task in determining fixation location. Linear mixed effects modeling of the PC1 scores demonstrated that task condition accounted for 41% of the variance, individual differences accounted for 28% of the variance, and stimulus exemplar for less than 1% of the variance. Fixation eigenimages provide a useful tool for investigating the relative importance of the different factors that drive human face-viewing behavior.
Assuntos
Movimentos Oculares/fisiologia , Reconhecimento Facial/fisiologia , Fixação Ocular/fisiologia , Análise de Componente Principal , Adolescente , Adulto , Feminino , Humanos , Masculino , Adulto JovemRESUMO
Audiovisual speech integration combines information from auditory speech (talker's voice) and visual speech (talker's mouth movements) to improve perceptual accuracy. However, if the auditory and visual speech emanate from different talkers, integration decreases accuracy. Therefore, a key step in audiovisual speech perception is deciding whether auditory and visual speech have the same source, a process known as causal inference. A well-known illusion, the McGurk Effect, consists of incongruent audiovisual syllables, such as auditory "ba" + visual "ga" (AbaVga), that are integrated to produce a fused percept ("da"). This illusion raises two fundamental questions: first, given the incongruence between the auditory and visual syllables in the McGurk stimulus, why are they integrated; and second, why does the McGurk effect not occur for other, very similar syllables (e.g., AgaVba). We describe a simplified model of causal inference in multisensory speech perception (CIMS) that predicts the perception of arbitrary combinations of auditory and visual speech. We applied this model to behavioral data collected from 60 subjects perceiving both McGurk and non-McGurk incongruent speech stimuli. The CIMS model successfully predicted both the audiovisual integration observed for McGurk stimuli and the lack of integration observed for non-McGurk stimuli. An identical model without causal inference failed to accurately predict perception for either form of incongruent speech. The CIMS model uses causal inference to provide a computational framework for studying how the brain performs one of its most important tasks, integrating auditory and visual speech cues to allow us to communicate with others.
Assuntos
Percepção Auditiva/fisiologia , Córtex Cerebral/fisiologia , Ilusões/fisiologia , Modelos Neurológicos , Percepção da Fala/fisiologia , Percepção Visual/fisiologia , Adulto , Simulação por Computador , Sinais (Psicologia) , Feminino , Humanos , Masculino , Mascaramento Perceptivo/fisiologia , Semântica , Enquadramento PsicológicoRESUMO
Human speech can be comprehended using only auditory information from the talker's voice. However, comprehension is improved if the talker's face is visible, especially if the auditory information is degraded as occurs in noisy environments or with hearing loss. We explored the neural substrates of audiovisual speech perception using electrocorticography, direct recording of neural activity using electrodes implanted on the cortical surface. We observed a double dissociation in the responses to audiovisual speech with clear and noisy auditory component within the superior temporal gyrus (STG), a region long known to be important for speech perception. Anterior STG showed greater neural activity to audiovisual speech with clear auditory component, whereas posterior STG showed similar or greater neural activity to audiovisual speech in which the speech was replaced with speech-like noise. A distinct border between the two response patterns was observed, demarcated by a landmark corresponding to the posterior margin of Heschl's gyrus. To further investigate the computational roles of both regions, we considered Bayesian models of multisensory integration, which predict that combining the independent sources of information available from different modalities should reduce variability in the neural responses. We tested this prediction by measuring the variability of the neural responses to single audiovisual words. Posterior STG showed smaller variability than anterior STG during presentation of audiovisual speech with noisy auditory component. Taken together, these results suggest that posterior STG but not anterior STG is important for multisensory integration of noisy auditory and visual speech.
Assuntos
Mapeamento Encefálico/métodos , Eletrocorticografia/métodos , Percepção da Fala/fisiologia , Lobo Temporal/fisiologia , Percepção Visual/fisiologia , Adulto , Epilepsia Resistente a Medicamentos/fisiopatologia , Feminino , Humanos , MasculinoRESUMO
Corvids (birds of the family Corvidae) display intelligent behavior previously ascribed only to primates, but such feats are not directly comparable across species. To make direct species comparisons, we used a same/different task in the laboratory to assess abstract-concept learning in black-billed magpies ( Pica hudsonia). Concept learning was tested with novel pictures after training. Concept learning improved with training-set size, and test accuracy eventually matched training accuracy-full concept learning-with a 128-picture set; this magpie performance was equivalent to that of Clark's nutcrackers (a species of corvid) and monkeys (rhesus, capuchin) and better than that of pigeons. Even with an initial 8-item picture set, both corvid species showed partial concept learning, outperforming both monkeys and pigeons. Similar corvid performance refutes the hypothesis that nutcrackers' prolific cache-location memory accounts for their superior concept learning, because magpies rely less on caching. That corvids with "primitive" neural architectures evolved to equal primates in full concept learning and even to outperform them on the initial 8-item picture test is a testament to the shared (convergent) survival importance of abstract-concept learning.
Assuntos
Comportamento Animal/fisiologia , Formação de Conceito/fisiologia , Passeriformes/fisiologia , Animais , Feminino , Masculino , Especificidade da EspécieRESUMO
The ability to learn abstract relational concepts is fundamental to higher level cognition. In contrast to item-specific concepts (e.g. pictures containing trees versus pictures containing cars), abstract relational concepts are not bound to particular stimulus features, but instead involve the relationship between stimuli and therefore may be extrapolated to novel stimuli. Previous research investigating the same/different abstract concept has suggested that primates might be specially adapted to extract relations among items and would require fewer exemplars of a rule to learn an abstract concept than non-primate species. We assessed abstract-concept learning in an avian species, Clark's nutcracker (Nucifraga columbiana), using a small number of exemplars (eight pairs of the same rule, and 56 pairs of the different rule) identical to that previously used to compare rhesus monkeys, capuchin monkeys and pigeons. Nutcrackers as a group (N = 9) showed more novel stimulus transfer than any previous species tested with this small number of exemplars. Two nutcrackers showed full concept learning and four more showed transfer considerably above chance performance, indicating partial concept learning. These results show that the Clark's nutcracker, a corvid species well known for its amazing feats of spatial memory, learns the same/different abstract concept better than any non-human species (including non-human primates) yet tested on this same task.
Assuntos
Formação de Conceito , Aprendizagem por Discriminação , Aves Canoras/fisiologia , AnimaisRESUMO
Humans combine visual information from mouth movements with auditory information from the voice to recognize speech. A common method for assessing multisensory speech perception is the McGurk effect: When presented with particular pairings of incongruent auditory and visual speech syllables (e.g., the auditory speech sounds for "ba" dubbed onto the visual mouth movements for "ga"), individuals perceive a third syllable, distinct from the auditory and visual components. Chinese and American cultures differ in the prevalence of direct facial gaze and in the auditory structure of their languages, raising the possibility of cultural- and language-related group differences in the McGurk effect. There is no consensus in the literature about the existence of these group differences, with some studies reporting less McGurk effect in native Mandarin Chinese speakers than in English speakers and others reporting no difference. However, these studies sampled small numbers of participants tested with a small number of stimuli. Therefore, we collected data on the McGurk effect from large samples of Mandarin-speaking individuals from China and English-speaking individuals from the USA (total n = 307) viewing nine different stimuli. Averaged across participants and stimuli, we found similar frequencies of the McGurk effect between Chinese and American participants (48 vs. 44 %). In both groups, we observed a large range of frequencies both across participants (range from 0 to 100 %) and stimuli (15 to 83 %) with the main effect of culture and language accounting for only 0.3 % of the variance in the data. High individual variability in perception of the McGurk effect necessitates the use of large sample sizes to accurately estimate group differences.
Assuntos
Comparação Transcultural , Idioma , Fonética , Percepção da Fala/fisiologia , Percepção Visual/fisiologia , Estimulação Acústica , Adolescente , Adulto , Atenção , Feminino , Humanos , Masculino , Estimulação Luminosa , Adulto JovemRESUMO
In the McGurk effect, visual speech from the face of the talker alters the perception of auditory speech. The diversity of human languages has prompted many intercultural studies of the effect in both Western and non-Western cultures, including native Japanese speakers. Studies of large samples of native English speakers have shown that the McGurk effect is characterized by high variability in the susceptibility of different individuals to the illusion and in the strength of different experimental stimuli to induce the illusion. The noisy encoding of disparity (NED) model of the McGurk effect uses principles from Bayesian causal inference to account for this variability, separately estimating the susceptibility and sensory noise for each individual and the strength of each stimulus. To determine whether variation in McGurk perception is similar between Western and non-Western cultures, we applied the NED model to data collected from 80 native Japanese-speaking participants. Fifteen different McGurk stimuli that varied in syllable content (unvoiced auditory "pa" + visual "ka" or voiced auditory "ba" + visual "ga") were presented interleaved with audiovisual congruent stimuli. The McGurk effect was highly variable across stimuli and participants, with the percentage of illusory fusion responses ranging from 3 to 78% across stimuli and from 0 to 91% across participants. Despite this variability, the NED model accurately predicted perception, predicting fusion rates for individual stimuli with 2.1% error and for individual participants with 2.4% error. Stimuli containing the unvoiced pa/ka pairing evoked more fusion responses than the voiced ba/ga pairing. Model estimates of sensory noise were correlated with participant age, with greater sensory noise in older participants. The NED model of the McGurk effect offers a principled way to account for individual and stimulus differences when examining the McGurk effect in different cultures.
RESUMO
The prevalence of synthetic talking faces in both commercial and academic environments is increasing as the technology to generate them grows more powerful and available. While it has long been known that seeing the face of the talker improves human perception of speech-in-noise, recent studies have shown that synthetic talking faces generated by deep neural networks (DNNs) are also able to improve human perception of speech-in-noise. However, in previous studies the benefit provided by DNN synthetic faces was only about half that of real human talkers. We sought to determine whether synthetic talking faces generated by an alternative method would provide a greater perceptual benefit. The facial action coding system (FACS) is a comprehensive system for measuring visually discernible facial movements. Because the action units that comprise FACS are linked to specific muscle groups, synthetic talking faces generated by FACS might have greater verisimilitude than DNN synthetic faces which do not reference an explicit model of the facial musculature. We tested the ability of human observers to identity speech-in-noise accompanied by a blank screen; the real face of the talker; and synthetic talking face generated either by DNN or FACS. We replicated previous findings of a large benefit for seeing the face of a real talker for speech-in-noise perception and a smaller benefit for DNN synthetic faces. FACS faces also improved perception, but only to the same degree as DNN faces. Analysis at the phoneme level showed that the performance of DNN and FACS faces was particularly poor for phonemes that involve interactions between the teeth and lips, such as /f/, /v/, and /th/. Inspection of single video frames revealed that the characteristic visual features for these phonemes were weak or absent in synthetic faces. Modeling the real vs. synthetic difference showed that increasing the realism of a few phonemes could substantially increase the overall perceptual benefit of synthetic faces, providing a roadmap for improving communication in this rapidly developing domain.
RESUMO
The prevalence of synthetic talking faces in both commercial and academic environments is increasing as the technology to generate them grows more powerful and available. While it has long been known that seeing the face of the talker improves human perception of speech-in-noise, recent studies have shown that synthetic talking faces generated by deep neural networks (DNNs) are also able to improve human perception of speech-in-noise. However, in previous studies the benefit provided by DNN synthetic faces was only about half that of real human talkers. We sought to determine whether synthetic talking faces generated by an alternative method would provide a greater perceptual benefit. The facial action coding system (FACS) is a comprehensive system for measuring visually discernible facial movements. Because the action units that comprise FACS are linked to specific muscle groups, synthetic talking faces generated by FACS might have greater verisimilitude than DNN synthetic faces which do not reference an explicit model of the facial musculature. We tested the ability of human observers to identity speech-in-noise accompanied by a blank screen; the real face of the talker; and synthetic talking faces generated either by DNN or FACS. We replicated previous findings of a large benefit for seeing the face of a real talker for speech-in-noise perception and a smaller benefit for DNN synthetic faces. FACS faces also improved perception, but only to the same degree as DNN faces. Analysis at the phoneme level showed that the performance of DNN and FACS faces was particularly poor for phonemes that involve interactions between the teeth and lips, such as /f/, /v/, and /th/. Inspection of single video frames revealed that the characteristic visual features for these phonemes were weak or absent in synthetic faces. Modeling the real vs. synthetic difference showed that increasing the realism of a few phonemes could substantially increase the overall perceptual benefit of synthetic faces.
RESUMO
To mitigate limitations in self-reported mood assessments, we introduce a novel affective bias task (ABT). The task quantifies instantaneous emotional state by leveraging the phenomenon of affective bias, in which people interpret external emotional stimuli in a manner consistent with their current emotional state. This study establishes task stability in measuring and tracking depressive symptoms in clinical and non-clinical populations. Initial assessment in a large non-clinical sample established normative ratings. Depressive symptoms were tracked relative to task performance in a non-clinical sample, as well as in a clinical cohort undergoing surgical evaluation for severe epilepsy. In both cohorts, a stronger negative affective bias was associated with higher Beck Depression Inventory (BDI-II) scores. The ABT exhibits high stability and interrater reliability, as well as construct validity in predicting depression levels in both cohorts, suggesting the task as a reliable proxy for mood and a diagnostic tool for detecting depressive symptoms.