A Multimodal Approach to Measuring Listening Effort: A Systematic Review on the Effects of Auditory Task Demand on Physiological Measures and Their Relationship.

OBJECTIVES: Listening effort involves the mental effort required to perceive an auditory stimulus, for example in noisy environments. Prolonged increased listening effort, for example due to impaired hearing ability, may increase risk of health complications. It is therefore important to identify valid and sensitive measures of listening effort. Physiological measures have been shown to be sensitive to auditory task demand manipulations and are considered to reflect changes in listening effort. Such measures include pupil dilation, alpha power, skin conductance level, and heart rate variability. The aim of the current systematic review was to provide an overview of studies to listening effort that used multiple physiological measures. The two main questions were: (1) what is the effect of changes in auditory task demand on simultaneously acquired physiological measures from various modalities? and (2) what is the relationship between the responses in these physiological measures? DESIGN: Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, relevant articles were sought in PubMed, PsycInfo, and Web of Science and by examining the references of included articles. Search iterations with different combinations of psychophysiological measures were performed in conjunction with listening effort-related search terms. Quality was assessed using the Appraisal Tool for Cross-Sectional Studies. RESULTS: A total of 297 articles were identified from three databases, of which 27 were included. One additional article was identified from reference lists. Of the total 28 included articles, 16 included an analysis regarding the relationship between the physiological measures. The overall quality of the included studies was reasonable. CONCLUSIONS: The included studies showed that most of the physiological measures either show no effect to auditory task demand manipulations or a consistent effect in the expected direction. For example, pupil dilation increased, pre-ejection period decreased, and skin conductance level increased with increasing auditory task demand. Most of the relationships between the responses of these physiological measures were nonsignificant or weak. The physiological measures varied in their sensitivity to auditory task demand manipulations. One of the identified knowledge gaps was that the included studies mostly used tasks with high-performance levels, resulting in an underrepresentation of the physiological changes at lower performance levels. This makes it difficult to capture how the physiological responses behave across the full psychometric curve. Our results support the Framework for Understanding Effortful Listening and the need for a multimodal approach to listening effort. We furthermore discuss focus points for future studies.

Learning effects in speech-in-noise tasks: Effect of masker modulation and masking release.

Previous research has shown that learning effects are present for speech intelligibility in temporally modulated (TM) noise, but not in stationary noise. The present study aimed to gain more insight into the factors that might affect the time course (the number of trials required to reach stable performance) and size [the improvement in the speech reception threshold (SRT)] of the learning effect. Two hypotheses were addressed: (1) learning effects are present in both TM and spectrally modulated (SM) noise and (2) the time course and size of the learning effect depend on the amount of masking release caused by either TM or SM noise. Eighteen normal-hearing adults (23-62 years) participated in SRT measurements, in which they listened to sentences in six masker conditions, including stationary, TM, and SM noise conditions. The results showed learning effects in all TM and SM noise conditions, but not for the stationary noise condition. The learning effect was related to the size of masking release: a larger masking release was accompanied by an increased time course of the learning effect and a larger learning effect. The results also indicate that speech is processed differently in SM noise than in TM noise.

Copresence Was Found to Be Related to Some Pupil Measures in Persons With Hearing Loss While They Performed a Speech-in-Noise Task.

OBJECTIVES: To assess if a manipulation of copresence was related to speech-in-noise task performance, arousal, and effort of persons with hearing loss. Task-related arousal and effort were measured by means of pupillometry. DESIGN: Twenty-nine participants (mean age: 64.6 years) with hearing loss (4-frequency pure-tone average [4F-PTA] of 50.2 dB HL [SD = 8.9 dB] in the right ear and 51.3 dB HL [SD = 8.7 dB] in the left ear; averaged across 0.5, 1, 2, and 4 kHz) listened to and repeated spoken Danish sentences that were masked by four streams of continuous speech. Participants were presented with blocks of 20 sentences, during which copresence was manipulated by having participants do the task either alone or accompanied by two observers who were recruited from a similar age group. The task was presented at two difficulty levels, which was accomplished by fixing the signal-to-noise ratio of the speech and masker to match the thresholds at which participants were estimated to correctly repeat 50% (difficult) or 80% (easy) of the sentences in a block. Performance was assessed based on whether or not sentences were repeated correctly. Measures of pupil size (baseline pupil size [BPS], peak pupil dilation [PPD], and mean pupil dilation [MPD]) were used to index arousal and effort. Participants also completed ratings of subjective effort and stress after each block of sentences and a self-efficacy for listening-questionnaire. RESULTS: Task performance was not associated with copresence, but was found to be related to 4F-PTA. An increase in BPS was found for copresence conditions, compared to alone conditions. Furthermore, a post-hoc exploratory analysis revealed that the copresence conditions were associated with a significantly larger pupil size in the second half of the task-evoked pupil response (TEPR). No change in PPD or MPD did was detected between copresence and alone conditions. Self-efficacy, 4F-PTA, and age were not found to be related to the pupil data. Subjective ratings were sensitive to task difficulty but not copresence. CONCLUSION: Copresence was not found to be related to speech-in-noise performance, PPD, or MPD in persons with HL but was associated with an increase in arousal (as indicated by a larger BPS). This could be related to premobilization of effort and/or discomfort in response to the observers' presence. Furthermore, an exploratory analysis of the pupil data showed that copresence was associated with greater pupil dilations in the second half of the TEPR. This may indicate that participants invested more effort during the speech-in-noise task while in the presence of the observers, but that this increase in effort may not necessarily have been related to listening itself. Instead, other speech-in-noise task-related processes, such as preparing to respond, could have been influenced by copresence.

The Presence of Another Individual Influences Listening Effort, But Not Performance.

OBJECTIVES: The aim of this study was to modify a speech perception in noise test to assess whether the presence of another individual (copresence), relative to being alone, affected listening performance and effort expenditure. Furthermore, this study assessed if the effect of the other individual's presence on listening effort was influenced by the difficulty of the task and whether participants had to repeat the sentences they listened to or not. DESIGN: Thirty-four young, normal-hearing participants (mean age: 24.7 years) listened to spoken Dutch sentences that were masked with a stationary noise masker and presented through a loudspeaker. The participants alternated between repeating sentences (active condition) and not repeating sentences (passive condition). They did this either alone or together with another participant in the booth. When together, participants took turns repeating sentences. The speech-in-noise test was performed adaptively at three intelligibility levels (20%, 50%, and 80% sentences correct) in a block-wise fashion. During testing, pupil size was recorded as an objective outcome measure of listening effort. RESULTS: Lower speech intelligibility levels were associated with increased peak pupil dilation (PPDs) and doing the task in the presence of another individual (compared with doing it alone) significantly increased PPD. No interaction effect between intelligibility and copresence on PPD was found. The results suggested that the change of PPD between doing the task alone or together was especially apparent for people who started the experiment in the presence of another individual. Furthermore, PPD was significantly lower during passive listening, compared with active listening. Finally, it seemed that performance was unaffected by copresence. CONCLUSION: The increased PPDs during listening in the presence of another participant suggest that more effort was invested during the task. However, it seems that the additional effort did not result in a change of performance. This study showed that at least one aspect of the social context of a listening situation (in this case copresence) can affect listening effort, indicating that social context might be important to consider in future cognitive hearing research.

Investigating the Influences of Task Demand and Reward on Cardiac Pre-Ejection Period Reactivity During a Speech-in-Noise Task.

OBJECTIVES: Effort investment during listening varies as a function of task demand and motivation. Several studies have manipulated both these factors to elicit and measure changes in effort associated with listening. The cardiac pre-ejection period (PEP) is a relatively novel measure in the field of cognitive hearing science. This measure, which reflects sympathetic nervous system activity on the heart, has previously been implemented during a tone discrimination task but not during a speech-in-noise task. Therefore, the primary goal of this study was to explore the influences of signal to noise ratio (SNR) and monetary reward level on PEP reactivity during a speech-in-noise task. DESIGN: Thirty-two participants with normal hearing (mean age = 22.22 years, SD = 3.03) were recruited at VU University Medical Center. Participants completed a Dutch speech-in-noise test with a single-interfering-talker masking noise. Six fixed SNRs, selected to span the entire psychometric performance curve, were presented in a block-wise fashion. Participants could earn a low (€0.20) or high (€5.00) reward by obtaining a score of ≥70% of words correct in each block. The authors analyzed PEP reactivity: the change in PEP measured during the task, relative to the baseline during rest. Two separate methods of PEP analysis were used, one including data from the whole task block and the other including data obtained during presentation of the target sentences only. After each block, participants rated their effort investment, performance, tendency to give up, and the perceived difficulty of the task. They also completed the need for recovery questionnaire and the reading span test, which are indices of additional factors (fatigue and working memory capacity, respectively) that are known to influence listening effort. RESULTS: Average sentence perception scores ranged from 2.73 to 91.62%, revealing a significant effect of SNR. In addition, an improvement in performance was elicited by the high, compared to the low reward level. A linear relationship between SNR and PEP reactivity was demonstrated: at the lower SNRs PEP reactivity was the most negative, indicating greater effort investment compared to the higher SNRs. The target stimuli method of PEP analysis was more sensitive to this effect than the block-wise method. Contrary to expectations, no significant impact of reward on PEP reactivity was found in the present dataset. Also, there was no physiological evidence that participants were disengaged, even when performance was poor. A significant correlation between need for recovery scores and average PEP reactivity was demonstrated, indicating that a lower need for recovery was associated with less effort investment. CONCLUSIONS: This study successfully implemented the measurement of PEP during a standard speech-in-noise test and included two distinct methods of PEP analysis. The results revealed for the first time that PEP reactivity varies linearly with task demand during a speech-in-noise task, although the effect size was small. No effect of reward on PEP was demonstrated. Finally, participants with a higher need for recovery score invested more effort, as shown by average PEP reactivity, than those with a lower need for recovery score.

Approaches to mathematical modeling of context effects in sentence recognition.

Probabilistic models to quantify context effects in speech recognition have proven their value in audiology. Boothroyd and Nittrouer [J. Acoust. Soc. Am. 84, 101-114 (1988)] introduced a model with the j-factor and k-factor as context parameters. Later, Bronkhorst, Bosman, and Smoorenburg [J. Acoust. Soc. Am. 93, 499-509 (1993)] proposed an elaborated mathematical model to quantify context effects. The present study explores existing models and proposes a new model to quantify the effect of context in sentence recognition. The effect of context is modeled by parameters that represent the change in the probability that a certain number of words in a sentence are correctly recognized. Data from two studies using a Dutch sentence-in-noise test were analyzed. The most accurate fit was obtained when using signal-to-noise ratio-dependent context parameters. Furthermore, reducing the number of context parameters from five to one had only a small effect on the goodness of fit for the present context model. An analysis of the relationships between context parameters from the different models showed that for a change in word recognition probability, the different context parameters can change in opposite directions, suggesting opposite effects of sentence context. This demonstrates the importance of controlling for the recognition probability of words in isolation when comparing the use of sentence context between different groups of listeners.

Informational masking with speech-on-speech intelligibility: Pupil response and time-course of learning.

Previous research has shown a learning effect on speech perception in nonstationary maskers. The present study addressed the time-course of this learning effect and the role of informational masking. To that end, speech reception thresholds (SRTs) were measured for speech in either a stationary noise masker, an interrupted noise masker, or a single-talker masker. The utterance of the single talker was either time-forward (intelligible) or time-reversed (unintelligible), and the sample of the utterance was either frozen (same utterance at each presentation) or random (different utterance at each presentation but from the same speaker). Simultaneously, the pupil dilation response was measured to assess differences in the listening effort between conditions and to track changes in the listening effort over time within each condition. The results showed a learning effect for all conditions but the stationary noise condition-that is, improvement in SRT over time while maintaining equal pupil responses. There were no significant differences in pupil responses between conditions despite large differences in the SRT. Time reversal of the frozen speech affected neither the SRT nor pupil responses.

In a Concurrent Memory and Auditory Perception Task, the Pupil Dilation Response Is More Sensitive to Memory Load Than to Auditory Stimulus Characteristics.

OBJECTIVES: Speech understanding may be cognitively demanding, but it can be enhanced when semantically related text cues precede auditory sentences. The present study aimed to determine whether (a) providing text cues reduces pupil dilation, a measure of cognitive load, during listening to sentences, (b) repeating the sentences aloud affects recall accuracy and pupil dilation during recall of cue words, and (c) semantic relatedness between cues and sentences affects recall accuracy and pupil dilation during recall of cue words. DESIGN: Sentence repetition following text cues and recall of the text cues were tested. Twenty-six participants (mean age, 22 years) with normal hearing listened to masked sentences. On each trial, a set of four-word cues was presented visually as text preceding the auditory presentation of a sentence whose meaning was either related or unrelated to the cues. On each trial, participants first read the cue words, then listened to a sentence. Following this they spoke aloud either the cue words or the sentence, according to instruction, and finally on all trials orally recalled the cues. Peak pupil dilation was measured throughout listening and recall on each trial. Additionally, participants completed a test measuring the ability to perceive degraded verbal text information and three working memory tests (a reading span test, a size-comparison span test, and a test of memory updating). RESULTS: Cue words that were semantically related to the sentence facilitated sentence repetition but did not reduce pupil dilation. Recall was poorer and there were more intrusion errors when the cue words were related to the sentences. Recall was also poorer when sentences were repeated aloud. Both behavioral effects were associated with greater pupil dilation. Larger reading span capacity and smaller size-comparison span were associated with larger peak pupil dilation during listening. Furthermore, larger reading span and greater memory updating ability were both associated with better cue recall overall. CONCLUSIONS: Although sentence-related word cues facilitate sentence repetition, our results indicate that they do not reduce cognitive load during listening in noise with a concurrent memory load. As expected, higher working memory capacity was associated with better recall of the cues. Unexpectedly, however, semantic relatedness with the sentence reduced word cue recall accuracy and increased intrusion errors, suggesting an effect of semantic confusion. Further, speaking the sentence aloud also reduced word cue recall accuracy, probably due to articulatory suppression. Importantly, imposing a memory load during listening to sentences resulted in the absence of formerly established strong effects of speech intelligibility on the pupil dilation response. This nullified intelligibility effect demonstrates that the pupil dilation response to a cognitive (memory) task can completely overshadow the effect of perceptual factors on the pupil dilation response. This highlights the importance of taking cognitive task load into account during auditory testing.

Assessment of the psychometric properties of the AQoL-4D questionnaire in Kannada language for use with adults with hearing loss.

OBJECTIVE: This study aimed to validate the translated (Kannada language) version of the Assessment of Quality of Life-4 Dimensions (AQoL-4D) questionnaire for use in Kannada speaking adults with hearing loss. DESIGN: The study involved a cross-sectional survey design. The original (English) and the translated versions of the AQoL-4D questionnaire along with two other questionnaires, that is, the Hearing Handicap Questionnaire and the Participation Scale were self-administered by the study participants. The Kannada AQoL-4D was filled in twice by roughly 50% of the study sample in two sessions that were 15 d apart. STUDY SAMPLE: In total, 103 Kannada-English biliterate adults with hearing loss participated in the study. RESULTS: Overall analysis involved testing the factor structure and various psychometric properties including the internal consistency, discriminant validity, convergent validity and possible floor/ceiling effects. The factor analysis indicated a four-factor structure, and the overall results showed acceptable psychometric properties of the scores of the full scale. However, poor internal consistency was obtained for three out of the four subscales extracted from the questionnaire. CONCLUSION: Based on the internal consistency of the subscales, we recommend using either the full scale or only the first subscale in Kannada speaking adults with hearing loss.

Relations Between Self-Reported Daily-Life Fatigue, Hearing Status, and Pupil Dilation During a Speech Perception in Noise Task.

OBJECTIVE: People with hearing impairment are likely to experience higher levels of fatigue because of effortful listening in daily communication. This hearing-related fatigue might not only constrain their work performance but also result in withdrawal from major social roles. Therefore, it is important to understand the relationships between fatigue, listening effort, and hearing impairment by examining the evidence from both subjective and objective measurements. The aim of the present study was to investigate these relationships by assessing subjectively measured daily-life fatigue (self-report questionnaires) and objectively measured listening effort (pupillometry) in both normally hearing and hearing-impaired participants. DESIGN: Twenty-seven normally hearing and 19 age-matched participants with hearing impairment were included in this study. Two self-report fatigue questionnaires Need For Recovery and Checklist Individual Strength were given to the participants before the test session to evaluate the subjectively measured daily fatigue. Participants were asked to perform a speech reception threshold test with single-talker masker targeting a 50% correct response criterion. The pupil diameter was recorded during the speech processing, and we used peak pupil dilation (PPD) as the main outcome measure of the pupillometry. RESULTS: No correlation was found between subjectively measured fatigue and hearing acuity, nor was a group difference found between the normally hearing and the hearing-impaired participants on the fatigue scores. A significant negative correlation was found between self-reported fatigue and PPD. A similar correlation was also found between Speech Intelligibility Index required for 50% correct and PPD. Multiple regression analysis showed that factors representing "hearing acuity" and "self-reported fatigue" had equal and independent associations with the PPD during the speech in noise test. Less fatigue and better hearing acuity were associated with a larger pupil dilation. CONCLUSIONS: To the best of our knowledge, this is the first study to investigate the relationship between a subjective measure of daily-life fatigue and an objective measure of pupil dilation, as an indicator of listening effort. These findings help to provide an empirical link between pupil responses, as observed in the laboratory, and daily-life fatigue.

Effects of Hearing Impairment and Hearing Aid Amplification on Listening Effort: A Systematic Review.

OBJECTIVES: To undertake a systematic review of available evidence on the effect of hearing impairment and hearing aid amplification on listening effort. Two research questions were addressed: Q1) does hearing impairment affect listening effort? and Q2) can hearing aid amplification affect listening effort during speech comprehension? DESIGN: English language articles were identified through systematic searches in PubMed, EMBASE, Cinahl, the Cochrane Library, and PsycINFO from inception to August 2014. References of eligible studies were checked. The Population, Intervention, Control, Outcomes, and Study design strategy was used to create inclusion criteria for relevance. It was not feasible to apply a meta-analysis of the results from comparable studies. For the articles identified as relevant, a quality rating, based on the 2011 Grading of Recommendations Assessment, Development, and Evaluation Working Group guidelines, was carried out to judge the reliability and confidence of the estimated effects. RESULTS: The primary search produced 7017 unique hits using the keywords: hearing aids OR hearing impairment AND listening effort OR perceptual effort OR ease of listening. Of these, 41 articles fulfilled the Population, Intervention, Control, Outcomes, and Study design selection criteria of: experimental work on hearing impairment OR hearing aid technologies AND listening effort OR fatigue during speech perception. The methods applied in those articles were categorized into subjective, behavioral, and physiological assessment of listening effort. For each study, the statistical analysis addressing research question Q1 and/or Q2 was extracted. In seven articles more than one measure of listening effort was provided. Evidence relating to Q1 was provided by 21 articles that reported 41 relevant findings. Evidence relating to Q2 was provided by 27 articles that reported 56 relevant findings. The quality of evidence on both research questions (Q1 and Q2) was very low, according to the Grading of Recommendations Assessment, Development, and Evaluation Working Group guidelines. We tested the statistical evidence across studies with nonparametric tests. The testing revealed only one consistent effect across studies, namely that listening effort was higher for hearing-impaired listeners compared with normal-hearing listeners (Q1) as measured by electroencephalographic measures. For all other studies, the evidence across studies failed to reveal consistent effects on listening effort. CONCLUSION: In summary, we could only identify scientific evidence from physiological measurement methods, suggesting that hearing impairment increases listening effort during speech perception (Q1). There was no scientific, finding across studies indicating that hearing aid amplification decreases listening effort (Q2). In general, there were large differences in the study population, the control groups and conditions, and the outcome measures applied between the studies included in this review. The results of this review indicate that published listening effort studies lack consistency, lack standardization across studies, and have insufficient statistical power. The findings underline the need for a common conceptual framework for listening effort to address the current shortcomings.

Cortisol, Chromogranin A, and Pupillary Responses Evoked by Speech Recognition Tasks in Normally Hearing and Hard-of-Hearing Listeners: A Pilot Study.

Pupillometry is one method that has been used to measure processing load expended during speech understanding. Notably, speech perception (in noise) tasks can evoke a pupil response. It is not known if there is concurrent activation of the sympathetic nervous system as indexed by salivary cortisol and chromogranin A (CgA) and whether such activation differs between normally hearing (NH) and hard-of-hearing (HH) adults. Ten NH and 10 adults with mild-to-moderate hearing loss (mean age 52 years) participated. Two speech perception tests were administered in random order: one in quiet targeting 100% correct performance and one in noise targeting 50% correct performance. Pupil responses and salivary samples for cortisol and CgA analyses were collected four times: before testing, after the two speech perception tests, and at the end of the session. Participants rated their perceived accuracy, effort, and motivation. Effects were examined using repeated-measures analyses of variance. Correlations between outcomes were calculated. HH listeners had smaller peak pupil dilations (PPDs) than NH listeners in the speech in noise condition only. No group or condition effects were observed for the cortisol data, but HH listeners tended to have higher cortisol levels across conditions. CgA levels were larger at the pretesting time than at the three other test times. Hearing impairment did not affect CgA. Self-rated motivation correlated most often with cortisol or PPD values. The three physiological indicators of cognitive load and stress (PPD, cortisol, and CgA) are not equally affected by speech testing or hearing impairment. Each of them seem to capture a different dimension of sympathetic nervous system activity.

The Influence of Linguistic Proficiency on Masked Text Recognition Performance in Adults With and Without Congenital Hearing Impairment.

OBJECTIVE: The authors first examined the influence of moderate to severe congenital hearing impairment (CHI) on the correctness of samples of elicited spoken language. Then, the authors used this measure as an indicator of linguistic proficiency and examined its effect on performance in language reception, independent of bottom-up auditory processing. DESIGN: In groups of adults with normal hearing (NH, n = 22), acquired hearing impairment (AHI, n = 22), and moderate to severe CHI (n = 21), the authors assessed linguistic proficiency by analyzing the morphosyntactic correctness of their spoken language production. Language reception skills were examined with a task for masked sentence recognition in the visual domain (text), at a readability level of 50%, using grammatically correct sentences and sentences with distorted morphosyntactic cues. The actual performance on the tasks was compared between groups. RESULTS: Adults with CHI made more morphosyntactic errors in spoken language production than adults with NH, while no differences were observed between the AHI and NH group. This outcome pattern sustained when comparisons were restricted to subgroups of AHI and CHI adults, matched for current auditory speech reception abilities. The data yielded no differences between groups in performance in masked text recognition of grammatically correct sentences in a test condition in which subjects could fully take advantage of their linguistic knowledge. Also, no difference between groups was found in the sensitivity to morphosyntactic distortions when processing short masked sentences, presented visually. CONCLUSIONS: These data showed that problems with the correct use of specific morphosyntactic knowledge in spoken language production are a long-term effect of moderate to severe CHI, independent of current auditory processing abilities. However, moderate to severe CHI generally does not impede performance in masked language reception in the visual modality, as measured in this study with short, degraded sentences. Aspects of linguistic proficiency that are affected by CHI thus do not seem to play a role in masked sentence recognition in the visual modality.

The eye as a window to the listening brain: neural correlates of pupil size as a measure of cognitive listening load.

An important aspect of hearing is the degree to which listeners have to deploy effort to understand speech. One promising measure of listening effort is task-evoked pupil dilation. Here, we use functional magnetic resonance imaging (fMRI) to identify the neural correlates of pupil dilation during comprehension of degraded spoken sentences in 17 normal-hearing listeners. Subjects listened to sentences degraded in three different ways: the target female speech was masked by fluctuating noise, by speech from a single male speaker, or the target speech was noise-vocoded. The degree of degradation was individually adapted such that 50% or 84% of the sentences were intelligible. Control conditions included clear speech in quiet, and silent trials. The peak pupil dilation was larger for the 50% compared to the 84% intelligibility condition, and largest for speech masked by the single-talker masker, followed by speech masked by fluctuating noise, and smallest for noise-vocoded speech. Activation in the bilateral superior temporal gyrus (STG) showed the same pattern, with most extensive activation for speech masked by the single-talker masker. Larger peak pupil dilation was associated with more activation in the bilateral STG, bilateral ventral and dorsal anterior cingulate cortex and several frontal brain areas. A subset of the temporal region sensitive to pupil dilation was also sensitive to speech intelligibility and degradation type. These results show that pupil dilation during speech perception in challenging conditions reflects both auditory and cognitive processes that are recruited to cope with degraded speech and the need to segregate target speech from interfering sounds.

The influence of informational masking on speech perception and pupil response in adults with hearing impairment.

A recent pupillometry study on adults with normal hearing indicates that the pupil response during speech perception (cognitive processing load) is strongly affected by the type of speech masker. The current study extends these results by recording the pupil response in 32 participants with hearing impairment (mean age 59 yr) while they were listening to sentences masked by fluctuating noise or a single-talker. Efforts were made to improve audibility of all sounds by means of spectral shaping. Additionally, participants performed tests measuring verbal working memory capacity, inhibition of interfering information in working memory, and linguistic closure. The results showed worse speech reception thresholds for speech masked by single-talker speech compared to fluctuating noise. In line with previous results for participants with normal hearing, the pupil response was larger when listening to speech masked by a single-talker compared to fluctuating noise. Regression analysis revealed that larger working memory capacity and better inhibition of interfering information related to better speech reception thresholds, but these variables did not account for inter-individual differences in the pupil response. In conclusion, people with hearing impairment show more cognitive load during speech processing when there is interfering speech compared to fluctuating noise.

Combining Cardiovascular and Pupil Features Using k-Nearest Neighbor Classifiers to Assess Task Demand, Social Context, and Sentence Accuracy During Listening.

In daily life, both acoustic factors and social context can affect listening effort investment. In laboratory settings, information about listening effort has been deduced from pupil and cardiovascular responses independently. The extent to which these measures can jointly predict listening-related factors is unknown. Here we combined pupil and cardiovascular features to predict acoustic and contextual aspects of speech perception. Data were collected from 29 adults (mean  =  64.6 years, SD  =  9.2) with hearing loss. Participants performed a speech perception task at two individualized signal-to-noise ratios (corresponding to 50% and 80% of sentences correct) and in two social contexts (the presence and absence of two observers). Seven features were extracted per trial: baseline pupil size, peak pupil dilation, mean pupil dilation, interbeat interval, blood volume pulse amplitude, pre-ejection period and pulse arrival time. These features were used to train k-nearest neighbor classifiers to predict task demand, social context and sentence accuracy. The k-fold cross validation on the group-level data revealed above-chance classification accuracies: task demand, 64.4%; social context, 78.3%; and sentence accuracy, 55.1%. However, classification accuracies diminished when the classifiers were trained and tested on data from different participants. Individually trained classifiers (one per participant) performed better than group-level classifiers: 71.7% (SD  =  10.2) for task demand, 88.0% (SD  =  7.5) for social context, and 60.0% (SD  =  13.1) for sentence accuracy. We demonstrated that classifiers trained on group-level physiological data to predict aspects of speech perception generalized poorly to novel participants. Individually calibrated classifiers hold more promise for future applications.

Effects of hearing acuity on psychophysiological responses to effortful speech perception.

In recent studies, psychophysiological measures have been used as markers of listening effort, but there is limited research on the effect of hearing loss on such measures. The aim of the current study was to investigate the effect of hearing acuity on physiological responses and subjective measures acquired during different levels of listening demand, and to investigate the relationship between these measures. A total of 125 participants (37 males and 88 females, age range 37-72 years, pure-tone average hearing thresholds at the best ear between -5.0 to 68.8 dB HL and asymmetry between ears between 0.0 and 87.5 dB) completed a listening task. A speech reception threshold (SRT) test was used with target sentences spoken by a female voice masked by male speech. Listening demand was manipulated using three levels of intelligibility: 20 % correct speech recognition, 50 %, and 80 % (IL20 %/IL50 %/IL80 %, respectively). During the task, peak pupil dilation (PPD), heart rate (HR), pre-ejection period (PEP), respiratory sinus arrhythmia (RSA), and skin conductance level (SCL) were measured. For each condition, subjective ratings of effort, performance, difficulty, and tendency to give up were also collected. Linear mixed effects models tested the effect of intelligibility level, hearing acuity, hearing asymmetry, and tinnitus complaints on the physiological reactivity (compared to baseline) and subjective measures. PPD and PEP reactivity showed a non-monotonic relationship with intelligibility level, but no such effects were found for HR, RSA, or SCL reactivity. Participants with worse hearing acuity had lower PPD at all intelligibility levels and showed lower PEP baseline levels. Additionally, PPD and SCL reactivity were lower for participants who reported suffering from tinnitus complaints. For IL80 %, but not IL50 % or IL20 %, participants with worse hearing acuity rated their listening effort to be relatively high compared to participants with better hearing. The reactivity of the different physiological measures were not or only weakly correlated with each other. Together, the results suggest that hearing acuity may be associated with altered sympathetic nervous system (re)activity. Research using psychophysiological measures as markers of listening effort to study the effect of hearing acuity on such measures are best served by the use of the PPD and PEP.

The effects of working memory capacity and semantic cues on the intelligibility of speech in noise.

This study examined how semantically related information facilitates the intelligibility of spoken sentences in the presence of masking sound, and how this facilitation is influenced by masker type and by individual differences in cognitive functioning. Dutch sentences were masked by stationary noise, fluctuating noise, or an interfering talker. Each sentence was preceded by a text cue; cues were either three words that were semantically related to the sentence or three unpronounceable nonwords. Speech reception thresholds were adaptively measured. Additional measures included working memory capacity (reading span and size comparison span), linguistic closure ability (text reception threshold), and delayed sentence recognition. Word cues facilitated speech perception in noise similarly for all masker types. Cue benefit was related to reading span performance when the masker was interfering speech, but not when other maskers were used, and it did not correlate with text reception threshold or size comparison span. Better reading span performance was furthermore associated with enhanced delayed recognition of sentences preceded by word relative to nonword cues, across masker types. The results suggest that working memory capacity is associated with release from informational masking by semantically related information, and additionally with the encoding, storage, or retrieval of speech content in memory.

Time-specific Components of Pupil Responses Reveal Alternations in Effort Allocation Caused by Memory Task Demands During Speech Identification in Noise.

Daily communication may be effortful due to poor acoustic quality. In addition, memory demands can induce effort, especially for long or complex sentences. In the current study, we tested the impact of memory task demands and speech-to-noise ratio on the time-specific components of effort allocation during speech identification in noise. Thirty normally hearing adults (15 females, mean age 42.2 years) participated. In an established auditory memory test, listeners had to listen to a list of seven sentences in noise, and repeat the sentence-final word after presentation, and, if instructed, recall the repeated words. We tested the effects of speech-to-noise ratio (SNR; -4 dB, +1 dB) and recall (Recall; Yes, No), on the time-specific components of pupil responses, trial baseline pupil size, and their dynamics (change) along the list. We found three components in the pupil responses (early, middle, and late). While the additional memory task (recall versus no recall) lowered all components' values, SNR (-4 dB versus +1 dB SNR) increased the middle and late component values. Increasing memory demands (Recall) progressively increased trial baseline and steepened decrease of the late component's values. Trial baseline increased most steeply in the condition of +1 dB SNR with recall. The findings suggest that adding a recall to the auditory task alters effort allocation for listening. Listeners are dynamically re-allocating effort from listening to memorizing under changing memory and acoustic demands. The pupil baseline and the time-specific components of pupil responses provide a comprehensive picture of the interplay of SNR and recall on effort.

The Influence of Hearing Loss on the Pupil Response to Degraded Speech.

PURPOSE: Current evidence regarding the influence of hearing loss on the pupil response elicited by speech perception is inconsistent. This might be partially due to confounding effects of age. This study aimed to compare pupil responses in age-matched groups of normal-hearing (NH) and hard of hearing (HH) listeners during listening to speech. METHOD: We tested the baseline pupil size and mean and peak pupil dilation response of 17 NH participants (Mage = 46 years; age range: 20-62 years) and 17 HH participants (Mage = 45 years; age range: 20-63 years) who were pairwise matched on age and educational level. Participants performed three speech perception tasks at a 50% intelligibility level: noise-vocoded speech and speech masked with either stationary noise or interfering speech. They also listened to speech presented in quiet. RESULTS: Hearing loss was associated with poorer speech perception, except for noise-vocoded speech. In contrast to NH participants, performance of HH participants did not improve across trials for the interfering speech condition, and it decreased for speech in stationary noise. HH participants had a smaller mean pupil dilation in degraded speech conditions compared to NH participants, but not for speech in quiet. They also had a steeper decline in the baseline pupil size across trials. The baseline pupil size was smaller for noise-vocoded speech as compared to the other conditions. The normalized data showed an additional group effect on the baseline pupil response. CONCLUSIONS: Hearing loss is associated with a smaller pupil response and steeper decline in baseline pupil size during the perception of degraded speech. This suggests difficulties of the HH participants to sustain their effort investment and performance across the test session.