Pitfalls in research on ecological validity of novel executive function tests: A systematic review and a call to action.

The term "ecological validity" (EV) has traditionally referred to test scores' ability to predict real-world functioning. However, a test's similarity to real-world tasks is sometimes mistaken for evidence of its ability to predict daily life, sometimes bypassing rigorous validation research. The goal of this systematic review was to examine the type and quality of evidence used to support claims of EV of novel face-valid tests of executive functions (EF). MEDLINE and PsychINFO databases were searched using the following terms: ecologic* AND neuropsychol* AND (executive function* OR executive dysfunction OR executive abilit*). Thirty-two articles that explicitly stated that their results demonstrated EV of a novel face-valid test of EF were identified. Results showed that only 60% of studies based their claims about EV on test scores' ability to predict functional outcomes, with the remaining 40% relying on other evidence (e.g., correlations with other measures, participant feedback, group differences). Among the studies that did base their conclusions on test scores' ability to predict outcomes (n = 19), an overwhelming majority relied on behavioral rating scales, utilized small sample sizes and participant-to-variable ratios, and failed to control for covariates and multiple comparisons. Poor scientific rigor was particularly pronounced in studies of "naturalistic" tests. The present systematic review reveals significant conceptual, methodological, and statistical flaws among an overwhelming majority of studies that claim to have found support for the EV of a novel face-valid test of EF. We call upon authors, reviewers, and editors to safeguard the scientific rigor of research in this area. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Conceptualization of the term "ecological validity" in neuropsychological research on executive function assessment: a systematic review and call to action.

OBJECTIVE: "Ecological validity" (EV) is classically defined as test's ability to predict real-world functioning, either alone or together with test's similarity to real-world tasks. In neuropsychological literature on assessment of executive functions (EF), EV is conceptualized inconsistently, leading to misconceptions about the utility of tests. The goal of this systematic review was to examine how EV is conceptualized in studies of EF tests described as ecologically valid. METHOD: MEDLINE and PsychINFO Databases were searched. PRISMA guidelines were observed. After applying inclusion and exclusion criteria, this search yielded 90 articles. Deductive content analysis was employed to determine how the term EV was used. RESULTS: About 1/3 of the studies conceptualized EV as the test's ability to predict functional outcomes, 1/3 as both the ability to predict functional outcome and similarity to real-world tasks, and 1/3 were either unclear about the meaning of the term or relied on notions unrelated to classical definitions (e.g., similarity to real-world tasks alone, association with other tests, or the ability to discriminate between populations). CONCLUSIONS: Conceptualizations of the term EV in literature on EF assessment vary grossly, subsuming the notions of criterion, construct, and face validity, as well as sensitivity/specificity. Such inconsistency makes it difficult to interpret clinical utility of tests that are described as ecologically valid. We call on the field to require that, at minimum, the term EV be clearly defined in all publications, or replaced with more concrete terminology (e.g., criterion validity).

Interpreting executive and lower-order error scores on the timed subtests of the Delis-Kaplan Executive Function System (D-KEFS) battery: Error analysis across the adult lifespan.

OBJECTIVE: The Delis-Kaplan Executive Function System (D-KEFS) is a battery of tests designed to measure executive functions (EF). Additionally, the D-KEFS contains lower-order tasks, designed to control for speed of visual scanning, sequencing, and verbal and graphomotor output. The construct and criterion validities of D-KEFS scores that are time-based are well established. However, the constructs measured by the D-KEFS error scores are poorly understood, making clinical interpretations of such scores difficult. This study examined the construct validity of D-KEFS errors committed on EF tasks and tasks designed to measure lower-order processes (i.e., non-EF tasks), across the adult lifespan. METHOD: Participants were 427 adults (18-93 years) who completed the timed subtests of the D-KEFS. Four hundred two participants also completed the Push-Turn-Taptap (PTT; a separate measure of EF) to allow cross-validation. RESULTS: General linear regressions showed that D-KEFS errors committed on the EF tests were associated with EF timed performance (assessed using the D-KEFS time-based scores and the PTT), but only among older adults. Importantly, errors committed on the D-KEFS tasks of lower-order processes were also associated with D-KEFS time-based EF performance, and this relationship held across the adult lifespan. CONCLUSIONS: These findings suggest that among older adults EF errors on the D-KEFS can be interpreted as indices of EF, but such interpretations are not automatically warranted for younger adults. Additionally, errors committed on non-EF tasks contained within the D-KEFS battery can be interpreted as reflecting EF weaknesses across the adult lifespan.

Chronic versus recent expressive suppression burdens are differentially associated with cognitive performance among older adults.

Introduction: Expressive suppression (ES; suppression of affective behavior) has been shown to have a deleterious impact on subsequently administered tests of executive functions (EF), threatening validity, and reliability of EF assessment. Past research has shown that recent ES (i.e., across 24 hours prior to testing) and chronic ES (i.e., across 2 weeks prior to testing) have differential impact on test performance. The present study compared the association of chronic vs. recent ES with speed vs. accuracy of performance on tests of EF and tests of lower-order processes. METHOD: Participants were 255 community-dwelling older adults aged 60 and older. Participants completed timed subtests of the Delis-Kaplan Executive Function System and the Burden of State Emotion Regulation Questionnaire. RESULTS: Hierarchical linear regressions examined the contributions of chronic vs. recent ES to test performance. Recent ES was related to scores of both speed and accuracy on EF tests. The association between recent ES and EF errors held beyond covariates (i.e., chronic ES, demographics, depression, and general cognitive status). In contrast, the association between recent ES and EF speed was fully explained by EF error scores. Chronic ES was associated only with speed of performance and only on lower-order tasks, but this relationship did not survive correction for cognitive, psychiatric, and demographic factors. CONCLUSIONS: Recent ES appears to be a risk factor for EF lapses. Chronic ES, while related to performance speed, seems to also relate to several other cognitive, psychiatric, and demographic factors, which themselves explain slower information processing.

The vulnerability of executive functioning: The additive effects of recent non-restorative sleep, pain interference, and use of expressive suppression on test performance.

Objective: Certain transient contextual factors (i.e. higher expressive suppression, pain, and non-restorative sleep) are all associated with temporary decrements in performance on measures of executive functioning (EF), making clinical interpretation of EF scores challenging. Additionally, pain, non-restorative sleep, and expressive suppression are thought to be mutually interrelated with one another and with depression. This study sought to investigate how these factors impact EF when considered simultaneously.Methods: A total of 95 community-dwelling older adults completed the Push-Turn-Taptap (PTT) task as a measure of EF, and self-report measures of expressive suppression, pain interference, non-restorative sleep, and depressive symptoms.Results: Expressive suppression and pain interference, but not non-restorative sleep, demonstrated unique relationships with EF performance when all factors were considered simultaneously. Expressive suppression fully mediated the relationship between non-restorative sleep and EF performance, but not when controlling for depression. When a single contextual burden composite was generated collapsing across transient contextual factors, a composite that included all three examined factors (i.e. including non-restorative sleep) accounted for more variance in EF scores beyond depression than a composite that only included expressive suppression and pain interference.Conclusion: Expressive suppression, pain interference, and non-restorative sleep share some overlap with one another and with depression, but nonetheless have an additive negative effect on EF performance beyond depression. Quantifying these transient contextual factors may improve the accuracy of EF assessment and, by extension, the utility of EF measures in predicting daily functioning. These transient contextual factors also represent targets that, if better managed, may reduce EF lapses in daily life.

Executive functioning interacts with complexity of daily life in predicting daily medication management among older adults.

Objective: Executive functioning (EF) is critical for performance of instrumental activities of daily living, including medication management. Under complex conditions, daily activities require greater EF resources. This study examined if older adults who exhibit weaknesses in EF are at an increased risk for mismanaging their medications under complex contexts.Method: Participants were 50 non-Hispanic white community-dwelling older adults, ages 60-85. At a baseline visit, EF was assessed using the Push-Turn-Taptap task, and daily life complexity and medication regimen complexity were assessed using a structured interview. Medication management across the subsequent 8 weeks was assessed by weekly pill counts.Results: A general linear regression using at-home medication management as the dependent variable showed that EF and the interaction between EF and life complexity were significant predictors (both p values < .001). Specifically, life complexity had a deleterious impact on medication management, but only for older adults whose EF performance was in the bottom 22% of the sample. There was no association between medication regimen complexity and accuracy of medication management.Conclusions: These findings demonstrate that older adults who have even mild EF weaknesses are at an increased risk for mismanaging their medication when leading complex daily lives. These findings add support for the Contextually Valid Executive Assessment (ConVExA) model, which asserts that EF interacts with contextual factors as a predictor of functional outcomes.

Walking, talking, and suppressing: Executive functioning mediates the relationship between higher expressive suppression and slower dual-task walking among older adults.

Objective: Dual-task walking, which is related to fall risk, has also been shown to relate to executive functioning (EF). EF is known to be vulnerable to the effects of an emotion-regulation strategy known as expressive suppression, such that higher engagement in expressive suppression is related to subsequent decrements in EF. However, it is unknown whether expressive suppression is also associated with slower dual-task walking. In addition, if such an association exists, it is unknown whether EF mediates the relationship between expressive suppression and dual-task gait speed.Methods: Ninety-five community-dwelling older adults completed tasks of EF and lower-order component process using the Delis Kaplan Executive Function System (D-KEFS), as well as self-report measures of expressive suppression use in the 24 hours prior to testing and a measure of depressive symptoms.Results: Higher self-reported expressive suppression not only related to poorer EF, but also to slower dual-task walking beyond age and depressive symptoms; however these results did not hold when individuals with possible undiagnosed MCI were excluded. EF mediated the relationship between expressive suppression and dual-task walking speed.Conclusion: Expressive suppression appears to weaken EF, which in turn impacts executive aspects of motor functioning (such as walking under cognitive load) for cognitively vulnerable individuals. Quantifying and accounting for the taxing effect of effortful emotion regulation may improve the accuracy of EF assessment. Expressive suppression represents a potentially modifiable target to help reduce EF lapses and motor failings among older adults.

Naturally Occurring Expressive Suppression is Associated with Lapses in Instrumental Activities of Daily Living Among Community-Dwelling Older Adults.

OBJECTIVES: Expressive suppression (i.e., effortful regulation of overt affect) has a deleterious impact on executive functioning (EF). This relationship has potential ramifications for daily functioning, especially among older adults, because a close relationship exists between EF and functional independence. However, past research has not directly examined whether expressive suppression impacts instrumental activities of daily living (IADL). The present study examined this association among older adults. METHODS: One hundred ten community-dwelling older adults completed a self-report measure of acute (past 24 hr) and chronic (past 2 weeks) expressive suppression, a timed test of IADL, and the Behavioral Dyscontrol Scale as a measure of EF. RESULTS: High chronic expressive suppression was related to slow IADL performance beyond covariates (age, IQ, depression), but only for individuals with low EF. High acute expressive suppression was associated with lower accuracy on IADL tasks beyond covariates (IQ, depression), but this association was fully explained by EF. CONCLUSIONS: The current results suggest that expressive suppression is associated with less efficient and more error-prone IADL performance. EF fully accounted for the relationship between acute expressive suppression and IADL performance, showing that suppression is a risk factor for both poorer EF performance and functional lapses in daily life. Furthermore, individuals with weaker EF may be particularly vulnerable to the effect of chronic expressive suppression. (JINS, 2019, 25, 718-728).

The Impacts of Sexual Arousal and Its Suppression on Executive Functioning.

Suppression of certain types of reflexive emotional responses is thought to temporarily deplete executive functions (EF), as evidenced by poorer performance on measures of EF, but does not deplete other, lower-order cognitive processes. This study examined whether similar decrements in performance on EF tests would occur following suppression of sexual arousal. A sample of 44 male college students underwent baseline cognitive assessment (EF and lower-order cognitive processes), followed by experimental manipulation consisting of exposure to sexually explicit audiovisual stimuli. Sexual arousal was monitored using penile plethysmography. In this study, 21 participants were assigned to a suppression condition and were instructed to suppress sexual arousal during the video, while 23 were assigned to an arousal condition and were instructed to allow themselves to become aroused. Following experimental manipulation, cognition was reassessed. Unexpectedly, results showed EF decrements in the arousal group but not in the suppression group. As expected, only EF was affected by experimental manipulation, with no group differences in lower-order cognitive processes. Thus, the findings suggest that sexual arousal is associated with temporary decrements in EF performance, at least among young, primarily White, male college students. The results contribute to understanding why sexually charged situations are sometimes associated with poor decisions or unsafe/reckless sexual practices.

Greater naturally occurring expressive suppression is associated with poorer executive functioning and motor-sequence learning among older adults.

Objectives: Unusually high engagement in expressive suppression (i.e., purposeful regulation of overt affect) has been associated with poorer performance on executive functioning (EF) and motor-sequence learning tasks. As such, expressive suppression represents one possible source of fluctuations in executive test performance. However, the relationship between expressive suppression and EF and motor performance has not yet been examined in older adults, who are more prone to EF and motor fluctuations than are younger adults. The purpose of this study was to test whether greater self-reported, naturally occurring expressive suppression is related to poorer EF performance and motor-sequence learning in older adults. Method: One hundred and ten community-dwelling older adults completed a self-report measure of expressive suppression, a battery of EF tests, and a computer-based measure of motor-sequence learning. Results: As expected, higher self-reported burden of expressive suppression in the 24 hours prior to testing was related to poorer performance on EF tests and on multiple aspects of motor-sequence learning (action planning latencies and sequencing errors) even after accounting for age, depressive symptoms, and component processes (e.g., processing speed). Conclusions: The current results suggest that naturally occurring expressive suppression depletes EF, which builds on previous findings from experimental studies that show that expressive suppression leads to reduced EF performance. Furthermore, this effect can be captured using self-report methods. These findings highlight expressive suppression as one source of intraindividual fluctuations in executive and motor functioning, which likely place older adults at risk for both functional and motor lapses (e.g., medication mistakes, falls).

Exaggerated reaction to novelty as a predictor of incipient cognitive decline among community-dwelling older adults.

OBJECTIVE: The ability to detect covert markers of incipient cognitive decline among older adults before cognitive decline becomes overtly evident on traditional cognitive tests represents an important topic of research. Exaggerated reactions to novelty, reflected in novelty-induced increases in action planning latencies ("novelty effect"; NE) and low openness to experience (openness), have been previously associated with incipient cognitive decline among older adults who appeared cognitively normal at baseline. The purposes of the present study were to extend prior research on the utility of these markers by examining whether (a) NE and openness each predict cognitive change uniquely, and (b) whether these indices predict cognitive change above and beyond measures of memory, executive functions, processing speed/efficiency, premorbid IQ, and depressive symptoms. METHOD: Sixty-one cognitively normal community-dwelling older adults were administered a battery of tests assessing the relevant constructs at baseline and one-year follow up. Changes in cognitive status were assessed using the Demetia Rating Scale, 2nd Edition, NE was assessed using the Push-Turn-Taptap task (an electronic motor sequence learning task), and openness using the NEO Personality Inventory-Revised. The Test of Premorbid Functioning, and subtests from Repeatable Battery for the Assessment of Neuropsychological Status, Delis-Kaplan Executive Function System, and Wechsler Adult Intelligence Scale, 4th Edition, were used for additional assessment of baseline cognition. Depressive symptoms were assessed using the Geriatric Depression Scale. RESULTS: Results confirmed our hypothesis that both NE and openness contribute to prediction of cognitive change beyond baseline cognition and depressive symptoms, but none of the covariates (i.e., depression, executive functions, processing efficiency, or memory) themselves contributed to the model. NE and openness each contributed unique variance and were independent of each other. CONCLUSIONS: Openness and NE have the potential to provide evidence-based methods for estimating risk of future cognitive change in persons with currently normal standardized test scores.

Cognitive and gait decrements among non-demented older adults with Type 2 diabetes or hypertension: a systematic review.

OBJECTIVE: Older adults with Type 2 diabetes (DM2) and hypertension (HTN) often experience cognitive weaknesses. Growing evidence suggests that such weaknesses place patients at risk for gait disturbance and falls. The current systematic review evaluated (a) the impact of DM2 and HTN on cognition and gait and (b) the association between cognition and gait in the context of DM2 and HTN. METHODS: PubMed was searched to identify studies published in English before 15 April 2017 that evaluated both cognition and gait among non-demented older adults either as a function of DM2 or HTN status or as a function of continuous variables that indicate disease severity and/or management (e.g. blood pressure, HbA1C). Risk of bias was assessed by examining the method of verifying HTN/DM2 and excluding dementia as well as included covariates. RESULTS: In the majority of studies reviewed (n = 17), both DM2 and HTN status were associated with poorer cognitive and/or gait functioning (8 out of 10 studies). In addition, in 10 out of 11 studies cognition was reliably associated with gait. For continuous variables, higher systolic blood pressure (measured with ambulatory methods) was consistently related to poorer cognition and slower gait, but other continuous variables (e.g. HbA1C) were studied inconsistently with inconclusive results. CONCLUSIONS: Cognitive and gait decrements are not only common in the context of HTN and DM2, but also are related to each other. This suggests that clinical neuropsychologists should address cognitive contributions to gait disturbance and falls in their clinical work and research with these patient populations.

Motor sequencing in older adulthood: relationships with executive functioning and effects of complexity.

OBJECTIVE: Older adults' motor sequencing performance is more reliant on executive functioning (EF) and more susceptible to complexity than that of younger adults. This study examined for which aspects of motor sequencing performance these relationships hold. METHODS: Fifty-seven younger and 90 non-demented, community-dwelling, older adults completed selected subtests from the Delis-Kaplan Executive Function System as indices of EF and component processes (CP; graphomotor speed; visual scanning; etc.), as well as a computerized motor sequencing task (Push Turn Taptap task; PTT). The PTT requires participants to perform motor sequences that become progressively more complex across the task's four blocks, and is designed to assess action planning, action learning, and motor control speed and accuracy. RESULTS: Hierarchical regressions using each discrete aspect of performance as the dependent variable revealed that action planning is the only aspect of motor sequencing that is uniquely related to EF (beyond the CP composite) for both age groups. Action learning and motor control accuracy are uniquely associated with EF for older adults only, and only if the sequences are complex. Component processes do not fully account for the unique relationships between motor sequencing and EF in older adults. CONCLUSIONS: These results clarify prior findings by showing (a) more aspects of motor sequencing relate to EF for older compared to younger adults and (b) for these unique relationships, EF is only related to action during the generation of sequences that are complex. These findings further our understanding of how aging shapes the links between EF and motor actions, and can be used in evidence-based and theoretically driven intervention programs that promote healthy aging.

Reported Expressive Suppression in Daily Life Is Associated with Slower Action Planning.

Objectives Growing evidence demonstrates that (a) executive functioning (EF) becomes deleteriously affected by engagement in the emotion regulation strategy known as expressive suppression and (b) EF shows considerable functional and neuroanatomical overlap with motor output. The current study aimed to bridge these two literatures by examining the relationships between naturally occurring expressive suppression and several different aspects of motor output, including action planning, action learning, and motor-control speed and accuracy. In addition, we investigated whether any identified relationships could be explained by EF. Methods Fifty-one healthy young adults completed selected subtests from the Delis-Kaplan Executive Function System as indices of EF, a self-report measure of expressive suppression, and a computerized motor sequencing task (Push Turn Taptap task; PTT) designed to assess action planning, action learning, and motor control speed and accuracy. Results Hierarchical regressions using each aspect of PTT performance as the dependent variable revealed that higher than usual self-reported expressive suppression on the day of testing (relative to the 2 weeks preceding testing) was associated with longer action-planning latencies. This relationship was fully explained by EF. No other PTT variables related to expressive suppression on the day of testing. Conclusions These results suggest that increased expressive suppression in daily life is associated with slower action planning, an aspect of motor output that is reliant on EF, highlighting the importance of factors that lead to intra-individual fluctuations in EF and motor performance. (JINS, 2016, 22, 671-681).

Working memory performance inversely predicts spontaneous delta and theta-band scaling relations.

Electrophysiological studies have strongly implicated theta-band activity in human working memory processes. Concurrently, work on spontaneous, non-task-related oscillations has revealed the presence of long-range temporal correlations (LRTCs) within sub-bands of the ongoing EEG, and has begun to demonstrate their functional significance. However, few studies have yet assessed the relation of LRTCs (also called scaling relations) to individual differences in cognitive abilities. The present study addressed the intersection of these two literatures by investigating the relation of narrow-band EEG scaling relations to individual differences in working memory ability, with a particular focus on the theta band. Fifty-four healthy adults completed standardized assessments of working memory and separate recordings of their spontaneous, non-task-related EEG. Scaling relations were quantified in each of the five classical EEG frequency bands via the estimation of the Hurst exponent obtained from detrended fluctuation analysis. A multilevel modeling framework was used to characterize the relation of working memory performance to scaling relations as a function of general scalp location in Cartesian space. Overall, results indicated an inverse relationship between both delta and theta scaling relations and working memory ability, which was most prominent at posterior sensors, and was independent of either spatial or individual variability in band-specific power. These findings add to the growing literature demonstrating the relevance of neural LRTCs for understanding brain functioning, and support a construct- and state-dependent view of their functional implications.

Neurocognitive and neurophysiological correlates of motor planning during familiar and novel contexts.

OBJECTIVE: Research suggests that behavioral indices of motor planning (i.e., latencies that precede motor output) (a) relate to processing speed (PS) and executive functioning (EF), but not working memory (WM), and (b) deteriorate in novel contexts. It is not clear whether an electrophysiological index of motor planning (i.e., movement-related cortical potentials; MRCPs) also relates to PS and EF, and whether it deteriorates in novel contexts. This study sought to clarify associations among these variables while manipulating contextual novelty. METHOD: Forty healthy adults completed standardized measures of PS, EF, and WM. Participants performed highly familiar motor sequences in familiar versus novel contexts during EEG recording, while motor planning latencies and peak MRCPs were obtained. Hierarchical regressions assessed the relative contributions of PS, EF, and WM to motor planning latencies and MRCPs. RESULTS: Novel contexts elicited longer planning latencies (gav = 1.96) and reduced MRCPs (gav = .24) compared to familiar contexts. PS predicted planning times in both familiar (R² = .12) and novel contexts (R² = .15), while EF contributed additional variance during novel contexts only (R² Change = .10). EF was the sole predictor of MRCPs in both familiar (R² = .12) and novel contexts (R² = .18). WM did not predict planning latencies or MRCPs. CONCLUSIONS: Contextual novelty alone can decrease performance and neural activation during complex sequencing. The general link between preparatory activation and EF suggests that capacity limitations drive novelty effects, and implies a common substrate underlying motor planning and higher-order behavioral control.