Six elements test vs D-KEFS: what does "Ecological Validity" tell us?

OBJECTIVE: Extensive research shows that tests of executive functioning (EF) predict instrumental activities of daily living (IADLs) but are nevertheless often criticized for having poor ecological validity. The Modified Six Elements Test (MSET) is a pencil-and-paper test that was developed to mimic the demands of daily life, with the assumption that this would result in a more ecologically valid test. Although the MSET has been extensively validated in its ability to capture cognitive deficits in various populations, support for its ability to predict functioning in daily life is mixed. This study aimed to examine the MSET's ability to predict IADLs assessed via three different modalities relative to traditional EF measures. METHOD: Participants (93 adults aged 60 - 85) completed the MSET, traditional measures of EF (Delis-Kaplan Executive Function System; D-KEFS), and self-reported and performance-based IADLs in the lab. Participants then completed three weeks of IADL tasks at home, using the Daily Assessment of Independent Living and Executive Skills (DAILIES) protocol. RESULTS: The MSET predicted only IADLs completed at home, while the D-KEFS predicted IADLs across all three modalities. Further, the D-KEFS predicted home-based IADLs beyond the MSET when pitted against each other, whereas the MSET did not contribute beyond the D-KEFS. CONCLUSIONS: Traditional EF tests (D-KEFS) appear to be superior to the MSET in predicting IADLs in community-dwelling older adults. The present results argue against replacing traditional measures with the MSET when addressing functional independence of generally high-functioning and cognitive healthy older adult patients.

Pitfalls in research on ecological validity of novel executive function tests: A systematic review and a call to action.

The term "ecological validity" (EV) has traditionally referred to test scores' ability to predict real-world functioning. However, a test's similarity to real-world tasks is sometimes mistaken for evidence of its ability to predict daily life, sometimes bypassing rigorous validation research. The goal of this systematic review was to examine the type and quality of evidence used to support claims of EV of novel face-valid tests of executive functions (EF). MEDLINE and PsychINFO databases were searched using the following terms: ecologic* AND neuropsychol* AND (executive function* OR executive dysfunction OR executive abilit*). Thirty-two articles that explicitly stated that their results demonstrated EV of a novel face-valid test of EF were identified. Results showed that only 60% of studies based their claims about EV on test scores' ability to predict functional outcomes, with the remaining 40% relying on other evidence (e.g., correlations with other measures, participant feedback, group differences). Among the studies that did base their conclusions on test scores' ability to predict outcomes (n = 19), an overwhelming majority relied on behavioral rating scales, utilized small sample sizes and participant-to-variable ratios, and failed to control for covariates and multiple comparisons. Poor scientific rigor was particularly pronounced in studies of "naturalistic" tests. The present systematic review reveals significant conceptual, methodological, and statistical flaws among an overwhelming majority of studies that claim to have found support for the EV of a novel face-valid test of EF. We call upon authors, reviewers, and editors to safeguard the scientific rigor of research in this area. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Conceptualization of the term "ecological validity" in neuropsychological research on executive function assessment: a systematic review and call to action.

OBJECTIVE: "Ecological validity" (EV) is classically defined as test's ability to predict real-world functioning, either alone or together with test's similarity to real-world tasks. In neuropsychological literature on assessment of executive functions (EF), EV is conceptualized inconsistently, leading to misconceptions about the utility of tests. The goal of this systematic review was to examine how EV is conceptualized in studies of EF tests described as ecologically valid. METHOD: MEDLINE and PsychINFO Databases were searched. PRISMA guidelines were observed. After applying inclusion and exclusion criteria, this search yielded 90 articles. Deductive content analysis was employed to determine how the term EV was used. RESULTS: About 1/3 of the studies conceptualized EV as the test's ability to predict functional outcomes, 1/3 as both the ability to predict functional outcome and similarity to real-world tasks, and 1/3 were either unclear about the meaning of the term or relied on notions unrelated to classical definitions (e.g., similarity to real-world tasks alone, association with other tests, or the ability to discriminate between populations). CONCLUSIONS: Conceptualizations of the term EV in literature on EF assessment vary grossly, subsuming the notions of criterion, construct, and face validity, as well as sensitivity/specificity. Such inconsistency makes it difficult to interpret clinical utility of tests that are described as ecologically valid. We call on the field to require that, at minimum, the term EV be clearly defined in all publications, or replaced with more concrete terminology (e.g., criterion validity).

Test of Practical Judgment (TOP-J): Construct, Criterion, and Incremental Validity in a Community Sample of Older Adults.

OBJECTIVE: The Test of Practical Judgment (TOP-J) is a stand-alone judgment measure that is considered to tap into aspects of executive functioning (EF) and inform clinical predictions of daily functioning in older adults. Past validation research is variable and has some limitations. The present study sought to examine the reliability and construct, criterion, and incremental validities of scores on TOP-J 9-item version (TOP-J/9). METHOD: Participants were 95 community-dwelling older adults aged 60 to 85. Participants completed TOP-J/9, measures of EF and global cognition, and three different modalities of instrumental activities of daily living (IADLs) (self-report, performance-based tasks conducted in the laboratory, and performance-based tasks completed at home over 3 weeks). RESULTS: TOP-J/9 scores showed adequate internal consistency (α = 0.73) after correcting for the low number of items. TOP-J/9 was correlated with global cognition and EF, although EF did not survive correction for lower-order processes. Finally, although TOP-J/9 scores were associated with home-based IADL tasks (but not with self-report and laboratory-based IADLs), providing some evidence of criterion validity, they did not incrementally contribute to home-based IADL performance beyond other cognitive measures. However, when two items pertaining to social/ethical judgment were removed, this modified version of TOP-J did relate to EF beyond lower-order processes and contributed uniquely to prediction of home-based IADLs beyond other measures. CONCLUSION: Results suggest that TOP-J/9 taps into global cognitive status (but not necessarily EF) and predicts "real-world" functioning (but not above and beyond other cognitive measures). TOP-J psychometrics may be improved by removing two social/ethical items.

It's complicated: Executive functioning moderates impacts of daily busyness on everyday functioning in community-dwelling older adults.

OBJECTIVE: Research shows that cognitively healthy older adults with mild executive function (EF) weaknesses are vulnerable to the negative impacts of life complexity (or daily busyness) when performing instrumental activities of daily living (IADLs). However, past research assessed life complexity only at one timepoint, not capturing daily fluctuations. Importantly, fluctuations in busyness can themselves have deleterious impacts on functioning. This study extended past research by examining whether (1) variability in daily busyness would be more detrimental than level of busyness to performance of IADLs, and (2) EF assessed at home would moderate deleterious impact of busyness on IADLs. METHOD: Fifty-two community-dwelling older adults aged 60 to 95 completed daily IADL tasks and daily measures of EF and busyness via ecological momentary assessment, independently at home for 18 days. RESULTS: (1) In a subset of participants with mild EF weaknesses, high variability in busyness across days was associated with fewer tasks completed correctly; and (2) across all participants (regardless of EF), high levels of daily busyness were associated with fewer tasks completed on time. CONCLUSIONS: Findings indicate that high variability in daily busyness, potentially reflecting a lack of daily routine, was associated with IADL errors among cognitively healthy older adults with mild EF weaknesses. Additionally, consistently high levels of busyness were associated with failures to complete tasks, or failures to complete them on time, regardless of EF. These results further support the Contextually Valid Executive Assessment (ConVExA) model, which posits that EF and contextual factors interact to predict functional outcomes.

Was the COVID-19 Pandemic Associated with Gender Disparities in Authorship of Manuscripts Submitted to Clinical Neuropsychology Journals?

OBJECTIVE: The COVID-19 pandemic exacerbated gender disparities in some academic disciplines. This study examined the association of the pandemic with gender authorship disparities in clinical neuropsychology (CN) journals. METHOD: Author bylines of 1,018 initial manuscript submissions to four major CN journals from March 15 through September 15 of both 2019 and 2020 were coded for binary gender. Additionally, authorship of 40 articles published on pandemic-related topics (COVID-19, teleneuropsychology) across nine CN journals were coded for binary gender. RESULTS: Initial submissions to these four CN journals increased during the pandemic (+27.2%), with comparable increases in total number of authors coded as either women (+23.0%) or men (+25.4%). Neither the average percentage of women on manuscript bylines nor the proportion of women who were lead and/or corresponding authors differed significantly across time. Moreover, the representation of women as authors of pandemic-related articles did not differ from expected frequencies in the field. CONCLUSIONS: Findings suggest that representation of women as authors of peer-reviewed manuscript submissions to some CN journals did not change during the initial months of the COVID-19 pandemic. Future studies might examine how risk and protective factors may have influenced individual differences in scientific productivity during the pandemic.

Ecological validity reconsidered: the Night Out Task versus the D-KEFS.

INTRODUCTION: Although executive functioning (EF) correlates with execution of instrumental activities of daily living (IADLs), tests of EF have been criticized for having poor ecological validity. Attempts have been made to develop new tests that approximate naturalistic daily tasks. However, the incremental utility of such tests has not been convincingly demonstrated. The Night Out Task (NOT) is a novel measure designed to increase ecological validity. This study examined whether the NOT correlates with traditional lab- and home-based measures of EF and IADLs, and whether it outperforms traditional measures of EF in predicting IADLs. METHOD: Participants (50 adults aged 60 to 95) completed (1) the Delis Kaplan Executive Function System (D-KEFS) and IADLs in the laboratory, and (2) ecological momentary assessment of EF and daily IADL tasks at home across three weeks (using the Daily Assessment of Independent Living and Executive Skills protocol; DAILIES). RESULTS: The NOT correlated with a lab-based measure of EF beyond covariates, and lab-based IADLs beyond covariates and beyond the D-KEFS. However, it was unrelated to at-home variables beyond covariates. In contrast, the D-KEFS was a significant predictor of at-home IADLs, and this association was mediated by at-home EF performance. CONCLUSION: This study provides a preliminary validation of the NOT as a correlate of office-based performances in a primarily college educated white sample. Despite its high face validity, the NOT does not appear to sufficiently tap EF processes needed for home-based IADLs as measured by the DAILIES, although small sample size limits the interpretability of this negative finding.

Reappraising cognitive reappraisal: The taxing impact of emotion regulation on executive functioning in older adults.

OBJECTIVE: Cognitive reappraisal (CR) and expressive suppression (ES) are two common emotion regulation strategies that share similar cognitive and neural underpinnings. Prior research has consistently shown that recent engagement in ES (both self-reported and experimentally manipulated) is associated with subsequent temporary decrements in executive functioning (EF). Thus far, only one study has examined the association between CR and EF, with null results. However, that study was limited by examining only zero-order correlations and by assessing only the speed, not accuracy, of EF performance. The present study examined multivariate relationships among recent CR, recent ES, and EF (both speed and accuracy), as well as the potential impacts of more chronic engagements in, and trait-level preferences between, the two emotion regulation strategies. METHOD: Participants were 201 community-dwelling older adults aged 60 to 93 who had participated in three separate studies examining the relationship between self-reported emotion regulation and EF. RESULTS: Recent CR was associated with EF performance accuracy above and beyond chronic CR. Both recent CR and ES contributed to EF performance accuracy uniquely beyond each other and beyond chronic and preferred emotion regulation. CONCLUSIONS: Both recent ES and CR appear to have a deleterious impact on EF performance accuracy, potentially due to utilization of similar resources; both should be accounted for when assessing emotion regulation and its impacts on EF.

Reliability and validity of composite scores from the timed subtests of the D-KEFS battery.

Many neuropsychological tests of executive functioning (EF) have test-retest reliabilities below what is considered acceptable. Combining several tests into a composite can improve reliability, although due to concerns about the multifaceted nature of the EF construct, this practice is rarely applied. The present study (a) examined internal consistencies and test-retest reliabilities of two composites of four EF tests (EF-Speed from timed scores and EF-Accuracy from error scores), and (b) preliminarily examined the criterion validity of the composites as a first step toward construct validation. Participants were 576 adults aged 18-93. Participants completed timed subtests of the Delis-Kaplan Executive Function System (D-KEFS) at baseline. A subset of participants was retested twice (at 30 min and 1 year), or once (at 6 weeks). An additional subsample completed Timed Instrumental Activities of Daily Living (TIADLs), which was used to preliminarily examine the composites' criterion validity. Internal consistency and test-retest reliabilities ranged from good to excellent for the EF-Speed composite (.809-.901) but were poor to acceptable for the EF-Accuracy composite (.510-.736). Both composites surpassed reliabilities of individual test or condition scores; one third of the examined individual speed-based scores and none of the individual error-based scores met the acceptable reliability threshold (i.e., < .7). The EF-Speed composite accounted for more variance in TIADL speed, and the EF-Accuracy in TIADL accuracy, than did any of the individual test scores, providing preliminary evidence of clinical utility and criterion validity. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Daily Assessment of Executive Functioning and Expressive Suppression Predict Daily Functioning among Community-Dwelling Older Adults.

OBJECTIVE: Executive functioning (EF) is known to be associated with performance of instrumental activities of daily living (IADLs). However, prior research has found that the degree to which EF fluctuates was more predictive of self-reported cognitive and IADL lapses than was average EF performance. One source of such EF fluctuations is engagement in an emotion regulation strategy known as expressive suppression (ES). Importantly, ES has also been shown to relate to IADL performance, presumably due to its impact on EF. However, past research is limited due to assessing IADLs only in the laboratory or via self-report. The present study examined (a) the association of daily EF and ES fluctuations with performance of actual IADL tasks in participants' homes, and (b) whether any significant association between ES fluctuations and daily IADLs would be mediated by daily EF variability. METHOD: Participants were 52 older adults aged 60 to 95. Over the course of 18 days while at home, participants completed daily IADL tasks as well as daily measures of EF and ES via ecological momentary assessment. RESULTS: Contrary to our hypothesis, average EF across days predicted at-home IADLs above and beyond daily EF variability, which itself was also predictive. ES variability also predicted daily IADLs, and this association was fully mediated by average daily EF. CONCLUSIONS: Daily fluctuations in ES appear to have a deleterious impact on performance of IADLs at home, likely due to the impact of such fluctuations on EF, although the average level of EF capacity is also important.

Interpreting executive and lower-order error scores on the timed subtests of the Delis-Kaplan Executive Function System (D-KEFS) battery: Error analysis across the adult lifespan.

OBJECTIVE: The Delis-Kaplan Executive Function System (D-KEFS) is a battery of tests designed to measure executive functions (EF). Additionally, the D-KEFS contains lower-order tasks, designed to control for speed of visual scanning, sequencing, and verbal and graphomotor output. The construct and criterion validities of D-KEFS scores that are time-based are well established. However, the constructs measured by the D-KEFS error scores are poorly understood, making clinical interpretations of such scores difficult. This study examined the construct validity of D-KEFS errors committed on EF tasks and tasks designed to measure lower-order processes (i.e., non-EF tasks), across the adult lifespan. METHOD: Participants were 427 adults (18-93 years) who completed the timed subtests of the D-KEFS. Four hundred two participants also completed the Push-Turn-Taptap (PTT; a separate measure of EF) to allow cross-validation. RESULTS: General linear regressions showed that D-KEFS errors committed on the EF tests were associated with EF timed performance (assessed using the D-KEFS time-based scores and the PTT), but only among older adults. Importantly, errors committed on the D-KEFS tasks of lower-order processes were also associated with D-KEFS time-based EF performance, and this relationship held across the adult lifespan. CONCLUSIONS: These findings suggest that among older adults EF errors on the D-KEFS can be interpreted as indices of EF, but such interpretations are not automatically warranted for younger adults. Additionally, errors committed on non-EF tasks contained within the D-KEFS battery can be interpreted as reflecting EF weaknesses across the adult lifespan.

Chronic versus recent expressive suppression burdens are differentially associated with cognitive performance among older adults.

Introduction: Expressive suppression (ES; suppression of affective behavior) has been shown to have a deleterious impact on subsequently administered tests of executive functions (EF), threatening validity, and reliability of EF assessment. Past research has shown that recent ES (i.e., across 24 hours prior to testing) and chronic ES (i.e., across 2 weeks prior to testing) have differential impact on test performance. The present study compared the association of chronic vs. recent ES with speed vs. accuracy of performance on tests of EF and tests of lower-order processes. METHOD: Participants were 255 community-dwelling older adults aged 60 and older. Participants completed timed subtests of the Delis-Kaplan Executive Function System and the Burden of State Emotion Regulation Questionnaire. RESULTS: Hierarchical linear regressions examined the contributions of chronic vs. recent ES to test performance. Recent ES was related to scores of both speed and accuracy on EF tests. The association between recent ES and EF errors held beyond covariates (i.e., chronic ES, demographics, depression, and general cognitive status). In contrast, the association between recent ES and EF speed was fully explained by EF error scores. Chronic ES was associated only with speed of performance and only on lower-order tasks, but this relationship did not survive correction for cognitive, psychiatric, and demographic factors. CONCLUSIONS: Recent ES appears to be a risk factor for EF lapses. Chronic ES, while related to performance speed, seems to also relate to several other cognitive, psychiatric, and demographic factors, which themselves explain slower information processing.

Executive functioning interacts with complexity of daily life in predicting daily medication management among older adults.

Objective: Executive functioning (EF) is critical for performance of instrumental activities of daily living, including medication management. Under complex conditions, daily activities require greater EF resources. This study examined if older adults who exhibit weaknesses in EF are at an increased risk for mismanaging their medications under complex contexts.Method: Participants were 50 non-Hispanic white community-dwelling older adults, ages 60-85. At a baseline visit, EF was assessed using the Push-Turn-Taptap task, and daily life complexity and medication regimen complexity were assessed using a structured interview. Medication management across the subsequent 8 weeks was assessed by weekly pill counts.Results: A general linear regression using at-home medication management as the dependent variable showed that EF and the interaction between EF and life complexity were significant predictors (both p values < .001). Specifically, life complexity had a deleterious impact on medication management, but only for older adults whose EF performance was in the bottom 22% of the sample. There was no association between medication regimen complexity and accuracy of medication management.Conclusions: These findings demonstrate that older adults who have even mild EF weaknesses are at an increased risk for mismanaging their medication when leading complex daily lives. These findings add support for the Contextually Valid Executive Assessment (ConVExA) model, which asserts that EF interacts with contextual factors as a predictor of functional outcomes.

Convergent and discriminant validity of the Immediate Postconcussion Assessment and Cognitive Testing Battery (ImPACT) in young athletes.

Sports concussions are recognized as significant injuries among young athletes. Research demonstrates that return-to-play prior to becoming asymptomatic has significant repercussions including sustained cognitive deficits. Many programs have begun to use computerized testing rather than traditional neuropsychological tests to (a) determine baseline performance, (b) track symptoms, and (c) measure symptoms following concussion. Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) is one such tool. The current study examined ImPACT's convergent and discriminant validity by comparing scores from sports-related concussion athletes (SRC) to those from nonconcussed controls (CTL). SRC included 29 athletes, ages 12-16, referred for neuropsychological assessment following sports-related concussions. CTL included 25 healthy athletes, ages 12-16, who were concussion-free in the past year. Overall, results showed general support for ImPACT, when used to screen cognition. In fact, all ImPACT domains successfully differentiated between SRC and CTL athletes. Evidence supporting appropriate convergent validity was best for the Visual Memory domain. Further, ImPACT domains demonstrated variable discriminant validity. Overall examination of validity demonstrated that ImPACT has some weaknesses but may have utility in detecting postconcussion cognitive impairment.

Utility of the Conners' Adult ADHD Rating Scale validity scales in identifying simulated attention-deficit hyperactivity disorder and random responding.

Recent concern about malingered self-report of symptoms of attention-deficit hyperactivity disorder (ADHD) in college students has resulted in an urgent need for scales that can detect feigning of this disorder. The present study provided further validation data for a recently developed validity scale for the Conners' Adult ADHD Rating Scale (CAARS), the CAARS Infrequency Index (CII), as well as for the Inconsistency Index (INC). The sample included 139 undergraduate students: 21 individuals with diagnoses of ADHD, 29 individuals responding honestly, 54 individuals responding randomly (full or half), and 35 individuals instructed to feign. Overall, the INC showed moderate sensitivity to random responding (.44-.63) and fairly high specificity to ADHD (.86-.91). The CII demonstrated modest sensitivity to feigning (.31-.46) and excellent specificity to ADHD (.91-.95). Sequential application of validity scales had correct classification rates of honest (93.1%), ADHD (81.0%), feigning (57.1%), half random (42.3%), and full random (92.9%). The present study suggests that the CII is modestly sensitive (true positive rate) to feigned ADHD symptoms, and highly specific (true negative rate) to ADHD. Additionally, this study highlights the utility of applying the CAARS validity scales in a sequential manner for identifying feigning. (PsycINFO Database Record

The effects of symptom information coaching on the feigning of adult ADHD.

College students without ADHD may feign symptoms of ADHD to gain access to stimulant medications and academic accommodations. Unfortunately, research has shown that it can be difficult to discriminate malingered from genuine ADHD symptomatology, especially when evaluations are based only on self-report questionnaires. The present study investigated whether nonclinical college students given no additional information could feign ADHD as successfully as those who were coached on symptoms of the disorder. Similar to Jasinski et al. (2011) and other research on feigned ADHD, a battery of neuropsychological, performance validity, and self-report tests was administered. Undergraduates with no history of ADHD or other psychiatric disorders were randomly assigned to 1 of 2 simulator groups: a coached group that was given information about ADHD symptoms, or a noncoached group that was given no such information. Both simulator groups were asked to feign ADHD. Their performance was compared to a genuine ADHD group and a nonclinical group asked to respond honestly. Self-report, neuropsychological, and performance validity test data are discussed in the context of the effect of coaching and its implications for ADHD evaluations. Symptom coaching did not have a significant effect on feigning success. Performance validity tests were moderately effective at detecting feigned ADHD. (PsycINFO Database Record