Six elements test vs D-KEFS: what does "Ecological Validity" tell us?

OBJECTIVE: Extensive research shows that tests of executive functioning (EF) predict instrumental activities of daily living (IADLs) but are nevertheless often criticized for having poor ecological validity. The Modified Six Elements Test (MSET) is a pencil-and-paper test that was developed to mimic the demands of daily life, with the assumption that this would result in a more ecologically valid test. Although the MSET has been extensively validated in its ability to capture cognitive deficits in various populations, support for its ability to predict functioning in daily life is mixed. This study aimed to examine the MSET's ability to predict IADLs assessed via three different modalities relative to traditional EF measures. METHOD: Participants (93 adults aged 60 - 85) completed the MSET, traditional measures of EF (Delis-Kaplan Executive Function System; D-KEFS), and self-reported and performance-based IADLs in the lab. Participants then completed three weeks of IADL tasks at home, using the Daily Assessment of Independent Living and Executive Skills (DAILIES) protocol. RESULTS: The MSET predicted only IADLs completed at home, while the D-KEFS predicted IADLs across all three modalities. Further, the D-KEFS predicted home-based IADLs beyond the MSET when pitted against each other, whereas the MSET did not contribute beyond the D-KEFS. CONCLUSIONS: Traditional EF tests (D-KEFS) appear to be superior to the MSET in predicting IADLs in community-dwelling older adults. The present results argue against replacing traditional measures with the MSET when addressing functional independence of generally high-functioning and cognitive healthy older adult patients.

Pitfalls in research on ecological validity of novel executive function tests: A systematic review and a call to action.

The term "ecological validity" (EV) has traditionally referred to test scores' ability to predict real-world functioning. However, a test's similarity to real-world tasks is sometimes mistaken for evidence of its ability to predict daily life, sometimes bypassing rigorous validation research. The goal of this systematic review was to examine the type and quality of evidence used to support claims of EV of novel face-valid tests of executive functions (EF). MEDLINE and PsychINFO databases were searched using the following terms: ecologic* AND neuropsychol* AND (executive function* OR executive dysfunction OR executive abilit*). Thirty-two articles that explicitly stated that their results demonstrated EV of a novel face-valid test of EF were identified. Results showed that only 60% of studies based their claims about EV on test scores' ability to predict functional outcomes, with the remaining 40% relying on other evidence (e.g., correlations with other measures, participant feedback, group differences). Among the studies that did base their conclusions on test scores' ability to predict outcomes (n = 19), an overwhelming majority relied on behavioral rating scales, utilized small sample sizes and participant-to-variable ratios, and failed to control for covariates and multiple comparisons. Poor scientific rigor was particularly pronounced in studies of "naturalistic" tests. The present systematic review reveals significant conceptual, methodological, and statistical flaws among an overwhelming majority of studies that claim to have found support for the EV of a novel face-valid test of EF. We call upon authors, reviewers, and editors to safeguard the scientific rigor of research in this area. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Conceptualization of the term "ecological validity" in neuropsychological research on executive function assessment: a systematic review and call to action.

OBJECTIVE: "Ecological validity" (EV) is classically defined as test's ability to predict real-world functioning, either alone or together with test's similarity to real-world tasks. In neuropsychological literature on assessment of executive functions (EF), EV is conceptualized inconsistently, leading to misconceptions about the utility of tests. The goal of this systematic review was to examine how EV is conceptualized in studies of EF tests described as ecologically valid. METHOD: MEDLINE and PsychINFO Databases were searched. PRISMA guidelines were observed. After applying inclusion and exclusion criteria, this search yielded 90 articles. Deductive content analysis was employed to determine how the term EV was used. RESULTS: About 1/3 of the studies conceptualized EV as the test's ability to predict functional outcomes, 1/3 as both the ability to predict functional outcome and similarity to real-world tasks, and 1/3 were either unclear about the meaning of the term or relied on notions unrelated to classical definitions (e.g., similarity to real-world tasks alone, association with other tests, or the ability to discriminate between populations). CONCLUSIONS: Conceptualizations of the term EV in literature on EF assessment vary grossly, subsuming the notions of criterion, construct, and face validity, as well as sensitivity/specificity. Such inconsistency makes it difficult to interpret clinical utility of tests that are described as ecologically valid. We call on the field to require that, at minimum, the term EV be clearly defined in all publications, or replaced with more concrete terminology (e.g., criterion validity).

Test of Practical Judgment (TOP-J): Construct, Criterion, and Incremental Validity in a Community Sample of Older Adults.

OBJECTIVE: The Test of Practical Judgment (TOP-J) is a stand-alone judgment measure that is considered to tap into aspects of executive functioning (EF) and inform clinical predictions of daily functioning in older adults. Past validation research is variable and has some limitations. The present study sought to examine the reliability and construct, criterion, and incremental validities of scores on TOP-J 9-item version (TOP-J/9). METHOD: Participants were 95 community-dwelling older adults aged 60 to 85. Participants completed TOP-J/9, measures of EF and global cognition, and three different modalities of instrumental activities of daily living (IADLs) (self-report, performance-based tasks conducted in the laboratory, and performance-based tasks completed at home over 3 weeks). RESULTS: TOP-J/9 scores showed adequate internal consistency (α = 0.73) after correcting for the low number of items. TOP-J/9 was correlated with global cognition and EF, although EF did not survive correction for lower-order processes. Finally, although TOP-J/9 scores were associated with home-based IADL tasks (but not with self-report and laboratory-based IADLs), providing some evidence of criterion validity, they did not incrementally contribute to home-based IADL performance beyond other cognitive measures. However, when two items pertaining to social/ethical judgment were removed, this modified version of TOP-J did relate to EF beyond lower-order processes and contributed uniquely to prediction of home-based IADLs beyond other measures. CONCLUSION: Results suggest that TOP-J/9 taps into global cognitive status (but not necessarily EF) and predicts "real-world" functioning (but not above and beyond other cognitive measures). TOP-J psychometrics may be improved by removing two social/ethical items.