Challenges in Measuring Applied Cognition: Measurement Properties and Equivalence of the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT) Applied Cognition Item Bank.

OBJECTIVE: To present challenges in assessment of applied cognition and the results of differential item functioning (DIF) analyses used to inform the development of a computerized adaptive test (CAT). DESIGN: Measurement evaluation cohort study. DIF analyses of 107 items were conducted across educational, age, and sex groups. DIF hypotheses informed the evaluation of the results. SETTING: Hospital-based rehabilitation from a single hospital system. PARTICIPANTS: A total of 2216 hospitalized patients (N=2216). INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: Applied cognition item pool from multiple sources. RESULTS: Many items were hypothesized to show DIF, particularly for age. Information was moderately high in the lower (cognitive disability) tail of the distribution, but some items were not informative. Reliability estimates were high (>0.89) across all studied groups, regardless of estimation method. There were 35 items with DIF of high magnitude and 19 with accompanying supportive hypotheses. CONCLUSIONS: A key clinical tool in inpatient rehabilitation medicine is assessment of applied functional cognitive ability to inform patient-centered rehabilitation strategies to improve function. This was the first study to evaluate measurement equivalence of the applied cognition item pool across large samples of hospitalized patients. Although about one-third of the item pool evidenced DIF or low discrimination, results supported placement of most items into the bank and its use across groups differing in education, age, and sex. Six items were classified with salient DIF, defined as consistent DIF of high magnitude and or impact, with confirmatory directional DIF hypotheses, generated by content experts. These were recommended for adjustment or removal from the bank; 4 were deleted from the bank and 2 had lowered CAT exposure (administration frequency) rates. Many items hypothesized to show DIF contained content measuring constructs other than applied cognition such as physical frailty, perceptual difficulties, or skills reflective of greater educational attainment. Challenges in measurement of this construct are discussed.

Examination of the Measurement Equivalence of the Functional Assessment in Acute Care MCAT (FAMCAT) Mobility Item Bank Using Differential Item Functioning Analyses.

OBJECTIVE: To assess differential item functioning (DIF) in an item pool measuring the mobility of hospitalized patients across educational, age, and sex groups. DESIGN: Measurement evaluation cohort study. Content experts generated DIF hypotheses to guide the interpretation. The graded response item response theory (IRT) model was used. Primary DIF tests were Wald statistics; sensitivity analyses were conducted using the IRT ordinal logistic regression procedure. Magnitude and impact were evaluated by examining group differences in expected item and scale score functions. SETTING: Hospital-based rehabilitation. PARTICIPANTS: Hospitalized patients (N=2216). INTERVENTIONS: Not applicable. MAIN OUTCOME MEASURES: A total of 111 self-reported mobility items. RESULTS: Two linking items among those used to set the metric across forms evidenced DIF for sex and age: "difficulty climbing stairs step-over-step without a handrail (alternating feet)" and "difficulty climbing 3-5 steps without a handrail." Conditional on the mobility state, the items were more difficult for women and older people (aged ≥65y). An additional 18 items were identified with DIF. Items with both high DIF magnitude and hypotheses related to age were difficulty "crossing road at a 4-lane traffic light with curbs," "jumping/landing on one leg," "strenuous activities," and "descending 3-5 steps with no handrail." Although DIF of higher magnitude was observed for several items, the scale-level effect was relatively small and the exposure rate for the most problematic items was low (0.35, 0.27, and 0.20). CONCLUSIONS: This was the first study to evaluate measurement equivalence of the hospital-based rehabilitation mobility item bank. Although 20 items evidenced high magnitude DIF, 5 of which were related to stairs, the scale-level effect was minimal; however, it is recommended that such items be avoided in the development of short-form measures. No items with salient DIF were removed from calibrations, supporting the use of the item bank across groups differing in education, age, and sex. The bank may thus be useful to assist clinical assessment and decision-making regarding risk for specific mobility restrictions at discharge as well as identifying mobility-related functions targeted for postdischarge interventions. Additionally, with the goal of avoiding long and burdensome assessments for patients and clinical staff, these results could be informative for those using the item bank to construct short forms.

Evaluation of measurement equivalence of the Family Satisfaction with the End-of-Life Care (FAMCARE): Tests of differential item functioning between Hispanic and non-Hispanic White caregivers.

OBJECTIVE: Although the psychometric properties of the Family Satisfaction with End-of-Life Care measure have been examined in diverse settings internationally; little evidence exists regarding measurement equivalence in Hispanic caregivers. The aim was to examine the psychometric properties of a short-form of the FAMCARE in Hispanics using latent variable models and place information on differential item functioning (DIF) in an existing family satisfaction item bank. METHOD: The graded form of the item response theory model was used for the analyses of DIF; sensitivity analyses were performed using a latent variable logistic regression approach. Exploratory and confirmatory factor analyses to examine dimensionality were performed within each subgroup studied. The sample included 1,834 respondents: 317 Hispanic and 1,517 non-Hispanic White caregivers of patients with Alzheimer's disease and cancer, respectively. RESULTS: There was strong support for essential unidimensionality for both Hispanic and non-Hispanic White subgroups. Modest DIF of low magnitude and impact was observed; flagged items related to information sharing. Only 1 item was flagged with significant DIF by both a primary and sensitivity method after correction for multiple comparisons: "The way the family is included in treatment and care decisions." This item was more discriminating for the non-Hispanic, White responders than for the Hispanic subsample, and was also a more severe indicator at some levels of the trait; the Hispanic respondents located at higher satisfaction levels were more likely than White non-Hispanic respondents to report satisfaction. SIGNIFICANCE OF RESULTS: The magnitude of DIF was below the salience threshold for all items. Evidence supported the measurement equivalence and use for cross-cultural comparisons of the short-form FAMCARE among Hispanic caregivers, including those interviewed in Spanish.

Follow-up with Callers to the National Suicide Prevention Lifeline: Evaluation of Callers' Perceptions of Care.

Continuity of care for suicidal individuals engaged with a variety of health and mental health care systems has become a national priority, and crisis hotlines are increasingly playing a part in the risk management and continuum of care for these individuals. The current study evaluated a national initiative to have crisis centers in the National Suicide Prevention Lifeline network provide follow-up care to suicidal callers. Data were obtained from 550 callers followed by 41 crisis counselors from 6 centers. Two main data sources provided the information for the current study: a self-report counselor questionnaire on the follow-up activities completed on each clinical follow-up call and a telephone interview with follow-up clients, providing data on their perceptions of the follow-up intervention's effectiveness. The majority of interviewed follow-up clients reported that the intervention stopped them from killing themselves (79.6%) and kept them safe (90.6%). Counselor activities, such as discussing distractors, social contacts to call for help, and reasons for dying, and individual factors, such as baseline suicide risk, were associated with callers' perceptions of the impact of the intervention on their suicide risk. Our findings provide evidence that follow-up calls to suicidal individuals can reduce the perceived risk of future suicidal behavior.

Methodological issues in measuring subjective well-being and quality-of-life: Applications to assessment of affect in older, chronically and cognitively impaired, ethnically diverse groups using the Feeling Tone Questionnaire.

Quality of life assessment includes measurement of positive affect. Methods artifacts associated with positively and negatively worded items can manifest as negative items loading on a second factor, despite the conceptual view that the items are measuring one underlying latent construct. Negatively worded items may elicit biased responses. Additionally, item-level response bias across ethnically diverse groups may compromise group comparisons. The aim was to illustrate methodological approaches to examining method factors and measurement equivalence in an affect measure with 9 positively and 7 negatively worded items: The Feeling Tone Questionnaire (FTQ). The sample included 4,960 non-Hispanic White, 1,144 non-Hispanic Black, and 517 Hispanic community and institutional residents receiving long-term supportive services. The mean age was 82 (s.d.=11.0); 73% were female. Two thirds were cognitively impaired. Methods effects were assessed using confirmatory factor analyses (CFA), and reliability with McDonald's omega and item response theory (IRT) generated estimates. Measurement equivalence was examined using IRT-based Wald tests. Methods effects associated with negatively worded items were observed; these provided little IRT information, and as a composite evidenced lower reliability. Both 13 and 9 item positive affect scales performed well in terms of model fit, reliability, IRT information, and evidenced little differential item functioning of high magnitude or impact. Both CFA and IRT approaches provided complementary methodological information about scale performance. The 9-item affect scale based on the FTQ can be recommended as a brief quality-of-life measure among frail and cognitively impaired individuals in palliative and long-term care settings.

A comparison of item response theory-based methods for examining differential item functioning in object naming test by language of assessment among older Latinos.

Object naming tests are commonly included in neuropsychological test batteries. Differential item functioning (DIF) in these tests due to cultural and language differences may compromise the validity of cognitive measures in diverse populations. We evaluated 26 object naming items for DIF due to Spanish and English language translations among Latinos (n=1,159), mean age of 70.5 years old (Standard Deviation (SD)±7.2), using the following four item response theory-based approaches: Mplus/Multiple Indicator, Multiple Causes (Mplus/MIMIC; Muthén & Muthén, 1998-2011), Item Response Theory Likelihood Ratio Differential Item Functioning (IRTLRDIF/MULTILOG; Thissen, 1991, 2001), difwithpar/Parscale (Crane, Gibbons, Jolley, & van Belle, 2006; Muraki & Bock, 2003), and Differential Functioning of Items and Tests/MULTILOG (DFIT/MULTILOG; Flowers, Oshima, & Raju, 1999; Thissen, 1991). Overall, there was moderate to near perfect agreement across methods. Fourteen items were found to exhibit DIF and 5 items observed consistently across all methods, which were more likely to be answered correctly by individuals tested in Spanish after controlling for overall ability.

Service use by at-risk youths after school-based suicide screening.

OBJECTIVE: We sought to examine follow-up service use by students identified at risk for suicidal behavior in a school-based screening program and assess barriers to seeking services as perceived by youths and parents. METHOD: We conducted a longitudinal study of 317 at-risk youths identified by a school-based suicide screening in six high schools in New York State. The at-risk teenagers and their parents were interviewed approximately 2 years after the initial screen to assess service use during the intervening period and identify barriers that may have interfered with seeking treatment. RESULTS: At the time of the screening, 72% of the at-risk students were not receiving any type of mental health service. Of these students, 51% were deemed in need of services and subsequently referred by us to a mental health professional. Nearly 70% followed through with the screening's referral recommendations. The youths and their parents reported perceptions about mental health problems, specifically relating to the need for treatment, as the primary reasons for not seeking service. CONCLUSIONS: Screening seems to be effective in enhancing the likelihood that students at risk for suicidal behavior will get into treatment. Well-developed and systematic planning is needed to ensure that screening and referral services are coordinated so as to facilitate access for youths into timely treatment.