Health system implementation of the PROMIS Cognitive Function Screener in the Medicare Annual Wellness Visit: framing as abilities versus concerns.

BACKGROUND: Cognitive assessment is a required component of the Medicare Annual Wellness Visit (AWV). In this prospective study, we evaluated acceptability and usefulness of a patient-reported outcome measure (the PROMIS® Cognitive Function Screener, or PRO-CS) to screen for cognitive impairment during the AWV. We compared two versions of the PRO-CS: Abilities and Concerns. METHODS: We developed PRO-CS Abilities and PRO-CS Concerns using items from the PROMIS Cognitive Function item banks. We partnered with a large health system in Pennsylvania to implement an electronic health record (EHR)-integrated version of the 4-item PRO-CS into their AWV workflow. PRO-CS Abilities was implemented in June 2022 and then replaced with PRO-CS Concerns in October 2022. We used EHR data to evaluate scores on Abilities versus Concerns and their association with patient characteristics. We gathered feedback from providers on experiences with the PRO-CS and conducted cognitive interviews with patients to evaluate their preferences for Abilities versus Concerns. RESULTS: Between June 2022 and January 2023, 3,088 patients completed PRO-CS Abilities and 2,614 patients completed PRO-CS Concerns. Mean T-scores for Abilities (54.8) were slightly higher (indicating better cognition) than for Concerns (52.6). 10% of scores on Abilities and 13% of scores on Concerns indicated concern for cognitive impairment (T-score < 45). Both Abilities and Concerns were associated with clinical characteristics as hypothesized, with lower scores for patients with cognitive impairment diagnoses and those requiring assistance with instrumental activities of daily living. Abilities and Concerns had similar negative correlations with depression (r= -0.31 versus r= -0.33) and anxiety (r= -0.28 for both), while Abilities had a slightly stronger positive correlation with self-rated health (r = 0.34 versus r = 0.28). In interviews, providers commented that the PRO-CS could be useful to facilitate conversations about cognition, though several providers noted potential limitations of patient self-report. Feedback from patients indicated a preference for PRO-CS Concerns. CONCLUSIONS: Our findings suggest potential utility of the PRO-CS for cognitive screening in the Medicare AWV. PRO-CS Abilities and Concerns had similar associations with patient clinical characteristics, but the Concerns version was more acceptable to patients.

Confirmatory Factor Analysis and Measurement Invariance of the Functional Assessment of Cancer Therapy Lung Cancer Utility Index (FACT-LUI).

Background. A portion of the Functional Assessment of Cancer Therapy-Lung (FACT-L) instrument contributed to a previously published utility index, the FACT Lung Utility Index or FACT-LUI. Six FACT items representing lung cancer quality of life covered fatigue, pain, dyspnea, cough, anxiety, and depression. Two FACT items had been previously combined by the index authors into one for nausea and/or appetite loss, resulting in 7 final domains. Methods. The objective was to perform measurement invariance testing within a confirmatory factor analysis (CFA) framework to support the feasibility of using the FACT-LUI for non-preference-based psychometric applications. The original index patients comprised group 1, and similar FACT patient data (n = 249) from another published study comprised group 2. One 2-factor model and two 1-factor CFA models were evaluated to assess measurement invariance across groups, using varying degrees of item parceling and a small number of residual covariances, all justified by the literature. Results. The 1-factor models were most optimal. A 1-factor model with 1 pair of items parceled showed invariance to the partial scalar level using usual fit criteria across groups, requiring 2 unconstrained intercepts. A 1-factor model with 3 pairs of justified parcels showed full configural, metric, and scalar invariance across groups. Conclusions. The FACT-LUI items fit a partially to fully invariant 1-factor model, suggesting feasibility for non-preference-based applications. Implications. Results suggest useful incorporation of the FACT-LUI into clinical trials with no substantial increased respondent burden, allowing preference-based and other psychometric applications from the same index items. Highlights: This work suggests that in addition to being originally designed for use as a utility index, the 7 FACT-LUI items together also fit simple CFA and measurement invariance models. This less expected result indicates that these items as a group are also potentially useful in non-preference-based applications.Clinical trials can make for challenging decisions concerning which patient-reported outcome measures to include without being burdensome. However, the literature suggests a need for improved reporting of quality of life in lung cancer in particular as well as cancer in general. Inclusion of more disease-specific items such as the FACT-LUI may allow for information gathering of both preference-based and non-preference-based data with less demand on patients, similar to what has been done with some generic instruments.

Evaluation of PROMIS Cognitive Function Scores and Correlates in a Clinical Sample of Older Adults.

In this study we assessed the utility of self-reported cognitive function using two PROMIS® Cognitive Function (PROMIS-CF) items in an observational clinical sample of patients aged 65 and older (n = 16,249) at a large health system. We evaluated the association of PROMIS-CF scores with clinical characteristics and Montreal Cognitive Assessment (MoCA) scores, and we used logistic regression to examine predictors of 1-year decline in PROMIS-CF scores among patients with available data. PROMIS-CF scores were associated with clinical characteristics as hypothesized, with lower (more impaired) scores for patients with cognitive impairment (CI) diagnoses, multiple comorbidities, and those taking cognitive enhancing or interfering medications. PROMIS-CF scores were also positively associated with MoCA scores. Predictors of 1-year decline in PROMIS-CF scores included CI diagnoses, use of cognitive enhancing medications, higher depression scores, and lower social role function. Our findings suggest potential utility of PROMIS-CF items in a brief patient-administered screening tool for CI.

Patient-clinician interactions and disparities in breast cancer care: the equality in breast cancer care study.

PURPOSE: To examine whether interpersonal aspects of patient-clinician interactions, such as patient-perceived medical discrimination, clinician mistrust, and treatment decision-making contribute to racial/ethnic/educational disparities in breast cancer care. METHODS: A telephone interview was administered to 542 Asian/Pacific Islander (API), Black, Hispanic, and White women identified through the Greater Bay Area Cancer Registry, ages 20 and older diagnosed with a first primary invasive breast cancer. Adjusted odds ratios (aOR) and 95% confidence intervals (CI) were calculated from logistic regression models that assessed associations between race/ethnicity/education, medical discrimination, clinician mistrust, and treatment decision-making with concordance to breast cancer treatment guidelines (guideline-concordant treatment) and perceived quality of care (pQoC). RESULTS: Approximately three-quarters of women received treatment that was guideline-concordant (76.6%) and reported that their breast cancer care was excellent (72.1%). Non-college-educated Black women had lower odds of guideline-concordant care (aOR (CI) = 0.29 (0.12-0.67)) vs. college-educated White women. Odds of excellent pQoC were lower among the following: college-educated Hispanic women (aOR (CI) = 0.09 (0.02-0.47)) and API women regardless of education (aORs ≤ 0.50) vs. college-educated White women, women reporting low and moderate levels of discrimination (aORs ≤ 0.44) vs. none, and women reporting any clinician mistrust (aOR (CI) = 0.50 (0.29-0.88)) vs. none. Disparities in guideline-concordant care and pQoC persisted after controlling for medical discrimination, clinician mistrust, and decision-making. CONCLUSIONS: Interpersonal aspects of the patient-clinician interaction had an impact on pQoC but not receipt of guideline-concordant treatment and did not explain disparities in either outcome. IMPLICATIONS FOR CANCER SURVIVORS: Although breast cancer survivors' interpersonal interactions with clinicians did not influence receipt of appropriate treatment, intervention strategies to improve patient-clinician relations may help attenuate disparities in survivors' pQoC.

State of the psychometric methods: patient-reported outcome measure development and refinement using item response theory.

BACKGROUND: This paper is part of a series comparing different psychometric approaches to evaluate patient-reported outcome (PRO) measures using the same items and dataset. We provide an overview and example application to demonstrate 1) using item response theory (IRT) to identify poor and well performing items; 2) testing if items perform differently based on demographic characteristics (differential item functioning, DIF); and 3) balancing IRT and content validity considerations to select items for short forms. METHODS: Model fit, local dependence, and DIF were examined for 51 items initially considered for the Patient-Reported Outcomes Measurement Information System® (PROMIS®) Depression item bank. Samejima's graded response model was used to examine how well each item measured severity levels of depression and how well it distinguished between individuals with high and low levels of depression. Two short forms were constructed based on psychometric properties and consensus discussions with instrument developers, including psychometricians and content experts. Calibrations presented here are for didactic purposes and are not intended to replace official PROMIS parameters or to be used for research. RESULTS: Of the 51 depression items, 14 exhibited local dependence, 3 exhibited DIF for gender, and 9 exhibited misfit, and these items were removed from consideration for short forms. Short form 1 prioritized content, and thus items were chosen to meet DSM-V criteria rather than being discarded for lower discrimination parameters. Short form 2 prioritized well performing items, and thus fewer DSM-V criteria were satisfied. Short forms 1-2 performed similarly for model fit statistics, but short form 2 provided greater item precision. CONCLUSIONS: IRT is a family of flexible models providing item- and scale-level information, making it a powerful tool for scale construction and refinement. Strengths of IRT models include placing respondents and items on the same metric, testing DIF across demographic or clinical subgroups, and facilitating creation of targeted short forms. Limitations include large sample sizes to obtain stable item parameters, and necessary familiarity with measurement methods to interpret results. Combining psychometric data with stakeholder input (including people with lived experiences of the health condition and clinicians) is highly recommended for scale development and evaluation.

Mental status evaluation in older adults with cancer: Development of the Mental Health Index-13.

OBJECTIVES: The Mental Health Index (MHI) is widely used as a measure of mental health status, but has not been evaluated in the geriatric oncology population. This study evaluated the MHI-17 in a geriatric oncology population, to establish validity and scoring rules. MATERIALS AND METHODS: The Carolina Senior Registry (NCT01137825) was used to obtain data for 686 patients with cancer 65 and older who completed the MHI-17. The 17-item patient-reported measure produces one total score summing across four domains: anxiety, depression, positive affect, and sense of belonging. Cronbach's alpha (α), confirmatory factor analyses (CFA), item-response theory (IRT) analyses, and differential item functioning (DIF) analyses were used to evaluate internal consistency and validity. RESULTS AND DISCUSSION: The revised MHI retained the 13 best-fitting items from the MHI-17 and resulted in a final model that included two subscales: anxiety (four items, RMSEA 0.11; CFI 0.99; TLI 0.98) and depression (9 items, RMSEA 0.10; CFI 0.96; TL 0.95). IRT analyses of the four anxiety items indicated good fit (RMSEA 0.08) and precise measurement of adults with poor mental health, and the nine depression items also fit well (RMSEA 0.05). No meaningful differences were found by sex, education, or treatment stage. Scores were developed to provide meaningful norms. The new MHI-13 is a shorter, more accurate way to assess mental health in older adults with cancer and most importantly allows clinicians to separately identify anxiety and/or depression - a clinically important distinction as treatment differs among these two types of mental health impairment.

Development and Validation of a Genomic Knowledge Scale to Advance Informed Decision Making Research in Genomic Sequencing.

BACKGROUND: This study evaluated the psychometric properties of a new, comprehensive measure of knowledge about genomic sequencing, the University of North Carolina Genomic Knowledge Scale (UNC-GKS). METHODS: The UNC-GKS assesses knowledge in four domains thought to be critical for informed decision making about genomic sequencing. The scale was validated using classical test theory and item response theory in 286 adult patients and 132 parents of pediatric patients undergoing diagnostic whole exome sequencing (WES) in the NCGENES study. RESULTS: The UNC-GKS assessed a single underlying construct (genomic knowledge) with good internal reliability (Cronbach's alpha = 0.90). Scores were most informative (able to discriminate between individuals with different levels of genomic knowledge) at one standard deviation above the scale mean or lower, a range that included most participants. Convergent validity was supported by associations with health literacy and numeracy (rs=0.41-0.46). The scale functioned well across subgroups differing in sex, race/ethnicity, education, and English proficiency. DISCUSSION: Findings supported the promise of the UNC-GKS as a valid and reliable measure of genomic knowledge among people facing complex decisions about WES and comparable sequencing methods. It is neither disease- nor population-specific, and it functioned well across important subgroups, making it usable in diverse populations.

Reliability and validity of PROMIS measures administered by telephone interview in a longitudinal localized prostate cancer study.

PURPOSE: To evaluate the reliability and validity of six PROMIS measures (anxiety, depression, fatigue, pain interference, physical function, and sleep disturbance) telephone-administered to a diverse, population-based cohort of localized prostate cancer patients. METHODS: Newly diagnosed men were enrolled in the North Carolina Prostate Cancer Comparative Effectiveness and Survivorship Study. PROMIS measures were telephone-administered pre-treatment (baseline), and at 3-months and 12-months post-treatment initiation (N = 778). Reliability was evaluated using Cronbach's alpha. Dimensionality was examined with bifactor models and explained common variance (ECV). Ordinal logistic regression models were used to detect potential differential item functioning (DIF) for key demographic groups. Convergent and discriminant validity were assessed by correlations with the legacy instruments Memorial Anxiety Scale for Prostate Cancer and SF-12v2. Known-groups validity was examined by age, race/ethnicity, comorbidity, and treatment. RESULTS: Each PROMIS measure had high Cronbach's alpha values (0.86-0.96) and was sufficiently unidimensional. Floor effects were observed for anxiety, depression, and pain interference measures; ceiling effects were observed for physical function. No DIF was detected. Convergent validity was established with moderate to strong correlations between PROMIS and legacy measures (0.41-0.77) of similar constructs. Discriminant validity was demonstrated with weak correlations between measures of dissimilar domains (-0.20--0.31). PROMIS measures detected differences across age, race/ethnicity, and comorbidity groups; no differences were found by treatment. CONCLUSIONS: This study provides support for the reliability and construct validity of six PROMIS measures in prostate cancer, as well as the utility of telephone administration for assessing HRQoL in low literacy and hard-to-reach populations.

PROMIS Pediatric Peer Relationships Scale: development of a peer relationships item bank as part of social health measurement.

OBJECTIVE: This study's objective was to develop a measure of social health using item response theory as part of the Patient Reported Outcomes Measurement Information System (PROMIS). METHODS: After candidate items were generated from review of prior literature, focus groups, expert input, and cognitive interviews, items were administered to youth aged 8-17 as part of the PROMIS pediatric large scale testing. Exploratory and confirmatory factor analyses were used to assess dimensionality and to identify instances of local dependence. Items that met the unidimensionality criteria were subsequently calibrated using Samejima's Graded Response Model. Differential item functioning was examined by gender and age. RESULTS: The sample included 3,048 youth who completed the questionnaire (51.8% female, 60% white, and 22.7% with chronic illness). The initial conceptualization of social function and sociability did not yield unidimensional item banks. Rather, factor analysis revealed dimensions contrasting peer relationships and adult relationships. The analysis also identified dimensions formed by responses to positively versus negatively worded items. The resulting 15-item bank measures quality of peer relationships and has strong psychometric characteristics as a full bank or an 8-item short form. CONCLUSIONS: The PROMIS pediatric peer relationships scale demonstrates good psychometric characteristics and addresses an important aspect of child health.

Single and Multiple Ability Estimation in the SEM Framework: A Non-Informative Bayesian Estimation Approach.

Latent variable models with many categorical items and multiple latent constructs result in many dimensions of numerical integration, and the traditional frequentist estimation approach, such as maximum likelihood (ML), tends to fail due to model complexity. In such cases, Bayesian estimation with diffuse priors can be used as a viable alternative to ML estimation. The present study compares the performance of Bayesian estimation to ML estimation in estimating single or multiple ability factors across two types of measurement models in the structural equation modeling framework: a multidimensional item response theory (MIRT) model and a multiple-indicator multiple-cause (MIMIC) model. A Monte Carlo simulation study demonstrates that Bayesian estimation with diffuse priors, under various conditions, produces quite comparable results to ML estimation in the single- and multi-level MIRT and MIMIC models. Additionally, an empirical example utilizing the Multistate Bar Examination is provided to compare the practical utility of the MIRT and MIMIC models. Structural relationships among the ability factors, covariates, and a binary outcome variable are investigated through the single- and multi-level measurement models. The paper concludes with a summary of the relative advantages of Bayesian estimation over ML estimation in MIRT and MIMIC models and suggests strategies for implementing these methods.

PROMIS Pediatric Anger Scale: an item response theory analysis.

PURPOSE: The Patient-Reported Outcomes Measurement Information System (PROMIS) aims to develop patient-reported outcome (PROs) instruments for use in clinical research. The PROMIS pediatrics (ages 8-17) project focuses on the development of PROs across several health domains (physical function, pain, fatigue, emotional distress, social role relationships, and asthma symptoms). The objective of the present study was to report on the psychometric properties of the PROMIS Pediatric Anger Scale. METHODS: Participants (n = 759) were recruited from public school settings, hospital-based outpatient, and subspecialty pediatrics clinics. The anger items (k = 10) were administered on one test form. A hierarchical confirmatory factor analytic model (CFA) was conducted to evaluate scale dimensionality and local dependence. Item response theory (IRT) analyses were then used to finalize the item scale and short form. RESULTS: CFA confirmed that the anger items are representative of a unidimensional scale, and items with local dependence were removed, resulting in a six-item short form. The IRT-scaled scores from summed scores and each score's conditional standard error were calculated for the new six-item PROMIS Pediatric Anger Scale. CONCLUSIONS: This study provides initial calibrations of the anger items and creates the PROMIS Pediatric Anger Scale, version 1.0.

Practical considerations in equating progress tests.

Progress testing offers a unique contribution to medical education assessment in that it provides a tool for monitoring examinees' growth over time and their progress toward graduation objectives. However, one of the most important psychometric requirements of progress testing has typically been neglected in past applications: scores across the time and test forms are not commonly placed on the same scale. Equating is a method used to achieve this property. This article discusses the basic principles of equating and the particular challenges posed by progress testing. A hybrid equating method, along with the details of a recent application, is presented as a potential solution, and alternate approaches to equating are discussed. It is hoped that future applications of progress testing will both utilize equating and demonstrate its value as a tool in medical education assessment.

An item response analysis of the pediatric PROMIS anxiety and depressive symptoms scales.

PURPOSE: The Patient-Reported Outcomes Measurement Information System (PROMIS) aims to develop self-reported item banks for clinical research. The PROMIS pediatrics (aged 8-17) project focuses on the development of item banks across several health domains (physical function, pain, fatigue, emotional distress, social role relationships, and asthma symptoms). The psychometric properties of the anxiety and depressive symptom item banks are described. METHODS: Participants (n = 1,529) were recruited in public school settings, hospital-based outpatient and subspecialty pediatrics clinics. The anxiety (k = 18) and depressive symptoms (k = 21) items were split between two test administration forms. Hierarchical confirmatory factor-analytic models (CFA) were conducted to evaluate scale dimensionality and local dependence. IRT analyses were then used to finalize item banks and short forms. RESULTS: CFA results confirmed that anxiety and depressive symptoms are separate constructs and indicative of negative affect. Items with local dependence and DIF were removed resulting in 15 anxiety and 14 depressive symptoms items. The psychometric differences between short forms and simulated computer adaptive tests are presented. CONCLUSIONS: PROMIS pediatric item banks were developed to provide efficient assessment of health-related quality of life domains. This sample provides initial calibrations of anxiety and depressive symptoms item banks and creates PROMIS pediatric instruments, version 1.0.

Item response theory detected differential item functioning between healthy and ill children in quality-of-life measures.

OBJECTIVE: To demonstrate the value of item response theory (IRT) and differential item functioning (DIF) methods in examining a health-related quality-of-life measure in children and adolescents. STUDY DESIGN AND SETTING: This illustration uses data from 5,429 children using the four subscales of the PedsQL 4.0 Generic Core Scales. The IRT model-based likelihood ratio test was used to detect and evaluate DIF between healthy children and children with a chronic condition. RESULTS: DIF was detected for a majority of items but canceled out at the total test score level due to opposing directions of DIF. Post hoc analysis indicated that this pattern of results may be due to multidimensionality. We discuss issues in detecting and handling DIF. CONCLUSION: This article describes how to perform DIF analyses in validating a questionnaire to ensure that scores have equivalent meaning across subgroups. It offers insight into ways information gained through the analysis can be used to evaluate an existing scale.

Practical issues in the application of item response theory: a demonstration using items from the pediatric quality of life inventory (PedsQL) 4.0 generic core scales.

BACKGROUND: Item response theory (IRT) is increasingly being applied to health-related quality of life instrument development and refinement. This article discusses results obtained using categorical confirmatory factor analysis (CCFA) to check IRT model assumptions and the application of IRT in item analysis and scale evaluation. OBJECTIVES: To demonstrate the value of CCFA and IRT in examining a health-related quality of life measure in children and adolescents. METHODS: This illustration uses data from 10,241 children and their parents on items from the 4 subscales of the PedsQL 4.0 Generic Core Scales. CCFA was applied to confirm domain dimensionality and identify possible locally dependent items. IRT was used to assess the strength of the relationship between the items and the constructs of interest and the information available across the latent construct. RESULTS: CCFA showed generally strong support for 1-factor models for each domain; however, several items exhibited evidence of local dependence. IRT revealed that the items generally exhibit favorable characteristics and are related to the same construct within a given domain. We discuss the lessons that can be learned by comparing alternate forms of the same scale, and we assess the potential impact of local dependence on the item parameter estimates. CONCLUSIONS: This article describes CCFA methods for checking IRT model assumptions and provides suggestions for using these methods in practice. It offers insight into ways information gained through IRT can be applied to evaluate items and aid in scale construction.