Using Simulated Annealing to Investigate Sensitivity of SEM to External Model Misspecification.

Sensitivity analyses encompass a broad set of post-analytic techniques that are characterized as measuring the potential impact of any factor that has an effect on some output variables of a model. This research focuses on the utility of the simulated annealing algorithm to automatically identify path configurations and parameter values of omitted confounders in structural equation modeling (SEM). An empirical example based on a past published study is used to illustrate how strongly related an omitted variable must be to model variables for the conclusions of an analysis to change. The algorithm is outlined in detail and the results stemming from the sensitivity analysis are discussed.

Multidimensional Computerized Adaptive Testing: A Potential Path Toward the Efficient and Precise Assessment of Applied Cognition, Daily Activity, and Mobility for Hospitalized Patients.

OBJECTIVE: To develop and evaluate an efficient and precise variable-length functional assessment of applied cognition, daily activity, and mobility to inform mobility preservation and rehabilitation service delivery among hospitalized patients. DESIGN: A multidimensional item bank tapping into these dimensions was developed, with all items calibrated using a multidimensional graded response model. The items were adaptively selected from the item banks to maximize the test information, and the test ended when a joint stopping rule was satisfied. A simulation study was conducted based on the completed instrument, the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT), to compare its measurement precision and efficiency capabilities relative to conventional unidimensional computerized adaptive testing. Precision was measured by the bias and root mean squared error between the estimated and true (ie, simulated) θ estimates, whereas efficiency was measured by average test length. Data were collected by an interviewer reading questions from a tablet computer and entering patients' responses. SETTING: A large Midwestern hospital. PARTICIPANTS: A total of 4143 patients hospitalized with medical diagnosis and/or surgical complications, with 2060 in the calibration sample and 2083 in the validation cohort. INTERVENTION: Not applicable. RESULTS: Among the 2083 patients in the validation sample, FAMCAT administration required an average of 6 (SD=3.11) minutes. Ninety-six percent had their tests terminated by the standard error rule after responding to an average of 22.05 (SD=7.98) items, whereas 15 were terminated by the change in θ rule, with an average test length of 45.27 (SD=11.49). The remaining 76 responded until reaching the maximum test length of 60 items. CONCLUSIONS: The FAMCAT has the potential to satisfy the need for structured, frequent, and precise assessment of functional domains among hospitalized patients with medical diagnosis and/or surgical complications. The results are promising and may be informative for others who wish to develop similar instruments when concurrent assessment of correlated domains is required.

Can Proxy Ratings Supplement Patient Report to Assess Functional Domains Among Hospitalized Patients?

OBJECTIVE: To (1) characterize the agreement between patient and proxy responses on a multidimensional computerized adaptive testing measure of function, and to (2) determine whether patient, proxy, or multidimensional computerized adaptive testing score characteristics identify when a proxy report can be used as a substitute for patient report in clinical decision making. DESIGN: A psychometric study of the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Testing (FAMCAT) and its 3 scales (Applied Cognition, Daily Activity, and Basic Mobility). SETTING: An upper midwestern quaternary academic medical center PARTICIPANTS: A total of 300 pairs of patients (average age 60.9 years; range, 19-89) hospitalized on general medical services or readmitted to surgical services for postoperative complications and their proxies (average age 60.5 years; range, 20-88). INTERVENTION: Not applicable. MAIN OUTCOME MEASURES: There were 3 outcomes: (1) agreement between patient and proxy scores on the FAMCAT domains, as well as age and sex, analyzed with univariate and multivariate analysis of variance (MANOVA); (2) associations of patient-proxy relationship and FAMCAT score characteristics with patient-proxy score agreement; and (3) presence of psychometrically significant intra-dyad differences in FAMCAT scores. RESULTS: The results of the MANOVA and follow-up ANOVAs indicated that there were no statistically significant differences in FAMCAT scale scores between patient and proxy estimates for either the Daily Activity or Basic Mobility scales. There were significant differences for the Applied Cognition scale (P<.005) between mean patient and proxy scores, with proxies rating patients as functioning at a higher level (mean=0.42) than patients did themselves (mean=0.00). However, psychometrically significant intra-dyadic Applied Cognition score differences occurred in only 14% of dyads, compared with 25% in the other 2 scales. Sex and age were associated with patient-proxy agreement, but the patterns were not sufficiently consistent to permit generalizations regarding the likely validity of a proxy's scores. CONCLUSIONS: Patient and proxy FAMCAT Daily Activity and Basic Mobility scores did not differ significantly, and proxy reporting offers a creditable surrogate for patient report on these domains. Low rates of psychometrically significant intra-dyadic score differences suggest that proxy report may serve as a low-resolution screen for functional deficits in all FAMCAT domains. Approximately half the proxies provided multi-domain profile ratings on the 3 scales that did not differ significantly from these of the associated patients, but more research is needed to identify situations in which proxy profiles could be used in place of those provided by patients.

Does the Mode of PROM Administration Affect the Responses of Hospitalized Patients?

OBJECTIVE: To determine whether a multidimensional computerized adaptive test, the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT), could be administered to hospitalized patients via a tablet computer rather than being orally administered by an interviewer. DESIGN: A randomized comparison of the responses of hospitalized patients to interviewer vs tablet delivery of the FAMCAT and its assessment of applied cognition, daily activity, and basic mobility. SETTING: Two quaternary teaching hospitals in the Upper Midwest. PARTICIPANTS: A total of 300 patients (127 men, 165 women), average age 61.2 (range, 18-97) hospitalized on medical services or rehospitalized on surgical services were randomly assigned to either a tablet (150) or an interview (150) group. INTERVENTION: Electronic tablet vs interview. MAIN OUTCOME MEASURES: Item response theory point estimates of the FAMCAT latent scales, their psychometric standard errors, number of items administered per domain, the determinant (an indicator of overall precision of the latent trait vector), as well as the time that patients required to complete their FAMCAT sessions. RESULTS: Of the 300 patients, 292 completed their assessments. The assessments of 4 individuals in each group was interrupted by clinical care and were not included in the analyses. A significant (P=.009) mode effect (ie, interview vs tablet) was identified when all outcome variables were considered simultaneously. However, the only outcome that was affected by the administration mode was test duration: tablet administration reduced the roughly 6-minute test time required by both approaches by only 20 seconds, which, though statistically significant, was clinically insignificant. CONCLUSIONS: The results of a FAMCAT assessment, at least for this cohort of hospitalized patients, are independent of administration via tablet computer or interview.

Hypothesis Testing Methods for Multivariate Multi-Occasion Intra-Individual Change.

In psychological and educational measurement, it is often of interest to assess change in an individual. The current study expanded on previous research by introducing methods that can evaluate individual change on multiple latent traits measured on multiple occasions. The four methods considered are the likelihood ratio test (LRT), the multivariate Wald test (MWT), the modified multivariate Wald test (MMWT), and the score test (ST). Simulation studies were conducted to examine the true positive rate (TPR) and the false positive rate (FPR) of the new methods under a conventional fixed-form test and a computerized adaptive test (CAT). Manipulated variables included the number of occasions, change magnitudes, patterns of change, and correlations between latent traits. Results revealed that, in terms of FPR, all methods except MWT had close adherence to the nominal significance level. Among the three methods, the LRT is recommended as it provided a balance between FPR and TPR. Larger change magnitude yielded higher TPR, regardless of the remaining factors. With the same test length, a CAT yielded higher TPR than a conventional test. Real-data examples are provided of identifying psychometrically significant change across two to four occasions using a multivariate adaptive self-report medical outcomes measure from hospitalized patients. The detection of significant change among the three methods agreed highly, and those patients identified as having significant change exhibited large profile differences, which provided support for the valid performance of the proposed methods.