Validation of the cognitive section of the Penn computerized adaptive test for neurocognitive and clinical psychopathology assessment (CAT-CCNB).

BACKGROUND: The Penn Computerized Neurocognitive Battery is an efficient tool for assessing brain-behavior domains, and its efficiency was augmented via computerized adaptive testing (CAT). This battery requires validation in a separate sample to establish psychometric properties. METHODS: In a mixed community/clinical sample of N = 307 18-to-35-year-olds, we tested the relationships of the CAT tests with the full-form tests. We compared discriminability among recruitment groups (psychosis, mood, control) and examined how their scores relate to demographics. CAT-Full relationships were evaluated based on a minimum inter-test correlation of 0.70 or an inter-test correlation within at least 0.10 of the full-form correlation with a previous administration of the full battery. Differences in criterion relationships were tested via mixed models. RESULTS: Most tests (15/17) met the minimum criteria for replacing the full-form with the updated CAT version (mean r = 0.67; range = 0.53-0.80) when compared to relationships of the full-forms with previous administrations of the full-forms (mean r = 0.68; range = 0.50-0.85). Most (16/17) CAT-based relationships with diagnostics and other validity criteria were indistinguishable (interaction p > 0.05) from their full-form counterparts. CONCLUSIONS: The updated CNB shows psychometric properties acceptable for research. The full-forms of some tests should be retained due to insufficient time savings to justify the loss in precision.

Computerized adaptive testing to screen pre-school children for emotional and behavioral problems.

Questionnaires to detect emotional and behavioral (EB) problems in preventive child healthcare (PCH) should be short; this potentially affects their validity and reliability. Computerized adaptive testing (CAT) could overcome this weakness. The aim of this study was to (1) develop a CAT to measure EB problems among pre-school children and (2) assess the efficiency and validity of this CAT. We used a Dutch national dataset obtained from parents of pre-school children undergoing a well-child care assessment by PCH (n = 2192, response 70%). Data regarded 197 items on EB problems, based on four questionnaires, the Strengths and Difficulties Questionnaire (SDQ), the Child Behavior Checklist (CBCL), the Ages and Stages Questionnaire: Social Emotional (ASQ:SE), and the Brief Infant-Toddler Social and Emotional Assessment (BITSEA). Using 80% of the sample, we calculated item parameters necessary for a CAT and defined a cutoff for EB problems. With the remaining part of the sample, we used simulation techniques to determine the validity and efficiency of this CAT, using as criterion a total clinical score on the CBCL. Item criteria were met by 193 items. This CAT needed, on average, 16 items to identify children with EB problems. Sensitivity and specificity compared to a clinical score on the CBCL were 0.89 and 0.91, respectively, for total problems; 0.80 and 0.93 for emotional problems; and 0.94 and 0.91 for behavioral problems.    Conclusion: A CAT is very promising for the identification of EB problems in pre-school children, as it seems to yield an efficient, yet high-quality identification. This conclusion should be confirmed by real-life administration of this CAT. What is Known: • Studies indicate the validity of using computerized adaptive test (CAT) applications to identify emotional and behavioral problems in school-aged children. • Evidence is as yet limited on whether CAT applications can also be used with pre-school children. What is New: • The results of this study show that a computerized adaptive test is very promising for the identification of emotional and behavior problems in pre-school children, as it appears to yield an efficient and high-quality identification.

Initial Validation of a Computerized Adaptive Test for Substance Use Disorder Identification in Adolescents.

PURPOSE: Computerized adaptive tests (CATs) are highly efficient assessment tools that couple low patient and clinician time burden with high diagnostic accuracy. A CAT for substance use disorders (CAT-SUD-E) has been validated in adult populations but has yet to be tested in adolescents. The purpose of this study was to perform initial evaluation of the K-CAT-SUD-E (i.e., Kiddy-CAT-SUD-E) in an adolescent sample compared to a gold-standard diagnostic interview. METHODS: Adolescents (N = 156; aged 11-17) with diverse substance use histories completed the K-CAT-SUD-E electronically and the substance related disorders portion of a clinician-conducted diagnostic interview (K-SADS) via tele-videoconferencing platform. The K-CAT-SUD-E assessed both current and lifetime overall SUD and substance-specific diagnoses for nine substance classes. RESULTS: Using the K-CAT-SUD-E continuous severity score and diagnoses to predict the presence of any K-SADS SUD diagnosis, the classification accuracy ranged from excellent for current SUD (AUC = 0.89, 95% CI = 0.81, 0.95) to outstanding (AUC = 0.93, 95% CI = 0.82, 0.97) for lifetime SUD. Regarding current substance-specific diagnoses, the classification accuracy was excellent for alcohol (AUC = 0.82), cannabis (AUC = 0.83) and nicotine/tobacco (AUC = 0.90). For lifetime substance-specific diagnoses, the classification accuracy ranged from excellent (e.g., opioids, AUC = 0.84) to outstanding (e.g., stimulants, AUC = 0.96). K-CAT-SUD-E median completion time was 4 min 22 s compared to 45 min for the K-SADS. CONCLUSIONS: This study provides initial support for the K-CAT-SUD-E as a feasible accurate diagnostic tool for assessing SUDs in adolescents. Future studies should further validate the K-CAT-SUD-E in a larger sample of adolescents and examine its acceptability, feasibility, and scalability in youth-serving settings.

Reliability, validity, and simplification of the Chinese version of the Global Pain Scale in patients with rheumatoid arthritis.

BACKGROUND: Persistent pain is the most reported symptom in patients with rheumatoid arthritis (RA); however, effective and brief assessment tools are lacking. We validated the Chinese version of the Global Pain Scale (C-GPS) in Chinese patients with RA and proposed a short version of the C-GPS (s-C-GPS). METHOD: The study was conducted using a face-to-face questionnaire survey with a multicenter cross-sectional design from March to December 2019. Patients aged > 18 years who met the RA diagnostic criteria were included. Based on the classical test theory (CTT) and the item response theory (IRT), we assessed the validity and reliability of the C-GPS and the adaptability of each item. An s-C-GPS was developed using IRT-based computerized adaptive testing (CAT) analytics. RESULTS: In total, 580 patients with RA (mean age, 51.04 ± 24.65 years; mean BMI, 22.36 ± 4.07 kg/m2), including 513 (88.4%) women, were included. Most participants lived in a suburb (49.3%), were employed (72.2%) and married (91.2%), reported 9-12 years of education (66.9%), and had partial medical insurance (57.8%). Approximately 88.1% smoked and 84.5% drank alcohol. Analysis of the CTT demonstrated that all items in the C-GPS were positively correlated with the total scale score, and the factor loadings of all these items were > 0.870. A significant positive relationship was found between the Visual Analog Scale (VAS) and the C-GPS. IRT analysis showed that discrimination of the C-GPS was between 2.271 and 3.312, and items 6, 8, 13, 14, and 16 provided a large amount of information. Based on the CAT and clinical practice, six items covering four dimensions were included to form the s-C-GPS, all of which had very high discrimination. The s-C-GPS positively correlated with the VAS. CONCLUSION: The C-GPS has good reliability and validity and can be used to evaluate pain in RA patients from a Chinese cultural background. The s-C-GPS, which contains six items, has good criterion validity and may be suitable for pain assessment in busy clinical practice. TRIAL REGISTRATION: This cross-sectional study was registered in the Chinese Clinical Trial Registry (ChiCTR1800020343), granted on December 25, 2018.

Just-in-time adaptive ecological momentary assessment (JITA-EMA).

Interest in just-in-time adaptive interventions (JITAI) has rapidly increased in recent years. One core challenge for JITAI is the efficient and precise measurement of tailoring variables that are used to inform the timing of momentary intervention delivery. Ecological momentary assessment (EMA) is often used for this purpose, even though EMA in its traditional form was not designed specifically to facilitate momentary interventions. In this article, we introduce just-in-time adaptive EMA (JITA-EMA) as a strategy to reduce participant response burden and decrease measurement error when EMA is used as a tailoring variable in JITAI. JITA-EMA builds on computerized adaptive testing methods developed for purposes of classification (computerized classification testing, CCT), and applies them to the classification of momentary states within individuals. The goal of JITA-EMA is to administer a small and informative selection of EMA questions needed to accurately classify an individual's current state at each measurement occasion. After illustrating the basic components of JITA-EMA (adaptively choosing the initial and subsequent items to administer, adaptively stopping item administration, accommodating dynamically tailored classification cutoffs), we present two simulation studies that explored the performance of JITA-EMA, using the example of momentary fatigue states. Compared with conventional EMA item selection methods that administered a fixed set of questions at each moment, JITA-EMA yielded more accurate momentary classification with fewer questions administered. Our results suggest that JITA-EMA has the potential to enhance some approaches to mobile health interventions by facilitating efficient and precise identification of momentary states that may inform intervention tailoring.

Methods for online calibration of Q-matrix and item parameters for polytomous responses in cognitive diagnostic computerized adaptive testing.

The ability to rapidly provide examinees with detailed and effective diagnostic information is a critical topic in psychology. Knowing what diagnostic criteria the examinees have met enables the practitioner to seek the solution to help them in a timely manner, and this can be achieved by cognitive diagnostic computerized adaptive testing (CD-CAT). However, the pervasive challenge of replenishing items in the CD-CAT item bank limits its practical application. Online calibration is a means to address item replenishment, but in CD-CAT, most existing online calibration methods that jointly calibrate the Q-matrix and item parameters of the new items are developed only for dichotomous responses and are time-consuming. Notably, previous studies pay no attention to polytomously scored items that are frequently observed in testing, even though they can offer additional evidence for the examinees' diagnosis. To fill this gap, we propose a SCAD-based method (SCAD-EM) to calibrate the Q-matrix and item parameters of the new items with polytomous response data in order to promote the application of CD-CAT in practice. The performance of the SCAD-EM was investigated in two comprehensive simulation studies and compared against the revised single-item estimation method (SIE-BIC). Results indicated that the SCAD-EM produces a higher calibration accuracy for the category-level Q-matrix and is computationally more efficient across all conditions, but it produces a lower calibration accuracy for the item-level Q-matrix. An empirical study further demonstrated the utility of the SCAD-EM and the SIE-BIC methods in calibrating new items with a real dataset. The advantages of the proposed method, its limitations, and possible future research directions are offered at the end.

Development and psychometric evaluation of item banks for memory and attention - supplements to the EORTC CAT Core instrument.

BACKGROUND: Cancer patients may experience a decrease in cognitive functioning before, during and after cancer treatment. So far, the Quality of Life Group of the European Organisation for Research and Treatment of Cancer (EORTC QLG) developed an item bank to assess self-reported memory and attention within a single, cognitive functioning scale (CF) using computerized adaptive testing (EORTC CAT Core CF item bank). However, the distinction between different cognitive functions might be important to assess the patients' functional status appropriately and to determine treatment impact. To allow for such assessment, the aim of this study was to develop and psychometrically evaluate separate item banks for memory and attention based on the EORTC CAT Core CF item bank. METHODS: In a multistep process including an expert-based content analysis, we assigned 44 items from the EORTC CAT Core CF item bank to the memory or attention domain. Then, we conducted psychometric analyses based on a sample used within the development of the EORTC CAT Core CF item bank. The sample consisted of 1030 cancer patients from Denmark, France, Poland, and the United Kingdom. We evaluated measurement properties of the newly developed item banks using confirmatory factor analysis (CFA) and item response theory model calibration. RESULTS: Item assignment resulted in 31 memory and 13 attention items. Conducted CFAs suggested good fit to a 1-factor model for each domain and no violations of monotonicity or indications of differential item functioning. Evaluation of CATs for both memory and attention confirmed well-functioning item banks with increased power/reduced sample size requirements (for CATs ≥ 4 items and up to 40% reduction in sample size requirements in comparison to non-CAT format). CONCLUSION: Two well-functioning and psychometrically robust item banks for memory and attention were formed from the existing EORTC CAT Core CF item bank. These findings could support further research on self-reported cognitive functioning in cancer patients in clinical trials as well as for real-word-evidence. A more precise assessment of attention and memory deficits in cancer patients will strengthen the evidence on the effects of cancer treatment for different cancer entities, and therefore contribute to shared and informed clinical decision-making.

Measuring glaucoma quality of life in an Asian population using item banking: psychometric evaluation and computerized adaptive testing simulations.

PURPOSE: To assess the psychometric properties of glaucoma-specific health-related quality of life (HRQoL) item banks (IBs), and explore their efficiency using computerized adaptive testing (CAT) simulations. METHODS: In this cross-sectional, clinical study, 300 Asian glaucoma patients answered 221 items within seven IBs: Ocular Comfort Symptoms (OS); Activity Limitation (AL); Lighting (LT); Mobility (MB); Glaucoma Management (GM); Psychosocial (PSY); and Work (WK). Rasch analysis was conducted to assess each IB's psychometric properties (e.g., item "fit" to the construct; unidimensionality) and a set of analytic performance criteria guiding decision making relating to retaining or dropping domains and items was employed. CAT simulations determined the mean number of items for 'high' and 'moderate' measurement precision (stopping rule: SEM 0.3 and 0.387, respectively). RESULTS: Participants' mean age was 67.2 ± 9.2 years (62% male; 87% Chinese). LT, MB, and GM displayed good psychometric properties overall. To optimize AL's psychometric properties, 16 items were deleted due to poor "fit", high missing data, item bias, low discrimination and/or a low clinical/patient importance rating. To resolve multidimensionality in PSY, we rehomed 16 items into a "Concern (CN)" domain. PSY and CN required further amendment, including collapsing of response categories, and removal of poorly functioning items (N = 7). Due to poor measurement precision, low applicability and high ceiling effect, low test information indices, and low item separation index the WK IB was not considered further. In CAT simulations on the final seven IBs (n = 182 items total), an average of 12.1 and 15.7 items per IB were required for moderate and high precision measurement, respectively. CONCLUSIONS: After reengineering our seven IBs, they displayed robust psychometric properties and good efficiency in CAT simulations. Once finalized, GlauCAT™-Asian may enable comprehensive assessment of the HRQoL impact of glaucoma and associated treatments.

Test-Retest Reliability and Responsiveness of the Computerized Adaptive Testing System of the Functional Assessment of Stroke.

OBJECTIVE: To examine the test-retest reliability, responsiveness, and clinical utility of the Computerized Adaptive Testing System of the Functional Assessment of Stroke (CAT-FAS) in persons with stroke. DESIGN: Repeated measurements design. SETTING: A department of rehabilitation in a medical center. PARTICIPANTS: 30 persons with chronic stroke (for test-retest reliability) and 65 persons with subacute stroke (for responsiveness) were recruited. To examine the test-retest reliability, the participants received measurements twice at 1-month intervals. To examine the responsiveness, the data were collected at admission and discharge from hospital. INTERVENTIONS: Not applicable. MAIN OUTCOME MEASUREMENT TOOL: CAT-FAS. RESULTS: The intra-class correlation coefficients of the CAT-FAS were ≥0.82, indicating good to excellent test-retest reliability. The Kazis' effect size and standardized response mean of the CAT-FAS were ≥0.96, indicating good group-level responsiveness. For individual-level responsiveness, approximately two-thirds of the participants exceeded the conditional minimal detectable change. On average, the CAT-FAS was completed within 9 items and 3 minutes per administration. CONCLUSIONS: Our results suggest the CAT-FAS is an efficient measurement tool with good to excellent test-retest reliability and responsiveness. In addition, the CAT-FAS can be used routinely in clinical settings to monitor progress of the crucial 4 domains for persons with stroke.

Remote Symptom Monitoring With Ecological Momentary Computerized Adaptive Testing: Pilot Cohort Study of a Platform for Frequent, Low-Burden, and Personalized Patient-Reported Outcome Measures.

BACKGROUND: Remote patient-reported outcome measure (PROM) data capture can provide useful insights into research and clinical practice and deeper insights can be gained by administering assessments more frequently, for example, in ecological momentary assessment. However, frequent data collection can be limited by the burden of multiple, lengthy questionnaires. This burden can be reduced with computerized adaptive testing (CAT) algorithms that select only the most relevant items from a PROM for an individual respondent. In this paper, we propose "ecological momentary computerized adaptive testing" (EMCAT): the use of CAT algorithms to reduce PROM response burden and facilitate high-frequency data capture via a smartphone app. We develop and pilot a smartphone app for performing EMCAT using a popular hand surgery PROM. OBJECTIVE: The aim of this study is to determine the feasibility of EMCAT as a system for remote PROM administration. METHODS: We built the EMCAT web app using Concerto, an open-source CAT platform maintained by the Psychometrics Centre, University of Cambridge, and hosted it on an Amazon Web Service cloud server. The platform is compatible with any questionnaire that has been parameterized with item response theory or Rasch measurement theory. For this study, the PROM we chose was the patient evaluation measure, which is commonly used in hand surgery. CAT algorithms were built using item response theory models derived from UK Hand Registry data. In the pilot study, we enrolled 40 patients with hand trauma or thumb-base arthritis, across 2 sites, between July 13, 2022, and September 14, 2022. We monitored their symptoms with the patient evaluation measure, via EMCAT, over a 12-week period. Patients were assessed thrice weekly, once daily, or thrice daily. We additionally administered full-length PROM assessments at 0, 6, and 12 weeks, and the User Engagement Scale at 12 weeks. RESULTS: The use of EMCAT significantly reduced the length of the PROM (median 2 vs 11 items) and the time taken to complete it (median 8.8 seconds vs 1 minute 14 seconds). Very similar scores were obtained when EMCAT was administered concurrently with the full-length PROM, with a mean error of <0.01 on a logit (z score) scale. The median response rate in the daily assessment group was 93%. The median perceived usability score of the User Engagement Scale was 4.0 (maximum possible score 5.0). CONCLUSIONS: EMCAT reduces the burden of PROM assessments, enabling acceptable high-frequency, remote PROM data capture. This has potential applications in both research and clinical practice. In research, EMCAT could be used to study temporal variations in symptom severity, for example, recovery trajectories after surgery. In clinical practice, EMCAT could be used to monitor patients remotely, prompting early intervention if a patient's symptom trajectory causes clinical concern. TRIAL REGISTRATION: ISRCTN 19841416; https://www.isrctn.com/ISRCTN19841416.

Novel item selection strategies for cognitive diagnostic computerized adaptive testing: A heuristic search framework.

The computerized adaptive form of cognitive diagnostic testing, CD-CAT, has gained increasing attention in the domain of personalized measurements for its ability to categorize individual mastery status of fine-grained attributes more accurately and efficiently through administering items tailored to one's ability progressively. How to select the next item based on previous response(s) is crucial for the success of CD-CAT. Previous item selection strategies for CD-CAT have often followed a greedy or semi-greedy approach, which makes it difficult to strike a balance between diagnostic performance and item bank utilization. To address this issue, this study takes a graph perspective and transforms the item selection problem in CD-CAT into a path-searching problem, in which paths refer to possible test construction and nodes refer to individual items. A heuristic function is defined to predict the prospect of a path, indicating how well the corresponding test can diagnose the current examinee. Two search mechanisms with different biases towards item exposure control are proposed to approximate the optimal path with the best prospect. The first unused item on the resulting path is selected as the next item. The above components compose a novel CD-CAT item selection framework based on heuristic search. Simulation studies are conducted under a variety of conditions regarding bank designs, bank-quality conditions, and testing scenarios. The results are compared with different types of classic item selection strategies in CD-CAT, showing that the proposed framework can enhance bank utilization at a smaller cost of diagnostic performance.

Determinants of higher education teachers' intention to use technology-based exams.

The replacement of existing technology or the introduction of novel technology into the day-to-day routines of higher education institutions is not a trivial task. Currently, many higher education institutions are faced with the challenge of replacing existing procedures for administering written exams with e-exams. To guide this process, this paper proposes the novel technology-based exams acceptance model (TEAM) and empirically evaluates its model structure and usefulness from the perspective of higher education teachers. The model can be used to guide the transition from paper-based exams to e-exams and the implementation of innovative (e.g., adaptive) e-exam formats. The model includes perceived usefulness, computer self-efficacy, computer anxiety, prior experience, facilitating conditions, and subjective norm as predictors of the behavioral intention to use e-exams. To test the model empirically, the responses of 992 teachers at 63 German universities to a standardized online questionnaire were analyzed using structural equation modeling. The model fit was acceptable. With 77% (conventional e-exams) and 82% (adaptive e-exams), a large proportion of the variance of the intention to use these types of exams was explained. With TEAM, a highly predictive model for explaining the behavioral intention to use e-exams is now available. It offers a theoretical basis that can be used for the successful implementation of e-exams in higher education.

Reducing the question burden of patient reported outcome measures using Bayesian networks.

Patient Reported Outcome Measures (PROMs) are questionnaires completed by patients about aspects of their health status. They are a vital part of learning health systems as they are the primary source of information about important outcomes that are best assessed by patients such as pain, disability, anxiety and depression. The volume of questions can easily become burdensome. Previous techniques reduced this burden by dynamically selecting questions from question item banks which are specifically built for different latent constructs being measured. These techniques analyzed the information function between each question in the item bank and the measured construct based on item response theory then used this information function to dynamically select questions by computerized adaptive testing. Here we extend those ideas by using Bayesian Networks (BNs) to enable Computerized Adaptive Testing (CAT) for efficient and accurate question selection on widely-used existing PROMs. BNs offer more comprehensive probabilistic models of the connections between different PROM questions, allowing the use of information theoretic techniques to select the most informative questions. We tested our methods using five clinical PROM datasets, demonstrating that answering a small subset of questions selected with CAT has similar predictions and error to answering all questions in the PROM BN. Our results show that answering 30% - 75% questions selected with CAT had an average area under the receiver operating characteristic curve (AUC) of 0.92 (min: 0.8 - max: 0.98) for predicting the measured constructs. BNs outperformed alternative CAT approaches with a 5% (min: 0.01% - max: 9%) average increase in the accuracy of predicting the responses to unanswered question items.

Identifying the content for an item bank and computerized adaptive testing system to measure the impact of age-related macular degeneration on health-related quality of life.

PURPOSE: We are developing an age-related macular degeneration (AMD) health-related quality of life (HRQoL) item bank, applicable to Western and Asian populations. We report primarily on content generation and refinement, but also compare the HRQoL issues reported in our study with Western studies and current AMD-HRQoL questionnaires. METHODS: In this cross-sectional, qualitative study of AMD patients attending the Singapore National Eye Centre (May-December 2019), items/domains were generated from: (1) AMD-specific questionnaires; (2) published articles; (3) focus groups/semi-structured interviews with AMD patients (n = 27); and (4) written feedback from retinal experts. Following thematic analysis, items were systematically refined to a minimally representative set and pre-tested using cognitive interviews with 16 AMD patients. RESULTS: Of the 27 patients (mean ± standard deviation age 67.9 ± 7.0; 59.2% male), 18 (66.7%), two (7.4%), and seven (25.9%) had no, early-intermediate, and late/advanced AMD (better eye), respectively. Whilst some HRQoL issues, e.g. activity limitation, mobility, lighting, and concerns were similarly reported by Western patients and covered by other questionnaires, others like anxiety about intravitreal injections, work tasks, and financial dependency were novel. Overall, 462 items within seven independent HRQoL domains were identified: Activity limitation, Lighting, Mobility, Emotional, Concerns, AMD management, and Work. Following item refinement, items were reduced to 219, with 31 items undergoing amendment. CONCLUSION: Our 7-domain, 219-item AMD-specific HRQoL instrument will undergo psychometric testing and calibration for computerized adaptive testing. The future instrument will enable users to precisely, rapidly, and comprehensively quantify the HRQoL impact of AMD and associated treatments, with item coverage relevant across several populations.

Efficient and precise Ultra-QuickDASH scale measuring lymphedema impact developed using computerized adaptive testing.

PURPOSE: This study aimed to evaluate and improve the accuracy and efficiency of the QuickDASH for use in assessment of limb function in patients with upper extremity lymphedema using modern psychometric techniques. METHOD: We conducted confirmative factor analysis (CFA) and Mokken analysis to examine the assumption of unidimensionality for IRT model on data from 285 patients who completed the QuickDASH, and then fit the data to Samejima's graded response model (GRM) and assessed the assumption of local independence of items and calibrated the item responses for CAT simulation. RESULTS: Initial CFA and Mokken analyses demonstrated good scalability of items and unidimensionality. However, the local independence of items assumption was violated between items 9 (severity of pain) and 11 (sleeping difficulty due to pain) (Yen's Q3 = 0.46) and disordered thresholds were evident for item 5 (cutting food). After addressing these breaches of assumptions, the re-analyzed GRM with the remaining 10 items achieved an improved fit. Simulation of CAT administration demonstrated a high correlation between scores on the CAT and the QuickDash (r = 0.98). Items 2 (doing heavy chores) and 8 (limiting work or daily activities) were the most frequently used. The correlation among factor scores derived from the QuickDASH version with 11 items and the Ultra-QuickDASH version with items 2 and 8 was as high as 0.91. CONCLUSION: By administering just these two best performing QuickDash items we can obtain estimates that are very similar to those obtained from the full-length QuickDash without the need for CAT technology.

Adaptive Measurement of Change: A Novel Method to Reduce Respondent Burden and Detect Significant Individual-Level Change in Patient-Reported Outcome Measures.

OBJECTIVE: To describe the adaptive measurement of change (AMC) as a means to identify psychometrically significant change in reported function of hospitalized patients and to reduce respondent burden on follow-up assessments. DESIGN: The AMC method uses multivariate computerized adaptive testing (CAT) and psychometric hypothesis tests based in item response theory to more efficiently measure intra-individual change using the responses of a single patient over 2 or more testing occasions. Illustrations of the utility of AMC in clinical care and estimates of AMC-based item reduction are provided using the Functional Assessment in Acute Care Multidimensional Computerized Adaptive Test (FAMCAT), a newly developed functional multidimensional CAT-based measurement of basic mobility, daily activities, and applied cognition. SETTING: Two quaternary hospitals in the Upper Midwest. PARTICIPANTS: Four hundred ninety-five hospitalized patients who completed the FAMCAT on 2 to 4 occasions during their hospital stay. INTERVENTION: N/A. RESULTS: Of the 495 patients who completed more than 1 FAMCAT, 72% completed 2 sessions, 13% completed 3, and 15% completed 4, with 22.1%, 23.4%, and 23.0%, respectively, exhibiting significant multivariate change. Use of the AMC in conjunction with the FAMCAT reduced respondent burden from that of the FAMCAT alone for follow-up assessments. On average, when used without the AMC, 22.7 items (range, 20.4-24.4) were administered during FAMCAT sessions. Post hoc analyses determined that when the AMC was used with the FAMCAT a mean±standard deviation reduction in FAMCAT number of items of 13.6 (11.1), 13.1 (9.8), and 18.1 (10.8) would occur during the second, third, and fourth sessions, respectively, which corresponded to a reduction in test duration of 3.0 (2.4), 3.0 (2.8), and 4.7 (2.6) minutes. Analysis showed that the AMC requires no assumptions about the nature of change and provides data that are potentially actionable for patient care. Various patterns of significant univariate and multivariate change are illustrated. CONCLUSIONS: The AMC method is an effective and parsimonious approach to identifying significant change in patients' measured CAT scores. The AMC approach reduced FAMCAT sessions by an average of 12.6 items (55%) and 2.9 minutes (53%) among patients with psychometrically significant score changes.

Development of an Adaptive Computer-Aided Soft Sensor Diagnosis System for Assessment of Executive Functions.

The main objective of the present study is to highlight the role of technological (soft sensor) methodologies in the assessment of the neurocognitive dysfunctions specific to neurodevelopmental disorders (for example, autism spectrum disorder (ASD), attention deficit hyperactivity disorder (ADHD), and specific learning disorder). In many cases neurocognitive dysfunctions can be detected in neurodevelopmental disorders, some of them having a well-defined syndrome-specific clinical pattern. A number of evidence-based neuropsychological batteries are available for identifying these domain-specific functions. Atypical patterns of cognitive functions such as executive functions are present in almost all developmental disorders. In this paper, we present a novel adaptation of the Tower of London Test, a widely used neuropsychological test for assessing executive functions (in particular planning and problem-solving). Our version, the Tower of London Adaptive Test, is based on computer adaptive test theory (CAT). Adaptive testing using novel algorithms and parameterized task banks allows the immediate evaluation of the participant's response which in turn determines the next task's difficulty level. In this manner, the subsequent item is adjusted to the participant's estimated capability. The adaptive procedure enhances the original test's diagnostic power and sensitivity. By measuring the targeted cognitive capacity and its limitations more precisely, it leads to more accurate diagnoses. In some developmental disorders (e.g., ADHD, ASD) it could be very useful in improving the diagnosis, planning the right interventions, and choosing the most suitable assistive digital technological service.

Psychometric properties of the Patient-Reported Outcomes Measurement Information System (PROMIS®) pediatric item bank peer relationships in the Dutch general population.

PURPOSE: This study aimed to validate the PROMIS Pediatric item bank v2.0 Peer Relationships and compare reliability of the full item bank to its short form, computerized adaptive test (CAT) and the social functioning (SF) subscale of the Pediatric Quality of Life Inventory (PedsQL™). METHODS: Children aged 8-18 (n = 1327), representative of the Dutch population completed the Peer Relationships item bank. A graded response model (GRM) was fit to the data. Structural validity was assessed by checking item-fit statistics (S-X2, p < 0.001 = misfit). For construct validity, a moderately strong correlation (> 0.50) was expected between Peer Relationships and the PedsQL SF subscale. Cross-cultural DIF between U.S. and NL was assessed using logistic regression, where an item with McFadden's pseudo R2 > 0.02 was considered to have DIF. Percentage of participants reliably measured was assessed using the standard error of measurement (SEM) < 0.32 as a criterion (reliability of 0.90). Relative efficiency ((1-SEM2)/nitems) was calculated to compare how well the instruments performed relative to the amount of items administered. RESULTS: In total, 527 (response rate: 39.7%) children completed the PROMIS v2.0 Peer Relationships item bank (nitems = 15) and the PedsQL™ (nitems = 23). Structural validity of the Peer Relationships item bank was sufficient, but one item displayed misfit in the GRM model (S-X2 < 0.001); 5152R1r ("I played alone and kept to myself"). The item 733R1r ("I was a good friend") was the only item that displayed cross-cultural DIF (R2 = 0.0253). The item bank correlated moderately high (r = 0.61) with the PedsQL SF subscale Reliable measurements were obtained at the population mean and > 2SD in the clinically relevant direction. CAT outperformed all other measures in efficiency. Mean T-score of the Dutch general population was 46.9(SD 9.5). CONCLUSION: The pediatric PROMIS Peer Relationships item bank was successfully validated for use within the Dutch population and reference data are now available.

Improving Efficiency of Patient-Reported Outcome Collection: Application of Computerized Adaptive Testing to DASH and QuickDASH Outcome Scores.

PURPOSE: Patient-reported outcome measures assess health status and treatment outcomes in orthopedic care, but they may burden patients with lengthy questionnaires. Predictive models using machine learning, known as computerized adaptive testing (CAT), offer a potential solution. This study evaluates the ability of CAT to improve efficiency of the 30-item Disabilities of the Arm, Shoulder, and Hand (DASH) and 11-item QuickDASH questionnaires. METHODS: A total of 2,860 DASH and 27,355 QuickDASH respondents were included in the analysis. The CAT system was retrospectively applied to each set of patient responses stored on the instrument to calculate a CAT-specific score for all DASH and QuickDASH entries. The accuracy of the CAT scores, viewed in the context of the minimal clinically important difference for both patient-reported outcome measures (DASH, 12; QuickDASH, 9), was determined through descriptive statistics, Pearson correlation coefficient, intraclass correlation coefficient, and distribution of scores and score differences. RESULTS: The CAT model required an average of 15.3 questions to be answered for the DASH and 5.8 questions for the QuickDASH, representing a 49% and 47% decrease in question burden, respectively. Mean CAT score was the same for DASH and 0.1 points lower for QuickDASH with similar SDs (DASH, 12.9 ± 19.8 vs 12.9 ± 19.9; QuickDASH, 32.7 ± 24.7 vs 32.6 ± 24.6). Pearson coefficients (DASH, 0.99; QuickDASH, 0.98) and intraclass correlation coefficients (DASH, 1.0; QuickDASH, 0.98) indicated strong agreement between scores. The difference between the CAT and full score was less than the minimal clinically important difference in 99% of cases for DASH and approximately 95% of cases for QuickDASH. CONCLUSIONS: The application of CAT to DASH and QuickDASH surveys demonstrated an ability to lessen the response burden with negligible effect on score integrity. CLINICAL RELEVANCE: In the case of DASH and QuickDASH, CAT is an appropriate alternative to full questionnaire implementation for patient outcome score collection.

Hypothesis Testing Methods for Multivariate Multi-Occasion Intra-Individual Change.

In psychological and educational measurement, it is often of interest to assess change in an individual. The current study expanded on previous research by introducing methods that can evaluate individual change on multiple latent traits measured on multiple occasions. The four methods considered are the likelihood ratio test (LRT), the multivariate Wald test (MWT), the modified multivariate Wald test (MMWT), and the score test (ST). Simulation studies were conducted to examine the true positive rate (TPR) and the false positive rate (FPR) of the new methods under a conventional fixed-form test and a computerized adaptive test (CAT). Manipulated variables included the number of occasions, change magnitudes, patterns of change, and correlations between latent traits. Results revealed that, in terms of FPR, all methods except MWT had close adherence to the nominal significance level. Among the three methods, the LRT is recommended as it provided a balance between FPR and TPR. Larger change magnitude yielded higher TPR, regardless of the remaining factors. With the same test length, a CAT yielded higher TPR than a conventional test. Real-data examples are provided of identifying psychometrically significant change across two to four occasions using a multivariate adaptive self-report medical outcomes measure from hospitalized patients. The detection of significant change among the three methods agreed highly, and those patients identified as having significant change exhibited large profile differences, which provided support for the valid performance of the proposed methods.